We Keep Coding

Converting SQL statement from LIMIT to TOP C#

I am working on an application that was implemented using a SQLite database. I am currently in the process of adding the ability to use MSSQL as well. The complicated part is that it will need to be able to use either engine depending on the needs. There are a handful of different syntax differences that have to be accounted for. The biggest problem child I have come across is LIMIT vs TOP. I have written some logic to convert the SQLite statements into the proper format for MSSQL. My function for converting the LIMIT to TOP seems to be working, but it ended up being pretty ugly. I wanted to post it here and see if anyone had ideas for a cleaner method of completing this. I also wanted to see if anyone noticed any glaring issues that I have missed. The biggest problem I ran into is the possibility of nested select statements with possible LIMIT statements on them as well. I ended up pulling the statement apart into its individual parts, changing them from LIMIT to TOP, and then rebuilding the statement. There might even be an overall better way to do this that I am missing. Thanks ahead of time if you spend the time to take a look.
private static string ConvertLimitToTop(string commandText)
{
string processCommand = commandText;
int start = -1;
List<string> commandParts = new List<string>();
//Running through the string looking for nested statemets starting with (
for (int i = 0; i < processCommand.Length; i++)
{
//Any time we find a new open ( we want to start there
if (processCommand[i] == '(')
start = i;
//If we find a close ) we will grab the nested statment and replace it
if (processCommand[i] == ')')
{
//Grab the 3 parts of the string
string preString = processCommand.Substring(0, start);
string nestedCommand = processCommand.Substring(start, i - start + 1);
string postString = processCommand.Substring(i + 1);
//Add the nested command to the list
commandParts.Add(nestedCommand);
//Update the commandText replacing the nested command we removed with its index in the list
processCommand = preString + "{" + (commandParts.Count - 1) + "}" + postString;
//Go back the the beginning of the command and look for the next nested command
i = 0;
start = -1;
}
}
//If start isnt -1 that means we found an open ( without a closing )
if (start == -1)
{
//We want to add the final command the the list for processing too
commandParts.Add(processCommand);
//We're going to go through the command parts and replace the LIMIT
for (int i = 0; i < commandParts.Count; i++)
{
string command = commandParts[i];
Console.WriteLine(command);
//We need to find the where the LIMIT is and extact the number
int limitIndex = command.IndexOf("LIMIT");
if (limitIndex != -1)
{
int startIndex = limitIndex + 6;
//Assuming after the limit will be ), a space, or the end of the string
int endIndex = command.IndexOf(')', startIndex);
if (endIndex == -1)
endIndex = command.IndexOf(' ', startIndex);
if (endIndex == -1)
endIndex = command.Length - 1;
Console.WriteLine(startIndex);
Console.WriteLine(endIndex);
//Extract the number
string limitNumber = command.Substring(startIndex, endIndex - startIndex);
//Remove the LIMIT command. There should always be a space before so take that out too.
command = command.Remove(limitIndex - 1, endIndex - limitIndex + 1);
//Insert the top command with the number
command = command.Replace("SELECT", "SELECT TOP " + limitNumber);
//Update the list
commandParts[i] = command;
}
}
start = -1;
//We need to go through the commands in reverse order and reassemble the complete command
for (int i = 0; i < processCommand.Length; i++)
{
//If we find a { its a part of the command that needs to be replaced
if (processCommand[i] == '{')
start = i;
if (processCommand[i] == '}')
{
string startString = processCommand.Substring(0, start);
string midString = processCommand.Substring(start, i - start + 1);
string endString = processCommand.Substring(i + 1);
//Get the index of the command we need from the list
int strIndex = Int32.Parse(midString.Substring(1, midString.Length - 2));
processCommand = processCommand.Replace(midString, commandParts[strIndex]);
//Go back to the start and look for the next
i = 0;
start = -1;
}
}
commandText = processCommand;
}
else
{
LogManager.Write(LogLevel.Error, "Unmatched parentheses were found while processing a SQL command. Command: " + commandText);
}
return commandText;
}

why does it appear a blank space between to string concatened c#

public string completeHour(string theTime)
{
string total="";
string[] timeArray = theTime.Split(new[] { ":" }, StringSplitOptions.None);
string h = timeArray[0];
string i = timeArray[1];
string j = timeArray[2];
MessageBox.Show(h + "+" + i + "+" + j);
if (h == " " || i == " " || j == " ")
{
if (h == " ")
{
h = "00";
total = (String.Concat("00",theTime)).Trim();
MessageBox.Show(total);
}
else if (i == " ")
{
i = "00";
total = timeArray[0] + i + timeArray[2];
//MessageBox.Show("m-=" + total);
}
//else if (j == "")
//{
// j = "00";
// theTime = timeArray[0] + timeArray[1] + j;
// MessageBox.Show("s-=" + theTime);
//}
}
return total;
}
Why total is 00 :52:04 (for instance) and not 00:52:04 that was supposed to be?

If you'd like to make sure there are no leading or trailing white characters, you could call
string h = timeArray[0].Trim();
And then instead of checking the value against " ", you could compare it to String.Empty or call h.IsNullOrEmpty().
However I'd strongly recommend you to use simpler approach, using a DateTime object.
DateTime timeObject;
DateTime.TryParse(theTime, out timeObject);
and then just work with Hour, Minute and Second properties. This way you get away from custom parsing and make your code more object-oriented, thus easier to read, instead of juggling multiple string objects.

Best way to avoid this is using Trim() when assigning value to total in following two lines:
total = (String.Concat("00",theTime.Trim())).Trim();
.
.
.
total = timeArray[0].trim() + i + timeArray[2].Trim();

Although I was using a MaskedTextBox I didn't define a custom mask so, anywhere (I think) the system assumed that 'theTime' was the type of DateTime.
So, the result of String.Concat("00",theTime) was '00 :32:99', for instance.
I´ve solved it by using the variables h, i and j instead of theTime.
Using a DateTime variable was not appropriated because I want to allow the user to insert NULL values for the hours or for the minutes.

For loop reading a file isn't working

I have this:
private void getAccount()
{
string[] acct = File.ReadAllLines(Environment.GetFolderPath(Environment.SpecialFolder.Desktop) + #"\Accts.txt");
for (int i = 0; i < acct[line].Length - 1; i++)
{
foreach (char c in acct[line])
{
if (c.ToString() == ":")
{
onPass = true;
i += 1;
}
if (onPass == false) { user += acct[line][i]; }
if (onPass == true) { pass += acct[line][i]; }
}
}
MessageBox.Show("Username is " + user + ". \n\nPassword is " + pass + ".");
onPass = false;
}
The file has this:
minicl55:mypass
However this outputs this:
These are the following problems:
The characters are repeated a lot
only "mmmmmmm" is considered part of the username, everything up until the colon should be part of the username, after is pass
The : is included in the password, it should be ignored completely (except to tell where the username stops and the password starts)

The first time you go through your for loop, i == 0. Then the foreach loop looks at each character in acct[line], but i never changes, so for all the characters prior to :, the acct[line][i] part keeps returning acct[line][0], or "m" 8 times. That's why the username appears to be "mmmmmmmm".
Then the colon is encountered and i is increased by 1. Now onPass == true, so pass ends up having acct[line][1], which is the character "i". This repeats for the rest of the string, so pass appears to be "iiiiiii" (from the colon to the end).
Now we go back to the for loop. Except i has been increased by 1 inside the loop (bad idea) so now the for loop is actually on i == 2. Again the beginning part executes 8 times (once for each character in the username), but always refers to acct[line][2], so the username is "nnnnnnnn". Except onPass is still true, so it gets appended to the password variable. Then you get 7 more "i"'s after i is increased.
The i variable is increased internally and in the for loop again, so next time you're using acct[line][4], which is "c" (8 times), then i is increased by 1 inside the foreach loop and you get acct[line][5] 7 times, which is "l".
So far, password is "iiiiiiinnnnnnnniiiiiiicccccccclllllll". Hopefully you can see the pattern.
You could eliminate some of the looping and complexity, and just use something like: (untested)
private void getAccount()
{
var allAccounts = File.ReadAllLines(Environment.GetFolderPath(Environment.SpecialFolder.Desktop) + #"\Accts.txt");
foreach (var account in allAccounts)
{
var pieces = account.Split(':');
MessageBox.Show(string.Format("Username is {0}. \n\nPassword is {1}.", pieces[0], pieces[1]));
}
}

Your outer loop is iterating over each char in acct[line]. Then you do the same in your inner loop, you just express it a little differently.

Please show your variables, but here's another approach:
private void getAccount()
{
string user = "";
string pass = "";
string[] user_pass = new string[0];
var accts = System.IO.File.ReadAllLines(Environment.GetFolderPath(Environment.SpecialFolder.Desktop) + #"\Accts.txt");
foreach(var acct in accts)
{
user_pass = acct.Split(':');
}
//Add iteration for multiple lines
if (user_pass.Length > 0)
{
MessageBox.Show("Username is " + user_pass[0] + ". \n\nPassword is " + user_pass[1] + ".");
}
else
{
MessageBox.Show("Chaos: Dogs and Cats Living Together!");
}
}
}
}

Well, I see your first loop gets the length of a specific line whose position doesnt change at all.
for (int i = 0; i < acct[line].Length - 1; i++)
And then you loop through every character of that only line
foreach (char c in acct[line])
The thing is that if your acct[line] has X length, you will loop through the acct[line] X times, hence why the repeated characters. You end up reading the same character X times.

As everyone else has commented/answered, your outer and inner loops are pretty much doing the exact same thing. I rewrote the for-loops so the outer loop loops through each line of the array of strings, and then the inside loop will go through all of the characters in that line.
for (int line = 0; line < acct.Length; line++)
{
int i = 0;
foreach (char c in acct[line])
{
if (c.ToString() == ":")
{
onPass = true;
}
else
{
if (!onPass)
user += acct[line][i];
else
pass += acct[line][i];
}
i++;
}
}
I do suggest however, for your own benefit, if you do NEED to loop through all of the characters to use this for the inner loop:
for (int i = 0; i < acct[line].Length; i++)
{
if (acct[line][i].ToString() == ":")
{
onPass = true;
}
else
{
if (!onPass)
user += acct[line][i];
else
pass += acct[line][i];
}
}
Or better yet replace everything with something simpler, and less prone to being broken by small changes:
for (int line = 0; line < acct.Length; line++)
{
if (acct[line].Contains(":"))
{
string[] parts = acct[line].Split(':');
user = parts[0];
pass = parts[1];
MessageBox.Show("Username is " + user + ". \n\nPassword is " + pass + ".");
}
}

CSV Validation in C# - Making sure each row has the same number of commas

I wish to implement a fairly simple CSV checker in my C#/ASP.NET application - my project automatically generates CSV's from GridView's for users, but I want to be able to quickly run through each line and see if they have the same amount of commas, and throw an exception if any differences occur. So far I have this, which does work but there are some issues I'll describe soon:
int? CommaCount = null;
StringBuilder sb = new StringBuilder();
StringWriter sw = new StringWriter(sb);
String Str = null;
//This loops through all the headerrow cells and writes them to the stringbuilder
for (int k = 0; k <= (grd.Columns.Count - 1); k++)
{
sw.Write(grd.HeaderRow.Cells[k].Text + ",");
}
sw.WriteLine(",");
//This loops through all the main rows and writes them to the stringbuilder
for (int i = 0; i <= grd.Rows.Count - 1; i++)
{
StringBuilder RowString = new StringBuilder();
for (int j = 0; j <= grd.Columns.Count - 1; j++)
{
//We'll need to strip meaningless junk such as <br /> and
Str = grd.Rows[i].Cells[j].Text.ToString().Replace("<br />", "");
if (Str == " ")
{
Str = "";
}
Str = "\"" + Str + "\"" + ",";
RowString.Append(Str);
sw.Write(Str);
}
sw.WriteLine();
//The below code block ensures that each row contains the same number of commas, which is crucial
int RowCommaCount = CheckChar(RowString.ToString(), ',');
if (CommaCount == null)
{
CommaCount = RowCommaCount;
}
else
{
if (CommaCount!= RowCommaCount)
{
throw new Exception("CSV generated is corrupt - line " + i + " has " + RowCommaCount + " commas when it should have " + CommaCount);
}
}
}
sw.Close();
And my CheckChar method:
protected static int CheckChar(string Input, char CharToCheck)
{
int Counter = 0;
foreach (char StringChar in Input)
{
if (StringChar == CharToCheck)
{
Counter++;
}
}
return Counter;
}
Now my problem is, if a cell in the grid contains a comma, my check char method will still count these as delimiters so will return an error. As you can see in the code, I wrap all the values in " characters to 'escape' them. How simple would it be to ignore commas in values in my method? I assume I'll need to rewrite the method quite a lot.

var rx = new Regex("^ ( ( \"[^\"]*\" ) | ( (?!$)[^\",] )+ | (?<1>,) )* $", RegexOptions.ExplicitCapture | RegexOptions.IgnorePatternWhitespace | RegexOptions.Multiline);
var matches = rx.Matches("Hello,World,How,Are\nYou,Today,This,Is,\"A beautiful, world\",Hi!");
for (int i = 1; i < matches.Count; i++) {
if (matches[i].Groups[1].Captures.Count != matches[i - 1].Groups[1].Captures.Count) {
throw new Exception();
}
}

You could just use a regular expression that matches one item and count the number of matches in your line. An example of such a regex is the following:
var itemsRegex =
new Regex(#"(?<=(^|[\" + separator + #"]))((?<item>[^""\" + separator +
#"\n]*)|(?<item>""([^""]|"""")*""))(?=($|[\" + separator + #"]))");

Just do something like the following (assuming you don't want to have " inside your fields (otherwise these need some extra handling)):
protected static int CheckChar(string Input, char CharToCheck, char fieldDelimiter)
{
int Counter = 0;
bool inValue = false;
foreach (char StringChar in Input)
{
if (StringChar == fieldDelimiter)
inValue = !inValue;
else if (!inValue && StringChar == CharToCheck)
Counter++;
}
return Counter;
}
This will cause inValue to be true while inside fields. E.g. pass '"' as fieldDelimiter to ignore everything between "...". Just note that this won't handle escaped " (like "" or \"). You'd have to add such handling yourself.

Instead of checking the resulting string (the cake) you should check the fields (ingredients) before you concatenate (mix) them. That would give you the change to do something constructive (escaping/replacing) and throwing an exception only as a last resort.
In general, "," are legal in .csv fields, as long as the string fields are quoted. So internal "," should not be a problem, but the quotes may well be.

c# Optimized Select string creation loop

I'm looking to optimized this piece of code. It will process 15000 - 20000 lines. For now I have 9000 lines and it take 30 sec approx. I know string concatenation is slow but I don't know how to do it another way.
//
// Check if composite primary keys existe in database
//
string strSelect = "SELECT * FROM " + _strTableName + " WHERE ";
for (int i = 0; i < strCompositeKeyField.Length; i++)
{
bool boolKeyProcess = false;
strSelect += _strHeaderLineSplitedArray[(int)arrayListCompositeKeyIndex[i]] + " = ";
DataColumn thisColomn = _dsProcessDataFromFileAndPutInDataSetDataSet.Tables["Repartition"].Columns[_strHeaderLineSplitedArray[(int)arrayListCompositeKeyIndex[i]]];
//_strProcessDataFromFileAndPutInDataSetLog += "Debug: Composite key : " + _strHeaderLineSplitedArray[(int)arrayListCompositeKeyIndex[i]] + " dataType : " + thisColomn.DataType.ToString() + " arrayListCompositeKeyIndex[i] = " + arrayListCompositeKeyIndex[i] + " \n";
// check if field is datetime to make convertion
if (thisColomn.DataType.ToString() == "System.DateTime")
{
DateTime thisDateTime = DateTime.ParseExact(strReadDataLineSplited[(int)arrayListCompositeKeyIndex[i]], _strDateConvertion, null);
strSelect += "'" + thisDateTime.ToString() + "'";
boolKeyProcess = true;
}
// check if field a string to add ''
else if (thisColomn.DataType.ToString() == "System.String")
{
strSelect += "'" + strReadDataLineSplited[(int)arrayListCompositeKeyIndex[i]] + "'";
boolKeyProcess = true;
}
// check if field need hour to second converstion
else
{
for (int j = 0; j < strHourToSecondConverstionField.Length; j++)
{
if (strCompositeKeyField[i] == strHourToSecondConverstionField[j])
{
DateTime thisDateTime = DateTime.ParseExact(strReadDataLineSplited[(int)arrayListCompositeKeyIndex[i]], _strHourConvertion, System.Globalization.CultureInfo.CurrentCulture);
strSelect += thisDateTime.TimeOfDay.TotalSeconds.ToString();
boolKeyProcess = true;
}
}
}
// if not allready process process as normal
if (!boolKeyProcess)
{
strSelect += strReadDataLineSplited[(int)arrayListCompositeKeyIndex[i]];
}
// Add " AND " if not last field
if (i != strCompositeKeyField.Length - 1)
{
strSelect += " AND ";
}
}
//_strProcessDataFromFileAndPutInDataSetLog += "Debug: SELECT = " + strSelect + "\n";
SqlDataAdapter AdapterCheckCompositePrimaryKeys = new SqlDataAdapter(strSelect, _scProcessDataFrinFileAndPutInDataSetSqlConnection);
DataSet DataSetCheckCompositePrimaryKeys = new DataSet();
AdapterCheckCompositePrimaryKeys.Fill(DataSetCheckCompositePrimaryKeys, "PrimaryKey");

You should definitely take a look at StringBuilder - it works wonders for scenarios like this one. In this case, i'd use a mix of AppendFormat and Append. I tend to like AppendFormat to make the strings a bit easier to follow.
//
// Check if composite primary keys existe in database
//
StringBuilder strSelect = "SELECT * FROM " + _strTableName + " WHERE ";
for (int i = 0; i < strCompositeKeyField.Length; i++)
{
bool boolKeyProcess = false;
strSelect.AppendFormat("{0} =",
_strHeaderLineSplitedArray[(int)arrayListCompositeKeyIndex[i]]);
DataColumn thisColomn =
_dsProcessDataFromFileAndPutInDataSetDataSet
.Tables["Repartition"]
.Columns[_strHeaderLineSplitedArray[(int)arrayListCompositeKeyIndex[i]]];
//_strProcessDataFromFileAndPutInDataSetLog += "Debug: Composite key : " + _strHeaderLineSplitedArray[(int)arrayListCompositeKeyIndex[i]] + " dataType : " + thisColomn.DataType.ToString() + " arrayListCompositeKeyIndex[i] = " + arrayListCompositeKeyIndex[i] + " \n";
// check if field is datetime to make convertion
if (thisColomn.DataType.ToString() == "System.DateTime")
{
DateTime thisDateTime =
DateTime.ParseExact(strReadDataLineSplited[(int)arrayListCompositeKeyIndex[i]],
_strDateConvertion, null);
strSelect.AppendFormat("'{0}'", thisDateTime.ToString());
boolKeyProcess = true;
}
// check if field a string to add ''
else if (thisColomn.DataType.ToString() == "System.String")
{
strSelect.AppendFormat("'{0}'",
strReadDataLineSplited[(int)arrayListCompositeKeyIndex[i]]);
boolKeyProcess = true;
}
// check if field need hour to second converstion
else
{
for (int j = 0; j < strHourToSecondConverstionField.Length; j++)
{
if (strCompositeKeyField[i] == strHourToSecondConverstionField[j])
{
DateTime thisDateTime = DateTime.ParseExact(
strReadDataLineSplited[(int)arrayListCompositeKeyIndex[i]],
_strHourConvertion,
System.Globalization.CultureInfo.CurrentCulture);
strSelect.Append(thisDateTime.TimeOfDay.TotalSeconds.ToString());
boolKeyProcess = true;
}
}
}
// if not allready process process as normal
if (!boolKeyProcess)
{
strSelect.Append(strReadDataLineSplited[(int)arrayListCompositeKeyIndex[i]]);
}
// Add " AND " if not last field
if (i != strCompositeKeyField.Length - 1)
{
strSelect.Append(" AND ");
}
}
//_strProcessDataFromFileAndPutInDataSetLog += "Debug: SELECT = " + strSelect + "\n";
SqlDataAdapter AdapterCheckCompositePrimaryKeys = new SqlDataAdapter(strSelect.ToString(), _scProcessDataFrinFileAndPutInDataSetSqlConnection);
DataSet DataSetCheckCompositePrimaryKeys = new DataSet();
AdapterCheckCompositePrimaryKeys.Fill(DataSetCheckCompositePrimaryKeys, "PrimaryKey");

Use a StringBuilder and its Append() method.

Make use of a StringBuilder rather than string concatenation.

Use StringBuilder for your string manipulation like strSelect += ...
instead use stringBuilder.Append("...");

Have u tried the StringBuilder object ?
http://msdn.microsoft.com/en-us/library/system.text.stringbuilder.aspx

To answer your direct question you'll almost certainly benefit from using StringBuilder to build the string up, then ToString() it at the end.
However, if you could give us an overview of the intention (so we don't have to wade through to deduce it) we can probably recommend a better way.

After a quick look, one thing that stands out is that you should be using the StringBuilder class to build up the string, instead of continually concatenating on to your strSelect string variable. Excerpt from the linked MSDN article:
The performance of a concatenation operation for a String or
StringBuilder object depends on how
often a memory allocation occurs. A
String concatenation operation always
allocates memory, whereas a
StringBuilder concatenation operation
only allocates memory if the
StringBuilder object buffer is too
small to accommodate the new data.
Consequently, the String class is
preferable for a concatenation
operation if a fixed number of String
objects are concatenated. In that
case, the individual concatenation
operations might even be combined into
a single operation by the compiler. A
StringBuilder object is preferable for
a concatenation operation if an
arbitrary number of strings are
concatenated; for example, if a loop
concatenates a random number of
strings of user input.

I'm a database guy, so hopefully I don't sound like an idiot here, but can you use the StringBuilder class? I don't know if that requires the .NET framework or not.

You've received a lot of good suggestions to use StringBuilder. To improve your performance more and since you are expecting a large string result, I would recommend as well, that you choose a good initial capacity to reduce the number of times the internal string buffer needs to be expanded.
StringBuilder strSelect = new StringBuilder("SELECT * FROM " + _strTableName + " WHERE ", 8192);
Note that I picked 8192 characters here, but you may want to initialize this to a larger number if you really are adding "9000" lines of data to your condition. Find out your typical size and set it the capacity to just a small amount above that.
The way StringBuilder works is that as you append to the string and reach the current capacity, it must create a new buffer and copy the contents of the old buffer to the new one then append the new characters. To optimize performance, the new buffer will be double the old size. Now, the default initial capacity is either 16 characters or the size of the initializing string. If your resulting string is 5000 characters long, then a StringBuilder created with a default size will have to be expanded 9 times - that requires new memory allocation and copying all the previous characters! If, however, you knew this would happen regularly and created the StringBuilder with the right capacity, there would be no additional allocation or copies.
Normally you should not need to worry about internal operations of an object, but you do sometimes for performance such as this. Since StringBuilder does provide a way for you to specific a recommended initial capacity, take advantage of that. You do not know if the doubly/copy algorithm will change in the future nor the initial default capacity may change, but your specification of an initial capacity will continue to allocate the right sized builder - because this is part of the public contract.