We Keep Coding

EF Core 6, { character in an SQL VALUES() block is causing string not in a correct format exception

This one can't be correct, probably is somehow a mis-configuration of EF Core in our project.
Background: We have to export an Oracle DB from Aqua Data studio to a file. I have been manually transposing the export from Oracle to SQL Server. We came up with the idea of putting the manual process into a Visual Studio project.
From the project, certain strings in the VALUES() section of some INSERT INTO "SomeTable" blocks fail.
In all cases the same code copied over to SSMS works with no modification to it.
This unit test does prove that the { character in the string throws an exception. Removing it from the string, the ExecuteRawSQL() method will now insert the data. I've put Bold / Italics around the {.
We are running the latest SQL Server Express, on local host. EF Core 6, .Net Core 6.
The entire input string for ANSWER_TEXT is after the Test Method code block.
I've tried escaping the {, etc. Also putting a space between { and Project. Not that I thought this would fix it, just to rule it out.
The { causing the issue can't possibly be right. The same block with the entire string will insert via SSMS.
It's got to be a Visual Studio project or EF Core config issue, something like that.
It's got me stumped -- didn't expect to trim this string down and be able to duplicate this malfunction.
[TestMethod]
public void Test_CE_USER_ANSWERS()
{
File.Delete(#"C:\ProjectsTFS-DOIT\Online CE\Docs\New_TSQL_DB\Test_CE_USER_ANSWERS.txt");
List<string> sqlDataStatements = new ();
string ceUserAnswers = #"INSERT INTO ""CE_TRANSMITTAL_ANSWERS""(""FORM_ID"", ""TRANSMITTAL_ID"", ""QUESTION_ID"", ""ANSWER_ORDER"", ""ANSWER_TEXT"")
VALUES(9804, 1, 808, 0, 'A history/architecture memo to file dated 3/25/ 2021 has been uploaded to Environet ***{*** Project File>Cultural Resources>Project ')
";
sqlDataStatements.Add(ceUserAnswers);
using (CETESTContext ctx = new())
{
foreach (string statement in sqlDataStatements)
{
try
{
ctx.Database.ExecuteSqlRaw(statement);
}
catch (Exception ex)
{
File.AppendAllText(#"C:\ProjectsTFS-DOIT\Online CE\Docs\New_TSQL_DB\Test_CE_USER_ANSWERS.txt", "Exception Details: \n ================ \n" + ex.ToString() + "\n ================ \n" + "SQL Statement: \n ================ \n" + statement + "\n ================ \nEnd of Exception and Statement Message \n ================ \n");
}
}
}
}
Here is the entire input string:
'<div>There are no previously recorded archaeological sites within the project APE. The project area is located within a suburban corridor with commercial and dense residential development. Areas of new right-of-way beyond the existing operational right-of-way have been disturbed by modern development including commercial development, gas stations, driveways and streets, parking areas, underground utilities, sidewalks, and residential development to the northeast and east. No further archaeological investigations are recommended.</div><div><br></div>The project area is largely developed with commercial buildings less than 50 years old. A history/architecture memo to file dated 3/25/ 2021 has been uploaded to Environet {Project File>Cultural Resources>Project Information>Memo-to-File History Architecture]. No history/architecture properties within the APE were recommended as eligible for the NRHP. Based on the scope of the project and review of the project APE, no additional research is recommended for history/architecture properties, and no NRHP-listed or eligible properties will be affected by the project.'
Edit 1
Regarding this from Lasse V. Karlsen from the comments:
If I'm not mistaken, whole ExecuteSqlRaw implementation is an anti-pattern because it uses the same syntax and features as string.Format, thus promoting SQL concatenation. Try doubling up the brackets, {{ instead of just {, see if this ends up inserting with just 1 {
This does work. But we have some 4.9 million lines of code with INSERT INTO ... VALUES(), any of the values might have { or } which need to now be {{ or }}.
Would prefer not to alter data although technically this really does insert one { or }.
Is there some way to tell EF to not worry about {} characters? That would be much easier.
Edit 2
I wasn't crazy about doing a blanket replace of { with {{, } with }} but this works.
Here is our code for our replacements -- takes out unneeded Oracle syntax and now replaces {}:
private static string RemoveOracleSyntax(string statement)
{
string cleanStatement = statement;
if (cleanStatement.Contains("TO_DATE"))
{
cleanStatement = cleanStatement.Replace("TO_DATE(", "");
cleanStatement = cleanStatement.Replace(",'YYYY-MM-DD HH24:MI:SS')", "");
}
if (cleanStatement.Contains("TO_CLOB"))
{
cleanStatement = cleanStatement.Replace("') || TO_CLOB('", "");
cleanStatement = cleanStatement.Replace("TO_CLOB(", "");
cleanStatement = cleanStatement.Replace("'))", "') ");
cleanStatement = cleanStatement.Replace("'),", "', ");
}
if (cleanStatement.Contains("{"))
{
cleanStatement = cleanStatement.Replace("{", "{{");
}
if (cleanStatement.Contains("}"))
{
cleanStatement = cleanStatement.Replace("}", "}}");
}
return cleanStatement;
}

The input from Lasse V. Karlsen is the answer.
If I'm not mistaken, whole ExecuteSqlRaw implementation is an anti-pattern because it uses the same syntax and features as string.Format, thus promoting SQL concatenation. Try doubling up the brackets, {{ instead of just {, see if this ends up inserting with just 1 {.
Our Lead Developer is wants to use ExecuteSQL, not ExecuteSQLRaw. It was brought up this is no longer in the API. This supposedly would not fail on a { or } character.

How to check if the string contains dynamic word that starts with first letter

I am trying to find out in the string if the word that changes but starts with letter 'F' (C#). The result output from service call is as below:
Exception_Remote_Call--VNQ DN ERROR CODE found ERROR CODE= F0123,
ERROR DESCRIPTION= NOT AVAILABLE
In the above string, F0123 word changes according to the different ERROR CODE. I tried as below but it works for F0123 and does not work if the output is F0111. I would like to find if it starts with 'F'.
var isStartsWithF = s.Contains("F0123");
I would really appreciate for the help. Thank you in advance!

This is a job for regular expressions. To make things clearer and easier to spot for future maintainers, I might include the ERROR CODE = as part of the expression:
var data = "Exception_Remote_Call--VNQ DN ERROR CODE found ERROR CODE = F0123, ERROR DESCRIPTION= NOT AVAILABLE";
var exp = new Regex(#"ERROR CODE\s?= (F\d{4,5})");
var result = exp.Match(data).Groups[1].Value;
See it work here:
https://dotnetfiddle.net/nOXOCt

C# Imap Sort command with special characters

I'm working on a problem with imap sort extention:
My command is the following:
var query = "icône";
//byte[] bytes = Encoding.Default.GetBytes(query);
//query = Encoding.UTF8.GetString(bytes);
var command = "SORT (REVERSE ARRIVAL) UTF-8 " + "{" + query.Length + "}";
var imapAnswerString = client.Command(command);
imapAnswerString = client.Command(query);
I get the following error:
BAD Error in IMAP command SORT: 8bit data in atom
I found this:
C# Imap search command with special characters like á,é
But I don't see how to prepare my code to send this request sucessfully.

If you want to stick with MailSystem.NET, the answer that arnt gave is correct.
However, as I point out here (and below for convenience), MailSystem.NET has a lot of architectural design problems that make it unusable.
If you use an alternative open source library, like MailKit, you'd accomplish this search query far more easily:
var query = SearchQuery.BodyContains ("icône");
var orderBy = new OrderBy[] { OrderBy.ReverseArrival };
var results = folder.Search (query, orderBy);
Hope that helps.
Architectural problems in MailSystem.NET include:
MailSystem.NET does not properly handle literal tokens - either sending them (for anything other than APPEND) or for receiving them (for anything other than the actual message data in a FETCH request). What none of the authors seem to have noticed is that a server may choose to use literals for any string response.
What does this mean?
It means that the server may choose to respond to a LIST command using a literal for the mailbox name.
It means that any field in a BODYSTRUCTURE may be a literal and does not have to be a quoted-string like they all assume.
(and more...)
MailSystem.NET, for example, also does not properly encode or quote mailbox names:
Example from MailSystem.NET:
public string RenameMailbox(string oldMailboxName, string newMailboxName)
{
string response = this.Command("rename \"" + oldMailboxName + "\" \"" + newMailboxName + "\"");
return response;
}
This deserves a Jean-Luc Picard and Will Riker face-palm. This code just blindly puts double-quotes around the mailbox name. This is wrong for at least 2 reasons:
What if the mailbox name has any double quotes or backslashes? It needs to escape them with \'s.
What if the mailboxName has non-ASCII characters or an &? It needs to encode the name using a modified version of the UTF-7 character encoding.
Most (all?) of the .NET IMAP clients I could find read the entire response from the server into 1 big string and then try and parse the response with some combination of regex, IndexOf(), and Substring(). What makes things worse is that most of them were also written by developers that don't know the difference between unicode character counts (i.e. string.Length) and octets (i.e. byte counts), so when they try to parse a response to a FETCH request for a message, they do this after parsing the "{}" value in the first line of the response:
int startIndex = response.IndexOf ("}") + 3;
int endIndex = startIndex + octets;
string msg = response.Substring (startIndex, endIndex - startIndex);
The MailSystem.NET developers obviously got bug reports about this not working for international mails, so their "fix" was to do this:
public string Body(int messageOrdinal)
{
this.ParentMailbox.SourceClient.SelectMailbox(this.ParentMailbox.Name);
string response = this.ParentMailbox.SourceClient.Command("fetch "+messageOrdinal.ToString()+" body", getFetchOptions());
return response.Substring(response.IndexOf("}")+3,response.LastIndexOf(" UID")-response.IndexOf("}")-7);
}
Essentially, they assume that the UID key/value pair will come after the message and use that as a hack-around for their incompetence. Unfortunately, adding more incompetence to existing incompetence only multiplies the incompetence, it doesn't actually fix it.
The IMAP specification specifically states that the order of the results can vary and that they may not even be in the same untagged response.
Not only that, but their FETCH request doesn't even request the UID value from the server, so it's up to the server whether to return it or not!
TL;DR
How to Evaluate an IMAP Client Library
The first thing you should do when evaluating an IMAP client library implementation is to see how they parse responses. If they don't use an actual tokenizer, you can tell right off the bat that the library was written by people who have no clue what they are doing. That is the most sure-fire warning sign to STAY AWAY.
Does the library handle untagged ("*") responses in a central place (such as their command pipeline)? Or does it do something retarded like try and parse it in every single method that sends a command (e.g. ImapClient.SelectFolder(), ImapClient.FetchMessage(), etc)? If the library doesn't handle it in a central location that can properly deal with these untagged responses and update state (and notify you of important things like EXPUNGE's), STAY AWAY.
If the library reads the entire response (or even just the "message") into a System.String, STAY AWAY.

You're almost there. Your final command should be something like
x sort (reverse arrival) utf-8 subject {6+}
icône
ie. you're just missing a search term to describe where the IMAP server should search for icône and sort the results. There are many other search keys, not just subject. See RFC3501 page 49 and following pages.
Edit: The + is needed after the 6 in order to send that as a single command (but requires that the server support the LITERAL+ extension). If the server doesn't support LITERAL+, then you will need to break up your command into multiple segments, like so:
C: a001 SORT (REVERSE ARRIVAL) UTF-8 SUBJECT {6}
S: + ok, whenever you are ready...
C: icône
S: ... <response goes here>

Thanks all for your answer.
Basically the way MailSystem.net sends requests (Command method) is the crux of this problem, and some others actually.
The command method should be corrected as follows:
First, when sending the request to imap, the following code works better than the original one:
//Convert stuff to have to right encoding in a char array
var myCommand = stamp + ((stamp.Length > 0) ? " " : "") + command + "\r\n";
var bytesUtf8 = Encoding.UTF8.GetBytes(myCommand);
var commandCharArray = Encoding.UTF8.GetString(bytesUtf8).ToCharArray();
#if !PocketPC
if (this._sslStream != null)
{
this._sslStream.Write(System.Text.Encoding.UTF8.GetBytes(commandCharArray));
}
else
{
base.GetStream().Write(System.Text.Encoding.UTF8.GetBytes(commandCharArray), 0, commandCharArray.Length);
}
#endif
#if PocketPC
base.GetStream().Write(System.Text.Encoding.UTF8.GetBytes(commandCharArray), 0, commandCharArray.Length);
#endif
Then, in the same method, to avoid some deadlock or wrong exceptions, improve the validity tests as follows:
if (temp.StartsWith(stamp) || temp.ToLower().StartsWith("* " + command.Split(' ')[0].ToLower()) || (temp.StartsWith("+ ") && options.IsPlusCmdAllowed) || temp.Contains("BAD Error in IMAP command"))
{
lastline = temp;
break;
}
Finally, update the return if as follows:
if (lastline.StartsWith(stamp + " OK") || temp.ToLower().StartsWith("* " + command.Split(' ')[0].ToLower()) && !options.IsSecondCallCommand || temp.ToLower().StartsWith("* ") && options.IsSecondCallCommand && !temp.Contains("BAD") || temp.StartsWith("+ "))
return bufferString;
With this change, all commands work fine, also double call commands. There are less side effects than with the original code.
This resolved most of my problems.

Escaping command line arguments in C# for Urls and Local and Network Paths

How do I provide an input string with automatic escaping to a console application?
I mean inside my code, I can do
public static void main(string[] args)
{
string myURL;
myFolder = #"C:\temp\january\"; //just for testing
myFolder = args[0]; // I want to do this eventually
}
How can I provide values to myFolder without me having to escape it manually via command line?
If possible, I want to avoid calling this app like this:
C:\test> myapplication.exe "C:\\temp\\january\\"
EDIT:
instead I'd prefer calling the app like this if possible
C:\test> myapplication.exe #"C:\temp\january\"
Thank you.
EDIT:
This is actually for a console application that calls Sharepoint Web services. I tried
string SourceFileFullPath, SourceFileName, DestinationFolder, DestinationFullPath;
//This part didn't work. Got Microsoft.SharePoint.SoapServer.SoapServerException
//SourceFileFullPath = args[0]; // C:\temp\xyz.pdf
//SourceFileName = args[1]; // xyz.pdf
//DestinationFolder = args[2]; // "http://myserver/ClientX/Performance" Reports
//This worked.
SourceFileFullPath = #"C:\temp\TestDoc2.txt";
SourceFileName = #"TestDoc2.txt";
DestinationFolder = #"http://myserver/ClientX/Performance Reports";
DestinationFullPath = string.Format("{0}/{1}", DestinationFolder, SourceFileName);

The requirement to escape \ inside a string if it is not a verbatim string (one that starts with #) is a C# feature. When you start your application from a console, you are outside of C#, and the console does not consider \ to be a special character, so C:\test> myapplication.exe "C:\temp\january" will work.
Edit: My original post had "C:\temp\january\" above; however, the Windows command line seems to also handle \ as an escape character - but only when in front of a ", so that command would pass C:\temp\january" to the application. Thanks to #zimdanen for pointing this out.
Please note that whatever you put between quotes in C# is a representation of a string; the actual string may be different - for instance, \\ represents a single \. If you use other means to get strings into the program, such as the command line arguments or by reading from a file, the strings do not need to follow C#'s rules for string literals. The command line has different rules for representation, in which a \ represents itself.

“The prefix “#” enables the use of keywords as identifiers, which is useful when interfacing with other programming languages. The character # is not actually part of the identifier, so the identifier might be seen in other languages as a normal identifier, without the prefix. An identifier with an # prefix is called a verbatim identifier. Use of the # prefix for identifiers that are not keywords is permitted, but strongly discouraged as a matter of style.”
You can use one of the reserved words of c# with the # symbol
ex:-
string #int = "senthil kumar";
string #class ="MCA";
2.Before a string specially when using the file paths
string filepath = #"D:\SENTHIL-DATA\myprofile.txt";
instead of
string filepath = "D:\\SENTHIL-DATA\\myprofile.txt";
For a Multi lined text
string ThreeIdiots = #"Senthil Kumar,
Norton Stanley,
and Pavan Rao!";
MessageBox.Show(ThreeIdiots);
instead of
string ThreeIdiots = #"Senthil Kumar,\n Norton Stanley,and Pavan Rao!";

ToLower() in Global.asax Application_BeginRequest spiking CPU and bombing app

Hoping someone can shed some light on this issue we are having because I'm at a loss here.
First, a little background:
I rewrote the URL rewriting for our application and implemented it a couple of weeks ago. I did this using Application_BeginRequest() in the global.asax file and everything was fine with our application except for a small oversight I had made. When I'm rewriting the URLs I'm simply checking for the existence of certain keywords in the path that the user requests and then rewriting the path accordingly. Pretty straight forward stuff, not inventing the wheel here. Dry code, really. However, the text I'm checking for is all lowercase while the path may come in with different cases.
For instance:
string sPath = Request.Url.ToString();
sPath = sPath.Replace(Request.Url.Scheme + "://", "")
.Replace(Request.Url.Host, "");
if (sPath.TrimStart('/').TrimEnd('/').Split('/')[0].Contains("reports") && sPath.TrimStart('/').TrimEnd('/').Split('/').Length > 2) {
string[] aVariables = sPath.TrimStart('/').TrimEnd('/').Split('/');
Context.RewritePath("/app/reports/report-logon.aspx?iLanguageID=" + aVariables[1] + "&sEventCode=" + aVariables[2]);
}
...if someone enters the pages as /Reports/, the rule will not match and they will receive a 404 error as a result.
Simple to fix, though, I thought. One only needs to force the requested path string to lowercase so that anything I attempt to match against it will be looking at a lowercase version of the requested path, and match successfully in cases such as the above. So I adjusted the code to read:
string sPath = Request.Url.ToString();
sPath = sPath.Replace(Request.Url.Scheme + "://", "")
.Replace(Request.Url.Host, "");
sPath = sPath.ToLower(); // <--- New line
if (sPath.TrimStart('/').TrimEnd('/').Split('/')[0].Contains("reports") && sPath.TrimStart('/').TrimEnd('/').Split('/').Length > 2) {
string[] aVariables = sPath.TrimStart('/').TrimEnd('/').Split('/');
Context.RewritePath("/app/reports/report-logon.aspx?iLanguageID=" + aVariables[1] + "&sEventCode=" + aVariables[2]);
}
With this fix, when I request any URL that matches against the URL rewriting, however, the CPU on the server spikes to 100% and my entire application crashes. I take out .ToLower(), kill the app pool, and the application is perfectly fine again.
Am I missing something here!?!? What gives? Why does such a simple method cause my application to explode? .ToLower() works everywhere else in our application, and although I'm not using it extensively, I am using it quite successfully in other places around the application.

Not sure exactly why ToLower would cause this (only thing I can think of is that it is modifying request.url, which sends asp.net into a frenzy), but there is an easy fix: use an ignorecase comparison rather than converting everything tolower.
Change:
sPath.TrimStart('/').TrimEnd('/').Split('/')[0].Contains("reports")
to:
sPath.TrimStart('/').TrimEnd('/').Split('/')[0].IndexOf("reports", StringComparison.InvariantCultureIgnoreCase) != -1
and remove your ToLower logic.

Though I can't say why .toLower() is bringing your server down
Why dont you try it with indexOf
if (sPath.TrimStart('/').TrimEnd('/').Split('/')[0].IndexOf("reports",StringComparison.InvariantCultureIgnoreCase)>=0 && sPath.TrimStart('/').TrimEnd('/').Split('/').Length > 2)
{
string[] aVariables = sPath.TrimStart('/').TrimEnd('/').Split('/');
Context.RewritePath("/app/reports/report-logon.aspx?iLanguageID=" + aVariables[1] + "&sEventCode=" + aVariables[2]);
}