Related
I am making a simple compiler, and am working on string parsing. At the moment, my code is:
while (stringToParse.Contains(" + ") || stringToParse.Contains("+ ") || stringToParse.Contains(" +")) {
stringToParse = stringToParse.Replace(" +", "+").Replace("+ ", "+").Replace(" + ", "+");
}
string[] splitString = stringToParse.Split("+");
But something like:
"\"hello \" + \"world \" + \" + \" + \"hello\""
Would return:
["\"hello "\", "\"world \"", "\"", "\"", ]
(without backslashes)
But something like:
""hello " + "world " + " + " + "hello""
Would return:
[""hello "", ""world "", """, """, ]
So how can I specify if a " + " is in a string or as a separator? is there maybe a way to detect for something like the following?
...(any number of non " or + characters)...+...(any number of " or + characters)
My expected output would be:
[""hello "", ""world "", ""+""]
Explicit State Machine
To do this, Without using any dedicated library, I suggest to build a state machine.
You will iterate over the characters of the string, and depending on which character you encounter you update the state of the machine. Optimizations are possible, however, let us begin with conventional clarity.
var characters = input.ToCharArray();
var results = new List<string>();
var current = string.Empty;
// 0 = not inside quotes, we expect +
// 1 = not inside quotes, we expect "
// 2 = inside quotes
var state = 1;
foreach (var character in characters)
{
switch (state)
{
case 0:
// We are not inside quotes, we expect +
if (character == '+')
{
state = 1;
continue;
}
if (char.IsWhiteSpace(character))
{
continue;
}
// error?
break;
case 1:
// We are not inside quotes, we expect "
if (character == '\"')
{
state = 2;
continue;
}
if (char.IsWhiteSpace(character))
{
continue;
}
// error?
break;
case 2:
// We are inside quotes, we expect "
if (character == '\"')
{
state = 0;
results.Add(current);
current = string.Empty;
continue;
}
current += character;
break;
default:
// error?
break;
}
}
if (state != 0)
{
// error
}
// You can use results.ToArray();
Possible optimizations:
We can use a StringBuilder instead of concatenations.
Also, we can use IndexOf to find the next relevant character.
We can check if a string (a chunk of characters) is empty or white space (perhaps using IsNullOrWhiteSpace).
We can use AsSpan so we can work with ReadOnlySpan instead.
You can also see how you can add support for your own escape sequences, or any other stuff.
Implicit State Machine (with helper class)
I want to point out that this is not the only way to organize this code. I would, if I were you, create a pseudo iterator class that had a method two methods:
A method that returns the next character... or better yet, that returns true if the next character matches a parameter (and advances), or false (and does not advance).
A method that returns all the characters until the next instance of a particular character (and advances to there).
The main advantage of such approach is that I would no longer have to step character by character, thus, I would not need to have a state variable. Instead I could allow the code structure to resemble the shape of my gramar.
Wait, I have wrote such class: StringProcessor. It is part of the Theraot.Core nuget, it is used to parse strings to BigInteger.
var processor = new Theraot.Core.StringProcessor(input);
var results = new List<string>();
while (!processor.EndOfString)
{
// SkipWhile skips all the characters that match
processor.SkipWhile(char.IsWhiteSpace);
// Read returns true (and advances after) if what is next matches the paramter
if (processor.Read('"'))
{
// ReadUntil advances after and returns everything found before the parameter
// Note: it does not advance after the parameter.
results.Add(processor.ReadUntil('"'));
processor.Read('"');
}
processor.SkipWhile(char.IsWhiteSpace);
if (!processor.Read('+'))
{
// error?
}
}
Please notice that a class such as the StringProcessor used above cuts a lot of fluff, which makes it viable for simple languages.
Custom Tokenizer
Of course, for something more complex you might want to look for a tokenizer.
To give you an example, consider that this is the "grammar" we have:
Document: Many
{
Whitespace
String:
{
QuoteSymbol
NonQuoteSymbol
QuoteSymbol
}
Whitespace
PlusSymbol
}
No, this not any of the usual metalanguages. However, written this way it is easier to see how the code we had above resembles the language.
Would it not be nice to write as follows?
var QuoteSymbol = Pattern.Literal("QuoteSymbol", '"');
var NonQuoteSymbol = Pattern.Custom("NonQuoteSymbol", s => s.ReadUntil('"'));
var String = Pattern.Conjunction("String", QuoteSymbol, NonQuoteSymbol, QuoteSymbol);
var WhiteSpace = Pattern.Custom("WhiteSpace", s => s.ReadWhile(char.IsWhiteSpace));
var PlusSymbol = Pattern.Literal("PlusSymbol", '+');
var Document = Pattern.Repetition(
Pattern.Conjunction(WhiteSpace, String, WhiteSpace, PlusSymbol)
);
var results = from TerminalSymbol symbol
in Document.Parse(input)
where symbol.Pattern == String
select symbol.ToString();
Writing code like that would make it easier to modify the language. Well, we are still writing code, however you could imagine parsing a file that has the grammar of the language you want to parse... Fancy!
As you might expect, it requires extra work to build the necesary code to make it work. Or, you know, get some code that already works (the linked code is built around on StringProcessor).
Language Toolkits
The code presented earlier is not suitable to be used for a prettyprinter and is not capable of recovering from a syntax error. It can be modified to do such things. Neither will it integrate with code editors at any level.
If you want a fully fledged solution. I have two suggestions:
Irony
Nitra
These are the kind of things you would use if you wanted to create a programming language ontop.
And of course, I should link you to "Compilers: Principles, Techniques, and Tools" usually just known as "The Dragon Book".
I know that the data should be correct. I have no control over the data and my boss is just going to tell me that I need to figure out a way to deal with someone else's mistake. So please don't tell me it's not my problem that the data is bad, because it is.
Anywho, this is what I'm looking at:
"Words","email#email.com","","4253","57574","FirstName","","LastName, MD","","","576JFJD","","1971","","Words","Address","SUITE "A"","City","State","Zip","Phone","",""
Data has been scrubbed for confidentiality reasons.
So as you see, the data contains quotation marks and there are commas inside some of these quoted fields. So I cannot remove them. But the "Suite A""" is throwing off the parser. There are too many quotation marks. >.<
I'm using the TextFieldParser in the Microsoft.VisualBasic.FileIO namespace with these settings:
parser.HasFieldsEnclosedInQuotes = true;
parser.SetDelimiters(",");
parser.TextFieldType = FieldType.Delimited;
The error is
MalformedLineException: Line 9871 cannot be parsed using the current
delimiters.
I would like to scrub the data somehow to account for this but I'm not sure how to do it. Or maybe there's a way to just skip this line? Although I suspect my higher ups will not approve of me just skipping data that we might need.
If you are only trying to get rid of the stray " marks in your csv, you can use the following regex to find them and replace them with '
String sourcestring = "source string to match with pattern";
String matchpattern = #"(?<!^|,)""(?!(,|$))";
String replacementpattern = #"$1'";
Console.WriteLine(Regex.Replace(sourcestring,matchpattern,replacementpattern,RegexOptions.Multiline));
Explanation:
#"(?<!^|,)""(?!(,|$))"; will find will find any " that is not preceded by the beginning of the string, or a , and that is not followed by the end of the string or a ,
I am not familiar with TextFieldParser. However with CsvHelper, you can add a custom handler for invalid data:
var config = new CsvConfiguration();
config.IgnoreReadingExceptions = true;
config.ReadingExceptionCallback += (e, row) =>
{
// you can add some custom patching here if possible
// or, save the line numbers and add/edit them manually later.
};
using(var file = File.OpenRead(".csv"))
using(var reader = new CsvReader(reader, config))
{
reader.GetRecords<YourDtoClass>();
}
My only addition to what everyone is saying (because we've all been there) is to try to attempt to rectify each new issue you encounter with code. There are some decent REGEX strings out there https://www.google.com/?ion=1&espv=2#q=c-sharp+regex+csv+clean or you could manually fix things using String.Replace (String.Replace("\"\"\"","").Replace("\"\","").Replace("\",,","\",") or such). Eventually, as you detect and find ways of correcting more and more mistakes, your manual recovery rate will be minimized substantially (most of your bad data will likely come from similar mistakes). Cheers!
PS - Idea-ish (it's been a while - the logic may neeed some tweaking as I'm writing from memory), but you'll get the gist:
public string[] parseCSVWithQuotes(string csvLine,int expectedNumberOfDataPoints)
{
string ret = "";
string thisChar = "";
string lastChar = "";
bool needleDown = true;
for(int i = 0; i < csvLine.Length; i++)
{
thisChar = csvLine.Substring(i, 1);
if (thisChar == "'"&&lastChar!="'")
needleDown = needleDown == true ? false : true;//when needleDown = true, characters are treated literally
if (thisChar == ","&&lastChar!=",") {
if (needleDown)
{
ret += "|";//convert literal comma to pipe so it doesn't cause another break on split
}else
{
ret += ",";//break on split is intended because the comma is outside the single quote
}
}
if (!needleDown && (thisChar == "\"" || thisChar == "*")) {//repeat for any undesired character or use RegEx
//do not add -- this eliminates any undesired characters outside single quotes
}
else
{
if ((lastChar == "'" || lastChar == "\"" || lastChar == ",") && thisChar == lastChar)
{
//do not add - this eliminates double characters
}else
{
ret += thisChar;
lastChar = thisChar;
//this character is not an undesired character, is no a double, is valid.
}
}
}
//we've cleaned as best we can
string[] parts = ret.Split(',');
if(parts.Length==expectedNumberOfDataPoints){
for(int i = 0; i < parts.Length; i++)
{
//go back and replace the temporary pipe with the literal comma AFTER split
parts[i] = parts[i].Replace("|", ",");
}
return parts;
}else{
//save ret to bad CSV log
return null;
}
}
I've had to do this before,
The first step is to parse the data using string.split(',')
The next step is to combine the segments that belong together.
What I essentially did was
make a new list representing the combined strings
if a string begins with a quote, push it onto your new list
if it does not begin with a quote, append it to the last string in your list
Bonus: throw exceptions when a string ends with a quote but the next one does not begin with a quote
Depending on what the rules are regarding what can actually appear in your data, you might have to change your code to account for that.
At the core of CSV's file format, each line is a row, each cell in that row is separated by a comma. In your case, your format also contains the (very unfortunate) stipulation that commas inside a pair of quotation marks do not count as separators and are instead part of the data. I say very unfortunate because a misplaced quotation mark affects the entire rest of the line, and since quotation marks in standard ASCII do not distinguish between open and closed, there really is nothing you can do to recover from this without knowing the original intent.
That is when you log a message in a way that the person who does know the original intent (the person that provided the data) can look at the file and correct the error:
if (parse_line(line, &data)) {
// save the data
} else {
// log the error
fprintf(&stderr, "Bad line: %s", line);
}
And since your quotation marks aren't escaping newlines, you can keep on going with the next line after running into this error.
ADDENDUM: And if your company has a choice (i.e. your data is being serialized by a company tool) don't use CSV. Use something like XML or JSON with a much more clearly defined parsing mechanism.
I had to do this once aswell. My approach was to go through a line and keep track on what I was reading.
Basicly, I coded my own scanner chopping off tokens from the input line which gave me full control over my faulty .csv data.
This is what I did:
For each character on a line of input.
1. when outside of a string meeting a comma => all of the previous string (which can be empty) is a valid token.
2. when outside of a sting meeting anything but a comma or a quote => now you have a real problem, unquoted tekst => handle as you see fit.
3. when outside of a string meeing a quote => found a start of string.
4. when inside of a string meeting a comma => accept the comma as part of the string.
5. when inside of the string meeting a qoute => trouble starts here, mark this point.
6. continue and when meeting a comma (skipping white space if desired) close the string, 'unread' the comma and continue. (than will bring you to point 1.)
7. or continue and when meeting a quote -> obviously, what was read must be part of the string, add it to the string, 'unread' the quote and continue. (that will you bring to point 5)
8. or continue and find an whitespace, then End Of Line ('\n') -> the last qoute must be the closing quote. accept the string as a value.
9. or continue and fine non-whitespace, then End Of Line. -> now you have a real problem, you have the start of a string but it is not closed -> handle the error as you see fit.
If the number of fields in your .csv file is fixed you can count the comma's you recognise as field seperators and when you see a End Of Line you know you have another problem or not.
With the stream of strings received from the input line you can build a 'clean' .csv line and this way build a buffer of accepted and cleaned input that you can use in your already existing code.
I'm currently trying to strip a string of data that is may contain the hyphen symbol.
E.g. Basic logic:
string stringin = "test - 9894"; OR Data could be == "test";
if (string contains a hyphen "-"){
Strip stringin;
output would be "test" deleting from the hyphen.
}
Console.WriteLine(stringin);
The current C# code i'm trying to get to work is shown below:
string Details = "hsh4a - 8989";
var regexItem = new Regex("^[^-]*-?[^-]*$");
string stringin;
stringin = Details.ToString();
if (regexItem.IsMatch(stringin)) {
stringin = stringin.Substring(0, stringin.IndexOf("-") - 1); //Strip from the ending chars and - once - is hit.
}
Details = stringin;
Console.WriteLine(Details);
But pulls in an Error when the string does not contain any hyphen's.
How about just doing this?
stringin.Split('-')[0].Trim();
You could even specify the maximum number of substrings using overloaded Split constructor.
stringin.Split('-', 1)[0].Trim();
Your regex is asking for "zero or one repetition of -", which means that it matches even if your input does NOT contain a hyphen. Thereafter you do this
stringin.Substring(0, stringin.IndexOf("-") - 1)
Which gives an index out of range exception (There is no hyphen to find).
Make a simple change to your regex and it works with or without - ask for "one or more hyphens":
var regexItem = new Regex("^[^-]*-+[^-]*$");
here -------------------------^
It seems that you want the (sub)string starting from the dash ('-') if original one contains '-' or the original string if doesn't have dash.
If it's your case:
String Details = "hsh4a - 8989";
Details = Details.Substring(Details.IndexOf('-') + 1);
I wouldn't use regex for this case if I were you, it makes the solution much more complex than it can be.
For string I am sure will have no more than a couple of dashes I would use this code, because it is one liner and very simple:
string str= entryString.Split(new [] {'-'}, StringSplitOptions.RemoveEmptyEntries)[0];
If you know that a string might contain high amount of dashes, it is not recommended to use this approach - it will create high amount of different strings, although you are looking just for the first one. So, the solution would look like something like this code:
int firstDashIndex = entryString.IndexOf("-");
string str = firstDashIndex > -1? entryString.Substring(0, firstDashIndex) : entryString;
you don't need a regex for this. A simple IndexOf function will give you the index of the hyphen, then you can clean it up from there.
This is also a great place to start writing unit tests as well. They are very good for stuff like this.
Here's what the code could look like :
string inputString = "ho-something";
string outPutString = inputString;
var hyphenIndex = inputString.IndexOf('-');
if (hyphenIndex > -1)
{
outPutString = inputString.Substring(0, hyphenIndex);
}
return outPutString;
I have a log file that I want to parse and load into a database. I'm struggling with the best way to go about parsing it.
The log file is in the format Category: Information
Case Number: CASE01
User ID: JOSM
Software: Microsoft Word
Date Started: 21-01-2010
Date Ended: 22-01-2010
Thing is, there's other bits and pieces thrown into the log file that mean the information isn't always present on the same line. I also only want the information, not the category.
So far, I've tried stick it all into an array separated by \r\n, but I have to know the index of the information I want in order to consistently retrieve it, and that changes. I've also tried feeding it through StreamReader and saying
if (line.Contains("Case Number"))
{
tbReport.AppendText("Case Number: " + line.Remove(0, 13) + "\r\n");
}
Which gets me the information I want, but makes it very hard to do anything with.
I feel I'm better off going down the array path, but I could do with some guidance on how to search the array for the the category, and then parse the information.
Once I can parse it accurately, adding it into a database should be fairly straight forward. As it's my first time attempting this, I'd be interested in any tips or guidance as to the best way to go about this though.
Thanks.
This will give you a collection with all key/value pairs.
List<KeyValuePair> items = new List<KeyValuePair>();
var line = reader.ReadLine();
while (line != null)
{
int pos = line.IndexOf(':');
items.Add(new KeyValuePair(line.Substring(0, pos), line.Substring(pos+1));
line = reader.ReadLine();
}
If you have a log class which contains all possible names as properties, you can use reflection instead:
class LogEntry
{
public string CaseNumber { get; set; }
public string User { get; set; }
public string Software{ get; set; }
public string DateStarted { get; set; }
public string DateEnded { get; set; }
}
List<LogEntry> items = new List<LogEntry>();
var line = reader.ReadLine();
var currentEntry = new LogEntry();
while (line != null)
{
if (line == "") //empty line = new log entry. Change to your delimiter.
{
items.Add(currentEntry);
currentEntry = new LogEntry();
}
int pos = line.IndexOf(':');
var name = line.Substring(0, pos).Replace(" ", string.Empty);
var value = line.Substring(pos+1);
var pi = entry.GetType().GetProperty(name);
pi.SetValue(entry, value, null);
line = reader.ReadLine();
}
Note that I've not tested the code (just written it directly in here). You have to add error checking and such. The last alternative is not very performant as it is, but should do OK.
Sounds like a good case candidate for RegExp :
http://www.regular-expressions.info/dotnet.html
They're not too easy to learn but once you get the basic understanding, they can't be beaten for that kind of tasks.
It's not really a simple answer, but have you maybe though about using a regular expression for parsing the information out?
Regular expressions is kinda hardcore stuff, but they can parsed advanced files quite easily.
So in what I can see, then its like:
If a line starts with A-Z, then (a-z or A-Z or 0-9 or space) from zero to many times, then followed by a : then a space, and then the value.
So if you make a regular expression for that (If you wait awhile I will try to make one for you), then you could test each line with that. If it matches, then we can also use regular expressions to take the last part out, and the "key". If it don't matches, then we just append it to the last key.
Beware that its not totally fool-proof, as a new line could just start this way, but its kinda the best thing we can do, i think.
As promised here is a starting point for your regular expression:
^(?'key'[A-Z][a-z,A-Z,0-9,\s]+):\s(?'value'.+)
So to try and tell what it does, we need to go though each part:
^ ensures that a match starts on the beginning of a line
(?'key' is a syntax to begin a "capture" group. The regular expression will then give us access to easily take the "key" part of the regular expression out.
We that with a [A-Z] - that is a group that will match any big letter. But only one
[a-z,A-Z,0-9,\s]+ - is like the previous group, but just for all big, or small letters, numbers and space (\s), the plus outside the group tells that it can match more than one.
Then we just end the group, and puts in out *: and then a space.
We then begin a new group the value group, just like the key group.
Then we just write . (that means everything), and then just a + after that to make it catch more than one
I actually think that you can just take the whole string, and just match a:
RegEx.Matches (or something like that), and loop over them.
Then just take match.Groups["key"] and match.Groups["value"] and put into your array. (Sorry i dont have a Visual Studio handy to test it out)
The code below is designed to take a string in and remove any of a set of arbitrary words that are considered non-essential to a search phrase.
I didn't write the code, but need to incorporate it into something else. It works, and that's good, but it just feels wrong to me. However, I can't seem to get my head outside the box that this method has created to think of another approach.
Maybe I'm just making it more complicated than it needs to be, but I feel like this might be cleaner with a different technique, perhaps by using LINQ.
I would welcome any suggestions; including the suggestion that I'm over thinking it and that the existing code is perfectly clear, concise and performant.
So, here's the code:
private string RemoveNonEssentialWords(string phrase)
{
//This array is being created manually for demo purposes. In production code it's passed in from elsewhere.
string[] nonessentials = {"left", "right", "acute", "chronic", "excessive", "extensive",
"upper", "lower", "complete", "partial", "subacute", "severe",
"moderate", "total", "small", "large", "minor", "multiple", "early",
"major", "bilateral", "progressive"};
int index = -1;
for (int i = 0; i < nonessentials.Length; i++)
{
index = phrase.ToLower().IndexOf(nonessentials[i]);
while (index >= 0)
{
phrase = phrase.Remove(index, nonessentials[i].Length);
phrase = phrase.Trim().Replace(" ", " ");
index = phrase.IndexOf(nonessentials[i]);
}
}
return phrase;
}
Thanks in advance for your help.
Cheers,
Steve
This appears to be an algorithm for removing stop words from a search phrase.
Here's one thought: If this is in fact being used for a search, do you need the resulting phrase to be a perfect representation of the original (with all original whitespace intact), but with stop words removed, or can it be "close enough" so that the results are still effectively the same?
One approach would be to tokenize the phrase (using the approach of your choice - could be a regex, I'll use a simple split) and then reassemble it with the stop words removed. Example:
public static string RemoveStopWords(string phrase, IEnumerable<string> stop)
{
var tokens = Tokenize(phrase);
var filteredTokens = tokens.Where(s => !stop.Contains(s));
return string.Join(" ", filteredTokens.ToArray());
}
public static IEnumerable<string> Tokenize(string phrase)
{
return string.Split(phrase, ' ');
// Or use a regex, such as:
// return Regex.Split(phrase, #"\W+");
}
This won't give you exactly the same result, but I'll bet that it's close enough and it will definitely run a lot more efficiently. Actual search engines use an approach similar to this, since everything is indexed and searched at the word level, not the character level.
I guess your code is not doing what you want it to do anyway. "moderated" would be converted to "d" if I'm right. To get a good solution you have to specify your requirements a bit more detailed. I would probably use Replace or regular expressions.
I would use a regular expression (created inside the function) for this task. I think it would be capable of doing all the processing at once without having to make multiple passes through the string or having to create multiple intermediate strings.
private string RemoveNonEssentialWords(string phrase)
{
return Regex.Replace(phrase, // input
#"\b(" + String.Join("|", nonessentials) + #")\b", // pattern
"", // replacement
RegexOptions.IgnoreCase)
.Replace(" ", " ");
}
The \b at the beginning and end of the pattern makes sure that the match is on a boundary between alphanumeric and non-alphanumeric characters. In other words, it will not match just part of the word, like your sample code does.
Yeah, that smells.
I like little state machines for parsing, they can be self-contained inside a method using lists of delegates, looping through the characters in the input and sending each one through the state functions (which I have return the next state function based on the examined character).
For performance I would flush out whole words to a string builder after I've hit a separating character and checked the word against the list (might use a hash set for that)
I would create A Hash table of Removed words parse each word if in the hash remove it only one time through the array and I believe that creating a has table is O(n).
How does this look?
foreach (string nonEssent in nonessentials)
{
phrase.Replace(nonEssent, String.Empty);
}
phrase.Replace(" ", " ");
If you want to go the Regex route, you could do it like this. If you're going for speed it's worth a try and you can compare/contrast with other methods:
Start by creating a Regex from the array input. Something like:
var regexString = "\\b(" + string.Join("|", nonessentials) + ")\\b";
That will result in something like:
\b(left|right|chronic)\b
Then create a Regex object to do the find/replace:
System.Text.RegularExpressions.Regex regex = new System.Text.RegularExpressions.Regex(regexString, System.Text.RegularExpressions.RegexOptions.IgnoreCase);
Then you can just do a Replace like so:
string fixedPhrase = regex.Replace(phrase, "");