I'm attempting to write an application to extract properties and code from proprietary IDE design files. The file format looks something like this:
HEADING
{
SUBHEADING1
{
PropName1 = PropVal1;
PropName2 = PropVal2;
}
SUBHEADING2
{
{ 1 ; PropVal1 ; PropValue2 }
{ 2 ; PropVal1 ; PropValue2 ; OnEvent1=BEGIN
MESSAGE('Hello, World!');
{ block comments are between braces }
//inline comments are after double-slashes
END;
PropVal3 }
{ 1 ; PropVal1 ; PropVal2; PropVal3 }
}
}
What I am trying to do is extract the contents under the subheading blocks. In the case of SUBHEADING2, I would also separate each token as delimited by the semicolons. I had reasonably good success with just counting the brackets and keeping track of what subheading I'm currently under. The main issue I encountered involves dealing with the code comments.
This language happens to use {} for block comments, which interferes with the brackets in the file format. To make it even more interesting, it also needs to take into account double-slash inline comments and ignore everything up to the end of the line.
What is the best approach to tackling this? I looked at some of the compiler libraries discussed in another article (ANTLR, Doxygen, etc.) but they seem like overkill for solving this specific parsing issue.
I'd suggest writing a tokenizer and parser; this will give you more flexibility. The tokenizer basically does a simple text-wise breakdown of the sourcecode and puts it into more usable data structure; the parser figures out what to do with it, often leveraging recursion.
Terms to google: tokenizer, parser, compiler design, grammars
Math expression evaluator: http://www.codeproject.com/KB/vb/math_expression_evaluator.aspx
(you might be able to take an example like this and hack it apart into what you want)
More info about parsing: http://www.codeproject.com/KB/recipes/TinyPG.aspx
You won't have to go nearly as far as those articles go, but, you're going to want to study a bit on this one first.
You should be able to put something together in a few hours, using regular expressions in combination with some code that uses the results.
Something like this should work:
- Initialize the process by loading the file into a string.
Pull each top-level block from the string, using regex tags to separately identify the block keyword and contents.
If a block is found,
Make a decision based on the keyword
Pass the content to this process recursively.
Following this, you would process HEADING, then the first SUBHEADING, then the second SUBHEADING, then each sub-block. For the sub-block containing the block comment, you would presumably know based on the block's lack of a keyword that any sub-block is a comment, so there is no need to process the sub-blocks.
No matter which solution you will choose, I'm pretty sure the best way is to have 2 parsers/tokenizers. One for the main file structure with {} as grouping characters, and one for the code blocks.
Related
I've got a bunch of code in c# that has here and there lines where an if or else statement without parentheses got some extra lines which result in false behavior. I am looking for a regex to find all possible places where this problem could occur so i can manually look for it.
To clarify what i mean. In code are a few places (i found until now) where the following code is wrong.
if (notInitialized)
Initialize();
AdditionalInitializationNotUseThisWhenAlreadyInitialized();
which should be
if (notInitialized) {
Initialize();
AdditionalInitializationNotUseThisWhenAlreadyInitialized();
}
I tried this if\s*\(.*\)([\n\r\s[^{]]*.*);* but it gives me not only the results i want. It has the if (notInitialized) { parts too. I have almost no experience in using regex.
How can i find all these cases without checking every if/else/else if in the code, just the ones without curly braces?
One problem you are facing is that the regex is matching as much as it can on the .* to find a pattern match. Therefore, and depending on the options used (e.g. . matches anything but \n, or anything), you'll get unsatisfactory results.
Another problem is that you'd need to match recursively, e.g. skip as many ) as there were nested '(' in the expression. Only a very few regex engines can do this; .NET fortunately can via "balancing groups" but it is tricky and highly advanced application of regex. Also, for that to work correctly, you'd have to also recognize string and character literals (in quotes) as to not count parens in these.
Edit This regex for .NET should pretty reliably find these if and else statements:
\b(if\s*\(((?:[^()"']|'(\\.|[^'\\])'|"(\\.|[^"\\])"|(?<parens>\()|(?<-parens>\)))*(?(parens)(?!))\))|else)\s*[^{\s]
While this shows how powerful regexes can be, it's very cryptic and the proper way to do this would really be with a real parser (such as Roslyn).
You can use this:
if \(.+?\)[^{]*?\n[^{]*?[^{]
If you dont have ifs in a format like this
if (notInitialized)
{
Initialize();
AdditionalInitializationNotUseThisWhenAlreadyInitialized();
}
this also works:
if \(.+?\)[^{]*?\n
It will detect, if a line with "if" has no { at the end. Its is also a little shorter.
I am trying to parse a file in MVC C#, see the format below. Since its not in JSON I cannot use the Javascript serializer to deserialize to an object. The other option is use to LINQ and read line by line and retrieve the desired values. Could any one recommend a more efficient way to do it.
The first field I need to retrieve is the ASSAY NUMBER (for example value 877) from ASSAYS
and then the ASSAY_STATUS field from TEST_REPLICATE which could be multiple nodes. Thanks
LOAD_HEADER
{
EXPERIMENT_FILE_NAME "xyz.json"
EXPERIMENT_START_DATE_TIME 05.21.2012 03:44:01
OPERATOR_ID "Q_SI"
}
ASSAYS
{
ASSAY_NUMBER 877
ASSAY_VERSION 4
ASSAY_CALIBRATION_VERSION 1
}
TEST_REPLICATE
{
REPLICATE_ID 1985
ASSAY_NUMBER 877
ASSAY_VERSION 4
ASSAY_STATUS Research
}
TEST_REPLICATE
{
REPLICATE_ID 1985
ASSAY_NUMBER 877
ASSAY_VERSION 4
ASSAY_STATUS Research
}
You could either hack something together or use a parser generator like ANTLR or Coco/R. Both can generate parsers in C#.
I'm more fond of using a parser-combinator (a tool for constructing parsers using parser building blocks) than parser generators. I've had passable experience with Piglet, which is written with/for C#, and is pretty easy to use, and amazing experience with FParsec, but it's written for F#.
As far as parser generators go, there are those suggested by stmax, and there is also TinyPG, which a member recommended me once.
You can also roll your own parser. I suggest basing it on some sort of state machine model, though in this simple case, like Kirk Woll suggested, you could probably get by with some plain old string manipulation.
I think the answer to this hinges upon whether or not there will ever be more than one ASSAY_NUMBER value in the file. If so, the easiest and surest way I know is to read the file line-by-line and get the data you desire.
If, however, you know that each file is unique to a specific ASSY_NUMBER, you have a much simpler answer: read the file as one string and use REGEX to pull out the information you desire. I am not an expert on REGEX, but there are enough examples online that you should be able to create one that works.
I need to parse and split C and C++ functions into the main components (return type, function name/class and method, parameters, etc).
I'm working from either headers or a list where the signatures take the form:
public: void __thiscall myClass::method(int, class myOtherClass * )
I have the following regex, which works for most functions:
(?<expo>public\:|protected\:|private\:) (?<ret>(const )*(void|int|unsigned int|long|unsigned long|float|double|(class .*)|(enum .*))) (?<decl>__thiscall|__cdecl|__stdcall|__fastcall|__clrcall) (?<ns>.*)\:\:(?<class>(.*)((<.*>)*))\:\:(?<method>(.*)((<.*>)*))\((?<params>((.*(<.*>)?)(,)?)*)\)
There are a few functions that it doesn't like to parse, but appear to match the pattern. I'm not worried about matching functions that aren't members of a class at the moment (can handle that later). The expression is used in a C# program, so the <label>s are for easily retrieving the groups.
I'm wondering if there is a standard regex to parse all functions, or how to improve mine to handle the odd exceptions?
C++ is notoriously hard to parse; it is impossible to write a regex that catches all cases. For example, there can be an unlimited number of nested parentheses, which shows that even this subset of the C++ language is not regular.
But it seems that you're going for practicality, not theoretical correctness. Just keep improving your regex until it catches the cases it needs to catch, and try to make it as stringent as possible so you don't get any false matches.
Without knowing the "odd exceptions" that it doesn't catch, it's hard to say how to improve the regex.
Take a look at Boost.Spirit, it is a boost library that allows the implementation of recursive descent parsers using only C++ code and no preprocessors. You have to specify a BNF Grammar, and then pass a string for it to parse. You can even generate an Abstract-Syntax Tree (AST), which is useful to process the parsed data.
The BNF specification looks like for a list of integers or words separated might look like :
using spirit::alpha_p;
using spirit::digit_p;
using spirit::anychar_p;
using spirit::end_p;
using spirit::space_p;
// Inside the definition...
integer = +digit_p; // One or more digits.
word = +alpha_p; // One or more letters.
token = integer | word; // An integer or a word.
token_list = token >> *(+space_p >> token) // A token, followed by 0 or more tokens.
For more information refer to the documentation, the library is a bit complex at the beginning, but then it gets easier to use (and more powerful).
No. Even function prototypes can have arbitrary levels of nesting, so cannot be expressed with a single regular expression.
If you really are restricting yourself to things very close to your example (exactly 2 arguments, etc.), then could you provide an example of something that doesn't match?
I am calling a web service and all I get back is a giant blob of text. I am left to process it myself. Problem is not all lines are necessarily the same. They each have 2 or 3 sections to them and they are similar. Here are the most common examples
text1 [text2] /text3/
text1/test3
text1[text2]/text3
text1 [text2] /text /3 here/
I am not exactly sure how to approach this problem. I am not too good at doing anything advanced as far as manipulating strings.
I was thinking using a regular expression might work, but not too sure on that either. If I can get each of these 3 sections broken up it is easier from there to do the rest. its just there doesn't seem to be any uniformity to the main 3 sections that I know how to work with.
EDIT: Thanks for mentioning i didn't actually say what I wanted to do.
Basically, I want to split these 3 sections of text into their own strings seperate stings so basically take it from one single string to an array of 3 strings.
string[0] = text1
string[1] = text2
string[2] = text3
Here is some of the text I get back from a call as an example
スルホ基 [スルホき] /(n) sulfo group/
鋭いナイフ [するどいナイフ] /(n) sharp knife/
鋭い批判 [するどいひはん] /(n) sharp criticism/
スルナーイ /(n) (See ズルナ) (obsc) surnay (Anatolian woodwind instrument) (per:)/zurna/
スルピリン /(n) sulpyrine/
スルファミン /(n) sulfamine/
剃る [そる(P);する] /(v5r,vt) to shave/(P)/
As the first line for an example I want to pull it out into an array
string[0] = スルホ基
string[0] = [スルホき]
string[0] = /(n) sulfo group/
Those example seem a bit random, there has to be some kind of order, isn't there a spec for the service? If not i suggest more example so that we can understand the rules.
Read up on some of the info here on finite state machines, and see if you can use some of the concepts on your input parsing problem.
If there is some order to the groups on each line, then maybe you can use a regex to separate the groups out.
Edit: after seeing your samples, you may get by with a regex, breaking on some of those specific delimiters. It will take maybe half an hour to test theory: pick yourself up a free regex tester, make yourself a regex that will isolate out just one of those groups, and pump a few sample lines through. If it performs reliably on the real data that you have, then expand it and see if you can also isolate out the other groups.
I should mention though that your regexes will break or just become a nightmare if there is any sort of vagaries in your data (and frequently there is). So test long and hard before settling on them. If you find you start to have exceptions in your data, then you will need to choose some sort of parsing algorithm (the FSM i mentioned above is a pattern you can follow if you implement a parsing mechanism).
The most stupid answer is "Use regex". But more information needed for better one.
I am wondering if it is possible to extract the index position in a given string where a Regex failed when trying to match it?
For example, if my regex was "abc" and I tried to match that with "abd" the match would fail at index 2.
Edit for clarification. The reason I need this is to allow me to simplify the parsing component of my application. The application is an Assmebly language teaching tool which allows students to write, compile, and execute assembly like programs.
Currently I have a tokenizer class which converts input strings into Tokens using regex's. This works very well. For example:
The tokenizer would produce the following tokens given the following input = "INP :x:":
Token.OPCODE, Token.WHITESPACE, Token.LABEL, Token.EOL
These tokens are then analysed to ensure they conform to a syntax for a given statement. Currently this is done using IF statements and is proving cumbersome. The upside of this approach is that I can provide detailed error messages. I.E
if(token[2] != Token.LABEL) { throw new SyntaxError("Expected label");}
I want to use a regular expression to define a syntax instead of the annoying IF statements. But in doing so I lose the ability to return detailed error reports. I therefore would at least like to inform the user of WHERE the error occurred.
I agree with Colin Younger, I don't think it is possible with the existing Regex class. However, I think it is doable if you are willing to sweat a little:
Get the Regex class source code
(e.g.
http://www.codeplex.com/NetMassDownloader
to download the .Net source).
Change the code to have a readonly
property with the failure index.
Make sure your code uses that Regex
rather than Microsoft's.
I guess such an index would only have meaning in some simple case, like in your example.
If you'll take a regex like "ab*c*z" (where by * I mean any character) and a string "abbbcbbcdd", what should be the index, you are talking about?
It will depend on the algorithm used for mathcing...
Could fail on "abbbc..." or on "abbbcbbc..."
I don't believe it's possible, but I am intrigued why you would want it.
In order to do that you would need either callbacks embedded in the regex (which AFAIK C# doesn't support) or preferably hooks into the regex engine. Even then, it's not clear what result you would want if backtracking was involved.
It is not possible to be able to tell where a regex fails. as a result you need to take a different approach. You need to compare strings. Use a regex to remove all the things that could vary and compare it with the string that you know it does not change.
I run into the same problem came up to your answer and had to work out my own solution. Here it is:
https://stackoverflow.com/a/11730035/637142
hope it helps