We Keep Coding

How to work with textual formats in otherwise procedural code?

This question sounds trivial but let me explain my scenario.
I am working in an object oriented programming language (C#) and most of the actual execution code is procedural, i.e. series of statements, sometimes branches and loops. Fairly standard.
Now I am presented with a task to deal with a textual format (PGN, but it could be anything other like VCard or some custom format). At least for me, the "standard" way to work with it would be to use a mix of:
regular expressions
if / switch statements
for-loops
storing regexp matches into some custom structure and / or outputting it to some result format
However, I don't like this procedural approach at all - regular expressions are prone to errors, the code is usually quite hard to understand and debug, it usually tends to have quite a high cyclomatic complexity etc.
Simply put, I'd like it to be declarative but I don't know what tools or libraries to use.
I remember that when I saw demos of the "M" language I thought that that was exactly I was looking for. There was a simple way to declare syntax of my textual format, the tool would then automatically parse input string into an in-memory representation of the textual DSL, I think that it was also possible to transform the format into another etc.
I have been also in touch with the people behind JetBrains MPS which is another tool for working with DSLs but my scenario doesn't seem to be a perfect match for what they are trying to provide.
So if anyone has any idea about how to elegantly deal with textual formats in otherwise procedural code base, I'd be happy to learn about the options.

Check out my open source project meta#. I think it sounds like exactly what you're looking for.

Automating complex refactoring tasks

I have the situation that the same repeating refactoring tasks have to be done for a huge number of methods in my code.
For example imagine a interface with 100 methods, each of them has one or more parameters as well as a return value. For each of these methods I need to jump to the implementation change the return type and add a line of code which converts the old return value to its new type for callers of the interface method.
Is there any way to quickly automate such refactorings?
I even thought to write a custom script to do it, but writing a intelligent script would approximately take longer than doing it maually.
A tool supporting such task can save a lot of time.

It's a good question, but in the time it took since you posted it (not to mention the time you spent searching for an answer before posting), you could have completed the changes manually.
I know, I know, it's utterly unsatisfying, but if you think of it as a form of mediation, and only do this once a year, it's not that bad.

If your problem is one interface with 100 methods, then I agree with another poster: just doing it may seem painful but it is limited in effort and you can be done really soon.
If you have this problem repeatedly, or you have very large code base (many, many interfaces for which you want to perform this task), then what you need is a tool for implementing automated change: a program transformation engine. Such a tool provides the ability to parse source code, build a program representation (an abstract syntax tree), and enables one to apply "scripted" operations on the tree either through procedural interfaces and/or through source-to-source transformation patterns.
OUr DMS Software Reengineering Toolkit is such a program transformation system. It has a C# Front End to enable its application to C# code. Configuring such a tool for a complex task is not a matter of hours, so it is not useful for "small scale" changes. For large scale changes, such tools can make it possible to do things simply not practical by hand.

Resharper and CodeRush both have features which can help with this kind of task.
Resharper's change signature functionality is probably the closest match.

Can't you generate a new interface from the class you have and then remove the ones you don't need! if it's that simple!!

change the return type : by changing... the return type, provided it is not a standard type (...), and the converter can be implemented by a TypeConverter.
When i have such boring task to do, i often switch VS2010 and use a tool that allow regex search and replace. In your example, maybe change 'return xxx;' by 'var yyy=convert(xxx); return yyy;'
(for example editor Notepad++ (free) allready offers quite some possiblities to change everything in a project (use with caution))

Creating a scripting language to be used to create web pages

I am creating a scripting language to be used to create web pages, but don't know exactly where to begin.
I have a file that looks like this:
mylanguagename(main) {
OnLoad(protected) {
Display(img, text, link);
}
Canvas(public) {
Image img: "Images\my_image.png";
img.Name: "img";
img.Border: "None";
img.BackgroundColor: "Transparent";
img.Position: 10, 10;
Text text: "This is a multiline str#ning. The #n creates a new line.";
text.Name: text;
text.Position: 10, 25;
Link link: "Click here to enlarge img.";
link.Name: "link";
link.Position: 10, 60;
link.Event: link.Clicked;
}
link.Clicked(sender, link, protected) {
Image img: from Canvas.FindElement(img);
img.Size: 300, 300;
}
}
... and I need to be able to make that text above target the Windows Scripting Host. I know this can be done, because there used to be a lot of Docs on it around the net a while back, but I cannot seem to find them now.
Can somebody please help, or get me started in the right direction?
Thanks

You're making a domain-specific language which does not exist. You want to translate to another language. You will need a proper scanner and parser. You've probably been told to look at antlr. yacc/bison, or gold. What went wrong with that?
And as an FYI, it's a fun exercise to make new languages, but before you do for something like this, you might ask a good solid "why? What does my new language provide that I couldn't get any other (reasonable) way?"

The thing to understand about parsing and language creation is that writing a compiler/interpreter is primarily about a set of data transformations done to an input text.
Generally, from an input text you will first translate it into a series of tokens, each token representing a concept in your language or a literal value.
From the token stream, you will generally then create an intermediate structure, typically some kind of tree structure describing the code that was written.
This tree structure can then be validated or modified for various reasons, including optimization.
Once that's done, you'll typically write the tree out to some other form - assembly instructions or even a program in another language - in fact, the earliest versions of C++ wrote out straight C code, which were then compiled by a regular C compiler that had no knowledge of C++ at all. So while skipping the assembly generation step might seem like cheating, it has a long and proud tradition behind it :)
I deliberately haven't gotten into any suggestions for specific libraries, as understanding the overall process is probably much more important than choosing a specific parser technology, for instance. Whether you use lex/yacc or ANTLR or something else is pretty unimportant in the long run. They'll all (basically) work, and have all been used successfully in various projects.
Even doing your own parsing by hand isn't a bad idea, as it will help you to learn the patterns of how parsing is done, and so then using a parser generator will tend to make more sense rather than being a black box of voodoo.

Languages similar to C# are not easy to parse - there are some naturally left-recursive rules. So you have to use a parser generator that can deal with them properly. ANTLR fits well.
If PEG fits better, try this: http://www.meta-alternative.net/mbase.html

So you want to translate C# programs to JavaScript? Script# can do this for you.

Rather than write your own language and then run a translator to convert it into Javascript, why not extend Javascript to do what you want it to do?
Take a look at jQuery - it extends Javascript in many powerful ways with a very natural and fluent syntax. It's almost as good as having your own language. Take a look at the many extensions people have created for it too, especially jQuery UI.

Assuming you are really dedicated to do this, here is the way to go. This is normally what you should do: source -> SCANNER -> tokens -> PARSER -> syntax tree
1) Create a scanner/ parser to parse your language. You need to write a grammar to generate a parser that can scan/parse your syntax, to tokenize/validate them.
I think the easiest way here is to go with Irony, that'll make creating a parser quick and easy. Here is a good starting point
http://www.codeproject.com/KB/recipes/Irony.aspx
2) Build a syntax tree - In this case, I suggest you to build a simple XML representation instead of an actual syntax tree, so that you can later walk the XML representation of your DOM to spit out VB/Java Script. If your requirements are complex (like you want to compile it or so), you can create a DLR Expression Tree or use the Code DOM - but here I guess we are talking about a translator, and not about a compiler.
But hey wait - if it is not for educational purposes, consider representing your 'script' as an xml right from the beginning, so that you can avoid a scanner/parser in between, before spitting out some VB/Java script/Html out of that.

I don't wan to be rude... but why are you doing this?
Creating a parser for a regular language is a non-trivial task. Just don't do it.
Why don't you just use html, javascript and css (and jquery as someone above suggested)
If you don't know where to begin, then you probably don't have any experience of this kind and probably you don't have a good reason, why to do this.
I want to save you the pain. Forget it. It's probably a BAD IDEA!
M.

Check out Constructing Language Processors for Little Languages. It's a very good intro I believe. In fact I just consulted my copy 2 days ago when I was having trouble with my template language parser.
Use XML if at all possible. You don't want to fiddle with a lexer and parser by hand if you want this thing in production. I've made this mistake a few times. You end up supporting code that you really shouldn't be. It seems that your language is mainly a templating language. XML would work great there. Just as ASPX files are XML. Your server side blocks can be written in Javascript, modified if necessary. If this is a learning exercise then do it all by hand, by all means.
I think writing your own language is a great exercise. So is taking a college level compiler writing class. Good luck.

You obviously need machinery designed to translate langauges: parsing, tree building, pattern matching, target-language tree building, target-language prettyprinting.
You can try to do all of this with YACC (or equivalents), but you'll discover that parsing
is only a small part of a full translator. This means there's a lot more work
to do than just parsing, and that takes time and effort.
Our DMS Software Reengineering Toolkit is a commercial solution to building full translators for relatively modest costs.
If you want to do it on your own from the ground up as an exercise, that's fine. Just be prepared for the effort it really takes.
One last remark: designing a complete language is hard if you want to get a nice result.

Personally I think that every self-imposed challenge is good. I do agree with the other opinions that if what you want is a real solution to a real life problem, it's probably better to stick with proved solutions. However, if as you said yourself, you have an academic interest into solving this problem, then I encourage you to keep on. If this is the case, I might point a couple of tips to get you on the track.
Parsing is not really an easy task, that is way we take at least a semester of it. However, it can be learned. I would recommend starting with Terrence Parr's book on language implementation patterns. There are many great books about compiling and parsing, probably the most loved and hated been the Dragon Book.
This is pretty heavy stuff, but if you are really into this, and have the time, you should definitely take a look. This would be the Robisson Crusoe's "i'll make it all by myself approach". I have recently written an LR parser generator and it took me no more than a long weekend, but that after reading a lot and taking a full two-semesters course on compilers.
If you don't have the time or simply don't want to learn to make a parser "like men do", then you can always try a commercial or academic parser generator. ANTLR is just fine, but you have to learn its meta-language. Personally I think that Irony is a great tool, specially because it stays inside C# and you can take a look at the source code and learn for yourself. Since we are here, and I'm not trying to make any advertisement at all, I have posted a tiny tool in CodePlex that could be useful for this task. Take a look for yourself, it's open-source and free.
As a final tip, don't get scared if someone tells you it cannot be done. Parsing is a difficult theoretical problem but it's nothing that can't be learned, and it really is a great tool to have in your portfolio. I think it speaks very good of a developer that he can write an descent-recursive parser by hand, even if he never has to. If you want to pursuit this goal to its end, take a college-level compilers course, you'll thank me in a year.

Extending C# .NET application - build a custom scripting language or not?

I need to build a scripting interface for my C# program that does system level testing of embedded firmware.
My application contains libraries to fully interact with the devices. There are separate libraries for initiating actions, getting output and tracking success/failure. My application also has a GUI for managing multiple devices and assigning many scripts to be run.
For the testers (non-programmers, but technical), I need to provide a scripting interface that will allow them to come up with different scenarios for testing and run them. They are just going to call my APIs and then return a result to my program (pass/fail and message).
A very basic example of what I want:
TURN_POWER_ON
TUNE_FREQUENCY frequency
WAIT 5
IF GET_FREQUENCY == frequency
REPORT_PASS "Successfully tuned to " + frequency
ELSE
REPORT_FAIL "Failed to tune to " + frequency
ENDIF
TURN_POWER_OFF
Where the reporting, power and frequency functions are provided by my C# libraries.
Will something like IronRuby or IronPython be good for this, or should I just build my own very basic language?
Does the Ruby/Python code get messy when trying to include a bunch of .NET compiled assemblies? I want it to be easy to learn and code for non-programmers and programmers alike.
EDIT:
Thanks for all the great responses. I chose IronPython as the answer since it had the most support, but I'll spend a bit of time with each of IronPython, Boo and IronRuby to see what the testers would prefer to write scripts in.

I've heard very good things about IronPython for exactly this type of scenario. I'd certainly risk spending a few hours on a quick proof-of-concept, to see how it pans out.
Michael Foord will happily wax lyrical about the success of IronPython in similar cases (most specifically for him, for spreadsheet savvy users), and his book covers (IIRC) a few pointers about hosting it from .NET.

You may want to look at Boo, another managed language that runs on the CLR, and which is particularly well suited to build DSL and make your applications scriptable.
The compilation pipeline is directly extensible from the language itself.
Reading the Boo Manifesto is a good starting point if you want to learn more about it.
[Edit] I forgot to mention that Ayende Rahien is writing a complete book on the topic:
Building Domain Specific Languages in Boo

It might be worth considering PowerShell for this sort of task. That can call into .Net just as any of the DLR languages, and has a more natural language type chunking for tasks in its cmdlet (command-let) concept. You have to write the cmdlets in a compiled language at v1 -- in v2 which is being rolled out starting with Win7 and working to older releases in the next few months (v2 for Vista/Win2k8 is at RC now), you can build those in PowerShell directly.

I agree with Marc G, though it's worth mentioning that the general concept is a Domain Specific Langugage. While IronRuby/IronPython aren't strictly domain-specific, they are full-featured and it would leave you to get on with your implementation.
Visual Studio has the DSL tools, and there's the 'M' Grammar stuff you can look into.
But yeah, IronPython.

From the DSL you're going for, I'd recommend to use CUCUMBER with IronRuby.
With Cucumber, testers write tests that look something like that:
Scenario: See all vendors
Given I am logged in as a user in the administrator role
And There are 3 vendors
When I go to the manage vendors page
Then I should see the first 3 vendor names
It is very easy to make this language fit your needs.
Just google "Cucumber and IronRuby" and you'll find several guides and blog posts to get you started.

We are using embedded Iron Python for pricing formula in one of our project. This is how a real sample on how it looks like.
E_DOCUMENT_CHECK = DCPAV * ADS_NUM
E_SPECIFIC_TAX = STV
E_RESOURCE_DEV = RDV
E_LP_ISSUANCE = LPIV
E_ANNUAL_FEES = APFCV * SA * SIDES_NUM
E_SERVICE_FEES= MAX(
MINSFV,
E_DOCUMENT_CHECK+E_SPECIFIC_TAX+E_RESOURCE_DEV+E_LP_ISSUANCE+E_ANNUAL_FEES)
TOTAL= E_DOCUMENT_CHECK+E_SPECIFIC_TAX+E_RESOURCE_DEV+E_LP_ISSUANCE+E_ANNUAL_FEES+E_SERVICE_FEES
It is really straightforward to implement. The Max() function for example is just one of the custom C# method that we import to the IronPython engine and it looks natural to use in a configuration settings.

You could just use C# itself as the scripting language as described here CrowsProgramming - Runtime Scripting in .Net

IronRuby is the most powerful for creating domain-specific languages, because it's syntax is much more flexible and forgiving than python's (your users are going to screw the whitespace up and get annoyed by the mandatory () to call methods).
You could write your sample script in IronRuby and it would look like this:
TURN_POWER_ON
TUNE_FREQUENCY frequency
WAIT 5
if GET_FREQUENCY == frequency
REPORT_PASS "Successfully tuned to " + frequency
else
REPORT_FAIL "Failed to tune to " + frequency
end
TURN_POWER_OFF
Here's a sample of a DSL our Testers are currently using to write automated tests against our UI
window = find_window_on_desktop "OurApplication"
logon_button = window.find "Logon"
logon_button.click
list = window.find "ItemList"
list.should have(0).rows
add_button = window.find "Add new item"
add_button.click
list.should have(1).rows
However, as things stand right now IronPython is much more mature and has much better performance than IronRuby, so you may prefer to use that.
I'd strongly recommend going with either IronPython or IronRuby over creating your own custom language... You'll save an unimaginable amount of effort (and bugs)

Irony could also be a good candidate here.
Here is an example which creates animation for the given image on the fly.
Set camera size: 400 by 300 pixels.
Set camera position: 100, 100.
Move 200 pixels right.
Move 100 pixels up.
Move 250 pixels left.
Move 50 pixels down.
The tutorial for creating such DSL is here:Writing Your First Domain Specific Language
It is an old tutorial but the author still maintains his library here: Github Irony

C# .net Mnemonics and use in general

I'm just starting out with C# and to me it seems like Microsoft Called their new system .Net because you have to use the Internet to look everything up to find useful functions and which class they stashed it in.
To me it seems nonsensical to require procedure/functions written and designed to stand alone ( non instantiated static objects) to have their class not also function as their namespace.
That is Why can't I use Write or WriteLine instead of Console.WriteLine ?
Then when I start to get used to the idea that the objects I am using ( like string) know how to perform operations I am used to using external functions to achieve ( like to upper, tolower, substring, etc) they change the rules with numbers, numbers don't know how to convert themselves from one numeric type to another for some reason, instead you have to invoke Convert class static functions to change a double to an int and Math class static functions to achieve rounding and truncating.. which quickly turns your simple( in other languages) statement to a gazillion character line in C#.
It also seems obsessed with strong typing which interferes somewhat with the thought process when I code. I understand that type safety reduces errors , but I think it also increases complexity, sometimes unnecessarily. It would be nice if you could choose context driven types when you wish without the explicit Casting or Converting or ToStringing that seems to be basic necessity in C# to get anything done.
So... Is it even possible to write meaningful code in notepad and use cl with out Internet access? What ref book would you use without recourse to autocomplete and Network access?
Any suggestions on smoothing the process towards grokking this language and using it more naturally?

I think you're suffering a bit from the fact that you've used to working in one way during some years, and now must take time to get yourself comfortable using / developing in a new platform.
I do not agree with you , that MS hasn't been consistent on the fact that a string knows how it should convert itself to another type, and other datatypes (like ints) do not.
This is not true, since strings do not know for themselves how they should be converted to another type as well. (You can use the Convert class to Convert types to other types).
It is however true that every type in .NET has a ToString() method, but, you should not rely on that method to convert whatever you have to a string.
I think you have never worked in an OO language before, and therefore, you're having some difficulties with the paradigm shift.
Think of it this way: it's all about responsabilities and behaviour. A class is (if it is well designed) responsible for doing one thing, and does this one thing good.

There is no excuse to use notepad to code a modern language. SharpDevelop or Visual C# Express provide the functionality to work with C# in a productive way.
And no, due to the complexity, not using the internet as a source of information is also not a good option.
You could buy a book that introduces you to the concepts of the language in a structured way, but to get up-to-date information, the internet is neccessary.
Yes, there are drawbacks in C#, like in any other language. I can only give you the advice to get used to the language. Many of the drawbacks become understandable after that, even if some of them don't become less annoying. I recommend that you ask clear, direct questions with example code if you want to know how some language constructs work or how you can solve specific problems more efficiently. That makes it easier to answer those questions.

For notepad, I have no useful advice, however I would advise you to use one of the free IDE's, Microsofts Express Editions, or Sharp Develop.
The IDE will speed the groking of the language, at which point, you can switch back to notepad.

Reading your post I was thinking that you worked mostly with C or dynamic languages previously. Maybe C# is just a wrong choice for you, there are IronPython, F# and a bunch of other languages that have necessary functionality (like functions outside of classes etc.)
I disagree with you about consistency. In fact there are small inconsistency between some components of .NET, but most part of FW is very consistent and predictable.
Strong typing is a huge factor in low defect count. Dynamic typing plays nice in small/intermediate projects (like scripts, etc). In more or less complex program dynamism can introduce a lot of complexity.
Regarding internet/autocomplete - I can hardly imagine any technology with size of .NET that doesn't require a lot of knowledge sources.

Programming in c# using notepad is like buying a ferrari to drive in dirt roads.
At least use Visual Studio Express Edition. For what you wrote I understand that you come from a non OO background, try to learn the OO concept and try to use it. You will eventually understand most design decisions made for .Net.
http://en.wikipedia.org/wiki/Object-oriented_programming

Oh boy where do i start with you(this will be a long post hahaha), well, lets go little by little:
"Microsoft called their system .NET because you have to use Intenet...", the reason why is called .NET is because the SUITE OF MICROSOFT LANGUAGUES(and now some other ones too like Phyton and Ruby, etc) CAN CALL ANY LIBRARY or DLLs, example you can "NET"(Network OR CALL) a DLL that was built in Visual Basic, F#, C++ from WITHIN C# or from any of those languagues you can also call(or ".NET") C# libraries. OK ONE DOWN!!!
NEXT ONE: "it seems nonsensical to require....to have their class not also function as their namespace", this is because a Namespace can have AS MANY CLASSES AS YOU WISH, and your question:
"That is Why can't I use Write or WriteLine instead of Console.WriteLine ?".
The reason is because: "Console"(System.Console hense the "Using" statement at the beginning of your program) Namespace is where "Write" and "WriteLine" LIVES!!(you can also FULLY qualify it (or "call It"). (all this seems to me that you need to study C# Syntax), ok NEXT:
"when I start to get used to the idea that the objects...", ok in simple words:
C# is a "Strongly Type-Safe language" so that SHOULD-MUST tell you what "you are getting in to" otherwise STAY WITH "WEAK or NO TYPE SAFE LANGUAGES" LIKE PHP or C , etc. this does NOT means is bad it just MEANS IS YOUR JOB TO MAKE SURE, as i tell my students: "IF YOU NEED AN INT THEN DEFINE AN INT INSTEAD LETTING THE COMPILER DO IT FOR YOU OTHERWISE YOU WILL HAVE A LOT OF BAD BUGS", or in other words do YOUR homework BEFORE DESIGNING A PIECE OF SOFTWARE.
Note: C# is IMPLICITY TYPE SAFE language SO IF YOU WANT YOU CAN RUN IT AS UNSAFE so from then it wiLL be your job to make sure, so dont complain later(for being lazy) when bugs arrive AT RUNTIME(and a lot of times when the customer is already using your crappy software).
...and last but not least : Whey do you wan to shoot yourself by using notepad? Studio Express is FREE, even the database SQL SERVER is FREE TOO!!, unless you work for a company I WILL ASK FOR PRO, ETC. all the "extra" stuff is for large companies, teams, etc, YOU CAN DO 99% OF THE STUFF WITH THE FREE VERSIONS(and you can still buy-update to full version once you want to scalate to Distributed Software or a Large Project, or if your software becomes a big hit, Example: if you need millions of queryes or hits PER SECOND from your database or 100 people are working on same project(code) but for the majority of times for 2 or 3 "normal" developers working at home or small office the FREE ONES ARE ENOuGH!!)
cherrsss!!! (PS: Software Developer since the 80's)