We Keep Coding

A New and Full Implementation of Generic Intellisense

I am interested in writing a generic Intellisense enabled editor for SQL and C# (et al. if possible!). I would like to do this in C# as an overridden or extended WPF richTextBox-type control. I know there are many example projects available and I have implemented a basic version of my own; but most of the examples that I have come across (and indeed my own) are just that, basic.
A couple of code examples are:
DIY Intellisense By yetanotherchris
CodeTextBox - another RichTextBox control with syntax highlighting and Intellisense By Tamas Honfi
I have however, found a great example of an SQL editor with Intellisense QueryCommander SQL Editor By Mikael Håkansson which seems to work well. Microsoft must use a XML library of command keywords, but my question is: How (in detail) do Microsoft implement their Intellisense (as-you-type Intellisense) and how hard would it be for me to create my own of the same standard?
Edit A: A year on and I have managed to develop my own editor control with basic intellisense mainly for my own "enjoyment". I thought I would come back provide a list of freely available .NET projects that helped me with my own development and can be used out-of-the-box and free of charge:
ICSharpCode (WinForms)
AvalonEdit (WPF)
ScintillaNET (WinForms)
Query Commander [for example of intellisense implementation] (WinForms)
Edit B: 15 months after the question was asked I am still looking for new improved editors. This one is nice...
RoslynPAD is cool!
Edit C: 2 years+ on from the question, I have found the following projects, both using WPF and backed by AvalonEdit.
CodeCompletion for AvalonEdit using NRefactory. This project is really nice and has a full implementation of intellisense using NRefactory.
ScriptCS ScriptCS makes it easy to write and execute C# with a simple text editor.

How (in detail) do Microsoft implement their as-you-type Intellisense?
I can describe it to any level of detail you care to name, but I don't have the time for more than a brief explanation. I'll explain how we do it in Roslyn.
First, we build an immutable model of the token stream using a data structure that can efficiently represent edits, since obviously edits are precisely what there are going to be a lot of.
The key insight to making it efficient for persistent reuse is to represent the character lengths of the tokens but not their character positions in the edit buffer; remember, a token at the end of the file is going to change position on every edit but the length of the token does not change. You must at all costs minimize the number of total re-lexings if you want to be efficient on extremely large files.
Once you have an immutable model that can handle inserts and deletions to build up an immutable token stream without re-lexing the entire file every time, you then have to do the same thing, but for grammatical analysis. This is in practice a considerably harder problem. I recommend that you obtain an undergraduate or graduate degree in computer science with an emphasis on parser theory if you have not already. We obtained the help of people with PhDs who did their theses on parser theory to design this particular bit of the algorithm.
Then, obviously, build a grammatical analyzer that can analyze C#. Remember, it has to analyze broken C#, not correct C#; IntelliSense has to work while the program is in a non-compiling state. So start by coming up with modifications to the grammar that have good error-recovery characteristics.
OK, so now you've got a parser that can efficiently do grammatical analysis without re-lexing or re-parsing anything but the edited region, most of the time, which means that you can do the work between keystrokes. I forgot to mention, of course you will need to come up with some mechanism to not block the UI thread while doing all of these analyses should the analysis happen to take longer than the time between two keystrokes. The new "async/await" feature of C# 5 should help with that. (I can tell you from personal experience: be careful with the proliferation of tasks and cancellation tokens. If you are careless, it is possible to get into a state where there are tens of thousands of cancelled tasks pending, and that is not fast.)
Now that you've got a grammatical analysis you need to build a semantic analyzer. Since you are only doing IntelliSense, it does not need to be a particularly sophisticated semantic analyzer. (Our semantic analyzer must do an analysis suitable for generating code from correct programs and correct error analysis from incorrect programs.) But of course, again it has to do good semantic analysis on broken programs, which does increase the complexity considerably.
My advice is to start by building a "top level" semantic analyzer, again using an immutable model that can persist the state of the declared-in-source-code types from edit to edit. The top level analyzer deals with anything that is not a statement or expression: type declarations, directives, namespaces, method declarations, constructors, destructors, and so on. The stuff that makes up the "shape" of the program when the compiler generates metadata.
Metadata! I forgot about metadata. You'll need a metadata reader. You need to be able to produce IntelliSense on expressions that refer to types in libraries, obviously. I recommend using the CCI libraries as your metadata reader, and not Reflection. Since you are only doing IntelliSense, obviously you don't need a metadata writer.
Anyway, once you have a top-level semantic analyzer, then you can write a statement-and-expression semantic analyzer that analyzes the types of the expressions in a given statement. Pay particular attention to name lookup and overload resolution algorithms. Method type inference will be particularly tricky, especially inside LINQ queries.
Once you've got all that, an IntelliSense engine should be easy; just work out the type of the expression at the current cursor position and display a dropdown appropriately.
how hard would it be for me to create my own of the same standard?
Well, we've got a team of, call it ten people, and it'll probably take, call it five years all together to get the whole thing done from start to finish. But we have lots more to do than just the IntelliSense engine. That's maybe only 40% of the work. Oh, and half those people work on VB, now that I think about it. But those people have on average probably five or ten years experience in doing this sort of work, so they're faster at it than you will be if you've never done this before.
So let's say it should take you about ten to twenty years of full time work, working alone, to build a Roslyn-quality IntelliSense engine for C# that can do acceptably-close-to-correct analysis of large programs in the time between keystrokes.
Longer if you need to do that PhD first, obviously.
Or, you could simply use Roslyn, since that's what it's for. That'll take you probably a few hours, but you don't get the fun of doing it yourself. And it is fun!
You can download the preview release here:
http://www.microsoft.com/download/en/details.aspx?id=27746

This is an area where Microsoft typically produces great results - Microsoft developer tools really are awesome. And there is a clear commercial advantage for sales of their developer tools and for sales of Windows to having the best intellisense so it makes sense for Microsoft to devote the kind of resources Eric describes in his wonderfully detailed answer. Still, I think it's worth pointing out a few of things:
Your customers may not actually need all the features that Microsoft's implementation provides. The Microsoft solution might be incredibly over-engineered in terms of the features that you need to provide to your customers/users. Unless you're actually implementing a generic coding environment that is intended to be competitive with Visual Studio, it is likely that there are aspects of your intended use that either simplify the problem, or that allow you to make compromises on the solution that Microsoft feels they cannot make. Microsoft will likely spend resources decreasing response times that are already measured in hundreds of milliseconds. That may not be something you need to do. Microsoft is spending time on providing an API for others to use for code analysis. That's likely not part of your plan. Prioritize your features and decide what "good enough" looks like for you and your customers then estimate the cost of implementing that.
In addition to bearing the obvious costs of implementing requirements that you may not actually have, Microsoft also carries some costs that may not be obvious if you haven't worked in a team. There are huge communication costs associated with teams. It's actually incredibly easy to have five smart people take longer to produce a solution than it takes for a single smart person to produce the equivalent solution. There are aspects of Microsoft's hiring practices and organizational structure that make this scenario more likely. If you hire a bunch of smart people with egos and then empower all of them to make decisions, you too can get a 5% better solution for 500% of the cost. That 5% better solution might be profitable for Microsoft, but it could be deadly for a small company.
Going from a 1 person solution to a 5 person solution increases the costs, but that's just the intra-team development costs. Microsoft has separate teams that are devoted to (roughly) design, development, and testing even for a single feature. The project-related communication between peers across these boundaries has higher friction than within each of the disciplines. This not only increases communication costs between individuals, but it also results in larger team sizes. And more than that - since it's not a single team of 12 individuals, but is instead 3 teams of 5 individuals, there is 3x the upward communication cost. More costs that Microsoft has chosen to carry that may not translate to similar costs for other companies.
My point here is not to describe Microsoft as an inefficient company. My point is that Microsoft makes a ton of decisions about everything from hiring, to team organization, to design and implementation that start from assumptions about profitability and risk that simply do not apply to companies that are not Microsoft.
In terms of the intellisense thing, there are various ways of thinking about the problem. Microsoft is producing a very generic, reusable solution that doesn't just solve intellisense, but also targets code navigation, refactoring, and various other uses for code analysis. You don't need to do things the same way if your sole goal is to make it easy for developers to enter code without having to type much. Targeting that feature doesn't take years of effort and there are all sorts of creative things you can do if you're not just providing an API, but you actually control the UI too.

Fast easy refactoring in visual studio

I have a large project that is quite a mess. There's God classes, poor variable naming, poor indentation and spacing...among other issues.
I'm looking for a quick and easy way to make marketable improvements to such a project. In Visual Studio 2010 CTRL + K and CTRL + D gives me fast code cleanup. Organize usings takes out unused namespaces. But, both only seem to work on 1 page. Can I apply these changes to a project? Are there any similar cleanup techniques or free and reliable 3rd party addons?

Try out Resharper-- it can do what you're looking for.

One third party extension that I use that is free is DevExpress' Coderush Xpress (For C# and VB), which is the free edition of their full product. The great thing about this is that the express edition still lets you add CodeRush plugins to it (in addition to several project-wide refactoring options), which opens you up to a ton more options than just the Visual Studio Plugins.
Here is a huge list of the plugins available for CR Xpress.

I have been using Visual Assist X for quite some time, and I found it very useful. It is not free, and it may not automate the sweeping changes you seem to be looking for, but it makes simple refactoring much easier.
P.S. I have been using it for C++, not C#. But if anything, I would assume that it would have more capabilities for C#.

Are Visual Studio Express products really only for "hobbyists, students and novices"?

I have used Visual Studio Professional 2008, and have been testing the free C# Express 2010 version recently. In general I'm amazed at how good it is for free, and how many of the full VS features it has. I'm thinking of using it for a commercial program and I know the license allows for that, it's just the description of it being for "non-professional developers like hobbyists, students and novice developers" concerns me a bit.
What I'm interested in knowing is what is stopping it being 'professional', that is:
Have you evaluated the express edition, and found a specific useful feature lacking that stopped you from using it ? Or did you initially use the express versions, but upgraded to full VS because of a feature lacking ? If so, what was that feature ?
I've searched for similar questions and found lists of differences between the full VS and express versions, but I'm more interested in knowing peoples personal experiences with it. It seems like many of the extra features in VS target developers working in large teams, so I'm mainly interested in hearing from either solo or small team developers where it seems like there's less compelling reasons to upgrade.
The limitations I've personally encountered are:
Extensions not being supported, though I can still use DotTrace, NUnit and an obfuscator outside of the VS integration, albeit it's a bit less convenient.
Limited refactoring, although the "Rename" and "Extract Method" are still there and I think they're the most useful. Edit: Not having 'Encapsulate Field' in Express used to be annoying though, though the introduction of automatic getters and setters has pretty much canceled that out I think.
More limited debugging for multi-threaded apps.
Edit: Another is that you can't easily switch between targeting "Any CPU/x86/x64" in Express like you can in VS. It is possible, but needs manually editing your project file to do so.
But the pluses seem to outweigh the minuses so far. Is there anything you found was a deal-breaker for you ?
Update: To come back to this a couple of months later, and after releasing a product built with the Express version, it is indeed possible to program professionally with the Express versions. The limitations within the program itself are pretty minor and can be worked around, but I've increasingly come to realise it's really the "extensions not supported" one that's the (only) major drawback. No Resharper, CodeRush or the like, and no source control, profiling, database explorer or unit testing and the like within VS itself. It's more of a productivity drag than a deal breaker, but it is annoying to come across a cool looking VS extension only to see that "Not supported in Express versions" notice.
If anyone else is in the same situation, I'd evaluate the extensions you use (or might be planning to use) first and see how important they are to you. Express is fine if you don't use any extensions or could live without them without a significant drop in productivity, otherwise stick with the Professional version.

I've used express for 5 years, it has everything you need to produce professional projects. There's one important feature that's missing, which is the extensible Database Explorer, e.g. if you need to generate an Entity Framework model from a MySQL database. But for that I use Visual Studio Shell (also free). For other stuff such as unit testing and source control I use other tools. You really don't need Visual Studio to have everything you need to manage a software project.

No add-ins => No ReSharper => No dice.

The biggest problem with the Express version is you don't have any source control. You can't professionally work without it, even if you are a lone freelancer. Keeping a change history on a project is vital when you mess something up and want to go to a previous version or simply want to compare different revisions.
Also I think in Express you can't have web and desktop applications in the same solution, or do other types of combination of projects. I'm not really sure about this one, because I don't use Express (for the first reason :))

Other answers have covert the fact that it can very well be enough, but if you are more than a hobbyists, students or novices, just get enrolled into the Microsoft BizSpark program.
You get all the tools for free (Visual Studio Ultimate, Team Foundation Server, All Windows Versions etc.) and you don't even need to be a company. Enrollment is super easy and uncomplicated.
The only requirements are (taken from the homepage):
You develop software
Privately held
Less than 3 years in business
< 1mio $ per year
This is not an ad. We have been very happy with the program and got access to everything we needed. Just wanted to let people know it existed.
Additional info:
You can be member of this program for 3 years
When you leave the program you have to pay a 200$ fee, its made for startups afterall

I had been using VS Express for several years but I recently upgraded to VS Ultimate. There is nothing that you cannot do in Express, but the integration of features in the full product makes everything more convenient.
When you have source control, unit tests, performance/code analysis, installers all in the same solution, it saves a lot of time and hassle jumping between different programs. Also one big issue for me was not being able to mix features that are spread across the different Express editions, such as mixing ASP.NET with WinForms, or even VB.NET with C#.
I got a lot of use out of Express, it's a great product, but I was very happy the day I got to uninstall ten programs and replace them with one.

The pros:
Absolutely free
Has enought functinality for you to develop commercial products (you know the saying: you need 20% of anything to get 80% done). I developed and deployed a wp7 (windows phone 7) app with the express edition and it was a pleasant experience.
The cons:
No integration with source control. I use svn for my projects and i have to use tortoise svn outside of vs to manage the code. Its a minor annoyance, not a deal breaker.
Can't develop web/desktop/mobile in the same solution. Its easier to have them all in the same project when you have a common dll for your business logic. Unfortunately with express edition you will end up using copy/paste for moving dlls that are common between projects. Again it's a minor annoyance, not a deal breaker.

Professional is not the big step up here - Ultimate is. VS2010 Ultimate, for example, comes with bunches of profilers and that kind of thing. They give Professional away to any old student, for example.

There's only one plus: the cost. Every other difference is a minus (or at best reduces the cost of the minimum useful system). That means that, if the extra features are worth the money, buy the higher-end version.
The free lower-end version is great for (as they say) students, hobbyists, and novices. Students often don't have much money, and hobbyists and novices often don't want to spend all that much money on their hobbies and introductions.
However, a software professional is making money writing software, and presumably will make more money if he or she becomes more efficient. Assuming our professional is making decent money, the cost of an upper-end version of VS isn't large compared to revenues, and if it leads to even modest productivity increases it's worth it.
When you're doing something for money, don't try to do things too much on the cheap. Taking more time and inconvenience to do things will cost real money in that case, and avoiding it is worth real money.

Microsoft's marketing approach to these Express editions may be is to introduce Visual studio flavors to users (any user). Later, bring them on board with more powerful versions. Also, Express editions allows you to write code anywhere without getting worried about licensing part. You can use them to try out your test bed projects, community projects etc. I must re-iterate again that free licensing is the biggest draw towards Express editions

The only substantive differences I've found are the integration of code control and the ability to create heterogeneous projects. The first is not a major issue if you are the only developer. With the Express versions, You can use whatever code control mechanism you like, just not built in.
The second is only an issue if you want to develop using some combination of C++, C# or VB in the same project. You need at least VS2010 Professional to do that. Given the extensive features in C#, the absolute need to use C++ or VB with it in the same project is highly unlikely. However, those situations exist if you need C++ in a C# project for some low level API or have a legacy VB system and need some feature from C# or C++ that is just too difficult to manage in VB.
As a friend who built racing engines once observed: Speed costs money. How fast do you want to go? Express is free, the others can be pricey, although the price for VS2010 Professional just plummeted with the advent of VS2012.

Why should I upgrade to c# 4.0?

I know there are some nice new features in C# 4.0 but I can't, for the life of me, think of a compelling reason for either upgrading existing projects or for switching to new projects.
I've seen some posts where people have said that if their hosting service didn't provide .Net 4 that they'd find another provider as .Net 4 was pinicle to their direction <?>.
Now my boss is trying to get me to agree to switch all our production environments to C# 4 and to do it now.
So the question is has anyone either began using, or converted a project to, C# 4 for a compelling reason? Was there a feature that you just had to have that would make your life so much easier?

There are some cool new features in C# 4.0:
Dynamic member lookup
Covariant and contravariant generic type parameters
Optional ref Keyword when using COM
Optional parameters and named arguments
Indexed properties
In his release blog post Scott Guthrie goes into detail about the features of .NET 4 in general. Another great resource is a white paper at http://www.asp.net/learn/whitepapers/aspnet4. However, I'd doubt you are going to need one / any of these new features right away. As Scott Hanselman blogged:
there's a lot of stuff that's new and
added in .NET 4, but not in that
"overwhelming-I-need-to-relearn-everything"
way.
Whether or not you should upgrade is therefore dependent on a variety of other factors. Some reasons that spring to mind:
Standardizing your development environment on a single platform VS2010 over VS2008.
Size of the .NET Framework is substantially reduced
Speed improvements if you are a Visual Studio Tools for Office developer
An open dialogue with your manager seems appropriate to understand his reasoning for the upgrade. I'd argue that because it's shiny isn't a compelling reason.
As a reference this dated Stack Overflow question "Why not upgrade to the latest .net framework" provides the inverse to your question.

Quite frankly System.Collections.Concurrent has made developing multi-threaded applications a breeze.
The new and improved System.Linq.Expressions makes writing dynamically compiled code seem like child's play.
The new named parameters feature means I can have big constructors and not get confused as to what each parameter is. Immutable objects are just that much easier.

Surprisingly not mentioned:
PLINQ
Task Parallel Library

Is your question specific to C# 4.0, or .NET 4.0?
In C# 4.0 there are only a couple of really nice new features. Covariance/contravariance is not useful all the time, but when you run into a need for it, it can really save a lot of pain. Optional method parameters can reduce a lot of ugly method overrides, and make certain method calls a lot cleaner. If you're using COM or IronPython or any of a few similar frameworks, the dynamic keyword can also be a real lifesaver.
.NET 4.0 in general has a ton of really interesting features across a variety of frameworks. Foreign Key support in Linq to Entities, for example, is making life a lot easier for us. A lot of people are really excited about POCO support. They also added support for some of the LINQ methods (e.g. Distinct) that were previously missing from the Entity Framework.
So it will really all boil down to which frameworks you're using and how you're using them, and how expensive it will be for you to make the switch.

First, what is compelling to me may mean nothing to you. Having said that, I would upgrade Visual Studio if budget allows. In fact, personally I think there is a huge career risk in staying with a company that doesn't keep your tools up to date. You will fall behind in your knowledge of the field without access to the latest tools.
As for converting all your projects just to convert them it seems like folly to me. Putting aside all the extra work distribution (and upgrading the machines to have .NET 4), you have to consider the chance that you will have something go wrong. (And if you are like me some things must be called from 3rd party programs using .NET 3.5 making them unable to convert.)
My first rule would be that nothing is converted unless you are working on it anyway. But I would seriously look to convert anything that could use improvement from either parallel code, or COM interop.
I do have a compelling project that was converted. I had a long running web method being called. In the version that exists now, I return from the method without knowing the results. Instead I gave the user a way to check later. By moving to a parallel foreach loop this works much better and I can let the user know if there were any errors.
The same project is also being converted to use RIA services which have greatly improved and reducing the amount of my own code.

I upgraded for the same reason everyone else did.
...so I can put it on my resume :)

If you're starting a new project today, it's probably best to start it on 4.0, since down the road you will have to migrate it at some point anyways (assuming it stays around long enough, older versions of .net will simply stop being supported).
C# 4 implies other things.. depending on your project... WCF 4, WPF 4, ASP.NET 4, MVC 2, Entity Framework 2, etc.. So don't just look at C# as the reason to change, you also have to look at the whole stack. If there's still nothing compelling, then staying where you're at is probably a wise choice.

If you're doing WPF / Silverlight, I would definitely recommend upgrading to Visual Studio 2010 (I know, you can write .NET 4.0 code without an IDE, but that's an edge case if ever there were one).
The multi monitor support is nifty but buggy. I spend a lot of time trying to get windows to refresh.
In terms of language, the COM interop (as #Gvs mentioned) is also vastly improved with the dynamic datatype and optional parameters.
UPDATE: Multiple monitor support is pretty rock solid with VS 2010 SP1.

If you can get your boss to pop for the $10,000+ Visual Studio Ultimate Edition, IntelliTrace is a compelling reason to upgrade your environment and justification enough the for investment.

COM integration is much easier with the dynamic datatype, and optional parameters.

For me there are two things:
optional arguments -- because I am sick of polluting classes with X versions of the same method (overloading)
dynamic keyword -- because the expressiveness of generics in C# is a joke, this way I can at least "write what I mean" without hoops, of course with execution speed penalty
The more compact the code (i.e. if you express the idea without addition "oh, how to avoid limitation Y of the language"), the better, because the code is much easier to maintain and it is harder to make a stupid (or worse) mistake.

There are no compelling stability or security reasons to switch. Shouldn't that be your boss's concern?

Visual Studio Code Analysis - Does Microsoft follow it themselves?

Did a quick search but could not find anything about this.
I guess all of you know that the Visual Studio Code Analysis is quite nitpicking and gives warnings about a lot of things. Does anybody know how well Microsoft follow this themselves..? That is, if I were to run a code analysis on their assemblies, would the warnings be none or very few (perhaps surpress warning with a justification..?).

Most of the things that code analysis (or FXCop) check are closely based on the ".NET Framework Library Design Guidelines" (either the book, or on MSDN).
However those guidelines (especially in the book) have caveats, not all apply in all circumstances.
There are plenty of indications that MS do use these tools, but I assume they do have reasons to not apply all the rules all the time, as any other project does.

There are two core tools used widely at Microsoft for Code Analysis: FXCop for managed code and PreFast for native C++.
Historically, while not every team has thoroughly enforced the use of CA when building their products, there's been a significant upswing over the last 3-4 years in particular in the number of teams that now enforce pretty stringent CA requirements on their feature teams and on the product as a whole.
For example, in Vista, the Windows team essentially took 3 months off product development and SAL-annotated the vast majority of their key method and function declarations. in Win7, they mandated that all new code had to comply with a set of requirements for SAL-annotating key scenarios (primarily to reduce the likelihood of buffer overruns). In Win8 they're going further still and are incorporating new SAL annotations for a number of key scenarios. Combined with improved compilers and tools like PreFast (now build into VS 2010 Pro and up), they and you can find and eliminate potential issues before the product is released.
Note that the warnings issues by CA (whichever CA tool you choose to use) will always require overrides - sometimes, there's a really good reason as to why the code has to do what it does. But you should only override if you're ABSOLUTELY sure it's necessary and appropriate. NEVER turn off a warning because you don't understand it and never turn off a warning if you can't be bothered to fix it.