We Keep Coding

Why we need Reflection at all?

I was studying Reflection, I got some of it but I am not getting everything related to this concept. Why do we need Reflection? What things we couldn't achieve that we need Reflection?

There are many, many scenarios that reflection enables, but I group them primarily into two buckets.
Reflection enables us to write code that analyzes other code.
Consider for example the most basic question about an assembly: what types are in it? Assemblies are self-describing and reflection is the mechanism by which that description is surfaced to other code.
Suppose for example you wanted to write a program which took an assembly and did a graphical display of the relationships between the various classes in that assembly, to help you understand that code. There are such tools. They're in Visual Studio. Someone wrote those tools. They did not appear by magic. Reflection is the mechanism designed into the .NET framework that enables you or me or anyone else to write tools that understand code.
Reflection enables us to move compile time bindings to runtime.
Suppose you have a static method Foo.Bar(). When you put a call to Foo.Bar() in your program, you know with 100% certainty that the method you think is going to be called is actually going to be called. We call static methods "static" because the binding from the name Bar to the code that gets called can be understood statically -- that is, without running the program.
Now consider a virtual method Blah() on a base class. When you call whatever.Blah() you don't know exactly which Blah() will be called at compile time, but you know that some method Blah() with no arguments will be called on some type that is the runtime type of whatever, and that type is equal to or derived from the type which declares Blah(). (In fact you know more: you know that it is equal to or derived from the compile time type of whatever.) Virtual binding is a form of dynamic binding, but it is not fully dynamic. There's no way for the user to decide that this call should be to a different method on a different type hierarchy.
Reflection enables us to make calls that are bound entirely at runtime, based entirely on user choices if we like. We pay a performance penalty, and we lose compile-time type safety, but we gain the flexibility to decide 100% at runtime what code we call. There are scenarios where that's a reasonable tradeoff.

Reflection is such a deep part of the .NET framework that you often don't know that you're doing it (see Attributes and LINQ for instance). And when you do know you're doing it, even if it feels wrong, it might be the only way to achieve a particular objective.
Apart from the two broad areas that Eric mentioned here are a few others. There are lots more, these are just some that come to mind immediately.
Serialization (and similar)
Whether you're using XML or JSON or rolling your own, serializing objects is much easier when you don't have to write specific code for each class to enable serialization. Reflection enables you to enumerate the properties in your object that have been flagged for (or not flagged against) serailization and write them to the output.
This isn't about saving state though. Reflection allows us to write generic methods that can produce business output too, like CSV or XLSX files from an arbitrary collection. I get a lot of mileage out of my ToCSV(...) and ToExcel(...) extensions for things like producing downloadable versions of data sets on my web-based reporting.
Accessing Hidden Data
Yes, I know, this is a dodgy one. And yeah, Eric is probably going to slap me for this, but...
There's a lot of code out there - I'm looking at you, ASP.NET - that hides interesting and useful stuff behind private or protected. Sometimes the only way to get them out is to use reflection. Sometimes it's not the only way, but it can be the simpler way.
Attributes
Every time you tag an Attribute onto one of your classes, methods, etc. you are implicitly providing data that is going to be accessed through reflection. Want to use those attributes yourself? Reflection is the only way you can get at them.
LINQ and Other Expressions
This is really important stuff these days. If you've ever used LINQ to SQL, Entity Frameworks, etc. then you've used Expression in some way. You write a simple little POCO to represent a row in your database table and everything else gets handled by reflection. When you write a predicate expression the system is using the reflection model to build structures that are then processed (visited) to build an SQL statement.
Expressions aren't just for LINQ either, you do some really interesting things yourself, once you know what you're doing. I have code to generate line parsers for CSV import that run pretty damn quickly when compiled to Func<string, TRecord>. These days I tend to use a mapper somebody else wrote, but at the time I needed to slice a few more % off the total import time for a file with 20K records that was uploaded to a website periodically.
P/Invoke Marshalling
This one is a big deal behind the scenes and occasionally in the foreground too. When you want to call a Windows API function or use a native DLL, P/Invoke gives you ways to achieve this without having to mess about with building memory buffers in both directions. The marshalling methods use reflection to do translation of certain things - strings and so on being the obvious example - so that you don't have to get your hands dirty. All based on the Type object that is the foundation of reflection.
Fact is, without reflection the .NET framework wouldn't be what it is. No Attributes, no Expressions, probably a lot less interop between the languages. No automatic marshalling. No LINQ... at least in the way we often use it now.

Define attributes as compile-time data?

I'm writing classes to be a "higher-level" representation of some binary structures as binary files, tcp packets, etc.
For this, and for the sake of readability, that would be very nice if I could define some custom attribute to determine some information about each class' field (e.g. the offset of that field in the binary buffer, the size of the field, etc).
I could accomplish this by declaring constant integers, but IMHO the code would be very ugly and mucky. So I thought about using attributes, which are a very elegant way to accomplish what I want. Features like the InteropServices.Marshal actually uses attributes (as StructLayout, MarshalAs and FieldOffset) to accomplish something very similar to what I want, so I can only assume the performance tradeoff is advantageous compared to the gain of readability (please correct me if I'm wrong).
So, how the forementioned InteropServices' attributes are handled by the compiler/CLR?
Do you guys think the forementioned tradeoff is worth? If yes, the best way to deal with the attributes is using the default method using Refletion? I'm assuming that there could be other ways to access attributes rather than Reflection because I know this is a bit expensive and Marshal uses it in almost every method.
Any helpful thought would be very appreciated, thanks.

What you are proposing sounds reasonable assuming the parallels to the Interop are as clear as you are describing. To avoid the performance issues of using reflection for every property access you can use reflection once, maybe via a static constructor, and build compiled Expressions for each property instead. The performance should be the equivalent of calling a virtual method I would think.
Here is a link to a blog post denoting the performance differences between different dynamic invocation types. Compiled expressions are ~10x faster then cached reflection and "only" 2x slower then compiled property access.
http://www.palmmedia.de/Blog/2012/2/4/reflection-vs-compiled-expressions-vs-delegates-performance-comparision

Best way to invoke each property in a class

I have a class in an SDK, for which every property I am in interested in calling. I know that the only way (I think the only way this is), is to use reflection, which most people claim as being slow etc (although I've seen articles which illustrate how in some cases it is not as slow as originally thought).
Is there a better way than to loop through and invoke each property in the target class?
Also, why is reflection deemed to be so slow?

It might be worth taking a looking at TypeDescriptors. As far as I am aware they have some performance benefits over using reflection and work in a slightly different way (they cache metadata for example). The MSDN article confused me in the way it describes how reflection is used by type descriptors, so you might need to find a more expansive explanation (therfore the 3rd link might be more helpful) .
The API for type descriptors is similar to that used for reflection.
Navigate to:
http://msdn.microsoft.com/en-us/library/ms171819.aspx
http://msdn.microsoft.com/en-us/library/system.componentmodel.typedescriptor.aspx
And
http://blogs.msdn.com/b/parthopdas/archive/2006/01/03/509103.aspx
Soom loose answers to your questions then:
1) Because of caching and a slightly different implementation to reflection TypeDescriptors my provide a performance improvement over relfection alone
2) You may be able to retrieve the properties and (invoke/set/get?) the properties in one fell swoop. This may be a case of calling an invoke type method and writing a lambda statement to peform some action on the collection returned?

You can use reflection to generate C# code accessing directly to all properties you are interested in. That would be a faster way to perform the calls.
I think Reflection is not a bad option for you though. It's not that slow.

I would use reflection to generate the code that calls all the properties. Then you don't have to worry about reflection being slow.

Reflection.Emit better than GetValue & SetValue :S

I've been told to use Reflection.Emit instead of PropertyInfo.GetValue / SetValue because it is faster this way.
But I don't really know what stuff from Reflection.Emit and how to use it to substitute GetValue and SetValue. Can anybody help me with this ?

Just an alternative answer; if you want the performance, but a similar API - consider HyperDescriptor; this uses Reflection.Emit underneath (so you don't have to), but exposes itself on the PropertyDescriptor API, so you can just use:
PropertyDescriptorCollection props = TypeDescriptor.GetProperties(obj);
props["Name"].SetValue(obj, "Fred");
DateTime dob = (DateTime)props["DateOfBirth"].GetValue(obj);
One line of code to enable it, and it handles all the caching etc.

If you're fetching/setting the same property many times, then using something to build a typesafe method will indeed be faster than reflection. However, I would suggest using Delegate.CreateDelegate instead of Reflection.Emit. It's easier to get right, and it's still blazingly fast.
I've used this in my Protocol Buffers implementation and it made a huge difference vs PropertyInfo.GetValue/SetValue. As others have said though, only do this after proving that the simplest way is too slow.
I have a blog post with more details if you decide to go down the CreateDelegate route.

Use PropertyInfo.GetValue/SetValue
If you have performance problems cache the PropertyInfo object (don't repeatedly call GetProperty)
If - and only if - the use of reflection is the performance bottleneck of your app (as seen in a profiler) use Delegate.CreateDelegate
If - and really really only if - you are absolutely sure that reading/writing the values is still the worst bottleneck it's time to start learning about the fun world of generating IL in runtime.
I really doubt it's worth it, each of those levels increase the complexity of the code more then they improve performance - do them only if you have to.
And if runtime access to properties is your performance bottleneck it's probably better going for compile time access (it's hard time to be both generic and super high performance at the same time).

The purpose of Reflection.Emit is completely different from that of PropertyInfo.Get/SetValue. Via Reflection.Emit, you can directly emit IL code, for example into dynamically compiled assemblies, and execute this code. Of course, this code could access your properties.
I seriously doubt that this will be much quicker then using PropertyInfo in the end, and it's not made for this purpose either. You could use Reflection.Emit as the code generator for a small compiler, for example.

Using Reflection.Emit seems a little too "clever", as well as a premature optimization. If you profile your application, and you find that the GetValue/SetValue Reflection is the bottleneck, then you could consider optimizing, but probably not even then...

How do you optimize the performance of getting/setting attribute-decorated properties?

In C#, I'm marking some classes' properties with attributes and I'm using reflection to find these properties in order to perform gets and sets on them. I've found, however, that getting/setting with reflection in this manner is approximately 10x as slow as POCO gets/sets. Besides dropping the fundamental scenario described above to use alternate techniques, are there any documented tricks to make this significantly performant, such as caching techniques of some sort?

Going beyond what casperOne has said (including the bit about checking that this is the bottleneck), you may find it very helpful to convert the getters/setters into delegates (a Func<T> and an Action<T> for the getter and setter respectively) using Delegate.CreateDelegate. This can make a massive difference, and isn't terribly hard. If you're already going to cache the PropertyInfo, just cache the delegate pair instead.
I've got a blog post about Delegate.CreateDelegate - I first used it in anger when porting Protocol Buffers (which can be reflection-heavy at times). It helped a lot in that case.

Well, you could always store the PropertyInfo instance for the property once you get it and map it to whatever key works for you. Since the type is static, the PropertyInfo is not going to change, and storing a local reference is not going to kill you.
Assuming you don't do anything foolish to cache the value (like placing all the PropertyInfo instances in one long list and iterate the list every time to find it), you should be ok.
Of course, make sure you are not suffering from premature optimization disease before you go down this path. Identify that constantly calling GetProperty/GetProperties on your Type instance is a bottleneck in your application before making changes.