We Keep Coding

C# Custom Encoding and Decoding/Serialization of Values into a Stream

I am stuck with the following problem: I am communicating with another application using TcpClients/Streams.
This application needs input parameters passed to it, encoded in a special way (something along first byte: parameter type identifier, next 4 bytes: content length; next length bytes: content).
The application processes the input and gives me the output back in the same format. My problem is that I usually don't know the exact order of different return values. The application could return nothing, one int, 5 ints, 2ints and an IntArray...
EDIT: To clarify this, I can not change the underlying protocol or the source code of the application I am communicating with!
The problem is that I want to be able to use the params as if they were "native" C# objects.
What I tried achieving is the following approach:
https://dotnetfiddle.net/ZjbUix
I am not really sure whether this is a good idea. Maybe it would be a better alternative to have a EncodingStream(NetworkStream) with Write() methods for all possible parameter types, and a DecodingStream(NetworkStream) with GetInt(), GetIntArray() etc. methods. This would, however, more or less disregard the type identifier and could only be prevented with throwing an Exception along "Attempted to read an Int value but found Int Array parameter type".
EDIT: I implemented this version here: https://dotnetfiddle.net/8WVXPz
Another requirement is that it should be possible to easily add new parameter types while at best detecting errors at compile-time. Additionally, I'd prefer a non-reflection solution.
Maybe someone can make up an even better architecture to support this?

How to properly work with non-primitive ClrInstanceField values using ClrMD?

I've got some really large memory dumps of a managed process that I'm trying to get a lot of statistics from--as well as be able to present an interactive view of--fairly deep object graphs on the heap. Think something comparable to !do <address> with prefer_dml 1 set in WinDbg with SOS, where you can continually click on the properties and see their values, only in a much friendlier UI for comparing many objects.
I've found Microsoft.Diagnostics.Runtime (ClrMD) to be particularly well suited for this task, but I'm having a hard time working with array fields and I'm a little confused about object fields, which I have working a little better.
Array:
If I target an array with an address directly off the heap and use ClrType.GetArrayLength and ClrType.GetArrayElementValue things work fine, but once I'm digging through the fields on another object, I'm not sure what value I'm getting from ClrInstanceField.GetValue when the ClrInstanceField.ElementType is ClrElementType.SZArray (I haven't encountered Array digging around in my object graph yet, but I should like to handle it as well).
Edit: I just decided to use the ClrType for System.UInt64 to dereference the array field (using parent address + offset of the array field to calculate the address where the array pointer is stored), then I can work with it the same as if I got it from EnumerateObjects. I am now having some difficulty with some arrays not supporting the ArrayComponentType property. I have yet to test with arrays of Structs so I am also wondering if that will be a C-style allocation of inline structs, as it is with int[] or if it will be an array of pointers to structs on the heap. Guid[] is one of the types I'm having an issue getting the ArrayComponentType from.
Object: Fixed (logic error)
With a ClrInstanceField that has a Type of ClrElementType.Object I get much better results, but still need a little more. Firstly, after calling GetFieldValue I get back a ulong address(?) which I can use ClrInstanceField.Type.Fields against just fine, so I can see the field names and values of the nested object. That said, I have to account for polymorphism, so I tried using ClrHeap.GetObjectType on the same address and it either returns NULL or something completely incorrect. It seems odd that the address would work in my first use case, but not the second.
String: Fixed (found workaround)
Because my real project already uses DbgEng w/ SOS, I have a different way to easily get the value of strings by address, but it seemed very odd that trying to use ClrInstanceField.GetFieldValue succeeded in returning a string, but with completely inaccurate results (a bunch of strange characters). Maybe I'm doing this wrong?
Edit: I have extracted an abstraction that now runs in LINQPad from my original code. It's a bit long to post here, but it's all here in a gist. It's still a little messy from all the copy/paste/refactor and I'll be cleaning it up further an likely posting the final source on either CodePlex or GitHub after I've got these issues fixed.
The code base is fairly large and specific to a project, but if it's absolutely necessary I may be able to extract out a sample set. That said, all access to the ClrMD objects is fairly simple. I get the initial addresses from SOS commands like !dumpheap -stat (which works fine for the root objects) and then I use ClrHeap.GetTypeByName or ClrHeap.GetObjectType. After that it relies exclusively on ClrType.Fields and ClrInstanceField members Type, ElementType, and GetFieldValue
As an added bonus, I did find a browser friendly version of the XML Docs provided with the NuGet package, though it's the same documentation IntelliSense provides.

It's going to be hard to answer very precisely without seeing what your code looks like, but basically, it goes like this:
The first thing you need to know in order to be able to call GetFieldAddress/GetFieldValue is if the object address you have is a regular pointer or an interior pointer. That is, if it directly points to an object on the heap, or to an interior structure within an actual object (think String vs. Struct field within an actual object).
If you're getting the wrong values out of GetFieldAddress/GetFieldValue, it usually means you're not specifying that you have an interior pointer (or you thought you had one when you didn't).
The second part is understanding what the values mean.
If field.IsPrimitive() is true: GetFieldValue() will get you the actual primitive value (i.e. an Int32, Byte, or whatever)
If field.IsValueClass() is true, then GetFieldAddress() will get you an interior pointer to the structure. Thus, any calls on GetFieldAddress/Value() that you use on that address you need to tell it that it is an interior pointer!
If field.ElementType is a ClrElementType.String, then I seem to remember you need to call GetFieldValue will get you the actual string contents (need to check, but this should be it).
Otherwise, you have an object reference, in which case GetFieldValue() will get you a regular pointer to the new reference object.
Does this make sense?

How to marshal a COM-Parameter as VT_ARRAY of VT_RECORD

I've already done some extensive search, but I can't seem to find anything matching my problem.
The task I'm currently working on is to create a WCF-Wrapper for some DCOM-Objects. This already works great for the most parts, but now I'm stuck with one invocation that expects a VT_ARRAY containing VT_RECORD-Objects.
Marshalling as VT_ARRAY is not a problem, but how can I tell COM that the elements in this array are VT_RECORDs?
This is the invocation as I current use it.
InitTestCase(testCaseName, parameterFileName, testCase, cellInfos.ToArray());
The parameter I'm talking about is the last one. It's defined as List<CellInfo>, CellInfo itself is already attributed with Guid("7D422961-331E-47E2-BC71-7839E9E77D39") and ComVisible(true). It's not a struct but a class.
This is the condition failing on the native side: if (VT_RECORD == varCellConfig.vt)...
Because of old software using these interfaces, changing the native side is not an option
Any idea?

What's the practical difference between a variable being mutable vs non-mutable

I'm just learning C# and working with some examples of strings and StringBuilder. From my reading, I understand that if I do this:
string greeting = "Hello";
greeting += " my good friends";
that I get a new string called greeting with the concatenated value. I understand that the run-time(or compiler, or whatever) is actually getting rid of the reference to the original string greeting and replacing it with a new concatenated one of the same name.
I was just wondering what practical application/ramification this has. Why does it matter to me how C# shuffles strings around in the background when the effect to me is simply that my initial variable changed value.
I was wondering if someone could give me a scenario where a programmer would need to know the difference. * a simple example would be nice, as I'm a relative beginner to this.
Thanks in advance..

Strings, again, are a good example. A very common error is:
string greeting = "Hello Foo!";
greeting.Replace("Foo", "World");
Instead of the proper:
string greeting = "Hello Foo!";
greeting = greeting.Replace("Foo", "World");
Unless you knew that string was an immutable class, you could suspect the first method would be appropriate.
Why does it matter to me how C# shuffles strings around in the background when the effect to me is simply that my initial variable changed value.
The other major place where this has huge advantages is when concurrency is introduced. Immutable types are much easier to deal with in a concurrent situation, as you don't have to worry about whether another thread is modifying the same value within the same reference. Using an immutable type often allows you to avoid the potentially significant cost of synchronization (ie: locking).

I understand that the run-time(or compiler, or whatever) is actually getting rid of the reference to the original string greeting and replacing it with a new concatenated one of the same name.
Pedantic intro: No. Objects do not have names -- variables do. It is storing a new object in the same variable. Thus, the name (variable) used to access the object is the same, even though it (the variable) now refers to another object. An object may also be stored in multiple variables and have multiple "names" at the same time or it might not be accessible directly by any variable.
The other parts of the question have already been succinctly answered for the case of strings -- however, the mutable/immutable ramifications are much larger. Here are some questions which may widen the scope of the issue in context.
What happens if you set a property of an object passed into a method? (There are these pesky "value-types" in C#, so it depends...)
What happens if a sequence of actions leaves an object in an inconsistent state? (E.g. property A was set and an error occurred before property B was set?)
What happens if multiple parts of code expect to be modifying the same object, but are not because the object was cloned/duplicated somewhere?
What happens if multiple parts of code do not expect the object to be modified elsewhere, but it is? (This applies in both threading and non-threading situations)
In general, the contract of an object (API and usage patterns/scope/limitations) must be known and correctly adhered to in order to ensure program validity. I generally find that immutable objects make life easier (as then only one of the above "issues" -- a meager 25% -- even applies).
Happy coding.

C# isn't doing any "shuffling", you are! Your statement assigns a new value to the variable, the referenced object itself did not change, you just dropped the reference.
The major reason immutability is useful is this:
String greeting = "Hello";
// who knows what foo does
foo(greeting);
// always prints "Hello" since String is immutable
System.Console.WriteLine(greeting);
You can share references to immutable objects without worrying about other code changing the object--it can't happen. Therefore immutable objects are easier to reason about.

Most of the time, very little effect. However, in the situation of concatenating many strings, the performance hit of garbage collecting all those strings becomes problematic. Do too many string manipulations with just a string, and the performance of your application can take a nosedive.
This is the reason why StringBuilder is more effective when you have a lot of string manipulation to do; leaving all those 'orphaned' strings out there makes a bigger problem for the Garbage Collector than simply modifying an in memory buffer.

I think the main benefit of immutable strings lies in make memory management easier.
C# allocates memory byte by byte for each object. If you create a string "Tom" it takes up three bytes. You may then allocate an integer and that would be four bytes. If you then tried to change the string "Tom" to "Tomas" it would require moving all the other memory to make room for the two new characters a and s.
To eliminate this pain, it's easier (and quicker) to just allocate five new bytes for the string "Tomas".
Does that help?

In performance terms, the advantage of immutuable is copying an object is cheap in terms of both CPU and memory since it only involves making a copy of a pointer. The downside is that writing to the object becomes more expensive since it must make a copy of the object in the process.

Override ToString or provide non-ToString named extension method for an interface?

My question is about naming, design, and implementation choices. I can see myself going in two different directions with how to solve an issue and I'm interested to see where others who may have come across similar concerns would handle the issue. It's part aesthetics, part function.
A little background on the code... I created a type called ISlice<T> that provides a reference into a section of a source of items which can be a collection (e.g. array, list) or a string. The core support comes from a few implementation classes that support fast indexing using the Begin and End markers for the slice to get the item from the original source. The purpose is to provide slicing capabilities similar to what the Go language provides while using Python style indexing (i.e. both positive and negative indexes are supported).
To make creating slices (instances of ISlice<T>) easier and more "fluent", I created a set of extension methods. For example:
static public ISlice<T> Slice<T>(this IList<T> source, int begin, int end)
{
return new ListSlice<T>(source, begin, end);
}
static public ISlice<char> Slice(this string source, int begin, int end)
{
return new StringSlice(source, begin, end);
}
There are others, such as providing optional begin/end parameters, but the above will suffice for where I'm going with this.
These routines work well and make it easy to slice up a collection or a string. What I also need is way to take a slice and create a copy of it as an array, a list, or a string. That's where things get "interesting". Originally, I thought I'd need to create ToArray, ToList extension methods, but then remembered that the LINQ variants perform optimizations if your collection implements ICollection<T>. In my case, ISlice<T>, does inherits from it, though much to my chagrin as I dislike throwing NotSupportedExceptions from methods like Add. Regardless, I get those for free. Great.
What about converting back into a string as there's no built-in support for converting an IEnumerable<char> easily back into a string? Closest thing I found is one of the string.Concat overloads, but it would not handle chars as efficiently as it could. Just as important from a design stand point is that it doesn't jump out as a "conversion" routine.
The first thought was to create a ToString extension method, but that doesn't work as ToString is an instance method which means it trumps extension methods and would never be called. I could override ToString, but the behavior would be inconsistent as ListSlice<T> would need to special case its ToString for times where T is a char. I don't like that as the ToString will give something useful when the type parameter is a char, but the class name in other cases. Also, if there are other slice types created in the future I'd have to create a common base class to ensure the same behavior or each class would have to implement this same check. An extension method on the interface would handle that much more elegantly.
The extension method leads me to a naming convention issue. The obvious is to use ToString, but as stated earlier it's not allowed. I could name it something different, but what? ToNewString? NewString? CreateString? Something in the To-family of methods would let it fall in with the ToArray/ToList routines, but ToNewString sticks out as being 'odd' when seen in the intellisense and code editor. NewString/CreateString are not as discoverable as you'd have to know to look for them. It doesn't fit the "conversion method" pattern that the To-family methods provide.
Go with overriding ToString and accept the inconsistent behavior hardcoded into the ListSlice<T> implementation and other implementations? Go with the more flexible, but potentially more poorly named extension method route? Is there a third option I haven't considered?
My gut tells me to go with the ToString despite my reservations, though, it also occurred to me... Would you even consider ToString giving you a useful output on a collection/enumerable type? Would that violate the principle of least surprise?
Update
Most implementations of slicing operations provide a copy, albeit a subset, of the data from whatever source was used for the slice. This is perfectly acceptable in most use cases and leaves for a clean API as you can simply return the same data type back. If you slice a list, you return a list containing only the items in the range specified in the slice. If you slice a string, you return a string. And so on.
The slicing operations I'm describing above are solving an issue when working with constraints which make this behavior undesirable. For example, if you work with large data sets, the slice operations would lead to unnecessary additional memory allocations not to mention the performance impact of copying the data. This is especially true if the slices will have further processing done on them before getting to your final results. So, the goal of the slice implementation is to have references into larger data sets to avoid making unnecessary copies of the information until it becomes beneficial to do so.
The catch is that at the end of the processing the desire to turn the slice-based processed data back into a more API and .NET friendly type like lists, arrays, and strings. It makes the data easier to pass into other APIs. It also allows you to discard the slices, thus, also the large data set the slices referenced.

Would you even consider ToString giving you a useful output on a collection/enumerable type? Would that violate the principle of least surprise?
No, and yes. That would be completely unexpected behavior, since it would behave differently than every other collection type.
As for this:
What about converting back into a string as there's no built-in support for converting an IEnumerable>char< easily back into a string?
Personally, I would just use the string constructor taking an array:
string result = new string(mySlice.ToArray());
This is explicit, understood, and expected - I expect to create a new string by passing an object to a constructor.

Perhaps the reason for your conundrum is the fact that you are treating string as a ICollection<char>. You haven't provide details about the problem that you are trying to solve but maybe that's a wrong assumption.
It's true that a string is an IEnumerable<char>. But as you've noticed assuming a direct mapping to a collection of chars creates problems. Strings are just too "special" in the framework.
Looking at it from the other end, would it be obvious that the difference between an ISlice<char> and ISlice<byte> is that you can concatenate the former into a string? Would there be a concatenate operation on the latter that makes sense? What about ISlice<string>? Shouldn't I be able to concatenate those as well?
Sorry I'm not providing specific answers but maybe these questions will point you at the right solution for your problem.