We Keep Coding

Protobuf.net How to use the Any data type?

I have seen example *.proto files where a google.protobuf.Any data type is used. E.g. from the Eclipse Tahu Sparkplug Version B specification where there is something like:
message Row {
repeated DataSetValue elements = 1;
repeated google.protobuf.Any details = 2;
}
When I use the converter from https://protogen.marcgravell.com/, I get an example message in C# where (of course) the type Google.Protobuf.WellKnownTypes.Any is used. The question is now: Is there an equivalent for protobuf.net or if not a workaround I can use?

Excellent question. I've been meaning to address this area for ages. The answer is a simple "no, I haven't even looked at it". I'd be happy to do so, but reality is that I have limited time and need to prioritise effort on features people are asking for. "Any" is something I've heard basically no noise about, until today. If you have a genuine use case for it, we can probably look. GitHub or email may be a better place to explore it, unless you're happy to handle it simply via byte[] and your own code.

C# & IB API - Passing variable while one method has to be declared as 'void'

Background: I am using the C# API of Interactive Brokers and Matlab to load financial data and to conduct an analisys on them. I am quiet used to coding in Matlab, but am at a beginner level with C#. I have created a dll file with all the respective functions to get market data and have implemented the libary into Matlab so that I can initiate the functions in Matlab.
Current Issue: Using Matlab I am calling a method ("top level method") that is requesting the real time market data via the Interactive Broker API. Besides others the following method is called during the process, which simply writes the tickSize into the Console:
public virtual void tickSize(int tickerId, int field, int size)
{
Console.WriteLine("Tick Size. Ticker Id:" + tickerId + ", Field: " + field + ", Size: " + size + "\n");
}
As far as my research has tought me, this method is declared as 'void', because no direct output variable is given back as the requested information is simply written in the console.
The problem is, that this method gets called via another more complex method within the IB API, which is also declared as 'void'.
Goal: I want to get the information of this method (tickSize) as an output when calling the top level method with Matlab.
Constraint: The method within the IB API calling 'tickSize' is also declared as a void method and is a complex C# method with many constraints and if clauses, which I would by all possibilities not like to modify on my own.
Attempts to solve the issue: I guess that the text book way to address this issue, would be to write a method defining the respective output type (for this example lets say 'int' if I only want to get the tickerId, then adjust the complex method of the IB API to also the same output type and last but not least to adjust the method posted above so that the variable can be passed on through the process. However, as outlined above I would not like to mess with the full trading API and was advised by the tech support of IB to write myself another 4th method which then could somehow save the output (int tickerId) and pass it on to the main method directly. Here is where my knowledge of C# certainly ended and also a long Google search did not seem to deliver any results for this specific issue.
Question: When having multiple levels of methods in C#, is there a way to overcome one method being declared as 'void' and send an output directly to the top level method?
If it would help, I could also provide the respective methods I am talking about, but as I said the API one is quiet complex and I did not want to spam all the code in here

'public virtual void tickSize...' is a method called by an event within the API.
So, you don't call it, and nothing is 'returned'. It is called automatically by the API and is the way in which the API provides you with data.
If you wish capture this data, the best method is probably along the lines of:
Create a tickSize class - attibutes: tickerId, field, size.
Create a class level List tickSizes list.
Inside the tickSize method you mention, instantiate a tickSize object, populate it with tickerId, field and size.
Add this object to the tickSizes list.
You will then have a list of tickSizes which you can use for your analysis.

C# Custom Encoding and Decoding/Serialization of Values into a Stream

I am stuck with the following problem: I am communicating with another application using TcpClients/Streams.
This application needs input parameters passed to it, encoded in a special way (something along first byte: parameter type identifier, next 4 bytes: content length; next length bytes: content).
The application processes the input and gives me the output back in the same format. My problem is that I usually don't know the exact order of different return values. The application could return nothing, one int, 5 ints, 2ints and an IntArray...
EDIT: To clarify this, I can not change the underlying protocol or the source code of the application I am communicating with!
The problem is that I want to be able to use the params as if they were "native" C# objects.
What I tried achieving is the following approach:
https://dotnetfiddle.net/ZjbUix
I am not really sure whether this is a good idea. Maybe it would be a better alternative to have a EncodingStream(NetworkStream) with Write() methods for all possible parameter types, and a DecodingStream(NetworkStream) with GetInt(), GetIntArray() etc. methods. This would, however, more or less disregard the type identifier and could only be prevented with throwing an Exception along "Attempted to read an Int value but found Int Array parameter type".
EDIT: I implemented this version here: https://dotnetfiddle.net/8WVXPz
Another requirement is that it should be possible to easily add new parameter types while at best detecting errors at compile-time. Additionally, I'd prefer a non-reflection solution.
Maybe someone can make up an even better architecture to support this?

Use ProtoBuff.Net to receive multiple types of Objects over Socket

I'm implementing a custom protocol that will be used by two apps to send various commands to each other (including file transfers i.e. large binary data chunks). After much deliberation, I think protobuff is the right serializer to use for this task. Serialization with protobuff-net seems pretty straight forward. The issue I'm having is trying to figure out how to do deserialize various objects on the receiving end, especially, when the data is coming off of the network/socket. And many times I probably won't even know whether I have the entire packet to deserialize yet and continuously calling TryDeserializeWithLengthPrefix doesn't seem like the right approach. My concerns are:
How do I implement a scheme where I can figure out not only the message type but also the packet size. I can't just use sizeof(Message) because in case of binary chunks, the actual binary data could vary. Should I add a MessageLength field as the first member of the BaseMessage that all messages inherit from. Then first call TryDeserializeWithLengthPrefix by specifying BaseMessage type, this will give me the actual length of the serialized Message, then I make sure the required number of bytes have been read. If they are, then I call TryDeserializeWithLengthPrefix with the actual MessageType? Any other way of doing this?
I saw some examples of TryDeserializeWithLengthPrefix and it requires some kind of a lookup table that needs to be passed in. In my case the types of messages will easily cross 50 and grow as the app matures. Implementing a lookup table and keeping track of all the message types seems like a bad idea. Is there an alternative option that will allow the code to automatically deserialize various objects types coming off the socket?

1) You could implement your objects so they define their type itself. Just create classes for each object. You can use inheritance if needed.
[ProtoContract]
public class TransportObject1
{
...
[ProtoMember(1)]
public byte[] data ...
}
If you want to inherit classes, simply make a normal C# inheritance and add something like [ProtoInclude(9, typeof(ChildClass))] to the base class.
Regarding your binary data, you could divide it in chunks by yourself (e.g. when reading from source stream etc) and create a transport object (class as shown above) with a byte[] member for the data. On the receiver side you could then receive these objects as any other object, process it and write it's data from the property to the Output stream.
Note: you don't need to worry about the packets you get to build up the "one big packet" that will be deserialized to your transport object (class). TCP handles this for you.
But you will need to care about ordering and processing data from the property (in which order you write it to the output stream on the receiver side). Because when you send multiple packets (chunks) at "the same time", the whole objects could be received in a different order. So you definitely want some kind of chunk ID in the objects.
2)
At some point when receiving data, you will need to specify which type of data you are receiving (which class represents the object).
Because from a socket you simply receive bytes and protocol buffers do not know anything about the type.
You could implement some kind of messages which have a fixed-size header (where the type is saved) and then read this header and determine the type of the object.
With the solution mentioned above, you then don't need any additional lookup tables when deserializing. Protobuf-net handles that for you.
Just use Protobuf-net's Deserialize(..) method with the type parameter you are expecting (by the type set in your message header).
You need that because protocol buffers does not save any information about the types in a object. The type is purely defined by the type you are expecting to be deserialized.
If you want some kind of auto-type mapping, you will need to use another protocol that serializes and deserializes information about the object type.

The way I've used protobuf is with the TcpClient. After you establish a connection and have a TcpClient you can use the GetStream() method to get the handle on a blocking stream. The simple way to handle this is basically:
while (true)
{
var datapacket = Serializer.DeserializeWithLengthPrefix<T>(tcpStream, PrefixStyle.Fixed32);
if (datapacket == null) break;//stream is done
//do somthing with the object here
}
The read will block until the protobuf gets enough data to deserialize an object

Seeking guidance reading .yaml files with C#

Two months later:
The YAML (Eve Online blueprint.yaml) file I tried to parse changed a huge deal which also made it much easier to parse using de deserializer. If someone (for whatever reason) would like to see the code, it's updated on https://github.com/hkraal/ParseYaml
Based on the comment of Steve Wellens I've adjusted the code to do less things at once. It didn't matter in the error itself. I've created another project (Example1) in my solution to test the actual example found on aaubry.net I referenced to earlier.
It gave me the same error when using an "dynamic" key which lead to my current conclusion:
There is a difference between:
items:
- part_no: A4786
and
items:
part_no: A4786
The first is being used in the example which I (wrongly) assumed I could apply to my .yaml file which is using the second syntax.
Now it remains to find out how I can get the 'child' elements of my key with the syntax used in my yaml file...
As C# is used at work I started thinking about a nice project to learn about various aspects of the language while having a direct goal to work towards. However I'm hitting my first wall quite early in my project parsing a Yaml file. My goal is to create an List of YamlBlueprint objects as defined in YamlBlueprint.cs but I don't even get to the end of the Yaml file.
I've setup a testcase on github which demonstrates the problem:
https://github.com/hkraal/ParseYaml
The example on http://www.aaubry.net/page/YamlDotNet-Documentation-Loading-a-YAML-stream works up untill I want to loop trough the items. Based on what I see I should be able to give myKey as parameter to the YamlScalarNode() to access the items below it.
var items = (YamlSequenceNode)mapping.Children[new YamlScalarNode(myKey)];
I'm gettting the following error if I do:
An unhandled exception of type 'System.InvalidCastException' occurred in yamldotnet.exe
Additional information: Unable to cast object of type 'YamlDotNet.RepresentationModel.YamlMappingNode' to type 'YamlDotNet.RepresentationModel.YamlSequenceNode'.
When passing "items" as parameter to YamlScalarNode() it just complains about the item not being there which is to be expected. As I'm not sure where my toughttrain is going wrong I would love a bit assistance on how to troubleshoot this further.

Your question has already been correctly answered, but I would like to point out that your approach is probably not the best one for parsing files. The YamlDotNet.RepresentationModel.* types offer an object model that directly represents the YAML stream and its various parts. This is useful if you are creating an application that processes or generates YAML streams.
When you want to read a YAML document into an object graph, the best approach is to use the Deserializer class. With it you can write your code as follows:
using(var reader = File.OpenText("blueprints.yaml")
{
var deserializer = new Deserializer();
var blueprintsById = deserializer.Deserialize<Dictionary<int, YamlBlueprint>>(reader);
// Use the blueprintsById variable
}
The only difference is that the Id property of the YamlBlueprint instances won't be set, but that's just a matter of adding this:
foreach(var entry in blueprintsById)
{
entry.Value.Id = entry.Key;
}

You have too much stuff going on in one line of code. Create a new YamlScalarNode object in one line, access the array in another line, cast the resultant object in another line. That way, you'll narrow down the problem area to a single step.
The message is telling you that you are retrieving a YamlMappingNode from the array but you are casting it to a YamlSequenceNode. Which is not allowed since the two types are obviously not related.

Well that was kinda stupid... it's kind of hard to create an mapping of something which only contains one element. I've edited the repo linked in the OP with an working example in case somebody runs into the same problem.