I am preparing to write a platform independent socket protocol. After some initial research protobuf seems the way to go. I am new to protobuf and I can't seem to figure out one specific issue.
My requirements are:
Completely platform independent client;
C# server;
Async message communication;
Work from a .proto file (so no inference from existing classes);
Must be able to send messages without the server/client knowing up front which message type to expect.
I have already found the (De)SerializeWithLengthPrefix methods, and they are a start. What I can't figure out is how to receive a message if I do not know the type up front.
I have already seen that protobuf-net supports inheritance and can detect the type of the message, so for protobuf-net, inheriting all messages from a common base type would work. However, this is not platform independent and I am looking for a platform independent way of doing this.
So my question is: how do I send my messages so that any client can deserialize them without knowing the type up front?
If the requirement is to work with multiple messages of different types:
Just associate a unique number with each different type of message, and use as a prefix to the message; you can do this trivially with the optional integer parameter to the overloaded SerializeWithLengthPrefix method. The client would have to pre-process this prefix (or it can be handled within protobuf-net by Serializer.NonGeneric, which has a deserialization method that provides a callback for obtaining the type from this prefix number).
If the requirement is to work with completely unknown data:
Given you requirements, I suspect Jon's port would be a better fit; since your emphasis is on platform independence and .proto, and not on inference (where protobuf-net excels) or other .NET specific extensions / utilities.
Note that while a .proto can be compiled (via protoc) to a regular protobuf stream, working with entirely foreign data at the receiver is... irregular. It can probably be done by treating everything as extensions or by working with the coded streams, but...
Edit following discussion in comments:
A simple schema here could be simply:
message BaseMessage {
optional SomeMessage someMessage = 1;
optional SomeOtherMessage someOtherMessage = 2;
optional SomeThirdMessage someThirdMessage = 3;
}
message SomeMessage {...}
message SomeOtherMessage {...}
message SomeThirdMessage {...}
(you could optionally add a discriminator, if that helps)
This is actually essentially how protobuf-net treats inheritance, but is easily represented from other clients, and handles everything like lengths of sub-messages automatically.
Related
I'm playing a bit with SignalR and relating it back to some previous Pub/Sub work. In it, we have a Base Event with a couple of mandatory properties and then several Derived Events for specific payloads.
With SignalR, it appears that I need to define a hub based on each of the derived events as Send is going to deal with a specific type. For example, if I create a hub for the base class I can send any of the derived types or the base type without error but I always get back a base type losing any of the derived type's properties.
Seems my choices are a hub for each type or putting the derived properties in some type of blob to be parsed out by the receiver.
How far off is my thinking?
AFAIK, SignalR is based on a dynamic way to describe and (de)serialize payloads, so its runtime tries to match the type specified on the receiving part, without trying to further match any derived type. It's a mechanism which has the advantage of being able to work without requiring to share types across clients and server, but the disadvantage you are experiencing. This should explain what you see.
You could base your solution on the usage of dynamic, if you want to keep your hierarchy of payloads you'll have to take care yourself of deserialize the received dynamic value into instances of those, maybe with the help of a "record type" member on the base class. You would not need to do a full parsing.
Is there any way to glue metadata to an object in C#?
Context: Framework which is sending messages between peers over the network. Messages can be arbitrary serializable user-defined .NET types.
Of course, when a message is sent by a peer, the framework could wrap the object into a Message class which saves the metadata, and the receiver could unwrap it. However, the processing method of the peer could decide to resend the message to another peer - however, I want to keep the original metadata. The user should not be required to use Message.RealMessage all the time except when resending it.
I thought about keeping the wrapped instance in a dictionary and upon resending looking up if there is already a wrapped instance in the dictionary and resending that one, however, as messages may not be resent at all (or resent multiple times) this would require more and more memory.
Any solutions? Maybe C# directly supports gluing additional information to an object? Normally I would go for an internal interface, however, the user would have to derive all its classes from a framework's base class, which is not possible.
Edit: I kind of want to say "here is an object of WrappedMessage but you are only allowed to use the interface provided by the class T".
There is the ConditionalWeakTable that should do what you want a little better than using directly a Dictionary.
To quote:
Enables compilers to dynamically attach object fields to managed objects.
You can ignore the part about the class being for compiler :-)
My client-server communication looks like this: there are some so called annoucements which are seperate messages used to exchange information. The idea is that annoucement is the common part of every message. Actually I suppose it will be the type of the message. The type decide what is the content. In UML class diagram Annoucement would be the class all other messages inherit.
I want to implement that idea in communication between two applications one written in C++ the other in C#. I thought I could write a message that contain one field with the type if the message (an enum field) . All additional information relevant to the type would be implemented as an extensions.
I have found some examples how to use extensions in C++, however I have no clue how to do it in C#. I know there are interfaces IExtensible and IExtension (in protobuf-net) but how can I use them? Internet resources seem to be poor in the matter.
I suppose in the past messages in C# used to be define similiar to fashion that they are still defined in C++ apps (using proto file and protoc). Can I use the same proto file to define the message in C#? How? Will extenions be interpreted or overriden?
If I could implement extensions, I would sent a message, parse it, check the type and use approriate function to maintain it. That sounds to me cool because I wouldn't have to take care of the type of the message I was going to read - I don't have to know the type before parsing.
There are a number of ways you could do this. I'm not actually sure extensions is the one I would leap for, but:
in your message type, you could have a set of fully defined fields for each sub-message, i.e.
base-message
{1-5} common fields
{optional 20} sub-message 1
{optional 21} sub-message 2
{optional 22} sub-message 3
{optional 23} sub-message 4
sub-message 1
{1-n} specific fields
where you would have exactly one of the sub-message object
alternatively, encapsulate the common parts inside the more specific message:
common field type
{1-n} fields
sub-message 1
{1} common field type
{2-m} specific fields
Either approach would allow you to deserialize; the second is trickier, IMO, since it requires you to know the type ahead of time. The only convenient way to do that is to prefix each with a different identifier. Personally I prefer the first. This does not, however, require extensions - since we know everything ahead of time. As it happens, the first is also how protobuf-net implements inheritance, so you could do that with type inheritance (4 concrete sub-types of an abstract base message type)and [ProtoInclude(...)]
Re extension data; protobuf-net does support that, however as mentioned in the blog this is not included in the current v2 beta. It will be there soon, but I had to put a line somewhere. It is included in the v1 (r282) download though
Note that protobuf-net is just one of several C#/.NET implementations. The wire format is the same, but you might also want to consider the directly ported version. If I had to summarise the difference I would say "protobuf-net is a .NET serializer that happens to be protobuf; protobuf-csharp-port is a protobuf serializer that happens to be .NET" - they both achieve the same end, but protobuf-net focuses on being idiomatic to C#/.NET where-as the port focuses more on having the same API. Either should work here of course.
I'm new to Java, I'm porting over our Windows Phone 7 library to run on Android. Due to syntax similarities this has been very simple so far. Our library is basically an abstracted http message queue that provides data persistence and integrity on mobile platforms. It only provides asynchronous methods which is a design choice. On WP7 I make use of delegates to call the user supplied callback when an async message has been processed and the servers response received.
To achieve the same thing on Android I've found two ways so far - A simple Java listener interface that contains OnSuccess and OnFailure methods that the user must implement, or using the Android handler class which provides a message queue between threads (http://developer.android.com/reference/android/os/Handler.html).
I've gone with the Handler at this stage as if I'm honest it is the most similar to a C# delegate. It also seems like less work for a user of our library to implement. Example of some user code to make use of our library:
connection.GetMessage("http://somerestservice.com", GetCallback);
Handler GetCallback = new Handler() {
public void handleMessage(Message message){
CustomMessageClass customMessage = (CustomMessageClass)message.obj;
if(customMessage.status == Status.Delivered) {
// Process message here,
// it contains various information about the transaction
// as well as a tag that can contain a user object etc.
// It also contains the servers response as a string and as a byte array.
}
}
};
Using this the user can create as many different handlers as they'd like, called whatever they'd like, and pass them in as method parameters. Very similar to a delegate...
The reason I'm wondering if I should move to a listener interface is because the more exposure I gain to Java the more it seems that's just how it's done and it's how third parties using our library would expect it to be done.
It's essentially the same process, except each time you wanted to do something different with the server response i.e. You might be fetching different types of data from different endpoints, you're going to have to create a custom class that implements our interface each time, as well as implementing any methods our interface has. Or of course you could have a single monolithic class that all server responses were funneled in to but have fun trying to figure out what to do with each individual response...
I may be a bit biased due to coming from C# but a listener seems a bit convoluted and I like the handler implementation better, do any Java developers have any thoughts/advice? It would be much appreciated.
Cheers!
The benefit of using the interface approach is loose coupling. This way, any class that implements your interface shouldn't be aware of (or be affected by) any thread management being done elsewhere and can handle the result object as appropriate within its scope.
BTW, I'm a big fan of AsyncTask. Have you tried using?
I don't think what you have there compiles.. you need to define the handler implementation before you use it?
But to the substance of your question, if you really do want a different handler implementation for each response, than the api you have seems fine.
I would use the listener pattern if all messages are handled in the same way, or the different handling only depends on the content in the message which could not be determined when making the getMessage call.
As an aside, typically in Java function and variable names begin with a lower case. Only class names begin with an upper case.
I'm writing some kind of Computing farm with central server giving tasks and nodes that compute them.
I wanted to write it in such way, that nodes don't know what exactly they are computing. They get (from server) an object that implements IComputable iterface, has one method, .compute() that returns IResult type object and send it to the server.
Server is responsible for preparing these object and serving them through .getWork() method on wcf service, and gets the results with .submitResult(IResult result) method.
Problem is, that worker nodes need to know not only the interface, but full object implementation.
I know that Java can serialize method (probably to bytecode) through RMI. Is it possible with c# ?
What you will have to do is put the type which implements the method you are describing into a separate assembly. You can then send the assembly as a byte array to your server, where it will load the assembly, insptect it for types that fit your interface, and then load them. This is the basic pattern for plug-ins using .Net.
Some care has to be taken though. If you are accepting code from arbitrary sources, you will have to lockdown what these loaded assemblies can do (and it is good practice to do even if you trust the source).
A good classic example for how to do this is the Terrarium project. It is a case study that Microsoft produced that involved the viral spreading of arbitrary assemblies in a secure fashion.
You can do
System.Expression.LambdaExpression<Func<result>> lambda = MyFunction;
and then you can serialize expression to string and deserialize on the server