I have simple c# client app which sends object to java based server app.
Note:I am using protobuf-net on clinet side. For this application I have simple .proto file with only one field and .java class is generated by protoc compiler.
.Proto file:
message Person {
required string id = 1;
}
C# client to send object
MemoryStream ms = new MemoryStream();
Person per = new Person();
per.id = "TestId001";
Serializer.Serialize<Person>(ms, per);
byte[] buffer = ms.ToArray();
clientSocket.SendTo(buffer, hostEP);
Java based server to receive object
DataInputStream inputStream=new DataInputStream(socket.getInputStream());
Person person = Person.parseFrom(socket.getInputStream());
System.out.println("Id: " + person.getId());
Problem: I not getting the serialized message as sent by the c# app.Even I am not getting
any errors. That's why I was unable to figureout the problem.
"Problem: I not getting the serialized message"
This sounds simply like the classic "sockets are streams" issue (second example here: http://marcgravell.blogspot.com/2013/02/how-many-ways-can-you-mess-up-io.html). If you have written data to a stream but haven't closed that stream, the receiving stream does not terminate. It also does not automatically know that the client sent (say) 117 bytes that should be considered a single message. You have two options:
close the outbound stream after writing (only suitable for sending single messages, not for continued discussions between two nodes)
introduce some form of framing - for example a length-prefix - so that the receiver knows to only try to read a certain amount of data, rather than trying to read to the EOF (which will never come if you haven't closed the outbound stream)
Note: in addition to the "sockets are streams" issue, also keep in mind that protobuf messages are appendable. A protobuf message, by itself, has no notion of where it ends. Fortunately, both the existing bullet points above will address this. But importantly, you can't just write 3 Person object to the stream in a frame (or close the stream), and then expect to be able to get back 3 Person objects at the other end: you will get one Person object. In this scenario, the simplest option is to add a wrapper object, i.e.
message SomeMessage {
repeated Person people = 1;
}
Try adding these attributes to your class in C# and check if the input is ok in Java.
[ProtoContract]
class Person
{
[ProtoMember(1)]
public string id = "1";
}
Related
Consider putting data onto a windows clipboard DataPackage using SetData and later retrieving it using GetDataAsync, like this:
IEnumerable<T> objects = ...;
var randomAccessStream = new InMemoryRandomAccessStream();
using (XmlDictionaryWriter xmlWriter = XmlDictionaryWriter.CreateTextWriter(randomAccessStream.AsStreamForWrite(), Encoding.Unicode)) {
var serializer = new DataContractSerializer(typeof(T), knownTypes);
foreach (T obj in objects) {
serializer.WriteObject(xmlWriter, obj);
}
}
dataPackage.SetData(formatId, randomAccessStream);
Then later on (e.g. in Clipboard.ContentsChanged),
randomAccessStream = await dataPackageView.GetDataAsync(formatId) as IRandomAccessStream;
xmlReader = XmlDictionaryReader.CreateTextReader(randomAccessStream.AsStreamForRead(), Encoding.Unicode, XmlDictionaryReaderQuotas.Max, (OnXmlDictionaryReaderClose?)null);
var serializer = new DataContractSerializer(typeof(T), knownTypes);
while (serializer.IsStartObject(xmlReader)) {
object? obj = serializer.ReadObject(xmlReader);
...
}
xmlReader.Dispose(); // in the real code, this is in a finally clause
The question I have is, when do I dispose the randomAccessStream? I've done some searching and all the examples I've seen using SetData and GetDataAsync do absolutely nothing about disposing the object that is put into or obtain from the data package.
Should I dispose it after the SetData, after the GetDataAsync, in DataPackage.OperationCompleted, in some combination of these, or none of them?
sjb
P.S. If I can squeeze in a second question here ... when I put a reference into a DataPackage using for example dataPackage.Properties.Add( "IEnumerable<T>", entities), does it create a security risk -- can other apps access the reference
and use it?
tldr
The Clipboard is designed to pass content between applications and can only pass string content or a references to files, all other content must be either serialized to string, or saved to a file, or must behave like a file, to be access across application domains via the clipboard.
There is support and guidance for passing custom data and formats via the clipboard, ultimately this involves discrete management around what is "how to prepare the content on the provider side" and "how to interpret the content on the consumer side". If you can use simple serialization for this, then KISS.
IEnumerable<Test> objectsIn = new Test[] { new Test { Name = "One" }, new Test { Name = "two" } };
var dataPackage = new DataPackage();
dataPackage.SetData("MyCustomFormat", Newtonsoft.Json.JsonConvert.SerializeObject(objectsIn));
Clipboard.SetContent(dataPackage);
...
var dataPackageView = Clipboard.GetContent();
string contentJson = (await dataPackageView.GetDataAsync("MyCustomFormat")) as string;
IEnumerable<Test> objectsOut = Newtonsoft.Json.JsonConvert.DeserializeObject<IEnumerable<Test>>(contentJson);
In WinRT the DataPackageView class implementation does support passing streams however the normal rules apply for the stream in terms of lifecycle and if the stream is disposed or not. This is useful for transferring large content or when the consumer might request the content in different formats.
If you do not have an advanced need for it, or you are not transmitting file or image based resources, then you do not need to use a stream to transfer your data.
DataPackageView - Remarks
During a share operation, the source app puts the data being shared in a DataPackage object and sends that object to the target app for processing. The DataPackage class includes a number of methods to support the following default formats: text, Rtf, Html, Bitmap, and StorageItems. It also has methods to support custom data formats. To use these formats, both the source app and target app must already be aware that the custom format exists.
OPs attempt to save a stream to the Clipboard is in this case an example of saving an arbitrary or custom object to the clipboard, it is neither a string or a pointer to a file, so the OS level does not have a native way to handle this information.
Historically, putting string data, or a file reference onto the clipboard is effectively broadcasting this information to ALL applications on the same running OS, however Windows 10 extends this by making your clipboard content able to be synchronised across devices as well. The DataTransfer namespace implementation allows you to affect the scope of this availability, but ultimately this feature is designed to allow you to push data outside of your current application sandboxed domain.
So whether you choose serialize the content yourself, or you want the DataTransfer implementation to try and do it for you, the content will be serialized if it is not already a string or file reference format, and that serialized content, if it succeeds, is what will be made available to consumers.
In this way there is no memory leak or security issue where you might inadvertently provide external processes access to your current process memory or execution context, but data security is still a concern, so don't use the clipboard to pass sensitive content.
A simpler example for Arbitrary or Custom data
OPs example is to put an IEnumerable<T> collection of objects onto the clipboard, and to retrieve them later. OP is choosing to use XML serialization via the DataContractSerializer however a reference to the stream used by the serializer was saved to the clipboard, and not the actual content.
There is a lot of plumbing and first principals logic going on that for little benefit, streams are useful if you are going to stream the content, so if you are going to allow the consumer to control the stream but if you were going to write to the stream in a single synchronous process, then it is better to close off the stream altogether and pass around the buffer that you filled via your stream, we don't even try to re-use the same stream at a later point in time.
The following solution works for Clipboard access in WinRT to pre-serialize a collection of objects and pass them to a consumer:
IEnumerable<Test> objectsIn = new Test[] { new Test { Name = "One" }, new Test { Name = "two" } };
var dataPackage = new DataPackage();
string formatId = "MyCustomFormat";
var serial = Newtonsoft.Json.JsonConvert.SerializeObject(objectsIn);
dataPackage.SetData(formatId, serial);
Clipboard.SetContent(dataPackage);
Then in perhaps an entirely different application:
string formatId = "MyCustomFormat";
var dataPackageView = Clipboard.GetContent();
object content = await dataPackageView.GetDataAsync(formatId);
string contentString = content as string;
var objectsOut = Newtonsoft.Json.JsonConvert.DeserializeObject<IEnumerable<Test>>(contentString);
foreach (var o in objectsOut)
{
Console.WriteLine(o);
}
The definition of Test, in both the provider and the consumer application contexts:
public class Test
{
public string Name { get; set; }
}
when do I dispose the randomAccessStream?
Only Dispose the stream when you have finished using it, when you have Diposed the stream it will be no longer usable in any other contexts, even if you have stored or passed multiple references to it in other object instances.
If you are talking about the original stream referenced in the SetData() logic then look at this from the other angle, If you dispose too early, the consuming code will no longer have access to the stream and will fail.
As a general rule we should try to design the logic such that at any given point in time there is a clear and single Owner for any given stream, in that way it should be clear who has responsibility for Disposing the stream. This response to a slightly different scenario explains it well, https://stackoverflow.com/a/8791525/1690217 however as a general pattern only the scope that created the stream should be responsible for Disposing it.
The one exception to that is that if you need to access the stream outside of the creating method, then the parent class should hold a reference to it, in that scenario you should make the parent class implement IDisposable and make sure it cleans up any resources that might be hanging around.
The reason that you don't see this in documentation is often that the nuances around the timing for calling Dispose() are out of scope or will get lost in examples that are contrived for other purposes.
Specifically for examples where streams are passed via any mechanism and later used, as with DataPackage, it is too hard to show all of the orchestration code to cover the time in between storing the stream with DataPackage.SetData(...) and later accessing the stream via DataPackage.GetDataAsync(...)
Also consider the most common scenario for DataPackage where the consumer is not only in a different logical scope, but most likely in an entirely different application domain, to include all the code to cover when or if to call dispose should encompass the entire code base for 2 different applications.
Is there any way to pass the Length of uncertain Stream to WCF Service?
Unsertain Stream means the stream of
The stream provides its length only after process and writing the data.
e.g. GZipStream
Background
I'm making a WCF Service receiving multiple Streams from client.
As WCF Streaming only allows one stream in the message, I decided to concatenate all streams into one stream and divide it in server code.
The streams client provides will contains variable kinds of stream like FileStream, MemoryStreamwith data from DataTable serialization and
using (var fileStream = new FileStream(filePath, FileMode.Open))
using (var memoryStream = new MemoryStream())
using (var concatStream = new ConcatenatedStream(fileStream, memoryStream))
{
client.UploadStreams(concatStream);
}
ConcatenatedStream is a Stream implementation suggested in c# - How do I concatenate two System.Io.Stream instances into one? - Stack Overflow.
In server side, Length of each Streams will be needed to divide single stream to multiple streams.
As I want to save memory in client side, I decided to use PullStream.
PullStream will Write buffer on demand of Read.
But this causes a big problem. I cannot get Length of PullStream before starting streaming.
Any helps will be appreciated.
Thanks
Let's make it simple:
If you have the length of a part of the stream on client before you start pushing it to server you can append a structure before the payload and read that structure on server. That is a standard data transfer template. Doing so i.e. appending a header before each payload you give your server a hint on how long the next part is going to be.
If you do not have the length of a part of the stream on client before you start pushing it to server, you are going to have to 'insert' the header inside the payload. That's not very intuitive and not that useful but it does work. I used such a thing when I had my data prepared asynchronously on client and the first buffers were ready before the length was known. In this scenario you are going to need a so called marker i.e. a set of bytes that could not be found anywhere in the stream but before the header.
This scenario is the toughest of the 3 to implement when done for the first time. Buckle up. In order to do it right you should create an artificial structure of your stream. Such a structure is used for streaming video over network and called Network Abstraction Layer or NAL, read about it. It is also called stream format AnnexB from the h264 standard. You should abstract from the field in which the standard is described, the idea is very versatile.
In short the payload is divided into parts, so called NAL Units or NALUs, each part has a byte sequence which marks it's start, then goes the type indicator and length of the current NALU, then follows the payload of the NALU. For your purposes you would need to implement NALUs of two types:
Main data payload
Metadata
After you imagine how your stream should look like, you have to grip on the idea of "stream encoding". Those are fearsome words but do not worry. You just have to ensure that the byte sequence that is used to mark the start of the NALU is never met inside the payload of the NALU. In order to achieve that you are going to implement some replacement tactic. Browse for samples.
When you are done thinking this through and before you dive into that, think twice about it. Might be the scenario 3 would fit you easier.
In the case you are sure you will never have to process a part of the streamed data you can greatly simplify the scenario i.e. totally skip the stream encoding and implement something like this:
Client Stream principal code:
private byte[] mabytPayload;
private int mintCurrentPayloadPosition;
private int? mintTotalPayloadLength;
private bool mblnTotalPayloadLengthSent;
public int Read(byte[] iBuffer, int iStart, int iLength)
{
if (mintTotalPayloadLength.HasValue && !mblnTotalPayloadLengthSent)
{
//1. Write the packet type (0)
//3. Write the total stream length (4 bytes).
...
mblnTotalPayloadLengthSent = true;
}
else
{
//1. Write the packet type (1)
//2. Write the packet length (iLength - 1 for example, 1 byte is for
//the type specification)
//3. Write the payload packet.
...
}
}
public void TotalStreamLengthSet(int iTotalStreamLength)
{
mintTotalPayloadLength = iTotalStreamLength;
}
Server stream reader:
Public void WCFUploadCallback(Stream iUploadStream)
{
while(!endOfStream)
{
//1. Read the packet type.
if (normalPayload)
{
//2.a Read the payload packet length.
//2.b Read the payload.
}
else
{
//2.c Read the total stream length.
}
}
}
In the scenario where your upload is no-stop and the metadata about the stream is ready on client long after the payload, that happens as well, you are going to need two channels i.e. one channel for payload stream and another channel with metadata where you server will answer to the client with another question like 'what did you just started sending me' or 'what have you sent me' and the client will explain itself in the next message.
If you are ready to stick to one of the scenarios, one could give you some further details and/or recommendations.
I have a .net Web service Method 'CheckCustomerLicense', and this method returns a C sharp serialized object, I am calling this method via Soap from PHP.
I am getting the serialized object which is binary formatted and not in XML.
I want to access that data as object in PHP, but to do this I must desirealize it, and since its a .NET object I want to use .NET built in class 'System.Runtime.Serialization.Formatters.Binary.BinaryFormatter' via COM.
The Code I am Using For This Is Shown Below:
<?php
class eToolsLicenseNew
{
}
$url='http://mail.ucm.com.au/eToolsLicenseWebService/eToolsLicenseWebUpdateService.asmx?WSDL';
$soap = new SoapClient($url, array(
"trace" => 1, // enable trace to view what is happening
"exceptions" => 1, // disable exceptions "cache_wsdl" => 1)
);
try {
$customer=array('customerId'=>'12345');
$result=$soap->CheckCustomerLicense($customer);
//print_r($result);
$obj = new COM("System.Runtime.Serialization.Formatters.Binary.BinaryFormatter");
$object=new eToolsLicenseNew();
$object=$obj->Deserialize($result); // call to deserilize method
}
catch (SoapFault $e)
{
echo "Error: {$e->faultstring}";
}
?>
But When I call 'Deserializing Method' Giving Internal Server Error....
Other than that every thing is fine, Can Any one is there to help me... Please...
The data your receive is a byte array. If it contains a .NET binary formatted object, you will need to have a BinaryFormatter to deserialize it. A binary formatters Deserialize method however, does not take a byte array. It takes a stream of any kind. The simplest conversion method from byte array to a stream is using a MemoryStream. This takes a byte array as constructor parameter.
var bytes = new byte[50]; // example byte array
using(var stream = new MemoryStream(bytes))
{
BinaryFormatter formatter = new BinaryFormatter();
var obj = (YourExpectedType)formatter.Deserialize(stream);
}
This is what the C# code would look like, you will need to adapt it to the COM/PHP variant.
Again, if this is really the case, someone is sending a binary serialized object via XML serialized SOAP to someone, then the first someone needs to learn how to code webservices. Because this is not an interoperable webservice, this is C#-to-C# communication wasting time using SOAP.
I'm trying to send a broker message to a service bus and I want the message to be a list of multiple types. I've tried using interfaces as well as objects and it works fine until I add more than one type to the list. I'm read several posts and online articles about doing something similar and they all seem to be specific to doing manual xml seralization or using WCF. In this case the seralization is happening automatically.
My code is like so:
Queue<Object> x = new Queue<Object>();
x.Enqueue(new VRequest());
x.Enqueue(new PRequest());
ServiceBus.TrackerClient.SendAsync(new BrokeredMessage(x) { ContentType = "BulkRequest" });
Then my broker message handler (where a seralization error occurs):
var bulk = message.GetBody<Queue<Object>>();
Any ideas on how I can send a single broker message with multiple types?
To anyone interested you can use a binary formatter and memory stream to accomplish this. It's super flexible since you are working binary data... You can even use interfaces, etc. You will need to convert to a byte array once you have the memory stream so you can send it over the network. Then you are able to deserialize it on the other end. Also make sure you mark your objects are serializable.
BinaryFormatter formatter = new BinaryFormatter();
MemoryStream stream = new MemoryStream();
Queue<IYodas> q = new Queue<IYodas>();
q.Enqueue(new Yoda());
q.Enqueue(new Yoda2());
formatter.Serialize(stream, q);
Byte[] package = stream.ToArray();
// Send broker message using package as the object to send
....
// Then on the other end (you will need a byte array to object function)
Queue<IYodas> result = (Queue<IYodas>)ByteArrayToObject(package);
In my job, I work with an application developped partly in c++ and C#. The C++ code is responsible to manage activeMQ (send, receive message).
I've developped an application to monitor the messages sent in the topic by subscribing myself with my C# application.
So when a message is sent to a topic, my application manage to handle the message, but the message is serialized in ActiveMQBytesMessage.
How can I deserialize this object ?
public void OnMessage(IMessage message)
{
if (message != null)
{
var content = (message as ActiveMQBytesMessage).Content; // This is a byte[], I tried to deserialize using BinaryFormatter but it throws an exception, I can't write it here because I'm at home.
}
}
I just noticed that ActiveMQBytesMessage inherits IBytesMessage from namespace Apache.NMS, but I see nothing which helps me to deserialize the message.
I use the last version of ActiveMQ with NMS
[NB] The goal of my C# application is to simply monitor what's happening inside an ActiveMQ channel. That's why I need to deserialize the ActiveMQBytesMessage so I can display the name of the object and its content in a gridview.
[Added more information]
Here's what i tried to deserialize.
var memoryStream = new MemoryStream((message as ActiveMQBytesMessage).Content);
var binaryFormatter = new BinaryFormatter();
memoryStream.Position = 0;
var deserializedMessage = binaryFormatter.Deserialize(memoryStream);
And I get this error when it deserializes:
The input stream is not a valid binary format. The starting contents (in bytes) are: 00-00-00-00-00-00-4F-8C-00-00-00-09-00-00-00-00-54 ...
(I am making a few assumptions here, since you didn't specify certain details.) The BinaryFormatter you are attempting to use will only work for .NET objects, not for C++ objects. Most likely, these objects have not been encoded in a platform neutral way, and are in a C++ format specific to that particular compiler and platform. Therefore, it is up to you to parse the binary code directly to determine what object is encoded, and then to manually decode the data. If these are non-trivial objects, this will be a difficult task. If at all possible, try to get the original application to encode the objects into a platform neutral format that can be easily parsed and instantiated in C#. (I prefer using a TextMessage and XML encoding.) It won't be as efficient as the direct C++ to C++ encoding/decoding that is apparently going on right now, but it will allow external monitoring of the message stream. When I do this, I put the full typename (including namespace) of the object in the NMSType header property. This then tells me the internal structure of message content, and I can instantiate the correct object for parsing the data out of the message.
If all of that doesn't help, or the assumption is wrong and you are using Managed C++, perhaps this question/answer will help you: What serialization method is used for an ActiveMQ NMS C# object message?