I am trying to send a serialized object through a network tunnel, and while sending it, it adds whitespace (blank) at the end of the byte[], so it fits the selected size. When I go to deserialize it, it throws an error:
SerializationException: End of Stream encountered before parsing was
completed
Code:
[Serializable]
class Foo
{
public int number;
public string str;
}
public class ExampleServer
{
public static void Main(string[] args)
{
TcpListener listener = new TcpListener(9000);
listener.Start();
TcpClient serverClient = listener.AcceptTcpClient();
IFormatter formatter = new BinaryFormatter();
Stream stream = new MemoryStream();
byte[] buffer = new byte[2048];
serverClient.GetStream().Read(buffer, 0, buffer.Length);
stream.Write(buffer, 0, buffer.Length);
Foo fo = (Foo)formatter.Deserialize(stream);
Console.WriteLine(fo.number);
}
}
public class ExampleClient
{
public static void Main(string[] args)
{
TcpClient client = new TcpClient();
client.Connect("127.0.0.1", 9000);
Foo fo = new Foo();
fo.number = 10;
fo.str = "str";
IFormatter formatter = new BinaryFormatter();
Stream stream = new MemoryStream();
formatter.Serialize(stream, fo);
byte[] buffer = new byte[2048];
stream.Read(buffer, 0, buffer.Length);
client.GetStream().Write(buffer, 0, buffer.Length);
}
}
How would I fix this?
In both code samples, the fundamental problem is that you write a buffer to a MemoryStream and then immediately try to read from it, without rewinding. MemoryStream is like a video tape: it only has one position, and if you don't rewind: you're in the wrong place. So fixing that will probably help.
However, that is not the only problem:
when reading from the network stream, you don't check the return of Read. That means that all you have is at least one byte. Calling Read with a buffer does not fill the buffer - it blocks until it can read at least one byte, or the end of the stream is detected; you must check the return value of Read
technically this also applies to MemoryStream, although in reality you will get away with it there
sending / expecting 2k is ... bizarre; send what the payload size is, not something arbitrarily oversized; you probably need to know about "framing", which in a binary format usually means: length-prefix
perhaps ironically, BinaryFormatter does framing itself anyway; if you'd just pointed BinaryFormatter at the network stream, it probably would have worked, but:
BinaryFormatter is a terrible choice, frankly; I could write many paragraphs explaining why - but I already did
As general advice: network code is hard. If you aren't an expert in that area, I'd strongly recommend using a pre-rolled RPC or similar layer.
Try to add [Serializable] tag to the class you are Serializing and check if this solve you problem.
add like this:
[Serializable]
public class TestSimpleObject {
public int member1;
public string member2;
public string member3;
public double member4;
}
Related
I am implementing a client-server communication. I need to send a response to the client whose size may vary, because it contains a serialized array:
[Serializable]
public struct ServerResponse
{
public ApplicationAction ApplicationAction { get; set; }
public Product[] AssociatedProducts { get; set; }
}
What I am currently doing is serializing my instance of ServerResponse to XML using XmlSerializer (it wouldn't bother me at all to use another method my data doesn't need to be human-readable) and writing it on the socket stream like so:
XmlSerializer xmlSerializer = new XmlSerializer(typeof(ServerResponse));
NetworkStream networkStream = client.GetStream();
if (networkStream.CanWrite)
{
xmlSerializer.Serialize(networkStream, response);
}
My problem is that the client does not know the size of the data it needs to read.
My guess is to first send the size of the data and then all the data, but to do that I need to know
how can I get the actual size of a serialized object ?
Thanks
Befor you will send any data you should serialize your Stream to the memory using MemoryStream. Then you can take its lenght and multiply by sizeof(byte)
Your code will look something like this:
XmlSerializer xmlSerializer = new XmlSerializer(typeof(JsonQuestion));
NetworkStream networkStream = client.GetStream();
MemoryStream memoryStream = new MemoryStream();
xmlSerializer.Serialize(memoryStream, response);
var length = memoryStream.Length * sizeof(byte);
//Send size
...
//Send data
if (networkStream.CanWrite)
{
memoryStream.CopyTo(networkStream);
}
I have the following C# code which is supposed to serialize arbitrary objects to a string, and then of course deserialize it.
public static string Pack(Message _message)
{
BinaryFormatter formatter = new BinaryFormatter();
MemoryStream original = new MemoryStream();
MemoryStream outputStream = new MemoryStream();
formatter.Serialize(original, _message);
original.Seek(0, SeekOrigin.Begin);
DeflateStream deflateStream = new DeflateStream(outputStream, CompressionMode.Compress);
original.CopyTo(deflateStream);
byte[] bytearray = outputStream.ToArray();
UTF8Encoding encoder = new UTF8Encoding();
string packed = encoder.GetString(bytearray);
return packed;
}
public static Message Unpack(string _packed_message)
{
UTF8Encoding encoder = new UTF8Encoding();
byte[] bytearray = encoder.GetBytes(_packed_message);
BinaryFormatter formatter = new BinaryFormatter();
MemoryStream input = new MemoryStream(bytearray);
MemoryStream decompressed = new MemoryStream();
DeflateStream deflateStream = new DeflateStream(input, CompressionMode.Decompress);
deflateStream.CopyTo(decompressed); // EXCEPTION
decompressed.Seek(0, SeekOrigin.Begin);
var message = (Message)formatter.Deserialize(decompressed); // EXCEPTION 2
return message;
}
But the problem is that any time the code is ran, I am experiencing an exception. Using the above code and invoking it as shown below, I am receiving InvalidDataException: Unknown block type. Stream might be corrupted. at the marked // EXCEPTION line.
After searching for this issue I have attempted to ditch the deflation. This was only a small change: in Pack, bytearray gets created from original.ToArray() and in Unpack, I Seek() input instead of decompressed and use Deserialize(input) instead of decompressed too. The only result which changed: the exception position and body is different, yet it still happens. I receive a SerializationException: No map for object '201326592'. at // EXCEPTION 2.
I don't seem to see what is the problem. Maybe it is the whole serialization idea... the problem is that somehow managing to pack the Message instances is necessary because these objects hold the information that travel between the server and the client application. (Serialization logic is in a .Shared DLL project which is referenced on both ends, however, right now, I'm only developing the server-side first.) It also has to be told, that I am only using string outputs because right now, the TCP connection between the servers and clients are based on string read-write on the ends. So somehow it has to be brought down to the level of strings.
This is how the Message object looks like:
[Serializable]
public class Message
{
public MessageType type;
public Client from;
public Client to;
public string content;
}
(Client right now is an empty class only having the Serializable attribute, no properties or methods.)
This is how the pack-unpack gets invoked (from Main()...):
Shared.Message msg = Shared.MessageFactory.Build(Shared.MessageType.DEFAULT, new Shared.Client(), new Shared.Client(), "foobar");
string message1 = Shared.MessageFactory.Pack(msg);
Console.WriteLine(message1);
Shared.Message mess2 = Shared.MessageFactory.Unpack(message1); // Step into... here be exceptions
Console.Write(mess2.content);
Here is an image showing what happens in the IDE. The output in the console window is the value of message1.
Some investigation unfortunately also revealed that the problem could lie around the bytearray variable. When running Pack(), after the encoder creates the string, the array contains 152 values, however, after it gets decoded in Unpack(), the array has 160 values instead.
I am appreciating any help as I am really out of ideas and having this problem the progress is crippled. Thank you.
(Update) The final solution:
I would like to thank everyone answering and commenting, as I have reached the solution. Thank you.
Marc Gravell was right, I missed the closing of deflateStream and because of this, the result was either empty or corrupted. I have taken my time and rethought and rewrote the methods and now it works flawlessly. And even the purpose of sending these bytes over the networked stream is working too.
Also, as Eric J. suggested, I have switched to using ASCIIEnconding for the change between string and byte[] when the data is flowing in the Stream.
The fixed code lies below:
public static string Pack(Message _message)
{
using (MemoryStream input = new MemoryStream())
{
BinaryFormatter bformatter = new BinaryFormatter();
bformatter.Serialize(input, _message);
input.Seek(0, SeekOrigin.Begin);
using (MemoryStream output = new MemoryStream())
using (DeflateStream deflateStream = new DeflateStream(output, CompressionMode.Compress))
{
input.CopyTo(deflateStream);
deflateStream.Close();
return Convert.ToBase64String(output.ToArray());
}
}
}
public static Message Unpack(string _packed)
{
using (MemoryStream input = new MemoryStream(Convert.FromBase64String(_packed)))
using (DeflateStream deflateStream = new DeflateStream(input, CompressionMode.Decompress))
using (MemoryStream output = new MemoryStream())
{
deflateStream.CopyTo(output);
deflateStream.Close();
output.Seek(0, SeekOrigin.Begin);
BinaryFormatter bformatter = new BinaryFormatter();
Message message = (Message)bformatter.Deserialize(output);
return message;
}
}
Now everything happens just right, as the screenshot proves below. This was the expected output from the first place. The Server and Client executables communicate with each other and the message travels... and it gets serialized and unserialized properly.
In addition to the existing observations about Encoding vs base-64, note you haven't closed the deflate stream. This is important because compression-streams buffer: if you don't close, it may not write the end. For a short stream, that may mean it writes nothing at all.
using(DeflateStream deflateStream = new DeflateStream(
outputStream, CompressionMode.Compress))
{
original.CopyTo(deflateStream);
}
return Convert.ToBase64String(outputStream.GetBuffer(), 0,
(int)outputStream.Length);
Your problem is most probably in the UTF8 encoding. Your bytes are not really a character string and UTF-8 is a encoding with different byte lengths for characters.
This means the byte array may not correspond to a correctly encoded UTF-8 string (there may be some bytes missing at the end for instance.)
Try using UTF16 or ASCII which are constant length encodings (the resulting string will likely contain control characters so it won't be printable or transmitable through something like HTTP or email.)
But if you want to encode as a string it is customary to use UUEncoding to convert the byte array into a real printable string, then you can use any encoding you want.
When I run the following Main() code against your Pack() and Unpack():
static void Main(string[] args)
{
Message msg = new Message() { content = "The quick brown fox" };
string message1 = Pack(msg);
Console.WriteLine(message1);
Message mess2 = Unpack(message1); // Step into... here be exceptions
Console.Write(mess2.content);
}
I see that the bytearray
byte[] bytearray = outputStream.ToArray();
is empty.
I did modify your serialized class slightly since you did not post code for the included classes
public enum MessageType
{
DEFAULT = 0
}
[Serializable]
public class Message
{
public MessageType type;
public string from;
public string to;
public string content;
}
I suggest the following steps to resolve this:
Check the intermediate results along the way. Do you also see 0 bytes in the array? What is the string value returned by Pack()?
Dispose of your streams once you are done with them. The easiest way to do that is with the using keyword.
Edit
As Eli and Marc correctly pointed out, you cannot store arbitrary bytes in a UTF8 string. The mapping is not bijective (you can't go back and forth without loss/distortion of information). You will need a mapping that is bijective, such as the Convert.ToBase64String() approach Marc suggests.
I have now an issue with deserializing an object sent over TCP.
When deserializing, I get the following SerializationException (code below):
Additional information: Binary stream '0' does not contain a valid
BinaryHeader. Possible causes are invalid stream or object version
change between serialization and deserialization.
Serialization code:
public static void SerializeRO(Stream stream, ReplicableObject ro) {
MemoryStream serializedObjectStream = new MemoryStream();
Formatter.Serialize(serializedObjectStream, ro);
BinaryWriter bw = new BinaryWriter(stream);
bw.Write(serializedObjectStream.Length);
serializedObjectStream.WriteTo(stream);
serializedObjectStream.Close();
bw.Close();
}
Deserialization code:
public static List<ReplicableObject> ParseStreamForObjects(Stream stream) {
List<ReplicableObject> result = new List<ReplicableObject>();
while (true) {
if (!(stream as NetworkStream).DataAvailable) break;
BinaryReader br = new BinaryReader(stream);
int length = br.ReadInt32();
byte[] bytes = br.ReadBytes(length);
MemoryStream ms = new MemoryStream(bytes);
ms.Position = 0;
// ERROR OCCURS ON THE LINE BELOW
result.Add((ReplicableObject) Formatter.Deserialize(ms));
ms.Close();
br.Close();
}
return result;
}
The objects are being serialized at runtime, so I don't think it's a versioning issue. I am new to streaming etc, so I may have missed something obvious.
I'd like to suggest what I think it could be, but I'm really stuck. :)
Thanks.
serializedObjectStream.Length is a long.
You're writing a 64-bit value to the network, but you're trying to read it as a 32-bit int.
I implemented a Client and Server model that uses Socket with thread
When I want to pass only a string from Client to the Server, it works. But I want to pass an object and it throws this error:
"Attempting to deserialize an empty stream"
Here is the code:
Client:
ASCIIEncoding asen = new ASCIIEncoding();
MemoryStream memoryStream = new MemoryStream();
BinaryFormatter binaryFormatter = new BinaryFormatter();
Command command = new Command("meuLogin", "minhaSenha");
binaryFormatter.Serialize(memoryStream, command);
stream.Write(memoryStream.ToArray(), 0, memoryStream.ToArray().Length);
Server:
byte[] message = new byte[4096];
int bytesRead = 0;
bytesRead = clientStream.Read(message, 0, 4096);
MemoryStream memoryStream = new MemoryStream(bytesRead);
BinaryFormatter bf1 = new BinaryFormatter();
memoryStream.Position = 0;
Command command = (Command)bf1.Deserialize(memoryStream);
Another question: I copied the class Command from Client and pasted at Server to deserialize. Is this correct?
Thank you
I also recommend WCF. But if you continue using sockets, the key element that you're missing in your protocol is message framing.
You never use the message that you read from the stream. The memory stream you are reading from is thus empty.
On a side note, why do you use these intermediate MemoryStreams?
To answer your second question: for maximum maintainability, the class Command should be in a separate assembly that both Client and Server reference.
To answer your first question: you are attempting to deserialize from an empty stream on your server, just as the exception tells you. You need to copy the bytes you read from the clientStream into the memoryStream before you deserialize from the memoryStream. Alternatively, use the clientStream directly rather than using the memoryStream; this may require reconsidering your protocol.
Finally, I wholeheartedly agree with #Andrey: consider using WCF. It's way way way better than raw sockets.
If you change your server code to use a different MemoryStream constructor, the problem will go away.
MemoryStream memoryStream = new MemoryStream(message, 0, bytesRead);
However, I agree with Stephen and others. Either use WCF, or use Message framing.
Another question: I copied the class Command from Client and pasted at Server to deserialize. Is this correct?
No. The namespace for your Client and Server are likely different, so the Server will be trying to deserialize a stream that matches its namespace.
You should create your Command Class using its own namespace and reference that from both your Client and the Server.
After that...
Client:
static void StreamToServer(TcpClient client, Command obj) {
using (NetworkStream ns = client.GetStream()) {
using (MemoryStream ms = new MemoryStream()) {
BinaryFormatter formatter = new BinaryFormatter();
formatter.Serialize(ms, obj);
byte[] buf = ms.ToArray();
ns.Write(buf, 0, buf.Length);
}
}
}
Server:
static Command ReadStream(TcpListener listener) {
Command obj = null;
using (TcpClient client = listener.AcceptTcpClient()) { // waits for data
using (NetworkStream ns = client.GetStream()) {
byte[] buf = new byte[client.ReceiveBufferSize];
int len = ns.Read(buf, 0, buf.Length);
using (MemoryStream ms = new MemoryStream(buf, 0, len)) {
BinaryFormatter formatter = new BinaryFormatter();
obj = formatter.Deserialize(ms) as Command;
}
}
}
return obj;
}
WCF or Message framing may be easier, but I don't often have the opportunity at work to sit around and read a book.
Instead of passing bytesRead to the MemoryStream which is actually the length of the byte stream, you should pass 'message', as it is the actual stream of bytes. Like,
MemoryStream memoryStream = new MemoryStream(message);
As you are passing an integer variable, the compiler is throwing exception that the stream is empty.
As for WCF, it is remarkable framework, but for applications that require low latency and high performance, WCF is a horrible answer because of its overheads, it is built upon sockets. So if you use sockets, that will be the lowest level implementation and thus, the fastest. That depends on your application which paradigm you should choose...
What is the best way to copy the contents of one stream to another? Is there a standard utility method for this?
From .NET 4.5 on, there is the Stream.CopyToAsync method
input.CopyToAsync(output);
This will return a Task that can be continued on when completed, like so:
await input.CopyToAsync(output)
// Code from here on will be run in a continuation.
Note that depending on where the call to CopyToAsync is made, the code that follows may or may not continue on the same thread that called it.
The SynchronizationContext that was captured when calling await will determine what thread the continuation will be executed on.
Additionally, this call (and this is an implementation detail subject to change) still sequences reads and writes (it just doesn't waste a threads blocking on I/O completion).
From .NET 4.0 on, there's is the Stream.CopyTo method
input.CopyTo(output);
For .NET 3.5 and before
There isn't anything baked into the framework to assist with this; you have to copy the content manually, like so:
public static void CopyStream(Stream input, Stream output)
{
byte[] buffer = new byte[32768];
int read;
while ((read = input.Read(buffer, 0, buffer.Length)) > 0)
{
output.Write (buffer, 0, read);
}
}
Note 1: This method will allow you to report on progress (x bytes read so far ...)
Note 2: Why use a fixed buffer size and not input.Length? Because that Length may not be available! From the docs:
If a class derived from Stream does not support seeking, calls to Length, SetLength, Position, and Seek throw a NotSupportedException.
MemoryStream has .WriteTo(outstream);
and .NET 4.0 has .CopyTo on normal stream object.
.NET 4.0:
instream.CopyTo(outstream);
I use the following extension methods. They have optimized overloads for when one stream is a MemoryStream.
public static void CopyTo(this Stream src, Stream dest)
{
int size = (src.CanSeek) ? Math.Min((int)(src.Length - src.Position), 0x2000) : 0x2000;
byte[] buffer = new byte[size];
int n;
do
{
n = src.Read(buffer, 0, buffer.Length);
dest.Write(buffer, 0, n);
} while (n != 0);
}
public static void CopyTo(this MemoryStream src, Stream dest)
{
dest.Write(src.GetBuffer(), (int)src.Position, (int)(src.Length - src.Position));
}
public static void CopyTo(this Stream src, MemoryStream dest)
{
if (src.CanSeek)
{
int pos = (int)dest.Position;
int length = (int)(src.Length - src.Position) + pos;
dest.SetLength(length);
while(pos < length)
pos += src.Read(dest.GetBuffer(), pos, length - pos);
}
else
src.CopyTo((Stream)dest);
}
.NET Framework 4 introduce new "CopyTo" method of Stream Class of System.IO namespace. Using this method we can copy one stream to another stream of different stream class.
Here is example for this.
FileStream objFileStream = File.Open(Server.MapPath("TextFile.txt"), FileMode.Open);
Response.Write(string.Format("FileStream Content length: {0}", objFileStream.Length.ToString()));
MemoryStream objMemoryStream = new MemoryStream();
// Copy File Stream to Memory Stream using CopyTo method
objFileStream.CopyTo(objMemoryStream);
Response.Write("<br/><br/>");
Response.Write(string.Format("MemoryStream Content length: {0}", objMemoryStream.Length.ToString()));
Response.Write("<br/><br/>");
There is actually, a less heavy-handed way of doing a stream copy. Take note however, that this implies that you can store the entire file in memory. Don't try and use this if you are working with files that go into the hundreds of megabytes or more, without caution.
public static void CopySmallTextStream(Stream input, Stream output)
{
using (StreamReader reader = new StreamReader(input))
using (StreamWriter writer = new StreamWriter(output))
{
writer.Write(reader.ReadToEnd());
}
}
NOTE: There may also be some issues concerning binary data and character encodings.
The basic questions that differentiate implementations of "CopyStream" are:
size of the reading buffer
size of the writes
Can we use more than one thread (writing while we are reading).
The answers to these questions result in vastly different implementations of CopyStream and are dependent on what kind of streams you have and what you are trying to optimize. The "best" implementation would even need to know what specific hardware the streams were reading and writing to.
Unfortunately, there is no really simple solution. You can try something like that:
Stream s1, s2;
byte[] buffer = new byte[4096];
int bytesRead = 0;
while (bytesRead = s1.Read(buffer, 0, buffer.Length) > 0) s2.Write(buffer, 0, bytesRead);
s1.Close(); s2.Close();
But the problem with that that different implementation of the Stream class might behave differently if there is nothing to read. A stream reading a file from a local harddrive will probably block until the read operaition has read enough data from the disk to fill the buffer and only return less data if it reaches the end of file. On the other hand, a stream reading from the network might return less data even though there are more data left to be received.
Always check the documentation of the specific stream class you are using before using a generic solution.
There may be a way to do this more efficiently, depending on what kind of stream you're working with. If you can convert one or both of your streams to a MemoryStream, you can use the GetBuffer method to work directly with a byte array representing your data. This lets you use methods like Array.CopyTo, which abstract away all the issues raised by fryguybob. You can just trust .NET to know the optimal way to copy the data.
if you want a procdure to copy a stream to other the one that nick posted is fine but it is missing the position reset, it should be
public static void CopyStream(Stream input, Stream output)
{
byte[] buffer = new byte[32768];
long TempPos = input.Position;
while (true)
{
int read = input.Read (buffer, 0, buffer.Length);
if (read <= 0)
return;
output.Write (buffer, 0, read);
}
input.Position = TempPos;// or you make Position = 0 to set it at the start
}
but if it is in runtime not using a procedure you shpuld use memory stream
Stream output = new MemoryStream();
byte[] buffer = new byte[32768]; // or you specify the size you want of your buffer
long TempPos = input.Position;
while (true)
{
int read = input.Read (buffer, 0, buffer.Length);
if (read <= 0)
return;
output.Write (buffer, 0, read);
}
input.Position = TempPos;// or you make Position = 0 to set it at the start
Since none of the answers have covered an asynchronous way of copying from one stream to another, here is a pattern that I've successfully used in a port forwarding application to copy data from one network stream to another. It lacks exception handling to emphasize the pattern.
const int BUFFER_SIZE = 4096;
static byte[] bufferForRead = new byte[BUFFER_SIZE];
static byte[] bufferForWrite = new byte[BUFFER_SIZE];
static Stream sourceStream = new MemoryStream();
static Stream destinationStream = new MemoryStream();
static void Main(string[] args)
{
// Initial read from source stream
sourceStream.BeginRead(bufferForRead, 0, BUFFER_SIZE, BeginReadCallback, null);
}
private static void BeginReadCallback(IAsyncResult asyncRes)
{
// Finish reading from source stream
int bytesRead = sourceStream.EndRead(asyncRes);
// Make a copy of the buffer as we'll start another read immediately
Array.Copy(bufferForRead, 0, bufferForWrite, 0, bytesRead);
// Write copied buffer to destination stream
destinationStream.BeginWrite(bufferForWrite, 0, bytesRead, BeginWriteCallback, null);
// Start the next read (looks like async recursion I guess)
sourceStream.BeginRead(bufferForRead, 0, BUFFER_SIZE, BeginReadCallback, null);
}
private static void BeginWriteCallback(IAsyncResult asyncRes)
{
// Finish writing to destination stream
destinationStream.EndWrite(asyncRes);
}
For .NET 3.5 and before try :
MemoryStream1.WriteTo(MemoryStream2);
Easy and safe - make new stream from original source:
MemoryStream source = new MemoryStream(byteArray);
MemoryStream copy = new MemoryStream(byteArray);
The following code to solve the issue copy the Stream to MemoryStream using CopyTo
Stream stream = new MemoryStream();
//any function require input the stream. In mycase to save the PDF file as stream
document.Save(stream);
MemoryStream newMs = (MemoryStream)stream;
byte[] getByte = newMs.ToArray();
//Note - please dispose the stream in the finally block instead of inside using block as it will throw an error 'Access denied as the stream is closed'