I'm making a chat system thing with tcp which requires to send things in byte arrays, but when I convert an image into a byte array, send it and then convert back it gives this error: 'End of Stream encountered before parsing was completed.'. With strings it works just fine.
public byte[] ObjectToByteArray(object obj)
{
BinaryFormatter formatter = new BinaryFormatter();
using (var stream = new MemoryStream())
{
formatter.Serialize(stream, obj);
return stream.ToArray();
}
}
public object ByteArrayToObject(byte[] bytes)
{
using (var stream = new MemoryStream())
{
var binForm = new BinaryFormatter();
stream.Write(bytes, 0, bytes.Length);
stream.Position = 0;
var obj = binForm.Deserialize(stream);
return obj;
}
}
There's two separate things here; firstly, and I cannot emphasize this enough; do not use BinaryFormatter. Ever. It will hurt you. Lots of serializers exist, and BinaryFormatter (and the cousin NetDataContractSerializer) is literally the absolute last you should use. I can expand on that if you like, or I can suggest alternatives if you like.
Now; as for the actual problem: I strongly suspect that it isn't what you think it is. I have a hunch, based on decades of working on network code, that the real problem here is "framing". By which I mean: TCP is a stream protocol, not a message/packet protocol. I strongly suspect that you have not correctly deframed the exact bytes that were sent. I can't say this for sure without seeing your socket code, but... as I say: it is an hunch based on lots of experience. To investigate this: note the length of the bytes you send, and note the length of the bytes you've received. I'm pretty sure you'll find they are different. If there's still doubt: get the base-64 or hex string of the sent payload and the received payload (Convert.ToBase64String, for example), and compare that string. I'm pretty sure they'll turn out to be different.
Ultimately, network code is hard; I could try and explain individual points, but "how to correctly send messages over a network" could fill a book. IMO, if you're not interested in specializing in writing network code for the next 5 years: use an existing tool that will do the job for you, for example gRPC. Lots and lots of other messaging RPC tools exist.
Related
I am trying to figure out how to work with protobuf-net. The below example is trivial on purpose. It does not reflect a real life application.
Please, observe:
var ms = new MemoryStream();
Serializer.Serialize(ms, 1234);
Serializer.Serialize(ms, 5678);
ms.Position = 0;
var n1 = Serializer.Deserialize<int>(ms);
var n2 = Serializer.Deserialize<int>(ms);
Debug.WriteLine(n1);
Debug.WriteLine(n2);
It outputs:
5678
0
What is wrong?
I am using protobuf-net 2.4.0
As said in the comments, Protobuf doesn't have a concept of 'packets' the way you'd expect. A stream is always fully read on deserialization, previously read values are overriden when they're seen again in the stream. Therefore, writing multiple messages to a single stream won't work the way you do it. Unfortunately, there is no easy fix, you have to implement packet-splitting yourself. One way to do it is to prefix any message by it's length, and split the input accordingly. This works good for bigger message, but your message consists only of a single int which would scale very badly. I'd really suggest not using protobuf at all in this scenario, instead you could take a look at their int-serialization routine (some dynamic-length encoding if I remember correctly) and serialize and deserialize single ints yourself.
You can use BitConverter for converting integers to byte arrays and then sending the arrays to memorystream:
Following link can help:
https://www.c-sharpcorner.com/uploadfile/mahesh/convert-integer-to-byte-array-in-C-Sharp/
then once you have the array of bytearrays, you can do this:
var memoryStream = new MemoryStream();
for(int temp = 0; temp < bytArray.Length; temp++)
memoryStream.Write(byteArray[temp], 0, byteArray[temp].Length);
Finally, use Protobuff to have memorystream.
I would like to be able to get the length of the data available from a TCP network stream in C# to set the size of the buffer before reading from the network stream. There is a NetworkStream.Length property but it isn't implemented yet, and I don't want to allocate an enormous size for the buffer as it would take up too much space. The only way I though of doing it would be to precede the data transfer with another telling the size, but this seems a little messy. What would be the best way for me to go about doing this.
When accessing Streams, you usually read and write data in small chunks (e.g. a kilobyte or so), or use a method like CopyTo that does that for you.
This is an example using CopyTo to copy the contents of a stream to another stream and return it as a byte[] from a method, using an automatically-sized buffer.
using (MemoryStream ms = new MemoryStream())
{
networkStream.CopyTo(ms);
return ms.ToArray();
}
This is code that reads data in the same way, but more manually, which might be better for you to work with, depending on what you're doing with the data:
byte[] buffer = new byte[2048]; // read in chunks of 2KB
int bytesRead;
while((bytesRead = networkStream.Read(buffer, 0, buffer.Length)) > 0)
{
//do something with data in buffer, up to the size indicated by bytesRead
}
(the basis for these code snippets came from Most efficient way of reading data from a stream)
There is no inherent length of a network stream. You will either have to send the length of the data to follow from the other end or read all of the incoming data into a different stream where you can access the length information.
The thing is, you can't really be sure all the data is read by the socket yet, more data might come in at any time. This is try even if you somehow do know how much data to expect, say if you have a package header that contains the length. the whole packet might not be received yet.
If you're reading arbitrary data (like a file perhaps) you should have a buffer of reasonable size (like 1k-10k or whatever you find to be optimal for your scenario) and then write the data to a file as its read from the stream.
var buffer = byte[1000];
var readBytes = 0;
using(var netstream = GetTheStreamSomhow()){
using(var fileStream = (GetFileStreamSomeHow())){
while(netstream.Socket.Connected) //determine if there is more data, here we read until the socket is closed
{
readBytes = netstream.Read(buffer,0,buffer.Length);
fileStrem.Write(buffer,0,buffer.Length);
}
}
}
Or just use CopyTo like Tim suggested :) Just make sure that all the data has indeed been read, including data that hasn't gotten across the network yet.
You could send the lenght of the incoming data first.
For example:
You have data = byte[16] you want to send. So at first you send the 16 and define on the server, that this length is always 2 (because 16 has two characters). Now you know that the incomingLength = 16. You can wait now for data of the lenght incomingLength.
Which one is better : MemoryStream.WriteTo(Stream destinationStream) or Stream.CopyTo(Stream destinationStream)??
I am talking about the comparison of these two methods without Buffer as I am doing like this :
Stream str = File.Open("SomeFile.file");
MemoryStream mstr = new MemoryStream(File.ReadAllBytes("SomeFile.file"));
using(var Ms = File.Create("NewFile.file", 8 * 1024))
{
str.CopyTo(Ms) or mstr.WriteTo(Ms);// Which one will be better??
}
Update
Here is what I want to Do :
Open File [ Say "X" Type File]
Parse the Contents
From here I get a Bunch of new Streams [ 3 ~ 4 Files ]
Parse One Stream
Extract Thousands of files [ The Stream is an Image File ]
Save the Other Streams To Files
Editing all the Files
Generate a New "X" Type File.
I have written every bit of code which is actually working correctly..
But Now I am optimizing the code to make the most efficient.
It is an historical accident that there are two ways to do the same thing. MemoryStream always had the WriteTo() method, Stream didn't acquire the CopyTo() method until .NET 4.
The MemoryStream.WriteTo() version looks like this:
public virtual void WriteTo(Stream stream)
{
// Exception throwing code elided...
stream.Write(this._buffer, this._origin, this._length - this._origin);
}
The Stream.CopyTo() implementation like this:
private void InternalCopyTo(Stream destination, int bufferSize)
{
int num;
byte[] buffer = new byte[bufferSize];
while ((num = this.Read(buffer, 0, buffer.Length)) != 0)
{
destination.Write(buffer, 0, num);
}
}
Stream.CopyTo() is more universal, it works for any stream. And helps programmers that fumble copying data from, say, a NetworkStream. Forgetting to pay attention to the return value from Read() was a very common bug. But it of course copies the bytes twice and allocates that temporary buffer, MemoryStream doesn't need it since it can write directly from its own buffer. So you'd still prefer WriteTo(). Noticing the difference isn't very likely.
MemoryStream.WriteTo: Writes the entire contents of this memory stream to another stream.
Stream.CopyTo: Reads the bytes from the current stream and writes them to the destination stream. Copying begins at the current position in the current stream.
You'll need to seek back to 0, to get the whole source stream copied.
So I think MemoryStream.WriteTo better option for this situation
If you use Stream.CopyTo, you don't need to read all the bytes into memory to start with. However:
This code would be simpler if you just used File.Copy
If you are going to load all the data into memory, you can just use:
byte[] data = File.ReadAllBytes("input");
File.WriteAllBytes("output", data);
You should have a using statement for the input as well as the output stream
If you really need processing so can't use File.Copy, using Stream.CopyTo will cope with larger files than loading everything into memory. You may not need that, of course, or you may need to load the whole file into memory for other reasons.
If you have got a MemoryStream, I'd probably use MemoryStream.WriteTo rather than Stream.CopyTo, but it probably won't make much difference which you use, except that you need to make sure you're at the start of the stream when using CopyTo.
I think Hans Passant's claim of a bug in MemoryStream.WriteTo() is wrong; it does not "ignore the return value of Write()". Stream.Write() returns void, which implies to me that the entire count bytes are written, which implies that Stream.Write() will block as necessary to complete the operation to, e.g., a NetworkStream, or throw if it ultimately fails.
That is indeed different from the write() system call in ?nix, and its many emulations in libc and so forth, which can return a "short write". I suspect Hans leaped to the conclusion that Stream.Write() followed that, which I would have expected, too, but apparently it does not.
It is conceivable that Stream.Write() could perform a "short write", without returning any indication of that, requiring the caller to check that the Position property of the Stream has actually been advanced by count. That would be a very error-prone API, and I doubt that it does that, but I have not thoroughly tested it. (Testing it would be a bit tricky: I think you would need to hook up a TCP NetworkStream with a reader on the other end that blocked forever, and write enough to fill up the wire buffers. Or something like that...)
The comments for Stream.Write() are not quite unambiguous:
Summary:
When overridden in a derived class, writes a sequence of bytes to the current
stream and advances the current position within this stream by the number
of bytes written.
Parameters: buffer:
An array of bytes. This method copies count bytes from buffer to the current stream.
Compare that to the Linux man page for write(2):
write() writes up to count bytes from the buffer pointed buf to the file referred to by the file descriptor fd.
Note the crucial "up to". That sentence is followed by explanation of some of the conditions under which a "short write" might occur, making it very explicit that it can occur.
This is really a critical issue: we need to know how Stream.Write() behaves, beyond all doubt.
The CopyTo method creates a buffer, populates its with data from the original stream and then calls the Write method passing the created buffer as a parameter. The WriteTo uses the memoryStream's internal buffer to write. That is the difference. What is better - it is up to you to decide which method you prefer.
Creating a MemoryStream from a HttpInputStream in Vb.Net:
Dim filename As String = MyFile.PostedFile.FileName
Dim fileData As Byte() = Nothing
Using binaryReader = New BinaryReader(MyFile.PostedFile.InputStream)
binaryReader.BaseStream.Position = 0
fileData = binaryReader.ReadBytes(MyFile.PostedFile.ContentLength)
End Using
Dim memoryStream As MemoryStream = New MemoryStream(fileData)
SO I am trying to get a simple dataset from the strInstallDataSet string into the dataset, using the code below, when I have the debugger connected I can see that strInstallDataSet has data, byteArray has data, but even after reading msDataset has nothing, length just sits at 0, I have tried setting the position before and after reading but it still just doesn't pick up any data. Any ideas?
MemoryStream msDataset = new MemoryStream();
if (strInstallDataSet != null)
{
// Convert string to byte array.
byte[] byteArray = Encoding.ASCII.GetBytes(strInstallDataSet);
msDataset.Read(byteArray, 0, byteArray.Length);
// Put stream back into dataset object.
dsInstallData.ReadXml(msDataset);
msDataset.Close();
msDataset.Dispose();
}
You probably want to be doing the following:
using(StringReader reader = new StringReader(strInstallDataSet))
{
dsInstallData.ReadXml(reader);
}
You are not writing anything to the stream, only reading msDataset.Read...
Side note 1: you are using very low level methods - there are Reader/Writer classes that will correctly take care of encoding already.
Side note 2: use "using" instead of manually calling Close or Dispose (and don't call 2 of them together as both do exactly the same things).
You're misunderstanding what the MemoryStream.Read() does, it reads into the byte array, not into the memorystream.
You want MemoryStream.Write() where you have MemoryStream.Read()
Or better yet...
MemoryStream xmlMemoryStream = new MemoryStream(byteArray);
You must use Write method instead of read. I think you want to write your bytearray into your memory stream.
From the internet I got the way to read a huge string from a NetworkStream.
static NetworkStream ns = null;
static StringBuilder sb = null;
static byte[] buffer = null;
static int position = 0;
//.......................................
//other codes skipped for simplicity
//.......................................
private static string Read()
{
if (ns.CanRead)
{
sb.Clear();
position = 0;
while (ns.DataAvailable)
{
position = ns.Read(buffer, 0, buffer.Length);
sb.Append(Encoding.Unicode.GetString(buffer, 0, position));
}
return sb.ToString().Trim();
}
else
{
return null;
}
}
However, I cannot find an example how to write a huge string to a NetworkStream.
Is there a "symmetrical" pattern for writing as we do for reading?
Thank you in advance.
That reading code is dangerously wrong in many ways:
By using static variables in this way, it's hopelessly unsuitable for multi-threaded tasks. (I hope that's just due to you simplifying it...)
It never initializes the variables to non-null values - again, hopefully that's not the real code
It uses the DataAvailable property to decide when it should be "done" - that's incredibly dangerous as it means if a packet is delayed in the stream, you could read half as much data as you expected to
It uses Encoding.Unicode always, which is rarely the best choice of encoding
It assumes that it will always read a whole number of characters. What if one character is split between reads? That's what the Encoder/Decoder classes are for... but you don't really need to use them here anyway - see below.
I would strongly suggest that you wrap the NetworkStream in a StreamReader for reading, and a StreamWriter for writing. That's what they're for. You can then read a line at a time, or just a char[] buffer, or to the end of the stream (which means "until the socket is closed"). This is fine for a text-only protocol.
If you've got a protocol which mixes text and binary data, life becomes a lot harder. Personally I like protocols which length-prefix messages - that way you can read only the data you're meant to, and then perform whatever conversion you want.
Anyway, I hope this random selection of thoughts helps... if you want more detailed assistance, please provide details of what protocol you're using.