I try to serialize list of 15 objects this way:
XmlSerializer xmlSerializer = new XmlSerializer(typeof(List<Employee>));
MemoryStream memStream = new MemoryStream();
// Serialize
xmlSerializer.Serialize(stream, allKnownWorkers);
memStream.Position = 0;
data = memStream.GetBuffer();
Console.WriteLine("Transmitting.....");
stream.Write(data, 0, data.Length); // NetworkStream
Deserialization looks like:
// Read the first batch of the TcpServer response bytes.
Int32 bytes = stream.Read(data, 0, data.Length);
XmlSerializer xmlSerializer = new XmlSerializer(typeof(List<Employee>));
MemoryStream memStream = new MemoryStream();
memStream.Write(data, 0, bytes);
memStream.Position = 0;
// Deserialize
workers.AddRange((List<Employee>)xmlSerializer.Deserialize(memStream));
I get an exception in last line of deserialization: Unexpected end of file has occurred. The following elements are not closed: ... When I send list with only few objects it works correctly. I suppose there is a problem with stream buffer length. How can I fix it?
Thank you very much!
You are treating your stream-based network connection as if it were message based. But it's not. So you can't count on a single read from the stream returning all of the data for a single object, or even a single transmission.
Instead, you need to design into your protocol a way to know when you've read all the data for a unit of processing (whatever that happens to be in the given context...here that seems to be an XML document).
There are lots of ways to accomplish this. I would say the two most straightforward would be to either transmit a byte count first, before the XML data, so that the receiver knows how many bytes to read before the try to read the XML, or to simply build the XML parsing into the stream reading.
On that latter point, you might try just handing the network stream to your XmlSerializer. I don't recall off the top of my head how well it will handle this, but it could work as long as the XmlSerializer stops reading once it's got a complete XML document, instead of trying to read all the way to the end of the stream. But even if XmlSerializer doesn't just give it to you for free, it should not be too hard to detect the opening tag for the XML document's root element and then just keep reading data until you read the closing tag.
Related
I'm exploring how to implement an HTTP server in C#. (And before you ask, I know there is Kestrel (and nothing else that isn't obsolete), and I want a much, much smaller application.) So, the response could be a Stream that cannot be seeked and has an unknown length. For this situation, chunked encoding can be used instead of sending a Content-Length header.
The response can also be compressed with gzip or br as indicated by the client. This can be accomplished with e.g. the GZipStream class. I had almost said "easily", because that's not really the case. I always find the GZipStream API confusing each time I use it. I usually bump into every exception there is until I finally get it right.
It seems like I can only write (push) to a GZipStream and the compressed data will trickle out the other end into the specified "base" stream. But that's not desirable because I can't just let the compressed data flow to the client. It needs to be chunked. That is, each bit of compressed data needs to be prefixed with its chunk size. Of course the GZipStream cannot produce that format.
Instead, I'd like to read (pull) from the compressing GZipStream, but that doesn't seem to be possible. The documentation says it will throw an exception if I try that. But there has to be some instance that brings the compressed bytes into the chunked format.
So how would I get the expected result? Can it even be achieved with this API? Why can't I pull from the compressing stream, only push?
I'm not trying to make up (non-functional) sample code because that would only be confusing.
PS: Okay, maybe this:
Stream responseBody = ...;
if (canCompress)
{
responseBody = new GZipStream(responseBody, CompressionMode.Compress); // <-- probably wrong
}
// not shown: add appropriate headers
while (true)
{
int chunkLength = responseBody.Read(buffer); // <-- not possible
if (chunkLength == 0)
break;
response.Write($"{chunkLength:X}\r\n");
response.Write(buffer.AsMemory()[..chunkLength]);
response.Write("\r\n");
}
response.Write("0\r\n\r\n");
Your usage of GZipStream is incomplete. While your input responseBuffer is the correct target buffer, you have to actually write the bytes TO the GZipStream itself.
In addition, once you are done writing, you must close the GZipStream instance to write all compressed bytes to your target buffer. This is the critical step because there is no such thing as "partial compression" of an input stream in GZip. You would have to analyze the entire input in order to properly compress it. As such, this is the critical missing link that MUST happen before you can continue to write the response.
Finally, you need to reset the position of your output stream so that you can read it into an intermediary response buffer.
using MemoryStream responseBody = new MemoryStream();
GZipStream gzipStream = null; // make sure to dispose after use
if (canCompress)
{
using MemoryStream gzipStreamBuffer = new MemoryStream(bytes);
gzipStream = new GZipStream(responseBody, CompressionMode.Compress, true);
gzipStreamBuffer.CopyTo(gzipStream);
gzipStream.Close(); // close the stream so that all compressed bytes are written
responseBody.Seek(0, SeekOrigin.Begin); // reset the response so that we can read it to the buffer
}
var buffer = new byte[20];
while (true)
{
int chunkLength = responseBody.Read(buffer);
if (chunkLength == 0)
break;
// write response
}
In my test example, my bytes input was 241 bytes, whereas the compressed bytes written to the buffer totaled 82 bytes.
I want to write a String to a Stream (a MemoryStream in this case) and read the bytes one by one.
stringAsStream = new MemoryStream();
UnicodeEncoding uniEncoding = new UnicodeEncoding();
String message = "Message";
stringAsStream.Write(uniEncoding.GetBytes(message), 0, message.Length);
Console.WriteLine("This:\t\t" + (char)uniEncoding.GetBytes(message)[0]);
Console.WriteLine("Differs from:\t" + (char)stringAsStream.ReadByte());
The (undesired) result I get is:
This: M
Differs from: ?
It looks like it's not being read correctly, as the first char of "Message" is 'M', which works when getting the bytes from the UnicodeEncoding instance but not when reading them back from the stream.
What am I doing wrong?
The bigger picture: I have an algorithm which will work on the bytes of a Stream, I'd like to be as general as possible and work with any Stream. I'd like to convert an ASCII-String into a MemoryStream, or maybe use another method to be able to work on the String as a Stream. The algorithm in question will work on the bytes of the Stream.
After you write to the MemoryStream and before you read it back, you need to Seek back to the beginning of the MemoryStream so you're not reading from the end.
UPDATE
After seeing your update, I think there's a more reliable way to build the stream:
UnicodeEncoding uniEncoding = new UnicodeEncoding();
String message = "Message";
// You might not want to use the outer using statement that I have
// I wasn't sure how long you would need the MemoryStream object
using(MemoryStream ms = new MemoryStream())
{
var sw = new StreamWriter(ms, uniEncoding);
try
{
sw.Write(message);
sw.Flush();//otherwise you are risking empty stream
ms.Seek(0, SeekOrigin.Begin);
// Test and work with the stream here.
// If you need to start back at the beginning, be sure to Seek again.
}
finally
{
sw.Dispose();
}
}
As you can see, this code uses a StreamWriter to write the entire string (with proper encoding) out to the MemoryStream. This takes the hassle out of ensuring the entire byte array for the string is written.
Update: I stepped into issue with empty stream several time. It's enough to call Flush right after you've finished writing.
Try this "one-liner" from Delta's Blog, String To MemoryStream (C#).
MemoryStream stringInMemoryStream =
new MemoryStream(ASCIIEncoding.Default.GetBytes("Your string here"));
The string will be loaded into the MemoryStream, and you can read from it. See Encoding.GetBytes(...), which has also been implemented for a few other encodings.
You're using message.Length which returns the number of characters in the string, but you should be using the nubmer of bytes to read. You should use something like:
byte[] messageBytes = uniEncoding.GetBytes(message);
stringAsStream.Write(messageBytes, 0, messageBytes.Length);
You're then reading a single byte and expecting to get a character from it just by casting to char. UnicodeEncoding will use two bytes per character.
As Justin says you're also not seeking back to the beginning of the stream.
Basically I'm afraid pretty much everything is wrong here. Please give us the bigger picture and we can help you work out what you should really be doing. Using a StreamWriter to write and then a StreamReader to read is quite possibly what you want, but we can't really tell from just the brief bit of code you've shown.
I think it would be a lot more productive to use a TextWriter, in this case a StreamWriter to write to the MemoryStream. After that, as other have said, you need to "rewind" the MemoryStream using something like stringAsStream.Position = 0L;.
stringAsStream = new MemoryStream();
// create stream writer with UTF-16 (Unicode) encoding to write to the memory stream
using(StreamWriter sWriter = new StreamWriter(stringAsStream, UnicodeEncoding.Unicode))
{
sWriter.Write("Lorem ipsum.");
}
stringAsStream.Position = 0L; // rewind
Note that:
StreamWriter defaults to using an instance of UTF8Encoding unless specified otherwise. This instance of UTF8Encoding is constructed without a byte order mark (BOM)
Also, you don't have to create a new UnicodeEncoding() usually, since there's already one as a static member of the class for you to use in convenient utf-8, utf-16, and utf-32 flavors.
And then, finally (as others have said) you're trying to convert the bytes directly to chars, which they are not. If I had a memory stream and knew it was a string, I'd use a TextReader to get the string back from the bytes. It seems "dangerous" to me to mess around with the raw bytes.
You need to reset the stream to the beginning:
stringAsStream.Seek(0, SeekOrigin.Begin);
Console.WriteLine("Differs from:\t" + (char)stringAsStream.ReadByte());
This can also be done by setting the Position property to 0:
stringAsStream.Position = 0
I had using the BinaryFormatter to Serialize an object through NetworkStream
The code like this
//OpenConnection ...
TCPClient client = server.AcceptTCPConnection();
Message message = new Message("bla bla"); // This is the serializable class
NetworkStream stream = client.GetStream(); // Get Stream
BinaryFormatter bf = new BinaryFormatter();
bf.Serialize(stream, message);
stream.Flush();
stream.Close(); //Close Connection
And in client Code, we just need to Read from stream
bf.Deserialize(stream) as Message
to get the object we just sent from Server.
But there is a problem here, if I delete the line stream.Close(); the client cannot read this Object. Or I can change to stream.Dispose();
However, I want to use this stream again to send another Message, how I can do? Please help, it make me feel so headache ##
UPDATE:
I found the reason of this issue. Because I used one machine to run both client and server. It definitely worked well in two different machines. Someone can tell me why? Get big problem with this for a couple day ago.
Sending multiple separate messages involves "framing" - splitting the single channel into separate chunks that don't ever require the client to "read to end". Oddly, though, I was under the impression that BinaryFormatter already implemented basic framing - but: I could be wrong. In the general case, when working with a binary protocol, the most common approach is to prefix each message with the length of the payload, i.e.
using(var ms = new MemoryStream()) {
while(...)
{
// not shown: serialize to ms
var len BitConverter.GetBytes((int)ms.Length);
output.Write(len, 0, 4);
output.Write(ms.GetBuffer(), 0, (int) ms.Length);
ms.SetLength(0); // ready for next cycle
}
}
the caller has to:
read exactly 4 bytes (at least, for the above), or detect EOF
determine the length
read exactly that many bytes
deserialize
repeat
If that sounds like a lot of work, maybe just use a serializer that does all this for you; for example, with protobuf-net, this would be:
while(...) { // each item
Serializer.SerializeWithLengthPrefix(output, PrefixStyle.Base128, 1);
}
and the reader would be:
foreach(var msg in Serializer.DeserializeItems<Message>(
input, PrefixStyle.Base128, 1))
{
// ...
}
(note: this does not use the same format / rules as BinaryFormatter)
I'm trying to make a map for a game that I'm planning to create. The map should have two data files, and a picture file.
I want to put them together, to form a single file, and I only want to use the default libraries.
How can I do this, and still be able to separate them later?
A solution would be compression, but I couldn't find a way to compress multiple files using the gzipstreamer class.
You could use SharpZipLib to create a ZIP file.
Did you consider embedding the files as resources in the assembly (or in a separate assembly?)
A lot depends on the reasons why you want to group them.
Compression will cost time and CPU power.
I think you should consider embedding the resources in the assembly as Erno suggests.
But if you really want to pack them into a single file, you could do so by simply writing the length of each stream before the stream itself. You could then read the length byte and afterwards return the next length bytes as a Stream. Reading/writing with ugly methods below. The target stream could eventually be gzipped. Note that the naive methods below reads and writes the entire string to a single buffer and assumes that no file is larger than int.MaxValue.
But I would not recommend using just the standard libraries.
static void Append(Stream source, Stream target)
{
BinaryWriter writer = new BinaryWriter(target);
BinaryReader reader = new BinaryReader(source);
writer.Write((long)source.Length);
byte[] buffer = new byte[1024];
int read;
do
{
read = reader.Read(buffer, 0, buffer.Length);
writer.Write(buffer, 0, read);
}
while (read > 0);
writer.Flush();
}
static Stream ReadNextStream(Stream packed)
{
BinaryReader reader = new BinaryReader(packed);
int streamLength = (int)reader.ReadInt64();
MemoryStream result = new MemoryStream();
byte[] buffer = new byte[streamLength];
reader.Read(buffer, 0, buffer.Length);
BinaryWriter writer = new BinaryWriter(result);
writer.Write(buffer, 0, buffer.Length);
writer.Flush();
result.Seek(0, SeekOrigin.Begin);
return result;
}
Gzip compression only works on one file (it only ever has). You could try ZIP, 7-ZIP or some other archive format that allows multiple files. Alternately you can TAR the files together first, which was common practice for the compression scheme Gzip was invented to replace.
I had a simiar question a while ago here about saving 2 XML files in one file.
See my answer with code.
"I ended up writing my own Stream, which can be thought of as a multistream. It allows you to treat one stream as multiple streams in succession. i.e. pass a multistream to an xml parser (or anything else) and it'll read up to a marker, which says 'this is the end of the stream'. If you then pass that same stream to another xml parser, it'll read from that marker, to the next one or EOF"
Your basic usage would be:
Writing:
Open File Stream
Create MultiStream passing in File Stream in constructor
Write data file to multistream
Call write end of stream marker on multistream
Write 2nd data file to multistream
Call write end of stream marker on multistream
Save picture to multistream
Close multistream
Close file stream
Reading:
Open File Stream
Create MultiStream passing in File Stream in constructor
Read data file
Call advance to next stream on multistream
Read 2nd data file
Call advance to next stream on multistream
Read image (Image.FromStream() etc.)
Close multistream
Close file stream
I am working on C# sockets and using XMLSerializer to send and receive data.
The XML data are sent from a server to a client over a network connection using TCP/IP protocol. The XML.Serializer.Serialize(stream) serializes the XML data and send them over the socket connection but when I want to use the XMLSerializer.Deserialize(stream) to read. The sent data returns a xml parse error.
Here is how I'm serializing:
Memory Stream ms = new MemoryStream();
FrameClass frame= new FrameClass ();
frame.string1 = "hello";
frame.string2 = "world";
XmlSerializer xmlSerializer = new XmlSerializer(frame.GetType());
xmlSerializer.Serialize(ms, frame);
socket.Send(ms.GetBuffer(), (int)ms.Length, SocketFlags.None);
Deserializing:
FrameClass frame;
XmlSerializer xml = new XmlSerializer(typeof(FrameClass));
frame= (FrameClass)xml.Deserialize(new MemoryStream(sockCom.SocketBuffer));
listbox1.Items.Add(frame.string1);
listbox2.Items.Add(frame.string2);
I think it has something to do with sending the data one right after another.
Can anyone teach me how to do this properly?
Have you received all of the data before attempting to deserialize (it's not clear from your code). I'd be inclined to receive all of the data into a local string and the deserialize from that rather than attempting to directly deserialize from the socket. It would also allow you to actually look at the data in the debugger before deserializing it.
Try this:
using (MemoryStream ms = new MemoryStream())
{
FrameClass frame= new FrameClass ();
frame.string1 = "hello";
frame.string2 = "world";
XmlSerializer xmlSerializer = new XmlSerializer(frame.GetType());
xmlSerializer.Serialize(ms, frame);
ms.Flush();
socket.Send(ms.GetBuffer(), (int)ms.Length, SocketFlags.None);
}
If you're sending the Frame XML one right after the other, then you're not sending an XML document. The XML Serializer will attempt to deserialize your entire document!
I don't have time to research this now, but look into the XmlReaderSettings property for reading XML fragments. You would then create an XmlReader over the memorystream with those settings, and call it in a loop.
The important thing is to flush the stream. It's also useful to put the stream in a using block to ensure it's cleaned up quickly.
Besides what #John said about the Flush call, your code looks alright.
You say you're sending multiple FrameClass data pieces, then the code should work sending just a single piece of data.
If you need to send multiple data objects, then you cannot send them all in one go, otherwise the deserialization process will stumble over the data.
You could setup some communication between the server & the client so the server knows what it's getting.
client: I have some data
Server: ok I'm ready, send it
client: sends
Server: done processing
repeat process...