I just spent quite a few hours reading up on TCP servers and my desired protocol I was trying to implement, and finally got everything working great. I noticed the code looks like absolute bollocks (is the the correct usage? Im not a brit) and would like some feedback on optimizing it, mostly for reuse and readability.
The packet formats are always int, int, int, string, string.
try
{
BinaryReader reader = new BinaryReader(clientStream);
int packetsize = reader.ReadInt32();
int requestid = reader.ReadInt32();
int serverdata = reader.ReadInt32();
Console.WriteLine("Packet Size: {0} RequestID: {1} ServerData: {2}", packetsize, requestid, serverdata);
List<byte> str = new List<byte>();
byte nextByte = reader.ReadByte();
while (nextByte != 0)
{
str.Add(nextByte);
nextByte = reader.ReadByte();
}
// Password Sent to be Authenticated
string string1 = Encoding.UTF8.GetString(str.ToArray());
str.Clear();
nextByte = reader.ReadByte();
while (nextByte != 0)
{
str.Add(nextByte);
nextByte = reader.ReadByte();
}
// NULL string
string string2 = Encoding.UTF8.GetString(str.ToArray());
Console.WriteLine("String1: {0} String2: {1}", string1, string2);
// Reply to Authentication Request
MemoryStream stream = new MemoryStream();
BinaryWriter writer = new BinaryWriter(stream);
writer.Write((int)(1)); // Packet Size
writer.Write((int)(requestid)); // Mirror RequestID if Authenticated, -1 if Failed
byte[] buffer = stream.ToArray();
clientStream.Write(buffer, 0, buffer.Length);
clientStream.Flush();
}
I am going to be dealing with other packet types as well that are formatted the same (int/int/int/str/str), but different values. I could probably create a packet class, but this is a bit outside my scope of knowledge for how to apply it to this scenario. If it makes any difference, this is the Protocol I am implementing.
http://developer.valvesoftware.com/wiki/Source_RCON_Protocol
Thoughts:
you arent really using the reader except for a few ints; otherwise, all you need is ReadByte you can do that from the Stream, and save some indirection/confusion
read the ints manually to avoid Endianness issues
reading byte by byte can be expensive; if possible, try to fill a buffer (or rather: read the right amount of data) by looping over Read rather that ReadByte
if multiple messages are coming down the same pipe, reading to EOF will probably fail (either corrupt the data or block forever); you usually need either a terminator sequence or a length-prefix. I prefer the latter, as it let's you use Read instead of ReadByte
I assume that is packetSize in your example; it is critical to use this: to separate the messages, to verify you have an entire message, and to deny over-sized data
consider whether async (BeginRead) is suitable - sometimes yes, sometimes no; and note that this makes disposal trickier as you can't use "using" with async
when using MemoryStream, using .GetBuffer() in combination with .Length has less overhead than using .ToArray()
The first thing that jumps out to me is to always use the using statement with any object that implements IDisposable. This will ensure that your objects are properly disposed of even in the event of an exception.
private void FillList(BinaryReader reader, List list)
{
while (reader.PeekChar() != -1)
{
list.Add(reader.ReadByte());
}
}
...
try
{
int packetsize, requestid, serverdata;
string string1, string2;
List<byte> str = new List<byte>();
using (BinaryReader reader = new BinaryReader(clientStream))
{
packetsize = reader.ReadInt32();
requestid = reader.ReadInt32();
serverdata = reader.ReadInt32();
Console.WriteLine("Packet Size: {0} RequestID: {1} ServerData: {2}", packetsize, requestid, serverdata);
FillList(reader, str);
// Password Sent to be Authenticated
string1 = Encoding.UTF8.GetString(str.ToArray());
str.Clear();
FillList(reader, str);
}
// NULL string
string2 = Encoding.UTF8.GetString(str.ToArray());
Console.WriteLine("String1: {0} String2: {1}", string1, string2);
// Reply to Authentication Request
using (MemoryStream stream = new MemoryStream())
using (BinaryWriter writer = new BinaryWriter(stream))
{
writer.Write((int)(1)); // Packet Size
writer.Write((int)(requestid)); // Mirror RequestID if Authenticated, -1 if Failed
byte[] buffer = stream.ToArray();
clientStream.Write(buffer, 0, buffer.Length);
clientStream.Flush();
}
}
Related
I'm tring to translate C code to C# and I stumbled upon a line of code which I'm having problems translating.
sprintf((char*)&u8FirmareBuffer[0x1C0] + strlen((char*)&u8FirmareBuffer[0x1C0]), ".B%s", argv[3]);
specifically this line.
u8FirmwareBuffer is a unsigned char array in C, a byte array in C# I would guess.
argv[3] is a string.
How can I translate this line to C#.
Thank you for your help.
Edit: This has been marked as a duplicate, but I think they differenciate because I am using pointers which don't work with the solutions presented on the marked post.
You could do something like:
string myString = "This is my string";
byte[] buffer = new byte[1024];
int offset = 0;
// if you pass a byte buffer to the constructor of a memorystream, it will use that, don't forget that it cannot grow the buffer.
using (var memStream = new MemoryStream(buffer))
{
// you can even seek to a specific position
memStream.Seek(offset, SeekOrigin.Begin);
// check your encoding..
var data = Encoding.UTF8.GetBytes(myString);
// write it on the current offset in the memory stream
memStream.Write(data, 0, data.Length);
}
It's also possible with a StreamWriter
string myString = "This is my string";
byte[] buffer = new byte[1024];
int offset = 0;
// if you pass a byte buffer to the constructor.....(see above)
using (var memStream = new MemoryStream(buffer))
using (var streamWriter = new StreamWriter(memStream))
{
// you can even seek to a specific position
memStream.Seek(offset, SeekOrigin.Begin);
streamWriter.Write(myString);
streamWriter.Flush();
// don't forget to flush before you seek again
}
public static byte[] ReadMemoryMappedFile(string fileName)
{
long length = new FileInfo(fileName).Length;
using (var stream = File.Open(fileName, FileMode.OpenOrCreate, FileAccess.Read, FileShare.ReadWrite))
{
using (var mmf = MemoryMappedFile.CreateFromFile(stream, null, length, MemoryMappedFileAccess.Read, null, HandleInheritability.Inheritable, false))
{
using (var viewStream = mmf.CreateViewStream(0, length, MemoryMappedFileAccess.Read))
{
using (BinaryReader binReader = new BinaryReader(viewStream))
{
var result = binReader.ReadBytes((int)length);
return result;
}
}
}
}
}
OpenFileDialog openfile = new OpenFileDialog();
openfile.Filter = "All Files (*.*)|*.*";
openfile.ShowDialog();
byte[] buff = ReadMemoryMappedFile(openfile.FileName);
texteditor.Text = BitConverter.ToString(buff).Replace("-"," "); <----A first chance exception of type 'System.OutOfMemoryException' occurred in mscorlib.dll
I get a System.OutOfMemory exception when trying to read large files.
I've read a lot for 4 weeks in all the web... and tried a lot!!! But still, I can't seem to find a good solution to my problem.
Please help me..
Update
public byte[] FileToByteArray(string fileName)
{
byte[] buff = null;
FileStream fs = new FileStream(fileName,
FileMode.Open,
FileAccess.Read);
BinaryReader br = new BinaryReader(fs);
long numBytes = new FileInfo(fileName).Length;
buff = br.ReadBytes((int)numBytes);
//return buff;
return File.ReadAllBytes(fileName);
}
OR
public static byte[] FileToByteArray(FileStream stream, int initialLength)
{
// If we've been passed an unhelpful initial length, just
// use 32K.
if (initialLength < 1)
{
initialLength = 32768;
}
BinaryReader br = new BinaryReader(stream);
byte[] buffer = new byte[initialLength];
int read = 0;
int chunk;
while ((chunk = br.Read(buffer, read, buffer.Length - read)) > 0)
{
read += chunk;
// If we've reached the end of our buffer, check to see if there's
// any more information
if (read == buffer.Length)
{
int nextByte = br.ReadByte();
// End of stream? If so, we're done
if (nextByte == -1)
{
return buffer;
}
// Nope. Resize the buffer, put in the byte we've just
// read, and continue
byte[] newBuffer = new byte[buffer.Length * 2];
Array.Copy(buffer, newBuffer, buffer.Length);
newBuffer[read] = (byte)nextByte;
buffer = newBuffer;
read++;
}
}
// Buffer is now too big. Shrink it.
byte[] ret = new byte[read];
Array.Copy(buffer, ret, read);
return ret;
}
I still get a System.OutOfMemory exception when trying to read large files.
If your file is 4GB, then BitConverter will turn each byte into XX- string, each char in string is 2 bytes * 3 chars per byte * 4 294 967 295 bytes = 25 769 803 770. You need +25Gb of free memory to fit entire string, plus you already have your file in memory as byte array.
Besides, no single object in a .Net program may be over 2GB. Theoretical limit for a string length would be 1,073,741,823 chars, but you also need to have a 64-bit process.
So solution in your case - open FileStream. Read first 16384 bytes (or how much can fit on your screen), convert to hex and display, and remember file offset. When user wants to navigate to next or previous page - seek to that position in file on disk, read and display again, etc.
You need to read the file in chunks, keep track of where you are in the file, page the contents on screen and use seek and position to move up and down in the file stream.
You will not be able to display 4Gb file reading all of it in memory first by any approach.
The approach is to virtualize the data, reading only the visible lines when user scrolls. If you need to do a read-only text viewer then you can use WPF ItemsControl with virtulizing stack panel and bind to custom IList collection which will lazily fetch lines from the file calculating file offset by for the line index.
I send the serialized data over LAN network but sometimes the information is lost! Process is the following:
Sender:
string mydata is being serialized
string mydata is converted to byte[] bytes_of_mydata
int size_of_mydata is the length of byte[] bytes_of_mydata
int size_of_mydata itself is turned into byte[] bytes_size_of_mydata
byte[] bytes_of_mydata and byte[] bytes_size_of_mydata are sent
Receiver:
i first receive byte[] bytes_size_of_mydata
retrieve length int size_of_mydata of the second message from byte[] bytes_size_of_mydata
i then receive byte[] bytes_of_mydata, knowing exact length!
i then convert byte[] bytes_of_mydata to string mydata
deserialize string mydata
This approach usually works in most situations, but sometimes my data is not transmitted fully, so the string can't be deserialized.
I've debugged the received byte[] on the "receiver" and here is what happens:
I get the size of second message:
int size_of_second_message = BitConverter.ToInt32(dataByteSize, 0); // 55185
I start to receive the second message to byte array:
Byte[] dataByte = new Byte[55185];
But starting from position 5840 I start to receive 0 (nulls), so the part "5840 - 55185" are all "0":
byte[5836] = 53;
byte[5837] = 57;
byte[5838] = 54;
byte[5839] = 49;
byte[5840] = 0; // information ends to flow
byte[5841] = 0;
byte[5842] = 0;
byte[5843] = 0;
//....
byte[55185] = 0;
The example from the above is taken from an actual debugger!
So what's the problem? It's like the connection is being lost during transmission!! Why is it happening and how do I counter this problem? It doesn't happen on "every-time" basis.
And here comes the code
Send:
//text_message - my original message
//Nw - network stream
MemoryStream Fs = new MemoryStream(ASCIIEncoding.Default.GetBytes(text_message));
Byte[] buffer = Fs.ToArray(); // total 55185 bytes (as in example)
Byte[] bufferSize = BitConverter.GetBytes(Fs.Length); // 32 bytes represent size
bufferSize = GetNewByteSize(bufferSize);
Nw.Write(bufferSize, 0, bufferSize.Length); // send size
Nw.Flush();
Nw.Write(buffer, 0, buffer.Length); // send message
Nw.Flush();
Receive:
//get first(SIZE) bytes:
int ReadSize = 0; int maxSize = 32; // 32 - constant!
Byte[] dataByteSize = new Byte[maxSize];
int origsize;
using (var strm = new MemoryStream())
{
ReadSize = Nw.Read(dataByteSize, 0, maxSize);
strm.Write(dataByteSize, 0, ReadSize);
strm.Seek(0, SeekOrigin.Begin);
origsize = BitConverter.ToInt32(dataByteSize, 0); // origsize = 55185
}
Nw.Flush();
//get next(MESSAGE) bytes:
string message = ""; int thisRead = 0;
int max = Convert.ToInt32(origsize); // origsize = 55185
Byte[] dataByte = new Byte[max];
using (var strm = new MemoryStream())
{
thisRead = Nw.Read(dataByte, 0, max);
strm.Write(dataByte, 0, thisRead);
strm.Seek(0, SeekOrigin.Begin);
using (StreamReader reader = new StreamReader(strm))
{
message = reader.ReadToEnd();
}
}
Nw.Flush();
// message - the message that is being transmitted partly (sometimes)!
I didn't want to post the code but you guys usually ask "show us what you've done", so here it is!
Edit
Temporary fix is to switch to StreamWriter, reader.
Receive + send (server):
NetworkStream Nw = new NetworkStream(handlerSocket.Client);
string toreceive = "";
StreamReader reader = new StreamReader(Nw);
toreceive = reader.ReadLine();
string text_message = "to send back";
StreamWriter writer = new StreamWriter(Nw);
writer.WriteLine(text_message);
writer.Flush();
Nw.Close();
Send + receive (client):
NetworkStream Nw = new NetworkStream(handlerSocket.Client);
StreamWriter writer = new StreamWriter(Nw);
writer.WriteLine("to send");
writer.Flush();
string toreceive = new StreamReader(Nw).ReadLine();
writer.Close();
Nw.Close();
I'm looking for a solution regarding original problem, but so far everything is working due to temporary fix.
TCP is a stream based protocol which lets you read as much data has yet been received. Just because you send data in one step, there is no guarantee that the data will be received in the same block. You have to loop on the receiving side until you have received all the data you anticipate.
By the way I think that your protocol with an explicit length field at the start is really good, it makes up for simple client code (as simple as it gets at least).
Some time back I asked a question on built in functionality to wait until X bytes of data is available: .NET blocking socket read until X bytes are available?. The answer is unfortunately no.
i'm implementing a wcf service that accepts image streams. however i'm currently getting an exception when i run it. as its trying to get the length of the stream before the stream is complete. so what i'd like to do is buffer the stream until its complete. however i cant find any examples of how to do this...
can anyone help?
my code so far:
public String uploadUserImage(Stream stream)
{
Stream fs = stream;
BinaryReader br = new BinaryReader(fs);
Byte[] bytes = br.ReadBytes((Int32)fs.Length);// this causes exception
File.WriteAllBytes(filepath, bytes);
}
Rather than try to fetch the length, you should read from the stream until it returns that it's "done". In .NET 4, this is really easy:
// Assuming we *really* want to read it into memory first...
MemoryStream memoryStream = new MemoryStream();
stream.CopyTo(memoryStream);
memoryStream.Position = 0;
File.WriteAllBytes(filepath, memoryStream);
In .NET 3.5 there's no CopyTo method, but you can write something similar yourself:
public static void CopyStream(Stream input, Stream output)
{
byte[] buffer = new byte[8192];
int bytesRead;
while ((bytesRead = input.Read(buffer, 0, buffer.Length)) > 0)
{
output.Write(buffer, 0, bytesRead);
}
}
However, now we've got something to copy a stream, why bother reading it all into memory first? Let's just write it straight to a file:
using (FileStream output = File.OpenWrite(filepath))
{
CopyStream(stream, output); // Or stream.CopyTo(output);
}
I'm not sure what you are returning (or not returning), but something like this might work for you:
public String uploadUserImage(Stream stream) {
const int KB = 1024;
Byte[] bytes = new Byte[KB];
StringBuilder sb = new StringBuilder();
using (BinaryReader br = new BinaryReader(stream)) {
int len;
do {
len = br.Read(bytes, 0, KB);
string readData = Encoding.UTF8.GetString(bytes);
sb.Append(readData);
} while (len == KB);
}
//File.WriteAllBytes(filepath, bytes);
return sb.ToString();
}
A string can hold up to 2 GB, I believe.
Try this :
using (StreamWriter sw = File.CreateText(filepath))
{
stream.CopyTo(sw);
sw.Close();
}
Jon Skeets answer for .Net 3.5 and below using a Buffer Read is actually done incorrectly.
The buffer isn't cleared between reads which can result in issues on any read that returns less than 8192, for example if the 2nd read, read 192 bytes, the 8000 last bytes from the first read would STILL be in the buffer which would then be returned to the stream.
My code below you supply it a Stream and it will return a IEnumerable array.
Using this you can for-each it and Write to a MemoryStream and then use .GetBuffer() to end up with a compiled merged byte[].
private IEnumerable<byte[]> ReadFullStream(Stream stream) {
while(true) {
byte[] buffer = new byte[8192];//since this is created every loop, its buffer is cleared
int bytesRead = stream.Read(buffer, 0, buffer.Length);//read up to 8192 bytes into buffer
if (bytesRead == 0) {//if we read nothing, stream is finished
break;
}
if(bytesRead < buffer.Length) {//if we read LESS than 8192 bytes, resize the buffer to essentially remove everything after what was read, otherwise you will have nullbytes/0x00bytes at the end of your buffer
Array.Resize(ref buffer, bytesRead);
}
yield return buffer;//yield return the buffer data
}//loop here until we reach a read == 0 (end of stream)
}
I'm having trouble reading a compressed (deflated) data file using C# .NET DeflateStream(..., CompressionMode.Decompress). The file was written earlier using DeflateStream(..., CompressionMode.Compress), and it seems to be just fine (I can even decompress it using a Java program).
However, the first Read() call on the input stream to decompress/inflate the compressed data returns a length of zero (end of file).
Here's the main driver, which is used for both compression and decompression:
public void Main(...)
{
Stream inp;
Stream outp;
bool compr;
...
inp = new FileStream(inName, FileMode.Open, FileAccess.Read);
outp = new FileStream(outName, FileMode.Create, FileAccess.Write);
if (compr)
Compress(inp, outp);
else
Decompress(inp, outp);
inp.Close();
outp.Close();
}
Here's the basic code for decompression, which is what is failing:
public long Decompress(Stream inp, Stream outp)
{
byte[] buf = new byte[BUF_SIZE];
long nBytes = 0;
// Decompress the contents of the input file
inp = new DeflateStream(inp, CompressionMode.Decompress);
for (;;)
{
int len;
// Read a data block from the input stream
len = inp.Read(buf, 0, buf.Length); //<<FAILS
if (len <= 0)
break;
// Write the data block to the decompressed output stream
outp.Write(buf, 0, len);
nBytes += len;
}
// Done
outp.Flush();
return nBytes;
}
The call marked FAILS always returns zero. Why? I know it's got to be something simple, but I'm just not seeing it.
Here's the basic code for compression, which works just fine, and is almost exactly the same as the decompression method with the names swapped:
public long Compress(Stream inp, Stream outp)
{
byte[] buf = new byte[BUF_SIZE];
long nBytes = 0;
// Compress the contents of the input file
outp = new DeflateStream(outp, CompressionMode.Compress);
for (;;)
{
int len;
// Read a data block from the input stream
len = inp.Read(buf, 0, buf.Length);
if (len <= 0)
break;
// Write the data block to the compressed output stream
outp.Write(buf, 0, len);
nBytes += len;
}
// Done
outp.Flush();
return nBytes;
}
Solved
After seeing the correct solution, the constructor statement should be changed to:
inp = new DeflateStream(inp, CompressionMode.Decompress, true);
which keeps the underlying input stream open, and the following line needs to be added following the inp.Flush() call:
inp.Close();
The Close() calls forces the deflater stream to flush its internal buffers. The true flag prevents it from closing the underlying stream, which is closed later in Main(). The same changes should also be made to the Compress() method.
In your decompress method, are reassigning inp to a new Stream (a deflate stream). You never close that Deflate stream, but you do close the underlying file stream in Main(). A similar thing is going on in the compress method.
I think that the problem is that the underlying file stream is being closed before the deflate stream's finalizers are automatically closing them.
I added 1 line of code to your Decompress and Compress methods:
inp.Close() // to the Decompressmehtod
outp.Close() // to the compress method.
a better practice would be to enclose the streams in a using clause.
Here's an alternative way to write your Decompress method (I tested, and it works)
public static long Decompress(Stream inp, Stream outp)
{
byte[] buf = new byte[BUF_SIZE];
long nBytes = 0;
// Decompress the contents of the input file
using (inp = new DeflateStream(inp, CompressionMode.Decompress))
{
int len;
while ((len = inp.Read(buf, 0, buf.Length)) > 0)
{
// Write the data block to the decompressed output stream
outp.Write(buf, 0, len);
nBytes += len;
}
}
// Done
return nBytes;
}
I had the same problem with GZipStream, Since we had the original length stored I had to re-write the code to only read the number of bytes expected in the original file.
Hopefully I'm about to learn that there was a better answer (fingers crossed).