i am new to the C# world. I am using it for fast deployment of a solution to capture a live feed which comes in this form (curly brackets for clarity): {abcde}{CompressedMessage}, where {abcde} constitutes 5 characters indicating the length of the compressed message. The CompressedMessage is compressed using XCeedZip.dll, and needs to be uncompressed using the dll's uncompress method. The uncompress method returns an integer value indicating success or failure (of various sorts, eg no license failure, uncompression failure etc). I am receiving failure 1003 http://doc.xceedsoft.com/products/XceedZip/ for reference of the return values from the uncompress method.
while(true){
byte[] receiveByte = new byte[1000];
sock.Receive(receiveByte);
string strData =System.Text.Encoding.ASCII.GetString(receiveByte,0,receiveByte.Length);
string cMesLen = strData.Substring(0,5); // length of compressed message;
string compressedMessageStr = strData.Substring(5,strData.Length-5);
byte[] compressedBytes = System.Text.Encoding.ASCII.GetBytes(compressedMessageStr);
//instantiating xceedcompression object
XceedZipLib.XceedCompression obXCC = new XceedZipLib.XceedCompression();
obXCC.License("blah");
// uncompress method reference http://doc.xceedsoft.com/products/XceedZip/
// visual studio displays Uncompress method signature as Uncompress(ref object vaSource, out object vaUncompressed, bool bEndOfData)
object oDest;
object oSource = (object)compressedBytes;
int status = (int) obXCC.Uncompress(ref oSource, out oDest, true);
Console.WriteLine(status); /// prints 1003 http://doc.xceedsoft.com/products/XceedZip/
}
So basically my question boils down to invocation of the uncompress method and correct way of passing the parameters. I am in unfamiliar territory in the .net world, so i won't be surprised if the question is really simplistic.
Thanks for replies ..
##################################### updates
I am now doing the following:
int iter = 1;
int bufSize = 1024;
byte[] receiveByte = new byte[bufSize];
while (true){
sock.Receive(receiveByte);
//fetch compressed message length;
int cMesLen = Convert.ToInt32(System.Text.Encoding.ASCII.GetString(receiveByte,0,5));
byte[] cMessageByte = new byte[cMesLen];
if (i==1){
if (cMesLen < bufSize){
for (int i = 5; i < 5+cMesLen; ++i){
cMessageByte[i-5] = b[i];
}
}
}
XceedZipLib.XceedCompression obXCC = new XceedZipLib.XceedCompression();
obXCC.License("blah");
object oDest;
object oSource = (object) cMessageByte;
int status = (int) obXCC.Uncompress(ref oSource, out oDest, true);
if (iter==1){
byte[] testByte = objectToByteArray(oDest);
Console.WriteLine(System.Text.Encoding.ASCII.GetString(testByte,0,testByte.Length));
}
}
private byte[] objectToByteArray(Object obj){
if (obj==null){
return null;
}
BinaryFormatter bf = new BinaryFormatter();
MemoryStream ms = new MemoryStream();
bf.Serialize(ms,obj);
return ms.ToArray();
}
Problem is the testByte writeline command prints out gibberish. Any suggestions on how to move forward on this ? the status variable of uncompress is good and equal to 0 now.
The first mistake, always, is not looking at the return value of Receive; you have no idea how much data you just read, nor whether it constitutes an entire message.
It seems likely to me that you have corrupted the message payload by treating the entire data as ASCII. Rather than doing a GetString on the entire buffer, you should use GetString specifying only to use 5 bytes.
Correct process:
keep calling Receive (buffering the data, or increasing the offset and decreasing the count) until you have at least 5 bytes
process these 5 bytes to get the payload length
keep calling Receive (buffering the data, or increasing the offset and decreasing the count) until you have at least the payload length
process the payload without ever converting to/from ASCII
Related
I am curently working on a GZIP HTTP decompression.
My server receives some data and im cropping and saving it in binary mode.
I've made a little script to download the gzip from stackoverflow and saved it to a .gz file.
Works fine!
But the "gzip" I receive from my fortigate-firewall ends up being corrupted.
Corrupted and working file here: https://gofile.io/d/j520Nr
The buffer is the corrupted file - and im not sure why.
Both files are extremely different (at least how I see it) - but the GZIP header is definitely present!
Can someone maybe compare these two files and tell me why they are that different?
Or maybe even show me how to fix it?
Thats the gzip html url for both of the files: What is the best way to parse html in C#?
My corrupted file is around 2KB larger!
I would be happy for every step in the right direction - maybe it is something that can be fixed really easy!
The following code should show you my workflow, "ReadAll" is pretty slow but reads all from the stream. It will be optimized ofc (maybe its the problem of the wrong gzip stream?)
public static byte[] ReadAll(NetworkStream stream, int buffer)
{
byte[] data = new byte[buffer];
using MemoryStream ms = new MemoryStream();
int numBytesRead;
while ((numBytesRead = stream.Read(data, 0, data.Length)) > 0)
{
ms.Write(data, 0, numBytesRead);
}
return ms.ToArray();
}
private bool Handled = false;
/// <summary>
/// Handles Client and passes matches to the parser for more investigation
/// </summary>
/// <param name="obj"></param>
private void HandleClient(object obj)
{
TcpClient client = (TcpClient)obj;
Out.Log(LogLevel.Verbose, $"Client {client.Client.RemoteEndPoint} connected");
Data = null; // Resets data after each received stream
// Get a stream object for reading and writing
NetworkStream stream = client.GetStream();
//MemoryStream memory = new MemoryStream();
// Wait to receive all the data sent by the client.
if (stream.CanRead)
{
Out.Log(LogLevel.Debug, "Can read stream");
StringBuilder c_completeMessage = new StringBuilder();
if (!Handled)
{
Out.Log(LogLevel.Warning, "Handling first and last client.");
Handled = true;
int breakPoint = 0;
byte[] res = ReadAll(stream, 1024);
for (int i = 0; i < res.Length; i++)
{
int xy = res[i];
int yy = res[i + 1];
if (res[i].Equals(31) && res[i + 1].Equals(139))
{
breakPoint = i;
Out.Log(LogLevel.Error, GZIP_MAGIC + $" found. Magic Number of GZIP at :{breakPoint}:");
break;
}
continue;
}
byte[] res2 = res.SubArray(breakPoint, res.Length - breakPoint - 7); // (7 for offset linebreaks, eol, etc)
res2.WriteToFile(#"C:\Users\--\Temporary\Buffer_ReadFully_cropped.gz");
As mentioned before, chunking and buffer size played a big role here.
Remember, ICAP uses chunking so you have to respond to the previous package with a CONTINUE, otherwise you will just receive the first X bytes from the server.
I am attempting to connect my laptop with my standalone pc using C# TCPClient class.
Laptop is running a simple console application and plays the role of the server.
PC is a Unity aplication (2018.1.6f1 with .Net4.x Mono)
The code for sending is
public void SendData() {
Debug.Log("Sending data");
NetworkStream ns = client.GetStream();
BinaryFormatter bf = new BinaryFormatter();
TCPData data = new TCPData(true);
using (MemoryStream ms = new MemoryStream()) {
bf.Serialize(ms, data);
byte[] bytes = ms.ToArray();
ns.Write(bytes, 0, bytes.Length);
}
}
The same code is used in the Laptop's project, except Debug.Log() is replaced by Console.WriteLine()
For data reception I use
public TCPData ReceiveData() {
Debug.Log("Waiting for Data");
using (MemoryStream ms = new MemoryStream()) {
byte[] buffer = new byte[2048];
int i = stream.Read(buffer, 0, buffer.Length);
stream.Flush();
ms.Write(buffer, 0, buffer.Length);
ms.Seek(0, SeekOrigin.Begin);
BinaryFormatter bf = new BinaryFormatter();
bf.Binder = new CustomBinder();
TCPData receivedData = (TCPData)bf.Deserialize(ms);
Debug.Log("Got the data");
foreach (string s in receivedData.stuff) {
Debug.Log(s);
}
return receivedData;
}
}
Again the same on both sides,
The data I am trying to transfer looks like this
[Serializable, StructLayout(LayoutKind.Sequential)]
public struct TCPData {
public TCPData(bool predefined) {
stuff = new string[2] { "Hello", "World" };
ints = new List<int>() {
0,1,2,3,4,5,6,7,8,9
};
}
public string[] stuff;
public List<int> ints;
}
The custom binder is from here
without it I get an assembly error
with it I get Binary stream '0' does not contain a valid BinaryHeader. Possible causes are invalid stream or object version change between serialization and deserialization.
Now the problem:
Sending this from PC to Laptop - 100% success rate
Sending this from Laptop to PC - 20% success rate (80% is the Exception above)
How is it even possible that it "sometimes" works ?
Shouldn't it be 100% or 0% ?
How do I get it to work ?
Thanks
E: Ok thanks to all the suggestions I managed to increase the chances of success, but it still occasionally fails.
I send a data size "packet" which is 80% of the time received correctly, but in some cases the number I get from the byte[] is 3096224743817216 (insanely big) compared to the sent ~500.
I am using Int64 data type.
E2: In E1 I was sending the data length packet separately, now I have them merged, which does interpret the length properly, but now I am unable to deserialize the data... every time I get The input stream is not a valid binary format. The starting contents (in bytes) are: 00-00-00-00-00-00-04-07-54-43-50-44-61-74-61-02-00 ...
I read the first 8 bytes from the stream and the remaining 'x' are the data, deserializing it on server works, deserializing the same data throws.
E3: Fixed it by rewriting the stream handling code, I made a mistake somewhere in there ;)
NetworkStream.Read() doesn't block until it reads the requested number of bytes:
"This method reads data into the buffer parameter and returns the number of bytes successfully read. If no data is available for reading, the Read method returns 0. The Read operation reads as much data as is available, up to the number of bytes specified by the size parameter. If the remote host shuts down the connection, and all available data has been received, the Read method completes immediately and return zero bytes."
You must
1) Know how many bytes you are expecting
and
2) Loop on Read() until you have received the expected bytes.
If you use a higher-level protocol like HTTP or Web Sockets they will handle this "message framing" for you. If you code on TCP/IP directly, then that's your responsibility.
Ack. I am trying to open a specific entry in a zip file archive and store the contents in a string, instead of saving it to a file. I cannot use disk space for this per the client.
Here's what I have:
string scontents = "";
byte[] abbuffer = null;
MemoryStream oms = new MemoryStream();
try
{
//get the file contents
ozipentry.Open().CopyTo(oms);
int length = (int)oms.Length; // get file length
abbuffer = new byte[length]; // create buffer
int icount; // actual number of bytes read
int isum = 0; // total number of bytes read
// read until Read method returns 0 (end of the stream has been reached)
while ((icount = oms.Read(abbuffer, isum, length - isum)) > 0)
{
isum += icount; // sum is a buffer offset for next reading
}
scontents = BytesToString(abbuffer); <----abbuffer is filled with Ascii 0
}
finally
{
oms.Close();
}
The variable abbuffer is supposed to hold that contents of the stream, but all it holds is a bunch of ascii zeros, which I guess means it didn't read (or copy) the stream! But I do not get any error messages or anything. Can someone tell me how to get this working?
I've looked everywhere on stack and on the web, and no where does anyone answer this question specifically for ASP.NET 4.5 ZipArchive library. I cannot use any other library, so if you offer an answer in that, while it would be educational, won't help me at all in this instance. Thanks so much for any help!
One more thing. 'ozipentry' is of type ZipArchiveEntry and is an element in a ZipArchive Entries array. (ie ozipentry = oziparchive.Entries[i])
Oops. One more thing! The function 'BytesToString' is not included, because it is irrelevant. Before the function is called, the abbuffer array is already filled with 0's
Ok. Sorry for being so dense. I realized I was overthinking this. I changed to function to do this:
osr = new StreamReader(ozipentry.Open(), Encoding.Default);
scontents = osr.ReadToEnd();
And it worked fine! Didn't even have to worry about Encoding...
Edit: Solution is at bottom of post
I am trying my luck with reading binary files. Since I don't want to rely on byte[] AllBytes = File.ReadAllBytes(myPath), because the binary file might be rather big, I want to read small portions of the same size (which fits nicely with the file format to read) in a loop, using what I would call a "buffer".
public void ReadStream(MemoryStream ContentStream)
{
byte[] buffer = new byte[sizePerHour];
for (int hours = 0; hours < NumberHours; hours++)
{
int t = ContentStream.Read(buffer, 0, sizePerHour);
SecondsToAdd = BitConverter.ToUInt32(buffer, 0);
// further processing of my byte[] buffer
}
}
My stream contains all the bytes I want, which is a good thing. When I enter the loop several things cease to work.
My int t is 0although I would presume that ContentStream.Read() would process information from within the stream to my bytearray, but that isn't the case.
I tried buffer = ContentStream.GetBuffer(), but that results in my buffer containing all of my stream, a behaviour I wanted to avoid by using reading to a buffer.
Also resetting the stream to position 0 before reading did not help, as did specifying an offset for my Stream.Read(), which means I am lost.
Can anyone point me to reading small portions of a stream to a byte[]? Maybe with some code?
Thanks in advance
Edit:
Pointing me to the right direction was the answer, that .Read() returns 0 if the end of stream is reached. I modified my code to the following:
public void ReadStream(MemoryStream ContentStream)
{
byte[] buffer = new byte[sizePerHour];
ContentStream.Seek(0, SeekOrigin.Begin); //Added this line
for (int hours = 0; hours < NumberHours; hours++)
{
int t = ContentStream.Read(buffer, 0, sizePerHour);
SecondsToAdd = BitConverter.ToUInt32(buffer, 0);
// further processing of my byte[] buffer
}
}
And everything works like a charm. I initially reset the stream to its origin every time I iterated over hour and giving an offset. Moving the "set to beginning-Part" outside my look and leaving the offset at 0 did the trick.
Read returns zero if the end of the stream is reached. Are you sure, that your memory stream has the content you expect? I´ve tried the following and it works as expected:
// Create the source of the memory stream.
UInt32[] source = {42, 4711};
List<byte> sourceBuffer = new List<byte>();
Array.ForEach(source, v => sourceBuffer.AddRange(BitConverter.GetBytes(v)));
// Read the stream.
using (MemoryStream contentStream = new MemoryStream(sourceBuffer.ToArray()))
{
byte[] buffer = new byte[sizeof (UInt32)];
int t;
do
{
t = contentStream.Read(buffer, 0, buffer.Length);
if (t > 0)
{
UInt32 value = BitConverter.ToUInt32(buffer, 0);
}
} while (t > 0);
}
I once again need your help figuring out this problem of mine...Been already a day and I can't seem to find out why this is happening in my code and output.
Ok.....so basically I am trying to implement the RCON Protocol of Valve in C#, so far I am getting the expected output given the code and sample usage below:
Usage:
RconExec(socket, "cvarlist");
Code:
private string RconExec(Socket sock, string command)
{
if (!sock.Connected) throw new Exception("Not connected");
//sock.DontFragment = true;
sock.ReceiveTimeout = 10000;
sock.SendTimeout = 10000;
//sock.Blocking = true;
Debug.WriteLine("Executing RCON Command: " + command);
byte[] rconCmdPacket = GetRconCmdPacket(command);
sock.Send(rconCmdPacket); //Send the request packet
sock.Send(GetRconCmdPacket("echo END")); //This is the last response to be received from the server to indicate the end of receiving process
RconPacket rconCmdResponsePacket = null;
string data = null;
StringBuilder cmdResponse = new StringBuilder();
RconPacket packet = null;
int totalBytesRead = 0;
do
{
byte[] buffer = new byte[4]; //Allocate buffer for the packet size field
int bytesReceived = sock.Receive(buffer); //Read the first 4 bytes to determine the packet size
int packetSize = BitConverter.ToInt32(buffer, 0); //Get the packet size
//Now proceed with the rest of the data
byte[] responseBuffer = new byte[packetSize];
//Receive more data from server
int bytesRead = sock.Receive(responseBuffer);
//Parse the packet by wrapping under RconPacket class
packet = new RconPacket(responseBuffer);
totalBytesRead += packet.String1.Length;
string response = packet.String1;
cmdResponse.Append(packet.String1);
Debug.WriteLine(response);
Thread.Sleep(50);
} while (!packet.String1.Substring(0,3).Equals("END"));
Debug.WriteLine("DONE..Exited the Loop");
Debug.WriteLine("Bytes Read: " + totalBytesRead + ", Buffer Length: " + cmdResponse.Length);
sock.Disconnect(true);
return "";
}
The Problem:
This is not yet the final code as I am just testing the output in the Debug window. There are a couple of issues occuring if I modify the code to it's actual state.
Removing Thread.Sleep(50)
If I remove Thread.Sleep(50), the output doesn't complete and ends up throwing an exception. I noticed the 'END' termination string is sent by the server pre-maturely. This string was expected to be sent by the server only when the whole list completes.
I tested this numerous times and same thing happens, if I don't remove the line, the list completes and function exits the loop properly.
Removing Debug.WriteLine(response); within the loop and outputting the string using Debug.WriteLine(cmdResponse.ToString()); outside the loop, only partial list data is displayed. If I compare the actual bytes read from the loop with the length of the StringBuilder instance, they're just the same? Click here for the output generated.
Why is this happening given the two scenarios mentioned above?
You are not considering that Socket.Receive very well could read fewer bytes than the length of the supplied buffer. The return value tells you the number of bytes that was actually read. I see that you are properly storing this value in a variable, but I cannot see any code that use it.
You should be prepared to make several calls to Receive to retrieve the entire package. In particular when you receive the package data.
I'm not sure that this is the reason for your problem. But it could be, since a short delay on the client side could be enough to fill the network buffers so that the entire package is read in a single call.
Try using the following code to retrieve package data:
int bufferPos = 0;
while (bufferPos < responseBuffer.Length)
{
bufferPos += socket.Receive(responseBuffer, bufferPos, responseBuffer.Length - bufferPos, SocketFlags.None);
}
Note: You should also support the case when the first call to Receive (the one where you receive the package's data length) doesn't return 4 bytes.