We Keep Coding

Read socket data Hex instead of Ascii

I am receiving data from a cnc machine every 5 seconds. Length of the data is 66 bytes. And every two byte has a special meaning according to the guide that I have. The device sends the data over socket to a specific ip and port. I have been told that I should read the data as hex instead of ascii.
This line of code returns
string data = Encoding.ASCII.GetString(data.buffer,0,66);
this;
"\0\u0004\0\u0001\0\0\0\0\0\0\0\0\0\0\0\0\0\r\0\r\0\0\0\0\0\0:a\u0002#\0?\0`\u001b?\u0015U\0\0\0\0\u0001\u0010\0\u0018\0\0\u000f\a\0\0\0\0\0\0\0\0\0\0\0\0\0\0u/"
and of course it is not useful to me.
I did tried to convert byte array to the hex string with that code;
StringBuilder sb = new StringBuilder();
foreach (byte b in buffer)
sb.Append(b.ToString("X2"));
string hexString = sb.ToString();
And got result as
00040001000000000000000000020000000000000000000000003A9D023F00A000601B841555000000000110001800000F070000000000000000000000000000752F
And when I try to convert this result as string, no success, nothing meaningfull.
GOAL
What I am trying to achieve is, read the incoming socket data as hex and use every two byte as a word to match a value. For example first 2 byte should match either 0 or 1. With i have it returns ? (question mark)
Thank you.

I have been told that I should read the data as hex instead of ascii
My gut feeling is this statement has been misquoted or misunderstood. There is no value in processing binary data as string hex representation just as there is no value in converting it to ascii... The only sane way to process binary data, is in binary unless you have a meaningful way to convert it.
You mention you need word (2byte) groupings, you could just convert this to an array of short, or ushort depending on your needs
var bytes = new byte[66];
var shortArray = new short[bytes.Length / 2];
Buffer.BlockCopy(bytes, 0, shortArray, 0, bytes.Length);
or
for (int i = 0; i < shortArray.Length; i++)
shortArray[i] = BitConverter.ToInt16(bytes[(i*2)..(i*2+2)]);
Disclaimer : This is just an example, be very careful of the endianess of your data, there are other ways to do this

How can i encode a string to UTF-8 to a pre existing byte array?

I have a simple bit of code that converts a C# string by encoding it to UTF-8 then creating a byte array from it. But i am wondering how can i encode to UTF-8 using a byte array i have already made at a starting index?
So this is how i am currently encoding and getting the resulting byte array:
byte[] result = Encoding.UTF8.GetBytes(myString);
But I have a byte array premade that i would prefer to write to at a specific index if that makes sense. Is there any built in method to do this, if not how would i go about it ?

GetBytes has another overload that writes to existing array:
byte[] bytes = new byte[1000]; // sample, make sure it has enough space
var specificIndex = 0;
var actualByteCount = Encoding.UTF8.GetBytes(
myString, 0, myString.Length, bytes, specificIndex);
Don't forget to handle result to know how many bytes in the array actually represent string (actualByteCount)
Note you may need to use GetByteCount to get correct array size or adjust number of characters to convert to fit into your buffer.

First you will need to convert your bytes into Base64String then convert that into bytes. Likes this:
byte[] random = new byte[] { 0x00C9, 0x00C9, 0x00C9 };
byte[] encodedBytes = Encoding.UTF8.GetBytes(Convert.ToBase64String(random));

Byte to ASCII conversion Generating Unneeded/Undesired Spacing Characters

I have a code that relies heavily on bytes for its speed while not writing to a file.
In either the read function or the datapoint conversion function, I am getting several unnecessary "space" characters from the byte array after converting it to an ASCII string, even after setting them to NULL. This generates a lot of undesired whitespace. Here's parts of the current code:
//Within Read Function
var charBuf = Enumerable.Repeat<byte>(0, 1024).ToArray(); //Set byte array to null
int ret = Read(ConnectionID, charBuf, 1024); //Call to a custom dll to retrieve data
if (0 <= ret)
{
return charBuf;
}
//Datapoint message is set as an empty byte that gets added to the list Datapoint
//The following converts the datapoint to a string depending on its input
var message = Encoding.ASCII.GetString(dataPoint.Message);
if (String.IsNullOrEmpty(message))
{
message = "ReadError";
}
Is there any way to eliminate these supposed characters without too much code or is there an error in my conversion? Either fix would be appreciated.

To prevent modifying the read function of the code too much further than changing the datatype in order to optimize the speed of the code, I decided to take care of the empty space issue by having the process of string simplification done once the read function was no longer in use, and all the data was being written to a file:
var message = Encoding.Default.GetString(dataPoint.Message);
int messageSize=0;
byte nullByte = 0x00;
for (int k=0; k < dataPoint.Message.Count(); k++)
{
if (dataPoint.Message.ElementAt(k).Equals(nullByte))
{
messageSize = k+1;
break;
}
else
{
continue;
}
}
message = message.Substring(0, messageSize);
message would then be appended to a text file line by line for each message per line.
This method ensures that despite being defined as 1024 null spaces, only data received by the read function will be accounted for (where the data being received sends no spaces).

Split a result from File.ReadAllBytes

My code here reads all the bytes of a image and stores it in the byte array. Is there a way to convert these bytes into ascii then split them up to 512-char(ascii char) long pieces? Like when you try splitting a string based on the length, you can do that. Can you do something similar to splitting this into 512 lengths? This is to send to the server.
byte[] imagesize;
imagesize = File.ReadAllBytes(#"C:\image.jpeg");
Console.Write(imagesize[1].ToString());
What I really want is to convert these bytes into plain ASCII format (Which in C# would be Encoding.ASCII), then split that long ASCII line from converting all the bytes into 512-char(?) long chunks into a byte array (byte[] chunks). So when I send the packets I can do
for(i=0; i<AmountOfChunks; i++)
{
Packet.payload = chunks[i];
//The "chunks" is the ASCII formated array.
}
If someone knows how to do this, it would greatly help. Thanks, if there's anything more, i'll try to explain it in more detail if i can.
If this is wrong, because i know a byte is 8-bit each. Then somehow to be able to do it, maybe the bytes into a list?

Not clear why you needs this, but you might be looking for Convert.ToBase64String() to get a string representation. For chunking you can just walk over the resulting string and split at the appropriate indexes:
byte[] imagesize = File.ReadAllBytes(#"C:\image.jpeg");
string base64String = Convert.ToBase64String(imagesize);
List<string> chunks = new List<string>();
for (int i = 0; i < base64String.Length; i+=512)
{
chunks.Add(base64String.Substring(i, Math.Min(512, base64String.Length - i)));
}

Try to make this
int i=0;
do
{
sendBytes = imagesize.Skip(512*i).Take(512).ToArray();
//Your function of send
i++;
}
while(imagesize.Count()-512*i>0)

Large file handling - Read algorithm breaks - C#

So I've got an algorithm that reads from a (very large, ~155+ MB) binary file, parses it according to a spec and writes out the necessary info (to a CSV, flat text). It works flawlessly for the first 15.5 million lines of output, which produces a CSV file of ~0.99-1.03 GB. This gets through hardly over 20% of the binary file. After this it breaks, as in suddenly the printed data is not at all what is shown in the binary file. I checked the binary file, the same pattern continues (data split up into "packets" - see code below). Due to how it's handled, mem usage never really increases (steady ~15K). The functional code is listed below. Is it my algorithm (if so, why would it break after 15.5 million lines?!)... are there other implications I'm not considering due to the large file sizes? Any ideas?
(fyi: each "packet" is 77 bytes in length, beginning with a 3byte "startcode" and ending with a 5byte "endcode" - you'll see the pattern below)
edit code has been updated based on the suggestions below... thanks!
private void readBin(string theFile)
{
List<int> il = new List<int>();
bool readyForProcessing = false;
byte[] packet = new byte[77];
try
{
FileStream fs_bin = new FileStream(theFile, FileMode.Open);
BinaryReader br = new BinaryReader(fs_bin);
while (br.BaseStream.Position < br.BaseStream.Length && working)
{
// Find the first startcode
while (!readyForProcessing)
{
// If last byte of endcode adjacent to first byte of startcod...
// This never occurs outside of ending/starting so it's safe
if (br.ReadByte() == 0x0a && br.PeekChar() == (char)0x16)
readyForProcessing = true;
}
// Read a full packet of 77 bytes
br.Read(packet, 0, packet.Length);
// Unnecessary I guess now, but ensures packet begins
// with startcode and ends with endcode
if (packet.Take(3).SequenceEqual(STARTCODE) &&
packet.Skip(packet.Length - ENDCODE.Length).SequenceEqual(ENDCODE))
{
il.Add(BitConverter.ToUInt16(packet, 3)); //il.ElementAt(0) == 2byte id
il.Add(BitConverter.ToUInt16(packet, 5)); //il.ElementAt(1) == 2byte semistable
il.Add(packet[7]); //il.ElementAt(2) == 1byte constant
for(int i = 8; i < 72; i += 2) //start at 8th byte, get 64 bytes
il.Add(BitConverter.ToUInt16(packet, i));
for (int i = 3; i < 35; i++)
{
sw.WriteLine(il.ElementAt(0) + "," + il.ElementAt(1) +
"," + il.ElementAt(2) + "," + il.ElementAt(i));
}
il.Clear();
}
else
{
// Handle "bad" packets
}
} // while
fs_bin.Flush();
br.Close();
fs_bin.Close();
}
catch (Exception e)
{
MessageBox.Show(e.ToString());
}
}

Your code is silently catching any exception that happens in the while loop and swallowing it.
This is a bad practice because it masks issues like the one you are running into.
Most likely, one of the methods you call inside the loop (int.Parse() for example) is throwing an exception because it encounters some problem in the format of the data (or your assumptions about that format).
Once an exception occurs, the loop that reads data is thrown off kilter because it is no longer positioned at a record boundary.
You should do several things to make this code more resilient:
Don't silently swallow exception in the run loop. Deal with them.
Don't read data byte by byte or field by field in the loop. Since your records are fixed size (77 bytes) - read an entire record into a byte[] and then process it from there. This will help ensure you are always reading at a record boundary.

Don't put an empty generic catch block here and just silently catch and continue. You should check and see if you're getting an actual exception in there and go from there.
There is no need for the byteToHexString function. Just use the 0x prefix before a hexadecimal number and it will do a binary comparison.
i.e.
if(al[0] == 0x16 && al[1] == 0x3C && al[2] == 0x02)
{
...
}
I don't know what your doConvert function does (you didn't provide that source), but the BinaryReader class provides many different functions, one of which is ReadInt16. Unless your shorts are stored in an encoded format, that should be easier to use than doing your fairly obfuscated and confusing conversion. Even if they're encoded, it would still be far simpler to read the bytes in and manipulate them, rather than doing several roundtrips with converting to strings.
Edit
You appear to be making very liberal use of the LINQ extension methods (particularly ElementAt). Every time you call that function, it enumerates your list until it reaches that number. You'll have much better performing code (as well as less verbose) if you just use the built-in indexer on the list.
i.e. al[3] rather than al.ElementAt(3).
Also, you don't need to call Flush on an input Stream. Flush is used to tell the stream to write anything that it has in its write buffer to the underlying OS file handle. For an input stream it won't do anything.
I would suggest replacing your current sw.WriteLine call with this:
sw.WriteLine(BitConverter.ToString(packet)); and see if the data you're expecting on the row where it starts to mess up is actually what you're getting.
I would actually do this:
if (packet.Take(3).SequenceEqual(STARTCODE) &&
packet.Skip(packet.Length - ENDCODE.Length).SequenceEqual(ENDCODE))
{
ushort id = BitConverter.ToUInt16(packet, 3);
ushort semistable = BitConverter.ToUInt16(packet, 5);
byte contant = packet[7];
for(int i = 8; i < 72; i += 2)
{
il.Add(BitConverter.ToUInt16(packet, i));
}
foreach(ushort element in il)
{
sw.WriteLine(string.Format("{0},{1},{2},{3}", id, semistable, constant, element);
}
il.Clear();
}
else
{
//handle "bad" packets
}