I am receiving data from a cnc machine every 5 seconds. Length of the data is 66 bytes. And every two byte has a special meaning according to the guide that I have. The device sends the data over socket to a specific ip and port. I have been told that I should read the data as hex instead of ascii.
This line of code returns
string data = Encoding.ASCII.GetString(data.buffer,0,66);
this;
"\0\u0004\0\u0001\0\0\0\0\0\0\0\0\0\0\0\0\0\r\0\r\0\0\0\0\0\0:a\u0002#\0?\0`\u001b?\u0015U\0\0\0\0\u0001\u0010\0\u0018\0\0\u000f\a\0\0\0\0\0\0\0\0\0\0\0\0\0\0u/"
and of course it is not useful to me.
I did tried to convert byte array to the hex string with that code;
StringBuilder sb = new StringBuilder();
foreach (byte b in buffer)
sb.Append(b.ToString("X2"));
string hexString = sb.ToString();
And got result as
00040001000000000000000000020000000000000000000000003A9D023F00A000601B841555000000000110001800000F070000000000000000000000000000752F
And when I try to convert this result as string, no success, nothing meaningfull.
GOAL
What I am trying to achieve is, read the incoming socket data as hex and use every two byte as a word to match a value. For example first 2 byte should match either 0 or 1. With i have it returns ? (question mark)
Thank you.
I have been told that I should read the data as hex instead of ascii
My gut feeling is this statement has been misquoted or misunderstood. There is no value in processing binary data as string hex representation just as there is no value in converting it to ascii... The only sane way to process binary data, is in binary unless you have a meaningful way to convert it.
You mention you need word (2byte) groupings, you could just convert this to an array of short, or ushort depending on your needs
var bytes = new byte[66];
var shortArray = new short[bytes.Length / 2];
Buffer.BlockCopy(bytes, 0, shortArray, 0, bytes.Length);
or
for (int i = 0; i < shortArray.Length; i++)
shortArray[i] = BitConverter.ToInt16(bytes[(i*2)..(i*2+2)]);
Disclaimer : This is just an example, be very careful of the endianess of your data, there are other ways to do this
I want to read from a text file a hex number, using the last digit to define my length to read a string, again a number and so on until the line will finish.
using (StreamReader sr = new StreamReader(fileName)){
String line = sr.ReadLine();
string hexText = line.Substring(0,9);
char c = hexText[8];
int con = c - '0'; //saving the value
string myHex = con.ToString("X");
int length = Convert.ToInt32(myHex, 16);
string fieldChars = line.Substring(0, length); //getting the key
string b = line.Substring(c, length); }
so for "5A3F00004olga" the length is correct and 4 (the last hex bit) but for some reason b is not Olga.Why?
Let's take a closer look at your code. You have:
char c = hexText[8];
int con = c - '0'; //saving the value
string myHex = con.ToString("X");
int length = Convert.ToInt32(myHex, 16);
string fieldChars = line.Substring(0, length); //getting the key
string b = line.Substring(c, length);
So c contains the value of the character at position 8. I don't have any idea why you're subtracting '0' from it, and then converting the result back to a string. You could just as easily write:
string myHex = c + "X";
Also, if the value at hexText[8] were 'A', then subtracting '0' would give you 17 rather than the 10 that you expected.
I also don't know what you expect the line that assigns fieldChars to do, but I can pretty much guarantee that it's not doing what you want.
The reason b doesn't contain "olga" is because the substring starting position in this case would be 4 (44 in decimal), and length could be totally wrong as described above.
I think the code you want is:
char c = hexText[8];
string myHex = con.ToString("X");
int length = Convert.ToInt32(myHex, 16);
string b = line.Substring(9, length);
I didn't include fieldChars there because it's unclear to me what you want value you want that variable to hold. If you want the key, then you'd want line.Substring(0, 8); If you want the entire field, including the key, the length, and the text, you'd write line.Substring(0, 9 + length);
As suggested in comments, you can use the C# debugger to walk through your code one line at a time, and inspect variables to see what is happening. That's an excellent way to see the results of the code you write. Better yet is to work a few examples by hand before you start writing code. That way you get a better understanding of what the code is supposed to do. It makes writing correct code a whole lot easier.
If you don't know how to use the debugger, now is the perfect time to learn.
Another way you can see what's happening is to put Console.WriteLine statements after each line. For example:
char c = hexText[8];
Console.WriteLine("c = " + c);
If you do that after each line, or after each important group of lines, displaying the values of important variables, you can see exactly where your program is going off into the weeds.
I am using FormsConnect an App for the iPad to export signatures to XML.
The output from the App is as follows:
0x1828023f020000001d1100000018000000e02f01000000000000000000000000a888223f010000005259414ca0a37e00000000000000000000000000000000000000000000000000000000000000000010b9611100000000000000000000000000000000308e660e00000000c021070030a46c0e00000000000000000000000044b0fb3e00beb0000100000001000000107b6111000030420000584200000000000000000000000000000000000080bf00000000000000000100000000000000000000000000000000000000000000001cb0fb3e0100000000000000000000000000000000000000000000000047864aa0860100a086010000c05411000070110047864a0000000000000000010000000000000000000000000000000000000018884d3f801200010900000101000000000000000000020074a54e3fb0b6690ec0b6690e00000000000000000000000044b0fb3e0038a9000100000001000000f0e4c008000030420000304200000000000000000000000000000000000080bf00000000000000000100000000000000000000000000000000000000000000001cb0fb3e0100000000000000000000000000000000000000000000000047864aa0860100a08601000020761100407c110047864a0000000000000000010000000000000000000000000000000000000018884d3f801200010900000101000000000000000000020074a54e3f20b6690e30b6690e000000000000000000000000a888223f010000005259414c808b730c00000000000000000000000000000000000000000000000000000000000000000cc0fb3e0000000000000000000000000400000000000000000000000000000000000000000000000000000000000000a888223f010000005259414ce06ebe0a00000000000000000000000000000000000000000000000000000000000000009ceefb3ea08c61110000000000000000000000000000000000000000000000000000000000000000280001c10400000000000000b017640000000000000000000000000000000000000000000000000000000000000020415846fc3ed08c6111000000000000000000000000000000000000000000000000000000000000003f000000004014630000887a0670c89b0c40146300000000000000000000000000000000000000000000000000100000800000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000e44200000000000000000000000000000000408d6111f08e61110000584200003042000000000000000000000000a888223f010000005259414c4040050d00000000000000000000000000000000000000000000000000000000000000002cd7fb3e108d61110000000000000000a08e6111d033120000000000000000000000000000000000200000c10000000000000000000000000000000000000000a888223f010000005259414c304d050d0000000000000000000000000000000000000000000000000000000000000000ccc8fb3eb08d611100000000d08c611100000000908b6111000000000000000000000000000000000000000000000000000000000150000000000000f08d611101000000000000000000e03f0000204100000000000000000000000000000000108e611104000000000000000000000084854d3f01000000110000000300000002000000e08d61110000000000000000f0c7fb3e908b61112952d53200000000d08d6111000000000000000000000000c08a4d3f8011000101000000000000000000000000000100c8c94e3f0000000094c9fb3e0100000000000000608e611100000000000000000000000000000000000000000000000000003442000000000000e83f000000606666d63f808e6111408d6111000000000000000000000000c08a4d3f8011000101000000000000000000000000000100c8c94e3f0000000084854d3f0000000001000000030000000100000000000000000000000000000084854d3f01000000110000000300000002000000c08e61110000000000000000408d6111000000000000000000000000d088223fe003fe07020000000800000002000000000000000000000000000000b4effb3e308f6111000000000000000000000000b0906111000000000000000000000000f0496300200000c100000000608f6111000080bf0200000000000000a888223f010000005259414cf04a050d000000000000000000000000000000000000000000000000000000000000000054a0fb3ed08f6111000000000000000000000000d0331200000000000000000000000000f0496300330000c100000000000000000000000020887a06201c630000000000e0a762001032d805000000000000803f0000000000000000010000000000c841000000000416000000000000a888223f010000005259414c9047050d0000000000000000000000000000000000000000000000000000000000000000d088223fc00bfe070100000008000000010000000000000000000000000000003ca9fb3e00000000b7b6363f9796163ffffefe3e0000803f304277060000000084854d3f01000000110000000300000007000000609061110000000000000000c0b96f0e000000000000000000000000a8894d3f800501012412010050596100000000000000000000000000040000009f9e9e3ea9a8a83eb3b2b23e0000803f00c66d0e00000000000000000000000084854d3f01000000110000000300000002000000c06e6d0e00000000000000009ceefb3ee09161110000000000000000000000000000000000000000000000000000000000000000280001c10400000000000000b017640000000000000000000000000000000000000000000000000000000000000020415846fc3e10926111000000000000000000000000000000000000000000000000000000000000003f000000004014630020629b0c10629a0c40146300000000000000000000000000000000000000000000000000100000800000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000e4420000000000000000000000000000000080926111309461110000584200003042000000000000000000000000a888223f010000005259414c1043050d00000000000000000000000000000000000000000000000000000000000000002cd7fb3e509261110000000000000000e0936111d033120000000000000000000000000000000000200000c10000000000000000000000000000000000000000a888223f010000005259414c804b050d0000000000000000000000000000000000000000000000000000000000000000ccc8fb3ef0926111000000001092611100000000d09061110000000000000000000000000000000000000000000000000000000001500000000000003093611101000000000000000000e03f00002041000000000000000000000000000000005093611104000000000000000000000084854d3f01000000110000000300000002000000209361110000000000000000f0c7fb3ed09061112952d5320000000010936111000000000000000000000000c08a4d3f8011000101000000000000000000000000000100c8c94e3f0000000094c9fb3e0100000000000000a093611100000000000000000000000000000000000000000000000000003442000000000000e83f000000606666d63fc093611180926111000000000000000000000000c08a4d3f8011000101000000000000000000000000000100c8c94e3f0000000084854d3f0000000001000000030000000100000000000000000000000000000084854d3f0100000011000000030000000200000000946111000000000000000080926111000000000000000000000000d088223f6007fe07020000000800000002000000000000000000000000000000b4effb3e7094611100000000000000000000000030976111000000000000000000000000f0496300200000c100000000a0946111000080bf0200000000000000a888223f010000005259414c0047050d000000000000000000000000000000000000000000000000000000000000000054a0fb3e10956111000000000000000000000000d0331200000000000000000000000000f0496300330000c100000000000000000000000040629b0c201c630000000000e0a762001032d805000000000000803f0000000000000000010000000000c841000000000416000000000000a888223f010000005259414c8042050d0000000000000000000000000000000000000000000000000000000000000000d088223fc00ffe07010000000800000001000000000000000000000000000000907c6211000000000000000000000000a0946111000000000000000000000000a8894d3f8006010200000000504f6c0e00100000000000000000000000000000000000000000000045f41836397925360000000000000000000000005854554d000000006028000000000000000000000000000000000000d09561110000000000000000d4956111000000000010ea0d0000000000000000000000000000000000000000000000000000000000000000a8894d3f8005010132e100005059610000000000000000000000000004000000e3e2e23ef1f0f03e8180003f0000803fa888223f010000005259414c5045050d0000000000000000000000000000000000000000000000000000000000000000a888223f010000005259414cc044050d000000000000000000000000000000000000000000000000000000000000000054a0fb3e202fc508000000000000000000000000d0331200000000000000000000000000f0496300330000c10000000000000000000000008054990c201c630000000000e0a762001032d805000000000000803f0000000000000000010000000000c841000000000416000000000000a0966111000000000000000000000000087c070000000000000000000000000084854d3f010000001100000003000000020000007095611100000000000000009ceefb3e409661110000000000000000000000000000000000000000000000000000000000000000280001c10400000000000000b017640000000000000000000000000000000000000000000000000000000000000020415846fc3e60986111000000000000000000000000000000000000000000000000000000000000003f0000000040146300402a930c20a5990c40146300000000000000000000000000000000000000000000000000100000800000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000e44200000000000000000000000000000000a0986111509a611100005842000030420000000000000000000000002cd7fb3e709661110000000000000000009a6111d033120000000000000000000000000000000000200000c10000000000000000000000000000000000000000ccc8fb3e10996111000000006098611100000000509761110000000000000000000000000000000000000000000000000000000001500000000000005099611101000000000000000000e03f00002041000000000000000000000000000000007099611104000000000000000000000084854d3f01000000110000000300000002000000409961110000000000000000f0c7fb3e509761112952d5320000000030996111000000000000000000000000c08a4d3f8011000101000000000000000000000000000100c8c94e3f0000000094c9fb3e0100000000000000c099611100000000000000000000000000000000000000000000000000003442000000000000e83f000000606666d63fe0996111a0986111000000000000000000000000c08a4d3f8011000101000000000000000000000000000100c8c94e3f0000000084854d3f00000000010000000300000001000000000000000000000000
Now I'm trying to convert this Hex string back into an image using this code:
Bitmap newImage = new Bitmap(new MemoryStream(StringToByteArray(hex)));
public static byte[] StringToByteArray(string hex)
{
hex = hex.Substring(2, hex.Length - 2);
if (hex.Length % 2 != 0)
{
throw new ArgumentException(String.Format(CultureInfo.InvariantCulture, "The binary key cannot have an odd number of digits: {0}", hex));
}
byte[] HexAsBytes = new byte[hex.Length / 2];
for (int index = 0; index < HexAsBytes.Length; index++)
{
string byteValue = hex.Substring(index * 2, 2);
HexAsBytes[index] = byte.Parse(byteValue, NumberStyles.HexNumber, CultureInfo.InvariantCulture);
}
return HexAsBytes;
}
But I keep getting Parameter is not valid, when attempting to Load the stream into a Bitmap. I posed the question to the developer of the App, and here is his response:
Dear Alex
The data is stored in binary, hex format. Here are the instructions on how to read the binary data and write it back out as a jpg.
byteData[i] = (digit [i * 2] shl 4) + (digit [i * 2 + 1])
The above assumes that the digits are already converted to numbers from 0 to 15. If you’re trying to optimize your script by writing to memory in larger chunks (integers, for instance) you need to take endianness into account. All major CPU architectures are little-endian which means the numbers are stored with their bytes arranged in the opposite order from their actual value (ie, 0x1234 is stored as 0x3412 in memory).
Below are two different scripts for extracting blob data from the form:
1. Python Script:
iterate over each photo attached to image field in FormConnect xml file
data = R[i]
to strip '0x' off front of each blob
a = data[2:]
iterate over each byte, write as binary int to array
for i in range(len(a)/2):
b = (int(bin(int(a[i*2],16)),2)<<4) + int(bin(int(a[i*2+1],16)),2)
data.append(b)
write array to binary file
F = file(outFile,'wb')
data.tofile(F)
F.close()
PHP Script:
$pic = "";
$field = substr($field,2);
header("Content-Type: image/jpeg");
$pic = pack("H*", $field);
echo $pic;
If anyone can help me on this, I would be very grateful.
Kind regards,
Alex
Your problem isn't the hex conversion routine, it's that the original data isn't encoded properly. Based on all the zeroes within it, I'd say it's just a raw bitmap! My guess is that the author of the program that generated that data forgot to encode the bitmap into a specific file format before outputting it.
My ASP.NET page has following query string parameter:
…?IDs=1000000012,1000000021,1000000013,1000000022&...
Here IDs parameter will always have numbers separated by something, in this case ,. Currently there are 4 numbers but normally they would be in between 3 and 7.
Now, I am looking for method to convert each big number from above into smallest possible value; specifically compressing value of IDs query string parameter. Both, compressing each number algorithm or compressing whole value of IDs query string parameter are welcome.
Encode or decode is not an issue; just compressing the value IDs query string parameter.
Creating some unique small value for IDs and then retrieving its value from some data source is out of scope.
Is there an algorithm to compress such big numbers to small values or to compress value of the IDs query string parameter all together?
You basically need so much room for your numbers because you are using base 10 to represent them. An improvement would be to use base 16 (hex). So for example, you could represent 255 (3 digits) as ff (2 digits).
You can take that concept further by using a much larger number base... the set of all characters that are valid query string parameters:
A-Z, a-z, 0-9, '.', '-', '~', '_', '+'
That gives you a base of 67 characters to work with (see Wikipedia on QueryString).
Have a look at this SO post for approaches to converting base 10 to arbitrary number bases.
EDIT:
In the linked SO post, look at this part:
string xx = IntToString(42,
new char[] { '0','1','2','3','4','5','6','7','8','9',
'A','B','C','D','E','F','G','H','I','J','K','L','M','N','O','P','Q','R','S','T','U','V','W','X','Y','Z',
'a','b','c','d','e','f','g','h','i','j','k','l','m','n','o','p','q','r','s','t','u','v','w','x'});
That's almost what you need. Just expand it by adding the few characters it is missing:
yz.-~_+
That post is missing a method to go back to base 10. I'm not going to write it :-) but the procedure is like this:
Define a counter I'll call TOTAL.
Look at the right most character and find it's position in the array.
TOTAL = (the position of the character in the array)
Example: Input is BA1. TOTAL is now 1 (since "1" is in position 1 in the array)
Now look at the next character left of the first one and find it's position in the array.
TOTAL += 47 * (the position of the character in the array)
Example: Input is BA1. TOTAL is now (47 * 11) + 1 = 518
Now look at the next character left of the previous one and find it's position in the array.
TOTAL += 47 * 47 * (the position of the character in the array)
Example: Input is BA1. Total is now (47 * 47 * 10) + (47 * 11) + 1 = 243508
And so on.
I suggest you write a unit test that converts a bunch of base 10 numbers into base 47 and then back again to make sure your conversion code works properly.
Note how you represented a 6 digit base 10 number in just 3 digits of base 47 :-)
What is the range of your numbers? Assuming they can fit in a 16-bit integer, I would:
Store all your numbers as 16-bit integers (2 bytes per number, range -32,768 to 32,767)
Build a bytestream of 16-bit integers (XDR might be a good option here; at very least, make sure to handle endianness correctly)
Base64 encode the bytestream, using the modified base64 encoding for URLs (net is about 3 characters per number)
As an added bonus you don't need comma characters anymore because you know each number is 2 bytes.
Alternatively, if that isn't good enough, I'd use zlib to compress your stream of integers and then base64 the zlib-compressed stream. You can also switch to 32-bit integers if 16-bit isn't a large enough range (i.e. if you really need numbers in the 1,000,000,000 range).
Edit:
Maybe too late, but here's an implementation that might do what you need:
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
namespace Scratch {
class Program {
static void Main(string[] args) {
//var ids = new[] { 1000000012, 1000000021, 1000000013, 1000000022 };
var rand = new Random();
var ids = new int[rand.Next(20)];
for(var i = 0; i < ids.Length; i++) {
ids[i] = rand.Next();
}
WriteIds(ids);
var s = IdsToString(ids);
Console.WriteLine("\nResult string is: {0}", s);
var newIds = StringToIds(s);
WriteIds(newIds);
Console.ReadLine();
}
public static void WriteIds(ICollection<Int32> ids) {
Console.Write("\nIDs: ");
bool comma = false;
foreach(var id in ids) {
if(comma) {
Console.Write(",");
} else {
comma = true;
}
Console.Write(id);
}
Console.WriteLine();
}
public static string IdsToString(ICollection<Int32> ids) {
var allbytes = new List<byte>();
foreach(var id in ids) {
var bytes = BitConverter.GetBytes(id);
allbytes.AddRange(bytes);
}
var str = Convert.ToBase64String(allbytes.ToArray(), Base64FormattingOptions.None);
return str.Replace('+', '-').Replace('/', '_').Replace('=', '.');
}
public static ICollection<Int32> StringToIds(string idstring) {
var result = new List<Int32>();
var str = idstring.Replace('-', '+').Replace('_', '/').Replace('.', '=');
var bytes = Convert.FromBase64String(str);
for(var i = 0; i < bytes.Length; i += 4) {
var id = BitConverter.ToInt32(bytes, i);
result.Add(id);
}
return result;
}
}
}
Here's another really simple scheme that should give good compression for a set of numbers of the form N + delta where N is a large constant.
public int[] compress(int[] input) {
int[] res = input.clone();
Arrays.sort(res);
for (int i = 1; i < res.length; i++) {
res[i] = res[i] - res[i - 1];
}
return res;
}
This should reduce the set {1000000012,1000000021,1000000013,1000000022} to the list [1000000012,1,9,1], which you can then compress further by representing the numbers in base47 encoding as described in another answer.
Using simple decimal encoding, this goes from 44 characters to 16 characters; i.e. 63%. (And using base47 will give even more compression).
If it is unacceptable to sort the ids, you don't get quite as good compression. For this example, {1000000012,1000000021,1000000013,1000000022} compresses to the list [1000000012,9,-8,9]. That is just one character longer for this example
Either way, this is better than a generic compression algorithm or encoding schemes ... FOR THIS KIND OF INPUT.
If the only issue is the URL length, you can convert numbers to base64 characters, then convert them back to numbers at the server side
how patterned are the IDs you are getting? if digit by digit, the IDs are random, then the method I am about to propose won't be very efficient. but if the IDs you gave as an example are representative of the types you'd be getting, then perhaps the following could work?
i motivate this idea by example.
you have for example, 1000000012 as ID that you'd like to compress. why not store it as [{1},{0,7},{12}]? This would mean that the first digit is a 1 followed by 7 zeros followed by a 12. Thus if we use the notation {x} that would represent one instance of x, while if we use {x,y} that would mean that x occurs y times in a row.
you could extend this with a little bit of pattern matching and/or function fitting.
for example, pattern matching: 1000100032 would be [{1000,2}{32}].
for example, function fitting:
if your IDs are 10 digits, then split the ID into two 5 digit numbers and store the equation of the line that goes through both points. if ID = 1000000012, the you have y1 = 10000 and y2 = 12. therefore, your slope is -9988 and your intercept is 10000 (assuming x1 = 0, x2 = 1). In this case, it's not an improvement, but if the numbers were more random, it could be. Equivalently, you could store the sequence of IDs with piecewise linear functions.
in any case, this mostly depends on the structure of your IDs.
I assume you are doing this as a workaround for request URL length restrictions ...
Other answers have suggested encoding the decimal id numbers in hex, base47 or base64, but you can (in theory) do a lot better than that by using LZW (or similar) to compress the id list. Depending on how much redundancy there is in your ID lists, you could get significantly more than 40% reduction, even after re-encoding the compressed bytes as text.
In a nut-shell, I suggest that you find an off-the-shelf text compression library implemented in Javascript and use it client side to compress the ID list. Then encode the compressed bytestring using base47/base64, and pass the encoded string as the URL parameter. On the server side do the reverse; i.e. decode followed by decompress.
EDIT: As an experiment, I created a list of 36 different identifiers like the ones you supplied and compressed it using gzip. The original file is 396 bytes, the compressed file is 101 bytes, and the compressed + base64 file 138 bytes. That is a 65% reduction overall. And the compression ratio could actually improve for larger files. However, when I tried this with a small input set (e.g. just the 4 original identifiers), I got no compression, and after encoding the size was larger than the original.
Google "lzw library javascript"
In theory, there might be simpler solution. Send the parameters as "post data" rather than in the request URL, and get the browser to apply the compression using one of the encodings that it understands. That will give you more savings too since there is no need to encode the compressed data into legal URL characters.
The problem is getting the browser to compress the request ... and doing that in a browser independent way.