We Keep Coding

Strange behavior form C# serial port

I'm getting strange behavior from simple program, which I'm using to learn how to use the serial port. The form have just one serial port control and one TextBox. Because it's just a test program I have disabled the thread call check.
I forgot to mention,that I'm using micro-controller to send 1000 bytes of data (read EEPROM).
The strange thing is that, when I read the data and just append it directly to the text box in the DataReceived event, everything is fine, but when I first pass the values to int[] array,and then use a loop to convert them to strings in HEX format and append them to the TextBox, there are some zeros, between the values.
Some code with results.
Case 1: read data and directly append to TextBox
private void sp1_DataReceived(object sender, SerialDataReceivedEventArgs e)
{
while (sp1.BytesToRead > 0)
{
textBox1.AppendText(sp1.ReadByte().ToString("X")+ " ");
}
}
And the result is (well,part of it,as i said there are 1000 bytes to receive...)
0 1 2 3 4 5 6 7 8 9 A B C D E F 10 11 12 13 14 15 16 17 18 19 1A 1B 1C 1D 1E 1F 20 21 22 23 24 25 26 27 28 29 2A 2B 2C 2D 2E 2F 30 31 32 33 34 35 36 37 38 39 3A 3B 3C 3D 3E 3F 40 41 42 43 44 45 46 47 48 49 4A 4B 4C 4D 4E 4F 50 51 52 53 54 55 56 57 58 59 5A 5B 5C 5D 5E 5F 60 61 62 63 64 65 66 67 68 69 6A 6B 6C 6D 6E 6F 70 71 72 73 74 75 76 77 78 79 7A 7B 7C 7D 7E 7F 80 81 82 83 84 85 86 87 88 89 8A 8B 8C 8D 8E 8F 90 91 92 93 94 95 96 97 98 99 9A 9B 9C..... and so on
Case 2: first store the values to int array,and than convert them to string, and append to TextBox
private void sp1_DataReceived(object sender, SerialDataReceivedEventArgs e)
{
int[] buffer = new int[1000];
int i = 0;
while (sp1.BytesToRead > 0)
{
//textBox1.AppendText(sp1.ReadByte().ToString("X")+ " ");
buffer[i] = sp1.ReadByte();
i++;
}
int j = 0;
while (j < 1000)
{
textBox1.AppendText(buffer[j].ToString("X"));
j++;
}
I get a lot of 0's at random places, and it reads 4-5 times more data, than the 1000 in the loop
0123456789ABCDEF101112131415161718191A1B1C1D1E1F00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000202122232425262728292A2B2C2D2E2F303132333435363738393A3B3C3D3E3F404142434445464748494A4B4C4D4E4F505152535455565758595A5B5C5D5E5F606162636465666768696A6B6C6D6E6F707172737475767778797A7B7C7D7E7F808182838485868788898A8B8C8D8E8F909192939495969798999A9B9C9D9E9FA0A1A2A3A4A5A6A7A8A9AAABACADAEAFB0B1B2B3B4B5B6B7B8B9BABBBCBDBEBFC0C1C2C3C4C5C6C7C8C9CACBCCCDCECFD0D1D2D3D4D5D6D7D8D9DADBDCDDDEDFE0E1E2E3E4E5E6E7E8E9EAEBECEDEEEFF0F1F2F3F4F5F6F7F8F9FAFBFCFDFEFF0123456789ABCDEF101112131415161718191A1B1C1D1E1F202122232425262728292A2B2C2D2E2F303132333435363738393A3B3C3D3E3F404142434445464748494A4B4C4D4E4F505152535455565758595A5B5C5D5E5F606162636465666768696A6B6C6D6E6F707172737475767778797A7B7C7D7E7F808182838485868788898A8B8C8D8E8F909192939495969798999A9B9C9D9E9FA0A1A2A3A4A5A6A7A8A9AAABACADAEAFB0B1B2B3B4B5B6B7B8B9BABBBCBDBEBFC0C1C2C3C4C5C6C7C8C9CACBCCCDCECFD0D1D2D3D4D5D6D7D8D9DADBDCDDDEDFE0E1E2E3E4E5E6E7E8E9EAEBECEDEEEFF0F1F2F3F4F5F6F7F8F9FAFBFCFDFEFF0123456789ABCDEF101112131415161718191A1B1C1D1E1F202122232425262728292A2B2C2D2E2F303132333435363738393A3B3C3D3E3F404142434445464748494A4B4C4D4E4F505152535455565758595A5B5C5D5E5F606162636465666768696A6B6C6D6E6F707172737475767778797A7B7C7D7E7F808182838485868788898A8B8C8D8E8F909192939495969798999A9B9C9D9E9FA0A1A2A3A4A5A6A7A8A9AAABACADAEAFB0B1B2B3B4B5B6B7B8B9BABBBCBDBEBFC0C1C2C3C4C5C6C7C8C9CACBCCCDCECFD0D1D2D3D4D5D6D7D8D9DADBDCDDDEDFE0E1E2E3E4E5E6E7E8E9EAEBECEDEEEFF0F1F2F3F4F5F6F7F8F9FAFBFCFDFEFF0123456789ABCDEF101112131415161718191A1B1C1D1E1F202122232425262728292A2B2C2D2E2F303132333435363738393A3B3C3D3E3F404142434445464748494A4B4C4D4E4F505152535455565758595A5B5C5D5E5F606162636465666768696A6B6C6D6E6F707172737475767778797A7B7C7D7E7F808182838485868788898A8B8C8D8E8F909192939495969798999A9B9C9D9E9FA0A1A2A3A4A5A6A7A8A9AAABACADAEAFB0B1B2B3B4B5B6B7B8B9BABBBCBDBEBFC0C1C2C3C4C5C6C7C8C9CACBCCCDCECFD0D1D2D3D4D5D6D7D8D9DADBDCDDDEDFE0E1E2E3E4E5E6E7000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
What might be the reason for this strange behavior?
Thanks in advance

sp1_DataReceived is typically not called only once. Typically your computer handles received data faster than received. This means at some point the loop
while (sp1.BytesToRead > 0)
is left before all 1000 bytes are received. Just a short time later sp1_DataReceived is already called again because more of the 1000 bytes are now available. Since your first implementation only appends the bytes it doesen't matter. But your second implementation differs because you are always appending 1000 characters to your text. This could lead to a result of a multiple of 1000 characters with zeros appended.
To fix your problem you need to combine the bytes of multiple events. One solution could be to use a list like
private List<byte> buffer = new List<byte>();
private void sp1_DataReceived(object sender, SerialDataReceivedEventArgs e)
{
while (sp1.BytesToRead > 0)
{
buffer.Add(sp1.ReadByte());
}
//Print if all bytes are available
if (buffer.Count >= 1000)
{
//Join the bytes to a string using LINQ
textBox1.Text = String.Join("", buffer.Select(b => b.ToString("X")));
buffer.Clear();
}
}
or an array like
private byte[] buffer = new byte[1000];
private int bufferIndex = 0;
private void sp1_DataReceived(object sender, SerialDataReceivedEventArgs e)
{
while (sp1.BytesToRead > 0 && bufferIndex < 1000)
{
buffer[bufferIndex ] = sp1.ReadByte();
bufferIndex ++;
}
//Print if all bytes are available
if (bufferIndex >= 1000)
{
//Join the bytes to a string using LINQ
textBox1.Text = String.Join("", buffer.Select(b => b.ToString("X")));
bufferIndex = 0;
}
}
Note that this are only some ideas and example implementations. Since I do not know if you are also receiving other messages at the port it is not possible to give a perfect suitable solution to solve your problem.

Why does DateTime.Now sets the highest bit in DateTime's usual binary representation?

As far as I know, the binary representation of DateTime and TimeSpan structures are 8-byte numbers of ticks (1 millisecond = 10000 ticks according to TimeSpan.TicksPerSecond). And values of Days, Hours, Minutes, etc. properties are obtained by integer division on TicksPerDay, TicksPerHour, TicksPerSecond etc. constants of TimeSpan.
For example if You run this code:
TimeSpan s1 = new TimeSpan(3, 5, 7, 9, 11).AddTicks(13));
long t1 = s1.Ticks;
You can get (if you use Visual Studio) something like that in you Memory windows:
0x061BE4D0 3d 2a c9 67 86 02 00 00
0x061BE4E0 3d 2a c9 67 86 02 00 00
where 0x061BE4D0 and 0x061BE4E0 are addresses of s1 and t1 respectively.
(Actually You should write 's1' and '&t1' instead of just 't1' in Address area of Memory window)
Now if You run another snippet of code:
DateTime d1 = new DateTime(1, 1, 3, 5, 7, 9, 11).AddTicks(13);
long t1 = d1.Ticks;
DateTime d2 = DateTime.Now;
long t2 = d2.Ticks;
You'll see for 'd1', '&t1', 'd2', '&t2' respectively the data like shown:
0x061AE438 3d 6a 5f 3d bd 01 00 00
0x061AE430 3d 6a 5f 3d bd 01 00 00
0x061AE424 bd 71 d5 02 3f 9d d0 88
0x061AE41C bd 71 d5 02 3f 9d d0 08
Why does DateTime.Now set the highest bit (0x 80 00 00 00 00 00 00 00) in its binary representation?

If you look at the source code here (not sure exactly which version this is, but you get the idea):
http://www.dotnetframework.org/default.aspx/DotNET/DotNET/8#0/untmp/whidbey/REDBITS/ndp/clr/src/BCL/System/DateTime#cs/1/DateTime#cs
you can see that high bits are applied depending on whether the time is local or not.
From a quick glance over the code, there's a const member called:
private const UInt64 KindLocal = 0x8000000000000000;
which looks as if it's used in the conversion. I'd suspect this is happening because you're using DateTime.Now, which is a local 'Now'. If you used 'UtcNow' it probably would set a different bit.
private const UInt64 KindUtc = 0x4000000000000000;
However, when you get it as 'ticks', it probably returns the unspecified form, which has no top bit set.
private const UInt64 KindUnspecified = 0x0000000000000000;
Basically, you're getting hung up on the inner workings of the struct. If you really want to understand it, then I'd suggest digging through the code. Otherwise, just use it as per the instructions, and it'll work fine for you!

Reverse Engineering Serial Commands

tl;dr;
What I need to be able to do is reverse engineer serial commands so that I can figure out how either the human readable values or the binary values are being serialized into raw serial commands.
IE:
if
66 = 'A1' or '0110 1'
6e = 'A2' or '0110 2'
e6 = 'B1' or '1110 1'
ee = 'B2' or '1110 2'
then
what is A3 or B3, etc.
Im working on a wrapper for an open source automation project.
There is a way to send raw commands, and in theory string multiple commands together.
I've sniffed a few serial commands, and this is what they look like.
[init] [HouseCode | DeviceCode] [ready] [HouseCode | FunctionCode]
04 66 06 62 // A1 ON
04 6e 06 62 // A2 ON
04 62 06 62 // A3 ON
04 6a 06 62 // A4 ON
04 61 06 62 // A5 ON
04 69 06 62 // A6 ON
04 65 06 62 // A7 ON
04 6d 06 62 // A8 ON
04 67 06 62 // A9 ON
04 6f 06 62 // A10 ON
04 63 06 62 // A11 ON
04 6b 06 62 // A12 ON
04 60 06 62 // A13 ON
04 68 06 62 // A14 ON
04 64 06 62 // A15 ON
04 6c 06 62 // A16 ON
04 e6 06 e2 // B1 ON
04 ee 06 e2 // B2 ON
04 e2 06 e2 // B3 ON
04 ea 06 e2 // B4 ON
....
04 ec 06 e2 // B16 ON
04 66 06 63 // A1 Off
04 e6 06 e3 // B1 Off
04 66 06 61 // All A lights On (using A1 as the starting point)
04 e6 06 e1 // All B lights On (using B1 as the starting point)
04 66 06 66 // All A lights Off (using A1 as the starting point)
04 e6 06 66 // All B lights Off (using A1 as the starting point)
04 66 06 64 2a // A1 Dim 20
04 66 06 64 2c // A1 Dim 21
04 66 06 64 2e // A1 Dim 22
04 66 06 65 2a // A1 Bright 20
04 66 06 65 69 // A1 Bright 50
What I need to be able to do is reverse engineer this so that I can make the serial calls programmatically, or better yet, figure out how either the human readable values or the binary values are being serialized into raw serial commands.
Yes, I could sniff ALL the commands and store each value separately, but I'd like to know how this has been done.
Here's my current observation.
The calls are broken up into two.
04 is initiated and tells the device what to look for
** tells the system which device is being controlled [HouseCode & DeviceCode]
hex 55 is returned to tell you it's ready
06 is initiated and tells the device what to expect
** tells the system the house code and command [HouseCode & FunctionCode]
** is optionally sent and is a value between 0 & 100 to reference a dim level
hex 55 is sent back again to tell you it's ready
The second pair uses the first character as the alphabetic code (HouseCode = A, B, C, etc) and the second character is the address (DeviceCode = 1, 2, 3, etc)
with this information, my personal guess is that...
6 must directly correspond to A
e must directly correspond to B
The forth pair starts with the same HouseCode as the second pair
The forth pair ends with the FunctionCode
1 = all on
2 = on
3 = off
4 = dim
5 = bright
6 = all off
etc..
The fifth pair only shows on a bright/dim command and represents the number between 0 and 100
Lastly, in the docs, each of the commands relate to Binary data, so it's probably not a matter of converting A1 to hex, but rather the binary to hex.
HouseCode DeviceCode Binary Value
A 1 0110
B 2 1110
C 3 0010
D 4 1010
E 5 0001
F 6 1001
G 7 0101
H 8 1101
I 9 0111
J 10 1111
K 11 0011
L 12 1011
M 13 0000
N 14 1000
O 15 0100
P 16 1100
FunctionCode Binary Value
All Units Off 0000
All Lights On 0001
On 0010
Off 0011
Dim 0100
Bright 0101
All Lights Off 0110
Extended Code 0111
Hail Request 1000
Hail Acknowledge 1001
Pre-set Dim (1) 1010
Pre-set Dim (2) 1011
Extended Data Transfer 1100
Status On 1101
Status Off 1110
Status Request 1111
Does anyone know how I might go about achieving this?

Heyu is a fantastic open source application for working with X10 devices. They have published a very comprehensive X10 protocol specification based on X10's original doc.
This should take the guess work out of your work. What is significant is that the housecode and unitcode are static maps and can not be computed. The protocol spec specifies exactly how the bitstream is formed. E.g.
PC Interface Description
0x04,0x66 Address A1
0x6a Checksum ((0x04 + 0x66)&0xff)
0x00 OK for transmission.
0x55 Interface ready.
0x86,0x64 Function: A Dim 16/22*100%
0xea Checksum ((0x86 + 0x64)&0xff)
0x00 OK for transmission.
0x55 Interface ready.

Encoding/Decoding hex packet

I want to send this hex packet:
00 38 60 dc 00 00 04 33 30 3c 00 00 00 20 63 62
39 62 33 61 36 37 34 64 31 36 66 32 31 39 30 64
30 34 30 63 30 39 32 66 34 66 38 38 32 62 00 06
35 2e 31 33 2e 31 00 00 02 3c
so i build the string:
string packet = "003860dc0000" + textbox1.text+ "00000020" + textbox2.text+ "0006" + textbox3.text;
then "convert" it to ascii:
conn_str = HexString2Ascii(packet);
then i send the packet... but i have this:
00 38 60 **c3 9c** 00 00 04 33 30 3c 00 00 00 20 63
62 39 62 33 61 36 37 34 64 31 36 66 32 31 39 30
64 30 34 30 63 30 39 32 66 34 66 38 38 32 62 00
06 35 2e 31 33 2e 31 00 00 02 3c **0a**
why??
Thank you!
P.S.
the function is:
private string HexString2Ascii(string hexString)
{
byte[] tmp;
int j = 0;
int lenght;
lenght=hexString.Length-2;
tmp = new byte[(hexString.Length)/2];
for (int i = 0; i <= lenght; i += 2)
{
tmp[j] =(byte)Convert.ToChar(Int32.Parse(hexString.Substring(i, 2), System.Globalization.NumberStyles.HexNumber));
j++;
}
return Encoding.GetEncoding(1252).GetString(tmp);
}
EDIT:
if i convert directly in byte, the hex packet in coded as string:
00000000 30 30 33 38 36 30 64 63 30 30 30 30 30 34 33 33 003860dc 00000433
00000010 33 30 33 43 30 30 30 30 30 30 32 30 33 34 33 32 303C0000 00203432
00000020 36 33 36 33 33 35 33 39 33 32 33 34 36 36 33 39 63633539 32346639
00000030 36 33 33 39 33 31 33 39 33 30 33 36 33 33 36 35 63393139 30363365
00000040 33 35 36 33 36 35 36 35 36 35 33 31 33 39 33 38 35636565 65313938
00000050 36 33 33 31 36 34 33 34 36 33 33 30 30 30 30 36 63316434 63300006
00000060 33 35 32 65 33 31 33 33 32 65 33 31 30 30 30 30 352e3133 2e310000
00000070 30 32 33 43 023C

You cannot convert raw binary data to string data and expect things to just work. They are not the same. This is especially true when you mix up your character encodings.
C# characters are not ASCII characters. They are Unicode characters, represented by Unicode code points. When you then turn around and write those characters out, you need to specify what kind of data to write out. When you read your byte array into a string, using Encoding.GetEncoding(1252), you are getting the characters corresponding to code page 1252, in which 0xdc is a Ü.
But when your string is being converted back into bytes to send over the network, it is being written out as UTF-8. In UTF-8, UTF-00DC cannot be encoded as a single byte, since that byte value is used to indicate the start of a multi-byte sequence. Instead, it's encoded as the multi-byte sequence 0xc3 0x9c. As far as C# is concerned, those two values are the same character. (I don't know where that extra 0x0a is coming from, but my guess is an errant line feed from one of your text boxes and/or some other part of your process).
Its not clear what exactly you're trying to do, but I suspect you are converting way too many times for it to work out correctly. If you know the byte sequence you want to send, why not just encode that as a byte[] directly? For example, use a MemoryStream and write the constant bytes you need into it.
To get the values out of your text boxes, your original code to "convert" the string of hex digits into a string of ASCII characters had the right idea. You just need to stop at the point where you have a byte array, since ultimately the byte array is what you want.
public byte[] GetBytesFrom(string hex)
{
var length = hex.Length / 2;
var result = new byte[length];
for (var i = 0; i < length; i++)
{
result[i] = byte.Parse(hex.Substring(i, 2), NumberStyles.HexNumber);
}
return result;
}
// Variable portions of packet structure.
var byte[] segment2 = GetBytesFrom(textbox1.Text);
var byte[] segment4 = GetBytesFrom(textbox2.Text);
var byte[] segment6 = GetBytesFrom(textbox3.Text);
MemoryStream output = new MemoryStream();
output.Write(new[] { 0x00, 0x38, 0x60, 0xdc, 0x00, 0x00 }, 0, 6);
output.Write(segment2, 0, segment2.Length);
output.Write(new[] { 0x00, 0x00, 0x00, 0x20 }, 0, 4);
output.Write(segment4, 0, segment4.Length);
output.Write(new[] { 0x00, 0x06 }, 0, 2);
output.Write(segment6, 0, segment6.Length);
From here, you could use MemoryStream.CopyTo() to copy it to another stream, or MemoryStream.Read() to read the entire packet into a new byte array, or MemoryStream.GetBuffer() to get the underlying buffer (though that last one is rarely what you want -- it includes unused padding bytes)

Reading a ASN.1 DER-encoded RSA Public key

I'm writing an app to get a better understanding of DKIM. The spec says I retrieve a "ASN.1 DER-encoded" public key from the domain TXT record. I can seen the key on "s1024._domainkey.yahoo.com" = "MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDrEee0Ri4Juz+QfiWYui/E9UGSXau/2P8LjnTD8V4Unn+2FAZVGE3kL23bzeoULYv4PeleB3gfm".
How can I use this key from .net? The examples I've seen get the key from a X509Certificate2, or an XML file containing the RSAParameters.
CORRECTION: I copy/pasted the key above from the network-tools.com DNS tool, which must've cut it short. nslookup gives me the full key:
s1024._domainkey.yahoo.com text =
"k=rsa; t=y; p=MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDrEee0Ri4Juz+QfiWYui/E9UGSXau2P8LjnTD8V4Unn+2FAZVGE3kL23bzeoULYv4PeleB3gfm"
"JiDJOKU3Ns5L4KJAUUHjFwDebt0NP+sBK0VKeTATL2Yr/S3bTxhy+1xtj4RkdV7fVxTn56Lb4udUnwuxK4V5b5PdOKj+XcwIDAQAB; n=A 1024 bit key;"
So abelenky was on the right track with BASE64..

This is the base64-encoding of the DER-encoding of an ASN.1 PublicKeyInfo containing an RSA public key.
Here is a translation:
0 30 159: SEQUENCE {
3 30 13: SEQUENCE {
5 06 9: OBJECT IDENTIFIER '1 2 840 113549 1 1 1'
16 05 0: NULL
: }
18 03 141: BIT STRING 0 unused bits, encapsulates {
22 30 137: SEQUENCE {
25 02 129: INTEGER
: 00 EB 11 E7 B4 46 2E 09 BB 3F 90 7E 25 98 BA 2F
: C4 F5 41 92 5D AB BF D8 FF 0B 8E 74 C3 F1 5E 14
: 9E 7F B6 14 06 55 18 4D E4 2F 6D DB CD EA 14 2D
: 8B F8 3D E9 5E 07 78 1F 98 98 83 24 E2 94 DC DB
: 39 2F 82 89 01 45 07 8C 5C 03 79 BB 74 34 FF AC
: 04 AD 15 29 E4 C0 4C BD 98 AF F4 B7 6D 3F F1 87
: 2F B5 C6 D8 F8 46 47 55 ED F5 71 4E 7E 7A 2D BE
: 2E 75 49 F0 BB 12 B8 57 96 F9 3D D3 8A 8F FF 97
: 73
157 02 3: INTEGER 65537
: }
: }
: }
The OBJECT IDENTIFIER indicates that the following BIT STRING contains the encoding of an RSAPublicKey. The INTEGERs are the modulus and the public exponent.
You can decode the base64 with Convert.FromBase64String, but I don't think .NET has built-in functionality for parsing PublicKeyInfos, so you need to use a 3rd party tool like BouncyCastle.

For anyone interested in this matter I would suggest the System.Security.Cryptography.X509Certificates.PublicKey which can be used to read a DER encoded public key.

That string looks like its some sort of base-64 encoding.
If you convert that string from base-64 to a BLOB, it should then be in valid ASN.1 format.

Try the bouncycastle library, it provides great functionality for such cases.