We Keep Coding

Why does this C# TcpClient code not always see a response?

I am using the following code to read a psuedo-HTTP request response. It works sometimes but not always and I do not understand.
Background: I have a device that takes HTTP GET requests and sends a chunked HTTP response. In one case, the response is not a proper chunked HTTP response. It leaves out the null chunk that indicates the end of data. I have fixed that problem in the device, but I am trying to figure out how to read the non-comforming HTTP response. I found code from Create http request using TcpClient that sometimes works and sometimes doesn't and I do not understand why.
If I use the code unaltered, it works fine. If I use it by replacing the "www.bing.com" with my device's IP, "192.1.168.89" in both places the string appears, for example, and change the GET command line to "GET /index.htm HTTP/1.1", it works fine. This version of the command returns a web page that is constructed by the device and sends several TCP buffers (about 1400 bytes in my device) of chunked data.
However, if I change to another command that my device understands, "GET /request.htm?T HTTP/1.1", but returns less than 500 bytes of chunked data, then I never see the response. In fact it never gets past the call to "CopyToAsync(memory)" and I do not understand why. The device sees the request, parses it and sends a proper HTTP response. (I know it is a proper response because I have code that uses HTTPClient to read the response and it sees the response fine. And I see the response data from the device side is exactly the same going out in both cases. I can see the device data because I am writing the device's firmware and can change it to printf() the data being sent out to the TCP routines.)
Anyone have an explanation for why the code below isn't always seeing a response?
private static async Task<string> HttpRequestAsync() {
string result = string.Empty;
using (var tcp = new TcpClient("www.bing.com", 80))
using (var stream = tcp.GetStream())
{
tcp.SendTimeout = 500;
tcp.ReceiveTimeout = 1000;
// Send request headers
var builder = new StringBuilder();
builder.AppendLine("GET /?scope=images&nr=1 HTTP/1.1");
builder.AppendLine("Host: www.bing.com");
//builder.AppendLine("Content-Length: " + data.Length); // only for POST request
builder.AppendLine("Connection: close");
builder.AppendLine();
var header = Encoding.ASCII.GetBytes(builder.ToString());
await stream.WriteAsync(header, 0, header.Length);
// Send payload data if you are POST request
//await stream.WriteAsync(data, 0, data.Length);
// receive data
using (var memory = new MemoryStream())
{
await stream.CopyToAsync(memory);
memory.Position = 0;
var data = memory.ToArray();
var index = BinaryMatch(data, Encoding.ASCII.GetBytes("\r\n\r\n")) + 4;
var headers = Encoding.ASCII.GetString(data, 0, index);
memory.Position = index;
if (headers.IndexOf("Content-Encoding: gzip") > 0)
{
using (GZipStream decompressionStream = new GZipStream(memory, CompressionMode.Decompress))
using (var decompressedMemory = new MemoryStream())
{
decompressionStream.CopyTo(decompressedMemory);
decompressedMemory.Position = 0;
result = Encoding.UTF8.GetString(decompressedMemory.ToArray());
}
}
else
{
result = Encoding.UTF8.GetString(data, index, data.Length - index);
//result = Encoding.GetEncoding("gbk").GetString(data, index, data.Length - index);
}
}
//Debug.WriteLine(result);
return result;
}
}
private static int BinaryMatch(byte[] input, byte[] pattern)
{
int sLen = input.Length - pattern.Length + 1;
for (int i = 0; i < sLen; ++i)
{
bool match = true;
for (int j = 0; j < pattern.Length; ++j)
{
if (input[i + j] != pattern[j])
{
match = false;
break;
}
}
if (match)
{
return i;
}
}
return -1;
}
=====================
Let me edit the function above to show what it is now and maybe clarify things.
static async Task<byte[]> getTcpClientHttpDataRequestAsync(string ipAddress, string request)
{
string result = string.Empty;
List<byte> arrayList = new List<byte>();
using (var tcp = new TcpClient("192.168.1.89", 80))
using (var stream = tcp.GetStream())
using (var memory = new MemoryStream())
{
tcp.SendTimeout = 500;
tcp.ReceiveTimeout = 10000;
tcp.NoDelay = true;
// Send request headers
var builder = new StringBuilder();
builder.AppendLine("GET /request.htm?x01011920000000000001 HTTP/1.1");
builder.AppendLine("Host: 192.168.1.89");
builder.AppendLine("Connection: Close");
builder.AppendLine();
var header = Encoding.ASCII.GetBytes(builder.ToString());
Console.WriteLine("======");
Console.WriteLine(builder.ToString());
Console.WriteLine("======");
await stream.WriteAsync(header, 0, header.Length);
do { } while (stream.DataAvailable == 0);
Console.WriteLine("Data available");
bool done = false;
do
{
int next = stream.ReadByte();
if (next < 0)
{
done = true;
}
else
{
arrayList.Add(Convert.ToByte(next));
}
} while (stream.DataAvailable && !done);
byte[] data = arrayList.ToArray();
return data;
}
}
The GET command is what my device is responding to. If the command starts with 'x' as shown then it responds with a proper HTTP response and the function above reads the data. If it starts with 'd' it is missing the 0 length chunk at the end and the function above never sees any data from the device.
With Wireshark, I am seeing the following responses for the 'x' and 'd' commands.
The 'x' command returns 2 TCP frames with the following data:
0000 1c 6f 65 d3 f0 e2 4c 60 de 41 3f 67 08 00 45 00 .oe...L`.A?g..E.
0010 00 9c 00 47 00 00 64 06 d2 49 c0 a8 01 59 c0 a8 ...G..d..I...Y..
0020 01 22 00 50 05 5d fc f5 9e 72 ad 75 e3 2c 50 18 .".P.]...r.u.,P.
0030 00 01 a9 cd 00 00 48 54 54 50 2f 31 2e 31 20 32 ......HTTP/1.1 2
0040 30 30 20 4f 4b 0d 0a 43 6f 6e 6e 65 63 74 69 6f 00 OK..Connectio
0050 6e 3a 20 63 6c 6f 73 65 0d 0a 43 6f 6e 74 65 6e n: close..Conten
0060 74 2d 54 79 70 65 3a 20 74 65 78 74 2f 68 74 6d t-Type: text/htm
0070 6c 0d 0a 43 61 63 68 65 2d 43 6f 6e 74 72 6f 6c l..Cache-Control
0080 3a 20 6e 6f 2d 63 61 63 68 65 0d 0a 54 72 61 6e : no-cache..Tran
0090 73 66 65 72 2d 45 6e 63 6f 64 69 6e 67 3a 20 63 sfer-Encoding: c
00a0 68 75 6e 6b 65 64 0d 0a 0d 0a hunked....
0000 1c 6f 65 d3 f0 e2 4c 60 de 41 3f 67 08 00 45 00 .oe...L`.A?g..E.
0010 00 45 00 48 00 00 64 06 d2 9f c0 a8 01 59 c0 a8 .E.H..d......Y..
0020 01 22 00 50 05 5d fc f5 9e e6 ad 75 e3 2c 50 18 .".P.].....u.,P.
0030 00 01 fc 20 00 00 30 30 31 0d 0a 2b 0d 0a 30 30 ... ..001..+..00
0040 37 0d 0a 01 85 86 00 00 0d 0a 0d 0a 30 30 30 0d 7...........000.
0050 0a 0d 0a ...
By comparison the 'd' command returns data in 2 TCP frames as:
0000 1c 6f 65 d3 f0 e2 4c 60 de 41 3f 67 08 00 45 00 .oe...L`.A?g..E.
0010 00 9c 00 4e 00 00 64 06 d2 42 c0 a8 01 59 c0 a8 ...N..d..B...Y..
0020 01 22 00 50 05 5e d3 c3 f9 f5 69 cc 6d a3 50 18 .".P.^....i.m.P.
0030 00 01 30 ae 00 00 48 54 54 50 2f 31 2e 31 20 32 ..0...HTTP/1.1 2
0040 30 30 20 4f 4b 0d 0a 43 6f 6e 6e 65 63 74 69 6f 00 OK..Connectio
0050 6e 3a 20 63 6c 6f 73 65 0d 0a 43 6f 6e 74 65 6e n: close..Conten
0060 74 2d 54 79 70 65 3a 20 74 65 78 74 2f 68 74 6d t-Type: text/htm
0070 6c 0d 0a 43 61 63 68 65 2d 43 6f 6e 74 72 6f 6c l..Cache-Control
0080 3a 20 6e 6f 2d 63 61 63 68 65 0d 0a 54 72 61 6e : no-cache..Tran
0090 73 66 65 72 2d 45 6e 63 6f 64 69 6e 67 3a 20 63 sfer-Encoding: c
00a0 68 75 6e 6b 65 64 0d 0a 0d 0a hunked....
0000 1c 6f 65 d3 f0 e2 4c 60 de 41 3f 67 08 00 45 00 .oe...L`.A?g..E.
0010 00 36 00 4f 00 00 64 06 d2 a7 c0 a8 01 59 c0 a8 .6.O..d......Y..
0020 01 22 00 50 05 5e d3 c3 fa 69 69 cc 6d a3 50 18 .".P.^...ii.m.P.
0030 00 01 64 c2 00 00 30 30 37 0d 0a 01 90 91 00 00 ..d...007.......
0040 0d 0a 0d 0a ....
The only discernible differences that I see is that in the second frame of the 'd' command it is missing a 1 byte chunk that is part of our protocol (and shouldn't have any effect on the TCP/HTTP function) and the last 7 bytes of data that the 'x' command provides, which is the 0 length chunk expected for HTTP.
Going back to the code in HttpRequestAsync(), if the 'd' command is sent then the code never sees stream.DataAvailable become true, even though the data has been sent. Why?

await stream.CopyToAsync()
will not complete until
stream.DataAvailable == false
You have indicated to the server, in the headers that you will close the TCP connection when done, but have not done so. The server will eventually close the connection when it thinks you're gone. The server is not obligated to obey your "Connection: close" request and that should be indicated in the headers the server returns.
Before you call stream.CopyToAsync() you should check the headers to determine if what Content-Length has been supplied and pass a buffer length to stream.CopyToAsync() and then call TcpClient.Close()

UTF8 conversion of C# is not same as Java

I have to develop an application that connect to Java server listening over a port, third party gave me the snippet of Java code to send and receive data over the port .
String request = "request";
try (Socket soc = new Socket("localhost", 1234)) {
DataOutputStream output = new DataOutputStream(soc.getOutputStream());
output.writeUTF(request);
output.flush();
DataInputStream input = new DataInputStream(soc.getInputStream());
String response = input.readUTF(); }
I have to write or convert same in C# so as to use by other users and as per company needs. I wrote a code in C# as follows, but it does differently while it encodes the request-
string Request = "request";
byte[] data = new byte[1024];
IPAddress ipAdress = IPAddress.Parse("127.0.0.1");
IPEndPoint ipEndpoint = new IPEndPoint(ipAdress, 1234);
Socket client = new Socket(ipAdress.AddressFamily, SocketType.Stream, ProtocolType.Tcp);
try
{
client.Connect(ipEndpoint);
byte[] sendmsg = Encoding.UTF8.GetBytes(ComRequest);
int n = client.Send(sendmsg);}
Bytes written over port by C# code is different than the Java code.
Is there any equivalent to writeUTF and readUTF function of Java in C#?
Third party also gave me the function of C++ which performs the same operation of string to UTF8 conversion, but I am bad in C++, don't know what c_str() function does?
C++ code-
void convertStringToModifiedUTF8(string content, char* outputBuffer){
strcpy(outputBuffer+2, content.c_str());
outputBuffer[0]= content.length() & 0xFF00;
outputBuffer[1]= content.length() & 0x00FF;
}
Can anyone help in either way to have Java equivalent function or C# code equivalent to above C++ code?
I am highlighting some of the bytes changes-
Java-
0000 02 00 00 00 45 00 00 f8 4c 4e 40 00 80 06 00 00 ....E...LN#..... 0010 7f 00 00 01 7f 00 00 01 c9 c0 16 e3 2f ac ea cf ............/... 0020 8b 6a 2e d1 50 18 01 00 58 71 00 00 00 ce 7b 22 .j..P...Xq....
C# -
0000 02 00 00 00 45 00 00 **fa** 4c **5f** 40 00 80 06 00 00 ....E...L_#..... 00 10 7f 00 00 01 7f 00 00 01 c9 dd 16 e3 81 e5 cd 90 ................ 0020 87 47 fb 46 50 18 01 00 76 64 00 00 7b 20 22 6a .G.FP...vd..

Passing pointer to a struct?

Hello I'm trying to pass data from a pointer to a struct but the values seem to be different.
struct somestruct
{
public file header;
public uint version;
}
unsafe struct file
{
public fixed char name[8];
public uint type;
public uint size;
}
Then in code somewhere..
public unsafe int ReadFile(string filepath)
{
somestruct f = new somestruct();
byte[] fdata = System.IO.ReadAllBytes( filepath );
fixed( byte* src = fdata )
{
f.header = *(file*)src;
MessageBox.Show( new string(f.header.name) ); //should be 'FILENAME' but it's like japanese.
}
return 0;
}
Offset(h) 00 01 02 03 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F
00000000 46 49 4C 45 4E 41 4D 45 00 00 00 01 00 00 00 30 FILENAME.......0
00000010 74 27 9F EF 74 77 F1 D7 C5 86 93 3D 39 0D 72 A9 t'Ÿïtwñ×ņ“=9.r©
00000020 63 8B 92 CF F6 7D 8A 14 45 9D 68 51 A4 8E A4 EE c‹’Ïö}Š.E.hQ¤Ž¤î
00000030 4E FE D0 66 45 0E C9 8D 96 BB F4 EE 52 1F 89 D3 NþÐfE.É.–»ôîR.‰Ó
00000040 5C 80 1A 71 8A 16 B1 8B 3A A8 1B A4 48 11 B8 E8 \€.qŠ.±‹:¨.¤H.¸è
Do you have any idea what's going on?

Each char is 2 bytes - a fixed buffer of 8 chars is 16 bytes. You are reading the first 8 bytes as only the first 4 characters in that buffer, and the high bytes will make it look. Like the eastern Unicode ranges.
I would say: deserialize it at the stream level. Don't do this.
Basically, read (at least) 20 bytes into a buffer, then decode manually, using:
string s = Encoding.ASCII.GetString(buffer, 0, 8);
For the string, and probably shift operations for the unsigned integers.
You could also use unsafe code to read the integers from the buffer, via the other meaning of fixed and a pointer-cast.

A char is UTF-16 and is 2 bytes. You need to convert the UTF-8/ANSI (1 byte) string to a UTF-16 string.

How to GetBytes from string appropriately?

I have a string variable from which I get the following bytes with the following loop:
Bytes I get: 1e 05 55 3c *e2 *91 6f 03 *fe 1a 1d *f4 51 6a 5e 3a *ce *d1 04 *8c
With that loop:
byte[] temp = new byte[source.Length];
string x = "";
for (int i = 0;i != source.Length;i++)
{
temp[i] = ((byte) source[i]);
}
Now I have wanted to simplify that operation and use Encoding's GetBytes.
The problem is I cannot fit an appropriate encoding. e.g. I get several bytes incorrect:
Encoding.ASCII.GetBytes(source): 1e 05 55 3c *3f *3f 6f 03 *3f 1a 1d *3f 51 6a 5e 3a *3f *3f 04 *3f
Encoding.Default.GetBytes(source): 1e 05 55 3c e2 3f 6f 03 3f 1a 1d f4 51 6a 5e 3a ce 4e 04 3f
How can I get rid of that loop and use Encoding's GetBytes?
Here is the summary:
Loop(correct bytes): 1e 05 55 3c *e2 *91 6f 03 *fe 1a 1d *f4 51 6a 5e 3a *ce *d1 04 *8c
Encoding.ASCII.GetBytes(source): 1e 05 55 3c *3f *3f 6f 03 *3f 1a 1d *3f 51 6a 5e 3a *3f *3f 04 *3f
Encoding.Default.GetBytes(source): 1e 05 55 3c e2 3f 6f 03 3f 1a 1d f4 51 6a 5e 3a ce 4e 04 3f
Thanks!
Addition:
I have a string input in hex, sth like: "B1807869C20CC1788018690341"
then I transfer this into string with the method:
private static string hexToString(string sText)
{
int i = 0;
string plain = "";
while (i < sText.Length)
{
plain += Convert.ToChar(Convert.ToInt32(sText.Substring(i, 2), 16));
i += 2;
}
return plain;
}

Your hexToString is transferring byte values (via hex) directly to unicode code-points in the range 0-255. As it happens, that ties into code-page 28591, so if you use:
Encoding enc = Encoding.GetEncoding(28591);
and use that enc, you should get the right data; however, a more important point here is that binary data is not the same as text data, and you should not use a string to hold arbitrary binary.

Presuming that you are trying to "decode" a string literal:
C# stores the strings as Unicode internally.
So you might want to use a encoding that (correctly) supports Unicode
such as:
Encoding.UTF8.GetBytes(source)
Encoding.UnicodeEncoding.GetBytes(source)
Note the caution given for Encoding.Default in MSDN

Windows C# implementation of linux dd command

I'm writing a C#.Net app to run on windows that needs to take an image of a removable disk and chuck it onto a Linux Live USB. The Live USB is the inserted into the target machine and boots, on start up it runs a script which uses the dd command like so to flash it onto another drive:
dd if=/path/to/file/from/csharp/program of=/dev/sdX
The problem I am having is creating the image on the windows side. I have tried my Live Linux out with files I have created on a Linux system using dd and that works fine, but I need to be able to create these files from within a C#.Net application on Windows. I'd rather not have to rely on cygwin or some other dependency so tried to use the Win32 CreateFile function to open the physical device.
CreateFile is called with the first arg set to "\.\F:" (if F: is the drive I want to image), like so:
SafeFileHandle TheDevice = CreateFile(_DevicePath, (uint)FileAccess.Read, (uint)(FileShare.Write | FileShare.Read | FileShare.Delete), IntPtr.Zero, (uint)FileMode.Open, (uint)FILE_ATTRIBUTE_SYSTEM | FILE_FLAG_SEQUENTIAL_SCAN, IntPtr.Zero);
if (TheDevice.IsInvalid)
{
throw new IOException("Unable to access drive. Win32 Error Code " + Marshal.GetLastWin32Error());
}
FileStream Dest = System.IO.File.Open(_SaveFile, FileMode.Create);
FileStream Src = new FileStream(TheDevice, FileAccess.Read);
Src.CopyTo(Dest);
Dest.Flush();
Src.Close();
Dest.Close();
But when the output file is dd'd back onto a disk using the Live Linux USB the result is not as expected (the disk isn't bootable etc, but from examining the output file in a hex editor, it looks like there is an MBR at the beginning etc).
Is this a problem with endianess or should I using something other than a FileStream to copy the data into the file.
Alternatively is there an example of dd for Windows source code (C# or C++, i've looked at the Delphi for http://www.chrysocome.net/dd and don't totally understand it or have a decent Delphi IDE to pick the code apart) so I can see how that works?
UPDATE/EDIT:
Here is a hex string of the first 512 Bytes that the dd output contains:
33 C0 FA 8E D8 8E D0 BC 00 7C 89 E6 06 57 8E C0 FB FC BF 00 06 B9 00 01 F3 A5 EA 1F 06
00 00 52 52 B4 41 BB AA 55 31 C9 30 F6 F9 CD 13 72 13 81 FB 55 AA 75 0D D1 E9 73 09 66
C7 06 8D 06 B4 42 EB 15 5A B4 08 CD 13 83 E1 3F 51 0F B6 C6 40 F7 E1 52 50 66 31 C0 66
99 E8 66 00 E8 21 01 4D 69 73 73 69 6E 67 20 6F 70 65 72 61 74 69 6E 67 20 73 79 73 74
65 6D 2E 0D 0A 66 60 66 31 D2 BB 00 7C 66 52 66 50 06 53 6A 01 6A 10 89 E6 66 F7 36 F4
7B C0 E4 06 88 E1 88 C5 92 F6 36 F8 7B 88 C6 08 E1 41 B8 01 02 8A 16 FA 7B CD 13 8D 64
10 66 61 C3 E8 C4 FF BE BE 7D BF BE 07 B9 20 00 F3 A5 C3 66 60 89 E5 BB BE 07 B9 04 00
31 C0 53 51 F6 07 80 74 03 40 89 DE 83 C3 10 E2 F3 48 74 5B 79 39 59 5B 8A 47 04 3C 0F
74 06 24 7F 3C 05 75 22 66 8B 47 08 66 8B 56 14 66 01 D0 66 21 D2 75 03 66 89 C2 E8 AC
FF 72 03 E8 B6 FF 66 8B 46 1C E8 A0 FF 83 C3 10 E2 CC 66 61 C3 E8 62 00 4D 75 6C 74 69
70 6C 65 20 61 63 74 69 76 65 20 70 61 72 74 69 74 69 6F 6E 73 2E 0D 0A 66 8B 44 08 66
03 46 1C 66 89 44 08 E8 30 FF 72 13 81 3E FE 7D 55 AA 0F 85 06 FF BC FA 7B 5A 5F 07 FA
FF E4 E8 1E 00 4F 70 65 72 61 74 69 6E 67 20 73 79 73 74 65 6D 20 6C 6F 61 64 20 65 72
72 6F 72 2E 0D 0A 5E AC B4 0E 8A 3E 62 04 B3 07 CD 10 3C 0A 75 F1 CD 18 F4 EB FD 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 19 16 9F 29 00 00 80 01 01 00 06 FE 3F 0E 3F 00 00 00 61 C8 03 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 55 AA
and here is what my code produces:
EB 76 90 4D 53 44 4F 53 35 2E 30 00 02 04 04 00 02 00 02 00 00 F8 F2 00 3F 00 FF 00 3F
00 00 00 61 C8 03 00 80 00 29 7A E8 21 04 4E 4F 20 4E 41 4D 45 20 20 20 20 46 41 54 31
36 20 20 20 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 E9 05 01 B4 0E 53 33 DB CD 10 5B C3 8A 07 3C 00 74 06 E8 EE FF 43 EB F4 C3
0D 4E 6F 20 42 50 42 3A 20 43 61 6E 27 74 20 62 6F 6F 74 20 75 73 69 6E 67 20 43 48 53
20 66 75 6E 63 74 69 6F 6E 73 00 50 B0 2E E8 BC FF 58 33 DB 8E 06 E4 01 F6 06 DC 01 02
75 42 F6 06 DC 01 04 75 07 80 3E E8 01 80 72 34 53 53 52 50 06 53 55 6A 10 8B F4 52 50
8A 16 E8 01 B8 00 42 F9 CD 13 8A EC 58 5A 8D 64 10 72 14 80 FD 00 75 0F 03 C5 83 D2 00
C3 BB 91 00 E8 78 FF F4 EB FD 83 3E 18 00 00 74 F0 52 50 8B CD F7 36 18 00 8B F2 03 D1
3B 16 18 00 76 06 8B 0E 18 00 2B CE 33 D2 F7 36 1A 00 88 16 E9 01 8B F8 8B D7 51 8A C1
8D 4C 01 C0 E6 06 0A CE 8A EA 8B 16 E8 01 B4 02 CD 13 59 73 15 80 FC 09 75 0A 49 EB DE
8A C4 04 30 E8 18 FF B4 00 CD 13 EB D1 58 5A 03 C1 83 D2 00 2B E9 74 07 C1 E1 09 03 D9
EB 94 C3 00 00 00 00 FA FC E8 00 00 5E 81 EE 85 01 2E 8B 84 E4 01 8E D8 8E C0 8E D0 2E
C7 84 7C 01 AF 01 2E 89 84 7E 01 B9 00 01 BF 00 00 F3 2E A5 2E FF AC 7C FF BC 00 0A FB
80 3E E8 01 FF 75 04 88 16 E8 01 83 06 E4 01 20 A1 E0 01 8B 16 E2 01 BD 02 00 E8 E9 FE
50 52 EB 74 90 00 00 00 00 00 00 00 00 00 00 00 D3 20 00 00 00 30 80 00 FF 00 68 41 00
40 09 FF 40 5A AC 04 00 00 AC 04 00 00 00 00 12 00 55 AA
This was taken from exactly the same CF card without any editing/writing etc happening, so i'm confused as to why they are so different, but both end with the correct 55 AA bytes too. Does Windows mangle the MBR's on cards when they're accessed this way or is some other weird under the hood stuff happening that I'm not aware of?

I think what you have should work - I've tried this myself using a bootable floppy disk image (mounted as a virtual drive using ImDisk) and the resulting file is binary identical to the original image.
For completeness here is the code I used (in its entirity):
using System;
using System.IO;
using System.Runtime.InteropServices;
using Microsoft.Win32.SafeHandles;
namespace ConsoleApplication1
{
public class Program
{
const int FILE_ATTRIBUTE_SYSTEM = 0x4;
const int FILE_FLAG_SEQUENTIAL_SCAN = 0x8;
[DllImport("Kernel32.dll", SetLastError = true, CharSet = CharSet.Auto)]
public static extern SafeFileHandle CreateFile(string fileName, [MarshalAs(UnmanagedType.U4)] FileAccess fileAccess, [MarshalAs(UnmanagedType.U4)] FileShare fileShare, IntPtr securityAttributes, [MarshalAs(UnmanagedType.U4)] FileMode creationDisposition, int flags, IntPtr template);
[STAThread]
static void Main()
{
using (SafeFileHandle device = CreateFile(#"\\.\E:", FileAccess.Read, FileShare.Write | FileShare.Read | FileShare.Delete, IntPtr.Zero, FileMode.Open, FILE_ATTRIBUTE_SYSTEM | FILE_FLAG_SEQUENTIAL_SCAN, IntPtr.Zero))
{
if (device.IsInvalid)
{
throw new IOException("Unable to access drive. Win32 Error Code " + Marshal.GetLastWin32Error());
}
using (FileStream dest = File.Open("TempFile.bin", FileMode.Create))
{
using (FileStream src = new FileStream(device, FileAccess.Read))
{
src.CopyTo(dest);
}
}
}
}
}
}
If this doesn't work then it seems to indicate that:
There is a problem with the original image.
The problem is with whatever is using the disk image that you've just written.
There is some subtle differences in dealing with the specific device you are accessing (although I can't think what)
The most likely culprit is step 2. What exactly is it that you are doing with the resulting disk image?
Update: This is written in the comments, but for completeness I thought I'd add it to my answer - it looks like whats happening is that the contents of the first partition of the disk is being written, when instead what is wanted is the contents of the entire disk.
When you take a look at the second hex string (the one produced by sample code) in something like HxD we see this:
ëv.MSDOS5.0..........øò.?.ÿ.?...aÈ..€.)zè!.NO NAME FAT16 ..
........................................................é..´.S3Û
Í.[Ê.<.t.èîÿCëôÃ.No BPB: Can't boot using CHS functions.P°.è¼ÿX
3ÛŽ.ä.ö.Ü..uBö.Ü..u.€>è.€r4SSRP.SUj.‹ôRPŠ.è.¸.BùÍ.ŠìXZ.d.r.€ý.u.
.ŃÒ.û‘.èxÿôëýƒ>...tðRP‹Í÷6..‹ò.Ñ;...v.‹...+Î3Ò÷6..ˆ.é.‹ø‹×QŠÁ.
L.Àæ..Ίê‹.è.´.Í.Ys.€ü.u.IëÞŠÄ.0è.ÿ´.Í.ëÑXZ.ÁƒÒ.+ét.Áá..Ùë”Ã....
úüè..^.î…..‹„ä.ŽØŽÀŽÐ.Ç„|.¯..‰„~.¹..¿..ó.¥.ÿ¬|ÿ¼..û€>è.ÿu.ˆ.è.ƒ.
ä. ¡à.‹.â.½..èéþPRët............Ó ...0€.ÿ.hA.#.ÿ#Z¬...¬.......Uª
This looks to me like the boot sector of a FAT16 partition - the presence of the strings "MSDOS5.0", "NO NAME" and "FAT16" near the start is a dead giveaway.
Compare this to the output of the first hex string (the one produced by dd):
3ÀúŽØŽÐ¼.|‰æ.WŽÀûü¿..¹..ó¥ê....RR´A»ªU1É0öùÍ.r..ûUªu.Ñés.fÇ...´B
ë.Z´.Í.ƒá?Q.¶Æ#÷áRPf1Àf™èf.è!.Missing operating system...f`f1Ò».
|fRfP.Sj.j.‰æf÷6ô{Àä.ˆáˆÅ’ö6ø{ˆÆ.áA¸..Š.ú{Í..d.faÃèÄÿ¾¾}¿¾.¹ .ó¥
Ãf`‰å»¾.¹..1ÀSQö.€t.#‰ÞƒÃ.âóHt[y9Y[ŠG.<.t.$.<.u"f‹G.f‹V.f.Ðf!Òu.
f‰Âè¬ÿr.è¶ÿf‹F.è ÿƒÃ.âÌfaÃèb.Multiple active partitions...f‹D.f.
F.f‰D.è0ÿr..>þ}Uª.….ÿ¼ú{Z_.úÿäè..Operating system load error...^
¬´.Š>b.³.Í.<.uñÍ.ôëý......................................Ÿ)..€.
...þ?.?...aÈ..................................................Uª
And we see something that looks to me a lot like a master boot record. Why? Because in the MBR all of the first 440 bytes is boot code, unlike a FAT boot sector which contains the distinctive bios parameter block (it looks like garbage above, but if you put that through a disassembler you get something that looks like valid 16 bit code).
Also, both of those look like valid and completely different boot sectors (complete with error messages). There is no way that a programming error could have "mangled" one to look like the other - it must just be that the wrong thing is being read.
In order to get CreateFile to return the disk instead of the partition it looks like you just need to pass it a different string, for example #"\\.\PhysicalDrive0" opens the first physical disk.
See:
Low Level Disk Access
INFO: Direct Drive Access Under Win32

This is what i've written to do get the \.\PhysicalDriveX path for a given drive letter. If Pass the drive letter into this and take the return value and pass into CreateFile as the first Param I should now get something similar to dd under Linux.
using System.Management; //Add in a reference to this as well in the project settings
public static string GetPhysicalDevicePath(char DriveLetter)
{
ManagementClass devs = new ManagementClass( #"Win32_Diskdrive");
{
ManagementObjectCollection moc = devs.GetInstances();
foreach(ManagementObject mo in moc)
{
foreach (ManagementObject b in mo.GetRelated("Win32_DiskPartition"))
{
foreach (ManagementBaseObject c in b.GetRelated("Win32_LogicalDisk"))
{
string DevName = string.Format("{0}", c["Name"]);
if (DevName[0] == DriveLetter)
return string.Format("{0}", mo["DeviceId"]);
}
}
}
}
return "";
}