We Keep Coding

C# Encoding.CreateTranscodingStream does not output file with desired target encoding

I have the following piece of code:
using (Stream inputFileStream = File.OpenRead("C:\\Users\\User\\Downloads\\test.txt"))
{
using (Stream transcodingStream = Encoding.CreateTranscodingStream(inputFileStream, Encoding.GetEncoding(500), new UnicodeEncoding(bigEndian: true, byteOrderMark: true)))
{
using (Stream outputStream = File.OpenWrite("C:\\Users\\User\\Downloads\\test.txt"))
{
await transcodingStream.CopyToAsync(outputStream, cancellationToken);
}
}
}
My file before transcoding has the following first 16 bytes and is of Ebcdic type encoding (code page 500):
F5 F1 F1 F0 F2 C2 D4 E6 40 40 40 40 40 40 F1 F1 = 51102BMW 11
After performing the transcoding to Unicode with Big-Endiant and Byte Order Markings, I expect the file to begin with:
FF FE
However, I get:
00 35 00 31 00 31 00 30 00 32 00 42 00 4D 00 57 = �5�1�1�0�2�B�M�W
Where am I going wrong with this?

It seems like the transcoding stream does not care about maintaining the BOM for target encoding and it's something you have to manage yourself.
I've implemented the following solution:
targetEncoding is of type Encoding
outputStream.Seek(0, SeekOrigin.Begin);
await outputStream.WriteAsync(targetEncoding.Preamble.ToArray(), 0, targetEncoding.Preamble.Length, cancellationToken);

What pixel format to use with Byte array to bitmap

I have been trying to get a simple byte array in to a Picture box using c#. Most examples I tried give me parameter invalid. If I don't use a using or converter and break it out like so (see code below) I do get an image but it looks wrong (all black with some random colored dots up top).
Bitmap bmp = new Bitmap(48, 32, PixelFormat.Format24bppRgb);
BitmapData bmpData = bmp.LockBits(
new Rectangle(0, 0, bmp.Width, bmp.Height),
ImageLockMode.WriteOnly, bmp.PixelFormat);
Marshal.Copy(blob, 0, bmpData.Scan0, blob.Length);
bmp.UnlockBits(bmpData);
return bmp;
The byte array looks like this.
01 c7 f0 70 3f 03 00 c3 f0 60 1e 01 80 63 ff c3 1c 71 88 21 ff c7 3c f3 8e 31 f0 c7 fc ff 86 31 f0 c7 fc ff c4 31 f0 c1 fc 3f c0 31 e0 e0 7e 0f c0 f1 e1 f8 3f 83 c0 31 e1 fe 1f e1 c6 31 e0 ff 9f f9 86 30 f0 fb 9c 79 8f 38 f0 e3 98 79 8f 38 f8 63 98 79 80 30 38 62 1c 61 80 70 10 70 3e 03 ff ff ff ff ff ff ff ff ff ff ff ff 00 7e 00 78 7f f0 88 3c 18 18 3f f0 8e 3c 7e 0f 0f c3 86 30 ff 07 87 87 c4 31 ff 87 e0 8f c0 71 ff e7 f8 0f c0 71 ff e7 f0 0f c0 71 ff e7 f0 0f c6 71 ff e7 f0 1f 86 10 ff 87 e1 1f 8f 18 ff 8f 87 87 8f 18 7f 0f 0f c3 80 1c 0c 18 3f f0 80 7e 00 38 7f f0
My image should be 192 bytes and a 48 x 32 res image. What am I doing wrong?

As Michael Liu commented, if your data is 192 bytes, and you need to get 32 lines of data out of that, then each line will be 6 bytes, meaning that to get 48 pixels out of such a line, you need to multiply it by 8, or, in other words, use 1 bit per pixel, which is PixelFormat.Format1bppIndexed.
As for your problems in visualizing the data, you need to take the stride into account; the actual amount of bytes per line of pixels.
Images created by the .Net framework always use a multiple of four bytes as width of each line of pixels, so the new 48x32 image you make will have 8 bytes on each line. However, your image data is completely compact data, so it only has 6 bytes per line.
So you need to copy it line by line, keeping the stride of both your input and output into account. This means you should replace the single Marshal.Copy by a loop over the height that does one Marshal.Copy per line, copying the data per 6 bytes from the input array to the correct start pointer of each line on the output.
The adjusted Marshal.Copy operation should be something like this:
Int32 newDataWidth = ((Image.GetPixelFormatSize(pixelFormat) * width) + 7) / 8;
Int32 targetStride = bmpData.Stride;
Int64 scan0 = bmpData.Scan0.ToInt64();
for (Int32 y = 0; y < height; y++)
Marshal.Copy(blob, y * stride, new IntPtr(scan0 + y * targetStride), newDataWidth);
...with stride the stride of your input array (so in this case, 6).
To automate the input stride calculation, if you know you have completely compact data, you can use the formula I already kind of showed in the previous block:
public static Int32 GetMinimumStride(Int32 width, Int32 bitsLength)
{
return ((bitsLength * width) + 7) / 8;
}
I already posted code here on SO to build an image from bytes in any format, so you can just use that, I guess. Using that BuildImage function, you can build your image like this:
public static Bitmap BuildMonoBitmap(Byte monoBytes, Int32 width, Int32 height)
{
Color[] pal = new Color[] {Color.Black, Color.White};
Int32 stride = GetMinimumStride(width, 1);
return BuildImage(monoBytes, width, height, stride, PixelFormat.Format1bppIndexed, pal, null);
}
The result, shown in my test program:

string to byte[] without encoding or changing actual bytes at string

assume i got the following byte[]
0C 00 21 08 01 00 00 00 86 1B 06 00 54 51 53 65 72 76 65 72
with bitconverter BitConverter.ToString i can convert it to
0C-00-21-08-01-00-00-00-86-1B-06-00-54-51-53-65-72-76-65-72
how do i convert it back from string to byte[] to get
0C 00 21 08 01 00 00 00 86 1B 06 00 54 51 53 65 72 76 65 72
ascii encoding and other methods always getting me the equivalent bytes to the string but what i really need is the string to be byte[] as it is, i know if i did a reversing operation (using getbytes then tostring) ill end up with the same string but what i care about is while at getbytes to get the exact bytes
as i said
to put
0C-00-21-08-01-00-00-00-86-1B-06-00-54-51-53-65-72-76-65-72
AS string
and get
0C 00 21 08 01 00 00 00 86 1B 06 00 54 51 53 65 72 76 65 72
As byte[]
thanks in advance

You need this
byte[] bytes = str.Split('-').Select(s => Convert.ToByte(s, 16)).ToArray();

You can use SoapHexBinary class in System.Runtime.Remoting.Metadata.W3cXsd2001 namespace
string s = "0C-00-21-08-01-00-00-00-86-1B-06-00-54-51-53-65-72-76-65-72";
byte[] buf = SoapHexBinary.Parse(s.Replace("-"," ")).Value;

Remenber that BitConverter.ToString returns an equivalent hexadecimal string representation,so
if you decide to stick with it converting back as follow:
string temp = BitConverter.ToString(buf);//buf is your array.
byte[] newbuf = temp.Split('-').Select(s => Convert.ToByte(s,16)).ToArray();
But the safest way to convert bytes to string and back is base64:
string str = Convert.ToBase64String(buf);
byte[] result = Convert.FromBase64String(str);

Passing pointer to a struct?

Hello I'm trying to pass data from a pointer to a struct but the values seem to be different.
struct somestruct
{
public file header;
public uint version;
}
unsafe struct file
{
public fixed char name[8];
public uint type;
public uint size;
}
Then in code somewhere..
public unsafe int ReadFile(string filepath)
{
somestruct f = new somestruct();
byte[] fdata = System.IO.ReadAllBytes( filepath );
fixed( byte* src = fdata )
{
f.header = *(file*)src;
MessageBox.Show( new string(f.header.name) ); //should be 'FILENAME' but it's like japanese.
}
return 0;
}
Offset(h) 00 01 02 03 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F
00000000 46 49 4C 45 4E 41 4D 45 00 00 00 01 00 00 00 30 FILENAME.......0
00000010 74 27 9F EF 74 77 F1 D7 C5 86 93 3D 39 0D 72 A9 t'Ÿïtwñ×ņ“=9.r©
00000020 63 8B 92 CF F6 7D 8A 14 45 9D 68 51 A4 8E A4 EE c‹’Ïö}Š.E.hQ¤Ž¤î
00000030 4E FE D0 66 45 0E C9 8D 96 BB F4 EE 52 1F 89 D3 NþÐfE.É.–»ôîR.‰Ó
00000040 5C 80 1A 71 8A 16 B1 8B 3A A8 1B A4 48 11 B8 E8 \€.qŠ.±‹:¨.¤H.¸è
Do you have any idea what's going on?

Each char is 2 bytes - a fixed buffer of 8 chars is 16 bytes. You are reading the first 8 bytes as only the first 4 characters in that buffer, and the high bytes will make it look. Like the eastern Unicode ranges.
I would say: deserialize it at the stream level. Don't do this.
Basically, read (at least) 20 bytes into a buffer, then decode manually, using:
string s = Encoding.ASCII.GetString(buffer, 0, 8);
For the string, and probably shift operations for the unsigned integers.
You could also use unsafe code to read the integers from the buffer, via the other meaning of fixed and a pointer-cast.

A char is UTF-16 and is 2 bytes. You need to convert the UTF-8/ANSI (1 byte) string to a UTF-16 string.

Object-to-bytes conversion

When I'm trying to convert an object into byte array I'm getting a wierd array.
this is the code:
using (MemoryStream ms = new MemoryStream())
{
BinaryFormatter bf = new BinaryFormatter();
bf.Serialize(ms, obj);
Console.WriteLine(ByteArrayToString(ms.ToArray()));
}
//int obj = 50;
//string ByteArrayToString(byte[] byteArr) the functionality of this method is pretty obvious
the result is this:
"00 01 00 00 00 FF FF FF FF 01 00 00 00 00 00 00 00 04 01 00 00 00 0C 53 79 73 74 65 6D 2E 49 6E 74 33 32 01 00 00 00 07 6D 5F 76 61 6C 75 65 00 08 32 00 00 00 0B "
Can somebody explain to me WHY?:) the optimal result should be only "32 00 00 00".

Since serializer needs to provide enough information to deserialize the data back, it must include some metadata about the object being serialized. Specifically, the
53 79 73 74 65 6D 2E 49 6E 74 33 32
part stands for System.Int32
If you use BinaryWriter and its Write(Int32) method instead, you'll get the desired effect: your memory stream will contain just the four bytes from your integer. You wouldn't be able to deserialize it without knowing that you wrote an Int32 into the stream.

You're conflating BinaryFormatter serialization with an object's in memory format. What is written to the stream is merely an implementation detail of the BinaryFormatter and should not be relied upon for any interprocess communication not using BinaryFormatter.
If you're looking for the byte representation of the built-in types, use BitConverter.GetBytes (for strings use the appropriate Encoding.GetBytes).

The serialized byte array has both the data itself and the type info. That's why you get more info than you expect. That's neccessary for later deserializing.

The extra stuff in the results would be the BinaryFormatter object. You're not just outputting int obj = 50, you're outputting everything included in the BinaryFormatter as well.

Serialization process uses extra bytes to store information about types - it's the only way to ensure that serialized data will be deserialized into same objects of same types.
If you absolutely sure in what you doing and want to avoid any extra bytes, you may use your own serialization and make your formatter and serializers, which is very complicated. Or, you could use marshalling:
var size = Marshal.SizeOf(your_object);
// Both managed and unmanaged buffers required.
var bytes = new byte[size];
var ptr = Marshal.AllocHGlobal(size);
// Copy object byte-to-byte to unmanaged memory.
Marshal.StructureToPtr(font.LogFont, ptr, false);
// Copy data from unmanaged memory to managed buffer.
Marshal.Copy(ptr, bytes, 0, size);
// Release unmanaged memory.
Marshal.FreeHGlobal(ptr);
And to convert bytes to object:
var bytes = new byte[size];
var ptr = Marshal.AllocHGlobal(size);
Marshal.Copy(bytes, 0, ptr, size);
var your_object = (YourType)Marshal.PtrToStructure(ptr, typeof(YourType));
Marshal.FreeHGlobal(ptr);
This is quite slow and unsafe to use in most cases, but it's easiest way to strictly convert object to byte[] without implementing serialization and without [Serializable] attribute.