We Keep Coding

C# How to fix loss of data during file to binary to string to binary conversion

I read a file as binary, convert to hex string, convert back to binary, and write to a new file.
I expect a duplicate, but get a corrupted file.
I have been trying different ways to convert the binary into the hex string but can't seem to retain the entire file.
byte[] binary1 = File.ReadAllBytes(#"....Input.jpg");
string hexString = "";
int counter1 = 0;
foreach (byte b in binary1)
{
counter1++;
hexString += (Convert.ToString(b, 16));
}
List<byte> bytelist = new List<byte>();
int counter2 = 0;
for (int i = 0; i < hexString.Length/2; i++)
{
counter2++;
string ch = hexString.Substring(i*2,2);
bytelist.Add(Convert.ToByte(ch, 16));
}
byte[] binary2 = bytelist.ToArray();
File.WriteAllBytes(#"....Output.jpg", binary2);
Counter 1 and counter 2 should be the same count, but counter 2 is always about 10% smaller.
I want to get a duplicate file output that I could have transferred around via that string value.

Converting byte 10 will give a single char, and not 2 characters. Your convert back method (logically) build on 2 chars per byte.
this case works
byte[] binary1 = new byte[] { 100 }; // convert will result in "64"
and this case fails
byte[] binary1 = new byte[] { 10 }; // convert will result in "a"
I quick fixed your code, by padding with a "0" in case of a single char.
so working code:
byte[] binary1 = new byte[] { 100 };
string hexString = "";
int counter1 = 0;
foreach (byte b in binary1)
{
counter1++;
var s = (Convert.ToString(b, 16));
// new
if (s.Length < 2)
{
hexString += "0";
}
// end new
hexString += s;
}
List<byte> bytelist = new List<byte>();
int counter2 = 0;
for (int i = 0; i < hexString.Length / 2; i++)
{
counter2++;
string ch = hexString.Substring(i * 2, 2);
var item = Convert.ToByte(ch, 16);
bytelist.Add(item);
}
byte[] binary2 = bytelist.ToArray();
Please note, your code could use some refactoring, e.g. don't string concat in a loop and maybe check the Single Responsibility Principle.
Update, got it fixed, but there are better solutions here: How do you convert a byte array to a hexadecimal string, and vice versa?

C# hex to byte array loop

I have the following function:
public void SetTagData(string _data)
{
string data = _data;
byte[] ba = Encoding.Default.GetBytes(data);
string hexString = BitConverter.ToString(ba);
hexString = hexString.Replace("-", "");
var blockStart = 0;
var bufferHexBlocks = String.Empty;
try
{
for (var i = 0; i < hexString.Length; i++)
{
var byteList = new List<byte>();
byte[] datablockKey = ConvertHelpers.ConvertHexStringToByteArray(i.ToString().PadLeft(2, '0'));
var block = hexString.Substring(blockStart, 8);
byte[] datablockValue = ConvertHelpers.ConvertHexStringToByteArray(block);
byteList.AddRange(datablockKey);
byteList.AddRange(datablockValue);
_reader.Protocol("wb", byteList.ToArray());
blockStart += 8;
}
}
catch (Exception ex)
{
console.log(ex.message);
}
}
The data coming in is a bunch of hex as a string. I need to split this hex string into batches of 8 characters, append an incrementing 0 padded hex number from 00 to 1f and send this new string as a byte array to the _reader.Protocol function, which accepts a string wb as first parameter and the block as the second.
For example incoming data is:
string data = "3930313B36313B5350542D53504C3B3830303B3B352E373B3B303B303B3B3B34353036383B4E3B4E3B"
I need to send the following to the _reader.Protocol object:
(incremented padded hex 01, 02, 03, ... , 0f) and the first 8 characters of the data string, then the next, and so on as a byte array.
[013930313B], [0236313B53], etc.
I think I'm getting close... but missing something...
My problem at the moment is that I can't figure out how to loop in blocks of 8 and if the hex string is say 82 characters instead of 80 (multiple of 8), then how would I grab the last two characters without getting a IndexOutofRange exception.
Note: This is for a Windows CE application, so no new C# features please.

This below will work fine in conjunction with this answer and the sample string data given.
public static byte[] Parse(string data)
{
var count = data.Length / 8; //Might be worth throwing exception with any remainders unless you trust the source.
var needle = 0;
List<byte> result = new List<byte>(); //Inefficient but I'm being lazy
for (int i = 0; i < count; i++)
{
char[] buffer = new char[8];
data.CopyTo(needle, buffer, 0, buffer.Length);
//To get around the odd number when adding the prefixed count byte, send the hex string to the convert method separately.
var bytes = ConvertHexStringToByteArray(new string(buffer)); //Taken From https://stackoverflow.com/a/8235530/6574422
//As the count is less than 255, seems safe to parse to single byte
result.Add(byte.Parse((i + 1).ToString()));
result.AddRange(bytes);
needle += 8;
}
return result.ToArray();
}

I'm figured it out. It might not be the most efficient solution but it works just fine. I did it using a for loop inside a for loop.
In case anyone is interested here is the final code:
public void SetTagData(string _data)
{
string data = _data;
byte[] ba = Encoding.Default.GetBytes(data);
string hexString = BitConverter.ToString(ba);
hexString = hexString.Replace("-", "");
var blockStart = 0;
try
{
_reader.Protocol("s");
for(var count = 0; count < 16; count++)
{
var byteList = new List<byte>();
byte[] datablockKey = ConvertHelpers.ConvertHexStringToByteArray(count.ToString("X2"));
byteList.AddRange(datablockKey);
for (var innerCount = 0; innerCount < 4; innerCount++)
{
var block = String.Empty;
if (!String.IsNullOrEmpty(hexString.Substring(blockStart, 2)))
{
block = hexString.Substring(blockStart, 2);
}
else
{
block = "20";
}
byte[] datablockValue = ConvertHelpers.ConvertHexStringToByteArray(block);
byteList.AddRange(datablockValue);
blockStart += 2;
}
_reader.Protocol("wb", byteList.ToArray());
}
}
catch (Exception)
{
}
}

Convert Binary String to its readable equivalence C#

I have a string array that contains some binary data, and I would like to convert the binary to its equivalent character representative.
Each element inside the array contains 8 bit "1 byte" of data and I need to know how to convert it to its character equivalence
Here is the string array:
IEnumerable<string> resultChunks = Enumerable.Range(0, result.Length / 8)
.Select(x => result.Substring(x * 8, 8));
string[] newRes = resultChunks.ToArray();
string tempRes="";
for (int i = 0; i < newRes.Length; i++)
{
tempRes+=Convert.ToString(newRes[i]);
}
Current "result" is "0010001111000100001010010011101111000111001100110110011100110110"

If your data is in a byte array then you could use this:
string result = Encoding.ASCII.GetString(bytes);

I guess, your string is base64 string.
Then you can use next method:
public static string FromBase64(string base64Str, Encoding encoding = null)
{
if (string.IsNullOrWhiteSpace(base64Str))
{
return string.Empty;
}
byte[] bytes;
try
{
bytes = System.Convert.FromBase64String(base64Str);
}
catch (FormatException)
{
return string.Empty;
}
var stringEncoding = encoding ?? Encoding.UTF8;
return stringEncoding.GetString(bytes);
}

Trying to reproduce PHP's pack("H*") function in C#

this is my code in C# :
public static String MD5Encrypt(String str, Boolean raw_output=false)
{
// Use input string to calculate MD5 hash
String output;
MD5 md5 = System.Security.Cryptography.MD5.Create();
byte[] inputBytes = System.Text.Encoding.ASCII.GetBytes(str);
byte[] hashBytes = md5.ComputeHash(inputBytes);
// Convert the byte array to hexadecimal string
StringBuilder sb = new StringBuilder();
for (int i = 0; i < hashBytes.Length; i++)
{
sb.Append(hashBytes[i].ToString("x2"));
}
output = sb.ToString();
if (raw_output)
{
output = pack(output);
}
return output;
}
public static String pack(String S)
{
string MultiByte = "";
for (int i = 0; i <= S.Length - 1; i += 2)
{
MultiByte += Convert.ToChar(HexToDec(S.Substring(i, 2)));
}
return MultiByte;
}
private static int HexToDec(String hex)
{
//Int32.Parse(hexString, System.Globalization.NumberStyles.HexNumber);
return Convert.ToInt32(hex, 16);
}
To reproduce what is done in php by this way :
md5($str, true);
OR
pack('H*', md5( $str ));
I tried many things but can't get the same on the two sides in some cases of word.
For example, Trying this test on the string "8tv7er5j"
PHP Side :
9c36ad446f83ca38619e12d9e1b3c39e <= md5("8tv7er5j");
œ6­DoƒÊ8ažÙá³Ãž <= md5("8tv7er5j", true) or pack("H*", md5("8tv7er5j"))
C# Side :
9c36ad446f83ca38619e12d9e1b3c39e <= MD5Encrypt("8tv7er5j")
6­DoÊ8aÙá³Ã <= MD5Encrypt("8tv7er5j", true) or pack( MD5Encrypt("8tv7er5j") )
Why ? Encoding problem ?
EDIT 1 :
I have the good result, but bad encoded with this this function for pack() :
if ((hex.Length % 2) == 1) hex += '0';
byte[] bytes = new byte[hex.Length / 2];
for (int i = 0; i < hex.Length; i += 2)
{
bytes[i / 2] = Convert.ToByte(hex.Substring(i, 2), 16);
}
return bytes;
So, System.Text.Encoding.UTF8.GetString(bytes) give me :
�6�Do��8a���Þ
And System.Text.Encoding.ASCII.GetString(bytes)
?6?Do??8a??????
...

I encountered same scenario where I am in need of php's pack-unpack-md5 functions in C#. Most important was that I need to match out of all these 3 functions with php.
I created my own functions and then validated(verified) my output with functions at onlinephpfunctions.com. The output was same when I parsed with DefaultEncoding. FYI, I checked my application's encoding(Encoding.Default.ToString()) and it was System.Text.SBCSCodePageEncoding
Pack
private static string pack(string input)
{
//only for H32 & H*
return Encoding.Default.GetString(FromHex(input));
}
public static byte[] FromHex(string hex)
{
hex = hex.Replace("-", "");
byte[] raw = new byte[hex.Length / 2];
for (int i = 0; i < raw.Length; i++)
{
raw[i] = Convert.ToByte(hex.Substring(i * 2, 2), 16);
}
return raw;
}
MD5
private static string md5(string input)
{
byte[] asciiBytes = Encoding.Default.GetBytes(input);
byte[] hashedBytes = MD5CryptoServiceProvider.Create().ComputeHash(asciiBytes);
string hashedString = BitConverter.ToString(hashedBytes).Replace("-", "").ToLower();
return hashedString;
}
Unpack
private static string unpack(string p1, string input)
{
StringBuilder output = new StringBuilder();
for (int i = 0; i < input.Length; i++)
{
string a = Convert.ToInt32(input[i]).ToString("X");
output.Append(a);
}
return output.ToString();
}
PS: User can enhance these functions with other formats

I guess that PHP defaults to Latin1 so the code should look like :
public static String PhpMd5Raw(string str)
{
var md5 = System.Security.Cryptography.MD5.Create();
var inputBytes = System.Text.Encoding.ASCII.GetBytes(str);
var hashBytes = md5.ComputeHash(inputBytes);
var latin1Encoding = System.Text.Encoding.GetEncoding("ISO-8859-1");
return latin1Encoding.GetString(hashBytes);
}

If you are going to feed the result as a key for HMAC-SHA1 hashing keep it as bytes[] and initialize the HMACSHA1 with the return value of this function: DO NOT convert it to a string and back to bytes, I have spent hours because of this mistake.
public static byte[] PackH(string hex)
{
if ((hex.Length % 2) == 1) hex += '0';
byte[] bytes = new byte[hex.Length / 2];
for (int i = 0; i < hex.Length; i += 2)
{
bytes[i / 2] = Convert.ToByte(hex.Substring(i, 2), 16);
}
return bytes;
}

I know this is an old question. I am posting my answer for anyone who might reach this page searching for it.
The following code is the full conversion of the pearl function pack("H*") to c#.
public static String Pack(String input)
{
input = input.Replace("-", " ");
byte[] hashBytes = new byte[input.Length / 2];
for (int i = 0; i < hashBytes.Length; i++)
{
hashBytes[i] = Convert.ToByte(input.Substring(i * 2, 2), 16);
}
return Encoding.UTF7.GetString(hashBytes); // for perl/php
}

I'm sorry. I didn't go with the questions completely. But if php code is as below,
$testpack = pack("H*" , "you value");
and if can't read the $testpack values(due to some non support format), then first do base64_encode as below and echo it.
echo base64_encode($testpack);
Then use Risky Pathak answer. For complete this answer I'll post his answer with some small modification like base 64 encoding etc.
var hex = "you value";
hex = hex.Replace("-", "");
byte[] raw = new byte[hex.Length / 2];
for (int i = 0; i < raw.Length; i++)
{
raw[i] = Convert.ToByte(hex.Substring(i * 2, 2), 16);
}
var res = Convert.ToBase64String(raw);
Console.WriteLine(res);
Now if you compare both of values, those should be similar.
And all credit should go to the Risky Pathak answer.

The same in c# can be reached with Hex.Decode() method.
And bin2hex() in php is Hex.Encode().

BinaryWriter problem - "code adds some byte between Write() method"

I am try to do some code using BinaryWriter and Then BinaryReader.
When I wanna write I use method Write().
But the problem is that between two lines of Write method there appears a new byte which is in ASCII table in decimal 31 (sometines 24).
You can see it on this image:
You can see that byte at index 4 (5th byte) is of ASCII decimal value 31. I didnt insert it there. As you can see 1st 4 bytes are reserved for a number (Int32), next are other data (some text mostly - this is not important now).
As you can see from the code i write:
- into 1st line a number 10
- into 2nd line text "This is some text..."
How come came that 5th byte (dec 31) in between??
And this is the code I have:
static void Main(string[] args)
{
//
//// SEND - RECEIVE:
//
SendingData();
Console.ReadLine();
}
private static void SendingData()
{
int[] commandNumbers = { 1, 5, 10 }; //10 is for the users (when they send some text)!
for (int i = 0; i < commandNumbers.Length; i++)
{
//convert to byte[]
byte[] allBytes;
using (MemoryStream ms = new MemoryStream())
{
using (BinaryWriter bw = new BinaryWriter(ms))
{
bw.Write(commandNumbers[i]); //allocates 1st 4 bytes - FOR MAIN COMMANDS!
if (commandNumbers[i] == 10)
bw.Write("This is some text at command " + commandNumbers[i]); //HERE ON THIS LINE IS MY QUESTION!!!
}
allBytes = ms.ToArray();
}
//convert back:
int valueA = 0;
StringBuilder sb = new StringBuilder();
foreach (var b in GetData(allBytes).Select((a, b) => new { Value = a, Index = b }))
{
if (b.Index == 0) //1st num
valueA = BitConverter.ToInt32(b.Value, 0);
else //other text
{
foreach (byte _byte in b.Value)
sb.Append(Convert.ToChar(_byte));
}
}
if (sb.ToString().Length == 0)
sb.Append("ONLY COMMAND");
Console.WriteLine("Command = {0} and Text is \"{1}\".", valueA, sb.ToString());
}
}
private static IEnumerable<byte[]> GetData(byte[] data)
{
using (MemoryStream ms = new MemoryStream(data))
{
using (BinaryReader br = new BinaryReader(ms))
{
int j = 0;
byte[] buffer = new byte[4];
for (int i = 0; i < data.Length; i++)
{
buffer[j++] = data[i];
if (i == 3) //SENDING COMMAND DATA
{
yield return buffer;
buffer = new byte[1];
j = 0;
}
else if (i > 3) //SENDING TEXT
{
yield return buffer;
j = 0;
}
}
}
}
}

If you look at the documentation for Write(string), you'll see that it writes a length-prefixed string. So the 31 is the number of characters in your string -- perfectly normal.

You should probably be using Encoding.GetBytes and then write the bytes instead of writing a string
for example
bw.Write(
Encoding.UTF8.GetBytes("This is some text at command " + commandNumbers[i])
);

When a string is written to a binary stream, the first thing it does is write the length of the string. The string "This is some text at command 10" has 31 characters, which is the value you're seeing.

You should check the documentation of methods you use before asking questions about them:
A length-prefixed string represents the string length by prefixing to
the string a single byte or word that contains the length of that
string. This method first writes the length of the string as a UTF-7
encoded unsigned integer, and then writes that many characters to the
stream by using the BinaryWriter instance's current encoding.
;-)
(Though in fact it is an LEB128 and not UTF-7, according to Wikipedia).

The reason this byte is there because you're adding a variable amount of information, so the length is needed. If you were to add two strings, where would you know where the first ended and the second began?
If you really don't want or need that length byte, you can always convert the string to a byte array and use that.

Ok, here is my edited code. I removed BinaryWriter (while BinaryReader is still there!!), and now it works very well - no more extra bytes.
What do you thing? Is there anytihng to do better, to make it run faster?
Expecially Im interesting for that foreach loop, which read from another method that is yield return type!!
New Code:
static void Main(string[] args)
{
//
//// SEND - RECEIVE:
//
SendingData();
Console.ReadLine();
}
private static void SendingData()
{
int[] commands = { 1, 2, 3 };
// 1 - user text
// 2 - new game
// 3 - join game
// ...
for (int i = 0; i < commands.Length; i++)
{
//convert to byte[]
byte[] allBytes;
using (MemoryStream ms = new MemoryStream())
{
// 1.st - write a command:
ms.Write(BitConverter.GetBytes(commands[i]), 0, 4);
// 2nd - write a text:
if (commands[i] == 1)
{
//some example text (like that user sends it):
string myText = "This is some text at command " + commands[i];
byte[] myBytes = Encoding.UTF8.GetBytes(myText);
ms.Write(myBytes, 0, myBytes.Length);
}
allBytes = ms.ToArray();
}
//convert back:
int valueA = 0;
StringBuilder sb = new StringBuilder();
foreach (var b in ReadingData(allBytes).Select((a, b) => new { Value = a, Index = b }))
{
if (b.Index == 0)
{
valueA = BitConverter.ToInt32(b.Value, 0);
}
else
{
sb.Append(Convert.ToChar(b.Value[0]));
}
}
if (sb.ToString().Length == 0)
sb.Append("ONLY COMMAND");
Console.WriteLine("Command = {0} and Text is \"{1}\".", valueA, sb.ToString());
}
}
private static IEnumerable<byte[]> ReadingData(byte[] data)
{
using (MemoryStream ms = new MemoryStream(data))
{
using (BinaryReader br = new BinaryReader(ms))
{
int j = 0;
byte[] buffer = new byte[4];
for (int i = 0; i < data.Length; i++)
{
buffer[j++] = data[i];
if (i == 3) //SENDING COMMAND DATA
{
yield return buffer;
buffer = new byte[1];
j = 0;
}
else if (i > 3) //SENDING TEXT
{
yield return buffer;
j = 0;
}
}
}
}
}