We Keep Coding

c# UTF8 GetString from bytes array not equal to php chr function

I'm trying to make one decoder. Basic system .Net 4.7 I'm trying to migrate this system into php, but I'm having trouble converting bytes. As far as I understand the default string UTF-16le on C#, I understood the ord and chr functions as UCS-2 on the PHP side. I want to do below and I do not get the same result there are codes. What can I do to fix this, thanks in advance
XOR Encoded Text Bytes = [101,107,217,78,40,68,234,218,162,67,139,81,44,166,24,148];
on C#
string result = System.Text.Encoding.UTF8.GetString(destinationArray);
On PHP
for($i=0;$i<sizeof($encoded);$i++){
echo "\t".$encoded[$i]." => ".chr($encoded[$i])."\n";
$tmpStr .= chr($encoded[$i]);
}
C# Result size=26:
ek�N(D�ڢC�Q,��
PHP Result size=16:
ek�N(D�ڢC�Q,��
the strings looks the same, but byte translation is quite different.
C# Result to Bytes array:
byte[] utf8 = System.Text.Encoding.Unicode.GetBytes(result);
Console.WriteLine(string.Join("-", utf8));
response =
101-0-107-0-253-255-78-0-40-0-68-0-253-255-162-6-67-0-253-255-81-0-44-0-253-255-24-0-253-255
PHP Result to Bytes Array:
echo implode("-",unpack("C*", $tmpStr));
response = 101-107-217-78-40-68-234-218-162-67-139-81-44-166-24-148
if php response convert to UTF-16le, results again different
echo implode("-",unpack("C*", mb_convert_encoding($tmpStr,'UTF-16le')));
response =
101-0-107-0-63-0-78-0-40-0-68-0-63-0-162-6-67-0-63-0-81-0-44-0-63-0-24-0-63-0

You are mixing quite different things here.
First, in the C# code, you are not using the same encoding when converting from bytes to a string and then from a string back to bytes: Encoding.UTF8 in the first case and Encoding.Unicode (which is .NET name for UTF-16) in the latter... Things cannot go well if you do this. And by the way, I'm not sure that PHP's UCS2 is equivalent to UTF-16:
UTF-8 encodes characters on 1, 2, 3 or 4 bytes depending on the character
UTF-16 encodes characters on 2 or 4 bytes depending on the character
UCS-2 always encodes characters on 2 bytes, and hence cannot encode more than 65536 characters...
Then what you pass to the 'bytes to string' conversions is not necessarily valid! Because you've XORed the input data (I assume it to be some secret string), the resulting bytes may or may not be a valid sequence in some encodings. For example:
It is not valid in ASCII because you have (in your example) bytes > 127
It is not valid in UTF-8 because 217 followed by 78 is recognized neither as a 1-, 2-, 3-, or 4-byte character by UTF-8; hence, the � you see before the N.
It seems to be invalid UTF-16 as well, but roundtripping works (I could get back the original array using .NET's Unicode.GetString, then Unicode.GetBytes. If I remove your last byte - and end up with an odd number of bytes - then UTF-16 roundtripping does not work any more...
Although I did not test it, it should also be invalid UCS-2 because UCS-2 'looks like' UTF-16 for 2-byte characters.
Roundtripping works with ANSI encodings sucha as windows-1252 because these encodings accept any byte. However, I would discourage using such trick because you have to be sure the same code page is used on both sides of the encoding/decoding process.
Therefore, I think, in your case, the best way to store your XORed bytes into a string would be to convert the array to base64. In C# you can do it this way:
// The code below gives you ZWt1TihEInY+QydRLEIYMA==
var converted = Convert.ToBase64String(array);
// And this one gives you back the initial array
var bytes = Convert.FromBase64String(converted);
Quick googling will tell you to use base64_encode and base64_decode in PHP.
Bottom note: if you want to really understand what's going on with al this encodings stuff, here is the must-read blog post on the subject: https://www.joelonsoftware.com/2003/10/08/the-absolute-minimum-every-software-developer-absolutely-positively-must-know-about-unicode-and-character-sets-no-excuses/

Why am I getting two different 'formats' of hex in my bytes while evaluating an HMAC?

I'm getting a signed payload from an authentication source that comes in a base64 encoded and URL encoded format. I'm getting confused somewhere while evaluating, and ending up with similar data in different 'formats'.
Here's my code:
//Split the message to payload and signature
string[] split = raw_message.Split('.');
//Payload
string base64_payload = WebUtility.UrlDecode(split[0]);
byte[] payload = Convert.FromBase64String(base64_payload);
//Expected signature
string base64_expected_sig = WebUtility.UrlDecode(split[1]);
byte[] expected_sig = Convert.FromBase64String(base64_expected_sig);
//Signature
byte[] signature = hmacsha256.ComputeHash(payload);
//Output as a string
var foo = System.Text.Encoding.UTF8.GetString(expected_sig);
var bar = BitConverter.ToString(signature);
The expected signature (foo) comes out like so:
76eba09fcb54877299dcbd1e1e35717e3bd42e066e7ecdb131c7d0161dec3418
The computed signature (bar) is as follows:
76-EB-A0-9F-CB-54-87-72-99-DC-BD-1E-1E-35-71-7E-3B-D4-2E-06-6E-7E-CD-B1-31-C7-D0-16-1D-EC-34-18
Obviously, when comparing bytes for bytes, this doesn't work.
I see that I'm having to convert the expected_sig and the signature in different ways to get them to display as a string, but I can't figure out how I need to change the expected signature to get to where I can compare bytes for bytes.
I can obviously work around the issue but simply converting the string bar, but that's dirty and I just don't like it.
Where am I going wrong here? What am I not understanding?

The good news is that the hash computation appears to be working.
The bad news is that you're receiving the hash in a brain-dead fashion. For some reason it seems that the authors decided it was a good idea to:
Compute the hash (fine)
Convert this binary data to text as hex (fine)
Convert the hex back into binary data by applying ASCII/UTF-8/anything-ASCII-compatible encoding (why?)
Convert the result back into text using base64 (what?)
URL-encode the result (which wouldn't even be necessary with hex...)
Using either base64 or hex on the original binary makes sense, but applying both is crazy.
Anyway, it's fairly easy for you to do the same thing. For example:
string hexSignature = string.Join("", signature.Select(b => b.ToString("x2")));
byte[] hexSignatureUtf8 = Encoding.UTF8.GetBytes(hexSignature);
string finalSignature = Convert.ToBase64String(hexSignatureUtf8);
That should now match WebUtility.UrlDecode(split[1]).
Alternatively, you can work backwards from what's in the result, but I wouldn't go as far as parsing the hex back to bytes - it would be simpler to keep the first line of the above, but use:
string expectedHexBase64 = WebUtility.UrlDecode(split[1]);
byte[] expectedHexUtf8 = Convert.FromBase64String(expectedHexBase64);
string expectedHex = Encoding.UTF8.GetString(expectedHexUtf8);
Then compare it with hexSignature.
Ideally, you should talk to whoever's providing you with the crazy format and hit them with a cluestick though...

How to convert a string from iso 8859-1 to utf-8? C# Windows phone 7 -

my question is very simple but at the moment i don't know how to do this. I have a string in ISO-8859-1 format and i need to convert this string to UTF-8. I need to do it in c# on windows phone 7 sdk. How can i do it? Thanks

The MSDN page for the Encoding class lists the recognized encodings.
28591 iso-8859-1 Western European (ISO)
For your question the correct choice is iso-8859-1 which you can pass to Encoding.GetEncoding.
var inputEncoding = Encoding.GetEncoding("iso-8859-1");
var text = inputEncoding.GetString(input);
var output = Encoding.Utf8.GetBytes(text);

Two clarifications on the previous answers:
There is no Encoding.GetText method (unless it was introduced specifically for the WP7 framework). The method should presumably be Encoding.GetString.
The Encoding.GetString method takes a byte[] parameter, not a string. All strings in .NET are internally represented as UTF-16; there is no way of having a “string in ISO-8859-1 format”. Thus, you must be careful how you read your source (file, network), rather than how you process your string.
For example, to read from a text file encoded in ISO-8859-1, you could use:
string text = File.ReadAllText(path, Encoding.GetEncoding("iso-8859-1"));
To save to a text file encoded in UTF-8, you could use:
File.WriteAllText(path, text, Encoding.UTF8);
Reply to comment:
Yes. You can use Encoding.GetString to decode your byte array (assuming it contains character values for text under a particular encoding) into a string, and Encoding.GetBytes to convert your string back into a byte array (possibly of a different encoding), as demonstrated in the other answers.
The concept of “encoding” relates to how byte sequences (be they a byte[] array in memory or the content of a file on disk) are to be interpreted. The string class is oblivious to the encoding that the text was read from, or should be saved to.

You can use Convert which works pretty well, especially when you have byte array:
var latinString = "Řr"; // år
Encoding latinEncoding = Encoding.GetEncoding("iso-8859-1");
Encoding utf8Encoding = Encoding.UTF8;
byte[] latinBytes = latinEncoding.GetBytes(latinString);
byte[] utf8Bytes = Encoding.Convert(latinEncoding, utf8Encoding, latinBytes);
var utf8String = Encoding.UTF8.GetString(utf8Bytes);

How to convert between string and byte[] without losing integrity

I know how to convert from a string to byte[] in C#. In this particular case, I'm working with the string representation of an HMAC-SHA256 key. Let's say the string representation of this key I get from an OpenID OP is:
"81FNybKWfcM539vVGtJrXRmoVMxNmZHY3OgUro8+pZ8="
I convert it to byte[] like this:
byte[] myByteArr = Encoding.UTF8.GetBytes("81FNybKWfcM539vVGtJrXRmoVMxNmZHY3OgUro8+pZ8=");
The problem I have with that is that it seems to be losing the original data. If I take the byte array from the previous step and convert it back to a string, it's different from the original string.
string check = Convert.ToBase64String(myByteArr);
check ends up being:
"ODFGTnliS1dmY001Mzl2Vkd0SnJYUm1vVk14Tm1aSFkzT2dVcm84K3BaOD0="
which is obviously not the same as the original string representation I started with.

With crypto keys, always use Convert.FromBase64String and Convert.ToBase64String. That way you'll be doing it the standard way and will not lose bytes due to encoding problems. Base 64 string may not be space efficient but it is the preferred method for storage and transport of keys in many schemes.
Here is a quick verification:
byte[] myByteArr = Convert.FromBase64String("81FNybKWfcM539vVGtJrXRmoVMxNmZHY3OgUro8+pZ8=");
string check = Convert.ToBase64String(myByteArr);
Console.WriteLine(check);
// Writes: 81FNybKWfcM539vVGtJrXRmoVMxNmZHY3OgUro8+pZ8=

The first function (Encoding.UTF8.GetBytes) takes a string (of any kind) and returns a byte[] that represents that string in a particular encoding -- in your case, UTF8.
The second function (Convert.ToBase64String) takes a byte array (of any kind) and returns a string in base64 format so that you can store this binary data in any ASCII-compatible field using only printable characters.
These functions are not counterparts. It looks like the string you're getting is a base64-encoded string. If this is the case, use Convert.FromBase64String to get the byte[] that it represents, not Encoding.UTF8.GetBytes.

The bytes you get when using byte[] Encoding.GetBytes(string) and decoding a base64 string are not the same things. The former gives you the bytes that represent the string. You however want to decode a base64 string back to the bytes it represents. In that case you want to use Convert.FromBase64String().
string encoded = "81FNybKWfcM539vVGtJrXRmoVMxNmZHY3OgUro8+pZ8=";
byte[] decoded = Convert.FromBase64String(encoded); // this gives the bytes that the encoded string represents

The encoding classes have a GetString method, to convert it from a byte array back to a string.
If you used the UTF8 encoding to create the byte array, you should use the same coding to get it back again.
var original = "81FNybKWfcM539vVGtJrXRmoVMxNmZHY3OgUro8+pZ8=";
var byteArray = Encoding.UTF8.GetBytes(original);
var copy = Encoding.UTF8.GetString(byteArray);
bool match = (copy == original); // This returns true

Can we simplify this string encoding code

Is it possible to simplify this code into a cleaner/faster form?
StringBuilder builder = new StringBuilder();
var encoding = Encoding.GetEncoding(936);
// convert the text into a byte array
byte[] source = Encoding.Unicode.GetBytes(text);
// convert that byte array to the new codepage.
byte[] converted = Encoding.Convert(Encoding.Unicode, encoding, source);
// take multi-byte characters and encode them as separate ascii characters
foreach (byte b in converted)
builder.Append((char)b);
// return the result
string result = builder.ToString();
Simply put, it takes a string with Chinese characters such as 鄆 and converts them to ài.
For example, that Chinese character in decimal is 37126 or 0x9106 in hex.
See http://unicodelookup.com/#0x9106/1
Converted to a byte array, we get [145, 6] (145 * 256 + 6 = 37126). When encoded in CodePage 936 (simplified chinese), we get [224, 105]. If we break this byte array down into individual characters, we 224=e0=à and 105=69=i in unicode.
See http://unicodelookup.com/#0x00e0/1
and
http://unicodelookup.com/#0x0069/1
Thus, we're doing an encoding conversion and ensuring that all characters in our output Unicode string can be represented using at most two bytes.
Update: I need this final representation because this is the format my receipt printer is accepting. Took me forever to figure it out! :) Since I'm not an encoding expert, I'm looking for simpler or faster code, but the output must remain the same.
Update (Cleaner version):
return Encoding.GetEncoding("ISO-8859-1").GetString(Encoding.GetEncoding(936).GetBytes(text));

Well, for one, you don't need to convert the "built-in" string representation to a byte array before calling Encoding.Convert.
You could just do:
byte[] converted = Encoding.GetEncoding(936).GetBytes(text);
To then reconstruct a string from that byte array whereby the char values directly map to the bytes, you could do...
static string MangleTextForReceiptPrinter(string text) {
return new string(
Encoding.GetEncoding(936)
.GetBytes(text)
.Select(b => (char) b)
.ToArray());
}
I wouldn't worry too much about efficiency; how many MB/sec are you going to print on a receipt printer anyhow?
Joe pointed out that there's an encoding that directly maps byte values 0-255 to code points, and it's age-old Latin1, which allows us to shorten the function to...
return Encoding.GetEncoding("Latin1").GetString(
Encoding.GetEncoding(936).GetBytes(text)
);
By the way, if this is a buggy windows-only API (which it is, by the looks of it), you might be dealing with codepage 1252 instead (which is almost identical). You might try reflector to see what it's doing with your System.String before it sends it over the wire.

Almost anything would be cleaner than this - you're really abusing text here, IMO. You're trying to represent effectively opaque binary data (the encoded text) as text data... so you'll potentially get things like bell characters, escapes etc.
The normal way of encoding opaque binary data in text is base64, so you could use:
return Convert.ToBase64String(Encoding.GetEncoding(936).GetBytes(text));
The resulting text will be entirely ASCII, which is much less likely to cause you hassle.
EDIT: If you need that output, I would strongly recommend that you represent it as a byte array instead of as a string... pass it around as a byte array from that point onwards, so you're not tempted to perform string operations on it.

Does your receipt printer have an API that accepts a byte array rather than a string?
If so you may be able to simplify the code to a single conversion, from a Unicode string to a byte array using the encoding used by the receipt printer.
Also, if you want to convert an array of bytes to a string whose character values correspond 1-1 to the values of the bytes, you can use the code page 28591 aka Latin1 aka ISO-8859-1.
I.e., the following
foreach (byte b in converted)
builder.Append((char)b);
string result = builder.ToString();
can be replaced by:
// All three of the following are equivalent
// string result = Encoding.GetEncoding(28591).GetString(converted);
// string result = Encoding.GetEncoding("ISO-8859-1").GetString(converted);
string result = Encoding.GetEncoding("Latin1").GetString(converted);
Latin1 is a useful encoding when you want to encode binary data in a string, e.g. to send through a serial port.