We Keep Coding

Cannot convert special characters to UTF-8

One of my DB query from a legacy app is returning string in this format "0.5°,0.6°, 0.7°,0.8°, 0.9° and 1.0°" which is not correct. I am expecting the strings to be in format.
"0.5°, 0.75°, 1.0°, 1.5°, 1.75°, and 2.0".
I tried to convert the windows 1252 to utf-8 by doing the following:
Encoding.RegisterProvider(CodePagesEncodingProvider.Instance);
Encoding w1252 = Encoding.GetEncoding(1252);
byte[] w125bytes = w1252.GetBytes(htmlNormalized);
byte[] utfbytes = Encoding.Convert(w1252, utf8Encoding, w125bytes);
htmlNormalized = Encoding.UTF8.GetString(utfbytes);
but I am still getting the same string with special characters. what am i doing wrong? and how can I fix it?

How can I reverse this text encoding?

We have the encoding method below that accepts an input string from a legacy system file (in the Unicode format of a VB6 string) and the name of an encoding. It applies the encoding and returns a string that displays correctly in our newer web applications. As our newer applications have a reporting backend that still depends on the old formats I have a need to reverse the encoding to allow newly translated strings to be stored in the legacy files. Here are two examples of conversions done by the Encode method.
Encode("µn¤J¦WºÙ", "BIG5") returns 登入名稱
Encode("Çàðåãèñòðèðîâàííîå èìÿ", "windows-1251") returns Зарегистрированное имя
To reverse these encodings I have been trying various encoding steps based on questions found here and elsewhere but have thus far succeeded only in producing output that appears identical to the input, is composed entirely of question marks or is composed of a mixture of ASCII characters and question marks different to the original input.
The encoding method was written by a departed colleague and I must admit that I don't fully understand why it has the coded loops, while all other examples I find simply get the characters from the string, then the bytes from those using the input encoding and finally get the string from the bytes using the output encoding. If I try removing the coded loops and just doing those three steps the method no longer returns the expected result.
Here is the encoding method, and my question is, how can I create a corresponding Decode method that reverses what it does?
private static string Encode(string src, string encoding)
{
if (String.IsNullOrWhiteSpace(encoding)) return src;
Encoding unicode = Encoding.Unicode;
Encoding sourceEncoding = Encoding.GetEncoding(encoding);
char[] srcChars = src.ToCharArray();
byte[] srcBytes = sourceEncoding.GetBytes(srcChars);
if (srcChars.Length == srcBytes.Length)
{
for (int i = 0; i < srcChars.Length; i++)
if ((int)srcChars[i] < 256)
srcBytes[i] = (byte)srcChars[i];
}
else
{
srcBytes = new byte[srcChars.Length];
for (int i = 0; i < srcChars.Length; i++)
srcBytes[i] = (byte)srcChars[i];
}
byte[] unicodeBytes = Encoding.Convert(sourceEncoding, unicode, srcBytes);
return unicode.GetString(unicodeBytes);
}

German character in Livelink

I want to extract file/folder and other item type that contains some German special characters in their name from Livelink server. Livelink server has encoding UTF-8.
Value is Test Dokument äöüß
var bytes = new List<byte>(value.Length);
foreach (var c in value)
bytes.Add((byte)c);
var retValue = Encoding.UTF8.GetString(bytes.ToArray());
the above code sample fixes s encoding problem in some character but ß is seen as ? character in Latin( ISO 8859-2) encoding. can anybody help me fix the problem.
Thanks in advance

You have to set UTF-8 encoding on the LL session:
LLSTATUS llSessionStatus = LL_SessionAllocEx( &llSession, server, port, "", login, password, NULL);
LLSTATUS status = LL_SetCodePage( llSession , LL_TRUE, LL_TRUE, (LLLONG) 65001 );
65001 - is the codepage for UTF-8

It doesn't make sense to store ISO-8859-1 in a C# string, since it stores Unicode characters.
What really makes sense is to convert the Unicode string into a byte[] representing the string in ISO-8859-1.
var test ="äöüßÄÖÜ";
var iso = Encoding.GetEncoding("ISO-8859-1");
var bytes = iso.GetBytes(test);
File.WriteAllBytes("Sample file ISO-8859-1.txt", bytes);
Try this and you'll see that the text file is correctly encoded.
You can even check with a hex editor or with the debugger that the ß is correctly encoded as 0xDF (see table on wikipedia)

Encoding not converting

An ASP.NET page (ashx) receives a GET request with a UTF8 string. It reads a SqlServer database with Windows-1255 data.
I can't seem to get them to work together. I've used information gathered on SO (mainly Convert a string's character encoding from windows-1252 to utf-8) as well as msdn on the subject.
When I run anything through the functions below - it always ends up the same as it started - not converted at all.
Is something done wrong?
EDIT
What I'm specifically trying to do (getData returns a Dictionary<int, string>):
getData().Where(a => a.Value.Contains(context.Request.QueryString["q"]))
Result is empty, unless I send a "neutral" character such as "'" or ",".
CODE
string windows1255FromUTF8(string p)
{
Encoding win = Encoding.GetEncoding(1255);
Encoding utf8 = Encoding.UTF8;
byte[] utfBytes = utf8.GetBytes(p);
byte[] winBytes = Encoding.Convert(utf8, win, utfBytes);
return win.GetString(winBytes);
}
string UTF8FromWindows1255(string p)
{
Encoding win = Encoding.GetEncoding(1255);
Encoding utf8 = Encoding.UTF8;
byte[] winBytes = win.GetBytes(p);
byte[] utfBytes = Encoding.Convert(win, utf8, winBytes);
return utf8.GetString(utfBytes);
}

There is nothing wrong with the functions, they are simply useless.
What the functions do is to encode the strings into bytes, convert the data from one encoding to another, then decode the bytes back to a string. Unless the string contains a character that is not possible to encode using the windows-1255 encoding, the returned value should be identical to the input.
Strings in .NET doesn't have an encoding. If you get a string from a source where the text was encoded using for example UTF-8, once it's decoded into a string it doesn't have that encoding any more. You don't have to do anyting to a string to use it when the destination has a specific encoding, whatever library you are using that takes the string will take care of the encoding.

For some reason this worked:
byte[] fromBytes = (fromEncoding.UTF8).GetBytes(myString);
string finalString = (Encoding.GetEncoding(1255)).GetString(fromBytes);
Switching encoding without the conversion...

C# Encoding.Converting Latin to Hebrew

I'm trying to fetch and parse an online excel document which is written in hebrew but unfortunately in a non-hebrew encoding.
As an example I'm trying to convert the following string: "âìéåï_1", which serves as the 1st sheet name to hebrew using C# code, but I'm unable to do so.
I know the above is convertible, since when I open it up in NotePad++ and select Encoding/Character Sets/Hebrew/Windows 1255, I can see: "גליון_1" which is the correct hebrew representation of the above string.
I'm using the below code
string str = "âìéåï_1";
Encoding windows = Encoding.GetEncoding("Windows-1255");
Encoding ascii = Encoding.GetEncoding("Windows-1252");
byte[] asciiBytes = ascii.GetBytes(str);
byte[] windowsBytes = Encoding.Convert(ascii, windows, asciiBytes);
char[] windowsChars = new char[windows.GetCharCount(windowsBytes, 0, windowsBytes.Length)];
windows.GetChars(windowsBytes, 0, windowsBytes.Length, windowsChars, 0);
string windowsString = new string(windowsChars);
I assumed that the encoding of the origin string is Windows-1252 since when I paste it in NotePad++ and change the encoding to Windows-1252 the string remains the same...
I'm probably doing something wrong here, anyone know how to convert the above correctly?
Thanks,
Mikey

const string Str = "âìéåï_1";
Encoding latinEncoding = Encoding.GetEncoding("Windows-1252");
Encoding hebrewEncoding = Encoding.GetEncoding("Windows-1255");
byte[] latinBytes = latinEncoding.GetBytes(Str);
string hebrewString = hebrewEncoding.GetString(latinBytes);
hebrewString:
גליון_1
In your supplied example "Window-1252" is not actualy ASCII, it is extended ASCII, and for some reason Encoding.Convert with these two encodings cannot convert extended range ASCII, so all +127 characters are converted as 63 (i.e. ?). When "converting" from one extended ASCII character byte[] to another, I would expect the bytes to be the same, it is only when you convert them to a .Net unicode string I would expect them to be different. Not sure why Convert is converting +127 chars to '?'.