We Keep Coding

UTF-8 CSV file created with C# shows  characters in Excel

When a CSV file is generated using C# and opened in Microsoft Excel it displays  characters before special symbols e.g. £
In Notepad++ the hex value for  is: C2
So before writing the £ symbol to file, I have tried the following...
var test = "£200.00";
var replaced = test.Replace("\xC2", " ");
StreamWriter outputFile = File.CreateText("testoutput.csv"); // default UTF-8
outputFile.WriteLine(replaced);
outputFile.Close();
When opening the CSV file in Excel, I still see the "Â" character before the £ symbol (hex equivalent \xC2 \xA3); It made no difference.
Do I need to use a different encoding? or am I missing something?

Thank you #Evk and #Mortalier, your suggestions lead me to the right direction...
I needed to update my StreamWriter so it would explicitly include UTF-8 BOM at the beginning http://thinkinginsoftware.blogspot.co.uk/2017/12/correctly-generate-csv-that-excel-can.html
So my code has changed from:
StreamWriter outputFile = File.CreateText("testoutput.csv"); // default UTF-8
To:
StreamWriter outputFile = new StreamWriter("testoutput.csv", false, new UTF8Encoding(true))
Or: Another solution I found here was to use a different encoding if you're only expecting latin characters...
http://theoldsewingfactory.com/2010/12/05/saving-csv-files-in-utf8-creates-a-characters-in-excel/
StreamWriter outputFile = new StreamWriter("testoutput.csv", false, Encoding.GetEncoding("Windows-1252"))
My system will most likely use latin & non-latin characters so I'm using the UTF-8 BOM solution.
Final code
var test = "£200.00";
StreamWriter outputFile = new StreamWriter("testoutput.csv", false, new UTF8Encoding(true))
outputFile.WriteLine(test);
outputFile.Close();

I tried your code and Excel does show AŁ in the cell.
Then I tried to open the csv with LibreOffice Clac. At first there too was AŁ, but
on import the program will ask you about encoding.
Once I chose UTF-8 the £ symbol was displayed correctly.
My guess is that in fact there is an issue with your encoding.
This might help with Excel https://superuser.com/questions/280603/how-to-set-character-encoding-when-opening-excel

C# .csv-file in WinForm with Ä, Ö, Ü [duplicate]

I'm using the code below to read a text file that contains foreign characters, the file is encoded ANSI and looks fine in notepad. The code below doesn't work, when the file values are read and shown in the datagrid the characters appear as squares, could there be another problem elsewhere?
StreamReader reader = new StreamReader(inputFilePath, System.Text.Encoding.ANSI);
using (reader = File.OpenText(inputFilePath))
Thanks
Update 1: I have tried all encodings found under System.Text.Encoding. and all fail to show the file correctly.
Update 2: I've changed the file encoding (resaved the file) to unicode and used System.Text.Encoding.Unicode and it worked just fine. So why did notepad read it correctly? And why didn't System.Text.Encoding.Unicode read the ANSI file?

You may also try the Default encoding, which uses the current system's ANSI codepage.
StreamReader reader = new StreamReader(inputFilePath, Encoding.Default, true)
When you try using the Notepad "Save As" menu with the original file, look at the encoding combo box. It will tell you which encoding notepad guessed is used by the file.
Also, if it is an ANSI file, the detectEncodingFromByteOrderMarks parameter will probably not help much.

I had the same problem and my solution was simple: instead of
Encoding.ASCII
use
Encoding.GetEncoding("iso-8859-1")
The answer was found here.
Edit: more solutions. This maybe more accurate one:
Encoding.GetEncoding(1252);
Also, in some cases this will work for you too if your OS default encoding matches file encoding:
Encoding.Default;

Yes, it could be with the actual encoding of the file, probably unicode. Try UTF-8 as that is the most common form of unicode encoding. Otherwise if the file ASCII then standard ASCII encoding should work.

Using Encoding.Unicode won't accurately decode an ANSI file in the same way that a JPEG decoder won't understand a GIF file.
I'm surprised that Encoding.Default didn't work for the ANSI file if it really was ANSI - if you ever find out exactly which code page Notepad was using, you could use Encoding.GetEncoding(int).
In general, where possible I'd recommend using UTF-8.

Try a different encoding such as Encoding.UTF8. You can also try letting StreamReader find the encoding itself:
StreamReader reader = new StreamReader(inputFilePath, System.Text.Encoding.UTF8, true)
Edit: Just saw your update. Try letting StreamReader do the guessing.

For swedish Å Ä Ö the only solution form the ones above working was:
Encoding.GetEncoding("iso-8859-1")
Hopefully this will save someone time.

File.OpenText() always uses an UTF-8 StreamReader implicitly. Create your own StreamReader
instance instead and specify the desired encoding.
like
using (StreamReader reader = new StreamReader(#"C:\test.txt", Encoding.Default)
{
// ...
}

I solved my problem of reading portuguese characters, changing the source file on notepad++.
C#
var url = System.Web.HttpContext.Current.Server.MapPath(#"~/Content/data.json");
string s = string.Empty;
using (System.IO.StreamReader sr = new System.IO.StreamReader(url, System.Text.Encoding.UTF8,true))
{
s = sr.ReadToEnd();
}

I'm also reading an exported file which contains french and German languages. I used Encoding.GetEncoding("iso-8859-1"), true which worked out without any challenges.

for Arabic, I used Encoding.GetEncoding(1256). it is working good.

I had a similar problem with ProcessStartInfo and the property StandardOutputEncoding. I set it for German language console output to code page 850. This way I could read the output like ausführen instead of ausf�hren.

Handling Special Characters (¦)

I'm a bit lost on how to read and write to/from text files in C# when special characters are present. I'm writing a simple script that does some cleanup on a .txt data file which contains the '¦' character as its delimiter.
foreach (string file in Directory.EnumerateFiles(#"path\raw txt","*.txt"))
{
string contents = File.ReadAllText(file);
contents = contents.Replace("¦", ",");
File.WriteAllText(file.Replace("raw txt", "txt"), contents);
}
However, when I open the txt file in Notepad++, the delimeter is now �. What exactly is going on? What even is this characters (¦) encoding / how would I determine that? I've tried adding things like:
string contents = File.ReadAllText(file, Encoding.UTF8);
File.WriteAllText(file.Replace("raw txt", "txt"), contents, Encoding.UTF8);

Everything is now working correctly by switching the encoding to 'default' when both reading/writing.
string contents = File.ReadAllText(file, Encoding.Default);
File.WriteAllText(file.Replace("raw txt", "txt"), contents, Encoding.Default);

Try change encoding of Notepad to UTF-8

Problems with strings in the CSV file

I have an application that reads information from a CSV file to write it to the database. But some characters (example: º ç) are appearing problems Gravalos base. Anyone know how to fix this problem?
Thank you.
I'm using these lines of code to read the information from the CSV file:
string directory = #"C:\test.csv";
StreamReader stream = new StreamReader(directory);
string line = "";
line = stream.ReadLine();
string[] column = line.Split(';');

StreamReader defaults to UTF8 encoding and your file is in a different encoding. Try specifying it like this...
var encoding = Encoding.UTF16;
StreamReader stream = new StreamReader(directory, encoding);
Note that you need to know what encoding the file is in to properly read it... I'm just guessing that it might be UTF16 but obviously I can't know what it is.

You should specify the right encoding when reading the file. The default is UTF-8. Your file is probably encoded with a different encoding.

This is most likely related to the Encoding that is used when reading the file. By default, UTF8 is assumed as the Encoding. In order to read the file correctly, you need to specify the right encoding, e.g.:
string directory = #"C:\test.csv";
using(StreamReader stream = new StreamReader(directory, Encoding.ASCII))
{
string line = "";
line = stream.ReadLine();
string[] column = line.Split(';');
}
You can try the following encodings (see this link for a complete list):
Encoding.Default for ANSI encoding based in the current windows code page.
Encoding.ASCII for ASCII encoding.
Encoding.UTF* for different Unicode encodings.
Please note that I enclosed the StreamReader in a using block so that it is disposed when it is not needed anymore.

How to retain similar character encoding

I have a logfile that contains the half character ½, I need to process this log file and rewrite certain lines to a new file, which contain that character. However, when I write out the file the characters appear in notepad incorrectly.
I know this is some kind of encoding issue, and i'm not sure if it's just that the files i'm writing don't contain the correct bom or what.
I've tried reading and writing the file with all the available encoding options in the Encoding enumeration.
I'm using this code:
string line;
// Note i've used every version of the Encoding enumeration
using (StreamReader sr = new StreamReader(file, Encoding.Unicode))
using (StreamWRiter sw = new StreamWriter(newfile, false, Encoding.Unicode))
{
while ((line = sr.ReadLine()) != null)
{
// process code, I do not alter the lines, they are copied verbatim
// but i do not write every line that i read.
sw.WriteLine(line);
}
}
When I view the original log in notepad, the half character displays correctly. When I view the new file, it does not. This tells me the problem is not with notepad being able to display the character, because it works in the original.
Can anyone help me to solve this?

The solution was PEBKAC.
I was changing the encodings in a different part of the program that wasn't creating these files. When I changed the correct files, using Encoding.Default, it displays correctly.
Thanks Jon and others.