We Keep Coding

Getting data from JSON file in different languages using C#

I tried to get data from JSON file into my app using this code
(Hebrew letters appear like this "??"):
private static T PopulateData<T>(string fileName)
{
var file = "TestingNav.Data." + fileName;
var assembly = Assembly.GetExecutingAssembly();
T obj;
var resourceStream = assembly.GetManifestResourceStream(file);
using (var reader = new StreamReader(resourceStream, Encoding.Default))
{
string data = reader.ReadToEnd();
byte[] byteArray = Encoding.Default.GetBytes(data);
MemoryStream stream = new MemoryStream(byteArray);
var serializer = new DataContractJsonSerializer(typeof(T));
obj = (T)serializer.ReadObject(stream);
}
return obj;
}
however, it didn't work on different languages i.e. Hebrew... although I did try to put encoding of different types in it.
the JSON part that the code cant read:
{
"itemName": "Hamburger",
"name": "Hamburger 52",
"description": "Hamburger 52 Y/O",
"itemDescription": "טעים מאוד hamburger with Angus beef grilled to perfection",
"itemRating": 4.5
},
update:
private static T PopulateData<T>(string fileName)
{
var file = "TestingNav.Data." + fileName;
T obj;
var assembly = IntrospectionExtensions.GetTypeInfo(typeof(T)).Assembly;
Stream stream = assembly.GetManifestResourceStream(file);
string text = "";
using (var reader = new StreamReader(stream))
{
text = reader.ReadToEnd();
obj = JsonConvert.DeserializeObject<T>(text);
}
return obj;
}

The solution was quite simple thank you for all the massive help #Panagiotis Kanavos #dbc #Heretic Monkey
the solution:
private static T PopulateData<T>(string fileName)
{
var file = "TestingNav.Data." + fileName;
T obj;
var assembly = IntrospectionExtensions.GetTypeInfo(typeof(T)).Assembly;
Stream stream = assembly.GetManifestResourceStream(file);
string text = "";
using (var reader = new StreamReader(stream, Encoding.GetEncoding(1255)))
{
text = reader.ReadToEnd();
obj = JsonConvert.DeserializeObject<T>(text);
}
return obj;
}

XMLSerializer - issue with UTF-8 vs UTF-16 Code

I am trying to serialize a simple object (5 string properties) into XML to save to a DB Image field. Then I need to DeSerialize it back into a string later in the program.
However, I am getting some errors - caused by the XML being saved thinking it is in UTF-16 - however, when I load it from the DB back into a string - it thinks it is a UTF 8 String.
The error I get is
InnerException {"There is no Unicode byte order mark. Cannot switch to Unicode."} System.Exception {System.Xml.XmlException}
-- Message "There is an error in XML document (0, 0)." string
Is this happening because of the two different ways I save and load the string to/from the DB? On the save I am using a StringBuilder - but on the load from DB I am using just a String.
Thoughts?
Serialize and Save to DB
// Now Save the OBject XML to the Query Tables
var serializer = new XmlSerializer(ExportConfig.GetType());
StringBuilder StringResult = new StringBuilder();
using (var writer = XmlWriter.Create(StringResult))
{
serializer.Serialize(writer, ExportConfig);
}
//MessageBox.Show("XML : " + StringResult);
// Now Save to the Query
try
{
string UpdateSQL = "Update ZQryRpt "
+ " Set ExportConfig = " + TAGlobal.QuotedStr(StringResult.ToString())
+ " where QryId = " + TAGlobal.QuotedStr(((DataRowView)bindingSource_zQryRpt.Current).Row["QryID"].ToString())
;
ExecNonSelectSQL(UpdateSQL, uniConnection_Config);
}
catch (Exception Error)
{
MessageBox.Show("Error Setting ExportConfig: " + Error.Message);
}
Load from DB And Deserialize
byte[] binaryData = (byte[])((DataRowView)bindingSource_zQryRpt.Current).Row["ExportConfig"];
string XMLStored = System.Text.Encoding.UTF8.GetString(binaryData, 0, binaryData.Length);
if (XMLStored.Length > 0)
{
IIDExportObject ExportConfig = new IIDExportObject();
var serializer = new XmlSerializer(ExportConfig.GetType());
//StringBuilder StringResult = new StringBuilder(XMLStored);
// Load the XML from the Query into the StringBuilder
// Now we need to build a Stream from the String to use in the XMLReader
byte[] byteArray = Encoding.UTF8.GetBytes(XMLStored);
MemoryStream stream = new MemoryStream(byteArray);
using (var reader = XmlReader.Create(stream))
{
ExportConfig = (IIDExportObject)serializer.Deserialize(reader);
}
}

John - thank you very much for the comment! It allowed me to complete the code and find a solution.
As you noted - using a stream reader was the solution - but I could not read the first line because there was only one 'line' in my string. However, I could use the line
using (StreamReader sr = new StreamReader(stream, false))
Which allows me to read the stream and ignore the "Byte Order Mark Detection" set to false.
string XMLStored = MainFormRef.GetExportConfigForCurrentQuery();
if (XMLStored.Length > 0)
{
IIDExportObject ExportConfig = new IIDExportObject();
try
{
var serializer = new XmlSerializer(ExportConfig.GetType());
// Now we need to build a Stream from the String to use in the XMLReader
byte[] byteArray = Encoding.UTF8.GetBytes(XMLStored);
MemoryStream stream = new MemoryStream(byteArray);
// Now we need to use a StreamReader to get around UTF8 vs UTF16 issues
// A little cumbersome - but it works
using (StreamReader sr = new StreamReader(stream, false))
{
using (var reader = XmlReader.Create(sr))
{
ExportConfig = (IIDExportObject)serializer.Deserialize(reader);
}
}
}
catch
{
}
I am not sure this is the best solution - but it works. I will be curious to see if anyone else has a better way of dealing with this.

Thanks to G Bradley, I took his answer and generalized it a bit to make it a bit easier to call.
public static string SerializeToXmlString<T>(T objectToSerialize)
{
XmlSerializer serializer = new XmlSerializer(typeof(T));
XmlWriterSettings settings = new XmlWriterSettings();
settings.Indent = false;
settings.Encoding = Encoding.UTF8;
StringBuilder builder = new StringBuilder();
using (XmlWriter writer = XmlWriter.Create(builder, settings))
{
serializer.Serialize(writer, objectToSerialize);
}
return builder.ToString();
}
public static T DeserializeFromXmlString<T>(string xmlString)
{
if (string.IsNullOrWhiteSpace(xmlString))
return default;
var serializer = new XmlSerializer(typeof(T));
byte[] byteArray = Encoding.UTF8.GetBytes(xmlString);
MemoryStream stream = new MemoryStream(byteArray);
using (StreamReader sr = new StreamReader(stream, false))
{
using (var reader = XmlReader.Create(sr))
{
return (T)serializer.Deserialize(reader);
}
}
}

Convert LINQ result to CSV or Excel

Using a LINQ query I need to export to Excel when a WebApi method is called. I have built the LINQ query that will return the correct data, now I need it to export to .csv or Excel file format.
I have tried using MemoryStream and StreamWriter but I think I am just chasing my tail now.
[HttpGet]
[Route("Download")]
public Task<IActionResult> Download(int memberId)
{
var results = (from violations in _db.tblMappViolations
where violations.MemberID == memberId
select new IncomingViolations
{
Contact = violations.ContactName,
Address = violations.str_Address,
City = violations.str_City,
State = violations.str_State,
Zip = violations.str_Zipcode,
Country = violations.str_Country,
Phone = violations.str_Phone,
Email = violations.str_Email,
Website = violations.str_WebSite,
}).FirstOrDefault();
MemoryStream stream = new MemoryStream(results);
StreamWriter writer = new StreamWriter(stream);
writer.Flush();
stream.Position = 0;
FileStreamResult response = File(stream, "application/octet-stream");
response.FileDownloadName = "violations.csv";
return response;
}

Here is how you can send CSV file to the user from server.
string attachment = "attachment; filename=MyCsvLol.csv";
HttpContext.Current.Response.Clear();
HttpContext.Current.Response.ClearHeaders();
HttpContext.Current.Response.ClearContent();
HttpContext.Current.Response.AddHeader("content-disposition", attachment);
HttpContext.Current.Response.ContentType = "text/csv";
HttpContext.Current.Response.AddHeader("Pragma", "public");
var sb = new StringBuilder();
// Add your data into stringbuilder
sb.Append(results.Contact);
sb.Append(results.Address);
sb.Append(results.City);
// and so on
HttpContext.Current.Response.Write(sb.ToString());
For Sending it from API
MemoryStream stream = new MemoryStream();
StreamWriter writer = new StreamWriter(stream);
// Write Your data here in writer
writer.Write("Hello, World!");
writer.Flush();
stream.Position = 0;
HttpResponseMessage result = new HttpResponseMessage(HttpStatusCode.OK);
result.Content = new StreamContent(stream);
result.Content.Headers.ContentType = new MediaTypeHeaderValue("text/csv");
result.Content.Headers.ContentDisposition = new ContentDispositionHeaderValue("attachment") { FileName = "Export.csv" };
return result;
Update:-
public HttpResponseMessage Download()
{
var results = (from violations in _db.tblMappViolations
where violations.MemberID == memberId
select new IncomingViolations
{
Contact = violations.ContactName,
Address = violations.str_Address,
City = violations.str_City,
State = violations.str_State,
Zip = violations.str_Zipcode,
Country = violations.str_Country,
Phone = violations.str_Phone,
Email = violations.str_Email,
Website = violations.str_WebSite,
});
var sb = new StringBuilder();
MemoryStream stream = new MemoryStream();
StreamWriter writer = new StreamWriter(stream);
foreach(var tempResult in results)
{
sb.Append(tempResult.Contact+",");
sb.Append(tempResult.Address+",");
sb.Append(tempResult.City+",");
sb.Append(tempResult.State+",");
sb.Append(tempResult.Zip+",");
sb.Append(tempResult.Country+",");
sb.Append(tempResult.Phone+",");
sb.Append(tempResult.Email+",");
sb.Append(tempResult.Website+",");
sb.Append(Enviroment.NewLine);
}
writer.Write(sb.ToString());
writer.Flush();
stream.Position = 0;
HttpResponseMessage result = new HttpResponseMessage(HttpStatusCode.OK);
result.Content = new StreamContent(stream);
result.Content.Headers.ContentType = new MediaTypeHeaderValue("text/csv");
result.Content.Headers.ContentDisposition = new ContentDispositionHeaderValue("attachment") { FileName = "Export.csv" };
return result;
}

First, to reuse the code in other areas, always create helper classes.
I adopted this method of converting list into a stream with headers as property names, if you want a file from this, essentially, I would just add another step to this:
STEP 1:
public static Stream ConvertToCSVStream<T>(IEnumerable<T> objects)
{
Type itemType = typeof(T);
var properties = itemType.GetProperties();
var mStream = new MemoryStream();
StreamWriter sWriter = new StreamWriter(mStream);
var values = objects.Select(o =>
{
return string.Join(",", properties.Select(p =>
{
var value = p.GetValue(o).ToString();
if (!Regex.IsMatch(value, "[,\"\\r\\n]"))
{
return value;
}
value = value.Replace("\"", "\"\"");
return string.Format("\"{0}\"", value);
})) + sWriter.NewLine;
});
var valuesInStrings = values.Aggregate((current, next) => current + next);
try
{
sWriter.Write(string.Join(",", properties.Select(x => x.Name.Replace("_", " "))) + sWriter.NewLine);
sWriter.Write(valuesInStrings);
}
catch (Exception e)
{
mStream.Close();
throw e;
}
sWriter.Flush();
mStream.Position = 0;
return mStream;
}
if your data is text, just convert it directly to a file result but if not, you must convert it to binary array and write it to stream, refer to this article for converting it to binary data, in our case, for csv, you could just use the FileStream result that you've implemented in a separate method:
STEP 2:
public FileStreamResult CreateFile(MemoryStream mStream, string path, string name)
{
//set values, names, content type, etc
//return filestream
}
or any other method you find better.

Save your result in a DataTable and then just use this
XLWorkbook workbook = new XLWorkbook();
DataTable table = GetYourTable();
workbook.Worksheets.Add(table);
And you should definitely use stream writer for this if you know which file its going to write to from the start, else stream reader and then stream writer.

Writing to Filestream and copying to MemoryStream

I want to overwrite or create an xml file on disk, and return the xml from the function. I figured I could do this by copying from FileStream to MemoryStream. But I end up appending a new xml document to the same file, instead of creating a new file each time.
What am I doing wrong? If I remove the copying, everything works fine.
public static string CreateAndSave(IEnumerable<OrderPage> orderPages, string filePath)
{
if (orderPages == null || !orderPages.Any())
{
return string.Empty;
}
var xmlBuilder = new StringBuilder();
var writerSettings = new XmlWriterSettings
{
Indent = true,
Encoding = Encoding.GetEncoding("ISO-8859-1"),
CheckCharacters = false,
ConformanceLevel = ConformanceLevel.Document
};
using (var fs = new FileStream(filePath, FileMode.OpenOrCreate, FileAccess.ReadWrite))
{
try
{
XmlWriter xmlWriter = XmlWriter.Create(fs, writerSettings);
xmlWriter.WriteStartElement("PRINT_JOB");
WriteXmlAttribute(xmlWriter, "TYPE", "Order Confirmations");
foreach (var page in orderPages)
{
xmlWriter.WriteStartElement("PAGE");
WriteXmlAttribute(xmlWriter, "FORM_TYPE", page.OrderType);
var outBound = page.Orders.SingleOrDefault(x => x.FlightInfo.Direction == FlightDirection.Outbound);
var homeBound = page.Orders.SingleOrDefault(x => x.FlightInfo.Direction == FlightDirection.Homebound);
WriteXmlOrder(xmlWriter, outBound, page.ContailDetails, page.UserId, page.PrintType, FlightDirection.Outbound);
WriteXmlOrder(xmlWriter, homeBound, page.ContailDetails, page.UserId, page.PrintType, FlightDirection.Homebound);
xmlWriter.WriteEndElement();
}
xmlWriter.WriteFullEndElement();
MemoryStream destination = new MemoryStream();
fs.CopyTo(destination);
Log.Progress("Xml string length: {0}", destination.Length);
xmlBuilder.Append(Encoding.UTF8.GetString(destination.ToArray()));
destination.Flush();
destination.Close();
xmlWriter.Flush();
xmlWriter.Close();
}
catch (Exception ex)
{
Log.Warning(ex, "Unhandled exception occured during create of xml. {0}", ex.Message);
throw;
}
fs.Flush();
fs.Close();
}
return xmlBuilder.ToString();
}
Cheers
Jens

FileMode.OpenOrCreate is causing the file contents to be overwritten without shortening, leaving any 'trailing' data from previous runs. If FileMode.Create is used the file will be truncated first. However, to read back the contents you just wrote you will need to use Seek to reset the file pointer.
Also, flush the XmlWriter before copying from the underlying stream.
See also the question Simultaneous Read Write a file in C Sharp (3817477).
The following test program seems to do what you want (less your own logging and Order details).
using System;
using System.IO;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Xml;
using System.Threading.Tasks;
namespace ReadWriteTest
{
class Program
{
static void Main(string[] args)
{
string filePath = Path.Combine(
Environment.GetFolderPath(Environment.SpecialFolder.Personal),
"Test.xml");
string result = CreateAndSave(new string[] { "Hello", "World", "!" }, filePath);
Console.WriteLine("============== FIRST PASS ==============");
Console.WriteLine(result);
result = CreateAndSave(new string[] { "Hello", "World", "AGAIN", "!" }, filePath);
Console.WriteLine("============== SECOND PASS ==============");
Console.WriteLine(result);
Console.ReadLine();
}
public static string CreateAndSave(IEnumerable<string> orderPages, string filePath)
{
if (orderPages == null || !orderPages.Any())
{
return string.Empty;
}
var xmlBuilder = new StringBuilder();
var writerSettings = new XmlWriterSettings
{
Indent = true,
Encoding = Encoding.GetEncoding("ISO-8859-1"),
CheckCharacters = false,
ConformanceLevel = ConformanceLevel.Document
};
using (var fs = new FileStream(filePath, FileMode.Create, FileAccess.ReadWrite))
{
try
{
XmlWriter xmlWriter = XmlWriter.Create(fs, writerSettings);
xmlWriter.WriteStartElement("PRINT_JOB");
foreach (var page in orderPages)
{
xmlWriter.WriteElementString("PAGE", page);
}
xmlWriter.WriteFullEndElement();
xmlWriter.Flush(); // Flush from xmlWriter to fs
xmlWriter.Close();
fs.Seek(0, SeekOrigin.Begin); // Go back to read from the begining
MemoryStream destination = new MemoryStream();
fs.CopyTo(destination);
xmlBuilder.Append(Encoding.UTF8.GetString(destination.ToArray()));
destination.Flush();
destination.Close();
}
catch (Exception ex)
{
throw;
}
fs.Flush();
fs.Close();
}
return xmlBuilder.ToString();
}
}
}
For the optimizers out there, the StringBuilder was unnecessary because the string is formed whole and the MemoryStream can be avoided by just wrapping fs in a StreamReader. This would make the code as follows.
public static string CreateAndSave(IEnumerable<string> orderPages, string filePath)
{
if (orderPages == null || !orderPages.Any())
{
return string.Empty;
}
string result;
var writerSettings = new XmlWriterSettings
{
Indent = true,
Encoding = Encoding.GetEncoding("ISO-8859-1"),
CheckCharacters = false,
ConformanceLevel = ConformanceLevel.Document
};
using (var fs = new FileStream(filePath, FileMode.Create, FileAccess.ReadWrite))
{
try
{
XmlWriter xmlWriter = XmlWriter.Create(fs, writerSettings);
xmlWriter.WriteStartElement("PRINT_JOB");
foreach (var page in orderPages)
{
xmlWriter.WriteElementString("PAGE", page);
}
xmlWriter.WriteFullEndElement();
xmlWriter.Close(); // Flush from xmlWriter to fs
fs.Seek(0, SeekOrigin.Begin); // Go back to read from the begining
var reader = new StreamReader(fs, writerSettings.Encoding);
result = reader.ReadToEnd();
// reader.Close(); // This would just flush/close fs early(which would be OK)
}
catch (Exception ex)
{
throw;
}
}
return result;
}

I know I'm late, but there seems to be a simpler solution. You want your function to generate xml, write it to a file and return the generated xml. Apparently allocating a string cannot be avoided (because you want it to be returned), same for writing to a file. But reading from a file (as in your and SensorSmith's solutions) can easily be avoided by simply "swapping" the operations - generate xml string and write it to a file. Like this:
var output = new StringBuilder();
var writerSettings = new XmlWriterSettings { /* your settings ... */ };
using (var xmlWriter = XmlWriter.Create(output, writerSettings))
{
// Your xml generation code using the writer
// ...
// You don't need to flush the writer, it will be done automatically
}
// Here the output variable contains the xml, let's take it...
var xml = output.ToString();
// write it to a file...
File.WriteAllText(filePath, xml);
// and we are done :-)
return xml;
IMPORTANT UPDATE: It turns out that the XmlWriter.Create(StringBuider, XmlWriterSettings) overload ignores the Encoding from the settings and always uses "utf-16", so don't use this method if you need other encoding.

Simultaneously writing and validating XML

I have a Write method that serializes objects which use XmlAttributes. It's pretty standard like so:
private bool WriteXml(DirectoryInfo dir)
{
var xml = new XmlSerializer(typeof (Composite));
_filename = Path.Combine(dir.FullName, _composite.Symbol + ".xml");
using (var xmlFile = File.Create(_filename))
{
xml.Serialize(xmlFile, _composite);
}
return true;
}
Apart from trying to read the file I have just written out (with a Schema validator), can I perform XSD validation WHILE the XML is being written?
I can mess around with memory streams before writing it to disk, but it seems in .Net there is usually an elegant way of solving most problems.

The way I've done it is like this for anyone interested:
public Composite Read(Stream stream)
{
_errors = null;
var settings = new XmlReaderSettings();
using (var fileStream = File.OpenRead(XmlComponentsXsd))
{
using (var schemaReader = new XmlTextReader(fileStream))
{
settings.Schemas.Add(null, schemaReader);
settings.ValidationType = ValidationType.Schema;
settings.ValidationEventHandler += OnValidationEventHandler;
using (var xmlReader = XmlReader.Create(stream, settings))
{
var serialiser = new XmlSerializer(typeof (Composite));
return (Composite) serialiser.Deserialize(xmlReader);
}
}
}
}
private ValidationEventArgs _errors = null;
private void OnValidationEventHandler(object sender, ValidationEventArgs validationEventArgs)
{
_errors = validationEventArgs;
}
Then instead of writing the XML to file, using a memory stream do something like:
private bool WriteXml(DirectoryInfo dir)
{
var xml = new XmlSerializer(typeof (Composite));
var filename = Path.Combine(dir.FullName, _composite.Symbol + ".xml");
// first write it to memory
var memStream = new MemoryStream();
xml.Serialize(memStream, _composite);
memStream.Position = 0;
Read(memStream);
if (_errors != null)
{
throw new Exception(string.Format("Error writing to {0}. XSD validation failed : {1}", filename, _errors.Message));
}
memStream.Position = 0;
using (var outFile = File.OpenWrite(filename))
{
memStream.CopyTo(outFile);
}
memStream.Dispose();
return true;
}
That way you're always validating against the schema before anything is written to disk.