We Keep Coding

How to handle invalid XML response with utf-16 header over a utf-8 stream

I'm getting the following error in when trying to read some XML.
Exception has occurred: CLR/System.Xml.XmlException
Exception thrown: 'System.Xml.XmlException' in System.Private.Xml.dll: 'There is no Unicode byte order mark. Cannot switch to Unicode.'
I've identified this as the API is serving the content as utf-8 but the header is utf-16.
<?xml version="1.0" encoding="utf-16"?>
I've confirmed this in tests from static files by deleting the encoding or saving the file in utf-16. I have also confirmed that the incoming response is utf-8 looking in the response Content.Headers.ContentType.
Unfortunately I don't maintain the API and don't think that this will be getting fixed any time soon.
Is there a way to make a System.Text.XmlReader ignore the header in the stream, would be nice if there were a flag to simply ignore the doctype if they can't be bothered to make it accurate?
I think you can correct the content of XML using some kind of Schema replacement prior to final parsing?
I could always think about re-encoding the same content but it seems a little mad.
var mockBytes = System.Text.Encoding.UTF8.GetBytes("<?xml version=\"1.0\" encoding=\"utf-16\"?>");
var mockStream = MemoryStream new(mockBytes);
XmlReaderSettings settings = new XmlReaderSettings();
settings.Async = true;
using (var reader = XmlReader.Create(mockStream, settings))
{
if (reader.ReadToFollowing("Message") & await reader.ReadAsync())
{
while (await reader.MoveToContentAsync() == XmlNodeType.Element)
{
...
}
}
}

Thank you for the comments. Using them I have been able to test that simply instantiating and passing a StreamReader is all that is required to stop the XmlReader interpreting the encoding meta in the document type definition.
var mockBytes = System.Text.Encoding.UTF8.GetBytes("<?xml version=\"1.0\" encoding=\"utf-16\"?>");
var mockStream = MemoryStream new(mockBytes);
var sr = new StreamReader(mockStream);
XmlReaderSettings settings = new XmlReaderSettings();
settings.Async = true;
using (var reader = XmlReader.Create(sr, settings))
{
if (reader.ReadToFollowing("Message") & await reader.ReadAsync())
{
while (await reader.MoveToContentAsync() == XmlNodeType.Element)
{
...
}
}
}
This is the simplest solution I can imagine other than a flag on the XmlReaderSettings that doesn't seem to exist.
Furthermore, as #Jereon says, skipping to particular characters or line endings would get very brittle and fall over if some other change happened at the API. You would really have to try and look more carefully, perhaps pushing elements into a stack between <? + ?> not easy and also fortunately not necessary.

Deleting element from Xml breaks format on reload

I am creating a system that stores vehicle data. When I serialize the data using Xml serialization, I get the correct format as shown in the example below:
<?xml version="1.0"?>
<ArrayOfVehicle xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<Vehicle>
<Registration>fake1</Registration>
<Model>123</Model>
<Make>test</Make>
<Year>1999</Year>
<Cost>100</Cost>
</Vehicle>
<Vehicle>
<Registration>fake2</Registration>
<Model>321</Model>
<Make>123</Make>
<Year>2000</Year>
<Cost>321</Cost>
</Vehicle>
</ArrayOfVehicle>
The serialization uses a list of vehicles that have the attributes seen in the Xml file. I am trying to figure out how I can delete a vehicle from the list and serialize it back to the Xml file without breaking the format shown above.
The method that I have tried to use to delete the records from the list and serialize and deserialize the data, but when I remove and item, it breaks the format. This is what the Xml file looks like when I remove an item from the list and serialize it:
fake1 123 test 1999 100
Here is my code for removing an item:
for (int i = Business.VehicleList.Count - 1; i >= 0; i--)
{ //Where Business.VehicleList is my list
if (Business.VehicleList[i].Registration == registration)
{
Business.VehicleList.RemoveAt(i);
Business.Save(); //Method for serialization
}
}
Here is the error it throws when I try to deserialize the data again:
System.InvalidOperationException: 'There is an error in XML document (10, 19). XmlException: There are multiple root elements. Line 10, position 19.'
These are my serialization and deserialization methods:
public static void Retrieve()
{
using (FileStream fileStream = new FileStream("C:\\temp\\data.xml", FileMode.OpenOrCreate))
{
using (var reader = new StreamReader(fileStream))
{
if (fileStream.Length <= 0)
{
return;
}
else
{
XmlSerializer deserializer = new XmlSerializer(typeof(List<Vehicle>),
new XmlRootAttribute("ArrayOfVehicle"));
_vehicleList = (List<Vehicle>)deserializer.Deserialize(reader); //This is where the error is thrown
}
}
}
}
public static void Save()
{
XmlSerializer serializer = new XmlSerializer(typeof(List<Vehicle>));
using (FileStream fileStream = new FileStream("C:\\temp\\data.xml", FileMode.Open))
{
serializer.Serialize(fileStream, VehicleList);
fileStream.Close();
}
}
Any suggestions on how to remove a vehicle from my list without it breaking the Xml file?
Here is the source after I tried deleting an item from the vehicle string
<?xml version="1.0"?>
<ArrayOfVehicle xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
<Vehicle>
<Registration>123</Registration>
<Model>123</Model>
<Make>23</Make>
<Year>2000</Year>
<Cost>123</Cost>
</Vehicle>
</ArrayOfVehicle><Registration>1321</Registration>
<Model>123123</Model>
<Make>312312</Make>
<Year>2000</Year>
<Cost>321</Cost>
</Vehicle>
</ArrayOfVehicle>

In the Save method, new FileStream("C:\\temp\\data.xml", FileMode.Open) will open the existing file without truncating it. So after you write the new XML data to the file, there will be remnants of the old content if the new content is shorter than the old one.
Changing this to new FileStream("C:\\temp\\data.xml", FileMode.Create) will fix the issue.

I think it's because you are trying to de-serialize a malformed xml. Please first, make sure that your serialization method produces correct xml. The reason may be because of closing the stream inside using statement. And also serializing the list before for-loop finishes.
Try removing fileStream.Close(); and also moving Business.Save(); to outside of for-loop.
Here, I made a fiddle with same conditions and it works.

API that returns XML. Can I use XmlTextWriter and would this be faster than creating an XmlDocument

Currently writing an .ashx file that returns some XML. Here is what I am essentially doing in the ProcessRequest sub
Dim xml As XmlDocument = getXML(context.Request.QueryString("products"))
xml.Save(context.Response.Output)
I was wondering if I could also somehow use an XmlTextWriter and if so would this be better/faster?
Speed is by far the most important factor but I'm very new to programming in VB.net so if there's anything else I should know that would be useful please let me know.
Xml content incase anyone is interested...It's this but with the possibility of returning up to 46 products.
<?xml version="1.0" encoding="utf-8"?>
<products>
<product>
<id>58</id>
<prices />
<instock>True</instock>
<shipping> - This product will ship today if ordered within the next 2 hours and 46 minutes.</shipping>
</product>
<product>
<id>59</id>
<prices />
<instock>False</instock>
<shipping>This product will be in stock soon</shipping>
</product>
</products>

I don't think that the first concern is how you serialize the XML. A maximum amount of 46 products does not seem to be a very large amount of data, so the effects of serializing faster or handling memory more efficiently are not too big.
Instead, I'd propose to cache the output if possible. If the data are not user-dependent and do not change to frequently, you could store the result in the cache and serve it from there. Instead of storing the XmlDocument in cache, store the serialized version so that you only need to write the string to the response. The following sample shows how to use the cache:
public void ProcessRequest(HttpContext context)
{
string productsKey = context.Request.QueryString("products");
string cacheKey = "Products_" + productsKey;
if (context.Cache[cacheKey] == null)
{
// lockObj declared on class level as
// private static readonly lockObj = new object();
lock(lockObj) {
if (context.Cache[cacheKey] == null)
{
using(StringWriter writer = new StringWriter())
{
var doc = getXml(productskey);
doc.Save(writer);
// Set caching options as required
context.Cache.Add(cacheKey, writer.GetStringBuilder().ToString(), null, Cache.NoAbsoluteExpiration, TimeSpan.FromMinutes(10));
}
}
}
context.Response.Write(context.Cache[cacheKey]);
return;
}
For details on the Cache class see this link.

xml.LoadData - Data at the root level is invalid. Line 1, position 1

I'm trying to parse some XML inside a WiX installer. The XML would be an object of all my errors returned from a web server. I'm getting the error in the question title with this code:
XmlDocument xml = new XmlDocument();
try
{
xml.LoadXml(myString);
}
catch (Exception ex)
{
System.IO.File.WriteAllText(#"C:\text.txt", myString + "\r\n\r\n" + ex.Message);
throw ex;
}
myString is this (as seen in the output of text.txt)
<?xml version="1.0" encoding="utf-8"?>
<Errors></Errors>
text.txt comes out looking like this:
<?xml version="1.0" encoding="utf-8"?>
<Errors></Errors>
Data at the root level is invalid. Line 1, position 1.
I need this XML to parse so I can see if I had any errors.

The hidden character is probably BOM.
The explanation to the problem and the solution can be found here, credits to James Schubert, based on an answer by James Brankin found here.
Though the previous answer does remove the hidden character, it also removes the whole first line. The more precise version would be:
string _byteOrderMarkUtf8 = Encoding.UTF8.GetString(Encoding.UTF8.GetPreamble());
if (xml.StartsWith(_byteOrderMarkUtf8))
{
xml = xml.Remove(0, _byteOrderMarkUtf8.Length);
}
I encountered this problem when fetching an XSLT file from Azure blob and loading it into an XslCompiledTransform object.
On my machine the file looked just fine, but after uploading it as a blob and fetching it back, the BOM character was added.

Use Load() method instead, it will solve the problem. See more

The issue here was that myString had that header line. Either there was some hidden character at the beginning of the first line or the line itself was causing the error. I sliced off the first line like so:
xml.LoadXml(myString.Substring(myString.IndexOf(Environment.NewLine)));
This solved my problem.

I Think that the problem is about encoding. That's why removing first line(with encoding byte) might solve the problem.
My solution for Data at the root level is invalid. Line 1, position 1.
in XDocument.Parse(xmlString) was replacing it with XDocument.Load( new MemoryStream( xmlContentInBytes ) );
I've noticed that my xml string looked ok:
<?xml version="1.0" encoding="utf-8"?>
but in different text editor encoding it looked like this:
?<?xml version="1.0" encoding="utf-8"?>
At the end i did not need the xml string but xml byte[]. If you need to use the string you should look for "invisible" bytes in your string and play with encodings to adjust the xml content for parsing or loading.
Hope it will help

Save your file with different encoding:
File > Save file as... > Save as UTF-8 without signature.
In VS 2017 you find encoding as a dropdown next to Save button.

Main culprit for this error is logic which determines encoding when converting Stream or byte[] array to .NET string.
Using StreamReader created with 2nd constructor parameter detectEncodingFromByteOrderMarks set to true, will determine proper encoding and create string which does not break XmlDocument.LoadXml method.
public string GetXmlString(string url)
{
using var stream = GetResponseStream(url);
using var reader = new StreamReader(stream, true);
return reader.ReadToEnd(); // no exception on `LoadXml`
}
Common mistake would be to just blindly use UTF8 encoding on the stream or byte[]. Code bellow would produce string that looks valid when inspected in Visual Studio debugger, or copy-pasted somewhere, but it will produce the exception when used with Load or LoadXml if file is encoded differently then UTF8 without BOM.
public string GetXmlString(string url)
{
byte[] bytes = GetResponseByteArray(url);
return System.Text.Encoding.UTF8.GetString(bytes); // potentially exception on `LoadXml`
}

I've solved this issue by directly editing the byte array.
Collect the UTF8 preamble and remove directly the header.
Afterward you can transform the byte[]to a string with GetString method, see below.
The \r and \t I've removed as well, just as precaution.
XmlDocument configurationXML = new XmlDocument();
List<byte> byteArray = new List<byte>(webRequest.downloadHandler.data);
foreach(byte singleByte in Encoding.UTF8.GetPreamble())
{
byteArray.RemoveAt(byteArray.IndexOf(singleByte));
}
string xml = System.Text.Encoding.UTF8.GetString(byteArray.ToArray());
xml = xml.Replace("\\r", "");
xml = xml.Replace("\\t", "");

If your xml is in a string use the following to remove any byte order mark:
xml = new Regex("\\<\\?xml.*\\?>").Replace(xml, "");

At first I had problems escaping the "&" character, then diacritics and special letters were shown as question marks and ended up with the issue OP mentioned.
I looked at the answers and I used #Ringo's suggestion to try Load() method as an alternative. That made me realize that I can deal with my response in other ways not just as a string.
using System.IO.Stream instead of string solved all the issues for me.
var response = await this.httpClient.GetAsync(url);
var responseStream = await response.Content.ReadAsStreamAsync();
var xmlDocument = new XmlDocument();
xmlDocument.Load(responseStream);
The cool thing about Load() is that this method automatically detects the string format of the input XML (for example, UTF-8, ANSI, and so on). See more

I have found out one of the solutions.
For your code this could be as follows -
XmlDocument xml = new XmlDocument();
try
{
// assuming the location of the file is in the current directory
// assuming the file name be loadData.xml
string myString = "./loadData.xml";
xml.Load(myString);
}
catch (Exception ex)
{
System.IO.File.WriteAllText(#"C:\text.txt", myString + "\r\n\r\n" + ex.Message);
throw ex;
}

if we are using XDocument.Parse(#"").
Use # it resolves the issue.

Using an XmlDataDocument object is much better than using an XDocument or XmlDocument object. XmlDataDocument works fine with UTF8 and it doesn't have problems with Byte Order Sequences. You can get the child nodes of each element using ChildNodes property.
Use a custom function such as the following one:
static public void ReadXmlDataDocument2(string xmlFilePath)
{
if (xmlFilePath != null)
{
if (File.Exists(xmlFilePath))
{
System.IO.FileStream fs = default(System.IO.FileStream);
try
{
fs = new System.IO.FileStream(xmlFilePath, System.IO.FileMode.Open, System.IO.FileAccess.Read);
System.Xml.XmlDataDocument k_XDoc = new System.Xml.XmlDataDocument();
k_XDoc.Load(fs);
fs.Close();
fs.Dispose();
fs = null;
XmlNodeList ndsRoot = k_XDoc.ChildNodes;
foreach (System.Xml.XmlNode xLog in ndsRoot)
{
foreach (System.Xml.XmlNode xLog2 in xLog.ChildNodes)
{
if (xLog2.Name == "ERRORs")
{
foreach (System.Xml.XmlNode xLog3 in xLog2.ChildNodes)
{
if (xLog3.Name == "ErrorCode")
{
// Do something
}
if (xLog3.Name == "Description")
{
// Do something
}
}
}
}
}
}
catch (Exception ex)
{
MessageBox.Show(ex.Message);
}
}
}
}

XmlDocument in WCF Service method not successfully saving to file using the class' save method

Hello and thanks in advance,
I am attempting to take the input from text boxes in a silverlight application and on an event fired by a button click, convert them to an xml string, pass the string and a specified file name to a WCF service call and in that call save the xml to the specifed file(via a string parameter). The code which captures the text into an xml string seems to be successfully working(based on what I see in the variables when debugging) and looks like this:
private void ServerInfoNext_Click(object sender, RoutedEventArgs e)
{
//new RegisterServerGroupObject instance
RegisterServerGroupObject groupInfo= new RegisterServerGroupObject(groupNameTB.Text,1,parentServerNameTB.Text,LeaderNameCB.SelectedItem.ToString());
var serializer = new XmlSerializer(typeof(RegisterServerGroupObject));
XmlSerializerNamespaces ns = new XmlSerializerNamespaces();
ns.Add("","");
XmlWriterSettings settings = new XmlWriterSettings();
settings.Encoding = Encoding.UTF8;
settings.Indent = true;
settings.CloseOutput = true;
StringBuilder sb = new StringBuilder();
using (XmlWriter writer = XmlWriter.Create(sb,settings))
{
serializer.Serialize(writer, groupInfo);
writer.Close();
}
//sb now contains the xml string with the information from the serialized class
string contentsString = sb.ToString();
//create instance of XmlWrite service
XMLWriteServiceClient xmlClient = new XMLWriteServiceClient();
xmlClient.WriteXmlToServerCompleted += new EventHandler<System.ComponentModel.AsyncCompletedEventArgs>(xmlClient_WriteXmlToServerCompleted);
xmlClient.WriteXmlToServerAsync("ServerGroups.xml", contentsString);
}
at this point when the variable contents string is passed to the service method, I can see that it has valid xml, as well as within the service method itself, which looks like this:
public class XMLWriteService : IXMLWriteService
{
public void WriteXmlToServer(string filename,string xmlString)
{
XmlDocument xDoc = new XmlDocument();
xDoc.LoadXml(xmlString.ToString());
try
{
xDoc.Save(filename);
}
catch (FileNotFoundException e)
{
Console.WriteLine(e.InnerException.ToString());
}
}
}
The try/catch block is not indicating that the file("ServerGroups.xml") is not found, and I currently have that xml file in the ClientBin of the server side portion of the project. (the .Web side). However, after the method terminates there is no new xml written to the file. Can someone please tell me what I am doing wrong? I don't know why the XmlDocument class instance is not saving its contents to the file. Thanks in advance!

You aren't passing a path, so it's just going to save the file to the current directory of the WCF service process, whatever that happens to be. Either find out what that is, or do a search on your whole server drive for that file name to see where it's saving it. Better yet, call Path.Combine to append a path to the begining of the file name before you save to it. For instance:
xDoc.Save(Path.Combine("C:\\ClientBin", filename));
To answer your question in the comment below, if you want to append the incoming XML data to the data that is already stored in the XML file on the server, that's a bit more involved. It all depends what the format of the XML is. Since you are using serialization, which by default will only allow one object per XML document (because it puts the object name as the root document element, of which there can only be one), then you would have to have a different XML format. For instance, on the server side, you would need to have some kind of root element on the document under which you could keep appending the incoming RegisterServerGroupObject objects. For instance, if your XML file on the server looked like this:
<?xml version="1.0" encoding="utf-8" ?>
<ListOfRegisterServerGroupObject>
</ListOfRegisterServerGroupObject>
Then, you could append the data by inserting new elements within that root element, like this:
<?xml version="1.0" encoding="utf-8" ?>
<ListOfRegisterServerGroupObject>
<RegisterServerGroupObject>
...
</RegisterServerGroupObject>
<RegisterServerGroupObject>
...
</RegisterServerGroupObject>
...
</ListOfRegisterServerGroupObject>
To do this, you would need to first load the XML document, then get the root element, then append the incoming XML as a child element. For instance:
public void WriteXmlToServer(string filename, string xmlString)
{
string filePath = Path.Combine("C:\\ClientBin", filename);
XmlDocument storage = New XmlDocument();
storage.Load(filePath);
XmlDocument incoming = New XmlDocument();
incoming.LoadXml(xmlString);
storage.DocumentElement.AppendChild(incoming.DocumentElement);
storage.Save(filePath);
}

You may need to 'map' the physical path to the output file within the service
string path = HostingEnvironment.MapPath("~/MyPath/MyFile.xml");