We Keep Coding

JsonConvert.Deserialize with truncated JSON? [duplicate]

My json file is mostly an array that contain objects but the list is incomplete, so I can't use the last entry. I would like to deserialize the rest of the file while discarding the last invalid entry
[ { "key" : "value1" }, { "key " : "value2"}, { "key
Please tell me if there is a way using Newtonsoft.Json library, or do I need some preprocessing.
Thank you!

Looks like on Json.NET 8.0.3 you can stream your string from a JsonTextReader to a JTokenWriter and get a partial result by catching and swallowing the JsonReaderException that gets thrown when parsing the truncated JSON:
JToken root;
string exceptionPath = null;
using (var textReader = new StringReader(badJson))
using (var jsonReader = new JsonTextReader(textReader))
using (JTokenWriter jsonWriter = new JTokenWriter())
{
try
{
jsonWriter.WriteToken(jsonReader);
}
catch (JsonReaderException ex)
{
exceptionPath = ex.Path;
Debug.WriteLine(ex);
}
root = jsonWriter.Token;
}
Console.WriteLine(root);
if (exceptionPath != null)
{
Console.WriteLine("Error occurred with token: ");
var badToken = root.SelectToken(exceptionPath);
Console.WriteLine(badToken);
}
This results in:
[
{
"key": "value1"
},
{
"key ": "value2"
},
{}
]
You could then finish deserializing the partial object with JToken.ToObject. You could also delete the incomplete array entry by using badToken.Remove().
It would be better practice not to generate invalid JSON in the first place though. I'm also not entirely sure this is documented functionality of Json.NET, and thus it might not work with future versions of Json.NET. (E.g. conceivably Newtonsoft could change their algorithm such that JTokenWriter.Token is only set when writing is successful.)

You can use the JsonReader class and try to parse as far as you get. Something like the code below will parse as many properties as it gets and then throw an exception. This is of course if you want to deserialize into a concrete class.
public Partial FromJson(JsonReader reader)
{
while (reader.Read())
{
// Break on EndObject
if (reader.TokenType == JsonToken.EndObject)
break;
// Only look for properties
if (reader.TokenType != JsonToken.PropertyName)
continue;
switch ((string) reader.Value)
{
case "Id":
reader.Read();
Id = Convert.ToInt16(reader.Value);
break;
case "Name":
reader.Read();
Name = Convert.ToString(reader.Value);
break;
}
}
return this;
}
Code taken from the CGbR JSON Target.

the second answer above is really good and simple, helped me out!
static string FixPartialJson(string badJson)
{
JToken root;
string exceptionPath = null;
using (var textReader = new StringReader(badJson))
using (var jsonReader = new JsonTextReader(textReader))
using (JTokenWriter jsonWriter = new JTokenWriter())
{
try
{
jsonWriter.WriteToken(jsonReader);
}
catch (JsonReaderException ex)
{
exceptionPath = ex.Path;
}
root = jsonWriter.Token;
}
return root.ToString();
}

How to get data out of xml XDocument in C#?

I have an XML file that I get via a web service. File looks like this:
<Z_STU_CRS_TRNS_DOC xmlns="http://testurl">
<Z_STATUS_CODE>0</Z_STATUS_CODE>
<Z_STATUS_MSG>Success</Z_STATUS_MSG>
<Z_STUDENT_ID_SUB_DOC xmlns="http://testurl">
<Z_STU_ID>000999999</Z_STU_ID>
</Z_STUDENT_ID_SUB_DOC>
<Z_CRSE_SUB_DOC xmlns="http://testurl">
<Z_COURSE xmlns="http://testurl">
<Z_CRSE_DATA>9999|199901|TEST|9999|1|S|Scuba Diving| |XX</Z_CRSE_DATA>
</Z_COURSE>
<Z_COURSE xmlns="testurl">
<Z_CRSE_DATA>9999|200001|TEST|999|3|A|English 101| |XX</Z_CRSE_DATA>
</Z_COURSE>
</Z_CRSE_SUB_DOC>
</Z_STU_CRS_TRNS_DOC>
I'm able to consume the service and check for errors but I'm having a tough time actually getting the data out of the XDocument xml file.
protected void webClient_DownloadStringCompleted(object sender,
DownloadStringCompletedEventArgs e)
{
if (e.Error != null)
{
errorLabel.Text =
"The transaction failed. If you feel that you have reached " +
"this in error, please contact the help desk at xxx-xxx-xxxx.";
errorLabel.Visible = true;
return;
}
XDocument xml = XDocument.Parse(e.Result);
XNamespace ns = "http://testurl";
//Look for error code from WS
var field = xml.Descendants(ns + "Z_STATUS_CODE").FirstOrDefault();
if (field != null)
{
if (Convert.ToInt32((string)field.Value) == 1)
{
errorLabel.Text =
"The transaction failed, due to an invalid student id. If you " +
"feel that you have reached this in error, please contact " +
"the help desk at xxx-xxx-xxxx.";
errorLabel.Visible = true;
return;
}
}
I tried many different ways but nothing seems to help. Can someone help?

I figured it out! Posting in case someone else has similar issue.
List<studentRecord> studentCourses = new List<studentRecord>();
XmlReader reader = xml.CreateReader();
// Get elements
while (reader.Read())
{
if ((reader.NodeType == XmlNodeType.Element) && (reader.Name == "Z_CRSE_DATA"))
{
reader.Read();
if (reader.NodeType == XmlNodeType.Text)
{
studentRecord stuRec = new studentRecord();
stuRec.rawData = reader.Value;
studentCourses.Add(stuRec);
}
}
}
reader.Close();

You could read the node list, and read the values of the nodes according to their name.
It takes a bit more work, as you have to work through all the nodes itself, but like this you could do it
class Program
{
static void Main(string[] args)
{
string xmldata = #"<Z_STU_CRS_TRNS_DOC xmlns=""http://testurl"">
<Z_STATUS_CODE>0</Z_STATUS_CODE>
<Z_STATUS_MSG>Success</Z_STATUS_MSG>
<Z_STUDENT_ID_SUB_DOC xmlns=""http://testurl"">
<Z_STU_ID>000999999</Z_STU_ID>
</Z_STUDENT_ID_SUB_DOC>
<Z_CRSE_SUB_DOC xmlns=""http://testurl"">
<Z_COURSE xmlns=""http://testurl"">
<Z_CRSE_DATA>9999|199901|TEST|9999|1|S|Scuba Diving| |XX</Z_CRSE_DATA>
</Z_COURSE>
<Z_COURSE xmlns=""testurl"">
<Z_CRSE_DATA>9999|200001|TEST|999|3|A|English 101| |XX</Z_CRSE_DATA>
</Z_COURSE>
</Z_CRSE_SUB_DOC>
</Z_STU_CRS_TRNS_DOC>";
string errorTag = "Z_STATUS_CODE",
statusTag = "Z_STATUS_MSG";
XDocument xml = XDocument.Parse(xmldata);
XNamespace ns = "http://testurl";
int errorCode = -1;
string statusMessage = string.Empty;
using (XmlReader reader = xml.CreateReader())
{
while (reader.Read())
{
if (reader.NodeType != XmlNodeType.Element)
{
continue;
}
if (!string.Equals(reader.Name, errorTag) &&
!string.Equals(reader.Name, statusTag))
{
continue;
}
string currentName = reader.Name;
while (reader.Read())
{
if (reader.NodeType == XmlNodeType.EndElement)
{
break;
}
if (reader.NodeType != XmlNodeType.Text)
{
continue;
}
if (string.Equals(currentName, errorTag))
{
errorCode = int.Parse(reader.Value);
}
if (string.Equals(currentName, statusTag))
{
statusMessage = reader.Value;
}
break;
}
}
}
if (errorCode == -1)
{
// no tag found
Console.WriteLine("No tag found named: {0}", errorTag);
}
else if (errorCode == 0)
{
Console.WriteLine("Operation was a success!");
}
else
{
Console.WriteLine("Operation failed with error code {0}", errorCode);
}
if (!string.IsNullOrWhiteSpace(statusMessage))
{
Console.WriteLine("Status message: {0}", statusMessage);
}
Console.ReadLine();
}
}

Unexpected SignatureConfirmation in IssuedTokenOverTransport response

I am building a WCF client to access a vendor web service. The service uses IssuedTokenOverTransport, SymmetricKey, and expects SAML. I have the request working, but the response back from the service includes a SignatureConfirmation element in the WS-Security header. My C# client chokes on this "signature confirmation is not expected in the security header" and I don't see a way to either ignore or process this element. It seems the only way to get near the handling of SignatureConfirmation in WCF is to abandon the IssuedTokenOverTransport binding and use something else, but this doesn't seem to be an option because the service requires this binding type. Is this a bug in WCF?

I solved this by using a custom Message Encoder. See this article from Carlos Figueroa for background:
http://blogs.msdn.com/b/carlosfigueira/archive/2011/11/09/wcf-extensibility-message-encoders.aspx
Essentially, the encoder can look for the SignatureConfirmation element in the incoming message and remove it from the header. The key piece of code is this private method that is called from the ReadMessage overrides:
private MemoryStream ProcessMemoryStream(Stream inputStream, bool dispose)
{
StreamWriter xmlStream = null;
var outputStream = new MemoryStream();
bool continueFilter = false;
try
{
xmlStream = new StreamWriter(outputStream);
using (var reader = XmlReader.Create(inputStream))
{
using (
var writer = XmlWriter.Create(xmlStream,
new XmlWriterSettings() {ConformanceLevel = ConformanceLevel.Auto}))
{
while (reader.Read())
{
if (reader.LocalName.Equals("SignatureConfirmation") &&
reader.NamespaceURI.Equals(
"http://docs.oasis-open.org/wss/oasis-wss-wssecurity-secext-1.1.xsd"))
{
if (!reader.IsEmptyElement) continueFilter = reader.IsStartElement();
}
else if (reader.LocalName.Equals("Signature") &&
reader.NamespaceURI.Equals("http://www.w3.org/2000/09/xmldsig#"))
{
if (!reader.IsEmptyElement) continueFilter = reader.IsStartElement();
}
else if (continueFilter)
{
// continue to next node
}
else
XmlHelper.WriteShallowNode(reader, writer);
}
writer.Flush();
}
reader.Close();
}
outputStream.Position = 0;
return outputStream;
}
catch (Exception ex)
{
// handle error
throw;
}
finally
{
if (xmlStream != null && dispose) xmlStream.Dispose();
}
}
The Xml Helper:
internal static class XmlHelper
{
internal static void WriteShallowNode(XmlReader reader, XmlWriter writer)
{
if (reader == null)
{
throw new ArgumentNullException("reader");
}
if (writer == null)
{
throw new ArgumentNullException("writer");
}
switch (reader.NodeType)
{
case XmlNodeType.Element:
writer.WriteStartElement(reader.Prefix, reader.LocalName, reader.NamespaceURI);
writer.WriteAttributes(reader, true);
if (reader.IsEmptyElement)
{
writer.WriteEndElement();
}
break;
case XmlNodeType.Text:
writer.WriteString(reader.Value);
break;
case XmlNodeType.Whitespace:
case XmlNodeType.SignificantWhitespace:
writer.WriteWhitespace(reader.Value);
break;
case XmlNodeType.CDATA:
writer.WriteCData(reader.Value);
break;
case XmlNodeType.EntityReference:
writer.WriteEntityRef(reader.Name);
break;
case XmlNodeType.XmlDeclaration:
case XmlNodeType.ProcessingInstruction:
writer.WriteProcessingInstruction(reader.Name, reader.Value);
break;
case XmlNodeType.DocumentType:
writer.WriteDocType(reader.Name, reader.GetAttribute("PUBLIC"), reader.GetAttribute("SYSTEM"),
reader.Value);
break;
case XmlNodeType.Comment:
writer.WriteComment(reader.Value);
break;
case XmlNodeType.EndElement:
writer.WriteFullEndElement();
break;
}
}
}

Regex XML parsing C#

I am trying to build a regex parser for a single XML block.
I know people will say that Regex is not a good plan for xml, but I am working with stream data and I just need to know if a complete xml block has been broadcast and is sitting in the buffer.
I am trying to handle for anything between the Opening and closing blocks of the XML and any data in parameters of the main block header.
My example code is below the broken down Regular Expression, if anyone has any input on how to make this as comprehensive as possible I would greatly appreciate it.
Here is my regular expression formatted for visual aid.
I am balancing the group, as well as the group and validating that they do not exist at the end of the expression segments.
/*
^(?<TAG>[<]
(?![?])
(?<TAGNAME>[^\s/>]*)
)
(?<ParamData>
(
(\"
(?>
\\\"|
[^"]|
\"(?<quote>)|
\"(?<-quote>)
)*
(?(quote)(?!))
\"
)|
[^/>]
)*?
)
(?:
(?<HASCONTENT>[>])|
(?<-TAG>
(?<TAGEND>/[>])
)
)
(?(HASCONTENT)
(
(?<CONTENT>
(
(?<inTAG>[<]\<TAGNAME>)(?<-inTAG>/[>])?|
(?<-inTAG>[<]/\<TAGNAME>[>])|
([^<]+|[<](?![/]?\<TAGNAME>))
)*?
(?(inTAG)(?!))
)
)
(?<TAGEND>(?<-TAG>)[<]/\<TAGNAME>[>])
)
(?(TAG)(?!))
*/
Within my class, I expect that any Null object returned means there was no xml block on the queue.
Here is the class I am using.
(I used a literal string (#"") to limit the escape requirements, All " characters were replaced with "" to format properly.
public class XmlDataParser
{
// xmlObjectExpression defined below to limit code highlight errors
private Regex _xmlRegex;
private Regex xmlRegex
{
get
{
if (_xmlRegex == null)
{
_xmlRegex = new Regex(xmlObjectExpression);
}
return _xmlRegex;
}
}
private string backingStore = "";
public bool HasObject()
{
return (backingStore != null) && xmlRegex.IsMatch(backingStore);
}
public string GetObject()
{
string result = null;
if (HasObject())
{
lock (this)
{
Match obj = xmlRegex.Match(backingStore);
result = obj.Value;
backingStore = backingStore.Substring(result.Length);
}
}
return result;
}
public void AddData(byte[] bytes)
{
lock (this)
{
backingStore += System.Text.Encoding.Default.GetString(bytes);
}
}
private static string xmlObjectExpression = #"^(?<TAG>[<](?![?])(?<TAGNAME>[^\s/>]*))(?<ParamData>((\""(?>\\\""|[^""]|\""(?<quote>)|\""(?<-quote>))*(?(quote)(?!))\"")|[^/>])*?)(?:(?<HASCONTENT>[>])|(?<-TAG>(?<TAGEND>/[>])))(?(HASCONTENT)((?<CONTENT>((?<inTAG>[<]\<TAGNAME>)(?<-inTAG>/[>])?|(?<-inTAG>[<]/\<TAGNAME>[>])|([^<]+|[<](?![/]?\<TAGNAME>)))*?(?(inTAG)(?!))))(?<TAGEND>(?<-TAG>)[<]/\<TAGNAME>[>]))(?(TAG)(?!))";
}

Just use XmlReader and feed it a TextReader. To read streams, you want to change the ConformanceLevel to Fragment.
XmlReaderSettings settings = new XmlReaderSettings();
settings.ConformanceLevel = ConformanceLevel.Fragment;
using (XmlReader reader = XmlReader.Create(tr,settings))
{
while (reader.Read())
{
switch (reader.NodeType)
{
// this is from my code. You'll rewrite this part :
case XmlNodeType.Element:
if (t != null)
{
t.SetName(reader.Name);
}
else if (reader.Name == "event")
{
t = new Event1();
t.Name = reader.Name;
}
else if (reader.Name == "data")
{
t = new Data1();
t.Name = reader.Name;
}
else
{
throw new Exception("");
}
break;
case XmlNodeType.Text:
if (t != null)
{
t.SetValue(reader.Value);
}
break;
case XmlNodeType.XmlDeclaration:
case XmlNodeType.ProcessingInstruction:
break;
case XmlNodeType.Comment:
break;
case XmlNodeType.EndElement:
if (t != null)
{
if (t.Name == reader.Name)
{
t.Close();
t.Write(output);
t = null;
}
}
break;
case XmlNodeType.Whitespace:
break;
}
}
}

What is the fastest way to load XML into an XDocument?

When you create a new XDocument using XDocument.Load, does it open the XML file and keep a local copy, or does it continuously read the document from the hard drive? If it does continuously read, is there a faster way to parse XML?
XDocument x = XDocument.Load("file.xml");

There are a couple of measurements to consider:
Linear traversal speed (e.g. reading/loading)
On-demand query speed
To answer the immediate question: XDocument uses an XmlReader to load the document into memory by reading each element and creating corresponding XElement instances (see code below). As such, it should be quite fast (fast enough for most purposes), but it may consume a large amount of memory when parsing a large document.
A raw XmlReader is an excellent choice for traversal if your needs are limited to that which can be done without retaining the document in memory. It will outperform other methods since no significant structure is created nor resolved with relation to other nodes (e.g. linking parent and child nodes). However, on-demand query ability is almost non-existent; you can react to values found in each node, but you can't query the document as a whole. If you need to look at the document a second time, you have to traverse the whole thing again.
By comparison, an XDocument will take longer to traverse because it instantiates new objects and performs basic structural tasks. It will also consume memory proportionate to the size of the source. In exchange for these trade-offs, you gain excellent query abilities.
It may be possible to combine the approaches, as mentioned by Jon Skeet and shown here: Streaming Into LINQ to XML Using C# Custom Iterators and XmlReader.
Source for XDocument Load()
public static XDocument Load(Stream stream, LoadOptions options)
{
XmlReaderSettings xmlReaderSettings = XNode.GetXmlReaderSettings(options);
XDocument result;
using (XmlReader xmlReader = XmlReader.Create(stream, xmlReaderSettings))
{
result = XDocument.Load(xmlReader, options);
}
return result;
}
// which calls...
public static XDocument Load(XmlReader reader, LoadOptions options)
{
if (reader == null)
{
throw new ArgumentNullException("reader");
}
if (reader.ReadState == ReadState.Initial)
{
reader.Read();
}
XDocument xDocument = new XDocument();
if ((options & LoadOptions.SetBaseUri) != LoadOptions.None)
{
string baseURI = reader.BaseURI;
if (baseURI != null && baseURI.Length != 0)
{
xDocument.SetBaseUri(baseURI);
}
}
if ((options & LoadOptions.SetLineInfo) != LoadOptions.None)
{
IXmlLineInfo xmlLineInfo = reader as IXmlLineInfo;
if (xmlLineInfo != null && xmlLineInfo.HasLineInfo())
{
xDocument.SetLineInfo(xmlLineInfo.LineNumber, xmlLineInfo.LinePosition);
}
}
if (reader.NodeType == XmlNodeType.XmlDeclaration)
{
xDocument.Declaration = new XDeclaration(reader);
}
xDocument.ReadContentFrom(reader, options);
if (!reader.EOF)
{
throw new InvalidOperationException(Res.GetString("InvalidOperation_ExpectedEndOfFile"));
}
if (xDocument.Root == null)
{
throw new InvalidOperationException(Res.GetString("InvalidOperation_MissingRoot"));
}
return xDocument;
}
// which calls...
internal void ReadContentFrom(XmlReader r, LoadOptions o)
{
if ((o & (LoadOptions.SetBaseUri | LoadOptions.SetLineInfo)) == LoadOptions.None)
{
this.ReadContentFrom(r);
return;
}
if (r.ReadState != ReadState.Interactive)
{
throw new InvalidOperationException(Res.GetString("InvalidOperation_ExpectedInteractive"));
}
XContainer xContainer = this;
XNode xNode = null;
NamespaceCache namespaceCache = default(NamespaceCache);
NamespaceCache namespaceCache2 = default(NamespaceCache);
string text = ((o & LoadOptions.SetBaseUri) != LoadOptions.None) ? r.BaseURI : null;
IXmlLineInfo xmlLineInfo = ((o & LoadOptions.SetLineInfo) != LoadOptions.None) ? (r as IXmlLineInfo) : null;
while (true)
{
string baseURI = r.BaseURI;
switch (r.NodeType)
{
case XmlNodeType.Element:
{
XElement xElement = new XElement(namespaceCache.Get(r.NamespaceURI).GetName(r.LocalName));
if (text != null && text != baseURI)
{
xElement.SetBaseUri(baseURI);
}
if (xmlLineInfo != null && xmlLineInfo.HasLineInfo())
{
xElement.SetLineInfo(xmlLineInfo.LineNumber, xmlLineInfo.LinePosition);
}
if (r.MoveToFirstAttribute())
{
do
{
XAttribute xAttribute = new XAttribute(namespaceCache2.Get((r.Prefix.Length == 0) ? string.Empty : r.NamespaceURI).GetName(r.LocalName), r.Value);
if (xmlLineInfo != null && xmlLineInfo.HasLineInfo())
{
xAttribute.SetLineInfo(xmlLineInfo.LineNumber, xmlLineInfo.LinePosition);
}
xElement.AppendAttributeSkipNotify(xAttribute);
}
while (r.MoveToNextAttribute());
r.MoveToElement();
}
xContainer.AddNodeSkipNotify(xElement);
if (r.IsEmptyElement)
{
goto IL_30A;
}
xContainer = xElement;
if (text != null)
{
text = baseURI;
goto IL_30A;
}
goto IL_30A;
}
case XmlNodeType.Text:
case XmlNodeType.Whitespace:
case XmlNodeType.SignificantWhitespace:
if ((text != null && text != baseURI) || (xmlLineInfo != null && xmlLineInfo.HasLineInfo()))
{
xNode = new XText(r.Value);
goto IL_30A;
}
xContainer.AddStringSkipNotify(r.Value);
goto IL_30A;
case XmlNodeType.CDATA:
xNode = new XCData(r.Value);
goto IL_30A;
case XmlNodeType.EntityReference:
if (!r.CanResolveEntity)
{
goto Block_25;
}
r.ResolveEntity();
goto IL_30A;
case XmlNodeType.ProcessingInstruction:
xNode = new XProcessingInstruction(r.Name, r.Value);
goto IL_30A;
case XmlNodeType.Comment:
xNode = new XComment(r.Value);
goto IL_30A;
case XmlNodeType.DocumentType:
xNode = new XDocumentType(r.LocalName, r.GetAttribute("PUBLIC"), r.GetAttribute("SYSTEM"), r.Value, r.DtdInfo);
goto IL_30A;
case XmlNodeType.EndElement:
{
if (xContainer.content == null)
{
xContainer.content = string.Empty;
}
XElement xElement2 = xContainer as XElement;
if (xElement2 != null && xmlLineInfo != null && xmlLineInfo.HasLineInfo())
{
xElement2.SetEndElementLineInfo(xmlLineInfo.LineNumber, xmlLineInfo.LinePosition);
}
if (xContainer == this)
{
return;
}
if (text != null && xContainer.HasBaseUri)
{
text = xContainer.parent.BaseUri;
}
xContainer = xContainer.parent;
goto IL_30A;
}
case XmlNodeType.EndEntity:
goto IL_30A;
}
break;
IL_30A:
if (xNode != null)
{
if (text != null && text != baseURI)
{
xNode.SetBaseUri(baseURI);
}
if (xmlLineInfo != null && xmlLineInfo.HasLineInfo())
{
xNode.SetLineInfo(xmlLineInfo.LineNumber, xmlLineInfo.LinePosition);
}
xContainer.AddNodeSkipNotify(xNode);
xNode = null;
}
if (!r.Read())
{
return;
}
}
goto IL_2E1;
Block_25:
throw new InvalidOperationException(Res.GetString("InvalidOperation_UnresolvedEntityReference"));
IL_2E1:
throw new InvalidOperationException(Res.GetString("InvalidOperation_UnexpectedNodeType", new object[]
{
r.NodeType
}));
}

It will parse the incoming stream (whether it is from a file or a string doesn't matter) when you call Load() and then keep a local instance of the document in memory. Since the source can be anything (could be a NetworkStream, a DataReader, a string entered by the user) it couldn't go back and try to read the data again since it wouldn't know the state of it (streams being closed etc).
If you really want speed on the other hand, XDocument isn't the fastets (all though it is easier to work with) since it will need to first parse the document and then retain it in memory. If you are working with really large documents using an approach with System.Xml.XmlReader is usually way faster since it can read the document as a stream and doesn't need to retain anything except the current element. This benchmark shows some interesting figures about this.

I do no think it continuously reads; the nice thing about the XDocument.Load method is that it uses XmlReader to read the XML into an XML tree. And since now you just created a tree which is most likely stored in your memory as a tree it no longer reads the document constantly. It manipulates the tree and since it is a tree all your reading and modification are done a lot faster. Although it does not not implement IDisposable it is automatically disposed.