I'm trying to consume a third party webservice. I craft a request message, send it to the httpclient and receive the httpResponseMessage.
Below is a snippet from the method that handles the call. xml = true for my situation.
HttpResponseMessage response = await HttpClientInstance.SendAsync(requestMessage);
if (response.IsSuccessStatusCode)
{
if (xml)
{
try
{
//This fails with "the data at the root level is invalid"
XmlMediaTypeFormatter xmlFormatter = new XmlMediaTypeFormatter { UseXmlSerializer = true };
var content = response.Content.ReadAsAsync<T>(new Collection<MediaTypeFormatter> { xmlFormatter });
return content.Result;
}
catch (Exception tempe)
{
var content = response.Content.ReadAsAsync<T>();
return content.Result;
}
}
else
{
var content = response.Content.ReadAsAsync<T>();
return content.Result;
}
}
else
{
_logWriter.LogProcessFault(operationContext, null, GetCurrentMethod(), $"An error occurred while calling the API. URI = {endpoint}. StatusCode = {response.StatusCode}", null);
throw new Exception(response.StatusCode.ToString());
}
After running this code, an error is thrown when attempting to deserialize the XML data in the stream. It fails with an error "the data at the root level is invalid".
I commented out the XMLMediaTypeFormatter and response.Content.Read... and replaced it with this
var fileStream = File.Create("D:\\Extract\\test.txt");
await response.Content.CopyToAsync(fileStream);
fileStream.Close();
which writes out valid XML to a file.
In the immediate window I ran response.Content.ReadAsStringAsync() and the returned string value has extra backslashes escaping content.
For instance, this is what is in the generated test.txt file:
<?xml version="1.0" encoding="utf-8"?>
and this is from ReadAsStringAsync:
<?xml version=\"1.0\" encoding=\"utf-8\"?>
I believe this is what is causing the deserialization to fail. Is there a clean fix for this or perhaps I am doing something wrong elsewhere?
The issue is in the following block of code
await response.Content.CopyToAsync(fileStream);
fileStream.Close();
//This fails with "the data at the root level is invalid"
XmlMediaTypeFormatter xmlFormatter = new XmlMediaTypeFormatter { UseXmlSerializer = true };
var content = response.Content.ReadAsAsync<T>(new Collection<MediaTypeFormatter> { xmlFormatter });
HTTP responses can be consumed only once. In other words, response is not read into some buffer from which multiple reads can be serviced. It reads it directly off of the socket. So once read, it cannot be read again.
Above, you copy response into a file. That consumes the response and it is no longer available to be read again. So by the time you try to read it again when assigning to content, it reads nothing, so the XML parser (in formatter) throws xml syntax error b/c it basically receives an empty string.
Not sure why you're saving to file, but once saved to file, you can simply read the file and send its contents to xml parser, and your code should now work. And of course if you remove saving to file, you should be able to read and parse contents of the response just fine.
Related
I have a request that calls a post method. It is posting XML in the request content (but sending it as raw text). In testing, the length of the xml is 106880 characters.
In the wep api post method, I process the request body to pull out the XML and store each element/value in a dictionary using the following:
var stream = new System.IO.StreamReader(Request.Body);
XmlReaderSettings settings = new XmlReaderSettings() { Async = true };
using (XmlReader r = XmlReader.Create(stream, settings))
{
bool rowsExist = true;
while (rowsExist && await r.ReadAsync())
{
if (nodeType == r.NodeType)
{
var name = r.Name;
rowsExist = await r.ReadAsync();
if (r.NodeType == XmlNodeType.Text)
{
xmlDic[name] = r.Value;
}
}
}
}
This works fine with small XML, however when the text value is relatively large, when calling the second ReadAsync method, the data is truncated and the XmlReader throws an exception saying
"Synchronous operations are disallowed. Call ReadAsync or set AllowSynchronousIO to true instead."
The exception makes no sense because ReadAsync is being called, but appears to be related to the size of the data, as it wasn't doing it with a smaller set of XML.
I tested a workaround which is to read the entire request body into a string, and then run the XmlReader using the entire body. However, that does use up more memory as it is loading the entire request into memory first, something that shouldn't be necessary.
I wondered if there might be a default max size/limit that stream or XmlReader uses, and see that the XmlReader settings class has 2 properties that control the Max characters:
settings.MaxCharactersFromEntities
settings.MaxCharactersInDocument
However the first has a default set to 10000000, which is way more than I am posting, and the second is set to zero, which means no limit. As a reault, these don't appear to make any difference.
What could be causing this to fail when reading the body using a StreamReader?
I am facing this exception
Error getting value from 'Position' on 'Amazon.Runtime.Internal.Util.MD5Stream'.
when trying to read file from aws s3 bucket.
Here is my c# code
try
{
var s3ObjectPath = $"users/{email.Id}/emailattachments/{item.AttachedFileName}";
var ifExists = await this.Exists(s3ObjectPath);
if (ifExists)
{
Stream attachment = await s3Client.GetObjectStreamAsync(attachmentS3BucketName, s3ObjectPath, dicData);
Attachment att = new Attachment(attachment, item.AttachedFileName);
attachments.Add(att);
}
}
catch (AmazonS3Exception ex)
{
}
However this is working sometimes. I searched everywhere but didn't find solution.
Thanks in advance!!
This is due to an issue with how Streams work. When you open a Stream you are not actually getting the content of the file, rather you are opening a connection enabling you to access the data.
The error you got is returned when your code is trying to access the stream but doesn't have the ability to do so (a connection being lost / no appropriate credentials in the code accessing the stream or any other reason).
The solution is either to solve the underlaying issue and make sure your code still as access to the Stream or to just read from the stream and return a string rather than a stream
The code addition would look something like this:
...
Stream attachment = await s3Client.GetObjectStreamAsync(attachmentS3BucketName, s3ObjectPath, dicData);
StreamReader reader = new StreamReader(attachment);
string attachmentText = reader.ReadToEnd();
Attachment att = new Attachment(attachmentText, item.AttachedFileName);
attachments.Add(att);
I have a very weird behaviour in a legacy repository that I cloned and try to work on.
There is an action for downloading files (files are hosted on AWS):
(redundant code omitted for clarity).
public virtual async Task<ActionResult> Entity(EntityModel entity)
{
using (Stream stream = downloadFileObject(model.Updates[GetUpdateNumberFromQueryString()].UpdateLink))
using (MemoryStream ms = new MemoryStream())
{
await stream.CopyToAsync(ms).ConfigureAwait(false);
return File(ms.ToArray(), "application/octet-stream", "The file.exe");
}
}
The DownloadFileObject method connects to AWS, creates a request and returns the response stream:
public Stream GetS3FileObject(string objectKey, string fileName)
{
GetObjectRequest request = new GetObjectRequest
{
BucketName = bucketName,
Key = objectKey,
};
GetObjectResponse response = s3Client.GetObject(request);
return response.ResponseStream;
}
This returns a proper stream with no errors, I can see that the lenght indicates something around 35 mb, which is the proper size of the file I am getting.
Now, I tried several approaches for returning this stream as downloadable file, and they all result in something very weird.
Instead of getting the file, the browser retrieves a 110kb text file which pretty much looks like my current view html...
The returned HTML contains the following error message (apart from the regular site content)
Error executing child request for handler
'System.Web.Mvc.HttpHandlerUtil+ServerExecuteHttpHandlerAsyncWrapper'.
OutputStream is not available when a custom TextWriter is used.
I tried several approaches, where the first and most obvious was just:
var stream = DownloadFileObject(model.Updates[GetUpdateNumberFromQueryString()].UpdateLink);
return File(stream, "application/octet-stream", "The file.exe");
Then I tried to copy that stream to a byte array and return that - however, in all cases the result is the same.
Up to the last moment, the debugger shows that my stream/byte array is 35mb large.
Here are the headers:
Also, the rendered Download button html is as below:
Download update
And the button is being created in the view without any HTML helpers (Action.Link() etc)
#Model.SiteLabelDownloadUpdate
Any ideas what can be going on?
I'm writing a Windows Service to listen and process messages from MSMQ. The listener has various error handling steps, but if all else fails, I want to save the body of the message to a text file so that I can look at it. However, I can't seem to extract the content of my messages when this condition is hit. The following code is a simple representation of the sections in question, and it always produces an empty text file even though I know the message I'm testing with is not empty. HOWEVER, if I comment-out the initial attempt to deserialize the XML, the fail safe does work and produces a text file with the message body. So I think the problem is something to do with how the deserialization attempt leaves the underlying Stream? Just to clarify, when the message contains valid XML that CAN be deserialized, the service all works fine and the fail-safe never comes into action.
MyClass myClass = null;
try
{
XmlSerializer serializer = new XmlSerializer(typeof(MyClass));
// Comment the following out and the fail safe works
// Let this run and fail and the text file below is always empty
myClass = (MyClass)serializer.Deserialize(m.BodyStream);
}
catch (Exception ex)
{
}
if (myClass == null)
{
string filePath = #"D:\path\file.txt";
m.Formatter = new ActiveXMessageFormatter();
StreamReader reader = new StreamReader(m.BodyStream);
File.WriteAllText(filePath, reader.ReadToEnd());
}
Depending on the formatter you are using:
For windows binary messages:
File.WriteAllText(<path>, (new UTF8Encoding()).GetString((byte[])msg.Body)); // for binary
For XML messages try this:
msg.Formatter = new XmlMessageFormatter(new String[] { "System.String, mscorlib" });
var text = msg.Body.ToString();
// write to file..
For neither binary or XML, use the native formatter:
msg.Formatter = new ActiveXMessageFormatter();
reader = new StreamReader(msg.BodyStream);
msgBody = reader.ReadToEnd();
// write to file..
I am trying to use the recommended code from this page http://blogs.msdn.com/b/nathana/archive/2011/03/31/deciphering-a-soap-fault-with-a-400-status-code.aspx as follows:
static FaultException ParseProtocolExecption(ProtocolException ex)
{
try
{
System.IO.Stream stream = (ex.InnerException as WebException).Response.GetResponseStream();
System.Xml.XmlReader xmr = System.Xml.XmlReader.Create(stream);
Message message = Message.CreateMessage(xmr, (int)stream.Length, MessageVersion.Soap12);
MessageFault mf = MessageFault.CreateFault(message, (int)stream.Length);
FaultException fe = new FaultException(mf);
message.Close();
return fe;
}
catch (Exception)
{
return new FaultException(ex.Message);
}
}
I am using VS 2012 with .NET 4.5 using WCF. When the app gets a 400 Bad Request and it passes the ProtocolException to ParseProtocolException, it throws an exception on this line:
Message message = Message.CreateMessage(xmr, (int)stream.Length, MessageVersion.Soap12);
with the System.ServiceModel.CommunicationException: "The size necessary to buffer the XML content exceeded the buffer quota."
The stream.Length = 2,704 bytes, which is not very big. I tried the solution suggested on this site http://blogs.msdn.com/b/drnick/archive/2007/08/07/increasing-the-maximum-fault-size.aspx. However, even with the MaxFaultSize = 1 Mb, it gets the same errror.
Instead of this line:
System.Xml.XmlReader xmr = System.Xml.XmlReader.Create(stream);
I've tried this:
xmr = XmlDictionaryReader.CreateTextReader(stream, XmlDictionaryReaderQuotas.Max);
which sets all the quotas to their maximum value (Int32.MaxValue); but, I still get the same error on the CreateMessage call.
A sample response stream from the System.Net.WebException is as follows:
<?xml version="1.0" encoding="UTF-8"?>
<SOAP-ENV:Envelope xmlns:SOAP-ENV="http://www.w3.org/2003/05/soap-envelope" xmlns:SOAP-ENC="http://www.w3.org/2003/05/soap-encoding" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:wsa5="http://www.w3.org/2005/08/addressing" xmlns:c14n="http://www.w3.org/2001/10/xml-exc-c14n#" xmlns:wsu="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-1.0.xsd" xmlns:ds="http://www.w3.org/2000/09/xmldsig#" xmlns:wsse="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd" xmlns:xmime5="http://www.w3.org/2005/05/xmlmime" xmlns:xop="http://www.w3.org/2004/08/xop/include" xmlns:tptz="http://www.onvif.org/ver20/ptz/wsdl" xmlns:tds2="http://www.onvif.org/ver10/schema" xmlns:tds3="http://docs.oasis-open.org/wsn/b-2" xmlns:tds4="http://docs.oasis-open.org/wsrf/bf-2" xmlns:tds5="http://docs.oasis-open.org/wsn/t-1" xmlns:timg="http://www.onvif.org/ver20/imaging/wsdl" xmlns:trt="http://www.onvif.org/ver10/media/wsdl" xmlns:tan="http://www.onvif.org/ver20/analytics/wsdl" xmlns:tds6="http://www.canon.com/ns/networkcamera/onvif/va/schema" xmlns:tmd="http://www.onvif.org/ver10/deviceIO/wsdl" xmlns:tev="http://www.onvif.org/ver10/events/wsdl" xmlns:tds9="http://docs.oasis-open.org/wsrf/r-2" xmlns:tds="http://www.onvif.org/ver10/device/wsdl" xmlns:ter="http://www.onvif.org/ver10/error">
<SOAP-ENV:Header></SOAP-ENV:Header>
<SOAP-ENV:Body>
<SOAP-ENV:Fault>
<SOAP-ENV:Code>
<SOAP-ENV:Value>SOAP-ENV:Sender</SOAP-ENV:Value>
<SOAP-ENV:Subcode>
<SOAP-ENV:Value>ter:NotAuthorized</SOAP-ENV:Value>
</SOAP-ENV:Subcode>
</SOAP-ENV:Code>
<SOAP-ENV:Reason>
<SOAP-ENV:Text xml:lang="en">Sender Not Authorized</SOAP-ENV:Text>
</SOAP-ENV:Reason>
<SOAP-ENV:Node>http://www.w3.org/2003/05/soap-envelope/node/ultimateReceiver</SOAP-ENV:Node>
<SOAP-ENV:Role>http://www.w3.org/2003/05/soap-envelope/role/ultimateReceiver</SOAP-ENV:Role>
<SOAP-ENV:Detail>The action requested requires authorization and the sender is not authorized</SOAP-ENV:Detail>
</SOAP-ENV:Fault>
</SOAP-ENV:Body>
</SOAP-ENV:Envelope>
Using an async web crawler that I wrote in F# Interactive, I found that some of the namespace url's were not resolvable. I corrected the erroneous ones and then ran the crawler again to sum the lengths of the namespace pages. The total is 715,965 bytes, which is much less than the Int32.MaxValue of all the quotas in XmlDictionaryReaderQuotas. Perhaps the XmlDictionaryReader has a bug, or the error it returns is not the real problem?
I finally got the Message creation to work by removing the namespace definitions that were not actually used in the SOAP-ENV:Body (i.e., keeping only the xmlns:ter used in the Subcode element). But, of course, this doesn't really solve the problem, because the service is generating the SOAP fault; and I cannot change the service implementation (it's a 3rd party device - an Onvif camera).
Morevover, I can't make the quotas any greater; so, how else to handle this exception?
The solution is to trap the CommunicationException and then do an alternate parse of the XML in the stream that doesn't require namespace resolution:
public static FaultException ParseProtocolException(System.ServiceModel.ProtocolException ex) {
var stream = (ex.InnerException as System.Net.WebException).Response.GetResponseStream();
try {
var xmr = XmlReader.Create(stream);
var message = Message.CreateMessage(xmr, (int)stream.Length, MessageVersion.Soap12);
var mf = MessageFault.CreateFault(message, (int)stream.Length);
message.Close();
return new FaultException(mf);
} catch (CommunicationException) { // If CreateMessage has a problem parsing the XML,
// then this error will be thrown. Most likely, there is an unresolvable namespace reference.
// Do an alternate parse
stream.Seek(0, System.IO.SeekOrigin.Begin);
var soapFault = GetSoapFault(stream);
return new FaultException(soapFault.Reason);
}
}
The catch resets the stream to the beginning and then uses the following to get the specifics of the SOAP Fault using an XmlReader:
private struct SoapFault {
public string Subcode;
public string Reason;
public string Detail;
}
private static string GetTextChild(XmlReader xmr, string childName) {
return xmr.ReadToDescendant(childName) ?
xmr.ReadString() : System.String.Empty;
}
private static SoapFault GetSoapFault(System.IO.Stream s) {
var xr = XmlReader.Create(s);
var fault = new SoapFault();
if (xr.ReadToFollowing("SOAP-ENV:Subcode")) {
fault.Subcode = GetTextChild(xr, "SOAP-ENV:Value");
if (xr.ReadToFollowing("SOAP-ENV:Reason")) {
fault.Reason = GetTextChild(xr, "SOAP-ENV:Text");
if (xr.ReadToFollowing("SOAP-ENV:Detail"))
fault.Detail = GetTextChild(xr, "SOAP-ENV:Text");
}
}
return fault;
}