I am trying to log all requests in my asp.net web API project to a text file. I am using DelegationHandler feature to implement logging mechanism in my application, below is the code snippet for that,
public class MyAPILogHandler : DelegatingHandler
{
protected override async Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
{
// Captures all properties from the request.
var apiLogEntry = CreateApiLogEntryWithRequestData(request);
if (request.Content != null)
{
await request.Content.ReadAsStringAsync()
.ContinueWith(task =>
{
apiLogEntry.RequestContentBody = task.Result;
}, cancellationToken);
}
return await base.SendAsync(request, cancellationToken)
.ContinueWith(task =>
{
var response = task.Result;
// Update the API log entry with response info
apiLogEntry.ResponseStatusCode = (int)response.StatusCode;
apiLogEntry.ResponseTimestamp = DateTime.Now;
if (response.Content != null)
{
apiLogEntry.ResponseContentBody = response.Content.ReadAsStringAsync().Result;
apiLogEntry.ResponseContentType = response.Content.Headers.ContentType.MediaType;
apiLogEntry.ResponseHeaders = SerializeHeaders(response.Content.Headers);
}
var logger = new LogManager();
logger.Log(new LogMessage()
{
Message = PrepareLogMessage(apiLogEntry),
LogTo = LogSource.File
});
return response;
}, cancellationToken);
}
}
Above implementation is working as expected and it is logging all required request/response information to the file.
But when we make any multipart/form-data POST api call with some images attached, after logging this request, log file becomes huge, because all image/binary content is getting converted into string and writing down it the text file. please find below log file content,
Body:
----------------------------079603462429865781513947
Content-Disposition: form-data; name="batchid"
22649EEE-3994-4225-AF73-D9A6B659CAE3
----------------------------079603462429865781513947
Content-Disposition: form-data; name="files"; filename="d.png"
Content-Type: image/png
PNG
IHDR í %v ¸ sRGB ®Îé gAMA ± üa pHYs à ÃÇo¨d ÿ¥IDATx^ìýX]K¶(
·îsß»ß÷þï{O÷iÛ Á2âîîîÁe¹âîî,<# Á$÷w_ÈZó5$Dwvv×}
----------------------------4334344396037865656556781513947
Content-Disposition: form-data; name="files"; filename="m.png"
Content-Type: image/png
PNG
IHDR í %v ¸ sRGB ®Îé gAMA ± üa pHYs à ÃÇo¨d ÿ¥IDATx^ìýX]K¶(
·îsß»ß÷þï{O÷iÛ Á2âîîîÁe¹âîî,<# Á$÷w_ÈZó5$Dwvv×}
I don't want to log the binary content of a request body, it could be sufficient to log only request body file contents like,
----------------------------079603462429865781513947
Content-Disposition: form-data; name="batchid"
22649EEE-3994-4225-AF73-D9A6B659CAE3
----------------------------079603462429865781513947
Content-Disposition: form-data; name="files"; filename="d.png"
Content-Type: image/png
----------------------------4334344396037865656556781513947
Content-Disposition: form-data; name="files"; filename="m.png"
Content-Type: image/png
Can you please suggest how to prevent logging binary content of request body and log only file contents of a request body.
From what I gather, you are implementing something similar to this approach.
When uploading a file (i.e., a request type "multipart/form-data") its actual content always begins with the "Content-Type: {ContentTypeValue}\r\n\r\n" sequence and the next header begins with "\r\n--" sequence (as it is illustrated in your logs). You can explore more info about raw request data parsing at ReferenceSource. So, you can strip everything about file (if exists), for example, via RegEx:
Content-Type: {ContentTypeOrOctetStream}\r\n\r\n{FileContentBytesToRemove}\r\n
using System.Text.RegularExpressions;
...
string StripRawFileContentIfExists(string input) {
if(input.IndexOf("Content-Type") == -1)
return input;
string regExPattern = "(?<ContentTypeGroup>Content-Type: .*?\\r\\n\\r\\n)(?<FileRawContentGroup>.*?)(?<NextHeaderBeginGroup>\\r\\n--)";
return Regex.Replace(input, regExPattern, me => me.Groups["ContentTypeGroup"].Value + string.Empty + me.Groups["NextHeaderBeginGroup"].Value);
}
...
//apiLogEntry.RequestContentBody = task.Result;
apiLogEntry.RequestContentBody = StripRawFileContentIfExists(task.Result);
I'd suggest separate huge content from log. like you encountered, it just screws everything up in the log. to some extent disables log functionality.
I'd suggest you organize those huge content in file system. like this:
---request-a/
|--request-a-body-multi-part1.txt
|--request-a-body-multi-part2.txt
and just maintain a link in your log to reference this file system path.
hope it helps.
I'm receiving a multipart content response that belongs to an OAuth batch request:
// batchRequest is a HttpRequestMessage, http is an HttpClient
HttpResponseMessage response = await http.SendAsync(batchRequest);
If I read its content as full text:
string fullResponse = await response.Content.ReadAsStringAsync();
This is what it contains:
--batchresponse_e42a30ca-0f3a-4c17-8672-22abc469cd16
Content-Type: application/http
Content-Transfer-Encoding: binary
HTTP/1.1 200 OK
DataServiceVersion: 3.0;
Content-Type: application/json;odata=minimalmetadata;streaming=true;charset=utf-8
{\"odata.metadata\":\"https://graph.windows.net/XXX.onmicrosoft.com/$metadata#directoryObjects/#Element\",\"odata.type\":\"Microsoft.DirectoryServices.User\",\"objectType\":\"User\",\"objectId\":\"5f6851c3-99cc-4a89-936d-4bb44fa78a34\",\"deletionTimestamp\":null,\"accountEnabled\":true,\"signInNames\":[],\"assignedLicenses\":[],\"assignedPlans\":[],\"city\":null,\"companyName\":null,\"country\":null,\"creationType\":null,\"department\":\"NRF\",\"dirSyncEnabled\":null,\"displayName\":\"dummy1 Test\",\"facsimileTelephoneNumber\":null,\"givenName\":\"dummy1\",\"immutableId\":null,\"isCompromised\":null,\"jobTitle\":\"test\",\"lastDirSyncTime\":null,\"mail\":null,\"mailNickname\":\"dummy1test\",\"mobile\":null,\"onPremisesSecurityIdentifier\":null,\"otherMails\":[],\"passwordPolicies\":null,\"passwordProfile\":{\"password\":null,\"forceChangePasswordNextLogin\":true,\"enforceChangePasswordPolicy\":false},\"physicalDeliveryOfficeName\":null,\"postalCode\":null,\"preferredLanguage\":null,\"provisionedPlans\":[],\"provisioningErrors\":[],\"proxyAddresses\":[],\"refreshTokensValidFromDateTime\":\"2016-12-02T08:37:24Z\",\"showInAddressList\":null,\"sipProxyAddress\":null,\"state\":\"California\",\"streetAddress\":null,\"surname\":\"Test\",\"telephoneNumber\":\"666\",\"thumbnailPhoto#odata.mediaEditLink\":\"directoryObjects/5f6851c3-99cc-4a89-936d-4bb44fa78a34/Microsoft.DirectoryServices.User/thumbnailPhoto\",\"usageLocation\":null,\"userPrincipalName\":\"dummy1test#XXX.onmicrosoft.com\",\"userType\":\"Member\"}
--batchresponse_e42a30ca-0f3a-4c17-8672-22abc469cd16
Content-Type: application/http
Content-Transfer-Encoding: binary
HTTP/1.1 200 OK
DataServiceVersion: 3.0;
Content-Type: application/json;odata=minimalmetadata;streaming=true;charset=utf-8
{\"odata.metadata\":\"https://graph.windows.net/XXX.onmicrosoft.com/$metadata#directoryObjects/#Element\",\"odata.type\":\"Microsoft.DirectoryServices.User\",\"objectType\":\"User\",\"objectId\":\"dd35d761-e6ed-44e7-919f-f3b1e54eb7be\",\"deletionTimestamp\":null,\"accountEnabled\":true,\"signInNames\":[],\"assignedLicenses\":[],\"assignedPlans\":[],\"city\":null,\"companyName\":null,\"country\":null,\"creationType\":null,\"department\":null,\"dirSyncEnabled\":null,\"displayName\":\"Max Admin\",\"facsimileTelephoneNumber\":null,\"givenName\":null,\"immutableId\":null,\"isCompromised\":null,\"jobTitle\":null,\"lastDirSyncTime\":null,\"mail\":null,\"mailNickname\":\"maxadmin\",\"mobile\":null,\"onPremisesSecurityIdentifier\":null,\"otherMails\":[],\"passwordPolicies\":null,\"passwordProfile\":null,\"physicalDeliveryOfficeName\":null,\"postalCode\":null,\"preferredLanguage\":null,\"provisionedPlans\":[],\"provisioningErrors\":[],\"proxyAddresses\":[],\"refreshTokensValidFromDateTime\":\"2016-12-05T15:11:51Z\",\"showInAddressList\":null,\"sipProxyAddress\":null,\"state\":null,\"streetAddress\":null,\"surname\":null,\"telephoneNumber\":null,\"thumbnailPhoto#odata.mediaEditLink\":\"directoryObjects/dd35d761-e6ed-44e7-919f-f3b1e54eb7be/Microsoft.DirectoryServices.User/thumbnailPhoto\",\"usageLocation\":null,\"userPrincipalName\":\"maxadmin#XXX.onmicrosoft.com\",\"userType\":\"Member\"}
--batchresponse_e42a30ca-0f3a-4c17-8672-22abc469cd16--
I need to get all these contents as objects (like classics HttpResponseMessage, not simple strings), in order to get the HTTP return code, the JSON content, etc. as properties and be able to treat them.
I know how to read separatly all these contents, but I can't figure how to get them as objects, I've only succeeded in getting a string content :
var multipartContent = await response.Content.ReadAsMultipartAsync();
foreach (HttpContent currentContent in multipartContent.Contents) {
var testString = currentContent.ReadAsStringAsync();
// How to get this content as an exploitable object?
}
In my example, testString contains:
HTTP/1.1 200 OK
DataServiceVersion: 3.0;
Content-Type: application/json;odata=minimalmetadata;streaming=true;charset=utf-8
{\"odata.metadata\":\"https://graph.windows.net/XXX.onmicrosoft.com/$metadata#directoryObjects/#Element\",\"odata.type\":\"Microsoft.DirectoryServices.User\",\"objectType\":\"User\",\"objectId\":\"5f6851c3-99cc-4a89-936d-4bb44fa78a34\",\"deletionTimestamp\":null,\"accountEnabled\":true,\"signInNames\":[],\"assignedLicenses\":[],\"assignedPlans\":[],\"city\":null,\"companyName\":null,\"country\":null,\"creationType\":null,\"department\":\"NRF\",\"dirSyncEnabled\":null,\"displayName\":\"dummy1 Test\",\"facsimileTelephoneNumber\":null,\"givenName\":\"dummy1\",\"immutableId\":null,\"isCompromised\":null,\"jobTitle\":\"test\",\"lastDirSyncTime\":null,\"mail\":null,\"mailNickname\":\"dummy1test\",\"mobile\":null,\"onPremisesSecurityIdentifier\":null,\"otherMails\":[],\"passwordPolicies\":null,\"passwordProfile\":{\"password\":null,\"forceChangePasswordNextLogin\":true,\"enforceChangePasswordPolicy\":false},\"physicalDeliveryOfficeName\":null,\"postalCode\":null,\"preferredLanguage\":null,\"provisionedPlans\":[],\"provisioningErrors\":[],\"proxyAddresses\":[],\"refreshTokensValidFromDateTime\":\"2016-12-02T08:37:24Z\",\"showInAddressList\":null,\"sipProxyAddress\":null,\"state\":\"California\",\"streetAddress\":null,\"surname\":\"Test\",\"telephoneNumber\":\"666\",\"thumbnailPhoto#odata.mediaEditLink\":\"directoryObjects/5f6851c3-99cc-4a89-936d-4bb44fa78a34/Microsoft.DirectoryServices.User/thumbnailPhoto\",\"usageLocation\":null,\"userPrincipalName\":\"dummy1test#XXX.onmicrosoft.com\",\"userType\":\"Member\"}
I can't just imagine to parse manually this string... So if someone has a clue or can explain me the good way to read the content, it would be nice.
Thanks,
Max
Here is a way it can be done. The key is to add a new content type "msgtype" header to the response:
var multipartContent = await response.Content.ReadAsMultipartAsync();
var multipartRespMsgs = new List<HttpResponseMessage>();
foreach (HttpContent currentContent in multipartContent.Contents) {
// Two cases:
// 1. a "single" response
if (currentContent.Headers.ContentType.MediaType.Equals("application/http", StringComparison.OrdinalIgnoreCase)) {
if (!currentContent.Headers.ContentType.Parameters.Any(parameter => parameter.Name.Equals("msgtype", StringComparison.OrdinalIgnoreCase) && parameter.Value.Equals("response", StringComparison.OrdinalIgnoreCase))) {
currentContent.Headers.ContentType.Parameters.Add(new NameValueHeaderValue("msgtype", "response"));
}
multipartRespMsgs.Add(await currentContent.ReadAsHttpResponseMessageAsync());
// The single object in multipartRespMsgs contains a classic exploitable HttpResponseMessage (with IsSuccessStatusCode, Content.ReadAsStringAsync().Result, etc.)
}
// 2. a changeset response, which is an embedded multipart content
else {
var subMultipartContent = await currentContent.ReadAsMultipartAsync();
foreach (HttpContent currentSubContent in subMultipartContent.Contents) {
currentSubContent.Headers.ContentType.Parameters.Add(new NameValueHeaderValue("msgtype", "response"));
multipartRespMsgs.Add(await currentSubContent.ReadAsHttpResponseMessageAsync());
// Same here, the objects in multipartRespMsgs contain classic exploitable HttpResponseMessages
}
}
}
Thanks to darl0026
I want to handle HEAD requests in my Web API 2 application. I copy-pasted the code from StrathWeb's blog post Adding HTTP HEAD support to ASP.NET Web API:
public class HeadHandler : DelegatingHandler
{
private const string Head = "IsHead";
protected override async Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, System.Threading.CancellationToken cancellationToken)
{
if (request.Method == HttpMethod.Head)
{
request.Method = HttpMethod.Get;
request.Properties.Add(Head, true);
}
var response = await base.SendAsync(request, cancellationToken);
object isHead;
response.RequestMessage.Properties.TryGetValue(Head, out isHead);
if (isHead != null && ((bool) isHead))
{
var oldContent = await response.Content.ReadAsByteArrayAsync();
var content = new StringContent(string.Empty);
content.Headers.Clear();
foreach (var header in response.Content.Headers)
{
content.Headers.Add(header.Key, header.Value);
}
content.Headers.ContentLength = oldContent.Length;
response.Content = content;
}
return response;
}
}
When I make a HEAD request from Fiddler:
HEAD http://localhost:54225/api/books/5 HTTP/1.1
User-Agent: Fiddler
Host: localhost:54225
I get the following response back:
HTTP/1.1 200 OK
Cache-Control: no-cache
Pragma: no-cache
Content-Length: 0
Content-Type: application/json; charset=utf-8
Expires: -1
Server: Microsoft-IIS/10.0
X-AspNet-Version: 4.0.30319
X-SourceFiles: =?UTF-8?B?QzpcVXNlcnNcc2ltb25lXERvY3VtZW50c1xTaW1vbnNEb2N1bWVudHNcSVRcQyNcRGVtb0NvZGVcUkVTVGZ1bFdlYlNlcnZpY2VDbGllbnRERU1PXEJvb2tTZXJ2aWNlXGFwaVxib29rc1w1?=
X-Powered-By: ASP.NET
Date: Sun, 19 Feb 2017 12:09:52 GMT
The Content-Length is 0 but it should be 104 bytes.
When I add a breakpoint to the HeadHandler and step through the code it seems to set the Content-Length header value correctly, to 104 bytes, before returning the response.
Could there be some other step in the pipeline that is running after the HeadHandler, which recognises the response has no body, and sets the Content-Length to 0? The only things in the WebApiConfig.cs that come after the HeadHandler has been added are the mapping of the HttpAttribute routes and the setting up of the default route, neither of which seem likely to me to result in the Content-Length header being reset. Alternatively, could a configuration setting somewhere possibly affect the Content-Length returned in the response?
All you need to do is set the Content-Length of the response's Content.Headers - no need to create new content for the response. The act of requesting HEAD removes the body content from the response anyway.
This may have been a change since the article you quoted was written (2013), and no longer needs to create content from scratch...
So this is all you should need:
public class HeadHandler : DelegatingHandler
{
private const string Head = "IsHead";
protected override async Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, System.Threading.CancellationToken cancellationToken)
{
if (request.Method == HttpMethod.Head)
{
request.Method = HttpMethod.Get;
request.Properties.Add(Head, true);
}
var response = await base.SendAsync(request, cancellationToken);
object isHead;
response.RequestMessage.Properties.TryGetValue(Head, out isHead);
if (isHead != null && ((bool)isHead))
{
var oldContent = await response.Content.ReadAsByteArrayAsync();
response.Content.Headers.ContentLength = oldContent.Length;
return response;
}
return response;
}
}
I'm making a pretty simple WebAPI using OWIN/Katana self-hosted in a Azure Worker role. Everything is fine from the host perspective since I receive the request and it is routed to my Action just fine.
The problem is that the action MUST return a XML for the API caller/invoker and it is return a wrong encoding string as follow:
RAW Response:
HTTP/1.1 200 OK
Content-Length: 150
Content-Type: text/xml; charset=utf-8
Server: Microsoft-HTTPAPI/2.0
Date: Wed, 16 Jul 2014 05:49:42 GMT
<string xmlns="http://schemas.microsoft.com/2003/10/Serialization/"><Response>
<Say>Hello World</Say>
</Response></string>
RAW Request:
GET http://localhost:81/v1/ivr/menu?from=+12345&to=645645&callsid=11111111 HTTP/1.1
User-Agent: Fiddler
Host: localhost:81
Content-Type: text/xml;
response.ToString() result:
<Response>
<Say>Hello World</Say>
</Response>
API Controller Action code:
[HttpGet]
[Route("menu")]
public IHttpActionResult Menu(string from, string to, string callSid)
{
var response = new TwilioResponse();
response.Say("Hello World");
return Ok(response.ToString());
}
Well, all I need is to return the XML as this:
<Response>
<Say>Hello World</Say>
</Response>
What exactly am I doing wrong? The response is coming between tags and with a weird xmlns with what seems to be a wrong encoding...
I've tried to add Content-Type and Accept headers on the request to text/xml(I can't use application/xml but even if I do, it don't changes the output response)...
Also I've tried to create a OWINMiddleware that force the Response content type to text/xml as this:
public class XmlResponseMiddleware : OwinMiddleware
{
public XmlResponseMiddleware(OwinMiddleware next) : base(next)
{
}
public override async Task Invoke(IOwinContext context)
{
context.Response.ContentType = "text/xml";
await this.Next.Invoke(context);
}
}
No lucky...
Any help would be appreciated.
Thanks!
[HttpGet]
[Route("menu")]
public HttpResponeMessage Menu(string from, string to, string callSid)
{
var response = new TwilioResponse();
response.Say("Hello World");
return new HttpResponseMessage()
{
Content = new StringContent(
response.ToString(),
Encoding.UTF8,
"text/xml"
)
};
}
I know this was already answered, but the answer required changing your response type from an IHttpActionResult to an HttpResponeMessage.
If you want to preserve the IHttpActionResult (and make it a little cleaner) you can do this:
using System.Xml.Linq;
[HttpGet]
[Route("menu")]
public IHttpActionResult Menu(string from, string to, string callSid)
{
var response = new TwilioResponse();
response.Say("Hello World");
return Ok(XElement.Parse(response.ToString()));
}
In your WebApiConfig, add this to the register.
It will cause XML to be the default.
Why Twilio doesn't include application/xml as an accept header is beyond me.
config.Formatters.Clear();
config.Formatters.Add(new XmlMediaTypeFormatter());
config.Formatters.Add(new JsonMediaTypeFormatter());
config.Formatters.Add(new FormUrlEncodedMediaTypeFormatter());