Large requests are truncated when logging - c#

I have a set of WCF SOAP web services. A few of the service operations allows users to submit data. I want to log the payload from these requests to a SQL Server database. For clients that just read data, I only want to log simple metadata.
This is my data model:
I have a class that implements the IParameterInspector interface. All service operations requests are logged in the BeforeCall method: time of request, operation name, userid and target endpoint.
public class LogParameterInspector : IParameterInspector
{
private static WebServiceLog WebServiceLog = WebServiceLog.GetInstance();
private bool IsOperationIgnorable(string operationName)
{
//Ignore GETs for service metadata
return string.Equals(operationName, "get", StringComparison.InvariantCultureIgnoreCase);
}
public void AfterCall(string operationName, object[] outputs, object returnValue, object correlationState)
{
if (IsOperationIgnorable(operationName))
return;
SsnWebServiceContext.Current.Response.CreatedDate = DateTime.Now;
//Calls stored procedure
WebServiceLog.LogServerResponseToDb(SsnWebServiceContext.Current.Response);
}
public object BeforeCall(string operationName, object[] inputs)
{
if (IsOperationIgnorable(operationName))
return null;
SsnWebServiceContext.Current.Request.CreatedDate = DateTime.Now;
SsnWebServiceContext.Current.Request.OperationName = operationName;
foreach (var item in inputs)
{
if (item == null) continue;
if (item.GetType().GetInterfaces().Any(x => x.IsGenericType && x.GetGenericTypeDefinition() == typeof(IRequest<>)))
{
var header = item
.GetType()
.GetProperties()
.Single(p => p.Name == "Header")
.GetValue(item, null) as IRequestHeader;
SsnWebServiceContext.Current.Request.UserIdentification = header?.UserName;
}
}
//Calls stored procedure
WebServiceLog.LogClientRequestToDb(SsnWebServiceContext.Current.Request);
return null;
}
}
The message payload is explicitly logged in the few service operations that require it. (Calling the same stored procedure as in BeforeCall. The procedure checks if a record with the given GUID already exists, and if so, updates it).
I have a class that implements IExtension<OperationContext>. This class gives me access to request/response data through the lifetime of a request.
public class SsnWebServiceContext : IExtension<OperationContext>
{
//Helper property for syntax purposes
public static SsnWebServiceContext Current => OperationContext.Current.Extensions.Find<SsnWebServiceContext>();
//Called when extension is added to OperationContext. Useful for initializing stuff.
public void Attach(OperationContext owner)
{
var tmp = Guid.NewGuid();
Request = Request ?? new ClientRequest();
Request.SetReadOnlyGuid(tmp);
Response = Response ?? new ServerResponse();
Response.SetReadOnlyGuid(tmp);
}
//Called when extension is removed from OperationContext. Use for for cleanup.
public void Detach(OperationContext owner)
{
}
public ClientRequest Request { get; set; }
public ServerResponse Response { get; set; }
}
My custom context is added to OperationContext through a class that implements IDispatchMessageInspector. This is also where I create a copy of the message stream for future reference.
public class ContextDispatchMessageInspector : IDispatchMessageInspector
{
public object AfterReceiveRequest(ref Message request, IClientChannel channel, InstanceContext instanceContext)
{
using (var buffer = request.CreateBufferedCopy(int.MaxValue))
{
OperationContext.Current.Extensions.Add(new SsnWebServiceContext
{
Request = new ClientRequest
{
Message = buffer.CreateMessage(),
TargetRole = OperationContext.Current.IncomingMessageHeaders?.To?.ToString()
}
});
request = buffer.CreateMessage();
}
return null; //Correlation state passed to BeforeSendReply
}
public void BeforeSendReply(ref Message reply, object correlationState)
{
OperationContext.Current.Extensions.Remove(SsnWebServiceContext.Current);
}
}
All my endpoints are configured for basicHttpBinding with the following settings:
<bindings>
<basicHttpBinding>
<binding name="SSNBinding" maxBufferSize="2147483647" maxReceivedMessageSize="2147483647" >
<security mode="None"></security>
</binding>
</basicHttpBinding>
</bindings>
Problem:
The logging works fine, except for larger messages. When I try to view some of the XML messages in SSMS, I get the following error:
If I copy the XML content into a text editor, it's clear that the entire message has not been logged. I have found that this occurs only for requests where users are uploading files bigger than around 2MB. Files are uploaded as base64 encoded strings, and the logged XML content is truncated somewhere in the middle of the encoded string.
Weirdly (to me) the file seems to have been transferred successfully, even if it hasn't been logged completely. I've reproduced the behaviour locally, using SOAP UI and base64 encoding a large image: The request took a while (timed out after 1 minute in SOAP UI), but still completed succesfully some time later; I downloaded and viewed the image, and it was not corrupt. I got the same result with a large Word document.
Any suggestions on what I can do to prevent my logs from being truncated would be highly appreciated!

Your problem is not that the data is truncated in the table, but that SSMS limits the size of data in the output. So do not worry about missing data. your code is fine. That's why no errors are reported.
To check read the data using code (using plain ADO.Net for example), you will see that all the data is stored.
See some solutions:
http://blog.extreme-advice.com/2013/03/05/error-fix-unable-to-show-xml-the-following-error-happened-there-is-an-unclosed-literal-string/
https://www.mssqltips.com/sqlservertip/2795/prevent-truncation-of-dynamically-generated-results-in-sql-server-management-studio/
SQL Server truncation and 8192 limitation

Related

How to return a value from a c# SignalR client

I have a hub that manages many worker processes. I want to build a UI that lets me connect to this hub and retrieve the processing log from any of these worker processes. Essentially this will be a client wanting to obtain a string from another client. I have been able to get the request from client A sent to client B, but i dont know how to return anything in that response.
I have a simple method in the hub
public void GetRunLog(int runid)
{
JobRunLog log = null;
JobRunClient client = this.GetClientByRunID(runid);
if(client != null)
{
var rawlog = Clients.Client(client.ConnectionID).GetRunLog();
log = JsonConvert.DeserializeObject<JobRunLog>(rawlog);
Clients.Client(Context.ConnectionId).GetRunLog(log);
}
}
This request gets picked up by the client, but I dont know how to make it return a value so that var rawlog actually contains something. For the moment, this is the best workaround i could come up with.
myHubProxy.On("GetRunLog", (uiconnectionid) =>
{
string connectionid = uiconnectionid;
myHubProxy.Invoke("ReturnRunLog", run.ID, run.Log, connectionid).ContinueWith(task => {});
});
This will then make the worker client send the log back in a separate request with a reference to the client that had requested the log, but it isnt actually returning a respnonse to the initial request. I cant see a way to make this happen. Rather than use Invoke, how would i just return the object directly to the method on the hub that initiated the request?
Unfortunatelly Hub doesn't keeps it's state:
Because instances of the Hub class are transient, you can't use them
to maintain state from one method call to the next. Each time the
server receives a method call from a client, a new instance of your
Hub class processes the message. To maintain state through multiple
connections and method calls, use some other method such as a
database, or a static variable on the Hub class, or a different class
that does not derive from Hub.
Try to move the logic into a separate class and store the instance object in a static dictionary related to the connection id (don't forget to clean it). Whenewer call comes to the Hub it repoints it to a appropriate instance,
here is the simplified sample
public class TestingLogHub : Hub
{
public static readonly Dictionary<string, TestInstance> Instances =
new Dictionary<string, TestInstance>();
public void SetParameter(string value)
{
Instances[Context.ConnectionId].ContinueWith(value);
}
...
}
public class TestInstance : IDisposable
{
public TestInstance(string basePath, IHubContext host, string connectionId)
{...
}
public void ContinueWith(string value)
{
if (_nextAction == null)
{
FinishExecution();
}
else
{
try
{
_nextAction(value);
}
catch (Exception exception)
{
Error(exception.Message);
FinishExecution();
}
}
}
public void RequestParameterFor(Action<string> action, string parameter, string defaultValue = null)
{
_nextAction = action;
_host.Clients.Client(_connectionId).requestParameter(parameter, defaultValue??GetRandomText());
}
}
So when Instance is started it's doing some work, but at the moment it requires some input it executes RequestParameterFor that set's the next function to be executed into an instance state and waits for the next call of ContinueWith.
it is a bit generic example, in your case you can send back an object and provide it to an instance, and maybe dispose the instance at the end of that request, if that was the only required call

IEndpointBehavior life cycle / logging service calls

I'm trying to log all outbound requests that go to service references, including the full request and response body. I thought I had a solution using behaviorExtensions but, after deploying, it became clear that the extension was shared between multiple requests.
Here's my current code:
public class LoggingBehaviorExtender : BehaviorExtensionElement
{
public override Type BehaviorType => typeof(LoggingRequestExtender);
protected override object CreateBehavior() { return new LoggingRequestExtender(); }
}
public class LoggingRequestExtender : IClientMessageInspector, IEndpointBehavior
{
public string Request { get; private set; }
public string Response { get; private set; }
#region IClientMessageInspector
public virtual object BeforeSendRequest(ref System.ServiceModel.Channels.Message request, System.ServiceModel.IClientChannel channel)
{
Request = request.ToString();
Response = null;
return null;
}
public virtual void AfterReceiveReply(ref System.ServiceModel.Channels.Message reply, object correlationState)
{
Response = reply.ToString();
}
#endregion
#region IEndpointBehavior
public void AddBindingParameters(ServiceEndpoint endpoint, BindingParameterCollection bindingParameters) { }
public void ApplyClientBehavior(ServiceEndpoint endpoint, ClientRuntime clientRuntime)
{
clientRuntime.MessageInspectors.Add(this);
}
public void ApplyDispatchBehavior(ServiceEndpoint endpoint, EndpointDispatcher endpointDispatcher) { }
public void Validate(ServiceEndpoint endpoint) { }
#endregion
}
Then, when I reach the point to log, I extract the behavior...
var lre = client.Endpoint.Behaviors.OfType<LoggingRequestExtender>().FirstOrDefault();
var req = lre?.Request;
var resp = lre?.Response;
Adding debugging logging to the LoggingRequestExtender, I found it was only instantiated once for multiple requests.
Is there a way to make sure this behavior class is instantiated fresh for each thread? Or is there a better way of getting the full request / response body when making service calls?
Edit / Partial answer:
Since writing this I have discovered that the value returned by BeforeSendRequest is passed into AfterReceiveReply as the correlationState so I can connect the request and response using a guid:
public virtual object BeforeSendRequest(ref System.ServiceModel.Channels.Message request, System.ServiceModel.IClientChannel channel)
{
var guid = Guid.NewGuid();
WebServiceLog.LogCallStart(guid, channel.RemoteAddress.ToString(), request.ToString());
return guid;
}
public virtual void AfterReceiveReply(ref System.ServiceModel.Channels.Message reply, object correlationState)
{
Guid guid = (Guid)correlationState;
WebServiceLog.LogCallEnd(guid, reply.ToString());
}
I see two flaws to this approach. One, which is livable, is that this requires a log insert and then update rather than a single insert.
The second is more of an issue: In the case of an exception (e.g. timeout), we never hit AfterRecieveSupply so the log doesn't know what happened. I can separately log the exception...
try
{
response = client.SomeFunction(request);
}
catch (Exception ex)
{
AppLog.Error("Some function failed", ex);
}
... but I can't see a way of accessing the guid outside of BeforeSendRequest / AfterReceiveReply so I have nothing to tie the exception log to the service request log.
There are several approaches to this.
1, The situation you have described with having to log calls separately doesn't have to be like that. If your WCF service is in a non load balanced server just add the request to a MemoryCache using the Guid as a key. When the request comes in then pull off the request and log in one go. To capture the timed out calls you could run a process on a thread that would check the MemoryCache every x minutes to pull out and log (using an adequate lock to ensure thread saftey).
If the WCF service is in a load balanced environment then all you do is the same as above but store to a no sql type data store.
2, Is the code that makes the outbound calls within your scope for change? If so, you can forgo creating a behavior extension and create a bespoke message logger instead. Using a class that implements IDisposable you can write nice code like this..
RequestMessage request = new RequestMessage();
ResponseMessage response = null;
using (_messageLogger.LogMessage(request, () => response, CallContextHelper.GetContextId(), enabled))
{
response = _outboundService.DoSomething(request);
}
This will then not need another process to capture any timed out threads which will be handled in the dispose method.
If you need more clarity then let me know, hopefully this helps you...

Returning a value in MVC Web API, and then run log code afterwards

Below is a very simple HelloWorld API Method
[HttpGet]
[Route("api/helloworld")]
public string SayHello()
{
try
{
return "Hello World - API Version 1 - " + DateTime.Now.ToLongTimeString();
}
finally
{
Log("I'd like logging to not hold up the string from getting returned");
}
}
Unfortunately, the finally code doesn't work this way, so the log method in this case would block the string from getting returned until after the log was complete.
Is it possible to return a value in the MVC Web API, and then run code afterwards? In my particular case I'd like to log afterwards, but there is no reason for the database logging to take up time for the client to receive a response, as it will not effect the response.
Yes, but you'd need to run it on a separate thread.
While WebAPI does not have a OnRequestExecuted method on filters, which is what you are probably looking for it, I think filters are still the correct approach.
What you will need is a filter combined with a derivative ObjectContent class that defers your post-request logic to after the response is written. I use this approach to automatically create NHibernate session and transaction at the beginning of a request, and commit them when the request is complete, which is similar to what you described in a comment. Please keep in mind this is greatly simplified to illustrate my answer.
public class TransactionAttribute : ActionFilterAttribute
{
public override void OnActionExecuting(HttpActionContext actionContext)
{
// create your connection and transaction. in this example, I have the dependency resolver create an NHibernate ISession, which manages my connection and transaction. you don't have to use the dependency scope (you could, for example, stuff a connection in the request properties and retrieve it in the controller), but it's the best way to coordinate the same instance of a required service for the duration of a request
var session = actionContext.Request.GetDependencyScope().GetService(typeof (ISession));
// make sure to create a transaction unless there is already one active.
if (!session.Transaction.IsActive) session.BeginTransaction();
// now i have a valid session and transaction that will be injected into the controller and usable in the action.
}
public override void OnActionExecuted(HttpActionExecutedContext actionExecutedContext)
{
var session = actionExecutedContext.Request.GetDependecyScope().GetService(typeof(ISession));
var response = actionExecutedContext.Response;
if (actionExecutedContext.Exception == null)
{
var content = response.Content as ObjectContent;
if (content != null)
{
// here's the real trick; if there is content that needs to be sent to the client, we need to swap the content with an object that will clean up the connection and transaction AFTER the response is written.
response.Content = new TransactionObjectContent(content.ObjectType, content.Value, content.Formatter, session, content.Headers);
}
else
{
// there is no content to send to the client, so commit and clean up immediately (in this example, the container cleans up session, so it is omitted below)
if (session.Transaction.IsActive) session.Transaction.Commit();
}
}
else
{
// an exception was encountered, so immediately rollback the transaction, let the content return unmolested, and clean up your session (in this example, the container cleans up the session for me, so I omitted it)
if (session.Transaction.IsActive) session.Transaction.Rollback();
}
}
}
And the magic happens in this ObjectContent derivative. Its responsibility is to stream the object to the response, supporting async actions, and do something after the response is sent down. You can add your logging, db, whatever in here. In this example, it merely commits a transaction after successfully writing the response.
public class TransactionObjectContent : ObjectContent
{
private ISession _session;
public TransactionObjectContent(Type type, object value, MediaTypeFormatter formatter, ISession session, HttpContentHeaders headers)
: base(type, value, formatter)
{
_session = session;
foreach (var header in headers)
{
response.Content.Headers.TryAddWithoutValidation(header.Key, header.Value);
}
}
protected async override Task SerializeToStreamAsync(Stream stream, TransportContext context)
{
await base.SerializeToStreamAsync(stream, context); // let it write the response to the client
// here's the meat and potatoes. you'd add anything here that you need done after the response is written.
if (_session.Transaction.IsActive) _session.Transaction.Commit();
}
protected override void Dispose(bool disposing)
{
base.Dispose(disposing);
if (disposing)
{
if (_session != null)
{
// if transaction is still active by now, we need to roll it back because it means an error occurred while serializing to stream above.
if (_session.Transaction.IsActive) _session.Transaction.Rollback();
_session = null;
}
}
}
}
Now you can either register the filter in your global filters, or add it directly to actions or controllers. You don't have to keep copying and pasting redundant code for executing your logic in each action in another thread; the logic just gets applied automagically to every action you target with the filter. Much cleaner, much easier, and DRY.
Example controller:
[Transaction] // will apply the TransactionFilter to each action in this controller
public DoAllTheThingsController : ApiController
{
private ISession _session;
public DoAllTheThingsController(ISession session)
{
_session = session; // we're assuming here you've set up an IoC to inject the Isession from the dependency scope, which will be the same as the one we saw in the filter
}
[HttpPost]
public TheThing Post(TheThingModel model)
{
var thing = new TheThing();
// omitted: map model to the thing.
// the filter will have created a session and ensured a transaction, so this all nice and safe, no need to add logic to fart around with the session or transaction. if an error occurs while saving, the filter will roll it back.
_session.Save(thing);
return thing;
}
}

Getting WCF message body before deserialization

I am implementing WCF service that exposes a method whose [OperationContract] is [XmlSerializerFormat]. I sometimes get request whose body is not valid XML. In such cases I want to log the original body, so I can know why it didn't constitute valid XML. However, I can't get it from the Message object, see my attempts (by implementing IDispatchMessageInspector interface):
public object IDispatchMessageInspector.AfterReceiveRequest(ref Message request, IClientChannel channel, InstanceContext instanceContext)
{
request.ToString(); // "... Error reading body: System.Xml.XmlException: The data at the root level is invalid. Line 1, position 1. ..."
request.WriteBody(...); // Serialization Exception, also in WriteMessage and other Write* methods
request.GetReaderAtBodyContents(...); // Same
HttpRequestMessageProperty httpRequest = (HttpRequestMessageProperty)request.Properties[HttpRequestMessageProperty.Name]; // no body in httpRequest
}
When looking in watch, request.messageData appears to contain the body - but that's a private member.
How can I get the message buffer without trying to deserialize it?
Yes, you need custom MessageEncoder, unlike message inspectors (IDispatchMessageInspector / IClientMessageInspector) it sees original byte content including any malformed XML data.
However it's not trivial how to implement this approach. You have to wrap a standard textMessageEncoding as custom binding element and adjust config file to use that custom binding.
Also you can see as example how I did it in my project - wrapping textMessageEncoding, logging encoder, custom binding element and config.
For the opposite direction (I am writing a WCF client and the server returns invalid XML), I was able to extract the raw reply message in IClientMessageInspector.AfterReceiveReply by
accessing the internal MessageData property of reply via Reflection, and then
accessing its Buffer property, which is an ArraySegment<byte>.
Something similar might be available for the request message on the server side; so it might be worth examining the request variable in the debugger.
I'm aware that this is not exactly what you are asking for (since you are on the server side, not on the client side), and I'm also aware that using reflection is error-prone and ugly. But since the correct solution is prohibitively complex (see baur's answer for details) and this "raw dump" is usually only required for debugging, I'll share my code anyways, in case it is helpful to someone in the future. It works for me on .NET Framework 4.8.
public void AfterReceiveReply(ref Message reply, object correlationState)
{
object messageData = reply.GetType()
.GetProperty("MessageData",
BindingFlags.NonPublic | BindingFlags.Instance)
.GetValue(reply, null);
var buffer = (ArraySegment<byte>)messageData.GetType()
.GetProperty("Buffer")
.GetValue(messageData, null);
byte[] rawData =
buffer.Array.Skip(buffer.Offset).Take(buffer.Count).ToArray();
// ... do something with rawData
}
And here's the full code of the EndpointBehavior:
public class WcfLogger : IEndpointBehavior
{
public byte[] RawLastResponseBytes { get; private set; }
// We don't need these IEndpointBehavior methods
public void AddBindingParameters(ServiceEndpoint endpoint, BindingParameterCollection bindingParameters) { }
public void ApplyDispatchBehavior(ServiceEndpoint endpoint, EndpointDispatcher endpointDispatcher) { }
public void Validate(ServiceEndpoint endpoint) { }
public void ApplyClientBehavior(ServiceEndpoint endpoint, ClientRuntime clientRuntime)
{
clientRuntime.ClientMessageInspectors.Add(new MessageCaptureInspector(this));
}
internal class MessageCaptureInspector : IClientMessageInspector
{
private WcfLogger logger;
public MessageCaptureInspector(WcfLogger logger)
{
this.logger = logger;
}
public void AfterReceiveReply(ref Message reply, object correlationState)
{
// Ugly reflection magic. We need this for the case where
// the reply is not valid XML, and, thus, reply.ToString()
// only contains an error message.
object messageData = reply.GetType()
.GetProperty("MessageData",
BindingFlags.NonPublic | BindingFlags.Instance)
.GetValue(reply, null);
var buffer = (ArraySegment<byte>)messageData.GetType()
.GetProperty("Buffer")
.GetValue(messageData, null);
logger.RawLastResponseBytes =
buffer.Array.Skip(buffer.Offset).Take(buffer.Count).ToArray();
}
public object BeforeSendRequest(ref Message request, IClientChannel channel)
{
return null;
}
}
}
Usage:
var logger = new WcfLogger();
myWcfClient.Endpoint.EndpointBehaviors.Add(logger);
try
{
// ... call WCF method that returns invalid XML
}
catch (Exception ex)
{
Console.WriteLine(ex.ToString());
}
File.SaveAllBytes(#"C:\temp\raw_response.bin", logger.RawLastResponseBytes);
// Use the exception message and examine raw_response.bin with
// a hex editor to find the problem.
UPDATE
Some others that have run into this issue appear to have created a Customer Message Encoder.
A message encoding binding element serializes an outgoing Message and
passes it to the transport, or receives the serialized form of a
message from the transport and passes it to the protocol layer if
present, or to the application, if not present.

WCF client inspector get name of web service it calls or calling method name or type or something

I have implemented a WCF inspector in my client application that consumes numerous web services.
I am using this inspector as a logging mechanism to log calls sent from the application to those web services and the responses they give back.
public class WcfClientInterceptor : IClientMessageInspector
{
protected static readonly ILog log4net = LogManager.GetLogger(MethodBase.GetCurrentMethod().DeclaringType);
private static readonly IMessageLogger Logger = new Log4NetLogger();
private MessageLogEntry LogEntry;// = new MessageLogEntry();
public void AfterReceiveReply(ref Message reply, object correlationState)
{
if (Logger.IsLogEnabled)
{
LogEntry.ResponseBody = reply.ToString();
Logger.Log(LogEntry);
}
}
public object BeforeSendRequest(ref Message request, IClientChannel channel)
{
if (Logger.IsLogEnabled)
{
LogEntry = LogEntry ?? new MessageLogEntry();
//instanceContext.GetServiceInstance().GetType().Name
//LogEntry.WebServiceIdentity = request.Headers.Action;
LogEntry.WebServiceIdentity = OperationContext.Current.IncomingMessageHeaders.Action;
LogEntry.RequestBody = request.ToString();
}
return null;
}
}
My problem is that I don't know what web service is called. I want to get some kind of reference to them and log it.
This is the only method that works request.Headers.Action but it doesn't always work. Most of the time this is String.Empty.
OperationContext.Current is null which I understand is normal on the client side.
Is there any other way of getting the name of the webservice that is called?
or the name of the calling method? or something?
Thank you
it works with LogEntry.WebServiceIdentity = request.Headers.Action;
I made a mistake

Categories