Is there any way to track if end-point is available for Tcp Sink logging ?
For example locally on my machine I do not have FileBeat setup, while its working on Staging machine.
The way I initialize Logger
private readonly ILogger _tcpLogger;
public TcpClient(IOptions<ElasticSearchConfig> tcpClientConfig)
{
var ip = IPAddress.Parse(tcpClientConfig.Value.TcpClientConfig.IpAddress);
_tcpLogger = new LoggerConfiguration()
.WriteTo.TCPSink(ip, tcpClientConfig.Value.TcpClientConfig.Port, new TcpOutputFormatter())
.CreateLogger();
}
and simple method just to submit log
public void SubmitLog(string json)
{
_tcpLogger.Information(json);
}
And in my case when its submitting json string locally, it just goes nowhere and I would like to get an exeption/message back.
ideally on json submit, but during initialization is Ok.
Writing to a Serilog logger is meant to be a safe operation and never throw exceptions, and that's by design. Thus any exceptions that happen when sending those messages would only appear in the SelfLog - if you enable it.
e.g.
// Write Serilog errors to the Console
Serilog.Debugging.SelfLog.Enable(msg => Console.WriteLine(msg));
The example above is, of course, just to illustrate the SelfLog feature... You'll choose where/how to display or store these error messages.
Now, if the operation you're logging is important enough that you'd want to guarantee it succeeds (or throws exception if it doesn't) then you should use Audit Logging, i.e. Use .AuditTo.TCPSink(...) instead of .WriteTo.TCPSink(...)
Related
I'm trying to use the IdempotentSagas in Rebus with MongoDb as a storage.
I enable idempotency when configuring Rebus like this:
Configure
...
.Options( o => { o.EnableIdempotentSagas(); } )
...
.Sagas( s => { s.StoreInMongoDb( mongoDatabase ); } )
I can see in debug that (during handling the message) the property IdempotencyData in the IdempotentSagaData instance stores the handled message id.
But when the saga data gets persisted, the IdempotencyData is always stored as an empty document:
{
"_id" : NUUID("0aa63d69-f8f9-46bd-ab29-f1e46411a166"),
"Revision" : 1,
"IdempotencyData" : {},
...
}
and thus it always appears empty when the saga data is loaded from the storage to handle a message.
It seems that this neglects all the idempotency checks, and later redelivered messages will be handled as if they are completely new. But the IdempotencyData class seems to be designed in a way which prevents it from being serialized by the default MongoDb BsonSerializer (get-only properties, private backing fields).
Is it an intentional behavior? Maybe I'm missing some configuration step which would allow the idempotency data to be persisted?
Thanks in advance for your help.
Rebus (all versions prior to Rebus 5.0.0-b14) had a BSON serializer-unfriendly IdempotencyData, thus making it impossible to properly roundtrip this pretty important piece of data when using IIdempotentSagaData.
It has been fixed in Rebus.MongoDb 5.0.0-b02 and Rebus 5.0.0-b14.
Rebus' IdempotencyData now has proper constructors in place, allowing for serializers to initialize the entire state that way.
Rebus.MongoDb now registers appropriate class maps during initialization of the saga storage.
Given the following middleware:
public class RequestDurationMiddleware
{
private readonly RequestDelegate _next;
private readonly ILogger<RequestDurationMiddleware> _logger;
public RequestDurationMiddleware(RequestDelegate next, ILogger<RequestDurationMiddleware> logger)
{
_next = next;
_logger = logger;
}
public async Task Invoke(HttpContext context)
{
var watch = Stopwatch.StartNew();
await _next.Invoke(context);
watch.Stop();
_logger.LogTrace("{duration}ms", watch.ElapsedMilliseconds);
}
}
Because of the pipeline, it occurs before the end of pipeline and logs different times:
WebApi.Middlewares.RequestDurationMiddleware 2018-01-10 15:00:16.372 -02:00 [Verbose] 382ms
Microsoft.AspNetCore.Server.Kestrel 2018-01-10 15:00:16.374 -02:00 [Debug] Connection id ""0HLAO9CRJUV0C"" completed keep alive response.
Microsoft.AspNetCore.Hosting.Internal.WebHost 2018-01-10 15:00:16.391 -02:00 [Information] "Request finished in 405.1196ms 400 application/json; charset=utf-8"
How can I capture the actual request execution time from WebHost (405.1196ms in the example) value in this case? I want to store this value in database or use it elsewhere.
I thought this question was really interesting, so I looked into this for a bit to figure out how the WebHost is actually measuring and displaying that request time. Bottom line is: There is neither a good nor an easy nor a pretty way to get this information, and everything feels like a hack. But follow along if you’re still interested.
When the application is started, the WebHostBuilder constructs the WebHost which in turn creates the HostingApplication. That’s basically the root component that is responsible to respond to incoming requests. It is the component that will invoke the middleware pipeline when a request comes in.
It is also the component that will create HostingApplicationDiagnostics which allows to collect diagnostics about the request handling. At the beginning of the request, the HostingApplication will call HostingApplicationDiagnostics.BeginRequest, and at the end of the request, it will call HostingApplicationDiagnostics.RequestEnd.
Not that surprisingly, HostingApplicationDiagnostics is the thing that will measure the request duration and also log that message for the WebHost that you have been seeing. So this is the class that we have to inspect more closely to figure out how to get the information.
There are two things the diagnostics object uses to report diagnostics information: A logger, and a DiagnosticListener.
Diagnostic listener
The DiagnosticListener is an interesting thing: It is basically a general event sink that you can just raise events on. And other objects can then subscribe to it to listen to these events. So this almost sounds perfect for our purpose!
The DiagnosticListener object that the HostingApplicationDiagnostics uses is passed on by the WebHost and it actually gets resolved from dependency injection. Since it is registered by the WebHostBuilder as a singleton, we can actually just resolve the listener from dependency injection and subscribe to its events. So let’s just do that in our Startup:
public void ConfigureServices(IServiceCollection services)
{
// …
// register our observer
services.AddSingleton<DiagnosticObserver>();
}
public void Configure(IApplicationBuilder app, IHostingEnvironment env,
// we inject both the DiagnosticListener and our DiagnosticObserver here
DiagnosticListener diagnosticListenerSource, DiagnosticObserver diagnosticObserver)
{
// subscribe to the listener
diagnosticListenerSource.Subscribe(diagnosticObserver);
// …
}
That’s already enough to get our DiagnosticObserver running. Our observer needs to implement IObserver<KeyValuePair<string, object>>. When an event occurs, we will get a key-value-pair where the key is an identifier for the event, and the value is a custom object that is passed by the HostingApplicationDiagnostics.
But before we implement our observer, we should actually look at what kind of events HostingApplicationDiagnostics actually raises.
Unfortunately, when the request ends, the event that is raised on the diagnostic lister just gets passed the end timestamp, so we would also need to listen to the event that is raised at the beginning of the request to read the start timestamp. But that would introduce state into our observer which is something we want to avoid here. In addition, the actual event name constants are prefixed with Deprecated which might be an indicator that we should avoid using these.
The preferred way is to use activities which are also closely related to the diagnostic observer. Activities are apparently states that track, well, activities as they appear in the application. They are started and stopped at some point, and also already record how long they run on their own. So we can just make our observer listen to the stop event for the activity to get notified when its done:
public class DiagnosticObserver : IObserver<KeyValuePair<string, object>>
{
private readonly ILogger<DiagnosticObserver> _logger;
public DiagnosticObserver(ILogger<DiagnosticObserver> logger)
{
_logger = logger;
}
public void OnCompleted() { }
public void OnError(Exception error) { }
public void OnNext(KeyValuePair<string, object> value)
{
if (value.Key == "Microsoft.AspNetCore.Hosting.HttpRequestIn.Stop")
{
var httpContext = value.Value.GetType().GetProperty("HttpContext")?.GetValue(value.Value) as HttpContext;
var activity = Activity.Current;
_logger.LogWarning("Request ended for {RequestPath} in {Duration} ms",
httpContext.Request.Path, activity.Duration.TotalMilliseconds);
}
}
}
Unfortunately there is just no solution without downsides… I found this solution to be very inaccurate for parallel requests (e.g. when opening a page that has also images or scripts which are requested in parallel). This is likely due to the fact that we are using a static Activity.Current to get the activity. However there does not really seem to be a way to get just the activity for a single request, e.g. from the key value pair that was passed.
So I went back and tried my original idea again, using those deprecated events. The way I understood it is btw. that they are just deprecated because using activities is recommended, not because they will be removed soon (of course we are working with implementation details and an internal class here, so these things could change at any time). To avoid problems with concurrency, we need to make sure we store the state inside of the HTTP context (instead of a class field):
private const string StartTimestampKey = "DiagnosticObserver_StartTimestamp";
public void OnNext(KeyValuePair<string, object> value)
{
if (value.Key == "Microsoft.AspNetCore.Hosting.BeginRequest")
{
var httpContext = (HttpContext)value.Value.GetType().GetProperty("httpContext").GetValue(value.Value);
httpContext.Items[StartTimestampKey] = (long)value.Value.GetType().GetProperty("timestamp").GetValue(value.Value);
}
else if (value.Key == "Microsoft.AspNetCore.Hosting.EndRequest")
{
var httpContext = (HttpContext)value.Value.GetType().GetProperty("httpContext").GetValue(value.Value);
var endTimestamp = (long)value.Value.GetType().GetProperty("timestamp").GetValue(value.Value);
var startTimestamp = (long)httpContext.Items[StartTimestampKey];
var duration = new TimeSpan((long)((endTimestamp - startTimestamp) * TimeSpan.TicksPerSecond / (double)Stopwatch.Frequency));
_logger.LogWarning("Request ended for {RequestPath} in {Duration} ms",
httpContext.Request.Path, duration.TotalMilliseconds);
}
}
When running this, we do actually get accurate results and we also have access to the HttpContext which we can use to identify the request. Of course, the overhead that’s involved here is very apparent: Reflection to access property values, having to store information in HttpContext.Items, the whole observer thing in general… that’s probably not a very performant way to do this.
Futher reading on diagnostic source and activities: DiagnosticSource Users Guid and Activity User Guide.
Logging
Somewhere above I mentioned that the HostingApplicationDiagnostics also reports the information to the logging facilities. Of course: This is what we are seeing in the console after all. And if we look at the implementation, we can see that this already calculates the proper duration here. And since this is structured logging, we could use this to grab that information.
So let’s attempt to write a custom logger that checks for that exact state object and see what we can do:
public class RequestDurationLogger : ILogger, ILoggerProvider
{
public ILogger CreateLogger(string categoryName) => this;
public void Dispose() { }
public IDisposable BeginScope<TState>(TState state) => NullDisposable.Instance;
public bool IsEnabled(LogLevel logLevel) => true;
public void Log<TState>(LogLevel logLevel, EventId eventId, TState state, Exception exception, Func<TState, Exception, string> formatter)
{
if (state.GetType().FullName == "Microsoft.AspNetCore.Hosting.Internal.HostingRequestFinishedLog" &&
state is IReadOnlyList<KeyValuePair<string, object>> values &&
values.FirstOrDefault(kv => kv.Key == "ElapsedMilliseconds").Value is double milliseconds)
{
Console.WriteLine($"Request took {milliseconds} ms");
}
}
private class NullDisposable : IDisposable
{
public static readonly NullDisposable Instance = new NullDisposable();
public void Dispose() { }
}
}
Unfortunately (you probably love this word by now, right?), the state class HostingRequestFinishedLog is internal, so we cannot use it directly. So we have to use reflection to identify it. But we just need its name, then we can extract the value from the read-only list.
Now all we need to do is register that logger (provider) with the web host:
WebHost.CreateDefaultBuilder(args)
.ConfigureLogging(logging =>
{
logging.AddProvider(new RequestDurationLogger());
})
.UseStartup<Startup>()
.Build();
And that’s actually all we need to be able to access the exact same information that the standard logging also has.
However, there are two problems: We don’t have a HttpContext here, so we cannot get information about which request this duration actually belongs to. And as you can see in the HostingApplicationDiagnostics, this logging call is actually only made when the log level is at least Information.
We could get the HttpContext by reading the private field _httpContext using reflection but there is just nothing we can do about the log level. And of course, the fact that we are creating a logger to grab information from one specific logging call is a super hack and probably not a good idea anyway.
Conclusion
So, this is all terrible. There simply is no clean way to retrieve this information from the HostingApplicationDiagnostics. And we also have to keep in mind that the diagnostics stuff actually only runs when it’s enabled. And performance critical applications will likely disable it at one point or another. In any way, using this information for anything outside of diagnostics would be a bad idea since it’s just too fragile in general.
So what is the better solution? A solution that works outsid of a diagnostics context? A simple middleware that runs early; just like you have already used. Yes, this is likely not as accurate as it will leave out a few paths from the outer request handling pipeline but it will still be an accurate measurement for the actual application code. After all, if we wanted to measure framework performance, we would have to measure it from the outside anyway: as a client, making requests (just like the benchmarks work).
And btw. this is also how Stack Overflow’s own MiniProfiler works. You just register the middleware early and that’s it.
I am using NServicebus(version 4.6.3) with SQLTransport in my ASP.net web api project. I have different connectionstrings for the queues for different environments (Dev,QA,etc). My configuration looks like below:
public class BusConfigurator
{
public static IStartableBus Bus { get; private set; }
public static void DisposeBus()
{
if (Bus == null)
return;
Bus.Shutdown();
Bus.Dispose();
Bus = null;
}
public static void InitializeServiceBus(string connectionString)
{
var configure = Configure.With()
.DefineEndpointName("MyEndPoint")
.Log4Net(new DebugAppender { Threshold = Level.Warn })
.UseTransport<SqlServer>(connectionString)
.PurgeOnStartup(false)
.SetDefaultTransactionLevel()
.UnicastBus(); // Error is thrown here on second call
configure.MyCustomSQLServerPersistence();
Bus = configure.CreateBus();
}
public static void StartBus()
{
Bus.Start(() => Configure.Instance.ForInstallationOn<NServiceBus.Installation.Environments.Windows>().Install());
}
}
I have a dropdown in the app so that the user can select the environment. Based on the selection, I want to reconfigure the bus. So, I call DisposeBus then pass the connection string to the IntializeServiceBus method followed by the startBus. It works first time but throws error below when it gets called again with different connectionstring:
Unable to set the value for key: NServiceBus.Transport.ConnectionString. The settings has been locked for modifications. Please move any configuration code earlier in the configuration pipeline
Source=NServiceBus.Core
Line=0
BareMessage=Unable to set the value for key: NServiceBus.Transport.ConnectionString. The settings has been locked for modifications. Please move any configuration code earlier in the configuration pipeline
Is NServicebus intended to be used/configured this way? (I am guessing probably not) If not then is there a workaround/different approach for this?
In V4 or below, there is no way to do it by normal human means. There is only one Bus per AppDomain. All of the configuration API is static, so if you try, you get exactly the problems you ran into.
By "human means", I mean that it might be possible to do something crazy with spinning up a new AppDomain within your process, setting up a Bus within that, and then tearing it down when you're finished. It might be possible. I haven't tried it. I wouldn't recommend it.
In V5, the configuration API is completely redesigned, is not static, and so this is possible:
var cfg = new BusConfiguration();
// Set up all the settings with the new V5 Configuration API
using (var justOneBus = NServiceBus.Bus.Create(cfg).Start())
{
// Use justOneBus, then it gets disposed when done.
}
That's right. It's disposable. Then you can do it again. In your case you wouldn't want to put it in a using block - you would want to set it up somewhere, and when the dropdown gets switched, call Dispose on the current instance and rebuild it with the new parameters.
Keep in mind, however, that the Bus is still pretty expensive to create. It's definitely still something you want to treat as an application-wide singleton (or singleton-like) instance. You definitely wouldn't want to spin up a separate one per web request.
We are currently using self-hosted NServiceBus to handle queuable messages in our system. Right now there are instances where a queued message might fail on the first try and work on the automatic retries.
Right now we are logging on all failures, but we really don't care (at least for alerts) if a message failed the first time but worked on a re-try. What we do want to get alerted to is if all retries failed and a message goes into the error queue.
Is there any way native to NServiceBus to have code run when it's moving a message to the error queue?
If you are using the rest of the Service Platform (and you should!) that means that your error queue will have ServiceControl sitting on top of it, reading messages out of error and audit and persisting the details to its database so that it can serve up that information via its REST API to ServicePulse (for monitoring system health and uptime) and ServiceInsight (for exploration and debugging.)
Assuming you are using ServiceControl, it's pretty easy to have an endpoint subscribe to MessageFailed events that are published by ServiceControl. I explained how to do it in my blog post Failed Message Notification with ServiceControl.
This way, each endpoint doesn't have to be responsible for this task, and it is accomplished asynchronously by a centralized error monitoring endpoint.
It appears the correct way to do this is to create a custom implementation of IManageMessageFailures and registering the custom fault manager curing configuration time.
An example of this is:
public class CustomFaultManager : IManageMessageFailures
{
private readonly IManageMessageFailures faultManager;
static CustomFaultManager()
{
Configure.Instance.MessageForwardingInCaseOfFault();
}
public CustomFaultManager()
{
faultManager = new FaultManager();
((FaultManager)faultManager).ErrorQueue = ConfigureFaultsForwarder.ErrorQueue;
}
void IManageMessageFailures.SerializationFailedForMessage(TransportMessage message, Exception e)
{
faultManager.SerializationFailedForMessage(message, e);
}
void IManageMessageFailures.ProcessingAlwaysFailsForMessage(TransportMessage message, Exception e)
{
faultManager.ProcessingAlwaysFailsForMessage(message, e);
//Custom code goes here
}
void IManageMessageFailures.Init(Address address)
{
faultManager.Init(address);
}
}
from https://github.com/Particular/NServiceBus/issues/463
I'm trying to write a custom trace listener for Enterprise Library Logging which sends all log messages to an arbitrary WCF endpoint. The idea behind this is that I can set up a simple console app, etc at the other end which prints out all log messages in real time.
My question is in two parts:
Is there a mechanism to do this already? I already looked at the MSMQ listener and I'm not interested in using that because I may have a need to use a different protocol/binding at some point.
The way I have it implemented below - is it efficient enough or is there a better way? My concern is that every time a message comes through from the Logger (which may be frequent) I'm opening a new channel and then slamming it shut. Will this cause performance issues?
In my sample RemoteClient derives from ClientBase<T>.
[ConfigurationElementType(typeof(CustomTraceListenerData))]
public class RemoteTraceListener : CustomTraceListener
{
public override void Write(string message)
{
RemoteClient client = new RemoteClient();
client.Open();
client.Write(message);
client.Close();
}
public override void WriteLine(string message)
{
RemoteClient client = new RemoteClient();
client.Open();
client.WriteLine(message);
client.Close();
}
public override void TraceData(TraceEventCache eventCache, string source, TraceEventType eventType, int id, object data)
{
if (data is LogEntry && this.Formatter != null)
{
WriteLine(this.Formatter.Format(data as LogEntry));
}
else
{
WriteLine(data.ToString());
}
}
}
How often is this writing? I suggest WCF streaming as a better alternative of you're going to be logging frequently.
Failing that, it's probably a good idea to keep the client instance around as long as possible. You could try pooling it.
I found an open-source project called 'CLog' which does exactly what I'm looking for: http://clog.codeplex.com/.
A brief glance at the source code shows that he's using a singleton object to keep track of all the open channels that will receive log messages, and he's going with ChannelFactory<TChannel> as opposed to ClientBase<T> to instantiate each channel proxy. There are some threading implications that need to be addressed but I still think this is the way to fly.
It shouldn't be too hard to use this as a starting point for the implementation of my own custom trace listener which logs to WCF endpoints.
I think you should use a SQL database on witch you should log to because if you logg in a console app you could not see for examle something before 2 days.While in SQL you can make a quote and get the right data you need.
Another solution is to use the log4net project.