I have ran into a problem where a long running singleton added to HttpApplicationState which does some data masking (GDPR), stops masking data after running in the background for some time.
It's hard to debug because it only happens in our UAT enviroment and it usually happens overnight.
The problem is that the data masking library is third party, and is still work in progress (or at the end of that work in progress).
But I'd appreciate if anyone with better GC knowledge could look at the init code below, and confirm this is out of GC domain.
The Translator.GetInstance() is a Lazy loader of the GDPR masking/translation singleton. So it's initialized the first time the user masks/unmasks the data.
protected void Application_Start()
{
if (Translator)
{
Application["MaskDataUtility"] = new MaskDataUtility(Translator.GetInstance());
}
else
{
Application["MaskDataUtility"] = new MaskDataUtility(new CustomTranslator());
}
}
Related
I'm using MVVMlight SimpleIoC container, particularly to store quite a large volume of data, which is accessible throughout the application. When the application is started, I launch an asynchronous process, which prepares the data.
private MethodOfOriginalThread()
{
(new Thread(this.LoadRegistry)).Start();
}
private void LoadRegistry()
{
int meetingId = SimpleIoc.Default.GetInstance<MeetingDetails>().MeetingId;
List<Shareholder> registry = this.dataService.SearchRegistry(meetingId, string.Empty);
SimpleIoc.Default.Register(() => registry);
}
The process of obtaining the data takes approximately 10 seconds. After the process is complete, I save the data to container, as seen in the code above.
Later from a different viewModel I'm accessing this data:
this.shareholders =
new NotifyObservableCollection<Shareholder>(
SimpleIoc.Default.GetInstance<List<Shareholder>>()
);
The problem arises when this access is made before the actual load completes. Since the viemodels are very loosely coupled, I cannot refer to one from another. Sending messages may result in hyge amount of code, since these results are widely used in the application, and each viewmodel should have the message handling code. How can I check and wait, whether the data has been successfully loaded? An endless loop, which waits for required object in the container, sounds halfway ok, since the operation should eventually be completed, but I'm worried that this is not the most elegant solution. And also I need to let the user have some feedback that the data is still being loaded, so he waits patiently, and does not press too many buttons. This is quite challenging with an endless loop and frozen application.
I am creating a Windows service in C#. Its purpose is to consume info from a feed on the Internet. I get the data by using zeromq's pub/sub architecture (my service is a subscriber only). To debug the service I "host" it in a WPF control panel. This allows me to start, run, and stop the service without having to install it. The problem I am seeing is that when I call my stop method it appears as though the service continues to write to the database. I know this because I put a Debug.WriteLine() where the writing occurs.
More info on the service:
I am attempting to construct my service in a fashion that allows it to write to the database asynchronously. This is accomplished by using a combination of threads and the ThreadPool.
public void StartDataReceiver() // Entry point to service from WPF host
{
// setup zmq subscriber socket
receiverThread = new Tread(SpawnReceivers);
receiverThread.Start();
}
internal void SpawnReceivers()
{
while(!stopEvent.WaitOne(0))
{
ThreadPool.QueueUserWorkItem(new WaitCallback(ProcessReceivedData), subscriber.Recv()); // subscriber.Recv() blocks when there is no data to receive (according to the zmq docs) so this loop should remain under control, and threads only created in the pool when there is data to process.
}
}
internal void ProcessReceivedData(Object recvdData)
{
// cast recvdData from object -> byte[]
// convert byte[] -> JSON string
// deserialize JSON -> MyData
using (MyDataEntities context = new MyDataEntities())
{
// build up EF model object
Debug.WriteLine("Write obj to db...");
context.MyDatas.Add(myEFModel);
context.SaveChanges();
}
}
internal void QData(Object recvdData)
{
Debug.WriteLine("Queued obj in queue...");
q.Enqueue((byte[])recvdData);
}
public void StopDataReceiver()
{
stopEvent.Set();
receiverThread.Join();
subscriber.Dispose();
zmqContext.Dispose();
stopEvent.Reset();
}
The above code are the methods that I am concerned with. When I debug the WPF host, and the method ProcessReceivedData is set to be queued in the thread pool everything seems to work as expected, until I stop the service by calling StopDataReceiver. As far as I can tell the thread pool never queues any more threads (I checked this by placing a break point on that line), but I continue to see "Write obj to db..." in the output window and when I 'Break All' in the debugger a little green arrow appears on the context.SaveChanges(); line indicating that is where execution is currently halted. When I test some more, and have the thread pool queue up the method QData everything seems to work as expected. I see "Queued obj in queue..." messages in the output window until I stop the service. Once I do no more messages in the output window.
TL;DR:
I don't know how to determine if the Entity Framework is just slowing things way down and the messages I am seeing are just the thread pool clearing its backlog of work items, or if there is something larger at play. How do I go about solving something like this?
Would a better solution be to queue the incoming JSON strings as byte[] like I do in the QData method then have the thread pool queue up a different method to work on clearing the queue. I feel that that solution will only shift the problem around and not actually solve it.
Could another solution be to write a new service dedicated to clearing that queue? The problem I see with writing another service would be that I would probably have to use WCF (or possibly zmq) to communicate between the two services which would obviously add overhead and possibly become less performant.
I see the critical section in all of this being the part of getting the data off the wire fast enough because the publisher I am subscribed to is set to begin discarding messages if my subscriber can't keep up.
We're using the following pattern to handle caching of universal objects for our asp.net application.
private object SystemConfigurationCacheLock = new object();
public SystemConfiguration SystemConfiguration
{
get
{
if (HttpContext.Current.Cache["SystemConfiguration"] == null)
lock (SystemConfigurationCacheLock)
{
if (HttpContext.Current.Cache["SystemConfiguration"] == null)
HttpContext.Current.Cache.Insert("SystemConfiguration", GetSystemConfiguration(), null, DateTime.Now.AddMinutes(1), Cache.NoSlidingExpiration, new CacheItemUpdateCallback(SystemConfigurationCacheItemUpdateCallback));
}
return HttpContext.Current.Cache["SystemConfiguration"] as SystemConfiguration;
}
}
private void SystemConfigurationCacheItemUpdateCallback(string key, CacheItemUpdateReason reason, out object expensiveObject, out CacheDependency dependency, out DateTime absoluteExpiration, out TimeSpan slidingExpiration)
{
dependency = null;
absoluteExpiration = DateTime.Now.AddMinutes(1);
slidingExpiration = Cache.NoSlidingExpiration;
expensiveObject = GetSystemConfiguration();
}
private SystemConfiguration GetSystemConfiguration()
{
//Load system configuration
}
The problem is that when under load (~100,000 users) we see a huge jump in TTFB as the CacheItemUpdateCallback blocks all the other threads from executing until it has finished refreshing the cache from the database.
So what I figured we needed is solution that when the first thread after an expiry of the cache attempts to access it, an asynchronous thread is fired off to update the cache but still allows all other executing threads to read from the old cache until it has sucessfully updated.
Is there anything built into the .NET framework that can natively handle what I'm asking, or will I have to write it from scratch? Your thoughts please...
A couple of things...
The use of the HttpContext.Current.Cache is incidental and not necessarily essential as we've got no problem using private members on a singleton to hold the cached data.
Please don't comment on the cache times, SPROC effeciency, why we're caching in the first place etc as it's not relevent. Thanks!
AppFabric might be a good fit for what you're looking for.
http://msdn.microsoft.com/en-us/windowsserver/ee695849
http://msdn.microsoft.com/en-us/library/ff383731.aspx
So it turns out after several hours of investigation that the problem is not the CacheItemUpdateCallback blocking other threads as I originally thought, in fact it did exactly what I wanted it to asynchronously but it was the garbage collector stopping everything to clean up the LOH.
I built a little web application that displays charts. I was thinking that it might be useful for the superuser of the app to do a complete data refresh, however this process takes around 10 minutes to complete. I was thinking perhaps the user could click a button that would start off a new thread to do a data refresh and subsequent clicks would kill the thread and restart the data population process. The user would then be free to browse about the site and view the charts as their data is populated.
Is there a simple method of accomplishing something like this?
You can twist ASP.NET to do this sort of thing, but it violates a few good general rules for ASP.NET development -- and could really cause problems in a server farm.
So, the most obvious route is to do this work in a web service. You can have the method return a chunk of HTML if you want. You could also add status methods to see how the thread is progressing.
Other options include: Handing the intense processing off to a database server (sounds like this might be a good use of OLAP) or, another cheap trick might be to set up the click to fire off a scheduled task that runs on the server. Can you provide some additional detail about the environment? Single server? Data storage platform, version of .net?
Ok, I didn't use either answer so here is what I did. I decided that it would be better if subsequent clicks would terminate instead of the currently executing one. Thanks for your answers guys.
//code behind
protected void butRefreshData_Click(object sender, EventArgs e)
{
Thread t = new Thread(new ThreadStart(DataRepopulater.DataRepopulater.RepopulateDatabase));
t.Start();
}
//DataRepopulater.cs
namespace DataRepopulater
{
public static class DataRepopulater
{
private static string myLock = "My Lock";
public static void RepopulateDatabase()
{
if(Monitor.TryEnter(myLock))
{
DoWork();
Monitor.Exit(myLock);
}
}
}
I've got a simple web application using ASP.NET MVC3 and Ninject.Web.MVC (the MVC3 version).
The whole thing is working fine, except when the application ends. Whenever it ends, the kernel is disposed, as seen in Application_End() in NinjectHttpApplication:
Reflector tells me this:
public void Application_End()
{
lock (this)
{
if (kernel != null)
{
kernel.Dispose();
kernel = null;
}
this.OnApplicationStopped();
}
}
What happens is that my webserver goes down with a StackOverflowException (I tried both IIS7 and the built-in webserver in VS2010). I can only assume this is where it's going wrong, as I haven't written any code myself on application end.
I figured out that the Kernel knows how to resolve IKernel (which returns the Kernel itself), might this be something that could cause the stack overflow? I could imagine something like this happens:
Kernel.Dispose()
Dispose all instances in the kernel
hey! look at this, the kernel is also in the kernel. Return to step 1.
In other words, the kernel gets disposed, disposes all references it holds (which includes a self-reference), which causes it to dispose itself.
Does this make any sense?
Edit:
It seems the problem is in NinjectHttpApplication. Take a look at this activation code:
public void Application_Start()
{
lock (this)
{
kernel = this.CreateKernel();
...
kernel.Bind<IResolutionRoot>().ToConstant(kernel).InSingletonScope();
...
}
}
It seems ok, but what's happening now is that whenever an IResolutionRoot is called, kernel is cached within itself. When disposing the kernel, the cache is emptied which disposes all cached objects, which causes a circular reference.
A simple solution for NinjectHttpApplication would be to simply change the binding. Change the constant binding to a method one:
kernel.Bind<IResolutionRoot>().ToConstant(kernel).InSingletonScope();
becomes
kernel.Bind<IResolutionRoot>().ToMethod(x => this.Kernel);
This solves the problem, but I am not sure if the whole circular dispose caching issue is a bug in ninject.
I encountered the same issue.
I ended up copying the code for NinjectHttpApplication and removing Kernel.Dispose() in the Application_End function.
public void Application_End()
{
lock (this)
{
if (kernel != null)
{
//kernel.Dispose();
kernel = null;
}
this.OnApplicationStopped();
}
}
That should fix the error. Not sure if there is a planned fix for it though.
There was a bug in MVC3. It's fixed in the latest revision and will be part of the RC2 comming next week. In the mean time take the build from the build server http://teamcity.codebetter.com