I've been getting two exceptions at random times in my asp.net mvc code running on iis7:
Exception type: InvalidOperationException
Exception message: Collection was modified; enumeration operation may not execute.
at System.ThrowHelper.ThrowInvalidOperationException(ExceptionResource resource)
at System.Collections.Generic.List'1.Enumerator.MoveNextRare()
at System.Collections.Generic.List'1.Enumerator.MoveNext()
at System.Web.Routing.RouteCollection.GetRouteData(HttpContextBase httpContext)
at System.Web.Routing.UrlRoutingModule.PostResolveRequestCache(HttpContextBase context)
at System.Web.Routing.UrlRoutingModule.OnApplicationPostResolveRequestCache(Object sender, EventArgs e)
at System.Web.HttpApplication.SyncEventExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)
and
Exception type: NullReferenceException
Exception message: Object reference not set to an instance of an object.
at System.Web.Routing.RouteCollection.GetRouteData(HttpContextBase httpContext)
at System.Web.Routing.UrlRoutingModule.PostResolveRequestCache(HttpContextBase context)
at System.Web.Routing.UrlRoutingModule.OnApplicationPostResolveRequestCache(Object sender, EventArgs e)
at System.Web.HttpApplication.SyncEventExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)
It's not consistently reproducible, but I assume it's something changing (or corrupting) RouteTable.Routes. The only place I access RouteTable.Routes in my project is in Global.asax.cs and I know that the code there is only being called once, so it's not the problem. Any idea on how to track it down?
In my case, it ended up being a HttpModule: Ext.Direct.Mvc (Ext.Direct for ASP.NET MVC). This module had a bug (Fixed in version 0.8.0) which registered routes again every time Init() was called for the IHttpModule. (which might be called multiple times). If the timing was right, it would corrupt the RouteTable.Routes collection, and cause one of the two exceptions above.
That error is consistent with a collection not being thread-safe in .Net.
As per the MSDN article on RouteCollection.GetWriteLock(), RouteTable.Routes is not thread-safe. This issue is because some piece of code tried to modify the RouteTable.Routes in a manner which was not thread-safe.
The best practice is to modify the RouteTable.Routes in the Application_Start. If you have to modify it in a way which can be accessed by multiple threads at the same time, make sure you make it in a thread-safe way. Check the example from MSDN, copied below for ease of reference:
using (RouteTable.Routes.GetWriteLock())
{
Route newRoute = new Route("{action}/{id}", new ReportRouteHandler());
RouteTable.Routes.Add(newRoute);
}
Same applies for when reading from the RouteTable - RouteCollection.GetReadLock().
Probably, if this was fixed by removing the HttpModule, it is because this HttpModule does not implement such locking mechanisms.
Other answers explained WHAT happens but provided little details on how to actually track it down. It may not be your own code causing all this trouble, so Ctrl-F is not enough..
The idea
What makes solving it difficult, is the fact that you usually get errors on a different requests than the ones actually modifying RouteCollection. A generally safe call to UrlHelper.GenerateUrl() or the like would fail at seemingly random times. To track it down, I chose to move exceptions from victims to the culprit by forbidding changing routes after the initial set up.
Step 1: Implement custom RouteCollection
IF a flag is set then yell hard when someone tries to change routes. It's a crude implementation but you get the idea.
public class RestrictedRouteCollection : RouteCollection
{
public Boolean EnableRestrictions { get; set; }
protected override void InsertItem(Int32 index, RouteBase item)
{
if (EnableRestrictions) { throw new Exception("Unexpected route added."); }
base.InsertItem(index, item);
}
protected override void SetItem(Int32 index, RouteBase item)
{
if (EnableRestrictions) { throw new Exception("Unexpected change of RouteCollection item."); }
base.SetItem(index, item);
}
protected override void RemoveItem(Int32 index)
{
if (EnableRestrictions) { throw new Exception("Unexpected removal from RouteCollection, index: " + index); }
base.RemoveItem(index);
}
protected override void ClearItems()
{
if (EnableRestrictions) { throw new Exception("Unexpected clearing of routecollection."); }
base.ClearItems();
}
}
Step 2: Replace the default collection
In Application_start in Global.asax replace the default RouteCollection with your instance before the routes are set up. As there does not seem to be an API to replace it then we force it with reflection on the private fields:
var routeTable = new RestrictedRouteCollection();
var field = typeof(RouteTable)
.GetField("_instance", BindingFlags.Static | BindingFlags.NonPublic);
if (field == null) { throw new Exception("Expected field _instance was not found."); }
field.SetValue(null, routeTable);
Debug.Assert(RouteTable.Routes == routeTable);
Step3: Restrict changes once intended routes are set up
RouteConfig.RegisterRoutes(routeTable);
routeTable.EnableRestrictions = true;
All done! Now watch the bad request light up in exception logs.
A possible culprit: ImageResizer
In my case it was a 3-rd party component named ImageResizer (v4.0.4) whose internal MvcRoutingShimPlugin took the liberty of adding/removing a route without locking. This specific bug was already reported and fixed (though at this time not yet officially released).
Related
The main goal: I am creating a website (ASP.NET MVC 5) where I need to add some additional Authorization and redirection logic: For example, a user can only view page C after completing pages A and B. If they have completed page A and not B, and try to access page C, they will be redirected to page B.
After a good amount of research, my plan is to create a custom AuthorizationAttribute , and override OnAuthorization(). I want to do this responsibly, so I looked at the [source code][1] for this method, and want to only add logic to it, not take it away. I've started by copying that code over to my own subclass (which I include at the end of this post):
The problem is there are two elements here that I apparently cannot access:
MvcResources.AuthorizeAttribute_CannotUseWithinChildActionCache
CacheValidateHandler
The first is in System.Web.Mvc.Properties , and the error I get is that 'MvcResources is inaccessible due to its protection level'. All of the help I've seen online regarding this advises that the programmer change the access modifier for their class, but I can't since I didn't write this class: it's system code.
The second (CacheValidateHandler) 'does not exist in the current context'. It's a method in my parent class (AuthorizeAttribute), but it's private.
So is there something I'm missing? Does my subclass have to be in a special location (right now it's in a folder called Helpers) or do I have to do something different with namespaces? I'm still pretty new to C#. How can I safely override OnAuthorize if I'm not even able to repeat what the parent method does?
namespace MyApp.Helpers
{
public class MyAppAuth : AuthorizeAttribute
{
public virtual void OnAuthorization(AuthorizationContext filterContext)
{
if (filterContext == null)
{
throw new ArgumentNullException("filterContext");
}
if (OutputCacheAttribute.IsChildActionCacheActive(filterContext))
{
// If a child action cache block is active, we need to fail immediately, even if authorization
// would have succeeded. The reason is that there's no way to hook a callback to rerun
// authorization before the fragment is served from the cache, so we can't guarantee that this
// filter will be re-run on subsequent requests.
throw new InvalidOperationException(MvcResources.AuthorizeAttribute_CannotUseWithinChildActionCache);
}
bool skipAuthorization = filterContext.ActionDescriptor.IsDefined(typeof(AllowAnonymousAttribute), inherit: true)
|| filterContext.ActionDescriptor.ControllerDescriptor.IsDefined(typeof(AllowAnonymousAttribute), inherit: true);
if (skipAuthorization)
{
return;
}
if (AuthorizeCore(filterContext.HttpContext))
{
// ** IMPORTANT **
// Since we're performing authorization at the action level, the authorization code runs
// after the output caching module. In the worst case this could allow an authorized user
// to cause the page to be cached, then an unauthorized user would later be served the
// cached page. We work around this by telling proxies not to cache the sensitive page,
// then we hook our custom authorization code into the caching mechanism so that we have
// the final say on whether a page should be served from the cache.
HttpCachePolicyBase cachePolicy = filterContext.HttpContext.Response.Cache;
cachePolicy.SetProxyMaxAge(new TimeSpan(0));
cachePolicy.AddValidationCallback(CacheValidateHandler, null /* data */);
}
else
{
HandleUnauthorizedRequest(filterContext);
}
}
}
}
I believe I have a solution, but I would really love if someone with more experience could comment if what I'm doing is in any way unsafe.
For the first issue, I simply replaced MvcResources.AuthorizeAttribute_CannotUseWithinChildActionCache with a string, since this is one of the possible overloads of InvalidOperationException.
For the second issue, I copy-pasted the parent's private CacheValidateHandler() method:
private void CacheValidateHandler(HttpContext context, object data, ref HttpValidationStatus validationStatus)
{
validationStatus = OnCacheAuthorization(new HttpContextWrapper(context));
}
I'm consuming a clunky WCF server that occasionally throws various exceptions, and additionally returns some of its errors as string. I have no access to the server code at all.
I want to override the inner WCF-client request invocation method and handle all inner exceptions and hard-coded errors returned by the server and raise the Fault event if an error occurs, pseudo:
class MyClient : MyServiceSoapClient
{
protected override OnInvoke()
{
object result;
try
{
result = base.OnInvoke();
if(result == "Error")
{
//raise fault event
}
catch
{
//raise fault event
}
}
}
So that when I call myClient.GetHelloWorld(), it goes thru my overridden method.
How can this be achieved?
I know I don't have to use the generated client, but I don't want to re-implement all the contracts again, and I want to use the generated ClientBase subclass or at least its channel.
What I need is control over the inner request call method.
Update
I read this answer, and looks it's partially what I'm looking for, but I'm wondering if there is a way to attach an IErrorHandler to the consumer (client) code only, I want to add it to the ClientBase<TChannel> instance somehow.
Update
This article also looks very promising but it doesn't work. The applied attribute doesn't seem to take effect.
I can't find a way to add IServiceBehavior to the client side.
Update
I tried attaching an IErrorHandler via IEndpointBehavior.ApplyClientBehavior calling:
public void ApplyClientBehavior(ServiceEndpoint endpoint, ClientRuntime clientRuntime)
{
clientRuntime.CallbackDispatchRuntime.ChannelDispatcher.ErrorHandlers
.Add(new ErrorHandler());
}
(clientRuntime is a parameter), but exceptions are still thrown directly skipping MyErrorHandler.
ApplyDispatchBehavior isn't called at all.
Conclusion
I need to achieve two aspects:
Wrap all exceptions that might occur during the lifetime of a BaseClient<TChannel> and decide whether to handle them or throw them on. This should take care of all operation (the service I'm consuming exposes few dozens)
Parse all server-replies and throw exceptions for some of them, so they're forwarded as in statement 1.
You could use and modify the Exception Handling WCF Proxy Generator, more specifically, the base class that it uses. It's basic idea (check this description too) is to provide connection resilience by catching connection faults, and retrying the failed operation. As you can imagine, for this purpose it needs to be able to catch thrown exceptions, and also, it can inspect the result of calls.
The main functionality is given by the ExceptionHandlingProxyBase<T> base class, which you use instead of the ClientBase<T>. This base class has an Invoke method as follows, you'd need to modify that.
Simplified Invoke:
protected TResult Invoke<TResult>(string operationName, params object[] parameters)
{
this.Open();
MethodInfo methodInfo = GetMethod(operationName);
TResult result = default(TResult);
try
{
this.m_proxyRecreationLock.WaitOne(this.m_proxyRecreationLockWait);
result = (TResult)methodInfo.Invoke(m_channel, parameters);
}
catch (TargetInvocationException targetEx) // Invoke() always throws this type
{
CommunicationException commEx = targetEx.InnerException as CommunicationException;
if (commEx == null)
{
throw targetEx.InnerException; // not a communication exception, throw it
}
FaultException faultEx = commEx as FaultException;
if (faultEx != null)
{
throw targetEx.InnerException; // the service threw a fault, throw it
}
//... Retry logic
}
return result;
}
You'll need to modify the throw targetEx.InnerException; part to handle the exceptions as you need, and obviously the resturn value shoudl also be inspected for your needs. Other then that you can leave the retry logic or throw it away if you don't expect connection problems. There is another variant of the Invoke for void return methods.
Oh, and by the way, it works with duplex channels as well, there is another base class for those.
If you don't want to use the generator (it might not even work in newer versions of VS), then you could just take the base class for example from here, and generate the actual implementation class with T4 from your service interface.
If the service isn't returning a true exception, but just a message, you probably want to add a ClientMessageInspector as a new client behavior. Please see: https://msdn.microsoft.com/en-us/library/ms733786.aspx
I've ended up using something based on the answers in this question.
It sticks to the generated client code, and allows invocation of the operations generically.
The code is incomplete, feel free to fork and edit it. Please notify me if you found any bugs or made any updates.
It's pretty bulky so I'll just share the usage code:
using (var proxy = new ClientProxy<MyServiceSoapClientChannel, MyServiceSoapChannel>())
{
client.Exception += (sender, eventArgs) =>
{
//All the exceptions will get here, can be customized by overriding ClientProxy.
Console.WriteLine($#"A '{eventArgs.Exception.GetType()}' occurred
during operation '{eventArgs.Operation.Method.Name}'.");
eventArgs.Handled = true;
};
client.Invoke(client.Client.MyOperation, "arg1", "arg2");
}
I'm trying to add a custom IHttpActionInvoker to my WebAPI application in order to prevent the need for lots of repeated exception handling code in my action methods.
There really doesn't seem to be much out there about how to do this other than this article. After writing my IHttpActionInvoker as per the article I added this code:
GlobalConfiguration.Configuration.Services.Remove(typeof(IHttpActionInvoker),
GlobalConfiguration.Configuration.Services.GetActionInvoker());
GlobalConfiguration.Configuration.Services.Add(typeof(IHttpActionInvoker),
new MyApiControllerActionInvoker());
Into a method within my Global.asax file. Now when executing a call to my API I get the following exception raised at the Remove() method:
The service type IHttpActionInvoker is not supported
I guess I have two questions.
Considering there doesn't seen to be an awful lot out there about writing custom IHttpActionInvoker classes is this considered a good approach to solve exception handling in WebAPI applications?
Does anyone know why I would get such an exception when executing the Remove() method and how best to fix this particular issue?
I suffered the same error you describe when attempting to remove the service.
I discovered I didn't need to remove anything from the global config, as it appears if you've registered the interface in your container then it will resolve this first.
For example, I'm using SimpleInjector and in my global.asax I have this:
container.Register<IHttpActionInvoker , MyApiControllerActionInvoker >();
// Register the dependency resolver.
GlobalConfiguration.Configuration.DependencyResolver =
new SimpleInjectorWebApiDependencyResolver(container);
At runtime, it is resolving MyApiControllerActionInvoker dependency when required.
You can then perform exception handling in your customer ActionInvoker and any dependencies set in your constructor will be wired up correctly. The reason I was looking at the ActionInvoker was to get the constructor injection, since injecting into Attributes appears to require property injection.
Also rather than the remove/insert, replace seems to work. (in Global.asax)
GlobalConfiguration.Configuration.Services.Replace(typeof(IHttpActionInvoker), new MyApiControllerActionInvoker(fooService));
Have you considered registering an exception filter instead? Here's some documentation about that:
http://www.asp.net/web-api/overview/web-api-routing-and-actions/exception-handling
You shouldn't have to fall down to the action invoker layer if all you want to do is handle some exceptions in a particular way.
As for me it works with IActionInvoker instead of IHttpActionInvoker. As I understand, IHttpActionInvoker is used for the async api calls, isn't it?
public class RepControllerActionInvoker : ControllerActionInvoker
{
ILogger _log;
public RepControllerActionInvoker()
: base()
{
_log = DependencyResolver.Current.GetService<ILogger>();
}
public override bool InvokeAction(ControllerContext controllerContext, string actionName)
{
try
{
return base.InvokeAction(controllerContext, actionName);
}
catch (Exception e)
{
_log.Error(e);
throw new HttpException(500, "Internal error");
}
}
}
I've written a C# MVC 3 with NHibernate as the ORM and I'm having some odd exceptions thrown on most page loads. They seem to mostly relate to closed sessions and the like and I've checked most of the common issues but found little to help. Some of the exceptions include:
[[NHibernate.Util.ADOExceptionReporter]] : System.InvalidOperationException: There is already an open DataReader associated with this Command which must be closed first.
at System.Data.SqlClient.SqlInternalConnectionTds.ValidateConnectionForExecute(SqlCommand command)
[[NHibernate.Transaction.AdoTransaction]] : Begin transaction failed
System.Data.SqlClient.SqlException (0x80131904): The server failed to resume the transaction. Desc:3b00000006.
[[NHibernate.Transaction.AdoTransaction]] : Commit failed
System.NullReferenceException: Object reference not set to an instance of an object.
[[NHibernate.Transaction.AdoTransaction]] : Commit failed
System.Data.SqlClient.SqlException (0x80131904): The COMMIT TRANSACTION request has no corresponding BEGIN TRANSACTION.
[[NHibernate.Util.ADOExceptionReporter]] : System.InvalidOperationException: The transaction is either not associated with the current connection or has been completed.
at System.Data.SqlClient.SqlCommand.ValidateCommand(String method, Boolean async)
[[NHibernate.Transaction.AdoTransaction]] : Begin transaction failed
System.InvalidOperationException: SqlConnection does not support parallel transactions.
I apologise for the wall of exceptions, I suspect they are related but there could potentially be another error in the code as well causing one or two.
I don't like to use the word random for these things but I can't seem to track down any specific line of code that calls them, they just seem to appear at lines of code relating to ISession objects. I have even had a "Session is closed" exception thrown on the BeginTranscation method in my Global.asax file.
The application uses the web option of current_session_context_class in hibernate.cfg.xml.
My suspicion is that it is related to my session management code. The website usually loads around 10 simultaneous AJAX requests and the errors seem to occur more often when multiple pages are loading at the same time. There are two session factories, one for each database being used.
Here is my relevant Global.asax code:
protected void Application_Start()
{
AreaRegistration.RegisterAllAreas();
HibernatingRhinos.Profiler.Appender.NHibernate.NHibernateProfiler.Initialize();
RegisterGlobalFilters(GlobalFilters.Filters);
RegisterRoutes(RouteTable.Routes);
_sessionFactory = (new WebClientSessionManager()).MakeSessionFactory();
_sessionFactoryNotWeb = ClientSessionManager.MakeSessionFactory();
}
protected void Application_BeginRequest(object sender, EventArgs e)
{
_session = _sessionFactory.OpenSession();
_sessionNotWeb = _sessionFactoryNotWeb.OpenSession();
CurrentSessionContext.Bind(_sessionNotWeb);
CurrentSessionContext.Bind(_session);
_session.BeginTransaction();
_sessionNotWeb.BeginTransaction();
}
protected void Application_EndRequest(object sender, EventArgs e)
{
//Same code is repeated for the _sessionFactoryNotWeb
ISession session = CurrentSessionContext.Unbind(_sessionFactory);
if (session != null)
{
if (session.Transaction.IsActive)
{
try
{
session.Transaction.Commit();
}
catch
{
session.Transaction.Rollback();
}
}
try
{
session.Dispose();
}
catch
{
}
}
I have had a look at the page running in NHibernate profiler. Sometimes sessions are not started with BeginTranscation, sometimes they are not Committed, sometimes neither; and most puzzlingly, sometimes they are started three times but not finished.
Any calls to the ISession object are managed through this code (there is one for each factory):
public static ISession WebSession()
{
if (CurrentSessionContext.HasBind(MvcApplication._sessionFactory))
{
if (MvcApplication._sessionFactory.GetCurrentSession().IsOpen)
{
return MvcApplication._sessionFactory.GetCurrentSession();
}
else
{
log4net.LogManager.GetLogger(typeof(DBHandler)).Debug("Unbinding NHibernate session");
CurrentSessionContext.Unbind(MvcApplication._sessionFactory);
return WebSession();
}
}
else
{
log4net.LogManager.GetLogger(typeof(DBHandler)).Debug("Initialising NHibernate session");
var session = MvcApplication._sessionFactory.OpenSession();
CurrentSessionContext.Bind(session);
session.BeginTransaction();
return session;
}
}
There are no calls to BeginTransaction or Commit throughout the app without them being flushed, committed, the session disposed of and then being reopened as per the code above.
Any light you guys could shed on this would be much appreciated!
You seem to store your session in a global (application wide) property in your Global.asax.cs. That means, the property contains the last session created, not the session for a specific request. Two requests at the same time and you don't even know if you still access the session you just created, because it might already have been overwritten by another session from the next request, running the same code. You shouldn't store your session somewhere unrelated to the web request if you want to follow a Session-per-Request pattern. You could for instance store your NH session in the HttpContext.Items-Collection. As another way to session management in MVC Ayende posted a nice example on how to wrap session management around a single MVC action instead of the whole request.
I need to instantiate a singleton object per web request, so that the data is processed once and is valid throughout the request, I was using HttpContext.Current.Items to share data during HTTP request, everything was fine until we needed the singleton object instance across multiple threads, the first thing that I came up with was to pass the HttpContext instance to the new thread:
HttpContext context = HttpContext.Current;
ThreadPool.QueueUserWorkItem(callback =>
{
HttpContext.Current = context;
// blah blah
});
Which I don't think is a thread-safe approach as noted here.
Using Reflector I figured HttpContext.Current.Items actually uses CallContext to store objects in each logical thread. So I changed the singleton interface to this:
public static SingletonType SingletonInstance
{
get { return CallContext.GetData(key) as SingletonType; }
set { CallContext.SetData(key, value); }
}
And simply overwrite SingletonInstance when starting any new thread! The code works fine, however it seems that somehow under heavy load, CallContext.GetData(key) returns null and the application crashes with with a null reference exception!
I was thinking, if CallContext.GetData is atomic? But it just doesn't seem right, the CallContext is thread specific data storage and must be atomic or I am missing the point!
My other guess is that setting the SingletonInstance (CallContext.SetData) happens in one thread while CallContext.GetData executes in another as noted here but I don't know how/why?
update:
We are keeping an instance of each online user in an array on the server. The singleton object is actually a reference to the object representing current user. Current user must be unique and available in each thread for database querying, logging, error handling and more, this is how it is done:
public static ApplicationUser CurrentUser
{
get { return CallContext.GetData("ApplicationUser") as ApplicationUser ; }
set { CallContext.SetData("ApplicationUser", value); }
}
ASP.NET may migrate request between threads if it's under load. Once request is received page constructor may execute on one thread and page load on another. In this thread switch CallContext and ThreadStatic are not migrated, but luckaly HttpContext is.
This may be misleading as HttpContext is call context, but this is a little quirk in ASP.NET, probably due to cutting corners to improve performance.
You'll have to remove dependencies to CallContext and use HttpContext entire way through.
You can read more details in this terrific blog post by Piers7.
This was resolved during a chat session.
In essence it involves long-running tasks and a suggestion of using an external service (Web, or regular Windows Service) was decided as the best solution to the problem.
Thread-safing your second method is the best approach.
This is thread-safe version of your singletone:
public sealed class SingletonType
{
#region thread-safe singletone
private static object _lock = new object();
private SingletonType() { }
public static SingletonType SingletonInstance
{
get
{
if (CallContext.GetData(key) == null)
{
lock (_lock)
{
if (CallContext.GetData(key) == null)
CallContext.SetData(key, new SingletonType());
}
}
return CallContext.GetData(key) as SingletonType;
}
}
#endregion
//
//
// SingletoneType members
//
//
}
NOTE : using a lock { } block is the key.