I'm working with an open source ecommerce system written using C# and ASP.NET MVC 5. This package makes heavy use of autofac and IoC. It presents a plugin architecture that developers who work with this ecommerce package can use to create plugins, and it uses that plugin architecture itself.
Under some very mysterious circumstances (not just when the application domain recycles) it's taking a full minute to perform a page load. I've been struggling for a while to figure what's taking so long.
I haven't been able to get the application to run from within Visual Studio using the built in web server, and I haven't even been able to get the Visual Studio debugger to attach to an IIS process that is running it. So I haven't been able to use profilers that require Visual Studio.
I have managed to get Glimpse working, but Glimpse is not capturing the event that is taking a long time. In the time line, it shows "Start Request", which takes 0 ms, and then 7 to 50 seconds later, the next event starts, which is, for example, "Authorization: Home.Index", which it says took .01 ms.
What in the world is going on during that 7 to 50 second interval in between? How can I find that out?
Here's a somewhat strange clue: Today, after no change that we're aware of at all, suddenly the problem has more-or-less vanished (most of the time, at least temporarily) when the browser is on our internal network, but when the browser is on a computer outside our network, the problem persists. It's as if some plugin is doing some kind of DNS look up that is taking much less time (to succeed or fail) internally than it takes externally.
So conceivably, a controller is getting instantiated and some dependency or another from some plugin or another is doing some kind of network look up at construction time or something like that. So I'm trying to figure out how to measure how long it's taking MVC to construct each of the dependencies of the Home controller. So far, the solution has elluded me.
I guess I can maybe track down all the implementations of each of the dependencies and figure out which implementation is ending up getting used, and put log messages into their constructor code. Except some of the plugins don't provide source code. At least I can do it for the ones that do, anyway, but is there some profiler or tool I can use that doesn't require Visual Studio and works with ASP.NET MVC processes running under IIS?
Related
I apologize for the length of the question, but I believe it is difficult to understand the “why” without the background.
Background: I have two applications running in a Windows Embedded Standard 7 environment. They should be the only two applications running on the machine. One, called “Controller”, is written in C++ the other, “DBconnector”, is written in c#. This is not new code. It has been in active use and development for almost 20 years.
The purpose of the software is to run a manufacturing machine for producing parts. These machines are big and dangerous if the program crashes. Long ago, I discovered that if the network went down for some reason, all the threads in the application would stall – not just the network thread. This was disastrous since leaving the controller in a state with the wrong relays on in extremely rare circumstances could cause the machine to literally explode. Note: Several things have been added to the software and hardware to prevent this now. While this danger doesn’t really exist anymore, stability is still extremely important. I never want the operator to be stuck in a state where they can’t hit the reset button. My solution at the time was to move the networking tasks into a separate application. The OS was windows XP based at the time. I have no idea if the problem still exists in windows 10 since I really don’t want to rewrite hundreds of thousands of lines of code to try and merge the two programs now.
The development of the two programs diverged such that the one that controlled the machine, Controller, was designed for extreme stability and the other, DBconnector, was where dangerous things like networking and most file I/O happened. Communication between the two programs is facilitated using a memory mapped file that they both can access. I have no problem sharing window handles or process id’s or any other data that might be needed between the two programs.
Here is my question. How can I make the Controller application display the GUI of DBconnector? For example, I have started to add functionality to Controller that requires DBconnector to display the quality control sheets that are held on a web site on company servers. I want for an operator to be able to pull up the quality control sheet directly on the machine. The operator currently only interacts with the Controller application. I don’t want Controller to be able to access the network. Also, C# has some tools to make displaying a web page easy. It seems to me that the place to do this is DBconnector. The problem is that DBconnector runs in the background and cannot currently be seen or accessed by a user. So, the question is how to solve this.
First option I have tried is to tell DBconnector to come forward and put Controller in the background. Then, when the user is done, Controller comes back to the front. I have made this to work using some hacks, but it is inconsistent. The trick I used was to minimize and then maximize DBconnector which seems to bring it to the front most of the time and try to hold focus on one or the other. There still might be a way to do it this way, but it needs to be something that is consistent.
The second option is to run the DBconnector application inside of one of Controller’s windows. I have no idea how to do this. I thought about using ATL or COM, but I think these run as threads within Controllers process rather than as a separate application.
The third option I’ve considered is to create a window inside Controller that intercepts and passes all user input messages directly to Dbconnector using a windows message handle and takes a screenshot of DBconnector whenever the it is invalidated and passes it through the memory mapped file. Currently, this is what I am swaying towards.
Are there any suggestions on how to do the first and last option better, or how to do the second option at all, or another solution that I have missed? Keep in mind that our current hardware is running Windows Embedded Standard 7. The project is currently in visual studio 2015. The C++ window technology is MFC implemented using libraries originally from around 2003 I think. DBconnector is in .NET framework 4 in C#.
Environment:
Visual Studio 2015 Update 1
IIS Express 10
Moderately sized MVC web application
.NET Framework 4.6.1 x64 debug builds
Newer Core I7 laptop, plenty of ram, SSD drive
Making a small change in one projects .cs file, hit the green arrow to test out the change. It takes about 8 seconds for the build to finish and Chrome to pop open a new tab. Not bad. But then it takes about ~30 seconds for the first page to show up.
What can be done to reduce that delay? Would pre-compiled views be the first order improvement here? What are some of the best current techniques to achieve that?
Try installing Glimpse. Correctly set up it will show you where the delay is including the database calls and their duration etc.
Install Redgate Ants and step through the code locally (potentially pointing at the production database if this is a live problem), this tool should be able to tell you where any slow down is.
One of the features is:
Jump straight to the slowest activity
The call tree in the .NET performance profiler shows you data for
every method and identifies the most expensive methods, database
queries, and web requests
There is a 14 day free trial, which should be enough time to diagnose your problem.
I'm currently finishing up a project for University that requires us to build an enterprise application using techniques described in Fowlers "Patterns of Enterprise Architecture".
It's your bog standard ASP MVC application which talks to a service layer which talks to a data layer.. etc.
We've also been asked to run several load test scenarios, ranging from 1-25 users. I've created a load test per scenario (1User.loadtest, 5User.loadtest, 10User.loadtest etc..) and I was wondering if there was any way to queue these up and leave them running, rather than starting one, coming back a few minutes later, starting another.. etc.
TL;DR - Anybody know a way to queue load tests?
Load Tests are automatically queued. When running a load test, you can still add a new test. However, you still require to click N times...
another solution is to use the command line tool.
> mstest /TestContainer:LoadTest1.loadtest
I have a large-ish Winforms application written in C# which is periodically unresponsive. The issue seems to occur once the application has been use for an hour or so. Exact timings are difficult to gather as users often go off to work on something selse, get back to it and find it has become unresponsive.
I think a memory leak can be ruled out as I'm not seeing excessive memory usage (I've asked users to send a screenshot of the task manager and memory usagage is the same as I would see when the application is runnning normally)
Similarly, CPU usage is normal (single digit %)
As I've so far been unable to recreate the issue on mydevelopment PC I am planning on sitting next to one of the affected users and mirror every action the user performs in order to recreate this. (I'll be setting up a laptop to RDP in to my main PC)
Recreating the issue is one thing, but I'll need to find out what is actually going on in the application.
Could anyone tell me if running in debug mode (through visual studio) will be sufficient or will I need to do something different?
I've searched through a few posts and I've seen mention of profiling software, however I'm not sure if this would only help with general performance issues or memory management issues.
Alternatively, if anyone has come across similar freezing issues then do you have any suggestions of the kind of causes for this?
Some technical details: Aplication is C#, compiled against .NET 3.5, winforms GUI. There are a few external libraries (most significant is ComponentFactory Krypton Suite). Data access is to a Microsoft SQL Server 2005 database. The solution contains 39 projects, I'm not sure if that might have something to do with it?
Any suggestions/pointers would be greatly appreciated.
The application is working much more reliably now, freezing issues still occur on occasion but nowhere near as often as before.
The issue appears to be related to the endpoint security (in this case, Cisco Security Agent) present in the environment I'm working in, application has been whitelisted and has has significantly rediced the instances of application hangs. The development system I work on does not have this endpoint security present, so it didn't show up in early stages of testing.
Thanks for all your feedback, I think there are still threading and garbage collection issues that need cleaning up, hopefully this should sort out the last few issues.
Over the last few weeks I've been subject to a sudden and significant performance deterioration when browsing locally hosted ASP.NET 3.5 MVC web applications (C#). Load times for a given page are on average 20 seconds (regardless of content); start up is usually over a minute. These applications run fast on production and even test systems (Test system is comparable to my development environment).
I am running IIS 6.0, VS2008, Vista Ultimate, SQL2005, .NET 3.5, MVC 1.0, and we use VisualSVN 1.7.
My SQL DB is local and IPv6 does not seem to be the cause. I browse in Firefox and IE8 outside of Debug mode using loopback, machine name, and 'localhost' and get the exact same results every time (hence DNS doesn't seem to be the issue either).
Below are screen shots of my dotTrace output.
http://www.glowfoto.com/static_image/28-100108L/3123/jpg/06/2010/img4/glowfoto
This issue has made it near impossible to debug/test any web app. Any suggestions very much appreciated!
SOLUTION: Complete re-installation of Windows, IIS, Visual Studio, etc. It wasn't the preferred solution, but it worked.
Surely the big red flag on that profiler output is the fact that AddDirectory is called 408 times and AddExistingFile is called 66,914 times?
Can you just confirm that there's not just a shed load of directories and files underneath your MVC app's root folder? Because it looks like the framework is busying itself trying to work out what files it needs to build (or add watches to) on startup.
[I am not au fait with MVC and so maybe this is not what is happening but 67k calls to a function with a name like "AddExistingFile" does smell wrong].
I've learnt that it's usually a "smell" when things fail near a power of two ...
Given
Over the last few weeks I've been subject to a sudden and significant performance deterioration
and
AddExistingFile is called 66,914 times
I'm wondering if the poor performance hit at about the time as the number of files exceeded 65,535 ...
Other possibilities to consider ...
Are all 66,914 files in the same directory? If so, that's a lot of directory blocks to access ... try a hard drive defrag. In fact, it's even more directory blocks if they're distributed across a bunch of directories.
Are you storing all the files in the same list? Are you preseting the capacity of that list, or allowing it to "grow" naturally and slowly?
Are you scanning for files depth first or breadth first? Caching by the OS will favor the performance of depth first.
Update 14/7
Clarification of Are you storing all the files in the same list?
Naive code like this first example doesn't perform ideally well because it needs to reallocate storage space as the list grows.
var myList = new List<int>();
for (int i=0; i<10000; i++)
{
myList.Add(i);
}
It's more efficient, if you know it, to initialize the list with a specific capacity to avoid the reallocation overhead:
var myList = new List<int>(10000); // Capacity is 10000
for (int i=0; i<10000; i++)
{
myList.Add(i);
}
Update 15/7
Comment by OP:
These web apps are not programmatically probing files on my hard disk, at least not by my hand. If there is any recursive file scanning, its by VS 2008.
It's not Visual Studio that's doing the file scanning - it is your web application. This can clearly be seen in the first profiler trace you posted - the call to System.Web.Hosting.HostingEnvironment.Initialize() is taking 49 seconds, largely because of 66,914 calls to AddExistingFile(). In particular, the read of the property CreationTimeUTC is taking almost all the time.
This scanning won't be random - it's either the result of your configuration of the application, or the files are in your web applications file tree. Find those files and you'll know the reason for your performance problems.
Try creating a new, default MVC2 application in a new web folder. Build and browse it. If your load times are okay with the new app, then there's something up with your application. If not, it's outside of the context of the app and you should start looking at IIS config, extensions, hardware, network, etc.
In your app, back up your web config and start with a new, default web.config. That should disable any extensions or handlers you've installed. If that fixes your load times, start adding stuff from the old web.config into the new one in small blocks until the problem reappears, and in that way isolate the offending item.
I call this "binary search" debugging. It's tedious, but actually works pretty quickly and will most likely identify the problem when we get stuck in one of those "BUT IT SHOULD WORK!!!" modes.
Update Just a thought: to rule out IIS config, try running the site under Cassini/built-in dev server.
The solution was to format and do a clean install of Vista, SQL Server 2005, Visual Studio 2008, IIS6 and the whole lot. I am now able to debug, without consequence, the very same webapp(s) I was experiencing the problems with initially. This leads me to believe the problem lay within one of the installations above and must have been aggravated by a software update or by the addition of software.
You could download Fidler to measure how long each call takes and get some measurements.
Link
This video might help...