I have an asp.net/c# 4.0 website on a shared server.
The os Windows Server 2012 is running IIS 7, has 32GB of ram and 3.29GHz of processor.
The server is running into difficulty now and again, such as problems RDP'ing and other PHP websites running slow.
Our sys admin has suggested my website as being a possible memory hog and cause of these issues.
At any given time the sites "Memory (private working set)" is 2GB and can peak as high as 15GB.
I ran a trial version of JetBrains dotMemory on the server, attached to the website's w3wp.exe process. My first time using this program, I am a complete novice here.
I took two memory snapshots using dotMemory.
And the basic snapshot comparison can be seen here: http://i.imgur.com/0Jk8yYE.jpg
From the above comparison we can see that System.String and BusinessObjects.Item are the two items with the most survived bytes.
Drilling down on system.string I could see that the main dominating survived object was System.Web.Caching.CacheEntry with 135MB. See the screengrab: http://i.imgur.com/t1hs8nd.jpg
Which leads me to suspect maybe I cache too much?
I cache all data that comes out of my database: HTML Page Content, Nav-Menu Items, Relationships between pages & children, articles etc. Using HttpContext.Current.Cache.Insert.
With the cache timeout set to 10080 minutes.
My Questions:
1 - is 2GB Memory (private working set) and a peak as high as 15GB to much for a server with 32 GB Ram?
2 - Have I used dotMemory correctly to identify an issue?
3 - Is my caching an issue?
4 - Possible other causes?
5 - Possible solutions?
Thanks
Usage of a large amount of memory itself can not slow down a program. It can be slowed down
if windows actively using swap file. Ask an admin to check this case
if .NET program causes garbage collecting too much (produces too much memory traffic)
You can see a memory traffic information in dotMemory. Unfortunately it can't gather such information if it attaches to the program, the only way to collect object creation stack traces and memory traffic information is to launch your program under dotMemory profiler, enabling corresponding settings.
BUT!
The problem is you can't evaluate is memory traffic amount "N" high or no, the only way to find a root of a performance problem is using performance profiler.
For example JetBrains dotTrace can show you, how much time you program spends in garbage collecting, and only in case this is a bottle neck, you should use memory profiler to find a root of traffic.
Conclusion: try to find a bottle neck using performance profiler first.
Then, if you still have a questions about dotMemory, ask me, I'll try to help you :)
Related
I am working to diagnose a series of OutOfMemoryException problems within an application of ours. This is an internal 32-bit (x86) OWIN-hosted WebAPI that runs within a console application and talks to a series of hardware components in parallel. For that reason it's creating around 20 instances of a library, and the sharp increase in "virtual size" memory matches when those instances are created.
From the output of Process Explorer, and dotMemory, it does not appear that we're allocating that much actual memory within this application:
From reading many, many SO answers I think I understand that our problem is either from fragmentation within the G0, G1, G2 & LOH heaps, or we're possibly bumping into the 2GB addressable memory limit for a 32-bit process running on Windows 7. This application works in batches where it collects a bunch of data from hardware devices, creates collections in memory to aggregate that data into a single object, and then saves it to be retrieved by a client app. This activity is the cause of the spikes in the dotMemory visual, but these data structures are not enormous, which I think the dotMemory chart shows.
Looking at the heaps has shown they rarely grow beyond 10-15MB in size, and I don't see much evidence that the LOH is growing too large or being severely fragmented. I'm really struggling with how to proceed to better understand what's happening here.
So my question is two-fold:
Is it conceivable that we could be hitting that 2GB limit for virtual memory, and that's a cause for these memory exceptions?
If that is a possible cause then am I right in thinking a 64-bit build would get around that?
We are exploring moving to a 64-bit build, but that would require updating some low-level libraries we use to also be 64-bit. It's certainly an option we will explore eventually (if not sooner), but we're trying to understand this situation better before investing the time required.
Update after setting the LARGEADDRESSFLAG
Based a recommendation I set that flag on the binary and interestingly saw the virtual size jump immediately to nearly 3GB. I don't know if I should be alarmed by that?!
I will monitor the application with this configuration for the next several hours.
In my case the advice provided by #ThomasWeller was indeed correct and enabling the "large address aware" flag has allowed this application to run for several days without throwing memory exceptions.
I suspect my C# server app (running on win server 2012) has a native memory leak, likely from one of the native libraries I'm using. My app reaches ~2.5GB commited memory after running for about 2 weeks. Looking at a dump using WinDbg, I see 1.9GB of native memory (which keeps increasing). Managed memory did not increase beyond a ±600MB-ish threshold.
Running the !heap -s command I see that two segments take up 1.7GB of heap memory combined, however inspecting those segments with !heap -stat -h (basically following the flow from https://www.codeproject.com/Articles/31382/Memory-Leak-Detection-Using-Windbg) shows me that the amount of busy bytes combined for both is about 30MB, which leaves me with no idea on what could cause the leak.
I have no option of live debugging the app (connecting with a profiler etc) at the moment, since it is running in production and a performance degradation is out of question. Unfortunately, this behavior is also not reproducible in any other environment available to me.
One of my initial guesses was that this is not a leak but rather just lazy reclamation of memory by the OS (since no other process requires this memory). I could verify this by spinning up processes which eat up a lot of memory, but if I'm wrong my app would crash and that is not a good possibility for me :)
Is there any other way I can understand whats going on in the dump? Any other tool perhaps?
Any help would be greatly appreciated!
As the load on our Azure website has increased (along with the complexity of the work that it's doing), we've noticed that we're running into CPU utilization issues. CPU utilization gradually creeps upward over the course of several hours, even as traffic levels remain fairly steady. Over time, if the Azure stats are correct, we'll somehow manage to get > 60 seconds of CPU per instance (not quite clear how that works), and response times will start increasing dramatically.
If I restart the web server, CPU drops immediately, and then begins a slow creep back up. For instance, in the image below, you can see CPU creeping up, followed by the restart (with the red circle) and then a recovery of the CPU.
I'm strongly inclined to suspect that this is a problem somewhere in my own code, but I'm scratching my head as to how to figure it out. So far, any attempts to reproduce this on my dev or testing environments have proven ineffectual. Nearly all the suggestions for profiling IIS/C# performance seem to presume either direct access to the machine in question or at least a "Cloud Service" instance rather than an Azure Website.
I know this is a bit of a long shot, but... any suggestions, either for what it might be, or how to troubleshoot it?
(We're using C# 5.0, .NET 4.5.1, ASP.NET MVC 5.2.0, WebAPI 2.2, EF 6.1.1, Azure System Bus, Azure SQL Database, Azure redis cache, and async for every significant code path.)
Edit 8/5/14 - I've tried some of the suggestions below. But when the website is truly busy, i.e., ~100% CPU utilization, any attempt to download a mini-dump or GC dump results in a 500 error, with the message, "Not enough storage." The various times that I have been able to download a mini-dump or GC dump, they haven't shown anything particularly interesting, at least, so far as I could figure out. (For instance, the most interesting thing in the GC dump was a half dozen or so >100KB string instances - those seem to be associated with the bundling subsystem in some fashion, so I suspect that they're just cached ScriptBundle or StyleBundle instances.)
Try remote debugging to your site from visual studio.
Try https://{sitename}.scm.azurewebsites.net/ProcessExplorer/ there you can take memory dumps an GC dumps of your w3wp process.
Then you can compare 2 GC dumps to find memory leaks and open the memory dump with windbg/VS for further "offline" debugging.
we have a 2008 SQL Server machine, for some reason which we still havent figured out, every two weeks this server stops responding around the same time and around the same day, it's either sunday or saturday, we have checked the logs and the only message we have found is this
A significant part of sql server process memory has been paged out.
On the operating system log we algo found a message
Application popup: Windows - Virtual Memory Minimum Too Low : Your system is low on virtual memory. Windows is increasing the size of your virtual memory paging file. During this process, memory requests for some applications may be denied. For more information, see Help.
so it looks like the operating system is out of physical memory, we do not undestand why this happens every two weeks, it seems as if memory never gets freed and two weeks is the period it takes to get full. Is there a way we could better diagnose this? we are also wondering if it is related to how we are using NHibernate? or is there any other cause?
SQL Server consumes more and more memory over time, this is normal. We faced this issue after the server has been up for a couple of MONTHS. SQL's memory consumption went up to several GB and Windows has eventually cut it down...
Setting up a "max server memory" for your SQL Server should help. On our 8 GB server we've set it to 5.5 GB.
PS. Setting up a "low memory" email alert is a good practice. It will let know just before the things are about to mess up. This blog post explains how you do this.
1) identify the process that consumes the memory. Use the Process object and identify the process that consumes memory (large Private Bytes and Virtual Bytes)
2) If the process turns out to be SQL Server, follow SQL Server memory diagnostics steps. See Monitoring Memory Usage and Using DBCC MEMORYSTATUS to Monitor SQL Server Memory Usage
Depending on the identified memory consumer process and posibly on the identified SQL Server memory clerk, appropiate actions and remedies can be recommended, but untill you do the due diligence and find the cause no advice can possibly be given.
We are trying to track down a performance issue on a asp.net solution on windows 2008.
The error page that has no database access and very little logic takes 10 sec. Other pages over 70 seconds.
Performance drops noticeably only at high load
total memory usage is low 5 GB of 16 available
W3wp.exe using 2.5 GB
several Connection_Dropped DefaultAppPool in the httperr file
ca. 1500 connections, Asp.net queue length is 10000
CPU usage is low
Anyone have an idea what I could check next?
Edit
I have now used VS 2010 to run a performance test against it in a test virtual server.
I ran 200 users with a stepped build up and no wait time.
The interesting thing was that the page time continued to increase even after the max number of users was reached. There did not appear to be any memory leaks, memory usage is flat. Time taken per page goes from 0.1 to 30.0 seconds.
All pages increase, the one that increases most is a get of the login page, no database access just a forms auth check to see if the user is logged in.
Upon reading your numbers (always answering too fast, am I?) I agree that you should probably profile the server-side first. See What Are Some Good .NET Profilers?
I suggest you use
google chrome
It has excellent profiling tools (under developer tools, Ctrl-Shift-I on my installation). I peruse
network
profiles
timeline
charts for the information.
Also, there is the more highlevel Y-Slow extension to Firefox. It is developed/used by yahoo and gives some rather to the point advice out of the box.
If you prefer Firefox, the Firebug extension comes pretty close to the Google developer tools
Ah. Wha about you just look it up?
Attach a profiler, make a profiling run, find out where the CPU spends itÄs time.
There are plenty of profilers around that offer 14 day free trials.
I wouldsay ou need more CPU - find out why ;)