I'm implementing a WCF service client which is aimed to test several service methods. That's done by using standard generated proxy class created by Add Web Reference (inherited from System.Web.Services.Protocols.SoapHttpClientProtocol). What I need to do is execute certain type of requests many times simultaneously to see how it will affect server performance (something like capacity testing for server infrastructure).
Here's the problem - each of responses to these requests is pretty large (~10-100 mb) and I see that only few calls like
// parametersList.Count = 5
foreach(var param in parametersList)
{
var serviceResponse = serviceWebReferenceProxy.ExecuteMethod(param);
// serviceResponse is not saved anywhere else,
// expected to be GC'd after iteration
}
causes Private bytes of process to jump to ~500 mb of memory and Working Set to 200-300 mb. I suspect running them in parallel and increasing iterations count to 100-200 as needed will definitely cause StackOverflow/OutOfMemoryException. How this can be done then? I'm expecting removal of assigning service method response to variable will help, but that's a problem because I need to see each response's size. I'm looking for some sort of instant and guaranteed memory cleanup after each iteration.
Refactored logic to reuse existing objects as much as possible, which gave an ability to run more clients. After certain period of time garbage collecting becomes very slow but performance is acceptable.
Related
We have one old ASP.Net asmx webservice in our application which receives bulk requests at sometime. Service is taking less than 5 seconds for a single request. But It is taking more than a minute when it receives 20 or more concurrent requests. Following is the way it is implemented,
1)receives a request with input data from external clients
2)Will get 20 possibilities from database for one request based on input data after validation
3)Then It will iterate all 20 possibilities using foreach and gets solutions either from other external service or data base based on possibility data. Here in old implementation we have used Parallel.Foreach to perform all 20 calls (service calls or DB calls) parallely to improve the performance.
4)After that Service will send back the all 20 solutions to the client.
This old approach is working fine for few (1or 2 ) requests and resonse time of asmx service is very fast(less than 5 seconds) considering external service calls which are taking 2-3 seconds .But This approach is taking more than 60 seconds when the number of concurrent requests are more than 20.Concurrent requests are pushing CPU utilization to 100% and thread pool starvation as per experts analysis and there by causing requests to queue for threads allocation.
So we got a recommendation to replace parallel extensions and complete service with async/await implementation from end to end.I have implemented async/await end to end and also replaced Parallel.foreach with Task.WhenAll in TPL. But response time has increased a lot after this implementation.for a single request 20 secconds and it its taking more than 2 minutes for bulk requests.
I also tried async foreach in place of parallel.foreach as mentioned in below article but still performance is really bad.
https://stackoverflow.com/questions/14673728/run-async-method-8-times-in-parallel/14674239#14674239
As per logs basic issue is with external service calls/DB calls inside foreach in both old parallel or new async/await implementations.But these service responses are very fast for a single request. Async implementation is taking more time in completing service calls than parallel extensions implementation.
I think service should not take more than 20 seconds for bulk request if it is lessa than 5 seconds for single request.
Can anyone please me what should be the way forward here to improve the performance ?
Thanks in advance.
Regards,
Raghu.
Looks like a lot of things happening here at the same time. I believe you have on nderlying issue that causes many side effects.
I will make the assumption that your server is sufficient in terms of CPU and memory to handle the concurrent connections (though the CPU 100% makes me wonder).
It seems to me that your problem, is that the parallel tasks (or threads), compete for the same resources. That would explain why multiple requests take much more time and why the async paradigm takes even more.
Let me explain:
The problem in practice
Parallel implementation: 1 or 2 request need minimum synchronization, so even if they compete for the same resources, it should be fine.
When 20 threads, try to access the same resources, a lot is happening and you come to a situation known as livelock.
When you switch to async, no requests await for a thread (they are waiting on the IO threads), so you make the problem even worse.
(I suspect that the problem is on your database. If your database server is the same machine, it would also explain the utilization).
The solution
Instead of trying to up the parallelism, find the contested resources and identify the problem.
If it's in your database (most probable scenario), then you need to identify the queries causing the trouble and fix them (indexes, statistics, query plans and whatnot). DB profilers showing locks and query execution plans are your friends for this.
If the problem is in your code, try to minimize the race conditions and imporve your algorithms.
To get a hint of where to look for, use the Visual Studio profiling tools: https://learn.microsoft.com/en-us/visualstudio/profiling/profiling-feature-tour?view=vs-2019 or any external .net profiling software.
I have a workagent agents that runs every 60 seconds checking processing tables for new work, each 60 seconds if new work is available. During processing the agent uses a static TraceHelper class for logging purpose. At the point of writing to the log file I also send a WebRequest to an external API to ship the log entry to a Logstash.
The Webrequest essentially sends off a json object for each Writeline. obviously for logging purposes, order is important so my question is, even though I am only calling one POST HttpWebRequest this is happening hundreds of times a minute. Should I be worried about syncing issues? Could there be a potential that the second Writeline requests gets called and processed by the HttpWebRequest before the first Writeline has a chance to send? Or am I looking into wrong?
Note: below is semi pseudo code
Say I have the below
Tracehelper.Writeline("foo")
Tracehelper.Writeline("baa");
static Tracehelper(){}
public static void Writeline(string msg)
{
File.WriteToFile(msg);
WebProcessHelper.SendLog(msg);
}
static WebProcessHelper() {}
public static void SendLog(string msg)
{
SendHttpRequest(msg);
}
Is there a potential that "baa" is sent ahead of "foo"?
I don't know what classes you are actually using for the web request, but I'll assume its the default async ones and answer it based on that.If you are using something else, the answer may vary.
Yes and no. The requests will all be fired by the client in order. However if you are doing this hundreds of times per second, you may have the messages accepted by the server on the other end not in the same order since they may stack up and exceed the handling capacity of the server. Totally depends on the number of connections, and the server software used.
All that said, making hundreds of HTTP calls a second is an awful idea. If you need to log, use a logging framework that lets you buffer and batch up the logging into a request every second or two. This will greatly save you bandwidth, add scalability, and relieve CPU load on the server you are logging to.
I can heartily recommend either nLog or the Semantic Application Logging Block from MS. Have used both, and they are both really flexible and handle load well.
It's safe to assume that there's the possibility, especially if you're dealing with multiple threads running at once. A safe option would be to set up a separate thread with a BlockingCollection or something that simply pulls strings out of the BlockingCollection and sends them off to logstash.
That way you could have it such that anyone can write something to the BlockingCollection, and they'll all get sent to the remote server in the right order.
I am calling a webservice in multi threaded environment. Lot of my calls fail due to operation time out or bad request but none of the calls fail if I do it in linear fashion which means there is problem with invoking webservice with multiple threads. After lot of analysis, I found out that there is limit of concurrent connection which is causing these exception so I fixed it by adding below code.
ServicePointManager.DefaultConnectionLimit = 2 * _ThreadCount;
What I dont know is the possible disadvantage of increasing this limit. By default, Connection limit is 2. If anyone knows any disadvantages, please do let me know.
The MSDN says:
Changing the DefaultConnectionLimit property has no effect on existing
ServicePoint objects; it affects only ServicePoint objects that are
initialized after the change. If the value of this property has not
been set either directly or through configuration, the value defaults
to the constant DefaultPersistentConnectionLimit.
and
Note
Any changes to the DefaultConnectionLimit property affect both HTTP
1.0 and HTTP 1.1 connections. It is not possible to separately alter the connection limit for HTTP 1.0 and HTTP 1.1 protocols. When used in
the server environment (ASP.NET) DefaultConnectionLimit defaults to
higher number of connections, which is 10.
No, there should not be any disadvantages other than that your AppDomain will consume more resources. But in your case it's trivial.
In fact, it can actually help you use less resources (memory) since pending requests are queued up internally in the ServicePoint. Read here for more information: Big size of ServicePoint object after several hours sending HTTP request in parallel
let me give you the picture....I have around 46K tasks,these task are ran in batch of 100 (each task will call webservice) so I have 100 threads calling webserivce simultaneously. is it still trivial? or will it have some impact in my case?
It will of course have an impact. But the impact depends on many factors. A service point is per host.
If your tasks are mostly against the same host, increase DefaultConnectionLimit to a larger value (expected number of tasks in the current execution batch).
If you are mostly doing requests against different hosts, the limit in your question works fine.
Regarding the usage of resources, it's probably fine as long as your server is not very busy by other applications.
You should also realize, that it may hurt you at the other end.
You may get throttled or black listed, if you put a big load on the webservice you are calling. See for example this blog.
I have a web application using ASP.NET, that is connecting to Oracle CRM as a back end. The ASP.Net uses some business objects to call into the Oracle CRM webservices, and this works fine.
Except, however, Oracle CRM has a limitation where they only allow you to make 20 web service calls per second (or one call per 50mS), and if you exceed this rate a SOAPException is returned "The maximum rate of requests was exceeded. Please try again in X ms."
The traffic to the site has increased recently, so we are now getting a lot of these SOAPExceptions, but as the code that calls the webservice is wrapped up in a business object, I thought I would modify it to ensure that the 50ms limit is never breached.
I use the following code
private static object lock_obj = new object();
lock (lock_obj)
{
call webservice;
System.Threading.Thread.Sleep(50);
}
However, I am still getting some SOAP Exceptions. I did try writing the code using mutexes instead of lock(), but the performance impact proved to be a problem.
Can anyone explain to me why my solution isn't workinf, and perhaps suggest an alternative?
Edit: Moved to answer. Possible due to > 1 IIS worker process. I don't think object locking spans worker processes so subsequent simultaneous threads could be started but I could be wrong
http://hectorcorrea.com/Blog/Log4net-Thread-Safe-but-not-Process-Safe
My suggestion would be an application variable which stores the tick of the last request, then from that you can work out when it's safe to fire the next.
As long as your application is running with only one ASP.NET worker process you should be ok with what you have, but there are a few things to potentially consider.
Are you using a Web Garden? If so this creates multiple worker processes and therefore a lock is only obtained per/process
Are you in a load balanced environment? If so you will need to go to a different method.
OK, it turns out that a compounding issue was that we have a windows service running on the same server that was also calling into some of the same objects every 4 minutes (running on a different process of course). When I turn it off (and having bumped the sleep up to 100 as per Mitchel's suggestion) the problem seems to have gone away almost entirely.
I say almost, because every so often I still get the odd mysterious soapexception, but I think by and large the problem is sorted. I'm still a bit mystified as to how we can get any of these Exceptions, but we will live with it for now.
I think Oracle should publicise this feature of Oracle CRM on Demand a little more widely.
I have a troublesome problem which I'm at a loss to explain. To put it simply, the CPU use is inexplicably high on the web servers in my web farm.
I have a large number of users hitting two front-end web servers. 99% of the page loads are Ajax requests and serve a simple JSON-serialized object which the web servers retrieve from a backend using WCF. In the typical case (again, probably 99% of the requests), all the ASPX page is doing is making a WCF call to get this data, serializing it into a JSON string and returning it.
The object is pretty small-- a guid, a couple short strings, a few ints.
The non-typical case is the initial page load, which does the same thing (WCF request) but injects the response into different parts of the page using asp:literals.
All three machines (2 web servers, one backend) have the same hardward specs. I would expect the backend to do the majority of the work in this situation, since it's managing all the data, doing the lookups, etc. BUT: the load on the backend is much less than the load on the front ends. The backend is a nice, level 10-20% CPU load. The front ends run an average of 30%, but they're all over the map, sometimes hitting spikes of 100% for 10 seconds and taking 600ms to serve these very simple pages.
When I run the front-end in profiler (ANTS), it flags the WCF communication as taking 80% of the CPU time. That's the whole call on the .NET-generated WCF proxy.
WCF Setup: the service is fully parallel. I have instancing set to "single" and concurrency set to "multiple". I opened up the maxConnections and listenBacklog on the service to 256. Under heavy strain (500 requests/s) I see about 75 connections open between both front-end servers and the service, so it's not hitting that wall. I have security set to 'none' all around. Bandwidth use is about 1/20th of the potential (4Mb/s on a 100Mb/s network).
On the client (the web servers), I create a static ChannelFactory for the service. Code to call the service looks like:
service = MyChannelFactory.CreateChannel();
try {
service.Call();
service.Close();
} catch {
service.Abort();
}
(simplified, but you get the basic picture)
What I don't understand is where all this load on the front end is coming from. What's strange about it is that it's never in the 30%-90% range. It's either in panic mode (100%) or doing OK (30% or less). Given the load on the backend, though, I'd expect both of these machines to be 10% or less. Memory use, handles, etc., all seem reasonable.
To add one more wrinkle: when I log how long it takes to service these calls on the backend, I get times consistently less than 15ms (maybe one or two spikes to 30ms every minute). On the front end, these calls can take up to 1s to return. I guess that could be because of the CPU problems, but it seems off to me.
So... does anyone have any ideas on where to look on this kind of thing? I'm running short on things to explore.
Clarification: The WCF service is hosted in a Windows service, and is using a netTcp binding. Also, I have the maxConnections on the client set to 128, FWIW.
It's hard to say what might be going on, but a wild guess would be that something is hitting a contention point and its spinning (instead of doing a wait).
By any chance, have you increased the number of allowed HTTP connections to the back-end server in the front-end server? You can do it through the config file. One common issue I see with WCF clients is that the limit is left to the default value of 2, which severely limits concurrency at the client proxy level.
Have you considered and tested for the possibility of external factors?
Process recycles?
Is Dynamic compression enabled?