Running OWIN self-hosting web service in Azure Service Fabric behind Azure Load Balancer and Azure Traffic Manager Profile with multiple instances.
About 2% of web requests stuck when service is trying to read the request body from incoming requests to string using HttpContent.ReadAsStringAsync. So the ReadAsStringAsync fails after timeout :
System.Net.Http.HttpRequestException: Error while copying content to a stream.
Web service receives request with different sizes most of them are small but some could be up to 60K. The number of concurrent pending requests/instance is about 900 and CPU load ~30%.
Scaling out number of services helps a little bit but does not solve it completely, I am also trying to understand what is the root cause.
Here is the setting I override in OwinHttpListener:
listener.SetRequestProcessingLimits(Environment.ProcessorCount * 5, int.MaxValue);
listener.SetRequestQueueLimit(10000);
Any ideas of the root cause and/or how to fix it?
Owin ver:
PackageReference Include="Microsoft.AspNet.WebApi.Owin" Version="5.2.6"
Related
I am serving static files (a web page) over a wcf azure relay, but one (or more) of the js files will return 504 gateway timeout, and the html page takes a while to load. However, there were no issues when serving the files directly through a port on the server. Will changing the ReceiveTimeout or SendTimeout on the WebHttpRelayBinding have any effect on this, or could it be something else related to the wcf configuration? If not, is there any other way to troubleshoot this problem?
I can post some more code if needed but the setup for the wcf connection is similar (but slightly different) to the wcf relay sample (https://github.com/Azure/azure-relay/blob/master/samples/WCF%20Relay/RelayHttpNoAuth/Service/Program.cs):
host.AddServiceEndpoint(
GetType(),
new WebHttpRelayBinding(
EndToEndWebHttpSecurityMode.None,
RelayClientAuthenticationType.None) {IsDynamic = true},
httpAddress)
.EndpointBehaviors.Add(
new TransportClientEndpointBehavior(
TokenProvider.CreateSharedAccessSignatureTokenProvider(listenToken)));
host.Open();
Edit:
This ended up being the cause of why the requests were timing out. It really wasn't directly related to Azure Service Bus at all.
I am using azure app service and DB for my C# ODATA API and DB as the backend of of my phone app.
I only have one app service that hosts 10s of endpoints. There are times when I need to publish new versions and I don't want any incoming requests during that time of deployment.
I don't mind that users are not able to finish their requests during the maintenance.
Is there anything in Azure or API that can let me:
1. turn off the api/app service manually?
2. Be able to inform the user that a maintenance is in progress?
This is my trial:
the only thing I can come up with is this. While users always use the "odata" in their url requests: https://myserverl/odata/Users
which is setup in the webapi.config like this:
config.MapODataServiceRoute("odata", "odata", builder.GetEdmModel());
I put the routePrefix (2nd odata) in a web.config.
When I need to turn off access, I change my web.config (which I can access manually even after the publish of code into Azure) to be like this:
<add key="odata" value="noaccess" />
and in my webapi.config:
string odata = ConfigurationManager.AppSettings["odata"].ToString();
config.MapODataServiceRoute("odata", odata, builder.GetEdmModel());
and then save the web.config which will reset the server and all incoming requests that has "odata" will result into error. I can always set it back later.
This method will stop the users from sending requests during maintenance but will not let them know what is going on.
I figured it out.
when I call the server from my client, I verify that the response is between 200 & 299 before parsing results or any other further processing.
So now, I check also for the possible response from the server that it could be either 403 (access is denied) or 503 (server is unavailable). That's where I can add code to notify the user.
In Azure, simply stopping the app service, will generate one of those 2 error codes.
Note: You must check for both: 403 & 503.
I've faced with the next issue related to web service request processing:
Preamble
I have
Web api service hosted on IIS 7.0 on local machine
Test harness console application on the same machine
and i'm trying to simulate web service load by hitting one with requests generated via test harness app.
Test harness core code:
static int HitsCount = 40;
static async void PerformHitting()
{
{
await Task.WhenAll(ParallelEnumerable.Range(0, HitsCount)
.Select(_ => HitAsync())
.WithDegreeOfParallelism(HitsCount));
}
}
static async Task HitAsync()
{
// some logging skipped here
...
await new HttpClient().GetAsync(TargetUrl, HttpCompletionOption.ResponseHeadersRead);
}
Expectation
Logging shows that all HitAsync() calls are made simultaneously: each hit via HttpClients had started in
[0s; 0.1s] time frame (timings are roughly rounded here and below). Hence, I'm expecting to catch all these requests in approximately the same time frame on web service side.
Reality
But logging on the service side shows that requests grouped in bunches 8-12 request each and service catches these bunches with ~1 second interval. I mean:
[0s, 0.3s] <- requests #0-#10
[1.2s, 1.6s] <- requests #10-#20
...
[4.1s, 4.5s] <- request #30-#40
And i'm getting really long execution time for any significant HitCount values.
Question
I suspect some kind of built-in service throttling mechanism or framework built-in concurrent connections limitation. Only I found related to such guesstimate is that, but i didn't get any success trying soulutions from there.
Any ideas what is the issue?
Thanks.
By default, HTTP requests on ASP.NET are limited to 12 times the number of cores. I recommend setting ServicePointManager.DefaultConnectionLimit to int.MaxValue.
Well, the root of the problems lies in the IIS + Windows 7 concurrent requests handling limit (some info about such limits here. Moving service out to the machine with Windows Server kicked out the problem.
I have an application in WPF which is using Java web service. Users can search some documents via application. Two days ago they(who have created the web service) told me that I must change url of the service. So, I did it. But after that the application began to get datas slower than previous. To tell the truth, i am making about 12 request to the web service in one searching. But it was getting all datas in approximately 0.52 second with the previuos web service which was using HTTPS. But the current web service is using HTTP and it takes about 8 seconds to get all datas. And in my opinion the problem might be protocol. But actually, processing time in HTTP must be greater than HTTPS.
So, what could be a problem?
Also, i am connecting to web service with that code:
HQRTXServiceWSService service = new HQRTXServiceWSService();
service.Url = " a url of the web service";
service.Credentials = new System.Net.NetworkCredential("user", "password");
service.PreAuthenticate = true;
Maybe the webservice implementation has been updated and they messed up with their performance?
It could also be their infrastructure, or if there are more people using their webservice... Could be many things but your code ;)
I don't think that HTTP should be slower than HTTPS, usually, it's the contrary, because there's a small overhead for encryption on HTTPS.
maybe you could help me. i am trying to use wcf to transfer a string between client and server. most of the time it is working. but at some clients (one on particular) i received the following error "An HTTP Content-Type header is required for SOAP messaging and none was found".
1. is this an error that returns from the server side ?
2. how can this be fixed ?
Thanks in advance
G.
I recently had this issue.
It turned out it is caused by the Azure web app load balancer time out
https://feedback.azure.com/forums/169385-web-apps/suggestions/36572656-make-web-app-timeout-of-230-seconds-configurable
Because the server took longer than 230 seconds to process the request, so before the web service return the response, the load balaner will timeout and cut the TCP connection caused the ticket client receve a ProtocolException with the message:
An HTTP Content-Type header is required for SOAP messaging and none was found.
Since we can not control or configure that timeout settings, eventually I have to modify the service to make the process shorter than 230 seconds.
I've fixed this problem. I've increased WCF service server CloseTimeout Binding property to 5 minutes from default 1. It's server side problem.