We Keep Coding

Calling WCF Service in a loop causes queueing and eventual timeouts

I have around 10-20 tasks that each perform a separate job. At some point during this job each task will eventually call the same WCF endpoint asynchronously to get data.
The WCF endpoint does exist on a server farm cluster of 8 servers. Within this endpoint there are multiple things happening (Stored Procedure calls, Data Calculation, SQL inserts, delete, updates)
It sounds like a lot but One single call takes less than 1 second to return.
The problem is running full throttle, the service starts queuing up the calls and the response times grow larger and larger and eventually timeout. Each has to be processed sequentially as the ProcessJob Webservice needs to insert the data into the database as soon as it can.
How can I prevent this queuing and timeout from happening?
private void StartAllJobs()
{
List<Task> task = new List<Task>();
foreach(job in JobList)
{
//Start the job on its own thread
task.Add(Task.Run(await () => RunMyJob(job)));
}
Task.WaitAll(task.ToArray());
}
private async Task<string> RunMyJob(Job job)
{
//Other processing below
var infoList = ProcessOtherStuff(job);
using(var proxy=new serviceclient())
{
Stopwatch sw = new Stopwatch();
sw.Start();
for(var i=0;i<infoList.Count;i++)
//The count can reach up to 20000 records to process
{
try
{
var value = await proxy.ProcessJob(info);
sw.Stop();
Console.Writeline("Process took this long:{0}", sw.Elapsed);
}
catch(Exception ex)
{
Console.Writeline("Exception {0}: Time:{1}, ex.Message, DateTime.Now.ToString());
}
}
return "Complete";
}
}
The error message is something like this
System.TimeoutException: The request channel timed out while waiting for a reply after 00:01:00. Increase the timeout value passed to the call to Request or increase the SendTimeout value on the Binding. The time allotted to this operation may have been a portion of a longer timeout. ---> System.TimeoutException: The HTTP request to 'http://localhost/ReportServer/ReportExecution2005.asmx' has exceeded the allotted timeout of 00:01:00. The time allotted to this operation may have been a portion of a longer timeout. ---> System.Net.WebException: The operation has timed out
here is the client WCF config (only part that is related to the reporting services, not the entire WCF config)
<bindings>
<wsHttpBinding>
<binding name="ReportExecutionServiceSoap" closeTimeout="00:01:00"
openTimeout="00:01:00" receiveTimeout="00:10:00" sendTimeout="00:01:00"
allowCookies="false" bypassProxyOnLocal="false" hostNameComparisonMode="StrongWildcard"
maxBufferSize="2147483647" maxBufferPoolSize="2147483647" maxReceivedMessageSize="2147483647"
messageEncoding="Text" textEncoding="utf-8" transferMode="Buffered"
useDefaultWebProxy="true">
<readerQuotas maxDepth="32" maxStringContentLength="2147483647" maxArrayLength="2147483647"
maxBytesPerRead="2147483647" maxNameTableCharCount="2147483647" />
<security mode="TransportCredentialOnly">
<transport clientCredentialType="Non" proxyCredentialType="None"
realm="" />
</security>
</binding>
</wsHttpBinding>
</bindings>
<client>
<endpoint address="https://serverfarm.com/MyServer/MyWebService.svc"
binding="wsHttpBinding" bindingConfiguration="ReportExecutionServiceSoap"
contract="ReportExecutionServiceReference.ReportExecutionServiceSoap"
name="ReportExecutionServiceSoap" />
</client>
Then my service has
<endpoint address="" binding="wsHttpBinding" bindingConfiguration="MyBinding" contract="MyWebService.IMyWebService"/>
<endpoint address="mex" binding="mexHttpBinding" contract="IMetadataExchange"/>
<behaviors>
<serviceBehaviors>
<behavior name="MyBehavior">
<serviceMetadata httpGetEnabled="true"/>
<serviceDebug includeExceptionDetailInFaults="true"/>
<serviceThrottling maxConcurrentInstances="70424" maxConcurrentCalls="10024" maxConcurrentSessions="60400" />
</behavior>
</serviceBehaviors>
</behaviors>
<bindings>
<wsHttpeBinding>
<binding name="MyBinding">
<readerQuotas
maxArrayLength="16384" maxBytesPerRead="4096" maxDepth="32" maxNameTableCharCount="16384" maxStringContentLength="102400"/>
<security mode="Transport">
<transportclientCredentialType="None"/>
<messageestablishSecurityContext="false"/>
</security>
</binding>

Additional information: The remote server returned an unexpected response: (413) Request Entity Too Large

I'm getting this error in my WCF Service: Additional information: The remote server returned an unexpected response: (413) Request Entity Too Large.
I tried to increase maxReceivedMessageSize to 2147483647 but it still doesn't work. Then, I measured the size of my file and I realized that it has 10 Kb.
Do you guys have any idea of what it might be?
I'm sending you a copy of my code:
Here, I'm basically sending a list of 1o Kb for the business logic layer (WCF service)
else if (comboBoxReason.Text == "Antecipação indevida - automática")
{
AdjustmentAmount = 0;
using (Stream s = new MemoryStream())
{
BinaryFormatter formatter = new BinaryFormatter();
formatter.Serialize(s, AnticipationAdjustment);
long size = s.Length;
}
AdjustmentAmount = Decimal.Round(FinancialAdjustmentsCalculations.AnticipationAutomaticRate(AnticipationAdjustment), 2); //THE ERROR IS HERE
textBoxAdjustment.Text = ValorAjuste.ToString();
AnticipationAdjustment.Clear();
}
That is my Web.Config from the Business Logic Layer (Ext)
<bindings>
<basicHttpBinding>
<binding name="BasicHttpBinding_IInt" closeTimeout="01:00:01" openTimeout="01:00:01"
receiveTimeout="02:00:01" sendTimeout="09:00:00" maxBufferPoolSize="2147483647"
maxBufferSize="2147483647" maxReceivedMessageSize="2147483647"/>
</basicHttpBinding>
And that is my App.Config from the interface:
<binding name="BasicHttpBinding_IExt"
closeTimeout="01:00:01"
openTimeout="01:00:01" receiveTimeout="02:00:01" sendTimeout="09:00:00"
maxBufferPoolSize="2147483647" maxBufferSize="2147483647" maxReceivedMessageSize="2147483647" />
</basicHttpBinding>
I tried to add the readerQuotas and maxItemsInObjectGraph="2147483647", but it still doesn't work:
<binding name="BasicHttpBinding_IExt" closeTimeout="01:00:01"
openTimeout="02:00:01" receiveTimeout="01:00:01" sendTimeout="09:00:00"
maxBufferPoolSize="524288000" maxBufferSize="65536000" maxReceivedMessageSize="65536000">
<readerQuotas maxDepth="2147483647" maxStringContentLength="2147483647" maxArrayLength="2147483647" maxBytesPerRead="2147483647" maxNameTableCharCount="2147483647" />
</binding>
</basicHttpBinding>
</bindings>
<client>
<endpoint address="http://localhost:56296/ExtService.svc" binding="basicHttpBinding"
bindingConfiguration="BasicHttpBinding_IExt" contract="ExtService.IExt"
name="BasicHttpBinding_IExt" behaviorConfiguration="Behaviors.EndpointBehavior" />
</client>
<behaviors>
<endpointBehaviors>
<behavior name="Behaviors.EndpointBehavior">
<dataContractSerializer maxItemsInObjectGraph="2147483647" />
</behavior>
</endpointBehaviors>
</behaviors>
</system.serviceModel>

Getting "413 Request Entity Too Large" when talking to WCF service

I have a WCF service, Server, which accepts requests from a console application Client. When sending the data (which is relatively large), I get the following exception:
System.ServiceModel.ProtocolException: The remote server returned an unexpected response: (413) Request Entity Too Large. ---> System.Net.WebException: The remote server returned an error: (413) Request Entity Too Large.
I thought I had the limits correctly set in the config files, but apparently I'm mistaken?
Here's the relevant part of Server's app.config:
<system.serviceModel>
<services>
<service name="Services.Server">
<endpoint bindingConfiguration="LargeWeb" binding="webHttpBinding" contract="Interfaces.IServer" />
</service>
</services>
<bindings>
<webHttpBinding>
<binding name="LargeWeb" maxBufferSize="2147483647" maxBufferPoolSize="2147483647" maxReceivedMessageSize="2147483647">
<readerQuotas maxDepth="32" maxStringContentLength="2147483647" maxArrayLength="2147483647" maxBytesPerRead="2147483647" maxNameTableCharCount="2147483647" />
</binding>
</webHttpBinding>
</bindings>
</system.serviceModel>
And here's Client's app.config:
<system.serviceModel>
<client>
<endpoint name="serverEndpoint" binding="webHttpBinding" bindingConfiguration="LargeWeb" contract="Interfaces.IServer" />
</client>
<bindings>
<webHttpBinding>
<binding name="LargeWeb" maxBufferSize="2147483647" maxBufferPoolSize="2147483647" maxReceivedMessageSize="2147483647">
<readerQuotas maxDepth="32" maxStringContentLength="2147483647" maxArrayLength="2147483647" maxBytesPerRead="2147483647" maxNameTableCharCount="2147483647" />
</binding>
</webHttpBinding>
</bindings>
</system.serviceModel>
And finally, this is how Client calls the Server's method:
using (var cf = new WebChannelFactory<IServer>("serverEndpoint", serverAddress))
{
var channel = cf.CreateChannel();
channel.SomeMethod(some, arguments);
}
What am I doing wrong?
Thank you for any help.

The issue was that Client was residing in a UnitTest project, and the issue arised while running a unit test. When running a unit test, only the UnitTest project's config file is taken into account, so the <service> parts of the Server config need to be copied into it.
This way, only one config file is ever used: both Server and Client read it for their configuration.
After that, things work fine.

EndpointNotFoundException on call to operation of running RESTful WCF service

I have a really strange problem with one of my WCF services. I'm pretty sure that most people who have experimented with WCF services have had an EndpointNotFoundException thrown when an endpoint had not been set up correctly in one of their configuration files. From the EndpointNotFoundException class page on MSDN:
The exception that is thrown when a remote endpoint could not be found or reached.
Further on, it continues:
The endpoint may not be found or reachable because the remote endpoint is down, the remote endpoint is unreachable, or because the remote network is unreachable.
This does not reflect my situation. So it seems that receiving an EndpointNotFoundException when working with WCF is not unusual, but this Exception is not thrown when first trying to access the service... instead, it is thrown only when trying to call one of the operations of the service:
using (ExportConfirmationServiceClient client = new ExportConfirmationServiceClient(
"WebHttpBinding_IExportConfirmationService")) // <-- Exception is NOT thrown here
{
...
component releaseConfirmation = DeserializeTestXmlFile(filePath);
client.AddExportConfirmation("5051275066302", releaseConfirmation);
// Exception is thrown on call to service operation on line above
...
}
Interestingly, the Exception message also includes the name of the operation in the mentioned file path:
There was no endpoint listening at http://domain/Folder/ServiceName.svc/OperationName that could accept the message. This is often caused by an incorrect address or SOAP action. See InnerException, if present, for more details.
The inner Exception has the following message:
The remote server returned an error: (404) Not Found.
This is particularly confusing for me as I can successfully browse to the service URL and see the default You have created a service page:
Also, if I navigate to the path in the Exception message, I see an Endpoint not found message on the page:
However, if I navigate to any operation page of one of my other, working WCF services, I get a standard 400 Bad Request error from the browser, even though the operation works fine. So it would seem to me as if this original EndpointNotFoundException could be a red herring... I'm really not sure though, as I've not spent much time working with WCF.
I'll show my web.config (server side) here just in case anyone needs to see it:
<?xml version="1.0"?>
<configuration>
<system.web>
<compilation debug="true" targetFramework="4.0" />
</system.web>
<system.serviceModel>
<behaviors>
<serviceBehaviors>
<behavior name="Midas.WebConfirmations.ExportConfirmationServiceBehaviour">
<serviceMetadata httpGetEnabled="true" />
<serviceDebug includeExceptionDetailInFaults="false" />
</behavior>
</serviceBehaviors>
<endpointBehaviors>
<behavior name="webHttp">
<webHttp />
</behavior>
</endpointBehaviors>
</behaviors>
<serviceHostingEnvironment multipleSiteBindingsEnabled="false" />
<services>
<service name="Midas.WebConfirmations.ExportConfirmationService" behaviorConfiguration="Midas.WebConfirmations.ExportConfirmationServiceBehaviour">
<endpoint address="" binding="webHttpBinding" contract="Midas.WebConfirmations.IExportConfirmationService" behaviorConfiguration="webHttp" />
<endpoint address="mex" binding="mexHttpBinding" contract="IMetadataExchange" />
</service>
</services>
</system.serviceModel>
<system.webServer>
<modules runAllManagedModulesForAllRequests="true"/>
</system.webServer>
</configuration>
This is the client App.config (bear in mind that this references two WCF services):
<?xml version="1.0" encoding="utf-8"?>
<configuration>
<system.serviceModel>
<bindings>
<basicHttpBinding>
<binding name="BasicHttpBinding_IDataService" closeTimeout="00:01:00" openTimeout="00:01:00" receiveTimeout="00:10:00" sendTimeout="00:01:00" allowCookies="false" bypassProxyOnLocal="false" hostNameComparisonMode="StrongWildcard" maxBufferSize="65536" maxBufferPoolSize="524288" maxReceivedMessageSize="65536" messageEncoding="Text" textEncoding="utf-8" transferMode="Buffered" useDefaultWebProxy="true">
<readerQuotas maxDepth="32" maxStringContentLength="8192" maxArrayLength="16384" maxBytesPerRead="4096" maxNameTableCharCount="16384" />
<security mode="None">
<transport clientCredentialType="None" proxyCredentialType="None" realm="" />
<message clientCredentialType="UserName" algorithmSuite="Default" />
</security>
</binding>
</basicHttpBinding>
<webHttpBinding>
<binding name="WebHttpBinding_IExportConfirmationService" allowCookies="true" maxReceivedMessageSize="20000000" maxBufferSize="20000000" maxBufferPoolSize="20000000">
<readerQuotas maxDepth="32" maxArrayLength="200000000" maxStringContentLength="200000000"/>
</binding>
</webHttpBinding>
</bindings>
<behaviors>
<serviceBehaviors>
<behavior name="Midas.WebConfirmations.ExportConfirmationServiceBehaviour">
<serviceMetadata httpGetEnabled="true" />
<serviceDebug includeExceptionDetailInFaults="false" />
</behavior>
</serviceBehaviors>
<endpointBehaviors>
<behavior name="webEndpointBehavior">
<webHttp defaultBodyStyle="Wrapped" defaultOutgoingResponseFormat="Xml" helpEnabled="true"/>
</behavior>
</endpointBehaviors>
</behaviors>
<client>
<endpoint address="http://devbucket.ministryofsound.mos.local/MidasWebServices/DataService.svc" binding="basicHttpBinding" bindingConfiguration="BasicHttpBinding_IDataService" contract="Midas.WebServiceClients.IDataService" name="BasicHttpBinding_IDataService" />
<endpoint address="http://devbucket.ministryofsound.mos.local/MidasWebConfirmations/ExportConfirmationService.svc" binding="webHttpBinding" bindingConfiguration="WebHttpBinding_IExportConfirmationService" behaviorConfiguration="webEndpointBehavior" contract="IExportConfirmationService" name="WebHttpBinding_IExportConfirmationService" />
</client>
</system.serviceModel>
</configuration>
So if any of the great minds that frequent Stack Overflow could shed some light on this problem for me, I'd greatly appreciate it.
UPDATE >>>
In response to the first few comments, I suspected that the problem could be caused by an Exception being thrown from the server side code, so I simplified the operation code dramatically... now all it does is this, but I still get the same error:
public void AddExportConfirmation(string upc, component ingestionFeedback)
{
WebOperationContext.Current.OutgoingResponse.StatusCode = HttpStatusCode.OK;
}
Also, I did set up tracing, but it only worked on the client side, so it only told me what I already knew. I will have a look at the link you provided #BigDaddy, in case it shows how to set up tracing on the server side.
In response to Tewr, I generated the service client using svcutil.exe, but I also tried adding a service reference and letting Visual Studio create the reference for me... both methods resulted in the same error. Also, I have been updating the service references all day, as I've been making changes. The includeExceptionDetailInFaults="true" setting didn't make any difference, but I'll try adding a dummy operation to the service and try to view it in the browser.
UPDATE 2 >>>
Ok, so I added a simple getter method to the service and updated the references all round like #Tewr suggested. This has just made me more confused... the method:
[XmlSerializerFormat()]
[OperationContract]
[WebGet()]
string GetString();
The implementation just returns a string and I see that value when I access the service in a web browser:
However, I still get the same error from code, even when calling this same new operation... what does that mean?
UPDATE 3 >>>
After taking the advice from the comments, I set up a Service Trace on the service again... I still couldn't get the one working on the server, but on the client, it did output a trace file. In that file, I see an InvalidOperationException with the following message:
Envelope Version 'EnvelopeNone (http://schemas.microsoft.com/ws/2005/05/envelope/none)' does not support adding Message Headers.
I'm just researching this now, so if you know what this error is about, please let me know.

The problem with WCF is that it is so complex, the Exceptions cover so many different errors each and the Exception messages are so vague. From my limited experience, it seems like an error message will say one thing, but it will often be wholly, or partly irrelevant to your actual problem. So I basically kept fixing errors, which unlocked new Exceptions (and I still haven't come to the end of them yet!), but for the purpose of this question, there is an answer.
So it turned out that the original EndpointNotFoundException was actually thrown because there was an unhandled Exception in the AddExportConfirmation service method implementation. Once I had simplified the code (as in the first question update), that particular Exception went away and was replaced with the next one.

Getting Exception when trying to upload a big files size

I'm using wshttpbinding for my service
<wsHttpBinding>
<binding name="wsHttpBinding_Windows" maxBufferPoolSize="9223372036854775807" maxReceivedMessageSize="2147483647">
<readerQuotas maxArrayLength="2147483647" maxBytesPerRead="2147483647" maxStringContentLength="2147483647" maxNameTableCharCount="2147483647"/>
<security mode="Message">
<message clientCredentialType="Windows"/>
</security>
</binding>
</wsHttpBinding>
<behavior name="ServiceBehavior">
<dataContractSerializer maxItemsInObjectGraph="6553600"/>
<serviceThrottling maxConcurrentCalls="2147483647" maxConcurrentInstances="2147483647" maxConcurrentSessions="2147483647"/>
</behavior>
and when i try to upload a file of 15Mb it throws the EndPointNotFoundException below:
Exception Message:
There was no endpoint listening at "MY SERVICE URL" that could accept the message. This is often caused by an incorrect address or SOAP action. See InnerException, if present, for more details.
Exception:
The remote server returned an error: (404) Not Found.

There are (2) settings maxRequestLength and maxAllowedContentLength on the server side in your WCF confgiration that you are going to need to increase in order for this exception to be resolved. In the .config for the WCF service server side make sure to add the following:
<system.web>
<!--Increase 'maxRequestLength' to needed value: 100mb (value is in kilobytes)-->
<httpRuntime maxRequestLength="102400"/>
</system.web>
<system.webServer>
<security>
<requestFiltering>
<!--Increase 'maxAllowedContentLength' to needed value: 100mb (value is in bytes)-->
<requestLimits maxAllowedContentLength="104857600" />
</requestFiltering>
</security>
</system.webServer>

The upload limit is set at 2 level, 1st by the Application and 2nd by the Server.
Your app configuration looks good to me.
Check your IIS settings as defined in your machine.config file.
I found a Microsoft KB article to configure this
You can configure the value at any of these places.