We are trying to build a web service that can send/receive large data files from kb~gb sizes.
I know that we should use streaming to preform that so is not going to crash the memory.
But problem is we don't want to change our design, or more like, now we are going to try to see what will happen when we transfers large file say just 150mb file in a byte[] buffered transfers mode with WCF. (I know is goign to buffer the whole file into memory... and crash/exception if we transfer gb size files...)
But even so, they want to see what happen, so I have my wsHttpBinding in WCF config:
<system.web>
<compilation debug="true"/>
<httpRuntime maxRequestLength="524288"/>
</system.web>
....
<wsHttpBinding>
<binding name="wsHttpBinding1" closeTimeout="00:10:00" openTimeout="00:10:00"
receiveTimeout="00:30:00" sendTimeout="00:30:00" bypassProxyOnLocal="false"
hostNameComparisonMode="StrongWildcard"
maxBufferPoolSize="52428800" maxReceivedMessageSize="65536000"
messageEncoding="Mtom" useDefaultWebProxy="true">
<security mode="None" />
</binding>
</wsHttpBinding>
Then my client app config:
<wsHttpBinding>
<binding name="WSHttpBinding_IPrintService" closeTimeout="00:01:00"
openTimeout="00:01:00" receiveTimeout="00:30:00" sendTimeout="00:30:00"
bypassProxyOnLocal="false" transactionFlow="false" hostNameComparisonMode="StrongWildcard"
maxBufferPoolSize="52428800" maxReceivedMessageSize="65536000"
messageEncoding="Mtom" textEncoding="utf-8" useDefaultWebProxy="true"
allowCookies="false">
<readerQuotas maxDepth="32" maxStringContentLength="8192" maxArrayLength="16384"
maxBytesPerRead="4096" maxNameTableCharCount="16384" />
<reliableSession ordered="true" inactivityTimeout="00:10:00"
enabled="false" />
<security mode="None"/>
</binding>
</wsHttpBinding>
When I transfers small data it works fine, but when I try to transfers 150mb file, it gives me exceptions:
InnerException = {"Unable to write data to the transport
connection: An operation on a socket could not be performed because
the system lacked sufficient buffer space or because a queue was
full."}
I tried to increase maxReceivedMessage and such, but it still having the same problem....
======
Edit: I tried to increase all possible value above to... 2147483647.... I can sent a 30mb file successfully..but still not 150mb.... And I also tried to use this CustomBinding:
<customBinding>
<binding name="LargeCustomBinding"
receiveTimeout="01:00:00"
openTimeout="01:00:00"
closeTimeout="01:00:00"
sendTimeout="01:00:00">
<binaryMessageEncoding
maxReadPoolSize="2147483647"
maxSessionSize="2147483647"
maxWritePoolSize="2147483647">
<readerQuotas
maxArrayLength="2147483647"
maxDepth="2147483647"
maxBytesPerRead="2147483647"
maxNameTableCharCount="2147483647"
maxStringContentLength="2147483647" />
</binaryMessageEncoding>
<httpTransport
maxBufferSize="2147483647"
maxReceivedMessageSize="2147483647"
maxBufferPoolSize="2147483647"/>
</binding>
</customBinding>
But I am still getting the same problem....
=========
Edit 2: I guess I should put concern on how I sending the data, maybe I was wrong, I have an Object MyData:
byte[] LargeData = File.ReadAllBytes(this.LargeDataPath.Text);
MyData.LargeData = new List<KeyValuePair<string, byte[]>>()
{
new KeyValuePair<string, byte[]>("DATA", LargeData)
};
myServiceClient.SendDataAsync( new SendDataRequest(MyData));
There should be no problem? All I did is just sending one big byte[] inside a list.....
You should be sending a smaller chunks of data with a separate send method calls.
Put the data you want to send into a byte array.
Then realize the logic on both ends, that will know how to handle sizes and just start sending the data in a smaller pieces, one by one. When you have sent an entire file, it will get concatenated into an array on a client and you will be able to do further processing.
Fix this first:
maxStringContentLength="8192"
maxArrayLength="16384"
And ensure you have
maxStringContentLength="2147483646"
maxArrayLength="2147483646"
On both client and server.
I was sending 30Mb over WCF without problems.
And check out this
Related
I am using WCF service. The problem I have is its starts using double memory.
I am using HTTPS binding
<wsHttpBinding>
<binding name="secureHttpBinding" closeTimeout="04:01:00" openTimeout="04:01:00" receiveTimeout="04:10:00" sendTimeout="04:01:00" allowCookies="false"
bypassProxyOnLocal="false" hostNameComparisonMode="StrongWildcard"
messageEncoding="Text" textEncoding="utf-8" useDefaultWebProxy="true">
<readerQuotas maxDepth="128" maxStringContentLength="2147483647" maxArrayLength="2147483647" maxBytesPerRead="2147483647" maxNameTableCharCount="2147483647"/>
<security mode="Transport" >
<transport clientCredentialType="None"/>
</security>
</binding>
</wsHttpBinding>
<endpoint address="https://localhost/test.svc"
binding="wsHttpBinding"
bindingConfiguration="secureHttpBinding"
contract="IWcfContract"
name="SecureBasicHttpBinding_WcfContract">
Here is the code I am using to upload
using (Stream fileStream = File.OpenRead(logsZipFullPath))
{
// Call web server
UploadFileResponse response = _webServiceHelper.UploadFile(fileStream, currentDate, ".zip", string.Empty);
fileStream.Close();
}
Here is my model
[MessageContract]
public class UploadFileRequest : IDisposable
{
[MessageBodyMember(Order = 1)]
public Stream FileByteStream;
[MessageHeader(MustUnderstand = true)]
public string FileDescription;
}
My zip file is of 80MB.
The problem I have is at the start of the service its using 26mb which is quite fine. At the first call it uses 136MB after call completes it goes down to 26mb. which is also fine. after the second call to upload it starts using 346MB
which again gets down to 26mb after service call. My question is why it is using 346MB when the file is of only 80MB? My GC and disponse has been called correctly. But, is this normal behaviour or I am missing anything?
finally found a work around for this. According to this post
wsHttpBinding is the full-blown binding, which supports a ton of WS-*
features and standards - it has lots more security features, you can
use sessionful connections, you can use reliable messaging, you can
use transactional control - just a lot more stuff, but wsHttpBinding
is also a lot *heavier" and adds a lot of overhead to your messages as
they travel across the network
I think this is the main reason why the memory usage is high. Our requirement was to use the HTTPS certificate and we are able to use this with basicHttpBinding. SO, I changed my settings like this
<basicHttpBinding>
<binding name="BasicHttpBinding_WcfContract" closeTimeout="04:01:00" openTimeout="04:01:00" receiveTimeout="04:10:00" sendTimeout="04:01:00" allowCookies="false"
bypassProxyOnLocal="false" hostNameComparisonMode="StrongWildcard" maxBufferSize="2147483647" maxBufferPoolSize="2147483647"
maxReceivedMessageSize="2147483647" messageEncoding="Text" textEncoding="utf-8" transferMode="Streamed" useDefaultWebProxy="true">
<readerQuotas maxDepth="128" maxStringContentLength="2147483647" maxArrayLength="2147483647" maxBytesPerRead="2147483647" maxNameTableCharCount="2147483647"/>
<security mode="Transport"> <-- this is the main change
<transport clientCredentialType="None" proxyCredentialType="None" realm=""/>
<message clientCredentialType="UserName" algorithmSuite="Default"/>
</security>
</binding>
</basicHttpBinding>
Note: I changed security mode none to transport to support https.
Now, everything works without the issue.
Two possible things which causes memory issues.
wsHttpBinding overheads
Stream mode is not supported in wsHttpBinding
i'm currently working on a client/server program which is using wshttpdualbinding for communication in vs with c#. So i'm calling webservices over this binding from the server.
Everything worked fine, but in the last days i'm encountering the following problem: if i call a service from the server, the client just stops working and stops reacting. Nothing gets send from the client.
I can't post much, because it is a big project, but i'll try to post the best snippets:
binding:
<bindings>
<basicHttpBinding>
<binding name="BasicHttpBinding_AuthenticationService" closeTimeout="03:10:00"
openTimeout="03:10:00" receiveTimeout="03:10:00" sendTimeout="03:10:00"
bypassProxyOnLocal="false" maxBufferPoolSize="2147483647" maxReceivedMessageSize="2147483647"
useDefaultWebProxy="true" allowCookies="false" messageEncoding="Text" textEncoding="utf-8" transferMode="Buffered" >
<readerQuotas maxDepth="2147483647" maxStringContentLength="2147483647"
maxArrayLength="2147483647" maxBytesPerRead="2147483647" maxNameTableCharCount="2147483647" />
<security mode="None">
<transport clientCredentialType="None" proxyCredentialType="None"
realm="" />
<message clientCredentialType="UserName" algorithmSuite="Default" />
</security>
</binding>
</basicHttpBinding>
<wsDualHttpBinding>
<binding name="WSDualHttpBinding_Service" closeTimeout="03:10:00"
openTimeout="03:10:00" receiveTimeout="03:10:00" sendTimeout="03:10:00"
bypassProxyOnLocal="false" maxBufferPoolSize="2147483647" maxReceivedMessageSize="2147483647"
useDefaultWebProxy="true" messageEncoding="Text" textEncoding="utf-8" >
<readerQuotas maxDepth="2147483647" maxStringContentLength="2147483647"
maxArrayLength="2147483647" maxBytesPerRead="2147483647" maxNameTableCharCount="2147483647" />
<security mode="None">
<message clientCredentialType="UserName" algorithmSuite="Default"/>
</security>
</binding>
</wsDualHttpBinding>
</bindings>
problem causing code:
Clusterrationalemapping cr = new Clusterrationalemapping
{
Textbricks = rationale,
Cluster = cluster,
Project = CurrentProject
};
var clusterrationaleId = 0;
if (cluster.AddClusterRationaleMapping(cr))
{
clusterrationaleId = _service.SaveItem(cr);
cr.Id = clusterrationaleId;
if(rationale.Clusterrationalemappings == null) rationale.Clusterrationalemappings = new List<Clusterrationalemapping>();
rationale.Clusterrationalemappings.Add(cr);
}
A function gets called which creates a new clusterrationalemapping for a specific cluster. The rational and CurrentProject variables are always the same. just cluster is different.
Most of the time everything works and service.SaveItem returns the new Id as expected. But for one special cluster it is not working. service.SaveItem just stops and freezes the tool until i get some exception (server did not respond in time/gave no answer).
I'm sure that the wsDualHttpBinding is set correctly. If i comment out the line Project = CurrentProject and replace it with Project = new Project(); everything works fine for this cluster again.
So it looks like clusterrationalmapping might get too big for sending. But the exception is not saying that. And i got it from the server - from that direction it worked.
I used Microsoft Service Trace Viewer for debugging purpose and i can see there, that it instantly throws an exception that the endpoint could not be reached. So i'm pretty sure that the service dies when serializing the object.
Is there any way to debug into the service to see when exactly, or better why, it fails?
Thanks in advance for your help.
Since I have not been able to find a solution - I still think that there has been a problem with the serializer and a too big/nested class, i changed the binding to nettcp and it is working now.
I need to send the combination (login name / User Password) from my web application to a web service that autenticate all employees in all the companie's programs. But what is happening rigth now is plain text going throught the wire. I was asked to use TLS (not SSL3) in order to encrypt the combination for decent security (asp.net 4.0 Framework, c#).
The problem is I don't even know where to start, everywhere I look for similar questions, I see peoples that are arleady very close to there goal, only needing one last line of code to confirm the certificate or one last little configuration to allow the "handshake" process.
The way I am accessing my web services seems TOO SIMPLE, maybe I have to start all over from the begining, that's is why I want to ask poeple first. Here is how i call the web service's fonctions :
In the Web.Config :
//Binding
<binding name="BasicHttpBinding_ICommun1" closeTimeout="00:01:00"
openTimeout="00:01:00" receiveTimeout="00:10:00" sendTimeout="00:01:00"
allowCookies="false" bypassProxyOnLocal="false" hostNameComparisonMode="StrongWildcard"
maxBufferSize="65536" maxBufferPoolSize="524288" maxReceivedMessageSize="65536"
messageEncoding="Text" textEncoding="utf-8" transferMode="Buffered"
useDefaultWebProxy="true">
<readerQuotas maxDepth="32" maxStringContentLength="8192" maxArrayLength="16384"
maxBytesPerRead="4096" maxNameTableCharCount="16384" />
<security mode="None">
<transport clientCredentialType="None" proxyCredentialType="None"
realm="" />
<message clientCredentialType="UserName" algorithmSuite="Default" />
</security>
//Declaring EndPoint
<endpoint address="http://CannotTellYouThat.svc"
binding="basicHttpBinding" bindingConfiguration="BasicHttpBinding_ICommun1"
contract="NameOfTheService.ICommun" name="BasicHttpBinding_ICommun1" />
In the C# class, I only need this :
public NameOfTheService.Utilisateur login(string ip, string nomUtilisateur, string userPass)
{
NameOfTheService.CommunClient instance = new NameOfTheService.CommunClient();
return instance.Authentifier(ip, nomUtilisateur, userPass, AuthentificationServices.SystemeEnum.AppName);
}
It returns a User (Utilisateur) with all the informations needed. This looks like I have to remove the web.Config binding and do all that in the C#. Any kind of help or advice would be welcome. Thanks.
I have a small application that uses WCF to communicate with a webserver. This program is used by some 200 clients, and each client is sending in about 5-20 requests/min.
Looking at the error logs I often get:
The request channel timed out while waiting for a reply after
00:00:59.9989999
The requests are made as follows:
ClientService Client = new ClientService();
Client.Open();
Client.updateOnline(userID);
Client.Close();
This is the app.config
<configuration>
<configSections>
</configSections>
<startup><supportedRuntime version="v2.0.50727"/></startup><system.serviceModel>
<bindings>
<wsHttpBinding>
<binding name="WSHttpBinding_IClientService" closeTimeout="00:01:00"
openTimeout="00:01:00" receiveTimeout="00:10:00" sendTimeout="00:01:00"
bypassProxyOnLocal="false" transactionFlow="false" hostNameComparisonMode="StrongWildcard"
maxBufferPoolSize="524288" maxReceivedMessageSize="65536"
messageEncoding="Text" textEncoding="utf-8" useDefaultWebProxy="true"
allowCookies="false">
<readerQuotas maxDepth="32" maxStringContentLength="8192" maxArrayLength="16384"
maxBytesPerRead="4096" maxNameTableCharCount="16384" />
<reliableSession ordered="true" inactivityTimeout="00:10:00"
enabled="false" />
<security mode="None">
<transport clientCredentialType="Windows" proxyCredentialType="None"
realm="" />
<message clientCredentialType="Windows" negotiateServiceCredential="true" />
</security>
</binding>
</wsHttpBinding>
</bindings>
<client>
<endpoint address="PATH TO SERVICE"
binding="wsHttpBinding" bindingConfiguration="WSHttpBinding_IClientService"
contract="IClientService" name="WSHttpBinding_IClientService">
<identity>
<dns value="localhost" />
</identity>
</endpoint>
</client>
</system.serviceModel>
Of the "many calls" each day about 200-800 fail. And about n-1 are ok. I am very confused what the issue can be. Looking at the server stats, it's hardly building up a sweat - each request takes <2 sec to process. The data it's sending consists of "int" or "very small strings" - so, its not due to size, if it were - there should be a consistent failure..
Suggestions?
Try adding the timeout values to both the service AND the client:
<binding name="BasicHttpBinding_SomeName" closeTimeout="00:01:00"
openTimeout="00:01:00" receiveTimeout="00:10:00"
sendTimeout="00:10:00" maxBufferPoolSize="2147483647"
maxBufferSize="2147483647" maxReceivedMessageSize="2147483647">
It seems that your requests are queuing up on server before being handled and then starts to time out. You can try couple of things here to see exactly whats going on
1) Try throttling in your WCF services e.g: Try increasing your concurrent sessions. Take a look
WCF Throttling
2) Try using PerCall rather than Using Sessions here to make sure that no session remains there.
Here is what you can do on your interface to remove session
[ServiceContract(Namespace="YOUR NAMESPACE", SessionMode=SessionMode.NotAllowed)]
and your contract classes implementing those interface
ServiceBehavior(InstanceContextMode = InstanceContextMode.PerCall)]
AND .... You should enable tracing to see exactly whats going on .
Check this performance counter - ASP.NET Requests Queued.
One web application can host many web pages and services. IIS processes only 12 concurrent requests per CPU core by default.
This means that even if your service is fast but other pages/services are slow your request has to wait in queue before they are executed.
ASP.NET Requests Queued should be zero or close to zero.
See Performance Counters for ASP.NET on MSDN.
My service works great under low load. But under high load I start to get connection errors.
I know about other settings but I am trying to change the listenBacklog parameter in particular for my TCP Buffered binding.
If I set listenBacklog="10" I am able to telnet into the port where my WCF service is running.
If I change listenBacklog to anything higher than 10 it will not let me telnet into my service when it is running.
No errors seem to be thrown.
What can I do?
I get the same problem when I change my maxConnections away from 10. All other properties of the binding I can set higher without a problem.
Here is what my binding looks like:
<bindings>
<netTcpBinding>
<binding name="NetTcpBinding_IMyService" closeTimeout="00:01:00"
openTimeout="00:01:00" receiveTimeout="00:10:00" sendTimeout="00:01:00"
transactionFlow="false" transferMode="Buffered" transactionProtocol="OleTransactions"
hostNameComparisonMode="StrongWildcard" listenBacklog="10" maxBufferPoolSize="524288"
maxBufferSize="1048576" maxConnections="10" maxReceivedMessageSize="1048576">
<readerQuotas maxDepth="2147483647" maxStringContentLength="2147483647"
maxArrayLength="2147483647" maxBytesPerRead="2147483647" maxNameTableCharCount="2147483647" />
<reliableSession ordered="true" inactivityTimeout="00:10:00"
enabled="false" />
<security mode="Transport">
<transport clientCredentialType="Windows" protectionLevel="EncryptAndSign">
</transport>
<message clientCredentialType="Windows" />
</security>
</binding>
...
I really need to increase the values of maxConnections and listenBacklog
If you're running on a Win2000, XP, Vista or Win7 machine, then the OS is limiting you to 10 concurrent TCP connections. Try running on a Windows Server machine to confirm.
This is almost certainly a throttling issue. Try specifying non-default values for the service throttling behavior values. See this page for some guidance.
Try to remove all references to HTTP bindings and MEX.