We have a service implmented in WCF. It works fine in all environment except in a particular environment it often throws below exception
System.ServiceModel.ProtocolException: The remote server returned an unexpected response: (502) Bad Gateway.
System.Net.WebException: The remote server returned an error: (502) Bad Gateway. at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)...
The called methods return a Boolean value. But it is a big transaction which takes time on DB end often 15 mins. Our configuration is done for 50 mins.
Sometime is completes the transaction. Sometime it throws above error just after 2 minutes.
On the target machine there are load balancers involve. I have doubt that is error is because of them.
If the reason is so.. then what are my options??
Related
When consuming WCF in a local environment, everything is working just fine.
But when consuming WCF hosted on some external location (Azure etc.), for request process that lasts longer than 1 minute, I get the error:
System.TimeoutException: 'The request channel timed out while waiting for a reply after 01:01:00. Increase the timeout value passed to the call to Request or increase the SendTimeout value on the Binding. The time allotted to this operation may have been a portion of a longer timeout.'
WebException: The remote server returned an error: (504) Gateway Timeout.
This error is showing after 1 minute, not after 1 hour as it says.
The service is using wsHttpBinding.
Any idea what could be a problem here?
Have a really crazy situation. I can't post specifics, so I'm just looking for general guidance. We have already opened a ticket with Oracle/MySql support. I'm just looking to see if anyone else has run into this situation or anything similar. Here is our scenario:
Windows 2012 R2 Server with .NET 4.7.1 running.
Simple Windows Forms .NET application.
We are trying to run a simple query against a Linux MySql Server. MySql is Enterprise Version 5.7.x.
On the first attempted connection, the Windows Forms app locks the UI, waits about 15 seconds, and then reports back that there is an error running the command. The error is shown below.
System.ApplicationException: An exception occurred on the following sql command:select * from tablename where compl_date >= '2019-12-17 04:44:34 PM' ---> MySql.Data.MySqlClient.MySqlException: Authentication to host 'ip address' for user 'userid' using method 'mysql_native_password' failed with message: Reading from the stream has failed. ---> MySql.Data.MySqlClient.MySqlException: Reading from the stream has failed. ---> System.IO.EndOfStreamException: Attempted to read past the end of the stream.
When this error pops up, if I click on the "Continue" button, subsequent calls to the database work as intended (at about a 95% rate).
On the server, the mysqld error logs are shown below for the first call. Subsequent calls do work.
2019-12-16T22:06:29.554171Z 3496 [Warning] IP address 'client ip address' could not be resolved: Name or service not known
2019-12-16T22:06:50.188443Z 3496 [Note] Aborted connection 3496 to db: 'drupaldb' user: 'userid' host: 'ip address' (Got an error reading communication packets)
2019-12-17T02:53:17.832725Z 0 [Note] InnoDB: page_cleaner: 1000ms intended loop took 11355ms. The settings might not be optimal. (flushed=0 and evicted=0, during the time.)
2019-12-17T03:25:18.200855Z 3527 [Note] Got an error reading communication packets
2019-12-17T03:25:37.167395Z 3528 [Note] Got packets out of order
2019-12-17T03:25:37.382512Z 3529 [Note] Got packets out of order
2019-12-17T03:25:47.688836Z 3530 [Note] Bad handshake
2019-12-17T14:26:33.619967Z 4803 [Note] Got timeout reading communication packets
2019-12-17T19:34:34.741441Z 4851 [Note] Got timeout reading communication packets
2019-12-17T19:47:47.595426Z 4853 [Note] Got timeout reading communication packets
2019-12-17T19:48:45.586357Z 4854 [Note] Got timeout reading communication packets
If you have some general ideas, let me know.
FYI, we have some other linux/mysql instances, and this runs just fine.
At this point, we think we have solved the problem, at least for the short term. Both server and client are sitting on a private network. We think that the database server is trying to send a certificate to the windows client. The windows client is also on this private network. We think the Windows Client is not accepting the ssl certificate and that this is causing the failure on the first connection attempt. By adding the option "SslMode=None", this seems to resolve the issue.
Blog post we found that helped us: https://blog.csdn.net/fancyf/article/details/78295964
I'm trying to move big bunch of messages to another folder and the operation take approximately 5-8 minutes depends of the servers' locations and amount of items to move.
And I always get an error (504) Gateway Timeout,
with the stack trace:
Microsoft.Exchange.WebServices.Data.ServiceRequestException: The request failed. The remote server returned an error: (504) Gateway Timeout. ---> System.Net.WebException: The remote server returned an error: (504) Gateway Timeout.
at System.Net.HttpWebRequest.GetResponse()
at Microsoft.Exchange.WebServices.Data.EwsHttpWebRequest.Microsoft.Exchange.WebServices.Data.IEwsHttpWebRequest.GetResponse()
at Microsoft.Exchange.WebServices.Data.ServiceRequestBase.GetEwsHttpWebResponse(IEwsHttpWebRequest request)
--- End of inner exception stack trace ---
at Microsoft.Exchange.WebServices.Data.ServiceRequestBase.GetEwsHttpWebResponse(IEwsHttpWebRequest request)
at Microsoft.Exchange.WebServices.Data.ServiceRequestBase.ValidateAndEmitRequest(IEwsHttpWebRequest& request)
at Microsoft.Exchange.WebServices.Data.MultiResponseServiceRequest`1.Execute()
at Microsoft.Exchange.WebServices.Data.ExchangeService.InternalMoveItems(IEnumerable`1 itemIds, FolderId destinationFolderId, Nullable`1 returnNewItemIds, ServiceErrorHandling errorHandling)
at Microsoft.Exchange.WebServices.Data.ExchangeService.MoveItems(IEnumerable`1 itemIds, FolderId destinationFolderId)
at EAS.Exchange.ExchangeProvider.MoveItemsToFolder(String folderUniqueId, String[] itemUniqueIds)
I was trying to increase timeout for ExchangeService:
var ews = new Microsoft.Exchange.WebServices.Data.ExchangeService();
ews.Timeout = 900000; // 15 minutes in milliseconds
but it doesn't solve the problem. By the way, the operation finishes successfully on the EWS and all the items are moved after that.
Could you please recommend another possible solution how to fix this issue?
The 504 error indicates that the remote server experienced a timeout. If you experienced a direct timeout, after all, then nothing would have had a chance to send back an HTTP status code.
So the issue is not due to the connection between your program and the Exchange service; the problem is due to one of the following:
A connection is timing out between the Exchange service and some other service that Exchange uses internally. You will need to figure out which component is timing out and adjust the timeout settings specific to that area of the system. There are a lot of timeouts to look at. You could start here.
Your access to the Exchange service is mediated by a proxy server, and the connection between the proxy server and Exchange is timing out. You may need to adjust the proxy server settings.
You may be able to determine which of the above is the issue by inspecting the W3C logs and searching for 504s. Or you may be able to tell what a 504 looks like when it comes from the proxy simply by its styling, depending on how you have things set up. Another possibility is to add the Exchange server's address to your proxy exclusion list and see if that has any effect.
I am using the plain C# await httpListenerContext.AcceptWebSocketAsync(subProtocol: null), everything seems to work great and data is being sent and transmitted from Chrome successfully, however sometimes the connection seems to get dropped for no reason or the client will send the data partially, this is especially annoying since the data is JSON and I will only get half the object (and I am certain I send it full on the client)
I am testing on localhost so any connection issues aren't a possibility (not that they would be anyway because of TCP)
I see that I sometimes receive the following exception:
Error while processing account:The 'System.Net.Web
Sockets.ServerWebSocket' instance cannot be used for communication because it ha
s been transitioned into the 'Aborted' state.:System.Net.WebSockets.WebSocketExc
eption (0x80004005): The 'System.Net.WebSockets.ServerWebSocket' instance cannot
be used for communication because it has been transitioned into the 'Aborted' s
tate. ---> System.OperationCanceledException: The operation was canceled.
at System.Threading.CancellationToken.ThrowOperationCanceledException()
at System.Net.WebSockets.WebSocketHttpListenerDuplexStream.<ReadAsyncCore>d__
I have a problem with wcf servcices.
Wcf services method is inovked by an application. This app calls service method very often (dozens of times per minute). The service method is called properly (with Close() at the end, or Abort() after exception). The most strange thing for me is that after few hours my app is getting errors from services:
An error occurred while receiving the HTTP response to http://domain.xx/MyService.svc. This could be due to the service endpoint binding not using the HTTP protocol. This could also be due to an HTTP request context being aborted by the server (possibly due to the service shutting down). See server logs for more details. The underlying connection was closed: An unexpected error occurred on a receive. Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host.
or this one:
*The request channel timed out while waiting for a reply after 00:15:00. Increase the timeout value passed to the call to Request or increase the SendTimeout value on the Binding. The time allotted to this operation may have been a portion of a longer timeout. The HTTP request to 'http://domain.xx/MyService.svc' has exceeded the allotted timeout of 00:15:00. The time allotted to this operation may have been a portion of a longer timeout. The operation has timed out *
What can couse such errors? Why services are working properly within few first hours?
I would check your application log. From my experience, those errors tend to me more server related than code related. IIS may be having problems.
I know you mentioned it but it looks your are not closing your channels right. Also, make sure you do NOT use the same client for many server calls. Just create one, use it for a single call, and dispose it.
Here's a good read about closing WCF channels.