We Keep Coding

Getting spikes in gRPC calls every hour

I have a website running with around 7 servers (C#). And there is a gRPC service (golang) running with 3 instances. Each web server connects to and makes calls to the gRPC service. There are around 8000 calls per minute to the service.
The call to this service is not that critical, so lately we reduced the deadline of the call to 20 milliseconds. Here we noticed something strange. There was a spike in the "deadline exceeded" errors every hour throughout the day. And it happens exactly at 0th minute i.e. 2pm 3pm 4pm etc.
Why does this happen?
I came across this link saying gRPC resets the connection every hour, but nothing more than that.
So my question is does gRPC internally refresh the connection every hour. If yes is there anyway to tweak this behavior. If no then can someone give some direction as to how I can debug why this is happening.

No, grpc-go does not refresh connections. The only time it initiates a disconnect is if you configure "max idle" (ref) and the connection has been idle for longer than that time limit. By default this is disabled, so it's unlikely to be the culprit in this case.

Time out error occur on c# application

When i am posting data from c# application(Windows server) to PHP page which runs on another server(Ubuntu) using POST method,
i am posting minimum 1000 request per second to PHP page,
c# application is a multi threading application, once it receives the data it post the data to php page
when i continuously posting datas i'm getting posting timeout error on c# application, once i restart the application it works for few hours.
[Note: due to php takes time to finish the task so new request are waiting , it creates queue and its waiting time exceed more than 2 min and im getting timeout error].
Both of our server use maximum 50% of CPU and RAM usage
i check on both c# code and PHP code both are working fine there is no issues or bugs
and i check on mysql configuration also fine but i dont know about apache config.
Apache config are set as default
what i think is may be i should config apache or php to handle 1000 request per second, i dont know exactly because same code working fine until clients request
increased.
thanks in advance buddy :)

I think you might be hitting a TCP Port Exhaustion issue. If you are making many sequential calls to another server, and dont manage the TCP connections properly your OS will not immediately release the TCP Port connection it created for the outgoing call, and will assign further OS resources to the next call. I think the default TCP Port release time can be as high as 2 minutes.
See How do I prevent Socket/Port Exhaustion? for further details. To be sure we'd need to see your C# code to see how you are releasing the resource you use when creating the WebClient call.
If it is a port exhaustion issue, then you are going to have to manage your outgoing calls to the PHP server using a manually created pool of WebClient instances - even releasing the WebClient may not immediately release the OS resources that the WebClient made use of.

thank u for kind reply bro,
it was config issue on ubuntu server i didnt enable fast cgi now its works fine

How To Keep My Web Services Awake?

I have written two web services that I am running on GoDaddy. One is a Microsoft WCF web service and the other is a RESTful Web API service. They are both working, but they rarely get traffic. If I don't call the web services for some period of time they seem to go to sleep. Then when I load the pages that call the web services they take some 20 to 30 seconds to retrieve data from the services. After that if I continue to call them repeatedly they load in just a second or two. Is this normal or did I do something wrong in my configuration? Is there some way to keep them active?

Entirely normal. You can either increase the recycle time limit in IIS (but you will still get recycled eventually) or you can write a quick scheduled task like the following to run every 10 minutes or so:
powershell Invoke-WebRequest -Uri "http://example.com"
Although I would caution that you should forcefully restart the service sometime during low usage hours just to clear the process memory / resource utilization.

SOAP - Operation Timed Out - Rule out my server

I am using SOAP in C# .Net 3.5 to consume a web service, from a video game company. I am having lots of SOAP Exceptions with the error "Operation Timed Out"
While one process is timing out, others fly by with no problems. I would like to rule out a problem on my end, but I have no idea where to begin. My timeout is 5 minutes. For every 5,000 requests, maybe 500 fail.
Anyone have some advice for diagnosing web services failures? The web service owner will probably give no support to helping me on this, as it's a free service.
Thanks

I've had to do a lot of debugging connecting to a SOAP Service using PHP and timeouts are the worst problem. Normally the problem is the 'client' doesn't have a high enough timeout and bombs after something like 30s.
I test making the calls using SoapUI. I keep using a higher client-side timeout using that until I find something that works. Once I find that out I use the newly found time to my client and re-test.
Your only solution may be to make sure your 'clients' have a high enough timeout that will work for everything. 5 minutes should be fine for most of your server-side timeouts.

OK this is a huge question and there is a lot that it could be.
Have you tackled HTTP two connection limit? http://madskristensen.net/post/Optimize-HTTP-requests-and-web-service-calls.aspx
Have you got enough IO threads to cater for the load? Use the performance monitoring to check this for your App Pool - I think there is a IO threads counter. A quick google turned this up - http://www.guidanceshare.com/wiki/ASP.NET_2.0_Performance_Guidelines_-_Threading
Are you exhausting your bandwidth? Use performance monitoring again to check the usage of your network card.
This is a really hard subject to broach textually, as it so dependent on environment but I hop these might help.
This also looks interesting - http://software.intel.com/en-us/articles/how-to-tune-the-machineconfig-file-on-the-aspnet-platform/

Is there a way to tell how many "Concurrent Calls" are being made to a WCF service?

I've a WCF service that at the moment 11 clients ping in to every 3 minutes. They have all been running fine for a couple weeks. Last night, they all of a sudden stopped being able to ping in due to timing out. So I looked at my server web.config. Specifically:
<serviceThrottling maxConcurrentCalls ="50" maxConcurrentSessions="200"/>
which should have no problems working.
I changed both values to 500, saved the file, and everything started working again. So it must have been some issue with how many connections where being made.
My question is: is there a way to view on the server app how many concurrent calls there are currently? Like some sort of monitoring system? This would help me to find why 50 possible calls was not enough for 11 clients.
Question 2: Does editing a service's web.config, then saving it, reset all the connections? Or was it just that I made the concurrent calls larger?

You might want to look at the WMI instrumentation available:
http://msdn.microsoft.com/en-us/library/ms735098.aspx
Windows Communication Foundation (WCF)
includes a large set of performance
counters to help you gauge your
application's performance.