I'm trying to send multiple API requests from C# webservice on every 5 minutes using HttpClient. After some time all sockets are exhausted because they are not closing after request is sent. How I can close all w3wp.exe sockets from web service method?
private static HttpClient client = new HttpClient();
public async Task sendReq()
{
var url = "http://url.com";
var sp = ServicePointManager.FindServicePoint(new Uri(url));
sp.ConnectionLeaseTimeout = 20 * 1000;
client.DefaultRequestHeaders.ConnectionClose = true;
client.DefaultRequestHeaders.Add("Connection", "Keep-Alive");
client.DefaultRequestHeaders.Add("Keep-Alive", "300");
client.DefaultRequestHeaders.ConnectionClose = true;
var res = await client.GetStringAsync(url);
}
Related
I have HTTP Listener application written in C# running on Win Server 2022:
HttpListener Listener = new HttpListener();
Listener.Prefixes.Add("https://test.mydomain.com:13001/");
Listener.Start();
while (true)
{
HttpListenerContext Context = Listener.GetContext();
HttpListenerRequest Req = Context.Request;
HttpListenerResponse Resp = Context.Response;
byte[] data = { };
if (Req.HttpMethod == "POST" && Req.HasEntityBody)
{
StreamReader reader = new StreamReader(Req.InputStream, Req.ContentEncoding);
string s = reader.ReadToEnd();
string Return = manageData(s); //code here calls function to work with data from http request and returns values
data = Encoding.UTF8.GetBytes(Return);
}
Resp.StatusCode = 200;
Resp.AppendHeader("Access-Control-Allow-Origin", "*");
Resp.ContentType = "text/plain; charset=utf-8";
Resp.ContentEncoding = Encoding.UTF8;
Resp.ContentLength64 = data.LongLength;
Resp.OutputStream.WriteAsync(data, 0, data.Length);
Resp.Close();
}
Then I have another application for clients. This code can be executed by anyone. This is the code:
HttpContent content = new StringContent("Request string", Encoding.UTF8, "application/json");
HttpResponseMessage response = await client.PostAsync(Uri, content);
response.EnsureSuccessStatusCode();
string responseBody = await response.Content.ReadAsStringAsync();
HttpStatusCode code = response.StatusCode;
Everything works well when port 13001 is fully opened, but doesn't work when only my application has allowed access. These are the firewall settings:
This rule has allowed access but the external apps can't connect. Am I missing something in code what tells the system to allow access to program? Or is there problem with firewall? I just want to increase server security by not opening port completely but only for my application.
Thanks for all advices
I am trying to download a HTML source code from a single website (https://www.faa.gov/air_traffic/flight_info/aeronav/aero_data/NASR_Subscription/) in C#.
The issue is that it takes 10 seconds to download a 30kb HTML page source. Internet connection is not an issue, as I am able to download 10Mb files in this program instantly.
The following has been executed both in a separate thread and in the main thread. It still takes 10-12 seconds to download.
1)
using (var httpClient = new HttpClient())
{
using (var request = new HttpRequestMessage(new HttpMethod("GET"), url))
{
var response = await httpClient.SendAsync(request);
}
}
2)
using (var client = new System.Net.WebClient())
{
client.Proxy = null;
response = client.DownloadString(url);
}
3)
using (var client = new System.Net.WebClient())
{
webClient.Proxy = GlobalProxySelection.GetEmptyWebProxy();
response = client.DownloadString(url);
}
4)
WebRequest.DefaultWebProxy = null;
using (var client = new System.Net.WebClient())
{
response = client.DownloadString(url);
}
5)
var client = new WebClient()
response = client.DownloadString(url);
6)
var client = new WebClient()
client.DownloadFile(url, filepath);
7)
System.Net.WebClient myWebClient = new System.Net.WebClient();
WebProxy myProxy = new WebProxy();
myProxy.IsBypassed(new Uri(url));
myWebClient.Proxy = myProxy;
response = myWebClient.DownloadString(url);
8)
using var client = new HttpClient();
var content = await client.GetStringAsync(url);
9)
HttpWebRequest myRequest = (HttpWebRequest)WebRequest.Create(Url);
myRequest.Method = "GET";
WebResponse myResponse = myRequest.GetResponse();
StreamReader sr = new StreamReader(myResponse.GetResponseStream(), System.Text.Encoding.UTF8);
string result = sr.ReadToEnd();
sr.Close();
myResponse.Close();
I want a faster way to do this in C#.
Any information or help you can provide is much appreciated.
I know that this is dated, but I think I found the cause: I've encountered this at other sites. If you look at the response cookies, you will find one named ak_bmsc. That cookie shows that the site is running the Akamai Bot Manager. It offers bot protection, thus blocks requests that 'look' suspicious.
In order to get a quick response from the host, you need the right request settings. In this case:
Headers:
Host: (their host data) www.faa.gov
Accept: (something like:) */*
Cookies:
AkamaiEdge = true
example:
class Program
{
private static readonly HttpClient _client = new HttpClient();
private static readonly string _url = "https://www.faa.gov/air_traffic/flight_info/aeronav/aero_data/NASR_Subscription/";
static async Task Main(string[] args)
{
var sw = Stopwatch.StartNew();
using (var request = new HttpRequestMessage(HttpMethod.Get,_url))
{
request.Headers.Add("Host", "www.faa.gov");
request.Headers.Add("Accept", "*/*");
request.Headers.Add("Cookie", "AkamaiEdge=true");
Console.WriteLine(await _client.SendAsync(request));
}
Console.WriteLine("Elapsed: {0} ms", sw.ElapsedMilliseconds);
}
}
Takes 896 ms for me.
by the way, you shouldn't put HttpClient in a using block. I know it's disposable, but it's not designed to be disposed.
This Question has stumped everyone I have asked. I have found a solution that I am going to stick with.
This solution does what I need it to do in 0.5 seconds on average. This will only work for windows from what I can tell. If the user does not have "CURL" I revert and go to the old way that takes 10 seconds to get what I need.
The solution creates a batch file in a temporary directory, calls that batch file to "CURL" the website, then output the result of CURL to a .txt file in the temp directory.
private static void CreateBatchFile()
{
string filePath = $"{tempPath}\\tempBat.bat";
string writeMe = "cd \"%temp%\\ProgramTempDir\"\n" +
"curl \"https://www.faa.gov/air_traffic/flight_info/aeronav/aero_data/NASR_Subscription/\">FAA_NASR.txt";
File.WriteAllText(filePath, writeMe);
}
private static void ExecuteCommand()
{
int ExitCode;
ProcessStartInfo ProcessInfo;
Process Process;
ProcessInfo = new ProcessStartInfo("cmd.exe", "/c " + $"{tempPath}\\tempBat.bat");
ProcessInfo.CreateNoWindow = true;
ProcessInfo.UseShellExecute = false;
Process = Process.Start(ProcessInfo);
Process.WaitForExit();
ExitCode = Process.ExitCode;
Process.Close();
}
private static void GetResponse()
{
string response;
string url = "https://www.faa.gov/air_traffic/flight_info/aeronav/aero_data/NASR_Subscription/";
CreateBatchFile();
ExecuteCommand();
if (File.Exists($"{tempPath}\\FAA_NASR.txt") && File.ReadAllText($"{tempPath}\\FAA_NASR.txt").Length > 10)
{
response = File.ReadAllText($"{tempPath}\\FAA_NASR.txt");
}
else
{
// If we get here the user does not have Curl, OR Curl returned a file that is not longer than 10 Characters.
using (var client = new System.Net.WebClient())
{
client.Proxy = null;
response = client.DownloadString(url);
}
}
}
Whenever I have to PUT a json string to a server, I launch a new thread which has this code inside a class. It works fine, but the thing is that a TCP connection is opened for each request. When I checked the ServicePoint hashcode, its the same for each request.
When I looked in TCPView, I cannot find those connections - I think its because its opened and closed within ~50ms.
So, 2 questions -
Is it an issue if I leave it like this? A new request will be raised every second from the client.
How do I reuse the same TCP connection? What if I set ServicePoint.KeepAlive to true?
public void SendRequest()
{
string sOutput="";
try
{
HttpWebRequest myWebRequest = (HttpWebRequest)WebRequest.Create(_uri);
myWebRequest.Timeout = Timeout;
myWebRequest.ReadWriteTimeout = Timeout;
myWebRequest.ContentType = "application/json";
myWebRequest.Method = "PUT";
myWebRequest.Proxy = WebRequest.GetSystemWebProxy();
ServicePointManager.CheckCertificateRevocationList = true;
using (StreamWriter myStreamWriter = new StreamWriter(myWebRequest.GetRequestStream()))
{
myStreamWriter.Write(_json);
}
using (HttpWebResponse myWebResponse = (HttpWebResponse)myWebRequest.GetResponse())
{
using (StreamReader myStreamReader = new StreamReader(myWebResponse.GetResponseStream()))
{
sOutput = myStreamReader.ReadToEnd();
sOutput = sOutput.Length == 0 ? myWebResponse.StatusDescription : sOutput;
ServicePoint currentServicePoint = myWebRequest.ServicePoint;
sOutput = currentServicePoint.GetHashCode().ToString();
currentServicePoint.ConnectionLimit = 5;
}
}
}
catch (Exception Ex)
{
sOutput = Ex.Message;
}
finally
{
callback?.Invoke(sOutput);
}
}
And here is how I launch the thread -
HTTPClass hTTPClass = new HTTPClass(cuURI, json, 5000, new MyCallback(ResultCallBack));
Thread t = new Thread(new ThreadStart(hTTPClass.SendRequest));
t.Start();
Here is the code after switching to HttpClient -
static HttpClient client = new HttpClient();
public async Task Write()
{
await WriteAsync(cuURI, json);
}
private async Task WriteAsync(Uri uri, string json)
{
StringContent content = new StringContent(json,Encoding.UTF8,"application/json");
await client.PutAsync(uri, content);
}
Here is the wireshark trace screenshot which shows a new connection for every request.
The client is setting the FIN flag on its own, and the server is not sending a FIN from its side. What is happening is that I see a lot of connections in the TIME_WAIT state on the server side.
I have written an API endpoint and created a simple .net core console app to send off multiple requests to the API endpoint simultaneously to test how the API endpoint works
The code looks as below
static async Task Main(string[] args)
{
// ARRANGE
int request_number = 200;
Task<HttpResponseMessage>[] tasks = new Task<HttpResponseMessage>[request_number];
Action[] actions = new Action[request_number];
for (int i = 0; i < request_number; i++)
{
int temp = i;
actions[temp] = () =>
{
tasks[temp] = CallMyAPI();
};
}
// ACT
Parallel.Invoke(actions);
await Task.WhenAll(tasks);
// ASSERT
string sample1 = await tasks[0].Content.ReadAsStringAsync();
for (int i = 1; i < request_number; i++)
{
string toBeTested = await tasks[i].Content.ReadAsStringAsync();
if (toBeTested != sample1)
{
Console.WriteLine("Wrong! i = " + i);
}
}
Console.WriteLine("finished");
Console.WriteLine("Press any key to complete...");
Console.Read();
}
static async Task<HttpResponseMessage> CallMyAPI()
{
var request = new HttpRequestMessage();
request.Method = HttpMethod.Post;
string contentString = "some json string as http body";
request.Content = new StringContent(contentString, System.Text.Encoding.UTF8, "application/json");
request.RequestUri = new Uri("http://myAPIendpoint.com");
HttpResponseMessage response;
using (HttpClient httpClient = new HttpClient())
{
response = await httpClient.SendAsync(request);
}
return response;
}
So basically what I have been trying to do by the code is to send off multiple requests once, and wait and record all the responses. Then I compare them to verify that they all return the same response.
Initially, when I set the variable request_number as small numbers, like 50, 100, the test app runs well. However, as the request_numbergoes up and reaches around 200, it starts to throw an exception that looks like:
Inner Exception 1:
IOException: Unable to read data from the transport connection: The I/O operation has been aborted because of either a thread exit or an application request.
Inner Exception 2:
SocketException: The I/O operation has been aborted because of either a thread exit or an application request
What is this kind of exception supposed to mean?
Your problem is this:
using (HttpClient httpClient = new HttpClient())
{..
}
Each of these uses a new socket.
Use a single httpclient, e.g. a static one
I have a simple console application which sends HTTP POST from multiple threads:
List<Task> tasks = new List<Task>();
for (int i = 0; i < 100; i++)
{
tasks.Add(Task.Factory.StartNew(() => SendQuery(url1, query1)));
tasks.Add(Task.Factory.StartNew(() => SendQuery(url2, query2)));
}
Task.WaitAll(tasks.ToArray());
SendQuery(string uri, string requestString) looks like this:
Uri url = new Uri(uri);
try
{
using (HttpClient client = new HttpClient { Timeout = new TimeSpan(0, 0, 10, 0) })
{
StringContent content = new StringContent(requestString);
content.Headers.ContentType = new MediaTypeHeaderValue("application/json");
HttpResponseMessage response = client.PostAsync(url, content).Result;
response.EnsureSuccessStatusCode();
}
}
catch (Exception ex)
{
Console.WriteLine(ex);
}
The program works without any errors, all the queries are processed finally, but after filling tasks list each thread hangs on client.PostAsync(url, content).Result, after several minutes IIS starts to process queries. Why does this delay occur? What's happening during this time? I am using IIS 7.5 running on Windows Server 2008 R2 to host web-services which provide url1 and url2.
Set this value to 500 or 1000 at the start of your program and let us know the effects.
Your requests maybe getting throttled at the default value of 2. (depending on the .net version)
ServicePointManager.DefaultConnectionLimit = 500;