My HttpClient uses digest authentication to connect to the server and expects search queries in response. These search queries can come in any time so the client is expected to leave the connection open at all times.
The connection is made using the following code:
public static async void ListenForSearchQueries(int resourceId)
{
var url = $"xxx/yyy/{resourceId}/waitForSearchRequest?token=abc";
var httpHandler = new HttpClientHandler { PreAuthenticate = true };
using (var digestAuthMessageHandler = new DigestAuthMessageHandler(httpHandler, "user", "password"))
using (var client = new HttpClient(digestAuthMessageHandler))
{
client.Timeout = TimeSpan.FromMilliseconds(Timeout.Infinite);
var request = new HttpRequestMessage(HttpMethod.Get, url);
var tokenSource = new CancellationTokenSource();
tokenSource.CancelAfter(TimeSpan.FromMilliseconds(Timeout.Infinite));
using (var response = await client.SendAsync(request, HttpCompletionOption.ResponseHeadersRead, tokenSource.Token))
{
Console.WriteLine("\nResponse code: " + response.StatusCode);
using (var body = await response.Content.ReadAsStreamAsync())
using (var reader = new StreamReader(body))
while (!reader.EndOfStream)
Console.WriteLine(reader.ReadLine());
}
}
}
This is how the method is being used in the Main method of a console application.
private static void Main(string[] args)
{
const int serviceId = 128;
.
.
.
ListenForSearchQueries(resourceId);
Console.ReadKey();
}
This is what the output on the console window looks like:
Response code: OK
--searchRequestBoundary
Even though the timeout for the client is set to infinity, the connection times out after roughly five minutes (which is not the default timeout of the HttpClient) after the first output, throwing the following exception.
System.IO.IOException occurred
HResult=0x80131620
Message=The read operation failed, see inner exception.
Source=System.Net.Http
StackTrace:
at System.Net.Http.HttpClientHandler.WebExceptionWrapperStream.Read(Byte[] buffer, Int32 offset, Int32 count)
at System.Net.Http.DelegatingStream.Read(Byte[] buffer, Int32 offset, Int32 count)
at System.IO.StreamReader.ReadBuffer()
at System.IO.StreamReader.get_EndOfStream()
at ConsoleTester.Program.<ListenSearchQueriesDigestAuthMessageHandler>d__10.MoveNext() in C:\Users\xyz\ProjName\ConsoleTester\Program.cs:line 270
Inner Exception 1:
WebException: The operation has timed out.
The DelegateHandler used for the authentication is a a rough adaption of this code (see the source section).
Why is the client timing out and how can I prevent this?
My ultimate goal is to make a call and wait indefinitely for a response. When a response does come, I don't want the connection to close because more responses might come in the future. Unfortunately, I can't change anything at the server end.
Although the default value for Stream.CanTimeout is false, returning a stream via the response.Content.ReadAsStreamAsync() gives a stream where the CanTimeout property returns true.
The default read and write time out for this stream is 5 minutes. That is after five minutes of inactivity, the stream will throw an exception. Much similar to the exception shown in the question.
To change this behavior, ReadTimeout and/or the WriteTimeout property of the stream can be adjusted.
Below is the modified version of the ListenForSearchQueries method that changes the ReadTimeout to Infinite.
public static async void ListenForSearchQueries(int resourceId)
{
var url = $"xxx/yyy/{resourceId}/waitForSearchRequest?token=abc";
var httpHandler = new HttpClientHandler { PreAuthenticate = true };
using (var digestAuthMessageHandler = new DigestAuthMessageHandler(httpHandler, "user", "password"))
using (var client = new HttpClient(digestAuthMessageHandler))
{
client.Timeout = TimeSpan.FromMilliseconds(Timeout.Infinite);
var request = new HttpRequestMessage(HttpMethod.Get, url);
var tokenSource = new CancellationTokenSource();
tokenSource.CancelAfter(TimeSpan.FromMilliseconds(Timeout.Infinite));
using (var response = await client.SendAsync(request, HttpCompletionOption.ResponseHeadersRead, tokenSource.Token))
{
Console.WriteLine("\nResponse code: " + response.StatusCode);
using (var body = await response.Content.ReadAsStreamAsync())
{
body.ReadTimeout = Timeout.Infinite;
using (var reader = new StreamReader(body))
while (!reader.EndOfStream)
Console.WriteLine(reader.ReadLine());
}
}
}
}
This fixed the exception which was actually being thrown by the stream but seemed like was being thrown by the HttpClient.
Make the method return a Task
public static async Task ListenForSearchQueries(int resourceId) {
//...code removed for brevity
}
Update the console's main method to Wait on the Task to complete.
public static void Main(string[] args) {
const int serviceId = 128;
.
.
.
ListenForSearchQueries(resourceId).Wait();
Console.ReadKey();
}
I solved this problem in the following way:
var stream = await response.Content.ReadAsStreamAsync();
while (b == 1)
{
var bytes = new byte[1];
try
{
var bytesread = await stream.ReadAsync(bytes, 0, 1);
if (bytesread > 0)
{
text = Encoding.UTF8.GetString(bytes);
Console.WriteLine(text);
using (System.IO.StreamWriter escritor = new System.IO.StreamWriter(#"C:\orden\ConSegu.txt", true))
{
if (ctext == 100)
{
escritor.WriteLine(text);
ctext = 0;
}
escritor.Write(text);
}
}
}
catch (Exception ex)
{
Console.WriteLine("error");
Console.WriteLine(ex.Message);
}
}
in this way I get byte to byte the answer and I save it in a txt
later I read the txt and I'm erasing it again. for the moment it is the solution I found to receive the notifications sent to me by the server from the persistent HTTP connection.
Related
I've a .NET Core web app that, once a web method (i.e. Test()) is invoked, call another remote api.
Basically, I do it this way (here's a POST example, called within a web method Test()):
public T PostRead<T>(string baseAddress, string url, out bool succeded, object entity = null)
{
T returnValue = default(T);
succeded = false;
try
{
using (HttpClient client = new HttpClient())
{
client.DefaultRequestHeaders.Add("Authorization", MyApiKey);
HttpResponseMessage res = null;
string json = JsonConvert.SerializeObject(entity);
var body = new StringContent(json, UnicodeEncoding.UTF8, "application/json");
var task = Task.Run(() => client.PostAsync($"{baseAddress}/{url}", body));
task.Wait();
res = task.Result;
succeded = res.IsSuccessStatusCode;
if (succeded)
{
returnValue = JsonConvert.DeserializeObject<T>(res.Content.ReadAsStringAsync().Result);
}
else
{
Log($"PostRead failed, error: {JsonConvert.SerializeObject(res)}");
}
}
}
catch (Exception ex)
{
Log($"PostRead error: {baseAddress}/{url} entity: {JsonConvert.SerializeObject(entity)}");
}
return returnValue;
}
Question is: if Test() in my Web App is called 10 times, in parallel, do the POST requests to the remote server are be called in parallel or in serial?
Because if they will be called in serial, the last one will take the time of the previous ones, and that's not what I want.
In fact, when I have huge list of requests, I often receive the message A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.
Question is: if Test() in my Web App is called 10 times, in parallel, do the POST requests to the remote server are be called in parallel or in serial?
They will be called in parallel. Nothing in the code would make it in a blocking way.
In fact, when I have huge list of requests, I often receive the message A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.
There are at least 2 things that need to be fixed here:
Properly using async/await
Properly using HttpClient
The first one is easy:
public async Task<T> PostRead<T>(string baseAddress, string url, out bool succeded, object entity = null)
{
succeded = false;
try
{
using (HttpClient client = new HttpClient())
{
client.DefaultRequestHeaders.Add("Authorization", MyApiKey);
string json = JsonConvert.SerializeObject(entity);
var body = new StringContent(json, UnicodeEncoding.UTF8, "application/json");
var responseMessage = await client.PostAsync($"{baseAddress}/{url}", body);
var responseContent = await responseMessage.Content.ReadAsStringAsync();
if (responseMessage.IsSuccessStatusCode)
{
succeeded = true;
return JsonConvert.DeserializeObject<T>();
}
else
{
// log...
}
}
}
catch (Exception ex)
{
// log...
}
return default; // or default(T) depending on the c# version
}
The second one is a bit more tricky and can quite possibly be the cause of the problems you're seeing. Generally, unless you have some very specific scenario, new HttpClient() is plain wrong. It wastes resources, hides dependencies and prevents mocking. The correct way to do this is by using IHttpClientFactory, which can be very easily abstracted by using the AddHttpClient extension methods when registering the service.
Basically, this would look like this:
services.AddHttpClient<YourClassName>(x =>
{
x.BaseAddress = new Uri(baseAddress);
x.DefaultRequestHeaders.Add("Authorization", MyApiKey);
}
Then it's a matter of getting rid of all using HttpClient in the class and just:
private readonly HttpClient _client;
public MyClassName(HttpClient client) { _client = client; }
public async Task<T> PostRead<T>(string url, out bool succeded, object entity = null)
{
succeded = false;
try
{
string json = JsonConvert.SerializeObject(entity);
var responseMessage = await _client.PostAsync($"{baseAddress}/{url}", body);
//...
}
catch (Exception ex)
{
// log...
}
return default; // or default(T) depending on the c# version
}
[HttpClient call using parallel and solve JSON Parse error]
You can use task. when for parallel call . but when you use N number of calls and try to parse as JSON this may throw an error because result concatenation change the Json format so we can use Jarray. merge for combining the result .
Code
public async Task < string > (string baseAddress, string url, out bool succeded, object entity = null) {
using(HttpClient client = new HttpClient()) {
string responseContent = string.Empty;
client.DefaultRequestHeaders.Add("Accept", "application/json");
client.DefaultRequestHeaders.Add("Authorization", MyApiKey);
string json = JsonConvert.SerializeObject(entity);
var body = new StringContent(json, UnicodeEncoding.UTF8, "application/json");
var tasks = new List < Task < string >> ();
var jArrayResponse = new JArray();
int count = 10; // number of call required
for (int i = 0; i <= count; i++) {
async Task < string > RemoteCall() {
var response = await client.PostAsync($"{baseAddress}/{url}", body);
return await response.Content.ReadAsStringAsync();
}
tasks.Add(RemoteCall());
}
await Task.WhenAll(tasks);
foreach(var task in tasks) {
string postResponse = await task;
if (postResponse != "[]") {
var userJArray = JArray.Parse(postResponse);
jArrayResponse.Merge(userJArray);
}
}
responseContent = jArrayResponse.ToString();
return responseContent;
}
}
I 'd like to make a websocket client to connect to a third-party web socket server.
I am using ClientWebSocket Class:
public WSClientService(ClientWebSocket client, ILogger<WSClientService> logger)
{
_client = client;
_logger = logger;
}
For receiving messages i use this method:
public async Task GetMessagesAsync()
{
while (_client.State == WebSocketState.Open)
{
var chunkSize = 1024 * 4;
var buffer = new ArraySegment<byte>(new byte[chunkSize]);
do
{
WebSocketReceiveResult result;
using var ms = new MemoryStream();
try
{
do
{
//here throws exception
result = await _client.ReceiveAsync(buffer, CancellationToken.None);
ms.Write(buffer.Array, buffer.Offset, result.Count);
} while (!result.EndOfMessage);
if (result.MessageType == WebSocketMessageType.Close)
{
break;
}
ms.Seek(0, SeekOrigin.Begin);
using var reader = new StreamReader(ms, Encoding.UTF8);
var message = await reader.ReadToEndAsync();
_logger.LogInformation(message);
}
catch (Exception ex)
{
_logger.LogError(ex.Message);
}
} while (_client.State != WebSocketState.Aborted);
}
}
But for some messages i get this exception message :
The WebSocket received a frame with one or more reserved bits set
I noticed that this occurs probably by some compression because i can receive small messages. The exception thrown when calling this result = await _client.ReceiveAsync(buffer, CancellationToken.None);
Does anybody knows how to solve this problem?
The solution came with .net 6 release. Well the problem occured because there was no way to add compression in built-in ClientWebSocket in .net 5 and back (Except if someone would create an extension method for compression). Now with .net 6 there is a new option named DangerousDeflateOptions. This adds the corresponding header as well.
_ws.Options.DangerousDeflateOptions = new WebSocketDeflateOptions
{
ServerContextTakeover = false,
ClientMaxWindowBits = 15 // this is the default value
};
await _ws.ConnectAsync(new Uri(_url), token).ConfigureAwait(false);
I have written an API endpoint and created a simple .net core console app to send off multiple requests to the API endpoint simultaneously to test how the API endpoint works
The code looks as below
static async Task Main(string[] args)
{
// ARRANGE
int request_number = 200;
Task<HttpResponseMessage>[] tasks = new Task<HttpResponseMessage>[request_number];
Action[] actions = new Action[request_number];
for (int i = 0; i < request_number; i++)
{
int temp = i;
actions[temp] = () =>
{
tasks[temp] = CallMyAPI();
};
}
// ACT
Parallel.Invoke(actions);
await Task.WhenAll(tasks);
// ASSERT
string sample1 = await tasks[0].Content.ReadAsStringAsync();
for (int i = 1; i < request_number; i++)
{
string toBeTested = await tasks[i].Content.ReadAsStringAsync();
if (toBeTested != sample1)
{
Console.WriteLine("Wrong! i = " + i);
}
}
Console.WriteLine("finished");
Console.WriteLine("Press any key to complete...");
Console.Read();
}
static async Task<HttpResponseMessage> CallMyAPI()
{
var request = new HttpRequestMessage();
request.Method = HttpMethod.Post;
string contentString = "some json string as http body";
request.Content = new StringContent(contentString, System.Text.Encoding.UTF8, "application/json");
request.RequestUri = new Uri("http://myAPIendpoint.com");
HttpResponseMessage response;
using (HttpClient httpClient = new HttpClient())
{
response = await httpClient.SendAsync(request);
}
return response;
}
So basically what I have been trying to do by the code is to send off multiple requests once, and wait and record all the responses. Then I compare them to verify that they all return the same response.
Initially, when I set the variable request_number as small numbers, like 50, 100, the test app runs well. However, as the request_numbergoes up and reaches around 200, it starts to throw an exception that looks like:
Inner Exception 1:
IOException: Unable to read data from the transport connection: The I/O operation has been aborted because of either a thread exit or an application request.
Inner Exception 2:
SocketException: The I/O operation has been aborted because of either a thread exit or an application request
What is this kind of exception supposed to mean?
Your problem is this:
using (HttpClient httpClient = new HttpClient())
{..
}
Each of these uses a new socket.
Use a single httpclient, e.g. a static one
Hopefully, someone can enlighten me on the following.
I have a client that will do a request to a controller endpoint (there is no view, c# to c# or even C++ later). That controller will have to send responses as it fetches them asynchronously as json (sends json1 to client, then json2, then json3 until it closes the connection or send a null terminated text or similar). The purpose is to stream the results back to the client so it can start processing while the server still works.
My controller endpoint looks like this:
[HttpGet("testStream")]
public async Task testStream()
{
var response = HttpContext.Response;
response.Headers[HeaderNames.TransferEncoding] = "chunked";
for (var i = 0; i < 10; ++i)
{
await response.WriteAsync($"6\r\ntest {i}\r\n");
await response.Body.FlushAsync();
await Task.Delay(1 * 1000);
}
await response.WriteAsync("0\r\n\r\n");
await response.Body.FlushAsync();
}
My test looks like this:
static async void DownloadPageAsync()
{
// ... Target page.
string page = "http://localhost:8080/api/Stream/testStream";
Console.WriteLine("test");
while (!Debugger.IsAttached) Thread.Sleep(500);
// ... Use HttpClient.
using (HttpClient client = new HttpClient())
using (var response = await client.GetAsync(page, HttpCompletionOption.ResponseHeadersRead))
using (HttpContent content = response.Content)
{
string result = await content.ReadAsStringAsync();
do
{
Console.WriteLine(result);
result = await content.ReadAsStringAsync();
}
while (result != "null");
}
Console.WriteLine("END");
}
[Fact]
public void Test1()
{
TestSurvey.DownloadPageAsync();
}
I am getting exception when I call content.ReadAsStringAsync();
System.Net.Http.HttpRequestException : Error while copying content to a stream.
[xUnit.net 00:01:14.5836121] ---- System.IO.IOException : The read operation failed, see inner exception.
[xUnit.net 00:01:14.5836496] -------- System.Net.Http.CurlException : Failure when receiving data from the peer
[xUnit.net 00:01:14.5846837] Stack Trace:
[xUnit.net 00:01:14.5857807] at System.Net.Http.HttpContent.<LoadIntoBufferAsyncCore>d__48.MoveNext()
EDIT: Exception was due to not sending the size of the chunk
await response.WriteAsync($"6\r\ntest {i}\r\n");
but now on the test/client side, I get all the chunks at once...
To solve this, I made use of SSE or Server Side Events.
here is the server side in asp.net core:
[HttpGet("testStream")]
public async Task testStream()
{
var response = HttpContext.Response;
response.StatusCode = 200;
response.ContentType = "text/event-stream";
for (var i = 0; i < 10; ++i)
{
//the tags are either 'events:' or 'data:' and two \n indicates ends of the msg
//event: xyz \n\n
//data: xyz \n\n
await response.WriteAsync($"data: test {i}\n\n");
response.Body.Flush();
await Task.Delay(5 * 1000);
}
await response.WriteAsync("data:\n\n");
await response.Body.FlushAsync();
}
and here is the client side:
string page = "http://localhost:8080/api/Stream/testStream";
//while (!Debugger.IsAttached) Thread.Sleep(500);
using (HttpClient client = new HttpClient())
using (var s = await client.GetStreamAsync(page))
{
using (StreamReader r = new StreamReader(s))
{
string line = null;
while (null != (line = r.ReadLine()))
{
Console.WriteLine(line);
}
}
}
Usage of ReadAsStringAsync forced wait of all the message in order to proceed.
I am writing a class to handle file downloads and i am using this code [simplified]:
var webRequest = (HttpWebRequest)WebRequest.Create(downloadOperation.Link);
webRequest.Proxy = null;
using (var webResponse = await webRequest.GetResponseAsync())
{
using (var downloadStream = webResponse.GetResponseStream())
{
using (var outputFileWriteStream = await outputFile.OpenStreamForWriteAsync())
{
var buffer = new byte[4096];
var downloadedBytes = 0;
var totalBytes = webResponse.ContentLength;
while (downloadedBytes < totalBytes)
{
//*************************THIS LINE TAKES ABOUT 32 SECONDS TO EXECUTE ON FIRST INVOKE, ALL NEXT INVOKES TAKE ABOUT 120MS***************************
var currentRead = await downloadStream.ReadAsync(buffer, 0, buffer.Length);
//*******************************************************************************************************************************************************************
await outputFileWriteStream.WriteAsync(buffer, 0, currentRead);
}
}
}
}
Can you please explain to me why is it taking that long on first invoke and not on the next ones? I am worried that it is downloading the entire file on the first read.
Note that the files are usually between 3~15MB.
I am worried that it is downloading the entire file on the first read.
That's precisely what's happening. You can change that by setting webRequest.AllowReadStreamBuffering to false.
So i found a way to fix this problem, but it doesn't use WebRequest class.
I am now using the HttpClient found in (Windows.Web.Http).
Here is the fixed code:
var client = new Windows.Web.Http.HttpClient(); // prepare the http client
//get the response from the server
using (var webResponse = await client.GetAsync(downloadOperation.Link, HttpCompletionOption.ResponseHeadersRead)) //***********Node the HttpCompletionOption.ResponseHeaderRead, this means that the operation completes as soon as the client receives the http headers instead of waiting for the entire response content to be read
{
using (var downloadStream = (await webResponse.Content.ReadAsInputStreamAsync()).AsStreamForRead() )
{
using (var outputFileWriteStream = await outputFile.OpenStreamForWriteAsync())
{
var buffer = new byte[4096];
var downloadedBytes = 0;
var totalBytes = webResponse.ContentLength;
while (downloadedBytes < totalBytes)
{
//*************************THIS LINE NO LONGER TAKES A LONG TIME TO PERFORM FIRST READ***************************
var currentRead = await downloadStream.ReadAsync(buffer, 0, buffer.Length);
//*******************************************************************************************************************************************************************
await outputFileWriteStream.WriteAsync(buffer, 0, currentRead);
}
}
}
}
Hope this will help someone out there ;)
thank you all