We Keep Coding

c# set timeout for multipartReader.ReadNextSectionAsync()

I'm using C# .net core to read upload data from multipart post user sending multiple files.
How can I prevent use waiting infinite after read last file in
try
{
var cancellationTokenSource = new CancellationTokenSource();
cancellationTokenSource.CancelAfter(1000);
section = await multipartReader.ReadNextSectionAsync(cancellationTokenSource.Token);
}
catch (Exception ex)
{
throw;
}
Altough I've set cancelationToken for 1 second but it goes infinite, and won't go to
next line if I will send another request.
public static async Task<HttpRequest> FromHttpContextAsync(HttpContext httpContext)
{
bool multipart = false;
HttpRequest retVal = new HttpRequest(httpContext);
var sb = new StringBuilder();
var sr = new StreamReader(httpContext.Stream, Encoding.UTF8);
{
var line1 = await sr.ReadLineAsync();
sb.AppendLine(line1);
var Line1Parts = (line1).Split(' ');
retVal.Methode = Line1Parts[0].ToLower();
retVal.RawUrl = System.Net.WebUtility.UrlDecode(Line1Parts[1]).Replace("&", "&");
var urlPart = retVal.RawUrl.Split('?');
retVal.Url = urlPart[0];
if (urlPart.Length > 1)
{
foreach (var part in urlPart[1].Split(new[] { '&' }, StringSplitOptions.RemoveEmptyEntries))
{
var tmp = part.Split(new[] { '=' }, StringSplitOptions.RemoveEmptyEntries);
try
{
retVal.QueryStrings.Add(tmp[0], tmp[1]);
}
catch (Exception ex)
{
}
}
}
string line = await sr.ReadLineAsync();
sb.AppendLine(line);
int contentLength = 0;
while (!string.IsNullOrEmpty(line))
{
var tmp = line.Split(':');
var key = tmp[0].Trim().ToLower();
retVal.Header.Add(new KeyValuePair<string, string>(tmp[0], tmp[1]));
switch (key)
{
case "cookie":
{
foreach (var part in tmp[1].Split(new[] { ';' }, StringSplitOptions.RemoveEmptyEntries))
{
var pares = part.Split('=');
if (pares.Length == 2)
{
retVal.Cookies.Add(new KeyValuePair<string, string>(pares[0], pares[1]));
}
}
break;
}
case "content-length":
{
contentLength = int.Parse(tmp[1]);
break;
}
}
line = await sr.ReadLineAsync();
sb.AppendLine(line);
}
if (sb.ToString().Contains("Content-Type: multipart/form-data"))
{
string boundary = FindBoundary(sb.ToString());
MultipartReader multipartReader = new MultipartReader(boundary, httpContext.Stream);
var section = await multipartReader.ReadNextSectionAsync();
while (section != null)
{
// process each image
const int chunkSize = 1024;
var buffer = new byte[chunkSize];
var bytesRead = 0;
var fileName = GetFileName(section.ContentDisposition);
using (var stream = new MemoryStream())
{
do
{
try
{
bytesRead = await section.Body.ReadAsync(buffer, 0, buffer.Length);
}
catch (Exception ex)
{
Console.Write(ex);
}
stream.Write(buffer, 0, bytesRead);
} while (bytesRead > 0);
retVal.Files.Add(new Tuple<string, string, byte[]>("", fileName, stream.ToArray()));
}
try
{
var cancellationTokenSource = new CancellationTokenSource();
cancellationTokenSource.CancelAfter(1000);
section = await multipartReader.ReadNextSectionAsync(cancellationTokenSource.Token);
}
catch (Exception ex)
{
throw;
}
}
foreach (var file in retVal.Files)
{
File.WriteAllBytes("d:\\" + file.Item2, file.Item3);
}
}
HttpContext is inline class of this project and this is the source of HttpContext :
public class HttpContext
{
public HttpRequest Request { get; private set; }
public HttpResponse Response { get; private set; }
public Stream Stream { private set; get; }
private HttpContext(Stream networkStream)
{
Stream = networkStream;
}
public async static Task<HttpContext> FromHttpContextAsync(Stream networkStream)
{
var retVal = new HttpContext(networkStream);
retVal.Request = await HttpRequest.FromHttpContextAsync(retVal);
retVal.Response = HttpResponse.FromHttpContext(retVal);
return retVal;
}
}

While the lack of details and context makes trying to reproduce this issue really hard, I suspect the problem here is due to the fact NetworkStreams (used, under the covers, by your MultipartReader instance) do not yet fully support CancellationTokens. In fact, almost every Socket-related operation on .NET Core just checks for the eventually passed CancellationToken upfront - which is useless, in my opinion.
The good news is that the .NET Core team is actively working on this and I believe the issue will be completely solved in .NET Core 3.0:
https://github.com/dotnet/corefx/issues/24430
As a temporary ugly workaround, you can change your code to wait for both a fabricated delay task and your call to ReadNextSectionAsync(), assuming you don't want to re-used that stalled socket / NetworkStream after that:
var timeoutTask = Task.Delay(TimeSpan.FromSeconds(1)).ConfigureAwait(false);
var readNextSectionTask = multipartReader.ReadNextSectionAsync().ConfigureAwait(false);
if (await Task.WhenAny(timeoutTask, readNextSectionTask).ConfigureAwait(false) == timeoutTask)
{
// TODO: Handle the timeout
}
else
{
section = await readNextSectionTask;
}
Additionally, the fact you are not configuring your awaitable may act as a possible deadlock source (it is unclear to me whether you are running this code on ASP.NET Core itself or on some other synchronization context provider). To exclude this possibility, I would suggest to call ConfigureAwait(false) right after your await calls, as you can see on my previous code block.

System.Net.Http.HttpClient Timeout seems to be ignored

I am using Xamarin.iOS Version: 8.10.5.26 (Indie Edition) and facing a very strange behaviour with timing out requests sent with HttpClient():
The following code tries to get a result from an url, and has 60 seconds timeout (1 minutes), but when the request is fired it is taking around 90 seconds for time out. When call is made, I manually turn off the network connectivity to check the time out. It is observed that it is taking more than 60 seconds.
CODE
public async Task<Dictionary<string,object>> GetPatientDataASync (string lUsername)
{
var lDict = new Dictionary<string,object> ();
try {
string lQuerystring = "{Email: '" + lUsername + "'}";
String lUrl = String.Format (Constants.mURLPatient + "?where={0}", JObject.Parse (lQuerystring));
var lClient = new HttpClient ();
lClient.BaseAddress = new Uri (lUrl);
lClient.DefaultRequestHeaders
.Accept
.Add (new MediaTypeWithQualityHeaderValue ("application/json"));
lClient.DefaultRequestHeaders.Add ("X-Parse-Application-Id", Constants.mKeyParseAppId);
lClient.DefaultRequestHeaders.Add ("X-Parse-REST-API-Key", Constants.mKeyRestAPIKey);
lClient.Timeout = new TimeSpan (0, 1, 0);
var request = new HttpRequestMessage ();
request.Method = HttpMethod.Get;
if (Utility.isNetworkConnected ()) {
bool responseStatus = false;
await lClient.SendAsync (request)
.ContinueWith (responseTask => {
if (responseTask != null) {
var response = responseTask.Result;
if (response != null) {
if (response.IsSuccessStatusCode) {
var responseContent = response.Content;
if (responseContent != null) {
string responseString = responseContent.ReadAsStringAsync ().Result;
if (!string.IsNullOrWhiteSpace (responseString)) {
JObject json = JObject.Parse (responseString);
if (json != null) {
if (json ["results"].Any ()) {
Patient user = Patient.Instance;
user.objectId = json.SelectToken (#"results[0].objectId").Value<string> ();
user.Email = json.SelectToken (#"results[0].Email").Value<string> ();
user.Name = json.SelectToken (#"results[0].Name").Value<string> ();
user.IsNotificationsEnabled = json.SelectToken (#"results[0].IsNotificationsEnabled").Value<string> ();
Application.Current.Properties ["IsNotificationsEnabled"] = json.SelectToken (#"results[0].IsNotificationsEnabled").Value<string> ();
if (json.SelectToken (#"results[0].DeviceToken") != null) {
var deviceToken = json.SelectToken (#"results[0].DeviceToken").Value<JArray> ();
if (deviceToken != null)
user.DeviceToken = deviceToken.ToObject < List<string>> ();
} else {
user.DeviceToken = new List<string> ();
}
var doctors = json.SelectToken (#"results[0].MyDoctors").Value<JArray> ();
user.AllergicTo = json.SelectToken (#"results[0].AllergicTo").Value<string> ();
user.ContactNo = json.SelectToken (#"results[0].ContactNo").Value<string> ();
user.BloodGroup = json.SelectToken (#"results[0].BloodGroup").Value<string> ();
user.MyDoctors = doctors != null ? doctors.ToObject<List<string>> () : new List<string> ();
responseStatus = true;
} else
responseStatus = false;
}
}
}
}
}
}
});
lDict.Add (SUCCESS_CODE, responseStatus);
return lDict;
} else {
lDict.Add (NO_INTERNET, Constants.mStringNoInternetMessage);
return lDict;
}
} catch (Exception e) {
Debug.WriteLine (e.Message + "\n " + e.StackTrace);
lDict.Add (EXCEPTION_OCCURED, e);
return lDict;
}
}
If there's some mistake in my code , please do let me know.
Also the same issues are reported here :-
First link
Second Link

This is a know bug which was opened, closed and re-opened several times over the years. It is already reported here and here.

There is work-around solution:
Define CancellationTokenSource and Set Token to http request;
Invoke cancel by timeout on CancellationTokenSource like this cts.CancelAfter(timeout);:
Don't forgot catch exception, like this.
try
{
}
catch(TaskCanceledException)
{
if(!cts.Token.IsCancellationRequested)
{// timeout
}
else
{//other reason
}
}

C# httpclient hangs at the last request

I have a program which send a couple of thousands of post requests to the server. Everything is working fine until the end of the transmission. After all but one request are sent the program hangs.
Always there is only one request left to make. I am using HttpClient and everything is done async.
Here I create the httpclient:
ServicePointManager.ServerCertificateValidationCallback +=
(sender, cert, chain, sslPolicyErrors) => true;
ServicePointManager.DefaultConnectionLimit = 45;
ServicePointManager.Expect100Continue = false;
var CurrentCredentials = new System.Net.NetworkCredential(Repo.Username, Repo.Password);
var Handler = new HttpClientHandler
{
Credentials = CurrentCredentials,
ClientCertificateOptions = ClientCertificateOption.Automatic,
UseProxy = false,
};
var Client = new HttpClient(Handler);
Client.Timeout = TimeSpan.FromSeconds(2500);
Client.DefaultRequestHeaders.Add("Connection", "Keep-alive");
WebServ = new WebService(Client, ref Repo);
Here is where I send the requests:
private async Task<string> SendData(string Url, HttpContent[] content, string[] name)
{
using (var FormData = new MultipartFormDataContent())
{
for (int i = 0; i < content.Length; ++i)
{
if (name[i] == "file")
FormData.Add(content[i], name[i], name[i] + i.ToString() + ".jpg");
else
FormData.Add(content[i], name[i]);
}
try
{
using (var Response = await Client
.PostAsync(Url, FormData)
.ConfigureAwait(continueOnCapturedContext:false))
{
await App.Current.Dispatcher.BeginInvoke(
(Action)delegate
{
Repo.CurrentItem++;
});
using (var StreamResume = new StreamWriter("resume.bkp"))
{
await StreamResume.WriteLineAsync(Repo.CurrentItem.ToString());
}
if (Response.IsSuccessStatusCode)
{
await App.Current.Dispatcher.BeginInvoke(
(Action)delegate
{
Repo.Log.Add(Url + " OK" + Repo.CurrentItem);
});
}
else
{
await App.Current.Dispatcher.BeginInvoke(
(Action)delegate
{
Repo.Log.Add(Url + " Error "
+ Response.Content.ToString());
});
}
if (Repo.CurrentItem == Repo.NumberOfItems)
{
await App.Current.Dispatcher.BeginInvoke(
(Action)delegate
{
Repo.Log.Add("Data has been imported");
Repo.CurrentState = "Finished!";
Repo.CurrentItem = 0;
Repo.Uploading = false;
Repo.NotUploading = true;
Repo.Resumable = false;
File.Delete("resume.bkp");
});
}
}
return null;
}
catch (OperationCanceledException)
{
}
catch (InvalidOperationException)
{
App.Current.Dispatcher.Invoke(
(Action)delegate
{
Repo.Log.Add("The url is not valid");
Repo.CurrentItem = 0;
});
Client.CancelPendingRequests();
}
return null;
}
}
I encounter no errors whatsoever only that some threads never exit and the program never terminates. If I have a smaller data set, where only about 180 requests are made, the program does the job never hanging. Thanks in advance for your help

Downloading file parts using HttpWebRequest C#

I am trying to download a 100GB file using HttpWebRequest. The download will be split into parts depending on a preset part size. Below is the code I use to download the file:
private static void AddRangeHeaderHack(WebHeaderCollection headers, long start, long end)
{
// Original workaround by Eric Cadwell, code taken from
// https://connect.microsoft.com/VisualStudio/feedback/ViewFeedback.aspx?FeedbackID=93714
Type type = headers.GetType();
System.Reflection.MethodInfo setAddVerified = type.GetMethod("SetAddVerified",
System.Reflection.BindingFlags.NonPublic | System.Reflection.BindingFlags.Instance | System.Reflection.BindingFlags.FlattenHierarchy
);
string rangeHeaderValue = String.Format("bytes={0}-{1}", start, end);
if (setAddVerified != null)
setAddVerified.Invoke(headers, new object[] { "Range", rangeHeaderValue });
}
private ulong GetRemoteFileSize(string URI)
{
ulong size = 0;
HttpWebRequest req = null;
try
{
req = (HttpWebRequest)WebRequest.Create(URI);
using (var res = (HttpWebResponse)req.GetResponse())
{
size = (ulong)res.ContentLength;
res.Close();
}
}
catch (Exception ex)
{
}
if (req != null)
{
try
{
req.Abort();
req = null;
}
catch (Exception)
{
}
}
return size;
}
private int DownloadFromLink(string sSource, string sDestination)
{
int nRetryCount = 0;
int nMaxRetry = 5;
var lastProgress = DateTime.Now;
ulong offset = 0;
var bRetrying = false;
var bResumable = false;
var fileSize = GetRemoteFileSize(sSource);
if (fileSize > 0)
bResumable = true;
while (true)
{
HttpWebRequest webRequest = null;
try
{
try
{
bRetrying = false;
do
{
try
{
if (bDownloadAbort)
{
return -1;
}
webRequest = (HttpWebRequest)WebRequest.Create(sSource);
webRequest.Timeout = 3600000;
if (offset > 0)
{
AddRangeHeaderHack(webRequest.Headers, (long)offset, (long)fileSize);
}
// Retrieve the response from the server
using (var webResponse = (HttpWebResponse)webRequest.GetResponse())
{
var acceptRanges = String.Compare(webResponse.Headers["Accept-Ranges"], "bytes", true) == 0;
// Open the URL for download
using (var streamResponse = webResponse.GetResponseStream())
{
if (streamResponse != null)
{
// Create a new file stream where we will be saving the data (local drive)
using (var streamLocal = new FileStream(sDestination, offset>0?FileMode.Append:FileMode.Create, FileAccess.Write, FileShare.ReadWrite))
{
// It will store the current number of bytes we retrieved from the server
int bytesSize = 0;
// A buffer for storing and writing the data retrieved from the server
byte[] downBuffer = new byte[/*16384*/ 1024 * 1024];
bool binitialtry = true;
int nRetries = 0;
if (offset > 0)
{
streamLocal.Seek((long)offset, SeekOrigin.Begin);
}
// Loop through the buffer until the buffer is empty
while ((bytesSize = streamResponse.Read(downBuffer, 0, downBuffer.Length)) > 0
|| (File.Exists(sDestination) && (offset < (ulong)fileSize) && nRetries < 5 && bResumable))
{
if (binitialtry && bytesSize == 0)
{
binitialtry = false;
}
if (!binitialtry && bytesSize == 0)
{
nRetries++;
bRetrying = nRetries<5;
break;
}
if (bDownloadAbort)
{
try { streamLocal.Close(); }
catch { }
return;
}
try
{
// Write the data from the buffer to the local hard drive
streamLocal.Write(downBuffer, 0, bytesSize);
offset += (ulong)bytesSize;
}
catch (IOException ex)
{
if (streamResponse != null)
streamResponse.Close();
if (streamLocal != null)
streamLocal.Close();
if (webRequest != null)
webRequest.Abort();
return -1;
}
Interlocked.Add(ref actualDownloaded, bytesSize);
}
// When the above code has ended, close the streams
if (streamResponse != null)
streamResponse.Close();
if (streamLocal != null)
try { streamLocal.Close(); }
catch { }
if (webRequest != null)
webRequest.Abort();
if (webRequest != null)
wcDownload.Dispose();
streamLocal.Close();
}
streamResponse.Close();
}
}
webResponse.Close();
}
if(!bRetrying)
break;
}
catch (IOException ex)
{
if (webRequest != null)
webRequest.Abort();
if (wcDownload != null)
wcDownload.Dispose();
if (nRetryCount <= nMaxRetry)
{
Thread.Sleep(10000);
nRetryCount++;
bRetrying = true;
}
else
{
break;
}
}
catch (UnauthorizedAccessException ex)
{
if (webRequest != null)
webRequest.Abort();
break;
}
catch (WebException ex)
{
if (webRequest != null)
webRequest.Abort();
if (wcDownload != null)
wcDownload.Dispose();
if (nRetryCount <= nMaxRetry)
{
Thread.Sleep(10000);
nRetryCount++;
bRetrying = true;
}
else
{
break;
}
}
finally
{
}
} while (bRetrying);
}
catch (Exception ex)
{
break;
}
}
catch
{
break;
}
if(!bRetrying)
break;
}
}
If I try to download the file in 1 part, with out adding the range header, the code runs smoothly and the file downloads normally. When I add a range header, say from 10GB to 15GB or frankly any value, the code reaches streamResponse.Read and hangs there for several minutes then it throws an exception:
Unable to read data from the transport connection: An existing
connection was forcibly closed by the remote host
When the code retries the connection after the exception, the download resumes normally and the client is able to read data from the stream.
Can someone help me determine why such thing is happening?
Just to clear the matter about the server, the file is currently hosted on an Amazon S3 server, and the download is done from a generated direct link.

It could be a server setting, according to http://www.w3.org/Protocols/rfc2616/rfc2616-sec8.html#sec8.1.4
Clients that use persistent connections SHOULD limit the number of simultaneous connections that they maintain to a given server. A single-user client SHOULD NOT maintain more than 2 connections with any server or proxy. A proxy SHOULD use up to 2*N connections to another server or proxy, where N is the number of simultaneously active users. These guidelines are intended to improve HTTP response times and avoid congestion.
Try FDM, and see if it has a problem. http://www.freedownloadmanager.org/

I don’t know how to download a file in parts using HttpWebRequest, but I found this example online to build an own implementation. The article is about the HttpClient in C#. There is also complete code and project you can find in the download section of this page.
The Problem is that not all server support partial download. So NuGet packages can be used that handle the exceptions e.g.:
https://www.nuget.org/packages/downloader
or
https://www.nuget.org/packages/Shard.DonwloadLibrary.
These Libraries will handle the chunks and convert them back into a readable file or stream.
Downloader:
var downloader = new DownloadService(new()
{
ChunkCount = 8,
});
string file = #"Your_Path\fileName.zip";
string url = #"https://file-examples.com/fileName.zip";
await downloader.DownloadFileTaskAsync(url, file);
or Download Library:
string file = "Your_Path";
string url = "https://file-examples.com/fileName.zip";
var downloader = new LoadRequest(url,new()
{
Chunks = 8,
DestinationPath= file,
});
await downloader.Task;
I hope I could help!

Parallel.Foreach do not finish stream

For load testing purposes I need to simulate multiple users trying to login to a system at the same time. I have code written by another developer that can send a login request to the system. With an ok login it will also return other information in xml.
I've tried using Parallel.ForEach, but dont have any real experience with parallel programming:
Parallel.ForEach(clientList, client =>
{
RunTest(client);
});
public void RunTest(object data)
{
if (!(data is IGprsClient))
{
return;
}
_noRunningTests += 1;
IGprsClient gprsClient = data as IGprsClient;
DateTime startTime = DateTime.Now;
Log(gprsClient.Id, "Test started.");
bool result = gprsClient.StartTest(20000);
DateTime endTime = DateTime.Now;
TimeSpan diff = endTime - startTime;
if (result == false)
{
Log(gprsClient.Id, "Test failed.");
}
Log(gprsClient.Id, "Test took {0}ms. ", (int)diff.TotalMilliseconds);
_noRunningTests -= 1;
}
override public bool StartTest(int timeout)
{
_testStarted = true;
try
{
LogDebug("Trying to connect.");
_client = new TcpClient(ipAddress, port);
LogDebug("Connected.");
bool result = false;
//Todo: insert testcase into client
switch (TestCaseName)
{
case "LoginTEST":
var testCase = new LoginTEST(this);
result = testCase.Execute(user, pwd, phoneNum);
break;
default:
Log("Unknown test case: " + TestCaseName);
break;
}
_client.Close();
return result;
}
catch (Exception ex)
{
if (_client != null)
_client.Close();
Log(ex.Message);
return false;
}
}
Which in turn will send the request and read the response.
public bool Execute(string user, string pwd, string phoneNum)
{
SendDataListRequest(userId);
string requiredDataResponse = Client.ReadMsg();
return true;
}
Run test will send a request and reads the message like so:
public string ReadMsg()
{
int msgLength = -1;
var stream = _client.GetStream();
while (_testStarted)
{
int b = stream.ReadByte();
if (b == -1)
{
return "";
}
else if (b == 0x02 || msgLength == -1)
{
while (b != 0x02 && _testStarted)
{
b = stream.ReadByte(); // Finds the start token
if (b == -1)
{
return "";
}
}
msgLength = 0; // Starts collecting data
}
else if (b == 0x03)
{
byte[] encryptedMsg = Convert.FromBase64String(
Encoding.UTF8.GetString(byteBuffer, 0, msgLength));
byte[] decryptedMsg = SttCipher.DecryptMessage(encryptedMsg);
MemoryStream ms = new MemoryStream(decryptedMsg);
GZipStream gZipStream = new GZipStream(ms, CompressionMode.Decompress, true);
var bufLen = ReadAllBytesFromStream(gZipStream, decompressedBuffer);
gZipStream.Close();
string completeMsg = Encoding.UTF8.GetString(decompressedBuffer, 0, bufLen);
if (completeMsg.Length < 500)
LogDebug("Received XML-data:\n{0}", completeMsg);
else
LogDebug("Received XML-data: {0} bytes\n{1}...", completeMsg.Length, completeMsg.Substring(0, 500));
return completeMsg;
}
else
{
if (byteBuffer.Length <= msgLength)
{
throw new Exception("XML message too long!");
}
byteBuffer[msgLength] = (byte)b;
msgLength++;
}
}
return "";
}
Running one client is fine and will wait for the response. The issue is with several clients, the responses gets cut off. Leaving me with unclosed xml in the response.But I cant't figure out why. Does anyone have a reasonable explanation and/or a solution or a better way of doing it?