We Keep Coding

GetResponseAsync stuck in Scheduled status

I'm using below code (this is slightly simplified) to make a webrequest:
public async Task<string> GetResponseAsync()
{
WebRequest webrequest = WebRequest.Create(url);
WebResponse response = null;
string content = string.Empty;
webrequest.Method = "GET";
webrequest.Timeout = 10000; // 10 seconds
response = await webrequest.GetResponseAsync();//this seems to not get started
using (Stream dataStream = response.GetResponseStream())
{
StreamReader reader = new StreamReader(dataStream);
content = await reader.ReadToEndAsync();
}
response?.Close();
return content;
}
This code has been working in production for months. Recently some changes have been made to the load balancer of the underlying service and now intermittently the line with GetResponseAsync gets stuck.
Below is a screenshot from the tasks debugging window. It will stay in this state for hours and the timeout does not work. The tasks window only shows tasks which are either "Awaiting" or "Scheduled". There is no task in any other state. Double clicking the task in red will go to line with GetResponseAsync method.
I feel like I might be missing something obvious here. What can be the reason of this getting stuck?

As per the link below, use ConfigureAwait to prevent deadlocks. Please read extensive doc on deadlocks due to async calls
https://blog.stephencleary.com/2012/07/dont-block-on-async-code.html
public static async Task<JObject> GetJsonAsync(Uri uri)
{
// (real-world code shouldn't use HttpClient in a using block; this is just example code)
using (var client = new HttpClient())
{
var jsonString = await client.GetStringAsync(uri).ConfigureAwait(false);
return JObject.Parse(jsonString);
}
}

It's likely that code outside of this method is placing restrictions on the ExecutionContext or SynchronizationContext your task needs to resume execution.

It turned out that the SSL handshake failed and that the timeout does not work in this case. The solution was to pass a CancellationToken with the timeout like this:
await webrequest.GetResponseAsync(new CancellationTokenSource(millisecondsDelay: 10000).Token)

C# Speed up parallel webrequests using async

so I have this code:
This is the main function,a parallel for loop that iterates through all the data that needs to be posted and calls a function
ParallelOptions pOpt = new ParallelOptions();
pOpt.MaxDegreeOfParallelism = 30;
Parallel.For(0, maxsize, pOpt, (index,loopstate) => {
//Calls the function where all the webrequests are made
CallRequests(data1,data2);
if (isAborted)
loopstate.Stop();
});
This function is called inside the parallel loop
public static void CallRequests(string data1, string data2)
{
var cookie = new CookieContainer();
var postData = Parameters[23] + data1 +
Parameters[24] + data2;
HttpWebRequest getRequest = (HttpWebRequest)WebRequest.Create(Parameters[25]);
getRequest.Accept = Parameters[26];
getRequest.KeepAlive = true;
getRequest.Referer = Parameters[27];
getRequest.CookieContainer = cookie;
getRequest.UserAgent = Parameters[28];
getRequest.Method = WebRequestMethods.Http.Post;
getRequest.AllowWriteStreamBuffering = true;
getRequest.ProtocolVersion = HttpVersion.Version10;
getRequest.AllowAutoRedirect = false;
getRequest.ContentType = Parameters[29];
getRequest.ReadWriteTimeout = 5000;
getRequest.Timeout = 5000;
getRequest.Proxy = null;
byte[] byteArray = Encoding.ASCII.GetBytes(postData);
getRequest.ContentLength = byteArray.Length;
Stream newStream = getRequest.GetRequestStream(); //open connection
newStream.Write(byteArray, 0, byteArray.Length); // Send the data.
newStream.Close();
HttpWebResponse getResponse = (HttpWebResponse)getRequest.GetResponse();
if (getResponse.Headers["Location"] == Parameters[30])
{
//These are simple get requests to retrieve the source code using the same format as above.
//I need to preserve the cookie
GetRequets(data1, data2, Parameters[31], Parameters[13], cookie);
GetRequets(data1, data2, Parameters[32], Parameters[15], cookie);
}
}
From what I have seen and been told,I understand that making these requests async is a better idea than using a parallel loop.My method is also heavy on the proccesor.I wonder how can I make these requests async,but also preserve the multithreaded aspect. I also need to keep the cookie,after the post requests finishes.

Converting the CallRequests method to an async is really just a case of switching the sync method calls for async ones with the await keyword and changing the method signature to return Task.
Something like this:
public static async Task CallRequestsAsync(string data1, string data2)
{
var cookie = new CookieContainer();
var postData = Parameters[23] + data1 +
Parameters[24] + data2;
HttpWebRequest getRequest = (HttpWebRequest)WebRequest.Create(Parameters[25]);
getRequest.Accept = Parameters[26];
getRequest.KeepAlive = true;
getRequest.Referer = Parameters[27];
getRequest.CookieContainer = cookie;
getRequest.UserAgent = Parameters[28];
getRequest.Method = WebRequestMethods.Http.Post;
getRequest.AllowWriteStreamBuffering = true;
getRequest.ProtocolVersion = HttpVersion.Version10;
getRequest.AllowAutoRedirect = false;
getRequest.ContentType = Parameters[29];
getRequest.ReadWriteTimeout = 5000;
getRequest.Timeout = 5000;
getRequest.Proxy = null;
byte[] byteArray = Encoding.ASCII.GetBytes(postData);
getRequest.ContentLength = byteArray.Length;
Stream newStream =await getRequest.GetRequestStreamAsync(); //open connection
await newStream.WriteAsync(byteArray, 0, byteArray.Length); // Send the data.
newStream.Close();
HttpWebResponse getResponse = (HttpWebResponse)getRequest.GetResponse();
if (getResponse.Headers["Location"] == Parameters[30])
{
//These are simple get requests to retrieve the source code using the same format as above.
//I need to preserve the cookie
GetRequets(data1, data2, Parameters[31], Parameters[13], cookie);
GetRequets(data1, data2, Parameters[32], Parameters[15], cookie);
}
}
However this, in itself, doesn't really get you anywhere because you still need to await the returned tasks in your main method. A very straightforward (if somewhat blunt) way of doing so would be to simply call Task.WaitAll() (or await Task.WhenAll() if the calling method itself is to become async). Something like this:
var tasks = Enumerable.Range(0, maxsize).Select(index => CallRequestsAsync(data1, data2));
Task.WaitAll(tasks.ToArray());
However, this is really pretty blunt and loses control over how many iterations are running in parallel, etc. I MUCH prefer use of the TPL dataflow library for this sort of thing. This library provides a way of chaining async (or sync for that matter) operations in parallel and passing them from one "processing block" to the next. It has a myriad of options for tweaking degrees of parallelism, buffer sizes, etc.
A detailed expose is beyond the possible scope of this answer so i'd encourage you to read up on it but one possible approach would be to simply push this to an action block - something like this:
var actionBlock = new ActionBlock<int>(async index =>
{
await CallRequestsAsync(data1, data2);
}, new ExecutionDataflowBlockOptions
{
MaxDegreeOfParallelism = 30,
BoundedCapacity = 100,
});
for (int i=0; i <= maxsize; i++)
{
actionBlock.Post(i); // or await actionBlock.SendAsync(i) if calling method is also async
}
actionBlock.Complete();
actionBlock.Completion.Wait(); // or await actionBlock.Completion if calling method is also async
Couple of additional points that are outside the scope of my answer that I should mention in passing:
it looks like your CallRequests method is updating some external variable with its results. Where possible it's best to avoid this pattern and have the method return the results for collation later (which the TPL Dataflow library handles through TransformBlock<>). If updating external state is unavoidable then make sure you have thought about the multithreaded implications (deadlocks, race conditions, etc.) which are outside the scope of my answer.
I am assuming there is some useful property of index which has been lost when you created a minimal description for your question? Does it index into a parameter list or something similar? If so, you can always just iterate over these directly and change the ActionBlock<int> to an ActionBlock<{--whatever the type of your parameter is--}>
Make sure you understand the difference between multi-threaded/parallel execution and asynchronous. There are some similarities/overlaps for sure but just making something async doesn't make it multithreaded nor is the converse true.

HttpListener Called twice

I am using this code to implement Http Server:
public Server()
{
_httpListener = new HttpListener();
_httpListener.Prefixes.Add(Server.UriAddress);
StartServer();
}
public void StartServer()
{
_httpListener.Start();
while (_httpListener.IsListening)
ProcessRequest();
}
void ProcessRequest()
{
var result = _httpListener.BeginGetContext(ListenerCallback, _httpListener);
result.AsyncWaitHandle.WaitOne();
}
void ListenerCallback(IAsyncResult result)
{
HttpListenerContext context = _httpListener.EndGetContext(result);
HttpListenerRequest request = context.Request;
string url = request.RawUrl;
url = url.Substring(1, url.Length - 1);
HttpListenerResponse response = context.Response;
string responseString = url;
byte[] buffer = System.Text.Encoding.UTF8.GetBytes(responseString);
response.ContentLength64 = buffer.Length;
System.IO.Stream output = response.OutputStream;
output.Write(buffer, 0, buffer.Length);
output.Close();
}
And i have a problem that if i wrote this in the browser(It's an example and it's occur on every call):
http://localhost:8888/Hello%20World
the ListenerCallback method is called twice,any idea how to fix it?

An answer has been accepted already, but I think it may still be useful to other people subsequently.
By default, most browsers given a URL would make at least two calls. One call to the request URL and the other to favicon.ico.
So a check should be made in ListenerCallback like
HttpListenerContext context = _httpListener.EndGetContext(result);
HttpListenerRequest request = context.Request;
string url = request.RawUrl;
url = url.Substring(1);
if (url == "favicon.ico")
{
return;
}
//normal request handling code
I hope this helps someone.

If your website requires several calls to the server, it will be called several times. This happens when you hav pictures or anything else on you page.
Try to call the synchronous method _httpListener.GetContext() or synchronize your calls with a lock or Mutex.

I am not sure but I think I see what might be a problem with your code.
You are mixing two patterns of async handling
You release the main thread by waiting on the waithandle in the async result.
But I think that only signals that you can call endgetcontext, not that another thread can use the listener.
If you are using the callback pattern, you should release the main thread using a wait handle other than that provided in the asyncresult

ManualResetEvent.set not getting fired in C#

I am working on Visual Web Express 2010.
I am trying to upload a file to server and block the calling function and release it once uploading is complete. However, the main thread never gets unblocked.
public partial class MainPage : UserControl
{
private FileStream fileStream;
private static String responseStr = "";
private static ManualResetEvent evt = new ManualResetEvent(false);
public MainPage()
{
InitializeComponent();
HtmlPage.RegisterScriptableObject("App", this);
}
public void sendPhoto()
{
uploadFile();
}
private void uploadFile()
{
uploadDataToServer(url);
evt.WaitOne();
postProcess();
}
public static void postProcess()
{
HtmlPage.Window.Invoke("postProcess2", responseStr);
}
void uploadDataToServer(String url)
{
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
request.ContentType = "multipart/form-data; boundary=---------------------------" + _boundaryNo;
request.Method = "POST";
request.BeginGetRequestStream(writeCallback, request);
}
private void writeCallback(IAsyncResult asynchronousResult)
{
HttpWebRequest request = (HttpWebRequest)asynchronousResult.AsyncState;
MemoryStream memoryStream = new MemoryStream();
fileStream.CopyTo(memoryStream);
if (memoryStream != null)
{
memoryStream.Position = 0;
byte[] img = memoryStream.ToArray();
Stream postStream = request.EndGetRequestStream(asynchronousResult);
postStream.Write(img, 0, img.Length);
request.BeginGetResponse(new AsyncCallback(GetResponseCallback), request);
}
}
private void GetResponseCallback(IAsyncResult asynchronousResult)
{
HttpWebRequest request = (HttpWebRequest)asynchronousResult.AsyncState;
HttpWebResponse response = (HttpWebResponse)request.EndGetResponse(asynchronousResult);
Stream streamResponse = response.GetResponseStream();
StreamReader streamRead = new StreamReader(streamResponse);
string responseString = streamRead.ReadToEnd();
streamRead.Close();
streamResponse.Close();
response.Close();
responseStr = responseString;
evt.Set();
}
}
Now, when I use evt.WaitOne() in uploadFile, the whole app goes waiting and no request is send to server, i.e. the code never reaches getResponseCallBack and hence the application never wakes up.
However, if I don't use evt.WaitOne(), then the request is successful, however I can't read the response text since it's set in writeCallBack() function and the request is async one. What should I do to get over this problem?
I can't figure out:
1. If the request is multi-threaded / Async, then why evt.WaitOne() makes the complete app waiting and the request doesn't complete?
2. If the request is single threaded, then why postProcess() [removing the evt.WaitOne()] on trying to access responseStr [set in the getResponseCallBack()] doesn't get the proper response set in it.
[Sorry, I am new to this and am confused].
Thanks.
Sorry, I forgot to mention one thing that, I am using silverlight.

There is no point in starting an async operation and then waiting for it to complete. Instead of async request.BeginGetRequestStream(writeCallback, request); use the synchronous request.GetRequestStream(...) and consume it's result. ManualResetEvent would not be needed then.

Here is the solution to the problem that I figured out:
Some have advised to use HttpWebRequest.GetRequestStream and make the call synchronous. However, as it seems that silverlight doesn't allow this to take place. Moreover, there was many confusion related to threading and parrallel-processing.
The main problem was that I wasn't able to access the UI thread from my callback function, so I used the dispatcher method and since it always makes sure that the actions are performed in UI thread, my objective got fulfilled. The following did the trick.
System.Windows.Deployment.Current.Dispatcher.BeginInvoke(() => { postProcess(); });

Async FTP call never ending

I'm trying to find all the ftp servers accepting anonymous connections on a given set of ips.
Basically, I get the IPs I want to check, and then try a ListDirectory on each of them. If I have no exception, the ftp exists and is accessible.
I'm using an asynchronous method to verify an IP, which make things much faster. However, I then need to wait until all the async calls returned. To do this, I keep a counter on the number of async calls I have, the problem is this counter never gets to 0.
My code looks as follows:
to iterate over the IPs:
static int waitingOn;
public static IEnumerable<Uri> GetFtps()
{
var result = new LinkedList<Uri>();
waitingOn = 0;
IPNetwork ipn = IPNetwork.Parse("192.168.72.0/21");
IPAddressCollection ips = IPNetwork.ListIPAddress(ipn);
foreach( var ip in ips )
{
VerifyFtpAsync(ip, result);
}
while (waitingOn > 0)
{
Console.WriteLine(waitingOn);
System.Threading.Thread.Sleep(1000);
}
return result;
}
and to verify each IP:
public async static void VerifyFtpAsync( IPAddress ip, LinkedList<Uri> ftps )
{
++waitingOn;
try
{
Uri serverUri = new Uri("ftp://" + ip);
FtpWebRequest request = (FtpWebRequest)WebRequest.Create(serverUri);
request.Method = WebRequestMethods.Ftp.ListDirectoryDetails;
request.Timeout = 10000;
request.Credentials = new NetworkCredential("anonymous", "roim#search.com");
FtpWebResponse response = (FtpWebResponse) await request.GetResponseAsync();
// If we got this far, YAY!
ftps.AddLast(serverUri);
}
catch (WebException)
{
}
--waitingOn;
}

Replace
FtpWebResponse response = (FtpWebResponse) await request.GetResponseAsync();
with sync code
FtpWebResponse response = (FtpWebResponse) request.GetResponse();
FtpWebRequest.GetResponseAsync is not overriden for FTP specific and comes from the base WebRequest class, which doesn't seem to be able to handle this right.

First, you should never use async void unless you're writing an event handler.
Next, you do need to protect variables and collections against multithreaded access if your async methods may run in parallel (e.g., if this code is run in a console app). In your case, it sounds like you may want to use Task.WhenAll instead of a manual counter, and remove the shared collection.
public async static Task<Uri> VerifyFtpAsync(IPAddress ip)
{
try
{
...
return serverUri;
}
catch (WebException)
{
return null;
}
}
...
var ipTasks = ips.Select(ip => VerifyFtpAsync(ip));
var allResults = await Task.WhenAll(ipTasks);
var result = allResults.Where(url => url != null).ToArray();