Am I misusing the HttpClient class incorrectly. I am trying to test the HTTP status of images and it seems to not be executing at all. I have a list of a complex object so I want to run test on all image urls to see which urls are broken by doing this:
var client = new HttpClient();
var tasks = ObjectViewModel.Select(a => a.UserUrl).Select(url =>
client.GetAsync(url).ContinueWith(t =>
{
var response = t.Result;
if (!response.IsSuccessStatusCode)
url = "/Content/Images/MissingPic.png";
}));
I was originally doing it this in a foreach loop like so:
foreach(var Model in ObjectViewModel)
{
Model.UserUrl= Model.UserUrl.GetHttpRequest() ? Model.UserUrl:
"/Content/Images/MissingImage.png";
//Model.state= Model.state.ValidName();// this line is something seperate
//Model.property= Model.state.propertyCheck();// this line is something seperate
}
public static bool GetHttpRequest(this string s)
{
HttpWebRequest webRequest = (HttpWebRequest)WebRequest
.Create(s);
webRequest.AllowAutoRedirect = false;
HttpStatusCode responseStatusCode;
try
{
HttpWebResponse response = (HttpWebResponse)webRequest.GetResponse();
responseStatusCode = response.StatusCode;
}
catch (WebException we)
{
responseStatusCode = ((HttpWebResponse)we.Response).StatusCode;
}
if (responseStatusCode.ToString() == "OK")
return true;
else
return false;
}
which works perfectly fine, but takes about 5 to 7 seconds to complete all items since they are all running separately which is very long for a request to respond to the UI.
Consider using AsParallel() while iterating over your enumeration, should speed it up considerably.
var UrlToReponseMap = new Dictionary<string, bool>();
ObjectViewModel.AsParallel().ForAll(x =>
{
UrlToReponseMap[x.UserUrl] = x.UserUrl.GetHttpRequest();
});
Linq is (generally) lazy. This means that Linq statements only represent a query. Work happens when you enumerate the query (materialize).
You never materialize your tasks query. To get it to actually run the code in your select statement, you need to materialize the query by causing it to be enumerated.
One way would be to simply call tasks.ToList()
Related
So I had to create dozens of API requests and get json to make it an object and put it in a list.
I also wanted the requests to be parallel because I do not care about the order in which the objects enter the list.
public ConcurrentBag<myModel> GetlistOfDstAsync()
{
var req = new RequestGenerator();
var InitializedObjects = req.GetInitializedObjects();
var myList = new ConcurrentBag<myModel>();
Parallel.ForEach(InitializedObjects, async item =>
{
RestRequest request = new RestRequest("resource",Method.GET);
request.AddQueryParameter("key", item.valueOne);
request.AddQueryParameter("key", item.valueTwo);
var results = await GetAsync<myModel>(request);
myList.Add(results);
});
return myList;
}
What creates a new problem, I do not understand how to put them in the list and it seems I do not use a solution that exists in a form ConcurrentBag
Is my assumption correct and I implement it wrong or should I use another solution?
I also wanted the requests to be parallel
What you actually want is concurrent requests. Parallel does not work as expected with async.
To do asynchronous concurrency, you start each request but do not await the tasks yet. Then, you can await all the tasks together and get the responses using Task.WhenAll:
public async Task<myModel[]> GetlistOfDstAsync()
{
var req = new RequestGenerator();
var InitializedObjects = req.GetInitializedObjects();
var tasks = InitializedObject.Select(async item =>
{
RestRequest request = new RestRequest("resource",Method.GET);
request.AddQueryParameter("key", item.valueOne);
request.AddQueryParameter("key", item.valueTwo);
return await GetAsync<myModel>(request);
}).ToList();
var results = await TaskWhenAll(tasks);
return results;
}
I am looking for a better solution for the following case. Currently I am relying on the same order of the two lists in the following example.
In .NET Core/C# I am using a SDK that performs REST calls under the hood. The post method is async.
The code looks like that:
var requestTasks = new List<Task<SdkResponseType>>();
foreach (SdkRequest request in requests)
{
requestTasks.Add(sdkClient.PostAsync(request);
}
var responses = await Task.WhenAll(requestTasks);
After that I want to update the DB with the values of the response.
I know that the order of responses fits to the order of requests (described here).
Unfortunately, the response objects does not contain a value that gives me a hint to the entry request list.
It feels ugly to rely on the same order of responses and requests. Is there a better solution that does not contain patching the SDK?
I think, I am surely missing some code pattern.
Thanks in advance.
var taskList = emailOperatableCollection.MailjetQueue
.Select( async emailOperatable => {
var request = CreateMailJetRequest(emailOperatable,authUseraccount);
return new {
Request = request,
Result = await sdkClient.PostAsync(request)
};
} );
var responses = await Task.WhenAll(taskList);
foreach(var response in responses){
var originalRequest = response.Request;
var result = response.Result;
...
}
#Akash Kava: I think it's easer, not to discuss on my pseudo code above. This is method with the foreach with an async. The request is an object from the Mailjet SDK.
private async void SendEmailCollectionAsync(EmailOperatableCollection emailOperatableCollection, AuthUseraccount authUseraccount)
{
var mailjetClient = new MailjetClient(RuntimeEnvironment.BroadcastAdapterMailMailjetPublicKey, RuntimeEnvironment.BroadcastAdapterMailMailjetPrivateKey);
var mailjetRequestTasks = new List<Task<MailjetResponse>>();
foreach (EmailOperatable emailOperatable in emailOperatableCollection.MailjetQueue)
{
var request = CreateMailjetRequest(emailOperatable, authUseraccount);
mailjetRequestTasks.Add(mailjetClient.PostAsync(request));
}
var mailjetResponses = await Task.WhenAll(mailjetRequestTasks);
}
Now I want every response in the the array mailjetResponses to be assigned to every emailOperatable in the List emailOperatableCollection.MailjetQueue.
I have a method that attempts to download data from several URLs in Parallel, and return an IEnumerable of Deserialized types
The method looks like this:
public IEnumerable<TContent> DownloadContentFromUrls(IEnumerable<string> urls)
{
var list = new List<TContent>();
Parallel.ForEach(urls, url =>
{
lock (list)
{
_httpClient.GetAsync(url).ContinueWith(request =>
{
var response = request.Result;
//todo ensure success?
response.Content.ReadAsStringAsync().ContinueWith(text =>
{
var results = JObject.Parse(text.Result)
.ToObject<IEnumerable<TContent>>();
list.AddRange(results);
});
});
}
});
return list;
}
In my unit test (I stub out _httpClient to return a known set of text) I basically get
Sequence contains no elements
This is because the method is returning before the tasks have completed.
If I add .Wait() on the end of my .ContinueWith() calls, it passes, but I'm sure that I'm misusing the API here...
If you want a blocking call which downloads in parallel using the HttpClient.GetAsync method then you should implement it like so:
public IEnumerable<TContent> DownloadContentFromUrls<TContent>(IEnumerable<string> urls)
{
var queue = new ConcurrentQueue<TContent>();
using (var client = new HttpClient())
{
Task.WaitAll(urls.Select(url =>
{
return client.GetAsync(url).ContinueWith(response =>
{
var content = JsonConvert.DeserializeObject<IEnumerable<TContent>>(response.Result.Content.ReadAsStringAsync().Result);
foreach (var c in content)
queue.Enqueue(c);
});
}).ToArray());
}
return queue;
}
This creates an array of tasks, one for each Url, which represents a GetAsync/Deserialize operation. This is assuming that the Url returns a Json array of TContent. An empty array or a single member array will deserialize fine, but not a single array-less object.
I'm trying to find all the ftp servers accepting anonymous connections on a given set of ips.
Basically, I get the IPs I want to check, and then try a ListDirectory on each of them. If I have no exception, the ftp exists and is accessible.
I'm using an asynchronous method to verify an IP, which make things much faster. However, I then need to wait until all the async calls returned. To do this, I keep a counter on the number of async calls I have, the problem is this counter never gets to 0.
My code looks as follows:
to iterate over the IPs:
static int waitingOn;
public static IEnumerable<Uri> GetFtps()
{
var result = new LinkedList<Uri>();
waitingOn = 0;
IPNetwork ipn = IPNetwork.Parse("192.168.72.0/21");
IPAddressCollection ips = IPNetwork.ListIPAddress(ipn);
foreach( var ip in ips )
{
VerifyFtpAsync(ip, result);
}
while (waitingOn > 0)
{
Console.WriteLine(waitingOn);
System.Threading.Thread.Sleep(1000);
}
return result;
}
and to verify each IP:
public async static void VerifyFtpAsync( IPAddress ip, LinkedList<Uri> ftps )
{
++waitingOn;
try
{
Uri serverUri = new Uri("ftp://" + ip);
FtpWebRequest request = (FtpWebRequest)WebRequest.Create(serverUri);
request.Method = WebRequestMethods.Ftp.ListDirectoryDetails;
request.Timeout = 10000;
request.Credentials = new NetworkCredential("anonymous", "roim#search.com");
FtpWebResponse response = (FtpWebResponse) await request.GetResponseAsync();
// If we got this far, YAY!
ftps.AddLast(serverUri);
}
catch (WebException)
{
}
--waitingOn;
}
Replace
FtpWebResponse response = (FtpWebResponse) await request.GetResponseAsync();
with sync code
FtpWebResponse response = (FtpWebResponse) request.GetResponse();
FtpWebRequest.GetResponseAsync is not overriden for FTP specific and comes from the base WebRequest class, which doesn't seem to be able to handle this right.
First, you should never use async void unless you're writing an event handler.
Next, you do need to protect variables and collections against multithreaded access if your async methods may run in parallel (e.g., if this code is run in a console app). In your case, it sounds like you may want to use Task.WhenAll instead of a manual counter, and remove the shared collection.
public async static Task<Uri> VerifyFtpAsync(IPAddress ip)
{
try
{
...
return serverUri;
}
catch (WebException)
{
return null;
}
}
...
var ipTasks = ips.Select(ip => VerifyFtpAsync(ip));
var allResults = await Task.WhenAll(ipTasks);
var result = allResults.Where(url => url != null).ToArray();
Okay so basically I have a function that returns a string, but to get that string it uses webrequest which means while it's doing that webrequest the form is locking up unless I put it in a different thread.
But I can't figure out a way to capture the returned data in a thread since it's started using thread.start and that's a void.
Any help please?
Current code if it matters to anyone:
string CreateReqThread(string UrlReq)
{
System.Threading.Thread NewThread = new System.Threading.Thread(() => CreateReq(UrlReq));
string ReturnedData = "";
return ReturnedData;
}
string CreateReq(string url)
{
try
{
WebRequest SendReq = WebRequest.Create(url);
SendReq.Credentials = CredentialCache.DefaultCredentials;
SendReq.Proxy = WebRequest.DefaultWebProxy; //For closed port networks like colleges
SendReq.Proxy.Credentials = CredentialCache.DefaultCredentials;
SendReq.Timeout = 15000;
System.IO.StreamReader Reader = new System.IO.StreamReader(SendReq.GetResponse().GetResponseStream());
string Response = Reader.ReadToEnd();
Reader.Close();
return Response;
}
catch (WebException e)
{
EBox(e.Message, "Unknown Error While Connecting");
return null;
}
}
A common means of doing this is to use a Task<T> instead of a thread:
Task<string> CreateReqThread(string UrlReq)
{
return Task.Factory.StartNew() => CreateReq(UrlReq));
// In .NET 4.5, you can use (or better yet, reimplement using await/async directly)
// return Task.Run(() => CreateReq(UrlReq));
}
You can then call Task<T>.Result to get the returned value (later), when it's needed, or schedule a continuation on the task which will run when it completes.
This could look something like:
var request = CreateReqThread(theUri);
request.ContinueWith(t =>
{
// Shove results in a text box
this.textBox.Text = t.Result;
}, TaskScheduler.FromCurrentSynchronizationContext());
This also works perfectly with the new await/async support in C# 5.