I'd like to ask about how to wait for multiple async http requests.
My code is like this :
public void Convert(XDocument input, out XDocument output)
{
var ns = input.Root.Name.Namespace;
foreach (var element in input.Root.Descendants(ns + "a"))
{
Uri uri = new Uri((string)element.Attribute("href"));
var wc = new WebClient();
wc.OpenReadCompleted += ((sender, e) =>
{
element.Attribute("href").Value = e.Result.ToString();
}
);
wc.OpenReadAsync(uri);
}
//I'd like to wait here until above async requests are all completed.
output = input;
}
Dose anyone know a solution for this?
There is an article by Scott Hanselman in which he describes how to do non blocking requests. Scrolling to the end of it, there is a public Task<bool> ValidateUrlAsync(string url) method.
You could modify it like this (could be more robust about response reading)
public Task<string> GetAsync(string url)
{
var tcs = new TaskCompletionSource<string>();
var request = (HttpWebRequest)WebRequest.Create(url);
try
{
request.BeginGetResponse(iar =>
{
HttpWebResponse response = null;
try
{
response = (HttpWebResponse)request.EndGetResponse(iar);
using(var reader = new StreamReader(response.GetResponseStream()))
{
tcs.SetResult(reader.ReadToEnd());
}
}
catch(Exception exc) { tcs.SetException(exc); }
finally { if (response != null) response.Close(); }
}, null);
}
catch(Exception exc) { tcs.SetException(exc); }
return tsc.Task;
}
So with this in hand, you could then use it like this
var urls=new[]{"url1","url2"};
var tasks = urls.Select(GetAsync).ToArray();
var completed = Task.Factory.ContinueWhenAll(tasks,
completedTasks =>{
foreach(var result in completedTasks.Select(t=>t.Result))
{
Console.WriteLine(result);
}
});
completed.Wait();
//anything that follows gets executed after all urls have finished downloading
Hope this puts you in the right direction.
PS. this is probably as clear as it can get without using async/await
Consider using continuation passing style. If you can restructure your Convert method like this,
public void ConvertAndContinueWith(XDocument input, Action<XDocument> continueWith)
{
var ns = input.Root.Name.Namespace;
var elements = input.Root.Descendants(ns + "a");
int incompleteCount = input.Root.Descendants(ns + "a").Count;
foreach (var element in elements)
{
Uri uri = new Uri((string)element.Attribute("href"));
var wc = new WebClient();
wc.OpenReadCompleted += ((sender, e) =>
{
element.Attribute("href").Value = e.Result.ToString();
if (interlocked.Decrement(ref incompleteCount) == 0)
// This is the final callback, so we can continue executing.
continueWith(input);
}
);
wc.OpenReadAsync(uri);
}
}
You then run that code like this:
XDocument doc = something;
ConvertAndContinueWith(doc, (finishedDocument) => {
// send the completed document to the web client, or whatever you need to do
});
Related
Well, I'm building web parsing app and having some troubles making it async.
I have a method which creates async tasks, and decorator for RestSharp so I can do requests via proxy. Basically in code it just does 5 tries of requesting the webpage.
Task returns RestResponse and it's status code is always 0. And this is the problem, because if I do the same synchronously, it works.
private static async Task<HtmlNode> GetTableAsync(int page)
{
ProxyClient client = new ProxyClient((name) =>ProxyProvider.GetCoreNoCD(name),
serviceName, 10000, 10000);
var task = client.TryGetAsync(new Uri(GetPageUrl(page)), (res) =>
{
return res.IsSuccessStatusCode && res.IsSuccessful;
},5);
HtmlDocument doc = new HtmlDocument();
doc.LoadHtml((await task).Content);
return doc.DocumentNode.SelectSingleNode("//div[#class=\"table_block\"]/table");
}
And this works as expected, but synchronously.
private static async Task<HtmlNode> GetTableAsync(int page)
{
ProxyClient client = new ProxyClient((name) =>ProxyProvider.GetCoreNoCD(name),
serviceName, 10000, 10000);
var task = client.TryGetAsync(new Uri(GetPageUrl(page)), (res) =>
{
return res.IsSuccessStatusCode && res.IsSuccessful;
},5);
task.Wait();
HtmlDocument doc = new HtmlDocument();
doc.LoadHtml(task.Result.Content);
return doc.DocumentNode.SelectSingleNode("//div[#class=\"table_block\"]/table");
}
ProxyClient's insides:
public async Task<RestResponse?> TryGetAsync(Uri uri,
Predicate<RestResponse> condition, int tryCount = 15,
List<KeyValuePair<string, string>> query = null,
List<KeyValuePair<string, string>> headers = null,
Method method = Method.Get, string body = null)
{
WebClient? client = null;
RestResponse? res = null;
for(int i = 0; i < tryCount; i++)
{
try
{
client = new WebClient(source.Invoke(serviceName), serviceName, timeout);
res = await client.GetResponseAsync(uri, query, headers, method, body);
if (condition(res))
return res;
}
catch(Exception)
{
///TODO:add log maybe?
}
finally
{
if (client != null)
{
client.SetCDToProxy(new TimeSpan(cd));
client.Dispose();
}
}
}
return res;
}
I have no idea how to make it work with async and don't understand why it doesn't work as expected.
I think it might have to do with the Task.Wait() I would consider changing to await like this.
private static async Task<HtmlNode> GetTableAsync(int page)
{
ProxyClient client = new ProxyClient((name) =>ProxyProvider.GetCoreNoCD(name),
serviceName, 10000, 10000);
var statusOk = false;
var result = await client.GetAsync(new Uri(GetPageUrl(page));
statusOk = result.IsSuccessStatusCode &&
result.StatusCode == HttpStatusCode.OK;
//do what you want based on statusOk
HtmlDocument doc = new HtmlDocument();
doc.LoadHtml(result.Content);
return doc.DocumentNode.SelectSingleNode("//div[#class=\"table_block\"]/table");
}
Just decided to try different solutions, and seems like it works only if I return task result
Like this:
ProxyClient client = new ProxyClient((name) => ProxyProvider.GetCoreNoCD(name),
serviceName, 10000, 10000);
return await client.TryGetAsync(new Uri(GetPageUrl(page)), (res) =>
{ return res.IsSuccessStatusCode && res.IsSuccessful; });
I thought it could be some kind of misunderstanding of async/await, but seems like no. Maybe some kind of RestSharp bug.
I think you're just checking the result too early. You need to look at the result after the await:
var task = client.TryGetAsync(...);
// Too early to check
var x = await task;
// Check now
HtmlDocument doc = new HtmlDocument();
doc.LoadHtml(x.Content);
I am trying to call HttpClient request inside for loop as follows. It needs to do multiple consecutive calls to third party rest api.
But it only gives me fist service call result while loop exit before getting result from rest of the service call.
private void Search()
{
try
{
var i = 1;
using (var httpClient = new HttpClient())
{
while (i < 5)
{
string url = "https://jsonplaceholder.typicode.com/posts/" + i;
var response = httpClient.GetAsync(url).Result;
string jsonResult = response.Content.ReadAsStringAsync().Result;
Console.WriteLine(jsonResult.ToString());
i++;
}
}
}
catch (Exception ex)
{
Console.WriteLine(ex.ToString());
}
}
When I run with debug points the program gives me all the result. But when I run it without debug points it gives me only the first result.
I tried this with using async, await methods too. It also gives me same result.
As I feel Program needs to wait until the async call returns data.
Please help me to solve this.
EDIT - async way
private async Task<string> SearchNew()
{
try
{
var i = 1;
var res = string.Empty;
using (var httpClient = new HttpClient())
{
while (i < 5)
{
string url = "https://jsonplaceholder.typicode.com/posts/" + i;
var response = httpClient.GetAsync(url).Result;
string jsonResult = await response.Content.ReadAsStringAsync();
res = res + jsonResult + " --- ";
i++;
}
}
return res;
}
catch (Exception ex)
{
return ex.Message;
}
}
Both are giving same result.
There's a few things here that you should be doing. First, move the HttpClient creation outside of your method and make it static. You only need one of them and having multiple can be really bad for stability (see here):
private static HttpClient _client = new HttpClient();
Next, extract the calls to the HttpClient into a single method, something simple like this:
//Please choose a better name than this
private async Task<string> GetData(string url)
{
var response = await _client.GetAsync(url);
return await response.Content.ReadAsStringAsync();
}
And finally, you create a list of tasks and wait for them all to complete asynchronously using Task.WhenAll:
private async Task<string[]> SearchAsync()
{
var i = 1;
var tasks = new List<Task<string>>();
//Create the tasks
while (i < 5)
{
string url = "https://jsonplaceholder.typicode.com/posts/" + i;
tasks.Add(GetData(url));
i++;
}
//Wait for the tasks to complete and return
return await Task.WhenAll(tasks);
}
And to call this method:
var results = await SearchAsync();
foreach (var result in results)
{
Console.WriteLine(result);
}
I need to save downloaded video to gallery on iPhone, but getting error:
The operation couldnt be completed. (Cocoa error -1/)
Tried also to do this through webClient.DownloadDataAsync(), getting same error. Here is my listing:
public async Task<string> DownloadFile(string fileUri)
{
var tcs = new TaskCompletionSource<string>();
string fileName = fileUri.Split('/').Last();
var documentsDirectory = Environment.GetFolderPath(Environment.SpecialFolder.Personal);
string videoFileName = System.IO.Path.Combine(documentsDirectory, fileName);
var webClient = new WebClient();
webClient.DownloadFileCompleted += (s, e) =>
{
var authStatus = await PHPhotoLibrary.RequestAuthorizationAsync();
if(authStatus == PHAuthorizationStatus.Authorized){
var fetchOptions = new PHFetchOptions();
var collections = PHAssetCollection.FetchAssetCollections(PHAssetCollectionType.Album, PHAssetCollectionSubtype.Any, fetchOptions);
collection = collections.firstObject as PHAssetCollection;
PHPhotoLibrary.SharedPhotoLibrary.PerformChanges(() => {
var assetCreationRequest = PHAssetChangeRequest.FromVideo(NSUrl.FromFileName(videoFileName));
var assetPlaceholder = assetCreationRequest.PlaceholderForCreatedAsset;
var albumChangeRequest = PHAssetCollectionChangeRequest.ChangeRequest(collection);
albumChangeRequest.AddAssets(new PHObject[] { assetPlaceholder });
}, delegate (bool status, NSError error) {
if (status)
{
Console.Write("Video added");
tcs.SetResult("success");
}
});
}
try
{
webClient.DownloadFileAsync(new Uri(fileUri), videoFileName);
}
catch (Exception e)
{
tcs.SetException(e);
}
return await tcs.Task;
}
Any help would be appreciated. Thanks.
(Cocoa error -1/)
Are you sure that you actually have valid data/mp4 from your download?
Are you using SSL (https), otherwise have you applied for an ATS exception in your info.plist?
Check the phone/simulator console output for errors concerning ATS
Note: I typically use NSUrlSession directly to avoid the HttpClient wrapper...
Example using NSUrlSession task:
var videoURL = NSUrl.FromString(urlString);
var videoPath = NSFileManager.DefaultManager.GetUrls(NSSearchPathDirectory.DocumentDirectory, NSSearchPathDomain.User)[0];
NSUrlSession.SharedSession.CreateDownloadTask(videoURL, (location, response, createTaskError) =>
{
if (location != null && createTaskError == null)
{
var destinationURL = videoPath.Append(response?.SuggestedFilename ?? videoURL.LastPathComponent, false);
// If file exists, it is already downloaded, but for debugging, delete it...
if (NSFileManager.DefaultManager.FileExists(destinationURL.Path)) NSFileManager.DefaultManager.Remove(destinationURL, out var deleteError);
NSFileManager.DefaultManager.Move(location, destinationURL, out var moveError);
if (moveError == null)
{
PHPhotoLibrary.RequestAuthorization((status) =>
{
if (status.HasFlag(PHAuthorizationStatus.Authorized))
{
PHPhotoLibrary.SharedPhotoLibrary.PerformChanges(() =>
{
PHAssetChangeRequest.FromVideo(destinationURL);
}, (complete, requestError) =>
{
if (!complete && requestError != null)
Console.WriteLine(requestError);
});
}
});
}
else
Console.WriteLine(moveError);
}
else
Console.WriteLine(createTaskError);
}).Resume();
Note: To confirm your code, try using a known valid secure URL source:
https://clips.vorwaerts-gmbh.de/big_buck_bunny.mp4 via “Video For Everybody” Test Page
What I am trying to accomplish here is to see how much this API can handle as far as requests per second. I am trying to consume the API in a console app that will ultimately be throwaway code. My idea was to make a for loop that would try to upload an xml document every 2 seconds. I've never done this sort of thing before so forgive my ignorance. Here's my Main method:
static void Main()
{
RunAsync().Wait();
}
And the RunAsync method:
static async Task RunAsync()
{
Uri apiUrl = new Uri("http://apiurl.com/upload/files/uploadfiles");
const string file = #"C:\simple.xml";
WebClient client = new WebClient();
for (int i = 0; i <= 100; i++)
{
client.UploadFileCompleted += FileUploadSuccess;
client.UploadFileAsync(apiUrl, file);
await Task.Delay(2000);
Console.WriteLine("Upload waiting 2 seconds...");
}
Console.WriteLine("Loop completed.");
}
And the success method:
private static void FileUploadSuccess(object sender, UploadFileCompletedEventArgs e)
{
string reply = System.Text.Encoding.UTF8.GetString(e.Result);
Console.WriteLine("The file result was: {0}", reply);
}
It throws an exception on the first time through on e.Result. Here's the exception:
An unhandled exception of type 'System.Reflection.TargetInvocationException' occurred in System.dll
After doing some research, apparently I can't call the API method (which returns an async Task) without await'ing it. Unfortunately it seems UploadFileAsync is not "awaitable."
Here's the API method:
public async Task<HttpResponseMessage> UploadFiles()
{
var pilotTokenObject = TokenHelper.CreatePilotTokenObject(Request);
byte[] fileBuffer = null;
HttpResponseMessage retVal = null;
if (pilotTokenObject != null)
{
var content = Request.Content;
if (content == null)
{
throw new PilotApiException("Empty request content", HttpStatusCode.NoContent);
}
if (!content.IsMimeMultipartContent())
{
throw new PilotApiException("Request does not contain not multi-part content");
}
var uploadModelController = new PilotUploadModelController();
//*SAVE STREAMED FILE*
string serverSavePath = ConfigurationManager.AppSettings["PilotUploadApiTempStoragePath"];
if (!Directory.Exists(serverSavePath))
Directory.CreateDirectory(serverSavePath);
var provider = new MultipartFormDataStreamProvider(serverSavePath);
await Request.Content.ReadAsMultipartAsync(provider);
var fileData = provider.FileData;
if (fileData == null || fileData.Count == 0)
{
throw new PilotApiException("No multipart/form file data present.");
}
bool uploaded = false;
//Loop through each file
fileData.ForEach((fileRequest) =>
{
if (RetryUntilFileReadable(Path.Combine(serverSavePath, fileRequest.LocalFileName), 1000, 5))
{
var fileHeader = fileRequest.Headers;
if (fileHeader != null && fileHeader.ContentDisposition != null)
{
var fileName = fileHeader.ContentDisposition.FileName.Replace("\"", "");
var fileBytes = File.ReadAllBytes(Path.Combine(serverSavePath, fileRequest.LocalFileName));
//Save File to DB
var upload = uploadModelController.UploadHelper
.AddUploadFileToDb(pilotTokenObject.CentralUserDbUserId, pilotTokenObject.ClientIp, pilotTokenObject.UserAgentString,
UploadEnums.UploadKind.PilotUploadApi, fileName, fileBytes.Length, fileBytes,
UploadEnums.EncryptionType.None);
if (upload != null)
uploaded = true;
}
}
});
if (uploaded)
{
retVal = Request.CreateResponse(HttpStatusCode.Accepted, new
{
Response = String.Format("file uploaded successfully.")
});
}
}
return retVal;
}
Am I going about this the completely wrong way? Is what I want to do even doable?
It seems to me that the following would work better in your scenario:
byte[] response = await Task.Run(() => client.UploadFile(apiUrl, file));
string reply = System.Text.Encoding.UTF8.GetString(response);
Console.WriteLine("The file result was: {0}", reply);
Console.WriteLine("Upload waiting 2 seconds...");
await Task.Delay(2000);
Trying to mix-and-match the older asynchronous API with the newer async/await doesn't seem fruitful in this case. Better to just wrap the synchronous version of the API with async/await-compatible code.
(Note that it seems to me you could just as well call Thread.Sleep(2000) instead of creating a new delay task to wait on, but the above should work fine too).
I am new to C# and using Task. I was trying to run this application but my application hangs every time. When I am adding task.wait(), it keeps waiting and never returns. Any help is much appreciated.
EDIT:
I want to call DownloadString Asynchronously. And when I am doing task.Start() as suggested by "Austin Salonen" I don't get the address of location but default value in location string from returnVal. It means that location got value assigned before task got completed. How can I make sure that after task is completed only then location gets assigned returnVal.
public class ReverseGeoCoding
{
static string baseUri = "http://maps.googleapis.com/maps/api/geocode/xml?latlng={0},{1}&sensor=false";
string location = "default";
static string returnVal = "defaultRet";
string latitude = "51.962146";
string longitude = "7.602304";
public string getLocation()
{
Task task = new Task(() => RetrieveFormatedAddress(latitude, longitude));
//var result = Task.Factory.StartNew(RetrieveFormatedAddress("51.962146", "7.602304"));
task.Wait();
//RetrieveFormatedAddress("51.962146", "7.602304");
location = returnVal;
return location;
}
public static void RetrieveFormatedAddress(string lat, string lng)
{
string requestUri = string.Format(baseUri, lat, lng);
using (WebClient wc = new WebClient())
{
wc.DownloadStringCompleted += new DownloadStringCompletedEventHandler(wc_DownloadStringCompleted);
wc.DownloadStringAsync(new Uri(requestUri));
}
}
static void wc_DownloadStringCompleted(object sender, DownloadStringCompletedEventArgs e)
{
var xmlElm = XElement.Parse(e.Result);
var status = (from elm in xmlElm.Descendants()
where elm.Name == "status"
select elm).FirstOrDefault();
if (status.Value.ToLower() == "ok")
{
var res = (from elm in xmlElm.Descendants()
where elm.Name == "formatted_address"
select elm).FirstOrDefault();
//Console.WriteLine(res.Value);
returnVal = res.Value;
}
else
{
returnVal = "No Address Found";
//Console.WriteLine("No Address Found");
}
}
}
You aren't actually running the task. Debugging and checking task.TaskStatus would show this.
// this should help
task.Start();
// ... or this:
Task.Wait(Task.Factory.StartNew(RetrieveFormatedAddress("51.962146", "7.602304")));
Though if you're blocking, why start another thread to begin with?
I don't understand why you use DownloadStringCompleted event and try to make it blocking. If you want to wait the result, just use blocking call DownloadString
var location = RetrieveFormatedAddress(51.962146, 7.602304);
string RetrieveFormatedAddress(double lat, double lon)
{
using (WebClient client = new WebClient())
{
string xml = client.DownloadString(String.Format(baseUri, lat, lon));
return ParseXml(xml);
}
}
private static string ParseXml(string xml)
{
var result = XDocument.Parse(xml)
.Descendants("formatted_address")
.FirstOrDefault();
if (result != null)
return result.Value;
else
return "No Address Found";
}
If you want to make it async
var location = await RetrieveFormatedAddressAsync(51.962146, 7.602304);
async Task<string> RetrieveFormatedAddressAsync(double lat,double lon)
{
using(HttpClient client = new HttpClient())
{
string xml = await client.GetStringAsync(String.Format(baseUri,lat,lon));
return ParseXml(xml);
}
}