I'm coding a plug-in for Excel. I'd like to add a new method to excel that can crawl a web page and get back the html code.
my problem is that i have a lot of URLs to proces and if I use a sync method, it will take a lot of time and freeze my excel.
let say, i have a cell A1 which contains "http://www.google.com", and in A2, my method "=downloadHtml(A1)".
I'm using HttpClient because it is already handling Async. So here is my code :
static void Main()
{
GetWebPage(new Uri("http://www.google.com"));
}
static async void GetWebPage(Uri URI)
{
string html = await HttpGetAsync(URI);
//Do other operations with html code
Console.WriteLine(html);
}
static async Task<string> HttpGetAsync(Uri URI)
{
try
{
HttpClient hc = new HttpClient();
Task<Stream> result = hc.GetStreamAsync(URI);
Stream vs = await result;
StreamReader am = new StreamReader(vs);
return await am.ReadToEndAsync();
}
catch (WebException ex)
{
switch (ex.Status)
{
case WebExceptionStatus.NameResolutionFailure:
Console.WriteLine("domain_not_found");
break;
//Catch other exceptions here
}
}
return "";
}
The probem is that, when i run the program, the program exits before the task complete.
If i add a
Console.ReadLine();
the program will not exit do to the readline instruction, and after a couple of seconds, i see the html printed into my screen (du to the console.writeline instruction). So the program works.
how can i handle this ?
GetWebPage is a fire-and-forget method (an async void), so you cannot wait for it to finish.
You should be using this instead:
static void Main()
{
string html = Task.Run(() => HttpGetAsync(new Uri("http://www.google.com"))).GetAwaiter().GetResult();
//Do other operations with html code
Console.WriteLine(html);
}
Also, you could simplify the download code to this:
using (var HttpClient hc = new HttpClient())
{
return await hc.GetStringAsync(URI);
}
Related
I'm completely new to Unity development and I'm trying to integrate an asynchronous functionality into an existing Coroutine but I have faced several issues so far.
My issues:
The Unity app completely freezes, probably because it was blocked by a thread. I implemented this code in traditional C# (console app) without having any issues.(Fixed now in Unity after some modifications)
The download task begins but it never finishes . This happens only when I run it as APK. On Unity debugging on PC works fine.
My code:
public void Validate()
{
StartCoroutine(DoWork());
}
private IEnumerator DoWork()
{
bool success;
//Perform some calculations here
..
..
success = true;
//
yield return new WaitForSeconds(3);
if(success){
GetInfo(_files);
}
}
async void GetInfo(List<string> files)
{
await StartDownload(files);
//After completing the download operation, perform some other actions in the background
...
...
//
//When done, change the active status of specific game objects
}
public async Task StartDownload(List<string> files){
var t1 = GetFileSizesAsync(files);
var t2 = DownloadFilesAsync(files);
await Task.WhenAll(t1, t2);
}
public async Task GetFileSizesAsync(List<string> urls)
{
foreach (var url in urls)
GetFileSize(url);
txtSize.text = totalSizeMb +"MB";
}
private void GetFileSize(string url)
{
var uri = new Uri(url);
var webRequest = HttpWebRequest.Create(uri);
webRequest.Method = "GET";
try
{
var webResponse = webRequest.GetResponse();
var fileSize = webResponse.Headers.Get("Content-Length");
var fileSizeInMegaByte = Math.Round(Convert.ToDouble(fileSize) / 1024.0 / 1024.0, 2);
totalSizeMb = totalSizeMb + fileSizeInMegaByte;
}
catch (Exception ex)
{
}
finally
{
webRequest.Abort();
}
}
public async Task<List<string>> DownloadFilesAsync(List<string> urls)
{
var result = new List<string>();
foreach (var url in urls)
var download = await DownloadFile(url);
if(download)
result.Add(url);
return response;
}
private async Task<bool> DownloadFile(string url)
{
WebClient webClient = new WebClient();
var uri = new Uri(url);
var saveHere = "C:\\...";
try
{
await webClient.DownloadFileTaskAsync(uri, saveHere);
return true;
}
catch (Exception ex)
{
return false;
}
}
Can you tell me what I'm doing wrong here? I've tried several ways but couldn't manage to find a proper solution.
Thank you!
I would rather make it
async Task GetInfo(List<string> files){ ... }
And then in your Coroutine do
var t = Task.Run(async () => await GetInfo(files));
while(!t.IsCompleted)
{
yield return null;
}
// Or also
//yield return new WaitUntil(() => t.IsCompleted);
if(!t.IsCompletedSuccesfully)
{
Debug.LogError("Task failed or canceled!");
yield break;
}
Note however:
When done, change the active status of specific game objects
This can't be done async! It has to happen in the Unity main thread! Therefore you would probably rather return something from your GetInfo task and activate the objects in the Coroutine when done.
after the loop and yielding you could then access the return value via
var result = t.Result;
Your web requests are currently totally redundant! You start get requests only to check how big the received content is but immediately throw away that received content ... then you start a second request to actually download the files (again).
In general I would recommend to rather use a UnityWebRequest.Get you can directly yield in the Coroutine in combination with a DownloadHandlerFile which allows you to directly download the content into a local file instead of into runtime memory.
Also
var saveHere = "C:\\...";
is hopefully not what you are trying to use as path on Android ;)
First of all.
The freezing is definitely being done by the IEneumerator as I don't see a yield statement. What's a yield statement? Uhhhh...Google it. lel
Second of all.
Wouldn't a normal void work just fine?
Third of all.
I don't know much about async's since I'm pretty new to them
but I'm fairly certain you don't need them for:
Task GetFileSizesAsync(List urls)
AND
async Task<List> DownloadFilesAsync(List urls)
I may be wrong tho.
I have an issue where i loop over about 31 webservice URLs.
If i put a Thread.Sleep(1000) in the top code, it will work perfectly, but if I remove this, I only get success on 10 (sometimes less and sometimes more) request out of 31. How do I make it wait?
Code
foreach(var item in ss)
{
//Call metaDataApi(url,conn,name,alias)
}
public static void metadataApi(string _url, string _connstring, string _spname, string _alias)
{
// Thread.Sleep(1000);
//Metadata creation - Table Creation
using (var httpClient = new HttpClient())
{
string url = _url;
using (HttpResponseMessage response = httpClient.GetAsync(url).GetAwaiter().GetResult())
using (HttpContent content = response.Content)
{
Console.WriteLine("CHECKING");
if (response.IsSuccessStatusCode)
{
Console.WriteLine("IS OK");
string json = content.ReadAsStringAsync().GetAwaiter().GetResult();
//Doing some stuff not relevant
}
}
}
}
How it can look
You should look to use async/await where you can, but you could try something like this:
// you should share this for connection pooling
public static HttpClient = new HttpClient();
public static void Main(string[] args)
{
// build a list of tasks to wait on, then wait
var tasks = ss.Select(x => metadataApi(url, conn, name, alias)).ToArray();
Task.WaitAll(tasks);
}
public static async Task metadataApi(string _url, string _connstring, string _spname, string _alias)
{
string url = _url;
var response = await httpClient.GetAsync(url);
Console.WriteLine("CHECKING");
if (response.IsSuccessStatusCode)
{
Console.WriteLine("IS OK");
string json = await content.ReadAsStringAsync();
//Doing some stuff not relevant
}
}
One thing to note, this will try to run many in parallel. If you need to run them all one after the other, may want to make another async function that waits on each result individually and call that from the Main. .Result is a bit of an antipattern (with modern c# syntax, you can use async on the main function) but for your script it should be "ok", but I'd minimize usage of it (hence why I wouldn't use .Result inside of a loop.
I searched and saw many posts and I don't know why this simple console call using httpclient.getasync and awaiting it causes it not to complete. here is the code:
using System;
using System.Net.Http;
using System.Threading.Tasks;
public class Program
{
public static void Main(string[] args)
{
GetAPI().Wait();
Console.WriteLine("Hello");
//Console.ReadLine();
}
public static async Task GetAPI()
{
using (HttpClient client = new HttpClient())
{
var response = await client.GetAsync("https://www.google.com/");
//var response = client.GetAsync("https://www.google.com/").Result
string content = await response.Content.ReadAsStringAsync();
Console.WriteLine(content);
}
}
}
If I change to use client.GetAsync("https://www.google.com/").Result; and remove "await" (see commented out code) it will work however I believe that is a no no because I am making an asynchronous function into synchronous. I've seen other posts and blogs about this and it all claims what I am trying to do is correct but when running the example app it just doesn't turn out that way.
You can call wait() in Main.
But be carefull, this only works in console applications and will dead lock in UI applications like WPF, WinForms or in ASP.NET context...
public static void Main(string[] args)
{
try
{
var t = GetAPI();
t.Wait();
Console.WriteLine(t.Result);
}
catch (Exception ex)
{
Console.WriteLine(ex.Message);
}
}
public static async Task<string> GetAPI()
{
using (HttpClient client = new HttpClient())
{
var response = await client.GetAsync("https://www.google.com/");
string content = await response.Content.ReadAsStringAsync();
return content;
}
}
MSDN
This code will work just fine in a console application but will
deadlock when called from a GUI or ASP.NET context.
The issue is that awaiting the client.GetAsync, after getting the result is waiting for main thread to be free and main thread is waiting for client.GetAsync to complete.
var response = await client.GetAsync("https://www.google.com/").ConfigureAwait(false);
string content = await response.Content.ReadAsStringAsync().ConfigureAwait(false);
Will tell that client.GetAsync to complete on other thread and then return the result back to main thread so that no more dead lock.
I have a WinForms application that has two roles. If no command line parameters are present, the Main function calls Application.Run, and presents the UI. If command line parameters are present, Application.Run is NOT called. Instead, I call an async method like this:
result = HandleCommandLine(args).GetAwaiter().GetResult();
(I am new to async/await, and this form was based on a SO answer).
The end goal is to loop through a list, and for each entry, start a new task. Each of those tasks should run in parallel with the others. The tasks are started like this:
runningTasks.Add(Task.Factory.StartNew((args) => HandlePlayback( (Dictionary<string,string>) ((object[])args)[0]), new object[] { runArgs } ));
The tasks are added to the collection of runningTasks, and I later call:
Task.WaitAll(runningTasks.ToArray());
In each of the runningTasks, I am trying to send web requests using HttpClient:
using (HttpResponseMessage response = await Client.SendAsync(message))
{
using (HttpContent responseContent = response.Content)
{
result = await responseContent.ReadAsStringAsync();
}
}
Once Client.SendAsync is called, the whole thing goes belly up. All of my runningTasks complete, and the application exits. Nothing past the Client.SendAsync executes in any of those tasks.
Since I am new at async/await, I have very few ideas about what exactly might be wrong, and hence few ideas about how to fix it. I imagine it has something to do with the SynchronizationContexts in this situation (WinForms app acting like a console app), but I'm not grasping what I need to do and where to keep the service request and the web request async calls from causing everything to complete too early.
I guess my question then is, why are (only some) 'awaited' calls causing all tasks to complete? What can I do about it?
UPDATE:
Two things. #Joe White: The WindowsFormsSynchronizationContext.Current is always null wherever I check.
#David Pine: Minimal (kind of :) ) complete viable example follows. You will either need to add a command line argument to the project, or force execution to the HandleCommandLine function. In this example, it tries to make a website request for each of three sites. It doesn't appear to matter if they exist. The code reaches the Client.SendAsync some number of times (usually not three), but timing appears to matter.
using System;
using System.Collections.Generic;
using System.Net;
using System.Net.Http;
using System.Text;
using System.Threading.Tasks;
using System.Windows.Forms;
namespace WindowsFormsApplication1
{
static class Program
{
static List<Task> runningTasks = new List<Task>();
[STAThread]
static int Main()
{
int result = 1; // true, optimism
Application.EnableVisualStyles();
Application.SetCompatibleTextRenderingDefault(false);
string[] args = Environment.GetCommandLineArgs();
if (args.Length > 1)
{
// do the command line work async, while keeping this thread active
result = HandleCommandLine(args).GetAwaiter().GetResult();
}
else
{
// normal interface mode
Application.Run(new Form1());
}
return result;
}
static async Task<int> HandleCommandLine(string[] args)
{
// headless mode
int result = 1; // true, optimism
result = await HandleControlMode(args);
return result;
}
private static async Task<int> HandleControlMode(string[] Arguments)
{
int result = 1; // optimism
try
{
List<string> sites = new List<string>() { #"http://localhost/site1", #"http://localhost/site2", #"http://localhost/site3" };
foreach (string site in sites)
{
Begin(site); // fire off tasks
// the HandleControlMode method is async because in other circumstances, I do the following:
//await Task.Delay(5000); // sleep 5 seconds
}
// wait while all test running threads complete
try
{
Task.WaitAll(runningTasks.ToArray());
}
catch (Exception)
{
// not really a catch all handler...
}
}
catch (Exception)
{
// not really a catch all handler...
}
return result;
}
private static void Begin(string site)
{
//runningTasks.Add(Task.Factory.StartNew(() => HandlePlayback(runArgs)));
runningTasks.Add(Task.Factory.StartNew((args) => HandlePlayback((string)((object[])args)[0]), new object[] { site }));
}
private static async Task<int> HandlePlayback(string site)
{
int result = 1;
try
{
PlaybackEngine engine = new PlaybackEngine(site);
bool runResult = await engine.RunCommandLine(site);
if (!runResult)
{
result = 0;
}
}
catch (Exception)
{
result = 0;
}
return result;
}
}
public class PlaybackEngine
{
private static HttpClientHandler ClientHandler = new HttpClientHandler()
{
AllowAutoRedirect = false,
AutomaticDecompression = System.Net.DecompressionMethods.GZip | DecompressionMethods.Deflate
};
private static HttpClient Client = new HttpClient(ClientHandler);
public string Target { get; set; }
public PlaybackEngine(string target)
{
Target = target;
}
public async Task<bool> RunCommandLine(string site)
{
bool success = true;
string response = await this.SendRequest();
return success;
}
private async Task<string> SendRequest()
{
string result = string.Empty;
string requestTarget = Target;
HttpMethod method = HttpMethod.Post;
var message = new HttpRequestMessage(method, requestTarget);
StringContent requestContent = null;
requestContent = new StringContent("dummycontent", Encoding.UTF8, "application/x-www-form-urlencoded");
message.Content = requestContent;
try
{
using (HttpResponseMessage response = await Client.SendAsync(message))
{
using (HttpContent responseContent = response.Content)
{
result = await responseContent.ReadAsStringAsync();
System.Diagnostics.Debug.WriteLine(result);
}
}
}
catch (Exception ex)
{
}
return result;
}
}
}
UPDATE2:
I put similar code online at http://rextester.com/CJS33330
It's a straight console app, and I've added .ConfigureAwait(false) to all awaits (with no effect). In separate testing, I tried 4 or 5 other ways to call the first async function from Main - which all worked but had the same behavior.
The problem with this code is that I am not waiting on the Tasks that I thought I was. The runningTasks collection accepts any kind of Task. I didn't realize that Task.Factory.StartNew returned different type than the Task I was trying to start. My function returns
Task<int>
but StartNew returns
Task<Task<int>>
Those tasks completed immediately, and so the main thread did not stay alive long enough for the actual routines to run. You have to wait on the inner task instead:
Task<Task<int>> wrappedTask = Task.Factory.StartNew(...);
Task<int> t = await wrappedTask;
runningTasks.Add(t);
...
Task allTasks = Task.WhenAll(runningTasks.ToArray());
await allTasks;
For some reason, I was not able to use the built in ".Unwrap" function that should be equivalent, but the above code does the job.
I am struggling around the (seems so) pretty famous problem of the exception handling by using the async/await pattern. Specifically my context is on a HTTP client, but I have also tried with a much simpler test, and it behaves the same.
Consider the below program, which is a super-simplified version of my original app's context.
class Program
{
static void Main(string[] args)
{
Test();
Console.Write("Press any key...");
Console.ReadKey();
Console.WriteLine();
}
static async void Test()
{
var c = new MyClient();
try
{
var uri = new Uri("http://www.google.com/"); //valid address
var s = await c.GetString(uri);
Console.WriteLine(s.Length);
}
catch (WebException ex)
{
Console.WriteLine(ex.Message);
}
try
{
var uri = new Uri("http://www.foo.bah/"); //non-existent address
var s = await c.GetString(uri);
Console.WriteLine(s.Length);
}
catch (WebException ex)
{
Console.WriteLine(ex.Message);
}
}
}
class MyClient
{
public async Task<string> GetString(Uri uri)
{
var client = new HttpClient();
return await client.GetStringAsync(uri);
}
}
When the program starts, it downloads the first web site's page as a string, then displays its length: that's fine. Afterward, when the same operation is performed against an invalid address, the client raises a WebException (that's what I want), but it's not caught.
UPDATE: as "not caught", I mean that the code actually does not flow through the "catch" branch and silently displays the exception message. Instead, the exception is shown by the VS IDE, and the debugging breaks.
Any decent solution to catch the exception?
Many thanks in advance.
Although you have already figured out the exception is HttpRequestException not WebException, still I would like to highlight few important things about async-await operator usage.
async void is of type fire & forget and is only & only for event handlers.
As soon as compiler reaches first await operator inside async method control returns to the caller.
Debugging your code :-
Since you are using async void in Test method so the control returns to the caller and execution continues to line Console.Write("Press any key..."); without having any information about the Task and then you are waiting for the user input.
In the meanwhile response from awaited method comes and the execution continues inside Test method.
If you comment out the line Console.ReadKey(); inside main() OR user provides input immediately then you'll notice that response may or may not get printed. This is because you are not waiting on the Task getting executed you simply trusted on the user that he will not enter anything till your Task completes.
Solution:-
Solution is to return Task from Test() and then wait till it finishes, below is the updated code also note adding Async at the end of method name is the naming convention you must follow to save you from the headache of distinguishing between asynchronous and synchronous methods.
class Program
{
static void Main(string[] args)
{
Task task = TestAsync();
Console.Write("Press any key...");
task.wait();
//Console.ReadKey();
Console.WriteLine();
}
static async Task<string> TestAsync()
{
var c = new MyClient();
try
{
var uri = new Uri("http://www.google.com/"); //valid address
var s = await c.GetStringAsync(uri);
Console.WriteLine(s.Length);
}
catch (HttpRequestException ex)
{
Console.WriteLine(ex.Message);
}
try
{
var uri = new Uri("http://www.foo.bah/"); //non-existent address
var s = await c.GetStringAsync(uri);
Console.WriteLine(s.Length);
}
catch (HttpRequestException ex)
{
Console.WriteLine(ex.Message);
}
//to avoid compiler error
return null;
}
}
class MyClient
{
public async Task<string> GetStringAsync(Uri uri)
{
var client = new HttpClient();
return await client.GetStringAsync(uri);
}
}