I have a project where I need multiple data for a list of projects. For each project I call an api to get me this information. The loop works, although it takes 4 to 5 minutes to finish(Which is A LOT).
The code used to look like this :
foreach (var project in projects)
{
string url = urlOneProject + project.name + secondPartUrl + "?authtoken=" + authToken;
HttpWebRequest request;
request = (HttpWebRequest)WebRequest.Create(url);
request.Accept = "application/json";
request.ContentType = "application/json";
var executions = new Execs();
var response = (HttpWebResponse)(await request.GetResponseAsync());
using (response)
{
using (var reader = new StreamReader(response.GetResponseStream()))
{
JavaScriptSerializer js = new JavaScriptSerializer();
var objText = reader.ReadToEnd();
executions = (Execs)js.Deserialize(objText, typeof(Execs));
}
}
execs.AddRange(executions.executions);
}
To improve performance I thought that using threads might be a good idea. So, I came up with something like this:
ManualResetEvent resetEvent = new ManualResetEvent(false);
int toProcess = projects.Count;
foreach (var project in projects)
{
new Thread(() =>
{
string url = urlOneProject + project.name + secondPartUrl + "?authtoken=" + authToken;
HttpWebRequest request;
request = (HttpWebRequest)WebRequest.Create(url);
request.Accept = "application/json";
request.ContentType = "application/json";
var executions = new Execs();
var response = (HttpWebResponse)(await request.GetResponseAsync());
using (response)
{
using (var reader = new StreamReader(response.GetResponseStream()))
{
JavaScriptSerializer js = new JavaScriptSerializer();
var objText = reader.ReadToEnd();
executions = (Execs)js.Deserialize(objText, typeof(Execs));
}
}
lock (execs)
{
execs.AddRange(executions.executions);
}
if (Interlocked.Decrement(ref toProcess) == 0)
resetEvent.Set();
}).Start();
}
The problem with this code is that the line:
var response = (HttpWebResponse)(await request.GetResponseAsync());
doesn't compile anymore from the moment I added Thread. And the error I get is
"The 'await' operator can only be used within an async lambda expression "
That wasn't a problem when I didn't use threads. The GetResponseAsync is an async function and the use of the await is compulsory. I tried deleting it (which wasn't logical I agree but I ran out of options) but the compiler tells me I need an await for an async function.
I don't quite understand what changes with the implementation of the Thread.
Didn't I use the threads mechanically correctly? What should I do to correct this or to implement what I want to do correctly?
You are mixing multiple paradigms of coding, viz async / await, and oldschool Thread starting and synchronization, which is likely to lead to trouble.
As per the above comments
The reason your code doesn't compile is because you are attempting to use await in otherwise synchronous code passed to the thread. You can qualify lambdas as async as well.
Task is a much safer paradigm than Thread, and TPL provides rich and expressive tools to assist in asynchrony and parallelism.
If you process each parallel task in isolation, without sharing any data (such as the collections that you are locking), but instead return the resultant data from each Task you can then use LINQ to collate the results in a thread-safe manner.
var myTasks = projects.Select(async project =>
{
var url = $"urlOneProject{project.name}{secondPartUrl}?authtoken={authToken}";
var request = (HttpWebRequest) WebRequest.Create(url);
request.Accept = "application/json";
request.ContentType = "application/json";
using (var response = (HttpWebResponse) (await request.GetResponseAsync()))
using (var reader = new StreamReader(response.GetResponseStream()))
{
var objText = await reader.ReadToEndAsync();
return JsonConvert.DeserializeObject<Execs>(objText);
}
});
var execs = (await Task.WhenAll(myTasks))
.SelectMany(result => result.executions);
Other notes
Don't use JavaScriptSerializer - even the MSDN docco says to use NewtonSoft Json
There's an async version of reader.ReadToEndAsync which I've included.
You can drop the locks and the ManualResetEvent - since each Task returns it's result, we'll leave it to Task.WhenAll to collate the data.
You can flatten the children of multiple executions with a SelectMany
Adjacent using clauses are stackable - it saves a bit of eyestrain on the indentation.
Related
I have this method that downloads data from the internet:
public ObservableCollection<Magnetka> ParseJSON() {
ObservableCollection<Models.Magnetka> Markers = new ObservableCollection<Models.Magnetka>();
HttpWebRequest request = (HttpWebRequest)WebRequest.Create("(URL)");
request.Headers.Add("Authentication-Token", "(KEY)");
request.Method = "GET";
request.ContentType = "application/json";
CookieContainer cookieContainer = new CookieContainer();
request.CookieContainer = cookieContainer;
var response = request.GetResponse();
var responseString = new StreamReader(response.GetResponseStream()).ReadToEnd();
......... JSON parsing)
return Markers;
Now I found out this isn't too fast and it freezes the app for two seconds or so, so I need to make it async. However, I call the method like this in another class:
GetMarkers getMarkers = new GetMarkers();
Markers = getMarkers.ParseJSON();
How can I make my method async? I don't understand the concept well and if I make ParseJSON() async Task, I don't know where to put the "await" keyword.
Can anyone help me?
first, mark the method async and use the async version of GetResponse
public async Task<ObservableCollection<Magnetka>> ParseJSON()
{
...
var response = await request.GetResponseAsync();
...
return Markers;
}
then when you call it use the await keyword
Markers = await getMarkers.ParseJSON();
You would change your method to something like this instead:
public async Task<ObservableCollection<Magnetka>> ParseJSON()
{
var markers = new ObservableCollection<Magnetka>();
HttpWebRequest request = (HttpWebRequest)WebRequest.Create("(URL)");
request.Headers.Add("Authentication-Token", "(KEY)");
request.Method = "GET";
request.ContentType = "application/json";
CookieContainer cookieContainer = new CookieContainer();
request.CookieContainer = cookieContainer;
using var response = await request.GetResponseAsync().ConfigureAwait(false);
using var responseStream = response.GetResponseStream();
using var streamReader = new StreamReader(responseStream);
var json = await streamReader.ReadToEndAsync().ConfigureAwait(false);
// json parsing
return markers;
}
Firstly, you would mark your method async. This signals the compiler to set up a state machine for this method for continuation. Nothing async happening yet though.
Another thing you need to do is to wrap your return value in a Task, which is similar to a promise in some other languages. Tasks are await-able, which is where the asynchronous stuff is happening.
Switching from using the synchronous methods in WebRequest to use the Async variants and awaiting their results makes your method async.
So from:
request.GetResponse()
To:
await request.GetResponseAsync()
Which returns a awaitable Task<WebResponse>.
The same goes for the stream reader code, you can use its async API too.
A tricky thing by using asynchronous code is that, it kind of forces you to do this all the way up. However, there are exceptions for lifecycle methods and events, where you don't necessarily have to change the signature to return a Task.
Another thing you might notice here. I've added ConfigureAwait(false) at the end of the async method calls. I've done this because out of the box, after returning from an awaited method, the code will try to return to the originating context. This can lead to a couple of issues such as dead-locks if the originating context is the UI Thread. Or, it can lead to bad performance trying to switch between context a lot. Adding ConfigureAwait(false) will just return on the same context the await was run on, which is fine as long as we don't need or expect to return on that context.
As for where you call ParseJson() is probably equally important. Consider deferring it to some lifecycle method or event where you are not doing much else.
This could be OnResume() on Android or ViewWillAppear on iOS. If you are using MVVM, use a Command to encapsulate it.
I am trying to use Microsoft Project OData by querying data in C#. I am having performances issues with delays around 1s for each query. I am trying to query 2 information at once using that method :
public static async Task<string> ReadXml(string url)
{
var request = (HttpWebRequest)WebRequest.Create(url);
request.Credentials = Credentials; // SharePointOnlineCredentials
request.Headers.Add("X-FORMS_BASED_AUTH_ACCEPTED", "f");
using (var response = (HttpWebResponse)await request.GetResponseAsync())
using (var stream = response.GetResponseStream())
using (var reader = new System.IO.StreamReader(stream))
{
var xml = await reader.ReadToEndAsync();
return xml;
}
}
It works fine if I call it and always wait for it to end before calling it again, but I never receive any response from the WebRequest if I call it multiple times at once :
// just an example. I usually put a condition to filter for the tasks of a single project
var query1 = ReadXml(#"https://something.sharepoint.com/sites/pwa/_api/ProjectData/Projects");
var query2 = ReadXml(#"https://something.sharepoint.com/sites/pwa/_api/ProjectData/Tasks");
Task.WaitAll(query1, query2);
If I "await" the first one and then do the second one it works fine, but not with the code above. And this is assuming there is < 300 tasks in the project, if more than that I have to query them in chunk of 300 leading to 4 or 5 seconds for the entire query since I can't to them all at once!
Is there a way to send multiple request at the same time ?
I am able to do it by simply entering the url in multiple chrome tabs really fast / have faster responses. I don't understand why it doesn't work with my code!
Thanks,
According to the following post Webrequest.Create could be the problem, it uses an internally blocking method C# Thread UI is getting blocked | Possible reason WebRequest.Create?.
The code below uses the newer HttpClient and shouldn't have this issue.
public static HttpClient _HttpClient { get; } = new HttpClient(new HttpClientHandler { Credentials=new NetworkCredential("","")});
public static async Task<string> ReadXml(string url)
{
using (var requestMessage = new HttpRequestMessage(HttpMethod.Get, url))
{
requestMessage.Headers.Add("X-FORMS_BASED_AUTH_ACCEPTED", "f");
using (var response = await _HttpClient.SendAsync(requestMessage))
{
return await response.Content.ReadAsStringAsync();
}
}
}
When I try to make POST requests in Xamarin the time the function GetRequestStream() takes is 10-20 seconds. The answer time on my server is below 1 second and so is the POST request I made from a website.
I already tried:
Setting proxy server to null so it won't look it up
Using blocks around the requests so it get's flushed
Increased the maximum connections
Did everything with async multithreading
Even tried another class called RestSharp - same result.
Nothing I did was actually helping to reduce the runtime even by 100 milliseconds. I just cannot imagine that this is Xamarins fault because I can't be the only one who decided to do some HTTP requests in his cross platform app. I already lost UWP since the ServiceManager, which I use to connect to TLS sites isn't supported in UWP - thank you Xamarin.
I really need solutions so please help :)
this is the code i used and optimized:
byte[] byteArray = Encoding.UTF8.GetBytes(query);
HttpWebRequest webRequest = new HttpWebRequest(uri);
webRequest.Method = "POST";
webRequest.Proxy = null;
webRequest.ContentType = "application/x-www-form-urlencoded";
webRequest.Timeout = 1000;
webRequest.ReadWriteTimeout = 1000;
Stream dataStream = await webRequest.GetRequestStreamAsync();
await dataStream.WriteAsync(byteArray, 0, byteArray.Length);
dataStream.Close();
WebResponse webResponse = await webRequest.GetResponseAsync();
using (StreamReader reader = new StreamReader(webResponse.GetResponseStream(), Encoding.UTF8))
{
string ret = await reader.ReadToEndAsync();
return ret;
}
and this is the code i tried with the ModernHttpClient
System.Net.Http.HttpClient client = new System.Net.Http.HttpClient(new NativeMessageHandler());
Dictionary<string, string> values = new Dictionary<string, string>();
string[] q = query.Split('&');
for (int i = 0; i < q.Length; i++)
values.Add(q[i].Split('=')[0], q[i].Split('=')[1]);
FormUrlEncodedContent content = new FormUrlEncodedContent(values);
HttpResponseMessage response;
try
{
response = await client.PostAsync(uri, content);
string answer = await response.Content.ReadAsStringAsync();
return answer;
}
catch (Exception e)
{
return "";
}
and of course i added these lines before calling the whole network stuff
ServicePointManager.ServerCertificateValidationCallback += (a, b, c, d) => { return true; };
ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12;
ServicePointManager.Expect100Continue = false;
ServicePointManager.DefaultConnectionLimit = 100;
It's definitely a Xamarin or an Android Problem. I don't have other devices to test it,but... what can i do now? I tested it on a .Net console application, so it's definitely not .Net related.
I just found the answer!
it speeds up when i set the Protocol Version. It's mentione basically nowhere but i googled for android post slowdowns and there i found it, finally.
setting the protocol version to HTTP 1.1 speeds things up very fast!
now i just have to fix the boot delay with xamarin >.<
Thank you so much for you help!
Try implement ModernHttpClientPlugin into your code.
Just add this line when initializing new HttpClient, and everything should run smoother.
var httpClient = new HttpClient(new NativeMessageHandler());
Another idea is to try your implementation in new blank console aplication. Then you will see if this is problem with your server/client or Xamarin itself. It should look something like this.
class Program
{
static void Main(string[]args)
{
MainAsync(null);
Console.ReadKey();
}
static async System.Threading.Tasks.Task MainAsync(string[] args)
{
System.Net.Http.HttpClient client = new System.Net.Http.HttpClient();
Dictionary<string, string> values = new Dictionary<string, string>();
var query = "user=mickey&passwd=mini";
string[] q = query.Split('&');
for (int i = 0; i < q.Length; i++)
values.Add(q[i].Split('=')[0], q[i].Split('=')[1]);
FormUrlEncodedContent content = new FormUrlEncodedContent(values);
HttpResponseMessage response;
try
{
response = await client.PostAsync(new Uri("https://www.example.com/login.php"), content);
string answer = await response.Content.ReadAsStringAsync();
}
catch (Exception e)
{
}
}
}
Please, give us feedback how fast that goes.
In my case 'WebClient' and 'HttpWebReuest' cause first-time delays about 20 secs (possible, during resolve of new hosts) on some specific Android 7 devices, but works good on others.
I tried different ways to solve this problem but for me only replacement to 'HttpClient' solves it.
I'm trying to use the Azure media services with REST api, in a xamarin shared project on visual studio 2013. This is the code i use to get the access token.
public HttpFactory()
{
string token = GetToken(serviceURI).Result;
//some more methods also async tasks
}
private async Task<string> GetToken(Uri serviceUri)
{
HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create(serviceURI);
request.Accept = "application/json";
request.Method = "GET";
request.Headers["Host"] = "media.windows.net";
request.Headers["x-ms-version"] = "2.9";
request.Headers["Authorization"] = "Bearer " + token;
var response = (HttpWebResponse) await request.GetResponseAsync();
if (response.StatusCode == HttpStatusCode.MovedPermanently)
{
serviceURI = new Uri(response.Headers["Location"]);
HttpWebRequest req = ( HttpWebRequest)WebRequest.Create(serviceURI);
var res = (HttpWebResponse)await req.GetResponseAsync();
using (Stream responseStream = res.GetResponseStream())
{
using (StreamReader reader = new StreamReader(responseStream))
{
string str = reader.ReadToEnd();
// var test = JsonConvert.DeserializeObject(str);
JToken jtoken = JsonConvert.DeserializeObject<JToken>(str);
return jtoken["access_token"].Value<string>();
}
}
}
return "";
}
But when the compiler reaches -
var response = (HttpWebResponse) await request.GetResponseAsync();
it skips the rest of the code, and i never get the response. I know the code is working - because it works just fine without the task, in a async void method.
Anyone knows how to fix this, or am i doing something wrong? I also tried this in vs2015 but its the same.
You have a deadlock over the UI thread.
You're blocking the thread with Task.Result when it is needed to complete the async method which will complete the task that it's waiting on.
That's why you shouldn't block synchronously on asynchronous code. You should await the task returned from GetToken instead:
string token = await GetToken(serviceURI);
If you can't use async in that method, either move that logic to a different method (e.g. OnLoad event handler).
Another solution would be to use ConfigureAwait on the GetResponseAsync task and so the rest of the method wouldn't run on the UI thread, avoiding the deadlock:
var response = (HttpWebResponse) await request.GetResponseAsync().ConfigureAwait(false);
My method looks like this:
public string Request(string action, NameValueCollection parameters, uint? timeoutInSeconds = null)
{
parameters = parameters ?? new NameValueCollection();
ProvideCredentialsFor(ref parameters);
var data = parameters.ToUrlParams(); // my extension method converts the collection to a string, works well
byte[] dataStream = Encoding.UTF8.GetBytes(data);
string request = ServiceUrl + action;
var webRequest = (HttpWebRequest)WebRequest.Create(request);
webRequest.AllowAutoRedirect = false;
webRequest.Method = "POST";
webRequest.ContentType = "application/x-www-form-urlencoded";
webRequest.ContentLength = dataStream.Length;
webRequest.Timeout = (int)(timeoutInSeconds == null ? DefaultTimeoutMs : timeoutInSeconds * 1000);
webRequest.Proxy = null; // should make it faster...
using (var newStream = webRequest.GetRequestStream())
{
newStream.Write(dataStream, 0, dataStream.Length);
}
var webResponse = (HttpWebResponse)webRequest.GetResponse();
string uri = webResponse.Headers["Location"];
string result;
using (var sr = new StreamReader(webResponse.GetResponseStream()))
{
result = sr.ReadToEnd();
}
return result;
}
The server sends JSON in response. It works fine for small JSON, but when I request a large one - something goes wrong. By large one I mean something that takes 1-2 minutes to appear in a browser (google chrome, including server side generation time). It's actually 412KB of text. When I try to ask for the same JSON with the method above I get a web exception (timeout). I changed the timeout to 10 minutes (at least 5 times longer than chrome). Still the same.
Any ideas?
EDIT
This seems to have something to do with MS technologies. On IE this JSON also won't load.
Make sure you close your request. Otherwise, once you hit the maximum number of allowed connections (as low as four, for me, on one occasion), you have to wait for the earlier ones to time out. This is best done using
using (var response = webRequest.GetResponse()) {...