Creating a load tester for an upload API with UploadFileAsync - c#

What I am trying to accomplish here is to see how much this API can handle as far as requests per second. I am trying to consume the API in a console app that will ultimately be throwaway code. My idea was to make a for loop that would try to upload an xml document every 2 seconds. I've never done this sort of thing before so forgive my ignorance. Here's my Main method:
static void Main()
{
RunAsync().Wait();
}
And the RunAsync method:
static async Task RunAsync()
{
Uri apiUrl = new Uri("http://apiurl.com/upload/files/uploadfiles");
const string file = #"C:\simple.xml";
WebClient client = new WebClient();
for (int i = 0; i <= 100; i++)
{
client.UploadFileCompleted += FileUploadSuccess;
client.UploadFileAsync(apiUrl, file);
await Task.Delay(2000);
Console.WriteLine("Upload waiting 2 seconds...");
}
Console.WriteLine("Loop completed.");
}
And the success method:
private static void FileUploadSuccess(object sender, UploadFileCompletedEventArgs e)
{
string reply = System.Text.Encoding.UTF8.GetString(e.Result);
Console.WriteLine("The file result was: {0}", reply);
}
It throws an exception on the first time through on e.Result. Here's the exception:
An unhandled exception of type 'System.Reflection.TargetInvocationException' occurred in System.dll
After doing some research, apparently I can't call the API method (which returns an async Task) without await'ing it. Unfortunately it seems UploadFileAsync is not "awaitable."
Here's the API method:
public async Task<HttpResponseMessage> UploadFiles()
{
var pilotTokenObject = TokenHelper.CreatePilotTokenObject(Request);
byte[] fileBuffer = null;
HttpResponseMessage retVal = null;
if (pilotTokenObject != null)
{
var content = Request.Content;
if (content == null)
{
throw new PilotApiException("Empty request content", HttpStatusCode.NoContent);
}
if (!content.IsMimeMultipartContent())
{
throw new PilotApiException("Request does not contain not multi-part content");
}
var uploadModelController = new PilotUploadModelController();
//*SAVE STREAMED FILE*
string serverSavePath = ConfigurationManager.AppSettings["PilotUploadApiTempStoragePath"];
if (!Directory.Exists(serverSavePath))
Directory.CreateDirectory(serverSavePath);
var provider = new MultipartFormDataStreamProvider(serverSavePath);
await Request.Content.ReadAsMultipartAsync(provider);
var fileData = provider.FileData;
if (fileData == null || fileData.Count == 0)
{
throw new PilotApiException("No multipart/form file data present.");
}
bool uploaded = false;
//Loop through each file
fileData.ForEach((fileRequest) =>
{
if (RetryUntilFileReadable(Path.Combine(serverSavePath, fileRequest.LocalFileName), 1000, 5))
{
var fileHeader = fileRequest.Headers;
if (fileHeader != null && fileHeader.ContentDisposition != null)
{
var fileName = fileHeader.ContentDisposition.FileName.Replace("\"", "");
var fileBytes = File.ReadAllBytes(Path.Combine(serverSavePath, fileRequest.LocalFileName));
//Save File to DB
var upload = uploadModelController.UploadHelper
.AddUploadFileToDb(pilotTokenObject.CentralUserDbUserId, pilotTokenObject.ClientIp, pilotTokenObject.UserAgentString,
UploadEnums.UploadKind.PilotUploadApi, fileName, fileBytes.Length, fileBytes,
UploadEnums.EncryptionType.None);
if (upload != null)
uploaded = true;
}
}
});
if (uploaded)
{
retVal = Request.CreateResponse(HttpStatusCode.Accepted, new
{
Response = String.Format("file uploaded successfully.")
});
}
}
return retVal;
}
Am I going about this the completely wrong way? Is what I want to do even doable?

It seems to me that the following would work better in your scenario:
byte[] response = await Task.Run(() => client.UploadFile(apiUrl, file));
string reply = System.Text.Encoding.UTF8.GetString(response);
Console.WriteLine("The file result was: {0}", reply);
Console.WriteLine("Upload waiting 2 seconds...");
await Task.Delay(2000);
Trying to mix-and-match the older asynchronous API with the newer async/await doesn't seem fruitful in this case. Better to just wrap the synchronous version of the API with async/await-compatible code.
(Note that it seems to me you could just as well call Thread.Sleep(2000) instead of creating a new delay task to wait on, but the above should work fine too).

Related

Connection reset issue in .net core api call (net::ERR_CONNECTION_RESET 200 (OK))

I have a .net core API service which is called from a angular client project.
When a user request a status of his payment, we will make call to this service api and this service will then call a payment gateway service to fetch the status of payment and the output result will return to the user.
When i try to integrate this i am facing this below error.
net::ERR_CONNECTION_RESET 200 (OK)
core.js:5967 ERROR Unknown Error
This above issue is not showing when i try to hit the service after putting one breakpoint. Its also returning the result.
This is how entire flow works
Client side call performs by user
this.dataservice.postFeed(method, JSON.stringify(this.initsearch)).subscribe(result => {
var response = result.body["data"];
console.log(response);
});
Server side code looks like
[HttpPost]
public async Task<IActionResult> Post([FromBody] ObjectModel searchValue)
{
ApiResponse<string> response = new ApiResponse<string>();
IBaseResult<string> result = await _adlerBo.GetPaymentStatus(searchValue);
response.Success = result.success;
response.Data = result.Data;
return Ok(response);
}
In BusinessObject.cs
public async Task<IBaseResult<string>> GetPaymentStatus(PaymentSearchModel requestModel){
string apiResponse = await PaymentStatusCheckUsingAPI(requestModel.orderid);
return apiResponse ;
}
private async Task<string> PaymentStatusCheckUsingAPI(string orderNumber)
{
string message = await PostPaymentRequestToGateway(statusApiUrl, authQueryUrlParam);
NameValueCollection param = await GetResponseMap(message);
string status = "";
string encResJson = "";
if (param != null && param.Count == 2)
{
for (int i = 0; i < param.Count; i++)
{
if ("status".Equals(param.Keys[i]))
{
status = param[i];
}
if ("enc_response".Equals(param.Keys[i]))
{
encResJson = param[i];
}
}
if (!"".Equals(status) && status.Equals("0"))
{
resJson = crypto.Decrypt(encResJson, workingKey);
}
else if (!"".Equals(status) && status.Equals("1"))
{
Console.WriteLine("failure response: " + encResJson);
}
}
return resJson;
}
private async Task<string> PostPaymentRequestToGateway(string queryUrl, string urlParam)
{
string message = "";
try
{
StreamWriter myWriter = null;// it will open a http connection with provided url
WebRequest objRequest = WebRequest.Create(queryUrl);//send data using objxmlhttp object
objRequest.Method = "POST";
//objRequest.ContentLength = TranRequest.Length;
objRequest.ContentType = "application/x-www-form-urlencoded";//to set content type
myWriter = new System.IO.StreamWriter(objRequest.GetRequestStream());
myWriter.Write(urlParam);//send data
myWriter.Close();//closed the myWriter object
// Getting Response
System.Net.HttpWebResponse objResponse = (System.Net.HttpWebResponse)objRequest.GetResponse();//receive the responce from objxmlhttp object
using (System.IO.StreamReader sr = new System.IO.StreamReader(objResponse.GetResponseStream()))
{
message = await sr.ReadToEndAsync();
//Response.Write(message);
}
}
catch (Exception exception)
{
Console.Write("Exception occured while connection." + exception);
}
return message;
}
private async Task<NameValueCollection> GetResponseMap(string message)
{
//await Task.Delay(2000); I did this with no Luck
NameValueCollection Params = new NameValueCollection();
if (message != null || !"".Equals(message))
{
string[] segments = message.Split('&');
foreach (string seg in segments)
{
string[] parts = seg.Split('=');
if (parts.Length > 0)
{
string Key = parts[0].Trim();
string Value = parts[1].Trim();
Params.Add(Key, Value);
}
}
}
return await Task.FromResult(Params);
}
Any idea how to fix this? Why its working when i put breakpoint and not otherwise.
Am i doing correct asynchronous implimentsion in my api?

Error during transfering Bytes xamarin forms

I try to upload at least 30 pictures to a web server but my program catch errors.
They appear randomly, sometimes at the first image, sometimes third etc.
{System.Net.Http.HttpRequestException: Error while copying content to a stream. ---> System.IO.IOException: Unable to write data to the transport connection: Connection reset by peer. ---> System.Net.Sockets.SocketException: Connection reset by peer
at System.Net.Sockets.Socket.EndSend (System.IAsyncResult asyncResult) [0x0000c] in /Users/builder/jenkins/workspace/archive-mono/2019-08/android/release/mcs/class/referencesource/System/net/System/Net/Sockets/Socket.cs:3874
at System.Net.Sockets.NetworkStream.EndWrite (System.IAsyncResult asyncResult) [0x00057] in /Users/builder/jenkins/workspace/archive-mono/2019-08/android/release/mcs/class/referencesource/System/net/System/Net/Sockets/NetworkStream.cs:1043
--- End of inner exception stack trace ---
OR
Read error: ssl=0x7e46332e08: I/O error during system call, Connection reset by peer
I explain you the process : I got an array with image path and I created a loop to upload each image on my web server.
public async void LaunchImagesUpload()
{
for (int i = startIndex; i < picturesToDisplay.Count; i++)
{
try
{
await UploadSinglePicture(i);
}
catch (Exception e)
{
if(e.Message.Equals("The operation was canceled."))
{
hasErrors = true;
}
else if(e.Message.Equals("Error while copying content to a stream."))
{
// Sometimes program goes here
hasErrors = true;
}
else
{
// Sometimes program goes here
currentPictureGrid.ActivateCancelledMode("error , retry please");
hasErrors = true;
}
}
}
}
Here is the http call
public static async Task<string> CreateUploadTask(string file, string id, string user_id, string user_login)
{
string requestResult = "";
var cont = new MultipartFormDataContent();
var image = new StreamContent(File.OpenRead(file));
image.Headers.ContentDisposition = new ContentDispositionHeaderValue("form-data") { Name = "file", FileName = "imageToUpload.jpeg" };
image.Headers.ContentType = new MediaTypeHeaderValue("image/jpeg");
cont.Add(image);
string id_level = id
string uri = // private uri
using (var client = new HttpClient())
{
var response = await client.PostAsync(uri, cont);
if (response.StatusCode != System.Net.HttpStatusCode.OK)
{
return "error";
}
requestResult = response.Content.ReadAsStringAsync().Result;
Console.WriteLine("DATA FINAL" + requestResult);
}
return requestResult;
}
How can I upload multiple image. Do I made a mistake. There is another way to uplad pictures thank's in advance
UPDATE
I used breakpoint to see when appear error and I added a handler.HttpSendProgress to see the transfer and error appear during transfer
Here is the nuew CreateUploadTask function
public static async Task<string> CreateUploadTask(string file, string vtour_id, string user_id, string user_login)
{
string requestResult = "";
HttpContent fileStreamContent = new StreamContent(File.OpenRead(file));
await Task.Delay(500);
fileStreamContent.Headers.ContentDisposition = new ContentDispositionHeaderValue("form-data") { Name = "file", FileName = "imageToUpload.jpeg" };
fileStreamContent.Headers.ContentType = new MediaTypeHeaderValue("image/jpeg");
string boundary = "---8393774hhy37373773";
using (var formData = new MultipartFormDataContent(boundary))
{
handler.HttpSendProgress += (s, e) =>
{
// error is here because I display transfer and not all Bytes are transfered
float prog = (float)e.BytesTransferred / (float)File.OpenRead(file).Length;
prog = prog > 1 ? 1 : prog;
if (prog > .99)
{
currentPictureGrid.ActivateTreatmentMode();
Console.WriteLine("IMAGE IMPORT --------- Transfert -------- Mode 'en cours de traitement' activé");
}
else
{
Console.WriteLine("step 0");
byteTransferedBeforeBug = (float)e.BytesTransferred;
byteTotalBeforeBug = (float)File.OpenRead(file).Length;
currentPictureGrid.UpdateUploadRate((float)e.BytesTransferred, (float)File.OpenRead(file).Length);
}
};
formData.Add(fileStreamContent);
string id_level = (jsoncontent != null && jsoncontent.GetValue(currentPath.ToString())["floor"].ToString() != null ? jsoncontent.GetValue(currentPath.ToString())["floor"].ToString() : "0");
var response = await client.PostAsync(MY_PRIVATE_URL, formData);
requestResult = response.Content.ReadAsStringAsync().Result.ToString();
Console.WriteLine("DATA RESULT : " + requestResult);
}
return requestResult;
}

HttpClient in while loop only executed once

I am trying to call HttpClient request inside for loop as follows. It needs to do multiple consecutive calls to third party rest api.
But it only gives me fist service call result while loop exit before getting result from rest of the service call.
private void Search()
{
try
{
var i = 1;
using (var httpClient = new HttpClient())
{
while (i < 5)
{
string url = "https://jsonplaceholder.typicode.com/posts/" + i;
var response = httpClient.GetAsync(url).Result;
string jsonResult = response.Content.ReadAsStringAsync().Result;
Console.WriteLine(jsonResult.ToString());
i++;
}
}
}
catch (Exception ex)
{
Console.WriteLine(ex.ToString());
}
}
When I run with debug points the program gives me all the result. But when I run it without debug points it gives me only the first result.
I tried this with using async, await methods too. It also gives me same result.
As I feel Program needs to wait until the async call returns data.
Please help me to solve this.
EDIT - async way
private async Task<string> SearchNew()
{
try
{
var i = 1;
var res = string.Empty;
using (var httpClient = new HttpClient())
{
while (i < 5)
{
string url = "https://jsonplaceholder.typicode.com/posts/" + i;
var response = httpClient.GetAsync(url).Result;
string jsonResult = await response.Content.ReadAsStringAsync();
res = res + jsonResult + " --- ";
i++;
}
}
return res;
}
catch (Exception ex)
{
return ex.Message;
}
}
Both are giving same result.
There's a few things here that you should be doing. First, move the HttpClient creation outside of your method and make it static. You only need one of them and having multiple can be really bad for stability (see here):
private static HttpClient _client = new HttpClient();
Next, extract the calls to the HttpClient into a single method, something simple like this:
//Please choose a better name than this
private async Task<string> GetData(string url)
{
var response = await _client.GetAsync(url);
return await response.Content.ReadAsStringAsync();
}
And finally, you create a list of tasks and wait for them all to complete asynchronously using Task.WhenAll:
private async Task<string[]> SearchAsync()
{
var i = 1;
var tasks = new List<Task<string>>();
//Create the tasks
while (i < 5)
{
string url = "https://jsonplaceholder.typicode.com/posts/" + i;
tasks.Add(GetData(url));
i++;
}
//Wait for the tasks to complete and return
return await Task.WhenAll(tasks);
}
And to call this method:
var results = await SearchAsync();
foreach (var result in results)
{
Console.WriteLine(result);
}

How to wait for a callback before returning

I am trying to port a code from .Net to Unity C# and I am stuck on a syntax including a callback.
Basically, I had to replace the .Net 'HttpClient' library by this one, also called 'HttpClient'. But the 'Get' syntax is not the same and uses a Callback. I am quite new to C# and Http queries and don't know how to deal with this syntax to get the expected return.
The original function written in .Net:
internal static JObject GetToBackOffice(string action)
{
var url = backOfficeUrl;
url = url + action;
HttpClient httpClient = new HttpClient();
var response =
httpClient.GetAsync(url
).Result;
if (response.StatusCode == System.Net.HttpStatusCode.OK)
{
var idProcess = Newtonsoft.Json.Linq.JObject.Parse(response.Content.ReadAsStringAsync().Result);
return idProcess;
}
else
return null;
The C# code I am writing for Unity:
internal class Utils
{
internal static JObject GetToBackOffice(string action)
{
var url = backOfficeUrl;
url = url + action;
HttpClient httpClient = new HttpClient();
JObject idProcess = new JObject();
httpClient.GetString(new Uri(url),
(response) =>
{
// Raised when the download completes
if (response.StatusCode == System.Net.HttpStatusCode.OK)
{
idProcess = Newtonsoft.Json.Linq.JObject.Parse(response.Data);
}
else
{
idProcess = null;
}
});
// Here I would like to wait for the response, so that idProcess is filled with the received data before returning
return idProcess;
}
public class Action
{
public bool SendData(string id, string secretKey, FakeData data)
{
var idProcess = Utils.GetToBackOffice(String.Format("Events/{0}/infos",id));
//...I do then something with idProcess
//Currently, when I use idProcess here, it is still empty since the GetString response hasn't been received yet when this line is executed
return true;
}
}
public class EventHubSimulator : MonoBehaviour
{
void Start()
{
//Fill the parameters (I skip the details)
string oId = ...;
string secretKey = ...;
var vh = ...;
Action action = new Action();
action.SendData(oId, secretKey, vh);
}
}
My issue is that after the GetToBackOffice function, my code directly uses 'idProcess' for something else but this object is empty because the response was not received yet. I would like to wait for the response before my function returns.
I hope I was clear enough. I know that similar question have already been posted but couldn't find a solution to my specific issue.
Edit:
Finally I used a coroutine as Nain suggested but couldn't get what I expected the way he said. This way seems to work (event though it might not be a good way to do it).
public class EventHubSimulator : MonoBehaviour
{
void Start()
{
//Fill the parameters (I skip the details)
string oId = ...;
string secretKey = ...;
var vh = ...;
Utils utils = new Utils();
StartCoroutine(utils.SendData(oId, secretKey, vh));
}
}
public class Utils: MonoBehaviour
{
private const string backOfficeUrl = "http://myurl/api/";
public CI.HttpClient.HttpResponseMessage<string> response;
public IEnumerator SendData(string id, string secretKey, FakeData data)
{
response = null;
yield return GetToBackOffice(String.Format("Events/{0}/infos", id)); //Make a Http Get request
//The next lines are executed once the response has been received
//Do something with response
Foo(response);
}
IEnumerator GetToBackOffice(string action)
{
var url = backOfficeUrl;
url = url + action;
//Make a Http Get request
HttpClient httpClient = new HttpClient();
httpClient.GetString(new Uri(url), (r) =>
{
// Raised when the download completes
if (r.StatusCode == System.Net.HttpStatusCode.OK)
{
//Once the response has been received, write it in the global variable
response = r;
Debug.Log("Response received : " + response);
}
else
{
Debug.Log("ERROR =============================================");
Debug.Log(r.ReasonPhrase);
throw new Exception(r.ReasonPhrase);
}
});
//Wait for the response to be received
yield return WaitForResponse();
Debug.Log("GetToBackOffice coroutine end ");
}
IEnumerator WaitForResponse()
{
Debug.Log("WaitForResponse Coroutine started");
//Wait for response to become be assigned
while (response == null)
{
yield return new WaitForSeconds(0.02f);
}
Debug.Log("WaitForResponse Coroutine ended");
}
}
One solution is polling for completion as Nain submitted as an answer. If you don't want polling you can use a TaskCompletionSource. This Q&A dives a bit deeper into the why and how.
Your code can then be written like this:
async Task CallerMethod()
{
JObject result = await GetToBackOffice(...);
// Do something with result
}
internal static Task<JObject> GetToBackOffice(string action)
{
var tsc = new TaskCompletionSource<JObject>();
var url = backOfficeUrl;
url = url + action;
HttpClient httpClient = new HttpClient();
JObject idProcess = new JObject();
httpClient.GetString(new Uri(url),
(response) =>
{
// Raised when the download completes
if (response.StatusCode == System.Net.HttpStatusCode.OK)
{
tsc.SetResult(Newtonsoft.Json.Linq.JObject.Parse(response.Data));
}
else
{
tsc.SetResult(null);
}
});
return tsc.Task;
}
See also the section Async Method Calls a Coroutine And Wait for Completion of this msdn blogpost.
NOTE Tasks, and async/await support is only available as beta functionality in Unity. See also this post.
Write a coroutine like
//Class scope variable is neede to hold Response other wise it will
//be destroied as soon as function is ended
ResponseType response;
IEnumerator WaitForResponce(ResponseType response)
{
this.response = response;
while(this.response.Data == null)
yield return new WaitForSeconds (0.02f);
//do what you want here
}
and call the coroutine
httpClient.GetString(new Uri(url),
(response) =>
{
// Raised when the download completes
if (response.StatusCode == System.Net.HttpStatusCode.OK)
{
//idProcess = Newtonsoft.Json.Linq.JObject.Parse(response.Data);
StartCoroutine(WaitForResponse(response));
}
else
{
idProcess = null;
}
});
If the GetString method is returning a Task object then you can use the Wait() method to let the Task finished processing before a result is returned.
var task = httpClient.GetString({
impl here..
});
task.Wait();
return idProcess;

How to wait for multiple async http request

I'd like to ask about how to wait for multiple async http requests.
My code is like this :
public void Convert(XDocument input, out XDocument output)
{
var ns = input.Root.Name.Namespace;
foreach (var element in input.Root.Descendants(ns + "a"))
{
Uri uri = new Uri((string)element.Attribute("href"));
var wc = new WebClient();
wc.OpenReadCompleted += ((sender, e) =>
{
element.Attribute("href").Value = e.Result.ToString();
}
);
wc.OpenReadAsync(uri);
}
//I'd like to wait here until above async requests are all completed.
output = input;
}
Dose anyone know a solution for this?
There is an article by Scott Hanselman in which he describes how to do non blocking requests. Scrolling to the end of it, there is a public Task<bool> ValidateUrlAsync(string url) method.
You could modify it like this (could be more robust about response reading)
public Task<string> GetAsync(string url)
{
var tcs = new TaskCompletionSource<string>();
var request = (HttpWebRequest)WebRequest.Create(url);
try
{
request.BeginGetResponse(iar =>
{
HttpWebResponse response = null;
try
{
response = (HttpWebResponse)request.EndGetResponse(iar);
using(var reader = new StreamReader(response.GetResponseStream()))
{
tcs.SetResult(reader.ReadToEnd());
}
}
catch(Exception exc) { tcs.SetException(exc); }
finally { if (response != null) response.Close(); }
}, null);
}
catch(Exception exc) { tcs.SetException(exc); }
return tsc.Task;
}
So with this in hand, you could then use it like this
var urls=new[]{"url1","url2"};
var tasks = urls.Select(GetAsync).ToArray();
var completed = Task.Factory.ContinueWhenAll(tasks,
completedTasks =>{
foreach(var result in completedTasks.Select(t=>t.Result))
{
Console.WriteLine(result);
}
});
completed.Wait();
//anything that follows gets executed after all urls have finished downloading
Hope this puts you in the right direction.
PS. this is probably as clear as it can get without using async/await
Consider using continuation passing style. If you can restructure your Convert method like this,
public void ConvertAndContinueWith(XDocument input, Action<XDocument> continueWith)
{
var ns = input.Root.Name.Namespace;
var elements = input.Root.Descendants(ns + "a");
int incompleteCount = input.Root.Descendants(ns + "a").Count;
foreach (var element in elements)
{
Uri uri = new Uri((string)element.Attribute("href"));
var wc = new WebClient();
wc.OpenReadCompleted += ((sender, e) =>
{
element.Attribute("href").Value = e.Result.ToString();
if (interlocked.Decrement(ref incompleteCount) == 0)
// This is the final callback, so we can continue executing.
continueWith(input);
}
);
wc.OpenReadAsync(uri);
}
}
You then run that code like this:
XDocument doc = something;
ConvertAndContinueWith(doc, (finishedDocument) => {
// send the completed document to the web client, or whatever you need to do
});

Categories