We Keep Coding

SSIS Script Task using client.GetAsync(url) not waiting for response

I have an API call using client.GetAsync(url) within a SSIS script task but for some reason its not waiting for response from API and jumping back to the entry point for the script task which is public void Main(). Done some reading and found out that it might result in a deadlock for some reason but tried all the variations I could find to get it to work but with no luck. Something else that I don't understand is the exact same code is running on a webpage and that works perfect and waits for response from the api and continuing the flow.
Script Task entry point
The response here for payload is: ID =5, Status = WaitingForActivation, Method = "{null}", Result = "{Not yet computed}"
Here if in debug mode and moving the process back to go through the process again I noticed there 2 threads one current executing and the old one with the response I was expecting on the first call but not too sure what this means.
public void Main() {
// TODO: Add your code here
try {
PackageDetails packageInfo = new PackageDetails {
PackageNumber = 1234567891, Env = "Development", UserName = "USER"
};
var payload = API.ListHeadByPackAsync(packageInfo);
//var test = GetResponse();
Dts.TaskResult = (int) ScriptResults.Success;
} catch (Exception ex) {
System.Console.Write(ex.Message);
Dts.TaskResult = (int) ScriptResults.Failure;
}
}
API Call
public static class API {
public static async Task<PackageDetails> ListHeadByPackAsync(PackageDetails package) {
PackageDetails packageInfo = new PackageDetails();
try {
using(var client = new ApiClient(requestUrl, authToken)) {
var response = await client.GetAsync(); //-> not waiting for response
}
} catch (Exception err) {
switch (err.Message) {
//TODO:
}
}
return packageInfo;
}
}
Client
public class ApiClient: IDisposable {
private readonly TimeSpan _timeout;
private HttpClient _httpClient;
private HttpClientHandler _httpClientHandler;
private readonly string _baseUrl;
private readonly string _credentials;
//private const string MediaTypeXml = "application/csv";
public ApiClient(string baseUrl, string authToken, TimeSpan ? timeout = null) {
_baseUrl = baseUrl;
_credentials = Base64Encode(authToken);
_timeout = timeout ?? TimeSpan.FromSeconds(90);
}
public async Task < string > GetAsync() {
EnsureHttpClientCreated();
using(var response = await _httpClient.GetAsync(_baseUrl).ConfigureAwait(continueOnCapturedContext: false))
//-> after executing above line it will go straight to public void Main(), dose not wait for response
{
response.EnsureSuccessStatusCode();
return await response.Content.ReadAsStringAsync();
}
}
public void Dispose() {
_httpClientHandler?.Dispose();
_httpClient?.Dispose();
}
private void CreateHttpClient() {
_httpClientHandler = new HttpClientHandler {
AutomaticDecompression = DecompressionMethods.Deflate | DecompressionMethods.GZip
};
_httpClient = new HttpClient(_httpClientHandler, false) {
Timeout = _timeout
};
if (!string.IsNullOrWhiteSpace(_baseUrl)) {
_httpClient.BaseAddress = new Uri(_baseUrl);
}
_httpClient.DefaultRequestHeaders.Add("Authorization", "Basic" + " " + _credentials);
}
private void EnsureHttpClientCreated() {
if (_httpClient == null) {
//ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12;
CreateHttpClient();
}
}
public static string Base64Encode(string token) {
var tokenBytes = System.Text.Encoding.UTF8.GetBytes(token);
return System.Convert.ToBase64String(tokenBytes);
}
}

UWP HTTP request will produce a delay time in receiving response?

I'm writing the function of HTTP request for my UWP project. And I accidentally got the data last night ,which shows me that it takes longer to make a HTTP request with HTTP Client on UWP project than the same operation on Android
As for seraching the answer,I found out another similar question:
UWP http client delay in getting response
So,My question is: What make this delay happen ? Just for the diff of application framework or others ?
Here is my HTTP request codes on UWP:
using ServerMonitor.Controls;
using System;
using System.Diagnostics;
using System.Net.Http;
using System.Threading;
using System.Threading.Tasks;
namespace ServerMonitor.Services.RequestServices
{
public class HTTPRequest : BasicRequest, IRequest
{
private const short HTTPPORT = 80;
private const short HTTPSPORT = 443;
private string requestInfo = null;
public static HTTPRequest Instance
{
get
{
return Nested.instance;
}
}
private TransportProtocol httpOrhttps = TransportProtocol.http;
public string Uri { set => uri = value; }
public TransportProtocol ProtocolType { set => httpOrhttps = value; }
public string RequestInfo { get => requestInfo; }
private HTTPRequest() { }
private async Task<bool> HttpRequest(string uri)
{
Stopwatch stopwatch = new Stopwatch();
try
{
HttpClientHandler handler = new HttpClientHandler
{
AllowAutoRedirect = true
};
using (HttpClient client = new HttpClient(handler))
{
client.DefaultRequestHeaders.Referrer = new Uri(uri);
client.Timeout = TimeSpan.FromSeconds(OverTime);
CancellationTokenSource cts = new CancellationTokenSource();
cts.CancelAfter(TimeSpan.FromSeconds(OverTime));
HttpResponseMessage message = null;
stopwatch.Start();
Task queryTask = Task.Run(async() =>
{
message = await client.GetAsync(uri, cts.Token);
stopwatch.Stop();
});
var ranTask = Task.WaitAny(queryTask, Task.Delay(OverTime));
if (0 != ranTask)
{
stopwatch.Stop();
if (!cts.IsCancellationRequested) {
cts.Cancel();
}
TimeCost = OverTime;
Status = "1002";
Exception e = new TaskCanceledException("Overtime");
requestInfo = e.ToString();
ErrorException = e;
return false;
}
if (null == message)
{
Status = "500";
TimeCost = (int)(OverTime * 1.5);
requestInfo = "Failed Request : Response Message Is Null";
}
else {
TimeCost = (int)stopwatch.ElapsedMilliseconds;
Status = ((int)Enum.Parse(typeof(System.Net.HttpStatusCode), message.StatusCode.ToString())).ToString();
requestInfo = string.Format("{0} in {1}ms",message.StatusCode, stopwatch.ElapsedMilliseconds);
Debug.WriteLine(requestInfo);
}
}
await Task.CompletedTask;
return true;
}
catch (TaskCanceledException e)
{
Debug.WriteLine("请求超时");
DBHelper.InsertErrorLog(e);
TimeCost = OverTime;
Status = "1002";
ErrorException = e;
requestInfo = "Request OverTime !";
return false;
}
catch (OperationCanceledException e)
{
Debug.WriteLine("请求失败" + e.Message);
DBHelper.InsertErrorLog(e);
TimeCost = (int)(OverTime*1.5);
ErrorException = e;
Status = "1002";
requestInfo = e.Message;
return false;
}
catch (Exception e)
{
Debug.WriteLine("请求失败" + e.Message);
DBHelper.InsertErrorLog(e);
TimeCost = (int)(OverTime * 1.5);
ErrorException = e;
Status = "1001";
requestInfo = e.Message;
return false;
}
}
public async Task<bool> MakeRequest()
{
bool result = false;
switch (httpOrhttps)
{
case TransportProtocol.http:
result = await HttpRequest(uri);
return result;
default:
return result;
}
}
private class Nested
{
static Nested()
{
}
internal static readonly HTTPRequest instance = new HTTPRequest();
}
}
public enum TransportProtocol
{
http,
https
};
}
And followed are some of my screen shots:
Request from uwp application:
Request on Android application:
My UWP environment:
Windows 10.16299 SDK + Framework 4.6 + Visual Studio 2017

C# HttpWebRequest.BeginGetResponse blocks in one class but not another

I am validating a list of proxies using HttpWebRequest.BeginGetResponse. It works really well, I can validate thousands of proxies in seconds and doesn't block.
In another class within my project, I am calling the same code and it blocks.
Proxy validation method (Doesn't block):
public void BeginTest(IProxyTest test, Action<ProxyStatus> callback, int timeout = 10000)
{
var req = HttpWebRequest.Create(test.URL);
req.Proxy = new WebProxy(this.ToString());
req.Timeout = timeout;
WebHelper.BeginGetResponse(req, new Action<RequestCallbackState>(callbackState =>
{
if (callbackState.Exception != null)
{
callback(ProxyStatus.Invalid);
}
else
{
var responseStream = callbackState.ResponseStream;
using (var reader = new StreamReader(responseStream))
{
var responseString = reader.ReadToEnd();
if (responseString.Contains(test.Validation))
{
callback(ProxyStatus.Valid);
}
else
{
callback(ProxyStatus.Invalid);
}
}
}
}));
}
WebHelper.BeginGetResponse
public static void BeginGetResponse(WebRequest request, Action<RequestCallbackState> responseCallback)
{
Task<WebResponse> asyncTask = Task.Factory.FromAsync<WebResponse>(request.BeginGetResponse, request.EndGetResponse, null);
ThreadPool.RegisterWaitForSingleObject((asyncTask as IAsyncResult).AsyncWaitHandle, new WaitOrTimerCallback(TimeoutCallback), request, request.Timeout, true);
asyncTask.ContinueWith(task =>
{
WebResponse response = task.Result;
Stream responseStream = response.GetResponseStream();
responseCallback(new RequestCallbackState(responseStream));
responseStream.Close();
response.Close();
}, TaskContinuationOptions.NotOnFaulted);
//Handle errors
asyncTask.ContinueWith(task =>
{
var exception = task.Exception;
responseCallback(new RequestCallbackState(exception.InnerException));
}, TaskContinuationOptions.OnlyOnFaulted);
}
Other class with a similar method that also calls WebHelper.BeginGetResponse, but blocks (why?)
public void BeginTest(Action<ProxyStatus> callback, int timeout = 10000)
{
var req = HttpWebRequest.Create(URL);
req.Timeout = timeout;
WebHelper.BeginGetResponse(req, new Action<RequestCallbackState>(callbackState =>
{
if (callbackState.Exception != null)
{
callback(ProxyStatus.Invalid);
}
else
{
var responseStream = callbackState.ResponseStream;
using (var reader = new StreamReader(responseStream))
{
var responseString = reader.ReadToEnd();
if (responseString.Contains(Validation))
{
callback(ProxyStatus.Valid);
}
else
{
callback(ProxyStatus.Invalid);
}
}
}
}));
}
Calling code which blocks
private async void validateTestsButton_Click(object sender, EventArgs e)
{
await Task.Run(() =>
{
foreach (var test in tests)
{
test.BeginTest((status) => test.Status = status);
}
});
}
Calling code which doesn't block:
public static async Task BeginTests(ICollection<Proxy> proxies, ICollection<ProxyJudge> judges, int timeout = 10000, IProgress<int> progress = null)
{
await Task.Run(() =>
{
foreach (var proxy in proxies)
{
proxy.BeginTest(judges.GetRandomItem(), new Action<ProxyStatus>(status =>
{
proxy.Status = status;
}), timeout);
}
});
}

Although this dosnt address your problem exactly, it might help you out a little
Here are a couple of problems
You are using APM (Asynchronous Programming Model)
You are using the ThreadPool class which seems a little old fashioned
You are doing IO bound work and blocking threads on the threadpool
You are using a weird mix of APM and TBA asynchronous models
And seemingly tying up your thread pool waiting for IO
So you are doing IO bound work, the best pattern to use as you might have guess is the TBA async await pattern. basically every time you wait for a an IO Completion port you want to give that thread back to the operating system and be nice to your system inturn freeing up resources for where its needed.
Also you obviously want some degree of parallelism and you are best to at least have some control over it.
I would suggest this is a nice job for TPL Dataflow and an ActionBlock
Given
public class Proxy
{
public ProxyStatus ProxyStatus { get; set; }
public string ProxyUrl { get; set; }
public string WebbUrl { get; set; }
public string Error { get; set; }
}
ActionBlock Example
public static async Task DoWorkLoads(List<Proxy> results)
{
var options = new ExecutionDataflowBlockOptions
{
MaxDegreeOfParallelism = 50
};
var block = new ActionBlock<Proxy>(CheckUrlAsync, options);
foreach (var proxy in results)
{
block.Post(proxy);
}
block.Complete();
await block.Completion;
}
CheckUrlAsync Example
// note i havent tested this, add pepper and salt to taste
public static async Task CheckUrlAsync(Proxy proxy)
{
try
{
var request = WebRequest.Create(proxy.Url);
if (proxy.ProxyUrl != null)
request.Proxy = new WebProxy(proxy.ProxyUrl);
using (var response = await request.GetResponseAsync())
{
using (var responseStream = response.GetResponseStream())
{
using (var reader = new StreamReader(responseStream))
{
var responseString = reader.ReadToEnd();
if (responseString.Contains("asdasd"))
proxy.ProxyStatus = ProxyStatus.Valid;
else
proxy.ProxyStatus = ProxyStatus.Invalid;
}
}
}
}
catch (Exception e)
{
proxy.ProxyStatus = ProxyStatus.Error;
proxy.Error = e.Message;
}
}
Usage
await DoWorkLoads(proxies to test);
Summary
The code is neater, you arnt throwing actions all over the place, you using async and await you have ditched APM, you have control of the degrees of parallel and you are being nice to the thread pool

I solved the problem by wrapping the code in the BeginTest method which was blocking mysteriously in an Action, and then calling BeginInvoke on that Action.
I deduced this was caused by not setting the Proxy property on the HttpWebRequest in that method, which seemed to be causing a synchronous lookup of my systems proxy.
public void BeginTest(Action<ProxyStatus> callback, int timeout = 10000)
{
var action = new Action(() =>
{
var req = HttpWebRequest.Create(URL);
req.Timeout = timeout;
WebHelper.BeginGetResponse(req, new Action<RequestCallbackState>(callbackState =>
{
if (callbackState.Exception != null)
{
callback(ProxyStatus.Invalid);
}
else
{
var responseStream = callbackState.ResponseStream;
using (var reader = new StreamReader(responseStream))
{
var responseString = reader.ReadToEnd();
if (responseString.Contains(Validation))
{
callback(ProxyStatus.Valid);
}
else
{
callback(ProxyStatus.Invalid);
}
}
}
}));
});
action.BeginInvoke(null, null);
}

BeginGetContext performance

I saw a lot of examples about GeginGetContext but i have filling that all of them are waste time. Maybe i am wrong. Lets find out. Lets take for instance really good example from the Multi-threading with .Net HttpListener topic:
public class HttpListenerCallbackState
{
private readonly HttpListener _listener;
private readonly AutoResetEvent _listenForNextRequest;
public HttpListenerCallbackState(HttpListener listener)
{
if (listener == null) throw new ArgumentNullException("listener");
_listener = listener;
_listenForNextRequest = new AutoResetEvent(false);
}
public HttpListener Listener { get { return _listener; } }
public AutoResetEvent ListenForNextRequest { get { return _listenForNextRequest; } }
}
public class HttpRequestHandler
{
private int requestCounter = 0;
private ManualResetEvent stopEvent = new ManualResetEvent(false);
public void ListenAsynchronously(IEnumerable<string> prefixes)
{
HttpListener listener = new HttpListener();
foreach (string s in prefixes)
{
listener.Prefixes.Add(s);
}
listener.Start();
HttpListenerCallbackState state = new HttpListenerCallbackState(listener);
ThreadPool.QueueUserWorkItem(Listen, state);
}
public void StopListening()
{
stopEvent.Set();
}
private void Listen(object state)
{
HttpListenerCallbackState callbackState = (HttpListenerCallbackState)state;
while (callbackState.Listener.IsListening)
{
callbackState.Listener.BeginGetContext(new AsyncCallback(ListenerCallback), callbackState);
int n = WaitHandle.WaitAny(new WaitHandle[] { callbackState.ListenForNextRequest, stopEvent});
if (n == 1)
{
// stopEvent was signalled
callbackState.Listener.Stop();
break;
}
}
}
private void ListenerCallback(IAsyncResult ar)
{
HttpListenerCallbackState callbackState = (HttpListenerCallbackState)ar.AsyncState;
HttpListenerContext context = null;
int requestNumber = Interlocked.Increment(ref requestCounter);
try
{
context = callbackState.Listener.EndGetContext(ar);
}
catch (Exception ex)
{
return;
}
finally
{
callbackState.ListenForNextRequest.Set();
}
if (context == null) return;
HttpListenerRequest request = context.Request;
if (request.HasEntityBody)
{
using (System.IO.StreamReader sr = new System.IO.StreamReader(request.InputStream, request.ContentEncoding))
{
string requestData = sr.ReadToEnd();
//Stuff I do with the request happens here
}
}
try
{
using (HttpListenerResponse response = context.Response)
{
//response stuff happens here
string responseString = "Ok";
byte[] buffer = Encoding.UTF8.GetBytes(responseString);
response.ContentLength64 = buffer.LongLength;
response.OutputStream.Write(buffer, 0, buffer.Length);
response.Close();
}
}
catch (Exception e)
{
}
}
}
Here we can see main part for this topic:
while (callbackState.Listener.IsListening)
{
callbackState.Listener.BeginGetContext(new AsyncCallback(ListenerCallback), callbackState);
int n = WaitHandle.WaitAny(new WaitHandle[] { callbackState.ListenForNextRequest, stopEvent});
...
}
I can saw this patterns in +- all examples.
We are starting Getting Contex (Getting request = getting network stream with request) after that we are blocking current thread with Wait/WaitAny method, so thread witch is getting request is doing nothing until it will got full request after that it will getting new request. For example we are having WCF
request with large object(which are deserialized in this request and are serialized in the same way on other side) so we will wait and WASTE TIME until we complete getting full steam with this request.
I see that really it is Sync not Async way. Instead of this we can starting getting second request
right after the starting getting first and not call Wait, not blocking thread. I think we will have much better performance in such way. What do you think? Why do all examples contains Wait?

ServerSentEvents, HttpTaskAsyncHandler and waiting

I'm trying to create a quite simple notifications system (don't want to use SignalIR or something else). I have the following testing code:
Client side:
var source = new EventSource('/notifications.axd');
source.onopen = function () {
Console.log("Connection open");
};
source.onerror = function () {
Console.log("Connection error");
};
source.onmessage = function (event) {
Console.log("Message: " + event.data);
};
Server side:
public class NotificationMessage {
public NotificationMessage() {
Id = Guid.NewGuid().ToString();
}
public string Id { get; private set; }
}
public class NotificationsHandler : HttpTaskAsyncHandler {
private const string CONTENT_TYPE = "text/event-stream";
private sealed class NotificationItem {
public ConcurrentQueue<NotificationMessage> Messages;
public CancellationTokenSource CancellationTokenSource;
}
private static readonly ConcurrentDictionary<string, NotificationItem> _tasks =
new ConcurrentDictionary<string, NotificationItem>();
public static void Notify(string hostId, string userId, NotificationMessage message) {
NotificationItem item;
if (!_tasks.TryGetValue(string.Format("{0}|{1}", hostId, userId), out item)) {
return;
}
var tokenSource = item.CancellationTokenSource;
item.Messages.Enqueue(message);
item.CancellationTokenSource = new CancellationTokenSource();
tokenSource.Cancel();
}
public override async Task ProcessRequestAsync(HttpContext context) {
HttpRequest request = context.Request;
NotificationItem item = _tasks.GetOrAdd(
string.Format("{0}|{1}", request.Url.Host, CsSession.Data.CurrentUser.Id),
k => new NotificationItem {
Messages = new ConcurrentQueue<NotificationMessage>(),
CancellationTokenSource = new CancellationTokenSource()
}
);
HttpResponse response = context.Response;
response.ContentType = CONTENT_TYPE;
response.CacheControl = "no-cache";
response.ContentEncoding = Encoding.UTF8;
response.AppendHeader("connection", "keep-alive");
response.BufferOutput = false;
bool supportsAsyncFlush = response.SupportsAsyncFlush;
bool shouldPing = true;
while (response.IsClientConnected) {
try {
NotificationMessage message = null;
if ((!item.Messages.IsEmpty && item.Messages.TryDequeue(out message)) || shouldPing) {
response.Write(string.Format("data:{0}\n\n", message == null ? "{}" : JsonMapper.Serialize(message)));
if (supportsAsyncFlush) {
await Task.Factory.FromAsync(response.BeginFlush, response.EndFlush, null);
} else {
response.Flush();
}
}
} catch (Exception) {
break;
}
var delay = Task.Delay(15000, item.CancellationTokenSource.Token);
await delay;
shouldPing = delay.Status == TaskStatus.RanToCompletion;
}
}
}
The problem is: the above doesn't works. I have two issues:
1) When the client connects, I receive an empty packet (that's ok). Then, if I don't enqueue any messages, after awaiting the Task.Delay, the loop tries to write an empty message again, but I don't know where. The response.Write line never returns (and nothing is being received on the client).
2) If I write to the queue, for some reason the connection is dropped. If I put a breakpoint on the line after the await delay, that line is never executed (while my logic says otherwise :) ). If I cancel the token, the delay task should quit, but it seems it is aborting the whole handler??