Here is my async download reactive extension for WebClient.
What is the best way to recall "DownloadStringAsync" again and again till the operation succeeds?
Something like this but in reactive way:
while (true)
{
var result = DownloadStringAsync();
if (result)
{
return;
}
}
MY CODE:
[Serializable]
public class WebClientException : Exception
{
public WebClientResponse Response { get; set; }
public WebClientException()
{
}
public WebClientException(string message)
: base(message)
{
}
public WebClientException(string message, Exception innerException)
: base(message, innerException)
{
}
protected WebClientException(SerializationInfo info, StreamingContext context)
: base(info, context)
{
}
}
public class WebClientResponse
{
public WebHeaderCollection Headers { get; set; }
public HttpStatusCode StatusCode { get; set; }
public string Result { get; set; }
public WebException Exception { get; set; }
}
public static IObservable<WebClientResponse> DownloadStringAsync(this WebClient webClient, Uri address, WebHeaderCollection requestHeaders)
{
var asyncResult =
Observable.FromEventPattern<DownloadStringCompletedEventHandler, DownloadStringCompletedEventArgs>
(ev => webClient.DownloadStringCompleted += ev, ev => webClient.DownloadStringCompleted -= ev)
.ObserveOn(Scheduler.TaskPool)
.Select(o =>
{
var ex = o.EventArgs.Error as WebException;
if (ex == null)
{
var wc = (WebClient) o.Sender;
return new WebClientResponse {Headers = wc.ResponseHeaders, Result = o.EventArgs.Result};
}
var wcr = new WebClientResponse {Exception = ex};
var r = ex.Response as HttpWebResponse;
if (r != null)
{
wcr.Headers = r.Headers;
wcr.StatusCode = r.StatusCode;
var s = r.GetResponseStream();
if (s != null)
{
using (TextReader tr = new StreamReader(s))
{
wcr.Result = tr.ReadToEnd();
}
}
}
throw new WebClientException {Response = wcr};
})
.Take(1);
if (requestHeaders != null)
{
foreach (var key in requestHeaders.AllKeys)
{
webClient.Headers.Add(key, requestHeaders[key]);
}
}
webClient.DownloadStringAsync(address);
return asyncResult;
}
Your method produces a hot observable, which means that it has already started loading when it returns and each new subscription does not create a new request to the web server. You need to wrap your method in another and use Observable.Create (in order to create a cold observable which does create a new request upon each subscription):
public static IObservable<WebClientResponse> DownloadStringAsync(this WebClient webClient, Uri address, WebHeaderCollection requestHeaders)
{
return Observable
.Create(observer =>
{
DownloadStringAsyncImpl(webClient, address, requestHeaders)
.Subscribe(observer);
return () => { webClient.CancelAsync(); };
});
}
Here, DownloadStringAsyncImpl is your previous implementation of DownloadStringAsync, while the public method has been replaced.
Now you can retry the async method until it succeeds as follows:
myWebClient
.DownloadStringAsync( /* args... */)
.Retry()
.Subscribe(result => {
/* now I've got a result! */
});
I think you have at least one decent "here is some code" answer, so I will focus on a more general hand holding.
The first thing I would look at is the design guidelines for Rx. It is a short (34 page) PDF document that helps change paradigm from pull "subscriptions" to push, or moving from IEnumerable to IObservable.
If you want to go a bit further, there are PDF HOLs (hands on labs) for both .NET and JavaScript. You can find other resources on the Rx pages (start here).
If it is an async function. Doing a repetitive checking means you turned it into a sync function call. Is this something you really want to do?
You can have a dedicated thread calling this async function and block itself after calling this function. When create this thread, pass it a delegate that should be called after the async function returns. Upon completion, call the delegate with error code.
Hope this answers your question.
Related
I am trying to understand why my application has the following issues
after running for an hour or so, memory usage is over 2GB
after running for an hour or so event processing slows from ~100m/s to ~6-10m/s
I am using Dotnet 5 with Autofac. Here is the method that is fired by the Consumer_Received event
private async Task ProcessEvent(string eventName, string message)
{
if (_subsManager.HasSubscriptionsForEvent(eventName))
{
await using var scope = _autofac.BeginLifetimeScope(AUTOFAC_SCOPE_NAME);
var subscriptions = _subsManager.GetHandlersForEvent(eventName);
foreach (var subscription in subscriptions)
{
var handler = scope.ResolveOptional(subscription.HandlerType);
if (handler == null) continue;
var eventType = _subsManager.GetEventTypeByName(eventName);
var integrationEvent = JsonConvert.DeserializeObject(message, eventType);
var concreteType = typeof(IIntegrationEventHandler<>).MakeGenericType(eventType);
await Task.Yield();
await ((Task) concreteType.GetMethod("Handle")?
.Invoke(handler, new[] {integrationEvent}))
.ConfigureAwait(false);
}
}
}
This is the event
public class MPTEvent
{
public string M {get;set;}
public string Number {get;set;}
public string Type {get;set;}
public bool T {get;set;}
public bool A {get;set;}
public bool LB {get;set;}
}
The above method is fired for EVERY event stored inside the Queue. i have several "INotification" handlers that process
the events (IE. Handle), here is an example of one (Edited for brievety)
public async Task Handle(MPTEvent tEvent)
{
try
{
// try to get the tag and device
var contextD = await _repository.GetByNumber(tEvent.M);
if (contextD == null)
{
_logger.LogError("Detailed Error Message");
return;
}
var contextT = await _repository.GetByNumber(tEvent.Number);
if (contextT == null)
{
_logger.LogWarning("Detailed Warning Message");
return;
}
if (tEvent.T == 1 && tEvent.A == 0 && tEvent.LB == 0)
{
contextT?.TriggerTEvent(contextD, tEvent.R, tEvent.CreatedDate);
}
await _repository.UpdateAsync(contextT);
}
catch (Exception ex)
{
_logger.LogCritical(ex, "Detailed Exception Message");
}
}
The above handler is fired more than any others and i believe is partly where the biggest issue resides, the questions i have are
Should the handler (in this case) have a public async Task and return something like _repository.UpdateAsync(contextT)?
would it be smart to replace the _repository.UpdateAsync and _repository.AddAsync with mediator commands?
I am relatively new to Autofac/DI/rabbitMq so any advice is greatly appreciated
I'm trying to extend the HttpClient with an EventHandler.
Is this possible?
I have an Extension on HttpClient as follows:
public static class HttpClientExtensions
{
public async static Task<T> GetSomthingSpecialAsync<T>(this HttpClient client, string url)
{
using var response = await client.GetAsync(url);
if (response.StatusCode != System.Net.HttpStatusCode.OK)
{
//I have an error and want to raise the HttpClientEventError
HttpClientErrorEvent(null, new HttpClientErrorEventArgs()
{
StatusCode = response.StatusCode,
Message = $"{response.StatusCode } {(int)response.StatusCode } "
});
return default(T);
}
response.EnsureSuccessStatusCode();
[... ]
}
}
public class HttpClientErrorEventArgs : EventArgs
{
public System.Net.HttpStatusCode StatusCode { get; set; }
public string Message { get; set; }
}
But how do I define the HttpClientErrorEvent?
I tried the following but it is not an extension to a specific HttpClient:
public static event EventHandler<HttpClientErrorEventArgs> HttpClientErrorEvent = delegate { };
Don't use an event to return errors. For starters, how are you going to identify which request raised which error? You'd have to register and unregister event handlers around each call but how would you handle concurrent calls? How would you compose multiple such calls?
Errors aren't events anyway. At best, you'd have to handle the event as if it was a callback - in which case why not use an actual callback?
public async static Task<T> GetSomethingSpecialAsync<T>(this HttpClient client, string url,Action<(HttpStatusCode Status,string Message)> onError)
{
...
if (response.StatusCode != System.Net.HttpStatusCode.OK)
{
onError(response.Status,....);
return default;
}
}
...
var value=await client.GetSomethingSpesialAsync(url,
(status,msg)=>{Console.WriteLine($"Calling {url} Failed with {status}:{msg}");}
);
async/await was created so people can get rid of callbacks and events though. It's almost impossible to compose multiple async calls with events, and hard enough to do so with callbacks. That's why a lot of languages (C#, JavaScript, Dart, even C++ in a way ) introduced promises and async/await to get rid of both the success and error callback.
Instead of calling a callback you can actually return either a result or an error from your function. This is a functional way embedded in eg F#, Rust and Go (through tuples). There are a lot of ways to do this in C#:
Return a tuple with the value and error, eg (T? value, string? error)
Create a record with the value and error
Create separate Success and Error classes that share a common IResult<T> interface
Pattern matching can be used with any option to retrieve either the error or value without a ton of if statements.
Let's say we have a specific error type, HttpError.:
record HttpError(HttpStatusCode Status,string Message);
Using tuples, the method becomes:
public async static Task<(T value,HttpError error> GetSomethingSpecialAsync<T>(this HttpClient client,string url)
{
...
if (response.StatusCode != System.Net.HttpStatusCode.OK)
{
return (default,new HttpError(response.Status,....);
}
}
And called :
var (value,error)=await client.GetSomethingSpecialAsync(url);
if(error!=null)
{
var (status,msg)=error;
Console.WriteLine($"Calling {url} Failed with {status}:{msg}");
...
}
Instead of a tuple, we can create a Result record:
record Result<T>(T? Value,HttpError? Error);
Or separate classes:
interface IResult<T>
{
bool IsSuccess{get;}
}
record Success<T>(T Value):IResult<T>
{
public bool IsSuccess=>true;
}
record Error<T>(HttpError Error):IResult<T>
{
public bool IsSuccess => false;
}
public async static Task<IResult<T>> GetSomethingSpecialAsync<T>(this HttpClient client,string url){...}
var result=await client.GetSomethingSpecialAsync(url);
In all cases pattern matching can be used to simplify handling the result, eg:
var result=await client.GetSomethingSpecialAsync<T>(url);
switch (result)
{
case Error<T> (status,message):
Console.WriteLine($"Calling {url} Failed with {Status}:{Message}");
break;
case Success<T> (value):
...
break;
}
Having a specific Result<T> or IResult<T> type makes it easy to write generic methods to handle success, errors or compose a chain of functions. For example, the following could be used to call the "next" function if the previous one succeeded, otherwise just propagate the "error" :
IResult<T> ThenIfOk(this IResult<T> previous,Func<T,IResult<T>> func)
{
return previous switch
{
Error<T> error=>error,
Success<T> ok=>func(ok.Value)
}
}
This would allow creating a pipeline of calls :
var finalResult=doSomething(url)
.ThenIfOk(value=>somethingElse(value))
.ThenIfOk(....);
This style is called Railway oriented programming and is very common in functional and dataflow (pipeline) programming
You could store the handlers in your extension class and do something like this ? Please note this code is not thread safe and need to be synchronized around dictionary and list access !
public static class HttpClientExtensions
{
private static Dictionary<HttpClient, List<Action<HttpClientErrorEventArgs>>> Handlers { get; set; }
static HttpClientExtensions()
{
Handlers = new Dictionary<HttpClient, List<Action<HttpClientErrorEventArgs>>>();
}
public async static Task<T> GetSomthingSpecialAsync<T>(this HttpClient client, string url)
{
////code ....
//I have an error and want to raise the HttpClientEventError
HttpClientErrorEventArgs args = null;
client.RaiseEvent(args);
return default(T);
////code
}
public static void AddHandler(this HttpClient client, Action<HttpClientErrorEventArgs> handler)
{
var found = Handlers.TryGetValue(client, out var handlers);
if (!found)
{
handlers = new List<Action<HttpClientErrorEventArgs>>();
Handlers[client] = handlers;
}
handlers.Add(handler);
}
public static void RemoveHandler(this HttpClient client, Action<HttpClientErrorEventArgs> handler)
{
var found = Handlers.TryGetValue(client, out var handlers);
if (found)
{
handlers.Remove(handler);
if (handlers.Count == 0)
{
Handlers.Remove(client);
}
}
}
private static void RaiseEvent(this HttpClient client, HttpClientErrorEventArgs args)
{
var found = Handlers.TryGetValue(client, out var handlers);
if (found)
{
foreach (var handler in handlers)
{
handler.Invoke(args);
}
}
}
}
I'm looking for a solution to improve my code.
I have in all my controllers this code:
var data = await _accountService.Create(value);
if (data.HasError)
{
return BadRequest(data);
}
return Ok(data);
I'm looking for a solution where I don't need to do this If to check if HasError Exist.
In all my services I return the following class:
public class ResponseEnvelope<T>
{
public T Result { get; set; }
public IList<ErrorDto> Errors { get; set; }
public IList<MessageDto> Messages { get; set; }
public bool HasError
{
get
{
return Errors != null && Errors.Any();
}
}
public bool HasMessage
{
get
{
return Messages != null && Messages.Any();
}
}
public ResponseEnvelope<T> AddError(string code, string message = null, string description = null)
{
Errors.Add(new ErrorDto(code, message, description));
return this;
}
}
If I get you right and you just invoke a method on servoce and return its response wrapped, then add an extension method to Controller:
public IActionResult WrapResponse<T>(this Controller controller, ResponseEnvelope<T> envelope){
return envelope.HasError ? controller.BadRequest(envelope) : controller.Ok(envelope);
}
and invoke it as this.WrapResponse(data)
.Net Core
You can extract this functionality using ActionFilter or Middleware.
Action Filter has OnActionExecuted - and you can place this logic there.
I would prefer to use Middleware you cand do this after the call next.Invoke()
.Net Framework
If you are using .Net Framework there are a few options like IHttpHandler, IHttpModule and ActionFilter. You can read about modules and handlers there IHttpHandler vs IHttpModule
The best way to archive it was to implement a middleware capable of understanding the response from the API.
Because I return always an Envelope with Data and Errors (if they exist) the middleware was very simple.
Also because I have middleware I was able to implement a single point of log.
public async Task Invoke(HttpContext context)
{
Stream originalBody = context.Response.Body;
using (var memStream = new MemoryStream())
{
context.Response.Body = memStream;
await _next(context);
memStream.Position = 0;
//get response body here after next.Invoke()
var reader = new StreamReader(memStream);
string responseBody = await reader.ReadToEndAsync();
if (!string.IsNullOrEmpty(responseBody) && !responseBody.StartsWith("["))
{
JObject json = JObject.Parse(responseBody);
if (json.ContainsKey("hasError") && (bool)json["hasError"] == true)
{
_logger.LogError($"{_correlationContext.CorrelationContext.CorrelationId} {responseBody}");
context.Response.StatusCode = (int)HttpStatusCode.BadRequest;
}
else
{
if (context.Request.Path.Value.StartsWith("/api/authentication/users"))
{
_logger.LogInformation($"{_correlationContext.CorrelationContext.CorrelationId} Logged");
}
else
{
_logger.LogInformation($"{_correlationContext.CorrelationContext.CorrelationId} {responseBody}");
}
}
}
memStream.Position = 0;
await memStream.CopyToAsync(originalBody);
context.Response.Body = originalBody;
}
I want to write some Unittests with NUnit for our wpf application.
The application downloads some data with System.Net.WebClient in the background using the observer pattern.
Here is an example:
Download.cs
public class Download : IObservable<string>
{
private string url { get; }
private List<IObserver<string>> observers = new List<IObserver<string>>();
private bool closed = false;
private string data = null;
public Download(string url)
{
this.url = url;
startDownload();
}
public IDisposable Subscribe(IObserver<string> observer)
{
if (!observers.Contains(observer))
{
if (!closed)
{
observers.Add(observer);
}
else
{
sendAndComplete(observer);
}
}
return new Unsubscriber(observer, observers);
}
private void startDownload()
{
WebClient client = new WebClient();
client.DownloadStringCompleted += new DownloadStringCompletedEventHandler((object sender, DownloadStringCompletedEventArgs e) => {
if (e.Error != null)
{
data = e.Result;
}
closed = true;
sendAndComplete();
});
client.DownloadStringAsync(new Uri(url));
}
private void sendAndComplete()
{
foreach (var observer in observers)
{
sendAndComplete(observer);
}
observers.Clear();
}
private void sendAndComplete(IObserver<string> observer)
{
if (data != null)
{
observer.OnNext(data);
}
else
{
observer.OnError(new Exception("Download failed!"));
}
observer.OnCompleted();
}
private class Unsubscriber : IDisposable
{
private IObserver<string> _observer { get; }
private List<IObserver<string>> _observers { get; }
public Unsubscriber(IObserver<string> _observer, List<IObserver<string>> _observers)
{
this._observer = _observer;
this._observers = _observers;
}
public void Dispose()
{
if (_observer != null && _observers.Contains(_observer))
{
_observers.Remove(_observer);
}
}
}
}
DownloadInspector.cs
public class DownloadInspector : IObserver<string>
{
private Action<string> onSuccessAction { get; }
private Action<Exception> onErrorAction { get; }
private Action onCompleteAction { get; }
public DownloadInspector(Action<string> onSuccessAction, Action<Exception> onErrorAction, Action onCompleteAction)
{
this.onSuccessAction = onSuccessAction;
this.onErrorAction = onErrorAction;
this.onCompleteAction = onCompleteAction;
}
public void OnCompleted()
{
onCompleteAction.Invoke();
}
public void OnError(Exception error)
{
onErrorAction.Invoke(error);
}
public void OnNext(string value)
{
onSuccessAction.Invoke(value);
}
}
example (usage)
Download download = new Download("http://stackoverflow.com");
DownloadInspector inspector = new DownloadInspector(
(string data) =>
{
Debug.WriteLine("HANDLE DATA");
},
(Exception error) =>
{
Debug.WriteLine("HANDLE ERROR");
},
() =>
{
Debug.WriteLine("HANDLE COMPLETE");
}
);
I'm still new in c# and not very familiar with asynchronous programming in that language. I know the await and async keywords and know that they work with NUnit, but the current construct don't use this keywords.
Can you help me creating a unit test for this case? Its okay to change/remove the observer pattern.
The constructor for the Download class starts the download, which means that I can't subscribe an observer until after the download has started. That's a race condition. It's possible (although unlikely) that observers will be notified before they can be subscribed.
public Download(string url)
{
this.url = url;
startDownload();
}
But I can go ahead and test because I'm subscribing an observer before that can happen. If you can I'd recommend not doing that. Allow the caller to construct the class in one step and then start the download with a method call.
I also had to change this method. I figured that testing for an error would be the easiest first step, but it needs to do data = e.Result if there is no error, not if there is an error.
private void StartDownload()
{
WebClient client = new WebClient();
client.DownloadStringCompleted += new DownloadStringCompletedEventHandler((object sender, DownloadStringCompletedEventArgs e) =>
{
if (e.Error == null) // <== because of this
{
data = e.Result;
}
closed = true;
sendAndComplete();
});
client.DownloadStringAsync(new Uri(url));
}
What I didn't see coming is that WebClient.DownloadStringAsync isn't actually async. It doesn't return a Task. It just takes a callback. What that means is that there's no sure way to know whether it's done except to wait for it to notify the observer that the download is complete.
My NUnit test runner wasn't running, so I used MsTest. It's the same thing.
The basic approach is that I'm creating some flags, and the inspector responds to notifications by setting the flags. That way I can see which notifications were raised.
The last problem is that because DownloadStringComplete is a callback, the test exits before the Assert. That means it will always pass. So in order to fix it I had to do something I've never seen before, which I found here:
[TestMethod]
public void download_raises_error_notification()
{
var success = false;
bool error = false;
bool complete = false;
var pause = new ManualResetEvent(false);
var download = new Download("http://NoSuchUrlAnywhere.com");
var inspector = new DownloadInspector(
onSuccessAction: s => success = true,
onCompleteAction: () =>
{
complete = true;
pause.Set();
},
onErrorAction: s => error = true
);
download.Subscribe(inspector);
// allow 500ms for the download to fail. This is a race condition.
pause.WaitOne(500);
Assert.IsTrue(error,"onErrorAction was not called.");
}
This is technically an integration test since it must actually attempt a download in order to run. That could be remedied by mocking WebClient.
I'm creating a generic loader and I want to kick off an HttpClient SendAsync request. However, some of these requests may take time, so I want to add the option to cancel, and notify upon completion.
This seems like a standard scenario imho.
I'm not sure if this is the correct way to go about this, but based on some examples I've looked at, here is where I'm at. If you look at the bottom of the code, my question is - at that point, do I check the response and raise a success or error event?
public bool StartFetch()
{
if (IsFetching) return false;
IsFetching = true;
mCancellationTokenSource = new CancellationTokenSource();
// this is not awaited, so execution should continue
StartTask(request, mCancellationTokenSource.Token);
return true;
}
public bool CancelFetch()
{
// send cancellation
if (mCancellationTokenSource != null)
mCancellationTokenSource.Cancel();
Cleanup();
return true;
}
private async Task StartTask(LFHttpRequest request, CancellationToken cancellationToken)
{
var message = new HttpRequestMessage(request.Method, request.Uri);
var response = await HttpClient.SendAsync(message, cancellationToken);
// at this point, do I take a look at response and raise a custom OnSuccess or OnError event???
// or do I want to grab the task from `SendAsync`, check for completed or faulted?
}
When you're looking at exposing task-related state like IsFetching, it's often cleaner and easier to just expose the Task itself.
Something like this:
public Task<T> FetchTask { get; private set; }
public bool StartFetch()
{
if (FetchTask != null) return false;
mCancellationTokenSource = new CancellationTokenSource();
FetchTask = FetchAsync(request, mCancellationTokenSource.Token);
return true;
}
public bool CancelFetch()
{
// send cancellation
if (mCancellationTokenSource != null)
mCancellationTokenSource.Cancel();
FetchTask = null;
return true;
}
private async Task<T> FetchAsync(LFHttpRequest request, CancellationToken cancellationToken)
{
var message = new HttpRequestMessage(request.Method, request.Uri);
var response = await HttpClient.SendAsync(message, cancellationToken);
response.EnsureSuccessStatusCode();
var ret = // Convert response.Content into T.
return ret;
}
I would recommend throwing InvalidOperationExceptions for the StartFetch and CancelFetch operations if the IsFetching state is invalid. This may seem annoying but it lets you catch programmer error and threading issues before they get to be a bigger, hidden problem.
As for your asynchronous approach, your method should return a result. So maybe something like private async Task<MyHttpResult> StartTask(...). Your result should contain a way determine success, failure, and cancellation.
For example:
public sealed class MyHttpResult
{
public HttpResponse Result { get; private set; }
public Exception Error { get; private set; }
public bool WasCancelled { get; private set; }
public MyHttpResult(HttpResponse result, Exception error, bool wasCancelled)
{
this.Result = result;
this.Error = error;
this.WasCancelled = wasCancelled;
}
}
Many async methods will throw a TaskCanceledException if they are cancelled, so you can catch that to signify, like so:
async Task<MyHttpResult> StartTask(LFHttpRequest request, CancellationToken cancellationToken)
{
var message = new HttpRequestMessage(new HttpMethod(request.Method), request.Uri);
HttpResponse response = null;
Exception lastError = null;
bool wasCancelled = false;
try
{
response = await MessageInvoker.SendAsync(message, cancellationToken);
}
catch(TaskCanceledException)
{
wasCancelled = true;
}
catch(Exception ex)
{
lastError = ex;
}
var result = new MyHttpResult(response, lastError, wasCancelled);
return result;
}
This is all assuming that your observers are also the callers, so they can await this method. If this is not the case, your idea of an EventHandler makes sense. Instead of returning the result, you could create a custom EventArgs class like so:
public delegate void TaskResultEventHandler<T>(object sender, TaskResultEventArgs<T> e);
public sealed class TaskResultEventArgs<T> : EventArgs
{
public T Result { get; private set; }
public Exception Error { get; private set; }
public bool WasCancelled { get; private set; }
public TaskResultEventArgs(T result, Exception error, bool wasCancelled)
{
this.Result = result;
this.Error = error;
this.WasCancelled = wasCancelled;
}
}
Then it's simply a matter of exposing a TaskResultEventHandler<HttpResponse> and your observers subscribing to it. You could invoke it like so:
var handler = this.HttpTaskCompleted;
if(handler != null)
handler(this, new TaskResultEventArgs<HttpResponse>(response, lastError, wasCancelled));
After you awaited the http call
var response = await HttpClient.SendAsync(message, cancellationToken);
You should test for cancellation:
if(cancellationToken.IsCancellationRequested)
//... do what you want, throw or return false or null, depending on how you want to handle this cancellation.
Or you can check and throw the Microsoft exception in one call :
cancel.ThrowIfCancellationRequested();