Kafka .Net Produce vs ProduceAsync - c#

I am using log4net (.Net) to write kafka appender and I am running into an issue where I cannot use await ProduceAsync.
Error
An asynchronous operation cannot be started at this time. Asynchronous operations may only be started within an asynchronous handler or module or during certain events in the Page lifecycle. If this exception occurred while executing a Page, ensure that the Page is marked <%# Page Async="true" %>. This exception may also indicate an attempt to call an "async void" method, which is generally unsupported within ASP.NET request processing. Instead, the asynchronous method should return a Task, and the caller should await it. ,
StackTrace : at
System.Web.AspNetSynchronizationContext.OperationStarted(at System.Runtime.CompilerServices.AsyncVoidMethodBuilder.Create()
Code
public class CustomAppender: AppenderSkeleton
{
private IProducer<Null, string> p;
public override void ActivateOptions()
{
// Setup kafka producer
}
protected override void Append(LoggingEvent loggingEvent)
{
// Get JSON from application
// Add additional data to the json
callBroker(json, topic);
}
private async void callBroker(string json, string topic)
{
var result = await p.ProduceAsync(Topic, new Message<Null, string>{Value=json});
}
}
I can return Task in my callBroker method but then there is no async override for Append method.
So my question is, Can I use Producer.Produce instead of ProduceAsync in a high volume environment? this program will be logging >500 messages/sec, is there a preference on which works better? I also need to handle some exceptions and take some action if it fails for specific error codes.

Sync version
protected override void Append(LoggingEvent loggingEvent)
{
CallBroker(topic, json);
}
private void CallBroker(string topic, string message)
{
producer.Produce(topic, new Message<Null, string> { Value = message });
}
Semi-async version
If you can't change the signature of the Append method
then you can call an async method in blocking mode via the following way:
protected override void Append(LoggingEvent loggingEvent)
{
CallBrokerAsync(topic, json).GetAwaiter().GetResult();
}
private async Task CallBrokerAsync(string topic, string message)
{
await producer.ProduceAsync(topic, new Message<Null, string> { Value = message });
}
Async shines when it is used all the way down (from the top most entry-point through all the layers till the lowest component which calls the async I/O operation)
As always measure, measure and measure to understand how does this change affect your application.

Related

Diference betwen Exception Handling and Async Exception handling

I have created a Web API app and don't understand how my Global Exception Handling works. The following code doesn't work:
public void Handle(ExceptionHandlerContext context){
if (context.Exception is ObjectNotFoundException)
{
var result = new HttpResponseMessage(HttpStatusCode.NotFound)
{
Content = new StringContent(context.Exception.Message),
ReasonPhrase = "Nothing here for you"
};
context.Result = new ObjectNotFoundException(context.Request, result);
}
}
But this works fine:
public override void Handle(ExceptionHandlerContext context){
if (context.Exception is ObjectNotFoundException)
{
var result = new HttpResponseMessage(HttpStatusCode.NotFound)
{
Content = new StringContent(context.Exception.Message),
ReasonPhrase = "Nothing here for you"
};
context.Result = new ObjectNotFoundException(context.Request, result);
}
}
In you code, the only difference is the override keyword. Hence that's the source of problem here.
Since the method Handles requires to be overridden from the base abstract class ExceptionHandler, you need to use the override keyword.
Without override, you are creating a new implementation and removing the base implementation out there. Hence the method invocation gets removed from pipeline and is equivalent to no such method being present out there.
if you want to do async handling, you need to use "HandleAsync" method instead of "Handle".. https://msdn.microsoft.com/en-us/library/system.web.http.exceptionhandling.exceptionhandler.handleasync(v=vs.118).aspx
Any async handler will be just enable non-blocking execution. Hence if your logging/handling takes quite long, or there's an expected load out there due to too many requests in pipeline, use async.

Running a Task in the background (PCL)

I have a class with an async method:
public static async Task GetData() { ... }
In the app framework I am using I need to start that process and forget about it when the app starts:
protected override void OnStart()
{
await MyService.GetData();
}
I can't make OnStart async. How do I start it in a background task and forget about it?
I can't make OnStart Async. How do I start it in a background task and
forget about it?
Why not? Nothing prevents you from making it async. The async modifier doesn't affect the CLR method signature, i.e., you can override a void method and make it async:
abstract class AppBase
{
protected abstract void OnStart();
}
class App: AppBase
{
public static async Task GetData() { await Task.Delay(1); }
protected override async void OnStart()
{
await GetData();
}
}
This way, at least you'll see an exception if GetData throws, unlike what the other answer suggests.
Make sure you understand how async void methods and Task error handling work in general, this material may be helpful.
Some other problems with Task.Run( () => MyService.GetData() ):
as GetData is already asynchronous, there's very little sense in wrapping it with Task.Run. It's usually only done in a client-side UI app and only if GetData has a long-running synchronous part (before it hits its 1st await). Otherwise, you might as well just call GetData() without Task.Run and without await (which also would be a bad idea: in either case, you'd be doing a fire-and-forget call without observing possible exceptions).
Task.Run will start GetData on a random pool thread without synchronization content, which may be a problem for either a UI app or an ASP.NET app.
If you want to fire this async operation and forget about it all you need to do is invoke the method without awaiting the returned task:
protected override void OnStart()
{
MyService.GetDataAsync();
}
However, since you're not observing the task you would never know if it completed successfully.
You should either keep a reference to the task and await it in a later time:
public Task _dataTask;
protected override void OnStart()
{
_dataTask = MyService.GetDataAsync();
}
public Task AwaitInitializationAsync()
{
return _dataTask;
}
Or add a continuation handling any exceptions:
protected override void OnStart()
{
MyService.GetDataAsync().ContinueWith(t =>
{
try
{
t.Wait();
}
catch (Exception e)
{
// handle exceptions
}
});
}
You shouldn't use Task.Run as Noseratio explained, however using async void is much worse since an exception in an async void method (which isn't a UI event handler) would tear down the entire process*.
You can try to make the method async void while making sure there won't be any exceptions thrown inside it with a try-catch block:
protected override async void OnStart()
{
try
{
await GetData();
}
catch (Exception e)
{
// handle e.
}
}
But I would still recommend against it since even the chance of a complete crash is dangerous.
*You can get around that by registering an even handler for AppDomain.CurrentDomain.UnhandledException but this should be a last resort, not a best practice

How to return data using await function in C# windows phone 8?

I'm new to C#. I
I've a problem related to async methods and await function in C# windows phone 8.0.
I've this http request and can get response. This is working fine and There is no issue...
public async static Task<List<MyAccountData>> GetBalance()
{
HttpClient = new HttpClient();
string response = await client.GetStringAsync("http://xxxx/xxx/xxx");
List<MyAccountData> data = JsonConvert.DeserializeObject<List<MyAccountData>>(response);
return data;
}
I've another class call MainModel
public class MainModel
{
public void LoadData()
{
}
}
So My problem is, I want to call that GetBalance method with in MainModel class and parse data to LoadData method(simply want 2 access Data with in LoadData method). LoadData method can't change return type or can't use async. So how is this possible?
If you want a responsive UI - i.e., one that has a chance of being accepted in the store - then blocking on the async operation is not an option.
Instead, you have to think a bit about how you want your UI to look while the operation is in progress. And while you're thinking about that, also think about how you would want your UI to respond if the operation errors out.
Then you can code up a solution. It's possible to do this with async void, if you catch all exceptions and handle them cleanly:
public async void LoadData()
{
try
{
... // Set up "loading" UI state.
var balance = await GetBalanceAsync();
... // Set up "normal" UI state.
Balance = balance;
}
catch
{
... // Set up "error" UI state.
}
}
However, I prefer to use a type I created called NotifyTaskCompletion, which is a data-bindable wrapper for Task<T> (described in my MSDN article). Using NotifyTaskCompletion, the LoadData becomes much simpler:
public void LoadData()
{
GetBalanceOperation = new NotifyTaskCompletion<Balance>(GetBalanceAsync());
}
public NotifyTaskCompletion<Balance> GetBalanceOperation // Raises INotifyPropertyChanged when set
Then your UI can data-bind to properties on NotifyTaskCompletion<T>, such as IsNotCompleted (for the "loading" state), IsSuccessfullyCompleted and Result (for the "normal" state), and IsFaulted and ErrorMessage (for the "error" state).
There is no difference to use async await in Windows Phone 8 dev:
public class MainModel
{
public async void LoadData()
{
var data = await Foo.GetBalance();
}
}
Depends on whether you want LoadData to be synchronous (not returning until all the data has been streamed in over HTTP, and locking up the UI until then), or to begin the process and return immediately. If you can't change LoadData to async, then those are your only two options.
If you want LoadData to be synchronous:
public void LoadData() {
var task = GetBalance();
var result = task.Result; // will lock up the UI until the HTTP request returns
// now do something with result
}
If you want it to start a background process and return immediately, but for some reason don't want to mark LoadData as async:
public void LoadData() {
BeginLoadData();
}
private async void BeginLoadData() {
var result = await GetBalance();
// now do something with result
}
Though really, there's no reason not to go ahead and make LoadData async. async void does not force you to change the callers in any way (unlike Async<Task<T>>), and it's assignment-compatible with plain old non-async delegates:
public async void LoadData() {
var result = await GetBalance();
// now do something with result
}
// ...
LoadData(); // works just fine
Action myAction = LoadData; // works just fine
As you are working on asynchronus operations you need to wait until the operation is completed.The return type async/await method is always Task(TResult), to access the result of the async/await you need to use Result Property.The get accessor of Result property ensures that the asynchronous operation is complete before returning.
public void LoadData()
{
var data = GetBalance().Result;
}

Use events to forward exceptions for producer/consumer using Dataflow

I’m trying to implement a producer/consumer queue using Dataflow for HTTP requests towards a web service. I found an excellent post from Stephen Cleary, which is covering exactly this scenario. However, in contrast to Stephen’s post, I cannot mark the producer queue as complete since clients shall be able to enqueue requests throughout the entire lifetime of the application. The idea behind this approach that the client can constantly produce requests and the consumer is able to handle requests differently if more than 1 request is pending (which is required).
This requirement leads also to the fact that the consumption of the requests cannot be started after the production was finished, but have to be started the first request was enqueued. This also requires me to start the consumption in a non-blocking way (otherwise it would lead to a deadlock). I’ve done this via an async-call which is not awaited, which unfortunately hampers the exception handling. Exceptions occurring during the consumption (implementing the HTTP requests) cannot bubble up since the call of the consume-function is not awaited. I’ve introduced and event to deal with this kind of problem, but this leads me to the following questions:
Is it a good idea to use an event to forward exceptions from the consumer to the client of the producer?
Is this a good idea to implement the producer/consumer pattern in that fashion for my use case?
Are there potentially other approaches, which are more beneficial under the given circumstances?
To make it a more explicit, I’ve prepared a code example illustrating the problem I described above:
public class HttpConnector
{
private BufferBlock<RequestPayload> queue;
public delegate void ConnectorExceptionHandler(object sender, Exception e);
public event ConnectorExceptionHandler ConnectorExceptionOccured;
public Task<bool> ProduceRequest(RequestPayload payload)
{
if(this.queue == null)
{
this.queue = new BufferBlock<RequestPayload>();
this.ConsumeRequestsAsync(queue); //this call cannot be awaited since it would lead to a deadlock
//however, by not awaiting this call all exceptions raised in
//ConsumeRequestsAsync will be lost
}
return await queue.SendAsync(payload)
}
public Task ConsumeRequestsAsync(BufferBlock<RequestPayload> queue)
{
while(await queue.OutputAvailableAsync())
{
try
{
var payload = await queue.ReceiveAsync();
//do the HTTP request...
}
catch (Exception e)
{
ConnectorExceptionOccured(this, e); //fire event to forward the exception to the client
}
}
}
}
public class Client
{
private HttpConnector connector = new HttpConnector();
public Task<bool> UpdateDataAsync()
{
connector += (object sender, Exception e ) //register an event handler to receive exceptions occur
//during the consumption of the requests
{
//handle exception or re-throw
};
connector.ProduceRequest(new RequestPayload()); //produce a request
}
}
Forwarding exceptions via an event has some severe drawbacks:
Natural exception handling is not possible. If developers are aware of this mechanism, they won't catch any exception.
You cannot use AppDomain#UnhandledException for unhandled exceptions during the application runtime. In fact, if you don't have a subscription to the 'Exception'-event, the exception is completely lost.
If you have only one event to subscribe to, your exception object needs a lot of context information in order to figure out which operation caused the exception.
For our problem it turned out that it is better to use TaskCompletionSource, which is a standard technique to synchronize different threads. An instance of TaskCompletionSource class is provided by each RequestPayload object. After the consumption the TaskCompletionSource.Task is completed (either with the result or with an exception). The producer doesn't return the Task for queue.SendAsync(payload) but payload.CompletionSource.Task:
public class RequestPayload
{
public IModelBase Payload { get; set; }
public TaskCompletionSource<IResultBase> CompletionSource { get; private set; }
}
public class HttpConnector
{
private BufferBlock<RequestPayload> queue;
public Task ProduceRequest(RequestPayload payload)
{
if(this.queue == null)
{
this.queue = new BufferBlock<RequestPayload>();
this.ConsumeRequestsAsync(queue);
}
await queue.SendAsync(payload);
return await payload.CompletionSource.Task;
}
public Task ConsumeRequestsAsync(BufferBlock<RequestPayload> queue)
{
while(await queue.OutputAvailableAsync())
{
try
{
var payload = await queue.ReceiveAsync();
//do the HTTP request...
payload.CompletionSource.TrySetResult(null);
}
catch (Exception e)
{
payload.CompletionSource.TrySetException(e)
}
}
}
}
public class Client
{
private HttpConnector connector = new HttpConnector();
public Task UpdateDataAsync()
{
try
{
await connector.ProduceRequest(new RequestPayload());
}
catch(Exception e) { /*handle exception*/ }
}
}

MvvmLight, Messenger, and Async method calls

I am using MvvmLight and have implemented communication between some of my ViewModels using the MessengerInstance.Send(...) method. It works great!
Recently, though, I have moved from using Synchronous methods to async methods to retrieve data and it looks like this breaks messaging (probably because it executes on a different thread). For example:
public ICommand SomeCommand { get { return new RelayCommand(DoSomething); } }
private async void DoSomething(object obj)
{
//Used to be SomeWcfService.DoSomething(); with some logic afterward
await SomeWcfService.DoSomethingAsync().ContinueWith(task => { //Some logic after method completes });
MessengerInstance.Send(SomeDataToSend, MessageIdentifer.DoSomething);
}
Instead of using a continuation, just put it after the await:
private async void DoSomething(object obj)
{
//Used to be SomeWcfService.DoSomething(); with some logic afterward
var result = await SomeWcfService.DoSomethingAsync();
// .ContinueWith(task => { //Some logic after method completes });
// use result here!
MessengerInstance.Send(SomeDataToSend, MessageIdentifer.DoSomething);
}
If there is no result returned from DoSomethingAsync, you can just leave out the result, and put your code in place.
The continuation, as you wrote it, will not run on the same synchronization context. The await keyword is actually asynchronously waiting your continuation, not the async method from WCF, as well.
If your "some logic" is asynchronous, you can use await within that code, as well.

Categories