We Keep Coding

Threading in c# while making put calls

I am new to threading world of c#. I read there are different ways to do threading like sequential.
My scenario is below. Which one would be more suitable for the below.
I have list of complex objects. I will be making calls to PUT endpoint for each object [body of put] separately. There can be 1000 or more objects in the list. And I cannot pass all the objects at one and hence I have to pass each object in every call to the put endpoint. In this way, I have to make 1000 calls separately if there are 1000 objects.
Each put call is independent of each other while I have to store the properties of the response back from each call.
I was thinking to apply threading concept to above but not sure which one and how to do it.
Any suggestions would be greatly appreciated.
Thanking in advance.
As per the comments below,
Putting the method signatures here and adding more details.
I have IEnumerable<CamelList>. For each camel, I have to make a put request call and update the table from the response of each call. I will write a new method that will accept this list and make use of below 2 methods to make call and update table. I have to ensure, I am making not more than 100 calls at the same time and the API I am calling can be called by the same user 100 times per minute.
We have a method as
public Camel SendRequest(handler, uri, route, Camel); //basically takes all the parameters and provide you the Camel.
We have a method as public void updateInTable(Entity Camel); //updates the table.

HTTP calls are typically made using the HttpClient class, whose HTTP methods are already asynchronous. You don't need to create your own threads or tasks.
All asynchronous methods return a Task or Task<T> value. You need to use theawaitkeyword to await for the operation to complete asynchronously - that means the thread is released until the operation completes. When that happens, execution resumes after theawait`.
You can see how to write a PUT request here. The example uses the PutAsJsonAsync method to reduce the boilerplate code needed to serialize a Product class into a string and create a StringContent class with the correct content type, eg:
var response = await client.PutAsJsonAsync($"api/products/{product.Id}", product);
response.EnsureSuccessStatusCode();
If you want to PUT 1000 products, all you need is an array or list with the products. You can use LINQ to make multiple calls and await the tasks they return at the end :
var callTasks = myProducts.Select(product=>client.PutAsJsonAsync($"api/products/{product.Id}", product);
var responses = await Task.WhenAll(callTasks);
This means that you have to wait for all requests to finish before you can check if any one succeeded. You can change the body of Select to await the response itself :
var callTasks = myProducts.Select(async product=>{
var response=await client.PutAsJsonAsync($"api/products/{product.Id}", product);
if (!response.IsSuccessStatusCode)
{
//Log the error
}
return response.StatusCode;
});
var responses=await Task.WhenAll(callTasks);
It's better to conver the lambda into a separate method though, eg PutProductAsync :
async Task<HttpStatusCode> PutProduct(Product product,HttpClient client)
{
var response=await client.PutAsJsonAsync($"api/products/{product.Id}", product);
if (!response.IsSuccessStatusCode)
{
//Log the error
}
return response.StatusCode;
};
var callTasks = myProducts.Select(product=>PutProductAsync(product));
var responses=await Task.WhenAll(callTasks);

I'm going to suggest using Microsoft's Reactive Framework for this. You need to NuGet "System.Reactive" to get the bits.
Then you can do this:
var urls = new string[1000]; //somehow populated;
Func<string, HttpContent, IObservable<string>> putCall = (u, c) =>
Observable
.Using(
() => new HttpClient(),
hc =>
from resp in Observable.FromAsync(() => hc.PutAsync(u, c))
from body in Observable.FromAsync(() => resp.Content.ReadAsStringAsync())
select body);
var callsPerTimeSpanAllowed = 100;
var timeSpanAllowed = TimeSpan.FromMinutes(1.0);
IObservable<IList<string>> bufferedIntervaledUrls =
Observable.Zip(
Observable.Interval(timeSpanAllowed),
urls.ToObservable().Buffer(callsPerTimeSpanAllowed),
(_, buffered_urls) => buffered_urls);
var query =
from bufferedUrls in bufferedIntervaledUrls
from url in bufferedUrls
from result in putCall(url, new StringContent("YOURCONTENTHERE"))
select new { url, result };
IDisposable subscription =
query
.Subscribe(
x => { /* do something with each `x.url` & `x.result` */ },
() => { /* do something when it is all finished */ });
This code is breaking the URLs into blocks (or buffers) of 100 and putting them on a timeline (or interval) of 1 minute apart. It then calls the putCall for each URL and returns the result.
It's probably a little advanced for you now, but I thought this answer might be useful just to see how clean this can be.

Get first IObservable event without blocking the thread/task that wants it

I am looking at using IObservable to get a response in a request-response environment within a c# async methods and replace some older callback based code, but I am finding that if a value is pushed (Subject.OnNext) to the observable but FirstAsync is not yet at the await, then the FirstAsync is never given that message.
Is there a straightforward way to make it work, without a 2nd task/thread plus synchronisation?
public async Task<ResponseMessage> Send(RequestMessage message)
{
var id = Guid.NewGuid();
var ret = Inbound.FirstAsync((x) => x.id == id).Timeout(timeout); // Never even gets invoked if response is too fast
await DoSendMessage(id, message);
return await ret; // Will sometimes miss the event/message
}
// somewhere else reading the socket in a loop
// may or may not be the thread calling Send
Inbound = subject.AsObservable();
while (cond)
{
...
subject.OnNext(message);
}
I cant put the await for the FirstAsync simply before I send the request, as that would prevent the request being sent.

The await will subscribe to the observable. You can separate the subscription from the await by calling ToTask:
public async Task<ResponseMessage> Send(RequestMessage message)
{
var id = Guid.NewGuid();
var ret = Inbound.FirstAsync((x) => x.id == id).Timeout(timeout).ToTask();
await DoSendMessage(id, message);
return await ret;
}

I took a closer look and there is a very easy solution to your problem by just converting hot into cold observable. Replace Subject with ReplaySubject. Here is the article: http://www.introtorx.com/content/v1.0.10621.0/14_HotAndColdObservables.html.
Here is the explanation:
The Replay extension method allows you take an existing observable
sequence and give it 'replay' semantics as per ReplaySubject. As a
reminder, the ReplaySubject will cache all values so that any late
subscribers will also get all of the values.

await on observable to complete

I have a method that do some work asynchronously with use of observable. I would like to know what is the best way to make this method async, so that I will be able to await on it and do some work after observable completes.
My first try was to use await on observable.
public async Task DoWorkAsync()
{
var observable = Observable.Create<int>(o =>
{
Task.Run(() =>
{
Thread.Sleep(1000);
Console.WriteLine("OnNext");
o.OnNext(1);
o.OnError(new Exception("exception in observable logic"));
//o.OnCompleted();
});
return Disposable.Empty;
});
//observable = observable.Publish().RefCount();
observable.Subscribe(i => Console.WriteLine(i));
Console.WriteLine("Awaiting...");
await observable;
Console.WriteLine("After Awaiting...");
}
Depending on the scenario I had different issues with that approach (+/- means that this part of code is uncommented/commented):
+OnNext +OnCompleted -OnError -RefCount: OnNext was invoked 2 times (observable was subscribed 2 times). This is what I would like to avoid.
+OnNext +OnCompleted -OnError +RefCount: When I use RefCount() method the code works.
-OnNext +OnCompleted -OnError +RefCount: "Sequence contains no element" exception is thrown when my observable doesn't raise OnNext.
+OnNext -OnCompleted +OnError -RefCount: OnNext was invoked 2 times. Exception raised.
+OnNext -OnCompleted +OnError +RefCount: Hangs after displaying 1 (probably because it wants to return to thread that is awaited). We can make it work (and raise exception) by using SubscribeOn(ThreadPoolScheduler.Instance)
Anyway in case when observable is empty (no OnNext rised) we get exception even if OnError is not called and we don't have any exception inside observable logic. Thats why awaiting on observable is not good solution.
That is why I tried another solution using TaskCompletionSource
public async Task DoWorkAsync()
{
var observable = Observable.Create<int>(o =>
{
Task.Run(() =>
{
Thread.Sleep(1000);
Console.WriteLine("OnNext");
o.OnNext(1);
o.OnError(new Exception("exception in observable logic"));
//o.OnCompleted();
});
return Disposable.Empty;
});
var tcs = new TaskCompletionSource<bool>();
observable.Subscribe(i => Console.WriteLine(i),
e =>
{
//tcs.TrySetException(e);
tcs.SetResult(false);
},
() => tcs.TrySetResult(true));
Console.WriteLine("Awaiting...");
await tcs.Task;
Console.WriteLine("After Awaiting...");
}
It works ok in all scenarios and in case of OnError is invoked we could either use tcs.SetResult(false) and don't have information about exception details in outside method or we could use tcs.TrySetException(e) and be able to catch the exception in the outside method.
Can you suggest me if there is some better/cleaner solution or my second solution is the way to go?
EDIT
So I would like to know if there is a better solution than my second solution that will:
not require to use .Publish().RefCount()
not require additional subscription (what happens in await observable under the hood - OnNext is invoked 2 times)
Of course I could wrap my solution in some async extension method for subscribing that returns Task

EDIT:
If you remove the subscription you can do the following:
await observable.Do(i => Console.WriteLine(i)).LastOrDefaultAsync();
As for your arbitrary requirements... Not having multiple subscriptions for a cold observable makes sense; so you publish it. Refusing to use .Publish().Refcount() doesn't make sense. I don't understand why you're rejecting a solution that solves your problem.
There's a lot there, but I'm assuming this is your main question:
Anyway in case when observable is empty (no OnNext rised) we get
exception even if OnError is not called and we don't have any
exception inside observable logic. Thats why awaiting on observable is
not good solution.
await observable is the same as await observable.LastAsync(). So if there is no element, you get an exception. Imagine changing that statement to int result = await observable; What should the value of result be if there's no elements?
If you change await observable; to await observable.LastOrDefaultAsync(); everything should run smoothly.
And yes, you should use .Publish().Refcount()

I'd clearly prefer the 2nd solution, because it only subscribes once.
But out of curiosity: what's the purpose of writing a method like this?
If it's to allow for configurable side effects, this would be equivalent:
public async Task DoWorkAsync()
{
Action<int> onNext = Console.WriteLine;
await Task.Delay(1000);
onNext(1);
throw new Exception("exception in DoWork logic"); // ... or don't
}

You could use ToTask extension method:
await observable.ToTask();

Can I check in observer if a new item arrived to the observable?

I'm accessing a memory area that belongs to a different computational process.
There are relatively infrequent changes in the area and I need to run a calculation when there are changes. I get notifications on change, but I need to wait a bit to make sure that no more changes are being made. I model it like this:
var readyToBeProcessed = changed
.Select(x => DateTime.Now)
.Throttle(TimeSpan.FromSeconds(5));
However my calculations take quite some time, and it is possible that memory changes while I'm doing them. In this case I need to mark this particular round of calculations as invalid.
But how do I know in my observer, when I finished the calculation, if another event arrived or not, while processing current event? If there are no events arrive since I started the calculation, then it's valid and I can store the result.
In practice, it's very rarely, that the event arrive in the pattern (fast enough) that allow the calculation to become invalid, still I'd like to cater for this case.
Note: I realize that I cannot guarantee to have always valid calculations. There is a small time between a change in memory is made and the time I receive the event. It is entirely possible, that the sequence is like this 1) I'm doing the calculation 2) memory changes 3) I finish the calculation and check the event, and decide the calculation is valid 4) memory change event arrives. I'm happy to live with this for now
readyToBeProcessed.Subscribe(x =>
{
Log.Info("Start work...");
// Do calculation here
...
// When finished
if (Is there a new item)
{
Log.Info("There were changes while we worked... Invalidating");
Invalidate();
}
else
{
Log.Info("Succeeded");
}
}, cancellationToken);
Is Reactive bad fit for this task?

Rx is actually a great choice here, I think, though you may need to model it a bit more explicitly.
Think of there being really five type of events: Item changes, do-Work-begins, and do-Work-ends, Invalidates, and Succeededs (I wish I could use better names, but I'm working off what you wrote).
Here's a marble diagram of how they would work:
t(sec) : 0--1--2--3--4--5--6--7--8--9--10-11-12-13-14-15-16...
item-change : *-*--**-----------------*-------------------------...
do-Work-begins: ---------------------*-----------------*----------...
do-Work-ends : -------------------------*------------------*-----...
invalidate : -------------------------*------------------------...
succeeded : --------------------------------------------*-----...
We begin work once there has been a 5 second lull in item changes. If there has been any changes during the work time, we want to invalidate upon work-completion. If not, we want to observe success.
var doWorkBegins = changed
.Select(x => DateTime.Now)
.Throttle(TimeSpan.FromSeconds(5));
var doWorkEnds = doWorkBegins
.SelectMany(x =>
{
Log.Info("Start work...");
// DoWork();
//
// should return an observable that returns a single value when complete.
// If DoWork is just a void, then can use
// return Observable.Return(Unit.Default);
});
var lists = changed
.Buffer(() => doWorkEnds)
.Publish().RefCount();
var succeeded = lists
.Where(l => l.Count == 0);
var invalidate = lists
.Where(l => l.Count > 0);
invalidate.Subscribe(x =>
{
Log.Info("There were changes while we worked... Invalidating");
Invalidate();
}, cancellationToken);
succeeded.Subscribe(x =>
{
Log.Info("Succeeded");
}, cancellationToken);

Ideally, I would recommend you use a Task for keeping track of your work, then you can use:
readyToBeProcessed
.Select(evt => Observable.StartAsync<Unit>(async (cancellationToken) =>
{
//pass cancellationToken to work
var result = await DoWork(cancellationToken);
//test token if needed
return result;
}))
.Switch()
.Subscribe();
When the next item arrives, the current token will be canceled.

How to set dispose action to Observable?

I know how to create an observable and assign a disposing action:
Observable.Create(o =>
{
// o.OnNext etc.
return Disposable.Create(() => { /* ... */ });
});
But now I produced an observable from query syntax:
var observable = from x in otherObservable
select x;
How to assign a disposing action to such query?

If I understand correctly, you want to "chain" or "listen" whenever the subscription is disposed. One way to do this is to use the Finally operator of IObservable<T>, as such:
var ob = from x in Observable.Interval(TimeSpan.FromSeconds(1))
select x;
// Use Finally to create an intermediate IObservable
var disposeSub = ob.Finally(() => Console.WriteLine("disposed"));
// Subscribe to the intermediate observable instead the original one
var yourSub = disposeSub.Subscribe(Console.WriteLine);
// Wait for some numbers to print
Thread.Sleep(TimeSpan.FromSeconds(4));
// "disposed" will be written on the console at this point
yourSub.Dispose();
Hope that helps!

I think you should clarify your question. It's not entirely clear what you mean by "disposing action".
Calling an action using Observable.Finally has been suggested, but this action would run when the first of the following conditions is met:
The Observable sends OnCompleted()
The Observable sends OnError()
The subscription handle is disposed.
i.e. You can't guarantee that the action will be executed precisely when you call Dispose on the subscription handle; it may have already been run - but calling Dispose ensures it will have been invoked before the call to Dispose returns.
This may be what you need - but taking you at your word, you only want the action to run in the last of these cases - on dispose of the handle, then you would need to attach the action to the subscription handle itself, ie:
var otherDisposable = /* your observable */;
Action disposingAction = () => Console.WriteLine("I am disposed!");
var subscription = otherDisposable.Subscribe(/* set your handlers here */);
var disposable = new CompositeDisposable(
subscription,
Disposable.Create(disposingAction));
/* The disposingAction is *only* run when this is called */
disposable.Dispose();
I can't think what scenario would require this though, I wonder if Observable.Finally, as suggested by Carlos, is a better fit!

You don't dispose an observable. You dispose a subscription to an observable.
Example:
var observable = from x in otherObservable
select x;
var sub = observable.Subscribe(DoStuff);
sub.Dispose();