We Keep Coding

non reentrant observable in c#

Given the following method:
If I leave the hack in place, my unit test completes immediately with "observable has no data".
If I take the hack out, there are multiple threads all attempting to login at the same time.
The host service does not allow this.
How do I ensure that only one thread is producing observables at any given point in time.
private static object obj = new object();
private static bool here = true;
public IObservable<Party> LoadAllParties(CancellationToken token)
{
var parties = Observable.Create<Party>(
async (observer, cancel) =>
{
// this is just a hack to test behavior
lock (obj)
{
if (!here)
return;
here = false;
}
// end of hack.
try
{
if (!await this.RequestLogin(observer, cancel))
return;
// request list.
await this._request.GetAsync(this._configuration.Url.RequestList);
if (this.IsCancelled(observer, cancel))
return;
while (!cancel.IsCancellationRequested)
{
var entities = await this._request.GetAsync(this._configuration.Url.ProcessList);
if (this.IsCancelled(observer, cancel))
return;
var tranche = this.ExtractParties(entities);
// break out if it's the last page.
if (!tranche.Any())
break;
Array.ForEach(tranche, observer.OnNext);
await this._request.GetAsync(this._configuration.Url.ProceedList);
if (this.IsCancelled(observer, cancel))
return;
}
observer.OnCompleted();
}
catch (Exception ex)
{
observer.OnError(ex);
}
});
return parties;
}
My Unit Test:
var sut = container.Resolve<SyncDataManager>();
var count = 0;
var token = new CancellationTokenSource();
var observable = sut.LoadAllParties(token.Token);
observable.Subscribe(party => count++);
await observable.ToTask(token.Token);
count.Should().BeGreaterThan(0);

I do think your question is suffering from the XY Problem - the code contains several calls to methods not included which may contain important side effects and I feel that going on the information available won't lead to the best advice.
That said, I suspect you did not intend to subscribe to observable twice - once with the explicit Subscribe call, and once with the ToTask() call. This would certainly explain the concurrent calls, which are occurring in two different subscriptions.
EDIT:
How about asserting on the length instead (tweak the timeout to suit):
var length = await observable.Count().Timeout(TimeSpan.FromSeconds(3));
Better would be to look into Rx-Testing and mock your dependencies. That's a big topic, but this long blog post from the Rx team explains it very well and this answer regarding TPL-Rx interplay may help: Executing TPL code in a reactive pipeline and controlling execution via test scheduler

Correct way to link Tasks together when return values are needed at different times #2

I asked a question yesterday and, unfortunately, even with the answers provided, I'm still hitting up on stumbling blocks about how to do things correctly... My issue is that my code actually works, but I'm a complete novice at concurrency programming and it feels like I'm not programming the correct way and, most importantly, I'm afraid of developing bad habits.
To make up a simplistic example to elaborate on yesterday's question, suppose I had the following methods:
static Task<IEnumerable<MyClass>> Task1(CancellationToken ct)
static Task<IEnumerable<int>> Task2(CancellationToken ct, List<string> StringList)
static Task<IEnumerable<String>> Task3(CancellationToken ct)
static Task<IEnumerable<Double>> Task4(CancellationToken ct)
static Task Task5(CancellationToken ct, IEnumerable<int> Task2Info, IEnumerable<string> Task3Info, IEnumerable<double> Task4Info)
static Task Task6(CancellationToken ct, IEnumerable<int> Task2Info, IEnumerable<MyClass> Task1Info)
And the code I've written that utilizes them looks as follows:
static Task execute(CancellationToken ct)
{
IEnumerable<MyClass> Task1Info = null;
List<string> StringList = null;
IEnumerable<int> Task2Info = null;
IEnumerable<string> Task3Info = null;
IEnumerable<double> Task4Info = null;
var TaskN = Task.Run(() =>
{
Task1Info = Task1(ct).Result;
}
, ct)
.ContinueWith(res =>
{
StringList = Task1Info.Select(k=> k.StringVal).ToList();
Task2Info = Task2(ct, StringList).Result;
}
, ct);
return Task.Run(() =>
{
return Task.WhenAll
(
TaskN,
Task.Run(() => { Task3Info = Task3(ct).Result; }, ct),
Task.Run(() => { Task4Info = Task4(ct).Result; }, ct)
)
.ContinueWith(res =>
{
Task5(ct, Task2Info, Task3Info, Task4Info).Wait();
}
, ct)
.ContinueWith(res =>
{
Task6(ct, Task2Info, Task1Info).Wait();
}
, ct);
});
}
In other words:
I need the results of Task1 to calculate StringList and to run Task2
Task2, Task3 and Task4 can all run concurrently
I need the return values from all of the above for later method calls
Once these are run, I use their results to run Task5
Once Task5 is run, I use all the results in running Task6
As a simple explanation, imagine the first portion is data gathering, the second is data cleansing and the third data reporting
Like I said, my challenge is that this actually runs, but I simply feel that it's more of a "hack" that the right way to program - Concurrency programming is very new to me and I definitely want to learn the best ways this should be done...

I would feel better about my answer if I could see how your TaskN methods were implemented. Specifically, I would want to validate the need for your TaskN method calls to be wrapped inside calls to Task.Run() if they are already returning a Task return value.
But personally, I would just use the async-await style of programming. I find it fairly easy to read/write. Documentation can be found here.
I would then envision your code looking something like this:
static async Task execute(CancellationToken ct)
{
// execute and wait for task1 to complete.
IEnumerable<MyClass> Task1Info = await Task1(ct);
List<string> StringList = Task1Info.Select(k=> k.StringVal).ToList();
// start tasks 2 through 4
Task<IEnumerable<int>> t2 = Task2(ct, StringList);
Task<IEnumerable<string>> t3 = Task3(ct);
Task<IEnmerable<Double>> t4 = Task4(ct);
// now that tasks 2 to 4 have been started,
// wait for all 3 of them to complete before continuing.
IEnumerable<int> Task2Info = await t2;
IEnumerable<string> Task3Info = await t3;
IEnumerable<Double> Task4Info = await t4;
// execute and wait for task 5 to complete
await Task5(ct, Task2Info, Task3Info, Task4Info);
// finally, execute and wait for task 6 to complete
await Task6(ct, Task2Info, Task1Info);
}

I don't fully understand the code in question but it is very easy to link tasks by dependency. Example:
var t1 = ...;
var t2 = ...;
var t3 = Task.WhenAll(t1, t2).ContinueWith(_ => RunT3(t1.Result, t2.Result));
You can express an arbitrary dependency DAG like that. This also makes it unnecessary to store into local variables, although you can do that if you need.
That way you also can get rid of the awkward Task.Run(() => { Task3Info = Task3(ct).Result; }, ct) pattern. This is equivalent to Task3(ct) except for the store to the local variable which probably should not exist in the first place.
.ContinueWith(res =>
{
Task5(ct, Task2Info, Task3Info, Task4Info).Wait();
}
This probably should be
.ContinueWith(_ => Task5(ct, Task2Info, Task3Info, Task4Info)).Unwrap()
Does that help? Leave a comment.

it feels like I'm not programming the correct way and, most importantly, I'm afraid of developing bad habits.
I definitely want to learn the best ways this should be done...
First, distinguish between asynchronous and parallel, which are two different forms of concurrency. An operation is a good match for "asynchronous" if it doesn't need a thread to do its work - e.g., I/O and other logical operations where you wait for something like timers. "Parallelism" is about splitting up CPU-bound work across multiple cores - you need parallelism if your code is maxing out one CPU at 100% and you want it to run faster by using other CPUs.
Next, follow a few guidelines for which APIs are used in which situations. Task.Run is for pushing CPU-bound work to other threads. If your work isn't CPU-bound, you shouldn't be using Task.Run. Task<T>.Result and Task.Wait are even more on the parallel side; with few exceptions, they should really only be used for dynamic task parallelism. Dynamic task parallelism is extremely powerful (as #usr pointed out, you can represent any DAG), but it's also low-level and awkward to work with. There are almost always better approaches. ContinueWith is another example of an API that is for dynamic task parallelism.
Since your method signatures return Task, I'm going to assume that they're naturally asynchronous (meaning: they don't use Task.Run, and are probably implemented with I/O). In that case, Task.Run is the incorrect tool to use. The proper tools for asynchronous work are the async and await keywords.
So, taking your desired behavior one at a time:
// I need the results of Task1 to calculate StringList and to run Task2
var task1Result = await Task1(ct);
var stringList = CalculateStringList(task1Result);
// Task2, Task3 and Task4 can all run concurrently
var task2 = Task2(ct, stringList);
var task3 = Task3(ct);
var task4 = Task4(ct);
await Task.WhenAll(task2, task3, task4);
// I need the return values from all of the above for later method calls
var task2Result = await task2;
var task3Result = await task3;
var task4Result = await task4;
// Once these are run, I use their results to run Task5
await Task5(ct, task2Result, task3Result, task4Result);
// Once Task5 is run, I use all the results in running Task6
await Task6(ct, task2Result, task1Result);
For more about what APIs are appropriate in which situations, I have a Tour of Task on my blog, and I cover a lot of concurrency best practices in my book.

How to cancel a RestSharp synchronous execute() call?

I have this line, which does a blocking (synchronous) web service call, and works fine:
var response = client.Execute<DataRequestResponse>(request);
(The var represents IRestResponse<DataRequestResponse>)
But, now I want to be able to cancel it (from another thread). (I found this similar question, but my code must stay sync - the code changes have to stay localized in this function.)
I found CancellationTokenSource, and an ExecuteTaskAsync() which takes a CancellationToken. (See https://stackoverflow.com/a/21779724/841830) That sounds like it could do the job. I got as far as this code:
var cancellationTokenSource = new CancellationTokenSource();
var task = client.ExecuteTaskAsync(request, cancellationTokenSource.Token);
task.Wait();
var response = task.Result;
The last line refuses to compile, telling me it cannot do an implicit cast. So I tried an explicit cast:
IRestResponse<DataRequestResponse> response = task.Result as IRestResponse<DataRequestResponse>;
That compiles, runs, but then crashes (throws a NullReferenceException, saying "Object reference not set to an instance of an object").
(By the way, once I have this working, then the cancellationTokenSource.Token will of course be passed in from the master thread, and I will also be adding some code to detect when an abort happened: I will throw an exception.)
My back-up plan is just to abort the whole thread this is running in. Crude, but that is actually good enough at the moment. But I'd rather be doing it "properly" if I can.
MORE INFORMATION:
The sync Execute call is here: https://github.com/restsharp/RestSharp/blob/master/RestSharp/RestClient.Sync.cs#L55 where it ends up calling Http.AsGet() or Http.AsPost(), which then ends up here: https://github.com/restsharp/RestSharp/blob/master/RestSharp/Http.Sync.cs#L194
In other words, under the covers RestSharp is using HttpWebRequest.GetResponse. That has an Abort function, but being a sync function (i.e. which does not return until the request is done) that is not much use to me!

The async counterpart of your call
var response = client.Execute<DataRequestResponse>(request);
is the public virtual Task<IRestResponse<T>> ExecuteTaskAsync<T>(IRestRequest request, CancellationToken token) RestSharp async method.
It takes a cancellation token, and returns a task with the correct return type signature. To use it and wait for its (cancellable) completion, you could change your sample code to:
var cancellationTokenSource = new CancellationTokenSource();
// ...
var task = client.ExecuteTaskAsync<DataRequestResponse>(
request,
cancellationTokenSource.Token);
// this will wait for completion and throw on cancellation.
var response = task.Result;

Given this sync call:
var response = client.Execute<MyData>(request);
process(response);
Change it to:
var response = null;
EventWaitHandle handle = new AutoResetEvent (false);
client.ExecuteAsync<MyData>(request, r => {
response = r;
handle.Set();
});
handle.WaitOne();
process(response);
That is equivalent to the sync version, no benefit is gained. Here is one way to then add in the abort functionality:
bool volatile cancelAllRequests = false;
...
var response = null;
EventWaitHandle handle = new AutoResetEvent (false);
RestRequestAsyncHandle asyncRequest = client.ExecuteAsync<MyData>(request, r => {
response = r;
handle.Set();
});
while(true){
if(handle.WaitOne(250))break; //Returns true if async operation finished
if(cancelAllRequests){
asyncRequest.WebRequest.Abort();
return;
}
}
process(response);
(Sorry, couldn't wait for the bounty to bear fruit, so had to work this out for myself, by trial and error this afternoon...)

To call method asynchronously from loop in c#

I have a requirement where I have pull data from Sailthru API.The problem is it will take forever if I make a call synchronously as the response time depends on data.I have very much new to threading and tried out something but it didn't seems to work as expected. Could anyone please guide.
Below is my sample code
public void GetJobId()
{
Hashtable BlastIDs = getBlastIDs();
foreach (DictionaryEntry entry in BlastIDs)
{
Hashtable blastStats = new Hashtable();
blastStats.Add("stat", "blast");
blastStats.Add("blast_id", entry.Value.ToString());
//Function call 1
//Thread newThread = new Thread(() =>
//{
GetBlastDetails(entry.Value.ToString());
//});
//newThread.Start();
}
}
public void GetBlastDetails(string blast_id)
{
Hashtable tbData = new Hashtable();
tbData.Add("job", "blast_query");
tbData.Add("blast_id", blast_id);
response = client.ApiPost("job", tbData);
object data = response.RawResponse;
JObject jtry = new JObject();
jtry = JObject.Parse(response.RawResponse.ToString());
if (jtry.SelectToken("job_id") != null)
{
//Function call 2
Thread newThread = new Thread(() =>
{
GetJobwiseDetail(jtry.SelectToken("job_id").ToString(), client,blast_id);
});
newThread.Start();
}
}
public void GetJobwiseDetail(string job_id, SailthruClient client,string blast_id)
{
Hashtable tbData = new Hashtable();
tbData.Add("job_id", job_id);
SailthruResponse response;
response = client.ApiGet("job", tbData);
JObject jtry = new JObject();
jtry = JObject.Parse(response.RawResponse.ToString());
string status = jtry.SelectToken("status").ToString();
if (status != "completed")
{
//Function call 3
Thread.Sleep(3000);
Thread newThread = new Thread(() =>
{
GetJobwiseDetail(job_id, client,blast_id);
});
newThread.Start();
string str = "test sleeping thread";
}
else {
string export_url = jtry.SelectToken("export_url").ToString();
TraceService(export_url);
SaveCSVDataToDB(export_url,blast_id);
}
}
I want the Function call 1 to start asynchronously(or may be after gap of 3 seconds to avoid load on processor). In Function call 3 I am calling the same function again if the status is not completed with delay of 3 seconds to give time for receiving response.
Also correct me if my question sounds stupid.

You should never use Thread.Sleep like that because among others you don't know if 3000ms will be enough and instead of using Thread class you should use Task class which is much better option because it provides some additional features, thread pool managing etc. I don't have access to IDE but you should try to use Task.Factory.StartNew to invoke your request asynchronously then your function GetJobwiseDetail should return some value that you want to save to database and then use .ContinueWith with delegate that should save your function result into database. If you are able to use .NET 4.5 you should try feature async/await.
Easier solution:
Parallel.ForEach
Get some details on the internet about it, you have to know nothing about threading. It will invoke every loop iteration on another thread.

First of all, avoid at all cost starting new Threads in your code; those threads will suck your memory like a collapsing star because each of those gets ~1MB of memory allocated to it.
Now for the code - depending on the framework you can choose from the following:
ThreadPool.QueueUserWorkItem for older versions of .NET Framework
Parallel.ForEach for .NET Framework 4
async and await for .NET Framework 4.5
TPL Dataflow also for .NET Framework 4.5
The code you show fits quite well with Dataflow and also I wouldn't suggest using async/await here because its usage would transform your example is a sort of fire and forget mechanism which is against the recommendations of using async/await.
To use Dataflow you'll need in broad lines:
one TransformBlock which will take a string as an input and will return the response from the API
one BroadcastBlock that will broadcast the response to
two ActionBlocks; one to store data in the database and the other one to call TraceService
The code should look like this:
var downloader = new TransformBlock<string, SailthruResponse>(jobId =>
{
var data = new HashTable();
data.Add("job_id", jobId);
return client.ApiGet("job", data);
});
var broadcaster = new BroadcastBlock<SailthruResponse>(response => response);
var databaseWriter = new ActionBlock<SailthruResponse>(response =>
{
// save to database...
})
var tracer = new ActionBlock<SailthruResponse>(response =>
{
//TraceService() call
});
var options = new DataflowLinkOptions{ PropagateCompletion = true };
// link blocks
downloader.LinkTo(broadcaster, options);
broadcaster.LinkTo(databaseWriter, options);
broadcaster.LinkTo(tracer, options);
// process values
foreach(var value in getBlastIds())
{
downloader.Post(value);
}
downloader.Complete();

try with async as below.
async Task Task_MethodAsync()
{
// . . .
// The method has no return statement.
}

Is it in general dubious to call Task.Factory.StartNew(async () => {}) in Subscribe?

I have a situation where I need to use a custom scheduler to run tasks (these need to be tasks) and the scheduler does not set a synchronization context (so no ObserveOn, SubscribeOn, SynchronizationContextScheduler etc. I gather). The following is how I ended up doing it. Now, I wonder, I'm not really sure if this is the fittest way of doing asynchronous calls and awaiting their results. Is this all right or is there a more robust or idiomatic way?
var orleansScheduler = TaskScheduler.Current;
var someObservable = ...;
someObservable.Subscribe(i =>
{
Task.Factory.StartNew(async () =>
{
return await AsynchronousOperation(i);
}, CancellationToken.None, TaskCreationOptions.None, orleansScheduler);
});
What if awaiting wouldn't be needed?
<edit: I found a concrete, and a simplified example of what I'm doing here. Basically I'm using Rx in Orleans and the above code is bare-bones illustration of what I'm up to. Though I'm also interested in this situation in general too.
The final code
It turns out this was a bit tricky in the Orleans context. I don't see how I could get to use ObserveOn, which would be just the thing I'd like to use. The problem is that by using it, the Subscribe would never get called. The code:
var orleansScheduler = TaskScheduler.Current;
var factory = new TaskFactory(orleansScheduler);
var rxScheduler = new TaskPoolScheduler(factory);
var someObservable = ...;
someObservable
//.ObserveOn(rxScheduler) This doesn't look like useful since...
.SelectMany(i =>
{
//... we need to set the custom scheduler here explicitly anyway.
//See Async SelectMany at http://log.paulbetts.org/rx-and-await-some-notes/.
//Doing the "shorthand" form of .SelectMany(async... would call Task.Run, which
//in turn runs always on .NET ThreadPool and not on Orleans scheduler and hence
//the following .Subscribe wouldn't be called.
return Task.Factory.StartNew(async () =>
{
//In reality this is an asynchronous grain call. Doing the "shorthand way"
//(and optionally using ObserveOn) would get the grain called, but not the
//following .Subscribe.
return await AsynchronousOperation(i);
}, CancellationToken.None, TaskCreationOptions.None, orleansScheduler).Unwrap().ToObservable();
})
.Subscribe(i =>
{
Trace.WriteLine(i);
});
Also, a link to a related thread at Codeplex Orleans forums.

I strongly recommend against StartNew for any modern code. It does have a use case, but it's very rare.
If you have to use a custom task scheduler, I recommend using ObserveOn with a TaskPoolScheduler constructed from a TaskFactory wrapper around your scheduler. That's a mouthful, so here's the general idea:
var factory = new TaskFactory(customScheduler);
var rxScheduler = new TaskPoolScheduler(factory);
someObservable.ObserveOn(rxScheduler)...
Then you could use SelectMany to start an asynchronous operation for each event in a source stream as they arrive.
An alternative, less ideal solution is to use async void for your subscription "events". This is acceptable, but you have to watch your error handling. As a general rule, don't allow exceptions to propagate out of an async void method.
There is a third alternative, where you hook an observable into a TPL Dataflow block. A block like ActionBlock can specify its task scheduler, and Dataflow naturally understands asynchronous handlers. Note that by default, Dataflow blocks will throttle the processing to a single element at a time.

Generally speaking, instead of subscribing to execute, it's better/more idiomatic to project the task parameters into the task execution and subscribe just for the results. That way you can compose with further Rx downstream.
e.g. Given a random task like:
static async Task<int> DoubleAsync(int i, Random random)
{
Console.WriteLine("Started");
await Task.Delay(TimeSpan.FromSeconds(random.Next(10) + 1));
return i * 2;
}
Then you might do:
void Main()
{
var random = new Random();
// stream of task parameters
var source = Observable.Range(1, 5);
// project the task parameters into the task execution, collect and flatten results
source.SelectMany(i => DoubleAsync(i, random))
// subscribe just for results, which turn up as they are done
// gives you flexibility to continue the rx chain here
.Subscribe(result => Console.WriteLine(result),
() => Console.WriteLine("All done."));
}