We Keep Coding

How to correctly wait for dependent Neo4j bulk load tasks in C#

I need to bulk load two groups of data to Neo4j. The second group builds on the nodes and relationships created by the first group, so the second group cannot be loaded until the first group is fully committed.
My current implementation reads like the following.
public override void Import()
{
using var session = _driver.AsyncSession(
x => x.WithDefaultAccessMode(AccessMode.Write));
var groupA = session.WriteTransactionAsync(async x =>
{
var result = await x.RunAsync(queryA);
return result.ToListAsync();
});
groupA.Result.Wait();
var groupB = session.WriteTransactionAsync(async x =>
{
var result = await x.RunAsync(queryB);
return result.ToListAsync();
});
groupB.Result.Wait();
}
Note that I cannot change the method to be async; hence I am not using await.
This method sometimes runs correctly, but sometimes I get an error that reads like the following.
Cannot access records on this result any more as the result has already been consumed or the query runner where the result is created has already been closed.
I can switch to using the following instead:
groupA.Wait();
Though the issue with this is that the Status of groupA can be RanToCompletion while the Status of the Result is Faulted (e.g., when there is a syntax error with the query). Hence, any possible errors during the execution of queryA are not captured, and I guess that requires checking groupA.Result.Exceptions.
Given these challenges, I am not sure what is the recommended way of waiting for dependent Neo4j tasks in a synchronous method.

What you need is to await the result.ToListAsync in the scope of the Func. Your transaction is completing before you read the results, and after the transaction is closed you can no longer access the results.
var groupA = session.WriteTransactionAsync(async x =>
{
var result = await x.RunAsync(queryA);
return await result.ToListAsync();
});
groupA.Wait();
var groupB = session.WriteTransactionAsync(async x =>
{
var result = await x.RunAsync(queryB);
return await result.ToListAsync();
});
groupB.Wait();
The reason it sometimes completes successfully is because the result.ToListAsync task completes before your transaction has been commited & closed.

Is there a way to combine LINQ and async

Basically I have a procedure like
var results = await Task.WhenAll(
from input in inputs
select Task.Run(async () => await InnerMethodAsync(input))
);
.
.
.
private static async Task<Output> InnerMethodAsync(Input input)
{
var x = await Foo(input);
var y = await Bar(x);
var z = await Baz(y);
return z;
}
and I'm wondering whether there's a fancy way to combine this into a single LINQ query that's like an "async stream" (best way I can describe it).

When you use LINQ, there are generally two parts to it: creation and iteration.
Creation:
var query = list.Select( a => a.Name);
These calls are always synchronous. But this code doesn't do much more than create an object that exposes an IEnumerable. The actual work isn't done till later, due to a pattern called deferred execution.
Iteration:
var results = query.ToList();
This code takes the enumerable and gets the value of each item, which typically will involve the invocation of your callback delegates (in this case, a => a.Name ). This is the part that is potentially expensive, and could benefit from asychronousness, e.g. if your callback is something like async a => await httpClient.GetByteArrayAsync(a).
So it's the iteration part that we're interested in, if we want to make it async.
The issue here is that ToList() (and most of the other methods that force iteration, like Any() or Last()) are not asynchronous methods, so your callback delegate will be invoked synchronously, and you’ll end up with a list of tasks instead of the data you want.
We can get around that with a piece of code like this:
public static class ExtensionMethods
{
static public async Task<List<T>> ToListAsync<T>(this IEnumerable<Task<T>> This)
{
var tasks = This.ToList(); //Force LINQ to iterate and create all the tasks. Tasks always start when created.
var results = new List<T>(); //Create a list to hold the results (not the tasks)
foreach (var item in tasks)
{
results.Add(await item); //Await the result for each task and add to results list
}
return results;
}
}
With this extension method, we can rewrite your code:
var results = await inputs.Select( async i => await InnerMethodAsync(i) ).ToListAsync();
^That should give you the async behavior you're looking for, and avoids creating thread pool tasks, as your example does.
Note: If you are using LINQ-to-entities, the expensive part (the data retrieval) isn't exposed to you. For LINQ-to-entities, you'd want to use the ToListAsync() that comes with the EF framework instead.
Try it out and see the timings in my demo on DotNetFiddle.

A rather obvious answer, but you have just used LINQ and async together - you're using LINQ's select to project, and start, a bunch of async Tasks, and then await on the results, which provides an asynchronous parallelism pattern.
Although you've likely just provided a sample, there are a couple of things to note in your code (I've switched to Lambda syntax, but the same principals apply)
Since there's basically zero CPU bound work on each Task before the first await (i.e. no work done before var x = await Foo(input);), there's no real reason to use Task.Run here.
And since there's no work to be done in the lambda after call to InnerMethodAsync, you don't need to wrap the InnerMethodAsync calls in an async lambda (but be wary of IDisposable)
i.e. You can just select the Task returned from InnerMethodAsync and await these with Task.WhenAll.
var tasks = inputs
.Select(input => InnerMethodAsync(input)) // or just .Select(InnerMethodAsync);
var results = await Task.WhenAll(tasks);
More complex patterns are possible with asynchronony and Linq, but rather than reinventing the wheel, you should have a look at Reactive Extensions, and the TPL Data Flow Library, which have many building blocks for complex flows.

Try using Microsoft's Reactive Framework. Then you can do this:
IObservable<Output[]> query =
from input in inputs.ToObservable()
from x in Observable.FromAsync(() => Foo(input))
from y in Observable.FromAsync(() => Bar(x))
from z in Observable.FromAsync(() => Baz(y))
select z;
Output[] results = await query.ToArray();
Simple.
Just NuGet "System.Reactive" and add using System.Reactive.Linq; to your code.

How to add new where condition in IQueryable when selecting a view model?

I am fairly new to C# and .Net, so apologies if something doesn't make sense, I will try my best to explain my problem.
I have two methods which are basically going to use the similar query but with slight differences. So instead of repeating the query in both methods I created a third private method which will return the common part of the query and then the functions can add more clauses in query as they require.
Here is a generic function which returns the IQueryable object with common part of the query
private IQueryable<OfferViewModel> GetOffersQueryForSeller(int sellerId)
{
return Db.Offers
.Where(o => o.Sku.SellerId == sellerId && o.IsActive && !o.IsDiscontinued)
.Select(o => new OfferViewModel
{
Id = o.Id,
Name = o.Sku.Name,
ImageUrl = o.Sku.ImageUrl ?? o.Sku.Upcq.Upc.ImageUrl,
QuantityName = o.Sku.QuantityName
});
}
Following are the two method which are reusing the IQueryable object
public async Task<List<OfferViewModel>> GetSellerOffers(int sellerId)
{
var query = GetOffersQueryForSeller(sellerId);
return await query.ToListAsync();
}
public async Task<List<OfferViewModel>> GetDowngradableSellerOffers(int sellerId)
{
var query = GetOffersQueryForSeller(sellerId);
return await query
.Where(o => o.Sku.Id == monthlySkuId)
.ToListAsync();
}
Now GetSellerOffers works just fine but GetDowngradableSellerOffers throws a run time error with message The specified type member 'Sku' is not supported in LINQ to Entities.. I asked around and one of the guys told me that I cannot add additional where after adding a select which uses a ViewModel because then my records will be mapped to ViewModel and LINQ will attempt to look up props of ViewModel instead of database columns.
Now I have two questions,
In the docs I read Entity Framework will only run query when I try to fetch the results with methods like ToList and if I haven't done that why it wouldn't allow me to apply conditions on database fields/
How can I reuse the common query in my scenario?

How about the following code:
(The type Offer should be replaced by the type of the Elements that Db.Offers holds)
private IQueryable<OfferViewModel> GetOffersQueryForSeller(int sellerId, Func<Offer,bool> whereExtension)
{
return Db.Offers
.Where(o => ... && whereExtension.Invoke(o))
.Select(o => new OfferViewModel { ... });
}
private IQueryable<OfferViewModel> GetOffersQueryForSeller(int sellerId)
{
return GetOffersQueryForSeller(sellerId, (o) => true);
}
And then call it in GetDowngradableSellerOffers like this:
public async Task<List<OfferViewModel>> GetDowngradableSellerOffers(int sellerId)
{
var query = GetOffersQueryForSeller(sellerId, (o) => o.Sku.Id == monthlySkuId);
return await query.ToListAsync();
}

Why does Task.WhenAll return void?

This is the code from the image above:
if (claims != null && claims.Any())
{
// firstly, why doesn't this work?
// var _claimResults = from claim in claims select UserManager.AddClaimAsync(user.Id, claim);
// but this does...
List<Task> _claimResults = new List<Task>();
foreach (var claim in claims)
{
_claimResults.Add(UserManager.AddClaimAsync(user.Id, claim));
}
// secondly, why does Task.WhenAll return void when it clearly says it returns Task?
Task claimsResult = await Task.WhenAll(_claimResults);
}
Why doesn't the LINQ expression work, yet the foreach does. The LINQ expression gives me a "underlying provider failed to open" exception on execution.
Why does Task.WhenAll() return void when it says it's return type is Task?
Edit: claims is a List<Claim> which I think is List<System.Security.Claim>.

WhenAll returns a Task, but then you're awaiting that task. Awaiting a plain Task (rather than a Task<T>) gives no result. So you either want:
Task claimsResult = Task.WhenAll(_claimResults);
or
await Task.WhenAll(_claimResults);
My suspicion is that the LINQ problem is because your foreach approach materializes the query immediately - the LINQ equivalent would be:
var _claimsResults = claims.Select(claim => UserManager.AddClaimAsync(user.Id, claim))
.ToList();
... where the ToList() method materializes the results immediately. (I've used the method call syntax rather than query expression syntax because query expressions are pretty pointless for trivial queries like this.)

Not sure about #1, but for #2 its because you called await on it
it should be
Task claimsResult = Task.WhenAll(_claimResults);
await claimsResult;
or omit the variable entirely if you don't need to delay the await somewhere else

Asynchronous LINQ

I was looking for an async .Where() but could not find one so after some research I've created one.
public static class LinqExtension
{
public static async Task<IEnumerable<T>> WhereAsync<T>(this IEnumerable<T> source, Func<T, Task<bool>> #delegate)
{
var tasks = source.Select(async t => new
{
Predicate = await #delegate(t).ConfigureAwait(false),
Value = t
}).ToList();
var results = await Task.WhenAll(tasks).ConfigureAwait(false);
IEnumerable<T> typeList = results.Where(pred => pred.Predicate).Select(val => val.Value);
return typeList;
}
}
When I try to use it i get runtime error
Cannot convert implicit type bool to Task and yes it's correct
This is how I've tried
var q = await context.StockHistories.WhereAsync(x => x.ProductId == productId);
I've tried
context.StockHistories.WhereAsync(Task.Run(() => { x => x.ProductId == productId; }));
but getting
Only assignment, call, increment, decrement, and new object
expressions can be used as a statement
Can please someone provide a solution and explain what I am doing wrong?

The async methods for EF are the ones that execute the query. So what you actually want is
var q = await context.StockHistories.Where(x => x.ProductId == productId).ToListAsync();
Basically there isn't an asynchronous Where method because it doesn't make sense to have one because it's just used to generate the actual SQL that will be executed on the DB. The query isn't actually run until you iterate the results, and all the methods that do that have an asynchronous version.