We Keep Coding

Data not being inserted into Cosmos C#

I have a list of items (32007 of them)
I am adding them in bulk
However, it seems as though some are not being inserted
The even stranger thing is that if I run my process several times from scratch (i.e. recreating my collection), the number items created varies, I have seen 32005, 32003
I have scaled up my collection to have a lot of RUs (Autoscaled) to 25000
I split the data into tranches of 100
My logic is below
public async Task<List<Account>> ProcessAccountsAsync(List<Account> accounts)
{
var cosmosConnection = await ConnectToDatabaseAsync().ConfigureAwait(false);
var failedAccounts = new List<Account>();
var accountsToInsert = new Dictionary<PartitionKey, Stream>(accounts.Count);
Parallel.ForEach(accounts, (account) =>
{
var stream = new MemoryStream();
var json = JsonConvert.SerializeObject(account, JsonHelper.DefaultSettings());
stream.Write(Encoding.Default.GetBytes(json));
stream.Position = 0;
accountsToInsert.Add(new PartitionKey(account.Id), stream);
});
var tasks = new List<Task>(accounts.Count);
foreach (var account in accountsToInsert)
{
tasks.Add(cosmosConnection.Container.CreateItemStreamAsync(account.Value, account.Key)
.ContinueWith((Task<ResponseMessage> task) =>
{
using (var response = task.Result)
{
if (!response.IsSuccessStatusCode)
{
var actualAccount = accounts.FirstOrDefault(x => account.Key.ToString().Contains(x.Id));
Debug.WriteLine($"Processing Account : {actualAccount?.ArcContactId} Received {response.StatusCode} ({response.ErrorMessage}).");
failedAccounts.Add(actualAccount);
}
}
}));
}
await Task.WhenAll(tasks);
return failedAccounts;
}
In my calling logic I retry all accounts that have failed
var tidiedJson = JsonConvert.SerializeObject(list, Formatting.Indented);
accounts = JsonConvert.DeserializeObject <List<DomainModels.Versions.v2.Account>>(tidiedJson);
var failedAccounts = await cosmosAccountRepository.ProcessAccountsAsync(accounts);
while (failedAccounts.Count > 0)
{
failedAccounts = await cosmosAccountRepository.ProcessAccountsAsync(failedAccounts);
}
I have no idea why accounts are not being inserted and why the behaviour is so random!
I have tried this with a variety of throughput and tranche sizes, no difference
Can anyone see anything obvious?
Failing this, I have the accounts with an ID field, is there a way of finding out which of the accounts are not in the database, without having to go through all 32007 of them one by one, which is obviously not a good plan!
Paul

Why is my IEnumerable<T>.Where iterator not executing inside a using block while Async call?

I'm sure this is not a Dapper issue however I am finding, in the following snippet, that the predicate supplied to the Where function is never executed.
private async Task<IEnumerable<Product>> GetProducts()
{
using (var connection = await _connectionFactory.Create())
{
var products = await connection.QueryAsync<Product>("select * from Products");
return products.Where(p => p.Active);
}
}
However if I move the operation to outside the using it is executed.
private async Task<IEnumerable<Product>> GetProducts()
{
var products = Enumerable.Empty<Product>();
using (var connection = await _connectionFactory.Create())
{
products = await connection.QueryAsync<Product>("select * from Products");
}
return products.Where(p => p.Active);
}
Is there some sort of deferred execution going on?

In the first example, if you can make the following modification in the return statement:
return products.Where(p => p.Active).ToList();, then it will work as expected.
Case 1:
Issue here is Where clause applied on the IEnumerable<Product> is deferred execution, which is returned wrapped up in the Task as follows Task<IEnumerable<Product>>, but now you need to run the Task, which shall execute the predicate too, Not sure how are you executing the Task or may be there's an issue with wrapping the deferred execution in this manner, but end result is predicate is not coming in effect as expected, even when its applied on the Dapper result, which is buffered by default (no-streaming)
Case 2:
It works in the second case, since you are completely getting rid of deferred execution, Enumerable.Empty<Product>() is ensuring that memory is first allocated, so predicate is executed moment its applied there's no deferred execution. In fact predicate is any way applied outside the using block
In the Async method, you are disposing the connection with the using block, mostly since Dapper internally allocates memory that's why all the data is sent across, connection is then disposed, and predicate is never executed. I have similar sample, which doesn't rely on database connection and it works as expected, therefore we can deduce that connection dispose play a role here in predicate not executing. In second case predicate is applied outside using block, so connection dispose has no role and memory is already allocated.
Sample Code (using LinqPad):
async Task Main()
{
var result = await GetTest();
result.Dump();
}
public async Task<IEnumerable<Test>> GetTest()
{
var value = await GetTestDb();
return value.Where(x => x.Id == 1);
}
public async Task<IEnumerable<Test>> GetTestDb()
{
return await Task.FromResult(
new List<Test>
{
new Test{Id = 1, Name = "M"},
new Test{Id = 2, Name = "S"}
}
);
}
public class Test
{
public int Id { get; set; }
public string Name { get; set; }
}
Result:

Your predicate is not actually working as predicate. It is simply a LINQ call.
return products.Where(p => p.Active);
When above line execute, products is already filled by all the rows from the table based on your query and QueryAsync call in earlier line.
Good thing about Dapper is that, it provides complete control of query writing to you. So, if you want to filter the records, why not write the query that way?
using(var connection = ....)
{
var param = new DynamicParameters();
param.Add("#Active", 1);
var products = await connection.QueryAsync<Product>("select * from Products where Active = #Active", param);
return products;
}
You should then remove the products.Where line.
About actual problem you asked in question:
I could not reproduce the problem. When I run following code to read the output in console application, it returns expected results.
DbDataReader dbDataReader = new DbDataReader();
IEnumerable<Product> activeProducts = dbDataReader.GetProducts().Result;
Console.WriteLine(activeProducts.Count());
Your method is little modified as below:
public class DbDataReader
{
string connectionString = #"....";
public async Task<IEnumerable<Product>> GetProducts()
{
using(var connection = await GetOpenConnection())
{
var products = await connection.QueryAsync<Product>("select * from Products;WAITFOR DELAY '00:00:05'");
return products.Where(p => p.Active);
}
}
private async Task<SqlConnection> GetOpenConnection()
{
SqlConnection sqlConnection = new SqlConnection(connectionString);
await sqlConnection.OpenAsync();
return sqlConnection;
}
}
Note that I have intentionally delayed the QueryAsync call with WAITFOR.

ContinueWith not being called async

I'm trying to httpget some values before I execute the next line in the statement. I need to wait for this call to return so I can use the values I deserialize into a list.
Since I want the async call to finish first, I wrapped this in a Task. It worked and it's successfully retrieving the JSON. I then can't get it to go into the ContinueWith block. Why is it not going in there, even when the task is completed(?).
How I'm calling it:
Task f = Task.Run(() =>
{
var task = RetrieveDataAsync();
}).ContinueWith((antecedent) =>
{
pokemonListActivityListView.Adapter = new PokemonListAdapter(this, pokemonList);
pokemonListActivityListView.FastScrollEnabled = true;
pokemonListActivityListView.ItemClick += PokemonListActivityListViewOnItemClick;
});
RetrieveDataAsync method:
private async Task RetrieveDataAsync()
{
string dataUri = "http://pokemonapp6359.azurewebsites.net/Pkmn/GetAllPokemon";
using (var httpClient = new HttpClient())
{
var uri = new Uri(string.Format(dataUri, string.Empty));
//DisplayProgressBar(BeforeOrAfterLoadState.Before, progressBarView);
var response = await httpClient.GetAsync(uri);
//DisplayProgressBar(BeforeOrAfterLoadState.After, progressBarView);
if (response.IsSuccessStatusCode)
{
var content = await response.Content.ReadAsStringAsync();
pokemonList = JsonConvert.DeserializeObject<List<PokemonDTO>>(content);
//canPressButtons = true; //fix this when implement local db
Utilities.Utilities.ShowToast(this, "Successfully fetched data", ToastLength.Short, GravityFlags.Center);
return;
}
else
{
Utilities.Utilities.ShowToast(this, "Failed to fetch data", ToastLength.Short, GravityFlags.Center);
return;
}
}
}
Why is my code not going into the ContinueWith when I've got the JSON ? Thanks!

Instead of just assigning the hot task, you are not waiting on it to finish. You have to call ContinueWith on that task:
var task = RetrieveDataAsync();
task.ContinueWith( ... );
Or await the task:
var result = await RetrieveDataAsync();
... // continue

The problem is that you're ignoring the task returned from RetrieveDataAsync. If you return that task from your lambda expression, then it will behave as you expect.
On a side note, you shouldn't use ContinueWith; it's a dangerous API. Use await instead of ContinueWith:
await Task.Run(() => RetrieveDataAsync());
pokemonListActivityListView.Adapter = new PokemonListAdapter(this, pokemonList);
pokemonListActivityListView.FastScrollEnabled = true;
pokemonListActivityListView.ItemClick += PokemonListActivityListViewOnItemClick;

Async/Await not working as expected

I wanted to call / run three tasks parallely and then wait for their results. From my service class what I did is ...
var _validChapterCodesTask = gd.validateChapterCodeDetails(_input1);
var _validGroupCodesTask = gd.validateGroupCodeDetails(_input1);
await Task.WhenAll(_validChapterCodesTask, _validGroupCodesTask);
var _validChapterCodes = await _validChapterCodesTask;
var _validGroupCodes = await _validGroupCodesTask;
And in my DAL Class , the definitions look like :
public async Task<IEnumerable<ChapterCodeValidationOutput>> validateChapterCodeDetails(GroupMembershipValidationInput gmvi)
{
Repository rep = new Repository();
if (!gmvi._chapterCodes.All(x => x.Equals("")))
{
var _validChapterCodes = await rep.ExecuteSqlQueryAsync(typeof(ChapterCodeValidationOutput),SQL.Upload.UploadValidation.getChapterCodeValidationSQL(gmvi._chapterCodes), null);
return (IEnumerable<ChapterCodeValidationOutput>)_validChapterCodes;
}
else
return new List<ChapterCodeValidationOutput>();
}
public async Task<IEnumerable<GroupCodeValidationOutput>> validateGroupCodeDetails(GroupMembershipValidationInput gmvi)
{
Repository rep = new Repository();
if (!gmvi._chapterCodes.All(x => x.Equals("")))
{
var _validGroupCodes = await rep.ExecuteSqlQueryAsync(typeof(GroupCodeValidationOutput), SQL.Upload.UploadValidation.getGroupCodeValidationSQL(gmvi._groupCodes), null);
return (IEnumerable<GroupCodeValidationOutput>)_validGroupCodes;
}
else
return new List<GroupCodeValidationOutput>();
}
But these are not working as expected ? The breakpoints hit show that one by one the method is getting called from service, going to DAL, execute the DB Query and comeback and assign to respective variables.
And as soon as it hits
await Task.WhenAll(_validChapterCodesTask, _validGroupCodesTask);
The next few lines are not hit and the control returns to the invoking controller.
How would I get back their results and fire them parallely and wait for their results only?
What am I doing wrong ?

await Task.WhenAll(_validChapterCodesTask, _validGroupCodesTask)
creates a Task that should be waited for. Use await in the invoking controller.

How to await in a method returning list?

i have a static method that should return list. but i want to do an await inside the method.
public static List<ContactModel> CreateSampleData()
{
var data = new List<ContactModel>();
StorageFolder musiclibrary = KnownFolders.MusicLibrary;
artists = (await musiclibrary.GetFoldersAsync(CommonFolderQuery.GroupByAlbumArtist)).ToList();
for (var i = 0; i < artists.Count; i++)
{
try
{
data.Add(new ContactModel(artists[i].Name));
}
catch { }
}
return data;
}
when i make it
public static async Task<List<ContactModel>> CreateSampleData(){//method contents}
i get error on another page for this code
Error: Task<List<ContactModel>> doesnt contain a definition for ToAlphaGroups
var items = ContactModel.CreateSampleData();
data = items.ToAlphaGroups(x => x.Name);

You have to await your async method:
var items = await ContactModel.CreateSampleData();
Your method now returns a Task, thats why you get the error message.

I don't know whether I should mention this because I agree with the answer of Jan-Patric Ahnen.
But since you said you cannot add await to your code: Task has a property called Result that returns the "result" of the Task.
var items = ContactModel.CreateSampleData().Result;
data = items.ToAlphaGroups(x => x.Name);
A few things before you use Result:
Result blocks the calling thread, if called from the UI thread your app might become unresponsive
You should try to avoid Result at all cost and try to use await since Result can produce unexpected results.