We Keep Coding

Find TOP (1) for each ID in array

I have a large (60m+) document collection, whereby each ID has many records in time series. Each record has an IMEI identifier, and I'm looking to select the most recent record for each IMEI in a given List<Imei>.
The brute force method is what is currently happening, whereby I create a loop for each IMEI and yield out the top most record, then return a complete collection after the loop completes. As such:
List<BsonDocument> documents = new List<BsonDocument>();
foreach(var config in imeiConfigs)
{
var filter = GetImeiFilter(config.IMEI);
var sort = GetImeiSort();
var data = _historyCollection.Find(filter).Sort(sort).Limit(1).FirstOrDefault();
documents.Add(data);
}
The end result is a List<BsonDocument> which contains the most recent BsonDocument for each IMEI, but it's not massively performant. If imeiConfigs is too large, the query takes a long time to run and return as the documents are rather large.
Is there a way to select the TOP 1 for each IMEI in a single query, as opposed to brute forcing like I am above?

have tried using the LINQ Take function?
List documents = new List();
foreach(var config in imeiConfigs)
{
var filter = GetImeiFilter(config.IMEI);
var sort = GetImeiSort();
var data = _historyCollection.Find(filter).Sort(sort).Take(1).FirstOrDefault();
documents.Add(data);
}
https://learn.microsoft.com/es-es/dotnet/api/system.linq.enumerable.take?view=netframework-4.8

I think bad performance come from "Sort(sort)", because the sorting forces it to go through all the collection.
But perhaps you can improuve time performance with parallel.
List<BsonDocument> documents;
documents = imeiConfigs.AsParallel().Select((config) =>
{
var filter = GetImeiFilter(config.IMEI);
var sort = GetImeiSort();
var data = _historyCollection.Find(filter).Sort(sort).Limit(1).FirstOrDefault();
return data;
}).ToList();

Count rows within partition in Azure table storage

I've seen various questions around SO about how to get the total row count of an Azure storage table, but I want to know how to get the number of rows within a single partition.
How can I do this while loading a minimal amount of entity data into memory?

As you may already know that there's no Count like functionality available in Azure Tables. In order to get the total number of entities (rows) in a Partition (or a Table), you have to fetch all entities.
You can reduce the response payload by using a technique called Query Projection. A query projection allows you to specify the list of entity attributes (columns) that you want table service to return. Since you're only interested in total count of entities, I would recommend that you only fetch PartitionKey back. You may find this blog post helpful for understanding about Query Projection: https://blogs.msdn.microsoft.com/windowsazurestorage/2011/09/15/windows-azure-tables-introducing-upsert-and-query-projection/.

https://azure.microsoft.com/en-gb/features/storage-explorer/ allows you to define a Query and you can use the Table Statistics toolbar item to get the total rows for the whole table or your query

Tested the speed using Stopwatch to fetch and count 100,000 entities in a Partition that have three fields in addition to the standard TableEntity.
I select just the PartitionKey and use a resolver to end up with just a list of strings, which once the entire Partition has been retrieved I count.
Fastest I have got it is around 6000ms - 6500ms. Here is the function:
public static async Task<int> GetCountOfEntitiesInPartition(string tableName, string partitionKey)
{
CloudTable table = tableClient.GetTableReference(tableName);
TableQuery<DynamicTableEntity> tableQuery = new TableQuery<DynamicTableEntity>().Where(TableQuery.GenerateFilterCondition("PartitionKey", QueryComparisons.Equal, partitionKey)).Select(new string[] { "PartitionKey" });
EntityResolver<string> resolver = (pk, rk, ts, props, etag) => props.ContainsKey("PartitionKey") ? props["PartitionKey"].StringValue : null;
List<string> entities = new List<string>();
TableContinuationToken continuationToken = null;
do
{
TableQuerySegment<string> tableQueryResult =
await table.ExecuteQuerySegmentedAsync(tableQuery, resolver, continuationToken);
continuationToken = tableQueryResult.ContinuationToken;
entities.AddRange(tableQueryResult.Results);
} while (continuationToken != null);
return entities.Count;
}
This is a generic function, all you need is the tableName and partitionKey.

You could achieve this by leveraging atomic batch operation of azure table storage service pretty efficiently. For every partition have an additional entity with the same partition key and a specific row key like "PartitionCount" etc. That entity will have a single int (or long ) property Count.
Every time you insert a new entity do an atomic batch operation to also increment the Count property of your partition counter entity. Your partition counter entity will have the same partition key with your data entity so that allows you to do an atomic batch operation with guaranteed consistency.
Every time you delete an entity, go and decrement the Count property of the partition counter entity. Again in a batch execute operation so these 2 operations are consistent.
If you want to just read the value of partition count then all you need to do is to make a single point query to the partition counter entity and its Count property will tell you the current count for that partition.

This can be done a bit shorter than #NickBrooks answer.
public static async Task<int> GetCountOfEntitiesInPartition<T>(
string tableName,string partitionKey)
where T : ITableEntity, new()
{
var tableClient = tableServiceClient.GetTableClient(tableName);
var results = _tableClient.QueryAsync<T>(t => t.PartitionKey == partitionKey,
select: new[] { "PartitionKey" });
return await results.CountAsync();
}
The results.CountAsync() comes from System.Linq.Async, a NuGet package which is officially supported by dotnet.

I think you can directly use the .Count in C#. You can use either this technique:
var tableStorageData = await table.ExecuteQuerySegmentedAsync(azQuery, null);
int count = tableStorageData.Count();
or
TableQuery<UserDetails> tableQuery = new TableQuery<UserDetails>();
var tableStorageData = table.ExecuteQuery(tableQuery,null);
count = tableStorageData .Count();
The count variable will have the number total number of rows depending on the query.

Searching for record in c# Winform (Entity Framework)

I have a c# winform with textboxes, connected via Entity Framework to a table, called Candidates (it has 700 records).
I'm using a BindingSource named candidatesBindingSource. Everything works as I want.
There is just one thing. I'm trying to implement searching candidates with surnames. So i have a Textbox, called textSurname and a Button with this code
for searching through my records:
var searchResults = (from a in _context.Candidates where (a.Surname.Contains(textSurname.Text)) select a.Id).ToList();
if (searchResults.Count > 0)
{
// Id of a record in searchResults is correct
var position = searchResults[0];
// This line moves focus to a wrong record
candidatesBindingSource.Position = position; //
}
If a record is found, I can get its Id. And here I have a problem. How can I reposition my candidatesBindingSource to the
record with the Id from my searchResults? For example, if I have an Id = 2638, the code above repositions my candidatesBindingSource
to the last record. I'm suspecting that this part candidatesBindingSource.Position actualy works as recordcount (700 in my table)
and is unable to go to the record nr. 2638 (not to the record with this Id). Am I right? So how can I implement a GOTO record with my found Id?
Do I really have to use a For loop with MoveNext command to compare my searched Id with all Id's?
Any hint would be very appreciated.

Ok, so this is how you initialize you binding source
candidatesBindingSource.DataSource = _context.Candidates.ToList();
Then you don't need to search the database, you can search the data source list using the List.FindIndex method like this:
var candidateList = (List<Candidate>)candidatesBindingSource.DataSource;
var searchText = textSurname.Text;
var firstMatchIndex = candidateList.FindIndex(c => c.Surname.Contains(searchText));
if (firstMatchIndex >= 0)
candidatesBindingSource.Position = firstMatchIndex;

I think you should set to candidatesBindingSource.Position index of item and not id.
That post will help you to get index of item correctly, witout read whole data again.
Get Row Index in a list by using entity framework
Also you can try get index from your binding source.

If you create a list out of your context it will have the same indexing as the databinding you set on your form. To set your form to look at the result of your search you can use the a match from the FindIndex() method of the list, and then set your .Position to that index.
using (Candidates _context = new Candidates())
{
var candidateList = _context.Candidate.ToList();
var firstCandidateMatchIndex = candidateList.FindIndex(c =>
c.Surname.Contains(textSurname.Text));
if (firstCandidateMatchIndex >= 0)
candidateBindingSource.Position = firstCandidateMatchIndex;
}

Getting the selected datalist by an array

I am going to ask a very basic question and probably a repeated one but I have a bit different situation.
I want to use "in" operator in Linq.
I have to get all the rows from table which has Id provided
by my array and returns the row if it has. How can I do it.
My array has
var aa="1091","1092","1093" and so on.
and my table uses these Ids as Primary keys
.I have to get all the rows whose Id is contained in the array and I do not want to use S.P.

You can use Enumerable.Contains,
var aa = new string[3] { "1091", "1092", "1093" };
var res = yourDataSource.Where(c => aa.Contains(c.ID));

IN statements are created by using Contains in your Where call. Assuming you use integers as IDs, you could write something like this:
var myArray=new[]{1091,1092,1094};
var myEntities=from entity in myTable
where myArray.Contains(entity.ID)
select entity;

LINQ with ManyToMany: Filtering based on multiple selection

I am a newbe to C# and have to use it for my master thesis. At the moment, I am facing a problem that is a bit to complex for me.
I have set up a database with a many-to-many relationship like this:
Table Relay:
- id (PK)
- Name
- Input
Table ProtectionFunction:
- id (PK)
- ANSI
- IEC
- Description
Table RelayConfig (junction table)
- RelayID (PK)
- ProtFuncID (PK)
- TimeToSaturate
- Remanence
The thing is, a Relay can have multiple protection functions, and for each it has specific values for TimeToSaturate and Remanence. Now I want to realize a filter. The user can select protection function via checkboxes in a DataGridView and a ListBox should show all Relays that support ALL of these protection functions.
I have already created the LINQ-to-SQL classes for my project. But now I am stuck because I don't know how to realize the filtering. All LINQ commands I have found so far would give me all Relays for one protection function.
I really hope one of you can give me a hint.

var ids = new int[]{ ... };
// if ids is null or ids.Length == 0 please return null or an empty list,
//do not go further otherwise you'll get Relays without any function filter
var query = Relays.AsQueryable();
foreach (var id in ids)
{
var tempId = id;
query = query.Where(r=>r.RelayConfigs.Any(rc=>rc.ProtFuncID == tempId));
}
var items = query.ToList();
Update
Just saw this on PredicateBuilder page:
The temporary variable in the loop is required to avoid the outer
variable trap, where the same variable is captured for each iteration
of the foreach loop.

It's easier if you start from the RelayConfigs. Something like this should work:
var protFuncIds = new[]{1,2,3};
var query = from rc in db.RelayConfigs
where protFuncIds.Contains(rc.ProtFuncID)
select rc.Relay;
var relays = query.Distinct().ToList();
UPDATE:
based on your comment, the following should work, however do monitor the SQL generated...
IQueryable<Relay> query = db.Relays
foreach (var id in ids)
query = relays.Where(r => r.RelayConfigs.Select(x => x.ProtFuncId).Contains(id));
var relays = query.ToList();

// Build a list of protection function ids from your checkbox list
var protFuncIDs = [1,2,3,4];
using(var dc = new MyDataContext())
{
var result = dc.Relays.Where(r=>protFuncIDs.Join(r.RelayConfigs, pf=>pf, rc=>rc.ProtFuncID, (pf,rc)=>pf).Count() == protFuncIDs.Length).ToArray();
}
It's not especially efficient, but that should do the trick for you.

I have done this in Lightswitch, and here was my preprocess query:
partial void UnusedContactTypesByContact_PreprocessQuery(int? ContactID, ref IQueryable<ContactType> query)
{
query = from contactType in query
where !contactType.ContactToContactTypes.Any(c => c.Contact.Id == ContactID)
select contactType;
}
Hope that helps.