We Keep Coding

MongoDB C# Driver - Fastest way to perform an "IN" query on _id

I'm trying to get values from a collection, based on items whose IDs are in a certain collection of IDs.
My current code to build the filter is:
IEnumerable<string> IDList;
using (var enumerator = IDList.GetEnumerator())
{
if (enumerator.MoveNext() == false) return null; // empty collection
// take the first key
var key = enumerator.Current;
filter = Builders<MyClass>.Filter.Eq(p => p.Key, key);
// take all the other keys
while (enumerator.MoveNext())
{
var innerKey = enumerator.Current;
filter = filter | Builders<MyClass>.Filter.Eq(p => p.Key, innerKey);
}
}
and then my code to get the items is:
List<MyClass> values = new List<MyClass>();
using (var cursor = await MyCollection.FindAsync(filter))
{
while (await cursor.MoveNextAsync())
{
values.AddRange(cursor.Current);
}
}
This code's performance seems pretty subpar, and I'm sure there has to be a faster way since MongoDB should have very good performance... Not to mention I'm querying an indexed field, which should make the query very fast. What can I do to speed this up, both in an async way and a sync way? From some Googling I've seen that there are many ways to query a collection, and I'm not sure which way would be the best for my particular case.
Running this query in RoboMongo takes 0.02 seconds, while running it in C# MongoDb.Driver takes a full second, sometimes even longer and I'm not sure why.
Thanks in advance.

How about a simple "$in" query?
using MongoDB.Bson;
using MongoDB.Driver;
using System;
using System.Collections.Generic;
using System.Linq;
using System.Threading.Tasks;
namespace ConsoleApp1
{
public class MyClass
{
public ObjectId Id;
public string Key;
}
public class Program
{
static void Main(string[] args)
{
IEnumerable<string> ids = new [] { "a", "b", "c" };
var collection = new MongoClient().GetDatabase("test").GetCollection<MyClass>("test");
foreach (var id in ids)
{
collection.InsertOne(new MyClass { Key = id });
}
// here comes the "$in" query
var filter = Builders<MyClass>.Filter.In(myClass => myClass.Key, ids);
// sync
List<MyClass> values = collection.Find(filter).ToList();
// async
var queryTask = collection.FindAsync(filter);
values = GetValues(queryTask).Result;
Console.ReadLine();
}
private static async Task<List<MyClass>> GetValues(System.Threading.Tasks.Task<IAsyncCursor<MyClass>> queryTask)
{
var cursor = await queryTask;
return await cursor.ToListAsync<MyClass>();
}
}
}

Unique Indexes convention in EF6

How to create custom index and key conventions for different type of indexes. I need different naming for following key or index types:
PK_TableName Primary keys
FK_SourceTable_Column_TargetTable for foreign keys
IX_TableName_Column1_Column2 Non-unique indexes
UX_TableName_Column1_Column2 Unique indexes
By defaults, Entity Framework uses following namings:
PK_schemaname.TableName for primary keys
FK_schemaname.SourceTable_schemaname.TargetTable_Column1 for foreign keys
IX_Column1 for non-unique indexes
ColumnName for unique indexes
I've found out that I can implement IStoreModelConvention<T>, but I haven't found particular type to use as type parameter.
Moreover, there're can be Custom Code-First Conventions, but my research is ended with no results. How I can get mentioned naming rules when I use Entity Framework Code First? It can be anything: package, sample, or just direction for following researches.

Mission impossible for PK and FK. The problems is that there is no special EdmModel property/attribute/annotation for naming the store constraint - in the model they are basically represented as list of columns (properties) and the naming convention is hardcoded inside the migration builder classes. Please note that some examples mentioned in the comments are showing how to rename the FK columns (properties), not the FK constraint itself.
Luckily for indexes, although not simple, but it's possible, thanks to the IndexAttribute and IndexAnnotation. This is because the annotation (with attribute) is associated with column (entity property), and then consolidated by an internal class called ConsolidatedIndex.
So in order to achieve the goal, you have to create IStoreModelConvention<EntityType>, prepare a consolidated index info from properties similar to how ConsolidatedIndex class does it, determine the new name based on your rules for the unnamed indexes or indexes with default name generated for FK constrains by the ForeignKeyIndexConvention, and update the corresponding IndexAnnotation of the properties.
With that being said, here is the code for applying your index name convention:
using System;
using System.Collections.Generic;
using System.ComponentModel.DataAnnotations.Schema;
using System.Data.Entity.Core.Metadata.Edm;
using System.Data.Entity.Infrastructure;
using System.Data.Entity.Infrastructure.Annotations;
using System.Data.Entity.Migrations.Model;
using System.Data.Entity.ModelConfiguration.Conventions;
using System.Linq;
public class IndexNameConvention : IStoreModelConvention<EntityType>
{
public void Apply(EntityType item, DbModel model)
{
// Build index info, consolidating indexes with the same name
var indexInfo = new List<IndexInfo>();
foreach (var p in item.Properties)
{
foreach (var mp in p.MetadataProperties)
{
var a = mp.Value as IndexAnnotation;
if (a == null) continue;
foreach (var index in a.Indexes)
{
var info = index.Name != null ? indexInfo.FirstOrDefault(e => e.Name == index.Name) : null;
if (info == null)
{
info = new IndexInfo { Name = index.Name };
indexInfo.Add(info);
}
else
{
var other = info.Entries[0].Index;
if (index.IsUnique != other.IsUnique || index.IsClustered != other.IsClustered)
throw new Exception("Invalid index configuration.");
}
info.Entries.Add(new IndexEntry { Column = p, Annotation = mp, Index = index });
}
}
}
if (indexInfo.Count == 0) return;
// Generate new name where needed
var entitySet = model.StoreModel.Container.EntitySets.First(es => es.ElementType == item);
foreach (var info in indexInfo)
{
var columns = info.Entries.OrderBy(e => e.Index.Order).Select(e => e.Column.Name);
if (info.Name == null || info.Name == IndexOperation.BuildDefaultName(columns))
{
bool unique = info.Entries[0].Index.IsUnique;
var name = string.Format("{0}_{1}_{2}", unique ? "UX" : "IX", entitySet.Table, string.Join("_", columns));
if (name.Length > 128) name = name.Substring(0, 128);
if (info.Name == name) continue;
foreach (var entry in info.Entries)
{
var index = new IndexAttribute(name);
if (entry.Index.Order >= 0)
index.Order = entry.Index.Order;
if (entry.Index.IsUniqueConfigured)
index.IsUnique = entry.Index.IsUnique;
if (entry.Index.IsClusteredConfigured)
index.IsClustered = entry.Index.IsClustered;
entry.Index = index;
entry.Modified = true;
}
}
}
// Apply the changes
foreach (var g in indexInfo.SelectMany(e => e.Entries).GroupBy(e => e.Annotation))
{
if (g.Any(e => e.Modified))
g.Key.Value = new IndexAnnotation(g.Select(e => e.Index));
}
}
class IndexInfo
{
public string Name;
public List<IndexEntry> Entries = new List<IndexEntry>();
}
class IndexEntry
{
public EdmProperty Column;
public MetadataProperty Annotation;
public IndexAttribute Index;
public bool Modified;
}
}
All you need is to add it to the DbModelBuilder.Conventions in your OnModelCreating:
modelBuilder.Conventions.Add<IndexNameConvention>();

Update Object without Select EF6 MySQL

Is it possible to update objects with Entity Framework, without grabbing them first?
Example: Here, I have a function that provides a Primary Key to locate the objects, pulls them, then updates them. I would like to eliminate having to pull the objects first, and simply run an UPDATE query. Removing the need for the SELECT query being generated.
public async Task<int> UpdateChecks(long? acctId, string payorname, string checkaccountnumber, string checkroutingnumber, string checkaccounttype)
{
using (var max = new Max(_max.ConnectionString))
{
var payments = await
max.payments.Where(
w =>
w.maindatabaseid == acctId && (w.paymentstatus == "PENDING" || w.paymentstatus == "HOLD")).ToListAsync();
payments.AsParallel().ForAll(payment =>
{
payment.payorname = payorname;
payment.checkaccountnumber = checkaccountnumber;
payment.checkroutingnumber = checkroutingnumber;
payment.checkaccounttype = checkaccounttype;
payment.paymentmethod = "CHECK";
payment.paymentstatus = "HOLD";
});
await max.SaveChangesAsync();
return payments.Count;
}
}

You can use the Attach() command to attach an entity you already know exists and then call SaveChanges() will will call the appropriate update method. Here is some sample code from the MSDN article on the topic:
on the subject:
var existingBlog = new Blog { BlogId = 1, Name = "ADO.NET Blog" };
using (var context = new BloggingContext())
{
context.Entry(existingBlog).State = EntityState.Unchanged;
// Do some more work...
context.SaveChanges();
}
Note that this is general EF logic, not related to any specific database implementation.

DapperExtensions: Add "insert ... update on duplicate key"

Now I'm using Dapper + Dapper.Extensions. And yes, it's easy and awesome. But I faced with a problem: Dapper.Extensions has only Insert command and not InsertUpdateOnDUplicateKey. I want to add such method but I don't see good way to do it:
I want to make this method generic like Insert
I can't get cached list of properties for particular type because I don't want to use reflection directly to build raw sql
Possible way here to fork it on github but I want to make it in my project only. Does anybody know how to extend it? I understand this feature ("insert ... update on duplicate key") is supported only in MySQL. But I can't find extension points in DapperExtensions to add this functionality outside.
Update: this is my fork https://github.com/MaximTkachenko/Dapper-Extensions/commits/master

This piece of code has helped me enormously in MySQL -related projects, I definitely owe you one.
I do a lot of database-related development on both MySQL and MS SQL. I also try to share as much code as possible between my projects.
MS SQL has no direct equivalent for "ON DUPLICATE KEY UPDATE", so I was previously unable to use this extension when working with MS SQL.
While migrating a web application (that leans heavily on this Dapper.Extensions tweak) from MySQL to MS SQL, I finally decided to do something about it.
This code uses the "IF EXISTS => UPDATE ELSE INSERT" approach that basically does the same as "ON DUPLICATE KEY UPDATE" on MySQL.
Please note: the snippet assumes that you are taking care of transactions outside this method. Alternatively you could append "BEGIN TRAN" to the beginning and "COMMIT" to the end of the generated sql string.
public static class SqlGeneratorExt
{
public static string InsertUpdateOnDuplicateKey(this ISqlGenerator generator, IClassMapper classMap, bool hasIdentityKeyWithValue = false)
{
var columns = classMap.Properties.Where(p => !(p.Ignored || p.IsReadOnly || (p.KeyType == KeyType.Identity && !hasIdentityKeyWithValue))).ToList();
var keys = columns.Where(c => c.KeyType != KeyType.NotAKey).Select(p => $"{generator.GetColumnName(classMap, p, false)}=#{p.Name}");
var nonkeycolumns = classMap.Properties.Where(p => !(p.Ignored || p.IsReadOnly) && p.KeyType == KeyType.NotAKey).ToList();
if (!columns.Any())
{
throw new ArgumentException("No columns were mapped.");
}
var tablename = generator.GetTableName(classMap);
var columnNames = columns.Select(p => generator.GetColumnName(classMap, p, false));
var parameters = columns.Select(p => generator.Configuration.Dialect.ParameterPrefix + p.Name);
var valuesSetters = nonkeycolumns.Select(p => $"{generator.GetColumnName(classMap, p, false)}=#{p.Name}").ToList();
var where = keys.AppendStrings(seperator: " and ");
var sqlbuilder = new StringBuilder();
sqlbuilder.AppendLine($"IF EXISTS (select * from {tablename} WITH (UPDLOCK, HOLDLOCK) WHERE ({where})) ");
sqlbuilder.AppendLine(valuesSetters.Any() ? $"UPDATE {tablename} SET {valuesSetters.AppendStrings()} WHERE ({where}) " : "SELECT 0 ");
sqlbuilder.AppendLine($"ELSE INSERT INTO {tablename} ({columnNames.AppendStrings()}) VALUES ({parameters.AppendStrings()}) ");
return sqlbuilder.ToString();
}
}

Actually I closed my pull request and remove my fork because:
I see some open pull requests created in 2014
I found a way "inject" my code in Dapper.Extensions.
I remind my problem: I want to create more generic queries for Dapper.Extensions. It means I need to have access to mapping cache for entities, SqlGenerator etc. So here is my way. I want to add ability to make INSERT .. UPDATE ON DUPLICATE KEY for MySQL. I created extension method for ISqlGenerator
public static class SqlGeneratorExt
{
public static string InsertUpdateOnDuplicateKey(this ISqlGenerator generator, IClassMapper classMap)
{
var columns = classMap.Properties.Where(p => !(p.Ignored || p.IsReadOnly || p.KeyType == KeyType.Identity));
if (!columns.Any())
{
throw new ArgumentException("No columns were mapped.");
}
var columnNames = columns.Select(p => generator.GetColumnName(classMap, p, false));
var parameters = columns.Select(p => generator.Configuration.Dialect.ParameterPrefix + p.Name);
var valuesSetters = columns.Select(p => string.Format("{0}=VALUES({1})", generator.GetColumnName(classMap, p, false), p.Name));
string sql = string.Format("INSERT INTO {0} ({1}) VALUES ({2}) ON DUPLICATE KEY UPDATE {3}",
generator.GetTableName(classMap),
columnNames.AppendStrings(),
parameters.AppendStrings(),
valuesSetters.AppendStrings());
return sql;
}
}
One more extension method for IDapperImplementor
public static class DapperImplementorExt
{
public static void InsertUpdateOnDuplicateKey<T>(this IDapperImplementor implementor, IDbConnection connection, IEnumerable<T> entities, int? commandTimeout = null) where T : class
{
IClassMapper classMap = implementor.SqlGenerator.Configuration.GetMap<T>();
var properties = classMap.Properties.Where(p => p.KeyType != KeyType.NotAKey);
string emptyGuidString = Guid.Empty.ToString();
foreach (var e in entities)
{
foreach (var column in properties)
{
if (column.KeyType == KeyType.Guid)
{
object value = column.PropertyInfo.GetValue(e, null);
string stringValue = value.ToString();
if (!string.IsNullOrEmpty(stringValue) && stringValue != emptyGuidString)
{
continue;
}
Guid comb = implementor.SqlGenerator.Configuration.GetNextGuid();
column.PropertyInfo.SetValue(e, comb, null);
}
}
}
string sql = implementor.SqlGenerator.InsertUpdateOnDuplicateKey(classMap);
connection.Execute(sql, entities, null, commandTimeout, CommandType.Text);
}
}
Now I can create new class derived from Database class to use my own sql
public class Db : Database
{
private readonly IDapperImplementor _dapperIml;
public Db(IDbConnection connection, ISqlGenerator sqlGenerator) : base(connection, sqlGenerator)
{
_dapperIml = new DapperImplementor(sqlGenerator);
}
public void InsertUpdateOnDuplicateKey<T>(IEnumerable<T> entities, int? commandTimeout) where T : class
{
_dapperIml.InsertUpdateOnDuplicateKey(Connection, entities, commandTimeout);
}
}
Yeah, it's required to create another DapperImplementor instance because DapperImplementor instance from base class is private :(. So now I can use my Db class to call my own generic sql queries and native queries from Dapper.Extension. Examples of usage Database class instead of IDbConnection extensions can be found here.

Entity Framework LINQ projection into custom type results in missing data

I have a many to many relationship between Contractors and SafetyCouncils. They are joined by a bridge table ContractorsSafetyCouncils which consists of ContractorId and SafetyCouncilId. These 2 columns form a composite key. This relationship is mapped correctly in EF4. The Contractor entity has the property:
public virtual ICollection<SafetyCouncil> SafetyCouncils
{
get;
set;
}
And the SafetyCouncil entity has the property:
public virtual ICollection<Contractor> Contractors
{
get;
set;
}
When accessing these properties via lazy loading from a single Contractor or SafetyCouncil entity, they work exactly as expected. But when accessing this relationship in a query:
from c in ContractorRepository.All()
where c.PQFs.Count() > 0
let psmAudits = c.PQFs.SelectMany(pqf => pqf.Audits)
let psmAudit = psmAudits.FirstOrDefault(audit => audit.CompletedDate == psmAudits.Max(a => a.CompletedDate))
let scsAudits = c.PQFs.SelectMany(pqf => pqf.SCSAudits)
let scsAudit = scsAudits.FirstOrDefault(audit => audit.CompletedDate == scsAudits.Max(a => a.CompletedDate))
select new MasterListItem()
{
AdministratorNotes = c.AdminFlags.Where(f => f.IsActive && f.ForPQF).Select(f => f.Text),
CanViewInfo = false,
ContractorName = c.ContractorName,
ContractorId = c.Id,
ContractorTaxId = c.TaxId,
SafetyCouncilIds = c.SafetyCouncils.Select(sc => sc.Id),
PQFSubmitted = c.PQFs.Max(p => p.PQFInfo.SubmittedDate.Value),
PSMAuditId = psmAudit.Id,
PSMAuditComplete = psmAudit.CompletedDate,
PSMAuditStatus = psmAudit.Status.Description,
SCSAuditId = scsAudit.Id,
SCSAuditComplete = scsAudit.CompletedDate
};
The problem occurs with:
SafetyCouncilIds = c.SafetyCouncils.Select(sc => sc.Id),
For every record the SafetyCouncilIds collection has 0 members, when based on the data in the database every record should have at least 1 SafetyCouncilId associated with it.
If I run the same query, but project into an anonymous type instead of the MasterListItem type, it works correctly. Why can't I project this query into my custom type?
Update:
My MasterListItem POCO contained the following properties:
public string SafetyCouncilIdsString
{
get;
set;
}
public IEnumerable<int> SafetyCouncilIds
{
set
{
StringBuilder sb = new StringBuilder(",");
foreach (var id in value)
{
sb.Append(id);
sb.Append(",");
}
this.SafetyCouncilIdsString = sb.ToString();
}
}
The SafetyCouncilIds property was the cause of the problem. I changed this to an automatic property and built the string elsewhere and projecting onto the POCO worked like a charm.

public IEnumerable<int> SafetyCouncilIds
{
set
{
StringBuilder sb = new StringBuilder(",");
foreach (var id in value)
{
sb = sb.Append(id).Append(","); // <-- try this
// *or sb = sb.AppendFormat("{0},", id);*
}
this.SafetyCouncilIdsString = sb.ToString();
}
}

I have two suggestions:
Try to isolate the the problem by removing any extra parts of the query.
Compare the two sql queries generated and find the differences.
Unfortunately, without access to your code or schema, I can't provide a better answer.

The SafetyCouncilIds property was the cause of the problem. I changed this to an automatic property and built the string elsewhere and projecting onto the POCO worked like a charm.