We Keep Coding

How can I speed up bulk client comparisons/update/inserts with EF Core?

I have a WebAPI that accepts an uploaded CSV file and parses the file into Entity objects. This happens very quickly and leaves me with a list of a little over 1000 objects. I then need to see if these uploaded Entity objects already exist in the database (i.e. is this an update or an insert).
Rather than have thousands of queries to the database, I read all existing clients and then do my comparisons in memory. However, this whole process takes WAY to long and my call times out. Somehow, I need to speed up this code. Is there anything I could do to speed this up?
public async Task SaveResultsAsync()
{
var allExisting = _dbContext.Clients
.Include(c => c.Bookings)
.IgnoreQueryFilters()
.Where(c => c.AgencyId == _agencyId)
.ToList();
Client searchItem = null;
Func<Client,bool> searchFunc = (c) => {
return c.AgencyReferenceNumber == searchItem.AgencyReferenceNumber ||
(c.LastName == searchItem.LastName && (c.SSN != null && c.SSN == searchItem.SSN)
|| (c.FirstName == searchItem.FirstName && c.Birthdate.HasValue && searchItem.Birthdate.HasValue
&& c.Birthdate.Value == searchItem.Value));
//Its the foreach loop that takes all the time!
foreach(var client in clients){
searchItem = client;
//Searches data queried -- if not found, searches any new clients that have already been added
var existing = allExisting.FirstOrDefault(searchFunc) ?? _dbContext.Clients.Local.FirstOrDefault(searchFunc);
if(existing != null){
//Update existing item
existing.Birthdate = client.Birthdate;
...
foreach(var booking in client.Bookings)
{
var b_existing = existing.Bookings.FirstOrDefault(b => b.AgencyReferenceNumber == booking.AgencyReferenceNumber);
if(b_existing != null)
{
b_existing.AdmissionTypeId = booking.AdmissionTypeId;
...
}
else{
c_existing.Bookings.Add(booking);
}
}
if(_dbContext.Entry(c_existing).State != EntityState.Added)
_dbContext.Update(c_existing);
}
else{
_dbContext.Add(client);
}
}
await _dbContext.BulkSaveChangesAsync() //EntityFramework Extensions
}

I was able to speed this all up significantly by adding .AsNoTracking() to my initial query for allExisting like so:
var allExisting = _dbContext.Clients
.AsNoTracking()
.Include(c => c.Bookings).AsNoTracking()
.IgnoreQueryFilters()
.Where(c => c.AgencyId == _agencyId)
.ToList();
Because I am using _dbContext.Update() and _dbContext.Add(), SaveChanges() still knows what to update.
Apparently, the change tracking built into EF must add much more overhead than I expected.

How can I append mutliple results from context to one list/variable LINQ C#

I'm working on small app for fetching products/articles, and I wrote a method that's getting articles by type. (types are contained in request arg).
What I'm trying to achieve is: append all results (from all if conditions if they are satisfied) to one main list which should be returned to customer..
When I'm debugging and checking query it says its returning type is IQueryable<Article> so basically my question is how can I append multiple IQueryables into one which should be returned to user..
This code below is not working because result is always empty..
I've tried also with var result = new List<Article>(); and later result.AddRange(query); and I've changed also return type to
return await result.AsQueryable().ToListAsync(); but obviously something breaks somewhere and I get an empty array at the end.
public async Task<IEnumerable<Article>> GetArticlesByType(ArticleObject request)
{
var result = new Article[] { }.AsQueryable();
IQueryable<ArticleDTO> query = null;
if (request.Food.HasValue && (bool)request.Food)
{
// Return type of query is IQueryable<Article>
query = _context.Articles.Where(x => x.Active == true && x.ArticleType == ArticleType.Food).Select(x => new Article
{
Id = x.Id,
ArticleName = x.ArticleName
});
// Here I just wanted if this condition is satisfied to add values to my result
result.AsQueryable().Union(query);
}
if (request.Drink.HasValue && (bool)request.Drink)
{
query = _context.Articles.Where(x => x.Active == true && x.ArticleType == ArticleType.Drink).Select(x => new Article
{
Id = x.Id,
ArticleName = x.ArticleName
});
// Again if there are any values in query add them to existing result values
result.AsQueryable().Union(query);
}
if (request.Candy.HasValue && (bool)request.Candy)
{
// When its candy I want also articles from food category
query = _context.Articles.Where(x => x.Active == true && x.ArticleType == ArticleType.Food || x.ArticleType == ArticleType.Candy).Select(x => new Article
{
Id = x.Id,
ArticleName = x.ArticleName
});
// Again if there are values in query add them to existing result
result.AsQueryable().Union(query);
}
//At the end return result and all the values in case all conditions were satisfied
return await result.ToListAsync();
}

Try with result.AsQueryable().Union(query.ToList());. This will fetch the object from database. So far query contains references to objects in database and not in your memory

MongoDB projection toListAsync() method not supported

I am trying to use projection in my query and get the following error:
"The result operation MongoDB.Driver.Linq.Expressions.ResultOperators.ListResultOperator is not supported."
Here is the code:
public async Task<IEnumerable<Listing>> LoadAllUserListings(string userId)
{
var result = _context.Listing.Aggregate().Match(l => l.OwnerId == userId || l.Sales.Any(a => a.Owner.Id == userId)).
Project(l => new Listing
{
Id = l.Id,
Reference = l.Reference,
OwnerId = l.OwnerId,
Sales = l.Sales.Where(a => a.Owner.Id == userId || a.Manager.Id == userId).ToList(),
Products = l.Products,
Status = l.Status,
DueDate = l.DueDate
}).ToListAsync();
return await result;
}
It does not appear to like the ToListAsync call. I got this code snippet from the following answer:
https://stackoverflow.com/questions/50904811/mongodb-c-sharp-filter-and-get-all-subdocuments
There reason I am using projection is to omit some fields which the user should not see (depending on the role). Any help on this would be appreciated.
Thanks in advance.

The problem occurs in that line:
Sales = l.Sales.Where(a => a.Owner.Id == userId || a.Manager.Id == userId).ToList()
What happens here ? MongoDB driver takes this expression and tries to translate it to aggregation framework syntax. There's a $filter operator which can be run on nested collection and driver is able to translate .Where() to that operator however there's nothing corresponding for .ToList() at the end of that expression and that's why it fails.
So the fix is fairly easy: you just need to use IEnumerable<T> instead of List<T> for Sales property and then get rid of that .ToList() so your code will look like this:
public async Task<IEnumerable<Listing>> LoadAllUserListings(string userId)
{
var result = _context.Listing.Aggregate().Match(l => l.OwnerId == userId || l.Sales.Any(a => a.Owner.Id == userId)).
Project(l => new Listing
{
Id = l.Id,
Reference = l.Reference,
OwnerId = l.OwnerId,
Sales = l.Sales.Where(a => a.Owner.Id == userId || a.Manager.Id == userId),
Products = l.Products,
Status = l.Status,
DueDate = l.DueDate
}).ToListAsync();
return await result;
}

How to delete a row from database using lambda linq?

I want to perform a delete operation to unfriend a user in a certain android application I'm developing. The following method returns "Done" but the data doesn't delete from the table. What is the problem here?
public string deleteFriend(int user, int friend) {
int i = db.Friends.Where(x => x.Person.Id == user && x.Person1.Id == friend).Select(x => x.Id).First();
var rec=db.Friends.ToList();
rec.RemoveAll(x=>rec.Any(xx=>xx.Id==i));
db.SaveChanges();
return "Done";
}
I'm working with Entity frame work LINQ.

Try something like this:
var friend = db.Friends.Where (x=>x.person.Id == user && x.Person1.Id == friend).FirstorDefault();
if (friend != null)
{
db.friends.Remove(friend);
db.SaveChanges()
}
if you have got multiple records than you can get rid of firstOrDefault and add .ToList() in the end.
And than use db.friends.RemoveRange(friend)
it is much cleaner and hope this helps
Thanks

using where id in (1,2,3)
List<int> ids = new List<int>(){1,2,3};
var table1 = _context.table1.Where(x => ids.Contains(x.Id)).ToList();
if (table1 != null && table1.Count > 0) {
_context.table1.RemoveRange(table1 );
_context.SaveChanges();
}

WHERE IN clause

I'm working in MVC3 project. I was browsing for a while and trying several examples but I could not get it working.
I need to get a list of record from OrderForm table whose DeptID are in another list I already have got.
I'm aware that I need to use Contains() replacing IN SQL clause, but every example that I could read are doing this in the same way
.Where(ListOfDepartments.Contains(q.DeptID))
This is my method at the controller, which obviously is not working:
public ActionResult ValidOrders(string installation, string orderpriority, string stockclass, string validity)
{
int instID = System.Convert.ToInt32(installation);
int orderpriorityID = System.Convert.ToInt32(orderpriority);
int stockclassID = System.Convert.ToInt32(stockclass);
string period = validity;
try
{
var departments = dba.Department
.Where (a => a.InstID == instID);
var valid = dba.OrderForm
.Where(q => q.FormType == 3
&& q.FormStatus == 2
&& q.OrderPriority.OrderPriorityID == orderpriorityID
&& q.StockClassID == stockclassID
&& departments.Contains(q.DeptID));
return View(valid.ToList());
}
catch (Exception)
{
return View("Error");
}
}
What I'm doing wrong?

you need a list of int, not Department.
var departments = dba.Department
.Where (a => a.InstID == instID)
.Select(d => d.Id);//Id is a guess, it maybe another property name
//.ToList();