I have the following logic:
loop through a list of ids, get the associated entity, and for that entity, loop through another list of ids and get another entity. Code is below:
foreach (var docId in docIds)
{
var doc = new EntityManager<Document>().GetById(docId);
foreach (var tradeId in tradeIds)
{
var trade = new EntityManager<Trade>().GetById(tradeId);
if (doc.Trade.TradeId != trade.TradeId)
{
Document newDoc = new Document(doc, trade, 0);
new EntityManager<Document>().Add(newDoc);
}
}
}
my question is mainly about sql performance. Obviously there will be a bunch of selects happening, as well as some adds. Is this a bad way to go about doing something like this?
Should I, instead, use a session and get a list of all entities that match the list of ids (with 1 select statement) and then loop after?
It depends only on my expirience. But you can test it yourselve.
If Trade entity isn't very big and count of entities wouldnt be over 1000 - reading all entities and loop after will be much preferable.
If count is more 1k - its better to call stored procedure with joining temp table, containing your ids.
Related
I am updating records on a SharePoint list based on data from a SQL database. Lets say my table looks something like this:
VendorNumber
ItemNumber
Descrpition
1001
1
abc
1001
2
def
1002
1
ghi
1002
3
jkl
There can be multiple keys in each table. I am trying to make a generic solution that will work for multiple different table structures. In the above example, VendorNumber and ItemNumber would be considered keys.
I am able to retrieve the SharePoint lists as c# List<Microsoft.SharePoint.Client.ListItem>
I need to search through the List to determine which individual ListItem corresponds to the current SQL datarow I am on. Since both ListItem and DataRow allow bracket notation to specify column names, this is pretty easy to do using LINQ if you only have one key column. What I need is a way to do this if I have anywhere from 1 key to N keys. I have found this solution but realize it is very inefficient. Is there a more efficient way of doing this?
List<string> keyFieldNames = new List<string>() { "VendorNumber", "ItemNumber" };
List<ListItem> itemList = MyFunction_GetSharePointItemList();
DataRow row = MyFunction_GetOneRow();
//this is the part I would like to make more efficient:
foreach (string key in keyFieldNames)
{
//this filters the list with each successive pass.
itemList = itemList.FindAll(item => item[key].ToString().Trim() == row[key].ToString().Trim());
}
Edited to Add: Here is a link to the ListItem class documentation:
Microsoft.SharePoint.Client.ListItem
While ListItem is not a DataTable object, its structure is very similar. I have intentionally designed it so that both the ListItem and my DataRow object will have the same number of columns and the same column names. This was done to make comparing them easier.
A quick optimization tip first:
Create a Dictionary<string, string> to use instead of row
List<string> keyFieldNames = new List<string>() { "VendorNumber", "ItemNumber" };
DataRow row = MyFunction_GetOneRow();
var rowData = keyFieldNames.ToDictionary(name=>row[name].ToString().Trim());
foreach (string key in keyFieldNames)
{
itemList = itemList.FindAll(item => item[key].ToString().Trim() == rowData[key]);
}
This will avoid doing the ToString & Trim on the same records over & over. That's probably taking 1/3rd to 1/2 the time of the loop. (The comparison is fast compared to the string manipulation)
Beyond that, all I can think of is to use reflection to build a specific function, on the fly to handle the comparison. BUT, that would be a big effort, and I don't see it saving that much time. Basically, whatever you do, will still have to do the same basics: Lookup the values by key, and compare them. That's what's taking the majority of the time.
After I stopped looking for an answer, I stumbled across one. I have now realized that using a .Where is implemented using deferred execution. This means that even though the foreach loop iterates several times, the LINQ query executes all at once. This was the part I was struggling to wrap my head around.
My new sudo code:
List<string> keyFieldNames = new List<string>() { "VendorNumber", "ItemNumber" };
List<ListItem> itemList = MyFunction_GetSharePointItemList();
DataRow row = MyFunction_GetOneRow();
//this is the part I would like to make more efficient:
foreach (string key in keyFieldNames)
{
//this filters the list with each successive pass.
itemList = itemList.Where(item => item[key].ToString().Trim() == row[key].ToString().Trim());
}
I know that the .ToString().Trim() is still inefficient, I will address this at some point. But for now at least my mind can rest knowing that the LINQ executes all at once.
I am trying to update all rows in IQueryable, that I retrieve from a database.
Is this the correct way to conduct this? When I run an Xunit test on this, the rows seem disappear.
foreach (var item in productCustomer) // productCustomer is IQueryable
{
item.isActiveStatus = (item.ExpirationYear < 2019);
}
_dbContext.SaveChanges();
Currently using EF Core 3.1
Note: Trying to refrain from creating totally new object or remapping everything; just want to update 1 member out of the 20 members.
You need to first get the objects, before they can be tracked for changes. However, if all you're doing with the IQueryable is to update a member without touching anything else, it would be faster to just execute the UPDATE query.
Consider:
using(var context = new SampleContext())
{
var commandText = "UPDATE Categories SET IsActiveStatus = ExpirationYear < 2019 --WHERE <YOUR CONDITIONS>";
//var name = new SqlParameter("#variable", "value");
context.Database.ExecuteSqlCommand(commandText/*,<params SqlParameter[]>*/);
}
change
> foreach (var item in productCustomer)
to
foreach (var item in productCustomer.ToList())
IQueryable is nothing more that a select statement on the table that triggers a database read.
No object is stored in memory until you pull it into a List or whatever datatype makes the most sense for you.
I got a database with members, each member has a list of sports they do.
now I want to loop through a listbox and add every selected item to my database.
This is my database :
And this is my code :
foreach (var item in sportCheckedListBox.CheckedIndices)
{
int sportindex = Convert.ToInt32(item.ToString()) + 1;
var queryResult = from sp in context.Sports
where sp.sportnr == sportindex
select sp;
foreach (var sport in queryResult)
{
myMember.Sports.Add(sport);
}
}
This looks kinda 'shady', how could I do this better ?
One thing I'd do for sure is move the query out of the loop. Queries should never exist in loops for performance and maintainability reasons. LINQ knows how to translate a (new int[] { 0, 1, 2, ... }).Contains(column) construct into a WHERE column IN (0, 1, 2, ...) statement, so let's use that:
// Get all checked items together
var lookupIndices = sportCheckedListBox.CheckedIndices.Select(i => Convert.ToInt32(item.ToString()) + 1);
// Find all matching sport numbers
var queryResult = from sp in context.Sports
where lookupIndices.Contains(sp.sportnr)
select sp;
// Now loop over the results
foreach (var sport in queryResult)
{
myMember.Sports.Add(sport);
}
// save changes
I think you can just do AddRange:
myMember.Sports.AddRange(queryResult);
myMember.Sports.SaveChanges()
You may need to covert queryResult to an IEnumerable type if it's not already though.
There is nothing fundamentally wrong with your approach, but you can achieve it more concisely with Linq.
Instead of your foreach loop, if you always want to assign a new list you could use
myMember.Sports = queryResult.ToList();
If you want to instead concatenate results to an existing list, you could use
myMember.Sports = myMember.Sports.Concat(queryResult.ToList());
If you wanted to do the same as above, but not have any duplicates (as defined by the object you are adding), instead
myMember.Sports = myMember.Sports.Union(queryResult.ToList());
I am hitting the database in loop. Pls suggest how i can avoid this ?
foreach (var assignedPricing in reviewingPricing)
{
var assignedUserId = _wfActivitySvc.GetPricingReviewer(assignedPricing.PricingID).UserId;
if (assignedUserId == UserId)
{
reviewingAssignedPricings.Add(assignedPricing);
}
}
Create a new query in your database service:
//Build an collection with just unique ids
var priceIds = reviewingPricing.Select(x => x.PricingId).Distinct();
//Return a key/value collection with all priceId/UserId
var reviewerMap = _wfActivitySvc.GetAllReviewersByPriceId(priceIds);
//now you can loop without db queries
foreach (var pricing in reviewingPricing)
{
var reviewer = reviewMap.FirstOrDefault(x => x.PricingId == pricing.PricingId);
if (reviewerMap == null)
continue;
if (reviewer.UserId == UserId)
{
reviewingAssignedPricings.Add(pricing);
}
}
1) You may want to insert all records at once. You can create stored procedure to do this. If you use SQL server you can use BulkInserter class: https://github.com/ronnieoverby/RonnieOverbyGrabBag/blob/master/BulkInserter.cs
For production I had to tweak it a little bit internally to reduce its initialization time, but for infrequent bulk inserts Github version is just fine.
Usage example:
var bulkInserter = new BulkInserter<YourClass>(SqlConnection, "Table Name");
bulkInserter.Insert(reviewingAssignedPricings);
bulkInserter.Flush();
2) If this reads from database every time
_wfActivitySvc.GetPricingReviewer(assignedPricing.PricingID).UserId;
then replace it with single call outside loop to get all reviweres from database, then add to dictionary (key = priceID, value = reviewer) and then get reviewers from dictionary within loop. If you use simple List and .FirstOrDefault(), then it can be noticeably slow for 100+ items in list. jgauffin answer describes this idea.
If an object is inserted 1 at a time, then the Id can be fetched from the object:
foreach (var object in objectList)
{
conn.Insert(object);
int id = object.Id; // Returns Id as expected
}
However, if an IEnumerable of objects is inserted, the Ids cannot be fetched properly:
conn.Insert(objectList);
foreach (var object in objectList)
{
int id = object.Id; // Returning 0
}
Is there a way to insert the list of objects and still get the Ids back without inserting 1 at a time?
Doesn't look like that has been implemented. See the code here. I would assume this is for performance reasons.