We Keep Coding

Adding items to the list inside foreach loop

epublic ActionResult ExistingPolicies()
{
if (Session["UserId"]==null)
{
return RedirectToAction("Login");
}
using(PMSDBContext dbo=new PMSDBContext())
{
List<Policy> viewpolicy = new List<Policy>();
var userid = Session["UserId"];
List<AddPolicy> policy= dbo.AddPolicies.Where(c => c.MobileNumber ==
(string)userid).ToList();
foreach(AddPolicy p in policy)
{
viewpolicy=dbo.Policies.Where(c => c.PolicyId ==p.PolicyId).ToList();
}
Session["Count"] = policy.Count;
return View(viewpolicy);
}
}
Here the policy list clearly has 2 items.But when I iterate through foreach,the viewpolicy list only takes the last item as its value.If break is used,it takes only the first item.How to store both items in viewpolicy list??
Regards
Surya.

You can iterate through policies and add them by one to list with Add, but I would say that often (not always, though) better option would be to just retrieve the whole list from DB in one query. Without knowing your entities you can do at least something like that:
List<AddPolicy> policy = ...
viewpolicy = dbo.Policies
.Where(c => policy.Select(p => p.PolicyId).Contains(c.PolicyId))
.ToList();
But if you have correctly set up entities relations, you should be able to do something like this:
var viewpolicy = dbo.AddPolicies
.Where(c => c.MobileNumber == (string)userid)
.Select(p => p.Policy) //guessing name here, also can be .SelectMany(p => p.Policy)
.ToList();

Of course; instead of adding to the list, you replace it with a whole new one on each pass of the loop:
viewpolicy=dbo.Policies.Where(c => c.PolicyId ==p.PolicyId).ToList()
This code above will search all the policies for the policy with that ID, turn it into a new List and assign to the viewpolicy variable. You never actually add anything to a list with this way, you just make new lists all the time and overwrite the old one with the latest list
Perhaps you need something like this:
viewpolicy.Add(dbo.Policies.Single(c => c.PolicyId ==p.PolicyId));
This has a list, finds one policy by its ID number (for which there should be only one policy, right? It's an ID so I figured it's unique..) and adds it to the list
You could use a Where and skip the loop entirely if you wanted:
viewpolicy=dbo.Policies.Where(c => policy.Any(p => c.PolicyId == p.PolicyId)).ToList();
Do not do this in a loop, it doesn't need it. It works by asking LINQ to do the looping for you. It should be converted to an IN query and run by the DB, so generally more performant than dragging the policies out one by one (via id). If the ORM didn't understand how to make it into SQL you can simplify things for it by extracting the ids to an int collection:
viewpolicy=dbo.Policies.Where(c => policy.Select(p => p.PolicyId).Any(id => c.PolicyId == id)).ToList();
Final point, I recommend you name your "collections of things" with a plural. You have a List<Policy> viewpolicy - this is a list that contains multiple policies so really we should call it viewPolicies. Same for the list of AddPolicy. It makes code read more nicely if things that are collections/lists/arrays are named in the plural

Something like:
viewpolicy.AddRange(dbo.Policies.Where(c => c.PolicyId ==p.PolicyId));

Replacing Include() calls to Select()

Im trying to eliminate the use of the Include() calls in this IQueryable definition:
return ctx.timeDomainDataPoints.AsNoTracking()
.Include(dp => dp.timeData)
.Include(dp => dp.RecordValues.Select(rv => rv.RecordKind).Select(rk => rk.RecordAlias).Select(fma => fma.RecordAliasGroup))
.Include(dp => dp.RecordValues.Select(rv => rv.RecordKind).Select(rk => rk.RecordAlias).Select(fma => fma.RecordAliasUnit))
.Where(dp => dp.RecordValues.Any(rv => rv.RecordKind.RecordAlias != null))
.Where(dp => dp.Source == 235235)
.Where(dp => dp.timeData.time >= start && cd.timeData.time <= end)
.OrderByDescending(cd => cd.timeData.time);
I have been having issues with the database where the run times are far too long and the primary cause of this is the Include() calls are pulling everything.
This is evident in viewing the table that is returned from the resultant SQL query generated from this showing lots of unnecessary information being returned.
One of the things that you learn I guess.
The Database has a large collection of data points which there are many Recorded values.
Each Recorded value is mapped to a Record Kind which may have a Record Alias.
I have tried creating a Select() as an alternative but I just cant figure out how to construct the right Select and also keep the entity hierarchy correctly loaded. I.e. the related entities are loaded with unnecessary calls to the DB.
Does anyone has alternate solutions that may jump start me to solve this problem.
Ill add more detail if needed.

You are right. One of the slower parts of a database query is the transport of the selected data from the DBMS to your local process. Hence it is wise to limit this.
Every TimeDomainDataPoint has a primary key. All RecordValues of this TimeDomainDataPoint have a foreign key TimeDomainDataPointId with a value equal to this primary key.
So If TimeDomainDataPoint with Id 4 has a thousand RecordValues, then every RecordValue will have a foreign key with a value 4. It would be a waste to transfer this value 4 a 1001 times, while you only need it once.
When querying data, always use Select and select only the properties you actually plan to use. Only use Include if you plan to update the fetched included items.
The following will be much faster:
var result = dbContext.timeDomainDataPoints
// first limit the datapoints you want to select
.Where(datapoint => d.RecordValues.Any(rv => rv.RecordKind.RecordAlias != null))
.Where(datapoint => datapoint.Source == 235235)
.Where(datapoint => datapoint.timeData.time >= start
&& datapoint.timeData.time <= end)
.OrderByDescending(datapoint => datapoint.timeData.time)
// then select only the properties you actually plan to use
Select(dataPoint => new
{
Id = dataPoint.Id,
RecordValues = dataPoint.RecordValues
.Where(recordValues => ...) // if you don't want all RecordValues
.Select(recordValue => new
{
// again: select only the properties you actually plan to use:
Id = recordValue.Id,
// not needed, you know the value: DataPointId = recordValue.DataPointId,
RecordKinds = recordValues.RecordKinds
.Where(recordKind => ...) // if you don't want all recordKinds
.Select(recordKind => new
{
... // only the properties you really need!
})
.ToList(),
...
})
.ToList(),
TimeData = dataPoint.TimeData.Select(...),
...
});
Possible imporvement
The part:
.Where(datapoint => d.RecordValues.Any(rv => rv.RecordKind.RecordAlias != null))
is used to fetch only datapoints that have recordValues with a non-null RecordAlias. If you are selecting the RecordAlias anyway, consider doing this Where after your select:
.Select(...)
.Where(dataPoint => dataPoint
.Where(dataPoint.RecordValues.RecordKind.RecordAlias != null)
.Any());
I'm not really sure whether this is faster. If your database management system internally first creates a complete table with all columns of all joined tables and then throws away the columns that are not selected, then it won't make a difference. However, if it only creates a table with the columns it actually uses, then the internal table will be smaller. This could be faster.

your problem is hierarchy joins in your query.In order to decrease this problem create other query for get result from relation table as follows:
var items= ctx.timeDomainDataPoints.AsNoTracking().Include(dp =>dp.timeData).Include(dp => dp.RecordValues);
var ids=items.selectMany(item=>item.RecordValues).Select(i=>i.Id);
and on other request to db:
var otherItems= ctx.RecordAlias.AsNoTracking().select(dp =>dp.RecordAlias).where(s=>ids.Contains(s.RecordKindId)).selectMany(s=>s.RecordAliasGroup)
to this approach your query do not have internal joins.

How to Performance Test This and Suggestions to Make Faster?

I seem to have written some very slow piece of code which gets slower when I have to deal with EF Core.
Basically I have a list of items that store attributes in a Json string in the database as I am storing many different items with different attributes.
I then have another table that contains the display order for each attribute, so when I send the items to the client I am order them based on that order.
It is kinda slow at doing 700 records in about 18-30 seconds (from where I start my timer, not the whole block of code).
var itemDtos = new List<ItemDto>();
var inventoryItems = dbContext.InventoryItems.Where(x => x.InventoryCategoryId == categoryId);
var inventorySpecifications = dbContext.InventoryCategorySpecifications.Where(x => x.InventoryCategoryId == categoryId).Select(x => x.InventorySpecification);
Stopwatch a = new Stopwatch();
a.Start();
foreach (var item in inventoryItems)
{
var specs = JObject.Parse(item.Attributes);
var specDtos = new List<SpecDto>();
foreach (var inventorySpecification in inventorySpecifications.OrderBy(x => x.DisplayOrder))
{
if (specs.ContainsKey(inventorySpecification.JsonKey))
{
var value = specs.GetValue(inventorySpecification.JsonKey);
var newSpecDto = new SpecDto()
{
Key = inventorySpecification.JsonKey,
Value = displaySpec.ToString()
};
specDtos.Add(newSpecDto);
}
}
var dto = new InventoryItemDto()
{
// create dto
};
inventoryItemDtos.Add(dto);
}
Now it goes crazy slow when I add EF some more columns that I need info from.
In the //create dto area I access some information from other tables
var dto = new InventoryItemDto()
{
// access brand columns
// access company columns
// access branch columns
// access country columns
// access state columns
};
By trying to access these columns in the loop takes 6mins to process 700 rows.
I don't understand why it is so slow, it's the only change I really made and I made sure to eager load everything in.
To me it almost makes me think eager loading is not working, but I don't know how to verify if it is or not.
var inventoryItems = dbContext.InventoryItems.Include(x => x.Branch).ThenInclude(x => x.Company)
.Include(x => x.Branch).ThenInclude(x => x.Country)
.Include(x => x.Branch).ThenInclude(x => x.State)
.Include(x => x.Brand)
.Where(x => x.InventoryCategoryId == categoryId).ToList();
so I thought because of doing this the speed would not be that much different then the original 18-30 seconds.
I would like to speed up the original code too but I am not really sure how to get rid of the dual foreach loops that is probably slowing it down.

First, loops inside loops is a very bad thing, you should refactor that out and make it a single loop. This should not be a problem because inventorySpecifications is declared outside the loop
Second, the line
var inventorySpecifications = dbContext.InventoryCategorySpecifications.Where(x => x.InventoryCategoryId == categoryId).Select(x => x.InventorySpecification);
should end with ToList(), because it's enumerations is happening within the inner foreach, which means that the query is running for each of "inventoryItems"
that should save you a good amount of time

I'm no expert but this part of your second foreach raises a red flag: inventorySpecifications.OrderBy(x => x.DisplayOrder). Because this is getting called inside another foreach it's doing the .OrderBy call every time you iterate over inventoryItems.
Before your first foreach loop, try this: var orderedInventorySpecs = inventorySpecifications.OrderBy(x => x.DisplayOrder); and then use foreach (var inventorySpec in orderedInventorySpecs) and see if it makes a difference.

To help you better understand what EF is running behind the scenes add some logging in to expose the SQL being run which might help you see how/where your queries are going wrong. This can be extremely helpful to help determine if your queries are hitting the DB too often. As a very general rule you want to hit the DB as few times as possible and retrieve only the information you need via the use of .Select() to reduce what is being returned. The docs for the logging are: http://learn.microsoft.com/en-us/ef/core/miscellaneous/logging
I obviously cannot test this and I am a little unsure where your specDto's go once you have them but I assume they become part of the InventoryItemDto?
var itemDtos = new List<ItemDto>();
var inventoryItems = dbContext.InventoryItems.Where(x => x.InventoryCategoryId == categoryId).Select(x => new InventoryItemDto() {
Attributes = x.Attributes,
//.....
// access brand columns
// access company columns
// access branch columns
// access country columns
// access state columns
}).ToList();
var inventorySpecifications = dbContext.InventoryCategorySpecifications
.Where(x => x.InventoryCategoryId == categoryId)
.OrderBy(x => x.DisplayOrder)
.Select(x => x.InventorySpecification).ToList();
foreach (var item in inventoryItems)
{
var specs = JObject.Parse(item.Attributes);
// Assuming the specs become part of an inventory item?
item.specs = inventorySpecification.Where(x => specs.ContainsKey(x.JsonKey)).Select(x => new SpecDto() { Key = x.JsonKey, Value = specs.GetValue(x.JsonKey)});
}
The first call to the DB for inventoryItems should produce one SQL query that will pull all the information you need at once to construct your InventoryItemDto and thus only hits the DB once. Then it pulls the specs out and uses OrderBy() before materialising which means the OrderBy will be run as part of the SQL query rather than in memory. Both those results are materialised via .ToList() which will cause EF to pull the results into memory in one go.
Finally the loop goes over your constructed inventoryItems, parses the Json and then filters the specs based on that. I am unsure of where you were using the specDtos so I made an assumption that it was part of the model. I would recomend checking the performance of the Json work you are doing as that could be contributing to your slow down.
A more integrated approach to using Json as part of your EF models can be seen at this answer: https://stackoverflow.com/a/51613611/621524 however you will still be unable to use those properties to offload execution to SQL as accessing properties that are defined within code will cause queries to fragment and run in several parts.

How to retrieve more than one instance of a unique Id with linq?

I have a post action in my API which creates an order based off some data it retrieves from a view model.
It needs to be able to retrieve all movies from a table of 'movies' that are passed to it via the view model and create an order. The view model passes the action the Ids of the movies it needs to retrieve.
I have a working solution, and when giving the action data like this it works:
{
"movieIds": [34, 35],
"customerId": 21
}
database:
However when I give the action data which contains two or more movies with the same Id, it only ever saves one movie.
{
"movieIds": [34, 34],
"customerId": 21
}
database:
After debugging the code, i found out that it's this linq statement which is causing the problem, it only ever saves one instance of the movie to 'movies'.
movies = _context.Movies.Where(m => newRental.MovieIds.Contains(m.Id)).ToList();
Does anyone know why it does this? and how to construct a linq statement that allows it to save multiple Ids?

If you think in SQL terms, what you ask is quite complex (not too much, but probably impossible for the Entity Framework LINQ translator-to-SQL).
The simplest solution is multiply the rows after the query, C#-side.
var movies = _context.Movies.Where(m => newRental.MovieIds.Contains(m.Id)).ToList();
var movies2 = (from x in newRental.MovieIds
join y in movies on x equals y.Id
select y).ToList();
We join ("inner") the newRental.MovieIds with the movies. In this way the rows are "multiplied" when necessary.

You need to select movies from your list passed in, NOT from the list of movies in the database.
In SQL terms, what you're saying is
select * from movies where movieId in (34,34)
Which will of course return only one row.
What you need instead is to select one movie for each entry in your list. This will be less efficient for longs lists of movies, but I assume that's unlikely to be a huge problem.
movies = newRental.MovieIds
.Select(rm => _context.Movies.FirstOrDefault(m => m.Id==rm)
.Where(x => x != null) //Just make sure no entries are NULL.. optional
.ToList();
That should do what you want.
For a more convoluted, but probably more DB-efficient solution, you could instead do this:
//Get list of matches from DB into a list in one hit.
var possibleMovies = _context.Movies.Where(m=>newRental.MovieIds.Contains(m.Id)).ToList();
//Match up entries in request to DB entries.
movies = newRental.MovieIds
.Select(rm => possibleMovies.FirstOrDefault(m => m.Id==rm))
.Where(x => x != null) //Just make sure no entries are NULL.. optional
.ToList();
And that would fetch all movies in one statement then use that list to match up the requested list. You could almost certainly do this in a more terse way, but this is clear and obvious - when you look at this in 2 month's time it won't confuse you..... :)

You can try below,May this will help you
i guess newRental.MovieIds is array which contain List of MovieIds
For Specific custmore
movies = _context.Movies
.Where(m => newRental.MovieIds.Contains(m.Movie_Id) && m.customerId==21)
.ToList();
For newRental.customerId Only
movies = _context.Movies
.Where(m => newRental.MovieIds.Contains(m.Movie_Id) && newRental.customerId.contains(m.Customer_Id ))
.ToList();

Edit: Do not use this! As xanatos correctly pointed out this is an antipattern. I'll let it stand as it is for learning purposes!
This line of code returns just one element since in _context.Movies is per se only one movie with the id 34 stored. I'd change the line to:
movies = newRental.MovieIds.Select(movieId => _context.Movies.SingleOrDefault(m => m.Id == movieId)).ToList();
This way you gain a result for each new rental no matter if it's stored only once.

Select Contains With CSV But Keep Sort Order

I have the scenario where I have a IList<Guid> in a variable called csv which are also in a specific order that I need to keep. I am then doing a select contains like so I can get back all my topics based in the list of guids I have.
The guids are from a lucene search which are ordered by the original score from each LuceneResult. Which is why I need to keep them in this order.
var results = _context.Topic
.Where(x => csv.Contains(x.Id));
However. I lose the order the guids came in as soon as I do this. Any idea how I can do this but keep the same order I hand the list of guids to the context and get the topics back in the same order based on the topid.Id?
I have tried the following as mentioned below, by doing a join but they still come out in the same order? Please note that I am paging these results too.
var results = _context.Topic
.Join(csv,
topic => topic.Id,
guidFromCsv => guidFromCsv,
(topic, guidFromCsv) => new { topic, guidFromCsv }
)
.Where(x => x.guidFromCsv == x.topic.Id)
.Skip((pageIndex - 1)*pageSize)
.Take(pageSize)
.Select(x=> x.topic);
** UPDATE **
So I have moved away from just using and guid and am attempting to pass in my lucene model which has the score property that I want to order by. Here is what I have
public PagedList<Topic> GetTopicsByLuceneResult(int pageIndex, int pageSize, int amountToTake, List<LuceneSearchModel> luceneResults)
{
var results = _context.Topic
.Join(luceneResults,
topic => topic.Id,
luceneResult => luceneResult.Id,
(topic, luceneResult) => new { topic, luceneResult }
)
.Where(x => x.luceneResult.Id == x.topic.Id)
.OrderByDescending(x => x.luceneResult.Score)
.Skip((pageIndex - 1) * pageSize)
.Take(pageSize)
.Select(x => x.topic);
var topicResults = results.ToList();
// Return a paged list
return new PagedList<Topic>(topicResults, pageIndex, pageSize, topicResults.Count);
}
However I am now getting the following error? Is what I am doing possible?
Unable to create a constant value of type 'LuceneSearchModel'. Only primitive types or enumeration types are supported in this context.

If I understand the question correctly, you want to filter the Topics based on the csv and you want to get back the results in the same order as the csv. If so:
var results = csv
.GroupJoin(_context.Topic, guid => guid, topic => topic.Id,
(guid, topics) => topics)
.SelectMany(topics => topics);
It is important to note that this treats the _context.Topic as an IEnumerable<T>; therefore, it will fetch all topics from the database and perform the GroupJoin on the client side, not on the database.
EDIT: Based on the comment below, this answer is NOT what you want. I'll just leave the answer here for documentation.