We Keep Coding

Linq query taking too long to process results

I have 3 tables
ERPEntry
ERPEntryType
ERPApp
I am trying to get data from these 3 tables using the below queries:.. The table ERPApp is in different context than the other 2 tables, that is the reason i am using 2 queries
var res = (from s in ERPDB.ERPEntrys
join t in ERPDB.ERPEntryTypes
on s.EntryTypeID equals t.EntryTypeID
where s.UserIDAdded == '250176'
select new {s.EntryTypeID, s.DateAdded, t.EntryTypeName, s.AppID }).OrderByDescending(d => d.DateAdded).Take(10).ToArray();
var erpResult = (from a in APPDB.ERPApps.AsEnumerable()
join b in res on a.AppId equals b.AppId
select new ERPInfo
{
EntryId = b.EntryID,
EntryType = b.EntryTypeName,
ERPApp = a.ApplicationName,
DateAdded = b.DateAdded
}).ToList();
I got the desired result but the 2nd query is taking almost 4 minutes to process and return result into varibale erpResult... any help on how can i resolve this performace issue ?

First, the answer:
var res = (from s in ERPDB.ERPEntries
join t in ERPDB.ERPEntryTypes
on s.EntryTypeID equals t.EntryTypeID
where s.UserIdAdded == "250176"
select new { s.EntryID, s.EntryTypeID, s.DateAdded, t.EntryTypeName, s.AppId })
.OrderByDescending(d => d.DateAdded).Take(10).ToArray();
/* Which immediately executes:
exec sp_executesql N'SELECT TOP (10) [t0].[EntryID], [t0].[EntryTypeID],
[t0].[DateAdded], [t1].[EntryTypeName], [t0].[AppId]
FROM [dbo].[ERPEntry] AS [t0]
INNER JOIN [dbo].[ERPEntryType] AS [t1] ON [t0].[EntryTypeID] = [t1].
[EntryTypeID]
WHERE [t0].[UserIdAdded] = #p0
ORDER BY [t0].[DateAdded] DESC',N'#p0 varchar(8000)',#p0='250176'
*/
// Get the distinct AppID values in res
// Executes in memory
var distinctApps = (from r in res select r.AppId).Distinct();
// Query APPDB with the distinct values
var matchingApps = (from a in APPDB.ERPApps where distinctApps
.Contains(a.AppId) select new { a.AppId, a.ApplicationName }).ToArray();
/* Which immediately executes this efficient query:
exec sp_executesql N'SELECT [t0].[AppId], [t0].[ApplicationName]
FROM [dbo].[ERPApp] AS [t0]
WHERE [t0].[AppId] IN (#p0, #p1, #p2, #p3)',N'#p0 bigint,#p1 bigint,#p2
bigint,#p3 bigint',#p0=101,#p1=123,#p2=125,#p3=129
*/
var erpResultWithAppNames = (from a in matchingApps
join b in res on a.AppId equals b.AppId
select new
{
EntryId = b.EntryID,
EntryType = b.EntryTypeName,
ERPApp = a.ApplicationName,
DateAdded = b.DateAdded
}).ToList();
Extra notes:
As already mentioned in the comments, the call to .AsEnumerable() on the APPDB.ERPApps table causes the whole table to be loaded into memory as objects.
Facts:
You had to load 25.000 rows into memory (and convert them to objects),
You are not only loading 25.000 rows. You are loading 25.000 x 173 cells and create 25.000 objects with 173 fields each.
if you could just load the two fields you need (AppId and ApplicationName) instead of all the 173 fields (with whatever data they have) the performance would improve, but would still be inefficient considering what you are trying to achieve.
The current performance problem is in part due to transferring the whole table with all its 173 fields to your server, and in part to mapping all these rows to objects and fields to properties. The impact of each can be measured as further research.
Linq To SQL. When checked with SqlProfiler:
from a in APPDB.ERPApps.AsEnumerable() join b ...
// PROFILED QUERY:
SELECT [t0].[AppId], [t0].[ApplicationName], [t0].[Field1], [t0].[Field2],
[t0].[Field3], [t0].[Field4], [t0].[Field5], [t0].[Field6], [t0].[Field7],
[t0].[Field8], [t0].[Field9], ... ... ... ... [t0].[Field176], [t0].[Field173],
FROM [dbo].[ERPApp] AS [t0]
But:
var erpResult = (from a in APPDB.ERPApps
select new { AppId = a.AppId, ApplicationName = a.ApplicationName }
).ToArray();
// PROFILED QUERY:
SELECT [t0].[AppId], [t0].[ApplicationName]
FROM [dbo].[ERPApp] AS [t0]

Linq return distinct values from table if not exist in another

I am trying to return all distinct rows from Staging below where Staging.CenterCode does not exist in Centers.CenterCode.
At the moment Stagings has around 850 distinct CenterCodes and Centers is empty so I should be getting all of the distinct rows, but count begs to differ :)
Any ideas?
var query =
(from s in db.Stagings
join t in db.Centers on s.CenterCode equals t.CenterCode into tj
from t in tj.DefaultIfEmpty()
where s.CenterCode != t.CenterCode
select s.CenterCode).Distinct();
var c = query.Count();
I only need the unique columns from staging so not sure if I actually need a join with the above as I am not ever using data returned from Centers - I have however tried both and get the same 0 value for count.
Any ideas?

I would not use a join, but use a Contains.
var centerCodesQuery = db.Centers.CenterCode
.Select(x => x.CenterCode);
var query = db.Staging
.Where(x => !centerCodesQuery.Contains(x.CenterCode))
.Select(x => x.CenterCode)
.Distinct();
var c = query.Count();

the join is an inner join. So, if none of the rows in 1 table match the other table on the specified identifier then it will return 0. In yours you are trying to join 1 table with 850 distinct rows with an empty table. This will return 0.
If you actually want to return only those rows in 1 table that aren't in another you can use Except:
var query = (from s in db.Stagings
select s.CenterCode)
.Except(from t in db.Centers
select t.CenterCode);
var c = query.Count();

Looks like you are trying to implement antijoin via left outer join, which is one of the possible ways, but in order to make it work, you need to change
where s.CenterCode != t.CenterCode
to
where t == null

Linq-to-SQL: arithmetic operation on consecutive elements

For example, I have a table:
Date |Value
----------|-----
2015/10/01|5
2015/09/01|8
2015/08/01|10
Is there any way using Linq-to-SQL to get a new sequence which will be an arithmetic operation between consecutive elements in the previously ordered set (for example, i.Value - (i-1).Value)? It must be executed on SQL Server 2008 side, not application side.
For example dataContext.GetTable<X>().OrderByDescending(d => d.Date).Something(.......).ToArray(); should return 3, 2.
Is it possible?

You can try this:
var q = (
from i in Items
orderby i.ItemDate descending
let prev = Items.Where(x => x.ItemDate < i.ItemDate).FirstOrDefault()
select new { Value = i.ItemValue - (prev == null ? 0 : prev.ItemValue) }
).ToArray();
EDIT:
If you slightly modify the above linq query to:
var q = (from i in Items
orderby i.ItemDate descending
let prev = Items.Where(x => x.ItemDate < i.ItemDate).FirstOrDefault()
select new { Value = (int?)i.ItemValue - prev.ItemValue }
).ToArray();
then you get the following TSQL query sent to the database:
SELECT ([t0].[ItemValue]) - ((SELECT [t2].[ItemValue]
FROM (SELECT TOP (1) [t1].[ItemValue]
FROM [Items] AS [t1]
WHERE [t1].[ItemDate] < [t0].[ItemDate]) AS [t2]
)) AS [Value]
FROM [Items] AS [t0]
ORDER BY [t0].[ItemDate] DESC
My guess now is if you place an index on ItemDate field this shouldn't perform too bad.

I wouldn't let SQL do this, it would create an inefficient SQL query (I think).
I could create a stored procedure, but if the amount of data is not too big I can also use Linq to objects:
List<x> items=dataContext.GetTable<X>().OrderByDescending(d => d.Date).ToList();//Bring data to memory
var res = items.Skip(1).Zip(items, (cur, prev) => cur.Value - prev.Value);
At the end, I might use a foreach for readability

Linq-to-entities, get results + row count in one query

I've seen multiple questions about this matter, however they were 2 years (or more) old, so I'd like to know if anything changed about this.
The basic idea is to populate a gridview and create custom paging. So, I need the results and row count as well.
In SQL this would be something like:
SELECT COUNT(id), Id, Name... FROM ... WHERE ...
Getting everything in a nice simple query. However, I'd like to be consistent and use Linq2Entities.
So far I'm using the approach with two queries (against sql server), because it just works. I would like to optimize it though and use a single query instead.
I've tried this:
var query = from o in _db.Products
select o;
var prods = from o in query
select new
{
Count = query.Count(),
Products = query
};
This produces a very nasty and long query with really unnecessary cross joins and other stuff which I don't really need or want.
Is there a way to get the paged results + count of all entities in a one simple query? What is the recommended approach here?
UPDATE:
Just tried FutureQueries and either I'm doing something wrong, or it actually executes two queries. This shows my sql profiler:
-- Query #1
SELECT
[GroupBy1].[A1] AS [C1]
FROM ( SELECT
COUNT(1) AS [A1]
FROM [dbo].[Products] AS [Extent1]
WHERE 1 = [Extent1].[CategoryID]
) AS [GroupBy1];
And next row:
-- Query #1
SELECT
[Extent1].[ID] AS [ID],
[Extent1].[Name] AS [Name],
[Extent1].[Price] AS [Price],
[Extent1].[CategoryID] AS [CategoryID]
FROM [dbo].[Products] AS [Extent1]
WHERE 1 = [Extent1].[CategoryID];
The C# code:
internal static List<Product> GetProducts(out int _count)
{
DatabaseEntities _db = new DatabaseEntities();
var query = from o in _db.Products
where o.CategoryID == 1
select o;
var count = query.FutureCount();
_count = count.Value;
return query.Future().ToList();
}
Did I miss something? According to my profiler it does exactly the same except that added row in the query (-- Query #1).

Have a look at Future Queries to do this in EntityFramework.Extended. The second example on that linked page uses FutureCount() to do exactly what you want. Adapted here:
var q = db.Products.Where(p => ...);
var qCount = q.FutureCount();
var qPage = q.Skip((pageNumber-1)*pageSize).Take(pageSize).Future();
int total = qCount.Value; // Both queries are sent to the DB here.
var tasks = qPage.ToList();

this 'EntityFramework.Extended' library is no longer supported use this one instead:
entityframework-plus and go here:
https://entityframework-plus.net/query-future to see how you can get count and records
in the same query.

Linq2Sql: Get every N'th row [duplicate]

Anybody know how to write a LINQ to SQL statement to return every nth row from a table? I'm needing to get the title of the item at the top of each page in a paged data grid back for fast user scanning. So if i wanted the first record, then every 3rd one after that, from the following names:
Amy, Eric, Jason, Joe, John, Josh, Maribel, Paul, Steve, Tom
I'd get Amy, Joe, Maribel, and Tom.
I suspect this can be done... LINQ to SQL statements already invoke the ROW_NUMBER() SQL function in conjunction with sorting and paging. I just don't know how to get back every nth item. The SQL Statement would be something like WHERE ROW_NUMBER MOD 3 = 0, but I don't know the LINQ statement to use to get the right SQL.

Sometimes, TSQL is the way to go. I would use ExecuteQuery<T> here:
var data = db.ExecuteQuery<SomeObjectType>(#"
SELECT * FROM
(SELECT *, ROW_NUMBER() OVER (ORDER BY id) AS [__row]
FROM [YourTable]) x WHERE (x.__row % 25) = 1");
You could also swap out the n:
var data = db.ExecuteQuery<SomeObjectType>(#"
DECLARE #n int = 2
SELECT * FROM
(SELECT *, ROW_NUMBER() OVER (ORDER BY id) AS [__row]
FROM [YourTable]) x WHERE (x.__row % #n) = 1", n);

Once upon a time, there was no such thing as Row_Number, and yet such queries were possible. Behold!
var query =
from c in db.Customers
let i = (
from c2 in db.Customers
where c2.ID < c.ID
select c2).Count()
where i%3 == 0
select c;
This generates the following Sql
SELECT [t2].[ID], [t2]. --(more fields)
FROM (
SELECT [t0].[ID], [t0]. --(more fields)
(
SELECT COUNT(*)
FROM [dbo].[Customer] AS [t1]
WHERE [t1].[ID] < [t0].[ID]
) AS [value]
FROM [dbo].[Customer] AS [t0]
) AS [t2]
WHERE ([t2].[value] % #p0) = #p1

Here's an option that works, but it might be worth checking that it doesn't have any performance issues in practice:
var nth = 3;
var ids = Table
.Select(x => x.Id)
.ToArray()
.Where((x, n) => n % nth == 0)
.ToArray();
var nthRecords = Table
.Where(x => ids.Contains(x.Id));

Just googling around a bit I haven't found (or experienced) an option for Linq to SQL to directly support this.
The only option I can offer is that you write a stored procedure with the appropriate SQL query written out and then calling the sproc via Linq to SQL. Not the best solution, especially if you have any kind of complex filtering going on.

There really doesn't seem to be an easy way to do this:
How do I add ROW_NUMBER to a LINQ query or Entity?
How to find the ROW_NUMBER() of a row with Linq to SQL
But there's always:
peopleToFilter.AsEnumerable().Where((x,i) => i % AmountToSkipBy == 0)
NOTE: This still doesn't execute on the database side of things!

This will do the trick, but it isn't the most efficient query in the world:
var count = query.Count();
var pageSize = 10;
var pageTops = query.Take(1);
for(int i = pageSize; i < count; i += pageSize)
{
pageTops = pageTops.Concat(query.Skip(i - (i % pageSize)).Take(1));
}
return pageTops;
It dynamically constructs a query to pull the (nth, 2*nth, 3*nth, etc) value from the given query. If you use this technique, you'll probably want to create a limit of maybe ten or twenty names, similar to how Google results page (1-10, and Next), in order to avoid getting an expression so large the database refuses to attempt to parse it.
If you need better performance, you'll probably have to use a stored procedure or a view to represent your query, and include the row number as part of the stored proc results or the view's fields.