We Keep Coding

SQL Code generated incorrectly using Entity Framework while using Left join

I am using Entity framework for executing code for left join and the sql code that is being generated has mapping issue for one parameter
This mapping issue is when i am using IQuerable object on end of the join and the other end is context object
If I use context object instead of Iquerable then the sql code generated is proper.
E.g. DB entity name: order
Property name: OrderId
Mapped to Table Name : Orders & column order_id
If in my Iquerable object if i alias OrderId as order_id that resolves the issue as both my alias and my db column name is same so there is no mapping issue.
If in my Iquerable object if i alias OrderId as order_id that resolves the issue as both my alias and my db column name is same so there is no mapping issue.
var query =(from o in context.Orders
where o.BranchKey == branchKey
Select new
{
o.OrderId,
o.BranchKey
}
);
query = query.Skip(firstRecord).Take(count).AsQueryable();
var orderDetails = (from o in query
join od in context.OrderDetails
on new {o.orderId , o.BranchKey } equals new { od.OrderId, od.BranchKey } into ordergroup
from grp in ordergroup.DefaultIfEmpty()
select new
{
o.order_id,
o.BranchKey
}
)
**Expected Result**
exec sp_executesql N'SELECT [t].[OrderId], [t].[BranchKey]
FROM (SELECT [o].[order_id] AS [OrderId], [o].[BranchKey]
FROM [dbo].[Orders] AS [o]
WHERE ((([o].[BranchKey] = #__branchKey_0)
OFFSET #__p_4 ROWS FETCH NEXT #__p_5 ROWS ONLY
) AS [t]
LEFT JOIN [dbo].[order_details] AS **[od] ON ([t].[orderId] = [od].[order_id])** AND ([t].[BranchKey] = [od].[BranchKey])
**Actual Result**
exec sp_executesql N'SELECT [t].[OrderId], [t].[BranchKey]
FROM (
SELECT [o].[order_id] AS [OrderId], [o].[BranchKey]
FROM [dbo].[Orders] AS [o]
WHERE ((([o].[BranchKey] = #__branchKey_0)
OFFSET #__p_4 ROWS FETCH NEXT #__p_5 ROWS ONLY
) AS [t]
LEFT JOIN [dbo].[order_details] AS **[od] ON ([t].[order_id] = [od].[order_id])** AND ([t].[BranchKey] = [od].[BranchKey])

Entity Framework 6 Stored Procedure C# side missing created column

I have the following stored procedure. I implemented the UNION to avoid writing 2 stored procedures. My main issue is that EF will not recognize the dynamically created column, even if I add it manually to the object_result class that queries the model. Does anyone know how to get such dynamically generated columns to work?
(What's weird is the RANK() VolumeRank below does work. But RefYear does not map).
SQL:
BEGIN
SET NOCOUNT ON;
SET FMTONLY OFF;
SELECT *
FROM
(SELECT
B.OS, B.DS, COUNT(DISTINCT A.PO) Volume,
RANK() OVER (ORDER BY COUNT(DISTINCT A.PO) DESC) VolumeRank,
YEAR(GETDATE()) RefYear
FROM
DB.DW.LoadData AS A
JOIN
DB.DW.Geo AS B ON A.PO = B.PO
INNER JOIN
DW.DMaster AS C ON A.LoadDateKey = C.DateKey
WHERE
C.IsT12Weeks_LastWeek = 1
AND a.PrimaryCustomerID = #CustomerId
GROUP BY
B.OS, B.DS) AS A
WHERE
VolumeRank <= 3
UNION
SELECT *
FROM
(SELECT
B.OS, B.DS,
COUNT(DISTINCT A.PO) Volume,
RANK() OVER (ORDER BY COUNT(DISTINCT A.PO) DESC) VolumeRank,
YEAR(getdate())-1 RefYear
FROM
DB.DW.LoadData AS A
JOIN
DB.DW.Geo AS B ON A.PO = B.PO
INNER JOIN
DW.DMaster AS C ON A.LoadDateKey = C.DateKey
WHERE
PrimaryCustomerID = #CustomerId
AND FWeek >= #FirstWeek
AND FWeek <= #LastWeek
AND FiscalYear = #FiscalYear
GROUP BY
B.OS, B.DS) AS A
WHERE
VolumeRank <= 3
ORDER BY
RefYear DESC, Volume DESC
END
C#:
List<spGetT_Result> result = new List<spGetT_Result>();
using (var db = new T_Entities())
{
result = db.spGetT(eid, firstWeek, lastWeek, year).ToList();
}
Model class includes this line, but it's always null when I run:
public Nullable<int> RefYear { get; set; }

LinqToSQL - ToList() appears to be very slow

I'm newish to LinqToSQL and the project that I am working on cannot be changed to something else. I am translating some old SQL code to Linq. Not being that hot at linq, I used Linqer to do the translation for me. The query took about 90 seconds to run, so I thought it must be the linqToSQL. However, when I copied the query that the LinqToSQL produced and ran an ExecuteQuery on the datacontext it was super quick as I expected. I've copied the full queries, rather than trying to distil it down, but it looks like the issue is with something LinqToSQL is doing behind the scenes.
To summarise, if I copy the T-SQL created by linq and run
var results = DB.ExecuteQuery<InvoiceBalanceCheckDTO.InvoiceBalanceCheck>(#"T-SQL created by Linq - see below").ToList()
it completes with expected results in about 0.5 seconds.
It runs about the same time directly in SSMS. However, if I use the linqToSQL code that creates the T-SQL and do ToList() it takes ages. The result is only 9 records, although without the constraint to check the balance <> 0, there would be around 19,000 records. It's as if it's getting all 19,000 and then checking <> 0 after it's got the records.
I have also changed the Linq to project into the class used above, rather than to an anonymous type, but it makes not difference
This is the original SQL :
SELECT InvoiceNum, Max(AccountCode), Sum(AmountInc) AS Balance
FROM
(SELECT InvoiceNum, AccountCode, AmountInc From TourBookAccount WHERE AccDetailTypeE IN(20,30) AND InvoiceNum >= 1000
UNION ALL
SELECT InvoiceNum, '<no matching invoice>' AS AccountCode, AccountInvoiceDetail.AmountInc
FROM AccountInvoiceDetail
INNER JOIN AccountInvoice ON AccountInvoiceDetail.InvoiceID=AccountInvoice.InvoiceID
WHERE AccDetailTypeE IN(20,30)
AND InvoiceNum >= 1000
) as t
GROUP BY InvoiceNum
HAVING (Sum(t.AmountInc)<>0)
ORDER BY InvoiceNum
and this is the linq
var test = (from t in
(
//this gets the TourBookAccount totals
from tba in DB.TourBookAccount
where
detailTypes.Contains(tba.AccDetailTypeE) &&
tba.InvoiceNum >= dto.CheckInvoiceNumFrom
select new
{
InvoiceNum = tba.InvoiceNum,
AccountCode = tba.AccountCode,
Balance = tba.AmountInc
}
)
.Concat //note that concat, since it's possible that the AccountInvoice record does not actually exist
(
//this gets the Invoice detail totals.
from aid in DB.AccountInvoiceDetail
where
detailTypes.Contains(aid.AccDetailTypeE) &&
aid.AccountInvoice.InvoiceNum >= dto.CheckInvoiceNumFrom &&
select new
{
InvoiceNum = aid.AccountInvoice.InvoiceNum,
AccountCode = "<No Account Records>",
Balance = aid.AmountInc
}
)
group t by t.InvoiceNum into g
where Convert.ToDecimal(g.Sum(p => p.Balance)) != 0m
select new
{
InvoiceNum = g.Key,
AccountCode = g.Max(p => p.AccountCode),
Balance = g.Sum(p => p.Balance)
}).ToList();
and this is the T-SQL that the linq produces
SELECT [t5].[InvoiceNum], [t5].[value2] AS [AccountCode], [t5].[value3] AS [Balance]
FROM (
SELECT SUM([t4].[AmountInc]) AS [value], MAX([t4].[AccountCode]) AS [value2], SUM([t4].[AmountInc]) AS [value3], [t4].[InvoiceNum]
FROM (
SELECT [t3].[InvoiceNum], [t3].[AccountCode], [t3].[AmountInc]
FROM (
SELECT [t0].[InvoiceNum], [t0].[AccountCode], [t0].[AmountInc]
FROM [dbo].[TourBookAccount] AS [t0]
WHERE ([t0].[AccDetailTypeE] IN (20, 30)) AND ([t0].[InvoiceNum] >= 1000)
UNION ALL
SELECT [t2].[InvoiceNum],'<No Account Records>' AS [value], [t1].[AmountInc]
FROM [dbo].[AccountInvoiceDetail] AS [t1]
INNER JOIN [dbo].[AccountInvoice] AS [t2] ON [t2].[InvoiceID] = [t1].[InvoiceID]
WHERE ([t1].[AccDetailTypeE] IN (20, 30)) AND ([t2].[InvoiceNum] >= 1000)
) AS [t3]
) AS [t4]
GROUP BY [t4].[InvoiceNum]
) AS [t5]
WHERE [t5].[value] <> 0

I would bet money, that the problem is in this line:
where Convert.ToDecimal(g.Sum(p => p.Balance)) != 0m
What is probably happening, is that it can't translate this to SQL and silently tries to get all rows from db to memory, and then do filtering on in memory objects (LINQ to objects)
Maybe try to change this to something like:
where g.Sum(p=>.Balance!=0)

Well, the answer turned out not to be LinqToSQL itself (although possibly the way it creates the query could be blamed) , but the way SQL server handles the query. When I was running the query on the database to check speed (and running the created T=SQL in DB.ExecuteQuery) I had all the variables hardcoded. When I changed it to use the exact sql that Linq produces (i.e. with variables that are substituted) it ran just as slow in SSMS.
Looking at the execution plans of the two, they are quite different. A quick search on SO brought me to this page : Why does a parameterized query produces vastly slower query plan vs non-parameterized query which indicated that the problem was SQL server's "Parameter sniffing".
The culprit turned out to be the "No Account Records" string
For completeness, here is the generated T-SQL that Linq creates.
Change #p10 to the actual hardcoded string, and it's back to full speed !
In the end I just removed the line from the linq and set the account code afterwards and all was good.
Thanks #Botis,#Blorgbeard,#ElectricLlama & #Scott for suggestions.
DECLARE #p0 as Int = 20
DECLARE #p1 as Int = 30
DECLARE #p2 as Int = 1000
DECLARE #p3 as Int = 20
DECLARE #p4 as Int = 30
DECLARE #p5 as Int = 1000
DECLARE #p6 as Int = 40
DECLARE #p7 as Int = 10
DECLARE #p8 as Int = 0
DECLARE #p9 as Int = 1
DECLARE #p10 as NVarChar(4000)= '<No Account Records>' /*replace this parameter with the actual text in the SQl and it's way faster.*/
DECLARE #p11 as Decimal(33,4) = 0
SELECT [t5].[InvoiceNum], [t5].[value2] AS [AccountCode], [t5].[value3] AS [Balance]
FROM (
SELECT SUM([t4].[AmountInc]) AS [value], MAX([t4].[AccountCode]) AS [value2], SUM([t4].[AmountInc]) AS [value3], [t4].[InvoiceNum]
FROM (
SELECT [t3].[InvoiceNum], [t3].[AccountCode], [t3].[AmountInc]
FROM (
SELECT [t0].[InvoiceNum], [t0].[AccountCode], [t0].[AmountInc]
FROM [dbo].[TourBookAccount] AS [t0]
WHERE ([t0].[AccDetailTypeE] IN (#p0, #p1)) AND ([t0].[InvoiceNum] >= #p2)
UNION ALL
SELECT [t2].[InvoiceNum], #p10 AS [value], [t1].[AmountInc]
FROM [dbo].[AccountInvoiceDetail] AS [t1]
INNER JOIN [dbo].[AccountInvoice] AS [t2] ON [t2].[InvoiceID] = [t1].[InvoiceID]
WHERE ([t1].[AccDetailTypeE] IN (#p3, #p4)) AND ([t2].[InvoiceNum] >= #p5) AND ([t2].[InvoiceStatusE] <= #p6) AND ([t2].[InvoiceTypeE] = #p7) AND ([t1].[BookNum] <> #p8) AND ([t1].[AccDetailSourceE] = #p9)
) AS [t3]
) AS [t4]
GROUP BY [t4].[InvoiceNum]
) AS [t5]
WHERE [t5].[value] <> #p11
SELECT [t5].[InvoiceNum], [t5].[value2] AS [AccountCode], [t5].[value3] AS [Balance]
FROM (
SELECT SUM([t4].[AmountInc]) AS [value], MAX([t4].[AccountCode]) AS [value2], SUM([t4].[AmountInc]) AS [value3], [t4].[InvoiceNum]
FROM (
SELECT [t3].[InvoiceNum], [t3].[AccountCode], [t3].[AmountInc]
FROM (
SELECT [t0].[InvoiceNum], [t0].[AccountCode], [t0].[AmountInc]
FROM [dbo].[TourBookAccount] AS [t0]
WHERE ([t0].[AccDetailTypeE] IN (20, 30)) AND ([t0].[InvoiceNum] >= 1000)
UNION ALL
SELECT [t2].[InvoiceNum], '<No Account Records>' AS [value], [t1].[AmountInc]
FROM [dbo].[AccountInvoiceDetail] AS [t1]
INNER JOIN [dbo].[AccountInvoice] AS [t2] ON [t2].[InvoiceID] = [t1].[InvoiceID]
WHERE ([t1].[AccDetailTypeE] IN (20, 30)) AND ([t2].[InvoiceNum] >= 0) AND ([t2].[InvoiceStatusE] <= 40) AND ([t2].[InvoiceTypeE] = 10) AND ([t1].[BookNum] <> 0) AND ([t1].[AccDetailSourceE] = 1)
) AS [t3]
) AS [t4]
GROUP BY [t4].[InvoiceNum]
) AS [t5]
WHERE [t5].[value] <> 0

How to convert following SQL Query into a LINQ to SQL query

I have following SQL Query and would like to convert to LINQ to SQL which I will use in entity framework 5.0
var internationalDesksList =
from internationalDesks in _context.InternationalDesks
from subsection in
_context.Subsections.Where(
s =>
internationalDesks.EBALocationId == s.LocationId ||
internationalDesks.FELocationId == s.LocationId).DefaultIfEmpty()
where subsection.PublicationId == 1
select new {internationalDesks.Id, subsection.LocationId};
I have referred the following posts and answers. Though no luck.
LINQ to SQL - Left Outer Join with multiple join conditions Linq
left join on multiple (OR) conditions
When I tried this query in LINQPad I got the following answer which is correct.
-- Region Parameters
DECLARE #p0 Int = 1
-- EndRegion
SELECT [t0].[Id], [t1].[Id] AS [Id1]
FROM [InternationalDesks] AS [t0]
LEFT OUTER JOIN [Subsection] AS [t1] ON (([t0].[FELocationId]) = [t1].[LocationId]) OR (([t0].[EBALocationId]) = [t1].[LocationId])
WHERE [t1].[PublicationId] = #p0
However in entity framework 5 ( DBContext ) it is not providing me with the correct query. When I checked in SQL profiler all columns in subsection table is selected. That's it.
Following is the result:
SELECT
[Extent1].[Id] AS [Id],
[Extent1].[Description] AS [Description],
[Extent1].[PracticeAreaId] AS [PracticeAreaId],
[Extent1].[LocationId] AS [LocationId],
...
FROM [dbo].[Subsection] AS [Extent1]
Don't know what could be problem. Please help me.

With LINQ you can't do LEFT OUTER JOIN on some boolean expression, only equijoins are supported. So, you can generate CROSS JOIN this way:
var internationalDesksList =
from internationalDesks in _context.InternationalDesks
from subsection in _context.Subsections
where subsection.PublicationId == 1 &&
(internationalDesks.EBALocationId == subsection.LocationId ||
internationalDesks.FELocationId == subsection.LocationId)
select new {
internationalDesks.Id,
subsection.LocationId
};
EF 5 will generate following SQL:
SELECT
[Extent1].[Id] AS [Id],
[Extent2].[LocationId] AS [LocationId]
FROM [dbo].[InternationalDesks] AS [Extent1]
CROSS JOIN [dbo].[Subsections] AS [Extent2]
WHERE (1 = [Extent2].[PublicationId]) AND
([Extent1].[EBALocationId] = [Extent2].[LocationId] OR
[Extent1].[FELocationId] = [Extent2].[LocationId])
As you can see, only required columns are selected. I also checked this query in LINQ to SQL - following query is generated:
DECLARE #p0 Int = 1
SELECT [t0].[Id], [t1].[LocationId]
FROM [InternationalDesks] AS [t0], [Subsections] AS [t1]
WHERE ([t1].[PublicationId] = #p0) AND
(([t0].[EBALocationId] = [t1].[LocationId]) OR
([t0].[FELocationId] = [t1].[LocationId]))

convert xml string in a sql table to dynamic columns

I have two tables (using table variables for illustration. You can run these directly in management studio) that are related by the Id column.
Items in the first table has some standard set of columns and the second table has some extended parameter data for the same record. I'm storing the extended set as xml as it is dynamic in all aspects (different per product or new values being added etc).
I'm able to join these two tables and flatten out the column list as you can see in the example below. But my query requires to have the dynamic columns be defined beforehand. I would like to have this truly dynamic in the sense that if I were to add a new column in the #extended table, it should automatically come out as a new column in the output column list.
Basically the list of additional columns should be determined by the xml for that record. column name should be the xml tag and value should be value for the xml tag for each id.
Any pointers? (and can it be fast too with around 100k records or more in each table)
declare #standard table
(
Id INT,
Column1 varchar(10),
Column2 varchar(10),
Column3 varchar(10)
)
declare #extended table
(
Id INT,
column1 xml
)
insert into #standard values (1,'11', '12', '13')
insert into #standard values (2,'21', '22', '23')
insert into #extended values (1,'<FieldSet><Field><id>1</id><column4>1x</column4><column5>4x</column5></Field></FieldSet>')
insert into #extended values (2,'<FieldSet><Field><id>2</id><column4>2x</column4><column5>5x</column5></Field></FieldSet>')
select s.column1, s.column2,
(
SELECT Item2.value('(column4)[1]', 'varchar(50)')
FROM
e.column1.nodes('/FieldSet') AS T(Item)
CROSS APPLY e.column1.nodes('/FieldSet/Field') AS T2(Item2)
) column4,
(
SELECT Item2.value('(column5)[1]', 'varchar(50)')
FROM
e.column1.nodes('/FieldSet') AS T(Item)
CROSS APPLY e.column1.nodes('/FieldSet/Field') AS T2(Item2)
) column5
from #extended e
join #standard s on s.Id = e.Id

First off you can simplify your current query a bit.
select s.column1,
s.column2,
e.column1.value('(/FieldSet/Field/column4)[1]', 'varchar(50)') as column4,
e.column1.value('(/FieldSet/Field/column5)[1]', 'varchar(50)') as column5
from extended as e
join standard as s
on s.Id = e.Id
To do what you want will not be easy or fast. You need to get a list of all name/value pairs in your XML.
select T1.X.value('.', 'int') as Id,
T2.X.value('local-name(.)', 'sysname') as Name,
T2.X.value('.', 'varchar(10)') as Value
from extended as e
cross apply e.column1.nodes('/FieldSet/Field/id') as T1(X)
cross apply e.column1.nodes('/FieldSet/Field/*[position() > 1]') as T2(X)
Use that in a pivot query and join to standard.
select S.column1,
S.column2,
P.column4,
P.column5
from standard as s
inner join
(
select id, P.column4, P.column5
from (
select T1.X.value('.', 'int') as Id,
T2.X.value('local-name(.)', 'sysname') as Name,
T2.X.value('.', 'varchar(10)') as Value
from extended as e
cross apply e.column1.nodes('/FieldSet/Field/id') as T1(X)
cross apply e.column1.nodes('/FieldSet/Field/*[position() > 1]') as T2(X)
) as e
pivot (min(Value) for Name in (column4, column5)) P
) P
on S.Id = P.Id
To do this with a dynamic number of columns returned you need to build this pivot query dynamically.
Store the Name/Value pairs in temp table, use that table to figure out the columns you need and to build your query.
create table #ext
(
Id int,
Name sysname,
Value varchar(10),
primary key(Id, Name)
)
insert into #ext(Id, Name, Value)
select T1.X.value('.', 'int') as Id,
T2.X.value('local-name(.)', 'sysname') as Name,
T2.X.value('.', 'varchar(10)') as Value
from extended as e
cross apply e.column1.nodes('/FieldSet/Field/id') as T1(X)
cross apply e.column1.nodes('/FieldSet/Field/*[position() > 1]') as T2(X)
declare #SQL nvarchar(max)
set #SQL =
'select S.column1,
S.column2,
[COLLIST]
from standard as s
inner join
(
select id, [COLLIST]
from #ext as e
pivot (min(Value) for Name in ([COLLIST])) P
) P
on S.Id = P.Id'
declare #ColList nvarchar(max)
set #ColList =
(select ','+Name
from #ext
group by Name
for xml path(''), type).value('.', 'nvarchar(max)')
set #SQL = replace(#SQL, '[COLLIST]', stuff(#ColList, 1, 1, ''))
exec (#SQL)
drop table #ext

I hope it will help you
SELECT #COUNT_XML=0
SELECT #COUNT_XML=(SELECT #xxxxx_GROUP_ID.value('count(/NewDataSet/position/ID)', 'INT'))
IF(#COUNT_XML > 0)
BEGIN
IF OBJECT_ID('tempdb..#TBL_TEMPOSITION') IS NOT NULL
DROP TABLE #TBL_TEMPOSITION
CREATE TABLE #TBL_TEMPOSITION (ID NUMERIC(18,0))
INSERT INTO #TBL_TEMPOSITION (ID)
SELECT XMLxxxxGroup.PositionGPItem.value('.','NUMERIC(18,0)')
FROM #xxxxx_GROUP_ID.nodes('/NewDataSet/position/ID') AS XMLPositionGroup(PositionGPItem)
SELECT #emp_cond =#emp_cond+ N' AND CM.STATIC_EMP_INFO.POSITION_ID IN (SELECT ID FROM #TBL_TEMPOSITION) '
END