We are facing a production issue with oracle db where the numeric postfix of command parameter in sql issued by EF keeps on increasing when LINQ is being used. For example, when run multiple times, the linq query
var collection = from obj in Query where obj.Val1==val1 select obj;
gets translated as (not exact, but sufficient to describe the idea)
select * from Table1 where Column1=#p__linq__0
select * from Table1 where Column1=#p__linq__1
select * from Table1 where Column1=#p__linq__2
...
select * from Table1 where Column1=#p__linq__1003
...
As a result, the oracle shared pool cache runs out of memory.
Does anyone have any idea of it?
Related
So basically I have a table containing a set of data. This data is then joined onto an organisation table to which multiple users can be apart of. Im then trying to get all files in the table where the user executing the query, has permission to access the organisation. To do this I'm using a where clause that checks the users permissions from the application, to the files that have them organisations linked. Im then selecting the top 100 results and counting the records returned. (I want to see if the user has access to 100+ files over all the organisations).
The problem is when I use the following LINQ query:
(from f in File
join o in Organisation on f.OrganisationId equals o.Id
where permissions.Contains(o.Id.ToString())
select f).Take(100).Count();
The take and the count aren't executed on the SQL server and are run in memory when I try a contains on a list which should convert to an IN (VALUES) query on SQL. I have 70,000+ File records and this is very slow and times out on a web server. This is expected as Entity Framework core is in early stages and does not support moderate or advanced LINQ queries yet.
My question is, is there a better alternative to raw SQL queries while still being able to filter by an array of items and still using Entity Framework core v1.1? Thanks.
Edit: I tried updating to the latest version, this still did not solve my issue as I still got the following output.
The LINQ expression '{permissions => Contains([o].Id.ToString())}' could not be translated and will be evaluated locally.
The LINQ expression 'Contains([o].Id.ToString())' could not be translated and will be evaluated locally.
The LINQ expression 'Take(__p_1)' could not be translated and will be evaluated locally.
The LINQ expression 'Count()' could not be translated and will be evaluated locally.
The warnings are misleading - the problem is the ToString() call which causes client evaluation of the query.
The following should produce the intended SQL query:
var idList = permissions.Select(int.Parse);
var result = (
from f in File
join o in Organisation on f.OrganisationId equals o.Id
where idList.Contains(o.Id)
select f).Take(100).Count();
which in my environment (EF Core v1.1.1) produces the following SQL with no warnings (as expected):
SELECT COUNT(*)
FROM (
SELECT TOP(#__p_1) [f].[Id], [f].[Name], [f].[OrganisationId]
FROM [Files] AS [f]
INNER JOIN [Organisations] AS [o] ON [f].[OrganisationId] = [o].[Id]
WHERE [o].[Id] IN (1, 3, 4)
) AS [t]
I have no experience with the .NET Entity Framework and I have some doubts about what exactly do this query:
using (MyCorpo.EarlyWarnings.Model.EarlyWarningsEntities context = new Model.EarlyWarningsEntities())
{
DBContext.SetTimeout(context);
model.VulnerabilitySeverityAverage = (from x in context.VulnerabilityAlertDocuments select x.Severity).Average();
}
(Where the type of the model.VulnerabilitySeverityAverage is simply a string)
So I think that VulnerabilityAlertDocuments map the VulnerabilityAlertDocument database table because into the EarlyWarningsEntities I have something this line:
public DbSet<VulnerabilityAlertDocument> VulnerabilityAlertDocuments { get; set; }
So I am executing a query on the VulnerabilityAlertDocuments DbSet object that represent a query on my VulnerabilityAlertDocument table on my database. Is it correct?
So what exatly do the previous query?
I think that it select the Severity field value of all records in the VulnerabilityAlertDocument table and calculate the avarage value from all these value.
Is it my reasoning correct?
How can I convert this entity query in a classic SQL query? Someone can help me?
Tnx
How can I convert this entity query in a classic SQL query?
To see actual SQL you can just call .ToString() method on your query;
var sql = (from x in context.VulnerabilityAlertDocuments select x.Severity).Average().ToString();
So I am executing a query on the VulnerabilityAlertDocuments DbSet
object that represent a query on my VulnerabilityAlertDocument table
on my database. Is it correct?
Yes
So what exatly do the previous query?
Your query will average value in Severity column of ValnerabilityAlertDocuments table.
your translated query would've looked simular to this:
SELECT
[GroupBy1].[A1] AS [C1]
FROM ( SELECT
AVG([Extent1].[Severity]) AS [A1]
FROM [dbo].[ValnerabilityAlertDocuments] AS [Extent1]
) AS [GroupBy1]
Also you could try to use such tool as SQL Server Profiler
UPDATE:
Just adding LinqPad to list of tools (thanks to Dismissile)
Select Average(x.Severity)
From VulnerabilityAlertDocuments x
Thats assuming your table is called "VulnerabilityAlertDocuments"
try again
This question already has an answer here:
Stored Proc slower from application than Management Studio
(1 answer)
Closed 9 years ago.
This is my dynamic query used on search form which runs in milliseconds in SSMS roughly between 300 to 400 ms:
exec sp_executesql N'set arithabort off;
set transaction isolation level read uncommitted;
With cte as
(Select ROW_NUMBER() OVER
(Order By Case When d.OldInstrumentID IS NULL
THEN d.LastStatusChangedDateTime Else d.RecordingDateTime End
desc) peta_rn,
d.DocumentID
From Documents d
Inner Join Users u on d.UserID = u.UserID
Inner Join IGroupes ig on ig.IGroupID = d.IGroupID
Inner Join ITypes it on it.ITypeID = d.ITypeID
Where 1=1
And (CreatedByAccountID = #0 Or DocumentStatusID = #1 Or DocumentStatusID = #2 )
And (d.JurisdictionID = #3 Or DocumentStatusID = #4 Or DocumentStatusID = #5)
AND ( d.DocumentStatusID = 9 )
)
Select d.DocumentID, d.IsReEfiled, d.IGroupID, d.ITypeID, d.RecordingDateTime,
d.CreatedByAccountID, d.JurisdictionID,
Case When d.OldInstrumentID IS NULL THEN d.LastStatusChangedDateTime
Else d.RecordingDateTime End as LastStatusChangedDateTime,
dbo.FnCanChangeDocumentStatus(d.DocumentStatusID,d.DocumentID) as CanChangeStatus,
d.IDate, d.InstrumentID, d.DocumentStatusID,ig.Abbreviation as IGroupAbbreviation,
u.Username, j.JDAbbreviation, inf.DocumentName,
it.Abbreviation as ITypeAbbreviation, d.DocumentDate,
ds.Abbreviation as DocumentStatusAbbreviation,
Upper(dbo.GetFlatDocumentName(d.DocumentID)) as FlatDocumentName
From Documents d
Left Join IGroupes ig On d.IGroupID = ig.IGroupID
Left Join ITypes it On d.ITypeID = it.ITypeID
Left Join Users u On u.UserID = d.UserID
Left Join DocumentStatuses ds On d.DocumentStatusID = ds.DocumentStatusID
Left Join InstrumentFiles inf On d.DocumentID = inf.DocumentID
Left Join Jurisdictions j on j.JurisdictionID = d.JurisdictionID
Inner Join cte on cte.DocumentID = d.DocumentID
Where 1=1
And peta_rn>=#6 AND peta_rn<=#7
Order by peta_rn',
N'#0 int,#1 int,#2 int,#3 int,#4 int,#5 int,#6 bigint,#7 bigint',
#0=44,#1=5,#2=9,#3=1,#4=5,#5=9,#6=94200,#7=94250
This sql is formed in C# code and the where clauses are added dynamically based on the value the user has searched in search form. It takes roughly 3 seconds to move from one page to 2nd. I already have necessary indexes on most of the columns where I search.
Any idea why would my Ado.Net code be slow?
Update: Not sure if execution plans would help but here they are:
It is possible that SQL server has created inappropriate query plan for ADO.NET connections. We have seen similar issues with ADO, usual solution is to clear any query plans and run slow query again - this may create better plan.
To clear query plans most general solution is to update statistics for involved tables. Like next for you:
update statistics documents with fullscan
Do same for other tables involved and then run your slow query from ADO.NET (do not run SSMS before).
Note that such timing inconsistencies may hint of bad query or database design - at least for us that is usually so :)
If you run a query repeatedly in SSMS, the database may re-use a previously created execution plan, and the required data may already be cached in memory.
There are a couple of things I notice in your query:
the CTE joins Users, IGroupes and ITypes, but the joined records are not used in the SELECT
the CTE performs an ORDER BY on a calculated expression (notice the 85% cost in (unindexed) Sort)
probably replacing the CASE expression with a computed persisted column which can be indexed speeds up execution.
note that the ORDER BY is executed on data resulting from joining 4 tables
the WHERE condition of the CTE states AND d.DocumentStatusID = 9, but AND's other DocumentStatusIDs
paging is performed on the result of 8 JOINed tables.
most likely creating an intermediate CTE which filters the first CTE based on peta_rn improves performance
.net by default uses UTF strings, which equates to NVARCHAR as opposed to VARCHAR.
When you are doing a WHERE ID = #foo in dot net, you are likely to be implicitly doing
WHERE CONVERT(ID, NVARCHAR) = #foo
The result is that this where clause can't be indexed, and must be table scanned. The solution is to actually pass each parameter into the SqlCommand as a DbParameter with the DbType set to VARCHAR (in the case of string).
A similar situation could of course occur with Int types if the .net parameter is "wider" than the SQL column equivalent.
PS The easiest way to "prove" this issue is to run your query in SSMS with the following above
DECLARE #p0 INT = 123
DECLARE #p1 NVARCHAR = "foobar" //etc etc
and compare with
DECLARE #p0 INT = 123
DECLARE #p1 VARCHAR = "foobar" //etc etc
I have a linq query which selects several fields from my Customer table.
Applied to this method are multiple filters, using Func<IQueryable<T>, IQueryable<T>> with .Invoke.
The original query is essentially select * from customer.
The filter method is essentially select top 10
The output SQL is select top 10 from (select * from customer)
My customer table has over 1,000,000 rows which causes this query to take about 7 seconds to execute in SSMS. If I alter the output SQL to select top 10 from (select top 10 * from customer) by running it in SSMS then the query is instant (as you'd expect).
I am wondering if anyone knows what might cause LINQ to not combine these in a nice way, and if there is a best practice/workaround I can implement.
I should note that my actual code isn't select * it is selecting a few fields, but there is nothing more complex.
I am using SQL Server 2008 and MVC 3 with entity framework (not sure what version)
Edit: I should add, it's IQueryable all the way, nothing is evaluated until the end, and as a result the long execution is confined to that single line.
I don't know why it's not being optimised.
If the filter method really is equivalent to SELECT TOP 10 then you should be able to do it like this:
return query.Take(10);
which would resolve to select top 10 * from customer rather than the more convoluted thing you ended up with.
If this won't work then I'm afraid I'll need a little more detail.
EDIT: To clarify, if you do this in LINQ:
DataItems.Take(10).Take(10)
you would get this SQL:
SELECT TOP (10) [t1].[col1], [t1].[col2]
FROM (
SELECT TOP (10) [t0].[col1], [t0].[col2]
FROM [DataItem] AS [t0]
) AS [t1]
So if you can somehow use a Take(n) you will be okay.
I have a very simple linq to sql query in C#:
int acctNum = 12345;
var query = from p in db.table
where p.ACCT_NO == acctNum
select p;
This generates the following SQL:
exec sp_executesql N'SELECT [t0].field1, [t0].field2, [t0].ACCT_NO
FROM [dbo].[table] AS [t0]
WHERE [t0].[ACCT_NO] = #p0', N'#p0 decimal(29,0)', #p0 = 12345
For some reason, this is taking an incredibly long time to run (several minutes). If I run an equivalent query in management studio (select * from table where acct_no = 12345), it takes less than a second on a very large table (~7MM rows). After some digging with the SQL profiler, I found that linq is passing the acctNum parameter as a Decimal(29,0) while the field is stored in the database as a Numeric(18,0). If I take the generated SQL and just change the parameter type from decimal to numeric, it runs in less than a second. In the profiler, I can see that the linq version uses almost 2 million reads versus about 400 for the numeric parameter query. How can I force linq to pass this parameter as numeric instead of decimal?
Most likely the problem lies with the type of p.ACCT_NO (in other words it was probably generated as a floating-point numeric type). Make sure that this property is also typed as an int and it should work.