I am using Telerik Open/Data Access ORM against an ORACLE.
Why do these two statements result in different SQL commands?
Statement #1
IQueryable<WITransmits> query = from wiTransmits in uow.DbContext.StatusMessages
select wiTransmits;
query = query.Where(e=>e.MessageID == id);
Results in the following SQL
SELECT
a."MESSAGE_ID" COL1,
-- additional fields
FROM "XFE_REP"."WI_TRANSMITS" a
WHERE
a."MESSAGE_ID" = :p0
Statement #2
IQueryable<WITransmits> query = from wiTransmits in uow.DbContext.StatusMessages
select new WITransmits
{
MessageID = wiTranmits.MessageID,
Name = wiTransmits.Name
};
query = query.Where(e=>e.MessageID == id);
Results in the following SQL
SELECT
a."MESSAGE_ID" COL1,
-- additional fields
FROM "XFE_REP"."WI_TRANSMITS" a
The query generated with the second statement #2 returns, obviously EVERY record in the table when I only want the one. Millions of records make this prohibitive.
Telerik Data Access will try to split each query into database-side and client-side (or in-memory LINQ if you prefer it).
Having projection with select new is sure trigger that will make everything in your LINQ expression tree after the projection to go to the client side.
Meaning in your second case you have inefficient LINQ query as any filtering is applied in-memory and you have already transported a lot of unnecessary data.
If you want compose LINQ expressions in the way done in case 2, you can append the Select clause last or explicitly convert the result to IEnumerable<T> to make it obvious that any further processing will be done in-memory.
The first query returns the full object defined, so any additional limitations (like Where) can be appended to it before it is actually being run. Therefore the query can be combined as you showed.
The second one returns a new object, which can be whatever type and contain whatever information. Therefore the query is sent to the database as "return everything" and after the objects have been created all but the ones that match the Where clause are discarded.
Even though the type were the same in both of them, think of this situation:
var query = from wiTransmits in uow.DbContext.StatusMessages
select new WITransmits
{
MessageID = wiTranmits.MessageID * 4 - 2,
Name = wiTransmits.Name
};
How would you combine the Where query now? Sure, you could go through the code inside the new object creation and try to move it outside, but since there can be anything it is not feasible. What if the checkup is some lookup function? What if it's not deterministic?
Therefore if you create new objects based on the database objects there will be a border where the objects will be retrieved and then further queries will be done in memory.
Related
I have a local List with entities, some hundreds, and I have a SQL Server table where I store the ID of the successful processed entities, some millions. I would like to know, which entities form my local set are not yet processed i.e. are not in the SQL Table.
The first approach is to iterate through the local list with the following Linq statement:
Entity entity = db.Entities.FirstOrDefault(m => m.ID == ID);
if (entity == null) { NewList.Add(ID) }
the NewList would then contain all the new entities. However this is very slow.
In LINQ, how would you send the entire local list to the SQL Server with one call and then return the ones not in the SQL table?
Do you really have to create a temporary table with my local list, then left-join on the already processed table and return the ones with a null?
Use .Contains method to retrieve already processed ids
and Except to create list of not yet processed ids.
var localList = new List<int> { 1, 2, 3 };
var processed = db.Entities
.Where(entity => localList.Contains(entity.Id))
.Select(entity => entity.Id)
.ToList();
var notProcessed = localList.Except(processed).ToList();
It will depend on provider, but .Contains should generate sql like:
SELECT Id FROM Entity WHERE Id IN (1, 2, 3)
suggestion:
create a temp table and insert your IDs
select your result on the SQL side
EDIT:
"Can you do that in LINQ?"
TL;DR:
yes* but that's an ugly piece of work, write the SQL yourself
*)depends on what you mean with "in" LINQ, because that is not in the scope of LINQ. In other words: a LINQ expression is one layer too abstract, but if you happen to have an LINQ accessible implementation for this, you can use this in your LINQ statements
on the LINQ expression side you have something like:
List<int> lst = new List<int>() { 1,2,3 };
List<int> result = someQueryable.Where(x=>lst.Contains(x.ID)).Select(x=>x.ID).ToList();
the question now is: what happens on the SQL side (assuming the queryable leads us to a SQL database)?
the queryable provider (e.g. Entity Framework) somehow has to translate that into SQL, execute it and come back with the result
here would be the place to modify the translation...
for example examine the expression tree with regard to the object that is the target for the Contains(...) call and if it is more than just a few elements, go for the temp table approach...
the very same LINQ expression can be translated into different SQL commands. The provider decides how the translation has to be done.
if your provider lacks support for large Contains(...) cases, you will probably experience poor performance... good thing is usually nobody forces you to use it this way ... you can skip linq for performance optimized queries, or you could write a provider extension yourself but then you are not on the "doing something with LINQ"-side but extending the functionality of your LINQ provider
if you are not developing a large scalable product that will be deployed to work with different DB-Backends, it is usually not worth the effort... the easier way to go is to write the sql yourself and just use the raw sql option of your db connection
I have a MVC controller action that does a database query this way:
var marcaciones = db.Marcacion
where db is the database context in Entity Framework, and Marcacion is a database table. After that instruction, marcaciones type becomes
System.Data.Entity.DbSet`1[CasinosCloud.Models.Marcacion]
That allows to add any filter before framework actually executes the query in database.
So far, so good.
However, depending on certain condition, marcaciones variable is assigned in a different way.
The database model is such that marcaciones entity in database is a child of another entity. To get that marcaciones list, I can do this:
var marcaciones = trabajador.ServicioSupervisado.SelectMany(s => s.Marcacion).AsQueryable();
As you can infer from instruction above, trabajador is a parent database entity that have many ServicioSupervisado entities, which, in turn, can have many Marcacion entities.
Since marcaciones variable is the same as the marcaciones variable I showed before, I have to convert to Queryable.
After executing the above instruction, marcaciones type becomes;
{System.Linq.Enumerable+<SelectManyIterator>d__17`2
[CasinosCloud.Models.Servicio,CasinosCloud.Models.Marcacion]}
That mean query is actually converted to an Enumerable List.
All that works when no other filter is applied. When I add query filter I got problems with the second form. First, the whole web page is slower because all filters are applied in a memory list, not in the database, and second, I have problems with string comparisons, specially when I try to find a text in lowercase when in database is stored in uppercase. Of course, nothing is found in such a case.
I think the problem is reduced by solving the type issue. Why after calling SelectMany, the query is actually executed and converted to an Enumerable List? Is there a way to avoid this and all that to be executed in database? Maybe I should rewrite that instruction not using SelectMany. I tried by using db.Marcacion.Insersect() to do the intersection with this code, but the same problem occurs:
trabajador.ServicioSupervisado.SelectMany(s => s.Marcacion)
EDIT:
Query I want to execute in database takes the following form:
For the first way:
SELECT m.*
FROM Marcacion m
For the second way:
SELECT m.*
FROM Marcacion m
INNER JOIN Servicio s ON s.ServicioId = m.ServicioId
INNER JOIN Trabajador t ON t.TrabajadorId = s.TrabajadorId
WHERE t.TrabajadorId = 1069
EDIT 2:
For the second way, I tried with:
marcaciones = marcaciones.Where(m => trabajador.ServicioSupervisado.Any(s => s.ServicioId == m.ServicioId));
After that, when query is actually executed in database, this error happens:
System.NotSupportedException: 'Unable to create a constant value of type 'CasinosCloud.Models.Servicio'. Only primitive types or enumeration types are supported in this context.'
I solved it by writing the query this way:
var servicios = trabajador.ServicioSupervisado.Select(s => s.ServicioId);
marcaciones = marcaciones.Where(m => servicios.Any(s => s == m.ServicioId));
That way query is executed when I call ToList(), and works in both cases I told about, being very fast in both since query is run directly in database with all filters applied.
By seeing the database log, the final query executed by Entity Framework was:
SELECT
[Extent1].[MarcacionId] AS [MarcacionId],
[Extent1].[MonitorId] AS [MonitorId],
[Extent1].[TrabajadorId] AS [TrabajadorId],
[Extent1].[EmpresaId] AS [EmpresaId],
[Extent1].[ServicioId] AS [ServicioId],
[Extent1].[MarcacionFechaHora] AS [MarcacionFechaHora],
[Extent1].[MarcacionEntradaSalida] AS [MarcacionEntradaSalida],
[Extent1].[MarcacionChecksum] AS [MarcacionChecksum],
[Extent1].[MarcacionEquipo] AS [MarcacionEquipo],
[Extent1].[MarcacionEsManual] AS [MarcacionEsManual],
[Extent1].[MarcacionCreadoEn] AS [MarcacionCreadoEn],
[Extent1].[MarcacionActualizadoEn] AS [MarcacionActualizadoEn],
[Extent1].[MarcacionIndice] AS [MarcacionIndice]
FROM [dbo].[Marcacion] AS [Extent1]
INNER JOIN [dbo].[Trabajador] AS [Extent2] ON [Extent1].[TrabajadorId] = [Extent2].[TrabajadorId]
WHERE ( EXISTS (SELECT
1 AS [C1]
FROM ( SELECT 1 AS X ) AS [SingleRowTable1]
WHERE 3 = [Extent1].[ServicioId]
)) AND ([Extent2].[TrabajadorNombres] + N' ' + [Extent2].[TrabajadorApellidos] LIKE #p__linq__0 ESCAPE N'~') AND ([Extent1].[MarcacionFechaHora] >= #p__linq__1) AND ([Extent1].[MarcacionFechaHora] <= #p__linq__2)
This is my code:
var query = context.SomeEntities.Select(e => new SomeDto
{
Name = e.Title,
})
The variable query is IQueryable.
var list = new List<SomeDto>
{
new SomeDto
{
Name = "SomeName1"
},
new SomeDto
{
Name = "SomeName2"
}
};
Now I need union these collections,I write
query = query.Union(list)
But in this place I get exception
Unable to create a constant value of type 'SomeDto'. Only primitive types or enumeration types are supported in this context
I know that I can do ToList() for first collection,but I need union collections without call to database.How can I do this?
As user3825493 pointed, there are answers regarding similar issues. The first link pretty much explains the whole thing.
Your problem is that due to the fact that Union is mapped to SQL UNION, arguments of the operation are to be converted to data types acceptable by Entity Framework. However, the only types acceptable by it for items inside IEnumerable input parameters are, as specified in the error you get, primitive types and enum's.
You should either:
var unionOfQueryWithList = query.AsEnumerable().Union(list)
or, if you like to use the SQL facilities, write a stored procedure which performs the union as part of its logic and find the way to pass your data into SQL, such as this.
The former loads all the results of the query into a memory before performing the union. It is convenient for the cases when you have all the pre-filtered data returned from EF and only want to use the output for later processing in your code.
The later takes you out of the EF zone and required for the cases when you have operations which will benefit from running on SQL engine, such as filtering or joining DB data based on a complex data you have access to in your code.
either you convert the query to IEnumerable or you can try this :
list.Union(query); // use Union() of list not query :)
it will work and you still don't need to get data from DB you can reverse the order if you want :)
We are using C#. VS2012, EF 5.0, and Oracle 11g. Approach is code first. I have a table that is defined, and it is plainly visible in looking at the code that it is defined with all the correct columns (and none that are not there.)
Still, when I run certain LINQ queries (joins) and attempt to select the results into a new object, things break. Here is the LINQ:
IQueryable<CheckWage> query =
from clientWage in context.ClientWages
join paycheckWage in context.PaycheckWages
on
new {clientWage.PermanentClientId, clientWage.WageId} equals
new {paycheckWage.PermanentClientId, paycheckWage.WageId}
where
(paycheckWage.PermanentClientId == Session.PermanentClientId) &&
(clientWage.PermanentClientId == Session.PermanentClientId)
select new CheckWage
{
CWage = clientWage,
PWage = paycheckWage
};
Now, here is the SQL it emits (as captured by Devart's DbMonitor tool):
SELECT
"Extent1".ASSOCIATE_NO,
"Extent1".PCLIENT_ID,
"Extent1".CLIENT_NO,
"Extent1".CLIENT_NAME,
"Extent1".ADDRESS1,
"Extent1".ADDRESS2,
"Extent1".CITY,
"Extent1".STATE,
"Extent1".ZIP,
"Extent1".COUNTRY,
"Extent1".CLIENT_TYPE,
"Extent1".DOING_BUSINESS_AS,
"Extent1".CONTACT,
"Extent1".PHONE,
"Extent1".EXTENSION,
"Extent1".FAX,
"Extent1".FAX_EXTENSION,
"Extent1".EMAIL,
"Extent1".NEXTEMP,
"Extent1".PAY_FREQ,
"Extent1".EMPSORT,
"Extent1".DIVUSE,
"Extent1".CLIENT_ACCESS_TYPE,
"Extent1".AUTOPAY_WAGE_ID,
"Extent1".FEIN,
"Extent1".HR_MODULE,
"Extent1".BANK_CODE,
"Extent1".ACH_DAYS,
"Extent1".ACH_COLLECT,
"Extent1".UPDATED,
"Extent1".IAT_FLAG,
"Extent1".ORIG_EMAIL,
"Extent1"."R1",
"Extent1"."R2"
FROM INSTANTPAY.CLIENT "Extent1"
WHERE "Extent1".PCLIENT_ID = :EntityKeyValue1'
There are no such columns as "R1" and "R2." I am guessing is has something to do with the join into a new object type with two properties, but I am pulling my hair out trying to figure out what I've done or haven't done that is resulting in this errant SQL. Naturally, the error from the Oracle server is "ORA-00904: "Extent1"."R2": invalid identifier." Strange that is doesn't choke on R1, but perhaps it only lists the last error or something...
Thanks in advance,
Peter
5/23/2014: I left out an important detail. The SQL is emitted when I attempt to drill into one of the CheckWage objects (using Lazy loading), as both of the contained objects have a navigation property to the "Client" entity. I can access the client table just fine in other LINQ queries that do not use a join, it is only this one that creates the "R1" and "R2" in the SELECT statement.
Peter
Using the Entity Framework, when one executes a query on lets say 2000 records requiring a groupby and some other calculations, does the query get executed on the server and only the results sent over to the client or is it all sent over to the client and then executed?
This using SQL Server.
I'm looking into this, as I'm going to be starting a project where there will be loads of queries required on a huge database and want to know if this will produce a significant load on the network, if using the Entity Framework.
I would think all database querying is done on the server side (where the database is!) and the results are passed over. However, in Linq you have what's known as Delayed Execution (lazily loaded) so your information isn't actually retrieved until you try to access it e.g. calling ToList() or accessing a property (related table).
You have the option to use the LoadWith to do eager loading if you require it.
So in terms of performance if you only really want to make 1 trip to the Database for your query (which has related tables) I would advise using the LoadWith options. However, it does really depend on the particular situation.
It's always executed on SQL Server. This also means sometimes you have to change this:
from q in ctx.Bar
where q.Id == new Guid(someString)
select q
to
Guid g = new Guid(someString);
from q in ctx.Bar
where q.Id == g
select q
This is because the constructor call cannot be translated to SQL.
Sql's groupby and linq's groupby return differently shaped results.
Sql's groupby returns keys and aggregates (no group members)
Linq's groupby returns keys and group members.
If you use those group members, they must be (re-)fetched by the grouping key. This can result in +1 database roundtrip per group.
well, i had the same question some time ago.
basically: your linq-statement is converted to a sql-statement. however: some groups will get translated, others not - depending on how you write your statement.
so yes - both is possible
example:
var a = (from entity in myTable where entity.Property == 1 select entity).ToList();
versus
var a = (from entity in myTable.ToList() where entity.Property == 1 select entity).ToList();