We Keep Coding

Using `System.String Concat` in `LINQ to Entities` generates `CAST`s instead of `CONCAT`

As I can understand from the docs, when one uses string.Concat from a querying technology, such as LINQ to Entities, this canonical function should be translated to the correct corresponding store function for the provider being used, meaning Concat for MsSQL's T-SQL. But when I'm running the following test code:
var items = (from item in testDB.Items
where list.Contains(string.Concat(item.Id,":"))
select new
{
Hit = string.Concat(item.Id,":"),
Item = item
}).ToList();
the following SQL is being produced:
SELECT
[Extent1].[Id] AS [Id],
CAST( [Extent1].[Id] AS nvarchar(max)) + N':' AS [C1],
FROM [testDB].[Items] AS [Extent1]
WHERE (CAST([Extent1].[Id] AS nvarchar(max)) + N':' -- N.B.
IN (N'1:', N'2:', N'3:')) AND (CAST([Extent1].[Id] AS nvarchar(max)) + N':' IS NOT NULL)
N.B.: +(Plus operator) with CAST instead of Concat is being used.
Obviously I'm doing something wrong, but what? The problem is that the CAST to NVARCHAR(MAX) takes enormous amount of time, especially when several fields are being concatenated.
It looks like it cannot be done, since Concat uses sql_variant type, which is not defined by SQLProviderManifest, overloading is not supported so only one function signature can be mapped(names in schema should be unique), so obviously they just made a short-circuit by concatenating using the plus operator with casts when needed, even when I mapped the Concat in code-first, the generated SQL was still using it so it became apparent. I think the only way is to utilize the DbSet.SqlQuery.

can use item.Id+":"
I used this usualy

I think you can remove the concatenation from the SQL all together. It seems like you have a list of strings that contain numbers followed by a colon. Why not strip the colon from the numbers and do direct numeric comparisons?
var list2 = list.Select(i => int.Parse(i.Replace(':','')));
var items = (from item in testDB.Items
where list2.Contains(item.Id,":"))
.AsEnumerable() // switch from DB query to memory query
.Select(item => new
{
Hit = string.Concat(item.Id,":"),
Item = item
}).ToList();

So to get things working we won't use LINQ, but DbSet.SqlQuery:
var contains = list.Aggregate(new System.Text.StringBuilder(),
(sb, s) => sb.Append($"N'{s}', "),
sb => (sb.Length > 0)
? sb.ToString(0, sb.Length - 2)
: null);
if (contains == null)
{
//ToDo: list is empty, we've got a problem
}
else
{
var query = testDB.Items.SqlQuery($#"SELECT [E1].[Id],..
FROM [testDB].[Items] AS [E1]
WHERE CONCAT_WS(':', [E1].[Id],..) IN ({contains })");
}
It should be done with SqlParameters of course, but this is the general idea for a workaround.

LINQ nested array and the ternary operator. The nested query is not supported. Operation1='Case' Operation2='Collect'

The following code produces the error
The nested query is not supported. Operation1='Case' Operation2='Collect'
The question is what am i doing so terribly wrong? How can i fix that?
IQueryable<Map.League> v = from ul in userLeagues
select new Map.League
{
id = ul.LeagueID,
seasons =
inc.Seasons ? (from ss in ul.Standings
where ss.LeagueID == ul.LeagueID
select new Map.Season
{
seasonId = ss.Season.SeasonId,
seasonName = ss.Season.SeasonName
}).ToList() : null,
};
Update
what i cannot follow is why this is working as a charm
seasons = (from ss in ul.Standings
where ss.LeagueID == ul.LeagueID
select new Map.Season
{
seasonId = ss.Season.SeasonId,
seasonName = ss.Season.SeasonName
}).Distinct(),
what is wrong with the ternary operator?

The exception indicates that you're using Entity Framework. Always good to mention the LINQ implementation in questions.
When LINQ runs against a SQL backend, the SQL provider tries to translate the whole statement into one SQL statement. This greatly reduces the types of operations that are supported, because SQL is far more restricted than LINQ. Note that the variable inc.Seasons should also part of the SQL statement. Now the problem is that a SQL can't return two different result set depending on a variable that's part of itself: there is always one fixed SELECT clause.
So there is a Case method in the expression in a place where it's not supported (and I guess that hence the subsequent Collect isn't supported either).
You could solve this by making the inclusion part of the where clause:
from ul in userLeagues
select new Map.League
{
id = ul.LeagueID,
seasons = from ss in ul.Standings
where inc.Seasons // here
&& ss.LeagueID == ul.LeagueID
select new Map.Season
{
seasonId = ss.Season.SeasonId,
seasonName = ss.Season.SeasonName
})
}

I think you simply can't put an if-else inside a linq query, at least not in that spot.
See this post for detailed explanation and discussion.
Oh, and specially look at the "hack" by user AyCabron, I think that could resolve your situation neatly (depending on what you exactly want and why you choose to have null pointers).

The problem is not with Linq in general but with Linq to objects.
since you are using IQueryable you expect the query to run in the DB,
in that context, you cannot use many operators including the ternary operator.
if you tried the same code using Linq to objects (i.e Enumerable) it will succeed.
see example here: working example

Error The nested query is not supported. Operation1='Case' Operation2='Collect' is generated by EF when you use null within a ? statement.
EF can not convert statements like condition ? object/list : null.
In your specific example, remove .ToList() as it will also produce error when there is no rows return. EF will automatically give you null when there is no items to select.

Linq Union: How to add a literal value to the query?

I need to add a literal value to a query. My attempt
var aa = new List<long>();
aa.Add(0);
var a = Products.Select(p => p.sku).Distinct().Union(aa);
a.ToList().Dump(); // LinqPad's way of showing the values
In the above example, I get an error:
"Local sequence cannot be used in LINQ to SQL implementation
of query operators except the Contains() operator."
If I am using Entity Framework 4 for example, what could I add to the Union statement to always include the "seed" ID?
I am trying to produce SQL code like the following:
select distinct ID
from product
union
select 0 as ID
So later I can join the list to itself so I can find all values where the next highest value is not present (finding the lowest available ID in the set).
Edit: Original Linq Query to find lowest available ID
var skuQuery = Context.Products
.Where(p => p.sku > skuSeedStart &&
p.sku < skuSeedEnd)
.Select(p => p.sku).Distinct();
var lowestSkuAvailableList =
(from p1 in skuQuery
from p2 in skuQuery.Where(a => a == p1 + 1).DefaultIfEmpty()
where p2 == 0 // zero is default for long where it would be null
select p1).ToList();
var Answer = (lowestSkuAvailableList.Count == 0
? skuSeedStart :
lowestSkuAvailableList.Min()) + 1;
This code creates two SKU sets offset by one, then selects the SKU where the next highest doesn't exist. Afterward, it selects the minimum of that (lowest SKU where next highest is available).
For this to work, the seed must be in the set joined together.

Your problem is that your query is being turned entirely into a LINQ-to-SQL query, when what you need is a LINQ-to-SQL query with local manipulation on top of it.
The solution is to tell the compiler that you want to use LINQ-to-Objects after processing the query (in other words, change the extension method resolution to look at IEnumerable<T>, not IQueryable<T>). The easiest way to do this is to tack AsEnumerable() onto the end of your query, like so:
var aa = new List<long>();
aa.Add(0);
var a = Products.Select(p => p.sku).Distinct().AsEnumerable().Union(aa);
a.ToList().Dump(); // LinqPad's way of showing the values

Up front: not answering exactly the question you asked, but solving your problem in a different way.
How about this:
var a = Products.Select(p => p.sku).Distinct().ToList();
a.Add(0);
a.Dump(); // LinqPad's way of showing the values

You should create database table for storing constant values and pass query from this table to Union operator.
For example, let's imagine table "Defaults" with fields "Name" and "Value" with only one record ("SKU", 0).
Then you can rewrite your expression like this:
var zero = context.Defaults.Where(_=>_.Name == "SKU").Select(_=>_.Value);
var result = context.Products.Select(p => p.sku).Distinct().Union(zero).ToList();

LINQ: Paging technique, using take and skip but need total records also - how to implement this?

I have implemented a paging routine using skip and take. It works great, but I need the total number of records in the table prior to calling Take and Skip.
I know I can submit 2 separate queries.
Get Count
Skip and Take
But I would prefer not to issue 2 calls to LINQ.
How can I return it in the same query (e.g. using a nested select statement)?
Previously, I used a paging technique in a stored procedure. I returned the items by using a temporary table, and I passed the count to an output parameter.

I'm sorry, but you can't. At least, not in a pretty way.
You can do it in an unpretty way, but I don't think you like that:
var query = from e in db.Entities where etc etc etc;
var pagedQuery =
from e in query.Skip(pageSize * pageNumber).Take(pageSize)
select new
{
Count = query.Count(),
Entity = e
};
You see? Not pretty at all.

There is no reason to do two seperate queries or even a stored procedure. Use a let binding to note a sub-query when you are done you can have an anon type that contains both your selected item as well as your total count. A single query to the database, 1 linq expression and your done. TO Get the values it would be jobQuery.Select(x => x.item) or jobQuery.FirstOrDefault().Count
Let expressions are an amazing thing.
var jobQuery = (
from job in jc.Jobs
let jobCount = (
from j in jc.Jobs
where j.CustomerNumber.Equals(CustomerNumber)
select
j
).Count()
where job.CustomerNumber.Equals(CustomerNumber)
select
new
{
item = job.OrderBy(x => x.FieldName).Skip(0).Take(100),
Count = jobCount
}
);

Using "Match" in a Linq statement

I have a table that has two records (there will be many at runtime). The deviceId of the records are, “DEVICE1” and “DEVICE2”. I want to use a regular expression to extract records.
The code below compiles but fails to return a result. When I hover the cursor on the “devices.ToList()” statement I get the following error:
base {System.SystemException} = {"LINQ to Entities does not recognize the method 'System.Text.RegularExpressions.MatchCollection Matches(System.String)' method, and this method cannot be translated into a store expression."}”
Can anyone show me how I can modify my query so that this would return records based on the expression?
filterText = #"DEVICE.";
Regex searchTerm = new Regex(filterText);
using (var ctx = new MyEntities())
{
var devices = from d in ctx.Devices
let matches = searchTerm.Matches(d.DeviceId)
where matches.Count > 0
select ((Device)d);
return devices.ToList();
}

I don't believe you can use regular expressions with LINQ to Entities. However, it looks like you're just trying to find devices which start with "DEVICE", so the query would be:
return ctx.Devices.Where(d => d.DeviceId.StartsWith("DEVICE"))
.ToList();
EDIT: If you actually need the flexibility of a regular expression, you should probably first fetch the device IDs (and only the device IDs) back to the client, then perform the regular expression on those, and finally fetch the rest of the data which matches those queries:
Regex regex = new Regex(...);
var deviceIds = ctx.Devices.Select(d => DeviceId).AsEnumerable();
var matchingIds = deviceIds.Where(id => regex.IsMatch(id))
.ToList();
var devices = ctx.Devices.Where(d => matchingIds.Contains(d.DeviceId));
That's assuming it would actually be expensive to fetch all the data for all devices to start with. If that's not too bad, it would be a simpler option. To force processing to be performed in process, use AsEnumerable():
var devices = ctx.Devices.AsEnumerable()
.Where(d => regex.IsMatch(d.DeviceId))
.ToList();

You should always remember that your LinqToEntities queries must be translated to SQL queries. Since SQL Server has no support for regular expressions, this can not work.
As suggested in the comment by Paul Ruane, StartsWith will work. This can be translated by LinqToEntities into WHERE DeviceId LIKE 'DEVICE%'.
If StartsWith isn't enough because you may need to look for strings in the middle of database columns, Contains will also work:
var devices = from d in ctx.Devices
where d.DeviceId.Contains("DEVICE")
select d;
This will result in the following: WHERE DeviceId LIKE '%DEVICE%'.

Remember when using Entity Framework or Linq to Sql that your query ends up being translated to SQL. SQL doesn't understand your regular expression object, and can't use its matches on the server side. To use your RegEx easily you could instead retrieve all the devices from the server first, and then use your existing logic. e.g.
using (var ctx = new MyEntities())
{
var devices = from Device d in ctx.Devices select d;
// Retrieve all the devices:
devices = devices.ToList();
devices = from d in devices
let matches = searchTerm.Matches(d.DeviceId)
where matches.Count > 0
select ((Device)d);
return devices.ToList();
}
This does carry the cost of retrieving more data than you need, potentially much more. For complex logic you may want to consider a Stored Procedure, which could potentially use the same RegEx via CLR functions.

LinqToEntities does not support pushing Regex's down to the database. Just do Contains (which gets converted to sql... where DeviceId Like '%DEVICE%').
filterText = #"DEVICE.";
using (var ctx = new MyEntities())
{
var devices = from d in ctx.Devices
d.DeviceId.Contains(filterText)
select d;
return devices.ToList();
}