I am working on a Linq query expression statement, and want to use a let statement to set a constant value as part of the query to reduce potential errors.
In my example... (this is total mock up). the let statement "validateCount" is the question.
from referenceRow in context.SomeTable
// We want 8 in the Take and the results to still be 8 after the where.
// instead of typing in the value twice (in this example), i wanted to use a constant value here
let validateCount = 8
// sub query expression in my example
let subQuery = from sub in context.SomeTable
where sub.SomeColumn == rowReference.SomeColumn
orderby sub.SomeColumn descending
select sub
// use the sub query, take X amount, apply filter, check to see if Count() is the same as Take()
where subQuery.Take(validateCount) // this is where we fail. if we put in the constant 8, we are ok.
.Where(x => x.SomeOtherColumn == "Some value")
.Count() == validateCount // this is fine
select referenceRow
Unfortunately it seems that the let expression "validateCount" which has a single value of 8 in this example, can only work in the comparison part of .Count() but cannot be passed into the .Take() without throwing an exception.
Limit must be a DbConstantExpression or a DbParameterReferenceExpression.
Parameter name: count
Looking for a solution to use some user-defined constant in a single code location that can be used in the rest of the query expression, both in the .Take() and .Count() without having to be updated several spots in the code.
our application allows users to supply their own linq query expression to build their queries. I can't define anything outside the scope of this query, and must be within, using something like a 'let' statement.
let statement generates intermediate anonymous type projection (Select call) in the query expression tree. EF query provider (as indicated by the exception message) requires Skip and Take arguments to be resolved to constant or variable values (i.e. to be able to be evaluated locally), hence the let cannot be used for that purpose.
Instead, the constants/variables used in Skip / Take expressions should be defined outside of the query and used inside.
To define a constant value you would use:
const int validateCount = 8;
var query = (from .... Take(validateCount) ...);
To define a variable value (SQL query parameter):
int validateCount = 8;
var query = (from .... Take(validateCount) ...);
Here the C# compiler will turn validateCount into closure and EF query provider will be happy to bind a parameter (with that value).
our application allows users to supply their own linq query expression to build their queries. I can't define anything outside the scope of this query, and must be within, using something like a 'let' statement.
When supplying their own queries, the users should follow the same Skip / Take argument rules as above, i.e. define their constants and variables outside of their queries.
Related
Is it possible to use custom method In query for example:
var result = from u in context.MyTable where MyMethod(u) == 10 select u;
As Pranay explains, you cannot have a custom (C#) method as part of the LINQ to SQL query, because LINQ to SQL wouldn't be able to look at the expression tree of the method and so it cannot translate it to SQL.
One option that you have is to write your function in SQL and store it as a SQL function on the SQL Server (possibly, you could also use SQL CLR, but I have not tried that). Then you can add the function to your DataContext type and LINQ to SQL will translate it to calls to the function on SQL server. Something like:
var result = from u in context.MyTable
where context.MyMethod(u) == 10 select u;
The problem, of course, is that you'll need to write the function in SQL (I think SQL CLR could also work - not sure about the performance and other possible complications though)
I also wrote an article (some time ago) that shows how to do this when you write the "method" as an expression tree way (as a value of type Expression<Func<...>>), which is possible, because in this case, the code is compiled as an expression tree. However, there is some postprocessing that has to be done and you can still write just a single expression that can be easily inlined in the LINQ query.
Check this full article : What is and what isn't possible with linq
Following is not possible
// function used in filter
static bool MyFunc(Nwind.Product p)
{
return p.ProductName.StartsWith("B");
}
// query that uses MyFunc
var q =
from p in db.Products
where MyPriceFunc(p.UnitPrice) > 30m
select p
It compiles with no errors, but when you execute it LINQ to SQL throws an exception saying: "Static method System.Boolean MyTest(LINQTest.Nwind.Product) has no supported translation to SQL."
The exception is actually thrown when you try to fetch results from q (for example using the foreach statement), because LINQ to SQL attempts to convert the expression trees to T-SQL only when the results are needed and the query must be executed.
To fix the example you can simply copy the code that checks whether product name starts with "B" to the where clause of the query and it would work fine.
Yes, but if you are using Linq-to-Sql - your method has to have special code to handle to SQL conversion.
Why out variables are not allowed in query clause?
If I use out variables here it fails:
string json = "{'PayDays':['2017-07-07','2017-07-21','2017-08-04','2017-08-18']}";
var pd = JsonConvert.DeserializeObject<Accounting>(json);
var rm = from item in pd.PayDays
where (DateTime.TryParse(item, out DateTime dateresult)) ?
dateresult.Subtract(DateTime.Now).Days >= 0 ? true : false : false
select item;
rm.Dump();
But old way works:
DateTime result;
var rm = from item in pd.PayDays
where DateTime.TryParse(item, out result) ? (result.Subtract(DateTime.Now).Days >= 0 ? true : false) : false
select item;
rm.Dump();
Here are the relevant LDM notes.
The conclusion:
We won't have time to do this feature in C# 7.0. If we want to leave ourselves room to do it in the future, we need to make sure that we don't allow expression variables in query clauses to mean something else today, that would contradict such a future.
The current semantics is that expression variables in query clauses
are scoped to only the query clause. That means two subsequent query
clauses can use the same name in expression variables, for instance.
That is inconsistent with a future that allows those variables to
share a scope across query clause boundaries.
Thus, if we want to allow this in the future we have to put in some
restrictions in C# 7.0 to protect the design space. We have a couple
of options:
Disallow expression variables altogether in query clauses
Require that
all expression variables in a given query expression have different
names
The former is a big hammer, but the latter requires a lot of
work to get right - and seems at risk for not blocking off everything
well enough.
and
We will neither do expression variables nor deconstruction in C# 7.0, but would like to do them in the future. In order to protect our ability to do this, we will completely disallow expression variables inside query clauses, even though this is quite a big hammer.
The problem is that the translation you showed works fine for normal LINQ to objects, but would be problematic for other providers, like PLINQ or Entity Framework.
For example, consider the translation you proposed for your query modified to use PLINQ:
DateTime result;
var rm = from item in pd.PayDays.AsParallel()
where DateTime.TryParse(item, out result) ? (result.Subtract(DateTime.Now).Days >= 0 ? true : false) : false
select item;
var list = rm.ToList();
This code is not thread-safe, because multiple threads share a single result variable and you're likely going to get incorrect results, because they will try to use it at the same time.
There are ways to correctly translate such queries, at least in some cases, but they're not trivial, so out variables are simply not allowed in LINQ query syntax. Though there is a proposal to allow them in a future version of C#.
I'm using SQL Server 2005, with a case sensitive database..
In a search function, I need to create a Linq To Entities (L2E) query with a "where" clause that compare several strings with the data in the database with these rules :
The comparison is a "Contains" mode, not strict compare : easy as the string's Contains() method is allowed in L2E
The comparison must be case insensitive : I use ToLower() on both elements to perform an insensitive comparison.
All of this performs really well but I ran into the following Exception :
"Argument data type ntext is invalid for argument 1 of lower function" on one of my fields.
It seems that the field is a NText field and I can't perform a ToLower() on that.
What could I do to be able to perform a case insensitive Contains() on that NText field ?
Never use .ToLower() to perform a case-insensitive comparison. Here's why:
It's possibly wrong (your client collation could be, say, Turkish, and your DB collation not).
It's highly inefficient; the SQL Emitted is LOWER instead of = with a case-insensitive collation.
Instead, use StringComparison.OrdinalIgnoreCase or StringComparison.CurrentCultureIgnoreCase:
var q = from f in Context.Foos
where f.Bar.Equals("hi", StringComparison.OrdinalIgnoreCase)
select f;
But for Contains() there's a problem: Unlike Equals, StartsWith, etc., it doesn't have an overload for a StringComparison argument. Why? Good question; ask Microsoft.
That, combined with SQL Server's limitation on LOWER means there's no simple way to do what you want.
Possible workarounds might include:
Use a full text index, and do the search in a procedure.
Use Equals or StartsWith instead, if possible for your task
Change the default collation of the column?
Use a lambda expression here and create an intermediary list that can handle the lower clause.
var q = Context.Foos.ToList().Where(s => s.Bar.ToLower().Contains("hi"));
Not terribly efficient, but it does work. If you have additional predicates in your where clause then it works to your advantage:
var q = Context.Foos.Where(p => p.f1 == "foo" && p.f2 == "bar").
ToList().Where(s => s.Bar.ToLower().Contains("hi"));
as we known , this is a very "bugged" situation.
and it bugs me a lot.
Today, i decide to create a view as:
select * from tableName
where theColumn like '%key%'
then load this view into EF.
life is getting easy!
I have some stats code that I want to use in various places to calculate success / failure percentages of schedule Results. I recently found a bug in the code and this was due to the fact it was replicated in each LINQ statement, I then decided it would be better to have common code to do this. The problem being, of course, is that a normal function, when executed on SQL server, throws a NotSupportedException because the fuinction doesnt exist in SQL Server.
How can I write a reusable stats code that gets executed on SQL server or is this not possible?
Here is the code I have written for Result
public class Result
{
public double CalculateSuccessRatePercentage()
{
return this.ExecutedCount == 0 ? 100 : ((this.ExecutedCount - this.FailedCount) * 100.0 / this.ExecutedCount);
}
public double CalculateCoveragePercentage()
{
return this.PresentCount == 0 ? 0 : (this.ExecutedCount * 100.0 / this.PresentCount);
}
}
And it is used like so (results is IQueryable, and throws the exception):
schedule.SuccessRatePercentage = (int)Math.Ceiling(results.Average(r => r.CalculateSuccessRatePercentage()));
schedule.CoveragePercentage = (int)Math.Ceiling(results.Average(r => r.CalculateCoveragePercentage()));
or like this (which works, because we do this on a single result)
retSchedule.SuccessRatePercentage = (byte)Math.Ceiling(result.CalculateSuccessRatePercentage());
retSchedule.CoveragePercentage = (byte)Math.Ceiling(result.CalculateCoveragePercentage());
Edit
As per #Fred's answer I now have the following code, which works for an IQueryable
schedule.SuccessRatePercentage = (int)Math.Ceiling(scheduleResults.Average(ScheduleResult.CalculateSuccessRatePercentageExpression()));
schedule.CoveragePercentage = (int)Math.Ceiling(scheduleResults.Average(ScheduleResult.CalculateCoveragePercentageExpression()));
The only problem, albeit a minor one, is that this code will not work for individual results i.e.
retSchedule.SuccessRatePercentage = (byte)Math.Ceiling(/* How do I use it here for result */);
You can't pass functions to SQL - you would need to declare the function on the actual SQL database and then call that from your code.
What you could do/try is this:
Expression<Func<Result, double>> CalculateCoveragePercentage()
{
return r => r.PresentCount == 0 ? 0 : (r.ExecutedCount * 100.0 / r.PresentCount);
}
It needs to be interpreted instead of executed so that EF can translate it to SQL. The problem is, I've only heard of this being possible when it's passed directly into a where clause.
Since you are able to do these calculations when you apply them directly inside of your LINQ query, I'm inclined to think that it should also be possible to declare those calculations as Expression<Func<..., ...>> and them pass them in.
The only way to know for sure is to try (unless you feel like looking into EF's ExpressionBuilder)
UPDATE:
I should have mentioned that, if this would work, you need to pass this expression into a Select statement:
// Assuming you have Results declared as a DbSet or IDbSet, such as:
DbSet<Result> Results
// You could do something like this (just to illustrate that
// it would be interpreted rather than executed):
List<double> allCoveragePercentages = Results.Select(CalculateCoveragePercentage)
.ToList();
UPDATE #2:
In order for this to work with individual results (or in any case whatsoever), you need to pass it into a clause that accepts the expression. Examples are Select, Where, Average (apparently), anything that does not returns results.
From the top of my head (I'm sure I'm missing a few):
List: ToArray, ToDictionary, ToList, ToLookup
Single result: First, FirstOrDefault, Single, SingleOrDefault, Last, LastOrDefault
Computation: Count, Sum, Max, Min
Since the above clauses return results, they (for as far as I know) only accept Predicates (a function that can only return 'true' or 'false')
You may have coincidentally got it right with your .Average(CalculateCoveragePercentage)
So if you were to get a single result with .FirstOrDefault(), you would pass in your expression inside of a select clause right before that: .Select(CalculateCoveragePercentage).FirstOrDefault(). That is, if you don't need the actual entity but just the calculation. Be aware though that this particular example will return 0 if there were no Result results. You may or may not want this behavior.
Of course, if you already have your result (it's not an IQueryable anymore) then you can simple do:
var coveragePercentage = CalculateCoveragePercentage().Compile().Invoke(result);
But that would kind of defeat the purpose of the expression - for this situation you should just add a method to your Result class that calculates the CoveragePercentage of a given instance.
I have a LINQ to SQL Query as below:
var carIds = from car in _db.Cars
where car.Color == 'Blue'
select car.Id;
The above will be translated into the below sql finally:
select Id from Cars
where Color = 'Blue'
I've read that the phases are:
LINQ to Lambda Expression
Lambda Expression to Expression Trees
Expression Trees to SQL Statements
SQL Statements gets executed
So, my questions are when and how does this get translated and executed?
I know that phase 4 happens at runtime when my "carIds" variable get accessed within a foreach loop.
foreach(var carId in carIds) // here?
{
Console.Writeline(carId) // or here?
}
What about the other phases? when do they occur? at compile time or runtime? at which line (on the definition or after the definition, when accessed or before it gets accessed)?
What you are talking about is deferred execution - essentially, the linq to SQL query will be executed at the time you attempt to enumerate the results (e.g. iterate round it, .ToArray() it etc).
In your example, the statement is executed on the following line:
foreach(var carId in carIds)
See this MSDN article on LINQ and Deferred Execution
is done by either you, the dev, or the C# compiler; you can write lambda expression manually, or with LINQ syntax they can be implied from LINQ (i.e. where x.Foo == 12 is treated as though it were .Where(x => x.Foo == 12), and then the compiler processes step 2 based on that)
is done by the C# compiler; it translates the "lambda expression" into IL that constructs (or re-uses, where possible) an expression tree. You could also argue that the runtime is what processes this step
and
are typically done when the query is enumerated, specifically when foreach calls GetEnumerator() (and possibly even deferred to the first .MoveNext)
the deferred execution allows queries to be composed; for example:
var query = ...something complex...
var items = query.Take(20).ToList();
here the Take(20) is performed as part of the overall query, so you don't bring back everything from the server and then (at the caller) filter it to the first 20; only 20 rows are ever retrieved from the server.
You can also use compiled queries to restrict the query to just step 4 (or steps 3 and 4 in some cases where the query needs different TSQL depending on parameters). Or you can skip everything except 4 if you write just TSQL to start with.
You should also add a step "5": materialization; this is quite a complex and important step. In my experience, the materialization step can actually be the most significant in terms of overall performance (hence why we wrote "dapper").