There is a known issue with CosmosDb where if you use an ORDER BY clause it excludes documents that do not have this property defined
To work round this I'm trying to create functionality that takes a LINQ query and replaces the Order clause with a check for documents where the property is not defined so we can then run both queries and combine the results.
So:
ordersDb.Where(x => x.Name == customerName).OrderBy(x => x.CompanyName)
would become:
ordersDb.Where(x => x.Name == customerName)
.Where(x => !x.CompanyName.IsDefined()) // IsDefined is a built in CosmosDb function
Using Expression Builder I've created the following. However, I'm having issues trying to call my expression as a Where method - :
private sealed class OrderByToIsNotDefinedVisitor : ExpressionVisitor
{
protected override Expression VisitMethodCall(MethodCallExpression node)
{
if (node.Method.DeclaringType == typeof(Queryable) &&
(node.Method.Name == "OrderBy" || node.Method.Name == "OrderByDescending"))
{
// Get the IsDefined method
var methodIsDefined = typeof(TypeCheckFunctionsExtensions).GetMethod("IsDefined",
BindingFlags.Static | BindingFlags.NonPublic | BindingFlags.Public, null,
new Type[] { typeof(object) }, null);
// Apply the IsDefined method to the property that was being used for OrderBy
var isDefinedItem = Expression.Call(methodIsDefined, node.Arguments[1]);
// Alter the expression to check for !IsDefined()
var isNotDefinedItem = Expression.Not(isDefinedItem);
var entityType = node.Method.GetGenericArguments()[0];
var genericWhere = BuildGenericWhere();
var methodWhere = genericWhere.MakeGenericMethod(entityType);
var param = Expression.Parameter(entityType);
Expression newExpression =
Expression.Call(
methodWhere,
node.Arguments[0],
Expression.Lambda(
typeof(Func<,>).MakeGenericType(entityType, typeof(bool)),
isNotDefinedItem,
param));
return newExpression;
}
return base.VisitMethodCall(node);
}
}
private static MethodInfo BuildGenericWhere()
{
var genericWhereMethod = typeof(Enumerable).GetMethods(BindingFlags.Public | BindingFlags.Static)
.Where(x => x.Name == "Where" && x.GetGenericArguments().Length == 1)
.Select(x => new { Method = x, Parameters = x.GetParameters() })
.Where(x => x.Parameters.Length == 2 &&
x.Parameters[0].ParameterType.IsGenericType &&
x.Parameters[0].ParameterType.GetGenericTypeDefinition() == typeof(IEnumerable<>) &&
x.Parameters[1].ParameterType.IsGenericType &&
x.Parameters[1].ParameterType.GetGenericTypeDefinition() == typeof(Func<,>))
.Select(x => x.Method)
.Single();
return genericWhereMethod;
}
This compiles OK, but when I execute the query I get:
Microsoft.Azure.Documents.Linq.DocumentQueryException: Expression with
NodeType Lambda is not supported.
...as documentDb cannot handle the lambda clause
I've also tried replacing the lambda clause with a direct call to the where method, so the call becomes:
var updatedQueryExpression = Expression.Call(node.Arguments[0], methodWhere, isNotDefinedItem);
return updatedQueryExpression;
...however, this results in:
System.ArgumentException: Static method requires null instance,
non-static method requires non-null instance
Firstly,we are told that we can only sort with properties of document, not derived values.
So,for me,i also recommand you following the suggestion which is mentioned by #Paul in the comment: Querying all the matching data from cosmos db,then try to sort the result list and take the top elements.I believe that you already know the expression How to get first N elements of a list in C#?:
var firstFiveArrivals = myList.OrderBy(i => i.ArrivalTime).Take(5);
Since you have to get the top elements and if your matching data set is large enough, whether you use LINQ or SQL, you will encounter thresholds which could be solved by Continuation Token.
Related
Task: I need to give some expression with parameters into LINQ's where to get some data from database, but have an error above
This example of working expression:
var shopExp = GetPersonForShop(PersonTypeIds.Director, new Guid("adda423f-8c38-40e0-9f39-6deceb787bc0")); // id
Where(shopExp)
But i need assign id dynamically, but got error above :
_repository.Persons
.Where(GetPersonForShop(PersonTypeIds.Director, person.PersonId)
And got error:
{"Unable to cast object of type 'System.Linq.Expressions.InstanceMethodCallExpression2' to type 'System.Linq.Expressions.LambdaExpression'."}
How does function for where(linq) look:
private Expression<Func<Person, bool>> GetPersonForShop(PersonTypeIds personTypeId, Guid personId)
{
return person => person .PeronTypeId== (int) personTypeId && person .PersonId == personId;
}
This is approximate look like out production, just change names of parametrs code
How can I add expression with parameters to Where clause??
Lambda expressions use => notation. Try something like this:
var idToFind = new Guid("adda423f-8c38-40e0-9f39-6deceb787bc0");
var result = _repository.Persons
.Where(p => p.TypeId == PersonTypeIds.Director && p.PersonId == idToFind);
In this expression, p represents each Person record in the Persons table, compared one-by-one using the boolean expression that follows it.
Depending on your datasource, the comparison for each p will either be done by .NET in memory, or it will happen inside your database using a SQL WHERE clause which is constructed from the boolean expression. The last would be optimal because it would mean that not the entire Persons table has to be transferred into .NET memory before comparison can take place.
Update - To apply the same condition multiple times without repeating it in your code, while still keeping the advantages of LINQ to SQL translation intact, you can put the condition in an Expression<Func<Person, bool>> object and then use that multiple times:
Expression<Func<Person, bool>> expression =
p => p.TypeId == PersonTypeIds.Director && p.PersonId == idToFind;
var result1 = datasource1.Where(expression);
var result2 = datasource2.Where(expression);
var result3 = datasource3.Where(expression);
Or through a method that produces the Expression object:
var result1 = datasource1.Where(GetExpression(idToFind));
var result2 = datasource2.Where(GetExpression(idToFind));
var result3 = datasource3.Where(GetExpression(idToFind));
public Expression<Func<Person, bool>> GetExpression(Guid idToFind)
{
return p => p.TypeId == PersonTypeIds.Director && p.PersonId == idToFind;
}
Or alternatively you can use a helper method:
var result1 = FilterByTypeAndId(datasource1, idToFind);
var result2 = FilterByTypeAndId(datasource2, idToFind);
var result3 = FilterByTypeAndId(datasource3, idToFind);
public IQueryable<Person> FilterByTypeAndId(IQueryable<Person> datasource, Guid idToFind)
{
return datasource.Where(p => p.TypeId == PersonTypeIds.Director && p.PersonId == idToFind);
}
based on the previous response, I am going to give you a few alternatives and suggestions.
var idToFind = new Guid("adda423f-8c38-40e0-9f39-6deceb787bc0");
var result = _repository
.Persons
.Where(p => p.TypeId == PersonTypeIds.Director)
.Where(p => p.PersonId == idToFind)
.ToList();
First is doing the where clause in 2 steps and then, adding the ToList(), with the ToList(), you will deal with collections and LINQ that is pretty useful. And by doing the where clause in 2 steps, is more for readable purposes.
Consider the following code which provides two methods: One to return an IQueryable, and one which leverages a compiled query to very efficient return the location matching a specific ID:
public IQueryable<Location> GetAllLocations()
{
return from location in Context.Location
where location.DeletedDate == null
&& location.Field1 = false
&& location.Field2 = true
&& location.Field3 > 5
select new LocationDto
{
Id = location.Id,
Name = location.Name
}
}
private static Func<MyDataContext, int, Location> _getByIdCompiled;
public Location GetLocationById(int locationId)
{
if (_getByIdCompiled == null) // precompile the query if needed
{
_getByIdCompiled = CompiledQuery.Compile<MyDataContext, int, Location>((context, id) =>
(from location in Context.Location
where location.DeletedDate == null
&& location.Field1 = false
&& location.Field2 = true
&& location.Field3 > 5
&& location.Id == id
select new LocationDto {
Id = location.Id,
Name = location.Name
})).First());
}
// Context is a public property on the repository all of this lives in
return _getByIdCompiled(Context, locationId);
}
This is a pretty big simplification of the actual code, but I think it gets the idea accross, and it works fine. The next thing I want to do is refactor the code, so that the common bit of the expression can be reused, since it will be used in many other types of compiled queries. In other words, this expression:
from location in Context.Location
where location.DeletedDate == null
&& location.Field1 = false
&& location.Field2 = true
&& location.Field3 > 5
select new LocationDto
{
Id = location.Id,
Name = location.Name
};
How can I somehow capture this in a variable or function and reuse it in multiple compiled queries? My attempts so far have led to errors complaining about things not being translatable to SQL, Member access not allowed, etc.
Update: Another potentially better way I could have asked this question is as follows:
Consider the two compiled queries below:
_getByIdCompiled = CompiledQuery.Compile<MyDataContext, int, LocationDto>((context, id) =>
(from location in Context.Location // here
where location.DeletedDate == null // here
&& location.Field1 = false // here
&& location.Field2 = true // here
&& location.Field3 > 5 // here
&& location.Id == id
select new LocationDto { // here
Id = location.Id, // here
Name = location.Name
})).First()); // here
_getByNameCompiled = CompiledQuery.Compile<MyDataContext, int, LocationDto>((context, name) =>
(from location in Context.Location // here
where location.DeletedDate == null // here
&& location.Field1 = false // here
&& location.Field2 = true // here
&& location.Field3 > 5 // here
&& location.Name == name
select new LocationDto { // here
Id = location.Id, // here
Name = location.Name // here
})).First()); // here
All of the lines marked // here are duplicate very un-dry pieces of code. (In my code base, this actually 30+ lines of code.) How do I factor it out and make it reusable?
So, this whole thing is somewhat odd in that the Compile method needs to not only see the Expression objects passed to each query operator (Where, Select, etc.) as something it can understand, but it needs to see the whole query, including the use of all of the operators, as something it can comprehend as Expression objects. This pretty much removes more traditional query composition as an option.
This is going to get a tad messy; more so than I would really like, but I don't see a whole lot of great alternatives.
What we're going to do is create a method to construct our queries. It's going to accept a filter as a parameter, and that filter will be an Expression that represents a filter for some object.
Next we're going to define a lambda that will look almost exactly like what you would pass to Compile, but with an extra parameter. That extra parameter will be of the same type as our filter, and it will represent that actual filter. We'll use that parameter, instead of the filter, throughout the lambda. We'll then use a UseIn method to replace all instances of that third parameter in our new lambda with the filter expression that we are providing to the method.
Here is the method to construct the query:
private static Expression<Func<MyDataContext, int, IQueryable<LocationDto>>>
ConstructQuery(Expression<Func<Location, bool>> filter)
{
return filter.UseIn((MyDataContext context, int id,
Expression<Func<Location, bool>> predicate) =>
from location in context.Location.Where(predicate)
where location.DeletedDate == null
&& location.Field1 == false
&& location.Field2 == true
&& location.Field3 > 5
&& location.Id == id
select new LocationDto
{
Id = location.Id,
Name = location.Name
});
}
Here is the UseIn method:
public static Expression<Func<T3, T4, T5>>
UseIn<T1, T2, T3, T4, T5>(
this Expression<Func<T1, T2>> first,
Expression<Func<T3, T4, Expression<Func<T1, T2>>, T5>> second)
{
return Expression.Lambda<Func<T3, T4, T5>>(
second.Body.Replace(second.Parameters[2], first),
second.Parameters[0],
second.Parameters[1]);
}
(The typing is a mess here, and I can't figure out how to give the generic types meaningful names.)
The following method is used to replace all instances of one expression with another:
public static Expression Replace(this Expression expression,
Expression searchEx, Expression replaceEx)
{
return new ReplaceVisitor(searchEx, replaceEx).Visit(expression);
}
internal class ReplaceVisitor : ExpressionVisitor
{
private readonly Expression from, to;
public ReplaceVisitor(Expression from, Expression to)
{
this.from = from;
this.to = to;
}
public override Expression Visit(Expression node)
{
return node == from ? to : base.Visit(node);
}
}
Now that we've gotten through this gory mess, comes the easy part. ConstructQuery should be able to be modified at this point to represent your real query, without much difficulty.
To call this method we simply need to provide whatever filter we want applied to this alteration of the query, such as:
var constructedQuery = ConstructQuery(location => location.Id == locationId);
Linq statements have to end in either select or group clause so you can't cut out part of the query and store it elsewhere, but if you will always be filtering by the same four conditions, you can use the lambda syntax instead, and then add any additional where clauses in new queries.
And as pointed out by #Servy, you need to call First() or FirstOrDefault() to get a single element from the query result.
IQueryable<Location> GetAllLocations()
{
return Context.Location.Where( x => x.DeletedDate == null
&& x.Field1 == false
&& x.Field2 == true
&& x.Field3 > 5
).Select( x => x );
}
Location GetLocationById( int id )
{
return ( from x in GetAllLocations()
where x.Id == id
select x ).FirstOrDefault();
}
//or with lambda syntax
Location GetLocationById( int id )
{
return GetAllLocations()
.Where( x => x.Id == id )
.Select( x => x )
.FirstOrDefault();
}
How to get single column with anonymous method using linq expression. Here's my code and it doesn't work:
public IEnumerable<object> GetPropertyValues<T>(string propName) where T : class
{
return base.Query<T>().AsEnumerable()
.Where(x => x.GetType().GetProperty(propName).Name == propName)
.Select(x => x.GetType().GetProperty(propName).GetValue(x, null));
}
Here's the code in non generic method:
base.Query<Product>().Select(x => x.ProductName).AsEnumerable();
Thanks in advance.
This condition is incorrect, because when the property propName is missing, it crashes, rather than returning false:
.Where(x => x.GetType().GetProperty(propName).Name == propName)
If you wanted to say "the dynamic type has the property propName", a proper condition for it would look like this:
.Where(x => x.GetType().GetProperty(propName) != null)
Note that this is necessary only when some, but not all, subclasses of T have the desired property propName. If the property is present in the T itself, you could get property upfront, and do the rest of the query like this:
var theProp = typeof(T)..GetProperty(propName);
return base.Query<T>().AsEnumerable().Select(x => theProp.GetValue(x, null));
Once you call AsEnumerable() all further filtering will happen in memory rather than SQL. Thus, you could do:
var prop = typeof(T).GetProperty(propName);
return base.Query<T>().AsEnumerable().Select(t => prop.GetValue(t));
but this will select all columns of T into memory before filtering down to just one. To do the filtering in SQL, you'll need to dynamically construct an expression:
var prop = typeof(T).GetProperty(propName);
var parameter = Expression.Parameter(typeof(T));
var memberAccess = Expression.MakeMemberAccess(parameter, prop);
var selector = Expression.Lambda<Func<T, TProperty>>(memberAccess, parameter);
return base.Query<T>().Select(selector);
For an enumeration of any type of objects...
public static IEnumerable<Object> Pluck<Object>(this IEnumerable<Object> source, string propName)
{
return source
.Select(x => new {
obj: x
prop: x.GetType().GetProperty(propName)
})
.Where(x => x.prop != null)
.Select(x => x.prop.GetValue(x.obj));
};
For an enumeration of a single type of object...
public static IEnumerable<Object> Pluck<T>(this IEnumerable<T> source, string, propName)
{
var prop = typeof(T).GetProperty(propName);
return prop == null
? Enumerable.Empty<Object>()
: source.Select(x => prop.GetValue(x));
};
I have looked at few examples here Calling a Method from an Expression and on MSDN but I have not been able to get the right method call/object type for Any() for the query below. I seem to be able to get the property call but not IEnumerable part of the child property.
billing_map_set_lu is the parent of billmaps_lu and is defined as an association in the Entity Framework.
The reason I am using expression trees is that
I need to be able to define the query at runtime with 1-n .SelectMany(p => p.billmaps_lu).Where(predicate) clauses. So I figured if I could build the expression trees I could handle all the different combinations I have for this system which are many.
var myResults = ctx.billing_map_set_lu
.Where(p => p.billmaps_lu.Any(b => b.billmap_columnname == "templatesittings_key" && b.billmap_columnvalue == 428264))
SelectMany(p => p.billmaps_lu)
.Where (b =>b.billmap_columnname =="locations_key" && b.billmap_columnvalue == 12445)
Select(z => z.billing_map_set_lu);
I have tried a quite a few attempts using the samples above...
ParameterExpression bms = Expression.Parameter(typeof(billmaps_lu));
Expression left1 = Expression.Property(bms, typeof(billmaps_lu).GetProperty("billmap_columnname"));
Expression right1 = Expression.Constant("templatesittings_key", typeof(string));
Expression InsideAny1 = Expression.Equal(left1, right1);
Expression left2 = Expression.Property(bms, typeof(billmaps_lu).GetProperty("billmap_columnvalue"));
Expression right2 = Expression.Constant(428264, typeof(int));
Expression InsideAny2 = Expression.Equal(left2, right2);
Expression myWhereClause1 = Expression.AndAlso(InsideAny1, InsideAny2);
The above part seems fine but when I try to do the .Any It is like I can't get the right property/method to get the right objects out. (I feel like I am on a physics problem where I am working with the wrong units.) I am hoping it is something simple that I am missing, I am pretty new to Expression Trees.. I have included non-working code to try to show you where my head is at and how someone can steer me in the right direction.
MethodInfo method = typeof(Enumerable).GetMethods().Where(m => m.Name == "Any" && m.GetParameters().Length == 2).Single().MakeGenericMethod(typeof(billing_map_set_lu).GetProperty("billmaps_lu").PropertyType);
ParameterExpression billMapSetParameter = Expression.Parameter(typeof(billing_map_set_lu), "p");
ParameterExpression billMaps = Expression.Parameter(typeof(billmaps_lu), "p1");
var myFunction = Expression.Lambda<Func<billmaps_lu, bool>>(Expression.Call(method, Expression.Property(billMapSetParameter, typeof(billing_map_set_lu).GetProperty("billmaps_lu")), myWhereClause1), billMaps)
Disclaimer, I haven't got any compiled working code.
2 problems.
First problem probably lies in:
ParameterExpression billMapSetParameter = Expression.Parameter(typeof(billing_map_set_lu), "p");
That's not a parameter you need in:
Expression.Lambda<Func<billmaps_lu, bool>>(Expression.Call(method, Expression.Property(**billMapSetParameter**, typeof(billing_map_set_lu).GetProperty("billmaps_lu")), myWhereClause1), billMaps)
Change the billMapSetParameter to the billMaps ParamterExpression, then you should be good to go. You are calling the PropertyExpression to obtain your billMapSet for you from the ParameterExpression.
2nd problem: (Not sure, but my gut feeling)
You may need to pass the Where clause as a ConstantExpression with type Expression<.Func<>>. .Any method takes two parameters, of which, the second is an Expression<.Func<>> (Or just a Func<>? can't remember).
var whereExpression = Expression.Lambda<.Func<.billmaps_lu, bool>>(myWhereClause1, bms);
var ce = Expression.Constant(whereExpression)
Then pass back ce into originally where you "myWhereClause1" is.
Cross finger it works
Edit- Scrap that, SHOW MI ZEH CODEZ
public class Foo
{
public List<string> Strings { get; set; }
}
class Program
{
static void Main(string[] args)
{
Func<Foo, bool> func =
a => a.Strings.Any(b => b == "asdf");
// b => b == "asdf";
var bParameter = Expression.Parameter(typeof (string));
var asdfConstant = Expression.Constant("asdf");
var compare = Expression.Equal(bParameter, asdfConstant);
var compareExpression = Expression.Lambda<Func<string, bool>>(compare, bParameter);
var ceCompareExpression = Expression.Constant(compareExpression.Compile());
// a => a.Strings.Any(compareExpression)
var parameter = Expression.Parameter(typeof (Foo));
var foosProperty = Expression.Property(parameter, typeof (Foo).GetProperty("Strings"));
MethodInfo method = typeof(Enumerable).GetMethods().Where(m => m.Name == "Any" && m.GetParameters().Length == 2).Single().MakeGenericMethod(typeof(string));
var anyMethod = Expression.Call(method, foosProperty, ceCompareExpression);
var lambdaExpression = Expression.Lambda<Func<Foo, bool>>(anyMethod, parameter);
// Test.
var foo = new Foo {Strings = new List<string> {"asdf", "fdsas"}};
Console.WriteLine(string.Format("original func result: {0}", func(foo)));
Console.Write(string.Format("constructed func result: {0}", lambdaExpression.Compile()(foo)));
Console.ReadKey();
}
}
What Linq expression would I use to select the 1rst overloaded method that accepts a single string parameter?
For example to return the DateTime.ToString(string format) instead of the DateTime.ToString(IFormatProvider provider).
t = typeof(DateTime);
string[] validMethods = { "ToString" };
return t.GetMethods().Where(a => validMethods.Contains(a.Name) & a.GetParameters().Length == 1).ToArray();
You can check the whole parameter list with a SequenceEqual call:
t = typeof(DateTime);
string[] validMethods = { "ToString" };
Type[] parameters = { typeof(string) };
return t.GetMethods()
.Where(a => validMethods.Contains(a.Name) &&
a.GetParameters().Select(p => p.ParameterType)
.SequenceEqual(parameters)).ToArray();
Something like this:
var match = (from method in type.GetMethods()
where validMethods.Contains(method.Name)
let parameters = method.GetParameters()
where parameters.Length == 1
where parameters[0].ParameterType == typeof(string)
select method).FirstOrDefault();
if (match != null)
{
...
}
Normally there couldn't be more than one method with a single string parameter, but it can happen, if you're looking at a derived class which hides a base class method with exactly the right parameter list and name. At that point I don't think it's specified which method you'll get first though...
I would have added this as a comment to Mehrdad's excellent answer, but don't think it would have formatted correctly...
Looking at deeply nested LINQ expressions makes my brain hurt (probably because I'm a LINQ newbie).
Here's my attempt to reformat it in a more readable way.
var t = typeof(DateTime);
string[] validMethods = { "ToString" };
var parameters = new[] { typeof(string) };
return t.GetMethods()
.Where
(
a => validMethods.Contains(a.Name)
&&
a.GetParameters()
.Select(p => p.ParameterType)
.SequenceEqual(parameters)
)
.ToArray();
Still makes my brain hurt - but not quite as much.