I'm using elasticsearch.net library in C# and I'm trying to query for objects matching specified filter.
I would like the query to return objects where at least one of input names from filter exists in object's Names collection.
The problem is that I always get 0 hits as result with this query, even tho I am certain that data matching specified filter does exist in the database and I would love to find out what's wrong with my query...
The model:
public class A
{
public int AId { get; set; }
public IEnumerable<string> Names { get; set; }
}
The filtering object:
public class Filter
{
public IEnumerable<string> NamesToSearch { get; set; }
}
The method for querying data:
public async Task<IEnumerable<A>> GetFilteredData(Filter filter)
{
var query = await _elasticClient.SearchAsync<A>(x => x.Query(q => q.Terms(a => a.Names, filter.NamesToSearch))
.Fields(a => a.AId, a => a.Names));
return query.Hits
.Select(x => new A
{
AId = x.Fields.FieldValues<A, int>(a => a.AId)[0]
})
.ToList();
}
I have also tried following query, but it didn't yield expected result neither:
var query = await _elasticClient.SearchAsync<A>(x => x.Query(q => q.Nested(n => n.Filter(f => f.Terms(y => y.Names, filter.NamesToSearch))))
.Fields(a => a.AId, a => a.Names));
SOLUTION WHICH WORKED FOR ME:
I have upgraded a bit code from Sławomir Rosiek's answer to actually compile using ElasticSearch.net 1.7.1 and be type-safe (no references to field name by string) and ended up with following extension method, which worked like a charm for my scenario:
public static QueryContainer MatchAnyTerm<T>(this QueryDescriptor<T> descriptor, Expression<Func<T, object>> field, object[] values) where T : class, new()
{
var queryContainer = new QueryContainer();
foreach (var value in values)
{
queryContainer |= descriptor.Term(t => t.OnField(field).Value(value));
}
return queryContainer;
}
and usage:
var query = await _elasticClient.SearchAsync<A>(x => x.Query(q =>
q.Bool(b =>
b.Should(s => s.MatchAnyTerm(a => a.Names, filter.NamesToSearch.ToArray()))
.Fields(a => a.AId, a => a.Names));
I think that your problem is that you tries to pass whole array to query. Instead of that you should treat that as OR expression.
Below is the raw query that you should use:
{
"query": {
"bool": {
"should": [
{ "term": {"names": "test" } },
{ "term": {"names": "xyz" } }
]
}
}
}
And that the C# code to achive that. First I have defined helper function:
private static QueryContainer TermAny<T>(QueryContainerDescriptor<T> descriptor, Field field, object[] values) where T : class
{
QueryContainer q = new QueryContainer();
foreach (var value in values)
{
q |= descriptor.Term(t => t.Field(field).Value(value));
}
return q;
}
And now the query:
string[] values = new[] { "test", "xyz" };
client.Search<A>(x => x.Query(
q => q.Bool(
b => b.Should(s => TermAny(s, "names", values)))));
Related
I have read through the question, and answer, Dynamic linq expression tree with nested properties. It does seem to be very similar, though a lack of understanding with Expressions is resulting in me not being able to translate the answer to my own scenario.
Given a class structure that looks a little like this:
public class Parent
{
public Parent()
{
ParentField = new HashSet<ParentField>();
}
public int ParentId { get; set; }
public ICollection<ParentField> ParentField { get; set; }
}
public class ParentField
{
public int ParentField { get; set; }
public Field Field { get; set; }
public string Value {get;set;}
}
public class Field
{
public int FieldId { get; set; }
public string Name { get; set; }
}
I am trying to build up a query that would be represented by this:
var query = _unitOfWork.ParentRepository.Queryable().Include("ParentField.Field");
query = query.Where(i =>
(
i.ParentField.Any(pField => pField.Field.Name.Equals("anId", StringComparison.OrdinalIgnoreCase)) &&
i.ParentField.Any(pField =>
// There may be multiple values to search for, so require the OR between each value
pField.Value.Equals("10", StringComparison.OrdinalIgnoreCase) || pField.Value.Equals("20", StringComparison.OrdinalIgnoreCase))
)
// There may be multiple Names to search for, so require the AND between each Any
&&
(
i.ParentField.Any(pField => pField.Field.Name.Equals("anotherId", StringComparison.OrdinalIgnoreCase)) &&
i.ParentField.Any(pField =>
pField.Value.Equals("50", StringComparison.OrdinalIgnoreCase) || pField.Value.Equals("60", StringComparison.OrdinalIgnoreCase))
));
The important parts to note is that there can be many "Field.Name"'s to search for, as well as multiple "Values" within each of the groups.
I'm not able to give much of an example of what I have tried so far, given I am not sure where to actually start.
Any pointers would be fantastic.
In this particular case there is no need to build dynamic expression predicate. The && can be achieved by chaining multiple Where, and || by putting the values into IEnumerable<string> and using Enumerable.Contains.
For single name / values filter it would be something like this:
var name = "anId".ToLower();
var values = new List<string> { "10", "20" }.Select(v => v.ToLower());
query = query.Where(p => p.ParentField.Any(
pf => pf.Field.Name == name && values.Contains(pf.Value.ToLower())));
And for multiple key / values pairs:
var filters = new Dictionary<string, List<string>>
{
{ "anId", new List<string> { "10", "20" } },
{ "anotherId", new List<string> { "50", "60" } },
};
foreach (var entry in filters)
{
var name = entry.Key.ToLower();
var values = entry.Value.Select(v => v.ToLower());
query = query.Where(p => p.ParentField.Any(
pf => pf.Field.Name == name && values.Contains(pf.Value.ToLower())));
}
#geraphl's answer presents the same idea, but w/o taking into account EF query provider specific requirements (no dictionary methods, only primitive value list Contains, no Equals with StringComparison.OrdinalIgnoreCase usage, but == and ToLower etc.)
I am not sure but i think you are searching for something like this.
First you have to put your searched values in a data-type that is capable of doing what you want, which is in this case a dictionary:
var nameValues = new Dictionary<string, string[]>(StringComparer.OrdinalIgnoreCase)
{
{ "anId" , new string[] {"10", "20"} },
{ "anotherId" , new string[] {"50", "60"} },
};
Then you first check if the name is in the dictionary and if so, you also check if one of the values also listed is in the list you search.
var query = _unitOfWork.ParentRepository.Queryable().Include("ParentField.Field");
query = query.Where(i =>
(
i.ParentField.Any(pFieldOuter => nameValues.ContainsKey(pFieldOuter.Field.Name) ?
i.ParentField.Any(pFieldInner => nameValues[pFieldOuter.Field.Name].Contains(pFieldInner.Value, StringComparer.OrdinalIgnoreCase))
: false
)
));
But if you want, that all your searched names and values are included you have to do this.
var names = new HashSet<string>(StringComparer.OrdinalIgnoreCase) { "anId", "anotherId" };
var values = new List<List<string>>() {
new List<string> { "10", "20" },
new List<string> { "50", "60" }
};
var query = _unitOfWork.ParentRepository.Queryable().Include("ParentField.Field");
query = query.Where(i =>
(
names.All(n => i.ParentField.Any(pField => pField.Name.Equals(n, StringComparison.OrdinalIgnoreCase))) &&
values.All(l => l.Any(v => i.ParentField.Any(pField => pField.Value.Equals(v, StringComparison.OrdinalIgnoreCase))))
));
Even if i am sure this is not the most effective way to do is, it may be a good hint for you.
I want to return a list of items in order of how many related items there are.
Imagine the following classes. and imagine they all had DbSets... context.A..., context.B...
class A
{
public ID { get; set; }
}
class B
{
public virtual A A { get; set; }
}
I am trying to get a list of A items in order of most related from B. The query might look like this:
IEnumerable<A> GetMostRelatedAs( int numberOfAsToReturn )
{
return this.context.A.SelectMany(
a => a.ID,
( whatever) => new
{
A = whatever,
RelatedBCount = this.context.B.Where( b => b.A.ID == whatever.ID)
}).OrderByDescending( x => x.RelatedBCount ).Take( numberOfAsToReturn );
}
Where am I going wrong in my query?
Due to this:
I am trying to get a list of A items in order of most related to from B.
to from makes this quite confusing, so on this basis I'm going to have a stab in the dark with this one:
IEnumerable<dynamic> GetMostRelatedAs( int numberOfAsToReturn )
{
var results = this.context.A
.GroupJoin(
this.context.B,
a => a.ID,
b => b.A.ID,
(singleA, multipleBs) => new {
// this is the projection, so take here what you want
numberOfBs = multipleBs.Count(),
name = singleA.Name,
singleA.ViewCount
}
)
.OrderByDescending(x => x.ViewCount)
.Take(numberOfAsToReturn)
.ToList();
// here you can use automapper to project to a type that you can use
// So you could add the following method calls after the ToList()
// .Project(this.mappingEngine)
// .To<ClassThatRepresentsStructure>()
// The reason you don't map before the ToList is that you are already doing a projection with that anonymous type.
return results;
}
Edit
To address the comments:
IEnumerable<A> GetMostRelatedAs( int numberOfAsToReturn )
{
var results = this.context.A
.GroupJoin(
this.context.B,
a => a.ID,
b => b.A.ID,
(singleA, multipleBs) => new {
// this is the projection, so take here what you want
numberOfBs = multipleBs.Count(),
name = singleA.Name,
singleA.ViewCount,
singleA
}
)
.OrderByDescending(x => x.ViewCount)
.Take(numberOfAsToReturn)
.ToList()
.Select(x => x.singleA);
return results;
}
I'm looking for a way to get total price count from the Costs list in my object. I can't get Projections.Sum to work in my QueryOver so I tried another way but I'm having problems with it. I want to use a unmapped property in my QueryOver. I found this example but it's giving an error.
Object:
public class Participant
{
public int Id { get; set; }
public double TotalPersonalCosts { get { return Costs.Where(x => x.Code.Equals("Persoonlijk") && x.CostApprovalStatus == CostApprovalStatus.AdministratorApproved).Sum(x => x.Price.Amount); } }
public IList<Cost> Costs { get; set; }
}
The property TotalPersonalCosts is not mapped and contains the total price count.
Extension Class:
public static class ParticipantExtensions
{
private static string BuildPropertyName(string alias, string property)
{
if (!string.IsNullOrEmpty(alias))
{
return string.Format("{0}.{1}", alias, property);
}
return property;
}
public static IProjection ProcessTotalPersonalCosts(System.Linq.Expressions.Expression expr)
{
Expression<Func<Participant, double>> w = r => r.TotalPersonalCosts;
string aliasName = ExpressionProcessor.FindMemberExpression(expr);
string totalPersonalCostName = ExpressionProcessor.FindMemberExpression(w.Body);
PropertyProjection totalPersonalCostProjection =
Projections.Property(BuildPropertyName(aliasName, totalPersonalCostName));
return totalPersonalCostProjection;
}
}
My QueryOver:
public override PagedList<AccountantViewInfo> Execute()
{
ExpressionProcessor.RegisterCustomProjection(
() => default(Participant).TotalPersonalCosts,
expr => ParticipantExtensions.ProcessTotalPersonalCosts(expr.Expression));
AccountantViewInfo infoLine = null;
Trip tr = null;
Participant pa = null;
Cost c = null;
Price p = null;
var infoLines = Session.QueryOver(() => tr)
.JoinAlias(() => tr.Participants, () => pa);
if (_status == 0)
infoLines.Where(() => pa.TotalCostApprovalStatus == TotalCostApprovalStatus.CostPrinted || pa.TotalCostApprovalStatus == TotalCostApprovalStatus.CostPaid);
else if (_status == 1)
infoLines.Where(() => pa.TotalCostApprovalStatus == TotalCostApprovalStatus.CostPrinted);
else
infoLines.Where(() => pa.TotalCostApprovalStatus == TotalCostApprovalStatus.CostPaid);
infoLines.WhereRestrictionOn(() => pa.Employee.Id).IsIn(_employeeIds)
.Select(
Projections.Property("pa.Id").WithAlias(() => infoLine.Id),
Projections.Property("pa.Employee").WithAlias(() => infoLine.Employee),
Projections.Property("pa.ProjectCode").WithAlias(() => infoLine.ProjectCode),
Projections.Property("tr.Id").WithAlias(() => infoLine.TripId),
Projections.Property("tr.Destination").WithAlias(() => infoLine.Destination),
Projections.Property("tr.Period").WithAlias(() => infoLine.Period),
Projections.Property("pa.TotalPersonalCosts").WithAlias(() => infoLine.Period)
);
infoLines.TransformUsing(Transformers.AliasToBean<AccountantViewInfo>());
var count = infoLines.List<AccountantViewInfo>().Count();
var items = infoLines.List<AccountantViewInfo>().ToList().Skip((_myPage - 1) * _itemsPerPage).Take(_itemsPerPage).Distinct();
return new PagedList<AccountantViewInfo>
{
Items = items.ToList(),
Page = _myPage,
ResultsPerPage = _itemsPerPage,
TotalResults = count,
};
}
Here the .Expression property is not found from expr.
I don't know what I'm doing wrong. Any help or alternatives would be much appreciated!
Solution with Projection.Sum() thx to xanatos
.Select(
Projections.Group(() => pa.Id).WithAlias(() => infoLine.Id),
Projections.Group(() => pa.Employee).WithAlias(() => infoLine.Employee),
Projections.Group(() => pa.ProjectCode).WithAlias(() => infoLine.ProjectCode),
Projections.Group(() => tr.Id).WithAlias(() => infoLine.TripId),
Projections.Group(() => tr.Destination).WithAlias(() => infoLine.Destination),
Projections.Group(() => tr.Period).WithAlias(() => infoLine.Period),
Projections.Sum(() => c.Price.Amount).WithAlias(() => infoLine.TotalPersonalCost)
);
You can't use unmapped columns as projection columns of a NHibernate query.
And the way you are trying to do it is conceptually wrong: the ParticipantExtensions methods will be called BEFORE executing the query to the server, and their purpose is to modify the SQL query that will be executed. An IProjection (the "thing" that is returned by ProcessTotalPersonaCosts) is a something that will be put between the SELECT and the FROM in the query. The TotalCosts can't be returned by the SQL server because the SQL doesn't know about TotalCosts
I am working on a simple mapping EntityFramework <> DTO's , it's working perfecly excepto for the deferred execution , I have the following code :
public abstract class Assembler<TDto, TEntity> : IAssembler<TDto, TEntity>
where TEntity : EntityBase , new ()
where TDto : DtoBase, new ()
{
public abstract TDto Assemble(TEntity domainEntity);
public abstract TEntity Assemble(TEntity entity, TDto dto);
public virtual IQueryable<TDto> Assemble(IQueryable<TEntity> domainEntityList)
{
List<TDto> dtos = Activator.CreateInstance<List<TDto>>();
foreach (TEntity domainEntity in domainEntityList)
{
dtos.Add(Assemble(domainEntity));
}
return dtos.AsQueryable();
}
public virtual IQueryable<TEntity> Assemble(IQueryable<TDto> dtoList)
{
List<TEntity> domainEntities = Activator.CreateInstance<List<TEntity>>();
foreach (TDto dto in dtoList)
{
domainEntities.Add(Assemble(null, dto));
}
return domainEntities.AsQueryable();
}
}
Sample Assembler :
public partial class BlogEntryAssembler : Assembler<BlogEntryDto, BlogEntry>, IBlogEntryAssembler
{
public override BlogEntry Assemble(BlogEntry entity, BlogEntryDto dto)
{
if (entity == null)
{
entity = new BlogEntry();
}
/*
entity.Id = dto.Id;
entity.Created = dto.Created;
entity.Modified = dto.Modified;
entity.Header = dto.Header;
*/
base.MapPrimitiveProperties(entity, dto);
this.OnEntityAssembled(entity);
return entity;
}
public override BlogEntryDto Assemble(BlogEntry entity)
{
BlogEntryDto dto = new BlogEntryDto();
//dto.Id = entity.Id;
//dto.Modified = entity.Modified;
//dto.Created = entity.Created;
//dto.Header = entity.Header;
base.MapPrimitiveProperties(dto, entity);
dto.CategoryName = entity.Category.Name;
dto.AuthorUsername = entity.User.Username;
dto.AuthorFirstName = entity.User.FirstName;
dto.AuthorLastName = entity.User.LastName;
dto.TagNames = entity.Tags.Select(t => t.Name)
.ToArray();
dto.TagIds = entity.Tags.Select(t => t.Id)
.ToArray();
dto.VotedUpUsernames = entity.BlogEntryVotes.Where(v => v.Vote > 0)
.Select(t => t.User.Username)
.ToArray();
dto.VotedDownUsernames = entity.BlogEntryVotes.Where(v => v.Vote < 0)
.Select(t => t.User.Username)
.ToArray();
// Unmapped
dto.FileCount = entity.BlogEntryFiles.Count();
dto.CommentCount = entity.BlogEntryComments.Count();
dto.VisitCount = entity.BlogEntryVisits.Count();
dto.VoteCount = entity.BlogEntryVotes.Count();
dto.VoteUpCount = entity.BlogEntryVotes.Count(v => v.Vote.Equals(1));
dto.VoteDownCount = entity.BlogEntryVotes.Count(v => v.Vote.Equals(-1));
dto.VotePuntuation = entity.BlogEntryVotes.Sum(v => v.Vote);
dto.Published = entity.Visible && entity.PublishDate <= DateTime.Now;
this.OnDTOAssembled(dto);
return dto;
}
}
my service class :
public virtual PagedResult<BlogEntryDto> GetAll(bool includeInvisibleEntries, string tag, string search, string category, Paging paging)
{
var entries = this.Repository.GetQuery()
.Include(b => b.Tags)
.Include(b => b.User)
.Include(b => b.Category)
.Include(b => b.BlogEntryFiles)
.Include(b => b.BlogEntryComments)
.Include(b => b.BlogEntryPingbacks)
.Include(b => b.BlogEntryVisits)
.Include(b => b.BlogEntryVotes)
.Include(b => b.BlogEntryImages)
.AsNoTracking();
if (!includeInvisibleEntries)
{
entries = entries.Where(e => e.Visible);
}
if (!string.IsNullOrEmpty(category))
{
entries = entries.Where(e => e.Category.Name.Equals(category, StringComparison.OrdinalIgnoreCase));
}
if (!string.IsNullOrEmpty(tag))
{
entries = entries.Where(e => e.Tags.Count(t => t.Name.Equals(tag, StringComparison.OrdinalIgnoreCase)) > 0);
}
if (!string.IsNullOrEmpty(search))
{
foreach (var item in search.Split(new char[] { ' ' }, StringSplitOptions.RemoveEmptyEntries))
{
entries = entries.Where(e => e.Header.Contains(item));
}
}
return this.Assembler.Assemble(entries).GetPagedResult(paging);
}
When I call the GetAll method it returns and converts all the entities in the table to Dto's and only then it pages the resulting collection, of course that's not what I was expecting.. I would like to execute the code inside my Assemble method once the paging has been done, any idea?
PS : I know I could use Automapper, just trying to learn the internals.
In your assembler you add the projected DTOs to a List<TDto>. That's detrimental in two ways. First, it is forced execution, because the list is filled and then returned. Second, at that moment you switch from LINQ to Entities to LINQ to objects and there is no way back. You can convert the list to IQueryable again by AsQueryable, but that does not re-inject the EF query provider. In fact, the conversion is useless.
That's why AutoMapper's ProjectTo<T> statement is so cool. It transfers expressions that come after the To all the way back to the original IQueryable and, hence, its query provider. If these expressions contain paging statements (Skip/Take) these will be translated into SQL. So I think that you'll quickly come to the conclusion you better use AutoMapper after all.
public virtual IQueryable<TDto> Assemble(IQueryable<TEntity> domainEntityList)
{
List<TDto> dtos = Activator.CreateInstance<List<TDto>>();
foreach (TEntity domainEntity in domainEntityList)
{
dtos.Add(Assemble(domainEntity));
}
return dtos.AsQueryable();
}
The foreach loop in the above code is the point at which you're executing the query on the database server.
As you can see from this line:
return this.Assembler.Assemble(entries).GetPagedResult(paging);
This method is getting called before GetPagedResult(paging).. so that is the reason paging happens on the full result set.
You should understand that enumerating a query (foreach) requires the query to run. Your foreach loop processes each and every record returned by that query.. it's too late at this point for a Paging method to do anything to stop it!
I have this method:
public CampaignCreative GetCampaignCreativeById(int id)
{
using (var db = GetContext())
{
return db.CampaignCreatives
.Include("Placement")
.Include("CreativeType")
.Include("Campaign")
.Include("Campaign.Handshake")
.Include("Campaign.Handshake.Agency")
.Include("Campaign.Product")
.AsNoTracking()
.Where(x => x.Id.Equals(id)).FirstOrDefault();
}
}
I would like to make the list of Includes dynamic. I tried:
public CampaignCreative GetCampaignCreativeById(int id, string[] includes)
{
using (var db = GetContext())
{
var query = db.CampaignCreatives;
foreach (string include in includes)
{
query = query.Include(include);
}
return query.AsNoTracking()
.Where(x => x.Id.Equals(id)).FirstOrDefault();
}
}
But it didn't compile. I got this error:
Cannot implicitly convert type 'System.Data.Entity.Infrastructure.DbQuery' to 'System.Data.Entity.DbSet'. An explicit conversion exists (are you missing a cast?)
Does anyone know how to make the list of Includes dynamic?
Thanks
I am more fond of the non-string expressive way of defining includes. Mainly because it doesn't rely on magic strings.
For the example code, it would look something like this:
public CampaignCreative GetCampaignCreativeById(int id) {
using (var db = GetContext()) {
return db.CampaignCreatives
.Include(cc => cc.Placement)
.Include(cc => cc.CreativeType)
.Include(cc => cc.Campaign.Select(c =>
c.Handshake.Select(h => h.Agency)))
.Include(cc => cc.Campaign.Select(c => c.Product)
.AsNoTracking()
.Where(x => x.Id.Equals(id))
.FirstOrDefault();
}
}
And to make those dynamic, this is how you do that:
public CampaignCreative GetCampaignCreativeById(
int id,
params Expression<Func<T, object>>[] includes
) {
using (var db = GetContext()) {
var query = db.CampaignCreatives;
return includes
.Aggregate(
query.AsQueryable(),
(current, include) => current.Include(include)
)
.FirstOrDefault(e => e.Id == id);
}
}
Which is used like this:
var c = dataService.GetCampaignCreativeById(
1,
cc => cc.Placement,
cc => cc.CreativeType,
cc => cc.Campaign.Select(c => c.Handshake.Select(h => h.Agency)),
cc => cc.Campaign.Select(c => c.Product
);
Make the query variable queryable:
public CampaignCreative GetCampaignCreativeById(int id, string[] includes)
{
using (var db = GetContext())
{
var query = db.CampaignCreatives.AsQueryable();
foreach (string include in includes)
{
query = query.Include(include);
}
return query
.AsNoTracking()
.Where(x => x.Id.Equals(id))
.FirstOrDefault();
}
}
Giving the compiler a hint by using IQueryable<CampaignCreative> instead of var will work too.
IQueryable<CampaignCreative> query = db.CampaignCreatives;
// or
DbQuery<CampaignCreative> query = db.CampaignCreatives;
When using var the compiler infers DbSet<T> for query which is more specific than the type returned by Include (which is DbQuery<T> (=base class of DbSet<T>) implementing IQueryable<T>), so you can't assign the result to the query variable anymore. Hence the compiler error on the query = query.Include(include) line.
I wrote this method to retrieve any set of entity dynamically based on their types. I used the IDbEntity interface to provide a valid key to search the userId in all the classes.
The Util.GetInverseProperties<T>() method is used to get the properties needed in the Include statement.
public IEnumerable<T> GetItems<T>(string userId) where T : class, IDbEntity
{
var query = db.Set<T>().Where(l => l.UserId==userId);
var props = Util.GetInverseProperties<T>();
foreach (var include in props)
query = query.Include(include.Name);
return query
.AsNoTracking()
.ToList();
}
public interface IDbEntity
{
public string UserId { get; set; }
}
public static List<PropertyInfo> GetInverseProperties<T>()
{
return typeof(T)
.GetProperties()
.Where(p => Attribute.IsDefined(p, typeof(InversePropertyAttribute)))
.ToList();
}