We Keep Coding

ElasticSearch (NEST) Query to exclude specific term results

I'm trying to find a QueryContainer query that I can perform on an ElasticSearch result that will essentially filter out any "A" status Items in my resultset. My ProductIndex document contains a field named "StatusCode", and I don't want to return these "A" status' to my search resultsets... I'm having the hardest time finding a way to remove these items.
This query properly finds these "A" status':
.Match(qm => qm
.Field(f => f.StatusCode)
.Query("A"));
But I want to do the opposite (not get all the "A" status items, but exclude them)
Based upon other threads I read on here, I came up with the following query, but it's not filtering out these results:
.Bool(b => b
.MustNot(mn => mn
.Terms(t => t
.Field(f => f.StatusCode)
.Terms("A")
)
));
and
.Bool(b => b
.MustNot(mn => mn
.Term(t => t
.Field(f => f.StatusCode).Value("A")
)
));
But neither removes results that have an "A" statuscode
Kibana value of a result that is still being returned, but has the status "A" code:

The MustNot filter continues to not work for me - I ended up resolving this by performing two search queryies - one based upon the result set obtained from from the first query
.Match(qm => qm
.Field(f => f.StatusCode)
.Query("A"));
This properly filtered the items to only return the ones I was trying to remove from my primary query. I obtained this document list, then applied this list as an exclusion filter to my underlying request, thus filtering out the items I no longer wanted to return.
Not as elegant as I would like, but got the job done.

How can I use not exist or null in elasticsearch query (EQL) by using Nest C#?

Everyone know that Nest Elasticsearch is not easy and boring to make clarified queries for looking something. I also stumpled upon this issue . As a result I could't use 'not empty' and null in my query.
var list = client.Count<LogMessage>(s => s.Index("xxx-*").Query(q =>
!q.Term(t => t.Field(f => f.Test.Suffix("keyword")).Value(null)) &&
q.Term(t => t.Field(f => f.Environment.Suffix("keyword")).Value("yyy")) &&
q.DateRange(t => t.Field(f => f.LogDate).GreaterThan(DateTime.Now.AddMinutes(-15)))));
Null or string.Empty is not working here. How can I use NOT null or Not Empty?

I think what you are looking for is:
.Query(q => q.Bool(b => b
.Must(m => m
.Exists(e => e
.Field(f => f.Test)
)
)
))
This checks for null values.

IMO this isn't much of a NEST usability issue as it's just non-trivial to do this in Elasticsearch itself. I have had success by negating a wildcard query (.Wilcard) on that field and/or using .Exists to find documents which do not have that field because null values are not stored on a document and empty values are difficult to search for in non-keyword text fields because the analyzers will not look for blank values.
Please see this answer and the one below it which is even shorter.
https://github.com/elastic/elasticsearch/issues/7515#issuecomment-158668403

C# elastic search query, match multiple queries

I have 2 elastic search queries that need to be matched by each fetched document in different ways. The "pools" query is a terms query. Each document has a list of pools attached to it, each of them being a string, and at least one of those pools must be in the provided list of pools in the "pools" query.
The other query is actually composed of multiple queries and at least 75% percent of them should be matched.
So in order for a document to be matched, the "pools" query must always be matched and from the other query, at least 75% must be matched.
I wrote my query like this:
var matchQuery = BuildQuery(searchCriteria);
var poolQuery = BuildPoolsQueryField(searchCriteria);
// prepare the data for elasticsearch
var result = await _elasticSearchClient.SearchAsync<ElasticPersonEntity>(
p => p.Query(q => q
.Bool(b => b.Must(poolQuery).Should(matchQuery.ToArray())
.MinimumShouldMatch(MinimumShouldMatch.Percentage(75))))).ConfigureAwait(false);
But I could not find anywhere on the internet if you can chain multiple Should and Must clauses and what it happens if you chain them like this.

According to your description, your query is wrong: you need to mustpoolQuery && matchQuery(75%) so
The .MinimumShouldMatch(MinimumShouldMatch.Percentage(75) should be inside your matchQuery :
I join an example (using my data, but this should solve your problem)
.Query(q => q
.Bool(b => b
.Must(
mu => mu.Term(te => te.CntCd, "FR"),
mu => mu.Bool(bo => bo
.Should(your should query).MinimumShouldMatch(75)
)
)
)
)

How do I construct an ElasticSearch search using NEST with fields from multiple types without magic strings

I'm setting up a system where my search indexed objects are Entities which map back to a normalized SQL database. So on the SQL side I may have a Company which is linked to one or more Company_Address records, as well as Company_Email records, etc. But all of the relevant searchable fields will be in a single Company indexed document tying back to the Company's unique id field.
I'm trying to setup queries which will return the relevant id based on comparing a user provided query string with specific fields in the indexed document. I can do so with a query similar to:
var searchResponse = this.client.Search<Company>(search => search.Query(
s => s.MultiMatch(
m => m.Query(query.QueryString)
.Fields(f => f.Field(c => c.Name).Field("Address1")) // Magic strings go here
.Fuzziness(Fuzziness.Auto)
)).Index<Company>());
I'd like to be able to replace "Address1" with ca.Address1 where ca is a CompanyAddress object. Is there a way to construct a collection of FieldDescriptor objects with multiple backing types and then supply that to the search query?

You can construct a Fields instance in a number of ways:
Building a Fields instance by chaining Field instances, constructing those from member access Lambda expressions on some type T
var searchResponse = this.client.Search<Company>(search => search
.Query(q => q
.MultiMatch(m => m
.Query("query")
.Fields(
Infer.Field<Company>(c => c.Name)
.And<CompanyAddress>(c => c.Address1)
)
.Fuzziness(Fuzziness.Auto)
)
)
.Index<Company>()
);
Using nameof with the Address1 property of CompanyAddress
var searchResponse = this.client.Search<Company>(search => search
.Query(q => q
.MultiMatch(m => m
.Query(query.QueryString)
.Fields(f => f
.Field(c => c.Name)
.Field(nameof(CompanyAddress.Address1))
)
.Fuzziness(Fuzziness.Auto)
)
)
.Index<Company>()
);

Cannot add same key to dictionary more than once

Here is my code:
IEnumerable<ServiceTicket> troubletickets = db.ServiceTickets.Include(t => t.Company).Include(t => t.UserProfile);
var ticketGroups = new Dictionary<string, List<ServiceTicket>>();
ticketGroups = troubletickets
.GroupBy(o => o.DueDate).ToDictionary(
group => {
var firstOrDefault = #group.FirstOrDefault();
return firstOrDefault != null
? firstOrDefault.DueDate.HasValue
? firstOrDefault.DueDate.Value.ToShortDateString()
: ""
: "";
},
group => group.ToList()
).OrderBy(g => g.Key).ToDictionary(g => g.Key, g => g.Value);
The error that I am getting is: 'An item with the same key has already been added.' This is because the DueDate value is occasionally repeated. My question is how can I keep the key from being added if it already exists in the dictionary?

It seems that you are grouping by one value (the DueDate value), but using a different value as the dictionary key.
Can you not just use the custom code for grouping instead?
ticketGroups = troubletickets
.GroupBy(o => o.DueDate.HasValue
? o.DueDate.Value.ToShortDateString()
: "")
.ToDictionary(g => g.Key, g => g.ToList());
Note that I took our the superfluous OrderBy and second ToDictionary call - I assumed you were trying to "order" the dictionary which won't work as a plain dictionary is not ordered.

You get duplicate keys because there are two ways to get an empty string as key, either an empty group, or an empty date. The duplicate will always be the empty string. I wonder if you really intended to get an empty string as key when the group is empty. Anyway, it's not necessary, you can always filter empty groups later.
It's easier to group by date (including null) first through the database engine and then apply string formatting in memory:
IQueryable<ServiceTicket> troubletickets = db.ServiceTickets
.Include(t => t.Company)
.Include(t => t.UserProfile);
Dictionary<string, List<ServiceTicket>> ticketGroups =
troubletickets
.GroupBy(ticket => ticket.DueDate)
.AsEnumerable() // Continue in memory
.ToDictionary(g => g.Key.HasValue
? g.Key.Value.ToShortDateString()
: string.Empty,
g => g.Select(ticket => ticket));
Now the grouping is by the Key value, not by the First element in the group. The Key is never null, it's always a Nullable<DateTime>, with or without a value.
Side note: you'll notice that EF will not generate a SQL group by statement, that's because the SQL statement is "destructive": it only returns grouped columns and aggregate data, not the individual records that a LINQ GroupBy does return. For this reason, the generated SQL is pretty bloated and it may enhance performance if you place the AsEnumerable before the .GroupBy.