We Keep Coding

Return no result for NEST C# but Query DSL return result

I have this Query DSL which return the correct result when I query directly at ElasticSearch
GET /person/_search
{
"query": {
"match": {
"nameDetails.nameValue.firstName": {
"query": "Fawsu"
}
}
}
}
}
But in NEST C#, it doesn't return any result. May I know what's wrong with my syntax?
var response = _elasticClient.Search<Person> (s => s
.Index("person")
.Query(q => q
.Match(m => m
.Field(f => f.NameDetails.Name.First().NameValue.FirstName)
.Query("Fawsu")
)
)
);
or
var response = _elasticClient.Search<Person> (s => s
.Index("person")
.Query(q => q
.Match(m => m
.Field(f => f.NameDetails.Name[0].NameValue.FirstName)
.Query("Fawsu")
)
)
);
How do I see the query generated by NEST to troubleshoot this?

The expression in .Field(f => f.NameDetails.Name[0].NameValue.FirstName) looks like it has an extra level, Name, to the object graph compared to the expected string. I would expect the output of the expression would be
nameDetails.name.nameValue.firstName
which will not match the field "nameDetails.nameValue.firstName" in the query.
You can see what NEST sends to Elasticsearch in a number of ways

NEST won't allow me to add a func parameter

Okay, I have a weird issue and I'm not sure if it's just because they're funcs and funcs are quite new to me (the whole idea of a func annoys me a bit).
I need to call the exact same query twice, because I need 2 graphs/data sets. One for each parameter. Because I don't want to duplicate my query, I've added a func parameter with my class, so I can strongly typed select the correct property on my class.
SearchDescriptor<Stats> Query(SearchDescriptor<Stats> qc, Func<Stats, double> field)
{
var query = qc
.Aggregations(aggs => aggs
.DateHistogram("histogram", d => d
.Field(f => f.Timestamp)
.FixedInterval("30m")
.MinimumDocumentCount(1)
.Aggregations(childAggs => childAggs
.Average("3", f => f
.Field(fi => field(fi))
)
)
)
)
.Query(q =>
q.Bool(b =>
b.Filter(
f => f.MatchPhrase(mp => mp
.Field(fi => fi.Name)
.Query(name)
),
f => f.DateRange(r => r
.Field(fi => fi.Timestamp)
.GreaterThanOrEquals(DateTime.UtcNow.AddDays(-1))
.LessThanOrEquals(DateTime.UtcNow)
)
)
)
);
return query;
}
var cpu = await GetByScrolling<Stats>(client, qc => Query(qc, fi => fi.CpuPercent));
var memory = await GetByScrolling<Stats>(client, qc => Query(qc, fi => fi.MemoryUsage));
This compiles and should work, but unfortunately the generated JSON turns out like this:
"aggs": {
"3": {
"avg": {
"field": "field"
}
}
}
As you can see, the value is field, which is definitely wrong. It's like the internal part of NEST does some kind of .ToString() on whatever argument is there, which is why it's returning field instead of what I put in my func.
Any idea if this is supposed to be like this? The aggs in my JSON also corresponds to the .Aggregations(aggs => aggs lambda naming.

Ah, as I said, I am new to funcs and also Expression, which turns out is what I need. It doesn't actually evaluate the func - I need to give it an expression.
SearchDescriptor<Stats> Query(SearchDescriptor<Stats> qc, Expression<Func<Stats, double>> field)
{
......
}
and then give it the expression:
.Average("3", f => f
.Field(field)
)
Then it evaluates it and works perfectly.

Can this query about finding missing keys be improved? (either SQL or LINQ)

I am developing a ASP.NET MVC website and is looking a way to improve this routine. It can be improved either at LINQ level or SQL Server level. I hope at best we can do it within one query call.
Here is the tables involved and some example data:
We have no constraint that every Key has to have each LanguageId value, and indeed the business logic does not allow such contraint. However, at application level, we want to warn the admin that a key is missing a/some language values. So I have this class and query:
public class LocalizationKeyWithMissingCodes
{
public string Key { get; set; }
public IEnumerable<string> MissingCodes { get; set; }
}
This method get the Key list, as well as any missing codes (for example, if we have en + jp + ch language codes, and the key only has values for en + ch, the list will contains jp):
public IEnumerable<LocalizationKeyWithMissingCodes> GetAllKeysWithMissingCodes()
{
var languageList = Utils.ResolveDependency<ILanguageRepository>().GetActive();
var languageIdList = languageList.Select(q => q.Id);
var languageIdDictionary = languageList.ToDictionary(q => q.Id);
var keyList = this.GetActive()
.Select(q => q.Key)
.Distinct();
var result = new List<LocalizationKeyWithMissingCodes>();
foreach (var key in keyList)
{
// Get missing codes
var existingCodes = this.Get(q => q.Active && q.Key == key)
.Select(q => q.LanguageId);
// ToList to make sure it is processed at application
var missingLangId = languageList.Where(q => !existingCodes.Contains(q.Id))
.ToList();
result.Add(new LocalizationKeyWithMissingCodes()
{
Key = key,
MissingCodes = missingLangId
.Select(q => languageIdDictionary[q.Id].Code),
});
}
result = result.OrderByDescending(q => q.MissingCodes.Count() > 0)
.ThenBy(q => q.Key)
.ToList();
return result;
}
I think my current solution is not good, because it make a query call for each key. Is there a way to improve it, by either making it faster, or pack within one query call?
EDIT: This is the final query of the answer:
public IQueryable<LocalizationKeyWithMissingCodes> GetAllKeysWithMissingCodes()
{
var languageList = Utils.ResolveDependency<ILanguageRepository>().GetActive();
var localizationList = this.GetActive();
return localizationList
.GroupBy(q => q.Key, (key, items) => new LocalizationKeyWithMissingCodes()
{
Key = key,
MissingCodes = languageList
.GroupJoin(
items,
lang => lang.Id,
loc => loc.LanguageId,
(lang, loc) => loc.Any() ? null : lang)
.Where(q => q != null)
.Select(q => q.Code)
}).OrderByDescending(q => q.MissingCodes.Count() > 0) // Show the missing keys on the top
.ThenBy(q => q.Key);
}

Another possibility, using LINQ:
public IEnumerable<LocalizationKeyWithMissingCodes> GetAllKeysWithMissingCodes(
List<Language> languages,
List<Localization> localizations)
{
return localizations
.GroupBy(x => x.Key, (key, items) => new LocalizationKeyWithMissingCodes
{
Key = key,
MissingCodes = languages
.GroupJoin( // check if there is one or more match for each language
items,
x => x.Id,
y => y.LanguageId,
(x, ys) => ys.Any() ? null : x)
.Where(x => x != null) // eliminate all languages with a match
.Select(x => x.Code) // grab the code
})
.Where(x => x.MissingCodes.Any()); // eliminate all complete keys
}

Here is the SQL logic to identify the keys that are missing "complete" language assignments:
SELECT
all.[Key],
all.LanguageId
FROM
(
SELECT
loc.[Key],
lang.LanguageId
FROM
Language lang
FULL OUTER JOIN
Localization loc
ON (1 = 1)
WHERE
lang.Active = 1
) all
LEFT JOIN
Localization loc
ON (loc.[Key] = all.[Key])
AND (loc.LanguageId = all.LanguageId)
WHERE
loc.[Key] IS NULL;
To see all keys (instead of filtering):
SELECT
all.[Key],
all.LanguageId,
CASE WHEN loc.[Key] IS NULL THEN 1 ELSE 0 END AS Flagged
FROM
(
SELECT
loc.[Key],
lang.LanguageId
FROM
Language lang
FULL OUTER JOIN
Localization loc
ON (1 = 1)
WHERE
lang.Active = 1
) all
LEFT JOIN
Localization loc
ON (loc.[Key] = all.[Key])
AND (loc.LanguageId = all.LanguageId);

your code seems to be doing a lot of database query and materialization..
in terms of LINQ, the single query would look like this..
we take the cartesian product of language and localization tables to get all combinations of (key, code) and then subtract the (key, code) tuples that exist in the relationship. this gives us the (key, code) combination that don't exist.
var result = context.Languages.Join(context.Localizations, lang => true,
loc => true, (lang, loc) => new { Key = loc.Key, Code = lang.Code })
.Except(context.Languages.Join(context.Localizations, lang => lang.Id,
loc => loc.LanguageId, (lang, loc) => new { Key = loc.Key, Code = lang.Code }))
.GroupBy(r => r.Key).Select(r => new LocalizationKeyWithMissingCodes
{
Key = r.Key,
MissingCodes = r.Select(kc => kc.Code).ToList()
})
.ToList()
.OrderByDescending(lkmc => lkmc.MissingCodes.Count())
.ThenBy(lkmc => lkmc.Key).ToList();
p.s. i typed this LINQ query on the go, so let me know if it has syntax issues..
the gist of the query is that we take a cartesian product and subtract matching rows.

Elastic search query making string match case insensitive

filterInputs.profileId = "d12";
var results = client
.Search<StockBaseEntity>(s => s
.Type("item")
.Take(1000)
.Filter(f => f
.Bool(bb => bb
.Must(ms =>
{
return
!ms.Term("profileId", filterInputs.profileId)
})))
.Sort(so => so.OnField("sortScore").Ascending())
);
"profileId" can be in small or capitals or mixed. I want to return result irrespective of the case.
How can i do this?
Currently, its treating d123 different from D123.
OR da different from DA, different from dA.
(All these should be same).
How can i do this?

If you are using the standard analyzer for the field profileId (which I guess you are otherwise you wouldn't be asking this question), then the values are stored in lowercase in Elasticsearch index. You need to lowercase the value of filterInputs.profileId and pass it in the Term() filter.
var results = client.Search<StockBaseEntity>(s => s
.Type("item")
.Take(1000)
.Filter(f => f
.Bool(bb => bb
.Must(ms =>
{
return !ms.Term("profileId", filterInputs.profileId.ToLowerInvariant());
})))
.Sort(so => so.OnField("sortScore").Ascending()));

Retain default order for Linq Contains

I want to retain the default order that comes from sql, after processing by Linq also.I know this question has been asked before. Here is a link Linq Where Contains ... Keep default order.
But still i couldn't apply it to my linq query correctly. could anyone pls help me with this? Thanks!
Here is the query
var x = db.ItemTemplates.Where(a => a.MainGroupId == mnId)
.Where(a => a.SubGruopId == sbId)
.FirstOrDefault();
var ids = new List<int> { x.Atribute1, x.Atribute2, x.Atribute3, x.Atribute4 };
var y = db.Atributes.Where(a => ids.Contains(a.AtributeId))
.Select(g => new
{
Name = g.AtributeName,
AtType = g.AtributeType,
Options = g.atributeDetails
.Where(w=>w.AtributeDetailId!=null)
.Select(z => new
{
Value=z.AtributeDetailId,
Text=z.AtDetailVal
})
});

Your assumption is wrong. SQL server is the one that is sending the results back in the order you are getting them. However, you can fix that:
var x = db.ItemTemplates.Where(a => a.MainGroupId == mnId)
.Where(a => a.SubGruopId == sbId)
.FirstOrDefault();
var ids = new List<int> { x.Atribute1, x.Atribute2, x.Atribute3, x.Atribute4 };
var y = db.Atributes.Where(a => ids.Contains(a.AtributeId))
.Select(g => new
{
Id = g.AtributeId,
Name = g.AtributeName,
AtType = g.AtributeType,
Options = g.atributeDetails
.Where(w=>w.AtributeDetailId!=null)
.Select(z => new
{
Value=z.AtributeDetailId,
Text=z.AtDetailVal
})
})
.ToList()
.OrderBy(z=>ids.IndexOf(z.Id));
Feel free to do another select after the orderby to create a new anonymous object without the Id if you absolutely need it to not contain the id.
PS. You might want to correct the spelling of Attribute, and you should be consistent in if you are going to prefix your property names, and how you do so. Your table prefixes everything with Atribute(sp?), and then when you go and cast into your anonymous object, you remove the prefix on all the properties except AtributeType, which you prefix with At. Pick one and stick with it, choose AtName, AtType, AtOptions or Name, Type, Options.