I have a use case where I need to do aggregation on multiple columns using C#.
I am using NEST libraries for this and I am facing the following issue
Query C# :
var searchRequest = new SearchRequest
{
SearchType = SearchType.Count,
Filter = filter,
Aggregations = new Dictionary<string, IAggregationContainer>
{
{ "a", new AggregationContainer
{
ExtendedStats = new ExtendedStatsAggregator()
{
Field = "a"
}
}
},
{ "b", new AggregationContainer
{
ExtendedStats = new ExtendedStatsAggregator()
{
Field = "b"
}
}
}
}
};
When I receive response from NEST, however I am getting only result for one aggregation. I am looking at SearchResult.Agg dictionary but it has only one entry for one aggregation field instead of two.
Let me know if I am missing soemthing or is it some issue with NEST libraries
if you are using term aggregation then you need to use aggregation with filter.
var qryRes1 = client.Search<object>(x => x
.Aggregations(ag => ag
.Filter("filter", (flt => flt
.Filter(f =>
{
FilterContainer filter = null;
filter &= f.Query(qr => qr.Term(wl => wl.OnField("a").Value("the value you need to filter for field a")));
return filter;
})
.Aggregations(agr => agr
.Terms("b", tr =>
{
TermsAggregationDescriptor<object> trmAggDescriptor = new TermsAggregationDescriptor<object>();
trmAggDescriptor.Field("b");
return trmAggDescriptor;
}))))
));
var terms = qryRes1.Aggs.Filter("filter").Terms("b");
Related
Does anyone know how to do multiple aggregations with nest?
I have found quite a few examples unfortunately none of them work.
Here's what I have:
Vehicles fields = new Vehicles();
//create a terms query
var query = new TermsQuery
{
IsVerbatim = true,
Field = "VehicleOwnerId",
Terms = new string[] { 25 },
};
var aggregations = new Dictionary<string, IAggregationContainer>
{
{ "years", new AggregationContainer
{
Terms = new TermsAggregation(nameof(fields.Year))
{
Field = new Field(nameof(fields.Year))
}
}
}
//,
//{ "makes", new AggregationContainer
// {
// Terms = new TermsAggregation("Make")
// {
// Field = new Field(nameof(fields.Make))
// }
// }
//}
};
//create the search request
var searchRequest = new SearchRequest
{
Query = query,
From = 0,
Size = 100,
Aggregations = aggregations
};
var result = client.SearchAsync<InventoryLiveView>(searchRequest).Result;
var years = result.Aggregations.Terms("years");
Dictionary<string, long> yearCounts = new Dictionary<string, long>();
foreach (var item in years.Buckets)
{
yearCounts.Add(item.Key, item.DocCount ?? 0);
}
If I just execute the code like this it works. Years returns the aggregates as expected. If I try to add another field (like the one commented out above) it fails and I get zero records.
How can I get multiple aggregates in one query? I see examples of it all over, but none of the examples I've tried seem to work and most seem to be outdated (including some in the Nest documentation).
I have also tried this approach which is pretty close to the documentation.
//create the search request
var searchRequest = new SearchRequest
{
Query = query,
From = 0,
Size = 100,
//Aggregations = aggregations
Aggregations = new AggregationDictionary
{
{
"childAgg", new ChildrenAggregation("childAgg", typeof(Vehicles ))
{
Aggregations = new AggregationDictionary
{
{"years", new TermsAggregation(nameof(fields.VehicleYear))},
{"makes", new TermsAggregation(nameof(fields.VehicleMakeName))},
{"models", new TermsAggregation(nameof(fields.VehicleModelName))},
}
}
}
}
};
var result = client.SearchAsync<Vehicles>(searchRequest).Result;
This just produces a null reference exception.
I guess I'll never have too worry about getting to proud as a programmer :)
It's too often that the solution to the problem makes me feel stupid when it reveals itself.
So my issue was that the field I was trying to use in the aggregation was text and couldn't be used. I switched everything to the ID fields and multiple aggregations work as expected.
So this version of the code works like a champ:
Vehicle fields = new Vehicle ();
//create a terms query
var query = new TermsQuery
{
IsVerbatim = true,
Field = "VehicleOwnerId",
Terms = new string[] { "30" },
};
string[] Fields = new[]
{
nameof(fields.Year),
nameof(fields.MakeId),
nameof(fields.ModelId)
};
var aggregations = new Dictionary<string, IAggregationContainer>();
foreach (string sField in Fields)
{
var termsAggregation = new TermsAggregation(sField)
{
Field = sField
};
aggregations.Add(sField, new AggregationContainer { Terms = termsAggregation });
}
//create the search request
var searchRequest = new SearchRequest
{
Query = query,
From = 0,
Size = 10,
Aggregations = aggregations
};
var result = client.SearchAsync<InventoryLiveView>(searchRequest).Result;
var years = result.Aggregations.Terms(nameof(fields.Year));
Dictionary<string, long> yearCounts = new Dictionary<string, long>();
foreach (var item in years.Buckets)
{
yearCounts.Add(item.Key, item.DocCount ?? 0);
}
The exact error from elasticsearch, which I saw using postman was:
Fielddata is disabled on text fields by default. Set fielddata=true on [MakeName] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead.
Here is my example using SearchDescriptors. My only problem is how to serialize returned results into a proper Key Value list. Is Looping through a fields list the best way to return results.
SearchDescriptor<Advert> agghDescriptor = new SearchDescriptor<Advert>();
agghDescriptor.Aggregations(ag => ag.Terms("make", a => a.Field(f => f.Make)) &&
ag.Terms("region", a => a.Field(f => f.Region)) &&
ag.Terms("city", a => a.Field(f => f.City)) &&
ag.Terms("category", a => a.Field(f => f.Category)) &&
ag.Terms("application", a => a.Field(f => f.Application)) &&
ag.Terms("portalId", a => a.Field(f => f.PortalId)) &&
ag.Terms("isActiveAuctionAdvert", a => a.Field(f => f.IsActiveAuctionAdvert)) &&
ag.Terms("isBargainAccount", a => a.Field(f => f.IsBargainAccount)) &&
ag.Terms("condition", a => a.Field(f => f.Condition))
);
agghDescriptor.Size(0);
var json2 = _client.RequestResponseSerializer.SerializeToString(agghDescriptor);
var aggregationResult = _client.Search<Advert>(agghDescriptor);
List<string> fields = new List<string>();
fields.Add("make");
fields.Add("category");
fields.Add("region");
List<Aggregation> aggregations = new List<Aggregation>();
foreach (var field in fields)
{
var aggrs = aggregationResult.Aggregations.Terms(field);
List<AggregateItem> aggregateItems = new List<AggregateItem>();
foreach (var item in aggrs.Buckets)
{
aggregateItems.Add(new AggregateItem()
{
Count = item.DocCount ?? 0,
Key = item.Key
});
}
aggregations.Add(new Aggregation()
{
Name = field,
Aggregates = aggregateItems
});
}
I created this form to generate a list of students with the ability to filter by some criteria (on the left) and to display any information needed (from the right)
When the form is initializing at the start I am grabbing the whole student list with Entity Framework
List<Student> students = await context.Students.ToListAsync().ConfigureAwait(false);
And I am saving it into two lists:
private List<Student> _listOfAllStudents = new List<Student>();
private List<Student> _filteredStudents = new List<Student>();
Then I am performing my logic against the lists like this:
private void PrepareFilteredStudentListAccordingToFilterCheckedBoxes()
{
_filteredStudents = _listOfAllStudents;
if (ColonieFilterCheckBox.Checked)
{
_filteredStudents = _filteredStudents.Intersect(_listOfAllStudents.Where(x => x.Colonie).Select(x => x).ToList()).ToList();
}
if (NatationFilterCheckBox.Checked)
{
_filteredStudents = _filteredStudents.Intersect(_listOfAllStudents.Where(x => x.Nataion).Select(x => x).ToList()).ToList();
}
if (ExcursionFilterCheckBox.Checked)
{
_filteredStudents = _filteredStudents.Intersect(_listOfAllStudents.Where(x => x.Excursion).Select(x => x).ToList()).ToList();
}
//Rest of the code is omitted but you get the idea...
}
Same logic is being done according to display checkboxes:
private void FillDataGridViewWithFilteredStudentAccordingToDisplayCheckBoxes()
{
FilteredStudentDataGridView.Columns.Add("Id", "Numero");
FilteredStudentDataGridView.Columns.Add("LastName", "Nom");
FilteredStudentDataGridView.Columns.Add("FirstName", "Prenom");
if (MiddleNameDisplayCheckBox.Checked)
{
FilteredStudentDataGridView.Columns.Add("MiddleName", "Nom Du Pere");
}
if (BirthdayDateDisplayCheckBox.Checked)
{
FilteredStudentDataGridView.Columns.Add("DateOfBirth", "Date De Naissance");
}
//Rest of the code omitted, but same concept.
foreach (Student student in _filteredStudents)
{
List<object> rowsValues = new List<object>();
foreach (object column in FilteredStudentDataGridView.Columns)
{
string columnName = ((DataGridViewTextBoxColumn)column).Name;
if (columnName == "Id")
{
rowsValues.Add(student.StudentId);
}
if (columnName == "FirstName")
{
rowsValues.Add(student.FirstName);
}
//Code omitted.
}
object[] arrayRowsValues = rowsValues.ToArray();
FilteredStudentDataGridView.Rows.Add(arrayRowsValues);
}
}
I was wondering if there is a way to use LINQ instead of all those if blocks to filter the data according to my conditions?
I would definitely move the filtering to the database:
IQueryable<Student> studentQuery = context.Students;
if (ColonieFilterCheckBox.Checked) {
studentQuery = studentQuery.Where(x => x.Colonie);
}
if (NatationFilterCheckBox.Checked) {
studentQuery = studentQuery.Where(x => x.Nataion);
}
if (ExcursionFilterCheckBox.Checked) {
studentQuery = studentQuery.Where(x => x.Excursion);
}
var _filteredStudents = await studentQuery.ToListAsync().ConfigureAwait(false);
If you prefer local filtering logic, you can combine the conditions a little or use Reflection to simplify the code but slow it down some.
For combined conditions, you can do
_filteredStudents = _listOfAllStudents
.Where(x => (!ColonieFilterCheckBox.Checked || x.Colonie) &&
(!NatationFilterCheckBox.Checked || x.Natation) &&
(!ExcursionFilterCheckBox.Checked || x.Excursion)).ToList();
but that checks the CheckBoxs per Student.
Instead, using Reflection, you can build the code dynamically for the filter (again, assuming you name the CheckBox controls after the fields):
IEnumerable<Student> queryStudents = _listOfAllStudents;
var xParm = Expression.Parameter(typeof(Student));
foreach (var filterField in new[] { "Colonie", "Natation", "Excursion" }) {
if (((CheckBox)Controls.Find($"{filterField}CheckBox")).Checked) {
var whereLambda = (Expression<Func<Student, bool>>)Expression.Lambda(Expression.PropertyOrField(xParm, filterField), xParm);
queryStudents = queryStudents.Where(whereLambda.Compile());
}
}
_filteredStudents = queryStudents.ToList();
For your display logic, I would rename all the check boxes to match the data field names, and then use a lot of Reflection:
private void FillDataGridViewWithFilteredStudentAccordingToDisplayCheckBoxes() {
var headerText = new Dictionary<string, string> { { "Id", "Numero" }, { "LastName", "Nom" }, { "FirstName", "Prenom" }, { "MiddleName", "Nom Du Pere" },
{ "DateOfBirth", "Date De Naissance" } };
var viewColumns = new List<string> { "Id", "LastName", "FirstName" };
foreach (var possibleColumn in headerText.Keys) {
var displayColumns = Controls.Find(possibleColumn + "DisplayCheckBox", true);
if (displayColumns.Length == 1) {
if (((CheckBox)displayColumns[0]).Checked)
viewColumns.Add(possibleColumn);
}
}
//Rest of the code omitted, but same concept.
foreach (var dataFieldName in viewColumns)
FilteredStudentDataGridView.Columns.Add(dataFieldName, headerText[dataFieldName]);
foreach (var student in _filteredStudents) {
var studentType = student.GetType();
var rowValues = new List<object>();
foreach (var dataFieldName in viewColumns)
rowValues.Add(studentType.GetProperty(dataFieldName).GetValue(student, null));
FilteredStudentDataGridView.Rows.Add(rowValues.ToArray());
}
}
Note that if you care about the order of the columns displayed, you will need some logic to order or sort the headerText.Keys. In my asp implementation of something similar, I just have a manual list of procedure calls with data names in the order that I want, and the procedure checks if the data item is to be displayed (in viewColumns), then adds the data and column header.
You could reuse your linq expressions in a single method. Try to use AsQueryable extension method from System.Linq namespace instead:
private ICollection<Student> FilterStudents(ICollection<Student> students)
{
var query = students.AsQueryable();
if (ColonieFilterCheckBox.Checked)
{
query = query.Where(x=>x.Colonie);
}
if (NatationFilterCheckBox.Checked)
{
query = query.Where(x=>x.Nation);
}
if (ExcursionFilterCheckBox.Checked)
{
query = query.Where(x=>x.Excursion);
}
return query.ToList();
}
I am very new to c# and LINQ, now I need to do some aggregation on the navigation tables nested one after the other, the result was generated as below:
[{
"ForumMain":"General Forum",
"Forums": [{
"Title":"My First Forum",
"TopicCount":4,
"PostCount":[2,0,1,0]
},
{
"Title":"My Second Forum",
"TopicCount":0,
"PostCount":[]
}]
}]
I wanted PostCount also be the sum, but it is giving debug null error when I try to sum PostCount, please see my linq query below:
var data = ForumCategories.Select(f => new
{
ForumMain = f.CategoryName,
Forums = f.Fora.Select(t => new
{
Title = t.ForumName,
TopicCount = t.Topics.Count,
PostCount = t.Topics
.Select(m => m.ForumPosts.Count == null ? 0 : m.ForumPosts.Count)
})
});
From what I understand what you want under PostCount to have the total number of post you have for all the topics in that Forum. Right?
If so:
var data = db.ForumCategories.Select(f => new
{
ForumMain = f.CategoryName,
Forums = f.Fora.Select(t => new
{
Title = t.ForumName,
TopicCount = t.Topics.Count,
PostCount = t.Topics.SelectMany(x => x.ForumPosts).Count()
})
});
I'm playing around with the new driver of mongodb 2.0, and looking for adding some facetted searchs (Temporary move ,before using elastic search).
Here is some method where I created to build the agreggation. I guess that it should work.
As parameter I passed also a filterdefinition in the method.
But I don't find how to limit my agreggation to the filter.
Any Idea ???
private void UpdateFacets(SearchResponse response, FilterDefinition<MediaItem> filter, ObjectId dataTableId)
{
response.FacetGroups =new List<SearchFacetGroup>();
SearchFacetGroup group = new SearchFacetGroup()
{
Code = "CAMERAMODEL",
Display = "Camera model",
IsOptional = false
};
using (IDataAccessor da = NodeManager.Instance.GetDataAccessor(dataTableId))
{
var collection = da.GetCollection<MediaItem>();
var list = collection.Aggregate()
.Group(x => ((ImageMetaData) x.MetaData).Exif.CameraModel, g => new { Model = g.Key, Count = g.Count() })
.ToListAsync().Result;
foreach (var l in list)
{
group.Facets.Add(new SearchFacetContainer()
{
Code = l.Model,
Display = l.Model,
Hits = l.Count,
IsSelected = false
});
}
}
response.FacetGroups.Add(group);
}
I haven't used facet, but with Mongo driver Aggregate has .Match operation that accepts a filterdefinition.
collection1.Aggregate().Match(filter)
I need to preform a query that check if a collection is in given collection, just like the regular in operation but for collections.
class Post
{
public string[] Tags {get;set;}
}
session.Queury<Post>.Where(x=>x.Tags.in(new[]{".net","c#","RavenDB"})).ToList();
so if i have in my DB:
new Post{Tags= new[]{"C#",".net"}};
it will be returned
but if i have:
new Post{Tags= new[]{"C#",".net","SQLServer"}};
it will not be returned.
Update:
what i am trying to do is this:
session.Query<Post>()
.Where(x => x.Tags.All(y => y.In(new[] { "C#", ".net", "RavenDB" })))
.ToList();
but i got System.NotSupportedException.
I manage to find a solution:
static void Main(string[] args)
{
var sessionStore = new EmbeddableDocumentStore
{
RunInMemory = true,
UseEmbeddedHttpServer = true,
Conventions =
{
DefaultQueryingConsistency = ConsistencyOptions.AlwaysWaitForNonStaleResultsAsOfLastWrite
}
};
sessionStore.Initialize();
using (var session = sessionStore.OpenSession())
{
var allTags = new[] {"C#", ".net", "RavenDB", "Linux", "Mac"};
var tagsCollection = new[] {"C#", ".net", "RavenDB"};
var complementTagsCollection = allTags.Except(tagsCollection).ToList();
session.Store(new Post
{
Tags = new List<string>{"C#",".net"}
});
session.SaveChanges();
// Posts where all their tags are in tagsCollection
var result = session.Query<Post>().Where(x => !x.Tags.In(complementTagsCollection)).ToList();
}
}
The way IN works, it matches ANY of them.
If you want to match all you have to do a separate check for each.