We Keep Coding

How to use GroupBy on an index in RavenDB?

I have this document, a post :
{Content:"blabla",Tags:["test","toto"], CreatedOn:"2019-05-01 01:02:01"}
I want to have a page that displays themost used tags since the last 30 days.
So far I tried to create an index like this
public class Toss_TagPerDay : AbstractIndexCreationTask<TossEntity, TagByDayIndex>
{
public Toss_TagPerDay()
{
Map = tosses => from toss in tosses
from tag in toss.Tags
select new TagByDayIndex()
{
Tag = tag,
CreatedOn = toss.CreatedOn.Date,
Count = 1
};
Reduce = results => from result in results
group result by new { result.Tag, result.CreatedOn }
into g
select new TagByDayIndex()
{
Tag = g.Key.Tag,
CreatedOn = g.Key.CreatedOn,
Count = g.Sum(i => i.Count)
};
}
}
And I query it like that
await _session
.Query<TagByDayIndex, Toss_TagPerDay>()
.Where(i => i.CreatedOn >= firstDay)
.GroupBy(i => i.Tag)
.OrderByDescending(g => g.Sum(i => i.Count))
.Take(50)
.Select(t => new BestTagsResult()
{
CountLastMonth = t.Count(),
Tag = t.Key
})
.ToListAsync()
But this gives me the error
Message: System.NotSupportedException : Could not understand expression: from index 'Toss/TagPerDay'.Where(i => (Convert(i.CreatedOn, DateTimeOffset) >= value(Toss.Server.Models.Tosses.BestTagsQueryHandler+<>c__DisplayClass3_0).firstDay)).GroupBy(i => i.Tag).OrderByDescending(g => g.Sum(i => i.Count)).Take(50).Select(t => new BestTagsResult() {CountLastMonth = t.Count(), Tag = t.Key})
---- System.NotSupportedException : GroupBy method is only supported in dynamic map-reduce queries
Any idea how can I make this work ? I could query for all the index data from the past 30 days and do the groupby / order / take in memory but this could make my app load a lot of data.

The results from the map-reduce index you created will give you the number of tags per day. You want to have the most popular ones from the last 30 days so you need to do the following query:
var tagCountPerDay = session
.Query<TagByDayIndex, Toss_TagPerDay>()
.Where(i => i.CreatedOn >= DateTime.Now.AddDays(-30))
.ToList();
Then you can the the client side grouping by Tag:
var mostUsedTags = tagCountPerDay.GroupBy(x => x.Tag)
.Select(t => new BestTagsResult()
{
CountLastMonth = t.Count(),
Tag = t.Key
})
.OrderByDescending(g => g.CountLastMonth)
.ToList();

#Kuepper
Based on your index definition. You can handle that by the following index:
public class TrendingSongs : AbstractIndexCreationTask<TrackPlayedEvent, TrendingSongs.Result>
{
public TrendingSongs()
{
Map = events => from e in events
where e.TypeOfTrack == TrackSubtype.song && e.Percentage >= 80 && !e.Tags.Contains(Podcast.Tags.FraKaare)
select new Result
{
TrackId = e.TrackId,
Count = 1,
Timestamp = new DateTime(e.TimestampStart.Year, e.TimestampStart.Month, e.TimestampStart.Day)
};
Reduce = results => from r in results
group r by new {r.TrackId, r.Timestamp}
into g
select new Result
{
TrackId = g.Key.TrackId,
Count = g.Sum(x => x.Count),
Timestamp = g.Key.Timestamp
};
}
}
and the query using facets:
from index TrendingSongs where Timestamp between $then and $now select facet(TrackId, sum(Count))

The reason for the error is that you can't use 'GroupBy' in a query made on an index.
'GroupBy' can be used when performing a 'dynamic query',
i.e. a query that is made on a collection, without specifying an index.
See:
https://ravendb.net/docs/article-page/4.1/Csharp/client-api/session/querying/how-to-perform-group-by-query

I solved a similar problem, by using AdditionalSources that uses dynamic values.
Then I update the index every morning to increase the Earliest Timestamp. await IndexCreation.CreateIndexesAsync(new AbstractIndexCreationTask[] {new TrendingSongs()}, _store);
I still have to try it in production, but my tests so far look like it's a lot faster than the alternatives. It does feel pretty hacky though and I'm surprised RavenDB does not offer a better solution.
public class TrendingSongs : AbstractIndexCreationTask<TrackPlayedEvent, TrendingSongs.Result>
{
public DateTime Earliest = DateTime.UtcNow.AddDays(-16);
public TrendingSongs()
{
Map = events => from e in events
where e.TypeOfTrack == TrackSubtype.song && e.Percentage >= 80 && !e.Tags.Contains(Podcast.Tags.FraKaare)
&& e.TimestampStart > new DateTime(TrendingHelpers.Year, TrendingHelpers.Month, TrendingHelpers.Day)
select new Result
{
TrackId = e.TrackId,
Count = 1
};
Reduce = results => from r in results
group r by new {r.TrackId}
into g
select new Result
{
TrackId = g.Key.TrackId,
Count = g.Sum(x => x.Count)
};
AdditionalSources = new Dictionary<string, string>
{
{
"TrendingHelpers",
#"namespace Helpers
{
public static class TrendingHelpers
{
public static int Day = "+Earliest.Day+#";
public static int Month = "+Earliest.Month+#";
public static int Year = "+Earliest.Year+#";
}
}"
}
};
}
}

LINQ sum not performed when values are identical

I need to sum elements of same type starting from 2 LINQ queries.
Below is my code:
var query1 = from d in _contextProvider.Context.Documents
where d.TransportId == transportId
group d by d.Type
into dg
select new { DocumentType = dg.Key.ToString(), DocumentCount = dg.Count() };
var query2 = from n in _contextProvider.Context.NotificationDocuments
where n.TransportId == transportId
group n by n.TransportId
into nd
select new { DocumentType = "Notification", DocumentCount = nd.Count() };
var query_collapsed = query1.Union(query2)
.GroupBy(p => new { DocumentType = p.DocumentType })
.Select(g => new DocumentCounters() { DocumentType = g.Key.DocumentType, DocumentCount = g.Sum(p => p.DocumentCount) });
Example: below let's analyse values for DocumentType equals to Notification.
Values of query1:
Values of query2:
The collapsed query :
That's correct: 1 + 2 = 3
The problem: I noticed that whenever the count for Notification in query1 is equals to the count for Notification in query2, then the sum is not performed.
Example:
2 + 2 = 2
or
3 + 3 = 3
Any ideas ?

LINQ Union will remove duplicate entries. If you want to merge the two sequences you can use Concat like so:
var query_collapsed = query1.Concat(query2)
.GroupBy(p => new { DocumentType = p.DocumentType })
.Select(g => new DocumentCounters() { DocumentType = g.Key.DocumentType, DocumentCount = g.Sum(p => p.DocumentCount) });

How to return Distinct Row using LINQ

I have two rows which have all the data same except one column.
I want to show only one row on the UI but one row which has different data should be shown as comma seperated values.
Sample Data
PricingID Name Age Group
1 abc 56 P1
1 abc 56 P2
Output should be :
PricingID Name Age Group
1 abc 56 P1,P2
I am using this approach but it is not working , it gives me two rows only but data i am able to concatenate with comma.
List<PricingDetailExtended> pricingDetailExtendeds = _storedProcedures.GetPricingAssignment(pricingScenarioName, regionCode, productCode, stateCode, UserId, PricingId).ToList();
var pricngtemp = pricingDetailExtendeds.Select(e => new
{
PricingID = e.PricingID,
OpportunityID = e.OpportunityID,
ProductName = e.ProductName,
ProductCD = e.ProductCD
});
pricingDetailExtendeds.ForEach(e=>
{
e.ProductCD = string.Join(",",string.Join(",", (pricngtemp.ToList().Where(p => p.PricingID == e.PricingID).Select(k => k.ProductCD).ToArray())).Split(',').Distinct().ToArray());
e.OpportunityID =string.Join(",", string.Join(",", (pricngtemp.ToList().Where(p => p.PricingID == e.PricingID).Select(k => k.OpportunityID).ToArray())).Split(',').Distinct().ToArray());
e.ProductName =string.Join(",", string.Join(",", (pricngtemp.ToList().Where(p => p.PricingID == e.PricingID).Select(k => k.ProductName).ToArray())).Split(',').Distinct().ToArray());
}
);
// pricingDetailExtendeds = GetUniquePricingList(pricingDetailExtendeds);
return pricingDetailExtendeds.Distinct().AsEnumerable();
Any body can suggest me better approach and how to fix this issue ?
Any help is appreciated.

You want to use the GroupBy linq function.
I then use the String.Join function to make the groups comma seperated.
So something like this:
var pricingDetailExtendeds = new[]
{
new
{
PricingID = 1,
Name = "abc",
Age = 56,
Group = "P1"
},
new
{
PricingID = 1,
Name = "abc",
Age = 56,
Group = "P2"
}
};
var pricngtemp =
pricingDetailExtendeds.GroupBy(pde => new {pde.PricingID, pde.Name, pde.Age})
.Select(g => new {g.Key, TheGroups = String.Join(",", g.Select(s => s.Group))}).ToList();
You can easily extrapolate this to the other fields.
To return the PricingDetailExtended, the just create it in the select. So something like this
.Select(g => new PricingDetailExtended {
PricingID = g.Key.PricingId,
TheGroups = String.Join(",", g.Select(s => s.Group))
}).ToList();
You won't have the field TheGroups though, so just replace that field with the proper one.

An example of what I was describing in my comment would be something along the lines of the following. I would expect this to be moved into a helper function.
List<PriceDetail> list = new List<PriceDetail>
{
new PriceDetail {Id = 1, Age = 56, Name = "abc", group = "P1"},
new PriceDetail {Id = 1, Age = 56, Name = "abc", group = "P2"},
new PriceDetail {Id = 2, Age = 56, Name = "abc", group = "P1"}
};
Dictionary<PriceDetailKey, StringBuilder> group = new Dictionary<PriceDetailKey, StringBuilder>();
for (int i = 0; i < list.Count; ++i)
{
var key = new PriceDetailKey { Id = list[i].Id, Age = list[i].Age, Name = list[i].Name };
if (group.ContainsKey(key))
{
group[key].Append(",");
group[key].Append(list[i].group);
}
else
{
group[key] = new StringBuilder();
group[key].Append(list[i].group);
}
}
List<PriceDetail> retList = new List<PriceDetail>();
foreach (KeyValuePair<PriceDetailKey, StringBuilder> kvp in group)
{
retList.Add(new PriceDetail{Age = kvp.Key.Age, Id = kvp.Key.Id, Name = kvp.Key.Name, group = kvp.Value.ToString()});
}
you could even convert the final loop into a LINQ expression like:
group.Select(kvp => new PriceDetail {Age = kvp.Key.Age, Id = kvp.Key.Id, Name = kvp.Key.Name, group = kvp.Value.ToString()});
Its worth noting you could do something similar without the overhead of constructing new objects if, for example, you wrote a custom equality comparer and used a list instead of dictionary. The upside of that is that when you were finished, it would be your return value without having to do another iteration.
There are several different ways to get the results. You could even do the grouping in SQL.

How to get row value where a corresponding value changed - LINQ

Sorry for the title being a little vague, couldn't think of a good one.
I have a list of objects that holds some maximum and minimum limit values along with a timestamp.
To illustrate, my grid used to show the contents of that list could be like this (very simplified):
LimitMin | LimitMax | Start Time
1 2 08:00
1 2 08:01
1 2 08:03
2 5 08:05
2 5 08:06
2 5 08:10
Right now, I just do a select distinct, to get the distinct limits and add them to a list, like this:
var limitdistinct = printIDSPC.Select(x => new { x.LimitMin, x.LimitMax }).Distinct();
But I would like to get the timestamp as well, where the limits changed (08:05 in the example above). I cannot seem to figure out, how to accomplish this. I thought about how Distinct actually works behind the scenes, and if you could somehow get the timestamp from the select statement. Do I have to go through the entire list in a foreach loop, and compare the values to see where it changed?
Any help?

The trick here is to use GroupBy instead of Distinct. You could then either get the minimum timestamp for each limits pair:
items
.GroupBy(x => new { x.LimitMin, x.LimitMax })
.Select(x => new {
x.Key.LimitMin,
x.Key.LimitMax,
MinStartTime = x.Min(y => y.StartTime)
});
or, as GroupBy preserves the order of the original items, get the first timestamp for each:
items
.GroupBy(x => new { x.LimitMin, x.LimitMax })
.Select(x => new {
x.Key.LimitMin,
x.Key.LimitMax,
FirstStartTime = x.First().StartTime
});

Try this:-
var limitdistinct = printIDSPC.GroupBy(x => new { x.LimitMax, x.LimitMin })
.Select(x => new
{
LimitMin = x.Key.LimitMin,
LimitMax = x.Key.LimitMax,
MinTime = x.OrderBy(y => y.StartTime).First().StartTime
});
Fiddle.

One solution would be to group by min/max, then order by start time and finally select the first time value:
var list = new List<Foo>
{
new Foo { LimitMin = 1, LimitMax = 2, StartTime = TimeSpan.Parse("08:00") },
new Foo { LimitMin = 1, LimitMax = 2, StartTime = TimeSpan.Parse("08:01") },
new Foo { LimitMin = 1, LimitMax = 2, StartTime = TimeSpan.Parse("08:03") },
new Foo { LimitMin = 2, LimitMax = 5, StartTime = TimeSpan.Parse("08:05") },
new Foo { LimitMin = 2, LimitMax = 5, StartTime = TimeSpan.Parse("08:06") },
new Foo { LimitMin = 2, LimitMax = 5, StartTime = TimeSpan.Parse("08:10") },
};
var tmp = list
.GroupBy(z => new { z.LimitMin, z.LimitMax })
.Select(z =>
new
{
Time = z.OrderBy(z2 => z2.StartTime).First().StartTime,
Min = z.Key.LimitMin,
Max = z.Key.LimitMax
})
.ToList();

How to merge two lists while adding metadata indicating value source with LINQ statement?

Given, for instance:
var a = new List<int>(){ 1 , 2 , 50 };
var b = new List<int>(){ 9 , 7 , 2 };
I need to merge them together to one sorted list, while adding some data indicating their origin (a or b). An output for example would be something like:
mergedList = { {1,false},{2,false},{2,true},{7,true},{9,true},{50,false} }
(true means it comes from a).
Edit start...
mergedList =
{ {1,IsB=false},{2,IsB=false},{2,IsB=true},{7,IsB=true},{9,IsB=true},{50,IsB=false} }
...Edit end
How can I do it with LINQ, preferably in query statement form (from ... select ...) ?

Not query form, but should work.
var ans = a.Select(i => new { Value = i, FromA = true })
.Concat(b.Select(i => new { Value = i, FromA = false }))
.OrderBy(i => i.Value);

You could create an anonymous type with the additional property:
var a = new List<int>(){ 1 , 2 , 50 };
var b = new List<int>(){ 9 , 7 , 2 };
var ax = a.Select(i => new{ Num = i, FromB = false });
var bx = b.Select(i => new{ Num = i, FromB = true});
var merged = ax.Concat(bx).OrderBy(x => x.Num);
Note that Enumerable.Concat will not eliminate dulicates, but since you want to add the origin i assume that this is desired.
Output:
foreach(var x in merged)
Console.WriteLine("Num: {0} From-B? {1}", x.Num, x.FromB);
Demo

var aItems = from aa in a
select new {Value = aa, Indicator = true};
var bItems = from bb in b
select new {Value = bb, Indicator = false};
var result = aItems.Concat(bItems).OrderBy(t => t.Value);
And pure method syntax:
var aItems = a.Select(aa => new {Value = aa, Indicator = true});
var bItems = b.Select(bb => new {Value = bb, Indicator = false});
var result = aItems.Concat(bItems).OrderBy(t => t.Value);

I think you can do this with anonymous types:
var mergedList = a.Select(x => new {val = x, tag = true})
.Union(b.Select(x => new {val = x, tag = false}))
.OrderBy(x => x.val);