We Keep Coding

How to sum periods between dates in linq c# till a specific period

I have a datatable that contains three columns, I need to check when each employee ID reached two or will reach two years by subtracting Date1 from Date2 and sum the difference by a LINQ query.
If Date2 value is null that means ID is still working till now.
ID Date1 Date2
> 100 10/01/2016 09/01/2017
> 100 20/09/2017 25/05/2019
> 101 05/07/2018
I need output like below:
ID two_years
> 100 19/09/2018
> 101 04/07/2020

Using my Scan extension which is based on the APL Scan operator (like Aggregate, only it returns the intermediate results):
public static class IEnumerableExt {
// TRes seedFn(T FirstValue)
// TRes combineFn(TRes PrevResult, T CurValue)
public static IEnumerable<TRes> Scan<T, TRes>(this IEnumerable<T> src, Func<T, TRes> seedFn, Func<TRes, T, TRes> combineFn) {
using (var srce = src.GetEnumerator()) {
if (srce.MoveNext()) {
var prev = seedFn(srce.Current);
while (srce.MoveNext()) {
yield return prev;
prev = combineFn(prev, srce.Current);
}
yield return prev;
}
}
}
}
Then assuming by two years, you mean 2 * 365 days, and assuming you count the beginning and ending dates of each period as part of the total, this LINQ will find the answer:
var ans = src.Select(s => new { s.ID, s.Date1, s.Date2, Diff = (s.Date2.HasValue ? s.Date2.Value-s.Date1 : DateTime.Now.Date-s.Date1).TotalDays+1 })
.GroupBy(s => s.ID)
.Select(sg => new { ID = sg.Key, sg = sg.Scan(s => new { s.Date1, s.Date2, s.Diff, DiffAccum = s.Diff }, (res, s) => new { s.Date1, s.Date2, s.Diff, DiffAccum = res.DiffAccum + s.Diff }) })
.Select(IDsg => new { IDsg.ID, two_year_base = IDsg.sg.FirstOrDefault(s => s.DiffAccum > twoYears) ?? IDsg.sg.Last() })
.Select(s => new { s.ID, two_years = s.two_year_base.Date1.AddDays(twoYears-(s.two_year_base.DiffAccum - s.two_year_base.Diff)).Date });
If your original data is not sorted in Date1 order, or ID+Date1 order, you will need to add OrderBy to sort by Date1.
Explanation:
First we compute the days worked(?) represented by each row, using today if we don't have an ending Date2. Then group by ID and work on each group. For each ID, we compute the running sum of days worked. Next find the Date1 that proceeds the two year mark using the running sum (DiffAccum) and compute the two year date from Date1 and the remaining time needed in that period.
If it is possible a particular ID could have a lot of periods, you could use another variation of Scan, ScanUntil which short circuits evaluation based on a predicate.

Finding consecutive rows to group based on arbitrary

I need to group a list (really a datatable but for simplicity) based on 3 columns where in the first column in really a representation of a fiscal year in integer form. I need to group on all rows where:
the current fiscal year has the second column (Service) equal to 1 and
the following fiscal year is equal to 1, and any consecutive fiscal years have 1 as the Service
The end result group will then have merged all consecutive fiscal years that have 1 as the Service, by taking the first year the group started to the last year it ended while summing the Service and Earnings.
var list = new List(){
{20002001, 1 , 100.00},
{20012002, .5 , 100.00},
{20022003, 1.0 , 100.00},
{20042005, 1.0 , 50.00},
{20052006, 1.0 , 50.00
};
Should produce the results:
20002001, 1 , 100.00
20012002, 1 , 100.00
20022006, 3 , 200.00
This is not a legible example to work off of but maybe it will shed light on what I am attempting to work towards:
var test = from r in list
where r.Item("FiscalYear") + 10001 = list.SkipWhile(r2 => !r.Equals(current)).Skip(1).FirstOrDefault(r3 => r3.Item("FiscalYear")) &&
r.Item("Service") = 1 &&
list.SkipWhile(r2 => !r.Equals(current)).Skip(1).FirstOrDefault(r => r.Item("Service") = 1D)
Select New { FiscalYear = $"{r.Item("FiscalYear") % 1000}{I have NO IDEA HOW TO DETERMINE THIS PART}", Service = list.Sum(r => r.Item("FiscalYear"), Earnings = list.Sum(r => r.Item("Earnings"))
Assuming the get next would work from http://www.herlitz.nu/2011/12/01/getting-the-previous-and-next-record-from-list-using-linq/. But it does not work in my scenerio.
I have considered a group by which I could do except I would miss out on the correct counts to group the Fiscal Year by.

Using the GroupAdjacent by function described at https://stackoverflow.com/a/4682163/6137718 :
public static class LinqExtensions
{
public static IEnumerable<IEnumerable<T>> GroupAdjacentBy<T>(
this IEnumerable<T> source, Func<T, T, bool> predicate)
{
using (var e = source.GetEnumerator())
{
if (e.MoveNext())
{
var list = new List<T> { e.Current };
var pred = e.Current;
while (e.MoveNext())
{
if (predicate(pred, e.Current))
{
list.Add(e.Current);
}
else
{
yield return list;
list = new List<T> { e.Current };
}
pred = e.Current;
}
yield return list;
}
}
}
}
The result you want can be gotten like this. Note that I use string for year, but this can be changes to use int, if needed. Using the following class structure:
class Entry
{
public Entry(string year, double service, double earnings)
{
this.Year = year;
this.Service = service;
this.Earnings = earnings;
}
public string Year;
public double Service;
public double Earnings;
}
You can get the result you desire by doing something like this:
var result = list.GroupAdjacentBy((x, y) => x.Service == 1 && y.Service == 1).
Select(g => new Entry(
g.First().Year.Substring(0,4) + g.Last().Year.Substring(4,4),
g.Sum(e => e.Service),
g.Sum(e => e.Earnings)));
An example of my code can be found at https://dotnetfiddle.net/RqmYa9 .
I'm unsure as to why, in your example result, the second entry has 1 for Service instead of 0.5. If you wanted all Services to be at least one, you can do a ternary in the query when you select the sum of Service.

finding sequential patterns of objects in a list with particular properties

I have a class like this:
public class TestResults
{
public String TestName {get;set;}
public Int32 StudentID {get;set;}
public Decimal Score {get;set;}
public Date TestTaken {get;set;}
}
So some objects mike look like this:
test.TestName = "Big Important Test";
test.StudentID = 17;
test.Score = 0.75M;
test.TestTaken = "1/1/2015";
tests.add(test);
test.TestName = "Big Important Test";
test.StudentID = 12;
test.Score = 0.89M;
test.TestTaken = "1/1/2015";
tests.add(test);
test.TestName = "Sneaky Pop Quiz in Chemistry";
test.StudentID = 17;
test.Score = 0.97M;
test.TestTaken = "2/1/2015";
tests.add(test);
test.TestName = "Sneaky Pop Quiz in Chemistry";
test.StudentID = 17;
test.Score = 0.97M;
test.TestTaken = "2/1/2015";
tests.add(test);
What I'm trying to determine is something like "For every student, show me students with large jumps in their scores?" I asked a similar question a while back in the dba.stackexchange.com world and have used the LEAD function, but now I'd like to move the logic into C#.
So a concrete question I'd want to code for would be (as an example):
Show me students who've jumped from the 60 and 70 percent range to the
90 range.
I know I can write a rat's nest of loops and branching logic, but was wondering if there are any more elegant and more comprehensive ways of identifying sequences of patterns in LINQ / C# land.
I've heard people talk about F#, but have no practical experience with that. Additionally, I think the "pattern matching" I'm talking about is a bit more involved than some of the simple string-pattern-matching I keep running across.

You could use LINQ to get the answer. Here is an example of a way you could do it:
var scores = tests.GroupBy(t => t.StudentID)
.Select(g => new { StudentID = g.Key, Min = g.Min(i => i.Score), Max = g.Max(i => i.Score) })
.Where(s => s.Max - s.Min > .20M);
foreach(var score in scores)
Console.WriteLine("Student: {0} Jump: {1}", score.StudentID, score.Max - score.Min);
The LINQ statement first groups by StudentID. Next it projects the StudentID and Min and Max scores from each group to a new anonymous type. Finally, apply a where condition that only returns items with a "large jump in score". I define "large jump in score" as the difference between max score and min score is greater than .20.
Note: this code will work even when a student has more than 2 scores in the list.
UPDATE:
Since you have updated your post I understand your question better. Here is an updated answer:
var scores = tests.GroupBy(t => t.StudentID)
.Select(g => new { StudentID = g.Key, Min = g.OrderBy(i => i.Score).First(), Max = g.OrderByDescending(i => i.Score).First() })
.Where(s => (s.Min.Score >= .60M & s.Min.Score < .80M) & s.Max.Score >= .90M & s.Min.TestTaken < s.Max.TestTaken);
foreach(var score in scores)
Console.WriteLine("Student: {0} Jump: {1}", score.StudentID, score.Max.Score - score.Min.Score);
This uses a similar approach, but instead of recording the min and max scores in the anonymous type, I record the TestResults instance having the min score and max score. In ther where clause we check that the TestResults having the min score is in the 60-80 range. We check that the TestResults having the max score is in the 90+ range. Finally, we check that the min score occurred on a date before the max one occurred.

You can do something like this:
const decimal differenceLimit = 0.05M;
var studentIdsWithJump = tests.GroupBy (g => g.StudentID)
.Where(g => g.OrderBy(c => c.Score)
.GroupAdjacentBy((first, second) =>
first.Score + differenceLimit < second.Score
).Count() > 1
)
.Select(g => g.Key);
With the helper method taken from here:
public static class LinqExtensions
{
public static IEnumerable<IEnumerable<T>> GroupAdjacentBy<T>(this IEnumerable<T> source, Func<T, T, bool> predicate)
{
using (var e = source.GetEnumerator())
{
if (e.MoveNext())
{
var list = new List<T> { e.Current };
var pred = e.Current;
while (e.MoveNext())
{
if (predicate(pred, e.Current))
{
list.Add(e.Current);
}
else
{
yield return list;
list = new List<T> { e.Current };
}
pred = e.Current;
}
yield return list;
}
}
}
}
This gives you the jumps for all ranges. If you want to narrow it down, you could add a further .Where() for scores > 60, and adjust the differenceLimit accordingly

LINQ: Collapsing a series of strings into a set of "ranges"

I have an array of strings similar to this (shown on separate lines to illustrate the pattern):
{ "aa002","aa003","aa004","aa005","aa006","aa007", // note that aa008 is missing
"aa009"
"ba023","ba024","ba025"
"bb025",
"ca002","ca003",
"cb004",
...}
...and the goal is to collapse those strings into this comma-separated string of "ranges":
"aa002-aa007,aa009,ba023-ba025,bb025,ca002-ca003,cb004, ... "
I want to collapse them so I can construct a URL. There are hundreds of elements, but I can still convey all the information if I collapse them this way - putting them all into a URL "longhand" (it has to be a GET, not a POST) isn't feasible.
I've had the idea to separate them into groups using the first two characters as the key - but does anyone have any clever ideas for collapsing those sequences (without gaps) into ranges? I'm struggling with it, and everything I've come up with looks like spaghetti.

So the first thing that you need to do is parse the strings. It's important to have the alphabetic prefix and the integer value separately.
Next you want to group the items on the prefix.
For each of the items in that group, you want to order them by number, and then group items while the previous value's number is one less than the current item's number. (Or, put another way, while the previous item plus one is equal to the current item.)
Once you've grouped all of those items you want to project that group out to a value based on that range's prefix, as well as the first and last number. No other information from these groups is needed.
We then flatten the list of strings for each group into just a regular list of strings, since once we're all done there is no need to separate out ranges from different groups. This is done using SelectMany.
When that's all said and done, that, translated into code, is this:
public static IEnumerable<string> Foo(IEnumerable<string> data)
{
return data.Select(item => new
{
Prefix = item.Substring(0, 2),
Number = int.Parse(item.Substring(2))
})
.GroupBy(item => item.Prefix)
.SelectMany(group => group.OrderBy(item => item.Number)
.GroupWhile((prev, current) =>
prev.Number + 1 == current.Number)
.Select(range =>
RangeAsString(group.Key,
range.First().Number,
range.Last().Number)));
}
The GroupWhile method can be implemented like so:
public static IEnumerable<IEnumerable<T>> GroupWhile<T>(
this IEnumerable<T> source, Func<T, T, bool> predicate)
{
using (var iterator = source.GetEnumerator())
{
if (!iterator.MoveNext())
yield break;
List<T> list = new List<T>() { iterator.Current };
T previous = iterator.Current;
while (iterator.MoveNext())
{
if (!predicate(previous, iterator.Current))
{
yield return list;
list = new List<T>();
}
list.Add(iterator.Current);
previous = iterator.Current;
}
yield return list;
}
}
And then the simple helper method to convert each range into a string:
private static string RangeAsString(string prefix, int start, int end)
{
if (start == end)
return prefix + start;
else
return string.Format("{0}{1}-{0}{2}", prefix, start, end);
}

Here's a LINQ version without the need to add new extension methods:
var data2 = data.Skip(1).Zip(data, (d1, d0) => new
{
value = d1,
jump = d1.Substring(0, 2) == d0.Substring(0, 2)
? int.Parse(d1.Substring(2)) - int.Parse(d0.Substring(2))
: -1,
});
var agg = new { f = data.First(), t = data.First(), };
var query2 =
data2
.Aggregate(new [] { agg }.ToList(), (a, x) =>
{
var last = a.Last();
if (x.jump == 1)
{
a.RemoveAt(a.Count() - 1);
a.Add(new { f = last.f, t = x.value, });
}
else
{
a.Add(new { f = x.value, t = x.value, });
}
return a;
});
var query3 =
from q in query2
select (q.f) + (q.f == q.t ? "" : "-" + q.t);
I get these results:

Use LINQ to group a sequence by date with no gaps

I'm trying to select a subgroup of a list where items have contiguous dates, e.g.
ID StaffID Title ActivityDate
-- ------- ----------------- ------------
1 41 Meeting with John 03/06/2010
2 41 Meeting with John 08/06/2010
3 41 Meeting Continues 09/06/2010
4 41 Meeting Continues 10/06/2010
5 41 Meeting with Kay 14/06/2010
6 41 Meeting Continues 15/06/2010
I'm using a pivot point each time, so take the example pivot item as 3, I'd like to get the following resulting contiguous events around the pivot:
ID StaffID Title ActivityDate
-- ------- ----------------- ------------
2 41 Meeting with John 08/06/2010
3 41 Meeting Continues 09/06/2010
4 41 Meeting Continues 10/06/2010
My current implementation is a laborious "walk" into the past, then into the future, to build the list:
var activity = // item number 3: Meeting Continues (09/06/2010)
var orderedEvents = activities.OrderBy(a => a.ActivityDate).ToArray();
// Walk into the past until a gap is found
var preceedingEvents = orderedEvents.TakeWhile(a => a.ID != activity.ID);
DateTime dayBefore;
var previousEvent = activity;
while (previousEvent != null)
{
dayBefore = previousEvent.ActivityDate.AddDays(-1).Date;
previousEvent = preceedingEvents.TakeWhile(a => a.ID != previousEvent.ID).LastOrDefault();
if (previousEvent != null)
{
if (previousEvent.ActivityDate.Date == dayBefore)
relatedActivities.Insert(0, previousEvent);
else
previousEvent = null;
}
}
// Walk into the future until a gap is found
var followingEvents = orderedEvents.SkipWhile(a => a.ID != activity.ID);
DateTime dayAfter;
var nextEvent = activity;
while (nextEvent != null)
{
dayAfter = nextEvent.ActivityDate.AddDays(1).Date;
nextEvent = followingEvents.SkipWhile(a => a.ID != nextEvent.ID).Skip(1).FirstOrDefault();
if (nextEvent != null)
{
if (nextEvent.ActivityDate.Date == dayAfter)
relatedActivities.Add(nextEvent);
else
nextEvent = null;
}
}
The list relatedActivities should then contain the contiguous events, in order.
Is there a better way (maybe using LINQ) for this?
I had an idea of using .Aggregate() but couldn't think how to get the aggregate to break out when it finds a gap in the sequence.

Here's an implementation:
public static IEnumerable<IGrouping<int, T>> GroupByContiguous(
this IEnumerable<T> source,
Func<T, int> keySelector
)
{
int keyGroup = Int32.MinValue;
int currentGroupValue = Int32.MinValue;
return source
.Select(t => new {obj = t, key = keySelector(t))
.OrderBy(x => x.key)
.GroupBy(x => {
if (currentGroupValue + 1 < x.key)
{
keyGroup = x.key;
}
currentGroupValue = x.key;
return keyGroup;
}, x => x.obj);
}
You can either convert the dates to ints by means of subtraction, or imagine a DateTime version (easily).

In this case I think that a standard foreach loop is probably more readable than a LINQ query:
var relatedActivities = new List<TActivity>();
bool found = false;
foreach (var item in activities.OrderBy(a => a.ActivityDate))
{
int count = relatedActivities.Count;
if ((count > 0) && (relatedActivities[count - 1].ActivityDate.Date.AddDays(1) != item.ActivityDate.Date))
{
if (found)
break;
relatedActivities.Clear();
}
relatedActivities.Add(item);
if (item.ID == activity.ID)
found = true;
}
if (!found)
relatedActivities.Clear();
For what it's worth, here's a roughly equivalent -- and far less readable -- LINQ query:
var relatedActivities = activities
.OrderBy(x => x.ActivityDate)
.Aggregate
(
new { List = new List<TActivity>(), Found = false, ShortCircuit = false },
(a, x) =>
{
if (a.ShortCircuit)
return a;
int count = a.List.Count;
if ((count > 0) && (a.List[count - 1].ActivityDate.Date.AddDays(1) != x.ActivityDate.Date))
{
if (a.Found)
return new { a.List, a.Found, ShortCircuit = true };
a.List.Clear();
}
a.List.Add(x);
return new { a.List, Found = a.Found || (x.ID == activity.ID), a.ShortCircuit };
},
a => a.Found ? a.List : new List<TActivity>()
);

Somehow, I don't think LINQ was truly meant to be used for bidirectional-one-dimensional-depth-first-searches, but I constructed a working LINQ using Aggregate. For this example I'm going to use a List instead of an array. Also, I'm going to use Activity to refer to whatever class you are storing the data in. Replace it with whatever is appropriate for your code.
Before we even start, we need a small function to handle something. List.Add(T) returns null, but we want to be able to accumulate in a list and return the new list for this aggregate function. So all you need is a simple function like the following.
private List<T> ListWithAdd<T>(List<T> src, T obj)
{
src.Add(obj);
return src;
}
First, we get the sorted list of all activities, and then initialize the list of related activities. This initial list will contain the target activity only, to start.
List<Activity> orderedEvents = activities.OrderBy(a => a.ActivityDate).ToList();
List<Activity> relatedActivities = new List<Activity>();
relatedActivities.Add(activity);
We have to break this into two lists, the past and the future just like you currently do it.
We'll start with the past, the construction should look mostly familiar. Then we'll aggregate all of it into relatedActivities. This uses the ListWithAdd function we wrote earlier. You could condense it into one line and skip declaring previousEvents as its own variable, but I kept it separate for this example.
var previousEvents = orderedEvents.TakeWhile(a => a.ID != activity.ID).Reverse();
relatedActivities = previousEvents.Aggregate<Activity, List<Activity>>(relatedActivities, (items, prevItem) => items.OrderBy(a => a.ActivityDate).First().ActivityDate.Subtract(prevItem.ActivityDate).Days.Equals(1) ? ListWithAdd(items, prevItem) : items).ToList();
Next, we'll build the following events in a similar fashion, and likewise aggregate it.
var nextEvents = orderedEvents.SkipWhile(a => a.ID != activity.ID);
relatedActivities = nextEvents.Aggregate<Activity, List<Activity>>(relatedActivities, (items, nextItem) => nextItem.ActivityDate.Subtract(items.OrderBy(a => a.ActivityDate).Last().ActivityDate).Days.Equals(1) ? ListWithAdd(items, nextItem) : items).ToList();
You can properly sort the result afterwards, as now relatedActivities should contain all activities with no gaps. It won't immediately break when it hits the first gap, no, but I don't think you can literally break out of a LINQ. So it instead just ignores anything which it finds past a gap.
Note that this example code only operates on the actual difference in time. Your example output seems to imply that you need some other comparison factors, but this should be enough to get you started. Just add the necessary logic to the date subtraction comparison in both entries.