We Keep Coding

Converting linq query to lambda syntax

I want to show heatmap when message sent by day and hour and I wrote some linq query.
Original SQL Query
SELECT weekday, hr , count(message.hour)
FROM (SELECT days.dow as weekday, hours.hour as hr from days left join hours,message on message.dow = days.dow group by days.dow, hours.hour)
LEFT JOIN message on message.dow = weekday and message.hour = hr group by weekday, hr
Message class
public class Message
{
public long Id { get; set; }
public string Text { get; set; }
public DateTime Timestamp { get; set; }
}
LINQ query(it works)
int[] Hours = { 0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23 };
int[] Days = { 0, 1, 2, 3, 4, 5, 6 };
List<Message> messages = new List<Message>();
/*
Load Message data from csv file
...
*/
var heatmap = from d in
(from message in Messages
join h in Hours on message.Timestamp.Hour equals h
join d in Days on (int)message.Timestamp.DayOfWeek equals d
orderby d, h
group new { d, h } by new { d, h } into g
select new { Weekday = g.Key.d, Hour = g.Key.h })
join message in Messages on new { d.Weekday, d.Hour } equals new { Weekday = (int)message.Timestamp.DayOfWeek, Hour = message.Timestamp.Hour }
group new { d.Weekday, d.Hour } by new { d.Weekday, d.Hour } into g
select new { Weekday = g.Key.Weekday, Hour = g.Key.Hour, Total = g.Count() };
Sample result heatmap data
Weekday Hour Total
[0] 0 0 120
[1] 0 1 57
[2] 0 2 79
...
[167] 6 23 89
and here is the code that I converted LINQ query to lambda syntax code but it doesn't work! :(
int[] Hours = { 0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23 };
int[] Days = { 0, 1, 2, 3, 4, 5, 6 };
List<Message> messages = new List<Message>();
/*
Load Message data from csv file
...
*/
var heatmapLambda = (Messages.Join(Hours, message => message.Timestamp.Hour, h => h, (m, h) => new { m, h })
.Join(Days, m => (int)m.m.Timestamp.DayOfWeek, d => d, (m,d) => new { m, d })
.OrderBy(m => new {m.d, m.m.h})
.GroupBy(g => new {g.d, g.m.h})
.Select(r => new {
Weekday = r.Key.d,
Hour = r.Key.h
}))
.Join(Messages, d => new {d.Weekday, d.Hour}, m => new { Weekday = (int)m.Timestamp.DayOfWeek, Hour = m.Timestamp.Hour}, (d,h) => new { d.Weekday, d.Hour})
.GroupBy(g => new { g.Weekday, g.Hour})
.Select(r => new { Weekday = r.Key.Weekday, Hour = r.Key.Hour, Count = r.Count() });
heatmapLambda has no any items. what is right lambda expression of my linq query?

It seems to me that you're doing quite a few unnecessary joins.
When you do:
from message in Messages
join h in Hours on message.Timestamp.Hour equals h
join d in Days on (int)message.Timestamp.DayOfWeek equals d
...it is the same as:
from message in Messages
let h = message.Timestamp.Hour
let d = (int)message.Timestamp.DayOfWeek
And then when you do:
join message in Messages on new { d.Weekday, d.Hour } equals new { Weekday = (int)message.Timestamp.DayOfWeek, Hour = message.Timestamp.Hour }
...you're basically rejoining back to your original messages.
I think that this is the equivalent query:
var heatmap =
from message in Messages
let Hour = message.Timestamp.Hour
let Weekday = (int)message.Timestamp.DayOfWeek
orderby Weekday, Hour
group message by new { Weekday, Hour } into gms
select new { gms.Key.Weekday, gms.Key.Hour, Count = gms.Count() };
...so, if that's the case, then this is what you need using lambdas:
var heatmap =
Messages
.OrderBy(message => (int)message.Timestamp.DayOfWeek)
.ThenBy(message => message.Timestamp.Hour)
.GroupBy(message => new { Weekday = (int)message.Timestamp.DayOfWeek, message.Timestamp.Hour })
.Select(gms => new { gms.Key.Weekday, gms.Key.Hour, Count = gms.Count() });

IMO the primary goal of a query should be to get a correct result in a best way, rather than the syntax used (query, method or mixed).
In your case, the query is far from optimal - unnecessary joins, grouping and ordering. From what I see, you are trying to get the count of the messages per each (weekday, hour) combination from the Days and Hours collections, which can be achieved by simple Cartesian product combined with group join like this
var query =
from key in (from d in Days from h in Hours select new { Weekday = d, Hour = h })
join m in Messages
on key equals new { Weekday = (int)m.Timstamp.DayOfWeek, m.Timestamp.Hour } into items
select new { key.Weekday, key.Hour, Total = items.Count() };
In general when working with joins and or groupings, using the query syntax is easier and more natural due to transparent identifiers. In this particular case method syntax is not so different, but you need to know how to map the constructs:
var query =
Days.SelectMany(d => Hours, (d, h) => new { Weekday = d, Hour = h })
.GroupJoin(Messages,
key => key, m => new { Weekday = (int)m.Timstamp.DayOfWeek, m.Timestamp.Hour },
(key, items) => new { key.Weekday, key.Hour, Total = items.Count() });

How to get row value where a corresponding value changed - LINQ

Sorry for the title being a little vague, couldn't think of a good one.
I have a list of objects that holds some maximum and minimum limit values along with a timestamp.
To illustrate, my grid used to show the contents of that list could be like this (very simplified):
LimitMin | LimitMax | Start Time
1 2 08:00
1 2 08:01
1 2 08:03
2 5 08:05
2 5 08:06
2 5 08:10
Right now, I just do a select distinct, to get the distinct limits and add them to a list, like this:
var limitdistinct = printIDSPC.Select(x => new { x.LimitMin, x.LimitMax }).Distinct();
But I would like to get the timestamp as well, where the limits changed (08:05 in the example above). I cannot seem to figure out, how to accomplish this. I thought about how Distinct actually works behind the scenes, and if you could somehow get the timestamp from the select statement. Do I have to go through the entire list in a foreach loop, and compare the values to see where it changed?
Any help?

The trick here is to use GroupBy instead of Distinct. You could then either get the minimum timestamp for each limits pair:
items
.GroupBy(x => new { x.LimitMin, x.LimitMax })
.Select(x => new {
x.Key.LimitMin,
x.Key.LimitMax,
MinStartTime = x.Min(y => y.StartTime)
});
or, as GroupBy preserves the order of the original items, get the first timestamp for each:
items
.GroupBy(x => new { x.LimitMin, x.LimitMax })
.Select(x => new {
x.Key.LimitMin,
x.Key.LimitMax,
FirstStartTime = x.First().StartTime
});

Try this:-
var limitdistinct = printIDSPC.GroupBy(x => new { x.LimitMax, x.LimitMin })
.Select(x => new
{
LimitMin = x.Key.LimitMin,
LimitMax = x.Key.LimitMax,
MinTime = x.OrderBy(y => y.StartTime).First().StartTime
});
Fiddle.

One solution would be to group by min/max, then order by start time and finally select the first time value:
var list = new List<Foo>
{
new Foo { LimitMin = 1, LimitMax = 2, StartTime = TimeSpan.Parse("08:00") },
new Foo { LimitMin = 1, LimitMax = 2, StartTime = TimeSpan.Parse("08:01") },
new Foo { LimitMin = 1, LimitMax = 2, StartTime = TimeSpan.Parse("08:03") },
new Foo { LimitMin = 2, LimitMax = 5, StartTime = TimeSpan.Parse("08:05") },
new Foo { LimitMin = 2, LimitMax = 5, StartTime = TimeSpan.Parse("08:06") },
new Foo { LimitMin = 2, LimitMax = 5, StartTime = TimeSpan.Parse("08:10") },
};
var tmp = list
.GroupBy(z => new { z.LimitMin, z.LimitMax })
.Select(z =>
new
{
Time = z.OrderBy(z2 => z2.StartTime).First().StartTime,
Min = z.Key.LimitMin,
Max = z.Key.LimitMax
})
.ToList();

How can I do a "one hit" Linq query with nullable values?

I have the following class:
public class MyTrelloCard
{
...
public DateTime? completed { get; set; }
public Decimal? estHours { get; set; }
public bool complete { get; set; }
}
...and I want to query the number of estHours on each date in the completed date. These have to be nullable values as not every card is completed, and not every card has estimated hours added to it.
At the moment I have to run two queries. The first one:
List<burnDownData> bData = (from c in cards
where c.complete
group c by new
{
date = Convert.ToDateTime(c.completed).Date
} into g
select new burnDownData
{
date = g.Key.date,
completedHours = g.Sum(x=>x.estHours) ?? 0
}).ToList();
returns all the cards that have been completed.
The second one is used to iterate through all dates between a start and end period and cumulatively add up the completed hours:
for (DateTime d = start; d.Date <= end; d = d.AddDays(1))
{
if ((d.DayOfWeek >= DayOfWeek.Monday) && (d.DayOfWeek <= DayOfWeek.Friday))
{
List<C> values = new List<C>();
Decimal xx = (from b in bData
where b.date.Date == d
select b.completedHours).SingleOrDefault();
total = total + xx;
values.Add(new C { v = d.ToString("d MMM") });
values.Add(new C { v = total.ToString() });
myRows.Add(new Row { c = values });
}
}
This seems inefficient though. Is it possible to do this directly? i.e. to replace this part of the loop:
Decimal xx = (from b in bData
where b.date.Date == d
select b.completedHours).SingleOrDefault();
With something that queries the cards data directly?

Interpreting your question:
I want to query the number of estHours on each date in the completed
date. These have to be nullable values as not every card is completed,
and not every card has estimated hours added to it
...as "I'm trying to get a sum of estimated hours for each completed date instance, and want to allow for null completed and estHours values".
You could include the check for null in your Where clause:
var x = cards.Where(c => c.completed != null )
.GroupBy(c => c.completed, (key, group) => new
{
dateComp = key.Value,
totEstHrs = group.Sum(i => i.estHours)
});
x.ToList().ForEach(
item => Debug.Print("{0:MM/dd/yyyy} {1}", item.dateComp, item.totEstHrs));
For example:
List<MyTrelloCard> cards = new List<MyTrelloCard>() {
new MyTrelloCard() { completed = new DateTime(2014, 4, 28), estHours = 5, complete = true },
new MyTrelloCard() { completed = null, estHours = 5, complete = false },
new MyTrelloCard() { completed = null, estHours = null, complete = true },
new MyTrelloCard() { completed = new DateTime(2014, 4, 28), estHours = 7, complete = false },
new MyTrelloCard() { completed = new DateTime(2014, 4, 29), estHours = null, complete = false },
new MyTrelloCard() { completed = new DateTime(2014, 4, 29), estHours = 3, complete = false },
};
Produces:
04/28/2014 12
04/29/2014 3
Edit:
To group on just the date portion (omitting the timestamp) you can group on the ToShortDateString() value:
var x = cards.Where(c => c.completed != null)
.GroupBy(c =>
((DateTime)c.completed).ToShortDateString(),
(key, group) => new
{
dateComp = key,
totEstHrs = group.Sum(i => i.estHours)
});

Ah - so you want an entry for every weekday in your range, even if no data exists for that day. In which case, build an array of eligible days, and groupjoin your cards.
Here's how:
var start = DateTime.Today;
var end = start.AddDays(14);
var cards = new[]{new {complete = true, completed = (DateTime?)DateTime.Now, estHours = new decimal?(3)} };
var days = Enumerable.Range(0, end.Subtract(start).Days)
.Select(x => start.AddDays(x))
.Where(x => !new []{DayOfWeek.Saturday,DayOfWeek.Sunday}.Contains(x.DayOfWeek));
var results = from d in days
join c in cards on d equals c.completed.GetValueOrDefault().Date
into cGrp
select new {d, completedHours = cGrp.Sum(x => x.estHours)};
of course you don't need the var cards = line, as you already have your own source for that!
Edit - All timestamps for a day are grouped into the 1 day.

Group by date range , count and sort within each group LINQ

I have a collection of dates stored in my object. This is sample data. In real time, the dates will come from a service call and I will have no idea what dates and how many will be returned:
var ListHeader = new List<ListHeaderData>
{
new ListHeaderData
{
EntryDate = new DateTime(2013, 8, 26)
},
new ListHeaderData
{
EntryDate = new DateTime(2013, 9, 11)
},
new ListHeaderData
{
EntryDate = new DateTime(2013, 1, 1)
},
new ListHeaderData
{
EntryDate = new DateTime(2013, 9, 15)
},
new ListHeaderData
{
EntryDate = new DateTime(2013, 9, 17)
},
new ListHeaderData
{
EntryDate = new DateTime(2013, 9, 5)
},
};
I now need to group by date range like so:
Today (1) <- contains the date 9/17/2013 and count of 1
within 2 weeks (3) <- contains dates 9/15,9/11,9/5 and count of 3
More than 2 weeks (2) <- contains dates 8/26, 1/1 and count of 2
this is my LINQ statement which doesn't achieve what I need but i think i'm in the ballpark (be kind if I'm not):
var defaultGroups = from l in ListHeader
group l by l.EntryDate into g
orderby g.Min(x => x.EntryDate)
select new { GroupBy = g };
This groups by individual dates, so I have 6 groups with 1 date in each. How do I group by date range , count and sort within each group?

Introduce array, which contains ranges you want to group by. Here is two ranges - today (zero days) and 14 days (two weeks):
var today = DateTime.Today;
var ranges = new List<int?> { 0, 14 };
Now group your items by range it falls into. If there is no appropriate range (all dates more than two weeks) then default null range value will be used:
var defaultGroups =
from h in ListHeader
let daysFromToday = (int)(today - h.EntryDate).TotalDays
group h by ranges.FirstOrDefault(range => daysFromToday <= range) into g
orderby g.Min(x => x.EntryDate)
select g;
UPDATE: Adding custom ranges for grouping:
var ranges = new List<int?>();
ranges.Add(0); // today
ranges.Add(7*2); // two weeks
ranges.Add(DateTime.Today.Day); // within current month
ranges.Add(DateTime.Today.DayOfYear); // within current year
ranges.Sort();

How about doing this?
Introduce a new property for grouping and group by that.
class ListHeaderData
{
public DateTime EntryDate;
public int DateDifferenceFromToday
{
get
{
TimeSpan difference = DateTime.Today - EntryDate.Date;
if (difference.TotalDays == 0)//today
{
return 1;
}
else if (difference.TotalDays <= 14)//less than 2 weeks
{
return 2;
}
else
{
return 3;//something else
}
}
}
}
Edit: as #servy pointed in comments other developers may confuse of int using a enum will be more readable.
So, modified version of your class would look something like this
class ListHeaderData
{
public DateTime EntryDate;
public DateRange DateDifferenceFromToday
{
get
{
//I think for this version no comments needed names are self explanatory
TimeSpan difference = DateTime.Today - EntryDate.Date;
if (difference.TotalDays == 0)
{
return DateRange.Today;
}
else if (difference.TotalDays <= 14)
{
return DateRange.LessThanTwoWeeks;
}
else
{
return DateRange.MoreThanTwoWeeks;
}
}
}
}
enum DateRange
{
None = 0,
Today = 1,
LessThanTwoWeeks = 2,
MoreThanTwoWeeks = 3
}
and use it like this
var defaultGroups = from l in ListHeader
group l by l.DateDifferenceFromToday into g // <--Note group by DateDifferenceFromToday
orderby g.Min(x => x.EntryDate)
select new { GroupBy = g };

Do you specifically want to achieve the solution in this way? Also do you really want to introduce spurious properties into your class to meet these requirements?
These three lines would achieve your requirements and for large collections willbe more performant.
var todays = listHeader.Where(item => item.EntryDate == DateTime.Today);
var twoWeeks = listHeader.Where(item => item.EntryDate < DateTime.Today.AddDays(-1)
&& item.EntryDate >= DateTime.Today.AddDays(-14));
var later = listHeader.Where(item => item.EntryDate < DateTime.Today.AddDays(-14));
also you then get the flexibility of different groupings without impacting your class.
[Edit: in response to ordering query]
Making use of the Enum supplied above you can apply the Union clause and OrderBy clause Linq extension methods as follows:
var ord = todays.Select(item => new {Group = DateRange.Today, item.EntryDate})
.Union(
twoWeeks.Select(item => new {Group = DateRange.LessThanTwoWeeks, item.EntryDate}))
.Union(
later.Select(item => new {Group = DateRange.MoreThanTwoWeeks, item.EntryDate}))
.OrderBy(item => item.Group);
Note that I'm adding the Grouping via a Linq Select and anonymous class to dynamically push a Group property again not effecting the original class. This produces the following output based on the original post:
Group EntryDate
Today 17/09/2013 00:00:00
LessThanTwoWeeks 11/09/2013 00:00:00
LessThanTwoWeeks 15/09/2013 00:00:00
LessThanTwoWeeks 05/09/2013 00:00:00
MoreThanTwoWeeks 26/08/2013 00:00:00
MoreThanTwoWeeks 01/01/2013 00:00:00
and to get grouped date ranges with count:
var ord = todays.Select(item => new {Group = DateRange.Today, Count=todays.Count()})
.Union(
twoWeeks.Select(item => new {Group = DateRange.LessThanTwoWeeks, Count=twoWeeks.Count()}))
.Union(
later.Select(item => new {Group = DateRange.MoreThanTwoWeeks, Count=later.Count()}))
.OrderBy(item => item.Group);
Output is:
Group Count
Today 1
LessThanTwoWeeks 3
MoreThanTwoWeeks 2

I suppose this depends on how heavily you plan on using this. I had/have a lot of reports to generate so I created a model IncrementDateRange with StartTime, EndTime and TimeIncrement as an enum.
The time increment handler has a lot of switch based functions spits out a list of times between the Start and End range based on hour/day/week/month/quarter/year etc.
Then you get your list of IncrementDateRange and in linq something like either:
TotalsList = times.Select(t => new RetailSalesTotalsListItem()
{
IncrementDateRange = t,
Total = storeSales.Where(s => s.DatePlaced >= t.StartTime && s.DatePlaced <= t.EndTime).Sum(s => s.Subtotal),
})
or
TotalsList = storeSales.GroupBy(g => g.IncrementDateRange.StartTime).Select(gg => new RetailSalesTotalsListItem()
{
IncrementDateRange = times.First(t => t.StartTime == gg.Key),
Total = gg.Sum(rs => rs.Subtotal),
}).ToList(),

Looking for a more elegant LINQ solution

I am trying to figure out an efficient way to retrieve the data I am after. I need to get a list of all of the most recent children by ParentId coupled with all parent entries that do NOT have children. I have created a visual guide to illustrate what the response should be.
The query needs to remain as IQueryable until ALL sorting and paging is completed.
Last and LastOrDefault are not supported by LINQ to Entities (as stated by the error messages I have received while using them).
Using First or FirstOrDefault will return the error "This method or operation is not implemented"
Original Data:
-------------------------------
- Id - ParentId - CreatedDate -
-------------------------------
- 1 - - 07/01/2013 -
- 2 - - 07/01/2013 -
- 3 - - 07/01/2013 -
- 4 - 1 - 07/02/2013 -
- 5 - 2 - 07/03/2013 -
- 6 - 2 - 07/04/2013 -
- 7 - 1 - 07/05/2013 -
-------------------------------
Data returned by query
-------------------------------
- Id - ParentId - CreatedDate -
-------------------------------
- 3 - - 07/01/2013 -
- 6 - 2 - 07/04/2013 -
- 7 - 1 - 07/05/2013 -
-------------------------------
Currently, my LINQ query looks like this:
// Retrieves parent records with NO children.
var q1 = myTable
.Where(x => x.ParentId == null)
.Except(myTable
.Where(x => myTable
.Any(c => (c.ParentId == x.Id))));
// Retrieves most recent child records for each parentId
var q2 =
(from a in myTable
join b in
(myTable.Where(a => a.ParentId != null)
.GroupBy(a => a.ParentId)
.Select(b => new { ParentId = b.Key, CreatedDate = b.Max(t => t.CreatedDate) }))
on a.ParentId equals b.ParentId
where a.CreatedDate == b.CreatedDate
select a);
q1 = q1.Union(q2);
The back-end is using Npgsql2 with PostgreSQL. I am looking for a more elegant solution for this query. I am very new to LINQ and would like to optimize this.
Sorting code (sloppy, but jTable returns these strings):
if (string.IsNullOrEmpty(sorting) || sorting.Equals("Name ASC")) {
q1 = q1.OrderBy(p => p.Customer.Name);
} else if (sorting.Equals("Name DESC")) {
q1 = q1.OrderByDescending(p => p.Customer.Name);
} else if (sorting.Equals("Date ASC")) {
q1 = q1.OrderBy(p => p.CreatedDate);
} else if (sorting.Equals("Date DESC")) {
q1 = q1.OrderByDescending(p => p.CreatedDate);
}
Paging code:
var result = pageSize > 0
? q1.Skip(startIndex).Take(pageSize).ToList()
: q1.ToList();

Use grouping:
Mock data:
public class Entry {
public int Id { get; set; }
public int? ParentId { get; set; }
public DateTime Date { get; set; }
};
var list = new List<Entry> {
new Entry{ Id = 1, ParentId = null, Date = new DateTime(2013, 7, 1) },
new Entry{ Id = 2, ParentId = null, Date = new DateTime(2013, 7, 1) },
new Entry{ Id = 3, ParentId = null, Date = new DateTime(2013, 7, 1) },
new Entry{ Id = 4, ParentId = 1, Date = new DateTime(2013, 7, 2) },
new Entry{ Id = 5, ParentId = 2, Date = new DateTime(2013, 7, 3) },
new Entry{ Id = 6, ParentId = 2, Date = new DateTime(2013, 7, 4) },
new Entry{ Id = 7, ParentId = 1, Date = new DateTime(2013, 7, 5) }
};
Query:
var query = from l in list
group l by l.ParentId into g
select new {
Items = g.OrderBy(x => x.Date).Last()
};
var res = query.OrderBy(x => x.Items.Id).Select(x => x.Items).ToList();
LinqPad result:
Id ParentId Date
3 null 01.07.2013 0:00:00
6 2 04.07.2013 0:00:00
7 1 05.07.2013 0:00:00

I can propose a different query, still in two phases
var firstQuery = myTable.Select(p => new { p.ID, ParentID = p.ParentID ?? p.ID, p.CreatedDate })
.GroupBy( p => p.ParentID).Select( q => new
{
el = q.OrderByDescending( k => k.CreatedDate).Take(1)
}).SelectMany(t => t.el);
var result = dc.TabellaId_ParentId.Where(p => test.Select(q => q.ID).Contains(p.ID));