We Keep Coding

How to group dates by week and weekday with linq?

I have data from which I should count rows by weeks and weekdays. As result I should get
Starting day of the week, weekday, count of data for that day
I have tried this code:
var GroupedByDate = from r in dsDta.Tables[0].Rows.Cast<DataRow>()
let eventTime = (DateTime)r["EntryTime"]
group r by new
{
WeekStart = DateTime(eventTime.Year, eventTime.Month, eventTime.AddDays(-(int)eventTime.DayOfWeek).Day),
WeekDay = eventTime.DayOfWeek
}
into g
select new
{
g.Key,
g.WeekStart,
g.WeekDay,
LoadCount = g.Count()
};
However, from DateTime(eventTime.Year, ...)
I get an error "C# non-invocable member datetime cannot be used like a method."
What to do differently?

The immediate error is due to you missing the new part from your constructor call. However, even with that, you'd still have a problem as you're using the month and year of the existing date, even if the start of the week is in a previous month or year.
Fortunately, you can simplify it very easily:
group r by new
{
WeekStart = eventTime.AddDays(-(int)eventTime.DayOfWeek)),
WeekDay = eventTime.DayOfWeek
}
Or if eventTime isn't always a date, use eventTime.Date.AddDays(...).
Alternatively, for clarity, you could extract a separate method:
group r by new
{
WeekStart = GetStartOfWeek(eventTime)
WeekDay = eventTime.DayOfWeek
}
...
private static DateTime GetStartOfWeek(DateTime date)
{
// Whatever implementation you want
}
That way you can test the implementation of GetStartOfWeek separately, and also make it more complicated if you need to without it impacting your query.

c# LINQ to dataset IN clause contains, group, min

I am new to LINQ but am trying to tackle a tough one right off the bat. I am trying to do LINQ to dataset and emulate the following query...
SELECT smID, MIN(entID) FROM table
WHERE exID = :exID
AND smID IN (1,2,3,4,5,6,7,8, etc)
GROUP BY smID
The code I have so far is as follows...
DataTable dt = ds.Tables["myTable"];
var query =
from g in dt.AsEnumerable()
where g.Field<string>("exID") == exID
&& smIDs.Contains(g.Field<string>("smID"))
group g by g.Field<string>("smID") into rowGroup
select new
{
smID = rowGroup.Key,
minEntID = rowGroup.Min(g => g.Field<int>("entID"))
};
exID is a string variable in the method and smIDs is a List of strings also created earlier in the method. I created the following code to try and see my results and it throws an "System.InvalidCastException" error at query.Count...
if (query.Count() > 0)
{
foreach (var item in query)
{
string s = item.smID;
int i = (int)item.minEntID;
}
}
I have been unable to figure out what I am doing wrong.
VS points to...
minEntID = rowGroup.Min(g => g.Field<int>("entID"))
This is the first two lines of the stack trace...
at System.Data.DataRowExtensions.UnboxT`1.ValueField(Object value)
at System.Data.DataRowExtensions.Field[T](DataRow row, String columnName)
Any pointers would be most appreciated. Thanks.

Judging by the exception and stack trace, the type you're specifying for the endID field in your query doesn't match the DataType for that column in the DataTable. These must match -- you cannot use the Field method to cast the value to a different type.

I used Linqer to come up with this code:
from t in db.Table // your C# table / collection here, of course
where t.ExId == stackoverflow.ExId
&& (new int[] {1, 2, 3 }).Contains(t.SmId)
group t by new { t.SmId } into g
select new {
SmId = g.Key.SmId,
minEntID = g.Min(p => p.EntId)
}

linq - fill out sparse table for plot

I have a table with an event id pk, date column and an event type column.
I want to use linq to get the count of events in each day.
The issue is that the table is sparse, i.e. values are not stored in days which did not have any events.
Since I want to use this data for a line chart, I need to fill out the data with the missing dates and give them a value of zero.
Is these any way to do this inside linq? or do I have to do this manually?
Is there any recommended method of doing this?
Edit:
I created the following method:
public string GetDailyData(int month, int year)
{
int days = DateTime.DaysInMonth(year,month);
DateTime firstOfTheMonth = new DateTime(year, month, 1);
PaymentModelDataContext db = new PaymentModelDataContext();
var q = from daynumber in Enumerable.Range(0, days)
let day = firstOfTheMonth.AddDays(daynumber)
join data in db.TrackingEvents on day equals data.timestamp.Day into d2
from x in d2.DefaultIfEmpty()
select Tuple.Create(x.Key, x.Value);
return ParseJson(q);
}
The problem is I get an error on the 'join' keyword:
"The type of one of the expressions in the join clause is incorrect. Type inference failed in the call to 'GroupJoin'"
Edit 2:
I made the changes suggested and tried to group the results.
When I send them to the parsing function, I get a null object ref error.
Here is the new code:
[WebMethod]
public string GetDailyData(int month, int year)
{
int days = DateTime.DaysInMonth(year, month);
DateTime firstOfTheMonth = new DateTime(year, month, 1);
PaymentModelDataContext db = new PaymentModelDataContext();
var q = from daynumber in Enumerable.Range(0, days)
let day = firstOfTheMonth.AddDays(daynumber)
join data in db.TrackingEvents on day equals data.timestamp.Date into d2
from x in d2.DefaultIfEmpty()
group x by x.timestamp.Date;
return ParseJson(q);
}
And the parsing function:
private string ParseJson<TKey, TValue>(IEnumerable<IGrouping<TKey, TValue>> q)
{
string returnJSON = "[{ \"type\" : \"pie\", \"name\" : \"Campaigns\", \"data\" : [ ";
foreach (var grp in q)
{
double currCount = grp.Count();
if (grp.Key != null)
returnJSON += "['" + grp.Key + "', " + currCount + "],";
else
returnJSON += "['none', " + currCount + "],";
}
returnJSON = returnJSON.Substring(0, returnJSON.Length - 1);
returnJSON += "]}]";
return returnJSON;
}

You should be able to use LINQ. One method is to use Enumerable.Range to create a collection of dates between the min and max dates, and then perform an outer join (using GroupJoin) against the sparse table. (See MSDN Reference: How to Perform Outer Joins (C# Programming Guide))
For instance, if numdays is the date range (in days), MinDate is the initial date, and SparseData is your sparse data, and SparseData has an instance property Day that specifies the date, then you might do:
var q = Enumerable.Range(0, numdays)
.Select(a => MinDate.AddDays(a))
.GroupJoin(
SparseData,
q=>q,
sd=>sd.Day,
(key, value) =>
Tuple.Create(
key,
value.DefaultIfEmpty().First()
)
);
Or, equivalently,
var q2 = from daynumber in Enumerable.Range(0, numdays)
let day = MinDate.AddDays(daynumber)
join data in SparseData on day equals data.Day into d2
from x in d2.DefaultIfEmpty()
select Tuple.Create(x.Key, x.Value);

The code I've written follows an almost identical approach to that suggested in #drf's answer - outer joining the aggregated results to the complete set of dates.
However, it's slightly simpler and I believe it produces the output format you want (also, I've compiled and run it, so it at least does what I expect it to :-))
I've assumed a collection called events, the members of which have a property timestamp
Note that I've assumed the timestamps may include times as well as dates - if this isn't the case you can simplify the code slightly by omitting the .Dates
Finally, I've determined the range of dates to be defined by the period you have data for - obviously you can change the startDate and endDate values
DateTime startDate = events.OrderBy(e=>e.timestamp).First().timestamp.Date;
DateTime endDate = events.OrderBy(e=>e.timestamp).Last().timestamp.Date;
var allDates = Enumerable.Range(0, (endDate - startDate).Days + 1)
.Select(a => startDate.AddDays(a))
.GroupJoin(events, d=>d.Date, e=>e.timestamp,
(d, e) =>
new{date = d, count = e.Count()});

Not in LINQ2SQL as far as I can figure out, but the standard trick when you write a stored procedure is to generate a list of all dates in the range, filter out those already in the list and take a union of the results.
This should be quite easy to do in LINQ2Objects once you have retrieved the sparse data.

linq goup by day

I have the following grouping statement:
var test = from a in MyDC.Table
where .....
group a by a.Date into daygroups
select new MyModel()
{
TheCount = (from c in daygroups
where c.AppointDate < "the date of the daygroups for this day").Sum( d =>d)
}
Basically, the query looks in a table for appointments within a certain month and does counts by day for each day of the month. Daygroups groups the results by days so I can do the daily counts. How do I specify the date within the daygroups?
Thanks.

Try
where c.AppointDate < daygroups.Key

Your date is in daygroups.Key
When grouping with LINQ, the value you are grouping on ends up in the Key property of the IGrouping object, daygroups in your case.
Something like this, perhaps:
var test = from a in MyDC.Table
where .....
group a by a.Date into daygroups
select new MyModel {
TheCount = daygroups.Where(d => d.AppointDate < daygroups.Key).Count()
}

Filling in missing dates using a linq group by date query

I have a Linq query that basically counts how many entries were created on a particular day, which is done by grouping by year, month, day. The problem is that because some days won't have any entries I need to back fill those missing "calendar days" with an entry of 0 count.
My guess is that this can probably be done with a Union or something, or maybe even some simple for loop to process the records after the query.
Here is the query:
from l in context.LoginToken
where l.CreatedOn >= start && l.CreatedOn <= finish
group l by
new{l.CreatedOn.Year, l.CreatedOn.Month, l.CreatedOn.Day} into groups
orderby groups.Key.Year , groups.Key.Month , groups.Key.Day
select new StatsDateWithCount {
Count = groups.Count(),
Year = groups.Key.Year,
Month = groups.Key.Month,
Day = groups.Key.Day
}));
If I have data for 12/1 - 12/4/2009 like (simplified):
12/1/2009 20
12/2/2009 15
12/4/2009 16
I want an entry with 12/3/2009 0 added by code.
I know that in general this should be done in the DB using a denormalized table that you either populate with data or join to a calendar table, but my question is how would I accomplish this in code?
Can it be done in Linq? Should it be done in Linq?

I just did this today. I gathered the complete data from the database and then generated a "sample empty" table. Finally, I did an outer join of the empty table with the real data and used the DefaultIfEmpty() construct to deal with knowing when a row was missing from the database to fill it in with defaults.
Here's my code:
int days = 30;
// Gather the data we have in the database, which will be incomplete for the graph (i.e. missing dates/subsystems).
var dataQuery =
from tr in SourceDataTable
where (DateTime.UtcNow - tr.CreatedTime).Days < 30
group tr by new { tr.CreatedTime.Date, tr.Subsystem } into g
orderby g.Key.Date ascending, g.Key.SubSystem ascending
select new MyResults()
{
Date = g.Key.Date,
SubSystem = g.Key.SubSystem,
Count = g.Count()
};
// Generate the list of subsystems we want.
var subsystems = new[] { SubSystem.Foo, SubSystem.Bar }.AsQueryable();
// Generate the list of Dates we want.
var datetimes = new List<DateTime>();
for (int i = 0; i < days; i++)
{
datetimes.Add(DateTime.UtcNow.AddDays(-i).Date);
}
// Generate the empty table, which is the shape of the output we want but without counts.
var emptyTableQuery =
from dt in datetimes
from subsys in subsystems
select new MyResults()
{
Date = dt.Date,
SubSystem = subsys,
Count = 0
};
// Perform an outer join of the empty table with the real data and use the magic DefaultIfEmpty
// to handle the "there's no data from the database case".
var finalQuery =
from e in emptyTableQuery
join realData in dataQuery on
new { e.Date, e.SubSystem } equals
new { realData.Date, realData.SubSystem } into g
from realDataJoin in g.DefaultIfEmpty()
select new MyResults()
{
Date = e.Date,
SubSystem = e.SubSystem,
Count = realDataJoin == null ? 0 : realDataJoin.Count
};
return finalQuery.OrderBy(x => x.Date).AsEnumerable();

I made a helper function which is designed to be used with anonymous types, and reused in as generic way as possible.
Let's say this is your query to get a list of orders for each date.
var orders = db.Orders
.GroupBy(o => o.OrderDate)
.Select(o => new
{
OrderDate = o.Key,
OrderCount = o.Count(),
Sales = o.Sum(i => i.SubTotal)
}
.OrderBy(o => o.OrderDate);
For my function to work please note this list must be ordered by date. If we had a day with no sales there would be a hole in the list.
Now for the function that will fill in the blanks with a default value (instance of anonymous type).
private static IEnumerable<T> FillInEmptyDates<T>(IEnumerable<DateTime> allDates, IEnumerable<T> sourceData, Func<T, DateTime> dateSelector, Func<DateTime, T> defaultItemFactory)
{
// iterate through the source collection
var iterator = sourceData.GetEnumerator();
iterator.MoveNext();
// for each date in the desired list
foreach (var desiredDate in allDates)
{
// check if the current item exists and is the 'desired' date
if (iterator.Current != null &&
dateSelector(iterator.Current) == desiredDate)
{
// if so then return it and move to the next item
yield return iterator.Current;
iterator.MoveNext();
// if source data is now exhausted then continue
if (iterator.Current == null)
{
continue;
}
// ensure next item is not a duplicate
if (dateSelector(iterator.Current) == desiredDate)
{
throw new Exception("More than one item found in source collection with date " + desiredDate);
}
}
else
{
// if the current 'desired' item doesn't exist then
// create a dummy item using the provided factory
yield return defaultItemFactory(desiredDate);
}
}
}
The usage is as follows:
// first you must determine your desired list of dates which must be in order
// determine this however you want
var desiredDates = ....;
// fill in any holes
var ordersByDate = FillInEmptyDates(desiredDates,
// Source list (with holes)
orders,
// How do we get a date from an order
(order) => order.OrderDate,
// How do we create an 'empty' item
(date) => new
{
OrderDate = date,
OrderCount = 0,
Sales = 0
});
Must make sure there are no duplicates in the desired dates list
Both desiredDates and sourceData must be in order
Because the method is generic if you are using an anonymous type then the compiler will automatically tell you if your 'default' item is not the same 'shape' as a regular item.
Right now I include a check for duplicate items in sourceData but there is no such check in desiredDates
If you want to ensure the lists are ordered by date you will need to add extra code

Essentially what I ended up doing here is creating a list of the same type with all the dates in the range and 0 value for the count. Then union the results from my original query with this list. The major hurdle was simply creating a custom IEqualityComparer. For more details here: click here

You can generate the list of dates starting from "start" and ending at "finish", a then step by step check the number of count for each date separately