We Keep Coding

C#: Groupby, Difference of data in dataset

Following is my dataset which I get from the database:
Id -- Date -- ClockIn -- ClockOut
1 -- 1/1/2016 -- 1/1/2016:09:00:00 -- 1/1/2016:17:03:00
1 -- 1/2/2016 -- 1/1/2016:09:00:00 -- 1/1/2016:11:30:00
1 -- 1/2/2016 -- 1/1/2016:13:00:00 -- 1/1/2016:15:03:00
Expected result
1 -- 1/1/2016(8 hrs) -- 09:00:00 -- 17:03:00
1 -- 1/2/2016(4 hrs) -- 09:00:00 -- 11:30:00
-- 13:00:00 -- 15:03:00
Question: I want a group-by on Date field and difference of clockIn/out and its sum for the day
Here is the code what I am trying but not able to get it work:
var resultSet = from newSet in Dtinfo.AsEnumerable()
group newSet by new
{
uid = newSet.Field<int>("UID"),
gDate = newSet.Field<DateTime>("wDate"),
inTime = newSet.Field<DateTime>("punchIn"),
outTime = newSet.Field<DateTime>("punchOut"),
location = newSet.Field<String>("locName"),
typeName = newSet.Field<String>("typeName"),
} into counter
select new
{
UID = counter.Key.uid,
wDate=counter.Key.gDate,
punchIn = counter.Key.inTime,
punchOut = counter.Key.outTime,
locName= counter.Key.location,
typeName= counter.Key.typeName,
diff = (counter.Key.outTime - counter.Key.InTime).TotalHours,
};
I am getting the difference but I need the group by and sum to work as well.
Any light on the path would be helpful.

here is the query.
I am returning anonymous class with
Date - date which was grouped by
TotalHours - sum of all entries for that day
Entries - entries, another anonymous class
It is probably better to replace them with named classes.
var res = from newset in Dtinfo.AsEnumerable()
group newset by newset.Field<DateTime>("wDate")
into counter
select new
{
Date = counter.Key,
TotalHours = counter.Select(a => a.Field<DateTime>("punchOut") - a.Field<DateTime>("punchIn")).Sum(a => a.TotalHours),
Entries = counter.Select(a => new
{
uid = a.Field<int>("UID"),
gDate = a.Field<DateTime>("wDate"),
/*other fields here
*
*
*/
})
};

Speed up the linq group by statement

I have a table like this
UserID Year EffectiveDate Type SpecialExpiryDate
1 2015 7/1/2014 A
1 2016 7/1/2015 B 10/1/2015
there is no ExpriyDate in the table because it is only valid for one year, so the expiry date can be calculated from the effective date by adding a year.
The result I want to get is like this (the current year's effective date and the next year's expiry date)
UserID EffectiveDate ExpiryDate
1 7/1/2014 7/1/2016
And If the user's type is B, then there will be a special expiry date, so for this person, the result will be
UserID EffectiveDate ExpiryDate
1 7/1/2014 10/1/2015
Here is the code I wrote
var result = db.Table1
.Where(x => x.Year>= 2015 && (x.Type == "A" || x.Type == "B"))
.GroupBy(y => y.UserID)
.OrderByDescending(x => x.FirstOrDefault().Year)
.Select(t => new
{
ID = t.Key,
Type = t.FirstOrDefault().Type,
EffectiveDate = t.FirstOrDefault().EffectiveDate,
ExpiryDate = t.FirstOrDefault().SpecialExpiryDate != null ? t.FirstOrDefault().SpecialExpiryDate : (t.Count() >= 2 ? NextExpiryDate : CurrentExpiryDate)
}
);
The code can get the result I need, but the problem is that in the result set there are about 10000 records which took about 5 to 6 seconds. The project is for a web search API, so I want to speed it up, is there a better way to do the query?
Edit
Sorry I made a mistake, in the select clause it should be
EffectiveDate = t.LastOrDefault().EffectiveDate
but in the Linq of C#, it didn't support this LastOrDefault function transfered to sql, and it cause the new problem, what is the easiest way to get the second item of the group?

You could generate the calculated data on the fly, using a View in your database.
Something like this (pseudocode):
Create View vwUsers AS
Select
UserID,
Year,
EffectiveDate,
EffectiveData + 1 as ExpiryDate, // <--
Type,
SpecialExpiryDate
From
tblUsers
And just connect your LINQ query to that.

Try this:
var result =
db
.Table1
.Where(x => x.Year>= 2015 && (x.Type == "A" || x.Type == "B"))
.GroupBy(y => y.UserID)
.SelectMany(y => y.Take(1), (y, z) => new
{
ID = y.Key,
z.Type,
z.EffectiveDate,
ExpiryDate = z.SpecialExpiryDate != null
? z.SpecialExpiryDate
: (t.Count() >= 2 ? NextExpiryDate : CurrentExpiryDate),
z.Year,
})
.OrderByDescending(x => x.Year);
The .SelectMany(y => y.Take(1) effectively does the .FirstOrDefault() part of your code. By doing this once rather than for many properties you may improve the speed immensely.
In a test I performed using a similarly structured query I got these sub-queries being run when using your approach:
SELECT t0.increment_id
FROM sales_flat_order AS t0
GROUP BY t0.increment_id
SELECT t0.hidden_tax_amount
FROM sales_flat_order AS t0
WHERE ((t0.increment_id IS NULL AND #n0 IS NULL) OR (t0.increment_id = #n0))
LIMIT 0, 1
-- n0 = [100000001]
SELECT t0.customer_email
FROM sales_flat_order AS t0
WHERE ((t0.increment_id IS NULL AND #n0 IS NULL) OR (t0.increment_id = #n0))
LIMIT 0, 1
-- n0 = [100000001]
SELECT t0.hidden_tax_amount
FROM sales_flat_order AS t0
WHERE ((t0.increment_id IS NULL AND #n0 IS NULL) OR (t0.increment_id = #n0))
LIMIT 0, 1
-- n0 = [100000002]
SELECT t0.customer_email
FROM sales_flat_order AS t0
WHERE ((t0.increment_id IS NULL AND #n0 IS NULL) OR (t0.increment_id = #n0))
LIMIT 0, 1
-- n0 = [100000002]
(This continued on for two sub-queries per record number.)
If I ran my approach I got this single query:
SELECT t0.increment_id, t1.hidden_tax_amount, t1.customer_email
FROM (
SELECT t2.increment_id
FROM sales_flat_order AS t2
GROUP BY t2.increment_id
) AS t0
CROSS APPLY (
SELECT t3.customer_email, t3.hidden_tax_amount
FROM sales_flat_order AS t3
WHERE ((t3.increment_id IS NULL AND t0.increment_id IS NULL) OR (t3.increment_id = t0.increment_id))
LIMIT 0, 1
) AS t1
My approach should be much faster.

Fetch records for past days excluding today using LinQ

I have database which consists of following columns:
App Icon, App Title, App Description, Date, Actual Price,
Itunes Link, AppID, UserID, Downloads Count
I need to know how many apps were inserted for past 7 days excluding today by Linq query. I tried goggling it and this is the code I found, but I am unable to modify it according to my needs
var q = from u in entity.Submit_App
where u.Date > DateTime.Now.AddDays(-30) &&
u.Date <= DateTime.Now
group u by EntityFunctions.TruncateTime(u.Date) into g
select new ChartTotal()
{
OrderDate = g.Key,
Total = g.Sum(y => y.GrandTotal)
};

Use DateTime.Today to get date without time part. Also you can introduce new range variable with let keyword to simplify your query:
var q = from u in entity.Submit_App
let date = EntityFunctions.TruncateTime(u.Date)
where date >= DateTime.Today.AddDays(-7) &&
date < DateTime.Today
group u by date into g
select new ChartTotal {
OrderDate = g.Key,
Total = g.Sum(x => x.GrandTotal)
};

MySQL entity framework Group and Select By Date

I have a MySQL database with a table 'Clicks'. There's a 'Created' column (datetime) which I'd like to group by and select the year, month and day part. I want to count the records per day within a specific range of dates (startDate and endDate).
var query = from c in scope.Entities.Clicks
where c.Created >= startDate && c.Created <= endDate
group c by new {c.Created.Year, c.Created.Month, c.Created.Day}
into grouped
select new {
Year = grouped.Key.Year,
Month = grouped.Key.Month,
Day = grouped.Key.Day,
Clicks = grouped.Count()
};
This produces a bad query:
SELECT
`GroupBy1`.`K1` AS `C1`,
`GroupBy1`.`K2` AS `C2`,
`GroupBy1`.`K3` AS `C3`,
`GroupBy1`.`K4` AS `C4`,
`GroupBy1`.`A1` AS `C5`
FROM (SELECT
COUNT(1) AS `A1`
FROM `Click` AS `Extent1`
WHERE (`Extent1`.`Created` >= #p__linq__0) AND (`Extent1`.`Created` <= #p__linq__1)
GROUP BY
1,
YEAR(`Extent1`.`Created`),
MONTH(`Extent1`.`Created`),
DAY(`Extent1`.`Created`)) AS `GroupBy1`
With an error: MySql.Data.MySqlClient.MySqlException: Can't group on 'A1'
What do I do wrong? Or is this a MySql connector bug? I tried MySQL connectors 6.5.4 and 6.6.5

DJ KRAZE I tried your approach and it turns out that the double "select" is necessary to produce a correct query by EF. Thanks! I post the full answer here if anyone needs it.
I tested both ways and both work:
var subQuery = from c in scope.Entities.Clicks
where c.Created >= startDate && c.Created <= endDate
select new { c.Created.Year, c.Created.Month, c.Created.Day };
var query = from c in subQuery
group c by new {c.Year, c.Month, c.Day}
into grouped
select new {
Year = grouped.Key.Year,
Month = grouped.Key.Month,
Day = grouped.Key.Day,
Clicks = grouped.Count()
};
And
var query = scope.Entities.Clicks.Where(c => c.Created >= startDate && c.Created <= endDate)
.Select(c => new { c.Created.Year, c.Created.Month, c.Created.Day})
.GroupBy(c => new {c.Year, c.Month, c.Day})
.Select(grouped => new { Clicks = grouped.Count(), grouped.Key.Year, grouped.Key.Month, grouped.Key.Day});
These give the correct mysql query:
SELECT
1 AS `C1`,
`GroupBy1`.`K1` AS `C2`,
`GroupBy1`.`K2` AS `C3`,
`GroupBy1`.`K3` AS `C4`,
`GroupBy1`.`A1` AS `C5`
FROM (SELECT
`Project1`.`C1` AS `K1`,
`Project1`.`C2` AS `K2`,
`Project1`.`C3` AS `K3`,
COUNT(1) AS `A1`
FROM (SELECT
YEAR(`Extent1`.`Created`) AS `C1`,
MONTH(`Extent1`.`Created`) AS `C2`,
DAY(`Extent1`.`Created`) AS `C3`
FROM `Click` AS `Extent1`
WHERE (`Extent1`.`Created` >= #p__linq__0) AND (`Extent1`.`Created` <= #p__linq__1)) AS `Project1`
GROUP BY
`Project1`.`C1`,
`Project1`.`C2`,
`Project1`.`C3`) AS `GroupBy1`
(I still think this is a bug and my first query should work as well btw)

How to perform aggregate function on last 4 rows of data?

I've got a table off the following model.
public class WeeklyNums
{
public int FranchiseId { get; set; }
public DateTime WeekEnding { get; set; }
public decimal Sales { get; set; }
}
I need a fourth column that calculates the minimum for this week and the previous three weeks. So the output would look like this.
1 7-Jan $1
1 14-Jan $2
1 21-Jan $3
1 28-Jan $4 **1**
1 4-Feb $4 **2**
1 11-Feb $6 **3**
1 18-Feb $4 **4**
1 25-Feb $8 **4**
1 3-Mar $7 **4**
I have no idea where to even start. Even some help with solving it in SQL would be helpful.
thx!

Consider using outer apply:
select yt1.*
, hist.four_week_min
from YourTable yt1
outer apply
(
select min(col1) as four_week_min
from YourTable yt2
where yt2.dt between dateadd(wk, -3, yt1.dt) and yt1.dt
) hist
Working example at SQL Fiddle.

var runningMins = from weekNum in data
select new
{
FranchiseId = weekNum.FranchiseId,
WeekEnding = weekNum.WeekEnding,
Sales = weekNum.Sales,
LastThreeWeeks = data.OrderByDescending( x => x.WeekEnding )
.Where( x => x.WeekEnding <= weekNum.WeekEnding )
.Take( 4 )
.Min( x => x.Sales )
};
SQL Query that will return minimum of the current and the three previous regardless of whether the dates are exactly three weeks apart:
With RnkItems As
(
Select DateVal, Sales
, Row_Number() Over ( Order By DateVal ) As Rnk
From SourceData
)
Select *
, (
Select Min(Sales)
From RnkItems As R1
Where R1.Rnk Between R.Rnk - 3 And R.Rnk
)
From RnkItems R
Order By 1
SQL Fiddle version

I know I'm too late, but here's the linq version:
var result = from w1 in db.Table
from w2 in db.Table.Where(x => x.WeekEnding >= w1.WeekEnding.AddDays(-28))
select new
{
FranchiseId = w1.FranchiseId,
WeekEnding = w1.WeekEnding,
Sales = w1.Sales,
SalesMin = w2.Min(x => x.Sales)
};