Aggregate and Group by Datetime - c#

Having this data (SQL):
I call (every minute with timer )data then group into 30 minutes interval with this method:
public void GetData()
{
try
{
using (crypto_dbEntities1 context = new crypto_dbEntities1(con))
{
var result = context.kraken_btc.Where(x => x.date >= LastRecordedPoint).ToList();
result = AggregateCandlesIntoRequestedTimePeriod(Period.Minute, Period.HalfAnHour, result);
foreach (var data in result.OrderBy(x => x.date))
{
data.date = DateTime.SpecifyKind(data.date.DateTime, DateTimeKind.Utc).ToUniversalTime();
Point pb = new Point
{
Date = data.date.DateTime.ToLocalTime(),
JDate = (long)data.javaDate,
Open = (double)data.open,
High = (double)data.hight,
Close = (double)data.close,
Low = (double)data.low,
Volume = (long)data.volume,
};
if (pb.Date <= LastRecordedPoint)
{
MainCollection.Last().Date = data.date.DateTime.ToLocalTime();
MainCollection.Last().Close = (double)data.close;
MainCollection.Last().High = (double)data.hight;
MainCollection.Last().Open = (double)data.open;
MainCollection.Last().Low = (double)data.low;
MainCollection.Last().Volume = (long)data.volume;
Debug.WriteLine(pb.Date + " Updated data ..");
}
else
{
MainCollection.Add(pb);
}
LastRecordedPoint = pb.Date;
}
}
}
catch (Exception err)
{
MessageBox.Show(err.ToString());
}
}
public enum Period
{
Minute = 1,
FiveMinutes = 5,
TenMinutes = 10,
QuarterOfAnHour = 15,
HalfAnHour = 30,
Hour = 60
}
private List<kraken_btc> AggregateCandlesIntoRequestedTimePeriod(Period rawPeriod, Period requestedPeriod, List<kraken_btc> candles)
{
int rawPeriodDivisor = (int)requestedPeriod;
candles = candles.GroupBy(g => new { TimeBoundary = new DateTime(g.date.Year, g.date.Month, g.date.Day, g.date.Hour, (g.date.Minute / rawPeriodDivisor) * rawPeriodDivisor, 0) })
.Select(s => new kraken_btc
{
date = s.Key.TimeBoundary,
hight = s.Max(z => z.hight),
low = s.Min(z => z.low),
open = s.First().open,
close = s.Last().close,
volume = s.Sum(z => z.volume),
})
.OrderBy(o => o.date)
.ToList();
return candles;
}
And it does the job of aggregating the data, but the problem is I have to wait 30 minutes for my serie finally update if you look at the last candle its time is 17.30 when database has data for 17.47 (im UTC +2.00).
So the question is how can I group data and start draw an incomplete candle at 18.00 like all exchange platform does...

The GroupBy statement rounds down the dates, so all data from 17:30 up to 17:47 is rounded down to 17:30.
I would move the code to calculate the TimeBoundary to its own method so you can unit test it fully, using the RoundUp/RoundDown methods in this question How can I round up the time to the nearest X minutes? by redent84
public static DateTime RoundUp(this DateTime dt, TimeSpan d)
{
var modTicks = dt.Ticks % d.Ticks;
var delta = modTicks != 0 ? d.Ticks - modTicks : 0;
return new DateTime(dt.Ticks + delta, dt.Kind);
}
public static DateTime RoundDown(this DateTime dt, TimeSpan d)
{
var delta = dt.Ticks % d.Ticks;
return new DateTime(dt.Ticks - delta, dt.Kind);
}

Related

pairing and calculating the hours of checkTypes

Continuing from here, I have a table named Attendancelogs which has all the sorted logs CHeckTypes In and Out accordingly. (Thanks to #StephenMuecke the previous question was pretty much resolved) However, what I am trying to achieve next is to make a pairList of all the In and Out in a particular period (StartDate and EndDate) and then calculate the total hours of the employee.
Different Cases, already been taken care of in the previous question
I have two method, one called getSingleDevicePairs(int EnrollNumber, DateTime StartDate, DateTime EndDate) which creates the pairList and the second method getTimeSpentEachDay(List<Pair> pairList) which calculates the total hours of each day.
Pair.cs
public class Pair {
public int id { get; set; }
public int RegisterationId { get; set; }
public int EmpID { get; set; }
public DateTime InnDateTime { get; set; }
public DateTime OutDateTime { get; set; }
}
public List<Pair> getSingleDevicePairs(int EnrollNumber, DateTime StartDate, DateTime EndDate) {
DateTime today = DateTime.Now;
List<Pair> pairList = new List<Pair>();
var logs = db.AttendanceLogs.Where(x => x.RegisterationId == EnrollNumber && x.Date >= StartDate &&
x.Date <= EndDate && x.isIgnore != true && (x.CheckType == "In" || x.CheckType == "Out")).Distinct().ToList();
int loopEnd = 0;
bool oddLogs = false;
if (logs.Count % 2 == 0) {
loopEnd = logs.Count;
} else {
loopEnd = logs.Count - 1;
oddLogs = true;
}
bool inn = true;
if (loopEnd > 1) {
Pair pair = new Pair();
for (int v = 0; v < loopEnd; v++) {
if (inn) {
pair.InnDateTime = logs[v].DateTime;
inn = false;
} else {
pair.OutDateTime = logs[v].DateTime;
inn = true;
pairList.Add(pair);
pair = new Pair();
}
}
}
Bogus bogus = new Bogus();
DateTime bogusDate = new DateTime();
if (oddLogs) {
bogus.MachineNum = logs[logs.Count - 1].DeviceID;
bogus.RegisterationId = logs[logs.Count - 1].RegisterationId;
bogus.DateTime = logs[logs.Count - 1].DateTime;
bogusDate = logs[logs.Count - 1].DateTime;
}
return pairList;
}
^I changed the above method with different approach, since the above approach would mess up with Case 1 shown in the link above.
public List<Pair> getSingleDevicePairs(int EnrollNumber, DateTime StartDate, DateTime EndDate) {
DateTime today = DateTime.Now;
List<Pair> pairList = new List<Pair>();
var logs = db.AttendanceLogs.Where(x => x.RegisterationId == EnrollNumber && x.Date >= StartDate &&
x.Date <= EndDate && x.isIgnore != true && (x.CheckType == "In" || x.CheckType == "Out")).Distinct().ToList();
bool isCheck = false;
Pair pair = new Pair();
DateTime previous = logs.FirstOrDefault().DateTime;
foreach (var log in logs) {
if (!isCheck) {
pair.InnDateTime = log.DateTime;
isCheck = true;
} else {
pair.OutDateTime = log.DateTime;
isCheck = false;
}
pairList.Add(pair);
pair = new Pair();
}
return pairList;
}
^This approach again will fail at Case 1 as it is sequentially adding In and Out in the pairList.
public List<DateAndTime> getTimeSpentEachDay(List<Pair> pairList) {
List<DateAndTime> list = new List<DateAndTime>();
if (pairList.Count > 0) {
for (int i = 0; i < pairList.Count; i++) {
TimeSpan span = TimeSpan.Zero;
// bool flag = false;
int result = -1;
do {
span = span + (pairList[i].OutDateTime - pairList[i].InnDateTime);
result = -1;
if (i < pairList.Count - 1) {
DateTime p = (DateTime)pairList[i].InnDateTime;
DateTime q = (DateTime)pairList[i + 1].InnDateTime;
result = DateTime.Compare(p.Date, q.Date);
}
if (result == 0) {
i++;
// flag = true;
}
} while (result == 0);
//if (i == pairList.Count - 1)
//{
// span = span + (pairList[i].OutDateTime - pairList[i].InnDateTime) ?? TimeSpan.Zero;
//}
DateAndTime dnt = new DateAndTime();
dnt.date = ((DateTime)pairList[i].InnDateTime).ToString("yyyy-MM-dd");
dnt.Time = span;
list.Add(dnt);
}
}
return list.ToList();
}
I am trying to get total hours for each pair as well as a way to take the odd In's together which I can display on the calendar.
Below is an image of the calendar view for an employee which displays only the paired hours, I even want to show a In that would indicate the employee that he either forgot to check out or a missing entry was made.
*Lets say he had a checkin for 9th April of around 08:00 PM that had no Check out on that day and also in the NightersLimit (12 AM - 7 AM) then there should be a single line displayed on the calendar which can only happen if I bring the bogus record for a day from previous method.
There is nothing wrong with the getTimeSpentEachDay(); just a little changes in the getSingleDevicePairs(); method, your first approach fails since it eliminates the last entry of the logs in case of an odd number and your second approach fails since it is not taking account of the sequence the logs are stored in the table.
public List<Pair> getSingleDevicePairs(int EnrollNumber, DateTime StartDate, DateTime EndDate, int? missingEntry)
{
var logs = db.AttendanceLogs.Where(x => x.RegisterationId == EnrollNumber &&
x.Date >= StartDate && x.Date <= EndDate && x.isIgnore != true
&& (x.CheckType == "In" || x.CheckType == "Out")).Distinct().ToList();
if (logs.Count > 0)
{
bool isCheck = false;
Pair pair = new Pair();
DateTime previous = logs.FirstOrDefault().DateTime;
foreach (var log in logs)
{
if (!isCheck)
{
if (log.CheckType == "In")
{
pair.InnDateTime = log.DateTime;
isCheck = true;
}
}
else
{
if (log.CheckType == "Out")
{
pair.OutDateTime = log.DateTime;
isCheck = false;
pairList.Add(pair);
pair = new Pair();
}
if (pair.OutDateTime == DateTime.MinValue)
{
pair.InnDateTime = log.DateTime;
}
}
}
}
return pairList;
}
This completes the pair only when there is an Out for an In.

c# get next execution datetime

I have the following code:
Dictionary<DayOfWeek, List<TimeSpan>> Daily = new Dictionary<DayOfWeek, List<TimeSpan>>();
The idea is that i can add a day with a time to the Daily dictionary. But a job can execute on the same day more than once.
so Daily can look like this:
{ "Monday" : [{"Hour":10, "Minute": 15}, {"Hour": 8, "Minute":5}] }
Now i would like to get the next execution datetime.
private void UpdateNextExecutionTime()
{
TimeSpan t = new TimeSpan(15, 15, 0);
DayOfWeek current = DateTime.Now.DayOfWeek;
DayOfWeek tmp = current;
TimeSpan time = new TimeSpan(DateTime.Now.Hour, DateTime.Now.Minute, DateTime.Now.Second);
int cur = (int)current;
for(int i = 0; i < 7; i++) {
if(Daily.ContainsKey((DayOfWeek)cur)) {
tmp = (DayOfWeek)cur;
/* stuck */
continue;
}
cur++;
if (cur >= 7)
cur = 0;
}
}
I managed to get the first upcoming day (including today) in the Daily dictionary but i'm stuck on the getting the time.
How would i be able to do this?
Assuming t is the time you're using as the point at which you'd like to find the next execution time, put this where you have 'stuck' (edit; just realized you changed to using TimeSpan in your list):
var execTimes = Daily[tmp];
if (execTimes != null) {
var nextTime = execTimes.OrderBy(x => x).FirstOrDefault(x => x > t);
if (nextTime != default(TimeSpan)) {
// do something...
}
}
You can also have the day in the TimeSpan,
Here is a simple example the finds the next date:
var schedule = new List<TimeSpan>{
new TimeSpan(0,16,30,0),
new TimeSpan(1,16,30,0),
new TimeSpan(5,16,30,0)
};
var monday = DateTime.Today.AddDays(-(int)DateTime.Today.DayOfWeek + 1);
var fromMonday = DateTime.Now - monday;
var next = schedule.OrderBy(t => t).FirstOrDefault(t => t > fromMonday);
Console.Write(monday + next - DateTime.Now);

Padding time series data?

I am receiving large amounts of time series data in a list timeSeries.Records. The object has a date time property of (10 minute intervals) TimeStamp and a data value property Data
The time series may have gaps which I need to pad with the appropriate date time stamp and a value of double.NaN. This will allow me to display the data correctly within a chart in the UI.
I have written the following code which works but is incredibly slow! I suspect this is because I am newing up new objects & returning them in the while loop. I assume this can be optimised significantly and maybe is the wrong approach completely but not sure where to start...
Here's the code which appends the data to the chart:
foreach (TimeSeriesRecord record in this.FillTimeSeriesGaps(timeSeries))
{
dataSeries.Append(record.TimeStamp, record.Data);
}
Filling the gaps:
private IEnumerable<TimeSeriesRecord> FillTimeSeriesGaps(ITimeSeriesProvider timeSeries)
{
// Get the min & max records by date time
TimeSeriesRecord minRecord = timeSeries.Records.OrderBy(r => r.TimeStamp).FirstOrDefault();
TimeSeriesRecord maxRecord = timeSeries.Records.OrderByDescending(r => r.TimeStamp).FirstOrDefault();
// 10 sec time interval
TimeSpan seriesIntervalTime = new TimeSpan(0, 10, 0);
DateTime workingDateTime = minRecord.TimeStamp;
while (minRecord.TimeStamp <= maxRecord.TimeStamp)
{
if (timeSeries.Records.All(r => r.TimeStamp != workingDateTime))
{
yield return new TimeSeriesRecord() {TimeStamp = workingDateTime, Data = double.NaN};
}
else
{
yield return new TimeSeriesRecord()
{
TimeStamp = workingDateTime,
Data = (from r in timeSeries.Records
where r.TimeStamp == workingDateTime
select r.Data).First()
};
workingDateTime = workingDateTime.Add(seriesIntervalTime);
}
}
}
For anyone who is interested this is how I sped things up:
private static IEnumerable<TimeSeriesRecord> FillTimeSeriesGaps(ITimeSeriesProvider timeSeries)
{
TimeSpan seriesIntervalTime = new TimeSpan(0, 10, 0);
DateTime previousDateTime = DateTime.MinValue;
foreach (GenericTimeSeriesRecord record in timeSeries.Records)
{
if (previousDateTime == DateTime.MinValue)
{
yield return record;
previousDateTime = record.TimeStamp;
continue;
}
if (previousDateTime + seriesIntervalTime == record.TimeStamp)
{
yield return record;
previousDateTime = record.TimeStamp;
continue;
}
else
{
yield return new TimeSeriesRecord() { TimeStamp = previousDateTime + seriesIntervalTime, Data = double.NaN };
previousDateTime = previousDateTime + seriesIntervalTime;
}
}
}

c# linq statement to calculate balance over 12 months

I'm trying to plot a flot chart. I want the x axis to be months from january to december and y axis to be account balances. I have the income and expense of the account over 12 months time but subtracting them would only give me the difference for the month it does not add the balance from the previous month.
Here is how I obtained the income and expenses over the range:
var monthsToDate = Enumerable.Range(1, 12)
.Select(m => new DateTime(DateTime.Today.Year, m, 1))
.ToList();
var sums = from month in monthsToDate
select new
{
month = month,
income = (from account in household.Accounts
from transaction in account.Transactions
where transaction.IsIncome && transaction.Created.Month == month.Month
select transaction.Amount).DefaultIfEmpty().Sum(),
expense = (from account in household.Accounts
from transaction in account.Transactions
where !transaction.IsIncome && transaction.Created.Month == month.Month
select transaction.Amount).DefaultIfEmpty().Sum(),
};
what I get back is this
.
.
.
[4] = { month = {5/1/2015 12:00:00 AM}, income = 3000, expense = 1804.75 }
[5] = { month = {6/1/2015 12:00:00 AM}, income = 2500, expense = 1560 }
[6] = { month = {7/1/2015 12:00:00 AM}, income = 0, expense = 550 }
.
.
.
You can add this reusable extension method to your code:
internal static class CollectionExtensions
{
/// <summary>
/// Returns a sequence whose first element is the first element of the source sequence (if any),
/// and every subsequent element is the result of applying a specified accumulator function to the
/// previous element of the resulting sequence and the next member of the source sequence.
/// </summary>
/// <typeparam name="T"></typeparam>
/// <param name="source"></param>
/// <param name="accumulator"></param>
/// <returns></returns>
public static IEnumerable<T> Accumulate<T>(this IEnumerable<T> source, Func<T, T, T> accumulator)
{
if (source == null) throw new ArgumentNullException("source");
if (accumulator == null) throw new ArgumentNullException("accumulator");
return source.AccumulateImpl(accumulator);
}
private static IEnumerable<T> AccumulateImpl<T>(this IEnumerable<T> source, Func<T, T, T> accumulator)
{
using (var enumerator = source.GetEnumerator())
{
T accumulation;
T next;
if (enumerator.MoveNext())
accumulation = enumerator.Current;
else yield break;
yield return accumulation;
if (enumerator.MoveNext())
next = enumerator.Current;
else yield break;
while (true)
{
accumulation = accumulator(accumulation, next);
yield return accumulation;
if (enumerator.MoveNext())
next = enumerator.Current;
else yield break;
}
}
}
}
Example usage:
var range = Enumerable.Range(0, 5); // 0, 1, 2, 3, 4
var accumulated = range.Accumulate((x, y) => x + y); // 0, 1, 3, 6, 10
Now, if you change your select to return a named type, instead of an anonymous one (I'm assuming you're using decimal for money - if not, you can adapt this code):
internal class MonthlyIncomeAndExpenses
{
public DateTime Month { get; set; }
public decimal Income { get; set; }
public decimal Expenses { get; set; }
}
var sums = from month in monthsToDate
select new MonthlyIncomeAndExpenses
{
Month = month,
Income = ..., // what you already have
Expense = ..., // what you already have
};
Then you only have to add one easy line:
var accumulated = sums.Accumulate((previous, next) => new MonthlyIncomeAndExpenses
{
Month = next.Month,
Income = previous.Income + next.Income,
Expense = previous.Expense + next.Expense,
});
Since you need an array of arrays for flot, you could just run a loop over your array and sum up the income and expense from all previous month. Something like this (after your existing code):
var flotDataAsList = new List<double[]>();
double balance = 0.0;
for (int i = 0; i <= 12; i++)
{
DateTime thisMonth = new DateTime(year, i, 1);
balance += sums.Where(m => m.month == thisMonth).Sum(m => m.income - m.expense);
flotDataAsList .Add(new double[] { GetJavascriptTimestamp(thisMonth), balance });
}
var flotDataAsArray = flotDataAsList.ToArray();
GetJavascriptTimestamp() method can be taken from flot documentation:
public static int GetJavascriptTimestamp(System.DateTime input)
{
System.TimeSpan span = new System.TimeSpan(System.DateTime.Parse("1/1/1970").Ticks);
System.DateTime time = input.Subtract(span);
return (long)(time.Ticks / 10000);
}

Time difference in C#, 3-digit-format minutes

I have a problem calculating difference between two timestamps, when minutes are represented with 3 digits, e.g. 180:22 = 180 minutes and 22 seconds.
So, can you help me how can I get difference between timestamps like:
180:22 and 122:11
or
232:21 and 31:34
etc.
UPDATE: I need to get difference between two times, defined as strings. What makes a problem is that minutes in those strings (times) are larger than 60, and they are over the limit. So I need to know how to find difference like in above examples (180:22 and 122:11, and 232:21 and 31:34)
Use System.TimeSpan structures:
var seconds=(new TimeSpan(0, 180, 22)-new TimeSpan(0, 122, 11)).TotalSeconds;
var minutes=(new TimeSpan(0, 232, 21)-new TimeSpan(0, 31, 34)).TotalMinutes;
Here's a class that will do this stuff:
public class CrazyTime
{
public TimeSpan TimeSpanRepresentation { get; set; }
public CrazyTime(TimeSpan timeSpan)
{
this.TimeSpanRepresentation = timeSpan;
}
public CrazyTime(string crazyTime)
{
// No error checking. Add if so desired
var pieces = crazyTime.Split(new[] { ':' });
var minutes = int.Parse(pieces[0]);
var seconds = int.Parse(pieces[1]);
TimeSpanRepresentation = new TimeSpan(0, minutes, seconds);
}
public static CrazyTime operator-(CrazyTime left, CrazyTime right)
{
var newValue = left.TimeSpanRepresentation - right.TimeSpanRepresentation;
return new CrazyTime(newValue);
}
public override string ToString()
{
// How should negative Values look?
return ((int)Math.Floor(TimeSpanRepresentation.TotalMinutes)).ToString() + ":" + TimeSpanRepresentation.Seconds.ToString();
}
}
Here's how it might be used:
var a = new CrazyTime("123:22");
var b = new CrazyTime("222:11");
var c = b - a;
Console.WriteLine(c);
This works:
string time1 = "180:22";
string time2 = "122:11";
TimeSpan span1 = getTimespan(time1);
TimeSpan span2 = getTimespan(time2);
TimeSpan diff = span1 - span2;
getTimespan just has to correctly parse the string. I decided on Regex to do that, but you could go any route, particularly if the delimiter ":" isn't ever going to change.
private static TimeSpan getTimespan(string time1)
{
Regex reg = new Regex(#"\d+");
MatchCollection matches = reg.Matches(time1);
if (matches.Count == 2)
{
int minutes = int.Parse(matches[0].Value);
int seconds = int.Parse(matches[1].Value);
return new TimeSpan(0, minutes, seconds);
}
return TimeSpan.Zero;
}

Categories