I'm working on list sorting in webjob.
It is working fine when i used c# List. But to increase performance i'm saving my data into Redis cache in the form of list.
my final target is only take out most recent last 5 minutes data.
working c# code -
public class MyObject
{
public uint InstrumentID { get; set; }
public decimal Close { get; set; }
public decimal High { get; set; }
public decimal Low { get; set; }
public decimal Open { get; set; }
public DateTime TimeStamp { get; set; }
public uint Volume { get; set; }
public DateTime Created { get; } = DateTime.Now;
public DateTime Ttl { get; } = DateTime.Now.AddMinutes(5);
public DateTime? Persisted { get; set; }
public bool IsDead => DateTime.Now > Ttl;
public bool IsPersisted => Persisted.HasValue;
public bool TimeToPersist => IsPersisted == false && DateTime.Now > Created.AddMinutes(5);
public DateTime GetStartOfPeriodByMins(int numMinutes)
{
int oldMinutes = TimeStamp.Minute;
int newMinutes = (oldMinutes / numMinutes) * numMinutes;
DateTime startOfPeriod = new DateTime(TimeStamp.Year, TimeStamp.Month, TimeStamp.Day, TimeStamp.Hour, newMinutes, 0);
return startOfPeriod;
}
}
var inputList = new SortedSet<MyObject>(new MyObjectComparer());
inputList.Add(new MyObject() { TimeStamp = DateTime.Now, Open = 9, High = 12, Low = 8, Close = 11, InstrumentID = 2526 });
Thread.Sleep(10000);
inputList.Add(new MyObject() { TimeStamp = DateTime.Now, Open = 9, High = 12, Low = 8, Close = 11, InstrumentID = 2526 });
Thread.Sleep(10000);
inputList.Add(new MyObject() { TimeStamp = DateTime.Now, Open = 9, High = 12, Low = 8, Close = 11, InstrumentID = 2526 });
Thread.Sleep(50000);
inputList.Add(new MyObject() { TimeStamp = DateTime.Now, Open = 9, High = 12, Low = 8, Close = 11, InstrumentID = 2526 });
var resultSet = inputList
.GroupBy(i => i.GetStartOfPeriodByMins(5))
.Select(gr =>
new
{
StartOfPeriod = gr.Key,
Min = gr.Min(item => item.Open),
Max = gr.Max(item => item.Open),
Open = gr.OrderBy(item => item.TimeStamp).First().Open,
Close = gr.OrderBy(item => item.TimeStamp).Last().Open
});
Now same records i'm continuously inserting into redis cache. and while tried to take last 5 minutes data I was thiking to use same GetStartOfPeriodByMins concept but it needs a list of MyObject class and redis return RedisValue[].
Redis code - using StackExchange.Redis package
var cache = RedisConnectorHelper.Connection.GetDatabase();
//int i = 0;
for (int i = 0; i < 10000; i++)
{
var tickDataHis = new MyObject()
{
InstrumentID = 2526,
Close = 14 + i,
High = 16 + i,
Low = 11 + i,
Open = 12 + i,
TimeStamp = DateTime.Now,
Volume = 11111
};
// insert into redis
cache.ListRightPush("tickData", JsonConvert.SerializeObject(tickDataHis));
Thread.Sleep(3000);
}
var inputList = cache.ListRange("tickData");
or is there any other way to get latest 5 minutes data from redis
cache?
I used Redis to store timeseries data in the past. In order to optimize data retrieval, I used a sorted set (in that case there were more than one, but the concept is the same) where the score was the unix timestamp of when the data were recorded, and I serialized my data using Newtonsoft.Json library.
The code was something like this:
var myData = new MyObject() { SomeProperty = "my text" };
long dataTimestamp = DateTimeOffset.Now.ToUnixTimeMilliseconds();
string serializedData = Newtonsoft.Json.JsonConvert.Convert(myData);
redisDb.SortedSetAdd("mySortedSet", dataTimestamp, serializedData);
Doing in this way, if you want to retrieve only the data of the last 5 minutes, you can directly filter the data loaded from Redis using SortedSetRangeByScore method and passing "now - 5 minutes" as starting score, so that you can deserialize only what you need (which is of course less expensive than deserializing the whole list):
var minutesToTake = 5;
long startingTime = DateTimeOffset.Now.AddMinutes(-minutesToTake).ToUnixTimeMilliseconds();
RedisValue[] redisData = redisDb.SortedSetRangeByScore("mySortedSet", startingTime);
After that you can easily deserialize your data with the help of linq (consider that RedisValue implements operator to string conversion):
MyObject[] myData = redisData.Select(d => JsonConvert.DeserializeObject<MyObject>(d)).ToArray();
Edit: I didn't use the package suggested by #Hasan Emrah Süngü. Maybe it is more efficient, I'm just explaining what I did at that time.
Edit 2: redisDb is my StackExchange.Redis.IDatabase instance.
Edit 3: Useful reference links:
data persistence on Redis: https://redis.io/topics/data-types-intro#redis-expires-keys-with-limited-time-to-live
Redis sorted sets: https://redis.io/topics/data-types-intro#redis-sorted-sets
sorted sets for timeseries: https://redislabs.com/redis-best-practices/time-series/sorted-set-time-series/
Related
I am building a NET 5 API and am unable to extract and calculate something. I have a table StockTransaction which among other has property Quantity (I skipped some properties for brevity):
public class StockTransaction : BaseEntity
{
public int Id { get; set; }
public decimal Price { get; set; }
public int Quantity { get; set; }
public bool Purchase { get; set; }
public int Resolved { get; set; }
}
Suppose I have 7 transactions, all of which are purchase:
List<StockTransaction> list = new List<StockTransaction>
{
new StockTransaction {Id = 1, Purchase = true, Quantity = 10, Resolved = 5, Price = 50},
new StockTransaction {Id = 2, Purchase = true, Quantity = 5, Resolved = 5, Price = 70},
new StockTransaction {Id = 3, Purchase = true, Quantity = 8, Resolved = 8, Price = 25},
new StockTransaction {Id = 4, Purchase = true, Quantity = 7, Resolved = 5, Price = 77},
new StockTransaction {Id = 5, Purchase = true, Quantity = 1, Resolved = 1, Price = 23},
new StockTransaction {Id = 6, Purchase = true, Quantity = 3, Resolved = 0, Price = 14},
new StockTransaction {Id = 7, Purchase = true, Quantity = 2, Resolved = 0, Price = 17},
};
And I would like to get the value of the last 7 quantities, which in this case gives 176 ((2 x 17) + (3 x 14) + (1 x 23) + (1 x 77)). (How) can this be done? Every help and hint is more then appreciated...
You can use
var last3=list.OrderByDescending(x=>x.CreatedDate).Take(3).Select(x=>x.Quantity * x.Price).Sum();
var requiredSum=last3+list.Where(x=>x.id==4).Select(x=>x.Price).FirstOrDefault();
Although you have tagged your question with LINQ, do it with a normal loop:
private static decimal SumOfLastTransaction(IEnumerable<StockTransaction> stockTransactions, int max)
{
decimal result = 0;
int sum = 0;
foreach(var stockTransaction in stockTransactions.OrderByDescending(x => x.Id))
{
if(sum + stockTransaction.Quantity <= max)
{
result += stockTransaction.Quantity * stockTransaction.Price;
sum += stockTransaction.Quantity;
}
else
{
result += (max - sum) * stockTransaction.Price;
return result;
}
}
return result;
}
You loop over the last items in your list. You sum the amount of transactions and check whether it is smaller than your maximum. As long as they are smaller, you can add the amount of transactions. If not, you check how much is missing until your maximum is reached and substract this from your quantity.
Online demo: https://dotnetfiddle.net/9oM4pj
I have 2 date time picker. assume that in my database already saved the data needed
//Status //Date //Time //name
----------------------------------------------
Check In 1/8/2016 12:30:36pm ali
Lunch 1/8/2016 2:40:36pm ali
Check In 1/8/2016 3:40:36pm ali
Check Out 1/8/2016 6:40:36pm ali
As i don't want to calculate the lunch time. as i want to calculate what my employee work for that day
6:40:36 PM - 12:30:36pm = 6 //total hours worked include lunch
So i have to subtract the Lunch - Checkk in which take 1 hours
6 - (3:40:36 PM - 2:40:36 PM) = 5 hours //total hours that worked
What kind of logic should i have to do this?
I already know all type of SQL clause to pick from my database. But i need a way that can calculate this easier without implementing huge lines of codes.
This isn't particularly robust and could probably do with some refactoring, but might give you a starting point to extend for your own logic.
Your states as an enum:
public enum Status
{
CheckIn,
CheckOut,
Lunch
}
Turn your data into a list of these:
public class EmployeeStatusChange
{
public int EmployeeId { get; set; }
public DateTime DateTime { get; set; }
public Status Status { get; set; }
}
And then use this extension method:
public static double CalculateHours(this List<EmployeeStatusChange> changes)
{
changes = changes.OrderBy(x => x.DateTime).ToList();
double hours = 0;
var start = DateTime.MinValue;
var lunch = DateTime.MinValue;
var checkedOut = false;
foreach (var change in changes)
{
// exclude anything before the first check in, or if we have already checked out
if ((start == DateTime.MinValue && change.Status != Status.CheckIn) || checkedOut)
{
continue;
}
// Set the start time
if (start == DateTime.MinValue && change.Status == Status.CheckIn)
{
start = change.DateTime;
continue;
}
switch (change.Status)
{
case Status.CheckIn:
if (lunch == DateTime.MinValue)
{
continue;
}
start = change.DateTime;
continue;
case Status.Lunch:
lunch = change.DateTime;
hours += (change.DateTime - start).TotalHours;
break;
case Status.CheckOut:
checkedOut = true;
hours += (change.DateTime - start).TotalHours;
break;
}
}
return hours;
}
This will return 6.5:
var items = new List<EmployeeStatusChange>();
items.Add(new EmployeeStatusChange { EmployeeId = 1, Status = Status.CheckIn, DateTime = new DateTime(2015, 1, 1, 9, 0, 0) });
items.Add(new EmployeeStatusChange { EmployeeId = 1, Status = Status.Lunch, DateTime = new DateTime(2015, 1, 1, 10, 30, 0) });
items.Add(new EmployeeStatusChange { EmployeeId = 1, Status = Status.CheckIn, DateTime = new DateTime(2015, 1, 1, 11, 0, 0) });
items.Add(new EmployeeStatusChange { EmployeeId = 1, Status = Status.Lunch, DateTime = new DateTime(2015, 1, 1, 12, 0, 0) });
items.Add(new EmployeeStatusChange { EmployeeId = 1, Status = Status.CheckIn, DateTime = new DateTime(2015, 1, 1, 13, 0, 0) });
items.Add(new EmployeeStatusChange { EmployeeId = 1, Status = Status.CheckOut, DateTime = new DateTime(2015, 1, 1, 17, 0, 0) });
items.CalculateHours();
If you have a collection of:
public class TestObj
{
public DateTime Key { get; set; }
public int MyProperty { get; set; }
public int MyProperty2 { get; set; }
}
The collection might have a count of 20,000 TestObj objects. I need to query using linq on the Key (DateTime) from DateTime t1 to DateTime t2.
I also want to insert and delete from the collection.
I see that the SortedList has an Add and a Remove method. Is SortedList efficient way to handle this problem? I am thinking that a List<T> would have to traverse through the entire list to be sure of getting all objects with t1 and t2.
You should not worry too much about performance of lists. 20,000 is nothing. If you have a list with billions of elements, then it might be more of a problem, but even then, optimization causes more problems than it solves.
If you want to make sure it doesn't matter, you can test it yourself:
var l = new SortedList<DateTime, TestObj>();
var key = new DateTime(1995, 1, 1);
for (var i = 0; i < 20000; i++)
{
var o = new TestObj();
o.Key = key;
key = key.AddDays(1);
l.Add(o.Key, o);
}
var sw = new Stopwatch();
var date1 = new DateTime(1995, 5, 5);
var date2 = new DateTime(2010, 5, 5);
sw.Start();
var between = l.Where(x => x.Key >= date1 && x.Key <= date2).ToArray();
Console.WriteLine(between.Count());
sw.Stop();
Console.WriteLine(sw.Elapsed);
Output is:
5480 (number of elements filtered out) and
00:00:00.0142387 (you wouldn't even notice)
I have some data which is queried from a database, but if there are no records for a particular financial year, it does not get returned in my list. I need all financial years returned, but the actual information (GrossEx, GST, GrossInc) should all be null.
public class FinancialData
{
public string FinancialYear { get; set; }
public string Type { get; set; }
public decimal? GrossEx { get; set; }
public decimal? GST { get; set; }
public decimal? GrossInc { get; set; }
}
How do I efficiently add FinancialYears, from 2004 to 2015 (with Gross, GST and GrossInc being null), to my list if they do not exist?
List<FinancialData> financialDataBucket = new List<FinancialData>();
FinancialData entry1 = new FinancialData { FinancialYear = "2012 - 2013", GrossEx = 1, GrossInc = 1, GST = 1 };
FinancialData entry2 = new FinancialData { FinancialYear = "2013 - 2014", GrossEx = 1, GrossInc = 1, GST = 1 };
financialDataBucket.Add(entry1);
financialDataBucket.Add(entry2);
I've tried doing a Union with a linq comparer, but for some reason it didn't work, and I can't figure out why? Is this the best way to solve my problem?
var merged = allFinancialYearsData.Union(financialDataBucket, new EqComparer());
public class EqComparer : IEqualityComparer<FinancialData>
{
public bool Equals( FinancialData x, FinancialData y )
{
return x.FinancialYear == y.FinancialYear;
}
public int GetHashCode( FinancialData obj )
{
return obj.GetHashCode ();
}
}
EDIT:
So, I'm thinking about either creating 10 different financial year objects and adding them to a list, OR
var currentYear = DateTime.Now.Year + (DateTime.Now.Month < 7 ? 0 : 1);
var earliestYear = 2005;
for (int i = earliestYear; i <= currentYear; i++) {
//Instantiate FinancialData here.....
financialDataItem.FinancialYear = (i-1) + " - " + i
}
Thanks!
You have to perform a group join. In code snippet below, allRecords create dummy FinancialData for "2005 - 2006" to "2014 - 2015".
var allRecords = Enumerable.Range(0, 10).Select(x => new FinancialData { FinancialYear = string.Format("{0} - {1}", 2005 + x, 2005 + x + 1) });
var myRecords = new[] { new FinancialData { FinancialYear = "2012 - 2013", GrossEx = 1, GrossInc = 1, GST = 1 }, new FinancialData { FinancialYear = "2013 - 2014", GrossEx = 1, GrossInc = 1, GST = 1 } };
var result = allRecords.GroupJoin(myRecords
, x => x.FinancialYear, y => y.FinancialYear, (x, y) => y.FirstOrDefault(u => u.FinancialYear == x.FinancialYear) ?? x);
I have a generic list of names whose number depends on what the user selects. I want to arrange those names in a range of times of the day that the user input before.
For example: we have 3 names, the user input 08:00-17:00. From 08:00-17:00 we have 9 hours -> 9/3=3, so arrange the names in a 3 hours templates.
This is how it looks in code:
List<string> myList = new List<string>();
myList.Add(Name1);
myList.Add(Name2);
myList.Add(Name3);
Console.WriteLine("Enter hours range: ");
Console.ReadLine(); //in this case the user input is 08:00-17:00
if (myList.Count == 3)
{
Console.WriteLine("8-11 " + myList.ElementAt(0)); //Name1
Console.WriteLine("11-14 " + myList.ElementAt(1)); //Name2
Console.WriteLine("14-17 " + myList.ElementAt(2)); //Name3
}
The problem comes, when there are more than 3 names and the hours are halved (lets say 17:30).
Any ideas on how to do this kind of thing?
You could do something like
private static void CalculateTimeIntervals()
{
var startTime = new DateTime(2014, 8, 15, 8, 0, 0);
var endTime = new DateTime(2014, 8, 15, 17, 0, 0);
var lengthOfTime = endTime - startTime;
var numberOfPeople = 4;
var dividedLengthOfTime = lengthOfTime.Ticks / numberOfPeople;
var people = new List<Person>();
for (int i = 1; i <= numberOfPeople; i++)
{
people.Add(
new Person
{
Id = i,
StartTime = startTime.AddTicks((dividedLengthOfTime * i) - dividedLengthOfTime),
EndTime = startTime.AddTicks(dividedLengthOfTime * i)
});
}
}
where Person looks like
public class Person
{
public int Id { get; set; }
public DateTime StartTime { get; set; }
public DateTime EndTime { get; set; }
}