If you have a collection of:
public class TestObj
{
public DateTime Key { get; set; }
public int MyProperty { get; set; }
public int MyProperty2 { get; set; }
}
The collection might have a count of 20,000 TestObj objects. I need to query using linq on the Key (DateTime) from DateTime t1 to DateTime t2.
I also want to insert and delete from the collection.
I see that the SortedList has an Add and a Remove method. Is SortedList efficient way to handle this problem? I am thinking that a List<T> would have to traverse through the entire list to be sure of getting all objects with t1 and t2.
You should not worry too much about performance of lists. 20,000 is nothing. If you have a list with billions of elements, then it might be more of a problem, but even then, optimization causes more problems than it solves.
If you want to make sure it doesn't matter, you can test it yourself:
var l = new SortedList<DateTime, TestObj>();
var key = new DateTime(1995, 1, 1);
for (var i = 0; i < 20000; i++)
{
var o = new TestObj();
o.Key = key;
key = key.AddDays(1);
l.Add(o.Key, o);
}
var sw = new Stopwatch();
var date1 = new DateTime(1995, 5, 5);
var date2 = new DateTime(2010, 5, 5);
sw.Start();
var between = l.Where(x => x.Key >= date1 && x.Key <= date2).ToArray();
Console.WriteLine(between.Count());
sw.Stop();
Console.WriteLine(sw.Elapsed);
Output is:
5480 (number of elements filtered out) and
00:00:00.0142387 (you wouldn't even notice)
Related
I count months and years from a given date to the present date
and from this list I have to subtract the months that were returned to me in the sql (linq) query.
I try to use "Except" on the results, but gives me an error in the picture below
var list = _ecpContext.Akceptacje_UnionAll_V
.Where(f => f.ADLogin == user)
.Select(f => new
{
Miesiac= f.Miesiac, //month
Rok= f.Rok // year
})
.ToList();
//-------------------------------------------------------------------
DateTime employmentDate = _ecpContext.Ustawienia.FirstOrDefault(x => x.UstLogin == user).EmploymentDate;
int employmentYear = employmentDate.Year;
int employmentMonth = employmentDate.Month;
DateTime now = DateTime.Now;
int currentYear = now.Year;
int currentMonth = now.Month;
var newList = Array.Empty<object>().Select(x => new { Month = 1, Year = 1 }).ToList();
for (var i = employmentYear; i <= currentYear; i++)
{
for (var x = employmentMonth; x <= currentMonth; x++)
{
newList.Add(new { Month = x, Year = i });
}
}
//-------------------------------------------------------------------
// i try
IEnumerable<DatesOfShortages> listMissingDates = list.Except(newList);
public class DatesOfShortages
{
public int Year { get; set; }
public int Month { get; set; }
}
new error
The Except method is a method which produces the set difference of two sequences so you need to invoke it.
IEnumerable<DatesOfShortages> listMissingDates = newList.Except(list);
You can't have one list A full of anonymous types, and another list B full of Tuples, and run a.Except(b) on them
Make a list of anonymous types instead of tuples:
var newList = Array.Empty<object>().Select(x => new { Month = 1, Year = 1 }).ToList();
for (var i = employmentYear; i <= currentYear; i++)
{
for (var x = employmentMonth; x <= currentMonth; x++)
{
newList.Add(new{ Month = x, Year = i});
}
}
For newList I suppose something like new [] { list.ElementAtOrDefault(-1) }.ToList(); would work too.. Whatever trick you feel like pulling to get a list of ATs!
I'm working on list sorting in webjob.
It is working fine when i used c# List. But to increase performance i'm saving my data into Redis cache in the form of list.
my final target is only take out most recent last 5 minutes data.
working c# code -
public class MyObject
{
public uint InstrumentID { get; set; }
public decimal Close { get; set; }
public decimal High { get; set; }
public decimal Low { get; set; }
public decimal Open { get; set; }
public DateTime TimeStamp { get; set; }
public uint Volume { get; set; }
public DateTime Created { get; } = DateTime.Now;
public DateTime Ttl { get; } = DateTime.Now.AddMinutes(5);
public DateTime? Persisted { get; set; }
public bool IsDead => DateTime.Now > Ttl;
public bool IsPersisted => Persisted.HasValue;
public bool TimeToPersist => IsPersisted == false && DateTime.Now > Created.AddMinutes(5);
public DateTime GetStartOfPeriodByMins(int numMinutes)
{
int oldMinutes = TimeStamp.Minute;
int newMinutes = (oldMinutes / numMinutes) * numMinutes;
DateTime startOfPeriod = new DateTime(TimeStamp.Year, TimeStamp.Month, TimeStamp.Day, TimeStamp.Hour, newMinutes, 0);
return startOfPeriod;
}
}
var inputList = new SortedSet<MyObject>(new MyObjectComparer());
inputList.Add(new MyObject() { TimeStamp = DateTime.Now, Open = 9, High = 12, Low = 8, Close = 11, InstrumentID = 2526 });
Thread.Sleep(10000);
inputList.Add(new MyObject() { TimeStamp = DateTime.Now, Open = 9, High = 12, Low = 8, Close = 11, InstrumentID = 2526 });
Thread.Sleep(10000);
inputList.Add(new MyObject() { TimeStamp = DateTime.Now, Open = 9, High = 12, Low = 8, Close = 11, InstrumentID = 2526 });
Thread.Sleep(50000);
inputList.Add(new MyObject() { TimeStamp = DateTime.Now, Open = 9, High = 12, Low = 8, Close = 11, InstrumentID = 2526 });
var resultSet = inputList
.GroupBy(i => i.GetStartOfPeriodByMins(5))
.Select(gr =>
new
{
StartOfPeriod = gr.Key,
Min = gr.Min(item => item.Open),
Max = gr.Max(item => item.Open),
Open = gr.OrderBy(item => item.TimeStamp).First().Open,
Close = gr.OrderBy(item => item.TimeStamp).Last().Open
});
Now same records i'm continuously inserting into redis cache. and while tried to take last 5 minutes data I was thiking to use same GetStartOfPeriodByMins concept but it needs a list of MyObject class and redis return RedisValue[].
Redis code - using StackExchange.Redis package
var cache = RedisConnectorHelper.Connection.GetDatabase();
//int i = 0;
for (int i = 0; i < 10000; i++)
{
var tickDataHis = new MyObject()
{
InstrumentID = 2526,
Close = 14 + i,
High = 16 + i,
Low = 11 + i,
Open = 12 + i,
TimeStamp = DateTime.Now,
Volume = 11111
};
// insert into redis
cache.ListRightPush("tickData", JsonConvert.SerializeObject(tickDataHis));
Thread.Sleep(3000);
}
var inputList = cache.ListRange("tickData");
or is there any other way to get latest 5 minutes data from redis
cache?
I used Redis to store timeseries data in the past. In order to optimize data retrieval, I used a sorted set (in that case there were more than one, but the concept is the same) where the score was the unix timestamp of when the data were recorded, and I serialized my data using Newtonsoft.Json library.
The code was something like this:
var myData = new MyObject() { SomeProperty = "my text" };
long dataTimestamp = DateTimeOffset.Now.ToUnixTimeMilliseconds();
string serializedData = Newtonsoft.Json.JsonConvert.Convert(myData);
redisDb.SortedSetAdd("mySortedSet", dataTimestamp, serializedData);
Doing in this way, if you want to retrieve only the data of the last 5 minutes, you can directly filter the data loaded from Redis using SortedSetRangeByScore method and passing "now - 5 minutes" as starting score, so that you can deserialize only what you need (which is of course less expensive than deserializing the whole list):
var minutesToTake = 5;
long startingTime = DateTimeOffset.Now.AddMinutes(-minutesToTake).ToUnixTimeMilliseconds();
RedisValue[] redisData = redisDb.SortedSetRangeByScore("mySortedSet", startingTime);
After that you can easily deserialize your data with the help of linq (consider that RedisValue implements operator to string conversion):
MyObject[] myData = redisData.Select(d => JsonConvert.DeserializeObject<MyObject>(d)).ToArray();
Edit: I didn't use the package suggested by #Hasan Emrah Süngü. Maybe it is more efficient, I'm just explaining what I did at that time.
Edit 2: redisDb is my StackExchange.Redis.IDatabase instance.
Edit 3: Useful reference links:
data persistence on Redis: https://redis.io/topics/data-types-intro#redis-expires-keys-with-limited-time-to-live
Redis sorted sets: https://redis.io/topics/data-types-intro#redis-sorted-sets
sorted sets for timeseries: https://redislabs.com/redis-best-practices/time-series/sorted-set-time-series/
IEnumerable<IGrouping<long, MyClass>> datas = list.GroupBy(x => x.PropertyXYOfMyClass);
// get all items from each group
foreach (var grouping in datas)
{
long groupKey = groupingByMyClass.Key;
//iterating through values
foreach (var item in groupingByMyClass)
{
long key = item.PropertyIntOfClassA;
string property = item.PropertyA;
}
}
Each group contains some items, wow to get values from first item of each group?
UPDATE
void Extract()
{
List<DataHolder> data = new List<DataHolder>();
List<DateTime> randomTimes = new List<DateTime>();
Random r = new Random();
DateTime d = new DateTime(2019, 9, 19, 7, 0, 0);
for (int i = 0; i < 100; i++)
{
DataHolder dh = new DataHolder();
TimeSpan t = TimeSpan.FromSeconds(r.Next(0, 14400));
dh.OID = i;
dh.Value = r.Next(50);
dh.Snapshottime = d.Add(t);
data.Add(dh);
}
data.OrderBy(o => o.Snapshottime).ToList();
List<DataHolder> SortedList = data.OrderBy(o => o.Snapshottime).ToList();
TimeSpan interval = new TimeSpan(0, 15, 0);
var result = SortedList.GroupBy(x => x.Snapshottime.Ticks / interval.Ticks) .OrderBy(x => x.Key);
}
public class DataHolder
{
public int OID { get; set; }
public double Value { get; set; }
public DateTime Snapshottime { get; set; }
}
Here from result i need to take first item from each group.
try this:
var finalResult = result.Select(gpr=>grp.First());
or if you want the earliest/Latest/etc you could order by first:
var finalResult = result.Select(gpr=>grp.OrderBy(x=>x.SnapShotTime).First());
You've already done the heavy lifting. Make a simple loop over the result:
var result = SortedList.GroupBy(x => x.Snapshottime.Ticks / interval.Ticks) .OrderBy(x => x.Key);
var resultList = new List<DataHolder>();
foreach(var group in result)
{
resultList.Add(group.First());
}
I hope this helps.
I have 2 Lists each of different objects. Each list contains a date element. What I am trying to do is pull items from each list in sequence and do something.
Object1
{
string description
date updateDate
int value
}
Object2
{
string description
date updateDate
string descritpion2
}
IE
List<object1>
object1.date = 10/1/2017
object1.date = 9/3/2017
List<object2>
object2.date = 10/15/2017
object2.date = 9/1/2017
I want to process these in order so i would do List 2 object 9/1, List 1 object 9/2, List 1 object 9/3, List 2 object 10/5
How can one achieve this?
How about this?
var list1 = new List<Object1>();
var list2 = new List<Object2>();
var newOrderedByDateCollection = list1
.Select(i => new TempClass(i, i.updateDate))
.Concat(list2
.Select(j => new TempClass(j, j.updateDate)))
.OrderBy(tmp => tmp.Date)
.Select(tmp => tmp.OrigItem);
//This could be replaced by a tuple like Tuple<object, DateTime> but I thought this would come across clearer
public class TempClass
{
public TempClass(object origItem, DateTime date)
{
OrigItem = origItem;
Date = date;
}
public object OrigItem { get; set; }
public DateTime Date { get; set; }
}
You now have a ordered list of type object. Which I can't see a way of getting around, So as you iterate through that list, you'll need to cast each object appropriately back by doing a switch and some pattern matching
Edit: for comepleteness here is the tuple version (I think its probably the best way to do it)
var newOrderedByDateCollection = list1
.Select(i => new Tuple<object,DateTime>(i, i.updateDate))
.Concat(list2
.Select(j => new Tuple<object, DateTime>(j, j.updateDate)))
.OrderBy(tmp => tmp.Item2)
.Select(tmp => tmp.Item1);
If you want to keep type safety (avoid object) and don't mind sorting the lists to new lists, you can do a loop with both indexes:
var l1count = l1.Count;
var l2count = l2.Count;
var ocount = l1count + l2count;
var l1o = l1.OrderBy(o => o.updateDate).ToList();
var l2o = l2.OrderBy(o => o.updateDate).ToList();
for (int j1 = 0, j2 = 0; j1 + j2 < ocount;) {
if (j1 < l1count && (l1o[j1].updateDate <= l2o[j2].updateDate || j2 >= l2count)) {
// process l1o[j1]
++j1;
}
else {
// process l2o[j2]
++j2;
}
}
I have a scenario where I have a list of objects with a datetime field in the object. I'm trying to find out if there is a way to use LINQ to group the list by sequential datetimes and return a subset of the list with the sequential datetimes as a range.
public virtual IList<LineItem> LineItems { get; set; }
...
public class LineItem
{
public virtual string Status { get; set; }
public virtual DateTime TransDate { get; set; }
...
}
So If I had 6 LineItems with Status = P for all and
TransDate = { 8/1/2011 , 8/2/2011 , 8/3/2011 , 8/5/2011 , 8/6/2011 , 8/9/2011 }
respectively, I'd like to return the following list:
{ (P, 8/1/2011-8/3/2011) , (P,8/5/2011-8/6/2011) , (P,8/9/2011) }
Any thoughts? I can do this with iterating through the list manually and checking the TransDate to see if it's sequential, I am just looking for a more elegant (preferably LINQ) way of doing it. Thanks!
I would use a helper method like this:
private static IEnumerable<ICollection<T>> PartitionByPredicate<T>(
this IEnumerable<T> seq, Func<T, T, bool> split)
{
var buffer = new List<T>();
foreach (var x in seq)
{
if (buffer.Any() && split(buffer.Last(), x))
{
yield return buffer;
buffer = new List<T>();
}
buffer.Add(x);
}
if (buffer.Any())
yield return buffer;
}
And then:
var sorted = LineItems.OrderBy(i => i.TransDate);
var split = sorted.PartitionByPredicate(
(x, y) => (y.TransDate.Date - x.TransDate.Date).TotalDays > 1)
(edit: cleaned it up slightly, my first version was silly.)
I suggest you go with an iterator block implementation, as suggested by #mquander.
But here's a fun, pure LINQ solution that will work (albeit inefficiently), assuming the dates are distinct and chronological:
var groups = from item in LineItems
let startDate = item.TransDate
group item by LineItems.Select(lineItem => lineItem.TransDate)
.SkipWhile(endDate => endDate < startDate)
.TakeWhile((endDate, index) =>
startDate.AddDays(index) == endDate)
.Last();
//If required:
var groupsAsLists = groups.Select(g => g.ToList()).ToList();
This works by choosing the last sequential date in any date-sequence as the key for that sequence.
I don't think this is very elegant, but its LINQ and it works :)
var list = new[] {
new DateTime(2011, 1, 1),
new DateTime(2011, 1, 2),
new DateTime(2011, 1, 3),
new DateTime(2011, 1, 5),
new DateTime(2011, 1, 6),
new DateTime(2011, 1, 8),
new DateTime(2011, 1, 10),
};
var ordered = list.OrderBy(d => d);
var accum = ordered.Aggregate(new Dictionary<DateTime, List<DateTime>>(), (dic, val) => {
if (!dic.Any())
{
dic.Add(val, new List<DateTime> { val });
}
else
{
if ((val - dic[dic.Keys.Last()].Last()).Days <= 1)
dic[dic.Keys.Last()].Add(val);
else
dic.Add(val, new List<DateTime> { val });
}
return dic;
});
In result accum will have 4 groups: 1-3, 5-6, 8 and 10.