We Keep Coding

Filter, merge, sort and page data from multiple sources

At the moment I'm retrieving data from the DB through a method that retrieves an IQueryable<T1>, filtering, sorting and then paging it (all these on the DB basically), before returning the result to the UI for display in a paged table.
I need to integrate results from another DB, and paging seems to be the main issue.
models are similar but not identical (same fields, different names, will need to map to a generic domain model before returning);
joining at the DB level is not possible;
there are ~1000 records at the moment between both DBs (added during
the past 18 months), and likely to grow at mostly the same (slow)
pace;
results always need to be sorted by 1-2 fields (date-wise).
I'm currently torn between these 2 solutions:
Retrieve all data from both sources, merge, sort and then cache them; then simply filter and page on said cache when receiving requests - but I need to invalidate the cache when the collection is modified (which I can);
Filter data on each source (again, at the DB level), then retrieve, merge, sort & page them, before returning.
I'm looking to find a decent algorithm performance-wise. The ideal solution would probably be a combination between them (caching + filtering at the DB level), but I haven't wrapped my head around that at the moment.

I think you can use the following algorithm. Suppose your page size is 10, then for page 0:
Get 10 results from database A, filtered and sorted at db level.
Get 10 results from database B, filtered and sorted at db level (in parallel with the above query)
Combine those two results to get 10 records in the correct sort order. So you have 20 records sorted, but take only first 10 of them and display in UI
Then for page 1:
Notice how many items from database A and B you used to display in UI at previous step. For example, you used 2 items from database A and 8 items from database B.
Get 10 results from database A, filtered and sorted, but starting at position 2 (skip 2), because those two you already have shown in UI.
Get 10 results from database B, filtered and sorted, but starting at position 8 (skip 8).
Merge the same way as above to get 10 records from 20. Suppose now you used 5 item from A and 5 items from B. Now, in total, you have shown 7 items from A and 13 items from B. Use those numbers for the next step.
This will not allow to (easily) skip pages, but as I understand that is not a requirement.
The perfomance should be effectively the same as when you are querying single database, because queries to A and B can be done in parallel.

I've created something here, I will come back with explications if needed.
I'm not sure my algorithm works correctly for all edge cases, it cover all of the cases what I had in mind, but you never know. I'll leave the code here for your pleasure, I will answer and explain what is done there if you need that, leave a comment.
And perform multiple tests with list of items with large gaps between the values.
using System;
using System.Collections.Generic;
using System.Linq;
namespace ConsoleApplication1
{
class Program
{
//each time when this objects are accessed, consider as a database call
private static IQueryable<model1> dbsetModel_1;
private static IQueryable<model2> dbsetModel_2;
private static void InitDBSets()
{
var rnd = new Random();
List<model1> dbsetModel1 = new List<model1>();
List<model2> dbsetModel2 = new List<model2>();
for (int i = 1; i < 300; i++)
{
if (i % 2 == 0)
{
dbsetModel1.Add(new model1() { Id = i, OrderNumber = rnd.Next(1, 10), Name = "Test " + i.ToString() });
}
else
{
dbsetModel2.Add(new model2() { Id2 = i, OrderNumber2 = rnd.Next(1, 10), Name2 = "Test " + i.ToString() });
}
}
dbsetModel_1 = dbsetModel1.AsQueryable();
dbsetModel_2 = dbsetModel2.AsQueryable();
}
public static void Main()
{
//generate sort of db data
InitDBSets();
//test
var result2 = GetPage(new PagingFilter() { Page = 5, Limit = 10 });
var result3 = GetPage(new PagingFilter() { Page = 6, Limit = 10 });
var result5 = GetPage(new PagingFilter() { Page = 7, Limit = 10 });
var result6 = GetPage(new PagingFilter() { Page = 8, Limit = 10 });
var result7 = GetPage(new PagingFilter() { Page = 4, Limit = 20 });
var result8 = GetPage(new PagingFilter() { Page = 200, Limit = 10 });
}
private static PagedList<Item> GetPage(PagingFilter filter)
{
int pos = 0;
//load only start pages intervals margins from both database
//this part need to be transformed in a stored procedure on db one, skip, take to return interval start value for each frame
var framesBordersModel1 = new List<Item>();
dbsetModel_1.OrderBy(x => x.Id).ThenBy(z => z.OrderNumber).ToList().ForEach(i => {
pos++;
if (pos - 1 == 0)
{
framesBordersModel1.Add(new Item() { criteria1 = i.Id, criteria2 = i.OrderNumber, model = i });
}
else if ((pos - 1) % filter.Limit == 0)
{
framesBordersModel1.Add(new Item() { criteria1 = i.Id, criteria2 = i.OrderNumber, model = i });
}
});
pos = 0;
//this part need to be transformed in a stored procedure on db two, skip, take to return interval start value for each frame
var framesBordersModel2 = new List<Item>();
dbsetModel_2.OrderBy(x => x.Id2).ThenBy(z => z.OrderNumber2).ToList().ForEach(i => {
pos++;
if (pos - 1 == 0)
{
framesBordersModel2.Add(new Item() { criteria1 = i.Id2, criteria2 = i.OrderNumber2, model = i });
}
else if ((pos -1) % filter.Limit == 0)
{
framesBordersModel2.Add(new Item() { criteria1 = i.Id2, criteria2 = i.OrderNumber2, model = i });
}
});
//decide where is the position of your cursor based on start margins
//int mainCursor = 0;
int cursor1 = 0;
int cursor2 = 0;
//filter pages start from 1, filter.Page cannot be 0, if indeed you have page 0 change a lil' bit he logic
if (framesBordersModel1.Count + framesBordersModel2.Count < filter.Page) throw new Exception("Out of range");
while ( cursor1 + cursor2 < filter.Page -1)
{
if (framesBordersModel1[cursor1].criteria1 < framesBordersModel2[cursor2].criteria1)
{
cursor1++;
}
else if (framesBordersModel1[cursor1].criteria1 > framesBordersModel2[cursor2].criteria1)
{
cursor2++;
}
//you should't get here case main key sound't be duplicate, annyhow
else
{
if (framesBordersModel1[cursor1].criteria2 < framesBordersModel2[cursor2].criteria2)
{
cursor1++;
}
else
{
cursor2++;
}
}
//mainCursor++;
}
//magic starts
//inpar skipable
int skipEndResult = 0;
List<Item> dbFramesMerged = new List<Item>();
if ((cursor1 + cursor2) %2 == 0)
{
dbFramesMerged.AddRange(
dbsetModel_1.OrderBy(x => x.Id)
.ThenBy(z => z.OrderNumber)
.Skip(cursor1*filter.Limit)
.Take(filter.Limit)
.Select(x => new Item() {criteria1 = x.Id, criteria2 = x.OrderNumber, model = x})
.ToList()); //consider as db call EF or Stored Procedure
dbFramesMerged.AddRange(
dbsetModel_2.OrderBy(x => x.Id2)
.ThenBy(z => z.OrderNumber2)
.Skip(cursor2*filter.Limit)
.Take(filter.Limit)
.Select(x => new Item() {criteria1 = x.Id2, criteria2 = x.OrderNumber2, model = x})
.ToList());
; //consider as db call EF or Stored Procedure
}
else
{
skipEndResult = filter.Limit;
if (cursor1 > cursor2)
{
cursor1--;
}
else
{
cursor2--;
}
dbFramesMerged.AddRange(
dbsetModel_1.OrderBy(x => x.Id)
.ThenBy(z => z.OrderNumber)
.Skip(cursor1 * filter.Limit)
.Take(filter.Limit)
.Select(x => new Item() { criteria1 = x.Id, criteria2 = x.OrderNumber, model = x })
.ToList()); //consider as db call EF or Stored Procedure
dbFramesMerged.AddRange(
dbsetModel_2.OrderBy(x => x.Id2)
.ThenBy(z => z.OrderNumber2)
.Skip(cursor2 * filter.Limit)
.Take(filter.Limit)
.Select(x => new Item() { criteria1 = x.Id2, criteria2 = x.OrderNumber2, model = x })
.ToList());
}
IQueryable<Item> qItems = dbFramesMerged.AsQueryable();
PagedList<Item> result = new PagedList<Item>();
result.AddRange(qItems.OrderBy(x => x.criteria1).ThenBy(z => z.criteria2).Skip(skipEndResult).Take(filter.Limit).ToList());
//here again you need db cals to get total count
result.Total = dbsetModel_1.Count() + dbsetModel_2.Count();
result.Limit = filter.Limit;
result.Page = filter.Page;
return result;
}
}
public class PagingFilter
{
public int Limit { get; set; }
public int Page { get; set; }
}
public class PagedList<T> : List<T>
{
public int Total { get; set; }
public int? Page { get; set; }
public int? Limit { get; set; }
}
public class Item : Criteria
{
public object model { get; set; }
}
public class Criteria
{
public int criteria1 { get; set; }
public int criteria2 { get; set; }
//more criterias if you need to order
}
public class model1
{
public int Id { get; set; }
public int OrderNumber { get; set; }
public string Name { get; set; }
}
public class model2
{
public int Id2 { get; set; }
public int OrderNumber2 { get; set; }
public string Name2 { get; set; }
}
}

Select random record from table with further conditions in C# code

I'm coding little process where I can select random record from table, where field with checkbox/boolean is also unchecked.
The moral of the story is to select random record, work with it and set checkbox/boolean field == Checked.
I have approach this work by selecting random record from Count() method, I got Integer in return and I can choose random number from that pool. But I also need to filter the query to select only records who have the checkbox/boolean field unchecked, but I haven't managed in the following code to go so far.
I have used Lambda expression as follows but maybe better way can exists.
public class Program
{
static void Main(string[] args)
{
DatabaseModelContainer mc = new DatabaseModelContainer();
MyTable myTable = new MyTable();
Random rnd = new Random();
int index = rnd.Next(myTable.Count);
string selectedRec = myTable.MyField.OrderBy(r => myTable.MyField).Take(index).ToString();
//Selected record to be shown at the screen, but will not appear (only reference point is shown)
Console.WriteLine(selectedRec);
//Selected random value is displayed here and that indeed worked, I get random value there, but I can't use the id to select record above based on that id.
Console.WriteLine(index.ToString());
Console.ReadLine();
}
}
Code above does NOT show who to choose random record WHERE CheckboxField == Checked, so that code needs to be added as well.
What approach is best?

Assuming your MyTable is a sequence, the correct way of doing it should be :
var randomRecord = myTable
.Where(x => x.CheckboxField)
.ToList()
.OrderBy(x => rnd.Next())
.FirstOrDefault();
Full Example:
See it here: https://dotnetfiddle.net/B4JLU0
using System;
using System.Collections.Generic;
using System.Linq;
public class Program
{
class TableRow
{
public int I {get;set;}
public bool CheckboxField {get;set;}
TableRow(int i)
{
this.I = i;
this.CheckboxField = (i % 3 == 0); //in our demo, assume that some are already checked, some aren't
}
public static IEnumerable<TableRow> GenerateTableRows(int howMany)
{
for (int i=0; i<howMany; i++)
{
yield return new TableRow(i);
}
}
}
public static void Main()
{
new Program();
Console.WriteLine("Done");
}
Program()
{
ICollection<TableRow> myTable = new List<TableRow>(TableRow.GenerateTableRows(12));
while (myTable.Count(x => !x.CheckboxField) > 0) //keep looping until we've processed all records
{
var randomRecord = GetRandomRecord(myTable);
Console.WriteLine(randomRecord == null ? "No Matching Rows Exist" : randomRecord.I.ToString());
}
}
TableRow GetRandomRecord(ICollection<TableRow> myTable)
{
var rnd = new Random(); //we could optionally provide this as a parameter too to avoid recreating each time
var randomRecord = myTable
.Where(x => !x.CheckboxField)
.ToList()
.OrderBy(x => rnd.Next())
.FirstOrDefault();
if(randomRecord != null) //avoid issue if all items are checked
{
randomRecord.CheckboxField = true; //mark this record as having been processed
}
return randomRecord;
}
}

Should be able to do this:
var randomNumberGenerator = new Random();
var randomNumber = randomNumberGenerator.Next(myTable.Count(r => r.CheckboxField)) - 1;
myTable
.Where(r => r.CheckboxField)
.Skip(randomNumber)
.Select(r => r.DesiredColumn)
.FirstOrDefault();

Creating a two-dimensional array

I am trying to create a two dimensional array and I am getting so confused. I was told by a coworker that I need to create a dictionary within a dictionary for the array list but he couldn't stick around to help me.
I have been able to create the first array that lists the the programs like this
+ project 1
+ project 2
+ project 3
+ project 4
The code that accomplishes this task is below-
var PGList = from x in db.month_mapping
where x.PG_SUB_PROGRAM == SP
select x;
//select x.PG.Distinct().ToArray();
var PGRow = PGList.Select(x => new { x.PG }).Distinct().ToArray();
So that takes care of my vertical array and now I need to add my horizontal array so that I can see the total amount spent in each accounting period. So the final output would look like this but without the dashes of course.
+ program 1-------100---200---300---400---500---600---700---800---900---1000---1100---1200
+ program 2-------100---200---300---400---500---600---700---800---900---1000---1100---1200
+ program 3-------100---200---300---400---500---600---700---800---900---1000---1100---1200
+ program 4-------100---200---300---400---500---600---700---800---900---1000---1100---1200
I have tried to use a foreach to cycle through the accounting periods but it doesn't work. I think I might be on the right track and I was hoping SO could provide some guidance or at the very least a tutorial for me to follow. I have posted the code that I written so far on the second array below. I am using C# and MVC 3. You might notice that their is no dictionary within a dictionary. If my coworker is correct how would I do something like that, I took a look at this question using dictionary as a key in other dictionary but I don't understand how I would use it in this situation.
Dictionary<string, double[]> MonthRow = new Dictionary<string, double[]>();
double[] PGContent = new double[12];
string lastPG = null;
foreach (var item in PGRow)
{
if (lastPG != item.PG)
{
PGContent = new double[12];
}
var MonthList = from x in db.Month_Web
where x.PG == PG
group x by new { x.ACCOUNTING_PERIOD, x.PG, x.Amount } into pggroup
select new { accounting_period = pggroup.Key.ACCOUNTING_PERIOD, amount = pggroup.Sum(x => x.Amount) };
foreach (var P in MonthList)
{
int accounting_period = int.Parse(P.accounting_period) - 1;
PAContent[accounting_period] = (double)P.amount;
MonthRow[item.PG] = PGContent;
lastPG = item.PG;
}
I hope I have clearly explained my issue, please feel free to ask for any clarification needed as I need to solve this problem and will be checking back often. Thanks for your help!

hope this helps.
// sample data
var data = new Dictionary<string, List<int>>();
data.Add("program-1", new List<int>() { 100, 110, 130 });
data.Add("program-2", new List<int>() { 200, 210, 230 });
data.Add("brogram-3", new List<int>() { 300, 310, 330 });
// query data
var newData = (from x in data
where x.Key.Contains("pro")
select x).ToDictionary(v => v.Key, v=>v.Value);
// display selected data
foreach (var kv in newData)
{
Console.Write(kv.Key);
foreach (var val in kv.Value)
{
Console.Write(" ");
Console.Write(val.ToString());
}
Console.WriteLine();
}
output is:
program-1 100 110 130
program-2 200 210 230

Don't try to use anonymous types or LINQ projection to create new data types, especially if you're a beginner, you will just get confused. If you want a specialized data type, define one; e.g.:
public class Account
{
public string Name { get; private set; }
public decimal[] MonthAmount { get; private set; }
readonly int maxMonths = 12;
public Account(string name, ICollection<decimal> monthAmounts)
{
if (name == null)
throw new ArgumentNullException("name");
if (monthAmounts == null)
throw new ArgumentNullException("monthAmounts");
if (monthAmounts.Count > maxMonths)
throw new ArgumentOutOfRangeException(string.Format(" monthAmounts must be <= {0}", maxMonths));
this.Name = name;
this.MonthAmount = new decimal[maxMonths];
int i = 0;
foreach (decimal d in monthAmounts)
{
this.MonthAmount[i] = d;
i++;
}
}
}
Use instances of this type directly, you do not have to convert them to arrays, dictionaries, lists, or anything else:
var accountPeriods = new List<Account>();
accountPeriods.Add(new Account("program-1", new decimal[] { 1, 2, 3, 4 }));
You can use LINQ or whatever to query or alter instances of your new type:
foreach (Account a in accountPeriods)
foreach (decimal d in a.MonthAmount)
DoSomethingWith(d);
That should be enough to get you started.

I want to thank #Ray Cheng and #Dour High Arch for their help but I have figured out another way to accomplish this task and I wanted to post my code so that the next person that is having the same trouble can figure out their problem faster.
Above I split my code into more managable sections to explain my problem as clearly as I could and the code below has all those parts combined so you can see the big picture. This code returns an array that contains the program and the amounts for every month.
public virtual ActionResult getAjaxPGs(string SP = null)
{
if (SP != null)
{
var PGList = from x in db.month_mapping
where x.PG_SUB_PROGRAM == SP
select x;
var PGRow = PGList.Select(x => new { x.PG }).Distinct().ToArray();
float[] PGContent = new float[12];
Dictionary<string,float[]> MonthRow = new Dictionary<string, float[]>();
foreach (var item in PGRow)
{
PGContent = new float[12];
var MonthList = from x in db.month_Web
where x.PG == item.PG
group x by new { x.ACCOUNTING_PERIOD, x.PG, x.Amount } into pggroup
select new { accounting_period = pggroup.Key.ACCOUNTING_PERIOD, amount = pggroup.Sum(x => x.Amount) };
foreach (var mon in MonthList)
{
int accounting_period = int.Parse(mon.accounting_period) - 1;
PGContent[accounting_period] = (float)mon.amount/1000000;
}
MonthRow[item.PG] = PGContent;
}
return Json(MonthRow, JsonRequestBehavior.AllowGet);
}
return View();
}
This code worked great for me since I am pulling from a Linq to SQL query instead of adding data directly into the code. My problems stemmed from mainly putting the data pulls outside of the foreach loops so it only pulled 1 piece of data from the SQL instead of all twelve months. I hope this helps some one else who is trying to pull data in from SQL data sources into multidimensional arrays.

Entity Random Select from DB C# MVC

Try to find the solution but i cant.
So problem is next one. I have the EDM model of database. I have a class with functions to get data from DB.
Like this:
public IQueryable<photos> FindUserPhotos(string userlogin)
{
return from m in db.photos
where m.userlogin == userlogin
select m;
}
How to get the Random 10 lines from DB?

I always use this method for get custom entity OrderBy(x => Guid.NewGuid())
public photos Find10RandomUserPhotos(string userlogin)
{
return db.photos.Where(x => x.userlogin == userlogin).OrderBy(x => Guid.NewGuid()).Take(10).ToList();
}

Following Random row from Linq to Sql
public photos FindRandomUserPhoto(string userlogin)
{
var qry = FindUserPhotos(userlogin);
int count = qry.Count();
int index = new Random().Next(count);
return qry.Skip(index).FirstOrDefault();
}
public Array<photos> Find10RandomUserPhotos(string userlogin)
{
var result = New Array<photos>;
for (i = 0; i < 10; i++) {
result.add(FindRandomUserPhoto(userlogin));
}
return result
}

LINQ, SelectMany with multiple possible outcomes

I have a situation where I have lists of objects that have to be merged. Each object in the list will have a property that explains how it should be treated in the merger. So assume the following..
enum Cascade {
Full,
Unique,
Right,
Left
}
class Note {
int Id { get; set; }
Cascade Cascade { get; set; }
// lots of other data.
}
var list1 = new List<Note>{
new Note {
Id = 1,
Cascade.Full,
// data
},
new Note {
Id = 2,
Cascade.Right,
// data
}
};
var list2 = new List<Note>{
new Note {
Id = 1,
Cascade.Left,
// data
}
};
var list3 = new List<Note>{
new Note {
Id = 1,
Cascade.Unique,
// data similar to list1.Note[0]
}
}
So then, I'll have a method ...
Composite(this IList<IList<Note>> notes){
return new List<Note> {
notes.SelectMany(g => g).Where(g => g.Cascade == Cascade.All).ToList()
// Here is the problem...
.SelectMany(g => g).Where(g => g.Cascade == Cascade.Right)
.Select( // I want to do a _LastOrDefault_ )
// continuing for the other cascades.
}
}
This is where I get lost. I need to do multiple SelectMany statements, but I don't know how to. But this is the expected behavior.
Cascade.Full
The Note will be in the final collection no matter what.
Cascade.Unique
The Note will be in the final collection one time, ignoring any duplicates.
Cascade.Left
The Note will be in the final collection, First instances superseding subsequent instances. (So then, Notes 1, 2, 3 are identical. Note 1 gets pushed through)
Cascade.Right
The Note will be in the final collection, Last instance superseding duplicates. (So Notes 1, 2, 3 are identical. Note 3 gets pushed trough)

I think you should decompose the problem in smaller parts. For example, you can implement the cascade rules for an individual list in a seperate extension method. Here's my untested take at it:
public static IEnumerable<Note> ApplyCascades(this IEnumerable<Note> notes)
{
var uniques = new HashSet<Note>();
Note rightToYield = null;
foreach (var n in notes)
{
bool leftYielded = false;
if (n.Cascade == Cascade.All) yield return n;
if (n.Cascade == Cascade.Left && !leftYielded)
{
yield return n;
leftYielded = true;
}
if (n.Cascade == Cascade.Right)
{
rightToYield = n;
}
if (n.Cascade == Cascade.Unique && !uniques.Contains(n))
{
yield return n;
uniques.Add(n);
}
}
if (rightToYield != null) yield return rightToYield;
}
}
This method would allow to implement the original extension method something like this:
List<Note> Composite(IList<IList<Note>> notes)
{
var result = from list in notes
from note in list.ApplyCascades()
select note;
return result.ToList();
}