We Keep Coding

Filter, merge, sort and page data from multiple sources

At the moment I'm retrieving data from the DB through a method that retrieves an IQueryable<T1>, filtering, sorting and then paging it (all these on the DB basically), before returning the result to the UI for display in a paged table.
I need to integrate results from another DB, and paging seems to be the main issue.
models are similar but not identical (same fields, different names, will need to map to a generic domain model before returning);
joining at the DB level is not possible;
there are ~1000 records at the moment between both DBs (added during
the past 18 months), and likely to grow at mostly the same (slow)
pace;
results always need to be sorted by 1-2 fields (date-wise).
I'm currently torn between these 2 solutions:
Retrieve all data from both sources, merge, sort and then cache them; then simply filter and page on said cache when receiving requests - but I need to invalidate the cache when the collection is modified (which I can);
Filter data on each source (again, at the DB level), then retrieve, merge, sort & page them, before returning.
I'm looking to find a decent algorithm performance-wise. The ideal solution would probably be a combination between them (caching + filtering at the DB level), but I haven't wrapped my head around that at the moment.

I think you can use the following algorithm. Suppose your page size is 10, then for page 0:
Get 10 results from database A, filtered and sorted at db level.
Get 10 results from database B, filtered and sorted at db level (in parallel with the above query)
Combine those two results to get 10 records in the correct sort order. So you have 20 records sorted, but take only first 10 of them and display in UI
Then for page 1:
Notice how many items from database A and B you used to display in UI at previous step. For example, you used 2 items from database A and 8 items from database B.
Get 10 results from database A, filtered and sorted, but starting at position 2 (skip 2), because those two you already have shown in UI.
Get 10 results from database B, filtered and sorted, but starting at position 8 (skip 8).
Merge the same way as above to get 10 records from 20. Suppose now you used 5 item from A and 5 items from B. Now, in total, you have shown 7 items from A and 13 items from B. Use those numbers for the next step.
This will not allow to (easily) skip pages, but as I understand that is not a requirement.
The perfomance should be effectively the same as when you are querying single database, because queries to A and B can be done in parallel.

I've created something here, I will come back with explications if needed.
I'm not sure my algorithm works correctly for all edge cases, it cover all of the cases what I had in mind, but you never know. I'll leave the code here for your pleasure, I will answer and explain what is done there if you need that, leave a comment.
And perform multiple tests with list of items with large gaps between the values.
using System;
using System.Collections.Generic;
using System.Linq;
namespace ConsoleApplication1
{
class Program
{
//each time when this objects are accessed, consider as a database call
private static IQueryable<model1> dbsetModel_1;
private static IQueryable<model2> dbsetModel_2;
private static void InitDBSets()
{
var rnd = new Random();
List<model1> dbsetModel1 = new List<model1>();
List<model2> dbsetModel2 = new List<model2>();
for (int i = 1; i < 300; i++)
{
if (i % 2 == 0)
{
dbsetModel1.Add(new model1() { Id = i, OrderNumber = rnd.Next(1, 10), Name = "Test " + i.ToString() });
}
else
{
dbsetModel2.Add(new model2() { Id2 = i, OrderNumber2 = rnd.Next(1, 10), Name2 = "Test " + i.ToString() });
}
}
dbsetModel_1 = dbsetModel1.AsQueryable();
dbsetModel_2 = dbsetModel2.AsQueryable();
}
public static void Main()
{
//generate sort of db data
InitDBSets();
//test
var result2 = GetPage(new PagingFilter() { Page = 5, Limit = 10 });
var result3 = GetPage(new PagingFilter() { Page = 6, Limit = 10 });
var result5 = GetPage(new PagingFilter() { Page = 7, Limit = 10 });
var result6 = GetPage(new PagingFilter() { Page = 8, Limit = 10 });
var result7 = GetPage(new PagingFilter() { Page = 4, Limit = 20 });
var result8 = GetPage(new PagingFilter() { Page = 200, Limit = 10 });
}
private static PagedList<Item> GetPage(PagingFilter filter)
{
int pos = 0;
//load only start pages intervals margins from both database
//this part need to be transformed in a stored procedure on db one, skip, take to return interval start value for each frame
var framesBordersModel1 = new List<Item>();
dbsetModel_1.OrderBy(x => x.Id).ThenBy(z => z.OrderNumber).ToList().ForEach(i => {
pos++;
if (pos - 1 == 0)
{
framesBordersModel1.Add(new Item() { criteria1 = i.Id, criteria2 = i.OrderNumber, model = i });
}
else if ((pos - 1) % filter.Limit == 0)
{
framesBordersModel1.Add(new Item() { criteria1 = i.Id, criteria2 = i.OrderNumber, model = i });
}
});
pos = 0;
//this part need to be transformed in a stored procedure on db two, skip, take to return interval start value for each frame
var framesBordersModel2 = new List<Item>();
dbsetModel_2.OrderBy(x => x.Id2).ThenBy(z => z.OrderNumber2).ToList().ForEach(i => {
pos++;
if (pos - 1 == 0)
{
framesBordersModel2.Add(new Item() { criteria1 = i.Id2, criteria2 = i.OrderNumber2, model = i });
}
else if ((pos -1) % filter.Limit == 0)
{
framesBordersModel2.Add(new Item() { criteria1 = i.Id2, criteria2 = i.OrderNumber2, model = i });
}
});
//decide where is the position of your cursor based on start margins
//int mainCursor = 0;
int cursor1 = 0;
int cursor2 = 0;
//filter pages start from 1, filter.Page cannot be 0, if indeed you have page 0 change a lil' bit he logic
if (framesBordersModel1.Count + framesBordersModel2.Count < filter.Page) throw new Exception("Out of range");
while ( cursor1 + cursor2 < filter.Page -1)
{
if (framesBordersModel1[cursor1].criteria1 < framesBordersModel2[cursor2].criteria1)
{
cursor1++;
}
else if (framesBordersModel1[cursor1].criteria1 > framesBordersModel2[cursor2].criteria1)
{
cursor2++;
}
//you should't get here case main key sound't be duplicate, annyhow
else
{
if (framesBordersModel1[cursor1].criteria2 < framesBordersModel2[cursor2].criteria2)
{
cursor1++;
}
else
{
cursor2++;
}
}
//mainCursor++;
}
//magic starts
//inpar skipable
int skipEndResult = 0;
List<Item> dbFramesMerged = new List<Item>();
if ((cursor1 + cursor2) %2 == 0)
{
dbFramesMerged.AddRange(
dbsetModel_1.OrderBy(x => x.Id)
.ThenBy(z => z.OrderNumber)
.Skip(cursor1*filter.Limit)
.Take(filter.Limit)
.Select(x => new Item() {criteria1 = x.Id, criteria2 = x.OrderNumber, model = x})
.ToList()); //consider as db call EF or Stored Procedure
dbFramesMerged.AddRange(
dbsetModel_2.OrderBy(x => x.Id2)
.ThenBy(z => z.OrderNumber2)
.Skip(cursor2*filter.Limit)
.Take(filter.Limit)
.Select(x => new Item() {criteria1 = x.Id2, criteria2 = x.OrderNumber2, model = x})
.ToList());
; //consider as db call EF or Stored Procedure
}
else
{
skipEndResult = filter.Limit;
if (cursor1 > cursor2)
{
cursor1--;
}
else
{
cursor2--;
}
dbFramesMerged.AddRange(
dbsetModel_1.OrderBy(x => x.Id)
.ThenBy(z => z.OrderNumber)
.Skip(cursor1 * filter.Limit)
.Take(filter.Limit)
.Select(x => new Item() { criteria1 = x.Id, criteria2 = x.OrderNumber, model = x })
.ToList()); //consider as db call EF or Stored Procedure
dbFramesMerged.AddRange(
dbsetModel_2.OrderBy(x => x.Id2)
.ThenBy(z => z.OrderNumber2)
.Skip(cursor2 * filter.Limit)
.Take(filter.Limit)
.Select(x => new Item() { criteria1 = x.Id2, criteria2 = x.OrderNumber2, model = x })
.ToList());
}
IQueryable<Item> qItems = dbFramesMerged.AsQueryable();
PagedList<Item> result = new PagedList<Item>();
result.AddRange(qItems.OrderBy(x => x.criteria1).ThenBy(z => z.criteria2).Skip(skipEndResult).Take(filter.Limit).ToList());
//here again you need db cals to get total count
result.Total = dbsetModel_1.Count() + dbsetModel_2.Count();
result.Limit = filter.Limit;
result.Page = filter.Page;
return result;
}
}
public class PagingFilter
{
public int Limit { get; set; }
public int Page { get; set; }
}
public class PagedList<T> : List<T>
{
public int Total { get; set; }
public int? Page { get; set; }
public int? Limit { get; set; }
}
public class Item : Criteria
{
public object model { get; set; }
}
public class Criteria
{
public int criteria1 { get; set; }
public int criteria2 { get; set; }
//more criterias if you need to order
}
public class model1
{
public int Id { get; set; }
public int OrderNumber { get; set; }
public string Name { get; set; }
}
public class model2
{
public int Id2 { get; set; }
public int OrderNumber2 { get; set; }
public string Name2 { get; set; }
}
}

How to compute rank of IEnumerable<T> and store it in type T

I want to compute rank of element in an IEnumerable list and assign it to the member. But below code works only when called 1st time. 2nd time call starts from last rank value. So instead of output 012 and 012, I'm getting 012 and 345
class MyClass
{
public string Name { get; set; }
public int Rank { get; set; }
}
public void SecondTimeRankEvaluvate()
{
MyClass[] myArray = new MyClass[]
{
new MyClass() { Name = "Foo" },
new MyClass() { Name = "Bar" },
new MyClass() { Name = "Baz" }
};
int r = 0;
var first = myArray.Select(s => { s.Rank = r++; return s; });
foreach (var item in first)
{
Console.Write(item.Rank);
}
// Prints 012
Console.WriteLine("");
foreach (var item in first)
{
Console.Write(item.Rank);
}
// Prints 345
}
I understand that the variable r is being captured (closure) and reused when called 2nd time. I don't want that behavior. Is there any clean way to compute rank and assign it?
Also r variable (in actual code) isn't in the same scope where foreach loop is present. It is in a function which returns var first

var first = myArray.Select((s, i) => { s.Rank = i; return s; });

LINQ uses lazy evaluation and runs the Select part every time you use myArray.
You can force evaluation to happen only once by storing the result in a List.
Change
var first = myArray.Select(s => { s.Rank = r++; return s; });
to
var first = myArray.Select(s => { s.Rank = r++; return s; }).ToList();
Another way would be to join myArray with a new sequence using Zip every time, like this
var first = myArray.Zip(Enumerable.Range(0, int.MaxValue), (s, r) =>
{
s.Rank = r;
return s;
});

You shouldn't use LINQ if you're not querying the collection.
To update each item in an array, use a for loop:
for (int i = 0; i < myArray.Length; i++)
{
myArray[i].Rank = i;
}
To update each item in an enumerable, use a foreach loop:
int r = 0;
foreach (var item in myArray)
{
item.Rank = r++;
}

remove list-items with Linq when List.property = myValue

I have the following code:
List<ProductGroupProductData> productGroupProductDataList = FillMyList();
string[] excludeProductIDs = { "871236", "283462", "897264" };
int count = productGroupProductDataList.Count;
for (int removeItemIndex = 0; removeItemIndex < count; removeItemIndex++)
{
if (excludeProductIDs.Contains(productGroupProductDataList[removeItemIndex].ProductId))
{
productGroupProductDataList.RemoveAt(removeItemIndex);
count--;
}
}
Now i want to do the same with linq. Is there any way for this?
The second thing would be, to edit each List-Item property with linq.

you could use RemoveAll.
Example:
//create a list of 5 products with ids from 1 to 5
List<Product> products = Enumerable.Range(1,5)
.Select(c => new Product(c, c.ToString()))
.ToList();
//remove products 1, 2, 3
products.RemoveAll(p => p.id <=3);
where
// our product class
public sealed class Product {
public int id {get;private set;}
public string name {get; private set;}
public Product(int id, string name)
{
this.id=id;
this.name=name;
}
}

Firstly corrected version of your current code that won't skip entries
List<ProductGroupProductData> productGroupProductDataList = FillMyList();
string[] excludeProductIDs = { "871236", "283462", "897264" };
int count = productGroupProductDataList.Count;
for (int removeItemIndex = 0; removeItemIndex < count; removeItemIndex++)
{
while (removeItemIndex < count && excludeProductIDs.Contains(productGroupProductDataList[removeItemIndex].ProductId)) {
productGroupProductDataList.RemoveAt(removeItemIndex);
count--;
}
}
}
This linq code would do the job.
List<ProductGroupProductData> productGroupProductDataList = FillMyList();
string[] excludeProductIDs = { "871236", "283462", "897264" };
productGroupProductDataList=productGroupProductDataList.Where(x=>!excludedProductIDs.Contains(x.ProductId)).ToList();
Alternatively using paolo's answer of remove all the last line would be would be
productGroupProductDataList.RemoveAll(p=>excludedProductIDs.Contains(p=>p.ProductId));
What you mean by "The second thing would be, to edit each List-Item property with linq."?

As per your comment here's a version that creates a set that excludes the elements rather than removing them from the original list.
var newSet = from p in productGroupProductDataList
where !excludeProductIDs.Contains(p.ProductId))
select p;
The type of newSet is IEnumerable if you need (I)List you can easily get that:
var newList = newSet.ToList();

C# Linq question

I have a text file in which I am storing entries for an address book.
The layout is like so:
Name:
Contact:
Product:
Quantity:
I have written some linq code to grab the name plus the next four lines, for a search by name feature.
I also want to be able to search by contact.
The challenge is to match the contact info, grab the next 3 lines, and also grab the line prior to the match.
That way if Search By Contact is used, the full list of info will be returned.
private void buttonSearch_Click(object sender, EventArgs e)
{
string[] lines = File.ReadAllLines("C:/AddressBook/Customers.txt");
string name = textBoxSearchName.Text;
string contact = textBoxContact.Text;
if (name == "" && contact == "")
{
return;
}
var byName = from line in lines
where line.Contains(name)
select lines.SkipWhile(f => f != line).Take(4);
//var byContact = from line in lines
// where line.Contains(name)
// select lines.SkipWhile(f => f != name).Take(4);
if (name != "")
{
foreach (var item in byName)
foreach (var line in item) { listBox2.Items.Add(line); }
listBox2.Items.Add("");
}
//if (contact != "")
//{
// foreach (var item in byContact)
// foreach (var line in item) { listBox2.Items.Add(line); }
//listBox2.Items.Add("");
}
}

Firstly i would recommend changing your data storage approach if you can.
Secondly i would recommend reading the file into an object, something like this:
public class Contact
{
public string Name {get; set;}
public string Contact {get; set;}
public string Product {get; set;}
public int Quantity {get; set;}
}
...
public IEnumerable<Contact> GetContacts()
{
//make this read line by line if it is big!
string[] lines = File.ReadAllLines("C:/AddressBook/Customers.txt");
for (int i=0;i<lines.length;i += 4)
{
//add error handling/validation!
yield return new Contact()
{
Name = lines[i],
Contact = lines[i+1],
Product = lines[i+2],
Quantity = int.Parse(lines[i+3]
};
}
}
private void buttonSearch_Click(object sender, EventArgs e)
{
...
var results = from c in GetContacts()
where c.Name == name ||
c.Contact == contact
select c;
...
}

See if this will work
var contactLinesList = lines.Where(l => l.Contains(name))
.Select((l, i) => lines.Skip(i - 1).Take(4)).ToList();
contactLinesList.ForEach(cl => listBox2.Items.Add(cl));

This is not the smallest code in earth but it shows how to do a couple of things. Although I don't recommend using it, because it is quite complex to understand. This is to be considered as a hobbyist, just learning code!!! I suggest you load the file in a well known structure, and do Linq on that... anyway... this is a C# Console Application that does what you proposed using Linq syntax, and one extension method:
using System;
using System.Collections.Generic;
using System.Linq;
namespace stackoverflow.com_questions_5826306_c_linq_question
{
public class Program
{
public static void Main()
{
string fileData = #"
Name: Name-1
Contact: Xpto
Product: Abc
Quantity: 12
Name: Name-2
Product: Xyz
Contact: Acme
Quantity: 16
Name: Name-3
Product: aammndh
Contact: YKAHHYTE
Quantity: 2
";
string[] lines = fileData.Replace("\r\n", "\n").Split('\n');
var result = Find(lines, "contact", "acme");
foreach (var item in result)
Console.WriteLine(item);
Console.WriteLine("");
Console.WriteLine("Press any key");
Console.ReadKey();
}
private static string[] Find(string[] lines, string searchField, string searchValue)
{
var result = from h4 in
from g4 in
from i in (0).To(lines.Length)
select ((from l in lines select l).Skip(i).Take(4))
where !g4.Contains("")
select g4
where h4.Any(
x => x.Split(new char[] { ':' }, 2)[0].Equals(searchField, StringComparison.OrdinalIgnoreCase)
&& x.Split(new char[] { ':' }, 2)[1].Trim().Equals(searchValue, StringComparison.OrdinalIgnoreCase))
select h4;
var list = result.FirstOrDefault();
return list.ToArray();
}
}
public static class NumberExtensions
{
public static IEnumerable<int> To(this int start, int end)
{
for (int it = start; it < end; it++)
yield return it;
}
}
}

If your text file is small enough, I'd recommend using regular expressions instead. This is exactly the sort of thing it's designed to do. Off the top of my head, the expression will look something like this:
(?im)^Name:(.*?)$ ^Contact:search_term$^Product:(.*?)$^Quantity:(.*?)$

How to delete an item from a generic list

I have a generic list
How do I remove an item?
EX:
Class Student
{
private number;
public Number
{
get( return number;)
set( number = value;)
}
private name;
public Name
{
get( return name;)
set( name = value;)
}
main()
{
static List<student> = new list<student>();
list.remove...???
}
}

Well, there is nothing to remove because your list is empty (you also didn't give it an identifier, so your code won't compile). You can use the Remove(T item) or RemoveAt(int index) to remove an object or the object at a specified index respectively (once it actually contains something).
Contrived code sample:
void Main(...)
{
var list = new List<Student>();
Student s = new Student(...);
list.Add(s);
list.Remove(s); //removes 's' if it is in the list based on the result of the .Equals method
list.RemoveAt(0); //removes the item at index 0, use the first example if possible/appropriate
}

From your comments I conclude that you read student name from input and you need to remove student with given name.
class Student {
public string Name { get; set; }
public int Number { get; set; }
public Student (string name, int number)
{
Name = name;
Number = number;
}
}
var students = new List<Student> {
new Student ("Daniel", 10),
new Student ("Mike", 20),
new Student ("Ashley", 42)
};
var nameToRemove = Console.ReadLine ();
students.RemoveAll (s => s.Name == nameToRemove); // remove by condition
Note that this will remove all students with given name.
If you need to remove the first student found by name, first use First method to find him, and then call Remove for the instance:
var firstMatch = students.First (s => s.Name == nameToRemove);
students.Remove (firstMatch);
If you want to ensure there is only one student with given name before removing him, use Single in a similar fashion:
var onlyMatch = students.Single (s => s.Name == nameToRemove);
students.Remove (onlyMatch);
Note that Single call fails if there is not exactly one item matching the predicate.

List<Student> students = new List<Student>();
students.Add(new Student {StudentId = 1, StudentName = "Bob",});
students.RemoveAt(0); //Removes the 1st element, will crash if there are no entries
OR to remove a known Student.
//Find a Single Student in the List.
Student s = students.Where(s => s.StudentId == myStudentId).Single();
//Remove that student from the list.
students.Remove(s);

Well, you didn't give your list a name, and some of your syntax is wonky.
void main()
{
static List<Student> studentList = new List<Student>();
}
// later
void deleteFromStudents(Student studentToDelete)
{
studentList.Remove(studentToDelete);
}
There are more remove functions detailed here: C# List Remove Methods

int count=queue.Count;
while(count>0)
{
HttpQueueItem item = queue[0];
/// If post succeeded..
if (snd.IsNotExceedsAcceptedLeadsPerDayLimit(item.DataSaleID, item.AcceptedLeadsPerDayLimit) && snd.PostRecord(item.DataSaleDetailID, item.PostString, item.duplicateCheckHours, item.Username, item.Password, item.successRegex))
{
if (item.WaitTime > 0)
Thread.Sleep(item.WaitTime);
queue.Remove(item);
}
///If Exceeds Accepted leads per day limit by DataSale..
else if (!snd.IsNotExceedsAcceptedLeadsPerDayLimit(item.DataSaleID, item.AcceptedLeadsPerDayLimit))
{
queue.RemoveAll(obj => obj.DataSaleID == item.DataSaleID);
}
/// If Post failed..
else //if (!snd.PostRecord(item.DataSaleDetailID, item.PostString, item.duplicateCheckHours, item.Username, item.Password, item.successRegex))
{
queue.Remove(item);
}
count = queue.Count;
}

To delete a row or record from generic list in grid view, just write this code-
List<Address> people = new List<Address>();
Address t = new Address();
people.RemoveAt(e.RowIndex);
grdShow.EditIndex = -1;
grdShow.DataSource = people;
grdShow.DataBind();

Maybe you can use a Dictionary<string, Student> instead of the List<Student>.
When you add a Student, add its ID or Name or whatever can uniquely identify it. Then you can simply call myStudents.Remove(myStudentId)

Try linq:
var _resultList = list.Where(a=>a.Name != nameToRemove).Select(a=>a);

I made a program that contains 7 cards, then shuffle and I hope to take in order to help them
class Program
{
static void Main(string[] args)
{
Random random = new Random();
var cards = new List<string>();
//CARDS VECRTOR
String[] listas = new String[] { "Card 1", "Card 2", "Card 3", "Card 4", "Card 5", "Card 6", "Card 7"};
for (int i = 0; i<= cards.Count; i++)
{
int number = random.Next(0, 7); //Random number 0--->7
for (int j = 0; j <=6; j++)
{
if (cards.Contains(listas[number])) // NO REPEAT SHUFFLE
{
number = random.Next(0, 7); //AGAIN RANDOM
}
else
{
cards.Add(listas[number]); //ADD CARD
}
}
}
Console.WriteLine(" LIST CARDS");
foreach (var card in cards)
{
Console.Write(card + " ,");
}
Console.WriteLine("Total Cards: "+cards.Count);
//REMOVE
for (int k = 0; k <=6; k++)
{
// salmons.RemoveAt(k);
Console.WriteLine("I take the card: "+cards.ElementAt(k));
cards.RemoveAt(k); //REMOVE CARD
cards.Insert(k,"Card Taken"); //REPLACE INDEX
foreach (var card in cards)
{
Console.Write(card + " " + "\n");
}
}
Console.Read(); //just pause
}
}