We Keep Coding

I have two large lists and I need get the diff between them

I have two large lists and I need get the diff between them.
The first list is from another system via webservice, the second list is from a database (destiny of data).
i will compare and get items from first list that not have in second list and insert in the database (second list source).
have another solution with best performance?
using List.Any(), the process take a lot of hours and not finish...
using for loop, the process take 10 hours or more.
Each list have 1.300.000 records
newItensForInsert = List1.Where(item1 => !List2.Any(item2 => item1.prop1 == item2.prop1 && item1.prop2 == item2.prop2)).ToList();
//or
for (int i = 0; i < List1.Count; i++)
{
if (!List2.Any(x => x.prop1 == List1[i].prop1 && x.prop2 == List1[i].prop2))
{
ListForInsert.Add(List1[i]);
}
}
//or
ListForInsert = List1.AsParallel().Except(List2.AsParallel(), IEqualityComparer).ToList();

You could use List.Except
List<object> webservice = new List<object>();
List<object> database = new List<object>();
IEnumerable<object> toPutIntoDatabase = webservice.Except(database);
database.AddRange(toPutIntoDatabase);
EDIT:
You can even use the new PLINQ (parallel LINQ) like this
IEnumerable<object> toPutIntoDatabase = webservice.AsParallel().Except(database.AsParallel());
EDIT:
Maybe you could use a Hashset to speed up lookups.
HashSet<object> databaseHash = new HashSet<object>(database);
foreach (var item in webservice)
{
if (databaseHash.Contains(item) == false)
{
database.Add(item);
}
{

If same data type then you can use List.Exists,
Else Better to go with inner join and emit
var newdata = from c in dblist
join p in list1 on c.Category equals p.Category into ps
from p in ps.DefaultIfEmpty()
it will select list if given data not present in dblist

HashSet<T> is optimized for executing this kind of set operations. In many cases it's worth the effort to create HashSets from Lists and do the set operation on the Hashsets. I demonstrated this with a little Linqpad program.
The program creates two lists containing 1,300,000 objects. It uses your method to get the difference (or better: attempted to used, because I ran out of patience). And it uses LINQ's Except and hashsets with ExceptWith, both with an IEqualityComparer. The program is listed below.
The result was:
Lists created: 00:00:00.9221369
Hashsets created: 00:00:00.1057532
Except: 00:00:00.2564191
ExceptWith: 00:00:00.0696830
So creating the HashSets and executing ExceptWith (together 0.18), beat Except (0.26s).
One caveat: creating HashSets may take too much memory since the large lists already take a fair amount of memory.
void Main()
{
var sw = Stopwatch.StartNew();
var amount = 1300000;
//amount = 50000;
var list1 = Enumerable.Range(0, amount).Select(i => new Demo(i)).ToList();
var list2 = Enumerable.Range(10, amount).Select(i => new Demo(i)).ToList();
sw.Stop();
sw.Elapsed.Dump("Lists created");
sw.Restart();
var hs1 = new HashSet<Demo>(list1, new DemoComparer());
var hs2 = new HashSet<Demo>(list2, new DemoComparer());
sw.Stop();
sw.Elapsed.Dump("Hashsets created");
sw.Restart();
// var list3 = list1.Where(item1 => !list2.Any(item2 => item1.ID == item2.ID)).ToList();
// sw.Stop();
// sw.Elapsed.Dump("Any");
// sw.Restart();
var list4 = list1.Except(list2, new DemoComparer()).ToList();
sw.Stop();
sw.Elapsed.Dump("Except");
sw.Restart();
hs1.ExceptWith(hs2);
sw.Stop();
sw.Elapsed.Dump("ExceptWith");
// list3.Count.Dump();
list4.Count.Dump();
hs1.Count.Dump();
}
// Define other methods and classes here
class Demo
{
public Demo(int id)
{
ID = id;
Name = id.ToString();
}
public int ID { get; set; }
public string Name { get; set; }
}
class DemoComparer : IEqualityComparer<Demo>
{
public bool Equals(Demo x, Demo y)
{
return (x == null && y == null)
|| (x != null && y != null) && x.ID.Equals(y.ID);
}
public int GetHashCode(Demo obj)
{
return obj.ID.GetHashCode();
}
}

Use List.Exists, it is better than List.Any Performance-wise

How i foreach loop in listitems values entity framework

I have 2 lists. One has 3 records like 1 , 2 , 3 and seconds list holds table records. All i wanna do add first list values to second list.
I hope helps.
foreach (var itemAgent in listofValues)
{
foreach (var item in formSorgu)
{
#region MyRegion
CrmDonusleri crmEkleme = new CrmDonusleri()
{
AradigiBolge = item.bolge,
AramaTarihi = Convert.ToDateTime(item.aratarihi),
Musno = item.musno,
GeriDonusYapildiMi = false,
AtanmaTarihi = DateTime.Now,
KanalAdi = item.kanal,
ProgramAdi = item.program,
AtananAgent = itemAgent
};
DbContext.CrmDonusleri.Add(crmEkleme);
#endregion
}
}
DbContext.SaveChanges();
listofValues hold 3 records and formSorgu hold 2000 records. listofValues as List One and formSorgu as List Two. And i want my final list like Picture below.
I don't think my code is right. Please show me the right way to write this query.
for (int i = formSorgu.Count + 1; i >= 0; i--)
{
foreach (var itemAgent in listofValues)
{
CrmDonusleri crmEkkle = new CrmDonusleri()
{
Musno = formSorgu.FirstOrDefault().musno,
AtananAgent = itemAgent
};
}
}
when i use this code it gets one record from formSorgu but add to listofValues 3 times i just want it to foreach one time and go out from foreach loop and carry on other for loop record.

actually, i love to share i am not like stackover mods. so i found an excellent answer on internet. you can find a link that equally selecting items from list and psuedo run.
solution link
that is a link i used. and i turn my code like this;
var randoAgent = new RandomPicker<string>(listofValues);
foreach (var itemSorgu in formSorgu)
{
var item = randoAgent.PickItem();
CrmDonusleri crmEkle = new CrmDonusleri()
{
Musno = itemSorgu.musno,
TelNo = itemSorgu.telno,
AtananAgent = item,
AradigiBolge = itemSorgu.bolge,
AramaTarihi = Convert.ToDateTime(itemSorgu.aratarihi),
AtanmaTarihi = DateTime.Now,
Ekleyen = itemSorgu.ekleyen,
GeriDonusYapildiMi = false,
KanalAdi = itemSorgu.kanal,
ProgramAdi = itemSorgu.program
};
DbContext.CrmDonusleri.Add(crmEkle);
}
DbContext.SaveChanges();
and works perfect.

Entity can't pass short (int16) values out

This is really holding up the process.
I have a db of marks stored as short, and want to extract a single entry out and create summary statistics on it. It seems simple but I seem to have to jump through a ton of hoops to get it so must be missing something basic.
Here is the basic method, and this works happily. however all i can do with it is pass it to the DataGridView.
private void MarksSummary(string StudentID)
{
int ID = Convert.ToInt32(StudentID);
//get the average of the marks using entity
using (var context = new collegestudentsEntities1())
{
var StudentMarks = (from m in context.Marks
where m.StudIDFK == ID
select new
{
m.Marks1,
m.marks2,
m.Marks3,
m.Marks4
});
dataGridView1.DataSource = StudentMarks.ToList();
Anything else, seems to be ridiculously long winded.
Eg: I can't do this
var Marklist = new List<Int16>();
StudentMarks.ToList().ForEach(m => Marklist.Add(m));
as I get "cannot convert from 'AnonymousType#1' to 'short'"
or this
Marklist = StudentMarks.ToList();
or this
double av = Marklist.Average();
Yet I can do a forEach which is silly on one row of data
foreach (var s in StudentMarks)
{
Marklist.Add(s.Marks1);
Marklist.Add(s.marks2);
Marklist.Add(s.Marks3);
Marklist.Add(s.Marks4);
}
and this works outputting happily
txtMarksOverFifty.Text = Marklist.Count(s => s > 50).ToString();
txtMarksFailed.Text = Marklist.Count(s => s < 50).ToString();
So what am I missing to get the values out of the query easily?
Thanks for your help :-)

Your foreach is trying to add an anonymous type
select new
{
m.Marks1,
m.marks2,
m.Marks3,
m.Marks4
} //...
To a List<Int16> so it's not surprising that fails. What it looks like you want to do with that is:
StudentMarks.ToList().ForEach(m => Marklist.AddRange(new [] { m.Marks1, m.marks2, m.Marks3, m.Marks4 }));
Edit: If you're just looking for a solution with less code you might try:
using (var context = new collegestudentsEntities1())
{
var StudentMarks = (from m in context.Marks
where m.StudIDFK == ID
select new[]
{
m.Marks1,
m.marks2,
m.Marks3,
m.Marks4
}).SelectMany(mark => mark).ToList();
}
or simply:
List<Int16> Marklist = context.Marks.Where(mark => mark.StudIDFK == ID)
.SelectMany(m => new [] { m.Marks1, m.marks2, m.Marks3, m.Marks4 })
.ToList();

Look at what you are creating here:
select new
{
m.Marks1,
m.marks2,
m.Marks3,
m.Marks4
});
This is an object that contains shorts.
StudentMarks.ToList().ForEach(m => Marklist.Add(m));
Here you are trying to add an object to a list of shorts. Try:
StudentMarks.ToList().ForEach(m => {
Marklist.Add(m.Mark1);
Marklist.Add(m.Mark2);
Marklist.Add(m.Mark3);
Marklist.Add(m.Mark4);
}
);

create an array of anonymous type

I am trying to get data from database for Google charts in my program. I would like to create an array of anonymous type (var) instead of repeating my code over and over again:
public JsonResult GetChartData(int sID, int regionID)
{
var testPathOne = from p in _rep.GetMetricsData().GetLHDb().page_loads
where p.t_3id == sID && p.test_path_id == 1
select new { time = p.time, created_at = p.created_at };
var testPathTwo = from p in _rep.GetMetricsData().GetLHDb().page_loads
where p.t_3id == sID && p.test_path_id == 2
select new { time = p.time, created_at = p.created_at };
var tOne = testPathOne.ToArray();
var tTwo = testPathTwo.ToArray();
var name = new { test1 = tOne, test2 = tTwo };
return Json(name);
}
i know that i will need a for loop so i can go through all the test path id's instead of hard coding them like this p.test_path_id == 1, but my question is how would i make this part dynamic var name = new { test1 = tOne, test2 = tTwo };
Edit:
I apologize, I would like to do something like this:
name is an array
for loop:
testPath = query
name.Add(testPath)
I hope that makes sense

The easiest solution in this particular case would be to just give a name to the class that is currently anonymous. While there are workarounds that you can use, when you need to start working really hard to use an anonymous type you probably shouldn't be using it. It's there to make certain tasks quicker and easier; if that isn't happening then you are likely better off with a real class.
That solution would look something like this:
//Please give me a real name
public class ClassToBeRenamed
{
public DateTime Time { get; set; }
public DateTime CreatedAt { get; set; }
}
List<ClassToBeRenamed[]> myList = new List<ClassToBeRenamed[]>();
for (int i = 0; i < 10; i++)
{
myList.Add((from p in _rep.GetMetricsData().GetLHDb().page_loads
where p.t_3id == sID && p.test_path_id == i
select new ClassToBeRenamed { Time = p.time, CreatedAt = p.created_at })
.ToArray());
}
Having said all of that, it's still possible.
var myList = new[]{
from p in _rep.GetMetricsData().GetLHDb().page_loads
where p.t_3id == sID && p.test_path_id == 1
select new { time = p.time, created_at = p.created_at }.ToArray()
}.ToList();
for (int i = 2; i < 10; i++)
{
myList.Add(from p in _rep.GetMetricsData().GetLHDb().page_loads
where p.t_3id == sID && p.test_path_id == i
select new { time = p.time, created_at = p.created_at }.ToArray()
);
}
var myArray = myList.ToArray();
If it's really, really important that you have an array, and not a list, then you could call ToArray on myList at the very end. It's important that you start out with a list, and only convert it to an array at the end because Arrays have a fixed size once they are created. You can mutate their contents, but you can't make them bigger or smaller. A List on the other hand, is designed to mutate it's size over time, so it can start out with 0 or 1 items and then add items over time, which is important for us in this particular context. (It's actually useful quite often, which is why it is frequently useful to use List over arrays.)

Instead of using LINQ use foreach loops. You are more limited with LINQ.
Also define your ArrayLists at the beginning and add to them as you go.
var test1 = new ArrayList();
var test2 = new ArrayList();
foreach(PageLoad p in _rep.GetMetricsData().GetLHDb().page_loads)
{
if(p.t_3id == sID)
{
var tr = new { time = p.time, created_at = p.created_at };
switch(p.test_path_id)
{
case 1: test1.Add(tr); break;
case 2: test2.Add(tr); break;
}
}
}
return Json(new { test1, test2, });
You do not need to define the names of properties anonymous types because they default to the variable names.

Strange array behavior

I observe a very strange behavior, maybe could you help me to see what happen.
Here the class:
public sealed class Sudoku
{
private SudokuCell[] _grid = new SudokuCell[81];
// ctor {}
private IEnumerable<SudokuCell> Grid
{
get { return _grid; }
}
private SudokuRow[] _rows;
public IEnumerable<SudokuRow> Rows
{
get
{
if (_rows == null)
{
_rows = new SudokuRow[9];
for (int i = 0, length = 9; i < length; i++)
{
_rows[i] = new SudokuRow(from cell in Grid
where cell.Row == i
select cell);
// Always print 9 (GOOD)
Trace.WriteLine("First Loop " + i + " : " + _rows[i].Cells.Count());
}
}
for (int i = 0; i < 9; i++)
{
// Always print 0 ! Huh !?
Trace.WriteLine("Second Loop " + i + " : " + _rows[i].Cells.Count());
}
return _rows;
}
}
}
public abstract class SudokuPart
{
public SudokuPart(IEnumerable<SudokuCell> cells)
{
Cells = cells;
}
public int Index
{ get; protected set; }
public IEnumerable<SudokuCell> Cells
{ get; protected set; }
}
public sealed class SudokuRow : SudokuPart
{
public SudokuRow(IEnumerable<SudokuCell> cells)
: base(cells)
{
base.Index = cells.First().Row;
}
}
Could anyone tell me why in the second loop it trace 0 instead of 9 !? I changed nothing between both loops !!!
Thanks...

This is the problem:
_rows[i] = new SudokuRow(from cell in Grid
where cell.Row == i
select cell);
That's capturing the loop variable (i)... within the loop, it has a sensible value, which is why you're seeing 9 matches.
However, when you count the matching values in the second loop, that single captured variable will have the value 9. Now no cell.Row has a value of 9, so you're not getting any matches. For more information on this, see Eric Lippert's great blog post, "Closing over the loop variable considered harmful."
Three fixes:
Capture a copy of the loop variable:
int copy = i;
_rows[i] = new SudokuRow(from cell in Grid
where cell.Row == copy
select cell);
Each iteration of the loop will get a separate copy.
Materialize the query in the loop:
_rows[i] = new SudokuRow((from cell in Grid
where cell.Row == i
select cell).ToList());
Or even:
_rows[i] = new SudokuRow(Grid.Where(cell => cell.Row == i).ToList());
Don't use LINQ at all! Why not just have an array of arrays to represent the grid? That's a much more natural approach, IMO.

I think Jon Skeet answer is great, but I just wanted to add a bit to it with an example of Deferred LINQ Queries. Once I saw this in action, it helped me understand a bit more about some of the nuances of this kind of code problem you ran into.
Try this code.
var numbers = new List<int> {1, 2, 3, 4, 5};
//Lets create an IEnumerable<int> with the values in our numbers list greater then 3.
var bignumbers = numbers.Where(n => n > 3);
//You may assume our variable bignumbers now contains the numbers 4 and 5
//now lets add another number to our original list that fits the criteria of our LINQ Where statement
numbers.Add(6);
foreach (var big in bignumbers) {
Console.WriteLine(big.ToString());
}
Our output from our foreach loop is going to be 4,5,6! This is because our query doesn't run until the foreach causes the enumeration of the items in our bignumbers variable.
Just something else to consider when your building lists within loops and your querying those lists outside of the loops. Your often going to get something other than what your expecting.