We Keep Coding

Best Way to compare 1 million List of object with another 1 million List of object in c#

i am differentiating 1 million list of object with another 1 million list of object.
i am using for , foreach but it takes too much of time to iterate those list.
can any one help me best way to do this
var SourceList = new List<object>(); //one million
var TargetList = new List<object>()); // one million
//getting data from database here
//SourceList with List of one million
//TargetList with List of one million
var DifferentList = new List<object>();
//ForEach
SourceList.ToList().ForEach(m =>
{
if (!TargetList.Any(s => s.Name == m.Name))
DifferentList.Add(m);
});
//for
for (int i = 0; i < SourceList .Count; i++)
{
if (!TargetList .Any(s => s == SourceList [i].Name))
DifferentList .Add(SourceList [i]);
}

I think it seems like a bad idea but IEnumerable magic will help you.
For starters, simplify your expression. It looks like this:
var result = sourceList.Where(s => targetList.Any(t => t.Equals(s)));
I recommend making a comparison in the Equals method:
public class CompareObject
{
public string prop { get; set; }
public new bool Equals(object o)
{
if (o.GetType() == typeof(CompareObject))
return this.prop == ((CompareObject)o).prop;
return this.GetHashCode() == o.GetHashCode();
}
}
Next add AsParallel. This can both speed up and slow down your program. In your case, you can add ...
var result = sourceList.AsParallel().Where(s => !targetList.Any(t => t.Equals(s)));
CPU 100% loaded if you try to list all at once like this:
var cnt = result.Count();
But it’s quite tolerable to work if you get the results in small portions.
result.Skip(10000).Take(10000).ToList();
Full code:
static Random random = new Random();
public class CompareObject
{
public string prop { get; private set; }
public CompareObject()
{
prop = random.Next(0, 100000).ToString();
}
public new bool Equals(object o)
{
if (o.GetType() == typeof(CompareObject))
return this.prop == ((CompareObject)o).prop;
return this.GetHashCode() == o.GetHashCode();
}
}
void Main()
{
var sourceList = new List<CompareObject>();
var targetList = new List<CompareObject>();
for (int i = 0; i < 10000000; i++)
{
sourceList.Add(new CompareObject());
targetList.Add(new CompareObject());
}
var stopWatch = new Stopwatch();
stopWatch.Start();
var result = sourceList.AsParallel().Where(s => !targetList.Any(t => t.Equals(s)));
var lr = result.Skip(10000).Take(10000).ToList();
stopWatch.Stop();
Console.WriteLine(stopWatch.Elapsed);
}
Update
I remembered what you can use Hashtable.Choos unique values from targetList and from sourceList next fill out the result whose values are not targetList.
Example:
static Random random = new Random();
public class CompareObject
{
public string prop { get; private set; }
public CompareObject()
{
prop = random.Next(0, 1000000).ToString();
}
public new int GetHashCode() {
return prop.GetHashCode();
}
}
void Main()
{
var sourceList = new List<CompareObject>();
var targetList = new List<CompareObject>();
for (int i = 0; i < 10000000; i++)
{
sourceList.Add(new CompareObject());
targetList.Add(new CompareObject());
}
var stopWatch = new Stopwatch();
stopWatch.Start();
var sourceHashtable = new Hashtable();
var targetHashtable = new Hashtable();
foreach (var element in targetList)
{
var hash = element.GetHashCode();
if (!targetHashtable.ContainsKey(hash))
targetHashtable.Add(element.GetHashCode(), element);
}
var result = new List<CompareObject>();
foreach (var element in sourceList)
{
var hash = element.GetHashCode();
if (!sourceHashtable.ContainsKey(hash))
{
sourceHashtable.Add(hash, element);
if(!targetHashtable.ContainsKey(hash)) {
result.Add(element);
}
}
}
stopWatch.Stop();
Console.WriteLine(stopWatch.Elapsed);
}

Scanning the target list to match the name is an O(n) operation, thus your loop is O(n^2). If you build a HashSet<string> of all the distinct names in the target list, you can check whether a name exists in the set in O(1) time using the Contains method.

//getting data from database here
You are getting the data out of a system that specializes in matching and sorting and filtering data, into your RAM that by default cannot yet do that task at all. And then you try to sort, filter and match yourself.
That will fail. No matter how hard you try, it is extremely unlikely that your computer with a single programmer working at a matching algorithm will outperform your specialized piece of hardware called a database server at the one operation this software is supposed to be really good at that was programmed by teams of experts and optimized for years.
You don't go into a fancy restaurant and ask them to give you huge bags of raw ingredients so you can throw them into a big bowl unpeeled and microwave them at home. No. You order a nice dish because it will be way better than anything you could do yourself.
The simple answer is: Do not do that. Do not take the raw data and rummage around in it for hours. Leave that job to the database. It's the one thing it's supposed to be good at. Use it's power. Write a query that will give you the result, don't get the raw data and then play database yourself.

Foreach performs a null check before each iteration, so using a standard for loop will provide slightly better performance that will be hard to beat.
If it is taking too long, can you break down the collection into smaller sets and/or process them in parallel?
Also you could look a PLinq (Parallel Linq) using .AsParallel()
Other areas to improve are the actual comparison logic that you are using, also how the data is stored in memory, depending on your problem, you may not have to load the entire object into memory for every iteration.
Please provide a code example so that we can assist further, when such large amounts of data are involved performance degredation is to be expected.
Again depending on the time that we are talking about here, you could upload the data into a database and use that for the comparison rather than trying to do it natively in C#, this type of solution is better suited to data sets that are already in a database or where the data changes much less frequently than the times you need to perform the comparison.

How to remove first occurence of element in List C# with LINQ?

I have the following list
List<string> listString = new List<string>() { "UserId", "VesselId", "AccountId", "VesselId" };
I would like to use a Linq operator which removes only the first occurrence of VesselId.

you can not change the original collection with LINQ, so the closest thing would be:
var index = listString.IndexOf("VesselId");
if(index>-1)
listString.RemoveAt(index);
EDIT 1:
According to research done by Igor Mesaros, I ended up implementing the logic which resides in the List.Remove method, so an even simpler solution would be:
listString.Remove("VesselId");

If you have a simple list of strings or some other primitive type, then you can just call:
listString.Remove("VesselId");
as mentioned by #Eyal Perry
If you have a huge none primitive list, this is the most efficient
class MyClass
{
string Name {get; set;}
}
var listString = new List<MyClass>() {/* Fill in with Data */};
var match = listString.FirstOrDefault(x => x.Name == "VesselId");
if(match != null)
listString.Remove(match);

If what you are looking to do is get a distinct list then you have an extension method for that listString.Distinct()

If you absolutely MUST use LINQ, you can use it to find the first instance of "VesselId" inside the Remove() method, like so:
listString.Remove((from a in listString
where a == "VesselId"
select a).First());

listString = listString.Union(new List<string>()).ToList();
OR
List<string> CopyString = new List<string>();
CopyString.AddRange(listString);
foreach (var item in CopyString)
{
var index = CopyString.IndexOf(item);
if (index >= 0 && listString.Count(cnt => cnt == item) > 1)
listString.RemoveAt(index);
}

Add a List to another List that is either null or not null

Consider the the piece of code below. I need to call the method CreateOrAddToLists() in a loop. First time the method is called the two lists casedata.Cases and casedata.Documents will be null, hence I can populate by assigning cases to casedata.Cases like this:
casedata.Cases = cases;
At any subsequent call to CreateOrAddToLists() lists casedata.Cases and casedata.Documents will not be null and I can add new data to the lists using AddRange():
casedata.Cases.AddRange(cases);
var casedata = new CaseData(); //contains lists to get populated
private void CreateOrAddToLists()
{
var cases = new List<Case>(); //gets data with from database
var documents = new List<Document>(); //gets data with from database
if (casedata.Cases == null)
{
casedata.Cases = cases;
}
else
{
casedata.Cases.AddRange(cases);
}
if (casedata.Documents == null)
{
casedata.Documents = documents;
}
else
{
casedata.Documents.AddRange(documents);
}
}
Is there a better or neater way to do a null-check before AddRange? Can I do it in on line of code?

In the constructor for CaseData instantiate the two list objects, then you'll be assured they won't be null and you can just use AddRange.
public CaseData()
{
Cases = new List<Case>();
Documents = new List<Document>();
}

It's more clear:
casedata.Cases = casedata.Cases ?? new List<Case>();
casedata.Cases.AddRange(cases);
casedata.Documents = casedata.Documents ?? new List<Document>();
casedata.Documents.AddRange(documents);

Add one list to another in C#

I'm trying to add one list to another . So I have this main list that I'm going to build. I loop through records and built a list and want this list to main list every time I loop. I'm trying to do this in C#.
I'm using following code. Add() function is not working.I'm getting syntax error.
IList<CgValDetail> cgValDetail = null;
//Get cgValDetails for each control
foreach (UiControlScreenMetaData tempUiControls in uiControls)
{
if (tempUiControls.CgValId == null)
{
continue;
}
IList<CgValDetail> tempCgValDetail = Retrieval<CgValDetail>.Search(new { CgValId = tempUiControls.CgValId }).ToList();
if (!tempCgValDetail.Any())
{
_foundationService.LogBusinessError(null, new ParameterBuilder("CgValId", tempUiControls.CgValId), "Invalid_CgValId_found");
return false;
}
//Add tempCgValDetail List to main list which is cgValDetail
cgValDetail.Add(tempCgValDetail);
}

Take a look at AddRange.
var firstList = new List<string>();
var secondList = new List<string>() { "a", "b" };
firstList.AddRange(secondList);
You mentioned that you don't have access to AddRange... The problem is that you're using an IList, which doesn't implement AddRange. Check this out for more on why: Why doesn't IList support AddRange
I would advise you to switch to List.

First, you need to instantiate cgValDetail as a new list, not a null.
Then you should decide if you want to declare cgValDetail as IList instead of a List. If yes, try this:
IList<CgValDetail> cgValDetail = new List<CgValDetail>();
//Get cgValDetails for each control
foreach (UiControlScreenMetaData tempUiControls in uiControls)
{
if (tempUiControls.CgValId == null)
{
continue;
}
IList<CgValDetail> tempCgValDetail = Retrieval<CgValDetail>.Search(new { CgValId = tempUiControls.CgValId }).ToList();
if (!tempCgValDetail.Any())
{
_foundationService.LogBusinessError(null, new ParameterBuilder("CgValId", tempUiControls.CgValId), "Invalid_CgValId_found");
return false;
}
//Add tempCgValDetail List to main list which is cgValDetail
((List<CgValDetail>)cgValDetail).AddRange(tempCgValDetail);
}
But I wonder why just not using a List instead of IList.
List<CgValDetail> cgValDetail = new List<CgValDetail>();
//Get cgValDetails for each control
foreach (UiControlScreenMetaData tempUiControls in uiControls)
{
if (tempUiControls.CgValId == null)
{
continue;
}
List<CgValDetail> tempCgValDetail = Retrieval<CgValDetail>.Search(new { CgValId = tempUiControls.CgValId }).ToList();
if (!tempCgValDetail.Any())
{
_foundationService.LogBusinessError(null, new ParameterBuilder("CgValId", tempUiControls.CgValId), "Invalid_CgValId_found");
return false;
}
//Add tempCgValDetail List to main list which is cgValDetail
cgValDetail.AddRange(tempCgValDetail);
}

Your cgValDetail is null. That's why when you add(...), you get syntax error.
Create a new IList<CgValDetail>
IList<CgValDetail> cgValDetail = new List<CgValDetail>();
Why didn't you use List<T> in stead of IList<T>?

You forgot to new it.
Just a simple example:
List<a> aList= new List<a>();
List<aList> List_aList = new List<aList>();
List_aList.add(new aList());
Here the link to same question.

Maybe, you can use "UNION" in Linq (if you don't really care about performance or result set is not big enough).
cgValDetail.Add(tempCgValDetail);
change to
cgValDetail = cgValDetail.Union(tempCgValDetail.Select(a => a)).ToList();

Method to check array list containing specific string

I have an ArrayList that import records from a database.
Is there any method to check whether the arrayList contains schname that i want to match to another list which is an api?
List<PrimaryClass> primaryList = new List<PrimaryClass>(e.Result);
PrimaryClass sc = new PrimaryClass();
foreach (string item in str)
{
for (int a = 0; a <= e.Result.Count - 1; a++)
{
string schname = e.Result.ElementAt(a).PrimarySchool;
string tophonour = e.Result.ElementAt(a).TopHonour;
string cca = e.Result.ElementAt(a).Cca;
string topstudent = e.Result.ElementAt(a).TopStudent;
string topaggregate = e.Result.ElementAt(a).TopAggregate;
string topimage = e.Result.ElementAt(a).TopImage;
if (item.Contains(schname))
{
}
}
}
This is what I have come up with so far, kindly correct any errors that I might have committed. Thanks.

How about ArrayList.Contains?

Try this
foreach( string row in arrayList){
if(row.contains(searchString)){
//put your code here.
}
}

Okay, now you've shown that it's actually a List<T>, it should be easy with LINQ:
if (primaryList.Any(x => item.Contains(x.PrimarySchool))
Note that you should really consider using foreach instead of a for loop to iterate over a list, unless you definitely need the index... and if you're dealing with a list, using the indexer is simpler than calling ElementAt.

// check all types
var containsAnyMatch = arrayList.Cast<object>().Any(arg => arg.ToString() == searchText);
// check strings only
var containsStringMatch = arrayList.OfType<string>().Any(arg => arg == searchText);