We Keep Coding

Best Way to compare 1 million List of object with another 1 million List of object in c#

i am differentiating 1 million list of object with another 1 million list of object.
i am using for , foreach but it takes too much of time to iterate those list.
can any one help me best way to do this
var SourceList = new List<object>(); //one million
var TargetList = new List<object>()); // one million
//getting data from database here
//SourceList with List of one million
//TargetList with List of one million
var DifferentList = new List<object>();
//ForEach
SourceList.ToList().ForEach(m =>
{
if (!TargetList.Any(s => s.Name == m.Name))
DifferentList.Add(m);
});
//for
for (int i = 0; i < SourceList .Count; i++)
{
if (!TargetList .Any(s => s == SourceList [i].Name))
DifferentList .Add(SourceList [i]);
}

I think it seems like a bad idea but IEnumerable magic will help you.
For starters, simplify your expression. It looks like this:
var result = sourceList.Where(s => targetList.Any(t => t.Equals(s)));
I recommend making a comparison in the Equals method:
public class CompareObject
{
public string prop { get; set; }
public new bool Equals(object o)
{
if (o.GetType() == typeof(CompareObject))
return this.prop == ((CompareObject)o).prop;
return this.GetHashCode() == o.GetHashCode();
}
}
Next add AsParallel. This can both speed up and slow down your program. In your case, you can add ...
var result = sourceList.AsParallel().Where(s => !targetList.Any(t => t.Equals(s)));
CPU 100% loaded if you try to list all at once like this:
var cnt = result.Count();
But it’s quite tolerable to work if you get the results in small portions.
result.Skip(10000).Take(10000).ToList();
Full code:
static Random random = new Random();
public class CompareObject
{
public string prop { get; private set; }
public CompareObject()
{
prop = random.Next(0, 100000).ToString();
}
public new bool Equals(object o)
{
if (o.GetType() == typeof(CompareObject))
return this.prop == ((CompareObject)o).prop;
return this.GetHashCode() == o.GetHashCode();
}
}
void Main()
{
var sourceList = new List<CompareObject>();
var targetList = new List<CompareObject>();
for (int i = 0; i < 10000000; i++)
{
sourceList.Add(new CompareObject());
targetList.Add(new CompareObject());
}
var stopWatch = new Stopwatch();
stopWatch.Start();
var result = sourceList.AsParallel().Where(s => !targetList.Any(t => t.Equals(s)));
var lr = result.Skip(10000).Take(10000).ToList();
stopWatch.Stop();
Console.WriteLine(stopWatch.Elapsed);
}
Update
I remembered what you can use Hashtable.Choos unique values from targetList and from sourceList next fill out the result whose values are not targetList.
Example:
static Random random = new Random();
public class CompareObject
{
public string prop { get; private set; }
public CompareObject()
{
prop = random.Next(0, 1000000).ToString();
}
public new int GetHashCode() {
return prop.GetHashCode();
}
}
void Main()
{
var sourceList = new List<CompareObject>();
var targetList = new List<CompareObject>();
for (int i = 0; i < 10000000; i++)
{
sourceList.Add(new CompareObject());
targetList.Add(new CompareObject());
}
var stopWatch = new Stopwatch();
stopWatch.Start();
var sourceHashtable = new Hashtable();
var targetHashtable = new Hashtable();
foreach (var element in targetList)
{
var hash = element.GetHashCode();
if (!targetHashtable.ContainsKey(hash))
targetHashtable.Add(element.GetHashCode(), element);
}
var result = new List<CompareObject>();
foreach (var element in sourceList)
{
var hash = element.GetHashCode();
if (!sourceHashtable.ContainsKey(hash))
{
sourceHashtable.Add(hash, element);
if(!targetHashtable.ContainsKey(hash)) {
result.Add(element);
}
}
}
stopWatch.Stop();
Console.WriteLine(stopWatch.Elapsed);
}

Scanning the target list to match the name is an O(n) operation, thus your loop is O(n^2). If you build a HashSet<string> of all the distinct names in the target list, you can check whether a name exists in the set in O(1) time using the Contains method.

//getting data from database here
You are getting the data out of a system that specializes in matching and sorting and filtering data, into your RAM that by default cannot yet do that task at all. And then you try to sort, filter and match yourself.
That will fail. No matter how hard you try, it is extremely unlikely that your computer with a single programmer working at a matching algorithm will outperform your specialized piece of hardware called a database server at the one operation this software is supposed to be really good at that was programmed by teams of experts and optimized for years.
You don't go into a fancy restaurant and ask them to give you huge bags of raw ingredients so you can throw them into a big bowl unpeeled and microwave them at home. No. You order a nice dish because it will be way better than anything you could do yourself.
The simple answer is: Do not do that. Do not take the raw data and rummage around in it for hours. Leave that job to the database. It's the one thing it's supposed to be good at. Use it's power. Write a query that will give you the result, don't get the raw data and then play database yourself.

Foreach performs a null check before each iteration, so using a standard for loop will provide slightly better performance that will be hard to beat.
If it is taking too long, can you break down the collection into smaller sets and/or process them in parallel?
Also you could look a PLinq (Parallel Linq) using .AsParallel()
Other areas to improve are the actual comparison logic that you are using, also how the data is stored in memory, depending on your problem, you may not have to load the entire object into memory for every iteration.
Please provide a code example so that we can assist further, when such large amounts of data are involved performance degredation is to be expected.
Again depending on the time that we are talking about here, you could upload the data into a database and use that for the comparison rather than trying to do it natively in C#, this type of solution is better suited to data sets that are already in a database or where the data changes much less frequently than the times you need to perform the comparison.

How to Move to next record while building an IEnumerable if the current row has data problems

I have a method that outputs a list of RSVPs for a class. The RSVP is based on a pull from a SQL table holding RSVP records, based on an input parameter of the class id. Then a dictionary of all students (people who RSVPd to that class) is made. Finally I output the IEnumerable of the RSVPs based on the data for each student.
The problem I'm running into is that I've got a couple students in the data that are "bad users": They aren't in the system. Potentially because of bad record deletions, or bad creations. Either way, I need to set up error handling for "bad student records" while building the IEnumerable.
My thought was to catch the potential error when evaluating the student id on this line:
var data = x.ToRsvpData(students[x.RawAgentId]);
And then just skip that record and move on to the next one. However, I'm not sure how to do that.
Here's the complete method:
public IEnumerable<RsvpData> GetAllRsvpsFor(Guid scheduledId)
{
var rsvps = _sors.AnyRsvpsIn(new[] { scheduledId })[scheduledId];
var certificates = _sors.CertificatesIn(rsvps.Select(x => x.RsvpId).ToList());
var students = _sors.StudentsBy(rsvps);
return rsvps.Select(x => {
var data = x.ToRsvpData(students[x.RawAgentId]);
if (x.Completed)
{
data.SignatureUrl = StaticContent.S3WebPrefixFor(string.Format("/schools/signatures/{0}.jpg", x.RsvpId.ToString()));
var cert = certificates.FirstOrDefault(c => c.Rsvp.RsvpId == x.RsvpId);
data.CertificateId = cert != null ? cert.CertId.ToString() : "";
}
return data;
}).OrderBy(x => x.LastName).ToList();
}
Update:
Here's the completed, working code:
public IEnumerable<RsvpData> GetAllRsvpsFor(Guid scheduledId)
{
var rsvps = _sors.AnyRsvpsIn(new[] { scheduledId })[scheduledId];
var certificates = _sors.CertificatesIn(rsvps.Select(x => x.RsvpId).ToList());
var students = _sors.StudentsBy(rsvps);
var cleanRsvpData = new List<RsvpData>();
foreach (var rsvp in rsvps)
{
try
{
var data = rsvp.ToRsvpData(students[rsvp.RawAgentId]);
if (rsvp.Completed)
{
data.SignatureUrl = StaticContent.S3WebPrefixFor(string.Format("/schools/signatures/{0}.jpg", rsvp.RsvpId.ToString()));
var cert = certificates.FirstOrDefault(c => c.Rsvp.RsvpId == rsvp.RsvpId);
data.CertificateId = cert != null ? cert.CertId.ToString() : "";
}
cleanRsvpData.Add(data);
}
catch (Exception ex)
{ //Bad Student record
Elmah.ErrorSignal.FromCurrentContext().Raise(ex);
}
}

Depending on what makes them particularly misbehaving records, simply wrapping your Enumerable in another Enumerable should fix the problem. Something like this:
IEnumerable<Record> GetCorrectRecords(IEnumerable<Record> records)
{
foreach(var record in records)
if(record.Valid) // up to you how you define this
yield return record;
}

Instead of using lambda expressions, your best bet would probably be to use a temp list variable and use a try catch block - so only the "clean" records will make it into your list:
List<RsvpData> cleanRsvpData = new List<RsvpData>();
foreach (RsvpData rsvp in rsvps)
{
try
{
RsvpData data = rsvp.ToRsvpData(students[x.RawAgentId]);
if (rsvp.Completed)
{
data.SignatureUrl = "test";
var cert = certificates.FirstOrDefault(c => c.Rsvp.RsvpId == x.RsvpId);
data.CertificateId = cert != null ? cert.CertId.ToString() : "";
}
cleanRsvpData.Add(data);
}
catch (Exception ex)
{ // handle error here
}
}
Since the try catch is INSIDE the loop, it won't break your whole loop if one of the items in the list throws an error.

You can do that using the famous try...catch block, something like this:
public IEnumerable<int> GetAllRsvpsFor(Guid scheduledId)
{
//all the code that precedes the loop
//for or foreach loop
{
//any code that you have to perform after this block
try
{
var data = x.ToRsvpData(students[x.RawAgentId]);
}
catch
{
continue; //continues to the next for iteration in case of any error
}
//any code that you have to perform after this block
}
}

How can I determine if this method is threadsafe?

private void PerformValuations(DateTime testDate, RegressionEffectivenessTest.RegressionDateWithValues date)
{
var valueDate = new LegacyWCFRateTimeStamp { Type = RateTimeStampType.EndOfDay, TimeStamp = date.ValueDate };
var curveDate = new LegacyWCFRateTimeStamp { Type = RateTimeStampType.EndOfDay, TimeStamp = date.CurveDate };
var shiftDate = new LegacyWCFRateTimeStamp { Type = RateTimeStampType.EndOfDay, TimeStamp = date.ShiftDate };
if (date.NeedHedgeValues)
{
date.HedgeCleanPrice = 0M;
date.HedgeCleanIntrinsicValue = 0M;
foreach (var tran in _hedgeTranQuoteIds)
{
var tranquoteId = tran.TranQuoteId;
CheckAndLoadTrade(testDate, valueDate, shiftDate, curveDate, tran, tranquoteId);
var result = ValueTrade(tranquoteId);
var rtnVal = !result.Succeeded
? HandleFailure(tranquoteId, shiftDate, curveDate, result, valueDate)
: CreateAccountingValuation(valueDate, result);
date.HedgeCleanIntrinsicValue += rtnVal.IntrinsicValue - rtnVal.AccruedInterest.GetValueOrDefault(0);
date.HedgeCleanPrice += rtnVal.CleanPrice;
}
}
So I'm trying to run a Parallel.ForEach on this method. There were a couple of things that I was worried about. The first one is within the CheckAndLoadTrade method, it accesses a private Dictionary of the class to possibly add an item to it if it isn't there, and then the ValueTrade method gets an item from this dictionary.
If I parallel this out, am I going to run into any thread safety issues with the dictionary being accessed? Or possibly anything else I didn't notice? All other method calls use variables defined in their own scope, it's really just this one Dictionary that I am worried about. Should I throw a lock before and after the actual dictionary access happens?

What part are you trying to parallelize? The foreach loop?
If you can, use a ConcurrentDictionary. within CheckAndLoadTrade. Another concern is what code runs when the trade is not in the dictionary. Is the "loadtrade" code thread-safe?

Sorting an IList

I have got a complete list of my brands to appear in my DropDownBox, however there appears to be no order (simply how they are input into the database) and I need to sort them into alphabetical order.
But it doesn't look like I can use the .Sort(); on an IList and there doesn't seem to be anything similar on ILists so I am at a bit of a loss, I have tried to convert the IList into a List and then using the List.Sort() method but I have had no luck with this as it just comes back unsorted again:
public void BrandListRetrieve()
{
var factory = new BrandFactory();
var customBool1State =
factory.ByCustomBoolean1(false, CoreHttpModule.Session);
if (customBool1State != null)
{
var brandDropDown = CoreHttpModule
.Session
.CreateCriteria(typeof(Brand)).List<Brand>();
foreach (Brand brand in brandDropDown)
{
this.Items.Add(brand.Name);
}
if (this.Items.Count < 0)
{
this.Items.Insert(0, new ListItem("Hello World", "Hello World"));
}
var brandList = brandDropDown as List<string>;
if (brandList != null)
brandList.Sort();
}
}

you should try this;
foreach (Brand brand in brandDropDown.OrderBy(b => b.Name))
You can certainly REMOVE the following lines from your code;
var brandList = brandDropDown as List<string>;
if (brandList != null)
brandList.Sort();

it seems that you need to use brand.Sort() with Compare method. Try to read this manual http://msdn.microsoft.com/en-us/library/w56d4y5z.aspx

Method to check array list containing specific string

I have an ArrayList that import records from a database.
Is there any method to check whether the arrayList contains schname that i want to match to another list which is an api?
List<PrimaryClass> primaryList = new List<PrimaryClass>(e.Result);
PrimaryClass sc = new PrimaryClass();
foreach (string item in str)
{
for (int a = 0; a <= e.Result.Count - 1; a++)
{
string schname = e.Result.ElementAt(a).PrimarySchool;
string tophonour = e.Result.ElementAt(a).TopHonour;
string cca = e.Result.ElementAt(a).Cca;
string topstudent = e.Result.ElementAt(a).TopStudent;
string topaggregate = e.Result.ElementAt(a).TopAggregate;
string topimage = e.Result.ElementAt(a).TopImage;
if (item.Contains(schname))
{
}
}
}
This is what I have come up with so far, kindly correct any errors that I might have committed. Thanks.

How about ArrayList.Contains?

Try this
foreach( string row in arrayList){
if(row.contains(searchString)){
//put your code here.
}
}

Okay, now you've shown that it's actually a List<T>, it should be easy with LINQ:
if (primaryList.Any(x => item.Contains(x.PrimarySchool))
Note that you should really consider using foreach instead of a for loop to iterate over a list, unless you definitely need the index... and if you're dealing with a list, using the indexer is simpler than calling ElementAt.

// check all types
var containsAnyMatch = arrayList.Cast<object>().Any(arg => arg.ToString() == searchText);
// check strings only
var containsStringMatch = arrayList.OfType<string>().Any(arg => arg == searchText);