We Keep Coding

C# MVC Loop through list and update each record efficiently

I have a list of 'Sites' that are stored in my database. The list is VERY big and contains around 50,000+ records.
I am trying to loop through each record and update it. This takes ages, is there a better more efficient way of doing this?
using (IRISInSiteLiveEntities DB = new IRISInSiteLiveEntities())
{
var allsites = DB.Sites.ToList();
foreach( var sitedata in allsites)
{
var siterecord = DB.Sites.Find(sitedata.Id);
siterecord.CabinOOB = "Test";
siterecord.TowerOOB = "Test";
siterecord.ManagedOOB = "Test";
siterecord.IssueDescription = "Test";
siterecord.TargetResolutionDate = "Test";
DB.Entry(siterecord).State = EntityState.Modified;
}
DB.SaveChanges();
}
I have cut the stuff out of the code to get to the point. The proper function code I am using basically pulls a list out from Excel, then matches the records in the sites list and updates each record that matches accordingly. The DB.Find is slowing the loop down dramatically.
[HttpPost]
public ActionResult UploadUpdateOOBList()
{
CheckPermissions("UpdateOOBList");
string[] typesallowed = new string[] { ".xls", ".xlsx" };
HttpPostedFileBase file = Request.Files[0];
var fname = file.FileName;
if (!typesallowed.Any(fname.Contains))
{
return Json("NotAllowed");
}
file.SaveAs(Server.MapPath("~/Uploads/OOB List/") + fname);
//Create empty OOB data list
List<OOBList.OOBDetails> oob_data = new List<OOBList.OOBDetails>();
//Using ClosedXML rather than Interop Excel....
//Interop Excel: 30 seconds for 750 rows
//ClosedXML: 3 seconds for 750 rows
string fileName = Server.MapPath("~/Uploads/OOB List/") + fname;
using (var excelWorkbook = new XLWorkbook(fileName))
{
var nonEmptyDataRows = excelWorkbook.Worksheet(2).RowsUsed();
foreach (var dataRow in nonEmptyDataRows)
{
//for row number check
if (dataRow.RowNumber() >= 4 )
{
string siteno = dataRow.Cell(1).GetValue<string>();
string sitename = dataRow.Cell(2).GetValue<string>();
string description = dataRow.Cell(4).GetValue<string>();
string cabinoob = dataRow.Cell(5).GetValue<string>();
string toweroob = dataRow.Cell(6).GetValue<string>();
string manageoob = dataRow.Cell(7).GetValue<string>();
string resolutiondate = dataRow.Cell(8).GetValue<string>();
string resolutiondate_converted = resolutiondate.Substring(resolutiondate.Length - 9);
oob_data.Add(new OOBList.OOBDetails
{
SiteNo = siteno,
SiteName = sitename,
Description = description,
CabinOOB = cabinoob,
TowerOOB = toweroob,
ManageOOB = manageoob,
TargetResolutionDate = resolutiondate_converted
});
}
}
}
//Now delete file.
System.IO.File.Delete(Server.MapPath("~/Uploads/OOB List/") + fname);
Debug.Write("DOWNLOADING LIST ETC....\n");
using (IRISInSiteLiveEntities DB = new IRISInSiteLiveEntities())
{
var allsites = DB.Sites.ToList();
//Loop through sites and the OOB list and if they match then tell us
foreach( var oobdata in oob_data)
{
foreach( var sitedata in allsites)
{
var indexof = sitedata.SiteName.IndexOf(' ');
if( indexof > 0 )
{
var OOBNo = oobdata.SiteNo;
var OOBName = oobdata.SiteName;
var SiteNo = sitedata.SiteName;
var split = SiteNo.Substring(0, indexof);
if (OOBNo == split && SiteNo.Contains(OOBName) )
{
var siterecord = DB.Sites.Find(sitedata.Id);
siterecord.CabinOOB = oobdata.CabinOOB;
siterecord.TowerOOB = oobdata.TowerOOB;
siterecord.ManagedOOB = oobdata.ManageOOB;
siterecord.IssueDescription = oobdata.Description;
siterecord.TargetResolutionDate = oobdata.TargetResolutionDate;
DB.Entry(siterecord).State = EntityState.Modified;
Debug.Write("Updated Site ID/Name Record: " + sitedata.Id + "/" + sitedata.SiteName);
}
}
}
}
DB.SaveChanges();
}
var nowdate = DateTime.Now.ToString("dd/MM/yyyy");
System.IO.File.WriteAllText(Server.MapPath("~/Uploads/OOB List/lastupdated.txt"),nowdate);
return Json("Success");
}

Looks like you are using Entity Framework (6 or Core). In either case both
var siterecord = DB.Sites.Find(sitedata.Id);
and
DB.Entry(siterecord).State = EntityState.Modified;
are redundant, because the siteData variable is coming from
var allsites = DB.Sites.ToList();
This not only loads the whole Site table in memory, but also EF change tracker keeps reference to every object from that list. You can easily verify that with
var siterecord = DB.Sites.Find(sitedata.Id);
Debug.Assert(siterecord == sitedata);
The Find (when the data is already in memory) and Entry methods themselves are fast. But the problem is that they by default trigger automatic DetectChanges, which leads to quadratic time complexity - in simple words, very slow.
With that being said, simply remove them:
if (OOBNo == split && SiteNo.Contains(OOBName))
{
sitedata.CabinOOB = oobdata.CabinOOB;
sitedata.TowerOOB = oobdata.TowerOOB;
sitedata.ManagedOOB = oobdata.ManageOOB;
sitedata.IssueDescription = oobdata.Description;
sitedata.TargetResolutionDate = oobdata.TargetResolutionDate;
Debug.Write("Updated Site ID/Name Record: " + sitedata.Id + "/" + sitedata.SiteName);
}
This way EF will detect changes just once (before SaveChanges) and also will update only the modified record fields.

I have followed Ivan Stoev's suggestion and have changed the code by removing the DB.Find and the EntitySate Modified - It now takes about a minute and a half compared to 15 minutes beforehand. Very suprising as I didn't know that you dont actually require that to update the records. Clever. The code is now:
using (IRISInSiteLiveEntities DB = new IRISInSiteLiveEntities())
{
var allsites = DB.Sites.ToList();
Debug.Write("Starting Site Update loop...");
//Loop through sites and the OOB list and if they match then tell us
//750 records takes around 15-20 minutes.
foreach( var oobdata in oob_data)
{
foreach( var sitedata in allsites)
{
var indexof = sitedata.SiteName.IndexOf(' ');
if( indexof > 0 )
{
var OOBNo = oobdata.SiteNo;
var OOBName = oobdata.SiteName;
var SiteNo = sitedata.SiteName;
var split = SiteNo.Substring(0, indexof);
if (OOBNo == split && SiteNo.Contains(OOBName) )
{
sitedata.CabinOOB = oobdata.CabinOOB;
sitedata.TowerOOB = oobdata.TowerOOB;
sitedata.ManagedOOB = oobdata.ManageOOB;
sitedata.IssueDescription = oobdata.Description;
sitedata.TargetResolutionDate = oobdata.TargetResolutionDate;
Debug.Write("Thank you, next: " + sitedata.Id + "\n");
}
}
}
}
DB.SaveChanges();
}

So first of all you should turn your HTTPPost in an async function
more info https://learn.microsoft.com/en-us/dotnet/csharp/programming-guide/concepts/async/
What you then should do is create the tasks and add them to a list. Then wait for them to complete (if you want/need to) by calling Task.WaitAll()
https://learn.microsoft.com/en-us/dotnet/api/system.threading.tasks.task.waitall?view=netframework-4.7.2
This will allow your code to run in parallel on multiple threads optimizing performance quite a bit already.
You can also use linq to for example reduce the size of allsites beforehand by doing something that will roughly look like this
var sitedataWithCorrectNames = allsites.Where(x => x //evaluate your condition here)
https://learn.microsoft.com/en-us/dotnet/framework/data/adonet/ef/language-reference/supported-and-unsupported-linq-methods-linq-to-entities
and then start you foreach (var oobdata) with the now foreach(sitedate in sitedataWithCorrectNames)
Same goes for SiteNo.Contains(OOBName)
https://learn.microsoft.com/en-us/dotnet/csharp/programming-guide/concepts/linq/getting-started-with-linq
P.S. Most db sdk's also provide asynchornous functions so use those aswell.
P.P.S. I didn't have an IDE so I eyeballed the code but the links should provide you with plenty of samples. Reply if you need more help.

How to filter values read from a text file

I want to get a couple of values from a textfile in C#. Example:
1.sex=male
1.name=barack
1.lastname=obama
1.age = 55
2.sex=female
2.name= kelly
2.lastname=clinton
2.age = 24
3.sex = male
3.firstname= mike
3.lastname= james
3.age= 19
I only want to get all the "name", "lastname" and ages from the textFile, not the "sex". How can I filter this? I have tried something like this, but it only shows 1 value.
var list = new List<string>();
var text = File.ReadAllLines(#"C:\Users\Jal\Desktop\Test.text");
foreach (var s in text)
{
if (s.Contains("Name"))
{
if (s.Contains("Name"))
{
var desc = s.Substring(s.IndexOf("=") + 1);
list.Add(desc);
ListView.Items.Add(desc);
}
}
}
I found this code on Stack Overflow, but it doesn't get all of the values I want.

var names = new List<string>();
var lastnames = new List<string>();
var text = File.ReadAllLines(#"C:\Users\Jal\Desktop\Test.text");
foreach (var s in text)
{
if (s.Contains("lastname"))
{
var lastname = s.Substring(s.IndexOf("=") + 1);
lastnames.Add(lastname);
continue;
}
if (s.Contains("name"))
{
var name = s.Substring(s.IndexOf("=") + 1);
names.Add(name);
continue;
}
}
And in same way you can add another properties.

s.Contains("Name") won't ever be true on this case because it's case-sensitive, and your string in the file is "name".
Try using s.Contains("name")
But you would be better off using a Regex for this kind of thing.

Associate XML Attribute with Class Property

I am building an MVC App in C# and am currently working on a method to load an xml document into a file stream and iterate through the nodes and save the data into my model objects.
It is all working fine but I am hard coding the relationship between the object property and the xml attribute name. I was wondering if there is a smart way to associated the two so I can run it all through a for each loop.
This is the code I have currently and it works, but I would like to make it more generic
OLD CODE
var xmlDoc = LoadXmlFileIntoStream("WSAPayCode.xml");
var elementCollection = ExtractDescendants(xmlDoc, "WSAPayCode");
foreach (var element in elementCollection)
{
var abbreviationChar = element.Attribute("AbbreviationChar");
var payCode = new PayCode
{
Name = element.Attribute("Name").Value,
AutoResolved = element.Attribute("AutoResolved").Value.IsBool(),
EditExcuseAbsn = element.Attribute("EditExcuseAbsn").Value.IsBool(),
PersistPceSw = element.Attribute("PersistPceSw").Value.IsBool(),
AbbreviationChar = (string)element.Attribute("AbbreviationChar"),
EditCntToCdotSw = element.Attribute("EditCntToCdotSw").Value.IsBool(),
EditAffShfTotal = element.Attribute("EditAffShfTotal").Value.IsBool(),
EditCntToOt = element.Attribute("EditCntToOt").Value.IsBool(),
PayUsingWeightedAverageRate = element.Attribute("PayUsingWeightedAverageRate").Value.IsBool(),
RequiresMgrApproval = element.Attribute("RequiresMgrApproval").Value.IsBool(),
WeightedAverageRateIsComputedDaily =
element.Attribute("WeightedAverageRateIsComputedDaily").Value.IsBool(),
JustAutoResExpAsWorked = element.Attribute("JustAutoResExpAsWorked").Value.IsBool(),
AssociatedDurationPayCodeName = element.Attribute("AssociatedDurationPayCodeName").Value,
WeightedAverageRateContributionsUseAnAdjustedRate =
element.Attribute("WeightedAverageRateContributionsUseAnAdjustedRate").Value.IsBool(),
ScheduleHoursType = element.Attribute("ScheduleHoursType").Value,
CheckAvlbltySw = element.Attribute("CheckAvlbltySw").Value.IsBool(),
WageAddition = (string)element.Attribute("WageAddition"),
VisibleInMainArea = element.Attribute("VisibleInMainArea").Value.IsBool(),
IsMoneyCategory = element.Attribute("IsMoneyCategory").Value.IsBool(),
AmountType = element.Attribute("AmountType").Value,
VisibleInReport = element.Attribute("VisibleInReport").Value.IsBool(),
ContributesToWeightedAverageRates =
element.Attribute("ContributesToWeightedAverageRates").Value.IsBool(),
UnjustAutoResExpAsWorked = (bool)element.Attribute("UnjustAutoResExpAsWorked"),
WageMultiply = (string)element.Attribute("WageMultiply"),
Type = (string)element.Attribute("Type"),
VisibleToUser = (bool)element.Attribute("VisibleToUser"),
CustomerId = _customerId,
};
_db.PayCodes.Add(payCode);
_db.SaveChanges();
}
New Code
I have written some code to interate through the xml file and pull out the names of the attributes - Code Below (Also works)
var xmlDoc = LoadXmlFileIntoStream("WSAPayCode.xml");
var elementCollection = ExtractDescendants(xmlDoc, "WSAPayCode");
var nodeAttributes = xmlDoc.Descendants("WSAPayCode").Select(x => x.Attributes());
foreach (var attrs in nodeAttributes)
{
var _attribute = "";
foreach (var attr in attrs)
{
// This successfully reads through each attribute and pulls out the name of the attribut
_attribute = attr.Name.ToString();
}
}
Problem I would like to solve
What I would like to do now is instantiate an object and iterate through the attribute names and save the values to the corresponding property in the object. i.e. replace the OLD code with something that dynamically assigns the values to the object properties.

I would check into using the XML Serializers / Deserializers. On your class, you have provide attributes as to whether the property is an element or attribute, and then let it handle the rest of it for you.
You could set the attributes to match what the XML document is providing, such as if the name is an element, or attribute, etc.
For example:
[XmlRoot("PayCode")]
public class PayCode
{
[XmlElement("Name")
public string Name { get; set;}
....
}
Then, to deserialize to your object:
XmlSerializer serializer = new
XmlSerializer(typeof(PayCode));
FileStream fs = new FileStream(filename, FileMode.Open);
XmlReader reader = XmlReader.Create(fs);
PayCode payCode = (PayCode)serializer.Deserialize(reader);

xml file comparison

Is there any way to compare two XML files in C#? I only want to compare the nodes of the first file with that of of second file. I don't want to append the missing nodes.
Is there any way to do this?
Here's what I have tried:
var docA = XDocument.Parse(#"<mind_layout></mind_layout>");
var docB = XDocument.Parse(#"<mind_layout></mind_layout>");
var rootNameA = docA.Root.Name;
var rootNameB = docB.Root.Name;
var equalRootNames = rootNameB.Equals(rootNameA);
var descendantsA = docA.Root.Descendants();
var descendantsB = docB.Root.Descendants();
for (int i = 0; i < descendantsA.Count(); i++)
{
var descendantA = descendantsA.ElementAt(i);
var descendantB = descendantsB.ElementAt(i);
var equalChildNames = descendantA.Name.Equals(descendantB.Name);
var valueA = descendantA.Value;
var valueB = descendantA.Value;
var equalValues = valueA.Equals(valueB);
}
where <mind_layout> is the root node in both the files.

If you just want to compare the file contents (including, for example, indentation), you coud do:
if (File.ReadAllText(#"C:\path\to\file1.xml") == File.ReadAllText(#"C:\path\to\file2.xml"))
{
// Same TEXT content
}
(Warning: this is not the most optimized check you could do!)
If you want to compare the XML content (regardless of the formatting), you can do:
var doc1 = XDocument.Load(File.OpenRead(#"C:\path\to\file1.xml"));
var doc2 = XDocument.Load(File.OpenRead(#"C:\path\to\file2.xml"));
if (XDocument.DeepEquals(doc1, doc2))
{
// Same XML content
}

How to create LINQ QUERY dynamically?

Is there any way to insert part of the code between { } dynamically?
LINQ QUERY:
var csvdata = from csvline in csvlines
let column = csvline.Split(';')
select new {
produkt = column[0],
cislo = column[1],
part = column[2],
serial = column[3]
};
I mean something like:
string qpart = "produkt = column[0], cislo = column[1], part = column[2], serial = column[3]";
var csvdata = from csvline in csvlines
let column = csvline.Split(';')
select new {
qpart
};
Thanks for answers..

Try investigating Dynamic Linq Query Library.
http://weblogs.asp.net/scottgu/archive/2008/01/07/dynamic-linq-part-1-using-the-linq-dynamic-query-library.aspx
http://naspinski.net/post/Writing-Dynamic-Linq-Queries-in-Linq-to-Entities.aspx
You'll need to convert the string array returned from Split into IQueryable for it to work but I think this is your best shot.
var results = columns
.Select("new(column[0] As produkt)");
Is how I'd I imagine it would work?.