We Keep Coding

C# MVC Loop through list and update each record efficiently

I have a list of 'Sites' that are stored in my database. The list is VERY big and contains around 50,000+ records.
I am trying to loop through each record and update it. This takes ages, is there a better more efficient way of doing this?
using (IRISInSiteLiveEntities DB = new IRISInSiteLiveEntities())
{
var allsites = DB.Sites.ToList();
foreach( var sitedata in allsites)
{
var siterecord = DB.Sites.Find(sitedata.Id);
siterecord.CabinOOB = "Test";
siterecord.TowerOOB = "Test";
siterecord.ManagedOOB = "Test";
siterecord.IssueDescription = "Test";
siterecord.TargetResolutionDate = "Test";
DB.Entry(siterecord).State = EntityState.Modified;
}
DB.SaveChanges();
}
I have cut the stuff out of the code to get to the point. The proper function code I am using basically pulls a list out from Excel, then matches the records in the sites list and updates each record that matches accordingly. The DB.Find is slowing the loop down dramatically.
[HttpPost]
public ActionResult UploadUpdateOOBList()
{
CheckPermissions("UpdateOOBList");
string[] typesallowed = new string[] { ".xls", ".xlsx" };
HttpPostedFileBase file = Request.Files[0];
var fname = file.FileName;
if (!typesallowed.Any(fname.Contains))
{
return Json("NotAllowed");
}
file.SaveAs(Server.MapPath("~/Uploads/OOB List/") + fname);
//Create empty OOB data list
List<OOBList.OOBDetails> oob_data = new List<OOBList.OOBDetails>();
//Using ClosedXML rather than Interop Excel....
//Interop Excel: 30 seconds for 750 rows
//ClosedXML: 3 seconds for 750 rows
string fileName = Server.MapPath("~/Uploads/OOB List/") + fname;
using (var excelWorkbook = new XLWorkbook(fileName))
{
var nonEmptyDataRows = excelWorkbook.Worksheet(2).RowsUsed();
foreach (var dataRow in nonEmptyDataRows)
{
//for row number check
if (dataRow.RowNumber() >= 4 )
{
string siteno = dataRow.Cell(1).GetValue<string>();
string sitename = dataRow.Cell(2).GetValue<string>();
string description = dataRow.Cell(4).GetValue<string>();
string cabinoob = dataRow.Cell(5).GetValue<string>();
string toweroob = dataRow.Cell(6).GetValue<string>();
string manageoob = dataRow.Cell(7).GetValue<string>();
string resolutiondate = dataRow.Cell(8).GetValue<string>();
string resolutiondate_converted = resolutiondate.Substring(resolutiondate.Length - 9);
oob_data.Add(new OOBList.OOBDetails
{
SiteNo = siteno,
SiteName = sitename,
Description = description,
CabinOOB = cabinoob,
TowerOOB = toweroob,
ManageOOB = manageoob,
TargetResolutionDate = resolutiondate_converted
});
}
}
}
//Now delete file.
System.IO.File.Delete(Server.MapPath("~/Uploads/OOB List/") + fname);
Debug.Write("DOWNLOADING LIST ETC....\n");
using (IRISInSiteLiveEntities DB = new IRISInSiteLiveEntities())
{
var allsites = DB.Sites.ToList();
//Loop through sites and the OOB list and if they match then tell us
foreach( var oobdata in oob_data)
{
foreach( var sitedata in allsites)
{
var indexof = sitedata.SiteName.IndexOf(' ');
if( indexof > 0 )
{
var OOBNo = oobdata.SiteNo;
var OOBName = oobdata.SiteName;
var SiteNo = sitedata.SiteName;
var split = SiteNo.Substring(0, indexof);
if (OOBNo == split && SiteNo.Contains(OOBName) )
{
var siterecord = DB.Sites.Find(sitedata.Id);
siterecord.CabinOOB = oobdata.CabinOOB;
siterecord.TowerOOB = oobdata.TowerOOB;
siterecord.ManagedOOB = oobdata.ManageOOB;
siterecord.IssueDescription = oobdata.Description;
siterecord.TargetResolutionDate = oobdata.TargetResolutionDate;
DB.Entry(siterecord).State = EntityState.Modified;
Debug.Write("Updated Site ID/Name Record: " + sitedata.Id + "/" + sitedata.SiteName);
}
}
}
}
DB.SaveChanges();
}
var nowdate = DateTime.Now.ToString("dd/MM/yyyy");
System.IO.File.WriteAllText(Server.MapPath("~/Uploads/OOB List/lastupdated.txt"),nowdate);
return Json("Success");
}

Looks like you are using Entity Framework (6 or Core). In either case both
var siterecord = DB.Sites.Find(sitedata.Id);
and
DB.Entry(siterecord).State = EntityState.Modified;
are redundant, because the siteData variable is coming from
var allsites = DB.Sites.ToList();
This not only loads the whole Site table in memory, but also EF change tracker keeps reference to every object from that list. You can easily verify that with
var siterecord = DB.Sites.Find(sitedata.Id);
Debug.Assert(siterecord == sitedata);
The Find (when the data is already in memory) and Entry methods themselves are fast. But the problem is that they by default trigger automatic DetectChanges, which leads to quadratic time complexity - in simple words, very slow.
With that being said, simply remove them:
if (OOBNo == split && SiteNo.Contains(OOBName))
{
sitedata.CabinOOB = oobdata.CabinOOB;
sitedata.TowerOOB = oobdata.TowerOOB;
sitedata.ManagedOOB = oobdata.ManageOOB;
sitedata.IssueDescription = oobdata.Description;
sitedata.TargetResolutionDate = oobdata.TargetResolutionDate;
Debug.Write("Updated Site ID/Name Record: " + sitedata.Id + "/" + sitedata.SiteName);
}
This way EF will detect changes just once (before SaveChanges) and also will update only the modified record fields.

I have followed Ivan Stoev's suggestion and have changed the code by removing the DB.Find and the EntitySate Modified - It now takes about a minute and a half compared to 15 minutes beforehand. Very suprising as I didn't know that you dont actually require that to update the records. Clever. The code is now:
using (IRISInSiteLiveEntities DB = new IRISInSiteLiveEntities())
{
var allsites = DB.Sites.ToList();
Debug.Write("Starting Site Update loop...");
//Loop through sites and the OOB list and if they match then tell us
//750 records takes around 15-20 minutes.
foreach( var oobdata in oob_data)
{
foreach( var sitedata in allsites)
{
var indexof = sitedata.SiteName.IndexOf(' ');
if( indexof > 0 )
{
var OOBNo = oobdata.SiteNo;
var OOBName = oobdata.SiteName;
var SiteNo = sitedata.SiteName;
var split = SiteNo.Substring(0, indexof);
if (OOBNo == split && SiteNo.Contains(OOBName) )
{
sitedata.CabinOOB = oobdata.CabinOOB;
sitedata.TowerOOB = oobdata.TowerOOB;
sitedata.ManagedOOB = oobdata.ManageOOB;
sitedata.IssueDescription = oobdata.Description;
sitedata.TargetResolutionDate = oobdata.TargetResolutionDate;
Debug.Write("Thank you, next: " + sitedata.Id + "\n");
}
}
}
}
DB.SaveChanges();
}

So first of all you should turn your HTTPPost in an async function
more info https://learn.microsoft.com/en-us/dotnet/csharp/programming-guide/concepts/async/
What you then should do is create the tasks and add them to a list. Then wait for them to complete (if you want/need to) by calling Task.WaitAll()
https://learn.microsoft.com/en-us/dotnet/api/system.threading.tasks.task.waitall?view=netframework-4.7.2
This will allow your code to run in parallel on multiple threads optimizing performance quite a bit already.
You can also use linq to for example reduce the size of allsites beforehand by doing something that will roughly look like this
var sitedataWithCorrectNames = allsites.Where(x => x //evaluate your condition here)
https://learn.microsoft.com/en-us/dotnet/framework/data/adonet/ef/language-reference/supported-and-unsupported-linq-methods-linq-to-entities
and then start you foreach (var oobdata) with the now foreach(sitedate in sitedataWithCorrectNames)
Same goes for SiteNo.Contains(OOBName)
https://learn.microsoft.com/en-us/dotnet/csharp/programming-guide/concepts/linq/getting-started-with-linq
P.S. Most db sdk's also provide asynchornous functions so use those aswell.
P.P.S. I didn't have an IDE so I eyeballed the code but the links should provide you with plenty of samples. Reply if you need more help.

Get current iteration from TFS

I need to get current iteration's path from TFS project. I'm able to use REST API query <server>/<project>/_apis/work/teamsettings/iterations?$timeframe=current&api-version=v2.0-preview but I don't want to perform query and parse JSON response. I want to use appropriate API in .NET client libraries for VSTS (and TFS).
I have an instance of the VssConnection. How can I get the path of current iteration from this object?

You can get the current iteration using the WorkHttpClient without having to iterate:
var creds = new VssBasicCredential(string.Empty, "personalaccesstoken");
VssConnection connection = new VssConnection(new Uri("url"), creds);
var workClient = connection.GetClient<WorkHttpClient>();
var teamContext = new TeamContext(teamId);
teamContext.ProjectId = projectId;
var currentIteration = await workClient.GetTeamIterationsAsync(teamContext, "current");

The simplest way I've found to do it was by using ICommonStructureService4 and TeamSettingsConfigurationService methods:
static TfsTeamProjectCollection _tfs = TfsTeamProjectCollectionFactory
.GetTeamProjectCollection("<tfsUri>")
(...)
static string GetCurrentIterationPath()
{
var css = _tfs.GetService<ICommonStructureService4>();
var teamProjectName = "<teamProjectName>";
var project = css.GetProjectFromName(teamProjectName);
var teamName = "<teamName>";
var teamSettingsStore =
_tfs.GetService<TeamSettingsConfigurationService>();
var settings = teamSettingsStore
.GetTeamConfigurationsForUser(new[] { project.Uri })
.Where(c => c.TeamName == teamName)
.FirstOrDefault();
if (settings == null)
{
var currentUser = System.Threading.Thread.CurrentPrincipal.Identity.Name;
throw new InvalidOperationException(
$"User '{currentUser}' doesn't have access to '{teamName}' team project.");
}
return settings.TeamSettings.CurrentIterationPath;
}
And returning the TeamSettings.CurrentIterationPath property.

I found a solution using VssConnection:
var workClient = connection.GetClient<WorkHttpClient>();
var iterations = workClient.GetTeamIterationsAsync(new TeamContext("project-name")).Result;
var currentDate = DateTime.Now.Date;
var currentIterationPath = iterations
.Select(i => new { i.Path, i.Attributes })
.FirstOrDefault(i => currentDate >= i.Attributes.StartDate &&
currentDate <= i.Attributes.FinishDate)
?.Path;

Here is a case provides a solution: Get the current iteration path from TFS
private static XmlNode currentIterationNode;
TfsTeamProjectCollection tpc = TFSConncetion(#"http://tfs/url");
ICommonStructureService4 css = tpc.GetService<ICommonStructureService4>();;
WorkItemStore workItemStore = new WorkItemStore(tpc);
foreach (Project teamProject in workItemStore.Projects)
{
if (teamProject.Name.Equals("TeamProjectNameGoesHere"))
{
NodeInfo[] structures = css.ListStructures(teamProject.Uri.ToString());
NodeInfo iterations = structures.FirstOrDefault(n => n.StructureType.Equals("ProjectLifecycle"));
if (iterations != null)
{
XmlElement iterationsTree = css.GetNodesXml(new[] { iterations.Uri }, true);
XmlNodeList nodeList = iterationsTree.ChildNodes;
currentIterationNode = FindCurrentIteration(nodeList);
String currentIterationPath = currentIterationNode.Attributes["Path"].Value;
}
}
}

How to add/update milestones in a feature using rally rest API and C#?

I am not able to add or update milestones field for the Features in the Rally. If anyone having the code available using C# to update the same, please share with me. I am searching and doing from last one week with no luck.
When I am trying to add/Update milestones in the Features. I am getting the error as "Could not read: Could not read referenced object null". My code is as follows:-
public DynamicJsonObject UpdateFeaturesbyName(string fea, string bFun)
{
//getting list of Feature.
Request feat = new Request("PortfolioItem/Feature");
feat.Query = new Query("Name", Query.Operator.Equals, fea);
QueryResult TCSResults = restApi.Query(feat);
foreach (var res in TCSResults.Results)
{
var steps = res["Milestones"];
Request tsteps = new Request(steps);
QueryResult tstepsResults = restApi.Query(tsteps);
foreach (var item in tstepsResults.Results)
{
}
if (res.Name == fea)
{
var targetFeature = TCSResults.Results.FirstOrDefault();
DynamicJsonObject toUpdate = new DynamicJsonObject();
//toUpdate["Milestones"] = "";
// CreateResult createResult = restApi.Create(steps._ref, toUpdate);
// String contentRef = steps._ref;
//String contentRef = createResult._ref;
string[] value = null;
string AccCri = string.Empty;
if (!string.IsNullOrWhiteSpace(bFun))
{
value = bFun.Split(new string[] { "\r\n", "\n" }, StringSplitOptions.None);
foreach (string item in value)
{
//if (string.IsNullOrWhiteSpace(AccCri))
// AccCri = item;
//else
// AccCri = AccCri + "<br/>" + item;
if (!string.IsNullOrWhiteSpace(item))
{
//Query for Milestone.
Request ms = new Request("Milestone");
ms.Fetch = new List<string>() { "Name", "ObjectID" };
ms.Query = new Query("Name", Query.Operator.Equals, item);
QueryResult msResults = restApi.Query(ms);
var targetMLResult = msResults.Results.FirstOrDefault();
long MLOID = targetMLResult["ObjectID"];
DynamicJsonObject tarML = restApi.GetByReference("Milestone", MLOID, "Name", "_ref", "DisplayColor");
DynamicJsonObject targetML = new DynamicJsonObject();
targetML["Name"] = tarML["Name"];
//targetML["_ref"] = tarML["_ref"];
targetML["_ref"] = "/milestone/" + Convert.ToString(MLOID);
targetML["DisplayColor"] = tarML["DisplayColor"];
// Grab collection of existing Milestones.
var existingMilestones = targetFeature["Milestones"];
long targetOID = targetFeature["ObjectID"];
// Milestones collection on object is expected to be a System.Collections.ArrayList.
var targetMLArray = existingMilestones;
var tagList2 = targetMLArray["_tagsNameArray"];
tagList2.Add(targetML);//
//targetMLArray.Add(targetML);
targetMLArray["_tagsNameArray"] = tagList2;
toUpdate["Milestones"] = targetMLArray;
OperationResult updateResult = restApi.Update(res._ref, toUpdate);
bool resp = updateResult.Success;
}
}
}
//toUpdate["c_AcceptanceCriteria"] = AccCri;
//OperationResult updateResult = restApi.Update(res._ref, toUpdate);
}
}
var features = TCSResults.Results.Where(p => p.Name == fea).FirstOrDefault();
var featuresref = features._ref;
return features;
}

Now that v3.1.1 of the toolkit has been released you can use the AddToCollection method to do this.
Otherwise, you can still always just update the full collection. The value should be an arraylist of objects with _ref properties.
Check out this example (which adds tasks to defects, but should be very similar to what you're doing): https://github.com/RallyCommunity/rally-dot-net-rest-apps/blob/master/UpdateTaskCollectionOnDefect/addTaskOnDefect.cs

How to convert TermCollection to Tree object via CSOM

I'm querying a SharePoint 2013 Term Store via the SharePoint Client Object Model in order to get a TermCollection.
I'd like to bind the results to a WPF TreeView control. Any idea how I can turn the TermCollection into something that the TreeView will understand?
public static TermCollection GetTaxonomyTerms(string webUrl, string libraryTitle, string fieldTitle)
{
var context = new ClientContext(webUrl);
var web = context.Web;
var list = web.Lists.GetByTitle(libraryTitle);
var fields = list.Fields;
var field = context.CastTo<TaxonomyField>(fields.GetByInternalNameOrTitle(fieldTitle));
context.Load(field);
var termStores = TaxonomySession.GetTaxonomySession(context).TermStores;
context.Load(termStores);
context.ExecuteQuery(); // TODO: Can this ExecuteQuery be avoided by using a LoadQuery statement?
var termStore = termStores.Where(t => t.Id == field.SspId).FirstOrDefault();
var termSet = termStore.GetTermSet(field.TermSetId);
var terms = termSet.GetAllTerms(); //TODO: Do we need a version that returns a paged set of terms? or queries the server again when a node is expanded?
context.Load(terms);
context.ExecuteQuery();
return terms;
}

I ended up writing my own code (please let me know if there's an easier way to do this).
My 'Term' object below is just a simple POCO with Name and Terms.
var terms = SharePointHelper.GetTaxonomyTerms(webUrl, libraryTitle, fieldTitle);
var term = terms.AsRootTreeViewTerm();
....
}
public static Term AsRootTreeViewTerm(this SP.TermCollection spTerms)
{
var root = new Term();
foreach (SP.Term spTerm in spTerms)
{
List<string> names = spTerm.PathOfTerm.Split(';').ToList();
var term = BuildTerm(root.Terms, names);
if (!root.Terms.Contains(term))
root.Terms.Add(term);
}
return root;
}
static Term BuildTerm(IList<Term> terms, List<string> names)
{
Term term = terms.Where(x => x.Name == names.First())
.DefaultIfEmpty(new Term() { Name = names.First() })
.First();
names.Remove(names.First());
if (names.Count > 0)
{
Term child = BuildTerm(term.Terms, names);
if (!term.Terms.Contains(child))
term.Terms.Add(child);
}
return term;
}

Get creation date for the file with SharpSvn

I'm playing with SharpSvn and have to analyze folder in repository
and I need to know when some file has be created there,
not last modification date, but when it was created,
Do you have any idea how to do that?
I started with the following:
Collection<SvnLogEventArgs> logitems;
var c = client.GetLog(new Uri(server_path), out logitems);
foreach (var i in logitems)
{
var properties = i.CustomProperties;
foreach (var p in properties)
{
Console.WriteLine(p.ToString());
Console.WriteLine(p.Key);
Console.WriteLine(p.StringValue);
}
}
But I don't see any creation date there.
Does someone know where to get it?

Looks like I can't do that. Here is how I solved this problem:
I'm getting the time if it was the SvnChangeAction.Add.
Here is the code (SvnFile is my own class, not from SharpSvn):
public static List<SvnFile> GetSvnFiles(this SvnClient client, string uri_path)
{
// get logitems
Collection<SvnLogEventArgs> logitems;
client.GetLog(new Uri(uri_path), out logitems);
var result = new List<SvnFile>();
// get craation date for each
foreach (var logitem in logitems.OrderBy(logitem => logitem.Time))
{
foreach (var changed_path in logitem.ChangedPaths)
{
string filename = Path.GetFileName(changed_path.Path);
if (changed_path.Action == SvnChangeAction.Add)
{
result.Add(new SvnFile() { Name = filename, Created = logitem.Time });
}
}
}
return result;
}

Slightly different code than the previous one:
private static DateTime findCreationDate(SvnClient client, SvnListEventArgs item)
{
Collection<SvnLogEventArgs> logList = new Collection<SvnLogEventArgs>();
if (item.BasePath != "/" + item.Name)
{
client.GetLog(new Uri(item.RepositoryRoot + item.BasePath + "/" + item.Name), out logList);
foreach (var logItem in logList)
{
foreach (var changed_path in logItem.ChangedPaths)
{
string filename = Path.GetFileName(changed_path.Path);
if (filename == item.Name && changed_path.Action == SvnChangeAction.Add)
{
return logItem.Time;
}
}
}
}
return new DateTime();
}