We Keep Coding

Compare Two Complex List of data

When I am Trying to save current list of data into database, I need to get already existing data from database, and need to compare with current list of data.
I have two lists one is PreviousList(existing data from DB) and other is CurrentList(Modified data)
public class SoftClose
{
public int ID = -1;
public int AID = -1;
public int WFID = -1;
public string PREFIX;
public DateTime SCDATE;
public string STATUS;
}
In CurrentList I modified Prefix to D2 where ID=1 and added new row(Id=4)...
My req is
When I am trying to save CurrentList to Db,
If there is any new Prefix in CurrentList that is not there in PreviousList I need to insert that new row and need to change Status to ADD for that row.
I changed Prefix to D2 where Id = 1 in CurrentList. D1 is there is DB and but not in CurrentList so i need to delete it. So i need to change the status to DELETE for that record. I should not insert D2 record where id=1 becuase D2 is already there. If I changed to D5 where Id = 1 then I need to insert it because D5 is not there in DB So i need to change the status to UPDATE.
How to do this? What is the best approach to compare lists

here is a solution you could try:
List<SoftClose> previousList = new List<SoftClose>(){
new SoftClose(){ID=1, Status = "NO_CHANGE",AID="19", Prefix = "D1"},
new SoftClose(){ID=2, Status = "NO_CHANGE",AID="20", Prefix = "D2"},
new SoftClose(){ID=3, Status = "NO_CHANGE",AID="21", Prefix = "D3"}
};
List<SoftClose> currentList = new List<SoftClose>(){
new SoftClose(){ID=1, Status = "NO_CHANGE",AID="19", Prefix = "D2"},
new SoftClose(){ID=2, Status = "NO_CHANGE",AID="20", Prefix = "D2"},
new SoftClose(){ID=3, Status = "NO_CHANGE",AID="21", Prefix = "D6"},
new SoftClose(){ID=4, Status = "NO_CHANGE",AID="22", Prefix = "D4"},
new SoftClose(){ID=5, Status = "NO_CHANGE",AID="22", Prefix = "D5"}
};
var addlist = currentList.Where(c => previousList.All(p => !p.ID.Equals(c.ID) && !p.Prefix.Equals(c.Prefix)));
foreach(var n in addlist)
{
var index = currentList.FindIndex(p => p.Prefix.Equals(n.Prefix));
currentList[index].Status = "ADD";
}
var updateORdeletelist = currentList.Where(c => c.Status.Equals("NO_CHANGE") && previousList.Exists(p => p.ID.Equals(c.ID) && !p.Prefix.Equals(c.Prefix)));
foreach (var n in updateORdeletelist)
{
var index = currentList.FindIndex(p => p.Prefix.Equals(n.Prefix));
if (previousList.FindIndex(p => p.Prefix.Equals(n.Prefix)) < 0)
currentList[index].Status = "UPDATE";
else
currentList[index].Status = "DELETE";
}
foreach (var item in currentList)
{
Console.WriteLine($"Id:{item.ID}, Desc1:{item.Prefix}, Status:{item.Status}");
}
output
Id:1, Desc1:D2, Status:DELETE
Id:2, Desc1:D2, Status:NO_CHANGE
Id:3, Desc1:D6, Status:UPDATE
Id:4, Desc1:D4, Status:ADD
Id:5, Desc1:D5, Status:ADD

There is a tool called Side by Side SQL Comparer in C# at https://www.codeproject.com/Articles/27122/Side-by-Side-SQL-Comparer-in-C.
basic use of the component:
using (TextReader tr = new StreamReader(#"c:\1.sql"))
{
sideBySideRichTextBox1.LeftText = tr.ReadToEnd();
}
using (TextReader tr = new StreamReader(#"c:\2.sql"))
{
sideBySideRichTextBox1.RightText = tr.ReadToEnd();
}
sideBySideRichTextBox1.CompareText();
You load the left and right sides to their respective variables sideBySideRichTextBox1.LeftText and sideBySideRichTextBox1.RightText and compare them with sideBySideRichTextBox1.CompareText();
In your case the 1.sql and 2.sql would be your PreviousList and CurrentList -database files.
There is more detailed documentation at the project-site.

How can I insert more than 2 tables in LINQ using foreign keys

I am apparently Inserting data using LINQ by creating classes of tables in the databases but it just has error that says object is null.
This is my sample code using C# LINQ:
using (dc = new linqDBDataContext(conn))
{
Subject_Curriculum sc;
Subject_Schedule ss;
Subject_Department sd;
Subject_Standing sst;
Pre_Requisite pr;
Pre_Requisite_Year_Standing prys;
Curriculum cu = new Curriculum();
cu.Curriculum_Title = curriculumName;
cu.Course_Number = courseNumber;
foreach (var s in ssd)
{
sc = new Subject_Curriculum();
sc.Course_Code = s.courseCode;
sc.Course_Title = s.courseTitle;
cu.Subject_Curriculums.Add(sc);
dc.Subject_Curriculums.InsertOnSubmit(sc);
for (int i = 0; i < s.numberOfSchedules; i++)
{
ss = new Subject_Schedule();
if (i == 0)
{
ss.Units = s.unitsLec;
ss.Schedule_Type = "Lecture";
ss.Number_Of_Hours = s.numberOfHoursLec;
}
else
{
ss.Units = s.unitsLab;
ss.Schedule_Type = "Laboratory";
ss.Number_Of_Hours = s.numberOfHoursLab;
}
sc.Subject_Schedules.Add(ss);
dc.Subject_Schedules.InsertOnSubmit(ss);
}
foreach (var sdl in s.department)
{
sd = new Subject_Department();
sd.Department_Number = sdl;
sc.Subject_Departments.Add(sd);
dc.Subject_Departments.InsertOnSubmit(sd);
}
sst = new Subject_Standing();
sst.Year = s.year;
sst.Semester = s.semester;
cu.Subject_Standings.Add(sst);
dc.Subject_Standings.InsertOnSubmit(sst);
if (s.yearStandingStatus)
{
prys = new Pre_Requisite_Year_Standing();
prys.Year_Standing = Convert.ToInt32(s.yearStanding.ToString().Substring(0, 1));
sc.Pre_Requisite_Year_Standings.Add(prys);
dc.Pre_Requisite_Year_Standings.InsertOnSubmit(prys);
}
else
{
if (s.prereq.Count == 0)
{
pr = new Pre_Requisite();
pr.Pre_Requisite_Code = null;
sc.Pre_Requisites.Add(pr);
dc.Pre_Requisites.InsertOnSubmit(pr);
}
else
{
foreach (var p in s.prereq)
{
pr = new Pre_Requisite();
pr.Pre_Requisite_Code = Convert.ToInt32(p);
sc.Pre_Requisites.Add(pr);
dc.Pre_Requisites.InsertOnSubmit(pr);
}
}
}
}
dc.Curriculums.InsertOnSubmit(cu);
dc.SubmitChanges();
return true;
}
As you can see in the code, the Curriculum table has the highest hierarchy in the database and the other tables inherits its primary key into Subject_Curriculum, Pre_Requisite, Subject_Standing and Pre_Requisite_Year_Standing. While Subject_Schedules and Subject_Department inherits Subject_Curriculum's primary key. What can I do to make this insertion possible to all table at once?

I already solved my question. It is just by adding all tables from their foreign keys and insert and submit changes at the end of the loop. This makes this thread close.

.NET CSV Uploader Allow Nulls

I've put together a CSV importer which I assume works, though I get this error, how do I allow this column to be null so when it adds it to the table it automatically sets the ID? I've tried:
csv.Configuration.WillThrowOnMissingFields = false;
but it doesn't recognise it, this is the error I get when attempting to upload:
CsvHelper.ValidationException: 'Header matching ['ID'] names at index 0 was not found. If you are expecting some headers to be missing and want to ignore this validation, set the configuration HeaderValidated to null. You can also change the functionality to do something else, like logging the issue.'
[HttpPost]
[ActionName("CreateBulk")]
public ActionResult CreateBulkUpload()
{
object db;
var file = Request.Files["attachmentcsv"];
using (var csv = new CsvReader(new StreamReader(file.InputStream), true))
{
var records = csv.GetRecords<Client>().ToList();
foreach (var item in records)
{
var strip = item.homePage.Replace("https://www.", "").Replace("http://www.", "")
.Replace("https://", "").Replace("http://", "").Replace("www.", "");
string[] URLtests =
{"https://www." + strip, "http://www." + strip, "https://" + strip, "http://" + strip};
string[] Metric = MajesticFunctions.MajesticChecker(URLtests);
var userId = User.Identity.GetHashCode();
var UserTableID = 1;
var newclient = new Client
{
clientN = item.clientN,
homePage = Metric[0],
clientEmail = item.clientEmail,
monthlyQuota = item.monthlyQuota,
TrustFlow = Int32.Parse(Metric[1]),
CitationFlow = Int32.Parse(Metric[2]),
RI = Int32.Parse(Metric[3]),
MJTopicsID = item.MJTopicsID,
UserTableID = UserTableID
};
ViewBag.newdomain = newclient;
return RedirectToAction("Index");
}
}
return RedirectToAction("Index");
}

Did you try out the suggestion mentioned in the error message?
like this?
csv.configuration.HeaderValidated = null;

The developer made some breaking changes this year, so the accepted answer will no longer work.
Instead, you have to create a configuration object in advance and inject it in the constructor:
var config = new CsvConfiguration(CultureInfo.InvariantCulture)
{
HeaderValidated = null
};
using (var reader = new StreamReader(file))
using (var csv = new CsvReader(reader, config))

Make sure to include both these lines:
csv.Configuration.HeaderValidated = null;
csv.Configuration.MissingFieldFound = null;

C# Reading CSV file with SQL conditions

I am using CsvHelper lib to read CSV file and I can successfully read the file with the lib. However I cannot use SQL condition to filter values. How can I do that without using SQL Server. I am really stuck on it.
It was very easy with Pandas and Pandasql libs in Python but it is being too hard in C#..
My Code:
public static void Main(string[] args)
{
var fileInfo = new FileInfo(#"filePath");
using (TextReader reader = fileInfo.OpenText())
using (var csvReader = new CsvReader(reader))
{
csvReader.Configuration.Delimiter = ",";
csvReader.Configuration.HasHeaderRecord = false;
csvReader.Configuration.IgnoreQuotes = true;
csvReader.Configuration.TrimFields = true;
csvReader.Configuration.WillThrowOnMissingField = false;
while (csvReader.Read())
{
var myStrinVar = csvReader.GetField<string>(0);
Console.Write(myStrinVar); //SELECT * FROM table...
}
}
}

I would suggest using LINQ to filter your results.
https://msdn.microsoft.com/en-us/library/bb397906.aspx
Say you have some class MyClass that you can serialize the lines in your file into.
For example:
public class MyClass
{
public int ID { get; set; }
}
var records = csv.GetRecords<MyClass>().ToList();
var filtered = records.Where(r => r.ID >= 10);
That example is a bit contrived but you can use any boolean expression you like in the where clause.

I know this is too late for OP, but the issue with the accepted answer is that you have to read in the entire result set to memory which may not be tenable for large files. Also, if you can extend this code below to get the top N rows without having to read the entire CSV if you find matches early in the file.
public static void Main(string[] args)
{
var fileInfo = new FileInfo(#"filePath");
var where = ""; //Code to set up where clause part of query goes here
using (TextReader reader = fileInfo.OpenText())
using (var csvReader = new CsvReader(reader))
{
csvReader.Configuration.Delimiter = ",";
csvReader.Configuration.HasHeaderRecord = false;
csvReader.Configuration.IgnoreQuotes = true;
csvReader.Configuration.TrimFields = true;
csvReader.Configuration.WillThrowOnMissingField = false;
DataTable dt = null;
while (csvReader.Read())
{
//Use the first row to initialize the columns.
if (dt == null)
{
dt = new DataTable();
for (var i = 0; i < csvReader.FieldCount; i++)
{
var fieldType = csvReader.GetFieldType(i);
DataColumn dc;
if (fieldType.IsNullableType())
{
dc = new DataColumn(csvReader.GetName(i), Nullable.GetUnderlyingType(fieldType));
dc.AllowDBNull = true;
}
else
dc = new DataColumn(csvReader.GetName(i), data.GetFieldType(i));
dt.Columns.Add(dc);
}
}
//Map DataReader to DataRow
var newRow = dt.Rows.Add();
foreach(DataColumn col in dt.Columns)
{
newRow[col.ColumnName] = csvReader[col.ColumnName];
}
//Create a temporary DataView and filter it with the where clause.
DataView dv = new DataView(dt);
dv.RowFilter = where;
var data = dv.Count > 0 ? dv[0] : null;
if(data != null)
{
//Row in here matches your where clause.
//Code to read this row or do something with it.
}
//Empty the temporary data table.
dt.Rows.Clear();
}
}
}

Trying to find the quickest (least costly) way to loop values in from a .csv through two tables

The application I am building allows a user to upload a .csv file, which will ultimately fill in fields of an existing SQL table where the Ids match. First, I am using LinqToCsv and a foreach loop to import the .csv into a temporary table. Then I have another foreach loop that loops the fields from the temporary table into an existing table where the Ids match. The only way I have gotten this to work consistently and successfully is nesting the second foreach loop within the first:
[HttpPost]
public ActionResult UploadValidationTable(HttpPostedFileBase csvFile)
{
var inputFileDescription = new CsvFileDescription
{
SeparatorChar = ',',
FirstLineHasColumnNames = true
};
var cc = new CsvContext();
var filePath = uploadFile(csvFile.InputStream);
var model = cc.Read<Credit>(filePath, inputFileDescription);
try
{
var entity = new TestEntities();
foreach (var item in model)
{
var tc = new TemporaryCsvUpload
{
Id = item.Id,
CreditInvoiceAmount = item.CreditInvoiceAmount,
CreditInvoiceDate = item.CreditInvoiceDate,
CreditInvoiceNumber = item.CreditInvoiceNumber,
CreditDeniedDate = item.CreditDeniedDate,
CreditDeniedReasonId = item.CreditDeniedReasonId,
CreditDeniedNotes = item.CreditDeniedNotes
};
entity.TemporaryCsvUploads.Add(tc);
var idMatches = entity.Authorizations.ToList().Where(x => x.Id == tc.Id);
foreach (var number in idMatches)
{
number.CreditInvoiceDate = tc.CreditInvoiceDate;
number.CreditInvoiceNumber = tc.CreditInvoiceNumber;
number.CreditInvoiceAmount = tc.CreditInvoiceAmount;
number.CreditDeniedDate = tc.CreditDeniedDate;
number.CreditDeniedReasonId = tc.CreditDeniedReasonId;
number.CreditDeniedNotes = tc.CreditDeniedNotes;
}
}
entity.SaveChanges();
entity.Database.ExecuteSqlCommand("TRUNCATE TABLE TemporaryCsvUpload");
TempData["Success"] = "Updated Successfully";
}
catch (LINQtoCSVException)
{
TempData["Error"] = "Upload Error: Ensure you have the correct header fields and that the file is of .csv format.";
}
return View("Upload");
}
The issue is speed. It takes about 1 minute and 49 seconds to search through an SQL table of 7000 entries, match the ids, and fill in the fields.
So, I looked at this and thought that the second loop really didn't need to be nested. I switched up the code like so:
[HttpPost]
public ActionResult UploadValidationTable(HttpPostedFileBase csvFile)
{
var inputFileDescription = new CsvFileDescription
{
SeparatorChar = ',',
FirstLineHasColumnNames = true
};
var cc = new CsvContext();
var filePath = uploadFile(csvFile.InputStream);
var model = cc.Read<Credit>(filePath, inputFileDescription);
try
{
var entity = new TestEntities();
var tc = new TemporaryCsvUpload();
foreach (var item in model)
{
tc.Id = item.Id;
tc.CreditInvoiceAmount = item.CreditInvoiceAmount;
tc.CreditInvoiceDate = item.CreditInvoiceDate;
tc.CreditInvoiceNumber = item.CreditInvoiceNumber;
tc.CreditDeniedDate = item.CreditDeniedDate;
tc.CreditDeniedReasonId = item.CreditDeniedReasonId;
tc.CreditDeniedNotes = item.CreditDeniedNotes;
entity.TemporaryCsvUploads.Add(tc);
}
var idMatches = entity.Authorizations.ToList().Where(x => x.Id == tc.Id);
foreach (var number in idMatches)
{
number.CreditInvoiceDate = tc.CreditInvoiceDate;
number.CreditInvoiceNumber = tc.CreditInvoiceNumber;
number.CreditInvoiceAmount = tc.CreditInvoiceAmount;
number.CreditDeniedDate = tc.CreditDeniedDate;
number.CreditDeniedReasonId = tc.CreditDeniedReasonId;
number.CreditDeniedNotes = tc.CreditDeniedNotes;
}
entity.SaveChanges();
entity.Database.ExecuteSqlCommand("TRUNCATE TABLE TemporaryCsvUpload");
TempData["Success"] = "Updated Successfully";
}
catch (LINQtoCSVException)
{
TempData["Error"] = "Upload Error: Ensure you have the correct header fields and that the file is of .csv format.";
}
return View("Upload");
}
This time around, it only took 19 seconds to complete. A vast improvement on the first. But when I checked the database, only one row of the 7 that should match was filled in. Can anybody spot a reason why the second code block would not be filling in all the rows it should be? Or a better way to optimize the first block? Thanks!