I previously asked the question and got answer to Best approach to write query but the problem is that if you have to save this result in a list then there duplication of records. For example
the resultant table of the join given EXAMPLE
See there are duplicate rows. How can you filter them out, and yet save the data of order number?
Of course there may be some ways but I am looking for some great ways
How can we store the data in list and not create duplicate rows in list?
My current code for my tables is
int lastUserId = 0;
sql_cmd = new SqlCommand();
sql_cmd.Connection = sql_con;
sql_cmd.CommandText = "SELECT * FROM AccountsUsers LEFT JOIN Accounts ON AccountsUsers.Id = Accounts.userId ORDER BY AccountsUsers.accFirstName";
SqlDataReader reader = sql_cmd.ExecuteReader();
if (reader.HasRows == true)
{
Users userToAdd = new Users();
while (reader.Read())
{
userToAdd = new Users();
userToAdd.userId = int.Parse(reader["Id"].ToString());
userToAdd.firstName = reader["accFirstName"].ToString();
userToAdd.lastName = reader["accLastName"].ToString();
lastUserId = userToAdd.userId;
Websites domainData = new Websites();
domainData.domainName = reader["accDomainName"].ToString();
domainData.userName = reader["accUserName"].ToString();
domainData.password = reader["accPass"].ToString();
domainData.URL = reader["accDomain"].ToString();
userToAdd.DomainData.Add(domainData);
allUsers.Add(userToAdd);
}
}
For second table I have custom list that will hold the entries of all the data in second table.
The table returned is table having joins and have multiple rows for same
Besides using the Dictionary idea as answered by Antonio Bakula...
If you persist the dictionary of users and call the code in your sample multiple times you should consider that a user account is either new, modifed, or deleted.
The algorithm to use is the following when executing your SQL query:
If row in query result is not in dictionary create and add new user to the dictionary.
If row in query result is in dictionary update the user information.
If dictionary item not in query result delete the user from the dictionary.
I'd also recommend not using SELECT *
Use only the table columns your code needs, this improves the performance of your code, and prevents a potential security breach by returning private user information.
i am not sure why are you not using distinct clause in your sql to fetch unique results. also that will be faster. did you look at using hashtables.
I would put users into Dictonary and check if allready exists, something like this :
Dictionary<int, Users> allUsers = new Dictionary<int, Users>()
and then in Reader while loop :
int userId = int.Parse(reader["Id"].ToString());
Users currUser = allUsers[userId];
if (currUser == null)
{
currUser = new Users();
currUser.userId = userId);
currUser.firstName = reader["accFirstName"].ToString();
currUser.lastName = reader["accLastName"].ToString();
allUsers.Add(userID, currUser);
}
Websites domainData = new Websites();
domainData.domainName = reader["accDomainName"].ToString();
domainData.userName = reader["accUserName"].ToString();
domainData.password = reader["accPass"].ToString();
domainData.URL = reader["accDomain"].ToString();
currUser.DomainData.Add(domainData);
Seems like the root of your problem is in your database table.
When you said duplicate data rows, are you saying you get duplicate entries in the list or you have duplicate data in your table?
Give 2 rows that are duplicate.
Two options:
First, prevent pulling duplicate data from sql by using a distinct clause like:
select distinct from where
Second option as mentioned Antonio, is to check if the list already has it.
First option is recommended unless there are other reasons.
Related
The application I am building allows a user to upload a .csv file containing multiple rows and columns of data. Each row contains a unique varchar Id. This will ultimately fill in fields of an existing SQL table where there is a matching Id.
Step 1: I am using LinqToCsv and a foreach loop to import the .csv fully into a temporary table.
Step 2: Then I have another foreach loop where I am trying to loop the rows from the temporary table into an existing table only where the Ids match.
Controller Action to complete this process:
[HttpPost]
public ActionResult UploadValidationTable(HttpPostedFileBase csvFile)
{
var inputFileDescription = new CsvFileDescription
{
SeparatorChar = ',',
FirstLineHasColumnNames = true
};
var cc = new CsvContext();
var filePath = uploadFile(csvFile.InputStream);
var model = cc.Read<Credit>(filePath, inputFileDescription);
try
{
var entity = new TestEntities();
var tc = new TemporaryCsvUpload();
foreach (var item in model)
{
tc.Id = item.Id;
tc.CreditInvoiceAmount = item.CreditInvoiceAmount;
tc.CreditInvoiceDate = item.CreditInvoiceDate;
tc.CreditInvoiceNumber = item.CreditInvoiceNumber;
tc.CreditDeniedDate = item.CreditDeniedDate;
tc.CreditDeniedReasonId = item.CreditDeniedReasonId;
tc.CreditDeniedNotes = item.CreditDeniedNotes;
entity.TemporaryCsvUploads.Add(tc);
}
var idMatches = entity.PreexistingTable.Where(x => x.Id == tc.Id);
foreach (var number in idMatches)
{
number.CreditInvoiceDate = tc.CreditInvoiceDate;
number.CreditInvoiceNumber = tc.CreditInvoiceNumber;
number.CreditInvoiceAmount = tc.CreditInvoiceAmount;
number.CreditDeniedDate = tc.CreditDeniedDate;
number.CreditDeniedReasonId = tc.CreditDeniedReasonId;
number.CreditDeniedNotes = tc.CreditDeniedNotes;
}
entity.SaveChanges();
entity.Database.ExecuteSqlCommand("TRUNCATE TABLE TemporaryCsvUpload");
TempData["Success"] = "Updated Successfully";
}
catch (LINQtoCSVException)
{
TempData["Error"] = "Upload Error: Ensure you have the correct header fields and that the file is of .csv format.";
}
return View("Upload");
}
The issue in the above code is that tc is inside the first loop, but the matches are defined after the loop with var idMatches = entity.PreexistingTable.Where(x => x.Id == tc.Id);, so I am only getting the last item of the first loop.
If I nest the second loop then it is way to slow (stopped it after 10 minutes) because there are roughly 1000 rows in the .csv and 7000 in the preexisting table.
Finding a better way to do this is plaguing me. Pretend that the temporary table didn't even come from a .csv and just think about the most efficient way to fill in rows in table 2 from table 1 where the id of that row matches. Thanks for your help!
As your code is written now, much of the work is being done by the application that could much more efficiently be done by SQL Server. You are making hundreds of unnecessary roundtrip calls to the database. When you are mass importing data you want a solution like this:
Bulk import the data. See this answer for helpful guidance on bulk import efficiency with EF.
Join and update destination table.
Processing the import should only require a single mass update query:
update PT set
CreditInvoiceDate = CSV.CreditInvoiceDate
,CreditInvoiceNumber = CSV.CreditInvoiceNumber
,CreditInvoiceAmount = CSV.CreditInvoiceAmount
,CreditDeniedDate = CSV.CreditDeniedDate
,CreditDeniedReasonId = CSV.CreditDeniedReasonId
,CreditDeniedNotes = CSV.CreditDeniedNotes
from PreexistingTable PT
join TemporaryCsvUploads CSV on PT.Id = CSV.Id
This query would replace your entire nested loop and apply the same update in a single database call. As long as your table is indexed properly this should run very fast.
After saving CSV record into second table which have same fileds as your primary table, execute following procedure in sqlserver
create proc [dbo].[excel_updation]
as
set xact_abort on
begin transaction
-- First update records
update first_table
set [ExamDate] = source.[ExamDate],
[marks] = source.[marks],
[result] = source.[result],
[dob] = source.[dob],
[spdate] = source.[spdate],
[agentName] = source.[agentName],
[companycode] = source.[companycode],
[dp] = source.[dp],
[state] = source.[state],
[district] = source.[district],
[phone] = source.[phone],
[examcentre] = source.[examcentre],
[examtime] = source.[examtime],
[dateGiven] = source.[dateGiven],
[smName] = source.[smName],
[smNo] = source.[smNo],
[bmName] = source.[bmName],
[bmNo] = source.[bmNo]
from tbUser
inner join second_table source
on tbUser.[UserId] = source.[UserId]
-- And then insert
insert into first_table (exprdate, marks, result, dob, spdate, agentName, companycode, dp, state, district, phone, examcentre, examtime, dateGiven, smName, smNo, bmName, bmNo)
select [ExamDate], [marks], [result], [dob], [spdate], [agentName], [companycode], [dp], [state], [district], [phone], [examcentre], [examtime], [dateGiven], [smName], [smNo], [bmName], [bmNo]
from second_table source
where not exists
(
select *
from first_table
where first_table.[UserId] = source.[UserId]
)
commit transaction
delete from second_table
The condition of this code is only that both table must have same id matching data. Which id match in both table, data of that particular row will be updated in first table.
As long as the probability of the match is high you should simply attempt update with every row from your CSV, with a condition that the id matches,
UPDATE table SET ... WHERE id = #id
I have a CheckedListbox which contains values from some table called products.
The idea is to check the products that are associated to a customer. Now it does save correctly in an link table, yet when loading it again, the items that were checked do not get loaded correctly into the CheckedListbox.
So from that link table where, I would like to get all rows from just one column. All tables are already loaded into the application so I don't want to use sql.
I've tried using linq, with no success, Ids is just empty here.
int[] Ids = (from m in dataset.Tables["LinkTable"].AsEnumerable()
where m.Field<int>("customerId") == customerId
select m.Field<int>("productId")).ToArray();
Then, if I do succeed to get those Id's, I would like to get the indexes of those primary keys so I can set the correct products to checked.
I've tired doing it like this, but this gives me error in other parts of the program, because I am setting a Primary key to a global datatable. Datagridviews don't like that.
DataColumn[] keyColumns = new DataColumn[1];
keyColumns[0] = dataset.Tables["products"].Columns["Id"];
currentPatient.GetTheDataSet.Tables["products"].PrimaryKey = keyColumns;
foreach (int Id in Ids)
{
DataRow row = dataset.Tables["Products"].Rows.Find(Id);
int index = dataset.Tables["Products"].Rows.IndexOf(row);
clbMedications.SetItemChecked(index, true);
}
I would like to do that last part without specifying a primary key, I couldn't find how to do that in linq.
I know it consists of 2 questions, but perhaps this can be done with just one linq statement so I better combine them.
[EDIT]
Finally, i think i've got what you need:
var qry = (from p in ds.Tables["products"].AsEnumerable()
select new {
Id = p.Field<int>("Id"),
Index = ds.Tables["products"].Rows.IndexOf(p),
Checked = ds.Tables["LinkTable"].AsEnumerable().Any(x=>x.Field<int>("productId") == p.Field<int>("Id") && x.Field<int>("customerId")==customerid)
}).ToList();
Above query returns the list, which you can bnid with CheckedListbox.
I'm new to writing LINQ queries, and I'm having trouble with string comparisons. I'm loading the data into a DataTable (confirmed that table in SQL DB and DataTable have same number of rows), but for some reason I can't find a value that I know exists in both.
The text box contains 'servername' while the datarows contain 'servername.mydomain.net', so here's what my code looks like
string strParameter = txtAutoComplete.ToString().ToLower();
//WUG TableAdapter and DataTable
dsCIInfoTableAdapters.DeviceTableAdapter taWUG;
taWUG = new dsCIInfoTableAdapters.DeviceTableAdapter();
dsCIInfo.DeviceDataTable dtWUG = new dsCIInfo.DeviceDataTable();
taWUG.Fill(dtWUG);
var qstWUG = (from row in dtWUG.AsEnumerable()
where row.Field<string>("sDisplayName").ToLower().Contains(strParameter)
select row.Field<string>("sDisplayName"));
Beleive in your LINQ statement dtWUG needs to be dtWUG.AsEnumerable(). Linq only works on data sources that implement the IEnumerable Interface.
You can debug it easier if you add some let statements where you can add breakpoints:
var qstWUG = (from row in dtWUG
let display = row.Field<string>("sDisplayName")
let lower = display.ToLower()
let contains = lower.Contains(strParameter)
where contains
select display).ToArray();
Also convert it to an array using .ToArray() at the end, will make it execute immediately (LINQ is lazy by paradigm, doesn't execute until it's needed), and also easier to look at in subsequent breakpoints.
Yeah, I feel stupid... I forgot to use the textbox.text to assign it to a string
string strParameter = txtAutoComplete.Text.ToLower();
//WUG TableAdapter and DataTable
dsCIInfoTableAdapters.DeviceTableAdapter taWUG;
taWUG = new dsCIInfoTableAdapters.DeviceTableAdapter();
dsCIInfo.DeviceDataTable dtWUG = new dsCIInfo.DeviceDataTable();
taWUG.Fill(dtWUG);
var qstWUG = (from row in dtWUG.AsEnumerable()
let display = row.Field<string>("sDisplayName")
where display.ToLower().Contains(strParameter)
select display).ToArray();
I'm stumped on this one.
I'm trying to merge two DataTables into one. Preferably I would use linq to perform this task, but the problem is I need to add conditions for the join dynamically. The data for each table comes from two different calls to stored procedures and which calls are used can be switched. The results can therefor vary in number of columns and which primary keys are available.
The goal is to replace regular strings in the first result set with a second database that can contain unicode (but only if it contains a value for that specific combination of primary keys).
My linq query would look like this:
var joined = (from DataRow reg in dt1.Rows
join DataRow uni in dt2.Rows
on new { prim1 = reg.ItemArray[0], prim2 = reg.ItemArray[1] }
equals new { prim1 = uni.ItemArray[0], prim2 = uni.ItemArray[1] }
select new
{
prim1 = reg.ItemArray[0],
prim2 = reg.ItemArray[1],
value1 = reg.ItemArray[4],
value2 = uni.ItemArray[3] ?? reg.ItemArray[3]
}
);
This works perfectly for what I want, but as I said I need to be able to define which columns in each table are primary keys, so this:
join DataRow uni in dt2.Rows
on new { prim1 = reg.ItemArray[0], prim2 = reg.ItemArray[1] }
equals new { prim1 = uni.ItemArray[0], prim2 = uni.ItemArray[1] }
needs to be replaced by something like creating a DataRelation between the tables or before performing the linq adding the primary keys dynamically.
ALSO, I need to make the select something like SQLs * instead of specifying each column, as I do not know the number of columns in the first result set.
I've also tried joining the tables by adding primary keys and doing a merge, but how do I then choose which column in dt2 to overwrite which one in dt1?
DataTable join = new DataTable("joined");
join = dt1.Copy();
join.Merge(dt2, false, MissingSchemaAction.Add);
join.AcceptChanges();
I'm using VS2012.
I ended up using a very simple approach, which doesn't involve creating primary key relations or joins at all. I'm sure there are more elegant or performance effective ways of solving the problem.
Basically I've adapted the solution in Linq dynamically adding where conditions, where instead of joining I dynamically add .Where-clauses.
That way I can loop through the rows and compare for each dynamically added primary key:
foreach (DataRow regRow in dt1.Rows)
{
//Select all rows in second result set
var uniRows = (from DataRow uniRow in dt2.Rows select uniRow);
//Add where clauses as needed
if (firstCondition) { uniRows = uniRows.Where(x => x["SalesChannel"] == "001"); }
else if (secondCondition) { uniRows = uniRows.Where(x => x["Language"] == "SV"); }
else (thirdCondition) { uniRows = uniRows.Where(x => x["ArticleNo"] == "242356"); }
// etc...
}
Each row gets compared to a diminishing list of rows in the second result set.
I have 2 data tables, one in which i have my data(D1) with a unique data Id and in another data table(D2) i have the Ids of all the records of my data table(D1) in a particular order.How can i sort my data table(D1) based on the order of Ids in D2.I am using c# asp.net
You could copy the rows in the ordering table into a Dicationary with an index. Assuming your key field is named Key the code might look like this:
static void Main(string[] args)
{
var dt = new DataTable("Data");
var dtOrder = new DataTable("Order");
// Insert some data here
int i = 0;
var orderDict = new Dictionary<object, int>();
foreach(DataRow row in dtOrder.Rows)
{
orderDict.Add(row["Key"], ++i);
}
var ordered = dt.Rows.Cast<DataRow>().OrderBy(r => orderDict[r["Key"]]);
}
As I read Peaceman71's comment, I think it is worth mentioning that this is a disconnected approach. Any proper database software will do this for you as well.
It depends on if you want to do this in the code or in the database.
In the database you would join the two tables, such as (MS-SQL/T-SQL):
SELECT D2.Sort, D1.* FROM D2 LEFT JOIN D1 ON D2.ID = D1.ID ORDER BY D2.Sort
In code it depends very much on where you keep the data. DataSet, DataTables etc.