I have a DataTable in C# and would like to send it to my SQL CE 4 server. One thing that makes it a bit more complicated is that when it encounters an duplicate, it should either ignore it and move on to the next row in the DataTable. I've looked around but a lot of information I find doesn't seem to work with the CE version of SQL Server. What's an efficient way of doing this?
Filter your DataTable to exclude the duplicate rows before uploading, using the DataTable.Select Method
e.g.
DataTable table = DataSet1.Tables["Orders"];
// Presuming the DataTable has a column named Date.
string expression;
expression = "Date > #1/1/00#"; // you will need logic to remove your duplicates
DataRow[] foundRows;
// Use the Select method to find all rows excluding duplicates
foundRows = table.Select(expression);
// .NET 3.5 onwards
DataTable filteredDataTable = foundRows.copyToDataTable();
Try this Logic.
var dt = new DataTable(); //Supposed that this is your DataTable
foreach(DataRow row in dt.Rows)
{
var find = MyFindMethod("Id"); 1. select statement that find if the id is on database
if(find.Rows > 0)
{
//Id exist do nothing
}
else
{
//Id not exist then 2. Do Insert to sql ce id I not exist
MyInsertMethod("Id");
}
}
Regards
Related
I am assigning data to a Dataset manually from the data of another Dataset, in which two conditions are met one when there is data from another Dataset where the date field matches the rows with this data, otherwise I add them as a new row. This is perfect.
The problem is that when you finish assigning the data to the Dataset, the data that shows first are the ones that are updated with the first condition.
I need the data to be sorted by Date either dd\MM\yyyy or yyyy/MM/dd without taking into account the hours, and regardless of whether it was updated or added, but to order it by Date of ASC mode.
//foreach (DataRow dr in ds.Tables[0].Rows)
foreach (DataRow dr in ds.Tables[0].AsEnumerable().OrderBy(x=>x.Field<DateTime>("date").Date).ToList())
{
var row=dataset_manually.Tables[0].AsEnumerable().Where(x=> x.Field<DateTime>("date").Date == ((DateTime)dr["date"]).Date).FirstOrDefault();
if (row!=null)
{
//Update the data
row["entryToTurn"]=(DateTime)dr["entryToTurn"];
row["departureToTurn"]=(DateTime)dr["departureToTurn"];
row["turn"]=dr["turn"].ToString();
}
else
{
//Add new rows
var row2= dataset_manually.Tables[0].NewRow();
row2["entryToTurn"] = dr["entryToTurn"];
row2["departureToTurn"] = dr["departureToTurn"];
row2["turn"] = dr["turn"].ToString();
dataset_manually.Tables[0].Rows.Add(fila2);
}
}
For some reason the OrderBy in the foreach does not work. adds the data but does not order it. foreach (DataRow dr in ds.Tables[0].AsEnumerable().OrderBy(x=>x.Field<DateTime>("date").Date).ToList())
I tried this but it did not work:
dataset_manually.Tables[0].DefaultView.Sort = "date asc";
And I can not do it like this:
datagridview1.Sort(datagridview1.Columns[0], ListSortDirection.Ascending);
Because this only shows the data ordered in the DataGridView but when using the Dataset in another side the data is not sorted and the positions of the values in the grid are not the same things in the Dataset.
Desired output:
Note: The data is NOT assigned by the index, it must be assigned where the date field matches
Environment: Visual Studio 2010 (WindowsForms C#) & .NET NetFramework 4
Your code is not working because you first sort your data set and then you add new rows to it (so after everything)
Solution for this is slow if you have a lot of rows inside data set so I AM RECOMMENDING TO DO NOT DO THIS BUT SORT DATA AFTER LOADING IT INTO CODE.
Solution to this problem could be:
Load Data from DB into dataGridView
Get dataSource from dataGridView as DataTable
Add new data to DataTable
Sort DataTable
Save sorted data
Sorting could be done with:
DataView dv = data.DefaultView;
dv.Sort = "PrimaryColumn asc" //desc for descending
data = dv.ToTable();
I have a U2/UniVerse database, and need to copy the data in one table into a SQL Server table. The table in question has approx 600,000 rows and just under 200 columns. I didn't create the table, and can't change it.
For other tables, I'm looping thru a UniDataSet one record at a time and adding it to a DataTable, and then using SqlBulkCopy to copy the records to the SQL Server. This works fine, but with the large table I seem to be running out of memory when creating the DataTable.
DataTable dt = new DataTable("myTempTable");
dt.Columns.Add("FirstColumn", typeof(string));
dt.Columns.Add("SecondColumn", typeof(string));
... //adding a bunch more columns here
dt.Columns.Add("LastColumn", typeof(string));
U2Connection con = GetU2Con();
UniSession us1 = con.UniSession;
UniSelectList s1 = us1.CreateUniSelectList(0);
UniFile f1 = us1.CreateUniFile("MyU2TableName")
s1.Select(f1);
UniDataSet uSet = f1.ReadRecords(s1.ReadListAsStringArray());
foreach (UniRecord uItem in uSet)
{
List<String> record = new List<String>(uItem.Record.ToString().Split(new string[] { "รพ" }, StringSplitOptions.None));
DataRow row = dt.NewRow();
row[0] = uItem.RecordID;
row[1] = record[0];
row[2] = record[1];
... //add the rest of the record
row[50] = record[49]
dt.Rows.Add(row);
}
con.Close();
So that copies the records from the UniDataSet into a DataTable. Then, I SqlBulkCopy the DataTable into a SQL table:
string SQLcon = GetSQLCon();
using (SqlBulkCopy sbc = new SqlBulkCopy(SQLcon))
{
sbc.DestinationTableName = "dbo.MySQLTableName";
sbc.BulkCopyTimeout = 0;
sbc.BatchSize = 1000; //I've tried anywhere from 50 to 50000
try
{
sbc.WriteToServer(dt);
}
catch
{
Console.WriteLine(ex.Message);
}
}
This works just fine for my U2 tables that have 50,000 or so rows, but it basically crashes the debugger (VS Express 2012) when the table has 500,000 rows. The PC I'm doing this on is Windows 7 x64 with 4GB ram. The VS process looks like it uses up to 3.5GB RAM before it crashes.
I'm hoping there's a way to write the UniDataSet right to SQL using SqlBulkCopy, but I'm not too familiar with the U2 .Net toolkit.
The problem I face is the UniDataSet records are multivalue, and I need to pick them apart before I can write them to SQL.
Thanks!
DataTable it gets too much bigger in memory, before inserting to Database.
Why don't you split the bulk insert operation? For example read the first 50.000 results and the insert to Sql server database, clear the DataTable Memory and start again with the next 50.000 rows.
if (dt.Rows.Count > 50000)
{
//do SqlbulkCopy
dt.Rows.Clear();
}
In U2 Toolkit for .NET v2.1.0 , we have implemented Native Access. Now you can create DataSet/DataTable from UniData/UniVerse File directly. You can specify WHERE and SORT Clause too. You will see performance improvement as it will not make too much Server Trip to get IDs. For example, if you have 1000 record IDs, it will make 1000 times Server Trip. Whereas if you use Native Access, it will make one Server Trip.
Please download U2 Toolkit for .NET v2.2.0 Hot Fix 1 and try the following code. For more information , please contact u2askus#rocketsoftware.com.
U2Connection con = GetU2Con();
U2Command cmd = lConn.CreateCommand();
cmd.CommandText = string.Format("Action=Select;File=MyU2TableName;Attributes=MyID,FirstColumn,SecondColumn,LastColumn;Where=MyID>0;Sort=MyID");
U2DataAdapter da = new U2DataAdapter(cmd);
DataSet ds = new DataSet();
da.Fill(ds);
DataTable dt = ds.Tables[0];
I have a DataTable I am populating from SQL table with the following example columns
ID
Type
Value
I am populating the DataTable with rows which are of a certain type. I want to select the rows 10 - 20 from my resulting DataTable:
Connect conn = new Connect();
SqlDataAdapter da = new SqlDataAdapter(SQL, conn.Connection());
//Creates data
DataTable d = new DataTable();
da.Fill(d);
DataRow[] result = d.Select();
In the above code I have omitted the main SQL, and currently I have no select for my DataRow array. I cannot find a way to reference the row numbers.
So for instance I am looking for something like Select("rownum > X && rownum < Y")
I have searched here, and a number of other resources to no avail. Any clues would be really handy, or just a simple not possible.
It's always better to select only what you need from the database(f.e. by using the TOP clause or a window function like ROW_NUMBER) instead of filtering it in memory.
However, you can use Linq-To-DataSet and Enumerable.Skip + Enumerable.Take:
var rows = d.AsEnumerable()
.Skip(9).Take(11); // select rows 10-20 as desired (so you want 11 rows)
If you want a new DataTable from the filtered result use CopyToDataTable, if you want a DataRow[] use rows.ToArray().
I would just do the 'take' and 'skip' command and keep it simple:
d.Select().Skip(10).Take(10); // skips 10 rows, then selects ten after that.
This would be assuming you have the Linq using set (using System.Linq)
I have a DataTable imported from Excel file.
Data i need is only unique from specific columns of the DataTable.
The unique data i meant is like when a command DISTINCT is used in SQL Select Query.
I want to get the list of the unique data from the DataTable Column and put them into List
I think LinQ can be used for this matter but i'm not so familiar with it.
I was thinking of code like this below
var data is from MyDataTable
where MyDataTable.ColumnName = "SpecificColumn"
select MyDataTable["SpecificColumn"]).UniqueData;
List<string> MyUniqueData = new List<string>();
foreach(object obj in data)
{
if(MyUniqueData.NotContain(obj))
MyUniqueData.add(obj);
}
I hope someone can drop off some knowledge to me.
var unique = data.Distinct().ToList();
What you're looking for is .Distinct(). See MSDN documentation here. You can specify your own comparer if you need something specific and it will return only unique records.
If you have a Datatable or DataView, inorder to get unique records from a column, you have to write this.
this would be simple.
DataTable dtNew = dt.DefaultView.ToTable(true, "ColName"); // for Datatable
DataTable dtnew= dv.ToTable(true, "ColName"); // for DataView
I have a DataTable available with me which contains thousands of rows. There is a column called EmpID which is containing '0' for some of the rows. I want to remove them from my current DataTable and want to create a new correct DataTable. I cannot go row by row checking it since it contains huge amount of data. Give me a suggestion to overcome this problem.
the best way would be to filter it at source (if possible) - so if you are creating it from a db, exclude all 0 values in your sql query itself using a where
starting .net 2.0, ms enhanced the filtering logic on the datatable to a great extent. so if you used the dataview (on top of your datatable) and added the where clause in there and added some sort of runtime indexes on this field, it would give you the desired results without looping over all records
You can use DataTable.Select("EmpID <> 0"). This will return an array of DataRows which you can create your new DataTable from if required.
Isn't it possible to first select the rows with EmpID = 0 and then iterate over these only ?
DataTable newTable = new DataTable();
foreach (DataRow dr in oldTable.Select("EmpID = '0'")) {
newTable.Rows.Add(dr);
oldTable.Rows.Remove(dr);
}
You can try
DataRow[] temp=
table.Select("EmpID ='0'");
foreach(DataRow dr in temp)
{
table.Rows.Remove(dr);
}
table.acceptchanges();