I used SQLite before, and adding multiple rows using Insert in a for loop was slow. The solution was using a transaction.
Now that I am using SQLiteAsyncConnection in SQLite.Net (for ORM), I also tried to use a transaction. It works but with only one problem. The insert order is not the order of the data.
Database.RunInTransactionAsync(
(SQLiteConnection conn) => {
foreach (var row in rows)
{
conn.InsertOrReplace(row);
}
conn.Commit();
}
);
If rows contained [1,2,3,4,5,6], the rows in the database was something like [3,1,2,6,4,5]. How can I keep the original order?
Note that I only mean newly inserted rows. Even thought the code is replacing existing rows, when testing there were no existing rows in the database to be replaced.
PS: The row has ID field which is the [PrimaryKey], but in the rows the rows are not sorted by ID. It seems that in the database the rows are sorted by ID. I do not want it to be sorted by ID but the original order to be maintained.
PS 2: I need to know the ID of the last-inserted row. When viewing the database using a GUI tool like DB Browser for SQLite or getting the last item by LIMIT 1, it seems the SQLite had automatically sorted the rows by ID. I did some Google search and it said by the rules of SQL, when there is no ORDER BY, the order of the returned rows are not guaranteed to be the physical order, anyway. Should I create another field and set it as the primary, auto-increasing field?
Currently, ID is guaranteed to be unique per row, but 'ID' is part of the data itself, not a field specially added for the use with the database.
SQL tables are logically unordered, so if you want a certain order, you always have to use ORDER BY in your queries.
If your data does not contain any values (e.g., timestamp) that corresponds to the insertion order, then you have to use the rowid, i.e., add a column declared as INTEGER PRIMARY KEY.
Related
I couldn't find any similar questions in Stack Overflow
Is there a way to create a virtual SQL database with indexes in memory? Or maybe a built in function for creating indexes on datatables to quickly searching a column in a table multiple times? Trying to compare each row of table A against the indexed entry in table B (rather than looping through every row of table B completely for each row in Table A).
Right now I'm creating a dictionary<T,int> index where T is the typeof the column being indexed, and int is the row. That way if I create an index, I cycle through all rows of the table and create a dictionary key on the column (with the dictionary value being the row number). This works for unique primary keys, and I've used a variation with int lists if there are multiple rows for a given key.
This works when trying to find the exact value in another table, but not if I want to perform a comparison and find all int keys greater than a specific value. I could probaby reinvent the wheel with a sorted binary search tree (especially since the table data would be static), but would rather use an existing solution without the risk of introducing my own code errors.
Create in-memory SQLite Database and then you can use all of its benefits.
https://sqlite.org/inmemorydb.html
Is there any way to find "DateTime" of the last change in rows in a table in SQL Server?
The changes (Insert / Update) are submitted by another windows application
And all I have in this table is insert_Date and there is no update_Date (and I can't add any columns or use triggers)
I've tried some queries, but all I got was the number of "User Updates" in a table, not the IDs of modified rows!
I want to get rows which are modified or inserted after a specific DateTime
If the information isn't stored in the table (or in another one by using a trigger for example) then it's impossible to track which rows were inserted after a determined datetime. You might find the time the last operation was executed at a table/index level (by querying sys.dm_db_index_usage_stats) but not at a record level.
You can't find data that doesn't exist!
I know I can do a bulk insert into my table with an identity column by not specifying the SqlBulkCopyOptions.KeepIdentity as mentioned here.
What I would like to be able to do is get the identity values that the server generates and put them in my datatable, or even a list. I saw this post, but I want my code to be general, and I can't have a version column in all my tables. Any suggestions are much appreciated. Here is my code:
public void BulkInsert(DataTable dataTable, string DestinationTbl, int batchSize)
{
// Get the DataTable
DataTable dtInsertRows = dataTable;
using (SqlBulkCopy sbc = new SqlBulkCopy(sConnectStr))
{
sbc.DestinationTableName = DestinationTbl;
// Number of records to be processed in one go
sbc.BatchSize = batchSize;
// Add your column mappings here
foreach (DataColumn dCol in dtInsertRows.Columns)
{
sbc.ColumnMappings.Add(dCol.ColumnName, dCol.ColumnName);
}
// Finally write to server
sbc.WriteToServer(dtInsertRows);
}
}
AFAIK, you can't.
The only way (that I know of) to get the values(s) of the identity field is by using either SCOPE_IDENTITY() when you insert row-by-row; or by using the OUTPUT approach when inserting an entire set.
The 'simplest' approach probably would be that you would SqlBulkCopy the records in the table and then fetch them back again later on. The problem might be that it could be hard to properly (and quickly) fetch those rows from the server again. (e.g. it would be rather ugly (and slow) to have a WHERE clause with IN (guid1, guid2, .., guid999998, guid999999) =)
I'm assuming performance is an issue here as you're already using SqlBulkCopy so I'd suggest to go for the OUTPUT approach in which case you'll firstly need a staging table to SqlBulkCopy your records in. Said table should then be including some kind of batch-identifier (GUID?) as to allow multiple treads to run side by side. You'll need a stored procedure to INSERT <table> OUTPUT inserted.* SELECT the data from the staging-table into the actual destination table and also clean-up the staging table again. The returend recordset from said procedure would then match 1:1 to the origanal dataset responsible for filling the staging table, but off course you should NOT rely on it's order. In other words : your next challenge than will be matching the returned Identity-fields back to the original records in your application.
Thinking things over, I'd say that in all cases -- except the row-by-row & SCOPY_IDENTITY() approach, which is going to be dog-slow -- you'll need to have (or add) a 'key' to your data to link the generated id's back to the original data =/
You can do a similar approach described above by deroby but instead of retrieving them back via a WHERE IN (guid1, etc... You match them back up to the rows inserted in memory based on their order.
So I would suggest to add a column onto the table to match the row to a SqlBulkCopy transaction and then do the following to match the generated Ids back to the in memory collection of rows you just inserted.
Create a new Guid and set this value on all the rows in the bulk copy mapping to the new column
Run the WriteToServer method of the BulkCopy object
Retrieve all the rows that have that same key
Iterate through this list which will be in the order they were added, these will be in the same order as the the in memory collection of rows so you then will know the generated id for each item.
This will give you better performance than giving each individual row a unique key. So after you bulk insert the data table you could do something like this (In my example I will have a list of objects from which I will create the data table and then map the generated ids back to them)
List<myObject> myCollection = new List<myObject>
Guid identifierKey = Guid.NewGuid();
//Do your bulk insert where all the rows inserted have the identifierKey
//set on the new column. In this example you would create a data table based
//off the myCollection object.
//Identifier is a column specifically for matching a group of rows to a sql
//bulk copy command
var myAddedRows = myDbContext.DatastoreRows.AsNoTracking()
.Where(d => d.Identifier == identiferKey)
.ToList();
for (int i = 0; i < myAddedRows.Count ; i++)
{
var savedRow = myAddedRows[i];
var inMemoryRow = myCollection[i];
int generatedId = savedRow.Id;
//Now you know the generatedId for the in memory object you could set a
// a property on it to store the value
inMemoryRow.GeneratedId = generatedId;
}
I am using fluentmigrator to add a new column to a table. I then want to update each row in the table with a unique value for that column.
Currently when I use:
Update.Table("Foo").InSchema("dbo").Set(new { Bar = Bar.Generate() }).AllRows();
It gives the same value for all the rows.
How do I ensure it calls that method for each row?
I'm not sure what Bar.Generate does but I am guessing it creates a GUID or unique id.
If so then you could use:
Execute.Sql("update dbo.Foo set Bar = NEWID()");
Or if you want sequential guids then you could use NEWSEQUENTIALID().
If you are adding a new column for this unique identier, then all you would need to do is specify the new column .AsGuid()
EDIT: FluentMigrator is a small fluent dsl and is not meant to cover a complicated case like this. There is no way (as far as I know) to do this with one sql UPDATE and therefore no easy way to do it with FluentMigrator. You'll have to get the row count for the table with ADO.NET or an ORM (Dapper/NHibernate) and then loop through each row and update the Bar column with the custom unique identifier. So if you have one million rows then you will have to make one million sql updates. If you can rewrite your Bar.Generate() method as an Sql function that is based on the NEWID() function like this or this then you could do it as one UPDATE statement and call it with FluentMigrator's Execute.Sql method.
You haven't mentioned which database you are working with. But some like Postgres have non-standard features that could help you.
i have a many-to-many relationship table in a typed DataSet.
For convenience on an update i'm deleting old relations before i'm adding the new(maybe the same as before).
Now i wonder if this way is failsafe or if i should ensure only to delete which are really deleted(for example with LINQ) and only add that one which are really new.
In SQL-Server is a unique constraint defined for the relation table, the two foreign keys are a composite primary key.
Is the order the DataAdapter updates the DataRows which RowState are <> Unchanged predictable or not?
In other words: is it possible that DataAdapter.Update(DataTable) will result in an exception when the key already exists?
This is the datamodel:
This is part of the code(LbSymptomCodes is an ASP.Net ListBox):
Dim daTrelRmaSymptomCode As New ERPModel.dsRMATableAdapters.trelRMA_SymptomCodeTableAdapter
For Each oldTrelRmaSymptomCodeRow As ERPModel.dsRMA.trelRMA_SymptomCodeRow In thisRMA.GettrelRMA_SymptomCodeRows
oldTrelRmaSymptomCodeRow.Delete()
Next
For Each item As ListItem In LbSymptomCodes.Items
If item.Selected Then
Dim newTrelRmaSymptomCodeRow As ERPModel.dsRMA.trelRMA_SymptomCodeRow = Services.dsRMA.trelRMA_SymptomCode.NewtrelRMA_SymptomCodeRow
newTrelRmaSymptomCodeRow.fiRMA = Services.IdRma
newTrelRmaSymptomCodeRow.fiSymptomCode = CInt(item.Value)
Services.dsRMA.trelRMA_SymptomCode.AddtrelRMA_SymptomCodeRow(newTrelRmaSymptomCodeRow)
End If
Next
daTrelRmaSymptomCode.Update(Services.dsRMA.trelRMA_SymptomCode)
Thank you in advance.
I think that the DataAdapter in ADO.NET is clever enough to perform the delete/inserts in the correct order.
However, if you really want to ensure that updates are done in the correct order you should do it manually by using the Select method to return an array of data rows for each particular row state. You could then call the Update method on the array of data rows
DataTable tbl = ds.Tables["YourTable"];
// Process any Deleted rows first
adapter.Update(tbl.Select(null, null, DataViewRowState.Deleted));
// Process any Updated/Modified rows
adapter.Update(tbl.Select(null, null, DataViewRowState.ModifiedCurrent));
// Process the Inserts last
adapter.Update(tbl.Select(null, null, DataViewRowState.Added));
Not sure about the DA but in theory DB transactions should be performed in the following order Deletes, Inserts, Updates.
looking at msdn the exact wording for the update method is
Blockquote
Attempts to save all changes in the DataTable to the database. (This includes removing any rows deleted from the table, adding rows inserted to the table, and updating any rows in the table that have changed.)
Blockquote
In regards to your solution of deleting items and possibly re-inserting the same items, typically speaking this should be avoided because it creates a load on the DB. In high volume applications you want to do everything you can to minimize calls to the DB as they are very expensive; computation time, from determining which row updates are spurious, is cheap.