We Keep Coding

Changing Auto Increment of a colum in SQLite

I am creating a table that at first keyID column is NOT in AUTO INCREMENT since i am going to insert a data that came from an old database where the keyID is already been set to each rows and it's not possible to be changed. So is it possible that i can set the column to AUTO INCREMENT after all the data has been moved? I've seen this:
ALTER TABLE tbl_name MODIFY COLUMN keyID_id INT auto_increment
So if ever i set the keyID to auto increment after i moved all the data, so for example 10 rows has been transferred and some numbers may be missing for example, keyIDs: 1,2,3,5,6,7...15
can I assume that after i set the keyID to auto increment, the next row will be keyID 16?

The first thing to say here is that SQLite has no syntax to change the settings of a column. You are forced to rename your original table, create a new table with the schema you want and then copy from the renamed table into the new one.
So your code to add the AUTOINCREMENT flag to your table is something like this
SQLiteCommand cmd = new SQLiteCommand();
cmd.Connection = yourConnection;
cmd.CommandText = "ALTER TABLE tblName RENAME TO tmp_tblName";
cmd.ExecuteNonQuery();
cmd = new SQLiteCommand(#"
CREATE TABLE tblName
(ID INTEGER PRIMARY KEY AUTOINCREMENT,
AnotherField NVARCHAR(100) NOT NULL)", cnn);
cmd.ExecuteNonQuery();
cmd.CommandText = "INSERT INTO tblName (ID, AnotherField) SELECT ID, AnotherField FROM tmp_tblName";
cmd.ExecuteNonQuery();
Now, when you create a table with the AUTOINCREMENT SQLite add a new row to an internal table called sqlite_sequence. This table could be read by the usual ADO.NET classes
cmd.CommandText = "SELECT seq FROM sqlite_sequence WHERE name = 'tblName'";
int lastInsertedValue = Convert.ToInt32(cmd.ExecuteScalar());
Console.WriteLine(lastInsertedValue);
If your run this test code you will see that the value for the field seq is equal to the maximum value inserted in the PRIMARY KEY field.
After this if you try to insert a new record in your table you will see that the AUTOINCREMENT flag has started to work as expected and your new record will receive the next value from the sequence....

Even though the SQLite documentation and some posts state that the alter table command in SQLite is very limited and should not allow making such modifications, it seems that it's still possible to alter PK properties. What worked for me was:
Create a table with CREATE TABLE "test" (Field1INTEGER,Field2TEXT,)
Insert arbitrary data
Change column definition for column Field1 with DB Browser for SQLite
Subsequent inserts should then auto increment the IDs in the column Field1.
I must admit that it might not be suitable if there's the requirement to do this programatically, but if some manual steps are acceptable, it may be a solution.

Identity column in SQL Server exposes strange behaviour

I have an identity (auto increment integer) column in my data table in a SQL Server database.
When I start my program and add new record to this table identity column always equals -1. For the next record it becomes -2 and so on. I add new record this way:
http://msdn.microsoft.com/en-us/library/5ycd1034.aspx
However when I restart my program all identity values are reordered (become 1, 2, ...).
Any ideas why this happens? It would be no issue if i could delete these records without restarting. I use SQL Server 2008.
Also is there any way to specify MAX size for column data type through GUI interface (when adding table in visual studio 2012 server explorer) ?

Why they are negative I don't know. However, when you reload the application those records already exist in the database and have id's that were assigned when they were committed to the database; that's why they have real values on restart.
But, you don't need to delete the records from the DataTable, you just need to refresh that row after committing it to the database. There are a number of ways to do this, and would depend significantly on exactly how you're accessing your data now, but you can do things like tack on the SELECT SCOPE_IDENTITY() command with the INSERT command and then use ExecuteScalar to commit the row, like this:
var insertCmd = "INSERT INTO tbl (fld1, fld2) VALUES (#fld1, #fld2); SELECT SCOPE_IDENTITY()";
using (var c = new SqlConnection(connString))
using (var cmd = new SqlCommand(insertCmd, c))
{
cmd.AddParameterWithValue("#fld1", fld1Value);
cmd.AddParameterWithValue("#fld2", fld2Value);
var result = cmd.ExecuteScalar();
int id;
if (int.TryParse(result, out id))
{
// update the DataTable row here
dataTable.Rows[index]["id_column"] = id;
dataTable.AcceptChanges();
}
}
You could even choose to reload the entire DataTable after performing the update.

Possible to insert with a Table Parameter, and also retrieve identity values?

I'm trying to insert records using a high performance table parameter method ( http://www.altdevblogaday.com/2012/05/16/sql-server-high-performance-inserts/ ), and I'm curious if it's possible to retrieve back the identity values for each record I insert.
At the moment, the answer appears to be no - I insert the data, then retrieve back the identity values, and they don't match. Specifically, they don't match about 75% of the time, and they don't match in unpredictable ways. Here's some code that replicates this issue:
// Create a datatable with 100k rows
DataTable dt = new DataTable();
dt.Columns.Add(new DataColumn("item_id", typeof(int)));
dt.Columns.Add(new DataColumn("comment", typeof(string)));
for (int i = 0; i < 100000; i++) {
dt.Rows.Add(new object[] { 0, i.ToString() });
}
// Insert these records and retrieve back the identity
using (SqlConnection conn = new SqlConnection("Data Source=localhost;Initial Catalog=testdb;Integrated Security=True")) {
conn.Open();
using (SqlCommand cmd = new SqlCommand("proc_bulk_insert_test", conn)) {
cmd.CommandType = CommandType.StoredProcedure;
// Adding a "structured" parameter allows you to insert tons of data with low overhead
SqlParameter param = new SqlParameter("#mytable", SqlDbType.Structured);
param.Value = dt;
cmd.Parameters.Add(param);
SqlDataReader dr = cmd.ExecuteReader();
// Set all the records' identity values
int i = 0;
while (dr.Read()) {
dt.Rows[i].ItemArray = new object[] { dr.GetInt32(0), dt.Rows[i].ItemArray[1] };
i++;
}
dr.Close();
}
// Do all the records' ID numbers match what I received back from the database?
using (SqlCommand cmd = new SqlCommand("SELECT * FROM bulk_insert_test WHERE item_id >= #base_identity ORDER BY item_id ASC", conn)) {
cmd.Parameters.AddWithValue("#base_identity", (int)dt.Rows[0].ItemArray[0]);
SqlDataReader dr = cmd.ExecuteReader();
DataTable dtresult = new DataTable();
dtresult.Load(dr);
}
}
The database is defined using this SQL server script:
CREATE TABLE bulk_insert_test (
item_id int IDENTITY (1, 1) NOT NULL PRIMARY KEY,
comment varchar(20)
)
GO
CREATE TYPE bulk_insert_table_type AS TABLE ( item_id int, comment varchar(20) )
GO
CREATE PROCEDURE proc_bulk_insert_test
#mytable bulk_insert_table_type READONLY
AS
DECLARE #TableOfIdentities TABLE (IdentValue INT)
INSERT INTO bulk_insert_test (comment)
OUTPUT Inserted.item_id INTO #TableOfIdentities(IdentValue)
SELECT comment FROM #mytable
SELECT * FROM #TableOfIdentities
Here's the problem: the values returned from proc_bulk_insert_test are not in the same order as the original records were inserted. Therefore, I can't programmatically assign each record the item_id value I received back from the OUTPUT statement.
It seems like the only valid solution is to SELECT back the entire list of records I just inserted, but frankly I'd prefer any solution that would reduce the amount of data piped across my SQL Server's network card. Does anyone have better solutions for large inserts while still retrieving identity values?
EDIT: Let me try clarifying the question a bit more. The problem is that I would like my C# program to learn what identity values SQL Server assigned to the data that I just inserted. The order isn't essential; but I would like to be able to take an arbitrary set of records within C#, insert them using the fast table parameter method, and then assign their auto-generated ID numbers in C# without having to requery the entire table back into memory.
Given that this is an artificial test set, I attempted to condense it into as small of a readable bit of code as possible. Let me describe what methods I have used to resolve this issue:
In my original code, in the application this example came from, I would insert about 15 million rows using 15 million individual insert statements, retrieving back the identity value after each insert. This worked but was slow.
I revised the code using high performance table parameters for insertion. I would then dispose of all of the objects in C#, and read back from the database the entire objects. However, the original records had dozens of columns with lots of varchar and decimal values, so this method was very network traffic intensive, although it was fast and it worked.
I now began research to figure out whether it was possible to use the table parameter insert, while asking SQL Server to just report back the identity values. I tried scope_identity() and OUTPUT but haven't been successful so far on either.
Basically, this problem would be solved if SQL Server would always insert the records in exactly the order I provided them. Is it possible to make SQL server insert records in exactly the order they are provided in a table value parameter insert?
EDIT2: This approach seems very similar to what Cade Roux cites below:
http://www.sqlteam.com/article/using-the-output-clause-to-capture-identity-values-on-multi-row-inserts
However, in the article, the author uses a magic unique value, "ProductNumber", to connect the inserted information from the "output" value to the original table value parameter. I'm trying to figure out how to do this if my table doesn't have a magic unique value.

Your TVP is an unordered set, just like a regular table. It only has order when you specify as such. Not only do you not have any way to indicate actual order here, you're also just doing a SELECT * at the end with no ORDER BY. What order do you expect here? You've told SQL Server, effectively, that you don't care. That said, I implemented your code and had no problems getting the rows back in the right order. I modified the procedure slightly so that you can actually tell which identity value belongs to which comment:
DECLARE #TableOfIdentities TABLE (IdentValue INT, comment varchar(20))
INSERT INTO bulk_insert_test (comment)
OUTPUT Inserted.item_id, Inserted.comment
INTO #TableOfIdentities(IdentValue, comment)
SELECT comment FROM #mytable
SELECT * FROM #TableOfIdentities
Then I called it using this code (we don't need all the C# for this):
DECLARE #t bulk_insert_table_type;
INSERT #t VALUES(5,'foo'),(2,'bar'),(3,'zzz');
SELECT * FROM #t;
EXEC dbo.proc_bulk_insert_test #t;
Results:
1 foo
2 bar
3 zzz
If you want to make sure the output is in the order of identity assignment (which isn't necessarily the same "order" that your unordered TVP has), you can add ORDER BY item_id to the last select in your procedure.
If you want to insert into the destination table so that your identity values are in an order that is important to you, then you have a couple of options:
add a column to your TVP and insert the order into that column, then use a cursor to iterate over the rows in that order, and insert one at a time. Still more efficient than calling the entire procedure for each row, IMHO.
add a column to your TVP that indicates order, and use an ORDER BY on the insert. This isn't guaranteed, but is relatively reliable, particularly if you eliminate parallelism issues using MAXDOP 1.
In any case, you seem to be placing a lot of relevance on ORDER. What does your order actually mean? If you want to place some meaning on order, you shouldn't be doing so using an IDENTITY column.

You specify no ORDER BY on this: SELECT * FROM #TableOfIdentities so there's no guarantee of order. If you want them in the same order they were sent, do an INNER JOIN in that to the data that was inserted with an ORDER BY which matches the order the rows were sent in.

Dataset autoincrement issue

I have a SQL Server Database with several tables. One of them has "ID" (primary key), "Name" and other columns that i won't mention here for sake of simplicity. "ID" column is auto increment, unique and when i add some row using "SQL Server management studio", "ID" column increments properly. Database is old and current auto increment is at 1244 or so.
Now, i have created a C# project that uses TYPED Dataset to work with data from database. My database starts empty, dataset is filled using table adapters, new rows are added using my program but there's a problem i have never stumbled upon so far: when my program adds new row to Dataset, then updates database (using table adapter), "ID" column in my database gets correct auto-incremented number (1245,1246 etc), BUT my "ID" column in dataset gets "-1", "-2" instead! What's the problem? How can i tell my dataset to use auto-increment seed specified by database instead generating it's own NEGATIVE (???) primary key numbers?
EDIT:
I get and compare rows using this:
dsNames.tbNamesRow[] TMP = basedataset.tbNames.Select() as dsNames.tbNamesRow[];
foreach (dsNames.tbNamesRow row in TMP)
{
string Name = row.Name;
bool Found = Name == Search;
if (CompareDelegate != null)
Found = CompareDelegate(Name, Search);
if (Found)
{
int ID = row.ID;
break;
}
}

My original comment was kind of incorrect, I assumed you were retrieving the value from the database and THAT dataset had incorrect values in it.
The way ADO.NET deals with preventing collisions with it's disconnected dataset, it assigns negative IDENTITY column values, because it wouldn't know a possible positive number that IS NOT a collision as it's disconnected. These (negative) values are unique in terms of that transaction.
When you try and commit your changes, the ADO.NET engine determines the proper SQL to produce the correct result.

Prevent negative id's from being inserted into Database from .NET DataSet

I have my data set filled and I allow a user to add a row to the data table. Once they confirm they want to add the row I add the row to the DataSet and proceed to update the Database DataTable.
MyTypedDataSet.MyDataTable newRow = dataSetObject.MyDataTable.NewMyDataTableRow();
newRow.Description = "New Row";
dataSetObject.MyDataTable.AddMyDataTableRow( newRow );
I can update the data table by using an SQLConnection and calling SQLDataAdapter.Update(), but the problem is, the ID value from the data set gets carried over into the database, which is still negative.
How do i update my DataTable so that newly added rows have the correctly incremented ID?

You didn't indicate the RDBMS you're using, but given your comments and my assumption, you're using SQL Server. And, you're using the IDENTITY property to increment the key value.
If new rows are not getting the correct next identity value, you can run the following to reset the value.
DBCC CHECKIDENT('table-name', RESEED)
This will reseed the identity value using the max value in the table.
You can view the current information about the identity using the IDENT_CURRENT, IDENT_INCR and IDENT_SEED functions.

Do not take the ID data from your client. Rather use a generated ID for your table.
May be you could use a stored procedure(CRUD) which captures the MAX(col) and puts the nextid in the new row created.