I have to update a number of rows to a table, if the updating row is not existing in the table I need to insert that row. I cannot use unique key, so no use with ON duplicate KEY UPDATE
I have to achieve something like this
DECLARE count DOUBLE;
SELECT count(uid)
INTO count
FROM Table
WHERE column1 ='xxxxx'
AND column2='xxxxx';
IF (count=0)
THEN
--peform insert
ELSE
--perform update
END IF
This is for a high performance application.Any ideas? Code level or Query level
FYI : Database is Mysql
You could work with a temporary table.
Put your data into a temporary table
Do an update of the "other" table via a JOIN
Delete the matching data from the temp table
Insert the remaining stuff from the temp table into the main table.
This will be faster than doing it record by record if you have loads of data.
That's the store procedure we use, could possibly work for you as well.
if not exists (select 1 from Table where column1 ='xxxxx' AND column2='xxxxx')
insert into Table ( column1,column2)
values ( #xxxx,xxxxx)
else
update Table
You can use EXISTS or check the cound of a sub select if its > 0 to know if the row exists allready
BEGIN TRAN
IF EXISTS ( SELECT *
FROM Table WITH ( UPDLOCK, SERIALIZABLE )
WHERE CONDITION)
BEGIN
UPDATE Table SET SOMETHING WHERE CONDITION
END
ELSE
BEGIN
INSERT INTO Table(Field1,....) VALUES (Value1,..... )
END
COMMIT TRAN
NOTE: Transaction are very good but using IF EXISTS is not good in case of Insertion/Updation with mulitple queries.
You may find useful REPLACE statement. Its syntax described here.
Related
I have to get the IDENTITY values from a table after SQLBULKCOPY to the same table. The volume of data could be thousands of records.
Can someone help me out on this ?
Disclaimer: I'm the owner of the project Bulk Operations
In short, this project overcomes SqlBulkCopy limitations by adding MUST-HAVE features like outputting inserted identity value.
Under the hood, it uses SqlBulkCopy and a similar method as #Mr Moose answer.
var bulk = new BulkOperation(connection)
// Output Identity Value
bulk.ColumnMappings.Add("CustomerID", ColumnMappingDirectionType.Output);
// Map Column
bulk.ColumnMappings.Add("Code");
bulk.ColumnMappings.Add("Name");
bulk.ColumnMappings.Add("Email");
bulk.BulkInsert(dt);
EDIT: Answer comment
can I simply get a IList or simply , I see its saved back in the customers table, but there is no variable where I can get a hold of it, can you please help with that. So, I an insert in Orders.CustomerID table
It depends, you can keep a reference to the Customer DataRow named CustomerRef in the Order DataTable.
So once you merged your customer, you are able to populate easily a column CustomerID from the column CustomerRef in your Order DataTable.
Here is an example of what I'm trying to say: https://dotnetfiddle.net/Hw5rf3
I've used a solution similar to this one from Marc Gravell in that it is useful to first import into a temp table.
I've also used the MERGE and OUTPUT as described by Jamie Thomson on this post to track data I have inserted into my temp table to match it with the id generated by the IDENTITY column of the table I want to insert into.
This is particularly useful when you need to use that ID as a foreign key reference to other tables you are populating.
Try this
CREATE TABLE #temp
(
DataRow varchar(max)
)
BULK INSERT #Temp FROM 'C:\tt.txt'
ALTER TABLE #temp
ADD id INT IDENTITY(1,1) NOT NULL
SELECT * FROM #temp
-- dummy schema
CREATE TABLE TMP (data varchar(max))
CREATE TABLE [Table1] (id int not null identity(1,1), data varchar(max))
CREATE TABLE [Table2] (id int not null identity(1,1), id1 int not null, data varchar(max))
-- imagine this is the SqlBulkCopy
INSERT TMP VALUES('abc')
INSERT TMP VALUES('def')
INSERT TMP VALUES('ghi')
-- now push into the real tables
INSERT [Table1]
OUTPUT INSERTED.id, INSERTED.data INTO [Table2](id1,data)
SELECT data FROM TMP
I'm working on a pet project that will allow me to store my game collection in a DB and write notes on those games. The single entries of games has been coded by inserting desired variables into my game_information table and outputting the PK (identity) of the newly created row from that table, so I can insert it into my game_notes table along with the note.
var id = db.QueryValue("INSERT INTO Game_Information (gamePrice, name, edition) output Inserted.gameId VALUES (#0, #1, #2)", gamePrice, name, edition);
db.Execute("INSERT INTO Game_Notes(gameId, notes, noteDate) VALUES (#0, #1, #2)", id, notes, noteDate);
I'm now playing with uploading data in bulk via csv but how can I write a BULK INSERT that would output all PKs of the newly created rows, so I can inserted them into my second table (game_notes) along with a variable called notes?
At the moment I have the following:
Stored Procedure that reads .csv and uses BULK INSERT to dump information into a view of game_information
#FileName nvarchar(200)
AS
BEGIN
DECLARE #sql nvarchar(MAX);
SET #sql = 'BULK INSERT myview
FROM ''mycsv.csv''
WITH
(
FIELDTERMINATOR = '','',
ROWTERMINATOR = ''\n'',
FIRSTROW = 2
)'
EXEC(#sql)
END
C# code that creates set up in WebMatrix
if ((IsPost) && (Request.Files[0].FileName!=" "))
{
var fileSavePath = "";
var uploadedFile = Request.Files[0];
fileName = Path.GetFileName(uploadedFile.FileName);
uploadedFile.SaveAs(//path +filename);
var command = "EXEC Procedure1 #FileName = #0";
db.Execute(command, //path +filename);
File.Delete(//path +filename);
}
Which allows for csv records to be inserted into game_information.
If this isn't feasible with BULK INSERT, would something along the lines of be a valid solution to attempt?
BULK INSERT into a temp_table
INSERT from temp_table to my game_information table
OUTPUT the game_Ids from the INSERT as an array(?)
then INSERT the Ids along with note into game_notes.
I've also been looking at OPENROWSET but I'm unsure if that will allow for what I'm trying to accomplish. Feedback on this is greatly appreciated.
Thank your for your input womp. I was able to get the desired results by amending my BULK INSERT as follows:
BEGIN
DECLARE #sql nvarchar(MAX);
SET #sql=
'CREATE TABLE #Temp (--define table--)
BULK INSERT #Temp --Bulk into my temp table--
FROM '+char(39)+#FileName+char(39)+'
WITH
(
FIELDTERMINATOR = '','',
ROWTERMINATOR = ''\n'',
FIRSTROW = 2
)
INSERT myDB.dbo.game_information(gamePrice, name, edition, date)
OUTPUT INSERTED.gameId, INSERTED.Date INTO myDB.dbo.game_notes(gameId, noteDate)
SELECT gamePrice, name, edition, date
FROM #Temp'
EXEC(#sql)
END
This placed the correct ids into game_notes and left the Note column of the table as Null for those entries. Which meant I could run a simple
"UPDATE game_notes SET Notes = #0 WHERE Notes IS NULL";
To push the desired note into the correct rows. I'm executing this and the stored bulk procedure in the same If (IsPost), so I feel like I'm protected from the wrong accidental note updates.
You have a few different options.
Bulk inserting into a temp table and then copying information into your permanent tables is definitely a valid solution. However, based on what you're trying to do I don't see the need for a temp table. Just bulk import into game_information, SELECT your ID's to your application, and then do your update of game_notes.
Another option would be to insert your keys. You can allow for IDENTITY_INSERT to be on for your tables and just have your keys as part of the CSV file. See here: https://msdn.microsoft.com/en-ca/library/ms188059.aspx?f=255&MSPPError=-2147217396. If you did this then you could do a BULK INSERT into your Game_information table, and then do a second BULK INSERT into your secondary tables by using a different CSV file. Be sure to re-enable key constraints and turn IDENTITY_INSERT off after its finished.
If you need more particular control over the data you're selecting from the CSV file then you can use OPENROWSET but there's not enough details in your post to comment further.
I am trying to implement bulk insert of data from Datatable. In my MS-SQL Table(Destination table) i have a column with primary key not Identity column, so i have to increment manually. But its not possible in Code because there will be multi Thread on the same table.Please give me suggestion if any.
public void BulkInsert(DataTable dtTable)
{
DataTable dtProductSold = dtTable;
//creating object of SqlBulkCopy
SqlBulkCopy objbulk = new SqlBulkCopy(ConStr.ToString());
//assigning Destination table name
objbulk.DestinationTableName = "BatchData_InvReportMapping";
//Mapping Table column
objbulk.ColumnMappings.Add("InvPK", "InvPK");
objbulk.ColumnMappings.Add("DateValue", "DateDalue");
objbulk.ColumnMappings.Add("TextValue", "TextValue");
objbulk.ColumnMappings.Add("NumericValue", "NumericValue");
objbulk.ColumnMappings.Add("ErrorValue", "ErrorValue");
//inserting bulk Records into DataBase
objbulk.WriteToServer(dtProductSold);
}
Thanks in advance,
This is too long for a comment.
If you have a primary key column, then you need to take responsibility for its being unique and non-NULL when you insert rows. SQL Server offers a very handy mechanism to help with this, which is the identity column.
If you do not have an identity, then I you basically have two options:
Load data that has a valid primary key column.
Create a trigger that assigns the value when rows are loaded in.
Oh, wait. The default option for bulk insert is not to fire triggers, so the second choice really isn't a good option.
Instead, modify the table to have an identity primary key column. Then define a view on the table without the primary key and do the bulk insert into the view. The primary key will then be assigned automatically.
EDIT:
There is a third option, which might be feasible. Load the data into a staging table. Then insert from the staging table into the final table, calculating the primary key value. Something like this:
insert into finaltable (pk, . . .)
select m.maxpk + seqnum, . . . .
from (select row_number() over (order by (select null)) as seqnum,
. . .
from stagingtable
) s cross join
(select max(pk) as maxpk
from finaltable
) m;
i had one idea
generally we use tables to store the records, even if you insert the data using front end finally it will be stored in table.So i am suggesting to use sequences with insert trigger on the table. which means when you insert the data into the table first the trigger will be called, sequence will be incremented the the increased value will be stored along with other values in the table. just try this. because in oracle 11g we don't have identity() hence we will use sequences and insert trigger for identity column
Create a Table called id's. VARCHAR(50) TableName, INT Id.
When you want to generate your ids read the relevant row and increment it by the number of rows you want to insert within the same transaction.
you can now bulk insert these rows whenever you want without worrying about other threads inserting them.
Similar to how Nhibernates HiLow generator works.
http://weblogs.asp.net/ricardoperes/making-better-use-of-the-nhibernate-hilo-generator
I have a windows service which basically watches a folder for any CSV file. Each record in the CSV file is inserted into a SQL table. If the same CSV file is put in that folder, it can lead to duplicate record entries in the table. How can I avoid duplicate insertions into the SQL table?
Try INSERT WHERE NOT EXISTS, where a, b and c are relevant columns, #a, #b and #c are relevant values.
INSERT INTO table
(
a,
b,
c
)
VALUES
(
#a,
#b,
#c
)
WHERE NOT EXISTS
(
SELECT 0 FROM table WHERE a = #a, b = #b, c = #c
)
The accepted answer has a syntax error and is not compatible with relational databases like MySQL.
Specifically, the following is not compatible with most databases:
values(...) where not exists
While the following is generic SQL, and is compatible with all databases:
select ... where not exists
Given that, if you want to insert a single record into a table after checking if it already exists, you can do a simple select with a where not exists clause as part of your insert statement, like this:
INSERT
INTO table_name (
primay_col,
col_1,
col_2
)
SELECT 1234,
'val_1',
'val_2'
WHERE NOT EXISTS (
SELECT 1
FROM table_name
WHERE primary_col=1234
);
Simply pass all values with the select keyword, and put the primary or unique key condition in the where clause.
Problems with the answers using WHERE NOT EXISTS are:
performance -- row-by-row processing requires, potentially, a very large number of table scans against table
NULL handling -- for every column where there might be NULLs you will have to write the matching condition in a more complicated way, like
(a = #a OR (a IS NULL AND #a IS NULL)).
Repeat that for 10 columns and viola - you hate SQL :)
A better answer would take into account the great SET processing capabilities that relational databases provide (in short -- never use row-by-row processing in SQL if you can avoid it. If you can't -- think again and avoid it anyway).
So for the answer:
load (all) data into a temporary table (or a staging table that can be safely truncated before load)
run the insert in a "set"-way:
INSERT INTO table (<columns>)
select <columns> from #temptab
EXCEPT
select <columns> from table
Keep in mind that the EXCEPT is safely dealing with NULLs for every kind of column ;) as well as choosing a high-performance join type for matching (hash, loop, merge join) depending on the available indexes and table statistics.
I am inserting records through a query similar to this one:
insert into tbl_xyz select field1 from tbl_abc
Now I would like to retreive the newly generated IDENTITY Values of the inserted records. How do I do this with minimum amount of locking and maximum reliability?
You can get this information using the OUTPUT clause.
You can output your information to a temp target table or view.
Here's an example:
DECLARE #InsertedIDs TABLE (ID bigint)
INSERT into DestTable (col1, col2, col3, col4)
OUTPUT INSERTED.ID INTO #InsertedIDs
SELECT col1, col2, col3, col4 FROM SourceTable
You can then query the table InsertedIDs for your inserted IDs.
##IDENTITY will return you the last inserted IDENTITY value, so you have two possible problems
Beware of triggers executed when inserting into table_xyz as this may change the value of ##IDENTITY.
Does tbl_abc have more than one row. If so then ##IDENTITY will only return the identity value of the last row
Issue 1 can be resolved by using SCOPE__IDENTITY() instead of ##IDENTITY
Issue 2 is harder to resolve. Does field1 in tbl_abc define a unique record within tbl_xyz, if so you could reselect the data from table_xyz with the identity column. There are other solutions using CURSORS but these will be slow.
SELECT ##IDENTITY
This is how I've done it before. Not sure if this will meet the latter half of your post though.
EDIT
Found this link too, but not sure if it is the same...
How to insert multiple records and get the identity value?
As far as I know, you can't really do this with straight SQL in the same script. But you could create an INSERT trigger. Now, I hate triggers, but it's one way of doing it.
Depending on what you are trying to do, you might want to insert the rows into a temp table or table variable first, and deal with the result set that way. Hopefully, there is a unique column that you can link to.
You could also lock the table, get the max key, insert your rows, and then get your max key again and do a range.
Trigger:
--Use the Inserted table. This conaints all of the inserted rows.
SELECT * FROM Inserted
Temp Table:
insert field1, unique_col into #temp from tbl_abc
insert into tbl_xyz (field1, unique_col) select field1, unique_col from tbl_abc
--This could be an update, or a cursor, or whatever you want to do
SELECT * FROM tbl_xyz WHERE EXISTS (SELECT top 1 unique_col FROM #temp WHERE unique_col = tbl_xyz.unique_col)
Key Range:
Declare #minkey as int, #maxkey as int
BEGIN TRANS --You have to lock the table for this to work
--key is the name of your identity column
SELECT #minkey = MAX(key) FROM tbl_xyz
insert into tbl_xyz select field1 from tbl_abc
SELECT #maxkey = MAX(key) FROM tbl_xyz
COMMIT Trans
SELECT * FROM tbl_xyz WHERE key BETWEEN #minkey and #maxkey