We Keep Coding

Build Where Clause Dynamically in Ado.net C#

I will be taking in around 1000 records at a given time and I have to determine if they are existing records or new records.
If they are existing I have to update the records, if new then just insert them. I will not know if any of them will be existing or if they all will be existing.
I thought that it might be best to do one query to the database and try to find if any of them exist in the db and store them in memory and check that collection in memory and check that.
Originally I was told I 1 field would be enough to determine uniqueness. So I thought I could just do 1 big in clause against 1 field in the database but now I found out that is not the case and I need to use 3 fields to determine if the record is existing or now.
This is basically an and clause
select * from where columnA = "1" and ColumnB = "2" and ColumnC = "3"
How can I properly right this in C# ado.net?
I am guessing I going to need to have like some super where clause?
select * from where (columnA = "1" and ColumnB = "2" and ColumnC = "3") or (columnA = "4" and ColumnB = "5" and ColumnC = "6") or [....998 more conditional clauses)
I am open to better ideas if possible. I still think doing it in 1 shot is better than doing 1000 separate queries.

I can only help you to write query for your request
var recordCount = 1000;
var query = "SELECT * FROM TableName WHERE";
for (var i = 1; i < recordCount - 2; i += 3)
{
query += " (columnA = " + i + " and ColumnB = " + (i + 1) + " and ColumnC = " + (i + 2) + ") or ";
}

I feel kinda silly writing this answer, because I think that you should be able to put the pieces together a complete answer from other posts - but this is not an exact duplicate of either of the questions I have in mind.
There already are questions and answers in Stackoverflow dealing with this issue - however in my search I only found answers that are not thread safe, and most of them are using merge.
There are different questions and answers I can refer you to such as
my answer to Adding multiple parameterized variables to a database in c#
where you can see how to work with table valued parameters on c#, and to Aaron Bertrand's answer to Using a if condition in an insert SQL Server where you can see how to create a safe upsert - however I didn't find any answer that covers this completely - so here you go:
First you need to create a user defined table type in your database:
CERATE TYPE MyTableType AS TABLE
(
Column1 int NOT NULL,
Column2 int NOT NULL,
Column3 int NOT NULL,
-- rest of the columns in your table goes here
PRIMARY KEY (Column1, Column2, Column3)
)
Then, you create the stored procedure:
CREATE stp_UpsertMyTable
(
#MyTableType dbo.MyTableType readonly -- table valued parameters must be readonly
)
AS
SET TRANSACTION ISOLATION LEVEL SERIALIZABLE;
BEGIN TRANSACTION;
UPDATE t
SET t.column4 = tvp.column4,
t.column5 = tvp.column5 -- and so on for all columns that are not part of the key
FROM dbo.MyTable AS t
INNER JOIN #MyTableType AS tvp
ON t.Column1 = tvp.Column1
AND t.Column2 = tvp.Column2
AND t.Column3 = tvp.Column3;
-- Note: <ColumnsList> should be replaced with the actual columns in the table
INSERT dbo.MyTable(<ColumnsList>)
SELECT <ColumnsList>
FROM #MyTableType AS tvp
WHERE NOT EXISTS
(
SELECT 1 FROM dbo.MyTable t
WHERE t.Column1 = tvp.Column1
AND t.Column2 = tvp.Column2
AND t.Column3 = tvp.Column3
);
COMMIT TRANSACTION;
GO
Then, the c# part is simple:
DataTable dt = new DataTable();
dt.Columns.Add("Column1", typeof(int));
dt.Columns.Add("Column2", typeof(int));
dt.Columns.Add("Column3", typeof(int));
dt.Columns.Add("Column4", typeof(string));
dt.Columns.Add("Column5", typeof(string));
// Fill your data table here
using (var con = new SqlConnection("ConnectionString"))
{
using(var cmd = new SqlCommand("stp_UpsertMyTable", con))
{
cmd.CommandType = CommandType.StoredProcedure;
cmd.Parameters.Add("#MyTable", SqlDbType.Structured).Value = dt;
con.Open();
cmd.ExecuteNonQuery();
}
}
Now you have a complete and safe upsert using a table valued parameter with only one round trip between c# and sql server.

How can I use more than 2100 values in an IN clause using Dapper?

I have a List containing ids that I want to insert into a temp table using Dapper in order to avoid the SQL limit on parameters in the 'IN' clause.
So currently my code looks like this:
public IList<int> LoadAnimalTypeIdsFromAnimalIds(IList<int> animalIds)
{
using (var db = new SqlConnection(this.connectionString))
{
return db.Query<int>(
#"SELECT a.animalID
FROM
dbo.animalTypes [at]
INNER JOIN animals [a] on a.animalTypeId = at.animalTypeId
INNER JOIN edibleAnimals e on e.animalID = a.animalID
WHERE
at.animalId in #animalIds", new { animalIds }).ToList();
}
}
The problem I need to solve is that when there are more than 2100 ids in the animalIds list then I get a SQL error "The incoming request has too many parameters. The server supports a maximum of 2100 parameters".
So now I would like to create a temp table populated with the animalIds passed into the method. Then I can join the animals table on the temp table and avoid having a huge "IN" clause.
I have tried various combinations of syntax but not got anywhere.
This is where I am now:
public IList<int> LoadAnimalTypeIdsFromAnimalIds(IList<int> animalIds)
{
using (var db = new SqlConnection(this.connectionString))
{
db.Execute(#"SELECT INTO #tempAnmialIds #animalIds");
return db.Query<int>(
#"SELECT a.animalID
FROM
dbo.animalTypes [at]
INNER JOIN animals [a] on a.animalTypeId = at.animalTypeId
INNER JOIN edibleAnimals e on e.animalID = a.animalID
INNER JOIN #tempAnmialIds tmp on tmp.animalID = a.animalID).ToList();
}
}
I can't get the SELECT INTO working with the list of IDs. Am I going about this the wrong way maybe there is a better way to avoid the "IN" clause limit.
I do have a backup solution in that I can split the incoming list of animalIDs into blocks of 1000 but I've read that the large "IN" clause sufferes a performance hit and joining a temp table will be more efficient and it also means I don;t need extra 'splitting' code to batch up the ids in to blocks of 1000.

Ok, here's the version you want. I'm adding this as a separate answer, as my first answer using SP/TVP utilizes a different concept.
public IList<int> LoadAnimalTypeIdsFromAnimalIds(IList<int> animalIds)
{
using (var db = new SqlConnection(this.connectionString))
{
// This Open() call is vital! If you don't open the connection, Dapper will
// open/close it automagically, which means that you'll loose the created
// temp table directly after the statement completes.
db.Open();
// This temp table is created having a primary key. So make sure you don't pass
// any duplicate IDs
db.Execute("CREATE TABLE #tempAnimalIds(animalId int not null primary key);");
while (animalIds.Any())
{
// Build the statements to insert the Ids. For this, we need to split animalIDs
// into chunks of 1000, as this flavour of INSERT INTO is limited to 1000 values
// at a time.
var ids2Insert = animalIds.Take(1000);
animalIds = animalIds.Skip(1000).ToList();
StringBuilder stmt = new StringBuilder("INSERT INTO #tempAnimalIds VALUES (");
stmt.Append(string.Join("),(", ids2Insert));
stmt.Append(");");
db.Execute(stmt.ToString());
}
return db.Query<int>(#"SELECT animalID FROM #tempAnimalIds").ToList();
}
}
To test:
var ids = LoadAnimalTypeIdsFromAnimalIds(Enumerable.Range(1, 2500).ToList());
You just need to amend your select statement to what it originally was. As I don't have all your tables in my environment, I just selected from the created temp table to prove it works the way it should.
Pitfalls, see comments:
Open the connection at the beginning, otherwise the temp table will
be gone after dapper automatically closes the connection right after
creating the table.
This particular flavour of INSERT INTO is limited
to 1000 values at a time, so the passed IDs need to be split into
chunks accordingly.
Don't pass duplicate keys, as the primary key on the temp table will not allow that.
Edit
It seems Dapper supports a set-based operation which will make this work too:
public IList<int> LoadAnimalTypeIdsFromAnimalIdsV2(IList<int> animalIds)
{
// This creates an IEnumerable of an anonymous type containing an Id property. This seems
// to be necessary to be able to grab the Id by it's name via Dapper.
var namedIDs = animalIds.Select(i => new {Id = i});
using (var db = new SqlConnection(this.connectionString))
{
// This is vital! If you don't open the connection, Dapper will open/close it
// automagically, which means that you'll loose the created temp table directly
// after the statement completes.
db.Open();
// This temp table is created having a primary key. So make sure you don't pass
// any duplicate IDs
db.Execute("CREATE TABLE #tempAnimalIds(animalId int not null primary key);");
// Using one of Dapper's convenient features, the INSERT becomes:
db.Execute("INSERT INTO #tempAnimalIds VALUES(#Id);", namedIDs);
return db.Query<int>(#"SELECT animalID FROM #tempAnimalIds").ToList();
}
}
I don't know how well this will perform compared to the previous version (ie. 2500 single inserts instead of three inserts with 1000, 1000, 500 values each). But the doc suggests that it performs better if used together with async, MARS and Pipelining.

In your example, what I can't see is how your list of animalIds is actually passed to the query to be inserted into the #tempAnimalIDs table.
There is a way to do it without using a temp table, utilizing a stored procedure with a table value parameter.
SQL:
CREATE TYPE [dbo].[udtKeys] AS TABLE([i] [int] NOT NULL)
GO
SET ANSI_NULLS ON
GO
SET QUOTED_IDENTIFIER ON
GO
CREATE PROCEDURE [dbo].[myProc](#data as dbo.udtKeys readonly)AS
BEGIN
select i from #data;
END
GO
This will create a user defined table type called udtKeys which contains just one int column named i, and a stored procedure that expects a parameter of that type. The proc does nothing else but to select the IDs you passed, but you can of course join other tables to it. For a hint regarding the syntax, see here.
C#:
var dataTable = new DataTable();
dataTable.Columns.Add("i", typeof(int));
foreach (var animalId in animalIds)
dataTable.Rows.Add(animalId);
using(SqlConnection conn = new SqlConnection("connectionString goes here"))
{
var r=conn.Query("myProc", new {data=dataTable},commandType: CommandType.StoredProcedure);
// r contains your results
}
The parameter within the procedure gets populated by passing a DataTable, and that DataTable's structure must match the one of the table type you created.
If you really need to pass more that 2100 values, you may want to consider indexing your table type to increase performance. You can actually give it a primary key if you don't pass any duplicate keys, like this:
CREATE TYPE [dbo].[udtKeys] AS TABLE(
[i] [int] NOT NULL,
PRIMARY KEY CLUSTERED
(
[i] ASC
)WITH (IGNORE_DUP_KEY = OFF)
)
GO
You may also need to assign execute permissions for the type to the database user you execute this with, like so:
GRANT EXEC ON TYPE::[dbo].[udtKeys] TO [User]
GO
See also here and here.

For me, the best way I was able to come up with was turning the list into a comma separated list in C# then using string_split in SQL to insert the data into a temp table. There are probably upper limits to this, but in my case I was only dealing with 6,000 records and it worked really fast.
public IList<int> LoadAnimalTypeIdsFromAnimalIds(IList<int> animalIds)
{
using (var db = new SqlConnection(this.connectionString))
{
return db.Query<int>(
#" --Created a temp table to join to later. An index on this would probably be good too.
CREATE TABLE #tempAnimals (Id INT)
INSERT INTO #tempAnimals (ID)
SELECT value FROM string_split(#animalIdStrings)
SELECT at.animalTypeID
FROM dbo.animalTypes [at]
JOIN animals [a] ON a.animalTypeId = at.animalTypeId
JOIN #tempAnimals temp ON temp.ID = a.animalID -- <-- added this
JOIN edibleAnimals e ON e.animalID = a.animalID",
new { animalIdStrings = string.Join(",", animalIds) }).ToList();
}
}
It might be worth noting that string_split is only available in SQL Server 2016 or higher or if using Azure SQL then compatibility mode 130 or higher. https://learn.microsoft.com/en-us/sql/t-sql/functions/string-split-transact-sql?view=sql-server-ver15

Copy a database table to another database table with a different structure

I am aware this was asked here. However it doesn't answer my question. I have 10 tables in a database called "merged". I am taking a bunch of other databases with an identical structure as "merged" except that "merged" has an extra column that is a combination of two columns in the table. I am trying to transfer all this data into merged but I believe the extra column I believe is preventing the transfer.
SqlCommand MergeDB = connDestination.CreateCommand();
string sqlMergeDB = "";
int a= 0;
for (a=0; a < tablenames.Length; a++){
string sqlMergeDB = "INSERT INTO sql_merged.dbo." + tablenames[a] + " SELECT * FROM sourceForMerged.dbo." + tablenames[a];
using (SqlDataReader reader = MergeDB.ExecuteReader()) {
while(reader.Read())
{
MessageBox.Show("Transfered a table");
}
}
}
The error occurs at the SqlDataReader row of the code above, which I believe means there is something wrong with the sql command. Please help. Thanks

If you name all the columns in both parts of the INSERT . . . SELECT statement you can map which source column gets inserted into which destination column.
If you imagine TargetTable (Name, ProductType, Date) and SourceTable (Date, Type, Name) then using:
INSERT INTO TargetTable (Name, ProductType, Date)
SELECT Name, Type, Date FROM SourceTable
would move the three columns into the appropriate columns even though the order doesn't match.
If you have "extra" columns in one table or the other you can either leave them out or provide expressions to fill them in:
INSERT INTO TargetTable (Name, ProductType, Date, CombinedValues)
SELECT Name, Type, Date, (ValCol1 + ' ' + ValCol2) FROM SourceTable
has four columns receiving data from four expressions, one of which concatenates two columns worth of data. (In real life, you may find that the concatenation expression is more complicated, but this is the basic idea).

You cannot use a:
Insert Into [Table] Select * From [Table2]
unless the tables schemas are identical. You would have to list out the columns for the Insert Statement.
If possible you could drop the column on the destination table and then add it back after the insert.
You could do something like this to build up you insert code if the table is very wide:
SELECT
'cmd.Parameter.Add("#' + column_name + '", SqlDbType.' + data_type + ');',
column_name 'Column Name',
data_type 'Data Type'
FROM information_schema.columns
WHERE table_name = 'TableName'

Oracle/c#: How do i use bind variables with select statements to return multiple records?

I have a question regarding Oracle bind variables and select statements.
What I would like to achieve is do a select on a number different values for the primary key. I would like to pass these values via an array using bind values.
select * from tb_customers where cust_id = :1
int[] cust_id = { 11, 23, 31, 44 , 51 };
I then bind a DataReader to get the values into a table.
The problem is that the resulting table only contains a single record (for cust_id=51). Thus it seems that each statement is executed independently (as it should), but I would like the results to be available as a collective (single table).
A workaround is to create a temporary table, insert all the values of cust_id and then do a join against tb_customers. The problem with this approach is that I would require temporary tables for every different type of primary key, as I would like to use this against a number of tables (some even have combined primary keys).
Is there anything I am missing?

Not asking the question as to why you would want to do this to begin with. Shouldn't the sql statement be something like
select * from tb_customers where cust_id = 11 or 23 or ...
Edit:
I am limited in Oracle but when I look at the documentation I think that you might have to do something like this:
variable i number
exec :i := 11
select * from tb_customers where cust_id = :i
This would allow you to take advantage of binding. You will have to add each record return to your own collection since it will still only return one at a time.

I know this was asked a while ago but not a brilliant answer.
I would do something like this - please excuse the crude psudo code
string bindList = "";
for(int ii=0;ii<cust_id.count;++ii)
{
if(ii == 0)
{
bindList += ":" + ii;
}
else
{
bindList += ",:" + ii;
}
OracleParameter param = new OracleParameter();
param.dbType = types.int;
param.value = cust_id[ii];
command.Parameters.Add(param);
}
query = "select * from tb_customers where cust_id in(" + bindList + ")";
So then query ends up having in(:1,:2,:3,etc) and each of these are bound separately.
There is also a similar question here: OracleParameter and IN Clause

Update multiple rows into SQL table

Suppose there is a fully populated array of data String[n][3] myData.
I want to do this:
for (String[] row : myData)
{
SQL = "update mytable set col3 = row[2]
where col1 = row[0] and col2=row[1];"
}
Obviously I've left a lot out, but I want to express the idea as succinctly as possible.
Is there a simple way of doing this in a single DB command?
How about a not so simple way?
EDITS:
Data is not coming from another table (it's a web form submission - Multiple Choice exam)
Seeing as the app is web facing, It's got to be injection proof. Parameterized Queries are my preferred way of going.
I'm using MS-SQL Server 2005
EDIT:Closing, and re-asking as Multiple DB Updates:
EDIT: Re-opened, as this appears to be a popular question

It depends on what database you are using. If you're using SQL Server 2008, you can use stored procedure TABLE parameters. This allows you to pass all your values into the stored procedure in a single table, then you can do:
update mytable set mytable.col1 = #tbl.col1
from mytable
inner join #tbl on mytable.col2 = #tbl.col2
If you're using SQL Server 2005, you can use XML. Format your values as XML, then use XQuery statements (ie, 'nodes' and 'value') to parse out the XML. This can also be done in a single SQL statement, and it doesn't require a stored procedure.

If you are using Sql Server you can use SqlBulkCopy. You would first have to put your data in a DataTable, which would be pretty easy since you already have it in a string array.
http://msdn.microsoft.com/en-us/library/system.data.sqlclient.sqlbulkcopy.aspx

You can make a big string like:
for (String[] row : myData)
{
SQL += "update mytable set col3 = row[2]
where col1 = row[0] and col2=row[1];"
}
sqlDriver.doInsertQuery(SQL); // change this to your way of inserting into the db
And just commit it all at once. I'm not very good with SQL so that's how i would do it.
The sql engine will just split it by the ';' and do separate inserts on its own. It's ok to add it all in a string though. It's kind the same as if u copy a big string with multiple updates/inserts into the sql prompt

This may not be the answer you want, but opening a transaction, executing your statements and then committing the transaction would, from a database point of view, do what you describe.
The state of the database does not change for other users of the database until the transaction has been completed, and that probably is the preferred effect.

If you are using Enterprise Library in your data access layer, you can create the transaction in .Net, iterate through your procedure calls, then commit/rollback all from .Net.
DbTransaction transaction = connection.BeginTransaction();
try
{
for (String[] row : myData)
{
ListDictionary params = new Specialized.ListDictionary();
params.add("#col3", row[2]);
params.add("#col1", row[0]);
params.add("#col2", row[1]);
executeNonQuery("myUpdateProcedure", params);
}
transaction.commit();
}
catch(Exception ex)
{
transaction.rollback();
throw ex;
}
finally
{
connection.close();
}

If for whatever reason you can't perform the update using one of the methods suggested above, the highly inefficient approach below would probably work for you.
SQL = "Update myTable Set Col3 = Case "
for (String[] row : myData)
{
SQL += "When Col1 = " + Row[0] + " and Col2 = " + Row[1] + " then " + row[2] + " "
}
SQL + = "Else Col3 end"

That looks like you want to make an update A, over rows that has coditions B and C. (A, B, C) are stored as tuples (rows) in myData. Isn't it?
Maybe (if you're using Microsoft SQL Server... I don't know if it exists in Oracle, could be) you can use a JOIN with an UPDATE. You can declare an update over a table joined with another one. If myData comes from another table then you could do (it's not the correct syntax) :
UPDATE whatchanges wc INNER JOIN changes c ON <yourcondition>
SET wc.col1 = c.newvalue
WHERE ....
(if you want to apply all changes in "changes" table you don't have to use WHERE of course, the INNER JOIN already has selected the correct rows).
Of course there are limitations to this kind of update. And it's MS SQL proprietary. So if it's your case I'd suggest to look for it on MS web (keywords: UPDATE and JOIN)

Not really. You could create the string with the same loop, then pass your values as parameters, but that will still be multiple database commands.
for each whatever
sql += "UPDATE ... ;"
end for
execute (sql)

I suspect you will need to use multiple SQL statements. You may find a wrapper to handle the minutiae but underneath I'd imagine it'd iteratively run a SQL statement for each UPDATE.

emit an update that goes against a values table:
UPDATE myTable SET col3=c FROM myTable JOIN (
SELECT 1 as a, 2 as b, 'value1' as c UNION ALL
SELECT 3 as a, 4 as b, 'value2' as c -- etc...
) x ON myTable.col1=x.a AND myTable.col2=x.b
so you just put this together like this:
// make one of these for each row
String.Format("SELECT {0} as a, {1} as b, '{2}' as c",
row[0], row[1], row[2].Replace("'","''"))
// put it together
string expr = "UPDATE myTable SET col3=c FROM myTable JOIN (" +
String.Join(stringformatarray, " UNION ALL ") +
") x ON myTable.col1=x.a AND myTable.col2=x.b"
or you can use StringBuilder to put this together.
and then, of course, you execute this one string.