We Keep Coding

Loop over and run SQL query several thousand times

I've read several dozen posts, many dating back years, and cannot come up with a modern, safe and reliable way to update a special value in several thousand records as a single query.
I loop over all the records in the table, determine a DateTime value based on some special logic and then run this simple query to update that value... over 3500 times. That's a lot of trips over the wire.
UPDATE ScheduleTickets
SET ScheduledStartUTC = #ScheduledStartUTC
WHERE ScheduleId = #ScheduleId AND PatchSessionId = #PatchSessionId
I've seen comments to not waste memory by saving to and using a DataTable. I've seen solutions that use a StringBuilder to dynamically create an update query but that feels insecure/dirty. Sure, the entire process takes less than a minute but there must be a better way.
So, after figuring out the DateTime value, I call...
UpdateScheduleTicketStart(ScheduleId, PatchSessionId, scheduledDateTime);
Which looks like this...
private static void UpdateScheduleTicketStart(long scheduleId, long patchSessionId, DateTime scheduledStartUTC)
{
using (SqlConnection c = ConnectVRS())
{
SqlCommand cmd = new SqlCommand(#"
UPDATE ScheduleTickets
SET ScheduledStartUTC = #ScheduledStartUTC
WHERE ScheduleId = #ScheduleId AND PatchSessionId = #PatchSessionId
", c);
cmd.Parameters.Add("#ScheduleId", SqlDbType.BigInt).Value = scheduleId;
cmd.Parameters.Add("#PatchSessionId", SqlDbType.BigInt).Value = patchSessionId;
cmd.Parameters.Add("#ScheduledStartUTC", SqlDbType.VarChar).Value = scheduledStartUTC;
cmd.ExecuteNonQuery();
}
}
How can I pass all the values to SQL Server in one call or how can I create a single SQL query to do the updates in one fell swoop?

Many people have suggested using a TableValueParameter, and I agree it would be a good method. Here is an example of how you could do that:
First Create a TVP and Stored Proc in SQL Server
CREATE TYPE [dbo].[SchdeuleTicketsType] As Table
(
ScheduledStartUTC DATETIME NOT NULL
, ScheduleId INT NOT NULL
, PatchSessionId INT NOT NULL
)
CREATE PROCEDURE [dbo].[usp_UpdateTickets]
(
#ScheduleUpdates As [dbo].[SchdeuleTicketsType] Readonly
)
AS
Begin
UPDATE t1
SET t1.ScheduledStartUTC = t2.ScheduledStartUTC
FROM ScheduleTickets AS t1
INNER JOIN #ScheduleUpdates AS t2
ON t1.ScheduleId = t2.ScheduleId AND
t1.PatchSessionId = t2.PatchSessionId
End
)
Next Modify your code to populate a table and pass that as a parameter to the stored proc:
private void Populate()
{
DataTable dataTable = new DataTable("SchdeuleTicketUpdates");
//we create column names as per the type in DB
dataTable.Columns.Add("ScheduledStartUTC", typeof(DateTime));
dataTable.Columns.Add("ScheduleId", typeof(Int32));
dataTable.Columns.Add("PatchSessionId", typeof(Int32));
//write you loop to populate here
//call the stored proc
using (var conn = new SqlConnection(connString))
{
var command = new SqlCommand("[usp_UpdateTickets]");
command.CommandType = CommandType.StoredProcedure;
var parameter = new SqlParameter();
//The parameter for the SP must be of SqlDbType.Structured
parameter.ParameterName = "#ScheduleUpdates";
parameter.SqlDbType = System.Data.SqlDbType.Structured;
parameter.Value = dataTable;
command.Parameters.Add(parameter);
command.ExecuteNonQuery();
}
}

If the values are in another table, use a join:
UPDATE st
SET ScheduledStartUTC = ot.ScheduledStartUTC
FROM ScheduleTickets st JOIN
OtherTable ot
ON st.ScheduleId = ot.ScheduleId AND st.PatchSessionId = ot.PatchSessionId;
You don't specify the special logic but you can probably express it in SQL.

SQL Server - Update table and return the Updated rows

I have a SQL Server database which has a lot of information inside.
I want to select top 50 rows in a single query (which I did, with no problem) but then I want to update a column from false to true, so next time I select I wont select the same, my code looks like this:
string Command = "UPDATE HubCommands SET [Alreadytaken] = 'true' FROM (SELECT TOP 50 [CommandId],[DeviceId],[Commandtext], [HashCommand],[UserId] FROM HubCommands) I WHERE [HubId] = '18353fe9-82fd-4ac2-a078-51c199d9072b'";
using (SqlConnection myConnection = new SqlConnection(SqlConnection))
{
using (SqlDataAdapter myDataAdapter = new SqlDataAdapter(Command, myConnection))
{
DataTable dtResult = new DataTable();
myDataAdapter.Fill(dtResult);
foreach (DataRow row in dtResult.Rows)
{
Guid CommandId, DeviceId, UserId;
Guid.TryParse(row["CommandId"].ToString(), out CommandId);
Guid.TryParse(row["DeviceId"].ToString(), out DeviceId);
Guid.TryParse(row["UserId"].ToString(), out UserId);
Console.WriteLine("CommandId" + CommandId);
}
}
}
This code does work, and it updates what I ask it to update, but I don't get nothing in the data table, its like it is always updating but not selecting.
If I do a normal select it does work and give information.
Does anyone have any idea how to update and get some data back, in a single query?

So your question is:
How can I update a table in SQL Server using C# and return the truly updated
rows as a DataTable ?
First You have multiple issues in your query.
You should use 1 and 0, not true or false. SQL-Server has a bit datatype and not a Boolean.
Second, this is how you should've constructed your query:
DECLARE #IDs TABLE
(
[CommandId] uniqueidentifier
);
INSERT INTO #IDs
SELECT [CommandId] FROM HubCommands
WHERE [HubId] = '18353fe9-82fd-4ac2-a078-51c199d9072b' AND [Alreadytaken] = 0;
UPDATE HubCommands
SET [Alreadytaken] = 1
WHERE CommandId IN
(
SELECT [CommandId] FROM #IDs
);
SELECT * FROM HubCommands
WHERE CommandId IN
(
SELECT [CommandId] FROM #IDs
);
Wrap all the above in a single string and use SqlDataReader. No need for an Adapter in you case (Since we're mixing commands unlike what the adapter usually does):
var sqlCommand = new SqlCommand(Command, myConnection);
SqlDataReader dataReader = sqlCommand.ExecuteReader();
DataTable dtResult = new DataTable();
dtResult.Load(dataReader);
I highly advise you to create a stored procedure accepting HubId as a parameter that does all the above work. It is neater and better for maintenance.

No results on MySql Select after an Update using a MySqlDataAdapter with a MySqlCommandBuilder

I have the following code that is meant to simply test and try to resolve an issue I'm having where MySQL appears to return no results immediately when attempting to select a record after another record has just been updated. I've tried creating all new objects without using a loop just to make sure something wasn't still in scope. I've tried adding Thread.Sleep(3000) between loop iterations to see if that would help. So far it's been consistent, the first select and update commands are successful, and the next select fails immediately.
If, however, I remove the update, the Select works on the second iteration.
for (int orderId = 1; orderId <= 2; orderId++)
{
string sql = "SELECT * FROM tblOrder WHERE orderId = " + orderId;
var conn = GetConnection(); //Returns a new connection instance
var cmd = new MySqlCommand(sql, conn);
var da = new MySqlDataAdapter(cmd);
var cb = new MySqlCommandBuilder(da);
var orderDataTable = new DataTable();
da.Fill(orderDataTable);
if (orderDataTable.Rows.Count == 0)
throw new Exception("Order ID [" + orderId + "] not found.");
Console.Write("Order ID [ " + orderId + "] was found...");
orderDataTable.Rows[0]["notes"] = orderDataTable.Rows[0]["notes"].ToString() + " ";
da.Update(orderDataTable);
Console.Write("Update Test Passed\r\n");
}
GetConnection looks like this...
private const string ConnectionString = "myConnectionString";
public static MySqlConnection GetConnection()
{
return new MySqlConnection(ConnectionString);
}
Should go without saying, but I've definitely checked to make sure the orderId I'm selecting is there.
I believe the issue has something to do with MySql locking mechanisms, but I'm more of a SQL Server guy so I'm not terribly familiar with the peculiarities of MySql. I tried changing the SELECT to end with "FOR UPDATE" and that did not seem to help.
Update: Writing my own update statement and using a separate MySqlCommand object works, but is not preferred since my actual code updates a lot of fields and logic and I would prefer not to have to form the update statement within that logic.

ODAC seems to be caching table schema?

I'm using Oracle's ODAC.NET for a .NET 3.5 project against an Oracle 11 Express database, and I'm seeing behavior that I can't explain (and can't seem to work around).
ODAC should be the latest, I just pulled it 3 days ago, but the versions are as follows:
Oracle.DataAccess.dll version 2.112.3.0 (release 5)
oci.dll (instant client) version 11.2.0.1
I have a Table, People, that has 3 columns:
ID
FirstName
LastName
In code I run an ALTER TABLE command, using OracleCommand.ExecuteNonQuery, to add a new column named "MIDDLE_NAME" to the table. That command succeeds. If I look at the table with Oracle SQL Developer, the columns shows up. All well and good.
Now if I run use OracleCommand.ExecuteReader with a command text of SELECT * FROM People right after I do the alter table, I get back data with only 3 columns, not 4!
Here is code that reproduces the problem:
public void FieldTest()
{
var sql1 = "CREATE TABLE People (" +
"ID NUMBER PRIMARY KEY, " +
"FirstName NVARCHAR2 (200), " +
"LastName NVARCHAR2 (200) NOT NULL)";
var sql2 = "ALTER TABLE People " +
"ADD Middle_Name NUMBER";
var sql3 = "SELECT * FROM People";
var sql4 = "SELECT column_name FROM all_tab_cols WHERE table_name = 'PEOPLE'";
var cnInfo = new OracleConnectionInfo("192.168.10.246", 1521, "XE", "system", "password");
var connectionString = BuildConnectionString(cnInfo);
using (var connection = new OracleConnection(connectionString))
{
connection.Open();
using (var create = new OracleCommand(sql1, connection))
{
create.ExecuteNonQuery();
}
using (var get = new OracleCommand(sql3, connection))
{
using (var reader = get.ExecuteReader())
{
Debug.WriteLine("Columns: " + reader.FieldCount);
// outputs 3, which is right
}
}
using (var alter = new OracleCommand(sql2, connection))
{
alter.ExecuteNonQuery();
}
using (var get = new OracleCommand(sql3, connection))
{
using (var reader = get.ExecuteReader())
{
Debug.WriteLine("Columns: " + reader.FieldCount);
// outputs 3, which is *wrong* <---- Here's the problem
}
}
using (var cols = new OracleCommand(sql4, connection))
{
using (var reader = cols.ExecuteReader())
{
int count = 0;
while (reader.Read())
{
count++;
Debug.WriteLine("Col: " + reader.GetString(0));
}
Debug.WriteLine("Columns: " + count.ToString());
// outputs 4, which is right
}
}
}
}
I've tried some things to prevent the behavior, and none of them give me back the 4th column:
I close the connection and re-open it
I use a new OracleConnection for the SELECT than for the ALTER
I use the same OracleConnection for the SELECT and for the ALTER
I use a new OracleCommand for the SELECT than for the ALTER
I use the same OracleCommand for the SELECT and for the ALTER
I call PurgeStatementCache on the connection between the ALTER and SELECT
I call FlushCache on the connection between the ALTER and SELECT
I explicitly Close and Dispose the OracleCommand and OracleConnection (as opposed to the using block) used for the ALTER and SELECT
Restarted the calling PC and the PC hosting the Oracle database.
If I look at the column list by doing a SELECT * FROM all_tab_cols, the new column is there.
The only thing that seems to work reliably is closing the app and re-starting it (well it's from a unit test, but it's a shutdown and restart of the test host). Then I get that 4th column. Sometimes I can use breakpoints and re-execute queries and the 4th column will appear, but nothing that is specifically repeatable with straight execution of code (meaning without setting a break point and moving the execution point back up).
Something in the bowels of ODAC seems to be caching the schema of that table, but I can figure out what, why or how to prevent it. Anyone have any experience with this, or ideas how I might prevent it?

I know this answer comes years later but if new readers run into problems with caching try setting:
Metadata Pooling = false, Self Tuning = False and Statement Cache Size = 0
...in the connection string. Keep in mind that there are performance implications for doing so.
https://docs.oracle.com/database/122/ODPNT/featConnecting.htm#GUID-0CFEB161-68EF-4BC2-8943-3BDFFB878602

Maybe post some of your C# code. The following is a test that behaves as expected, meaning I can see the new column immediately after adding it. This is using odp 11.2 rel 5 hitting an 11g db, using 4.0 framework:
The test table is:
CREATE TABLE T1
(
DTE DATE default sysdate
);
Drop and recreate it after each run of the following C# code (a bit dirty but anyway):
string connStr = "User Id=xxx;Password=yyy;Data Source=my11gDb;";
using (OracleConnection con = new OracleConnection(connStr))
{
string s = "ALTER TABLE T1 ADD (added_col VARCHAR2(10))";
using (OracleCommand cmd = new OracleCommand(s, con))
{
con.Open();
cmd.ExecuteNonQuery();
string s2 = "select column_name from all_tab_columns where table_name = 'T1'";
//con.FlushCache(); // doesn't seem to matter, works with or without
using (OracleCommand cmd2 = new OracleCommand(s2, con))
{
OracleDataReader rdr = cmd2.ExecuteReader();
for (int i = 0; rdr.Read(); i++)
{
Console.WriteLine("Column {0} => {1}",i+1,rdr.GetString(0));
}
rdr.Close();
}
}
}
Output:
Column 1 => DTE
Column 2 => ADDED_COL
Edit:
Ah, ok, I see what you're saying, it looks like statement caching. I played around with changing the cache size to 0 (in conn string, use "Statement Cache Size=0"), and also tried cmd.AddToStatementCache = false, but these did not work.
One thing that does work is to use a slightly different string, like adding a space. I know its a hack, but this is all I can get to work for me anyway.
Try your example with:
var sql3 = "SELECT * FROM People";
var sql5 = "SELECT * FROM People "; // note extra space
And use sql3 before adding column, and sql5 after adding a column.
Hope that helps

In C#, is "SELECT TOP 0 * FROM (/* ... */) s" used in conjuction with ADO.NET a good way to determine the column information in a SELECT statement?

I have a SQL SELECT statement which will not be known until runtime, which could contain JOIN's and inner selects. I need to determine the names and data types of each of the columns of the returned result of the statment from within C#. I am inclined to do something like:
string orginalSelectStatement = "SELECT * FROM MyTable";
string selectStatement = string.Format("SELECT TOP 0 * FROM ({0}) s", orginalSelectStatement);
SqlConnection connection = new SqlConnection(#"MyConnectionString");
SqlDataAdapter adapter = new SqlDataAdapter(selectStatement, connection);
DataTable table = new DataTable();
adapter.Fill(table);
foreach (DataColumn column in table.Columns)
{
Console.WriteLine("Name: {0}; Type: {1}", column.ColumnName, column.DataType);
}
Is there a better way to do what I am trying to do? By "better" I mean either a less resource-intensive way of accomplishing the same task or a more sure way of accomplishing the same task (i.e. for all I know the code snippet I just gave will fail in some situations).
SOLUTION:
First of all, my TOP 0 hack is bad, namely for something like this:
SELECT TOP 0 * FROM (SELECT 0 AS A, 1 AS A) S
In other words, in a sub-select, if two things are aliased to the same name, that throws an error. So it is out of the picture. However, for completeness sake, I went ahead and tested it, along with the two proposed solutions: SET FMTONLY ON and GetSchemaTable.
Here are the results (in milliseconds for 1,000 queries, each):
Schema Time: 3130
TOP 0 Time: 2808
FMTONLY ON Time: 2937
My recommendation would be GetSchemaTable since it's more likely to be future-proofed by a removal of the SET FMTONLY ON as valid SQL and it solves the aliasing problem, even though it is slightly slower. However, if you "know" that duplicate column names will never be an issue, then TOP 0 is faster than GetSchemaTable and is more future-proofed than SET FMTONLY ON.
Here is my experimental code:
int schemaTime = 0;
int topTime = 0;
int fmtOnTime = 0;
SqlConnection connection = new SqlConnection(#"MyConnectionString");
connection.Open();
SqlCommand schemaCommand = new SqlCommand("SELECT * FROM MyTable", connection);
SqlCommand topCommand = new SqlCommand("SELECT TOP 0 * FROM (SELECT * FROM MyTable) S", connection);
SqlCommand fmtOnCommand = new SqlCommand("SET FMTONLY ON; SELECT * FROM MyTable", connection);
for (int i = 0; i < 1000; i++)
{
{
DateTime start = DateTime.Now;
using (SqlDataReader reader = schemaCommand.ExecuteReader(CommandBehavior.SchemaOnly))
{
DataTable table = reader.GetSchemaTable();
}
DateTime stop = DateTime.Now;
TimeSpan span = stop - start;
schemaTime += span.Milliseconds;
}
{
DateTime start = DateTime.Now;
DataTable table = new DataTable();
SqlDataAdapter adapter = new SqlDataAdapter(topCommand);
adapter.Fill(table);
DateTime stop = DateTime.Now;
TimeSpan span = stop - start;
topTime += span.Milliseconds;
}
{
DateTime start = DateTime.Now;
DataTable table = new DataTable();
SqlDataAdapter adapter = new SqlDataAdapter(fmtOnCommand);
adapter.Fill(table);
DateTime stop = DateTime.Now;
TimeSpan span = stop - start;
fmtOnTime += span.Milliseconds;
}
}
Console.WriteLine("Schema Time: " + schemaTime);
Console.WriteLine("TOP 0 Time: " + topTime);
Console.WriteLine("FMTONLY ON Time: " + fmtOnTime);
connection.Close();

You could use GetSchemaTable to do what you want.
There is an example of how to use it here.

If using SQL Server, I would try using SET FMTONLY ON
Returns only metadata to the client. Can be used to test the format of
the response without actually running the query.
Apparently on SQL Server 2012, there's a better way. All is specified in the linked MSDN article.
BTW, this technique is what LINQ To SQL uses internally to determine the result set returned by a stored procedure, etc.

Dynamic SQL is always a bit of a minefield, but you could the SET FMTONLY ON on your query - this means the query will only return Metadata, the same as if no results were returned. So:
string selectStatement = string.Format("SET FMTONLY ON; {0}", orginalSelectStatement);
Alternatively, if you aren't tied to ADO, could you not go down the Linq-to-SQL route and generate a data context which will map out all of your database schemas in to code and their relevant types? You could also have a look at some of the Micro ORMs out there, such as Dapper.Net
There are plenty of other ORMs out there too.