We Keep Coding

Return ID of newly inserted row on a PostgreSQL database using C# and Npgsql?

I'm building a WinForms project in C# using a PostgreSQL database and the Npgsql framework.
For inserting a record, I need to return the ID of the new record. This SO question says to add SELECT SCOPE_IDENTITY() to the query string passed to cmd. So my query string looks like this:
string insertString = "INSERT INTO sometable (company_name, category, old_value, old_desc, new_value, new_desc, reference1, reference2) VALUES (#comp, #cat, #oldValue, #oldDesc, #newValue, #newDesc, #ref1, #ref2); SELECT SCOPE_IDENTITY();";
and then get the ID with something like
int modified = cmd.ExecuteNonQuery();
But that's likely SQL Server-specific. If I use that method, I get an exception at the above line saying, "fuction scope_identity() does not exist".
I wasn't able to find anything that seemed to address this on the Npgsql documentation.
Per the linked SO question and Denis' suggestions I've tried adding both
RETURNING id;
and
CURRVAL(pg_get_serial_sequence('my_tbl_name','id_col_name'))
to the query string, replacing SELECT SCOPE_IDENTITY(); with those statements in the code above. In both cases they work as intended in DBeaver on an insert, but in my C# code in my WinForm project, modified was set to "1".
NOTE: I re-titled the question and added more information about what I've done.

Add "returning idcolumn" to the end of the sql query, then run the command with the ExecuteScalar() method instead of ExecuteNonQuery(). It should return with an int.
string insert = "insert into table1 (col1) values (something) returning idcol";
int id = cmd.ExecuteScalar();

All the comments above were almost nearly spot on and got me to a solution but didn't exactly wrap it in a bow -- so I thought i'd post my implementation that works (with silly fake example tables of course).
private int? InsertNameIntoNamesTable(string name)
{
int? id = null;
using (var dbcon = new NpgsqlConnection(_connectionString))
{
dbcon.Open();
StringBuilder sb = new StringBuilder();
var sql = $#"
insert into names_table
(name)
values
({name})
returning id;
";
sb.Append(sql);
using (var cmd = new NpgsqlCommand(sql, dbcon))
{
id = (int)cmd.ExecuteScalar();
}
dbcon.Close();
}
return id;
}

Loop over and run SQL query several thousand times

I've read several dozen posts, many dating back years, and cannot come up with a modern, safe and reliable way to update a special value in several thousand records as a single query.
I loop over all the records in the table, determine a DateTime value based on some special logic and then run this simple query to update that value... over 3500 times. That's a lot of trips over the wire.
UPDATE ScheduleTickets
SET ScheduledStartUTC = #ScheduledStartUTC
WHERE ScheduleId = #ScheduleId AND PatchSessionId = #PatchSessionId
I've seen comments to not waste memory by saving to and using a DataTable. I've seen solutions that use a StringBuilder to dynamically create an update query but that feels insecure/dirty. Sure, the entire process takes less than a minute but there must be a better way.
So, after figuring out the DateTime value, I call...
UpdateScheduleTicketStart(ScheduleId, PatchSessionId, scheduledDateTime);
Which looks like this...
private static void UpdateScheduleTicketStart(long scheduleId, long patchSessionId, DateTime scheduledStartUTC)
{
using (SqlConnection c = ConnectVRS())
{
SqlCommand cmd = new SqlCommand(#"
UPDATE ScheduleTickets
SET ScheduledStartUTC = #ScheduledStartUTC
WHERE ScheduleId = #ScheduleId AND PatchSessionId = #PatchSessionId
", c);
cmd.Parameters.Add("#ScheduleId", SqlDbType.BigInt).Value = scheduleId;
cmd.Parameters.Add("#PatchSessionId", SqlDbType.BigInt).Value = patchSessionId;
cmd.Parameters.Add("#ScheduledStartUTC", SqlDbType.VarChar).Value = scheduledStartUTC;
cmd.ExecuteNonQuery();
}
}
How can I pass all the values to SQL Server in one call or how can I create a single SQL query to do the updates in one fell swoop?

Many people have suggested using a TableValueParameter, and I agree it would be a good method. Here is an example of how you could do that:
First Create a TVP and Stored Proc in SQL Server
CREATE TYPE [dbo].[SchdeuleTicketsType] As Table
(
ScheduledStartUTC DATETIME NOT NULL
, ScheduleId INT NOT NULL
, PatchSessionId INT NOT NULL
)
CREATE PROCEDURE [dbo].[usp_UpdateTickets]
(
#ScheduleUpdates As [dbo].[SchdeuleTicketsType] Readonly
)
AS
Begin
UPDATE t1
SET t1.ScheduledStartUTC = t2.ScheduledStartUTC
FROM ScheduleTickets AS t1
INNER JOIN #ScheduleUpdates AS t2
ON t1.ScheduleId = t2.ScheduleId AND
t1.PatchSessionId = t2.PatchSessionId
End
)
Next Modify your code to populate a table and pass that as a parameter to the stored proc:
private void Populate()
{
DataTable dataTable = new DataTable("SchdeuleTicketUpdates");
//we create column names as per the type in DB
dataTable.Columns.Add("ScheduledStartUTC", typeof(DateTime));
dataTable.Columns.Add("ScheduleId", typeof(Int32));
dataTable.Columns.Add("PatchSessionId", typeof(Int32));
//write you loop to populate here
//call the stored proc
using (var conn = new SqlConnection(connString))
{
var command = new SqlCommand("[usp_UpdateTickets]");
command.CommandType = CommandType.StoredProcedure;
var parameter = new SqlParameter();
//The parameter for the SP must be of SqlDbType.Structured
parameter.ParameterName = "#ScheduleUpdates";
parameter.SqlDbType = System.Data.SqlDbType.Structured;
parameter.Value = dataTable;
command.Parameters.Add(parameter);
command.ExecuteNonQuery();
}
}

If the values are in another table, use a join:
UPDATE st
SET ScheduledStartUTC = ot.ScheduledStartUTC
FROM ScheduleTickets st JOIN
OtherTable ot
ON st.ScheduleId = ot.ScheduleId AND st.PatchSessionId = ot.PatchSessionId;
You don't specify the special logic but you can probably express it in SQL.

A better way to achieve INSERT without hitting the database multiple times

I have the following, I could make it work as I want to but I think i'm doing it the wrong way, could you please explain how this could be done in a more efficient way ? While also looping on Categories and doing the same as with Districts within the same Insert() Method.
Thanks in advance.
#region Methods
public int Insert(List<District> Districts, List<Category> Categories)
{
StringBuilder sqlString = new StringBuilder("INSERT INTO Stores (name, image) VALUES (#Name, #Image);");
using (SqlConnection sqlConnection = new
SqlConnection(ConfigurationManager.ConnectionStrings["OahuDB"].ConnectionString))
{
SqlCommand sqlCommand = new SqlCommand(sqlString.ToString(), sqlConnection);
sqlCommand.Parameters.AddWithValue("#Name", this.Name);
sqlCommand.Parameters.AddWithValue("#Image", this.Image);
sqlConnection.Open();
int x = (int)sqlCommand.ExecuteScalar();
sqlString.Clear();
sqlCommand.Parameters.Clear();
foreach (District item in Districts)
{
sqlString.AppendLine("INSERT INTO districts_has_stores (district_id, store_id) VALUES (#DistrictID, #StoreID);");
sqlCommand.CommandText = sqlString.ToString();
sqlCommand.Parameters.AddWithValue("#DistrictID", item.ID);
sqlCommand.ExecuteNonQuery();
}
return x;
}
}
EDIT
Is is wrong to achieve the above by doing the following ?
sqlString.Clear();
sqlCommand.Parameters.Clear();
sqlString.AppendLine("INSERT INTO districts_has_stores (district_id, store_id) VALUES (#DistrictID, #StoreID);");
sqlCommand.CommandText = sqlString.ToString();
sqlCommand.Parameters.AddWithValue("#StoreID", x);
foreach (District item in Districts)
{
sqlCommand.Parameters.AddWithValue("#DistrictID", item.ID);
sqlCommand.ExecuteNonQuery();
}
sqlString.Clear();
sqlCommand.Parameters.Clear();
sqlString.AppendLine("INSERT INTO categories_has_stores (category_id, store_id) VALUES (#CategoryID, #StoreID);");
sqlCommand.CommandText = sqlString.ToString();
sqlCommand.Parameters.AddWithValue("#StoreID", x);
foreach (Category item in Categories)
{
sqlCommand.Parameters.AddWithValue("#CategoryID", item.ID);
sqlCommand.ExecuteNonQuery();
}

The first obvious thing is to move the invariant part of the sqlCommand out of the loop
sqlCommand.Parameters.Clear();
sqlString.Clear();
sqlString.AppendLine("INSERT INTO districts_has_stores (district_id, store_id) VALUES (#DistrictID, #StoreID);");
sqlCommand.CommandText = sqlString.ToString();
sqlCommand.Parameters.AddWithValue("#DistrictID", 0); // as dummy value
sqlCommand.Parameters.AddWithValue("#StoreID", x); // invariant
foreach (District item in Districts)
{
sqlCommand.Parameters["#DistrictID"].Value = item.ID;
sqlCommand.ExecuteNonQuery();
}
But this doesn't answer your fundamental problem. How to avoid hitting the database multiple times.
You could build a query with multiple inserts like this
sqlString.Clear();
sqlString.Append("INSERT INTO districts_has_stores (district_id, store_id) VALUES (");
foreach(District item in Districts)
{
sqlString.Append(item.ID.ToString);
sqlString.Append(", ")
sqlString.Append(x.ToString());
sqlString.Append("),");
}
sqlString.Length--;
sqlCommand.CommandText = sqlString.ToString()
But string concatenation is really a bad practice and I present this solution just as an example and I don't want to suggest this kind of approach.
The last possibility are Table-Valued Parameters (Only from SqlServer 2008).
First you need to create a Sql Type for the table you will pass in
CREATE TYPE dbo.DistrictsType AS TABLE
( DistrictID int, StoreID int )
and a StoredProcedure that will insert the data from the datatable passed in
CREATE PROCEDURE usp_InsertDistricts
(#tvpNewDistricts dbo.DistrictsType READONLY)
AS
BEGIN
INSERT INTO dbo.Districts (DistrictID, StoreID)
SELECT dt.DistrictID, dt.StoreID FROM #tvpNewDistricts AS dt;
END
then, back to your code you pass the district into the storedprocedure
(Probably you need to convert your List in a DataTable)
DataTable dtDistricts = ConvertListToDataTable(Districts);
SqlCommand insertCommand = new SqlCommand("usp_InsertDistricts", sqlConnection);
SqlParameter p1 = insertCommand.Parameters.AddWithValue("#tvpNewDistricts", dtDistricts);
p1.SqlDbType = SqlDbType.Structured;
p1.TypeName = "dbo.DistrictsType";
insertCommand.ExecuteNonQuery();
Well, if you look back at the link above, you will find other ways to pass your data in a single step to the database backend.... (Scroll to the end and you will find also a method that doesn't require a stored procedure on the database)

Assuming Stores has an identity column, in SQL Server, create a table type and a table-valued parameter to take advantage of it:
CREATE TYPE dbo.DistrictsTVP AS TABLE
(
DistrictID INT -- PRIMARY KEY? I hope so.
);
GO
CREATE PROCEDURE dbo.InsertStoreAndDistricts
#Name NVARCHAR(255),
#Image <some data type???>,
#Districts dbo.DistrictsTVP READONLY
AS
BEGIN
SET NOCOUNT ON;
DECLARE #StoreID INT;
INSERT dbo.Stores(name, [image]) SELECT #Name, #Image;
SET #StoreID = SCOPE_IDENTITY();
INSERT dbo.district_has_stores(district_id, store_id)
SELECT DistrictID, #StoreID
FROM #Districts;
END
GO
Then in C#, you can pass your List in directly without any looping:
using (...)
{
SqlCommand cmd = new SqlCommand("dbo.InsertStoreAndDistricts", sqlConnection);
cmd.CommandType = CommandType.StoredProcedure;
SqlParameter tvparam = cmd.Parameters.AddWithValue("#Districts", Districts);
tvparam.SqlDbType    = SqlDbType.Structured;
// other params here - name and image
cmd.ExecuteNonQuery();
}

Recently in my project i used XML as a data type in my stored proc and did insert update and delete in just one shot instead of hitting the database multiple times .
Sample Stored proc
ALTER PROCEDURE [dbo].[insertStore]
#XMLDATA xml,
#name varchar(50),
#image datatype
AS
Begin
INSERT INTO Store
(name
,image
)
Select XMLDATA.item.value('#name[1]', 'varchar(10)') AS Name,
XMLDATA.item.value('#image[1]', 'yourData type') AS Image
FROM #XMLDATA.nodes('//Stores/InsertList/Store') AS XMLDATA(item)
END
Similarly you can write for update and delete .In C# u need to create the xml
public string GenerateXML(List<District> Districts)
var xml = new StringBuilder();
var insertxml = new StringBuilder();
xml.Append("<Stores>");
for (var i = 0; i < Districts.Count; i++)
{ var obj = Districts[i];
insertxml.Append("<Store");
insertxml.Append(" Name=\"" + obj.Name + "\" ");
insertxml.Append(" Image=\"" + obj.Image + "\" ");
insertxml.Append(" />");
}
xml.Append("<InsertList>");
xml.Append(insertxml.ToString());
xml.Append("</InsertList>");
SqlCommand cmd= new SqlCommand("insertStore",connectionString);
cmd.CommandType=CommandType.StoredProcedure;
SqlParameter param = new SqlParameter ();
param.ParameterName ="#XMLData";
param.value=xml;
paramter.Add(param);
cmd.ExecuteNonQuery();

Personally, I would create a stored procedure for the insert and pass in a Table-Valued param, which would allow you to do
INSERT tbl (f1, f2, ... fN)
SELECT * FROM #TVP
http://msdn.microsoft.com/en-us/library/bb510489.aspx
Unless you're using SQL 2005, then I would use an XML param in my stored proc and Serialize a collection to be inserted.

Think about your system design. Where is the data that you need to insert coming from? If it's already in the database, or another database, or some other kind of data store, you should be able to achieve a more bulk kind of transfer, simply inserting from one database to the other in a loop in stored procedure.
If the data is coming from a user, or some incompatible data store, like say an export from some third party program, then you basically have to realize that to get it into the database will involve quite of few round-trips to the database. You can use some tables, or XML or such , but those are actually closer to doing a bulk insert using other methods.
The bottom line is that SQL databases are designed to do inserts one at a time. This is 99% of the time OK because you are never asking users using the UI to type in thousands of things at one time.

How to pass sqlparameter to IN()? [duplicate]

This question already has answers here:
Pass Array Parameter in SqlCommand
(11 answers)
Closed 6 years ago.
For some reason the Sqlparameter for my IN() clause is not working. The code compiles fine, and the query works if I substitute the parameter with the actual values
StringBuilder sb = new StringBuilder();
foreach (User user in UserList)
{
sb.Append(user.UserId + ",");
}
string userIds = sb.ToString();
userIds = userIds.TrimEnd(new char[] { ',' });
SELECT userId, username
FROM Users
WHERE userId IN (#UserIds)

You have to create one parameter for each value that you want in the IN clause.
The SQL needs to look like this:
SELECT userId, username
FROM Users
WHERE userId IN (#UserId1, #UserId2, #UserId3, ...)
So you need to create the parameters and the IN clause in the foreach loop.
Something like this (out of my head, untested):
StringBuilder sb = new StringBuilder();
int i = 1;
foreach (User user in UserList)
{
// IN clause
sb.Append("#UserId" + i.ToString() + ",");
// parameter
YourCommand.Parameters.AddWithValue("#UserId" + i.ToString(), user.UserId);
i++;
}

Possible "cleaner" version:
StringBuilder B = new StringBuilder();
for (int i = 0; i < UserList.Count; i++)
YourCommand.Parameters.AddWithValue($"#UserId{i}", UserList[i].UserId);
B.Append(String.Join(",", YourCommand.Parameters.Select(x => x.Name)));

If you are using SQL 2008, you can create a stored procedure which accepts a Table Valued Parameter (TVP) and use ADO.net to execute the stored procedure and pass a datatable to it:
First, you need to create the Type in SQL server:
CREATE TYPE [dbo].[udt_UserId] AS TABLE(
[UserId] [int] NULL
)
Then, you need to write a stored procedure which accepts this type as a parameter:
CREATE PROCEDURE [dbo].[usp_DoSomethingWithTableTypedParameter]
(
#UserIdList udt_UserId READONLY
)
AS
BEGIN
SELECT userId, username
FROM Users
WHERE userId IN (SELECT UserId FROM #UserIDList)
END
Now from .net, you cannot use LINQ since it does not support Table Valued Parameters yet; so you have to write a function which does plain old ADO.net, takes a DataTable, and passes it to the stored procedure: I've written a generic function I use which can do this for any stored procedure as long as it takes just the one table-typed parameter, regardless of what it is;
public static int ExecStoredProcWithTVP(DbConnection connection, string storedProcedureName, string tableName, string tableTypeName, DataTable dt)
{
using (SqlConnection conn = new SqlConnection(connection.ConnectionString))
{
SqlCommand cmd = new SqlCommand(storedProcedureName, conn);
cmd.CommandType = CommandType.StoredProcedure;
SqlParameter p = cmd.Parameters.AddWithValue(tableName, dt);
p.SqlDbType = SqlDbType.Structured;
p.TypeName = tableTypeName;
conn.Open();
int rowsAffected = cmd.ExecuteNonQuery(); // or could execute reader and pass a Func<T> to perform action on the datareader;
conn.Close();
return rowsAffected;
}
}
Then you can write DAL functions which use this utility function with actual names of stored procedures; to build on the example in your question, here is what the code would look like:
public int usp_DoSomethingWithTableTypedParameter(List<UserID> userIdList)
{
DataTable dt = new DataTable();
dt.Columns.Add("UserId", typeof(int));
foreach (var userId in updateList)
{
dt.Rows.Add(new object[] { userId });
}
int rowsAffected = ExecStoredProcWithTVP(Connection, "usp_DoSomethingWithTableTypedParameter", "#UserIdList", "udt_UserId", dt);
return rowsAffected;
}
Note the "connection" parameter above - I actually use this type of function in a partial DataContext class to extend LINQ DataContext with my TVP functionality, and still use the (using var context = new MyDataContext()) syntax with these methods.
This will only work if you are using SQL Server 2008 - hopefully you are and if not, this could be a great reason to upgrade! Of course in most cases and large production environments this is not that easy, but FWIW I think this is the best way of doing this if you have the technology available.

SQL Server sees your IN clause as:
IN ('a,b,c')
What it needs to look like is:
IN ('a','b','c')
There is a better way to do what you're trying to do.
If the user id's are in the DB, then the IN clause should be changed to a subquery, like so:
IN (SELECT UserID FROM someTable WHERE someConditions)
This is a hack -- it doesn't work well with indexes, and you have to be careful it works right with your data, but I've used it successfully in the past:
#UserIDs LIKE '%,' + UserID + ',%' -- also requires #UserID to begin and end with a comma

How do I read the results of a stored proc into it's complex type

I've created my stored procedure, I've created the complex type using the entity model. Now assuming I have successfully established a connection to the database - I'm now ready to run the store procedure and store the rows in a List<ComplexType>. How do I do this in the best, most efficient way? I'm aware that I can iterate through the columns and rows of an SQLDataReader but it sort of feels like I'd be missing the point of the entity framework.
Thanks a lot.

You can add the stored procedure as a function import and then call it in your assembly directly:
Using (var context = new NorthwindEntities())
{
Var query = context.GetEmployeeNames(); // we import the stored procedure as a function GetEmployeeNames().
//…
}
If you use OUTPUT parameter in your stored procedure, you need to add ObjectParameters to get the return value. For example,
Using(var context = new NorthwindEntities())
{
ObjectParameter firstname = new ObjectParameter(“firstname”, typeof(String));
ObjectParameter lastname = new ObjectParameter(“lastname”, typeof(String));
Var query = context.GetEmployeeByID(123, firstname, lastname);
// Console.WriteLine(“Employee {0}’s name is: {1}.{2}.”, 123, firstname, lastname);
}
Here is a live sample:
ReferralVisitors is a Stored Procedure.