We Keep Coding

Is there a way to create a new table type and pass a data-table without a procedure?

I'm using C# and SQL Server in Windows environment with Visual Studio 2017. I'm trying to pass a datatable (called #profiles) to the SQL script.
In order to do so, I first must create a type of a table that matches the data table passed.
Problem is that in every way I tried to populate a new table with the datatable passed I'm getting one of two exceptions:
"Column, parameter, or variable #profiles. : Cannot find data type ProfileIdTableType."
"The table type parameter '#profiles' must have a valid type name."
From what I searched I could find that a datatable with new table type is generally used with a procedure, but no matter - I still get the above exceptions.
I tried to declare a new table type and use the #profiles with it with no success.
When I declare the SqlParameter I'm using to pass it, I generally encounter the first exception (can't find the type)
I should mention that I can't find the created type in the "Programmability" section of SQL Server (but my type is temp and so it should be)
These are 2 ways I'm using to pass the datatable to the script from C#:
SqlParameter #profiles = new SqlParameter("#profiles", profileIds.Tables[0]);
profiles.TypeName = "ProfileIdTableType";
or:
DbParameter #profiles = new SqlParameter("#profiles", profileIds.Tables[0]);
and then use it:
updatedProfiles = (int)DbAdminOps.ExecuteNonQueryCommand(updateProfileSettingsCommand, CommandType.Text, new DbParameter[] { #profiles, #updatedTemplate }, null);
This is the SQL script i used last (but tried many variations not presented here)
-- create a table type of profile Ids passed by user
CREATE TYPE ProfileIdTableType AS TABLE (ID INT)
go
DECLARE #PRFL ProfileIdTableType
GO
CREATE PROCEDURE PopulateTable
#profiles ProfileIdTableType READONLY
AS
INSERT INTO #PRFL(ID)
SELECT [ID] FROM #profiles
GO
#profiles ProfileIdTableType
EXEC PopulateTable #profiles
go
I expected #profiles be recognized as a table so i can use it in my script but all i get really is exception. I put a lot of effort into it but just couldn't.
Went through all the stack overflow questions, youtubes, microsoft documentation and web.
If there's any information I left out and is important - let me know.
Would really appreciate some advice.
Cheers!

The key point is to specify SqlDbType as Structured plus to define TypeName as shown in the following snippet.
comm.Parameters.AddWithValue("#tvpEmails", dt);
// EMAIL.TVP_Emails should exist on your SQL instance under UDDT types
comm.Parameters[comm.Parameters.Count - 1].TypeName = "EMAIL.TVP_Emails";
comm.Parameters[comm.Parameters.Count - 1].SqlDbType = SqlDbType.Structured;
See the complete code down below. Please let me know if you have any difficulties.
using System.Data;
using System.Data.SqlClient;
using System.Net.Mail;
namespace ConsoleApp10
{
class Program
{
static void Main(string[] args)
{
var mm = new MailMessage();
using (var conn = new SqlConnection("your connection string"))
{
using (var comm = new SqlCommand())
{
comm.Connection = conn;
conn.Open();
comm.CommandText =
#"INSERT INTO [EMail].[MailAttachments] (fileName,fileSize,attachment)
SELECT fileName, fileSize, attachment FROM #tvpEmails";
var dt = CreateTable();
foreach (var eml in mm.Attachments)
{
var newRow = dt.NewRow();
newRow["FileName"] = eml.Name;
newRow["FileSize"] = eml.ContentStream.Length;
var allBytes = new byte[eml.ContentStream.Length];
newRow["Attachment"] = allBytes;
eml.ContentStream.Position = 0;
dt.Rows.Add(newRow);
}
comm.Parameters.AddWithValue("#tvpEmails", dt);
comm.Parameters[comm.Parameters.Count - 1].TypeName = "EMAIL.TVP_Emails";
comm.Parameters[comm.Parameters.Count - 1].SqlDbType = SqlDbType.Structured;
comm.ExecuteNonQuery();
if (conn.State == ConnectionState.Open)
conn.Close();
}
}
}
private static DataTable CreateTable()
{
var dt = new DataTable();
dt.Columns.Add("FileName", typeof(string));
dt.Columns.Add("FileSize", typeof(long));
dt.Columns.Add("Attachment", typeof(byte[]));
return dt;
}
}
}

mysql stored procedure bulk insert

I have a stored procedure that looks like that:
InsertItem: INSERT INTO (IN itemId INT, name TEXT);
Is there a way I could execute a bulk of it?
like instead of executing something like that:
using (MySqlConnection connection = new MySqlConnection(_connectionString))
{
connection.Open();
foreach (Item item in GetItems())
{
using (MySqlCommand command = new MySqlCommand("InsertItem", connection))
{
command.CommandType = CommandType.StoredProcedure;
command.Parameters.AddWithValue("#itemId", item.ItemId);
command.Parameters.AddWithValue("#name", item.Name);
command.ExecuteNonQuery();
}
}
}
I'm trying to achieve code looking like that without successing:
using (MySqlConnection connection = new MySqlConnection(_connectionString))
{
connection.Open();
using (MySqlCommandBulk command = new MySqlCommand("InsertItem", connection))
{
command.CommandType = CommandType.StoredProcedure;
for (Item item in GetItems())
{
MySqlCommandBulkItem bulkItem = new MySqlCommandBulkItem();
bulkItem["itemId"] = item.ItemId;
bulkItem["name"] = item.Name;
command.BulkItems.Add(bulkItem);
}
command.Execute();
}
}
My point is that the command will send all of the data at once, and will not send each query alone.
Any ideas?

The Oracle connector for the Dotnet framework allows the use of arrays in place of scalars on parameters. But the MySQL connector doesn't.
There are two ways to accelerate bulk loads in MySQL.
One of them applies to InnoDB tables but doesn't help with MyISAM tables. Start a transaction. Then, after every few hundred rows, COMMIT it and start another one. That will commit your table inserts in bunches, which is faster than autocommiting them individually.
The other is to use MySQL's LOAD DATA INFILE command to slurp up a data file and bulk-insert it into the database. This is very fast, but you have to be diligent about formatting your file correctly.

Transferring data between two Access databases

I am writing a simple reporting tool that will need to move data from a table in one Access database to a table in another Access database (the table structure is identical). However, I am new to C# and am finding it hard to come up with a reliable solution.
Any pointers would be greatly appreciated.

Access SQL supports using an IN clause to specify that a table resides in a different database. The following C# code SELECTs rows from a table named [YourTable] in Database1.accdb and INSERTs them into an existing table named [YourTable] (with the identical structure) in Database2.accdb:
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Data.OleDb;
namespace oleDbTest
{
class Program
{
static void Main(string[] args)
{
string myConnectionString;
myConnectionString =
#"Provider=Microsoft.ACE.OLEDB.12.0;" +
#"Data Source=C:\Users\Public\Database1.accdb;";
using (var con = new OleDbConnection())
{
con.ConnectionString = myConnectionString;
con.Open();
using (var cmd = new OleDbCommand())
{
cmd.Connection = con;
cmd.CommandType = System.Data.CommandType.Text;
cmd.CommandText =
#"INSERT INTO YourTable IN 'C:\Users\Public\Database2.accdb' " +
#"SELECT * FROM YourTable WHERE ID < 103";
cmd.ExecuteNonQuery();
}
con.Close();
}
Console.WriteLine("Done.");
}
}
}

Many ways.
0) If it's only once, copy and paste the table.
1) If you want to do this inside Access, the easiest way is to create a linked table in the new database, and then a make table query in the new database.
2) You can reference the second table directly.
SELECT *
FROM TableInDbX IN 'C:\SomeFolder\DB X';
3) In a macro, you can use the TransferDatabase method of the DoCmd object to link relevant tables and then run suitable append and update queries to synchronize.
4) VBA
http://www.techonthenet.com/access/questions/new_mdb.php

Given column names Col1, Col2, and Col3:
private static void Migrate(string dbConn1, string dbConn2) {
// DataTable to store your info into
var table = new DataTable();
// Modify your SELECT command as needed
string sqlSelect = "SELECT Col1, Col2, Col3 FROM aTableInOneAccessDatabase ";
// Notice this uses the connection string to DB1
using (var cmd = new OleDbCommand(sqlSelect, new OleDbConnection(dbConn1))) {
cmd.Connection.Open();
table.Load(cmd.ExecuteReader());
cmd.Connection.Close();
}
// Modify your INSERT command as needed
string sqlInsert = "INSERT INTO aTableInAnotherAccessDatabase " +
"(Col1, Col2, Col3) VALUES (#Col1, #Col2, #Col3) ";
// Notice this uses the connection string to DB2
using (var cmd = new OleDbCommand(sqlInsert, new OleDbConnection(dbConn2))) {
// Modify these database parameters to match the signatures in the new table
cmd.Parameters.Add("#Col1", DbType.Int32);
cmd.Parameters.Add("#Col2", DbType.String, 50);
cmd.Parameters.Add("#Col3", DbType.DateTime);
cmd.Connection.Open();
foreach (DataRow row in table.Rows) {
// Fill in each parameter with data from your table's row
cmd.Parameters["#Col1"].Value = row["Col1"];
cmd.Parameters["#Col2"].Value = row["Col2"];
cmd.Parameters["#Col3"].Value = row["Col3"];
// Insert that data
cmd.ExecuteNonQuery();
}
cmd.Connection.Close();
}
}
Now, I do not work with Access databases very often, so you may need to tweak something up there.
That should get you well on your way, though.
Worth noting:
If I remember correctly, Access does NOT pay attention to your OleDbParameter names! You could call them whatever you want, and in fact most people just use a question mark ? for the parameter fields.
So, you have to add and update these parameters in the same order that your statement calls them.
So, why did I name the parameters #Col1, #Col2, #Col3? Here, it just to help you and me understand where each parameter is intended to map to. It is also good practice to get into. If you ever migrate to a better database, hopefully it will pay attention to what the parameters are named.

A better way to achieve INSERT without hitting the database multiple times

I have the following, I could make it work as I want to but I think i'm doing it the wrong way, could you please explain how this could be done in a more efficient way ? While also looping on Categories and doing the same as with Districts within the same Insert() Method.
Thanks in advance.
#region Methods
public int Insert(List<District> Districts, List<Category> Categories)
{
StringBuilder sqlString = new StringBuilder("INSERT INTO Stores (name, image) VALUES (#Name, #Image);");
using (SqlConnection sqlConnection = new
SqlConnection(ConfigurationManager.ConnectionStrings["OahuDB"].ConnectionString))
{
SqlCommand sqlCommand = new SqlCommand(sqlString.ToString(), sqlConnection);
sqlCommand.Parameters.AddWithValue("#Name", this.Name);
sqlCommand.Parameters.AddWithValue("#Image", this.Image);
sqlConnection.Open();
int x = (int)sqlCommand.ExecuteScalar();
sqlString.Clear();
sqlCommand.Parameters.Clear();
foreach (District item in Districts)
{
sqlString.AppendLine("INSERT INTO districts_has_stores (district_id, store_id) VALUES (#DistrictID, #StoreID);");
sqlCommand.CommandText = sqlString.ToString();
sqlCommand.Parameters.AddWithValue("#DistrictID", item.ID);
sqlCommand.ExecuteNonQuery();
}
return x;
}
}
EDIT
Is is wrong to achieve the above by doing the following ?
sqlString.Clear();
sqlCommand.Parameters.Clear();
sqlString.AppendLine("INSERT INTO districts_has_stores (district_id, store_id) VALUES (#DistrictID, #StoreID);");
sqlCommand.CommandText = sqlString.ToString();
sqlCommand.Parameters.AddWithValue("#StoreID", x);
foreach (District item in Districts)
{
sqlCommand.Parameters.AddWithValue("#DistrictID", item.ID);
sqlCommand.ExecuteNonQuery();
}
sqlString.Clear();
sqlCommand.Parameters.Clear();
sqlString.AppendLine("INSERT INTO categories_has_stores (category_id, store_id) VALUES (#CategoryID, #StoreID);");
sqlCommand.CommandText = sqlString.ToString();
sqlCommand.Parameters.AddWithValue("#StoreID", x);
foreach (Category item in Categories)
{
sqlCommand.Parameters.AddWithValue("#CategoryID", item.ID);
sqlCommand.ExecuteNonQuery();
}

The first obvious thing is to move the invariant part of the sqlCommand out of the loop
sqlCommand.Parameters.Clear();
sqlString.Clear();
sqlString.AppendLine("INSERT INTO districts_has_stores (district_id, store_id) VALUES (#DistrictID, #StoreID);");
sqlCommand.CommandText = sqlString.ToString();
sqlCommand.Parameters.AddWithValue("#DistrictID", 0); // as dummy value
sqlCommand.Parameters.AddWithValue("#StoreID", x); // invariant
foreach (District item in Districts)
{
sqlCommand.Parameters["#DistrictID"].Value = item.ID;
sqlCommand.ExecuteNonQuery();
}
But this doesn't answer your fundamental problem. How to avoid hitting the database multiple times.
You could build a query with multiple inserts like this
sqlString.Clear();
sqlString.Append("INSERT INTO districts_has_stores (district_id, store_id) VALUES (");
foreach(District item in Districts)
{
sqlString.Append(item.ID.ToString);
sqlString.Append(", ")
sqlString.Append(x.ToString());
sqlString.Append("),");
}
sqlString.Length--;
sqlCommand.CommandText = sqlString.ToString()
But string concatenation is really a bad practice and I present this solution just as an example and I don't want to suggest this kind of approach.
The last possibility are Table-Valued Parameters (Only from SqlServer 2008).
First you need to create a Sql Type for the table you will pass in
CREATE TYPE dbo.DistrictsType AS TABLE
( DistrictID int, StoreID int )
and a StoredProcedure that will insert the data from the datatable passed in
CREATE PROCEDURE usp_InsertDistricts
(#tvpNewDistricts dbo.DistrictsType READONLY)
AS
BEGIN
INSERT INTO dbo.Districts (DistrictID, StoreID)
SELECT dt.DistrictID, dt.StoreID FROM #tvpNewDistricts AS dt;
END
then, back to your code you pass the district into the storedprocedure
(Probably you need to convert your List in a DataTable)
DataTable dtDistricts = ConvertListToDataTable(Districts);
SqlCommand insertCommand = new SqlCommand("usp_InsertDistricts", sqlConnection);
SqlParameter p1 = insertCommand.Parameters.AddWithValue("#tvpNewDistricts", dtDistricts);
p1.SqlDbType = SqlDbType.Structured;
p1.TypeName = "dbo.DistrictsType";
insertCommand.ExecuteNonQuery();
Well, if you look back at the link above, you will find other ways to pass your data in a single step to the database backend.... (Scroll to the end and you will find also a method that doesn't require a stored procedure on the database)

Assuming Stores has an identity column, in SQL Server, create a table type and a table-valued parameter to take advantage of it:
CREATE TYPE dbo.DistrictsTVP AS TABLE
(
DistrictID INT -- PRIMARY KEY? I hope so.
);
GO
CREATE PROCEDURE dbo.InsertStoreAndDistricts
#Name NVARCHAR(255),
#Image <some data type???>,
#Districts dbo.DistrictsTVP READONLY
AS
BEGIN
SET NOCOUNT ON;
DECLARE #StoreID INT;
INSERT dbo.Stores(name, [image]) SELECT #Name, #Image;
SET #StoreID = SCOPE_IDENTITY();
INSERT dbo.district_has_stores(district_id, store_id)
SELECT DistrictID, #StoreID
FROM #Districts;
END
GO
Then in C#, you can pass your List in directly without any looping:
using (...)
{
SqlCommand cmd = new SqlCommand("dbo.InsertStoreAndDistricts", sqlConnection);
cmd.CommandType = CommandType.StoredProcedure;
SqlParameter tvparam = cmd.Parameters.AddWithValue("#Districts", Districts);
tvparam.SqlDbType    = SqlDbType.Structured;
// other params here - name and image
cmd.ExecuteNonQuery();
}

Recently in my project i used XML as a data type in my stored proc and did insert update and delete in just one shot instead of hitting the database multiple times .
Sample Stored proc
ALTER PROCEDURE [dbo].[insertStore]
#XMLDATA xml,
#name varchar(50),
#image datatype
AS
Begin
INSERT INTO Store
(name
,image
)
Select XMLDATA.item.value('#name[1]', 'varchar(10)') AS Name,
XMLDATA.item.value('#image[1]', 'yourData type') AS Image
FROM #XMLDATA.nodes('//Stores/InsertList/Store') AS XMLDATA(item)
END
Similarly you can write for update and delete .In C# u need to create the xml
public string GenerateXML(List<District> Districts)
var xml = new StringBuilder();
var insertxml = new StringBuilder();
xml.Append("<Stores>");
for (var i = 0; i < Districts.Count; i++)
{ var obj = Districts[i];
insertxml.Append("<Store");
insertxml.Append(" Name=\"" + obj.Name + "\" ");
insertxml.Append(" Image=\"" + obj.Image + "\" ");
insertxml.Append(" />");
}
xml.Append("<InsertList>");
xml.Append(insertxml.ToString());
xml.Append("</InsertList>");
SqlCommand cmd= new SqlCommand("insertStore",connectionString);
cmd.CommandType=CommandType.StoredProcedure;
SqlParameter param = new SqlParameter ();
param.ParameterName ="#XMLData";
param.value=xml;
paramter.Add(param);
cmd.ExecuteNonQuery();

Personally, I would create a stored procedure for the insert and pass in a Table-Valued param, which would allow you to do
INSERT tbl (f1, f2, ... fN)
SELECT * FROM #TVP
http://msdn.microsoft.com/en-us/library/bb510489.aspx
Unless you're using SQL 2005, then I would use an XML param in my stored proc and Serialize a collection to be inserted.

Think about your system design. Where is the data that you need to insert coming from? If it's already in the database, or another database, or some other kind of data store, you should be able to achieve a more bulk kind of transfer, simply inserting from one database to the other in a loop in stored procedure.
If the data is coming from a user, or some incompatible data store, like say an export from some third party program, then you basically have to realize that to get it into the database will involve quite of few round-trips to the database. You can use some tables, or XML or such , but those are actually closer to doing a bulk insert using other methods.
The bottom line is that SQL databases are designed to do inserts one at a time. This is 99% of the time OK because you are never asking users using the UI to type in thousands of things at one time.

How can I insert each element in a 1D array into a new row in an SQL Database using C#?

I have an array containing a list of file paths that I want to insert into an SQL database.
string[] filePaths = Directory.GetFiles(#"C:\Test Folder");
I am confident in setting up and connecting to the database, I just can't figure out how to take each element and place it in a new row in the database.
Thanks,
Matt

It depends on which technology you are using (Please note that when inserting lots of rows the use of SqlBulkCopy is recommended).
ADO.NET
foreach (var path in filePaths)
{
var command = new SqlCommand("INSERT INTO mytable(col1) VALUES(#param1)", connection);
command.Parameters.AddWithValue("#param1", path);
command.ExecuteNonQuery();
}
LINQ-to-SQL
var projection = filePaths.Select(a => new MyTable() { Col1 = a });
myContext.InsertAllOnSubmit(projection);
myContext.SubmitChanges();
LINQ-to-Entities
foreach (var path in filePaths)
{
myModel.MyTable.AddObject(new MyTable() { Col1 = path });
}
myModel.SaveChanges();

foreach(string fp in filePaths)
{
InsertIntoDb(fp);
}
//Method
public void InsertIntoDb(string insert)
{
SqlConnection con = //Setup DB connection
SqlCommand command = new SqlCommand();
command.Connection = con;
command.CommandText = "Insert #insert into Table";
command.Parameters.AddWithValue("#insert", insert);
command.ExecuteNonQuery();
}
This leaves out a lot, but it should point you in the right direction. Essentially, you want to set up your connection, instantiate a SqlCommand object where the command text is the SQL Text to insert your value (Better would be a stored procedure or some other sanitized way to insert the data to avoid Sql Injection), and then call ExecuteNonQuery() to insert the actual data.

The most efficient way is to use SqlBulkCopy
you will have to project your data into a DataTable, DataRow[], etc. OR IDataReader (which is more efficient - refer this discussion and example) in order to use SqlBulkCopy