I have a data table with 1000 of records. I want to perform a bulk insert operation from C# to PGSQL. I know one way which copies a text file to pgtable.
The example syntax is as follows:
using (NpgsqlConnection conn = new NpgsqlConnection("Server=127.0.0.1;User Id=postgres;Password=postgres;Database=postgres;"))
{
onn.Open();
NpgsqlCommand command = new NpgsqlCommand("copy \"schema\".\"tablename\" (\"col1\",\"col2\") from 'C:\\datafile.txt'", conn);
command.ExecuteNonQuery();
conn.Close();
}
Is there any other function which I can use to pass the data table instead of writing data into a text file? I m using Npgsql.dll.
You should probably fully read the PostgreSQL docs for COPY.
COPY can either be used to import a file that exists in the PostgreSQL server filesystem (as your code sample shows), or it can be used to copy data from the client, which is probably what you're looking for. The latter is triggered by substituting STDIN for the filename.
If you want to import data from your client program using Npgsql, please read the Npgsql COPY docs as well. For textual data import you'll likely need to call NpgsqlConnection.BeginTextImport(), there's a sample for that in the docs.
public bool CopyFileToPostgress(String tableName, String filePath,String delimiter)
{
NpgsqlConnection conn = new NpgsqlConnection("Host=xx.xx.xx.xx;port=xxxx;Database=xxxx;Username=xxxx;Password=xxxx;Pooling=false;Timeout=300;CommandTimeout=300");
NpgsqlCommand cmd = new NpgsqlCommand();
Boolean result = true;
try
{
conn.Open();
NpgsqlTransaction transaction = conn.BeginTransaction();
if (File.Exists(filePath))
{
try
{
NpgsqlCommand command = new NpgsqlCommand($"COPY {tableName} FROM '{filePath}' (DELIMITER '{delimiter}')", conn);
command.ExecuteNonQuery();
}
catch (Exception e)
{
result = false;
transaction.Rollback();
throw e;
}
finally
{
if (result)
{
transaction.Commit();
}
transaction.Dispose();
}
}
}
catch (Exception ex)
{
result = false;
}
finally
{
cmd.Dispose();
conn.Close();
conn.Dispose();
}
return result;
}
Related
I was trying to write a program to perform large table, around 2 billions records, to another table. It looks to me that SqlBulkCopy will need to wait until all data is read from the SqlDataReader before inserting. If I use the same query and table in SSIS, SSIS was started right away and I could see data inserting from the target table.
Am I coding it correctly? How can I make it SSIS alike?
using (SqlConnection conn = new SqlConnection(connectionString))
{
conn.Open();
using (SqlCommand cmd = new SqlCommand(bukCopyData.SourceQuery, conn))
{
cmd.CommandTimeout = 0;
using (SqlDataReader reader = cmd.ExecuteReader())
{
performBulkCopy(connectionStringDest, DestinationTable, reader);
}
}
}
private void performBulkCopy(string connectionString, string destinationTable, SqlDataReader reader)
{
using (SqlBulkCopy sbc = new SqlBulkCopy(connectionString,
SqlBulkCopyOptions.KeepIdentity | SqlBulkCopyOptions.CheckConstraints | SqlBulkCopyOptions.KeepNulls
))
{
sbc.DestinationTableName = destinationTable;
sbc.BatchSize = 102400;
sbc.BulkCopyTimeout = 0;
try
{
sbc.WriteToServer(reader);
} catch (Exception e)
{
throw;
} finally
{
reader.Close();
}
}
}
Actualy, the EnableStreaming property seems to work. I had to make sure the column order in the query is matching with the target table. Not sure why it needs a long time to throw the error.
I'm trying to copy some columns from one table located on the MySQL Server, to a local MySQL database using C#. How can I do it ?
I'm not sure if i should use MySQLDataAdapter or Bulk Copy (which i don't understand much).
Bulk copy would be an easier method. It takes a previously created datatable and uploads it to the database.
You can use the following code to help. The below code allows you to return a datatable:
public DataTable FillTable(string sql)
{
MySqlCommand sqlQuery = new MySqlCommand();
cmd.Connection = ;// insert the connection details of the DB to transfer FROM
cmd.CommandText = sql;
DataTable dt = new DataTable();
try
{
conn.Open();
dt.Load(cmd.ExecuteReader());
conn.Close();
}
catch (SqlException ex)
{
Console.WriteLine("fillTable: "+ ex.Message);
}
return dt;
}
Then using the following you can send the data over to your new database:
Datatable newTable = FillTable("SELECT * FROM MyOldTable");
using (MySqlBulkCopy destination = new MySqlBulkCopy("MyNewDatabaseConnection String"))
{
destination.DestinationTableName = "myNewTable";
try
{
destination.WriteToServer(newTable);
}
catch (Exception Ex)
{
Console.WriteLine(Ex.Message);
}
}
Reference MSDN
We're looking for a hands-on way to create a database from a given connection string with minimum code possible (e.g., to include in LINQPad scripts or C# scriptlets).
Is there a one-liner or something similar to create a database from a connection string? Most preferably something like "Static.CreateDatabase("connection string").
I have created a static class Scripts which contains a static method CreateDatabase
// USE Class Like this
Scripts.CreateDatabase("Connectionstring")
//Class
static class Scripts
{
static bool CreateDatabase(string Connectionstr)
{
bool result =false;
SqlConnection Conn = new SqlConnection(Connectionstr); // pass connection string and user must have the permission to create a database,
string Query = "CREATE DATABASE Exampledatabase ";
SqlCommand Command = new SqlCommand(Query, Conn);
try
{
Conn .Open();
Command.ExecuteNonQuery();
result =true;
}
catch (System.Exception ex)
{
result =false;
}
finally
{
if (Conn.State == ConnectionState.Open)
{
Conn.Close();
}
}
return result;
}
}
SqlConnection myConn = new SqlConnection ("Server=localhost;Integrated security=SSPI;database=master");
String sql = "CREATE DATABASE MyDatabase";
SqlCommand myCommand = new SqlCommand(sql, myConn);
try {
myConn.Open();
myCommand.ExecuteNonQuery();
// OK
} catch (System.Exception ex) {
// failed
} finally {
if (myConn.State == ConnectionState.Open) {
myConn.Close();
}
}
Download nuget add as a reference to LINQPad.
Then use comand:
void Main()
{
var database = new ProductivityTools.CreateSQLServerDatabase.Database("XXX", "Server=.\\SQL2019;Trusted_Connection=True;");
database.Create();
}
I'm trying to change my program from an SqlConnection to an OleDbConnection but I've hit a little block.
My program works as expected using SqlConnection but I can't get it to work with OleDb.
My program reads the results of a stored procedure (XML), sends it to a web service, then stores the results in a table.
I'm having issues with reading the XML from the stored procedure.
Here's my code for the first part:
public static bool BuildXml()
{
using (OleDbCommand buildXml = new OleDbCommand("usp_BUILD_RISKCALC_XML", SqlOleDbConnection))
{
buildXml.CommandType = CommandType.StoredProcedure;
try
{
OleDbDataReader reader = buildXml.ExecuteScalar();
while (reader.Read())
{
SendXml = reader.GetString(0);
}
}
catch (Exception ex)
{
WriteLog(ex.Message, 101);
return false;
}
}
}
I'm getting an InvalidOperationException the text of which reads
Specified cast is not valid
For reader.GetString(0).
I am 100% sure the stored procedure is working as I have tested using an SqlConnection as well as running in SQL Server Management Studio.
Your code should be;
public static bool BuildXml()
{
using (OleDbCommand buildXml = new OleDbCommand("usp_BUILD_RISKCALC_XML", SqlOleDbConnection))
{
buildXml.CommandType = CommandType.StoredProcedure;
try
{
object value = buildXml.ExecuteScalar();
SendXml = Convert.ToString(value);
}
catch (Exception ex)
{
WriteLog(ex.Message, 101);
return false;
}
}
}
The code buildXml.ExecuteScalar() returns an object. So casting it to an OleDbDataReader will end up in the InvalidOperationException.
ExecuteScalar return type's is object and you can't convert it to OleDbDataReader. When you have used ExecuteScalar means that you are sure that you want to return only one value from your stored procedure. You don't need OleDbDataReader just try following:
SendXml = buildXml.ExecuteScalar().ToString;
I am reading rows from a database which contains BLOB's. I take each of these blobs and saves them as a file to my disk. Afterwards I want to delete the rows I have just read. Now the problem: When I debug, all works like a charm. When I run without having a debug breakpoint, it doesn't delete anything! And it doesn't come up with an error.
I use C# and MS-SQL server.
Here is my code:
class Program
{
static void Main(string[] args)
{
if (args[0] == "/?" || args.Length != 1)
{
Console.WriteLine("BlobReader");
Console.WriteLine("Will read blob from database and write is as a file to destination specified in database.");
Console.WriteLine();
Console.WriteLine("BlobReader <AppName>");
Console.WriteLine();
Console.WriteLine("<AppName>: Application name which identifies which files to extract and save.");
EndProgram();
}
string now = DateTime.Now.ToString("yyyy-MM-dd hh:mm:ss.fff");
Console.WriteLine("writing files to disk");
bool success = SqlBlob2File(args[0], now);
if (success)
{
Console.WriteLine("Deleting db");
DeleteRows(args[0], now);
Console.WriteLine("Db deleted");
}
EndProgram();
}
static void EndProgram()
{
Console.WriteLine();
Console.WriteLine("Press any key to end.");
Console.ReadLine();
}
static private void DeleteRows(string app, string now)
{
try
{
string connectionString = ConfigurationManager.ConnectionStrings["dbConn"].ToString();
SqlConnection connection = new SqlConnection(connectionString);
string sql = string.Format("DELETE FROM Blobs WHERE Application = '{0}' AND CreatedDate < '{1}'", app,
now);
SqlCommand cmd = new SqlCommand(sql, connection);
connection.Open();
cmd.BeginExecuteNonQuery();
connection.Close();
}
catch (Exception ex)
{
Console.WriteLine(ex.Message);
}
}
static private bool SqlBlob2File(string app, string now)
{
bool success;
string connectionString = ConfigurationManager.ConnectionStrings["dbConn"].ToString();
SqlConnection connection = new SqlConnection(connectionString);
try
{
int blobCol = 0; // the column # of the BLOB field
string sql =
string.Format(
"SELECT Blob, Drive, Folder, FileName FROM Blobs WHERE Application='{0}' AND CreatedDate < '{1}'",
app, now);
SqlCommand cmd = new SqlCommand(sql, connection);
connection.Open();
SqlDataReader dr = cmd.ExecuteReader();
while (dr.Read())
{
string destFilePath = string.Format("{0}{1}{2}", dr["Drive"], dr["Folder"], dr["FileName"]);
Byte[] b = new Byte[(dr.GetBytes(blobCol, 0, null, 0, int.MaxValue))];
dr.GetBytes(blobCol, 0, b, 0, b.Length);
System.IO.FileStream fs = new System.IO.FileStream(destFilePath, System.IO.FileMode.Create,
System.IO.FileAccess.Write);
fs.Write(b, 0, b.Length);
fs.Close();
Console.WriteLine("Blob written to file successfully");
}
dr.Close();
success = true;
}
catch (SqlException ex)
{
success = false;
Console.WriteLine(ex.Message);
}
finally
{
connection.Close();
}
return success;
}
}
If I set up a breakpoint in method DeleteRows it does delete from the database. If I don't, nothing is deleted.
If you use this cmd.BeginExecuteNonQuery(),you have use EndExecuteNonquery().
Otherwise,it wont be deleted and it may create memory leak problem.
Best way is to use cmd.ExecuteNonQuery();
Does this also happen when you remove the Connection.Close? It might be that since you are removing async that the connection is closed before the operation can complete unlike when you have a breakpoint and the code execution waits.
Also instead of using SqlConnection the way you do, you might want to use:
using (sqlConnection con = new SqlConnection())
{
// your code
}
This way the connection is automatically closed when it is out of scope or when something goes wrong.
I agree with MahaSwetha that cmd.ExecuteNonQuery() would make your life easier. Also cmd.ExecuteNonQuery() returns an int telling you how much rows have changed. Can come in handy sometimes.