I need to write a huge amount of data to a postgredb. The PG connection_limit is set to 200. And I need to write ~5000 values per second. If I open a new connection for every query, I get too many clients errors. But if I use only one connection, its to slow to insert ~5000 values.
What is the correct method to manage the connection in this case?
At the moment I open and close the connection for every query
public static async Task WriteValueAsync(Item Item)
{
try
{
using (NpgsqlConnection con = new NpgsqlConnection(constring))
{
con.Open();
String sql = "INSERT INTO \"" + Item.ID + "\" (...) VALUES (...)";
NpgsqlCommand cmd = con.CreateCommand();
cmd.CommandText = sql;
await cmd.ExecuteNonQueryAsync();
cmd.Dispose();
con.Close();
}
}
catch (Exception e)
{
Debug.WriteLine("DB WriteValueAsync: " + e.ToString());
}
}
Related
I want to execute an stored procedure located in a linked server database. Currently, I'm using this in SSMS:
INSERT INTO myTable
EXEC [LINKEDSERVER\LINKED].[Data_Base_Name].[Store].usp_GetInfo 1, 1, NULL, 'H'
This will insert into my Local DB the result data from the Stored procedure located in LINKEDSERVER\LINKED.
I want to be able to do this with a command from C#, is there a proper way to do it?
Thanks!
You could execute SP from DataContext:
using (DataContext ctx = DataContext())
{
int result = ctx.SP_ProcedureName("1", "2", "3");
}
But first you have to add it to DataContext Diagram from your database as you add tables but from "Stored Procedures" folder.
that is more defensive and neat solution. but if you prefer to use raw command line at least use parameterized query for it like this example :
string sqlText = "SELECT columnName FROM Test_Attachments WHERE Project_Id =#PID1 AND [Directory] = #Directory";
SqlCommand myCommand = new SqlCommand(sqlText, SqlConnection);
myCommand.Parameters.AddWithValue("#PID1", 12);
myCommand.Parameters.AddwithValue("#Directory", "testPath");
It is way for avoiding SQL injection to your code.
Also you could use finally block for close connection :
finally
{
command.Connection.Close();
}
Thank you guys for the help. Oleg thanks for your suggestion as well. What I did was this:
qSQL = "INSERT INTO " + tableName + " EXEC [LINKEDSERVER\\LINKED].[Data_Base_Name]." + spName;
using (SqlConnection _connection = new SqlConnection(connectionString))
{
try
{
command = new SqlCommand();
command.Connection = _connection;
command.Connection.Open();
command.CommandText = _qSQL;
command.CommandTimeout = 300; //Because it takes long
SqlTransaction transaction;
transaction = connection.BeginTransaction();
try
{
command.Transaction = _transaction;
command.ExecuteNonQuery();
transaction.Commit();
Debug.WriteLine("Done");
}
catch (SqlException e)
{
Debug.WriteLine("Exception [{0}:{1}]", e.Number, e.Message);
transaction.Rollback();
}
//close connection
command.Connection.Close();
}
catch (SqlException e)
{
command.Connection.Close();
Debug.WriteLine("exception error number: " + e.Number + ": " + e.Message);
}
}
}
If you have any suggestions to improve this let me know.
I have a c# console app client and I am trying to connect to mysql database.
I have a main method called select.
This method return n of rows.
I am looping through results, and for each one I am calling another method call GetProductAttributes and after I get the results I am calling another method called UpdateProductAttributesJsonField
Here is my code :
public void Select(int shopId)
{
using(MySqlCommand cmd = new MySqlCommand("GetProducts", new MySqlConnection(connection.ConnectionString)))
{
cmd.Parameters.Add(new MySqlParameter("#last_modified_date", ""));
cmd.Parameters.Add(new MySqlParameter("#ShopId", shopId));
cmd.CommandType = CommandType.StoredProcedure;
cmd.Connection.Open();
MySqlDataReader dataReader = cmd.ExecuteReader(CommandBehavior.CloseConnection);
while (dataReader.Read())
{
var id = int.Parse(dataReader["Id"].ToString());
var attributes = GetProductAttributes(id);
var json = JsonConvert.SerializeObject(attributes.ToArray());
UpdateProductAttributesJsonField(id, json);
}
//dataReader.Close();
}
}
public Dictionary<string, string> GetProductAttributes(int Id)
{
var result = new Dictionary<string, string>();
using(MySqlCommand cmd = new MySqlCommand("GetProductAttributes", new MySqlConnection(connection.ConnectionString)))
{
cmd.Parameters.Add(new MySqlParameter("#Id", Id));
cmd.CommandType = CommandType.StoredProcedure;
cmd.Connection.Open();
//CommandBehavior.CloseConnection
MySqlDataReader dataReader = cmd.ExecuteReader(CommandBehavior.CloseConnection);
while (dataReader.Read())
{
result.Add(dataReader["name"].ToString(), dataReader["value"].ToString());
}
//dataReader.Close();
}
return result;
}
public void UpdateProductAttributesJsonField(int productId, string json)
{
using( MySqlCommand cmd = new MySqlCommand("UpdateArticleAttributes", new MySqlConnection(connection.ConnectionString)))
{
cmd.Parameters.Add(new MySqlParameter("#articleId", productId));
cmd.Parameters.Add(new MySqlParameter("#json", json));
cmd.CommandType = CommandType.StoredProcedure;
cmd.Connection.Open();
cmd.ExecuteNonQuery();
}
//cmd.EndExecuteNonQuery(CommandBehavior.CloseConnection); // cmd.ExecuteReader(CommandBehavior.CloseConnection);
}
Here is my initialize method
private void Initialize()
{
server = ConfigurationManager.AppSettings["ShopwareDBServer"];
database = ConfigurationManager.AppSettings["DatabaseName"];
uid = ConfigurationManager.AppSettings["DatabaseUser"];
password = ConfigurationManager.AppSettings["DatabasePassword"];
port = "3306";
string connectionString;
connectionString = "SERVER=" + server + ";" + "Port=" + port + ";" + "DATABASE=" +
database + ";" + "UID=" + uid + ";" + "PASSWORD=" + password + ";" + "Convert Zero Datetime=True";
connection = new MySqlConnection(connectionString);
}
I am getting this error :
{"error connecting: Timeout expired. The timeout period elapsed prior
to obtaining a connection from the pool. This may have occurred
because all pooled connections were in use and max pool size was
reached."}
Since you are using Data Reader in your code, you need to close the connection before opening a new one.
It may seem to be a suggestion rather than answering the question, but let me show you another way to do the same thing.
You want to call 3 procedures then why you are not combining these three procedures and create a new one in the database and call that procedure. I think, this way you need only one connection and your problem will be solved.
By the way, this is not possible to call three procedures on the same connection that uses "Data Reader".
You can check whether the connection state is opened or closed; this can help you decrease your maximum connection pool size.
You can write the code like this:
If(Connection.State == Closed) {
Connection.open
}
This is because you use maximum connection pool size may be in your code you no where write connection.close() so connection size is increase with time so just make sure you close connections in finally block.
First restart MySQL so it will close all the connection and change your code accordingly. And this issue is not due to calling 3 stored procedures in one connection and if you want to do it is done in multiple way.
Either you call one stored procedure and call other procedures from the first one.
You also call stored procedures step by step from c# code as well.
I have a problem running a simple insert query from my C# app towards a Postrges DB.
This is the function that builds the query:
string push = "INSERT INTO \"Tasks\"( \"TName\", \"Desc\", \"TType\", \"DCreated\", \"DEnd\") VALUES (\'" + this.Name + "\',\'" + this.Descr + "\'," + this.Type + ",\'" + this.StartDate + "\', \'" + this.EndDate + "\');";
GenericDbClass.ExecutePush(push);
And this is the string that gets passed to the DB:
INSERT INTO "Tasks"( "TName", "Desc", "TType", "DCreated", "DEnd") VALUES ('dddddd','dddddddddddd',3,'13.04.2015 17:00', '24.04.2015 16:42');
If I copy the string and run it inside pgAdmin it works right of the bat, but from here it doesn't do anything - no exceptions thrown,no errors, nothing in the logs, as if it just doesn't reach the server.
In addition here is the push method:
public static void ExecutePush(string sql)
{
try
{
NpgsqlConnection conn = new NpgsqlConnection(GenericDbClass.GetDbConnString());
conn.Open();
NpgsqlDataAdapter da = new NpgsqlDataAdapter(sql, conn);
conn.Close();
}
catch (Exception msg)
{
MessageBox.Show(msg.ToString());
throw;
}
}
Edit: This is the working solution I found
public static void ExecutePush(string sql)
{
try
{
NpgsqlConnection conn = new NpgsqlConnection(GenericDbClass.GetDbConnString());
conn.Open();
NpgsqlCommand nc = new NpgsqlCommand(sql, conn);
nc.ExecuteNonQuery();
conn.Close();
}
catch (Exception msg)
{
MessageBox.Show(msg.ToString());
throw;
}
}
NpgsqlDataAdapter da = new NpgsqlDataAdapter(sql, conn);
Means "Please create a data-adaptor that uses the INSERT SQL passed as sql to do a selection". That doesn't make much sense, and then you don't do anything with it, anyway.
conn.CreateCommand(sql).ExecuteNonQuery();
Seems more like what you want.
I'm using MS Sql database in my application and when I do SELECT operation from database it stucks for a few minutes.
And it happens a few times per day. All other SELECTS take several seconds only.
Also I noticed if I close app while SELECT operation is in progress. It will not work at all on any next app starts UNTIL I restart database engine...
Why could it be?
Here is the code snippet:
using (SqlConnection myConnection = new SqlConnection(connString))
{
myConnection.Open();
SqlCommand command = myConnection.CreateCommand();
SqlTransaction transaction;
transaction = myConnection.BeginTransaction("LockTransaction");
command.Connection = myConnection;
command.Transaction = transaction;
try
{
int recordsAtOnce = maxToLock;
command.CommandText =
"SELECT TOP 1000 id, productName from Parts WHERE part_used is NULL;";
List<string> idList = new List<string>();
SqlDataReader myReader = command.ExecuteReader(CommandBehavior.Default);
while (myReader.Read())
{
string id = myReader.GetString(1);
string name = myReader.GetInt32(0).ToString();
idList.Add(id);
}
myReader.Close();
string idsStr = "";
for(int i = 0; i < idList.Count; i++)
{
if (i != 0)
{
idsStr += ", ";
}
idsStr += idList[i];
}
// lock record
command.CommandText = "UPDATE Parts SET part_used=\'rt\' WHERE id in (" + idsStr + ")";
command.Parameters.Clear();
command.CommandType = CommandType.Text;
command.ExecuteNonQuery();
transaction.Commit();
}
catch (Exception ex)
{
transaction.Rollback();
}
I think your reader values are being assigned to the wrong variables.
Try changing:
string id = myReader.GetString(1);
string name = myReader.GetInt32(0).ToString();
idList.Add(id);
To:
string id = myReader.GetInt32(0);
string name = myReader.GetString(1).ToString();
idList.Add(id);
This is most probably because of a lock. If another transaction is writing into the Parts table, your select needs to wait.
How big is the Parts table? That could cause performance problems. Or it could be another transaction locking before you.
Other things:
Call dispose on your connection, command and reader when done with them via using
Parameterize the second query rather than string concating. It's more secure and significantly more performant because of how Sql Server works internally.
minor: If you're looking at thousands of items, use StringBuilder instead of concatenating.
I'm sending a command to the database and it is returning that 1 rows are affected, but when i look inside the database, there are no records. I am receiving no errors. I checked to make sure the string was building correctly and it is. Any ideas? I'm not using parameterized queries here, I know. I will later. Here is the code from the database layer:
public int InsertStartTime(certificate cert, DateTime startTime, string lineNumber)
{
string sql = "INSERT INTO checkLog(userID,lineNumber,startTime) VALUES(" +
cert.userID + ", '" + lineNumber + "', '" + startTime + "');";
int result = 0;
try
{
conn.Open();
comm.CommandText = sql;
result = comm.ExecuteNonQuery();
}
catch (Exception ex)
{
MessageBox.Show(ex.Message);
}
finally
{
conn.Close();
}
MessageBox.Show(result.ToString() + " rows affected");
return result;
}
Using an access 2000 db file:
string connStr = #"Provider=Microsoft.Jet.OLEDB.4.0;Data Source=|DataDirectory|\assets\users.mdb;Persist Security Info=True";
Right click your database file in VS and look at the properties. Is it set to "Copy Always"? By default, visual studio will make a copy of your database for debugging and any changes will be made only to this copy and will not be reflected in the original. You can set it to copy "Never" if you want to work on the "real" database file even in debug mode.
Depending on the database / data provider you are using, your SQL command may not be executing in auto-commit mode.
Try committing your transaction explicitly. Something like this:
conn.Open();
using (var tran = conn.BeginTransaction()) {
comm.Transaction = tran; // Possibly redundant, depending on database.
comm.CommandText = sql;
result = comm.ExecuteNonQuery();
tran.Commit();
}
Nobody else pointed this out so I will. PLEASE DO NOT USE SQL this way. Use parameters. You leave yourself wide open to sql attacks otherwise.
string sql = "INSERT INTO checkLog(userID,lineNumber,startTime) VALUES(#ID, #line, #starttime);
try
{
conn.Open();
comm.CommandText = sql;
comm.Parameters.Add("ID").Value = cert.userID;
comm.Parameters.Add("line").Value = lineNumber ;
comm.Parameters.Add("starttime").Value = startTime ;
result = comm.ExecuteNonQuery();
}
catch (Exception ex)
{
MessageBox.Show(ex.Message);
}