Problem: The memory leaks (accumulates) over time, and reaches 99% capacity eventually.
I have the following C# code that constantly pushes data into PostgreSQL DB using while loop. I'm really struggling because I'm not a C# programmer. My main language is Python. I've been trying to look up C# references to solve my issue, but failed cuz I simply don't understand lots of syntax. The C# code is written by someone else in my company, but he's not available now.
Here is the code:
var connString = "Host=x.x.x.x;Port=5432;Username=postgres;Password=password;Database=database";
using (var conn = new Npgsql.NpgsqlConnection(connString)){
conn.Open();
int ctr = 0;
// Insert some data
while(#tag.TerminateTimeScaleLoop == 100)
{
#Info.Trace("Pushed Data: PostGre A " + ctr.ToString());
using (var cmd = new Npgsql.NpgsqlCommand())
{
cmd.Connection = conn;
cmd.CommandText = "INSERT INTO TORQX VALUES (#r,#p)";
cmd.Parameters.AddWithValue("r", System.DateTime.Now.ToUniversalTime());
cmd.Parameters.AddWithValue("p", #Tag.RigData.Time.TORQX);
cmd.ExecuteNonQuery();
cmd.Parameters.Clear();
cmd.CommandText = "INSERT INTO BLKPOS VALUES (#s,#t)";
cmd.Parameters.AddWithValue("s", System.DateTime.Now.ToUniversalTime());
cmd.Parameters.AddWithValue("t", #Tag.RigData.Time.BLKPOS);
cmd.ExecuteNonQuery();
cmd.Parameters.Clear();
// #Info.Trace("Pushed Data: PostGre " + ctr.ToString());
}
ctr = ctr + 1;
}
#Info.Trace("Pushed Data: PostGre A Terminated");
The code successfully established a connection in the beginning, and uses only one connection the entire time. It correctly inserts data to DB. But after memory capacity reaches 99%, its not inserting very well. The source of issue I can think of is that this code is constantly creating new object, but does not clear that object after one iteration is done. Can anyone tell me where the source of problem is & provide possible solution to this?
++ please understand that I'm not a C# programmer... I'm not too familiar with the concept of memory handling. But I will try my best to understand
Here is something you can try. Notice the instantiation of the command and parameters happens outside of the loop, not on every iteration.
I am recycling the parameters. As a result I am using Add(), not AddWithValue() and you must fill in the database type for the second parameter and consider using precision and scale parameters too as appropriate.
This will only work if the two commands use the same parameter types. You might consider creating two commands, one for each query.
Know that variable names beginning with # makes me cringe as a c# developer....
var connString = "Host=x.x.x.x;Port=5432;Username=postgres;Password=password;Database=database";
using (var conn = new Npgsql.NpgsqlConnection(connString))
{
conn.Open();
int ctr = 0;
#Info.Trace("Pushed Data: PostGre A " + ctr.ToString());
using (var cmd = new Npgsql.NpgsqlCommand())
{
cmd.Connection = conn;
var par_1 = cmd.Parameters.Add("#p1", /*< appropriate datatype here >*/);
var par_2 = cmd.Parameters.Add("#p2", /*< appropriate datatype here >*/);
while(#tag.TerminateTimeScaleLoop == 100)
{
cmd.CommandText = "INSERT INTO TORQX VALUES (#p1,#p2)";
par_1.Value = System.DateTime.Now.ToUniversalTime());
par_2.Value = #Tag.RigData.Time.TORQX;
cmd.ExecuteNonQuery();
cmd.CommandText = "INSERT INTO BLKPOS VALUES (#p1,#p2)";
par_1.Value = System.DateTime.Now.ToUniversalTime());
par_2.Value = #Tag.RigData.Time.BLKPOS;
cmd.ExecuteNonQuery();
ctr = ctr + 1;
}
}
}
#Info.Trace("Pushed Data: PostGre A Terminated");
Related
I am trying to modify a table data using a SQL statement
foreach (Words words in Words_DB.Records)
{
string _IPAUS = words.IPAUS;
string _IPAUK = words.IPAUK;
query = "UPDATE Words SET IPAUK='" + _IPAUK + "',IPAUS='" + _IPAUS + "' WHERE WORD='" + words.Word + "'";
command.Parameters.Clear();
command.CommandText = query;
//command.Parameters.AddWithValue("#IPAUK", _IPAUK);
//command.Parameters.AddWithValue("#IPAUS", _IPAUS);
//command.Parameters.AddWithValue("#WORD", words.Word);
int a = command.ExecuteNonQuery();
}
A example of query is UPDATE Words SET IPAUK='ɑːd.vɑːk',IPAUS='ɑːrd.vɑːrk' WHERE WORD='aardvark'
The problem is when a read the database data I receive :
But, when I use the MySql Tools to execute the Query the result is right.
What I am doing wrong?
Regards
The question concatenates raw input to generate a SQL query which exposes to SQL injection and bugs like this one. If _IPAUK contained '; -- all the data in that column would be lost.
In this case it seems the code is trying to pass Unicode data using ASCII syntax, resulting in mangled data.
The solution to both SQL injection and conversion issues is to use parameterized queries. In a parameterized query, the actual parameter values never become part of the query itself. The server compiles the SQL query into an execution plan and executes that using the parameter values.
await using var connection = new MySqlConnection(connString);
await connection.OpenAsync();
// Insert some data
using (var cmd = new MySqlCommand())
{
cmd.Connection = connection;
cmd.CommandText = "UPDATE Words SET IPAUK=#IPAUK,IPAUS=#IPAUS WHERE WORD=#Word";
cmd.Parameters.AddWithValue("IPAUK", words.IPAUK);
cmd.Parameters.AddWithValue("IPAUS", words.IPAUS);
cmd.Parameters.AddWithValue("Word", words.Word);
await cmd.ExecuteNonQueryAsync();
}
The example uses the open source MySQLConnector ADO.NET Driver instead of Oracle's somewhat ... buggy driver.
The code can be simplified even more by using Dapper to construct the command, parameters and handle the connection automagically. Assuming words only has the IPAUK, IPAUS and Word properties, the code can be reduced to three lines :
var sql="UPDATE Words SET IPAUK=#IPAUK,IPAUS=#IPAUS WHERE WORD=#Word";
await using var connection = new MySqlConnection(connString);
await connection.ExecuteAsync(sql,words);
Dapper will construct a MySqlCommand, add parameters based on the properties of the parameter object (words), open the connection, execute the command and then close the connection
Thanks a lot for your helps.
This is my final code working properly.
string query = "UPDATE Words SET IPAUK=#IPAUK,IPAUS=#IPAUS WHERE WORD=#WORD";
var command = DatabaseConnection.MySql_Connection.CreateCommand();
try
{
foreach (Words words in Words_DB.Records)
{
MySqlParameter IPAUSp = new MySqlParameter("#IPAUS", MySqlDbType.VarChar, 60);
MySqlParameter IPAUKp = new MySqlParameter("#IPAUK", MySqlDbType.VarChar, 60);
MySqlParameter WORD = new MySqlParameter("#WORD", MySqlDbType.VarChar, 50);
command.Parameters.Clear();
command.CommandText = query;
command.Parameters.AddWithValue(IPAUKp.ToString(), words.IPAUK);
command.Parameters.AddWithValue(IPAUSp.ToString(), words.IPAUS);
command.Parameters.AddWithValue(WORD.ToString(), words.Word);
int a = command.ExecuteNonQuery();
}
}
Try it like this:
command.CommandText = "UPDATE Words SET IPAUK= #IPAUK, IPAUS= #IPAUS WHERE WORD= #Word;";
// Match these to the column type and length in the DB
command.Parameters.Add("#IPAUK", MySQlDbType.VarChar, 30);
command.Parameters.Add("#IPAUS", MySQlDbType.VarChar, 30);
command.Parameters.Add("#Word", MySQlDbType.VarChar, 30);
foreach (Words words in Words_DB.Records)
{
command.Parameters["#IPAUK"].Value = words.IPAUK;
command.Parameters["#IPAUS"].Value = words.IPAUS;
command.Parameters["#Word"].Value = words.Word;
command.ExecuteNonQuery();
}
Notice how the above minimizes the work done in the loop, which should improve performance, while also fixing the HUGE GAPING SECURITY ISSUE in the question from using string concatenation to build the query.
Separately, I have the impression Words_DB.Records is the result of a prior query. It's highly likely you could eliminate this entire section completely by updating the prior query to also do the update in one operation on the server. Not only would that greatly reduce your code, it will likely improve performance here by multiple orders of magnitude.
I would like to do the same thing as oracle sqldeveloper do with parametered queries (but from c#, not java).
Let's say there's an arbitrary, user supplied query, eg
select * from dual where 1 = :parameter
My task is to parse safely similar strings, identify the parameters, ask them from user and execute the query.
Which is the right / safe approach? I guess, there's some oracle client api to do this. Or is the right way using some pl/sql stuff (eg. from DBMS_SQL)? I couldn't find such a thing yet...
Update / clarification: see the example code below:
// user enters the query string with parameters somehow:
string sql = AskUserForSelectString();
// now the value of sql is:
// "select column0 from tablename where column1 = :param1 and column2 = :param2 ;"
// this is my original question: HOW TO DO THIS?
List<string> param_names = OracleParseQueryAndGiveMyParameters(sql);
// param_names is now a list of ":param1",":param2"
// ask user again for parameter values:
var param_values = new List<string>();
foreach (var param_name in param_names)
{
string param_value = AskUserForParameterValue(param_name);
param_values.Add(param_value);
}
// give the parameter values for the query in safe way:
using (var cmd = new SqlCommand(sql, myDbConnection))
{
for (int i=0; i< param_names.Count; i++)
cmd.Parameters.AddWithValue(param_names[i], param_values[i]);
var result = cmd.ExecuteReader();
// process result...
}
The key point is that I don't know the parameters in advance. This is exactly what SqlDeveloper can do.
(That isn't an issue if EF expects # before the parameter name instead of colon, that can be worked out easily.)
You can do it like this:
var sql = "INSERT INTO myTable (myField1, myField2) " +
"VALUES (#someValue, #someOtherValue);";
using (var cmd = new SqlCommand(sql, myDbConnection))
{
cmd.Parameters.AddWithValue("#someValue", someVariable);
cmd.Parameters.AddWithValue("#someOtherValue", someTextBox.Text);
cmd.ExecuteNonQuery();
}
What you absolutly must NOT do is:
var sql = "INSERT INTO myTable (myField1, myField2) " +
"VALUES ('" + someVariable + "', '" + someTextBox.Text + "');";
var cmd = new SqlCommand(sql, myDbConnection);
cmd.ExecuteNonQuery();
The problem with the second example is that it opens your code to an SQL Injection attack.
One (hacky but accurate?!) way with the original ":parameter" bind variable syntax is to call out to C and use Oracle OCI functions to do the parsing for you.
Prepare the statement with OCIStmtPrepare2() and then call
OCIStmtGetBindInfo() to get the variable names.
I tried to update a paragraph from mysql table,but i got error like this
"You have an error in your SQL syntax; check the manual that
corresponds to your MySQL server version for the right syntax to use
near 's first-ever super-villainess."
My mysql Query
cmd.CommandText = "UPDATE `moviemaster` SET `Runtime`='" + runtime + "',`DateMasterId`='" + dateid + "',`Trailer`='" + trailer + "',`Synopsis`='" + synopsis + "' WHERE `MovieMasterId`='" + movieid + "'";
I got error in 'synopsis',it's a big data containing a large paragraph.If i romove 'Synopsis' section from the query,everything working fine.What exactly the problem.How can i resolve this?
#SonerGönül:Ok,fine.. then please show me an example of parameterised
query
Sure. I also wanna add a few best practice as well.
Use using statement to dispose your connection and command automatically.
You don't need to escape every column with `` characters. You should only escape if they are reserved keywords for your db provider. Of course, at the end, changing them to non-reserved words is better.
Do not use AddWithValue method. It may generate upexpected and surprising result sometimes. Use Add method overload to specify your parameter type and it's size.
using (var con = new SqlConnection(conString))
using(var cmd = con.CreateCommand())
{
cmd.CommandText = #"UPDATE moviemaster
SET Runtime = #runtime, DateMasterId = #dateid, Trailer = #trailer, Synopsis = #synopsis
WHERE MovieMasterId = #movieid";
cmd.Parameters.Add("#runtime", MySqlDbType.VarChar).Value = runtime; ;
cmd.Parameters.Add("#dateid", MySqlDbType.VarChar).Value = dateid;
cmd.Parameters.Add("#trailer", MySqlDbType.VarChar).Value = trailer;
cmd.Parameters.Add("#synopsis", MySqlDbType.VarChar).Value = synopsis;
cmd.Parameters.Add("#movieid", MySqlDbType.VarChar).Value = movieid;
// I assumed your column types are VarChar.
con.Open();
cmd.ExecuteNonQuery();
}
Please avoid using inline query. Your database can be subjected to SQL Injection. See this example, on what can be done using SQL Injection.
And use paramterized query instead. Here is the example taken from here. This way, even if your string has special characters, it will not break and let you insert/update/select based on parameters.
private String readCommand = "SELECT LEVEL FROM USERS WHERE VAL_1 = #param_val_1 AND VAL_2 = #param_val_2;";
public bool read(string id)
{
level = -1;
MySqlCommand m = new MySqlCommand(readCommand);
m.Parameters.AddWithValue("#param_val_1", val1);
m.Parameters.AddWithValue("#param_val_2", val2);
level = Convert.ToInt32(m.ExecuteScalar());
return true;
}
and finally, your query will become
cmd.CommandText = "UPDATE `moviemaster` SET `Runtime`= #param1,`DateMasterId`= #dateid, `Trailer`= #trailer,`Synopsis`= #synopsis WHERE `MovieMasterId`= #movieid";
cmd.Parameters.AddWithValue("#param1", runtime);
cmd.Parameters.AddWithValue("#dateid", dateid);
cmd.Parameters.AddWithValue("#trailer", trailer);
cmd.Parameters.AddWithValue("#synopsis", synopsis);
cmd.Parameters.AddWithValue("#movieid", movieid);
Ok, so here's the problem I have to solve. I need to write a method in C# that will modify a table in SQL Server 2008. The table could potentially contain millions of records. The modifications include altering the table by adding a new column and then calculating and setting the value of the new field for every row in the table.
Adding the column is not a problem. It's setting the values efficiently that is the issue. I don't want to read in the whole table into a DataTable and then update and commit for obvious reasons. I'm thinking that I would like to use a cursor to iterate over the rows in the table and update them one by one. I haven't done a whole lot of ADO.NET development, but it is my understanding that only read-only server side (firehose) cursors are supported.
So what is the correct way to go about doing something like this (preferably with some sample code in C#)? Stored procedures or other such modifications to the DB are not allowed.
jpgoody,
Here is an example to chew on using the NerdDinner database and some SQLConnection, SQLCommand, and SQLDataReader objects. It adds one day to each of the Event Dates in the Dinners table.
using System;
using System.Data.SqlClient;
namespace NerdDinner
{
public class Class1
{
public void Execute()
{
SqlConnection readerConnection = new SqlConnection(Properties.Settings.Default.ConnectionString);
readerConnection.Open();
SqlCommand cmd = new SqlCommand("SELECT DinnerID, EventDate FROM Dinners", readerConnection);
SqlDataReader reader = cmd.ExecuteReader();
SqlConnection writerConnection = new SqlConnection(Properties.Settings.Default.ConnectionString);
writerConnection.Open();
SqlCommand writerCommand = new SqlCommand("", writerConnection);
while (reader.Read())
{
int DinnerID = reader.GetInt32(0);
DateTime EventDate = reader.GetDateTime(1);
writerCommand.CommandText = "UPDATE Dinners SET EventDate = '" + EventDate.AddDays(1).ToString() + "' WHERE DinnerID = " + DinnerID.ToString();
writerCommand.ExecuteNonQuery();
}
}
}
}
Your problem looks like something that you should be solving using T-SQL and not C#, unless there is some business rule that you are picking up dynamically and calculating the column values T-SQL should be the way to go. Just write a stored procedure or just open up Management studio and write the code to make your changes.
If this does not help then please elaborate on what exactly you want to do to the table, then we can help you figure out if this can be done via T-SQL or not.
[EDIT] you can do something like this
string sql = " USE " + paramDbName;
sql+= " ALTER TABLE XYZ ADD COLUMN " + param1 + " datatype etc, then put semicolon to separate the commands as well"
sql+= " UPDATE XYZ SET Columnx = " + some logic here
cmd.CommandText = sql;
cmd.ExecuteNonQuery();
get this executed on the required instance of Sql Server 2008.
If you have too many lines of text then use StringBuilder.
Here's a suggestion:
You can read data using a DataReader , create a update command for current row and add it to a list of commands.Then run update commands in a transaction.
something like this:
var commands=new List<SqlCommand>();
while(dr.Read())
{
var cmd=new SqlCommand();
cmd.CommandText="Add your command text here";
commands.Add(cmd);
}
using(var cnn=new SqlConnection("Connection String"))
{
IDbTransaction transaction;
try
{
cnn.Open();
transaction=cnn.BeginTransaction();
foreach(var cmd in commands)
{
cmd.Transaction=transaction;
cmd.ExecuteNonQuery();
cmd.Dispose();
}
transaction.Commit();
}
catch(SqlException)
{
if(transaction!=null)
transaction.Rollback();
throw;
}
}
Replaces Question: Update multiple rows into SQL table
Here's a Code Snippet to update an exam results set.
DB structure is as given, but I can submit Stored Procedures for inclusion (Which are a pain to modify, so I save that until the end.)
The question: Is there a better way using SQL server v 2005.,net 2.0 ?
string update = #"UPDATE dbo.STUDENTAnswers
SET ANSWER=#answer
WHERE StudentID =#ID and QuestionNum =#qnum";
SqlCommand updateCommand = new SqlCommand( update, conn );
conn.Open();
string uid = Session["uid"].ToString();
for (int i= tempStart; i <= tempEnd; i++)
{
updateCommand.Parameters.Clear();
updateCommand.Parameters.AddWithValue("#ID",uid);
updateCommand.Parameters.AddWithValue("#qnum",i);
updateCommand.Parameters.AddWithValue("#answer", Request.Form[i.ToString()]);
try
{
updateCommand.ExecuteNonQuery();
}
catch { }
}
A few things stand out:
You don't show where the SqlConnection is instantiated, so it's not clear that you're disposing it properly.
You shouldn't be swallowing exceptions in the loop - better to handle them in a top level exception handler.
You're instantiating new parameters on each iteration through the loop - you could just reuse the parameters.
Putting this together it could look something like the following (if you don't want to use a transaction, i.e. don't care if some but not all updates succeed):
using (SqlConnection conn = new SqlConnection(connectionString))
{
conn.Open();
using (SqlCommand updateCommand = new SqlCommand(update, conn))
{
string uid = Session["uid"].ToString();
updateCommand.Parameters.AddWithValue("#ID", uid);
updateCommand.Parameters.AddWithValue("#qnum", i);
updateCommand.Parameters.Add("#answer", System.Data.SqlDbType.VarChar);
for (int i = tempStart; i <= tempEnd; i++)
{
updateCommand.Parameters["#answer"] = Request.Form[i.ToString()];
updateCommand.ExecuteNonQuery();
}
}
}
Or to use a transaction to ensure all or nothing:
using (SqlConnection conn = new SqlConnection(connectionString))
{
conn.Open();
using (SqlTransaction transaction = conn.BeginTransaction())
{
using (SqlCommand updateCommand = new SqlCommand(update, conn, transaction))
{
string uid = Session["uid"].ToString();
updateCommand.Parameters.AddWithValue("#ID", uid);
updateCommand.Parameters.AddWithValue("#qnum", i);
updateCommand.Parameters.Add("#answer", System.Data.SqlDbType.VarChar);
for (int i = tempStart; i <= tempEnd; i++)
{
updateCommand.Parameters["#answer"] = Request.Form[i.ToString()];
updateCommand.ExecuteNonQuery();
}
transaction.Commit();
}
} // Transaction will be disposed and rolled back here if an exception is thrown
}
Finally, another problem is that you are mixing UI code (e.g. Request.Form) with data access code. It would be more modular and testable to separate these - e.g. by splitting your application into UI, Business Logic and Data Access layers.
For 30 updates I think you're on the right track, although the comment about the need for a using around updateCommand is correct.
We've found the best performing way to do bulk updates (>100 rows) is via the SqlBulkCopy class to a temporary table followed by a stored procedure call to populate the live table.
An issue I see is when you are opening your connection.
I would at least before every update call the open and then close the connection after the update.
If your loop takes time to execute you will have your connection open for a long time.
It is a good rule to never open your command until you need it.
You can bulk insert using OpenXML. Create an xml document containing all your questions and answers and use that to insert the values.
Edit: If you stick with your current solution, I would at least wrap your SqlConnection and SqlCommand in a using block to make sure they get disposed.
emit a single update that goes against a values table:
UPDATE s SET ANSWER=a FROM dbo.STUDENTAnswers s JOIN (
SELECT 1 as q, 'answer1' as a
UNION ALL SELECT 2, 'answer2' -- etc...
) x ON s.QuestionNum=x.q AND StudentID=#ID
so you just put this together like this:
using(SqlCommand updateCommand = new SqlCommand()) {
updateCommand.CommandType = CommandType.Text;
updateCommand.Connection = conn;
if (cn.State != ConnectionState.Open) conn.Open();
StringBuilder sb = new StringBuilder("UPDATE s SET ANSWER=a FROM dbo.STUDENTAnswers s JOIN (");
string fmt = "SELECT {0} as q, #A{0} as a";
for(int i=tempStart; i<tempEnd; i++) {
sb.AppendFormat(fmt, i);
fmt=" UNION ALL SELECT {0},#A{0}";
updateCommand.Parameters.AddWithValue("#A"+i.ToString(), Request.Form[i.ToString()]);
}
sb.Append(") x ON s.QuestionNum=x.q AND StudentID=#ID");
updateCommand.CommandText = sb.ToString();
updateCommand.Parameters.AddWithValue("#ID", uid);
updateCommand.ExecuteNonQuery();
}
This has the advantages of being an all other nothing operation (like if you'd wrapped several updates in a transaction) and will run faster since:
The table and associated indexes are looked at/updated once
You only pay for the latency between your application and the database server once, rather than on each update