reader.Read() is sometimes false when looping and not all table rows are read. During this skipped state reader.Read() never becomes true. Waiting for it before iterating locks it up, of course. Is there a SQL connection status that will help mutex the loop? This happens 1 - 10% of the time during the loop.
** Answered: I was referencing the wrong sql column in my query. Adding reader.NextResult(); helped as well.
using System;
using System.Diagnostics;
using Microsoft.Data.SqlClient;
public string StringTableRows(string conn_str, int row_count)
{
string s = "";
Query query = new Query();
for (int i = 1; i < row_count;)
{
// **Fix: I was referencing the wrong column.
string dyn_query = query.RowQuery(ids[i]);
try
{
SqlConnection conn = new SqlConnection(conn_str);
using (conn)
{
SqlCommand cmd = new SqlCommand(dyn_query, conn);
using (cmd)
{
cmd.Connection.Open();
SqlDataReader reader = comm.ExecuteReader();
using (reader)
{
Debug.WriteLine("Outside: " + i);
while (reader.Read())
{
for (int j = 0; j < reader.FieldCount; j++)
{
s = s + reader.GetString(0);
Debug.WriteLine("Inside: " + i);
}
}
// **Fix: This is the fix per answer.
reader.NextResult();
}
}
}
}
catch (Exception ex)
{
Debug.WriteLine("Failed query: " + dyn_query);
Debug.WriteLine("Execute Exception: " + ex);
}
i++;
}
s = s.Insert(0,"<Tables>");
s = s + "</Tables>";
return s;
}
reader. Read() is sometimes false when looping and not all table rows are read
That will only happen if the additional rows are in a separate resultset because the CommandText has multiple SELECT statements or calls a stored procedure with multiple SELECT statements. In that case you have to call SqlDataReader.NextResult() before you call Read() again.
Related
private void GetTextFile()
{
NpgsqlConnection npgsqlConnection = new NpgsqlConnection();
npgsqlConnection.ConnectionString = "Server=127.0.0.1;Port=5432;User
Id=postgres;Password=rutuparna;Database=Employee";
npgsqlConnection.Open();
NpgsqlCommand command = new NpgsqlCommand("Select * from employee_details", npgsqlConnection);
NpgsqlDataReader dataReader = command.ExecuteReader();
using (System.IO.StreamWriter writer = new System.IO.StreamWriter(#"D:\Rutu\txtfile.txt", false, Encoding.UTF8))
{
while (dataReader.Read())
{
writer.WriteLine(dataReader[0] + "; " + dataReader[1] + ";" + dataReader[2] + ";" + dataReader[3]);
}
}
MessageBox.Show("Data fetched Properly");
}
Here I have done how to convert data into a text file.
Can somebody please give me the code of how to export Text File data to the SQL database using C# ?
Why not something like this
using (SqlConnection con = new SqlConnection(#"your connection string"))
{
con.Open();
using(StreamReader file = new StreamReader(#"D:\Rutu\txtfile.txt"))
{
while((line = file.ReadLine()) != null)
{
string[] fields = line.Split(',');
SqlCommand cmd = new SqlCommand("INSERT INTO employee_details(column1, column2, column3,column4) VALUES (#value1, #value2, #value3, #value4)", con);
cmd.Parameters.AddWithValue("#value1", fields[0].ToString());
cmd.Parameters.AddWithValue("#value2", fields[1].ToString());
cmd.Parameters.AddWithValue("#value3", fields[2].ToString());
cmd.Parameters.AddWithValue("#value4", fields[3].ToString());
cmd.ExecuteNonQuery();
}
}
}
I don't know the names of your columns, so you'll need to replace them.
If speed is a consideration (and IMHO it always should be) and you are using PostgreSQL (which you seem to be), then you should take a look at the COPY function. Single inserts are always slower than a bulk operation.
It is relatively easy to wrap the COPY function within c#. The following is a cut down version of a method, to illustrate the point. My method loops through the rows of a DataTable, but it is easy to adapt for a file situation (ReadLine() etc).
using (var pgConn = new NpgsqlConnection(myPgConnStr))
{
using (var writer = pgConn.BeginBinaryImport("COPY " + destinationTableName + " (" + commaSepFieldNames + ") FROM STDIN (FORMAT BINARY)"))
{
//Loop through data
for (int i = 0; i < endNo; i++)
{
writer.StartRow();
//inner loop through fields
for (int j = 0; j < fieldNo; j++)
{
//test for null
if (true)
{
writer.WriteNull();
}
else
{
//Write data using column types
writer.Write(value, type);
}
}
}
writer.Complete();
}
}
WriteNull() is a special method for correctly adding NULL to the stream, otherwise you use Write<T>(). This latter has three overloads, just the value, value with type enumeration, or value with type name. I prefer to use the enumeration. commaSepFieldNames is a comma separated list of the field names. Other variables should (I hope) be self-explanatory.
I'm trying to make sure that both my insert and delete below work completely or not at all. I have my connection object outside of my transaction scope which I believe is correct by not 100% sure.
I do know that this code is not working as I intent. After the first part (the insert runs) and then I abort by terminating on a break point, the rows are indeed inserted even though I never called scope.complete.
Please point out the flaw in my thinking and logic here.
sqlConnection.Open();
int numFound = 1;
int max = 99;
int iteration = 0;
while (iteration < max && numFound > 0)
{
iteration++;
var ids = new List<int>();
using (var sqlCommand0 = new SqlCommand(sql0, sqlConnection))
{
using (SqlDataReader reader1 = sqlCommand0.ExecuteReader())
{
while (reader1.Read())
{
ids.Add(reader1.GetInt32(0));
}
}
}
numFound = ids.Count;
if (numFound > 0)
{
using (var scope = new TransactionScope())
{
string whereClause = $"WHERE Id IN ({string.Join(",", ids)})";
string sql1 = string.Format(sqlTemplate1, whereClause);
using (var sqlCommand1 = new SqlCommand(sql1, sqlConnection))
{
sqlCommand1.ExecuteNonQuery();
}
// BREAK POINT HERE - ABORTED PROGRAM AND sql1 had been committed.
var sql2 = "DELETE FROM SendGridEventRaw " + whereClause;
using (var sqlCommand2 = new SqlCommand(sql2, sqlConnection))
{
sqlCommand2.ExecuteNonQuery();
}
scope.Complete();
total += numFound;
Console.WriteLine("deleted: " + whereClause);
}
}
}
}
I think it's because you open your connection before starting your transaction. You could try to fix your issue by first starting your transaction and then opening your connection.
Just from what I am seeing and from what I am assuming is what you intend to happen is this:
If your first query gets some records, then the next query executes, hence the statement:
if (numFound > 0)
If that is the case, and where you put your breakpoint is true, of course the insert statement will fire. Reason is:
using (var sqlCommand1 = new SqlCommand(sql1, sqlConnection))
{
sqlCommand1.ExecuteNonQuery();
}
is within that if statement. You're saying "if there are any rows, execute the insert query."
If you're trying to actually get the scope object to do the query, then you're going to have to have all of the query construction happening within the object and then having scope.complete() doing the execution.
For example:
//In TransactionScope class
public string Complete(var ids, int numFound, SqlConnection sqlConnection, string sqlTemplate1)
{
string whereClause = $"WHERE Id IN ({string.Join(",", ids)})";
string sql1 = string.Format(sqlTemplate1, whereClause);
using (var sqlCommand1 = new SqlCommand(sql1, sqlConnection))
{
sqlCommand1.ExecuteNonQuery();
}
var sql2 = "DELETE FROM SendGridEventRaw " + whereClause;
using (var sqlCommand2 = new SqlCommand(sql2, sqlConnection))
{
sqlCommand2.ExecuteNonQuery();
}
return whereClause;
}
//in your Main class
if (num > 0)
{
string whereClause = scope.Complete(ids, numFound, sqlConnection, sqlTemplate1);
Console.WriteLine("deleted" + whereClause"." );
}
I am of course just going off of the assumptions I stated above. If I am incorrect, please let me know.
Hope it helps.
I have table that contains 350632 records currently. I have recently added a new column to the table which I am trying to populate using this code in C#:
List<int> listOfInts = new List<int>();
dbConnect.Open();
int counter = 1;
string toExecute = "select * from tempwords";
string insertQuery = "update tempwords set rownum=#toInsert";
using (SQLiteTransaction transaction = dbConnect.BeginTransaction())
{
using (SQLiteCommand newCommand = new SQLiteCommand(toExecute, dbConnect))
{
using (SQLiteDataReader reader = newCommand.ExecuteReader())
{
while (reader.Read())
{
listOfInts.Add(counter);
counter++;
}
}
}
transaction.Commit();
dbConnect.Dispose();
}
Console.WriteLine(listOfInts.Count.ToString());
dbConnect.Open();
int iterator = 0;
using (SQLiteTransaction transactionx = dbConnect.BeginTransaction())
{
using (SQLiteCommand command = new SQLiteCommand(insertQuery, dbConnect))
{
command.Transaction = transactionx;
while (iterator <= listOfInts.Count - 1)
{
command.Parameters.AddWithValue("#toInsert", listOfInts[iterator]);
command.ExecuteNonQuery();
iterator++;
Console.WriteLine((iterator + 1).ToString() + Environment.NewLine);
}
}
transactionx.Commit();
dbConnect.Dispose();
}
I think the logic is fine and it would all be done properly but the update is so slow(even though I have an index onn the rownum column). Is there any way I can speed it up to some realistic time?
Thanks in advance.
This command:
update tempwords set rownum=#toInsert
updates all 360632 rows (with the same value).
When you execute this command 360632 times, you end up updating 122942799424 rows.
If you want to update only a single row with each command execution, you have to tell the database which row that is:
update tempwords set rownum = #toInsert where _id = #id_of_the_row
I am currently working on an C# project and I am trying to get the number of rows returned from MySQL Data Reader.
I know there is no direct function so I am trying to write my own. In the function, I pass it the MySQLDataReader object and then loop through the MySQL Data Reader and increment a counter and return the value of the counter.
This then seems to lock up the program, I guess because I am Reader.read() after I've got the count I'm already at the end. Instead I have tried creating a copy of the reader and then loop through the temp version but I get the same result.
Below is my code where I am executing the query and calling the function.
string query = "SELECT * FROM reports, software, platforms, versions "
+ "WHERE EmailVerified = #verified AND reports.SoftwareID = software.id AND reports.PlatformID = platforms.id "
+ "AND reports.VersionID = versions.id AND BugReportAcceptedNotificationSent = #notificationSent";
using (MySqlCommand cmd = new MySqlCommand(query, db.conn))
{
cmd.Parameters.AddWithValue("#verified", "1");
cmd.Parameters.AddWithValue("#notificationSent", "0");
using (MySqlDataReader reader = cmd.ExecuteReader())
{
totalEmails = HelperClass.totalRowsInMySQLDataReader(reader);
while (reader.Read())
{
currentEmailCount++;
EmailNotifications emailNotification = new EmailNotifications(reader);
emailNotification.sendNewBugReportAfterVerificationEmail(currentEmailCount, totalEmails);
}
}
}
Below is my function that gets the row count
public static int totalRowsInMySQLDataReader(MySqlDataReader reader)
{
MySqlDataReader tempReader = reader;
ILibraryInterface library = GeneralTasks.returnBitsLibrary(Configuration.databaseSettings, Configuration.engineConfig.logFile);
string methodInfo = classDetails + MethodInfo.GetCurrentMethod().Name;
try
{
int count = 0;
while (tempReader.Read())
{
count++;
}
tempReader = null;
return count;
}
catch (Exception ex)
{
string error = string.Format("Failed to get total rows in MySQL Database. Exception: {0}", ex.Message);
library.logging(methodInfo, error);
library.setAlarm(error, CommonTasks.AlarmStatus.Medium, methodInfo);
return -1;
}
}
Make use of a DataTable to load your results, e.g.
DataTable dt = new DataTable();
dt.Load(reader);
int numberOfResults = dt.Rows.Count;
You can then also iterate over the rows to read the values, e.g.
foreach(DataRow dr in dt.Rows)
{
var value = dr["SomeResultingColumn"]
}
The other option is to issue two separate SQL statements, however you would need to ensure both statements were enclosed within a transaction with a Serializable isolation level, this is needed to make sure records aren't inserted between the execution of the two SQL statements.
To avoid multiple queries, how about including the total in the select itself?
SELECT COUNT(*) AS TotalNORows, * FROM reports, software, platforms, versions etc
i think without executing another command it's not possible...as there is no method available for count in reader class
you can try this... if it works..
string query = "SELECT * FROM reports, software, platforms, versions "
+ "WHERE EmailVerified=#verified AND reports.SoftwareID=software.id AND reports.PlatformID=platforms.id "
+ "AND reports.VersionID=versions.id AND BugReportAcceptedNotificationSent=#notificationSent";
using (MySqlCommand cmd = new MySqlCommand(query, db.conn))
{
cmd.Parameters.AddWithValue("#verified", "1");
cmd.Parameters.AddWithValue("#notificationSent", "0");
using (MySqlDataReader reader = cmd.ExecuteReader())
{
// create a new connection db.conn2 then
MySqlCommand cmd2 = new MySqlCommand(query, db.conn2))
cmd2.Parameters.AddWithValue("#verified", "1");
cmd2.Parameters.AddWithValue("#notificationSent", "0");
MySqlDataReader reader2 = cmd2.ExecuteReader();
int numberofrow=0;
while(reader2.Read())
numberofrow++;
//your codes......
}
Was working on the same problem. I hate having to iterate if a method is already available, but this is was the shortest bit I could come up with:
MySqlCommand cmd = new MySqlCommand(query, connection);
MySqlDataReader reader = cmd.ExecuteReader();
int rowcount = 0;
while(reader.Read()){
rowcount++;
}
First, create this class:
public static class Extensions
{
public static int Count(this MySqlDataReader dr)
{
int count = 0;
while(dr.Read())
count++;
return count;
}
}
This will implement .Count () on MySqlDataReader.
int count = reader.Count();
Exemple:
string sql= "SELECT * FROM TABLE";
MySqlCommand cmd = new MySqlCommand(sql, connection);
MySqlDataReader reader = cmd.ExecuteReader();
int count = reader.Count();
Maybe you could look things the other way around
You could just do a select count(*) and get the row count
or
use a data adapter to fill a container (like a DataTable) and count the rows
Unfortunatelly solution from Jan Van #Herck will return one row only, so in case you are interested in getting all rows and their number in one select, this isn't what you need.
In that case I suggest uou to try this:
select * , (select count(*) from my_table) AS numRow from my_table;
or read this:
Getting total mysql results returned from query with a limit: FOUND_ROWS error
You can use follwoing SQL Query to get the total rows Count.
SELECT COUNT(*) FROM [MYTABLE]
from the Code you can use ExecuteScalar() method to get the total number of rows returned by QUERY.
Try This:
int GetRowsCount(MySqlCommand command)
{
int rowsCount=Convert.ToIn32(command.ExecuteScalar());
return rowsCount;
}
Use above function as below:
MySqlCommand command=new MySlCommand("Select count(*) from MyTable",connectionObj);
int totalRows = GetRowsCount(command)
OleDbDataReader dbreader = new OleDbDataReader();
int intcount = 0;
if (dbreader.HasRows == true)
{
if (dbreader.Read())
{
intcount = dbreader.RecordsAffected;
}
}
"dbreader.RecordsAffected" will give you the number rows changed,inserted or deleted by the last statement of SQL
I've data in DataTable with 2 rows and 3 columns. I want to insert that data into Oracle table.
How can I insert? please give me with some example.
And also
How can I pass datatable to storedprocedure in ORACLE...
I pass datatable in below mensioned manner, but datatable type problem is comming. how can I solve this?
cmd.Parameters.Add("#Details",dtSupplier);
(OR)
cmd.Parameters.Add("Details", DbType.Single).Value = dtSupplier.ToString();
want to insert dataset or a datatable into ORACLE,
create an ORACLE data adapter.
create a command object for insertion,
set the CommandType to StoredProcedure.
Update command of the data adapter,
pass the dataset or datatable as parameter.
like this:
OracleDataAdapter da = new OracleDataAdapter();
OracleCommand cmdOra = new OracleCommand(StoredProcedureName, Connection);
cmdOra.CommandType = CommandType.StoredProcedure;
da.InsertCommand = cmdOra;
da.Update(dsDataSet);
OR
if above dont work than pass datatable as xml prameter than than process it
For details check : ADO.NET DataTable as XML parameter to an Oracle/SQL Server Database Stored Procedure
OR
Check this thread on Oracle site : Thread: Pass data table to Oracle stored procedure
Check existing answer : How to Pass datatable as input to procedure in C#?
I'm very late for this answer, but I elaborated a bit to have some more readable (I hope) code, and to avoid all those .ToString() for the values so nulls and other less common values can be handled; here it is:
public void Copy(String tableName, DataTable dataTable)
{
var insert = $"insert into {tableName} ({GetColumnNames(dataTable)}) values ({GetParamPlaceholders(dataTable)})";
using (var connection = /*a method to get a new open connection*/)
{
for (var row = 0; row < dataTable.Rows.Count; row++)
{
InsertRow(dataTable, insert, connection, row);
}
}
}
private static void InsertRow(DataTable dataTable, String insert, OracleConnection connection, Int32 row)
{
using (var command = new OracleCommand(insert, connection))
{
AssembleParameters(dataTable, command, row);
command.ExecuteNonQuery();
}
}
private static void AssembleParameters(DataTable dataTable, OracleCommand command, Int32 row)
{
for (var col = 0; col < dataTable.Columns.Count; col++)
{
command.Parameters.Add(ParameterFor(dataTable, row, col));
}
}
private static OracleParameter ParameterFor(DataTable dataTable, Int32 row, Int32 col)
{
return new OracleParameter(GetParamName(dataTable.Columns[col]), dataTable.Rows[row].ItemArray.GetValue(col));
}
private static String GetColumnNames(DataTable data) => (from DataColumn column in data.Columns select column.ColumnName).StringJoin(", ");
private static String GetParamPlaceholders(DataTable data) => (from DataColumn column in data.Columns select GetParamName(column)).StringJoin(", ");
private static String GetParamName(DataColumn column) => $":{column.ColumnName}_param";
Hope this can be still useful to somebody
The best idea would be follow the step mentioned below
Create a transaction
Begin the transaction
Loop through you data table
call your procedure
If no error occurred commit transaction
else roll back transaction
Regarding this part of your question:
cmd.Parameters.Add("#Details",dtSupplier);
(OR)
cmd.Parameters.Add("Details", DbType.Single).Value = dtSupplier.ToString();
What is the type of the "Details" parameter? Is it a Single? Then you would have to pick one (1) value from your DataTable and pass it to your parameter, something like dtSupplier.Rows[0]["col"].
If you use dtSupplier.ToString() you are just making a string of the entire DataTable (which i guess will always be the type name of DataTable).
First of all, you need to add Oracle.DataAccess.dll as reference in Visual Studio. In most cases, you can find this dll in the directory C:\ProgramData\Oracle11g\product\11.2.0\client_1\ODP.NET\bin\2.x\Oracle.DataAccess.dll
If just you need to insert the records from DataTable to Oracle table, then you can call the below function. Consider that your DataTable name is dt.
string error = "";
int noOfInserts = DataTableToTable(dt,out error);
1. Without using Oracle Parameters(special character non-safe)
The definition of the function is given below. Here, we are just making the query dynamic for passing this as a sql statement to the InsertWithQuery function.
public int DataTableToTable(DataTable dt,out string error)
{
error = "";
for (int i = 0; i < dt.Rows.Count; i++)
{
finalSql = "INSERT INTO TABLENAME SELECT ";
for (int j = 0; j < dt.Columns.Count; j++)
{
colValue += "'" + dt.Rows[i][j].ToString() + "',";
}
colValue = colValue.Remove(colValue.Length - 1, 1);
finalSql += colValue + " FROM DUAL";
InsertWithQuery(finalSql, out error);
if (error != "")
return error;
inserts++;
colValue = "";
}
}
The code for InsertWithQuery function is given below. Here, in the connection string you have to place you database details like Host,Username,Password etc.
public int InsertWithQuery(string query, out string error)
{
error = "";
int rowsInserted = 0;
if (error == "")
{
OracleConnection con = new OracleConnection("Data Source=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=)(PORT=)))(CONNECT_DATA=(SERVER=DEDICATED)(SID=)));User Id=;Password=");
OracleTransaction trans = con.BeginTransaction();
try
{
error = "";
OracleCommand cmd = new OracleCommand();
cmd.Transaction = trans;
cmd.Connection = con;
cmd.CommandText = query;
rowsInserted = cmd.ExecuteNonQuery();
trans.Commit();
con.Dispose();
return rowsInserted;
}
catch (Exception ex)
{
trans.Rollback();
error = ex.Message;
rowsInserted = 0;
}
finally
{
con.Dispose();
}
}
return rowsInserted;
}
2. With using Oracle Parameters(special character safe)
This can handle special characters like single quotes like scenarios in the column values.
public int DataTableToTable(DataTable dt,out string error)
{
error = "";
string finalSql = "";
List<string> colValue = new List<string>();
List<string> cols = new List<string>() {"COLUMN1","COLUMN2","COLUMN3"};
for (int i = 0; i < dt.Rows.Count; i++)
{
finalSql = "INSERT INTO TABLENAME(COLUMN1,COLUMN2,COLUMN3) VALUES(:COLUMN1,:COLUMN2,:COLUMN3) ";
for (int j = 0; j < dt.Columns.Count; j++)
{
colValue.Add(dt.Rows[i][j].ToString());
}
objDAL.InsertWithParams(finalSql,colValue,cols, out error);
if (error != "")
return error;
inserts++;
colValue.Clear();
}
}
And the InsertWithParams is given below
public string InsertWithParams(string sql, List<string> colValue, List<string> cols, out string error)
{
error = "";
try
{
OracleConnection con = new OracleConnection("Data Source=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=)(PORT=)))(CONNECT_DATA=(SERVER=DEDICATED)(SID=)));User Id=;Password=");
OracleCommand command = new OracleCommand(sql, con);
for (int i = 0; i < colValue.Count; i++)
{
command.Parameters.Add(new OracleParameter(cols[i], colValue[i]));
}
command.ExecuteNonQuery();
command.Connection.Close();
}
catch (Exception ex)
{
error = ex.Message;
}
return null;
}
try {
//Suppose you have DataTable dt
string connectionString = #"Provider=Microsoft.ACE.OLEDB.12.0;" +
#"Data Source='Give path of your access database file here';Persist Security Info=False";
OleDbConnection dbConn = new OleDbConnection(connectionString);
dbConn.Open();
using (dbConn)
{
int j = 0;
for (int i = 0; i < 2; i++)
{
OleDbCommand cmd = new OleDbCommand(
"INSERT INTO Participant_Profile ([column1], [column2] , [column3] ) VALUES (#c1 , #c2 , #c3 )", dbConn);
cmd.Parameters.AddWithValue("#c1", dt.rows[i][j].ToString());
cmd.Parameters.AddWithValue("#c2", dt.rows[i][j].ToString());
cmd.Parameters.AddWithValue("#c3", dt.rows[i][j].ToString());
cmd.ExecuteNonQuery();
j++;
}
}
}
catch (OleDbException exception)
{
Console.WriteLine("SQL Error occured: " + exception);
}
I know it's been a big WHILE upon the matter, but the same need: "to insert data from a datatable to an Oracle table" has happened to me. I found this thread. I also tried the answers and came to the conclusion that executing a
...
cmd.ExecuteNonQuery();
...
in a loop, is bad. Reeaaally bad. The first thing that is bad is performance, the second is unnecessary complexity, the third is unnecessary Oracle Objects (stored proc). The time it takes to complete, lets say 200 rows, is almost 1 minute and that's me rounding it down. So in the hope that someone else will find this helpful here's my experience.
I got stubborn and searched some more, so I found out this, true it's from 2018. But I'm in 2021 myself...
So the base code is:
using Oracle.ManagedDataAccess.Client; // you don't need other dll, just install this from nuget gallery
using System.Data;
public static void Datatable2Oracle(string tableName, DataTable dataTable)
{
string connString = "connection string";
OracleBulkCopy copy= new(connString, OracleBulkCopyOptions.UseInternalTransaction /*I don't know what this option does*/);
copy.DestinationTableName = tableName;
copy.WriteToServer(dataTable);
copy.Dispose();
}
This should match a raw oracle DDL performance:
create table table_name as select * from other_table_name