We Keep Coding

Method with MySql query for LOAD DATA IN FILE never finishes

I am populating a MySQL table in C# using the LOAD DATA INFILE method.
Therefore I wrote a method for creating the query string...
public static string LoadDataInFile(string TempCsv, string MatchId)
{
StringBuilder MySqlString = new StringBuilder();
MySqlString.AppendFormat(#"LOAD DATA INFILE '{0}' INTO TABLE {1}_trackingdata
FIELDS TERMINATED BY ';'
IGNORE 1 LINES", TempCsv, MatchId);
return MySqlString.ToString();
}
and one that executes raw queries:
public static void ExecuteRawQuery(string Query, string ConnectionString)
{
using (MySqlConnection conn = new MySqlConnection(ConnectionString.ToString()))
{
try
{
conn.Open();
MySqlCommand cmd = new MySqlCommand(Query, conn);
cmd.ExecuteNonQuery();
conn.Close();
Console.WriteLine("Import probably done...");
return;
}
catch (MySqlException SqlException)
{
Console.ForegroundColor = ConsoleColor.DarkRed;
Console.WriteLine("SqlException: {0}", SqlException.Message);
Console.WriteLine("Exit program.");
Console.ResetColor();
Environment.Exit(0);
}
}
}
The table gets populated as expected (checked with Sequel Pro), but the application never continues.
public static void Main(string[] args)
{
// ...
ExecuteRawQuery(LoadDataInFile("myFile.csv", "123456"));
Console.WriteLine("I will never be displayed..");
// ...
}
Is there any reason why ExecuteRawQuery() never finishes? When I execute other queries (SELECT, UPDATE) everything works as expected.

I don't know why, but using the MySqlClient.MySqlBulkLoader-class solves the problem. Leaving this here for anybody experiencing the same issue:
public static void ImportCSV(string CSV, string TableName, string ConnectionString)
{
using (MySqlConnection conn = new MySqlConnection(ConnectionString.ToString()))
{
MySqlBulkLoader Bulkloader = new MySqlBulkLoader(conn);
Bulkloader.TableName = TableName;
Bulkloader.FileName = CSV;
Bulkloader.Timeout = 0;
Bulkloader.NumberOfLinesToSkip = 1;
Bulkloader.FieldTerminator = ";";
var result = Bulkloader.Load();
Console.WriteLine("Imported {0} rows into the database", result);
}
}

C# ASP.Net Processing Issue - Reading Table While Deleting Rows

Simply, I have an application that has one page that deletes and then re-adds/refreshes the records into a table every 30 seconds. I have another page that runs every 45 seconds that reads the table data and builds a chart.
The problem is, in the read/view page, every once in a while I get a 0 value (from a max count) and the chart shows nothing. I have a feeling that this is happening because the read is being done at the exact same time the delete page has deleted all the records in the table but has not yet refreshed/re-added them.
Is there a way in my application I can hold off on the read when the table is being refreshed?
Best Regards,
Andy
C#
ASP.Net 4.5
SQL Server 2012
My code below is run in an ASP.Net 4.5 built Windows service. It deletes all records in the ActualPlot table and then refreshes/adds new records from a text file every 30 seconds. I basically need to block (lock?) any user from reading the ActualPlot table while the records are being deleted and refreshed. Can you PLEASE help me change my code to do this?
private void timer1_Tick(object sender, ElapsedEventArgs e)
{
// Open the SAP text files, clear the data in the tables and repopulate the new SAP data into the tables.
var cnnString = ConfigurationManager.ConnectionStrings["TaktBoardsConnectionString"].ConnectionString;
SqlConnection conn = new SqlConnection(cnnString);
SqlConnection conndetail = new SqlConnection(cnnString);
SqlConnection connEdit = new SqlConnection(cnnString);
SqlCommand cmdGetProductFile = new SqlCommand();
SqlDataReader reader;
string sql;
// Delete all the records from the ActualPlot and the ActualPlotPreload tables. We are going to repopulate them with the data from the text file.
sql = "DELETE FROM ActualPlotPreload";
try
{
conn.Open();
SqlCommand cmd = new SqlCommand(sql, conn);
cmd.ExecuteNonQuery();
}
catch (System.Data.SqlClient.SqlException ex)
{
string msg = "Delete Error:";
msg += ex.Message;
Library.WriteErrorLog(msg);
}
finally
{
conn.Close();
}
sql = "DELETE FROM ActualPlot";
try
{
conn.Open();
SqlCommand cmd = new SqlCommand(sql, conn);
cmd.ExecuteNonQuery();
}
catch (System.Data.SqlClient.SqlException ex)
{
string msg = "Delete Error:";
msg += ex.Message;
Library.WriteErrorLog(msg);
}
finally
{
conn.Close();
}
// Read the SAP text file and load the data into the ActualPlotPreload table
sql = "SELECT DISTINCT [BoardName], [ProductFile], [ProductFileIdent] FROM [TaktBoards].[dbo].[TaktBoard] ";
sql = sql + "JOIN [TaktBoards].[dbo].[Product] ON [Product].[ProductID] = [TaktBoard].[ProductID]";
cmdGetProductFile.CommandText = sql;
cmdGetProductFile.CommandType = CommandType.Text;
cmdGetProductFile.Connection = conn;
conn.Open();
reader = cmdGetProductFile.ExecuteReader();
string DBProductFile = "";
string DBTischID = "";
string filepath = "";
string[] cellvalues;
DateTime dt, DateCheckNotMidnightShift;
DateTime ldSAPFileLastMod = DateTime.Now;
string MyDateString;
int FileRecordCount = 1;
while (reader.Read())
{
DBProductFile = (string)reader["ProductFile"];
DBTischID = (string)reader["ProductFileIdent"];
filepath = "c:\\inetpub\\wwwroot\\WebApps\\TaktBoard\\FilesFromSAP\\" + DBProductFile;
FileInfo fileInfo = new FileInfo(filepath); // Open file
ldSAPFileLastMod = fileInfo.LastWriteTime; // Get last time modified
try
{
StreamReader sr = new StreamReader(filepath);
FileRecordCount = 1;
// Populate the AcutalPlotPreload table from with the dates from the SAP text file.
sql = "INSERT into ActualPlotPreload (ActualDate, TischID) values (#ActualDate, #TischID)";
while (!sr.EndOfStream)
{
cellvalues = sr.ReadLine().Split(';');
if (FileRecordCount > 1 & cellvalues[7] != "")
{
MyDateString = cellvalues[7];
DateTime ldDateCheck = DateTime.ParseExact(MyDateString, "M/dd/yyyy", null);
DateTime dateNow = DateTime.Now;
string lsDateString = dateNow.Month + "/" + dateNow.Day.ToString("d2") + "/" + dateNow.Year;
DateTime ldCurrentDate = DateTime.ParseExact(lsDateString, "M/dd/yyyy", null);
string lsTischID = cellvalues[119];
if (ldDateCheck == ldCurrentDate)
{
try
{
conndetail.Open();
SqlCommand cmd = new SqlCommand(sql, conndetail);
cmd.Parameters.Add("#ActualDate", SqlDbType.DateTime);
cmd.Parameters.Add("#TischID", SqlDbType.VarChar);
cmd.Parameters["#TischID"].Value = cellvalues[119];
MyDateString = cellvalues[7] + " " + cellvalues[55];
dt = DateTime.ParseExact(MyDateString, "M/dd/yyyy H:mm:ss", null);
cmd.Parameters["#ActualDate"].Value = dt;
// Ignore any midnight shift (12am to 3/4am) units built.
DateCheckNotMidnightShift = DateTime.ParseExact(cellvalues[7] + " 6:00:00", "M/dd/yyyy H:mm:ss", null);
if (dt >= DateCheckNotMidnightShift)
{
cmd.ExecuteNonQuery();
}
}
catch (System.Data.SqlClient.SqlException ex)
{
string msg = "Insert Error:";
msg += ex.Message;
Library.WriteErrorLog(msg);
}
finally
{
conndetail.Close();
}
}
}
FileRecordCount++;
}
sr.Close();
}
catch
{ }
finally
{ }
}
conn.Close();
// Get the unique TischID's and ActualDate from the ActualPlotPreload table. Then loop through each one, adding the ActualUnits
// AcutalDate and TischID to the ActualPlot table. For each unique TischID we make sure that we reset the liTargetUnits to 1 and
// count up as we insert.
SqlCommand cmdGetTischID = new SqlCommand();
SqlDataReader readerTischID;
int liTargetUnits = 0;
string sqlInsert = "INSERT into ActualPlot (ActualUnits, ActualDate, TischID) values (#ActualUnits, #ActualDate, #TischID)";
sql = "SELECT DISTINCT [ActualDate], [TischID] FROM [TaktBoards].[dbo].[ActualPlotPreload] ORDER BY [TischID], [ActualDate] ASC ";
cmdGetTischID.CommandText = sql;
cmdGetTischID.CommandType = CommandType.Text;
cmdGetTischID.Connection = conn;
conn.Open();
readerTischID = cmdGetTischID.ExecuteReader();
DBTischID = "";
DateTime DBActualDate;
string DBTischIDInitial = "";
while (readerTischID.Read())
{
DBTischID = (string)readerTischID["TischID"];
DBActualDate = (DateTime)readerTischID["ActualDate"];
if (DBTischIDInitial != DBTischID)
{
liTargetUnits = 1;
DBTischIDInitial = DBTischID;
}
else
{
liTargetUnits++;
}
try
{
conndetail.Open();
SqlCommand cmd = new SqlCommand(sqlInsert, conndetail);
cmd.Parameters.Add("#ActualUnits", SqlDbType.Real);
cmd.Parameters.Add("#ActualDate", SqlDbType.DateTime);
cmd.Parameters.Add("#TischID", SqlDbType.VarChar);
cmd.Parameters["#TischID"].Value = DBTischID;
cmd.Parameters["#ActualDate"].Value = DBActualDate;
cmd.Parameters["#ActualUnits"].Value = liTargetUnits;
cmd.ExecuteNonQuery();
cmd.Parameters.Clear();
}
catch (System.Data.SqlClient.SqlException ex)
{
string msg = "Insert Error:";
msg += ex.Message;
Library.WriteErrorLog(msg);
}
finally
{
conndetail.Close();
}
}
conn.Close();
Library.WriteErrorLog("SAP text file data has been imported.");
}

If the data is being re-added right back after the delete (basically you know what to re-add before emptying the table), you could have both operation within the same SQL transaction, so that the data will be available to the other page only when it has been re-added.
I mean something like that :
public bool DeleteAndAddData(string connString)
{
using (OleDbConnection conn = new OleDbConnection(connString))
{
OleDbTransaction tran = null;
try
{
conn.Open();
tran = conn.BeginTransaction();
OleDbCommand deleteComm = new OleDbCommand("DELETE FROM Table", conn);
deleteComm.ExecuteNonQuery();
OleDbCommand reAddComm = new OleDbCommand("INSERT INTO Table VALUES(1, 'blabla', 'etc.'", conn);
reAddComm.ExecuteNonQuery();
tran.Commit();
}
catch (Exception ex)
{
tran.Rollback();
return false;
}
}
return true;
}

If your queries don't take too long to execute, you can start the two with a difference of 7.5 seconds, as there is a collision at every 90 seconds when the read/write finishes 3 cycles, and read/view finishes 2 cycles.
That being said, it's not a fool-proof solution, just a trick based on assumptions, in case you wan't to be completely sure that read/view never happens when read/write cycle is happening, try considering having a Read Lock. I would recommend reading Understanding how SQL Server executes a query and Locking in the Database Engine
Hope that helps.

I would try a couple of things:
Make sure your DELETE + INSERT operation is occurring within a single transaction:
BEGIN TRAN
DELETE FROM ...
INSERT INTO ...
COMMIT
If this isn't a busy table, try locking hints your SELECT statement. For example:
SELECT ...
FROM Table
WITH (UPDLOCK, HOLDLOCK)
In the case where the update transactions starts while your SELECT statement is running, this will cause that transaction to wait until the SELECT is finished. Unfortunately it will block other SELECT statements too, but you don't risk reading dirty data.

I was not able to figure this out but I changed my code so the program was not deleting all the rows in the ActualPlot table but checking to see if the row was there and if not adding the new row from the text file.

C# Unable to detach database

I have a class that creates a database that saves the mdf file in a specified location. Then it copies tables from an existing database. Then creates stored procedures from an sql file. Then detaches the database created from the start once the process is done. My problem is that my detach method won't work throwing an exception saying that the database is in use. I have disposed my connections properly.
This is in-line with my previous question.
Here is my class:
Event
private void btnFullBackup_Click(object sender, EventArgs e)
{
progressBar.Value = 0;
lblStatus.Text = "Starting full backup...";
CreateDB("FULL");
progressBar.Value = 20;
lblStatus.Text = "Copying tables...";
CopyTables("FULL");
progressBar.Value = 60;
lblStatus.Text = "Creating stored procedures...";
CreateStoredProcedures("FULL");
progressBar.Value = 70;
progressBar.Value = 80;
DetachBackup("FULL");
lblStatus.Text = "Done";
progressBar.Value = 100;
MessageBox.Show("Backup was created successfully", "",
MessageBoxButtons.OK, MessageBoxIcon.Information);
}
Methods used:
void CreateDB(string type)
{
//define and browse location to save mdf
lblStatus.Text = "Creating pysical database...";
FolderBrowserDialog folderBrowserDialog = new FolderBrowserDialog();
folderBrowserDialog.ShowDialog();
lblStatus.Text = "Checking folder permission...";
string selectedFolder = folderBrowserDialog.SelectedPath + "\\";
newBackupLocation = selectedFolder;
//check permission
if (WriteAccessToFolder(selectedFolder) == false)
{
MessageBox.Show("The folder you have chosen does not have write permission", "Monytron",
MessageBoxButtons.OK, MessageBoxIcon.Error);
folderBrowserDialog.ShowDialog();
return;
}
//create DB
lblStatus.Text = "Creating database...";
string connectionString = ConfigurationManager.ConnectionStrings["ConnectionString"].ConnectionString;
var query = GetDbCreationQuery(selectedFolder, type);
using (var conn = new SqlConnection(connectionString))
using (var command = new SqlCommand(query, conn))
{
try
{
conn.Open();
command.ExecuteNonQuery();
folderBrowserDialog.Dispose();
}
catch (Exception ex)
{
MessageBox.Show(ex.ToString());
}
finally
{
if ((conn.State == ConnectionState.Open))
{
conn.Close();
}
}
}
}
void CopyTables(string backupDBName)
{
string connectionString = ConfigurationManager.ConnectionStrings["ConnectionString"].ConnectionString;
var query = CopyQuery(backupDBName + DateTime.Now.ToString("yyyyMMdd"));
using (var conn = new SqlConnection(connectionString))
using (var command = new SqlCommand(query, conn))
{
try
{
conn.Open();
command.ExecuteNonQuery();
}
catch (Exception ex)
{
MessageBox.Show(ex.ToString());
}
finally
{
if ((conn.State == ConnectionState.Open))
{
conn.Close();
}
}
}
}
void CreateStoredProcedures(string type)
{
string connectionString = ConfigurationManager.ConnectionStrings["ConnectionString"].ConnectionString;
using (var conn = new SqlConnection(connectionString + ";database=" + type + DateTime.Now.ToString("yyyyMMdd")))
{
string spLocation = File.ReadAllText("CreateStoredProcedures.sql");
Server server = new Server(new ServerConnection(conn));
try
{
server.ConnectionContext.ExecuteNonQuery(spLocation);
}
catch (Exception ex)
{
MessageBox.Show(ex.ToString());
}
}
}
bool DetachBackup(string backupDBName)
{
string connectionString = ConfigurationManager.ConnectionStrings["ConnectionString"].ConnectionString;
var builder = new SqlConnectionStringBuilder(connectionString);
string serverName = builder.DataSource;
string dbName = builder.InitialCatalog;
try
{
Server smoServer = new Server(serverName);
smoServer.DetachDatabase(backupDBName + DateTime.Now.ToString("yyyyMMdd"), false);
return true;
}
catch (Exception ex)
{
MessageBox.Show(ex.ToString());
return false;
}
}

The Connection to the database is in most cases placed in a pool after using. This way you can re-connect quickly using the same connection string, but on the other hand, I suspect this pool of connections is blocking you from detaching a database.
You can probably do something like this:
Put use master as the last statement in each query against database before you close the connection, or
Modify connection string so it doesn't use pooling (uid=...; pwd=...; pooling=false;)
Hope it helps.

You should first kill connections to the database if you want to keep connection pooling. You can do it setting the database in single user access with rollback_immediate clause before calling the detach method.
Have a look here to use C#:
Is there a way to set the DB as Single User Mode in C#?
Or here to run T-SQL script:
https://serverfault.com/questions/76432/how-can-i-detach-a-database-that-is-in-use

C# Mysql Connection must be valid and open

First of all: I got my code running without using oop. I declared all my variables inside the same class and opened/closed the connection right before and after passing the query to the db. That worked! Now with some new experiences I tried to split my code into different classes. Now it wont work anymore.
It tells me "Connection must be valid and open". Enough text, here's my current code:
Services.cs
public static MySqlConnection conn // Returns the connection itself
{
get
{
MySqlConnection conn = new MySqlConnection(Services.ServerConnection);
return conn;
}
}
public static string ServerConnection // Returns the connectin-string
{
get
{
return String.Format("Server={0};Port=XXXX;Database=xxx;Uid=xxx;password=xxXxxXxXxxXxxXX;", key);
}
}
public static void DB_Select(string s, params List<string>[] lists)
{
try
{
MySqlCommand cmd = conn.CreateCommand();
cmd.CommandType = CommandType.Text;
string command = s;
cmd.CommandText = command;
MySqlDataReader sqlreader = cmd.ExecuteReader();
while (sqlreader.Read())
{
if (sqlreader[0].ToString().Length > 0)
{
for (int i = 0; i < lists.Count(); i++)
{
lists[i].Add(sqlreader[i].ToString());
}
}
else
{
foreach (List<string> save in lists)
{
save.Add("/");
}
}
}
sqlreader.Close();
}
catch (Exception ex)
{
MessageBox.Show("Error while selecting data from database!\nDetails: " + ex);
}
}
LoginForm.cs
private void checkUser(string username, string password)
{
using (Services.conn)
{
Services.conn.Open();
Services.DB_Select("..a short select statement..");
Services.conn.Close();
}
I guess this is all we need. I have shortened my code to get a focus on the problem.
I created Services.cs to get a global way to access the db from all forms without copy&pasting the connection info. Now when I reach my LoginForm.cs it throws an error "Connection must be valid and open". I've already debugged my code. It's all time closed. Even when passing conn.Open() it stays closed. Why?
Another try: I've also tried placing conn.Open() and conn.Close() inside Services.DB_Select(..) at the beginning and end. Same error here.
I have to say: The code worked before and I've used the same connection-string. So the string itself is surely valid.
I appreciate any help given here!

The problem is that you don't store the connection that was returned from your factory property. But don't use a property like a method. Instead use it in this way:
using (var con = Services.conn)
{
Services.conn.Open();
Services.DB_Select("..a short select statement..", con ));
//Services.conn.Close(); unnecessary with using
}
So use the same connection in the using that was returned from the property(or better created in the using) and pass it to the method which uses it. By the way, using a property as factory method is not best practise.
But in my opinion it's much better to create the connection where you use it, best place is in the using statement. And throw the con property to the garbage can, it is pointless and a source for nasty errors.
public static void DB_Select(string s, params List<string>[] lists)
{
try
{
using(var conn = new MySqlConnection(Services.ServerConnection))
{
conn.Open();
MySqlCommand cmd = conn.CreateCommand();
cmd.CommandText = s;
using( var sqlreader = cmd.ExecuteReader())
while (sqlreader.Read())
{
if (sqlreader[0].ToString().Length > 0)
{
for (int i = 0; i < lists.Count(); i++)
{
lists[i].Add(sqlreader[i].ToString());
}
}
else
{
foreach (List<string> save in lists)
{
save.Add("/");
}
}
} // unnecessary to close the connection
} // or the reader with the using-stetement
}
catch (Exception ex)
{
MessageBox.Show("Error while selecting data from database!\nDetails: " + ex);
}
}

Try to restructure your Services class as follows
public static MySqlConnection conn // Returns the connection itself
{
get
{
MySqlConnection conn = new MySqlConnection(Services.ServerConnection);
return conn;
}
}
private static string ServerConnection // Returns the connectin-string - PRIVATE [Improved security]
{
get
{
return String.Format("Server={0};Port=XXXX;Database=xxx;Uid=xxx;password=xxXxxXxXxxXxxXX;", key);
}
}
// Rather than executing result here, return the result to LoginForm - Future improvement
public static void DB_Select(MySqlConnection conn ,string s, params List<string>[] lists)
{
try
{
MySqlCommand cmd = conn.CreateCommand();
cmd.CommandType = CommandType.Text;
string command = s;
cmd.CommandText = command;
MySqlDataReader sqlreader = cmd.ExecuteReader();
while (sqlreader.Read())
{
if (sqlreader[0].ToString().Length > 0)
{
for (int i = 0; i < lists.Count(); i++)
{
lists[i].Add(sqlreader[i].ToString());
}
}
else
{
foreach (List<string> save in lists)
{
save.Add("/");
}
}
}
sqlreader.Close();
}
catch (Exception ex)
{
MessageBox.Show("Error while selecting data from database!\nDetails: " + ex);
}
}
In LoginForm.cs use returning connection and store it there. When you need to execute query, use
MySqlConnection conn=Services.conn(); // Get a new connection
Services.DB_Select(conn,"..a short select statement.."); // Executing requirement
Services.conn.Close();
Additional - I suggest you need to return MySqlDataReader to LoginForm and handle results there

private MySqlConnection _conn;
public MySqlConnection conn // Returns the connection itself
{
get
{
if(_conn == null)
_conn = new MySqlConnection(Services.ServerConnection);
return _conn;
}
}

SQL query doesn't get inserted

I've been trying to get my query to work for some time it runs but doesn't insert anything nor does it return any errors.
The database connection is open and is successfuly connection.
The Table is called errorlog and holds the following data
- id (int autoincremental, Primary key, Unique)
- exception (varchar)
- time (DateTime)
exception = String(error message)
time = DateTime.Now
Here's the code:
public void insertError(string error, DateTime time)
{
SqlCeParameter[] sqlParams = new SqlCeParameter[]
{
new SqlCeParameter("#exception", error),
new SqlCeParameter("#time", time)
};
try
{
cmd = new SqlCeCommand();
cmd.Connection = connection;
cmd.CommandType = CommandType.Text;
cmd.CommandText = "INSERT INTO errorlog (exception, time) VALUES(#exception, #time)";
cmd.Parameters.AddRange(sqlParams);
cmd.ExecuteNonQuery();
}
catch (Exception e)
{
Console.WriteLine(e.Message);
}
}
Any help would be appreciated, Thanks in advance.
EDIT
Removed quotes around #exception
Heres the connection:
protected DataController()
{
try
{
string appPath = System.IO.Path.GetDirectoryName(Assembly.GetAssembly(typeof(DataController)).CodeBase).Replace(#"file:\", "") + #"\";
string strCon = #"Data Source = " + appPath + #"Data\EasyShop.sdf";
connection = new SqlCeConnection(strCon);
}
catch (Exception e)
{
}
connection.Open();
}
Finally the way it gets called:
public bool log(string msg, bool timestamp = true)
{
DataController dc = DataController.Instance();
dc.insertError(msg, DateTime.Today);
return true;
}

Debug your application and see if connection points exactly to the
database you want. Also check if you look for the inserted records
in the same database.
If your connection belongs to the transaction, check if it's committed. You will not see those records inserted until transaction is committed.
It seems to me, that you INSERT is wrong. Remove quotes around #exception
Open SQL Server Profiler, connect to your database and check if your INSERT appears in there.