Issue with MySqlBulkLoader from C# app - c#

I am writing an app to export some data from MSSQL to a MySQL server on a nightly basis. I use a simple query to grab all the data for the previous day and have then tried a few different approaches for getting it to MySQL. The fastest approach is using MySqlBulkLoader but for some reason it isn't moving all the data. After I do the insert, I am comparing the records in the text file that was generated to the number of records that are in MySQL and the counts are off from 1 in some cases all the way up to 10.
If I do the same approach for getting the data to the text file but loop through each row of the text file rather than bulk upload and do insert statements, all the records get imported.
Here is the bulk upload code I am currently using. I recently added the FieldQuotationCharacter to see if that would help and it didn't (when I added that, I made the text generation script enclose fields in quotes).
uploader.TableName = "testtable";
uploader.FieldTerminator = "\t";
uploader.LineTerminator = "\r\n";
uploader.NumberOfLinesToSkip = 0;
uploader.FileName = updateFile; //this is a variable pointing to the current file
uploader.Timeout = 120;
uploader.FieldQuotationCharacter = '"';
int totalExported = uploader.Load();
Any ideas?

Seems a little strange but on a whim I decided to write my output file with a blank line at the top and then set NumberOfLinesToSkip = 1. After doing this, everything worked and no records went missing. Kinda strange. Seems that setting it to 0 didn't work and may not be supported.

I was having a similar problem where it was loading about 2/3 of the rows in my file and the problem turned out to be the behavior of the local keyword in the load data sql that is generated by MySqlBulkLoader.
By default, it seems that MySqlBulkLoader.Local == true, this allows it to use a local file, but also affects the error handling for invalid rows. Instead of throwing an error when a row is invalid, it will give you "warnings". You can verify this and view the warning by running an actual sql command for load data (which is what MySqlBulkLoader uses):
load data local infile '/Temp/bb7dd81c-c79f-49c7-9ae4-fdc8e48df6d5.csv'
ignore into table my_staging_tbl
fields terminated by ',' enclosed by '"' escaped by '\\'
lines terminated by '\n'
This outputs the count of affected rows and a list of all warnings
In my case, I had the LineTerminator as "\n" instead of "\r\n" and after the warnings were suppressed it still ended up importing most of the rows.
You can override this behavior by setting MySqlBulkLoader.ConflictOption to MySqlBulkLoaderConflictOption.Replace instead of Ignore which is the default. Note that this also affects the way it handles duplicate keys.
More on the load data stuff for mysql: http://dev.mysql.com/doc/refman/5.7/en/load-data.html

If you stumble upon this post looking for a solution to trap those warnings, here it is. You actually need to add an event handler on the underlying connection to grab them. Like this:
connection.InfoMessage += new MySqlInfoMessageEventHandler(OnInfoMessage);
connection.Open();
MySqlBulkLoader bulkLoader = new MySqlBulkLoader(connection);
void OnInfoMessage(object sender, MySqlInfoMessageEventArgs e) {
MySqlCommand myCommand = new MySqlCommand("SHOW WARNINGS", (MySqlConnection)sender);
MySqlDataReader reader = myCommand.ExecuteReader();
while (reader.Read()) {
Console.WriteLine(reader[0].ToString() + " " + reader[1].ToString() + " " + reader[2].ToString());
}
}

Related

Access database acting randomly and inconsistently after being updated with OLEDB c#

So I am having a very weird issue, At first my code to insert items into my access database stopped working, it is a simple code like this:
using (OleDbConnection myCon = new OleDbConnection())
{
OleDbCommand cmd = new OleDbCommand()
{
CommandType = CommandType.Text,
CommandText = "insert into applicationSalts ([appName],[salt]) values (?,?)"
};
cmd.Parameters.AddWithValue("#appName", appName);
cmd.Parameters.AddWithValue("#salt", salt);
myCon.ConnectionString = publicDbConnectionString;
cmd.Connection = myCon;
myCon.Open();
int result = cmd.ExecuteNonQuery();
myCon.Close();
}
So it is very straight forward, inserting those values and I checked that result = 1 (query finished successfully).
Now it gets a bit weird, The database happens to not show any new values. Which is weird, Then I tried renaming the database and then all those records appear later! I tried using the database from another location (I was using it from C:/databaseName then now used it from D:/databaseName) then it worked. I moved the database file that worked back into C, then suddenly the records that appeared before disappeared.
Much weirder is that at the beginning one of the tables in the same database used to be working with an ideantical call as the one prior to this. But this one didnt! Then when I was trying to find the issue that one also stopped working..
Also after I manually changed some records when it was working through ms access it later ignored the changes..
I am slowly going insane as I am not really understanding what is going on, This is using access 2000 file format.
Edit: After further experimentation it is still getting weirder for me, While the file is called onlineDb.mdb it had a single record, deleting that made it disappear, Now renaming it to onDb.mdb made that record come back, adding other records that were missing. Then renaming to onlineDB.mdb again made all records disappear.

How can I display a message if a value already exists in database?

I am currently writing my first .Net & C# application with Visual Studio, and have a need to write generated values to MySQL from the application.
At present, I can write values fine - but I need to be able to check to see if a value exists and display that line if it does exist, otherwise insert new line to table. My connection string is defined at the top of the form.
I have the following defined already, and it writes to the database successfully if no duplicate values exist in the LicenseKey column. If a duplicate exists, it throws an unhandled exception.
private void SaveDetails()
{
// MySQL 'insert' command
string InsertNewLicense = "insert into BCOM.LicenseDetails(LicenseeName,ComputerName,ContactName,ContactEmail,LicenseKey,CreationDate) values('" +this.textBoxLicenseeName.Text+ "','" +this.textBoxComputerName.Text+ "','" +this.textBoxContactName.Text+ "','" +this.textBoxContactEmail.Text+ "','" +this.textBoxLicenseKey.Text+ "','" +this.textBoxCreationDate.Text+ "');";
//MySQL instance details
MySqlConnection InsertLicenseDetails = new MySqlConnection(LicenseDatabaseConnection);
//MySQL command execution
MySqlCommand InsertCommand = new MySqlCommand(InsertNewLicense, InsertLicenseDetails);
// Handles command outputs.
MySqlDataReader InsertReader;
//Opens connection to run query on database
InsertLicenseDetails.Open();
// Here our query will be executed and data saved into the database.
MessageBox.Show("License Details Saved. Please ensure you have emailed the license to the customer.");
while (InsertReader.Read())
{
}
InsertLicenseDetails.Close();
}
What I want to happen is for a check to be run on the LicenseKey column to see if the value exists, before different actions are taken.
If the value does not exist, I would like to insert the new line to the table (like my existing command does).
If, however, the value does exist, I would like to pop up a form showing the values from the line that the duplicate appears in as a form.
Where would I put in an event handler to read MySQLException values? What exception would I have to respond to for a duplicate value or no database response?
I agree with what the others have said in their comments, you could change the SQL Query to do the check instead of having 2.
IF(SELECT ... WHERE A = B)
RETURN THAT THE VALUE ALREADY EXISTS
ELSE
INSERT NEW VALUE
Also there was a good comment about SQL Injection and parameterized queries. The query string should look a bit more like
INSERT into BCOM.LicenseDetails(LicenseeName,ComputerName,ContactName,ContactEmail,LicenseKey,CreationDate) values(#LicenseeName, #ComputerName, #ContactName ...);
and your SqlCommand be parameterized
InsertCommand.Paramaters.AddWithValue("#LicenseeName", this.textBoxLicenseeName.Text);
InsertCommand.Paramaters.AddWithValue("#ComputerName", this.textBoxComputerName.Text);
...
That should be a good start to get you going.
After looking at the queries for a while I decided to try a different tack - instead of using a direct check if it's there, I opted to use a count(*) query. When I click the save button on the form, the buttonClick_event calls SaveDetails(), which runs the following:
private void SaveDetails()
{
string InsertNewLicense = "INSERT into BCOM.LicenseDetails(LicenseeName,ComputerName,ContactName,ContactEmail,LicenseKey,CreationDate) values(#LicenseeName, #ComputerName, #ContactName, #ContactEmail, #LicenseKey, #CreationDate)";
string LicenseExistence = "SELECT COUNT(*) FROM BCOM.LicenseDetails WHERE LicenseKey LIKE #LicenseKey";
MySqlConnection LicenseDetails = new MySqlConnection(LicenseDatabaseConnection);
MySqlCommand InsertCommand = new MySqlCommand(InsertNewLicense, LicenseDetails);
InsertCommand.Parameters.AddWithValue("#LicenseeName", this.textBoxLicenseeName.Text);
InsertCommand.Parameters.AddWithValue("#ComputerName", this.textBoxComputerName.Text);
InsertCommand.Parameters.AddWithValue("#ContactName", this.textBoxContactName.Text);
InsertCommand.Parameters.AddWithValue("#ContactEmail", this.textBoxContactEmail.Text);
InsertCommand.Parameters.AddWithValue("#LicenseKey", this.textBoxLicenseKey.Text);
InsertCommand.Parameters.AddWithValue("#CreationDate", this.textBoxCreationDate.Text);
MySqlCommand QueryCommand = new MySqlCommand(LicenseExistence, LicenseDetails);
QueryCommand.Parameters.AddWithValue("#LicenseKey", this.textBoxLicenseKey.Text);
MySqlDataReader InsertReader;
LicenseDetails.Open();
if ((int)(long)QueryCommand.ExecuteScalar() >0)
{
MessageBox.Show("This license already exists in the database.");
}
else
{
InsertReader = InsertCommand.ExecuteReader();
MessageBox.Show("License Details Saved. Please ensure you have emailed the license to the customer.");
while (InsertReader.Read())
{
}
}
LicenseDetails.Close();
So, if the query against the license keys returns with any results at all (more than 0 rows returned), a messagebox pops up showing that the key already exists. If the resultant number of rows is 0, the insert command gets run.
This was figured out with a look through MySQL command notes, testing with phpMyAdmin, matching against existing projects online, and support from the following:
The SELECT query was figured out with great support from #Seige.
The query was parameterized with help from Seige, following on from the advice of Sani Huttunen. Many thanks to them both.
Changing to the count method was done on the advice of a fellow coder in another community online - a good friend and brilliant coder.

Can't Access a xml file at random by C# console application

I have a C# console application which creates, parses and deletes multiple xml files at runtime. The application used to run fine in Windows 2003 server with .Net 2.0.
Recently, the Application framework was upgraded to >net 4.0 and the Windows Server OS to Windows 2008 64-bit.
Since then, the application encounters the following exception at random:
Access to the path 'D:\Content\iSDC\GDCOasis\GATE_DATA\LOG\635125008068192773\635125008074911566\SOD\AllRespId.xml' is denied.
at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
at System.IO.File.Delete(String path)
at ProcessGateFile.SOD.saveFile(String psFile, String psXMLString, Boolean isNonAscii)
The code for the creation, parsing and deletion is as follows:
saveFile(tmpPath + "\\SOD\\AllRespId.xml", "<?xml version= \"1.0\" ?><XML>" + sbldDistinctResp.ToString() + "</XML>", isChinese);
//Save list of Distinct responsibilities for User
sbldDistinctResp.Remove(0, sbldDistinctResp.Length);
xmlCase.Load(tmpPath + "\\SOD\\AllRespId.xml");
arrResps.Clear();
//Start preparing Responsibility selection criteria
RespNodes = xmlCase.SelectNodes("//row");
sRespCriteria = "";
if (RespNodes.Count > 0)
{
foreach (XmlNode RespNode in RespNodes)
{
string RespName = RespNode.Attributes.GetNamedItem("RespId").Value.ToString();
if (!arrResps.Contains(RespName))
{
arrResps.Add(RespName);
}
}
for (int i = 0; i < arrResps.Count; i++)
{
sbldDistinctResp.Append("(#RespId = '" + arrResps[i].ToString() + "') or ");
}
sbldDistinctResp.Remove(sbldDistinctResp.Length - 4, 4);
sRespCriteria = sbldDistinctResp.ToString();
if (!sRespCriteria.Equals(""))
{
sRespCriteria = "(" + sRespCriteria + ")";
}
}
File.Delete(tmpPath + "\\SOD\\AllRespId.xml");
I repeat, the error is happening at random, i.e. it works at times and does not at other times during the same process.
Any idea what might be causing this and how to resolve?
Just a couple of observations:
Why are you saving and then immediately loading the file again? In fact, why do you even need to save this file - you already have all the information you need in the sbldDistinctResp variable to generate the XML you need to work with (as evidenced by the saveFile call at the start of the code) - couldn't you just make a copy of it, surround it with the same XML as you did during saveFile, and work with that?
"It happens randomly" is a very subjective observation :). You should profile this (run it 10,000 times in a loop for example) and record the pattern of errors. You may well be surprised that what seems random at first actually shows a clear pattern over a large number of runs. This may help you to make a connection between the problem and some other apparently unrelated event on the server; or it may confirm that it truly is random and therefore outside of your control.
If you really can't find the problem and you go with the idea of anti-virus, etc, then you could wrap the loading code in a try/catch and re-try a couple of times if you get the error. It's hacky but it would work, assuming you have accepted that the initial error is beyond your control.

replace linebreak with \r\n when sending back to database

Newbie here again.
I have a form in c# that retrieves by calling a Web Service information from a database. One field it gets is just "notes" regarding this client.
The form i have allows multiple lines in the Notes field, but when i save it (send it via webservice to the database back) it doesn't recognize line breaks. How can i tell it to replace line breaks with the /r/n/ (is that right?) code, so when it retrieves it next time from the database it will know how to display it in the notes field?
This is the code i am using now which doesn't work.
if (MasterRecord.Customer.Notes.Trim().Length < 1)
{
string s = "";
foreach (object o in txtNotes.Text)
{
s += o.ToString() + ("\r\n");
}
}
Just do:
var stringToInsert = txtNotes.Text.Replace("\n", "\r\n");
When you are saving multiline text in sql server then replace carriage return with your Custom NewLine specifier for example : %NL%
txtNote.Text.Replace(Environment.NewLine, "%NL%");
and when you retrieving from database field then follow the reveres approach:
MasterRecord.Customer.Notes.Replace("%NL%", Environment.NewLine);
if you access this on .net platform then all ok.. i am not confirm about the the another platforms which acess this webserivce..
it works but it is trick to process data through application it saves %NL% in the field value.

Delete from database in c#

I'm writing an app in C# WPF with VS10 express.
I have to say I'm a very beginner in C# and VS but I'v searched a lot of examples on Google, I really tried to solve this problem on my own..
I have a local database (mydatabase.sdf) and at the load of my window I fill a table of that database with some data. One of the fields of that table needs a unique value, so I want to put in every load the same data, but I get an error than off course.
I want to delete all the data from the database before I refill, this seems to be so easy but I don't get it working...
I tried
dataset.Tables["mytable"].Clear()
that doesn't work, it seems to be deleting only data from the datagrid (dataTable) but not really from the datastore.
also I tried:
for (int i = 0; i < dataset.Tables["mytable"].Rows.Count; i++)
{
dataset.Tables["mytable"].Rows[i].Delete();
}
this.TableAdapter.Update(this.dataset);
But at startup the dataset.Tables["mytable"].Rows.Count statement returns zero at startup, but if I put in my data I get the "unique-value error".
The only way to get it deleted is to delete it manually from the datagrid and then push an Update button, that really deletes it from the datastore.
It is no option to make that field in the database not-unique because of development reasons.
How can I delete really data from the datastore/database (mydatabase.sdf) in the load of my program??
EDIT
Here is the code how I fill the database with data:
public void FillInternet()
{
klantenTableAdapter1.ClearBeforeFill = false;
string MyConString = "SERVER=myserver;" +
"DATABASE=mydb;" +
"UID=myuid;" +
"PASSWORD=mypass;";
MySqlConnection connection = new MySqlConnection(MyConString);
MySqlCommand command = connection.CreateCommand();
MySqlDataReader Reader;
command.CommandText = "SELECT klantnr, voorletters, roepnaam, achternaam, tussenvoegsel, meisjesnaam, straat, huisnr, subhuisnr, postcode, plaats, telthuis, telmobiel, telwerk, fax, email, geboortedatum FROM klanten ORDER BY klantnr";
connection.Open();
Reader = command.ExecuteReader();
try
{
while (Reader.Read())
{
DataRow newLogRow = dataset1.Tables["klanten"].NewRow();
var thisrow = "";
for (int i = 0; i < Reader.FieldCount; i++)
{
thisrow = Reader.GetValue(i).ToString();
newLogRow[Reader.GetName(i)] = thisrow;
}
dataset1.Tables["klanten"].Rows.Add(newLogRow);
this.klantenTableAdapter1.Update(this.dataset1);
}
connection.Close();
}
catch (Exception ex)
{
MessageBox.Show("Error: " + ex.Message,"Fout",MessageBoxButton.OK,MessageBoxImage.Error);
}
dataset1.AcceptChanges();
//Fill from internet
//da.Fill(dataset1.klanten);
//Fill from local database
klantenTableAdapter1.Fill(dataset1.klanten);
this.klantenTableAdapter1.Update(this.dataset1);
this.DataContext = dataset1.klanten.DefaultView;
}
ADO.NET uses a "disconnected" recordset model. It keeps a copy of the data in client-side structures (DataSet and DataTable). Updates/inserts/deletions made to the client-side structures need to be pushed back out to the database. You need to read up on ADO.NET to get a basic understanding of this process and to get a sense of the ADO.NET event-model, which will be necessary if you want to do anything that involves typical real-world complications. There are many books written on ADO.NET because it is a feature-rich middle-tier data layer with significant complexities.
For your purposes, you could read up on the ADO.NET Command object and the SQL "delete" command. You will also need to explore how ADO.NET handles autoincrementing primary keys, which is one of the trickiest aspects of the disconnected model.
If the database itself defines an autoincrementing key, you cannot supply that value when inserting new rows unless you turn the auto-increment off temporarily in the back-end. That is not an ADO.NET issue, BTW. That is 100% back-end.
From your other posts, I'm going on the assumption that your database on this too is MySQL. You mention a unique column which typically means an auto-increment column. If you "delete" the entries after you've built them (say starting 1-10), and then try to re-add your next cycle the same 1-10 items, it can choke on you giving you this message. If you add numbers to your table starting with the last one used... see if that helps.

Categories