I come to you after a lot of hours googling and reading other discussions about SQLite on StackOverflow, but I definitely can't find any explanation to my problem, so here it is :
The context :
I'm developping an application for iPad wich has to deal with some "large" amounts of data, in several occasions. In one of them, I must import points coordinates from a .kml file (Google's xml for geographical data) into my database, in order to reuse them later with a MKMapView and load them faster than by parsing xml when it needs to show a specific layer.
The details :
The import thing is quite easy : when dealing with those files, I'm only concerned with 2 tables :
One containing zones definitions and details : for the moment, an integer as an id, and a text for naming.
One containing two real for coordinate storage and an integer referencing the first table for knowing which zone point is part of.
So as long as reading my file, I first create an entry for the new zone, and then I insert points into the second table, with ID of the last zone created in the first table...nothing complicated!
But...
The problem :
After running fine a while, I get an exception from SQLite with the famous message "Unable to open the database file", and then it comes I can't do anything more with the database. This exception can randomly occur in the zone creation or the points insertion methods.
My reflexions :
Considering the numerous points in those files, I suspected memory or disk saturation but other parts of my app discarded those points (to my mind).
First, memory : it comes that when the exception occurs, the app is using about 10 or 12 MB of RAM. It can seems quite huge, but it's due to the ~10MB .kml file loaded in memory, so it's explainable. And above it all, the MKMapView thing of my app deals with tons of high-res tiles layers above map, and so leads to memory peaks which can afford 20 or even 25MB without making the iPad to crash.
Second, disk : when reseting my database and filling only the 2 tables described above, the db file size when the exception occurs is always about 2.2 or 2.5MB, but when I fill other tables (the other parts of my apps works well!) the db file is about 6 or 7MB, and the device doesn't complain at all.
So what?!
CPU-angryness and panic? I don't think so because some of the other tables of my database are filled at the same rythm without problem... and running my app in simulator crashes too, with a core i7 just laughing at the job.
SQLite bad use? There we go! To my mind, it's the only solution left! But I really can't understand what's going on here because I process my requests the same way I do in other app's parts which - repeating myself - work like a charm!
SQLite details :
I have a DB class which is a singleton I use to avoid creating/releasing an SqliteConnection object each request I do, and all my methods dealing with database are contained in this class to be sure I don't play with the connection anywhere else without knowing it. Here are concerned methods of this class :
public void saveZone(ObjZone zone) { //at this point, just creates an entry with a name and let sqlite give it a new id
lock (connection) { //SqliteConnection object
try {
openConnection();
SqliteCommand cmd = connection.CreateCommand();
cmd.CommandText = zone.id == 0 ?
"insert into ZONES (Z_NAME) values (" + format(zone.name) + ") ;" :
"update ZONES set Z_NAME = " + format(zone.name) + " where Z_ID = " + format(zone.id) + " ;";
cmd.ExecuteNonQuery();
if (zone.id == 0) {
cmd.CommandText = "select Z_ID from ZONES where ROWID = last_insert_rowid() ;";
zone.id = uint.Parse(cmd.ExecuteScalar().ToString());
}
cmd.Dispose();
}
catch (Exception e) {
Log.failure("DB.saveZone(" + zone.ToString() + ") : [" + e.GetType().ToString() + "] - " +
e.Message + "\n" + e.StackTrace); //custom Console.WriteLine() method with some formating
throw e;
}
finally {
connection.Close();
}
}
}
public void setPointsForZone(List<CLLocationCoordinate2D> points, uint zone_id) { //registers points for a given zone
lock (connection) {
try {
openConnection();
SqliteCommand cmd = connection.CreateCommand();
cmd.CommandText = "delete from ZONESPOINTS where Z_ID = " + format(zone_id);
cmd.ExecuteNonQuery();
foreach(CLLocationCoordinate2D point in points) {
cmd.CommandText = "insert into ZONESPOINTS values " +
"(" + format(zi_id) + ", " + format(point.Latitude.ToString().Replace(",", ".")) + ", "
+ format(point.Longitude.ToString().Replace(",", ".")) + ");";
cmd.ExecuteNonQuery();
cmd.Dispose();
}
}
catch (Exception e) {
Log.failure("DB.setPointsForZone(" + zone_id + ") : [" + e.GetType().ToString() + "] - " + e.Message);
throw e;
}
finally {
connection.Close();
}
}
}
And to be as clear as I can, here are some of the methods referenced in the two above (I use this custom openConnection() method because I use foreign keys constraints in most of my tables and cascading behaviours are not enabled by default, but I need them.) :
void openConnection() {
try {
connection.Open();
SqliteCommand cmd = connection.CreateCommand();
cmd.CommandText = "PRAGMA foreign_keys = ON";
cmd.ExecuteNonQuery();
cmd.Dispose();
}
catch (Exception e) {
Log.failure("DB.openConnection() : [" + e.GetType().ToString() + "] - " + e.Message);
throw e;
}
}
public static string format(object o) {
return "'" + o.ToString().Replace("'", "''") + "'";
}
Well, sorry for the novel, I may already thank you for reading all that stuff, no?! Anyway, if I missed something that could be useful, let me know and I'll document it as soon as possible.
I hope someone will be able to help me, anyway, thank you by advance!
(And my apologies for my poor frenchie's english.)
EDIT
My problem is "solved"! After a few changes for debugging pourposes, no big modifications, and no success, I put back the code in the state I posted it... and now it works. But I really would appreciate if some someone could give me an explanation of what may have happened! It seems like SQLite behaviour (on iPad at least - never used it anywhere else) can be quite obscure at some times... :/
I wouldn't cross my fingers for this but I would try two things:
If possible, pre-process the KML file to a second SQLite database and use this database to import data in the main database (thinking of lower memory/processor requirements)
Transaction the imported data in small batches.
HTH
EDIT: you might have checked this already, but anyway: unable to open database.
Related
I have a string array in a C# program that I need to load into a temp table in SQL Server. I am currently using a foreach loop to run the insert query. When the SQL connection closes, the temp table disappears so I end up with a table that only has 1 row when there are hundreds in the array.
I have tried adding
using (SqlConnection sqlconnection2 = new SqlConnection())
statement at the top of the method (removing the existing connection line, inside the If(ConnSucceeds) statement, just inside the try block, and inside the foreach loop. when its inside the foreach loop i have the same issue. Putting it anywhere else I get an error stating that the connection isn't open.
Ultimately I will need to add at least one more foreach loop to run another SQL query to manipulate the data and then find some way to export it to a text file all using the same connection.
private void ImportToTempTable()
{
this.GetProgramInfoForSQLQuery();
this.GetInstallFolder();
string config = this._InstallFolder + #"\" + this._Version + #"\" + this._brandName + #".exe.config";
GetInstanceName(config);
string connStr = "<proprietary conn string parameters>";
bool ConnSucceeds = false;
SqlConnection sqlConnection = new SqlConnection();
StringBuilder errorMessages = new StringBuilder();
if (!ConnSucceeds)
{
try
{
sqlConnection.ConnectionString = connStr;
sqlConnection.Open();
this.WriteNote("SQL Connection Succeeded");
this.WriteNote("");
ConnSucceeds = true;
}
catch (Exception ex)
{
ProjectData.SetProjectError(ex);
int num = (int)Interaction.MsgBox((object)(#"Unable to connect to SQL Server:" + sqlConnection.ConnectionString + #"
Does the " + this._brandName + " Database Live on this Machine?"), MsgBoxStyle.Exclamation, (object)"SQL Connection Error");
ProjectData.ClearProjectError();
}
}
if (ConnSucceeds)
{
string filename = #"C:\Program Folder\DC_Imports\dc_raw.txt";
try
{
StreamReader s = new StreamReader(filename);
string fileContents = s.ReadToEnd();
int removeHeader = fileContents.IndexOf('\n');
string contentsNoHeader = fileContents.Substring(removeHeader);
string contentsFixed = contentsNoHeader.Replace("'", "''");
string delim = "\n";
string[] Rows = contentsFixed.Split(delim.ToCharArray());
foreach (string row in Rows)
{
string query = #"USE DBName IF (NOT EXISTS (SELECT * FROM tempdb.sys.tables WHERE name LIKE '%#DCImportTable%'))
BEGIN
CREATE TABLE #DCImportTable (Main varchar (8000));
INSERT INTO #DCImportTable (Main) VALUES ('" + row + #"');
END
ELSE
INSERT INTO #DCImportTable (Main) VALUES ('" + row + "');";
SqlCommand command = new SqlCommand(query, sqlConnection);
command.ExecuteNonQuery();
this.WriteNote(row);
}
this.WriteNote("Check Table");
this.WriteNote("");
}
catch (SqlException ex)
{
for (int i = 0; i < ex.Errors.Count; i++)
{
errorMessages.Append("Error \n" +
"Message: " + ex.Errors[i].Message + "\n");
}
this.WriteNote(errorMessages.ToString());
sqlConnection.Close();
this.WriteNote("SQL Connection Terminated");
}
}
else
{
this.WriteNote("SQL Login Incorrect");
sqlConnection.Close();
this.WriteNote("SQL Connection Terminated");
}
}
Any help would be greatly appreciated! This is probably the most complex thing I've ever tried to code and I have to use temp tables to do it.
In order to show how this might look I need to move around the pieces of this code a little bit. It gets much easier if you separate as much of this as possible into different pieces. Otherwise the logic and flow become complicated and difficult to follow. There's more than one way to do this. This is just a high-level example to show how you can break this up.
First, read the rows:
public class FileRowReader
{
public string[] ReadRows(string filename)
{
StreamReader s = new StreamReader(filename);
string fileContents = s.ReadToEnd();
int removeHeader = fileContents.IndexOf('\n');
string contentsNoHeader = fileContents.Substring(removeHeader);
string contentsFixed = contentsNoHeader.Replace("'", "''");
string delim = "\n";
return contentsFixed.Split(delim.ToCharArray());
}
}
By itself that part is much easier to understand. (I only spent a few seconds on the names. It would be good to rename them to something more descriptive.)
Then, write the rows to SQL. Again, to keep it simple, this can be separate from all of the initialization and configuration. I'm not addressing the issue of whether it makes sense or not to execute a single command for each row, or the risk of SQL injection that happens when we build SQL by concatenating strings. The SQL should be modified so that it uses parameters.
public class SqlRowWriter
{
private readonly string _connectionString;
public SqlRowWriter(string connectionString)
{
_connectionString = connectionString;
}
public void WriteRows(string[] rows)
{
using (var connection = new SqlConnection(_connectionString))
{
connection.Open();
foreach (var row in rows)
{
// Write each row
}
}
}
}
One of the benefits of this approach is that unrelated parts of the code are decoupled. The part that reads from a text file doesn't know what you're going to do with the text. It doesn't care. Its job it just to read and give you the text.
The part that writes to SQL doesn't care where the data came from. It just takes whatever data you give it and operates on it. Each part is simpler and easier to understand because it's doing one thing, not trying to juggle a few at once.
Another benefit is that both classes are now easier to test. If you want to verify that your reader works, test that. If you want to test that your writer works, test that. You don't have to test them both at the same time.
I didn't include the part that collects and writes individual SQL errors. You could add it back in. The reason I left it out is that it would be really difficult to figure out what to do next if you try to write 50 rows but 25 of them throw exceptions.
What I particularly recommend is trying to send all of the rows to SQL at once instead of sending it one row at a time. You would do that using a stored procedure on the SQL Server end (instead of passing the whole SQL command as a string) and then passing all of the rows as a table-valued parameter. It's like sending rows of data in one execution instead of multiple executions. That's a slightly different answer, but by keeping the SQL part of this separate it becomes much easier to modify your code later to make that change. Here's an example.
Sending them all at once also means that you can either do one bulk insert or do separate inserts in a transaction. That way the whole thing succeeds or the whole thing fails. You're not left in a weird state where part of your data has been inserted.
Now the method you started with doesn't have to do all this stuff. It can just use these two other classes:
private void ImportToTempTable()
{
// I don't know what this does. I just left it in because it's
// part of your original code.
this.GetProgramInfoForSQLQuery();
this.GetInstallFolder();
string config = this._InstallFolder + #"\" + this._Version + #"\" + this._brandName + #".exe.config";
GetInstanceName(config);
string filename = "<whatever your file name is>";
string connStr = "<proprietary conn string parameters>";
var rowReader = new FileRowReader();
var rows = rowReader.ReadRows(filename);
var writer = new SqlRowWriter(connStr);
writer.WriteRows(rows);
}
What if you read all of the rows but then you wanted to inspect them all to find out ahead of time if there was anything in them that you didn't want to write to SQL, like blank rows or headers? Now it's really easy to insert another class or method that takes an array of strings and filters them or cleans them up somehow.
I haven't included any exception handling yet. That's also easier if you do it in the outer method. If you want to handle all exceptions from reading or writing the same, you could just do this:
private void ImportToTempTable()
{
// I don't know what this does. I just left it in because it's
// part of your original code.
this.GetProgramInfoForSQLQuery();
this.GetInstallFolder();
string config = this._InstallFolder + #"\" + this._Version + #"\" + this._brandName + #".exe.config";
GetInstanceName(config);
string filename = "<whatever your file name is>";
string connStr = "<proprietary conn string parameters>";
try
{
var rowReader = new FileRowReader();
var rows = rowReader.ReadRows(filename);
var writer = new SqlRowWriter(connStr);
writer.WriteRows(rows);
}
catch (Exception ex)
{
// log the exception
}
}
Even if you don't add more detail, it will be easy to tell if it's an exception from reading the file or from writing to SQL. If it's SQL, it will be easy to tell if it was from opening the connection or from executing a command. Chances are that will tell you everything you need. If you need more detail you could put separate try/catch blocks around different method calls, or you could catch specific exception types like SqlException or IOException. But odds are that just catching and logging will tell you everything you need to know.
Could someone help me with an Insert ? I'm trying to give insert in the DB but I have to convert the dateTimePicker pro to the right format, I do not know if it is exactly like this, I already tried google and tested it in several ways and none worked for me.
The first image is the INSERT, which is in a class.
The second image is the own button to insert.
The third image is the Database (yes I did in phpmyadmin of wamp msm pq is a very small and simple DB).
The error that happens when I try to insert is simply in that throw; Of the insert, and I think it's the question of help. Thank you to anyone who can help.
i just cant format the "Datav".
public void inserir_produtov()
{
try
{
string inserir1 = "INSERT INTO venda VALUES (null, '" + Datav.ToString("yyyy-MM-dd HH:mm:ss") + "', '" +
Id + "','" +
Quantidadev + "','" +
Valorv.ToString().Replace(',', '.') + "');";
bancodedados1.ExecutarComandos(inserir1);
}
catch
{
throw;
}
}
Here is the Button to insert:
private void btngravar1_Click(object sender, EventArgs e)
{
vendas.Id = int.Parse(txtidproduto.Text);
vendas.Quantidadev = int.Parse(txtquantidade2.Text);
vendas.Valorv = double.Parse(txtvalor.Text);
vendas.Datav = DateTime.Parse(dateTimePicker1.Text);
if (txtidvenda.Text == "")
{
vendas.inserir_produtov();
Limpar();
MessageBox.Show("Dados inseridos com sucesso.");
}
Use parameters instead so you don't need to format. This will also help prevent SQL injection too.
Have a look at this link:
SQL Insert Query Using C#
I created a small application that queries a db2 database and returns information. I created a windows form that accepts input and returns the information from the query. My closing statement is a:
finally
{
conn.close();
}
I was curious -- does the connection (conn) actually close when I hit the little red box on the form? I searched the other questions here and the web but could not really find a definitive answer.
Here's the full try-catch-finally block (with some info obfuscated --> *****):
`try
{
conn.Open();
string queryString = String.Format("SELECT * " +
"FROM ***** " +
"WHERE USERPRF LIKE '%{0}%' " +
"ORDER BY TIMESTMP DESC " +
"FETCH FIRST 1 ROWS ONLY", userNameInput);
using (OdbcCommand com = new OdbcCommand(queryString, conn))
{
using (OdbcDataReader reader = com.ExecuteReader())
{
if (reader.Read())
{
string ***** = reader["*****"].ToString();
string ***** = reader["*****"].ToString();
string user = reader["USERPRF"].ToString();
string timeStamp = reader["TIMESTMP"].ToString();
listBox1.Items.Clear();
listBox1.Items.Add("Username: " + user);
listBox1.Items.Add("*****" + *****);
listBox1.Items.Add("*****: " + *****);
listBox1.Items.Add("Last Changed: " + timeStamp);
}
else
{
listBox1.Items.Clear();
listBox1.Items.Add("There was no data to return! Try again.");
}
}
}
}
catch (Exception ex)
{
string errorMessage = ex.Message;
}
finally
{
conn.Close();
}`
If the connection is owned by the application, then yes - it should close.
It is generally bad practice to leave a connection open for long duration's as it constitutes a security risk. (Someone could inject code into your application to reuse the open the connection, to do dodgy stuff)
using (SqlConnection cn = new SqlConnection(strConnectString))
{
// Stuff
}
I would make sure that you handle the onClosing event of your windows form, and tell it to dispose of the SqlConnection explicitly, or at least attempt to do so.
Better safe than sorry.
Note - I have heard some talk that SqlConnections can be shared in the SQLConnectionPool. If this is the case, you can modify your connection string to disable or enable ConnectionPooling.
I have been working on a project related to database (.mdf). I have created some windows forms in visual studio using C#. Basically these forms work together to store, update and delete data from the Service Based Database i created.
The problem is when i build the project, it builds fine, no errors. It inserts a data provided from textboxes to the datagridview too as intended. But as soon as i stop the current debugging, and then rerun it again, all the data provided previously is lost from the datagridview!!
I cant understand why this is happening. anyone please help me. Im totally new to this stuff.. a bit of guidance would be heartily appreciated.
when i had previously used MySQL for the same purpose, the updated data would be permanently stored to the database, but since i migrated from the MySQL to SQL Server's Service Based Database, i get such confusing error.
......
void loadData()
{
SqlConnection con = new SqlConnection(System.Configuration.ConfigurationManager.ConnectionStrings["baikalpik_bidhut_sewaConnectionString"].ToString());
SqlCommand cmd = new SqlCommand("SELECT SNo,Customer_ID, Citizenship_No, Name, Subscription_Date, Phone_No, Location,Locality,Bulbs,Deposit,Monthly_Charge FROM customerinformation;", con);
try
{
SqlDataAdapter adp = new SqlDataAdapter();
adp.SelectCommand = cmd;
DataTable dt = new DataTable();
adp.Fill(dt);
dataGridViewCustomerInformation.DataSource = dt;
}
catch (Exception ex)
{
MessageBox.Show(ex.Message);
}
}
private void buttonAdd_Click(object sender, EventArgs e)
{
try
{
float m_chrg = Convert.ToInt64(textBoxBulbs.Text)*500;
SqlConnection con = new SqlConnection(System.Configuration.ConfigurationManager.ConnectionStrings["baikalpik_bidhut_sewaConnectionString"].ToString());
SqlCommand cmd = new SqlCommand("INSERT INTO customerinformation(SNo,Customer_ID,Citizenship_No,Name,Subscription_Date,Location,Locality,Bulbs,Deposit,Phone_No,Monthly_Charge) values('" + textBoxSNo.Text + "','" + textBoxCustomerID.Text + "','" + textBoxCitizenshipNumber.Text + "','" + textBoxName.Text + "','" + textBoxSubscriptionDate.Text + "','" + textBoxLocation.Text + "','" + textBoxLocality.Text + "','" + textBoxBulbs.Text + "','" + textBoxDeposit.Text + "','" + textBoxPhoneNumber.Text + "','" + m_chrg + "')", con);
con.Open();
SqlDataReader reader = cmd.ExecuteReader();
dt = new DataTable();
dt.Load(reader);
con.Close();
dataGridViewCustomerInformation.DataSource = dt;
loadData();
MessageBox.Show("Entry Added!");
fillListbox();
textBoxSNo.Clear();
textBoxBulbs.Clear();
textBoxCitizenshipNumber.Clear();
textBoxCustomerID.Clear();
textBoxDeposit.Clear();
textBoxLocality.Clear();
textBoxLocation.Clear();
textBoxPhoneNumber.Clear();
textBoxName.Clear();
textBoxSubscriptionDate.Clear();
}
catch(Exception ex)
{
MessageBox.Show(ex.Message);
}
}
If you have your MDF listed in your project files then the property Copy To Output Directory handles how your MDF file is copied to the output directory (BIN\DEBUG or BIN\RELEASE).
If this property is set to Copy Always, then when you run your program a fresh copy of your database file is copied from the project folder to the output effectively destroying every data that you have inserted, modified, deleted in the previous run of your program.
The best solution to this dilemma is to add the MDF file as a permanent database inside your current install of Sql Server and adjust your connection string. And of course set the property to Copy Never
In alternative you could set to Copy if newer. This will allow to change the database schema inside the Server Explorer and let the Visual Studio IDE copy the new structure only after you have made changes.
UPDATE BUT IMPORTANT Not related to your actual question, but your insert query is a serious problem. Do not use string concatenation to build a sql command text. Particularly if the partial text comes from user input. You could face syntax errors (if someone places a single quote inside a text box) or worst, a Sql Injection problem (see here for a funny explanation)
To find good examples of Insert search for 'parametrized query'
As I can read from SQLite FAQ it supports multiple processes reading (SELECT) and only one process writing (INSERT, UPDATE, DELETE) database at any moment in time:
SQLite uses reader/writer locks to control access to the database.
When any process wants to write, it must lock the entire database file
for the duration of its update. But that normally only takes a few
milliseconds. Other processes just wait on the writer to finish then
continue about their business
I'm using System.Data.SQLite adapter via c#.
Could someone expalin me plz, how exactly this process is going on?
Will this process work automatically and writing SQLiteCommand will simply wait if there is another writing SQLiteCommand already executing over the same database?
Or maybe it will throw an exception? What kind of it?
Sorry but I found no information about this mechanics :)
Thank you.
UPDATE:
I've found post saying that exception will be raised with a specific errorcode
Is that statement correct?
I've investigated it by myself:
I created a sample SQLite database c:\123.db with one table Categories containing two fields: ID (uniqueidentifier) and Name (nvarchar).
I then wrote some multi-thread code to emulate multiple write access to the database (don't forget to add a System.Data.SQLite reference to your project if you use this code):
using System;
using System.Data.SQLite;
using System.Threading.Tasks;
namespace SQLiteTest
{
class Program
{
static void Main(string[] args)
{
var tasks = new Task[100];
for (int i = 0; i < 100; i++)
{
tasks[i] = new Task(new Program().WriteToDB);
tasks[i].Start();
}
foreach (var task in tasks)
task.Wait();
}
public void WriteToDB()
{
try
{
using (SQLiteConnection myconnection = new SQLiteConnection(#"Data Source=c:\123.db"))
{
myconnection.Open();
using (SQLiteTransaction mytransaction = myconnection.BeginTransaction())
{
using (SQLiteCommand mycommand = new SQLiteCommand(myconnection))
{
Guid id = Guid.NewGuid();
mycommand.CommandText = "INSERT INTO Categories(ID, Name) VALUES ('" + id.ToString() + "', '111')";
mycommand.ExecuteNonQuery();
mycommand.CommandText = "UPDATE Categories SET Name='222' WHERE ID='" + id.ToString() + "'";
mycommand.ExecuteNonQuery();
mycommand.CommandText = "DELETE FROM Categories WHERE ID='" + id.ToString() + "'";
mycommand.ExecuteNonQuery();
}
mytransaction.Commit();
}
}
}
catch (SQLiteException ex)
{
if (ex.ReturnCode == SQLiteErrorCode.Busy)
Console.WriteLine("Database is locked by another process!");
}
}
}
}
The result on my Core2Duo E7500 is that Exception is never raised!
Looks like SQLite is optimised enough for my needs (locking/unlocking is really fast and normally only takes a few milliseconds as SQLite FAQ tells us) - Great!
Note that there is no need to retrieve an integer ErrorCode for an SQLiteException - you can use a special enum ReturnCode field instead. All codes are described here.
Hope this information will help somebody.