I'm developing a front end for retro gaming. Upon boot I'm parsing a lot of xml files from another application so I end up with a List of "systems" each containing a list of "games". Parsing the xml data is really fast, but writing all this to the sqlite database is not. Currently it takes ~25 seconds (20.000 records) which may not be too bad, but hopefully I can get some ideas on how to make it even faster.
Ideally I want to be able to compare my object containing all the data with the sqlite database. If anything has changed in the xml files/parsed object the data should be updated or deleted from the database. Is there a better way of solving this then doing a initial import and then dump all of the database and compare it to the object? Basically the whole import code backwards...
This is my current code:
public static void PopulateDatabase(List<RetroBoxSystem> systems)
{
using (SQLiteConnection con = new SQLiteConnection("Data Source=RetroBox.db;Version=3;"))
{
con.Open();
using (SQLiteTransaction tr = con.BeginTransaction())
{
using (SQLiteCommand cmd = con.CreateCommand())
{
foreach (var system in systems)
{
cmd.CommandText = #"INSERT OR IGNORE INTO systems(system_id, name)
VALUES ((SELECT system_id FROM systems WHERE name = #name), #name)";
cmd.Parameters.Add(new SQLiteParameter("#name", system.Name));
cmd.ExecuteNonQuery();
}
}
using (SQLiteCommand cmd = con.CreateCommand())
{
cmd.CommandText = #"INSERT OR IGNORE INTO games(game_id, system_id, name, description, cloneof, manufacturer, genre, rating, year)
VALUES ((SELECT game_id FROM games WHERE name = #name), (SELECT system_id FROM systems WHERE name = #system), #name, #description, #cloneof, #manufacturer, #genre, #rating, #year)";
foreach (var system in systems)
{
foreach (var g in system.GameList)
{
cmd.Parameters.Add(new SQLiteParameter("#system", system.Name));
cmd.Parameters.Add(new SQLiteParameter("#name", g.Name));
cmd.Parameters.Add(new SQLiteParameter("#description", g.Description));
cmd.Parameters.Add(new SQLiteParameter("#cloneof", g.CloneOf));
cmd.Parameters.Add(new SQLiteParameter("#manufacturer", g.Manufacturer));
cmd.Parameters.Add(new SQLiteParameter("#genre", g.Genre));
cmd.Parameters.Add(new SQLiteParameter("#rating", g.Rating));
cmd.Parameters.Add(new SQLiteParameter("#year", g.Year));
cmd.ExecuteNonQuery();
}
}
}
tr.Commit();
}
}
}
Your best bet would be the Entity Framework for comparing objects without coding everything yourself, however depending on your project type you might not have access to it (such as Windows Phone projects).
Your insert query seems pretty optimized, but maybe you can make it faster using async inserts with ConfigureAwait(false).
More details about ConfigureAwait here: How to massively improve SQLite Performance (using SqlWinRT)
Related
How would I delete a row from a sql database, either with stored procedures or without, right now I have tried without, using a button press.
This is what I have so far, _memberid has been sent over from a differnt form from the database(For context).
private void btnDelete_Click(object sender, EventArgs e)
{
SqlCommand cmd = new SqlCommand();
cmd.Connection = Lib.SqlConnection;
cmd.CommandType = CommandType.Text;
cmd.CommandText = "Delete * From Members where MemberId = " + _memberId;
SqlDataAdapter adapter = new SqlDataAdapter();
adapter.DeleteCommand = cmd;
adapter.Fill(MembersDataTable); // Im fairly sure this is incorrect but i used it from old code
DialogResult = DialogResult.OK;
}
If you're trying to do a simple ADO.Net-based delete, then it would be somehting like his:
private void DeleteById(int memberId)
{
// or pull the connString from config somewhere
const string connectionString = "[your connection string]";
using (var connection = new SqlConnection(connectionString))
{
connection.Open();
using (var command = new SqlCommand("DELETE FROM Members WHERE MemberId = #memberId", connection))
{
command.Parameters.AddWithValue("#memberId", memberId);
command.ExecuteNonQuery();
}
}
Use parameter to prevent SQL injection.
There are essentially three main things I'm seeing...
One
You don't need the * in the query. DELETE affects the whole row, so there's no need to specify columns. So just something like:
DELETE FROM SomeTable WHERE SomeColumn = 123
Two
There's no need for a SqlDataAdapter here, all you need to do is execute the query. For example:
cmd.ExecuteNonQuery();
The "non query" is basically a SQL command which doesn't query data for results. Inserts, updates, and deletes are generally "non queries" in this context. What it would return is simply the number of rows affected, which you can use to double-check that it matches what you expect if necessary.
Three
Don't do this:
cmd.CommandText = "Delete From Members where MemberId = " + _memberId;
This kind of string concatenation leads to SQL injection. While it looks intuitively like you're using _memberId as a query value, technically you're using it as executable code. It's less likely (though not impossible) to be a problem for numeric values, but it's a huge problem for string values because it means the user can send you any string and you'll execute it as code.
Instead, use query parameters. For example, you might do something like this:
cmd.CommandText = "Delete From Members where MemberId = #memberId";
cmd.Parameters.Add("#memberId", SqlDbType.Int);
cmd.Parameters["#memberId"].Value = _memberId;
This tells the database engine itself that the value is a value and not part of the executing query, and the database engine knows how to safely handle values.
You could use a DataAdapter, but since you aren't using a datatable, it's just easier to do it without like this:
var sql = "DELETE FROM Members WHERE MemberId=#MemberId";
using(var cmd = new SqlCommand(sql, Lib.SqlConnection))
{
cmd.Connection.Open();
cmd.Parameters.Add("#MemberId",SqlDbType.Int).Value = _memberId;
cmd.ExecuteNonQuery();
}
And if you are using Dapper, you can do this:
Lib.SqlConnection.Execute("DELETE FROM Members WHERE MemberId=#MemberId", new {MemberId=_memberId});
If you are still using DataTables, I would highly recommend you look into using this (or something like this) to simplify your database accesses. It'll make CRUD logic on a database a breeze, and your code will me a lot more maintainable because you can get rid of all the odd needs to do casting, boxing/unboxing, and reduce the chances of runtime bugs because of the use of magic strings that happens so often with DataTables (column names). Once you start working with POCO classes, you'll hate having to use DataTables. That said, there are a few places where DataTables are a better solution (unknown data structures, etc), but those are usually pretty rare.
In my orders table I have PK INVOICE ID for each order item within the invoice
and a FK CUSTOMER ID I want to keep the CUSTOMER ID sequence same for same item within below code is doing the job but I just want to know if I am doing it with right way or there is a better way to do it
string connstr = "Data Source=JDT; User Id=admin; password=admin;";
string seqcmdtxt = #"SELECT CUSTOMER_ID_SEQ.NEXTVAL AS CUSTSEQ FROM DUAL";
string insertcmdtxt = #"INSERT INTO ORDERS (ORDER_ID,
CUSTOMER_ID,
PRODUCT_ID,
QUANTITY,
UNIT_PRICE,
ORDER_STATUS,
NOTES,
CREATED_BY,
CREATED_ON,
UPDATE_BY,
UPDATE_ON)
VALUES
(ORDER_ID_SEQ.NEXTVAL, --ORDER_ID
:TB_INVOICE_ID, --CUSTOMER_ID
:DGV_PRODUCT_DESC, --PRODUCT_ID
:DGV_QUANTITY, --QUANTITY
:DGV_UNIT_PRICE, --UNIT_PRICE
NULL, --ORDER_STATUS
:DGV_NOTES, --NOTES
'SYSTEM', --CREATED_BY
SYSDATE, --CREATED_ON
NULL, --UPDATE_BY
NULL) --UPDATE_ON
RETURNING ORDER_ID INTO :OUT_ORDER_ID"; //~ Returning VAULES ~//
using (OracleConnection conn = new OracleConnection(connstr))
using (OracleCommand cmd = new OracleCommand(insertcmdtxt, conn))
{
try
{
conn.Open();
cmd.CommandText = seqcmdtxt;
OracleDataReader reader = cmd.ExecuteReader();
while (reader.Read())
{
TB_INVOICE_ID.Text = reader["CUSTSEQ"].ToString();
}
cmd.CommandText = insertcmdtxt;
for (int i = 0; i < DGV_INVOICE.Rows.Count; i++)
{
//~ refreshing parameters ~//
cmd.Parameters.Clear();
cmd.Parameters.Add(new OracleParameter("TB_INVOICE_ID", TB_INVOICE_ID.Text));
cmd.Parameters.Add(new OracleParameter("DGV_PRODUCT_DESC", DGV_INVOICE.Rows[i].Cells[1].Value));
cmd.Parameters.Add(new OracleParameter("DGV_QUANTITY", DGV_INVOICE.Rows[i].Cells[2].Value));
cmd.Parameters.Add(new OracleParameter("DGV_UNIT_PRICE", DGV_INVOICE.Rows[i].Cells[3].Value));
cmd.Parameters.Add(new OracleParameter("DGV_NOTES", DGV_INVOICE.Rows[i].Cells[5].Value));
cmd.Parameters.Add(":OUT_ORDER_ID", OracleDbType.Decimal, ParameterDirection.Output);
//cmd.Parameters.Add(":OUT_CUSTOMER_ID", OracleDbType.Decimal, ParameterDirection.Output);
cmd.ExecuteNonQuery();
}
TB_INVOICE_ID.Text = (cmd.Parameters[":OUT_ORDER_ID"].Value).ToString();
//TB_NOTES.Text = (cmd.Parameters[":OUT_CUSTOMER_ID"].Value).ToString();
}
catch (Exception EX)
{ MessageBox.Show(EX.Message, "error msg", MessageBoxButtons.OK, MessageBoxIcon.Error); }
}
A few pointers.
Consider separating the code that gets the Customer ID from code that adds the invoice rows
int GetCustomerId();
void AddInvoice(int customerId);
You seem to be storing the CUSTSEQ in, what looks like, a global variable (input box). I would considering storing that in a local variable
string customerSequenceId = reader["CUSTSEQ"].ToString();
cmd.Parameters.Add(new OracleParameter("TB_INVOICE_ID", customerSequenceId));
Typically you don't need to be too concerned with the order in which records in the database are stored. (It makes some difference with regard to performance, but from the perspective of your application it doesn't matter.) What matters is the order in which records are sorted when you retrieve them. And that can vary. In one scenario you might want them ordered by date, in another by quantity, etc.
So what matters first is your 'ORDER BY' clause in your SQL query. Even though a database might appear to return records in some predictable sequence, it's important to always specify the order unless it absolutely doesn't matter.
Then if you build collections like List<Order> from the results of that query in your application you'll want to also ensure that if the sequence matters, you also specify an .OrderBy or .OrderByDescending in your LINQ expressions. Many operations, like .Select, will always return records in their original sequence, so if you know the sort order when you start you don't need to specify it again and again with every operation. But if you're selecting distinct groups or joining, or passing collections from one function to another so that the order may be uncertain, then you may need to sort them again if the sort order matters.
I have a stored procedure that looks like that:
InsertItem: INSERT INTO (IN itemId INT, name TEXT);
Is there a way I could execute a bulk of it?
like instead of executing something like that:
using (MySqlConnection connection = new MySqlConnection(_connectionString))
{
connection.Open();
foreach (Item item in GetItems())
{
using (MySqlCommand command = new MySqlCommand("InsertItem", connection))
{
command.CommandType = CommandType.StoredProcedure;
command.Parameters.AddWithValue("#itemId", item.ItemId);
command.Parameters.AddWithValue("#name", item.Name);
command.ExecuteNonQuery();
}
}
}
I'm trying to achieve code looking like that without successing:
using (MySqlConnection connection = new MySqlConnection(_connectionString))
{
connection.Open();
using (MySqlCommandBulk command = new MySqlCommand("InsertItem", connection))
{
command.CommandType = CommandType.StoredProcedure;
for (Item item in GetItems())
{
MySqlCommandBulkItem bulkItem = new MySqlCommandBulkItem();
bulkItem["itemId"] = item.ItemId;
bulkItem["name"] = item.Name;
command.BulkItems.Add(bulkItem);
}
command.Execute();
}
}
My point is that the command will send all of the data at once, and will not send each query alone.
Any ideas?
The Oracle connector for the Dotnet framework allows the use of arrays in place of scalars on parameters. But the MySQL connector doesn't.
There are two ways to accelerate bulk loads in MySQL.
One of them applies to InnoDB tables but doesn't help with MyISAM tables. Start a transaction. Then, after every few hundred rows, COMMIT it and start another one. That will commit your table inserts in bunches, which is faster than autocommiting them individually.
The other is to use MySQL's LOAD DATA INFILE command to slurp up a data file and bulk-insert it into the database. This is very fast, but you have to be diligent about formatting your file correctly.
I am writing a small app using WPF/C# & SQLITE.
One of the functions inserts a record containing two date/time values into a table.
I want to know if there is a proper way to do this (to ensure that the date/month fields are stored consistently).
I have created an INSERT query that uses parameters alongside date variables (clsVariables.DatActivityEnd = DateTime.Now;).
String cntnStr = ClsVariables.StrDb;
var connection = new SQLiteConnection(cntnStr);
connection.Open();
using (SQLiteCommand cmd = connection.CreateCommand())
{
cmd.CommandText =
"INSERT INTO tblActivity ([Activity_Category], [Activity_Category_Sub], [Activity_Start], [Activity_End], [Activity_Duration]) VALUES (#ActivityCategory, #ActivityCategorySub, #ActivityStart, #ActivityEnd, #ActivityDuration);";
cmd.Parameters.Add(new SQLiteParameter("#ActivityCategory", ClsVariables.StrActivityCurrent));
cmd.Parameters.Add(new SQLiteParameter("#ActivityCategorySub", ClsVariables.StrActivitySubCurrent));
cmd.Parameters.Add(new SQLiteParameter("#ActivityStart", ClsVariables.DatActivityStart.ToString()));
cmd.Parameters.Add(new SQLiteParameter("#ActivityEnd", ClsVariables.DatActivityEnd.ToString()));
cmd.Parameters.Add(new SQLiteParameter("#ActivityDuration", ClsVariables.DblActivityDuration));
cmd.ExecuteNonQuery();
}
connection.Close();
Is there anything else I should do - Thank you?
You have to use sql formatted strings:
string sqlFormattedDate = myDateTime.ToString("yyyy-MM-dd HH:mm:ss");
Update:
Today I would prefer AddWithValues with Type Safety and without any Conversion:
cmd.Parameters.AddWithValue("#ActivityStart", ClsVariables.DatActivityStart);
I'm using query, where the piece is:
...where code in ('va1','var2'...')
I have about 50k of this codes.
It was working when I has 30k codes, but know I get:
The query processor ran out of internal resources and could not produce a query plan. This is a rare event and only expected for extremely complex queries or queries that reference a very large number of tables or partition
I think that problem is related with IN...
So now I'm planning use foreach(string code in codes)
...where code =code
Is it good Idea ??
I suggest you create a temporary table containing all the codes you want to match, and join against that instead.
However, we haven't really seen enough of your code to comment for sure.
First, create a temp table, call it #tmp0, with just one column. Then:
SqlConnection conn = new SqlConnexion(connection_string);
SqlCommand cmd = new SqlCommand("INSERT INTO #tmp0 (code) VALUE (#code)", conn);
conn.Open();
foreach (string s in bigListOfCodes)
{
cmd.Parameters.Clear();
cmd.Parameters.AddWithValue("#code", s);
cmd.ExecuteNonQuery();
}
cmd = new SqlCommand("SELECT * FROM tbl " +
"WHERE code IN (SELECT CODE FROM #tmp0)", conn);
SqlDataReader rs = cmd.ExecuteReader();
while (rs.Read())
{
/* do stuff */
}
cmd = new SqlCommand("DROP TABLE #tmp0", conn);
cmd.ExecuteNonQuery();
conn.Close();
I know this seems like a lot of work for the server, but it's really pretty fast.
I'm not sure where you got those 50k values, but if it is from another query, just JOIN to this table and get all the data at one time from one query.