Edit 3:
I guess my issue is resolved for the moment... I changed both my service and test app to run as the SYSTEM account instead of the NetworkService account. It remains to be seen if the benefits of changing the user account will persist, or if it will only be temporary.
Original Question:
I've noticed that my small 224kB SQLite DB is very slow to open in my C# application, taking anywhere from some small number of ms, to 1.5 seconds or more. Below is my code, with all the extra debugging statements I've added this afternoon. I've narrowed it down to the call to cnn.Open(); as shown in the logs here:
2014-03-27 15:05:39,864 DEBUG - Creating SQLiteConnection...
2014-03-27 15:05:39,927 DEBUG - SQLiteConnection Created!
2014-03-27 15:05:39,927 DEBUG - SQLiteConnection Opening...
2014-03-27 15:05:41,627 DEBUG - SQLiteConnection Opened!
2014-03-27 15:05:41,627 DEBUG - SQLiteCommand Creating...
2014-03-27 15:05:41,627 DEBUG - SQLiteCommand Created!
2014-03-27 15:05:41,627 DEBUG - SQLiteCommand executing reader...
2014-03-27 15:05:41,658 DEBUG - SQLiteCommand executed reader!
2014-03-27 15:05:41,658 DEBUG - DataTable Loading...
2014-03-27 15:05:41,767 DEBUG - DataTable Loaded!
As you can see, in this instance it took 1.7 SECONDS to open the connection. I've tried repeating this, and cannot predict whether subsequent connections will open nearly immediately, or be delayed like this.
I've considered using some form of connection pooling, but is it worthwhile to pursue that for a single-instance single-threaded application? Right now, I'm creating an instance of my SQLiteDatabase class, and calling the below function for each of my queries.
public DataTable GetDataTable(string sql)
{
DataTable dt = new DataTable();
try
{
Logging.LogDebug("Creating SQLiteConnection...");
using (SQLiteConnection cnn = new SQLiteConnection(dbConnection))
{
Logging.LogDebug("SQLiteConnection Created!");
Logging.LogDebug("SQLiteConnection Opening...");
cnn.Open();
Logging.LogDebug("SQLiteConnection Opened!");
Logging.LogDebug("SQLiteCommand Creating...");
using (SQLiteCommand mycommand = new SQLiteCommand(cnn))
{
Logging.LogDebug("SQLiteCommand Created!");
mycommand.CommandText = sql;
Logging.LogDebug("SQLiteCommand executing reader...");
using (SQLiteDataReader reader = mycommand.ExecuteReader())
{
Logging.LogDebug("SQLiteCommand executed reader!");
Logging.LogDebug("DataTable Loading...");
dt.Load(reader);
Logging.LogDebug("DataTable Loaded!");
reader.Close();
}
}
cnn.Close();
}
}
catch (Exception e)
{
throw new Exception(e.Message);
}
return dt;
}
Edit:
Sure, dbConnection is the connection string, set by the following function. inputFile is just the string path of the filename to open.
public SqLiteDatabase(String inputFile)
{
dbConnection = String.Format("Data Source={0}", inputFile);
}
And at this point, I think sql is irrelevant, as it's not making it to that point when the cnn.Open() stalls.
Edit 2:
Ok, I've done some more testing. Running the testing locally, it completes a 1000 iteration loop in ~5 seconds, for about 5ms per call to cnn.Open(). Running the test from the same windows installer that I did on my local PC, it completes in ~25 minutes, averaging 1468ms per call to cnn.Open().
I made a small test program that only calls the TestOpenConn() function from the service program (same exact code that is running in the Windows service), running against a copy of the file located in a test directory. Running this on the server or my local PC results in acceptable performance (1.95ms per call on the server, 4ms per call on my local PC):
namespace EGC_Timing_Test
{
class Program
{
static void Main(string[] args)
{
Logging.Init("log4net.xml", "test.log");
var db = new SqLiteDatabase("config.sqlite");
db.TestOpenConn();
}
}
}
Here's the test function:
public void TestOpenConn()
{
// TODO: Remove this after testing loop of opening / closing SQLite DB repeatedly:
const int iterations = 1000;
Logging.LogDebug(String.Format("Running TestOpenConn for {0} opens...", iterations));
var startTime = DateTime.Now;
for (var i = 0; i < iterations; i++)
{
using (SQLiteConnection cnn = new SQLiteConnection(dbConnection))
{
Logging.LogDebug(String.Format("SQLiteConnection Opening, iteration {0} of {1}...", i, iterations));
var startTimeInner = DateTime.Now;
cnn.Open();
var endTimeInner = DateTime.Now;
var diffTimeInner = endTimeInner - startTimeInner;
Logging.LogDebug(String.Format("SQLiteConnection Opened in {0}ms!", diffTimeInner.TotalMilliseconds));
cnn.Close();
}
}
var endTime = DateTime.Now;
var diffTime = endTime - startTime;
Logging.LogDebug(String.Format("Done running TestOpenConn for {0} opens!", iterations));
Logging.LogInfo(String.Format("{0} iterations total:\t{1}", iterations, diffTime));
Logging.LogInfo(String.Format("{0} iterations average:\t{1}ms", iterations, diffTime.TotalMilliseconds/iterations));
}
I guess my issue is resolved for the moment... I changed both my service and test app to run as the SYSTEM account instead of the NetworkService account. It remains to be seen if the benefits of changing the user account will persist, or if it will only be temporary.
I'm assuming you're using the open source System.Data.SQLite library.
If that's the case, it's easy to see through the Visual Studio Performance Profiler that the Open method of the SQLiteConnection class has some serious performance issues.
Also, have a lookthrough the source code for this class here: https://system.data.sqlite.org/index.html/artifact/97648754af51ffd6
There's an awful lot of disk access being made to read XML configuration and Windows environment variable(s).
My suggestion is to try and call Open() as seldom as possible, and try and keep a reference to this open SQLiteConnection object around in memory.
A performance ticket is raised with SQLite Forum
Having had the same problem, I was looking into this and it seems to be related to permissions on the file or it's parent folders, who created it, and/or how it was created. In my case, the SQLite database file was being created by a script run as a regular user, and then an IIS-hosted service would access the file under a different domain service account.
Every time the service opened a connection, it took over 1.5 seconds, but otherwise operated correctly (it could eventually access the file). A stand-alone program running as the regular user could open a connection to the same file in the same place in a few milliseconds.
Analysis of a procmon trace revealed that in the case of the service, we were getting several ACCESS DENIED logs on the file over the course of about 1.5 seconds, that were not present in the trace when running as the regular user.
Not sure what's going on there. The service worked fine and was able to eventually query the data in the file, albeit slowly.
When we made the service account the owner of the parent folder of the file, and gave him write permission, the ACCESS DENIED logs disappeared and the service operated at full speed.
You can add "Modify" permissions of appropriate user to folder with your database.
Right Click on folder > Properties > Security > Edit > Add (I added IIS_Users) > Select "Modify" checkbox > OK
Related
I have been searching over to find if there is any easy way to get the Execution time of a ADO.NET command object.
I know i can manually do a StopWatch start and stop. But wanted to if there are any easy way to do it in ADO.NET
There is a way, but using SqlConnection, not command object. Example:
using (var c = new SqlConnection(connectionString)) {
// important
c.StatisticsEnabled = true;
c.Open();
using (var cmd = new SqlCommand("select * from Error", c)) {
cmd.ExecuteReader().Dispose();
}
var stats = c.RetrieveStatistics();
var firstCommandExecutionTimeInMs = (long) stats["ExecutionTime"];
// reset for next command
c.ResetStatistics();
using (var cmd = new SqlCommand("select * from Code", c))
{
cmd.ExecuteReader().Dispose();
}
stats = c.RetrieveStatistics();
var secondCommandExecutionTimeInMs = (long)stats["ExecutionTime"];
}
Here you can find what other values are contained inside dictionary returned by RetrieveStatistics.
Note that those values represent client-side statistics (basically internals of ADO.NET measure them), but seems you asked for analog of Stopwatch - I think that's fine.
The approach from the answer of #Evk is very interesting and smart: it's working client side and one of the main key of such statistics is in fact NetworkServerTime, which
Returns the cumulative amount of time (in milliseconds) that the
provider spent waiting for replies from the server once the
application has started using the provider and has enabled statistics.
so it includes the network time from the DB server to the ADO NET client.
An alternative, more DB server oriented, would be running SET STATISTICS TIME ON and then retrieve the InfoMessage.
A draft of the code of the delegate (where I'm simply writing to the debug console, but you may want to replace it with a StringBuilder Append)
internal static void TrackInfo(object sender, SqlInfoMessageEventArgs e)
{
Debug.WriteLine(e.Message);
foreach (var element in e.Errors) {
Debug.WriteLine(element.ToString());
}
}
and usage
conn.InfoMessage += TrackInfo;
using (var cmd = new SqlCommand(#"SET STATISTICS TIME ON", conn)) {
cmd.ExecuteNonQuery();
}
using (var cmd = new SqlCommand(yourQuery, conn)) {
var RD = cmd.ExecuteReader();
while (RD.Read()) {
// read the columns
}
}
I suggest you move to SQL Server 2016 and use the Query Store feature. This will track execution time and performance changes over time for each query you submit. Requires no changes in your application. Track all queries, including those executed inside stored procedures. Track any application, not only your own. Is available in all editions, including Express, and including the Azure SQL DB Service.
If you track on the client side, you must measure the time yourself, using a wall clock. I would add and expose performance counters and then use the performance counters infrastructure to capture and store the measurements.
As a side not, simply tracking the execution time of a batch sent to SQL Server yields very coarse performance info and is seldom actionable. Read How to analyse SQL Server performance.
I have following C# code in a console application.
Whenever I debug the application and run the query1 (which inserts a new value into the database) and then run query2 (which displays all the entries in the database), I can see the new entry I inserted clearly. However, when I close the application and check the table in the database (in Visual Studio), it is gone. I have no idea why it is not saving.
using System;
using System.Collections.Generic;
using System.Data.Entity;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using System.Data.SqlServerCe;
using System.Data;
namespace ConsoleApplication1
{
class Program
{
static void Main(string[] args)
{
try
{
string fileName = "FlowerShop.sdf";
string fileLocation = "|DataDirectory|\\";
DatabaseAccess dbAccess = new DatabaseAccess();
dbAccess.Connect(fileName, fileLocation);
Console.WriteLine("Connected to the following database:\n"+fileLocation + fileName+"\n");
string query = "Insert into Products(Name, UnitPrice, UnitsInStock) values('NewItem', 500, 90)";
string res = dbAccess.ExecuteQuery(query);
Console.WriteLine(res);
string query2 = "Select * from Products";
string res2 = dbAccess.QueryData(query2);
Console.WriteLine(res2);
Console.ReadLine();
}
catch (Exception e)
{
Console.WriteLine(e);
Console.ReadLine();
}
}
}
class DatabaseAccess
{
private SqlCeConnection _connection;
public void Connect(string fileName, string fileLocation)
{
Connect(#"Data Source=" + fileLocation + fileName);
}
public void Connect(string connectionString)
{
_connection = new SqlCeConnection(connectionString);
}
public string QueryData(string query)
{
_connection.Open();
using (SqlCeDataAdapter da = new SqlCeDataAdapter(query, _connection))
using (DataSet ds = new DataSet("Data Set"))
{
da.Fill(ds);
_connection.Close();
return ds.Tables[0].ToReadableString(); // a extension method I created
}
}
public string ExecuteQuery(string query)
{
_connection.Open();
using (SqlCeCommand c = new SqlCeCommand(query, _connection))
{
int r = c.ExecuteNonQuery();
_connection.Close();
return r.ToString();
}
}
}
EDIT: Forgot to mention that I am using SQL Server Compact Edition 4 and VS2012 Express.
It is a quite common problem. You use the |DataDirectory| substitution string. This means that, while debugging your app in the Visual Studio environment, the database used by your application is located in the subfolder BIN\DEBUG folder (or x86 variant) of your project. And this works well as you don't have any kind of error connecting to the database and making update operations.
But then, you exit the debug session and you look at your database through the Visual Studio Server Explorer (or any other suitable tool). This window has a different connection string (probably pointing to the copy of your database in the project folder). You search your tables and you don't see the changes.
Then the problem get worse. You restart VS to go hunting for the bug in your app, but you have your database file listed between your project files and the property Copy to Output directory is set to Copy Always. At this point Visual Studio obliges and copies the original database file from the project folder to the output folder (BIN\DEBUG) and thus your previous changes are lost.
Now, your application inserts/updates again the target table, you again can't find any error in your code and restart the loop again until you decide to post or search on StackOverflow.
You could stop this problem by clicking on the database file listed in your Solution Explorer and changing the property Copy To Output Directory to Copy If Newer or Never Copy. Also you could update your connectionstring in the Server Explorer to look at the working copy of your database or create a second connection. The first one still points to the database in the project folder while the second one points to the database in the BIN\DEBUG folder. In this way you could keep the original database ready for deployment purposes and schema changes, while, with the second connection you could look at the effective results of your coding efforts.
EDIT Special warning for MS-Access database users. The simple act of looking at your table changes the modified date of your database ALSO if you don't write or change anything. So the flag Copy if Newer kicks in and the database file is copied to the output directory. With Access better use Copy Never.
Committing changes / saving changes across debug sessions is a familiar topic in SQL CE forums. It is something that trips up quite a few people. I'll post links to source articles below, but I wanted to paste the answer that seems to get the best results to the most people:
You have several options to change this behavior. If your sdf file is part of the content of your project, this will affect how data is persisted. Remember that when you debug, all output of your project (including the sdf) if in the bin/debug folder.
You can decide not to include the sdf file as part of your project and manage the file location runtime.
If you are using "copy if newer", and project changes you make to the database will overwrite any runtime/debug changes.
If you are using "Do not copy", you will have to specify the location in code (as two levels above where your program is running).
If you have "Copy always", any changes made during runtime will always be overwritten
Answer Source
Here is a link to some further discussion and how to documentation.
I've got two processes with connections to the same SQL CE .sdf database file. One inserts items into a table and the other reads all the records from the table. After the insert I can confirm the rows are there with the Server Explorer but my query from the second process does not show them:
this.traceMessages.Clear();
SqlCeCommand command = new SqlCeCommand("SELECT AppName, Message, TraceId FROM Messages", this.connection);
using (var reader = command.ExecuteReader())
{
while (reader.Read())
{
this.traceMessages.Add(
new TraceMessage
{
AppName = reader.GetString("AppName"),
Message = reader.GetString("Message"),
TraceId = reader.GetString("TraceId")
});
}
}
It can generally load up correctly the first time but doesn't pick up updates, even after restarting the process. The connection string just has a simple Data Source that I've confirmed is pointing to the same file on both processes.
Anyone know why this is happening? Is there some setting I can enable to get updates from separate processes to work?
This is because unlike "traditional" databases, the data that you write is not flushed to disk immediately, it is deferred and happens some time later.
You have two choices in the writing program:
1) Add the Flush Interval parameter to your connection string and set it to 1. This will have a lag of up to a second before the data is flushed to the sdf.
2) When you call Commit, use the parameterized overload that allows you to specify CommitMode.Immediate. This will flush data to disk immediately.
I am getting a MSDTC exception in a Transaction in a C# application. The functionality is to upload one lakh (one hundred thousand) zipcode records into database tables after reading from a csv file. This operation is done in around 20 batch database operations (each batch containing 5000 records). The functionality is working fine, if I don’t use transaction.
The interesting part is that other functionalities that uses transactions are able to complete their transactions. This leads me to an assumption that the exception message is a misleading one.
Any thoughts on what could be the issue?
Exception: “Network access for Distributed Transaction Manager (MSDTC) has been disabled. Please enable DTC for network access in the security configuration for MSDTC using the Component Services Administrative tool.”
Source: System.Transactions
Inner Exception: “The transaction manager has disabled its support for remote/network transactions. (Exception from HRESULT: 0x8004D024)”
Note: There is a for loop inside the transaction. Is it causing any issue?
The actual requirement is: There are some existing zipcodes in zipcode table. Each month the administrator will upload the new zipcode csv file. The new items from csv get inserted. Zipcodes which are not available in csv (but present in database) are considered to be retired and is to be deleted. The list of retired zip codes is to be returned to the User Interface. The newly added zip codes also need to be returned.
private void ProcessZipCodes(StringBuilder dataStringToProcess, int UserID)
{
int CountOfUnchangedZipCode = 0;
string strRetiredZipCode = "";
string strNewZipCode = "";
dataStringToProcess.Remove(dataStringToProcess.Length - 1, 1);
if (dataStringToProcess.Length > 0)
{
List<string> batchDataStringList = GetDataStringInBatches(dataStringToProcess);
//TimeSpan.FromMinutes(0) - to make transaction scope as infinite.
using (TransactionScope transaction = TransactionScopeFactory.GetTransactionScope(TimeSpan.FromMinutes(0)))
{
foreach (string dataString in batchDataStringList)
{
PerformDatabaseOperation(dataString, UserID);
}
transaction.Complete();
}
}
}
private List<string> GetDataStringInBatches(StringBuilder dataStringToProcess)
{
List<string> batchDataStringList = new List<string>();
int loopCounter = 0;
string currentBatchString = string.Empty;
int numberOfRecordsinBacth = 5000;
int sizeOfTheBatch = 0;
List<string> individualEntriesList = new List<string>();
string dataString = string.Empty;
if (dataStringToProcess != null)
{
dataString = dataStringToProcess.ToString();
}
individualEntriesList.AddRange(dataString.Split(new char[] { '|' }));
for (loopCounter = 0; loopCounter < individualEntriesList.Count; loopCounter++)
{
if (String.IsNullOrEmpty(currentBatchString))
{
currentBatchString = System.Convert.ToString(individualEntriesList[loopCounter]);
}
else
{
currentBatchString = currentBatchString+"|"+System.Convert.ToString(individualEntriesList[loopCounter]);
}
sizeOfTheBatch = sizeOfTheBatch + 1;
if (sizeOfTheBatch == numberOfRecordsinBacth)
{
batchDataStringList.Add(currentBatchString);
sizeOfTheBatch = 0;
currentBatchString = String.Empty;
}
}
return batchDataStringList;
}
private void PerformDatabaseOperation(string dataStringToProcess, int UserID)
{
SqlConnection mySqlConnection = new SqlConnection("data source=myServer;initial catalog=myDB; Integrated Security=SSPI; Connection Timeout=0");
SqlCommand mySqlCommand = new SqlCommand("aspInsertUSAZipCode", mySqlConnection);
mySqlCommand.CommandType = CommandType.StoredProcedure;
mySqlCommand.Parameters.Add("#DataRows", dataStringToProcess.ToString());
mySqlCommand.Parameters.Add("#currDate", DateTime.Now);
mySqlCommand.Parameters.Add("#userID", UserID);
mySqlCommand.Parameters.Add("#CountOfUnchangedZipCode", 1000);
mySqlCommand.CommandTimeout = 0;
mySqlConnection.Open();
int numberOfRows = mySqlCommand.ExecuteNonQuery();
}
Dev Env: Visual Studion 2005
Framework: .Net 3.0
DB: SQL Server 2005
When I run the query SELECT [Size],Max_Size,Data_Space_Id,[File_Id],Type_Desc,[Name] FROM MyDB.sys.database_files WHERE data_space_id = 0 --It says the size (of log) is 128
UPDATE
We have three different databases used in our application. One for data, one for history and one for logging. When I put enlist = false in the above connectionstring, for the time being, it is working. But it is in my development environment. I am skeptic about whether it will work in production also. Any thought on potential risks?
Thanks
Lijo
When you are opening more than one connection within a TransactionScope, the running transaction will automatically be escalated to a distributed transaction. For distributed transactions to work, the MSDTC on both SQL Server and the machine running the application must be configured to allow network access. SQL Server and the local DTC communicate when running distributed transactions.
The problem in your case is most likely that MSDTC on the machine running your application does not allow network access because this is the default for workstations. To fix this do the following:
Go to "Control Panel" -> "Aministration" -> "Component Services".
Browse through the tree until you get to a node called "Local DTC" or something like that.
Right-click and choose "Properties".
Go to "Security" and make sure that you allow network access and also allow inbound and outbound communication with DTC.
Click "Ok".
You will probably be prompted to restart DTC. There seems to be a bug in the UI because even though you accept a restart of the DTC, it will not be restarted. Instead you have to restart the DTC service manually in the service manager.
BTW, remember to close the connection after use in PerformDatabaseOperation. It is good practice to put it in a using block:
using (SqlConnection mySqlConnection = new .....)
{
// Some code here...
mySqlConnection.Open();
// Some more code ...
}
Is it possible that aspInsertUSAZipCode interacts with a linked server? If it does then it will try and promote your current local transaction to a distributed transaction (with transactional integrity preserved between your server and the linked server).
It may be necessary to do this outside of a transaction, if the MSDTC on the remote server cannot be configured for distributed transactions. I think the best way to do this is to create a temporary table and then SqlBulkCopy your records into it and then execute aspInsertUSAZipCode using the temporary table on the server. You may need to use a cursor.
The purpose of the temporary table is so that if something goes wrong, the table is removed at the termination of your connection.
You are probably hitting some max amount of data in a transaction limit.
Check your event log for any msdtc errors http://technet.microsoft.com/en-us/library/cc774415(WS.10).aspx
Having 100,000 rows in a single transaction is going to give you problems.
I do not think that a single transaction is going to work in this case, you need to look at why you are using a transaction here, and find a different way to accomplish it.
I'm trying to read a Paradox 5 table into a dataset or simular data structure with the view to putting it into an SQL server 2005 table. I've trawled google and SO but with not much luck. I've tried ODBC:
public void ParadoxGet()
{
string ConnectionString = #"Driver={Microsoft Paradox Driver (*.db )};DriverID=538;Fil=Paradox 5.X;DefaultDir=C:\Data\;Dbq=C:\Data\;CollatingSequence=ASCII;";
DataSet ds = new DataSet();
ds = GetDataSetFromAdapter(ds, ConnectionString, "SELECT * FROM Growth");
foreach (String s in ds.Tables[0].Rows)
{
Console.WriteLine(s);
}
}
public DataSet GetDataSetFromAdapter(DataSet dataSet, string connectionString, string queryString)
{
using (OdbcConnection connection = new OdbcConnection(connectionString))
{
OdbcDataAdapter adapter = new OdbcDataAdapter(queryString, connection);
connection.Open();
adapter.Fill(dataSet);
connection.Close();
}
return dataSet;
}
This just return the error
ERROR [HY000] [Microsoft][ODBC Paradox Driver] External table is not in the expected format.
I've also tired OELDB (Jet 4.0) but get the same External table is not in the expected format error.
I have the DB file and the PX (of the Growth table) in the Data folder... Any help would be much appriciated.
I've had the same error. It appeared when I started my C# project on Win2008 64 (previos OS was Win2003 32). Also I found out that it worked fine in console apps and gave different errors in winforms. It seems that problem comes from the specifics of 32 ODBC driver working on 64-bit systems.
My solution was:
// Program.cs
static void Main()
{
Application.EnableVisualStyles();
Application.SetCompatibleTextRenderingDefault(false);
// it is important to open paradox connection before creating
// the first form in the project
if (!Data.OpenParadoxDatabase())
return;
Application.Run(new MainForm());
}
The connectionstring is common:
string connStr = #"Driver={{Microsoft Paradox Driver (*.db )}};DriverID=538;
Fil=Paradox 7.X;DefaultDir=C:\\DB;Dbq=C:\\DB;
CollatingSequence=ASCII;";
After opening connection you may close it in any place after creating first Form (if you need to keep DB closed most of time), for example:
private void MainForm_Load(object sender, EventArgs e)
{
Data.CloseParadoxDatabase();
}
After doing that you may open and close connection every time you want during execution of your application and you willn't get any exceptions.
Maybe this will help you out,
http://support.microsoft.com/support/kb/articles/Q237/9/94.ASP?LN=EN-US&SD=SO&FR=1
http://support.microsoft.com/support/kb/articles/Q230/1/26.ASP
It appears that the latest version of the Microsoft Jet Database Engine
(JDE) does not fully support Paradox unless the Borland Database Engine
(BDE) is also installed.
Try to Run all The Applications with the "Run As Administrator" privileges especially run the VS.NET with "Run As Administrator "... and I am sure your problem get solved
This isn't an answer, but more of a question: any particular reason you're trying to use C# to do the data manipulation as opposed to using SQL Server tools to load the data directly? Something like DTS or SSIS would seem like a better tool for the job.
Thanks, I'll give that a try. I wanted to use C# so I can put it on some web pages without the extra set of putting it in SQL server.