I recently saw this, see below. I got OutofMemoryException when loading 2.5 million records with DataTable. And near the bottom, there is a table.Dispose(). Memory usage: 560Mb! Why use DataTable anyway?
public string[] GetIDs()
{
DataTable table = new DataTable();
using (SqlConnection dwConn = new SqlConnection(this.ConnectionString))
{
dwConn.Open();
SqlCommand cmd = dwConn.CreateCommand();
cmd.CommandText = "SELECT ID FROM Customer";
SqlDataReader reader = cmd.ExecuteReader();
table.Load(reader);
}
var result = new string[table.Rows.Count];
for(int i = 0; i < result.Length; i++ )
{
result[i] = table.Rows[i].ItemArray[0].ToString();
}
table.Dispose();
table = null;
return result;
}
I turned this in the following, and the memory used was now 250Mb for 2.5 million records, same as above. The memory used is now less than 45% of the original.
public IEnumerable<String> GetIDs()
{
var result = new List<string>();
using (var dwConn = new SqlConnection(ConnectionString))
{
dwConn.Open();
SqlCommand cmd = dwConn.CreateCommand();
cmd.CommandText = "SELECT ID FROM Customer";
using (SqlDataReader reader = cmd.ExecuteReader())
{
while (reader.Read())
{
result.Add(reader["ID"].ToString());
}
}
}
return result;
}
Its good to see that you have a problem to your solution, but i would also recommend to have a look at this discussion which depicts that DataReader is a better solution than DataTable, but it depends on it use as well. After reading this you will understand memory consumption is expected to be less in case of DataReader.
Another advantage using SqlDataReader is documented in MSDN documentation is:
A part from Remarks:
Changes made to a result set by another process or thread while data
is being read may be visible to the user of the SqlDataReader.
So it is a possible reason that you are getting this difference in observation.
Hope it is useful for you and others as well.
Related
I know only one way how to count rows in results after ExecuteReader with this code:
while (er.Read()) {
count++;
}
How can I check with no while?
Short answer: you can't.
Long answer: ExecuteReader returns a forward-only sequential stream iterator, each time you advance one position you will point to and read one record until it reaches the end of the available data. Therefore it is impossible to know how many records you have until you have read all of them.
Here is a solution that may work for you, though.
Imagine you want to run this simple query: select * from Users
Even if you run this in your Oracle database directly, you won't know how many records you have until the query is executed. If you wanted to know the number of records you would use a count query, something like select count(*) from Users.
You can do the same using c#:
int count = 0;
using (var conn = new OracleConnection("Some connection string"))
{
conn.Open();
using (var cmd = conn.CreateCommand())
{
cmd.CommandText = "select count(*) from users";
count = (int)cmd.ExecuteScalar();
}
using (var cmd = conn.CreateCommand())
{
cmd.CommandText = "select * from users";
using(var reader = cmd.ExecuteReader())
{
while (reader.Read()) {
[...]
}
}
}
}
You could load your results into a DataTable and get the row count of of it:
using(var dt = new DataTable())
{
dt.Load(er); //assuming er is the OracleDataReader
}
Now you can read the rowCount with dt.Rows.Count.
I am trying to populate a group of labels in a C# windows form with some values that are in a certain attribute (PlayerName) in a database that I have in access.
Currently the only code I have is:
OleDbConnection connection = new OleDbConnection(CONNECTION STRING HERE);
OleDbCommand command = new OleDbCommand();
command.Connection = connection;
command.CommandText = "SELECT PlayerName FROM [TotalPlayerName] WHERE Team = 1 AND SportID = " + Form1.IDNumber;
I need a list or array that holds these values so I can use them to populate the labels, but I am unaware of how to do this.
You need to call ExecuteReader to obtain a data reader and then loop through the rows of the result set like this:
List<string> result = new List<string>();
using (var reader = command.ExecuteReader())
{
while (reader.Read())
{
result.Add(reader.GetString(0));
}
}
Before you do this, don't forget to open the connection like this:
connection.Open();
There are a couple of things here..... for sake of best practice well its more standard practice... as I like to say!
Use USING as this cleans up after connection.. see here for great examples in a "using" block is a SqlConnection closed on return or exception?
using (OdbcDataReader DbReader = DbCommand.ExecuteReader())
{
int fCount = DbReader.FieldCount;
while (DbReader.Read())
{
Label1 = DbReader.GetString(0);
Label2 = DbReader.GetString(1);
Label3 = DbReader.GetString(2);
Label4 = DbReader.GetString(3);
for (int i = 0; i < fCount; i++)
{
String col = DbReader.GetString(i);
Console.Write(col + ":");
}
Console.WriteLine();
}
}
NB your SQL only return 1 field /String at the moment
while reading the data fill the list like
List<string> players = new List<string>();
OleDbDataReader rdr = command.ExecuteReader();
While(rdr.Read())
{
players.Add(rdr["PlayerName"].ToString());
}
You need to create a OleDbReader object to read the response from the query. You will also need to create a List to store the data Something like this:
List<string> playerNameList = new List<string>();
using (OleDbReader r = command.ExecuteReader())
{
while(reader.Read())
{
playerNameList.Add(reader.GetString(0));
}
}
One option might be using OleDbDataAdapter to fill a DataTable those values that returns your query;
var dt = new DataTable();
using(var da = new OleDbDataAdapter(command))
{
da.Fill(dt);
}
And since your query return one column, you can use AsEnumerable to that datatable to get them as a string like;
List<string> list = dt.AsEnumerable()
.Select(r => r.Field<string>("PlayerName"))
.ToList();
You can read: Queries in LINQ to DataSet
By the way, you should always use parameterized queries. This kind of string concatenations are open for SQL Injection attacks.
Also use using statement to dispose your connection and command automatically as I did for OleDbDataAdapter in my example.
Tried finding similar from my problem but it seems there are too many unreachable code detectedmy table consist of 4 rows and I tried this code
using (MySqlConnection conn = new MySqlConnection(myConnection))
{
conn.Open();
MySqlCommand cmd = new MySqlCommand(query, conn);
int num = Convert.ToInt32(cmd.ExecuteScalar());
MySqlCommand cmd1 = new MySqlCommand(query2, conn);
MySqlDataReader reader = cmd1.ExecuteReader();
while (reader.Read())
{
for (int a = 0; a <= num; a++)
{
List li = new List();
li.linkLabel1.Text = reader["TitleAnime"].ToString();
flowLayoutPanel1.Controls.Add(li);
// break;
}
}
conn.Close();
}
but it gives me 16 values and thats too much then I tried putting break inside the for loop and I was able to achieve my goal and it gives me 4 values which is the same on my table but there seems to be an error called unreachable code detected..should I ignore it since I was able to get what I need? or is there another way for that
I'm pretty sure you are doing one query too much there, and you are getting NxN results because of that first query and of that for loop.
Try something like this:
using (MySqlConnection conn = new MySqlConnection(myConnection))
{
conn.Open();
MySqlCommand cmd1 = new MySqlCommand(query2, conn);
MySqlDataReader reader = cmd1.ExecuteReader();
while (reader.Read())
{
List li = new List();
li.linkLabel1.Text = reader["TitleAnime"].ToString();
flowLayoutPanel1.Controls.Add(li);
}
conn.Close();
}
If that does the job, consider in changing the name of the query1 and cmd1 to query and cmd since now you'll have only one of each
It's quite simple. You're running a for loop based on the number of entries you have in your database from the command object.
The read method will iterate four times if that's how many records you have in your database. Refer to this article for more information: http://msdn.microsoft.com/en-us/library/system.data.sqlclient.sqldatareader.read(v=vs.110).aspx.
In short, just remove the for loop and you'll get the result you want.
Try something like this:
while (reader.Read())
{
flowLayoutPanel1.Controls
.Add(new List { linkLabel1.Text = reader["TitleAnime"].ToString()});
}
I created an list named 'PTNList', and everything I needed added to it just fine. Now I am attempting to write code to retrieve each element from that list and run it against an SQL query. I have a feeling I'm not sure exactly how to about this. The CompareNumbers.txt file generates, but nothing is printed to it. Any help is greatly appreciated.
Below is the section of code I believe needs to be worked with.
using (FileStream fs = new FileStream("c:/temp/CompareNumbers.txt", FileMode.Append, FileAccess.Write))
using (StreamWriter sw = new StreamWriter(fs))
foreach (var ptn in PTNList)
{
//create sql for getting the count using "ptn" as the variable thats changing
//call DB with sql
//Get count from query, write it out to a file;
Console.WriteLine("Running Query");
string query2 = #"SELECT COUNT(PRODUCT_TYPE_NO)
AS NumberOfProducts
FROM dbo.PRODUCT
Where PRODUCT_TYPE_NO = " + ptn;
SqlCommand cmd2 = new SqlCommand(query2);
cmd2.Connection = con;
rdr = cmd2.ExecuteReader();
while (rdr.Read())
{
sw.WriteLine(rdr["NumberOfProducts"]);
}
rdr.Close();
}
You haven't used apostrophes around the values. But you should use parameters anyway. You could use one query instead of one for every type. For example with this approach:
string sql = #"SELECT COUNT(PRODUCT_TYPE_NO) AS NumberOfProducts
FROM dbo.PRODUCT
Where PRODUCT_TYPE_NO IN ({0});";
string[] paramNames = PTNList.Select(
(s, i) => "#type" + i.ToString()
).ToArray();
string inClause = string.Join(",", paramNames);
using (SqlCommand cmd = new SqlCommand(string.Format(sql, inClause)))
{
for (int i = 0; i < paramNames.Length; i++)
{
cmd.Parameters.AddWithValue(paramNames[i], PTNList[i]);
}
// con.Open(); // if not already open
int numberOfProducts = (int) cmd.ExecuteScalar();
}
Update: maybe you really just want to loop them and get their count. Then you don't need this complex approach. But you should still use sql-parameters to prevent sql-injection and other issues like missing apostrophes etc.
You'll want to convert the column back to a type, e.g.
sw.WriteLine(rdr["NumberOfProducts"] as string);
Also, note that your query is prone to SqlInjection attacks and should be parameterized, and that SqlCommand is also disposable. You can squeeze a bit more performance by reusing the SqlCommand:
string query2 = #"SELECT COUNT(PRODUCT_TYPE_NO)
AS NumberOfProducts
FROM dbo.PRODUCT
Where PRODUCT_TYPE_NO = #ptn";
using (var cmd2 = new SqlCommand(query2))
{
cmd2.Connection = con;
cmd2.Parameters.Add("#ptn", SqlDbType.Varchar);
foreach (var ptn in PTNList)
{
cmd2.Parameters["#ptn"].Value = ptn;
Console.WriteLine("Running Query");
using var (rdr = cmd2.ExecuteReader())
{
if (rdr.Read())
{
sw.WriteLine(rdr["NumberOfProducts"] as string);
}
}
}
}
Are you sure your query give an result and sw.WriteLine is executed? I would redesign your code like this, becfause if you have an error in your data query, you might get into trouble. I always like to use this (schema):
IDataReader reader = null;
try
{
// create every thing...
}
catch(Exception ex)
{
// catch all exceptions
}
finally()
{
if(reader != null)
{
reader.Close();
}
}
And use the same for your connection, so that you can be sure, it is closed correct.
I made function in c# to read line by line and then load lines to sqlite (s3db).
private void LoadFromDictionary()
{
Encoding enc = Encoding.GetEncoding(1250);
using (StreamReader r = new StreamReader("c:\\Temp2\\dictionary.txt", enc))
{
string line = "";
while ((line = r.ReadLine()) != null)
{
line = line.Trim();
AddWord(line);
}
}
MessageBox.Show("Finally :P", "Info");
}
private void AddWord(string w)
{
String insSQL = "insert into Words values(\"" + w + "\")";
String strConn = #"Data Source=C:\Temp2\dictionary.s3db";
SQLiteConnection conn = new SQLiteConnection(strConn);
SQLiteDataAdapter da = new SQLiteDataAdapter(insSQL, strConn);
da.Fill(dt);
dataGridView1.DataSource = dt.DefaultView;
}
But is it any faster way? I created table by sqlite administrator application.
Can sqlite load itself file and make it as a table?
I am talking about 3+ millions words (one word in one line).
PS. please correct my topic if there is something wrong :)
Yes, there is a much, much faster method using the following techniques:
1) Only open a connection to the database one time
2) Use a parameterized command for better performance and lower overhead (don't have to use new strings on each pass).
3) Wrap the entire operation in a transaction. As a general rule, this will improve your performance.
Note that I do not show transaction rollback or closing the connection, which are also best practices that should be implemented.
private void LoadFromDictionary()
{
Encoding enc = Encoding.GetEncoding(1250);
string strConn = #"Data Source=C:\Temp2\dictionary.s3db";
SqliteConnection conn = new SqliteConnection(strConn);
conn.Open();
string insSQL = "insert or ignore into wyrazy values(#Word)";
DbCommand oCommand = conn.CreateCommand();
oCommand.Connection = conn;
oCommand.CommandText = insSQL;
DbParameter oParameter = oCommand.CreateParameter();
oParameter.Name = "#Word";
oParameter.DbType = DbType.String;
oParameter.Size = 100;
oCommand.Parameters.Add(oParameter);
DbTransaction oTransaction = conn.BeginTransaction();
using (StreamReader r = new StreamReader("c:\\Temp2\\dictionary.txt", enc))
{
string line = "";
while ((line = r.ReadLine()) != null)
{
line = line.Trim();
if (!string.IsNullOrEmpty(line)) {
oParameter.Value = line;
oCommand.ExecuteNonQuery();
}
}
}
oTransaction.Commit();
conn.Close();
MessageBox.Show("Finally :P", "Info");
}
You could try bulk insert. By reading this article please pay special attention to the parametrized queries being used there and which you should use instead of the string concatenations in your sample in the insSQL variable.
Using Transactions usually speed things up quite a bit, depending on your desired batch size. I'm not 100% as familiar with DataAdapters and DataSources but instead of creating a new connection every time to insert one row, modify your code to use one connection and use SQLiteConnection.BeginTransaction() and the when you are done call Transaction.Commit().
I just did this the other day, first use a transaction, and parameterized queries. I was able to load 16 million rows in about a minute doing this.
internal static void FastInsertMany(DbConnection cnn)
{
using (DbTransaction dbTrans = cnn.BeginTransaction())
{
using (DbCommand cmd = cnn.CreateCommand())
{
cmd.CommandText = "INSERT INTO TestCase(MyValue) VALUES(?)";
DbParameter Field1 = cmd.CreateParameter();
cmd.Parameters.Add(Field1);
for (int n = 0; n < 100000; n++)
{
Field1.Value = n + 100000;
cmd.ExecuteNonQuery();
}
}
dbTrans.Commit();
}
}