We Keep Coding

C# - DataTable Out of Memory exception in application to catch SQL Server "INSERT" events

I have been tasked with creating an application that monitors any "INSERT" events on a specific table. I was going to go about this using SqlDependency to create a notification link between the DB and the C# app, but it turns out I am not able to do this due to security issues.
Due to this, I have modeled my application as follows:
This is well and good, but as it turns out, the SQL table I am querying has a rather large size. The table has nearly 3.5 Million rows 55 columns. When loading into the C# DataTable object, I am getting an out of memory exception.
internal static DataTable ExecuteQuery(string query, Dictionary<string,string> parameters = null)
{
try
{
using (SqlConnection dbconn = new SqlConnection(SQLServer.Settings.ConnectionString))
using (SqlCommand cmd = new SqlCommand())
{
dbconn.Open(); // Open the connection
cmd.CommandText = query; // Set the query text
cmd.Connection = dbconn;
if (parameters != null)
{
foreach (var parameter in parameters) // Add filter parameters
cmd.Parameters.AddWithValue(parameter.Key, parameter.Value);
}
var dt = new DataTable();
using (SqlDataAdapter adpt = new SqlDataAdapter(cmd)){adpt.Fill(dt);} // MY ERROR OCCURS HERE!
dbconn.Close();
queryError = false;
return dt;
}
}
catch(Exception ex)
{
queryError = true;
EventLogger.WriteToLog("ExecuteQuery()", "Application", "Error: An error has occured while performing a database query.\r\nException: " + ex.Message);
return null;
}
}
When running the code above, I get the following error at the line for SqlDataAdapter.Fill(dt)
Exception of type 'System.OutOfMemoryException' was thrown.
Is there a way that I can either restructure my application OR prevent this incredibly high memory consumption from the DataTable class? SQL server seems capable enough to do a select * from the table but when I fill a DataTable with the same data, I use up over 6GB of RAM! Why is there so much overhead when using DataTable?
Here is a link to my flowchart.

I was able to resolve this issue by making use of the SqlDataReaderclass. This class lets you "stream" the sql result set row by row rather bringing back the entire result set all at once and loading that into memory.
So now in step 5 from the flow chart, I can query for only the very first row. Then in step 6, I can query again at a later date and iterate through the new result set one row at a time until I find the original row I started at. All the while, I am filling a DataTable with the new results. This accomplishes two things.
I don't need to load all the data from the query all at once into local memory.
I can immediately get the "inverse" DataSet. AKA... I can get the newly inserted rows that didn't exist the first time I checked.
Which is exactly what I was after. Here is just a portion of the code:
private static SqlDataReader reader;
private static SqlConnection dbconn = new SqlConnection(SQLServer.Settings.ConnectionString);
private void GetNextRows(int numRows)
{
if (dbconn.State != ConnectionState.Open)
OpenConnection();
// Iterate columns one by one for the specified limit.
int rowCnt = 0;
while (rowCnt < numRows)
{
while (reader.Read())
{
object[] row = new object[reader.FieldCount];
reader.GetValues(row);
resultsTable.LoadDataRow(row, LoadOption.PreserveChanges);
rowCnt++;
sessionRowPosition++;
break;
}
}
}
The whole class would be too large for me to post here but one of the caveats was that the interval between checks for me was long, on the order of days, so I needed to close the connection between checks. When closing the connection with a SqlDataReader, you loose your row position so I needed to add a counter to keep track of that.

Check you query for select. You probably get from database many rows.

How can create a new table in database if the table does not exist (c#)

Here is my code, it works when the database has the table "UserInformation"
public bool Save()
{
using (SqlConnection connection = new SqlConnection(ConnectionString)
{
connection.Open();
using (SqlTransaction transaction = connection.BeginTransaction())
{
try
{
using (var adapter = new UserInformationTableAdapter())
{
adapter.Connection = connection;
adapter.Transaction = transaction;
var table = new HelloDataSet.UserInformationDataTable();
HelloDataSet.UserInformationRow row = table.NewUserInformationRow();
row.UserName = userName;
row.Password = password;
row.Brithday = brithday;
table.Rows.Add(row);
adapter.Update(table);
transaction.Commit();
return true;
}
}
catch (Exception e)
{
transaction.Rollback();
return false;
}
finally
{
connection.Close();
}
}
}
However, when there is no table in the database, it will not create the "UserInformation" table in the database, it will jump to "catch" exception in line "adapter.Update(table);"
So my question is how can I create a new table in database if there is no "UserInformation" table in it. In addition, if the database already has the table "UserInformation" can I add a new column "Position" in that table?

Finally, I got the answer and want to share it out. First, I have to say that I put my question in a wrong way. What I really want is I have an application, and this application is connected with a database. However, I am allowed user to switch database. So, when the user switch to a new database I would like the application copy the entire database structure (not including the data) from the old one to the new one. Also, if I make some change (could be add a new column for one or more table, or add another new table) for the database in my application code, I would like every other database know the updates and make the same change by running my new application code.
So, here is my solution. I write a framework called "SchemaManager." It will create an additional table in each database, this table contains the version of the database. So, every time when I run my application the "SchemaManager" will check my hard code database version number with the database version number, if my hard code database version number is greater than the database version number, the "SchemaManager" will check the change and do the update for me.
I know my solution is not the best, but this is what I did. If anyone have anyother solution, please share with me and other people.

Displaying data from local SQL Server CE database in a gridview C#

I am working on a small application for personal use. It is about keeping some data readily available for me.
It just consists of a local database, functions to add, erase or modify 4 or 5 columns of data and displaying the table in a datagridview.
I have managed to add data to the table and I have managed to use a
SELECT * FROM mytable
statement to get the data and iterate through it but I want to bind the table to the datagridview.
Here is my current method of trying to bind the data:
private void button2_Click(object sender, EventArgs e)
{
string query = "SELECT * FROM myTable";
SqlCeConnection conn = new SqlCeConnection(conString);
using (SqlCeDataAdapter adap = new SqlCeDataAdapter(query, conn))
{
//the adapter will open and close the connection for you.
DataTable dat = new DataTable();
adap.Fill(dat);
dataGridView1.DataSource = dat;
}
}
When I run this code it does not throw an exception and if I change the name of the table to something that does not exists then it causes an exception telling that the table does not exists so I know that it is fetching my table. It simply is not showing it.
Any ideas?
Thanks

C# Login Screen - Find User in SQL Table and Update Field

I am currently writing a small application to keep track of monetary ins and outs, something just to improve my general C# skills. For my Login Screen currently I have the following Code
private void Login_Load(object sender, EventArgs e)
{
// TODO: This line of code loads data into the 'spendingInsAndOutsDataSet.Users' table. You can move, or remove it, as needed.
this.usersTableAdapter.Fill(this.spendingInsAndOutsDataSet.Users);
}
private void button1_Click(object sender, EventArgs e)
{
string userNameText = userName.Text;
string passwordText = password.Text;
foreach (DataRow row in spendingInsAndOutsDataSet.Users)
{
if (row.ItemArray[4].Equals(userNameText) && row.ItemArray[5].Equals(passwordText))
{
MessageBox.Show("Login Successful");
MainGUI newForm = new MainGUI();
this.Visible = false;
newForm.Show();
break;
}
else
{
userName.Text = String.Empty;
password.Text = String.Empty;
MessageBox.Show("Login Failed");
break;
}
}
}
What I am looking to do when the Login is Sucessful is to write the MachineName of the current PC to a field in Users table in my SQL Database. That way when I come to start creating records I can quickly find my UsersId (which is a foreign key in my Transactions table).
I know you can get the Active Machine Name using the System.Enviroments path but Im unsure exactly how to go about writing the update. I know how to do it using a SqlCommand but I am wondering if there is a simpler way using the DataRows I have used within the ForEach loop.
Thanks in advance, any questions let me know.
James

Assuming it is an Access database (If not then make the necessary changes):
Use an Adapter to fill a table with your results. Then compare the row columns with the information provided by the user. Don't forget to use parameters to avoid injections that may potentially ruin your database or expose your user's information to a hacker.
DataTable dt = new DataTable();
String sql = "SELECT * FROM users WHERE user = #user and password=#password"
OleDbConnection connection = getAccessConnection();
OleDbDataAdapter da = new OleDbDataAdapter(sql, connection);
da.SelectCommand.Parameters.Add("#user", OleDbType.VarChar).Value = userNameText;
da.SelectCommand.Parameters.Add("#password", OleDbType.VarChar).Value = password.Text;
try
{
connection.Open();
da.Fill(dt);
connection.Close();
}
catch(OleDbException ex)
{
connection.Close();
MessageBox.Show(ex.ToString());
}
if(dt.Rows.Count == 1)
return true; //username && password matches
else if(dt.Rows.Count == 0)
return false; // does not match
You could also use AddWithValue for your parameters.
da.SelectCommand.Parameters.AddWithValue("#user", userNameText);
getAccessConnection() is a predefined OleDbConnection function that has the connection to the database setup and creates a new instance of the connection for you (that I have created for myself).
public OleDbConnection getAccessConnection()
{
this.connection = new OleDbConnection();
this.connection.ConnectionString = #"Provider=Microsoft.ACE.OLEDB.12.0;Data Source="
+ Classified.SOURCE + ";Jet OLEDB:Database Password="
+ Classified.PASS + ";";
return this.connection;
}
It is preferred to create classes for all of these functions for developers who may join in on the project. Also read up on C# test driven development as well.
Also it looks like your loop will break even if that one record fails, only allowing it to go to it's first record.
Onto creating your own data set and filling it with queried tables is also useful. Here is a brief example:
DataSet ds = new DataSet();
ds.Tables.Add(dt, "userSearchedTable");
ds.Tables["userSearchedTable"].Rows[0][1].ToString();
Then you can declare a specific data table within the set when ever you need to.

In your foreach loop, set the MachineName of the current PC on relevant row then at the end of the method call:
this.usersTableAdapter.Update(this.spendingInsAndOutsDataSet.Users);
This will update the database with the machine name
However looking at your code there are a few additional comments to make I'd like to add to improve what you have:
You are loading the entire data table and then checking it for the username and password. Really you query for the user ID in the database, load that single row and check the password. If you have many users, your current implementation will create a lot of network traffic.
Instead of:
foreach (DataRow row in spendingInsAndOutsDataSet.Users)
Consider using something like:
foreach (SpendingInsAndOutsDataSet.UsersRow row in spendingInsAndOutsDataSet.Users)
i.e. the strongly typed version of the data row object. This means you can use:
row.Username.Equals(userNameText)
instead of
row.ItemArray[4].Equals(userNameText)
Also if you are anticipating that this will be used over a network, you should look to encrypt the passwords.

How I can save the data from GridView to the database?

I am developing an ASP.Net C# Web Application that contains a GridView to display the records of a certain table from my database which I use ODBC Connection to connect to it and a DataSet to save data in it and edit it then I should save data to the database using the changes made in the DataSet.
I could access the database succefully using the fill() method of the OdbcDataAdapter and I could do databinding so that the data is viewed in the GridView.
My question is how I can save the gridview to the dataset then to the database when any updates or changes done [the vice versa of the operation done before]?
My sample code that is used inside a web form class is as follow:-
private void SelectFromDatabase()
{
string OdbcConnectionString1 = getConnectionString();
OdbcConnection OdbcConnection1 = new OdbcConnection(OdbcConnectionString1);
string OdbcSelectText1 = "SELECT * FROM table";
OdbcCommand OdbcSelectCommand1 = new OdbcCommand(OdbcSelectText1, OdbcConnection1);
OdbcDataAdapter OdbcDataAdapter1 = new OdbcDataAdapter();
try
{
OdbcConnection1.Open();
OdbcDataAdapter1.SelectCommand = OdbcSelectCommand1;
OdbcDataAdapter1.AcceptChangesDuringFill = true;
int FillResult = OdbcDataAdapter1.Fill(myDataSet, TableName);
myDataSet.AcceptChanges();
fillGridViewbyDataset(myGridView, myDataSet, TableName);
Response.Write("<br/>SelectFromDatabase() Fill Result: " + FillResult);
}
catch (Exception Exception1)
{
Response.Write("<br/> SelectFromDatabase() Exception: " + Exception1.Message);
}
finally
{
OdbcConnection1.Close();
}
}
private void fillGridViewbyDataset(GridView gv, DataSet ds, string dt)
{
gv.DataSource = ds;
gv.DataMember = dt;
gv.DataBind();
}
what I need is something like:-
how to save Gridview to the DataSet then save the DataSet to the database as i got the gridview updates but the database still without any updates !!
if I have a DataSet called myDs and I edit a field in it by direct access in a loop like the following:-
for (int i = 0; i < myDS.Tables[TableName].Rows.Count; i++)
{
//some function or web method to get the id value of the record being updated
int n = getNewNumber();
//updating the dataset record according to some condition
if (n == 0)
{
myDS.Tables[TableName].Rows[i]["id"] = n;
myDS.Tables[TableName].Rows[i]["description"] = "some data";
}
else
{
myDS.Tables[TableName].Rows[i]["id"] = n;
myDS.Tables[TableName].Rows[i]["description"] = "new data";
}
}
How I make these changes done in the database as I could see it in the GridView when I do databind() but the database is not affected and I try using the fill & update methods of OdbcDataAdapter and OdbcCommandBuilder ??
Please this is urgent as I need it in developing an important application..
Thanks in advance for your replies and answers .....

Everything you need to know about saving from the GridView to the DataSet and to the DB is explained is this article.
Hope this helps!

If it's "an important application", I'd recommend using Stored Procedures and grant only the EXECUTE privilege to the database user on the package. If the user has full DML privileges, your data might be more vulnerable.
Here's a basic tutorial on calling stored procedures
If you have time, I'd also look at the Microsoft Enterprise Library.