We Keep Coding

C# - DataTable Out of Memory exception in application to catch SQL Server "INSERT" events

I have been tasked with creating an application that monitors any "INSERT" events on a specific table. I was going to go about this using SqlDependency to create a notification link between the DB and the C# app, but it turns out I am not able to do this due to security issues.
Due to this, I have modeled my application as follows:
This is well and good, but as it turns out, the SQL table I am querying has a rather large size. The table has nearly 3.5 Million rows 55 columns. When loading into the C# DataTable object, I am getting an out of memory exception.
internal static DataTable ExecuteQuery(string query, Dictionary<string,string> parameters = null)
{
try
{
using (SqlConnection dbconn = new SqlConnection(SQLServer.Settings.ConnectionString))
using (SqlCommand cmd = new SqlCommand())
{
dbconn.Open(); // Open the connection
cmd.CommandText = query; // Set the query text
cmd.Connection = dbconn;
if (parameters != null)
{
foreach (var parameter in parameters) // Add filter parameters
cmd.Parameters.AddWithValue(parameter.Key, parameter.Value);
}
var dt = new DataTable();
using (SqlDataAdapter adpt = new SqlDataAdapter(cmd)){adpt.Fill(dt);} // MY ERROR OCCURS HERE!
dbconn.Close();
queryError = false;
return dt;
}
}
catch(Exception ex)
{
queryError = true;
EventLogger.WriteToLog("ExecuteQuery()", "Application", "Error: An error has occured while performing a database query.\r\nException: " + ex.Message);
return null;
}
}
When running the code above, I get the following error at the line for SqlDataAdapter.Fill(dt)
Exception of type 'System.OutOfMemoryException' was thrown.
Is there a way that I can either restructure my application OR prevent this incredibly high memory consumption from the DataTable class? SQL server seems capable enough to do a select * from the table but when I fill a DataTable with the same data, I use up over 6GB of RAM! Why is there so much overhead when using DataTable?
Here is a link to my flowchart.

I was able to resolve this issue by making use of the SqlDataReaderclass. This class lets you "stream" the sql result set row by row rather bringing back the entire result set all at once and loading that into memory.
So now in step 5 from the flow chart, I can query for only the very first row. Then in step 6, I can query again at a later date and iterate through the new result set one row at a time until I find the original row I started at. All the while, I am filling a DataTable with the new results. This accomplishes two things.
I don't need to load all the data from the query all at once into local memory.
I can immediately get the "inverse" DataSet. AKA... I can get the newly inserted rows that didn't exist the first time I checked.
Which is exactly what I was after. Here is just a portion of the code:
private static SqlDataReader reader;
private static SqlConnection dbconn = new SqlConnection(SQLServer.Settings.ConnectionString);
private void GetNextRows(int numRows)
{
if (dbconn.State != ConnectionState.Open)
OpenConnection();
// Iterate columns one by one for the specified limit.
int rowCnt = 0;
while (rowCnt < numRows)
{
while (reader.Read())
{
object[] row = new object[reader.FieldCount];
reader.GetValues(row);
resultsTable.LoadDataRow(row, LoadOption.PreserveChanges);
rowCnt++;
sessionRowPosition++;
break;
}
}
}
The whole class would be too large for me to post here but one of the caveats was that the interval between checks for me was long, on the order of days, so I needed to close the connection between checks. When closing the connection with a SqlDataReader, you loose your row position so I needed to add a counter to keep track of that.

Check you query for select. You probably get from database many rows.

Database method to work with any type of provider

So far my program only dealt with Sql Server for any type of data work. I would like my program to work with MySql as well. And now that I making this change, I would like to avoid code repetition as much as I can. So I started thinking about having a Factory Pattern with two concrete factories SqlServerFactory and MySqlFactory. This will work! But many of my database methods have pretty much the same structure except that in one, we are using an SqlConnection and SqlCommand and the other we are using a MySqlConnection and the rest of the method code is basically the same. Below is a example of one of my SELECT methods:
public DataTable GetSpyList()
{
DataTable dt = new DataTable();
using (OleDbConnection conn = new OleDbConnection(cs))
{
try
{
string query = "SELECT * FROM SpyList";
OleDbDataAdapter da = new OleDbDataAdapter(query, conn);
conn.Open();
da.Fill(dt);
conn.Close();
return dt;
}
catch (Exception ex)
{
return null;
}
}
}
How can I reduce code repetition by making the above method work for any type of data provider? Is this something that can be achieved by using DbProviderFactories?

Yes, the static DbProviderFactories and the DbProviderFactory class are fit for your purposes.
Of course, every abstraction like this is based on the common functionality of the base classes DbConnection, DbCommand, DbDataReader, DbDataAdapter.
For example. In the MySqlDataReader class you could find an overload of the method GetString that takes as input the name of the column to retrieve. This overload doesn't exists in the SqlDataReader and you need to use the ordinal position of the column to retrieve data. So, in case of DbProviderFactory you could use only the common version of GetString between the MySql provider and the Sql Server provider (IE using the GetString method that takes the ordinal position). Luckily, Intellisense helps a lot to avoid this error.
You could rewrite your code as
public DataTable GetSpyList()
{
DbProviderFactory fac = DbProviderFactories.GetFactory("MySQL.Data.MySqlClient");
DataTable dt = new DataTable();
using(DbConnection conn = fac.CreateConnection())
{
cn.ConnectionString = cs
DbDataAdapter da = fac.CreateDataAdapter();
da.SelectCommand = conn.CreateCommand();
da.SelectCommand.CommandText = "SELECT * FROM SpyList";
try
{
conn.Open();
da.Fill(dt);
return dt;
}
catch (Exception ex)
{
return null;
}
}
}
This example uses a fixed name for the Data Provider, but of course, you could read this name from some kind of configuration file and have your program change its underlying database changing the entry in the configuration file.

Using select query in C#.NET for web service application

I am not unable to understand the way in which a select * query is written while we create a web service using C#.NET to connect to an SQL Database.
Basically, I have 4 columns . I am taking a particular input from the user , which is an existing value of 1st column. Now depending on that value I want to select all the records of the remaining 3 columns. I need to write a SELECT * query but I don't in what form will I get those records. I have heard of SqlDataAdapter but then will it return me the records in a row-column format or do have to store the result in some sort of List and then use it for other purposes.
Can someone please help me to understand how such a query can be written?
This code is wrong but it will help understand what I need
I want to get the records of other columns based upon my "where clause" condition
using System;
using System.Collections;
using System.ComponentModel;
using System.Data;
using System.Linq;
using System.Web;
using System.Web.Services;
using System.Web.Services.Protocols;
using System.Xml.Linq;
using System.Data.SqlClient;
namespace statistics
{
/// <summary>
/// Summary description for Service1
/// </summary>
[WebService(Namespace = "http://tempuri.org/")]
[WebServiceBinding(ConformsTo = WsiProfiles.BasicProfile1_1)]
[ToolboxItem(false)]
// To allow this Web Service to be called from script, using ASP.NET AJAX, uncomment the following line.
// [System.Web.Script.Services.ScriptService]
public class Service1 : System.Web.Services.WebService
{
[WebMethod]
public int retrieve(String rollno)
{
int rows=0;
SqlConnection myConnection = new SqlConnection(#"Data Source=.\SQLEXPRESS;Initial Catalog=student;User ID=sa;Password=123");
try
{
myConnection.Open();
SqlCommand myCommand = new SqlCommand();
myCommand.Connection = myConnection;
String strsql = "SELECT * FROM checkrecord values WHERE rollno=#rollno";
DataSet dataSet = new DataSet();
SqlDataAdapter dataAdapter = new SqlDataAdapter(strsql, myConnection);
myCommand.Parameters.Add("#rollno", SqlDbType.VarChar).Value = rollno;
rows = myCommand.ExecuteNonQuery();
SqlDataReader myReader = myCommand.ExecuteReader()
}
catch (Exception ex)
{
Console.WriteLine(ex.Message);
}
finally
{
myConnection.Close();
}
return rows;
}
}
}

You need to connect to the database, then use SQLDataAdapter to send the user input SELECT query to the database. Something like,
SELECT col2,col3,col4 FROM yourtable WHERE col1 = ?
There are different ways to use the SQLDataAdapter, and specify the query and parameters. The SQLDataAdapter will return a DataSet that contains a DataTable. You can then use the DataSet or DataTable to populate the control of your choice (ie; DataGrid) with the results. There is a walk-through of this on MSDN: http://msdn.microsoft.com/en-us/library/aa984467%28v=vs.71%29.aspx
Based on your edit, you don't want to use ExecuteNonQuery. To get the rows you use ExecuteReader. Example here:
http://msdn.microsoft.com/en-us/library/system.data.sqlclient.sqlcommand.executereader%28v=VS.71%29.aspx

I think you are talking about the return type of the webservice?
Yes, the database query will return you a list of the record values. With the .net webservices however, you can put these values into a custom class written by you, and the structure of this class will be included in your webservice.
[WebMethod]
public Myclass retrieve(String rollno)
{
return new Myclass("variable1", "variable2");
}
It will be included in the WSDL and thus can be implemented by any other application using your web service.

I recommend looking at the documentation for the class in question, in this case, the SQLDataAdapter
http://msdn.microsoft.com/en-us/library/system.data.sqlclient.sqldataadapter.aspx
What I think you might want is to fill the results into a dataset.

How to populate Crystal Reports with SQL

Since there doesn't seem to be any way to load old VB6 (ActiveReports) reports in .Net, I need to recreate dozens of reports in .Net. I'd like to do this in the least painful way possible.
In VB6, the original authors simply did something like this for every report:
adoConn.ConnectionString = globalConnectionObject.ConnectionString
adoConn.Source = ReportFunctions.GetAllUsers()
GetAllUsers() returns an SQL string which selects a bunch of fields; those fields are then used in the report.
Now:
How can I do something similar in .Net (either using the built-in Crystal Reports, or the built-in "Microsoft Reporting Technology")?
I can't get "Database Expert" to recognize globalConnectionObject (an ADODB.Connection object); and if I fill a dataset and do
report.SetDataSource(dataSet)
It tells me "The report has no tables."
How do I populate a Crystal Reports report!? (the connection string/data location is not known at compile time)

Creating Connection:
/// <summary>
/// Sets the connection.
/// </summary>
public void SetConnection()
{
_connection = new SqlConnection(Settings.Default.connectionString);
if (_connection.State == System.Data.ConnectionState.Open)
{
_connection.Close();
}
_connection.Open();
}
/// <summary>
/// Closes the connection.
/// </summary>
public void CloseConnection()
{
if (_connection.State == System.Data.ConnectionState.Open)
{
_connection.Close();
_connection.Dispose();
}
}
using above connection Populate dataset manually or Add a dataset to your solution to get data without writing connection codes. Say You have added Employee.XSD. Add a tableadapter in the XSD which will help you in pulling data from database, by auto generating queries and datatables. After doing all these things. Create a method somewhere in your project,
Public DataTable GetAllEmployees()
{
Employee.EmployeeTableAdapter adapt = New Employee.EmployeeTableAdapter();
DataTable dt = New DataTable();
dt = adapt.GetData(); // you can also use fill based on your criteria.
return dt; //your datatable with data
}
Now on Your Form add a reportviewer control.
ReportViewer1 rpt = New ReportViewer();
ReportDocument rptDoc = new ReportDocument();
rptDoc.Load("path of rpt file");
rptDoc.SetDataSource(GetAllEmployees());
rpt.Document = rptDoc;
rpt.Refresh();
Before that From Crystal Report , add the fields of table on the report as per your requirement.
Another Way To achieve
Crystal Reports can be used over various objects. If you have scenario to bind it dynamically, see my answer below Then u can do one thing that is , add a new dataset to solution. Create a datatable and add required columns with appropriate data type. DO not add query or table adapter. Now from your code add a class file and create properties exactly similar to the datatable columns. Now set the datatable as source to report and add its column on the report.
For example , if you have columns
ID - integer
EmpName - string
Salary - double
Department - string
Create a class
public class Employee
{
private SqlConnection _connection;
public int ID {get;set;}
public string EmpName {get;set;}
public double Salary {get;set;}
public string Department {get;set;}
/// <summary>
/// Sets the connection.
/// </summary>
public void SetConnection()
{
//assuming connection string is placed in settings file from Project Properties.
_connection = new SqlConnection(Settings.Default.connectionString);
if (_connection.State == System.Data.ConnectionState.Open)
{
_connection.Close();
}
_connection.Open();
}
/// <summary>
/// Closes the connection.
/// </summary>
public void CloseConnection()
{
if (_connection.State == System.Data.ConnectionState.Open)
{
_connection.Close();
_connection.Dispose();
}
}
public DataTable GetEmployees()
{
DataTable dt = new DataTable("Employee");
// using above connection
SetConnection();
using(SqlCommand command = new SqlCOmmand("commandText",_connection))
{
using(SqlDataReader reader = command.ExecuteReader())
{
dt.Load(reader);
}
}
return dt;
}
}
Now create another function that will populate the datatable created inside the dtataset.
public void PopulateDataTable()
{
DataTable dt = GetEmployee();
Employee dsEmployee = New DataSet();
dsEmployee.EmployeeDataTable dtEmp = new dsEmployee.EmployeeDataTable();
dtEmp = dt;
}

We do something similar using Crystal Reports, and it has caused me to really dislike Crystal Reports. Using the database expert I created a new connection to the database and a new command with the SQL that I want to use. This defines the columns that Crystal Reports can use and allows you to create the report. Then when displaying the report we do:
ReportDocument rd = new ReportDocument();
rd.Load(MapPathSecure("NameOfMyReport.rpt"));
rd.SetDataSource(dataSet.Tables[0]);
I have only done it with one table, so I don't know how having multiple tables in the DataSet would affect it. Basically, you set the structure of the report with the Database Expert and the command. Then you overwrite the actual data at runtime. I really wish that Crystal Reports had a more robust way of doing this, or that I understood it if it exists.

c# windows forms / basic data grid

One of the problems I am having with c# is that there seems to be so much information online that I am having trouble finding the right answer to the most basic of questions.
I am trying to do something simple:
I have a button, I click it, it queries the database and populates a datagrid on my windows form.
private void button1_Click(object sender, EventArgs e)
{
SqlConnection c = new SqlConnection("Data Source = (local); Integrated Security = true; Initial Catalog = pubs; ");
c.Open();
// 2
// Create new DataAdapter
SqlCommand cmd = c.CreateCommand();
cmd.CommandText = #" SELECT * FROM Authors ";
SqlDataReader reader = cmd.ExecuteReader();
dataGridView1.DataSource = reader;
dataGridView1.DataBind();
}
Error 1 'System.Windows.Forms.DataGridView' does not contain a definition for 'DataBind' and no extension method 'DataBind' accepting a first argument of type 'System.Windows.Forms.DataGridView' could be found.....
I am probably missing a "using directive" but which one? Multiple Google searches tell me how to bind a Yahoo RSS Feed to a gridview or provide various obscure details on "using directives".
Maybe I am using the SqlDataReader incorrectly. Should I be using SqlAdapter instead? What happened to all the good basic tutorials online for windows c# forms? A few months ago I found a couple great tutorials, but they seem to have lost their pageranking and I cannot find them anymore using basic google searches.

You're not missing a using directive; it's just that the WinForms DataGridView doesn't have a DataBind method. Just assigning DataSource is enough to get the binding to happen; you don't need to call a method as well.
However, I don't think you can assign a SqlDataReader as the DataSource. According to the DataSource property documentation in MSDN, the DataSource must be an IList, an IListSource, an IBindingList or an IBindingListView. You will probably instead need to load the data into a DataTable or DataSet (or an object data source populated using an object-relational mapper), and use that as the DataSource.

Try this instead:
using (SqlConnection conn = new SqlConnection("your connection string"))
{
conn.Open();
using (SqlCommand cmd = new SqlCommand(#"SELECT * FROM Authors", conn))
{
using (SqlDataAdapter adap = new SqlDataAdapter(cmd))
{
DataTable dt = new DataTable();
adap.Fill(dt);
dataGridView1.DataSource = dt;
}
}
}
The DataGridView does not have a DataBind() method because it doesn't need one. Setting the DataSource property handles the binding for you. The using() blocks will automatically close and dispose of everything for you as well.
Note: you should replace "your connection string" with a valid connection string. I left yours out of my sample to avoid the horizontal scrollbars, and I'm not sure yours is valid anyway. You may get a runtime error when you run the code using your connection string. www.connectionstrings.com is a great resource for figuring out a valid connection string.
Update: instead of the nested using() blocks, you can also do it like this:
using (SqlConnection conn = new SqlConnection("..."))
using (SqlCommand cmd = new SqlCommand(#" SELECT * FROM Authors", conn))
using (SqlDataAdapter adap = new SqlDataAdapter(cmd))
{
conn.Open();
DataTable dt = new DataTable();
adap.Fill(dt);
dataGridView1.DataSource = dt;
}
I prefer the nested style, but it's "half of one, six dozen of the other" to me. Typically, I would encapsulate code like this into a class (called "DataGetter" or whatever) with a static method like:
public static DataTable GetData(string query)
{
// do all the connecting and adapting and filling and so forth
}
so that the code in your button click would be as simple as:
dataGridView1.DataSource = DataGetter.GetData("SELECT * FROM AUTHORS");
However, I would not do this in any performance-critical section of my code, since you sometimes want to keep a SqlCommand object (and its SqlParameter collection) around between calls. You do not need to keep SqlConnection objects around between calls, thanks to connection pooling (in fact, you don't want to keep them around under any circumstances).