We Keep Coding

C# MYSQL query every second

So, I have a program I'm working on where more than one client is connected to a MYSQL Database and I want to keep them all current. When one client updated the database, the information in all clients updates. I'm new and still studying in college, so the only way I could think to do this is to make a column that held each records update time and then each second use this query:
if (sqlHandler.userLastUpdate < sqlHandler.LastUpdate())
{
//Loads the products from the Database
sqlHandler.ReadDB();
//Set's this clients last update time to now so it doesn't keep refreshing
sqlHandler.userLastUpdate = DateTimeOffset.UtcNow.ToUnixTimeSeconds();
}
public double LastUpdate()
{
using var con = new MySqlConnection(CONNECTION_STRING);
con.Open();
string sql = "SELECT MAX(LastUpDate) FROM products";
using var cmd = new MySqlCommand(sql, con);
using MySqlDataReader rdr = cmd.ExecuteReader();
rdr.Read();
double time = rdr.GetDouble(0);
return time;
}
This seems horribly inefficient. Is there a better way to do this. I've had a couple clients running on the same MYSQL server and it seemed to be fine, but it just seems to be a better way to do this.

C# Low in memory and Insert into Oracle DB data - 1 million rows - OutOfMemory Exception

The Problem:
I have a web application where people can upload xml, xmls, csv files.
I then take their content and insert it into my Oracle DB.
Technical details:
I recently had a problem where I get OutOfMemory Exception trying to use the data.
The previous developer created a list of lists on the data in order to manage them. However, this is giving us OutOfMemory Exception.
We are using the LinqToExcel library.
Sample code:
excel = new ExcelQueryFactory(excelFile);
IEnumerable<RowNoHeader> data = from row in excel.WorksheetNoHeader(sheetName)
select row;
List<List<string>> d = new List<List<string>>(data.Count());
foreach (RowNoHeader row in data)
{
List<string> list = new List<string>();
foreach (Cell cell in row)
{
string cellValue = cell.Value.ToString().Trim(' ').Trim(null);
list.Add(cellValue);
}
d.Add(list);
}
I have tried to change the code and instead did this:
string connectionstring = string.Format(#"Provider=Microsoft.ACE.OLEDB.12.0;Data Source={0};Extended Properties='Excel 12.0;HDR=YES;';", excelFile);
OleDbConnection connection = new OleDbConnection();
connection.ConnectionString = connectionstring;
OleDbCommand excelCommand = new OleDbCommand();
excelCommand.Connection = connection;
excelCommand.CommandText = String.Format("Select * FROM [{0}$]", sheetName);
connection.Open();
DataTable dtbl = CreateTable(TableColumns);
OleDbDataReader reader = excelCommand.ExecuteReader();
while (reader.Read())
{
DataRow row = dtbl.NewRow();
dtbl.Rows.Add(row);
}
using (OracleCommand command = new OracleCommand(selectCommand, _oracleConnection))
{
using (OracleDataAdapter adapter = new OracleDataAdapter(command))
{
using (OracleCommandBuilder builder = new OracleCommandBuilder(adapter))
{
OracleTransaction trans = _oracleConnection.BeginTransaction();
command.Transaction = trans;
adapter.InsertCommand = builder.GetInsertCommand(true);
adapter.Update(dtbl);
trans.Commit();
}
}
}
However, I still get the same OutOfMemory Exception.
I have read online and've seen that I should make my project x64 and use the following:
<runtime>
<gcAllowVeryLargeObjects enabled="true" />
</runtime>
However, I can't change my web application to run on x64.
My solution was to make this in batches like this:
int rowCount = 0;
while (reader.Read())
{
DataRow row = dtbl.NewRow();
dtbl.Rows.Add(row);
if (rowCount % _batches == 0 && rowCount != 0)
{
DBInsert(dtbl, selectCommand);
dtbl = CreateTable(TableColumns);
}
}
private void DBInsert(DataTable dt, string selectCommand)
{
using (OracleCommand command = new OracleCommand(selectCommand, _oracleConnection))
{
using (OracleDataAdapter adapter = new OracleDataAdapter(command))
{
using (OracleCommandBuilder builder = new OracleCommandBuilder(adapter))
{
OracleTransaction trans = _oracleConnection.BeginTransaction();
command.Transaction = trans;
adapter.InsertCommand = builder.GetInsertCommand(true);
adapter.Update(dt);
trans.Commit();
}
}
}
}
}
It works, however this is very slow. I was wondering if there is a way to either solve the problem with the memory serially or write in memory in parallel.
I have tried to insert the data in parallel using threads but this takes a lot of memory and throws OutOfMemory Exception as well.

Just don't load 1M rows into a DataTable. Use whatever bulk import mechanism is available to load a stream of rows. Oracle, like SQL Server offers several ways to bulk import data.
Collections like List or DataTable use an internal buffer to store data that they reallocate when it fills up, using twice the original size. With 1M rows that leads to a lot of reallocations and a lot of memory fragmentation. The runtime may no longer be able to even find a contiguous block of memory large enough to store 2M entries. That's why it's important to set the capacity parameter when creating a new List.
Apart from that, it doesn't serve any purpose to load everything in memory and then send it to the database. It's actually faster to send the data as soon as each file is read, or as soon as a sufficiently large number is loaded. Instead of trying to load 1M rows at once, read 500 or 1000 of them each time and send them to the database.
Furthermore, Oracle's ADO.NET provider includes the OracleBulkCopy class that works in a way similar to SqlBulkCopy for SQL Server. The WriteToServer method can accept a DataTable or a DataReader. You can use the DataTable overload to send batches of item. An even better idea is to use the overload that accepts a reader and have the class collect the batch and send it to the database.
Eg :
using(var bcp=OracleBulkCopy(connectionString))
{
bcp.BatchSize=5000;
bcp.DestinationTableName = "MyTable";
//For each source/target column pair, add a mapping
bcp.ColumnMappings.Add("ColumnA","ColumnA");
var reader = excelCommand.ExecuteReader();
bcp.WriteToServer(reader);
}

C# - DataTable Out of Memory exception in application to catch SQL Server "INSERT" events

I have been tasked with creating an application that monitors any "INSERT" events on a specific table. I was going to go about this using SqlDependency to create a notification link between the DB and the C# app, but it turns out I am not able to do this due to security issues.
Due to this, I have modeled my application as follows:
This is well and good, but as it turns out, the SQL table I am querying has a rather large size. The table has nearly 3.5 Million rows 55 columns. When loading into the C# DataTable object, I am getting an out of memory exception.
internal static DataTable ExecuteQuery(string query, Dictionary<string,string> parameters = null)
{
try
{
using (SqlConnection dbconn = new SqlConnection(SQLServer.Settings.ConnectionString))
using (SqlCommand cmd = new SqlCommand())
{
dbconn.Open(); // Open the connection
cmd.CommandText = query; // Set the query text
cmd.Connection = dbconn;
if (parameters != null)
{
foreach (var parameter in parameters) // Add filter parameters
cmd.Parameters.AddWithValue(parameter.Key, parameter.Value);
}
var dt = new DataTable();
using (SqlDataAdapter adpt = new SqlDataAdapter(cmd)){adpt.Fill(dt);} // MY ERROR OCCURS HERE!
dbconn.Close();
queryError = false;
return dt;
}
}
catch(Exception ex)
{
queryError = true;
EventLogger.WriteToLog("ExecuteQuery()", "Application", "Error: An error has occured while performing a database query.\r\nException: " + ex.Message);
return null;
}
}
When running the code above, I get the following error at the line for SqlDataAdapter.Fill(dt)
Exception of type 'System.OutOfMemoryException' was thrown.
Is there a way that I can either restructure my application OR prevent this incredibly high memory consumption from the DataTable class? SQL server seems capable enough to do a select * from the table but when I fill a DataTable with the same data, I use up over 6GB of RAM! Why is there so much overhead when using DataTable?
Here is a link to my flowchart.

I was able to resolve this issue by making use of the SqlDataReaderclass. This class lets you "stream" the sql result set row by row rather bringing back the entire result set all at once and loading that into memory.
So now in step 5 from the flow chart, I can query for only the very first row. Then in step 6, I can query again at a later date and iterate through the new result set one row at a time until I find the original row I started at. All the while, I am filling a DataTable with the new results. This accomplishes two things.
I don't need to load all the data from the query all at once into local memory.
I can immediately get the "inverse" DataSet. AKA... I can get the newly inserted rows that didn't exist the first time I checked.
Which is exactly what I was after. Here is just a portion of the code:
private static SqlDataReader reader;
private static SqlConnection dbconn = new SqlConnection(SQLServer.Settings.ConnectionString);
private void GetNextRows(int numRows)
{
if (dbconn.State != ConnectionState.Open)
OpenConnection();
// Iterate columns one by one for the specified limit.
int rowCnt = 0;
while (rowCnt < numRows)
{
while (reader.Read())
{
object[] row = new object[reader.FieldCount];
reader.GetValues(row);
resultsTable.LoadDataRow(row, LoadOption.PreserveChanges);
rowCnt++;
sessionRowPosition++;
break;
}
}
}
The whole class would be too large for me to post here but one of the caveats was that the interval between checks for me was long, on the order of days, so I needed to close the connection between checks. When closing the connection with a SqlDataReader, you loose your row position so I needed to add a counter to keep track of that.

Check you query for select. You probably get from database many rows.

Npgsql C# throwing Operation is not valid due to the current state of the object suddenly

I have a C# winform application and everything was working fine, but after prolonged use of the app I started getting this error
Operation is not valid due to the current state of the object
The error is from a function that executes every 5 seconds to get a list of names from the database
NpgsqlCommand cmd = new NpgsqlCommand(
String.Format("select pid,name from queue order by id"), conn);
NpgsqlDataReader reader = cmd.ExecuteReader();
while (reader.Read())
{
queue[(int)reader["pid"]] = (string)reader["name"];
}
This list contains names in a queue and needs to be updated in as short time as possible
From what I have read, it seems like a new limitation from .net framework..
Any better way to do this or a workaround to avoid this error?
Edit: and btw, I dont understand the limitation! I added a function that enters more than 100000 entries into the database and I didn't get this error!

Do you dispose reader and cmd after use? This could be memory leak-related where the postgres-provider ends up running out of an internal resource after some time.
You should follow a using-pattern like described on their homepage: http://www.npgsql.org/doc/
using (NpgsqlCommand cmd = new NpgsqlCommand(
String.Format("select pid,name from queue order by id"), conn))
{
using (NpgsqlDataReader reader = cmd.ExecuteReader())
{
while (reader.Read())
{
queue[(int)reader["pid"]] = (string)reader["name"];
}
}
}

How to Update a BLOB column, error ORA-00932, while Insert works

I cannot update a BLOB field, but Insert works, see code below.
My guess is that it has something to do with the problem of storing one BLOB value in lots of records, involving copying large data.
In my case, I know that only one record will be updated, but Oracle might be of the opinion that potentially several records may need to be updated. With Insert, there is guaranteed only 1 record involved, but not always with Update. Now how do I get around this problem?
NB: the ArtNr field in the Where-clause is a primary key with a Unique index.
By the way, I find it worrysome that there are lots of code examples for Insert BLOB on the internet, but I could not find one for Update BLOB.
using Oracle.DataAccess.Client;//needs reference to Oracle.DataAccess.dll
using Oracle.DataAccess.Types; //OracleBlob
public static bool StoreBlobImage(OracleConnection conn, string ArtNr, byte[] bImageJpg)
{
bool Ok = false;
#if true // this is what I need, but does not work
string Sql = "update MyTable set Image = :Image where ArtNr = :ArtNr";
#else // this works
string Sql = "insert into MyTable (ArtNr, Image) values (:ArtNr, :Image)";
#endif
using (OracleCommand cmd = new OracleCommand(Sql, conn))
{
//cmd.Connection = conn;
//cmd.CommandType = CommandType.Text;
cmd.Parameters.Add("ArtNr", OracleDbType.Varchar2, 8).Value = ArtNr;
#if false // tried method 1
cmd.Parameters.Add("Image", OracleDbType.Blob).Value = bImageJpg;
#else // now trying method 2
OracleParameter blobParameter = new OracleParameter();
blobParameter.OracleDbType = OracleDbType.Blob;
blobParameter.ParameterName = "Image";
blobParameter.Value = bImageJpg;
blobParameter.Direction = ParameterDirection.Input;
blobParameter.IsNullable = true;
cmd.Parameters.Add(blobParameter);
#endif
try
{
conn.Open();
cmd.ExecuteNonQuery(); // ORA-00932: inconsistent datatypes: expected - got BLOB
}
catch (Exception TheException)
{
}// debug breakpoint
}
return Ok;
}

I really though you were imagining things when I read your post. Out of curiousity, I tried it and was amazed that this error really does occur.
There is good news. I poked around and found this:
How can I update data in CLOB fields using a >> prepared query << with ODP (Oracle.DataAccess)?
It turns out when using an update statement with an LOB, the LOB has to be declared first in the parameters. With that in mind, I got the same error you did with your code, but this worked perfectly:
public static bool StoreBlobImage(OracleConnection conn, string ArtNr, byte[] bImageJpg)
{
bool Ok = false;
string Sql = "update MyTable set Image = :Image where ArtNr = :ArtNr";
using (OracleCommand cmd = new OracleCommand(Sql, conn))
{
cmd.Parameters.Add("Image", OracleDbType.Blob).Value = bImageJpg;
cmd.Parameters.Add("ArtNr", OracleDbType.Varchar2, 8).Value = ArtNr;
try
{
cmd.ExecuteNonQuery();
}
catch (Exception TheException)
{
}
}
return Ok;
}
Simply by switching the parameters.
I gave a kudo to the question and answer of that original question (same guy, in this case).
You are right, there is precious little on the web in the way of help for updates on BLOBs in Oracle.
Great question. I feel like I learned something today.
-- EDIT --
Per OP's suggestion, there is another fix, per the same thread referenced above, that can prevent the necessity of rearranging the parameters. My guess is this might also come in handy if you are updating multiple LOBs.
Switching the BindByName Property appears to also resolve the issue:
cmd.BindByName = true;