We Keep Coding

Too many Oracle connections open

I have a .NET Core web api on IIS running and I'm using oracle as database and I do connect to it like this.
using(OracleConnection con = new OracleConnection(connectionString))
{
OracleCommand cmd = new OracleCommand();
//some other code here
con.Open();
}
I'm not using EF or so. I (de)serialze the data from the reader directly into a json or xml string.
I have a small batch file to test the api. The batch sends a request using curl every second and when I run the batch file 5 times, to simulate a little bit of traffic, (I know there are some tools for that, but thats not the problem) the api has open 7 connections to the database. Why are 7 db connections open, how to handle it, so that a maximum of 2 or 3 simultaneously are open, even if a request has to wait?
I do not want to "just make it work", I want it to work the right way. Because of that, I want to now, is "connection pooling" the keyword here? Especially the max pool size, could I just set it to 3, or did I forget something to set up, or something?

Frequent Opening and Closing of MySQL Database over Cloud

I have a local server MS-SQL situated in one of our branch. I want to transfer new data from the local server to MySql table (over the cloud) every 1 minute.
I have coded a small C# application which opens both the server connections, search and insert the new rows into MySql Database and then close the connection.
Now my question is, as I am continously opening, updating and closing the Mysql connection every minute; will there be any issues? Is there any other alternate method through which I can establish a single connection to MySql database and then keep on inserting the new rows every minute.
I appreciate your valuable support.
Below is the coding part I use to open, update and close the connection: Remember, the below method is executed every minute.
if (queryValues != "")
{
queryValues = queryValues.Remove(queryValues.Length - 1);
query = query + queryValues + ")";
MyCommand3 = new MySqlCommand(query, MyConn3);
if (MyConn3.State == 0)
MyConn3.Open();
MyReader3 = null;
MyReader3 = MyCommand3.ExecuteReader();
MyCommand3.Dispose();
MyReader3.Dispose();
MyConn3.Close();
}

It is not a problem.
Connector/NET, the MySQL driver for C#, offers connection pooling by default. That is, it keeps the connection between your program and MySQL open, so you can re-use it with another .Open() operation.
It only closes the connections when they get to be three minutes old, so you should be fine with a once-per-minute operation. And, it manages stuff like lost connections pretty well, so you don't have to.
What's more, opening a connection isn't a high-overhead operation. Only if you do it multiple times a second does the overhead get out of hand.

Database connection on .mdb slows down over time

Our application connects to a .mdb-file on Network. All was fine, untill we swapped the Computer from a 32-bit Windows 7 to 64 bit Windows 10. Since this action, connecting to the database through our C# code is getting slower over time. It starts with something lower than 1 second after starting the application. After running the application ca. 8 hours the opening of the database file takes more than 10 seconds, and rising. Ca. every 5 seconds the access database gets connected.
After a restart of the application all is fine. The frequently restart isn't an long term option for our customer. The queries itself are done in some ms and closing is ok, too.
I've seen that the processor usage is only about 10%. Memory i don't know at the moment.
Has anybody an idea why connecting to the database get slowed down over time?
public void OpenDb ( string _sOpenString )
{
this.sFilePathToAccessDb = _sOpenString;
this.sConnectionString =
#"Provider=Microsoft.ACE.OLEDB.12.0;DataSource=" + this.sFilePathToAccessDb;
this.oOleDbConnection =
new System.Data.OleDb.OleDbConnection( this.sConnectionString);
this.oOleDbConnection.Open();
}
public void CloseDb()
{
if (this.oOleDbConnection != null)
{
this.oOleDbConnection.Close();
this.oOleDbConnection.Dispose();
}
this.oOleDbConnection = null;
}
public void foo()
{
OpenDb(#"\\fooserver\databases\bar.mdb");
//do some stuff
CloseDb();
}

Hence after several testing it turned out, that we've installed Microsoft Access Database Engine 2010 version 14 on the machine. After uninstall this version and install version 12.0 all is fine. Opening a Database on a network drive takes stable 30 ms.

Azure Worker role processing is very slow

I have a C# ETL process which run once in a week and it takes 6 hours to complete on-premises windows server.
Here is the C# class structure.
Source database: Firebird database files downloaded everytime on disk, takes 20 mins to download.
Destination: SQL Server (on-prem)
Load process steps are here.
Do something
Do something
Execute parallel
We have 5 firebird independent db files which work on 5 different tables so set the max degree of parallelism to 5.
Parallel.ForEach(destTables, new ParallelOptions { MaxDegreeOfParallelism = 5}, (eachtable) =>
{
var tableName = eachtable.ToString(CultureInfo.InvariantCulture);
lock (tableName)
{
Thread.Sleep(10000);
readTable.BulkLoad(tableName, srcConForMainFile, destConForSQL);
Thread.Sleep(10000);
}
});
Do something
Complete the process
Now, I moved this process to Azure worker role.
Source database: Firebird database files downloaded on worker role disk (local storage=100GB set in .csdef file), taking 20-30 mins to download which is fine.
Destination db: Dedicated SQL Azure database of S3 Standard (100 DTUs) created in the same region where is worker role is created
I have setup a Large size worker role (4 cores, 7 GB RAM, High Net Bandwidth, 999 GB disk size) but the process took 20 hours to complete.
I also noticed that the CPU utilization has gone upto maximum 25% at some time and RAM used upto 2.5 or 3GB. That's it.
Is Parallel.ForEach working really in the worker role VM ?
How to verify Parallel execution is happening in worker role VM ?
Should we still increase the db to a higher pricing tier ?
Are there any other settings should be made to worker role VM so that the process runs much faster - 6hrs vs 20hrs ?

Error - LINQ/TransactionScope with multiple database connections

I'm having a helluva time wrapping a couple transactions to 2 different databases on the same SQL Server. I initially was having trouble with network DTC access and I resolved that. Now, the error that I continue to get is "Communication with the underlying transaction manager has failed."
We have some customer profiles in a database and when these profiles become outdated we want to move them to an 'archive' database for storage. The move is simply (italics for humor) adding them to the archive database and deleting them from the main/live database. I have a DataContext for each database. The code below performs the Add and then gets the error on the Delete when trying to use the second DataContext. I've only been working with LINQ for a few months and I've scoured articles for the past couple of days. I'd like to know if anything is wrong with my code or if there is still something not configured properly with the DTC or ???
We're running on VMware for my workstation and the server.
- Workstation is Windows 7 SP1
- Server is Windows and SQL Server 2008R2
Routine for the 'Move':
private int MoveProfileToArchiveDB( int iProfileId )
{
int rc = RC.UnknownError;
// get new Archive profile object
ProfileArchive.ProfileInfo piArchive = new ProfileArchive.ProfileInfo();
// 'Live' DataContext
using ( ProfileDataContext dbLive = new ProfileDataContext() )
{
// get Live profile
ProfileInfo piLive = ProfileInfo.GetProfile( dbLive, iProfileId );
// copy Live data to Archive profile object... including the id
ProfileArchive.ProfileInfo.CopyFromLive( piLive, piArchive, true );
}
bool bArchiveProfileExists = ProfileArchive.ProfileInfo.ProfileExists( piArchive.id );
// make the move a transaction...
using ( TransactionScope ts = new TransactionScope() )
{
// Add/Update to Archive db
using ( ProfileArchiveDataContext dbArchive = new ProfileArchiveDataContext() )
{
// if this profile already exists in the Archive db...
if ( bArchiveProfileExists )
{
// update the personal profile in Archive db
rc = ProfileArchive.ProfileInfo.UpdateProfile( dbArchive, piArchive );
}
else
{
// add this personal profile to the archive db
int iArchiveId = 0;
piArchive.ArchiveDate = DateTime.Now;
rc = ProfileArchive.ProfileInfo.AddProfile( dbArchive, piArchive, ref iArchiveId );
}
// if Add/Update was successful...
if ( rc == RC.Success )
{
// Delete from the Live db
using ( ProfileDataContext dbLive = new ProfileDataContext() )
{
// delete the personal profile from the Profile DB
rc = ProfileInfo.DeleteProfileExecCmd( dbLive, iProfileId ); // *** ERROR HERE ***
if ( rc == RC.Success )
{
// Transaction End (completed)
ts.Complete();
}
}
}
}
}
return rc;
}
NOTES:
I have a few different methods for the Delete and they all work outside the TransactionScope.
ProfileInfo is the main profile table and is roughly the same for both Live and Archive databases.
Any help is greatly appreciated! Thanks much...

Rather than continue criss cross comments, I decided to post this as an answer instead.
don't use error codes. That's what exceptions are for. The code flow is more difficult to read and error code returns invite to be ignored. Exceptions make the code easier to read and far less error prone.
If you use a TransactionScope, remember to always set the isolation level explicitly. See using new TransactionScope() Considered Harmful. The implicit isolation level of SERIALIZABLE is almost never called for and has tremendous negative scale impact.
Transaction escalation. Whenever multiple connections are opened inside a transaction scope they can escalate the transaction to a distributed transaction. The behavior differ from version to version, some have tried to document it, eg. TransactionScope: transaction escalation behavior:
SQL Server 2008 is much more intelligent then SQL Server 2005 and can
automatically detect if all the database connections in a certain
transaction point to the same physical database. If this is the case,
the transaction remains a local transaction and it is not escalated to
a distributed transaction. Unfortunately there are a few caveats:
If the open database connections are nested, the transaction is still
escalated to a distributed transaction.
If in the transaction, a
connection is made to another durable resource, the transaction is
immediately escalated to a distributed transaction.
Since your connection (from the two data contextes used) point to different databases, even on SQL Server 2008 your TransactionScope will escalate to a distributed transaction.
Enlisting your application into DTC is harmful in at least two ways:
throughput will sink through the floor. A database can support few thousand local transactions per second, but only tens (maybe low hundreds) of distributed transactions per second. Primarily this is because of the complexity of two phase commit.
DTC requires a coordinator: MSDTC. The [security enhancements made to MSDTC] make configuration more challenging and it certainly is unexpected for devs to discover that MSDTC is required in their app. The steps described in the article linked are probably what you're missing right now. For Windows Vista/Windows 7/Windows Server 2008/Windows Server 2008R2 the steps are described in MSDTC in Windows Vista and Windows Server 2008, in How to configure DTC on Windows 2008 and other similar articles.
Now if you fix MSDTC communication following the articles mentioned above, your code should be working, but I still believe this archiving should not occur in the client code running EF. There are far better tools, SSIS being a prime example. A nightly scheduled job running SSIS would transfer those unused profiles far more efficiently.