I have a local server MS-SQL situated in one of our branch. I want to transfer new data from the local server to MySql table (over the cloud) every 1 minute.
I have coded a small C# application which opens both the server connections, search and insert the new rows into MySql Database and then close the connection.
Now my question is, as I am continously opening, updating and closing the Mysql connection every minute; will there be any issues? Is there any other alternate method through which I can establish a single connection to MySql database and then keep on inserting the new rows every minute.
I appreciate your valuable support.
Below is the coding part I use to open, update and close the connection: Remember, the below method is executed every minute.
if (queryValues != "")
{
queryValues = queryValues.Remove(queryValues.Length - 1);
query = query + queryValues + ")";
MyCommand3 = new MySqlCommand(query, MyConn3);
if (MyConn3.State == 0)
MyConn3.Open();
MyReader3 = null;
MyReader3 = MyCommand3.ExecuteReader();
MyCommand3.Dispose();
MyReader3.Dispose();
MyConn3.Close();
}
It is not a problem.
Connector/NET, the MySQL driver for C#, offers connection pooling by default. That is, it keeps the connection between your program and MySQL open, so you can re-use it with another .Open() operation.
It only closes the connections when they get to be three minutes old, so you should be fine with a once-per-minute operation. And, it manages stuff like lost connections pretty well, so you don't have to.
What's more, opening a connection isn't a high-overhead operation. Only if you do it multiple times a second does the overhead get out of hand.
Related
I have a MySql server with the following config changes:
max_allowed_packet = 16M
max_connections = 2000
Innodb_cache_pool_size = 90G
I have a multi-threaded .netcore2 application executing 1000+ select queries per thread every couple of seconds.
My server environment is Ubuntu 16.04 cloud with the mysql package
This starts out fine and everything works fast but after a short amount of time, most connections change to "Sleep" mode and those who don't perform 5-10 selects every couple of seconds.
As far as resources go, I am using about 70% of all my cpus and abour 30% of my RAM.
my server is on an intranet network so I cannot copy the entire contents, but I'll try and give all the relevant info possible:
128GB RAM,
20 VCPUS (i7 cpus),
Dedicated server
All DBS are using the InnoDB engine with the Barracuda type.
Any help would be appreciated!
EDIT: as mentioned this is on an intranet network and I'm not allowed to copy any code.. I'll do my best to provide something similiar:
List<string> itemLst=new List<string>();
//fills the list here
ParallelOptions po=new ParallelOptions();
po.MaxDegreeOfParallelism = 45;
Parallel.ForEach(itemLst,po,item=>
{
//Open connection to MySql server
//do some SELECT queries
//do 1 insert and 1 delete query
//close and dispose connection
}
In my Unity3D game the Player plays himself through short levels, which at the beginning take about 4 seconds. The goal is to get the best possible clear time in each level. I am currently saving these clear times locally, but I would like to upload them to an SQL Server, to be able to create a leaderboard for each level.
Since performing the SqlConnection.Open() command takes about 1-2 seconds, sometimes also 3, I was wondering whether I should keep a static connection always open, ready to execute any queries that I want to run.
Are there any unwanted and dangerous side-effects when doing it?
Edit: This is the code that I use for opening the SqlConnection.
SqlConnectionStringBuilder builder = new SqlConnectionStringBuilder()
{
DataSource = dataServer,
UserID = userId,
Password = password
};
SqlConnection connection = new SqlConnection(builder.ToString());
connection.Open();
First I'll answer this question:
Are there any unwanted and dangerous side-effects when doing it?
Assuming you keep this code in your Game (client) and the SQL Server is not client-side but in a server of yours located somewhere, a simple Client Reverse Engineer will be able to grab your credentials from your connection string and use them to gain Unauthorized access to your database. (NEVER trust the client)
With that being said, I would suggest you use a simple server-side technology (php,Socket App, Java servlet, etc..) along with the SQL that the client will send the information there and then to the database.
Example:
1) Client-> Posts Data -> PHP
2) PHP -> Connects -> SQL Database
3) PHP -> Inserts data -> SQL Database
Like this you can also retrieve the results of your ladder from the database by asking php (or whatever technology you are using) to send your data to the client.
Best wishes in your progress, and feel free to contact me for any additional information!
I am having a very strange problem and am hoping someone out there has had a similar experience.
My companies application for one client is getting "banned" from the SQL Server at the beginning of our application. The behavior is strange. I'll write it out in point form.
SQL Connections are created, data is retrieved, the connections are closed, talk to another datasource and then denied access to SQL Server.
Here's the long winded version:
.NET application connects to database multiple times. Gets some data, does some work. It then goes to get some more data and then gets an error that the "SQL Server cannot be found or access is denied". If the process is started over again without re-starting the app then no more connections are able to be made to SQL Server. All new connections result in "SQL Server cannot be found or access is denied". If the application is restarted then it will repeat the above process.
This is the first in 5 years of my experience with the software to have this problem. The application does have code written in Delphi 7. The dephi 7 / VBA code has not issues. My .NET code that performs the actual query looks like:
protected abstract DbConnection GetConnection();
protected abstract DbDataAdapter GetDataAdapter(DbCommand cmd);
protected abstract DbCommand GetCommand(DbConnection conn, String sql);
protected abstract DbCommandBuilder GetCommandBuilder(DbDataAdapter adapter);
public virtual DataTable Query(string sql)
{
var dt = new DataTable();
using (var conn = GetConnection())
{
try
{
using (var cmd = GetCommand(conn, sql))
{
using (var adapter = GetDataAdapter(cmd))
{
adapter.Fill(dt);
}
}
}
catch (Exception ex)
{
throw new SqlStatementException(sql, ex);
}
}
return dt;
}
It is my own quite and dirty DAL. When it is used it is using an OleDbConnection.
Note: Due to legacy code the connection string is configured for OleDbConnection. After taking a moment to review my code I do have the ability to change the connection type to SqlConnection. I haven't tried that yet.
On the client's machine I have not been able to reproduce the issue outside of the main application. I tried creating a little app that would make 100 calls back to back using the format above with an OleDbConnection but it executed successfully.
The failure in the main app happens in the same spot. That should give me a clue except I cannot make sense of it since it is making duplicate query, getting the same data. But I will say that the application talks to two data sources and transfers data from one to the other. Before it does the transfer it does some validation on the sources. So it talks to another database (proprietary file based) via ODBC and comes back successfully and then fails when trying to talk to SQL Server through OleDbConnection.
My suspicion is something is happening in the connection pool. That is causing a failure which in turns causes a denial of access.
Other interesting points. All worked fine for about a year, client got a new machine a couple of months ago, all work fine and then suddenly stopped. I put the application on another machine at the client's site and all worked well for a week and then the same issue appeared. We turned everything off on the client's machine but the issue persisted. I thought firewall but no luck there.
Any assistance is greatly appreciated.
Was gonna put this in a comment, but it got too big :-)
I see your connection-creating methods are abstract. This of course means that derivatives can do all sorts of bad things when they create the connection. I'd look there first.
One thing I found in a similar situation...if you're doing something in the code that creates the connection that makes the connection string unique, you won't be reusing those pooled connections. So...doing something like adding an "App=MyApp" + an incrementing number, date/time, or guid, it will destroy your ability to use pooled connections. When this happened to me, it took me forever to figure it out.
If your application was "slow enough" in the past, such that "old" pooled connections fall out of the pool, you might never see a problem...but then, say a customer gets hot new hardware...and blam...weird errors from nowhere! This might not be what's happening to you, but maybe it will give you some ideas about where to look. Good luck!
I am getting a timeout when executing an SQL SELECT.
I connect to the database thus:
using (SQLConnection conn = SQLConnection(#"Data Source=mydbServer;Initial Catalog=mydb;Integrated Security=true;Timeout=180");
{
conn.Open();
and that successfully connects.
Inside the using{} I set:
string query = #"SELECT a.field, a.timestamp " +
"FROM mydb.dbo.myTable1 a WITH(NOLOCK) " +
"LEFT JOIN [myOtherdbServer].myOtherdb.dbo.MyTable2 b WITH(NOLOCK) " +
"ON a.field = b.field " +
"WHERE b.field is NULL " +
"AND a.timestamp >= '2015-05-01 00:00:00.000' " +
"AND a.timestamp < '2015-06-01 00:00.00.000'";
and execute the command thus:
using (SQLCommand queryCmd = new SQLCommand(query, conn)
{
queryCmd.CommandTimeout = 300;
using (SQLDataReader rdr = queryCmd.ExecuteReader())
{
The SQLCommand throws a timeout exception: "The timeout period elapsed prior to the completion of the operation or the server is not responding".
If I use SQL Server Management Studio on the same system my program is running on, and as the same user under which my program is running, and execute the same SELECT, it completes in under 1 second and returns several thousand rows.
This is happening consistently now. It was working a couple of days ago. What should I be looking for? I'm baffled because the same SELECT works from SQL Server Management Studio. Does SQL SMS use a different connection?
"If I use SQL Server Management Studio on the same system my program is running on, and as the same user under which my program is running, and execute the same SELECT, it completes in under 1 second and returns several thousand rows."
There can be several reasons to why the execution time in SQL server management studio is better. For once, results are Cached, it may also very well be that you are lacking indexes on the timestamp column
In addition, is your application server located on the same server as you sql server? If not, this may increase latency and cause timeouts.
Linked servers may be an issue, except for latency considerations, I'm not sure the NOLOCK statement is sufficient to ensure what youre trying to achieve on the remote server. There's a good chance you may need to create a view on the remote server that contains the NOLOCK statement.
This is happening consistently now. It was working a couple of days ago. What should I be looking for? I'm baffled because the same SELECT works from SQL Server Management Studio. Does SQL SMS use a different connection?
Usually when something was working and now it has stopped working, then something was changed. Start troubleshooting until you find what it is, look for index changes, code architecture modifications, even windows updates, anything which may give you a lead, until you are able to restore it.
Additional advice,
try limiting the select statement to TOP 10, just to see you are able to get back results, this may indicate the issue is in the object's size / query execution time, and not with your server / application configuration.
Recently our QA team reported a very interesting bug in one of our applications. Our application is a C# .Net 3.5 SP1 based application interacting with a SQL Server 2005 Express Edition database.
By design the application is developed to detect database offline scenarios and if so to wait until the database is online (by retrying to connect in a timely manner) and once online, reconnect and resume functionality.
What our QA team did was, while the application is retrieving a bulk of data from the database, stop the database server, wait for a while and restart the database. Once the database restarts the application reconnects to the database without any issues but it started to continuously report the exception "Could not find prepared statement with handle x" (x is some number).
Our application is using prepared statements and it is already designed to call the Prepare() method again on all the SqlCommand objects when the application reconnects to the database. For example,
At application startup,
SqlCommand _commandA = connection.CreateCommand();
_commandA.CommandText = #"SELECT COMPANYNAME FROM TBCOMPANY WHERE ID = #ID";
_commandA.CommandType = CommandType.Text;
SqlParameter _paramA = _commandA.CreateParameter();
_paramA.ParameterName = "#ID";
_paramA.SqlDbType = SqlDbType.Int;
_paramA.Direction = ParameterDirection.Input;
_paramA.Size = 0;
_commandA.Parameters.Add(_paramA);
_commandA.Prepare();
After that we use ExceuteReader() on this _commandA with different #ID parameter values in each cycle of the application.
Once the application detects the database going offline and coming back online, upon reconnect to the database the application only executes,
_commandA.Prepare();
Two more strange things we noticed.
1. The above situation on happens with CommandType.Text type commands in the code. Our application also uses the same exact logic to invoke stored procedures but we never get this issue with stored procedures.
2. Up to now we were unable to reproduce this issue no matter how many different ways we try it in the Debug mode in Visual Studio.
Thanks in advance..
I think with almost 3 days of asking the question and close to 20 views of the question and 1 answer, I have to conclude that this is not a scenario that we can handle in the way we have tried with SQL server.
The best way to mitigate this issue in your application is to re-create the SqlCommand object instance again once the application detects that the database is online.
We did the change in our application and our QA team is happy about this modification since it provided the best (or maybe the only) fix for the issue they reported.
A final thanks to everyone who viewed and answered the question.
The server caches the query plan when you call 'command.Prepare'. The error indicates that it cannot find this cached query plan when you invoke 'Prepare' again. Try creating a new 'SqlCommand' instance and invoking the query on it. I've experienced this exception before and it fixes itself when the server refreshes the cache. I doubt there is anything that can be done programmatically on the client side, to fix this.
This is not necessarily related exactly to your problem but I'm posting this as I have spent a couple of days trying to fix the same error message in my application. We have a Java application using a C3P0 connection pool, JTDS driver, connecting to a SQL Server database.
We had disabled statement caching in our the C3P0 connection pool, but had not done this on the driver level. Adding maxStatements=0 to our connection URL stopped the driver caching statements, and fixed the error.