Today I've implemented a nasty hack in my code where every request to the database opens it's own connection due to the fact that I couldn't find any way to enable MARS (multiple active record sets) when communicating with a MySQL database.
In my C# program I do a lot of parallel work, which isn't a problem regarding databases such as MSSQL 2005 and 2008 (append ;MultipleActiveResultSets=true to your connection string) and SQLite (supports it "out of the box") and you are able to retrieve two datasets from the database at the same time.
Things that I do know: it's expensive to open a connection to the database and their for I would like to keep these to a minimum.
Any suggestions?
Maybe a best way to handle this scenario type is to implement that parallel data processing into your database, by using a store procedure or a cursor, so you don't need to deal with a very specific database feature.
Any suggestions?
I don't know if there really is no way to enable MARS with MySQL, but if that's correct, then my best suggestion is to implement connection-pool.
Look at the MySQL documentation for connection string parameters (no MARs) -
http://dev.mysql.com/doc/refman/5.5/en/connector-net-connection-options.html
Things that I do know: it's expensive to open a connection to the database and their for I would like to keep these to a minimum.
Utilize connection pooling all the way! (just make sure you use the exact same connection string each time).
Mysql doesn't support MARS,instead you have to save the data and close the reader or use another connection for the new reader.
Related
I am current building (in C#) a fairly basic point-of-sale program for a local community in Uganda to use in tracking business at their sunflower seed press. I was thinking that I would need some sort of database (like a SQL database), but I've never set up a database before, so I'm wondering what the best way to do this is. Maybe a database isn't the best way. The program will not have internet access, so everything will have to be done locally on the machine.
I think your first step should be designing out what data you need to store. Build an Entity Relationship Model and decide what your domain model is going to be. There are many different Database Engines out there that you can use that have different features, installation requirements, etc. A database engine can be installed locally, or on a remote machine to connect to. If you're writing a C# app, you'll probably want to use the System.Data namespace. You can use plain ADO .NET, or use something like Linq To Enttiies to help create proxy classes for your data tables.
You can access a SQL database using the same API for queries / record extraction regardless of the DB Engine uses. In some caess, you may need to use a seperate library that provides an implementation (or a better one), as in the case of an Oracle Database and the Oracle Data Access Components. Right out of the gate, .NET works very well with Microsoft SQL Server, but other options would work.
The details of what database engine are not as important as defining a good set of data tables to represent your data.
Yes. If it has lots of data you have to consider using database. Whether you have internet or not, as long as you have local network, you can easily do database.
Set up a database server ( maybe sql)
Do your database and install it on the database server
Do your application and connect to your database through connection string.
You are on the right track to use a database to store data. It is pretty easy to accomplish. Your computer does not need to be connected to the internet.
SQL Server Express Edition is free with a limit of 10 gigs of data. This will probably be much, much more space than you will need.
From C#, use ADO.NET. It is very simple if you know some SQL. Code samples here.
I've installed StreamInsight 2.1. I've been able to turn it on and Server.Create(), expose a WCF endpoint, and connect to it with the Event Flow Debugger. I've found plenty of examples of creating fake event sources.
I'd like to know how to create what I think to be possible: a standing query against a SQL 2012 database. I haven't found any sample code for connecting to an RDB - just statements that "This is just for show, for real you'll maybe connect to an RDB".
Can it only use LINQ-to-SQL? Entity Framework 4? plainer ADO.NET? Some MS SQL-specific protocol?
I guess the part that really confuses me is how these standing queries are supposed to work. It would seem odd to me, these days, for Microsoft to have an exposed capability in SQL Server that is ONLY accessible by a certain independent SKU (StreamInsight). Therefore, it must be one of these two:
Is SQL aware of the standing-ness of the query or does StreamInsight just modify a LINQ statement to poll efficiently (like by adding AND #LastExecution <= InsertedOn and proper indexes).
If SQL is aware of it as a special kind of query, what mechanism does StreamInsight use to create such a standing query? Is it something that a programmer can utilize without StreamInsight Services and so on?
The standing queries you mentioned are the ones written using LINQ that run inside the StreamInsight engine and not SQL Server itself. With StreamInsight, we the developers are on the hook for writing the adapter/source/sink code to get data in and out.
Can it only use LINQ-to-SQL? Entity Framework 4? plainer ADO.NET? Some MS SQL-specific protocol?
Yes. I have written this code before and I've had good luck with just plain ADO.NET. For what our needs were, it was way more flexible than LINQ-to-SQL and Entity Framework.
Do you have control over the schema of the table(s) you are querying? If you do, I would suggest adding some kind of LastUpdated datetime column that gets updated when the data in your table changes. That way you can write the query in such a way that only pulls back the changes without returning the whole table itself. Your adapter/source will need to have some kind of timer that executes your SQL statement periodically. You'll also need to maintain the last executed datetime so you can plug that into your SQL statement.
You could create a StreamInsight input adapter for SSSB to generate events based on messages from Server Broker queues.
I've not worked through this solution, but it seems to me that it would be possible to implement Service Broker or a SQL query notification in SQL Server to message the application on data change, then use Streaminsight to process and respond to this altered state.
I think this would give the effect of a "standing query" and avoid the (potentially wasteful) overhead of polling.
I'd like to know how to create what I think to be possible: a standing query against a SQL
2012 databas
And here we go. Not possible. Period. Why you think that is possible? Need to poll, the best is the ever waiting poll for the message queue, but no, standing query NOT POSSIBLE. Too bad.
Where did you get the funny idea it is possible?
Right now I am having a customer who is working with several businesses. He is working with their data but is not allowed to directly access their databases. We thought of using SQLite or SQL CE and storing a copy/part of the original database as a file on a network share. Now the problem is that SQL CE is not supporting it and SQLite highly recommends not to do so.
First of all the performance is a huge problem, since our customer is working with a lot of data (up to several gb). The second problem is that SQLite has problems (actually the underlying os functionality for file locking is the problem) with concurrent usage of the database, when it is stored on a network share. I did a lot of research on that topic and many people say that it is just a matter of time that the database gets currupt.
Does anyone know a better solution to that problem or a workaroung which lets me use SQLite? It does not need to be a file based database, as long as nothing needs to be installed or run on the server.
Thanks, David.
If you are going to store data on a network share and have concurrent users accessing it you are going to need a db that can handle concurrent access. MS Access will quickly die if under concurrent access as will SQL Lite.
SQL Server Express is free and works very well. PostgreSQL as suggested by Maxim is an open source full featured db that will do the job very well but may be overkill.
You could also look at Redis ... fast lightweight in memory no sql db that also has capability to persist to file.
You can try PostgreSQL. It is very easy to configure, and is rather reliable. It also support server export/import options.
And any of this makes sense, if you client is able to get his hands on an exported database somehow.
I am trying to migrate data from an Informix database to SQL Server 2008. I've got quite a lot of data to move. I've been try multiple methods to get the data over, and so far SQLBulkCopy in multiple chunks seems to be the fastest that I can find. Does anyone know of a faster means of getting the data over? I'm trying to cut down on the transfer time so that on my cut-over date I don't run out of time to do the full cut-over. Thanks.
As you mentioned, I think that the bcp command is the fastest solution.
you can make csv file from your data and then import those to your db by bcp command.
There isn't much more you can do to get this work completed faster. One thing you might want to look at though is the recover model for the sql database. If it's currently set to Full, you're going to end up slowing down quite a bit as the transaction log fills up.
http://msdn.microsoft.com/en-us/library/ms189275.aspx
Hope that helps.
If you can use an Ole or ODBC connection to your Informix database, then SSIS may be the best option.
I asked a question a while ago about which local DB was right for my situation. I needed to access the DB from both .NET code and VB6. The overwhelming response was SQLite. However, I decided to pass on SQLite, because the only OLE DB provider for it charges royalties for every deployed copy of my software. It also requires an activation procedure to be run on every single PC.
After evaluating other options (SQL Server Compact edition - barely functional OLE DB provider, Firebird - don't want to have to pay for another driver, etc...), I've come to conclusion that the only viable choice is using .MDB files created by Microsoft Access (or the Jet engine).
I haven't used it since late 90s, so I have the following questions to those who have experience with it.
Have they resolved the problem where the database would corrupt every now and then.
Is access to the MDB from c# accomplished via the ADO.NET OLEDB Provider or is there a native solution (i can't seem to find it).
Is there a viable alternative to the really crappy SQL Editor in Access?
Thanks.
Rather then going "back" to Access, I'd stick with SQLite and use the System.Data.SQLite provider for SQLite data access within the .NET code.
Then I'd just create a simple COM interop .NET class for use by VB6 that wraps any required SQLite data access functionality. Finally, just reference and use it like a standard COM object from your VB6 projects.
My knowledge of Access is probably a bit dated and biased by bad experiences, but within reason I would try most other options before resorting to the Access route.
Have you considered SQL Server 2008 Express Edition (as oppose to SQL Server CE)?
1) Personally, I found that most times that Access DBs corrupted it was due to code that didn't clean up after it self, or there was a faulty Network card involved.
2)
string connectionString = #“Provider = Microsoft.Jet.OLEDB.4.0; " +
#"Data Source = C:\data\northwind.mdb; " +
#"User Id = guest; Password = abc123”
using (OleDbConnection oleDbConnection = New OleDbConnection())
{
oleDbConnection.ConnectionString = connectionString;
oleDbConnection.Open();
...
}
3) SQL Server 2008 Express Edition
MDB corruption is largely due to failures that occur in client machines, file servers, and networks while the database is open. If you put the MDB on a file share this is always a risk, if on a local hard drive and used by one user the problems are much rarer.
I would not expect SQLite to be any different, and if anything worse.
Periodically running JetComp.exe (a Microsoft download) will fix many problems and compact index tables and such. Backups are important no matter what you use.
You don't need MS Access at all to use Jet MDBs. There are some 3rd party tools for designing the database schema and doing interactive queries, both command line and GUI.
Since the MDB format is more or less deprecated, your late 90s knowledge is quite up to date. See this MSDN page
You could also try SQL Anywhere it runs on various OS and has a small footprint. Works for me :)
AngryHacker asked:
Q1. Have they resolved the problem where the database would corrupt every now and then.
Er, what?
There was never any corruption problem in properly engineered apps properly deployed in properly maintained environments. I haven't seen a corrupted MDB in 3 or 4 years, and I have dozens of my apps in full-time production use by many clients in many different types of operating environments.
I think that most people who experience corruption are those who try to share an MDB file among many users (whether split or unsplit). Since you're not contemplating using Access, that's not really an issue.
Q2. Is access to the MDB from c# accomplished via the ADO.NET OLEDB Provider or is there a native solution (i can't seem to find it).
The native solution would be DAO, but that's COM, so you might not want to use that. From C#, I'd say OLEDB is your best bet, but that's not my area of expertise so take it with a grain of salt. I believe that Michael Kaplan reported that the Jet ADO/OLEDB provider was thread-safe, while DAO is not. This doesn't mean he recommended ADO/OLEDB over DAO, though, but his comments also came in an Access context, and not C#.
Q3. Is there a viable alternative to the really crappy SQL Editor in Access?
Why would you be using that when you're not actually using Access? You could use any SQL editor you like as long as you test that the SQL you write is compatible with Jet's SQL dialect.
I, for one, don't see what the issue is with Access's SQL editor (other than the inability to set the font size), but then, I write a lot of my SQL using the QBE and don't ever even look at the SQL view.
To answer your question regarding the really crappy SQL editor in Access - I wholeheartedly agree. The font stinks, MSAccess always badly reformats the query, it sometimes adds in metacharacters that break my SQL, and lastly but worstly, if it can't parse the SQL, it won't let you have access to it!
My solution is to use external code. I use DAO to instantiate MSAccess and can then directly edit the queries using the QueryDefs collection. It lets you do most things - create, rename, edit, etc. There are a couple of things you cannot do this way though - for example, you do not have access to the query metadata (description, hidden, etc).
External code is also great because you can build a suite of test cases, specifying expected return values, etc.