I'm trying to learn to use SQLite, but I'm very frustrated and confused. I've gotten as far as finding System.Data.SQLite, which is apparently the thing to use for SQLite in C#.
The website has no documentation whatsoever. The "original website", which is apparently obsolete from 2010 onwards, has no documentation either. I could find a few blog tutorials, but from what I can tell their method of operation is basically:
Initialize a database connection.
Feed SQL statements into the connection.
Take out stuff that comes out of the connection.
Close connection.
I don't want to write SQL statements in my C# code, they're ugly and I get no assistance from the IDE because I have to put the SQL code in strings.
Can't I just:
Create a DataSet.
Tell the DataSet that it should correspond to the SQLite database MyDB.sqlite.
Manipulate the DataSet using its member functions.
Not worry about SQLite because the DataSet automatically keeps itself in sync with the SQLite database on disc.
I know that I can fill a DataSet with the contents of a database, but if I want access to the entire database I will have to fill the DataSet with all of its contents. If my database is 1 GB, I have just used up 1 GB of RAM (not to mention the time needed to write all of it at once).
Can't I simply take a SQLite database connection and pretend it's just an ordinary DataSet (that perhaps needs to be asked occasionally if it's done syncing yet)?
The answer to the question is no.
No you cannot simply take a SQLite connection pretend it's just a DataSet.
If you don't want to code SQL statements then consider Entity Framework.
Using SQLite Embedded Database with Entity Framework and Linq-to-SQL
You shouldn't treat a DataSet as a database. It's just a result of a query.
You query the database to get a subset of data (you never want ALL the data from your DB) and this subset is used to populate your DataSet.
You are required to synchronize your changes manually because DataSet doesn't know which updates should be a part of which transaction. This is your system knowledge.
The DataSet is an in memory cache and will only synchronize to the underlying data store when the developer allows it. You could put a timer wrapper around in and do it on a schedule but you still need to keep the Dataset and data store synchronized manually.
Storing 1GB+ of data is really not recommended as the memory usage would be very high and the performance very low. You also don't want to be sending that amount of data over a network or god forbid an internet connection.
Why would you want to keep 1GB of data in memory?
Related
I have almost the exact same issue as the scenario (linked) below, but unfortunately i'm unable to recreate the solutions succesfully.
I have a c# application using SQL Bulk Import with a datareader and writetoserver, where it's the SQLDatReader or an OracleDataReader, and i need to add columns to the result set.
I can not do it on the source sql statement.
I can not load a data table first and modify it (as it's 100's of gb's of data, almost a terabyte).
How to add columns to DataReader
can anyone provide a working example an help "push" me over this problem?
I temporarily found a solution of using SQL Server Integration Services (SSIS), but what i found while watching it run is it downloads all the data to a dts_buffer, than does the column modifications and then pumps the data into sql server, try doing that with a couple 100gb of data and it is not a good performing thing, if you can even get your infrastructure to build you a 24 core VM with 128gb of memory).
I finally have a small working example, the codeproject (jdweng) was helpful.
I will pose followup, i've tested with sql server (sqldatareader), need to do a test with oracle data reader.
One of the cases i was trying was converting a oracle unique id (stored as a string) to sql server as a uniqueidentifier. I want convert that on the fly, there is no way to adjust the source oracle statement (ADyson) to return a compatible datatype to sql server. Altering a 1tb table afterwards from varchar(40) to uniqueidentifier is painful, but if i could just change as part of the bulk insert, it'd be quick.
and i think now i will be able to.
I used to work in VB.net and used Dataset's all the time when working with ADO, but now i'm Working in C# and the Research is showing me a lot of Recordsets.
Are they the same thing?
If no what is the difference?
Datasets vs RecordSets
Essentially it is to do with how it fetches the data and allows you to intereact with it, recordsets would typically only allow you to fetch data from one table at a time (with default settings) whereas datasets can retrieve the entire set of data. there is more information at that link
Dataset is a connectionless data holder whereas RecordSet is connection oriented Data holder.Though DataSet you can refer more than 1 table at a time, but in the case of Recordset only 1 table is processed at a time. Through Dataset you can process more than 1 record,but in case of recordset recordset you have to make travesel to each record and after that you can make processing.
a direct quote backing up what i said
Difference between ADO.NET Dataset and ADO Recordset?
ADO.NET is an object-oriented set of libraries that allows you to interact with data sources.
http://www.job4india.in/interview-questions/net-interview-questions
ADO :-
1.It is a COM based library.
2.Classic ADO requires active connection with the data store.
3.Locking feature is available.
4.Data is stored in binary format.
5.XML integration is not possible.
ADO.NET :-
1.It is a CLR based library.
2.ADO.NET architecture works while the data store is disconnected.
3.Locking feature is not available.
4.Data is stored in XML.
Read More :- http://www.job4india.in/net-interview-questions/what-difference-between-ado-and-adonet
No, the DataSet class (System.Data namespace) has the same name in C#. I'm not familiar with anything called recordset in ADO.NET (whereas I am with DataSet). Could you post some examples?
i have a sql proc that returns 20,000 + records and want to get this data into a CSV for a SQL2005 bulk load operation.
i think using a data set is overkill since i need only forward only read access to the data.
now i have a data reader but dont think iterating the data-reader is a good idea cause it will lock the oracle DB i am getting the 20,000 records from for some time while its done doing its thing.
logically i am thinking to create a disconnected snapshot of the data in a data table maybe and use that to generate my csv file.
i dont often develop such ETL apps so i wanted to know whats the gold standard on this type of operation.
thoughts?
also, allow me to mention that this needs to be a console app since CORP rules wont allow linked servers and anything cool - so that means SSIS is out.
Since you are worried about doing the iteration of the datareader yourself i could recommend using SqlBulkCopy class.
It lets you load data into an sql server database from any source than can be read with an IDataReader instance
Might solve your potential locking issue.
I have tables and a lot of sotred procedure that work with sql database.
For demonstration purposes I want to load data into memory (maybe dataset that i can then store in session- the demnonstration is limited so server memory cap won't be a problem ?) from my sql tables and manipulate it with my stored procedures.
Is it possible? Or i need to rewrite all my stored procedures or even replace them with code that works with data set?
Stored Procedures are stored in the database. There's no way to make one of them, all the way in the DB server, operate on some in-memory structure back on the web server--a completely different place!
What you're asking to do is a bit like saying you ordered a pizza, and now that it's been delivered to your house you want the pizza parlor to switch the crust for you on the fly, without the pizza leaving your house. Without some seriously advanced technology that can selectively distort the space-time continuum and perhaps even effect time travel, this will never happen.
If you must change the pizza once it is at your house, you either have to order a new one, or modify it yourself with your own tools.
If you wish for SQL Server to manage your table in memory, there is no guaranteed way to.
You can create a table variable and fill it with the dataset, and it'll probably be stored in memory.
Check this answer for more information:
Does MS-SQL support in-memory tables?
Aditionally, with some server configuration you may be able to put your tempdb into a ramdisk, which would effectively give you leeway to operate not just with table variables and hope, but you can store your dataset in a temporary table and be sure it'll be in RAM. Check this Microsoft's article:
http://support.microsoft.com/?scid=kb;en-us;917047&x=17&y=9
EDIT: I would expect that if the dataset fits in server memory (and their configured per process limit) it would be stored in server's memory. But that's just an educated guess as I'm not familiar with ASP.NET's architecture
You can in theory load your entire database in memory and even store a copy of the database for each session. Whether that is a sound thing to do, is a different discussion. All successful ASP applications I hear of are stateless.
Its absolutely impossible to have SQL Server manipulate your process memory where the ADO.Net data sets reside, through Transact-SQL procedures or any other technology. You are going to have to rewrite all your procedures as CLR methods in your ASP.Net application to operate on the ADO.Net data sets.
Have you considered an in-memory DB like SQLite? Check out my answer to this SO thread. There are other alternatives too.
As a mater of fact, you can do it. It's silly but possible.
You load all tables from SQL server into app.
Create XML document containing data.
Send XML to SQL stored procedure (you have to create new ones) and process it
Talking about pizzas, this is like ordering all possible ingredients to be sent from Domino to your house, you open all boxes, reorder them, and then send evrithing back to Domino to make you pizza you want.
Good luck :D
I just saw this topic: Datatable vs Dataset
but it didn't solve my doubt .. Let me explain better, I was doing connection with database and needed to show the results in a GridView. (I used RecordSet when I worked with VB6 while ago and DataSet is pretty similar to it so was much easier to use DataSet.)
Then a guy told me DataSet wasn't the best method to do ..
So, should I 'learn' DataReader or keep using DataSet ? DataTable ?
What are the pros/cons ?
That is essentially: "which is better: a bucket or a hose?"
A DataSet is the bucket here; it allows you to carry around a disconnected set of data and work with it - but you will incur the cost of carrying the bucket (so best to keep it to a size you are comfortable with).
A data-reader is the hose: it provides one-way/once-only access to data as it flies past you; you don't have to carry all of the available water at once, but it needs to be connected to the tap/database.
And in the same way that you can fill a bucket with a hose, you can fill the DataSet with the data-reader.
The point I'm trying to make is that they do different things...
I don't personally use DataSet very often - but some people love them. I do, however, make use of data-readers for BLOB access etc.
It depends on your needs. One of the most important differences is that a DataReader will retain an open connection to your database until you're done with it while a DataSet will be an in-memory object. If you bind a control to a DataReader then it's still open. In addition, a DataReader is a forward only approach to reading data that can't be manipulated. With a DataSet you can move back and forth and manipulate the data as you see fit.
Some additional features: DataSets can be serialized and represented in XML and, therefore, easily passed around to other tiers. DataReaders can't be serialized.
On the other hand if you have a large amount of rows to read from the database that you hand off to some process for a business rule a DataReader may make more sense rather than loading a DataSet with all the rows, taking up memory and possibly affecting scalability.
Here's a link that's a little dated but still useful: Contrasting the ADO.NET DataReader and DataSet.
Further to Marc's point: you can use a DataSet with no database at all.
You can fill it from an XML file, or just from a program. Fill it with rows from one database, then turn around and write it out to a different database.
A DataSet is a totally in-memory representation of a relational schema. Whether or not you ever use it with an actual relational database is up to you.
Different needs, different solutions.
As you said, dataset is most similar to VB6 Recordset. That is, pull down the data you need, pass it around, do with it what you will. Oh, and then eventually get rid of it when you're done.
Datareader is more limited, but it gives MUCH better performance when all you need is to read through the data once. For instance, if you're filling a grid yourself - i.e. pull the data, run through it, for each row populate the grid, then throw out the data - datareader is much better than dataset. On the other hand, dont even try using datareader if you have any intention of updating the data...
So, yes, learn it - but only use it when appropriate. Dataset gives you much more flexibility.
DataReader vs Dataset
1) - DataReader is designed in the connection-oriented architecture
- DataSet is designed in the disconnected architecture
2) - DataReader gives forward-only access to the data
- DataSet gives scrollable navigation to the data
3) - DataReader is read-only we can’t make changes to the data present under it
- DataSet is updatable we can make changes to the data present under it and send those changes back to the data source
4) - DataReader does not provide options like searching and sorting of data
- DataSet provides options like searching and sorting of data
To answer your second question - Yes, you should learn about DataReaders. If anything, so you understand how to use them.
I think you're better of in this situation using DataSets - since you're doing data binding and all (I'm thinking CPU cycles vs Human effort).
As to which one will give a better performance. It very much depends on your situation. For example, if you're editing the data you're binding and batching up the changes then you will be better off with DataSets
DataReader is used to retrieve read-only and forward-only data from a database. It read only one row at a time and read only forward, cannot read backward/random. DataReader cannot update/manipulate data back to database. It retrieve data from single table. As it is connected architecture, data is available as long as the connection exists.
DataSet is in-memory tables. It is disconnected architecture, automatically opens the connection and retrieve the data into memory, closes the connection when done. It fetches all the data at a time from the datasource to its memory. DataSet helps to fetch data from multiple tables and it can fetch back/forth/randomly.DataSet can update/insert/manipulate data.