When I load from the database I use one store procedure which loads the DataItem and any Data associated with it. This comes back in one DataSet with two tables, the first table has one row and describes the DataItem and each row in the other table describing the related Data.
This DataSet is then used to populate my objects.
My problem comes when I have to save the objects back to the database. I am currently saving the DataItem and then looping through all of my Data and performing a save on each one. Completely horrible way to go about doing it, I know. It's both slow and it's not transactional.
So what I'd ideally like to do is convert my objects back into my DataSet and then save it all back to the database in one efficient transactional operation. What code do I need on the C# side to make this transactional and to allow me to pass back a DataSet. I presume this will involve using a TableAdapter. But given that I have two tables how will this work? What do I use on the SQL side - Can I use store procedures? (I would like to avoid having SQL in my C# project) Would I need to write something that will handle cycling through a datatable to save each record?
What's the best way to go about doing all this? This will form the lynchpin of a project I'm working on so I want it to be as fast and efficient as it can be!
(.NET 4.0 and SQL 2005)
Did not use TableAdapter in the end as it was more effort than it was worth.
From the comments:
http://msdn.microsoft.com/en-us/library/4esb49b4.aspx
Related
I'm currently working with an import file that has 460,000 rows of data within it. Each row consists of a ID and a quantity (eg. "1,120"). This information is read from the file, then should be used to update each individual row within a database (eg. UPDATE item SET quantity = QTY WHERE id = 1).
The problem I'm having, though, is actually being able to actually run the query efficiently. If I run an individual query for each line, it's really not going to work (As I've found out the hard way).
I'm not in any way a SQL user and I'm currently learning, but from what I've seen, the web doesn't seem to have any useful results on this.
I was wondering if anybody had experience with updating such a large dataset, and if so, would they be willing to share the methods that they used to achieve this?
460k rows isn't a lot, so you should be okay there.
I'd recommend importing the entire dataset into a temporary table, or table variable. To get the solution working, start by creating an actual physical table, which you can either DROP or TRUNCATE while you are getting this working.
Create the table, then import all the data into it. Then, do your table update based on a join to this import table.
Discard the import table when appropriate. Once this is all working how you want it to, you can do the entire thing using a stored procedure, and use a temporary table to handle the imported data while you are working with it.
460000 rows is a small dataset. Really small.
Bulk insert into tempoary table, then use an update command to run the update on the original data in one run.
I'm trying to learn to use SQLite, but I'm very frustrated and confused. I've gotten as far as finding System.Data.SQLite, which is apparently the thing to use for SQLite in C#.
The website has no documentation whatsoever. The "original website", which is apparently obsolete from 2010 onwards, has no documentation either. I could find a few blog tutorials, but from what I can tell their method of operation is basically:
Initialize a database connection.
Feed SQL statements into the connection.
Take out stuff that comes out of the connection.
Close connection.
I don't want to write SQL statements in my C# code, they're ugly and I get no assistance from the IDE because I have to put the SQL code in strings.
Can't I just:
Create a DataSet.
Tell the DataSet that it should correspond to the SQLite database MyDB.sqlite.
Manipulate the DataSet using its member functions.
Not worry about SQLite because the DataSet automatically keeps itself in sync with the SQLite database on disc.
I know that I can fill a DataSet with the contents of a database, but if I want access to the entire database I will have to fill the DataSet with all of its contents. If my database is 1 GB, I have just used up 1 GB of RAM (not to mention the time needed to write all of it at once).
Can't I simply take a SQLite database connection and pretend it's just an ordinary DataSet (that perhaps needs to be asked occasionally if it's done syncing yet)?
The answer to the question is no.
No you cannot simply take a SQLite connection pretend it's just a DataSet.
If you don't want to code SQL statements then consider Entity Framework.
Using SQLite Embedded Database with Entity Framework and Linq-to-SQL
You shouldn't treat a DataSet as a database. It's just a result of a query.
You query the database to get a subset of data (you never want ALL the data from your DB) and this subset is used to populate your DataSet.
You are required to synchronize your changes manually because DataSet doesn't know which updates should be a part of which transaction. This is your system knowledge.
The DataSet is an in memory cache and will only synchronize to the underlying data store when the developer allows it. You could put a timer wrapper around in and do it on a schedule but you still need to keep the Dataset and data store synchronized manually.
Storing 1GB+ of data is really not recommended as the memory usage would be very high and the performance very low. You also don't want to be sending that amount of data over a network or god forbid an internet connection.
Why would you want to keep 1GB of data in memory?
This might be a dumb question. But does ADO.net support updating to a database without having to write SQL commands?
Example:
I have a method that reads from database and keeps the rows in memory. I then modify some of the rows. Is it then possible to make ADO.net update the newest changes to the database without having to write an update SQL statements but instead let ADO.net figure it out?
I am asking this because I might want to update at a much later point. I could just store the SQL statements in a list but then I would be doing many updates instead of just one big one which would take longer time.
What you need is some sort of ORM, and ADO is not an ORM. So, no. You must write the SQL. You could maybe simplify things by writing a stored procedure, though. Then you can use ADO parameters
If you want, you can save your changes as objects in memory until you need to actually persist them. Then you can have a mapper that will take the object and write the SQL for you. However, then you are redoing some of the work of what is already done in an ORM
Like the sql you used to get the data, you need sql to put the data. It also needs to update what column to update. I don't think it can be automatic. Or use the Entity Framework. Probably saving the objects to be updates (IDs) is the way to go or update instantly.
ADO.NET supports DataAdapters and DataSets which allow you to do the following:
Manipulate data within your DataSet.
Push changes to the database by passing your DataSet as a parameter to the Update method of the DataAdapter.
In order to get the DataAdapter to push the changes it is necessary to specify insert, update, and delete commands. You will have to specify some sql in your command configuration but it is like a template of the sql statement that will update each row that you operate upon rather than your having to manually track changes.
Once you have configured your commands, use the UPDATE method with the DataSet as the parameter and it will persist your changes based on your commands. You will not need to track the individual sql changes.
A sample of configuring commands can be found here.
A sample of calling the update can be found here.
I want to refactor a logic to filter the grid of my application. I'd like to implement this logic only in the Database.
So, I have a grid which displays the data from more tables from DB. There are some textBoxes and comboBoxes where it sets the data for each filter it wants.
Now, it is a HUGE Stored Procedure in Database which works this way:
Initially selects [ALL DATA] from tables into a temporary table ,
after that,
according to fields that were filled with data (from application filter), it removes from [ALL DATA] that info which is NOT LIKE SELECTED FILTER
And so on foreach parameter which is set in the filter.
This way consumes much time, because initially selects all data, and then slowly removes those which don't need.
I don't want to create SQL queries on client side. I'd like to do that only to Database, or .... iimmmm, i don't know...
Which would be a best way, very optimized, which would run fast and return results in short time as possible?
I use C# and .NET 4.0 for client side, and MSSQL DB.
Thank you for advices.
Amend your stored procedure so that it only selects the required data into the temprorary table in the first place, rather than selecting everything and then deleting what isn't required.
That's what ORMs are for. Don't build a dynamic SQL statement on the client side. Rather, build a LINQ query filter by filter. You'll get the benefits of being entirely dynamic, without the risk of SQL injection.
You can look at this for an example.
I am trying to figure out the best way to design my C# application which utilizes data from a SQL Server backend.
My application periodically has to update 55K rows each one at a time from a loop in my application. Before it does an update it needs to check if the record to be updated exists.
If it exists it updates one field.
If not it performs an insert of 4 fields.
The table to be updated has 600K rows.
What is the most efficient way to handle these updates/inserts from my application?
Should I create a data dictionary in c# and load the 600K records and query the dictionary first instead of the database?
Is this a faster approach?
Should I use a stored procedure?
What’s the best way to achieve maximum performance based on this scenario?
You could use SqlBulkCopy to upload to a temp table then have a SQL Server job do the merge.
You should try to avoid "update 55K rows each one at a time from a loop". That will be very slow.
Instead, try to find a way to batch the updates (n at a time). Look into SQL Server table value parameters as a way to send a set of data to a stored procedure.
Here's an article on updating multiple rows with TVPs: http://www.sqlmag.com/article/sql-server-2008/using-table-valued-parameters-to-update-multiple-rows
What if you did something like this, instead?
By some means, get those 55,000 rows of data into the database; if they're not already there. (If you're right now getting those rows from some query, arrange instead for the query-results to be stored in a temporary table on that database. (This might be a proper application for a stored procedure.)
Now, you could express the operations that you need to perform, perhaps, as two separate SQL queries: one to do the updates, and one or more others to do the inserts. The first query might use a clause such as "WHERE FOO IN (SELECT BAR FROM #TEMP_TABLE ...)" to identify the rows to be updated. The others might use "WHERE FOO NOT IN (...)"
This is, to be precise, exactly the sort of thing that I would expect to need to use a stored procedure to do, because, if you think about it, "the SQL server itself" is precisely the right party to be doing the work, because he's the only guy around who already has the data on-hand that you intend to manipulate. He alone doesn't have to "transmit" those 55,000 rows anywhere. Perfect.