I'm building a C# application where the end users won't have any sql database manager. I need a way of storing around 3 million objects (with roughly 10-20 string fields) so that the program can import the info. I've tried binary formatting but im getting out of memory exceptions due to the large amount of data. Is there any other way to store the objects in a speedy and size efficient way?
Thank You
This is what databases are made for. It doesn't matter that the end user won't have a database system already. There are a number of light-weight options available that you can deploy with your app. Here are three:
If you are Windows-only, then the Access database engine is built into the operating system. Just build a database with only table definitions (no records), and your app will be able to use it.
There is an implementation of sqlite completely in managed code, allowing you to compile it directly into your app.
Sql Server Compact Edition is just two *.dll files that are easy to deploy with the app.
None of these options require you to have anything running in the background on the target system.
You could also consider HDF5. This is designed for large-volume, high IO, distributed and high-performance computations. It is very flexible and imposes no install restriction on your application (can be completely self contained).
Hope this helps!
Related
I built a software for a farm using C#, the program was meant to keep track of the inventory and the financial exchanges related to the farms work.
The software was built to be installed on the manager's computer who then entered the farms data and retrieved reports and so. While the accountant used the same PC to use the financial part of the program with a different account.
Now the farm's business grew and more users need to use the system, can I move the database with the old data to a server so users can log in the system from different PC's at the time and continue the old tasks?
If I can - what do I change in my code?
P.S. the database was done in MS Access.
Not a lot of information to go on here. I can tell you that Access is a file based database system, and so whilst you could put the database files on a server or a NAS device with no problem with multiple users you should expect to run into the usual problems of Windows file sharing - VERY SLOW performance as a minimum.
It is also possible that the database may have been limited to a single user at a time, and without any more information impossible to know whether the developers have allowed for multi-user at all or whether you could have a situation where if several people do open the file at once one person may be overwriting another's data leading to corruption.
The short answer is that if the original developers are no longer around and you cannot ask the question of them then you probably need a new dedicated application to do the work which would mean either a complete rewrite or an alternative commercial application.
For multi-user Microsoft SQL Server, MySql, or even Firebird or another dedicated database back end would be the way to go. The front end could be anything - Winforms, WPF, even a web application if that is what you want, but it would have to be written.
I hope that this is helpful.
I'm looking into developing Universal Windows Applications with C# and XAML. If I were to design an application to log user's meals (nutrition log app) I'd need a database of foods (nutrition data - protein, fat etc.), user's data (age, weight, height, goals, calorie needs etc.) and meals (time, foods consumed, quantity of each etc.).
Some of these would be local, some not.
QUESTION: What's the best approach to take here, what kind of databases should I be looking into, what form to store data in, what's the best way to store a lot of data and have it readily available, what do others use with Universal Windows Applications?
Side note: I've written a simple C# Console Application that takes my data, calculates my needs, allows me to input food's nutrition data and then use that data to log meals. This data is written in separate XML files (user.xml, meal.xml and food.xml). I can do all kinds of queries using LINQ - what I've eaten in the last X days, how much do I still need to eat today, how much protein does food X contain... But XML can't be the best way to store and handle lots of data.
Help appreciated!
You can use SQLite. It is a full-blown SQL relational database which stores its data in a single file. It has ports for PCL and Universal apps, and has a .NET connector which allows it to work with ADO and LINQ. See download page.
We have been using SQLIte and its working fine for us.
You can use either SQLIte PCL if you wish to work with raw SQL queries or SQLite NET which gives you object instances to play with.
Only catch is that SQLite is a native component which forces you to build against three CPU archs x86, x64 and ARM. So you need to have three different build output. Some details here http://codifyit.blogspot.in/
For your data which you does not get updated too often, you can use some kind of database caching controlled from your code.
cheers,
Saurav
We are using Azure Mobile Apps and leveraging;
using Microsoft.WindowsAzure.MobileServices.Sync;
using Microsoft.WindowsAzure.MobileServices.SQLiteStore;
With these you define a local data store;
MobileServiceSQLiteStore store = new MobileServiceSQLiteStore(ApplicationData.Current.RoamingFolder.Path + "\\MyLocalSQLite.db");
Debug.WriteLine("Database Location: " + ApplicationData.Current.RoamingFolder.Path + "\\MyLocalSQLite.db");
store.DefineTable<Account>();
await App.MobileService.SyncContext.InitializeAsync(store);
This allows you to store data in the cloud as well as locally using
private IMobileServiceSyncTable<Account> accountsTable = App.MobileService.GetSyncTable<Account>();
You will need to read up on using them a bit, but its fairly strait forward.
https://blogs.msdn.microsoft.com/azuremobile/2014/04/07/deep-dive-on-the-offline-support-in-the-managed-client-sdk/ got me started.
These tables can convert to collections;
Accounts = await accountsTable.ToCollectionAsync();
which you can then bind to with your UI.
All in all its a pretty slick system once you make it up the learning curve, some of it is still pretty new though, and you will run into a few gotchas. Happy Coding!
So far, in my applications, I stored all my data via serialization, so I never really needed anything else. But now I am working with more complicated data and I need more than a simple file to store them. Is there some inbuilt storage engine in Windows I can use, which will allow me to pull and edit data with SQL queries? (Since I doubt the user will be willing to install and configure standalone MySQL server just for my application and I don't really want to use 3rd party solutions)
And if there is, how can I access such database engine?
Yes, Windows does have an embedded database engine, which has been there since Windows 2000. It is called Esent.
There is a project called Managed Esent to expose the features of Esent to managed code. There is a NuGet package for it.
I do not know whether it fits your exact need of updates via SQL queries but it does boast a number of features:
ACID transactions with savepoints, lazy commits, and robust crash recovery.
Snapshot isolation.
Highly concurrent database access.
Flexible meta-data (tens of thousands of columns, tables, and indexes are possible).
Indexing support for integer, floating point, ASCII, Unicode, and binary columns.
Update
I have since had some experience with Esent in C# via the nuget package. It does NOT have SQL query capabilities. There is support for indices and basic cursors over them but nothing like the power of SQL. My suggestion if you need a light-weight SQL is definitely SQLite.
SQLite is a software library that implements a self-contained, serverless, zero-configuration, transactional SQL database engine.
http://www.sqlite.org
SQLite will work without requiring your App User to install SQLExpress or any other database engine, making it a good option for desktop applications.
Use SQLite as standalone database and integrated part of your applications.
Main benefits:
data is stored inside a single file, easy to pull/edit
ships as part of your application as dll, no separate
installation/configuration needed
i found some info about the database on the Microsoft website database info
there is a embedded database engine. Esent
If it is small set of user specific data, storing at windows registry is a good option.
No, there is no inbuilt database available.
But there are good alternatives, like the mqsql express server or a SQLITE database, which is basically a single database file which is accessed through a library in your project. So if you use SQLITE, you do not need to instal anything.
Is it possible to have an embedded database as a file shared on a network disk and used simultaneously by multiple users (to both read and write)?
Slowness is not an issue, but no database software should have to be installed. Is Microsoft SQL Server Compact appropriate for such a purpose?
I found this page on the msdn which suggests that version 3.5 and above of MS SQL Server Compact should be o.k. in this scenario.
To support multiple applications that
access the same database at the same
time, SQL Server Compact 3.5 provides
multiuser support. Multiuser support
enables multiple users of a database
to synchronize data without having to
disconnect the database before they
use merge replication or remote data
access (RDA). For more information
about multiuser synchronization, see
Multiuser Access and RDA and Multiuser
Access and Synchronization.
However I, personally, have no experience with this so could not say for sure if this will definitely work.
Have you tried VistaDB?
I believe it should meet your requirements.
SQLIte for .NET should work for you!
SQLite is a small C library that
implements a self-contained,
embeddable, zero-configuration SQL
database engine. Features include:
Transactions are atomic, consistent, isolated, and durable (ACID) even
after system crashes and power
failures. Zero-configuration - no
setup or administration needed.
Implements most of SQL92. (Features not supported)
A complete database is stored in a single disk file.
Database files can be freely shared between machines with different byte
orders.
Supports databases up to 2 terabytes (241 bytes) in size.
Sizes of strings and BLOBs limited only by available memory.
Small code footprint: less than 30K lines of C code, less than 250KB code
space (gcc on i486)
Faster than popular client/server database engines for most common
operations.
Simple, easy to use API.
TCL bindings included. Bindings for many other languages available
separately.
Well-commented source code with over 95% test coverage.
Self-contained: no external dependencies.
Sources are in the public domain. Use for any purpose.
I want to create a folder with around 3000 files on a user PC for a WPF C# application. I am worried whether after I implement, if the lookup of a filename will be too slow. Can someone please suggest whether it should work, or if there is a better alternative?
Basically, my main goal is to use Clickonce to deploy the application on any user PC, and start it to run without imposing requirements of downloading database application.
Background:
I have created a C# application to find correlated pairs from yahoo finance. It currently downloads quotes for 3000 symbols and stores it in a Mysql database on the user PC. I am interested in creating a new version of the application which does not depend on Mysql database. I am thinking of saving all the quotes in a separate files in a folder on the user PC.Not sure if this is a good idea, or if some other alternatives like creating small object database on the user PC, using some helper class library which manages cache of large arrays, etc. would be ideal.
PS : It is an open source education project, so I cannot afford to buy commercial product.
I'd recommend using (as you've said that you're willing to look at alternatives to the file system) Sql Server Compact 4.0. It's:
Lightweight
Doesn't require a separate installer to be run
Doesn't need a separate service to be running
Light on memory footprint
Very easy to code against as you can use DataTable, DataSet and a set of SqlServerCe based SqlConnection / SqlDataAdapter classes that will be very familiar.
Free!
A good candidate for upgrading to Sql Server Express in the future if you need to.
Has tooling available (in Webmatrix, and I think Visual Studio 2010 SP1) that allows you to manipulate the database structure without having to resort to code.
You'll probably be able to achieve a swap-over in a very short space of time as "most" of the work can be achieved by adding a reference to SqlServerCe to your project and performing a search & replace on "MySql" for "SqlServerCe". That's a slight oversimplification, but not by much! =)
How about using SQLite? You wouldn't need to install anything else than your software. Here is a wrapper for .NET.
To answer your question regarding looking up the filename on the drive, that depends on what kind of drive it is. If it is an SSD it will probably not take long at all, if it is a mechanical drive and you are using indexing it will also be fast. However if you don't use indexing and have a slower mechanical drive, it will be slow.
You could also consider a RAM Disk / RAM Drive if you really want performance and want to have it stored on a filesystem. However, you would need to store it on a persistant storage as well.
You could use a simple file-based database such as SQLite.
http://www.sqlite.org/
There is a .NET driver for it as well.
http://code.google.com/p/csharp-sqlite/
I wouldn't advise falling back to the file system in lieu of a database in this case. Have you considered SQL Server Compact Edition or SQLite? Both run in-process, meaning that you don't have to install a database server - it's part of your application. And, of course, they still support SQL.
3000 files will be fine, 300,000 files and you'd have a problem :)
The only way to know for sure is to measure!
Create folders with 10 zero length files , 100 files 1000 files . . . 10,000,000.
Then measure the time it takes to find a file in each of the folders, you'll get a better idea which are "too slow".
P.s. "Too slow" is completely subjective. is 10ms too slow, or is 10 miunutes too slow?
You need to decide what "Too slow" is and work from there.
P.P.S Think carefully before deciding to use the file system in this way, there are better alternatives (as shown in other anawers).
I would say consider using a XML file to store your 3000 pairs of data, rahter than 3000 separate files, and there are plenty of LINQ to XML you can use to access data faster.