I'm in the early stages of building a winform C# app based on Access db (can't use other types of DB for different reasons).
My main issue is how to design the DB since the amount of data is vast (based on daily data) and it will fill up the its size limit within a month or so.
I thought of creating a new DB for every month, but how will I be able to compare data between the different DB, for example, between months? I want the C# app to execute the queries.
Are there any tutorials, books? I have no experience of using and linking front and back-end Access.
Any ideas?
Thanks!
You probably don't want to hear this, but starting a project with MS Access as the backend is not a good idea when you already know in advance that you will hit Access' size limit after only a month.
You say in a comment:
I'm stuck with Access because of these: 1. High cost of SQL server. 2. I'm not familiar with SQL server. 3. SQL server express (the free edition) also has a size limit, though larger (10gb). Are there other DB free and without size limitation? Are there other DB free
I agree with you that SQL Server is not a good solution in your situation - high price and size limitation are valid arguments against SQL Server (both full version and Express Edition).
So, in my opinion using a different database engine is the only real solution here.
Your third argument against SQL Server was "I'm not familiar with it", but I strongly advise you to become familiar with another database engine than Access, because using Access in your situation (size limit!!!) will be a pain in the long run.
(Note to all nitpickers: No, I'm not bashing Access in general - I'm making a living with it myself.
However, it has its limits and when you know in advance that you'll hit its size limit within a month, it's not a good idea to use it here.)
Yes, you could do some hack and use a different Access database for each month, but you will really feel the pain as soon as your users will need to load data from several months at once, or as soon as your boss asks you for a "quick report about our sales in the last three years" :-)
But you can use a different database engine. Yes, you will have to invest time to become familiar with it, learn how to set it up and so on.
But believe me, it will pay off in the long run because you don't have to deal with the hassle of one database file per month.
There are lots of free and capable database engines available, the most known are:
PostgreSQL
MySQL
Firebird
To connect to the MS Access database(s) you can use the code shown here and then you can go about 'joining' the data in your C# front-end.
You might end up writing a subset of a DB engine in C# though and I thoroughly support the comments provided by Bernard and Bryan.
Your DB design should be isolated from your front-end technology decisions, and the reverse is also true. See Multitier Architecture.
Using a multi-tier architecture will help you separate presentation from business logic from data access, allowing you to design and implement each of these components in a modular and robust fashion.
Searches for N-Tier architecture or Multitier architecture will find a wealth of information and help on how to implement multi-tier solutions and why you should go to the trouble.
Here are a few links that might help.
http://msdn.microsoft.com/en-us/library/aa288452(v=vs.71).aspx
http://bytes.com/topic/net/answers/516465-c-insert-statement-using-oledb
Related
I'm part of a small team that currently uses an Access database for scheduling a larger team's availability. This has presented some issues with corruption of the Access database. Additionally, I want to implement additional functionality over time.
I've set out to create an application for the 4-5 of us to use that will solve the concurrent database issue, as well as give the team more functionality.
Since this is a shared network drive, I won't have access to SQL Server (from my guess). I thought maybe a web service would be the way to go, but I don't really want to front the bill for this. Additionally, when I eventually leave the team I don't want to maintain this.
Some ideas I've come up with is an application written in C# that acts as the front-end with SQLite embedded as the back-end. However, I've spent days trying to get the Entity Framework to work with SQLite and am at the point of giving up.
I'm trying to decide what else I can do to solve this issue. Is there another technology I can use?
As was said, it sounds like you try to reinvent the DMBS wheel.
If you have a Database that multiple clients can use at the same time, "sharing a access file on a network share" will simply not cut it. You need a proper DBMS. You have simply outgrown the scale Access was designed for. propably even the scale it was intended for.
You said cost might be an issue, but it is not really: There are dozens of DBMS out there, with a number being Freeware. MySQL is a shining example of a free DBMS. Conver that whole Access thing into a MySQL Database. Write a frontend for the MySQL Database. Done.
If you already have a computer providing the share across the network, that same computer can provide the MySQL Server. Setting up a DBMS with 1+ instances can be a bit more involved then just enabling a share, but not much more then programming a WebService.
I normally use SQL Server and C# for all projects I do, however I am looking upon a project that could potentially span to billions of rows of data and I don't feel comfortable doing this in SQL Server .
The data I will be storing is
datetime
ipAddress
linkId
possibly other string related data
I have only ever dealt with relational databases before and hence was looking for some guidance on what database technology would be best suited for this type of data storage. One that could scale and do so at a low cost (when compared to sharding SQL Server)
I would then need to pull this data out based on linkId.
Also would I be able to do ordering within the query to the DB or would that be best done in the application?
EDIT: It will be cloud based. Hence I was looking at SQL Azure, which I have used extensively, however it just starts causing issues as the row count goes up.
Since you are looking for general guidance, I feel it is ok to provide an answer that you have prematurely dismissed ;-). Microsoft SQL Server can definitely handle this situation (in the generic sense of having a table of those fields and billions of rows). I have personally worked on a Data Warehouse that had 4 nodes, each of which had the main fact table holding 1.2 - 1.5 Billion rows (and growing) and responded to queries quickly enough, despite some aspects of the data model and indexing that could have been done better. It is a web-based application with many users hitting it all day long (though some periods of the day much harder than others). Also, that fact table was much wider than the table you are describing, unless that "possibly other string related data" is rather large (but there are ways to properly model that as well). True, the free Express edition might not meet your needs, but Standard Edition likely would and it is not super expensive. Enterprise has a nice feature for doing online index rebuilds, but that alone might not warrant the huge jump in license fees.
Keep in mind that with little to no description of what you are actually trying to accomplish with this data, it is hard for me to say that MS SQL Server will definitely meet your needs. But, given that you seemed to have ruled it out entirely on the basis of the large number of rows you might possibly get, I can at least speak to that situation: with good data modeling, good index design, and regular index maintenance, MS SQL Server can definitely handle billions of rows. Now, whether or not it is the best choice for your project depends on what you are trying to do, what the client is comfortable with maintaining, etc.
Good luck :)
EDIT:
When I said (above) that the queries came back "quickly enough", I
meant anywhere from 1 to 90 seconds, depending on various factors.
Keep in mind that these were not simple queries, and in my opinion,
several improvements could be made to the data modeling and index
strategy.
I intentionally left out the Table Partitioning feature not only
because it is only in Enterprise Edition, but also because it is more
often misunderstood and hence misused than understood and used
properly. Table/Index partitioning in SQL Server is not a means of
"sharding".
I also did not mention Column Store indexes because they are only
available in Enterprise Edition. However, for projects large enough
to justify the cost, Column Store indexes are certainly worth
investigating. They were introduced in SQL Server 2012 and came with
the restriction that the table could not be updated once the Column
Store index was created. You can get around that, to a degree, using
Table Partitioning, but in SQL Server 2014 that restriction will be
removed.
Given that this needs to be cloud-based and that you use .Net / C#, if you really are only talking about a few tables (so far just the stated one and the implied "Link" table--source of LinkID) and hence might not need relationships or some of the other RDBMS features, then one option is to use Amazon's DynamoDB. DynamoDB is part of AWS (Amazon Web Services) and is a NoSQL database. Development and even the initial stage of rolling out a project are made a bit easier by their low-end, free tier. As of 2013-11-04, the main DynamoDB page states that:
AWS Free Tier includes 100MB of Storage, 5 Units of Write Capacity,
and 10 Units of Read Capacity with Amazon DynamoDB.
Here is some documentation: Overview, How to Query with .Net, and general .Net SDK.
BE AWARE: When looking into how much you think it might cost, be sure to include related AWS pieces, such as Network usage, etc.
I am starting on an analytics project that will be getting data from several different sources and comparing them to one another. Sources can be anything from an API such as google analytics API to a locally hosted database.
Should I build a single database to import this data into on a regular basis?
Can anyone suggest some best practices, patterns or articles? I really don't know where to start with this so any information would be great! Thanks!
I will be using SQL Server 2008 R2, C# 4.0.
That's a big question, Mike - plenty of people have entire careers doing nothing but Data Warehousing.
I would give a qualified "yes" to your first question - one of the main attractions of a DWH is that you can consolidate multiple data sources into a single source of information. (The qualification is that there may be circumstances where you don't want to do this - for example, for security or performance reasons.)
As ever, Wikipedia is a reasonable first stop for information on this subject. Since your question is already tagged with data-warehouse, StackOverflow is another possible source.
The canonical books on the subject are probably:
Building the Data Warehouse - WH Inmon
The Data Warehouse Toolkit - Ralph Kimball, Margy Ross
The Data Warehouse Lifecycle Toolkit - Ralph Kimball, Margy Ross, Warren Thornthwaite, Joy Mundy, Bob Becker
Note that the Inmon and Kimball approaches are radically different - Inmon concentrates on a top-down, normalised relational approach to constructing an enterprise DWH, while Kimball's approach is more bottom-up, dimensional, functional datamart-based.
The DWH Toolkit concentrates on the technical aspects of building a DWH, while The DWH Lifecycle Toolkit is based as much on the organisational challenges as on the technical details.
Good luck!
I would start with SSIS which is a data integration technology that comes with SQL Server. It may handle a lot of the data sources you need. If you are using APIs such as Googles to get data you may need to put that in a staging table first.
Start with a single staging database which you will use as your primary source to load data into Analysis Services and see how that works out. Use SSIS to populate that staging database.
You need to take up the following steps:
1. First you need to pick up the ETL platform like SSIS, Informatica, or other ETL tools, etc.
2. Then, you need to pick up the appropriate database like Oracle or SQL server, etc.
3. Thereafter, you need to make the logical data warehouse modeling (Star or Snowflake) and
4. Finally, you need to develop the whole data ware house.
I would advise making two databases, i.e.
1. ODS for storing the data from different sources and for cleansing and
2. Warehouse database for storing all the relevant data.
I am working on a Sometimes Connected CRUD application that will be primarily used by teams(2-4) of Social Workers and Nurses to track patient information in the form of a plan. The application is a revisualization of a ASP.Net app that was created before my time. There are approx 200 tables across 4 databases. The Web App version relied heavily on SP's but since this version is a winform app that will be pointing to a local db I see no reason to continue with SP's. Also of note, I had planned to use Merge Replication to handle the Sync'ing portion and there seems to be some issues with those two together.
I am trying to understand what approach to use for the DAL. I originally had planned to use LINQ to SQL but I have read tidbits that state it doesn't work in a Sometimes Connected setting. I have therefore been trying to read and experiment with numerous solutions; SubSonic, NHibernate, Entity Framework. This is a relatively simple application and due to a "looming" verion 3 redesign this effort can be borderline "throwaway." The emphasis here is on getting a desktop version up and running ASAP.
What i am asking here is for anyone with any experience using any of these technology's(or one I didn't list) to lend me your hard earned wisdom. What is my best approach, in your opinion, for me to pursue. Any other insights on creating this kind of App? I am really struggling with the DAL portion of this program.
Thank you!
If the stored procedures do what you want them to, I would have to say I'm dubious that you will get benefits by throwing them away and reimplementing them. Moreover, it shouldn't matter if you use stored procedures or LINQ to SQL style data access when it comes time to replicate your data back to the master database, so worrying about which DAL you use seems to be a red herring.
The tricky part about sometimes connected applications is coming up with a good conflict resolution system. My suggestions:
Always use RowGuids as your primary keys to tables. Merge replication works best if you always have new records uniquely keyed.
Realize that merge replication can only do so much: it is great for bringing new data in disparate systems together. It can even figure out one sided updates. It can't magically determine that your new record and my new record are actually the same nor can it really deal with changes on both sides without human intervention or priority rules.
Because of this, you will need "matching" rules to resolve records that are claiming to be new, but actually aren't. Note that this is a fuzzy step: rarely can you rely on a unique key to actually be entered exactly the same on both sides and without error. This means giving weighted matches where many of your indicators are the same or similar.
The user interface for resolving conflicts and matching up "new" records with the original needs to be easy to operate. I use something that looks similar to the classic three way merge that many source control systems use: Record A, Record B, Merged Record. They can default the Merged Record to A or B by clicking a header button, and can select each field by clicking against them as well. Finally, Merged Records fields are open for edit, because sometimes you need to take parts of the address (say) from A and B.
None of this should affect your data access layer in the slightest: this is all either lower level (merge replication, provided by the database itself) or higher level (conflict resolution, provided by your business rules for resolution) than your DAL.
If you can install a db system locally, go for something you feel familiar with. The greatest problem I think will be the syncing and merging part. You must think of several possibilities: Changed something that someone else deleted on the server. Who does decide?
Never used the Sync framework myself, just read an article. But this may give you a solid foundation to built on. But each way you go with data access, the solution to the businesslogic will probably have a much wider impact...
There is a sample app called issueVision Microsoft put out back in 2004.
http://windowsclient.net/downloads/folders/starterkits/entry1268.aspx
Found link on old thread in joelonsoftware.com. http://discuss.joelonsoftware.com/default.asp?joel.3.25830.10
Other ideas...
What about mobile broadband? A couple 3G cellular cards will work tomorrow and your app will need no changes sans large pages/graphics.
Excel spreadsheet used in the field. DTS or SSIS to import data into application. While a "better" solution is created.
Good luck!
If by SP's you mean stored procedures... I'm not sure I understand your reasoning from trying to move away from them. Considering that they're fast, proven, and already written for you (ie. tested).
Surely, if you're making an app that will mimic the original, there are definite merits to keeping as much of the original (working) codebase as possible - the least of which is speed.
I'd try installing a local copy of the db, and then pushing all affected records since the last connected period to the master db when it does get connected.
Okay, I've seen but haven't programmed in C# before. You can assume I'm competent in C++, and advanced in C (for what good that'll do me). I understand inheritance, polymorphism, etc so OO concepts aren't going to be a huge problem.
Let's say I've been given a task to prototype a quick and dirty program that won't be much different than what I could do in access in a short time.
It'll have a DB with 5-6 tables (mostly small, a few with have several thousand rows but only 4 or so columns, etc)
I'll need to have forms generated dynamically from one of the DBs
The results of the forms will be stored in another table
The DB isn't multiuser
Basically your run of the mill access app... except without access. I'm sure I can muddle my way through and create horrendously bad code, but I'm equally sure lots of people here can give me a push in the right direction (tutorials, wizards, info, differences and killers moving from C/C++ to C#, etc).
Is there a simple DB I can plug in to get started aside from mdb, or is that the best choice for this particular nail? I'm aiming for a quick and dependency-less install.
Thanks!
-Adam
It sounds like for this app, you could use Microsoft Dynamic Data or Castle Active Record, and have the application working a few minutes after you finished the database. These tools connect to a database and generate forms for inputing data. Take a look at them.
Access is probably your best choice for database. MS Sql 2005/2008 Express would also work well, but that would require an install.
If you're programming in C#, Visual Studio comes with an added install for SQL Server Express. If you're looking to get something up quick and dirty, it would pretty easy to leverage that database in building your app.
If concurrency is not an issue, then I'd say go with SQLite. An ADO.NET provider can be found here: http://sourceforge.net/projects/sqlite-dotnet2/files. As for the concurrency issue, when performing a bunch of inserts, that operation needs to be enclosed within a transaction. However, that transaction places an exclusive lock on the database until the transaction is either commited or rolled back. At least that's been my experience, but I've only been using it for a week or two myself.
Hope that helps!
If you already have MS Access installed, then yes the mdb is probably your quickest way to get started.
Also you will want to just started with a quick ADO.NET tutorial. There are hundreds of these (well almost hundreds, I haven't counted).
How about using SQLlite instead of access for db? I have never used it but have heard that its good for some light weight and quick db tasks.
SqlExpress would be your best bet, simply because you already have all the support that you need in System.Data.SqlClient. Other then that, there are some decent help on the MSDN.
SQLite is exactly what you're after. It's a C library for accessing a file based relational database. The referred site has full visual studio support (System.Data.SQLite).
I would suggest using SubSonic to generate your data access code and scaffold your screens.