I have searched back and forth but seemingly could not get a hold of what I need. I am sorry if this has been answered of late. A redirection to the discussion will do me good.
This is the scenario. I have been instructed to move from Microsoft Visual Foxpro (MS is withdrawing support come 2015) to .Net C# by my boss. For the sake of good foundation and adoption of best practices, I have decided to first learn, piece pertinent information together, then start coding. This is the second year.
We are a bureau company that offer payroll processing outsource services to over 50 clients. Each client currently has their own database. The databases have tables with completely identical structures.
I am a newbie. Totally new to .net world.
I had started off with raw SQL using datatables, datareaders but in my research I got some discussions discouraging this. Many were of the view that Entity Framework should serve the purpose. But one is allowed to mix approaches especially when complex queries are involved.
Can someone point me to some 'good read' where I can implement Entity Framework with over 50 indentical databases. Each database is totally independent and has nothing to dowith any other. When the user logs in, they select which client they need to process payroll for, then EF points to that database.
EF needs 2 different pieces of information to work with data from a database:
1) The database schema: This is included as compiled code in your application and cannot normally be changed at runtime.
2) The connection string: This is provided at runtime, normally from a config file.
In your case, all the databases have the same schema, so you can just model one database and it will work for all the others.
The piece you want to change is the connection string. This tells EF how to find the database and can be provided at runtime.
There is an overload of the DbContext constructor which takes a connection string as a parameter: MSDN: DbContext Constructor (String)
And there are even classes in the framework that help create connection strings for you:
MSDN: EntityConnectionStringBuilder Class
MSDN: Connection String Builders
It is very simple
I had,
//WMSEntities is conection string name in web.config
//also the name of Entitiframework
public WMSEntities() : base("name=WMSEntities")
{
}
already in autogenerated Model.Context.cs of edmx folder
To connect to multiple database in runtime, I created another constructor that takes connection string as parameter like below in same file Model.Context.cs
public WMSEntities(string connStringName)
: base("name=" + connStringName)
{
}
Now, I added other connection string in Web.Config for example
<add name="WMSEntities31" connectionString="data source=TESTDBSERVER_NAME;
initial catalog=TESTDB;userid=TestUser;password=TestUserPW/>
<add name="WMSEntities" connectionString="data source=TESTDBSERVER_NAME12;
initial catalog=TESTDB12;userid=TestUser12;password=TestUserPW12/>
Then, when connecting to database I call below method passing connetionString name as parameter
public static List<v_POVendor> GetPOVendorList(string connectionStringName)
{
using (WMSEntities db = new WMSEntities(connectionStringName))
{
vendorList = db.v_POVendor.ToList();
}
}
Hrmmm I happen to really like EF Code First but I'm not certain it suits what you're doing. How often does your schema change?
Should You Be Using EF?
Advantages of EF
If the schema changes somewhat regularly, the Migrations part of EF Code First might save you a lot of time and effort because you can often do away with SQL scripts for schema upgrades - schema changes end up in your source repository with the rest of your code instead. You'd start here:
https://stackoverflow.com/a/8909092/176877
I also happen to really like how easy EF is to setup, and how easy it is to write LINQ queries against it and return exactly the POCOs I built from the DB.
But EF might not be the best fit.
Other ORMs to consider
Many other ORMs support LINQ and POCOs with better support for existing databases (there are things that can be pretty difficult to map in EF Code First), --and existing support for asynchronous operation (EF is on 5.0 right now; 6.0 has async)-- (update: EF6 is the latest and its async support is great. Its bulk delete is terrible though and should be avoided like plague, drop to plain SQL for that).
In particular NHibernate is the beast on the scene for existing db support, but it's a bit of a configuration chore and what appears to be political infighting has caused the documentation to be conflicting for different versions and forks of it.
Much simpler are many "Micro ORMs" - that link is to a short list from 2011 but if you poke around you'll find 30 or so in .Net. Some generate better or less optimal queries, some none at all, some make you write SQL (don't use those) - you'll have to poke around to decide which is for you. This can be a bigger research task but I suspect the easy configuration and small learning curve for one of these best suits what you're trying to do.
Answer to your specific question
Talk to All client Dbs at once
If you're connecting to all 50 databases from one app at the same time you'll need to instantiate 50 DbContexts like:
var dbClient1 = new DbClient1();
var dbClient2 = new DbClient2();
Assuming you went around making little wrapper classes like:
public class DbClient1 : CoreDbContext
{
public DbClient1()
: base("DbClient1") // Means use the connection string named "DbClient1" in Web.Config
Where CoreDbContext is the main EF class in your Project that extends DbContext (standard part of any EF project).
Talk to just one at a time
If you're using just the one per app then any EF tutorial will do.
The only major trick will be migrating those Dbs when schema changes occur. Two basic approaches there. Either way you grab a backup and restore a copy of them locally so you can test your migrations against them (update-database -f -verbose). If you don't you risk data errors like changing a column to NOT NULL and finding your local test instance had no nulls, one client's did, kaboom. Once you get them working, you're onto deciding how you want to update Production. There are a lot of ways you might do this ranging from writing a custom roll-forward/back tool (or finding one) with SQL scripts checked into git, hiring a DBA, or much simpler:
The Obvious - SQL Script
Dump the migration to SQL (update-database -script) and run it against the actual production database.
My Crazy Way for Small Numbers of Dbs
Add entries for each db to Web.Config, and create a Project Configuration for each of them like "DbDeployClient1," "DbDeployClient2," etc. In each of those make a build define like DbDeployClient1, and then add this to your DbContext class:
public CoreDbContext()
#if DbDeployClient1
: base("DbDeployClient1")
#elseif DbDeployClient2
: base("DbDeployClient2")
// etc
#endif
{
That allows you to quickly switch to your DbDeploy config and run the migration directly from Visual Studio against the target database. Obviously if you're doing this you'll need to temporarily open a port, preferably only allowing in your IP, on the actual SQL Server instance you're migrating. One nicety is you get clear errors from your migration right there, and full rollback capability, without any real work - all that rollback support you're leveraging is just part of EF. And one dev can do it without a bunch of other bottlenecks. But it has a lot of opportunities to reduce risk and improve automation.
Related
we are at the beginning of a new project, which will replace a legacy project. The legacy one is written in .Net Framework 4.0 (SOA with WCF) + SQL Server. The connection with SQL is made by ADO.Net + stored procedures. There is a structural mistake by having most of the logic on the stored procedures, and on top of that, it is a monolytic.
The new project will be made with .Net 6 APIs and in some cases, it will have SQL Server as well, for operational data.
So, looking at the new product the question was raised: should we move from ADO.Net to EF? This is tempting since it reduces the development effort, but performance is a concern.
Taking a look at the technical must haves:
Get the product to be as fast as possible (performance is a concern)
The new project is expected to live at least for the next 15 years
Operations are executed against tables with 30 to 50 million records
We must be able to run operations against the regular database, but also against the readonly one (AlwaysOn)
We must be able to perform some resiliency policies such as retries in case of deadlocks
We don't have much room for changes if we choose one path and somewhere along the way we realize we should had gone with the other option
Quite honestly, IMHO, based on our tech requirements I feel should move forward with ADO.Net + Stored procedures (without any business logic) + some sort of package that translates the SQL results to my objects in a fast manner, but I'd like to give EF a shot, at least on this stage of the process where we are investigating possibilities.
I'd like to gather if possible opinions, specially if there is someone out there that went to EF with requirements as similar as ours, or someone who didn't go to EF or had to change from EF to ADO.Net somewhere along the way.
Thanks.
The only thing in your requirements that could support using ADO.NET over EF is
Get the product to be as fast as possible (performance is a concern)
Which is a nonsense requirement, as you can always write more code and make things more complex to make things marginally faster. You need a real performance requirement so you can measure different approaches.
EF is so widely used staff but I don't realize how I should use it. I met a lot of issues with EF on different projects with different approaches. So some questions brought together in my head. And answers leads me to use pure ado.net with stored procedures.
So the questions are:
How to deal with EF in n-tier application?
For example, we have some DAL with EF. I saw a lot of articles and projects that used repository, unit of work patterns as some kind of abstraction for EF. I think such approach kills most of benefits that increase development speed and leads to few things:
remapping of EF load results in some DTO that kills performance(call some select to get table data - first loop, second loop - map results to some composite type generated by ef, next - filter mapped data using linq and, at last, map it to some DTO). Exactly remapping to DTO is killer of one of the biggest efs benefit;
or
leads to strong cohesion between EF (and it's version) and app. It will be something like 2-tier app with dal and presentation with bll or dal with bll and presentation. I guess it's not best practice. And the same loading process as we have for previous thing except mapping, so again performance issue raised up. We could try to use EF as DAL without any abstraction under them. But we will get similar issues in some other way.
Should I use one context per app\thread\atomic operation? Using approach - one context per app\thread may slightly increase performance and possibilities to call navigation properties, but we meet another problem - updating this context and growing loaded data in context, also I'm not sure about concurrency with one dbcontext per app\thread. Using context per operation will lead us to remapping EF results to our DTO's. So you see that we again pushed back to question no.1.
Could we try to use EF + stored procedures only? Again we have issues from previous questions. What is the reason to use EF if the biggest part of functionality will not be used?
So, yes EF is great to start project. It so convenient when we have few screens and crud operations.
But what next?
All this text is just unsorted thoughts. I know that pure ado.net will lead to another kind of challenges.
So, what is your opinion about this topic?
By following the naming conventions , you will find it's called : ADO.NET Entity Framework , which means that Entity Framework sits on top of ADO.NET so it can't be faster , It may perform both in equal time , but let's look at EF provides :
You will no more get stuck with writing queries without any clue about if what you're writing is going to compile or not .
It makes you rely on C# or your favorite .NET language on writing your own data constraints that you wish to accept from the target user directly inside your model classes .
Finally : EF and LINQ give a lot of power in maintaining your applications later .
There are three different models with the Entity Framework : Model First , Database First and Code First get to know each of 'em .
-The Point about killing performance when remapping is on process , it's because that on the first run , EF loads metadata into memory and that takes time as it builds in-memory representation of model from edmx file.
ADO. Net is an object oriented framework that allows you to interact with database system (SQL, Oracle, etc).
Entity framework is a techniques of manipulating data in databases like (collection of queries (inert table name , select * from like this )).
it is uses with LINQ.
Entity Framework is not efficient in any case as in most tools or toolboxes designed to achieve 'faster' results.
Access to database should be viewed as a separate tier using store procedures as the interface. There is no reason for any application to have more than absolutely require CRUD operations. Less is more principle. Stored procedures are easy to write, secure, maintain and is de facto fastest way. It's easy to write tools to generate desired codes for POCO and DbContext through stored procedures.
Application well designed should have a limited numbers of connection strings to database and none of which should be the all mighty God. Using schema to support connection rights.
Lazy loading are false statements added to solve a problem that should never exist and introduced with ORM and its plug and play features. Data should only be read when needed. Developers should be responsible to implement this logic base on application context.
If your application logic has a problem to maintain states, no tool will help. It will in fact, make it worse by cover up the real problem until it's too late.
Database first is the only solution for a well designed application. Civilization realized long time ago the important of solid aqueduct and sewer system. High level code can and will be replaced anytime but data stays. Rewrite an entire application is matter of days if database is well designed.
Applications are just glorified database access. Still true in most cases.
This is my conclusion after many years in business applications debugging through codes produced by many different tools or toolboxes. The faster results advertised are not even close to cover the amount of time/energy wasted later trying to clean up the mess. Performance issues are rarely if not ever caused by high demand but the sum of all 'features' added through unusable tools.
ADO.NET provides consistent access to data sources such as SQL Server and XML, and to data sources exposed through OLE DB and ODBC. Data-sharing consumer applications can use ADO.NET to connect to these data sources and retrieve, handle, and update the data that they contain.
Entity Framework 6 (EF6) is a tried and tested object-relational mapper (O/RM) for .NET with many years of feature development and stabilization. An ORM like EF has the following advantage
ORM lets developers focus on the business logic of the application thereby facilitating huge reduction in code.
It eliminates the need for repetitive SQL code and provides many benefits to development speed.
Prevents writing manual SQL queries; & many more..
In an n-tier application,it depends on the amount of data your application is handling and your database is managing. According to my knowledge DTO's don't kill performance. They are data container for moving data between layers and are only used to pass data and does not contain any business logic. They are mostly used in service classes.See DTO.
One DBContext is always a best practice.
There is no such combination of EF + SP(Stored Procedure) as per my knowledge. If you wish to use an ORM like EF and an SP at the same time try micro-ORMs like Dapper,BLToolkit, etc..It was build for that purpose and is heck lotta fast than EF. Here is a good article on Dapper ORM.
Here is a related thread on a similar topic: What is the difference between an orm and ADO.net?
I have been looking for solution for a few days already and could find anything that would help to solve my problem.
I have a WCF service for which I have to make some unit tests. The problem is that service takes data from database in this way:
using (var context = new MyProjectEntities())
{
//here goes the actions
}
MyProjectEntities is autogenerated from edmx model i guess.. (Database first)
So this way it takes all the data from the database and operates on it.
My question is: whats the correct way to feed service with fake data for testing, instead of data from database?
The most trivial way is to use a live database. This isn't too flexible, because you need a new database in a fixed initial state for every single run, and also multiple developers can't use the same database at the same time.
What we do at my company is this: use a single-file database, namely SQL Server CE. If your code is database-engine independent, this can totally work, just change your connection string and you can even put your database to a fixed state by copying a template datafile to the right place. This isn't a really isolated unit test, but it's very simple to implement, it doesn't have the above problems, and you basically get what you need in the end. If your code is database-engine dependent, now you have one more reason to use an ORM solution like NHibernate or Entity Framework.
The best, most flexible, and also most complex solution is using a dependency injection or mocking framework. This is textbook stuff, there's tons of literature on that, it will give you all the flexibility there is.
In writing an application that runs on Fluent Nhibernate/Nhibernate, something has me a bit concerned. I suppose this would be true of any ORM (and even without using an ORM), but what is the ... I guess the word is 'field of study' that relates to the best practices and methods for updating a database after deployment?
In nHibernate, I establish a SessionFactory and have an initial run where it writes the database out based on the mappings. That's fine and good, I can even write the database out manually. But what about when my client comes back and wants something new added? Can I append to the database without losing my data? I am completely new to all of this and it has been troubling me since the start of this project, and I really do not know what direction to go to make sure I can manage the program after it is deployed.
I have looked at other stack overflow questions that I could find regarding this topic - one of which did not even have an accepted answer (though the question itself was kind of vague), but I did discover the tool http://www.red-gate.com/products/sql-development/sql-compare/ from the question
Tool to upgrade SQL Express database after deployment though I am wondering just how good of a 'strategy' that is.
There are a couple of options, use the AutoMapping feature in Fluent NHibernate to minimize the mapping code you write. If your schema changes comply with the AutoMap conventions then you only need to work with the corresponding domain object changes.
Another less optimal option is to take a database first approach and have something like MyGeneration automatically generate the domain classes and NHibernate mapping files from the schema. This works if you have complete control of the database schema and it can be made to implement a good domain model design (both conditions which very rarely ever happen...)
In either approach, these tools can help handle the database scripting needed to "migrate" the schema changes to a new version
from my experience, after deployment you have to manually keep your db structure up-to-date.
that means that whenever you add / change your db structure, you do so using a script with DDL commands.
when you're ready to deploy, you just run those DDL scripts against your production db.
for example, if you add a 'bar' column to your 'foo' table, your script would be something like (pseudo-code):
ALTER TABLE foo ADD COLUMN 'bar' int(32) not null default(0);
(EDIT: I made it a community wiki as it is more suited to a collaborative format.)
There are a plethora of ways to access SQL Server and other databases from .NET. All have their pros and cons and it will never be a simple question of which is "best" - the answer will always be "it depends".
However, I am looking for a comparison at a high level of the different approaches and frameworks in the context of different levels of systems. For example, I would imagine that for a quick-and-dirty Web 2.0 application the answer would be very different from an in-house Enterprise-level CRUD application.
I am aware that there are numerous questions on Stack Overflow dealing with subsets of this question, but I think it would be useful to try to build a summary comparison. I will endeavour to update the question with corrections and clarifications as we go.
So far, this is my understanding at a high level - but I am sure it is wrong...
I am primarily focusing on the Microsoft approaches to keep this focused.
ADO.NET Entity Framework
Database agnostic
Good because it allows swapping backends in and out
Bad because it can hit performance and database vendors are not too happy about it
Seems to be MS's preferred route for the future
Complicated to learn (though, see 267357)
It is accessed through LINQ to Entities so provides ORM, thus allowing abstraction in your code
LINQ to SQL
Uncertain future (see Is LINQ to SQL truly dead?)
Easy to learn (?)
Only works with MS SQL Server
See also Pros and cons of LINQ
"Standard" ADO.NET
No ORM
No abstraction so you are back to "roll your own" and play with dynamically generated SQL
Direct access, allows potentially better performance
This ties in to the age-old debate of whether to focus on objects or relational data, to which the answer of course is "it depends on where the bulk of the work is" and since that is an unanswerable question hopefully we don't have to go in to that too much. IMHO, if your application is primarily manipulating large amounts of data, it does not make sense to abstract it too much into objects in the front-end code, you are better off using stored procedures and dynamic SQL to do as much of the work as possible on the back-end. Whereas, if you primarily have user interaction which causes database interaction at the level of tens or hundreds of rows then ORM makes complete sense. So, I guess my argument for good old-fashioned ADO.NET would be in the case where you manipulate and modify large datasets, in which case you will benefit from the direct access to the backend.
Another case, of course, is where you have to access a legacy database that is already guarded by stored procedures.
ASP.NET Data Source Controls
Are these something altogether different or just a layer over standard ADO.NET?
- Would you really use these if you had a DAL or if you implemented LINQ or Entities?
NHibernate
Seems to be a very powerful and powerful ORM?
Open source
Some other relevant links;
NHibernate or LINQ to SQL
Entity Framework vs LINQ to SQL
I think LINQ to SQL is good for projects targeted for SQL Server.
ADO.NET Entity Framework is better if we are targeting different databases. Currently I think a lot of providers are available for ADO.NET Entity Framework, Provider for PostgreSQL, MySQL, esql, Oracle and many other (check http://blogs.msdn.com/adonet/default.aspx).
I don't want to use standard ADO.NET anymore because it's a waste of time. I always go for ORM.
Having worked on 20+ different C#/ASP.NET projects I always end up using NHibernate. I often start with a completely different stack - ADO.NET, ActiveRecord, hand rolled wierdness. There are numerous reasons why NHibernate can work in a wide range of situations, but the absolutely stand out for me is the saving in time, especially when linked to code generation. You can change the datamodel, and the entities get rebuilt, but most/all the other code doesn't need to be changed.
MS does have a nasty habit of pushing technologies in this area that parallel existing open source, and then dropping them when they don't take off. Does anyone remember ObjectSpaces?
Added for new technologies:
With Microsoft Sql Server out for Linux in Beta right now, I think it's ok to not be database agnostic. The .Net Core Path and MS-SQL route allows you to run on Linux servers like Ubuntu entirely with no windows dependencies.
As such, imo, a very good flow is to not use a full ORM framework or data controls and leverage the power of SSDT Visual Studio Projects (Sql Server Data Tools) and a Micro ORM.
In Visual Studio you can create a Sql Server Project as a legit Visual Studio Project. Doing so allows you to create the entire database via table designers or raw query editing right inside visual studio.
Secondly, you get SSDT's Schema Compare tool which you can use to compare your database project to a live database in Microsoft Sql Server and update it. You can sync your Visual Studio Project with the server causing updates in your project to go out to the server. Or you can sync the server with your project causing your source code to update. Via this route you can easily pick up changes the DBA made in maintenance last night and push out your new development changes for a new feature easily with a simple tool.
Using that same tool you can compute the migration script without actually running it, if you need to pass that off to an operations department and submit a change order, it works for that flow to.
Now for writing code against you MS-SQL Database, I recommend PetaPoco.
Because PetaPoco works Perfectly inline with the above SSDT solution. PetaPoco comes with T4 text templates you can use to generate all your data entity classes, and it generates the bulk data layer classes for you.
The catch is, you have to write queries yourself, which isn't a bad thing.
So you end up with something like this:
var people = dbContext.Fetch<Person>("SELECT * FROM People where Username Like '%#0%'", "bob");
PetaPoco automatically handles parameterizing #0 for you, it also has the handy Sql class for building queries.
Furthermore, PetaPoco is an order of magnitude faster than EF6 and 8+ times faster than EF7.
So in total, this solution involves using SSDT for SCHEMA management, and PetaPoco for code integration at the gain of high maintainability, customization, and very good performance.
The only downfall to this approach, is that you're hard tieing yourself to Microsoft Sql Server. However, imo, Microsoft Sql Server is one of the best RDBM's out there.
It's got DBMail, Jobs, CLR object capabilities, and on and on. Plus the integration between Visual Studio and MS-SQL server is phenomenal and you don't get any of that if you choose a different RDBMS.
I must say that I never used NHibernate for the immense time that needed to start using... time wasted on the XML setup.
I recently did a web application in MVC2, where I did choose ADO Entities Framework and I use Linq all the time.
I must say, I was impressed with the speed! and our site was having around 35 000 unique visitors per day, in around 60Gb bandwidth per day (I reduced radically this 60Gb number by hosting all static files in Amazon S3 - Great .NET wrapper they have, I must say).
I will always go this way. It's easy to start (just add new data item, choose tables and that's it! for every change in the database we just need to refresh the model - made automatically in just 2 clicks) and it's fun to use - Linq rules!