I often come to a point that when developing an application that connects to a MS SQL database for basic CRUD functions, I need to iterate through the returned results and populate a list of a particular data type that matches the table design.
Specific to .NET, I would just create a method (e.g. GetAllObjA()) and use ADO.NET to specify a SqlCommand that hooks up to a Stored Proc to fetch the data from the return SqlDataReader. I would then iterate through the returned rows, creating a new object and adding that to a list for each.
If however, I wanted to fetch data for different data type, I would rewrite this for methods GetAllObjB(), GetAllObjC() and so forth with the list's data type being different, which feels like a complete waste of rewriting code.
I realise Generics have a purpose here where the data type can be specified in the one method such as GetAll< T >(). Unfortunately this would still require me to define which table I would be fetching the data from and still require me to match the column names with the member names in the object, which doesn't really solve the problem as the application code has no way of knowing how the table is designed.
Is this the extent of Generics' usefulness in this instance? As the applications I am building are fairly small scale, I don't feel an ORM is warranted if it is possible to hand-code the solution instead.
I agree with the comments that micro-ORM's can either solve your scenario or give you some good ideas. Massive is just plain fun to read because it fits in one file. It uses the dynamic functionality of the Framework to solve your issue. Dapper is oriented towards the mapping aspect.
Also, take a look at the Data Access Application Block. Although this is now Open Source, it was originally maintained by Microsoft. It was an overall enterprise application framework, so there are several bloated dependencies that you do not need. But the data access block has some great prototypes for exactly what you are asking for: mapping a IDataReader resultset to a POCO using generics. So you write the mapping code only once, and the only thing you define per table is the actual reader-to-poco property mapping.
Sometimes a table or its mapping may have some quirks, so you want to keep the mapping definition by hand. For other simple tables with basic primitives, the use of dynamics as demonstrated by Rob Connery's Massive combined with the generic rowset mapper can make for some very easy-to-read-and-maintain code.
Disclaimer: this is not a judgement on the merits of this approach over EF. It is simply suggesting a simple, non-EF approach.
So, you might have a small library defining an interface:
public interface IResultSetMapper<TResult>
{
Task<List<TResult>> MapSetAsync(IDataReader reader);
}
and a generic ADO.Net helper class that processes any reader:
// a method with a class that manages the Command (_Command) and Connection objects
public async Task<List<TOut>> ExecuteReaderAsync<TOut>(IResultSetMapper<TOut> resultsMapper)
{
List<TOut> results = null;
try
{
using(var connection = await _DbContext.CreateOpenedConnectionAsync())
{
_Command.Connection = connection;
// execute the reader and iterate the results; when done, the connection is closed
using(var reader = await _Command.ExecuteReaderAsync())
{
results = await resultsMapper.MapSetAsync(reader);
}
}
return results;
}
catch(Exception cmdEx)
{
// handle or log exception...
throw;
}
}
So, the above code would be a helper library, written once. Then your mapper in your application might look like;
internal class ProductReaderMap : IResultSetMapper<Product>
{
public async Task<List<Product>> MapSetAsync(IDataReader reader)
{
List<Product> results = new List<Product>();
using(reader)
{
results.Add(new Product
{
ProductId = r.GetInt32(0),
ProductName = r.GetString(1),
SupplierId = r.GetInt32(2),
UnitsInStock = r.GetInt16(3)
});
}
return results;
}
}
You could break this out even further, defining a row mapper rather that a row set mapper, since the iteration over the reader can be abstracted as well.
Related
I am creating a Data Access Layer in C# for an SQL Server database table. The data access layer contains a property for each column in the table, as well as methods to read and write the data from the database. It seems to make sense to have the read methods be instance based. The question I have is regarding handling the database generated primary key property getter/setter and the write method. As far as I know I have three options...
Option 1: Using a static method while only allowing a getter on the primary key would allow me to enforce writing all of the correct values into the database, but is unwieldy as a developer.
Option 2: Using and instance based write method would be more maintainable, but I am not sure how I would handle the get/set on the primary key and it I would probably have to implement some kind of validation of the instance prior to writing to the database.
Option 3: Something else, but I am wary of LINQ and drag/drop stuff, they have burned me before.
Is there a standard practice here? Maybe I just need a link to a solid tutorial?
You might want to read up on active record patterns and some examples of them, and then implement your own class/classes.
Here's a rough sketch of a simple class that contains some basic concepts (below).
Following this approach you can expand on the pattern to meet your needs. You might be OK with retrieving a record from the DB as an object, altering its values, then updating the record (Option2). Or if that is too much overhead, using a static method that directly updates the record in the database (Option1). For an insert, the database (SP/query) should validate the natural/unique key on the table if you need to, and probably return a specific value/code indicating a unique constraint error). For updates, the same check would need to be performed if allowing natural key fields to be updated.
A lot of this depends on what functionality your application will allow for the specific table.
I tend to prefer retrieving an object from the DB then altering values and saving, over static methods. For me, it's easier to use from calling code and can handle arcane business logic inside the class easier.
public class MyEntityClass
{
private int _isNew;
private int _isDirty;
private int _pkValue;
private string _colValue;
public MyEntityClass()
{
_isNew = true;
}
public int PKValue
{
get {return _pkValue;}
}
public string ColValue
{
get {return _colValue;}
set
{
if (value != _colValue)
{
_colValue = value;
_isDirty = true;
}
}
}
public void Load(int pkValue)
{
_pkValue = pkValue;
//TODO: query database and set member vars based on results (_colVal)
// if data found
_isNew = false;
_isDirty = false;
}
public void Save()
{
if (_isNew)
{
//TODO: insert record into DB
//TODO: return DB generated PK ID value from SP/query, and set to _pkValue
}
else if (_isDirty)
{
//TODO: update record in DB
}
}
}
Have you had a look at the Entity Framework. I know you said you are wary of LINQ, but EF4 takes care of a lot of the things you mentioned and is a fairly standard practice for DALs.
I would stick with an ORM Tool (EF, OpenAccess by Telerik, etc) unless you need a customized dal that you need (not want) total control over. For side projects I use an ORM - at work however we have our own custom DAL with provider abstractions and with custom mappings between objects and the database.
Nhibernate is also a very solid tried and true ORM with a large community backing it.
Entity Framework is the way to go for your initial DAL, then optimize where you need it: Our company actually did some benchmarking in comparing EF vs SQL reader, and found that for querying the database for one or two tables worth of information, the speed is about 6's (neither being appreciably faster than the other). After two tables there is a performance hit, but its not terribly significant. The one place that writing your own SQL statements became worthwhile was in batch commit operations. At which point EF allows you to directly write the SQL queries. So save your self some time and use EF for the basic heavy lifting, and then use its direct connection for the more complicated operations. (Its the best of both worlds)
I’ve done some Googling but I have yet to find a solution, or even a definitive answer to my problem.
The problem is simple. I want to dynamically create a table per instance of a dynamically named/created object. Each table would then contain records that are specific to the object. I am aware that this is essentially an anti-pattern but these tables could theoretically become quite large so having all of the data in one table could lead to performance issues.
A more concrete example:
I have a base class/interface ACCOUNT which contains a collection of transactions. For each company that uses my software I create a new concrete version of the class, BOBS_SUB_SHOP_ACCOUNT or SAMS_GARAGE_ACCOUNT, etc. So the identifying value for the class is the class name, not a field within the class.
I am using C# and Fluent nHibernate.
So my questions are:
Does this make sense or do I need to clarify more? (or am I trying
to do something I REALLY shouldn’t?)
Does this pattern have a name?
Does nHibernate support this?
Do you know of any documentation on
the pattern I could read?
Edit: I thought about this a bit more and I realized that I don't REALLY need dynamic objects. All I need is a way to tie objects with some identifier to a table through NHibernate. For example:
//begin - just a brain dump
public class Account
{
public virtual string AccountName { get; set; }
public virtual IList Stuff { get; set; }
}
... somewhere else in code ...
//gets mapped to a table BobsGarageAccount (or something similar)
var BobsGarage = new Account{AccountName="BobsGarage"};
//gets mapped to a table StevesSubShop(or something similar)
var StevesSubShop = new Account{AccountName="StevesSubShop"};
//end
That should suffice for what i need, assuming NHibernate would allow it. I am trying to avoid a situation where one giant table would have the heck beat out of it if high volume occurred on the account tables. If all accounts were in one table... it could be ugly.
Thank you in advance.
Rather than creating a class on the fly, I would recommend a dynamic object. If you implement the right interfaces (one example is here, and in any case you can get there by inheriting from DynamicObject), you can write
dynamic bobsSubShopAccount = new DynamicAccount("BOBS_SUB_SHOP_ACCOUNT");
Console.WriteLine("Balance = {0}", bobsSubShopAccount.Balance);
in your client code. If you use the DLR to implement DynamicAccount, all these calls get intercepted at runtime and passed to your class at runtime. So, you could have the method
public override bool TryGetMember(GetMemberBinder binder, out object result)
{
if (DatabaseConnection.TryGetField(binder.Name, out result))
return true;
// Log the database failure here
result = null;
return false; // The attempt to get the member fails at runtime
}
to read the data from the database using the name of the member requested by client code.
I haven't used NHibernate, so I can't comment with any authority on how NHibernate will play with dynamic objects.
Those classes seem awfully smelly to me, and attempt to solve what amounts to be an actual storage layer issue, not a domain issue. Sharding is the term that you are looking for, essentially.
If you are truly worried about performance of the db, and your loads will be so large, perhaps you might look at partitioning the table instead? Your domain objects could easily handle creating the partition key, and you don't have to do crazy voodoo with NHibernate. This will also more easily permit you to not do nutty domain level things in case you change your persistence mechanisms later. You can create collection filters in your maps, or map readonly objects to a view. The latter option would be a bit smelly in the domain though.
If you absolutely insist on doing some voodoo you might want to look at NHibernate.Shards, it was intended for easy database sharding. I can't say what the current dev state and compatibility is, but it's an option.
I am working on converting a web application over to a WPF desktop application. I want to use the existing data access layer so I don't have to rewrite all the SQL queries.
But, as it stands, everything is populated almost purely from DataTables returned from SQL queries. To make things a bit easier to manage, in some instances, it would really be nice to convert these things into objects.
For example, I have a query that pulls out report information. I may get 500 results with columns like ReportID, ReportTitle, ReportDate.
I would like to make a report class that has these public attributes and somehow convert the SQL query results into a collection of these report objects.
What is the best way to go about doing this?
Super bonus points if there is an easy way of going backwards (updating the database if the objects are changed).
You should learn about Object-Relational Mapping (ORM). A good ORM can save you tonnes of work in the future, and gets those nasty queries out of your code.
I'd recommend NHibernate or Entity Framework 4.0
While I would also like to suggest ORM (NHibernate is the way to go:)) a possible solution is:
public IEnumerable<Report> ToReportList(DataTable dt)
{
return dt.AsEnumerable().Select(dr => new Report
{
member1 = dr["column1"].ToString(),
...
});
}
Report is your class here by the way.Such as,
internal class Report
{
public string member1{ get; set;}
...
}
You may also want to check this,
http://msdn.microsoft.com/en-us/library/Bb399375(v=VS.100).aspx
I think if you search stackoverflow, you will find nicer examples as I remember learning this from here.
By the way, if you use NHibernate, you won't have to rewrite your queries at all. All you have to do is map your tables to a class and booya you are good to go. It will handle all your DML stuff (well mostly) and you can easily tell the ORM to do LazyLoad, Batch processing etc which is pretty cool.
Super bonus points if there is an easy
way of going backwards (updating the
database if the objects are changed).
For that , go for ORM i.e. NHibernate (I know I am biased :)). For LINQ to SQL examples check the 2 links below:
http://msdn.microsoft.com/en-us/library/bb386931.aspx
http://csainty.blogspot.com/2008/01/linq-to-sql-insertupdatedelete.html
+1 ORM. Entity Framework is good, LINQ to SQL is good too, but you'd need a good database design and is better for pretty basic SQL CRUD actions. For custom entities from multiple datasources, I'd go EF.
As far as backwards updating - LINQ to SQL has an easy-peasy implementation, sorta like this - say you've got a db called MyDatabase with Dog entities in it:
using(MyDatabaseDataContext db = new MyDatabaseDataContext())
{
//LINQ will automatically pluralize your items (entity named Dog becomes Dogs)
Dog d = db.Dogs.Where(x=>x.DogName.Equals(dogName));
d.Owner = "Steve";
db.SubmitChanges();
//adding new items is easy too
Dog newdog = new Dog();
newDog.DogName = "Scruff";
newDog.Owner = "Jim";
db.Dogs.Add(newDog);
db.SubmitChanges();
}
Check out this method. First you can create a class that inherits DataContext. And then you can use methods of DataContext like to ExecuteQuery<> to convert your results in to objects. With this method you can directly use your queries you wrote earlier. Also, I feel that this way is a lot better than maintaining a .dbml file, because of the issues associated to synchronization with the actual database schema.
Consider the following example. First you need to define a class for interacting with your database
public class DBManagerDataContext : DataContext
{
private static string connectionString = ""; // Your connection string
public static DBManagerDataContext CreateInstance()
{
return new DBManagerDataContext(connectionString);
}
protected DBManagerDataContext(string connectionString)
: base(connectionString, new AttributeMappingSource())
{
}
}
Then you can use this context to execute queries and convert them in to objects as shown below:
public class Report
{
public int ReportID;
public string ReportTitle;
public DateTime ReportDate;
private static string query = "select ReportID, ReportTitle, ReportDate from dbo.Reports"; // Your query
public static List<Report> GetReportList()
{
DBManagerDataContext context = DBManagerDataContext.CreateInstance();
return context.ExecuteQuery<Report>(query).ToList();
}
}
You can use the method "GetReportList()" given above like this for example:
List<Report> reports = Report.GetReportList();
In the case of updates, the method suggested by "jdandison" is a nice option, apart from using the data context as above. In the case of updates it would be "ExecuteCommand" though. Please explore the DataContext class for more information.
Edit: Please note that the query column names should match the definition in the object
We are designing a product which could support multiple databases. We are doing something like this currently so that our code supports MS SQL as well as MySQL:
namespace Handlers
{
public class BaseHandler
{
protected string connectionString;
protected string providerName;
protected BaseHandler()
{
connectionString = ApplicationConstants.DatabaseVariables.GetConnectionString();
providerName = ApplicationConstants.DatabaseVariables.GetProviderName();
}
}
}
namespace Constants
{
internal class ApplicationConstants
{
public class DatabaseVariables
{
public static readonly string SqlServerProvider = "System.Data.SqlClient";
public static readonly string MySqlProvider = "MySql.Data.MySqlClient";
public static string GetConnectionString()
{
return ConfigurationManager.ConnectionStrings["CONNECTION_STRING"].ConnectionString;
}
public static string GetProviderName()
{
return ConfigurationManager.ConnectionStrings["CONNECTION_STRING"].ProviderName;
}
}
}
}
namespace Handlers
{
internal class InfoHandler : BaseHandler
{
public InfoHandler() : base()
{
}
public void Insert(InfoModel infoModel)
{
CommonUtilities commonUtilities = new CommonUtilities();
string cmdInsert = InfoQueryHelper.InsertQuery(providerName);
DbCommand cmd = null;
try
{
DbProviderFactory provider = DbProviderFactories.GetFactory(providerName);
DbConnection con = LicDbConnectionScope.Current.GetOpenConnection(provider, connectionString);
cmd = commonUtilities.GetCommand(provider, con, cmdInsert);
commonUtilities.PrepareCommand(cmd, infoModel.AccessKey, "paramAccessKey", DbType.String, false, provider, providerName);
commonUtilities.PrepareCommand(cmd, infoModel.AccessValue, "paramAccessValue", DbType.String, false, provider, providerName);
cmd.ExecuteNonQuery();
}
catch (SqlException dbException)
{
//-2146232060 for MS SQL Server
//-2147467259 for MY SQL Server
/*Check if Sql server instance is running or not*/
if (dbException.ErrorCode == -2146232060 || dbException.ErrorCode == -2147467259)
{
throw new BusinessException("ER0008");
}
else
{
throw new BusinessException("GENERIC_EXCEPTION_ERROR");
}
}
catch (Exception generalException)
{
throw generalException;
}
finally
{
cmd.Dispose();
}
}
}
}
namespace QueryHelpers
{
internal class InfoQueryHelper
{
public static string InsertQuery(string providerName)
{
if (providerName == ApplicationConstants.DatabaseVariables.SqlServerProvider)
{
return #"INSERT INTO table1
(ACCESS_KEY
,ACCESS_VALUE)
VALUES
(#paramAccessKey
,#paramAccessValue) ";
}
else if (providerName == ApplicationConstants.DatabaseVariables.MySqlProvider)
{
return #"INSERT INTO table1
(ACCESS_KEY
,ACCESS_VALUE)
VALUES
(?paramAccessKey
,?paramAccessValue) ";
}
else
{
return string.Empty;
}
}
}
}
Can you please suggest if there is any better way of doing it? Also what are the pros and cons of the approach?
Whatever you do, don't write your own mapping code. Its already been done before, and its probably been done a million times better than whatever you could write by hand.
Without a doubt, you should use NHibernate. Its an object-relational mapper which makes database access transparent: you define a set of DAL classes which represent each table in your database, and you use the NHibernate providers to perform queries against your database. NHibernate will dynamically generate the SQL required to query the database and populate your DAL objects.
The nice thing about NHibernate is that it generates SQL based on whatever you've specified in the config file. Out of the box, it supports SQL Server, Oracle, MySQL, Firebird, PostGres and a few other databases.
I'd use NHibernate.
Here's nice beginner tutorial
For your current need, I agree with NHibernate...
Just want to point out something with your class hierarchy...
You will better to use Interface
Like (Just check the doc or Internet for exact syntax)
Interface IDBParser
Function1
Function2
class MSSQLParser : IDBParser
Function1
Function2
class MySQLParser : IDBParser
Function1
Function2
Then in your code you can use the interface
Main()
IDBParser dbParser;
if(...)
dbParser = new MSSQLParser();
else
dbParser = new MySQLParser();
SomeFunction( dbParser );
// the parser can be sent by parameter, global setting, central module, ...
SomeFunction( IDBParser dbParser)
dbParser.Function1();
That way it will be easier to manage and your code won't be full of the same if/else condition. It will be also a lot easier to add others DB. Another advantage is that it could help you with unit testing by sending mock object.
If you have to code it yourself and not use a product which provides unified access, remember that objects like SqlDataAdapter and OracleDataAdapter inherit from the common DbDataAdapter (at least in later versions of the runtime). If you cast down to DbDataAdapter, you can write code that will work with both databases in the places where you would do the same for both databases. Some of your code will look a little like this:
DbDataAdapter adapter = GetOracleDataAdapter() as DbDataAdapter;
Once you've casted down, it doesn't matter if it is a SqlDataAdapter or a OracleDataAdapter. You can call it the same way.
However, remember that coding for two databases means using features that exist only within both while having to work around the shortcomings of both. It's not really a good idea.
If you need a mapping from database entries to Objects I suggest you go with the solution other already suggested: NHibernate.
If this seems like overkill for your application and you want to go with the Ado.net approach and do not need a O/RM-soultion, you should have a look on what the Spring.net guys did and learn about the Ado.Net Provider Abstraction.
There are object-relational mapping layers out there that will support multiple database technologies, like Entity Spaces.
What's always good in such cases is to create a layered architecture, where all the DB related stuff is JUST in the data access layer. Then you could have different implementations of your DAO layer, one for Oracle, SQL Server etc...
You should separate the business layer from the DAO layer with interfaces, such that your Business layer just uses them to access the DAO layer. So you could perfectly exchange the underlaying implementation of the DAO layer to run on an Oracle DB or whatever system you like.
Another good suggestion is to take a look at object-relational mappers like Scott already suggested. I'd take a look at NHibernate or Entity framework.
Many people have suggested an O/R mapping framework such as NHibernate. This is quite a reasonable approach unless you don't want to use an O/R mapper for some reason. Something like NHibernate will probably get you 95%+ of the way but you may need to write some custom SQL. Don't panic if this is the case; you can still do an ad-hoc solution for the rest.
In this case, take the bits that do need the custom SQL for and separate them out into a platform specific plugin module. Write Oracle, MySQL, SQL Server (etc.) plugins as necessary for the individual database platforms you want to support.
ADO.Net makes it fairly easy to wrap sprocs, so you might be able to move the platform dependent layer down into some stored procedures, presenting a more-or-less consitent API to the middle tier. There are still some platform dependencies (such as the '#' prefix on SQL Server variable names), so you would need to make a generic sproc wrapper mechanism (which is not all that hard).
With any luck, the specific operations you need to break out in this manner will be fairly small in number so the amount of work to maintain the plugins will be limited.
One approach to this problem is to design your application to work entirely with disconnected DataSets, and write a data access component that handles fetching data from the different database brands you'll be supporting, as well as persisting changes made to the DataSets by your application back to the original databases.
Pros: DataSets in .Net are well-written, easy-to-use and powerful, and do an excellent job of providing methods and tools for working with table-based data.
Cons: This method can be problematic if your application needs to work with extremely large sets of data on the client side.
Right now, Microsoft's Entity Framework has a few shortcommings, some of them that can be deal breakers, depending on the application's intended architecture.
From what I've seen and read about V2, which will be shipped with .Net 4, I think it will certainly deserve to be looked at.
I am trying to insert 61,000+ objects, obtained via a remote call, into a SQL Server 2005 database in the fastest method possible. Any suggestions?
I have looked at using SQLBulkCopy, but am having a few problems working out how to get the data into the right format since I am not starting with a DataTable, but instead have a list of objects. So if answers could contain code samples that would be appreciated.
I am trying to insert the data into a temp table before processing it to keep memory usage down.
Edit...
#JP - this is something that will run every night as a scheduled batch job with an IIS ASP.NET application.
Thanks.
If this is something you are doing one time or only periodically, you should look at using SSIS (it's basically DTS on steroids). You could build a package that gets the data from one datasource and inserts it into another. There are also features for stop/start and migration tracking. Without more details on your situation, I can't really provide code, but there are a lot of code samples out there on SSIS. You can learn more and play around with SSIS in Virtual Labs.
If you intend on using the SQLBulkCopy class I would suggest that you create a custom class that implements IDataReader that will be responsible for mapping the 61000 source data objects to the appropriate columns in the destination table and then using this custom class as a parameter to the SQLBulkCopy WriteToServer method.
The only tricky part will be implementing the IDataReader interface in your class. But even that shouldn't be too complicated. Just remember that your goal is to have this class map your 610000 data objects to column names. And that your class will be called by the SQLBulkCopy class to provide the data. The rest should come together pretty easily.
class CustomReaderClass : IDataReader
{
// make sure to implement the IDataReader inferface in this class
// and a method to load the 61 000 data objects
void Load()
{
// do whatever you have to do here to load the data..
// with the remote call..?!
}
}
//.. later you use it like so
SQLBulkCopy bulkCopyInstance;
CustomReaderClass aCustomReaderClass = new aCustomReaderClass();
aCustomReaderClass.Load();
// open destination connection
// .. and create a new instance of SQLBulkCopy with the dest connection
bulkCopyInstance.WriteToServer(aCustomReaderClass);
// close connection and you're done!
I hope the above "pseudo-code" makes some sense..
#Miky D had the right approach, but I would like to expand the details. Implementing IDataReader is not really that hard.
To get IDataReader working with a bulk inserter you should look at implementing:
Dispose();
FieldCount {
object GetValue(int i);
GetSchemaTable();
Read();
The rest can be stubs that throw NotImplementedExceptions, see this sample
Getting the schema table is also pretty easy. Just select one row from the target table and call GetSchemaTable().
To keep stuff clearer I like to have an abstract class that throws NotImplementedException on the non essential methods, perhaps down the line that abstract class can implement the missing bits for added robustness.
A couple of BIG caveats with this approach:
Which methods to implement is not documented in SQLBulkCopy
With the follow on that, in later versions of the framework/hotfixes or service pack may break you. So if I had mission critical code I would take the bite and implement the whole interface.
I think, that its pretty poor that SQLBulkCopy does not have an additional minimal interface for bulk inserting data, IDataReader is way to fat.