I need to execute read-only queries against a database that I don't control. My top choice would be Linq to SQL, however, the column names differ slightly between our Dev, QA, and Production environments.
For example, take a FolderName column. We might have:
Dev: u34_FolderName
QA: u74_FolderName
PROD: u56_FolderName
I want to do queries like this:
var query = from c in DepartmentReviews
where c.FolderName == "Test"
Can I use the Entity Framework to solve this problem?
What a potential solution look like? 3 assemblies, 1 for each of my environments? Can I create common interfaces or base classes for each of these and code against those?
Changing schemas are always a problem. Though I am not a big advocate of storing XMLs in a database, your problem sounds like it can make use of this approach.
This solution requires you to make one (and only one schema change).It may/may not be feasable in your application.
In your table you probably would have a column for unique identifier and a column for xml (SQL Server 2005/2008 naturally support it). You can serialize it as XML (you probably would end up using a generic XML Serializer- Serializer<T> whose type you would infer at run time using reflection). Deserialize it and you can get your object. You can read more about it here.
So your query would be like
var myXML = from c in ObjectContext.Table
where c.FolderName == "Test" select MyXmlColumn;
var myType = InferTypeFromConfig();
var serializer = new XmlSerializer<myType>();
var object = (myType )serializer.Deserialize(TheXMLStreamCreatedFrom(myXml));
my $0.02
You may need to use an XmlMappingSource instead of the default AttributeMappingSource. With an XmlMappingSource you can customise the column mappings in an xml file, independent of the main application. This should allow you to have different mappings from dev, qa and production.
Have you considered making your own data access layer? If your queries are limited to a specific set of cases, it wouldn't be a fully-fledged framework. You would have a common vocabulary sit atop the three database schemas, and you would pick the correct raw column name at runtime. You'd just stitch up your queries the old fashioned way.
Although you would have to write your queries manually, you still retain a lot of the client-side power of LINQ if your native data structures are IEnumerable.
Using Entity Framework you could manipulate the EDMX file at build time to achieve the desired result for each environment, see http://msdn.microsoft.com/en-us/library/cc982042.aspx
You could for example create a project that does the pre-processing for the appropriate environment at build time and include this project file in MSBUILD as the first project that gets built.
Or you could generate multiple CSDL, MSL, SSDL files and switch between them at runtime instead of using the default behavior which is to load them from resources that are embedded during the build process.
In the EF 4 you can do this fairly easily with Code First, which is currently in CTP.
Related
I'm dynamically importing data into a database where I create new tables on the fly and store the metadata so that I can access those tables later via dynamically constructed SQL. My question is, for C#, is there a library out there that I can use that can abstract away some of the details of the SQL itself? The situation I'm running into is with sequences (although there are others). In Oracle accessing a sequence looks like this
select foo.nextVal from dual;
In Postgres...
select currval('foo_id_seq');
For my project I don't know what the final database will be and I don't like the idea of running through the project fixing a bunch of errors due to bad SQL.
I looked at NHibernate and it looks like tools like that (Linq to SQL) require an existing object model in place. I don't have an object model because all of my data is dynamically provided and I don't know the number of columns, data types, etc.
Any suggested approach to this problem is appreciated.
If the data you're trying to store has a dynamic structure, then it really sounds like a relational database may not be the best choice. It's strengths rely on data being statically structured and well defined. You might be better served with a document oriented store like MongoDB which is designed for dynamic schemas. If you used something like MongoDB, I think your question around abstracting query generation for dynamically changing schemas goes away.
That said, some relational databases like SQL Server have good support for XML data types which allow you to specify an arbitrary structure within your static schema. SQL Server also allows you to query directly into XML data types and even index them, which means you can query on the server side without the need for transferring the XML back to the client, deserializing, etc. To decide if this will perform well enough for your needs you'll have to test with data that will represent your production load.
So far in my .Net coding adventures I've only had a need to save information to files. So I've used XmlSerializer and DataContractSerializer to serialize attributed classes to XML files. My next project, however, requires that I save and retrieve information from a SQL server database. I'm wondering what my options are for doing this.
The current version of the app, which was not created by me, uses a lot of hard coded SQL commands. But now I'm trying to avoid doing anything where I have to read or write individual fields to or from the database or objects. I especially want to avoid a lot of hard coded SQL in my code. I like how the serializer classes just figure out how to read and write XML files based on the attributes and or public properties of the class. Is there something similar for a database rather then XML?
Object Relational Mapping
There are bunch of products out there, most notorious one being NHibernate, there are couple of competing products offered by Microsoft in Linq 2 Sql and Entity Framework (you're supposed to use the later, but everyone uses the first as is waaaay simpler).
You can see a nice (although I suspect biased) comparison of ORM offerings at http://ormbattle.net/
I believe you're referring to Object Relational Mappers. These provide a wealth of functionality, including simple object CRUD plumbing.
Check out:
NHibernate
Entity Framework
Linq to SQL
There are many others, but that'll get you going.
There is no generic object type when you deal with databases. Only tables and fields.
The combination of these could make an object though. Your best bet is to use stored procedures if you are concerned with hard coded SQL on the client code.
I'm also mainly referring to the actual field types in a database. ORM's are a different story. If you want look into nHibernate if you want an object relational mapper that can help with INSERTs, SELECTs, etc.
Depending on the project an ORM like NHibernate might be what you're looking for. Something where you map your database information to classes and the ORM takes care of the inserts, deletes, and selects for you without hand-written SQL. This also allows for migration to a different database system without a ton of rewrite.
I say it depends on the project because other things come into play here like performance and how the data is actually structured.
I think you should read up on Linq to SQL. This will allow you to work "primarily" with classes that are representations of your database tables and their relations.
DataContext context = new DataContext();
var obj = context.Table1.Single(row => row.Id == 1234);
obj.Name = "Test1234";
context.SubmitChanges();
This could be a good place to start to learn about Linq to SQL
Hope this is what you are looking for.
I agree with (and prefer) the previous suggestions to use an ORM. Just to make sure you have a full menu of options here is another option. If you're comfortable with the XML representation, (de)serialization, etc... you could also look into using SQLXML. With that said, you should not use this to avoid doing proper database design although this can be totally reasonable for some solutions.
Which ORM will give me compile-tested queries?
Is linqtosql compile time tested?
Edit:
Say I write a query that references a column named 'TotalSales'. I then rename the column in my database to TotalSales2 (and any other config file like: Employee.cfg.xml in nHibernate).
When I compile the project, I want Visual Studio to tell me the column 'totalSales' doesn't exist and then I will go and change it.
There aren't any as far as I'm aware. They will often let you create a LINQ query that cannot be translated into SQL for example. Also, I am not aware of any compile time checking that your mappings map to your database correctly.
You can, and should in my opinion, perform all these checks within tests. Most ORMs make this easy to do.
I use LLBLGen but it has to be "refreshed" when data model changes are made. I don't think you'll get an ORM that will AT COMPILE TIME check for modifications against the database. You're asking for quite a bit there.
In DataObjects.Net properties marked by [Field] attribute are always bound to field in database, so you can be sure that query will be translated. If you use not persistent field or another not supported statement, query translator will fail in runtime or performs such operation on fetched objects (on client).
Generally compile time validation is impossible or theoretically can be performed with special post-build tasks, that will scan compiled code, find all queries and validate them. But such checks will seriously slow down compilation process.
Perhaps not exactly what you're looking for but if using the Entity Framework and selecting "Update Model From Database" from the designer you will get messages saying the fields are no longer mapped if you change the names.
This doesn't happen automatically when you build a project.
Basically, you need 2 features together:
Compile-time checked queries (= an ORM with LINQ implementation). This is normally not a problem - at least some tools support this.
Pre-build step updating your entities based on database schema. AFAIK this is rarely implemented this way - normally you must explicitly update the model by the schema. Note that this part is normally rather costly.
SubSonic can do that if you include the code generation step as a pre-build event.
I used a Java tool called DODS, which was developed ca. 2000 with the Enhydra application server. DODS is still around here: http://www.enhydra.org/tech/dods/
The way DODS works, and which meets your goal of compile-time validation, is that it's a code generation tool. It generates Java classes corresponding to the tables in your database. Object instances of these classes have getters and setters for each column in the table. Of course if you change your database structure, you have to re-generation the Java code using DODS.
As long as you keep the generated code up to date with the structure of your database, it provides compile-time validation that any application code that uses these classes is querying valid tables and columns.
Anyway, I realize you tagged your question with C# and ASP.NET. A tool that generates Java code isn't going to be that helpful for you. But there could be another tool more specifically for .NET that works on the same principle of generating code that maps to database structure. So I'd suggest narrowing your search to .NET ORM tools that say something about code generation.
I will likely be responsible for porting a vb6 application to c#. This application is a windows app that interacts with an access db. The data access is encapsulated in basic business objects. One class for one table basically. The existing vb6 business objects read and write to the DB via DAO. I have written DALs and ORMs a few times before but they all targeted SQL Server only. This one will need to target access and sql server. In previous projects, I would place the SQL strings in the private parts of the business object and maybe move the redundant sql code like connecting, creating command, in into a common base class to reduce the code.
This time, i'm thinking about writing the SQL strings into a .settings file or some other key/value type text file. I would then write a sql utility to edit this file and allow me to run and test the parameterized queries. These queries would be referenced by name in the business object instead of embedding the sql into code.
I know a standard approach is to create a DAL for each targeted database and have the configuration state which DAL to use. I really don't want to create the two DAL classes for each database. It seems like it would be less code if I just referenced the correct query by keyname and have the proper type of connection.
So, are you guys doing things like this? How would or have you approached this problem?
What works best for you?
Thanks!
Well, there's a lot of options - so it really depends on what your most pressing needs are :-)
One approach might be to create SQL statements as text files inside your VS solution, and mark them as "embedded resource" in the "build action". That way, the SQL is included in your resulting assembly, and can be retrieved from it at runtime using the ResourceManifestStream of the .NET framework:
private string LoadSQLStatement(string statementName)
{
string sqlStatement = string.Empty;
string namespacePart = "ConsoleApplication1";
string resourceName = namespacePart + "." + statementName;
using(Stream stm = Assembly.GetExecutingAssembly().GetManifestResourceStream(resourceName))
{
if (stm != null)
{
sqlStatement = new StreamReader(stm).ReadToEnd();
}
}
return sqlStatement;
}
You need to replace "ConsoleApplication1" with your actual namespace, in which the sql statement files reside. You need to reference them by means of the fully qualified name. Then you can load your SQL statement with this line:
string mySQLStatement = LoadSQLStatement("MySQLStatement.sql");
This however makes the queries rather "static", e.g. you cannot configure and change them at runtime - they're baked right into the compiled binary bits. But on the other hand, in VS, you have a nice clean separation between your C# program code, and the SQL statements.
If you need to be able to possibly tweak and change them at runtime, I'd put them into a single SQL table which contains e.g. a keyword and the actual SQL query as fields. You can then retrieve them as needed, and execute them. Since they're in the database table, you can also change, fix, amend them at will - even at runtime - without having to re-deploy your whole app.
Marc
When I really need it, I put the queries into individual *.sql files, then include them into Resources.resx. There is a 'Files' section in it, which allows you to include Embedded Resource files.
After that, I can use generated Resources.MyQuery property which both guarantees that resource exists and saves me from writing a custom resource load method.
LINQ to DataSet sounds like the way to go for you.
If you havent used the .NET 3.5 before / LINQ then you're in for a treat. LINQ will save you writing your raw sql in string literals and provide you with a more logical way to creating querys.
Anyway, check this link out for using LINQ on Access databases - http://msdn.microsoft.com/en-us/library/bb386977.aspx
If i'd had to create application for both SQL and Access, I'd use some IDAL interface, DALCommon with common functionality implementation and separate DALSql and DALAccess, inherited from DALCommon, with some specific stuff, like exceptions, transactions handling, security etc.
I used to keep stored procedure names or queries in resource files.
I'll tell where I won't put it ever, something I saw done in some code I inherited. It was in Java, but applies to any language
A base class that declared protected static member variables for for SQL statements, inited to null, with a get method that returns individual SQL statements
A sub class for each supported database server, with an init method that assigns to the base class member variables
Several DA classes that use the base class method to retrieve SQL statements
The application start-up class with the responsibility to create the correct sub-class object and call its init method
I will also not go into explaining why I will not do this ever :-)
One method we used is to have a class that would connect to the DB and methods to call procedures and in the method parameter you would provide the procedure name. so all the SQL code is in the procedure. we would use overloads for the different return types
class ConnectToSQL()
{
//connectSql code (read from setting file i assume)
XMLDataDocument runProcedure(string procedureName);
int runProcedure(string procedureName);
//etc....
}
Sometimes, like with custom reporting apps, you really need to embrace the impedance mismatch, and give special importance to the SQL. In these cases I recommend the following: For each module that contains SQL strings, create a single static "SQL" class to hold them all. Some of the SQL strings will likely require parameters, so be consistent and put each string behind it's own static method.
I only do this for the occasional custom reporting app, but it always works out great and feels refreshing and liberating. And it's quite nice to come back months later to make an enhancement, and find all of the SQL waiting for you in a single SQL.cs file. Just by reading that one file, it all comes back, and often this is the only file that needs to be changed.
I don't see a need in these cases for hiding the SQL in resources or elsewhere. When SQL is important, then it's important. Interestingly, more and more developers are now freely mixing SQL with C#, including I believe this site, because essentially, that's what LINQ is.
Finally, as always, make sure you are not susceptible to SQL injection attacks. Especially if user input is involved, make sure you are using some kind of parameterization and that you are not using string concatenation.
Embedding solutions shown above may not work if SQL Query has a "where" cause like , but for the same Query the next run needs PropertyID='113' as the PropertyID is read-in.
Glad you asked! Put your sql in a QueryFirst .sql template.
It's automatically compiled into your app as an embedded resource, but you don't care. You just write it, in a real sql window, connected to your DB, with syntax validation and intellisense for tables and columns, then use it, via the generated Execute() methods, with intellisense for your inputs and results.
disclaimer : I wrote QueryFirst.
I am trying to leverage ORM given the following requirements:
1) Using .NET Framework (latest Framework is okay)
2) Must be able to use Sybase, Oracle, MSSQL interchangeably
3) The schema is mostly static, BUT there are dynamic parts.
I am somewhat familiar with SubSonic and NHibernate, but not deeply.
I get the nagging feeling that the ORM can do what I want, but I don't know how to leverage it at the moment.
SubSonic probably isn't optimal, since it doesn't currently support Sybase, and writing my own provider for it is beyond my resources and ability right now.
For #3 (above), there are a couple of metadata tables, which describe tables which the vendors can "staple on" to the existing database.
Let's call these MetaTables, and MetaFields.
There is a base static schema, which the ORM (NHibernate ATM) handles nicely.
However, a vendor can add a table to the database (physically) as long as they also add the data to the metadata tables to describe their structure.
What I'd really like is for me to be able to somehow "feed" the ORM with that metadata (in a way that it understands) and have it at that point allow me to manipulate the data.
My primary goal is to reduce the amount of generic SQL statement building I have to do on these dynamic tables.
I'd also like to avoid having to worry about the differences in SQL being sent to Sybase,Oracle, or MSSQL.
My primary problem is that I don't have a way to let ORM know about the dynamic tables until runtime, when I'll have access to the metadata
Edit: An example of the usage might be like the one outlined here:
IDataReader rdr=new Query("DynamicTable1").WHERE("ArbitraryId",2).ExecuteReader();
(However, it doesn't look like SubSonic will work, as there is no Sybase provider (see above)
Acording to this blog you can in fact use NHibernate with dynamic mapping. It takes a bit of tweaking though...
We did some of the using NHibernate, however we stopped the project since it didn't provide us with the ROI we wanted. We ended up writing our own ORM/SQL layer which worked very well (worked since I no longer work there, I'm guessing it still works).
Our system used a open source project to generate the SQL (don't remember the name any more) and we built all our queries in our own Xml based language (Query Markup Language - QML). We could then build an xmlDocument with selects, wheres, groups etc. and then send that to the SqlEngine that would turn it into a Sql statement and execute it. We discusse, but never implemented, a cache in all of this. That would've allowed us to cache the Qmls for frequently used queries.
I am a little confused as to how the orm would be used then at runtime? If the ORM would dynamically build something at runtime, how does the runtime code know what the orm did dynamically?
"have it at that point allow me to manipulate the data" - What is manipulating the data?
I may be missing something here and i aplogize if thats the case. (I only have really used bottom up approach with ORM)
IDataReader doesn't map anything to an object you know. So your example should be written using classic query builder.
Have you looked into using the ADO.NET Entity Framework?
MSDN: LINQ to Entities
It allows you to map database tables to an object model in such a manner that you can code without thinking about which database vendor is being used, and without worrying about minor variations made by a DBA to the actual tables. The mapping is kept in configuration files that can be modified when the db tables are modified without requiring a recompile.
Also, using LINQ to Entities, you can build queries in an OO manner, so you aren't writing actual SQL query strings.