We Keep Coding

Where to migrate data from SQL Server?

I'm working on an application which was previously developed with EF, SQL Server, C# and WinForms. For some reasons we need to remove SQL Server and use some and put data in some other container like XML, Excel sheet or in memory classes.
DAL was designed using EF with repository and UOW pattern. Can you please suggest me what can be the best way to migrate and what format will be easy to transfer data?

Since the data access is decoupled into a repository you should be able to write a new repository instance that will allow you to retrieve your data in its new format. Specifically if you wanted to hold the data in XML you could just write a new repository that uses Linq-to-XML to retrieve your data.
I don't know of a good quick way to transfer your data from SQL Server into XML. If I were doing it I'd probably write some custom application to make the conversion, but I suspect there are better solutions out there for this type of migration.

Your question duplicates this questions:
Entity Framework with XML Files
How to use Entity Framework 4.0 with Xml or in-memory Storage (non-SQL)
There seems to be a provider for virtuoso xml:
http://virtuoso.openlinksw.com/dataspace/dav/wiki/Main/VirtAdoNet35Provider
Some hints can be found here:
http://social.msdn.microsoft.com/Forums/en/adodotnetentityframework/thread/6f6164f6-fced-43de-b3fc-eccbd74dd482
http://blogs.msdn.com/b/adonet/archive/2009/11/05/model-first-with-the-entity-framework-4.aspx
http://en.wikipedia.org/wiki/ADO.NET_Entity_Framework
EF ist basically developed for database abstraction. You will need to write your own xml database provider if you try to strore XML instead.
Your options are:
move to SQL compact (but not in XML)
use XPath and rewrite your code
use Xpath for your own database provider
Be aware that your xml file size and speed depends on the amount of data! Using multiple xml files can end up in a mess.

Using custom file system with Entity Framework 4.0

Is it possible, or even recommended, to use a custom file system with the Entity Framework? I am working on a learning project to see if I can take some custom intermediate files (the majority of them binary files) from an older file system and use the Entity Framework to pass data through a WCF service with OData.
I'm having quite a bit of trouble finding tutorials on how to possibly do this. Would this not be the recommended approach with a custom filesystem? I really like the capability of WCF services and want to take advantage of them for sure. I am not sold on the entity framework, is there a better way to pass this data into a WCF service?
Thank you.

I would advise against this. Essentially you want to use EF as a database engine. You need to ensure that your binary files are queryable. And that you have a streaming architecture to avoid doing a total in memory copy of the data.
However if the data is small enough then I would simple create an object model that represents your data and use basic Linq for the interrogation of it.
There is another solution that allows streaming from a text file (Implementing an IEnumerable on your object and use yield from a tokenized file stream.
However I suspect the aim of your problem is to use WCF Data Services to make your life easier, not more complicated. In which case write this idea off and go back to the drawing board, as this is NOT going to be trivial

I don't see why not. EF is designed to work with various data providers. There are open source projects like the Postgresql provider for EF that you can study to find out how to do it.

Is there a method of serialization that is similar to a database

So far all the serialization examples I have found on the web are related to storing arrays or list in a file. With each class of object having to be serialized into their own file such as a ".bin". The root of my problem is that I want to have the information for my product local stored, but I'm so use to working with sql. It's hard for me to visualize how to store information locally. If C# is anything like asp I should be able to connect to an Access database, but that pretty much defeats one of the ideas of serialization which is user non-readability. Is there a serialization method similar to using table and fields or at least allowing you to store all user information in one file?

You could use a ADO.NET DataSet that is serialized and stored locally. It will contain all of the data structures that you're familiar with and allow you to query the data the way you seem to want to and if you serialize it with a Binary Serializer, it will be unreadable to end-users.
Also, you could look at SQLite as an alternative to using DataSets.
SQLite is a software library that
implements a self-contained,
serverless, zero-configuration,
transactional SQL database engine.
SQLite is the most widely deployed SQL
database engine in the world. The
source code for SQLite is in the
public domain.
NHibernate with SQLite is a great combination as well.
Cheers.

Check out NHibernate. That will give you your 'database-like' storage.
If it's human-readability you're after, consider serializing your objects using XML. .Net has decent support for serializing (and deserializing) objects using both XML and binary formats.
The tutorial I used for learning serialization in C# is this CodeProject article.
Update:
I misread one point you made: serialization does not necessarily mean human-readable or not - if you decide to serialize, figure out if you want the data readable or not. Binary serialization is likely to be more compact and less readable.

Save objects to a database?

So far in my .Net coding adventures I've only had a need to save information to files. So I've used XmlSerializer and DataContractSerializer to serialize attributed classes to XML files. My next project, however, requires that I save and retrieve information from a SQL server database. I'm wondering what my options are for doing this.
The current version of the app, which was not created by me, uses a lot of hard coded SQL commands. But now I'm trying to avoid doing anything where I have to read or write individual fields to or from the database or objects. I especially want to avoid a lot of hard coded SQL in my code. I like how the serializer classes just figure out how to read and write XML files based on the attributes and or public properties of the class. Is there something similar for a database rather then XML?

Object Relational Mapping
There are bunch of products out there, most notorious one being NHibernate, there are couple of competing products offered by Microsoft in Linq 2 Sql and Entity Framework (you're supposed to use the later, but everyone uses the first as is waaaay simpler).
You can see a nice (although I suspect biased) comparison of ORM offerings at http://ormbattle.net/

I believe you're referring to Object Relational Mappers. These provide a wealth of functionality, including simple object CRUD plumbing.
Check out:
NHibernate
Entity Framework
Linq to SQL
There are many others, but that'll get you going.

There is no generic object type when you deal with databases. Only tables and fields.
The combination of these could make an object though. Your best bet is to use stored procedures if you are concerned with hard coded SQL on the client code.
I'm also mainly referring to the actual field types in a database. ORM's are a different story. If you want look into nHibernate if you want an object relational mapper that can help with INSERTs, SELECTs, etc.

Depending on the project an ORM like NHibernate might be what you're looking for. Something where you map your database information to classes and the ORM takes care of the inserts, deletes, and selects for you without hand-written SQL. This also allows for migration to a different database system without a ton of rewrite.
I say it depends on the project because other things come into play here like performance and how the data is actually structured.

I think you should read up on Linq to SQL. This will allow you to work "primarily" with classes that are representations of your database tables and their relations.
DataContext context = new DataContext();
var obj = context.Table1.Single(row => row.Id == 1234);
obj.Name = "Test1234";
context.SubmitChanges();
This could be a good place to start to learn about Linq to SQL
Hope this is what you are looking for.

I agree with (and prefer) the previous suggestions to use an ORM. Just to make sure you have a full menu of options here is another option. If you're comfortable with the XML representation, (de)serialization, etc... you could also look into using SQLXML. With that said, you should not use this to avoid doing proper database design although this can be totally reasonable for some solutions.

How can I leverage an ORM for a database whose schema is unknown until runtime?

I am trying to leverage ORM given the following requirements:
1) Using .NET Framework (latest Framework is okay)
2) Must be able to use Sybase, Oracle, MSSQL interchangeably
3) The schema is mostly static, BUT there are dynamic parts.
I am somewhat familiar with SubSonic and NHibernate, but not deeply.
I get the nagging feeling that the ORM can do what I want, but I don't know how to leverage it at the moment.
SubSonic probably isn't optimal, since it doesn't currently support Sybase, and writing my own provider for it is beyond my resources and ability right now.
For #3 (above), there are a couple of metadata tables, which describe tables which the vendors can "staple on" to the existing database.
Let's call these MetaTables, and MetaFields.
There is a base static schema, which the ORM (NHibernate ATM) handles nicely.
However, a vendor can add a table to the database (physically) as long as they also add the data to the metadata tables to describe their structure.
What I'd really like is for me to be able to somehow "feed" the ORM with that metadata (in a way that it understands) and have it at that point allow me to manipulate the data.
My primary goal is to reduce the amount of generic SQL statement building I have to do on these dynamic tables.
I'd also like to avoid having to worry about the differences in SQL being sent to Sybase,Oracle, or MSSQL.
My primary problem is that I don't have a way to let ORM know about the dynamic tables until runtime, when I'll have access to the metadata
Edit: An example of the usage might be like the one outlined here:
IDataReader rdr=new Query("DynamicTable1").WHERE("ArbitraryId",2).ExecuteReader();
(However, it doesn't look like SubSonic will work, as there is no Sybase provider (see above)

Acording to this blog you can in fact use NHibernate with dynamic mapping. It takes a bit of tweaking though...

We did some of the using NHibernate, however we stopped the project since it didn't provide us with the ROI we wanted. We ended up writing our own ORM/SQL layer which worked very well (worked since I no longer work there, I'm guessing it still works).
Our system used a open source project to generate the SQL (don't remember the name any more) and we built all our queries in our own Xml based language (Query Markup Language - QML). We could then build an xmlDocument with selects, wheres, groups etc. and then send that to the SqlEngine that would turn it into a Sql statement and execute it. We discusse, but never implemented, a cache in all of this. That would've allowed us to cache the Qmls for frequently used queries.

I am a little confused as to how the orm would be used then at runtime? If the ORM would dynamically build something at runtime, how does the runtime code know what the orm did dynamically?
"have it at that point allow me to manipulate the data" - What is manipulating the data?
I may be missing something here and i aplogize if thats the case. (I only have really used bottom up approach with ORM)

IDataReader doesn't map anything to an object you know. So your example should be written using classic query builder.

Have you looked into using the ADO.NET Entity Framework?
MSDN: LINQ to Entities
It allows you to map database tables to an object model in such a manner that you can code without thinking about which database vendor is being used, and without worrying about minor variations made by a DBA to the actual tables. The mapping is kept in configuration files that can be modified when the db tables are modified without requiring a recompile.
Also, using LINQ to Entities, you can build queries in an OO manner, so you aren't writing actual SQL query strings.