We Keep Coding

Transform (large) XML-files into relational SQL

I've been tasked with the job of importing a set of XML files, transform them and upload them to an SQL database, and then re-transforming them to a different XML-format.
The XML files are rather large, and some of them a little complex, so I'm unsure of the best way to do this. I'd of course like to automate this process somehow - and was actually hoping there'd be some kind of Entity Framework-esque solution to this.
I'm quite new to handling and dealing with XML in .NET, so I don't really know what my options are. I've read about XSLT, but that seems to me, to be a "language" I need to learn first, making it kind of not a solution for me.
Just to set a bit of context, the final solution actually needs to import new/updated versions of the XML on a weekly basis, uploading the new data to sql, and re-exporting as the other XML-format.
If anyone could give me any ideas as to how to proceed, I'd be much obliged.
My first instict was to use something like XSD2DB or XML SPY to first create the database structure, but I don't really see how I'm then supposed to proceed either.
I'm quite blank in fact :)

XSLT is language used by XML processors to transform XML document in one format to XML document in another format. XSLT would be your choice if you don't need to store data in database as well.
All tools like XSD2DB or XML SPY will create some database schema for you but the quality of the schema will be very dependent on quality of XML document and XSD (do you have XSD or are you going to generate it from sample XML?). The generated database will probably not be to much useful for EF.
If you have XSD you can use xsd.exe tool shipped with Visual studio and generate classes representing data of your XML files in .NET code. You will be able to use XmlSerializer to deserialize the XML document into your generated classes. The problem is that some XSD constructs like choice are modeled in .NET code by very ugly way. Another problem can be performance if your XML files are really huge because deserialization must read all data at once. The last problem can be again EF - classes generated by XSD will most probably not be usable as entities and you will not be able to map them.
So either use EF and in such case you will have to analyze XSD and create custom entities and mapping to your own designed database and you will fill your classes either from XmlReader (best performance), XmlDocument or XDocument or use some tool helping you creating classes or database from XML and in such case use direct SQL to work with a database.
Reverse operation will again require custom approach. You will have data represented either by your custom EF entities or by some autogenerated classes and you will have to transform them to a new format. You can again use xsd.exe to get classes for a new format and write a custom .NET code filling new classes from old ones (and use XmlSerializer to persist a new structure to XML) or you can use XmlWriter, XDocument or XmlDocument to build target XML document directly.
Data migration in any form is not easy task with ready to use solution. In case of really huge data processing you can use tools like SQL Server Integration Services where you will interact with XML and SQL directly and process data in batches.

Have a look at SQLXML 4.0. It does exactly what you want (in upload part).

On-Disk database storage, best practices

If this question seems common to you, I apologise, I did a quick search around this site and a few google searches and could not find a satisfying answer.
My question is this;
I have only been a software developer for 3-4 years now. This may seem like a time long enough to answer this question myself however in all my time, I have never had to develop software where the main body of data-storage is not required to be in an on-line database. This time however, my latest development requires only for its data to be stored only to disk.
The actual data itself is light-weight. In-code the main asset will be a class with only a few, string based properties on it which must be persisted. My initial thoughts are on simple serialisation. On application close new assets are simply serialised and stored on disk as a file. I also though maybe for backup purposes (or if it is somehow a better option to a serialised class) an XML file would be appropriate.
I cannot think of any distinct disadvantages of either of these approaches, it is this fact which causes me to ask this question publicly. In my experience, there is rarely a solution to a problem which does not have it's downsides.

Serialization (binary or XML) is appropriate for a small amount of data. The problem with this approach is when you get large amounts of data (that you may need to query).
If you are on a windows platform and in need of a proper database, you can use the embedded database engine that comes with windows - ESENT. It is the backing store of Exchange and RavenDB.
Here are the .NET wrapper libraries for it.
ManagedEsent provides managed access to ESENT, the embeddable database engine native to Windows. ManagedEsent uses the esent.dll that is part of Microsoft Windows so there are no extra unmanaged binaries to download and install.

The most lightweight solution, is of course to use XML and serialization. The main advantage of that is that it is very easy, requiring little code, and is easily editable using a text editor. The other advantage of this is being able to have multiple files, and they will be easy to transfer from PC to PC.
Here is a nice tutorial on XML serialization.
However, if your application is going to be reading, writing, and changing the data a lot, and there is only one source of data, it would be better to use a light-weight database. Many people like SQLite, while I personally prefer Firebird.
See this question for using SQLite with C#, and see here for information for using Firebird with .net.

Another embedded database option is Sql Server Compact Edition. The latest version of this is v4 and it seems to be much improved over previous versions.
It's functionally equivalent to using an XML file, or an access database, or even a plain old text file, in that you don't need to have a Sql Server service running or install anything special on the machine that your application runs on.

I've been using Sqlite in a project and it works very well and it's easy to use too, one thing to keep it mind when using Sqlite though is that it's designed to be used in a single user environment, so if you use it as the database for the backend of a website for instance you're likely to find that it'll struggle under the slightest of load..
Check out this link for the C# wrapper:
http://sqlite.phxsoftware.com/
I also use NHibernate and NHibernate.Linq to interact with the data, you can get a build of both which are compatible here: http://www.dennisdoomen.net/2009/07/nhibernate-210-ga-with-linq-and-fluent.html
NHibernate.Linq allows you to use those nice Linq query syntax on your Sqlite db:
var onePiece = from s in session.Linq() where s.Name == "One Piece" select s;

Is there a method of serialization that is similar to a database

So far all the serialization examples I have found on the web are related to storing arrays or list in a file. With each class of object having to be serialized into their own file such as a ".bin". The root of my problem is that I want to have the information for my product local stored, but I'm so use to working with sql. It's hard for me to visualize how to store information locally. If C# is anything like asp I should be able to connect to an Access database, but that pretty much defeats one of the ideas of serialization which is user non-readability. Is there a serialization method similar to using table and fields or at least allowing you to store all user information in one file?

You could use a ADO.NET DataSet that is serialized and stored locally. It will contain all of the data structures that you're familiar with and allow you to query the data the way you seem to want to and if you serialize it with a Binary Serializer, it will be unreadable to end-users.
Also, you could look at SQLite as an alternative to using DataSets.
SQLite is a software library that
implements a self-contained,
serverless, zero-configuration,
transactional SQL database engine.
SQLite is the most widely deployed SQL
database engine in the world. The
source code for SQLite is in the
public domain.
NHibernate with SQLite is a great combination as well.
Cheers.

Check out NHibernate. That will give you your 'database-like' storage.
If it's human-readability you're after, consider serializing your objects using XML. .Net has decent support for serializing (and deserializing) objects using both XML and binary formats.
The tutorial I used for learning serialization in C# is this CodeProject article.
Update:
I misread one point you made: serialization does not necessarily mean human-readable or not - if you decide to serialize, figure out if you want the data readable or not. Binary serialization is likely to be more compact and less readable.

Best practice for Uploading Excel data in SQL Server using ASP.NET

I am looking for best practice for uploading excel data in Sql server 2000 database through asp.net web application. Excel data will be predefined Format with almost 42 columns and out of 42 10 fields are mandatory and rest are conditional mandatory. i.e. if data exists it should be in defined format. I also need to validate for special character, length, specified format and so on.
After validating, i need to store valid data into sql server table and provide export to excel functionality for invalid data for exporting in same excel format with indicator to identity the invalid cells.
Can any one suggest me to do the same in optimized way.
Thank you...

You can use ADO.NET to read the data in from the spreadsheet, as outlined here.
Read it in to memory and parse all the data as necessary. Store the parsed data into a DataTable, and then you can persist that data in bulk to the database using a couple of possible methods.
The quickest, most efficient way to bulkload data into SQL Server is using SqlBulkCopy. The alternative method is to use an SqlDataAdapter. I recently outlined both approaches, with examples and performance comparisons here.

You can, but you are, AFAIK, not allowed to use Excel COM Interop on a web server. And it is definately not recommended and supported (source).
So you are left with 2 options:
Try to switch to a different format (XML, CSV) or use an Excel XML format, that you can read and write using System.XML or System.XML.Linq.
Find a component that can read and write Excel binary files. There are commercial and open source components available.

FileHelpers for .net is a decent library that will do alot of the processing for you if you are looking for something quick and efficient without having to build a ton of it yourself. They have an example of loading excel files into a sql database like you describe.

Simple, linq-able, file based database

Hey, what's the easiest way to use a file-based database with LINQ in C#? It would be best if I could use it without installing extra components.
EDIT: I want to use it for a file index. Not the whole file system, but the database should be not too slow and not too big.

I'd recommend MS SQL Server Compact Edition. Its embedable, small footprint, good performance and you can use Linq2Sql to query it easily. Also it integrates well with Visual Studio IDE and SQL Management Studio.

Are you opposed to using XML?
That's basically what XML is (or, rather, is a major use of XML), and Linq to XML is very powerful.

The way i've implemented this kind of thing previously is to load a csv file into a C# List structure (couldn't say which is best without information on the data), and use Linq to access data from there.
This may not be the best solution for you, but you have given limited information on what you're looking for.

You will always need to install database drivers to talk to a specific database. The only way to avoid that is to create your own driver to talk to a database, or even create your own database engine.
However, both SQLite and VistaDB have LINQ to SQL now. Other alternatives are to use a text file or XML file for your "database."