We Keep Coding

Best practice when querying restored databases

I am developing an application that is differentiating between an existing database and a backup-file (.bak). When generally connecting and retrieving data from SQL Server, I prefer to use Linq, since I find it easier to use this when building queries.
In this case I do not see how I can do this, my procedure is that I construct a string with the SQL query for restoring the database (from the .bak file) and then retrieve whatever data that differs from the current one, with this one.
All queries that I use are strings which I format to add the database-name and schema. The strings that I am using become quite large and I find that it is very cluttering (I like clean code).
The structure of the databases are all the same and do not ever change between the different backups or the real deal. New backups come each day, and have to be checked against, so it isn't really an option to go in and add a new connection-file each time there is a new backup.
Is there any way that I can restore the database, using Linq, and then retrieve the data using Linq queries? Or am I doing it as I should?

You can only use Linq for queries so you can not use it to restore a database.
Also, you can't use different databases in the same Linq query, you can however materialize the results of each query to the database and make a third Linq query based on them, as you can see in this answer.

Using dynamic temporary table

I have the following situation: i need to create some temporary tables to optimize a load problem the recently has ocurred. It seems that LINQ to SQL doesn't work well with temporary table, unless they are mapped on the DBML. I, honestly, still don't understand how scope works on LINQ to SQL. With that in mind i went to define every temporary table on the DBML.
But, as always, things can't be that easy. I can't define on compilation time (which is what linq needs) what name my temporary table will have, because it will be defined when an user logs on the system. To add more: i will have several of these dynamic temporary table, so there's no way i can map it all to the DBML.
When i tried to create my temporary tables through executeCommand, select its results and cast it to strong type (TempTableDefinition). However, when i tried to insert values on this new created temporary table i got a SQLException saying 'Invalid object name #NewTempTable' (this was the same name i used to create the table).
It appears that i will have to use pure old plain ADO.NET to create every temporary table and map it's properties to a strong typed object (i prefer this approach). I really wouldn't like to mix ADO.NET with LINQ, since i just read that it's a bad ideia. Plus, i prefer linq approach of strong type objects to the ADO.NET way.
Resume:
So, do you know or is it even possible to create dynamic temporary tables that linq to sql can work with? I can't define it's name on compilation time, only on execution time. Any tips will be appreciated.

The problem seems to be that L2S by default opens and closes the connection for each logical request. That kills your temp tables.
Either open the connection manually (and close it, of course) or wrap everything in a TransactionScope which integrates with L2S and keeps the connection open.

to optimize a load problem
Linq-2-sql and batch/bulk will anyway not work together. Every insert/update/delete will result in a single statement (ok, one transaction but still). For hardcore performance, avoid Linq-2-sql and once you have your data in, use Linq with all the advantages like strong typing etc.

Why do I need LINQ if I use NHibernate-like ORMs?

I was reading this SO question but still I am not clear about one specific thing.
If I use NHibernate, why do I need LINQ?
The question in my mind becomes more aggravated when I knew that NHibernate also included LINQ support.
LINQ to NHibernate?
WTF!

LINQ is a query language. It allows you to express queries in a way that is not tied in to your persistence layer.
You may be thinking about the LINQ 2 SQL ORM.
Using LINQ in naming the two is causes unfortunate confusions like yours.
nHibernate, EF, LINQ2XML are all LINQ providers - they all allow you to query a data source using the LINQ syntax.

Well, you don't need Linq, you can always do without it, but you might want it.
Linq provides a way to express operations that behave on sets of data that can be queried and where we can then perform other operations based on the state of that data. It's deliberately written so as to be as agnostic as possible whether that data is in-memory collections, XML, database, etc. Ultimately it's always operating on some sort of in-memory object, with some means of converting between in-memory and the ultimate source, though some bindings go further than others in pushing some of the operations down to a different layer. E.g. calling .Count() can end up looking at a Count property, spinning through a collection and keeping a tally, sending a Count(*) query to a database or maybe something else.
ORMs provide a way to have in-memory objects and database rows reflect each other, with changes to one being reflected by changes to the other.
That fits nicely into the "some means of converting" bit above. Hence Linq2SQL, EF and Linq2NHibernate all fulfil both the ORM role and the Linq provider role.
Considering that Linq can work on collections you'd have to be pretty perverse to create an ORM that couldn't support Linq at all (you'd have to design your collections to not implement IEnumerable<T> and hence not work with foreach). More directly supporting it though means you can offer better support. At the very least it should make for more efficient queries. For example if an ORM gave us a means to get a Users object that reflected all rows in a users table, then we would always be able to do:
int uID = (from u in Users where u.Username == "Alice" select u.ID).FirstOrDefault();
Without direct support for Linq by making Users implement IQueryable<User>, then this would become:
SELECT * FROM Users
Followed by:
while(dataReader.Read())
yield return ConstructUser(dataReader);
Followed by:
foreach(var user in Users)
if(user.Username == "Alice")
return user.ID;
return 0;
Actually, it'd be just slightly worse than that. With direct support the SQL query produced would be:
SELECT TOP 1 id FROM Users WHERE username = 'Alice'
Then the C# becomes equivalent to
return dataReader.Read() ? dataReader.GetInt32(0) : 0;
It should be pretty clear how the greater built-in Linq support of a Linq provider should lead to better operation.
Linq is an in-language feature of C# and VB.NET and can also be used by any .NET language though not always with that same in-language syntax. As such, ever .NET developer should know it, and every C# and VB.NET developer should particularly know it (or they don't know C# or VB.NET) and that's the group NHibernate is designed to be used by, so they can depend on not needing to explain a whole bunch of operations by just implementing them the Linq way. Not supporting it in a .NET library that represents queryable data should be considered a lack of completeness at best; the whole point of an ORM is to make manipulating a database as close to non-DB related operations in the programming language in use, as possible. In .NET that means Linq supprt.

First of all LINQ alone is not an ORM. It is a DSL to query the objects irrespective of the source it came from.
So it makes perfect sense that you can use LINQ with Nhibernate too
I believe you misunderstood the LINQ to SQL with plain LINQ.

Common sense?
There is a difference between an ORM like NHibernate and compiler integrated way to express queries which is use full in many more scenarios.
Or: Usage of LINQ (not LINQ to SQL etc. - the langauge, which is what you talk about though I am not sure you meant what you said) means you dont have to deal with Nhibernate special query syntax.
Or: Anyone NOT using LINQ - regardless of NHibernate or not - de qualifies without good explanation.

You don't need it, but you might find it useful. Bear in mind that Linq, as others have said, is not the same thing as Linq to SQL. Where I work, we write our own SQL queries to retrieve data, but it's quite common for us to use Linq to manipulate that data in order to serve a specific need. For instance, you might have a data access method that allows you to retrieve all dogs owned by Dave:
new DogOwnerDal().GetForOwner(id);
If you're only interested in Dave's daschunds for one specific need, and performance isn't that much of an issue, you can use Linq to filter the response for all of Dave's dogs down to the specific data that you need:
new DogOwnerDal().GetForOwner(id).Where(d => d.Breed == DogBreeds.Daschund);
If performance was crucial, you might want to write a specific data access method to retrieve dogs by owner and breed, but in many cases the effort expended to create the new data access method doesn't increase efficiency enough to be worth doing.
In your example, you might want to use NHibernate to retrieve a lump of data, and then use Linq to break that data into lots of individual subsets for some form of processing. It might well be cheaper to get the data once and use Linq to split it up, instead of repeatedly interrogating the database for different mixtures of the same data.

Why write a custom LINQ provider?

What is the benefit of writing a custom LINQ provider over writing a simple class which implements IEnumerable?
For example this quesiton shows Linq2Excel:
var book = new ExcelQueryFactory(#"C:\Users.xls");
var administrators = from x in book.Worksheet<User>()
where x.Role == "Administrator"
select x;
But what is the benefit over the "naive" implementation as IEnumerable?

A Linq provider's purpose is to basically "translate" Linq expression trees (which are built behind the scenes of a query) into the native query language of the data source. In cases where the data is already in memory, you don't need a Linq provider; Linq 2 Objects is fine. However, if you're using Linq to talk to an external data store like a DBMS or a cloud, it's absolutely essential.
The basic premise of any querying structure is that the data source's engine should do as much of the work as possible, and return only the data that is needed by the client. This is because the data source is assumed to know best how to manage the data it stores, and because network transport of data is relatively expensive time-wise, and so should be minimized. Now, in reality, that second part is "return only the data asked for by the client"; the server can't read your program's mind and know what it really needs; it can only give what it's asked for. Here's where an intelligent Linq provider absolutely blows away a "naive" implementation. Using the IQueryable side of Linq, which generates expression trees, a Linq provider can translate the expression tree into, say, a SQL statement that the DBMS will use to return the records the client is asking for in the Linq statement. A naive implementation would require retrieving ALL the records using some broad SQL statement, in order to provide a list of in-memory objects to the client, and then all the work of filtering, grouping, sorting, etc is done by the client.
For example, let's say you were using Linq to get a record from a table in the DB by its primary key. A Linq provider could translate dataSource.Query<MyObject>().Where(x=>x.Id == 1234).FirstOrDefault() into "SELECT TOP 1 * from MyObjectTable WHERE Id = 1234". That returns zero or one records. A "naive" implementation would probably send the server the query "SELECT * FROM MyObjectTable", then use the IEnumerable side of Linq (which works on in-memory classes) to do the filtering. In a statement you expect to produce 0-1 results out of a table with 10 million records, which of these do you think would do the job faster (or even work at all, without running out of memory)?

You don't need to write a LINQ provider if you only want to use the LINQ-to-Objects (i.e. foreach-like) functionality for your purpose, which mostly works against in-memory lists.
You do need to write a LINQ provider if you want to analyse the expression tree of a query in order to translate it to something else, like SQL. The ExcelQueryFactory you mentioned seems to work with an OLEDB-Connection for example. This possibly means that it doesn't need to load the whole excel file into memory when querying its data.

In general performance. If you have some kind of index you can do a query much faster than what is possible on a simple IEnumerable<T>.
Linq-To-Sql is a good example for that. Here you transform the linq statement into another for understood by the SQL server. So the server will do the filtering, ordering,... using the indexes and doesn't need to send the whole table to the client which then does it with linq-to-objects.
But there are simpler cases where it can be useful too:
If you have a tree index over the propery Time then a range query like .Where(x=>(x.Time>=now)&&(x.Time<=tomorrow)) can be optimized a lot, and doesn't need to iterate over every item in the enumerable.

LINQ will provide deferred execution as much as maximum possible to improve the performance.
IEnumurable<> and IQueryable<> will totally provide different program implementations. IQueryable will give native query by building expression tree dynamically which provides good performance indeed then IEnumurable.
http://msdn.microsoft.com/en-us/vcsharp/ff963710.aspx
if we are not sure we may use var keyword and dynamically it will initialize a most suitable type.

A Better DataTable

I have an application that uses DataTables to perform grouping, filtering and aggregation of data. I want to replace datatables with my own data structures so we don't have any unnecessary overhead that we get from using datatables. So my question is if Linq can be used to perform the grouping, filtering and aggregation of my data and if it can is the performance comparable to datatables or should I just hunker down and write my own algorithms to do it?
Thanks
Dan R.

Unless you go for simple classes (POCO etc), your own implementation is likely to have nearly as much overhead as DataTable. Personally, I'd look more at using tools like LINQ-to-SQL, Entity Framework, etc. Then you can use either LINQ-to-Objects against local data, or the provider-specific implementation for complex database queries without pulling all the data to the client.
LINQ-to-Objects can do all the things you mention, but it involves having all the data in memory. If you have non-trivial data, a database is recommended. SQL Server Express Edition would be a good starting point if you look at LINQ-to-SQL or Entity Framework.
Edited re comment:
Regular TSQL commands are fine and dandy, but you ask about the difference... the biggest being that LINQ-to-SQL will provide the entire DAL for you, which is a huge time saver, as well as making it possible to get a lot more compile-time safety. But is also allows you to use the same approach to look at your local objects and your database - for example, the following is valid C# 3.0 (except for [someDataSource], see below):
var qry = from row in [someDataSource]
group row by row.Category into grp
select new {Category = grp.Key, Count = grp.Count(),
TotalValue = grp.Sum(x=>x.Value) };
foreach(var x in qry) {
Console.WriteLine("{0}, {1}, {2}", x.Category, x.Count, x.TotalValue);
}
If [someDataSource] is local data, such as a List<T>, this will execute locally; but if this is from your LINQ-to-SQL data-context, it can build the appropriate TSQL at the database server. This makes it possible to use a single query mechanism in your code (within the bounds of LOLA, of course).

You'd be better off letting your database handle grouping, filtering and aggregation. DataTables are actually relatively good at this sort of thing (their bad reputation seems to come primarily from inappropriate usage), but not as good as an actual database. Moreover, without a lot of work on your part, I would put my money on the DataTable's having better performance than your homegrown data structure.

Why not use a local database like Sqlserver CE or firebird embedded? (or even ms access! :)). Store the data in the local database, do the processing using simple sql queries and pull the data back. Much simpler and likely less overhead, plus you don't have to write all the logic for grouping/aggregates etc. as the database systems already have that logic built in, debugged and working.

Yes, you can use LINQ to do all those things using your custom objects.
And I've noticed a lot of people suggest that you do this type of stuff in the database... but you never indicated where the database was coming from.
If the data is coming from the database then at the very least the filtering should probably happen there, unless you are doing something specialized (e.g. working from a cached set of data). And even then, if you are working with a significant amount of cached data, you might do well to put that data into an embedded database like SQLite, as someone else has already mentioned.