We have a couple of SQL queries as strings:
public class Query
{
public static string CreditTransferId(string expectedValue, string table, int statusId, int messageTypeId, int destination103, int destination202, string StoreStatus202Id)
{
return $"SELECT top 1 Id from {table} where MessageId = '{expectedValue}' and FlowId=3 and StatusId={statusId} and MessageTypeId={messageTypeId} and " +
$" Destination103={destination103} and Destination202={destination202} and StoreStatus103Id is null and StoreStatus202Id {StoreStatus202Id}";
}
}
We have them returned as strings from methods inside the Query class. We want to refactor the code, since we have a method with more than 3 parameters which is pretty hard to use.
How would you go about this? What's the cleanest way to organize SQL queries which need a lot of parameters?
Dynamic SQL is a very bad thing for a start as these are open to SQL injection, you should use parameterise queries and return a string.
"eg: SELECT top 1 Id from [Table] where [MessageId] = #messageId" etc
So you dont need to pass in any values, you would add these to your list of SqlParamater's
The table name is probably pointless as this is related to the sql, so probably just add that into the sql string
This doesn't really need an extra class, just create the sql variable where you call it, so it right there if you need it?
..or use Stored Procedures
..or use Entity Framework
EF is great and you have to decide if that's what you want. There are some cases where it is not suitable. Of u decide to stick with plain text queries how about dividing queries into parts:
FromBuilder
JoinBuilder
GroupBuilder
Condition builder etc
ex.:
return #"
"+new TableIdSelectorBuilder(table).Get() +#"
"+new FromBuilder().Get(table) +#"
"+new TableConditionByIdBuilder(table).Get(I'd)+#"
";
EDIT:
Stored procedures allow to change queries without publishing new app version but is a bit pain in the ass to work on a living organism. At least sometimes.
Hopefully this helps you a bit. I was figuring this out a long time ago too.
Use nameOf instead of hardcoded column names
One advice that I have is: Try to use nameOf() for column names (in case your domain model matches exactly your database structure).
Refactoring becomes a lot easier as well.
Create a query builder
I created a class which allows me to create simple to medium complex queries (Select, Insert, Update, Joins, Deletes, Limit, Offset, ...). For most of the queries that I write I can use this query builder to get what I need and the code will look something like this:
var query = new QueryBuilder()
.Update(nameof(MyTable))
.Set(nameof(MyTable.Name), entity.Name)
.Set(nameof(MyTable.Firstname), entity.Firstname)
.WhereEquals(nameof(MyTable.Id), entity.Id)
.Build();
Unfortunately, I haven't found a good QueryBuilder out there yet. Therefore I created my own.
Use string literals
Microsoft Documentation
Another solution that I recently encountered is by doing the following:
var query = $$"""
SELECT * FROM {{nameof(MyTable)}
WHERE
XX
""";
This at least allows you to indent correctly and the code will remain clean.
Outsource your queries in separate files
I was also thinking about having bigger queries in separate files and include them in the Assembly. When you want to execute the query, you load the query from the file and run it like this.
You might also want to cache that query then, to not always load it from the file.
Final words
Personally, I try to use the QueryBuilder as often as possible. It creates clean code which looks somewhat like LINQ
Backup approach for me is to use the string literals as showed above
Trying to create separate files for queries, I have never tried yet, because there was no use for myself
I would also avoid using Stored Procedures, because they are pain to write and also hard to debug in my opinion
The cleanest option would be to use Entity Framework, but if you want to use Micro-ORMs like Dapper, try one of the solutions above
Related
Is it possible to generate a raw SQL statement for insert/update operations using nHibernate without actually executing them? Assuming of course that everything (mappings, connectionStrings, etc.) is properly configured?
The closest thing I've found is to call:
Session.SessionFactory.GetClassMetadata(typeof(Client))
Which returns an object of type SingleTableEntityPersister containing SQLIdentityInsertString, that looks like this:
INSERT INTO Client (FirstName, LastName) values (?, ?)
But it would still require me to bind all of the properties manually, and on top of that SQLIdentityInsertString is a protected property. Are there any proper ways of doing that?
Okay, the closest thing I've found is to construct your own sql query with a string builder. First you need to extract your class metadata:
var metaData = Session.SessionFactory.GetClassMetadata(typeof(Client)) as SingleTableEntityPersister;
Then you can retrieve other information, such as:
var propertyNames = metaData.PropertyNames;
var tableName = metaData.TableName;
var firstPropertyValue = metaData.GetPropertyValue(client, propertyNames[0], EntityMode.Poco);
Once you have that information you can construct your own query manually. Not exactly the solution I wanted, but it's as close as it gets, I think.
However, one thing to note is that Session.CreateSQLQuery(string) method is currently bugged, and as a result SetParameter method doesn't work with more than 10 named parameters. There already seems to be a bug report created for this on NHbiernate's Jira.
I am using C# MVC4 with Linq.
I have used dependency injection for my project which resulted in me having a separate Model's project along with a separate Repository project (and one for testing ect). All this no problem.
I moved my queries out of the controllers (old style) and into the repository (new DI style), and injected them. It works fine.
I have a standard linq query (pick any example, they are basic enough), which returns a set of items from the database as normal. No problems here either.
My problem is, that I want to implement paging, and I taught it would be simple enough to. Here is my steps:
Take in the results of the linq query from the repository (injected into the controller) store it in a var. It looks something like:
var results = _someInjectedCode.GetListById(SomeId);
Before, I was able to do something simple like:
results.Count()
results.Skip(SomeNum).Take(SomeOtherNum)
But now that I want paging, I need to do my Skip Take something like this:
var results = from xyz in _someInjectedCode.GetListById(SomeId).SomeId).Skip(SomeNum).Take(SomeOtherNum)
select new[] {a,id, a.fName, a.lName .....}
The problem with this is that I no longer have access to the total count of items before the list was shortened to the Pre Skip...Take state unless I do two queries which means hitting the DB twice.
What is the best way to resolve this issue.
I just do it like this:
var result = (from n in mycollection
where n.someprop == "some value"
select n).ToList();
var count = result.Count;
There are probably other ways, but this is the simplest that I know of.
Thinking about it from a SQL point of view, I can't think of a way in a single normal query to retrieve both the total count and a subset of the data, so I don't think you will be able to do it in LINQ either.
To avoid creating two separate commands, only thing I can think of is a stored proc that returns two tables (one with just the count, the other with your subset of results). It would still execute two queries, but in a single connection. You'd lose your LINQ though. So if you want to keep your LINQ query, you might be stuck with making two separate calls.
The other way is to retrieve the entire unpaged resultset into memory, then run your Take and Skip against the array, but this is pretty wasteful and probably worse than two calls.
You can either add additional parameters to your repository interface/class which will provide paging parameters and return count alongside your result or modify your interfaces to return IQueryable and then apply count and then skip/take before query is compiled and sent for execution.
I need to do a query on my database that might be something like this where there could realistically be 100 or more search terms.
public IQueryable<Address> GetAddressesWithTown(string[] towns)
{
IQueryable<Address> addressQuery = DbContext.Addresses;
addressQuery.Where( x => towns.Any( y=> x.Town == y ) );
return addressQuery;
}
However when it contains more than about 15 terms it throws and exception on execution because the SQL generated is too long.
Can this kind of query be done through Entity Framework?
What other options are there available to complete a query like this?
Sorry, are we talking about THIS EXACT SQL?
In that case it is a very simple "open your eyes thing".
There is a way (contains) to map that string into an IN Clause, that results in ONE sql condition (town in ('','',''))
Let me see whether I get this right:
addressQuery.Where( x => towns.Any( y=> x.Town == y ) );
should be
addressQuery.Where ( x => towns.Contains (x.Town)
The resulting SQL will be a LOT smaller. 100 items is still taxing it - I would dare saying you may have a db or app design issue here and that requires a business side analysis, I have not me this requirement in 20 years I work with databases.
This looks like a scenario where you'd want to use the PredicateBuilder as this will help you create an Or based predicate and construct your dynamic lambda expression.
This is part of a library called LinqKit by Joseph Albahari who created LinqPad.
public IQueryable<Address> GetAddressesWithTown(string[] towns)
{
var predicate = PredicateBuilder.False<Address>();
foreach (string town in towns)
{
string temp = town;
predicate = predicate.Or (p => p.Town.Equals(temp));
}
return DbContext.Addresses.Where (predicate);
}
You've broadly got two options:
You can replace .Any with a .Contains alternative.
You can use plain SQL with table-valued-parameters.
Using .Contains is easier to implement and will help performance because it translated to an inline sql IN clause; so 100 towns shouldn't be a problem. However, it also means that the exact sql depends on the exact number of towns: you're forcing sql-server to recompile the query for each number of towns. These recompilations can be expensive when the query is complex; and they can evict other query plans from the cache as well.
Using table-valued-parameters is the more general solution, but it's more work to implement, particularly because it means you'll need to write the SQL query yourself and cannot rely on the entity framework. (Using ObjectContext.Translate you can still unpack the query results into strongly-typed objects, despite writing sql). Unfortunately, you cannot use the entity framework yet to pass a lot of data to sql server efficiently. The entity framework doesn't support table-valued-parameters, nor temporary tables (it's a commonly requested feature, however).
A bit of TVP sql would look like this select ... from ... join #townTableArg townArg on townArg.town = address.town or select ... from ... where address.town in (select town from #townTableArg).
You probably can work around the EF restriction, but it's not going to be fast and will probably be tricky. A workaround would be to insert your values into some intermediate table, then join with that - that's still 100 inserts, but those are separate statements. If a future version of EF supports batch CUD statements, this might actually work reasonably.
Almost equivalent to table-valued paramters would be to bulk-insert into a temporary table and join with that in your query. Mostly that just means you're table name will start with '#' rather than '#' :-). The temp table has a little more overhead, but you can put indexes on it and in some cases that means the subsequent query will be much faster (for really huge data-quantities).
Unfortunately, using either temporary tables or bulk insert from C# is a hassle. The simplest solution here is to make a DataTable; this can be passed to either. However, datatables are relatively slow; the over might be relevant once you start adding millions of rows. The fastest (general) solution is to implement a custom IDataReader, almost as fast is an IEnumerable<SqlDataRecord>.
By the way, to use a table-valued-parameter, the shape ("type") of the table parameter needs to be declared on the server; if you use a temporary table you'll need to create it too.
Some pointers to get you started:
http://lennilobel.wordpress.com/2009/07/29/sql-server-2008-table-valued-parameters-and-c-custom-iterators-a-match-made-in-heaven/
SqlBulkCopy from a List<>
So using EF4, I'm running a search query that ends with this common function:
query = query.Skip(5).Take(10);
Those records in the database have a column called ImpressionCount, which I intend to use to count the number of times that each record displayed on a page of search results.
What's the most efficient way to do this? Off the top of my head, I'm just going to look at the result set, get a list of ID's and then hit the database again using ADO.NET to do something like:
UPDATE TableName SET ImpressionCount = ImpressionCount + 1 WHERE Id IN (1,2,3,4,5,6,7,8,9,10)
Seems simple enough, just wondering if there's a more .NET 4 / Linq-ish way to do this that I'm not thinking of. One that doesn't involve another hit to the database would be nice too. :)
EDIT: So I'm leaning towards IAbstract's response as the answer since it doesn't appear there's a "built in" way to do this. I didn't think there was but it never hurts to ask. However, the only other question I think I want to throw out there is: is it possible to write a SQL trigger that could only operate on this particular query? I don't want ImpressionCount to update on EVERY select statement for the record (for example, when someone goes to view the detail page, that's not an impression -- if an admin edits the record in the back end, that's not an impression)...possible using LINQ or no?
SQL Server would somehow need to be able to identify that the query was generated by that Linq command, not sure if that's possible or not. This site is expecting relatively heavy traffic so I'm just trying to optimize where possible, but if it's overkill, I might just go ahead and hit the database again one time for each page of results.
In my opinion, it is better to go ahead and run with the SQL command as you have it. Just because we have LINQ does not mean it is always the best choice. Your statement gives you a one-call process to update impression counts and should be fairly quick.
You can technically use a .Select to modify each element in a returned result, but it's not idiomatic C#/linq, so you're better off using a foreach loop.
Example:
var query = query.ToList().Select(x => { x.ImpressionCount++; return x; });
As IAbstract said, be careful of performance issues. Using the example above, or a foreach will execute 10 updates. Your one update statement is better.
I know Linq2NHibernate has this same issue - trying to stick with Linq just isn't any good for updates (that's why it's called "language integrated query");
Edit:
Actually there's probably no reason why EF4 or NHibernate couldn't parse the select expression, realize it's an update, and translate it into an update statement an execute it, but certainly neither framework will do that. If that were something that could happen, you'd want a new .Update() extension method for IQueryable<T> to explicitly state that you're modifying data. Using .Select() for it is a dirty hack.
... which means there's no reason you couldn't write your own .Update(x => x.ImpressionCount++) extension method for IQueryable<T> that output the SQL you want and call ExecuteStoreCommand, but it would be a lot of work.
It is second nature for me to whip up some elaborate SQL set processing code to solve various domain model questions. However, the trend is not to touch SQL anymore. Is there some pattern reference or conversion tool out there that helps convert the various SQL patterns to Linq syntax?
I would look-up ways to code things like the following code: (this has a sub query):
SELECT * FROM orders X WHERE
(SELECT COUNT(*) FROM orders Y
WHERE Y.totalOrder > X.totalOrder) < 6
(Grab the top five highest total orders with side effects)
Alternatively, how do you know Linq executes as a single statement without using a debugger? I know you need to follow the enumeration, but I would assume just lookup the patterns somewhere.
This is from the MSDN site which is their example of doing a SQL difference. I am probably wrong, but I wouldn't think this uses set processing on the server (I think it pulls both sets locally then takes the difference, which would be very inefficient). I am probably wrong, and this could be one of the patterns on that reference.
SQL difference example:
var differenceQuery =
(from cust in db.Customers
select cust.Country)
.Except
(from emp in db.Employees
select emp.Country);
Thanks
-- Update:
-- Microsoft's 101 Linq Samples in C# is a closer means of constructing linq in a pattern to produce the SQL you want. I will post more as I find them. I am really looking for a methodology (patterns or a conversion tool) to convert SQL to Linq.
-- Update (sql from Microsoft's difference pattern in Linq):
SELECT DISTINCT [t0].[field] AS [Field_Name]
FROM [left_table] AS [t0]
WHERE NOT (EXISTS(
SELECT NULL AS [EMPTY]
FROM [right_table] AS [t1]
WHERE [t0].[field] = [t1].[field]
))
That's what we wanted, not what I expected. So, that's one pattern to memorize.
If you have hand-written SQL, you can use ExecuteQuery, specifying the type of "row" class as a function template argument:
var myList = DataContext.ExecuteQuery<MyRow>(
"select * from myview");
The "row" class exposes the columns as public properties. For example:
public class MyRow {
public int Id { get; set; }
public string Name { get; set; }
....
}
You can decorate the columns with more information:
public class MyRow {
....
[Column(Storage="NameColumn", DbType="VarChar(50)")]
public string Name { get; set; }
....
}
In my experience linq to sql doesn't generate very good SQL code, and the code it does generate breaks down for large databases. What linq to sql does very well is expose stored procedures to your client. For example:
var result = DataContext.MyProcedure(a,b,c);
This allows you to store SQL in the database, while having the benefits of an easy to use, automatically generated .NET wrapper.
To see the exact SQL that's being used, you can use the SQL Server Profiler tool:
http://msdn.microsoft.com/en-us/library/ms187929.aspx
The Linq-to-Sql Debug Visualizer:
http://weblogs.asp.net/scottgu/archive/2007/07/31/linq-to-sql-debug-visualizer.aspx
Or you can write custom code to log the queries:
http://goneale.wordpress.com/2008/12/31/log-linq-2-sql-query-execution-to-consoledebug-window/
This is why Linq Pad was created in the first place. :) It allows you to easily see what the output is. What the results of the query would be etc. Best is it's free. Maybe not the answer to your question but I am sure it could help you.
If you know exactly the sql you want, then you should use ExecuteQuery.
I can imagine a few ways to translate the query you've shown, but if you're concerned that "Except" might not be translated.
Test it. If it works the way you want then great, otherwise:
Rewrite it with items you know will translate, for example:
db.Customers.Where(c => !db.Employees.Any(e => c.Country == e.Country) );
If you are concerned about the TSQL generated, then I would suggest formalising the queries into stored procedures or UDFs, and accessing them via the data-context. The UDF approach has slightly better metadata and composability (compared to stored procedure) - for example you can add addition Where/Skip/Take etc to a UDF query and have it run at the database (but last time I checked, only LINQ-to-SQL (not Entity Framework) supported UDF usage).
You can also use ExecuteQuery, but there are advantages of letting the database own the fixed queries.
Re finding what TSQL executed... with LINQ-to-SQL you can assign any TextWriter (for example, Console.Out) to DataContext.Log.
I believe the best way is to use stored procedures. In this case you has full control on the SQL.