We Keep Coding

Where to do pagination/filtering? In the database or in the code?

I have to write the code for the following method:
public IEnumerable<Product> GetProducts(int pageNumber, int pageSize, string sortKey, string sortDirection, string locale, string filterKey, string filterValue)
The method will be used by a web UI and must support pagination, sorting and filtering. The database (SQL Server 2008) has ~250,000 products. My question is the following: where do I implement the pagination, sorting and filtering logic? Should I do it in a T-SQL stored procedure or in the C# code?
I think that it is better if I do it in T-SQL but I will end up with a very complex query. On the other hand, doing that in C# implies that I have to load the entire list of products, which is also bad...
Any idea what is the best option here? Am I missing an option?

You would definitely want to have the DB do this for you. Moving ~250K records up from the database for each request will be a huge overhead. If you are using LINQ-to-SQL, the Skip and Take methods will do this (here is an example), but I don't know exactly how efficient they are.

I think other (and potentionaly best) option is to use some higher level framework that shield you from complexity of query writing. EntityFramework, NHibernate and LINQ(toSQL) help you a lot. That said database is typically best place to do it in your case.

today itself I implement pagination for my website. I have done with stored procedure though I am using Entity-Framework. I found that executing a complex query is better then fetching all records and doing pagination with code. So do it with stored procedure.
And I see your code line, which you have attached, I have implemented in same way only.

I would definatly do it in a stored procedure something along the lines of :
SELECT * FROM (
SELECT
ROW_NUMBER() OVER (ORDER BY Quantity) AS row, *
FROM Products
) AS a WHERE row BETWEEN 11 AND 20
If you are using linq then the Take and Skip methods will take care of this for you.

Definitely in the DB for preference, if at all possible.
Sometimes you can mix things up a bit such as if you have the results returned from a database function (not a stored procedure, functions can be parts of larger queries in ways that stored procedures cannot), then you can have another function order and paginate, or perhaps have Linq2SQL or similar call for a page of results from said function, producing the correct SQL as needed.
If you can at least get the ordering done in the database, and will usually only want the first few pages (quite often happens in real use), then you can at least have reasonable performance for those cases, as only enough rows to skip to, and then take, the wanted rows need be loaded from the db. You of course still need to test that performance is reasonable in those rare cases where someone really does look for page 1,2312!
Still, that's only a compromise for cases where paging is very difficult indeed, as a rule always page in the DB unless it's either extremely difficult for some reason, or the total number of rows is guaranteed to be low.

Execute query only if it is select statement. How?

string sqlQuery = "unknown";
I need to write a function which receives a sql query as parameter e.g. sqlQuery. I would like to execute it only if it is select statement and return data. In other case, if parameter sqlQuery contains delete, update or truncate, the function should return null.
I wonder if there is way to achieve this without parsing contents of parameter sqlQuery.
I would like to do this using c sharp for oracle queries.
Any tips. Thanks.
Update:
This should work for all kinds of users with all privileges.

Run the query in the context of a user who only has select privileges. Any other type of query will error out.

SET TRANSACTION READ ONLY, then execute the string. If it attempts to modify data, it will generate an ORA-01456 error. You can trap this and return whatever you want.

If you really have to work with a constructed string that will operate on the database, you should use the DBMS_ASSERT database package to make sure you have a pure query that's not subject to SQL injection. There's a nice paper on the Oracle site about that here.
The basics are:
only give the minimum privileges necessary, for example only giving the user "select" as described in an earlier reply. And then only on the minimum necessary set of tables. Views are really helpful here in limiting access.
Use bind variables where that's possible.
If you can't use bind variables then check the purity of your statement using DBMS_ASSERT

You can probably search the string for keywords like "update", "delete", "truncate" and all the other ways you can do ddl or dml on the table, but it is very error-prone. You have to eliminate strings in the query which might have these keywords and there are a lot of keywords that you have to take into account.
If your requirement is to return null, Why not give just the select privilege on the necessary objects and return null if you encounter the Insufficient Privileges error?
http://download.oracle.com/docs/cd/E11882_01/server.112/e17069/strms_trapply.htm#STRMS1065

I would not allow the client to specify a SQL select string. Too many possible attack vectors.
Have you considered using Linq? The caller could pass a Func<T, bool> that could be passed to a Where clause. Since Linq will generate the select statement for you, there's no possibility of a non-select statement occuring.

Bear in mind a SELECT column FROM table FOR UPDATE will still take an exclusive lock on every row on that table. And it only needs SELECT privileges (none of INSERT, UPDATE or DELETE are required).

You can use ADO.NET SqlCommand http://msdn.microsoft.com/en-us/library/system.data.sqlclient.sqlcommand.aspx. It has methods ExecuteReader for a select type query and ExecuteNonQuery for other sql expression, you jest set the CommandText string attribute. If I'm right it throws exception if the query is not a select in ExecuteReader but you must check it.

string sqlQuery = "("+evil_sql+")";
Only a subquery can start with a parentheses. This will stop DML, DDL, and the FOR UPDATE issue that Gary mentioned. You still have to execute everything, just catch all the errors. I've done this on a public-facing website without any issues.
Even if your user is not directly granted anything you'll need to check for unnecessary PUBLIC grants. And of course keep your system patched. There have been exploits in functions that can be called in a SELECT.

SQL column true or false - use a view?

Simple question, I have an application and dependant on whether you are in the administration system or on the public website I want to show different results.
Example: in the database for a news story in the administration I may set the column value 'showonsite' to false. So I would like this to show in the administration panel only and not on the live site.
My question is, because I require the same information just with the only one column change, live site to only show true values and administration to show both. What is the most effective way of achieving this without copy paste of code?
Thanks

Please see the second query.
I may be missing something. For non-adminstrative users, could you simply not SELECT news stories with the showonsite column set to false?
For example:
SELECT
*
FROM
dbo.NewsStory
WHERE
showOnSite = 1
Edit:
Oh! I see. You're talking about copying and pasting the query. It's late. >.<
If you have a stored procedure, you can pass a value to indicate whether the given user is an administrator:
SELECT
*
FROM
dbo.NewsStory
WHERE
showOnSite = 1
OR #isAdmin = 1

Depends on where you write your code - if you use stored procedures, then just a parameter like #liveonly should be sufficient. Sameway if you use an inline sql query directly in a C# method, then a parameter live_only should be good enough, to determine whether the extra where condition will be added or not.
For eg. in the SP
SELECT *
FROM news
WHERE ((#live_only=true AND showonsite=true) or (#live_only=false))
This way it can be called with #live_only = false for admin panel, but with value true for the actual site.

No, I wouldn't use a view for that, an I wouldn't copy any code. It sounds like this is just a conditional filter on your queries.
You don't make it clear how you are doing the data-access, but this might mean aasdij an extra line of TSQL, adding adding a so parameter, adding an extra LINQ .Where clause, etc

Building SQL Query with SQL Data Adapter? Generating query off of configuration section

I'm not sure if I'm approaching this the correct way, but am very open and appreciative of any suggestions.
I am building a chart off of data in a SQL database.
The chart has 'Age' and 'Percentile' or those are the 2 fields I am most concerned with.
I have a config section in my web.config called 'Percentiles'. This is so the percentiles to be displayed can be identified in this config section. For example:
<percentiles>
<add percentile="P50" visible="true">
<add percentile="P75" visible="true">
<add percentile="P85" visible="false">
<add percentile="P95" visible="true">
...
</percentiles>
I will parse the list of percentiles and would like to either filter my DataSet or generate my query based on that list. The percentiles are stored as columns in the database. So for example I want my query to look like this based on that configuration
SELECT P50, P75, P95 FROM MyData
Now I suppose I could just do SELECT * and remove those columns from my DataSet, or just not add them as 'Series' to my chart. However, I am a novice developer looking to improve the way I do things. In my little experience it seems grabbing all of the data (albeit a small amount) is not a good practice.
So, this is where I arrive at the SQLDataAdapter. Is it possible to build my query via code to only include the columns I want? The psuedocode I am wondering if is possible is something like this:
foreach(Percentile p in PercentileConfiguration)
{
myDataAdapter.Select.Columns.Add(p.Percentile);
}
Is this a good approach? Any and all suggestions very appreciated! Thank you!

I'm confused. At first glance, this seems to be a question purely about how to query a database. And the answer to that is that you construct a SQL command to do that. I can't understand why you are storing database field names in the web.config. If you want a query to look like "SELECT P50, P75, P95 FROM MyData", then make it so.

SELECT * .... is going to slightly slower than SELECT P50, P75, etc. But only marginally. If your only running the query once ( as opposed to multiple times in a loop, or something), and your table doesnt have a massive amount of columns, or columns with huge data, then the performance hit is going to be so nominal that the effort and maintenance in selecting specific columns will probably not be worth it. Someone once told me that if you have to choice between code that is readable(maintainable) and code that will run a millisecond faster go with the readable code.
That said, if your going to do it, you just need to construct the sql text and then pass that to your adapter.
Something like:
StringBuilder columns = new StringBuilder();
foreach(Percentile p in PercentileConfiguration)
{
if(columns.ToString().Length > 0) columns.Append(", ");
columns.Append(p.Percentile);
}
string sql = string.Format("SELECT {0} FROM myTable", columns.ToString());
SqlDataAdapter da = new SqlDataAdapter(sql, connectionString);
...
...
...

Regex to Extract Update Columns from a Sql Statement

I need a Regex Statement (run in c#) that will take a string containing a Sql Update statement as input, and will return a list of columns to be updated. It should be able to handle columns surrounded by brackets or not.
// Example Sql Statement
Update Employees
Set FirstName = 'Jim', [LastName] = 'Smith', CodeNum = codes.Num
From Employees as em
Join CodeNumbers as codes on codes.EmployeeID = em.EmployeeID
In the end I would want to return an IEnumerable or List containing:
FirstName
LastName
CodeNum
Anyone have any good suggestions on implementation?
Update: The sql is user-generated, so I have to parse the Sql as it is given. The purpose of extracting the column names in my case is to validate that the user has permission to update the columns included in the query.

You're doing it backwards. Store the data in a broken out form, with the table to be updated, the column names, and the expressions to generate the new values all separate. From this canonical representation, generate both the SQL (when you need it) and the list of columns being updated (when you need that instead).
If you absolutely must pull the column names out of a SQL statement, I don't think that regular expressions are the correct way to go. For example, in the general case you may need to skip over new value expressions that contain arbitrarily nested parenthesis. You will probably want a full SQL parser. The book Lex & Yacc by Levine, Mason, and Brown has a chapter on parsing SQL.
Response to update:
You are in for a world of hurt. The only way to do what you want is to fully parse the SQL, because you also need to make sure that you don't have any subexpressions that perform unauthorized actions.
I very, very strongly recommend that you come up with another way to do whatever it is that you are doing. Maybe break out the modifiable fields into a separate table and use access controls? Maybe come up with another interface for them to use in specifying what they want done? Whatever it is that you're doing, there is almost certainly a better way to do it. Down that path there be dragons.

Regular expressions cannot do this task, because SQL is not a regular language.
You can do this, but not with a regular expression. You need a full-blown parser.
You can use ANTLR to generate parsers in C#, and there are free grammars available for parsing SQL in ANTLR.
However, I agree with Glomek that allowing user-supplied SQL to be run against your system, even after you have tried to validate that it includes no "unauthorized actions," is foolish. There are too many cases that may circumvent your validation.
Instead, if you have only a single text field, you should define a simplified Domain-Specific Language that permits users to specify only actions that they are authorized to do. From this input, you can build the SQL yourself.

SQL has a complex recursive grammer, and, there will always be some sub select, group by, or literal that will break your regex based parser.
Why don't use a sql parser to achieve what you need, here is an article shows you how to achieve what you need within 3 minutes.