We Keep Coding

How to rewrite C# Linq query with state variable into T-SQL

I have a Linq query (against in-memory objects) which uses local dateTimeLast variable to keep state:
IEnumerable<CacheEntry> entries = await db.Caches.OrderBy(e => e.Time).ToListAsync();
DateTime? dateTimeLast = null;
IEnumerable<CacheEntry> progression = entries.Where(e =>
{
bool isProgress = ((dateTimeLast == null) || (dateTimeLast >= e.DateAndTime));
if (isProgress)
dateTimeLast = e.DateAndTime;
return isProgress;
});
var result = progression.ToList();
How can I rewrite that Linq query into plain a T-SQL (SQL Server) query?
I do not know how to translate Where condition with state variable dateTimeLast in T-SQL.
Source table grew a lot in size and loading all into memory is too slow now.
Of course query is very simplified so there would be additional WHERE conditions, like SELECT * FROM Caches WHERE <search_condition> ORDER BY Time, but they are not the issue.
Source table Caches has 2 columns: Time, DateAndTime (they are no related).
For example I was looking at LAG function, but not useful.

You can achieve what you want with some SQL code using a combination of:
A temporary #dateTimeLast variable, as suggested by Tetsuya Yamamoto
A temporary table to store the valid rows
A cursor to iterate the rows in the table
You can use the following code as guidance (NOT tested, please consider it as pseudocode), it assumes that the name of the table is entries and that it has an integer id column:
DECLARE #dateTimeLast DATETIME
DECLARE #isProgress BIT
DECLARE #id INT
DECLARE #entryDateTime DATETIME
SELECT TOP 0 * INTO #temp FROM entries
DECLARE the_cursor CURSOR FOR SELECT Id, DateAndTime FROM entries ORDER BY Time
OPEN the_cursor
FETCH NEXT FROM the_cursor INTO #id, #entryDateTime
WHILE ##FETCH_STATUS = 0
BEGIN
SELECT #isProgress = (#dateTimeLast IS NULL) OR (#dateTimeLast >= #entryDateTime)
IF #isProgress
BEGIN
INSERT INTO #temp (SELECT * FROM entries WHERE id = #id)
#dateTimeLast = #entryDateTime
END
FETCH NEXT FROM the_cursor INTO #id, #entryDateTime
END
CLOSE the_cursor
DEALLOCATE the_cursor
SELECT * FROM #temp
Another option is to create the temporary table to store just the ids of the rows, and in the end do something like SELECT * FROM entries WHERE id IN (SELECT id FROM #temp)

Can you only parameterize the where clause?

To prevent sql injection, I am trying to use parameterized queries. But it is not clear to me whether I should only paramterize the where clause or other parts of the query. For example, I am trying to improve the following query:
string strQ = #";WITH lstTable as (
SELECT ROW_NUMBER() OVER(ORDER BY " + sort + #") AS RowNum, *
FROM (
SELECT *
FROM SystemMessage
WHERE Deleted = 0 ";`
This query is being used in grid and based on user's selection, it will sort by the column name. Do I needed paramaterize 'sort' in this scenario?

You are correct that you can't use parameters directly for this use case. However the fact if you need to use parameters or not depends on how sort gets populated.
If it is a list of column names that are hard coded and the user just picks which indexes in the list are going to be chosen you don't need to worry about parameterizing, the user does not have direct input in the query so they can't inject code in to it.
If the user is providing the column names directly you must clean up the user input before passing it in, a way to do that is use the sql function QUOTENAME to clean up the input.
string strQ = #"
declare #query nvarchar(max)
set #query = ';WITH lstTable as (
SELECT ROW_NUMBER() OVER(ORDER BY' + QUOTENAME(#sortColumn) + ') AS RowNum, *
FROM (
SELECT *
FROM SystemMessage
WHERE Deleted = 0 '
exec sp_executesql #query";
What that will do is whatever string you pass in to #sortColumn it will properly wrap [ ] around that string value. It then uses that excaped value in the dynamicly generated string and runs that with sp_executesql.
One important note, this example only works with a single column name currently, you would need a QUOTENAME and a new parameter per column you wanted to add in to the query. If you did try passing in FirstName, LastName it would become
;WITH lstTable as (
SELECT ROW_NUMBER() OVER(ORDER BY [FirstName, LastName]) AS RowNum, *
FROM (
SELECT *
FROM SystemMessage
WHERE Deleted = 0
when executed so it would attempt to find a column named "[FirstName, LastName]" and would fail.

Selecting data using varyng source column

I am wondering if there is a way to select "like".
For example i have columns h1, h2, h3 i want my select to pick which column to grab based on user input.
Select like "user var will be 1,2 or 3" from table
Can you have a varying select or will i have to do
Select * from table and then filter my data table?
Another though I had was through having it add the h onto the users var in C#
Select 'h" + var + " ' from table

You could build the query on the fly:
int col = Convert.ToInt32(request["usercolumn"]);
string q = "SELECT h" + col + " FROM table";
My c# is a bit weak, but the gist here is take the user input, make sure it's an integer and build your query using the user input.

SELECT * is typically bad practice. If you did something like the data table filtering I would specify the possible columns explicitly. I would also look at using dynamic SQL. Dynamically choose column in SQL query

Robbert code will work but if use will enter greater than 3 then that will give error, better to use case like this
DECLARE #col int = 1
select case #col when 1 then h1
When 2 then h2
Else h3 END
From table

INSERT with SELECT DISTINCT and another field included?

I have a products table and each product have a category field (this is not up to me to change and I need to read the data from it as it is), from the product table I wanted to update a category table with unique categories only (I also use a IFNULL to deal with empty categories), this is what I have so far that works:
string query = "INSERT INTO categories (name)
SELECT DISTINCT(IFNULL(category, \"Uncategorized\")) AS category_name
FROM products
WHERE NOT EXISTS (
SELECT name FROM categories
WHERE name = category_name)";
With the above query I am able to update the categories with unique category fields and every time I need to run it, it will add only categories that does not exist to the table.
But besides the above I also need to include a profile id with each category inserted something like this, which does not work:
string query = "INSERT INTO categories (profile_id, name)
SELECT " + profileId.ToString() + ",
DISTINCT(IFNULL(category, \"Uncategorized\")) AS category_name
FROM products
WHERE NOT EXISTS (
SELECT name FROM categories
WHERE name = category_name
AND profile_id = #profileId)";
Is there a way I could acomplish this with a query ?
The above gives me a message:
SQLite error near "DISTINCT": syntax error
SqliteMan error:
Query Error: near "DISTINCT": syntax error Unable to execute statement
Let me know if I havent been clear and what should I be more clear about.

You are using the DISTINCT keyword incorrectly: it is not a function. Try instead:
"INSERT INTO categories (profile_id, name)
SELECT DISTINCT '"+profileId.ToString()+"', IFNULL(category, 'Uncategorized')
FROM products
WHERE category_name NOT IN (
SELECT name FROM categories WHERE profile_id = "'+profileId.ToString()+"'
)
"
One assumes that profileId.ToString() is guaranteed to be safe from SQL injection, or else you would be escaping it/passing it as a parameter to a prepared statement?

How to optimize SQL query?

I have 2 tables ('keys' is consists of about 6 fields, 'stats' is consists of about 65 fields).
I want to insert rows in both of tables without dublication of phrase text. I use something like this:
UPDATE Keys SET CommandType = 'ADDED', CommandCode = #CommandCode WHERE
KeyText = #KeyText AND Tab_ID = #TabID AND CommandType = 'DELETED';
INSERT INTO Keys (IsChecked, KeyText, AddDateTime, Tab_ID, KeySource_ID, CommandCode, CommandType)
SELECT 0, #KeyText, datetime(), #TabID, #KeySourceID, #CommandCode, 'ADDED'
WHERE NOT EXISTS (SELECT 1 FROM Keys WHERE Tab_ID = #TabID AND KeyText = #KeyText);
INSERT INTO Statistics (Key_ID)
SELECT ID FROM Keys WHERE KeyText = #KeyText AND Tab_ID = #TabID AND (CommandType IS NULL OR CommandType <> 'DELETED') AND
NOT EXISTS (SELECT 1 FROM Statistics WHERE Key_ID = (SELECT ID FROM Keys WHERE KeyText = #KeyText AND Tab_ID = #TabID AND (CommandType IS NULL OR CommandType <> 'DELETED') LIMIT 1));
How can I optimize it? I create indexes for all used in this query fields. Maybe you can recommend me some solution?
Thanks for help and sorry for my bad english.

Creating indices slows down insert and update queries because the index must be updated along with the data. To optimize your particular insert statements, get rid of any indices you don't need for your typical select statements. Then work on simplifying those "and not exists" clauses. Those are the only source of any performance gains you're going to get. Try creating indices to speed that subquery up once you've simplified it.

You can combine an insert/update statement into a single statement with a MERGE statement.
If you want to copy modifications of keys into statistics, you can use an OUTPUT statement.
You'd have to add your indexes to the question in order to be able to comment on their effectiveness, but basically you want a single index on each table that contains all of the columns in your where clause. You want to use include columns for anything in your select that is not in the where clause.
The best way to optimize is to get an estimated/actual query plan and see which parts of the query are slow. In SQL Server this is done from the "query" menu. Basically, look out for anything that says "scan", that means you're missing an index. "seek" is good.
However, a query plan is mostly helpful for fine-tuning. In this case, using a different algorithm (like merge/output) will make a more drastic difference.
In SQL Server, the results would look somewhat like this:
INSERT INTO [Statistics] (ID)
SELECT ID FROM
(
MERGE [Keys] AS TARGET
USING (
SELECT #KeyText AS KeyText, #TabID AS TabId, #CommandCode AS CommandCode, #KeySourceID AS KeySourceID, 'Added' AS CommandType
) AS SOURCE
ON (target.KeyText = source.KeyText AND target.Tab_Id = #TabID)
WHEN MATCHED AND CommandType = 'DELETED' THEN
UPDATE SET Target.CommandType = Source.CommandType, Target.CommandCode = Source.CommandCode
WHEN NOT MATCHED BY TARGET THEN
INSERT (IsChecked, KeyText, AddDateTime, Tab_Id, KeySource_ID, CommandCode, CommandType) VALUES (0, KeyText, getdate(), TabId, KeySourceId, CommandCode, CommandType)
OUTPUT $Action, INSERTED.ID
) AS Changes (Action, ID)
WHERE Changes.Action = 'INSERT'
AND NOT EXISTS (SELECT 1 FROM Statistics b WHERE b.ID = Changes.ID)

The problem was in bad indexes for my tables. I reconstruct it and replace some query parametrs with static content and it works great!