We Keep Coding

Ideas on incorrect ORDER BY results

I want to emphasize that I'm looking for ideas, not necessarily a concrete answer since it's difficult to show what my queries look like, but I don't believe that's needed.
The process looks like this:
Table A keeps filling up, like a bucket - an SQL job keeps calling SP_Proc1 every minute or less and it inserts multiple records into table A.
At the same time a C# process keeps calling another procedure SP_Proc2 every minute or less that does an ordered TOP 5 select from table A and returns the results to the C# method. After C# code finishes processing the results it deletes the selected 5 records from table A.
I bolded the problematic part above. It is necessary that the records from table A be processed 5 at a time in the order specified, but a few times a month SP_Proc2 selects the ordered TOP 5 records in a wrong order even though all the records are present in table A and have correct column values that are used for ordering.
Something to note:
I'm ordering by integers, not varchar.
The C# part is using 1 thread.
Both SP_Proc1 and SP_Proc2 use a transaction and use READ COMMITTED OR READ COMMITTED SNAPSHOT transaction isolation level.
One column that is used for ordering is a computed value, but a very simple one. It just checks if another column in table A is not null and sets the computed column to either 1 or 0.
There's a unique nonclustered index on primary key Id and a clustered index composed of the same columns used for ordering in SP_Proc2.
I'm using SQL Server 2012 (v11.0.3000)
I'm beginning to think that this might be an SQL bug or maybe the records or index in table A get corrupted and then deleted by the C# process and that's why I can't catch it.
Edit:
To clarify, SP_Proc1 commits a big batch of N records to table A at once and SP_Proc2 pulls the records from table A in batches of 5, it orders the records in the table and selects TOP 5 and sometimes a wrong batch is selected, the batch itself is ordered correctly, but a different batch was supposed to be selected according to ORDER BY. I believe Rob Farley might have the right idea.

My guess is that your “out of order TOP 5” is ordered, but that a later five overlaps. Like, one time you get 1231, 1232, 1233, 1234, and 1236, and the next batch is 1235, 1237, and so on.
This can be an issue with locking and blocking. You’ve indicated your processes use transactions, so it wouldn’t surprise me if your 1235 hasn’t been committed yet, but can just be ignored by your snapshot isolation, and your 1236 can get picked up.
It doesn’t sound like there’s a bug here. What I’m describing above is a definite feature of snapshot isolation. If you must have 1235 picked up in an earlier batch than 1236, then don’t use snapshot isolation, and force your table to be locked until each block of inserts is finished.

An alternative suggestion would be to use a table lock (tablock) for the reading and writing procedures.
Though this is expensive, if you desire absolute consistency then this may be the way to go.

Clean memory after getting an element from queue

Suppose we have the following situation:
We have 2-3 tables in database with a huge amount of data (let it be 50-100mln of records) and we want to add 2k of new records. But before adding them we need to check our db on duplicates. So if this 2k contains records which we have in our DB we should ignore them. But to find out whether new record is a duplicate or not we need info from both tables (for example we need to make left join).
The idea of solution is: one task or thread create a suitable data for comparison and pushes data into queue (by batches, not record by record), so our queue(or concurrentQueue) is a global variable. The second thread gets batch from queue and look it through. But there's a problem - memory is growing...
How can I clean memory after I've surfed through the batch?
P.S. If smb has another idea how to optimize this process - please describe it...

This is not the specific answer to the question you are asking, because what you are asking, doesn't really make sense to me.
if you are looking to update specific rows:
INSERT INTO tablename (UniqueKey,columnname1, columnname2, etc...)
VALUES (UniqueKeyValue,value1,value2, etc....)
ON DUPLICATE KEY
UPDATE columnname1=value1, columnname2=value2, etc...
If not, simply ignore/remove the update statement.
This would be darn fast, considering, it would use the unique index of whatever field you want to be unique, and just do an insert or update. No need to validate in a separate table or anything.

Linq to SQL, Update a lot of Data before One Insert

Before insert new value to table, I need change one field in all rows of that table.
What the best way to do this? in c# code, ore use trigger? if C# can you show me the code?
UPD
*NEW VERSION of Question*
Hello. Before insert new value to table, I need change one field in all rows of that table with specific ID( It is FK to another table).
What the best way to do this? in c# code, ore use trigger? if C# can you show me the code?

You should probably consider changing your design this doesn't sound like it will scale well, i would probably do it with a trigger if it is always required, but if not, id use ExecuteCommand.
var ctx = new MyDataContext();
ctx.ExecuteCommand("UPDATE myTable SET foo = 'bar'");

Looking at your comment on Paul's answer, I feel like I should chime in here. We have a few tables where we need to keep a history of each entry in that table. We implement this by creating a separate table for each. For example, we may have a Comment table, and then a CommentArchive table with a foreign key reference to the CommentId in the Comment table.
A trigger on the Comment table ensures that each time certain fields in the Comment table are updated, the "old" version (which is accessible via the deleted table in the trigger) gets pushed to the CommentArchive table. Obviously, this means several CommentArchive entries may exist for each Comment, but if you're only looking for the "active" comments, you just look in the Comment table. And if you need information about the history of a comment, you can easily use LINQ to SQL to jump from the Comment you're interested in to the CommentArchives that reference it.
Because the triggers we use in the above example only insert a single value into the Archive table for each update, they run very quickly and we get good performance. We had issues recently where I tried making the triggers more complex and we started getting dead-locks with as few as 15 concurrent transactions. So the lesson is that you should make these triggers simple, and make them touch as few rows in as few tables as possible.

Check if it is safe to delete a row

I want to be able to check if deleting a row from a table in SQL Server 2008 will fail because of a Foreign Key violation without attempting to delete it.
Basically, I don't want to show the user a delete button if they are not going to be able delete it because the key is used elsewhere.
I am needing this in many places in the application so don't really want to have to write the checks manually to see if it safe to delete the row. Any suggestions on the best way to achieve this?
I am using entity framework for access to the data.

There is no quick and easy way to check this. You could probably build something dynamic using information_schema, but it will undoubtedly be ugly and not very fast.
The absolute best choice is the few lines of custom code that it takes to verify per location.
Another option would be to start a transaction, try the delete. If it fails, then you know. If it succeeds, roll back the transaction, and you know the delete is possible. This is still ugly and is using transactions in a somewhat broken way, but it would work. Make sure cascade deletes aren't on for the table, though.

When you query, do a LEFT JOIN to the child table. Use CanDelete computed value to decide if the button should be shown. The COUNT here removed duplicates if you more than 1 child row per parent row.
SELECT
Col1, Col2, Col3, ...,
CASE C.Existence WHEN 0 THEN 1 ELSE 0 END AS CanDelete
FROM
ParentTable P
LEFT JOIN
(
SELECT COUNT(*) AS Existence, FKColumn
FROM Childtable GROUP BY FKColumn
) C ON P.FKColumn = C.FKColumn
WHERE
P.Col = ...
Another way might be
SIGN(C.Existence) AS HasChildRows

I've done this sort of thing in prior applications. I created a function named something like TryDelete(). Inside the method, I attempted to delete the desired row. If I got a FK exception, I caught it and returned false. In either case, true or false, I encapsulated the delete in a transaction and then rolled it back.

You could add in a partial class of your entity a method that would check if the referenced objects exist.
For example, lets say you have Entity1 which has collections of Entity2. Basically, in each of the entity partial classes you'd write a property IsReferenced that would:
For Entity1 return true if Entity1 has any item in Entity2
For Entity2 return ture if there's a reference to Entity1
As you're guessing, you'll need to make sure that you include referenced values always in your fetch, or, if you're working attached to context, you could use .Load() in IsReferenced to fetch entities before checking. It is an overhead, it just depends if you're willing to 'pay' for it.
Then, you can show/hide the 'delete' button based on that element property wherever needed thus avoiding having to repeat the checks each time.

I think you have 2 possible choices here.Since you cannot garantee that all relations will be mapped in your OM, you would have to check it on the database.
You can either try an actual delting inside a transaction that is rolled back afterwards, but this would also work if you have to contraint configured with cascading deletes...
Another way would be extracting all contraints from the sysobjects table, and verify that each table has no records. But that would require some dynamic SQL, which can also get quite messy.

if you're at the database level I would join all the tables where a conflict could exist.
any records that return can not be deleted which means the remaining set can be.

Assuming that the database is used by multiple users (which the vast majority are) - there's going to be a window of opportunity between "checking" the delete is possible, and the user possibly deciding to delete the row, during which someone else might perform some activity that negates the result of the test.
This means that you might display the Delete button, but by the time you attempt the delete, it's no longer possible. Also, you might not display a Delete button, but by the time the user has decided they want to delete the row (but can't find the button), they should be allowed to.
There's no way to avoid these kind of races. I'd just let people attempt the delete if they want to, but be prepared to deal with failures due to Foreign Keys.

How Do SQL Transactions Work?

I have not been working in SQL too long, but I thought I understood that by wrapping SQL statements inside a transaction, all the statements completed, or none of them did. Here is my problem. I have an order object that has a lineitem collection. The line items are related on order.OrderId. I have verified that all the Ids are set and are correct but when I try to save (insert) the order I am getting The INSERT statement conflicted with the FOREIGN KEY constraint "FK_OrderItemDetail_Order". The conflict occurred in database "MyData", table "dbo.Order", column 'OrderId'.
psuedo code:
create a transaction
transaction.Begin()
Insert order
Insert order.LineItems <-- error occurs here
transaction.Commit
actual code:
...
entity.Validate();
if (entity.IsValid)
{
SetChangedProperties(entity);
entity.Install.NagsInstallHours = entity.TotalNagsHours;
foreach (OrderItemDetail orderItemDetail in entity.OrderItemDetailCollection)
{
SetChangedOrderItemDetailProperties(orderItemDetail);
}
ValidateRequiredProperties(entity);
TransactionManager transactionManager = DataRepository.Provider.CreateTransaction();
EntityState originalEntityState = entity.EntityState;
try
{
entity.OrderVehicle.OrderId = entity.OrderId;
entity.Install.OrderId = entity.OrderId;
transactionManager.BeginTransaction();
SaveInsuranceInformation(transactionManager, entity);
DataRepository.OrderProvider.Save(transactionManager, entity);
DataRepository.OrderItemDetailProvider.Save(transactionManager, entity.OrderItemDetailCollection); if (!entity.OrderVehicle.IsEmpty)
{
DataRepository.OrderVehicleProvider.Save(transactionManager, entity.OrderVehicle);
}
transactionManager.Commit();
}
catch
{
if (transactionManager.IsOpen)
{
transactionManager.Rollback();
}
entity.EntityState = originalEntityState;
}
}
...
Someone suggested I need to use two transactions, one for the order, and one for the line items, but I am reasonably sure that is wrong. But I've been fighting this for over a day now and I need to resolve it so I can move on even if that means using a bad work around. Am I maybe just doing something stupid?

I noticed that you said you were using NetTiers for your code generation.
I've used NetTiers myself and have found that if you delete your foreign key constraint from your table, add it back to the same table and then run the build scripts for NetTiers again after making your changes in the database might help reset the data access layer. I've tried this on occasion with positive results.
Good luck with your issue.

Without seeing your code, it is hard to say what the problem is. It could be any number of things, but look at these:
This is obvious, but your two insert commands are on the same connection (and the connection stays open the whole time) that owns the transaction right?
Are you retrieving your ID related to the constraint after the first insert and writing it back into the data for second insert before executing the command?
The constraint could be set up wrong in the DB.
You definitely do not want to use two transactions.

Looks like your insert statement for the lineItems is not correctly setting the value for the order .. this should be a result of the Insert order step. Have you looked (and tested) the individual SQL statements?
I do not think your problem has anything to do with transaction control.

I have no experience with this, but it looks like you might have specified a key value that is not available in the parent table. Sorry, but I cannot help you more than this.

The problem is how you handle the error. When an error occurs, a transaction is not automatically rolled back. You can certainly (and probably should) choose to do that, but depending on your app or where you are you may still want to commit it. And in this case, that's exactly what you're doing. You need to wrap some error handling code around there to rollback your code when the error occurs.

The error looks like that the LineItems are not being given the proper FK OrderId that was autogenerated by the the insert of the Order to the Order Table. You say you have checked the Ids, Have you checked the FKs in the order details as well ?