I have some code that reads results into a List from a SqlDataReader, parsing them into domain objects as it goes using reader.GetXXX(int ordinal) methods inside a while reader.Read() loop.
This code generally works fine over very large datasets and across a wide range of queries and tables, but one query hangs midway through reading this list of results and eventually times out with a "Wait Operation Timed Out" error.
I've repeated this a lot of times, and it always hangs on the same record (roughly 336k records into a 337k record set).
If I pause execution while it's hanging i can see that it is midway through parsing a record, and it is hanging on a reader.GetXXX call.
I have tried the following:
executing the proc manually in ssms (works fine)
Chunking the calls to the database such that it reads 250k records in a chunk and then requeries to get the rest (it still hung on the same record, but now the record was in the second batch)
Checking the ID for each record before parsing, and skipping the one that it hangs on (the record after that is parsed but it hangs on the next record).
Updating stats (parsing gets about 3000 records further before hanging after this)
I'm tempted to blame the database, but given the query runs without a hitch in SSMS, I'm not convinced. Any help much appreciated!
other stuff that may help:
the proc takes a table valued parameter and joins onto that along with another database table to get its results
this was originally spotted on a VM running a long way from the client machine, but i have subsequently reproduced it by restoring and connecting to a machine that is under my desk.
edit: as requested, the query is:
CREATE PROCEDURE [trading].[Trade_SelectLegalEntityPositionChunked]
#PartyPositions trading.PartyPositionType READONLY,
#Offset INT,
#ChunkSize INT
AS
BEGIN
SET NOCOUNT ON;
DECLARE #OffsetLocal INT = #Offset;
DECLARE #ChunkSizeLocal INT = #ChunkSize;
SELECT /* stuff */
FROM [trading].[Trade]
JOIN [refdata].[Book] ON [refdata].[Book].[BookId] = [trading].[Trade].[BookId]
JOIN #PartyPositions pos
ON pos.[PartyId] = [refdata].[Book].[LegalEntityId]
AND [trading].[Trade].[TradeType] = pos.[TradeType]
AND [trading].[Trade].[InstrumentId] = pos.[InstrumentId]
ORDER BY [trading].[Trade].[TradeId]
OFFSET #OffsetLocal ROWS FETCH NEXT #ChunkSizeLocal ROWS ONLY;;
END
edit: i've also checked:
other threads - there are none that should be getting in the way: the only threads that are running are the one running the query and the usual supporting threads such as the main thread and message loop (I'm testing from a CUI)
edit (bored yet?)
if i reduce the scope of the query a bit i can see it blocking while parsing a record for about 4 mins. It then carries on reading stuff until it hangs again.
I still cannot see anything much going on in the client - i now have the GcNotification class from CLR via C# running and there isn't any GC going on.This points to the database, as #Crowcoder says but the fact it runs fine in SSMS and it pausing midway through reading a record means I am loathe to blame it. This is probably down to my lack of knowledge about databases though!
Related
I want to emphasize that I'm looking for ideas, not necessarily a concrete answer since it's difficult to show what my queries look like, but I don't believe that's needed.
The process looks like this:
Table A keeps filling up, like a bucket - an SQL job keeps calling SP_Proc1 every minute or less and it inserts multiple records into table A.
At the same time a C# process keeps calling another procedure SP_Proc2 every minute or less that does an ordered TOP 5 select from table A and returns the results to the C# method. After C# code finishes processing the results it deletes the selected 5 records from table A.
I bolded the problematic part above. It is necessary that the records from table A be processed 5 at a time in the order specified, but a few times a month SP_Proc2 selects the ordered TOP 5 records in a wrong order even though all the records are present in table A and have correct column values that are used for ordering.
Something to note:
I'm ordering by integers, not varchar.
The C# part is using 1 thread.
Both SP_Proc1 and SP_Proc2 use a transaction and use READ COMMITTED OR READ COMMITTED SNAPSHOT transaction isolation level.
One column that is used for ordering is a computed value, but a very simple one. It just checks if another column in table A is not null and sets the computed column to either 1 or 0.
There's a unique nonclustered index on primary key Id and a clustered index composed of the same columns used for ordering in SP_Proc2.
I'm using SQL Server 2012 (v11.0.3000)
I'm beginning to think that this might be an SQL bug or maybe the records or index in table A get corrupted and then deleted by the C# process and that's why I can't catch it.
Edit:
To clarify, SP_Proc1 commits a big batch of N records to table A at once and SP_Proc2 pulls the records from table A in batches of 5, it orders the records in the table and selects TOP 5 and sometimes a wrong batch is selected, the batch itself is ordered correctly, but a different batch was supposed to be selected according to ORDER BY. I believe Rob Farley might have the right idea.
My guess is that your “out of order TOP 5” is ordered, but that a later five overlaps. Like, one time you get 1231, 1232, 1233, 1234, and 1236, and the next batch is 1235, 1237, and so on.
This can be an issue with locking and blocking. You’ve indicated your processes use transactions, so it wouldn’t surprise me if your 1235 hasn’t been committed yet, but can just be ignored by your snapshot isolation, and your 1236 can get picked up.
It doesn’t sound like there’s a bug here. What I’m describing above is a definite feature of snapshot isolation. If you must have 1235 picked up in an earlier batch than 1236, then don’t use snapshot isolation, and force your table to be locked until each block of inserts is finished.
An alternative suggestion would be to use a table lock (tablock) for the reading and writing procedures.
Though this is expensive, if you desire absolute consistency then this may be the way to go.
I'm calling a simple stored procedure that returns around 650 rows. There are several joins and the procedure takes about 5-6 seconds. No problem.
Enumerating the results, however, is taking about a minute to complete.
using (var context = new DBContext())
{
var results = context.GetResults(param); //5-6 seconds
var resultList = results.ToList(); //1 minute+
}
I don't use Entity Framework much, but this seems abnormal. Am I doing something wrong? Is there something I can look at to speed this up? The table is huge, but the way I read it, this code should only be enumerating the 650 results... which should take no time at all.
Note: Not sure if this is related, but the time it takes to select all rows from said table is about the same (around a minute)
The solution to my problem was to disable parameter sniffing by creating a copy of the input parameter.
alter procedure dbo.procedure
#param int
as
begin
set nocount on;
declare #paramCopy int
set #paramCopy = #param
...
Based on your recent edit, I have an idea of what's happening. I think that the .GetResults() call is simply getting the query ready to be run, utilizing deferred execution. Only when you are calling .ToList() in the next line is it actually going out and trying to build the entities themselves (hence the time difference).
So why is it taking so long to load? That could be for a number of reasons, including:
You might have lazy loading disabled. This will cause all of the records to be fully loaded, with all of their respective navigational properties as well, and have all of that be tracked by the DbContext. That makes for a lot of memory consumption. You might want to consider turning it on (but not everyone likes having lazy loading enabled).
You are allowing the tracker to track all of the records, which takes up memory. Instead of this, if the data you're grabbing is going to be read-only anyway, you might want to consider the use of AsNoTracking, like in this blog post. That should reduce the load time.
You could be grabbing a lot of columns. I don't know what your procedure returns, but if it's a lot of rows, with lots of different columns, all of that data being shoved into memory will take a loooong time to process. Instead, you might want to consider only selecting as few columns as needed (by using a .Select() before the call to .ToList()) to only grab what you need.
I'm developing an ASP.NET app that analyzes Excel files uploaded by user. The files contain various data about customers (one row = one customer), the key field is CustomerCode. Basically the data comes in form of DataTable object.
At some point I need to get information about the specified customers from SQL and compare it to what user uploaded. I'm doing it the following way:
Make a comma-separated list of customers from CustomerCode column: 'Customer1','Customer2',...'CustomerN'.
Pass this string to SQL query IN (...) clause and execute it.
This was working okay until I ran into The query processor ran out of internal resources and could not produce a query plan exception when trying to pass ~40000 items inside IN (...) clause.
The trivial ways seems to:
Replace IN (...) with = 'SomeCustomerCode' in query template.
Execute this query 40000 times for each CustomerCode.
Do DataTable.Merge 40000 times.
Is there any better way to work this problem around?
Note: I can't do IN (SELECT CustomerCode FROM ... WHERE SomeConditions) because the data comes from Excel files and thus cannot be queried from DB.
"Table valued parameters" would be worth investigating, which let you pass in (usually via a DataTable on the C# side) multiple rows - the downside is that you need to formally declare and name the data shape on the SQL server first.
Alternatively, though: you could use SqlBulkCopy to throw the rows into a staging table, and then just JOIN to that table. If you have parallel callers, you will need some kind of session identifier on the row to distinguish between concurrent uses (and: don't forget to remove your session's data afterwards).
You shouldn't process too many records at once, because of errors as you mentioned, and it is such a big batch that it takes too much time to run and you can't do anything in parallel. You shouldn't process only 1 record at a time either, because then the overhead of the SQL server communication will be too big. Choose something in the middle, process eg. 10000 records at a time. You can even parallelize the processing, you can start running the SQL for the next 10000 in the background while you are processing the previous 10000 batch.
I have an Oracle database that I access using Devart and Entity Framework.
There's a table called IMPORTJOBS with a column STATUS.
I also have multiple processes running at the same time. They each read the first row in IMPORTJOBS that has status 'REGISTERED', put it to status 'EXECUTING', and if done put it to status 'EXECUTED'.
Now because these processes are running in parallel, I believe the following could happen:
process A reads row 10 which has status REGISTERED,
process B also reads row 10 which has still status REGISTERED,
process A updates row 10 to status EXECUTING.
Process B should not be able to read row 10 as process A already read it and is going to update its status.
How should I solve this? Put read and update in a transaction? Or should I use some versioning approach or something else?
Thanks!
EDIT: thanks to the accepted answer I got it working and documented it here: http://ludwigstuyck.wordpress.com/2013/02/28/concurrent-reading-and-writing-in-an-oracle-database.
You should use the built-in locking mechanisms of the database. Don't reinvent the wheel, especially since RDBMS are designed to deal with concurrency and consistency.
In Oracle 11g, I suggest you use the SKIP LOCKED feature. For example each process could call a function like this (assuming id are number):
CREATE OR REPLACE TYPE tab_number IS TABLE OF NUMBER;
CREATE OR REPLACE FUNCTION reserve_jobs RETURN tab_number IS
CURSOR c IS
SELECT id FROM IMPORTJOBS WHERE STATUS = 'REGISTERED'
FOR UPDATE SKIP LOCKED;
l_result tab_number := tab_number();
l_id number;
BEGIN
OPEN c;
FOR i IN 1..10 LOOP
FETCH c INTO l_id;
EXIT WHEN c%NOTFOUND;
l_result.extend;
l_result(l_result.size) := l_id;
END LOOP;
CLOSE c;
RETURN l_result;
END;
This will return 10 rows (if possible) that are not locked. These rows will be locked and the sessions will not block each other.
In 10g and before since Oracle returns consistent results, use FOR UPDATE wisely and you should not have the problem that you describe. For instance consider the following SELECT:
SELECT *
FROM IMPORTJOBS
WHERE STATUS = 'REGISTERED'
AND rownum <= 10
FOR UPDATE;
What would happen if all processes reserve their rows with this SELECT? How will that affect your scenario:
Session A gets 10 rows that are not processed.
Session B would get the same 10 rows, is blocked and waits for session A.
Session A updates the selected rows' statuses and commits its transaction.
Oracle will now (automatically) rerun Session B's select from the beginning since the data has been modified and we have specified FOR UPDATE (this clause forces Oracle to get the last version of the block).
This means that session B will get 10 new rows.
So in this scenario, you have no consistency problem. Also, assuming that the transaction to request a row and change its status is fast, the concurrency impact will be light.
Each process can issue a SELECT ... FOR UPDATE to lock the row when they read it. In this scenario, process A will read and lock the row, process B will attempt to read the row and block until process A releases the lock by committing (or rolling back) its transaction. Oracle will then determine whether the row still meets B's criteria and, in your example, won't return the row to B. This works but it means that your multi-threaded process may now be effectively single-threaded depending on how your transaction control needs to work.
Possible ways to improve scalability
A relatively common approach on the consumer to resolving this is to have a single coordinator thread that reads the data from the table, parcels out work to different threads, and updates the table appropriately (including knowing how to re-assign a job if the thread that was assigned it has died).
If you are using Oracle 11.1 or later, you can use the SKIP LOCKED clause on your FOR UPDATE so that each session gets back the first row that meets their criteria and is not locked (the clause existed in earlier versions but was not documented so it may not work correctly).
Rather than using a table for ImportJobs, you can use a queue with multiple consumers. This will allow Oracle to distribute messages to each process without you needing to build any additional locking (Oracle queues are doing it all behind the scenes).
Use versioning and optimistic concurrency.
The IMPORTJOBS table should have a timestamp column that you mark as ConcurrencyMode = Fixed in your model. Now when EF tries to do an update the timestamp column is incorporated in the update statement: WHERE timestamp = xxxxx.
For B, the timestamp changed in the mean time, so a concurrency exception is raised, which, in this case, you handle by skipping the update.
I'm from a SQL server background and I don't know the Oracle equivalent of timestamp (or rowversion), but the idea is that it's a field that auto-updates when an update is made to a record.
I understand the concepts around multithreading and using Thread Pools. One concept I am trying to figure out is how to keep track of what emails have been sent to on each thread. So imagine, each thread is responsible for pulling x number of records, iterating through those emails, applying an email template, then saving the email to a pick up directory. Obviously, I need a way to tell each thread not to pull the same data as another thread.
One solution I was thinking was to page the data, have a global variable or array to keep track of the pages already sent to, have each thread examine that variable and start from the next available page. The only issue I can think of is if the data changes, then the pages available might get out of sync.
another solution is to set a boolean value in the database to determine if an account has been emailed to or not. So, EF would pull X amount of records and update those records as being ready to email on. This way each query would only look for emails that are not ready to be emailed to.
I wanted to get some other suggestions, if possible, or expand on the solutions I provided.
Given that you may one day want to scale to more than one app server, memory synchronization implementations might also not be sufficient to guarantee that emails are not duplicated.
One of the simplest ways to solve is to implement a batch processing mechanism would be at the database level.
Under a Unit of Work
Read N x records, with Pessimistic Locking (i.e. preventing concurrent reads by other threads pulling the same emails)
Stamp these records with a batch id (or a IsProcessed Indicator)
Return the records to your app
e.g. a Batching PROC in SQL server might look something like (Assuming table = dbo.Emails, which has a PK EmailId and a processed indicator BIT field IsProcessed):
CREATE PROC dbo.GetNextBatchOfEmails
AS
BEGIN
-- Identify the next N emails to be batched. UPDLOCK is to prevent another thread batching same emails
SELECT top 100 EmailId
INTO #tmpBatch
FROM dbo.Emails WITH (UPDLOCK)
WHERE IsProcessed = 0
-- Stamp emails as sent. Assumed that PROC is called under a UOW. The batch IS the UOW
UPDATE e
SET e.IsProcessed = 1
FROM dbo.Emails e
INNER JOIN #tmpBatch t
on e.EmailId = t.EmailId
-- Return the batch of emails to caller
SELECT e.*
FROM dbo.Emails e
INNER JOIN #tmpBatch t
on e.EmailId = t.EmailId
END
Then expose the PROC as an EF Function Import mapped to your Email Entity. Under a TransactionScope ts, you can then call the EF Function Import, and send emails, and call ts.Complete() on success.
In addition to nonnb's method, you can accomplish it all in one statement if you wish if you are using SQL Server 2005+.
;WITH q AS
(
SELECT TOP 10 *
FROM dbo.your_queue_table
WHERE
IsProcessing = 0
--you can obviously include more filtering criteria to meet your needs
)
UPDATE q WITH (ROWLOCK, READPAST)
SET IsProcessing = 1
OUTPUT INSERTED.*
There is also some great information located here about using database tables as queues.