We Keep Coding

Use linq to iterate through large DB tables

I have two tables: Foo and Bar. For each row in Foo, I now want to add a row in Bar which references the respective Foo record. Foo will likely contain several millions of records.
Normally this answer would have been perfect: linq to sql - loop through table data and set value. But as it says on the tin, using the following line is not particularly ideal for large tables.
List<User> users = dc.Users.ToList();
Since caching the entire table in a List<> is not going to work, what other options do I have? Is there an elegant way to "page through" the records, for instance? Since I am quite sure that this is a relatively common problem, I think it's likely that there is a best practice for this too. I have not been able to find it, however.

Your talking about several million rows of data, then Linq is not your friend.
Consider using a stored procedure or, if you like, DbContext.ExecuteCommand.
Both will result in a huge performance gain.

You can work with predefined batches using .Skip() and .Take() methods. Another thing to consider is using a trigger so that you don't need to worry about the second table at all.

fastest way to search huge list of big texts

I have a windows application written in C# that needs to load load 250,000 rows from database and provide a "search as you type" feature which means as soon as user types something in a text box, the application needs to search all 250,000 records (which are btw, single column with 1000 characters each row) using like search and display the found records.
The approach I followed was:
1- The application loads all the records into a typed List<EmployeeData>
while (objSQLReader.Read())
{
lstEmployees.Add(new EmployeesData(
Convert.ToInt32(objSQLReader.GetString(0)),
objSQLReader.GetString(1),
objSQLReader.GetString(2)));
}
2- In TextChanged event, Using LINQ, I search (with combination of Regular Expression) and attach the IEnumerable<EmployeesData> to a ListView which is in Virtual Mode.
String strPattern = "(?=.*wood*)(?=.*james*)";
IEnumerable<EmployeesData> lstFoundItems = from objEmployee in lstEmployees
where Regex.IsMatch(Employee.SearchStr, strPattern, RegexOptions.IgnoreCase)
select objEmployee;
lstFoundEmployees = lstFoundItems;
3- RetrieveVirtualItem event is handled to display items in ListView to display the item.
e.Item = new ListViewItem(new String[] {
lstFoundEmployees.ElementAt(e.ItemIndex).DateProjectTaskClient,
e.ItemIndex.ToString() });
Though the lstEmployees is loaded relatively fast (1.5 seconds) for loading the list from SQL Server, to search on TextChanged, it takes more than 7 minutes to search using LINQ. Searching thru SQL Server directly by performing a LIKE search takes less than 7 seconds.
What am I doing wrong here? How can I make this search faster (not more 2 seconds)? This is a requirement from my client. So, any help is highly appreciated. Please Help...

Does the database column that stores the text data have an index on it? If so, something similar to the trie structure that Nicholas described is already in use. Indexes in SQL Server are implemented using B+ trees, which have a an average search time on the order of log base 2 of n, where n is the height of the tree. This means that if you have 250,000 records in the table the number of operations required to search are log base 2 ( 250,000 ) or approximately 18 operations.
When you load all of the information into a data reader and then use a LINQ expression it's a linear operation, (O) n, where n is the length of the list. So worst case, it's going to be 250,000 operations. If you use a DataView there will be indexes that can be used to help with searching, which will drastically improve performance.
At the end of the day if there will not be too many requests submitted against the database server leverage the query optimizer to do this. As long as the LIKE operation isn't performed with a wildcard at the front of the string (i.e. LIKE %some_string) (negates the use of an index) and there is an index on the table you will have really fast performance. If there are just too many requests that will be submitted to the database server, either put all of the information into a DataView so an index can be used, or use a dictionary as Tim suggested above, which has a search time of O(1) (on the order of one), assuming the dictionary is implemented using a hash table.

You'd be wanting to preload things and build yourself a data structure called a trie
It's memory-intensive, but it's what the doctor ordered in this case.

See my answer to this question. If you need instant response (i.e. as fast as a user types), loading the data into memory can be a very attractive option. It may use a bit of memory, but it is very fast.
Even though there are many characters (250K records * 1000), how many unique values are there? An in-memory structure based off of keys with pointers to records matching those keys really doesn't have to be that big, even accounting for permutations of those keys.
If the data it truly won't fit into memory or changes frequently, keep it in the database and use SQL Server Full Text Indexing, which will handle searches such as this much better than a LIKE. This assumes a fast connection from the application to the database.
Full Text Indexing offers a powerful set of operators/expressions which can be used to make searches more intelligent. It's available with the free SQL Expression Edition, which will handle up to 10GB of data.

If the records can be sorted, you may want to go with a binary search, which is much, much faster for large data sets. There are several implementations in .NET collections, like List<T> and Array.

Reducing Parse Calls in Oracle

I am noticing the parse_calls are equal to the number of executions in our Oracle 11g database.
select parse_calls, executions
from v$sql order by parse_calls desc;
Running the above query gives the following result.
"PARSE_CALLS" "EXECUTIONS"
87480 87480
87475 87476
87044 87044
26662 26662
21870 21870
21870 21870
As I'm aware this is a major performance drawback. All of these SQL statements are either stored procedures or using bind variables. I'm also reusing the command objects that are calling the stored procedures from C#.
How do I reduce the number of parse calls in this?
Also, is there some method I can distinguish between hard parses and soft parses?
EDIT:
As #DCookie mentioned I ran the following query on the database.
SELECT s2.name, SUM(s1.value)
FROM v$sesstat s1 join v$statname s2 on s1.statistic# = s2.statistic#
WHERE s2.name LIKE '%parse count%'
GROUP BY s2.name
ORDER BY 1,2;
The result is as below
"NAME" "SUM(S1.VALUE)"
"parse count (describe)" 0
"parse count (failures)" 29
"parse count (hard)" 258
"parse count (total)" 11471
So the number of hard parses seem to be very low compared to the number of parses. Thanks to everyone for their responses :)
FINAL UPDATE:
The main issue for the parsing was because we had connection pooling turned off in the connection string. After turning on connection pooling I was able to completely resolve the parsing problem.

Start with this:
SELECT name, SUM(value)
FROM v$sesstat s1 join v$statname s2 on s1.statistic# = s2.statistic#
WHERE s1.name LIKE '%parse count%'
GROUP BY name
ORDER BY 1,2;
This will give you the number of hard parses and total parses. The parse_calls values in your query is total parses, hard and soft.
What does your SQL do? Not much cursor processing, mostly single statements? You are getting pretty much a 1-1 ratio of executions to parses, which, if they are soft parses means you are not doing much cursor processing.
EDIT:
Unless you can modify your code to open and hang on to a cursor for each of your SQL statements, and reuse them as much as possible within a session, I don't think you can avoid the parses.

A parse call has to occur any time a new cursor is created, even if the statement is in the library cache. It is the parse call that checks the library cache. If the statement is found in the library cache, it is a soft parse.
#DCookie has given an answer to your question about checking hard vs. soft parse count. I expect you will find most parse calls are soft parses. Note that you shouldn't expect the counts returned from v$sysstat to be very close to total parse calls from v$sql, since the former is a count since instance startup and the latter just shows the statements that are currently in the library cache.
The only way to avoid parse calls entirely at all is to keep a handle to an existing cursor, and execute it when needed, binding new values in when appropriate. This will occur sometimes by caching of cursors -- it's out of your explicit control, although I believe there are parameters you can change to affect it. In PL/SQL code you can explicitly hold onto cursors that you create and manage using the DBMS_SQL package. I would expect that C# has corresponding capabilities.
In any case, what you're looking at is probably not a problem. Just because counts seem high in no way implies that parsing is a bottleneck on your system.
First of all, you should check whether the SQL statements with those high parse counts are even in your control. When I did a modified version of your query on one of my systems:
select parse_calls, executions, parsing_schema_name,sql_text
FROM v$sql
ORDER BY parse_calls DESC;
I found that the statements with the highest number of parse calls were all recursive SQL parsed by SYS. This may not be the case for you depending on your usage, but it is something to check.

DataTable Select vs List<T> LINQ Performance

I have an app that does an SQL and loads a set of data into a datatable. As part of the processing there are 6 or 7 DataTable.Select() to filter some data. Each item that needs processing takes 300ms. There are 5000 items to process so takes 25 mins. This is unacceptable.
Would creating POCO's and loading them into a List and then using LINQ to query the list be quicker than using DataTable.Select?
Thanks
UPDATE: I have delved in a bit more and there are 2 datatables each with around 15000 records. The 2 queries used to populate the datatables take a second each. It then takes 25mins to loop over 5000 items in a Dictionary's values property and do 5 DataTable.Select's
eg/
foreach (OutputRecord Mailpiece in DictionaryMailpieces.Values)
{
try
{
DataRow[] R = DataTable1.Select("MAILPIECE = " + Mailpiece.MailpieceSetSequenceNumber + " AND (STATUS = 4034 OR STATUS = 4037)", "DAL_DATE desc");
if (R != null && R.Length > 0)
{
}
}
catch
{
}
}

Funny there is no "SQL" tag associated with your question. I suggest, you learn how to use the SQL language and its benefits. From what you say, it is likely you are, with your code, creating a lot of Cartesian products, instead of leveraging the Relational Database facilities (joins, indexes, etc.)
Using cross joins of DataTables or Lists or anything similar will always lead to heavy performance degradation, whatever language or platform is used.
That said, you could use LINQ because it's capable of producing smart SQL (dynamically), but you still want to avoid all ToList(), ToArray() and similar extension methods on IEnumerable(T) that summon all the underlying data (keep it enumerable from end to end and leverage "object streaming" whenever possible). If you understand really what is a Relational Database and how to use it efficiently, you will be a better LINQ developer.

Almost anything will be faster than manipulating an ADO.NET DataTable -- they are not designed for fast retrieval in any sense. You should also put the objects into an appropriate data structure; a DataTable is a red-black binary tree of rows, so if you don't want that, you shouldn't use one.
If you're just using the DataTable as a sequential collection of rows with fields, then you'll probably see a factor of 2 or more speedup just by replacing the DataTable with a List<T> and replacing your Select calls with Where calls, although it depends on what you're doing with it.
EDIT: Actually, I changed my mind. Nothing you could be doing sorting-or-filtering-wise with 5000 items in a DataTable implies a cost of anywhere close to 300ms, so the bottleneck is probably unrelated.

Using LINQ will most likely not provide a huge speed improvement, in and of itself. That being said, you could potentially use PLINQ to simplify the parallelization of the processing, which could allow this to scale better on multicore systems. This tends to be much simpler when using POCOs instead of DataTable, as DataTable is not thread safe, and has concurrency issues.
That being said - I suspect that profiling this process would, overall, give you a much better potential improvement, as it would allow you to find and correct any bottlenecks. If there are no specific bottlenecks, and the process just requires that amount of raw processing, caching may also help. In addition, it's possible that leaving the data on the database and using some form of ORM may help, as well, as the "6 or 7" filter operations could be run on a scalable server instead of locally. All of this is highly dependent on the nature of your data and algorithm, however, so it would require some careful consideration to determine whether it would be beneficial or detrimental overall.

Would creating POCO's and loading them into a List and then using LINQ to query the list be quicker than using DataTable.Select?
We have no idea, you didn't give us enough information. We have no idea how your method is coded (maybe you have an errant Thread.Sleep(300) buried in your code; we can't tell).
More importantly, we need to know where the bottleneck is. To figure that out, you need a profiler. Get one and then once you know what the bottleneck is, we can probably help you eek out some extra performance.
That said, switching to LINQ probably isn't going to single-handedly be the solution to your performance problems. Something else is wrong, and whether it is coded using DataTables and LINQ is mostly irrelevant. The performance gains are going to come from having the right plan of attack to your problem; DataTables and LINQ are just ways of implementing that plan of attack.

Where to do pagination/filtering? In the database or in the code?

I have to write the code for the following method:
public IEnumerable<Product> GetProducts(int pageNumber, int pageSize, string sortKey, string sortDirection, string locale, string filterKey, string filterValue)
The method will be used by a web UI and must support pagination, sorting and filtering. The database (SQL Server 2008) has ~250,000 products. My question is the following: where do I implement the pagination, sorting and filtering logic? Should I do it in a T-SQL stored procedure or in the C# code?
I think that it is better if I do it in T-SQL but I will end up with a very complex query. On the other hand, doing that in C# implies that I have to load the entire list of products, which is also bad...
Any idea what is the best option here? Am I missing an option?

You would definitely want to have the DB do this for you. Moving ~250K records up from the database for each request will be a huge overhead. If you are using LINQ-to-SQL, the Skip and Take methods will do this (here is an example), but I don't know exactly how efficient they are.

I think other (and potentionaly best) option is to use some higher level framework that shield you from complexity of query writing. EntityFramework, NHibernate and LINQ(toSQL) help you a lot. That said database is typically best place to do it in your case.

today itself I implement pagination for my website. I have done with stored procedure though I am using Entity-Framework. I found that executing a complex query is better then fetching all records and doing pagination with code. So do it with stored procedure.
And I see your code line, which you have attached, I have implemented in same way only.

I would definatly do it in a stored procedure something along the lines of :
SELECT * FROM (
SELECT
ROW_NUMBER() OVER (ORDER BY Quantity) AS row, *
FROM Products
) AS a WHERE row BETWEEN 11 AND 20
If you are using linq then the Take and Skip methods will take care of this for you.

Definitely in the DB for preference, if at all possible.
Sometimes you can mix things up a bit such as if you have the results returned from a database function (not a stored procedure, functions can be parts of larger queries in ways that stored procedures cannot), then you can have another function order and paginate, or perhaps have Linq2SQL or similar call for a page of results from said function, producing the correct SQL as needed.
If you can at least get the ordering done in the database, and will usually only want the first few pages (quite often happens in real use), then you can at least have reasonable performance for those cases, as only enough rows to skip to, and then take, the wanted rows need be loaded from the db. You of course still need to test that performance is reasonable in those rare cases where someone really does look for page 1,2312!
Still, that's only a compromise for cases where paging is very difficult indeed, as a rule always page in the DB unless it's either extremely difficult for some reason, or the total number of rows is guaranteed to be low.