We Keep Coding

Effective way of displaying results of stored procedure with interactive button columns

Apologies if the title is vague - I will explain in as much detail as possible the problem.
I have an ASP.NET webpage which queries a TSQL database using a stored procedure, the query is run on Page_Load. The procedure returns a dataset (with ID column, RequestID). The contents are essentially a list of data-specific requests that need to be approved or rejected as exceptions to another workflow (not entirely relevant but may help explain my approach)
I'd like to display the results of this procedure in a grid (currently this is easy enough) however I'm running into problems when trying to add buttons that allow the user to interact with the data, e.g. I'd like 'Reject/Approve' buttons.
When one of these buttons is pressed the relevant stored procedure would be run, with the RequestID of the row pressed passed as a parameter. The idea is to have the stored procedure update an audit log and update the relevant tables accordingly (each request represent a complex data object spanning multiple tables) - for this reason the default controls aren't sufficient for select/edit/delete.
I feel like I'm doing this entirely wrong from the basic approach, which is using a gridview and trying to add custom SelectMethod/DeleteMethod's. What would be a best practice approach for doing this?
EDIT*
Here is the data table and how i'm populating it:
DataTable Requests = new DataTable();
Requests = GetRequests(connString); //Method that returns a datatable from a stored procedure
Displaying the data in a gridview as follows:
gridView.DataSource = Requests;
gridView.DataBind();
However when i add the following code, which is me trying to fire a custom method when using the default select/delete controls in a gridview i get an exception:
gridView.AutoGenerateDeleteButton = true;
gridView.AutoGenerateSelectButton = true;
gridView.SelectMethod = "ApproveRequest";
gridView.DeleteMethod = "RejectRequest";
Here is the exceptiopn (which make sense, but makes me think i'm going about this the wrong way).
Message=DataSource or DataSourceID cannot be defined on 'gridView' when it uses model binding.

Millions of rows in the database, only so much needed

Problem summary:
C# (MVC), entity framework 5.0 and Oracle.
I have a couple of million rows in a view which joins two tables.
I need to populate dropdownlists with filter-posibilities.
The options in these dropdownlists should reflect the actual contents
of the view for that column, distinct.
I want to update the dropdownlists whenever you select something, so
that the new options reflect the filtered content, preventing you
from choosing something that would give 0 results.
Its slow.
Question: whats the right way of getting these dropdownlists populated?
Now for more detail.
-- Goal of the page --
The user is presented with some dropownlists that filter the data in a grid below. The grid represents a view (see "Database") where the results are filtered.
Each dropdownlist represents a filter for a column of the view. Once something is selected, the rest of the page updates. The other dropdownlists now contain the posible values for their corresponding columns that complies to the filter that was just applied in the first dropdownlist.
Once the user has selected a couple of filters, he/she presses the search button and the grid below the dropdownlists updates.
-- Database --
I have a view that selects almost all columns from two tables, nothing fancy there. Like this:
SELECT tbl1.blabla, tbl2.blabla etc etc
FROM table1 tbl1, table2 tbl2
WHERE bsl.bvz_id = bvz.id AND bsl.einddatum IS NULL;
There is a total of 22 columns. 13 VARCHARS (mostly small, 1 - 20, one of em has a size of 2000!), 6 DATES and 3 NUMBERS (one of them size 38 and one of them 15,2).
There are a couple of indexes on the tables, among which the relevant ID's for the WHERE clause.
Important thing to know: I cannot change the database. Maybe set an index here and there, but nothing major.
-- Entity Framework --
I created a Database first EDMX in my solution and also mapped the view. There are also classes for both tables, but I need data from both of them, so I don't know if I need them. The problem by selecting things from either table would be that you can't apply half of the filtering, but maybe there are smart way's I didn't think of yet.
-- View --
My view is strongly bound to a viewModel. In there I have a IEnumerable for each dropdownlist. The getter for these gets its data from a single IEnumerable called NameOfViewObjects. Like this:
public string SelectedColumn1{ get; set; }
private IEnumerable<SelectListItem> column1Options;
public IEnumerable<SelectListItem> Column1Options
{
get
{
if (column1Options == null)
{
column1Options= NameOfViewObjects.Select(item => item.Column1).Distinct()
.Select(item => new SelectListItem
{
Value = item,
Text = item,
Selected = item.Equals(SelectedColumn1, StringComparison.InvariantCultureIgnoreCase)
});
}
return column1Options;
}
}
The two solutions I've tried are:
- 1 -
Selecting all columns in a linq query I need for the dropdownlists (the 2000 varchar is not one of them and there are only 2 date columns), do a distinct on them and put the results into a Hashset. Then I set NameOfViewObjects to point towards this hashset. I have to wait for about 2 minutes for that to complete, but after that, populating the dropdownlists is almost instant (maybe a second for each of them).
model.Beslissingen = new HashSet<NameOfViewObject>(dbBes.NameOfViewObject
.DistinctBy(item => new
{
item.VarcharColumn1,
item.DateColumn1,
item.DateColumn2,
item.VarcharColumn2,
item.VarcharColumn3,
item.VarcharColumn4,
item.VarcharColumn5,
item.VarcharColumn6,
item.VarcharColumn7,
item.VarcharColumn8
}
)
);
The big problem here is that the object NameOfViewObject is probably quite large, and even though using distinct here, resulting in less than 100.000 results, it still uses over 500mb of memory for it. This is unacceptable, because there will be a lot of users using this screen (a lot would be... 10 max, 5 average simultaniously).
- 2 -
The other solution is to use the same linq query and point NameOfViewObjects towards the IQueryable it produces. This means that every time the view wants to bind a dropdownlist to a IEnumerable, it will fire a query that will find the distinct values for that column in a table with millions of rows where most likely the column it's getting the values from is not indexed. This takes around 1 minute for each dropdownlist (I have 10), so that takes ages.
Don't forget: I need to update the dropdownlists every time one of them has it's selection changed.
-- Question --
So I'm probably going at this the wrong way, or maybe one of these solutions should be combined with indexing all of the columns I use, maybe I should use another way to store the data in memory, so it's only a little, but there must be someone out there who has done this before and figured out something smart. Can you please tell me what would be the best way to handle a situation like this?
Acceptable performance:
having to wait for a while (2 minutes) while the page loads, but
everything is fast after that.
having to wait for a couple of seconds every time a dropdownlist
changes
the page does not use more than 500mb of memory

Of course you should have indexes on all columns and combinations in WHERE clauses. No index means table scan and O(N) query times. Those cannot scale under any circumstance.
You do not need millions of entries in a drop down. You need to be smarter about filtering the database down to manageable numbers of entries.
I'd take a page from Google. Their type ahead helps narrow down the entire Internet graph into groups of 25 or 50 per page, with the most likely at the top. Maybe you could manage that, too.
Perhaps a better answer is something like a search engine. If you were a Java developer you might try Lucene/SOLR and indexing. I don't know what the .NET equivalent is.

First point you need to check is your DB, make sure you have to right indexes and entity relations in place,
next if you want to dynamical build your filter options then you need to run the query with the existing filters to obtain what the next filter can be. there are several ways to do this,
firstly you can query the data and extract the values from the return, this has a huge load time and wastes time returning data you don't want (unless you are live updating the results with the filter and dont have paging, in which case you might aswell just get all the data and use linqToObjects to filter)
a second option is to have a parallel queries for each filter that returns the possible filters, so filter A = all possible values of A from data, filter b = all possible values of B when filtered by A in the data, C = all possible values of C when filtered by A & B in the data, etc. this is better than the first but not by much
another option is the use aggregates to speed things up, ie you have a parallel query as above but instead of returning the data you return how many records are returned, aggregate functions are always quicker so this will cut your load time dramatically but you are still repeatedly querying a huge dataset to it wont be exactly nippy.
you can tweak this further using exist to just return a 0 or 1.
in this case you would look at a table with all possible filters and then remove the ones with no values from the parallel query
the next option will be the fastest by a mile is to cache the filters in the DB, with a separate table
then you can query that and say from Cache, where filter = ABC select D, the problem with this maintaining the cache, which you would have to do in the DB as part of the save functions, trigggers etc.

Another solution that can be added in addition to the previous suggestions is to use the /*+ result_cache */ hint, if your version of Oracle supports it (Oracle version 11g or later). If the output of the query is small enough for a drop-down list, then when a user enters criteria that matches the same criteria another user used, the results are returned in a few milliseconds instead of a few seconds or minutes. Result cache is wonderful for queries that return a small set of rows out of millions.
select /*+ result_cache */ item_desc from some_table where item_id ...
The result cache is automatically flushed when any insert/updates/deletes occur on the database tables.

I've done something 'kind of' similar in the past - if you can add a table to the database then I'd explore introducing a 'scratchpad' type table where results are temporarily stored as the user refines their search. Since multiple users could be working simultaneously the table would have to have an additional column for identifying the user.
I'd think you'd see some performance benefit since all processing is kept server-side and your app would simply be pulling data from this table. Since you're adding this table you would also have total control over it.
Essentially I'd imagine the program flow would go something like:
User selects some filters and clicks 'Search'.
Server populates scratchpad table with results from that search.
App populates results grid from scratchpad table.
User further refines search and clicks 'Search'.
Server removes/adds rows to scratchpad table as necessary.
App populates results grid from scratchpad table.
And so on.
Rather than having all the users results in one 'scratchpad' table you could possibly explore having temporary 'scratchpad' tables per user.

Best caching strategy in this particular scenario

I am retrieving a big list of news items.
I want to utilise caching of this list for obvious reasons (running a query any more than necessary won't do).
I bind this list to a repeater on which I have extended to enable paging (flicking between pages causes page load and so a retrieval of the list again).
The complication is the fact that these news items can also be queried by date from a query string 'year' that is present only when querying the news items by date.
Below is pseudo code for what I have so far (pasting the real code here would take too much time trimming out all of the bits that just add confusion):
if(complete news items list are not cached OR querystring["year"] != null)
{
Int year = queryString["year"] (could be null)
Cache.Add(GetNewsItems(year));
}
else
{
return cached newsItems;
}
The problem is that when the news items page loads (because of a paging control's postback) the querystring's [year] parameter will still be populated and so will re-do the GetNewsItems. Also - even if the home URL (i.e. no query string) is then navigated to - in effect there is still a cached version of the news items so it will not bother to try and retrieve them - but they MAY be there for a particular year and so aren't relevant to an 'all years' kind of search.
Do I add a new cache entry to flag what most recent search was done?
What considerations do I have to make here to cache timing out?
I can add a new query string if needs be (preferably not) - would this solve the problem?

Could you get the complete list on the first query, and cache this (presuming it is not too enormous)? Then for any subsequent queries get the data from the cache and query it to filter out just the year you want.
Maybe you could store the data in the cache in the form of
IDictionary<int, IEnumerable<NewsItem>>
(assuming there are more than one NewsItem for a year) where the key is the year so you can just retrieve a single dictionary value pair to get a year's data.
Alternatively cache the data by year, as you are doing in your sample, and implement a mechanism to get data from the cache for each year if it exists. When getting all, you can either just get all from the data store and cache this separately or implement some mechanism to determine what years are missing from the cache and just get these. Personally, I think I would go with caching all and filtering data from the cache so as to not clog up the memory with too much cached data.
A useful TryGetValue pattern to get from cache is something like:
private bool TryGetValue<U>(string key, out U value)
{
object cachedValue = HttpContext.Current.Cache.Get(key);
if (cachedValue == null)
{
value = default(U);
return false;
}
else
{
try
{
value = (U)cachedValue;
return true;
}
catch
{
value = default(U);
return false;
}
}
}

Not sure if this might help for your case, but you can add the repeater to an user control, and enable OutputCache for it, to be invalidated by POST/GET params:
<%# OutputCache Duration="100" VaryByParam="year" %>

How to access a cell in a DataRowView based on the columns DataPropertyName?

I have a Windows Forms application with a DataSet. I've simply used the Data | Add New DataSource to add the Products table of the Northwind database to my DataSources and created a DataGridView showing the contents of the Products table. I just dragged the Products table from the Data Sources window to the form, so all the columns are created automatically.
Now, I want the rows containing a product where the Discontinued column is true to be painted in a different color. I've created a CellPainting event handler for it, but I'm having trouble locating the value for the Discontinued column.
Because the DataGridView is created automatically, the Columns in it have names like dataGridViewTextBoxColumn1, which has a DataPropertyName of "ProductID".
My question is: how can I find the value for Discontinued based on the DataPropertyName? Or am I required to use the name of the column itself? (In which case I better give it a meaningful name)
My code is:
private void productsDataGridView_CellPainting(object sender,
DataGridViewCellPaintingEventArgs e)
{
if (e.RowIndex >= 0)
{
var row = productsDataGridView.Rows[e.RowIndex];
if ((bool) (row.Cells[ nameOrIndexOfColumn ].Value))
{
e.CellStyle.ForeColor = Color.DarkGray;
}
}
}
How can I access the Value of the Column with DataPropertyName "Discontinued"?
Solution
Based on Neil Barnwell's answer, this seems to be a way.
private void productsDataGridView_CellPainting(object sender, DataGridViewCellPaintingEventArgs e)
{
if (e.RowIndex >= 0)
{
var productView = (DataRowView) productsDataGridView.Rows[e.RowIndex].DataBoundItem;
var product = productView.Row as NorthwindDataSet.ProductsRow;
if (product != null && product.Discontinued)
{
e.CellStyle.ForeColor = Color.DarkGray;
}
}
}
The big advantage of this is that the Discontinued value doesn't have to be an actual column on the DataGridView.

Don't get the value from the column in the grid, get the value from the actual datarow that populates the gridrow. This way you can avoid all the magic strings etc.
There's a little bit of casting to be done because the [Type]DataRow is hidden inside a DataView that's attached to the grid, but it is a more elegant approach (and far less brittle in case of future changes) if you integrate it nicely into your code than relying on magic strings.
Here's an old blog post of mine that describes in detail how to do it:
http://koder.wordpress.com/2010/04/09/getting-data-from-a-winforms-datagridview/
UPDATE
You've mentioned that you're using Northwind and that you've "simply dragged the Products table to the form", so I'm guessing this isn't a mission-critical piece of software, but for the benefit of other people reading, I just wanted to suggest that this wouldn't be a typical approach any more in a real application.
Typically these days we'd consider having a Domain Model, perhaps using an ORM to obtain our Domain Objects from the data store (of course datasets aren't a real ORM), then possibly using things like MVVM to build data structures optimised for binding to UI elements from those Domain Objects.
Using this approach, because you have the actual data to hand in your ViewModel, you can calculate such rules as colours etc from the real data, and the UI simply displays the results of applying those business rules.

Why don't you try giving a name to that particular column yourself and then you know how to access it. Visual studio shouldn't have any problem with this even if its auto generated. Because it was generated to help you do all the binding and column creating stuff in the background. But it is still available for editing, autogenerated doesn't mean it generated at run-time . Its just helping you do the usual stuff, but you can still edit it.
row.Cells[ nameOrIndexOfColumn ]

You can try something on these lines
if (dataGridView1.Columns[e.ColumnIndex].DataPropertyName == "Discontinued")
{
if (dataGridView1[reqdColumnIndex, e.RowIndex].FormattedValue as bool)
{
e.CellStyle.ForeColor = Color.Gray;
}
}
reqdColumnIndex is the column which has your bool value

Server Side Pagination

I am loading data into my DataGrid via the ItemsSource property. I have a DataPager as well for pagination. The Grid is populated by calling a WCF service which returns a List.
public void webService_GetProductsCompleted(object sender, GetServiceReference.GetProductsCompletedEventArgs e)
{
PagedCollectionView pagingCollection = new PagedCollectionView(e.Result);
pgrProductGrids.Source = pagingCollection;
grdProductGrid.ItemsSource = pagingCollection;
}
Now there is a new requiremement that I want to load data with server side pagination. I'm a newbie learning Silverlight and for me the concept of server side paging is completely new too. So I came here to know what's required for server side pagination. Any good examples, tutorials,step-by-step guidelines that can give me a direction? I have to complete this task in a limited time. Please guide seniors

I typically use 1 of these 2 options:
1: If your data is sorted/paged by a field that has a unique value in it, let the database do the heavy lifting by utilizing the TOP feature and an ORDER BY. This way the smallest amount of data is returned from the server to the page. In the following example, MyTable has a field "NAME" that is unique and is how I want the data sorted. I am getting 10 records per page.
SELECT TOP 10 * FROM MyTable Where Name> [The last name in the previous 10 results] ORDER BY Name
You could use this for multiple fields if you wanted to use a windowing function like ROW_NUMBER() (I don't know what database you are using, this assumes SQL Server)
2: If this is not the case, then it gets ugly. You need to get all the data and get all the records between [The last page * Number per page] and the number of records per page in some iterative code. (Ugly, slow, lots of memory, not possible on large data sets.)