Details
VS-2008 Professional SP1
Version .net 3.5
Language:C#
I have a WPF Datagrid which loads from Linq-sql query Datacontext data item.The result set contains around 200k rows and its it very slow loading them,sorting,filtering etc.
What is the simple and easy way to improve the speed?
Couple of things I saw searching are
Scrollview,Data virtualization etc people also talk about Paging,Profiling etc
Loading data: 200k rows is a lot of data that no one (user) wants to see in one place. It will definitely reduce your UI user experience. So your best bet is to filter your data just to reduce the amount of it (for example do not show closed orders, just show the open ones). If you can't do so, you should use Virtualization. I didn't see any applications that use pagination in order to show data (Of course except in web). Most of the time it isn't such a good approach. But if you are talking about a type of data that is like search engines results you must use it. But keep in mind that most users won't exceed page 10 in search engines results.
Filtering: I would suggest doing it on your server side for such a huge amount of data (SQL Server here), or as I said first filter the whole 200k to reduce the amount on server side and then filter it (for user) in order to find something, on the client side. You might also find the following link helpful:
http://www.codeproject.com/KB/WPF/DataGridFilterLibrary.aspx
Sorting: Again I would suggest server-client solution but you might also find following links helpful:
http://blogs.msdn.com/b/jgoldb/archive/2008/08/26/improving-microsoft-datagrid-ctp-sorting-performance.aspx
http://blogs.msdn.com/b/jgoldb/archive/2008/08/28/improving-microsoft-datagrid-ctp-sorting-performance-part-2.aspx
http://blogs.msdn.com/b/jgoldb/archive/2008/10/30/improving-microsoft-datagrid-sorting-performance-part-3.aspx
Many people don't use default SortMemberPath of WPF datagrid just because it uses reflection on every single record and this will highly reduce the performance of the sorting process.
Hosein
Here is a very good sample of Data Virtualization (Not UI Virtualization):
http://www.codeproject.com/KB/WPF/WpfDataVirtualization.aspx
Althogh it doesn't support the LINQ IQueryable objects directly but you can use this sample as it is. Of course I'm now wokring to improve it to work with IQueryable objects directly. I think it's not so hard.
Wow, 200K rows is a lot of data. Paging sounds like a good idea. Try to decide how many rows per page you want, say 50. Upon showing the screen the first time, show only the first 50. Then give the user the option to move between pages.
Sorting might be trickier this way though.
Virtualization can be another option, sadly, I have yet to work with virtualization.
Sometimes you may have only ~30 visible rows to load and if those rows + whatever columns are expensive to load due to their number and complexity of the each cell (it's template, or how many wpf elements it has), none of the above comments really make a difference. Each row will take it's sweet time to load!
What helps is to stagger or lazily load each row on the UI, so that the user sees that the ui is doing something rather than just freezing for ~10+ seconds..
For simplicity, assuming that the datagrid ItemSource="{Binding Rows}", and Rows is IEnumerable, where Row is some class you created : add a property IsVisible to Row (don't forget to raise property changed, of course)
you could do something like this:
private void OnFirstTimeLoad()
{
Task.Factory.StartNew(() =>
{
foreach (var row in ViewModel.Rows)
{
/*this is all you really need,
note: since you're on the background thread, make sure IsVisible executes on the UI thread, my utils method does just that*/
myUtils.ExecuteOnUiThread(() => row.IsVisible = true);
/*optional tweak:
this just forces Ui to refresh so each row repaint staggers nicely*/
Application.Current.Dispatcher
.Invoke(DispatcherPriority.Background, (SendOrPostCallback) delegate { }, null);
}
});
}
oh, and don't forget to trigger in XAML:
<DataGrid.ItemContainerStyle>
<Style TargetType="{x:Type DataGridRow}">
<Setter Property="Visibility" Value="{Binding Path=IsVisible, Converter={StaticResource BoolToVisibility}}"/>
........
Is your datagrid inside a Scrollviewer? Because it does the entire datagrid (all the rows) to be rendered. I had a similar problem and removing the scrollviewer solved the problem with the slow loading.
Question that you should be asking is:
Are users groing to look through 200K rows of data?
How much data is too much for the users? May be alert the user that the query returned too many rows and you are listing the first 1000
Is it worth your time & money to program paging, Data Virtualization etc if users do not look beyond the first 1000 rows.
Related
I know this is a stupid question, I need to let you guys know that I am fully aware that it is useless in 99% of situations to make a listbox with this many elements in c#:
That being said I need this to be done...is there any way to populate a listbox with 40000 elements without it completely destroying performance/freezing up, thanks!
note: I have tried it, this is per the exact requirements of a professor...when adding 40000 elements through a DataSource and DataBind the application freezes up
You tell me.
for(i=0;i<40000;i++)
{
listBox1.Items.Add("click me");
}
Even if is possible (I never tried it), the usability for this form will be 0.
In that cases a more usable implementation is via lookup text-boxes and lists, where the user can enter a text to search record that matches this text and displays them in a any kind of list.
It is of course possible to do it, but not very practicable.
When using a desktop technology like WinForms or WPF, for a large number of items like this you are better off using something like an auto complete textbox, and have it set to filter/search after the user has typed two or three characters. In this case you can also use a control that offers scrolling virtualisation - this means that there is only a limited number of UI elements created in the scrolling portion of the dropdown, and those elements get reused when a scroll occurs. If you don't use virtualisation then a new element gets created for every list item that gets scrolled in to view. (Note that Silverlight controls have this functionality - just in case it's an option).
For ASP.NET though I would suggest that you do not want to do anything that would cause a large transfer of data (large items, or small items but lots of them) as it won't be performant. Instead you should look to do what Google does - retrieve search results in a paged fashion.
I have a ComboBox which when I repopulate it, it seems to be quite a time consuming task. After doing some profiling I have found that the majority of the time is spent in the ComboBox.Items.AddRange(Array) method. I have included an example method below which shows how I carryout the repopulating of the ComboBox.
public void Repopulate(IList<MyType> sortedList)
{
MyComboBox.BeginUpdate();
try
{
MyComboBox.Items.Clear();
MyComboBox.Items.AddRange(sortedList.ToArray());
}
finally
{
MyComboBox.EndUpdate();
}
}
sortedList contains around 280 items, and there are up to 53 ComboBox's which need to be repopulated. So it can take quite some time to repopulated all these controls (approx. 700ms on a high spec machine, 8000ms on a low spec machine) which is too slow for my requirements. I tried adding the sortedList to a new IList and that took around 1ms (on my high spec machine).
I need to get the repopulating of the ComboBox so that it takes significantly less time, ideally similar times to the IList but any performance increase would be good. So far I have been unable to find any way to improve the speed of the re-population.
Does anyone have any idea's on how I can reduce the time taken to re-populate the ComboBox?
Your problem could be that you have enabled the combobox's Sorted property. When this is enabled and you call AddRange the combobox sorts all those items, which if your items are already sorted, is unnecessary.
To prove my point, I created two combobox's that were populated using 10,000 sorted ints and AddRange. The only difference was that one combobox had the Sorted property enabled, the other one didn't. Here are the times in milliseconds of the resulting AddRange call
notSortedCombo: 5ms
sortedCombo: 1140ms
Could this be your problem? Could you have 53 comboboxes that have the sorted property enabled?
AddRange already calls BeginUpdate and EndUpdate under the hood, so you aren't gaining anything by calling it yourself.
This shaved a few milliseconds off of it for me:
public void Repopulate(IList<string> sortedList) {
comboBox1.BeginUpdate();
comboBox1.Items.Clear();
foreach (string item in sortedList) {
comboBox1.Items.Add(item);
}
comboBox1.EndUpdate();
}
The bigger problem is probably the design: 53 combo boxes is a lot of combo boxes to throw at a user — the user won't be able to interact with all 53 controls at once. You could get a little hacky and just populate the combo boxes with the visible value (1 item) and then populate the list as the control gets focused or in a background timer.
But consider reducing the number of controls on the screen. White space is considered a good thing.
You can improve performance by changing property FormattingEnabled of your combobox to False
The problem is that Visual Studio keeps it as True if you don't specify items "manually" in the designer, but add them programmatically.
Try, it solved issue for me
Regarding UI virtualization: Virtualization exists both for WPF (Example) and WinForms. But only some controls support it, like ListView, DataGridView, TreeView, etc. Those are the controls designed for bigger amounts of data. If possible you could switch to one of these controls.
Are all controls visible on screen at the same time? Maybe only updating the visible ones will help.
An alternative is to update the combo boxes asynchronously. If you're lucky enough to work with the new async/await stuff in .NET this is easily done. If you want to do it manually, you can just update a single combobox and schedule the update of the next combobox a few milliseconds in the future (use a timer or the TPL). This way the UI is at least still responsive while the update takes place.
Yet another way is to update the list only when the user focusses a certain ComboBox. There's usually no need to update the contents because the user will only see it when he uses the ComboBox.
You could also try to hide the combobox before you update its data. Or you could tell windows not to redraw the UI while modifying the contents (StackOverflow topic here). But I haven't tested whether this really improves the performance.
I developed an application that uses a DataGridView, and can have upwards of 500k rows in it. It is structured like this currently:
DataGridView.DataSource is a BindingSource
BindingSource.DataSource = AggregateBindingListView
AggBLV.SourceLists = {Lists of Data}
AggBLV.Sort("PropertyName")
AggBLV.ApplyFilter(Predicate)
...
The AggregateBindingListView is a collection that implements:
Component, IBindingListView, IList, IRaiseItemChangedEvents, ICancelAddNew, ITypedList
It is excellent piece of code developed by http://blogs.warwick.ac.uk/andrewdavey
Anyways. We've been using it for 4 years, and its become a performance bottleneck.
So here is my question:
I have a Collection of 500k items, and would like to bind them to a DataGridView. I need it to have Multi-Column sorting, Predicate Filtering, and A priority on performance.
We've just upgraded to C# 4.0.
Can I do better than what I have? I can post timing statistics and the such, but i'd need something for comparison.
You could try to use data virtualization as shown here. It's intended for WPF, but it could probably work in WinForms with minor adaptations. The idea is that instead of holding all the data at once in memory, you only load the necessary "pages" as needed, and unload them when they're not needed anymore.
What WPF control should I use when I need to have a spreadsheet/datagrid with MASSIVE amounts of columns and rows with data?
At most there will be over 26000 colums and rows.
Best Regards, Kjetil
I think you need to take a step back and ask "why?"
Is a human really going to scroll through 26k columns looking for information?
Perhaps this is an opportunity to create a better UI metaphor for analyzing all that data. I’d like to help provide a solution, but without knowing the business domain it would be a futile gesture.
I would give DataGrid a shot for this. WPF DataGrid has built-in support for virtualization. You can also try to set the VirtualizationMode property to Recycling, to see if it gives you better performance.
<DataGrid VirtualizingStackPanel.IsVirtualizing="True"
VirtualizingStackPanel.VirtualizationMode="Recycling">
</DataGrid>
Recently, I encountered similar situation with you. I resolved the issue with UI Virtualization and Data Virtualization.
What is the best for me is this, where famous Virtualization approaches, Paul McClean and Vincent Van Den Berghe, were upgraded.
I would recommend a DataGrid, but you can use whatever control you want as long as it's virtualizing.
The issue I see is that if you try to bind to a standard array or other collection as your backing to store the data you'll end up with 676 million cells allocated (26000 times 26000) even if there's little to no data in there. My suggestion for your backing store would be to create your own class based on SortedList<int, SortedList<int, object>> that only contains data for cells that are populated. You can then create a this[row, col] operator that gets or sets cell data and that can be used to bind into your control. Then only the cells that have data will be allocated.
Do not use a datagrid.
Use something that doesn't even attempt to work with such huge datasets. Write something that has two scrollbars - when you scroll, it retrieves/updates the subset of data you want to view and displays it in some useful way.
Dealing with such a large amount of data will require some careful and in-depth design and consideration. I suggest you write your own control. If you are unsure how to tackle this very complex task, I suggest you start from first principles, come up with a plan and give it a go - then come back with the questions you will likely have in managing such a huge amount of information.
Which control would be best for showing a huge (300.000+) list of filenames?
I've tried DataGridView, but it seems to be overkill and also slow.
Are there better alternatives?
None.
No USER will be able to handle a single list of 300.000+ entries in a meaningful way. Looks like your design is seriously flawed - do you really have to present the complete list?
Consider using a search box and let the users search the file names (use auto completion/suggestions like Google et.al.) or create a separate list for every starting letter (like most address books do). Or find another way to reduce the number of entries from which the user has to select.
The standard ListView control has a virtual mode designed specifically for your situation. I've used it with a million row list previously and it does the job well.
It is a true virtual mode. In other words memory allocation and list population time remains low regardless of the size of the overall list. This is unlike the DataGridView that really starts to slowdown and use memory on large lists.
To use virtual mode set:
VirtualListMode = true
VirtualListSize= 300000
(or whatever size your list currently is)
Then handle the RetrieveVirtualItem event to populate the list on demand from your list. You may also wnat/need to handle the CacheVirtualItems and SearchForVirtualItem events.
Set up pagination and restrict the number of rows displayed by the DataGrid. You can add a combobox to jump between pages. This is a standard solution.
also see this post https://stackoverflow.com/questions/2125963/need-help-in-gridview-and-table
Have you tried the ListView with report style? This is the control used by Windows natively in its file browsers.
Following on from gotch4's answer. Here is a good article from CodeProject on how to do paging with a DataGridView.
You might want to check out ObjectListView, specifically the VirtualObjectListView:
http://objectlistview.sourceforge.net/cs/index.html
I forget off hand what license it has been released under so you might want to look at that before using it in a commercial application.