I have an application in which such a large amount of data is loaded at the beginning that the waiting time for the users is no longer justifiable.
At first only data is loaded to fill a listbox explorer, which serves as browser to load the remaining information when selecting the item. So much for the data model.
I now intend to maintain a local data source and only update the data that the user selects, but I have to deal with the question if I should keep the finished objects for the model or the raw data.
Has anyone played around with the different approaches and can say/link to what is best approach in terms of maintenance and performance? I work with .NET
Related
I am doing a small application that queries a web service for in-game prices for items in a particular game. This games obviously has over 200 items in game (with their associated uint IDs), and has different item types (ore, combat, etc). In my application, I have a view that allows the user to specify for with item he wants to query for the price, and it has 2 comboboxes: one for item type and the 2nd one that will show items of that specific type (so when the first combobox changes, the second one shows the items associated to the selected item type).
Also, I do not have direct access to the game's database with all the item types, items and their associated IDs. I would have to replicate that information (that is available online) in a database of my own, or in an XML file, or another container of the sort.
Knowing that, my question is what would be the best: loading the whole database (or parsing the whole XML file) into a List<GameItem> at the opening of the application, or querying the database (or parsing a part of the XML file) each time the user changes the item type combobox? If I do the whole loading at the beginning of the application, maybe I would run into the application taking A LOT of memory for nothing, but on the other hand if I query the database (or parse the XML file) each time the user changes the item type combobox, maybe there would be a problem where there would be a "delay" in the application each time he would do that operation.
I would start an asynchronous method after starting the app, where it loads the game items. This way it won't also block the UI while user do what ever it do in your app. I've done this in my app where user is reading an ebook and it loads 200 books at the same time. This way user is able to continue it reading etc while it load books in a background.
First thing you want to do is establish a high-level interface that doesn't bother with or mention these details so that you can change your mind later if necessary and change as few things as possible in response. Make the interface focus on what it should do rather than how it should do it. Hide away all those 'hows', make them private.
Optimization is best applied in hindsight, with profilers and measurements in your hand, and code that can be optimized without being too intrusive/invasive and creating cascading breakages throughout your codebase (by being tucked under a good interface).
Second, keep in mind that a million 32-bit floating point variables just takes 4 megabytes of RAM. I came originally from a time where 4 megabytes was considered a massive amount of memory. Today we're talking pennies. 200 items is typically nowhere near enough data to concern yourself with the added expense and difficulty of implementing a disk indexing structure unless each item stores like a million sub-elements each.
So unless you're working with exceptionally massive items, I'd suggest starting with a basic solution of loading them into memory on startup.
A bigger concern for your case and scale if there's store logic involved might be security and atomicity much more than performance, to ensure that item transactions are either completed 100% successfully or fails/rolls back 100% as a whole (never half-finished). You might also want to periodically write to disk anyway to make sure that you don't lose the data in the case of an unexpected server shutdown, but you don't necessarily have to be using that file structure for anything more than a safety backup. Though I wasn't clear if you were handling that store-side logic or just providing a convenient client for the customers. If the latter, you can forget about this stuff.
I have a web project in asp.net/C#/Mysql where the data can be up to some 30,000 rows of data to process.
Its a reporting tool, and I have to show statistics like counts and sum at several levels.
I want to know which would be the better way to go around this.
I can filter my data to limited columns though which I can query.
Now, Is is a good way to get the data (all rows) to my application on load and whenever user queries I can filter that data and do my calculations in the code and show my statistics.
or I can have a stored procedure do all my calculations and every time user queries I can call the stored procedure and get my statistics.
Thanks
Databases are optimized to do this kind of data manipulation. And since you reduce network load as well i would vote for the second option.
Another possibility is to consider some kind of OLAP solution where transactional data is already consolidated into smaller chunks of data which in turn can be easily queried.
I would definetly go with the second option. You are talking about a web application, if You want to get all the data at load, then you must store it somewhere to preserve it during postbacks. If you chose to store it in a session state, you will end up consuming the Web server memory, since you have more than one user accessing your site.
If you store it in view state, then you will end up in a very big client response,which will make your page very slow to load on the client side, and will cause network traffic.
Option 2 is the best, because stored procedures are precompiled, which means they are much better in terms of performance. You will also reduce network traffic.
I am currently storing about a million rows into a DataTable,
this takes about 600 Mb RAM.
So, when you run the application it with store a 1 000 000 rows in a DataTable and display that on a GridView. When you close the Application this will obviously clear the memory.
I will Like to give the user this option, in other words when the user tries to close the form he will be asked whether he wants to clear memory or not.
The reason for doing this is that does not have to wait for the data to be read into the Datatable each time he runs the application.
.... I am fairly new to C#, so I apologize if this is an in appropriate question here.
You can only keep something in memory if there is a running application / process with that data in memory - if your application closes (and there is only 1 instance of your application open with that data in memory) then the memory will be released. If you want to keep it in memory then your only options would be
Don't close your application, e.g. just hide the window instead (generally a very bad idea that your users won't thank you for)
Keep that data in another separate process that stays running even when your application isn't, for example a Windows service (again possibly not something that your users are going to thank you for, unless you are actually trying to develop a form of service as opposed to a Windows application).
A much better idea would be change the way your application works with large data sets so that it doesn't need to keep it all in memory, for example if you display all of that data in a large list view then use a virtual list view so that only those rows currently being displayed to the user are loaded into memory. You can then store your data either in an external database or a semi-permanent file (such as a SQLite database).
Or just load the data set each time like you currently are.
I suggest you keep it in a file or database instead of in memory. You can query some data that you want to use during runtime. For instance, 20 or 50 records at a time.
It does not make sense if you display 1M records in datagrid at once and I bet that even you will not look through them.
Try compressing a particular region of data(set of objects corresponds to rows) and extract when needed.
Refer,
Compressing Object in Dot Net
I am working on a real time trace logging application in wpf.I am reading logged data from a UDP port and converting it in terms of my modal class.The thing i am worried is, consider if the user keeps the application open for a long period of time,the application will be using a large amount of memory.I display the log information in a scrollable list,so the user has the provision to scroll up to and get to a previous log.I'm looking for a design approach so that i can deliver the best results with the optimal use of memory.So which is the best design approach for this kind of application ?
"Real Time" mean as soon as data is available Application should pick up it and display. No other way.
You can consider something like cleanup of the already previewed logging information if this is appropriate from user perspectives and load historical data on demand.
Also one of the possible solutions is to optimize LogInformationdata model so entities which you are displaying would require less memory, this could be significant improvement considering that a lot of entries are displayed and each single saved byte may result in MegaBytes of saved memory, so please share some code of entities which are bound to UI and indicate which fields/properties really need to be displayed to end user
For some kind of data you can implement Lazy Loading and request data from DB/file system on demand. For instance when an user opening Details form for a particular LogInfo entry in the UI list you are requesting advanced information like full description and so on, so you do not need to keep it always in memory whilst user do not request it by opening "More Details" form
If DB calls is high cost for your Applicaiton you can store some information on the file system in serialized format and load it On Demand in Lazy Loading manner.
Really it is hard to suggest something concrete without of knowledge of the Use Cases and standard workflows. So please provide more details how this applicaiton is used by an user so more ideas coudl come in.
One idea that might work is to add a "listener" that would see if the program was altered, or after lets say 5 minutes of seeming idle a popup would ask if you want to continue. It would then reconnect once the user clicks "ok"
Would that work? I am assuming that the memory use is on the log server.
It seems like your have two much data to display. You can use UI Virtualization. VirtualizingStackPanel can do what you want by not creating UI elements for all your log lines untill the user scrolls to it. A example would be too long for stackoverflow. There are plenty of examples on the web.
http://bea.stollnitz.com/blog/?p=338
On the other hand if you are memory requirements are too high just because there is too much log data. Consider writing it to a database on disk.
Requirement
.NET Windows application using C# interacts with Oracle DB for retrieving and saving data
Issue
With huge volume of data, the performance is slow and the memory usage is high, the application displays the entire data in the screen. Response time is high due to database call and client side data processing
Proposed Solution
Using pagination (from Oracle DB) to display partial data in the screen, response time of the application will be faster; however, it will make DB calls for each page.
We are looking at a solution to get the 1st page data from DB and start the application, after which there will be a background job to get the rest of the data from DB to a local XML DB. So, in case of next page, the data will be loaded from XML instead of making a DB call.
Is this design possible?
Is synchronization possible between local XML DB and the Oracle DB?
Personally I am not sure you really want to go that far, as synchronoization, and overall disk IO could be very "interesting" at best.
Typically what I have found to be good in the past if you REALLY must have "pre-fetched" records for more of the result set is that you can cache say the next 2 and previous 2 pages in memory, that way the users transition is smooth, and after you navigate the page, a backend thread will go out and pre-fetch the next page so taht you have it.
Otherwise, if you do what you are talking about, you are only deferring the performance impacts and introducing data synchronization and other issues.