I am working on a real time trace logging application in wpf.I am reading logged data from a UDP port and converting it in terms of my modal class.The thing i am worried is, consider if the user keeps the application open for a long period of time,the application will be using a large amount of memory.I display the log information in a scrollable list,so the user has the provision to scroll up to and get to a previous log.I'm looking for a design approach so that i can deliver the best results with the optimal use of memory.So which is the best design approach for this kind of application ?
"Real Time" mean as soon as data is available Application should pick up it and display. No other way.
You can consider something like cleanup of the already previewed logging information if this is appropriate from user perspectives and load historical data on demand.
Also one of the possible solutions is to optimize LogInformationdata model so entities which you are displaying would require less memory, this could be significant improvement considering that a lot of entries are displayed and each single saved byte may result in MegaBytes of saved memory, so please share some code of entities which are bound to UI and indicate which fields/properties really need to be displayed to end user
For some kind of data you can implement Lazy Loading and request data from DB/file system on demand. For instance when an user opening Details form for a particular LogInfo entry in the UI list you are requesting advanced information like full description and so on, so you do not need to keep it always in memory whilst user do not request it by opening "More Details" form
If DB calls is high cost for your Applicaiton you can store some information on the file system in serialized format and load it On Demand in Lazy Loading manner.
Really it is hard to suggest something concrete without of knowledge of the Use Cases and standard workflows. So please provide more details how this applicaiton is used by an user so more ideas coudl come in.
One idea that might work is to add a "listener" that would see if the program was altered, or after lets say 5 minutes of seeming idle a popup would ask if you want to continue. It would then reconnect once the user clicks "ok"
Would that work? I am assuming that the memory use is on the log server.
It seems like your have two much data to display. You can use UI Virtualization. VirtualizingStackPanel can do what you want by not creating UI elements for all your log lines untill the user scrolls to it. A example would be too long for stackoverflow. There are plenty of examples on the web.
http://bea.stollnitz.com/blog/?p=338
On the other hand if you are memory requirements are too high just because there is too much log data. Consider writing it to a database on disk.
Related
I have an application in which such a large amount of data is loaded at the beginning that the waiting time for the users is no longer justifiable.
At first only data is loaded to fill a listbox explorer, which serves as browser to load the remaining information when selecting the item. So much for the data model.
I now intend to maintain a local data source and only update the data that the user selects, but I have to deal with the question if I should keep the finished objects for the model or the raw data.
Has anyone played around with the different approaches and can say/link to what is best approach in terms of maintenance and performance? I work with .NET
I am doing a small application that queries a web service for in-game prices for items in a particular game. This games obviously has over 200 items in game (with their associated uint IDs), and has different item types (ore, combat, etc). In my application, I have a view that allows the user to specify for with item he wants to query for the price, and it has 2 comboboxes: one for item type and the 2nd one that will show items of that specific type (so when the first combobox changes, the second one shows the items associated to the selected item type).
Also, I do not have direct access to the game's database with all the item types, items and their associated IDs. I would have to replicate that information (that is available online) in a database of my own, or in an XML file, or another container of the sort.
Knowing that, my question is what would be the best: loading the whole database (or parsing the whole XML file) into a List<GameItem> at the opening of the application, or querying the database (or parsing a part of the XML file) each time the user changes the item type combobox? If I do the whole loading at the beginning of the application, maybe I would run into the application taking A LOT of memory for nothing, but on the other hand if I query the database (or parse the XML file) each time the user changes the item type combobox, maybe there would be a problem where there would be a "delay" in the application each time he would do that operation.
I would start an asynchronous method after starting the app, where it loads the game items. This way it won't also block the UI while user do what ever it do in your app. I've done this in my app where user is reading an ebook and it loads 200 books at the same time. This way user is able to continue it reading etc while it load books in a background.
First thing you want to do is establish a high-level interface that doesn't bother with or mention these details so that you can change your mind later if necessary and change as few things as possible in response. Make the interface focus on what it should do rather than how it should do it. Hide away all those 'hows', make them private.
Optimization is best applied in hindsight, with profilers and measurements in your hand, and code that can be optimized without being too intrusive/invasive and creating cascading breakages throughout your codebase (by being tucked under a good interface).
Second, keep in mind that a million 32-bit floating point variables just takes 4 megabytes of RAM. I came originally from a time where 4 megabytes was considered a massive amount of memory. Today we're talking pennies. 200 items is typically nowhere near enough data to concern yourself with the added expense and difficulty of implementing a disk indexing structure unless each item stores like a million sub-elements each.
So unless you're working with exceptionally massive items, I'd suggest starting with a basic solution of loading them into memory on startup.
A bigger concern for your case and scale if there's store logic involved might be security and atomicity much more than performance, to ensure that item transactions are either completed 100% successfully or fails/rolls back 100% as a whole (never half-finished). You might also want to periodically write to disk anyway to make sure that you don't lose the data in the case of an unexpected server shutdown, but you don't necessarily have to be using that file structure for anything more than a safety backup. Though I wasn't clear if you were handling that store-side logic or just providing a convenient client for the customers. If the latter, you can forget about this stuff.
I'm planning on creating a live analytics's page for my website - A bit like Google Analytic but will real live data which will change as new users load a page on my site etc.
The main site is/will be written using Asp.Net/C# as the back end with a MS SQL database and the front end will support things like JavaScript (JQuery), CSS3, HTML5 (If required).
I was wondering what methods can I use to have the live analytic in terms of; How to get the data onto the analytic's page, what efficient graphing can I use, and storing the data with fast input/output.
The first thing that came to my mind is to use Node.js - Could I use this to achieve a live analytic's page? Is a good idea? Are there any better alternatives? Any drawbacks with this?
Would I need a C# Application running on a server to use Node.js to send/receive all the data to and from the website?
Would using a MS SQL database be fast enough? Would I need to store all the data live, or could I store it in chunks every x amount of seconds/minutes? (Which would be more efficient?)
This illustrates my initial thoughts on the matter -
Edit:
I'm going to be using this system over multiple sites, I could be getting 10 hits at a time to around 1,000,000 (Highly unlikely, but still possible). I want to be able to scale this system and adapt it to the environment it's in.
It really depends on how "real time" the realtime data needs to be. For example, I made this recently:
http://www.reed.co.uk/labs/realtime/
Which shows job applications coming into the system. Obviously there is way too much going on during busy periods to actually be querying the main database in realtime - so, what we do is query a sliding "window" and cache it on the server - this is a chunk of the last 5minutes worth of events.
We then play this back to the user as is it's happening "now". having a little latency as part of a SLA (wherein the users don't really care) can make the whole system vastly more scalable.
[EDIT- further explanation]
The data is just retrieved from a basic stored procedure call - naturally, a big system like reed has hundreds of transactions/second - so we cant keep hitting the main cluster, for every user.
All we do, is make sure we have a current window, in this case the last 5min of data cached on the server. When a client comes to the site, we get that last 5min of data, and play it back like it's happening right now - the end user is none-the-wiser - but what it means is that all clients are reading off the cache. Once the cache is 5min old, we invalidate it, and start again. This means a max of 1 DB hit, every five min - thus making teh system vastly more scalable (not that it really needs to be - as it's just for fun, really)
Just so you are aware Google analytics's already offers live user tracking. when inside the dashboard of a site on Google analytics's. click the home button on the top bar, and then the real time button on the left bar. Considering the design work and quality of this service, it seems this may be a better option then to attempt to recreate its service. If you do choose to proceed to create your own, then you can at least use their services as a benchmark for the desired features.
Using Api's like the googles charting API https://developers.google.com/chart/ would be a good approach to displaying the output of your stored data, with decreased development time. If you provide more information on the number of hits you exspect, and the scale of the server this software will be hosted, then it will be easier to give you answers to the speed questions.
Situation: multiple front-ends (e.g. Silverlight, ASP) sharing a single back-end server (WCF RIA or other web service).
I am looking for a standard to prevent multiple people from editing the same form. I understand that this is not an easy topic, but requirements are requirements.
Previously I used the DB last modified date against the submitted data and give a warning or error if the data was modified since it was loaded. The initial system simply overrode the data without any warning. The problem is that I have a new requirement to prevent both these situations. There will be many UIs, so a locking system might be a challenge, and there is obviously no guarantee that the client will not close the window/browser in the middle of an edit.
I would appreciate any help.
If I'm correct, it seems what you are talking about is a form of check-out/edit/check-in style workflow. You want when one user is editing a record, no other users can even begin to edit the same record.
This is a form of pessimistic concurrency. Many web and data access frameworks have support for (the related) optimistic concurrency - that is, they will tell you that someone else already changed the record when you tried to save. Optimistic has no notion of locking, really - it makes sure that no other user saved between the time you fetched and the time you save.
What you want is not an easy requirement over the web, since the server really has no way to enforce the check-in when a user aborts an edit (say, by closing the browser). I'm not aware of any frameworks that handle this in general.
Basically what you need is to hold checkout information on the server. A user process when editing would need to request a checkout, and the server would grant/deny this based on what they are checking out. The server would also have to hold the information that the resource is checked out. When a user saves the server releases the lock and allows a new checkout when requested. The problem comes when a user aborts the edit - if it's through the UI, no problem... just tell the server to release the lock.
But if it is through closing the browser, powering off the machine, etc then you have an orphaned lock. Most people solve this one of two ways:
1. A timeout. The lock will eventually be released. The upside here is that it is fairly easy and reliable. The downsides are that the record is locked for a while where it's not really in edit. And, you must make your timeout long enough that if the user takes a really, really long time to save they don't get an error because the lock timed out (and they have to start over).
2. A heartbeat. The user has a periodic ping back to the server to say "yep, still editing". This is basically the timeout option from #1, but with a really short timeout that can be refreshed on demand. The upside is that you can make it arbitrarily short. The downside is increased complexity and network usage.
Checkin/checkout tokens are really not that hard to implement if you already have a transacted persistant store (like a DB): the hard part is integrating it into your user experience.
I have a web app that will send out daily, weekly email updates depending on the user permissions and their alert settings (daily, weekly, monthly, or none).
Each email to an account (which would have multiple users) requires a few DB calls and calculations. Thus making these daily/weekly emails pretty expensive as the number of users increase.
Are there any general tips on writing these services? I'm looking for some architecture tips or patterns and not really topics like email deliverability.
I would cache the data before the processing time, if you are having to handle very large sets of information, so that the DB 'calculations' can be omitted from the processing cycle at the specific times. Effectively break the processing up so that the DB intensive stuff is done a bit before the scheduled processing of the information. When it comes time to actually send these emails out, I would imagine you can process a very large volume quickly without a whole lot of tuning up front. Granted, I also don't know what kind of volume we're talking about here.
You might also thread the application so that your processing data is further split into logical chunks to reduce the overall amount of data that has to be processed all at once, depending on your situation it might streamline things, granted, I normally don't recommend getting into threading unless there is a good reason to, and you may have one. At the very least, use a background worker type of threaded process and fire off a few dependent on how you segment your data.
When handling exceptions, remember to now let those bring your processing down, handle them through logging of some sort or notification and then move on, you wouldn't want an error to mess things up for further processing, I'm sure you probably planned for that though.
Also, send your emails asynchronously so they don't block processing, it's probably an obvious observance but sometimes little things like that are overlooked and can create quite the bottleneck when sending out lots of emails.
Lastly, test it with a reasonable load beforehand, and shoot for well over capacity.
You may want to check out sql reporting services.
You may have to translate the current setup into the sql reporting format but in return you'll get a whole administrative interface for scheduling the report generation, allowing users to modify the report inputs, caching historical/current reports, and the ability for users to manage their own email subscriptions.
http://msdn.microsoft.com/en-us/library/ms160334.aspx