I have a requirement to produce a report screen for different financial periods. As this is quite a large data set with a lot of rules the process could take a long time to run (well over an hour for some of the reports to return).
What is the best way of handling this scenario inside MVC?
I am concerned about:
screen locking
performance
usability
the request timing out
Those are indeed valid concerns.
As some of the commenters have already pointed out: if the reports do not depend on input from the user, then you might want to generate the reports beforehand, say, on a nightly basis.
On the other hand, if the reports do depend on input from the user, you can circumvent your concerns in a number of ways, but you should at least split the operation into multiple steps:
Have a request from the browser kick off the process of generating the report. You could start a new thread and tell it to generate the report, or you could put a "Create report" message on a queue and have a service consume messages and generate reports. Whatever you do, make sure this first request finishes quickly. It should return some kind of identifier identifying the task just started. At this time, you can inform the user that the system is processing the request.
Use Ajax to repeated poll the server for completion of the report using the given identifier. Preferably, the process generating the report should report its progress and this information should be provided to the user via the Ajax polling. If you want to get fancy, you could use SignalR to notify the browser of progress.
Once the report is ready, return a link to the user where he/she can access the report.
Depending on how you implement this, the user may be able to close the browser, have a sip of coffee and come back to a completed report.
In case your app is running on Windows Server with IIS your ASP.Net code can create a record in db table which will mean that report should be created.
Then you can use Windows Service or Console App which might be running on the same server and constantly checking if there any new fields in the table. This Service would create a report and during creation it should update table field to indicate progress.
Your ASP.net page might be displaying progress bar, getting progress indication from db using ajax requests or simply refreshing the page every several seconds.
If you are running on Windows Azure cloud you might use WebWorker instead of Windows Service
For screen locking on your page you may use jquery Block-UI library
Related
We have a web application that is hosted in IIS. In our database that serves the application we have all kinds of different data values. We are trying to figure out a way to have an email sent to a client if a certain data value exists or exceeds a threshold value.
Generic Example:
Say we have a table that lists widgets and their 'in inventory' quantity. Every time someone sells a widget, this quantity value would be depleted. We want to send an email to the manager when the widget quantity gets below 5 and tell him to reorder more widgets.
We don't want to have sql triggers that check the quantity any time a 'depletion' transaction takes place. Instead, we want some type of background monitoring process that checks the level of the widgets on a timed basis. How can we accomplish this? Windows Service / WinForm application? Something built into IIS that will run ASP.net C# code?
Polling based monitoring should be your last resort. It uses too many resources for a simple task and most of the time it will only see that it's not the case to do anything. And it doesn't even scale when your data grows.
Instead, you should focus on the code that changes those values and act then, on in spot. And the check will also be lighter: only one item being checked not all, and only once, not every x seconds/minutes/hours/...
Apart from the architectural considerations, to answer your question, just as Jonathan said: anything that can read a database and send emails will do, but I'd consider a Windows Service for this job because that's what they were made for: background jobs running all the time, unrelated to the host users. You also get some extra benefits like automatic startup and recovery options.
Anything that can read the database and send an email could accomplish this - console app, winforms app, web app -- it doesn't really matter.
It may be more efficient to monitor when the values are changed (what changes them? A web application?) and have that application also send notifications
I have an c# asp.net management system with a button that calls a SQL Server Query to get 90,000 strings of text in multiple languages and categorized into sections. This in turn is sorted and 150 Binary files made before saving as a .ZIP and emailing the user with the results. The total time to process this and email the results is about 6 minutes. In this time the Web Page is sat waiting for the whole process to complete. I would like to be able to press the start process button and then allow this to work away in the background while I continue using the web management system, but I am unsure what is the most efficient method for doing this. I initially created an asmx file thinking this would work but the result is the same and so I am now looking at async and await. Can anyone give me any pointers on this and let me know if I am on the right track. I am currently not getting anything back to let me know the process has completed successfully as I can handle this by emailing the user to say something went wrong. The reason for this is the user could be on any number of pages.
There are probably a few ways to go about tackling this problem. Your options will vary based on what version of .NET you are using, so I'll not post code directly; however, the you can implement the concept I describe using ASMX web services, WCF, MVC, and so on.
Start-and-poll Approach
The classic response for this kind of problem is to implement a StartSomething() method and a GetProgress() method. A very-simple example of this approach using ASMX-based web services is presented here.
In the example, one service method is used to start a process on a background thread. Myself, I would change the example by having the start method return a value to the client to identify which background process was started, as you could feasibly have several going on at a time.
The client then can call a separate method to get progress updates, and continue polling until the process is complete.
There are a number of reasons why you should prefer to do lengthy background processing in a non-IIS service. I recommend using a Windows service to protect yourself from IIS somewhat-randomly restarting your application pool in the middle of a big job.
WebSockets
Another option worth some exploration on your part is to use WebSockets, which allow the server to contact a modern browser when the process is complete. The main advantage of this approach is that the client does not need to busily poll the service for updates. Its primary disadvantage is that WebSockets are new enough that there are still plenty of browsers that could not be clients for such a service.
Good luck!
I've been tasked with allowing a user to upload an excel file and validating the data within the file.
I've researched ways on how to validate the data before the user submits the excel file on the client side( in javascript ), but it looks like you can't do this.
Is seems that the only way is to post this file to the server and do the validation on the server.
This seems like it could be time intensive. As of right now we don't really know how big this files can be, so we really don't want to bog down our server.
Is there any good way to do this?
Also, I am not saving the excel file on the server I just care about the data in it.
This is a ASP .NET application.
There are tools to parse excel files in javascript, so it is doable, but you need to provide more info. What makes a valid file? Is there a particular format, or do you just want to see if it will open in excel?
This sort of validation definitely should be done server-side. Even if there is client-side validation, you can't trust it or assume that it ran properly and would want to re-validate server-side anyway.
If the file is potentially large and validation could take a long time (more than a few moments) then you'll want to perform this sort of task asynchronously. As a general architectural guideline, in the past I've done this as follows:
Have two database tables to hold the spreadsheet data. One for the data itself (record-level) and one for the overall status of the uploaded spreadsheet (file-level). Something like this:
Spreadsheets
----------
ID
User
Status
DateUploaded
DateProcessed
etc.
SpreadsheetRecords
----------
ID
SpreadsheetID
SomeValueFromTheData
TheNextValueFromTheData
etc.
ValidationMessage
User uploads the spreadsheet, which is converted to strongly-typed data and persisted somewhere. This happens in-line immediately in the handling of the request in the web application (synchronously). If this fails, the file itself is invalid to the point that it can't be run through the business logic and an error is displayed to the user. You can use any number of techniques/libraries/tools to convert spreadsheets into in-memory objects, depending on the format of the spreadsheet. (A Google search for "convert OpenXML to DataTable in C#" might be a place to start, for example.)
User is presented with a response indicating that the upload was successful and processing will begin. At this point all of the "records" in the spreadsheet are saved in a table and linked to a record in another table which maintains the status of the processing, the user associated with it, etc. Any time the user re-loads the page they just see whatever the current status of the processing is. This part depends heavily on the UI. What I had before was a page where the user can check the status of their previous uploads. They can upload in bulk and the system will just process each file in turn, for example.
A background process (Windows Service in my case) continually polls the Spreadsheets table for new records which are in a "pending" status and "processes" that data. This is where the "validation" takes place, which could mean just about anything depending on the business rules. For each record in SpreadsheetRecords perform the business logic and store the result (for example, if there's an error put that error in ValidationMessage).
When processing completes for all of the records in that data, update the record in Spreadsheets with the status. (Passed, Failed, etc.) At this point you can notify the user that processing as completed, whether via email or some notification on the website (something like Facebook's notification bar, for example).
When the user checks the page again to see the status of the processing, they can see the result of the processing including any specific error messages about specific records in the data.
This is all very high-level, of course. There are a lot of assumptions here that would need to be tweaked for your own setup/architecture. The main point is the asynchronous nature of the whole thing. It's entirely possible to do all of this synchronously as well, which just eliminates the steps of telling the user that it's begun and notifying them when it's complete. But synchronous processing for something like this runs the risk of timing out (or at least presenting a bad user experience) for long-running tasks.
My environment - C# 3.5 and ASP.NET 4.0 and VS 2010
Apologies - am a bit new to some of the concepts related to threading and Async methods.
My scenario is this:
My site will periodically make a couple of GET/POSTS to an external site and collect some data
This data will be cached in a central cache
The periodic action will happen once in about 5 minutes, and will happen for every new member who registers on my site. The querying for the member will stop based on certain conditions
The user does NOT need to be logged in for these periodic queries - they register on the site, and then off my async code goes - it keeps working 24/7 and messages the user once a while via email depending on certain trigger condition. So essentially it all should happen in the background regardless of whether the user is explicitly logged in or not.
Load Expected - I anticipate about 100 total running members a day (accounting for new members + old ones leaving/stopping).
the equation is ~ 100 visitors / day x 4 POST per fetch x 12 fetches / hour x 8 hours / day
In my mind - I'm running 100 threads a day, and each 'thread' wakes up once in 5 minutes and does some stuff. The threads will interact with a static central cache which is shared among all of them.
I've read some discussions on ThreadPools, AsyncPage etc - all a bit new territory. In my scenario what would you suggest? What's the best approach to doing this so it's efficient?
In your response I would appreciate if you mention specific classes/methods/links to use so I can chase this. Thanks a bunch!
You will not be able to do it with ASP.net as such, you will not be able to keep the "threads" running with any level of reliability. IIS could decide to restart the appication pool (I.E. the whole process) at any point in time. Really what you would need is some kind of windows service that runs and makes the requests. You could the use HttpWebRequest.BeginGetResponse method to make your calls. This will fire off the relevent delegate when the response comes back and .net will manage the threading.
Agreeing with Ben, I would not use threading in IIS with ASP.NET. It's not the same as using it in a desktop application.
If you're going to use some kind of polling or timed action, I recommend having a handler (.ashx) or asp.net page (aspx) that can take the request that you want to run in the background and return XML or JSON as a response. You can then set some javascript in your pages to do an AJAX request to that URI and get whatever data you need. That handler can do the server side operations that you need. This will let you run background processes and update the front-end for your users if need be, and will take advantage of the existing IIS thread pool, which you can scale to fit the traffic you're getting.
So, for instance
ajaxRequest.ashx : Processes "background" request, takes http POST/GET parameters.
myPage.aspx : your UI
someScript.js : javascript file with functions to call ajaxRequest.ashx from myPage.aspx (or any other page) when certain actions or intervals occur.
jQuery.js : No need to write all the AJAX code or event handlers yourself :)
You will need to create a separate Windows service(or a console app that runs using the Windows scheduler) to poll the remote server.
If you need to trigger requests based on user interation with your site, the best way is to use some kind of queuing system(eg MSMQ) that your service monitors.
I just posted the question how-to-determine-why-the-browser-keeps-trying-to-load-a-page and discovered that my problem is with Gravatar.
I also noticed that StackOverflow is suffering from the same outage.
Does anyone know of a graceful way to determine if Gravatar, or any third party website for that matter, is up or not, before trying to retrieve avatar icons from them?
This would eliminate the long page load and the never ending busy cursor ... I shouldn't say never ending ... it just takes a long time to go away and is very confusing to the user as they sit there and wait ... for nothing.
You can have a different process that is periodically checking the status of the site. Set a rule about what is down for you, for instance you could say: "ping time > 1500 ms = down". Have this process to leave a note in a database table or config file. Then you check this value on each page rendering at almost no cost.
Depending on how critical is this external site, you can do the check more or less often.
This process could be an out of the web stack program, or a page only accessible through localhost that gets executed via Scheduled Tasks or an ASP.NET facility like mentioned in the comments.
For Gravatar you can cache all theses images instead of taking them from their server everytime. Of course, if user change their icon, it might not refresh as fast as it would be if it were direct access to the main server but at least you do not have to request gravar server everytime.