Background
When writing front-end tests, we often need to wait until the web-application is done fetching data and updating the DOM before we want to interact with the page. With Selenium C#, this means a lot of explicit waits on the state of the page that is tailored to the specific scenario (maybe waiting for a loading indicator or a specific element to appear). However, most of the time this visual indicator is just a proxy for an async task like an HTTP request. Other solutions such as Protractor and Cypress have easy solutions for waiting for HTTP requests (this is the default in Protractor).
Question
One of the frameworks I maintain is written in C#, and I'm trying to find a solution to easily wait for any outstanding HTTP requests, rather than writing custom explicit waits against the DOM. Is there a solution for this? I'm open to using an additional open-source solution if needed.
I assumed I might need to set up a proxy so that I can manipulate and hook into HTTP requests. I looked into BrowserUp (continuation of the BrowserMobProxy project, which seems to no longer be maintained), but can't tell from docs if this sort of use case is possible or intended.
I remember years ago trying to solve this in Ruby. We settled on a hybrid JavaScript and Ruby solution. Each time an Ajax request got sent, we set a global JavaScript variable to true. When all pending requests had finished, we set it to false. We still had flaky tests. They were brittle and inconsistent even with some JavaScript gymnastics going on behind the scenes.
Even though Ajax (or background HTTP requests) might be complete, JavaScript still needs additional processing time to do something with the response. It was mere milliseconds, but remember that Selenium and your browser run in different threads — everything is a race condition. We kept getting intermittent test failures because the HTTP requests were done, but the browser was still in the process of evaluating element.innerHTML = response.responseText when Selenium would attempt to interact with an element that was supposed to be on screen after the request was complete. We still had to use explicit waits.
Basically, you are stuck with explicit waits in order to achieve stable tests. I've jumped through a lot of hoops over the years to get things working any other way. The only saving grace I've found is the Page Object Model Pattern, which at least centralizes this ugly code in one place for any particular use case.
So, yeah. The code is ugly. You need to use explicit waits. It turns out test code needs to be just as purposefully architected as the application code it tests.
Related
I have an c# asp.net management system with a button that calls a SQL Server Query to get 90,000 strings of text in multiple languages and categorized into sections. This in turn is sorted and 150 Binary files made before saving as a .ZIP and emailing the user with the results. The total time to process this and email the results is about 6 minutes. In this time the Web Page is sat waiting for the whole process to complete. I would like to be able to press the start process button and then allow this to work away in the background while I continue using the web management system, but I am unsure what is the most efficient method for doing this. I initially created an asmx file thinking this would work but the result is the same and so I am now looking at async and await. Can anyone give me any pointers on this and let me know if I am on the right track. I am currently not getting anything back to let me know the process has completed successfully as I can handle this by emailing the user to say something went wrong. The reason for this is the user could be on any number of pages.
There are probably a few ways to go about tackling this problem. Your options will vary based on what version of .NET you are using, so I'll not post code directly; however, the you can implement the concept I describe using ASMX web services, WCF, MVC, and so on.
Start-and-poll Approach
The classic response for this kind of problem is to implement a StartSomething() method and a GetProgress() method. A very-simple example of this approach using ASMX-based web services is presented here.
In the example, one service method is used to start a process on a background thread. Myself, I would change the example by having the start method return a value to the client to identify which background process was started, as you could feasibly have several going on at a time.
The client then can call a separate method to get progress updates, and continue polling until the process is complete.
There are a number of reasons why you should prefer to do lengthy background processing in a non-IIS service. I recommend using a Windows service to protect yourself from IIS somewhat-randomly restarting your application pool in the middle of a big job.
WebSockets
Another option worth some exploration on your part is to use WebSockets, which allow the server to contact a modern browser when the process is complete. The main advantage of this approach is that the client does not need to busily poll the service for updates. Its primary disadvantage is that WebSockets are new enough that there are still plenty of browsers that could not be clients for such a service.
Good luck!
I have a really long submit()-type function on one of my web pages that runs entirely on the server, and I'd like to display a progress bar to the client to show the, well, progress.
I'd be ok with updating it at intervals of like 20% so long as I can show them something.
Is this even possible? Maybe some kind of control with runat="server"? I'm kind of lost for ideas here.
It's possible, but it's quite a bit harder to do in a web based environment than in, for example, a desktop based environment.
What you'll have to do is submit a request to the server, have the server start the async task and then send a response back to the client. The client will then need to periodically poll the server (likely/ideally using AJAX) for updates. The server will want to, within the long running task's body, set a Session value (or use some other method of storing state) that can be accessed by the client's polling method.
It's nasty, and messy, and inefficient, so you wouldn't want to do this if there are going to be lots of users executing this.
Here is an example implementation by Microsoft. Note that this example uses UpdatePanel objects, ASP timers, etc. which make the code quite a bit simpler to write (and it's still not all that pretty) but these components are fairly "heavy". Using explicity AJAX calls, creating web methods rather than doing full postbacks, etc. will improve the performance quite a bit. As I said though, even in the best of cases, it's a performance nightmare. Don't do this if you have a lot of users or if this is an operation performed very much. If it's just for occasional use by a small percentage of admin users then that may not be a concern, and it does add a lot from the user's perspective.
I would take a look at .net 4.5's async and await.
Using Asynchronous Methods in ASP.NET MVC 4 -- (MVC example I know sorry)
Then check out this example using a progress bar
I am currently working on a project, this project is basically a web site which as its main function, runs a long calculation task (this task is made up of between 1-10 sub tasks) - (takes about 30-40 seconds to complete on average) and returns a result to the user, as it stands the code uses multiple threading in the site itself (which i don't really like the idea of), so the site continues to run after a button click, whilst this thread in the background continues the calculation.
I'd ideally like to move this calculation into a separate service, to stop the issues related to running threading in a web app. My idea is that it should be a separate service that runs every X seconds to see if any jobs are present in the database, if there are, run them, but i have little experience in writing a reoccurring windows service, and was wondering if anyone has any ideas on the matter, is this the right way to go about such a task, does anyone have any experience of doing something similar, and can anybody recommend any particular ways I should proceed?
I am working on a similar project. I was thinking about create a service, like you, but I didn't see the benefits in my case. I have to do some long calculations, and put it in the cache. I don't know if this is the best practice but what I did was to create a timer on global.asax, perform my calculations and put it into cache. When the request comes I give the last cached value.
I hope this help you.
In a situation like this I always prefer to create a tcp listener that accepts the requests from the clients. If I am using .net, my first choice would be wcf service with tcp or named pipe binding. Then when I receive a request, I start a thread to proceed the request. When the long running process finally ends, I return a callback to the client, which was a url that specified in the first request to the scheduler. IMHO this is the best way to do it.
However you may want to build your application on a proven framework like NServiceBus, RhinoBus, etc. You may implement the same business over them too.
I'm tasked to create a web application. I'm currently using c# & asp.net (mvc - but i doubt its relevant to the question) - am a rookie developer and somewhat new to .net.
Part of the logic in the application im building is to make requests to an external smsgateway by means of hitting a particular url with a request - either as part of a user-initiated action in the webapp (could be a couple of messages send) or as part of a scheduledtask run daily (could and will be several thousand message send).
In relation to a daily task, i am afraid that looping - say - 10.000 times in one thread (especially if im also to take action depending on the response of the request - like write to a db) is not the best strategy and that i could gain some performance/timesavings from some parallelization.
Ultimately i'm more afraid that thousands of users at the same time (very likely) will perform the action that triggers a request. With a naive implementation that spawns some kind of background thread (whatever its called) for each request i fear a scenario with hundreds/thousands of requests at once.
So if my assumptions are correct - how do i deal with this? do i have to manually spawn some appropriate number of new Thread()s and coordinate their work from a producer/consumer-like queue or is there some easy way?
Cheers
If you have to make 10,000 requests to a service then it means that the service's API is anemic - probably CRUD-based, designed as a thin wrapper over a database instead of an actual service.
A single "request" to a well-designed service should convey all of the information required to perform a single "unit of work" - in other words, those 10,000 requests could very likely be consolidated into one request, or at least a small handful of requests. This is especially important if requests are going to a remote server or may take a long time to complete (and 2-3 seconds is an extremely long time in computing).
If you do not have control over the service, if you do not have the ability to change the specification or the API - then I think you're going to find this very difficult. A single machine simply can't handle 10,000 outgoing connections at once; it will struggle with even a few hundred. You can try to parallelize this, but even if you achieve a tenfold increase in throughput, it's still going to take half an hour to complete, which is the kind of task you probably don't want running on a public-facing web site (but then, maybe you do, I don't know the specifics).
Perhaps you could be more specific about the environment, the architecture, and what it is you're trying to do?
In response to your update (possibly having thousands of users all performing an action at the same time that requires you to send one or two SMS messages for each):
This sounds like exactly the kind of scenario where you should be using Message Queuing. It's actually not too difficult to set up a solution using WCF. Some of the main reasons why one uses a message queue are:
There are a large number of messages to send;
The sending application cannot afford to send them synchronously or wait for any kind of response;
The messages must eventually be delivered.
And your requirements fit this like a glove. Since you're already on the Microsoft stack, I'd definitely recommend an asynchronous WCF service backed by MSMQ.
If you are working with SOAP, or some other type XML request, you may not have an issue dealing with the level of requests in a loop.
I set up something similar using a SOAP server with 4-5K requests with no problem...
A SOAP request to a web service (assuming .NET 2.0 and superior) looks something like this:
WebServiceProxyClient myclient = new WebServiceProxyClient();
myclient.SomeOperation(parameter1, parameter2);
myclient.Close();
I'm assuming that this code will will be embedded into your business logic that you will be trigger as part of the user initiated action, or as part of the scheduled task.
You don't need to do anything especial in your code to cope with a high volume of users. This will actually be a matter of scalling on your platform.
When you say 10.000 request, what do you mean? 10.000 request per second/minute/hour, this is your page hit per day, etc?
I'd also look into using an AsyncController, so that your site doesn't quickly become completely unusable.
I'll have an ASP.net page that creates some Excel Sheets and sends them to the user. The problem is, sometimes I get Http timeouts, presumably because the Request runs longer than executionTimeout (110 seconds per default).
I just wonder what my options are to prevent this, without wanting to generally increase the executionTimeout in web.config?
In PHP, set_time_limit exists which can be used in a function to extend its life, but I did not see anything like that in C#/ASP.net?
How do you handle long-running functions in ASP.net?
If you want to increase the execution timeout for this one request you can set
HttpContext.Current.Server.ScriptTimeout
But you still may have the problem of the client timing out which you can't reliably solve directly from the server. To get around that you could implement a "processing" page (like Rob suggests) that posts back until the response is ready. Or you might want to look into AJAX to do something similar.
I've not really had to face this issue too much yet myself, so please keep that in mind.
Is there not anyway you can run the process async and specify a callback method to occur once complete, and then keep the page in a "we are processing your request.." loop cycle. You could then open this up to add some nice UI enhancements as well.
Just kinda thinking out loud. That would probably be the sort of thing I would like to do :)