We Keep Coding

Request timeout error while processing long tasks

I have an c# asp.net management system with a button that calls a SQL Server Query to get 90,000 strings of text in multiple languages and categorized into sections. This in turn is sorted and 150 Binary files made before saving as a .ZIP and emailing the user with the results. The total time to process this and email the results is about 6 minutes. In this time the Web Page is sat waiting for the whole process to complete. I would like to be able to press the start process button and then allow this to work away in the background while I continue using the web management system, but I am unsure what is the most efficient method for doing this. I initially created an asmx file thinking this would work but the result is the same and so I am now looking at async and await. Can anyone give me any pointers on this and let me know if I am on the right track. I am currently not getting anything back to let me know the process has completed successfully as I can handle this by emailing the user to say something went wrong. The reason for this is the user could be on any number of pages.

There are probably a few ways to go about tackling this problem. Your options will vary based on what version of .NET you are using, so I'll not post code directly; however, the you can implement the concept I describe using ASMX web services, WCF, MVC, and so on.
Start-and-poll Approach
The classic response for this kind of problem is to implement a StartSomething() method and a GetProgress() method. A very-simple example of this approach using ASMX-based web services is presented here.
In the example, one service method is used to start a process on a background thread. Myself, I would change the example by having the start method return a value to the client to identify which background process was started, as you could feasibly have several going on at a time.
The client then can call a separate method to get progress updates, and continue polling until the process is complete.
There are a number of reasons why you should prefer to do lengthy background processing in a non-IIS service. I recommend using a Windows service to protect yourself from IIS somewhat-randomly restarting your application pool in the middle of a big job.
WebSockets
Another option worth some exploration on your part is to use WebSockets, which allow the server to contact a modern browser when the process is complete. The main advantage of this approach is that the client does not need to busily poll the service for updates. Its primary disadvantage is that WebSockets are new enough that there are still plenty of browsers that could not be clients for such a service.
Good luck!

Potential performance issues when calling web service from SQL CLR

I have a need to call external web service on certain events in my application. I don't want to modify my application and make any dependencies of that external web service. So, I need to think of a way to do this with some sort of external component.
One possible approach is that I make database view which will get filled up when some events in my application occurs. Then I will set up trigger on that view which will call CLR function. In that CLR function I will make call to external web service. By doing this, I will get "real-time" integration which is good. But, this approach has downsides. Major one is that it seems that calling web service from CLR is not a good idea since it will block main SQL thread (?!) until CLR receives some answer.
Until now, I have only found that setting this property will help with performance issues:
System.Net.ServicePointManager.DefaultConnectionLimit = 9999
More about it you can find here.
Now, since you know my needs (that is real-time or at least close-to-real-time integration without any calls from my application to exteranl web service) is there some better way to do it?
One other approach I can think of is having some service which will periodically check for changes in my DB that needs to trigger calls to external web service. Once this service detects such change, it will call web service and transfer data. This is not true real-time integration of course. I must admit that, except for performance issues, I like having triggers and CLR much more since it guarantees real-time integration and has no affect on my application whatsoever.

I am not sure that I would agree with the design of moving the web-service call to a database. However, I am sure there are reasons as to why you wouldn't want to change the application.
Here are a couple of options that you can try -
1) Instead of a database, and CLR making web-service calls, use a message queue. NServiceBus is a good choice for passing event occurrences as message, which can trigger this call
2) If you are stuck with using SQL server to store the events, look at SQL server Service broker

Update progress bar from codebehind

I have a really long submit()-type function on one of my web pages that runs entirely on the server, and I'd like to display a progress bar to the client to show the, well, progress.
I'd be ok with updating it at intervals of like 20% so long as I can show them something.
Is this even possible? Maybe some kind of control with runat="server"? I'm kind of lost for ideas here.

It's possible, but it's quite a bit harder to do in a web based environment than in, for example, a desktop based environment.
What you'll have to do is submit a request to the server, have the server start the async task and then send a response back to the client. The client will then need to periodically poll the server (likely/ideally using AJAX) for updates. The server will want to, within the long running task's body, set a Session value (or use some other method of storing state) that can be accessed by the client's polling method.
It's nasty, and messy, and inefficient, so you wouldn't want to do this if there are going to be lots of users executing this.
Here is an example implementation by Microsoft. Note that this example uses UpdatePanel objects, ASP timers, etc. which make the code quite a bit simpler to write (and it's still not all that pretty) but these components are fairly "heavy". Using explicity AJAX calls, creating web methods rather than doing full postbacks, etc. will improve the performance quite a bit. As I said though, even in the best of cases, it's a performance nightmare. Don't do this if you have a lot of users or if this is an operation performed very much. If it's just for occasional use by a small percentage of admin users then that may not be a concern, and it does add a lot from the user's perspective.

I would take a look at .net 4.5's async and await.
Using Asynchronous Methods in ASP.NET MVC 4 -- (MVC example I know sorry)
Then check out this example using a progress bar

Networked Client-Server application advice

I'm trying to design an application that will allow two users over a network to play the prisoner's
dilemma game (http://en.wikipedia.org/wiki/Prisoner%27s_dilemma).
Basically, this involves:
Game starts (Round 1).
Player 1 chooses to either cooperate, or betray.
Player 2 chooses to either cooperate, or betray.
Each other's decisions are then displayed
Round 2 begins
Etc.
I've done some thinking and searching and I think the application should contain the following:
Server class that accepts incoming tcp/ip connections
Gui clients (Seperate program)
For each connection (maximum 2) the server will create a new ConnectedClient class. This class will contain the details of the two player's machines/identities.
The Server class and the ConnectedClient class will connect/subscribe events to each so they can alert one another when e.g. server instruction ready to transmit to players, or players have transmitted their inputs to the server.
I'm not sure whether the best approch is to use a single thread to do or the work, or have it multithreaded. Single threaded would obviously be easier, but I'm not sure whether it is possible for this situation - I've never made a application before requiring TCP/IP connections, and I'm not sure if you can listen for two incoming connections on one thread.
I've found the following guide online, but it seems that it opens two clients on two threads, and they communicate directly to each other - bypassing the server (which I will need to control the game logic): http://www.codeproject.com/Articles/429144/Simple-Instant-Messenger-with-SSL-Encryption-in-Cs
I'm very interested and would be grateful on any advice on how you would go about implementing the application (mainly the server class).
I hope I've explained my intentions clearly. Thanks in advance.

My 1st advice would be to forget about TCP/IP and sockets here. You definitely can do it with that technology stack, but you would also get a lot of headache implementing all the things you want. And the reason is it too low level technology for such a class of tasks. I would go with tcp/ip and sockets only for academic interest, or if I need tremendous control over the communication, or if I have very high performance requirements.
So, my 2nd advice would be to look at WCF technology. Don't be afraid if you haven't used it before. It's not that difficult. And if you were ready to use sockets for your app, you can handle WCF definitely. For you task you can create basic communication withing 1-2 hours from scratch using any WCF tutorial.
So, I would create a server WCF service which will have some API functions containing your business logic. It can be hosted within a windows service, IIS, or even a console application.
And your clients would use that WCF service, calling their functions like it's functions from another local class in your project. WCF could also help you do the events which you want (it's a little bit more advanced topic though). And you can even forget about threading here, most of the things will be working out of the box.

First, as others have said, separate your game logic as much as you can, so the basic funcionality won't depend too much on your comunication infrastructure.
For the communication, WCF can handle the task. You can make your clients send a request to a service hosted in IIS, doing some kind of identification/authentication, and open a Duplex channel from where your service can push results and comunicate the start of new rounds.
Once one client connects, it waits for another. When it happens, it notifies the first client using the Duplex Channel callback and awaits for its choice. Then it asks the second user, awaits for its response. When it comes, it notifies the result to both and restarts the game.
Going a little bit deeper in the implementation:
You will have a service with some operations (like Register, PushDecision, more if needed). You will also define a callback interface, with the operations your service will need to push to the client (NotifyResult, RequestDecision, again, these are examples). You then create proxies for your clients that maps to your service operations and implement the callback operations in a way it expose events and raise them when the service pushs messages.
A use case:
Client A creates the proxy, calls Register on the server. The server receives the call, register the cilent and saves the callback object in a state. A duplex connection will be established. What does that mean? It means that (if you using the PollingDuplexBinding, as you probably will) from now on the proxy object in Client A will be doing long poll requests to the server, checking if there is a callback message. If there isnt, then it long polls again. If there is, it calls the method of the callback in the proxy passing the data the server has push. The callback method in the proxy will tipically raise an event, or execute a delegate, its up to you to choose.
Client B connects (calling Register), does the same as it did to A, and the server, noticing that two clients are connected, requests a response to A through its saved callback. This can happen during the processing of the B's Register call, or it can be triggered to execute in a new thread (or better, run in the ThreadPool or start a new Task) in B's register call.
Client A will receive the server callback requesting its choice. It can then notify the user and get the choice through the UI. A new call is made to the server (PushDecision, for example). The server receives Client A choice, asks B the same way. Once it has both responses, it calculates the result and pushes the outcome to the Clients.
An advantage of using Duplex Channels with PollingDuplex with WPF is that, as it uses long polling, there will be no need to use other ports than 80.
This is by no means a final implementation, is just a little guide to give you some ideas instead of just giving you some misty advices. Of course, there may be a bunch of other ways of doing that with WCF.
We can first assume that the application can handle only two users per time and then, if you want, you can scale up, making your service keep some form of state with a mapping table with locked access, as another example.
Some thoughts on WCF: There is an easy path to start developing with WCF using the Visual Studio tools (svcutil) but I don't like that approach. You don't "get to know" the WCF infrastructure well, you become tied to the verbose magic with which it generates your proxies, and you lose flexibility, especially in special scenarios, like Duplex polling that you may want to use.
The other way, that is to manually create your services and your proxies, is not that hard, though, and gets very interesting once you realize what you can do with it. Related to that I can give you one advice: do everything you can to make your proxy operations use Task-based Async Pattern (you can see the different ways to implement proxy operations here). This will make your code much cleaner and straight forward when combined with the new C# async/await keywords and your UI will be a joy to implement.
I can recommend some links to get you started. Some of them are old, but very didactic.
There used to be a fantastic article of WCF in this link but it seems to be now offline. Luckily, I found the content available there in a file in this link.
This one covers your hosting options.
Topics on WCF infrastructure: link
Topics on Duplex Services: link link link
Topics on Task-based Async Pattern: link link link

Well one advice I can give you if you insist that all user communicate through server and you want your application to scale:
Separate your logic (by understanding each part of the logic you want to build on the server)
Make your classes such that it can handle multiple users per transaction
Use IOCP whenever possible
it depends on the structure of your application if you need authentication and user profiles etc .. you may introduce the WCF or whatever web-service for user and hide your actual action in the background (this will cost you performance but it might be the only suitable solution you have) , so you may have your authentication framework at the top of your server logic, and a pipelined action logic in the behind .. i.e. users get authenticated to be able to access the services presented by the server, but these services pipeline all users and handle as many as possible simultaneously — if you don't need authentication then you might directly communicate to your server logic and you may use completion ports on user's request - a lot of work to be done here.

What is the best Design/Way to keep user connected?

Am working on a POC for self learning in which I want to keep my user connected in LIVE pattern. For example, A game in which 4 user can play at a time , here I need to keep this user connected to my game .
M not good at Socket type of programming and love to do that in Services way.What i wana know is 'What is the best way of doing this'. According to my initial Brain Storming, I have decided that I will use SilverLight(In Browser Or Out of Browser) as Front end [I have no issue in that].
I m more concern in back end.
Either I make an handler or make a WCF service or use full duplex service and use pooling mechanism for that. As a random thought I come up with a Timer type logic that will fire every after 10 seconds at clients end and get status like
Is it now Its turn to roll a dice
Home many user left (in case if
some of them left)
What are connected user status in
game like there score/points ect and
update
game view according to this at his end
Kindly place your best answers here that will help me to learn this.
Regards and thanks in Advance
EDIT:
Starting Bounty as i need more feedback.
FH

Fasih,
Since HTTP is stateless, you cannot make 2 way communication from your code. But there is a workaround if you are using AJAX. As you said timer is a one way. Another one is called COMET or Reverse AJAX.
This simulates the two way communication without relying on timer. To accomplish this you have to make a long running AJAX calls to the server, and the call is only returned if there is a change to update. Assume simple web chat scenario. 2 users make a long AJAX calls to the server, and both are polling the common medium (say DB), if the user1 sends some text, it will get updated and the user 2's waiting AJAX call pick up the text and return. And again both users will make a long running call to listen each other.
As you already decided to go ahead with silverlight, you can use WCF duplex channel to emulate the 2 way communication. As i explained earlier, dont go with timer logic. Its not instant if you are polling the server for 10 sec (anything can happen in a game within 10 sec), and it will increase the server load if you poll for each second.
So avoid timer logic and use long running AJAX calls.
If you are looking for options other than WCF duplex channels, HTML5 web sockets and COMETs are other ways to go.
check out this post for browsers supporting web socokets.

Basically it is a question of being able to push data to the client from the server.
So I was thinking is a subscriber publisher architecture, you can create a queue(in a db table for ex) on the server for each of users that are connected, and have an ajax calling a web service that will pull data from the table.
Every message should be encapuslated as a command for the client. So you can use different messages for each operation that the client is capable of. {command:display,text:"user blah blah has logged in"} another command could look like {command:rolldice, text:"roll the dice"}
Let me know what you think...

If you've decided to go for WCF then I would suggest you to use callbacks.
More info here: WCF: Working with One-Way Calls, Callbacks, And Events
-- Pavel