Scaling up of Azure Webjobs - c#

I have an azure web app with 2 webjobs. I need to scale up just one webjob. Is there any way of scaling it up/out independent of the site and the other webjob?

Assuming your WebJobs are continuous,
From https://github.com/projectkudu/kudu/wiki/WebJobs-API
If a continuous job is set as singleton it'll run only on a single instance opposed to running on all instances. By default, it runs on all instances.
To set a continuous job as singleton during deployment (without the need for
the REST API) you can simply create a file called settings.job with the content: { "is_singleton": true } and put it at the root of the (specific) WebJob directory.
WebJobs that you'd like to stay on one instance, set as singleton.
The rest of them will scale automagically with your App Service Plan.
Triggered WebJobs only run on one instance.
Source: same URL as above.
Invoke a triggered job
Note: if the site has multiple instances, the job will run on one of them arbitrarily.

Related

Run a single while(true) on Azure

I'm new to Azure and I need to DoSomeWork() in Azure. This DoSomeWork() should be running periodically, every N minutes or so. However, DoSomeWork() can't be executed twice at the same time. In other words, any DoSomeWork() execution can't start before a prior DoSomeWork() execution finished.
I've been taken a look at Azure Web Jobs, particularly Continuous Azure Web Jobs. This seems the way to go but it's not clear on how to start, especially with the starting code that you get in VS:
static void Main()
{
var config = new JobHostConfiguration();
if (config.IsDevelopment)
{
config.UseDevelopmentSettings();
}
var host = new JobHost(config);
// The following code ensures that the WebJob will be running continuously
host.RunAndBlock();
}
There is also Functions class that takes an input a QueueTrigger decorated parameter, but I don't want the code to be triggered by any queue message or so.
How can I get a simple Console.WriteLine("hello world") running e.g. every minute but without overlapping? If Azure Web Jobs is not the way to go, what should I use instead (should be Azure-based)?
As mentioned in the comment, azure webjobs supports TimerTrigger feature(scheduled WebJob) which can be ran every xxx minutes as per your need.
It's quite simple when using azure webjob. For example, in visual studio, just create a console project -> Add this line of code: Console.WriteLine("hello world") -> then build the project -> then zip all the necessary files including the .exe into a .zip file -> at last, upload the .zip file into your webjob, and set the schedule(like execute the code every 5 minutes).
Please refer to this doc for more details about creating a scheduled WebJob.
You can also consider using other azure services which supports timerTrigger feature, like azure function.
WebJobs can be complicated to set up and maintain. Especially when dealing with a CI/CD pipeline. I have a few running right now and am getting ready to move them into one of the following more dependable and maintainable solutions:
The way we set up scheduled work is to use an Azure Function that runs via CRON schedule. It's super dependable and durable since it's managed by Azure. You just set it up, throw your code up and the rest is up to Azure to make sure it fires off when you configured it to.
If you want to do this in your own application, take a look at running a background service in an ASP.NET Core application. You can run a timer in the background service that will fire off and do some work. Keep in mind that if your app scales horizontally, you will be running two timers, which probably isn't good in your situation.
You could do something fancy like setting up a Azure Function to hit an endpoint on your WebAPI at a scheduled time. Then you could send the work to a BackgroundService which is singleton, so you could block a second request if you are currently running your job.
We tend to go the last route. Azure fires off a timer, the function executes, sends a message to an endpoint, the endpoint places work in the background.
There are tons of options outside of what I mentioned, but these are the only ones I have had the privilege to architect.

Can I run an Azure storage queue processor in a webAPI project?

Within Azure Functions and WebApps it is possible to automate execution of Storage queues by setting up a method linked to that queue by a QueueTrigger. I would like to accomplish the same thing, but I want it all compiled as part of my Web API so it is all in one project/compile/upload.
I have played around and I can setup the functions within my webapi project and compile them. I have added all the necessary packages and configuration, but the queue does not fire and no errors occur.

Is there an alternative for Azure Cloud Services that can scale out faster?

We currently run our service as an Azure Cloud Service with a web role. It's rather convenient to deploy (from the .cspkg) and manage OS changes (the Guest OS images thing) and we even download rarely changing dependencies from blob storage and unpack them.
However scaling out is very slow - our code computes various metrics and at some moment it decides to scale out and issues a Change Deployment Configuration Management Service request. Then some magic happens inside Azure and something like 5-10 minutes later we have new service instances running our startup code - control finally gets into RoleEntryPoint.OnStart(). That's very slow for us - we'd like to have RoleEntryPoint.OnStart() invoked something like 5-10 seconds instead of 5-10 minutes after the scale-out request.
Is there any PaaS-style alternative to Cloud Services within Azure which is known to scale out faster?
The alternative would be App Service (web apps). You won't have the same control as with cloud services but they can still be customized pretty well, and they scale out (and up) really fast.

creating azure worker rules from a vm

is it possible to start Worker Role Instances dynamically from a c# application running on azure windows vm?
in azure i have a Medium virtual machine, on it there is a c# console application that runs automatically on 11:00PM daily and it keeps processing data until about 7:00AM, my data is getting bigger and thus needs more time to be processed and i need to finish processing all data before 5:00AM.
is it possible to use Worker rule to run an instance of the application an pass it a part of the data to process?
note that my process makes http requests to external websites and the processed data gets written to a mongodb.
i am not sure where to start, and i am not sure if using worker rules is better than creating couple vms.
in general how would you solve this problem with the tools available on azure?
Is it possible to start Worker Role Instances dynamically from a c#
application running on azure windows vm?
Absolutely Yes. In order to do so, you would need to consume Service Management API. You could either write code yourself to consume this API or there's a Windows Azure Management Library available to do so which you can install from Nuget. To learn more about this API, you may find this blog post useful: http://www.bradygaster.com/post/getting-started-with-the-windows-azure-management-libraries.
Generally speaking Worker Roles are equivalent to Windows Services in the sense that both are used to perform background tasks. Since you're performing background tasks through your VM, I can't see any reason why you can't do the same though a Worker Role instance. My recommendation would be to go through tutorials available online or Windows Azure Platform Training Kit to become familiar with Worker Role concepts and how you could make use of them in your project.
For your specific scenario you may want to look at the auto scale rules that are now available; In the configuration for the worker role, in the Azure Management Console, you can specify, for example, that you want at least two workers running between certain times each day.
The Service Management API gives you a lot more control, but the auto scale is quick and easy to start with.
Incidentally, if the work your worker has to do can be divided into atomic chunks, then you may want to use a storage queue to write all the tasks to and then have the worker role pull tasks off that queue. You can then configure the autoscale to monitor the length of the queue and start and stop workers as required.

Spawning and executing a Worker process in Azure

The google has really failed me on this one. I am new to Azure and am only intermediate at .NET
I have an Azure solution going and I've written some code in a Web Role which runs great. What I would like to do now is move some of this code into an Azure Worker, which will be initialized by a controller function in the Web Role
What on earth do I need to do to get this going locally? I have created the Worker project within the SLN. I just need to know how to fire it up and run it.
I think part of my problem is I am assuming these workers behave like Heroku workers... is this the case? Because what I need is something like a queue system (a bunch of "worker tasks" in one big queue).
A lot of the links I've found for tutorials seem to tap dance around how to actually initialize the process from a Web Role.
Workers in Windows Azure are not tasks; they're entire VMs. To make your life easier, memorize this little detail: Web Role instances are Windows Server 2008 with IIS running, and Worker Roles are the same thing but with IIS disabled.
When you added that worker role to your project, you actually now have a new set of virtual machines running (at least one, depending on the instance count you set). These VMs have their own OnStart() and Run() methods you can put code into, for bootstrapping purposes.
If you grab the Windows Azure training kit, you'll see a few labs that show how to communicate between your various role instances (a common pattern being the use of Windows Azure queues). There's a good example of background processes with the Guestbook hands-on lab (the very first lab).
More info on this, as I've gotten it going now..
If you're coming from a Heroku background, then an Azure Worker is more or less the function in Rails that you'd actually execute with the queue. Unlike Heroku queued operations, an Azure Worker just runs endlessly and keeps polling for new stuff to do... hence the templated sleep(10000) in the Run() function.
The most conventional way I've found to make a Web and Worker talk to each other is by queue messages via Azure ServiceBus which is currently NOT emulated, meaning you need a functioning Azure account to make this work, and it will work even if you are running locally. You just need internet access.
A ServiceBus message can pass an entire object over to the Worker (so long as the Worker proj has the right dependencies in it), so that's kind of nice.
I think you're just having trouble starting the azure emulator along with your worker/web roles? Just set the azure configuration project as the start up project and run that. It'll boot up the emulator along with all your roles.

Categories