I'm currently running some computationally intensive simulations, but they are taking a long time to complete. I've already split the workload across all the available physical cores in my processor. What I'm wondering is how to go about splitting the workload further and assigning it to other computers. I'm contemplating buying a couple Xeon servers and using them for the number crunching.
The one big issue I have is that I'm currently running the program within Visual Studio (Ctrl F5) as there are two methods which I'm constantly making small changes to.
Any suggestions on how/if it's possible to assign the workload to other computers / if it's possible to still run the program with VS or would I need to create an *.exe each time I wanted to run it?
It depends on the problem you're solving.
You can use map/reduce and Hadoop if it's easily parallelizable, like SETI#Home.
You can use something like MPI if it's not, like linear algebra.
Isn't the crux of your problem in this statement "The one big issue i have is that im currently running the program within Visual Studio (Ctrl F5) as there are two methods which im constantly making small changes to."?
Is it the "one big issue" because if you distribute then you can't afford modifying the code on all of the nodes when doing the job so you think about something distributing it for you? If this is the case then I assume that you already know how to split the algo or data in a way that nodes can take take of small parts of the job.
If it's the case - sorry if I misunderstood - then externalise the part that you are "constantly making small changes to" into a file or a DataBase encoded in some simple or more elaborate form depending on what you are changing so you don't need to have your nodes change constantly. Deploy the code on all nodes, connect them to the DB or file which contains the varying bit and enjoy your new Ferrari!
You could use the WMI service to start your process on the remote computers. You would build your exe to a shared directory that is visible to the remote computer, then use WMI on the remote computer to launch the exe.
There are plenty of examples out there to do this, but to illustrate, a simple method which assumes no authentication complications is to use a .VBS script file:
strComputer = "acomputer"
strCommandLine = "calc.exe"
Set objWMIService = GetObject("winmgmts:\\" & strComputer & "\root\cimv2")
Set objProcess = objWMIService.Get("Win32_Process")
intReturnValue = objProcess.Create(strCommandLine, , , intPID)
WScript.Echo "Process ID: " & intPID
You can also use PsExec from SysInternals to handle all the details of making this work.
After building the exe in Visual Studio, you could run it on your local machine to ensure it does what you want, then when you are ready to launch it on the remote systems, you can execute a batch script similar to the above VBS to launch the exe on the remote systems.
You will still need to provide some mechanism to divide up the workload so that each client knows what part of the problem it is supposed to work on. You could provide this information in the command line used to start the remote apps, in a config file in the directory with the exe, in a database table, or use a separate command-and-control type server that the clients connect back to (although with that approach you'll soon get to the stage where you would have been better off with learning to use an existing solution rather than rolling your own).
You may also want to include a remote 'kill switch' of some sort. You could use PsKill from SysInternals, or if you want a more graceful shutdown, something simple like the existence of a particular file in the same directory as the exe can serve as a flag for the remote processes to shut themselves down.
You could also consider adding CSScript support to the client so that the remote client programs are static and load and compile a CSScript file to do the work. This might be useful if you encounter some kind of difficulty in frequently redeploying and restarting the client programs, or if you need them to all be slightly different (you might write a program to generate separate script files for each client for example).
Related
I run an assembly line at work that I'm trying to automate.
The current software is two stand alone excitable. One is an editor that adds the variables to an sql database and the second reads that database and controls the cnc. Both are on the same pc, running at the same time and both are written in vb6. When you hit the 'add' button in the first .exe (editor) it somehow tells the second .exe to reload the sql database and load any updates.
My problem is I've written a software that takes the barcode and inserts the variables into the database automatically which will bypass the first software but the second software doesn't know when to revisit the database for updates.
Are there any common ways for one .exe to talk to a second .exe and how do I listen so I can duplicate it?
Thanks
Sam
EDIT :
sorry what i meant by 'bypass' is make the first .exe redundant. My software inserts in to the sql rather then their editor software.
You can look at something called named pipes. This is how it is commonly done.
You can read about it here at msdn, there is a good example.
https://msdn.microsoft.com/en-us/library/bb546085(v=vs.110).aspx
Pipes, mentioned in another response, are one common mechanism. My preference when small amounts of data are exchanged are UDP sockets.
Using the Winsock control you can set up peer-peer or 'talker-listener' communication in just a few lines of code. Your second program can listen for either a simple signal or a more complete packet of data from the first program and act accordingly.
A side benefit is that if this PC is on a local network, it's trivial to move the two programs to separate PCs if that becomes desirable at a later date.
MSMQ private machine queues are easy to set up and use in VB6. They almost sound ideal for the sort of thing you are doing, and you may be able to ditch the database entirely.
We run a Windows Forms application developed in C# in our company, and one problem is giving us headaches.
When we run the application from a local machine, in drive C:, for example, the application loads and runs fast. It's heavily database-based, which means it does a lot of queries to our MSSQL server, and it runs all queries in less than 1 second, while running from a local drive.
If we run the same application from a mapped network drive (not a UNC path, a M: mapped drive), it loads fast, but the queries takes ages to complete, and hardly we can see the result.
ClickOnce is not an option for us (due to reasons that are not subject to discussion here), and we have several other 3rd party applications that runs fast, loaded from the same mapped M: drive.
I did some research, and the closest question I could find is this one:
http://stackoverflow.duapp.com/questions/2554716/my-c-net-application-is-running-slower-when-the-exe-is-located-on-the-network
When I right-click the application there's no "unblock" option available, which tells me that there's no secondary stream attached to the file and it's "trusted" by the machine.
Also, I tried adding <loadFromRemoteSources enabled="true"/> in the .config file, but it caused no changes in the application performance so far.
The application is not signed, and the slowness happens with both debug and release versions of the application.
What are we doing wrong ?
PS: I'm still trying to pinpoint the exact command that's taking longer to work, but no luck so far.
EDIT: Adding new information. It seems that the problem wasn't the network "per se", but the fact that the application was doing a background task and failing because it was running from the network. This failure wasn't wrapped around a try-catch block, and was preventing the background task to return properly, creating a major delay on the application response.
That means it was our development bug, not Windows fault. Thanks for the answers, I'll vote to close this question.
I have recently found one scenario where exactly this was happening in .net winforms sql-server application.
On one machine, the application was lightning-fast, on another one, queries took seconds.
Second machine was configured to use VPN dialed via PPTP. The VPN was automatically reconnecting whenever the computer got online – even if the machine was in company network (where no VPN was needed). VPN auto-redial trick always seemed to be very useful... until I found that connection to the SQL server basically always went through the VPN because of this. Manually disconnecting the VPN instantly helped: responses got fast again.
I do not say this is definite solution in your case but this is one of things what causes almost unacceptable slowness of queries. I observed this first hand.
I would like to be able to do an "inplace" update with my program. Basically, I want to be able to login remotely where the software is deployed, install it while other users are still using it (in a thin client way), and it update their program.
Is this possible without too much of a hassle? I've looked into clickonce technology, but I don't think that's really what I'm looking for.
What about the way firefox does it's updates? Just waits for you to restart the program, and notifies you when it's been updated.
UPDATE: I'm not remoting into the users' PC. This program is ran on a server, and I remote in and update it, the users run it directly off the server through remote access.
ClickOnce won't work because it requires a webserver.
I had some example code that I can't find right now but you can do something similar to Firefox with the System.Deployment.Application namespace.
If you use the ApplicationDeployment class, you should be able to do what you want.
From MSDN, this class...
Supports updates of the current deployment programmatically, and handles on-demand downloading of files.
Consider the MS APIs with BITS, just using bitsadmin.exe in a script or the Windows Update Services.
Some questions:
Are the users running the software locally, but the files are located on a networked share on your server?
Are they remoting into the same server you want to remote into, and execute it there?
If 2. are they executing the files where they are placed on the server, or are they copying them down to a "private folder"?
If you cannot change the location of the files, and everyone is remoting in, and everyone is executing the files in-place, then you have a problem. As long as even 1 user is running the program, the files will be locked. You can only update the files once everyone is out.
If, on the other hand, the users are able to run their own private copy of the files, then I would set up a system where you have a central folder with the latest version of the files, and when a user starts his program, it checks if the central folder has newer versions than the user is about to execute. If it does, copy the new version down first.
Or, if that will take too long, and the user will get impatient (what, huh, users getting impatient?), then having the program check the versions after startup, and remind the user to exit would work instead. In this case, the program would set a flag that upon next startup would do the copying, only now the user is aware of it happening.
The copying part would easily be handled by either having a separate executable that does the actual copying, and executing that instead, or the program could copy itself temporarily to another location and run that copy with parameters that says "update the original files".
While you can design your code to modify itself (maybe not in C#?), this is generally a bad idea. This means that you must restart something to get the update. (In Linux you are able to replace files that are in use, however an update does not happen until the new data is loaded into memory i.e. application restart)
The strategy used by Firefox (never actually looked into it) is storing the updated executable in a different file which is checked for when program starts to load. This allows the program to overwrite the program with the update before the resource is locked by the OS. You can also design you program more modular so that portions of it can be "restarted" without requiring a restart of the entire program.
How you actually do this is probably provided by the links given by others.
Edit:: In light of a response given to Lasse V. Karlsen
You can have your main program looking for the latest version of the program to load (This program wouldn't be able to get updates without everyone out). You then can remove older versions once people are no longer using it. Depending on how frequent people restart their program you may end up with a number of older programs versions.
ClickOnce and Silverlight (Out of browser) both support your scenario, if we talk about upgrades. Remote login to your users machine? Nope. And no, Firefox doesn't do that either as far as I can tell..
Please double-check both methods and add them to your question, explaining why they might not do what you need. Otherwise it's hard to move on and suggest better alternatives.
Edit: This "I just updated, please restart" thing you seem to like is one method call for Silverlight applications running outside of the browser. At this point I'm fairly certain that this might be the way to go for you.
ClickOnce doesn't require a webserver, it will let you publish updates while users are running the software. You can code your app to check for new update every few minutes and prompt the user to restart the app if a new version is found which will then take them through the upgrade process.
Another option is a Silverlight OOB application, but this would be more work if your app is already built as WinForms/WPF client app.
Various deployment/update scenarios (for .NET applications) are discussed with there pros and cons in Microsoft's Smart Client Architecture and Design Guide. Though a little bit old I find that most still holds today, as it is describing rather the basic architectural principles than technical details. There is a PDF version, but you find it online as well:
Deploying and Updating Smart Client Applications
Is this possible without too much of a hassle?
Considering the concurrency issues with thin clients and the complexity of Windows installations, yes hot updates will be a hassel without doing it the way the system demands.
I've created an exe file that does some maintainance work on my server.
I want to be able to launch it from the website that sits on the server.
The exe has to be launched on the server itself and not on the client.
My instincts tell me it's not possible but I've had to check with you guys.
If I need to set certain permissions / security - I can.
Yes, it can be done, but it's not recommended.
An ideal solution for running maintenance scripts/executables is to schedule them using cron on Unix/Linux systems or using Scheduled Tasks in Windows. Some advantages of the automated approach vs. remote manual launch:
The server computer is self-maintaining. Clients can fail and people can forget. As long as the server is running the server will be keeping itself up to date, regardless of the status of client machines or persons.
When will the executable be launched? Every time a certain page is visited? What if the page is refreshed? For a resource-intensive script/executable this can severely degrade server performance. You'll need to code rules to handle multiple requests and monitor running maintenance processes. Cron & scheduled tasks handle these possibilities already.
A very crude option, Assuming IIS: Change Execute Access from "Scripts Only" or "None" to "Scripts and Executables"
To make this less crude, you should have the executable implement a CGI interface (if that is under your control.
And, if you want to use ASP.NET to add autorization/authentication, the code (C#) to do this would be:
System.Diagnostics.Process process;
var startInfo = New System.Diagnostics.ProcessStartInfo("C:\file.exe")
process.StartInfo = startInfo;
process.Start();
process.WaitForExit();
It's possible, but almost certainly it's a bad idea. What environment/webserver? You should just need to set the relevant 'execute' permissions for the file.
I really suggest that you don't do this, however, and configure the task to run automatically in the background. The reasoning is that, configured badly, you could end up letting people run any executable, and depending on other factors, completely take over your machine.
Depends what language you're using; most server side scripting languages give you a way to exectue shell commands, for example:
$result=`wc -l /etc/passwd`;
executed a unix command from perl.
Most web languages (I know at least Java and PHP) allow you to execute a command line argument from within a program.
I need to process large image files into smaller image files. I would like to distribute the work to many "slave" servers, rather than tasking my main server with this. I am using Windows Server 2005/2008, C#, and ASP.NET. I have a lot of web application development experience but have not developed distributed systems. I had a notion that this could be designed as follows:
1) Files would be placed in a shared network drive
2) Slave servers would periodically poll the drive for new content
3) Slave servers would rename newly found files to something like UNPROCESSED_appIDXXXX_jidXXXXX_photoidXXXXX.tif and begin processing that file.
4) Other slave servers would avoid trying to process files that are in process by examining file name, i.e. if something has been named "UNPROCESSED" they will not attempt to process.
I am wondering a few things:
1) Will there be issues with two slave servers trying to "grab" and rename the file at once, or will Windows Server automatically lock the file?
2) What do you think the best mechanism for notification of new content for processing should be? One simple idea is to write a basic aspx page on each slave system and have it running on a timer. A better idea might be to write a Windows Service that utilizes SystemFileWatcher and have it running on each slave system. A third idea is to have a central server somehow dispatch instructions to a given slave server to attempt a processing job, but I do not know of ways of invoking that kind of communication beyond a very hack-ish approach of having the master server pass a message via HTTP.
I'd much appreciate any guidance you have to offer.
Cheers,
-KF
If you don't want to go all the way with a compute cluster type solution. You should consider having a job manager running somewhere that will parcel out the work. That way, when a server becomes available to do work, it asks the job manager for a new bit of work to do. It can then tell the job manager that it's finished and the job manager can inform your "client" when the work on the whole job is complete. That way, it's easy to register work and know it's complete and the job manager can parcel out the work without the worry of race conditions on file renames. :)