Supercomputers are great team players, but they work best on a tight schedule. And on-demand private clouds are providing a way to schedule some of their toughest jobs.
Before getting into the convergence of high-performance computers and cloud, let’s talk workload management in general
The popularity of a cluster computing, joining high-performance computers and other hardware into a single system over a network, is growing. So is grid computing, which takes it further by bringing HPC systems, even spread over the world, into lock-step. The advantages are clear: jobs are sent to the right place, at the right time, and thus, complex tasks can be completed faster and more cheaply,
But it’s the software that has to be in charge of managing it all. And until recently (the 1990s), it was fairly primitive. After a quick evolution though, HPC workload management software is now far far more sophisticated. For example, I spoke to one vendor, Adaptive Computing, which has developed such a platform, known as Moab.
According to Wolfgang Baumann, business development manager at the company, Moab can schedule tasks intelligently over a cluster or grid, sending the right jobs to remote locations, calculating the capacity necessary to run them, whether it’s thousands of small jobs or one large one, and working within defined limits.
These limits could be cost or power consumption (or both). A few days ago, for example, Adaptive announced it was partnering with Cray to provide Moab to a research institute and university in Germany, which share a massive distributed Cray XC30 grid.
Enter the cloud
While grids and clusters, and the software to manage them, were growing, cloud was going through its own evolution. And as it turns out, HPC systems can thrive in a private cloud environment under the right circumstances.
How? Baumann used the example of a financial institution that’s paying for a 100-node cluster over a month to process thousands of small transactions. But at month’s end, it needs to run a simulation on its entire portfolio in one shot–one gigantic problem to solve. Adaptive Computing can then build an “on-demand” private cloud to do it using a whopping 1500 cores.
While HPC is traditionally seen as something in the domain of science labs and universities, I found this an interesting use case in the enterprise, and something I’m going to look into more.