There are currently four pre-installed tools on the host workflow.iu.xsede.org that allow users to run scientific workflows directly on XSEDE machines without having to install or maintain the software themselves. The host machine works just like login.xsede.org. If you have an XSEDE account and an active allocation, you already have an account on workflow.iu.xsede.org.
“This new capability will allow our users to try out scientific workflow software on XSEDE without needing to install or maintain the software,” said Marlon Pierce, manager for XSEDE science gateways and scientific workflows. “We want XSEDE to be responsive to the needs of the community at large, and this technology recommendation came straight from the users themselves — a BoF session at XSEDE14, actually — and we based an XSEDE15 tutorial on this.”
The four workflow technologies that are seamlessly inserted into XSEDE architecture are Swift, Makeflow/Work Queue, Pegasus and RADICAL-Pilot. An additional resource regarding these new workflows includes in-depth examples and documentation to allow users to see what each workflow can do for them.
- Tutorial guides for the tools are available from https://sites.google.com/site/xsedeworkflows.
- Further questions can be directed to Pierce at marpierc@iu.edu.
- You can also join the workflow community mailing list. See instructions at https://www.xsede.org/ecss-workflows.
Swift
Swift is a simple language for writing parallel scripts that runs many copies of ordinary programs concurrently as soon as their inputs are available, reducing the need for complex parallel programming. The same Swift script runs on multi-core computers, clusters, grids, clouds, and supercomputers. Swift is thus a useful tool for moving computations from a laptop or workstation to any XSEDE resource. Swift can run a million programs, thousands at a time, launching hundreds of programs per second.
Makeflow/Work Queue
Makeflow is a workflow engine for executing workflows that range from just a handful of tasks on a single machine to millions of tasks on large clusters. The same workflow can be run without modification on a variety of batch systems, including HTCondor, SGE, Torque, PBS, SLURM, and the (included) Work Queue system. The Makeflow language is very similar to that of classic Make, so it is very easy to get started using existing applications.
Pegasus
Pegasus WMS is a production ready configurable system for mapping and executing workflows over a wide range of execution environment, including XSEDE. Workflows are described by users as portable abstract directed acyclic graphs (DAG) which are then mapped into executable workflows optimized for a particular environment. Pegasus comes with full support for complex data management, automatic provenance collection, and sharing of reproducible workflows.
RADICAL-Pilot
RADICAL-Pilot is a simple way to run multiple jobs — many simulations, coupled simulations — concurrently. RADICAL-Pilot offers a Python API that gives the developer complete freedom on selecting which resources to allocate and how to distribute the tasks over the resources. RADICAL-Pilot can provide the basic capabilities needed to run different scenarios of multiple jobs, whether running a bag-of-tasks on a single XSEDE resource, or several pipelines of dependent tasks on multiple distinct XSEDE resources.
Pierce added: “There are a lot more workflow tools out there that may be of interest to the XSEDE community. We certainly want to hear from both XSEDE researchers and workflow software developers on what will be useful to them.”
You can sign up to receive Impact by XSEDE — the project’s monthly e-newsletter to update subscribers on the activities of XSEDE — by visiting https://illinois.edu/gm/subscribe/22179.
The Extreme Science and Engineering Discovery Environment (XSEDE) is the most advanced, powerful, and robust collection of integrated advanced digital resources and services in the world. It is a single virtual system that scientists can use to interactively share computing resources, data, and expertise. XSEDE accelerates scientific discovery by enhancing the productivity of researchers, engineers, and scholars by deepening and extending the use of XSEDE’s ecosystem of advanced digital services and by advancing and sustaining the XSEDE advanced digital infrastructure. XSEDE is a five-year, $121-million project and is supported by the National Science Foundation.
