Workflow Managment systems
Date:
Workflows are a schematic representation of what you as a scientist need to do to convert raw input (simulation parameters, observational/experimental data) into the scientific products you are interested in (mostly plots and tables). They help you keep track of the dependencies between different steps in an analysis or simulation pipeline, and make it easier for other people to see what it is exactly that you do. Workflow managements systems (WMSs) help you to run workflows automatically. They read your workflow and then schedule it on the hardware you make available in an efficient and robust way, without requiring any manual intervention. This makes running your workflows much more efficient and means you have to spend less time on trivial computing tasks, which means you have more time to do interesting science. In this talk, I give a very basic introduction to the concept of scientific workflows using a real example from astrophysics. I start from a very simple workflow and then gradually increase its complexity to illustrate the usefulness of WMSs. I then show how to run this example workflow using the Makeflow WMS.
The example workflow is available from my personal github page (https://github.com/bwvdnbro/HydroCodeSpherical1D/tree/master/paper_workflows).
More information on the Makeflow WMS can be found on http://ccl.cse.nd.edu/software/manuals/makeflow.html. I would also encourage you to check out the Blue Waters webinar series on scientific workflows: https://bluewaters.ncsa.illinois.edu/webinars/workflows.