12-16 November 2018
Australian Synchrotron
Australia/Melbourne timezone

Lightflow - a lightweight, distributed workflow system

16 Nov 2018, 09:50
20m
Oliphant Auditorium (Australian Synchrotron)

Oliphant Auditorium

Australian Synchrotron

800 Blackburn Road 3168 Clayton
Plenary Talk Experiment Control Plenary Talks

Speaker

Andreas Moll (Australian Synchrotron)

Description

The Australian Synchrotron, is one of Australia’s most important pieces of research infrastructure. After more than 10 years of operation,
the beamlines at the Australian Synchrotron are well established
and the demand for automation of research tasks is
growing. Such tasks routinely involve the reduction of TBscale
data, online (realtime) analysis of the recorded data to
guide experiments, and fully automated data management
workflows.

In order to meet these demands, a generic, distributed
workflow system was developed. It is based on well established
Python libraries and tools. The individual tasks
of a workflow are arranged in a directed acyclic graph and
one or more directed acyclic graphs form a workflow. Workers
consume the tasks, allowing the processing of a workflow
to scale horizontally. Data can flow between tasks and a variety
of specialised tasks is available.

Lightflow has been released as open source on the Australian
Synchrotron GitHub page

Primary authors

Andreas Moll (Australian Synchrotron) Stephen Mudie (ANSTO)

Presentation Materials

Your browser is out of date!

Update your browser to view this website correctly. Update my browser now

×