User Meeting 2016

Merger, a Software Pipeline to Merge Datasets at the Australian Synchrotron MX beamlines

Not scheduled

15m

National Centre for Synchrotron Science

Poster Structural Biology

Speaker

Jun Aishima (Australian Synchrotron)

Description

The ability to determine the progress of an experiment is critical to efficiently use beamtime during synchrotron visits. While processing results from individual datasets are sufficient for many crystallography experiments, chemical crystallography experiments that require use of the minikappa to collect datasets from multiple orientations, and sensitive samples where partial datasets can be collected from multiple macromolecular crystals in particular would benefit from the ability to combine results from multiple data collections. With these applications in mind, we have developed a dataset merging pipeline. The Merger pipeline is designed to automatically run dataset merging using three different programs – Phenix scale_and_merge, Blend, and XScale. The user simply selects the datasets to combine using the web application, selecting a reference dataset if desired, and press the 'Merge Selected Datasets' button to trigger the merging process. Written entirely using the Python programming language, the Merger pipeline is run within the Airflow pipeline framework which was originally developed by Airbnb. Several steps are first performed to sanitize the input to the different merging programs, removing incompatible (different space group/unit cell parameters) datasets, reindexing datasets if necessary, and after running XScale, further analysis software is run to derive dataset statistics. The merging results from XScale, the standard of dataset merging software, are inserted into the database to allow users to readily compare results using the web application.