Pipeline

class skypy.pipeline.Pipeline(configuration)[source] [edit on github]

Bases: object

Class for running pipelines.

This is the main class for running pipelines of functions with dependencies and using their results to generate variables and tables.

Construct the pipeline.

Parameters
configurationdict-like

Configuration for the pipeline.

Notes

Each step in the pipeline is configured by a dictionary specifying a variable name and the associated value.

If a function argument is a string $variable_name, it refers to the values of previous step in the pipeline.

‘configuration’ should contain the name and configuration of each variable and/or an entry named ‘tables’. ‘tables’ should contain a set of nested dictionaries, first containing the name of each table, then the name and configuration of each column and optionally an entry named ‘init’ with a configuration that initialises the table. If ‘init’ is not specificed the table will be initialised as an empty astropy Table by default.

See [1] for examples of pipeline configurations in YAML format.

References

1

https://github.com/skypyproject/skypy/tree/main/examples

Methods Summary

depend(args)

get dependencies from function args

evaluate(value)

evaluate an item in the pipeline

execute([parameters])

Run a pipeline.

read(filename)

Read a pipeline from a configuration file.

write(filename[, overwrite])

Write pipeline results to disk.

write_fits(filename[, overwrite])

Write pipeline results to a FITS file.

write_hdf5(filename[, overwrite])

Write pipeline results to a HDF5 file.

Methods Documentation

depend(args)[source] [edit on github]

get dependencies from function args

returns a list of all references found

evaluate(value)[source] [edit on github]

evaluate an item in the pipeline

execute(parameters={})[source] [edit on github]

Run a pipeline.

This function runs a pipeline of functions to generate variables and the columns of a set of tables. It uses a Directed Acyclic Graph to determine a non-blocking order of execution that resolves any dependencies, see [1].

Parameters
parametersdict

Updated parameter values for this execution.

References

1

https://networkx.github.io/documentation/stable/

classmethod read(filename)[source] [edit on github]

Read a pipeline from a configuration file.

Parameters
filenamestr

The name of the configuration file.

write(filename, overwrite=False)[source] [edit on github]

Write pipeline results to disk.

Parameters
filenamestr

Name of output file to be written. It must have one of the supported file extensions for FITS (.fit .fits .fts) or HDF5 (.hdf5 .hd5 .he5 .h5).

overwritebool

If filename already exists, this flag indicates whether or not to overwrite it (without warning).

write_fits(filename, overwrite=False)[source] [edit on github]

Write pipeline results to a FITS file.

Parameters
filenamestr

Name of output file to be written.

overwritebool

If filename already exists, this flag indicates whether or not to overwrite it (without warning).

write_hdf5(filename, overwrite=False)[source] [edit on github]

Write pipeline results to a HDF5 file.

Parameters
filenamestr

Name of output file to be written.

overwritebool

If filename already exists, this flag indicates whether or not to overwrite it (without warning).