Pipeline

class skypy.pipeline.Pipeline(configuration)

Bases: object

Class for running pipelines.

This is the main class for running pipelines of functions with dependencies and using their results to generate variables and tables.

Construct the pipeline.

Parameters:

configurationdict-like

Configuration for the pipeline.

Notes

Each step in the pipeline is configured by a dictionary specifying a variable name and the associated value.

If a function argument is a string $variable_name, it refers to the values of previous step in the pipeline.

‘configuration’ should contain the name and configuration of each variable and/or an entry named ‘tables’. ‘tables’ should contain a set of nested dictionaries, first containing the name of each table, then the name and configuration of each column and optionally an entry named ‘init’ with a configuration that initialises the table. If ‘init’ is not specificed the table will be initialised as an empty astropy Table by default.

See [1] for examples of pipeline configurations in YAML format.

References

[1]

https://github.com/skypyproject/skypy/tree/main/examples

Methods Summary

depend(args)	get dependencies from function args
evaluate(value)	evaluate an item in the pipeline
execute([parameters])	Run a pipeline.
read(filename)	Read a pipeline from a configuration file.
write(filename[, overwrite])	Write pipeline results to disk.
write_fits(filename[, overwrite])	Write pipeline results to a FITS file.
write_hdf5(filename[, overwrite])	Write pipeline results to a HDF5 file.

Methods Documentation

depend(args)

get dependencies from function args

returns a list of all references found

evaluate(value)

evaluate an item in the pipeline

execute(parameters={})

Run a pipeline.

This function runs a pipeline of functions to generate variables and the columns of a set of tables. It uses a Directed Acyclic Graph to determine a non-blocking order of execution that resolves any dependencies, see [1].

Parameters:

parametersdict

Updated parameter values for this execution.

References

[1]

https://networkx.github.io/documentation/stable/

classmethod read(filename)

Read a pipeline from a configuration file.

Parameters:

filenamestr

The name of the configuration file.

write(filename, overwrite=False)

Write pipeline results to disk.

Parameters:

filenamestr

Name of output file to be written. It must have one of the supported file extensions for FITS (.fit .fits .fts) or HDF5 (.hdf5 .hd5 .he5 .h5).

overwritebool

If filename already exists, this flag indicates whether or not to overwrite it (without warning).

write_fits(filename, overwrite=False)

Write pipeline results to a FITS file.

Parameters:

filenamestr

Name of output file to be written.

overwritebool

If filename already exists, this flag indicates whether or not to overwrite it (without warning).

write_hdf5(filename, overwrite=False)

Write pipeline results to a HDF5 file.

Parameters:

filenamestr

Name of output file to be written.

overwritebool

If filename already exists, this flag indicates whether or not to overwrite it (without warning).