Pipeline

class skypy.pipeline.Pipeline(configuration)

Bases: object

Class for running pipelines.

This is the main class for running pipelines of functions with dependencies and using their results to generate variables and tables.

Construct the pipeline.

Parameters

configurationdict-like

Configuration for the pipeline.

Notes

Each step in the pipeline is configured by a dictionary specifying a variable name and the associated value.

A value that is a tuple (function, args) specifies that the value will be the result of a function call. The first item is a callable, and the second value specifies the function arguments.

If a function argument is a string $variable_name, it refers to the values of previous step in the pipeline.

‘configuration’ should contain the name and configuration of each variable and/or an entry named ‘tables’. ‘tables’ should contain a set of nested dictionaries, first containing the name of each table, then the name and configuration of each column and optionally an entry named ‘init’ with a configuration that initialises the table. If ‘init’ is not specificed the table will be initialised as an empty astropy Table by default.

See [1] for examples of pipeline configurations in YAML format.

References

1

https://github.com/skypyproject/skypy/tree/master/examples

Methods Summary

execute([parameters])	Run a pipeline.
get_args(args)	parse function arguments
get_deps(args)	get dependencies from function args
get_value(value)	return the value of a field
read(filename)	Read a pipeline from a configuration file.
write([file_format, overwrite])	Write pipeline results to disk.

Methods Documentation

execute(parameters={})

Run a pipeline.

This function runs a pipeline of functions to generate variables and the columns of a set of tables. It uses a Directed Acyclic Graph to determine a non-blocking order of execution that resolves any dependencies, see [1].

Parameters

parametersdict

Updated parameter values for this execution.

References

1

https://networkx.github.io/documentation/stable/

get_args(args)

parse function arguments

strings beginning with $ are references to other fields

get_deps(args)

get dependencies from function args

returns a list of all references found

get_value(value)

return the value of a field

tuples specify function calls (function name, function args)

classmethod read(filename)

Read a pipeline from a configuration file.

Parameters

filenamestr

The name of the configuration file.

write(file_format=None, overwrite=False)

Write pipeline results to disk.

Parameters

file_formatstr

File format used to write tables. Files are written using the Astropy unified file read/write interface; see [1] for supported file formats. If None (default) tables are not written to file.

overwritebool

Whether to overwrite any existing files without warning.

References

1

https://docs.astropy.org/en/stable/io/unified.html