Skip to content

Abstracts

The core abstract APIs that define various class behaviors.

This module contains all of the abstract classes and APIs for defining objects. Abstracts include abstract data structures (like a graph), APIs for concepts such as queueing adapters and environment APIs, as well as fundamental data structures.

Dependency

Bases: Substitution

Abstract object representing a dependency.

The Dependency base class is intended to be used to capture external items the workflow is dependent on. These items include (but are not limited to):

* Remotely stored repositories (such as bitbucket)
* Paths located on the filesystem that hold required files or binaries
* Binaries that are required to be installed using a specific package
  manager
* External APIs that a workflow needs to pull data from

The goal of this base class is to make it so that this package is able to pull external dependencies in a consistent manner.

Source code in maestrowf/abstracts/envobject.py 
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
@six.add_metaclass(ABCMeta)
class Dependency(Substitution):
    """
    Abstract object representing a dependency.

    The Dependency base class is intended to be used to capture external items
    the workflow is dependent on. These items include (but are not limited to):

        * Remotely stored repositories (such as bitbucket)
        * Paths located on the filesystem that hold required files or binaries
        * Binaries that are required to be installed using a specific package
          manager
        * External APIs that a workflow needs to pull data from

    The goal of this base class is to make it so that this package is able to
    pull external dependencies in a consistent manner.
    """

    @abstractmethod
    def acquire(self, substitutions=None):
        """
        Acquire the dependency as specfied by the class instance.

        Subclasses that implement this interface should raise exceptions during
        acquisition should they be unable to retrieve their specified
        dependency. It is assumed that if acquiring throws an exception that
        the study cannot proceed forward.

        :param substitutions: List of Substitution objects that can be applied.
        """

acquire(substitutions=None) abstractmethod

Acquire the dependency as specfied by the class instance.

Subclasses that implement this interface should raise exceptions during acquisition should they be unable to retrieve their specified dependency. It is assumed that if acquiring throws an exception that the study cannot proceed forward.

Parameters:

Name Type Description Default
substitutions

List of Substitution objects that can be applied.

 None
Source code in maestrowf/abstracts/envobject.py 
145
146
147
148
149
150
151
152
153
154
155
156
@abstractmethod
def acquire(self, substitutions=None):
    """
    Acquire the dependency as specfied by the class instance.

    Subclasses that implement this interface should raise exceptions during
    acquisition should they be unable to retrieve their specified
    dependency. It is assumed that if acquiring throws an exception that
    the study cannot proceed forward.

    :param substitutions: List of Substitution objects that can be applied.
    """

Graph

An abstract graph data structure.

Source code in maestrowf/abstracts/graph.py 
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
@six.add_metaclass(ABCMeta)
class Graph:
    """An abstract graph data structure."""

    # NOTE: fdinatal -- 04/07/2017
    # This class mandates that a graph class be searchable currently.
    # That requirement should be filtered out into another abstract interface
    # most likely (think C# interfaces). I could imagine cases where a coder/
    # developer would want a leaner object without the frills.

    @abstractmethod
    def add_node(self, name, obj):
        """
        Method to add a node to the graph.

        :param name: String identifier of the node.
        :param obj: An object representing the value of the node.
        """

    @abstractmethod
    def add_edge(self, src, dest):
        """
        Add the edge (src, dest) to the graph.

        :param src: Source vertex name.
        :param dest: Destination vertex name.
        """

    @abstractmethod
    def remove_edge(self, src, dest):
        """
        Remove edge (src, dest) from the graph.

        :param src: Source vertex name.
        :param dest: Destination vertex name.
        """

add_edge(src, dest) abstractmethod

Add the edge (src, dest) to the graph.

Parameters:

Name Type Description Default
src

Source vertex name.

 required
dest

Destination vertex name.

 required
Source code in maestrowf/abstracts/graph.py 
54
55
56
57
58
59
60
61
@abstractmethod
def add_edge(self, src, dest):
    """
    Add the edge (src, dest) to the graph.

    :param src: Source vertex name.
    :param dest: Destination vertex name.
    """

add_node(name, obj) abstractmethod

Method to add a node to the graph.

Parameters:

Name Type Description Default
name

String identifier of the node.

 required
obj

An object representing the value of the node.

 required
Source code in maestrowf/abstracts/graph.py 
45
46
47
48
49
50
51
52
@abstractmethod
def add_node(self, name, obj):
    """
    Method to add a node to the graph.

    :param name: String identifier of the node.
    :param obj: An object representing the value of the node.
    """

remove_edge(src, dest) abstractmethod

Remove edge (src, dest) from the graph.

Parameters:

Name Type Description Default
src

Source vertex name.

 required
dest

Destination vertex name.

 required
Source code in maestrowf/abstracts/graph.py 
63
64
65
66
67
68
69
70
@abstractmethod
def remove_edge(self, src, dest):
    """
    Remove edge (src, dest) from the graph.

    :param src: Source vertex name.
    :param dest: Destination vertex name.
    """

PickleInterface

A mixin class that implements a general pickle interface using dill.

Source code in maestrowf/abstracts/__init__.py 
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
class PickleInterface:
    """A mixin class that implements a general pickle interface using dill."""

    @classmethod
    def unpickle(cls, path):
        """
        Load a pickled instance from a pickle file.

        :param path: Path to a pickle file containing a class instance.
        """
        with open(path, 'rb') as pkl:
            obj = dill.load(pkl)

        if not isinstance(obj, cls):
            msg = "Object loaded from {path} is of type {type}. Expected an" \
                  " object of type '{cls}.'".format(path=path, type=type(obj),
                                                    cls=type(cls))
            LOGGER.error(msg)
            raise TypeError(msg)

        return obj

    def pickle(self, path):
        """
        Generate a pickle file of of a class instance.

        :param path: The path to write the pickle to.
        """
        with open(path, 'wb') as pkl:
            dill.dump(self, pkl)

pickle(path)

Generate a pickle file of of a class instance.

Parameters:

Name Type Description Default
path

The path to write the pickle to.

 required
Source code in maestrowf/abstracts/__init__.py 
77
78
79
80
81
82
83
84
def pickle(self, path):
    """
    Generate a pickle file of of a class instance.

    :param path: The path to write the pickle to.
    """
    with open(path, 'wb') as pkl:
        dill.dump(self, pkl)

unpickle(path) classmethod

Load a pickled instance from a pickle file.

Parameters:

Name Type Description Default
path

Path to a pickle file containing a class instance.

 required
Source code in maestrowf/abstracts/__init__.py 
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
@classmethod
def unpickle(cls, path):
    """
    Load a pickled instance from a pickle file.

    :param path: Path to a pickle file containing a class instance.
    """
    with open(path, 'rb') as pkl:
        obj = dill.load(pkl)

    if not isinstance(obj, cls):
        msg = "Object loaded from {path} is of type {type}. Expected an" \
              " object of type '{cls}.'".format(path=path, type=type(obj),
                                                cls=type(cls))
        LOGGER.error(msg)
        raise TypeError(msg)

    return obj

Singleton

Bases:

Single type to allow for classes to be typed as a singleton.

Source code in maestrowf/abstracts/__init__.py 
 97
 98
 99
100
class Singleton(_Singleton('SingletonMeta', (object,), {})):
    """Single type to allow for classes to be typed as a singleton."""

    pass

Source

Bases: EnvObject

Abstract class representing classes that alter environment sourcing.

WARNING: The API for this class is still in development. The Source environment class is meant to provide a way to programmatically set environment settings that binaries or other scripts may require in the workflow. Such settings that are intended to be captured are:

* Exporting of shell/environment variables (using 'export')
* Setting of an environment package with the 'use' command
Source code in maestrowf/abstracts/envobject.py 
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
class Source(EnvObject):
    """
    Abstract class representing classes that alter environment sourcing.

    WARNING: The API for this class is still in development.
    The Source environment class is meant to provide a way to programmatically
    set environment settings that binaries or other scripts may require in the
    workflow. Such settings that are intended to be captured are:

        * Exporting of shell/environment variables (using 'export')
        * Setting of an environment package with the 'use' command
    """

    @abstractmethod
    def apply(self, data):
        """
        Apply the Source to some string data.

        Subclasses of Source should use this method in order to apply an
        environment altering change. The 'data' parameter should be a string
        representing a command to apply Source to or a list of other commands
        that Source should be included with.

        :param data: A string representing a command or set of other sources.
        :returns: A string with the Source applied.
        """

apply(data) abstractmethod

Apply the Source to some string data.

Subclasses of Source should use this method in order to apply an environment altering change. The 'data' parameter should be a string representing a command to apply Source to or a list of other commands that Source should be included with.

Parameters:

Name Type Description Default
data

A string representing a command or set of other sources.

 required

Returns:

Type Description

A string with the Source applied.
Source code in maestrowf/abstracts/envobject.py 
109
110
111
112
113
114
115
116
117
118
119
120
121
@abstractmethod
def apply(self, data):
    """
    Apply the Source to some string data.

    Subclasses of Source should use this method in order to apply an
    environment altering change. The 'data' parameter should be a string
    representing a command to apply Source to or a list of other commands
    that Source should be included with.

    :param data: A string representing a command or set of other sources.
    :returns: A string with the Source applied.
    """

Specification

Abstract class for loading and verifying a Study Specification

Source code in maestrowf/abstracts/specification.py 
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
@six.add_metaclass(ABCMeta)
class Specification:
    """
    Abstract class for loading and verifying a Study Specification
    """

    @abstractclassmethod
    def load_specification(cls, path):
        """
        Method for loading a study specification from a file.

        :param path: Path to a study specification.
        :returns: A specification object containing the information loaded
                  from path.
        """

    @abstractclassmethod
    def load_specification_from_stream(cls, stream):
        """
        Method for loading a study specification from a stream.

        :param stream: Raw text stream containing specification data.
        :returns: A specification object containing the information in string.
        """

    @abstractmethod
    def verify(self):
        """
        Verify the whole specification.
        """

    @abstractmethod
    def get_study_environment(self):
        """
        Generate a StudyEnvironment object from the environment in the spec.

        :returns: A StudyEnvironment object with the data in the specification.
        """

    @abstractmethod
    def get_parameters(self):
        """
        Generate a ParameterGenerator object from the global parameters.

        :returns: A ParameterGenerator with data from the specification.
        """

    @abstractmethod
    def get_study_steps(self):
        """
        Generate a list of StudySteps from the study in the specification.

        :returns: A list of StudyStep objects.
        """

    @abstractproperty
    def output_path(self):
        """
        Return the OUTPUT_PATH variable (if it exists).

        :returns: Returns OUTPUT_PATH if it exists, empty string otherwise.
        """

    @abstractproperty
    def name(self):
        """
        Getter for the name of a study specification.

        :returns: The name of the study described by the specification.
        """

    @name.setter
    def name(self, value):
        """
        Setter for the name of a study specification.

        :param value: String value representing the new name.
        """

    @abstractproperty
    def desc(self):
        """
        Getter for the description of a study specification.

        :returns: A string containing the description of the study
            specification.
        """

    @desc.setter
    def desc(self, value):
        """
        Setter for the description of a study specification.

        :param value: String value representing the new description.
        """

desc(value)

Setter for the description of a study specification.

Parameters:

Name Type Description Default
value

String value representing the new description.

 required
Source code in maestrowf/abstracts/specification.py 
124
125
126
127
128
129
130
@desc.setter
def desc(self, value):
    """
    Setter for the description of a study specification.

    :param value: String value representing the new description.
    """

get_parameters() abstractmethod

Generate a ParameterGenerator object from the global parameters.

Returns:

Type Description

A ParameterGenerator with data from the specification.
Source code in maestrowf/abstracts/specification.py 
75
76
77
78
79
80
81
@abstractmethod
def get_parameters(self):
    """
    Generate a ParameterGenerator object from the global parameters.

    :returns: A ParameterGenerator with data from the specification.
    """

get_study_environment() abstractmethod

Generate a StudyEnvironment object from the environment in the spec.

Returns:

Type Description

A StudyEnvironment object with the data in the specification.
Source code in maestrowf/abstracts/specification.py 
67
68
69
70
71
72
73
@abstractmethod
def get_study_environment(self):
    """
    Generate a StudyEnvironment object from the environment in the spec.

    :returns: A StudyEnvironment object with the data in the specification.
    """

get_study_steps() abstractmethod

Generate a list of StudySteps from the study in the specification.

Returns:

Type Description

A list of StudyStep objects.
Source code in maestrowf/abstracts/specification.py 
83
84
85
86
87
88
89
@abstractmethod
def get_study_steps(self):
    """
    Generate a list of StudySteps from the study in the specification.

    :returns: A list of StudyStep objects.
    """

load_specification(path)

Method for loading a study specification from a file.

Parameters:

Name Type Description Default
path

Path to a study specification.

 required

Returns:

Type Description

A specification object containing the information loaded from path.
Source code in maestrowf/abstracts/specification.py 
42
43
44
45
46
47
48
49
50
@abstractclassmethod
def load_specification(cls, path):
    """
    Method for loading a study specification from a file.

    :param path: Path to a study specification.
    :returns: A specification object containing the information loaded
              from path.
    """

load_specification_from_stream(stream)

Method for loading a study specification from a stream.

Parameters:

Name Type Description Default
stream

Raw text stream containing specification data.

 required

Returns:

Type Description

A specification object containing the information in string.
Source code in maestrowf/abstracts/specification.py 
52
53
54
55
56
57
58
59
@abstractclassmethod
def load_specification_from_stream(cls, stream):
    """
    Method for loading a study specification from a stream.

    :param stream: Raw text stream containing specification data.
    :returns: A specification object containing the information in string.
    """

name(value)

Setter for the name of a study specification.

Parameters:

Name Type Description Default
value

String value representing the new name.

 required
Source code in maestrowf/abstracts/specification.py 
107
108
109
110
111
112
113
@name.setter
def name(self, value):
    """
    Setter for the name of a study specification.

    :param value: String value representing the new name.
    """

output_path()

Return the OUTPUT_PATH variable (if it exists).

Returns:

Type Description

Returns OUTPUT_PATH if it exists, empty string otherwise.
Source code in maestrowf/abstracts/specification.py 
91
92
93
94
95
96
97
@abstractproperty
def output_path(self):
    """
    Return the OUTPUT_PATH variable (if it exists).

    :returns: Returns OUTPUT_PATH if it exists, empty string otherwise.
    """

verify() abstractmethod

Verify the whole specification.

Source code in maestrowf/abstracts/specification.py 
61
62
63
64
65
@abstractmethod
def verify(self):
    """
    Verify the whole specification.
    """

Substitution

Bases: EnvObject

Abstract class representing classes that perform value replacements.

Source code in maestrowf/abstracts/envobject.py 
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
@six.add_metaclass(ABCMeta)
class Substitution(EnvObject):
    """Abstract class representing classes that perform value replacements."""

    @abstractmethod
    def substitute(self, data):
        """
        Perform a replacement of some substring into data.

        The method takes the input string data and performs a replacement. This
        API is used to represent concepts such as variables or parameters that
        would want to be replaced within the string data.

        :param data: A string to perform a replacement on.
        :returns: A string equal to the original string data with substitutions
            made (if any were performed).
        """

substitute(data) abstractmethod

Perform a replacement of some substring into data.

The method takes the input string data and performs a replacement. This API is used to represent concepts such as variables or parameters that would want to be replaced within the string data.

Parameters:

Name Type Description Default
data

A string to perform a replacement on.

 required

Returns:

Type Description

A string equal to the original string data with substitutions made (if any were performed).
Source code in maestrowf/abstracts/envobject.py 
81
82
83
84
85
86
87
88
89
90
91
92
93
@abstractmethod
def substitute(self, data):
    """
    Perform a replacement of some substring into data.

    The method takes the input string data and performs a replacement. This
    API is used to represent concepts such as variables or parameters that
    would want to be replaced within the string data.

    :param data: A string to perform a replacement on.
    :returns: A string equal to the original string data with substitutions
        made (if any were performed).
    """