Skip to content

Envobject

The collection of basic classes that can be used for an environment.

Dependency

Bases: Substitution

Abstract object representing a dependency.

The Dependency base class is intended to be used to capture external items the workflow is dependent on. These items include (but are not limited to):

* Remotely stored repositories (such as bitbucket)
* Paths located on the filesystem that hold required files or binaries
* Binaries that are required to be installed using a specific package
  manager
* External APIs that a workflow needs to pull data from

The goal of this base class is to make it so that this package is able to pull external dependencies in a consistent manner.

Source code in maestrowf/abstracts/envobject.py 
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
@six.add_metaclass(ABCMeta)
class Dependency(Substitution):
    """
    Abstract object representing a dependency.

    The Dependency base class is intended to be used to capture external items
    the workflow is dependent on. These items include (but are not limited to):

        * Remotely stored repositories (such as bitbucket)
        * Paths located on the filesystem that hold required files or binaries
        * Binaries that are required to be installed using a specific package
          manager
        * External APIs that a workflow needs to pull data from

    The goal of this base class is to make it so that this package is able to
    pull external dependencies in a consistent manner.
    """

    @abstractmethod
    def acquire(self, substitutions=None):
        """
        Acquire the dependency as specfied by the class instance.

        Subclasses that implement this interface should raise exceptions during
        acquisition should they be unable to retrieve their specified
        dependency. It is assumed that if acquiring throws an exception that
        the study cannot proceed forward.

        :param substitutions: List of Substitution objects that can be applied.
        """

acquire(substitutions=None) abstractmethod

Acquire the dependency as specfied by the class instance.

Subclasses that implement this interface should raise exceptions during acquisition should they be unable to retrieve their specified dependency. It is assumed that if acquiring throws an exception that the study cannot proceed forward.

Parameters:

Name Type Description Default
substitutions

List of Substitution objects that can be applied.

 None
Source code in maestrowf/abstracts/envobject.py 
145
146
147
148
149
150
151
152
153
154
155
156
@abstractmethod
def acquire(self, substitutions=None):
    """
    Acquire the dependency as specfied by the class instance.

    Subclasses that implement this interface should raise exceptions during
    acquisition should they be unable to retrieve their specified
    dependency. It is assumed that if acquiring throws an exception that
    the study cannot proceed forward.

    :param substitutions: List of Substitution objects that can be applied.
    """

EnvObject

An abstract class representing objects that exist in a study's environment.

The EnvObject is meant to be used to represent entities in the larger environment that affect the execution of a study (and therefore jobs). This abstract base class should be used to represent things such as data dependencies, code dependencies, variables, aliases, etc. The only method we require is _verify, to allow users to verify that they've provided the minimal information for the object to be valid.

Source code in maestrowf/abstracts/envobject.py 
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
@six.add_metaclass(ABCMeta)
class EnvObject:
    """
    An abstract class representing objects that exist in a study's environment.

    The EnvObject is meant to be used to represent entities in the larger
    environment that affect the execution of a study (and therefore jobs).
    This abstract base class should be used to represent things such as data
    dependencies, code dependencies, variables, aliases, etc. The only method
    we require is _verify, to allow users to verify that they've provided the
    minimal information for the object to be valid.
    """

    @abstractmethod
    def _verify(self):
        """
        Verify that the object is valid.

        Subclasses that inherit from the EnvObject abstract class are expected
        to provide a method for asserting that the contents contained within
        the object are valid. Valid can range anywhere from asserting that all
        expected member variables are populated to asserting specific values of
        members, etc.

        :returns: True if the EnvObject is verified, False otherwise.
        """

    def _verification(self, error):
        """
        A wrapper method for verifying for using a custom error message.

        :param error: String containing a custom error message.
        """
        if not self._verify():
            LOGGER.exception(error)
            raise ValueError(error)

Source

Bases: EnvObject

Abstract class representing classes that alter environment sourcing.

WARNING: The API for this class is still in development. The Source environment class is meant to provide a way to programmatically set environment settings that binaries or other scripts may require in the workflow. Such settings that are intended to be captured are:

* Exporting of shell/environment variables (using 'export')
* Setting of an environment package with the 'use' command
Source code in maestrowf/abstracts/envobject.py 
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
class Source(EnvObject):
    """
    Abstract class representing classes that alter environment sourcing.

    WARNING: The API for this class is still in development.
    The Source environment class is meant to provide a way to programmatically
    set environment settings that binaries or other scripts may require in the
    workflow. Such settings that are intended to be captured are:

        * Exporting of shell/environment variables (using 'export')
        * Setting of an environment package with the 'use' command
    """

    @abstractmethod
    def apply(self, data):
        """
        Apply the Source to some string data.

        Subclasses of Source should use this method in order to apply an
        environment altering change. The 'data' parameter should be a string
        representing a command to apply Source to or a list of other commands
        that Source should be included with.

        :param data: A string representing a command or set of other sources.
        :returns: A string with the Source applied.
        """

apply(data) abstractmethod

Apply the Source to some string data.

Subclasses of Source should use this method in order to apply an environment altering change. The 'data' parameter should be a string representing a command to apply Source to or a list of other commands that Source should be included with.

Parameters:

Name Type Description Default
data

A string representing a command or set of other sources.

 required

Returns:

Type Description

A string with the Source applied.
Source code in maestrowf/abstracts/envobject.py 
109
110
111
112
113
114
115
116
117
118
119
120
121
@abstractmethod
def apply(self, data):
    """
    Apply the Source to some string data.

    Subclasses of Source should use this method in order to apply an
    environment altering change. The 'data' parameter should be a string
    representing a command to apply Source to or a list of other commands
    that Source should be included with.

    :param data: A string representing a command or set of other sources.
    :returns: A string with the Source applied.
    """

Substitution

Bases: EnvObject

Abstract class representing classes that perform value replacements.

Source code in maestrowf/abstracts/envobject.py 
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
@six.add_metaclass(ABCMeta)
class Substitution(EnvObject):
    """Abstract class representing classes that perform value replacements."""

    @abstractmethod
    def substitute(self, data):
        """
        Perform a replacement of some substring into data.

        The method takes the input string data and performs a replacement. This
        API is used to represent concepts such as variables or parameters that
        would want to be replaced within the string data.

        :param data: A string to perform a replacement on.
        :returns: A string equal to the original string data with substitutions
            made (if any were performed).
        """

substitute(data) abstractmethod

Perform a replacement of some substring into data.

The method takes the input string data and performs a replacement. This API is used to represent concepts such as variables or parameters that would want to be replaced within the string data.

Parameters:

Name Type Description Default
data

A string to perform a replacement on.

 required

Returns:

Type Description

A string equal to the original string data with substitutions made (if any were performed).
Source code in maestrowf/abstracts/envobject.py 
81
82
83
84
85
86
87
88
89
90
91
92
93
@abstractmethod
def substitute(self, data):
    """
    Perform a replacement of some substring into data.

    The method takes the input string data and performs a replacement. This
    API is used to represent concepts such as variables or parameters that
    would want to be replaced within the string data.

    :param data: A string to perform a replacement on.
    :returns: A string equal to the original string data with substitutions
        made (if any were performed).
    """