Skip to content

Scriptadapter

Abstract Script Interfaces for generating scripts.

ScriptAdapter

Bases: object

Abstract class representing the interface for constructing scripts.

The ScriptAdapter abstract class is meant to provide a consistent high level interface to generate scripts automatically based on an ExecutionDAG. Adapters as a whole should only interface with the ExecutionDAG because it is ultimately the DAG that manages the state of tasks. Adapters attempt to bridge the 'how' in an abstract way such that the interface is refined to methods such as: - Generating a script with the proper syntax to submit. - Submitting a script using the proper command. - Checking job status.

Source code in maestrowf/abstracts/interfaces/scriptadapter.py 
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
@six.add_metaclass(ABCMeta)
class ScriptAdapter(object):
    """
    Abstract class representing the interface for constructing scripts.

    The ScriptAdapter abstract class is meant to provide a consistent high
    level interface to generate scripts automatically based on an ExecutionDAG.
    Adapters as a whole should only interface with the ExecutionDAG because it
    is ultimately the DAG that manages the state of tasks. Adapters attempt to
    bridge the 'how' in an abstract way such that the interface is refined to
    methods such as:
    - Generating a script with the proper syntax to submit.
    - Submitting a script using the proper command.
    - Checking job status.
    """

    def __init__(self, **kwargs):
        """
        Initialize a new instance of a ScriptAdapter.

        :param kwargs: The key value arguments for the ScriptAdapter instance.
        """
        self._exec = kwargs.pop("shell", "/bin/bash")
        LOGGER.debug("Shell set to '%s'.", self._exec)

    @abstractmethod
    def check_jobs(self, joblist):
        """
        For the given job list, query execution status.

        :param joblist: A list of job identifiers to be queried.
        :returns: The return code of the status query, and a dictionary of job
            identifiers to their status.
        """
        pass

    @abstractmethod
    def cancel_jobs(self, joblist):
        """
        For the given job list, cancel each job.

        :param joblist: A list of job identifiers to be cancelled.
        :returns: The return code to indicate if jobs were cancelled.
        """
        pass

    @abstractmethod
    def _write_script(self, ws_path, step):
        """
        Write a script to the workspace of a workflow step.

        The job_map optional parameter is a map of workflow step names to job
        identifiers. This parameter so far is only planned to be used when a
        study is configured to be launched in one go (more or less a script
        chain using a scheduler's dependency setting). The functionality of
        the parameter may change depending on both future intended use and
        derived classes.

        :param ws_path: Path to the workspace directory of the step.
        :param step: An instance of a StudyStep.
        :returns: Boolean value (True if the workflow step is to be scheduled,
            False otherwise) and the path to the written script.
        """
        pass

    def write_script(self, ws_path, step):
        """
        Generate the script for the specified StudyStep.

        :param ws_path: Workspace path for the step.
        :param step: An instance of a StudyStep class.
        :returns: A tuple containing a boolean set to True if step should be
            scheduled (False otherwise), path to the generate script, and path
            to the generated restart script (None if step cannot be restarted).
        """
        to_be_scheduled, script_path, restart_path = \
            self._write_script(ws_path, step)
        st = os.stat(script_path)
        os.chmod(script_path, st.st_mode | stat.S_IXUSR)

        if restart_path:
            st = os.stat(restart_path)
            os.chmod(restart_path, st.st_mode | stat.S_IXUSR)

        LOGGER.debug(
            "---------------------------------\n"
            "Script path:   %s\n"
            "Restart path:  %s\n"
            "Scheduled?:    %s\n"
            "---------------------------------\n",
            script_path, restart_path, to_be_scheduled
        )
        return to_be_scheduled, script_path, restart_path

    @abstractmethod
    def submit(self, step, path, cwd, job_map=None, env=None):
        """
        Submit a script to the scheduler.

        If cwd is specified, the submit method will operate outside of the path
        specified by the 'cwd' parameter.
        If env is specified, the submit method will set the environment
        variables for submission to the specified values. The 'env' parameter
        should be a dictionary of environment variables.

        :param step: An instance of a StudyStep.
        :param path: Path to the script to be executed.
        :param cwd: Path to the current working directory.
        :param job_map: A map of workflow step names to their job identifiers.
        :param env: A dict containing a modified environment for execution.
        :returns: The return code of the submission command and job identiifer.
        """
        pass

    @abstractproperty
    def extension(self):
        """
        Returns the extension that generated scripts will use.

        :returns: A string of the extension
        """
        pass

    @abstractproperty
    def key(self):
        """
        Return the key name for a ScriptAdapter..

        This is used to register the adapter in the ScriptAdapterFactory
        and when writing the workflow specification.
        """
        pass

__init__(**kwargs)

Initialize a new instance of a ScriptAdapter.

Parameters:

Name Type Description Default
kwargs

The key value arguments for the ScriptAdapter instance.

 {}
Source code in maestrowf/abstracts/interfaces/scriptadapter.py 
56
57
58
59
60
61
62
63
def __init__(self, **kwargs):
    """
    Initialize a new instance of a ScriptAdapter.

    :param kwargs: The key value arguments for the ScriptAdapter instance.
    """
    self._exec = kwargs.pop("shell", "/bin/bash")
    LOGGER.debug("Shell set to '%s'.", self._exec)

cancel_jobs(joblist) abstractmethod

For the given job list, cancel each job.

Parameters:

Name Type Description Default
joblist

A list of job identifiers to be cancelled.

 required

Returns:

Type Description

The return code to indicate if jobs were cancelled.
Source code in maestrowf/abstracts/interfaces/scriptadapter.py 
76
77
78
79
80
81
82
83
84
@abstractmethod
def cancel_jobs(self, joblist):
    """
    For the given job list, cancel each job.

    :param joblist: A list of job identifiers to be cancelled.
    :returns: The return code to indicate if jobs were cancelled.
    """
    pass

check_jobs(joblist) abstractmethod

For the given job list, query execution status.

Parameters:

Name Type Description Default
joblist

A list of job identifiers to be queried.

 required

Returns:

Type Description

The return code of the status query, and a dictionary of job identifiers to their status.
Source code in maestrowf/abstracts/interfaces/scriptadapter.py 
65
66
67
68
69
70
71
72
73
74
@abstractmethod
def check_jobs(self, joblist):
    """
    For the given job list, query execution status.

    :param joblist: A list of job identifiers to be queried.
    :returns: The return code of the status query, and a dictionary of job
        identifiers to their status.
    """
    pass

extension()

Returns the extension that generated scripts will use.

Returns:

Type Description

A string of the extension
Source code in maestrowf/abstracts/interfaces/scriptadapter.py 
154
155
156
157
158
159
160
161
@abstractproperty
def extension(self):
    """
    Returns the extension that generated scripts will use.

    :returns: A string of the extension
    """
    pass

key()

Return the key name for a ScriptAdapter..

This is used to register the adapter in the ScriptAdapterFactory and when writing the workflow specification.

Source code in maestrowf/abstracts/interfaces/scriptadapter.py 
163
164
165
166
167
168
169
170
171
@abstractproperty
def key(self):
    """
    Return the key name for a ScriptAdapter..

    This is used to register the adapter in the ScriptAdapterFactory
    and when writing the workflow specification.
    """
    pass

submit(step, path, cwd, job_map=None, env=None) abstractmethod

Submit a script to the scheduler.

If cwd is specified, the submit method will operate outside of the path specified by the 'cwd' parameter. If env is specified, the submit method will set the environment variables for submission to the specified values. The 'env' parameter should be a dictionary of environment variables.

Parameters:

Name Type Description Default
step

An instance of a StudyStep.

 required
path

Path to the script to be executed.

 required
cwd

Path to the current working directory.

 required
job_map

A map of workflow step names to their job identifiers.

 None
env

A dict containing a modified environment for execution.

 None

Returns:

Type Description

The return code of the submission command and job identiifer.
Source code in maestrowf/abstracts/interfaces/scriptadapter.py 
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
@abstractmethod
def submit(self, step, path, cwd, job_map=None, env=None):
    """
    Submit a script to the scheduler.

    If cwd is specified, the submit method will operate outside of the path
    specified by the 'cwd' parameter.
    If env is specified, the submit method will set the environment
    variables for submission to the specified values. The 'env' parameter
    should be a dictionary of environment variables.

    :param step: An instance of a StudyStep.
    :param path: Path to the script to be executed.
    :param cwd: Path to the current working directory.
    :param job_map: A map of workflow step names to their job identifiers.
    :param env: A dict containing a modified environment for execution.
    :returns: The return code of the submission command and job identiifer.
    """
    pass

write_script(ws_path, step)

Generate the script for the specified StudyStep.

Parameters:

Name Type Description Default
ws_path

Workspace path for the step.

 required
step

An instance of a StudyStep class.

 required

Returns:

Type Description

A tuple containing a boolean set to True if step should be scheduled (False otherwise), path to the generate script, and path to the generated restart script (None if step cannot be restarted).
Source code in maestrowf/abstracts/interfaces/scriptadapter.py 
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
def write_script(self, ws_path, step):
    """
    Generate the script for the specified StudyStep.

    :param ws_path: Workspace path for the step.
    :param step: An instance of a StudyStep class.
    :returns: A tuple containing a boolean set to True if step should be
        scheduled (False otherwise), path to the generate script, and path
        to the generated restart script (None if step cannot be restarted).
    """
    to_be_scheduled, script_path, restart_path = \
        self._write_script(ws_path, step)
    st = os.stat(script_path)
    os.chmod(script_path, st.st_mode | stat.S_IXUSR)

    if restart_path:
        st = os.stat(restart_path)
        os.chmod(restart_path, st.st_mode | stat.S_IXUSR)

    LOGGER.debug(
        "---------------------------------\n"
        "Script path:   %s\n"
        "Restart path:  %s\n"
        "Scheduled?:    %s\n"
        "---------------------------------\n",
        script_path, restart_path, to_be_scheduled
    )
    return to_be_scheduled, script_path, restart_path