abacusai.pipeline — AbacusAI documentation

# abacusai.pipeline

## Classes

`PythonFunctionArgument` | A config class for python function arguments
`CodeSource` | Code source for python-based custom feature groups and models
`PipelineReference` | A reference to a pipeline to the objects it is run on.
`PipelineStep` | A step in a pipeline.
`PipelineVersion` | A version of a pipeline.
`AbstractApiClass` |
`Pipeline` | A Pipeline For Steps.

## Module Contents

class abacusai.pipeline.PythonFunctionArgument

  Bases: `abacusai.api_class.abstract.ApiClass`

  A config class for python function arguments

  Parameters:

    - variable_type (PythonFunctionArgumentType) – The type of the python function argument

    - name (str) – The name of the python function variable

    - is_required (bool) – Whether the argument is required

    - value (Any) – The value of the argument

    - pipeline_variable (str) – The name of the pipeline variable to use as the value

  variable_type: abacusai.api_class.enums.PythonFunctionArgumentType
  name: str
  is_required: bool
  value: Any
  pipeline_variable: str

class abacusai.pipeline.CodeSource(client, sourceType=None, sourceCode=None, applicationConnectorId=None, applicationConnectorInfo=None, packageRequirements=None, status=None, error=None, publishingMsg=None, moduleDependencies=None)

  Bases: `abacusai.return_class.AbstractApiClass`

  Code source for python-based custom feature groups and models

  Parameters:

    - client (ApiClient) – An authenticated API Client instance

    - sourceType (str) – The type of the source, one of TEXT, PYTHON, FILE_UPLOAD, or APPLICATION_CONNECTOR

    - sourceCode (str) – If the type of the source is TEXT, the raw text of the function

    - applicationConnectorId (str) – The Application Connector to fetch the code from

    - applicationConnectorInfo (str) – Args passed to the application connector to fetch the code

    - packageRequirements (list) – The pip package dependencies required to run the code

    - status (str) – The status of the code and validations

    - error (str) – If the status is failed, an error message describing what went wrong

    - publishingMsg (dict) – Warnings in the source code

    - moduleDependencies (list) – The list of internal modules dependencies required to run the code

  __repr__()

    Return repr(self).

  to_dict()

    Get a dict representation of the parameters in this class

    Returns:

      The dict value representation of the class parameters

    Return type:

      dict

  import_as_cell()

    Adds the source code as an unexecuted cell in the notebook.

class abacusai.pipeline.PipelineReference(client, pipelineReferenceId=None, pipelineId=None, objectType=None, datasetId=None, modelId=None, deploymentId=None, batchPredictionDescriptionId=None, modelMonitorId=None, notebookId=None, featureGroupId=None)

  Bases: `abacusai.return_class.AbstractApiClass`

  A reference to a pipeline to the objects it is run on.

  Parameters:

    - client (ApiClient) – An authenticated API Client instance

    - pipelineReferenceId (str) – The id of the reference.

    - pipelineId (str) – The id of the pipeline for the reference.

    - objectType (str) – The object type of the reference.

    - datasetId (str) – The dataset id of the reference.

    - modelId (str) – The model id of the reference.

    - deploymentId (str) – The deployment id of the reference.

    - batchPredictionDescriptionId (str) – The batch prediction description id of the reference.

    - modelMonitorId (str) – The model monitor id of the reference.

    - notebookId (str) – The notebook id of the reference.

    - featureGroupId (str) – The feature group id of the reference.

  __repr__()

    Return repr(self).

  to_dict()

    Get a dict representation of the parameters in this class

    Returns:

      The dict value representation of the class parameters

    Return type:

      dict

class abacusai.pipeline.PipelineStep(client, pipelineStepId=None, pipelineId=None, stepName=None, pipelineName=None, createdAt=None, updatedAt=None, pythonFunctionId=None, stepDependencies=None, cpuSize=None, memory=None, timeout=None, pythonFunction={}, codeSource={})

  Bases: `abacusai.return_class.AbstractApiClass`

  A step in a pipeline.

  Parameters:

    - client (ApiClient) – An authenticated API Client instance

    - pipelineStepId (str) – The reference to this step.

    - pipelineId (str) – The reference to the pipeline this step belongs to.

    - stepName (str) – The name of the step.

    - pipelineName (str) – The name of the pipeline this step is a part of.

    - createdAt (str) – The date and time which this step was created.

    - updatedAt (str) – The date and time when this step was last updated.

    - pythonFunctionId (str) – The python function_id.

    - stepDependencies (list[str]) – List of steps this step depends on.

    - cpuSize (str) – CPU size specified for the step function.

    - memory (int) – Memory in GB specified for the step function.

    - timeout (int) – Timeout for the step in minutes, default is 300 minutes.

    - pythonFunction (PythonFunction) – Information about the python function for the step.

    - codeSource (CodeSource) – Information about the source code of the step function.

  __repr__()

    Return repr(self).

  to_dict()

    Get a dict representation of the parameters in this class

    Returns:

      The dict value representation of the class parameters

    Return type:

      dict

  delete()

    Deletes a step from a pipeline.

    Parameters:

      pipeline_step_id (str) – The ID of the pipeline step.

  update(function_name=None, source_code=None, step_input_mappings=None, output_variable_mappings=None, step_dependencies=None, package_requirements=None, cpu_size=None, memory=None, timeout=None)

    Creates a step in a given pipeline.

    Parameters:

      - function_name (str) – The name of the Python function.

      - source_code (str) – Contents of a valid Python source code file. The source code should contain the transform feature group functions. A list of allowed imports and system libraries for each language is specified in the user functions documentation section.

      - step_input_mappings (List) – List of Python function arguments.

      - output_variable_mappings (List) – List of Python function outputs.

      - step_dependencies (list) – List of step names this step depends on.

      - package_requirements (list) – List of package requirement strings. For example: [‘numpy==1.2.3’, ‘pandas>=1.4.0’].

      - cpu_size (str) – Size of the CPU for the step function.

      - memory (int) – Memory (in GB) for the step function.

      - timeout (int) – Timeout for the pipeline step, default is 300 minutes.

    Returns:

      Object describing the pipeline.

    Return type:

      PipelineStep

  rename(step_name)

    Renames a step in a given pipeline.

    Parameters:

      step_name (str) – The name of the step.

    Returns:

      Object describing the pipeline.

    Return type:

      PipelineStep

  refresh()

    Calls describe and refreshes the current object’s fields

    Returns:

      The current object

    Return type:

      PipelineStep

  describe()

    Deletes a step from a pipeline.

    Parameters:

      pipeline_step_id (str) – The ID of the pipeline step.

    Returns:

      An object describing the pipeline step.

    Return type:

      PipelineStep

class abacusai.pipeline.PipelineVersion(client, pipelineName=None, pipelineId=None, pipelineVersion=None, createdAt=None, updatedAt=None, completedAt=None, status=None, error=None, stepVersions={}, codeSource={}, pipelineVariableMappings={})

  Bases: `abacusai.return_class.AbstractApiClass`

  A version of a pipeline.

  Parameters:

    - client (ApiClient) – An authenticated API Client instance

    - pipelineName (str) – The name of the pipeline this step is a part of.

    - pipelineId (str) – The reference to the pipeline this step belongs to.

    - pipelineVersion (str) – The reference to this pipeline version.

    - createdAt (str) – The date and time which this pipeline version was created.

    - updatedAt (str) – The date and time which this pipeline version was updated.

    - completedAt (str) – The date and time which this pipeline version was updated.

    - status (str) – The status of the pipeline version.

    - error (str) – The relevant error, if the status is FAILED.

    - stepVersions (PipelineStepVersion) – A list of the pipeline step versions.

    - codeSource (CodeSource) – information on the source code

    - pipelineVariableMappings (PythonFunctionArgument) – A description of the function variables into the pipeline.

  __repr__()

    Return repr(self).

  to_dict()

    Get a dict representation of the parameters in this class

    Returns:

      The dict value representation of the class parameters

    Return type:

      dict

  refresh()

    Calls describe and refreshes the current object’s fields

    Returns:

      The current object

    Return type:

      PipelineVersion

  describe()

    Describes a specified pipeline version

    Parameters:

      pipeline_version (str) – Unique string identifier for the pipeline version

    Returns:

      Object describing the pipeline version

    Return type:

      PipelineVersion

  reset(steps=None, include_downstream_steps=True)

    Reruns a pipeline version for the given steps and downstream steps if specified.

    Parameters:

      - steps (list) – List of pipeline step names to rerun.

      - include_downstream_steps (bool) – Whether to rerun downstream steps from the steps you have passed

    Returns:

      Object describing the pipeline version

    Return type:

      PipelineVersion

  list_logs()

    Gets the logs for the steps in a given pipeline version.

    Parameters:

      pipeline_version (str) – The id of the pipeline version.

    Returns:

      Object describing the logs for the steps in the pipeline.

    Return type:

      PipelineVersionLogs

  skip_pending_steps()

    Skips pending steps in a pipeline version.

    Parameters:

      pipeline_version (str) – The id of the pipeline version.

    Returns:

      Object describing the pipeline version

    Return type:

      PipelineVersion

  wait_for_pipeline(timeout=1200)

    A waiting call until all the stages in a pipeline version have completed.

    Parameters:

      timeout (int) – The waiting time given to the call to finish, if it doesn’t finish by the allocated time, the call is said to be timed out.

  get_status()

    Gets the status of the pipeline version.

    Returns:

      A string describing the status of a pipeline version (pending, running, complete, etc.).

    Return type:

      str

class abacusai.pipeline.AbstractApiClass(client, id)
  __eq__(other)

    Return self==value.

  _get_attribute_as_dict(attribute)

class abacusai.pipeline.Pipeline(client, pipelineName=None, pipelineId=None, createdAt=None, notebookId=None, cron=None, nextRunTime=None, isProd=None, warning=None, createdBy=None, steps={}, pipelineReferences={}, latestPipelineVersion={}, codeSource={}, pipelineVariableMappings={})

  Bases: `abacusai.return_class.AbstractApiClass`

  A Pipeline For Steps.

  Parameters:

    - client (ApiClient) – An authenticated API Client instance

    - pipelineName (str) – The name of the pipeline this step is a part of.

    - pipelineId (str) – The reference to the pipeline this step belongs to.

    - createdAt (str) – The date and time which the pipeline was created.

    - notebookId (str) – The reference to the notebook this pipeline belongs to.

    - cron (str) – A cron-style string that describes when this refresh policy is to be executed in UTC

    - nextRunTime (str) – The next time this pipeline will be run.

    - isProd (bool) – Whether this pipeline is a production pipeline.

    - warning (str) – Warning message for possible errors that might occur if the pipeline is run.

    - createdBy (str) – The email of the user who created the pipeline

    - steps (PipelineStep) – A list of the pipeline steps attached to the pipeline.

    - pipelineReferences (PipelineReference) – A list of references from the pipeline to other objects

    - latestPipelineVersion (PipelineVersion) – The latest version of the pipeline.

    - codeSource (CodeSource) – information on the source code

    - pipelineVariableMappings (PythonFunctionArgument) – A description of the function variables into the pipeline.

  __repr__()

    Return repr(self).

  to_dict()

    Get a dict representation of the parameters in this class

    Returns:

      The dict value representation of the class parameters

    Return type:

      dict

  refresh()

    Calls describe and refreshes the current object’s fields

    Returns:

      The current object

    Return type:

      Pipeline

  describe()

    Describes a given pipeline.

    Parameters:

      pipeline_id (str) – The ID of the pipeline to describe.

    Returns:

      An object describing a Pipeline

    Return type:

      Pipeline

  update(project_id=None, pipeline_variable_mappings=None, cron=None, is_prod=None)

    Updates a pipeline for executing multiple steps.

    Parameters:

      - project_id (str) – A unique string identifier for the pipeline.

      - pipeline_variable_mappings (List) – List of Python function arguments for the pipeline.

      - cron (str) – A cron-like string specifying the frequency of the scheduled pipeline runs.

      - is_prod (bool) – Whether the pipeline is a production pipeline or not.

    Returns:

      An object that describes a Pipeline.

    Return type:

      Pipeline

  rename(pipeline_name)

    Renames a pipeline.

    Parameters:

      pipeline_name (str) – The new name of the pipeline.

    Returns:

      An object that describes a Pipeline.

    Return type:

      Pipeline

  delete()

    Deletes a pipeline.

    Parameters:

      pipeline_id (str) – The ID of the pipeline to delete.

  list_versions(limit=200)

    Lists the pipeline versions for a specified pipeline

    Parameters:

      limit (int) – The maximum number of pipeline versions to return.

    Returns:

      A list of pipeline versions.

    Return type:

      list[PipelineVersion]

  run(pipeline_variable_mappings=None)

    Runs a specified pipeline with the arguments provided.

    Parameters:

      pipeline_variable_mappings (List) – List of Python function arguments for the pipeline.

    Returns:

      The object describing the pipeline

    Return type:

      PipelineVersion

  create_step(step_name, function_name=None, source_code=None, step_input_mappings=None, output_variable_mappings=None, step_dependencies=None, package_requirements=None, cpu_size=None, memory=None, timeout=None)

    Creates a step in a given pipeline.

    Parameters:

      - step_name (str) – The name of the step.

      - function_name (str) – The name of the Python function.

      - source_code (str) – Contents of a valid Python source code file. The source code should contain the transform feature group functions. A list of allowed imports and system libraries for each language is specified in the user functions documentation section.

      - step_input_mappings (List) – List of Python function arguments.

      - output_variable_mappings (List) – List of Python function outputs.

      - step_dependencies (list) – List of step names this step depends on.

      - package_requirements (list) – List of package requirement strings. For example: [‘numpy==1.2.3’, ‘pandas>=1.4.0’].

      - cpu_size (str) – Size of the CPU for the step function.

      - memory (int) – Memory (in GB) for the step function.

      - timeout (int) – Timeout for the step in minutes, default is 300 minutes.

    Returns:

      Object describing the pipeline.

    Return type:

      Pipeline

  describe_step_by_name(step_name)

    Describes a pipeline step by the step name.

    Parameters:

      step_name (str) – The name of the step.

    Returns:

      An object describing the pipeline step.

    Return type:

      PipelineStep

  unset_refresh_schedule()

    Deletes the refresh schedule for a given pipeline.

    Parameters:

      pipeline_id (str) – The id of the pipeline.

    Returns:

      Object describing the pipeline.

    Return type:

      Pipeline

  pause_refresh_schedule()

    Pauses the refresh schedule for a given pipeline.

    Parameters:

      pipeline_id (str) – The id of the pipeline.

    Returns:

      Object describing the pipeline.

    Return type:

      Pipeline

  resume_refresh_schedule()

    Resumes the refresh schedule for a given pipeline.

    Parameters:

      pipeline_id (str) – The id of the pipeline.

    Returns:

      Object describing the pipeline.

    Return type:

      Pipeline

  create_step_from_function(step_name, function, step_input_mappings=None, output_variable_mappings=None, step_dependencies=None, package_requirements=None, cpu_size=None, memory=None)

    Creates a step in the pipeline from a python function.

    Parameters:

      - step_name (str) – The name of the step.

      - function (callable) – The python function.

      - step_input_mappings (List[PythonFunctionArguments]) – List of Python function arguments.

      - output_variable_mappings (List[OutputVariableMapping]) – List of Python function ouputs.

      - step_dependencies (List[str]) – List of step names this step depends on.

      - package_requirements (list) – List of package requirement strings. For example: [‘numpy==1.2.3’, ‘pandas>=1.4.0’].

      - cpu_size (str) – Size of the CPU for the step function.

      - memory (int) – Memory (in GB) for the step function.

  wait_for_pipeline(timeout=1200)

    A waiting call until all the stages of the latest pipeline version is completed.

    Parameters:

      timeout (int) – The waiting time given to the call to finish, if it doesn’t finish by the allocated time, the call is said to be timed out.

  get_status()

    Gets the status of the pipeline version.

    Returns:

      A string describing the status of a pipeline version (pending, running, complete, etc.).

    Return type:

      str