kfp.dsl package

kfp.dsl.RUN_ID_PLACEHOLDER

kfp.dsl.EXECUTION_ID_PLACEHOLDER

class kfp.dsl.BaseOp(name: str, init_containers: Optional[List[kfp.dsl._container_op.UserContainer]] = None, sidecars: Optional[List[kfp.dsl._container_op.Sidecar]] = None, is_exit_handler: bool = False)[source]

Bases: object

Base operator.

Parameters

name – the name of the op. It does not have to be unique within a pipeline because the pipeline will generates a unique new name in case of conflicts.
init_containers – the list of UserContainer objects describing the InitContainer to deploy before the main container.
sidecars – the list of Sidecar objects describing the sidecar containers to deploy together with the main container.
is_exit_handler – Deprecated.

add_affinity(affinity: kubernetes.client.models.v1_affinity.V1Affinity)[source]

Add K8s Affinity.

Parameters

affinity – Kubernetes affinity For detailed spec, check affinity definition
https – //github.com/kubernetes-client/python/blob/master/kubernetes/client/models/v1_affinity.py

Example:

V1Affinity(
    node_affinity=V1NodeAffinity(
        required_during_scheduling_ignored_during_execution=V1NodeSelector(
            node_selector_terms=[V1NodeSelectorTerm(
                match_expressions=[V1NodeSelectorRequirement(
                    key='beta.kubernetes.io/instance-type',
                    operator='In',
                    values=['p2.xlarge'])])])))

add_init_container(init_container: kfp.dsl._container_op.UserContainer)[source]

Add a init container to the Op.

Parameters: init_container – UserContainer object.

add_node_selector_constraint(label_name: Union[str, kfp.dsl._pipeline_param.PipelineParam], value: Union[str, kfp.dsl._pipeline_param.PipelineParam])[source]

Add a constraint for nodeSelector.

Each constraint is a key-value pair label. For the container to be eligible to run on a node, the node must have each of the constraints appeared as labels.

Parameters

label_name (Union[str, PipelineParam]) – The name of the constraint label.
value (Union[str, PipelineParam]) – The value of the constraint label.

add_pod_annotation(name: str, value: str)[source]

Adds a pod’s metadata annotation.

Parameters

name – The name of the annotation.
value – The value of the annotation.

add_pod_label(name: str, value: str)[source]

Adds a pod’s metadata label.

Parameters

name – The name of the label.
value – The value of the label.

add_sidecar(sidecar: kfp.dsl._container_op.Sidecar)[source]

Add a sidecar to the Op.

Parameters: sidecar – SideCar object.

add_toleration(tolerations: kubernetes.client.models.v1_toleration.V1Toleration)[source]

Add K8s tolerations.

Parameters: tolerations – Kubernetes toleration For detailed spec, check toleration definition https://github.com/kubernetes-client/python/blob/master/kubernetes/client/models/v1_toleration.py

add_volume(volume)[source]

Add K8s volume to the container.

Parameters

volume – Kubernetes volumes For detailed spec, check volume definition
https – //github.com/kubernetes-client/python/blob/master/kubernetes/client/models/v1_volume.py

after(*ops)[source]: Specify explicit dependency on other ops.

apply(mod_func)[source]

Applies a modifier function to self.

The function should return the passed object. This is needed to chain “extention methods” to this class.

Example:

from kfp.gcp import use_gcp_secret
task = (
    train_op(...)
        .set_memory_request('1G')
        .apply(use_gcp_secret('user-gcp-sa'))
        .set_memory_limit('2G')
)

attrs_with_pipelineparams = ['node_selector', 'volumes', 'pod_annotations', 'pod_labels', 'num_retries', 'init_containers', 'sidecars', 'tolerations']

property inputs: List of PipelineParams that will be converted into input parameters (io.argoproj.workflow.v1alpha1.Inputs) for the argo workflow.

set_caching_options(enable_caching: bool) → kfp.dsl._container_op.BaseOp[source]

Sets caching options for the Op.

Parameters: enable_caching – Whether or not to enable caching for this task.
Returns: Self return to allow chained setting calls.

set_display_name(name: str)[source]

set_retry(num_retries: int, policy: Optional[str] = None, backoff_duration: Optional[str] = None, backoff_factor: Optional[float] = None, backoff_max_duration: Optional[str] = None)[source]

Sets the number of times the task is retried until it’s declared failed.

Parameters

num_retries – Number of times to retry on failures.
policy – Retry policy name.
backoff_duration – The time interval between retries. Defaults to an immediate retry. In case you specify a simple number, the unit defaults to seconds. You can also specify a different unit, for instance, 2m (2 minutes), 1h (1 hour).
backoff_factor – The exponential backoff factor applied to backoff_duration. For example, if backoff_duration=”60” (60 seconds) and backoff_factor=2, the first retry will happen after 60 seconds, then after 120, 240, and so on.
backoff_max_duration – The maximum interval that can be reached with the backoff strategy.

set_timeout(seconds: int)[source]

Sets the timeout for the task in seconds.

Parameters: seconds – Number of seconds.

class kfp.dsl.Condition(condition, name=None)[source]

Bases: kfp.dsl._ops_group.OpsGroup

Represents an condition group with a condition.

Parameters

condition (ConditionOperator) – the condition.
name (str) – name of the condition

Example: ::

with Condition(param1==’pizza’, ‘[param1 is pizza]’): op1 =: ContainerOp(…) op2 = ContainerOp(…)

after(*ops): Specify explicit dependency on other ops.

remove_op_recursive(op)

class kfp.dsl.ContainerOp(name: str, image: str, command: Optional[Union[str, List[str]]] = None, arguments: Optional[Union[str, int, float, bool, kfp.dsl._pipeline_param.PipelineParam, List]] = None, init_containers: Optional[List[kfp.dsl._container_op.UserContainer]] = None, sidecars: Optional[List[kfp.dsl._container_op.Sidecar]] = None, container_kwargs: Optional[Dict] = None, artifact_argument_paths: Optional[List[kfp.dsl._container_op.InputArgumentPath]] = None, file_outputs: Optional[Dict[str, str]] = None, output_artifact_paths: Optional[Dict[str, str]] = None, is_exit_handler: bool = False, pvolumes: Optional[Dict[str, kubernetes.client.models.v1_volume.V1Volume]] = None)[source]

Bases: kfp.dsl._container_op.BaseOp

Represents an op implemented by a container image.

Parameters

name – the name of the op. It does not have to be unique within a pipeline because the pipeline will generates a unique new name in case of conflicts.
image – the container image name, such as ‘python:3.5-jessie’
command – the command to run in the container. If None, uses default CMD in defined in container.
arguments – the arguments of the command. The command can include “%s” and supply a PipelineParam as the string replacement. For example, (‘echo %s’ % input_param). At container run time the argument will be ‘echo param_value’.
init_containers – the list of UserContainer objects describing the InitContainer to deploy before the main container.
sidecars – the list of Sidecar objects describing the sidecar containers to deploy together with the main container.
container_kwargs – the dict of additional keyword arguments to pass to the op’s Container definition.
artifact_argument_paths – Optional. Maps input artifact arguments (values or references) to the local file paths where they’ll be placed. At pipeline run time, the value of the artifact argument is saved to a local file with specified path. This parameter is only needed when the input file paths are hard-coded in the program. Otherwise it’s better to pass input artifact placement paths by including artifact arguments in the command-line using the InputArgumentPath class instances.
file_outputs –
Maps output names to container local output file paths. The system will take the data from those files and will make it available for passing to downstream tasks. For each output in the file_outputs map there will be a corresponding output reference available in the task.outputs dictionary. These output references can be passed to the other tasks as arguments. The following output names are handled specially by the frontend and

backend: “mlpipeline-ui-metadata” and “mlpipeline-metrics”.
output_artifact_paths –
Deprecated. Maps output artifact labels to local artifact file paths. Deprecated: Use file_outputs instead. It now

supports big data outputs.
is_exit_handler – Deprecated. This is no longer needed.
pvolumes –
Dictionary for the user to match a path on the op’s fs with a V1Volume or it inherited type.

E.g {“/my/path”: vol, “/mnt”: other_op.pvolumes[“/output”]}.

Example:

from kfp import dsl
from kubernetes.client.models import V1EnvVar, V1SecretKeySelector
@dsl.pipeline(
    name='foo',
    description='hello world')
def foo_pipeline(tag: str, pull_image_policy: str):
  # any attributes can be parameterized (both serialized string or actual PipelineParam)
  op = dsl.ContainerOp(name='foo',
                      image='busybox:%s' % tag,
                      # pass in init_container list
                      init_containers=[dsl.UserContainer('print', 'busybox:latest', command='echo "hello"')],
                      # pass in sidecars list
                      sidecars=[dsl.Sidecar('print', 'busybox:latest', command='echo "hello"')],
                      # pass in k8s container kwargs
                      container_kwargs={'env': [V1EnvVar('foo', 'bar')]},
  )
  # set `imagePullPolicy` property for `container` with `PipelineParam`
  op.container.set_image_pull_policy(pull_image_policy)
  # add sidecar with parameterized image tag
  # sidecar follows the argo sidecar swagger spec
  op.add_sidecar(dsl.Sidecar('redis', 'redis:%s' % tag).set_image_pull_policy('Always'))

add_affinity(affinity: kubernetes.client.models.v1_affinity.V1Affinity)

Add K8s Affinity.

Parameters

affinity – Kubernetes affinity For detailed spec, check affinity definition
https – //github.com/kubernetes-client/python/blob/master/kubernetes/client/models/v1_affinity.py

Example:

V1Affinity(
    node_affinity=V1NodeAffinity(
        required_during_scheduling_ignored_during_execution=V1NodeSelector(
            node_selector_terms=[V1NodeSelectorTerm(
                match_expressions=[V1NodeSelectorRequirement(
                    key='beta.kubernetes.io/instance-type',
                    operator='In',
                    values=['p2.xlarge'])])])))

add_init_container(init_container: kfp.dsl._container_op.UserContainer)

Add a init container to the Op.

Parameters: init_container – UserContainer object.

add_node_selector_constraint(label_name: Union[str, kfp.dsl._pipeline_param.PipelineParam], value: Union[str, kfp.dsl._pipeline_param.PipelineParam]) → kfp.dsl._container_op.ContainerOp[source]

Sets accelerator type requirement for this task.

When compiling for v2, this function can be optionally used with set_gpu_limit to set the number of accelerator required. Otherwise, by default the number requested will be 1.

Parameters

label_name – The name of the constraint label. For v2, only ‘cloud.google.com/gke-accelerator’ is supported now.
value – The name of the accelerator. For v2, available values include ‘nvidia-tesla-k80’, ‘tpu-v3’.

Returns

self return to allow chained call with other resource specification.

add_pod_annotation(name: str, value: str)

Adds a pod’s metadata annotation.

Parameters

name – The name of the annotation.
value – The value of the annotation.

add_pod_label(name: str, value: str)

Adds a pod’s metadata label.

Parameters

name – The name of the label.
value – The value of the label.

add_pvolumes(pvolumes: Optional[Dict[str, kubernetes.client.models.v1_volume.V1Volume]] = None)[source]

Updates the existing pvolumes dict, extends volumes and volume_mounts and redefines the pvolume attribute.

Parameters: pvolumes – Dictionary. Keys are mount paths, values are Kubernetes volumes or inherited types (e.g. PipelineVolumes).

add_sidecar(sidecar: kfp.dsl._container_op.Sidecar)

Add a sidecar to the Op.

Parameters: sidecar – SideCar object.

add_toleration(tolerations: kubernetes.client.models.v1_toleration.V1Toleration)

Add K8s tolerations.

Parameters: tolerations – Kubernetes toleration For detailed spec, check toleration definition https://github.com/kubernetes-client/python/blob/master/kubernetes/client/models/v1_toleration.py

add_volume(volume)

Add K8s volume to the container.

Parameters

volume – Kubernetes volumes For detailed spec, check volume definition
https – //github.com/kubernetes-client/python/blob/master/kubernetes/client/models/v1_volume.py

after(*ops): Specify explicit dependency on other ops.

apply(mod_func)

Applies a modifier function to self.

The function should return the passed object. This is needed to chain “extention methods” to this class.

Example:

from kfp.gcp import use_gcp_secret
task = (
    train_op(...)
        .set_memory_request('1G')
        .apply(use_gcp_secret('user-gcp-sa'))
        .set_memory_limit('2G')
)

property arguments

attrs_with_pipelineparams = ['node_selector', 'volumes', 'pod_annotations', 'pod_labels', 'num_retries', 'init_containers', 'sidecars', 'tolerations']

property command

property container

Container object that represents the container property in io.argoproj.workflow.v1alpha1.Template. Can be used to update the container configurations.

Example:

import kfp.dsl as dsl
from kubernetes.client.models import V1EnvVar

@dsl.pipeline(name='example_pipeline')
def immediate_value_pipeline():
  op1 = (dsl.ContainerOp(name='example', image='nginx:alpine')
          .container
              .add_env_variable(V1EnvVar(name='HOST',
              value='foo.bar'))
              .add_env_variable(V1EnvVar(name='PORT', value='80'))
              .parent # return the parent `ContainerOp`
          )

property container_spec

property env_variables

property image

property inputs: List of PipelineParams that will be converted into input parameters (io.argoproj.workflow.v1alpha1.Inputs) for the argo workflow.

property is_v2

set_caching_options(enable_caching: bool) → kfp.dsl._container_op.BaseOp

Sets caching options for the Op.

Parameters: enable_caching – Whether or not to enable caching for this task.
Returns: Self return to allow chained setting calls.

set_display_name(name: str)

set_retry(num_retries: int, policy: Optional[str] = None, backoff_duration: Optional[str] = None, backoff_factor: Optional[float] = None, backoff_max_duration: Optional[str] = None)

Sets the number of times the task is retried until it’s declared failed.

Parameters

num_retries – Number of times to retry on failures.
policy – Retry policy name.
backoff_duration – The time interval between retries. Defaults to an immediate retry. In case you specify a simple number, the unit defaults to seconds. You can also specify a different unit, for instance, 2m (2 minutes), 1h (1 hour).
backoff_factor – The exponential backoff factor applied to backoff_duration. For example, if backoff_duration=”60” (60 seconds) and backoff_factor=2, the first retry will happen after 60 seconds, then after 120, 240, and so on.
backoff_max_duration – The maximum interval that can be reached with the backoff strategy.

set_timeout(seconds: int)

Sets the timeout for the task in seconds.

Parameters: seconds – Number of seconds.

class kfp.dsl.ExitHandler(exit_op: kfp.dsl._container_op.ContainerOp)[source]

Bases: kfp.dsl._ops_group.OpsGroup

Represents an exit handler that is invoked upon exiting a group of ops.

Parameters: exit_op – An operator invoked at exiting a group of ops.
Raises: ValueError – Raised if the exit_op is invalid.

Example

exit_op = ContainerOp(...)
with ExitHandler(exit_op):
  op1 = ContainerOp(...)
  op2 = ContainerOp(...)

after(*ops): Specify explicit dependency on other ops.

remove_op_recursive(op)

class kfp.dsl.InputArgumentPath(argument, input=None, path=None)[source]: Bases: object

class kfp.dsl.ParallelFor(loop_args: Union[List[Union[int, float, str, Dict[str, Any]]], kfp.dsl._pipeline_param.PipelineParam], parallelism: Optional[int] = None)[source]

Bases: kfp.dsl._ops_group.OpsGroup

Represents a parallel for loop over a static set of items.

Example

In this case op1 would be executed twice, once with case args=['echo 1'] and once with case args=['echo 2']:

with dsl.ParallelFor([{'a': 1, 'b': 10}, {'a': 2, 'b': 20}]) as item:
  op1 = ContainerOp(..., args=['echo {}'.format(item.a)])
  op2 = ContainerOp(..., args=['echo {}'.format(item.b])

TYPE_NAME = 'for_loop'

after(*ops): Specify explicit dependency on other ops.

remove_op_recursive(op)

class kfp.dsl.PipelineConf[source]

Bases: object

PipelineConf contains pipeline level settings.

add_op_transformer(transformer)[source]

Configures the op_transformers which will be applied to all ops in the pipeline. The ops can be ResourceOp, VolumeOp, or ContainerOp.

Parameters: transformer – A function that takes a kfp Op as input and returns a kfp Op

property data_passing_method

set_default_pod_node_selector(label_name: str, value: str)[source]

Add a constraint for nodeSelector for a pipeline.

Each constraint is a key-value pair label.

For the container to be eligible to run on a node, the node must have each of the constraints appeared as labels.

Parameters

label_name – The name of the constraint label.
value – The value of the constraint label.

set_dns_config(dns_config: kubernetes.client.models.v1_pod_dns_config.V1PodDNSConfig)[source]

Set the dnsConfig to be given to each pod.

Parameters: dns_config – Kubernetes V1PodDNSConfig For detailed description, check Kubernetes V1PodDNSConfig definition https://github.com/kubernetes-client/python/blob/master/kubernetes/docs/V1PodDNSConfig.md

Example

import kfp
from kubernetes.client.models import V1PodDNSConfig, V1PodDNSConfigOption
pipeline_conf = kfp.dsl.PipelineConf()
pipeline_conf.set_dns_config(dns_config=V1PodDNSConfig(
    nameservers=["1.2.3.4"],
    options=[V1PodDNSConfigOption(name="ndots", value="2")],
))

set_image_pull_policy(policy: str)[source]

Configures the default image pull policy.

Parameters: policy – the pull policy, has to be one of: Always, Never, IfNotPresent. For more info: https://github.com/kubernetes-client/python/blob/10a7f95435c0b94a6d949ba98375f8cc85a70e5a/kubernetes/docs/V1Container.md

set_image_pull_secrets(image_pull_secrets)[source]

Configures the pipeline level imagepullsecret.

Parameters: image_pull_secrets – a list of Kubernetes V1LocalObjectReference For detailed description, check Kubernetes V1LocalObjectReference definition https://github.com/kubernetes-client/python/blob/master/kubernetes/docs/V1LocalObjectReference.md

set_parallelism(max_num_pods: int)[source]

Configures the max number of total parallel pods that can execute at the same time in a workflow.

Parameters: max_num_pods – max number of total parallel pods.

set_pod_disruption_budget(min_available: Union[int, str])[source]

PodDisruptionBudget holds the number of concurrent disruptions that you allow for pipeline Pods.

Parameters: min_available (Union[int, str]) – An eviction is allowed if at least “minAvailable” pods selected by “selector” will still be available after the eviction, i.e. even in the absence of the evicted pod. So for example you can prevent all voluntary evictions by specifying “100%”. “minAvailable” can be either an absolute number or a percentage.

set_timeout(seconds: int)[source]

Configures the pipeline level timeout.

Parameters: seconds – number of seconds for timeout

set_ttl_seconds_after_finished(seconds: int)[source]

Configures the ttl after the pipeline has finished.

Parameters: seconds – number of seconds for the workflow to be garbage collected after it is finished.

class kfp.dsl.PipelineExecutionMode(value)[source]

Bases: enum.Enum

An enumeration.

V1_LEGACY = 1

V2_COMPATIBLE = 2

V2_ENGINE = 3

class kfp.dsl.PipelineParam(name: str, op_name: Optional[str] = None, value: Optional[str] = None, param_type: Optional[Union[str, Dict]] = None, pattern: Optional[str] = None)[source]

Bases: object

Representing a future value that is passed between pipeline components.

A PipelineParam object can be used as a pipeline function argument so that it will be a pipeline parameter that shows up in ML Pipelines system UI. It can also represent an intermediate value passed between components.

Parameters

name – name of the pipeline parameter.
op_name – the name of the operation that produces the PipelineParam. None means it is not produced by any operator, so if None, either user constructs it directly (for providing an immediate value), or it is a pipeline function argument.
value – The actual value of the PipelineParam. If provided, the PipelineParam is “resolved” immediately. For now, we support string only.
param_type – the type of the PipelineParam.
pattern – the serialized string regex pattern this pipeline parameter created from.

Raises: ValueError in name or op_name contains invalid characters, or both: op_name and value are set.

property full_name: Unique name in the argo yaml for the PipelineParam.

ignore_type()[source]: ignore_type ignores the type information such that type checking would also pass.

to_struct()[source]

class kfp.dsl.PipelineVolume(pvc: Optional[str] = None, volume: Optional[kubernetes.client.models.v1_volume.V1Volume] = None, **kwargs)[source]

Bases: kubernetes.client.models.v1_volume.V1Volume

Representing a volume that is passed between pipeline operators and is.

to be mounted by a ContainerOp or its inherited type.

A PipelineVolume object can be used as an extention of the pipeline function’s filesystem. It may then be passed between ContainerOps, exposing dependencies.

TODO(https://github.com/kubeflow/pipelines/issues/4822): Determine the: stability level of this feature.

Parameters

pvc – The name of an existing PVC
volume – Create a deep copy out of a V1Volume or PipelineVolume with no deps

Raises

ValueError – If volume is not None and kwargs is not None If pvc is not None and kwargs.pop(“name”) is not None

after(*ops)[source]

Creates a duplicate of self with the required dependecies excluding the redundant dependenices.

Parameters: *ops – Pipeline operators to add as dependencies

attribute_map = {'aws_elastic_block_store': 'awsElasticBlockStore', 'azure_disk': 'azureDisk', 'azure_file': 'azureFile', 'cephfs': 'cephfs', 'cinder': 'cinder', 'config_map': 'configMap', 'csi': 'csi', 'downward_api': 'downwardAPI', 'empty_dir': 'emptyDir', 'fc': 'fc', 'flex_volume': 'flexVolume', 'flocker': 'flocker', 'gce_persistent_disk': 'gcePersistentDisk', 'git_repo': 'gitRepo', 'glusterfs': 'glusterfs', 'host_path': 'hostPath', 'iscsi': 'iscsi', 'name': 'name', 'nfs': 'nfs', 'persistent_volume_claim': 'persistentVolumeClaim', 'photon_persistent_disk': 'photonPersistentDisk', 'portworx_volume': 'portworxVolume', 'projected': 'projected', 'quobyte': 'quobyte', 'rbd': 'rbd', 'scale_io': 'scaleIO', 'secret': 'secret', 'storageos': 'storageos', 'vsphere_volume': 'vsphereVolume'}

property aws_elastic_block_store

E501

Returns: The aws_elastic_block_store of this V1Volume. # noqa: E501
Return type: V1AWSElasticBlockStoreVolumeSource
Type: Gets the aws_elastic_block_store of this V1Volume. # noqa

property azure_disk

E501

Returns: The azure_disk of this V1Volume. # noqa: E501
Return type: V1AzureDiskVolumeSource
Type: Gets the azure_disk of this V1Volume. # noqa

property azure_file

E501

Returns: The azure_file of this V1Volume. # noqa: E501
Return type: V1AzureFileVolumeSource
Type: Gets the azure_file of this V1Volume. # noqa

property cephfs

E501

Returns: The cephfs of this V1Volume. # noqa: E501
Return type: V1CephFSVolumeSource
Type: Gets the cephfs of this V1Volume. # noqa

property cinder

E501

Returns: The cinder of this V1Volume. # noqa: E501
Return type: V1CinderVolumeSource
Type: Gets the cinder of this V1Volume. # noqa

property config_map

E501

Returns: The config_map of this V1Volume. # noqa: E501
Return type: V1ConfigMapVolumeSource
Type: Gets the config_map of this V1Volume. # noqa

property csi

E501

Returns: The csi of this V1Volume. # noqa: E501
Return type: V1CSIVolumeSource
Type: Gets the csi of this V1Volume. # noqa

property downward_api

E501

Returns: The downward_api of this V1Volume. # noqa: E501
Return type: V1DownwardAPIVolumeSource
Type: Gets the downward_api of this V1Volume. # noqa

property empty_dir

E501

Returns: The empty_dir of this V1Volume. # noqa: E501
Return type: V1EmptyDirVolumeSource
Type: Gets the empty_dir of this V1Volume. # noqa

property fc

E501

Returns: The fc of this V1Volume. # noqa: E501
Return type: V1FCVolumeSource
Type: Gets the fc of this V1Volume. # noqa

property flex_volume

E501

Returns: The flex_volume of this V1Volume. # noqa: E501
Return type: V1FlexVolumeSource
Type: Gets the flex_volume of this V1Volume. # noqa

property flocker

E501

Returns: The flocker of this V1Volume. # noqa: E501
Return type: V1FlockerVolumeSource
Type: Gets the flocker of this V1Volume. # noqa

property gce_persistent_disk

E501

Returns: The gce_persistent_disk of this V1Volume. # noqa: E501
Return type: V1GCEPersistentDiskVolumeSource
Type: Gets the gce_persistent_disk of this V1Volume. # noqa

property git_repo

E501

Returns: The git_repo of this V1Volume. # noqa: E501
Return type: V1GitRepoVolumeSource
Type: Gets the git_repo of this V1Volume. # noqa

property glusterfs

E501

Returns: The glusterfs of this V1Volume. # noqa: E501
Return type: V1GlusterfsVolumeSource
Type: Gets the glusterfs of this V1Volume. # noqa

property host_path

E501

Returns: The host_path of this V1Volume. # noqa: E501
Return type: V1HostPathVolumeSource
Type: Gets the host_path of this V1Volume. # noqa

property iscsi

E501

Returns: The iscsi of this V1Volume. # noqa: E501
Return type: V1ISCSIVolumeSource
Type: Gets the iscsi of this V1Volume. # noqa

property name

E501

Volume’s name. Must be a DNS_LABEL and unique within the pod. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names # noqa: E501

Returns: The name of this V1Volume. # noqa: E501
Return type: str
Type: Gets the name of this V1Volume. # noqa

property nfs

E501

Returns: The nfs of this V1Volume. # noqa: E501
Return type: V1NFSVolumeSource
Type: Gets the nfs of this V1Volume. # noqa

openapi_types = {'aws_elastic_block_store': 'V1AWSElasticBlockStoreVolumeSource', 'azure_disk': 'V1AzureDiskVolumeSource', 'azure_file': 'V1AzureFileVolumeSource', 'cephfs': 'V1CephFSVolumeSource', 'cinder': 'V1CinderVolumeSource', 'config_map': 'V1ConfigMapVolumeSource', 'csi': 'V1CSIVolumeSource', 'downward_api': 'V1DownwardAPIVolumeSource', 'empty_dir': 'V1EmptyDirVolumeSource', 'fc': 'V1FCVolumeSource', 'flex_volume': 'V1FlexVolumeSource', 'flocker': 'V1FlockerVolumeSource', 'gce_persistent_disk': 'V1GCEPersistentDiskVolumeSource', 'git_repo': 'V1GitRepoVolumeSource', 'glusterfs': 'V1GlusterfsVolumeSource', 'host_path': 'V1HostPathVolumeSource', 'iscsi': 'V1ISCSIVolumeSource', 'name': 'str', 'nfs': 'V1NFSVolumeSource', 'persistent_volume_claim': 'V1PersistentVolumeClaimVolumeSource', 'photon_persistent_disk': 'V1PhotonPersistentDiskVolumeSource', 'portworx_volume': 'V1PortworxVolumeSource', 'projected': 'V1ProjectedVolumeSource', 'quobyte': 'V1QuobyteVolumeSource', 'rbd': 'V1RBDVolumeSource', 'scale_io': 'V1ScaleIOVolumeSource', 'secret': 'V1SecretVolumeSource', 'storageos': 'V1StorageOSVolumeSource', 'vsphere_volume': 'V1VsphereVirtualDiskVolumeSource'}

property persistent_volume_claim

E501

Returns: The persistent_volume_claim of this V1Volume. # noqa: E501
Return type: V1PersistentVolumeClaimVolumeSource
Type: Gets the persistent_volume_claim of this V1Volume. # noqa

property photon_persistent_disk

E501

Returns: The photon_persistent_disk of this V1Volume. # noqa: E501
Return type: V1PhotonPersistentDiskVolumeSource
Type: Gets the photon_persistent_disk of this V1Volume. # noqa

property portworx_volume

E501

Returns: The portworx_volume of this V1Volume. # noqa: E501
Return type: V1PortworxVolumeSource
Type: Gets the portworx_volume of this V1Volume. # noqa

property projected

E501

Returns: The projected of this V1Volume. # noqa: E501
Return type: V1ProjectedVolumeSource
Type: Gets the projected of this V1Volume. # noqa

property quobyte

E501

Returns: The quobyte of this V1Volume. # noqa: E501
Return type: V1QuobyteVolumeSource
Type: Gets the quobyte of this V1Volume. # noqa

property rbd

E501

Returns: The rbd of this V1Volume. # noqa: E501
Return type: V1RBDVolumeSource
Type: Gets the rbd of this V1Volume. # noqa

property scale_io

E501

Returns: The scale_io of this V1Volume. # noqa: E501
Return type: V1ScaleIOVolumeSource
Type: Gets the scale_io of this V1Volume. # noqa

property secret

E501

Returns: The secret of this V1Volume. # noqa: E501
Return type: V1SecretVolumeSource
Type: Gets the secret of this V1Volume. # noqa

property storageos

E501

Returns: The storageos of this V1Volume. # noqa: E501
Return type: V1StorageOSVolumeSource
Type: Gets the storageos of this V1Volume. # noqa

to_dict(): Returns the model properties as a dict

to_str(): Returns the string representation of the model

property vsphere_volume

E501

Returns: The vsphere_volume of this V1Volume. # noqa: E501
Return type: V1VsphereVirtualDiskVolumeSource
Type: Gets the vsphere_volume of this V1Volume. # noqa

class kfp.dsl.ResourceOp(k8s_resource=None, action: str = 'create', merge_strategy: Optional[str] = None, success_condition: Optional[str] = None, failure_condition: Optional[str] = None, set_owner_reference: Optional[bool] = None, attribute_outputs: Optional[Dict[str, str]] = None, flags: Optional[List[str]] = None, **kwargs)[source]

Bases: kfp.dsl._container_op.BaseOp

Represents an op which will be translated into a resource template.

TODO(https://github.com/kubeflow/pipelines/issues/4822): Determine the: stability level of this feature.

Parameters

k8s_resource – A k8s resource which will be submitted to the cluster
action – One of “create”/”delete”/”apply”/”patch” (default is “create”)
merge_strategy – The merge strategy for the “apply” action
success_condition – The successCondition of the template
failure_condition – The failureCondition of the template For more info see: https://github.com/argoproj/argo-workflows/blob/master/examples/k8s-jobs.yaml
attribute_outputs – Maps output labels to resource’s json paths, similarly to file_outputs of ContainerOp
kwargs – name, sidecars. See BaseOp definition

Raises

ValueError – if not inside a pipeline if the name is an invalid string if no k8s_resource is provided if merge_strategy is set without “apply” action

add_affinity(affinity: kubernetes.client.models.v1_affinity.V1Affinity)

Add K8s Affinity.

Parameters

affinity – Kubernetes affinity For detailed spec, check affinity definition
https – //github.com/kubernetes-client/python/blob/master/kubernetes/client/models/v1_affinity.py

Example:

V1Affinity(
    node_affinity=V1NodeAffinity(
        required_during_scheduling_ignored_during_execution=V1NodeSelector(
            node_selector_terms=[V1NodeSelectorTerm(
                match_expressions=[V1NodeSelectorRequirement(
                    key='beta.kubernetes.io/instance-type',
                    operator='In',
                    values=['p2.xlarge'])])])))

add_init_container(init_container: kfp.dsl._container_op.UserContainer)

Add a init container to the Op.

Parameters: init_container – UserContainer object.

add_node_selector_constraint(label_name: Union[str, kfp.dsl._pipeline_param.PipelineParam], value: Union[str, kfp.dsl._pipeline_param.PipelineParam])

Add a constraint for nodeSelector.

Each constraint is a key-value pair label. For the container to be eligible to run on a node, the node must have each of the constraints appeared as labels.

Parameters

label_name (Union[str, PipelineParam]) – The name of the constraint label.
value (Union[str, PipelineParam]) – The value of the constraint label.

add_pod_annotation(name: str, value: str)

Adds a pod’s metadata annotation.

Parameters

name – The name of the annotation.
value – The value of the annotation.

add_pod_label(name: str, value: str)

Adds a pod’s metadata label.

Parameters

name – The name of the label.
value – The value of the label.

add_sidecar(sidecar: kfp.dsl._container_op.Sidecar)

Add a sidecar to the Op.

Parameters: sidecar – SideCar object.

add_toleration(tolerations: kubernetes.client.models.v1_toleration.V1Toleration)

Add K8s tolerations.

Parameters: tolerations – Kubernetes toleration For detailed spec, check toleration definition https://github.com/kubernetes-client/python/blob/master/kubernetes/client/models/v1_toleration.py

add_volume(volume)

Add K8s volume to the container.

Parameters

volume – Kubernetes volumes For detailed spec, check volume definition
https – //github.com/kubernetes-client/python/blob/master/kubernetes/client/models/v1_volume.py

after(*ops): Specify explicit dependency on other ops.

apply(mod_func)

Applies a modifier function to self.

The function should return the passed object. This is needed to chain “extention methods” to this class.

Example:

from kfp.gcp import use_gcp_secret
task = (
    train_op(...)
        .set_memory_request('1G')
        .apply(use_gcp_secret('user-gcp-sa'))
        .set_memory_limit('2G')
)

attrs_with_pipelineparams = ['node_selector', 'volumes', 'pod_annotations', 'pod_labels', 'num_retries', 'init_containers', 'sidecars', 'tolerations']

delete(flags: Optional[List[str]] = None)[source]: Returns a ResourceOp which deletes the resource.

property inputs: List of PipelineParams that will be converted into input parameters (io.argoproj.workflow.v1alpha1.Inputs) for the argo workflow.

property resource: Resource object that represents the resource property in io.argoproj.workflow.v1alpha1.Template.

set_caching_options(enable_caching: bool) → kfp.dsl._container_op.BaseOp

Sets caching options for the Op.

Parameters: enable_caching – Whether or not to enable caching for this task.
Returns: Self return to allow chained setting calls.

set_display_name(name: str)

set_retry(num_retries: int, policy: Optional[str] = None, backoff_duration: Optional[str] = None, backoff_factor: Optional[float] = None, backoff_max_duration: Optional[str] = None)

Sets the number of times the task is retried until it’s declared failed.

Parameters

num_retries – Number of times to retry on failures.
policy – Retry policy name.
backoff_duration – The time interval between retries. Defaults to an immediate retry. In case you specify a simple number, the unit defaults to seconds. You can also specify a different unit, for instance, 2m (2 minutes), 1h (1 hour).
backoff_factor – The exponential backoff factor applied to backoff_duration. For example, if backoff_duration=”60” (60 seconds) and backoff_factor=2, the first retry will happen after 60 seconds, then after 120, 240, and so on.
backoff_max_duration – The maximum interval that can be reached with the backoff strategy.

set_timeout(seconds: int)

Sets the timeout for the task in seconds.

Parameters: seconds – Number of seconds.

class kfp.dsl.Sidecar(name: str, image: str, command: Optional[Union[str, List[str]]] = None, args: Optional[Union[str, List[str]]] = None, mirror_volume_mounts: Optional[bool] = None, **kwargs)[source]

Bases: kfp.dsl._container_op.UserContainer

Creates a new instance of Sidecar.

Parameters

name – unique name for the sidecar container
image – image to use for the sidecar container, e.g. redis:alpine
command – entrypoint array. Not executed within a shell.
args – arguments to the entrypoint.
mirror_volume_mounts – MirrorVolumeMounts will mount the same volumes specified in the main container to the sidecar (including artifacts), at the same mountPaths. This enables dind daemon to partially see the same filesystem as the main container in order to use features such as docker volume binding
**kwargs – keyword arguments available for Container

add_env_from(env_from) → kfp.dsl._container_op.Container

Add a source to populate environment variables int the container.

Parameters

env_from – Kubernetes environment from source For detailed spec, check environment from source definition
https – //github.com/kubernetes-client/python/blob/master/kubernetes/client/models/v1_env_var_source.py

add_env_variable(env_variable) → kfp.dsl._container_op.Container

Add environment variable to the container.

Parameters

env_variable – Kubernetes environment variable For detailed spec, check environment variable definition
https – //github.com/kubernetes-client/python/blob/master/kubernetes/client/models/v1_env_var.py

add_port(container_port) → kfp.dsl._container_op.Container

Add a container port to the container.

Parameters

container_port – Kubernetes container port For detailed spec, check container port definition
https – //github.com/kubernetes-client/python/blob/master/kubernetes/client/models/v1_container_port.py

add_resource_limit(resource_name, value) → kfp.dsl._container_op.Container

Add the resource limit of the container.

Parameters

resource_name – The name of the resource. It can be cpu, memory, etc.
value – The string value of the limit.

add_resource_request(resource_name, value) → kfp.dsl._container_op.Container

Add the resource request of the container.

Parameters

resource_name – The name of the resource. It can be cpu, memory, etc.
value – The string value of the request.

add_volume_devices(volume_device) → kfp.dsl._container_op.Container

Add a block device to be used by the container.

Parameters

volume_device – Kubernetes volume device For detailed spec, volume device definition
https – //github.com/kubernetes-client/python/blob/master/kubernetes/client/models/v1_volume_device.py

add_volume_mount(volume_mount) → kfp.dsl._container_op.Container

Add volume to the container.

Parameters

volume_mount – Kubernetes volume mount For detailed spec, check volume mount definition
https – //github.com/kubernetes-client/python/blob/master/kubernetes/client/models/v1_volume_mount.py

property args

E501

Arguments to the entrypoint. The docker image’s CMD is used if this is not provided. Variable references $(VAR_NAME) are expanded using the container’s environment. If a variable cannot be resolved, the reference in the input string will be unchanged. The $(VAR_NAME) syntax can be escaped with a double $$, ie: $$(VAR_NAME). Escaped references will never be expanded, regardless of whether the variable exists or not. Cannot be updated. More info: https://kubernetes.io/docs/tasks/inject-data-application/define-command-argument-container/#running-a-command-in-a-shell # noqa: E501

Returns: The args of this V1Container. # noqa: E501
Return type: list[str]
Type: Gets the args of this V1Container. # noqa

attribute_map = {'args': 'args', 'command': 'command', 'env': 'env', 'env_from': 'envFrom', 'image': 'image', 'image_pull_policy': 'imagePullPolicy', 'lifecycle': 'lifecycle', 'liveness_probe': 'livenessProbe', 'mirror_volume_mounts': 'mirrorVolumeMounts', 'name': 'name', 'ports': 'ports', 'readiness_probe': 'readinessProbe', 'resources': 'resources', 'security_context': 'securityContext', 'startup_probe': 'startupProbe', 'stdin': 'stdin', 'stdin_once': 'stdinOnce', 'termination_message_path': 'terminationMessagePath', 'termination_message_policy': 'terminationMessagePolicy', 'tty': 'tty', 'volume_devices': 'volumeDevices', 'volume_mounts': 'volumeMounts', 'working_dir': 'workingDir'}

property command

E501

Entrypoint array. Not executed within a shell. The docker image’s ENTRYPOINT is used if this is not provided. Variable references $(VAR_NAME) are expanded using the container’s environment. If a variable cannot be resolved, the reference in the input string will be unchanged. The $(VAR_NAME) syntax can be escaped with a double $$, ie: $$(VAR_NAME). Escaped references will never be expanded, regardless of whether the variable exists or not. Cannot be updated. More info: https://kubernetes.io/docs/tasks/inject-data-application/define-command-argument-container/#running-a-command-in-a-shell # noqa: E501

Returns: The command of this V1Container. # noqa: E501
Return type: list[str]
Type: Gets the command of this V1Container. # noqa

property env

E501

List of environment variables to set in the container. Cannot be updated. # noqa: E501

Returns: The env of this V1Container. # noqa: E501
Return type: list[V1EnvVar]
Type: Gets the env of this V1Container. # noqa

property env_from

E501

List of sources to populate environment variables in the container. The keys defined within a source must be a C_IDENTIFIER. All invalid keys will be reported as an event when the container is starting. When a key exists in multiple sources, the value associated with the last source will take precedence. Values defined by an Env with a duplicate key will take precedence. Cannot be updated. # noqa: E501

Returns: The env_from of this V1Container. # noqa: E501
Return type: list[V1EnvFromSource]
Type: Gets the env_from of this V1Container. # noqa

get_resource_limit(resource_name: str) → Optional[str]

Get the resource limit of the container.

Parameters: resource_name – The name of the resource. It can be cpu, memory, etc.

get_resource_request(resource_name: str) → Optional[str]

Get the resource request of the container.

Parameters: resource_name – The name of the resource. It can be cpu, memory, etc.

property image

E501

Docker image name. More info: https://kubernetes.io/docs/concepts/containers/images This field is optional to allow higher level config management to default or override container images in workload controllers like Deployments and StatefulSets. # noqa: E501

Returns: The image of this V1Container. # noqa: E501
Return type: str
Type: Gets the image of this V1Container. # noqa

property image_pull_policy

E501

Image pull policy. One of Always, Never, IfNotPresent. Defaults to Always if :latest tag is specified, or IfNotPresent otherwise. Cannot be updated. More info: https://kubernetes.io/docs/concepts/containers/images#updating-images # noqa: E501

Returns: The image_pull_policy of this V1Container. # noqa: E501
Return type: str
Type: Gets the image_pull_policy of this V1Container. # noqa

property inputs: A list of PipelineParam found in the UserContainer object.

property lifecycle

E501

Returns: The lifecycle of this V1Container. # noqa: E501
Return type: V1Lifecycle
Type: Gets the lifecycle of this V1Container. # noqa

property liveness_probe

E501

Returns: The liveness_probe of this V1Container. # noqa: E501
Return type: V1Probe
Type: Gets the liveness_probe of this V1Container. # noqa

property name

E501

Name of the container specified as a DNS_LABEL. Each container in a pod must have a unique name (DNS_LABEL). Cannot be updated. # noqa: E501

Returns: The name of this V1Container. # noqa: E501
Return type: str
Type: Gets the name of this V1Container. # noqa

openapi_types = {'args': 'list[str]', 'command': 'list[str]', 'env': 'list[V1EnvVar]', 'env_from': 'list[V1EnvFromSource]', 'image': 'str', 'image_pull_policy': 'str', 'lifecycle': 'V1Lifecycle', 'liveness_probe': 'V1Probe', 'mirror_volume_mounts': 'bool', 'name': 'str', 'ports': 'list[V1ContainerPort]', 'readiness_probe': 'V1Probe', 'resources': 'V1ResourceRequirements', 'security_context': 'V1SecurityContext', 'startup_probe': 'V1Probe', 'stdin': 'bool', 'stdin_once': 'bool', 'termination_message_path': 'str', 'termination_message_policy': 'str', 'tty': 'bool', 'volume_devices': 'list[V1VolumeDevice]', 'volume_mounts': 'list[V1VolumeMount]', 'working_dir': 'str'}

property ports

E501

List of ports to expose from the container. Exposing a port here gives the system additional information about the network connections a container uses, but is primarily informational. Not specifying a port here DOES NOT prevent that port from being exposed. Any port which is listening on the default “0.0.0.0” address inside a container will be accessible from the network. Cannot be updated. # noqa: E501

Returns: The ports of this V1Container. # noqa: E501
Return type: list[V1ContainerPort]
Type: Gets the ports of this V1Container. # noqa

property readiness_probe

E501

Returns: The readiness_probe of this V1Container. # noqa: E501
Return type: V1Probe
Type: Gets the readiness_probe of this V1Container. # noqa

property resources

E501

Returns: The resources of this V1Container. # noqa: E501
Return type: V1ResourceRequirements
Type: Gets the resources of this V1Container. # noqa

property security_context

E501

Returns: The security_context of this V1Container. # noqa: E501
Return type: V1SecurityContext
Type: Gets the security_context of this V1Container. # noqa

set_cpu_limit(cpu: Union[str, kfp.dsl._pipeline_param.PipelineParam]) → kfp.dsl._container_op.Container

Set cpu limit (maximum) for this operator.

Parameters: cpu (Union[str, PipelineParam]) – A string which can be a number or a number followed by “m”, which means 1/1000.

set_cpu_request(cpu: Union[str, kfp.dsl._pipeline_param.PipelineParam]) → kfp.dsl._container_op.Container

Set cpu request (minimum) for this operator.

Parameters: cpu (Union[str, PipelineParam]) – A string which can be a number or a number followed by “m”, which means 1/1000.

set_env_variable(name: str, value: str) → kfp.dsl._container_op.Container

Sets environment variable to the container (v2 only).

Parameters

name – The name of the environment variable.
value – The value of the environment variable.

set_ephemeral_storage_limit(size) → kfp.dsl._container_op.Container

Set ephemeral-storage request (maximum) for this operator.

Parameters: size – a string which can be a number or a number followed by one of “E”, “P”, “T”, “G”, “M”, “K”.

set_ephemeral_storage_request(size) → kfp.dsl._container_op.Container

Set ephemeral-storage request (minimum) for this operator.

Parameters: size – a string which can be a number or a number followed by one of “E”, “P”, “T”, “G”, “M”, “K”.

set_gpu_limit(gpu: Union[str, kfp.dsl._pipeline_param.PipelineParam], vendor: Union[str, kfp.dsl._pipeline_param.PipelineParam] = 'nvidia') → kfp.dsl._container_op.Container

Set gpu limit for the operator.

This function add ‘<vendor>.com/gpu’ into resource limit. Note that there is no need to add GPU request. GPUs are only supposed to be specified in the limits section. See https://kubernetes.io/docs/tasks/manage-gpus/scheduling-gpus/.

Parameters

gpu (Union[str, PipelineParam]) – A string which must be a positive number.
vendor (Union[str, PipelineParam]) – Optional. A string which is the vendor of the requested gpu. The supported values are: ‘nvidia’ (default), and ‘amd’. The value is ignored in v2.

set_image_pull_policy(image_pull_policy) → kfp.dsl._container_op.Container

Set image pull policy for the container.

Parameters: image_pull_policy – One of Always, Never, IfNotPresent.

set_lifecycle(lifecycle) → kfp.dsl._container_op.Container

Setup a lifecycle config for the container.

Parameters

lifecycle – Kubernetes lifecycle For detailed spec, lifecycle definition
https – //github.com/kubernetes-client/python/blob/master/kubernetes/client/models/v1_lifecycle.py

set_liveness_probe(liveness_probe) → kfp.dsl._container_op.Container

Set a liveness probe for the container.

Parameters

liveness_probe – Kubernetes liveness probe For detailed spec, check probe definition
https – //github.com/kubernetes-client/python/blob/master/kubernetes/client/models/v1_probe.py

set_memory_limit(memory: Union[str, kfp.dsl._pipeline_param.PipelineParam]) → kfp.dsl._container_op.Container

Set memory limit (maximum) for this operator.

Parameters: memory (Union[str, PipelineParam]) – a string which can be a number or a number followed by one of “E”, “P”, “T”, “G”, “M”, “K”.

set_memory_request(memory: Union[str, kfp.dsl._pipeline_param.PipelineParam]) → kfp.dsl._container_op.Container

Set memory request (minimum) for this operator.

Parameters: memory (Union[str, PipelineParam]) – a string which can be a number or a number followed by one of “E”, “P”, “T”, “G”, “M”, “K”.

set_mirror_volume_mounts(mirror_volume_mounts=True)

Setting mirrorVolumeMounts to true will mount the same volumes specified in the main container to the container (including artifacts), at the same mountPaths. This enables dind daemon to partially see the same filesystem as the main container in order to use features such as docker volume binding.

Parameters: mirror_volume_mounts – boolean flag

set_readiness_probe(readiness_probe) → kfp.dsl._container_op.Container

Set a readiness probe for the container.

Parameters

readiness_probe – Kubernetes readiness probe For detailed spec, check probe definition
https – //github.com/kubernetes-client/python/blob/master/kubernetes/client/models/v1_probe.py

set_security_context(security_context) → kfp.dsl._container_op.Container

Set security configuration to be applied on the container.

Parameters

security_context – Kubernetes security context For detailed spec, check security context definition
https – //github.com/kubernetes-client/python/blob/master/kubernetes/client/models/v1_security_context.py

set_stdin(stdin=True) → kfp.dsl._container_op.Container

Whether this container should allocate a buffer for stdin in the container runtime. If this is not set, reads from stdin in the container will always result in EOF.

Parameters: stdin – boolean flag

set_stdin_once(stdin_once=True) → kfp.dsl._container_op.Container

Whether the container runtime should close the stdin channel after it has been opened by a single attach. When stdin is true the stdin stream will remain open across multiple attach sessions. If stdinOnce is set to true, stdin is opened on container start, is empty until the first client attaches to stdin, and then remains open and accepts data until the client disconnects, at which time stdin is closed and remains closed until the container is restarted. If this flag is false, a container processes that reads from stdin will never receive an EOF.

Parameters: stdin_once – boolean flag

set_termination_message_path(termination_message_path) → kfp.dsl._container_op.Container

Path at which the file to which the container’s termination message will be written is mounted into the container’s filesystem. Message written is intended to be brief final status, such as an assertion failure message. Will be truncated by the node if greater than 4096 bytes. The total message length across all containers will be limited to 12kb.

Parameters: termination_message_path – path for the termination message

set_termination_message_policy(termination_message_policy) → kfp.dsl._container_op.Container

Indicate how the termination message should be populated. File will use the contents of terminationMessagePath to populate the container status message on both success and failure. FallbackToLogsOnError will use the last chunk of container log output if the termination message file is empty and the container exited with an error. The log output is limited to 2048 bytes or 80 lines, whichever is smaller.

Parameters: termination_message_policy – File or FallbackToLogsOnError

set_tty(tty: bool = True) → kfp.dsl._container_op.Container

Whether this container should allocate a TTY for itself, also requires ‘stdin’ to be true.

Parameters: tty – boolean flag

property startup_probe

E501

Returns: The startup_probe of this V1Container. # noqa: E501
Return type: V1Probe
Type: Gets the startup_probe of this V1Container. # noqa

property stdin

E501

Whether this container should allocate a buffer for stdin in the container runtime. If this is not set, reads from stdin in the container will always result in EOF. Default is false. # noqa: E501

Returns: The stdin of this V1Container. # noqa: E501
Return type: bool
Type: Gets the stdin of this V1Container. # noqa

property stdin_once

E501

Whether the container runtime should close the stdin channel after it has been opened by a single attach. When stdin is true the stdin stream will remain open across multiple attach sessions. If stdinOnce is set to true, stdin is opened on container start, is empty until the first client attaches to stdin, and then remains open and accepts data until the client disconnects, at which time stdin is closed and remains closed until the container is restarted. If this flag is false, a container processes that reads from stdin will never receive an EOF. Default is false # noqa: E501

Returns: The stdin_once of this V1Container. # noqa: E501
Return type: bool
Type: Gets the stdin_once of this V1Container. # noqa

property termination_message_path

E501

Optional: Path at which the file to which the container’s termination message will be written is mounted into the container’s filesystem. Message written is intended to be brief final status, such as an assertion failure message. Will be truncated by the node if greater than 4096 bytes. The total message length across all containers will be limited to 12kb. Defaults to /dev/termination-log. Cannot be updated. # noqa: E501

Returns: The termination_message_path of this V1Container. # noqa: E501
Return type: str
Type: Gets the termination_message_path of this V1Container. # noqa

property termination_message_policy

E501

Indicate how the termination message should be populated. File will use the contents of terminationMessagePath to populate the container status message on both success and failure. FallbackToLogsOnError will use the last chunk of container log output if the termination message file is empty and the container exited with an error. The log output is limited to 2048 bytes or 80 lines, whichever is smaller. Defaults to File. Cannot be updated. # noqa: E501

Returns: The termination_message_policy of this V1Container. # noqa: E501
Return type: str
Type: Gets the termination_message_policy of this V1Container. # noqa

to_dict(): Returns the model properties as a dict

to_str(): Returns the string representation of the model

property tty

E501

Whether this container should allocate a TTY for itself, also requires ‘stdin’ to be true. Default is false. # noqa: E501

Returns: The tty of this V1Container. # noqa: E501
Return type: bool
Type: Gets the tty of this V1Container. # noqa

property volume_devices

E501

volumeDevices is the list of block devices to be used by the container. # noqa: E501

Returns: The volume_devices of this V1Container. # noqa: E501
Return type: list[V1VolumeDevice]
Type: Gets the volume_devices of this V1Container. # noqa

property volume_mounts

E501

Pod volumes to mount into the container’s filesystem. Cannot be updated. # noqa: E501

Returns: The volume_mounts of this V1Container. # noqa: E501
Return type: list[V1VolumeMount]
Type: Gets the volume_mounts of this V1Container. # noqa

property working_dir

E501

Container’s working directory. If not specified, the container runtime’s default will be used, which might be configured in the container image. Cannot be updated. # noqa: E501

Returns: The working_dir of this V1Container. # noqa: E501
Return type: str
Type: Gets the working_dir of this V1Container. # noqa

class kfp.dsl.SubGraph(parallelism: int)[source]

Bases: kfp.dsl._ops_group.OpsGroup

TYPE_NAME = 'subgraph'

after(*ops): Specify explicit dependency on other ops.

remove_op_recursive(op)

class kfp.dsl.UserContainer(name: str, image: str, command: Optional[Union[str, List[str]]] = None, args: Optional[Union[str, List[str]]] = None, mirror_volume_mounts: Optional[bool] = None, **kwargs)[source]

Bases: kfp.dsl._container_op.Container

Represents an argo workflow UserContainer (io.argoproj.workflow.v1alpha1.UserContainer) to be used in UserContainer property in argo’s workflow template (io.argoproj.workflow.v1alpha1.Template).

UserContainer inherits from Container class with an addition of mirror_volume_mounts attribute (mirrorVolumeMounts property).

See https://github.com/argoproj/argo-workflows/blob/master/api/openapi-spec/swagger.json

Parameters

name – unique name for the user container
image – image to use for the user container, e.g. redis:alpine
command – entrypoint array. Not executed within a shell.
args – arguments to the entrypoint.
mirror_volume_mounts – MirrorVolumeMounts will mount the same volumes specified in the main container to the container (including artifacts), at the same mountPaths. This enables dind daemon to partially see the same filesystem as the main container in order to use features such as docker volume binding
**kwargs – keyword arguments available for Container

swagger_types

The key is attribute name and the value is attribute type.

Type: dict

Example

from kfp.dsl import ContainerOp, UserContainer
# creates a `ContainerOp` and adds a redis init container
op = (ContainerOp(name='foo-op', image='busybox:latest')
   .add_initContainer(UserContainer(name='redis', image='redis:alpine')))

add_env_from(env_from) → kfp.dsl._container_op.Container

Add a source to populate environment variables int the container.

Parameters

env_from – Kubernetes environment from source For detailed spec, check environment from source definition
https – //github.com/kubernetes-client/python/blob/master/kubernetes/client/models/v1_env_var_source.py

add_env_variable(env_variable) → kfp.dsl._container_op.Container

Add environment variable to the container.

Parameters

env_variable – Kubernetes environment variable For detailed spec, check environment variable definition
https – //github.com/kubernetes-client/python/blob/master/kubernetes/client/models/v1_env_var.py

add_port(container_port) → kfp.dsl._container_op.Container

Add a container port to the container.

Parameters

container_port – Kubernetes container port For detailed spec, check container port definition
https – //github.com/kubernetes-client/python/blob/master/kubernetes/client/models/v1_container_port.py

add_resource_limit(resource_name, value) → kfp.dsl._container_op.Container

Add the resource limit of the container.

Parameters

resource_name – The name of the resource. It can be cpu, memory, etc.
value – The string value of the limit.

add_resource_request(resource_name, value) → kfp.dsl._container_op.Container

Add the resource request of the container.

Parameters

resource_name – The name of the resource. It can be cpu, memory, etc.
value – The string value of the request.

add_volume_devices(volume_device) → kfp.dsl._container_op.Container

Add a block device to be used by the container.

Parameters

volume_device – Kubernetes volume device For detailed spec, volume device definition
https – //github.com/kubernetes-client/python/blob/master/kubernetes/client/models/v1_volume_device.py

add_volume_mount(volume_mount) → kfp.dsl._container_op.Container

Add volume to the container.

Parameters

volume_mount – Kubernetes volume mount For detailed spec, check volume mount definition
https – //github.com/kubernetes-client/python/blob/master/kubernetes/client/models/v1_volume_mount.py

property args

E501

Arguments to the entrypoint. The docker image’s CMD is used if this is not provided. Variable references $(VAR_NAME) are expanded using the container’s environment. If a variable cannot be resolved, the reference in the input string will be unchanged. The $(VAR_NAME) syntax can be escaped with a double $$, ie: $$(VAR_NAME). Escaped references will never be expanded, regardless of whether the variable exists or not. Cannot be updated. More info: https://kubernetes.io/docs/tasks/inject-data-application/define-command-argument-container/#running-a-command-in-a-shell # noqa: E501

Returns: The args of this V1Container. # noqa: E501
Return type: list[str]
Type: Gets the args of this V1Container. # noqa

attribute_map = {'args': 'args', 'command': 'command', 'env': 'env', 'env_from': 'envFrom', 'image': 'image', 'image_pull_policy': 'imagePullPolicy', 'lifecycle': 'lifecycle', 'liveness_probe': 'livenessProbe', 'mirror_volume_mounts': 'mirrorVolumeMounts', 'name': 'name', 'ports': 'ports', 'readiness_probe': 'readinessProbe', 'resources': 'resources', 'security_context': 'securityContext', 'startup_probe': 'startupProbe', 'stdin': 'stdin', 'stdin_once': 'stdinOnce', 'termination_message_path': 'terminationMessagePath', 'termination_message_policy': 'terminationMessagePolicy', 'tty': 'tty', 'volume_devices': 'volumeDevices', 'volume_mounts': 'volumeMounts', 'working_dir': 'workingDir'}

property command

E501

Entrypoint array. Not executed within a shell. The docker image’s ENTRYPOINT is used if this is not provided. Variable references $(VAR_NAME) are expanded using the container’s environment. If a variable cannot be resolved, the reference in the input string will be unchanged. The $(VAR_NAME) syntax can be escaped with a double $$, ie: $$(VAR_NAME). Escaped references will never be expanded, regardless of whether the variable exists or not. Cannot be updated. More info: https://kubernetes.io/docs/tasks/inject-data-application/define-command-argument-container/#running-a-command-in-a-shell # noqa: E501

Returns: The command of this V1Container. # noqa: E501
Return type: list[str]
Type: Gets the command of this V1Container. # noqa

property env

E501

List of environment variables to set in the container. Cannot be updated. # noqa: E501

Returns: The env of this V1Container. # noqa: E501
Return type: list[V1EnvVar]
Type: Gets the env of this V1Container. # noqa

property env_from

E501

List of sources to populate environment variables in the container. The keys defined within a source must be a C_IDENTIFIER. All invalid keys will be reported as an event when the container is starting. When a key exists in multiple sources, the value associated with the last source will take precedence. Values defined by an Env with a duplicate key will take precedence. Cannot be updated. # noqa: E501

Returns: The env_from of this V1Container. # noqa: E501
Return type: list[V1EnvFromSource]
Type: Gets the env_from of this V1Container. # noqa

get_resource_limit(resource_name: str) → Optional[str]

Get the resource limit of the container.

Parameters: resource_name – The name of the resource. It can be cpu, memory, etc.

get_resource_request(resource_name: str) → Optional[str]

Get the resource request of the container.

Parameters: resource_name – The name of the resource. It can be cpu, memory, etc.

property image

E501

Docker image name. More info: https://kubernetes.io/docs/concepts/containers/images This field is optional to allow higher level config management to default or override container images in workload controllers like Deployments and StatefulSets. # noqa: E501

Returns: The image of this V1Container. # noqa: E501
Return type: str
Type: Gets the image of this V1Container. # noqa

property image_pull_policy

E501

Image pull policy. One of Always, Never, IfNotPresent. Defaults to Always if :latest tag is specified, or IfNotPresent otherwise. Cannot be updated. More info: https://kubernetes.io/docs/concepts/containers/images#updating-images # noqa: E501

Returns: The image_pull_policy of this V1Container. # noqa: E501
Return type: str
Type: Gets the image_pull_policy of this V1Container. # noqa

property inputs: A list of PipelineParam found in the UserContainer object.

property lifecycle

E501

Returns: The lifecycle of this V1Container. # noqa: E501
Return type: V1Lifecycle
Type: Gets the lifecycle of this V1Container. # noqa

property liveness_probe

E501

Returns: The liveness_probe of this V1Container. # noqa: E501
Return type: V1Probe
Type: Gets the liveness_probe of this V1Container. # noqa

property name

E501

Name of the container specified as a DNS_LABEL. Each container in a pod must have a unique name (DNS_LABEL). Cannot be updated. # noqa: E501

Returns: The name of this V1Container. # noqa: E501
Return type: str
Type: Gets the name of this V1Container. # noqa

openapi_types = {'args': 'list[str]', 'command': 'list[str]', 'env': 'list[V1EnvVar]', 'env_from': 'list[V1EnvFromSource]', 'image': 'str', 'image_pull_policy': 'str', 'lifecycle': 'V1Lifecycle', 'liveness_probe': 'V1Probe', 'mirror_volume_mounts': 'bool', 'name': 'str', 'ports': 'list[V1ContainerPort]', 'readiness_probe': 'V1Probe', 'resources': 'V1ResourceRequirements', 'security_context': 'V1SecurityContext', 'startup_probe': 'V1Probe', 'stdin': 'bool', 'stdin_once': 'bool', 'termination_message_path': 'str', 'termination_message_policy': 'str', 'tty': 'bool', 'volume_devices': 'list[V1VolumeDevice]', 'volume_mounts': 'list[V1VolumeMount]', 'working_dir': 'str'}

property ports

E501

List of ports to expose from the container. Exposing a port here gives the system additional information about the network connections a container uses, but is primarily informational. Not specifying a port here DOES NOT prevent that port from being exposed. Any port which is listening on the default “0.0.0.0” address inside a container will be accessible from the network. Cannot be updated. # noqa: E501

Returns: The ports of this V1Container. # noqa: E501
Return type: list[V1ContainerPort]
Type: Gets the ports of this V1Container. # noqa

property readiness_probe

E501

Returns: The readiness_probe of this V1Container. # noqa: E501
Return type: V1Probe
Type: Gets the readiness_probe of this V1Container. # noqa

property resources

E501

Returns: The resources of this V1Container. # noqa: E501
Return type: V1ResourceRequirements
Type: Gets the resources of this V1Container. # noqa

property security_context

E501

Returns: The security_context of this V1Container. # noqa: E501
Return type: V1SecurityContext
Type: Gets the security_context of this V1Container. # noqa

set_cpu_limit(cpu: Union[str, kfp.dsl._pipeline_param.PipelineParam]) → kfp.dsl._container_op.Container

Set cpu limit (maximum) for this operator.

Parameters: cpu (Union[str, PipelineParam]) – A string which can be a number or a number followed by “m”, which means 1/1000.

set_cpu_request(cpu: Union[str, kfp.dsl._pipeline_param.PipelineParam]) → kfp.dsl._container_op.Container

Set cpu request (minimum) for this operator.

Parameters: cpu (Union[str, PipelineParam]) – A string which can be a number or a number followed by “m”, which means 1/1000.

set_env_variable(name: str, value: str) → kfp.dsl._container_op.Container

Sets environment variable to the container (v2 only).

Parameters

name – The name of the environment variable.
value – The value of the environment variable.

set_ephemeral_storage_limit(size) → kfp.dsl._container_op.Container

Set ephemeral-storage request (maximum) for this operator.

Parameters: size – a string which can be a number or a number followed by one of “E”, “P”, “T”, “G”, “M”, “K”.

set_ephemeral_storage_request(size) → kfp.dsl._container_op.Container

Set ephemeral-storage request (minimum) for this operator.

Parameters: size – a string which can be a number or a number followed by one of “E”, “P”, “T”, “G”, “M”, “K”.

set_gpu_limit(gpu: Union[str, kfp.dsl._pipeline_param.PipelineParam], vendor: Union[str, kfp.dsl._pipeline_param.PipelineParam] = 'nvidia') → kfp.dsl._container_op.Container

Set gpu limit for the operator.

This function add ‘<vendor>.com/gpu’ into resource limit. Note that there is no need to add GPU request. GPUs are only supposed to be specified in the limits section. See https://kubernetes.io/docs/tasks/manage-gpus/scheduling-gpus/.

Parameters

gpu (Union[str, PipelineParam]) – A string which must be a positive number.
vendor (Union[str, PipelineParam]) – Optional. A string which is the vendor of the requested gpu. The supported values are: ‘nvidia’ (default), and ‘amd’. The value is ignored in v2.

set_image_pull_policy(image_pull_policy) → kfp.dsl._container_op.Container

Set image pull policy for the container.

Parameters: image_pull_policy – One of Always, Never, IfNotPresent.

set_lifecycle(lifecycle) → kfp.dsl._container_op.Container

Setup a lifecycle config for the container.

Parameters

lifecycle – Kubernetes lifecycle For detailed spec, lifecycle definition
https – //github.com/kubernetes-client/python/blob/master/kubernetes/client/models/v1_lifecycle.py

set_liveness_probe(liveness_probe) → kfp.dsl._container_op.Container

Set a liveness probe for the container.

Parameters

liveness_probe – Kubernetes liveness probe For detailed spec, check probe definition
https – //github.com/kubernetes-client/python/blob/master/kubernetes/client/models/v1_probe.py

set_memory_limit(memory: Union[str, kfp.dsl._pipeline_param.PipelineParam]) → kfp.dsl._container_op.Container

Set memory limit (maximum) for this operator.

Parameters: memory (Union[str, PipelineParam]) – a string which can be a number or a number followed by one of “E”, “P”, “T”, “G”, “M”, “K”.

set_memory_request(memory: Union[str, kfp.dsl._pipeline_param.PipelineParam]) → kfp.dsl._container_op.Container

Set memory request (minimum) for this operator.

Parameters: memory (Union[str, PipelineParam]) – a string which can be a number or a number followed by one of “E”, “P”, “T”, “G”, “M”, “K”.

set_mirror_volume_mounts(mirror_volume_mounts=True)[source]

Setting mirrorVolumeMounts to true will mount the same volumes specified in the main container to the container (including artifacts), at the same mountPaths. This enables dind daemon to partially see the same filesystem as the main container in order to use features such as docker volume binding.

Parameters: mirror_volume_mounts – boolean flag

set_readiness_probe(readiness_probe) → kfp.dsl._container_op.Container

Set a readiness probe for the container.

Parameters

readiness_probe – Kubernetes readiness probe For detailed spec, check probe definition
https – //github.com/kubernetes-client/python/blob/master/kubernetes/client/models/v1_probe.py

set_security_context(security_context) → kfp.dsl._container_op.Container

Set security configuration to be applied on the container.

Parameters

security_context – Kubernetes security context For detailed spec, check security context definition
https – //github.com/kubernetes-client/python/blob/master/kubernetes/client/models/v1_security_context.py

set_stdin(stdin=True) → kfp.dsl._container_op.Container

Whether this container should allocate a buffer for stdin in the container runtime. If this is not set, reads from stdin in the container will always result in EOF.

Parameters: stdin – boolean flag

set_stdin_once(stdin_once=True) → kfp.dsl._container_op.Container

Whether the container runtime should close the stdin channel after it has been opened by a single attach. When stdin is true the stdin stream will remain open across multiple attach sessions. If stdinOnce is set to true, stdin is opened on container start, is empty until the first client attaches to stdin, and then remains open and accepts data until the client disconnects, at which time stdin is closed and remains closed until the container is restarted. If this flag is false, a container processes that reads from stdin will never receive an EOF.

Parameters: stdin_once – boolean flag

set_termination_message_path(termination_message_path) → kfp.dsl._container_op.Container

Path at which the file to which the container’s termination message will be written is mounted into the container’s filesystem. Message written is intended to be brief final status, such as an assertion failure message. Will be truncated by the node if greater than 4096 bytes. The total message length across all containers will be limited to 12kb.

Parameters: termination_message_path – path for the termination message

set_termination_message_policy(termination_message_policy) → kfp.dsl._container_op.Container

Indicate how the termination message should be populated. File will use the contents of terminationMessagePath to populate the container status message on both success and failure. FallbackToLogsOnError will use the last chunk of container log output if the termination message file is empty and the container exited with an error. The log output is limited to 2048 bytes or 80 lines, whichever is smaller.

Parameters: termination_message_policy – File or FallbackToLogsOnError

set_tty(tty: bool = True) → kfp.dsl._container_op.Container

Whether this container should allocate a TTY for itself, also requires ‘stdin’ to be true.

Parameters: tty – boolean flag

property startup_probe

E501

Returns: The startup_probe of this V1Container. # noqa: E501
Return type: V1Probe
Type: Gets the startup_probe of this V1Container. # noqa

property stdin

E501

Whether this container should allocate a buffer for stdin in the container runtime. If this is not set, reads from stdin in the container will always result in EOF. Default is false. # noqa: E501

Returns: The stdin of this V1Container. # noqa: E501
Return type: bool
Type: Gets the stdin of this V1Container. # noqa

property stdin_once

E501

Whether the container runtime should close the stdin channel after it has been opened by a single attach. When stdin is true the stdin stream will remain open across multiple attach sessions. If stdinOnce is set to true, stdin is opened on container start, is empty until the first client attaches to stdin, and then remains open and accepts data until the client disconnects, at which time stdin is closed and remains closed until the container is restarted. If this flag is false, a container processes that reads from stdin will never receive an EOF. Default is false # noqa: E501

Returns: The stdin_once of this V1Container. # noqa: E501
Return type: bool
Type: Gets the stdin_once of this V1Container. # noqa

property termination_message_path

E501

Optional: Path at which the file to which the container’s termination message will be written is mounted into the container’s filesystem. Message written is intended to be brief final status, such as an assertion failure message. Will be truncated by the node if greater than 4096 bytes. The total message length across all containers will be limited to 12kb. Defaults to /dev/termination-log. Cannot be updated. # noqa: E501

Returns: The termination_message_path of this V1Container. # noqa: E501
Return type: str
Type: Gets the termination_message_path of this V1Container. # noqa

property termination_message_policy

E501

Indicate how the termination message should be populated. File will use the contents of terminationMessagePath to populate the container status message on both success and failure. FallbackToLogsOnError will use the last chunk of container log output if the termination message file is empty and the container exited with an error. The log output is limited to 2048 bytes or 80 lines, whichever is smaller. Defaults to File. Cannot be updated. # noqa: E501

Returns: The termination_message_policy of this V1Container. # noqa: E501
Return type: str
Type: Gets the termination_message_policy of this V1Container. # noqa

to_dict(): Returns the model properties as a dict

to_str(): Returns the string representation of the model

property tty

E501

Whether this container should allocate a TTY for itself, also requires ‘stdin’ to be true. Default is false. # noqa: E501

Returns: The tty of this V1Container. # noqa: E501
Return type: bool
Type: Gets the tty of this V1Container. # noqa

property volume_devices

E501

volumeDevices is the list of block devices to be used by the container. # noqa: E501

Returns: The volume_devices of this V1Container. # noqa: E501
Return type: list[V1VolumeDevice]
Type: Gets the volume_devices of this V1Container. # noqa

property volume_mounts

E501

Pod volumes to mount into the container’s filesystem. Cannot be updated. # noqa: E501

Returns: The volume_mounts of this V1Container. # noqa: E501
Return type: list[V1VolumeMount]
Type: Gets the volume_mounts of this V1Container. # noqa

property working_dir

E501

Container’s working directory. If not specified, the container runtime’s default will be used, which might be configured in the container image. Cannot be updated. # noqa: E501

Returns: The working_dir of this V1Container. # noqa: E501
Return type: str
Type: Gets the working_dir of this V1Container. # noqa

class kfp.dsl.VolumeOp(resource_name: Optional[str] = None, size: Optional[str] = None, storage_class: Optional[str] = None, modes: Optional[List[str]] = None, annotations: Optional[Dict[str, str]] = None, data_source=None, volume_name=None, generate_unique_name: bool = True, **kwargs)[source]

Bases: kfp.dsl._resource_op.ResourceOp

Represents an op which will be translated into a resource template which will be creating a PVC.

TODO(https://github.com/kubeflow/pipelines/issues/4822): Determine the: stability level of this feature.

Parameters

resource_name – A desired name for the PVC which will be created
size – The size of the PVC which will be created
storage_class – The storage class to use for the dynamically created PVC
modes – The access modes for the PVC
annotations – Annotations to be patched in the PVC
data_source – May be a V1TypedLocalObjectReference, and then it is used in the data_source field of the PVC as is. Can also be a string/PipelineParam, and in that case it will be used as a VolumeSnapshot name (Alpha feature)
volume_name – VolumeName is the binding reference to the PersistentVolume backing this claim.
generate_unique_name – Generate unique name for the PVC
kwargs – See kfp.dsl.ResourceOp

Raises

ValueError – if k8s_resource is provided along with other arguments if k8s_resource is not a V1PersistentVolumeClaim if size is None if size is an invalid memory string (when not a PipelineParam) if data_source is not one of (str, PipelineParam, V1TypedLocalObjectReference)

add_affinity(affinity: kubernetes.client.models.v1_affinity.V1Affinity)

Add K8s Affinity.

Parameters

affinity – Kubernetes affinity For detailed spec, check affinity definition
https – //github.com/kubernetes-client/python/blob/master/kubernetes/client/models/v1_affinity.py

Example:

V1Affinity(
    node_affinity=V1NodeAffinity(
        required_during_scheduling_ignored_during_execution=V1NodeSelector(
            node_selector_terms=[V1NodeSelectorTerm(
                match_expressions=[V1NodeSelectorRequirement(
                    key='beta.kubernetes.io/instance-type',
                    operator='In',
                    values=['p2.xlarge'])])])))

add_init_container(init_container: kfp.dsl._container_op.UserContainer)

Add a init container to the Op.

Parameters: init_container – UserContainer object.

add_node_selector_constraint(label_name: Union[str, kfp.dsl._pipeline_param.PipelineParam], value: Union[str, kfp.dsl._pipeline_param.PipelineParam])

Add a constraint for nodeSelector.

Each constraint is a key-value pair label. For the container to be eligible to run on a node, the node must have each of the constraints appeared as labels.

Parameters

label_name (Union[str, PipelineParam]) – The name of the constraint label.
value (Union[str, PipelineParam]) – The value of the constraint label.

add_pod_annotation(name: str, value: str)

Adds a pod’s metadata annotation.

Parameters

name – The name of the annotation.
value – The value of the annotation.

add_pod_label(name: str, value: str)

Adds a pod’s metadata label.

Parameters

name – The name of the label.
value – The value of the label.

add_sidecar(sidecar: kfp.dsl._container_op.Sidecar)

Add a sidecar to the Op.

Parameters: sidecar – SideCar object.

add_toleration(tolerations: kubernetes.client.models.v1_toleration.V1Toleration)

Add K8s tolerations.

Parameters: tolerations – Kubernetes toleration For detailed spec, check toleration definition https://github.com/kubernetes-client/python/blob/master/kubernetes/client/models/v1_toleration.py

add_volume(volume)

Add K8s volume to the container.

Parameters

volume – Kubernetes volumes For detailed spec, check volume definition
https – //github.com/kubernetes-client/python/blob/master/kubernetes/client/models/v1_volume.py

after(*ops): Specify explicit dependency on other ops.

apply(mod_func)

Applies a modifier function to self.

The function should return the passed object. This is needed to chain “extention methods” to this class.

Example:

from kfp.gcp import use_gcp_secret
task = (
    train_op(...)
        .set_memory_request('1G')
        .apply(use_gcp_secret('user-gcp-sa'))
        .set_memory_limit('2G')
)

attrs_with_pipelineparams = ['node_selector', 'volumes', 'pod_annotations', 'pod_labels', 'num_retries', 'init_containers', 'sidecars', 'tolerations']

delete(flags: Optional[List[str]] = None): Returns a ResourceOp which deletes the resource.

property inputs: List of PipelineParams that will be converted into input parameters (io.argoproj.workflow.v1alpha1.Inputs) for the argo workflow.

property resource: Resource object that represents the resource property in io.argoproj.workflow.v1alpha1.Template.

set_caching_options(enable_caching: bool) → kfp.dsl._container_op.BaseOp

Sets caching options for the Op.

Parameters: enable_caching – Whether or not to enable caching for this task.
Returns: Self return to allow chained setting calls.

set_display_name(name: str)

set_retry(num_retries: int, policy: Optional[str] = None, backoff_duration: Optional[str] = None, backoff_factor: Optional[float] = None, backoff_max_duration: Optional[str] = None)

Sets the number of times the task is retried until it’s declared failed.

Parameters

num_retries – Number of times to retry on failures.
policy – Retry policy name.
backoff_duration – The time interval between retries. Defaults to an immediate retry. In case you specify a simple number, the unit defaults to seconds. You can also specify a different unit, for instance, 2m (2 minutes), 1h (1 hour).
backoff_factor – The exponential backoff factor applied to backoff_duration. For example, if backoff_duration=”60” (60 seconds) and backoff_factor=2, the first retry will happen after 60 seconds, then after 120, 240, and so on.
backoff_max_duration – The maximum interval that can be reached with the backoff strategy.

set_timeout(seconds: int)

Sets the timeout for the task in seconds.

Parameters: seconds – Number of seconds.

class kfp.dsl.VolumeSnapshotOp(resource_name: Optional[str] = None, pvc: Optional[str] = None, snapshot_class: Optional[str] = None, annotations: Optional[Dict[str, str]] = None, volume: Optional[kubernetes.client.models.v1_volume.V1Volume] = None, api_version: str = 'snapshot.storage.k8s.io/v1alpha1', **kwargs)[source]

Bases: kfp.dsl._resource_op.ResourceOp

Represents an op which will be translated into a resource template which will be creating a VolumeSnapshot.

TODO(https://github.com/kubeflow/pipelines/issues/4822): Determine the: stability level of this feature.

Parameters

resource_name – A desired name for the VolumeSnapshot which will be created
pvc – The name of the PVC which will be snapshotted
snapshot_class – The snapshot class to use for the dynamically created VolumeSnapshot
annotations – Annotations to be patched in the VolumeSnapshot
volume – An instance of V1Volume
kwargs – See kfp.dsl.ResourceOp

Raises

ValueError – if k8s_resource is provided along with other arguments if k8s_resource is not a VolumeSnapshot if pvc and volume are None if pvc and volume are not None if volume does not reference a PVC

add_affinity(affinity: kubernetes.client.models.v1_affinity.V1Affinity)

Add K8s Affinity.

Parameters

affinity – Kubernetes affinity For detailed spec, check affinity definition
https – //github.com/kubernetes-client/python/blob/master/kubernetes/client/models/v1_affinity.py

Example:

V1Affinity(
    node_affinity=V1NodeAffinity(
        required_during_scheduling_ignored_during_execution=V1NodeSelector(
            node_selector_terms=[V1NodeSelectorTerm(
                match_expressions=[V1NodeSelectorRequirement(
                    key='beta.kubernetes.io/instance-type',
                    operator='In',
                    values=['p2.xlarge'])])])))

add_init_container(init_container: kfp.dsl._container_op.UserContainer)

Add a init container to the Op.

Parameters: init_container – UserContainer object.

add_node_selector_constraint(label_name: Union[str, kfp.dsl._pipeline_param.PipelineParam], value: Union[str, kfp.dsl._pipeline_param.PipelineParam])

Add a constraint for nodeSelector.

Each constraint is a key-value pair label. For the container to be eligible to run on a node, the node must have each of the constraints appeared as labels.

Parameters

label_name (Union[str, PipelineParam]) – The name of the constraint label.
value (Union[str, PipelineParam]) – The value of the constraint label.

add_pod_annotation(name: str, value: str)

Adds a pod’s metadata annotation.

Parameters

name – The name of the annotation.
value – The value of the annotation.

add_pod_label(name: str, value: str)

Adds a pod’s metadata label.

Parameters

name – The name of the label.
value – The value of the label.

add_sidecar(sidecar: kfp.dsl._container_op.Sidecar)

Add a sidecar to the Op.

Parameters: sidecar – SideCar object.

add_toleration(tolerations: kubernetes.client.models.v1_toleration.V1Toleration)

Add K8s tolerations.

Parameters: tolerations – Kubernetes toleration For detailed spec, check toleration definition https://github.com/kubernetes-client/python/blob/master/kubernetes/client/models/v1_toleration.py

add_volume(volume)

Add K8s volume to the container.

Parameters

volume – Kubernetes volumes For detailed spec, check volume definition
https – //github.com/kubernetes-client/python/blob/master/kubernetes/client/models/v1_volume.py

after(*ops): Specify explicit dependency on other ops.

apply(mod_func)

Applies a modifier function to self.

The function should return the passed object. This is needed to chain “extention methods” to this class.

Example:

from kfp.gcp import use_gcp_secret
task = (
    train_op(...)
        .set_memory_request('1G')
        .apply(use_gcp_secret('user-gcp-sa'))
        .set_memory_limit('2G')
)

attrs_with_pipelineparams = ['node_selector', 'volumes', 'pod_annotations', 'pod_labels', 'num_retries', 'init_containers', 'sidecars', 'tolerations']

delete(flags: Optional[List[str]] = None): Returns a ResourceOp which deletes the resource.

property inputs: List of PipelineParams that will be converted into input parameters (io.argoproj.workflow.v1alpha1.Inputs) for the argo workflow.

property resource: Resource object that represents the resource property in io.argoproj.workflow.v1alpha1.Template.

set_caching_options(enable_caching: bool) → kfp.dsl._container_op.BaseOp

Sets caching options for the Op.

Parameters: enable_caching – Whether or not to enable caching for this task.
Returns: Self return to allow chained setting calls.

set_display_name(name: str)

set_retry(num_retries: int, policy: Optional[str] = None, backoff_duration: Optional[str] = None, backoff_factor: Optional[float] = None, backoff_max_duration: Optional[str] = None)

Sets the number of times the task is retried until it’s declared failed.

Parameters

num_retries – Number of times to retry on failures.
policy – Retry policy name.
backoff_duration – The time interval between retries. Defaults to an immediate retry. In case you specify a simple number, the unit defaults to seconds. You can also specify a different unit, for instance, 2m (2 minutes), 1h (1 hour).
backoff_factor – The exponential backoff factor applied to backoff_duration. For example, if backoff_duration=”60” (60 seconds) and backoff_factor=2, the first retry will happen after 60 seconds, then after 120, 240, and so on.
backoff_max_duration – The maximum interval that can be reached with the backoff strategy.

set_timeout(seconds: int)

Sets the timeout for the task in seconds.

Parameters: seconds – Number of seconds.

kfp.dsl.component(func)[source]

Decorator for component functions that returns a ContainerOp.

This is useful to enable type checking in the DSL compiler.

Example

@dsl.component
def foobar(model: TFModel(), step: MLStep()):
  return dsl.ContainerOp()

kfp.dsl.get_pipeline_conf()[source]: Configure the pipeline level setting to the current pipeline Note: call the function inside the user defined pipeline function.

kfp.dsl.graph_component(func)[source]

Decorator for graph component functions.

This decorator returns an ops_group.

Example

# Warning: caching is tricky when recursion is involved. Please be careful
# and set proper max_cache_staleness in case of infinite loop.
import kfp.dsl as dsl
@dsl.graph_component
def flip_component(flip_result):
  print_flip = PrintOp(flip_result)
  flipA = FlipCoinOp().after(print_flip)
  flipA.execution_options.caching_strategy.max_cache_staleness = "P0D"
  with dsl.Condition(flipA.output == 'heads'):
    flip_component(flipA.output)
  return {'flip_result': flipA.output}

kfp.dsl.importer(*args, **kwargs)[source]

kfp.dsl.pipeline(name: Optional[str] = None, description: Optional[str] = None, pipeline_root: Optional[str] = None)[source]

Decorator of pipeline functions.

Example

@pipeline(
  name='my-pipeline',
  description='My ML Pipeline.'
  pipeline_root='gs://my-bucket/my-output-path'
)
def my_pipeline(a: PipelineParam, b: PipelineParam):
  ...

Parameters

name – The pipeline name. Default to a sanitized version of the function name.
description – Optionally, a human-readable description of the pipeline.
pipeline_root – The root directory to generate input/output URI under this pipeline. This is required if input/output URI placeholder is used in this pipeline.

kfp.dsl.python_component(name, description=None, base_image=None, target_component_file: Optional[str] = None)[source]

Decorator for Python component functions.

This decorator adds the metadata to the function object itself.
Args:
name: Human-readable name of the component description: Optional. Description of the component base_image: Optional. Docker container image to use as the base of the

component. Needs to have Python 3.5+ installed.

target_component_file: Optional. Local file to store the component
definition. The file can then be used for sharing.

Returns:
The same function (with some metadata fields set).

Example:
@dsl.python_component(
  name='my awesome component',
  description='Come, Let's play',
  base_image='tensorflow/tensorflow:1.11.0-py3',
)
def my_component(a: str, b: int) -> str:
  ...

Deprecated since version 0.2.6: This decorator does not seem to be used, so we deprecate it. If you need this decorator, please create an issue at https://github.com/kubeflow/pipelines/issues

kfp.dsl.types module