This section covers common ways to customize your Dagster Helm deployment.
Dagster allows you to pass custom configuration to the Kubernetes Jobs and Pods created by Dagster during execution.
If your instance is using the K8sRunLauncher, you can configure custom configuration for every run launched by Dagster by setting the k8sRunLauncher.runK8sConfig dictionary in the Helm chart.
k8sRunLauncher.runK8sConfig is a dictionary with the following keys:
containerConfig: The Pod's Container.podSpecConfig: The Pod's PodSpec.podTemplateSpecMetadata: The Pod's Metadata.jobSpecConfig: The Job's JobSpec.jobMetadata: The Job's Metadata.The values for each of these keys is a dictionary with the YAML configuration for the underlying Kubernetes object. The Kubernetes object fields can be configured using either snake case (for example, volume_mounts) or camel case (volumeMounts). For example:
runLauncher: type: K8sRunLauncher config: k8sRunLauncher: runK8sConfig: containerConfig: # raw config for the pod's main container resources: limits: cpu: 100m memory: 128Mi podTemplateSpecMetadata: # raw config for the pod's metadata annotations: mykey: myvalue podSpecConfig: # raw config for the spec of the launched's pod nodeSelector: disktype: ssd jobSpecConfig: # raw config for the kubernetes job's spec ttlSecondsAfterFinished: 7200 jobMetadata: # raw config for the kubernetes job's metadata annotations: mykey: myvalue
The dagster-k8s/config tag allows you to pass custom configuration to the Kubernetes Jobs and Pods created by Dagster for specific jobs or ops.
dagster-k8s/config is a dictionary with the following keys:
container_config: The Pod's Container.pod_spec_config: The Pod's PodSpec.pod_template_spec_metadata: The Pod's Metadata.job_spec_config: The Job's JobSpec.job_metadata: The Job's Metadata.The values for each of these keys is a dictionary with the YAML configuration for the underlying Kubernetes object. The Kubernetes object fields can be configured using either snake case (for example, volume_mounts) or camel case (volumeMounts).
If your instance is using the K8sRunLauncher or CeleryK8sRunLauncher, you can use the dagster-k8s/config tag on a Dagster job. For example:
@job( tags={ "dagster-k8s/config": { "container_config": { "resources": { "requests": {"cpu": "250m", "memory": "64Mi"}, "limits": {"cpu": "500m", "memory": "2560Mi"}, }, "volume_mounts": [ {"name": "volume1", "mount_path": "foo/bar", "sub_path": "file.txt"} ], }, "pod_template_spec_metadata": { "annotations": {"cluster-autoscaler.kubernetes.io/safe-to-evict": "true"} }, "pod_spec_config": { "volumes": [{"name": "volume1", "secret": {"secret_name": "volume_secret_name"}}], "affinity": { "node_affinity": { "required_during_scheduling_ignored_during_execution": { "node_selector_terms": [ { "match_expressions": [ { "key": "beta.kubernetes.io/os", "operator": "In", "values": ["windows", "linux"], } ] } ] } } }, }, }, }, ) def my_job(): my_op()
If your Dagster job is configured with an executor that runs each op in its own pod, like the k8s_job_executor or celery_k8s_job_executor, you can also use the dagster-k8s/config tag on a Dagster op to control the Kubernetes configuration for that specific op. For example:
@op( tags={ "dagster-k8s/config": { "container_config": { "resources": { "requests": {"cpu": "200m", "memory": "32Mi"}, } }, } } ) def my_op(context): context.log.info("running") @job(executor_def=k8s_job_executor) def my_job(): my_op()
Non-k8s run launchers and executors will ignore the dagster-k8s/config tag.
If a Kubernetes configuration dictionary (like container_config) is specified at both the instance level in the Helm chart and in a specific Dagster job or op, the two dictionaries will be shallowly merged. The more specific configuration takes precedence if the same key is set in both dictionaries.
For example, if k8sRunLauncher.runK8sConfig.podSpecConfig is set to {"nodeSelector": {"disktype": "ssd"}, "dns_policy": "ClusterFirst"} in the Helm chart, but a specific job has the pod_spec_config key in the dagster-k8s/config tag set to {"nodeSelector": {"region": "east"}}, the node selector from the job and the DNS policy from the Helm chart will be applied, since only the node selector is overridden in the job.
In a real deployment, users will likely want to set up an external PostgreSQL database and configure the postgresql section of values.yaml.
postgresql: enabled: false postgresqlHost: "postgresqlHost" postgresqlUsername: "postgresqlUsername" postgresqlPassword: "postgresqlPassword" postgresqlDatabase: "postgresqlDatabase" service: port: 5432
Supplying .Values.postgresql.postgresqlPassword will create a Kubernetes Secret with key postgresql-password, containing the encoded password. This secret is used to supply the Dagster infrastructure with an environment variable that's used when creating the storages for the Dagster instance.
If you use a secrets manager like Vault, it may be convenient to manage this Secret outside of the Dagster Helm chart. In this case, the generation of this Secret within the chart should be disabled, and .Values.global.postgresqlSecretName should be set to the name of the externally managed Secret.
global: postgresqlSecretName: "dagster-postgresql-secret" generatePostgresqlPasswordSecret: false
Users will likely want to permission a ServiceAccount bound to a properly scoped Role to launch Jobs and create other Kubernetes resources.
Users will likely want to use Secrets for managing secure information such as database logins.
It may be desirable to manage two Helm releases for your Dagster deployment: one release for the Dagster infrastructure, which consists of Dagit and the Daemon, and another release for your User Code, which contains the definitions of your pipelines written in Dagster. This way, changes to User Code can be decoupled from upgrades to core Dagster infrastructure.
To do this, we offer the dagster chart and the dagster-user-deployments chart.
$ helm search repo dagster NAME CHART VERSION APP VERSION DESCRIPTION dagster/dagster 0.11.0 0.11.0 Dagster is a system for building modern data ap... dagster/dagster-user-deployments 0.11.0 0.11.0 A Helm subchart to deploy Dagster User Code dep...
To manage these separate deployments, we first need to isolate Dagster infrastructure to its own deployment. This can be done by disabling the subchart that deploys the User Code in the dagster chart. This will prevent the dagster chart from creating the services and deployments related to User Code, as these will be managed in a separate release.
dagster-user-deployments: enableSubchart: false
Next, the workspace for Dagit must be configured with the future hosts and ports of the services exposing access to the User Code.
dagit: workspace: enabled: true servers: - host: "k8s-example-user-code-1" port: 3030 - ...
Finally, the dagster-user-deployments subchart can now be managed in its own release. The list of possible overrides for the subchart can be found in its values.yaml.
helm upgrade --install user-code dagster/dagster-user-deployments -f /path/to/values.yaml
If you use a Kubernetes distribution that supports the TTL Controller, then Completed and Failed Jobs (and their associated Pods) will be deleted after 1 day. The TTL value can be modified in your job tags:
@job( tags = { 'dagster-k8s/config': { 'job_spec_config': { 'ttl_seconds_after_finished': 7200 } } } ) def my_job(): my_op()
If you do not use a Kubernetes distribution that supports the TTL Controller, then you can run the following commands:
Delete dagster Jobs older than one day
kubectl get job | grep -e dagster-run -e dagster-step | awk 'match($4,/[0-9]+d/) {print $1}' | xargs kubectl delete job
Delete completed Pods older than one day
kubectl get pod | grep -e dagster-run -e dagster-step | awk 'match($3,/Completed/) {print $0}' | awk 'match($5,/[0-9]+d/) {print $1}' | xargs kubectl delete pod
You should now be familiar with the common ways to customize your Dagster Helm deployment.