Skip to main content

Takes a deployment in your kubernetes cluster and turns its pod template into a KubernetesPodOperator object.

Project description

Airflow KubernetesPodOperatorFromDeployment

PyPI version PyPI license PyPI pyversions

Or airkupofrod for short, is a tiny package which does one thing - takes a deployment in your kubernetes cluster and turns allows you to use its pod template as a KubernetesPodOperator object. It does this by providing the KubernetesPodOperatorFromDeployment operator.

airkupofrod supports 1.10.9<=airflow<2

Installation and usage

Ensure your airflow image has the python package airkupofrod installed

pip install airkupofrod

Then in your dags:

from airkupofrod.operator import KubernetesPodOperatorFromDeployment

my_kupofrod_task = KubernetesPodOperatorFromDeployment(
    deployment_labels={"app": "my-app"}, # deployment labels to lookup by
    deployment_fields={"metadata.name": "my-app-deploy-template"}, # deployment fields to lookup by
    deployment_namespace="some-ns", # where the deployment lives
    namespace="default", # where the pod will be deployed
    task_id="my-kupofrod-task", 
    dag=dag,
    in_cluster=True, 
) 

You will also need to make sure that a service account attached to your airflow pods has the a role capable of listing deployments bound to it. See role-binding for an example of this.

This is in addition to the role bindings necessary for the KubernetesPodOperator to work which can be seen in the airflow helm chart

Developing

Skaffold is used to test and develop inside kubernetes.

After ensuring you have:

  • Skaffold
  • Helm
  • Some type of k8s cluster available

Run:

skaffold dev --force=false --cleanup=false --status-check=false --port-forward

Then navigate to http://localhost:8080 and enable and trigger a run of the test deployments dag.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

airkupofrod-0.7.0.tar.gz (25.1 kB view hashes)

Uploaded Source

Built Distribution

airkupofrod-0.7.0-py3-none-any.whl (11.3 kB view hashes)

Uploaded Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page