Entry point for the Nessie Airflow provider.
Project description
Nessie Airflow Provider
Documentation: https://projectnessie.github.io/nessie_provider
Source Code: https://github.com/projectnessie/nessie_provider
Usage
To use in airflow install via pip pip install airflow-provider-nessie
. See Nessie Documentation for
instructions on starting and using a Nessie server.
Operators and Hooks
To interact with Nessie from an airflow DAG you have the following options:
- Nessie Hook: register as a connection w/ Airflow and store your Nessie url and credentials
- Create reference operator: Create a Branch or Tag as part of an airflow DAG
- Delete reference operator: Delete a Branch or Tag as part of an airflow DAG
- Commit operator: commit objects to the Nessie database on a given branch
- Merge operator: merge one branch into another
These can be seen in action by looking at the Example DAGs.
The basic_nessie.py
DAG shows each operator in action and the spark_nessie_iceberg.py
DAG shows a more complicated example of performing an iceberg
transaction in Nessie from the Spark operator.
Development
Setup environement
You should have Pipenv installed. Then, you can install the dependencies with:
pipenv install --dev
After that, activate the virtual environment:
pipenv shell
Run unit tests
You can run all the tests with:
make test
Alternatively, you can run pytest
yourself:
pytest
Format the code
Execute the following command to apply isort
and black
formatting:
make format
License
This project is licensed under the terms of the Apache Software License 2.0.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for airflow-provider-nessie-0.1.2.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 05636b5fcd484f9d0eda7c68f4eb010d0c41cffa270f2974d59b40b948045834 |
|
MD5 | 1a233bec3e0e174897b477ebb565e90a |
|
BLAKE2b-256 | 68a87ba727ee5d31446f56bd5b4cdb110d00432556fc897d9db637d984f7b24b |
Hashes for airflow_provider_nessie-0.1.2-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | bd004f20b58b3d393d5035803fa1ead0ef9ad4ea16193f0a7c20e9fcaf29d7a1 |
|
MD5 | a11750028668a799a741dc2064582f9c |
|
BLAKE2b-256 | 350f82ef17ea4aed05c2a8fae9aebbb19c7e2815b5c5b7cedb8f01c8592fa127 |