The RisingWave adapter plugin for dbt
Project description
dbt-risingwave
A RisingWave adapter plugin for dbt.
RisingWave is a cloud-native streaming database that uses SQL as the interface language. It is designed to reduce the complexity and cost of building real-time applications. https://www.risingwave.com
dbt enables data analysts and engineers to transform their data using the same practices that software engineers use to build applications. Use dbt for data transformations in RisingWave
Getting started
The package has not been published to PyPI, please install it via git.
- Install
dbt-risingwave
python3 -m pip install dbt-risingwave
- Get
RisingWaverunning
Please follow this guide to setup a functional RisingWave instance.
- Configure
dbtprofile file
The profile file is located in ~/.dbt/profiles.yml. Here's an example of how to use it with RisingWave.
default:
outputs:
dev:
type: risingwave
host: 127.0.0.1
user: root
pass: ""
dbname: dev
port: 4566
schema: public
target: dev
- Run
dbt debugto check whether configuration is correct.
Models
The dbt models for managing data transformations in RisingWave is similar to typical dbt sql models. The main differences are the materializations. We customized the materializations to fit the data processing model of RisingWave.
| Materializations | INFO |
|---|---|
| materialized_view | Create a materialized view. This materialization is corresponding to the incremental one in dbt. To use this materialization, add {{ config(materialized='materialized_view') }} to your model SQL files. |
| materializedview | (Deprecated) only for backward compatibility, use materialized_view instead |
| ephemeral | This materialization uses common table expressions in RisingWave under the hood. To use this materialization, add {{ config(materialized='ephemeral') }} to your model SQL files. |
| table | Create a table. To use this materialization, add {{ config(materialized='table') }} to your model SQL files. |
| view | Create a view. To use this materialization, add {{ config(materialized='view') }} to your model SQL files. |
| incremental | Use materialized_view instead if possible, since RisingWave is designed to use materialized view to manage data transformation in an incremental way. From v1.7.3, dbt-risingwave support incremental model to give users better control of when to update their model. This model will update table in a batch way incrementally. |
| source | Define a source {{ config(materialized='source') }}. You need to provide your create source statement as a whole in this model. |
| table_with_connector | Define a table with a connector {{ config(materialized='table_with_connector') }}. You need to provide your create table with connector statement as a whole in this model. Because dbt table has its own semantics, RisingWave use table_with_connector to distinguish itself from it. |
| sink | Define a sink {{ config(materialized='sink') }}. You need to provide your create sink statement as a whole in this model. |
To learn how to use, you can check RisingWave offical example dbt_rw_nexmark.
DBT RUN behavior
dbt run: only create new models (if not exists) without dropping any models.dbt run --full-refresh: drop models and create the new ones. This command can make sure your streaming pipelines are consistent with what you define in dbt models.
Graph operators
Graph operators is useful when you want to only recreate a subset of your models.
dbt run --select "my_model+" # select my_model and all children
dbt run --select "+my_model" # select my_model and all parents
dbt run --select "+my_model+" # select my_model, and all of its parents and children
Tests
All items below have been tested against the the latest RisingWave daily build verison.
- Offical example jaffle_shop is tested.
- RisingWave offical example dbt_rw_nexmark is tested.
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for dbt_risingwave-1.7.4-py3-none-any.whl
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | f07fe113421bd9b5329d40beb02e1adb52f4c0456b6d19d7dad124edee73bb6e |
|
| MD5 | 923538a419c0273adf00fdcf23fd4585 |
|
| BLAKE2b-256 | 0b70052cc17f6adc6d2c0fab41b028a55f67e2482fd9cc0150fb209b41181f05 |
