Postgres to Elasticsearch/OpenSearch sync

Navigation

Verified details

These details have been verified by PyPI
Maintainers
Avatar for taina from gravatar.com taina

Unverified details

These details have not been verified by PyPI
Project links
Meta
  • License: MIT License (MIT)
  • Author: Tolu Aina
  • Maintainer: Tolu Aina
  • Tags change data capture, elasticsearch, opensearch, pgsync, postgres
  • Requires: Python >=3.9.0
Classifiers
Report project as malware

Project description

PostgreSQL to Elasticsearch/OpenSearch sync

Requirements

Postgres setup

Enable logical decoding in your Postgres setting.

  • You also need to set up two parameters in your Postgres config postgresql.conf

    wal_level = logical

    max_replication_slots = 1

Installation

You can install PGSync from PyPI:

$ pip install pgsync

Config

Create a schema for the application named e.g schema.json

Example schema

Example spec

.. code-block::

[
    {
        "database": "[database name]",
        "index": "[Elasticsearch or OpenSearch index]",
        "nodes": {
            "table": "[table A]",
            "schema": "[table A schema]",
            "columns": [
                "column 1 from table A",
                "column 2 from table A",
                ... additional columns
            ],
            "children": [
                {
                    "table": "[table B with relationship to table A]",
                    "schema": "[table B schema]",
                    "columns": [
                      "column 1 from table B",
                      "column 2 from table B",
                      ... additional columns
                    ],
                    "relationship": {
                        "variant": "object",
                        "type": "one_to_many"
                    },
                    ...
                },
                {
                    ... additional children
                }
            ]
        }
    }
]

Environment variables

Setup environment variables required for the application

SCHEMA='/path/to/schema.json'

ELASTICSEARCH_HOST=localhost
ELASTICSEARCH_PORT=9200

PG_HOST=localhost
PG_USER=i-am-root # this must be a postgres superuser or replication user
PG_PORT=5432
PG_PASSWORD=*****

REDIS_HOST=redis
REDIS_PORT=6379
REDIS_DB=0
REDIS_AUTH=*****

Running

Bootstrap the database (one time only)

  • $ bootstrap --config schema.json

Run pgsync as a daemon

  • $ pgsync --config schema.json --daemon

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page