django-elastic-appsearch 2.1.0

Installing

Install Django Elastic App Search:

pip install django_elastic_appsearch

Add it to your INSTALLED_APPS:

INSTALLED_APPS = (
    ...
    'django_elastic_appsearch',
    ...
)

Add the Elastic App Search URL and Key to your settings module:

APPSEARCH_HOST = 'localhost:3002'
APPSEARCH_API_KEY = 'some_appsearch_api_token'

Configuring app search indexable models

Single engine

Configure the Django models you want to index to Elastic App Search. To index to one engine you can do this by inheriting from the AppSearchModel, and then setting some meta options.

AppsearchMeta.appsearch_engine_name - Defines which engine in your app search instance your model will be indexed to.

AppsearchMeta.appsearch_serialiser_class - Defines how your model object will be serialised when sent to your elastic app search instance. The serialiser and fields used here derives from Serpy, and you can use any of the serpy features like method fields.

Example:

from django_elastic_appsearch.orm import AppSearchModel
from django_elastic_appsearch import serialisers

class CarSerialiser(serialisers.AppSearchSerialiser):
    full_name = serialisers.MethodField()
    make = serialisers.StrField()
    model = serialisers.StrField()
    manufactured_year = serialisers.Field()

    def get_full_name(self, instance):
        return '{} {}'.format(make, model)


class Car(AppSearchModel):

    class AppsearchMeta:
        appsearch_engine_name = 'cars'
        appsearch_serialiser_class = CarSerialiser

    make = models.CharField(max_length=100)
    model = models.CharField(max_length=100)
    manufactured_year = models.CharField(max_length=4)

Multi engine

Configure the Django models you want to index to Elastic App Search. To index to multiple engines you can do this by inheriting from the AppSearchMultiEngineModel, and then setting a meta option.

AppsearchMeta.appsearch_serialiser_engine_pairs - A list of tuples of serialisers then engines that define which engine in your app search instance your model will be indexed to and how your model object will be serialised when sent to your elastic app search instance. The serialiser and fields used here derives from Serpy, and you can use any of the serpy features like method fields.

Example:

from django_elastic_appsearch.orm import AppSearchModel
from django_elastic_appsearch import serialisers

class CarSerialiser(serialisers.AppSearchSerialiser):
    full_name = serialisers.MethodField()
    make = serialisers.StrField()
    model = serialisers.StrField()
    manufactured_year = serialisers.Field()

    def get_full_name(self, instance):
        return '{} {}'.format(make, model)


class Truck(AppSearchMultiEngineModel):
    """A truck."""

    class AppsearchMeta:
        appsearch_serialiser_engine_pairs = [(CarSerialiser, "trucks")]

    make = models.TextField()
    model = models.TextField()
    year_manufactured = models.DateTimeField()

Using model and queryset methods to index and delete documents

Then you can call index_to_appsearch and delete_from_appsearch from your model objects.

Send the car with id 25 to app search.

from mymodels import Car

car = Car.objects.get(id=25)
car.index_to_appsearch()

Delete the car with id 21 from app search.

from mymodels import Car

car = Car.objects.get(id=21)
car.delete_from_appsearch()

Calling these on an AppSearchModel will return a single response object, and calling them on an AppSearchMultiEngineModel will return a list of response objects.

You can also call index_to_appsearch and delete_from_appsearch on QuerySets of AppSearchModel

Send all cars where the make is ‘Toyota’ to app search.

cars = Car.objects.filter(make='Toyota')
cars.index_to_appsearch()

Delete all cars where the make is ‘Saab’ from app search

cars = Car.objects.filter(make='Saab')
cars.delete_from_appsearch()

index_to_appsearch methods on the QuerySet and your model also supports an optional update_only parameter which takes in a boolean value. If update_only is set to True, the operation on the app search instance will be carried out as a PATCH operation. This will be useful if your Django application is only doing partial updates to the documents.

This will also mean that your serialisers can contain a subset of the fields for a document. This will be useful when two or more Django models or applications are using the same app search engine to update different sets of fields on a single document type.

Example below (Continued from the above Car example):

from django.db import models
from django_elastic_appsearch.orm import AppSearchModel
from django_elastic_appsearch import serialisers

class CarVINNumberSerialiser(serialisers.AppSearchSerialiser):
    vin_number = serialisers.StrField()

class CarVINNumber(AppSearchModel):

    class AppsearchMeta:
        appsearch_engine_name = 'cars'
        appsearch_serialiser_class = CarVINNumberSerialiser

    car = models.OneToOneField(
        Car,
        on_delete=models.CASCADE,
        primary_key=True
    )
    vin_number = models.CharField(max_length=100)

    def get_appsearch_document_id(self):
        return 'Car_{}'.format(self.car.id)

from mymodels import CarVINNumber

car_vin = CarVINNumber.objects.filter('car__id'=25).first()
car_vin.vin_number = '1M8GDM9A_KP042788'
car_vin.save()
car_vin.refresh_from_db()
car_vin.index_to_appsearch(update_only=True)

You’ll notice that we’ve set the appsearch_engine_name to cars so that the VIN number updates will go through to the same engine. You’ll also notice that we’ve overridden the get_appsearch_document_id method to make sure that VIN number updates do go through the same related car document.

The above example will update the car document with id 25 with the new VIN number and leave the data for the rest of the fields intact.

Important note: PATCH operations on Elastic App Search cannot create new schema fields if you submit schema fields currently unknown to your engine. So always make sure you’re submitting values for existing schema fields on your engine.

Use with your own custom queryset managers

If you want to specify custom managers which also has this functionality, you can inherit from django_elastic_appsearch.orm.AppSearchQuerySet

from django_elastic_appsearch.orm import AppSearchModel, AppSearchQuerySet

class MyCustomQuerySetManager(AppSearchQuerySet):
    def my_custom_queryset_feature(self):
        # Do Something cool
        pass

class MyCustomModel(AppSearchModel):
    field_1 = models.CharField(max_length=100)

    # Set the custom manager
    objects = MyCustomQuerySetManager.as_manager()

Use a custom document id for appsearch

By default, the unique document ID which identifies your model objects in app search is set to <model_name>_<object_id>. If we take the car example above, a Car object with an id of 543 will have the document ID Car_543 in app search.

You can customise this value by overriding the get_appsearch_document_id method on your model class.

Eg. You can do the following to make sure that the document ID on appsearch is exactly the same as the ID on your model object.

class Car(AppSearchModel):

    class AppsearchMeta:
        appsearch_engine_name = 'cars'
        appsearch_serialiser_class = CarSerialiser

    make = models.CharField(max_length=100)
    model = models.CharField(max_length=100)
    manufactured_year = models.CharField(max_length=4)

    def get_appsearch_document_id(self):
        return self.id

Settings

This package provides various Django settings entries you can use to configure your connection to the Elastic App Search instance you’re using.

APPSEARCH_HOST = 'localhost:3002'

APPSEARCH_API_KEY = 'private-key'

APPSEARCH_USE_HTTPS = False

APPSEARCH_CHUNK_SIZE = 50

APPSEARCH_INDEXING_ENABLED = True

APPSEARCH_HOST = 'localhost:3002'
APPSEARCH_API_KEY = 'private-key'
APPSEARCH_USE_HTTPS = False
APPSEARCH_CHUNK_SIZE = 50
APPSEARCH_INDEXING_ENABLED = True

Writing Tests

This package provides a test case mixin called MockedAppSearchTestCase which makes it easier for you to write test cases against AppSearchModel’s and AppSearchMultiEngineModel’s without actually having to run an Elastic App Search instance during tests.

All you have to do is inherit the mixin, and all the calls to Elastic App Search will be mocked. Example below.

from django.test import TestCase
from django_elastic_appsearch.test import MockedAppSearchTestCase
from myapp.test.factories import CarFactory

class BookTestCase(MockedAppSearchTestCase, TestCase):
    def test_indexing_book(self):
        car = CarFactory()
        car.save()
        car.index_to_appsearch()

        self.assertAppSearchModelIndexCallCount(1)

You will have access to the following methods to check call counts to different mocked app search methods.

self.assertAppSearchQuerySetIndexCallCount — Check the number of times index_to_appsearch was called on a appsearch model querysets.

self.assertAppSearchQuerySetDeleteCallCount — Check the number of times delete_from_appsearch was called on an appsearch model querysets.

self.assertAppSearchModelIndexCallCount — Check the number of times index_to_appsearch was called on an appsearch model objects.

self.assertAppSearchModelDeleteCallCount — Check the number of times delete_from_appsearch was called on an appsearch model objects.

If you are using a subclass of AppSearchQuerySet that overrides methods without calling the super class version you can use the queryset_class key word argument to the setUp function to mock it. Example below.

from django.test import TestCase
from django_elastic_appsearch.test import MockedAppSearchTestCase

class BusTestCase(MockedAppSearchTestCase, TestCase):
    """Test the `MockedAppSearchTestCase`."""

    def setUp(self, *args, **kwargs):
        """Load test data."""
        kwargs['queryset_class'] = 'example.querysets.CustomQuerySet.'
        super().setUp(*args, **kwargs)

Using the elastic app search python client

We use the official elastic app search python client under the hood to communicate with the app search instance. So if needed, you can access the app search instance directly and use the functionality of the official elastic app search client. Example below.

from django_elastic_appsearch.clients import get_api_v1_enterprise_search_client

client = get_api_v1_enterprise_search_client()
client.search('cars', 'Toyota Corolla', {})

1.1.5 (2021-05-11)

• Transfer ownership to Infoxchange

• Update documentation

• Dependency upgrades

1.1.4 (2021-05-05)

• Updated documentation

• Dependency upgrades

1.1.3 (2021-04-12)

• Add Django 3.2 to the test matrix

• Dependency upgrades

• Minor fixes

1.1.2 (2021-02-11)

• Security patch

1.1.1 (2021-02-04)

• Dependency upgrades

• Security patch

• Improve returned responses

1.1.0 (2021-01-27)

• Dependecy upgrades

• Add ability to index a model object into multiple app search engines

1.0.2 (2020-12-29)

• Dependency upgrades

1.0.1 (2020-12-15)

• Dependency upgrades

1.0.0 (2020-11-26)

• Python 3.9 support

• Update the project status to stable

0.7.6 (2020-11-26)

• Dependency upgrades

• Update documentation

0.7.5 (2020-10-28)

• Security patch

0.7.4 (2020-10-05)

• Add support for testing overridden queryset methods

• Update documentation

0.7.3 (2020-08-25)

• Remove support for Django 2.0 and Django 2.1

• Add support for Django 3.1

• Update documentation

0.7.2 (2020-08-25)

• Dependency upgrades

0.7.1 (2020-07-31)

• Dependency upgrades

0.7.0 (2020-07-30)

• Implement ability to do partial updates to documents

• Dependency upgrades

0.6.11 (2020-06-22)

• Fix failing dependency check with pipenv

• Dependency upgrades

0.6.9 (2020-05-15)

• Dependency upgrades

0.6.8 (2020-03-31)

• Dependency upgrades

• Security patches

0.6.7 (2020-02-25)

• Dependency upgrades

0.6.6 (2020-02-01)

• Dependency upgrades

0.6.3 (2020-01-03)

• Move from Travis CI to Github Actions

• Documentation updates

0.6.2 (2020-01-02)

• Dependency upgrades

• Documentation improvements

• Add linting for CI

• Setup automatic PyPI releases

0.6.1 (2019-12-24)

• Dependency upgrades

0.6.0 (2019-12-04)

• Remove support for Python 3.5

• Add support for Python 3.8

• Add support for Django 3

• Dependency upgrades

• Bump development status to Beta

0.5.6 (2019-12-03)

• Dependency upgrades

0.5.5 (2019-11-14)

• Dependency upgrades

0.5.4 (2019-10-02)

• Dependency upgrades

0.5.3 (2019-08-28)

• Improve documentation

• Refactor settings name APPSEARCH_URL -> APPSEARCH_HOST

0.5.1 (2019-08-26)

• Improve test coverage

• Improve documentation

• Add serpy as an official dependency

• Bump dependency versions

• Add code of conduct

0.4.2 (2019-08-16)

• Switch to the new official Elastic App Search python client

• Documentation improvements

0.2.3 (2019-08-02)

• Use Pipenv for dependency management

• Configure Dependabot for automatic dependency upgrades

• Remove support for Python 3.4

• Documentation improvements

0.2.2 (2019-07-29)

• Bug fixes

• Documentation improvements

0.1.0 (2019-07-26)

• First release on PyPI.