Rocket-Store Python

Using the filesystem as a searchable database.

Rocketstore is a Python library for data storage. It provides an interface for storing and retrieving data in various formats.

ORIGINAL / NODE VERSION: https://github.com/Paragi/rocket-store/

You can download from PyPi repository: https://pypi.org/project/Rocket-Store/

Basic terminology

Rocket-Store was made to replace a more complex database, in a setting that required a low footprint and high performance.

Rocket-Store is intended to store and retrieve records/documents, organized in collections, using a key.

Terms used:

• Collection: name of a collections of records. (Like an SQL table)
• Record: the data store. (Like an SQL row)
• Data storage area: area/directory where collections are stored. (Like SQL data base)
• Key: every record has exactly one unique key, which is the same as a file name (same restrictions) and the same wildcards used in searches.

Compare Rocket-Store, SQL and file system terms:

Features

• Support for file locking.
• Support for creating data storage directories.
• Support for adding auto incrementing sequences and GUIDs to keys.

Usage

To use Rocketstore, you must first import the library:

from Rocketstore import Rocketstore

rs = Rocketstore()

usage of constants:

#method 1:
rs = Rocketstore()
rs.post(..., rs._FORMAT_JSON)

#or 

rs.post(..., Rocketstore._FORMAT_JSON)

Post

rs.post(collection="delete_fodders1", key="1", record={"some":"json input"}, flags=Rocketstore._FORMAT_JSON)
# or
rs.post("delete_fodders1", "1", {"some":"json input"}, Rocketstore._FORMAT_JSON)

Stores a record in a collection identified by a unique key

Collection name to contain the records.

Key uniquely identifying the record

No path separators or wildcards etc. are allowed in collection names and keys. Illigal charakters are silently striped off.

Content Data input to store

Options

• _ADD_AUTO_INC: Add an auto incremented sequence to the beginning of the key
• _ADD_GUID: Add a Globally Unique IDentifier to the key

Returns an associative array containing the result of the operation:

• count : number of records affected (1 on succes)
• key: string containing the actual key used

If the key already exists, the record will be replaced.

If no key is given, an auto-incremented sequence is used as key.

If the function fails for any reason, an error is thrown.

Get

Find and retrieve records, in a collection.

rs.get(collection="delete_fodders1")
# or
rs.get("delete_fodders1")
# Get wildcar
rs.get("delete_*")

Collection to search. If no collection name is given, get will return a list of data base assets: collections and sequences etc.

Key to search for. Can be mixed with wildcards '*' and '?'. An undefined or empty key is the equivalent of '*'

Options:

• _ORDER : Results returned are ordered alphabetically ascending.
• _ORDER_DESC : Results returned are ordered alphabetically descending.
• _KEYS : Return keys only (no records)
• _COUNT : Return record count only

Return an array of

• count : number of records affected
• key : array of keys
• result : array of records

NB: wildcards are very expensive on large datasets with most filesystems. (on a regular PC with +10^7 records in the collection, it might take up to a second to retreive one record, whereas one might retrieve up to 100.000 records with an exact key match)

Delete

Delete one or more records, whos key match.

# Delete database
rs.delete()

# Delete collection with content
rs.delete("delete_fodders1")

# Delete wild collection 
rs.delete("delete_*")

# Delete exact key
rs.delete("delete_fodders1", "1")

Collection to search. If no collection is given, THE WHOLE DATA BASE IS DELETED!

Key to search for. Can be mixed with wildcards '*' and '?'. If no key is given, THE ENTIRE COLLECTION INCLUDING SEQUENCES IS DELETED!

Return an array of

• count : number of records or collections affected

Options

Can be called at any time to change the configuration values of the initialized instance

Options:

• data_storage_area: The directory where the database resides. The default is to use a subdirectory to the temporary directory provided by the operating system. If that doesn't work, the DOCUMENT_ROOT directory is used.
• data_format: Specify which format the records are stored in. Values are: _FORMAT_NATIVE - default. and RS_FORMAT_JSON - Use JSON data format.

rs.options(data_format=Rocketstore._FORMAT_JSON)
# or
rs.options(**{
  "data_format": Rocketstore._FORMAT_JSON,
  ...
})

Inserting with Globally Unique IDentifier key

Another option is to add a GUID to the key. The GUID is a combination of a timestamp and a random sequence, formatet in accordance to RFC 4122 (Valid but slightly less random)

If ID's are generated more than 1 millisecond apart, they are 100% unique. If two ID's are generated at shorter intervals, the likelyhod of collission is up to 1 of 10^15.

Contribute

Contributions are welcome. Please open an issue to discuss what you would like to change.

Docs:

• https://packaging.python.org/en/latest/tutorials/packaging-projects/
• https://realpython.com/pypi-publish-python-package/

Publish to Pypi

Local:

python -m pip install build twine
python3 -m build   
twine check dist/*
twine upload dist/*

Live: No need do nothing GitHub have Workflow action its publish auto

Local dev

In root folder run create virtual env virtualenv ./venv && . ./venv/bin/activate and run pip install -e .