Flexible Data Integrator
Project description
Flexible Dataset Integrator (FDI) helps data producers and processors to build connections into isolated heterogeneous datasets. to assemble, organize, and integrate data into self-describing, modular, hierarchical, persistent, referenceable Products, whose component datasets keep their own characteristicss and are easily accessible.
FDI provides scripting-friendly APIs and tools to define custom Products and generating Python class files. An integrated Product takes care of inter-platform compatibility, string representation, serialisation to simple exchange format, persistence to disk or server, and carrying references of other Products, enabling context-building and lazy-loading.
FDI’s base data model is defined in sub-package dataset. Persistent data access using storage pools, referencing, and Universal Resource Names, and referencie-carrying product Context, are defined in sub-package pal. A reference RESTful API server designed to communicate with a data processing server/docker using the data model, and a reference HTTP pool server are in sub-package pns.
Install/Uninstall
<a href=”http://mercury.bao.ac.cn:9006/mh/fdi/-/commits/master”><img alt=”pipeline status” src=”http://mercury.bao.ac.cn:9006/mh/fdi/badges/master/pipeline.svg” /></a>
For Users
If you need to use fdi, install from PyPI:
python3 -m pip install fdior from git repo:
python3 -m pip install http://mercury.bao.ac.cn:9006/mh/fdi/-/archive/master/fdi-master.tar.gzIf you want to install the develop dependencies:
python3 -m pip install fdi[DEV]or
python3 -m pip install http://mercury.bao.ac.cn:9006/mh/fdi/-/archive/develop/fdi-develop.tar.gz#egg=fdi[DEV]If you do not need to run tests, remove [DEV] to save time and disk space.
To uninstall:
python3 -m pip uninstall fdiFor Developers (or Those who are Not Sure which to Choose)
To install
FDIINSTDIR=/tmp # change this to your installation dir
cd $FDIINSTDIR
git clone http://mercury.bao.ac.cn:9006/mh/fdi.git
cd fdi
git checkout develop
make install EXT="[DEV]"If you want to install the master branch, remove the git checkout develop line above.
To test your installation
make testTo Generate Python Product Source Code
Re-generate baseproduct.py and product.py from YAML schema files in fdi/dataset/resources:
make pyLearn/Try the HTTP Pool Server APIs
The APIs are documented in fdi/httppool/schema/pools.yml with OpenAPI 3. Run this to see and try out with Swagger API Docs when the server is running:
http://127.0.0.1:5000/apidocsModify/Generate Documents
If you plan to compile documents in the docs directory, generate diagrams, API files, or HTML pages, run (in that order, respectively):
First run this once to install necessary packages:
python3 -m pip install -U -e .[PUB]Then when you need to make new class diagrams, API docs, or HTML pages:
make docs_plots
make docs_api
make docs_htmlThe generated HTML page is at docs/html/index.html.
Run Servers and Quick Tests
If you plan to run the pns and/or the http pool server locally, first install the dependencies:
python3 -m pip install -e .[SERV]
make installpnsTo test your pns servers installation, in one window, run:
make runserverin another window run:
make testpnsTo test your httppool servers installation, make sure to stop other server such as pnsserver above, then in one window, run:
make runpoolserverin another window run:
make testhttpDocker Containers
fdi
Get the fdi docker with running FDI environment:
docker pull mhastro/fdior build the image locally:
make build_dockerLaunch and login
make launch_docker
make ithttppool
Also available is a HttpPool server made from Ubuntu and apache:
docker pull mhastro/httppoolTo build the image locally:
make build_serverLaunch and connect:
make launch_server
curl -i http://127.0.0.1:9884/v0.8/Run the above in the package root directory of fdi. A file named .secret is needed by the build and launch commands. This is an example:
PNS_PORT=9884 PNS_USERNAME=... PNS__PASSWORD=... PNS_MQ_HOST=123.45.67.89 PNS_MQ_PORT=9876 PNS_MQ_USER=... PNS_MQ_PASS=...
For More
For more examples see tests/test_*.py.
Read more on package introduction, description, quick start, and API documents on readthedocs.io.
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
