Tool to build UNIHAN dataset into datapackage / simple data format.
Project description
cihaidata-unihan - tool to build unihan into simple data format CSV format. Part of the cihai project.
Unihan’s data is disperved across multiple files in the format of:
U+3400 kCantonese jau1 U+3400 kDefinition (same as U+4E18 丘) hillock or mound U+3400 kMandarin qiū U+3401 kCantonese tim2 U+3401 kDefinition to lick; to taste, a mat, bamboo bark U+3401 kHanyuPinyin 10019.020:tiàn U+3401 kMandarin tiàn
cihaidata_unihan/process.py will download Unihan.zip and build all files into a single tabular CSV (default output: ./data/unihan.csv):
char,ucn,kCantonese,kDefinition,kHanyuPinyin,kMandarin 丘,U+3400,jau1,(same as U+4E18 丘) hillock or mound,,qiū 㐁,U+3401,tim2,"to lock; to taste, a mat, bamboo bark",10019.020:"tiàn,tiàn"
process.py supports command line arguments. See cihaidata_unihan/process.py CLI arguments for information on how you can specify custom columns, files, download URL’s and output destinations.
Being built against unit tests. See the Travis Builds and Revision History.
Usage
To download and build your own unihan.csv:
$ ./cihaidata_unihan/process.pyCreates data/unihan.csv.
See cihaidata_unihan/process.py CLI arguments for advanced usage examples.
Structure
# dataset metadata, schema information.
datapackage.json
# (future) when this package is stable, unihan.csv will be provided
data/unihan.csv
# stores downloaded Unihan.zip and it's txt file contents (.gitignore'd)
data/build_files/
# script to download + build a SDF csv of unihan.
cihaidata_unihan/process.py
# unit tests to verify behavior / consistency of builder
tests/*
# python 2/3 compatibility modules
cihaidata_unihan/_compat.py
cihaidata_unihan/unicodecsv.py
# python module, public-facing python API.
__init__.py
cihaidata_unihan/__init__.py
# utility / helper functions
cihaidata_unihan/util.pyCihai is not required for:
data/unihan.csv - simple data format compatible csv file.
cihaidata_unihan/process.py - create a data/unihan.csv.
When this module is stable, data/unihan.csv will have prepared releases, without requires using cihaidata_unihan/process.py. process.py will not require external libraries.
Examples
Related links:
CSV Simple Data Format (SDF): http://data.okfn.org/standards/simple-data-format
Tools: http://data.okfn.org/tools
Python support |
Python 2.7, >= 3.3, pypy/pypy3 |
Source |
|
Docs |
|
Changelog |
https://cihaidata-unihan.git-pull.com/en/latest/history.html |
API |
|
Issues |
|
Travis |
|
Test coverage |
|
pypi |
|
OpenHub |
|
License |
MIT. |
git repo |
|
install dev |
|
tests |
|
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
