Testing Framework for SparkPipelineFramework
Project description
SparkPipelineFramework.Tests
Testing framework that can tests SparkPipelineFramework library by just providing input files to setup before running the transformer and output files to use for verifying the output
Usage
- Create a folder structure similar to the folder structure of your library in SparkPipelineFramework (This is how the Testing Framework finds the Transformer to run)
- Create an input folder and put in files that represent the input views. These files can be csv, json or parquet
- (Optionally) Create an input_schema folder and put in any schemas you want applied to the above views. This follows the Spark Json Schema format.
- (Optional) Create an output folder and put in files that represent the output views you expect. These files can be csv, json or parquet
- (Optional) Create an output_schema folder and put in any schemas you want applied to the output views
- Copy the following test code and put it in a test file in this folder
from pathlib import Path
from pyspark.sql import SparkSession
from spark_pipeline_framework_testing.test_runner import SparkPipelineFrameworkTestRunner
def test_folder(spark_session: SparkSession) -> None:
data_dir: Path = Path(__file__).parent.joinpath('./')
SparkPipelineFrameworkTestRunner.run_tests(spark_session=spark_session, folder_path=data_dir)
- Now just run this test.
Note: the test finds files in sub-folders too.
Example
For the transformer defined here: https://github.com/imranq2/SparkPipelineFramework.Testing/tree/main/library/features/people/my_people_feature You can find the test here: https://github.com/imranq2/SparkPipelineFramework.Testing/tree/main/tests/library/features/people/my_people_feature
Publishing a new package
- Create a new release
- The GitHub Action should automatically kick in and publish the package
- You can see the status in the Actions tab
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Close
Hashes for sparkpipelineframework.testing-2.1.2.tar.gz
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 2002829f71dd91a8b02a8c9879653e5944df9f3d215ada26aac353edd4358f9f |
|
| MD5 | a7780a1676ad26813fd2094dae694203 |
|
| BLAKE2b-256 | b1612d0b330e14b20d184d0e4bb5366c0b712dbbe32e8d5dfdeaea2546fe5024 |
Close
Hashes for sparkpipelineframework.testing-2.1.2-py3-none-any.whl
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 27cf96e411e3f31106223a1826edec92e7156bd392d609e96cbdc4905d48ce0a |
|
| MD5 | 4617a85caa765ccd1e196d2262d1b5bc |
|
| BLAKE2b-256 | 773ba87d6987a080814c6d93304c7c5ab067c1799e6dc63bd419aa4fead2be25 |
