Developer guide

Information: The build system changed in NeoFox v1.2.0 to poetry (https://python-poetry.org/).

Installation of neofox via poetry

Setup poetry once as described under Setup poetry .

Go to your local repository, add poetry to you path and install neofox into a virtual environment managed by poetry. If a poetry.lock file is given in the path, herein defined package versions are used. Otherwise the dependencies listed in pyproject.toml are used to generate the environment:

cd /path/to/neofox/addannot

export PATH="$PATH:/path/to/env/poetry/bin/"

poetry  install

If a wheel archive is needed, it can be generated optionally using (generates them in the dist directory):

poetry  build

To use the installed neofox package, use poetry run e.g.:

poetry  run neofox --help

Installation via pip

If a wheel file was generated with poetry build Installation of neofox via poetry, neofox can also be installed via pip.

pip install dist/neofox-x.y.z.whl

Testing

To run the tests make sure you have a file .env that contains the environment variables described in the configuration section.

Activate the virtual environment generated by poetry:

source .venv/bin/activate

Run integration tests

Run the integration tests as follows:

python -m unittest discover neofox.tests.integration_tests

The integration tests run over some real datasets and they take some time to run.

The integration test that runs the whole program over a relevant dataset can be run as follows:

python -m unittest neofox.tests.integration_tests.test_neofox

Regression tests

This last test (ie: test_neofox) writes its output to a file named neofox/tests/resources/output_yyyymmddHHMMSS.txt. If there is an existing file named neofox/tests/resources/output_previous.txt then it loads both files in memory and compares them. It outputs whether there are some lost or gained columns and for the common columns it evaluates if the values are the same. If they are the same the file output_previous.txt is overwritten by the new file, otherwise it outputs the details of the differing columns.

Run unit tests

The unit tests do not have any dependency and they finish in seconds.

Run the unit tests as follows:

python -m unittest discover neofox.tests.unit_tests

Logging

Logs are written to the standard error and to the output folder by default. Optionally they can be written to a file by setting the environment variable NEOFOX_LOGFILE pointing to the desired file.

Build the protocol buffers models

The protocol buffers model rely on the betterproto library. Install it as follows:

pip install "betterproto[compiler]"

The models and the required scripts are in the folder neofox/models

Build the models into Python code with make models.

Build the HTML documentation with make html (this requires docker).

Setup poetry

Setup poetry once on your system.

E.g. use mamba to install poetry:

mamba create -p /path/to/env/poetry conda-forge::poetry

The required python version for the neofox package can also be installed via mamba (currently python3.7):

# installs python v3.11
mamba create -p /path/to/env/python3.11 conda-forge::python=3.11

Go to your local neofox repository and set the python executable for poetry. This generates the python venv /path/to/neofox/addannot/.venv.

export PATH="$PATH:/path/to/env/poetry/bin/"

cd /path/to/neofox/addannot

# environments are installed in the current project
poetry config virtualenvs.in-project true

poetry env use /path/to/env/python3.11/bin/python3.11

Get information on the currently used python environment variable by running the following command. This will also show the location of the virtual environment in which the package is or will be installed.

poetry env info