CLI Functions

Hypergol provides a set of functions to autogenerate components of the framework. All functions can be called in the command line, boolean flags can be specified as --dryrun or --dryrun=False. For example:

$ python3 -m hypergol.cli.create_<...>  <parameters>

The same command can be executed in python as:

from hypergol.cli.create_<...> import create_<...>
create_<...>(<parameters>)

Hypergol uses Python Fire to wrap python functions and enable CLI execution.

create_project - Autogenerate project directories

hypergol.cli.create_project.create_project(projectName, dryrun=None, force=None)[source]

Generates the project directories and files

Fails if the target directory already exists unless force=True or --force in CLI is set.

Directories:

  • data_models with __init__.py

  • pipelines with __init__.py

  • tasks with __init__.py

  • models with __init__.py

  • modelslocks with __init__.py

  • tests

Executables:

  • make_venv.sh to create a virtual environment

  • run_tests.sh to run tests

  • run_pylint.sh to run linting

Misc:

  • requirements.txt

  • .gitignore

  • README.md

  • LICENSE <- Don’t forget to add current year and your name or change it to the one you want

  • pylintrc

Parameters

  • projectName (string (CamelCase)) – Name of the project to be created

  • dryrun (bool (default=None)) – If set to True it returns the generated code as a string

  • force (bool (default=None)) – If set to True it overwrites the target file

create_data_model - Autogenerate domain classes

hypergol.cli.create_data_model.create_data_model(className, *args, projectDirectory='.', dryrun=None, force=None, project=None)[source]

Generates domain class from the parameters derived from BaseData

Fails if the target file already exists unless force=True or --force in CLI is set.

Parameters

  • className (string (CamelCase)) – Name of the class to be created

  • projectDirectory (string (default='.')) – Location of the project directory, the code will be created in projectDirectory/data_models/class_name.py.

  • dryrun (bool (default=None)) – If set to True it returns the generated code as a string

  • force (bool (default=None)) – If set to True it overwrites the target file

  • *args (List of strings) – member variables string representation of the member variable in “name:type”, “name:List[type]” or “name:type:id” format

Returns

content – The generated code if dryrun is specified

Return type

string

create_task - Autogenerate processing tasks

hypergol.cli.create_task.create_task(className, *args, projectDirectory='.', dryrun=None, force=None, source=False)[source]

Generates task class from the parameters derived from Task

Fails if the target file already exists unless force=True or --force in the CLI is set.

Setting the --source will generate a different template that have stubs with the functions that need to be overwritten.

Parameters

  • className (string (CamelCase)) – Name of the class to be created

  • projectDirectory (string (default='.')) – Location of the project directory, the code will be created in projectDirectory/data_models/class_name.py.

  • dryrun (bool (default=None)) – If set to True it returns the generated code as a string

  • force (bool (default=None)) – If set to True it overwrites the target file

  • source (bool (default=False)) – If set to True the class will generate stubs for functions to be overwritten

  • *args (List of strings (CamelCase)) – Classes to be imported into the generated code from the datamodel, fails if class not found

Returns

content – The generated code if dryrun is specified

Return type

string

create_pipeline - Autogenerate processing pipelines

hypergol.cli.create_pipeline.create_pipeline(pipeLineName, *args, projectDirectory='.', dryrun=None, force=None, project=None)[source]

Generates a pipeline script from the parameters

Fails if the target file already exists unless force=True or --force in CLI is set.

Generates pipe_line_name.py in pipelines, imports all the classes listed in *args and creates stubs for them to be filled. Also creates the executable pipe_line_name.sh in the project directory with examples how to pass parameters from the shell.

Parameters

  • pipeLineName (string (CamelCase)) – Name of the pipeline to be created

  • projectDirectory (string (default='.')) – Location of the project directory, the code will be created in projectDirectory/data_models/class_name.py.

  • dryrun (bool (default=None)) – If set to True it returns the generated code as a string

  • force (bool (default=None)) – If set to True it overwrites the target file

  • *args (List of strings (CamelCase)) – Classes to be imported into the generated code from the data model, fails if class not found in either data_models or tasks

Returns

  • content (string) – The generated code if dryrun is specified

  • scriptContent (string) – The generated shell script to run the pipeline if dryrun is specified

create_model_block - Autogenerate Tensorflow/Torch model blocks

hypergol.cli.create_model_block.create_model_block(className, projectDirectory='.', dryrun=None, force=None)[source]

Generates a Model Block class.

The file will be located in project_name/models/blocks/block_name.py

Parameters

className (string (CamelCase)) – Name of the class to be created

create_model - Autogenerate Tensorflow/Torch models

Also generates training script and batch data manager class

hypergol.cli.create_model.create_model(modelName, trainingClass, evaluationClass, inputClass, outputClass, *args, projectDirectory='.', dryrun=None, force=None)[source]

Generates stubs for the Tensorflow model, data processing class and training script and shell script to run it from the command line. Shell scripts will be located in the project main directory (which should be the current directory when running them) and model files will be located in project_name/models/model_name/*.py.

After creation the user must implement the process_training_batch() , process_evaluation_batch(), process_input_batch() and process_output_batch member functions that take trainingClass, evaluationClass, inputClass and outputClass respectively.

The model must implement the get_loss(), produce_metrics() and get_outputs() functions (see documentation of BaseTensorflowModel and the Tutorial for more detailed instructions)

The training script is generated with example stubs that should be modified to align with the created model.

Parameters

  • modelName (string) – Name of the model

  • trainingClass (BaseData) – Datamodel class (must exist) of the Dataset that contains the training data

  • evaluationClass (BaseData) – Datamodel class (must exist) that will contain the evaluation data

  • inputClass (BaseData) – Datamodel class (must exist) that will be used as the input when serving the model

  • outputClass (BaseData) – Datamodel class (must exist) that will be returned as output when serving the model

  • *args (BaseTensorflowModelBlock) – Names of blocks that will build up the model

list_datasets - List existing datasets

hypergol.cli.list_datasets.list_datasets(dataDirectory, projectDirectory='.', pattern=None)[source]

Convenience function to list existing datasets in the project

Parameters

  • dataDirectory (string) – location of the project data

  • projectDirectory (string (default .)) – location of the data directory

Please see list_datasets() in HypergolProject for details.

diff_data_model - Print differences between data model versions

hypergol.cli.diff_data_model.diff_data_model(commit, *args, projectDirectory='.', dryrun=None, force=None)[source]

Convenience function to compare old data model class definitions to the current one

Please see diff_data_model() in HypergolProject for details.

create_old_data_model - Create an older version of the datamodel from git

hypergol.cli.create_old_data_model.create_old_data_model(commit, *args, projectDirectory='.', dryrun=None, force=None)[source]

Create an older version of a data model class from git

Please see create_old_data_model() in HypergolProject for details.