Data Related Classes

Hypergol provides a set of classes to define, store and process domain objects.

BaseData - Base class for domain objects

class hypergol.BaseData

Base class for all domain objects.

Extends the Repr convenience base class that provides printing facilities.

Provides to_data and from_data serialisation interfaces.

Provides get_id, get_hash_id interfaces.

Provides test capabilities for all of the above so changes to the derived classes can be checked quickly.

get_id()

Returns the class’s id if exists

get_hash_id()

Returns the class’s hash id if exists, defaults to get_id()

to_data()

Converts class to a dictionary, usually overridden

static to_string(data)

static from_string(data)

classmethod from_data(data)

Creates a class from data, usually overridden

Parameters

data (dictionary form of data representing the object) – Usually the result of a previous to_data() call.

test_get_hash_id()

Tests if the derived class correctly returns a tuple for an id

test_to_data()

Tests if the output of the derived class’s to_data() function can be converted to a string by json.dumps()

test_from_data()

Tests if a roundtrip of self.from_data(self.to_data()) modifies the class

Dataset - Class for storing domain objects

class hypergol.Dataset(dataType, location, project, branch, name, repoData=None, chunkCount=16)

Dataset class to store BaseData objects that is readable/writable in a parallel manner.

Files will be stored in: location/project/branch/name/name_???.jsonl.gz

__init__(dataType, location, project, branch, name, repoData=None, chunkCount=16)

Parameters

• dataType (BaseData) – Type of this dataset, only dataType objects can be stored in this dataset

• location (str) – path the project is in

• project (str) – project name

• branch (str) – branch name

• name (str) – name of this dataset

• repoData (RepoData = None) – stores the commit information at the creation of the dataset

• chunkCount (int = {16 ( default), 256, 4096}) – How many files the data will be stored in, sets the granularity of multithreaded processing

add_dependency(dataset)

Adds the .def file of a dataset to the .def file of this dataset so data lineage can be retraced

Parameters

dataset (Dataset) – dataset that contributes to the generation of this dataset

property directory

Full path of the directory this dataset will be in

init(mode)

Checks the existence of the dataset

Parameters

mode (str = ('w', 'r')) – The mode the dataset is about to be opened

Based on the mode if

• mode==’w’ : fails if the dataset already exists otherwise creates the .def file

• mode==’r’ : fails if the dataset doesn’t exist otherwise compares the data in the .def. file to the definition in the class.

• otherwise : fails due to unknown mode

open(mode)

Opens the dataset for reading or writing

Parameters

mode (str = ('w', 'r')) – The mode the dataset is about to be opened

Returns a DatasetWriter or DatasetReader object that handles the reading or writing of the files through the dataset’s chunks.

get_chunk_ids()

Returns the list of chunk id-s

get_data_chunks(mode)

Initialises the dataset and creates all the Datachunk classes

get_object_chunk_id(objectHashId)

Finds out which chunk the object belongs based on the hash id

delete()

Deletes the files and the directory of the dataset

exists()

True if the dataset’s .def file exists

DatasetWriter - Class for writing datasets

class hypergol.dataset.DatasetWriter(dataset)

Class to write into a dataset

__init__(dataset)

Opens all chunks at once and puts them in a dictionary for easy lookup

Implements context manager for proper file open/close.

Parameters

dataset (Dataset) – Dataset to be written into, at this point it is already established that it doesn’t yet exist.

append(elem)

Writes a single object into the right chunks

close()

Closes the files and writes the .chk file

DatasetReader - Class for reading datasets

class hypergol.dataset.DatasetReader(dataset)

Class to read from a dataset

Implements context manager and iterator. It doesn’t open any file until any reading actually happens and then opens each chunk one by one.

__init__(dataset)

Initialize self. See help(type(self)) for accurate signature.

DataChunk - Internal class for handling files

class hypergol.dataset.DataChunk(dataset, chunkId, mode)

This class represents the file that the data is actually stored in

When opened for writing it implements the append() method and when reading the __iter__() iterator. Upon close, it returns the checksum (SHA1 hash) of the content that was written into it.

__init__(dataset, chunkId, mode)

Parameters

• dataset (Dataset) – The dataset this class chunk belongs to

• chunkId (str) – The hexadecimal identified of this chunk

• mode (str = ('w' or 'r')) – The mode this chunk was created to be opened in, determined by Dataset.get_data_chunks()

property fileName

Name of the file the data will be stored

open()

Opens the chunk according to the mode specified at creation

close()

Closes the file handler and gets the checksum and returns it as DataChunkChecksum object

append(value)

Adds a data model object to the file, raises an error if the type doesn’t match the dataset’s type or the hash of the object doesn’t match the chunkId

Parameters

value (object) – Data model object matching the type of the Dataset this chunk belongs to

write(data)

Writes into the file and updates the hash, used in multithreaded rechunking in Task

DatasetFactory - Convenience class to create datasets

class hypergol.DatasetFactory(location, project, branch, chunkCount, repoData=None)

Convenience class to create lots of datasets at once. Used in pipelines where multiple datasets are created into the same location, project, branch

__init__(location, project, branch, chunkCount, repoData=None)

Parameters

• location (str) – path the project is in

• project (str) – project name

• branch (str) – branch name

• repoData (RepoData) – stores the commit information at the creation of the dataset

• chunkCount (int = {16 , 256, 4096}) – How many files the data will be stored in, sets the granularity of multithreaded processing

property projectDirectory

property branchDirectory

get(dataType, name, branch=None, chunkCount=None)

Creates a dataset with the parameters given and the factory’s own parameters

Parameters

• dataType (BaseData) – Type of the dataset

• branch (str=None) – Name of the branch to load the dataset from (if None, defaults to current)

• name (str) – Name of the dataset (recommended to be in snakecase)

• chunkCount (int=None) – Number of chunks, if None, the factory’s own value will be used

Pipelining Related Classes

Task - Class for create datasets from other datasets

class hypergol.Task(outputDataset: hypergol.dataset.Dataset, inputDatasets: List[hypergol.dataset.Dataset] = None, loadedInputDatasets: List[hypergol.dataset.Dataset] = None, logger=None, threads=None, logAtEachN=0, debug=False, force=False)

Class to create other datasets, created domain objects in run() must be appended to the output with self.output.append(object) (any number of the same type)

__init__(outputDataset: hypergol.dataset.Dataset, inputDatasets: List[hypergol.dataset.Dataset] = None, loadedInputDatasets: List[hypergol.dataset.Dataset] = None, logger=None, threads=None, logAtEachN=0, debug=False, force=False)

Parameters

• outputDataset (Dataset) – Return values of .run() functions will be saved into the chunks of this dataset.

• inputDatasets (List[Dataset]) – Chunks of these will be loaded line by line and passed onto the .run() function in each thread.

• loadedInputDatasets (List[Dataset] = None) – Data from these will be available in each job as a list.

• logger (Logger) – Standard logger class for each of the jobs

• threads (int = None) – Number of threads this task should run parallel

• logAtEachN (int = 0) – Log progress at each of the value, default = 0 means no logs

• debug (bool = False) – If true errors during execution stop the pipeline, otherwise they just get logged.

• force (bool = False) – All input object’s hashes must match in a single run() call. Use force=True to override this.

check_if_output_exists()

_get_temporary_dataset(jobId)

Based on the input chunk creates a temporary dataset and opens all chunks for writing so that the various output classes can be appended to the right chunk

log(message)

Standard logging

log_exception(ex)

log_counter(final=False)

get_jobs()

Generates a list of Job to be processed

execute(job: hypergol.job.Job)

Organising the execution of the task, see Tutorial/Task for a detailed description of steps

Parameters

job (Job) – parameters of chunks to be opened

initialise()

After opening input chunks and loading loaded inputs, creates delayed classes, initialises the results to be returned in JobReports and calls the task’s custom init()

init()

User-defined initialisation in each thread. Load files or complex classes here (e.g. a spacy model)

_open_input_chunks(job)

Opens input chunks and loads loaded input chunks

source_iterator(parameters)

run(*args)

This is the main computation of the task

Parameters

args (List[object]) – list of single domain objects one from each inputDataset in the same order as in the task construction, after these a list of domain objects which is the entire list from the loadedInputDatasets list.

_close_input_chunks()

Closes input chunks

finish_job(jobReport)

User-defined finalisation in each thread. Close file handlers or release memory of non-python objects here if necessary

finalise(jobReports, threads)

After func:execute finished, all the temporary datasets are opened and copied into the output dataset in a multithreaded way.

Parameters

• jobReports (List[JobReport]) – Reports on the executed jobs

• threads – Number of concurrent threads to do the merging

finish_task(jobReports, threads)

User-defined finalisation at the end of the task.

Pipeline - Class for organising tasks and datasets together

class hypergol.Pipeline(tasks: List[hypergol.task.Task], logger: hypergol.logger.Logger = None)

A simple pipeline that enables multithreaded execution of tasks

__init__(tasks: List[hypergol.task.Task], logger: hypergol.logger.Logger = None)

Parameters

• tasks (List[Task]) – List of tasks to be executed in the given order

• logger (Logger) – logger class

log(message)

run(threads=1, onlyTasks=None)

Runs each task Opens a set of threads, creates the list of job and calls the task’s execute() function. Upon finishing it calls finalise to create the .chk file of the output dataset.

Parameters

• threads (int = 1) – Number of threads to run

• onlyTasks (list = None) – list of task numbers in the task list to run (for debugging purpose). Add –onlyTasks=0 or –onlyTasks=1,2 parameter to the shell script in the CLI

Job - Class for passing information on chunks to tasks

class hypergol.job.Job(id_, total, parameters: Dict = None, inputChunks: List[hypergol.datachunk.DataChunk] = None, loadedInputChunks: List[hypergol.datachunk.DataChunk] = None)

Class for passing information on chunks to tasks

__init__(id_, total, parameters: Dict = None, inputChunks: List[hypergol.datachunk.DataChunk] = None, loadedInputChunks: List[hypergol.datachunk.DataChunk] = None)

Parameters

• id (int) – what’s the order of this job in the queue

• number (int) – number of total jobs in this task

• parameters (object) – any information to be passed to the source_iterator()

• inputChunks (List[DataChunk]) – these chunks will be iterated over while run() function is called

• loadedInputChunks (List[DataChunk]) – these chunks will be fully loaded before any run() function called

JobReport - Class for passing results of a task back to the pipeline

class hypergol.job_report.JobReport(jobId, exceptions, results=None)

Class for passing information from the tasks to the pipeline

__init__(jobId, exceptions, results=None)

Parameters

• jobId (int) – Id of the job executed

• exceptions (bool) – Was there any exception during execute()

• results (dict) – Any information to be passed to the finish() function

Delayed - Helper class to allow passing any class to threads

class hypergol.Delayed(classType, *args, **kwargs)

Enables delayed creation of classes so they can be declared in the main script and passed to a thread pickled.

__init__(classType, *args, **kwargs)

Parameters

• classType (Class) – type of the class to be created

• kwargs (args,) – parameters of the class’s constructor. If they cannot be pickled, they should be turned into Delayed as well.

static _make(v)

Enables recursive creation of the delayed classes

make()

In a Task.initialise(), each Delayed’d classes make() function is called that creates the class here:

Model Creation Related Classes

BaseBatchProcessor - Class for handling input/output for a model

class hypergol.BaseBatchProcessor(inputDataset, inputBatchSize, outputDataset)

Base class for batch processors.

Converts Hypergol datasets to batches for training models and converts TensorFlow model outputs to data model classes and then saves them.

__init__(inputDataset, inputBatchSize, outputDataset)

Parameters

• inputDataset (Dataset) – Hypergol dataset to retrieve batches from

• inputBatchSize (int) – number of samples to retrieve

• outputDataset (Dataset) – Hypergol dataset to save batches of model-processed samples

start()

TensorflowModelManager calls this to open the output for writing

read_batch()

Reads in Hypergol dataset as a generator

process_input_batch(batch)

Process data model class into input tensors in inference

Must return a dictionary of tensors, where keys matching the last arguments of functions get_loss(), produce_metrics() and get_outputs() in the implemented model.

Parameters

batch (List[BaseData]) – Values to be converted into tensors

process_training_batch(batch)

Returns Tensorflow compatible training data

Must return (inputs, targets) tuple, where inputs is a dictionary of tensors, where keys matching the last arguments of functions get_loss(), produce_metrics() and get_outputs() in the implemented model.

Parameters

batch (List[BaseData]) – Values to be converted into tensors

save_batch(inputs, targets, outputs)

Saves batch of model inputs + targets + outputs into Hypergol dataset

process_output_batch(outputs)

Process the TensorFlow model’s output into data model classes

Parameters

batch (List[BaseData]) – Values to be converted into tensors

process_evaluation_batch(inputs, targets, outputs)

Processing code for saving batches of model inputs + targets + outputs into Hypergol dataset Must return an instance compatible with self.outputDataset.

Parameters

• inputs – Created by process_input_batch()

• targets – Created by process_input_batch()

• outputs – Model response to inputs

finish()

TensorflowModelManager calls this to close the output after writing

This function runs even if the training was stopped by Ctrl-C. Otherwise, the Dataset would remain in an undefined state (with no .chk file).

Classes for organising Tensorflow code

class hypergol.BaseTensorflowModelBlock(*args, **kwargs)

Subclasses TensorFlow-Keras layers to provide logical groupings of functionality/layers.

Neither call() nor build() should be used as those won’t be called eventually by the model

There are no delayed initialisation in Hypergol models.

__init__(*args, blockName=None, **kwargs)

If further keras layers are required, they should be created here

get_config()

Function to get the configuration parameters for serialisation

build(input_shape)

This should not be called

call(*args, **kwargs)

This should not be called

class hypergol.BaseTensorflowModel(*args, **kwargs)

Main Tensorflow model class, responsible for implementing the interface of the model and the connection between the model blocks

It is recommended to avoid overwriting keras’s call() and build() functions because of their side effects. Model blocks should be initialised at construction, and if additional tensors are needed in this model, they should be initialised in the __init__() function as well.

Inputs for the three (get_loss(), produce_metrics() and get_outputs()) implemented functions must match and match with the return value of the model’s batchprocessor’s process_training_batch() function. The output of get_outputs()

__init__(modelName=None, longName=None, inputDatasetChkFileChecksum=None, **kwargs)

Parameters

• block1 (BaseTensorFlowBlock) – pass in block instances rather than construct them here

• modelName (string) – informal name of the model, defaults to the class name

• longName (string) – name of the model that uniquely identifies this current instance. It is set to <class name>_<training date>_<commit hash> by the generated training script

• inputDatasetChkFileChecksum (string) – checksum of the .chk file of the input dataset used in the training. This enables datalineage.

get_long_name()

get_input_dataset_chk_file_checksum()

get_model_blocks()

Returns blocks that are both member variables and arguments of the constructor for serialisation

call(inputs, training=None, mask=None)

This function is obsolete

get_loss(targets, training, **kwargs)

Must return a single value as the loss between targets and the inputs passed in through **kwargs

Parameters

• targets – Expected output values

• training (bool) – True if the loss is calculated during training, False otherwise

• **kwargs – Input values as keyword arguments

produce_metrics(targets, training, globalStep, **kwargs)

This function records values for TensorBoard, it runs in the context of a tf.summary.SummaryWriter so the implementation just need to call tf.summary.scalar or any other summary functions to record data to TensorBoard. See https://www.tensorflow.org/api_docs/python/tf/summary for further documentation on this.

It should not return anything as it will be discarded.

Parameters

• targets – Expected output values

• training (bool) – True if the loss is calculated during training, False otherwise

• globalStep (int) – The training/evaluation global step to be used as step in the summary functions

• **kwargs – Input values as keyword arguments

get_outputs(**kwargs)

This function returns the models response for an input. When the model is deployed, this is the interface to the calculations.

The get_outputs() function must specify the signature with the @tf.function decorator because this enables the model to be packaged and to be deployed as a standalone model. See the Tutorial for an example.

Must return a single value that will be converted to the output data model class by the model’s batch processor.

Parameters

**kwargs – Input values as keyword arguments

get_evaluation_outputs(**kwargs)

If the evaluation is different from the training, that can be implemented here. During evaluation TensorflowModelManager calls this actually and in turn this calls self.get_outputs.

class hypergol.TensorflowModelManager(model, optimizer, batchProcessor, project, restoreWeightsPath=None)

Class for managing TensorFlow model training.

__init__(model, optimizer, batchProcessor, project, restoreWeightsPath=None)

Parameters

• model (BaseModel) – model subclassed from BaseModel that is to be trained

• optimizer (TensorFlow optimizer) – optimizer from TensorFlow package to use for training

• batchProcessor (Dataset) – Hypergol dataset to use for training

• project (HypergolProject) – Hypergol project to handle directories

• restoreWeightsPath (path) – path to restore variables from a previously trained model

save_model()

Saves TensorFlow model, block definitions, and weights

restore_model_weights()

Restores TensorFlow model weights

train(withTracing)

Runs a single training step for the model

Parameters

withTracing (bool) – log TensorFlow graph metadata for the step

evaluate(withTracing)

Runs a single evaluation step for the model

Parameters

withTracing (bool) – log TensorFlow graph metadata for step

start()

Prepares to run the training cycle by creating the model data directories, create the SummaryWriters for Tensorboard for training and evaluation, initialises the batchprocessor (opens the output dataset for writing) and reloads the weights if restoreWeightsPath is specified.

run(stepCount, evaluationSteps, tracingSteps)

Runs a training schedule

Model is saved in every evaluation step and at the very last step.

Parameters

• stepCount (int) – num total steps in the schedule

• evaluationSteps (List[int]) – which steps to produce metrics on an evaluation sample

• tracingSteps (List[int]) – which steps to log graph metadata (components, memory consumption, etc.) to Tensorboard

finish()

Finishes training run by closing the output dataset.

Runs even if the training was interrupted by an exception (e.g. Ctrl-C)

Classes for organising Torch code

Miscellaneous Classes

Repr - Convenience class for default printing

class hypergol.Repr

Convencience class to automatically add standard __repr__() and __str__() functions to class.

Uses __dict__ property.

NameString - Convert string between cases

class hypergol.name_string.NameString(name, plural=None)

Class to generate strings of various cases

NameString stores the input string in components. The components are detected by splitting up the input name by single capital letters and underscores.

An irregular plural version can be supplied; otherwise an ‘es’ or ‘s’ will be attached to the end to get plural form.

__init__(name, plural=None)

Parameters

• name (string) – String in any case

• plural (string (default=None)) – If the plural form of a word is irregular, it can be provided here the same style as the name

property asFileName

Returns the standard python filename

e.g.: HelloWorld -> hello_world.py

property asSnake

Returns the string as snakecase

e.g.: HelloWorld -> hello_world

property asClass

Returns the string as a class name (PascalCase)

e.g.: HelloWorld -> HelloWorld

property asVariable

Returns the string as a variable name (camelCase)

e.g.: HelloWorld -> helloWorld

property asPluralVariable

Returns the string as a plural variable name

e.g.: HelloWorld -> helloWorlds

Used for autogenerating dataset variable names

property asPluralSnake

Returns the string as a plural snakecase name

e.g.: HelloWorld -> hello_worlds

Used for autogenerating dataset file names

HypergolProject - Class to manage information about the project

class hypergol.hypergol_project.HypergolProject(projectDirectory=None, dataDirectory='.', chunkCount=16, dryrun=None, force=None, repoManager=None)

Owner of all information about the project

CLI functions define what needs to be created, and this class creates them. It also consistently handles the mode flags (normal/dryrun/force)

It also verifies if a requested class exists in the respective directory (data_models, tasks) and identifies its type, e.g.: for HelloWorld it checks if data_models/hello_world.py or tasks/hello_world.py exists and assumes its role from that. Used in create_data_model() and create_pipeline()

__init__(projectDirectory=None, dataDirectory='.', chunkCount=16, dryrun=None, force=None, repoManager=None)

Parameters

• projectDirectory (string) – location of the project: e.g.: ~/repo_name, models will be in ~/repo_name/models

• projectDirectory – location of the data for the project project: e.g.: ~/data, files will be stored in ~/data/repo_name

• dryrun (bool (default=None)) – If set to True it returns the generated code as a string

• force (bool (default=None)) – If set to True it overwrites the target file

property isDryRun

property modeMessage

cli_final_message(creationType, name, content)

create_model_directory(modelName)

create_project_directory()

create_data_models_directory()

create_tasks_directory()

create_pipelines_directory()

create_blocks_directory()

create_models_directory()

create_tests_directory()

is_data_model_class(value: hypergol.name_string.NameString)

Checks if a name is a data_model class (based on if the snakecase .py file exists)

is_task_class(value: hypergol.name_string.NameString)

Checks if a name is in tasks class (based on if the snakecase .py file exists)

is_model_block_class(value: hypergol.name_string.NameString)

Checks if a name is in blocks class (based on if the snakecase .py file exists)

check_dependencies(dependencies)

Raises an error if any dependency is unknown

create_text_file(filePath, content)

render(templateName, templateData, filePath)

Creates a file from a template using jinja2

Parameters

• templateName (string) – filename of the template

• templateData (dict) – data to fill the template with

• filePath (Path) – full path of the destination file (ignored if self.mode != Mode.DRY_RUN)

make_file_executable(filePath)

render_executable(templateName, templateData, filePath)

render_simple(templateName, filePath)

list_datasets(pattern=None, asCode=False)

Convenience function to list datasets for a project

Returns a list of data loaded from the .def files in the directory

Parameters

• pattern (string (None)) – Regex pattern to filter on dataset names, if unspecified, defaults to .*

• asCode (bool (False)) – If True prints a code snippet that allows the dataset to be loaded (with imports and path updates)

diff_data_model(commit, *args)

Convenience function to compare old data model class definitions to the current one

Prints the diffs from the specified commit to the current commit

Parameters

• commit (string) – The git commit from where the comparison starts

• *args (List[string]) – List of class names to compare, if empty it compares all

create_old_data_model(commit, *args)

Convenience function to generate data model classes at an old commit to be able to load datasets created then

Full commit hash required.

project.create_old_data_model(commit='fbd8110b7194425e2323f68ef54dac15bb01ee7b', 'OneClass', 'TwoClass')

Will create data_models/one_class_fbd8110.py and data_models/two_class_fbd8110.py and replaces all occurences of OneClass and TwoClass to OneClassFBD8110 and TwoClassFBD8110 in each file.

Parameters

• commit (string) – git commit to retrieve classes from

• args (List[string]) – List of class names to generate, if empty it generates all

RepoManager - Helper class for storing Repository information

class hypergol.hypergol_project.RepoManager(repoDirectory=None, raiseIfDirty=True)

Wrapper class around git that provides all information about the repo connected to the project.

__init__(repoDirectory=None, raiseIfDirty=True)

Parameters

• repoDirectory (string) – directory where the the .git directory is located

• raiseIfDirty (bool) – if set and the repo contains uncommitted code, it raises an error

RepoData - Data class for storing Repository informations

class hypergol.dataset.RepoData(branchName, commitHash, commitMessage, comitterName, comitterEmail)

Stores the information about the repository in the dataset

__init__(branchName, commitHash, commitMessage, comitterName, comitterEmail)

Parameters

• branchName (str) –

• commitHash (str) –

• commitMessage (str) –

• comitterName (str) –

• comitterEmail (str) –

classmethod get_dummy()

Creates an empty RepoData if the Dataset was created outside a git repository

Logger - Class for logging

class hypergol.logger.Logger(path=None, level=20, overWrite=False, enabled=True)

Helper class to manage file-based and screen-based logging

CRITICAL = 50

ERROR = 40

WARNING = 30

INFO = 20

DEBUG = 10

NOTSET = 0

__init__(path=None, level=20, overWrite=False, enabled=True)

Parameters

• path (str (default=None)) – Optional path to save logs into a file

• level (int (default=Logger.INFO)) – Log level of this logger

• overWrite (bool (default=False)) – Indicate if an existing log file should be overwritten or appended

disable()

enable()

log(message)

Convenience method for Logger.INFO level

critical(message)

error(message)

exception(message)

warning(message)

info(message)

debug(message)