Module `dlkoopman.hyp_search`

Ready-to-use hyperparameter search module.

Functions

def run_hyp_search(dh, hyp_options, numruns=None, avg_ignore_initial_epochs=0, sort_key='avg_pred_anae_va') ‑> pathlib.Path

Perform many runs of a type of predictor - either StatePred or TrajPred - with different hyperparameter combinations (which are the argument values passed to the class and its methods) on given data, and save the metrics for each run.

Use the results to pick good hyperparameter combinations.

The method can be interrupted at any time and the intermediate results will be saved.

Saved statistics

For each of the following <x>:

'recon_loss', 'recon_anae' - Reconstruction loss and ANAE.
'lin_loss', 'lin_anae' - Linearity loss and ANAE.
'pred_loss', 'pred_anae' - Predicton loss and ANAE.
'total_loss' - Total loss.

save the following statistics:

If only training data is provided:

avg_<x>_tr - Average training <x> across all epochs.
final_<x>_tr - Final training <x> after last epoch.

If both training and validation data is provided (recommended), save the above, and additionally:

avg_<x>_va - Average validation <x> across all epochs.
best_<x>_va - Best validation <x> across all epochs.
bestep_<x>_va - Epoch number at which best validation <x> was achieved.

For more details on losses and ANAEs, refer to metrics.

Parameters

dh (StatePredDataHandler, or TrajPredDataHandler) - Data handler providing data. Model to be run is inferred from data, i.e. either StatePred or TrajPred.
hyp_options (dict[str,list]) - Input arguments to model and its methods will be swept over these values across runs. As an example, when dh is a StatePredDataHandler:

hyp_options = {
    ## arguments to __init__()
    'rank': [6,8], # must be provided since it's a required argument
    'encoded_size': 50, # must be provided since it's a required argument; this input stays constant across runs
    'encoder_hidden_layers': [[100,100],[200,100,50]]

    ## arguments to train_net()
    'numepochs': list(range(100,501,100)), # ranges must be provided as lists
    'clip_grad_norm': 5 # this input stays constant across runs
}

# this results in 2*2*5=20 possible runs

numruns (int, optional) - The total number of possible model runs is $N =$ the number of elements in the Cartesian product of the values of hyp_options (in the above example, this is 20). If numruns is None or $>N$ , run $N$ runs, otherwise run numruns runs.
Since each run takes time, it is recommended to set numruns to a reasonably smaller value when $N$ is large.
avg_ignore_initial_epochs (int, optional) - When collecting average statistics across epochs, ignore this many initial epochs. This is useful because the loss / ANAE values can be quite high early on, which can skew the averages and make it meaningless to compare runs based on the averages.
sort_key (str, optional) - Results in the final CSV will be sorted in ascending order of this column. For possible options, see Saved statistics. Note that sorting only happens at the end, and thus will not take place if execution is interrupted. Set to None to skip sorting.

Returns

output_folder (Path) - The path to a newly created folder containing:

Results CSV file = hyp_search_results.csv
Log file = hyp_search_log.log
If any of the individual model runs resulted in errors, their log files will be stored as well so that you can take a closer look at what went wrong.

The results CSV contains:

-------------------------------------------------------------------------
| UUID | <hyperparameters swept over> | <loss results> | <anae results> |
-------------------------------------------------------------------------
| run1 |              ...             |       ...      |       ...      |
| run2 |              ...             |       ...      |       ...      |
| run3 |              ...             |       ...      |       ...      |
-------------------------------------------------------------------------