Hyperparameter tuning

Machine learning algorithms usually have (a lot of) hyperparameters. You need to tune these, for which there is a recipe, but which is also partially an art.

Intended for: BSc, MSc, PhD

1. Definition of hyperparameter

2. Always tune your hyperparameters

Hyperparameters are absolute crucial for the performance of a learning algorithm. If they are off, your algorithms will not work, no matter how smart it is. Therefore: always tune (at least the most important) hyperparameters. 

3. Tune the most important parameters first

Alright, you know you need to tune hyperparameters, but where to start. You typically have a lot of hyperparameters, and you do not have the budget to vary all of them. Therefore, always tune the most important parameters first. This includes: 

4. Perform a grid search (or more advanced method)

Once you determined which hyperparameters you will tune, you  have to decide on an actual approach. 

5. Vary hyperparameters in orders of magnitude. 

When you try different values for hyperparameters, you usually want to vary them in orders of magnitude (this applies to many hyperparameters, such as the learning rate). 

6. Get intuition for reasonable values (hyperparameter optimization is more art than science)

This is probably the most difficult alspect of hyperparameter optimization. You will simply not have the budget to optimize over all hyperparameters. Therefore, you will have to fix many parameters to 'reasonable' values, where you determine reasonable. 

7. Other resources

Hyperparameter optimization is such a crucial aspect of machine learning, but we actually talk and write very little about it.