What is an hyperparameter?

An hyperparameter is a parameter of the machine-learning algorithm. While the parameters are learned by the machine-learning algorithm, an hyperparameter dictates how the algorithm learns.

For instance, recall the training objective for a ridge regression given a train-set $\trainset$:

$$\g(\linmodel{\linparamv}, \trainset) = \lmse(\linmodel{\linparamv}, \trainset) + \hyperparam\normtwo{\linparamv}^{2}$$

The variable $\hyperparam$ regulates the weight of the regularizer term in the training objective. Changing $\hyperparam$ will change the learned parameter $\linparamv$. It is an hyperparameter.

Other hyperparameters often used are the depth and width of a neural network, of the maximal degree of a polynomial feature augmentation.

Choosing the best hyperparameters

The process of choosing the best hyperparameters is called model selection.