An hyperparameter is a parameter of the machine-learning algorithm. While the parameters are learned by the machine-learning algorithm, an hyperparameter dictates how the algorithm learns.
For instance, recall the training objective for a ridge regression given a train-set :
The variable regulates the weight of the regularizer term in the training objective. Changing will change the learned parameter . It is an hyperparameter.
Other hyperparameters often used are the depth and width of a neural network, of the maximal degree of a polynomial feature augmentation.
Choosing the best hyperparameters
The process of choosing the best hyperparameters is called model selection.