Supervised learning is a well studied problem. However, in practice in can still be challenging to learn a rule that generalizes well or, in other words, it is advisable to be wary of jumping to conclusions especially in settings where data is scarce, which can have an even more severe impact for high-dimensional problems. In such cases one would often employ a method for dimensionality reduction, such as feature subset selection, which has the added value of straigth-forward interpretability of the outcome in terms of the set of most predictive input variables. Yet, here we face the same dilemma as previously with the actual learning algorithm and it would be advantageous to have a simple method that could increase the confidence in the selection. In this light, consistently selecting the same features over different data sets, i.e. a stable solution, is a very natural and intuitive indicator for that purpose. Hence, originally motivated by a high-dimenstional classification problem in the field of cancer research, we study the problem of stability of feature subset selection in the context of supervised binary classification, i.e. the behaviour of a feature subset selection algorithm under perturbations of the input. Therefore, we assume a probabilistic perspective to propose a generic framework to analyse stability on top of existing algorithms.