Trade-offs between formative and summative evaluation

From RecSysWiki
Jump to navigation Jump to search

Formative evaluation is quicker and cheaper to conduct than summative evaluation, and the results are more straightforward. However, the method is less suitable for adaptive systems (including recommender systems), because it is hard to find out what exactly causes (problems with) the usability or user experience. In summative evaluation you need to define hypotheses beforehand, you can only focus on a few aspects at a time, and the analysis is more complex. You also need more test users to ascertain adequate statistical power. On the other hand, it is easier to test adaptive systems because you can single out the effect of specific features. Moreover, the results of summative evaluation are more generalizable and can be statistically validated.