I have an interesting question which was posed to me by a friend. A board of peers is asked to rate employees based on performance. For the sake of this example, we will say the categories being scored include creativity, communication, and overall effectiveness. Up to ten points can be awarded in each category. The scores are averaged and the people with the highest score in each category receive an award and a small bonus. Due to the vagueness of the prompt, some judges think to award a total of ten point in each category across all candidates, while others think to award a certain amount of points out of ten for each candidate. For instance, using the first method on twenty candidates for a given category, one person could get three points (a very good score), one get two points, five get one point, and thirteen get no points. This brings the total of distributed points to the mandated ten. Using the second method, one person could get a score of 5/10, another 2/10, and so on for all twenty candidates. This means that a total of 200 points are up for grabs in each category given a set of 20 (max of ten per person). The question is: what method is more fair? I feel like their are multiple answers to this question. A mathematical approach seems the easiest. I thought their might be a way to apply game theory to this situation, but I am not sure. Any thoughts? I've reach my own conclusion which can be read by highlighting below as to avoid biasing other answers.
I thought about this problem in terms of accuracy and precision. Accuracy being how close a score is to what a judge feels an employee deserves. Precision being a measure of how exact a score is. Assuming that judges always want to achieve 100% accuracy(give the candidate the score they deserve), it would follow that increasing the precision rating system would lead to higher accuracy and a more fair assessment. The second method is more precise, as there are more points available. Even a very dedicated judge will most likely not go beyond awarding quarter points using the first method. Therefore, the second method is more fair.