Online since 2002. Over 3300 puzzles, 2600 worldwide members, and 270,000 messages. Forum

It is currently Wed Jul 23, 2014 8:10 am

All times are UTC - 5 hours

Post new topic Reply to topic  [ 1 post ] 
Author Message
 Post subject: Question about rating systems
PostPosted: Tue Jun 10, 2014 12:36 am 
User avatar

Joined: Sat Apr 10, 2010 8:07 pm
I have an interesting question which was posed to me by a friend. A board of peers is asked to rate employees based on performance. For the sake of this example, we will say the categories being scored include creativity, communication, and overall effectiveness. Up to ten points can be awarded in each category. The scores are averaged and the people with the highest score in each category receive an award and a small bonus. Due to the vagueness of the prompt, some judges think to award a total of ten point in each category across all candidates, while others think to award a certain amount of points out of ten for each candidate. For instance, using the first method on twenty candidates for a given category, one person could get three points (a very good score), one get two points, five get one point, and thirteen get no points. This brings the total of distributed points to the mandated ten. Using the second method, one person could get a score of 5/10, another 2/10, and so on for all twenty candidates. This means that a total of 200 points are up for grabs in each category given a set of 20 (max of ten per person). The question is: what method is more fair? I feel like their are multiple answers to this question. A mathematical approach seems the easiest. I thought their might be a way to apply game theory to this situation, but I am not sure. Any thoughts? I've reach my own conclusion which can be read by highlighting below as to avoid biasing other answers.

I thought about this problem in terms of accuracy and precision. Accuracy being how close a score is to what a judge feels an employee deserves. Precision being a measure of how exact a score is. Assuming that judges always want to achieve 100% accuracy(give the candidate the score they deserve), it would follow that increasing the precision rating system would lead to higher accuracy and a more fair assessment. The second method is more precise, as there are more points available. Even a very dedicated judge will most likely not go beyond awarding quarter points using the first method. Therefore, the second method is more fair.

Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 1 post ] 

All times are UTC - 5 hours

Who is online

Users browsing this forum: No registered users and 6 guests

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  

Forum powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group