On Judging

For the annual digital contest in 2015, it was decided to move to a top ten contest. All previous winners would be entrants. Realizing the import of this contest—You had number one images competing, on an equal basis, with honors images. It was incumbent to have a contest that was above reproach. To this end it was decided to have three judges.
It was also decided to get diverse judges. One judge was from academy, one was a professional photographer, and one was an experienced judge from another camera club. The images were furnished to the judges without their knowing the previous award or the identity of the maker.
This was going to be interesting—how to combine the three judges' opinions into one. First, we knew if they only chose ten images we would have weird results. We had them pick their top twenty (Remember we were after the top ten). Next how to combine the scores?

This is the process that was selected.

We asked each judge to assign from 100 to 1 points to each image among their top twenty. They could weigh these any way they wanted. For example if no 1 and no 2 were virtually tied they could award 100 and 99 points. It they wanted to spread them out they could go from 100 to 5 as in 100 95 90 etc. We made no suggestions beyond the rules.

Now, How to combine the scores. We could not total the raw scores since that would give judges who skewed to the high side more say in the outcome. By definition each of the three judges had an equal vote. So we normalized the total points for each judge to 100. This was done by totaling their total points awarded and calculating the percentage of each. that became their vote.

You can see the results of this in the following table. These are real results from actual judges and real images.

What we learned. If you lose a contest, or, for that matter if you win a contest the victory is very, very dependent upon the judge. More so than you ever though. There are many examples of Number One winners in quarterly contest who did not place in the top twenty,  Also there were honors winners who made it into the top ten. Yes themes did make some difference. But?

N. B.: Only three images were placed in the top twenty by all three judges. Images not included were not in any judge's top twenty.
N. B.: There is a sort feature in which you can see each judges winning order. Click the sort arrow on the total points for each judge or sort on total points to see the overall winners.

[supsystic-tables id='13']