Recognizing Excellence - Our Philosophy
Why did that album win? How are the results determined?
Our awards programs have been around for over 20 years, under multiple program directors and with input from countless volunteers. The awards team has decided to be more transparent about the process both in the hopes of giving the community a glimpse behind the scenes and to help nominators and judges understand the process ahead of diving in to the awards season.
Interested in getting involved? Let us know!!!
The Contemporary A Cappella Society, founded by Deke Sharon in 1991, began the Contemporary A Cappella Recording Awards (CARAs) in 1992. Since then, the process of submissions, nominations, and judging has evolved.
We believe in transparency, and have chosen to share with you, the community, what goes on behind the scenes and how the nominees and final results are determined. We’ve described the process below in detail and included answers to some common questions.
The A Cappella Video Awards (AVAs) had its first inception as just a few CARA categories in 2016 and then graduated to its own program in 2017. It follows the same process outlined below.
The A Cappella Community Awards (ACAs), to be relaunched within the year, however, uses community-driven category suggestions, nominations AND voting, and therefore follows a much simpler, direct process not described here. This article will focus on the judging process for CARAs and AVAs.
It is a stated goal of our organization to "recognize excellence in the a cappella community".
We ask our nominators to nominate tracks or albums in categories where they believe it is worthy of the award. We believe that receiving a nomination, in and of itself, is an honor that anyone should be proud of.
While we give awards with titles such as “Best Original Song”, the word “Best” is a bit of a misnomer. In reality, the nominees for a category, as a group, represent the “best”. Once submissions for each category have been whittled down to the short list of nominees, the eligibility, relevance of category, and standard of quality have already been established. Judges have a much harder job, and that is to rank all of the outstanding nominees against one another.
“Forced Rank” vs “First Past the Post” Voting
For our judging round, we look first and foremost for clear consensus among the judges. The spread of votes is an input to the decision. The main goal of the forced-rank system is to find consensus and optimize for a result the most people voting would find acceptable.
What is the difference between consensus and just going off of most 1st place votes?
Because we’re in the world of art, quality is not easily defined, and unless every nominee used the same arrangement, or production company, or voices, it is impossible to compare apples to apples. Because of that, sometimes one subset of judges will really like a submission and place it first, and another subset will not and place it last. Take the below scenario:
While two judges placed submission A first, an overwhelming majority of judges placed it fourth place or lower. Meanwhile, all eight judges placed submission B third or higher. In this case, while submission A has more first place votes, submission B shows more of a consensus among the judges. It is important to take these nuances into account when reviewing the judging data.
How do you deal with bias or conflict of interest?
Nominators and judges are asked to disclose all groups that they have been in or worked closely* with. They will not be assigned to a category if a submission or nominee represents a conflict of interest.
For our review committee, we acknowledge that bias is a part of being human, and while we put measures in place to ensure we are as objective as possible, it is impossible to weed out all bias. Therefore our strategy is to balance it, by making sure that any one decision is not made by any one person alone, and making sure we have a group made up of people with various backgrounds and connections.
*Note: ‘Closely’ is difficult to define, but we use our best judgment to determine an appropriate cut-off when assigning nominators and judges to categories.
While the exact dates will differ, both the CARAs and AVAs follow a similar process:
- Open submissions
- Open Nominator / Judge Interest Form
- Select Nominators and Judges based on their experience and past year reliability
- Draft Nominator category assignments and check for conflict of interest
- Review submissions for eligibility (on an ongoing basis)
- Close submissions
- Open nominations
- Close nominations
- Review nomination data
- Finalize / publicize list of nominees
- Open judging
- Close judging
- Review judging data, finalizing categories where consensus is clear
- Obtain tiebreaker votes
- Finalize / publicize results
We use a custom web application specifically built for CASA that allows for submissions, nominations, and judging all within the same application.
Submissions are accepted as soon as the submission form is ready, until the submission deadline. Our online system allows the artist to upload and provide genre, group type, and arranger information for their submissions, which is used to determine eligibility for award categories.
Nominator / Judge Selection
Nominators / judges must express interest, indicating their time availability and experience in a cappella so that we can match them up to appropriate categories by volume of submissions and genre and/or category type. All volunteers are notified in advance of the beginning of nominations and asked to confirm their availability.
How can I become a nominator / judge?
Submissions must be confirmed eligible before they will be presented to nominators. As submissions are received, we do our best to ensure the following before confirming a submission:
- The track(s) have been uploaded properly and play without difficulty
- The album or single was released within the appropriate calendar year (each award ceremony recognizes recorded music from the previous calendar year)
- The genres selected during the submission process are appropriate
- The group type selected is accurate
- The arranger type (per track) selected is accurate, to the best of our knowledge
Nominations open as soon as the nominator category assignments are finalized and enough submissions are confirmed for the nominators to get started. We do our best to give nominators about 4-6 weeks to get through all of the material, depending on volume.
Our nominators are an invaluable part of going through a very large volume of material and helping us:
- Find any eligibility issues we may have missed
- Find the tracks and/or albums that truly stand out amongst the pack
Our nominators are not presented with the group name, album title or any other identifying information. While they may recognize something upon listening, the main goal of blind nominating is to make sure that all submissions are equally likely to be listened to completely and without any specific expectations.
Nominators are encouraged to nominate tracks or albums based on whether they believe it could, in the end, win the award.
Why is an album that I would consider R&B nominated in the Pop category?
The genre selection for both albums and tracks are part of the submission form, and, with few exceptions**, are selected by the artist themselves.
**While we may at times add an album or track to a particular genre where we felt it was missing, it is very rare to remove it from one. One example of a case where the team may remove a genre from a submission: a group listed all of their tracks as barbershop with the misunderstanding that ‘barbershop’ simply meant ‘a cappella’. Not a single track on the album was actually ‘barbershop’, so the album and tracks were removed from the ‘barbershop’ category but remained in all other genre categories listed at the time of submission.
I selected a genre for my album when I submitted, but my friend said they didn’t see the album in that genre’s category, and instead saw only individual tracks. Why would that be?
Our eligibility guidelines state that for an album to be eligible for a genre award, 50% plus one of the tracks must be of that genre. For instance, in order to be considered in the jazz category, an album with 6 tracks must be both marked with the jazz genre by the person creating the submission AND have a minimum of 4 tracks marked as jazz.
Once nominations are closed, the review committee will review the nominations data and select the final list of nominees. This process takes about two weeks due to the need to seek tiebreaker votes for categories that are not clear.
How do you decide how many nominees there will be per category?
The review committee looks at the number of nominations from judges who completed listening to all submissions in a category. Then we look for a cut-off that, given only the top track/album per group, comes in at, ideally, between 4-8 nominees. Sometimes that number is smaller or larger based on what the data looks like. For instance, we’ve had scenarios where a cutoff of 4 or more votes from nominators would mean only 1 nominee, while a cutoff of 3 or more would mean 10 nominees.
What if my group has two tracks with the same number of nominations in a category?
The review committee will seek tiebreaker votes between the two tracks, or, in the case that tiebreaker votes do not actually break the tie, select the track that we believe has the best chance of winning the award.
But wait - I’ve seen the same group with two nominations in an arrangement category!
With our Professional Arrangement categories, where the arranger is not a member, faculty member, or friend of the group doing arrangements pro bono, we do consider the arranger to be the actual nominee rather than the performing group. So it is possible to see two arrangements in a Professional Arrangement category from Group A, but arranged by two different arrangers. However, we do only nominate one professional arrangement per arranger per category.
What if a nominator finishes their nominations after you have finalized the nominees, and it would have changed the results?
The review committee only considers the data available as of the nominating deadline, and then will seek tiebreaker votes from late and/or additional nominators where needed.
Once the nominees are finalized and judging assignments have been made/adjusted to account for any conflicts of interest, judges are given access to the system and will start ranking nominees in their assigned categories. We do our best to give our judges at least 4 weeks to complete their rankings.
How do you collect data for the judging round?
Judges navigate to each assigned category from their dashboard in our online system. They can listen to each nominee and drag and drop to reorder and rank them. Once they are happy with their rankings, they click a button to mark the category as finished and let us know their rankings are final. Similar to the nominating process, judges cannot see the artist name or album title of the nominees, although they may be familiar with the nominees already since the nominee list is published.
Is the judge ranking data all that determines the final result?
If at the end of the judging round there is a clear consensus among judges, then yes! It’s important to understand, however, what exactly we mean by a clear consensus. Please review the ‘Philosophy’ section above.
It is also important to understand that the results are finalized on a specific date, and if a judge submits rankings after the deadline, the data will have changed, but the original results will stand.
Once the judging deadline comes around, judging closes and the review committee reviews the rankings to determine the final results.
I’ve heard you have some sort of weighted score. How does that affect the results?
The weighted score is still in an experimental phase. It allows the review committee to look at the ranking data in different ways, and be alerted to possible scenarios where the vote spread should be taken into account. Over time, we will likely refine the weights assigned to different rankings and make more use of it, however at this time it is still somewhat arbitrary and unscientific.
How do you decide to break ties vs. award them?
For ties for runner-ups, we will typically just award the tie.
However, for first place, we prefer to award just one winner where possible. If there is a tie for first place at the time of the initial data review, we will attempt to break the tie with additional judges. If that still doesn’t provide clear consensus, then the review committee itself may vote. If there is still not a consensus, then we will award the tie, and there will be no runner-up.
How and from whom do you collect tiebreaker votes?
Tiebreaker votes are sought from:
- Judges that were assigned to the category but didn’t finish it before the deadline
- Judges that volunteered after the initial deadline and weren’t originally assigned
- Judges that finished all of their assigned categories and have expressed that they have the bandwidth to take on more
In the event tiebreaker votes are needed, having the tiebreaker judges rank the entire category is actually not usually helpful. Often we will ask the judges simply to vote between the top two. This means that sometimes the judge puts their votes in the online system, but more often than not they simply communicate their vote to the review committee directly (i.e. via email), and we add their numbers to the judge ranking report to determine which nominee comes out ahead.
If the tiebreaker votes do not provide clarity, however, a tie will be awarded, and there will be no runner up.
I’m surprised by the results. How could [enter group name here] not have won?
Judging art is inherently a subjective activity. While we take measures to be as objective as possible when reviewing the judging data, the data itself represents a subjective judgment.
It’s important to realize that every nominee for an award category has been deemed worthy of the award as a prerequisite (in fact, that’s how we ask the nominators to evaluate submissions). It follows that all nominees will be of extremely high quality, and therefore it can be very difficult for individual judges to rank them against each other. It is extremely rare that multiple judges will rank the same set of nominees in remotely the same way. Ranking Group A first and Group B last does not at all mean that the judge thinks Group B unworthy of the award. It simply means that they, subjectively, think that Group A represents the award best out of all the options.
At no point will we override a clear consensus from the judges, unless there is an eligibility issue caught after the nominations round.
The awards programs at CASA are an exciting and well-respected honor for all of our nominees and winners. Though it requires a great deal of time and effort from many volunteers, we consider it an immense privilege to recognize excellence in a cappella through our awards programs. We hope this article has provided a helpful explanation of our process. If you have a question we didn’t answer, please let us know by emailing firstname.lastname@example.org.
Still think the process could be better? Then we need you on our team! Apply for a position on our awards team and let us know how we can improve!