Beer Competition Registration

Reggie+eJudge+vJudge V9.41
©2005-2024 by Nelson Crowle (Nelson@ReggieBeer.com)

About Virtual Judging

What is Virtual Judging?

Virtual Judging is a way of running some or all flights in a competition where the judges evaluate entries over a video-chat (like Zoom) connection instead of F2F (Face to Face). This allows for 'No Contact' judging.

Use Reggie eJudging - all scoresheet recording and recordkeeping is done on each judges cell phone/tablet/laptop. Reggie eJudging works on all operating systems (Windows, Linux, MacOS, iPhone, Android, Chromebook) using the recommended current Chrome browser (or a modern HTML5 browser like Internet Explorer, Edge, Firefox, etc.)

Advantages

No need to set up a judging venue: judges arrange their own times to get flights evaluated. That means that there are no issues with paying for a venue or staff, no need to store and keep bottles cold for all judges, no need to provide food or water for the judges.
No paperwork for 1st Round or MiniBOS judging, if you're using eJudging and eScoresheets.
Much less concern about no-show judges at scheduled F2F sessions - judges can arrange dates and times with their judge partners without affecting any of the other judges.
Judges can commit much smaller time increments (down to judging just one entry) rather than committing to a full day. This gives the judges much more flexibility in scheduling, optionally spreading out a single flight at convenient times. It's more likely that you can recruit judges without requiring specific judging dates.
If you are using eScoresheets and eJudging (highly recommended), the scoresheets returned to the entrants are completely readable - no need to decipher chicken scratch, no worries about how light or dark the written words are.
All judging can be done from home, so no concerns about drinking and driving. This also allows judges to finish off (after judging is complete) any entries that were particularly good.
With eScoresheets, printable PDF files can be viewed/printed/saved by both the entrant and the judge. Scoresheets are normally available to entrants only after the competition has completed and awards have been announced.

Disadvantages

Much less social interaction that normally occurs at a competition. You get to meet with each judge partner for judging a particular flight, but no random roaming around and chatting with other participants.
As Competition Organizer, you need to provide an extra bottle to the judges. Typically, you'll need two bottles of each entry - one for each of two Virtual judges (since they will likely be each judging from their own homes and can not share a bottle). You'll also need a bottle for Best Of Show and/or Pro/Am judging. So if you've previously asked for 2 bottles, you'll need to ask for 3 instead.
Although larger medal groups (like 40 IPAs) can be handled with Virtual Judging, the medal determination is not as good as if the entries were judged F2F. Unless you ask for more than 3 bottles, you will not have samples available for the 2nd Round/MiniBOS judges. Reggie provides three solutions:
1. All entries are judged by the same judge pair. This can be done over several sessions (4 sessions of 10 entries each, for example). The advantage to this method is that since all of the entries are judged by only one judge pair, the bias between different judge pairs is eliminated, and the scores should be on the same scale. The amount of work done by this judge pair is equivalent to the amount of work done by 4 judge pairs, each judging a flight of 10 (in this example). After evaluating all entries, the judge pair can review all of their scores and their (well written!) scoresheets to determine medal awards, without the need to re-taste each sample.
2. Entries are split up between (for this example) 4 judge pairs. Each judge pair pushes 2-3 entries to MiniBOS (as in a non-Virtual session). The Judge Captain (most senior judge-pair-lead) chooses the MiniBOS judges (as in a regular session), and these 2 or 3 MiniBOS judges review scores and scoresheets from all of the entries pushed to MiniBOS to award medals (again, this is done without the ability to taste the samples). One disadvantage to this method is that judge pairs are likely not calibrated to each other, so one judge pair may (for example) have their scores running higher or lower than other judge pairs. This places a burden on the MiniBOS judges to be able to evaluate entries from (well written!) scoresheet comments, and not just by score.
3. The Comp Organizer may optionally limit entries either per Style (such as 21:IPA) or per SubStyle (such as 21A:American IPA and 21B:Specialty IPA). For example, the limit may be set at 12, which means that no more than 12 IPAs could be registered for entry in the competition. This balances out each medal group so that no MiniBOS is needed, removing the need to choose between options 1 or 2 above; however, it may compromise the goals of the competition, particularly if it normally has a larger entry count.
Since each of two judges judging the same entry is judging from a different bottle, it's possible that the samples of the same entry in different bottles may be different (oxidation, contamination, etc.)

Considerations for Competition Organizer

Entry bottles can still be collected by shipping/delivery, often to one or more LHBS.
Set up Virtual Judging sessions, planning for each judge pair in each session to judge one Medal Group. So, if you would like to offer judges the opportunity to judge 2-3 sessions (as in a typical F2F competition for a single day), set up one Virtual Session for each. You can always add Virtual Judging sessions if you have judges who are willing to judge more. Some competitions may set up 8 to 10 F2F sessions including previous weekends, Friday night, all day Saturday, etc., so Virtual Sessions can be set up similarly (but without requiring a particular date or time). All judges can be lumped together in Virtual Session #1 since they won't conflict with each other, and then additional Virtual Sessions may have less and less judges signed up, depending on how many sessions each judge wants to judge. The judge assignments *MUST* be done before the Labeling/Sorting Party so that the Sorting can organize appropriate sample bottles for each judge.
The Labeling/Sorting Party still needs to happen, and must be planned with minimal staff and appropriate sanitizing/air quality/distancing considerations. IMPORTANT - for Virtual Sessions, the bottles should be organized (in cases, etc.) BY JUDGE, and not by medal group. Instead of having a case of Lagers and a case of Stouts for First Round, another case of each for MiniBOS, and a another case of each for Best Of Show, you might have a case for Judge John containing 8 Lagers, 8 Stouts, and 8 Porters. Judge Joe might have a case with 8 Lagers, 8 Pilsners, and 8 ESBs - and Judges Joe and John are judging Lagers together, but the Stouts, Porters, Pilsners, and ESBs are all matched up with other judges.
Once all judges are assigned to flights and the Sorting/Labeling is done, you need to get the entry bottles to the judges. For smaller competitions or smaller judge pools, it may be reasonable for the Comp Organizer (or assignee) to drive the cases of bottles around to judge houses, pop the trunk, let the judge come out and take their case, and drive off with no contact. Bottles may be shipped to select judges, although this may be a considerable expense. Bottles may also be reverse distributed to "hubs" (possibly a LHBS) - i.e., the entries for judges in one geographical area may be delivered to the hub, and then the judges can drive by for a no-contact pickup, just like no-contact purchases from the LHBS. Be sure to let the hub personnel know the amount of work involved (a judge calls, the hub finds their case(s) of entries and prepares it, the judge drives by, the hub delivers the case(s) to the car) - don't overburden the LHBS with distributing your entries to judges. You might also tell judges in one geographical area that you will have all of their cases available during a certain timeframe at a certain location (your local brewery taproom comes to mind - sit yourself down, have a beer, go out to your car each time a judge shows up, etc.). Distribution to judges is the largest deviation from running a regular F2F competition - this is a big job, but once it's done, most of your work is complete.
We recommend that the competition include either a branded tasting glass in each judge's case of bottles, or an unopened sleeve of good quality clear hard plastic disposable tasting cups. This way, judges will be able to use consistent glassware, eliminating one more variable from the evaluation experience.
After all judges have received their cases of bottles, the Competition Organizer can sit back and watch the results come in as judge pairs complete each flight and medal group. The big part of your job is over. If you're using eJudging and eScoresheets, you won't have any work to do as each flight completes - no need to enter judge names or entry scores from the BJCP Flight Summary sheets or Reggie Flight Info sheets. The ONLY thing you need to do at this point is to monitor the judging progress, and maybe encourage slower judges to complete their flights.
Best Of Show judging would typically be set up as a F2F (since you only have one bottle per entry) session with appropriate health safeguards.
When pairing judges, it's helpful to match delivery methods to minimize the differences in handling the bottles. For example, if you are shipping entries to one judge, it may be more consistent to also choose a partner judge where bottles will be shipped rather than delivered.
Use eJudging and eScoresheets whenever possible. Using written scoresheets requires that the judge has the ability and technology to scan and transmit the scoresheets to you and also requires that you do extra work in verifying and uploading the scoresheets to Reggie. Also, scanned scoresheets (or PDF form scoresheets) still require the production and transmission of the BJCP Flight Summary sheet (which lists the judge names and all the scores) and that requires the Competition Organizer to enter this information manually (as you would for a regular non-eJudged competition). This introduces several areas for potential human error (transcription, etc.) that are eliminated by using eJudging. In addition, an eScoresheet takes about 4K of space, compared to a scanned scoresheet at 200K-500K, and eScoresheets are readable!
If you use the PDF form from the BJCP website (recommend to use eScoresheets instead), be aware that the fonts on some fields are excessively large (so not enough space to see your whole input), the tab order is not intuitive, and the form will allow the judge to enter invalid values (like too large of a number). Also, depending on how the scoresheet is distributed, the entrant may be able to alter their own copy of their scoresheet (but not the copy on Reggie). Maybe update the PDF form to address these issues before using it.
If you are using the PDF Form scoresheet (again, recommending eScoresheets/eJudging instead), email the PDF Scoresheet Form file to all judging who will use it, and instruct the judges to save it as a "master" copy of the scoresheet, then for each entry judged, make a copy of the master to fill out and save for that entry. For each flight, instruct the judge to take the master copy and type in their name, email, judge id, rank, and category information and save as a sub-master for that flight. That way, the judge can copy the sub-master for each entry and have those things already filled in. Tell judges: after judging each entry, DO NOT do a Save; instead do a Print and print as PDF, giving the file the appropriate name (see File Naming Convention below). Do not check the Fill And Sign option. This removes the Form capability from the final scoresheet PDF that you will upload.
If you ask the judges to use paper scoresheets, which they must then scan and transmit to you, email the scoresheet that you want the judges to use - so that all judges are using the same version of the scoresheet. Please give them the recommended scanning parameters:
- The size of a single page of a scanned scoresheet should be somewhere around 200-400K
- Greyscale (NOT color - takes too much space, doesn't add value; also NOT black and white - the contrast is too high to be readable)
- Resolution - I've found that 200dpi works very well - going higher doesn't add much in quality but costs significantly more space
- Contrast - depending on the darkness of your writing, you might need to bump up the contrast slightly - do a test sheet and see if it's reasonably readable
- Output type - scan to PDF (not TIFF, JPG, PNG, or other image file)
For either PDF Form or scanned paper scoresheets, here is the file naming convention. You MUST follow this for Reggie to be able to match up the right entries with the right PDFs.
- 1st Round scoresheets are named with the Judged Entry Number (the number written on the scoresheet), e.g., 1234.pdf. The Judged Entry Number will typically be a 4-digit number for Randomized numbered entries, but may be more or less digits for other bottle numbering methods.
- For 1st Round scoresheets from Virtual Judging (where each judge scans their own, so you get two PDFs for each entry), follow the Judged Entry Number with -1 for the Lead Judge, -2 for the second judge in a judge pair, -3 if you have a third judge on that flight, e.g., 1234-1.pdf or 1234-2.pdf.

Virtual Judging - Asynchronous vs. Synchronous judging

Reggie provides the capability to handle both Synchronous and Asynchronous judging. Synchronous Judging is where a judge pair judges the same entry at the same time (just like F2F sitting across a table) then discusses that entry and assigns a consensus score before moving on to the next entry. Asynchronous Judging lets each judge of the judge pair judge all of the entries of that flight on their own time, without any discussion or feedback from the other judge on those entries. Only after both judges have judged all entries, does Asynchronous Judging allow the judges to meet (virtually, over a video chat) to review all entries and scores, and to assign consensus scores to all entries.

Synchronous judges have the advantage of instant feedback (if one judge is detecting a defect, they can discuss while the beer is still in front of them). This also allows for mentoring and training of newer judges while compromising on the scores to be within the usual 7 (or 5, depending on the competition) points before coming to consensus. Asynchronous judging requires the judges to discuss their scoresheets without the ability to retaste the entry or change their notes or scores (which may have been assigned days ago).

Recommendations: In general, Synchronous judging should be used whenever possible. Asynchronous judging introduces some issues where judges may not agree on something but have no way to retaste the entry to make scoresheet adjustments. Asynchronous judging should only be used if there are scheduling issues that require it.

Virtual Judging recommendations from users

Plan for a gusher - open and pour your samples away from the electronics. Have a dump bucket and towels available.
Before opening the bottle, always show your label to the camera so you and the other judge can verify each other's entry number. You may also want to show the poured glass to the other judge.
Any audio or video communication will work, but the video adds nuances that help you and the other judge work together better.
Keep in mind that judging two bottles of the same beer may have different sensory evaluations - be considerate of your judge partner if they find something that you don't, or they observe something different (clarity, gusher, floaters, etc.) - the difference may be because the bottles are different.
If you are not the Lead Judge (or Judge Captain), and your judge partner has moved on to the next entry, just click Update Status.
Do one or two Practice scoresheets to get familiar with the eScoresheet. Plan to have the first entry take up to 30 minutes, but once you have done a couple of eScoresheets, you should be able to get them done in 7 to 10 minutes each (just like the BJCP Structured Scoresheet), and complete each entry (with consensus) in under 15 minutes. Two experienced eScoresheet judges should be able to judge 4 to 6 entries per hour.
Technology recommendations: none in particular - all seemed to work out fine. Zoom is popular, and the 40-minute limitation is waived for conferences of only 2 people. Google Meet and Skype worked well. If you're unable to set up a video conference, you can still reasonably well judge over a voice-only phone call, but video is highly recommended.
Do NOT try to cap and retaste an entry after more than an hour or so. The entry may oxidize and will lose carbonation - so it will not be in the best condition to represent what the brewer originally sent. Definitely do not cap and retaste after several days - it's unfair to the entrant.
For a single computer screen (tablet or laptop, for example), resize the right side of the eJudging screen to compact it to show only enough space left-to-right for the scoresheet, then put your video chat screen in the remaining area. That way, you can do the eJudging while still maintaining the video chat.
When doing MiniBOS for flights judged by multiple judge pairs, here is a recommended strategy:
- Start by reviewing MiniBOS entries by score from highest to lowest
- Review comments and possibly give more weight when judges agreed on sensory evaluation
- Review scores that have a wider spread from the consensus - particularly looking to see if the high score was justified based on the comments
- Assign medals based on score and scoresheet comment information - NOT just on score alone
Write complete and informative scoresheets! Not only are you providing better feedback for the entrants, but you may need to refer to your notes when discussing entries with your judge partner, including for Asynchronous judging, and for MiniBOS judging.

Judging from home - and setting up your judging environment

No spicy/greasy/oily food before judging.
Judge away from the kitchen, laundry, or any other distracting aromas.
Use a consistent glass for all entries (possibly a branded glass provided by the competition), rinsing in between entries, or a sleeve of hard clear plastic taster glasses provided by the competition.
Store all entries in the fridge at 38-40F (3-4C). Pull first entry from fridge 15 minutes before judging, then pull each "next" entry as you start judging the current entry.
Set up the entry opening/tasting area away from your electronics - expect a gusher!
Judge away from visual or audible distractions - a quiet, well lit, peaceful location is preferred - tell family members that you are "working" to minimize disturbances.
If you judge outside, consider that you may lose aroma to wind, and you may alter the beer by sunlight (e.g., skunking).
Whether paper or eScoresheets, write extensive notes about each entry, particularly in the Overall Impression section - this not only gives great feedback to the entrant, but allows you to recall the sensations of a particular entry days later.

Go to REGGIE