How We Shortlisted 12 People from 50 Candidates

Shortlisting can feel like a mammoth task. Take a too high-level approach and you’ll be left with the sneaking suspicion that you missed something important. Get bogged down in the nitty-gritty, and everyone gets overwhelmed and demoralised.

Our co-founder Andy recently directed a shortlisting process to narrow around fifty applicants to a twelve-strong interview list. Saying goodbye to three-quarters of an applicant pool is never easy, so we decided to share his shortlist process in the hope that it could help others attempting a similarly gruelling task.

(Please note: none of the screenshots shared here include anything sensitive or confidential! We recreated the process with dummy info to share it with you all.)

The first stage was Review: somewhat similar to an academic peer review. Each application was independently assessed twice. Reviewers included organisation staff as well as people with relevant industry experience at other organisations (universities, consultancies, and other businesses). A review score sheet was provided which had sections for comments and a final recommendation regarding interview: Yes, No, or Maybe.

All of the independent reviews were collated in a spreadsheet. Ahead of the shortlist meeting, each panellist was randomly assigned a subset of the applications (each with two reviews) to read in detail. The idea here was that the shortlist panel would have a combined, deep understanding of each applicant, so that the meeting could revolve around discussion instead of presentations (we’ve written before about what makes a good meeting).

On the day of the shortlist meeting, all panellists were invited to an Ideaflip board.

A schedule for the day showed work blocks, each containing a rough goal of how many applications to be processed. Regular breaks allowed everyone involved to see the anticipated pacing.

A yellow note was created for each application. These were input via Quick Entry by copying and pasting a column from the summary spreadsheet. Notes contained only the application’s project title and company: just the right amount of information to be memorable, but not overwhelming. This was also an opportunity to think about how to remove bias from the process.

Groups were set up for each area of the board, including a very clear one in the middle for the current application under discussion. This helped retain focus and avoid “which one are we on?” questions!

Once everyone had joined the Zoom call, panellists presented the applications they’d been assigned to read through. The day started with the ‘No/No’s (applications which had both received two No recommendations) followed by the ‘Yes/Yes’s, with everyone discussing whether they should indeed go forward for interview, or be put in the permanent ‘No’ pile. These tended to be the easiest decisions, helping the panel to establish working benchmarks of low and high quality applications.

The most polarising applications were next. Anything with a Yes/No, Yes/Maybe, or No/Maybe required much discussion. Maybe/Maybes were then addressed; experience dealing with previous applications made these difficult decisions slightly easier to process.

Regardless of who had presented which application, all panellists were encouraged to ask questions and discuss implications of moving a note into a group. The temptation was to press on and get it all done, but regular screen breaks proved important to keep the whole shortlist process sustainable. The schedule was constantly updated to reflect progress, giving everyone an idea of pace and progression.

A key advantage to Ideaflip is its fluidity. As the applications were reviewed, they were placed in whichever zone felt right: Yes, Maybe, No (or in between, if there wasn’t a clear answer). The notes were put in roughly the right place and ensuing discussion created a gradual movement towards a consensus. The extra time taken was more than worth it, as everyone involved felt their views had been heard.

As the target shortlist number of twelve drew near, colours and badges were added and themes were noted, as the panellists needed to curate a balance. These visual differentiations helped to inform some of the final decisions and movements between groups. This was also a way of noting applicants who could be forwarded onto other opportunities better suited to their projects.

Two applications were earmarked as backups in case any of the people invited to interview had experienced a change in circumstances, and no longer wanted to participate.

Once every application had been considered in turn, and the panel was happy with its decisions, the board was exported. There are numerous file formats an Ideaflip board can be exported to. A CSV allowed easy sorting in a spreadsheet, along with any annotations. Panellists received a PDF. The final stage was for everyone involved to sleep on their decisions to ensure it continued to feel like the right course of action.

We appreciate that we’ve gone quite detailed with the shortlist process used. In summation, here’s how to do it:

1. Create a score sheet so reviewers can score applicants on key areas, write a brief summary, and recommend Yes, No, or Maybe for interview.
2. Collate all reviews in a spreadsheet and give each a short, memorable title.
3. Randomly assign applicants (each with two independent reviews) to panellists so they can read and understand ahead of the shortlist meeting.
4. Import the titles into Notes on an Ideaflip board using Quick Entry and group them.
5. Create a shortlist meeting agenda on the left of the applicant pool (remember to factor in overrun time and plenty of breaks).
6. Add groups to the right of the applicant pool: Current, Yes, Maybe, No.
7. Panellists present applicants for discussion. Everyone works together to decide which group the applicant belongs in.
8. Use buttons, badges, text and different colours to help sort, and record feedback and feelings about positioning. Allow plenty of space is allowed for disagreement and compromises.
9. Once all the applications are in an approximate order, invite discussion on the final positioning (for example: review earlier positionings in light of applications considered later in the day).
10. Export the Ideaflip board as a CSV or PDF to share with everyone who attended.
11. Invite applicants to interview safe in the knowledge your shortlist process was robust!

Have you used a similar approach before?

We’d love to hear about how your organisation creates an interview shortlist from a large pool.

The topic isn’t widely discussed, but sharing best practice has the potential to improve the experience of both applicants and panellists. Get in touch on Twitter or LinkedIn.