AI for unbiased board recruitment

The challenge

Boardway is a fast-moving tech-startup devoted to building better company boards. In using LLM matching techniques to identify ideal candidates for board positions based on competence, they're challenging classic executive search methods and practices. But building executive search algorithms poses challenges in data collection, bias removal (most LLMs are not neutral off-the-shelf) and model fine-tuning in low-signal environments.

What we did

In assisting Boardway in improving search precision and data infrastructure, we evaluated the status quo, by conducting a combination of pure search retrieval tests alongside LLMaaJs to identify areas of improvement for both retrieval, ranking, cost and answer generation. We identified both simple improvements in swapping embedding models for retrieval - and recommended more efficient LLM selection. In addition, we supported Boardway in setting forth an evaluation strategy involving construction of gold data, stronger signal capture on platform and ongoing analytics in order to compound learnings more quickly.

A huge help while we built our AI search and matching platform. ctrl+eval brought a structured, data-driven perspective to difficult ML problems, helped us prioritize the right product initiatives, and challenged many of our assumptions. They also helped us identify and evaluate strong data science candidates as we grew the team.

Results

Boardway learned about the constraints of their systems and made quick progress towards improving them, which resulted in reduced token spend, faster search, improved accuracy and product experience. In addition, we supported Boardway in screening candidates for full-time positions to support their ongoing mission of making executive search equitable.

Learnings

LLMs don't always agree, with themselves In testing criteria-checking consistency we saw the non-deterministic nature of LLMs at play.

As depicted in the graph, we tested a handful of LLMs' self-agreement rate across a set of criteria-checks, to identify which models behaved best for our purposes. When holding the objective, prompt, context and temperature fixed, we observed, that some models are much more creative (or volatile) than others. This might not impact your day-to-day interactions with ChatGPT; but will be critical when building systems that need to behave predictably over time.