Question Targeting, Relevancy, and Validation

Question targeting and relevancy

When you are choosing which question to use, one of the most important things to get right is to choose a relevant question that is testing fundamental piece of knowledge (e.g. working with collections of data) and not something that only professionals with a specific type of experience had encountered (e.g. sound processing).

For example, it’s common to find algorithmic problems that test a very specific algorithm that candidates don't have a way of designing if they haven't seen that exact algorithm before. In our case we always put simple unknown algorithmic problems that don't have an official name and that way we are sure we are testing algorithm design or algorithmic thinking instead of just remembering and reproducing an algorithm candidates have already memorized.

Therefore our questions are targeting only highly relevant pieces of knowledge. We would really appreciate it if you could let us know your thoughts about this topic as we’d love to understand what our users think and what they consider important.

Validation studies

Some users ask if our tests are valid as defined by The U.S. Equal Employment Opportunity Commission (EEOC). The legal answer is no, because:

A validation study has to be done against a job description which we don't have access to nor control over.

The EEOC says: "Employers should ensure that employment tests and other selection procedures are properly validated for the positions and purposes for which they are used. The test or selection procedure must be job-related and its results appropriate for the employer’s purpose. While a test vendor’s documentation supporting the validity of a test may be helpful, the employer is still responsible for ensuring that its tests are valid under UGESP."

A validation study should cover the whole test which we don't have control over.

We provide a question library from which employers choose a set of questions they want to use. We also provide a functionality for employers to add their own custom questions that we have no control over.

Question quality assurance

We have a quality assurance process in place for our questions. In short, apart from the question author, at least two independent subject matter experts (typically with interviewing experience) have to review the question and ensure that the question is testing a fundamentally important concept that can separate good from bad candidates. On top of that, each question is presented in a test setting to tens of candidates claiming to be somewhat experienced in the relevant field where we make sure they can understand the question. They start off in the right direction if a reasonable number of them can solve the question completely. Only after that do we release the question to production.

Once released, we continuously monitor score statistics and feedback for our questions and take appropriate actions where needed.

Work-sample testing

The main idea behind EEOC's regulation on testing is to prohibit employers from using selection procedures that have the effect of disproportionately excluding persons based on race, color, religion, sex, or national origin, where the tests or selection procedures are not “job-related and consistent with business necessity.” The way we fight that is to focus on work-sample based testing where it's very easy to show that a person who doesn't do well is obviously not fit for the job irrespective of which group they belong to.

Obsolete and Deprecated questions

Our procedure for handling different versions of a question is the following:

- When we release a new version of a question due to a bug, the old version is deprecated. This means that the question is unsuitable for candidates and at that moment you will have to create a new test or clone the existing one. Cloning a test creates an exact copy with the same settings and questions, but automatically updates all the questions to their latest version.

- When we release a new version of a question as a minor improvement, but nothing is significantly wrong with the old version, we mark the old version as obsolete. You can continue inviting candidates to the test containing the obsolete question, but when you add the question to a new test or clone an existing test the question will automatically be updated to the latest version.

- We also deprecate questions which have been obsolete for over a year. There will be a warning about this a month earlier to notify you if your tests will be affected.

We are doing all this to make sure that our customers rely on questions whose quality we fully stand behind.