Decision matrix, multi-criteria evaluation: turning subjective opinions into a decision!

As we have just released a number of important upgrades to our rating feature, it is important to know how to facilitate an unbiased multi-criteria evaluation workshop. Let's start!

What is a multi-criteria evaluation, and why is it so important?

Also called decision matrix among insiders, a multi-criteria evaluation is a collaborative decision-making tool to assess and compare several alternatives based on a set of criteria.

Whether you’re trying to hire the right candidate for the job, deciding how to allocate your R&D budget or choosing the next market to invest in, multi-criteria evaluation are key elements of your daily professional life, and permeates every single important decision you make to grow your business. That’s why it’s so important to get it right.

Turning subjective opinions and ideas into objective data and ratings, a multi-criteria evaluation is key to making the right decision through the evaluation of all the most significant aspects of the problem at hand. It neutralises and gets rid of the main personal biases to reach the most comprehensive assessment thanks to a well-crafted set of criteria.

The collaborative aspect also plays an instrumental part here: it enables your teams and organisation to find common ground and reach a consensus during a process that’s often full of mutual anger, frustration and resentment between colleagues at odds with one another.

In other words, the efficiency, robustness and reliability of a multi-criteria evaluation rid you of the most troublesome aspects of decision-making, and provide the guarantee for your teams to get behind and support the final decision.

It goes without saying, but all this is true only when the multi-criteria evaluation is prepared, organised and facilitated properly. How does one do that?

When should I use a multi-criteria evaluation?

The first question to answer, evidently, is when to use the multi-criteria evaluation? Anytime you have to choose between a short number of alternatives. As briefly mentioned above, this highly versatile and nearly omnipotent tool can be used in a myriad of different contexts, and can be applied in the most diverse business settings. Here are a few examples:

• Human Resources: selecting a future employee among different top candidates;
• Strategy: evaluating strategic alternatives for the company’s future business growth;
• Marketing: selecting the best pricing model for a product/service;
• Risks: assessing risks;
• Supply: selecting a supplier or a product to buy;
• Communication: choosing the best logo for a campaign;
• Innovation: managing an innovation project portfolio;
• Budget: allocating funds to the most promising areas of growth.

In other words, any group of professionals can benefit from using a decision matrix and end up taking better informed decisions by carefully evaluating all the alternatives.

How can I facilitate a multi-criteria evaluation workshop?

Despite all the best intentions in the world, organisations can end up doing more harm than good when organising a multi-criteria evaluation. Here are all the most basic steps to follow to avoid making common mistakes.

Prerequisites

• A list of well-documented alternatives, which everyone involved in the process understands and is familiar with.
• If possible, make sure to keep your list of alternatives to 7 or less (with a voting session or a $100 budget game, for instance), or the decision matrix will become chaotic and counterproductive. Any additional alternative that wasn't shortlisted can, if needed, be put on a waiting list for future use.
• The right evaluation team: as obvious as it may sound, having the right people in the room is an aspect that shouldn’t be underestimated, at every step of the process: to select the alternatives, choose the criteria and take part in the evaluation.

Step 1. Create powerful criteria

1. Brainstorm a list of criteria
2. Select and refine to make sure they’re the best ones that will be able to deliver the most comprehensive evaluation
3. Assign the appropriate weights and define scales to each criterion
4. Clarify their meaning by providing a short definition of each criteria and indicating labels for every graduation of the scale. This is to make sure participants all have the same understanding of the criteria and the scales.

Step 2. Evaluate alternatives, one by one

1. Present and clarify the first alternative to rate
2. Participants silently rate the first alternative with the different criteria
3. Debrief, debate and share opinions
4. Participants can change their individual ratings until a consensus is reached
5. Repeat the same process with all the alternatives

‘Low-cost’ version: Instead of carefully evaluating every single alternative one by one, you can allow participants to evaluate all of them at the same time, and then organise a debrief. While the aforementioned process is the most robust one, this ‘low-cost’ version can work if you have very little time and a large amount of people to evaluate a set of alternatives.

Rather than a serious decision matrix, this process is more reminiscent of a survey to take the pulse of your participants, and is unlikely to reach a consensus or lead to a decision by the end of the session.

Step 3. Compare alternatives and take a decision 

The primary objective of any multi-criteria evaluation is to gather insights in order to make the best decision. Once your decision matrix is completed, the time has come to do exactly that. But contrary to what many organisations still do to this day, the decision outcome should not be binary (“It’s a go!” or “It’s a no go”), but ternary: Go / No Go / Refine.

Be careful not to count your chickens before they’re hatched. Whether an alternative has been found to be incredibly promising or on the contrary failed to make the cut, packing up your bags with definite “winners” and “losers” in mind is a premature move. A “winning alternative” can always be improved, while some of the losing ones can contain highly promising aspects that need to be nurtured and enriched to fully reveal their potential.

Enter the greenhousing process, a powerful instrument and mindset to grow your seeds of ideas into robust and actionable concepts.

The basics: Tips and tricks for a great multi-criteria evaluation

To help you grasp the most important aspects of a multi-criteria evaluation, here’s a quick list of the ultimate tips and tricks to stay on the right track.

Put the right people in the room

The importance of choosing the right people based on their post, expertise and experience to take part in the evaluation cannot be stressed enough. All the rest depends on it.

Choose the criteria wisely

If you don’t use one of the most popular and established criteria templates (like NUF: New, Useful, Feasible), take some time to choose them wisely.

You can always use the MECE method to ensure that the variables you’ve chosen are both “Mutually Exclusive” and “Collectively Exhaustive”. In other words, choose criteria that don’t overlap but whose combination encompasses everything that needs to be taken into account. A simple example: Costs and Revenues.

Double-check your criteria

You can never be too careful when it comes to choosing the criteria. One thing you need to double-check and triple-check is to avoid any possible misunderstanding of the variable. Every participant has to be on the same page. To come back to the previous example: is “Costs” really clear as a criteria, especially if you bring in people from different departments? What are we talking about here: Production costs? Logistics? Marketing costs? You get the idea.

Make sure the rating is consistent between raters

Everyone also needs to be on the same page when it comes to the scale and its graduations. Instinctively, we tend to see the highest value as the “best” value. Again, let’s use the “Costs” criteria. A rating of 10 on a scale from 0 to 10 could mean two things: “costs are great” (aka low) or “costs are very high”.

To avoid this type of confusion, add precise labels to the scale and make sure everyone understands what it’s about. Another alternative is to use one alternative as a baseline - in what is known as a Pugh matrix - and evaluate the other alternatives in reference to that baseline, rather than in absolute terms.

Beware of group biases

Negative group biases are always lurking around the corner, so keep an eye out for them. The most common ones include failing to be neutral while presenting the different alternatives, using different schemes to influence people’s ratings during the evaluation, or even changing the weights of criteria after the evaluation to ensure it fits one’s own agenda.

Don’t trust average values

Instead of relying on average values, which fail to describe how people truly voted, set your sights on the level of consensus. That means looking at the detailed distribution of ratings along the scales, rather than making hasty conclusions based on the mere average. 

Why use Stormz to facilitate a multi-criteria evaluation session?

Thanks to some of its one-of-a-kind and powerful features, Stormz will help you conduct groundbreaking multi-criteria evaluation sessions. Both in person and remotely.

Before the decision-making session

• Document each alternative. You can list as many alternatives as you need in Stormz cards. Stormz cards are much more than virtual post-it notes, and allow you to describe the concept in detail, add illustrations or even attach documents.
• Weigh the pros and cons. Before the evaluation, you can use the upgraded Stormz list feature to collect all the strengths and weaknesses of each alternative.
• Shortlist alternatives. Trying to evaluate too many alternatives is a bad idea. Use one of Stormz voting dynamics to shortlist them and keep the ones that truly matter.

To help you choose the best criteria

• Brainstorm the criteria. Create and facilitate a full-blown brainstorming session with Stormz to generate and select the best criteria.
• Benefit from unparalleled flexibility to parameter custom-made rating scales. Use Stormz’ advanced settings to parameter the chosen criteria to the smallest details: give them different weights, clarify their meaning with a description, define custom linear or non-linear scales, and add precise labels on each graduation.

During the multi-criteria evaluation session

• Customize the evaluation process to your needs. Use Stormz’ steps system to prepare and facilitate a custom-made collaborative session for your participants.
• Neutralize group biases. During the rating process, use Stormz’ advanced permission system to hide other people's ratings to keep them from influencing one another, and randomize the order of the alternatives.
• Use advanced visual tools to analyze the results. Stormz has released new real-time charts to analyze and compare the strength of each alternative. Once the evaluation is completed, use the in-card charts to visualize the level of consensus and facilitate an after-session discussion.

After the session

• Export detailed results in one click. All the rating data, including data for each participant, can be exported in an Excel file for further analysis.
• Improve weaker alternatives. If some of the promising options weren’t shortlisted, you can always run a greenhousing session with Stormz using POWER, GPS or any other concept development framework you’re comfortable with.