A curious development in our age of information overload is that while organisations today are awash in data, the explosion has not led to a commensurate enhancement of insight. In fact, many businesses can be described as suffering an insight deficit. This shortfall is especially damaging to both companies and clients as they miss valuable opportunities for growth and improved profitability.

For me, insight means a new understanding that makes a difference to the organisation’s thinking, decision-making and actions. This working definition includes two parts:

• a new understanding (ie, about customers, competitors, regulatory change, shifts in financial condition, etc); and
• what difference it makes for the organisation’s assumptions, decision-making and actions.

Here are six steps to help guide you in moving from data to insight. I will use a case from a recent workshop to illustrate the sequence.

Analysis is always purpose-driven. It typically addresses a ‘why’ question, such as:

• why have our overhead costs suddenly escalated compared with two years ago?; or
• why is our principal competitor’s profitability consistently higher than ours in the major geographic markets?

At the workshop, the managers started by focusing on these questions: why are our margins declining in one product area, and what can we do about it?

Once you have your ‘why’ question, the next step is to ask: what factors might influence the noted change, now or in the future? You can identify a surprisingly wide range of possibilities if you solicit ideas from individuals with different perspectives, ie, people from different functions, levels in the organisation, backgrounds or types of expertise.

At the workshop, I asked: what factors might influence the direction of margin change, now or in the future? Interestingly, some analysts from the finance department argued strongly that the recent fall-off in product quality compared with rivals, and the subsequent decline in sales revenue and margins, was partly down to the company’s failure to invest in the latest technologies.

But when we asked engineering personnel for their explanation, they all believed that the answer was obvious: a recent switch to a lower-cost supplier of a key component. Both potential factors – outdated technologies and inferior components – were added to a list that grew to include several other issues.

When assembling your list, be sure to extend the search beyond the historical factors, keeping both the present and the future in mind.

Analysis teams often get lost in their efforts to refine the data. Spreadsheet manipulation can become a substitute for thinking. One way to seek an understanding that may not be evident in the data is to ask the question: what indicators are relevant to each change factor? Unfortunately, an abundance of data does not mean that the relevant change indicators are apparent.

The indicators specific to the factors noted by the aforementioned finance and engineering functions are radically different. For investment in manufacturing technologies, they include:

• types of technologies available;
• types purchased;
• suppliers offering each type;
• costs involved;
• purchase decisions made or delayed; and
• time of installation.

By contrast, for the change in supplier, they include:

• number of available suppliers;
• products they provide;
• raw materials used in manufacturing the component;
• location in which it is manufactured;
• costs and terms associated with each product; and
• delivery dates.

Similarly, outcome indicators consist of:

• product throughput;
• product quality;
• sales units in different channels; and
• prices.

Leading indicators are critical to deriving insight. They help anticipate change before it occurs. For example, a change in the raw materials of a supplier’s component may reveal potential issues with the quality of the product manufactured by the firm that purchased the part.

Indicators – or more precisely, the change along them – do not speak for themselves. As analysts, we must derive meaning from the change. Thus, perhaps the most critical step is to get individuals to draw inferences – that is, tell us what the change in the indicator suggests to them.

Each suggestion needs to indicate the data and offer reasoning that supports it. To revert to the engineering team’s explanation, the impact of the change of supplier led to two significant inferences:

• finding a new supplier that would collaborate more closely with our purchasing team could lead to lower logistics costs; and
• based on its technology reputation, a new supplier could deliver a superior-quality component resulting in greater customer value and higher sales.

Again, to attain richer insights, it is imperative to ask individuals to come up with more than one conclusion and to provide the reasoning supporting them.

Drawing inferences provides the raw material for moving toward the possibility of new understanding. It involves the following three steps.

• First, group the inferences into topics or subjects, for example, cost structures, revenues, margins, profitability, current supplier, key current and future decisions, emerging business issues, marketplace change, rivals’ behaviours, etc.
• Second, ask individuals to review, synthesise and write down any overarching realisations that emerge about individual topics. You could ask: what new understanding seems to be emerging about cost management, or as an explanation of declining profitability, or how a rival is successfully charging prices higher than us? In our declining margin case, an unexpected realisation for the finance function involved HR policy changes. The finance team had not appreciated that the commitment to meet the wage levels in the region of the manufacturing plant had recently increased due to enhanced competition for labour because of several manufacturers moving into the area.
• Third, compare the new understanding to the former in order to ensure that there is a significant difference – the greater it is, the more likely that it will give rise to important business implications. Previously, the finance function had assumed that labour costs would increase only marginally, if at all.

In the workshop, it appeared that a combination of isolated but significant changes in management policies across multiple departments, including purchasing, HR, finance and sales, had all contributed to the unexpected decline in margins.

The value of insight resides not just in attaining the new understanding but in determining its implications for three aspects of the organisation: its thinking, decision-making and actions. Otherwise, it might be nice to know, but have little business value, if any.

• Thinking: How does the new understanding impact thinking, such as its effects on your management’s assumptions? In our margin case, the finance function’s initial assumptions about profitability for the financial year had to be modestly revised downward.
• Decision-making: The choice to change to a lower-cost supplier had to be revisited. Unless the supplier could remedy the issue of component quality, the decision would likely be made to seek another supplier, perhaps even reverting to the original supplier.
• Actions: One immediate action would be to create a cross-functional team to identify how each function’s decisions and actions affected the firm’s margin, and whether and how their decisions and actions were positively or negatively reinforcing it.

It is worthwhile following the steps that I have outlined above if you want to add more value either to your company or to your client. You can provide insights that can help to transform an organisation into a more successful and profitable enterprise in the future.

Dr Liam Fahey, Professor of Management Practice at Babson College USA, Co-founder of Leadership Forum, and author of new book The Insight Discipline (Emerald Publishing, £15.99)

• Strategy, risk and innovation hub

Further reading into strategy, data science and analytics is available through the article and ebooks below.

These articles and others are available below to logged-in ICAEW members and ACA students.

Online articles

The Library provides access to leading business, finance and management journals. These journals are available to logged-in ICAEW members, ACA students and other entitled users subject to suppliers' terms of use.