Audit firms, methodology providers, regulators and standard-setters have all, over recent years, sharpened their focus on professional judgements around audit sampling approaches, the justification of sample sizes and the quality of associated documentation.

There have been revisions to International Standards on Auditing, such as ISA (UK) 315 and ISA (UK) 530, a Thematic review of sampling by the UK’s Financial Reporting Council (FRC), and developments such as the removal of sample size caps by some methodology providers.

All of this has led to uncertainty among some auditors about what these changes mean for them.

The Audit and Assurance Faculty has been providing ongoing commentary, context, clarification, and a range of practical support resources – and a list with links follows this article.

Making sense of sampling and substantive analytical review was the subject of a faculty webinar, in March 2024, exploring robust sampling and effective substantive analytical procedures in the context of the developments above. It prompted lots of discussion and this article explores some of the key questions that arose.

Is it true that sample size caps are now banned?

This is not true. However, comments made by the FRC in recent years have highlighted the disadvantages of capping sample sizes on significant risk areas and key audit matters. This has led to a change in approach from some methodology providers, at least one of which has removed caps. In its 2022/23 Audit monitoring report, ICAEW also commented that “methodologies that apply a judgemental cap on sample sizes can give a false sense of security and it is not an approach that was ever intended to be simply taken by default”.

Is it true that some audit firms are changing methodology provider, so that they can still apply a sample size cap when performing substantive tests of detail? 

This has been happening – but it seems a very draconian step to take. Evidence to support the auditor’s opinion regarding different assertions can be gathered in different ways, which means that sample sizes for substantive tests of detail need not be prohibitively high. The need to invest time at planning to perform a proper risk assessment was also highlighted during the webinar. Auditors can sometimes find themselves doing too much work in some areas because they’ve not really thought about the nature and extent of the misstatement risk. 

When performing P&L testing, are auditors perhaps too reliant on transaction testing?

This was explored during the webinar. Many firms slavishly default to transaction tests without thinking about alternative testing approaches, such as substantive analytical procedures and/or the audit data analytics tools that are increasingly used by independent firms. Although some substantive tests of detail are likely to be required when auditing P&L figures, bringing other sources of evidence to bear will lead to a potentially more efficient approach and allow the auditor to get a better and broader understanding of the entity.

Might a general increase in methodology-driven sample sizes make auditors think more about testing controls?

Absolutely – and this is something that more and more firms are interested in exploring. Auditors do sometimes test controls without realising they’re doing it. For example, substantive tests of detail on expenditure often involve checking for the existence of a purchase invoice when testing occurrence – and for evidence of that invoice’s authorisation. This is surely a substantive test of detail and a test of control. Shouldn’t we therefore be ‘taking credit’ for both?

Care must be exercised here. In order to rely on this authorisation control the auditor needs reasonable assurance that it can’t be circumvented or overridden. How do we know that the control will be applied in the way that it should every time? However, by spurning tests of controls, some firms may be missing a good opportunity for an audit that is both more effective and efficient.

We do walkthrough tests, so aren’t we already testing controls?

Not necessarily. A walkthrough is performed to get evidence of implementation of systems and controls, a requirement for all ISA audits, whether a controls-based approach is or is not being adopted – although it is possible to carry out a walkthrough and a test of control at the same time. During the faculty webinar, auditors were encouraged to consider using the term ‘walkthrough’ rather than ‘walkthrough test’ to help them to avoid thinking of it as an actual audit test for which ‘credit’ can always be taken.

If we’re testing creditors, is it right to base the sample size on the total balance?

This is a question that was explored during the webinar. ‘Blind’ application of an audit methodology often results in an auditor inputting a total population balance into a sample size calculator – after stripping out individually material balances that require separate scrutiny – then seeing what sample size pops out for scrutiny.

However, this approach might be flawed. Principles of directional testing can often lead us to test credits for understatement. So, if we’re testing creditors for what is not there, why focus exclusively on what is there as the start point for the calculation? A better approach would be to focus on key suppliers based on purchase ledger activity and to perform a robust review of post year-end activity to check for missing accruals.

The FRC’s 2023 thematic review also deals with the application of judgement when sampling. The review specifically highlights the importance of explaining relevant judgements clearly on the audit file – and judgement could be required here.

We are regularly told that there can be problems when using sample size calculators on very small and very large populations. How so?

This is true. Where a population contains many thousands of items and the audit approach just involves a substantive test of detail, firms are urged to challenge themselves on whether a different line of attack is required. Paragraph 8b of ISA (UK) 330 states that tests of control will be required “where substantive procedures alone cannot provide sufficient appropriate audit evidence at the assertion level”. This might be a case in point.

Small samples have an increased risk of being unrepresentative and unreliable. Where a total population comprises a small number of items, alternative procedures should always be considered. 

Can we rely on our methodology provider’s sampling tool, or should we be developing our own?

It’s fine to base your sampling approach on a standard methodology, but care does need to be exercised. As a minimum, firms need to understand the methodology and be able to appreciate when it will and won’t work.

ISQM (UK) 1 states that even when a firm belongs to a network and makes use of network resources, the firm ‘remains responsible for its system of quality management, including professional judgements made in the design, implementation and operation of the system of quality management’. 

In its sampling thematic review, the FRC emphasises that firms must ensure they have a proper and full understanding of the sampling techniques … (they use) … and are able to understand and apply those methodologies.

Peter Herbert, Director, Insight Training