In my recent blog on measures and sampling, I said:
“A key principle for the effective use of data is to ensure it’s viewed in context. By this I mean not comparing one number with another but having sufficient data to analyse properly and doing so in graphical form. One of the best ways of doing this is using time series data graphs.”
A number of clients have asked me if it’s okay to use sample data and if so, is there a risk of looking selectively and not seeing the big picture or the whole system.
Generally, I wouldn’t use sampled data for the main measures of a whole system. It’s preferable to use all the data in a specific period to build a time series data chart. The only exception to this is when it’s impossible to get full data (perhaps due to the IT system or no historical data being available).
In such cases it might be necessary to use a sample, but with caution. At the end of the day, the purpose of measures is to make the system’s performance visible – only looking at a small part is probably ineffective.
We are also interested in seeing if the process looks to be ‘in control’ (variation is randomly distributed about the mean) or if it looks ‘out of control’ (variation not randomly distributed about the mean) and if there is ‘special cause variation’ (factors impacting the system performance in a non random way).
In the case of service systems, there is usually very little true special cause variation. The variation is mostly caused by factors within the system, not external to it. In contrast to the service sector, manufacturing focus is upon standardising to limit variation, whereas in a service system variation needs to be absorbed and dealt with by the system – and true system performance needs to be visible.
In very high volume transactional systems it can be necessary to summarise the data using daily averages (such as the average end to end time for cases completed on a day). This can have two effects: since averages are used variation is less visible, and rather than seeing the variation between cases they show system variation.
Sampling isn’t necessarily wrong but to be meaningful, it’s important that it’s understood in context and if used, that we understand that the sample is sufficiently representative of the whole.
Jaime Beckett, Principal Organisational Change Practitioner – firstname.lastname@example.org