Agusto & Co, a credit rating and research agency, committed a serious error in its methodology on its report titled “2018 Agusto & Co. Consumer Digital Banking Satisfaction Index”.
The 2018 Agusto & Co. Consumer Digital Banking Satisfaction Index, comprising a survey and a scorecard, examines customers’ preferences and attitude towards digital banking platforms hosted by their respective Banks. A focus group of respondents were drawn from the formal and informal sector segments of the economy. Respondents were sampled from various geopolitical regions within Nigeria including the South-West, South-East, South-South, North-Central and North-West regions. Respondents were also a combination of students, self-employed and employed customers of various commercial banks in Nigeria. The data collection technique used was a questionnaire designed to gain insight into the behavioural pattern of the sample population. The survey focused on issues around service quality and ease of carrying out transactions. The questionnaire comprised of multiple choice, closed and open ended questions. The questionnaire was administered both electronically and physically, thus encouraging a wider pool of respondents across the country.
It is very strange that banks affected have not pushed back. But in Nigeria, intellectual conversation is not in our styles. Agusto & Co failed in its core objective: “to create an independent appraisal of the ease of using digital banking platforms by the Nigerian populace considering that banks have invested significantly in digitalization.“ Simply, they had a fudge factor in the data collection and that rendered the conclusion meaningless.
This is the full report from Augusto & Co (download here). It made a big mistake on the sample collected in Question 3 ‘Who do you bank with?”. Here are SOME of the main issues looking at the plot:
- GTBank provided 20% of the participants. That is a big problem. GTBank does not command 20% of the market by number of customers in Nigerian banking. First Bank (I do think) is the largest bank (by customer size) in Nigeria. So, not making the sample to represent reality is a huge problem.
- Even if we assume we can ignore the actual market share, there is still an issue. Yes, when you have many GTBank customers, above a clear mean, in the study, you are essentially moving many elements which may affect the bank differently from its peers. Largely, StanbicIBTC Bank should not have been included in the final rating as it did not meet the minimum participation level for a scientific assessment. You cannot have one bank providing 20% and another 2%, of the sample if you have not bothered to consider the actual market shares (by customer size), and expect the inferences to be uncorrelated.
- Consider this scenario: you went to Ajegunle Lagos and meet 100 people. You ask them about their banking experiences. In all, one person used StanbicIBTC and he is happy; you give StanbicIBTC 5 stars. But you sampled 10 Union bank customers; 9 are happy while one is not. For that you issue 4.9 stars. You have not done justice to the whole process: StanbicIBTC might have done well but that is not reality. To get a real scientific result, your sample must mimic the actual customer distribution to a large extent and you need to sample many customers.
- Look at Access Bank (13.5%) and Zenith Bank (7.7%), you are having nearly 2x participation ratio between them. That is creating a fictitious market share in this industry where Access Bank is nearly twice of Zenith Bank! And you know there is trouble when you have a sample size with more Access Bank than First Bank customers in Nigeria. (The report did not explain how it managed all that.)
- I can go on and on but let me spare you the troubles
In short, the report is totally not balanced. I picked the press release from Guardian but did not read the report before sharing the conclusion. But one of my data engineers looked at it and flagged it: “Prof, if I did what Agusto did, I would be in trouble with you.”
Sure, while it is possible StanbicIBTC is doing well and GTBank has issues (I spent 46 minutes, last week, in its Ajao Estate branch for a very simple transaction as the banking hall was full), this report is not a scientific way to make sense of anything. We can all rant over services in the banking sector. But this report offers no credible insights. GTBank is not 5th and StanbicIBTC is not 1st either – they are all working and fixing market frictions but no scientific work exists, yet, to rank them.
Agusto & Co should withdraw it and re-do the data collection and analysis. It needs to calibrate out the impacts of the sampling to get a better picture of the state of our banking innovation. And they need to include how many customers that were sampled, not just the percentage distribution by banks. Did they sample 100 or 10,000 customers? You cannot miss such in a very impactful report like this.
Update
This comment on LinkedIn explains my point better.
[Name] not necessarily how much more but what are the probabilities or assumptions involved in the sampling? For example, what is the share of banking customers in Nigeria as a population? What share does each bank have? What is the regional split of the banking population? All these make up to decisions on sampling. As commented earlier, the approach would have been more suitable with a quota sampling than random.
