Anyone considering using cognitive training apps is faced with a difficult question…
Is the evidence compelling enough to commit a few weeks of time to it? Does it work beyond placebo effect — or practice effects on the standardized tests?
Consider how you might answer this question?
Online reviews?
A recent publication in the popular press?
A peer-reviewed scientific paper?
Highest Grade Evidence: Meta-Analyses
What is the highest grade evidence possible for any kind of intervention?
The answer: A meta-analyses
A meta-analysis is a statistical analysis that combines the results of multiple, independent scientific studies in different labs around the world — all those which have been published (or unpublished) looking at the type of training in question.
A meta-analysis sifts through conflicting results in different publications and finds the overall statistical effect (if there is one).
A meta-analysis is a part of a systematic review procedure. In developing medical interventions, for instance, a meta-analysis may be conducted on several clinical trials of a medical treatment, in an effort to obtain a better understanding of how well the treatment works.
Most Studied Cognitive Training Types
For cognitive training, two types of cognitive training have received the most scientific scrutiny — with the most peer-reviewed studies and publications — by far:
Working memory (WM) training (e.g. dual n-back)
Attention control training (also known as executive control training).
Both types of training have been studied enough by independent labs to provide a wealth of high-quality data for conducting meta-analyses to look for the overall effect sizes on different aspects of IQ and cognitive functioning.
Dual N-Back Meta-Analyses
The most recent and most authoritative meta-analysis of the effectiveness of DNB training was published by Soveri and colleagues in 2017 (full paper here). It analyzed 33 published dual n-back experiments from independent labs from around the world.
The study shows definitively that dual n-back training does significantly increase different broad abilities underlying IQ — including fluid intelligence (Gf) and working memory (Gwm).
Moreover, in case there was any lingering doubt, the results of this latest meta-analysis are completely consistent with the previous meta-analyses on the dual n-back (e.g. ref 1, ref 2). All meta-analyses of dual n-back training have said essentially reporting the same results (See Table 1 of the Soveri study).
So dual n-back is effective — but how effective is it?
Effect Sizes
The Soveri meta-analysis found the following significant effect sizes (Hedge’s g) for non-verbal fluid intelligence (Gf Non-Verbal), verbal fluid intelligence (Gf Verbal), working memory- verbal and spatial (WM), and attention control (Cognitive Control).
These effect sizes are shown in the table below (from their paper).
0.00 means no effect compared to control groups. If all the dots fell on the zero line, it would tell us that dual n-back training has no effect or simply based on a placebo effect.
Note that the ‘active’ studies control for placebo effects — so we are not looking at placebo effects here. These are genuine ‘far transfer’ effects on IQ and attention.
(Although interestingly there is no significant training effect on verbal fluid intelligence (verbal reasoning skills) — just as we saw in the recent cognitive training plus exercise study.)
If anyone questions whether dual n-back training has a real effect beyond placebo and test practice, the table above can be considered the final word on the data!
The question is not whether DNB training has an effect on IQ — it does. The question is whether the effect is large enough to be practically useful. On that question see the follow-up e-mail.
Attention Control Training Meta-Analyses
The other major type of brain training investigated by scientists is attention control training.
The meta-analysis for this area of research combining 15 individual studies was published in 2015: Does attention training work? A selective meta-analysis to explore the effects of attention training and moderators
The abstract from this meta-analysis is shown below. It is the best evidence we currently have for this type of brain training.
This meta-analysis found (averaging over all studies):
(1) a significant training effect on attention skills (g = 0.25)
(2) attention training significantly transferred to non-trained tasks (academic and cognitive
skills) (g = 0.24)
The non-training tasks were not differentiated, and averaged over “wordreading, writing, comprehension, working memory, visual skills, reasoning, dual-task performance, meta-cognition, executive function skills, and nonverbal IQ.”
Interestingly, the study also found the following larger effect sizes if studies were segregated into different types of attention training and different groups:
(1) Training attention orienting works better than other types of attention training
(g = 1.18!)
(2) Attention training improves attention very markedly for ADHD (g = 0.52)
However, because the number of studies is relatively limited, these results should be interpreted with caution.
Attention Orienting Training
The most effective type of attention training was found to be attention orienting training — with a very large effect size
What is attention orienting? It is selective attention — the selection of specific targets among multiple stimuli. It is also also the ability to disengage attention from one stimulus when needed and rapidly switch to another.
This kind of attention training, combined with dual n-back training, is implemented in i3 Mindware or HighIQPro). It is also implemented in what is called ‘Useful Field Of View’ training which is now known to be another effective brain training intervention for improving cognition generally.
Effect Sizes. How To Interpret Them.
How should we interpret these ‘effect sizes’?
An effect size (Hedge’s g) is the difference between the intervention/training group and the control group in standard deviation units. Effect sizes are conventionally interpreted as follows (with examples from the meta-analyses we’ve been looking at):
0.20: Small Effect Size (e.g. dual n-back effects on fluid intelligence)
0.50: Medium Effect Size (e.g. attention training effect on ADHD)
0.80: Large Effect size (e.g. orienting attention training effect on attention skills)
So the effect sizes on IQ for classic dual n-back training in laboratory settings are typically ‘small’ — around the 0.20 to 0.25 mark.
The effect size of dual n-back training on working memory(Gwm) gains is 0.24. Let’s take another kind of intervention to give this some context. 0.24 is virtually the same effect size of antidepressants such as Prozac in treating depression (ref). See the diagram below from another meta-analysis, this time looking at multiple studies on the effectiveness of anti-depressants.
When we consider that the dual n-back is a highly constrained type of brain training adopted by individuals who are often not very motivated in lab settings, and start to look at combining dual n-back training with attention control training (as in the IQ Mindware apps) and fitness training (see this article as reviewed in the previous e-mail messages), we are now looking to be on very firm footing for answering the original question:
Is the evidence compelling enough to commit a few weeks to? Does it work beyond placebo effect — or practice effects on the standardized tests?
Certainly compared to the standards of pharmaceuticals we can answer this in the affirmative!
And of course, this explains why studies investigating cognitive training continue to proliferate exponentially rather than fitter out — as you’d expect if there was no useful effect.
This was written by Mark Ashton Smith, Ph.D. - originally published in his excellent newsletter for IQ Mindware.
Share this post