It is the attainment gap fallacy that is damaging the life chances of FSM children in the north of England and elsewhere

The ‘north/south attainment gap’ claimed to exist by The Sutton Trust, The Social Mobility Foundation, the DfE, the National Schools Commissioner and virtually the entire English educational establishment is a fallacy. The actions taken by the government to ‘close the gap’ through market pressure on allegedly under-performing Northern schools from league tables and OfSTED  is counter productive and having the opposite effect to that which is intended. This has been revealed by this recent BBC News investigation.

The under performing schools  allegation is based on the comparatively poor attainment of  north of England FSM pupils at GCSE, compared to their KS2 SATs scores. A comparison is made with pupils suffering comparable levels of socio-economic deprivation in London Boroughs, where this ‘attainment gap’ is not found. The true explanation for this lies in the different cognitive ability and ethnicity profiles, featured in the BBC report, of London Boroughs compared to ‘white working class’ districts targeted by the ‘attainment gap’ allegations.

The BBC rightly draws attention to the perverse outcomes of the DfE’s ‘Progress 8’ school accountability measure, which uses KS2 SATs as the baseline for measuring the progress made by students at their secondary schools. The fact that neither the SATs themselves, nor their statistical manipulation by the DfE, are fit for purpose gives rise to even greater concerns than those raised in the BBC article.

Recent research by The Education Endowment Foundation (EEF) casts serious doubts about the validity of the ‘attainment gap’ claim which is discussed in my articles on the subject here and here.

I have been supported and joined in my investigations by John Mountford, a retired headteacher and former OfSTED inspector. His inquiries have revealed that it is not just Free School Meals (FSM) children in the north of England that are affected, but also in the more prosperous south. This is the letter he has sent to his MP.

Dear Mr Rees-Mogg

Thank you for your prompt reply to my initial inquiry regarding testing in schools (copied below for your convenience). I note that you have referred the matter to Rt. Hon. Nick Gibb and are awaiting his response. I will, in due course take the opportunity to attend one of your surgeries, as suggested. In the meantime, there have been further developments I wish to bring to your attention.

The limited research my colleague and I are engaged upon is, even at this early stage, yielding results that require a response from the DfE. 

The KS2 SATs were revised for 2016. In the original version, the raw marks from the exams were used to set National Curriculum Levels, with the raw mark thresholds for each level determined each year by the DfE. The DfE then determined the minimum acceptable proportion of pupils in every primary school that should achieve Level 4. This was the ‘floor target’ with schools failing to meet it being placed in ‘Requires Improvement’ or ‘Special Measures’ categories by OfSTED. SATs have always been ‘attainment tests’ based on specified content set out by DfE.

The post 2016 SATs are different. The concept of National Curriculum levels has been abandoned. The DfE now report SATs results on a ‘scale’ with a mean of 100, a minimum of 80 and a maximum of 120. This is ‘explained’ here, except that there is no explanation, just a set of conversion charts, changed each year, to convert raw SATs exam marks to a score on the 80 – 120 scale.

There is no explanation of what ‘attainment descriptors’ apply to the scaled score of 100, nor to any other scaled score including the minimum and maximum of 80 and 120. The only valid statistical alternative to criterion referenced attainment descriptors is norm referenced percentiles. For example the IQ/cognitive ability scale enables percentiles to be obtained for every standard score. It is not clear that the SATs ‘score’ is a standard score at all in the statistical sense. If it was, then the DfE could state the percentile represented by the minimum expected score of 100 for all pupils. In the 2017 SATs, DfE announced that 61% of pupils had met the ‘expected standard’ and attained a scaled score of at least 100. It is therefore clear that the ‘expected minimum scaled score’ of 100 cannot be the 50th percentile if  61 percent attained it last year.

We have asked respected academics of international standing to comment but none have so far made any statistical sense of it. We invite you to take advice from your own contacts in the academic world alongside any response you get from the DfE. It appears that the SATs ‘scale’ of 80 – 120 is not a ‘standard scale’ of any kind. It appears to be an arbitrary creation, along with the conversion tables for converting raw marks into SATs scores. In this context, it is important to note that Cognitive Ability Tests, in contrast, are standardised according to established statistical procedures, which is why they are still employed by grammar schools for their 11 plus selection tests.

The data acquired as part of our research confirms that the SATs results are inflated when compared to Non-Verbal Reasoning test standard scores, and especially for pupils attaining the lower NVR scores. This has serious implications for secondary schools, especially in relation to setting Attainment 8 and Progress 8 targets, especially for those schools with high numbers of FSM children on roll, because we know from data published by GL Assessment, who provide the Cognitive Ability Tests, that FSM children, on average, have lower cognitive abilities.

For example, we have used FoI to obtain the data for the nine  Non Verbal Reasoning Test bands used [for admissions purposes] by one large school. These give the number of pupils in each band in brackets. Underneath each band the mean scaled SATs score for reading and for maths are provided, in that order.

NVR Band 1 69-73 (9) corresponds to -2 SD (2nd percentile)

SATs 95, 97

Band 3 82-88 (25) to -1 SD (16th percentile)

NVR Band 1 pupils should presumably be performing at the 2nd SATs percentile. We do not know what SATs score this corresponds to, but it is certainly not 95/97

NVR Band 3 pupils should be performing at the 16th SATs percentile. This cannot be 102/104

This is like taking the bottom of the regression line and moving it up so that -2SD becomes 96 instead of 70, which we are informed is statistical nonsense. We hypothesise that this pattern results from primary schools, having a high proportion of low NVR pupils, resorting to cramming and coaching methods to meet the DfE floor target. Such children will have understood little and forgotten most of it by the end of the summer holidays, which is what hundreds of secondary schools report as the reason why they buy the Cognitive Ability Tests (CATs) to reliably inform diagnostic and target setting interventions for their pupils. 

So, it emerges from our analysis that SATs scores are systematically inflated for pupils of lower cognitive ability, and the lower the cognitive ability score, the greater the inflation.

 I apologise for the volume of detail contained herein, but real life is rarely simple.  As you will appreciate this is a matter of great importance. The fate of individual pupils and schools depends on this system being transparent and reliable. Clearly, this could potentially threaten the robustness of the whole examination system. As such, we believe it to be an urgent matter, requiring a thorough investigation.

Respectfully

John Mountford

These data represent clear evidence of the general inflation of SATs scores compared to Cognitive Ability Test (CAT) scores for the same pupils, especially for those of lower cognitive ability.

Put simply, instead of cramming our children with knowledge for SATs and GCSEs in ways that inhibit cognitive development, our pupils deserve educational experiences of the highest quality that make them cleverer and wiser, as well as more knowledgeable. Ways of achieving this are well established (eg by the EEF), although not specifically recognised by OfSTED or promoted by the DfE.

This leads to the inevitable conclusion that SATs are not fit for purpose and that the DfE, OfSTED, The Sutton Trust and the Social Mobility Foundation have got the ‘attainment gap’ completely wrong.

Advertisements
This entry was posted in Blogs, Uncategorized. Bookmark the permalink.

1 Response to It is the attainment gap fallacy that is damaging the life chances of FSM children in the north of England and elsewhere

  1. John Mountford says:

    Whether measured in Yr6, as our northern schools data confirm, or in Yr7, as for our southern schools, measuring progress in achievement between Yrs 6 and 11 using SATs scores is deeply flawed. SATs results are inflated for FSM children in particular, as our research has already established. This is why Academy chains use CATs rather than SATs to drive fair banding admission systems where these exist (there are many reports of some of these schools using this to bar access to more cognitively challenged children, thereby ‘feathering their own nest’ when OfSTED and the DfE come calling).

    We do not only question whether social mobility is a genuine phenomenon. We know, however, it has been hijacked for political purposes. This is why we maintain that teaching for cognitive development (which has the potential to raise general intelligence) is of greater value to students in real terms than teaching to the test. The changes in teaching strategies our project is calling for will inevitably raise academic achievement and support the development of general intelligence, because of the plasticity of the brain under these more favourable learning conditions.

    We are not concerned about measuring progress in intelligence between SATs and GCSEs. We are about trying to level up the playing field. Because the system makes crude comparisons, neglecting the subtleties about intake cohorts, those schools that cannot skew their intake favourably, suffer most. If we cannot bring about the revolution in teaching strategies so crucially need at this time, the least we can do is to campaign to scrap SATs and use CATs in their place. Such a move would at least avoid the duplication in funding at a time when all schools are strapped for cash.

    This quote from Joe Nutt’s blog helps remind us why we are in the business of education. “Great teachers – no matter where they are – are always driven to help children learn, not because they might get into a Russell Group university or land a well-paid job, but because they know learning is the surest route to them becoming free adults, whatever their disadvantages, wherever they started from.”
    https://www.tes.com/news/schools-and-teachers-must-be-allowed-get-their-work-without-being-hamstrung-obsession-social

    In our view, there is no place for the sort of high-stakes, pass/fail testing typified by SATs. Currently, the only reason we are interested in CATs is as an alternative to the mainly meaningless SATs. Our inquiry into the specifics of the spurious ‘standardisation’ process has revealed a worrying level of political interference in the new process. If it is left to the Secretary of State for Education on an annual basis to decide on a ‘pass mark’, that is not acceptable. Statistically, it is a shameful mess which is why we, and a growing chorus of other commentators, declare it unfit for purpose. However, CATs are a handy tool, proven over time to produce useful information about individual potential, not a strategy for raising individual achievement.

    Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s