Methodology behind the Guardian University Guide 2023
We use nine measures of performance, covering all stages of the student life cycle, to put together a league table for 66 subjects. We regard each provider of a subject as a department and ask each provider to tell us which of their students count within each department.
Our intention is to indicate how each department is likely to deliver a positive all-round experience to future students and in order to assess this we refer to how past students in the department have fared.
We quantify the resources and staff contact that have been dedicated to past students, we look at the standards of entry and the likelihood that students will be supported to continue their studies, before looking at how likely students are to be satisfied, to exceed expectations of success and to have positive outcomes after completing the course. Bringing these measures together, we get an overall score for each department and rank departments against this.
For comparability, the data we use focusses on full time first degree students. For prospective undergraduates who want to know where institutions rank in relation to one another, the Guardian scores have been averaged for each institution across all subjects to generate an overall university league table.
What’s changed for 2023?
The structure and methodology of the rankings has remained broadly constant since 2008 but, for the first time since 2014, this year’s guide follows a review of the subjects for which we calculate rankings. Managing the discontinuity in subject classifications has led to some exceptional methodology adjustments while other changes have been made to accommodate changing data availability.
Subject changes
The 2022 edition had 54 subjects but this edition has 66. The changes have come about as follows:
31 subjects have not changed whatsoever.
3 subjects have been renamed but otherwise stay the same.
8 subjects have seen changes to their composition, usually with this reflected in their title.
7 subjects have had one or more major area of activity removed to become a subject in its own right. The remaining subject has sometimes had an adjustment to its title.
2 subjects have been amalgamated into one.
2 subjects have been discontinued.
Finally, 16 subjects have been created.
Our handling of the transitions varied by subject and by metric. The 49 subjects that had retained their identity, albeit with some adjustments to their composition, remained connected to their prior year data for value-added scores, meaning that two-year averages were available when needed for these subjects. The newly created subjects were not able to draw on data from previous years.
Because the continuation metric always uses at least two years of data, a broader connection was needed to connect data from previous years. New subjects were associated with the most relevant subject from previous years in order to pick up continuation data from those years. For subjects that had changed, the continuation methodology was adjusted to create multi-year averages that were not weighted by the number of first years in each academic year. This is a temporary change to methodology.
Career prospects data for 2018/19 was commissioned according to the new subject boundaries, though these first needed to be expressed using JACS codes instead of HECOS. Data was again aggregated across years in an unweighted manner but this is intended as a permanent design feature rather than as a temporary means of handling the subject transition.
Tariff scores were not affected by the transition as they do not use multi-year averages. SSRs operate at the cost centre level and were mainly unaffected by the changes to subject boundaries, though the amalgamation of subject pairs that have distinct cost centres did necessitate the creation of two aggregated cost centres. NSS data picked up the new subject mappings for all years and no extraordinary adjustments were needed.
Career Prospects
After delays prevented the 2022 edition from using the data for the 2018/19 academic year, the data for 2019/20 was released in time for this year’s edition. This enabled us to use two years together, though care was taken to make sure that the results were averaged in an unweighted manner rather than as a simple aggregation. This was because there were some signals that the graduating cohort of 2018/19 were more adversely affected by the pandemic than the 2019/20 cohort were.
Expenditure
The spend per student metric depends on information about how much each university spends on academic services and on subject delivery. Release of this information was delayed until 14 July 2022 which was too late for the data to be processed. Data from 2019/20 was used for a second year.
Average Entry Tariff
Our average entry tariff metric takes the tariff points held by young entrants and divides them by the number of young entrants with non-zero tariffs. In previous years there was an additional rule that the majority of entrants with tariffable qualifications had to be young (under 21) on entry. This year we have removed this rule but replaced it with extra conditions concerning the number of young tariff-bearing entrants relative to the total size of the department.
Value Added
The value added scores work by assigning each qualifying student a probability of being awarded a 1st or a 2:1 and then using this likelihood to produce a value added score if they are successful. When the continuation metric was introduced it used a similar concept, but was more sophisticated in how it assigned probabilities, often taking exact grades into account along with the age of the graduate when they started their course. This more sophisticated approach was transplanted into the value added score methodology for this year’s guide but the data received from HESA/JISC failed to differentiate BTEC and non-BTEC students when assigning probabilities. The results used are correct and meaningful but the additional enhancement to recognise BTEC students has been postponed to next year.
What are the metrics?
Entry standards
This measure seeks to approximate the aptitude of fellow students who a prospective student can anticipate and reports the observed average grades of students joining the department – not the conditions of admission to the course that may be advertised. Average tariffs are determined by taking the total tariff points of first-year, first-degree, full-time entrants who were aged under 21 at the start of their course, if the qualifications that they entered with could all be expressed using the tariff system devised by UCAS. There must be more than seven students in any meaningful average and only students entering year 1 of a course (not a foundation year) with certain types of qualification are included.
This metric contributes 15% to the total score of a department, and refers to those who entered the department in 2020/21.
Student-staff ratios
Student-staff ratios seek to approximate the levels of staff contact that a student can expect to receive by dividing the volume of students who are taking modules in a subject by the volume of staff who are available to teach it. Thus a low ratio is treated positively – it indicates that more staff contact could be anticipated.
Staff and students are reported on a ‘full time equivalent’ basis and research-only staff are excluded from the staff volume. Students on placement or on a course that is franchised to another provider have their volume discounted accordingly.
At least 28 students and three staff (both FTE) must be present in an SSR calculation using 2020/21 data alone. Smaller departments that had at least seven student and two staff FTE in 2019/20, and at least 30 student FTE in total across 2019/20 and 2020/21, have a two-year average calculated.
This metric contributes 15% to the total score of a department. It is released at HESA cost centre level, and we map each cost centre to one or more of our subjects.
Expenditure per student
In order to approximate the level of resources that a student could expect to have dedicated to their provision, we look at the total expenditure in each subject area and divide it by the volume of students taking the subject. We exclude academic staff costs as the benefits of high staff volumes are already captured by the student-staff ratios but recognise that many costs of delivery are centralised: we add the amount of money each provider has spent per students on academic services such as libraries and computing facilities per student, over the past 2 years.
This metric is expressed as points/10 and contributes 5% to the total score of a department. It was not updated in the 2023 guide due to delayed data availability of financial information for 2020/21.
Continuation
Taking a degree-level course is a positive experience for most students but is not suited to everybody and some students struggle and discontinue their studies. Providers can do a lot to support their students – they might promote engagement with studies and with the broader higher education experience - and this measure captures how successful each department is in achieving this. We look at the proportion of students who continue their studies beyond the first year and measure the extent to which this exceeds expectations based on entry qualifications.
To achieve this, we take all first-year students on full-time first-degree courses that are scheduled to take longer than a year to complete and look ahead to the first of December in the following academic year to observe the proportion who are still active in higher education. This proportion is viewed positively, regardless of whether the student has switched course, transferred to a different provider, or been required to repeat their first year – only those who become inactive in the UK’s HE system are counted negatively.
To take the effect of entry qualifications into account we create an index score for each student who has a positive outcome, using their expectation of continuation up to a maximum of 97%. To calculate the score there must have been 35 entrants in the most recent cohort and 65 across the last 2 or 3 years.
This index score, aggregated across the last 2 or 3 years, contributes 15% to the total score of non-medical departments and 5% to those of the medical subjects. However, it is the percentage score – also averaged over 2 or 3 years - that is displayed.
Student satisfaction
The National student survey asks final year students for the extent to which they agree with 27 positive statements about their academic experience of the course and support that they received. Responses are on a 5-point Likert scale (1. definitely disagree to 5. definitely agree) and we take the responses from full-time first-degree students registered at the provider course to produce two statistics: a satisfaction rate and an average response. The satisfaction rate looks across the questions concerned and reports the proportion of responses that were ‘definitely agree’ or ‘mostly agree’ while the average response gives the average Likert score between 1 and 5 that was observed in the responses to those questions.
To assess the teaching quality that a student can expect to experience we took responses from the 2021 and 2022 NSS surveys and aggregated them for the following questions:
Staff are good at explaining things
Staff have made the subject interesting
The course is intellectually stimulating
My course has challenged me to achieve my best work
The overall satisfaction rate for each provider is displayed, and the average response is used with a 8% weighting.
To assess the likelihood that a student will be satisfied with assessment procedures and the feedback they receive we took responses from the 2021 and 2022 NSS surveys and aggregated them for the following questions:
The criteria used in marking have been clear in advance
Marking and assessment has been fair
Feedback on my work has been timely
I have received helpful comments on my work
The overall satisfaction rate for each provider is displayed, and the average response is used with a 8% weighting.
To assess the overall satisfaction of students with their courses we aggregated responses from the 2021 and 2022 NSS surveys for the question “overall, I am satisfied with the quality of the course”.
The overall satisfaction rate for each provider is displayed, and the average response is used with a 4% weighting.
Data was released at the CAH (common aggregation hierarchy) levels of aggregation and we used details of how these map to HECOS (Higher Education classification of subjects) to weight and aggregate results for each of our 66 subjects, prioritising results from the most granular level. Our aggregation rules required that 2022 results are available and that the total number of relevant respondents across the two years was 23 or more.
Value added
In order to assess the extent to which each department will support its students towards achieving good grades, we use value added scores to track students from enrolment to graduation. A student’s chances of getting a good classification of degree (a 1st or a 2:1) are already affected by the qualifications that they start with so our scores take this into account and report the extent to which a student exceeded expectations.
Each full-time student is given a probability of achieving a 1st or 2:1, based on the qualifications that they enter with or, if they have vague entry qualifications, the total percentage of good degrees expected for the student in their department. If they manage to earn a good degree, then they score points that reflect how difficult it was to do so (in fact, they score the reciprocal of the probability of getting a 1st or 2:1). Otherwise they score zero. Students completing an integrated masters award are always regarded as having a positive outcome.
At least 30 students must be in a subject for a meaningful value added score to be calculated using the most recent year of data alone. If there are more than 15 students in both the most recent year and the preceding year, then a two-year average is calculated.
This metric is expressed as points/10 and contributes 15% to the total score of a department.
Career prospects
Using results from the Graduate Outcomes survey for the graduating cohorts of 2018/19 and 2019/20, we seek to assess the extent to which students have taken a positive first step in the 15 months after graduation, in the hope that similar patterns will repeat for future cohorts. We value students that enter graduate level occupations (approximated by SOC groups 1-3: professional, managerial & technical occupations) and students that go on to further study at a professional or HE level and treat these students as positive.
Students report one or more activities, and for each of these give more detail. If students are self-employed or working for an employer, we treat them as positive if the occupation is in SOC groups 1-3, if they have either finished a course or are presently taking one then we look at the level and treat them positively accordingly. Students who have no activity that is regarded positively, but who either reported that they were unable to work, or only partially completed the survey leaving details of an activity incomplete, are excluded from the metric.
The metric refers only to students who graduated from full-time first-degree courses and we only use results if at least 15 students in a department responded in each of the two years or if at least 22.5 students responded in the most recent year. Partial responses are used if the respondent provided details for any of the activities that they reported undertaking. We exclude the responses if, for an activity, we are unable to determine if it should be treated as a positive outcome.
We have always avoided averaging results across years for this metric because the national economic environment that leavers find themselves in can have such a big effect on employment and this is especially true when a pandemic affects the economy. Unfortunately, response rates for the graduate outcomes survey are not high enough to maintain this stance. We therefore average the career prospects statistics across the two years in an unweighted manner, in order to avoid any advantage or disadvantage for a department that had a higher response for a cohort in which economic conditions were better/worse. In situations where only the most recent year of data meets the threshold for usage we have applied the year-on-year sector difference observed for the subject concerned in order to simulate what a 2-year average might have looked like given changing economic conditions.
This metric is worth 15% of the total score in all the non-medical subjects.
How do we use the metric results?
First of all, we determine if a department has enough data to support a ranking. Often individual metrics are missing and we seek to keep the department in the rankings where we can. An institution can only be included in the table if the weighting value of any indicators that are missing add up to 40% or less, and if the institution’s relevant department teaches at least 35 full time first degree students. There must also be at least 25 students (FTE) in the relevant cost centre.
For those institutions that qualify for inclusion in the subject table, each score is compared to the average score achieved by the other institutions that qualify, using standard deviations to gain a normal distribution of standardised scores (S-scores). The standardised score for student: staff ratios is negative, to reflect that low ratios are regarded as better. We cap certain S-scores – extremely high NSS, expenditure and SSR figures – at three standard deviations. This is to prevent a valid but extreme value from exerting an influence that far exceeds that of all other measures.
For metrics in subjects where there are very few datapoints we refer to the distribution of scores observed for a higher aggregation of subjects (CAH1). We also set a minimum standard deviation for each metric and make adjustments to the mean tariff that is referenced by departments with students who entered with Scottish highers or advanced highers.
Although we don’t display anything, we need to plug the gap left in the total score that is left by any missing indicators. We use a substitution process that firstly looks for the corresponding standardised score in the previous year and then, if nothing is available, resorts to looking at whether the missing metric is correlated to general performance in that subject. If it is, the department’s performance in the other metrics is used – effectively assuming that it would have performed as well in the missing metric as it did in everything else. If not, the average score achieved by other providers of the subject is used.
Using the weighting attached to each metric, the standardised scores are weighted and totalled to give an overall departmental score (rescaled to 100) against which the departments are ranked.
The institutional ranking
The institutional table ranks institutions according to their performance in the subject tables but considers two other factors when calculating overall performance.
First, the number of students in a department influences the extent to which that department’s total standardised score contributes to the institution’s overall score. And second, the number of institutions included in the subject table determines the extent to which a department can affect the institutional table.
The number of full-time undergraduates in each subject is expressed as a percentage of the total number of full-time undergraduates counted in subjects for which the institution is included within the subject table. For each subject, the number of institutions included within the table is counted and the natural logarithm of this value is calculated. The total S-Score for each subject – which can be negative or positive – is multiplied by these two values, and the results are summed for all subjects, to give an overall S-score for each institution. Institutions are ranked according to this overall S-score, though the value displayed in the published table is a scaled version of this, that gives the top university 100 points and all the others a smaller (but positive) points tally.
Each institution has overall versions of each of the indicators displayed next to its overall score out of 100, but these are crude institutional averages that are otherwise disconnected from the tables and give no consideration to subject mix. Therefore these institutional averages cannot be used to calculate the overall score or ranking position.
The indicators of performance for value added and for expenditure per student are treated slightly differently, because they need to be converted into points out of 10 before being displayed. Therefore these indicators do read from the subject level tables, again using student numbers to create a weighted average.
Institutions that appear in fewer than eight subject tables are not included in the main ranking of universities.
Course directory
The KIS database of courses, to which institutions provide regular updates to describe courses that students will be able to apply for in future years, is the data source of the courses that we list under each department in each subject group.
We have associated each full-time course with one or more subject groups, based on the subject data associated with the courses. We gave institutions the freedom to adjust these associations with subjects and also to change details of the courses. We include courses that are not at degree level, even though such provision is excluded from the data used to generate scores and rankings.
