Juergen Wastl is Director of Academic Relations and Consultancy at Digital Science. He previously headed up the Research Information team at the University of Cambridge’s Research Strategy Office and worked for BASF managing BMBF-funded projects internationally.

Briony Fane is a Research Analyst at Digital Science. She has a higher education background, having gained a PhD from City, University of London, and has worked as both a researcher and a research manager.

Bo Alroe has worked with research management and administration since 2004,  and currently as Director of Strategy with Digital Science. Bo is from Aalborg, Denmark, where he studied and currently lives with his family.

Introduction

In this analysis, we have used Dimensions on Google BigQuery to analyse the Impact Case Studies from REF 2014, the first exercise to evaluate the impact of research outside of academia. Impact Case Studies form an important part of all university REF submissions, with each describing both the research and its impact beyond academia. The inclusion of impact as a component of the REF reflects a growing interest in demonstrating the value of academic research in society.

Of the 6,975 Impact Case Studies submitted to REF 2014, 6,737 were included in the publicly available HEFCE dataset. Each case study has a unique digital object identifier (DOIs). We have taken these DOIs and used Dimensions on Google BigQuery to connect the REF Impact Case Studies to SDG categories using Dimensions’ SDG classification scheme launched last year.

Using Google BigQuery to connect external datasets to Dimensions to gain important insights

Thanks to HEFCE’s (now Research England) freely available database of REF research submissions, we were able to merge this with Dimensions. By applying Dimensions’ SDG filter it was easy to ascertain the proportion of research output associated with sustainable development submitted to REF 2014 underpinning the Impact Case Studies. We were also able to determine other trends within this information, including ‘types of impact’ associated with the case studies using publicly available code in the Dimensions BigQuery Lab; these include technological, societal, environmental, political, legal, cultural, health and economic types of impact, as defined in the post-REF report we published with King’s College London and HEFCE (now Research England).

In this case, we were able to leverage Dimensions data by enriching it with additional information in the form of impact case study data via Google BigQuery. This levels the playing field across databases, allowing users to carry out informative analysis and gain a greater depth of understanding of REF Impact data. This methodology provided the basis for the following two analyses.

Prevalence of SDG-related research underpinning Impact Case Studies

By analysing the number of SDG-related research publications from total publications between 1993 to 2013 (the two decades before REF 2014 submission) and the number and percentage submitted to the ‘output’ (REF2) and ‘impact’ (REF3) elements of REF 2014, the data show that SDG-related research was significant before the UN SDG goals were introduced in 2015. We see that proportionately more research with a sustainable development focus was submitted to demonstrate impact in REF 2014 than for research outputs. Interestingly, there was 25% overlap of impact case study research publication references with outputs submitted to REF2 in 2014¹.

Alignment of SDGs within the Dimensions categorisation and REF-related ‘types of impact’

SDGs can be clustered into three overarching pillars: Environmental, Economic and Social. The Stockholm Resilience Institute³ has assembled the SDGs by these overarching groups in a diagram that resembles a wedding cake (with Social labelled Biosphere here). This hierarchical aggregation of the individual goals allows us to better visualise and analyse overall trends, and enables comparisons with the REF Main Panel structure (A-D) to retrieve a high level view of sustainable development goals in REF Impact.

Underpinning research categorised by the HEFCE ‘impact types’ and by the three pillars of SDGs as outlined above nicely validates the outcome of the SDG classification system in Dimensions. Table 2 reveals that 62% of underpinning research classified within SDG13, Climate Action, correlates with HEFCE’s Environmental impact type. 45% of research classified as SDG3 Good Health and Well-Being correlates with HEFCE’s Health impact type, and 47% of research associated with SDG16 Peace, Justice and Strong Institutions correlates with HEFCE’s Societal impact type.

As we have already noted, the time period for REF 2014 precedes the formalisation of the UN SDGs but, as we demonstrated in our last blog, UK research exhibits a high proportion of SDG-tagged publications. This is even more pronounced in the underpinning research accompanying the REF Impact Case Studies, showcasing the existing contribution of research in the context of sustainable development to both ‘Impact’ and ‘Excellence’ in the REF 2014 submission.

Comparing citation averages for SDG-related underpinning research versus non SDG-related underpinning research

REF Impact Case Studies were categorised into eight types by HEFCE (now Research England): Cultural, Economic, Environmental, Health, Legal, Political, Societal and Technological. Figure 2 outlines the citation averages for underpinning research publications categorised by impact type that have been SDG-tagged in Dimensions, versus those not SDG-tagged. In particular, we see that citation averages are twice as high for SDG-related publications associated with Economic and Environmental impact types than those that are not SDG-related. Conversely, citation averages for underpinning research outputs associated with Technological and Cultural impact types fall below the citation average of those outputs that are not associated with SDGs. It could be that, for both Environmental and Economic impact types, there is an existing awareness of SDG-related sustainable development thinking, particularly environmental impact; we know that climate change didn’t emerge in 2015 with the introduction of the SDGs. For research related to Technological impact, the focus may have been on more non-sustainable development related technologies.

Looking ahead, we might expect that research underpinning Impact Case Studies in REF 2021 will see a notable increase in SDG-related publications as we continue the UN’s decade of action for realising the sustainable development goals.

Weaving the four REF Main Panels into the story provides us with a slightly different insight into the contribution that the overarching REF Panels made to sustainable development research in REF 2014, as seen in Figure 3:

• Main Panel C (social sciences) contributes most and is evident in all SDGs across all HEFCE impact types. This Panel contains the greatest quantity of underpinning research publications associated with Impact Case Studies overall and confirms the contribution of this Panel to REF2 outputs
• Unsurprisingly, Main Panel A (medicine, health and life sciences) contributes predominantly through SDG3 Health and Well-Being, and across to HEFCE’s Health impact type related Impact Case Studies
• Main Panel B (physical sciences, engineering and maths) mostly contributes to SDG7 Affordable and Clean Energy, and SDG13 Climate Action through to Technology and Environmental ‘impact types’
• Finally, Main Panel D (arts and humanities) which accommodates a more diverse portfolio of underpinning research associated with SDGs, but dominated by SDG16 associated research, does not solely contribute to cultural impact type categorised case studies as might be expected

Research in the context of sustainable development (using Dimensions SDG Classification scheme) acts as a pivot between HEFCE’s ‘Impact Type’ and the four REF Main Panels, showcasing the diverse interaction between the underpinning research in the Impact Case Studies component of REF 2014, its discipline and contribution to the UN SDGs, even before these were formally implemented.

Conclusion

This blog has revealed a starting point for us looking at the scope for interpreting SDGs through the lens of Impact. Utilising the reference lists submitted with the case studies provided us with a context within which to explore research associated with sustainable development in REF 2014, ahead of the formalisation of the UN SDGs in 2015.

Using Google BigQuery allows us to draw interesting conclusions about the impact of research submitted to REF 2014. By integrating Dimensions data with a relevant external dataset we have been able to showcase the potential for future application in developing insights on and beyond sustainable development with impact case studies from the forthcoming REF 2021.

Footnotes

1: The document can be viewed here

2: Each underpinning research publication may be associated with more than one SDG classification. The total number of publications we are working with is 21,829 but, with the overlap of SDG- and UoA-associations, these 21,829 publications are represented 32,684 times

3: The SDG ‘wedding cake’, developed at the Stockholm Resilience Centre, Stockholm University by the Centre’s science director Carl Folke

The post In search of SDGs in REF Impact Case Studies appeared first on Digital Science.

We support UK HEIs

Our diverse yet specialised portfolio companies can support UK HEIs by collating evidence for impact and excellence.

Dimensions

Find out more

Symplectic

Find out more

Altmetric

Find out more

Consultancy

Find out more

An overview of how we can help with your REF preparations

Plan Ahead With Our Webinars

For more information, get in touch

Juergen Wastl | Director of Academic Relations and Consultancy

info@digital-science.com

The post REF 2021 appeared first on Digital Science.

Research assessment will continue to include a significant impact component for funders both in the European Union and the United Kingdom. Universities – and research institutes – will need to be able to articulate social and economic impact effectively and comprehensively both for such assessment and as part of wider stakeholder engagement. This research will provide a platform for stakeholders to share their experience of capturing evidence of research impact across all subjects and disciplines covering:

• Disciplinary experiences
• Impact types
• Effective impact evidence types

The invite-only, consultation workshop will be facilitated by Vertigo Ventures, with members of the Main Panels and impact managers who had hands on experience in REF2014.

To share your views click here to complete a brief survey by 17th March 2016: www.smartsurvey.co.uk/s/7BSKC/

This post originally appeared here. 

The post HEFCE Commissions Research on Impact Evidence appeared first on Digital Science.

The UK’s Research Excellence Framework (REF2014) produced a fantastic new database of evidence about the impact of university research. In the past we have had data on research funding, training and publications. Now we can add to that the impact of research on the economy, society, the environment, health, education and across the arts.

REF2014 included a total of 6,975 case studies capturing the work of 50,000 researchers working in 154 institutions and grouped into 36 disciplinary units of assessment (UOAs) in four overarching subject panels: life sciences; engineering and physical sciences; social sciences; and arts and humanities. Each four page study included the underpinning research, details of the impact, and corroborative materials.

In 2014, the Higher Education Funding Council for England (HEFCE, which runs the REF) commissioned Digital Science to turn these case studies into a searchable database and website. We worked with the Policy Institute at King’s College London to analyse the contents and King’s produced a great overview report delving into that.

It’s a lot of data. How do you work out what’s there? Each case study focusses on priority areas in science, policy and society – but what areas? How closely are they related? Digital Science started with a detailed text analysis and then compared pairs of documents: on the website you can find click to ‘View similar case studies’.

That’s fine, but what does the network of similarity look like? The online version (accessible here) shows you the knowledge clusters, it identifies the main subject panels of of the REF that the work is drawn from, and it also allows you to go straight to each case study in a cluster to look at the research. This makes it very easy to get both the big picture and the detail that it contains.

The UOA categories are too broad to tell you much detail about the research underlying the case studies, so we also looked at the text from a different angle to see which specific Fields of Research (FoRs) the researchers were drawing on. We assigned up to three FoRs to a case study because we found that most case studies crossed disciplinary boundaries. Impactful research is usually also interdisciplinary research, sometimes on a grand scale.

The case studies are linked by FoRs they have in common, this creates a second network which we can visualise and which allows people to quickly see how research fields link to one another. The way in which the fields link up in our visualisation looks remarkably like the ultimate key to human innovation – the brain! So we have called these networks Digital Science BrainScans, and we can create a BrainScan for each university and college that makes up the underlying database.

Click on the image for the interactive visualisation:

The post Visualising Research Impact appeared first on Digital Science.

Laura Wheeler, Community Manager at Digital Science will be hosting our second webinar, “Articulating Research Impact – Strategies From Around The Globe”. Ben McLeish from Altmetric will be questioning and moderating the panel consisting of: Jonathan Adams, Digital Science’s Chief Scientist, Stacy Konkiel, Research Metrics Consultant at Altmetric and Daniel S. Katz, Senior Fellow in the Computation Institute (CI) at the University of Chicago.

The webinar will be taking place on Wednesday 24th June at 4pm BST / 11am ET. If you wish to tweet about the webinar the hashtag to use will be #DSwebinar.

Research outcomes are diverse, complex and realised over a wide spectrum of time. High quality research, whether basic or applied, delivers economic, social and health benefits, and many in very unexpected and complex ways. This is not always easy to communicate and articulate.

Jonathan Adams and his Consultancy Group in Digital Science have recently created a searchable database of impact case studies, with descriptive metadata, from the 2014 Research Exercise Framework (REF) in the United Kingdom. He is therefore very well placed to be able to give an overview of the different types of impact coming out of the research from one of the world’s most established and high quality academies.

Jonathan will share some interesting and diverse examples of impact from the case studies database, and speak to the expectations that administrators have when it comes to selecting tools to help them collect key indicators of impact, pinpoint that impact and then report on it accurately.

Building upon Jonathan’s insights, Stacy Konkiel will examine specific examples of impact that weren’t submitted in the REF2014 impact case studies, but that can showcase certain important types of impact like the influence of research on public policy, technology commercialization, and more. She’ll discuss how tools like Altmetric can help researchers easily discover where their work is making a difference in academia and beyond.

Daniel S. Katz will discuss how reviewers at the National Science Foundation (USA) consider the “intellectual merit” and “broader impacts” criteria for funding and in particular how metrics might help applicants understand their impacts in these areas.Dan will also talk about how reviewers might use qualitative and quantitative altmetrics data to inform their peer reviews for grant applications. He will address many of the salient questions around this use of metrics, for example, do reviewers take metrics seriously and what types of metrics are of most value to them?

Ben will then wrap up and open up to an audience Q&A. Attendees will be able to submit questions within the webinar or on Twitter via the #DSwebinar hashtag.

REGISTRATION NOW CLOSED

The post Digital Science Webinar: Articulating Research Impact – Strategies from Around the Globe appeared first on Digital Science.