Internal and external validity of energy social research during the pandemic

In this blog post Dr Mike Fell (UCL Energy Institute) and colleagues argue that those conducting social research in the energy domain should give special consideration to the internal and external validity of their work conducted during this pandemic period. They have co-produced a set of principles that researchers can consider in order to give themselves and research users greater confidence that findings and recommendations will still be applicable in years to come. While their focus is on energy, many of the principles are likely to be applicable for other subjects of social research. 

 

National and global events continuously shape social worlds. But the magnitude, speed, and reach of the changes to our lives right now are of a different order to anything that most people alive today have experienced. Given the scale and rapidity of change, how can we ensure that conclusions drawn from data collected during the pandemic are valid, representative, and generalisable to a post-pandemic world?

While the answer is inherently unknowable there are a number of principles that we believe energy social science researchers can take to help mitigate this uncertainty, and ease future interpretation of research findings. 

We co-produced these principles as energy researchers who represent a variety of disciplinary perspectives and subject interests, and are based in a range of institutions and countries. Our recommendations are likely to be most applicable to researchers employing quantitative research methods. However, we hope that as a set of considerations they will be helpful to a broad range of social science scholars, both in energy and beyond.

Challenges to validity during pandemic

Decisions about validity concern the tradeoffs and priorities of a given research study. A researcher might prioritise internal validity (or “the degree to which a study establishes the cause-and-effect relationship”)  by conducting a randomized controlled laboratory experiment. Artificial laboratory conditions enable strong experimental control, but limit generalisability across diverse, complex real-world situations. A field study, in contrast, might prioritise external validity (or “the generalization of research findings [...] to settings and populations other than those studied”), but surrenders some ability to control and measure variables. In short, external validity is key if findings are to be useful beyond the precise context of the study (as is often the case in applied research), but without internal validity generalisability is anyway meaningless.

We argue that the pandemic merits explicit consideration for validity for a number of reasons. 

  • The response to COVID-19 is a situation far removed from the conditions under which knowledge is ordinarily produced and applied, and questions around the validity of findings generated during this circumstance are inevitable.
  • An important consideration for external validity is how stable findings are over time. While there is always uncertainty about how closely the future will resemble the present, we argue that this uncertainty is now especially high.  
  • While research conducted now can shed light on how the extraordinary measures in place might impact energy use, it is challenging to disentangle these impacts from those that result from measures deployable in the absence of a pandemic.

Many of the principles we set out next could simply be viewed as good research practice, but we think that they merit explicit attention at this pressing moment. 

Principle 1: Capture and report on extra relevant data

We suggest that additional and/or modified variables may need to be collected and reported for studies carried out during or after the COVID-19 pandemic. Given the large number of possible new factors to be taken into account, we suggest researchers take a ‘core and consider’ approach (see full paper for an options table) . 

Policies and restrictions: As in all studies, reporting of contextual factors should encompass date(s), place(s), and duration of data collection. We suggest that this should now be supplemented with information on pandemic-related national and local policies that were in force at the time and place of data collection and any changes in these. This could include levels of restriction of people’s freedom to move around outside the home, including self-imposed precautionary behaviour, and the open/closed status of specific relevant services such as schools and certain businesses. 

Severity of the pandemic: Researchers may consider it to be important for context to give a sense of the severity of the pandemic at a given point in time (including health, social and economic impacts, as relevant). 

Impacts at participant level: A further core consideration is that measures affect individuals and households in diverse ways. This will differ by locality, but could include whether someone is considered a ‘key worker’ (and hence still regularly leaves the home during lockdown) or comes under a high-risk category and has to observe stricter measures. 

Demographics: Standard demographic variables may need amendment depending on the study aims. For instance, employment status can include categories such as being placed on government-subsidised furlough, working reduced hours, or working fully from home. Capturing information on recent (and risk of future) changes in income, employment status, receipt of benefits, or self-reported financial satisfaction may take on greater importance. Factors such as gender and ethnicity are also associated with differential impacts of the pandemic.

Cognitive-Behavioural: We anticipate that COVID-19 response measures will be associated with important changes in behaviour, but also cognitive, affective, and other social and material dimensions. Changes in energy-related behaviours and decision-making might be more apparent and measurable, but pandemic-related shifts in energy-related beliefs, attitudes, emotions, and judgments may be just as important to apprehend. 

Measurement instruments: We suggest the use of standardised approaches to measurement, such as widely used and validated scales employed in regular national surveys. This will minimise construct and instrument validity challenges and the resource-intensive efforts associated with developing new measures.

Principle 2: Adapt research design to the pandemic context

It is important to consider how study design, sample selection and recruitment, data collection methodology, and interpretation of findings may need to adapt to and accommodate the pandemic. Examples of ways to report such considerations are provided in the appendix to the full paper. Particular challenges to look out for include:

  • Spurious correlations, if the pandemic (and policies to mitigate it) affect both the independent and dependent variables of interest. Researchers can anticipate this and test whether the study might be vulnerable.
  • Difficulty in manipulating variables in experimental settings. For example, the salience of risk due to fossil fuel usage may be less malleable if people are preoccupied with other worries.
  • Between and within country comparisons, where COVID-19 impacts and restrictions differ. Researchers should contextualize their research by considering how circumstances might affect their results, or mitigate this by collecting a sample that is relatively homogeneous (e.g. in orders to stay at home) or increasing sample size.

Longitudinal Design: One approach that is likely to be particularly powerful is the introduction of longitudinal elements to study designs. Researchers could consider building replications into their research plan. This can be done by splitting data collection over waves separated by a period of time. Variables that remain constant over this period are likely to be less affected by pandemic response measures than those which show variation. Research groups could consider teaming up to add variables of interest onto the end of each other’s studies, saving on budget and introducing opportunities for new analyses. 

Replication: Another possible approach to demonstrate the robustness of research findings over time could be through attempting to reproduce previous research findings. This could help ‘calibrate’ the more recent research and give some insight into whether or not the domain of interest is more or less impacted by the pandemic (also accepting that failure to reproduce findings is not an unusual occurrence).
 
Mixed Methods: Researchers can also explore the possibility of using more than one method to investigate the same research question. This could help mitigate concern around validity risks related to use of a single method under unusual circumstances (e.g. conducting video interviews while other household members may be able to listen in). This is another area where collaboration between research groups could bring significant additional value.
 
Interpretation: Finally, researchers should pay explicit attention to the extent to which pandemic response measures might have contributed to particular results. Researchers should attempt to communicate and justify their best estimate as to the impact such factors could have had on findings. Transparency in reporting is likely to be key in allowing research users to make informed judgements of their own. 
 
Principle 3: Take opportunities to improve research
 
Employing the principles set out above presents a number of opportunities that go beyond simply mitigating threats to validity.
New insights: The introduction of longitudinal elements can provide important insights on stable and dynamic determinants of energy-relevant outcomes, especially if combined with new contextual, behavioural, and other data that may not previously have been collected. 
 
Natural experiments: Where data suggests that different groups of people have been (or will be) systematically exposed to different conditions as a result of the pandemic, natural experiments could be possible. These fleeting windows of opportunity can provide novel research opportunities and should be considered by energy researchers. 
 
Convergence research:  We already highlighted the possible benefits that could accrue from collaboration with other groups to facilitate replication and support validity, but there is also a wider convergence research opportunity (a way of addressing complex problems through highly integrated interdisciplinary approaches). Such collaboration may also provide a route to adding in important contextual data, for example through matching datasets.
 
Transparency and reproducibility: Responding to validity challenges presented by the COVID-19 crisis is an opportunity for the energy research field to step up and embrace practices around transparency and reproducibility that are now seen as standard practice in other areas of research. It is possible that the need to demonstrate validity at present will result in increased familiarity with tools such as pre-registration, potentially enhancing the overall validity of research in the field. Some resources for researchers working in multidisciplinary areas can be found here.
 
Principle 4: Be aware of ‘Body of Knowledge’ validity
 
In much of the social sciences, knowledge on the most severe and pressing problems is often difficult to create and therefore ‘low-hanging fruits’ can receive excessive coverage. Consider, for instance, the wealth of scholarship on local and urban energy initiatives in the UK versus the relatively thin body of work on energy practices in rural Sub-Saharan Africa. Both issues merit attention but the latter affects over a billion people, many of whom experience relatively severe degrees of energy poverty, and yet hardly registers in terms of volume in relevant energy social science research. 
 
We detect a risk that curtailment of field-based empirical research, especially in regions that face severe energy challenges, will exacerbate existing biases in representation in terms of volume (more desk study over ethnographic research than usual), methodology (more conceptual work over evidence-based research) and regional coverage (less pandemic-impacted areas over more pandemic-impacted areas).
 
Such exacerbation of an existing bias could cloud future accounts and understandings of the true effects of the pandemic on energy research. But it is not inevitable - it is an artefact of choices we make as a community. Informed by recognition of likely biases, we (and our funders) could commit to more even coverage based on relevance to real-world problems. We can work toward a reflexive understanding of our role as a scholarly community at this time of crisis and opportunity.
 
Principle 5:  Continue with good research management
 
While we think the validity challenges we have raised are important, we also recognise that any responses to them must fit within existing research plans, budgets, timelines, labour constraints, and the heightened need for affective care, including researchers’ own wellbeing under personal stress-inducing conditions. Ethical and data protection concerns, while not directly related to validity, must be borne prominently in mind.  For example, researchers should be mindful of the extra burden to participants that introducing additional data collection could bring.
 
Conclusion
 
Many of the principles we have set out can simply be viewed as good research practice, whatever the circumstances. However, we hope they might help provide some consistency in response and, moreover, some reassurance for researchers that they are covering off what are likely to be the most important challenges to validity during this time.
 
We think that these principles can be employed with relatively minimal impact on resources and timescales required for research. We all hope that the period of direct applicability of this paper will be as short as possible, and that measures to control the spread of COVID-19 will soon no longer be needed. Nonetheless, we also think that the considerations we raise here have enduring relevance for energy social science in general, and the potential to contribute to more widespread use of transparent, contextually aware and valid research practices in the long term. 
 
This blog post is based on a pre-print published on 6th May at SocArXiv Papers: Validity of social research during and after the pandemic: challenges, considerations and responses
 
AUTHOR BIO: Michael Fell is a senior research fellow at UCL Energy Institute. His work focuses on social aspects of energy use, in particular people's desire and ability to offer flexibility to electricity systems. He is currently working on two main projects: as researcher co-investigator and work package co-lead on the Energy Revolution Research Consortium (EnergyREV) on smart local energy systems; and as a researcher on the UKRI Centre for Research into Energy Demand Solutions on distributed ledger technology and energy retail markets

 

Full list of Authors:

Michael J. Fell, UCL Energy Institute, University College London, UK. michael.fell@ucl.ac.uk
Laura Pagel, Swiss Center for Affective Sciences, University of Geneva, Switzerland. Laura.Pagel@etu.unige.ch
Chien-fei Chen, Center for Ultra-wide-area Resilient Electrical Energy Transmission Networks (CURENT), University of Tennessee, USA. cchen26@utk.edu
Matthew H. Goldberg, Yale Program on Climate Change Communication, Yale University, USA. matthew.goldberg@yale.edu
Mario Herberz, Department of Psychology and Swiss Center for Affective Sciences, University of Geneva, Switzerland. mario.herberz@unige.ch
Gesche M. Huebner, UCL Energy Institute and Institute for Environmental Design & Engineering, University College London, UK. g.huebner@ucl.ac.uk
Siddharth Sareen, Department of Geography & Centre for Climate and Energy Transformation, University of Bergen, Norway. Siddharth.Sareen@uib.no
Ulf J. J. Hahnel, Department of Psychology and Swiss Center for Affective Sciences, University of Geneva, Switzerland. Ulf.Hahnel@unige.ch