SAVE: Data sources

The LCNF funded Solent Achieving Value from Efficiency (SAVE) project is collecting a range of data on household electricity demand. The data comprises electrical power demand, electrical energy consumption and a range of survey data on a large (n > 3,000) sample of households in the South East of England.

Staff and students at the University of Southampton can apply to use anonymised versions of the data for research purposes. Requests for access should be sent to Dr Ben Anderson in the first instance.

The SAVE sample

The SAVE sample is a stratified random address-based sample of households recruited by our research partner BMG Research from:

  • the County of Hampshire
  • the City of Southampton
  • the City of Portsmouth
  • the Isle of Wight

With some cautions this means that the sample is representative of households across these areas – but not of the UK in general.

BMG initially recruited over 4,000 households in late 2016 but due to attrition this has slowly declined. There are ongoing efforts to refresh and re-recruit new households to replace those that have left. As of May 2018 there were around 3,000 households with active monitoring equipment.

Household attributes

All households were asked to complete a survey during the recruitment process in order to collect information about the dwelling, its occupants and their electricity-using habits. Not all did so and there is ongoing work to complete the remaining ~20% of surveys. The survey was repeated annually with a subset of questions to provide updates on, for example, occupancy and appliance ownership.

In addition a number of Time Use Diaries modelled on recent national diary surveys were conducted with sub-samples of the households for particular research purposes.

Anonymised versions of the survey and time-use data are passed to the University of Southampton by BMG Research for secure storage and archiving as a .xlsx file whenever they are updated.

Surveys:

  • Initial recruitment survey (PDF with annotations)
  • Yearly update survey (subset of recruitment survey, PDF)

Data columns are:

  • household id (to link to other data)
  • a range of variables as specified above

Area level attributes

We have linked a number of neighborhood level indicators to the survey data using the Census LSOA or OA code in which the recruited household was found. As these are potentially disclosive, access to these additional variables is restricted. Examples of these variables include:

Electricity consumption

We are measuring whole-dwelling electricity consumption (Wh) every 15 minutes using the Loop energy monitor. This data is passed to the University of Southampton by the Loop system for secure storage and archiving as .csv.gz files. We receive one file per week (containing n households * 96 * 7 observations) and overall are receiving ~130 Mb of this data per week.

This data has some built-in resilience as the Loop system buffers (stores) up to 30 days of kWh observations in the Loop monitoring device in the dwelling. The device will continue to try to re-send this data after a communications outage but observations over 30 days old will eventually be over-written and lost. As the consumption readings are cumulative it is always possible to calculate the total consumption over a missing data period, but impossible to know exactly when it was consumed during that period.

This dataset can generally be used on a low-end desktop with care and sensible date filtering. We generally use RStudio and RMarkdown for data processing and analysis.

The data columns are:

  • household id (to link to other data)
  • observed time stamp (milliseconds, unix time, UTC)
  • received time stamp (milliseconds, unix time, UTC)
  • energy consumed (Wh) 

Electricity power demand

We are also measuring the instantaneous power (W) demand in these households every 10 seconds using the Loop energy monitor. This data is also passed to the University of Southampton by the Loop system for secure storage and archiving as .csv.gz files. We receive roughly one file per minute (containing n households * 6 observations) and overall are receiving ~11 Gb of this data per week.

There is no redundancy or resilience in this data – if an observation is missing due to communication problems it will never return!

This dataset is extremely large – do not expect to do much with it on a low-end desktop! You might want to investigate HPC services such as Iridis 5 or a commercial provider instead…

The data columns are:

  • household id (to link to other data)
  • observed time stamp (milliseconds, unix time, UTC)
  • received time stamp (milliseconds, unix time, UTC)
  • whole household instantaneous power (W, measured every 10 seconds just prior to sending)

Data analysis examples

These include:

Future archiving and data access

At the end of the SAVE project (June 2019), we plan to submit an anonymised version of all of the above data to the UK Data Service for archiving and future research re-use.

Posted in May, 2018