Using computer vision to improve on-screen representation

17 March 2021

Explore the research series

Diversity and Inclusion

Part one: Gaps in the evidence base

Authors: Raphael Leung, Bartolomeo Meletti

From presence to prominence: How can computer vision widen evidence base of on-screen representation

Research Report
Diversity and Inclusion

Part two: A framework for measuring on-screen representation

Authors: Raphael Leung, Bartolomeo Meletti

From presence to prominence: How can computer vision widen evidence base of on-screen representation

Research Report
Diversity and Inclusion

Part three: An illustrative demo measuring prominence of TV characters

Authors: Raphael Leung, Bartolomeo Meletti

From presence to prominence: How can computer vision widen evidence base of on-screen representation

Research Report

A series about improving issues of diversity in the characters that we see on our screens

The UK screen industry has a diversity (specifically, representation) problem which it has found difficult to solve. Progress has been made but structural inequalities remain. 

Representation is commonly defined as how the media portray individuals, social groups, events and issues to an audience. 

Computer vision is an interdisciplinary field that develops techniques to help computers ‘see’ or make sense of the visual world. For example, models can be trained to detect objects like faces in an image or video.

Due to the persistent nature of the diversity issue, it's crucial that we continue to evidence it. In this series of blogs, we explore whether and how computer vision might provide a new method of measuring on-screen representation. In particular:

  • We argue that diversity evaluation needs to consider more than on-screen presence - it should also consider prominence and portrayal. 
  • We discuss the ethics of applying computer vision to study on-screen characters, via a conceptual framework for measuring on-screen representation. We argue that computer vision can be usefully applied to identify character occurrences, rather than demographic classification. 
  • We provide an illustrative example of measuring character prominence using a short video clip.

This blog series is an output of a pilot project conducted by Nesta in partnership with Learning on Screen. The project team consists of Raphael Leung (Principal Data Scientist, Nesta); Bartolomeo Meletti (Education and Research Executive, Learning on Screen); Dr Cath Sleeman (Head of Data Discovery, Nesta); Gabriel A. Hernández (Chief Digital Officer, Learning on Screen); and Gil Toffell (Academic Research Manager, Learning on Screen).  

This output builds upon a presentation that Raphael and Bartolomeo gave at the International Federation of Television Archives (FIAT/IFTA) Conference 2020 and a short paper by Raphael for the IJCAI 2021 AI for Social Good workshop. It is also inspired by PEC research on generating more subtle, nuanced measures of creative diversity using machine learning, e.g. She said more, which analysed gendered pronouns that are male and female in news articles about the creative industries.

Image by Lê Minh