Description

A research collaboration between Bournemouth University, University of Dundee, and ITV, supported by XR Network+, resulting in three working prototype tools that repurpose subtitle data for Virtual Production workflows.

XR Subtitle Suite – Repurposing Subtitle Data for Virtual Production

Project partners: Bournemouth University, University of Dundee, ITV
Funded by: XR Network+ (Prototyping, Impact & Acceleration – Round 2)

This research project brought together Bournemouth University and the University of Dundee in collaboration with ITV to explore how subtitle data could be reconceptualised within Virtual Production (VP). Subtitles – typically viewed as an accessibility or translation asset – were reframed as a generative, structural, and editorial resource capable of shaping new forms of media interaction.

The six-month project was supported by XR Network+ and aimed not only to enhance an existing prototype, but to push further into the unexplored creative value of subtitle files. Rather than being static text layers, this project treated subtitles as narrative metadata, offering insight into character, tone, pacing, and scene structure. Through rapid iteration, collaboration with ITV, and iterative evaluation, the research produced a suite of three fully working tools.

Why Subtitles?

Subtitles are available for enormous volumes of broadcast media, yet their use is usually confined to linear display. This project asked a simple but transformative question:

What if subtitles became an input for editing, automation, and creative transformation instead of just an output layer?

By analysing dialogue structure, speaker turns, silence, pacing, sentiment, and contextual markers, subtitles can reveal:

• who is speaking, how often, and with what emotional intent
• how scenes flow, break, accelerate, or centre around conflict
• where meaning is created through timing, rhythm, or contradiction
• opportunities for accessibility improvements, reduction, summarisation, and alternate viewing modes

This shift in perspective opened up new workflows for VP teams, editors, researchers, and audience-facing experiences.

The Three Prototype Tools

Each tool works independently but draws from the same subtitle data foundation. Together, they form a toolkit for segmentation, transformation, and creative media interpretation.

1. Subtitle-Based Media Segmentation

A browser interface that allows users to divide and export media based on:

• Speaker — e.g. all of one character’s dialogue for training, analysis, or character-centred cuts
• Chapters — using structural metadata to isolate scenes or narrative beats
• Time-based ranges — rapidly extracting windows for review, summarisation, or short-form editing

This makes it possible to generate themed edits, accessibility-first versions, or character-focused highlight reels in seconds, rather than manually.

2. PrintedMedia – Graphic Novel & Print-Style Output Generator

This tool transforms video + subtitle data into visual print artefacts:

• comic-style page layouts built from extracted video frames
• speech balloons generated directly from subtitle text
• automatic condensation of scenes into narrative summaries

Designed not only for accessibility but for media literacy, education, and creative reinterpretation, it demonstrates how broadcast content can become visual storytelling material, teaching resource, or archival object.

3. Sentiment & Tone-Driven Stylised Output

A more experimental prototype that uses subtitle data to infer:

• emotional tone (anger, tension, calm, humour)
• character relationships and dynamics
• pacing changes, escalation, conflict

Visual output changes dynamically — applying colour, composition, or motion effects that reflect mood. This tool tests how data-led rendering might one day allow automated editorial stylisation, mood summaries, or emotion-led narrative versions of broadcast media.

Evaluation & Impact

All three tools were evaluated through participant interviews with industry professionals and educators. Key findings included:

• intuitive interfaces with low learning curves
• valuable potential for editorial, archive, and VP workflow integration
• applications for education, accessibility, and short-form content creation

Feedback directly shaped the final refinements, confirming future development value across industry and research contexts.

Future Work

Building on these prototypes, future research with ITV will explore how structured media metadata — including subtitles, scripts, and audio descriptions — can drive automation, adaptive rendering, and enhanced engagement in Virtual Production workflows.

There is clear opportunity to develop:

• adaptive editing pipelines
• real-time narrative visualisation
• archive search + retrieval tools
• accessibility-first content transformations

This project is one of four funded through the XR Network+ PIA initiative, awarding up to £10,000 for UK research exploring new ideas in Virtual Production. Work began in September 2024 and concluded with three publicly demonstrable prototypes.

Image credit: Benjamin Gorman.