Media overlays with VideoSync

Firetail allows you to import video and media data for replay in the context of tag sensor and GPS data using the VideoSync module.

Start the module via Data > Sync Video Data.

Video footage (and images) from camera traps can be synchronized with the main timeline to replay and understand location or acceleration data using visual evidence. This can be a powerful way to annotate sections in the data or to evaluate predicted behavioral annotations.

The module will refer to resources as videos reflecting its main use case, but images and audio will typically work fine as well.

Key Features

With the VideoSync module you can

• add video resources
• apply systematic offsets to UTC to video resources
• fine-adjust UTC start timestamps
• add notes for a resource
• save/load/combine video mappings
• query timestamps from the main timeline

Prerequisites

Starting from Firetail 14, duplicate resources are allowed, i.e. you can assign the same resource multiple times and apply different offsets. This can be useful a single resource contains evidence for multiple animals

Linking mpv player

For video replay and to support a wide range of video codecs, Firetail relies on the freely available MPV player (https://www.mpv.io). Installation and linking the mpv player binary (exe) to the VideoSync module requires a few easy steps (no admin rights required):

• Open “Data > Sync Video Data” from the main menu
• follow the link to the MPV Player (https://www.mpv.io)
• download one of the official builds (e.g. zip files), Nightly builds will work well for Win/Mac (Linux: third-party builds available)
• extract it to any location of your choice
• in the VideoSync module, select the path to the mpv binary

Typical Workflow

1. Apply a UTC Offset: adjust the offset to UTC for captured resources

   • choose if the offset is UTC plus X or UTC minus X
   • Pre-select an offset to UTC for the video timestamps (hour/min/second)
   • these can be adjusted to the millisecond-level to correct for systematic errors
   • 0h 0m 0s indicates your footage is UTC timestamped

2. Load Video/Audio resources

   • Add new media files. The selected UTC offset will be applied when loading.
   • Note: Firetail attempts to extract recording timestamps and durations from the files. You can always adjust timestamps/durations manually and save your work (Quick Save)

3. Double-clicking the “Resource Path” will start the video and jump to the corresponding position (UTC Video Start) in the main timeline.

   • the timeline is intitially paused
   • the video will play if the main timeline ‘Play’ is activated
   • Note: the video will only be in sync for ‘1 second equals 1 second’

4. Adjust Offsets:

• Fine adjust timestamps by hovering it in the ‘UTC video start’ timestamp, then:

• Quick Save or Save your adjustments (quick save/export/import)

Manual timestamp edit

Double-click on a timestamp to manually set the UTC start. If the timestamp cannot be validated (invalid format or not a valid date), the cell will flash red and reset its value to the previous setting.

Adding notes to a resource

Double-click items in the ‘notes’ column to add a note for a resource. Notes will be saved along with the video mapping. A mapping is automatically saved when the VideoSync window is closed, but may be discarded when the application is closed. Use ‘Quick Save’ or ‘Save Video Mapping’ to make sure crucial changes are kept.

Clear list

Remove all resources.

Save Video Mapping

Save the current list (JSON format).

Load Video Mapping

Load one or more mappings (JSON format). If multiple video mappings are selected they are stacked in the VideoSync list. This allows for a separation of resources by animal or type in individual files.

Search for Overlapping Media

The “Timestamp” field is updated from the main timeline. Using “Search for overlapping media” will search for a resources that overlap the current timestamp. The first match will start playing with an appropriate offset (paused depending on main time control ‘Play’).

Other matches will be highlighted in case of overlapping resources.

Removing/Duplicating resources

Select one or more videos using standard Shift-select or Control-select mouse patterns. Right-click the selection and choose Duplicate, Remove or Remove all selected. Remove will remove a single item in the list (with confirmation). Remove all selected will remove all selected items (with confirmation). Duplicate will add the current item to the end of the list, e.g. to experiment with other offsets or for other sections of the data.

Event-based resource alignment

A typical way to synchronize tag events and video events is to seek for matching events in either resource first, then shift the resource to the correct offset.

Doing so via timestamps can be tedious in the presence of time offsets.

Therefore, the Firetail way to do this is to

1. open the video to be synchronized
2. navigate to the event of interest in-video
3. click Video Event
4. switch to acceleration/map view
5. navigate to the same event in-data, e.g., some distinct acceleration peak
6. click Data Event
7. Shift Selection will show the offset that will be applied
8. Click Shift Selection by: ... to align the video to the data event

                        ====================================
                        Event-based Resource Synchronization
                        ====================================
                                    
(1)
Video  --------(event)------------
               ^ 
               navigate here, hit [Video Event]
(2)                
Data  -----------------------(event)------------------
                             ^
                             navigate here, hit [Data Event]
(3)                             
Offset         [-------------]
               computed offset
             
(3)
Shift Selection by result:
Video                --------(event)------------
Data  -----------------------(event)------------------

Note that for multiple selected resources, the offset will be applied to all items.

Forward Synchronization

The workflow is centered around the video resources in the sense that the main timeline will update when an event (seek, start, pause) is triggered from the VideoSync module. Vice versa, a seek event will not trigger an update in the Video viewport.

The reason is that the main timeline control is typically too coarse to find specific sequences within a video snippet.

Note that changing the speed of either data or video replay will lead to an offset.

Keyboard shortcuts in mpv viewport

Another advantage of mpv player is its great set of keyboard shortcuts that can pause, seek or even zoom within the current viewport. Below, we list a subset of shortcuts that work within the mpv viewport and that may come in handy for the work with Firetail.