Acceleration annotation

Annotation of acceleration data is increasingly important for a wide range of projects. Firetail provides you with a set of tools for manual and machine learning assisted (FireSOM) annotation. You can rapidly annotate

• Movebank individuals
• Movebank tags
• Movebank deployments

Core concepts

Firetail features layers as highest level of organisation. A layer can hold arbitrary many categories. Each category resides on one specific layer. A burst (see also Burst vs. continuous recording) can be assigned to one or more categories.

Add new categories

To add a new category press Add Category above the acceleration window

Assign a name to the category

The category will show up above the acceleration window

Save and Load Annotation Categories

For a proper annotation workflow a common ground set of categories is crucial. With Firetail you can save the current set of categories and layers using File > Annotations > Export Annotation Categories. To add the categories to your project, use File > Annotations > Import Annotation Categories.

Assign a category to a region

Once you have defined categories, you can use the selection mechanism described here to define regions that you’d like to annotate with one of your categories.

1. Select a region within the bursts or the map
2. Press on the category you want to assign this region to

The newly assigned category will be shown in a row/layer above the acceleration data.

Each category can be assigned to a single layer (visual group shown as a single row of data).

Note that overlapping categories shown in a single layer are possible, yet may not be visually sensible.

Right-clicking a category will yield a popup-menu for:

• category modification
• annotation deletion

Saving Annotations and Removing a category

You can remove existing categories via right-clicking on the respective category and select delete category. You are prompted that all annotations contained will be lost. There is currently no undo for this, so make sure to use

File > Annotations > Save Annotations

to save the current state.

Expert hint: the annotation resides in the respective $USER/movebank_downloads/study_<num> folder as annotation2.csv (to ensure compatiblity with AccelerationViewer, this will likely change in future versions). Make sure to include this folder in you regular backup procedure

FireSOM: Machine-learning assisted acceleration segmentation

IMPORTANT: Firetail’s annotation features are activated when loading data via Download Movebank by ID/Tag/Deployment. Your annotation is saved locally (on your machine) along with the downloaded data.

When working with acceleration data manual annotation of large-scale data may be time-consuming or near-impossible. Via Data > Acc: FireSOM Segmentation you can trigger Firetail’s automated segmentation algorithm FireSOM.

Below, we provide a rough overview of the FireSOM procedure:

Overview of FireSOM

1. calculate a set of user-selected features for each burst
2. normalise (standardization) extracted features
3. train the FireSOM (self-organizing map) using the bursts mapped into feature space
4. assign each burst to a category

The model dimension determines the number of predicted categories. For a 4x5 sized map you’ll retrieve 20 categories.

Video Tutorial

Here you can find a video tutorial on FireSOM, see below for more details

Parameters

Model Constraints

Width: the number of horizontal nodes, we suggest to use at least 2 depending on the complexity of data patterns

Height: the number of vertical nodes, we suggest to use at least 3 depending on the complexity of data patterns

Annotation Prefix: Each predicted category will be prefixed with this string

Clear existing Annotations with the same Prefix: Any previous annotation with this prefix will be removed. Consider renaming categories to avoid data loss

Feature Selection

Use Axis Statistics: add mean, median, and quantile statistics to the set of features, each axis Use ODBA: add overall dynamic body acceleration as a feature, all axes Use axis fold-changes: add statistics over log-2 fold-changes for selected pairs of axes mean-corrected norm: add the mean-corrected norm value for each burst continuity: add a measure of sum of changes

Click OK to train a model and see your results as new layers in the acceleration window.

Inspecting the model

While it is easy to train some model in Firetail you should not expect that your initial model will explain your data perfectly. Inspecting a predicted model is crucial. Luckily, Firetail provides you with a lot of context that makes it easier to interpret the predictions.

Overall, the model is based on the similarity of two bursts in feature space, so similar patterns should be assigned to identical or nearby categories. Neighborhood is defined in a tabular row-column sense, i.e node (1,4) is a direct neighbor of (1,3), (1,5), (0,4) and so on.

Think of patterns in terms of activities like rest, running or feeding. The more categories you predict the more behavioural complexity can be detected. For increasingly many categories it may become necessary to join predictions via layers to group similar categories.

Start simple

Therefore, start by training a simple model with dimension 3 by 3 and few features. You can easily train a quick model as a preview or even experiment with how well a single feature works for discrimination of visual patterns or regions of known behaviour.

Increase complexity if required

An increasing number of features means there is more freedom to discriminate bursts. The result model will be harder to interpret though. Incrementally increase model complexity and add features when two regions you deem dissimilar are placed in the same category.

Refine using local models

If the data features local specificities it makes sense to run the segmentation on a selection.

Make sure that at least a few hundred bursts are selected to avoid a poorly fitted model.

In this context, the option Restrict to Selection must be enabled. The algorithm will predict categories only for selected bursts. Choose a distinct model prefix to make sure to maintain both global and local predictions.

Refine specific categories

Right clicking on an annotation layer allows you to select annotatated regions by choosing “Annotations to Selections”.

This leads to a powerful workflow:

1. select regions that could not be segmented properly by the initial, coarse model
2. make sure that the regions to be refined are selected
3. re-run FireSOM using Restrict to Selection and a new model prefix
4. evaluate the local model via merge/delete semantics as discussed below

Assign, Rename and group

The machine-assigned group names cannot magically predict “running”, “feeding” or “flying”. Screen the predicted categories visually and in their best possible context to assign a category to a possibly preliminary behaviour. Repeated

• application of the segmentation
• changes in parameters and
• successive joining of categories into layers
• refinement of coarse categories
• deletion of superfluous categories

will lead to sensible annotations for a complete dataset.

Saving a model

You can save a trained model for later use on another dataset or selection. This provides a very powerful way to avoid the inclusion artifacts or to build more specific models. The core idea is to sort bursts into buckets defined in another context. Therefore, this strategy should work best on similar input data, although there is no technical constraint keeping the user from applying models cross-species or across tag-types.

A common use case is to save global and refined models for one dataset (see refinement) and then re-apply the models on another dataset.

An appropriate setup could also help to enforce consistency checks and enable in-depth validation.

The key steps are:

1. check Save Model after Training
2. set the required parameters and features
3. press OK to train the model
4. select a file to save the training state

A model is a json file that includes required map-weights, parameter settings and the selection of features. The application of this model on different data requires the same parameters and features.

Using a previously trained model

To apply a pre-trained model:

1. load the model via Load Trained Model
2. optional: choose a new prefix to avoid losing existing annotations
3. note that the feature selection and most parameters will be deactivated
4. press OK

The (selected) bursts are then classified using the pre-defined model.

General

Combine the concepts of

• selecting regions
• selecting by layer content
• merge categories into layers
• delete categories
• train local models
• train global models
• vary parameters
• vary features
• predict locally

to build models specific to your use-case.