Schäuffelhut Berger – Software Engineering
DE|EN

← All posts

· Matthias Berger

Keep a Movebank Study on Your Own Computer

“ChatGPT, why does my script download the whole study again?”

If you work with Movebank data in R or Python, you know the pattern: the script starts, asks Movebank for the study, and waits while the whole thing comes down the wire. Change one line of your analysis, run it again, and the same download starts over. For a small study that's a short wait. For a large study with acceleration data it takes much longer, and it puts the same load on Movebank's servers every single time.

The data hardly changed in the meantime. So why not keep a copy?

Movebank mirror in one sentence

movebank-mirror copies Movebank studies to your own computer as plain files: a first run catches up the history in chunks, and later runs only fetch what's new.

It comes as a command-line tool and as a Java library, and it uses the official Movebank API with your own Movebank account. Under the hood it talks to Movebank through movebank-api-client, a Java client for the Movebank REST API that we built as well; if you write Java tools for Movebank yourself, it may save you some work. You can only mirror what your account is allowed to see.

What ends up on your disk

First the metadata, then the data of each tag. The layout is deliberately boring:

<mirror-dir>/
├── 000002911040.json              study metadata: study, sensors, tags,
│                                  deployments, individuals, attribute lists
├── 000002911040-license.json      the licence you accepted for this study
└── 000002911040/                  one directory per study
    └── <tagId>/                   one directory per tag
        ├── state_<sensorTypeId>.json                   how far this tag got
        ├── <sensorTypeId>_<yyyyMMddHHmmss>.csv         catch-up chunks
        └── <sensorTypeId>_update_<yyyyMMddHHmmss>.csv  later updates

No database, no special format: JSON for the metadata and CSV for the events, as Movebank delivers them. Anything that reads CSV can read your mirror.

For a test run we mirrored the open study Galapagos Albatrosses (study 2911040, licence CC0). It has 28 tags with GPS and acceleration data. On disk that came to 56 CSV files, about 115,000 rows and 73 MB.

Getting started

You need Java 21. Build the command-line tool and put it on your PATH:

./gradlew :cli:installDist
export PATH="$PWD/cli/build/install/movebank-mirror/bin:$PATH"
export MOVEBANK_USER=...           # your Movebank username
export MOVEBANK_PASSWORD=...       # or use --password-file, see the README

See what your account can access, without downloading anything:

movebank-mirror metadata --list

Mirror one study and keep it up to date:

movebank-mirror -d /var/lib/mvb sync --study 2911040

Or run it once a day from cron, which refreshes and then exits:

movebank-mirror -d /var/lib/mvb sync --once

Interrupt it. It doesn't mind.

A long first download will get interrupted sooner or later: a laptop lid, a lost connection, a Ctrl-C. Each tag keeps a small state file (one per tag and sensor type) that remembers how far it got. Start the mirror again and it carries on where it stopped. Tags that are already done just get a quick check for updates.

Chunks are written under a temporary name and renamed only when they are complete, so a CSV file in your mirror is never half a download.

Staying current without hammering Movebank

Once a tag has caught up, the mirror switches to update mode: it asks Movebank only for rows that changed since the last pass. Tags without news are asked less and less often (5 minutes, then 15 minutes, 1 hour, 6 hours, 24 hours, and finally 7 days), and any new row resets the clock. A tag that is still sending stays fresh; a tag that stopped transmitting years ago costs almost nothing.

Licences stay with the data

Many Movebank studies ask you to accept their terms before you can download. By default movebank-mirror accepts and writes a record of each acceptance next to the study's metadata (<studyId>-license.json), so you can always see what you agreed to. If you'd rather not accept anything automatically, --license reject makes those studies fail instead. Either way, the study's terms apply to your local copy just as they do on Movebank.

What it doesn't do (yet)

The mirror assumes Movebank studies mostly grow: new rows arrive, old rows stay as they are. Where that's not true, your copy can drift:

  • Metadata is fetched once. A tag deployed later on an existing study, or an edited deployment window, won't show up until you run movebank-mirror metadata again. A weekly cron job does it.
  • Corrections and deletions. If a row is corrected on Movebank, the new version arrives as an update, but the old one stays in its earlier file. Deleted rows stay in your mirror. For studies that are actively curated, delete the study's directory now and then and let it download again.
  • No built-in verification against Movebank's row counts yet.

All of this is on the roadmap. If one of these gaps blocks you, open an issue and it moves up the list.

What's Next?

Files on your disk are nice, but your R scripts talk to the Movebank API, not to a folder. In the next post: a small local server that speaks Movebank's language, so those scripts run against your mirror without changing a line.

Building your own tracking database instead? Our series Basics of Animal Tracking Databases starts with the data model. Questions or a use case we haven't thought of? Get in touch.

About the Author
Matthias Berger has been a key member of the Movebank Development Team since its inception in 2008, contributing to the advancement of animal tracking databases and supporting researchers worldwide.