Data download & update#

Data location#

If you are in the SEASTERS project, you should have access to a shared sDrive workspace dedicated to the project. If not, you can contact the project’s PIs so they can add you to this workspace. Once in the group, you should see a DR14_UMR5566_SEASTERS shared project. In this folder, the Database subfolder contains the data we want to access with seastersdb. It is only about 5 GB and can thus easily be downloaded to work on it offline.

This Database folder should already be synchronized on the three clusters used in SEASTERS:

  • At CNRM: /cnrm/tropics/commun/DATACOMMUN/SEASTERS

  • On HILO: /data/projects/LOTUS/SEASTERS

  • On Calmip: /tmpdir/desmet/SEASTERS

Note

This is the directory you will need to provide when running seastersdb-locate at installation. You may also download the database locally; in this case, simply provide the corresponding local directory.

Download & update: rclone#

You may download the database manually from your browser. However, depending on your connection and browser, this may take a while and/or you may struggle in case of network instability or interruption. For this reason, we recommend using rclone to connect to sDrive using the WebDAV protocol.

Installation#

If you have root permission, rclone can be installed with:

sudo apt install rclone

Otherwise, it can also be installed inside a Conda environment using:

conda install conda-forge::rclone

Retrieve WebDAV access information to sDrive#

To configure a remote with rclone, we need a link, login and password. Let us retrieve those on sDrive.

  1. Go online and login to your sDrive account

  2. Click “File settings” in the lower left corner

  3. In the WebDAV section: save the link

  4. Click the link below that indicates something to do with 2FA

  5. Create a new app password with a name relating to the machine you will connect from

  6. A pop-up appears: save the provided login and password

Configure rclone#

  1. Run rclone config

  2. Create a new remote (e.g., with name “sdrive”)

  3. Follow to the instructions: you should choose stuff relating to “WebDAV”, and enter the link, login and password you’ve previously saved

  4. No bearer token

  5. No advanced config

Download & update#

We can now use rclone copy to download our database locally, from the sdrive remote we have registered. We recommend the following (changing the paths adequately):

rclone copy sdrive:/DR14_UMR5566_SEASTERS/Database /path/to/SEASTERS --verbose --stats=5s --stats-one-line > rclone.log 2>&1 &

Error

It is possible that your machine badly supports multithreading, leading rclone to raise errors. In this case, simply add the --multi-thread-streams=0 option.

rclone copy copies files from source to destination, skipping identical files. The same command is thus also relevant for updating.