Content from What is Rclone

Last updated on 2025-02-19 | Edit this page

Estimated time: 8 minutes

Overview

Questions

  • What is Rclone?
  • Why might you use this application?
  • How might Rclone help you manage your data?

Objectives

  • Basic understanding of Rclone usage
  • Know where to download the software and documentation
  • Know where to get help and examples

Introduction: What is Rclone?

Rclone is a command line program to manage files on cloud storage. Rclone is primarily used for transferring, syncing, and moving data between local systems and various cloud storage services.

Test Illustration
Test Illustration

What can you do with Rclone?

With Rclone, you can handle your data in several efficient ways:

  • Rclone enables efficient file handling for quickly transferring large or multiple files.
  • It lets you create reliable backups to secure your data.
  • You can perform file transfers seamlessly between your local system and cloud storage.
  • It supports directory synchronization to keep folders identical across locations.
  • It facilitates direct cloud migration by copying files directly between cloud providers.

How do you think you might use Rclone?

Consider the specific challenges you face with file transfers or cloud storage, and share how you might use Rclone to address these issues.

Rclone command syntax

rclone [command] source:source-folder destination:destination-folder

List of Rclone commands: https://rclone.org/commands/

Real-World Scenarios for Using Rclone

Understanding how Rclone works can be easier if you relate it to everyday tasks. Here are a few simple examples:

  • Copy:
    Imagine you have a folder of vacation photos on your laptop and you want to back them up to Google Drive without removing them from your computer. Using rclone copy will duplicate your photos to the cloud, leaving the originals intact.

  • Sync:
    Suppose you’re working on a project both at home and in the office. You keep a copy of your project folder on your computer and a backup in the cloud. With rclone sync, any changes you make locally are automatically mirrored in the cloud folder, keeping them exactly the same.

  • Move:
    After completing a project, you might want to free up space on your computer by archiving files to the cloud. The rclone move command transfers your project files to cloud storage and then deletes them from your local drive.

These examples show how each rclone command can help you manage your files based on your needs.

Key Points

  • Uses for Rclone
  • Where to find versions for specific operating systems
  • Where to get help.

Content from Creating Remote Connections

Last updated on 2025-02-19 | Edit this page

Estimated time: 15 minutes

Overview

Questions

  • What are Rclone remotes?
  • How are they used?

Objectives

  • Understand the concept of a remote or local connection

Creating a remote connection

A remote is any storage location not on your local machine. Use Rclone config to connect to services like Google Drive or Amazon S3. Rclone remotes can be used to transfer and synchronize files between cloud storage services, servers, and machines.

This can be an S3 bucket, Google Drive, Box or even a local machine.

rclone connections
rclone connections

CALLOUT (callout)

By default, rclone uses its own client_id and client_secret for services like Google Workspace and Box. This shared configuration can lead to rate limiting, which may slow down your transfers when many users are accessing these services. To improve performance, consider setting up your own client_id and client_secret. Detailed setup instructions are available in the rclone documentation for Google Drive and the Box documentation.

Key Points

  • Rclone remote and local connections
  • Types of cloud storage services that can be connected

Content from Rclone config command

Last updated on 2025-03-04 | Edit this page

Estimated time: 20 minutes

Overview

Questions

  • Once installed, how is Rclone setup or configured?
  • How to check for and edit existing configurations
  • what is a remote/headless machine

Objectives

  • Use config to create remote connections
  • Knowledge of useful command flags (or ‘options’)
  • Configure a remote connection for a machine with no browser available (ie. headless)

Rclone config command:

The *rclone config” command enteres an interactive configuration session where you can setup new remotes, edit and manage existing ones.

syntax:

Rclone config [flags]

rclone config
sand

BASH 

rclone config

OUTPUT 

output goes here
rclone config edit
rclone config edit

Rclone command flags

There are numerous command flags but these three are especially worth remembering:

  • -n, –dry-run Do a trial run with no permanent changes
  • -i, –interactive Enable interactive mode
  • -v, –verbose count Print lots more stuff (repeat for more) - useful when debugging

[live-code a setting up a simple remote]

Walk through configuring a local machine

Walk through configuring a remote that requires authorization

Configuring rclone on a remote/headless machine

A headless machine is one that has no browser available. For example you are using Windows WSL2 Linux subsystem or you logged into a remote machine (HPC, AWS, etc.) using your termnial client.

Some of the configurations (those involving oauth2) require an Internet connected web browser.

We are going to walk through this but here is an example from the Rclone documentation https://rclone.org/remote_setup/#configuring-using-rclone-authorize.

rclone config with headless machine
rclone config with headless machine

Documentation Specific to Remote

Rclone includes extensive documentation that is specific to particular remotes.

Examples:

Reference

https://rclone.org/remote_setup/

https://rclone.org/commands/rclone_config/

https://rclone.org/flags/#important

Key Points

  • Understand how configuration is used.
  • Familiar with useful command flags: dry-run, interactive and verbose.
  • Use config to create remote connections
  • Know where to get help.

Content from File Transfers, Listing, and Verification

Last updated on 2025-02-10 | Edit this page

Estimated time: 25 minutes

Overview

Questions

  • How are files moved or copied?
  • What does the rclone sync do?
  • How do you see what is already in the destination?

Objectives

  • Understand the difference between copy and sync
  • Be able to list what is already in the destination
  • Command line using Linux vs Windows WSL2 subsystem linux

Moving files around

Rclone is most frequently used to move files, individually or as a group from one place to another.

The syntax for the places, to or from is:

BASH 

source:folder  destination:folder

Moving, syncing and knowing what is already there

Lists contents of a remote:

BASH 

> rclone ls remote:path

Copy local files to remote:

BASH 

> rclone copy /local/path remote:path # copies /local/path to the remote

Sync local files to remote:

BASH 

> rclone sync --interactive /local/path remote:path # syncs /local/path to the remote

Filtering

Rclone commands are often similar to bash commands, such as ls command. However, Rclone files using the –include flag.

Some Examples:

My home backup of local to an external drive (windows using linux subsystem)

BASH 

> rclone copy /mnt/d/work-related /mnt/f/work-related-backup

Copying files from google drive and filtering for ’*.txt”

BASH 

> rclone copy rclone-intro-google:rclone-intro-google rclone-intro-box:rclone-intro --include "*.txt"

Checking a result before running by using the -n flag

BASH 

> rclone sync rclone-intro-google:rclone-intro-google rclone-intro-box:rclone-intro -n

OUTPUT 

2025/02/09 14:52:11 NOTICE: Beans, Snap and Italian – Pieces, Green and Wax - National Center for Home Food Preservation.pdf: Skipped copy as --dry-run is set (size 91.581Ki)
2025/02/09 14:52:11 NOTICE: For Safety's Sake - National Center for Home Food Preservation.pdf: Skipped copy as --dry-run is set (size 69.982Ki)
2025/02/09 14:52:11 NOTICE: Preserving_Food__Using_Pressure_Canners.pdf: Skipped copy as --dry-run is set (size 3.839Mi)
2025/02/09 14:52:11 NOTICE: Selecting the Correct Processing Time - National Center for Home Food Preservation.pdf: Skipped copy as --dry-run is set (size 123.316Ki)
2025/02/09 14:52:11 NOTICE: test01.txt.docx: Skipped copy as --dry-run is set
2025/02/09 14:52:11 NOTICE: USDA-Complete-Guide-to-Home-Canning-2015-revision.pdf: Skipped copy as --dry-run is set (size 16.518Mi)
2025/02/09 14:52:11 NOTICE: Potatoes, Sweet – Pieces or Whole - National Center for Home Food Preservation.pdf: Skipped copy as --dry-run is set (size 87.602Ki)
2025/02/09 14:52:11 NOTICE:
Transferred:       21.079 MiB / 21.079 MiB, 100%, 0 B/s, ETA -
Checks:                 3 / 3, 100%
Transferred:            8 / 8, 100%
Elapsed time:         2.0s

[add 2 more examples]

Different operating systems have slightly different syntax

Windows syntax:

BASH 

rclone ls C:\\Users\\jjamison\\rclone
  • Linux and macOS:

BASH 

rclone copy rclone-intro-box:rclone-intro rclone-intro-google:rclone-intro-google    
rclone copy rclone-intro-box:rclone-intro rclone-intro-google:rclone-intro-google -n

Windows WSL2 Linux subsystem* Linux syntax:

BASH 

rclone ls /mnt/c/Users/jjamison/rclone

Valid remote names

https://rclone.org/docs/#valid-remote-names

Reference:

https://rclone.org/docs/#subcommands

Windows https://rclone.org/docs/#windows

Linux https://rclone.org/docs/#linux-osx

Key Points

  • Difference between Copy and Sync
  • See what is already in the destination

Content from Choosing the Right Rclone Command: Copy, Sync, or Move

Last updated on 2025-02-18 | Edit this page

Estimated time: 25 minutes

Overview

Questions

  • What is the difference between copy, sync and move?
  • When should you choose one command over the others?

Objectives

  • Be able to pick the most appropriate command to mvoe file(s)
  • Determine which command is most appropriate for different file management scenarios.

Copy, Sync, or Move

When managing your files with rclone, you have three primary commands: copy, sync, and move. Each command handles your data differently, so it’s important to know their unique behaviors to choose the right one for your task.

Copy

The copy command duplicates files from the source to the destination. It compares files (by size, modification time, or MD5 checksum) and transfers only those that are new or have changed. Copying does not delete any files at the destination, making it safe when you simply want to update or add files.

Syntax:

BASH 

rclone copy source:sourcepath dest:destpath

Notes:

  • Only the contents of a directory are copied — not the directory itself.
  • Use the copyto command for copying single files.
  • If the destination path does not exist, it will be created.

Sync

The sync command makes the destination an exact mirror of the source. It copies new or updated files and deletes files in the destination that are not present in the source. Use sync when you need both locations to be identical, but be cautious as it can remove files from the destination.

Syntax:

BASH 

rclone sync source:path dest:path

Move

The move command transfers files from the source to the destination and then deletes them from the source after a successful transfer. This is useful when you want to relocate files rather than keep copies in both places.

Important Note: Since this can cause data loss, test first with the –dry-run or the –interactive/-i flag.

Syntax:

BASH 

rclone move source:path dest:path [flags]

Reference: https://rclone.org/commands/rclone_move/

Key Points

  • Different ways to move or copy files: copy, sync, move
  • Understand the difference between copy and sync

Content from Getting Help

Last updated on 2025-02-10 | Edit this page

Estimated time: 15 minutes

Overview

Questions

  • Where do I get help using Rclone?
  • question 2

Objectives

  • Know where to find help and information about Rclone.
  • objective 2

Rcopy has extensive documentation at https://rclone.org/

You can search on config commands and storage names.

Key Points

  • Rclone documentation.
  • keypoint 2