OSS工具-rclone

参考文献

• rclone
• Install

rclone

安装

• 使用一键安装脚本

  sudo -v ; curl https://rclone.org/install.sh | sudo bash

• 使用二进制文件安装

  # 获取二进制压缩包,并解压
  curl -O https://downloads.rclone.org/rclone-current-linux-amd64.zip
  unzip rclone-current-linux-amd64.zip
  cd rclone-*-linux-amd64
  
  # 将rclone命令放入全局命令目录,并赋予权限
  sudo cp rclone /usr/bin/
  sudo chown root:root /usr/bin/rclone
  sudo chmod 755 /usr/bin/rclone
  
  # 安装manpage
  sudo mkdir -p /usr/local/share/man/man1
  sudo cp rclone.1 /usr/local/share/man/man1/
  sudo mandb

配置

• rclone config

• 以配置Minio为例

  ➜  ~ rclone config
  Current remotes:
  
  Name                 Type
  ====                 ====
  aliyun               s3
  aliyun-demo          s3
  minio                s3
  minio-demo           s3
  
  e) Edit existing remote
  n) New remote
  d) Delete remote
  r) Rename remote
  c) Copy remote
  s) Set configuration password
  q) Quit config
  e/n/d/r/c/s/q> n
  
  Enter name for new remote.
  name> test
  
  Option Storage.
  Type of storage to configure.
  Choose a number from below, or type in your own value.
   1 / 1Fichier
     \ (fichier)
   2 / Akamai NetStorage
     \ (netstorage)
   3 / Alias for an existing remote
     \ (alias)
   4 / Amazon Drive
     \ (amazon cloud drive)
   5 / Amazon S3 Compliant Storage Providers including AWS, Alibaba, Ceph, China Mobile, Cloudflare, ArvanCloud, Digital Ocean, Dreamhost, Huawei OBS, IBM COS, IDrive e2, IONOS Cloud, Lyve Cloud, Minio, Netease, RackCorp, Scaleway, SeaweedFS, StackPath, Storj, Tencent COS, Qiniu and Wasabi
     \ (s3)
   6 / Backblaze B2
     \ (b2)
   7 / Better checksums for other remotes
     \ (hasher)
   8 / Box
     \ (box)
   9 / Cache a remote
     \ (cache)
  10 / Citrix Sharefile
     \ (sharefile)
  11 / Combine several remotes into one
     \ (combine)
  12 / Compress a remote
     \ (compress)
  13 / Dropbox
     \ (dropbox)
  14 / Encrypt/Decrypt a remote
     \ (crypt)
  15 / Enterprise File Fabric
     \ (filefabric)
  16 / FTP
     \ (ftp)
  17 / Google Cloud Storage (this is not Google Drive)
     \ (google cloud storage)
  18 / Google Drive
     \ (drive)
  19 / Google Photos
     \ (google photos)
  20 / HTTP
     \ (http)
  21 / Hadoop distributed file system
     \ (hdfs)
  22 / HiDrive
     \ (hidrive)
  23 / In memory object storage system.
     \ (memory)
  24 / Internet Archive
     \ (internetarchive)
  25 / Jottacloud
     \ (jottacloud)
  26 / Koofr, Digi Storage and other Koofr-compatible storage providers
     \ (koofr)
  27 / Local Disk
     \ (local)
  28 / Mail.ru Cloud
     \ (mailru)
  29 / Mega
     \ (mega)
  30 / Microsoft Azure Blob Storage
     \ (azureblob)
  31 / Microsoft OneDrive
     \ (onedrive)
  32 / OpenDrive
     \ (opendrive)
  33 / OpenStack Swift (Rackspace Cloud Files, Memset Memstore, OVH)
     \ (swift)
  34 / Oracle Cloud Infrastructure Object Storage
     \ (oracleobjectstorage)
  35 / Pcloud
     \ (pcloud)
  36 / Put.io
     \ (putio)
  37 / QingCloud Object Storage
     \ (qingstor)
  38 / SMB / CIFS
     \ (smb)
  39 / SSH/SFTP
     \ (sftp)
  40 / Sia Decentralized Cloud
     \ (sia)
  41 / Storj Decentralized Cloud Storage
     \ (storj)
  42 / Sugarsync
     \ (sugarsync)
  43 / Transparently chunk/split large files
     \ (chunker)
  44 / Union merges the contents of several upstream fs
     \ (union)
  45 / Uptobox
     \ (uptobox)
  46 / WebDAV
     \ (webdav)
  47 / Yandex Disk
     \ (yandex)
  48 / Zoho
     \ (zoho)
  49 / premiumize.me
     \ (premiumizeme)
  50 / seafile
     \ (seafile)
  Storage> 5
  
  Option provider.
  Choose your S3 provider.
  Choose a number from below, or type in your own value.
  Press Enter to leave empty.
   1 / Amazon Web Services (AWS) S3
     \ (AWS)
   2 / Alibaba Cloud Object Storage System (OSS) formerly Aliyun
     \ (Alibaba)
   3 / Ceph Object Storage
     \ (Ceph)
   4 / China Mobile Ecloud Elastic Object Storage (EOS)
     \ (ChinaMobile)
   5 / Cloudflare R2 Storage
     \ (Cloudflare)
   6 / Arvan Cloud Object Storage (AOS)
     \ (ArvanCloud)
   7 / Digital Ocean Spaces
     \ (DigitalOcean)
   8 / Dreamhost DreamObjects
     \ (Dreamhost)
   9 / Huawei Object Storage Service
     \ (HuaweiOBS)
  10 / IBM COS S3
     \ (IBMCOS)
  11 / IDrive e2
     \ (IDrive)
  12 / IONOS Cloud
     \ (IONOS)
  13 / Seagate Lyve Cloud
     \ (LyveCloud)
  14 / Minio Object Storage
     \ (Minio)
  15 / Netease Object Storage (NOS)
     \ (Netease)
  16 / RackCorp Object Storage
     \ (RackCorp)
  17 / Scaleway Object Storage
     \ (Scaleway)
  18 / SeaweedFS S3
     \ (SeaweedFS)
  19 / StackPath Object Storage
     \ (StackPath)
  20 / Storj (S3 Compatible Gateway)
     \ (Storj)
  21 / Tencent Cloud Object Storage (COS)
     \ (TencentCOS)
  22 / Wasabi Object Storage
     \ (Wasabi)
  23 / Qiniu Object Storage (Kodo)
     \ (Qiniu)
  24 / Any other S3 compatible provider
     \ (Other)
  provider> 14
  
  Option env_auth.
  Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars).
  Only applies if access_key_id and secret_access_key is blank.
  Choose a number from below, or type in your own boolean value (true or false).
  Press Enter for the default (false).
   1 / Enter AWS credentials in the next step.
     \ (false)
   2 / Get AWS credentials from the environment (env vars or IAM).
     \ (true)
  env_auth>
  
  Option access_key_id.
  AWS Access Key ID.
  Leave blank for anonymous access or runtime credentials.
  Enter a value. Press Enter to leave empty.
  access_key_id> minio
  
  Option secret_access_key.
  AWS Secret Access Key (password).
  Leave blank for anonymous access or runtime credentials.
  Enter a value. Press Enter to leave empty.
  secret_access_key> miniostorage
  
  Option region.
  Region to connect to.
  Leave blank if you are using an S3 clone and you don't have a region.
  Choose a number from below, or type in your own value.
  Press Enter to leave empty.
     / Use this if unsure.
   1 | Will use v4 signatures and an empty region.
     \ ()
     / Use this only if v4 signatures don't work.
   2 | E.g. pre Jewel/v10 CEPH.
     \ (other-v2-signature)
  region>
  
  Option endpoint.
  Endpoint for S3 API.
  Required when using an S3 clone.
  Enter a value. Press Enter to leave empty.
  endpoint> http://192.168.253.246:9000
  
  Option location_constraint.
  Location constraint - must be set to match the Region.
  Leave blank if not sure. Used when creating buckets only.
  Enter a value. Press Enter to leave empty.
  location_constraint>
  
  Option acl.
  Canned ACL used when creating buckets and storing or copying objects.
  This ACL is used for creating objects and if bucket_acl isn't set, for creating buckets too.
  For more info visit https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl
  Note that this ACL is applied when server-side copying objects as S3
  doesn't copy the ACL from the source but rather writes a fresh one.
  Choose a number from below, or type in your own value.
  Press Enter to leave empty.
     / Owner gets FULL_CONTROL.
   1 | No one else has access rights (default).
     \ (private)
     / Owner gets FULL_CONTROL.
   2 | The AllUsers group gets READ access.
     \ (public-read)
     / Owner gets FULL_CONTROL.
   3 | The AllUsers group gets READ and WRITE access.
     | Granting this on a bucket is generally not recommended.
     \ (public-read-write)
     / Owner gets FULL_CONTROL.
   4 | The AuthenticatedUsers group gets READ access.
     \ (authenticated-read)
     / Object owner gets FULL_CONTROL.
   5 | Bucket owner gets READ access.
     | If you specify this canned ACL when creating a bucket, Amazon S3 ignores it.
     \ (bucket-owner-read)
     / Both the object owner and the bucket owner get FULL_CONTROL over the object.
   6 | If you specify this canned ACL when creating a bucket, Amazon S3 ignores it.
     \ (bucket-owner-full-control)
  acl>
  
  Option server_side_encryption.
  The server-side encryption algorithm used when storing this object in S3.
  Choose a number from below, or type in your own value.
  Press Enter to leave empty.
   1 / None
     \ ()
   2 / AES256
     \ (AES256)
   3 / aws:kms
     \ (aws:kms)
  server_side_encryption>
  
  Option sse_kms_key_id.
  If using KMS ID you must provide the ARN of Key.
  Choose a number from below, or type in your own value.
  Press Enter to leave empty.
   1 / None
     \ ()
   2 / arn:aws:kms:*
     \ (arn:aws:kms:us-east-1:*)
  sse_kms_key_id>
  
  Edit advanced config?
  y) Yes
  n) No (default)
  y/n>
  
  Configuration complete.
  Options:
  - type: s3
  - provider: Minio
  - access_key_id: minio
  - secret_access_key: miniostorage
  - endpoint: http://192.168.253.246:9000
  Keep this "test" remote?
  y) Yes this is OK (default)
  e) Edit this remote
  d) Delete this remote
  y/e/d>   

配置文件

查看配置文件内容

➜  ~ rclone config show
[test]
type = s3
provider = Minio
access_key_id = minio
secret_access_key = miniostorage
endpoint = http://192.168.253.246:9000

查看配置文件位置

➜  ~ rclone config file
Configuration file is stored at:
/home/test/.config/rclone/rclone.conf

命令语法

# 本地到网盘
rclone [功能选项] <本地路径> <网盘名称:路径> [参数] [参数] ...

# 网盘到本地
rclone [功能选项] <网盘名称:路径> <本地路径> [参数] [参数] ...

# 网盘到网盘
rclone [功能选项] <网盘名称:路径> <网盘名称:路径> [参数] [参数] ...

• 网络名称即配置名称,即上面的[test]中的test

常用功能选项

• rclone copy - 复制
• rclone move - 移动，如果要在移动后删除空源目录，请加上 --delete-empty-src-dirs 参数
• rclone sync - 同步：将源目录同步到目标目录，只更改目标目录。
• rclone size - 查看网盘文件占用大小。
• rclone delete - 删除路径下的文件内容。
• rclone purge - 删除路径及其所有文件内容。
• rclone mkdir - 创建目录。
• rclone rmdir - 删除目录。
• rclone rmdirs - 删除指定灵境下的空目录。如果加上 --leave-root 参数，则不会删除根目录。
• rclone check - 检查源和目的地址数据是否匹配。
• rclone ls - 列出指定路径下的所有的文件以及文件大小和路径。
• rclone lsl - 比上面多一个显示上传时间。
• rclone lsd 列出指定路径下的目录
• rclone lsf - 列出指定路径下的目录和文件

常用参数

• -n = --dry-run - 测试运行，用来查看 rclone 在实际运行中会进行哪些操作。
• -P = --progress - 显示实时传输进度，500mS 刷新一次，否则默认 1 分钟刷新一次。
• --cache-chunk-size SizeSuffi - 块的大小，默认5M，理论上是越大上传速度越快，同时占用内存也越多。如果设置得太大，可能会导致进程中断。
• --cache-chunk-total-size SizeSuffix - 块可以在本地磁盘上占用的总大小，默认10G。
• --transfers=N - 并行文件数，默认为4。在比较小的内存的VPS上建议调小这个参数，比如128M的小鸡上使用建议设置为1。
• --config string - 指定配置文件路径，string为配置文件路径。
• --ignore-errors - 跳过错误。比如 OneDrive 在传了某些特殊文件后会提示Failed to copy: failed to open source object: malwareDetected: Malware detected，这会导致后续的传输任务被终止掉，此时就可以加上这个参数跳过错误。但需要注意 RCLONE 的退出状态码不会为0。

日志

• rclone 有 4 个级别的日志记录，ERROR，NOTICE，INFO 和 DEBUG。默认情况下，rclone 将生成 ERROR 和 NOTICE 级别消息。

  • -q - rclone将仅生成 ERROR 消息。

  • -v - rclone将生成 ERROR，NOTICE 和 INFO 消息，推荐此项。

  • -vv - rclone 将生成 ERROR，NOTICE，INFO和 DEBUG 消息。

  • --log-level LEVEL - 标志控制日志级别。

输出日志到文件

• 使用 --log-file=FILE 选项，rclone 会将 Error，Info 和 Debug 消息以及标准错误重定向到 FILE，这里的 FILE 是你指定的日志文件路径。

数据同步

rclone sync -v  minio:/data/ aliyun:/demo-minio/ > "./sync.log" 2>&1

• 将本地minio下data目录下的数据同步到aliyun OSS中的demo-minio目录下,并将日志输出到当前目录下的sync.log中