OSS工具-rclone
参考文献
rclone
安装
-
使用一键安装脚本
1
sudo -v ; curl https://rclone.org/install.sh | sudo bash
-
使用二进制文件安装
1
2
3
4
5
6
7
8
9
10
11
12
13
14# 获取二进制压缩包,并解压
curl -O https://downloads.rclone.org/rclone-current-linux-amd64.zip
unzip rclone-current-linux-amd64.zip
cd rclone-*-linux-amd64
# 将rclone命令放入全局命令目录,并赋予权限
sudo cp rclone /usr/bin/
sudo chown root:root /usr/bin/rclone
sudo chmod 755 /usr/bin/rclone
# 安装manpage
sudo mkdir -p /usr/local/share/man/man1
sudo cp rclone.1 /usr/local/share/man/man1/
sudo mandb
配置
-
rclone config
-
以配置
Minio
为例1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298➜ ~ rclone config
Current remotes:
Name Type
==== ====
aliyun s3
aliyun-demo s3
minio s3
minio-demo s3
e) Edit existing remote
n) New remote
d) Delete remote
r) Rename remote
c) Copy remote
s) Set configuration password
q) Quit config
e/n/d/r/c/s/q> n
Enter name for new remote.
name> test
Option Storage.
Type of storage to configure.
Choose a number from below, or type in your own value.
1 / 1Fichier
\ (fichier)
2 / Akamai NetStorage
\ (netstorage)
3 / Alias for an existing remote
\ (alias)
4 / Amazon Drive
\ (amazon cloud drive)
5 / Amazon S3 Compliant Storage Providers including AWS, Alibaba, Ceph, China Mobile, Cloudflare, ArvanCloud, Digital Ocean, Dreamhost, Huawei OBS, IBM COS, IDrive e2, IONOS Cloud, Lyve Cloud, Minio, Netease, RackCorp, Scaleway, SeaweedFS, StackPath, Storj, Tencent COS, Qiniu and Wasabi
\ (s3)
6 / Backblaze B2
\ (b2)
7 / Better checksums for other remotes
\ (hasher)
8 / Box
\ (box)
9 / Cache a remote
\ (cache)
10 / Citrix Sharefile
\ (sharefile)
11 / Combine several remotes into one
\ (combine)
12 / Compress a remote
\ (compress)
13 / Dropbox
\ (dropbox)
14 / Encrypt/Decrypt a remote
\ (crypt)
15 / Enterprise File Fabric
\ (filefabric)
16 / FTP
\ (ftp)
17 / Google Cloud Storage (this is not Google Drive)
\ (google cloud storage)
18 / Google Drive
\ (drive)
19 / Google Photos
\ (google photos)
20 / HTTP
\ (http)
21 / Hadoop distributed file system
\ (hdfs)
22 / HiDrive
\ (hidrive)
23 / In memory object storage system.
\ (memory)
24 / Internet Archive
\ (internetarchive)
25 / Jottacloud
\ (jottacloud)
26 / Koofr, Digi Storage and other Koofr-compatible storage providers
\ (koofr)
27 / Local Disk
\ (local)
28 / Mail.ru Cloud
\ (mailru)
29 / Mega
\ (mega)
30 / Microsoft Azure Blob Storage
\ (azureblob)
31 / Microsoft OneDrive
\ (onedrive)
32 / OpenDrive
\ (opendrive)
33 / OpenStack Swift (Rackspace Cloud Files, Memset Memstore, OVH)
\ (swift)
34 / Oracle Cloud Infrastructure Object Storage
\ (oracleobjectstorage)
35 / Pcloud
\ (pcloud)
36 / Put.io
\ (putio)
37 / QingCloud Object Storage
\ (qingstor)
38 / SMB / CIFS
\ (smb)
39 / SSH/SFTP
\ (sftp)
40 / Sia Decentralized Cloud
\ (sia)
41 / Storj Decentralized Cloud Storage
\ (storj)
42 / Sugarsync
\ (sugarsync)
43 / Transparently chunk/split large files
\ (chunker)
44 / Union merges the contents of several upstream fs
\ (union)
45 / Uptobox
\ (uptobox)
46 / WebDAV
\ (webdav)
47 / Yandex Disk
\ (yandex)
48 / Zoho
\ (zoho)
49 / premiumize.me
\ (premiumizeme)
50 / seafile
\ (seafile)
Storage> 5
Option provider.
Choose your S3 provider.
Choose a number from below, or type in your own value.
Press Enter to leave empty.
1 / Amazon Web Services (AWS) S3
\ (AWS)
2 / Alibaba Cloud Object Storage System (OSS) formerly Aliyun
\ (Alibaba)
3 / Ceph Object Storage
\ (Ceph)
4 / China Mobile Ecloud Elastic Object Storage (EOS)
\ (ChinaMobile)
5 / Cloudflare R2 Storage
\ (Cloudflare)
6 / Arvan Cloud Object Storage (AOS)
\ (ArvanCloud)
7 / Digital Ocean Spaces
\ (DigitalOcean)
8 / Dreamhost DreamObjects
\ (Dreamhost)
9 / Huawei Object Storage Service
\ (HuaweiOBS)
10 / IBM COS S3
\ (IBMCOS)
11 / IDrive e2
\ (IDrive)
12 / IONOS Cloud
\ (IONOS)
13 / Seagate Lyve Cloud
\ (LyveCloud)
14 / Minio Object Storage
\ (Minio)
15 / Netease Object Storage (NOS)
\ (Netease)
16 / RackCorp Object Storage
\ (RackCorp)
17 / Scaleway Object Storage
\ (Scaleway)
18 / SeaweedFS S3
\ (SeaweedFS)
19 / StackPath Object Storage
\ (StackPath)
20 / Storj (S3 Compatible Gateway)
\ (Storj)
21 / Tencent Cloud Object Storage (COS)
\ (TencentCOS)
22 / Wasabi Object Storage
\ (Wasabi)
23 / Qiniu Object Storage (Kodo)
\ (Qiniu)
24 / Any other S3 compatible provider
\ (Other)
provider> 14
Option env_auth.
Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars).
Only applies if access_key_id and secret_access_key is blank.
Choose a number from below, or type in your own boolean value (true or false).
Press Enter for the default (false).
1 / Enter AWS credentials in the next step.
\ (false)
2 / Get AWS credentials from the environment (env vars or IAM).
\ (true)
env_auth>
Option access_key_id.
AWS Access Key ID.
Leave blank for anonymous access or runtime credentials.
Enter a value. Press Enter to leave empty.
access_key_id> minio
Option secret_access_key.
AWS Secret Access Key (password).
Leave blank for anonymous access or runtime credentials.
Enter a value. Press Enter to leave empty.
secret_access_key> miniostorage
Option region.
Region to connect to.
Leave blank if you are using an S3 clone and you don't have a region.
Choose a number from below, or type in your own value.
Press Enter to leave empty.
/ Use this if unsure.
1 | Will use v4 signatures and an empty region.
\ ()
/ Use this only if v4 signatures don't work.
2 | E.g. pre Jewel/v10 CEPH.
\ (other-v2-signature)
region>
Option endpoint.
Endpoint for S3 API.
Required when using an S3 clone.
Enter a value. Press Enter to leave empty.
endpoint> http://192.168.253.246:9000
Option location_constraint.
Location constraint - must be set to match the Region.
Leave blank if not sure. Used when creating buckets only.
Enter a value. Press Enter to leave empty.
location_constraint>
Option acl.
Canned ACL used when creating buckets and storing or copying objects.
This ACL is used for creating objects and if bucket_acl isn't set, for creating buckets too.
For more info visit https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl
Note that this ACL is applied when server-side copying objects as S3
doesn't copy the ACL from the source but rather writes a fresh one.
Choose a number from below, or type in your own value.
Press Enter to leave empty.
/ Owner gets FULL_CONTROL.
1 | No one else has access rights (default).
\ (private)
/ Owner gets FULL_CONTROL.
2 | The AllUsers group gets READ access.
\ (public-read)
/ Owner gets FULL_CONTROL.
3 | The AllUsers group gets READ and WRITE access.
| Granting this on a bucket is generally not recommended.
\ (public-read-write)
/ Owner gets FULL_CONTROL.
4 | The AuthenticatedUsers group gets READ access.
\ (authenticated-read)
/ Object owner gets FULL_CONTROL.
5 | Bucket owner gets READ access.
| If you specify this canned ACL when creating a bucket, Amazon S3 ignores it.
\ (bucket-owner-read)
/ Both the object owner and the bucket owner get FULL_CONTROL over the object.
6 | If you specify this canned ACL when creating a bucket, Amazon S3 ignores it.
\ (bucket-owner-full-control)
acl>
Option server_side_encryption.
The server-side encryption algorithm used when storing this object in S3.
Choose a number from below, or type in your own value.
Press Enter to leave empty.
1 / None
\ ()
2 / AES256
\ (AES256)
3 / aws:kms
\ (aws:kms)
server_side_encryption>
Option sse_kms_key_id.
If using KMS ID you must provide the ARN of Key.
Choose a number from below, or type in your own value.
Press Enter to leave empty.
1 / None
\ ()
2 / arn:aws:kms:*
\ (arn:aws:kms:us-east-1:*)
sse_kms_key_id>
Edit advanced config?
y) Yes
n) No (default)
y/n>
Configuration complete.
Options:
- type: s3
- provider: Minio
- access_key_id: minio
- secret_access_key: miniostorage
- endpoint: http://192.168.253.246:9000
Keep this "test" remote?
y) Yes this is OK (default)
e) Edit this remote
d) Delete this remote
y/e/d>
配置文件
查看配置文件内容
1 | ➜ ~ rclone config show |
查看配置文件位置
1 | ➜ ~ rclone config file |
命令语法
1 | # 本地到网盘 |
- 网络名称即配置名称,即上面的
[test]
中的test
常用功能选项
rclone copy
- 复制rclone move
- 移动,如果要在移动后删除空源目录,请加上--delete-empty-src-dirs
参数rclone sync
- 同步:将源目录同步到目标目录,只更改目标目录。rclone size
- 查看网盘文件占用大小。rclone delete
- 删除路径下的文件内容。rclone purge
- 删除路径及其所有文件内容。rclone mkdir
- 创建目录。rclone rmdir
- 删除目录。rclone rmdirs
- 删除指定灵境下的空目录。如果加上--leave-root
参数,则不会删除根目录。rclone check
- 检查源和目的地址数据是否匹配。rclone ls
- 列出指定路径下的所有的文件以及文件大小和路径。rclone lsl
- 比上面多一个显示上传时间。rclone lsd
列出指定路径下的目录rclone lsf
- 列出指定路径下的目录和文件
常用参数
-n
=--dry-run
- 测试运行,用来查看 rclone 在实际运行中会进行哪些操作。-P
=--progress
- 显示实时传输进度,500mS 刷新一次,否则默认 1 分钟刷新一次。--cache-chunk-size SizeSuffi
- 块的大小,默认5M,理论上是越大上传速度越快,同时占用内存也越多。如果设置得太大,可能会导致进程中断。--cache-chunk-total-size SizeSuffix
- 块可以在本地磁盘上占用的总大小,默认10G。--transfers=N
- 并行文件数,默认为4。在比较小的内存的VPS上建议调小这个参数,比如128M的小鸡上使用建议设置为1。--config string
- 指定配置文件路径,string
为配置文件路径。--ignore-errors
- 跳过错误。比如 OneDrive 在传了某些特殊文件后会提示Failed to copy: failed to open source object: malwareDetected: Malware detected
,这会导致后续的传输任务被终止掉,此时就可以加上这个参数跳过错误。但需要注意 RCLONE 的退出状态码不会为0
。
日志
-
rclone 有 4 个级别的日志记录,
ERROR
,NOTICE
,INFO
和DEBUG
。默认情况下,rclone 将生成ERROR
和NOTICE
级别消息。-
-q
- rclone将仅生成ERROR
消息。 -
-v
- rclone将生成ERROR
,NOTICE
和INFO
消息,推荐此项。 -
-vv
- rclone 将生成ERROR
,NOTICE
,INFO
和DEBUG
消息。 -
--log-level LEVEL
- 标志控制日志级别。
-
输出日志到文件
- 使用
--log-file=FILE
选项,rclone 会将Error
,Info
和Debug
消息以及标准错误重定向到FILE
,这里的FILE
是你指定的日志文件路径。
数据同步
1 | rclone sync -v minio:/data/ aliyun:/demo-minio/ > "./sync.log" 2>&1 |
- 将本地
minio
下data
目录下的数据同步到aliyun OSS
中的demo-minio
目录下,并将日志输出到当前目录下的sync.log
中
本博客所有文章除特别声明外,均采用 CC BY-NC-SA 4.0 许可协议。转载请注明来自 HoleLin's Blog!