Configuration¶
Configuration file¶
Global¶
[global]
user = demo
auth = http://192.168.190.21/auth/v1.0
key = demo
concurrency = 10
# default segment_size is 10MB
segment_size = 10485760
segment_container_prefix = .segment_
recursive = True
preserve_path = True
user¶
User name for obtaining an auth token.
auth¶
URL for obtaining an auth token.
key¶
Key for obtaining an auth token.
concurrency¶
It allows watch folder spawns how many threads to handle the upload jobs in the waiting queue. The default value is10
.
segment_size¶
The segments size for Static Large Object(SLO). The default value is10485760
(10MB
)
segment_container_prefix¶
The prefix string of segment container when gowf creates new segment container, the default value is.segment_
recursive¶
Watch subdirectories for file changes if setTrue
. The default value isFalse
.
preserve_path¶
Constructing an object name (include the relative path) when it is True. The default value is False
For example, if you have a file called woof.txt
, and it is under the folder /<the_folder_you_watched>/dog/staffy/`
, the relative path of the file is dog/staffy/woof.txt
, so the object name in Swift with be `dog/staffy/woof.txt
.
When False
, objects will always be named after the basename of the source file (eg, woof.txt
).
If you have some files have the same name (0ab5.db
, 53ef.db
, … etc) in different subfolders (`312/0ab5
,``435/0ab5``) in below,
└── subfolder └── 38965 ├── 312 │ ├── 0ab5 │ │ └── 0ab5.db │ ├── 53ef │ │ └── 53ef.db │ ├── 9c3a │ │ └── 9c3a.db │ └── f19ed │ └── f19ed.db └── 435 ├── 0ab5 │ └── 0ab5.db ├── 53ef │ └── 53ef.db ├── 9c3a │ └── 9c3a.db └── f19ed └── f19ed.db
When you set recurisive = True
and preserve_path = False
, watch folder tries to overwrite the target object in remote (0ab5.db
) when one of local files is updated (subfolder/38965/312/0ab5/0ab5.db
or subfolder/435/312/0ab5/0ab5.db
). So the remote object will be the one is updated recently, please be aware this use case.
Folder section¶
[/tmp/b1]
storage_policy = Standard-Replica
container = b1RR
# segment_container = b1RR+mysegment
# split with comma
file_patterns = *.txt, *.log
# expired remote object in `expired_after`
# y: year, w: week, d: day, h: hour, m: minute, s: second
expired_after = 60s
# Metadata for objects
metadata = key1:val1, key2:val2
# dedup parameters
anchor = 0
anchor_upper_bound = 536870912
anchor_lower_bound = 1048576
anchor_divider = 128
min_divider = 2
max_multiplier = 2
buffer_read_gate = 536870912
buffer_read_size = 1073741824
storage_policy¶
The option allows you to create containers and segment containers under the specific storage policy. Default value is ` ` (Empty) and it’ll use the default policy that defines in the Swift cluster.
Note:
If the container or segment container is exist and has a different storage policy, the watch folder won’t upload any objects until you correct the policy in config file or use a new container/segment container.You’ll see some errors in below,
2017-08-28 08:33:01,502 - watch-uploader - INFO - Created Container(post_container) testSWF with Standard-Replica policy 2017-08-28 08:33:01,502 - watch-uploader - INFO - Created Container(post_container) testSWF with Standard-Replica policy 2017-08-28 08:33:01,671 - watch-uploader - INFO - Get(stat_container) container (testSWF_seg), policy: Reduced-Redundancy 2017-08-28 08:33:01,671 - watch-uploader - ERROR - Current container(testSWF_seg) policy(Reduced-Redundancy) is mismatch! (Standard-Replica)
container¶
The remote container name.
segment_container¶
The remote segment container name.
file_patterns¶
Only upload files that matched the patterns, split with comma
file_patterns = *.log.gz file_patterns = *.gz,*.zip
expired_after¶
When you set this option in the folder stion, the uploaded files under this folder will expired after the value you set. If you want to keep the upload files forever, please remove this optoin under the folder secion.
# 300 seconds expired_after = 300s # 5 minutes expired_after = 5m # 24 hours expired_after = 24h # 90 days expired_after = 90d # 7 years expired_after = 7y # 2 dyas 10 hours expired_after = 2d10h
metadata¶
You can add metadata for each files that you want to upload from this folder.
metadata = key-1:value-1,key-2:value-2
Run¶
# Run with single configuration file
$ gowf -config gowf.conf
$ gowf -config gowf.conf -config gowf-2nd.conf
# run it in debug mode (turn on log-level to debug)
$ gowf -config gowf.conf -log-file log.txt -log-level DEBUG
# run it in debug mode and store statistic report in stats folder.
$ gowf -config gowf.conf \
-log-file log.txt \
-log-level DEBUG \
-stats-folder stats
Help message¶
$ ./gowf -h
Usage of ./gowf:
-config value
A config file. To run multiple config files, please try '-config <config1> -config <config2>'
-dedup
Enable dedup feature for large object
** THIS IS AN EXPERIMENTAL FEATURE, PLEASE CONTACT SWIFTSTACK FIRST **
-log-file string
Log file (default "gowf.log")
-log-level string
Log level (DEBUG or INFO) (default "INFO")
-stats-folder string
Store statistic output in the folder
-update
Update to the latest version
-version
Version