Configuration

Configuration file

Global

[global]
user = demo
auth = http://192.168.190.21/auth/v1.0
key = demo

concurrency = 10

# default segment_size is 10MB
segment_size = 10485760
segment_container_prefix = .segment_

recursive = True
preserve_path = True

user

User name for obtaining an auth token.

auth

URL for obtaining an auth token.

key

Key for obtaining an auth token.

concurrency

It allows watch folder spawns how many threads to handle the upload jobs in the waiting queue. The default value is 10.

segment_size

The segments size for Static Large Object(SLO). The default value is 10485760 (10MB)

segment_container_prefix

The prefix string of segment container when gowf creates new segment container, the default value is .segment_

recursive

Watch subdirectories for file changes if set True. The default value is False.

preserve_path

Constructing an object name (include the relative path) when it is True. The default value is False

For example, if you have a file called woof.txt , and it is under the folder /<the_folder_you_watched>/dog/staffy/`, the relative path of the file is dog/staffy/woof.txt, so the object name in Swift with be `dog/staffy/woof.txt.

When False, objects will always be named after the basename of the source file (eg, woof.txt).

If you have some files have the same name (0ab5.db, 53ef.db, … etc) in different subfolders (`312/0ab5,``435/0ab5``) in below,

└── subfolder
          └── 38965
              ├── 312
              │   ├── 0ab5
              │   │   └── 0ab5.db
              │   ├── 53ef
              │   │   └── 53ef.db
              │   ├── 9c3a
              │   │   └── 9c3a.db
              │   └── f19ed
              │       └── f19ed.db
              └── 435
                  ├── 0ab5
                  │   └── 0ab5.db
                  ├── 53ef
                  │   └── 53ef.db
                  ├── 9c3a
                  │   └── 9c3a.db
                  └── f19ed
                      └── f19ed.db

When you set recurisive = True and preserve_path = False, watch folder tries to overwrite the target object in remote (0ab5.db) when one of local files is updated (subfolder/38965/312/0ab5/0ab5.db or subfolder/435/312/0ab5/0ab5.db). So the remote object will be the one is updated recently, please be aware this use case.

Folder section

[/tmp/b1]
storage_policy = Standard-Replica
container = b1RR
# segment_container = b1RR+mysegment

# split with comma
file_patterns = *.txt, *.log

# expired remote object in `expired_after`
# y: year, w: week, d: day, h: hour, m: minute, s: second
expired_after = 60s

# Metadata for objects
metadata = key1:val1, key2:val2

# dedup parameters
anchor = 0
anchor_upper_bound = 536870912
anchor_lower_bound = 1048576
anchor_divider = 128
min_divider = 2
max_multiplier = 2
buffer_read_gate = 536870912
buffer_read_size = 1073741824

storage_policy

The option allows you to create containers and segment containers under the specific storage policy. Default value is ` ` (Empty) and it’ll use the default policy that defines in the Swift cluster.

Note:

If the container or segment container is exist and has a different storage policy, the watch folder won’t upload any objects until you correct the policy in config file or use a new container/segment container.

You’ll see some errors in below,

2017-08-28 08:33:01,502 - watch-uploader - INFO - Created Container(post_container) testSWF with Standard-Replica policy
2017-08-28 08:33:01,502 - watch-uploader - INFO - Created Container(post_container) testSWF with Standard-Replica policy
2017-08-28 08:33:01,671 - watch-uploader - INFO - Get(stat_container) container (testSWF_seg), policy: Reduced-Redundancy
2017-08-28 08:33:01,671 - watch-uploader - ERROR - Current container(testSWF_seg) policy(Reduced-Redundancy) is mismatch! (Standard-Replica)

container

The remote container name.

segment_container

The remote segment container name.

file_patterns

Only upload files that matched the patterns, split with comma

file_patterns = *.log.gz
file_patterns = *.gz,*.zip

expired_after

When you set this option in the folder stion, the uploaded files under this folder will expired after the value you set. If you want to keep the upload files forever, please remove this optoin under the folder secion.

# 300 seconds
expired_after = 300s

# 5 minutes
expired_after = 5m

# 24 hours
expired_after = 24h

# 90 days
expired_after = 90d

# 7 years
expired_after = 7y

# 2 dyas 10 hours
expired_after = 2d10h

metadata

You can add metadata for each files that you want to upload from this folder.

metadata = key-1:value-1,key-2:value-2

Dedup options (under Folder section)

anchor

anchor_upper_bound

anchor_lower_bound

anchor_divider

min_divider

max_multiplier

buffer_read_gate

buffer_read_size

Run

# Run with single configuration file
$ gowf -config gowf.conf

$ gowf -config gowf.conf -config gowf-2nd.conf

# run it in debug mode (turn on log-level to debug)
$ gowf -config gowf.conf -log-file log.txt -log-level DEBUG

# run it in debug mode and store statistic report in stats folder.
$ gowf -config gowf.conf \
       -log-file log.txt \
       -log-level DEBUG \
       -stats-folder stats

Help message

$ ./gowf -h
Usage of ./gowf:
-config value
    A config file. To run multiple config files, please try '-config <config1> -config <config2>'
-dedup
    Enable dedup feature for large object
    ** THIS IS AN EXPERIMENTAL FEATURE, PLEASE CONTACT SWIFTSTACK FIRST **
-log-file string
    Log file (default "gowf.log")
-log-level string
    Log level (DEBUG or INFO) (default "INFO")
-stats-folder string
    Store statistic output in the folder
-update
    Update to the latest version
-version
    Version