Synchronizer
Synchronizer is a powerful utility with functionality of a basic backup application. It is able to copy entire folders into the cloud and back to a local drive or even between two cloud buckets, providing retention policies and many other options.
The high performance of sync is credited to parallelization of:
listing local directory contents
listing bucket contents
uploads
downloads
Synchronizer spawns threads to perform the operations listed above in parallel to shorten the backup window to a minimum.
Sync Options
Following are the important optional arguments that can be provided while initializing Synchronizer class.
compare_version_mode: When comparing the source and destination files for finding whether to replace them or not, compare_version_mode can be passed to specify the mode of comparison. For possible values seeb2sdk.v2.CompareVersionMode. Default value isb2sdk.v2.CompareVersionMode.MODTIMEcompare_threshold: It’s the minimum size (in bytes)/modification time (in seconds) difference between source and destination files before we assume that it is new and replace.newer_file_mode: To identify whether to skip or replace if source is older. For possible values seeb2sdk.v2.NewerFileSyncMode. If you don’t specify this the sync will raiseb2sdk.v2.exception.DestFileNewerin case any of the source file is older than destination.keep_days_or_delete: specify policy to keep or delete older files. For possible values seeb2sdk.v2.KeepOrDeleteMode. Default is DO_NOTHING.keep_days: if keep_days_or_delete isb2sdk.v2.KeepOrDeleteMode.KEEP_BEFORE_DELETEthen this specifies for how many days should we keep.
>>> from b2sdk.v2 import ScanPoliciesManager
>>> from b2sdk.v2 import parse_folder
>>> from b2sdk.v2 import Synchronizer
>>> from b2sdk.v2 import KeepOrDeleteMode, CompareVersionMode, NewerFileSyncMode
>>> import time
>>> import sys
>>> source = '/home/user1/b2_example'
>>> destination = 'b2://example-mybucket-b2'
>>> source = parse_folder(source, b2_api)
>>> destination = parse_folder(destination, b2_api)
>>> policies_manager = ScanPoliciesManager(exclude_all_symlinks=True)
>>> synchronizer = Synchronizer(
max_workers=10,
policies_manager=policies_manager,
dry_run=False,
allow_empty_source=True,
compare_version_mode=CompareVersionMode.SIZE,
compare_threshold=10,
newer_file_mode=NewerFileSyncMode.REPLACE,
keep_days_or_delete=KeepOrDeleteMode.KEEP_BEFORE_DELETE,
keep_days=10,
)
We have a file (hello.txt) which is present in destination but not on source (my local), so it will be deleted and since our mode is to keep the delete file, it will be hidden for 10 days in bucket.
>>> no_progress = False
>>> with SyncReport(sys.stdout, no_progress) as reporter:
synchronizer.sync_folders(
source_folder=source,
dest_folder=destination,
now_millis=int(round(time.time() * 1000)),
reporter=reporter,
)
upload f1.txt
delete hello.txt (old version)
hide hello.txt
We changed f1.txt and added 1 byte. Since our compare_threshold is 10, it will not do anything.
>>> with SyncReport(sys.stdout, no_progress) as reporter:
synchronizer.sync_folders(
source_folder=source,
dest_folder=destination,
now_millis=int(round(time.time() * 1000)),
reporter=reporter,
)
We changed f1.txt and added more than 10 bytes. Since our compare_threshold is 10, it will replace the file at destination folder.
>>> with SyncReport(sys.stdout, no_progress) as reporter:
synchronizer.sync_folders(
source_folder=source,
dest_folder=destination,
now_millis=int(round(time.time() * 1000)),
reporter=reporter,
)
upload f1.txt
Let’s just delete the file and not keep - keep_days_or_delete = DELETE You can avoid passing keep_days argument in this case because it will be ignored anyways
>>> synchronizer = Synchronizer(
max_workers=10,
policies_manager=policies_manager,
dry_run=False,
allow_empty_source=True,
compare_version_mode=CompareVersionMode.SIZE,
compare_threshold=10, # in bytes
newer_file_mode=NewerFileSyncMode.REPLACE,
keep_days_or_delete=KeepOrDeleteMode.DELETE,
)
>>> with SyncReport(sys.stdout, no_progress) as reporter:
synchronizer.sync_folders(
source_folder=source,
dest_folder=destination,
now_millis=int(round(time.time() * 1000)),
reporter=reporter,
)
delete f1.txt
delete f1.txt (old version)
delete hello.txt (old version)
upload f2.txt
delete hello.txt (hide marker)
As you can see, it deleted f1.txt and it’s older versions (no hide this time) and deleted hello.txt also because now we don’t want the file anymore. also, we added another file f2.txt which gets uploaded.
Now we changed newer_file_mode to SKIP and compare_version_mode to MODTIME. also uploaded a new version of f2.txt to bucket using B2 web.
>>> synchronizer = Synchronizer(
max_workers=10,
policies_manager=policies_manager,
dry_run=False,
allow_empty_source=True,
compare_version_mode=CompareVersionMode.MODTIME,
compare_threshold=10, # in seconds
newer_file_mode=NewerFileSyncMode.SKIP,
keep_days_or_delete=KeepOrDeleteMode.DELETE,
)
>>> with SyncReport(sys.stdout, no_progress) as reporter:
synchronizer.sync_folders(
source_folder=source,
dest_folder=destination,
now_millis=int(round(time.time() * 1000)),
reporter=reporter,
)
As expected, nothing happened, it found a file that was older at source but did not do anything because we skipped.
Now we changed newer_file_mode again to REPLACE and also uploaded a new version of f2.txt to bucket using B2 web.
>>> synchronizer = Synchronizer(
max_workers=10,
policies_manager=policies_manager,
dry_run=False,
allow_empty_source=True,
compare_version_mode=CompareVersionMode.MODTIME,
compare_threshold=10,
newer_file_mode=NewerFileSyncMode.REPLACE,
keep_days_or_delete=KeepOrDeleteMode.DELETE,
)
>>> with SyncReport(sys.stdout, no_progress) as reporter:
synchronizer.sync_folders(
source_folder=source,
dest_folder=destination,
now_millis=int(round(time.time() * 1000)),
reporter=reporter,
)
delete f2.txt (old version)
upload f2.txt
Handling encryption
The Synchronizer object may need EncryptionSetting instances to perform downloads and copies. For this reason, the sync_folder method accepts an EncryptionSettingsProvider, see Server-Side Encryption for further explanation and Sync Encryption Settings Providers for public API.