Synchronizer

Synchronizer is a powerful utility with functionality of a basic backup application. It is able to copy entire folders into the cloud and back to a local drive or even between two cloud buckets, providing retention policies and many other options.

The high performance of sync is credited to parallelization of:

  • listing local directory contents

  • listing bucket contents

  • uploads

  • downloads

Synchronizer spawns threads to perform the operations listed above in parallel to shorten the backup window to a minimum.

Sync Options

Following are the important optional arguments that can be provided while initializing Synchronizer class.

  • compare_version_mode: When comparing the source and destination files for finding whether to replace them or not, compare_version_mode can be passed to specify the mode of comparison. For possible values see b2sdk.v2.CompareVersionMode. Default value is b2sdk.v2.CompareVersionMode.MODTIME

  • compare_threshold: It’s the minimum size (in bytes)/modification time (in seconds) difference between source and destination files before we assume that it is new and replace.

  • newer_file_mode: To identify whether to skip or replace if source is older. For possible values see b2sdk.v2.NewerFileSyncMode. If you don’t specify this the sync will raise b2sdk.v2.exception.DestFileNewer in case any of the source file is older than destination.

  • keep_days_or_delete: specify policy to keep or delete older files. For possible values see b2sdk.v2.KeepOrDeleteMode. Default is DO_NOTHING.

  • keep_days: if keep_days_or_delete is b2sdk.v2.KeepOrDeleteMode.KEEP_BEFORE_DELETE then this specifies for how many days should we keep.

>>> from b2sdk.v2 import ScanPoliciesManager
>>> from b2sdk.v2 import parse_folder
>>> from b2sdk.v2 import Synchronizer, SyncReport
>>> from b2sdk.v2 import KeepOrDeleteMode, CompareVersionMode, NewerFileSyncMode
>>> import time
>>> import sys

>>> source = '/home/user1/b2_example'
>>> destination = 'b2://example-mybucket-b2'

>>> source = parse_folder(source, b2_api)
>>> destination = parse_folder(destination, b2_api)

>>> policies_manager = ScanPoliciesManager(exclude_all_symlinks=True)

>>> synchronizer = Synchronizer(
        max_workers=10,
        policies_manager=policies_manager,
        dry_run=False,
        allow_empty_source=True,
        compare_version_mode=CompareVersionMode.SIZE,
        compare_threshold=10,
        newer_file_mode=NewerFileSyncMode.REPLACE,
        keep_days_or_delete=KeepOrDeleteMode.KEEP_BEFORE_DELETE,
        keep_days=10,
    )

We have a file (hello.txt) which is present in destination but not on source (my local), so it will be deleted and since our mode is to keep the delete file, it will be hidden for 10 days in bucket.

>>> no_progress = False
>>> with SyncReport(sys.stdout, no_progress) as reporter:
        synchronizer.sync_folders(
            source_folder=source,
            dest_folder=destination,
            now_millis=int(round(time.time() * 1000)),
            reporter=reporter,
        )
upload f1.txt
delete hello.txt (old version)
hide   hello.txt

We changed f1.txt and added 1 byte. Since our compare_threshold is 10, it will not do anything.

>>> with SyncReport(sys.stdout, no_progress) as reporter:
        synchronizer.sync_folders(
            source_folder=source,
            dest_folder=destination,
            now_millis=int(round(time.time() * 1000)),
            reporter=reporter,
        )

We changed f1.txt and added more than 10 bytes. Since our compare_threshold is 10, it will replace the file at destination folder.

>>> with SyncReport(sys.stdout, no_progress) as reporter:
        synchronizer.sync_folders(
            source_folder=source,
            dest_folder=destination,
            now_millis=int(round(time.time() * 1000)),
            reporter=reporter,
        )
upload f1.txt

Let’s just delete the file and not keep - keep_days_or_delete = DELETE You can avoid passing keep_days argument in this case because it will be ignored anyways

>>> synchronizer = Synchronizer(
        max_workers=10,
        policies_manager=policies_manager,
        dry_run=False,
        allow_empty_source=True,
        compare_version_mode=CompareVersionMode.SIZE,
        compare_threshold=10,  # in bytes
        newer_file_mode=NewerFileSyncMode.REPLACE,
        keep_days_or_delete=KeepOrDeleteMode.DELETE,
    )

>>> with SyncReport(sys.stdout, no_progress) as reporter:
    synchronizer.sync_folders(
        source_folder=source,
        dest_folder=destination,
        now_millis=int(round(time.time() * 1000)),
        reporter=reporter,
    )
delete f1.txt
delete f1.txt (old version)
delete hello.txt (old version)
upload f2.txt
delete hello.txt (hide marker)

As you can see, it deleted f1.txt and it’s older versions (no hide this time) and deleted hello.txt also because now we don’t want the file anymore. also, we added another file f2.txt which gets uploaded.

Now we changed newer_file_mode to SKIP and compare_version_mode to MODTIME. also uploaded a new version of f2.txt to bucket using B2 web.

>>> synchronizer = Synchronizer(
        max_workers=10,
        policies_manager=policies_manager,
        dry_run=False,
        allow_empty_source=True,
        compare_version_mode=CompareVersionMode.MODTIME,
        compare_threshold=10,  # in seconds
        newer_file_mode=NewerFileSyncMode.SKIP,
        keep_days_or_delete=KeepOrDeleteMode.DELETE,
    )
>>> with SyncReport(sys.stdout, no_progress) as reporter:
    synchronizer.sync_folders(
        source_folder=source,
        dest_folder=destination,
        now_millis=int(round(time.time() * 1000)),
        reporter=reporter,
    )

As expected, nothing happened, it found a file that was older at source but did not do anything because we skipped.

Now we changed newer_file_mode again to REPLACE and also uploaded a new version of f2.txt to bucket using B2 web.

>>> synchronizer = Synchronizer(
        max_workers=10,
        policies_manager=policies_manager,
        dry_run=False,
        allow_empty_source=True,
        compare_version_mode=CompareVersionMode.MODTIME,
        compare_threshold=10,
        newer_file_mode=NewerFileSyncMode.REPLACE,
        keep_days_or_delete=KeepOrDeleteMode.DELETE,
    )
>>> with SyncReport(sys.stdout, no_progress) as reporter:
    synchronizer.sync_folders(
        source_folder=source,
        dest_folder=destination,
        now_millis=int(round(time.time() * 1000)),
        reporter=reporter,
    )
delete f2.txt (old version)
upload f2.txt

Handling encryption

The Synchronizer object may need EncryptionSetting instances to perform downloads and copies. For this reason, the sync_folder method accepts an EncryptionSettingsProvider, see Server-Side Encryption for further explanation and Sync Encryption Settings Providers for public API.

Public API classes

class b2sdk.v2.ScanPoliciesManager(exclude_dir_regexes=(), exclude_file_regexes=(), include_file_regexes=(), exclude_all_symlinks=False, exclude_modified_before=None, exclude_modified_after=None, exclude_uploaded_before=None, exclude_uploaded_after=None)[source]

Policy object used when scanning folders, used to decide which files to include in the list of files.

Code that scans through files should at least use should_exclude_file() to decide whether each file should be included; it will check include/exclude patterns for file names, as well as patterns for excluding directories.

Code that scans may optionally use should_exclude_directory() to test whether it can skip a directory completely and not bother listing the files and sub-directories in it.

Parameters:
__init__(exclude_dir_regexes=(), exclude_file_regexes=(), include_file_regexes=(), exclude_all_symlinks=False, exclude_modified_before=None, exclude_modified_after=None, exclude_uploaded_before=None, exclude_uploaded_after=None)[source]
Parameters:

  • exclude_dir_regexes (Iterable[str | Pattern]) – regexes to exclude directories

  • exclude_file_regexes (Iterable[str | Pattern]) – regexes to exclude files

  • include_file_regexes (Iterable[str | Pattern]) – regexes to include files

  • exclude_all_symlinks (bool) – if True, exclude all symlinks

  • exclude_modified_before (Optional[int]) – optionally exclude file versions (both local and b2) modified before (in millis)

  • exclude_modified_after (Optional[int]) – optionally exclude file versions (both local and b2) modified after (in millis)

  • exclude_uploaded_before (Optional[int]) – optionally exclude b2 file versions uploaded before (in millis)

  • exclude_uploaded_after (Optional[int]) – optionally exclude b2 file versions uploaded after (in millis)

The regex matching priority for a given path is: 1) the path is always excluded if it’s dir matches exclude_dir_regexes, if not then 2) the path is always included if it matches include_file_regexes, if not then 3) the path is excluded if it matches exclude_file_regexes, if not then 4) the path is included

should_exclude_relative_path(relative_path)[source]
Parameters:

relative_path (str) –

should_exclude_local_path(local_path)[source]

Whether a local path should be excluded from the scan or not.

This method assumes that the directory holding the path_ has already been checked for exclusion.

Parameters:

local_path (LocalPath) –

should_exclude_b2_file_version(file_version, relative_path)[source]

Whether a b2 file version should be excluded from the scan or not.

This method assumes that the directory holding the path_ has already been checked for exclusion.

Parameters:

  • file_version (FileVersion) –

  • relative_path (str) –

should_exclude_b2_directory(dir_path)[source]

Given the path of a directory, relative to the scan point, decide if all of the files in it should be excluded from the scan.

Parameters:

dir_path (str) –

should_exclude_local_directory(dir_path)[source]

Given the path of a directory, relative to the scan point, decide if all of the files in it should be excluded from the scan.

Parameters:

dir_path (str) –

class b2sdk.v2.Synchronizer(max_workers, policies_manager=<b2sdk._internal.scan.policies.ScanPoliciesManager object>, dry_run=False, allow_empty_source=False, newer_file_mode=NewerFileSyncMode.RAISE_ERROR, keep_days_or_delete=KeepOrDeleteMode.NO_DELETE, compare_version_mode=CompareVersionMode.MODTIME, compare_threshold=None, keep_days=None, sync_policy_manager=<b2sdk._internal.sync.policy_manager.SyncPolicyManager object>, upload_mode=UploadMode.FULL, absolute_minimum_part_size=None)[source]

Copies multiple “files” from source to destination. Optionally deletes or hides destination files that the source does not have.

The synchronizer can copy files:

  • From a B2 bucket to a local destination.

  • From a local source to a B2 bucket.

  • From one B2 bucket to another.

  • Between different folders in the same B2 bucket. It will sync only the latest versions of files.

By default, the synchronizer:

  • Fails when the specified source directory doesn’t exist or is empty. (see allow_empty_source argument)

  • Fails when the source is newer. (see newer_file_mode argument)

  • Doesn’t delete a file if it’s present on the destination but not on the source. (see keep_days_or_delete and keep_days arguments)

  • Compares files based on modification time. (see compare_version_mode and compare_threshold arguments)

Parameters:
__init__(max_workers, policies_manager=<b2sdk._internal.scan.policies.ScanPoliciesManager object>, dry_run=False, allow_empty_source=False, newer_file_mode=NewerFileSyncMode.RAISE_ERROR, keep_days_or_delete=KeepOrDeleteMode.NO_DELETE, compare_version_mode=CompareVersionMode.MODTIME, compare_threshold=None, keep_days=None, sync_policy_manager=<b2sdk._internal.sync.policy_manager.SyncPolicyManager object>, upload_mode=UploadMode.FULL, absolute_minimum_part_size=None)[source]

Initialize synchronizer class and validate arguments

Parameters:

  • max_workers (int) – max number of workers

  • policies_manager – object which decides which files to process

  • dry_run (bool) – test mode, does not actually transfer/delete when enabled

  • allow_empty_source (bool) – if True, do not check whether source folder is empty

  • newer_file_mode (b2sdk.v2.NewerFileSyncMode) – setting which determines handling for destination files newer than on the source

  • keep_days_or_delete (b2sdk.v2.KeepOrDeleteMode) – setting which determines if we should delete or not delete or keep for keep_days

  • compare_version_mode (b2sdk.v2.CompareVersionMode) – how to compare the source and destination files to find new ones

  • compare_threshold (int) – should be greater than 0, default is 0

  • keep_days (int) – if keep_days_or_delete is b2sdk.v2.KeepOrDeleteMode.KEEP_BEFORE_DELETE, then this should be greater than 0

  • sync_policy_manager (SyncPolicyManager) – object which decides what to do with each file (upload, download, delete, copy, hide etc)

  • upload_mode (UploadMode) – determines how file uploads are handled

  • absolute_minimum_part_size (Optional[int]) – minimum file part size for large files

  • sync_policy_manager –

  • upload_mode –

  • absolute_minimum_part_size –

sync_folders(source_folder, dest_folder, now_millis, reporter, encryption_settings_provider=<b2sdk._internal.sync.encryption_provider.ServerDefaultSyncEncryptionSettingsProvider object>)[source]

Syncs two folders. Always ensures that every file in the source is also in the destination. Deletes any file versions in the destination older than history_days.

Parameters:
class b2sdk.v2.SyncReport(stdout, no_progress)[source]

Handle reporting progress for syncing.

Print out each file as it is processed, and puts up a sequence of progress bars.

The progress bars are:

  • Step 1/1: count local files

  • Step 2/2: compare file lists

  • Step 3/3: transfer files

This class is THREAD SAFE, so it can be used from parallel sync threads.

Parameters:
update_compare(delta)[source]

Report that more files have been compared.

Parameters:

delta (int) – number of files compared

end_compare(total_transfer_files, total_transfer_bytes)[source]

Report that the comparison has been finished.

Parameters:

  • total_transfer_files (int) – total number of transferred files

  • total_transfer_bytes (int) – total number of transferred bytes

update_transfer(file_delta, byte_delta)[source]

Update transfer info.

Parameters:

  • file_delta (int) – number of files transferred

  • byte_delta (int) – number of bytes transferred

UPDATE_INTERVAL = 0.1
__init__(stdout, no_progress)
Parameters:

Add a circular symlink error message to the list of warnings.

Parameters:

path (str) – file path

Return type:

None

close()

Perform a clean-up.

end_total()

Total files count is done. Can proceed to step 2.

Return type:

None

error(message)

Print an error, gracefully interleaving it with a progress bar.

Parameters:

message (str) – an error message

Return type:

None

has_errors_or_warnings()

Check if there are any errors or warnings.

Return type:

bool

Returns:

True if there are any errors or warnings

invalid_name(path, error)

Add an invalid filename error message to the list of warnings.

Parameters:

  • path (str) – file path

  • error (str) –

Return type:

None

local_access_error(path)

Add a file access error message to the list of warnings.

Parameters:

path (str) – file path

Return type:

None

local_permission_error(path)

Add a permission error message to the list of warnings.

Parameters:

path (str) – file path

Return type:

None

print_completion(message)

Remove the progress bar, prints a message, and puts the progress bar back.

Parameters:

message (str) – an error message

Return type:

None

Parameters:

path (str) –

Return type:

None

update_count(delta)

Report that items have been processed.

Parameters:

delta (int) –

Return type:

None

update_total(delta)

Report that more files have been found for comparison.

Parameters:

delta (int) – number of files found since the last check

Return type:

None

stdout: TextIOWrapper
no_progress: bool

Sync Encryption Settings Providers

class b2sdk.v2.AbstractSyncEncryptionSettingsProvider[source]

Object which provides an appropriate EncryptionSetting object for sync, i.e. complex operations with multiple sources and destinations

abstract get_setting_for_upload(bucket, b2_file_name, file_info, length)[source]

Return an EncryptionSetting for uploading an object or None if server should decide.

Parameters:

  • bucket (Bucket) –

  • b2_file_name (str) –

  • file_info (Optional[dict]) –

  • length (int) –

Return type:

Optional[EncryptionSetting]

abstract get_source_setting_for_copy(bucket, source_file_version)[source]

Return an EncryptionSetting for a source of copying an object or None if not required

Parameters:

  • bucket (Bucket) –

  • source_file_version (FileVersion) –

Return type:

Optional[EncryptionSetting]

abstract get_destination_setting_for_copy(bucket, dest_b2_file_name, source_file_version, target_file_info=None)[source]

Return an EncryptionSetting for a destination for copying an object or None if server should decide

Parameters:

  • bucket (Bucket) –

  • dest_b2_file_name (str) –

  • source_file_version (FileVersion) –

  • target_file_info (Optional[dict]) –

Return type:

Optional[EncryptionSetting]

abstract get_setting_for_download(bucket, file_version)[source]

Return an EncryptionSetting for downloading an object from, or None if not required

Parameters:

  • bucket (Bucket) –

  • file_version (FileVersion) –

Return type:

Optional[EncryptionSetting]

class b2sdk.v2.ServerDefaultSyncEncryptionSettingsProvider[source]

Encryption settings provider which assumes setting-less reads and a bucket default for writes.

class b2sdk.v2.BasicSyncEncryptionSettingsProvider(read_bucket_settings, write_bucket_settings)[source]

Basic encryption setting provider that supports exactly one encryption setting per bucket for reading and one encryption setting per bucket for writing

Parameters:
__init__(read_bucket_settings, write_bucket_settings)[source]
Parameters: