Download Registry

The download registry provides reliable file fetching with retry, mirror fallback, hash verification, local caching, and extraction of compressed archives. It is built on the Pooch library.

REvoDesign.tools.download_registry.FileDownloadRegistry

Bases: CitableModuleAbstract

A file download registry manager for handling remote file resources.

This class implements automatic file downloading, verification, and cache management based on the pooch library. It supports specifying file hash values through a registry and provides convenient methods to fetch and verify remote files.

:param name: Module name, used to construct the default data directory path. :param base_url: Base URL for remote files. :param registry: File registry where keys are filenames and values are corresponding hash values (optional). :param version: Optional version number for creating versioned data directories. :param customized_directory: Optional custom download directory path. If not provided, uses the default user data directory. :param alternative_base_urls: Optional list of alternative base URLs for downloading files. :param retry_count: Number of retries for downloading files.

Retry mechanism

The function will run a nested loop to retry downloading files. It firstly tries to download files from the primary base URL. If the download fails, it will retry with the alternative base URLs if provided. For each base url, a certain number of retries will be attempted.

list_all_files property

Get a list of all files in the registry.

:return: List of filenames.

preprocess_registry(registry) staticmethod

Preprocess registry to ensure all hash values conform to pooch requirements.

:param registry: Original registry. :return: Processed registry.

setup(item)

Download and return the local path and related information of the specified file.

:param item: Filename to download. :return: DownloadedFile object containing file information. :raises NetworkError: Raises network error exception if download fails.

has(item)

Check if the specified file exists in the registry.

:param item: Filename. :return: True if exists, False otherwise.

prepare_registry_from_md5(md5_contents) staticmethod

Parse and generate registry from MD5 content string.

:param md5_contents: String containing MD5 values and filenames, each line in 'hash filename' format. :return: Parsed registry dictionary.

REvoDesign.tools.download_registry.DownloadedFile dataclass

Represents a downloaded file with its metadata

Attributes:

Name	Type	Description
name	str	Name of the file
version	Optional[str]	Version of the file, can be None
url	str	Download URL of the file
downloaded	str	Local path where the file is downloaded
registry	Optional[str]	Registry information, defaults to None

flatten_dir property

Get or create the flatten directory path

Creates directory if it doesn't exist, with name based on downloaded path plus '_flatten' suffix

Returns:

Name	Type	Description
str	str	Path to the flatten directory

flatten_archive property

Extract archive file to flatten directory

If flatten directory is empty, extracts the downloaded archive file to that directory and returns the list of extracted files

Returns:

Type	Description
	List[str]: List of extracted file names

Usage Pattern

from REvoDesign.tools.download_registry import FileDownloadRegistry

registry = FileDownloadRegistry(
    name="my_data",
    base_url="https://example.com/data/",
    registry={
        "file1.pdb": "md5:abc123def456",
        "file2.pkl": None,            # no hash verification
    },
    alternative_base_urls=["https://mirror.example.com/data/"],
    retry_count=3,
)

# Download with automatic retry and mirror fallback
result = registry.setup("file1.pdb")
print(result.downloaded)            # local path
print(result.url)                   # original URL
print(result.registry)              # hash string

# List available files
print(registry.list_all_files)

# Check if a file is in the registry
print(registry.has("file2.pkl"))

# Extract compressed archives
result.flatten_archive              # extracts to {path}_flatten/

Retry Mechanism

The FileDownloadRegistry.setup() method implements a cascading retry strategy:

1. For each base URL (primary first, then alternatives), Pooch's built-in retry mechanism is used (controlled by retry_count).
2. If all retries for a URL fail, the next alternative URL is tried.
3. If all URLs are exhausted, a NetworkError is raised.

Process

The setup() method returns a DownloadedFile dataclass that includes the local filesystem path. The archive extraction feature (flatten_archive) can automatically unpack .tar.gz and .zip archives into a _flatten/ subdirectory.

Creating Registries from MD5 Checksums

md5_text = """\
d41d8cd98f00b204e9800998ecf8427e  file1.pdb
e99a18c428cb38d5f260853678922e03  file2.pdb
"""
registry = FileDownloadRegistry.prepare_registry_from_md5(md5_text)