rez.package_cache

class rez.package_cache.PackageCache

Bases: object

Package cache.

A package cache is responsible for storing copies of variant payloads into a location that would typically be on local disk. The intent is to avoid fetching a package’s files over shared storage at runtime.

A package cache is used like so:

  • A rez-env is performed;

  • The context is resolved;

  • For each variant in the context, we check to see if it’s present in the current package cache;

  • If it is, the variant’s root is remapped to this location.

A package cache is not a package repository. It just stores copies of variant payloads - no package definitions are stored.

Payloads are stored into the following structure:

/<cache_dir>/foo/1.0.0/af8d/a/<payload>
                           /a.json

Here, ‘af8d’ is the first 4 chars of the SHA1 hash of the variant’s ‘handle’, which is a dict of fields that uniquely identify the variant. To avoid hash collisions, the variant is then stored under a subdir that is incrementally named (‘a’, ‘b’, …, ‘aa’, ‘ab’, …). The ‘a.json’ file is used to find the correct variant within the hash subdir. The intent is to keep cached paths short, and avoid having to search too many variant.json files to find the matching variant.

VARIANT_NOT_FOUND = 0

Variant was not found

VARIANT_FOUND = 1

Variant was found

VARIANT_CREATED = 2

Variant was created

VARIANT_COPYING = 3

Variant payload is still being copied to this cache

VARIANT_COPY_STALLED = 4

Variant payload copy has stalled

VARIANT_PENDING = 5

Variant is pending caching

VARIANT_REMOVED = 6

Variant was deleted

VARIANT_SKIPPED = 7
STATUS_DESCRIPTIONS = {0: 'was not found', 1: 'was found', 2: 'was created', 3: 'payload is still being copied to this cache', 4: 'payload copy has stalled.\nSee https://rez.readthedocs.io/en/stable/caching.html#cleaning-the-cache for more information.', 5: 'is pending caching', 6: 'was deleted', 7: 'is not being cached due to cache size limit'}
__init__(path)

Create a package cache.

Parameters:

path (str) – Path on disk, must exist.

get_cached_root(variant)

Get location of variant payload copy.

Parameters:

variant (Variant) – Variant to search for.

Returns:

Cached variant root path, or None if not found.

Return type:

str

get_variant_size(free, variant_root)

Get the size of the variant root.

Parameters:

  • free – The available free cache space.

  • variant_root – The rez resolved variant root.

Returns:

The size in bytes of the variant root (may exceed buffer slightly).

Return type:

int

cache_near_full()

Get the cache available space

Returns:

True if available cache space is below buffer, otherwise False.

Return type:

bool

variant_meets_space_requirements(rez_variant_root)

Check if the cache usage is above config.package_cache_used_threshold. If it is, start throttling the cache by checking each variants size to make sure it’s not going to take the cache size below the minimum buffer we set.

Parameters:

variant_root – The rez resolved variant root.

Returns:

  • True if the cache space used is below config.package_cache_used_threshold.

  • False if (free - variant_size) < config.package_cache_space_buffer, otherwise False.

Return type:

bool

add_variant(variant, force=False, wait_for_copying=False, logger=None)

Copy a variant’s payload into the cache.

The following steps are taken to ensure muti-thread/proc safety, and to guarantee that a partially-copied variant payload is never able to be used:

  1. The hash dir (eg ‘/<cache_dir>/foo/1.0.0/af8d’) is created;

  2. A file lock mutex (‘/<cache_dir>/.lock’) is acquired;

  3. The file ‘/<cache_dir>/foo/1.0.0/af8d/.copying-a’ (or -b, -c etc) is created. This tells rez that this variant is being copied and cannot be used yet;

  4. The file ‘/<cache_dir>/foo/1.0.0/af8d/a.json’ is created. Now another proc/thread can’t create the same local variant;

  5. The file lock is released;

  6. The variant payload is copied to ‘/<cache_dir>/foo/1.0.0/af8d/a’;

  7. The ‘.copying-a’ file is removed.

Note that the variant will not be cached in the following circumstances, unless force is True:

  • The variant is not cachable as determined by Variant.is_cachable;

  • The variant is from a local package, and ‘config.package_cache_local’ is False;

  • The variant is stored on the same disk device as this cache, and config.package_cache_same_device’ is False.

Parameters:

  • variant (Variant) – The variant to copy into this cache

  • force (bool) – Copy the variant regardless. Use at your own risk (there is no guarantee the resulting variant payload will be functional).

  • wait_for_copying (bool) – Whether the caching step should block when one of the pending variants is marked as already copying.

  • logger (None | logging.Logger) – If a logger is provided, log information to it.

Returns:

2-tuple: - str: Path to cached payload - int: One of VARIANT_FOUND, VARIANT_CREATED, VARIANT_COPYING, VARIANT_COPY_STALLED, VARIANT_SKIPPED

Return type:

tuple

remove_variant(variant)

Remove a variant from the cache.

Since this removes the associated cached variant payload, there is no guarantee that this will not break packages currently in use by a context.

Note that this does not actually free up associated disk space - you must call clean() to do that.

Returns:

One of: - VARIANT_REMOVED - VARIANT_NOT_FOUND - VARIANT_COPYING

Return type:

int

add_variants_async(variants)

Update the package cache by adding some or all of the given variants.

This method is called when a context is created or sourced. Variants are then added to the cache in a separate process.

Deprecated since version 3.2.0: Use add_variants() instead.

add_variants(variants, package_cache_async=True)

Add the given variants to the package payload cache.

get_variants()

Get variants and their current statuses from the cache.

Returns:

List of 3-tuple:

  • Variant: The cached variant

  • str: Local cache path for variant, if determined (’’ otherwise)

  • int: Status. One of: - VARIANT_FOUND - VARIANT_COPYING - VARIANT_COPY_STALLED - VARIANT_PENDING

Return type:

tuple

run_daemon()

Run as daemon and copy pending variants.

Called via rez-pkg-cache –daemon.

clean(time_limit=None)

Delete unused package cache files.

This should be run periodically via ‘rez-pkg-cache –clean’.

This removes:

  • Variants that have not been used in more than ‘config.package_cache_max_variant_days’ days;

  • Variants that have stalled;

  • Variants that are already pending deletion (remove_variant() was used).

Parameters:

time_limit (float) – Perform cleaning operations only up until this limit, resulting in a possibly incomplete cleanup. This is used to keep the cache size down without having to periodically run ‘rez-pkg-cache –clean’.