Running the Cache

Installation

If you are going to cache a Shotgun installation of any significant size, you should install the only non-Python dependency: PostgreSQL

You will also need virtualenv for a non-system install (as is recommended).

A typical install looks like:

# Grab the code
git clone git@github.com:westernx/sgcache
cd sgcache

# Create a virtualenv, and install the Python dependencies
virtualenv venv
. venv/bin/activate
pip install -r requirements.txt
pip install -r requirements-westernx.txt

Configuration

Configuration is provided by a cascade of information sourced from:

1. the default config in sgcache/config.py;
2. environment variables prefixed with SGCACHE_;
3. colon-delimited list of Python files specified by CONFIG

The recommended configuration setup is to write your own Python file with your configuration changes, and refer to it via $SGCACHE_CONFIG.

Database

SGCache generally operates with either of SQLite, or PostgreSQL (and maybe others, as it uses SQLAlchemy as its database access layer).

SQLite is usually reserved for very small installations, or development. Production environments should use PostgreSQL.

Schema

You must select the subset of the Shotgun schema that you want to cache, and set SCHEMA to point to the containing YAML file. The schema/keystone-basic.yml file demonstrates the format, and is generated from our live Shotgun and basic rules via:

./schema/dump > schema/keystone-full.yml
./schema/filter --absent -f schema/basic-filters.txt schema/keystone-full.yml > schema/keystone-basic.yml

Changes

To update the schema asserting the cache does not return invalid data, you must:

1. Update the schema YAML file.

2. Restart the event watcher and the periodic scanner.

3. Perform a complete scan of the modified types:

   sgcache-scanner --full --scan-types Shot,Task
   

4. Restart the web server only once the full scan finishes.

Restarting the web server prematurely will result in it assuming that the new fields are empty, and return incomplete data.

Priming the Cache

SGCache assumes that it knows about every entity in your Shotgun, so in order to return correct results, you must perform a full scan:

sgcache-scanner --full

If you are testing, you can specify individual projects to cache:

sgcache-scanner --full --scan-projects 66,67

Running the Daemons

The three primary daemons are sgcache-scanner, sgcache-events, and sgcache-web. It is recommended to run them seperately, but for convenience there is a sgcache-auto which will run them all.

All configuration options are exposed as command-line flags, but it is recommended to create a config.py file, and refer to it via $SGCACHE_CONFIG.

Reverse Proxying with Nginx

It is recommended to run SGCache behind an Nginx reverse proxy. This allows Nginx to directly transfer of large files, as we have experienced trouble with getting the cache to upload massive files itself.

Here is the Nginx config at Western Post:

server {
    listen       80;
    server_name  sgcache.westernx;

    # Fails fast with file uploads without this.
    client_max_body_size 1G;

    # Pass large uploads/downloads to Shotgun.
    location ~ ^/(upload|thumbnail|file_serve) {
        proxy_set_header Host keystone.shotgunstudio.com;
        proxy_pass https://keystone.shotgunstudio.com;
    }

    # Everything else goes to SGCache.
    location / {
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_pass http://127.0.0.1:8010;
    }

}

Configuration Reference

Shotgun

sgcache.config.SHOTGUN_URL

Base URL of the Shotgun server. If this is not set, we will use the follwing code to get arguments:

import shotgun_api3_registry
sg = Shotgun(*shotgun_api3_registry.get_args())

sgcache.config.SHOTGUN_SCRIPT_NAME

Script name for Shotgun server.

sgcache.config.SHOTGUN_API_KEY

API key for Shotgun server.

Core

sgcache.config.SQLA_URL

SQLAlchemy url for the primary data store. Default uses SQLite, but Postgresql is highly recommended for production use:

postgres:///sgcache

sgcache.config.SCHEMA

The relative path to the schema file to use.

Daemon

sgcache.config.WATCH_EVENTS

Should we watch the event log for changes? This is generally required to stay in sync with Shotgun.

sgcache.config.SCAN_CHANGES

Should we periodically scan for changed entities? This is a secondary method of staying in sync, designed to catch any data that the event watcher may have missed.

Events

sgcache.config.WATCH_IDLE_DELAY

Delay (in seconds) between polls of the event log.

Scanner

sgcache.config.SCAN_INTERVAL

Delay (in seconds) between scans of changed entities.

sgcache.config.SCAN_SINCE

How far back to scan (in seconds) on initial scan; subsequent scans will only scan for changes since the previous scan.

Warning

Setting this to a falsy value will result in a complete scan of your Shotgun server.

sgcache.config.SCAN_TYPES

Which entity types should be scanned? Falsy values default to all types. Delimit with commas in the command-line.

sgcache.config.SCAN_PROJECTS

Which projects (by ID) should be scanned? Empty values default to all projects. Delimit with commas in the command-line.

sgcache.config.AUTO_LAST_ID

Should we automatically detect when the last time the cache was updated? This affects the start location of both the event log and scanner (overriding SCAN_SINCE).

Web

sgcache.config.PORT

The port of the web server for the API proxy; set to something falsey to disable the web server entirely. Also set via $PORT envvar.

sgcache.config.GUNICORN_WORKERS

Number of web workers.

sgcache.config.GUNICORN_WORKER_CLASS

Class used for driving web workers.

sgcache.config.MAX_CONTENT_LENGTH

None

Logging

sgcache.config.SQLA_ECHO

Should SQLA engines log everything they do? Useful for development and debugging.

sgcache.config.LOGGING_FILE_DIR

Directory for log files, relative to DATA_ROOT

sgcache.config.LOGGING_FILE_LEVEL

Python logging level to capture into files; default os logging.INFO.

sgcache.config.LOGGING_SMTP_ARGS

SMPT settings for emailing error logs; is a tuple of arguments for a logging.handlers.SMTPHandler:

('mail.westernx', 'sgevents@mail.westernx', ['mboers@mail.westernx'], 'SGCache Log Event')

sgcache.config.LOGGING_SMTP_LEVEL

Python logging level to email; default is logging.ERROR.

sgcache.config.CLEAR_LOGGERS

Should existing handlers be cleared from the root of Python’s logging system? Useful if you have site-wide capture of Python logging that you don’t want to pollute with the minutia of the cache.

Environ

sgcache.config.CONFIG

List of external configration files to include; usually set via $SGCACHE_CONFIG as a colon-delimited list.

Core

sgcache.config.TESTING

Allows for unsafe behaviour for unit testing.

sgcache.config.DATA_ROOT

Where runtime data (logs, sockets, locks, etc..) is stored.