Running the Cache¶
Installation¶
If you are going to cache a Shotgun installation of any significant size, you should install the only non-Python dependency: PostgreSQL
You will also need virtualenv for a non-system install (as is recommended).
A typical install looks like:
# Grab the code
git clone git@github.com:westernx/sgcache
cd sgcache
# Create a virtualenv, and install the Python dependencies
virtualenv venv
. venv/bin/activate
pip install -r requirements.txt
pip install -r requirements-westernx.txt
Configuration¶
Configuration is provided by a cascade of information sourced from:
- the default config in
sgcache/config.py
; - environment variables prefixed with
SGCACHE_
; - colon-delimited list of Python files specified by
CONFIG
The recommended configuration setup is to write your own Python file with
your configuration changes, and refer to it via $SGCACHE_CONFIG
.
Database¶
SGCache generally operates with either of SQLite, or PostgreSQL (and maybe others, as it uses SQLAlchemy as its database access layer).
SQLite is usually reserved for very small installations, or development. Production environments should use PostgreSQL.
Schema¶
You must select the subset of the Shotgun schema that you want to cache,
and set SCHEMA
to point to the containing YAML file.
The schema/keystone-basic.yml
file demonstrates the format, and is
generated from our live Shotgun and basic rules via:
./schema/dump > schema/keystone-full.yml
./schema/filter --absent -f schema/basic-filters.txt schema/keystone-full.yml > schema/keystone-basic.yml
Changes¶
To update the schema asserting the cache does not return invalid data, you must:
Update the schema YAML file.
Restart the event watcher and the periodic scanner.
Perform a complete scan of the modified types:
sgcache-scanner --full --scan-types Shot,Task
Restart the web server only once the full scan finishes.
Restarting the web server prematurely will result in it assuming that the new fields are empty, and return incomplete data.
Priming the Cache¶
SGCache assumes that it knows about every entity in your Shotgun, so in order to return correct results, you must perform a full scan:
sgcache-scanner --full
If you are testing, you can specify individual projects to cache:
sgcache-scanner --full --scan-projects 66,67
Running the Daemons¶
The three primary daemons are sgcache-scanner
, sgcache-events
, and sgcache-web
.
It is recommended to run them seperately, but for convenience there is a sgcache-auto
which will run them all.
All configuration options are exposed as command-line flags, but it is recommended
to create a config.py
file, and refer to it via $SGCACHE_CONFIG
.
Reverse Proxying with Nginx¶
It is recommended to run SGCache behind an Nginx reverse proxy. This allows Nginx to directly transfer of large files, as we have experienced trouble with getting the cache to upload massive files itself.
Here is the Nginx config at Western Post:
server {
listen 80;
server_name sgcache.westernx;
# Fails fast with file uploads without this.
client_max_body_size 1G;
# Pass large uploads/downloads to Shotgun.
location ~ ^/(upload|thumbnail|file_serve) {
proxy_set_header Host keystone.shotgunstudio.com;
proxy_pass https://keystone.shotgunstudio.com;
}
# Everything else goes to SGCache.
location / {
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_pass http://127.0.0.1:8010;
}
}
Configuration Reference¶
Shotgun¶
-
sgcache.config.
SHOTGUN_URL
¶ Base URL of the Shotgun server. If this is not set, we will use the follwing code to get arguments:
import shotgun_api3_registry sg = Shotgun(*shotgun_api3_registry.get_args())
-
sgcache.config.
SHOTGUN_SCRIPT_NAME
¶ Script name for Shotgun server.
-
sgcache.config.
SHOTGUN_API_KEY
¶ API key for Shotgun server.
Core¶
-
sgcache.config.
SQLA_URL
¶ SQLAlchemy url for the primary data store. Default uses SQLite, but Postgresql is highly recommended for production use:
postgres:///sgcache
-
sgcache.config.
SCHEMA
¶ The relative path to the schema file to use.
Daemon¶
-
sgcache.config.
WATCH_EVENTS
¶ Should we watch the event log for changes? This is generally required to stay in sync with Shotgun.
-
sgcache.config.
SCAN_CHANGES
¶ Should we periodically scan for changed entities? This is a secondary method of staying in sync, designed to catch any data that the event watcher may have missed.
Events¶
-
sgcache.config.
WATCH_IDLE_DELAY
¶ Delay (in seconds) between polls of the event log.
Scanner¶
-
sgcache.config.
SCAN_SINCE
¶ How far back to scan (in seconds) on initial scan; subsequent scans will only scan for changes since the previous scan.
Warning
Setting this to a falsy value will result in a complete scan of your Shotgun server.
-
sgcache.config.
SCAN_TYPES
¶ Which entity types should be scanned? Falsy values default to all types. Delimit with commas in the command-line.
-
sgcache.config.
SCAN_PROJECTS
¶ Which projects (by ID) should be scanned? Empty values default to all projects. Delimit with commas in the command-line.
-
sgcache.config.
AUTO_LAST_ID
¶ Should we automatically detect when the last time the cache was updated? This affects the start location of both the event log and scanner (overriding
SCAN_SINCE
).
Web¶
-
sgcache.config.
PORT
¶ The port of the web server for the API proxy; set to something falsey to disable the web server entirely. Also set via
$PORT
envvar.
-
sgcache.config.
GUNICORN_WORKERS
¶ Number of web workers.
-
sgcache.config.
GUNICORN_WORKER_CLASS
¶ Class used for driving web workers.
-
sgcache.config.
MAX_CONTENT_LENGTH
¶
None
Logging¶
-
sgcache.config.
SQLA_ECHO
¶ Should SQLA engines log everything they do? Useful for development and debugging.
-
sgcache.config.
LOGGING_FILE_DIR
¶ Directory for log files, relative to
DATA_ROOT
-
sgcache.config.
LOGGING_FILE_LEVEL
¶ Python logging level to capture into files; default os
logging.INFO
.
-
sgcache.config.
LOGGING_SMTP_ARGS
¶ SMPT settings for emailing error logs; is a tuple of arguments for a
logging.handlers.SMTPHandler
:('mail.westernx', 'sgevents@mail.westernx', ['mboers@mail.westernx'], 'SGCache Log Event')
-
sgcache.config.
LOGGING_SMTP_LEVEL
¶ Python logging level to email; default is
logging.ERROR
.
-
sgcache.config.
CLEAR_LOGGERS
¶ Should existing handlers be cleared from the root of Python’s logging system? Useful if you have site-wide capture of Python logging that you don’t want to pollute with the minutia of the cache.