Skip to content

PSSM_GREMLIN Server

What It Is

The PSSM_GREMLIN server is a backend compute service that runs two key bioinformatics analyses on protein sequences:

  • PSSM (Position-Specific Scoring Matrix) -- generated by PSI-BLAST against the UniRef90 database. The resulting PSSM profile captures evolutionary conservation at each residue position, which REvoDesign uses as a mutagenesis guide.
  • GREMLIN (Generative REgularized ModeLs of proteINs) -- a co-evolution analysis built on TensorFlow 1.x. GREMLIN identifies pairs of residue positions that co-vary across a multiple sequence alignment (generated by HHblits against UniRef30), revealing structurally or functionally coupled sites.

Users upload a FASTA file via the server's web UI or REST API. The server queues the job, dispatches it to a Docker container (the "runner"), and makes results available for download once finished.

PSSM/GREMLIN server — task submission page
PSSM/GREMLIN server: upload a FASTA file to submit a computation task
PSSM/GREMLIN server — task dashboard overview
PSSM/GREMLIN server dashboard: monitor tasks, view status, and download results

The service uses an asynchronous task processing architecture to support concurrent multi-user submissions. When a user uploads a FASTA file, the system computes its MD5 hash as a unique task identifier, creates an isolated directory for input/output data, and enqueues the job onto a Celery message queue. Task state progresses through: pending → running → packing results → finished (or failed). The actual computation runs in an isolated Docker container per task, preventing interference between concurrent jobs. When complete, all result files (PSSM, co-evolution coupling scores, MSA files) are packaged into a ZIP archive for one-click download from the dashboard. Users can also cancel queued or running tasks.

Architecture

The server has four containerized services, orchestrated by Docker Compose:

                ┌──────────────────┐
                │     Browser /    │
                │    curl client   │
                └────────┬─────────┘
                         │ HTTP (port 8080)
                         ▼
┌─────────────────────────────────────────┐
│              web (Flask + Gunicorn)     │
│  - REST API (/PSSM_GREMLIN/api/...)    │
│  - Web UI (create_task, dashboard)     │
│  - Basic-auth via Flask-HTTPAuth       │
│  - Dispatches runner containers        │
│  - Needs /var/run/docker.sock          │
└────┬────────────────────────────────┬───┘
     │ Celery tasks                   │ docker socket
     ▼                                ▼
┌──────────┐               ┌──────────────────┐
│  redis   │◄──────────────│  worker (Celery) │
│ (broker) │  task queue   │  Background job   │
└──────────┘               │  executor         │
                           │  Needs docker.sock│
                           └────────┬─────────┘
                                    │ launches
                                    ▼
                    ┌──────────────────────────┐
                    │  runner container         │
                    │  (conda + GREMLIN env)    │
                    │  Runs REvoDesign_PSSM_    │
                    │  GREMLIN.sh script        │
                    │  Mounts: FASTA, DB        │
                    │  dirs, output dir         │
                    └──────────────────────────┘
REvoDesign evolutionary data calculation service architecture
Service architecture design: layered topology from reverse proxy down to Docker compute containers

The service stack is fully containerized and orchestrated via Docker Compose. Deployment requires only database files, environment variables, and a single command. All services run as non-root users to prevent privilege escalation; database directories are mounted read-only for data safety. User task data is organized by user identity and task MD5, with strict permission isolation.

Services

Service Base Image Role
web python:3.12-slim Flask + Gunicorn HTTP server. Serves the web UI and REST API.
worker Same as web Celery worker that receives run_gremlin_task jobs from Redis.
redis redis:7.2-alpine Celery message broker and result backend.
runner condaforge/mambaforge On-demand container that runs the PSSM/GREMLIN computation. Launched dynamically by web/worker via the Docker socket.

Key Design Decisions

  • Docker-out-of-Docker: The web and worker containers bind-mount /var/run/docker.sock to create and manage runner containers on the host Docker daemon. This isolates the heavy bioinformatics dependencies (older Python, TensorFlow 1.x, HHsuite) into the runner image, keeping the server image lean.
  • Celery for async tasks: Long-running jobs (potentially hours) are dispatched via Celery so the HTTP request returns immediately with a task ID for polling.
  • SQLite persistence: A lightweight SQLite database (via SQLAlchemy) tracks task state, metadata, and results locations.

API Endpoints

All endpoints are behind HTTP Basic Authentication (Flask-HTTPAuth) and are mounted under the /PSSM_GREMLIN path prefix.

Task Management

Method Endpoint Description
GET /PSSM_GREMLIN/create_task Web form to upload a FASTA file
POST /PSSM_GREMLIN/api/post Upload a FASTA file via API (multipart form)
GET /PSSM_GREMLIN/api/running/<md5sum> Poll task status
GET /PSSM_GREMLIN/api/results/<md5sum> Redirect to download URL when finished
GET /PSSM_GREMLIN/api/download/<md5sum> Download result ZIP archive
POST /PSSM_GREMLIN/api/cancel/<md5sum> Cancel a pending or running task
DELETE /PSSM_GREMLIN/api/delete/<md5sum> Delete a single task (soft-delete)
POST /PSSM_GREMLIN/api/delete Batch delete (JSON body: {"md5sums": [...]})

Dashboard & UI

Method Endpoint Description
GET /PSSM_GREMLIN/dashboard HTML dashboard showing task list, status, wall time
GET /favicon.ico Server favicon
GET /PSSM_GREMLIN/logo.svg REvoDesign logo

Task States

State Description
pending Uploaded, waiting for Celery worker
running Computation in progress
packing results Runner finished; archiving output
finished Results ready for download
failed Computation error
cancelled User-cancelled task
deleted:finshed Soft-deleted after completion
deleted:cancel Soft-deleted before completion

Running Trace Stages

During execution, tasks report their current stage:

  1. hhblits -- HHblits search against UniRef30
  2. hhfilter -- Filter MSA at 90% identity / 75% coverage
  3. gremlin -- GREMLIN co-evolution calculation (TensorFlow)
  4. blast -- PSI-BLAST search for PSSM profile

Deployment

Docker Images

Two Docker images are built and pushed to Docker Hub:

Image Dockerfile Purpose
yaoyinying/revodesign-pssm-gremlin-non-root server/docker/runner/Dockerfile Runner: conda env with GREMLIN dependencies
yaoyinying/revodesign-pssm-gremlin-server-non-root server/docker/server/Dockerfile Server: Flask + Celery + Gunicorn

The GitHub Actions workflow at .github/workflows/docker-image.yml builds both images on workflow_dispatch, tags them with the date and latest, and pushes to Docker Hub.

Environment Configuration

Required environment variables (defined in docker-compose.yml):

Variable Description
SERVER_DIR Host root for uploads, SQLite, and result folders
LOG_DIR Host directory for Gunicorn/Celery logs
DB_UNIREF30 UniRef30 HHsuite database prefix path
DB_UNIREF90 UniRef90 BLAST database prefix path
USERS_FILE Path to basic-auth credentials file
RUNNER_UID / RUNNER_GID Non-root user for runner containers
DOCKER_GID Group ID of the Docker socket on the host
NPROC CPU threads for runner
MAXMEM Memory cap (GB) for HHblits (-maxmem)
PORT Public HTTP port (default: 8080)
PUBLIC_DASHBOARD Per-user task isolation (default: false)
ADMIN_USERS Comma-separated admin usernames
TZ Timezone for logs

Setup Steps

  1. Prepare sequence databases on the host:
  2. Download and extract UniRef90, build BLAST database with makeblastdb

  3. Download and extract UniRef30 HHsuite archive

  4. Configure environment: 

    cp server/.env.example server/.env.production
# Edit .env.production with your paths and settings

  5. Configure basic auth users in the file referenced by USERS_FILE: 

    username:password

  6. Build and run using the helper script: 

    REVODESIGN_SERVER_ENV=server/.env.production \
  bash server/run/restart_pssm_flask.sh setup
REVODESIGN_SERVER_ENV=server/.env.production \
  bash server/run/restart_pssm_flask.sh restart

  7. Access the web UI at http://<host>:<port>/PSSM_GREMLIN/dashboard

Alternative Access

  • Cloudflare Tunnel: Expose the service without opening inbound ports.
  • NGINX reverse proxy: A configuration template is at server/nginx_sites/REvoDesign_PSSM_GREMLIN.app.

Runner Script

The runner script at server/REvoDesign_PSSM_GREMLIN.sh performs the full computation pipeline:

  1. hhblits -- searches the query sequence against UniRef30, producing a multiple sequence alignment (A3M format)
  2. hhfilter -- filters the MSA at 90% sequence identity and 75% coverage
  3. Lower-case removal -- post-processes inserts from the A3M
  4. GREMLIN_TFv1.py -- runs the TensorFlow 1.x GREMLIN model to compute co-evolutionary couplings (output: .mrf.pkl)
  5. psiblast -- runs PSI-BLAST against UniRef90 to produce the PSSM profile (output: ASCII matrix file and checkpoint)

Singularity Runner / Direct Execution

The script also carries a #SBATCH header for Slurm and can activate a conda environment (GREMLIN or tensorflow1.13) for native execution without Docker, though the production deployment is Docker-only.

Conda Environment

The runner conda environment is defined at server/env/GREMLIN.yml:

  • Python 3.6 (compatible with TensorFlow 1.x)
  • TensorFlow 1.13.1, NumPy, SciPy, Pandas, Matplotlib, Biopython
  • BLAST 2.13, HHsuite 3.3, HMMER 3.3.2
  • Channels: defaults, conda-forge, bioconda

The server's Python dependencies are in server/docker/server/requirements.txt and include Flask 3.x, Celery 5.x (with Redis), SQLAlchemy 2.x, and the Docker SDK for Python.

REvoDesign Plugin Integration

The REvoDesign plugin does not connect to the PSSM_GREMLIN server directly. The server is an independent HTTP service; users upload FASTA files via the web dashboard or curl.

Separately, REvoDesign includes a WebSocket peer-collaboration feature that allows multiple REvoDesign instances to share views, mutant trees, and data in real time. This is configured under ui.socket in main.yaml:

socket:
  server_url: localhost
  server_port: 63189
  use_key: true

The client-side implementation is at src/REvoDesign/clients/QtSocketConnector.py (see API Reference for details).