Self-Host Whisper API on a GPU Droplet — faster-whisper + FastAPI

Disclosure: Some links in this guide are affiliate links. If you sign up for DigitalOcean through them, ToolsHubKit earns a small commission — at no cost to you. We only recommend services we use.

Hosted transcription APIs charge per minute of audio and require you to upload potentially sensitive recordings to a third party. If you process a lot of audio — or any audio you'd rather not hand off — running your own Whisper API on a GPU Droplet gives you fixed, predictable cost and full data ownership. This guide deploys faster-whisper behind a FastAPI service with a job queue and API-key auth.

Prerequisites:

Basic Linux comfort (SSH, sudo, Python)
A domain or subdomain for the API (optional but recommended for HTTPS)
Audio files to test with (.mp3, .wav, .m4a)

1. Why Self-Host Whisper

Factor	Hosted API	Self-hosted on GPU Droplet
Pricing model	Per minute of audio	Fixed hourly Droplet cost
Data privacy	Audio leaves your infra	Audio never leaves your server
Cost at high volume	Scales linearly, gets expensive	Flat — cheaper past a break-even
Cold starts	None	None (always-on Droplet)
Model control	Fixed by provider	Any Whisper size you want

The trade-off: you manage the box. For steady or high-volume workloads, or privacy-sensitive audio, self-hosting wins on both cost and control.

2. Provision a GPU Droplet

GPU Droplets are a distinct product from regular Droplets. In the DigitalOcean console go to Create → GPU Droplets. Get $200 in free credits for 60 days when you sign up.

Sizing guidance:

Workload	GPU	Whisper model	Notes
Occasional / testing	Single NVIDIA H100 (fractional if available)	`small` / `medium`	Fastest to start
Steady production	Single H100	`large-v3`	Best accuracy
Batch transcription	Single H100	`large-v3` + int8	Throughput-optimized

Choose the AI/ML-ready Ubuntu 24.04 image if offered — it ships with NVIDIA drivers and CUDA preinstalled, saving you Step 3. Add your SSH key at creation.

Cost note: GPU Droplets bill hourly and are significantly more expensive than standard Droplets. Destroy the Droplet when you're not using it, or use it for batch jobs and tear it down after. For always-on low-volume needs, the small/medium models on a smaller GPU are far cheaper.

3. Verify CUDA (if not using the AI/ML image)

bash

ssh root@YOUR_GPU_DROPLET_IP

# Check the GPU is visible
nvidia-smi

If nvidia-smi fails, install the drivers and CUDA toolkit:

bash

apt update && apt upgrade -y
apt install -y nvidia-driver-550 nvidia-cuda-toolkit
reboot

After reboot, nvidia-smi should list your GPU, driver version, and CUDA version.

4. Install faster-whisper

faster-whisper is a CTranslate2 reimplementation of Whisper — up to 4× faster and lower memory than the reference implementation.

bash

# Create a non-root user for the service
adduser whisper
usermod -aG sudo whisper
su - whisper

# Python environment
sudo apt install -y python3-venv python3-pip ffmpeg
python3 -m venv ~/whisper-env
source ~/whisper-env/bin/activate

pip install --upgrade pip
pip install faster-whisper fastapi "uvicorn[standard]" python-multipart

ffmpeg is required for decoding audio formats. Test the model loads and uses the GPU:

python

# test.py
from faster_whisper import WhisperModel

model = WhisperModel("large-v3", device="cuda", compute_type="float16")
segments, info = model.transcribe("sample.mp3", beam_size=5)
print(f"Detected language: {info.language} ({info.language_probability:.2f})")
for seg in segments:
    print(f"[{seg.start:.2f} -> {seg.end:.2f}] {seg.text}")

bash

python test.py

5. The FastAPI Service

Create ~/app/main.py:

python

import os
import uuid
import tempfile
from fastapi import FastAPI, UploadFile, File, Header, HTTPException, BackgroundTasks
from fastapi.responses import JSONResponse
from faster_whisper import WhisperModel

API_KEY = os.environ["WHISPER_API_KEY"]
MODEL_SIZE = os.environ.get("WHISPER_MODEL", "large-v3")

app = FastAPI(title="Private Whisper API")

# Load the model once at startup (kept warm in GPU memory)
model = WhisperModel(MODEL_SIZE, device="cuda", compute_type="float16")

# Simple in-memory job store (swap for Redis in production)
jobs: dict[str, dict] = {}


def check_key(authorization: str | None):
    if authorization != f"Bearer {API_KEY}":
        raise HTTPException(status_code=401, detail="Invalid or missing API key")


def run_transcription(job_id: str, audio_path: str):
    try:
        segments, info = model.transcribe(audio_path, beam_size=5)
        text = " ".join(seg.text.strip() for seg in segments)
        jobs[job_id] = {
            "status": "done",
            "language": info.language,
            "duration": info.duration,
            "text": text,
        }
    except Exception as e:
        jobs[job_id] = {"status": "error", "error": str(e)}
    finally:
        os.remove(audio_path)


@app.post("/transcribe")
async def transcribe(
    background_tasks: BackgroundTasks,
    file: UploadFile = File(...),
    authorization: str | None = Header(default=None),
):
    check_key(authorization)

    job_id = str(uuid.uuid4())
    suffix = os.path.splitext(file.filename or "")[1] or ".tmp"
    fd, tmp_path = tempfile.mkstemp(suffix=suffix)
    with os.fdopen(fd, "wb") as f:
        f.write(await file.read())

    jobs[job_id] = {"status": "processing"}
    background_tasks.add_task(run_transcription, job_id, tmp_path)

    return JSONResponse({"job_id": job_id, "status": "processing"})


@app.get("/result/{job_id}")
async def result(job_id: str, authorization: str | None = Header(default=None)):
    check_key(authorization)
    if job_id not in jobs:
        raise HTTPException(status_code=404, detail="Job not found")
    return jobs[job_id]


@app.get("/health")
async def health():
    return {"status": "ok", "model": MODEL_SIZE}

Run it:

bash

export WHISPER_API_KEY="$(openssl rand -hex 24)"
echo "Your API key: $WHISPER_API_KEY"

uvicorn app.main:app --host 0.0.0.0 --port 8000

6. Concurrency and Queueing Long Files

A single GPU processes one transcription at a time efficiently. The BackgroundTasks approach above serializes work per worker, which is what you want — running two large-v3 jobs simultaneously on one GPU is slower than running them back-to-back.

For heavier production use, replace the in-memory store with a Redis-backed queue (RQ or Celery):

bash

pip install redis rq
sudo apt install -y redis-server

python

# worker.py — a dedicated queue worker
from redis import Redis
from rq import Worker, Queue

redis_conn = Redis()
if __name__ == "__main__":
    Worker([Queue(connection=redis_conn)], connection=redis_conn).work()

Enqueue jobs from the API instead of BackgroundTasks, and run one worker per GPU. This gives you durable jobs, retry on crash, and a clean concurrency limit (one in-flight job per GPU worker).

7. Run as a systemd Service

Create /etc/systemd/system/whisper-api.service:

ini

[Unit]
Description=Private Whisper Transcription API
After=network.target

[Service]
Type=simple
User=whisper
WorkingDirectory=/home/whisper
Environment=WHISPER_API_KEY=your_generated_key_here
Environment=WHISPER_MODEL=large-v3
ExecStart=/home/whisper/whisper-env/bin/uvicorn app.main:app --host 0.0.0.0 --port 8000
Restart=always
RestartSec=5

[Install]
WantedBy=multi-user.target

bash

sudo systemctl enable --now whisper-api
sudo journalctl -u whisper-api -f

8. Secure the Endpoint

Nginx reverse proxy + HTTPS + rate limiting. Install Nginx and Certbot:

bash

sudo apt install -y nginx certbot python3-certbot-nginx

Add a rate-limit zone in /etc/nginx/nginx.conf (inside the http block):

nginx

limit_req_zone $binary_remote_addr zone=whisper:10m rate=10r/m;

Create /etc/nginx/sites-available/whisper:

nginx

server {
    listen 80;
    server_name whisper.yourdomain.com;

    client_max_body_size 200M;  # allow large audio uploads

    location / {
        limit_req zone=whisper burst=5 nodelay;
        proxy_pass http://127.0.0.1:8000;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_read_timeout 600s;  # long transcriptions
    }
}

bash

sudo ln -s /etc/nginx/sites-available/whisper /etc/nginx/sites-enabled/
sudo nginx -t && sudo systemctl reload nginx
sudo certbot --nginx -d whisper.yourdomain.com

# Firewall: only SSH + HTTPS exposed
sudo ufw allow OpenSSH
sudo ufw allow 'Nginx Full'
sudo ufw enable

Now the API is reachable only over HTTPS, rate-limited, and gated by your bearer token.

9. Using the API

bash

API=https://whisper.yourdomain.com
KEY=your_generated_key_here

# Submit a file
JOB=$(curl -s -X POST "$API/transcribe" \
  -H "Authorization: Bearer $KEY" \
  -F "file=@meeting.mp3" | jq -r .job_id)

# Poll for the result
curl -s "$API/result/$JOB" -H "Authorization: Bearer $KEY" | jq

10. Cost per Hour of Audio — Break-Even

The math depends on live GPU Droplet pricing (verify current rates before committing), but the shape is always the same:

Hosted APIs charge a fixed rate per minute → cost rises linearly with volume.
A GPU Droplet charges a fixed hourly rate and transcribes many hours of audio per wall-clock hour with large-v3 (real-time factors of 10–30× are common on a modern GPU).

Break-even is typically reached at a few dozen hours of audio per month. Past that, self-hosting is dramatically cheaper — and for batch workloads, spin the Droplet up, process everything, and destroy it, paying only for the hours you actually used.

Always re-verify current GPU Droplet and hosted-API pricing before publishing your own numbers — both drift.

11. Wiring It Into ToolsHubKit

If you use ToolsHubKit's audio tools, point them at your private endpoint instead of a hosted API — same request shape (POST /transcribe with a bearer token), but your audio never leaves infrastructure you control. That's the whole ToolsHubKit ethos: local-first, privacy by default, no third-party data handoff.

Related Guides

Self-hosted background-removal service with DigitalOcean Spaces — another AI-inference-on-a-Droplet pattern
GitHub Actions auto-deploy to a Droplet — automate updates to this service
Self-host your entire developer stack on one Droplet — the broader self-hosting playbook

Resources

Resource	Link
DigitalOcean GPU Droplets (+ $200 free credit)	Sign up here
faster-whisper	https://github.com/SYSTRAN/faster-whisper
OpenAI Whisper	https://github.com/openai/whisper
FastAPI	https://fastapi.tiangolo.com
RQ (Redis Queue)	https://python-rq.org