Clipify

Problem

Content creators sit on hours of long-form footage (podcasts, streams, YouTube VODs) and repurpose it into short-form clips for TikTok, Shorts, and Reels. The manual path is cutting an 8-hour podcast into ten 45-second clips, writing hook titles, burning captions, and re-rendering per clip per platform. Most creators pay an editor several hundred dollars per long video or skip distribution entirely. I wanted to build a pipeline that took a YouTube URL or MP4 upload, ran it through transcription, picked the actual highlights at a quality close to what a human editor would choose, rendered them captioned and cropped to target aspect, and delivered download-ready clips without a manual pass. The non-obvious constraint was that moment-picking had to earn its cost: GPT scoring at a per-minute rate across hours of transcript adds up fast, and a naïve “send the whole transcript to GPT” approach was both expensive and worse than a heuristic-first pipeline.

Approach & Architecture

I built Clipify as a pnpm monorepo with two deployed services: a Next.js 15 App Router web application handling uploads, billing, and dashboard, and a Dockerized Node 20 worker that owned the heavy pipeline. Three BullMQ queues on Upstash Redis connected them (q_transcribe at concurrency 1 and API-bound, q_detect at concurrency 2, q_render at concurrency 0.5 and FFmpeg CPU-bound) with per-job timeouts (20 / 10 / 15 minutes), three retries with 2s exponential backoff, and stalled-job detection under a 10-minute lock.

The transcription stage sent audio to OpenAI Whisper with verbose_json and word-level timestamp granularity. For videos over 45 minutes I auto-split audio into 10-minute segments, processed each independently, and merged word timestamps back to global time. Memory stayed bounded and 8+ hour podcasts went through without API timeouts. An adaptive per-request timeout scaled with video length (minimum 15 minutes, up to 0.8× duration).

Moment detection was the interesting piece. I ran a heuristic-first scoring pass across the transcript (energy peaks via excited lexicon and fast speech, story-arc markers, viral hook patterns like “plot twist” and “wait until you see this”, and speaker-change dynamics) to generate variable-length candidate windows (35-75s) at three positions around each detected moment, deduplicated against >50% overlap. Only the top 15 candidates went to GPT-4o-mini for re-ranking against a structured rubric: Immediate Engagement 30%, Emotional Resonance 25%, Standalone Value 25%, Shareability 20%. The model returned scores plus hook titles in one call. Final selection picked 2-4 clips with 15-second minimum spacing. Rendering ran each selected clip through FFmpeg (H.264 High Profile at CRF 18, 10 Mbps max, 30 fps, AAC 192 kbps) with smart cropping per aspect ratio and ASS subtitle burning (Arial Black 68px, line breaks at punctuation with a 35-character limit).

A three-tier S3 lifecycle tiered storage by plan (users/free/ vs users/paid/): free-tier files expired after 90 days, paid-tier transitioned to Glacier, and all uploads and downloads went through presigned URLs (5-minute and 1-hour TTLs) so files never touched the server. Stripe drove three billing plans (Free at 60 min/month watermarked, Pro at $29/mo for 600 min, Agency at $99/mo for 2000 min) with five webhook events persisted as BillingEvent rows for audit.

Tradeoffs

A single-pass “send the whole transcript to GPT-4o and pick clips” was the obvious simplification, and I chose against it. Heuristic-first filtering gated the expensive GPT call behind cheap regex and lexicon scoring, cutting per-video cost meaningfully on long podcasts without noticeably hurting selection quality. The three-stage queue split added orchestration complexity but meant a crashed render couldn’t invalidate a completed transcription that cost real OpenAI money. Deducting quota minutes at transcription start rather than completion was a deliberate anti-abuse choice: it prevents users from abandoning jobs to reclaim minutes, at the cost of billing fairly for failed renders. Whisper API over a self-hosted model traded per-minute cost for zero ops work and word-level timestamp accuracy I couldn’t match locally.

Outcome

Clipify runs as a Next.js 15 web app and a Dockerized Node 20 worker across 17 API routes, three BullMQ queues, 12 Prisma models on Neon serverless Postgres, and a typed SSE event stream driving real-time queue position on the dashboard. The moment-detection pipeline runs five stages (heuristic scoring, candidate generation, deduplication, GPT re-ranking, final selection) and supports optional GPT completeness scoring for podcasts over 30 minutes. Long-form videos auto-chunk at the 45-minute boundary into 10-minute audio segments. The billing system covers three plan tiers with server-side quota enforcement at both project creation and transcription start, premium feature gating at the API layer, and Stripe signature verification on every webhook. Clips export as individual downloads, streaming-ZIP bulk downloads, JSON metadata, and CMX3600 EDL for professional NLEs.

Learnings

The hardest problem was the moment-detection cost/quality curve. My first version sent the entire transcript to GPT-4o for scoring, which produced reasonable clips but cost multiple dollars per hour and got slower as transcripts grew. The fix was inverting the pipeline: cheap heuristic scoring on 100% of transcript, variable-window candidate generation, dedupe, then hand only the top 15 candidates to GPT-4o-mini for a structured rubric. The deeper lesson was that LLM scoring is a last-mile refinement, not a first-pass filter. The tempting shape of “LLM sees everything” is the one that burns money. The second lesson was idempotency: detection deleted existing clips before creating new ones, rendering checked for double-attempts, and queue jobs used deduplication IDs. Without that discipline retry logic produced duplicate clips on every transient Whisper failure.