When I started building a Video Retrieval system, the embedding model seemed like the hard part. I had to decide where to split a video, how many frames to sample from each segment, and how to preserve temporal information. I also assumed that inference would eventually dominate the runtime as the model grew.

The first production runs told a different story. Inference had finished, but the next batch was nowhere to be found. The CPU was still decoding video while the GPU sat idle and waited for frames. Making the model faster would not fix that kind of bottleneck.

I moved scene splitting onto GPU decoding, then isolated the decoder in a separate process so that one malformed video could not freeze the entire API. When its heartbeat stopped, the parent terminated that worker and resumed with CPU decoding. End-to-end throughput improved from roughly 6~7× real time to 15~20×.

GPU decoding produced the headline number. Process isolation and fallback made it possible to finish an archive measured in hundreds of thousands of hours.

A Video Model Adds One More Axis

An image model commonly receives a tensor shaped as $B \times C \times H \times W$. Video adds a temporal axis $T$.

$$B \times C \times T \times H \times W$$

Some preprocessing pipelines first build $B \times T \times C \times H \times W$ and permute the axes inside the model. My Video Indexer follows that pattern. It samples eight frames from every scene, stacks them as $B \times T \times C \times H \times W$, and converts them to $B \times C \times T \times H \times W$ before the vision encoder.

Filling that temporal axis is where the trouble begins. A single image requires one read. A video first needs segment boundaries, then a fixed set of frames from every segment.

The first version of the pipeline was straightforward.

video
  └─ scene split
       └─ frame extraction for every range
            └─ batch inference

Scene splitting converts adjacent frames to HSV and measures the change in Hue, Saturation, and Value. A difference above the threshold starts a new scene. Minimum and maximum scene lengths prevent tiny segments and force a cut when one scene grows too long.

If the boundaries were known in advance, the decoder could read only the frames required by the model. Finding those boundaries already requires a sequential pass. The original design decoded once for scene detection and revisited the file to collect model inputs. The algorithm naturally created redundant decoding and random seeks.

Fixing the Feed Before the Model

Profiling showed lower GPU utilization than expected. Memory was available and batch inference was fast enough. The long gaps appeared between batches while the CPU decoder prepared the next set of frames.

Optimizing the model further would have targeted the wrong stage. The kitchen was fast; the ingredients were arriving one box at a time.

Decord supports CPU and GPU contexts for VideoReader, along with get_batch for retrieving multiple frames together. Its PyPI package is CPU-only, so using NVDEC required a source build with CUDA enabled. I also patched several source compatibility issues for the FFmpeg 6.x and CUDA environment used by the service.

The GPU worker keeps two VideoReader instances open. One scans frames sequentially and computes HSV differences. The other calls get_batch as soon as a scene range becomes available. Each range is divided into eight bins and the midpoint of every bin becomes a model input, keeping the temporal dimension fixed across scenes of different lengths.

def linspace_midpoints(start, end, n=8):
    bins = np.linspace(start, end + 1, n + 1, dtype=int)
    return [
        (int(bins[i]) + int(bins[i + 1]) - 1) // 2
        for i in range(n)
    ]

The model does not run once per scene. Several scenes accumulate into a chunk and become one inference batch. The decoder also stays alive for the duration of the video instead of reopening for every segment, reducing initialization and seek overhead.

Frame delivery became about two to three times faster than the CPU path. End-to-end analysis improved from 6–7× real time to roughly 15–20×. That gain did not come from swapping one decoder alone. It also included the removal of repeated decoder setup and the steadier flow of scene batches into inference.

Same Codec, Different Failure

The faster pipeline made a second problem easier to see.

The archive contained digitally produced material such as YouTube videos, old low-resolution broadcasts, and recent high-resolution programs. Every file had gone through a basic transcoding step. With the codec and resolution reasonably normalized, I expected decoding behavior to be similarly predictable.

It was not.

Some videos ran normally until Decord reached a particular frame and stopped. There was no exception and the process did not exit. The call simply never returned. I could not group the failures by codec, resolution, or production period.

Two H.264 files can still differ in profile, level, GOP structure, timestamps, reference frames, and damaged packets. Successfully reading the beginning of a file says little about a problematic frame near the end.

From an API perspective, silence was worse than an exception. try/except only works after control returns to Python. A native decoder blocked inside a call has no Python exception to catch. I first tried putting the work in threads, but there was no safe way to terminate only the stuck native call. The server gradually acquired a collection of zombie work that nobody wanted.

Building a Boundary That Can Be Killed

The decoder eventually moved into its own process. Multiprocessing was not primarily a throughput optimization here. I needed a boundary that could be terminated without taking the model server with it.

CUDA subprocesses start with the spawn context. PyTorch's multiprocessing guidance recommends spawn or forkserver for CUDA because copying an initialized accelerator runtime with fork can cause the poison-fork problem. A new interpreter and CUDA context cost more to create, but their ownership is explicit.

The parent and GPU worker communicate through a command queue and a response queue. At startup the worker reports FPS and total frame count. During scanning it emits the current frame as a heartbeat. It also reports scene ranges and acknowledges completed get_batch requests.

Checking whether the process is alive is not enough because the failure is a live process blocked in native code. The parent records the last heartbeat timestamp. When that silence exceeds scan_watchdog_sec, it terminates the worker. A separate timeout covers frame-batch retrieval.

The watchdog pauses while the embedding model is running. Otherwise normal inference time could look like a stalled decoder. A timeout needs more than a duration; it needs a precise definition of whose time is being measured.

Shared Memory Is Not Zero-Copy

Splitting the process creates another question: how should a large frame tensor cross the boundary? Putting a NumPy array directly on a queue adds serialization and copying. Eight frames per scene multiplied by several scenes per chunk makes that cost visible.

I used Python's multiprocessing.shared_memory, available in the standard library since Python 3.8. The worker writes a batch into a fixed CPU shared-memory block. The parent opens a NumPy view over the same buffer and copies it into a CUDA tensor.

It is important not to call this a zero-copy GPU pipeline.

Decord GPU decode
  → CPU shared memory
  → CUDA tensor in the parent
  → embedding model

The data travels from GPU to CPU and back to GPU. Shared memory still avoids queue serialization and preserves process isolation. At this stage the goal was not the purest memory path. It was a recoverable decoder with enough throughput to keep the model fed.

CUDA IPC or DLPack could reduce the device-memory round trip later. They would also make ownership, lifetime, and abnormal worker exits harder to reason about. Removing one copy is not automatically cheaper if it weakens failure recovery.

Resume on CPU Instead of Starting Over

When the watchdog detects a timeout, the parent terminates and joins the GPU worker. Restarting the video from frame zero would discard all embeddings already produced. The parent therefore records the end frame of the last successfully processed scene.

OpenCV resumes scene scanning from last_end_idx + 1. Decoding moves to CPU, while HSV computation and embedding inference still use CUDA tensors. CPU fallback is not a full CPU implementation. It replaces only the decoder with a more conservative path.

0 ───────────── last successful scene │ remaining frames ───────── end
             GPU decode                │      CPU decode

Most videos finish on the fast GPU path. Only unusual inputs pay for the slower decoder. The entire archive no longer has to run at the speed of its worst file, and an operator does not have to find and restart a frozen container.

This design changed how I thought about availability. A component that never fails would be ideal. In a heterogeneous archive, deciding how far each failure can spread was more practical.

Reject Cheap Failures Early

Runtime fallback is not the first line of defense. Before analysis, the API uses ffprobe to inspect the video stream and codec. FFmpeg then decodes the first ten seconds to catch common NAL-unit errors and empty frames.

This is a cheap filter rather than a proof of integrity. It cannot detect corruption near the end of a file or a decoder hang triggered by one particular frame. The worker watchdog remains necessary even after the file passes validation.

The API also separates download failures, empty uploads, integrity failures, decoder initialization failures, missing scenes, and model errors. A single failed status is not useful in a migration this large. Failure counts by stage decide what to fix next.

A semaphore limits request concurrency. More concurrent requests do not translate directly into more throughput when every request owns a decoder, CUDA context, and GPU memory. Past a certain point, context switching and memory pressure grow faster than useful work. A fast function and a stable service are different optimization targets.

The Result That Mattered More Than 20×

The final pipeline processed video at close to 20× real time on average, typically between 15× and 20× depending on resolution, scene frequency, and batch size. These figures come from internal end-to-end measurements against the playback duration of the input.

A reproducible benchmark would record more than total video hours. Codec and resolution distributions, FPS, scene count, batch size, GPU and CPU models, and fallback rate all matter. I would also record what share of frames used CPU decoding. Without that context, 20× is a good run rather than a useful baseline.

The most satisfying result was not the benchmark. It was a log sequence where one worker stopped sending heartbeats, the watchdog closed that process, and new scenes appeared from the CPU path moments later.

Embedding quality and retrieval accuracy remain central to a Video Retrieval system. At archive scale one more question belongs beside them:

When the system meets one strange video, will the rest of the migration still be running tomorrow morning?

GPU decoding created the speed. Process isolation and fallback converted it into throughput that could survive production data. The fallback looked like a detour, but it was the shorter route to finishing the migration.