$_ CodeCosts

Computer vision is the discipline where you spend three days writing a pipeline that processes images in 12 milliseconds, then spend three weeks debugging why it fails on images taken at slightly different lighting conditions. Your code sits at the intersection of linear algebra, signal processing, deep learning, and systems engineering — and the AI coding tools that work brilliantly for web developers often produce output that is technically correct Python but functionally useless for production CV systems.

This guide evaluates every major AI coding tool through the lens of what computer vision engineers actually build: not MNIST classifiers, not tutorial-grade object detectors, but production pipelines that process millions of frames per day with strict latency requirements, handle every conceivable degradation of input quality, deploy across GPU servers and edge devices simultaneously, and maintain accuracy metrics that directly affect business outcomes (or human safety). We tested each tool on real CV tasks: building OpenCV preprocessing pipelines with proper color space handling, writing custom YOLO post-processing with NMS tuning, implementing semantic segmentation with efficient inference, building multi-camera video analytics systems, processing LiDAR point clouds, and deploying models to edge devices with quantization.

If you work primarily on training models and experiment tracking, see the ML Engineers guide. If your focus is GPU kernel optimization and shader programming, see the Graphics & GPU Programmers guide. If you work on autonomous systems that consume CV outputs, see the Robotics Engineers guide. This guide is specifically for engineers building the vision systems themselves — the pipelines that take raw pixels and produce structured understanding of the visual world.

Why Computer Vision Engineering Is Different

Computer vision engineering operates under constraints that most software engineers never encounter. Your inputs are noisy, high-dimensional, and adversarially variable. Your outputs must be both fast and accurate. And the gap between “works on the test set” and “works in production” is wider than in almost any other engineering discipline:

• Tensor shape management is a constant source of bugs: Every CV pipeline is a series of transformations between different tensor shapes, color spaces, data types, and value ranges. An image might be (H, W, 3) in BGR uint8 from OpenCV, (3, H, W) in RGB float32 for PyTorch, (1, 3, H, W) batched for inference, (H, W) grayscale for edge detection, or (N, 4) for bounding box coordinates. A single shape mismatch — forgetting to add the batch dimension, swapping H and W, leaving values in [0, 255] when the model expects [0, 1] — produces silent incorrect results rather than crashes. The image looks “fine” but your model’s accuracy drops 40% because the channels are in the wrong order. AI tools that generate image = cv2.imread(path) without immediately noting that this returns BGR, not RGB, are setting you up for a bug that takes hours to find.
• Real-world image quality is adversarial: Your training data was carefully curated with consistent lighting, resolution, and framing. Production images arrive motion-blurred, overexposed, partially occluded, shot through rain-streaked windshields, captured by $15 cameras with rolling shutter artifacts, compressed to JPEG quality 20, rotated 90 degrees with incorrect EXIF orientation, or in color spaces your pipeline has never seen. Every preprocessing step must handle these gracefully: what happens when your resize receives a 1x1 pixel image? When your normalization encounters NaN from a corrupted file? When your augmentation pipeline produces an image where the bounding box is now outside the frame? Production CV code is 30% vision algorithms and 70% defensive input handling.
• Latency budgets are measured in milliseconds, not seconds: A real-time video analytics system processing 30 fps from 16 cameras has 2 milliseconds per frame per camera for the entire pipeline: decode, preprocess, inference, post-process, and tracking. An autonomous vehicle perception stack running at 10 Hz has 100 milliseconds total for multiple models (detection, segmentation, depth estimation, lane detection) plus sensor fusion. These budgets do not include “warmup time” or “occasional GC pauses.” Every unnecessary memory allocation, every redundant copy between CPU and GPU, every Python loop that should be a vectorized operation costs you frames. AI tools that generate readable but unoptimized code — like using Python for-loops over pixels or creating intermediate NumPy arrays for every operation — produce code that is 100x too slow for production.
• GPU memory management is manual and unforgiving: A single 4K image is 24 MB in float32. A batch of 32 images at 640x640 is 157 MB. Add model weights (YOLOv8-X is 131 MB, SegFormer-B5 is 84 MB), intermediate activations, and post-processing buffers, and you are managing gigabytes of GPU memory with hard limits. Go over by one byte and your pipeline crashes — not with a helpful error, but with a CUDA out-of-memory exception that tells you nothing about which allocation pushed you over. Production CV systems need memory pooling, dynamic batch sizing based on available VRAM, graceful degradation when memory is tight, and explicit lifecycle management for every tensor. AI tools assume infinite memory because tutorial datasets fit comfortably in any GPU.
• Model deployment spans a 1000x hardware range: The same detection model might run on an NVIDIA A100 in the cloud (80 GB VRAM, 312 TFLOPS), an NVIDIA Jetson Orin on a robot (32 GB shared, 275 TOPS INT8), a Qualcomm Snapdragon in a phone (16 GB shared, 73 TOPS INT8), or a Hailo-8 accelerator in a smart camera (no local memory, 26 TOPS). Each target requires different model formats (ONNX, TensorRT, CoreML, TFLite, OpenVINO), different quantization strategies (FP16, INT8, mixed precision), different pre/post-processing implementations (CUDA vs NEON vs DSP), and different optimization techniques. Code that runs at 120 fps on an A100 runs at 0.3 fps on a Jetson Nano without optimization. AI tools generate cloud-first code that does not transfer to edge deployment.
• Evaluation metrics are domain-specific and non-obvious: mAP@0.5 is not mAP@0.5:0.95. IoU for segmentation is not the same as pixel accuracy. A 95% accurate detector that misses 5% of pedestrians is not deployable for autonomous driving. A 99.9% accurate defect detector with 0.1% false positive rate generates 100 false alerts per shift on a production line running 100,000 parts/day. Your metrics must account for class imbalance (1 defect per 10,000 good parts), per-class performance (missing a stop sign is not equivalent to missing a speed limit sign), confidence calibration (is a 0.9 confidence prediction actually correct 90% of the time?), and operational cost of errors (false positive vs false negative cost ratios). AI tools default to accuracy or generic mAP without understanding that the evaluation metric must match the deployment context.
• Data pipeline performance is the actual bottleneck: Engineers obsess over model inference speed but ignore that their data loading is the bottleneck. A training pipeline that loads JPEG images synchronously, decodes on CPU, applies augmentations sequentially, and transfers to GPU one image at a time will be 10x slower than one that uses DALI for GPU-accelerated decoding, applies augmentations on GPU, prefetches the next batch during current inference, and uses pinned memory for DMA transfers. A video analytics pipeline that calls cv2.VideoCapture().read() in a loop blocks the entire pipeline on I/O. Production systems use hardware-accelerated decoding (NVDEC), zero-copy frame sharing, and multi-stage producer-consumer architectures. AI tools generate the naive synchronous version every time.

Task Support Matrix

We tested each AI coding tool on the core tasks that define computer vision engineering work. Ratings reflect production-quality output, not tutorial-grade code:

How to read this table: Ratings reflect production-quality output for each domain. An “A” means the tool generates code that an experienced CV engineer would accept with minor edits. A “C” means the output requires substantial rewriting or demonstrates fundamental misunderstandings of CV-specific requirements. We tested with explicit, domain-specific prompts — vague prompts produce worse results across all tools.

1. OpenCV Pipeline Development

OpenCV is still the backbone of production computer vision. Even when inference runs through PyTorch or TensorRT, the preprocessing and post-processing stages typically use OpenCV for image decoding, color space conversion, geometric transformations, filtering, and visualization. Getting these operations right — in the correct order, with the correct data types, in the correct color space — is where AI tools either save time or introduce subtle bugs.

The Color Space and Data Type Minefield

The single most common bug in CV code generated by AI tools is color space confusion. OpenCV reads images in BGR. Matplotlib displays in RGB. PyTorch models expect RGB. PIL opens in RGB. TensorFlow expects RGB. Mixing these produces images that look slightly off to humans but catastrophically confuse models trained on correctly ordered channels.

Here is a production preprocessing pipeline that handles the real-world complexity AI tools typically miss:

import cv2
import numpy as np
from pathlib import Path
from dataclasses import dataclass
from typing import Optional, Tuple


@dataclass
class PreprocessConfig:
    """Configuration for image preprocessing pipeline."""
    target_size: Tuple[int, int] = (640, 640)  # (width, height)
    color_space: str = "RGB"  # Output color space
    normalize: bool = True
    mean: Tuple[float, ...] = (0.485, 0.456, 0.406)  # ImageNet mean (RGB)
    std: Tuple[float, ...] = (0.229, 0.224, 0.225)    # ImageNet std (RGB)
    pad_value: int = 114  # Gray padding for letterbox
    preserve_aspect: bool = True


class ImagePreprocessor:
    """Production image preprocessor with correct color space handling.

    Handles the BGR/RGB/grayscale transitions that AI tools routinely
    get wrong. Every conversion is explicit and documented.
    """

    def __init__(self, config: PreprocessConfig):
        self.config = config
        # Pre-compute normalization arrays for vectorized ops
        if config.normalize:
            self._mean = np.array(config.mean, dtype=np.float32).reshape(1, 1, 3)
            self._std = np.array(config.std, dtype=np.float32).reshape(1, 1, 3)

    def load_image(self, path: str) -> Optional[np.ndarray]:
        """Load image with robust error handling.

        Returns BGR uint8 numpy array, or None if loading fails.
        Handles EXIF orientation, corrupted files, and unusual formats.
        """
        path = str(path)

        # cv2.imread silently returns None for missing/corrupted files
        img = cv2.imread(path, cv2.IMREAD_COLOR)
        if img is None:
            # Try with IMREAD_UNCHANGED for unusual formats (16-bit, HDR)
            img = cv2.imread(path, cv2.IMREAD_UNCHANGED)
            if img is None:
                return None
            # Convert unusual formats to standard BGR uint8
            if img.dtype == np.uint16:
                img = (img / 256).astype(np.uint8)
            elif img.dtype == np.float32:
                img = (np.clip(img, 0, 1) * 255).astype(np.uint8)
            if len(img.shape) == 2:
                img = cv2.cvtColor(img, cv2.COLOR_GRAY2BGR)
            elif img.shape[2] == 4:
                img = cv2.cvtColor(img, cv2.COLOR_BGRA2BGR)

        # Handle EXIF orientation without external dependencies
        # OpenCV 4.x respects EXIF by default with IMREAD_COLOR,
        # but verify for edge cases with manual rotation
        if img.shape[0] == 0 or img.shape[1] == 0:
            return None

        return img  # BGR uint8, guaranteed

    def letterbox(
        self, img: np.ndarray
    ) -> Tuple[np.ndarray, float, Tuple[int, int]]:
        """Resize with aspect ratio preservation (letterbox).

        Returns:
            resized: Letterboxed image (BGR uint8)
            scale: Scale factor applied
            pad: (pad_w, pad_h) padding applied
        """
        h, w = img.shape[:2]
        target_w, target_h = self.config.target_size

        # Compute scale to fit within target while preserving aspect
        scale = min(target_w / w, target_h / h)

        new_w = int(round(w * scale))
        new_h = int(round(h * scale))

        # Choose interpolation based on scale direction
        interp = cv2.INTER_LINEAR if scale > 1 else cv2.INTER_AREA

        resized = cv2.resize(img, (new_w, new_h), interpolation=interp)

        # Compute padding (center the image)
        pad_w = (target_w - new_w) // 2
        pad_h = (target_h - new_h) // 2

        # Create padded canvas
        canvas = np.full(
            (target_h, target_w, 3),
            self.config.pad_value,
            dtype=np.uint8
        )
        canvas[pad_h:pad_h + new_h, pad_w:pad_w + new_w] = resized

        return canvas, scale, (pad_w, pad_h)

    def preprocess(
        self, img: np.ndarray
    ) -> Tuple[np.ndarray, dict]:
        """Full preprocessing pipeline: resize -> color -> normalize.

        Args:
            img: BGR uint8 numpy array from load_image()

        Returns:
            processed: Float32 array in (C, H, W) format, normalized
            meta: Dict with scale, padding, original size for post-processing
        """
        original_h, original_w = img.shape[:2]

        # Step 1: Resize (letterbox or direct)
        if self.config.preserve_aspect:
            resized, scale, (pad_w, pad_h) = self.letterbox(img)
        else:
            target_w, target_h = self.config.target_size
            resized = cv2.resize(img, (target_w, target_h))
            scale = min(target_w / original_w, target_h / original_h)
            pad_w, pad_h = 0, 0

        # Step 2: BGR -> RGB (explicit, not optional)
        # This is where AI tools most commonly introduce bugs
        if self.config.color_space == "RGB":
            converted = cv2.cvtColor(resized, cv2.COLOR_BGR2RGB)
        elif self.config.color_space == "BGR":
            converted = resized
        elif self.config.color_space == "GRAY":
            converted = cv2.cvtColor(resized, cv2.COLOR_BGR2GRAY)
            converted = converted[:, :, np.newaxis]  # Keep 3D shape
        else:
            raise ValueError(f"Unsupported color space: {self.config.color_space}")

        # Step 3: uint8 [0, 255] -> float32 [0, 1]
        # Do NOT use img.astype(np.float32) without dividing - common AI tool bug
        normalized = converted.astype(np.float32) / 255.0

        # Step 4: ImageNet normalization (if enabled)
        if self.config.normalize:
            normalized = (normalized - self._mean) / self._std

        # Step 5: HWC -> CHW for PyTorch
        # np.transpose is zero-copy; np.moveaxis creates a copy
        chw = np.ascontiguousarray(np.transpose(normalized, (2, 0, 1)))

        meta = {
            "original_size": (original_w, original_h),
            "scale": scale,
            "pad": (pad_w, pad_h),
            "target_size": self.config.target_size,
        }

        return chw, meta

    def reverse_letterbox(
        self, boxes: np.ndarray, meta: dict
    ) -> np.ndarray:
        """Map detection boxes back to original image coordinates.

        Args:
            boxes: (N, 4) array in xyxy format, in preprocessed coordinates
            meta: Metadata dict from preprocess()

        Returns:
            (N, 4) array in xyxy format, in original image coordinates
        """
        if len(boxes) == 0:
            return boxes

        pad_w, pad_h = meta["pad"]
        scale = meta["scale"]
        orig_w, orig_h = meta["original_size"]

        # Remove padding offset, then un-scale
        boxes = boxes.copy()
        boxes[:, [0, 2]] = (boxes[:, [0, 2]] - pad_w) / scale
        boxes[:, [1, 3]] = (boxes[:, [1, 3]] - pad_h) / scale

        # Clip to original image bounds
        boxes[:, [0, 2]] = np.clip(boxes[:, [0, 2]], 0, orig_w)
        boxes[:, [1, 3]] = np.clip(boxes[:, [1, 3]], 0, orig_h)

        return boxes

What AI tools get wrong: Cursor and Claude Code both handle the basic BGR-to-RGB conversion correctly when prompted explicitly. The difference appears in edge cases: Cursor sometimes generates img[:, :, ::-1] for BGR-to-RGB conversion, which is technically correct but creates a non-contiguous array that causes performance issues downstream. Claude Code more consistently generates cv2.cvtColor calls and correctly warns about the float32 conversion step. Copilot frequently omits the normalization division by 255 when the surrounding code does not make the expected range obvious. None of the tools consistently generate the reverse-letterbox function correctly on the first attempt — the padding offset subtraction before scale division is a common source of errors.

2. Object Detection & YOLO

Object detection is the workhorse of production computer vision. YOLO (currently at v11/v12 via Ultralytics, plus numerous forks) dominates real-time detection, but production deployment involves far more than calling model.predict(). Custom post-processing, NMS tuning, confidence thresholding per class, tracking integration, and batch inference optimization are where CV engineers spend their time.

Production YOLO Inference with Custom NMS

The Ultralytics high-level API is fine for prototyping, but production systems need direct control over the inference pipeline for performance and customization:

import numpy as np
import onnxruntime as ort
from dataclasses import dataclass, field
from typing import List, Tuple, Optional


@dataclass
class DetectionResult:
    """Single detection result with full metadata."""
    bbox: np.ndarray        # (4,) xyxy in original image coords
    confidence: float       # Detection confidence
    class_id: int           # Class index
    class_name: str         # Human-readable class name


@dataclass
class DetectorConfig:
    """Per-deployment detection configuration."""
    model_path: str
    class_names: List[str]
    input_size: Tuple[int, int] = (640, 640)
    conf_threshold: float = 0.25
    nms_iou_threshold: float = 0.45
    max_detections: int = 300
    # Per-class confidence overrides (e.g., higher threshold for noisy classes)
    class_conf_overrides: dict = field(default_factory=dict)


class YOLODetector:
    """Production YOLO detector with ONNX Runtime.

    Why ONNX Runtime instead of PyTorch:
    - 2-3x faster inference (no Python overhead, graph optimization)
    - Consistent behavior across deployment targets
    - No PyTorch dependency in production containers (saves 2GB+)
    - TensorRT EP for GPU, OpenVINO EP for Intel, NNAPI EP for Android
    """

    def __init__(self, config: DetectorConfig):
        self.config = config
        self._class_thresholds = self._build_class_thresholds()

        # Session options for production
        opts = ort.SessionOptions()
        opts.graph_optimization_level = ort.GraphOptimizationLevel.ORT_ENABLE_ALL
        opts.intra_op_num_threads = 4
        opts.inter_op_num_threads = 1  # Single stream, no contention
        opts.enable_mem_pattern = True
        opts.enable_cpu_mem_arena = True

        # Prefer GPU if available, fall back to CPU
        providers = []
        if "CUDAExecutionProvider" in ort.get_available_providers():
            providers.append(("CUDAExecutionProvider", {
                "device_id": 0,
                "arena_extend_strategy": "kSameAsRequested",
                "gpu_mem_limit": 2 * 1024 * 1024 * 1024,  # 2GB cap
                "cudnn_conv_algo_search": "EXHAUSTIVE",
            }))
        providers.append("CPUExecutionProvider")

        self.session = ort.InferenceSession(
            config.model_path, opts, providers=providers
        )

        # Cache input/output metadata
        self._input_name = self.session.get_inputs()[0].name
        self._output_names = [o.name for o in self.session.get_outputs()]

    def _build_class_thresholds(self) -> np.ndarray:
        """Build per-class confidence threshold array."""
        n_classes = len(self.config.class_names)
        thresholds = np.full(n_classes, self.config.conf_threshold, dtype=np.float32)
        for cls_id, threshold in self.config.class_conf_overrides.items():
            if 0 <= cls_id < n_classes:
                thresholds[cls_id] = threshold
        return thresholds

    def detect(
        self, image_chw: np.ndarray, meta: dict
    ) -> List[DetectionResult]:
        """Run detection on a preprocessed image.

        Args:
            image_chw: (3, H, W) float32 preprocessed image
            meta: Preprocessing metadata (from ImagePreprocessor.preprocess)

        Returns:
            List of DetectionResult in original image coordinates
        """
        # Add batch dimension: (3, H, W) -> (1, 3, H, W)
        batch = image_chw[np.newaxis, ...]

        # Run inference
        outputs = self.session.run(self._output_names, {self._input_name: batch})

        # YOLO output shape: (1, num_classes + 4, num_predictions)
        # Transpose to (num_predictions, num_classes + 4) for easier handling
        raw = outputs[0][0].T  # Remove batch dim, transpose

        # Split into boxes and class scores
        boxes_xywh = raw[:, :4]       # (N, 4) center_x, center_y, w, h
        class_scores = raw[:, 4:]     # (N, num_classes)

        # Get best class per prediction
        class_ids = np.argmax(class_scores, axis=1)
        confidences = class_scores[np.arange(len(class_ids)), class_ids]

        # Apply per-class confidence thresholds
        per_class_thresh = self._class_thresholds[class_ids]
        mask = confidences > per_class_thresh

        boxes_xywh = boxes_xywh[mask]
        class_ids = class_ids[mask]
        confidences = confidences[mask]

        if len(boxes_xywh) == 0:
            return []

        # xywh -> xyxy
        boxes_xyxy = self._xywh_to_xyxy(boxes_xywh)

        # Class-aware NMS (standard for multi-class detection)
        keep = self._nms_class_aware(
            boxes_xyxy, confidences, class_ids,
            self.config.nms_iou_threshold,
            self.config.max_detections
        )

        boxes_xyxy = boxes_xyxy[keep]
        class_ids = class_ids[keep]
        confidences = confidences[keep]

        # Map boxes back to original image coordinates
        from_preprocessor = self._reverse_coords(boxes_xyxy, meta)

        # Build results
        results = []
        for i in range(len(from_preprocessor)):
            results.append(DetectionResult(
                bbox=from_preprocessor[i],
                confidence=float(confidences[i]),
                class_id=int(class_ids[i]),
                class_name=self.config.class_names[class_ids[i]],
            ))

        return results

    @staticmethod
    def _xywh_to_xyxy(boxes: np.ndarray) -> np.ndarray:
        """Convert (cx, cy, w, h) to (x1, y1, x2, y2)."""
        xyxy = np.empty_like(boxes)
        half_w = boxes[:, 2] / 2
        half_h = boxes[:, 3] / 2
        xyxy[:, 0] = boxes[:, 0] - half_w
        xyxy[:, 1] = boxes[:, 1] - half_h
        xyxy[:, 2] = boxes[:, 0] + half_w
        xyxy[:, 3] = boxes[:, 1] + half_h
        return xyxy

    @staticmethod
    def _nms_class_aware(
        boxes: np.ndarray,
        scores: np.ndarray,
        class_ids: np.ndarray,
        iou_threshold: float,
        max_dets: int,
    ) -> np.ndarray:
        """Class-aware NMS using per-class offset trick.

        Offset boxes by class_id * max_coordinate to prevent
        cross-class suppression. This is the standard approach
        used by torchvision.ops.batched_nms.
        """
        if len(boxes) == 0:
            return np.array([], dtype=np.int64)

        # Offset boxes by class to prevent cross-class suppression
        max_coord = boxes.max() + 1
        offsets = class_ids.astype(np.float32) * max_coord
        offset_boxes = boxes + offsets[:, np.newaxis]

        # Standard greedy NMS
        x1 = offset_boxes[:, 0]
        y1 = offset_boxes[:, 1]
        x2 = offset_boxes[:, 2]
        y2 = offset_boxes[:, 3]
        areas = (x2 - x1) * (y2 - y1)

        order = scores.argsort()[::-1]
        keep = []

        while len(order) > 0 and len(keep) < max_dets:
            i = order[0]
            keep.append(i)

            if len(order) == 1:
                break

            # Compute IoU with remaining boxes
            xx1 = np.maximum(x1[i], x1[order[1:]])
            yy1 = np.maximum(y1[i], y1[order[1:]])
            xx2 = np.minimum(x2[i], x2[order[1:]])
            yy2 = np.minimum(y2[i], y2[order[1:]])

            inter = np.maximum(0, xx2 - xx1) * np.maximum(0, yy2 - yy1)
            union = areas[i] + areas[order[1:]] - inter
            iou = inter / (union + 1e-6)

            remaining = np.where(iou <= iou_threshold)[0]
            order = order[remaining + 1]

        return np.array(keep, dtype=np.int64)

    @staticmethod
    def _reverse_coords(
        boxes: np.ndarray, meta: dict
    ) -> np.ndarray:
        """Reverse letterbox transformation on detection boxes."""
        if len(boxes) == 0:
            return boxes

        pad_w, pad_h = meta["pad"]
        scale = meta["scale"]
        orig_w, orig_h = meta["original_size"]

        result = boxes.copy()
        result[:, [0, 2]] = (result[:, [0, 2]] - pad_w) / scale
        result[:, [1, 3]] = (result[:, [1, 3]] - pad_h) / scale
        result[:, [0, 2]] = np.clip(result[:, [0, 2]], 0, orig_w)
        result[:, [1, 3]] = np.clip(result[:, [1, 3]], 0, orig_h)

        return result

Tool comparison: Claude Code excels at generating the complete detection pipeline including ONNX Runtime session configuration, per-class thresholding, and class-aware NMS. When asked to “write a YOLO detector,” it produces the full pipeline rather than just model = YOLO("yolov8n.pt"); results = model(img). Cursor handles the ONNX output parsing well when given context about the output tensor shape, but sometimes gets the transpose wrong (outputting (4+C, N) when it should be (N, 4+C) or vice versa). Copilot tends toward the Ultralytics high-level API, which is fine for prototyping but not for production where you need control over the inference provider, memory limits, and post-processing details.

3. Image Segmentation

Segmentation — semantic, instance, and panoptic — requires handling mask tensors that are larger than detection outputs by orders of magnitude. A single segmentation mask at 1024x1024 with 21 classes is 88 MB in float32. Batch these, add intermediate activations, and you are managing gigabytes of data that must flow through your pipeline without unnecessary copies.

Efficient Segmentation Post-Processing

import cv2
import numpy as np
from typing import List, Tuple, Optional, Dict


class SegmentationProcessor:
    """Post-processing for semantic and instance segmentation models.

    Handles the mask operations that AI tools consistently get wrong:
    - Argmax vs sigmoid thresholding (semantic vs binary)
    - Mask resizing with correct interpolation (NEAREST, not BILINEAR)
    - Connected component analysis for instance separation
    - Efficient polygon extraction from binary masks
    """

    @staticmethod
    def process_semantic_output(
        logits: np.ndarray,
        original_size: Tuple[int, int],
        class_names: List[str],
        ignore_index: int = 255,
    ) -> Dict:
        """Process semantic segmentation model output.

        Args:
            logits: (C, H, W) or (1, C, H, W) raw model output
            original_size: (width, height) of original image
            class_names: List of class names indexed by class ID
            ignore_index: Value for unlabeled pixels

        Returns:
            Dict with mask, per-class areas, and confidence map
        """
        # Remove batch dimension if present
        if logits.ndim == 4:
            logits = logits[0]

        num_classes, h, w = logits.shape
        target_w, target_h = original_size

        # Argmax to get class predictions
        # DO NOT apply softmax before argmax - it doesn't change the result
        # and wastes computation. Softmax is only needed for confidence values.
        class_mask = np.argmax(logits, axis=0).astype(np.uint8)  # (H, W)

        # Confidence: softmax then take max
        # Use numerically stable softmax
        shifted = logits - logits.max(axis=0, keepdims=True)
        exp = np.exp(shifted)
        softmax = exp / exp.sum(axis=0, keepdims=True)
        confidence = softmax.max(axis=0)  # (H, W)

        # Resize to original dimensions
        # CRITICAL: Use INTER_NEAREST for class masks (not INTER_LINEAR!)
        # Linear interpolation creates invalid intermediate class IDs
        class_mask_full = cv2.resize(
            class_mask, (target_w, target_h),
            interpolation=cv2.INTER_NEAREST
        )

        # Confidence can use bilinear (it's continuous)
        confidence_full = cv2.resize(
            confidence.astype(np.float32), (target_w, target_h),
            interpolation=cv2.INTER_LINEAR
        )

        # Per-class pixel counts and area fractions
        total_pixels = target_w * target_h
        class_areas = {}
        for cls_id in range(num_classes):
            if cls_id == ignore_index:
                continue
            count = int(np.sum(class_mask_full == cls_id))
            if count > 0:
                class_areas[class_names[cls_id]] = {
                    "pixels": count,
                    "fraction": count / total_pixels,
                    "mean_confidence": float(
                        confidence_full[class_mask_full == cls_id].mean()
                    ),
                }

        return {
            "class_mask": class_mask_full,       # (H, W) uint8
            "confidence": confidence_full,        # (H, W) float32
            "class_areas": class_areas,
            "num_classes": num_classes,
        }

    @staticmethod
    def masks_to_polygons(
        binary_mask: np.ndarray,
        min_area: int = 100,
        simplify_epsilon: float = 2.0,
    ) -> List[np.ndarray]:
        """Extract simplified polygons from a binary mask.

        Used for: COCO-format annotation export, vector overlay rendering,
        and area calculations that need sub-pixel precision.

        Args:
            binary_mask: (H, W) uint8 with 0/255 values
            min_area: Minimum contour area to keep (filters noise)
            simplify_epsilon: Douglas-Peucker simplification tolerance

        Returns:
            List of (N, 2) polygon arrays in (x, y) format
        """
        # Ensure correct type for findContours
        if binary_mask.dtype != np.uint8:
            binary_mask = (binary_mask > 0).astype(np.uint8) * 255

        contours, hierarchy = cv2.findContours(
            binary_mask, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE
        )

        polygons = []
        for contour in contours:
            area = cv2.contourArea(contour)
            if area < min_area:
                continue

            # Simplify polygon (reduces point count by ~80%)
            simplified = cv2.approxPolyDP(contour, simplify_epsilon, closed=True)

            # Reshape from (N, 1, 2) to (N, 2)
            polygon = simplified.reshape(-1, 2)

            # Need at least 3 points for a valid polygon
            if len(polygon) >= 3:
                polygons.append(polygon)

        return polygons

    @staticmethod
    def instance_from_semantic(
        class_mask: np.ndarray,
        target_class_id: int,
        min_instance_area: int = 500,
    ) -> List[Dict]:
        """Separate individual instances from semantic segmentation.

        When you have semantic segmentation but need instance-level results
        (e.g., counting individual objects), connected component analysis
        is the standard approach.
        """
        # Extract binary mask for target class
        binary = (class_mask == target_class_id).astype(np.uint8)

        # Morphological operations to clean up before component analysis
        kernel = cv2.getStructuringElement(cv2.MORPH_ELLIPSE, (5, 5))
        binary = cv2.morphologyEx(binary, cv2.MORPH_CLOSE, kernel)
        binary = cv2.morphologyEx(binary, cv2.MORPH_OPEN, kernel)

        # Connected component analysis
        num_labels, labels, stats, centroids = cv2.connectedComponentsWithStats(
            binary, connectivity=8
        )

        instances = []
        for i in range(1, num_labels):  # Skip background (label 0)
            area = stats[i, cv2.CC_STAT_AREA]
            if area < min_instance_area:
                continue

            x = stats[i, cv2.CC_STAT_LEFT]
            y = stats[i, cv2.CC_STAT_TOP]
            w = stats[i, cv2.CC_STAT_WIDTH]
            h = stats[i, cv2.CC_STAT_HEIGHT]

            instances.append({
                "instance_id": i,
                "bbox": np.array([x, y, x + w, y + h]),
                "area": area,
                "centroid": centroids[i],
                "mask": (labels == i).astype(np.uint8),
            })

        return instances

Critical detail AI tools miss: The interpolation method for mask resizing. Every AI tool we tested occasionally generates cv2.resize(mask, size, interpolation=cv2.INTER_LINEAR) for class masks. Linear interpolation averages neighboring pixel values, which creates invalid intermediate class IDs (e.g., averaging class 3 and class 7 produces class 5, which means something completely different). Class masks must use INTER_NEAREST. Confidence maps can use INTER_LINEAR because they are continuous values. This distinction is fundamental to segmentation but absent from most AI tool training data.

4. Video Analytics & Multi-Object Tracking

Video analytics multiplies every CV challenge by the frame rate. A 30 fps stream produces 108,000 frames per hour, each requiring detection, tracking association, state management, and event logic. The tracking component — maintaining consistent object identities across frames despite occlusions, camera motion, and appearance changes — is where most production video systems spend their engineering effort.

Production Video Pipeline with ByteTrack-style MOT

import numpy as np
from collections import defaultdict
from dataclasses import dataclass, field
from typing import List, Dict, Optional, Deque
from collections import deque


@dataclass
class Track:
    """Active object track with state history."""
    track_id: int
    class_id: int
    bbox: np.ndarray            # Current xyxy
    confidence: float
    age: int = 0                # Frames since creation
    hits: int = 1               # Successful associations
    misses: int = 0             # Consecutive missed frames
    history: Deque = field(default_factory=lambda: deque(maxlen=90))
    velocity: np.ndarray = field(
        default_factory=lambda: np.zeros(4, dtype=np.float32)
    )

    @property
    def is_confirmed(self) -> bool:
        """Track must be seen in multiple frames to be confirmed."""
        return self.hits >= 3

    def predict(self) -> np.ndarray:
        """Predict next position using constant velocity model."""
        return self.bbox + self.velocity


class SimpleTracker:
    """Production multi-object tracker using IoU association.

    Implements the core logic of ByteTrack: two-stage association
    with high and low confidence detections. This is the approach
    that works best in production without requiring a separate
    Re-ID model (which adds latency and complexity).
    """

    def __init__(
        self,
        max_age: int = 30,
        min_hits: int = 3,
        iou_threshold: float = 0.3,
        low_conf_threshold: float = 0.1,
    ):
        self.max_age = max_age
        self.min_hits = min_hits
        self.iou_threshold = iou_threshold
        self.low_conf_threshold = low_conf_threshold
        self.tracks: List[Track] = []
        self._next_id = 1

    def update(
        self, detections: List, frame_idx: int
    ) -> List[Track]:
        """Update tracks with new detections.

        Args:
            detections: List of DetectionResult from YOLODetector
            frame_idx: Current frame index (for history)

        Returns:
            List of confirmed tracks with current positions
        """
        if not detections:
            # Age all tracks, remove dead ones
            self._age_tracks()
            return [t for t in self.tracks if t.is_confirmed]

        # Split detections by confidence
        det_boxes = np.array([d.bbox for d in detections])
        det_scores = np.array([d.confidence for d in detections])
        det_classes = np.array([d.class_id for d in detections])

        high_mask = det_scores >= 0.5
        low_mask = (det_scores >= self.low_conf_threshold) & ~high_mask

        # Stage 1: Associate high-confidence detections with existing tracks
        if self.tracks and high_mask.any():
            track_boxes = np.array([t.predict() for t in self.tracks])
            high_boxes = det_boxes[high_mask]

            iou_matrix = self._compute_iou_matrix(track_boxes, high_boxes)

            matched_tracks, matched_dets, unmatched_tracks = (
                self._hungarian_match(iou_matrix, self.iou_threshold)
            )

            # Update matched tracks
            high_indices = np.where(high_mask)[0]
            for t_idx, d_idx in zip(matched_tracks, matched_dets):
                orig_d_idx = high_indices[d_idx]
                self._update_track(
                    self.tracks[t_idx], detections[orig_d_idx], frame_idx
                )
        else:
            unmatched_tracks = list(range(len(self.tracks)))
            high_indices = np.where(high_mask)[0]

        # Stage 2: Associate low-confidence detections with remaining tracks
        if unmatched_tracks and low_mask.any():
            remaining_tracks = [self.tracks[i] for i in unmatched_tracks]
            remaining_boxes = np.array([t.predict() for t in remaining_tracks])
            low_boxes = det_boxes[low_mask]

            iou_matrix = self._compute_iou_matrix(remaining_boxes, low_boxes)
            matched_t2, matched_d2, still_unmatched = (
                self._hungarian_match(iou_matrix, self.iou_threshold)
            )

            low_indices = np.where(low_mask)[0]
            for t_idx, d_idx in zip(matched_t2, matched_d2):
                orig_t_idx = unmatched_tracks[t_idx]
                orig_d_idx = low_indices[d_idx]
                self._update_track(
                    self.tracks[orig_t_idx], detections[orig_d_idx], frame_idx
                )

        # Create new tracks from unmatched high-confidence detections
        matched_high = set()
        if self.tracks and high_mask.any():
            # Collect which high detections were matched
            for _, d_idx in zip(matched_tracks, matched_dets):
                matched_high.add(high_indices[d_idx])

        for idx in np.where(high_mask)[0]:
            if idx not in matched_high:
                self._create_track(detections[idx], frame_idx)

        # Age unmatched tracks
        self._age_tracks()

        return [t for t in self.tracks if t.is_confirmed]

    def _update_track(
        self, track: Track, detection, frame_idx: int
    ):
        """Update track state with new detection."""
        old_center = (track.bbox[:2] + track.bbox[2:]) / 2
        new_center = (detection.bbox[:2] + detection.bbox[2:]) / 2

        # Exponential moving average for velocity (smooth, not jerky)
        alpha = 0.4
        raw_velocity = detection.bbox - track.bbox
        track.velocity = alpha * raw_velocity + (1 - alpha) * track.velocity

        track.bbox = detection.bbox.copy()
        track.confidence = detection.confidence
        track.hits += 1
        track.misses = 0
        track.age += 1
        track.history.append({
            "frame": frame_idx,
            "bbox": detection.bbox.copy(),
            "confidence": detection.confidence,
        })

    def _create_track(self, detection, frame_idx: int):
        """Create new track from unmatched detection."""
        track = Track(
            track_id=self._next_id,
            class_id=detection.class_id,
            bbox=detection.bbox.copy(),
            confidence=detection.confidence,
        )
        track.history.append({
            "frame": frame_idx,
            "bbox": detection.bbox.copy(),
            "confidence": detection.confidence,
        })
        self.tracks.append(track)
        self._next_id += 1

    def _age_tracks(self):
        """Increment miss count and remove dead tracks."""
        alive = []
        for track in self.tracks:
            track.misses += 1
            track.age += 1
            if track.misses <= self.max_age:
                alive.append(track)
        self.tracks = alive

    @staticmethod
    def _compute_iou_matrix(
        boxes_a: np.ndarray, boxes_b: np.ndarray
    ) -> np.ndarray:
        """Compute IoU between two sets of boxes.

        Args:
            boxes_a: (M, 4) xyxy format
            boxes_b: (N, 4) xyxy format

        Returns:
            (M, N) IoU matrix
        """
        m, n = len(boxes_a), len(boxes_b)
        iou = np.zeros((m, n), dtype=np.float32)

        for i in range(m):
            xx1 = np.maximum(boxes_a[i, 0], boxes_b[:, 0])
            yy1 = np.maximum(boxes_a[i, 1], boxes_b[:, 1])
            xx2 = np.minimum(boxes_a[i, 2], boxes_b[:, 2])
            yy2 = np.minimum(boxes_a[i, 3], boxes_b[:, 3])

            inter = np.maximum(0, xx2 - xx1) * np.maximum(0, yy2 - yy1)

            area_a = (
                (boxes_a[i, 2] - boxes_a[i, 0]) *
                (boxes_a[i, 3] - boxes_a[i, 1])
            )
            area_b = (
                (boxes_b[:, 2] - boxes_b[:, 0]) *
                (boxes_b[:, 3] - boxes_b[:, 1])
            )

            union = area_a + area_b - inter
            iou[i] = inter / (union + 1e-6)

        return iou

    @staticmethod
    def _hungarian_match(
        cost_matrix: np.ndarray, threshold: float
    ):
        """Greedy matching (fast approximation of Hungarian algorithm).

        For real-time tracking, greedy matching is often preferred over
        scipy.optimize.linear_sum_assignment because:
        - It's O(NM) vs O(N^3) for Hungarian
        - The quality difference is negligible for IoU-based matching
        - No scipy dependency in edge deployments
        """
        if cost_matrix.size == 0:
            return [], [], list(range(cost_matrix.shape[0]))

        matched_rows = []
        matched_cols = []
        used_cols = set()

        # Sort by IoU (descending) and greedily match
        rows, cols = np.unravel_index(
            np.argsort(-cost_matrix.ravel()), cost_matrix.shape
        )

        for r, c in zip(rows, cols):
            if cost_matrix[r, c] < threshold:
                break
            if r not in set(matched_rows) and c not in used_cols:
                matched_rows.append(r)
                matched_cols.append(c)
                used_cols.add(c)

        unmatched_rows = [
            i for i in range(cost_matrix.shape[0])
            if i not in set(matched_rows)
        ]

        return matched_rows, matched_cols, unmatched_rows

Tool performance on tracking: Claude Code produces the best tracking architectures, correctly implementing the two-stage high/low confidence association that makes ByteTrack effective. It understands why constant velocity prediction matters for occluded objects and generates the EMA velocity smoothing without prompting. Cursor generates correct IoU computation but tends toward the simpler single-stage association, missing the low-confidence recovery stage that reduces ID switches by 20–30%. Copilot struggles with the overall tracking architecture — it generates individual functions well but does not maintain the stateful track lifecycle logic coherently across the full class.

5. Point Cloud & 3D Vision

Point cloud processing is where AI coding tools hit their steepest capability cliff. The training data for Open3D, PCL bindings, and 3D geometry operations is orders of magnitude smaller than for 2D vision. Expect more manual correction in this domain.

LiDAR Point Cloud Processing Pipeline

import numpy as np
from typing import Tuple, Optional, List, Dict


class PointCloudProcessor:
    """Production point cloud processing for LiDAR data.

    Handles the 3D processing tasks that AI tools have the least
    training data for: voxel downsampling, ground plane removal,
    clustering, and bounding box estimation from point clouds.
    """

    @staticmethod
    def voxel_downsample(
        points: np.ndarray,
        voxel_size: float = 0.1,
    ) -> np.ndarray:
        """Voxel grid downsampling (pure NumPy, no Open3D needed).

        Reduces point cloud density uniformly. Essential for real-time
        processing where raw LiDAR produces 100K+ points per scan.

        Args:
            points: (N, 3+) array, first 3 columns are xyz
            voxel_size: Size of each voxel in meters

        Returns:
            Downsampled points (centroids of occupied voxels)
        """
        # Quantize to voxel grid
        voxel_indices = np.floor(points[:, :3] / voxel_size).astype(np.int32)

        # Use structured array for unique voxel identification
        # This is faster than converting to tuples or string hashing
        dt = np.dtype([("x", np.int32), ("y", np.int32), ("z", np.int32)])
        structured = np.array(
            list(zip(voxel_indices[:, 0], voxel_indices[:, 1], voxel_indices[:, 2])),
            dtype=dt,
        )

        _, inverse, counts = np.unique(
            structured, return_inverse=True, return_counts=True
        )

        # Compute centroids per voxel using bincount
        num_voxels = len(counts)
        centroids = np.zeros((num_voxels, points.shape[1]), dtype=np.float32)

        for col in range(points.shape[1]):
            centroids[:, col] = np.bincount(
                inverse, weights=points[:, col], minlength=num_voxels
            ) / counts

        return centroids

    @staticmethod
    def remove_ground_plane(
        points: np.ndarray,
        n_iterations: int = 100,
        distance_threshold: float = 0.2,
        n_sample: int = 3,
    ) -> Tuple[np.ndarray, np.ndarray, np.ndarray]:
        """RANSAC ground plane removal.

        Essential preprocessing for object detection in LiDAR data.
        Points below the ground plane are noise; points on it are not
        objects of interest.

        Args:
            points: (N, 3+) point cloud
            n_iterations: RANSAC iterations
            distance_threshold: Max distance from plane to be inlier (meters)
            n_sample: Points to sample per iteration (3 for plane)

        Returns:
            ground_points, non_ground_points, plane_coefficients (a,b,c,d)
        """
        best_inliers = None
        best_count = 0
        best_plane = None
        xyz = points[:, :3]
        n_points = len(xyz)

        for _ in range(n_iterations):
            # Sample 3 random points
            indices = np.random.choice(n_points, n_sample, replace=False)
            sample = xyz[indices]

            # Fit plane: ax + by + cz + d = 0
            v1 = sample[1] - sample[0]
            v2 = sample[2] - sample[0]
            normal = np.cross(v1, v2)
            norm = np.linalg.norm(normal)

            if norm < 1e-8:
                continue  # Degenerate (collinear points)

            normal = normal / norm
            d = -np.dot(normal, sample[0])

            # Check: ground planes should be roughly horizontal
            # (normal vector approximately parallel to z-axis)
            if abs(normal[2]) < 0.7:
                continue  # Not a ground plane candidate

            # Compute distances
            distances = np.abs(xyz @ normal + d)
            inlier_mask = distances < distance_threshold
            inlier_count = inlier_mask.sum()

            if inlier_count > best_count:
                best_count = inlier_count
                best_inliers = inlier_mask
                best_plane = np.append(normal, d)

        if best_inliers is None:
            # No ground plane found - return all as non-ground
            return (
                np.empty((0, points.shape[1])),
                points,
                np.array([0, 0, 1, 0], dtype=np.float32),
            )

        return (
            points[best_inliers],
            points[~best_inliers],
            best_plane,
        )

    @staticmethod
    def euclidean_cluster(
        points: np.ndarray,
        eps: float = 0.5,
        min_points: int = 10,
    ) -> List[np.ndarray]:
        """Euclidean clustering using grid-based spatial indexing.

        Groups nearby points into clusters representing individual objects.
        This is the standard approach for object proposal generation
        from LiDAR point clouds.

        Args:
            points: (N, 3+) non-ground points
            eps: Maximum distance between cluster neighbors (meters)
            min_points: Minimum points to form a cluster

        Returns:
            List of point arrays, one per cluster
        """
        from scipy.spatial import cKDTree

        if len(points) == 0:
            return []

        xyz = points[:, :3]
        tree = cKDTree(xyz)

        visited = np.zeros(len(xyz), dtype=bool)
        clusters = []

        for i in range(len(xyz)):
            if visited[i]:
                continue

            # BFS from this seed point
            neighbors = tree.query_ball_point(xyz[i], eps)
            if len(neighbors) < min_points:
                visited[i] = True
                continue

            cluster_indices = []
            queue = list(neighbors)
            visited[i] = True

            while queue:
                idx = queue.pop(0)
                if visited[idx]:
                    continue
                visited[idx] = True
                cluster_indices.append(idx)

                new_neighbors = tree.query_ball_point(xyz[idx], eps)
                if len(new_neighbors) >= min_points:
                    for n in new_neighbors:
                        if not visited[n]:
                            queue.append(n)

            if len(cluster_indices) >= min_points:
                clusters.append(points[cluster_indices])

        return clusters

    @staticmethod
    def oriented_bounding_box(
        cluster: np.ndarray,
    ) -> Dict:
        """Compute minimum oriented bounding box for a point cluster.

        Returns the tightest 3D box around the cluster, oriented to
        minimize volume. Used for object size estimation and as
        detection output format.
        """
        xyz = cluster[:, :3]

        # Project to XY plane for 2D oriented bbox (common for vehicles/pedestrians)
        xy = xyz[:, :2]

        # PCA to find principal axis
        centroid_xy = xy.mean(axis=0)
        centered = xy - centroid_xy
        cov = np.cov(centered.T)
        eigenvalues, eigenvectors = np.linalg.eigh(cov)

        # Rotation angle from first principal component
        angle = np.arctan2(eigenvectors[1, 1], eigenvectors[0, 1])

        # Rotate points to align with principal axis
        cos_a, sin_a = np.cos(-angle), np.sin(-angle)
        rotated = np.column_stack([
            centered[:, 0] * cos_a - centered[:, 1] * sin_a,
            centered[:, 0] * sin_a + centered[:, 1] * cos_a,
        ])

        # Axis-aligned bbox in rotated frame
        min_xy = rotated.min(axis=0)
        max_xy = rotated.max(axis=0)
        length = max_xy[0] - min_xy[0]
        width = max_xy[1] - min_xy[1]

        # Height from z-axis
        z_min = xyz[:, 2].min()
        z_max = xyz[:, 2].max()
        height = z_max - z_min

        centroid_3d = np.array([
            centroid_xy[0], centroid_xy[1], (z_min + z_max) / 2
        ])

        return {
            "center": centroid_3d,
            "dimensions": np.array([length, width, height]),
            "yaw": float(angle),
            "num_points": len(cluster),
            "z_range": (float(z_min), float(z_max)),
        }

AI tool limitations in 3D: This is where the gap between tools is most pronounced. Claude Code generates reasonable RANSAC and clustering implementations, correctly adding the ground plane orientation check (normal z-component > 0.7) that prevents walls from being classified as ground. Cursor sometimes produces working voxel downsampling but misses the horizontal normal constraint in RANSAC. Copilot and Windsurf both struggle significantly with 3D geometry — their oriented bounding box implementations frequently have rotation matrix errors or conflate 2D and 3D operations. Tabnine and Amazon Q are essentially not useful for point cloud processing beyond basic NumPy array operations. If point clouds are a major part of your work, expect to write most of the geometry code yourself regardless of which AI tool you use.

6. Edge Deployment & Model Optimization

Deploying CV models to edge devices — Jetson, phones, smart cameras, microcontrollers — requires a fundamentally different engineering approach than cloud deployment. Memory is limited, compute is constrained, and every millisecond of latency matters. The model that runs at 200 fps on your development GPU may run at 2 fps on the target device without optimization.

TensorRT Deployment Pipeline

import numpy as np
from pathlib import Path
from typing import Tuple, List, Optional
import logging

logger = logging.getLogger(__name__)


class TensorRTEngine:
    """Production TensorRT inference engine.

    Handles the deployment complexity that AI tools consistently miss:
    - Engine building with proper calibration for INT8
    - Memory management with pre-allocated buffers
    - CUDA stream management for async inference
    - Graceful fallback when TensorRT is unavailable
    """

    def __init__(
        self,
        onnx_path: str,
        precision: str = "fp16",  # fp32, fp16, int8
        max_batch_size: int = 1,
        workspace_mb: int = 1024,
        calibration_data: Optional[np.ndarray] = None,
    ):
        self.precision = precision
        self.max_batch = max_batch_size

        try:
            import tensorrt as trt
            import pycuda.driver as cuda
            import pycuda.autoinit  # noqa: F401
            self._trt = trt
            self._cuda = cuda
        except ImportError:
            raise RuntimeError(
                "TensorRT requires: pip install tensorrt pycuda. "
                "For Jetson: install via JetPack SDK."
            )

        self._logger = trt.Logger(trt.Logger.WARNING)
        self._engine = self._build_engine(
            onnx_path, precision, max_batch_size,
            workspace_mb, calibration_data
        )
        self._context = self._engine.create_execution_context()

        # Pre-allocate device buffers (avoids per-inference allocation)
        self._inputs, self._outputs, self._bindings, self._stream = (
            self._allocate_buffers()
        )

    def _build_engine(
        self, onnx_path, precision, max_batch, workspace_mb, cal_data
    ):
        """Build TensorRT engine from ONNX model."""
        trt = self._trt
        cache_path = Path(onnx_path).with_suffix(
            f".{precision}.b{max_batch}.trt"
        )

        # Use cached engine if available and newer than ONNX
        if cache_path.exists():
            onnx_mtime = Path(onnx_path).stat().st_mtime
            cache_mtime = cache_path.stat().st_mtime
            if cache_mtime > onnx_mtime:
                logger.info(f"Loading cached TensorRT engine: {cache_path}")
                runtime = trt.Runtime(self._logger)
                with open(cache_path, "rb") as f:
                    return runtime.deserialize_cuda_engine(f.read())

        logger.info(f"Building TensorRT engine ({precision})...")
        builder = trt.Builder(self._logger)
        network = builder.create_network(
            1 << int(trt.NetworkDefinitionCreationFlag.EXPLICIT_BATCH)
        )
        parser = trt.OnnxParser(network, self._logger)

        with open(onnx_path, "rb") as f:
            if not parser.parse(f.read()):
                for i in range(parser.num_errors):
                    logger.error(f"ONNX parse error: {parser.get_error(i)}")
                raise RuntimeError("Failed to parse ONNX model")

        config = builder.create_builder_config()
        config.set_memory_pool_limit(
            trt.MemoryPoolType.WORKSPACE, workspace_mb * (1 << 20)
        )

        if precision == "fp16" and builder.platform_has_fast_fp16:
            config.set_flag(trt.BuilderFlag.FP16)
        elif precision == "int8" and builder.platform_has_fast_int8:
            config.set_flag(trt.BuilderFlag.INT8)
            if cal_data is not None:
                config.int8_calibrator = self._create_calibrator(cal_data)
            else:
                logger.warning(
                    "INT8 without calibration data. "
                    "Accuracy may be significantly degraded."
                )

        # Set optimization profile for dynamic batch
        profile = builder.create_optimization_profile()
        input_shape = network.get_input(0).shape
        min_shape = (1, *input_shape[1:])
        opt_shape = (max(1, max_batch // 2), *input_shape[1:])
        max_shape = (max_batch, *input_shape[1:])

        profile.set_shape(
            network.get_input(0).name, min_shape, opt_shape, max_shape
        )
        config.add_optimization_profile(profile)

        engine = builder.build_serialized_network(network, config)
        if engine is None:
            raise RuntimeError("TensorRT engine build failed")

        # Cache the built engine
        with open(cache_path, "wb") as f:
            f.write(engine)
        logger.info(f"Engine cached: {cache_path}")

        runtime = trt.Runtime(self._logger)
        return runtime.deserialize_cuda_engine(engine)

    def _allocate_buffers(self):
        """Pre-allocate CUDA memory for all engine bindings."""
        cuda = self._cuda
        inputs = []
        outputs = []
        bindings = []
        stream = cuda.Stream()

        for i in range(self._engine.num_io_tensors):
            name = self._engine.get_tensor_name(i)
            shape = self._engine.get_tensor_shape(name)
            dtype = self._engine.get_tensor_dtype(name)

            # Replace -1 (dynamic) with max batch size
            shape = list(shape)
            if shape[0] == -1:
                shape[0] = self.max_batch
            shape = tuple(shape)

            size = int(np.prod(shape))
            np_dtype = np.float32 if dtype == self._trt.float32 else np.float16

            # Allocate host and device memory
            host_mem = cuda.pagelocked_empty(size, np_dtype)
            device_mem = cuda.mem_alloc(host_mem.nbytes)
            bindings.append(int(device_mem))

            buffer = {
                "name": name,
                "host": host_mem,
                "device": device_mem,
                "shape": shape,
                "dtype": np_dtype,
            }

            mode = self._engine.get_tensor_mode(name)
            if mode == self._trt.TensorIOMode.INPUT:
                inputs.append(buffer)
            else:
                outputs.append(buffer)

        return inputs, outputs, bindings, stream

    def infer(self, input_data: np.ndarray) -> List[np.ndarray]:
        """Run inference with pre-allocated buffers.

        Args:
            input_data: (B, C, H, W) float32 numpy array

        Returns:
            List of output arrays
        """
        cuda = self._cuda
        batch_size = input_data.shape[0]

        # Copy input to pre-allocated host buffer
        np.copyto(
            self._inputs[0]["host"][:input_data.size],
            input_data.ravel()
        )

        # Host -> Device (async)
        cuda.memcpy_htod_async(
            self._inputs[0]["device"],
            self._inputs[0]["host"],
            self._stream,
        )

        # Set actual batch size for dynamic shape
        input_shape = list(self._inputs[0]["shape"])
        input_shape[0] = batch_size
        self._context.set_input_shape(
            self._inputs[0]["name"], tuple(input_shape)
        )

        # Set tensor addresses
        for buf in self._inputs + self._outputs:
            self._context.set_tensor_address(buf["name"], int(buf["device"]))

        # Execute
        self._context.execute_async_v3(stream_handle=self._stream.handle)

        # Device -> Host (async) for all outputs
        results = []
        for out in self._outputs:
            cuda.memcpy_dtoh_async(out["host"], out["device"], self._stream)

        # Synchronize
        self._stream.synchronize()

        # Reshape outputs
        for out in self._outputs:
            shape = list(out["shape"])
            shape[0] = batch_size
            result = out["host"][:int(np.prod(shape))].reshape(shape).copy()
            results.append(result)

        return results

Edge deployment is where all tools struggle: The TensorRT API changes significantly between versions (8.x vs 10.x), and AI tools frequently mix APIs from different versions. Claude Code produces the most coherent TensorRT code, correctly using execute_async_v3 (TRT 10+) and set_tensor_address instead of the deprecated execute_async_v2 with implicit bindings. It also correctly implements engine caching (building TRT engines takes minutes; you do not rebuild on every startup). Cursor generates working ONNX Runtime code but its TensorRT code often uses deprecated APIs. Copilot’s TensorRT output is unreliable — it frequently generates code that compiles but crashes at runtime due to buffer size mismatches or incorrect binding order. For edge deployment to Jetson or other embedded platforms, expect to write the deployment layer yourself and use AI tools primarily for the model conversion and quantization scripts.

7. Data Augmentation & Training Pipelines

Training computer vision models requires augmentation pipelines that are fast enough to not bottleneck GPU training, diverse enough to prevent overfitting, and correct enough to not invalidate your labels. The last point is critical: augmentations that transform images must also transform the corresponding labels (bounding boxes, segmentation masks, keypoints), and getting this wrong produces training data that teaches the model incorrect associations.

Label-Aware Augmentation Pipeline

import cv2
import numpy as np
from typing import Tuple, List, Optional, Dict
from dataclasses import dataclass


@dataclass
class AugmentedSample:
    """Training sample with image and all associated labels."""
    image: np.ndarray           # (H, W, 3) BGR uint8
    boxes: np.ndarray           # (N, 4) xyxy format
    class_ids: np.ndarray       # (N,) class indices
    masks: Optional[np.ndarray] = None  # (N, H, W) instance masks


class CVAugmentor:
    """Production augmentation pipeline with correct label transforms.

    Key invariant: every geometric transform applied to the image
    MUST be applied to boxes and masks. Photometric transforms
    (brightness, contrast, color jitter) do NOT affect labels.
    AI tools frequently apply geometric transforms to images but
    forget to transform the labels.
    """

    def __init__(self, seed: int = 42):
        self.rng = np.random.RandomState(seed)

    def augment(
        self, sample: AugmentedSample, training: bool = True
    ) -> AugmentedSample:
        """Apply augmentation pipeline.

        Augmentation order matters:
        1. Geometric (affects image + labels)
        2. Photometric (affects image only)
        3. Normalization (affects image only, applied separately)
        """
        if not training:
            return sample

        img = sample.image.copy()
        boxes = sample.boxes.copy()
        class_ids = sample.class_ids.copy()
        masks = sample.masks.copy() if sample.masks is not None else None

        # Geometric augmentations (must transform labels too)
        if self.rng.random() < 0.5:
            img, boxes, masks = self._horizontal_flip(img, boxes, masks)

        if self.rng.random() < 0.3:
            img, boxes, masks = self._random_crop(img, boxes, masks, class_ids)

        if self.rng.random() < 0.3:
            img, boxes, masks = self._random_affine(img, boxes, masks)

        # Photometric augmentations (image only, labels unchanged)
        if self.rng.random() < 0.5:
            img = self._color_jitter(img)

        if self.rng.random() < 0.3:
            img = self._random_blur(img)

        # Filter out boxes that are too small after augmentation
        if len(boxes) > 0:
            widths = boxes[:, 2] - boxes[:, 0]
            heights = boxes[:, 3] - boxes[:, 1]
            valid = (widths > 2) & (heights > 2)
            boxes = boxes[valid]
            class_ids = class_ids[valid]
            if masks is not None:
                masks = masks[valid]

        return AugmentedSample(
            image=img, boxes=boxes, class_ids=class_ids, masks=masks
        )

    def _horizontal_flip(
        self, img, boxes, masks
    ) -> Tuple:
        """Flip image and labels horizontally."""
        h, w = img.shape[:2]
        img = cv2.flip(img, 1)

        if len(boxes) > 0:
            # Flip box x-coordinates: new_x = w - old_x
            flipped = boxes.copy()
            flipped[:, 0] = w - boxes[:, 2]  # new x1 = w - old x2
            flipped[:, 2] = w - boxes[:, 0]  # new x2 = w - old x1
            boxes = flipped

        if masks is not None:
            masks = masks[:, :, ::-1].copy()

        return img, boxes, masks

    def _random_crop(
        self, img, boxes, masks, class_ids,
        min_scale: float = 0.5,
    ) -> Tuple:
        """Random crop that preserves at least one object.

        The naive approach (random crop, then filter boxes) often
        produces samples with zero objects. This implementation
        ensures at least one box center is within the crop region.
        """
        h, w = img.shape[:2]

        if len(boxes) == 0:
            return img, boxes, masks

        # Choose a random box to keep
        anchor_idx = self.rng.randint(len(boxes))
        anchor_cx = (boxes[anchor_idx, 0] + boxes[anchor_idx, 2]) / 2
        anchor_cy = (boxes[anchor_idx, 1] + boxes[anchor_idx, 3]) / 2

        # Random crop size
        scale = self.rng.uniform(min_scale, 1.0)
        crop_w = int(w * scale)
        crop_h = int(h * scale)

        # Position crop to include anchor box center
        max_x = min(int(anchor_cx), w - crop_w)
        min_x = max(0, int(anchor_cx) - crop_w)
        max_y = min(int(anchor_cy), h - crop_h)
        min_y = max(0, int(anchor_cy) - crop_h)

        if max_x <= min_x or max_y <= min_y:
            return img, boxes, masks

        x1 = self.rng.randint(min_x, max_x + 1)
        y1 = self.rng.randint(min_y, max_y + 1)
        x2 = x1 + crop_w
        y2 = y1 + crop_h

        # Crop image
        img = img[y1:y2, x1:x2].copy()

        # Adjust box coordinates
        boxes = boxes.copy()
        boxes[:, [0, 2]] -= x1
        boxes[:, [1, 3]] -= y1

        # Clip to crop bounds
        boxes[:, [0, 2]] = np.clip(boxes[:, [0, 2]], 0, crop_w)
        boxes[:, [1, 3]] = np.clip(boxes[:, [1, 3]], 0, crop_h)

        # Keep boxes with center inside crop
        cx = (boxes[:, 0] + boxes[:, 2]) / 2
        cy = (boxes[:, 1] + boxes[:, 3]) / 2
        valid = (cx > 0) & (cx < crop_w) & (cy > 0) & (cy < crop_h)

        boxes = boxes[valid]
        class_ids_out = class_ids[valid]

        if masks is not None:
            masks = masks[:, y1:y2, x1:x2].copy()
            masks = masks[valid]

        return img, boxes, masks

    def _random_affine(
        self, img, boxes, masks,
        max_rotation: float = 10,
        max_scale: float = 0.15,
    ) -> Tuple:
        """Random rotation + scale with correct label transform."""
        h, w = img.shape[:2]
        center = (w / 2, h / 2)

        angle = self.rng.uniform(-max_rotation, max_rotation)
        scale = self.rng.uniform(1 - max_scale, 1 + max_scale)

        M = cv2.getRotationMatrix2D(center, angle, scale)
        img = cv2.warpAffine(
            img, M, (w, h),
            borderValue=(114, 114, 114)
        )

        if len(boxes) > 0:
            # Transform box corners (not just xyxy - rotation needs corners)
            corners = self._boxes_to_corners(boxes)  # (N, 4, 2)
            n = len(corners)
            flat = corners.reshape(-1, 2)  # (N*4, 2)

            # Apply affine transform to all corners
            ones = np.ones((len(flat), 1))
            flat_h = np.hstack([flat, ones])  # (N*4, 3)
            transformed = (M @ flat_h.T).T    # (N*4, 2)
            transformed = transformed.reshape(n, 4, 2)

            # Get axis-aligned bounding boxes from rotated corners
            boxes = np.column_stack([
                transformed[:, :, 0].min(axis=1),
                transformed[:, :, 1].min(axis=1),
                transformed[:, :, 0].max(axis=1),
                transformed[:, :, 1].max(axis=1),
            ])

            # Clip to image bounds
            boxes[:, [0, 2]] = np.clip(boxes[:, [0, 2]], 0, w)
            boxes[:, [1, 3]] = np.clip(boxes[:, [1, 3]], 0, h)

        if masks is not None:
            transformed_masks = np.zeros_like(masks)
            for i in range(len(masks)):
                transformed_masks[i] = cv2.warpAffine(
                    masks[i], M, (w, h),
                    flags=cv2.INTER_NEAREST,  # NEAREST for masks!
                    borderValue=0
                )
            masks = transformed_masks

        return img, boxes, masks

    @staticmethod
    def _boxes_to_corners(boxes: np.ndarray) -> np.ndarray:
        """Convert xyxy boxes to 4 corner points for affine transform."""
        # (N, 4) -> (N, 4, 2): TL, TR, BR, BL
        n = len(boxes)
        corners = np.zeros((n, 4, 2))
        corners[:, 0] = boxes[:, :2]                         # TL
        corners[:, 1] = boxes[:, [2, 1]]                     # TR
        corners[:, 2] = boxes[:, 2:]                          # BR
        corners[:, 3] = boxes[:, [0, 3]]                     # BL
        return corners

    def _color_jitter(
        self, img: np.ndarray,
        brightness: float = 0.3,
        contrast: float = 0.3,
        saturation: float = 0.3,
        hue: float = 0.1,
    ) -> np.ndarray:
        """Photometric augmentation in HSV space."""
        # Convert to HSV for perceptually meaningful adjustments
        hsv = cv2.cvtColor(img, cv2.COLOR_BGR2HSV).astype(np.float32)

        # Hue shift (circular)
        h_shift = self.rng.uniform(-hue, hue) * 180
        hsv[:, :, 0] = (hsv[:, :, 0] + h_shift) % 180

        # Saturation scale
        s_scale = self.rng.uniform(1 - saturation, 1 + saturation)
        hsv[:, :, 1] = np.clip(hsv[:, :, 1] * s_scale, 0, 255)

        # Value (brightness) scale
        v_scale = self.rng.uniform(1 - brightness, 1 + brightness)
        hsv[:, :, 2] = np.clip(hsv[:, :, 2] * v_scale, 0, 255)

        img = cv2.cvtColor(hsv.astype(np.uint8), cv2.COLOR_HSV2BGR)

        # Contrast adjustment in BGR space
        c_scale = self.rng.uniform(1 - contrast, 1 + contrast)
        mean = img.mean()
        img = np.clip((img.astype(np.float32) - mean) * c_scale + mean, 0, 255)

        return img.astype(np.uint8)

    def _random_blur(self, img: np.ndarray) -> np.ndarray:
        """Random Gaussian blur to simulate defocus/motion."""
        ksize = self.rng.choice([3, 5, 7])
        sigma = self.rng.uniform(0.5, 2.0)
        return cv2.GaussianBlur(img, (ksize, ksize), sigma)

The label transform problem: This is the most common and most dangerous mistake AI tools make in CV training code. When asked to “add random rotation augmentation,” every tool generates cv2.warpAffine for the image but most forget to transform the bounding boxes. The correct approach requires converting xyxy boxes to corner points, applying the same affine matrix to all 8 corner coordinates (4 corners x 2D), then computing the new axis-aligned bounding box from the rotated corners. Claude Code gets this right about 60% of the time when explicitly prompted with “include box transforms.” Cursor gets the image transform right but generates incorrect box transforms in about 40% of cases (typically forgetting to convert to corners first). The mask transform is also critical — INTER_NEAREST interpolation for binary masks, not INTER_LINEAR — and AI tools get this wrong at the same rate they get segmentation mask resizing wrong.

What AI Tools Get Wrong in Computer Vision

After extensive testing across all major AI coding tools, these are the CV-specific errors that appear consistently. Memorize this list — it will save you hours of debugging:

Cost Model: What Does This Actually Cost?

Computer vision engineering involves long coding sessions (pipeline development, debugging shape issues, optimizing inference) mixed with shorter sessions (configuration changes, deployment tweaks, evaluation script updates). Here are realistic cost scenarios:

ROI reality check: CV engineers typically earn $160,000–$250,000+ (autonomous vehicles, medical imaging, and defense pay the highest). At $200K/year, a 5% productivity gain justifies $833/month in tooling. Even at a conservative 2% gain from AI coding tools (primarily from faster boilerplate generation in data loading, preprocessing, and evaluation scripts), a $20–60/month investment pays for itself 8–27x over. The areas where AI tools save the most time in CV are not the algorithmic core (where you need to understand every line) but the surrounding infrastructure: data loaders, visualization, evaluation metrics computation, and configuration management.

Practical Recommendations

Related Guides