Native Multimodal Ingestion — Mastering the Gemini API

Omni-Modal Ingestion: Video, Audio, PDFs & Images

Unlike early generation AI systems that stitched together external OCR libraries and Whisper transcription pipelines, Gemini was architected natively as an omni-modal network. It directly processes interleaved streams of high-resolution video, multi-track audio, complex PDFs, and high-density images alongside text within the exact same attention layers.

Token Consumption Across Modalities

Understanding the exact tokenization formulas across modalities is vital for sizing context budgets on Vertex AI:

• Video Frames: Video is sampled at 1 frame per second (1 FPS). Each video frame consumes a fixed 258 input tokens. A 60-second video snippet consumes approximately \(60 \times 258 = 15,480\) tokens.
• Audio Streams: Raw audio is tokenized at 32 tokens per second (approximately 1,920 tokens per minute), preserving vocal inflections, background ambiance, and acoustic anomalies.
• PDF Documents: Every page in a rendered PDF is visually ingested as an image tile, consuming 258 tokens per page while simultaneously extracting native digital text streams.
• High-Resolution Images: Scaled to 768px minimum bounding boxes and mapped to 258 visual tokens per tile.

Handling Enterprise Media with the Unified SDK

For payloads exceeding 20MB, never pass raw Base64 strings over HTTP. Instead, leverage the Google Cloud Storage integration (gs://) or the unified Files API for zero-copy streaming:

import { GoogleGenAI } from "@google/genai";

const ai = new GoogleGenAI({ vertexAI: true, project: "my-gcp-project", location: "us-central1" });

// Ingest enterprise GCS video without downloading to local disk
const response = await ai.models.generateContent({
  model: "gemini-3.8-flash",
  contents: [
    {
      fileData: {
        fileUri: "gs://enterprise-video-archive/board-meeting-2026.mp4",
        mimeType: "video/mp4"
      }
    },
    {
      text: "Identify all timestamp intervals where compliance risks or budget discrepancies were discussed."
    }
  ]
});

console.log(response.text);