Hands-On with Hugging Face Transformers

The Transformers library provides high-level APIs that abstract away model complexity while giving you full control when you need it.

The pipeline() API — Instant Inference

The fastest way to use any model — one line of code for text generation, classification, summarization, translation, and more:

from transformers import pipeline

# Text generation
generator = pipeline("text-generation", model="mistralai/Mistral-Small-4", device_map="auto")
result = generator("Explain Docker networking in simple terms", max_new_tokens=256)
print(result[0]["generated_text"])

# Sentiment analysis
classifier = pipeline("sentiment-analysis")
print(classifier("I love open-source AI!"))  # [{'label': 'POSITIVE', 'score': 0.999}]

# Summarization
summarizer = pipeline("summarization", model="facebook/bart-large-cnn")
print(summarizer(long_article, max_length=130))

AutoModel Variants

The Trainer Class

HF's Trainer handles the training loop, evaluation, logging, checkpointing, mixed precision, and distributed training:

from transformers import Trainer, TrainingArguments
from datasets import load_dataset

dataset = load_dataset("imdb", split="train[:1000]")

args = TrainingArguments(
    output_dir="./results",
    num_train_epochs=3,
    per_device_train_batch_size=8,
    learning_rate=2e-5,
    fp16=True,
    logging_steps=10,
    evaluation_strategy="epoch",
    save_strategy="epoch",
)

trainer = Trainer(
    model=model,
    args=args,
    train_dataset=dataset,
    eval_dataset=eval_dataset,
    tokenizer=tokenizer,
)
trainer.train()

The Datasets Library

HF datasets provides streaming, memory-mapped, and cached access to 500K+ datasets:

from datasets import load_dataset

# Load from Hub
ds = load_dataset("tatsu-lab/alpaca", split="train")

# Stream large datasets without downloading fully
ds = load_dataset("allenai/c4", split="train", streaming=True)
for example in ds:
    process(example)

# Load from local files
ds = load_dataset("json", data_files="my_data.jsonl")