Rahul Kashyap
← Back to blog
Build Log

PIT: Why LLM Prompts Need Version Control

Production prompts are code. PIT brings Git-like semantic versioning, bisect, and replay to prompt engineering workflows.

Published

2026-02-28

Tags

LLMs · tooling · open source · prompt engineering

The Problem with Prompt Management

If you ship products powered by LLMs, you have a prompt management problem. Prompts are modified constantly. Small wording changes cause large behavioral shifts. There is no easy way to identify which change broke a previously working behavior.

Most teams manage prompts in:

  • Hardcoded strings in application code
  • Shared documents or spreadsheets
  • Environment variables
  • Custom internal tools

None of these give you what Git gives code: history, diff, bisect, rollback, and semantic understanding of change impact.

What PIT Does

PIT is a command-line tool that treats prompts as first-class versioned artifacts.

Core operations:
  • pit init creates a prompt repository
  • pit commit saves a prompt version with metadata
  • pit diff shows semantic differences between versions
  • pit bisect finds the exact version that introduced a regression
  • pit replay re-runs a prompt across all versions to visualize behavioral drift

The key design decision is that PIT understands prompt semantics, not just text diffs. A change from "be concise" to "be brief" is flagged as low-impact. A change from "never mention competitors" to removing that instruction is flagged as high-risk.

Design Principles

Three principles guided the architecture:

1. Files are truth. Every prompt version is a plain text file. No database, no server, no vendor lock-in.

2. History is cheap. Storing every version of every prompt adds negligible disk usage. There is no reason to lose history.

3. Bisect is the killer feature. When a prompt regresses, being able to binary-search through history to find the breaking change saves hours of manual debugging.

Current Status

PIT is open source and available on GitHub. It is intentionally minimal. The roadmap includes team collaboration features and integration with CI/CD pipelines for automated prompt regression testing.

← All writing