Your First Compilation

Learn how to use the compiler to automatically optimize your prompts.

What is Compilation?

Compilation is the process of automatically finding the optimal prompt configuration (instructions + few-shot examples) based on evaluation data. Instead of manually crafting prompts, you provide examples of good input/output pairs and let the compiler find what works best.

1. Prepare Training Data

Create a dataset of example input/output pairs:

const trainset = [
  { 
    input: { text: "Bill Gates founded Microsoft." }, 
    output: { names: ["Bill Gates"] } 
  },
  { 
    input: { text: "Elon Musk runs Tesla and SpaceX." }, 
    output: { names: ["Elon Musk"] } 
  },
  { 
    input: { text: "Jeff Bezos started Amazon." }, 
    output: { names: ["Jeff Bezos"] } 
  },
  { 
    input: { text: "Tim Cook leads Apple after Steve Jobs." }, 
    output: { names: ["Tim Cook", "Steve Jobs"] } 
  },
  { 
    input: { text: "Satya Nadella is the CEO of Microsoft." }, 
    output: { names: ["Satya Nadella"] } 
  },
];

Dataset Size

Start with 5-10 examples. The compiler will use some for few-shot learning and some for validation. More examples generally lead to better results.

2. Create an Evaluator

Evaluators score how well the model's output matches the expected output:

import { exactMatch, arrayOverlap } from "@mzhub/promptc";

// For exact matching (returns 1.0 if equal, 0.0 otherwise)
const evaluator = exactMatch();

// For array comparisons (Jaccard similarity)
const arrayEvaluator = arrayOverlap();

3. Run the Compiler

The BootstrapFewShot compiler finds the best few-shot examples:

import { BootstrapFewShot } from "@mzhub/promptc";

const compiler = new BootstrapFewShot(exactMatch());

const compiled = await compiler.compile(program, trainset, {
  candidates: 10,      // Try 10 different few-shot combinations
  concurrency: 5,      // Run 5 evaluations in parallel
  examplesPerCandidate: 3,
  onProgress: ({ candidatesEvaluated, totalCandidates, currentBestScore }) => {
    console.log(`Progress: ${candidatesEvaluated}/${totalCandidates}`);
    console.log(`Best score: ${(currentBestScore * 100).toFixed(1)}%`);
  }
});

4. View Results

The compiler returns metadata about the best configuration found:

console.log("Best score:", compiled.meta.score);
console.log("Token usage:", compiled.meta.tokenUsage.totalTokens);
console.log("API calls:", compiled.meta.tokenUsage.calls);
console.log("Strategy:", compiled.meta.strategy);

5. Save the Compiled Config

Save the optimized configuration for production use:

import { writeFileSync } from "fs";

// Save as JSON
writeFileSync("prompt.json", JSON.stringify(compiled, null, 2));

// The JSON contains:
// - meta.score: Best validation score
// - meta.tokenUsage: Total tokens used during compilation
// - config.instructions: The optimized instructions
// - config.fewShotExamples: The selected few-shot examples

6. Use in Production

Create a CompiledProgram for production inference:

import { createCompiledProgram, loadCompiledProgram } from "@mzhub/promptc";

// Option 1: Create from compilation result
const production = createCompiledProgram(program, compiled);

// Option 2: Load from saved JSON
const savedJson = readFileSync("prompt.json", "utf-8");
const production = loadCompiledProgram(savedJson, program);

// Use in production
const result = await production.run({
  text: "Jensen Huang founded NVIDIA."
});

Reproducibility

Saving compiled prompts as JSON ensures reproducible results. You can version control these files and deploy specific versions to production.

Complete Example

compile.js

import { 
  defineSchema, 
  ChainOfThought, 
  BootstrapFewShot,
  createProvider, 
  exactMatch,
  createCompiledProgram,
  z 
} from "@mzhub/promptc";
import { writeFileSync } from "fs";

// Schema
const NameExtractor = defineSchema({
  description: "Extract proper names of people from text",
  inputs: { text: z.string() },
  outputs: { names: z.array(z.string()) }
});

// Provider & Program
const provider = createProvider("openai", {
  apiKey: process.env.OPENAI_API_KEY
});
const program = new ChainOfThought(NameExtractor, provider);

// Training data
const trainset = [
  { input: { text: "Bill Gates founded Microsoft." }, output: { names: ["Bill Gates"] } },
  { input: { text: "Elon Musk runs Tesla." }, output: { names: ["Elon Musk"] } },
  { input: { text: "Jeff Bezos started Amazon." }, output: { names: ["Jeff Bezos"] } },
  { input: { text: "Tim Cook leads Apple." }, output: { names: ["Tim Cook"] } },
  { input: { text: "Satya Nadella is CEO." }, output: { names: ["Satya Nadella"] } },
];

// Compile
const compiler = new BootstrapFewShot(exactMatch());
const result = await compiler.compile(program, trainset, {
  candidates: 10,
  concurrency: 5
});

console.log("Score:", result.meta.score);
writeFileSync("prompt.json", JSON.stringify(result, null, 2));

// Use compiled program
const compiled = createCompiledProgram(program, result);
const output = await compiled.run({ text: "Jensen Huang founded NVIDIA." });
console.log("Names:", output.result.names);

Next: Learn about Schemas →