The Context Window Problem

Understanding Context Windows

Every LLM has a context window - the maximum amount of text it can process at once. Think of it as the model's working memory.

While 200K tokens sounds like a lot, consider a real coding project:

Typical E-Commerce Codebase
├── Source files:        ~50K tokens
├── Dependencies docs:   ~30K tokens
├── Conversation history: ~40K tokens
├── Tool outputs:        ~30K tokens
├── Error messages:      ~20K tokens
└── Remaining space:     ~30K tokens

The Core Challenge

Anthropic's engineering team identified the fundamental problem:

> "The core challenge of long-running agents is that they must work in discrete sessions, and each new session begins with no memory of what came before."

Context Anxiety

A fascinating phenomenon observed in long-running agents is context anxiety:

// What the agent "feels" as context fills up
const contextUsage = {
  '0-50%': 'Confident, exploratory, thorough',
  '50-75%': 'More focused, less exploration',
  '75-90%': 'Rushing, cutting corners',
  '90-100%': 'Panic mode - premature completion'
};

As agents perceive their context window filling up, they:

• Skip important steps
• Declare work "complete" prematurely
• Avoid exploring edge cases
• Make assumptions instead of verifying

> "We observed that agents would sometimes prematurely conclude work as they perceived approaching token limits, even when significant features remained incomplete."

Compaction: A Partial Solution

Compaction Strategies

interface CompactionStrategy {
  // Summarize old messages
  summarizeHistory(messages: Message[]): string;

  // Keep only relevant file content
  pruneFileContext(files: FileContent[]): FileContent[];

  // Compress tool outputs
  summarizeToolOutputs(outputs: ToolOutput[]): string;
}

// Example implementation
const compactor: CompactionStrategy = {
  summarizeHistory(messages) {
    // Keep last 10 messages verbatim
    const recent = messages.slice(-10);
    // Summarize older messages
    const older = messages.slice(0, -10);
    const summary = generateSummary(older);
    return [summary, ...recent];
  },

  pruneFileContext(files) {
    return files.map(file => ({
      path: file.path,
      // Keep only function signatures, not implementations
      content: extractSignatures(file.content)
    }));
  },

  summarizeToolOutputs(outputs) {
    return outputs.map(o =>
      `${o.tool}: ${o.success ? 'Success' : 'Failed'} - ${o.summary}`
    ).join('\n');
  }
};

Why Compaction Isn't Enough

Anthropic's key finding:

> "Compaction isn't sufficient for preventing context exhaustion across multiple windows."

Problems with compaction alone:

• Information loss - Summaries miss crucial details
• Compounding errors - Each summary loses fidelity
• No cross-session memory - New sessions start fresh
• Context anxiety persists - Agent still "feels" the pressure

The Solution: Structured State Management

Instead of trying to fit everything in context, use external state:

Key Insight from Anthropic

> "The key insight here was finding a way for agents to quickly understand the state of work when starting with a fresh context window, which is accomplished with the claude-progress.txt file alongside the git history."

Context Reset vs Context Compaction

Two different approaches:

Compaction (In-Place Summarization)

Old Context → Summarize → Continue in same session
[100K tokens] → [30K tokens] → [30K + new work]

Context Reset (Clean Slate)

Session 1 ends → Save state to files → Session 2 starts fresh
[Full context] → [External state] → [Load state into new context]

• No accumulated cruft - Each session starts clean
• No context anxiety - Full window available
• Explicit handoff - Forces clear documentation
• Better for evaluation - Can grade work between sessions

Designing for Multi-Session Work

// Session startup sequence
async function initializeSession(projectDir: string) {
  // 1. Check current directory
  const cwd = await bash('pwd');

  // 2. Read progress from previous sessions
  const progress = await readFile('claude-progress.txt');

  // 3. Review git history
  const gitLog = await bash('git log --oneline -20');

  // 4. Load feature list
  const features = await readJSON('feature-list.json');

  // 5. Find next task
  const nextFeature = features.find(f => !f.passes);

  return {
    context: { progress, gitLog, features },
    currentTask: nextFeature
  };
}

E-Commerce Example: Session Handoff

## claude-progress.txt

### Session 1 - 2025-01-15 14:30
- Initialized project with Vite + React + TypeScript
- Set up TailwindCSS
- Created basic layout components
- Started authentication system

### Current State:
- Auth UI complete: Login, Register, Forgot Password
- Backend routes stubbed but not implemented
- JWT logic not yet added

### Next Steps:
1. Implement JWT token generation in /api/auth/login
2. Add token validation middleware
3. Connect frontend to backend auth

### Known Issues:
- None currently

// Agent reads progress file and immediately knows:
// 1. What's been done (auth UI, layout)
// 2. What's in progress (JWT implementation)
// 3. Exact next steps (token generation)
// 4. Current issues (none)

Key Takeaways

• Context windows are limited - Even 200K tokens isn't enough for complex projects
• Context anxiety is real - Agents rush as context fills up
• Compaction helps but isn't enough - Information loss compounds
• External state is essential - Files outlive context windows
• Structured handoffs enable multi-session work - Progress files + git history

Practice Exercise

Design a state management system for a multi-session agent:

• What files would you create?
• What information goes in each file?
• How does the agent know what to do on startup?
• How do you prevent the agent from repeating work?

Next Lesson

Now that we understand the context problem, let's explore the Two-Agent Architecture Pattern - Anthropic's first solution involving an initializer and a coder agent.