What makes an AI agent project worth building
An AI agent is a language model in a loop: it decides which tool to call, reads the result, and decides what to do next. That loop takes about 40 lines of Python. Recruiters and interviewers have seen it many times. What they have not seen is proof that your agent works.
So every project on this page has three things on top of the loop:
- Real tools on real (or realistic) data. A CSV export of your spending, a public GitHub repo, a SQLite database, a website you run locally. Not "the model pretends to look things up".
- One hard part. Every idea below names the part where agents actually fail: dates, refusing to answer, approval before money moves, not looping forever, not editing the tests to make them pass.
- A test you can run. A fixed set of questions or tasks with known right answers, and a score. "It answered right 9 times out of 11, and here are the 2 it got wrong" is the line that makes a project stand out.
Anthropic's guide to building agents says the same thing from the other side: "we recommend finding the simplest solution possible, and only increasing complexity when needed." It also separates workflows ("LLMs and tools are orchestrated through predefined code paths") from agents (LLMs that "dynamically direct their own processes and tool usage"). Several ideas below work fine as workflows. Saying so in your README is a point in your favour, not against you.
13 AI agent project ideas at a glance
The last column is how many public GitHub repositories match a search for the idea in their name or description, counted on 21 November 2026. Treat it as a rough sign of how crowded the idea is, not as a quality ranking. The exact queries are listed in Honest limits.
| # | Project | Level | The hard part | How you prove it works | GitHub repos |
|---|---|---|---|---|---|
| 1 | Personal expense agent | Beginner | "Last month", category synonyms | 11 fixed questions, scored (starter kit below) | 442 |
| 2 | GitHub issue triage agent | Beginner | Each repo has its own labels | Label accuracy on closed issues | 295 |
| 3 | Resume vs job description gap agent | Beginner | Synonyms, never inventing skills | Precision and recall on hand-labelled JDs | 1,132 |
| 4 | Study planner with a calendar | Beginner | Hard constraints, no double booking | Constraint checker: 0 violations | 425 |
| 5 | Docs Q&A that cites and says "I don't know" | Intermediate | Refusing when the answer is not there | Right answers + right refusals | 2,020 |
| 6 | Text-to-SQL analyst with a read-only guard | Intermediate | Wrong joins, dangerous queries | Execution accuracy + attack prompts | 1,989 |
| 7 | Support agent with refund approval | Intermediate | Pause, resume, never refund twice | Refund runs exactly once per approval | 11,680 |
| 8 | Web research agent on a budget | Intermediate | Stopping, quoting only what pages say | Every quote found on its cited page | 20,942 |
| 9 | Pull request review agent | Intermediate | Too many useless comments | Planted bugs caught vs false alarms | 7,177 |
| 10 | Your own MCP server for a public API | Advanced | Tool descriptions a model understands | Same eval through two clients | 178,524 |
| 11 | Test-fixing coding agent | Advanced | Sandboxing, not editing the tests | Planted bugs fixed, tests untouched | 151 |
| 12 | Browser agent on a local test site | Advanced | Flaky pages, unsafe clicks | Success rate over 20 runs | 3,546 |
| 13 | Evaluation and tracing harness | Advanced | Deciding what "correct" means | Catches a regression you plant | 2,867 |
Beginner projects: one agent, two or three tools
Start here if you have written Python but not an agent. Each one fits in a weekend. Use a free local model through Ollama so it costs nothing to run, or any hosted model with tool calling.
1. Personal expense agent
What it does: answers questions like "how much did I spend on food last month?" from a CSV export of your UPI or bank transactions.
Tools: total_spent(category, month), top_merchants(month, n), budget_left(category, month). The model never adds numbers itself. It only picks the tool and the arguments.
The hard part: turning "last month" into 2026-10, and "cabs and autos" into the travel category. Models also tend to guess a number instead of saying "I can't see that".
Prove it: this is the project our starter kit builds, with an 11-question test and real results from a free 3B model.
Stretch: read a real bank statement export, and add a set_budget tool that only runs after the user types "yes".
2. GitHub issue triage agent
What it does: reads a new issue on a public repo and suggests a label (bug, docs, feature, question) and a one-line summary.
Tools: list_issues, read_issue and list_labels through the GitHub REST API. Keep it read-only at first.
The hard part: every repo uses its own labels, and long issues bury the actual problem under logs.
Prove it: closed issues already carry the labels maintainers chose. That is free ground truth. Hide the labels, run your agent on 50 closed issues of one repo and report how often it matches.
Stretch: post the suggested label as a comment, but only after a person approves it.
3. Resume vs job description gap agent
What it does: compares your resume with a job description and lists the skills you have, the skills you lack, and one small project that would close each gap.
Tools: extract_skills(text), compare(resume_skills, jd_skills), suggest_project(skill).
The hard part: synonyms ("JS" and "JavaScript", "REST" and "REST APIs"), and never putting a skill in the "you have" list that is not in the resume.
Prove it: hand-label the skills in 20 real job descriptions. Report precision (how many extracted skills are right) and recall (how many real skills it found). Add a hard check: zero invented skills on the "you have" side.
4. Study planner with a calendar
What it does: turns a syllabus and an exam date into a day-by-day plan that fits your free time.
Tools: list_free_slots from an .ics calendar file, add_event in dry-run mode, topics_for(subject).
The hard part: hard rules. No more than 3 hours a day, revision before the exam, no clash with classes. Models break rules like these quietly.
Prove it: write a plain Python checker that reads the plan and counts rule violations. Run 20 different syllabi. The target is zero. When it is not zero, feed the checker's message back to the agent and measure how often one retry fixes it.
Intermediate projects: retrieval, memory, approval, safety
5. Docs Q&A agent that cites and says "I don't know"
What it does: answers questions from a fixed set of documents, such as your college handbook or an open-source project's docs, with a citation for every claim.
Tools: search_docs(query) returning passages with their source. Our RAG guide builds this search step with code.
The hard part: refusing. When the answer is not in the documents, most chatbots answer anyway.
Prove it: 30 questions, 10 of which have no answer in the documents. Report right answers, right refusals, and whether each cited passage contains the fact it is cited for.
6. Text-to-SQL analyst with a read-only guard
What it does: answers business questions ("which city had the most orders in October?") by writing and running SQL on a SQLite database.
Tools: list_tables, describe_table, run_query on a read-only connection, SELECT only, with a row limit.
The hard part: wrong joins that return a plausible but wrong number, and prompts like "delete all cancelled orders".
Prove it: write 30 questions with a correct SQL query for each. Compare the rows your agent's query returns with the rows of the correct query. This is execution accuracy, the metric the BIRD text-to-SQL benchmark ranks models on. Add 5 attack prompts that must be refused. Brush up first with the free SQL fundamentals course.
7. Support agent with human approval for refunds
What it does: handles "where is my order" and "I want a refund" for a small invented shop. Looking things up is automatic. Money moving needs a person to approve.
Tools: get_order, days_since_delivery, issue_refund (gated).
The hard part: pausing for approval and resuming later without issuing the refund twice. Our LangGraph tutorial shows the trap: on resume, the paused step runs again from the top.
Prove it: a test that the refund function runs exactly once per approval and never without one, including when the process restarts between the pause and the approval.
8. Web research agent on a budget
What it does: answers a question by searching, reading a few pages, and writing a short answer with quotes and links.
Tools: search(query), fetch_page(url), with a hard cap such as 6 fetches per question.
The hard part: knowing when to stop, and quoting only what the pages actually say.
Prove it: for 15 questions, check automatically that every quoted sentence appears on the page it links to. This check needs no human and catches made-up quotes.
9. Pull request review agent
What it does: reads a pull request diff, runs the tests and a linter, and leaves review comments.
Tools: get_diff, read_file, run_tests, run_linter.
The hard part: noise. An agent that leaves 20 low-value comments gets switched off.
Prove it: plant 10 known bugs in a small repo across 10 pull requests. Report bugs caught and false alarms per pull request. Fewer, better comments win.
Advanced projects: protocols, sandboxes, browsers, evaluation
10. Your own MCP server for a public API
What it does: wraps a public data source, such as a government open-data API or your college timetable, as an MCP server so any MCP-capable assistant can use it. MCP is "an open-source standard for connecting AI applications to external systems".
The hard part: writing tool names, descriptions and argument schemas that a model picks correctly. Our tool calling guide measures how much descriptions matter.
Prove it: run the same 15-question test through two different MCP clients and report the scores side by side. With 178,524 repos matching "mcp server", a generic one will not stand out. A server for one specific, useful data source with a published test will. The free MCP and tool use course covers the protocol.
11. Test-fixing coding agent
What it does: given a repo with a failing test, it reads the code, edits it, reruns the tests and repeats until they pass or the budget runs out.
Tools: read_file, write_file, run_tests, all inside a container with no network.
The hard part: sandboxing, and an agent that "fixes" a test by editing the test.
Prove it: plant 20 bugs in a small project. Report how many are fixed with the test files unchanged (check their hashes). This is the idea behind SWE-bench, whose Verified set is "a human-filtered subset of 500 instances". It is also the least crowded idea here: 151 matching repos. The free agent harnesses course covers multi-agent coding setups.
12. Browser agent on a local test site
What it does: completes a task on a website, such as finding a product and filling a checkout form, by driving a real browser with Playwright.
The hard part: pages change and load slowly, and an agent with a browser can click things it should not. Run it against a site you host yourself, never your real accounts.
Prove it: run the same task 20 times and report the success rate, plus a hard rule checked in code: it never presses "Pay". The free Playwright course builds the test site skills.
13. Evaluation and tracing harness
What it does: wraps any agent above and records every model call, tool call, cost and time, then prints a report each time you change a prompt or a model.
The hard part: deciding what "correct" means for each task, and keeping the test set fixed while everything else changes.
Prove it: plant a regression and show that the report catches it. The starter kit below does this in a small way: one scripted answer is wrong on purpose, and the scorer flags it. The free production AI agents course goes further.
Ideas that are overdone, and how to rescue them
Some ideas show up in almost every "AI projects" list. Here is how many GitHub repositories matched each on 21 November 2026, next to two from our list:
| Search | Matching repos |
|---|---|
| "pdf chatbot" | 6,572 |
| "chat with pdf" | 4,827 |
| "travel planner" + agent | 3,446 |
| "youtube summarizer" | 2,064 |
| "issue triage" + agent | 295 |
| "failing tests" + agent | 151 |
You can still build a PDF chatbot. Just do the part the thousands of others skip: a refusal test (project 5), a citation check, and a score in the README. A common idea with a published test beats a rare idea with none.
Starter kit: a tested agent and its scorer
Here is project 1 built end to end in plain Python, with no framework and no API key. Copy the structure for any idea above: swap the tools and the test questions, keep the loop and the scorer.
AI agent project folder structure
This is the real folder we ran. Tools, loop and model are separate files so you can test each one alone, and the test questions live outside the code so nobody "fixes" a test by editing the agent.
expense-agent/
├── agent/
│ ├── loop.py
│ ├── models.py
│ └── tools.py
├── data/
│ └── transactions.csv
└── evals/
├── cases.jsonl
└── run_evals.py
The 30 transactions in data/transactions.csv are invented: rent, canteen, metro, recharge and so on across October and November 2026.
The tools
Each tool is a normal function plus a JSON schema the model sees. The schema restricts the category to the ones in the data and the month to YYYY-MM.
agent/tools.py
# tools.py: the agent's tools (plain functions) and the JSON schemas the model sees.
import csv, pathlib
from collections import Counter
ROWS = list(csv.DictReader(open(pathlib.Path(__file__).parent.parent / "data" / "transactions.csv")))
CATEGORIES = sorted({r["category"] for r in ROWS})
BUDGETS = {"food": 2000, "travel": 1500, "shopping": 2500, "education": 2000, "recharge": 400, "rent": 6500}
def total_spent(category: str, month: str) -> dict:
hits = [int(r["amount"]) for r in ROWS if r["category"] == category and r["date"].startswith(month)]
return {"category": category, "month": month, "total_inr": sum(hits), "transactions": len(hits)}
def top_merchants(month: str, n: int = 3) -> dict:
spend = Counter()
for r in ROWS:
if r["date"].startswith(month):
spend[r["merchant"]] += int(r["amount"])
return {"month": month, "top": [{"merchant": m, "total_inr": t} for m, t in spend.most_common(n)]}
def budget_left(category: str, month: str) -> dict:
spent = total_spent(category, month)["total_inr"]
return {"category": category, "month": month, "budget_inr": BUDGETS[category], "left_inr": BUDGETS[category] - spent}
MONTH = {"type": "string", "pattern": r"^\d{4}-\d{2}$", "description": "Month as YYYY-MM, e.g. 2026-10"}
CATEGORY = {"type": "string", "enum": CATEGORIES, "description": "Spending category"}
TOOLS = {
"total_spent": (total_spent, "Total rupees spent in one category in one month.",
{"type": "object", "properties": {"category": CATEGORY, "month": MONTH}, "required": ["category", "month"]}),
"top_merchants": (top_merchants, "The merchants where the most money went in a month.",
{"type": "object", "properties": {"month": MONTH, "n": {"type": "integer", "description": "How many, default 3"}}, "required": ["month"]}),
"budget_left": (budget_left, "Rupees left in this month's budget for one category (negative = over budget).",
{"type": "object", "properties": {"category": CATEGORY, "month": MONTH}, "required": ["category", "month"]}),
}
The loop and the argument check
The loop is the whole agent. Note two things: arguments are checked against the schema before any tool runs, and a bad call is returned to the model as an error it can fix, instead of crashing. max_steps stops a model that never finishes.
agent/loop.py
# loop.py: the agent loop. Model asks for a tool -> we validate and run it -> model sees the result.
import json, re
from agent.tools import TOOLS
SYSTEM = ("You answer questions about the user's spending using the tools. Today is 2026-11-21, "
"so 'this month' is 2026-11 and 'last month' is 2026-10. Amounts are in rupees. "
"Always call a tool for numbers; never guess.")
def check_args(schema: dict, args: dict) -> str | None:
"""A small JSON-schema check: required keys, enums, patterns, types. Returns an error or None."""
for key in schema["required"]:
if key not in args:
return f"missing argument '{key}'"
for key, value in args.items():
spec = schema["properties"].get(key)
if spec is None:
return f"unknown argument '{key}'"
if "enum" in spec and value not in spec["enum"]:
return f"'{value}' is not one of {spec['enum']}"
if "pattern" in spec and not re.fullmatch(spec["pattern"], str(value)):
return f"'{value}' does not look like {spec['description']}"
if spec["type"] == "integer" and not isinstance(value, int):
return f"'{key}' must be a whole number"
return None
def run_agent(model, question: str, max_steps: int = 5) -> dict:
messages = [{"role": "system", "content": SYSTEM}, {"role": "user", "content": question}]
calls = []
for step in range(1, max_steps + 1):
reply = model.chat(messages, TOOLS)
messages.append({"role": "assistant", "content": reply.get("content", ""), "tool_calls": reply.get("tool_calls", [])})
if not reply.get("tool_calls"):
return {"answer": reply.get("content", ""), "calls": calls, "steps": step}
for call in reply["tool_calls"]:
name, args = call["name"], call["arguments"]
if name not in TOOLS:
result = {"error": f"no tool called '{name}'"}
elif (error := check_args(TOOLS[name][2], args)):
result = {"error": error} # the model sees the error and can retry
else:
result = TOOLS[name][0](**args)
calls.append({"name": name, "arguments": args, "result": result})
messages.append({"role": "tool", "tool_name": name, "content": json.dumps(result)})
return {"answer": "Stopped: too many steps.", "calls": calls, "steps": max_steps}
Two models with the same interface
ScriptedModel follows a script, so you can test the loop and the scorer without any AI. OllamaModel calls a free local model. Ollama returns tool calls as message.tool_calls and takes results back as role: "tool" messages with a tool_name.
agent/models.py
# models.py: two interchangeable "brains" with the same chat() method.
import json, urllib.request
class ScriptedModel:
"""For testing the plumbing: replies come from a script, no AI involved."""
def __init__(self, plans: dict):
self.plans = plans # question -> (tool, args, answer template)
def chat(self, messages, tools):
question = messages[1]["content"]
tool, args, template = self.plans[question]
if tool is None: # a question the agent should not answer
return {"content": template}
if messages[-1]["role"] == "user": # first turn: ask for the tool
return {"tool_calls": [{"name": tool, "arguments": args}]}
result = json.loads(messages[-1]["content"]) # second turn: answer from the tool result
return {"content": template.format(**result)}
class OllamaModel:
"""A free local model through Ollama's /api/chat (tool calling supported)."""
def __init__(self, name="qwen2.5:3b", url="http://localhost:11434/api/chat"):
self.name, self.url = name, url
def chat(self, messages, tools):
specs = [{"type": "function", "function": {"name": n, "description": d, "parameters": s}}
for n, (_, d, s) in tools.items()]
msgs = [{k: v for k, v in m.items() if v} for m in messages]
for m in msgs: # Ollama wants {"function": {...}} wrappers
m["tool_calls"] = [{"function": c} for c in m.get("tool_calls", [])] or None
if m["tool_calls"] is None:
del m["tool_calls"]
body = {"model": self.name, "messages": msgs, "tools": specs, "stream": False,
"options": {"temperature": 0, "seed": 42}}
req = urllib.request.Request(self.url, json.dumps(body).encode(), {"Content-Type": "application/json"})
msg = json.load(urllib.request.urlopen(req, timeout=900))["message"]
return {"content": msg.get("content", ""),
"tool_calls": [c["function"] for c in msg.get("tool_calls") or []]}
The test questions and the scorer
Eleven questions. Ten need a tool. The last one, "What is my bank balance?", must be refused, because the agent cannot see it. Each question records the tool, the arguments and the number that should appear in the answer.
evals/cases.jsonl
{"q": "How much did I spend on food last month?", "tool": "total_spent", "args": {"category": "food", "month": "2026-10"}, "expect": "1340"}
{"q": "What did I spend on travel this month?", "tool": "total_spent", "args": {"category": "travel", "month": "2026-11"}, "expect": "1235"}
{"q": "Where did most of my money go in October 2026?", "tool": "top_merchants", "args": {"month": "2026-10", "n": 3}, "expect": "Hostel rent"}
{"q": "How much of my shopping budget is left this month?", "tool": "budget_left", "args": {"category": "shopping", "month": "2026-11"}, "expect": "701"}
{"q": "Am I over my food budget for November?", "tool": "budget_left", "args": {"category": "food", "month": "2026-11"}, "expect": "650"}
{"q": "Total spent on books and courses in October?", "tool": "total_spent", "args": {"category": "education", "month": "2026-10"}, "expect": "1149"}
{"q": "How much did I pay for phone recharge last month?", "tool": "total_spent", "args": {"category": "recharge", "month": "2026-10"}, "expect": "349"}
{"q": "Top 2 places I spent at this month?", "tool": "top_merchants", "args": {"month": "2026-11", "n": 2}, "expect": "Shoes"}
{"q": "How much did I spend on cabs and autos in November?", "tool": "total_spent", "args": {"category": "travel", "month": "2026-11"}, "expect": "1235"}
{"q": "How much travel budget was left last month?", "tool": "budget_left", "args": {"category": "travel", "month": "2026-10"}, "expect": "620"}
{"q": "What is my bank balance right now?", "tool": null, "args": null, "expect": null}
The scorer gives three marks per question: right tool, right arguments, right answer. Arguments are compared after filling in defaults, so top_merchants without n counts the same as n: 3.
evals/run_evals.py
# run_evals.py: score the agent on fixed questions.
# Usage: python -m evals.run_evals scripted|ollama [first question number, default 1]
import json, re, sys, time, pathlib
from agent.loop import run_agent
from agent.models import ScriptedModel, OllamaModel
CASES = [json.loads(line) for line in open(pathlib.Path(__file__).parent / "cases.jsonl")]
DEFAULTS = {"top_merchants": {"n": 3}}
def grade(case, run):
first = run["calls"][0] if run["calls"] else None
if case["tool"] is None: # should refuse: no tool, no made-up number
ok = first is None and not re.search(r"\d", run["answer"])
return ok, ok, ok
tool_ok = first is not None and first["name"] == case["tool"]
args_ok = tool_ok and {**DEFAULTS.get(case["tool"], {}), **first["arguments"]} == case["args"]
answer_ok = case["expect"] in run["answer"].replace(",", "").replace("₹", "")
return tool_ok, args_ok, answer_ok
def scripted():
plans = {c["q"]: (c["tool"], c["args"], "") for c in CASES}
for q, (tool, args, _) in plans.items(): # answer templates read the tool result
plans[q] = (tool, args, {"total_spent": "You spent Rs {total_inr} across {transactions} payment(s).",
"top_merchants": "Top: {top}", "budget_left": "Rs {left_inr} left of {budget_inr}.",
None: "I can only see the transactions you uploaded, not your bank balance."}[tool])
q = "Total spent on books and courses in October?"
plans[q] = ("total_spent", {"category": "shopping", "month": "2026-10"}, plans[q][2]) # a planted mistake
return ScriptedModel(plans)
model = scripted() if sys.argv[1] == "scripted" else OllamaModel()
first = int(sys.argv[2]) if len(sys.argv) > 2 else 1
totals, steps, start = [0, 0, 0], 0, time.time()
for i, case in list(enumerate(CASES, 1))[first - 1:]:
t = time.time()
run = run_agent(model, case["q"])
marks = grade(case, run)
totals = [a + b for a, b in zip(totals, marks)]
steps += run["steps"]
call = run["calls"][0] if run["calls"] else {"name": "-", "arguments": {}}
print(f"{i:2} {'tool' if marks[0] else '----'} {'args' if marks[1] else '----'} {'answer' if marks[2] else '------'}"
f" {call['name']}{json.dumps(call['arguments'])} ({time.time() - t:.0f} s)")
print(f" Q: {case['q']}\n A: {run['answer'][:160]}")
n = len(CASES) - first + 1
print(f"\nright tool {totals[0]}/{n}, right arguments {totals[1]}/{n}, right answer {totals[2]}/{n}, "
f"{steps / n:.1f} model calls per question, {time.time() - start:.0f} s total")
Run 1: scripted model (tests the scorer)
First we check the scorer itself. The script answers 10 questions correctly and gets question 6 wrong on purpose (shopping instead of education). A scorer that never fails is not a scorer, so this run must show exactly one miss.
python -m evals.run_evals scripted
1 tool args answer total_spent{"category": "food", "month": "2026-10"} (0 s)
Q: How much did I spend on food last month?
A: You spent Rs 1340 across 7 payment(s).
2 tool args answer total_spent{"category": "travel", "month": "2026-11"} (0 s)
Q: What did I spend on travel this month?
A: You spent Rs 1235 across 3 payment(s).
3 tool args answer top_merchants{"month": "2026-10", "n": 3} (0 s)
Q: Where did most of my money go in October 2026?
A: Top: [{'merchant': 'Hostel rent', 'total_inr': 6500}, {'merchant': 'Clothes shop', 'total_inr': 1200}, {'merchant': 'Earphones', 'total_inr': 899}]
4 tool args answer budget_left{"category": "shopping", "month": "2026-11"} (0 s)
Q: How much of my shopping budget is left this month?
A: Rs 701 left of 2500.
5 tool args answer budget_left{"category": "food", "month": "2026-11"} (0 s)
Q: Am I over my food budget for November?
A: Rs 650 left of 2000.
6 tool ---- ------ total_spent{"category": "shopping", "month": "2026-10"} (0 s)
Q: Total spent on books and courses in October?
A: You spent Rs 2459 across 3 payment(s).
7 tool args answer total_spent{"category": "recharge", "month": "2026-10"} (0 s)
Q: How much did I pay for phone recharge last month?
A: You spent Rs 349 across 1 payment(s).
8 tool args answer top_merchants{"month": "2026-11", "n": 2} (0 s)
Q: Top 2 places I spent at this month?
A: Top: [{'merchant': 'Hostel rent', 'total_inr': 6500}, {'merchant': 'Shoes', 'total_inr': 1799}]
9 tool args answer total_spent{"category": "travel", "month": "2026-11"} (0 s)
Q: How much did I spend on cabs and autos in November?
A: You spent Rs 1235 across 3 payment(s).
10 tool args answer budget_left{"category": "travel", "month": "2026-10"} (0 s)
Q: How much travel budget was left last month?
A: Rs 620 left of 1500.
11 tool args answer -{} (0 s)
Q: What is my bank balance right now?
A: I can only see the transactions you uploaded, not your bank balance.
right tool 11/11, right arguments 10/11, right answer 10/11, 1.9 model calls per question, 0 s total
It does: question 6 loses its arguments and answer marks, everything else passes.
Run 2: a free local model (qwen2.5:3b)
Now the same 11 questions with a real model: qwen2.5:3b on Ollama, temperature 0, seed 42, on a laptop CPU. The first run stopped after question 9: the laptop ran out of memory with other jobs running (the 3B model alone held about 1.9 GB of RAM). We ran questions 10 and 11 with the same settings using the start-number argument. Each question starts a fresh conversation, so splitting the run changes nothing.
python -m evals.run_evals ollama (stopped after question 9)
1 tool args answer total_spent{"category": "food", "month": "2026-10"} (111 s)
Q: How much did I spend on food last month?
A: Last month you spent a total of 1340 rupees on food. You made 7 transactions in the category of food last month.
2 tool args answer total_spent{"category": "travel", "month": "2026-11"} (33 s)
Q: What did I spend on travel this month?
A: This month in the travel category, you spent a total of 1235 rupees. You made 3 transactions in this category.
3 tool args answer top_merchants{"month": "2026-10"} (43 s)
Q: Where did most of my money go in October 2026?
A: In October, the most money went to Hostel rent with a total of ₹6500. Followed by Clothes shop with ₹1200 and Earphones at ₹899.
4 tool args answer budget_left{"category": "shopping", "month": "2026-11"} (38 s)
Q: How much of my shopping budget is left this month?
A: This month you have Rs. 701 left in your shopping budget. If you've spent more than that, you are over budget by the amount calculated.
5 tool args answer budget_left{"category": "food", "month": "2026-11"} (47 s)
Q: Am I over my food budget for November?
A: For this month (November), your food budget of 2000 INR has left you with a surplus of 650 INR. This means you are not over your budget; in fact, you are under
6 tool args answer total_spent{"category": "education", "month": "2026-10"} (60 s)
Q: Total spent on books and courses in October?
A: In October, you spent a total of ₹1149 on books and courses. This was made up of 2 transactions.
Could you please specify if these were all for education or if
7 tool args answer total_spent{"category": "recharge", "month": "2026-10"} (127 s)
Q: How much did I pay for phone recharge last month?
A: Last month you spent 349 rupees on phone recharge.
8 tool ---- answer top_merchants{"month": "2026-11"} (131 s)
Q: Top 2 places I spent at this month?
A: In this month, the top two places you spent money at are:
1. Hostel rent - Rs 6500
2. Shoes - Rs 1799
Would you like to know more about your spending?
9 tool args answer total_spent{"category": "travel", "month": "2026-11"} (66 s)
Q: How much did I spend on cabs and autos in November?
A: In November, you spent a total of ₹1235 on cabs and autos. This amount includes 3 transactions.
python -m evals.run_evals ollama 10
10 tool args answer budget_left{"category": "travel", "month": "2026-10"} (169 s)
Q: How much travel budget was left last month?
A: Last month, there was a budget of 1500 rupees for travel. The amount left in this category's budget is 620 rupees.
If you need the details for other categorie
11 tool args answer -{} (11 s)
Q: What is my bank balance right now?
A: To help you understand your spending better, could you please tell me how much you've spent in the 'recharge' category this month and last month? This informati
right tool 2/2, right arguments 2/2, right answer 2/2, 1.5 model calls per question, 180 s total
Across both parts: right tool 11 of 11, right arguments 10 of 11, right answer 11 of 11 by the scorer's rules. Full marks on 10 of 11 questions. What the runs show:
- Dates worked because the prompt says what today is. Every "last month" and "this month" became the right
YYYY-MM. Leave the date out of the system prompt and small models guess a year; our tool calling guide shows that failure. - Synonyms worked. "Books and courses" became
educationand "cabs and autos" becametravel. The enum in the schema does a lot of that work: the model can only choose from the six real categories. - One argument miss, with a right answer. For "top 2 places", the model left out
n=2, got three merchants back and picked two itself. The answer was right this time. The scorer still marks it, because on a long list it would not be. - The "pass" on question 11 is hollow. The model did not invent a balance, so the scorer's rule (no tool, no number) passes it. But its reply asked how much the user spent on recharge, which helps nobody. Automatic checks catch made-up numbers. They do not catch unhelpful replies. Read the passes as well as the failures, and tighten the rule (for example, require the reply to say it cannot see the balance).
- Unasked follow-up questions. Questions 6 and 10 ended with offers or questions nobody asked. Harmless here, annoying in a product. A line in the system prompt usually fixes it, and the scorer will tell you if the fix broke anything else.
- Speed. 11 to 169 seconds per question on a busy laptop CPU (the 11 was the refusal, with no tool call), 656 seconds for the first nine. Fine for testing, too slow for users. A hosted model or a GPU usually answers in a few seconds, and the same scorer then tells you whether the switch changed any answers.
Questions to be ready for about your agent project
If you put an agent project on your resume, expect to be asked about the parts around the model, not the model itself:
- How do you know it works? Show the test set and the score, including the misses.
- What happens when the model calls a tool with bad arguments? Point to your argument check and the retry.
- How do you stop it looping? A step limit, a budget, or both.
- What can it break? Which tools are read-only, and which need a person to approve.
- What does one task cost, and how long does it take? Measure it. Our CPU run above took seconds to minutes per question.
- Why an agent and not a fixed workflow? Sometimes the honest answer is that a workflow would do, and that is a good answer.
Honest limits
- The GitHub counts are rough. They come from GitHub's repository search (
in:name,description) on 21 November 2026. A repo can match without being that kind of project, and a real one can be missed if it is named differently. The exact queries were:"expense tracker" agent in:name,description(442),"issue triage" agent in:name,description(295),resume "job description" agent in:name,description(1,132),"study planner" agent in:name,description(425),rag citations agent in:name,description(2,020),"text to sql" agent in:name,description(1,989),"customer support" agent in:name,description(11,680),"research agent" in:name,description(20,942),"code review" agent in:name,description(7,177),"mcp server" in:name,description(178,524),"failing tests" agent in:name,description(151),"browser agent" in:name,description(3,546),"agent evaluation" in:name,description(2,867),"chat with pdf" in:name,description(4,827),"pdf chatbot" in:name,description(6,572),"travel planner" agent in:name,description(3,446),"youtube summarizer" in:name,description(2,064). - One small model, one run. The Ollama result is qwen2.5:3b at temperature 0 on 11 questions. A bigger model will likely score higher, and a different seed or prompt can change individual answers. Use the numbers to see the kinds of mistakes, not to rank models.
- The data is invented. The transactions, budgets and shop are made up for teaching. The code, the runs and the outputs are real.
- The project test plans are suggestions. Projects 2 to 13 were not built for this page. Their "prove it" steps describe how we would test them, using methods such as execution accuracy and planted bugs that the benchmarks linked above use at a larger scale.
Frequently asked questions
What are some good AI agent project ideas for beginners?
Pick an agent with two or three tools over data you control: an expense agent that answers questions from your transaction CSV, a GitHub issue triage agent that suggests labels, a resume vs job description gap agent, or a study planner that reads a calendar file. Each fits in a weekend with a free local model through Ollama.
Which AI agent projects look good on a resume?
The ones with proof. Add a fixed test set with known right answers and publish the score, including the misses. Also name the hard part you handled, such as refusing unanswerable questions, a human approval step before money moves, or a step limit that stops loops. A common idea with a published test beats a rare idea with none.
How should I structure an AI agent project folder?
Keep the tools, the agent loop and the model in separate files (for example agent/tools.py, agent/loop.py, agent/models.py), the data in data/, and the test questions plus the scorer in evals/. Separate files let you test each piece alone, and keeping test questions outside the code stops anyone from tuning the agent to the test.
Can I build an AI agent project for free?
Yes. Run a small open model such as qwen2.5:3b locally with Ollama, which supports tool calling through its /api/chat endpoint, and write the agent in plain Python with no paid API. A scripted test model lets you test the loop and the scorer without any model at all.
Which AI agent project ideas are overdone?
On 21 November 2026, GitHub search found 6,572 repositories matching "pdf chatbot", 4,827 matching "chat with pdf" and 3,446 matching "travel planner" with "agent". You can still build these, but add what most skip: a refusal test, a citation check and a score.
Do I need LangChain or LangGraph for an agent project?
No. The core agent loop is about 40 lines of plain Python, as the starter kit on this page shows. Frameworks such as LangGraph help when you need memory saved between runs, pausing for human approval or several agents, so add one when your project needs those, and say why in your README.
Crio AI Engineering
Build agents that work outside a demo
Crio's AI Engineering program is project-first: you build LLM apps, RAG pipelines and tool-using agents, then learn to evaluate, secure and ship them. Talk to the team about whether it fits you.
Sources
- Building effective agents (workflows vs agents; "finding the simplest solution possible"), 19 December 2024 · Anthropic
- Ollama tool calling: /api/chat tools, message.tool_calls, role tool with tool_name · Ollama
- GitHub REST API: search endpoints, total_count, rate limits · GitHub
- Searching for repositories: the in:name and in:description qualifiers · GitHub
- BIRD text-to-SQL benchmark (12,751 question-SQL pairs, 95 databases, execution accuracy) · BIRD
- SWE-bench Verified: a human-filtered subset of 500 instances · SWE-bench
- What is the Model Context Protocol (MCP)? · Model Context Protocol
- Playwright for Python: installation · Playwright
The starter kit was run on 21 November 2026 with Python 3 (standard library only) and Ollama serving qwen2.5:3b (temperature 0, seed 42) on a laptop CPU; both Output blocks are the real output. The transactions and budgets are invented. GitHub repository counts were fetched on 21 November 2026 from the GitHub search API with in:name,description; the queries are listed under Honest limits. Sources were read on 21 November 2026.
Last updated: