Behind the scenes, two months later: How we ship Google Agent Skills like products

extract

Behind the scenes, two months later: How we ship Google Agent Skills like products Remigiusz Samborski Lead Developer Relations Engineer In my previous post, I took you behind the scenes of Google Agent Skills, covering how the project started, how we maintain quality at scale, and how we handle governance. In this post, I'll share what we've built since then: the automated quality gates that every change must pass, the AI review agent that inspects skill quality, and the lessons that we learned along the way. You can adopt similar gates and review rules in your own skill repository. Since my last post, the Google Agent Skills repository has passed 21,000 GitHub stars, up from 15,000. It now hosts over 170 skills from teams across Google, and we distribute them across multiple AI platforms. Building agent skills for a single developer or a small project is straightforward. You write a SKILL.md file, define a few instructions, test it manually, and it works. At our scale, manual workflows stop being effective. Unvetted edits can introduce subtle bugs, and it becomes difficult to catch issues before they reach users. The core philosophy: Ship skills like products One key lesson from my previous post: a skill is a living product, not a one-off document. In practice, that means we ship every skill update like a product release. It's tempting to treat a skill as documentation. But documentation explains concepts to a human reader, while a skill steers an AI agent's reasoning loop, tool selection, and execution path. A misplaced phrase in a skill can cause an agent to invoke invalid command flags, enter an infinite loop, or waste tokens. That makes every instruction edit a behavioral change. You wouldn't ship a product update without testing it, and the same goes for a skill. This principle drives three core requirements: - Strict semantic versioning: Every behavioral change requires an explicit version bump. - Deterministic automated validation: Linters and validators check every proposed change before a human or AI reviewer sees it. - Mandatory evaluation evidence: Every change includes an evaluation report. It compares the skill against its previous version, or—if the skill is new—against an agent without the skill. Automated conformance gates: The pull request pipeline In my previous post, I briefly listed the checks that every skill passes on check-in: linters, link checkers, and AI-assisted checklists. Here's what's inside them, plus two gates that I haven't covered before: semantic versioning and evaluation suite validation. When an author submits a change to our internal skills repository, an automated continuous integration and continuous delivery (CI/CD) pipeline triggers immediately. Before a skill is exported to GitHub, it must pass these six automated gates and receive human approval: - Structural conformance: Directory layout, allowed files, and taxonomy. - Skill linters: Frontmatter, folder naming, and Markdown rules. - Semantic versioning: Version bump for every behavioral change. - Evaluation suite: Valid eval suite and a fresh eval report. - Link validation: Links are valid. - AI review agent: Skill quality and design standards. - Human approval: Final approval from a human reviewer (a subject matter expert). Structural conformance and skill linters Our automated validation suite inspects every pull request for structural and syntactical correctness. Some of the rules include these requirements: - Folder name and metadata name must match. - SKILL.md and description length limits are enforced. - All required fields in the frontmatter metadata are present. - Only a curated list of files and directories is allowed within a skill directory. - All Markdown files follow formatting rules. - The main README.md file and the machine-readable index.json registry are up to date. Semantic versioning Every skill declares a semantic version in its SKILL.md frontmatter metadata: First, our automated checks verify that the version is a quoted string. YAML parses unquoted values like 2.0 as numbers, which causes unexpected parsing errors in tools that expect a string. Second, the validator checks the version transition itself. New skills must start at 1.0.0, and any change to an existing skill's instructions, scripts, or references requires a version bump or the pull request is blocked. Standard reset rules apply: bumping MAJOR (for example, 1.4.2 to 2.0.0) resets MINOR and PATCH to 0, and bumping MINOR (for example, 1.4.2 to 1.5.0) resets PATCH to 0. Why enforce strict versioning if the agent only reads the Markdown instructions? The version number is for the developer and their package tooling, not for the model at runtime. Skills don't auto-update in the background, which means there's no network check or latency overhead each time an agent invokes a skill. Instead, explicit versions let developers compare their locally installed skills against the upstream catalog and decide when to pull changes—for example, by running the npx skills update command. This versioning also supports workflows where teams want to lock a critical skill to a known-good version to guarantee reproducible agent behavior across their projects. Evaluation suite validation Our continuous evaluations (evals) compare agent performance with and without each skill. Those results are only as reliable as the evaluation suites behind them. If a suite contains broken file references or compound expectations, test results become noisy and untrustworthy. Our validator parses each evaluation suite and the pull request description. Then it checks several strict invariants: - Resolution of file-based prompts: When a test prompt references a file (such as @file:test_prompt.txt), the validator confirms that the target file exists and that it isn't empty. - Atomic expectations: Each expectation must check exactly one criterion. The validator blocks compound expectations until authors split them. Lesson 2 later in this post shows why. - Duplicate detection: The validator flags duplicate test prompts and conflicting expectations within the same evaluation suite. - Mandatory evaluation report: Every pull request that modifies a skill must link to an evaluation report in its description. - Eval staleness check: Authors usually iterate on a pull request as reviewers leave feedback. The validator confirms that the linked report ran against the latest commit. Link validation Over 170 skills link to documentation for fast-evolving Google products, and those pages can move long after a skill passes review. Rather than waiting for users to report broken links, we validate links at two stages. Before merge, pull request checks verify all relative links, such as files in references/. Then, every Monday, a GitHub Actions workflow runs lychee across all external links in the public repository and it reports any broken links to skill owners for remediation. AI review agent Linters and schema validators catch syntactic errors, but they can't evaluate overall quality. They can't tell you if an instruction is confusing, if a reference guide is too verbose, or if a task should have been a bash script instead of a large language model (LLM) instruction. To solve this, we integrated an AI review agent into our review pipeline. When a change affects a skill, the AI reviewer inspects it using a dedicated review skill. Yes, we use a skill to review skills. Here are the core rules the AI reviewer evaluates. You can adopt these rules in your own skill development workflows. 1. Skill suitability and the deterministic trap The first question our AI reviewer asks is simple: Does this task belong in an instruction? We often see authors fall into the deterministic trap. They write rigid, numbered sequences of command-line interface (CLI) commands directly into Markdown text. LLMs are probabilistic engines. If an operation requires a strict sequence of commands where a variation can cause an error, embedding those commands in the instruction text invites hallucinations. Our AI reviewer enforces a clear boundary between deterministic and probabilistic logic. Rigid CLI sequences, JSON parsers, and deployment steps must live in executable scripts under scripts/. Markdown instructions are reserved for tasks that require judgment, such as choosing an architecture, diagnosing error logs, or synthesizing code. This rule complements the Model Context Protocol (MCP) guidance that I shared last time: prefer remote MCP tools whenever possible. When you must fall back to CLI or API calls, wrap them in a script instead of spelling them out in the instructions. 2. The L1/L2/L3 modular architecture Context windows are valuable. Loading a 2,000-line skill into an agent consumes context and degrades reasoning. Our AI reviewer enforces a three-tier information hierarchy that follows the progressive disclosure model of Agent Skills: - Level 1 (Metadata): The YAML frontmatter (name, description). The agent uses this concise summary for discovery and routing. - Level 2 (Core instructions): The main SKILL.md file. The body must stay compact (under 500 lines). It provides workflow overviews, decision trees, and guardrails. - Level 3 (Reference docs): Detailed API references, schemas, and other reference tables live in a references/ directory. Our AI reviewer enforces two distinct rules for Level 3: flat directory structure and single-hop indirection. Keeping references/ flat (no subdirectories) is a repository convention that discourages sprawling documentation trees and makes file discovery simpler. More importantly for runtime performance, the reviewer blocks chained references, where SKILL.md links to a reference file that links to yet another file. Each hop requires an extra tool call, adding latency and cost, and our evaluations showed that agents frequently stop before following a second level of chained references. 3. Non-interactive script design When skills bundle helper scripts in scripts/, we expect agents to execute them directly in terminal sandboxes. Our AI reviewer flags any script that blocks on interactive prompts. Every script must accept inputs through command-line flags or environment variables, provide --help documentation, and include a --dry-run or confirmation flag for destructive operations. Scripts also need to separate output cleanly, writing structured data (usually JSON) to standard output and writing actionable error messages to standard error. Example AI review checklist snippet The review skill also checks frontmatter for spec compliance and discovery, and it turns the L1/L2/L3 architecture into concrete structure rules. Here's an excerpt from those criteria: Human approval After it passes all automated checks, every pull request must receive approval from a human reviewer. The reviewer is typically the skill's subject matter expert and, for new skills, a member of the Central Skills team. Lessons learned from repository history Six months of commit history and pull request discussions reveal several practical takeaways. 1. Scope CI checks carefully Early in the project, we configured automated pull request checks with broad directory patterns. One check required an approved bug tracking ticket on every change. Because the pattern matched too broadly, engineers making a one-line typo fix were blocked for missing a tracking ticket. Meanwhile, newly added skills in deeper subdirectories went uninspected. Takeaway: Match automated check rules to specific file operations, such as added versus modified files, and verify exact path depths. Treat minor edits to existing skills differently from new skills. 2. Atomic expectations When we started writing evaluation suites, we saw authors write compound expectations like this: If the agent configured the region and lifecycle rules but missed the uniform access flag, the LLM-as-a-judge scorer returned inconsistent ratings. Migrating to atomic expectations resolved the inconsistent ratings: Takeaway: Check exactly one criterion per expectation. Atomic expectations make evaluation results clear, reliable, and debuggable. 3. Documentation drift happens fast In a multi-contributor repository, documentation tables drift within days if maintained manually. Takeaway: If you publish an index or catalog, never rely on contributors to update it manually. Instead, write a generator script and enforce it with a CI unit test. That's the origin of the README.md and index.json checks in our linters. Our index.json even opens with a warning: "This file is generated. Do not edit it by hand." 4. Gate expensive LLM checks behind fast deterministic ones Running full evaluation suites and an AI review agent on every change can get expensive fast, both in token usage and CI runtime—especially when authors push frequent small commits during review. In our pipeline, fast deterministic checks run first. Structural conformance, skill linters, and semantic versioning act as blocking gates: if any of them fail, the pipeline stops immediately and it won't trigger the AI reviewer or evaluation checks until those issues are fixed. Takeaway: Stage your pipeline so cheap, deterministic checks block broken changes before LLM-powered gates run. When you build a similar system, consider adding further cost guardrails, such as manual triggers for heavy checks (for example, a /run-evals pull request comment) or only triggering automatic re-runs when the diff exceeds a specific threshold. Conclusion Building a scalable Agent Skills repository requires more than good prompt engineering. It requires solid developer infrastructure: - Ship skills like products: version, validate, and evaluate every change. - Enforce semantic versioning and atomic expectations with automated CI gates. - Use an AI reviewer to catch design anti-patterns, such as the deterministic trap. - Check links before merge and on a schedule. Links and further reading To get started with Google Agent Skills, check out these resources: - Google Agent Skills repo: github.com/google/skills - Previous post: Behind the scenes: How we build, test, and scale Google Agent Skills - Launch announcement: Level Up Your Agents: Announcing Google's Official Skills Repository - Skills tutorial: Introduction: What Are Google Cloud Agent Skills? What's next? If you found this post helpful, share it with your friends on social media and connect with me on LinkedIn, X, or Bluesky. Thanks for reading!