Schema Quality

Schema quality

Seven weighted checks that grade how well a store's tool schemas guide a model. Each tool, and the store overall, gets a score from 0 to 100 and a grade from A to F.

A model can only shop as well as the schemas let it. When a tool has no required fields, a model guesses which parameters are mandatory. When a parameter has no description, the model invents its meaning. Most runs where "the agent did something odd" trace back to a schema that left the model guessing. Schema quality measures that guidance before a model ever runs.

The seven checks

Each tool's inputSchema is scored against seven checks. Checks that cover several parameters give partial credit: three of four parameters described earns three quarters of that check's points.

CheckPointsWhy it matters
Has an inputSchema10 Without one, models may send arbitrary parameters. No schema stops scoring here.
Has properties15 An empty schema tells the model nothing about what to send.
Declares required fields10Models won't know which parameters are mandatory.
Every property has a description20 The heaviest check: descriptions do more than anything else to steer a model.
Every property has an explicit type15Untyped parameters invite wrong shapes.
Array properties define items15 Without items, models can't know what goes inside the array.
Object properties define nested properties15 An object with no declared structure is a guess the model has to make.

Grades

GradeScore
A90–100
B75–89
C60–74
D40–59
F0–39

The overall score is the average of the tool scores. Every issue names the parameters involved, so the list of issues is also your list of fixes.

Note

The API returns the grade as schema_quality: in the answer to POST /api/v1/inspect/tools, and on each run from GET /api/v1/sessions/{ulid}. One grader produces it, so the same tool list always gets the same grade. Schema quality is a measured fact about the store's published schemas, not a judgement of the store. It grades UCP tools only; the tools a store's website registers (WebMCP) are not graded.

Why letter grades here and not elsewhere

Schema quality is a weighted scorecard over many small criteria, which is where letter grades are honest (the same reason TLS scanners grade configurations A to F). Questions about behaviour with a few distinct outcomes, like identity enforcement, get words instead. A grade on a yes-or-no question would read as certification, which the Playground doesn't do.

Improving a grade

  1. Add required: the most common miss, and cheap to fix.
  2. Describe every parameter: write for a model that has never seen your store, with units, formats and allowed values.
  3. Type everything, including array items and nested object properties: the deep structure is where models invent shapes.
  4. Grade again after each change: fetch the tools again and the grade is worked out fresh; nothing is cached.

Something wrong on this page?