From 65e50f1c0b840868a9d35d64fc3cddb9b090f833 Mon Sep 17 00:00:00 2001 From: Marshall Greenblatt Date: Tue, 21 Jul 2026 17:02:28 -0400 Subject: [PATCH] ai: Add testing guidance for tools/ --- tools/AGENTS.md | 111 ++++++++++++++++++++++++++++++++++++++++++++++++ tools/CLAUDE.md | 1 + 2 files changed, 112 insertions(+) create mode 100644 tools/AGENTS.md create mode 100644 tools/CLAUDE.md diff --git a/tools/AGENTS.md b/tools/AGENTS.md new file mode 100644 index 000000000..58bbbb27e --- /dev/null +++ b/tools/AGENTS.md @@ -0,0 +1,111 @@ +# CEF Tools Contributor and Agent Guidelines + +This file applies to `//cef/tools` and its descendants. + +## Generator inventory and coverage + +- Every direct-child, non-test `make_*.py` generator must have a matching + `make_*_test.py` module. +- `make_generator_inventory_test.py` enforces the direct-child inventory. When + adding or removing a generator, update its expected count intentionally and + keep nested generators, helpers, and existing tests out of the inventory. +- Exercise every top-level generator function through at least one successful + test. Cover meaningful explicit error branches, writer behavior, and the + generator's CLI style; do not treat filename coverage as function coverage. +- Use standard-library `unittest`, `tempfile`, and `unittest.mock`. Run from the + CEF root: + + ```sh + python3 -m unittest discover -s tools -p '*_test.py' + python3 -m unittest discover -s tools -p 'make_example_test.py' -v + ``` + +- Run the complete suite twice for generator-wide work. These modules share + parser state, so the second run detects ordering and cleanup leaks. + +## Hermetic fixtures and goldens + +- Put deterministic inputs and reviewed goldens under + `tools/testdata/make_generators/`; use `generator_test_util.py` for shared + fixture, manifest, archive, and CLI helpers. +- Mock or fix time, year, Git metadata, versions, platforms, environment + variables, external commands, and formatter results when they affect output. +- Goldens must originate from a reviewed baseline, never from ignored + working-tree output, and tests must not contain an auto-update path. +- Use inline expectations for small output and reviewed full-file goldens for + larger generated C/C++ content. Add targeted assertions that explain the + behavior protected by a golden. +- Prove that customized CppToC/CToCpp bodies survive regeneration, source + fixtures are not modified, and relevant paths work when they contain spaces. + +## Writer contracts + +- Test every writer on its first generation and an identical second + generation. +- For `write_file_if_changed`, assert changed/unchanged returns and preserved + mtime. For unconditional writers, assert stable bytes across rewrites. +- Multi-output writers must report a change when any output changes. Test + repairs where only one output was corrupted. +- Pre-publication failures must leave outputs missing or unchanged. Characterize + partial I/O-failure output before changing it; atomic publication is a + separate behavior change. + +## CLI contracts + +- Cover invalid status, stdout, stderr, and partial artifacts, plus successful + stdout, single-file, multi-output, argparse, and distribution CLI styles. +- If a generator function returns metadata with its content, such as + `(content, customized)`, unpack the result at the CLI boundary and write only + the string content. Do not change the underlying generator return contract + merely to fix a CLI. +- Assert progress messages exactly when callers may depend on them, and return + explicit nonzero statuses for rejected invocations. + +## Parser and module state + +- Restore parser notification context and other module globals with + `try/finally`; add test cleanup so failing regressions cannot leak state. +- Generator modules intended for reuse must be importable without parsing + arguments, exiting, mutating files, calling Git, or launching subprocesses. + Put orchestration in `main(argv=None)` and use a conventional guarded entry + point when refactoring for importability. + +## Output equivalence + +- Establish an exact pre-refactor baseline and keep testability refactors + separate from behavior fixes. +- Testability refactors must preserve generated bytes, paths, ordering, + comments, whitespace, guards, encoding, newlines, modes, and symlink targets. +- Compare fixes to the latest approved baseline while retaining the original + refactor baseline as an immutable reference. +- Return-value, status, diagnostic, and CLI changes must not alter generated + files unless that is their documented purpose. +- For translator changes, run a repository-scale comparison covering C API, + CppToC, CToCpp, wrapper, dylib, and GYPI outputs. +- For distribution changes, compare the complete normalized tree. Compare + archive member paths, types, uncompressed bytes, and modes; normalize archive + timestamps only. Do not use raw archive-container hashes as equivalence. +- Use the same Python interpreter as the recorded baseline. Python-version + differences can change comment wrapping and produce misleading byte deltas. + +## Bug-fix workflow + +- Reproduce a bug with a focused regression before changing production code. +- Keep coverage/refactor checkpoints separate from behavior fixes so output + differences cannot be attributed to the wrong change. +- Keep unrelated bugs independently reviewable. Closely related instances of + the same root cause may be grouped when explicitly requested. + +## Formatting and validation + +- From the CEF root, format only explicit changed Python paths: + + ```sh + python3 tools/fix_style.py tools/make_example.py tools/make_example_test.py + ``` + +- Inspect formatter output immediately. The formatter may rewrap unrelated + legacy code; revert that churn with targeted patches. +- Compile every changed generator, helper, and test. +- Run `git diff --check` and scan untracked files for trailing whitespace. +- Re-run focused tests after formatting, then run the complete suite twice. diff --git a/tools/CLAUDE.md b/tools/CLAUDE.md new file mode 100644 index 000000000..43c994c2d --- /dev/null +++ b/tools/CLAUDE.md @@ -0,0 +1 @@ +@AGENTS.md