A task

A spec for a library, and a set of problems that a good library makes short. task.yaml names the language, the production library and the problems.

The clirs task: the design instruction and the clap production library each fill /library, which every one of the 13 evaluation problems imports. Each problem holds an instruction, tests, and clap and no-library reference solutions. One environment image sits under all of it, with no package registries.

Design instruction

design/instruction.md
Designer
The open spec: what the library is for, what it should support, and the packages it may use. The designer sees nothing else.

Production library

existing_library/clap/
Image build
A pinned real-world library, here clap 4.6.1. The production-library arm gets its source at /library with the crates installed; every other arm is checked to be without it.

Problems

evaluation/<problem>/ · 13 for clirs
ImplementerGrader only
  • instruction.md and workspace/ go to the implementer: a behavioral spec that names no library, and a starter project.
  • tests/ and solution/ stay hidden until grading. The production-library solution's metrics are the reference that simplicity is measured against.

Environment

environment/Dockerfile
Image build
One image for the design phase and every problem, with no package registries. Pre-installed lists everything in it.