Task and repository state together
The task statement and the repository state it applies to, kept together.
Each package pairs one repository issue with a runnable environment, a reference patch, a test patch, and a grading script — so a task can be attempted, checked, and scored without manual setup.
The task statement and the repository state it applies to, kept together.
An OCI layout under docker/ that starts without extra setup.
One accepted resolution, kept as a diff against the packaged state.
Checks are added by the test patch; grade.sh scores an attempt against them.
| Path | Contents |
|---|---|
task.json | Task statement, repository reference, and runtime settings. |
code.patch | Reference resolution as a diff. |
test.patch | Test additions applied before scoring. |
grade.sh | Scoring script whose output decides pass or fail. |
docker/ | OCI image layout for the sandbox. |
| Stage | How it is used |
|---|---|
| SFT | Issue-to-patch trajectories with a reference answer. |
| RL | Sandboxed attempts with automatic scores. |
| Evaluation | End-to-end resolution checks on fixed packages. |
| Layer | Contents |
|---|---|
| manifest.json | Every file with name, size, date, and collection ID. |
| UTF-8 JSONL | One record per line for text and annotations. |
| Documents | Data card, field schema, and a PDF spec for each collection. |
| Packaged tasks | docker/ (OCI layout) and json/ (task, patches, grading script). |
Tell us the language mix, task count, and evaluation setup. We reply within one business day. Sample packs, data cards, and field schemas are shared after your request is reviewed.
Each package ships an OCI image with its runtime settings. Network and resource requirements are confirmed per collection.
Yes. The test patch is separate, so additional checks can be layered on.
grade.sh runs the tests and returns the score used for pass or fail.