A useful software-factory evaluation asks two questions. Does the foundation remain under the team’s control as agents, models, workflows, and repositories change? Does it keep operating when an agent dies halfway through real work?
This is Gas City, Inc.’s evaluation method. It deliberately values engine ownership, worker independence, durable external work, portable factory methods, and an extensible ecosystem. Those are the properties that let a factory become an organizational capability instead of a temporary wrapper around one agent product. Operational results still need the evidence package described below.
Score the foundation before the demo
A polished run can hide where the factory’s durable value lives. Score these properties before comparing task completion:
| Foundation property | Passing question | Evidence |
|---|---|---|
| Engine ownership | Can the team inspect, run, extend, and retain the machinery that operates the factory? | Source, license, deployment artifacts, extension seams |
| Worker independence | Can different coding-agent CLIs and custom harnesses serve as workers without redefining the factory around one vendor? | Agent configuration and the same workflow run with different workers |
| Durable external work | Do identity, assignment, dependencies, provenance, and history survive outside any one agent session or computer process? | Work records, history, crash handoff, ready-work query |
| Method portability | Can the team version, carry, fork, and reuse its agents, workflows, automations, prompts, and skills? | Inspectable configuration and export or installation path |
| Open extension and reuse | Can factory methods accumulate through an inspectable ecosystem rather than only inside one account? | Extension source, registry or distribution mechanism, installation record |
These are not generic procurement criteria. They measure whether the buyer is acquiring a factory capability that can compound. Product polish, managed compute, governance dashboards, and vendor support belong in the decision, but they do not substitute for ownership of the factory’s work and method.
Hold the test conditions still
Choose a small repository with a fast test suite and a task that requires at least two dependent steps. A bug with a reproducible failing test works well. A narrow feature with an acceptance test works too.
Run each system with the same:
- starting commit and repository state;
- task statement and acceptance criteria;
- model and model-serving setup where the products permit it;
- token, spend, and wall-clock budget;
- network and tool permissions;
- point where the active agent process is terminated.
Prompts and configuration may differ because they are part of each system. Publish them. Hidden tuning can turn a fair-looking table into an unreproducible demo.
Then score five operating properties
Use pass, partial, and fail. Record operator interventions separately instead of hiding them inside a weighted score.
| Property | Test | Evidence |
|---|---|---|
| Routing | Include independent, blocked, and review work. Observe what starts and why. | Work graph, assignment record, routing config, timestamps |
| Durability | Kill the active agent after it records progress. Start a fresh session. | Persisted work, decisions, dependency state, recovery transcript |
| Visibility | Give a second operator access during the run. Ask what is active, blocked, failed, and complete. | CLI or UI capture plus the underlying state queried |
| Completion | Let the workflow reach its stated outcome. Run acceptance tests and inspect the diff. | Test output, artifacts, review notes, final repository state |
| Operator load | Count every rescue, re-prompt, manual state repair, and hidden handoff. | Intervention log with reason and duration |
A pass means the system completed the property under its normal operating model and left readable evidence. Partial means it reached the result with a documented intervention or lost some of the expected record. Fail means the evaluator had to reconstruct state, bypass the tested mechanism, or could not reach the outcome.
Routing: test readiness, not dispatch volume
Create at least one task that is ready, one blocked by unfinished work, and one reserved for review. Start enough workers to make a collision possible.
A passing system sends ready work to an eligible agent, leaves blocked work alone, and records ownership clearly enough that a second agent does not start the same task. It should also explain the route through configuration or state an evaluator can inspect.
Starting three agents quickly earns no credit by itself.
Durability: kill the process on purpose
Terminate an agent after it has made a decision and changed some code, but before the work is complete. Do not preserve the original chat as a secret recovery aid.
Start a fresh session and ask it to recover from the system’s normal work record. The evaluator should be able to identify the active task, previous decisions, blockers, and next action. Code on disk is useful evidence, though it does not replace work status and reasoning.
With Gas City’s default bd work-store provider, the bead and Dolt history survive while the session disappears. The maintained Beads recovery docs cover database and synchronization recovery; the coding-agent crash recovery guide focuses on the session handoff.
Visibility: ask someone who did not author the workflow
The workflow author already knows what the system was supposed to do. Use a second operator for the visibility test.
Give them the ordinary CLI, dashboard, logs, or API. Ask four questions: what is running, what is blocked, what failed, and what outcome has been produced? Note every time they must read implementation code or ask the author.
Visibility claims should name the underlying source. Gas City materializes graph-workflow runs as root and step beads, then projects the local run view by folding retained bead lifecycle events and enriching it with live or linked detail. Its event list and stream APIs are documented in the events reference. It does not capture a separate governed team operational record. Evaluating Gas City should not give it credit for Beads Team Server capabilities.
Completion: define the finish before the run
“Agent exited successfully” is not a software outcome.
Write the acceptance test, required review evidence, and expected repository result before the evaluation. For a code-delivery workflow, the outcome might be a reviewed pull request ready for human review. Another workflow may produce a diagnosis, migration plan, or generated artifact.
Run the actual tests. Inspect the diff. Check that blocked steps did not disappear and that review addressed the stated criteria. Completion quality belongs beside operational reliability.
Operator load: keep a rescue log
Every manual intervention gets one line:
time | action | reason | minutes | changed system state?Distinguish normal policy gates from rescue. A planned human approval is part of the workflow. Re-prompting an agent because the orchestrator lost its assignment is operator load. Repairing a corrupted work record is more serious than answering a designed question.
Report the count and total time. Do not compress them into a proprietary score.
Publish the evidence package
A credible result includes the repository and starting commit, task, acceptance tests, prompts, configuration, commands, model identifiers, budgets, tool permissions, logs, work-state exports, failure-injection point, artifacts, intervention log, scoring decisions, and review date.
If privacy prevents publishing a real repository, build a public fixture with the same failure shape. Redact secrets rather than omitting the operating details that would let another evaluator reproduce the result.
Gas City’s side of a test can begin with the Quickstart and How Gas City works. Those links document the implementation; they do not predetermine a passing score.
Compare products inside one category
Use this rubric to compare software factories with software factories. Gas City, Factory.ai, and 8090 may organize work differently, but each claims to operate continuing software production. Hold the outcome and evidence standard fixed, then disclose the configuration each candidate needed.
Agent frameworks need a different evaluation. Their job is to help developers implement agent applications, so node APIs, state models, tool integration, checkpoint semantics, and deployment ergonomics belong in that rubric. Running a framework against a factory scorecard mostly measures the factory infrastructure the evaluator had to build around it.
The software factories versus agent frameworks guide draws that boundary before the test begins.
The multi-agent orchestration guide explains why these five properties matter.
Run the rubric on one real failure. Publish the receipts before the score.
Related articles
- Orchestrating multiple AI agents: Understand why ownership, dependencies, recovery, visibility, and gates are the tested properties.
- Gas City vs Factory.ai: Apply the rubric to two products in the software-factory category.
- Software factories vs agent frameworks: Choose the right evaluation set before scoring products.
- Recover agent work after a crash: Use a concrete failure sequence when designing the durability test.