AI agents are taking on more of the work that keeps a property running. As that responsibility grows, the industry needs a rigorous way to measure whether they can deliver the outcomes residents and operators depend on.
At EliseAI, building agents for housing means working through the details that determine whether a request is actually resolved. A payment has to reconcile with the ledger. A repair has to reach the right team. An exception has to follow the community’s policy. Reliability depends on getting these decisions right across the entire workflow.
Today, we’re introducing ResidentBench, a benchmark for AI agents in property management, developed with General Context Labs. It brings that operational standard into a shared research framework, with 100 tasks across five workflow areas, expert-informed rubrics, and verification of the actions and records an agent leaves behind.
We built ResidentBench to make the next stage of progress in housing AI measurable. It lets us examine what our agent system contributes around a frontier model, identify the capabilities that still need to improve, and give the industry a common foundation for advancing them.
Comparing agent systems
A central question for housing AI is how much a purpose-built agent system adds to an increasingly capable underlying model. We tested six models inside EliseAI’s agent and their native agents, Claude Code or Codex, using the same tasks, starting state, and relevant tool access. Five independent runs per task measure how consistently each system completes the work.
Across all six models, EliseAI’s agent completes more tasks. The comparison also shows where substantial research work remains.
- Domain-specific agent design improves outcomes across models. EliseAI’s agent improves task completion by 16–22 percentage points across all six models, showing a consistent contribution from the system around the model.
- The strongest pairing completes 63% of tasks. Claude Opus 5.5 inside EliseAI’s agent leads the comparison, followed by GPT-6.1 Sol at 61%. Evaluating across model families lets us track how advances at the frontier translate into completed property-management work.
- The remaining failures define the next research problems. Even the leading pairing leaves more than a third of tasks incomplete. Each failure gives us a specific opportunity to improve how agents gather context, apply policy, coordinate actions, and recognize when human judgment is required.
For operators, this makes the value of an agent system concrete. For researchers, it makes the result inspectable. Per-task evidence connects each score to the agent’s actions, resident response, and final records, so improvements can be traced to the capabilities that produced them.
Bringing operational expertise into AI research
Housing is a demanding environment for autonomous agents. Information is spread across systems. Policies vary by community. A resident’s request can arrive with missing details, and resolving it can require several dependent actions. The consequences reach people in their homes.
Building for this environment gives us a detailed view of what reliable performance requires. ResidentBench turns that knowledge into tasks, environments, and grading criteria that other researchers and operators can examine. It makes the decisions property teams make every day explicit enough to evaluate systematically.
As agents become more capable, we need evaluations that keep pace with the work we expect them to do. A benchmark grounded in housing operations can help model builders identify meaningful capability gaps and help operators judge where new capabilities are ready to be used.
What’s in ResidentBench
We organized ResidentBench around five areas of property operations. The tasks reflect the breadth of work an agent needs to handle, including routine requests, exceptions, and decisions that require human approval.
Property-specific questions, unit availability, and tour coordination.
Request triage, work-order creation, access constraints, and urgent escalation.
Ledger reconciliation, payment disputes, and authorized adjustments.
Eligibility, approved offers, lease-term questions, and follow-up.
Documented requirements, information handling, and required approvals.
Each task runs in a synthetic property environment with resident records, documents, and tools for taking action. Incomplete requests, conflicting records, and approval boundaries recreate the conditions that make the work difficult. This lets us study operational complexity without using live customer data. Every task has a reference outcome and a rubric identifying the evidence required to verify completion.
A resident message or delegated assignment
Property records, policies, and connected tools
Decisions, tool actions, and resident communication
Expert rubric and final-state checks
Defining a task
A late-fee dispute illustrates the standard. A resident says they paid within the grace period, but a charge remains on the account. Resolving that request requires the agent to reconcile the evidence, understand its authority, and follow through with a consistent outcome across the resident response and account records.
The agent receives the resident message and access to the relevant account. It must determine which records matter, check the policy, and decide whether it can act directly or needs approval. The deliverable includes its response to the resident, any permitted ledger action, and the resulting case record.
Evaluating a task
The grading standard follows the work through to completion. Expert-informed rubrics check the evidence the agent finds, the policy it applies, the actions it takes, and the outcome it communicates. Every criterion maps to something a property team needs to trust before considering the request resolved.
| Dimension | Example criterion | Evidence |
|---|---|---|
| Record retrieval | Uses the correct resident’s payment timestamp and ledger entry. | Tool trace |
| Policy application | Applies the supplied grace-period and fee rules correctly. | Policy and decision |
| Authorized action | Uses the permitted adjustment or approval pathway. | Tool trace and final state |
| Communication | Explains the verified outcome and any remaining next step. | Resident response |
| Record consistency | Leaves a case record consistent with the actions taken. | Final state |
We report two metrics side by side.
- Rubric score. The fraction of applicable criteria satisfied within each task, averaged across tasks. This reveals which parts of a workflow an agent can complete.
- Task completion. The fraction of tasks where every required criterion passes. This measures how often the agent produces a fully acceptable outcome.
An agent may retrieve the right record and explain the policy but fail to obtain a required approval. Its rubric score captures those demonstrated capabilities, while the completion metric records the unresolved requirement. Together, the metrics help us study progress in individual capabilities and the reliability of the complete workflow.
Each applicable criterion has equal weight within its task. A task passes only when every required criterion passes, including authorization and escalation requirements. Tool and state checks verify structured actions, while criteria involving communication or judgment are assessed against task-specific reference outcomes.
Evaluation protocol and reproducibility
Every model runs all 100 tasks in each agent setting, with five independent runs per task. The comparison spans 6,000 task runs. Each run starts from a fresh copy of the same synthetic property environment, with matching records, policies, and available actions.
The evaluation records model and agent versions, prompts, tool adapters, and resource limits alongside the results. Task-completion rate is calculated over repeated runs within each task, then averaged across tasks. Rubric score follows the same task-level averaging, so tasks with longer rubrics do not dominate the aggregate.
Runs interrupted by infrastructure failures are retried from the original starting state. Agent errors, incorrect actions, and missed requirements count as task failures. Versioned task specifications, rubrics, and run traces connect each reported score to the configuration and evidence behind it.
Advancing housing AI together
We want ResidentBench to give the housing and AI research communities a shared way to understand progress. Operators bring the knowledge of which outcomes matter. Researchers bring new approaches to planning, retrieval, tool use, and evaluation. Bringing those perspectives together can raise the quality of both the benchmark and the agents it measures.
For EliseAI, this work connects research directly to product improvement. A failed task identifies a capability to strengthen. A successful run provides evidence for the work an agent can take on. Repeating that process as models and systems improve keeps our research accountable to the needs of the people using them.
We welcome property teams who want to contribute difficult scenarios, researchers who want to audit the rubrics or improve the evaluation methods, and agent builders who want to test new approaches. Their contributions can expand task coverage, challenge our assumptions, and make the results more useful across the industry.
We believe the next advances in housing AI will come from combining stronger models with deeper operational understanding and rigorous evaluation. ResidentBench is our contribution to making those advances visible, reproducible, and useful to the residents and teams they ultimately serve.