A System or a Demonstration: Seven Tests for AI Infrastructure in Institutional Real Estate

An investment committee is watching a vendor present. The deal on the screen is one the firm closed eighteen months ago; the vendor asked for it in advance, and the data room was opened to the system a week before the meeting. The system has read the leases, populated the firm’s own model template and drafted a memo in the house format. The passing rent agrees with the closing rent roll. The exit yield is the one the deal team used. A clause reference sits beside each figure, and the two the chair asks to see lead to the right page of the right lease. The committee approves a procurement.

What the hour tested was an artefact: one output, on one deal the vendor had studied, checked by people who already knew the answer. Nothing in it established what the system does on documents it has not seen, in the hands of an analyst the vendor has not coached, or how anyone would find out later that a figure was wrong. The committee did to a system what it would never do to a memo: it accepted a conclusion because the figures on the page matched the figures it expected.

The standard this desk applies to a memo has two halves: a stranger can follow each material figure to a document outside the memo, and the model that produced the figure performs the calculation the memo attributes to it. Neither half is a property of an output. Both are properties of a process, and they become properties of a system only when they hold for a run nobody in the room has prepared. That is the test that should have been run, and it differs in kind from the one that was.

The object matters because the memo is what the system is being bought to make. In a survey of 54 institutions by Addepar and the Stanford Initiative in Long-Term Investing, 97% had a formal memo template and process, 78% required a memo to be approved before any investment was made, and the write-and-review cycle averaged two months. A firm buying a system to shorten that cycle is buying a memo-production process, to be judged by the memo’s standard. Most deployments so far have placed a conversational interface over an existing document store and added none of the controls an institution would demand of an analyst doing the same job.

The Smart Bricks Research Lab has been working through what an evaluation of the process, as distinct from the output, should contain, drawing on bank model-risk guidance, the RICS professional standard on artificial intelligence and the published benchmark literature on document extraction. What follows is seven tests, the vocabulary they borrow, and a fortnight’s protocol for running them.

The instrument for examining a process already exists, written by people who examined models for a living long before language models arrived.

In April 2011 the Federal Reserve and the Office of the Comptroller of the Currency issued SR 11-7, their supervisory guidance on model risk management. It defines a model as a quantitative method, system or approach that processes input data into estimates, and gives it three components: an information input component, a processing component that turns inputs into estimates, and a reporting component that translates the estimates into something a business can use. It states that model risk increases with greater complexity, higher uncertainty about inputs and assumptions, broader use and larger potential impact. It names the discipline that manages that risk: effective challenge, meaning critical analysis by objective, informed parties who are separate from the model’s development and able to identify its limitations. And it is explicit about bought-in products. A vendor’s expertise sits outside the bank and its components are proprietary, so the bank is expected to obtain developmental evidence, testing results and a statement of limitations from the vendor, and to validate its own use of the product.

The Prudential Regulation Authority’s SS1/23, published in May 2023 and in force from May 2024, restates this as five principles: model identification and risk classification, governance, development and use, independent validation, and risk mitigants. Its definition of a model is wider than the American one. It includes methods whose input data are qualitative or based on expert judgement and whose outputs are qualitative, which describes a system that reads a lease and drafts a memo without any need for argument.

These documents govern banks and insurers rather than real estate investors, and what those firms report about their own AI use is instructive. In the Bank of England and Financial Conduct Authority survey of 118 financial firms published in November 2024, 75% were already using AI. About a third of use cases were third-party implementations, up from 17% two years earlier. Asked how well they understood the technologies they had deployed, 46% of firms reported a partial understanding and 34% a complete one. A decade of model-risk supervision has not closed the gap between adoption and comprehension.

Closer to the asset class, RICS published its professional standard on the responsible use of artificial intelligence in surveying practice in September 2025, with effect from 9 March 2026. For RICS-regulated firms several things are now mandatory: a written assessment of whether AI is the appropriate tool for a task, a register of AI systems whose outputs have a material impact, written due diligence on third-party suppliers, a written reliability decision on material outputs prepared by or under a named surveyor who accepts responsibility for it, and randomised dip sampling wherever AI produces outputs at volume. In the United States, NIST’s AI Risk Management Framework of January 2023 is voluntary, and names validity and reliability as the necessary condition of a trustworthy system.

Put SR 11-7’s anatomy and its three elements of validation, conceptual soundness, ongoing monitoring and outcomes analysis, against the two-part memo standard and seven tests fall out. Exhibit 1 sets them against the evidence a committee should ask for, and against what a prepared demonstration shows instead.

Test	What you do	Instrument	What failing looks like
1. Moved source	What you doChange one input in one source document and re-run; ask a stranger to retrace ten outputs in a minute each	InstrumentShare of ten outputs whose figure, path and reference all moved; share retraced inside a minute	What failing looks likeThe figure changes and the reference does not; the system explains where a number came from instead of showing it
2. Unfamiliar document	What you doBring three documents the vendor has not seen and the firm has abstracted by hand: a side letter that overrides, a bilingual divergence, an escalation with floor and cap	InstrumentField accuracy against the firm's abstract; share of low-confidence fields routed to a person	What failing looks likeEvery field filled, plausibly, including the ones it got wrong
3. Removal	What you doCount export-and-load steps; switch the system off for a day and list what the firm still holds in its own systems	InstrumentCount of manual steps; count of outputs that exist only in the vendor's store	What failing looks likeAbstracts and models that evaporate with the subscription
4. Assumption register	What you doAsk for the register from the last run; re-run on the firm's defaults and show the delta	InstrumentShare of chosen inputs with a named owner, default, deviation, reason and approval	What failing looks likeAssumptions inseparable from arithmetic; a growth rate nobody chose
5. Stop	What you doAsk for the incident log from a real run: restarts, escalations, declined actions, the stopping rule	InstrumentEscalations per hundred outputs; existence of a written stopping rule	What failing looks likeNo log, or a log with nothing in it
6. Wrong number	What you doPlant one contradicted error in a rent roll line and follow it through reconciliation, model, memo and correction	InstrumentCaught at reconciliation, yes or no; time from detection to corrected downstream outputs; count of outputs reached	What failing looks likeThe error is found by a person, corrected by hand in the memo, and survives in the model
7. Two-hundredth underwrite	What you doPrice the tenth and the two-hundredth underwrite with compute, reviewer minutes, exception queue and false-positive rate	InstrumentFully loaded cost per defensible underwrite at both volumes	What failing looks likeReview cost that does not fall with volume; a licence price offered in place of a cost

Each test is run on one closed deal the firm already holds, with the controller and the analyst who will use the system in the room. The instrument column is the number the buyer writes down; the last column is the answer that ends the evaluation.

Source: Smart Bricks Research Lab, drawing on the two-part standard in this desk's earlier notes

Test 1. Replay by a stranger. A run of the system leaves a record: the documents it read, the fields it extracted and where in each document it found them, the operations it applied, and the defaults it supplied where the documents were silent. The test is whether someone who was neither on the deal team nor employed by the vendor can take that record, re-execute the run and arrive at the same figure by the same trail. If re-execution needs the vendor’s help, or produces a different figure with an equally plausible explanation, the trail was a narrative written alongside the number rather than a log of how the number was made. The earlier note’s check, changing one input at source and re-running to see whether the stated provenance moves with the figure, is the single-number form of the same test.

The reference design for such a record is already in European law. Article 12 of the EU AI Act requires high-risk systems to be technically capable of automatically recording events across their lifetime. Commercial underwriting is unlikely to fall within the Annex III categories that make a system high-risk, and those obligations now apply from 2 December 2027, so this is not a duty most real estate firms carry. It is the design regulators reached for, and a committee can ask for it by contract.

Test 2. The holdout. The firm assembles a set of its own executed lease stacks, including amended ones, ones where a side letter contradicts the lease and ones where a commencement letter moved a date, together with one closed deal and its committee memo, and withholds all of it from the vendor until the day of the test. The system extracts and underwrites; the firm scores each field against the document that governs.

The published benchmarks explain why the firm’s own documents are the ones that count. CUAD, the expert-annotated contract review dataset published in 2021, took 510 commercial contracts and had lawyers trained for 70 to 100 hours label more than 13,000 clauses across 41 categories. The best model tested reached an area under the precision-recall curve of 47.8%, and 44.0% precision at 80% recall: to find four in five of the clauses that mattered, it flagged more wrong ones than right. Models have improved considerably since 2021, and the number is no ceiling; the design is the point. Held-out, expert-labelled documents in the domain, scored clause type by clause type, is the one measurement that tells a buyer anything.

Position matters as well as content. Liu and colleagues, in work published in the Transactions of the Association for Computational Linguistics in 2023, found that when the passage answering a question sat in the middle of a long context rather than at either end, accuracy fell by more than 20 points, and in the settings with 20 and 30 documents it fell below the 56.1% the same model scored with no documents at all. For a lease stack the consequence is direct: the governing term is often the one in the third amendment in the middle of the pile, and the holdout should put it there deliberately.

The deliverable is a distribution. Complex provisions, rent floors and caps, percentage rent, co-tenancy, are where extraction accuracy is known to fall away for people and machines alike, so the holdout over-samples them, and the result is reported by clause type and by position in the document, alongside the rate at which the system routed low-confidence fields to a reviewer rather than filling them. A headline accuracy averaged across easy fields is the demonstration again, in a different form.

• April 2011

  US supervisory guidance defines a model as input, processing and reporting components and requires 'effective challenge' separated from development

  Federal Reserve and OCC, SR 11-7

• May 2023

  PRA extends five model-risk principles, including independent validation, to AI and machine learning in modelling

  Bank of England PRA, SS1/23

• 7 characteristics

  Trustworthy AI per NIST, with accuracy claims to be paired with realistic test sets and external validity beyond training conditions

  NIST AI RMF 1.0, January 2023

• 1% of NAV

  Tolerance above which a NAV error in a Luxembourg real estate vehicle must be notified, corrected and compensated; 5% cap for professional-investor vehicles

  CSSF Circular 24/856, in force 1 January 2025

• January 2025

  Performance template ships the cash flow table with the metrics and reports IRR and TVPI with and without subscription lines

  ILPA Performance Template

Bank supervisors and fund regulators have written what a model is, who must challenge it, and what happens when its output is wrong.

Source: Federal Reserve SR 11-7 (4 April 2011); Bank of England PRA SS1/23 (17 May 2023); NIST AI 100-1 (January 2023); CSSF Circular 24/856 (28 March 2024); ILPA (22 January 2025)

Test 3. What the firm keeps. Every figure the system produces during the contract, the trail behind it and the library of assumptions it drew on live in the firm’s own systems of record, in formats the firm can open, so that after the contract ends every number can still be explained and reproduced. This is the buyer’s side of what SR 11-7 asks of a bank using a vendor model: the bank obtains the developmental evidence, the test results and the stated limitations, validates its own use, and expects the vendor to keep monitoring performance and to disclose the results. A firm that cannot open its own trails after exit has not bought infrastructure; it has rented an explanation.

The BoE and FCA figures show why this deserves a test of its own: third-party implementations roughly doubled as a share of use cases in two years while 46% of firms reported only a partial understanding of what they were running. Dependence is growing faster than comprehension in a population supervised on exactly this point for a decade, and a real estate firm with no supervisor asking the question has to ask it of itself.

Test 4. Who owns the default. A system that underwrites will supply defaults where the documents are silent: rent growth, exit yield, downtime between lettings, cost inflation, capital allowance. Each is an assumption, and the test is that the system treats it as one. Every default is labelled as an assumption rather than shown as though it were read from a document; each carries the name of a person inside the firm who owns it; each is versioned, with a change log that records when it moved and why; and none changes between two runs without the change appearing in both. A system whose defaults improve quietly between quarters has changed the memo without changing its author.

The memo literature already treats assumptions this way. The Addepar and Stanford brief describes memos as time-stamped artefacts whose assumptions should be tracked over time in case they later change, and reports that most surveyed firms have no structured process for going back over memos after the decision. Absent that process, a silently altered default is not caught at all. RICS has now written the governance of assumptions into a professional standard: the written reliability decision on a material output must state the assumptions made and any concerns about the datasets beneath it, under a named surveyor who takes responsibility for it.

A system that has never reported a failure has not been measured. It has been presented.

Smart Bricks Research Lab

Test 5. The failure log. Before any demonstration, the vendor produces its record of failure on documents like the firm’s: the fields it could not fill, the questions it declined, the answers later found to be wrong. Then the firm watches what the system does when it does not know. The acceptable behaviours are to route the field to a person, to refuse, or to flag; the unacceptable one is to fill the field with something plausible.

The best evidence on the trade-off comes from adjacent territory. Researchers at Stanford tested three retrieval-grounded legal research products built for professional use, on 202 pre-registered queries, alongside a general-purpose model. The professional tools produced a hallucinated answer on 17%, 33% and 17% of queries against 43% for the general model, and incomplete answers on 18%, 25% and 62%. The tool with the lowest error rate got there partly by declining to answer, and the longest answers carried the most checkable claims and took longest to check. The numbers belong to legal research and should not be carried across; the shape should. Any retrieval-grounded system trades coverage against error, and a committee should be shown where the vendor set the trade. In an earlier study by the same group, general-purpose models answered questions about US federal court cases with hallucination rates of 58% to 88%, and frequently failed to correct a mistaken premise in the question. The real estate analogue is a system that accepts the seller’s rent roll figure as its premise and never opens the lease.

The RICS standard requires members to understand a system’s limitations and failure modes and the risk of erroneous output, and where an output cannot reasonably be relied on, to tell the client so in writing. Article 14 of the EU AI Act supplies the design pattern for oversight: the person watching must be able to disregard, override or reverse an output, and to interrupt the system. A system with no refusal path leaves that person nowhere to stand.

Test 6. Whose name is on the wrong number. Some figure will be wrong. The test is what the record then shows: who found it, on what date, what changed downstream of it, and whose name authorised the output in the first place. A system output that reaches a committee carries an approver’s name, and the approver is a person in the firm, not the vendor and not the system.

Two external facts sharpen this. On 18 March 2024 the SEC settled with two investment advisers, Delphia and Global Predictions, for USD 225,000 and USD 175,000 respectively, over claims about AI capabilities the firms did not have; the regulator described the conduct as AI washing. The respondents were investment advisers, and the point is narrow: a claim about what a system can do is itself a regulated statement, and a firm that repeats a vendor’s claim to its investors has made that claim its own. Second, limited partners already inspect the correction record at fund level. Question 14.5 of ILPA’s Due Diligence Questionnaire 2.0 asks a general partner to name three to five investments below 1.0x TVPI and to say what went wrong, what was done and when outside experts were brought in. An LP who asks that of a portfolio will ask it of the system that underwrote the portfolio.

None of this requires a technology programme. It requires a fortnight, three people and a written request.

The material is the firm’s own. Twenty executed lease stacks, chosen to include amendments, side letters, a bilingual pair and at least one commencement letter that contradicts the lease it follows. One closed deal, with the committee memo that approved it and the realised first-year figures. The firm’s assumption library. All of it is withheld from the vendor until the test day.

The people are the controller who will sign off the outputs, the analyst who will live inside the system, and one person from audit or compliance who had no part in the deal. That third seat is effective challenge in miniature, and it is the seat a vendor’s demonstration never fills.

The request goes to the vendor in writing before the fortnight begins: run records in a form the firm can re-execute, the failure log on comparable documents, the assumption change log, and a sample export in the format the firm would keep at exit. The RICS standard is the template: due diligence is made in writing and followed up in writing, the answers and the extent of practical testing are recorded, and anything the supplier declines to provide is logged as a risk in the register. A refusal is a finding.

The scorecard has seven lines. Replay passes when the audit seat reaches the same figure and trail from the record without the vendor’s help. The holdout passes when the error distribution by clause type and position is produced, whatever it shows, and low-confidence fields were routed rather than filled. Exit passes when the sample export opens and explains a number without the system. Ownership passes when every default in the closed deal carries a name and a version. The failure log passes by existing, and by matching what the fortnight observed. Accountability passes when a deliberately planted error is found, dated and traced downstream. The closed deal’s own cycle, from document upload to a memo the controller would sign, is timed against the two-month average the memo survey reports.

In SR 11-7’s terms, replay, exit and ownership test conceptual soundness, the holdout against realised figures is outcomes analysis, and the failure log and the correction record become ongoing monitoring once the contract is signed. Exhibit 3 sets the tests against that vocabulary and the RICS requirements.

• 29%

  Share of 39,522 CMBS loans (USD 650bn, 2013-2019) whose underwritten income exceeded actual NOI by 5% or more

  Griffin and Priest, Journal of Finance, 2023

• 72%

  Share of 2005-2007 vintage US closed-end real estate funds that underperformed the index on both unlevered and levered bases

  MSCI, August 2022

• 44%

  Precision at 80% recall of the best model in a 510-contract, 41-category legal clause extraction benchmark; human annotators needed 70-100 hours of training

  Hendrycks et al., CUAD, NeurIPS 2021

• Factor of two

  Approximate muting of volatility from a spreadsheet dividing by a sum instead of an average, in a VaR model approved with a known copy-and-paste process and found only after the losses

  JPMorgan Chase Management Task Force, January 2013

• Two months

  Average write-and-review cycle per investment memo across 54 institutions; the majority run no structured ex-post review

  Addepar and Stanford Long-Term Investing, 2022

The failures the seven tests are built to catch have measured precedents: chosen numbers that drifted in one direction at scale, documents that defeated purpose-built models, a risk model approved with a known manual process, and a memo cycle whose cost the firm already pays.

Source: Griffin and Priest (2023); MSCI (2022); Hendrycks, Burns, Chen and Ball (2021); JPMorgan Chase & Co. Management Task Force (2013); Addepar and Stanford Long-Term Investing (2022)

Test 7. The hundredth underwrite. The price of computation has collapsed. Stanford’s AI Index for 2025 reports that the cost of inference for a system performing at the level of GPT-3.5 fell more than 280-fold between November 2022 and October 2024. Whatever the vendor charges per document, the token bill is not where the firm’s money will go. It will go on the exceptions a person has to clear, the low-confidence fields the system routed, and the dip samples the firm must run to know that the exception rate has not moved. The RICS standard says as much for regulated firms: where AI produces outputs at volume it is not proportionate to scrutinise every one, but the firm remains accountable for each, and must draw randomised samples at regular intervals. That sampling is a recurring budget line with a named person against it. The question to the vendor is the marginal cost of the hundredth underwrite with reviewer time and sampling included, and the exception rate observed at that volume in a comparable firm. A price per document without an exception rate is a quote for the cheap half. SR 11-7’s observation that risk grows with breadth of use, and the PRA’s tiering of validation effort by materiality, say the same thing from the supervisor’s side: a system that passed on ten underwrites is a different risk on a thousand.

Then what the tests do not do. They judge whether a system is infrastructure, not whether its assumptions are good; a system that passes all seven and carries the firm’s exit yield produces an opinion wherever the input was one, and labels it. None of the borrowed frameworks binds a typical real estate fund manager. SR 11-7 and SS1/23 apply to banks; the EU AI Act’s general provisions apply from 2 August 2026, its Annex III high-risk obligations from 2 December 2027 and Annex I from 2 August 2028, and commercial underwriting is unlikely to be caught by any of them; NIST’s framework is voluntary. For all but RICS-regulated firms the discipline is self-imposed, and a committee that wants it has to ask for it by name. The tests are a procurement instrument. Passing them once buys nothing after the contract is signed unless the failure log, the correction record and the dip samples are kept as standing work, budgeted as a role.

Return to the committee room. The same vendor, the same closed deal, now on the last day of the fortnight. The passing rent still agrees with the closing rent roll. The difference is that the person from audit reached it from the record alone, the error the controller planted in the third amendment was found and dated, the exit yield carries the name of the person who chose it, and the vendor’s log shows what the system did on the four leases where it could not find the term. The output has not changed. What the committee knows about it has.