Can we trust AI? Or should we live with inconsistency?

Inconsistency is one of the most common objections raised when in-house counsel consider AI-assisted contract review. The concern is understandable. Legal work depends on predictability. If a tool produces different outputs from the same input, the reasoning goes, how can it be relied upon? 

It is a fair question. But it is worth asking a harder one alongside it: how consistent is manual review in practice?

The (in)consistency we already accept

Contract review in construction is rarely performed under ideal conditions. Timelines are tight, document volumes are high, and the same lawyer is often reviewing similar clauses across multiple projects simultaneously. In that environment, the quality and depth of review is not always uniform.

A clause that receives close scrutiny on a Monday morning may receive a lighter touch late on a Friday. A risk that is escalated on one project may be accepted without comment on another, not because the risk has changed, but because the reviewer is stretched.

Familiar-looking amendments can be waved through simply because they look like something that has been seen and managed before.

Lower value contracts may not be reviewed in full or at all. That is the reality of limited resources, internal policies designed to streamline workflows and avoid bottlenecks.

None of this reflects poorly on in-house counsel. It reflects the reality of working under sustained pressure. But it does mean that inconsistency is already present in the system and is largely invisible.

AI inconsistency and how to manage it

It is true that AI outputs are not identical every time. The nature of large language models means that phrasing, emphasis, and structure may vary across reviews. For general-purpose tools applied to complex legal documents, that variability can be a genuine problem. 

Specialist tools, built for a defined domain and set of tasks, are designed to contain that variability within acceptable limits. The analytical framework applied to each document remains the same. The risk categories are fixed. The playbooks against which deviations are assessed do not change between reviews. What varies is how the tool expresses a finding, not whether it identifies one.

Two experienced lawyers reviewing the same clause will often reach the same conclusion through slightly different reasoning or frame the same risk in different terms. That is not inconsistency in any meaningful sense. It is the normal variation of professional judgement. Well-designed AI tools operate within similar bounds.

Risk profiles and Playbooks: Making consistency structural

The most effective way to manage AI consistency is to embed it into the system from the outset. This is where domain specific understanding, risk profiles and playbooks become essential.

Specialist tools such as Contrafly are designed to perform specific tasks and are structured and provided with the domain specific knowledge which inform every analysis they undertake.

A risk profile defines what an organisation considers acceptable across different clause types: which deviations are tolerable, which require escalation, and which are non-negotiable. 

A playbook captures the organisation's established positions on those clauses and the preferred responses when deviations are identified.

When a specialist AI tool operates within those parameters, the outputs it produces reflect agreed positions rather than variable judgements. The tool does not decide what is acceptable. It measures what it finds against what has already been decided by the legal team and flags accordingly.

This is a more reliable form of consistency than relying on individual reviewers to remember and apply those same positions under time pressure, across multiple projects, without a structured framework to guide them.

The risks that fall through the gaps

Manual review carries its own form of inconsistency that is rarely discussed openly: the risk that something is missed not because the reviewer is unskilled, but because the volume of material makes it genuinely difficult to maintain the same level of attention across a long document.

In heavily amended FIDIC or NEC contracts, risk often accumulates across multiple clauses rather than residing in any single provision. A change to a limitation of liability clause only becomes significant when read alongside an amendment to the indemnity provisions and a modified dispute resolution mechanism. Identifying that combination requires systematic cross-referencing that is time consuming to do manually, and easy to do incompletely.

Specialist AI tools perform that cross-referencing systematically on every document, every time. The risk is not that the tool will miss something because it is tired or distracted. The risk of omission is managed structurally rather than being managed by individual effort.

A more honest comparison

The right comparison is not between a perfect manual review and an imperfect AI review. It is between real-world manual review, with all of the time pressures and resource constraints it involves, and a specialist AI tool operating within a structured framework of defined risk positions.

That comparison looks different from the one that tends to be assumed. When AI tools are built for a specific domain, constrained by defined parameters, and used as the first stage of a review process that still involves human judgement, the consistency they provide is not theoretical. It is structural.

Contrafly is built on that principle. The consistency that matters is not whether every output looks identical. It is whether every material risk is identified and presented to the right person, with the information they need to make a sound decision. 

If you would like to see how that works in practice, we are offering a small number of walkthroughs to construction legal teams ahead of our formal launch. Follow Contrafly on LinkedIn or get in touch directly to arrange a session.