AI Capability Case Study — Vision and Speech AI: Understanding AI Probability, Confidence and Fitness for Use
- Apr 14
- 6 min read
Updated: 1 day ago
1. Organisational Problem
A residential property management organisation was exploring the use of AI-enabled video monitoring to improve visibility of operationally relevant events across multiple sites.
This included events such as parcel deliveries, visitor arrivals and contractor attendance that would otherwise require manual review of recorded video footage.
The organisation believed that AI-powered image recognition could reduce manual review effort and improve the speed at which important events were identified.
However, Vision and Speech AI systems often generate outputs based on learned patterns, probabilities and confidence levels rather than certainty.
This meant the key challenge was not simply whether the system could detect events. It was whether performance would be sufficiently reliable, appropriately governed and fit for operational use.
Vision and Speech AI includes the ability to analyse images, video, speech and audio in order to detect objects, recognise events, interpret environments and support decision making.
In the Orr Consulting AI Universe, Vision and Speech AI helps address the question:
How can machines interpret the physical world?
2. Situation
The organisation identified parcel detection as an initial use case.
The objective appeared straightforward.
When a delivery driver approached a property and deposited a parcel, the system would automatically identify the event and generate a notification.
However, the organisation quickly discovered that defining success was more complex than expected.
Questions emerged such as:
What counts as a parcel?
How should the system distinguish between parcels and shopping bags?
What happens if the parcel is partially obscured?
What if delivery occurs during darkness or poor weather?
What level of accuracy is acceptable?
How many incorrect notifications are tolerable?
An important question therefore emerged.
The issue was not whether the AI could identify parcels some of the time. The issue was determining what level of performance would be sufficiently reliable for operational use.
This shifted the focus from technology capability towards evidence, governance and fitness for purpose.
Leaders recognised that successful deployment would depend not only on the AI model itself but also on testing, governance and evidence-based decision making.
Orr Consulting was engaged to support the organisation through a structured AI transformation approach.
3. Background
The initiative was approached using the Orr Consulting AI Transformation Process, a structured strategic framework for selecting, designing and delivering AI opportunities through a Discover, Design and Deliver approach.
Rather than starting with assumptions about the technology, the approach focused on defining the business outcome, establishing how success would be measured and determining what level of performance would be required before operational deployment could proceed with confidence.
The process identified a need to move beyond assumptions regarding AI capability and establish an evidence-based view of operational fitness for use.
The key activities and decisions are summarised below.
3.1 Discover
In the AI Transformation Process, the purpose of the Discover stage is to build understanding, assess readiness and identify realistic AI opportunities before committing to strategy or investment.
The Discover stage confirmed that the business problem was genuine and that Vision and Speech AI represented a potentially suitable response. The Discover stage also highlighted the importance of organisational understanding regarding AI confidence, probability and operational suitability, reflecting themes commonly explored through Orr Consulting's AI Capability and Maturity Assessment approach.
For this use case, suitability was assessed against the key AI Use Case Discovery criteria:
Alignment to Business Strategy — The use case supported organisational objectives to improve visibility of activity across sites, reduce manual review effort and improve responsiveness to important events.
Cost, Complexity and Risk — The proposed solution involved deploying an existing Vision and Speech AI capability within a controlled pilot, creating a manageable implementation profile while allowing performance and governance requirements to be evaluated.
Impact and Benefits — Potential benefits included improved operational efficiency, reduced manual monitoring effort, faster event identification and improved user experience.
Data Readiness — Suitable video data was available across multiple properties, allowing the organisation to evaluate performance across a representative range of operating conditions.
These characteristics created a strong foundation for practical Vision and Speech AI adoption.
3.2 Design
In the AI Transformation Process, the purpose of the Design stage is to define direction, establish governance and control and justify investment before delivery begins.
The Design stage identified that the key challenge was not whether the AI could detect parcel deliveries, but what level of performance would be sufficient for operational use.
The organisation agreed that deployment decisions would be evidence-based rather than assumption-based.
Testing criteria were defined, including:
Detection accuracy
False positive rates
False negative rates
Performance under varying lighting conditions
Performance under different weather conditions
Consistency across locations
User satisfaction
Operational usefulness
AI Governance and Assurance activities established acceptance criteria, testing controls, monitoring requirements and decision-making responsibilities.
The business case recognised that AI performance would never be perfect.
Instead, the objective was to determine whether performance was sufficient for the intended purpose.
This created a clear basis for evidence-based deployment decisions supported by testing, governance and agreed acceptance criteria.
3.3 Deliver
In the AI Transformation Process, the purpose of the Deliver stage is to deliver AI initiatives in a controlled way and embed them into business-as-usual operations.
Following Design stage approval of the AI Business Case, the pilot was delivered through structured AI Project Management.
The objective was to evaluate performance under real-world operating conditions before making wider deployment decisions.
The objective was not immediate deployment at scale. The objective was to establish whether the capability was sufficiently reliable for operational use.
4. Action Taken
During the Deliver stage, the organisation implemented a controlled pilot.
Video footage was collected across a representative sample of properties and operating conditions.
The AI system was tested against real-world events including:
Parcel deliveries
Visitor arrivals
Residents entering properties
Pets moving through camera views
Weather-related movement
Vehicle activity
Each event was manually reviewed and compared against the AI output.
This allowed the organisation to establish an evidence base for performance.
The testing programme demonstrated several important findings.
The system correctly identified most parcel deliveries.
However, performance varied depending on environmental conditions.
Accuracy reduced during darkness, heavy rain and where parcels were partially obscured.
The pilot also identified false positives, where the AI incorrectly classified events as deliveries.
Importantly, this did not mean the system had failed.
It demonstrated that AI outputs were probabilistic rather than deterministic.
The organisation therefore introduced confidence thresholds, review processes and operational controls to ensure outputs were used appropriately.
Deployment decisions were based on measured performance rather than marketing claims or assumptions.
5. Outcomes
All benefits were monitored and tracked in line with Orr Consulting’s AI Benefits Realisation approach.
5.1 Improved AI Understanding
Leaders developed a more realistic understanding of how AI systems operate.
The focus moved away from expecting perfection and toward understanding probabilities, confidence levels and operational suitability.
5.2 Evidence-Based Decisions
Deployment decisions were supported by measured performance data rather than assumptions about AI capability.
This improved governance and reduced implementation risk.
5.3 Improved User Experience
Users received timely notifications for genuine delivery events while maintaining appropriate expectations regarding occasional inaccuracies.
5.4 Better Governance
The organisation established clear criteria for testing, acceptance and ongoing monitoring.
This created stronger assurance arrangements and clearer accountability.
5.5 Adoption Confidence
Structured testing provided evidence that the system could operate effectively within defined boundaries.
This increased organisational confidence while maintaining realistic expectations.
5.6 Improved Fitness for Use
Perhaps the most important outcome was a shift in mindset.
The key question was not:
"Is the AI perfect?"
The key question became:
"Is the AI sufficiently reliable for this specific purpose?"
This principle applies across almost every AI capability area.
6. Final Thoughts
For Vision and Speech AI, the critical question is not whether the AI is perfect, but whether it is sufficiently reliable for the purpose for which it is being used.
AI systems operating in the physical world will rarely achieve perfect accuracy. The more important consideration is whether performance is sufficient to support the intended operational outcome within appropriate governance and assurance arrangements.
More broadly, successful AI transformation depends on evaluating AI systems through evidence, testing and operational fitness for use rather than assumptions about capability.
The most successful outcome was not the deployment of the technology itself. It was establishing a structured approach for assessing AI performance, governance and operational suitability before wider adoption decisions were made.
The same principle applies across the AI Universe of capabilities. AI systems should not be judged on whether they are perfect. They should be judged on whether they are sufficiently reliable, appropriately governed and demonstrably fit for purpose.
This AI Capability Case Study is part of the Orr Consulting AI Insights Library — structured thinking for AI transformation leaders and decision makers.
If your organisation is considering Vision and Speech AI and wants to understand how to assess AI systems safely, effectively and proportionately, we would be pleased to discuss your next AI steps.
Subscribe to Orr Consulting to receive occasional emails with practical AI Insights and updates.
