What to Measure in the First 90 Days of IoT and How to Review Pilot Value
DBR77 Team · Published
In the first 90 days of IoT, measure response time to stops and the share of downtime with no reason first, and OEE last. Response time is under your control from the day the floor becomes visible, while OEE is an outcome that follows weeks later.
In week three OEE has usually not moved yet, so a pilot judged on OEE looks like a failure. Response time and reason quality show within days whether the plant uses the data. This article gives a 90-day measurement plan, a pilot review format and the questions to ask at six months.
Why response time moves before OEE
On one corrugated-packaging line, where the plant has confirmed the numbers, average response time to a stop fell from 24 to 9 minutes before OEE began to rise (full case).
OEE multiplies availability, performance and quality. Vorne notes that most manufacturers are closer to 60% OEE than to the 85% often called world class. Moving that number takes time, because it depends on many losses at once.
Response time depends mainly on how fast the right person learns about a stop. Once every stop appears live and goes to an owner, the minutes start to fall in the first week. Vorne notes that most companies do not accurately track minor stops well under five minutes. The first weeks of live data make these stops visible, and fixing them is what moves OEE later.
Agree the decision contract before the pilot starts
Before the first device is installed, agree a written decision contract with the line owner. It covers:
- Five metrics that matter on this line, for example response time, share of "unknown" downtime, availability, output per shift and OEE.
- One owner for each metric, by name or role, on the shift where it is measured.
- A threshold that triggers action, such as "response time above 15 minutes for two days in a row means a review with maintenance".
- The success criteria for the pilot, written down and signed by the sponsor.
An illustrative example of success criteria: response time below 10 minutes and "unknown" below 10% of downtime by week eight. Use your own baseline to set the numbers. Our first 30 days article shows how to record that baseline in week one.
Keep definitions stable. ISO 22400-2 gives each manufacturing KPI a formula, unit and time behaviour. Every line should calculate OEE the same way, or later comparisons will not hold.
The 30/60/90-day measurement plan
The first month checks that the data is complete, the second that people act on it, and the third that the result is worth money. The targets below are illustrative; set yours from the baseline.
| Day range | Metric | Target example | Who owns it |
|---|---|---|---|
| 1–30 | Machines reporting live status | All machines in scope on the pilot line | Project lead with IT/OT |
| 1–30 | Stops with a reason captured at the machine | Above 80% of stops | Line owner |
| 31–60 | Average response time to a stop | Below 10 minutes | Maintenance lead |
| 31–60 | Share of "unknown" downtime | Below 10% | Shift leaders |
| 31–60 | Alerts acknowledged within 5 minutes | Above 90% | Shift leaders |
| 61–90 | Availability and OEE vs baseline | Clear upward trend for 4 weeks | Plant manager |
| 61–90 | Value of the gain in money | Agreed with finance | Project sponsor |
Do not add more metrics in the middle of the pilot. If a new question comes up, note it for the scaling phase.
Know what one OEE point is worth before the pilot starts
The ROI calculator turns your shifts, rates and margins into a value per OEE point, so the day-90 review ends with a number finance accepts.
How to run the pilot review meeting
Hold a short review every two weeks and a full review at day 90. The full review takes about an hour. Invite the sponsor, the line owner, the maintenance lead, one shift leader and someone from finance. Use the same agenda each time:
- Baseline vs today for the five contract metrics, on one page.
- Top three losses by minutes, and what was done about each.
- Alerts that were ignored or escalated, and why.
- What operators and shift leaders say, collected before the meeting.
- Value in money, using the method agreed with finance at the start.
- The decision: go, extend or stop, with a date for the next step.
Give real weight to what the shift leader reports, because when the floor finds the alerts useful, the numbers usually improve as well.
The go/no-go decision to scale
McKinsey describes how many companies get stuck in "pilot purgatory", running pilots that never scale. The cure is a decision date written into the contract. At day 90 there are three outcomes:
- Go. Success criteria met. Approve the next lines and a rollout plan. See how to go from one successful pilot to a plant standard.
- Extend. Data is complete but behaviour has not changed yet, for example reasons are captured but response time is flat. Extend by four to six weeks with one specific fix.
- Stop. Machines are connected but nobody owns the response, and the sponsor cannot change that. Stopping is cheaper than a pilot that runs for a year.
A go decision needs a measured gain against the baseline and a team that can repeat the setup. The numbers do not have to be perfect.
What to review after six months
At six months the question is whether the gain holds and spreads. Review four things:
- Drift. Has response time crept back up? Are reasons slipping back to "other"?
- Reason lists. Remove reasons nobody uses and split the ones that hide two different problems.
- Alert load. Count alerts per owner per shift. Rising numbers often mean thresholds need tuning.
- Value. Compare OEE and downtime minutes with the original business case, in money.
Deloitte's smart factory study found that leading adopters reported about twice the labor productivity gains of other groups. A framework such as the Smart Industry Readiness Index helps decide what to add next.
FAQ
What is the single most important metric in the first 90 days?
Response time from a stop to the first action. It is controllable from the first week and shows whether people use the data. OEE confirms the result later.
When should OEE start to improve?
Usually in the second half of the pilot, once reasons are captured consistently and the top losses are being worked on. By then response time should already have fallen.
Who should own the pilot review?
The project sponsor chairs it, but the line owner presents the numbers. If IT or the vendor presents them, production will treat the pilot as someone else's project.
What if the pilot misses its targets?
Find out which part failed: the data, the response or the value. Data gaps are fixed with hardware, slow response with owners and escalation rules. Only a missing value case is a reason to stop.
Conclusion
Most of the setup is done before day one: the contract metrics with named owners, the success criteria signed by the sponsor and the date of the day-90 decision. From then on, the 30/60/90 table tells each owner what to watch and when. One item to settle with finance before the first machine is connected is how one OEE point will be valued, since the day-90 review ends on that number.
Plan a pilot with the success criteria written first
We agree the baseline, the five metrics, the owners and the day-90 decision with you before the first machine is connected.
Sources
- Vorne, World-Class OEE, 2026
- Vorne, The Six Big Losses, 2026
- ISO, ISO 22400-2: Key performance indicators for manufacturing operations management, 2014
- McKinsey & Company, IoT value set to accelerate through 2030, 2021
- Deloitte Insights, Driving value from smart factory technologies, 2019
- DBR77, Smart Industry Readiness Index (SIRI), 2026
- DBR77 IIoT pilot data, corrugated-packaging line (client anonymous), 2026