“Malfunctioning clock” sparked JT outage

Wednesday 29 July 2020

A “malfunctioning clock” was the root cause of a six-hour JT network blackout that caused difficulties with 999 calls, a new report by the operator has said.

The telecoms provider’s network went down for a total of six hours on Sunday 12 July, during which time customers were unable to make calls, and suffered difficulties with broadband and internet services.

999 calls were also unable to be made via landline, but remained possible on mobile devices.

Having investigated the cause of the major incident, JT released its findings in a report yesterday – just as Jersey’s competition watchdog launched an independent probe into the technical issue, as well as governance and security at the telecoms company.

Pictured: 999 calls were not possible from landlines during this time.

JT said that the failure was triggered by the malfunctioning of a clock connected to the network’s 100 IP routers.

At 18:55, one generated a wrong date (27/11/2000), which sparked what JT’s Chief Technology and Information Officer Thierry Berthouloux described as a “sequence of events that was almost impossible to foresee”, causing the outage.

Mr Berthouloux said JT investigated whether this could have been the result of a malicious activity, but had ruled out these possibilities.

Instead, the likely cause was identified as a hardware failure that caused the clock to return to factory settings. 

With mobile and internet communications hit by the outage, JT lost access to its corporate arm, making it difficult to keep customers informed.

It said that the need for “more robust customer communications during service incidents” was one of “many learnings” following the outage.

Pictured: Broadband services were affected in the outage.

Speaking yesterday, CEO Graeme Millar described the failure as “the worst in JT’s 132-year history”, saying that the network had already “taken the steps needed to prevent any repetition”.

He added that the firm’s CFO will be conducting a review of the effectiveness of these steps.

However, due to the impact on 999 services, the competition authority has nonetheless decided to launch its own, independent review into what happened and has appointed telecoms experts Cognitio Consultants to conduct it.

At the time of the outage, the Jersey Competition Regulatory Authority (JCRA) had already been completing a separate investigation into system failures with the 999 service provided by JT and Sure earlier this year.

However, the six-hour outage in early July led the authority to decide that a more in-depth probe would be needed.

Pictured: The Home Affairs Minister and JCRA's CEO.

Responding to questions from Express yesterday, Home Affairs Minister Constable Len Norman explained that the previous failures had involved some 999 calls being dropped. 

In those instances, the Emergency Services were able to “trace and make contact” with callers, he explained.

However, the Minister added: “…In the last major outage that was not possible, so we don’t know who made calls from landlines and was unable to get to the 999 service. We have no reason to suspect that there was any major problem because of that, but we can’t be 100% sure because we can’t trace the calls. 

“That is why this investigation is so so important.”

Cognitio’s investigation is expected to last between two and four weeks.

Pictured: The investigation is believed to cost around £120,000.

Beyond the impact on Emergency Services, JCRA CEO Tim Ringsdore said that the outage also had implications for the local economy and Jersey’s connectivity.

The cost of the independent investigation – which will be covered by JT - is understood to be in the region of £120,000.  

Mr Ringsdore would not confirm this when asked, saying that the details of the contract were commercially sensitive. 

At the end of the review, Cognitio will make a number of findings and recommendations. If the findings are found to have violated JT’s operating licence, Mr Ringsdore warned that the authority had the ability to order a fine of up to 10% of the company’s revenue.

Pictured: JCRA's new Chair, Stephanie Liston.

It’s the first major probe to be launched by the newly-independent JCRA, which officially split from its Guernsey counterpart earlier this month.

The board is still yet to be complete, with two more non-executive directors to be appointed.

Mr Ringsdore said he did not expect the recent split to pose a challenge to the local competition authority in conducting its JT review. 

“Our core activities haven’t been affected and I’m confident we can cope with this type of investigation. We work with experts in this area,” he commented. 

JCRA’s new Chair Stephanie Liston - a telecoms expert who arrived the day after the outage - said of the review: “The Board has reviewed the initial report provided to us by JT and considered the recommendations of our officers and legal advice. Working closely with the Government of Jersey, we have concluded that this investigation is critically important to the citizens and consumers of Jersey. The resilience of Jersey's communication services nationally and internationally is fundamental to our safety and economic growth."

Constable Len Norman added: "I appreciate that JT has assured States Members it will conduct an internal review but I believe it is also essential to give the public confidence that an independent, transparent investigation is carried out by the authority to establish what went wrong, why and what actions are necessary to prevent a recurrence.

“I fully support the authority in its investigation, which is in the best interests of the island’s government, economy, citizens and consumers.”