Two router replacements overloaded the network, causing nationwide communications failure

Questions mount after outage struck during upgrade from copper to fiber-optic lines

Passenger jets wait at Chicago O'Hare International Airport. [AFP]
Passenger jets wait at Chicago O'Hare International Airport. [AFP]

The communications failure that halted flights across the United States on Friday (local time) originated from two newly installed routers, the Wall Street Journal reported Saturday, citing government and industry officials.

Defense contractor L3Harris installed the new routers at a Federal Aviation Administration communications facility in El Dorado, Arkansas, on Friday. Immediately after installation, the equipment began flooding the FAA's entire communications network with large amounts of data. The same phenomenon appeared at a radar facility in Amarillo, Texas.

Officials said the two routers sent 1,000 connection signals to the entire network every four to five seconds. The resulting overload caused communications failures at 17 of the FAA's 22 national air route traffic control centers — the direct trigger for the nationwide ground stop the FAA issued Friday afternoon.

The system stabilized only after the FAA disconnected the two routers from the network. Reuters also reported, citing the FAA safety employees' union and other sources, that the problem was confined to the two routers updated that day.

The cause has not yet been determined. FAA officials have identified a feature in the routers they believe caused the data overload and have since disabled it on all Juniper routers in the FAA's air traffic control network. The agency is investigating whether the problem stemmed from a hardware defect or a configuration error by the contractor.

The routers in question are products of network equipment maker Juniper Networks, which Hewlett Packard Enterprise acquired last year. About 60 Juniper routers are currently operating normally in the FAA's air traffic control network. An HPE spokesperson told the Wall Street Journal that the equipment appeared to have functioned as intended, confirming "it did not malfunction."

Transportation Secretary Sean Duffy told reporters at a news conference that "what we're trying to figure out is whether it was incompetence on the part of the vendor or whether it was a faulty router," adding that the cause of the defect had not yet been identified.


andy@heraldcorp.com