»The manufacturer's backdoor in Alcatel Lucent devices caused huge losses for many operators. Someone used it to "brick" tens of thousands of devices - Niebezpiecznik.pl -
The attack took place on October 16, 2017 at dawn Polish time, simultaneously on GPON access terminals in several different countries. Were used, among others logins and passwords embedded in the firmware of subscriber devices [provided by Alcatel-Lucent. Someone read the access data to the administration account embedded in the firmware and logged in to thousands of devices using this data, physically corrupting them].
This article was written by Piotr Marciniak and was originally published on the pages of telko.in. We reprint it with the consent of Łukasz Dec, the editor-in-chief of Telko.in. We provide additional information on this attack at the end of the article.FROM THE AUTHOR: This text is the result of my fatigue and helplessness after six months of correspondence and meetings with Nokia representatives and passing new deadlines for delivering a defect-free product. The text is also a description of the problem that every operator can face on any equipment, which - in my opinion - we have no chance to independently detect and 100 percent. secure. Finally, I would like this article to be a voice in the discussion about the responsibility of hardware and software manufacturers for their products.
Domino effect
On October 16, 2017, after 6:10 a.m. OLT devices (optical line termination) in our network logged the first in a series of alarms for loss of communication with ONT (optical network termination). Everything was happening very quickly and it looked like a domino effect. There were several dozen terminals per minute. In fact, there was no time to react, because - as the subsequent analyzes of various types of logs showed - attacks on selected networks were prepared earlier, and they were carried out, inter alia, login and password saved in one of the ONT software modules developed by a third party, other than the terminal provider, which in our case is Alcatel-Lucent, which was acquired by Nokia in 2016.
The preparations of the attackers were wider and thanks to them, at the time of the attack, there was no need to hack the devices or scan the IP of vulnerable devices. This process has been done before. At a critical moment, it was enough to run the scripts.
The scale of the damage
Unauthorized access to the ONTs (and thus, for example, a reset of network settings) would be a nuisance, but manageable event based on the operator's own resources. The attacker, however, led to the destruction of the contents of the flash chips, preventing the devices from starting in any mode that would allow even new software to be loaded. The failure affected over a dozen percent of services provided based on the ONT model vulnerable to the attack. Phones rang - subscribers demanded an immediate solution to the problem. It became necessary to physically replace hundreds of damaged customer tips with new ones, while our own stocks did not provide such a massive replacement.
In this critical situation, it was not possible to obtain additional devices for replacement from the manufacturer. Regardless of whether we consider it an urgent service intervention due to the fault of the manufacturer or a priority purchase order. There was no equipment from the 240 series, or any replacements in Nokia's warehouses. Only after a few weeks, we managed to buy only 130 new devices. Of course they helped, but they arrived late and in too little quantity.
Therefore, the action of restoring services was based on a patchwork of various solutions. First of all, we replaced devices for triple-play services based on our own supplies. In other cases, we bought and implemented (fortunately tested earlier) competitive 1-port terminals available on the spot, adding to them, where necessary, additional RG (residential gateway). Some subscribers decided to buy routers on their own.
Unfortunately, due to the scale, it took several weeks to replace the devices. The problem was not only the lack of Nokia equipment, but also the number of devices that each service team could replace each day. Of course, expenses on equipment were followed by discounts for customers for lack of services and, unfortunately, a certain number of resignations from people who did not want or could not wait for the replacement.
Attack mechanism
As already mentioned, the logins and passwords embedded in the terminal software were used. Sometimes it is introduced by the manufacturer himself, explaining that it is a "service password" in case the customer forgets the access data. I have always believed that in such cases there should be a password recovery procedure rather than a backdoor. The problem is, in fact, that it is impossible to protect this type of logins and passwords from unauthorized persons.
Such a login gate is usually completely invisible to the device administrator. Without knowing about it, it's hard to tell that such a hidden account exists. At the same time, it cannot be deleted or its password changed. The total blockade of remote terminal management on superior firewalls means cutting off or limiting the administration of the equipment. In the discussed case, the ONTs - if they are routers - have a public IP, which is usually associated with the policy of the least possible restrictions on port blocking. At the same time, there is no rational argument to place this type of backdoor in home terminals for service reasons. After all, the "soap dishes" have a reset button. And the default login and password are printed on the box or sticker.
Disassembled Alcatel-Lucent ONT I-240W-A terminals waiting for replacement of damaged flash memory modules. (Source: TPnets.com)
In the situation described, it seems otherwise. According to our findings, this particular backdoor was not introduced by the producers, but by the software vendor. It turns out that terminal software is often not a proprietary vendor production, but a cluster of modules purchased from various sources. The key question is: does the device manufacturer control purchases and are responsible for what they sell? I have no doubts that the producer should be responsible, and his own claims for damages to customers can and should be directed to his suppliers. He should not shirk responsibility for sold and dangerous trash. When we look at, for example, the automotive industry, we will notice service calls from time to time to replace defective airbags or brakes. Items often produced by third parties.
It would also be difficult to assume that when, for example, we install anti-burglary doors, and their manufacturer provides keys to them to anyone interested, the costs of "burglary" should be borne by the buyer of the door. The producer's responsibility cannot be made dependent on whether the buyer has invested in wire mesh and hired a security guard. Nor to argue that the lock on the door is just an ornament. Unfortunately, we meet such arguments when we hear that we can block access to our ONTs on other devices in the network.
What does the producer say?
In the case of many network devices, it is not possible to choose software other than the manufacturer or to check whether newer versions have a vulnerability identified by us. These are closed platforms. Hence, in the case of GPON hardware under the Alcatel-Lucent brand, it is necessary to provide the corrected software by Nokia.
As I mentioned, due to the scale, we informed the producer about the incident on the same afternoon. We assumed that the problem might also apply to other networks (we received confirmation later). Within two days, the manufacturer received two damaged ONT terminals from our network. Our diagnosis of flash memory corruption has been confirmed. In the first days, it seemed that we would work out software solutions (patch) and set the rules for replacing defective hardware. Unfortunately, the process slowed down quickly.
The manufacturer assumed that since we do not have an active service contract, it will not help, regardless of the nature of the event and our inability to purchase such a contract in the past. He did not feel obliged to secure even the possibility of purchasing the necessary number of new terminals. At the same time, it was suggested that since we have a problem with the old version of the software, then ... we can buy a new one at a price described by a 6-digit number. Importantly, when asked if the new version is free from the documented defect, we heard that there is no such guarantee. We were denied the possibility of testing the latest version of the software for security.
The piquancy of the whole situation is added by the fact that, as a result of our search, we have established that videos showing how to hack 240 series terminals were published on YouTube in June 2016. We shared this knowledge with the producer, but to this day we have not received any formal statement in this regard. The only reaction was another offer to sell solutions with greater discounts in the cost of the software, while requiring the purchase of Gold level support.
A soldering station used in the process of removing and surface mounting damaged memory modules. (Source.TPnets.com)
We provided a lot of data (device logs and links to materials). If anyone has analyzed them, the results of this work have not been received by any known users of problematic devices of these devices. Neither are we.
Own corrective actions and costs
Unable to replace the damaged terminals, or even buy new ones instead, we asked the manufacturer for support in a possible repair. Unfortunately, we got nothing from Nokia again. Even the technical documentation of the devices.
At this point, a deep tribute to Marcin Kuczera (Leon sp. Z oo), who carried out fault diagnostics in his company, and then determined the source of the problem and the method of repair.
As a result of his work, we developed a procedure for removing, programming and surface mounting of new flash chips. Removable memories were often not only overwritten with random data, but also contained physically damaged cells that prevented reprogramming and reassembly. It was necessary to organize new memory chips, programmers and a soldering station. In the absence of a production line, manual repair is a tedious and therefore costly process that has so far consumed hundreds of man-hours. We had no choice. Some devices are still waiting for repair.
It also turned out that the AMS system installed on us is also leaky. We were forced to turn off the server with this system (we can buy new software that may work better ...).
The total costs of losses incurred by us include damaged and purchased equipment, discounts and subscriber cancellations, hundreds of man-hours, travel costs, etc. Additionally, what our Customer Service Office experienced by some customers was extremely toxic.
The key question is: what is the process of supervision over software branded by large hardware vendors? We have not regained some of the system functionality to this day. For now, locks are in effect on parent devices. But we don't know if there are other flaws in the software. To date, Nokia has not sent us any communication or technical bulletin on this matter. She also did not release the update.
Who is such an attack for?
Looking at the international scale and type of the event, there was a thesis that I quote: "it looks like tests of the vulnerability of telecommunications networks of neighboring countries." This is evidenced by both the fact of a parallel attack in several countries on various platforms and the use of a vulnerability embedded in one of the software modules without the vendors' knowledge. From the perspective of our considerations and network security, however, the key question is what is the process of supervision over software developed by large hardware vendors? In this case, it turned out to be ineffective.
Over the next few weeks, we recreated the course of events in cooperation with other attacked networks and friendly experts. All materials and analyzes were provided on an ongoing basis to device manufacturers (at least two), which - to our knowledge - were then successfully attacked. The first of the producers took actions quite efficiently, including: sending their customers e-mails describing the problem, suggestions for ad-hoc actions, and finally releasing relevant software updates. We were not so lucky.
End
The text was not written until six months after the events described. First, we saved the situation, then we resuscitated the equipment and had long, fruitless talks with the manufacturer. I do not hide that we have been waiting for his effective and adequate help all this time.
At this point, I would like to thank all the people who shared with us their knowledge, logos and information about the events described above. Special thanks go to the experts and institutions who devoted their often private time to analyzing the materials we collect.
I would also like to thank the few employees of the old Alcatel-Lucent Polska team who - despite the lack of corporate commitment - helped us as much as they could.
Comment from the editorial office of Niebezpiecznik.pl
The incident described by Piotr Marciniak has been known to us since October in the context of another ONT device manufacturer, ZTE. ZTE devices were hit by the problem on October 14. Tomasz Brol, CTO at Syrion, gave a great presentation about this incident at the PLNOG conference.
Key information? This is not a hybrid war, but an attack with the "Brickerbot" botnet - its author has published his thoughts here (worth reading). In turn, courtesy of Tomasz, we can provide you with the entire presentation on this incident.
And here we could end this comment under the reprinted article and let you already mess with the stupidity of leaving service passwords in the firmware. But there is one more thing related to this incident that needs to be presented for a complete picture. This is the position of the manufacturer, Alcatel-Lucent. There is no official, but the post about this article on the Telko.in fanpage has Ms Wioleta Jędrzejewska, who works at Alcatel-Lucent and shares information about this incident from a different perspective than Piotr Marciniak.
Ms Wioleta confirms what Piotr Marciniak himself admitted in the article. His company did not purchase technical support:
But should such support be needed to remove the "defect" generated by the manufacturer himself, without the fault of the user or the operator? Andrzej Karpiński from Orange summarizes it accurately:
However, the manufacturer has a different view on this issue. Here is Mrs. Wiola's reply:
It is not for us to judge whether this is what an employee of the manufacturer should say in a public forum. So, finally, let's give the floor to an employee of another operator, who summed it up like this:
Either way, the dialogue with Alcatel should be happy. We know very well how difficult it is to get any information from this company in the press - we tried while reporting holes in Alcatel Link OneTouch devices to Alcatel, see.
A lesson and a dilemma
First of all, the incident described by Piotr Marciniak and Tomasz Brol is a great example of the fact that each device must be hardened. Cut everything you can and not connect with interfaces in the network / VLANs to which the device should not have access (with one of the operators only one ONT was exposed to the public internet. The rest was not. But this one was connected with the rest of the management network ... ).
Second: Should the manufacturer be responsible for bugs (holes) in the software? If so, how long? And only for customers who have purchased "support"?
PS. What happened to the ONTs may soon happen to Mikrotiki. A botnet is currently prowling, targeting unhardened routers of this manufacturer. Patch them before it's too late - see the description of the attack and recommended steps here.
Update 26/04/2018, 14:31 We were contacted by Ms. Wioleta Jędrzejewska, who informed us that:
In line with the above, we have detailed the sentence that could suggest that Ms. Wiola's statement is an unofficial position of Alcatel-Lucent. It is not, it is the private opinion of Mrs. Wiola. At the same time, we asked Alcatel-Lucent to comment on this matter. When we get a reply, we'll update this article.
Update 04/27/2018, 12:14 Here is the statement of Nokia's press spokesman:
