Conquering Notpetya Conquering NotPetya: Two Weeks On The Front Line

Recently the world has got to know the details of the NotPetya attack suffered by A. P. Moller – Maersk in June last year. The company chairman has recalled the events, listing the losses incurred by the company, but also praising human resilience which allowed it to recover relatively quickly from such a damaging attack.

Among the teams bringing Maersk’s systems back online were also several Predicans – and here is their story.

Discovering the problem

It was the end of June, a nice, summer day. The morning started out like any other. Two of our Predicans, Service Manager Fruzsina Zacs and Predica CEO Paweł Szczecki, were on site at the Maersk UK office for vendor meetings.

The planned meeting began and discussions about the new Managed Service operations had just commenced, when a large commotion spread across the office. Suddenly, the monitoring screens started to show all systems turning to red, and not long after all laptops started to reboot.

Meanwhile in Poland, the team had arrived at the office, made coffee, switched on their laptops and were checking for project updates or verifying the systems like they would on a normal day. But for some of them, something was different.

The team working on the Managed Service engagement with Maersk suddenly received a call from their Service Manager Fru. She stated that all connections to the company are to be terminated immediately. They asked how they were expected to perform their duties that day. The response was: “You are not expected to. Not today.” They moved on to other things.

High alert

At that point, nobody knew what had actually happened. The first assumption was that something went wrong with the monitoring systems. However, once the laptops had started to reboot, the suspicions of ransomware were raised.

The extent of the problem was still unknown. Our colleagues on site were told that the meeting would be postponed to the next day but there was nothing they could do to help and they should return to their hotels.

The next morning, as they made their way back to the client office, it was evident that the issue had not been resolved. As they entered, they noticed several war rooms had been established and the vendor’s Identity teams were in planning.

After getting a very quick update on what was known at the time, the team in Poland was once again informed that they should keep their laptops disconnected from the Maersk network. At the same time, a quick decision was made to bring in our first experts ASAP.

That same afternoon our Cloud Identity and Active Directory experts landed in Heathrow. In the evening, the news finally broke: “Maersk is down.”

Sending reinforcements

It was soon starting to become apparent that the situation was extreme. As our Team Lead Tomasz Gościmiński recalls:

“In the morning I emailed Tomek who was our AD expert on site, asking him what was going on. At 5 pm I finally got a reply: ‘Tom, in a minute!’”

Once we finally had a clearer picture of the severity of the situation, we’ve sent in more team members to help. In the end, they were around for four weeks on rotating shifts, helping and advising the various IT and support teams to bring the systems back to life.

Chairman Jim Hagemann Snabe said at the World Economic Forum that the “complete infrastructure” of the company has been shut down by ransomware. For 10 days, Maersk had to switch to manual operations to manage their ships which were docking and unloading their cargo worldwide at all hours. Employees, senior management, partners and suppliers all worked around the clock to reinstate Maersk’s systems.

Powering through, powering up

Our team was present on the premises for four weeks to recover and stabilize the core infrastructure services. They also ensured a quick turnaround for requests during the reinstatement of critical business systems. The primary and key objective was to reestablish the Active Directory for identity management. This was necessary to reenable resource access which was the first step towards switching back to automated operations.

At the same time, we have enabled login to Office 365 via a web portal and upgraded the authentication solution. This way, while the company hardware was still unavailable, employees could communicate using their mobile devices or alternative machines.

While on site, we have also implemented cloud solutions which enhanced company’s IT security. These services will protect the company from damage to critical systems in case of any future attacks.

Our team was also there to provide general support to Maersk’s employees while introducing the new services. This helped to make the adoption process as easy as possible under the circumstances and guide users through any required processes.

The aftermath

After 2 weeks of very intense focus, round-the-clock shifts and gallons of coffee, the core systems were operational and stable once again. Maersk employees were going back to their stations and returning to their regular duties. The majority of our team was able to leave the UK and come back to our Warsaw office.

But the work on Maersk’s infrastructure is by no means over. The introduction of new systems means that there are still challenges with adoption which need to be addressed. The disruption also caused long-term effects on the systems which are continuously being resolved as they arise.

The NotPetya attack on Maersk has been one of the most challenging projects we have had to contend with. However, our team, together with Maersk’s IT department and other partners, has worked tirelessly to overcome the difficulties, minimizing the losses and bringing the company back online as quickly as possible.

It took time to fully recover all systems, and new challenges still appear, but even so, the infrastructure is now stronger than ever – as are all the people who worked to restore it. It was the experience of a lifetime!

Securing the systems for the future

Our Digital Advisor and Board Member Andrzej Lipka added some final remarks on securing organizations against attacks such as this.

We were able to react really fast to the difficult situation at Maersk because we had a managed service agreement (which started just a few months prior) and acted immediately. Right now, we are doing a lot of work on fixing the issues arising in the aftermath of the incident.

However, we are also optimizing and preparing the infrastructure in the event of future incidents of such a scale, e.g. preparing and test running DR strategies, designing regular security reviews and ‘penetration tests’ of the infrastructure we support. We are also deploying new cloud technologies to help alleviate or at least minimize the risk of these events happening again.

To get more advice on defending your organization from ransomware, read our previous post on the subject. If you want to be sure that your security strategy is up to scratch, or would like to know more about our managed services offering – contact us now.