Inside a Cyber Attack

How do you react when all marina operations come to a complete stop—no warning, no practice drill, just a curt text message that says to stop using email because all, not only marina, but entire busy port operations, shut down? The marina staff at the Port of Seattle were faced with just that question last August, when they were hit with a cyberattack. Through shock, creativity and recovery, the lessons learned were invaluable as attacks like these become more commonplace.

For Shelby Allman, Giuseppe Alvarado and Simon Wright, their jobs at the port’s marinas, as they knew them, transformed with a text, warning all personnel to refrain from using any communication except text messages. The initial thinking was that the port communication systems were down.

“I had just returned from vacation and was still at home when I started reading texts. At that moment, we didn’t know if this was temporary, just 24 hours, or what caused the problem. There was no mention of cyberattack,” said Alvarado, business analyst for the marinas at the port—Shilshole, Bell Harbor, Fishermen’s Terminal, Salmon Bay and Harbor Island. They soon learned the entire port was impacted, and it started to become apparent this was not going to be a quick fix.

The Impacts
Shelby Allman, harbor operations for recreational boating at the port, explained there were levels of priorities. Despite maritime covering some 17 miles across the port, with marinas encompassing about a mile, restoring function to SeaTac airport was the first order of business.

What was formerly done via computers had to be done by hand. There were thousands of bags that had to be sorted with handwritten bag checks and receipts. Marina staff were reassigned to help check in passengers and their bags. Workers from all areas of the port were asked to volunteer to get these critical operations running again.

While some staff headed to help in different parts of the port, those who were still at the marina had a lot of problems to contend with. The systems and processes that were never before given a second thought were now in front of mind as nothing seemed to work as it should.

Wright, facilities manager at the marinas, said none of the key fobs worked. “That meant the gates weren’t working so we had to open them for customers as they showed up. Same with the restrooms. Our showers are also controlled by fobs, which was a big problem for our liveaboards. We had to do a physical plumbing bypass for each of our about 60 showers at around $1,000 each,” he said. The fobs also controlled the trash compactors, causing a backlog of trash.

Allman said payments could no longer be processed digitally. She had to teach younger staff how to manually swipe credit cards and handwrite receipts and checks. “Communication was hard. We had one phone for all marina customers to call for any information and questions. They wanted to know what was going on, and some wanted to just ask if we knew the Port had been cyber attacked,” Allman said. With so many calls, it became difficult to take moorage requests and then coordinate those requests across the properties.

The marina conference room was turned into a triage center where staff met regularly to share information and set priorities. The days turned into weeks, and more and more problems were identified. Allman said the whiteboard calendar in the room, which was initially used to cross off days they successfully completed, got too depressing to continue.

Hiring was put on hold, as was any procurement. “We couldn’t access vendor agreements or purchase orders, so over time, orders were forgotten,” Wright said. “I ordered 10 tons of asphalt, which still hasn’t arrived, and I have no idea what happened to it.” Allman added that she recently had 25 dock carts show up unexpectedly because she’d forgotten they were ordered more than eight months ago. Tracking systems and records were gone.

Alvarado said that employees were not even certain they’d be paid since payroll was shut down. “Everything went silent and I’m sitting home wondering what I could be doing,” he commented, adding that during this time customers still came to the marina, still wanted to use their boats and the marina amenities and still expected some level of customer service.

“Marina maintenance personnel were essential and continued to work as usual, but others throughout the port were sent home or reassigned,” Wright said. “The port tried to reassign to avoid people having to take leave without pay that would come from being unable to do their regular jobs.”

Recovery Begins
Although nothing was operating as it should, the marina had to continue to function on a day-to-day basis. Maintenance was still needed, customers still arrived and popular events still needed to happen.

“We didn’t have written records of our customers’ phone numbers. We didn’t even have staff phone numbers written, so we had a huge number of calls coming in,” Allman said. All those calls were coming to the one landline they were allowed to use, since cellphone communication was prohibited while the extent of the cyberattack was determined.

Phone trees were implemented to relay messages efficiently across all staff. To further streamline communications, the Washington Ports stepped in to become the temporary tenant alert system. They provided blanket updates that reduced the number of calls, allowing staff to focus on the more critical on-site needs of the marina.

“Fortunately, email came back quickly, well before the phones, so that gave us access to anything we’d been working on before the attack happened. Documents and information we had been sharing were back in our hands,” Alvarado said. He cited an instance where they had recently shared a customer list via email, so they were able to access that bit of information.

The port implemented its Continuity of Operations Procedures (COOP) plan, which is more intended for actions during and after natural disasters, but it helped as a map and timeline for activities. Having backups offsite also helped, but Wright said they learned many more records should have been backed up.

He emphasized that there is even a role for keeping paper copies since hackers can often break into off-site storage servers and use those as the point of entry to the main operating systems.

Lessons Learned
Plans, copies and secondary operating procedures are now in effect or underway at the marinas and the port. Alvarado cautioned that attacks like these will happen to everyone at some time and emphasized that it is critical for all marinas to plan now for the inevitable.

“Go through an exercise of this happening. Do you have a backup plan for just how to bill? How do you keep cash flowing? Is it possible to have a customer list with billing information kept with a third-party vendor who can send things on your behalf and have payments go directly to the bank? What would you do if all systems stopped? Could you text, email, do you know what you’re billing for,” Alvarado questioned. “These are things we’re starting to investigate, although right now we’re still recovering and not quite ready to talk in ‘what ifs’.”

The reports of exactly what happened are still coming out, but it appears hackers stole three terabytes of data and requested $6 million in ransom to have it returned. The Port chose not to pay and instead rebuilt all their systems from the hardware up and reprogrammed everything that touched the port’s server.

Local and state law enforcement, along with the FBI are still investigating the attack, but it appears the attack came through the port’s internal server with the hackers likely having had access for some time before they actually took over the data. When and how they got in remains unknown.