Ransomeware

Digital art by Tamara Rees
Sources: Adobe Stock/corund/nikolam, Shutterstock/Gaidamashchuk/LiquidLayout

Imagine treating ailing animals without information about their vaccinations, medications, allergies, recent test results, previous surgeries, or behavioral issues, such as a history of biting. It's the stuff of veterinarians' nightmares. For practitioners at some hospitals owned by National Veterinary Associates (NVA) recently, it wasn't just a bad dream.

On Oct. 25, an undivulged number of NVA's more than 670 general practice, emergency and specialty hospitals were hit by a malware attack. Malicious code, presumably delivered when an NVA employee opened an infected link or file in an email, spread through the company's network, barring veterinarians and support staff from the medical records they rely on to treat animals and communicate with clients. In instances described to the VIN News Service, the records lockout lasted from a couple days to nearly two weeks, followed by another month or more of limited access.

A cybersecurity blogger described the event as a ransomware attack, which means the attackers managed to encrypt records on the company's practice information management software and demanded payment to decrypt the files to restore access.

Several veterinarians at affected hospitals posted about the attack anonymously on the Veterinary Information Network, an online community for the profession and parent of VIN News. They described their experience on the front line of a malware attack as frustrating, stressful and overwhelming.

"We have a lot of clients angry about the slow down, and the staff is under extreme stress," one wrote in mid-November. "I'm not sure how we're going to come out the other side."

NVA has released very little information to the public about the attack, and the company did not respond to specific questions from VIN News for nearly two months. When told about the anonymous veterinarians’ experiences, the company arranged, and a representative sat in on, telephone interviews with two managing veterinarians and a practice manager. In addition, NVA provided the following statement on January 9, which reads in part:

"In late October, some of NVA hospitals were hit by a malware attack. Since every hospital is set up differently, the impact of the malware attack varied widely by hospital. Despite it all, the hospital teams and NVA support teams worked tremendously hard to help hospitals in their primary goal — continuing care of patients and clients. NVA is incredibly proud of our hospital teams and their care of their patients and clients.

"Today our hospitals' servers and networks are recovered."

Founded in 1996 by Dr. Stan Creighton with the stated aim of keeping veterinarians in charge of their own practices and now owned by a private equity group, NVA has approximately 670 hospitals in the United States, Canada, Australia and New Zealand. The company reportedly employs 2,500 veterinarians. Last year, JAB Investors based in Luxembourg signed an agreement to purchase NVA. JAB owns Compassion-First Pet Hospitals, along with food brands Keurig Dr. Pepper, Panera Bread, Krispy Kreme Doughnuts, Peet's Coffee & Tea, among others.

The malware attack involved the practice information management software AVImark, according to NVA veterinarians.

However, the company that owns AVImark, Covetrus, told VIN News by email that the problem was not at their end. "This is not an AVImark issue nor does it have anything to do with Covetrus systems."

Locked out

Word of the records shutdown leaked in early November on message boards on Reddit and VIN. The first news report was published Nov. 19 on a cybersecurity blog written by former Washington Post reporter Brian Krebs. His report is based on a source "close to" security firms brought in by NVA to remediate the attack.

Krebs reported that the ransomware attack affected approximately 400 locations. He also wrote that NVA spokesperson Laura Koester confirmed ransomware, declined to answer questions about the attack or whether a ransom had been paid, and said access to all the patient records had been fully restored.

"For a few days, some [pet owners] couldn't do online bookings, and some hospitals had to look at different records for their patients," he quotes Koester as saying. "But throughout this whole thing, if there was a sick animal, we saw them. No one closed their doors."

Two veterinarians at different NVA hospitals provided a less sanguine description of the experience. By their accounts, while their clinics stayed open, working conditions were extremely challenging. Initially, they had no access to records and no clear plan for getting back online. The challenges lasted for weeks, not days, according to the veterinarians.

They posted anonymously about their respective experiences on the VIN message board, and they answered questions from VIN News by email on condition of not being identified. Both said they had been instructed by NVA not to speak publicly about the incident, and feared doing so would cost their jobs. But as one explained, frustration and anger about what the employee saw as an insensitive and inadequate response to the attack led the veterinarian to speak out.

In the midst of the event, in November, one of the veterinarians, located in the West, wrote on VIN, "While it's true we haven't turned anyone away (thank god it's the slow season), we are working twice as hard to get half as much done."

The veterinarian expressed annoyance at NVA’s boast about not closing a single practice. "I, for one, think that closing for a few days at the onset might have been the right thing to do," the clinician told VIN News. "And we absolutely did refuse to see a number of clients once we realized that we couldn’t process appointments faster than one per hour per doctor. Additional pets that needed to be seen were sent elsewhere, more often than not without any of their medical records."

The second veterinarian, located in the Midwest, told VIN News that doctors, having no access to patient histories, were apprehensive about performing operations but did not cancel procedures because "management required us to go ahead with surgeries, etc." The practitioners could, at least, obtain some patient information through in-house blood analyzers, the veterinarian noted.

The attempt to continue business as usual was stymied in other ways, though. The veterinarian said, "We did have a lot of no-shows due to lack of reminder calls."

It's unclear whether a directive to continue with surgeries, regardless, applied throughout the chain. VIN News heard of but could not confirm a case of a client who was upset about a cancelled surgery at an NVA hospital.

NVA alluded to variability in management and care decisions in its Jan. 9 written statement to VIN News:

"NVA is a community of locally managed animal hospitals. Every hospital has their own unique operation and is led by their on-site Managing Veterinarian and their team. All patient decisions are exclusively under the purview of the local veterinary team. NVA believes veterinary medicine is best practiced when local veterinarians craft their own approach. The NVA community is united by our love of animals and the people who love them."

An alternative view from the trenches

Employees supplied by NVA for interviews with VIN News described the malware attack and recovery as stressful at times but not unmanageable. They said there were a few "traffic jams" in the early days but that no surgeries or treatments were postponed and the practice of medicine was never compromised by the attack.

All three said NVA kept them informed, including providing regular updates and timelines for reestablishing functionality. They said NVA also provided online resources and moral support through the incident.

"Anytime we needed anything, they were on the phone," said Dr. James Gilchrist, who runs a 10-doctor NVA hospital in Waterville, New York. The biggest challenge, he said, was probably "writer's cramp," from having to keep records by hand at the high-volume practice.

Kyra McCormack, a practice manager in Independence, Missouri, seconded Gilchrist's praise of NVA. She and Gilchrist said the experience of working through the malware attack was "bonding" and "team-building" at their hospitals. 

Since each NVA hospital has different computer capabilities and software, and access to different client and patient data, strategies for collecting information and tracking patient visits varied to some extent by location. To varying degrees, the practices said they relied on Google Docs, Excel, Word, and paper forms. In some cases, software not affected by the malware contained important patient and client information.

Dr. John Paulson, who runs an NVA hospital in Shoreline, just north of Seattle, was able to retrieve some patient information from a reminder service that had recently downloaded client and patient data from his AVImark database but was unaffected by the malware. 

He said NVA provided his practice with a single computer that could access records from their most recent backup within a couple days. The anonymous veterinarians reported that their wait was closer to two weeks.

By that point, it was hard for them to feel grateful. "One of the things that upset me the most," one said, "was the delay from the initial ‘crash' to the single laptop arriving with AVImark and patient data. That was a very long 12 days of practicing blindly." The veterinarian said doctors had to ask clients questions such as, "Was this lump here last year? Does it look the same to you?"

While the servers at that practice, in the West, were back online by the third week of November, it took until the second week of December for the Midwest hospital.

The veterinarian at that location commented that having one computer to do the work of what normally took a dozen computers was a marginal improvement, almost as bad as no access. "We can't look up doses for med refills, vaccine due dates, or old histories," the veterinarian wrote on VIN. "We can't keep track of fecals or other things sent to outside labs. Everything put in place by management is a band aid on a massive hemorrhage."

All the veterinarians interviewed for this article said their practice information management systems were fully or nearly fully operational by mid-January.

Keeping it mum

As reported by Krebs, NVA distributed talking points to staff instructing them not to share facts about the incident nor to give a count of the number of hospitals involved, and to "use the verbiage ‘computer outage.' "

"The directive not to share the truth of the matter with clients has been one of the hardest things," one of the veterinarians wrote on VIN. "Even though it would tremendously help our clients to know ahead of time what is going on, instead of being told individually as they come in and wonder why we don't know their dog's history."

Employees were told that it wouldn't serve to reveal to the hackers "how much this hurt us," the veterinarian said.

One veterinarian told VIN News in November: "We were not allowed to make any kind of blanket announcement, but if people ask, we tell them. We are at the point where many clients have been here multiple times and are quite astonished, appalled, even angry that the computers are still down. They are questioning why we haven't fixed it, why we don't hire someone, etc. So we are forced to ‘come clean' about our corporate ownership to some clients."

NVA has a reputation for shielding its relationship to practices it owns. The veterinarian confirmed that employees have been told explicitly in the past not to tell clients the practice is corporate-owned.

As for NVA's decision to keep mum about the attack, that is common practice, said Nick DiPasquale, a security professional at Set Solutions, a cybersecurity company based in Texas. He said companies often wait to acknowledge an attack occurred because "addressing an ongoing security incident can jeopardize the ongoing investigation and remediation efforts."

While that may be true for companies in general, veterinarians have certain obligations to communicate with their clients.

Raphael Moore, general counsel for VIN, said corporations are within their rights to tell employees what to say, how to say it, and when to say it. "However, licensing rules and the requirement of a veterinarian to meet regulatory requirements for the practice of veterinary medicine would necessitate a conversation," he said. "I think the ‘fine line' of being directed how to practice medicine by corporate is being blurred, if not trampled on completely, when a veterinarian is told not to discuss the loss of medical information."

For example, he said, a veterinarian might be obligated by their state's practice act to divulge to the client that they are missing information before prescribing medication or embarking on surgery.

Moore said the loss of medical history raises tricky questions such as: If the doctor needs to perform diagnostics to replace missing information, who pays for the test? And suppose clients depend on reminders from their clinic to bring in a pet for heartworm testing or vaccinations, and they aren't notified because records are inaccessible. Does the practice bear any liability?

No practice is immune

Dr. Sarah Vineyard, owner of a one-doctor practice in San Diego, California, was the victim of ransomware attacks that locked her software — coincidentally also AVImark — in 2016 and 2018. She described the second time as "crippling."

"We were down for 12 days and they were some of the most terrifying, stress-inducing days I have ever had, as I was terrified of missing something crucial or making a mistake," Vineyard said.

Vineyard did not pay a ransom. Instead, she hired an information technology consultant, who successfully decrypted her files. She said she was helped along by a software product she used for client communications and appointments, which works with AVImark. 

Tapping information in that system, Vineyard was able to access her schedule to avoid double-booking. The software also gave her access to client contact information, so she sent emails alerting clients and asking for their patience.

Sharing her experience in the VIN message board discussion about the NVA malware attack, Vineyard wrote: "We are a small independently owned practice. We have tight relations with our clients. Transparency goes a long way toward tolerance."

She elaborated: "I was able to remember much, but in the event that I needed to verify details, the clients were very understanding," she said. "We've all had computer problems in the past and more often than not, they felt awful for us. There were even a few that called to check on us after the fact."

To help practitioners affected by the NVA attack, Vineyard posted some hard-earned tips for responding to an attack. See a shortened version. 

‘Massive disruption'

Ransomware attacks are becoming more targeted, sophisticated and costly, according to an FBI public service announcement posted in October. Since early 2018, the FBI said, the incidence of broad, indiscriminate ransomware campaigns has sharply declined, but complaints to the bureau's Internet Crime Complaint Center and case information indicate that losses from ransomware attacks have increased significantly.

In the first nine months of 2019, at least 621 government entities, health-care service providers and educational institutions in the U.S. were affected by ransomware, according to a report by Emsisoft, an international anti-malware company based in New Zealand. That tally is based only on publicly disclosed cases.

"The attacks have caused massive disruption: municipal and emergency services have been interrupted, medical practices have permanently closed, ER patients have been diverted, property transactions halted, the collection of property taxes and water bills delayed, medical procedures canceled, schools closed and data lost," the report said.

The health-care sector accounted for the lion's share of attacks. "Cybercriminals understand that healthcare providers are often more inclined to pay the ransom as failure to do so may result in data loss that could potentially put lives at risk," the report states. At least two health-care providers reportedly went out of business in 2019 as a result of ransomware attacks.

DiPasquale, who has no direct knowledge of the incident, speculated that the NVA attackers might have thought they were targeting human clinics, which are additionally valuable to hackers as sources of data for identity theft. "But so many organizations are being targeted now," he said, "that it's also a possibility that this chain of clinics was identified as a soft target due to their lack of security."

Victims of attacks may, like Vineyard, choose not to pay the ransom and marshal other resources to restore and rebuild data. Others pay it.

The FBI recommends against paying ransom because there is no guarantee the hacker will unlock the data and because capitulating provides an incentive for more criminal activity. But tell that to companies facing massive disruption. To them, paying ransom is more expedient and potentially more affordable than trying to restore information from backups, according to DiPasquale.

While paying a ransom is risky because the hacker might take the money and fail to restore access, the transaction can be surprisingly business-like. "The hacker world is based on reputation," DiPasquale said. "If word gets out that the ransom doesn't get your data freed, nobody will pay."

Protecting your system

Malicious code can enter a computer through multiple means. It might come through a phishing campaign, which involves email appearing to be from a reputable source that lures the recipient to click on a link or download an attachment that then infects the computer. It could be a program that crawls the internet looking for vulnerabilities in systems.

The former is the more common, DiPasquale said: "People will always be your weakest link."

There is no way to be entirely protected from malware. However, cybersecurity consultants and the FBI agree on a few fundamentals.

• Be sure everyone in your workplace understands how to identify and avoid phishing lures.
• Back up data regularly to an offline source.
• Be sure operating systems and applications on all the devices used are the most current and patched versions.
• Keep anti-malware software up to date.
• If a message appears on a device warning of an infection, immediately disconnect the device from the internet and from the workplace network to try to prevent the problem from spreading.

"Get comfortable with the thought of being vigilant about your security," DiPasquale advised, comparing it to using care with a lit candle. "You would blow it out before you leave."

Bringing in a cybersecurity consultant is another option. An annual or semi-annual review with a consultant can fine-tune a staff's cybersecurity practices and help business owners identify vulnerabilities. DiPasquale admits consultants can be expensive. "The industry is run by fear and the knowledge that a malware attack can take your company down," he said. Rates of $500 to $700 an hour are common.

Krebs reported that NVA hired two outside security firms to investigate and remediate the attack. He also said the company was installing software from Carbon Black, a cloud-based cybersecurity company, on all NVA properties.

Aside from keeping systems up and running, there are intangibles to consider, including why some practices came through the attack better than others.

For some, the extended outage, especially if perceived as poorly managed, caused lingering distress.

"There are going to be long-term, if not permanent consequences to our mental and physical health, and to our relationships with each other and our clients," one veterinarian wrote. "Overall I love my clinic and my job. I love them too much to look elsewhere … But this really, really, really, really sucks on a level that most people just do not understand."

Paulson, on the other hand, found several silver linings in the experience. The act of updating medical records from handwritten travel sheets, he said, caused him to brush up his understanding of AVImark's diagnosis and problem codes. “Now I feel like my record-keeping is better and more efficient,” he said.  

He also said his fear of malware is reduced. “I’m less worried about that now," he said. "It’s always been on my mind: What would we do if we didn’t have computers? Now the answer is, we would be OK. You can have a high-quality delivery of medicine and surgery without one computer in the building.”