FAIRFAX
COUNTY UNPLUGGED:
THE
SCOURGE OF NIMDA
by Sharon D. Nelson and John W. Simek*
NIMDA is Admin spelled backwards. There is nary a network administrator in the world who did not feel the sting of NIMDA when the most pernicious virus/worm hybrid of all time raced through the wild on September 18, 2001. A week to the day after the World Trade Center ceased to exist and the Pentagon was badly damaged, NIMDA appeared. Coincidence? Perhaps. But technologists universally agree that we have never seen anything quite as clever as NIMDA, nothing that could exploit so many vulnerabilities, operate in so many ways, and even lie dormant until springing back to life again.
Fairfax County is the largest county in Virginia and certainly the county most heavily dependent on technology. Long a leader in e-government, Fairfax County has justifiable pride in its innovative and far-reaching methods of using technology to further the ends of government. One of its greatest strengths in the modern era became an Achilles' heel on the day NIMDA spread like wildfire across the globe.
The County became aware of NIMDA on the second day of its existence, when it appeared on County computers. The exact method by which NIMDA first penetrated the County network will probably never be known, according to David Molchavy, the Chief Information Officer for Fairfax County. NIMDA exploits so many different methods of propagation that tracing its entry is difficult. As we know now, NIMDA crawls through back doors left open from the Code Red virus, through e-mails, through visits to infected web sites via JavaScript, and through various other server software vulnerabilities.
The first few cases on Wednesday didn't set off major alarms, as network administrators sought to follow the advice of the experts and cleaned NIMDA from infected machines. But by Thursday, NIMDA's frantic self-replication had reached crisis proportions and the County made a decision to unplug itself and undertake a full scale scrubbing of its network. Down came e-government on the County's web site (www.co.fairfax.va.us) and down came most local government operations, including the courts.
This was not the first time that Fairfax County had made the decision to unplug itself. When the Love Bug virus struck, the County also briefly unplugged itself while it managed the situation. It is worth stressing that NIMDA did not bring the County network down. While it might have done so in time, the County sensibly and prudently surveyed the situation and chose to prevent further harm by disconnecting all machines from the County network until NIMDA had been eradicated from the County network.
Because NIMDA is so insidious, this meant a 45-60 minute process of working on each of the 9000 workstations and 300 servers attached to the County network, scrubbing NIMDA where it was found, and installing the patches that would prevent reinfection. For the first time, the County established a "Command Center" to deal with a virus/worm and followed a "triage-like' methodology of treating those with highest priorities first. More than 150 technicians worked around the clock, catching naps on cots in a nearby room, in a Herculean effort to bring the County's systems back up as soon as humanly possible.
Courts were high on the mission critical list, just behind public safety and human services. The County pulled the plug on the courts on Thursday night. On Friday the 21st and Monday the 24th, there were no filings accepted as the Land Records office of Fairfax County Circuit Court. Though a considerable nuisance to all parties impacted, there was only minor financial damage in the form of an extra day or two of interest charges and the difficulties of complying with the Wet Settlement Act. Generally, the Court receives 1000-1300 land records filings a day, including deeds, trusts, releases, certificates of satisfaction, assignments, and powers of attorney. The court estimates that less than 1000 of the would-be filers suffered the minor financial harm of the delay during the two days of downtime. As soon as the Court was back up on Tuesday morning, employees worked overtime, and succeeded in becoming completely caught up by Thursday.
Clearly, other court business did not proceed normally, but the courts were open and operational. Though paper filings were accepted, there were no electronic filings and no electronic searching capabilities. Dockets were updated by hand. As one court official said, "it wasn't great, but we could do what we needed to and we muddled through." Like all the other County agencies, the Court could only wait for the County to "green light" its computers to resume normal operations.
David Ferris, the Network Administrator for the Fairfax County Police Department, said that several individual workstations and a file server were infected by NIMDA. Fortunately, the Computer Aided Dispatch system, which sends policemen to crime scenes, firemen to fires, and ambulances to the injured, is NOT connected to the County system and was therefore unaffected. But one of the first critical systems to come up was that which supports the County's emergency helicopter. At the time of NIMDA's assault, the Department did not have a written priority list of whom to bring up first, but Ferris stated that it certainly is developing one now. Though having a formal list is a good idea, Ferris said the priorities were self-evident and handled appropriately.
The greatest impact in the Police Department was on administrative functions, which were the last to return to service. Emergency services were dealt with first, then Central Records, then the Massey Building (where police headquarters is housed), and then the individual stations. As with the courts, there was no loss of data on the servers. Like many other County agencies, the Police Department is now thoughtfully considering back up procedures in case of another technological catastrophe. Though no changes have yet been implemented, Ferris expects a multi-level defensive plan to be developed so that back up systems are in place for all police functions dependent upon computers.
Emphatically, no court or police data was lost or compromised. Though this is the major fear expressed by technophobes, it is impossible under current back up policies to lose any data other than that of the current day. Restoring from back up media may be a nuisance, but no more than that. Likewise, the County lost no data from its servers, although the Police Department noted that some individuals might have lost minor data they stored on their local machines.
Did we all return to paper? No, even though paper alternatives were sometimes available, as within the Police Department, the tendency was to simply wait for the machines to come back online. We have become creatures of the tech world and are "on hold" when our technology is on the fritz. A room full of police detectives, when queried by the authors about what they did when NIMDA brought their computers down, cheerfully answered "Not much." Most of them had a 4-5 day breather, refreshing no doubt, but what if the computers had been out longer or if a national emergency occurred?
Is NIMDA really gone? No one knows for sure. We already know that NIMDA can lie dormant and rear its ugly head in some new form. Some preventive measures are in place against what we know, but we have learned to be wary that what we know today may prove woefully inadequate when tomorrow's virus mutation starts burrowing into our systems. This has been especially true of NIMDA, as the virus specialists continue to update their virus signatures as more of NIMDA's devilish attributes become apparent and more patches and fixes are needed to combat them.
We have a tendency to forget how desperately dependent we are on our technology. Whether struck by lightning, cyberterrorism, or viruses, we can be rendered all but helpless in short order. We can live without being able to check out a library book or reserving tee times online at County golf courses, but it a different matter when we need to dispatch a helicopter to airlift an injured driver from the Beltway or when we are unable to process court records or pull up criminal records.
Fairfax received more than the usual amount of publicity because it had to advise the public that they could not pay their personal property taxes via the Internet during the NIMDA cleanup. Not the kind of publicity the County likes, but it is hard not to applaud how professionally the NIMDA scourge was handled by the Command Center and its cadre of technologists.
While the County has always been mindful of computer security, even to the point of hiring outside specialists to make sure County systems are as well protected as possible, NIMDA is an example of a new phenomenon against which no protection was available when it was released into the wild. As Molchany notes ruefully, "this is the cost of doing business in the information age." And costly it is -- while there are no precise figures, experts concur that NIMDA is the costliest virus/worm yet, with billions of dollars expended on its eradication.
Fairfax County, for all the glare of publicity, probably did the prudent thing in unplugging itself. As Molchany said, "we successfully moved into incident recovery mode and, through long hours of volunteering, brought the systems back up with relative speed." The aftermath has consisted of reporting to the Board of Supervisors on the incident, developing future Command Center procedures, and setting forth precise lists of computing priorities in the event the network must be taken down in the future. Have new security measures been taken as well? Yup, but don't ask what they are because Molchany isn't talking (nor should he!). NIMDA was a rude wake up call, but Fairfax County (and every other government entity) has had fair warning and should be on constant alert.
There are elements out there, whether mischievous, criminal, or terrorist, that will seek to wreak technological havoc. Security, always an issue, is now the greatest concern of e-government. If biological and chemical warfare present enormous health hazards to our country, cyberwarfare can potentially do untold damage to our national infrastructure. As always, prevention is better than a cure after the fact, and preparedness is now a byword among network administrators around the globe.
* Sharon D. Nelson and John W. Simek are President and Vice President, respectively, of Sensei Enterprises, Inc., a legal information technology firm based in Fairfax, Virginia. Sensei is the developer of the electronic filing pilot for Fairfax County Circuit Court and a provider of computer forensic services to courts and law firms. They may be reached at: (703) 716-0085 (phone); sensei@senseient.com (e-mail); http://www.senseient.com (web site).