Today, April 21, 2010, McAfee thought they were doing the world a good thing when they issued DAT update 5958. Not too long thereafter, there were cries from the IT trenches of machines that would no longer boot.
As it turns out, McAfee borked up the DAT file. The borked up DAT file resulted in McAfee classifying the critical Windows executable svchost.exe as a virus. McAfee would then delete and zero-write the executable, resulting in the system rebooting itself. Svchost controls critical system services like the RPC service, which thus caused other services not to load, like the critical network services. Without these services, you can log on as a local admin, but the machine will not get an IP address, it will not load themes, drivers for things like USB flash drives will not install, and trying to view properties on event viewer items results in nothing happening.
Initially, our sleuthing was turning up evidence that a virus had tampered with system files. We were pretty happy yet extremely alarmed when we found out that it was McAfee.
So, how to fix it? Well, that was an interesting dilemma. We were able to copy svchost.exe from an unaffected machine to the machines with the messed up svchost file. However, we cannot just boot the machine back up normally, as McAfee still had DAT 5958, which means that the executable would be intercepted and quarantined again when rebooted. Thanks to the hive-mind of Twitter, we got a few new ideas that worked great:

1) Boot the machine into safe mode. Navigate to the McAfee program files directory, and within the VirusScan Enterprise folder (in our case), we renamed mcshield.exe to something else. I chose ___mcshield.exe. Replace the svchost executable using a CD with the good executable as the source. Reboot normally. Quickly open the McAfee VirusScan Console and disable On-Access Scanner, then run a DAT Rollback. With the newer versions of DAT files released, you can simply update the app and it should go past the wonky 5958 update.

2) The method I preferred to do involved a bootable USB stick with the installer for Windows Vista, Windows 7, or Windows Server 2008. I created a folder on the flash drive and placed a good copy of svchost.exe from an XP machine in there. Using the USB drive, I then booted into the installer and chose to Repair the installation. In the repair options, I opened up the command prompt and copied the good svchost executable from the flash drive to the affected system. After the file copies, I reboot the machine into safe mode and rename the McAfee mcshield.exe file as described above. Reboot normally and either Rollback DATs or update to above DAT 5959.

We later ran into problems where in order to get to DAT 5959 from 5957, McAfee had to update to DAT 5958. Pete did most of the back-end work to solve that problem, so I don't really have any input on how to fix that. I do know, however, that update 5959 is safe, but takes some care to update to it.

So how do we plan on mitigating such problems in the future? Well, we are considering putting in a time delay before updates are pushed to the clients, but this has mixed issues. We wouldn't be able to respond as effectively to fast-moving threats. Alternatively, we briefly considered putting svchost.exe to a white-list, but that leaves a huge, huge door open to later attacks.
Ultimate solution: Spend a day devoting all man-hours to fixing McAfee and get it working properly.

McAfee claims that there was minimal impact to customers due to this problem. If taking employees off of important tasks and having unusable workstations is 'minimal impact,' I think McAfee has a slap to their face coming.

-Elekt