As much as possible we need to automate security testing. I know this always starts a flame war, however, with the complexity of networks growing, especially with virtualization, it’s more important. Example: Netflix released their network management approaches, and it’s a sharp contrast to the way we've managed in the past. In the past, it’s been a "OMG don't touch it, you might break it." Well, if you can break it, there's something that needs to be fixed, and if you don't know what's broken, how can you fix it? Netflix even goes so far as to take down portions of the network and see how it reacts. Now, as Mortman says, “careful with live ammo.” But this is how I always wanted to manage a network. In a controlled environment test performance, reliability, and security. Then, fix the problems you find. If you have fail-over, force it to fail over. Scan the network constantly, if stuff crashes or has vulnerabilities, fix them. It’s almost as if we need a QA department within every IT department to test it on a regular basis and track the fixes. Better you find the weaknesses than wait for an attacker or "network anomaly" to find it for you and go into "firefighting mode" by trying to fix it with management breathing down your neck.
That... And this post is merely an excuse to have a picture of the Simpsons Smoking Monkeys..
Remember, good security is really just good systems administration. We strongly recommend looking at the documents that are available from visible operations.