The point of caster gear is to up your poison dagger damage. The only reason you wouldn't want that gear is if you were just going to be a melee necro with poison dagger as sort of a bonus. If you're relying on poison to do your damage, you're going to need +skills stuff to get that poison damage up.

As for low defense, you just need to keep bone armor up. It's a hit-and-run build. You stab once or twice and then back up to put bone armor back on.

What is it they say about silence being golden?