It might be a bit much, but why not a brief comment on why you've repped +ve or -ve? I mean, if we ban someone we give a comment as to why, so why not do it with reps, too?

Makes it easier to moderate, and the rep-ee can report abuse of the system judging by the comment (eg, a -ve rep and the comment "Yo mom sukz ass craka!!11!!ONE").