Do what seems logical to you, logic defeats things like this. You say you know that you're tired of being part of this group, so stop being part of the group. Don't worry about hurting them, don't worry about what might happen, just do what you feel is right. You're more important than them, you can't place their feelings above yours.