Every one of us at some point in our lives must spend time with people we would rather not. Whether it’s at work, school, socially or family, my guess is we all have those people that try us in some form or fashion. It is an inevitability of life, therefore my advice: learn to live with it, artfully.
My general advice, keep your mouth shut as much as possible. Very little good comes from instigating or reacting to inflammatory actions. There are people in the world that must have drama in their daily life to feel fulfilled, why I do not know (I’m sure we could analyze the psychology of it easily enough but that’s already spending too much time on people we are trying to avoid). I personally find it utterly exhausting. Thus, I recommend engaging those people minimally and sharing little of your own personal circumstances. You can be polite without having to be involved and you certainly don’t want to create any misunderstanding surrounding your relationship.
When you know you are good person, that you try your best even though you still make mistakes, that you will always work to keep your priority in focus (whether it be work, kids, etc.), don’t feel guilty about how you feel, we are all entitled to our own feelings and hopefully you are doing your best. And, perhaps the people you want to avoid are doing their best too; they are just incapable of better. Who knows? So when it gets really tough and you either want to wring someone’s neck or completely walk away, remember, you would most likely be where you are or doing what you are doing, with or without those annoying folks, they are just white noise.