The reason for suggesting that children have responsibility towards their parents is because of the strong ties of debt relating to the care the parents have generally provided towards the children. A good person acknowledges their debts towards others and acts in ways that repay those debts. Your responsibilities towards your parents relate to your understanding and appreciation of their help towards you.
Responsibilities like this are conventional truth, that you should acknowledge and allow to guide your choices in the world. They do not relate directly to ultimate reality, since on an ultimate level there are no beings, there is just experience.
When others are abusive, this I think weighs negatively on any responsibility the victim might have towards them. Again, this relates to conventional reality. On an ultimate level, the goal is to understand abuse as merely arising and ceasing experiences of seeing, hearing, etc.
So, practically, you have to do two things:
First, you have to weigh the good people have done for you with the bad to determine how you should act towards them on a conventional level, independent of your affection or aversion towards them.
Second, you have to address your affection or aversion towards those you come in contact with, working to rise above partiality to see your experiences clearly.
It sounds like the manipulation you refer to relates to the first imperative, and the not wanting to deal with negativity and complaints relates to the second. You should try to separate your aversion from the things people do or have done to you.