Family is the people that love you, not necessarily those related to you. It sucks when you go through the hurt of actually deciding to cut people out of your life, but if someone if making you miserable and being toxic, being family isn't an excuse for them to keep doing it. The best thing I ever did was cut off my entire family. I know some people think you should do whatever you can to keep family 'together' and keep up the appearance to the outside world that things are just peachy, but life is too short in my opinion to put up with people who are toxic. I look back now and wonder why I didn't cut them off years before.
There's nothing wrong with not being close to your family, and also choosing to only have one kid. You have to look at the best decisions for you and your husband and baby now.