Is this right to do...?

Yes what you did was wrong. I would not stay with a man that wanted to desert his child because she was the past. You both did great harm to this girl, this girl that you claim to love. I hope he does not do the same to you and your children.