We both love each other, share same dreams, and so compatible but he needs me to give my kids back to my ex.?

I really love this guy so much and he feels the same, he is always there for mi and so caring and all but our one problem are my kids from previous marriage. He wants mi to give them back to my ex- so we can carry on with our life. We have been dating for 1yr now and It hurs mi so bad dat i am going to loose him. I need help. wot should i do???