This is a situation that unfortunately is not uncommon and the ongoing changing of the access goalposts can go on for years after the relationship has been dissolved. It seems that your ex-husband is still emotionally involved with you in that his current reaction seems to be in response to your new, stable relationship and this has invoked an old response pattern in him.
When you were together, he was abusive and tried to exercise power over you. This current situation seems like another version of this, and I wonder if you need to look at your past pattern and learn from your history. In the past, you suffered for a long time before accepting that you had to take action, no matter how difficult and upsetting it was. Now, you may again have to take action that is challenging and unsavoury. It is particularly difficult because of your huge desire to protect your children.