BF says I have inappropriate relationship with ex DH
Ex DH and I have 2 children who are 6 and 2. We divorced when I was pregnant with our 2 year old due to his drug use. He was in jail for about a year (drug charges) and was released end of November. He slowly started seeing the kids again, and has been seeing them regularly since February. I have full custody but he has visitation which is supervised by me (states this in our court order, we both agreed to it).
BF and I have been together 1.5 year and did not have any issues related to my ex DH since he was not around. It seems like since ex DH has been around, it has been nonstop issues. BF accuses me of having an inappropriate relationship with him.
For the past couple months, we have been meeting at my mom's house for his visits. My mom lives in the same neighborhood as him and I was visiting her on weekends anyway. Ex DH typically plays outside with the kids while I hang out inside with my mom. Everyone is happy with this arrangement besides BF. He says I am "spending my weekends with ex DH." This is his visitation that he is legally entitled to. We agreed on the days and location because it works for us. I told him he doesn't have to like it. I even asked if he wanted to come a few times but he never wanted to. We got into it again this past weekend and he left again.
I am not sure where to go from here. BF thinks there is something going on between ex DH and I and there truly isn't. I have not seen him without the kids since we have been divorced. My kids want to spend time with him and as long as he is clean, I want them to see him as well. I won't tell my kids they can't see their dad because BF isn't comfortable. BF was well aware of our custody arrangement and that ex DH would eventually be out of jail. I'm just baffled. I don't want my relationship to end over this but I don't know what other option I have.