I would find this a terrible situation to be in, but there is not much you can do except move. Never, ever let her know it bothers you. Are there young children involved? As far as how to handle it, make sure you know what the custody order says and decide now how flexible you want to be with unexpected "drop-ins."

If you think she might be a problem, get an alarm system, deadbolt locks, and good exterior cameras. So sorry this is happening to you!

We used to live a 5 minute drive away from BM - we lived near her for 17 yrs - I think we did our time!  We moved an hour away in 2020 and it was the best thing to put some distance between us and her.  Having said that, if you decide to stay I'd agree with the previous post about good locks, alarms and cameras.  Discourage BM coming around very firmly.  I would do this by checking your door cam and just not opening the door to her.  

"I know that I am probably sounding dramatic,"

No, we get it. All i've got is sympathy and a suggestion to talk with your DH about how this move will affect communication and the schedule (it will be much easier for the kids to bounce back and forth at BM's whim, i know this because BM here would bring the skids back and forth multiple times per day without asking.) Boundaries must be maintained. 

. . .  his controlling and narcisstic ex

That's all you needed to say for anyone here to understand. 

This happened with a couple I know. The crazy and out-of-control alcoholic ex moved up the road from them. The way they handled the situation was they remained steadfast. Firm boundaries. They did not budge. Of course, it made the crazy ex crazy and, as always, she pushed the boundaries. Eventually, this game did not pay off for the ex, and in about a year's time, SHE moved. 

Keep the focus on YOUR right to a PEACEFUL home at all costs. I hope you and your spouse are on the same page with all of this. Stick around here for support! 

This is going to depend a lot on how your DH is?  Is he easily persuaded to do things for the kids? We can see BMs house from ours. Not sure if she did it so she could keep track of my SO or not but it is not a good situation. BM is always asking for him to ride SS everywhere. Her house is not out of the way so it makes it easy for my SO to do it and he is a doormat to them. The kids don't stay anymore but when they did they could forget things and be able to make my SO take them to get it and there were frequent walk ins by the kids grabbing stuff they forgot here. It is not pleasant. If you have a DH with a backbone and boundaries it might be better. 

This sucks and I feel for you. I have always preferred to have a couple of highways and a river between us and BM. She did move closer to us, once, for a short while, but SD was 18 and married so BM had no valid reason to bother us and my DH had made it clear to her that he wanted nothing to do with her, ever, after SD was emancipated.

Like everyone has already stated, you and your husband need to agree on boundaries and stick to them for dear life.

This really sucks. At the beginning of our relationship my husbands ex wife lived nearby. She used to pop in when we were gone on vacation until he locked her out.

When the kids got older if there was any sort of disagreement at his place when the kids were acting up  then the kids would run down the street to their moms house to escape from the consequences of their actions... Something to keep a watch for.

How old are the kids? I agree with the others to have strong boundaries, ring doorbell, cameras etc and do not allow her in your house. Do not allow the kids to invite her over either. 

Pick up drop off go as in CO.  No talking. Everything by text,   No dropping over. No babysitter kids before school / after school.  No cutting  her grass fixing her car ect.  Nothing will change. How's it is going now will continue.  Make a line in the. Sand.