No, you aren't wrong.

If he's not working, you control the money, remove his access to your account and give him a little "allowance". 

Tell him he has one week to find a job or he can find a new place to live.

If it's your money, then don't let him have access to it. Take his debit card, and if he needs cash for something, he needs to ask you for it. If he wants spending money for his kids, then he gets a job.

I really hope he's responsible in other areas, because otherwise that situation is unlivable.  I agree take all forms of card away and give him a cash allowance. 

I would just tell him that of course you don't consider his kids as your kids, they are quite literally his kids, you had no part in creating them, and no, you will likely never love them like your own, especially if he's determined to spoil them rotten.

This is what I would do. Sit down with him and tell him that you two need to sort out finances. Do not bring up his kids. Go over what your household income is, what your outgoings are and work out what money will be left. Estimate what will be needed for groceries and gas. Discuss how after all of these essentials and a little money set asside for building savings there is very little money left for discressionary spending and ask how he wants to deal with this discressionary money. Does he want to spend it collectively on things that you both agree on or does he want to split it between you two and each have a small amount of physical cash you can spend as you wish? Give him some options. Arrange with him that you two will have another discussion about finances in a month or two and go over how this budget and plans for discressionary money has worked out and that you can both raise any challenges then and make changes if needed. Tell him that you want to work with him on this and if he objects tell him the alternative is that you sort all the finances and give him a small allowance out of your paycheck and that you would rather be a partner than a parent to him. 

Being part of the process of working out a plan and having autonomy within that plan to spend even a small portion of money as he wishes without being accused of being frivolous or unfair could help him stick with the rest of the budget. Working on finances collectively when you have different financial habits outlooks is challenging and requires both partners to adapt. Finding a solution that works for you both might not be easy but it will be worth the effort. 

Again do not bring up his kids. He is being unreasonable about it all but it will only cause him to throw baseless accusations at you. Stick to finances and your budget as that is what the core issue is. If you choose to both decide how money is spent then you can advocate for your child. If you choose to split the discressionary spending then you have 50% of the money to spend on you and your son if you choose and he has 50% to split between him and his 4 kids. If he wants more money, he can work on getting a job.  

This is good advice. My DH was not great with money when we met and no way were we going to share one account. We, as reasonable adults, sat down and did what Beth says ^^^.

If your DH refuses then you have a bigger problem than finances.

My DH is 300% better with money now. I still hold the joint debit card, but would trust him now id he needed it. Debt is being paid down.

You aren't going wrong, I just don't get men I really don't. My fiancé did that around November of last year when he was strapped on cash and he bought SD a 30$ doll and didn't buy our daughter a toy. 

The sneakiness is a whole other issue. He's eroding the trust in the relationship. The fact he goes behind your back means he KNOWS you don't approve and he's doing what he wants anyway. That's not how it works in a true partnership.

I would definitely examine the fact that he is gaslighting you and doing the Deflection Dance.

He is accusing you of not liking his kids. He is accusing you of "not loving them like your own". He is sneaking around behind your back and not telling you about his uses of marital assetss (this is called "lies by omission")

To answer your question, no, you are not wrong. The fact that you are asking this shows how his manipulations are working on you. Well here is a dose of clear cold reality: He is committing Financial Infidelity. Cheating you out of your money that you are saving up for your own kiddo. Very sad.

Have a HUGE talk with him. That from now on you will be taking control of the finances. That he will be given what he needs and that he needs to get out there and get a job, any job. So that he can support his own kiddos. No more toys for kiddos, from YOUR money. He very clear about how you feel about his actions. And give him repercussions.

My husband and I have separate finances. Previously when I was under employed I only helped a little bit. NOW, we make the same but everything is shared. Anything that we have left over is for our own decision how to spend/save.

"You don't like my kids" (TM) strawman argument.

"You're mean"  "They're juuuuusssttt kiiiiiids"  "I want them to feel at home"

Uncontrollable spending on skids then hiding the evidence..(shudder)  you're giving me flashbacks

So why didn't he also buy your shared child a toy too? 

Some of these guys... it's like they see their 1st family kids as their kids but shared kids as your kid.

My formerSO seems to be like that and further evidenced by him not ever seeing our shared child after I left him. He only did until he realized I wasn't coming back, now hasn't seen him in years. 

It's called deflection. He throws that at you when he doesn't want to talk about the core issue. Call him on it next time. " This is not about how I feel about your kids, this conversation is about the finances. Every time I bring up the budget, you throw that out at me to change the subject. If you continue to change the subject , I will have no choice but to take control of the budget without your input so the necessities get covered."    It then puts it back into his court to either discuss the budget or not. If not then " ok since you chose not to discuss this , you leave me no choice to handle it on my own".  Take him off all accounts and keep him away from your money. If he complains, he can go earn it if he wants money.