Firstly, find out how to checkmate with just a rook. It's a real game-changer. Nextly, check out some mating patterns. Usually, if you don't know how to checkmate, just simplify until you've got a rook and the opponent doesn't have anything, then checkmate from there.
Trouble Checkmating Without Queen?

There are free drills for king and queen checkmate and king and rook checkmate. You should try practicing it.

I would reccomend the course checkmate patterns manual, althought i should be using it more myself, im 92/1000 variations leanred. I find for myself i forget to see where the king can run too instead of setting a mating net an being patient.

Learn rook mate and also try going to L*chess.com and looking into the "Practice" section to learn piece mates and mating patterns.
Is there any videos/techniques I can learn or become knowledgeable about to help me improve my endgame without a queen? Is it normal to struggle to checkmate without a queen?