How do I beat the Scandinavian?

I enjoy playing 1.e4 but the scandinavian is the thorn in my side. I cannot find a way to fight it. Often times black will play Bg4 and then castle queenside and attack my d4 pawn and I cannot find a way to counter this. What should I do? Please response knowledgable members as this is the only opening holding me back from playing 1.e4 with confidence.

Possibly of interest:
My First Chess Opening Repertoire for White by Vincent Moret
https://www.newinchess.com/media/wysiwyg/product_pdf/9033.pdf
https://www.chess.com/article/view/how-to-understand-openings

Opening Repertoire: 1 e4

https://www.newinchess.com/media/wysiwyg/product_pdf/7819.pdf

A SIMPLE CHESS OPENING REPERTOIRE FOR WHITE by Sam Collins
http://www.gambitbooks.com/pdfs/A_Simple_Chess_Opening_Repertoire_for_White.pdf
http://www.jeremysilman.com/shop/pc/A-Simple-Chess-Opening-Repertoire-for-White-76p3916.htm
Playing 1.e4 - Caro-Kann, 1...e5 and Minor Lines by John Shaw
http://www.qualitychess.co.uk/ebooks/Playing1e4CaroKannandothers-excerpt.pdf
A Chess Opening Repertoire for Blitz and Rapid by Evgeny Sveshnikov and Vladimir Sveshnikov
https://www.newinchess.com/media/wysiwyg/product_pdf/9020.pdf

Keep in mind that you're not "beating" the opening, you're getting a playable middle game.

Cool. Thanks for the insights.

Glad to help.

If pressure on d4, is the issue then play d3 lol.  3. Nc3, 4. g3 like this

 No mention of 2. f3? No loss of center control after exchange on e4. 

That sounds like a good idea.

Bazing-ga

blacks queen is not doing much there

People wayyy over think the opening.  KISS...Keep It Simple Stupid.