Is there a secret to this opening, or is it just terrible?

Your games aren't exactly evidence of it being bad.

I've won every modern Sicilian I've played as white apart from one draw, I'm pretty sure that's not a bad opening for black.

Atm I haven't lost playing either side of the Scandinavian. Clearly it's a forced win for both sides.

There is not much reason why the queen should be better placed on a5 than on d8. It is certainly better placed on c7 which takes a move in any way. But on a5 it just looks developed while getting pushed around and landing on d8 in most cases several moves later.

It might be slightly better on d6, which made this line popular some years ago.

With Qd8 black will usually look for a passive and solid position without any obvious weaknesses or bad pieces but a white space advantage and often a white bishop pair.

Here is a sample from a GM-game:

  Clearly true of most mainline openings.

But admitting to it would put a terrible damper on this discussion.

Qd8 It is called the "Banker" Scandinavian  

lol! Qd8 xD thats terrible do Qd6 instead.

Aha!  So it's 6 Bd3 that's terrible!

It's bad, but not as bad as it looks. The idea is to give up the center and then play against it, with an open d-file and either Bg4 or a King's-side fianchetto... very much in the style of a Gruenfeld or Alekhine's Defense.