I have no issues on either. One may argue I am equally bad at both....!
Lately, I have had more success taking opponents out of their comfort zone and playing the kind of chess I like.... so no Slav's or King's Indian, No Centre Counter or Caro - Just Gambits. Noteably the Latvian Gambit against e4 and Budapest Defence (not always the gambit lines) against d4.
So could style of play also be affecting you play against d4 and c4?
I find e4 much easier to play against. d4 not so much and positively hate black against c4
Have toyed with everything over the years against d4 but never settled on anything. QGA my latest, see how long that lasts