Is The person with black peices at a disadvantage because they go second ?
I think so. It is small. But statistically, and over a long time period, white tends to win more often than black. White moves first and may also have a psychological advantage.
Maybe your problem with the black pieces is that white plays openings you are not familiar with as a white player?