You were. Guy was an idiot.
USCF Rules, do you know them ?

ask this guy BishopCannons who has played for 35 yrs and believe there should be a chess.com chess rule same as the FIDE !
http://www.chess.com/forum/view/general/whites-clock-starts-first

I would say that 16J, "Black not present", and 16M , "Equipment needed to start the clock" support your position.
16J "If Black is not present for the start of the game, White shall start his own clock, make his move on the board, and start Black's clock."
16M "...A late opponents clock may not be started until the board and pieces are in place..."
39A "Choice of equipment" is more explicit. "...If Black does not provide standard equipment and White does, Black does not have the right to delay the start of the game to search for alternate equipment..."
You'll have to excuse my obsession with the rules. I'm a former soccer referee, which causes me to be a rules junkie.

I think you are right. Since your opponent was late for the match, his clock is allowed to be started. He also forfeits the right to pick out his pieces/board/clock since he was late. The way I look at it is: rules aren't meant to "save" someone when they are late. Trying to get out of the clock penalty via another rule is clearly not the intent of the rule book.
I have a feeling that the TD made a decision you don't agree with (if he did, I disagree with him too), but I could very well be wrong. What did the TD say?

I finally found it! Rule 39A1. "If Black is late for the start of a round, White makes the choice of equipment."

Last tournament I played in, I took a pawn en passant. My opponent, undoes my move, puts his pawn back, says "I'm a little fuzzy on the en passant rule", and starts my clock back. I paused the clock, made the move again, and said, "Then you need to talk to the TD, not undo my move." It was funny, but I'm not sure what he expected. I guess he wanted me to make a different move. Anyway, the TD straightened him out.
Years ago I was playing in a tournament in NC and showed up to the game on time and it was time for the round to start. I was assigned the white pieces in this round and my opponent wasnt present. I set up my board and pieces and clock, sat down made my move and started my opponents clock. A few minutes later my opponent showed up and started putting my set aside in order to use his pieces and he told me as the player with the black pieces he was the one who gets choice of equipment by uscf rules. I said he was right BUT if you are late to the game that right/choice is forfeited and the player who is on time gets to choose. He didnt agree and went to get the TD while I waited....... who was right ? Was I right or my opponent ?