Can my opponent move my chess piece if it was an illegal move?
I don't believe that he should touch/move my chess piece.
Probably not, technically, but if all he does is to put the piece back where it started, I can see no harm.
Now on the other hand, he were to tip over your king, that would be going too far.
/ You didn't say if you were talking about an over the board slow tournament game or a blitz game and what rules you think should be applied.
Since USCF rules aren't available online, let's look at FIDE. It says the position before the illegal move should be reinstated. The rule doesn't demand that one side or the other do the reinstating. Or if the arbiter should be involved.

Therefore, touching your piece in order to move it back seems reasonable.
You made the illegal move. You ought to cooperate and help get things straight.
Can my opponent move my chess piece if it was an illegal move?
I don't believe that he should touch/move my chess piece.