No, I think there are valid and invalid reasons. Not wanting to play another game period, or having something else to do, sure. My only point is that in a percentage of players who decline rematches that were otherwise going to play again anyway, are doing so out of cowardice. My only point is that it is, in some circumstances, one of the psychological factors in declining.
At amateur level chess is played for fun. Fun is a 100% psychological factor. How comes that declining a rematch for psychological reasons is not a valid reason for you?
I personally don't accept rematches most of the time, but when I do, I think it's good practice to play at least 5 games with that person if you have the time. I think that rematches are good for developing a larger opening repertoire, but it's perfectly fine and not cowardly to refuse a rematch. Sometimes you just don't want to commit to chess for a certain duration.