If the position has a clear idea and I know what's going on (I'm comfortable in the position basically) then I kind of like music, it helps my calculation chug along.
If it's a difficult position though, and I need to concentrate to survive basically, then music definitely hurts my play.
So music is almost always better for bullet, because you have no time to dig into the position, you just need your calculation to chug along and find ok moves quickly. So I agree with you, and don't use music in my long games such as tournaments.
So, over the last couple of weeks I've been experimenting to see if music would help me to play better or if it destroyed any hope of winning for me. I tried both "Blitz" (G/3) and "Classical" (G/120) time controls.
I tried different types of music, anything from "Rap" to "Mozart".
I found that while listening to rap it became harder to concentrate and keep a "balanced pace" throughout the game.
On the other hand, "Classical" music such as Mozart and Beethoven seemed to put my mind in a "rhythm" and at times helped me concentrate and keep my mind on the game.
However, overall I played better without the music.
But those are just my results. Post your thoughts in the comments below. :)