I had a "weird" instance like this recently (perhaps not your exact problem).
I was developing a tube screamer circuit, and had a basic TS circuit on my breadboard (see below). I was noticing that when my volume was fully off, there was still some signal coming through, as if it didn't turn off! I checked every connection and it seemed OK. Then I noticed...there is a 47 uF cap to ground on the power section where the reference 4.5V supply is formed (as in the schematic). The volume control is connected to this reference voltage. I had forgotten to add this cap to my breadboard, and when I did add it, the problem went away. Apparently that cap is there to prevent signal oscillations from affecting the reference voltage source (effectively damping them out).
So, yes the power supply components are very important, and making sure you have stable supply and reference voltages is key to reducing noise and unexpected effects such as the one I encountered. I also believe that some of the high pitched oscillation you get with some high gain distortions may be due to a combination of layout and poor power filtering. Perhaps for my own designs I may experiment with using an op amp to supply a stable reference voltage versus a simple resistor voltage divider. Perhaps overkill, but...