I hadn't really thought much about it but my understanding has increased each time I've been asked and struggled to get across my views on it. I think I'm now ready to try and give a full answer here...
I'm going to make a connection with our primaeval past and primaeval instincts and emotions as the main parts of my answer. To do that I need first to make some observations that will stimulate the right ideas in your mind.
Primaeval InstinctsThe most basic primaeval instincts are fear and the need to procreate. That's one strand. The other is that at night, in dim and dark places we don't see colour - the rods and cones in our retinas are designed for colour vision when there's plenty of light around and black and white when there's little light around.
Procreation InstinctAn observation - when dating has gone so far along its course, we go out for a meal in a romantic, intimate, low-light setting. Ever wondered why the need for low light and candles? Whatever the reason, isn't that a direct connection with our primaeval procreation instinct?
Fear InstinctFear is heightened at night / in low light, when you can't really see any threats that might be around. Have you and some mates ever 'talked' yourself into a right case of the willies when in a spooky place?
In both these situations, a common factor is that you are seeing in (or close to) black and white. So doesn't it make sense that a photo / image / film in black and white triggers or stimulates at least a little, those more primaeval feelings? When this happens they are nearer the conscious 'surface' and therefore flavour whatever else is going on in your mind when viewing the work.
In a nutshell...So in a nut shell, our primitive subconscious associates black and white with low-light and the possibilities of fight, flight or procreate (hah! managed to avoid the alliterative verb so as not offend any sensibilities!). And when we see an image in black and white it makes those primitive feelings come to the surface to 'colour' our emotional response to the image.
So what do you think? Do you see the connection? Is it a (fool /) full-ish answer?