Bad Fuel Pump?

[41M3]

Hi All,
I have a 03 e39 530i 73k miles. Love it!
I've started experiencing some strange starting problems. On cold start my engine fires right up. After I drive, or the car idles for a while during warm up ( I live in CT) and I turn off, I can't seem to start back up. I then have to wait an hour or so, then I'll go to start and it fires right up. I've been thinking it's the fuel pump but I am second guessing myself as the car starts up in the morning just fine or after sitting for a while. But if I go to get gas and turn the car off at the pump, I can't start it back up immediately. No performance issues at all. Have any of you experienced this? Thanks in advance!