To take part in that fantasy when you have a big penis yourself, requires a search for bigger dicks than your own, or at the very least ones of equal size.

There was some recent polling done, indicating that 1 in 5 gay men have rejected a hook-up based entirely on penis size. So yes, to an extent, well endowed gays (especially hung bottoms\versatiles) may tend to seek out big gay dicks.