This will sound combative, but please don't take it that way: How is it more difficult? It is the hub of every greater metro area transportation system. Barring it being in one's outer-borough backyard (which would itself be applicable only to a population orders of magnitude smaller than any "backyard" within the city proper), Manhattan is the easiest place to get both with regards to frequency and options of mode. Parking would be expensive, But if you make the choice you can pay the price IMO. Otherwise, do the world and yourself a favor and be a good metro citizen: park and ride via one of the 5 gazillion trains or buses.
My perception of the reason there hasn't been another pro sports venue has little to do with what you're suggesting. I think it comes down to 2 (hugely difficult to overcome) issues: 1) finding suitable ground space that 2) meets whatever financial parameters are in play for the prospective developer/tenant/financiers.
Financial concerns aren't the only reason why there aren't other stadiums in Manhattan. They are many reasons why, most of which I've already mentioned.
Anyway, in my opinion the other boroughs make more sense. Manhattan doesn't have the space, and can't accomodate people who drive in from outside the city.