The map is tough for every spot except maybe red and green (on the minimap), but I find that the main factor is what other players do. Basically those who encounter resistance early from more than one side lose. It becomes sort of a "luck of the draw" scenario, where you just hope to get lucky
Pillager, I don't find the island player to be a slow start. They have a strong advantage in mobility, and they're not subject to encountering early resistance to expansion. But this map generally doesn't play out like a balanced map, and I find it hard to pin down any specific cause beyond the erratic layout. Maybe if you played 5-10 games with Fog of War turned off you could gauge what is happening and work to balance the map with unit availability and income? There are also many, many map editor options now that would allow you to fine-tune the map without compromising its design at all.