Maybe not so much, because the only performance acceptable solution that comes to my mind is keeping the x*y array like it's now but marking the planquadrats that are outside the map somehow. This whould ofc save memory because that positions won't generate extra data structures, but will reside memory too, even it's low.
The only way to get around this whould be using some bi-dimensional hash structure to store the tiles, to allow constant time access to the tiles, and those nodes will have to be linked also to adjacent tiles to allow for iterative map processing (or maybe not). The gaps whould in that case not ocuppy memory in that case, but will imply recoding most of the simutrans inner functions, I'm quite sure this task whould take lots and lots of coding, maybe with little advantage compared with current implementation.
Introducing this whould also create lots of problems designing how the player can activate that disabled areas for playing, to layout them (they can default to water level, shore, be random...), this has also to be implemented in the network multiplay, re-design the mini-map, the vehicles routing... It's quite a big change, and I don't personally think it's worth the work it requires, atm. I'd focus other aspects of the game that need to be improved imho.