There is still a small community living just outside palace grounds. A few old Malay mansions built in the early days are remnants of the pride and joy of the early sultanate, lay scattered around the area.