However, your stop is at Sag Harbor, one of the island's oldest towns. Settled sometime in the early 18th century by English colonists, Sag Harbor was a whaling town and the first port of entry ...