Southwest Harbor is a town in Hancock County, Maine, United States. Located on Mount Desert Island, the population was 1,764 at the 2010 census.