By 1880 white settlers had flooded Woolsey, steadily pushing its native Creek Indian residents out of the area. Soon after this first influx of settlers, Woolsey grew into a prosperous town, offering a school, churches, a post office, farms, plantations and other businesses.
Woolsey, residents say, remembers a prominent doctor who served in the Confederacy during the Civil War.