Clinton College was first located in a frame building on Academy Street, between North Broad and Elizabeth Streets in downtown Clinton. The building had been used for the Clinton Female Academy as early as 1857. Starting in 1872, it was used as the Clinton High School. When the High School Association became the Clinton College Association in 1880, the building was used for the first college classes.

In 1886, Clinton College (then Presbyterian College of South Carolina) moved to a new building on the present-day Thornwell Orphanage campus, and the old frame building was given to the school board around 1899. It was moved by 1902 to an unknown location, and the Academy Street School was built on the site.