After the Great Fire of 1901 destroyed most of downtown Jacksonville, Andrew Carnegie offered the city $50,000 to build a new library, provided the City had a site and appropriated $5,000 a year for library support. The City agreed and in 1905 the Jacksonville Free Public Library opened on the northeast corner of Ocean and Adams streets. The building was designed by architect Henry John Klutho in the Neo-Classic Revival style, with four columns on the facade. Today, it houses a law firm and is on the National Register of Historic Places.

By 1960, the library was too small to serve a community of 455,000 people and the City approved building a new library, to be located diagonally across from the Carnegie library, on the southwest corner of Ocean and Adams streets. To make room for the new building, the City demolished its old City Hall — a classic structure with a great dome, also designed by Henry Klutho.

The library trustees commissioned Hardwick & Lee, Architects, to design the new library.