This is not the red rock sandstone of the desert southwest, but the sand and clay of Georgia.  The canyons of Providence Canyon State Park did not form over millions of years like the canyons of the southwest, but over the course of a hundred years.  

When the early 19th century settlers cleared the area for farming, perhaps unaccustomed to farming on slopes, they ran their rows up/downhill.  The furrows of their fields made for perfect drainage channels, helping water running downhill to cut through the top clay layer to the soft sand layer below and setting off a rapid erosion process.  Within 50 years, the cuts were 3-6 feet deep and within 100 years the land was useless for farming.

The sand of Providence Canyon was laid down 65-70 million years ago when seas were higher and this area was likely a chain of barrier islands.  The sand was protected from erosion by a layer of clay laid down 55-65 million years ago.  Iron content in the clay contributes to both the red colors of the sand as well as an iron-rich layer between the clay and sand.  Look for iron nodules in the runoff from the canyon edge.