The "so" at the beginning of the line means "just as," which means that in line, the speaker is comparing mankind's fall from the Garden of Eden to the change from gold to green he described in the first half of the poem.
Just like the flowers subsided, or were downgraded to become leaves, Eden also sinks. This refers to the biblical fall of Adam and Eve from the Garden of Eden. When Eve ate forbidden fruit, she and Adam, and all their descendents, were forever punished by being banished from Eden and subject to all of the grief that we humans know today.