The basic gist of this poem is that there are many beautiful and perfect sights, moments, experiences and memories in life, but, they are pretty short-lived. The rest of the time, life is pretty dull and drab; however, there are those moments of beauty and perfection that we have to draw on during the down times. Johnson compares these moments to "a lily of a day" and how it is much "fairer in May" even though it lives at other moments too, and "it fall and die at night." While it was alive though, it was a "flower of light." It, while it was alive and in its prime, a beautiful and inspiring lily. This is just like our lives--he concludes with the idea that "in short measures life may perfect be," meaning, life is perfect in only short, transient moments that often fade and die.
He contrasts our lives with their temporary perfect moments to a tree that is perfect and beautiful for the duration of its life, for "three hundred year" before it falls "at last" to become an ugly log. So, "It's Not Growing Like a Tree," refers to how our lives are not constantly beautiful, with perfect moments consistently growing, for centuries. Rather, we are more like the lilies, who have brief but exultant lives of moments of beauty and perfection.
I hope that these thoughts help you to understand the poem a bit better; good luck!