As the flame flows down the tea bag, hot air starts to accumulate within the cylinder. Eventually, this hot air concentrates in the bottom of the cylinder as the flame reaches the bottom of the tea bag.
Because the air molecules in the hot air are relatively spread out (less dense), they push up through the cylinder toward the colder (more dense) air, creating a current. The force of this current isn't enough to lift the tea bag until it's almost completely burned. Once the ashen tea bag skeleton burns thoroughly, it becomes light enough for the convection current to launch it into the air.