The effect of silica on the glass transition and mechanical properties of PET was studied using MD simulations. The results of the density–temperature curve, the specific volume versus temperature for the nanocomposites in the glass transition process shown that adding silica decreased the Tg of PET and adding hydroxylated silica increased the Tg of PET. It was found that adding silica can increase the mobility of the polymer chains, decrease the free volume of the system and decrease many kinds of energy of the system, so the glass transition was determined by the balance of these effects. The addition of the hydroxylated silica cause stronger mechanical interlockings between the silica and the polymer molecules, which is a neg- ative effect on molecular chain mobility, so Tg increases. Furthermore, the addition of a rigid silica material and a stronger interaction between the modified silica and polymer chains resulted in the higher mechanical properties of nanocomposite in comparison with those in the pure PET system. In conclusion, molecular dynamic simulation may be applied to calculate the Tg and mechanical properties of realistic complex system. This method will provide help for the materials design and processing.