Gartner's duct cysts develop on the side walls of the vagina. This duct is present while a baby is developing in the womb but will most often disappear after birth. If parts of the duct remain, they may collect fluid and develop into a vaginal wall cyst later in life.