Absolutely, and the technology to do so has been in use since the 90s. Concentrated solar power (CSP) heats up molten salt and stores it in a tank, allowing it to be fed to the turbines not only as base load, but also as dispatchable or load following plants. The only achilles heel is that CSP plants must be located where there is plenty of blue sky. But on the flipside, such areas are called deserts and there happens to be a lot of space there that are not used for anything else. And about 80% of the world's population live in countries where there are suitable resources for CSP, so this is a most relevant and highly scalable solution.
Solar Two in the Mojave desert started operating with storage in 1995, and was decommissioned in 2009.
As to the cost of baseload CSP, this is simply a function of scale. With more deployments, the cost invariably will go down as demonstrated with PV over the last few years. By 2020, the US DOE SunShot program has a goal of reaching 6 cents/kWh, which would make it cheaper than coal.