GISextensions add functionality directly to a particular software. For instance in the proprietary world, ESRIs Spatial Analyst
Extension adds functions for (raster) analysis to ESRIs ArcGIS Desktop product.
Plug-insact in the same manner, but they typically
add a narrow set of abilities, e.g.
one particular raster analysis method, whereas an extension can add a wider set of new abilities,
e.g. a toolbox of raster analysis methods.
For that reason in OpenJUMP an extension is actually considered a collection of plug-ins.
However, we could not find a reference in the literature that sufficiently explains the difference between plug-in and extension.