Yes, actually deleting the folder is exactly what you should do. A "pure" VSIX extension is just a unzip of the files to a folder under
It's a good idea to check though that it wasn't installed through MSI or some other installer technology by verifying in Windows Add/Remove Programs.