The original structure built by Indian Muslim traders in 1748 was a wooden building and in 1872, it was rebuilt in brick. The mosque is one of the traditional mosques in Melaka, which still retains its original design. The architectural design of the mosque is a cross between Sumatran, Chinese, Hindu, and the Melaka Malay. The minaret, ablution pool and entrance arch were built at the same time with the main building. The kampung kling mosque is named based on the place where Indian traders dwell in that place called Kampung Kling.
The minaret resembles a pagoda. The mosque also has a blend of English and Portuguese glazed tiles, Corinthian columns with symmetrical arches in the main prayer hall, a Victorian chandelier, a wooden pulpit with Hindu and Chinese-style carvings, and Moorish cast iron lamp-posts in the place of ablution for pre-prayer cleansing.
The Department of Museums and Antiquities completed conservation works on the mosque in the 1990s.
The mosque should not be confused with Kampung Hulu Mosque, located a few blocks to the north; it also has a pagoda-like minaret.