Raffles was born on the ship Ann off the coast of Port Morant, Jamaica, to Captain Benjamin Raffles (d. June 1797) and Anne Raffles (née Lyde). His father was a Yorkshireman who had a burgeoning family and little luck in the West Indies trade during the American Revolution, sending the family into debt. The little money the family had went into schooling Raffles. He attended a boarding school. In 1795, at the age of 14, Raffles started working as a clerk in London for the British East India Company, the trading company that shaped many of Britain's overseas conquests. In 1805 he was sent to what is now Penang in the country of Malaysia, then called the Prince of Wales Island, starting his long association with Southeast Asia. He started with a post under the Honourable Philip Dundas, the Governor of Penang.
He was appointed assistant secretary to the new Governor of Penang in 1805 and married Olivia Mariamne Fancourt, a widow who was formerly married to Jacob Cassivelaun Fancourt, an assistant surgeon in Madras who had died in 1800. At this time he also made the acquaintance of Thomas Otho Travers, who would accompany him for the next twenty years.