In 1888 a client, Mary Morstan, comes with two puzzles for Holmes. The first is the disappearance of her father, British Indian Army Captain Arthur Morstan in December 1878. According to Mary, her father had telegraphed her upon his safe return from India and requested her to meet him at the Langham Hotel in London. When Mary arrived at the hotel, she was told her father had gone out the previous night and not returned. Despite all efforts, no trace has ever been found of him. Mary contacted her father's only friend who was in the same regiment and had since retired to England, one Major John Sholto, but he denied knowing her father had returned. The second puzzle is that she has received six pearls in the mail from an anonymous benefactor, one per year since 1882 after answering an anonymous newspaper query inquiring for her. With the last pearl she has received a letter remarking that she has been a wronged woman and asking for a meeting. Holmes takes the case and soon discovers that Major Sholto had died in 1882 and that within a short span of time Mary began to receive the pearls, implying a connection. The only clue Mary can give Holmes is a map of a fortress found in her father's desk with the names of Jonathan Small and three Sikhs named Dost Akbar, Abdullah Khan, and Mahomet Singh.
Holmes, Watson, and Mary meet Thaddeus Sholto, the son of the late Major Sholto and the anonymous sender of the pearls. Thaddeus confirms the Major had seen Mary's father the night he died; they had arranged a meeting to divide a priceless treasure Sholto had brought home from India. While quarreling over the treasure, Captain Morstan — long in weak health — suffered a heart attack. Not wanting to bring attention to the object of the quarrel to public notice, Sholto disposed of the body and hid the treasure. However, he himself suffered from poor health and an enlarged spleen (both due to malaria, as a quinine bottle stands by his bed). His own health became worse when he received a letter from India in early 1882. Dying, he called his two sons and confessed to Morstan's death and was about to divulge the location of the treasure when he suddenly cried "Keep him out!" before falling back and dying. The puzzled sons glimpsed a face in the window but the only trace was a single footstep in the dirt. On their father's body is a note reading "The Sign of Four". Both brothers quarreled over whether a legacy should be left to Mary Morstan, and Thaddeus left his brother Bartholomew, taking a chaplet and sending its pearls to Mary. The reason he sent the letter is that Bartholomew has found the treasure and possibly Thaddeus and Mary might confront him for a division of it.
Bartholomew is found dead in his home from a poison dart and the treasure is missing. While the police wrongly take Thaddeus in as a suspect, Holmes deduces that there are two persons involved in the murder: a one-legged man, Jonathan Small, as well as another "small" accomplice. He traces them to a boat landing where Small has hired a launch named the Aurora. With the help of his dog Toby and the Baker Street Irregulars and his own disguise, Holmes traces the launch. In a police launch Holmes and Watson chase the Aurora and capture it, but in the process end up killing the "small" companion after he attempts to kill Holmes with a poisoned dart shot from a blow-pipe. Small tries to escape but is captured. However the iron treasure box is empty; Small claims to have dumped the treasure over the side during the chase.
Small confesses that years before he was a soldier of the Third Buffs in India and lost his right leg in a swimming accident to a crocodile. After some time, when he was an overseer on a tea plantation, the Indian Rebellion of 1857 occurred and he was forced to flee for his life to the Agra fortress. While standing guard one night he was overpowered by two Sikh troopers who gave him a choice of being killed or being an accomplice to waylaying a disguised servant of a Rajah who sent the servant with a valuable fortune in pearls and jewels to the British for safekeeping. The robbery and murder took place and the crime was discovered, although the jewels were not. Small got penal servitude on the Andaman Islands, and after twenty years he overheard that John Sholto has lost money gambling. Small sees his chance and makes a deal with Sholto and Arthur Morstan: Sholto will recover the treasure and in return send a boat to pick up Small and the Sikhs. Sholto double-crosses both Morstan and Small and steals the treasure for himself. Small vows vengeance and escapes the Andaman Islands with an islander named Tonga. It was the news of his escape that shocked Sholto into his fatal illness. Small arrives too late to hear of the treasure's location but leaves the note which refers to the name of the pact between himself and his three Sikh accomplices. When Bartholomew found the treasure Small planned to only steal it but claims a miscommunication led Tonga to kill him as well.
Mary Morstan is left without the bulk of the Agra treasure, although she will apparently receive the rest of the Chaplet. John Watson falls in love with Mary, and we learn at the end that he has proposed to her and she has accepted.