In 1955 Duffy began freelancing as a fashion artist for Harper's Bazaar where he first came into contact with commercial photography. Inspired by the photographic contact sheets he saw passing through the art director's desk he sought a job as a photographers assistant. Unsuccessfully, he applied for a job with John French and was subsequently employed at Carlton studios and then at Cosmopolitan Artists. Duffy went on to work as an assistant to the photographer Adrian Flowers and whilst working for Flowers he received his first photographic commission from Ernestine Carter, who was the fashion editor of The Sunday Times.