James was married to Mary Robertson Walsh (1810–1882), the sister of a fellow Princeton seminarian, on July 28, 1840, by the mayor of New York, in his bride's house on Washington Square. The couple lived in New York, and had five children: William (1842), Henry Jr. (1843), Garth Wilkinson (1845), Robertson (1846), and Alice (1848). Three of his children: William James, Henry James, and Alice James, have been of interest to posterity as writers.