According to tradition, Arlington Mansion was constructed by John Hampton White and his wife, Jane Surget White, residents of Elizabethtown, New Jersey. The exact date when construction began remains uncertain, but scholars estimate it to be around 1819-1820.

The identity of the architect responsible for Arlington Mansion is unknown. While it has been historically attributed to John Hampton White, there is no existing documentation to confirm this claim. What is known is that Lewis Evans, a wealthy planter from Natchez, acquired the land upon which Arlington stands in 1806. He established a plantation and built a house there. In 1814, he sold a portion of the plantation, including the house, to Jonathan Thompson, a land speculator. Thompson later sold the Arlington property to Mrs. Jane Surget White in December 1818. Jane White was the daughter of Pierre Surget, a French immigrant from one of Natchez’s prominent families. John Hampton White passed away on October 15, 1819, during a yellow fever epidemic, while Jane White died on July 1, 1825.

Architectural historians consider Arlington Mansion to be one of the four significant Federal Style villas that set the standard for later antebellum homes in Natchez. Situated atop a gentle natural hill, the red brick house consists of two full stories above a low basement. The main block measures 64 by 87 feet (20 m × 27 m).

The front facade is five bays wide, with the central three bays adorned by a tetrastyle portico featuring four prominent Doric columns. The front entrances on both levels are embellished with radiating brick voussoirs, carved marble impost blocks, and keystones. Each opening is flanked by two sidelights with decorative muntins, and a fanlight crowning a central door. All window openings are adorned with carved marble sills and lintels.

Inside, the main levels are divided by a central hall plan, with two rooms on each side. The eastern side features a stair-hall, connected to the main hall by a fanlight-topped entrance. The interior showcases refined Federal style woodwork, with the carved overdoor panels being particularly noteworthy.

Service wings are located at the rear of the main house. At the southeastern corner, there is an attached one-story brick wing, fronted by a porch with slender turned columns. Adjacent to this wing is a detached two-story brick outhouse, serving as the kitchen and chef’s quarters. Attached to the rear wall of this structure is a one-story brick carriage house. A two-story barn is situated approximately 150 feet (46 m) west of the rear courtyard, separated by a geometric boxwood garden.

During the mid-19th century, a full-width grand veranda was added to the rear, featuring Doric columns that match those at the front, along with Greek Revival woodwork below. At this time, a cast iron balcony was installed above a central doorway leading to the stair-hall on the east side of the house, and the attached wing underwent renovation.

Arlington Mansion suffered a devastating fire on September 17, 2002, which destroyed the main roof and a significant portion of the second floor. Shortly after the fire, the Historical Natchez Foundation undertook efforts to construct a new roof. Subsequently, vandals damaged numerous windows and defaced or destroyed much of the interior and exterior woodwork. Following a lawsuit filed by the Natchez Preservation Commission, the absentee owner was fined on December 2, 2009, in the local court of Natchez after being convicted of demolition by neglect.