The building housed all of the club's athletic activities, as well as living and dining spaces. The 35-story Downtown Athletic Club building was designed in the Art Deco style by Starrett & van Vleck, who also designed the adjacent 21 West Street at the same time. Following the September 11 attacks on the nearby World Trade Center, the surrounding neighborhood was blockaded, and the club closed down, having filed for bankruptcy due to the clubhouse's inaccessibility. The club started admitting female members in 1977, and after facing further financial troubles in the late 1990s, sold off part of its building. The building was sold off in 1936 following the club's bankruptcy, but was reacquired in 1950. The club bought land for their building near the Hudson River in 1927 and completed the structure in 1930. The Downtown Athletic Club was founded in 1926 as an all-male club.

The Downtown Athletic Club was known for issuing the Heisman Trophy, an annual award for outstanding college football players that was named after John Heisman, the club's first athletic director. The Downtown Athletic Club, also known as the Downtown Club, was a private social and athletic club that operated from 1926 to 2002 at 20 West Street, within the Financial District of Lower Manhattan in New York City.

City sports club hours