The Flagstaff House was built in the Greek Revival style. Like many other colonial buildings, however, its roof was constructed with local materials in Chinese style, as transporting tiles all the way from Europe was costly and inefficient. Over the years, alterations to the architecture of the building were made to accommodate the Hong Kong climate and to the building's changing needs.

The most significant of all was the addition of verandahs in the 1860s.

A verandah is a light, external, open gallery or porch built around the central structure of a building. Due to the sub-tropical climate of Hong Kong and the lack of modern air-conditioning equipment, buildings heated up rapidly in the summer, and often suffered from humidity during spring. The verandah was therefore a functional installation to the building. It provided insulation, blocking the sun from directly shining on to the building and keeping the temperature of the building and its interiors stable. Though a relatively less common feature of Western architecture, verandahs are regarded as a useful feature for natural cooling and sunshade. It was in effect an early form of natural air conditioning.

Of the six buildings covered on this website, the Legislative Council building, the Kom Tong Hall, and some buildings in the Central Police Station Compound all have verandahs.

In 1981, the Flagstaff House was put under the management of the Urban Council, with the intention of converting the building into a museum devoted to Chinese tea culture.

Conversion works went underway shortly thereafter, with three major objectives.

The first was public safety. The building had to be structurally reinforced for opening to the public and refurbished to satisfy relevant building regulations.

The second was museum needs. Humidity, for instance, is a major concern for any museum, as it could affect the conditions of the exhibits. A second layer of transparent material was therefore added to the windows to reduce the amount of humidity entering the building, air-conditioning, and other amenities were also installed for humidity and temperature control.

The third, and perhaps the most significant objective architecturally, was to restore the building to its original mid-19th-century appearance as much as possible, so that its exemplary architectural and interior qualities could be preserved and displayed.

For example, the main staircase was removed and the two staircases at the two ends of the building were restored. The front porch was also removed.

The Flagstaff House Museum of Tea Ware officially opened its doors in 1984. The building was declared a monument by the Antiquities and Monuments Ordinance in 1989.

With an additional donation of rare Chinese ceramics and seals to the Urban Council in 1994, the original space in the Flagstaff House became insufficient for use. An extension to the main building was constructed the same year, becoming the K.S. Lo Gallery to house the donated items. The Gallery matched the outlook of the Flagstaff House for its exterior design, while its interiors followed traditional Chinese-style design.