The House of Memories was constructed by Judge Hans A. Anderson and his wife, Oline, in 1931, and dedicated to the memory of their recently-deceased daughter, Hazel (Anderson) MacMurray. Containing Judge Anderson's extensive personal collection of historical relics and documents, it was gifted to the city then and its contents subsequently expanded by donations of antiques and papers by others. The city deeded the museum over to Trempealeau County in 1936, but for much of the 1940s it was closed to the public and not maintained. There were attempts in the late 1940s to to reopen it and reconstitute the Trempealeau County Historical Society, which Judge Anderson had helped found in 1910. In 1952, the county board voted to deed it back to the heirs of Judge and Mrs. Anderson, who both had passed away in 1939. The contents of the museum had been dispersed to several locations or returned to the the heirs and other donors. The Andersons' heirs then sold it to Whitehall Lodge No. 271, Free and Accepted Masons, and its appendant bodies in 1954. The Masons used the building until 2003, when Lodge No. 271 surrendered its charter and was merged into the Galesville Lodge. The 72-year-old structure, located across Main St. from the county courthouse, was then demolished and replaced by an office building.