To answer tigger's question, yes, 'vulgo' exists in English, but like many less well known Latin terms, it's nearly archaic. I had to go to the OED to find it in a dictionary, which translates it as 'commonly' or 'popularly.'
My (fuzzy) recollection is that it was used mainly in formal or academic writing, mainly in the days when all educated English speakers also knew Latin (namely, the 19th century and before). Some modern academics might still understand it, but not all, and certainly not the general public (say, tourists).
And the larger problem here, as RE1 and tigger point out, is that the whole custom of calling not only the house but also generation after generation of its occupants by a historic family surname that is no longer their real surname has to be explained to outsiders anyway. (Does it really also apply to Saba's mansions, or more farmhouses or village houses?)
If it's really only about the houses and not the people or the town, CM2DD's suggestion 'known as the Hofer mansion' is surely fine. If the readers need to know more about the background, I suspect Saba will just have to write a footnote or something. After the explanation, something as short as 'commonly' or 'popularly,' or either of those with 'called' or 'known as,' should be okay.