MacDonald and his wife, feminist and social reformer Margaret Ethel Gladstone, wanted to build The Hillocks on a hilltop site in Lossiemouth.
The plot was near a quarry where MacDonald had collected fossils in his youth.
But Mrs Kielhorn said some people already living in the area were unimpressed by his political views and upbringing.
She said her grandfather was told to build his house in “the slums” among the town’s fisherfolk.
The Hillocks was constructed in Moray Street, near Lossiemouth’s Seatown fishing community.
The house was originally built for MacDonald’s mother Anne so she could look after his five children while he and Gladstone were on political tours, including of Australia.
The cost of building the house, about £400, was paid for by Gladstone.
Mrs Kielhorn never met her grandmother. Gladstone died in 1911 from blood poisoning.
Following his own death – of heart failure aged 71 and while at sea on an ocean liner – MacDonald was buried alongside his wife at a graveyard in Spynie, near Lossiemouth.
The house has remained in the MacDonald family ever since.