The first zoo penguin, a king penguin from the Falkland Islands, arrived in England and was put on display in 1865. Lovecraft very much could've seen a live specimen.
https://www.londonzoo.org/zoo-stories/blog/penguins-through-...