Any point to a double-ground Golem?

No. TM the Golems to double rock and focus on ground Rhydons. Rhydon is much stronger than Golem and Golem is the only relatively strong Mon that can learn double rock. Unless Rhydon candy is extremely rare and Golem candy is plentiful.

Compared to Rhydon, Golem has more defense while Rhydon has a little more attack and a lot more HP (and CP). But if you want variety and not use 6 Rhydons, its still a solid choice for the eventual Raikou and Entei, and possibly pure electric type raid bosses.

Apart from slightly worse attack stat than Rhydon, is there any other reason why double ground Golems get so much hate?
I don't think slightly worse stats can justify double TMs, if you happen to roll double ground.

As has been mentioned above, double ground golem is a slightly worse version of double ground rhydon.

The only use for the double ground moveset will be against any pure electric type mons such as the raid boss Raikou or Electabuzz (since it is their only weakness).

once the birds are gone might struggle to find a use for double rock golems as well

Double ground golem is good for pure electric, or pure fire. Rhydon is better in this role. I am keeping mine only because I don't have 6 double ground Rhydons. It is reasonable to change it to double rock, as the only other marginally servicable double rock user is Sudowoodo (emphasis on marginally).