As suggested in the Postgres slack channel by lukasfittl, I disabled hashagg on my old server, and ran the query again. That changed one piece to a groupagg (like was used on the new server) and the performance was similar to the 9.6.1 box.
9.5.5 w/ hashagg disabled: https://explain.depesz.com/s/SBVt
