Are you running on a Mac? Perhaps there are no system fontconfig caches?This seems precisely to be the problem. I do have a ~/.fontconfig directory containing cache files. When I delete it, the performance of the command line script is just as slow as the cgi script until the next time the command is run (e.g. after the cache is remade). So I guess the question becomes: how do I get apache and the CGI script to see and use the .fontconfig caches?
Try updating the fontconfig cache as root:
sudo fc-cache
Or try the same command as the user running the cgi-bin programs (apache?)
John