Could not resolve host name error in psycopg2

[derwin theduck <hotchipsveg@xxxxxxxxx>]

We have been getting this error intermittently (about once a week) in Django with channels since switching from a local database server to a hosted one:

could not translate host name "timescaledb" to address: Name or service not known

Connecting directly to the server with psycopg2 in the python interpreter works ok:

>>> import psycopg2 as pg
>>> pg.__version__
'2.7.7 (dt dec pq3 ext lo64)'
>>> conn = pg.connect(host='timescaledb', database=*, user=*, password=*)        >>> conn.server_version
110005

The resident database / network expert maintains that everything is ok on the networking side, and that django, once it establishes a connection, should not be attempting to reconnect.

The problem happens not on startup, but in the middle of the day after processing transactions without any issues for several hours.

Any idea how to go about troubleshooting / fixing this? :'(

 (  File "/home/dduck/.local/lib/python3.7/site-packages/django/db/models/query.py", line 276, in __iter__)
 (    self._fetch_all())
 (  File "/home/dduck/.local/lib/python3.7/site-packages/django/db/models/query.py", line 1261, in _fetch_all)
 (    self._result_cache = list(self._iterable_class(self)))
 (  File "/home/dduck/.local/lib/python3.7/site-packages/django/db/models/query.py", line 57, in __iter__)
 (    results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size))
 (  File "/home/dduck/.local/lib/python3.7/site-packages/django/db/models/sql/compiler.py", line 1135, in execute_sql)
 (    cursor = self.connection.cursor())
 (  File "/home/dduck/.local/lib/python3.7/site-packages/django/utils/asyncio.py", line 24, in inner)
 (    return func(*args, **kwargs))
 (  File "/home/dduck/.local/lib/python3.7/site-packages/django/db/backends/base/base.py", line 260, in cursor)
 (    return self._cursor())
 (  File "/home/dduck/.local/lib/python3.7/site-packages/django/db/backends/base/base.py", line 236, in _cursor)
 (    self.ensure_connection())
 (  File "/home/dduck/.local/lib/python3.7/site-packages/django/utils/asyncio.py", line 24, in inner)
 (    return func(*args, **kwargs))
 (  File "/home/dduck/.local/lib/python3.7/site-packages/django/db/backends/base/base.py", line 220, in ensure_connection)
 (    self.connect())
 (  File "/home/dduck/.local/lib/python3.7/site-packages/django/db/utils.py", line 90, in __exit__)
 (    raise dj_exc_value.with_traceback(traceback) from exc_value)
 (  File "/home/dduck/.local/lib/python3.7/site-packages/django/db/backends/base/base.py", line 220, in ensure_connection)
 (    self.connect())
 (  File "/home/dduck/.local/lib/python3.7/site-packages/django/utils/asyncio.py", line 24, in inner)
 (    return func(*args, **kwargs))
 (  File "/home/dduck/.local/lib/python3.7/site-packages/django/db/backends/base/base.py", line 197, in connect)
 (    self.connection = self.get_new_connection(conn_params))
 (  File "/home/dduck/.local/lib/python3.7/site-packages/django/utils/asyncio.py", line 24, in inner)
 (    return func(*args, **kwargs))
 (  File "/home/dduck/.local/lib/python3.7/site-packages/django/db/backends/postgresql/base.py", line 185, in get_new_connection)
 (    connection = Database.connect(**conn_params))
 (  File "/usr/local/lib64/python3.7/site-packages/psycopg2/__init__.py", line 130, in connect)
 (    conn = _connect(dsn, connection_factory=connection_factory, **kwasync))
 (  could not translate host name "timescaledb" to address: Name or service not known)