Search Postgresql Archives

Re: Testing of a fast method to bulk insert a Pandas DataFrame into Postgres

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hello, Adrian Klaver,
What is the robust way to upgrade Pandas?
Regards,
David

On Monday, 4 October 2021, Adrian Klaver <adrian.klaver@xxxxxxxxxxx> wrote:
On 10/4/21 9:20 AM, Shaozhong SHI wrote:
Hello, Adrian Klaver,

Pandas version is 0.23.0.

The reason the below does not work is method did not show up until pandas 0.24.0.


I used the following code:

def psql_insert_copy(table, conn, keys, data_iter):
     # gets a DBAPI connection that can provide a cursor
     dbapi_conn = conn.connection
     with dbapi_conn.cursor() as cur:
         s_buf = StringIO()
         writer = csv.writer(s_buf)
         writer.writerows(data_iter)
         s_buf.seek(0)

         columns = ', '.join('"{}"'.format(k) for k in keys)
         if table.schema:
             table_name = '{}.{}'.format(table.schema, table.name <http://table.name>)
         else:
             table_name = table.name <http://table.name>

         sql = 'COPY {} ({}) FROM STDIN WITH CSV'.format(
             table_name, columns)
         cur.copy_expert(sql=sql, file=s_buf)
engine = create_engine('postgresql+psycopg2://:5432/postgres')
try:
     df.to_sql('test1', engine, schema='public', if_exists='append', index=False, method=psql_insert_copy)

I could not find obvious reasons.



Regards,

David



--
Adrian Klaver
adrian.klaver@xxxxxxxxxxx

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[Index of Archives]     [Postgresql Jobs]     [Postgresql Admin]     [Postgresql Performance]     [Linux Clusters]     [PHP Home]     [PHP on Windows]     [Kernel Newbies]     [PHP Classes]     [PHP Databases]     [Postgresql & PHP]     [Yosemite]

  Powered by Linux