Re: Moving delta data faster

Adrian Klaver <adrian.klaver@xxxxxxxxxxx> · Thu, 4 Apr 2024 17:39:54 -0700

On 4/4/24 13:42, yudhi s wrote:

On Thu, Apr 4, 2024 at 9:04 PM Adrian Klaver <adrian.klaver@xxxxxxxxxxx 
<mailto:adrian.klaver@xxxxxxxxxxx>> wrote:

    On 4/3/24 22:24, yudhi s wrote:
     >
     > On Thu, Apr 4, 2024 at 10:16 AM Adrian Klaver
    <adrian.klaver@xxxxxxxxxxx <mailto:adrian.klaver@xxxxxxxxxxx>
     > <mailto:adrian.klaver@xxxxxxxxxxx
    <mailto:adrian.klaver@xxxxxxxxxxx>>> wrote:

    S3 is not a database. You will need to be more specific about '... then
    from the S3 it will be picked and gets merged to the target postgres
    database.'

The data from S3 will be dumped into the stage table and then the 
upsert/merge from that table to the actual table.

The S3 --> staging table would be helped by having the data as CSV and 
then using COPY. The staging --> final table step could be done as 
either ON CONFLICT or MERGE, you would need to test in your situation to 
verify which works better.

--
Adrian Klaver
adrian.klaver@xxxxxxxxxxx