Re: [External] Multiple COPY on the same table

Ron <ronljohnsonjr@xxxxxxxxx> · Mon, 20 Aug 2018 12:49:31 -0500



    Maybe he just has a large file that needs to be loaded into a
    table...

    
    On 08/20/2018 11:47 AM, Vijaykumar Jain
      wrote:

    
      Hey Ravi,

      
      What is the goal you are trying to achieve here.

      To make pgdump/restore faster?

      To make replication faster?

      To make backup faster ?

      
      Also no matter how small you split the files into, if network is
      your bottleneck then I am not sure you can attain n times the
      benefit my simply sending the files in parallel but yeah maybe
      some benefit.

      But then for parallel processing you also need to ensure your
      server is having relevant resources or else it will just be a lot
      of context switching I guess ?

      Pg dump has an option to dump in parallel

      pgbasebackup is single threaded I read but pgbackrest can allow
      better parallel processing in backups.

      There is also logical replication where you can selectively
      replicate your tables to avoid bandwidth issues.

      I might have said a lot and nothing may be relevant, but you need
      to let us know the goal you want to achieve :)

      
      Regards,

      Vijay
      
      From: Ravi
          Krishna <sravikrishna@xxxxxxx>

          Sent: Monday, August 20, 2018 8:24:35 PM

          To: pgsql-general@xxxxxxxxxxxxxxxxxxxx

          Subject: [External] Multiple COPY on the same table
         
      
            Can I split a large file into multiple files and then run
            copy using each file.  The table does not contain any 
          
            serial or sequence column which may need serialization. Let
            us say I split a large file to 4 files.  Will the
          
            performance boost by close to 4x??
          
            
            ps: Pls ignore my previous post which was without a subject
            (due to mistake)
        
      
    -- 

      Angular momentum makes the world go 'round.