Re: ERROR: unexpected chunk number 0 (expected 1) for toast value 76753264 in pg_toast_10920100

Achilleas Mantzios <achill@xxxxxxxxxxxxxxxxxxxxx> · Fri, 9 Jun 2017 15:13:25 +0300



    On 09/06/2017 14:44, Harry Ambrose
      wrote:

    
        Hi Tom,
        

        Thanks for attempting to replicate the issue.
        

            Anyway,
              the bad news is I couldn't reproduce the problem then and
              I can't

              now.  I don't know if it's a timing issue or if there's
              something critical

              about configuration that I'm not duplicating.  Can you
              explain what sort

              of platform you're testing on, and what nondefault
              configuration settings

              you're using?

            
        Further details about the environment
          that I can replicate on below:
        

        - Non default postgresql.conf settings:
        
          checkpoint_segments
              = 192
          checkpoint_completion_target
              = 0.9
          checkpoint_timeout
              = 5min
          wal_keep_segments
              = 256
          wal_writer_delay
              = 200ms
          archive_mode
              = on
          archive_command
              = 'rsync -e ssh -arv  /wal/pg_xlog/%f postgres@<removed
              hostname>:/wal/pg_xlog'
          archive_timeout
              = 60
          syslog_facility
              = 'LOCAL0'
          log_statement
              = 'mod'
          syslog_ident
              = 'postgres'
          log_line_prefix
              = '%h %m  %p %c %u %a  %e '
          log_timezone
              = 'GB'
          track_activities
              = on
          track_counts
              = on
          datestyle
              = 'iso, mdy'
          timezone
              = 'GB'
          default_text_search_config
              = 'pg_catalog.english'
          array_nulls
              = on
          sql_inheritance
              = on
          standard_conforming_strings
              = on
          synchronize_seqscans
              = on
          transform_null_equals
              = off
          - Two node master/slave setup using streaming
              replication (without slots).
          - CentOS 6.9 (2.6.32-696.el6.x86_64).
          - PostgreSQL 9.4.10 on
            x86_64-unknown-linux-gnu, compiled by gcc (GCC) 4.4.7
            20120313 (Red Hat 4.4.7-17), 64-bit.
          - 64GiB RAM.
          - AMD Opteron(TM) Processor 6238.
          - pg_default sat on 2 disk RAID-1
            conifugration (ext3 filesystem).
          - Custom tablespaces (where the
            errors occur) sat on 4 disk RAID-10 (ext3 filesystem).
          - All disks are HP 600G SAS 6.0Gbps
            with P420(i) controllers and battery backed cache enabled.
          

    Maybe you could give some info on :

    - your ext3 mkfs and mount options (journal, barriers, etc)

    - your controller setup (battery should be working good and cache
    mode set to write back)

    - your disks setup (write cache should be disabled)

    - you should check your syslogs/messages for any errors related to
    storage

    - is your RAM ECC? Did you run any memtest?

    - is your CPU overheating ?

    - have you experienced any crashes/freezes ?

    
          Please let me know if you require
            further info.
          

          Best wishes,
          Harry
          

    -- 
Achilleas Mantzios
IT DEV Lead
IT DEPT
Dynacom Tankers Mgmt