Re: RHEL6.6 dm-cache and dm-era

Romu Hu <huruomu@xxxxxxxxx> · Mon, 29 Sep 2014 10:31:08 +0800



    On 2014/9/26 18:33, Heinz Mauelshagen wrote:

    
      Romu,

      
      did you initialize the metadata device before trying to create the
      era target?

      Zeroes in the first KBs will do.

    
    Thanks Heinz, it solved the problem.  Is it the proper way to
    initialize the metadata device of an era target or just a
    workaround?

    
    Thanks

    Romu

    
     Yes,
      check for NULL in era_destroy is needed to  handle the error path
      you hit properly.

      
      Heinz

      
      On 09/26/2014 05:16 AM, Romu Hu
        wrote:

      
        On 2014/9/25 22:57, Heinz Mauelshagen wrote:

        
          Romu,

          
          "dmsetup create MyEra --table "0 20971520 /dev/mapper/mpathbp1
          /dev/mapper/mpathap1 8"

          
          should read

          
          dmsetup create MyEra --table "0 20971520 era
          /dev/mapper/mpathbp1 /dev/mapper/mpathap1 8"

        
        Sorry my bad.

        
        I ran 'dmsetup create MyEra --table "0 20971520 era
        /dev/mapper/mpathbp1 /dev/mapper/mpathap1 8"' but kernel
        (2.6.32-494.el6.i686) oopsed:

        
        Sep 25 18:34:11 localhost kernel: device-mapper: era:
          sb_check failed: magic 0: wanted 2126579579

        Sep 25 18:34:11 localhost kernel: device-mapper: block
          manager: superblock validator check failed for block 0

        Sep 25 18:34:11 localhost kernel: device-mapper: era:
          couldn't read_lock superblock

        Sep 25 18:34:11 localhost kernel: BUG: unable to handle
          kernel NULL pointer dereference at 00000008

        Sep 25 18:34:11 localhost kernel: IP:
          [<f7e2b3c7>] era_destroy+0x7/0x60 [dm_era]Sep 25
          18:34:11 localhost kernel: *pdpt = 0000000032457001 *pde =
          00000003fb5fa067

        Sep 25 18:34:11 localhost kernel: Oops: 0000 [#1] SMP

        Sep 25 18:34:11 localhost kernel: last sysfs file:
          /sys/module/dm_persistent_data/initstate

        
        Perhaps this should be cherry-picked?

        https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=989f26f5ad308f40a95f280bf9cd75e558d4f18d

        
        The log says sb_check failed, anything wrong with the dmsetup
        command parameters?

        
        Thanks

        Romu

        
          On 09/25/2014 04:02 PM, Romu
            wrote:

          
                2014-09-25 18:19 GMT+08:00
                  Heinz Mauelshagen <heinzm@xxxxxxxxxx>:

                  
                        On 09/25/2014 08:10 AM, Romu Hu wrote:

                         On 2014/9/24 12:10,
                          Brassow Jonathan wrote:

                          On Sep 22, 2014, at
                            7:35 AM, Romu wrote:
                            

                              I
                                  tried the following command to set up
                                  my era target but the command
                                  immediately panics the system and the
                                  system reboots.
                                  

                                  # dmsetup create MyEra --table "0
                                    41941903 era
                                    /dev/mapper/VG-CacheDataLV_cmeta
                                    /dev/mapper/VG-OriginLV 4096"

                                  
                                  The metadata dev and the origin
                                    dev are all part of a LVM cache LV. 
                                    VG-CacheDataLV_cmeta is the cache
                                    metadata LV on the smaller and
                                    faster device, VG-OriginLV is the
                                    origin LV on the faster and slower
                                    device, 41941903 is the total sector
                                    number of the device of OriginLV
                                    (the LV takes 100% space of the
                                    device), 4096 is the block size of
                                    OriginLV, I have run 'mkfs.ext4
                                    /dev/mapper/VG-OriginLV' before
                                    running the dmsetup command.

                                  
                                  Below is the message in
                                    /var/log/messages after running the
                                    dmsetup comnmand:
                                  

                                  kernel: device-mapper: era:
                                    sb_check failed: magic 1623043:
                                    wanted 2126579579

                                  
                                    kernel: device-mapper: block
                                      manager: superblock validator
                                      check failed for block 0
                                    kernel: device-mapper: era:
                                      couldn't read_lock superblock
                                  
                                  
                                  Any idea?
                                

                            Sorry, I haven't used dm-era yet. 
                              However, it does appear that you are
                              perhaps specifying the wrong devices when
                              creating the era device?  Looks like you
                              might be allowing the era target and the
                              cache target to use the same metadata area
                              at the same time - causing them to corrupt
                              each other's metadata area?
                            

                             brassow
                          
                          
                          I think the dmsetup command to set up an era
                          target should be something like this:

                          
                          #
                              dmsetup create MyEra --table "0
                              sector_number era metadeta_dev origin_dev
                              block_size"

                          
                          My questions:

                          
                          1) How to calculate sector_number?  According
                          to https://www.kernel.org/doc/Documentation/device-mapper/era.txt,
                          I guess it should be (4 * nr_blocks) bytes +
                          buffers, but what is nr_blocks and what is
                          buffers?

                        
                       No calculation: it's the size of your era
                      target in sectors. Typically "blockdev --getsz
                      origin_dev"

                        
                         2) Any documentation
                          for dmsetup tables?

                        
                       Look for examples in the kernel source
                      trees Documentaion/device.mapper.

                      
                      table line syntax is: "start_sector
                      length_in_sectors <target>
                      <target_arguments{0,N}>"

                        
                         3) Both my metadata_dev
                          and origin_dev are 10G partitions, is this all
                          right?

                        
                       Metadata device size is plenty but that
                      depends on how many eras with how many updates you
                      want to have.

                        
                         4) My origin_dev is a
                          ext4 filesystem with a block size of 4096, so
                          the block_size in the command line should also
                          be 4096?

                        
                       You may use 4096 = _8_ sectors, you used
                      4096 sectors = 2MB below.

                      It's the granularity the era target housekeeps
                      blocks for.

                        
                          I tried the following command:

                          
                          # dmsetup create MyEra --table "0 160000
                            era /dev/mapper/mpathap1
                            /dev/mapper/mpathbp1 4096"

                        
                       May not be  same physical device for data
                      and metadata!

                      If it is, thta'd explain your oops.

                      
                      # dmsetup create MyEra --table "0 $(blockdev
                        --getsz /dev/mapper/mpathbp)
                        era whatever_disctinct_metadata_device
                        /dev/mapper/mpathbp1 8"

                      
                      Would use 8 sector block size (which is tiny!)
                      with disctinct metadata and data devices
                      (presumabyl your ext4 sits on /dev/mapper/mpathbp1)

                          
                          Heinz

                        
                  Heinz, thank you for your help!
                  

                  My origin dev is 10G so total sector number
                    is 20971520.  /dev/mapper/mpathbp1 (/dev/dm-6) is
                    the metadata dev, /dev/mapper/mpathap1 (/dev/dm-7)
                    is the origin dev, they are on different LUNs.
                  

                  I tried the following commands:
                  

                  # dmsetup create MyEra --table "0 20971520
                    /dev/mapper/mpathbp1 /dev/mapper/mpathap1 8"

                  
                  # Target type name /dev/mapper/mpathbp1 is too
                    long.
                  
                    # Command failed
                  
                  
                  # dmsetup create MyEra --table "0 20971520
                    /dev/dm-6 /dev/dm-7 8"

                  
                  # device-mapper: reload ioctl on MyEra failed:
                    Invalid argument
                  
                    # Command failed
                  
                  
                  And when executing the second command I see the
                    following in /var/log/messages:
                  

                    Sep 25 06:37:48 localhost kernel:
                      device-mapper: table: 253:8: /dev/dm-6: unknown
                      target type
                    Sep 25 06:37:48 localhost kernel:
                      device-mapper: ioctl: error adding target to table
                    Sep 25 06:37:48 localhost multipathd: dm-8:
                      remove map (uevent)
                    Sep 25 06:37:48 localhost multipathd: dm-8:
                      devmap not registered, can't remove
                    Sep 25 06:37:48 localhost multipathd: dm-8:
                      remove map (uevent)
                    Sep 25 06:37:48 localhost multipathd: dm-8:
                      devmap not registered, can't remove
                  
                  
                  Then I run 'mkfs.ext4 /dev/dm-6' and tried the
                    second command again but got the same result.
                  

                  Any idea?
                  

                  Thanks
                  Romu
                
              
            --
dm-devel mailing list
dm-devel@xxxxxxxxxx
https://www.redhat.com/mailman/listinfo/dm-devel
          
          
          --
dm-devel mailing list
dm-devel@xxxxxxxxxx
https://www.redhat.com/mailman/listinfo/dm-devel
        
        
        --
dm-devel mailing list
dm-devel@xxxxxxxxxx
https://www.redhat.com/mailman/listinfo/dm-devel
      
      
      -- 
===============================================================
Heinz Mauelshagen                               +49 2626 141200
Consulting Development Engineer             FAX +49 2626 924446
Red Hat GmbH
Am Sonnenhang 11
56242 Marienrachdorf
Germany                                       heinzm@xxxxxxxxxx
===============================================================
      

      --
dm-devel mailing list
dm-devel@xxxxxxxxxx
https://www.redhat.com/mailman/listinfo/dm-devel
    
    
--
dm-devel mailing list
dm-devel@xxxxxxxxxx
https://www.redhat.com/mailman/listinfo/dm-devel