Resolve partial lvm logical volume

  1. Server rebooted during lvm snapshot and put volume group in inconsistent state.
    # vgs
      VG   #PV #LV #SN Attr   VSize   VFree
      vg0    1   3   0 wz--n- 229.66G 9.66G
      vg1    2   3   0 wz-pn- 279.46G 9.46G
    

    Note the "p" attribute in vg1 which put the group in partial mode.

  2. To resolve, I first vgreduce then re-created the volume group from a good known archive:
    # vgreduce --removemissing --force vg1
      vsrv31_var_lib_mysql is expected to have only one segment using it, while it has 0
      Failed to find mirror_seg for vsrv31_var_lib_mysql
      vsrv31_var_log_mysql is expected to have only one segment using it, while it has 0
      Failed to find mirror_seg for vsrv31_var_log_mysql
      vsrv01_var_log is expected to have only one segment using it, while it has 0
      Failed to find mirror_seg for vsrv01_var_log
      Wrote out consistent volume group vg1
    
  3. Upon listing the volume groups, it automatically removed /dev/sdd1 from vg1.
    # vgs
      WARNING: Inconsistent metadata found for VG vg1 - updating to use version 1675
      Removing PV /dev/sdd1 (2tfAMr-erXr-fm1G-dicV-W08Q-eqg6-CL8nbH) that no longer belongs to VG vg1
      VG   #PV #LV #SN Attr   VSize   VFree  
      vg0    1   3   0 wz--n- 229.66G   9.66G
      vg1    1   0   0 wz--n- 139.73G 139.73G
    
    #pvs
      PV         VG   Fmt  Attr PSize   PFree  
      /dev/md2   vg0  lvm2 a-   229.66G   9.66G
      /dev/md3   vg1  lvm2 a-   139.73G 139.73G
      /dev/sdd1       lvm2 --   139.73G 139.73G
    
  4. Extended vg1 to included sdd1:
    # vgextend vg1 /dev/sdd1
      Volume group "vg1" successfully extended
    
    # pvs
      PV         VG   Fmt  Attr PSize   PFree  
      /dev/md2   vg0  lvm2 a-   229.66G   9.66G
      /dev/md3   vg1  lvm2 a-   139.73G 139.73G
      /dev/sdd1  vg1  lvm2 a-   139.73G 139.73G
    
    # vgs
      VG   #PV #LV #SN Attr   VSize   VFree  
      vg0    1   3   0 wz--n- 229.66G   9.66G
      vg1    2   0   0 wz--n- 279.46G 279.46G
    
    # lvs
      LV                 VG   Attr   LSize   Origin Snap%  Move Log Copy%  Convert
      lv0_145186_var_www vg0  -wi-ao  16.00G                                      
      lv0_swap           vg0  -wi-ao   4.00G                                      
      lv0_vz             vg0  -wi-ao 200.00G  
    
  5. Then did a listing of vg1 configs from archives (files located in /etc/lvm/archive):
    # vgcfgrestore -l vg1
    
      
      File:         /etc/lvm/archive/vg1_02156.vg
      VG name:      vg1
      Description:  Created *before* executing '/usr/sbin/lvcreate --size 5000m --snapshot --name snap_vsrv31 /dev/vg1/vsrv31_var_lib_mysql'
      Backup Time:  Thu Feb 27 02:39:26 2014
    
       
      File:         /etc/lvm/archive/vg1_02157.vg
      VG name:      vg1
      Description:  Created *before* executing '/usr/sbin/lvremove -f /dev/vg1/snap_vsrv31'
      Backup Time:  Thu Feb 27 07:35:34 2014
    
       
      File:         /etc/lvm/archive/vg1_02158.vg
      VG name:      vg1
      Description:  Created *before* executing '/usr/sbin/lvcreate --size 5000m --snapshot --name snap_vsrv31 /dev/vg1/vsrv31_var_lib_mysql'
      Backup Time:  Fri Feb 28 02:37:38 2014
    
  6. Restored the last known good archive:
    # vgcfgrestore -f /etc/lvm/archive/vg1_02158.vg vg1
      Restored volume group vg1
    
    # vgs
      VG   #PV #LV #SN Attr   VSize   VFree
      vg0    1   3   0 wz--n- 229.66G 9.66G
      vg1    2   3   0 wz--n- 279.46G 9.46G
    
    # pvs
      PV         VG   Fmt  Attr PSize   PFree
      /dev/md2   vg0  lvm2 a-   229.66G 9.66G
      /dev/md3   vg1  lvm2 a-   139.73G    0 
      /dev/sdd1  vg1  lvm2 a-   139.73G 9.46G
    
    # lvs
      LV                   VG   Attr   LSize   Origin Snap%  Move Log Copy%  Convert
      lv0_145186_var_www   vg0  -wi-ao  16.00G                                      
      lv0_swap             vg0  -wi-ao   4.00G                                      
      lv0_vz               vg0  -wi-ao 200.00G                                      
      vsrv01_var_log       vg1  -wi---  10.00G                                      
      vsrv31_var_lib_mysql vg1  -wi--- 250.00G                                      
      vsrv31_var_log_mysql vg1  -wi---  10.00G
    
  7. Activated the volume group:
    # vgchange -ay vg1
      3 logical volume(s) in volume group "vg1" now active
    
    # lvs
      LV                   VG   Attr   LSize   Origin Snap%  Move Log Copy%  Convert
      lv0_145186_var_www   vg0  -wi-ao  16.00G                                      
      lv0_swap             vg0  -wi-ao   4.00G                                      
      lv0_vz               vg0  -wi-ao 200.00G                                      
      vsrv01_var_log       vg1  -wi-a-  10.00G                                      
      vsrv31_var_lib_mysql vg1  -wi-a- 250.00G                                      
      vsrv31_var_log_mysql vg1  -wi-a-  10.00G
    
  8. I then got around to force check the logical volumes prior to mounting it and luckily all data was intact.
Comment