29 thoughts on - “No Free Sectors Available” While Try To Extend Logical Volumen In A Virtual Machine Running CentOS 6.5

• Rebooting your system, then run fdisk /dev/sda

  Then run P
  N
  P
  3
  8e
  ……so on

  —–Original Message—

• Can’t pass from here, it says:

  Partition number (1-4): 3
  No free sectors available

  Why?

• This is for actual partitions, not LVM which seems to be what you want per the rest of your message.

  It’s telling you the truth. Sounds like you want another Logical Volume (LV) not partition.

  Unless you know what you’re doing, you _really_ shouldn’t do this in anything but a VM where you won’t lose your data. First rule of LVM resizing is to adjust the size (grow or shrink depending on your goal) of the file system before resizing the LV “container”.

  Remember there are a few “layers” here you have to keep in mind. disk –> partition –> LVM Phys Volume –> LVM Vol Group –> LVM Logical Vol –> File System (ext4, xfs, etc)

  If there are free extents in the VG, then you can probably create a LV. Depends on the extent size (defaults can vary between releases and/or an admin’s configuration).

  Sounds like you destroyed one or more of your LVs through all this.

  Please read the following documentation before forging further ahead. And you might spin up a VM or live CD to experiment with LVM operations before going any further as well.
  – speaks about extents [0]
  – read the entire Chapter 2 on LVM [1] as it applies to your scenario (ex:
  snapshots probably don’t)
  – dated/older, but it may prove helpful [2]

  [0]
  https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Logical_Volume_Manager_Administration/lv_overview.html
  [1]
  https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Logical_Volume_Manager_Administration/LVM_components.html
  [2] http://www.tldp.org/HOWTO/html_single/LVM-HOWTO/

• Hi SilverTip nice answer and very helpful, I’ll try to get some more help here since as I said in the main post I’m not an expert on Linux or a Administrator I’m just a developer trying to setup a development enviroment so …

  It’s telling you the truth. You’re right, what I need is a new LV but how I do that?

  Probable and I’m pretty sure I do it :-(

  Fine, I read it but know doubts persist on my mind. First, I’m running OS
  in a Vmware Workstation VM and I’ll not like to loose every I have there since then I’ll need to reconfigure all from scratch but if there is not another option to save my mess the we should go through it. Now I’m almost sure what I need here is a “Linear Volumes” configuration why? Well because my VM disks have 30GB in first and now I resize it to 80GB and that’s the space I want to see in my Linux and can’t get it. In order to get it working again, what steps I should follow? That’s my concern and what I’ve clear at all

  Thanks

• If I were in your position, I think I would:
  * Create a new, 80GB disk using VMWare
  * Partition that disk into your /boot and LVM partitions
  * pvcreate
  * vgcreate
  * lvcreate the disk structure you want in your new disk, making sure all LVs are at least a little bigger than the old ones.
  * use dd to copy disks from old drives to corresponding old drives
  * use resize2fs to expand your file system to the full size of each of the LVs you created.
  * detach old virtual disk from your VM
  * reboot, and see if you succeeded

  If I forgot something here, hopefully someone else will chime in. The idea is to dump your corrupted LVM structure without loosing its content.

  Ted Miller Elkhart, IN, USA

• Uppsss I think this goes more and more advanced all the time but here I go
  …. more doubts

  Not problem at all

  How I do that out of the box? I mean should I mount that disk in the VM and partition from there, right?

  Ok, create physical volume and volume group

  Here I get lost, what structure should I create here? I only have one LV
  lv_root you mean create the same and of course bigger than the old one, right?

  And here I declare myself complete lost, this is the first time I see this command and don’t know how to use it

• Those I/O errors are alarming. They suggest that you have a disk that is failing. Does anything about disk sda appear in /var/log/messages when you do that? You should indeed have 29GB available for growing lv_root, but perhaps the disk error is what is preventing the tool from finding the LV’s UUID.

• On Mon, Oct 27, 2014 at 11:21 PM, Robert Nichols

• what do you get from the commands:

  pvs -v
  vgs -v
  lvs

  and, if pvs shows any /dev/mdXX devices, the output of mdadm –detail
  /dev/mdXX

  ? example output…

  # pvs -v
  Scanning for physical volume names
  PV VG Fmt Attr PSize PFree DevSize PV UUID
  /dev/md127 vgdata lvm2 a– 1.82t 19.68g 1.82t pPuDNs-AVQ8-92tw-TXcT-WWyD-nPhQ-dZqpx0
  /dev/sda2 vg_myhost lvm2 a– 476.45g 5.36g 476.45g EWe4ws-1Z6S-v9d6-gvQ2-e7QE-K58b-Sd1W5z

  # mdadm –detail /dev/md127
  /dev/md127:
  Version : 1.2
  Creation Time : Sat Jun 14 13:18:25 2014
  Raid Level : raid1
  Array Size : 1953383232 (1862.89 GiB 2000.26 GB)
  Used Dev Size : 1953383232 (1862.89 GiB 2000.26 GB)
  Raid Devices : 2
  Total Devices : 2
  Persistence : Superblock is persistent

  Update Time : Mon Oct 27 22:35:55 2014
  State : clean
  Active Devices : 2
  Working Devices : 2
  Failed Devices : 0
  Spare Devices : 0

  Name : myhost:0 (local to host myhost)
  UUID : d9c90fda:9a0e5d4f:d27cf1f6:19d0b43a
  Events : 441

  Number Major Minor RaidDevice State
  0 8 17 0 active sync /dev/sdb1
  1 8 33 1 active sync /dev/sdc1

  # vgs -v
  Finding all volume groups
  Finding volume group “vgdata”
  Finding volume group “vg_myhost”
  VG Attr Ext #PV #LV #SN VSize VFree VG
  UUID VProfile
  vg_myhost wz–n- 4.00m 1 6 0 476.45g 5.36g cX6DQy-iDY2-mL0Q-zM1m-pgf2-kLdE-8zWtIG
  vgdata wz–n- 4.00m 1 1 0 1.82t 19.68g USqdKh-VIv7-TrCE-2RXn-52oG-7Qed-01URWg

  # lvs
  LV VG Attr LSize Pool Origin Data% Move Log Cpy%Sync Convert
  lv_home vg_myhost -wi-ao—- 29.30g
  lv_root vg_myhost -wi-ao—- 50.00g
  lv_swap vg_myhost -wi-ao—- 11.80g
  lvimages vg_myhost -wi-ao—- 150.00g
  lvpgsql vg_myhost -wi-ao—- 30.00g
  lvtest vg_myhost -wi-a—– 200.00g
  lvhome2 vgdata -wi-ao—- 1.80t

• Hi John, here are the results:

  #pvs -v
  Scanning for physical volume names
  /dev/root: read failed after 0 of 4096 at 27522957312: Input/output error
  /dev/root: read failed after 0 of 4096 at 27523014656: Input/output error
  Wiping cache of LVM-capable devices
  /dev/root: read failed after 0 of 4096 at 27522957312: Input/output error
  /dev/root: read failed after 0 of 4096 at 27523014656: Input/output error
  Couldn’t find device with uuid vSbuSJ-o1Kh-N3ur-JYkM-Ktr4-WEO2-JWe2wS.
  There are 1 physical volumes missing.
  There are 1 physical volumes missing.
  There are 1 physical volumes missing.
  PV VG Fmt Attr PSize PFree DevSize PV UUID
  /dev/sda2 vg_devserver lvm2 a– 29.51g 0 79.51g ij17Vf-kY56-jfg0-j769-Z1E5-Nk7C-RqYG18
  unknown device vg_devserver lvm2 a-m 29.99g 29.96g 0
  vSbuSJ-o1Kh-N3ur-JYkM-Ktr4-WEO2-JWe2wS

  # vgs -v
  Finding all volume groups
  /dev/root: read failed after 0 of 4096 at 27522957312: Input/output error
  /dev/root: read failed after 0 of 4096 at 27523014656: Input/output error
  Finding volume group “vg_devserver”
  Wiping cache of LVM-capable devices
  /dev/root: read failed after 0 of 4096 at 27522957312: Input/output error
  /dev/root: read failed after 0 of 4096 at 27523014656: Input/output error
  Couldn’t find device with uuid vSbuSJ-o1Kh-N3ur-JYkM-Ktr4-WEO2-JWe2wS.
  There are 1 physical volumes missing.
  There are 1 physical volumes missing.
  VG Attr Ext #PV #LV #SN VSize VFree VG
  UUID VProfile
  vg_devserver wz-pn- 4.00m 2 2 0 59.50g 29.96g VidRBE-37ri-HYfd-Sd1x-6lZX-ph9I-F3wvCG

  # lvs
  /dev/root: read failed after 0 of 4096 at 27522957312: Input/output error
  /dev/root: read failed after 0 of 4096 at 27523014656: Input/output error
  Couldn’t find device with uuid vSbuSJ-o1Kh-N3ur-JYkM-Ktr4-WEO2-JWe2wS.
  LV VG Attr LSize Pool Origin Data% Move Log Cpy%Sync Convert
  lv_root vg_devserver -wi-ao–p- 25.63g
  lv_swap vg_devserver -wi-ao—- 3.91g

  And there is not mdXX device so I ommit that command

• Something very strange is going on. /dev/root is presumably
  /dev/mapper/vg_devserve-lv_root. It is odd that LVM should be looking inside that LV for another PV, and disturbing that there should be an I/O error at those offsets, which are within the 27523022848 byte size of /dev/mapper/vg_devserver-lv_root .

  Please post the output from
  ls -l /dev/root /dev/mapper and the contents of file /etc/lvm/backup/vg_devserver .

• On Tue, Oct 28, 2014 at 10:31 AM, Robert Nichols dm-2

  /dev/mapper:
  total 0
  crw-rw—- 1 root root 10, 58 Oct 27 17:55 control lrwxrwxrwx 1 root root 7 Oct 27 17:55 vg_devserver-lv_root -> ../dm-2
  lrwxrwxrwx 1 root root 7 Oct 27 17:55 vg_devserver-lv_root-missing_1_0
  -> ../dm-1
  lrwxrwxrwx 1 root root 7 Oct 27 17:55 vg_devserver-lv_swap -> ../dm-0

  # cat /etc/lvm/backup/vg_devserver
  # Generated by LVM2 version 2.02.100(2)-RHEL6 (2013-10-23): Mon Oct 20
  15:27:17 2014

  contents = “Text Format Volume Group”
  version = 1

  description = “Created *after* executing ‘vgreduce –removemissing vg_devserver'”

  creation_host = “localhost” # Linux localhost
  2.6.32-431.29.2.el6.x86_64 #1 SMP Tue Sep 9 21:36:05 UTC 2014 x86_64
  creation_time = 1413835037 # Mon Oct 20 15:27:17 2014

  vg_devserver {
  id = “VidRBE-37ri-HYfd-Sd1x-6lZX-ph9I-F3wvCG”
  seqno = 6
  format = “lvm2” # informational
  status = [“RESIZEABLE”, “READ”, “WRITE”]
  flags = []
  extent_size = 8192 # 4 Megabytes
  max_lv = 0
  max_pv = 0
  metadata_copies = 0

  physical_volumes {

  pv0 {
  id = “ij17Vf-kY56-jfg0-j769-Z1E5-Nk7C-RqYG18”
  device = “/dev/sda2” # Hint only

  status = [“ALLOCATABLE”]
  flags = []
  dev_size = 61888512 # 29.5107 Gigabytes
  pe_start = 2048
  pe_count = 7554 # 29.5078 Gigabytes
  }

  pv1 {
  id = “vSbuSJ-o1Kh-N3ur-JYkM-Ktr4-WEO2-JWe2wS”
  device = “unknown device” # Hint only

  status = [“ALLOCATABLE”]
  flags = [“MISSING”]
  dev_size = 62906520 # 29.9962 Gigabytes
  pe_start = 2048
  pe_count = 7678 # 29.9922 Gigabytes
  }
  }

  logical_volumes {

  lv_root {
  id = “Ee2apF-rS1m-7Xny-4XYk-u2ZM-lMp7-IBQPnL”
  status = [“READ”, “WRITE”, “VISIBLE”]
  flags = []
  creation_host = “devserver”
  creation_time = 1361816267 # 2013-02-25
  13:47:47 -0430
  segment_count = 2

  segment1 {
  start_extent = 0
  extent_count = 6554 # 25.6016 Gigabytes

  type = “striped”
  stripe_count = 1 # linear

  stripes = [
  “pv0”, 0
  ]
  }
  segment2 {
  start_extent = 6554
  extent_count = 8 # 32 Megabytes

  type = “striped”
  stripe_count = 1 # linear

  stripes = [
  “pv1”, 0
  ]
  }
  }

  lv_swap {
  id = “hW0BAn-BzE4-aHXm-b18h-8rg0-JyBS-ycbOVd”
  status = [“READ”, “WRITE”, “VISIBLE”]
  flags = []
  creation_host = “devserver”
  creation_time = 1361816271 # 2013-02-25
  13:47:51 -0430
  segment_count = 1

  segment1 {
  start_extent = 0
  extent_count = 1000 # 3.90625 Gigabytes

  type = “striped”
  stripe_count = 1 # linear

  stripes = [
  “pv0”, 6554
  ]
  }
  }
  }
  }

• By looking at fdisk output pasted in your first mail, I don’t see more than
  2 partition in your first disk – /dev/sda . The first partition is – /dev/sda1 – is allocated for /boot and second partition is – /dev/sda2 – for your PV/VG/LV where root resided. However, the above output indicates you had 2 PVs , which is contradicting with the information you have shared so far.

  I assume that, you have added an additional disk , then extended the VG, but later removed ? OR tried another vgcreate ( afaik, lvm shouldn’t allow this but a wild guess ) on actual PV – /dev/sda2 ?

  Cheers,

• You’re right in some moment I added a new disk and try to extend the VG and yes once again I removed by trying to do the same from Gparted Live CD and I messed all, this is what I’m trying to fix if it’s possible so what procedure I should follow having this extra info? (apologies perhaps I
  should say this from the begining)

• Ok. I think, I should know

  – whether you have rebooted your machine/VM right after extended the disk
  ? If not, please don’t reboot now :)
  – Other than attaching an additional disk, have you tried any lvm/dd commands or any other procedure that’s missed in your previous update ?

• Yes, that would have helped a lot. I can see in file
  /etc/lvm/backup/vg_devserver that the lv_root LV has been extended by a mere 32 Megabytes (yes, Mega) on the missing physical volume. Do you still have the additional disk? The best thing to do would be to put it back, make a partition on it, and then re-create the missing PV. I’ll assume that the new partition is /dev/sdb1. Adjust all references in the following if it si something else.

  Make a copy of /etc/lvm/backup/vg_devserver on vg_devserver.bak, and then run

  pvcreate -v –restorefile vg_devserver.bak \
  –uuid “vSbuSJ-o1Kh-N3ur-JYkM-Ktr4-WEO2-JWe2wS” \
  /dev/sdb1

  (I’ve broken that up into multiple lines to avoid word wrap problems.)

  You should then be able to run pvs and lvs successfully. To properly remove this PV from the LVM structure, run

  lvreduce –extents -8 /dev/vg_devserver/lv_root
  vgreduce vg_devserver /dev/sdb1
  pvremove /dev/sdb1

  That lvreduce will get rid of the 32MB that were allocated on that added PV. You will probably get a warning about possible data loss and will have to confirm. The vgreduce and pvremove should then proceed without any issues.

• On Tue, Oct 28, 2014 at 11:45 AM, Robert Nichols http://imgur.com/jMuhM03 this is how my disk looks like

  This is the result of the command above:

  # pvcreate -v –restorefile /etc/lvm/backup/vg_devserver.bak –uuid
  “vSbuSJ-o1Kh-N3ur-JYkM-Ktr4-WEO2-JWe2wS” /dev/sdb1
  /dev/root: read failed after 0 of 4096 at 27522957312: Input/output error
  /dev/root: read failed after 0 of 4096 at 27523014656: Input/output error
  Wiping cache of LVM-capable devices
  /dev/root: read failed after 0 of 4096 at 27522957312: Input/output error
  /dev/root: read failed after 0 of 4096 at 27523014656: Input/output error
  Couldn’t find device with uuid vSbuSJ-o1Kh-N3ur-JYkM-Ktr4-WEO2-JWe2wS.
  Wiping cache of LVM-capable devices
  /dev/root: read failed after 0 of 4096 at 27522957312: Input/output error
  /dev/root: read failed after 0 of 4096 at 27523014656: Input/output error
  Device /dev/sdb1 not found (or ignored by filtering).

• You will have to find the archived configuration from before you tried to extend that LV, and restore that configuration. Let’s see what file you have. Post the output from

  grep -H ‘description =’ /etc/lvm/archive/vg_devserver*

• Here,

  # grep -H ‘description =’ /etc/lvm/archive/vg_devserver*
  /etc/lvm/archive/vg_devserver_00000-1387802225.vg:description = “Created
  *before* executing ‘/sbin/vgs –noheadings -o name –config
  ‘log{command_names=0 prefix=\” \”}””
  /etc/lvm/archive/vg_devserver_00001-1037597683.vg:description = “Created
  *before* executing ‘vgextend vg_devserver /dev/sda3′”
  /etc/lvm/archive/vg_devserver_00002-1876503.vg:description = “Created
  *before* executing ‘lvextend -L+29.99 /dev/vg_devserver/lv_root'”
  /etc/lvm/archive/vg_devserver_00003-1263624397.vg:description = “Created
  *before* executing ‘vgreduce –removemissing vg_devserver'”
  /etc/lvm/archive/vg_devserver_00004-313693030.vg:description = “Created
  *before* executing ‘vgreduce –removemissing vg_devserver –force'”

• That all there are??? By default that archive should go back a minimum of
  30 days and a minimum of 10 files, whichever is larger. All I see here is a history of your recovery efforts. Did you make that change from a live CD, or something?

  If necessary I could edit the backup file that you have now and clean out all reference to the missing PV, but I’d rather not do that if there is a safer way.

• Well seem to be not working

  # vgcfgrestore -v –file LVMconfig.new vg_devserver –test File descriptor 7 (pipe:[18995]) leaked on vgcfgrestore invocation. Parent PID 3790: bash
  TEST MODE: Metadata will NOT be updated and volumes will not be
  (de)activated.
  Parse error at byte 81 (line 2): unexpected token
  Couldn’t read volume group metadata.
  Restore failed.
  Test mode: Wiping internal cache
  Wiping internal VG cache

• Nevermind I fixed this is the result of running the command in test mode:

  # vgcfgrestore -v –file LVMconfig.new vg_devserver –test File descriptor 7 (pipe:[18995]) leaked on vgcfgrestore invocation. Parent PID 3790: bash
  TEST MODE: Metadata will NOT be updated and volumes will not be
  (de)activated.
  /dev/root: read failed after 0 of 4096 at 27522957312: Input/output error
  /dev/root: read failed after 0 of 4096 at 27523014656: Input/output error
  Restored volume group vg_devserver
  Test mode: Wiping internal cache
  Wiping internal VG cache

  Should I go?

• That looks good. I understand the I/O error now. It is from an attempt to read from the missing PV. The one question I have is, “Did you ever resize the root filesystem?” I suspect not, since you would have seen errors from the missing blocks. If you _did_, then it should be “interesting” trying to get the filesystem back into its available space. In any case, fixing the LVM structure has to come first, so I say go ahead without the “–test”
  option.

• Ok, done:

  vgcfgrestore -v –file LVMconfig.new vg_devserver File descriptor 7 (pipe:[18995]) leaked on vgcfgrestore invocation. Parent PID 3790: bash
  /dev/root: read failed after 0 of 4096 at 27522957312: Input/output error
  /dev/root: read failed after 0 of 4096 at 27523014656: Input/output error
  Restored volume group vg_devserver

  What’s next?

• See if things look sane. Do vgs and lvs run without error now?

• Again, your attachments aren’t coming through. Did you try running vgs and lvs before rebooting? What was the result?

• That’s weird maybe the list is configured to not allow attachments and no I
  didn’t try any just reboot after the latest command and now get kernel panic, VM do not start

• The list does allow attachments. The file I sent you was an attachment.

  I wish you hadn’t jumped right into rebooting. My only consolation is that the result you got is probably what would have happened had you tried to reboot _without_ changing anything.

  Are you seeing any messages prior to the kernel panic? If not, reboot and press TAB to see the GRUB menu. Press “a” to get to the kernel parameters and then remove the parameters “rhgb” and “quiet”. Then you will get to see what leads to the panic.

• I do not know why then my attachment are not attached, anyway here I
  uploaded the image http://imgur.com/B7YWY10

  Yes, before the latest change I reboot several times and system always pass trough

  I leave you the printscreen on the image link, anyway is a VM the only problem is that I lost all the data and need to reinstall it all from scratch but recently CentOS 6.6 comes out so I’ll give a try to this one and this time creating the proper LVM with a lot of space and keep this email secure for the future

  Thanks to everyone here one the list, great support