Growing the disk in a vagrant basebox

I love using vagrant when making big changes to my puppet code. It allows you to really easily start with a blank slate for each iteration. It gives me a reasonable amount of certainty that should^H^H^H when one my instances blow up, a single puppet pass will do everything needed to generate a new one.

So, today, I’m happily working away on a major restructure of some puppet code when I run into this error.

  Error: Execution of '/usr/local/sbin/pkg install -qy javaservicewrapper' returned 3: pkg: Not enough space in /var/cache/pkg, needed 467 KiB available -100 KiB

Ah crap, I’ve run out of space in the vagrant instance. This basebox only has a 4GB root.

  $ df -h
  Filesystem        Size    Used   Avail Capacity  Mounted on
  /dev/ada0p2       3.7G    3.4G   -1.5M   100%    /

I need to make a new basebox with more space. One option is to just create a brand new basebox from scratch. But, instead, I decide to try and resize the disk in this existing basebox.

First of all, I created a minimal Vagrantfile. I just want to spin up a new VM based on my existing basebox image as a starting point so I run vagrant up with this Vagrantfile.

Once the instance comes up, I stop it with the virtualbox cli. Note, the VM that the Vagrantfile created is called ‘basebox’.

  $ VBoxManage controlvm basebox acpipowerbutton

Lets see how the disk look first.

  $ VBoxManage showvminfo basebox | grep IDE
  Storage Controller Name (0):            IDE
  IDE (0, 0): /home/chrskly/VirtualBox VMs/basebox/box-disk1.vmdk (UUID: ab38bc49-9e84-4635-9035-188038496198)
  IDE (1, 0): Empty

The disk I want to resize is a vmdk. The virtualbox cli can’t resize vmdk images so we have to convert it to a vdi, grow it, then convert it back to a vmdk.

  $ cd ~/VirtualBox\ VMs/basebox/
  $ VBoxManage clonehd box-disk1.vmdk box-disk1.vdi --format vdi
  0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100%
  Clone medium created in format 'vdi'. UUID: c2eb30af-96b3-4224-b26f-232d9e382589
  $ VBoxManage modifyhd box-disk1.vdi --resize 32768
  0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100%
  $ VBoxManage clonehd box-disk1.vdi box-disk1-32gb.vmdk --format vmdk
  0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100%
  Clone medium created in format 'vmdk'. UUID: f8760621-d8bb-441f-a394-7509e26e5986

Now we have a clone of our disk image, only it’s 32GB instead of 4GB. We need to attach this disk to our VM.

  $ VBoxManage storageattach basebox --storagectl "IDE" --device 0 --port 0 --type hdd --medium /home/chrskly/VirtualBox\ VMs/basebox/box-disk1-32gb.vmdk
  $ VBoxManage showvminfo basebox | grep IDE
  Storage Controller Name (0):            IDE
  IDE (0, 0): /home/chrskly/VirtualBox VMs/basebox/box-disk1-32gb.vmdk (UUID: f8760621-d8bb-441f-a394-7509e26e5986)
  IDE (1, 0): Empty

Note that there is no detach command for vbox storage. To replace a disk, you attach the new disk to old device/port address. This kicks off the old disk.

Now I want to boot the instance into single user mode to resize the filesystem. Because I’m lazy and don’t want to walk into the next room, turn on a monitor and type a few commands there, I set up serial console access to the VM. First, I had a look at the serial port config for the VM.

  $ VBoxManage showvminfo basebox | grep -i uart
  UART 1:          disabled
  UART 2:          disabled
  UART 3:          disabled
  UART 4:          disabled

The serial port is not enabled. We need to enable it.

  $ VBoxManage modifyvm basebox --uart1 0x3F8 4
  $ VBoxManage modifyvm basebox --uartmode1 server /tmp/basebox.raw

0x3F8 + IRQ4 is COM1. You can get more info on this in the virtualbox docs here: https://www.virtualbox.org/manual/ch08.html#vboxmanage-modifyvm-other https://www.virtualbox.org/manual/ch03.html#serialports

We then use socat to create a pty and connect to it with screen.

  socat UNIX-CONNECT:/tmp/basebox.raw PTY,link=/tmp/basebox.pty &
  screen /tmp/basebox.pty

When you boot your VM, you should now see console output in your screen session. You may need to set console="comconsole" in your /boot/loader.conf in your VM.

Press 2 for single user.

Now lets grow the disk from the OS-es perspective. We have a swap partition here that we need to delete temporarily while we grow the root partition.

  # gpart show
  =>     34  8388541  ada0  GPT  (32G) [CORRUPT]
         34     1024     1  freebsd-boot  (512K)
       1058  7965696     2  freebsd-ufs  (3.8G)
    7966754   419840     3  freebsd-swap  (205M)
    8386594     1981        - free -  (991K)

  # gpart recover ada0
  ada0 recovered
  # gpart show
  =>      34  67108797  ada0  GPT  (32G)
          34      1024     1  freebsd-boot  (512K)
        1058   7965696     2  freebsd-ufs  (3.8G)
     7966754    419840     3  freebsd-swap  (205M)
     8386594  58722237        - free -  (28G)

  # gpart delete -i 3 ada0
  ada0p3 deleted
  # gpart show
  =>      34  67108797  ada0  GPT  (32G)
          34      1024     1  freebsd-boot  (512K)
        1058   7965696     2  freebsd-ufs  (3.8G)
     7966754  59142077        - free -  (28G)

  # gpart resize -i 2 -s 31g ada0
  ada0p2 resized
  # gpart add -t freebsd-swap ada0
  ada0p3 added
  # gpart show
  =>      34  67108797  ada0  GPT  (32G)
          34      1024     1  freebsd-boot  (512K)
        1058  65011712     2  freebsd-ufs  (31G)
    65012770   2096061     3  freebsd-swap  (1.0G)

  # growfs /dev/ada0p2
  It's strongly recommended to make a backup before growing the file system.
  OK to grow filesystem on /dev/ada0p2, mounted on /, from 3.8GB to 31GB? [Yes/No] yes
  super-block backups (for fsck_ffs -b #) at:
   8975872, 10258112, 11540352, 12822592, 14104832, 15387072, 16669312, 17951552,
   19233792, 20516032, 21798272, 23080512, 24362752, 25644992, 26927232,
   28209472, 29491712, 30773952, 32056192, 33338432, 34620672, 35902912,
   37185152, 38467392, 39749632, 41031872, 42314112, 43596352, 44878592,
   46160832, 47443072, 48725312, 50007552, 51289792, 52572032, 53854272,
   55136512, 56418752, 57700992, 58983232, 60265472, 61547712, 62829952, 64112192

Time to reboot.

  # reboot
  Mar 31 22:59:27 init: single user shell terminated.
  Waiting (max 60 seconds) for system process `vnlru' to stop...done
  Waiting (max 60 seconds) for system process `bufdaemon' to stop...done
  Waiting (max 60 seconds) for system process `syncer' to stop...
  Syncing disks, vnodes remaining...0 0 0 0 0 0 0 0 0 0 done
  All buffers synced.
  Uptime: 56m7s
  $ vagrant ssh
  Last login: Fri Mar 31 22:02:36 2017 from 10.0.2.2
  FreeBSD 10.3-RELEASE-p11 (GENERIC) #0: Mon Oct 24 18:49:24 UTC 2016

  $ df -h
  Filesystem     Size    Used   Avail Capacity  Mounted on
  /dev/ada0p2     30G    2.5G     25G     9%    /
  devfs          1.0K    1.0K      0B   100%    /dev
  fdescfs        1.0K    1.0K      0B   100%    /dev/fd

Excellent! We can now package our VM to make a new basebox and I can get back to my puppet code.

  $ vagrant package basebox
  ==> basebox: Attempting graceful shutdown of VM...
  ==> basebox: Clearing any previously set forwarded ports...
  ==> basebox: Exporting VM...
  ==> basebox: Compressing package to: /home/chrskly/Code/basebox/package.box