Gentoo on AWS

Published on Author Artem Butusov9 Comments

Introduction

There are a lot of ready Gentoo instance images available for free on Amazon Marketplace, but those images are not trusted at least by me, so I have to install Gentoo from scratch.

There are also a lot of different articles about how to run Gentoo on AWS or on any other cloud. My article based on my experience and how I’m doing that.

Ok, we are logged in Ubuntu in screen, we have second drive to install gentoo, so just follow with
official gentoo handbook to install. Below I will show what I’m doing to get gentoo installed.

This article recommended for people who read Gentoo Handbook and know how to install Gentoo locally. If you are not familiar with Gentoo you will need definitly to start from Gentoo Handbook. Also if you are not familiar with gentoo I recommend to try to install on virtual machine locally firstly.

Also this article is not an answer on question: “Why Gentoo and not CentOS, Debian, Ubuntu, RedHat, SuSE etc?”

I will try to keep article hierarchy similar to Gentoo Handbook.

As example I will use Hardened Gentoo x86_64.

You can use power computing instance to speedup building, use cheapest instance but it will take more time, or use technics like distcc or binpkg which are not covered by this article.

Virtualization

Amazon has two virtualization available for linux platform: PVM and HVM. PVM didn’t work for Windows because using para-virtual devices and HVM emulates everything so can be used to install Windows. A long time ago PVM was faster than HVM but for now difference is so small because HVM provide the same para-virtual devices for HVM as for PVM.

Amazon has less support for PVM in comparison to HVM. A lot of instance types including new available only if you use HVM.

So we will go forward with HVM, but let me know if you are intrested in PVM. May it will motivate me to share my PVM experience too.

Also, you need to know that Amazon currently using XEN as primary virtualization hypervisor.

Workflow

What do we expect in the end of procedure? We expect to see Amazon Image which can be used to spawn new HVM instance with fresh clean Gentoo default installation which allow as to logon via SSH based on ssh public keys provided via AWS to instance.

There are few ways to reach this goal.

The Way #1:

  • run any instance with linux os on first drive
  • install gentoo on second drive
  • reboot to gentoo on second drive
  • clone gentoo from second drive to first drive
  • reboot to gentoo on first drive
  • create snapshot from first drive
  • create image from created snapshot

The Way #2:

  • run any instance with linux os on first drive
  • install gentoo on second drive
  • create snapshot from second drive
  • create image from created snapshot

The Way #3:

  • prepare gentoo image on any virtualization software locally
  • import AMI image
  • https://aws.amazon.com/ec2/vm-import/

Just for education purposes here I will show how to replace original linux installation with gentoo based on Way #1.

Spawning linux instance

Logon to AWS console and create instance:

STEP 1: CHOOSE AN AMAZON MACHINE IMAGE (AMI)

I will use Ubuntu Server 14.04 LTS (HVM), SSD Volume Type as example.

STEP 2: CHOOSE AN INSTANCE TYPE

I will use t2.micro but cpu optimized instance recommended.

STEP 3: CONFIGURE INSTANCE DETAILS

You can skip this screen with one exception: if you would like to attach earlier created volume to instance you need to be sure that instance and volume located in the same availability zone and sometimes you have to manually choose valid availability zone.

STEP 4: ADD STORAGE

Use 20GiB for Root (target) and 20GiB for /dev/sdb (temporary).

STEP 5: TAG INSTANCE

You can skip.

STEP 6: CONFIGURE SECURITY GROUP

Let amazon create default security group which allows to connect via SSH.

STEP 7: REVIEW INSTANCE LAUNCH

Create or import SSH public key and confirm.

Connecting to instance

Grub IP address for running instance from AWS console, 1.2.3.4, for example.

Ubuntu instance has “ubuntu” default user name.

If you are on unix-like os: ssh ubuntu@1.2.3.4

If you are on windows os: download PuTTY, install and use IP address and “ubuntu” username to connect to SSH server.

After logging into instance run screen to save terminal with running commands even if for some reason you will loose connection. In that case you will need just to reconnect and run screen -r to restore session.

On Ubuntu we need also to run sudo bash to get root access.

Note: You can register domain in Route 53 on amazon and associate instance public IP address with domain name. I recommend also to use Elastic IP for that purposes because it will allow you to terminate instance and create againg with the same IP address without any needs to change something in DNS records.

Installing Gentoo in second volume

Preparing the disks

We will perform all operations on second disk because we are running Ubuntu from first disk, so first disk is busy when we boot os from that disk.

Designing a partition scheme

List all available disks:

lsblk
NAME    MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
xvda    202:0    0  20G  0 disk
└─xvda1 202:1    0  20G  0 part /
xvdb    202:16   0  20G  0 disk

We are creating first partition from 4096 sector to be sure that it will be well aligned for any device (physical or virtual). We are printing partition table after each operation to be sure that everything is ok. Also to get start for next partition you need to select next sector after end of previous partition.

We are creating one 18Gb partition for root and remaining space ~2Gb for swap.

Using parted to partition the disk

parted /dev/xvdb

Below is an example for 13GB partition, but it is recommended to use a little bit larger volume to not run out of space after compilation (18Gb root + 2Gb swap).

(parted) unit s
(parted) mklabel msdos
(parted) print
(parted) mkpart primary 4096s 13G
(parted) print
(parted) mkpart primary 25391104s 100%
(parted) print
(parted) quit

Creating file systems

mkfs.ext4 /dev/xvdb1
e2label /dev/xvdb1 temp-rootfs
mkswap --label swap /dev/xvdb2

Mounting

mkdir -p /mnt/gentoo
mount /dev/xvdb1 /mnt/gentoo
swapon /dev/xvdb2

Stage3 files

Look on http://distfiles.gentoo.org/releases/ for fresh stage3 file and use url to download latest version.

Downloading files

cd /mnt/gentoo
wget http://distfiles.gentoo.org/releases/amd64/autobuilds/current-stage3-amd64-hardened/stage3-amd64-hardened-20150108.tar.bz2
wget http://distfiles.gentoo.org/releases/snapshots/current/portage-latest.tar.xz

Unpacking files

tar xvpf stage3-*.tar.bz2
tar xvf portage-latest.tar.xz -C /mnt/gentoo/usr

Chrooting

Copy DNS settings from active environment to new one:

cp /etc/resolv.conf /mnt/gentoo/etc/

Mount proc/dev:

mount -t proc none /mnt/gentoo/proc
mount -o bind /dev /mnt/gentoo/dev

Chrooting:

chroot /mnt/gentoo /bin/bash
env-update && source /etc/profile

Configuring

Compiler (GCC)

Options
– Use -mtune=generic to get system which will works on any AWS equipment and any instance type.
– Use -jN where N is number of cpu plus 1 to compile everything with more threads.
– Use custom USE value to provide more information about what will you use.
– Use USE="-perl" to get rid of a lot of useless perl packages.
– Use USE="-bindist" to recompile everything under selected hardened toolchain.
– Use USE="unicode" to use unicode as default everywhere.
– Use USE="threads" to use thread everywhere where it’s supported.

nano /etc/portage/make.conf

Example:

CFLAGS="-O2 -pipe -mtune=generic"
CXXFLAGS="${CFLAGS}"
CHOST="x86_64-pc-linux-gnu"
USE="unicode threads -perl -bindist"
MAKEOPTS="-j2"
PORTDIR="/usr/portage"
DISTDIR="${PORTDIR}/distfiles"
PKGDIR="${PORTDIR}/packages"

OpenRC

Edit rc.conf:

nano /etc/rc.conf

Change values:

rc_logger="YES"
unicode="YES"
rc_sys=""

Locale

Edit locale.gen:

nano /etc/locale.gen

Uncomment this one:

en_US.UTF-8 UTF-8

Generate locales:

locale-gen

Use eselect locale list and eselect locale set to set default locale to UTF-8:

# eselect locale list
Available targets for the LANG variable:
  [1]   C
  [2]   en_US.utf8
  [3]   POSIX
  [ ]   (free form)
# eselect locale set 2
Setting LANG to en_US.utf8 ...
Run ". /etc/profile" to update the variable in your shell.

Keymaps

There is no real keyboard so we no need that service. This service is enabled by default and we need to disable it:

rc-update delete keymaps boot

Console Fonts

There is no real console screen so we no need that service. It’s not enabled by default so no any actions required.

Timezone

We will use “US/Eastern” as example.

ln -sf /usr/share/zoneinfo/US/Eastern /etc/localtime
echo "US/Eastern" > /etc/timezone

Hostname

Edit hostname:

nano /etc/conf.d/hostname

Set your hostname:

hostname="yapb.info"

Please keep in mind that when your creating new instance cloud unit script will replace hostname with default value.

Network

Create new interface and add it to auto start:

ln -s /etc/init.d/net.lo /etc/init.d/net.eth0
rc-update add net.eth0 default

DHCP will be used by default.

Edit hosts:

nano /etc/hosts

Add hostname aliases and FQDN:

127.0.0.1       yapb.info yapb localhost

fstab

Edit fstab:

nano /etc/fstab

We need to comment example lines and add ours:

LABEL="temp-rootfs"     /               ext4            noatime         0 1
LABEL="swap"            none            swap            sw              0 0

Kernel

Kernel sources

Install kernel sources:

emerge gentoo-sources

If you don’t want to recompile kernel on fresh update then protecting current kernel version will be a good choice:

emerge --ask --noreplace gentoo-sources:X.Y.Z

Genkernel

Install genkernel:

emerge genkernel

Edit genkernel:

nano /etc/genkernel.conf

I recommend to tune genkernel options, first related to compile threads and second related to genkernel verbosity level:

MAKEOPTS="-j2"
LOGLEVEL=2

Build kernel

Let’s run auto kernel compilation with menu configuration:

genkernel all --menuconfig

We will use default kernel config. You can optimize it later in any moment, remove useless drivers and tune for your needs. I will focus here on requirements to get gentoo run on AWS XEN HVM.

Most of required XEN modules will be enabled by default with selected options below:

Processor type and features  --->
    [*] Linux guest support  --->
        [*]   Enable paravirtualization code
        [ ]     paravirt-ops debugging (NEW)
        [*]     Paravirtualization layer for spinlocks
        [*]     Xen guest support
        [*]       Support for running as a PVH guest
        [ ]     KVM Guest support (including kvmclock) (NEW)
        [*]     Paravirtual steal time accounting

As of 2015-01-19 XEN frame buffer device is not available for XEN HVM in AWS, and we can get this boot
delay (25 seconds delay):

disable xenbus_probe_frontend driver, otherwise you will get 25 seconds delay in boot with message:
[    6.050120] xenbus_probe_frontend: Waiting for devices to initialise: 25s...20s...15s...10s...5s...0s...
[   31.052175] xenbus_probe_frontend: Timeout connecting to device: device/vfb/0 (local state 3, remote state 1)

To fix that disable xen framebuffer:

Device Drivers  --->
    Graphics support  --->
        Frame buffer Devices  --->
            < > Xen virtual frame buffer support

If you are planning to use benefits of Enhanced Networking on some instances then please also set:

CONFIG_IXGBEVF=y

Module ixgbvf with Intel(R) 10GbE PCI Express Virtual Function Ethernet support available starting from Linux 3.14. If it’s not availabe it could be installed from https://github.com/sormy/gentoo-overlay/tree/master/net-misc/ixgbevf .

Instance should be also tagged to use Enhanced Networking and AMI should be tagged so all instances spawn from that AMI should be able to use Enhanced Networking too.

More details what is that and why you need that is here: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/enhanced-networking.html

Bootloader

We need just to emerge bootloader, we will configure it later:

emerge grub

SSH

Remove password for root and lock password (we will use public key authentication):

passwd -d -l root

Add SSH to autostart:

rc-update add sshd default

Cloud init

You need to install small script, which will set hostname and auto add ssh public keys for root user in case we create new instance from snapshot.

This script is already exists in portage and we can install it. But this scripts is not maintained anymore and masked. The easiest way is to just create /etc/init.d/amazon-ec2 manually:

touch /etc/init.d/amazon-ec2
chmod +x /etc/init.d/amazon-ec2
rc-update add amazon-ec2 boot
nano /etc/init.d/amazon-ec2

Content:

#!/sbin/openrc-run

depend() {
         before hostname
         need net.eth0
}

start() {
        local instance_id=$(wget -t 2 -T 5 -q -O - http://169.254.169.254/latest/meta-data/instance-id)
        [ -f "/var/lib/amazon-ec2" ] && [ "$(cat /var/lib/amazon-ec2)" = "$instance_id" ] && exit 0

        einfo "Fetching metadata from EC2 servers"

        ebegin "  hostname"
        local hostname=$(wget -t 2 -T 5 -q -O - http://169.254.169.254/latest/meta-data/local-hostname)
        echo "hostname=${hostname}" >> /etc/conf.d/hostname
        eend $?

        ebegin "  SSH keys"

        mkdir -p /root/.ssh

        local keys=$(wget -t 2 -T 5 -q -O - http://169.254.169.254/latest/meta-data/public-keys/ | cut -d = -f 1 | xargs echo)
        [ -n "${keys}" ] && \
            wget -t 2 -T 5 -q -O - $(for key in $keys; do echo "http://169.254.169.254/latest/meta-data/public-keys/$key/openssh-key"; done) \
            >> /root/.ssh/authorized_keys \
            2>/dev/null

        if [ -f /root/.ssh/authorized_keys ]; then
            chown root:root /root/.ssh/authorized_keys
            chmod 0600 /root/.ssh/authorized_keys
        fi

        echo "$instance_id" > "/var/lib/amazon-ec2"

        eend $?
}

This init script will fetch AWS metadata and if instance id is different from previous launch then will set hostname and inject ssh keys.

Boot Gentoo from second volume

Exiting from chroot

Exit from chroot env:

exit

Copying kernel

Copy Gentoo kernel/ramdisk to first volume:

cp -v /mnt/gentoo/boot/*genkernel* /boot/
‘/mnt/gentoo/boot/initramfs-genkernel-x86_64-3.17.7-hardened-r1’ -> ‘/boot/initramfs-genkernel-x86_64-3.17.7-hardened-r1’
‘/mnt/gentoo/boot/kernel-genkernel-x86_64-3.17.7-hardened-r1’ -> ‘/boot/kernel-genkernel-x86_64-3.17.7-hardened-r1’
‘/mnt/gentoo/boot/System.map-genkernel-x86_64-3.17.7-hardened-r1’ -> ‘/boot/System.map-genkernel-x86_64-3.17.7-hardened-r1’

Configuring bootloder

Now we are ready to boot into freshly build Gentoo located on second volume. To get that we will modify Ubuntu’s bootloader config.

Edit Ubuntu’s grub config:

nano /boot/grub/grub.cfg

Edit first Ubuntu entry to use gentoo kernel, gentoo ramdisk, gentoo kernel options and gentoo root device:

menuentry 'Ubuntu' ... {
    ...
    linux /boot/kernel-genkernel-x86_64-3.17.7-hardened-r1 root=LABEL=temp-rootfs net.ifnames=0 console=tty1 console=ttyS0
    initrd /boot/initramfs-genkernel-x86_64-3.17.7-hardened-r1
}

Rebooting

reboot

Ok, after successful reboot you will have working gentoo on second volume and inactive ubuntu on first volume, so we can completely erase first drive and start cloning gentoo from second volume to first volume.

Don’t forget that now you need to logon to gentoo under username “root”.

If something will go wrong you can terminate instance, create it again and connect second drive
with gentoo to continue your work:

sudo bash
mkdir /mnt/gentoo && mount /dev/xvdb1 /mnt/gentoo

Clone Gentoo to first volume

Prepare disk

mkfs.ext4 /dev/xvda1
e2label /dev/xvda1 cloudimg-rootfs
mkdir -p /mnt/gentoo
mount /dev/xvda1 /mnt/gentoo

Copy OS

mkdir /mnt/gentoo/run
mkdir /mnt/gentoo/proc
mkdir /mnt/gentoo/mnt
touch /mnt/gentoo/mnt/.keep
mkdir /mnt/gentoo/sys
mkdir /mnt/gentoo/tmp -m 1777
cp -rp /bin /boot /dev /etc /home /lib /lib32 /lib64 /media /opt /root /sbin /usr /var /mnt/gentoo

Configure bootloader

We need to chroot into gentoo on first volume to get easier process:

Mount proc/dev:

mount -t proc none /mnt/gentoo/proc
mount -o bind /dev /mnt/gentoo/dev

Chroot:

chroot /mnt/gentoo /bin/bash
env-update && source /etc/profile

Install bootloader:

grub-install /dev/xvda

Edit grub defaults:

nano /etc/default/grub

Recommended values are:

GRUB_CMDLINE_LINUX="net.ifnames=0 console=tty1 console=ttyS0"
GRUB_TERMINAL=console
GRUB_DEFAULT=0
GRUB_TIMEOUT=0

Explanation:
net.ifnames=0 will prevent kernel to use network device name other than eth0
console=tty1 console=ttyS0 will redirect console so it cab be shown in amazon console
GRUB_TERMINAL=console required to prevent grub using framebuffer device
GRUB_DEFAULT=0 boot first entry by default
GRUB_TIMEOUT=0 boot without waiting

Generate grub config file:

grub-mkconfig -o /boot/grub/grub.cfg

Fix fstab – change LABEL=”rootfs” (second volume) to LABEL=”cloudimg-rootfs” (first volume):

nano /etc/fstab

Rebooting

reboot

Finalize Gentoo installation

Fix hostname

nano /etc/conf.d/hostname
hostname="gentoo"

Fix hosts

nano /etc/hosts
127.0.0.1       gentoo.local gentoo localhost

Cleanup

Remove stage3 and snapshot files from /:

rm -v /stage3-* /portage-*

Detach and remove in AWS console attached second volume.

3rd party tools

Install everything you need for your minimal image:

emerge --update --newuse --deep world && revdep-rebuild
emerge eix && eix-update
emerge gentoolkit
emerge app-misc/mc
emerge syslog-ng
emerge logrotate
...

Force filesystem check and fix on reboot

touch /forcefsck

AMI image

Ok, now you can create snapshot from first volume, create AMI image from snapshot and use created
image to spawn new gentoo instances in a few seconds.

Alternatives

Compiling in cloud on cheap t2 instances could be very very very slow if you don’t have enough CPU credits.

There are a few different options you could consider:

  • You could spawn CPU-optimized to build image and then spawn instance you need from created image.
  • You could build image locally and import it as EC2 image and then spawn instance you need from created image.
  • You could create local binary repo and make your cloud instance install packages from precompiled binaries prepared from your local repo. Good for enterprise, you could have one or a few build servers and distribute binary packages to whole gentoo fleet

Сonclusion

Now you have custom minimal AMI image which can be used to spawn instances.
You can create your custom AMI images for different purposes based on this minimal AMI image.

Please let me know if you found something wrong in this article.

Good luck 😉

9 Responses to Gentoo on AWS

  1. Hi Artem,

    Great blog about Gentoo on AWS.

    I made a mistake in my current Gentoo EC2 instance.
    I used -march=native in CFLAGS and now i can not start another instance from latest snapshot.

    Do i need to recompile whole system with CFLAGS=”-O2 -pipe” or only some important packages?
    How can i find necessary packages for the recompilation.

    Thank you.

    • You will need to recompile world with new compile options: emerge world –emptytree
      Update: You are right, you could recompile only packages with wrong GCC options if you know what are the packages. You could try to identify that list in /var/log/portage/elog/summary.log. Otherwise the only 100% way is to recompile world with –emptytree

  2. One gotcha for me was:

    mount -o bind /dev /mnt/gentoo/dev

    Should be:

    mount –rbind /dev /mnt/gentoo/dev

    After that, everything went swimmingly well.

    Thanks for making this guide!

  3. Worked like a charm! was able to create an AMI and launch a new instance no problem. Only two things I had to do differently was that the /etc/init.d/amazon-ec2 file didn’t exist but since you had the entire file was able to recreate and i had to do grub-install instead of grub2-install. Thanks!!!

  4. Great blog. One thing I noticed is that in my experience it requires more than 13G to genkernel. Probably you want to allocate more disk space for /dev/sdb to avoid the interrupt.

    BTW, c4.large is way more faster than t2.micro to get the kernel generated like 2 hours vs 1 day.

    • t* instances has burstable performance, if you run out of cpu credits then compilation could take 1 day 🙂 Based on my experience t2.micro could compile kernel in 1 hour if you have enough cpu credits.

  5. I can’t get past “ClientError: Unsupported kernel version 4.xx.yy”. Amazon Linux uses 4.9.38 but since that’s not available from either vanilla-sources or gentoo-sources I just pulled down the tarball from kernel.org and configured it per the instructions here. I don’t know if maybe this is the error that’s thrown if there’s something about the kernel config that’s unworkable or if it’s maybe comparing checksums against stock kernels that come with specific distro versions. Any ideas?

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.