hadooppowered.com

Setup Puppetmaster with PuppetDB and Puppetboard

Intro

Managing your infrastructure will become a nightmare if you have to manually configure ten's or thousands of servers. There are some systems to handle that job for you.

To name a few:

All of them have their pro's and con's but I like puppet the most and will show you how to get up and running with it today on a Debian system. You should have at least two servers at hand. The best solution for testing is probably Vagrant.

If you have no idea what Vagrant is, try to follow my Vagrant Tutorial.

All steps below are fully tested with Debian Wheezy (Debian 7.x), especially for the puppetmaster, the puppetdb and the puppetboard. The puppet agent will also run on Debian Squeeze (Debian 6.x).

Setup the puppet master

First you have to install the puppet master. The master itself will be under control of puppet later. Keep in mind that this service may need some more ressources (RAM, HDD and CPU) if you add more and more nodes.

$ cd /tmp/
$ wget http://apt.puppetlabs.com/puppetlabs-release-wheezy.deb
$ dpkg -i puppetlabs-release-wheezy.deb
$ apt-get update

$ apt-get install -y puppetmaster puppet

The puppet agents look by default for the master at the hostname puppet. You should configure your hosts file to have the puppetmaster there. Here's the sample /etc/hosts file for the master:

$ vi /etc/hosts
127.0.0.1   localhost
127.0.1.1   master.vagrantup.com master puppet

Setup your hostname correctly! Here's the sample /etc/hostname file for the master:

$ vi /etc/hostname
master

Make sure facter fqdn and hostname -f return the correct fqdn for your master:

$ facter fqdn
master.vagrantup.com

$ hostname -f
master.vagrantup.com

Just to make sure everything works properly, restart the services.

$ service puppetmaster restart
$ service puppet restart

Setup the puppet agent(s)

This step must be done on all machines you want to configure with your puppet master.

For Debian Wheezy (Debian 7.x)

$ cd /tmp/
$ wget http://apt.puppetlabs.com/puppetlabs-release-wheezy.deb
$ dpkg -i puppetlabs-release-wheezy.deb
$ apt-get update

$ apt-get install -y puppet

For Debian Squeeze (Debian 6.x)

$ cd /tmp/
$ wget https://apt.puppetlabs.com/puppetlabs-release-squeeze.deb # note the difference in the package name
$ dpkg -i puppetlabs-release-squeeze.deb
$ apt-get update

$ apt-get install -y puppet

The puppet agents look by default for the master at the hostname puppet. You should configure your hosts file to have the puppetmaster there. Here's the sample /etc/hosts file for one agent:

$ vi /etc/hosts
127.0.0.1   localhost
127.0.1.1   agent.vagrantup.com agent

172.28.128.3    master.vagrantup.com    master  puppet

Setup your hostname correctly! Here's the sample /etc/hostname file for one agent:

$ vi /etc/hostname
agent

Make sure facter fqdn and hostname -f return the correct fqdn for your agents

$ facter fqdn
agent.vagrantup.com

$ hostname -f
agent.vagrantup.com

Just to make sure everything works properly, restart the services.

service puppet restart

Add the agents to the master

To add a new node (agent) to the puppet master you must generate certificates.

On every agent node:

$ puppet agent --waitforcert 60 --test # should work, if not use the command below 
$ puppet agent --server master.vagrantup.com --waitforcert 60 --test

On the puppet master:

$ puppet cert --list
    --> "agent.vagrantup.com" (MD5) B3:F7:6D:B7:D3:0E:CD:CC:C2:DF:A3:ED:0A:E4:8B:6D

$ puppet cert sign agent.vagrantup.com

Create a default manifest

The first thing you need is a basic manifest for configuring your nodes.

Must-Have default modules

Puppet can use community created modules in your manifests. Here are a few modules I used for this tutorial. You must install them the follow the steps below.

$ puppet module install saz-locales            # https://forge.puppetlabs.com/saz/locales
$ puppet module install saz-timezone           # https://forge.puppetlabs.com/saz/timezone
$ puppet module install saz-vim                # https://forge.puppetlabs.com/saz/vim
$ puppet module install puppetlabs-git         # https://forge.puppetlabs.com/puppetlabs/git
$ puppet module install stephenrjohnson-puppet # https://forge.puppetlabs.com/stephenrjohnson/puppet
$ puppet module install puppetlabs-ntp         # https://forge.puppetlabs.com/puppetlabs/ntp

Create your first default manifest

The easiest step is to create the default site.pp in /etc/puppet/manifests/.

We will use the "old" method here and use an import for included host manifests and you will get a warning later:

This is method is deprecated and will not work in Puppet 4.0.

But let's stick to the "old" method because it's easier to understand and to get started quickly.

# /etc/puppet/manifests/site.pp

import 'nodes/*.pp'

node default {

    # configure the puppet agents
    class { 'puppet::agent':
        puppet_run_style    => 'service',
        puppet_server       => 'master.vagrantup.com',
        environment         => 'production',
        puppet_run_interval => '10',
        report              => 'true'
    }

    # set the default locales
    class { 'locales':
        default_locale => 'en_US.UTF-8',
        locales        => ['en_US.UTF-8 UTF-8','de_DE.UTF-8 UTF-8']
    }

    # configure ntp
    class { '::ntp':
        servers => [ 'ptbtime1.ptb.de','ptbtime2.ptb.de','ptbtime3.ptb.de',
            '0.europe.pool.ntp.org','1.europe.pool.ntp.org',
            '2.europe.pool.ntp.org','3.europe.pool.ntp.org'
            ]
    }

    # set the timezone on all servers to UTC
    class { 'timezone':
        timezone => 'UTC'
    }

    # install some common system tools
    Package { ensure => 'installed' }
    $systemtools = [ 'dstat','htop','screen','sudo',
        'ack','bzip2','less','lsof','netcat','rsync',
        'strace','telnet','traceroute','wget','zip'
        ]
    package { $systemtools: }

    # install vim and set some .vimrc defaults
    class { 'vim':
        opt_misc => ['nocompatible','hlsearch', 'showcmd',
        'showmatch','ignorecase','smartcase',
        'incsearch','autowrite','hidden',
        'smartindent','tabstop=4','shiftwidth=4',
        'expandtab']
    }

    # install source code management systems
    class { 'git': }

}

Install puppetdb

Install the puppet module for puppetdb first:

$ puppet module install puppetlabs-puppetdb # https://forge.puppetlabs.com/puppetlabs/puppetdb , https://docs.puppetlabs.com/puppetdb/latest/install_via_module.html

Create the manifest for the master-host:

# /etc/puppet/manifests/nodes/master.vagrantup.com.pp

node 'master.vagrantup.com' inherits default {
    ...
    # Configure puppetdb and its underlying database
    class { 'puppetdb':
        # database_username => 'puppetdb',
        # database_password => 'puppetdb',
        # database_name     => 'puppetdb',
        report_ttl        => '7d'
    }

    # Configure the puppet master to store reports
    class { 'puppetdb::master::config':
        manage_report_processor => true,
        enable_reports          => true
    }
    ...
}

Install puppetboard

Install the puppet module for puppetboard and apache webserver first:

$ puppet module install nibalizer-puppetboard # https://forge.puppetlabs.com/nibalizer/puppetboard
$ puppet module install puppetlabs-apache # https://forge.puppetlabs.com/puppetlabs/apache

Create the manifest for the master-host:

# /etc/puppet/manifests/nodes/master.vagrantup.com.pp

node 'master.vagrantup.com' inherits default {
...
    # Configure python
    class { 'python':
        virtualenv => true,
        pip        => true
    }

    # Configure Apache
    class { 'apache': }
    class { 'apache::mod::wsgi': }

    # Configure puppetboard
    class { 'puppetboard':
        unresponsive => '1'
    }

    # Configure vhost for puppetboard
    class { 'puppetboard::apache::vhost':
        vhost_name => 'master.vagrantup.com',
        port       => '8088'
    }
...
}

Disable the default vhost and reload the apache config:

a2dissite *default*
service apache2 reload

Site is served on: master.vagrantup.com:8088

Full sample manifests from this tutorial

site.pp

# /etc/puppet/manifests/site.pp

import 'nodes/*.pp'

node default {

    # configure the puppet agents
    class { 'puppet::agent':
        puppet_run_style    => 'service',
        puppet_server       => 'master.vagrantup.com',
        environment         => 'production',
        puppet_run_interval => '10',
        report              => 'true'
    }

    # set the default locales
    class { 'locales':
        default_locale => 'en_US.UTF-8',
        locales        => ['en_US.UTF-8 UTF-8','de_DE.UTF-8 UTF-8']
    }

    # configure ntp
    class { '::ntp':
        servers => [ 'ptbtime1.ptb.de','ptbtime2.ptb.de','ptbtime3.ptb.de',
            '0.europe.pool.ntp.org','1.europe.pool.ntp.org',
            '2.europe.pool.ntp.org','3.europe.pool.ntp.org'
            ]
    }

    # set the timezone on all servers to UTC
    class { 'timezone':
        timezone => 'UTC'
    }

    # install some common system tools
    Package { ensure => 'installed' }
    $systemtools = [ 'dstat','htop','screen','sudo','ack' ]
    package { $systemtools: }

    # install vim and set some .vimrc defaults
    class { 'vim':
        opt_misc => ['nocompatible','hlsearch', 'showcmd',
        'showmatch','ignorecase','smartcase',
        'incsearch','autowrite','hidden',
        'smartindent','tabstop=4','shiftwidth=4',
        'expandtab']
    }

    # install source code management systems
    class { 'git': }

}

master.vagrantup.com.pp

# /etc/puppet/manifests/nodes/master.vagrantup.com.pp

node 'master.vagrantup.com' inherits default {

    # Configure puppetdb and its underlying database
    class { 'puppetdb':
        # database_username => 'puppetdb',
        # database_password => 'puppetdb',
        # database_name     => 'puppetdb',
        report_ttl        => '7d'
    }

    # Configure the puppet master to store reports
    class { 'puppetdb::master::config':
        manage_report_processor => true,
        enable_reports          => true
    }

    # Configure python
    class { 'python':
        virtualenv => true,
        pip        => true
    }

    # Configure Apache
    class { 'apache': }
    class { 'apache::mod::wsgi': }

    # Configure puppetboard
    class { 'puppetboard':
        unresponsive => '1'
    }

    # Configure vhost for puppetboard
    class { 'puppetboard::apache::vhost':
        vhost_name => 'master.vagrantup.com',
        port       => '8088'
    }

}

Notes

Read the puppet documentation!

Use puppet modules!

Whenever possible use the already existing puppet modules from puppetlabs.com. Report bugs and send pull requests for enhancements back!

Validate and lint your files!

$ puppet parser validate filename.pp
$ puppet-lint filename.pp

Install puppet-lint for your vim environment! --> ToDo: Create a tutorial for this!

Multiple default providers warning

The warning following warnig is an open issue:

Warning: Found multiple default providers for vcsrepo: bzr, dummy, svn, hg, cvs, git; using bzr

Follow the ticket: https://tickets.puppetlabs.com/browse/MODULES-428

Workaround

$ mkdir -p /etc/puppet/modules/vcsrepo/lib/facter
$ vi /etc/puppet/modules/vcsrepo/lib/facter/vcsrepo.rb
Facter.add('vcsrepo') do
  setcode do
    "dummy"
  end
end

Puppet deprecated warning

When running puppet parser validate /etc/puppet/manifests/site.pp you will get the following warning

Warning: The use of 'import' is deprecated at /etc/puppet/manifests/site.pp:5. See http://links.puppetlabs.com/puppet-import-deprecation

Use the external node classifier to prevent the deprecated warning. You can find the guide for configuring ENC at the Puppetlabs References.


No public comments, but you can email comments@hadooppowered.com or tweet @hadooppowered.


← Older
Newer →