Keeping You Honest

Extending Puppet 3’s Databindings to Types

by , under Puppet

I know a lot of you are looking forward to Part 3 It is being worked on, but writing the modules for it is going a little slow.

I’ve been toying with some ideas, and been discussing them with various members of the community and PuppetLabs devs. So I figured I would give it a quick write up to hash out a few last details before I submit a feature One request.

One of the biggest new things to happen to Puppet was databinding. This feature allows you to automagically externalize your data just be setting the class up as a parametrized class.

class foo (
  $param1 = 'bar',
){
  ...
}

And then just add the following to the proper yaml file based on your hierarchy: foo::param1: ‘bat’

But this is limited to classes. And I started to wonder… why? Why can’t this be used with defined types? And of course, if it is used with defined types, then why not let it be used with built in types and custom types? And I can’t really see any reason why databindings shouldn’t be available across the board.

This, however, brings us to a problem.  With a class, the lookup key is class::param_name. Classes are singletons, so this is not a problem.  However, defined types and built in resources are not – they can be used over and over as long as the title/name is unique. So how then do we address the lookups?

My first thought was to use type::title::param. For example, if we have a module named apache2 and it has a define for vhosts, it would be apache2::vhosts when you reference it. Common parameters for a vhost define are documentroot, port, template etc.  These could use a key of apache2::vhosts::<$title>::documentroot or apache2::vhosts::<$title>::port. For most cases, and I believe most if not all built in types, this will work fine.

However, there are some modules that use top level defines. For example, the concat module has a define that is simply named “concat”. And there is a concat::setup class. Now, let’s say that the concat::setup class and the concat define both take a parameter called x. (this is a bit contrived, but it illustrates the point, and concat does use such a top level define). The parameter for the class would be concat::setup::x. Now, we are using the define, and we add concat { ‘setup': } to the catalog.  Now we have a problem… this would also be referenced as concat::setup::x.

So that is one potential problem – and a big one for those cases where it pops up. Personally, I don’t know of too many “real” cases where this would be a problem, but I have been told it could happen quite often.

Another possible idea would be to put all resources into a special namespace for the key lookups.  Such as type::resource::<$title>::param. Using our example from above, the lookup key would then be type::concat::setup:x .  Of course, tthe fixed string doesn’t have to be “type” – it could be whatever is deemed reasonable. This, of course, doesn’t entirely remove the problem, just narrows the scope in which it can happen. In all honesty, I am not sure on what the best answer is. But I would really like to be able like to be able to do something like user { ‘bob': } and have it automagically pull in the parameters.

 

I do realize that I can do something close to this using create_resources(), but you can’t chain to functions and wrapping create_resources() in anchor tags is a bit tedious, not to mention it makes the code harder to read.

If you have any thoughts or ideas on how to do this, please leave me a comment here, look me up on IRC (FriedBob in #puppet and #puppet-dev on Freenode) or twitter ( @FriedBob).

Posted on Feb.08, 2013

1 Comment :, , , , more...

Puppet Roles and Profiles with a Simple Module Structure (part 2)

by , under Puppet

In part 1 I discuss the inspiration for this series and also the overall goals. If you haven’t read it, it would be beneficial to read it before reading this article.

I will be using a top down approach for the design, but will build the modules bottom up.

Node List

In this environment we will have:

  • 1 database cluster
  • 3 app server clusters
  • 1 web server cluster
  • 1 ldap server cluster
  • 1 load balancer cluster
  • 1 content server.

Each cluster, other than the app and web server clusters, each cluster will consist of two nodes, in either a Master/slave or “hot standby” type configuration. The web and app servers will each have three nodes.

Role List

For this environment we will have 19 nodes that we will be managing. Of these 19, there are only 6 distinct roles, so we can go ahead and define them here.

  • Database Role
  • App Server Role
  • Web Server Role
  • LDAP Role
  • Load Balancer Role
  • Content Server Role

These roles will become the top level classes that will be assigned by either our ENC or in the site.pp based node defs. Which one doesn’t really matter and is more a matter of personal taste and company policy and what works for you and your team. For simplicity’s sake, I will assume we are using the site.pp method.

Profile List

Now we can define our profiles. Each role has a few different variants, or profiles that will be needed. We shall list these now.

  • Database
    • Master
    • Slave
  • App Server
    • Cluster 1
    • Cluster 2
    • Cluster 3
  • Web Server
  • LDAP
    • Master
    • Replica
  • Load Balancer
    • Master
    • Backup
  • Content Server

You may notice that the content server and the web server roles do not actually have any defined profiles. These will be using the “default” profile for their roles, as each node in the web server cluster should be configured the same, with the host name as the exception. And the content server is a singleton. Not a true snowflake, but not clustered at this point in time.

 Module List

At this point, we have worked our way down to the point where we can start talking about application stacks and specific configurations and environments. For the purpose of this article, everything will be in a single environment, but it’d be trivial to do this with additional environments. Instead of Database -> Master profile,  you’d use something like Database -> Master -> Test or Database -> Master_test for the profile, and just create additional Master and Slave profiles per environment.

There are certain things that are common to all nodes, this would be things like puppet, ssh, 7zip, unzip, ntp, admin users, package manager repo sources (apt/yum/whatever), etc.. These common, shared items will all go into a base module. The exact name doesn’t matter, but I would I would avoid ‘default’ or ‘common’ to avoid conflicts. Names like ‘base’ or ‘basenode’ are fairly common.

Even though each role has a couple different profiles, the core application stack is the same. The only difference is how they are configured.

  • Database Server
    • PostgreSQL
  • App Server
    • Tomcat
    • Java
  • Web Server
    • Apache
  • LDAP
    • OpenLDAP
  • Load Balancer
    • Keepalived
  • Content Server
    • Alfresco

You may have noticed, but at this point, we still don’t say anything about versions or such. That will be a parametrized option that will be discussed in part 3, as long as the other code level details for much of this.

 

Posted on Jan.12, 2013

2 Comments :, , more...

Puppet Roles and Profiles with a Simple Module Structure (part 1)

by , under Puppet

Introduction and Background

Since starting to work with Puppet Labs’ Puppet at my current position, I have been constantly looking to better understand how to use puppet, and for ways to improve how my team uses puppet so we can do more with less. In the process of this, I do a lot of reading and try to stay active on the mailing list, IRC and the new ask.puppetlabs site, both to further my own understanding and to give back to the community some.

I recently came across a pair of blog posts that really caught my eye.  The first one was “Simple Puppet Module Structure Redux” by R.I.Pienaar. This post describes a very nice pattern that is a blend of the package – config – service pattern and a modified anchor pattern. It does rely on Puppet 3’s databinding, so does not use the params.pp pattern yet does have the same sort of flexibility. After reading it, I really liked it. But I did realize that it was really only applicable to a certain type of puppet module. What I consider to be the building blocks of a node, or application level modules. For modules like the classic example of NTP, or Apache2 or Tomcat, this pattern is amazing. However, it doesn’t necessarily fit with higher level modules, that are pulling in the different building block modules and how those are bound together to create servers and clusters an infrastructure. I was giving this some thought, and trying to catch R.I. online and I came across another blog post that was referenced on the users mailing list.

This post was titled “Designing Puppet – Roles and Profiles” and written by Craig Dunn. This one is about using layers of abstraction to reduce the learning curve of an infrastructure, make it more self-documenting and reduce code duplication. After some thought, I decided to implement both of these patterns with my own puppet use. I have not yet done so, but have started reworking them. After some discussions on IRC, I have decided to document this process and give some examples on how these patterns can work together. I will not be sharing the actual code I use at work, but I will be creating some real world examples and documenting the general process.

For the purposes of this series of posts, I will be puppetizing the following infrastructure:

  • 1 database cluster
  • 3 app server clusters
  • 1 web server cluster
  • 1 ldap cluster
  • 1 load balancer cluster
  • 1 content server

Each web server and app server cluster consists of three nodes that are identical save for the hostname. The ldap has one master and one replica. They both serve reads but only the master handles writes. The load balancer cluster is two servers, one an active master and the other a hot standby. The database cluster consists of a master and a slave that is replicated to and can serve as a hot standby. The load balancer is in front, and balances ldap and web requests. Each web server node also serves as a balancer for the app server clusters. All nodes will be running the same OS, in order to simplify the blog post, and because that is how I do things at my job.

Current State of Affairs

Currently, at least in my environment, my puppet code is a mix of things from back before “I knew better”, prior best practice “flavor of the week” patterns, and code that lays somewhere in between. It is inconsistent, and not always easy to maintain or understand.  This is further complicated by the fact that there have been several different people working on the codebase, each with their own style and idea of what good puppet code should look like.

Part two will focus on how the planning and design phases, of how to get an existing mess into something that will be easy to maintain and manage. Part three will be filled with examples – both in terms of how to lay things out on the file system to be able to take advantage of the autoloader and Puppet 3’s databindings (I’ll even throw in a way you can get databinding like functionality in puppet 2.7.x) and actual code examples for roles, profiles and the application/service level modules that form the foundation for it all.

Posted on Jan.11, 2013

3 Comments :, , , more...

Looking for something?
You can search for it here.

Custom Search

Still not finding what you're looking for? Drop a comment on a post or contact us so we can take care of it!