What is the -top- extension, and how do I use it in CFEngines Mustache templating method?

The -top- extension to the mustache template method, first introduced in CFEngine 3.9.0, is a special key representing the complete data given to the templating engine. This is useful for iterating over the top level of a container {{#-top-}} ... {{/-top-}} and rendering json representation of data given with $ and %. Note, when iterating over -top- you can expand the current iterations key with @ and value with ..

Let's take a look at a couple small examples:

Iterating over -top- for rendering simple key=value configuration files

Many configuration file formats like /etc/sysctl.conf are simple key = value.

In this example, we see how data for /etc/sysctl.conf can be modeled for simple rendering using -top-.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
  bundle agent main
  {
    vars:

        # A simple data structure holding sysctl keys and values
        "c" data => '{
    "net.ipv6.conf.forwarding": "0",
    "net.ipv6.conf.all.forwarding": "0",
    "fs.protected_hardlinks": "1",
    "fs.protected_symlinks": "1",
    "vm.swappiness": "10"
  }';

    reports:
        "CFEngine $(sys.cf_version)";

        # A report to see what a rendered file would look like
        "/etc/sysctl.conf$(const.n)$(with)"
          with => string_mustache( "# Rendered by CFEngine$(const.n){{#-top-}}{{{@}}}={{{.}}}$(const.n){{/-top-}}",
                                   c);
  }

The above policy results in this output:

R: CFEngine 3.12.0
R: /etc/sysctl.conf
# Rendered by CFEngine
net.ipv6.conf.forwarding=0
net.ipv6.conf.all.forwarding=0
fs.protected_hardlinks=1
fs.protected_symlinks=1
vm.swappiness=10

It's easy to take this reports prototype and turn it into a real files promise. Let's take this example one step further and make it a bit more practical.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
  bundle agent main
  {
    vars:

        # Let's make it easy for this policy to be applied to a temporary
        # location. Note, an empty string will result in no prefix. This is a
        # useful pattern when combined with classes.

        "PREFIX" string => "/tmp";

        # A simple data structure holding sysctl keys and values
        "c" data => '{
    "net.ipv6.conf.forwarding": "0",
    "net.ipv6.conf.all.forwarding": "0",
    "fs.protected_hardlinks": "1",
    "fs.protected_symlinks": "1",
    "vm.swappiness": "10"
  }';

    files:
        "$(PREFIX)/etc/sysctl.conf"
          create => "true",
          template_method => "inline_mustache",
          edit_template_string => "# Rendered by CFEngine$(const.n){{#-top-}}{{{@}}}={{{.}}}$(const.n){{/-top-}}",
          template_data => @(c);


    reports:
        "CFEngine $(sys.cf_version)";

        # A report to see what a rendered file would look like
        "${PREFIX}/etc/sysctl.conf"
          printfile => cat( $(this.promiser) );
  }

Here is the policy output:

R: CFEngine 3.12.0
R: /tmp/etc/sysctl.conf
R: # Rendered by CFEngine
R: net.ipv6.conf.forwarding=0
R: net.ipv6.conf.all.forwarding=0
R: fs.protected_hardlinks=1
R: fs.protected_symlinks=1
R: vm.swappiness=10

Rendering data for direct use and debugging

The -top- extension, combined with $ or % is able to render data provided to the template as JSON. This is useful in preparing requests for REST APIs and general debugging in cases where you are unsure of the exact data provided.

For example, if data is not explicitly provided to a template via template_data, datastate() is used. Let's take a look at a small example where we render a couple of data structures.

This example shows rendering JSON data for use in communication with a REST API.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
  bundle agent main
  {

    vars:

        "request"
          data => '{ "name": "morpheus", "job": "leader" }';

        "response"
          string => execresult( 'curl -s -X POST https://reqres.in/api/users -d @/tmp/request.json -H "Content-Type: application/json"', useshell),
          depends_on => { "request_ready" };

    files:
        "/tmp/request.json"
          handle => "request_ready",
          create => "true",
          template_method => "inline_mustache",
          edit_template_string => "{{$-top-}}",
          template_data => @(request);

    reports:
        "CFEngine $(sys.cf_version)";

        "Posted Request"
          printfile => cat( "/tmp/request.json" );

        "Response$(const.n)$(response)"
          if => isvariable( response );
  }

We use the freely available https://reqres.in/ to show a simple way to interact with an API.

And here is the policy output:

R: CFEngine 3.12.0
R: Posted Request
R: {"job":"leader","name":"morpheus"}
R: Response
{"job":"leader","name":"morpheus","id":"771","createdAt":"2018-11-16T15:47:56.473Z"}

Many times it's useful to inspect a data structure and the % renders multi-line JSON to make it easier to read. In this example, we find all the variables in the paths bundle and merge them together into a single data structure.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
  bundle agent main
  {

    vars:

        "paths" data => variablesmatching_as_data( "default:paths\..*");

    reports:
        "CFEngine $(sys.cf_version)";

        "Variable data from bundle paths$(const.n)$(with)"
          # WARNING the escaped % is an artifacct of hugo, remove the backslash!
          with => string_mustache( "{{\%-top-}}", paths );
  }

Here is the policy output:

R: CFEngine 3.12.0
R: Variable data from bundle paths
{
  "default:paths.all_paths": [
    "groupadd",
    "usermod",
    "iptables_save",
    "ifconfig",
    "groupdel",
    "dmidecode",
    "egrep",
    "getfacl",
    "createrepo",
    "mailx",
    "logger",
    "ethtool",
    "service",
    "free",
    "diff",
    "curl",
    "netstat",
    "virtualenv",
    "apt_config",
    "cat",
    "sysctl",
    "tr",
    "hostname",
    "bc",
    "dc",
    "svc",
    "dpkg_divert",
    "apt_get",
    "echo",
    "crontab",
    "nologin",
    "test",
    "npm",
    "cksum",
    "ip",
    "userdel",
    "awk",
    "ping",
    "groupmod",
    "dpkg",
    "lsof",
    "env",
    "dig",
    "perl",
    "cut",
    "domainname",
    "apt_cache",
    "wc",
    "tar",
    "pgrep",
    "update_alternatives",
    "aptitude",
    "useradd",
    "apt_key",
    "pip",
    "update_rc_d",
    "find",
    "init",
    "df",
    "grep",
    "printf",
    "realpath",
    "sort",
    "shadow",
    "systemctl",
    "chkconfig",
    "ls",
    "crontabs",
    "lsattr",
    "wget",
    "getent",
    "sed",
    "iptables",
    "git"
  ],
  "default:paths.apt_cache": "/usr/bin/apt-cache",
  "default:paths.apt_config": "/usr/bin/apt-config",
  "default:paths.apt_get": "/usr/bin/apt-get",
  "default:paths.apt_key": "/usr/bin/apt-key",
  "default:paths.aptitude": "/usr/bin/aptitude",
  "default:paths.awk": "/usr/bin/awk",
  "default:paths.bc": "/usr/bin/bc",
  "default:paths.cat": "/bin/cat",
  "default:paths.chkconfig": "/sbin/chkconfig",
  "default:paths.cksum": "/usr/bin/cksum",
  "default:paths.createrepo": "/usr/bin/createrepo",
  "default:paths.crontab": "/usr/bin/crontab",
  "default:paths.crontabs": "/var/spool/cron/crontabs",
  "default:paths.curl": "/usr/bin/curl",
  "default:paths.cut": "/usr/bin/cut",
  "default:paths.dc": "/usr/bin/dc",
  "default:paths.df": "/bin/df",
  "default:paths.diff": "/usr/bin/diff",
  "default:paths.dig": "/usr/bin/dig",
  "default:paths.dmidecode": "/usr/sbin/dmidecode",
  "default:paths.domainname": "/bin/domainname",
  "default:paths.dpkg": "/usr/bin/dpkg",
  "default:paths.dpkg_divert": "/usr/bin/dpkg-divert",
  "default:paths.echo": "/bin/echo",
  "default:paths.egrep": "/bin/egrep",
  "default:paths.env": "/usr/bin/env",
  "default:paths.ethtool": "/sbin/ethtool",
  "default:paths.find": "/usr/bin/find",
  "default:paths.free": "/usr/bin/free",
  "default:paths.getent": "/usr/bin/getent",
  "default:paths.getfacl": "/usr/bin/getfacl",
  "default:paths.git": "/usr/bin/git",
  "default:paths.grep": "/bin/grep",
  "default:paths.groupadd": "/usr/sbin/groupadd",
  "default:paths.groupdel": "/usr/sbin/groupdel",
  "default:paths.groupmod": "/usr/sbin/groupmod",
  "default:paths.hostname": "/bin/hostname",
  "default:paths.ifconfig": "/sbin/ifconfig",
  "default:paths.init": "/sbin/init",
  "default:paths.ip": "/sbin/ip",
  "default:paths.iptables": "/sbin/iptables",
  "default:paths.iptables_save": "/sbin/iptables-save",
  "default:paths.logger": "/usr/bin/logger",
  "default:paths.ls": "/bin/ls",
  "default:paths.lsattr": "/usr/bin/lsattr",
  "default:paths.lsof": "/usr/bin/lsof",
  "default:paths.mailx": "/usr/bin/mailx",
  "default:paths.netstat": "/bin/netstat",
  "default:paths.nologin": "/usr/sbin/nologin",
  "default:paths.npm": "/usr/bin/npm",
  "default:paths.path[apt_cache]": "/usr/bin/apt-cache",
  "default:paths.path[apt_config]": "/usr/bin/apt-config",
  "default:paths.path[apt_get]": "/usr/bin/apt-get",
  "default:paths.path[apt_key]": "/usr/bin/apt-key",
  "default:paths.path[aptitude]": "/usr/bin/aptitude",
  "default:paths.path[awk]": "/usr/bin/awk",
  "default:paths.path[bc]": "/usr/bin/bc",
  "default:paths.path[cat]": "/bin/cat",
  "default:paths.path[chkconfig]": "/sbin/chkconfig",
  "default:paths.path[cksum]": "/usr/bin/cksum",
  "default:paths.path[createrepo]": "/usr/bin/createrepo",
  "default:paths.path[crontab]": "/usr/bin/crontab",
  "default:paths.path[crontabs]": "/var/spool/cron/crontabs",
  "default:paths.path[curl]": "/usr/bin/curl",
  "default:paths.path[cut]": "/usr/bin/cut",
  "default:paths.path[dc]": "/usr/bin/dc",
  "default:paths.path[df]": "/bin/df",
  "default:paths.path[diff]": "/usr/bin/diff",
  "default:paths.path[dig]": "/usr/bin/dig",
  "default:paths.path[dmidecode]": "/usr/sbin/dmidecode",
  "default:paths.path[domainname]": "/bin/domainname",
  "default:paths.path[dpkg]": "/usr/bin/dpkg",
  "default:paths.path[dpkg_divert]": "/usr/bin/dpkg-divert",
  "default:paths.path[echo]": "/bin/echo",
  "default:paths.path[egrep]": "/bin/egrep",
  "default:paths.path[env]": "/usr/bin/env",
  "default:paths.path[ethtool]": "/sbin/ethtool",
  "default:paths.path[find]": "/usr/bin/find",
  "default:paths.path[free]": "/usr/bin/free",
  "default:paths.path[getent]": "/usr/bin/getent",
  "default:paths.path[getfacl]": "/usr/bin/getfacl",
  "default:paths.path[git]": "/usr/bin/git",
  "default:paths.path[grep]": "/bin/grep",
  "default:paths.path[groupadd]": "/usr/sbin/groupadd",
  "default:paths.path[groupdel]": "/usr/sbin/groupdel",
  "default:paths.path[groupmod]": "/usr/sbin/groupmod",
  "default:paths.path[hostname]": "/bin/hostname",
  "default:paths.path[ifconfig]": "/sbin/ifconfig",
  "default:paths.path[init]": "/sbin/init",
  "default:paths.path[ip]": "/sbin/ip",
  "default:paths.path[iptables]": "/sbin/iptables",
  "default:paths.path[iptables_save]": "/sbin/iptables-save",
  "default:paths.path[logger]": "/usr/bin/logger",
  "default:paths.path[ls]": "/bin/ls",
  "default:paths.path[lsattr]": "/usr/bin/lsattr",
  "default:paths.path[lsof]": "/usr/bin/lsof",
  "default:paths.path[mailx]": "/usr/bin/mailx",
  "default:paths.path[netstat]": "/bin/netstat",
  "default:paths.path[nologin]": "/usr/sbin/nologin",
  "default:paths.path[npm]": "/usr/bin/npm",
  "default:paths.path[perl]": "/usr/bin/perl",
  "default:paths.path[pgrep]": "/usr/bin/pgrep",
  "default:paths.path[ping]": "/bin/ping",
  "default:paths.path[pip]": "/usr/bin/pip",
  "default:paths.path[printf]": "/usr/bin/printf",
  "default:paths.path[realpath]": "/usr/bin/realpath",
  "default:paths.path[sed]": "/bin/sed",
  "default:paths.path[service]": "/usr/sbin/service",
  "default:paths.path[shadow]": "/etc/shadow",
  "default:paths.path[sort]": "/usr/bin/sort",
  "default:paths.path[svc]": "/usr/sbin/service",
  "default:paths.path[sysctl]": "/sbin/sysctl",
  "default:paths.path[systemctl]": "/bin/systemctl",
  "default:paths.path[tar]": "/bin/tar",
  "default:paths.path[test]": "/usr/bin/test",
  "default:paths.path[tr]": "/usr/bin/tr",
  "default:paths.path[update_alternatives]": "/usr/bin/update-alternatives",
  "default:paths.path[update_rc_d]": "/usr/sbin/update-rc.d",
  "default:paths.path[useradd]": "/usr/sbin/useradd",
  "default:paths.path[userdel]": "/usr/sbin/userdel",
  "default:paths.path[usermod]": "/usr/sbin/usermod",
  "default:paths.path[virtualenv]": "/usr/bin/virtualenv",
  "default:paths.path[wc]": "/usr/bin/wc",
  "default:paths.path[wget]": "/usr/bin/wget",
  "default:paths.perl": "/usr/bin/perl",
  "default:paths.pgrep": "/usr/bin/pgrep",
  "default:paths.ping": "/bin/ping",
  "default:paths.pip": "/usr/bin/pip",
  "default:paths.printf": "/usr/bin/printf",
  "default:paths.realpath": "/usr/bin/realpath",
  "default:paths.sed": "/bin/sed",
  "default:paths.service": "/usr/sbin/service",
  "default:paths.shadow": "/etc/shadow",
  "default:paths.sort": "/usr/bin/sort",
  "default:paths.svc": "/usr/sbin/service",
  "default:paths.sysctl": "/sbin/sysctl",
  "default:paths.systemctl": "/bin/systemctl",
  "default:paths.tar": "/bin/tar",
  "default:paths.test": "/usr/bin/test",
  "default:paths.tr": "/usr/bin/tr",
  "default:paths.update_alternatives": "/usr/bin/update-alternatives",
  "default:paths.update_rc_d": "/usr/sbin/update-rc.d",
  "default:paths.useradd": "/usr/sbin/useradd",
  "default:paths.userdel": "/usr/sbin/userdel",
  "default:paths.usermod": "/usr/sbin/usermod",
  "default:paths.virtualenv": "/usr/bin/virtualenv",
  "default:paths.wc": "/usr/bin/wc",
  "default:paths.wget": "/usr/bin/wget"
}

Try finding and merging variables by tag, rendering datastate(), or bundlestate().