Removing nodes: Understanding "puppet node purge", "node-ttl", and "node-purge-ttl" in Puppet Enterprise

You can remove nodes from different components in your deployment using puppet node purge <NODE NAME>, node-ttl, and node-purge-ttl.  If you need to remove agentless nodes, those steps are in our documentation. You can also learn how to release licenses from inactive nodes.

Version and installation information

PE version: All supported versions

Solution

Remove nodes from your deployment with puppet node purge

Use puppet node purge to completely remove nodes from your deployment. The puppet node purge command is a wrapper around two separate commands:

puppet node deactivate marks the node as deactivated in PuppetDB. Although the node remains present in PuppetDB, it no longer shows up in the console under Nodes > Inventory. The node's report data is still available in the console.

puppet node clean deletes the information cache, including certs and cached catalogs, for the deactivated node on the primary server (called the master in older versions of PE). This command creates a new CRL (certificate revocation list) and applies it to the primary server.

Besides calling puppet node deactivate and puppet node clean, the puppet node purge command also removes any node information from the database.

View nodes deactivated by puppet node deactivate by querying the deactivated field:

curl -X GET http://localhost:8080/pdb/query/v4 --data-urlencode 'query=nodes { deactivated is not null }' | python -m json.tool

Maintain your deployment with node-ttl and node-purge-ttl

Use node-ttl and node-purge-ttl to maintain your deployment. Both settings remove outdated node information from PuppetDB and free disk space. However, node removal is not permanent and certificates are not revoked. When a node is expired and purged with node-ttl and node-purge-ttl, the next Puppet run on that node adds it back to PuppetDB.

node-ttl marks the node as expired in PuppetDB after a specified amount of time. Although the node remains present in PuppetDB, it no longer shows up in the console under Nodes > Inventory. The node's report data is still available in the console.

node-purge-ttl affects only deactivated or expired nodes in PuppetDB. It automatically purges nodes that have been deactivated or expired for a specified amount of time from PuppetDB. All facts, catalogs, and reports for the relevant nodes are deleted.

Understand when nodes are expired and deleted from PuppetDB with node-ttl and node-purge-ttl

Neither node-ttl nor node-purge-ttl affects nodes immediately.

node-ttl is triggered when nodes have had no activity for a specified amount of time (no new catalogs, facts, or reports). If you set node-ttl = 7d, you might expect your nodes to expire 7 days after the last new catalog, fact or report. However, nodes won't be evaluated and expired until the first garbage collection (GC) 7 days after the last new catalog, fact or report.

node-purge-ttl is triggered a specified amount of time after the node is deactivated or expired. If you set node-purge-ttl=1h, you might expect your node to be purged from PuppetDB one hour after your node expires or deactivates. However, it won't be evaluated and purged until the first GC after the hour has elapsed.

How often GC happens and when node-ttl and node-purge-ttl are effective is governed by the gc-interval setting. For example, if gc-interval is set to 60 minutes (the default) and node-ttl is set to 5 minutes, the node exists in PuppetDB as an active node and is visible in the console for up to 60 minutes. During a GC run, node-ttl is evaluated. The node auto-expires if the node-ttl time interval (in this case 5 minutes) has passed since the last new activity on the node.

When an interval is specified for node-purge-ttl, it is applied during the GC run. For example:

gc-interval = 5m
node-ttl = 4m
node-purge-ttl = 1h

Note: d stands for days, m stands for minutes, s stands for seconds. All example PuppetDB logs are located on the PuppetDB node (the primary server in a standard deployment) at /var/log/puppetlabs/puppetdb/puppetdb.log.

2016-09-30 14:17:13,299 INFO  [p.p.c.services] Starting sweep of stale nodes (threshold: 4 minutes)
2016-09-30 14:17:13,306 INFO  [p.p.c.services] Auto-expired node nagios-client
2016-09-30 14:17:13,306 INFO  [p.p.c.services] Auto-expired node nagios-server
2016-09-30 14:17:13,307 INFO  [p.p.c.services] Finished sweep of stale nodes (threshold: 4 minutes)
2016-09-30 14:17:13,308 INFO  [p.p.c.services] Starting purge deactivated and expired nodes (threshold: 1 hour)
2016-09-30 14:17:13,309 INFO  [p.p.c.services] Finished purge deactivated and expired nodes (threshold: 1 hour)
2016-09-30 14:17:13,309 INFO  [p.p.c.services] Starting sweep of stale reports (threshold: 14 days)
2016-09-30 14:17:13,312 INFO  [p.p.c.services] Finished sweep of stale reports (threshold: 14 days)
2016-09-30 14:17:13,312 INFO  [p.p.c.services] Starting database garbage collection
2016-09-30 14:17:13,325 INFO  [p.p.c.services] Finished database garbage collection

Using the default values:

node-ttl = 7d
node-purge-ttl = 0s
report-ttl = 14d

Note: report-ttl automatically deletes reports that are older than the specified amount of time.

Setting node-purge-ttl with a value of 0s is equivalent to leaving the value unset. The GC doesn't purge nodes.

gc-interval = 5m
node-ttl = 4m
node-purge-ttl = 0s

2016-09-30 14:27:06,272 INFO  [p.p.c.services] Starting sweep of stale nodes (threshold: 4 minutes)
2016-09-30 14:27:06,291 INFO  [p.p.c.services] Auto-expired node nagios
2016-09-30 14:27:06,294 INFO  [p.p.c.services] Finished sweep of stale nodes (threshold: 4 minutes)
2016-09-30 14:27:06,294 INFO  [p.p.c.services] Starting sweep of stale reports (threshold: 14 days)
2016-09-30 14:27:06,299 INFO  [p.p.c.services] Finished sweep of stale reports (threshold: 14 days)
2016-09-30 14:27:06,300 INFO  [p.p.c.services] Starting database garbage collection
2016-09-30 14:27:06,314 INFO  [p.p.c.services] Finished database garbage collection

Leaving node-ttl and node-purge-ttl unset is equivalent to setting the value to 0 seconds. The GC doesn't expire or purge nodes.

gc-interval = 5m

node-ttl - unset
node-purge-ttl - unset

2016-09-30 15:13:01,324 INFO  [p.p.c.services] Starting sweep of stale nodes (threshold: 7 days)
2016-09-30 15:13:01,329 INFO  [p.p.c.services] Finished sweep of stale nodes (threshold: 7 days)
2016-09-30 15:13:01,329 INFO  [p.p.c.services] Starting sweep of stale reports (threshold: 14 days)
2016-09-30 15:13:01,339 INFO  [p.p.c.services] Finished sweep of stale reports (threshold: 14 days)
2016-09-30 15:13:01,340 INFO  [p.p.c.services] Starting database garbage collection
2016-09-30 15:13:01,356 INFO  [p.p.c.services] Finished database garbage collection

In the case where both values are set to 0s, exactly the same thing happens; neither interval is applied, and nodes remain in PuppetDB and the console. For example:

gc-interval = 5m
node-ttl = 0s
node-purge-ttl = 0s

2016-09-30 14:34:44,294 INFO  [p.p.c.services] Starting sweep of stale reports (threshold: 14 days)
2016-09-30 14:34:44,308 INFO  [p.p.c.services] Finished sweep of stale reports (threshold: 14 days)
2016-09-30 14:34:44,308 INFO  [p.p.c.services] Starting database garbage collection
2016-09-30 14:34:44,323 INFO  [p.p.c.services] Finished database garbage collection

Configure node-ttl and node-purge-ttl

View configuration settings for node-ttl and node-purge-ttl in the PuppetDB configuration file. By default, this config file is located at /etc/puppetlabs/puppetdb/conf.d on your PuppetDB node, under the [database] section. Both settings must be edited in the console in the PE PuppetDB group in the puppet_enterprise::puppetdb class.

By default, the settings maintaining your node information in PuppetDB are:

node-ttl = 7d
node-purge-ttl = 0s
report-ttl = 14d

Giving any of these three settings a value of 0s is equivalent to leaving the value unset and your nodes won't be expired or purged.

View nodes impacted by node-ttl and node-purge-ttl

View expired nodes in PuppetDB by using PQL to query the PuppetDB API root endpoint using the expired field:

curl -X GET http://localhost:8080/pdb/query/v4 --data-urlencode 'query=nodes { node_state = "inactive" }' | python -m json.tool

View expired and deactivated nodes by combining both queries using or.

curl -X GET http://localhost:8080/pdb/query/v4 --data-urlencode 'query=nodes { expired is not null or deactivated is not null }' | python -m json.tool