How do I capture and decrypt SSL traffic between primary server (master) and agent nodes?

Troubleshoot network problems and verify the integrity of traffic between the primary server (called the master in older versions of Puppet Enterprise) and agent nodes by using Wireshark, an open-source network protocol analyzer.

Note: We cannot troubleshoot third-party software. If you have issues with Wireshark, request help from their mailing lists. If you have issues with WinSCP, request help on their forums.

Version and installation information

PE version: All
Wireshark tested version: 2.0.2
OS: All
Installation type: Any

Solution

Complete the following steps:

1. Download and install Wireshark on your local machine.

2. Capture traffic between the primary server and an agent from the primary server node. As root on the primary server, run the following, replacing network interface and IP with values from your infrastructure:

   tcpdump -i <network interface> -c 10000 -w /var/tmp/puppet-run-capture.pcap host <IP of the agent node>

   Note: Including "-c 10000" stops the capture at 10,000 packets and prevents the capture file from filling your hard drive. If required, increase the size of the capture by changing 10000 to a larger value.

3. Transfer the network capture file and SSL private keys from your primary server and agent nodes to the machine where you installed Wireshark. For example, for a primary server and agent running PE 2016.1.1, transfer the files via scp by running the following on the computer with Wireshark installed:

   scp user@server:/var/tmp/puppet-run-capture.pcap
scp user@server:/etc/puppetlabs/puppet/ssl/private_keys/<<SERVER HOSTNAME>>.pem
scp user@agent:/etc/puppetlabs/puppet/ssl/private_keys/<<AGENT HOSTNAME>>.pem

   Note: If you are using Windows, you can transfer the files using WinScp.

   Complete the following steps in Wireshark.

4. Decrypt data by loading the SSL certs into the Wireshark SSL dissector. Go to Preferences > Protocols > SSL. Click the Edit button for the RSA keys list. Add the primary server and agent keys.

   
   

5. Enter:

   A. 0.0.0.0 for the IP address.

   B. 0 for the Port.

   C. Data for the Protocol.

   D. In the Key Field value, select the primary server and agent keys.

   

    

6. Open the binary capture file in Wireshark to view the traffic between the agent and primary server. Right clicking on a packet and select "Decode As..." Add the primary server's TCP port stream by clicking plus in the lower left corner. In the Current field, select SSL, in the Value field, select port 8140, in Field select TCP Port. Repeat the same steps to add the TCP port stream for the agent, selecting the only other available high numbered port.

   
   

   When the file is successfully decoded, the capture shows a list of:

   • An SSL handshake when the TCP connection between the primary server and agent is established.

   • Data packets in cleartext.

7. Check the capture to verify traffic and data are as expected. View the cleartext data packets using the following steps.

   A. In the Info field, find any packet with a value of Application Data.

   B. Select and right clicking the packet.

   C. Select Follow SSL Stream.

8. If you captured multiple TCP streams, repeat steps 6 and 7.

Part of a sample capture follows, showing the facts sent from the agent to the primary server:

POST /puppet/v3/catalog/burnside-agent1?environment=production HTTP/1.1 Accept: pson, yaml, dot, binary X-Puppet-Version: 4.4.1 Accept-Encoding: gzip;q=1.0,deflate;q=0.6,identity;q=0.3 User-Agent: Ruby Host: burnside-server:8140 Content-Length: 17886 Content-Type: application/x-www-form-urlencoded

environment=production&facts_format=pson&facts=%257B%2522name%2522%253A%2522burnside-agent1%2522%252C%2522values%2522%253A%257B%2522aio_agent_build%2522%253A%25221.4.1%2522%252C%2522aio_agent_version%2522%253A%25221.4.1%2522%252C%2522architecture%2522%253A%2522amd64%2522%252C%2522augeas%2522%253A%257B%2522version%2522%253A%25221.4.0%2522%257D%252C%2522augeasversion%2522%253A%25221.4.0%2522%252C%2522bios_release_date%2522%253A%252201%252F01%252F2011%2522%252C%2522bios_vendor%2522%253A%2522Seabios%2522%252C%2522bios_version%2522%253A%25220.5.1%2522%252C%2522blockdevice_vda_size%2522%253A8589934592%252C%2522blockdevice_vda_vendor%2522%253A%25220x1af4%2522%252C%2522blockdevices%2522%253A%2522vda%2522%252C%2522chassistype%2522%253A%2522Other%2522%252C%2522custom_auth_conf%2522%253Afalse%252C%2522dhcp_servers%2522%253A%257B%2522eth0%2522%253A%2522192.168.0.1%2522%252C%2522system%2522%253A%2522192.168.0.1%2522%257D%252C%2522disks%2522%253A%257B%2522vda%2522%253A%257B%2522size%2522%253A%25228.00%2BGiB%2522%252C%2522size_bytes%2522%253A8589934592%252C%2522vendor%2522%253A%25220x1af4%2522%257D%257D%252C%2522dmi%2522%253A%257B%2522bios%2522%253A%257B%2522release_date%2522%253A%252201%252F01%252F2011%2522%252C%2522vendor%2522%253A%2522Seabios%2522%252C%2522version%2522%253A%25220.5.1%2522%257D%252C%2522chassis%2522%253A%257B%2522type%2522%253A%2522Other%2522%257D%252C%2522manufacturer%2522%253A%2522Fedora%2BProject%2522%252C%2522product%2522%253A%257B%2522name%2522%253A%2522OpenStack%2BNova%2522%252C%2522serial_number%2522%253A%2522eeda805d-c7e4-4eea-8f2e-da428547637a%2522%252C%2522uuid%2522%253A%252253864FD8-9ECB-4AAD-A78A-C91A691D76C5%2522%257D%257D%252C%2522ec2_metadata........................

HTTP/1.1 200 OK
Date: Fri, 08 Apr 2016 15:38:03 GMT
Content-Type: text/pson; charset=ISO-8859-1
X-Puppet-Version: 4.4.1
Transfer-Encoding: chunked
Server: Jetty(9.2.z-SNAPSHOT)

8000
{"tags":["puppet_enterprise","puppet_enterprise::profile::agent","profile","agent","puppet_enterprise::profile::mcollective::agent","mcollective","settings","default","puppet_enterprise::params","params","puppet_enterprise::symlinks","symlinks","puppet_enterprise::pxp_agent","pxp_agent","puppet_enterprise::pxp_agent::service","service","puppet_enterprise::mcollective::server","server","puppet_enterprise::mcollective::server::plugins","plugins","puppet_enterprise::mcollective::service","puppet_enterprise::mcollective::server::logs","logs","puppet_enterprise::mcollective::server::certs","certs","puppet_enterprise::mcollective::server::facter","facter","puppet_enterprise::mcollective::cleanup","cleanup","node","class"],"name":"burnside-agent1","version":1460129883,"code_id":null,"catalog_uuid":"7160f650-59aa-4a5c-bfe2-21c47bc1d408","catalog_format":1,"environment":"production","resources":[{"type":"Stage","title":"main","tags":["stage"],"exported":false,"parameters":{"name":"main"}},{"type":"Class","title":"Settings","tags":["class","settings"],"exported":false}............