This article includes best practices, tips, and tricks for troubleshooting failed deployments and conveys an understanding of Pipelines deployment steps and structure.
This article is specific to deployments done in Pipelines for Applications. This does not cover Kubernetes deployments created with Pipelines for Containers.
This article assumes the reader has knowledge of navigating *nix operating systems.
Let's take a look at the best way to approach troubleshooting deployment issues.
Table of contents
- Log files
- Log debugging with Pipelines manifest
- Pipelines deployment directory structure
- Change to the distelli user
- Check environment variables
- Inspecting deployment steps
- Which step occurs in which directory
- Our library of example deployment tutorials is steadily growing. You can find them by visiting the Pipelines for Applications docs site and selecting Deployments from the menu on the left.
- Learn more about manifests from our library of example manifests.
- A great reference for bash scripting can be found here.
- If you have a Puppet Support account, you can open a ticket with us. Learn more about Puppet Products and Support.
When deploying fails you should first look at the log files. Logs are captured during each phase of deployment.
The logs are easily viewed in the Pipelines web UI. Execute the following steps to see the logs:
- Log into your Pipelines account.
- Navigate to the application where the deployment failed.
Click the Failed Deploy link in the app history timeline.
You will find yourself at the deployment details screen. For each server in the deployment there will be a Logs button. Click this to see the logs. These are the deployment manifest execution logs.
You can find more logging in the
stdout log. Click the log selector to select
stdout log the following phase is logged:
Log debugging with Pipelines manifest
It is helpful to include
echo statements in your
distelli-manifest.yml to quickly identify which step in which phase you are in while viewing the logs. Here is an example Pipelines manifest file with echo statements:
Note: Some sections do not log.
jdoe/SimpleNodeApp: PkgInclude: # This section does not log - server.js PkgExclude: # This section does not log - node_modules/ PreInstall: - echo "Distelli PreInstall Starting." # Check to see if node is already installed - if [ -a /usr/local/bin/node ]; then # node is already installed - echo "node exists" - else # Install node from nodejs.org - echo "node does not exist. Installing..." - wget http://nodejs.org/dist/v0.12.2/node-v0.12.2-linux-x64.tar.gz - sudo tar -C /usr/local --strip-components 1 -zxvf node-v0.12.2-linux-x64.tar.gz - fi - echo "Distelli PreInstall Complete." PostInstall: - echo "Distelli PostInstall Starting." - sudo npm config set loglevel verbose - echo "Distelli PostInstall Complete." Exec: # You should not use echo in the Exec: section - node server.js
Now when you view the logs you can easily identify which step of which phase an issue may have occurred in.
Look in the log for errors. There typically is some indication of what may have failed. Also look in the stdout to see why the application may have failed to execute.
Some other commands that may be helpful to include in a
distelli-manifest.yml for troubleshooting might be:
pwd- Show the present working directory the deployment phase is in.
env- Show the current environment variables during the deployment phase.
set- Show the current environment variables and aliases during the deployment phase.
df- Show the current free disk space.
netstat -plnt- Show the current TCP ports being listened to.
You can include these commands in the
distelli-manifest.yml phases. For example:
jdoe/SimpleJavaApp: Env: - CLASSPATH: 'target/*:lib/*' - JAVA_HOME: "/usr/local/jre" - JVM_ARGS: '"-Duser.timezone=UTC -Xmx128M -Xms128M"' - PORT: "8080" - ARGS: "$PORT" PreInstall: - echo "Starting PreInstall Section" - pwd - env - echo "End PreInstall Section" PostInstall: - echo "Starting PostInstall Section" - pwd - env - echo "------------------------------" - netstat -plnt - echo "------------------------------" - echo "End PostInstall Section" Exec: - exec $JAVA_HOME/bin/java -cp $CLASSPATH $JVM_ARGS com.example.app.SimpleJavaApp $ARGS PkgInclude: - target/*.jar - lib/*.jar
The above steps can assist with troubleshooting a deployment problem by adding debugging to the Pipelines manifest and looking at the logs.
Learn more about manifests from our library of example manifests.
Pipelines deployment directory structure
When you install the Pipelines agent a directory structure is built to accommodate deployments. It is helpful to understand the structure on the build server when troubleshooting. This structure sits at the root in a directory called
distelli owned by the user and group
With an application called
SimplePHPApp deployed to an environment called
TEST_PHP, the directory structure will look similar to this:
/distelli ├── _agent │ ├── agent_data.yml │ ├── agent-stdout.log │ ├── distelli-agent.log │ ├── dtk │ │ └── │ ├── dtk-stdout.log │ ├── logs │ │ └── │ ├── manifests │ │ └── │ ├── server-id.yml │ └── supervisor │ ├── supervisord.conf │ └── supervisord.log ├── apps │ └── ├── _apps │ └── SimplePHPApp.TEST_PHP.d-e6kksc38lmed3764772c │ ├── bin │ │ ├── distelli-env.sh │ │ ├── distelli-preinstall.sh │ │ └── distelli-run.sh │ └── simpleapp.php ├── .dtk │ ├── agent.sock │ ├── app_state.secret │ ├── app_state.sqlite │ ├── cli_state.sqlite │ └── supervisor.sock ├── envs │ └── TEST_PHP │ ├── bin -> /distelli/_apps/SimplePHPApp.TEST_PHP.d-e6kksc38lmed3764772c/bin │ ├── distelli-stdout.log │ └── simpleapp.php -> /distelli/_apps/SimplePHPApp.TEST_PHP.d-e6kksc38lmed3764772c/simpleapp.php ├── logs │ ├── agent.log │ ├── deployment │ │ └── d-e6kksc38lmed3764772c.log │ ├── install.log │ ├── supervise.log │ └── tasks │ └── ├── _packages │ └── └── tasks └──
On Windows the Pipelines root directory can be found at:
Let’s first look at the naming structure that is created when you deploy. You can see the below naming included in several places in the directory hierarchy above.
SimplePHPApp.TEST_PHP.d-e6kksc38lmed3764772c | | | | |application | env | deployment ID |
Remember, this example directory structure is based on deploying one application called
SimplePHPApp to environment
In this scenario, the Pipelines specific environment variable
DISTELLI_INSTALLHOME is the below directory. It is this directory where your application release is unbundled during the Install phase.
The Pipelines specific environment variable
DISTELLI_APPHOME is below. It is here that a symlink to the contents of the above directory is created after PostInstall.
For more information on deployment environment variables see the Environment variables reference guide.
When doing a deployment the
distelli-manifest.yml shell scripts will be here in the
Change to the
You can switch your user to the
distelli user with the following command:
sudo su - distelli
This will cause you to become user
distelli so that when you execute any deployment scripts, you will be executing them with the same user and permissions that are used when Pipelines does the deploy; and you can see the exact behavior.
It is important to have the same PATH as when the
distelli user deploys, do the following:
export PATH=$(getconf PATH) echo $PATH
For more information on the Agent PATH, see Agent deploy path.
Check environment variables
The file below holds the script that initializes the deployment environment variables. For more information on using environment variables see the Environment variables reference guide.
cat command you can look at the contents of this file and the environment variables used for this applications deployment.
$ cat distelli-env.sh
export DISTELLI_APPNAME=SimplePHPApp export DISTELLI_INSTALLHOME=/distelli/_apps/SimplePHPApp.TEST_PHP.d-e6kksc38lmed3764772c export DISTELLI_APPHOME=/distelli/envs/TEST_PHP # Variable Injected from Manifest export PORT=8006
When troubleshooting it is helpful to
distelli-env.sh file to your environment so that the environment variables are populated in your shell:
$ source bin/distelli-env.sh
Note: When Pipelines for Applications runs the various scripts
getconf PATHis called. This is important to note as the PATH may not be what you expect. For more info, see Agent deploy path.
distelli-manifest.yml phase files
Each phase of the deployment, for example PreInstall, PostInstall, Run (Exec), Restart, Terminate, and others will have an individual bash shell script (or batch on Windows) file in:
The naming format for these files is:
Note: This naming format is true except for the Exec section which is instead run.
distelli-preinstall.sh distelli-install.sh distelli-postinstall.sh distelli-run.sh <Exec phase distelli-preterminate.sh distelli-start.sh …and more
Inspecting deployment steps
You can view and execute the various deployment phase scripts manually. It is suggested that if you do run a script, ensure you have switched to user distelli. For example:
$ cat bin/distelli-preinstall.sh export DISTELLI_APPNAME=SimplePHPApp export DISTELLI_INSTALLHOME=/distelli/_apps/SimplePHPApp.TEST_PHP.d-e6kksc38lmed3764772c export DISTELLI_APPHOME=/distelli/envs/TEST_PHP set -e # Variable Injected from Manifest export PORT=8006 sudo apt-get -y install php5-cli
Note that Pipelines for Applications inserts the
set -e option. This causes the entire script to exit on any single failure. It may be helpful in troubleshooting to add an option:
set +e- Revert the
set -ebehavior so that a single failure does not cause the entire script to end
set -e- Change the behavior back to Distelli default
set -x- Bash will provide debugging information on commands being executed
set +x- Turn off bash debugging
An alternative may be to use the
|| true at the end of a command that may fail. Here is an example:
eval "$(~/.rbenv/bin/rbenv init -) || true"
Some notes regarding the
Exec: section in the
distelli-manifest.yml file. Here is an example:
Exec: - echo "Distelli Exec Starting." - node server.js
The last line of this section will be the process that is monitored by Pipelines for Applications. In the above example, the
node server.js process will be monitored. That is why I did not include a final echo statement in this section.
On Windows, the
Exec: section should only have 1 line (entry).
Which step occurs in which directory