Skip to content

2020-06-14 Stackstorm

Stackstorm

Stackstorm#

  • Platform for integration and automation across services and tools
  • Ties together existing infrastructure and application environment

  • Focuses on taking actions in response to events

  • Facilitated troubleshooting

  • Automated remediation
  • Continuous deployment

Compose these rules and workflows as code. StackStorm integrates with other systems through something called packs

How it Works#

  1. Events are aggregated (Push/Pull) from various services via sensors
  2. Events are compared against triggers and generate actions
  3. Processed actions areplaced on a message queue (rabbit mq)
  4. Actions reach out to services to perform the actions
  5. Log/audit history is pushed to the db (mongo)
  6. Processed results are sent back to the rules engine for further processing

Terms:

  • Sensors - python plugins. Inbound or outbound, receive or watch for events. A trigger is generated by sdtackstorm.
  • Triggers - stackstorm representations of external events
  • Actions - Outbound integrations. Eg. SSH, rest call, openstack, docker, ansible,
  • Rules - Map triggers to actions or workflows.
  • Workflows - Stitch actions together, define order, transitions and passing data.
  • Packs - units of content deployment (plugins) found on stackstorm exchange
  • Audit trail - actions are recorded and stored.

ST2 Services#

  • Provides the main stackstorm functionality
  • Located at /opt/stackstorm/st2
  • Share a virtualenv
  • Configured via /etc/st2/st2.conf

Read more about the reference deployment on stackstorm

Getting Started#

Check the st2 version:

st2 --version
st2 3.0.0, on Python 2.7.12

Login:

st2 login st2admin -p Ch@ngeMe

List actions from the core pack:

st2 action list --pack=core

List all the triggers:

st2 trigger list

List rules:

st2 rule list

Actions#

Run a local shell command:

st2 run core.local -- date -R

See the execution results:

st2 execution list

Run a command on a remote host:

st2 run core.remote hosts='localhost' -- uname -a

All stackstorm operations are also available via the REST Api

List all actions:

st2 action list

Get action metadata:

st2 action get core.http

Get details and parameters for an action:

st2 run core.http --help

To run an action from the cli use:

st2 run <action> key=value positional arguments

Eg. st2 run -j core.http url="https://docs.stackstorm.com" method="GET"

Use core.remote to run linux commands on multiple hosts via ssh. It assumes passwordless ssh is configured for the host.

For core.local and core.remote use -- to seperate the command run on the host or use the cmd parameter.

st2 run core.local -- ls -al
st2 run core.local cmd="ls -al"

Executions#

Get a list of executions:

st2 execution list

Get a single execution:

st2 execution get 5cc6f3766cb8de088e9dab3c

Get the last 5 executions:

st2 execution list -n 5

Creating a Rule#

  • StackStorm uses rules to run actions or workflows when events happen.
  • Events are typically monitored by sensors.
  • When a sensor catches an event, it fires a trigger.
  • Trigger trips the rule, the rule checks the criteria and if it matches, it runs an action.

A rule definition is a yaml file with 3 sections: trigger, criteria and action.

Define a rule, based on the rule structure:

---
    name: "sample_rule_with_webhook"
    pack: "examples"
    description: "Sample rule dumping webhook payload to a file."
    enabled: true

    trigger:
        type: "core.st2.webhook"
        parameters:
            url: "sample"

    criteria:
        trigger.body.name:
            pattern: "st2"
            type: "equals"

    action:
        ref: "core.local"
        parameters:
            cmd: "echo \"{{trigger.body}}\" >> ~/st2.webhook_sample.out ; sync"
  • The trigger is a webhook…so going to https://your-server/api/v1/webhooks/sample trips the rule.
  • The criteria will check that the body name contains the pattern st2
  • The action will append the payload to that file on the stackstorm local host node

The payload of a trigger is reffered to with {{ trigger }}, if it is a valid json object it can be accessed with {{trigger.path.to.parameter}}

Remember to check available triggers:

# View a list of triggers
st2 trigger list

# Check details of a trigger
st2 trigger get core.st2.webhook

You can do alot more with criteria comparisons

Deploy a Rule#

Stackstorm can autoload rules, or can be deployed with API or CLI

Create a rule from CLI#

st2 rule create /usr/share/doc/st2/examples/rules/sample_rule_with_webhook.yaml

# List the rules
st2 rule list --pack=examples

# Get the rule you just created
st2 rule get examples.sample_rule_with_webhook

Executing the action that triggers that rule#

Check your apikey:

st2 apikey list

Make the call to the webhook endpoint:

Install httpie:

sudo apt update
sudo apt install httpie

Make the request to the /sample endpoint:

I got the auth token using the --debug command

http --verify=no https://10.10.10.10/api/v1/webhooks/sample foo=bar bill=baz name=st2 X-Auth-Token:cc29ec9adca840eeb44734cdbeaa8a4f

Check if the action was executed:

st2 execution list -n 1

Verify it actually worked:

sudo tail /home/stanley/st2.webhook_sample.out 
{u'foo': u'bar', u'bill': u'baz', u'name': u'st2'}

By default st2 runs as the stanley user

You can even run an action from stackstorm:

st2 run core.http method=POST body='{"you": "too", "name": "st2"}' url=https://10.10.10.10/api/v1/webhooks/sample headers='x-auth-token=cc29ec9adca840eeb44734cdbeaa8a4f,content-type=application/json' verify_ssl_cert=False

You will get a 202 accepted response…it did not give me the execution id

It is best to store your custom packs (of custom actions, rules, sensors and triggers) as code in a repo as .yml files

To deploy them, copy them to: /opt/stackstorm/packs/

# Copy examples to st2 content directory and set permissions
sudo cp -r /usr/share/doc/st2/examples/ /opt/stackstorm/packs/
sudo chown -R root:st2packs /opt/stackstorm/packs/examples
sudo chmod -R g+w /opt/stackstorm/packs/examples

# Run setup
st2 run packs.setup_virtualenv packs=examples

# Reload stackstorm context
st2ctl reload --register-all

Datastore#

To store and share common variables you can use stackstorm’s datastore, which can be referenced in rules and workflows with: {{st2kv.system.my_parameter}}

Create a key value:

st2 key set user stanley

View a list of key value stores:

st2 key list

StackStorm API Docs#

View the Stackstorm API docs

Actions#

In StackStorm, we can accomplish such things through actions

Eg.

  • Push a config change to a network device
  • Restart a service on a server
  • Retrieve information about a virtual machine
  • Bounce a switchport
  • Send a message to slack

They accept input, do work, and usually provide some output

Run a hello world on the local machine

st2 run core.local "echo Hello World!"

Get help about an action

st2 run core.local -h

Run an action in the background (sometimes you don’t want to have to wait for an action to complete)

st2 run -a core.local "sleep 300"

Get the result of an action

st2 execution get 5cc81e6e6cb8de0867e220ca

id: 5cc81e6e6cb8de0867e220ca
status: running (27s elapsed)
parameters: 
cmd: sleep 300
result: None

The execution describes an instance of a running action

See a list of recent executions

st2 execution list

Using a pack#

View the actions in a pack

st2 action list --pack=napalm

Credentials are stored in pack configurations

cat /opt/stackstorm/configs/napalm.yaml

Stackstorm Help#

CLI for StackStorm event-driven automation platform. https://stackstorm.com

positional arguments:
{run,action,action-alias,auth,login,whoami,apikey,execution,inquiry,key,pack,policy,policy-type,rule,webhook,timer,runner,sensor,trace,trigger,trigger-instance,rule-enforcement,workflow,service-registry,role,role-assignment}
    run                 Invoke an action manually.
    action              An activity that happens as a response to the external
                        event.
    action-alias        Action aliases.
    auth                Authenticate user and acquire access token.
    login               Authenticate user, acquire access token, and update
                        CLI config directory
    whoami              Display the currently authenticated user
    apikey              API Keys.
    execution           An invocation of an action.
    inquiry             Inquiries provide an opportunity to ask a question and
                        wait for a response in a workflow.
    key                 Key value pair is used to store commonly used
                        configuration for reuse in sensors, actions, and
                        rules.
    pack                A group of related integration resources: actions,
                        rules, and sensors.
    policy              Policy that is enforced on a resource.
    policy-type         Type of policy that can be applied to resources.
    rule                A specification to invoke an "action" on a "trigger"
                        selectively based on some criteria.
    webhook             Webhooks.
    timer               Timers.
    runner              Runner is a type of handler for a specific class of
                        actions.
    sensor              An adapter which allows you to integrate StackStorm
                        with external system.
    trace               A group of executions, rules and triggerinstances that
                        are related.
    trigger             An external event that is mapped to a st2 input. It is
                        the st2 invocation point.
    trigger-instance    Actual instances of triggers received by st2.
    rule-enforcement    Models that represent enforcement of rules.
    workflow            Commands for workflow authoring related operations.
                        Only orquesta workflows are supported.
    service-registry    Service registry group and membership related
                        commands.
    role                RBAC roles.
    role-assignment     RBAC role assignments.

Debugging commands#

Using the --debug flag you can see all the requests being made to the stackstorm api.

**The --debug flag must be just after st2

Eg.

st2 --debug run core.local date

Stackstorm packs#

Stackstorm is highly extensible It focuses on the primitives necessary to enable event-driven automation

Anytime you want to integrate with a third party system or upload some custom rules or workflows, you do this through a pack.

Install a pack

st2 pack install napalm

Packs can also be installed directly from a git repo:

st2 pack install https://github.com/emedvedev/chatops_tutorial

Actions#

pieces of code that can perform arbitrary automation or remediation tasks

Examples:

  • Restart a service on a server
  • Create a cloud server
  • Acknowledge a nagios, or pagerduty alert
  • Send a notification via email or sms
  • Send a message to slack
  • Snapshot a vm
  • Run a nagios check

Executed when a rule is triggered. Multiple actions can be strung together in a workflow.

CLI#

View all actions

st2 action list

List all actions in linux packs

st2 action list -p linux

Get info of a particular action

st2 action get linux.check_loadavg

To execute an action manually

st2 action execute <action with parameters>

Eg.

st2 action execute core.http url="http://httpbin.org/get"

Action Runners#

An execution environment for user-implemented actions to be run remotely via ssh (or locally).

Available runners:

  • local-shell-cmd - Executes linux commands on the host of Stackstorm
  • local-shell-script - Executes scipts on the host of Stackstorm
  • remote-shell-cmd - Executes linux command on one or more remote hosts provided by the user
  • remote-shell-script - Remote runner for scripts
  • python-script - Python runner, implemented as a run() method on python classes.
  • http-request - HTTP client
  • action-chain - Executes simple linear workflows
  • mistral-v2 - Executes complex Mistral Openstack workflows
  • cloudslang - Complex workflows, built on top of CloudSlang
  • inquirer - Implements core logic of inquiries
  • winrm-cmd - Run CLI on windows hosts
  • winrm-ps-cmd - Run powershell on windows hosts
  • winrm-ps-script - Run powershell scripts on hosts

Actions inherit the runner’s parameters.

Writing Custom Actions#

An action is composed of: 1. A Yaml metadata file describing the aciton and inputs 2. A script that implements the logic

Can use any programming language as long as it exits with 0 on success and 1 on error. All log messages should be printed to stderr

Action Metadata#

Describes the action.

  • name - Name of the action
  • runner_type - Type of runner
  • enabled
  • entry_point - Location of the action launch script relative to: /opt/stackstorm/packs/${pack_name}/actions/
  • parameters - dictionary of parameters for the action
  • tags - Supplemental information

Eg.

---
name: "send_sms"
runner_type: "python-script"
description: "This sends an SMS using twilio."
enabled: true
entry_point: "send_sms.py"
parameters:
    from_number:
        type: "string"
        description: "Your twilio 'from' number in E.164 format. Example +14151234567."
        required: true
        position: 0
    to_number:
        type: "string"
        description: "Recipient number in E.164 format. Example +14151234567."
        required: true
        position: 1
        secret: true
    body:
        type: "string"
        description: "Body of the message."
        required: true
        position: 2
        default: "Hello {% if system.user %} {{ st2kv.system.user }} {% else %} dude {% endif %}!"
Output Schema#

Disabled by default. To enable it set: validate_output_schema = True in /etc/st2/st2.conf [system]

Modelling the output of an action. This aids in workflow development and error handling.

For example a python action outputs a tuple:

---
...
output_schema:
    errors:
       type: array
       items:
           type: string
   output:
       required: true
       type: array
       items:
           type: number
   status_code:
       required: true
       type: integer

If the action does not return the correct fields it will fail validation and the action itself will fail. This prevents propagating corrupt data to other actions in a workflow that could lead to unpredictable results.

Paramters in Action#

You can access variable from the key-value store at st2kv.system. YOu can get access to action context variable with:

parameters:
  user:
    type: "string"
    description: "User of this action."
    required: true
    default: "{{action_context.api_user}}"

The prefix action_context is used.

You can also access config_context which contains key-value contents of your pack configuration

In actionchains and workflows, every task can access the parent’s execution_id

Action Registration#

To register a new action:

  1. Place it into the content location
  2. Tell the system the action is available

Actions are grouped in packs and located at: /opt/stackstorm/packs

For hacking one off actions create you action in /opt/stackstorm/packs/default/actions, once tested add it to a dedicated pack.

Register an action with:

st2 action create my_action_metadata.yaml

Reload all actions:

st2ctl reload --register-actions
Built-in Parameters#
  • args
  • cmd
  • cwd
  • env
  • dir
Overriding Runner Parameters#

The linux.rsync action overrides the cmd in the remote-shell-cmd runner, with rsync:

---
    name: 'rsync'
    runner_type: 'remote-shell-cmd'
    description: 'Copy file(s) from one place to another w/ rsync'
    enabled: true
    entry_point: ''
    parameters:
        source:
            type: 'string'
            description: 'List of files/directories to to be copied'
            required: true
        dest_server:
            type: 'string'
            description: "Destination server for rsync'd files"
            required: true
        destination:
            type: 'string'
            description: 'Destination of files/directories on target server'
            required: true
        cmd:
            immutable: true
            default: 'rsync {{args}} {{source}} {{dest_server}}:{{destination}}'
        connect_timeout:
            type: 'integer'
            description: 'SSH connect timeout in seconds'
            default: 30
        args:
            description: 'Command line arguments passed to rysnc'
            default: '-avz -e "ssh -o ConnectTimeout={{connect_timeout}}"'
Environment variables for actions#

Runners make the following environment variables available:

  • ST2_ACTION_PACK_NAME - name of pack the current action belongs to
  • ST2_ACTION_EXECUTION_ID - Execution id of the action being executed
  • ST2_ACTION_API_URL - Full URL to the public API endpoint
  • ST2_ACTION_AUTH_TOKEN - Auth token which is available to the action until it completes

You can use these in a shell script:

#!/usr/bin/env bash

# Retrieve a list of actions by hitting the API using cURL and the information provided
# via environment variables

RESULT=$(curl -H "X-Auth-Token: ${ST2_ACTION_AUTH_TOKEN}" ${ST2_ACTION_API_URL}/actions)
echo ${RESULT}
Converting Existing Scripts into Actions#

Converting an existing script

  1. Make sure script conforms to conventions

Make sure it exits on 0 for success, non-zero for error

  1. Create a metadata file

You will probably use: remote-shell-script

  1. Update argument parsing in the script
  • named - parameters that do not include the position attribute

  • positional - include a position attribute

    script.sh –param1=value –param2=value –param3=value

types of arguments:

  • string, integer and float - serialized as a string
  • boolean - serialized as a string 1 (true) or 0 (false)
  • array - serialized as a comma delimited string
  • object - serialized as json

Alot more info in the docs…

Sensors and Triggers#

Sensors#

Python that periodically poll external events or passively wait for inbound events. They inject triggers, which can be matched by rules for potencial action execution.

Triggers#

Identify the incoming events to StackStorm A trigger is a tuple of type (string) and optional parameters (object).

Info on creating sensors and triggers in the docs

Rules#

Capture operational patterns as automations

Rules map triggers to actions (or workflows), apply matching criteria and map trigger payloads to action inputs

Example:

---
    name: "rule_name"                      # required
    pack: "examples"                       # optional
    description: "Rule description."       # optional
    enabled: true                          # required

    trigger:                               # required
        type: "trigger_type_ref"

    criteria:                              # optional
        trigger.payload_parameter_name1:
            type: "regex"
            pattern : "^value$"
        trigger.payload_parameter_name2:
            type: "iequals"
            pattern : "watchevent"

    action:                                # required
        ref: "action_ref"
        parameters:                        # optional
            foo: "bar"
            baz: "{{ trigger.payload_parameter_1 }}"
  • name - name of the rule
  • pack - the pack a rule belongs to, default is assumed
  • description
  • enabled
  • trigger - type of trigger emitted (get with st2 trigger list)
  • criteria - optional, type of comparison and a pattern to match against.
  • action - the action to execute

Each rule can only have a single action, for multiple actions you need to create a workflow

Criteria#

Rules needed to be matched against

Eg.

criteria:
    trigger.payload_parameter_name1:
        type: "regex"
        pattern : "^value$"
    trigger.payload_parameter_name2:
        type: "iequals"
        pattern : "watchevent"

The above criteria are AND

If you want OR criteria, have multiple criteria sections

Criteria Types#
  • equals - Values are equal
  • nequals - Values not equal
  • lessthan - trigger value is less than provided value
  • greaterthan
  • matchwildcard - simple grok style linux matching
  • regex - regular expression - re.search('pattern', trigger_value)
  • iregex - case insensitive regex
  • iequals - case insensitve string match
  • contains - contains string case sensitive
  • ncontains - does not contain the provided value
  • icontains - contains string case insensitive
  • startswith - beginning matches a string
  • endswith - ends with a given string
  • iendswith - ends with, case insensitivity
  • timediff_lt - time difference between trigger time and current time less than
  • timediff_gt - time difference between trigger time and current time greater than
  • inside - checking if trigger.payload is within provided value (opposite of contains)
  • ninside - not inside
  • search - way to match json with any or all

Action#

Sets the action or workflow to be executed on successful match of a trigger and an optional set of criteria. A rule can also specify parameters.

Example

action:                                # required
    ref: "action_ref"
    parameters:                        # optional
        foo: "bar"
        baz: 1

To pass along the context of a trigger to an action, you can interpolate variable with Jinja templating syntax.

action:
    ref: "action_ref"
    parameters:
        foo: "bar"
        baz: "{{ trigger.payload_parameter_1 }}"

Both null and None can be a trigger variable so use the following filter: {{ trigger.payload_parameter_1 | use_none }}

Managing Rules#

To deploy a rule:

st2 rule create ${PATH_TO_RULE}

Eg.

st2 rule create /usr/share/doc/st2/examples/rules/sample_rule_with_webhook.yaml

You can update a rule with:

st2 rule update /usr/share/doc/st2/examples/rules/sample_rule_with_webhook.yaml

Reload all rules with

st2ctl reload --register-rules

List all rules

st2 rule list

View a single rule

st2 rule get examples.sample_rule_with_webhook

Delete a rule

st2 rule delete ${RULE_NAME_OR_ID}

Rule Location#

Rules are located at: /opt/stackstorm/packs/<pack_name>/rules

Testing Rules#

A st2-rule-tester is provided to evaluate rules against trigger instances without running any of the stackstorm components.

st2-rule-tester --rule=${RULE_FILE} --trigger-instance=${TRIGGER_INSTANCE_DEFINITION} --config-file=/etc/st2/st2.conf
echo $?
  • RULE_FILE - yaml rule file
  • TRIGGER_INSTANCE_DEFINITION - specification of the trigger with accompanying payload

Example:

---
    trigger: "irc.pubmsg"
    payload:
    source:
        nick: "Kami_"
        host: "gateway/web/irccloud.com/x-uvv"
    channel: "#stackstorm"
    timestamp: 1419166748,
    message: "stackstorm is cool!"
Timers#

Allow running a particular action repeatedly based on a defined time interval, or at a specific point in time. Like cron jobs.

Trigger types:

  • core.st2.IntervalTimer - run at specific intervals
  • core.st2.DateTimer - run an action at the specified date and time
  • core.st2.CronTimer - Run an action when current time matches the time constraint defined in UNIX cron format
core.st2.IntervalTimer#

Eg.

trigger:
  type: "core.st2.IntervalTimer"
  parameters:
      unit: "seconds"
      delta: 30
core.st2.CronTimer#
trigger:
  type: "core.st2.CronTimer"
  parameters:
      timezone: "UTC"
      day_of_week: 6 # or day_of_week: "sun"
      hour: 0
      minute: 0
      second: 0

Workflows#

Multiple actions across various systems. Strings atomic actions into higher level automation, orchestrates their executions and inputs, it maintains state between actions and provides reliability and transparency to the execution.

Workflows can be called by other workflows

Workflow runners#

  • Orquesta - New workflow engine, replaces both ActionChains and Mistral
  • ActionChain - Legacy no frills workflow runner. No complex logic.
  • Mistral - orignated in Openstack: complex, nested, error handling, forks, joins etc.

Orquestra#

Runs natively on stackstorm. Defined in yaml, does not require a seperate authentication system and database like Mistral.

  • A graph based workflow engine
Getting Started#

Requires an action metadata file: /opt/stackstorm/packs/<mypack>/actions

Basic example: examples.orquesta-basic

Save in /opt/stackstorm/packs/examples/actions/workflows/orquesta-basic.yaml:

version: 1.0

description: A basic workflow that runs an arbitrary linux command.

input:
- cmd
- timeout

tasks:
task1:
    action: core.local cmd=<% ctx(cmd) %> timeout=<% ctx(timeout) %>
    next:
    - when: <% succeeded() %>
        publish:
        - stdout: <% result().stdout %>
        - stderr: <% result().stderr %>

output:
- stdout: <% ctx(stdout) %>

Define the workflow in /opt/stackstorm/packs/examples/actions/orquesta-basic.yaml:

---
name: orquesta-basic
pack: examples
description: Run a local linux command
runner_type: orquesta
entry_point: workflows/orquesta-basic.yaml
enabled: true
parameters:
cmd:
    required: true
    type: string
timeout:
    type: integer
    default: 60

Then create the action:

st2 action create /opt/stackstorm/packs/examples/actions/orquesta-basic.yaml

Run the workflow:

st2 run examples.orquesta-basic cmd=date -a
Inspection#

The workflow is inspected before execution.

Other Stackstorm stuff#

  • It has a powerful workflow engine, where a shell script is linear satackstorm can branch and do various actions.
  • You can use core.ask for asking a question to a user - continue or abort

Storing secrets#

It is recommended to store encrypted values for passwords and private keys.

To store a secret:

st2 key set api_token SECRET_TOKEN --encrypt

To get the key as plain text use:

st2 key get api_token --decrypt

To use variables stored in jinja use decrypt_kv:

aws_key: "{{st2kv.system.aws_key | decrypt_kv}}"

To store dictionaries or lists:

```# Pass the result of this expression to the action st2.kv.set {{ {‘complex’: ‘structure’, ‘foo’: [‘x’, True]} | to_json_string }}

Or set it on the CLI#

st2 key set foo ‘{“complex”: “structure”, “foo”: [“x”, True]}’

Read the data back in using the st2kv and from_json_string filters#

{{ st2kv.system.foo | from_json_string }}```

In these cases it is very important to check the examples in the repo: st2/contrib/examples/actions/workflows

From the examples there are various ways to intepolate the key value variable:

With yaql:

    password: <% st2kv('system.my_password', decrypt => true) %>

with yaml:

    cmd: 'echo "{{ st2kv.system.foobar }}"'

or:

    cmd: echo "{{ st2kv('system.foobar', decrypt=True) }}"

for some reason this does not work:

    password: "{{ st2kv.system.password | decrypt_kv }}"

Troubleshooting a Rule#

Excellent post on troubleshooting a rule with stackstorm

Your don’t need a pack for everything#

why can’t you just install the Packer CLI on your StackStorm box? FYI that’s exactly how we use it. Same with Terraform, we install the Terraform CLI on the StackStorm host - NMaludy (stackstorm slack 2019)

If there is a command line tool you want to use, you can use it simply with using a core.local action.

If the tool is sitting on another server, then you could also use core.remote

Installing packs from private repos#

Sometimes you will have your own private repo to deploy on a stackstorm instance.

You can view this link on installing packs from private repos

Which comes down to doing:

st2 pack install https://<user>:<token>@github.com/username/repo.git

st2 variable#

Apparently you can use the st2 variable in a workflow: ctx(st2)

vars:
- username: {{ ctx(st2).api_user }}

Sources#