We're sometimes asked, "Can you run Bash or Python scripts natively in Tines?" and today, we're sharing the answer, and weighing in on the debate between full-code, low-code and no-code automation.
The short answer is yes, you can run scripts and Linux commands natively in Tines, however, you might not necessarily want to. Many of our customers avoid doing so for security, usability and performance reasons. Let's take a closer look at some of these potential pitfalls.
Security risks. By allowing the use of custom code, you’re also allowing any engineer or analyst with access to Tines to potentially write unstable or dangerous code. You then have to deal with importing arbitrary libraries. This introduces new risks of supply chain compromise, additional dependency management, and potentially conflicts between version requirements between workflows.
Maintenance. Your workflow automation platform is likely outside the scope of organizational supply-chain controls, so you’ll have to create new processes to manage the security of the custom code you're using in workflows.
Performance issues. You'll have to deal with long-running code, timeouts, and max memory usage.
At Tines, we have some customers that want to accept the risk of custom code. For these customers, we provide a feature called Command-over-HTTP. This acts as a hardened sandbox and allows users to execute scripts and commands in a “safer” environment. You'll still have to deal with long-running code, timeouts, and max memory usage, and it's worth noting that these aren’t skills your typical analyst would or should have. Read our how-to guide to learn more.
Given all the above, we find that our customers, even those with a Python development background, prefer using the native Tines builder. With Tines, you can do everything you can do with scripting, but it’s faster, more secure, and more collaborative.
Automation provides a practical solution to the challenges of increasing workloads and higher-than-average levels of burnout. And when it comes to automation, more and more practitioners are considering alternatives to Python, PowerShell, or Bash. This is where Tines comes in.
Common building components
When automating tasks, there are building patterns common to all technologies. These are HTTP requests, Webhooks, Data Manipulation, and Scheduling.
HTTP Requests
One of the most common tasks in automation is making HTTP requests to retrieve data. Below are examples of how Tines and Python can be used to make the same request to VirusTotal.
Webhooks
Another way of triggering automation is through webhooks. A webhook listens for an HTTP request. These remove the need to constantly query a service to catch an event.
Data Manipulation
Automation is used to ingest, process, and output data in a more user-friendly or usable form. The act of processing the data is called data manipulation.
Scheduling
Automations are built to run autonomously, to speed up repetitive tasks. Many of these tasks have to run at regular intervals. Scheduling handles tasks such as:
generating weekly reports,
gathering daily tasks, or
deactivating unused assets or credentials
Credentials
When accessing data on a system, a process will need to be authenticated. This is to verify that the system accessing the information has the required permissions. When using HTTP requests, this is done through a variety of methods, such as basic auth, OAuth2.0, session keys, and API tokens. Credentials and secrets must be handled carefully to avoid credential leakage.
Layout and Readability
Python, like most programming languages, uses newlines and indentation to divide itself into sections. Functions can also be used for extra readability. This can lead to long blocks of code which can be difficult to read and intimidating when complicated programming features are used, especially for newer developers.
In Tines, these are naturally broken into simple function-like elements called actions. Each action has unique styling, quickly informing the user what is does. These actions can be grouped to make single actions into user-defined functions. They can also be moved around on the story page to help the user see what actions are working together. The connections between them quickly let a user know what each action follows on from.
Manageability
As a Python script grows, functions can be used to reduce the amount of code duplication. In Tines, Send to Stories can be used to create a similar solution. The added advantage of using Send to Stories is that these stories can be reused across different automation jobs, much like importing a new Python file.
Case Study: Automation - Slack News Feed
The Hacker News is a well-known and reputable website when it comes to cybersecurity news. There are a few benefits to introducing a feed:
Help a security team stay up to date with potential threats
Help employees keep visibility on issues and alerts important to the company
RSS feeds use XML to distribute their information. An RSS Reader works by querying a feed and sending the information to the user. In terms of Slack, you would want to remove all articles previously displayed and only send a message if the article is new. In this case, we are doing this by querying the feed daily and removing any article older than a day. This can be changed to query hourly, or even once every minute.
Slack Preparation
Before we can build this automation, we need to create a Slack channel to post into, and a Slack bot to post the messages.
Slack Channel
To create a Slack channel:
Go to your workspace
On the top of the second section of the sidebar, click the Channels dropdown
Select Create and Create a Channel
Name the channel hacker-news-feed.
Slackbot
To create a new Slackbot, go to api.slack.com/apps and select Create New App. In the menu that appears, select From an app manifest, choose your workspace, and paste the manifest below into the text box (Make sure YAML is selected).
display_information:
name: Hacker News Feed
description: The Hacker News RSS Feed
background_color: "#3732b3"
features:
bot_user:
display_name: Hacker News Feed
always_online: false
oauth_config:
scopes:
bot:
- chat:write
settings:
org_deploy_enabled: false
socket_mode_enabled: false
token_rotation_enabled: falseThe manifest above creates a bot called Hacker News Feed. When added to a channel, the bot has permission to write messages within a channel it’s in.
After creating the Slack bot, you will be in the bot’s settings, in the Basic Information tab. You should see a section called Install your app with a button Install to Workspace. Press this button to install your new bot onto your workspace.
After installing the bot, go to the OAuth & Permissions tab under Features on the left taskbar. There, you will find a section called OAuth Tokens for Your Workspace. Copy the Bot User OAuth Token. This is your Slack bot Credential (slack_auth_token).
Add the Bot to your Channel
With both your channel and bot created, go back to the Slack channel. In the message bar, tag your slack bot (@hacker-news-feed) and select the button Add to Channel.
Python
Requirements
For this, we are going to need three libraries in Python. One of these comes installed with Python, but the other two need to be installed. These libraries are:
datetime
Used to convert the article published date into a DateTime object
requests
Used to make an HTTP request to retrieve the RSS feed
xmltodict
Converts the RSS feed from XML to a Python dictionary object
The libraries needed to be installed are requests and xmltodict. Below you can see the commands that need to be run to install the packages.
$ pip3 install requests
$ pip3 install xmltodictPython Code
Script Configuration
Before we create the RSS feed, the packages should be imported, and important variables should be set. This includes:
thn_feed_url
The RSS feed URL we are subscribing to
slackbot_url
Slack API URL to post a message to a channel
slack_channel_name
Name of the Slack channel to post in (make sure to include the #)
slack_auth_token
Slackbot’s OAuth token (Blocked out for security)
feed_threshold
The age limit of the articles to post (created in the last day)
from datetime import datetime
import requests
import xmltodict
thn_feed_url = 'https://feeds.feedburner.com/TheHackersNews'
slackbot_url = 'https://slack.com/api/chat.postMessage'
slack_channel_name = '#hacker-news-feed'
slack_auth_token = 'xoxb-█████████████-█████████████-████████████████████████'
feed_threshold = datetime.date.today() - datetime.timedelta(days=1)Retrieving the RSS Feed
When retrieving an RSS feed from The Hacker News, no data needs to be passed to the server. Therefore, making an HTTP GET request returns the feed in XML format. This is then parsed using the xmltodict library to get a format we can manipulate the feed with.
thn_feed = requests.get(
thn_feed_url
)
dict_feed = xmltodict.parse(thn_feed.content)Extracting the New Articles
To extract the new articles for today, we need to loop over the feed in dictionary format. Each article is stored in the article variable and the published date is extracted. If the publication date is newer than the threshold date, we can post the article to Slack.
for article in dict_feed['rss']['channel']['item']:
published = datetime.strptime(article['pubDate'], '%a, %d %b %Y %H:%M:%S %z')
if published > feed_threshold:Posting to Slack
Before the article can be posted to Slack, it needs to be formatted into an acceptable format. In this case, it is in the format:
Title
Link
DescriptionOnce the message is created, the headers and payload are set with the required variables, and the HTTP POST request is made to Slack.
slack_message = '*{}*\n{}\n\n{}'.format(article['title'], article['link'], article['description'])
headers = {
'Content-type': 'application/json',
'Authorization': 'Bearer ' + slack_auth_token
}
payload = {
'channel': slack_channel_name,
'text': slack_message
}
slack_response = requests.post(
slackbot_url,
headers=headers,
json=payload
)Tines
Requirements
Before creating the story, there are some values we can set. Tines has variables split into Resources and Credentials. Both can be used in multiple stories, but where resources are plain text JSON objects, Credentials are used for hiding generating, and restricting access to auth tokens.
Credentials
After navigating to the Credentials folder in your team, multiple types of credentials can be created after selecting New Credential. You can read more on each type of credential here, but for this story, we will be using a Text Credential for our Slack bots authentication token.
After selecting the Text Credential, give it a name like slackbot_thn and paste The Hacker News Feed’s Bot User OAuth Token into value. For added security, slack.com can be added to the domain field. This ensures that this credential will only be sent for Slack API calls and will throw an error if it tries to be used elsewhere.
Creating the Story
Retrieving the RSS Feed
When retrieving the RSS feed, we need to use an HTTP Request. After providing the URL to the feed and specifying that we are using a GET method, we can retrieve the feed. Optionally, we can specify the content type to be XML as this is the data format used by RSS feeds.
After exploding all the items into articles, we can then pass them through an event trigger. These triggers act like if statements, letting us emit events if they pass certain conditions. In this case, we only want to emit an article if it is younger than 1 day old.
To do this, we can use the Date formula to convert the published date into Unix time. To specify the time limit, we use the Date formula again to get the current Unix time and subtract the length of a day (86400 seconds). Instead of putting the day length into the formula, we can use a Local value. This acts as a variable only present in a single action.
There are multiple reasons you would want to do this. You can use a local value to clean up and simplify an input before putting it into an action, or if you wanted to use a value in multiple parts of the action, or in this case, to make the action more descriptive and improve readability and editability later.
Posting to Slack
When posting to Slack, we will be using an HTTP request again. Like before, the URL and method need to be specified. We are also choosing to specify the content type but this is not necessary. Unlike the last HTTP request, a payload and header need to be specified. If these are not present in your HTTP request action, they can be added by going to the + ption button at the bottom of the build action pane.
The payload acts as the body of an HTTP request. In this case, we need to specify the channel name and the text body. To format the body of the text, we can type in this field and add newlines like a simple text editor. In this case, we will be using the same format as used in the Python script.
In order to bring in the article text, we have to use pills in the text field. To create a pill, press ctrl + space. Pills allow you to use formulas and specify values from previous actions. Using the values as seen in the example below will pull the relevant article data into the story.
Finally, we need to specify the authentication token. The headers of an HTTP request are automatically generated but can be added to or changed using the Headers option. To use the Headers option, add a header key-pair value with the plus symbol to the right. This will create two boxes, the first of which is the header and the second is the value. In this case, we are using an Authentication header with the value being the slack bot token we created earlier.
As we created the credential in the Tines credential store, we can reference this using a pill. Create the pill as before, however, using the CREDENTIAL formula key, we can specify an credential we have access to from the credential store.
And that's it - we're done!
