Chatbots for Security and IT Teams – Part 3: Creating A Slack Chatbot

August 19, 2019 in Blog

In Part 1 and Part 2 of our Chatbots for Security and IT Teams blog series we examined how to build Chatbots for Microsoft Teams. While Microsoft Teams has now overtaken Slack in terms of popularity, Slack retains the hearts and minds of many, and healthy competition will no doubt result in better features for both. This blog will examine how to create a Chatbot in Slack to improve communication and collaboration within your team.

This process of collaborating within a chat tool is commonly called “chatops” – “a collaboration model that connects people, tools, process, and automation into a transparent workflow” according to Atlassian.

In order for ChatOps to be successful it requires both the ability to kick-off automated actions from within the chat application, and for an automation solution to send alerts and data to the chat program both proactively and reactively.

In this blog, we will examine how to send basic notifications in Slack to a single channel, then we’ll examine how to interact with our Chatbot from within Slack, and lastly we’ll learn how to send proactive notifications to individual Slack users.

As mentioned in Part 1, the idea of Chatbots for security and IT teams is not new – security teams in Slack, Netflix and Dropbox, among others, have created open source Slack Chatbots for alerting purposes and for indicator enrichment. Creating your own Chatbot which fits your own internal processes allows you to be more flexible in your tool and process choice, and keeps your information private, however you should check out their blogs for useful ideas!

The first step to setting up your Chatbot in Slack is to create an OAuth application here – https://api.slack.com/apps. You’ll need to be an admin on your workspace, or be working with an admin, in order to do this. When you follow the link, you should be presented with a page like this.

Creating a Chatbot within Slack

Create an application called “Tines Chatbot” (or whatever name you choose). When you create your Slack App you have to choose your workspace. If this is your first time setting up an application, it may make sense to test in a demo or development workspace.

On the next page you’ll be presented with several options on what features and functions you’d like for your application:

  • Incoming Webhooks – post messages from Tines directly to a channel (e.g. the Incident Response channel). Use this to send Prompt messages to a specific channel where the channel details will not change e.g.
    • Post information details of new high priority incidents into an InfoSec channel or IT channel
    • Post alerts for incidents close to SLA limits
    • Inform teams of new servers deployed or new vulnerabilities found
  • Interactive Components – creates shortcut to specific actions with right-click actions on messages
  • Slash Commands – allow a user to interact with Tines through specific, user-defined commands e.g.
    • /searchdomain – search for traffic to a domain in logs
    • /lookupdomain – lookup a domain’s reputation
    • /lookupuser – find user profile information in active director
    • /lockaccount – lock a user’s account
    • /quarantinedevice – quarantine a device proactively
    • /escalateticket – escalate a ticket
    • /blockdomain – block a domain on the firewall
  • Bots – allow two way text communication with a user (i.e. not just clicking links to prompts)

To quickly demonstrate how to get an alert from Tines into Slack, let’s select “Incoming Webhooks”

Incoming Webhooks 

The most simple interaction with Slack are notifications sent to a specific channel via a Slack webhook. Let’s click on “Incoming Webhooks” in the screen above. If you’ve move past this screen, click “incoming webhooks” in the features menu. Then click “On” in the top right hand corner

Now click “Add New Webhook to Workspace” at the bottom of the page

Now choose a channel to post these notifcations to. I’m going to select my personal channel, @thomas. Now click “Install”

Slack will return a webhook URL. Let’s send this webhook URL a message from within Tines.

In your Tines tenant create a new story called “Slack Chatbot”. In that Story create a New Agent using the template “Post Message to a Slack Channel”

  • Give it the name “Post Message to Webhook”
  • Enter the Webhook URL you just created in the URL parameter
  • Click Save
  • Now click “Run Agent”

You should receive an alert within your chosen slack channel:

Congratulations! You’ve just sent your first message to Slack!

Next, let’s make the App look slightly more professional – choose a logo for your Chatbot in the “Basic Information” settings

Click “Save Changes” at the bottom of the page. The alert should now look a little much better:

You can use this alert to send messages to an IT or Incident Channel every time a new alert is created, for example. If you have alerts specific to an individual team you can simply install a webhook for their team channel to alert them for specific actions. You can read more about formatting Slack messages here.

Slash Commands

There are obviously significant limitations to a simple webhook – the most obvious is there is no way to communicate back to your Tines Chatbot. Fortunately this is relatively easy to do in Slack using “Slash Commands“. You can send “Slash Commands” to proactively kick off automation stories within Tines.

In this example, we’ll use the command /analyzedomain to kick-off the analysis of a domain within Tines from within any Slack Channel in your workspace.

First, let’s setup a new webhook within our Tines Story. Let’s call it “Receive Commands”

Then, in Slack, let’s go to the Slash Commands section of your application. You can choose this section in the “Features” menu on the left hand side of your page

Then click “Create New Command”

Enter the command “analyzedomain”, and your webhook URL in the “Request URL” section. Add a short description, and a usage hint. You can leave everything else as it is.

Now click “Save

Note: When you change specific permissions in slack you should get a notification saying the scopes have changed – you’ll need to “reinstall the app”

Now, let’s go to a test channel within our Workspace – you can choose whichever one you want.

Start typing “/an…” and the Slash Command should pop up

Type “/analyzedomain tines.io”. You should get the response: 

Let’s look at the data received in Tines – we can see that we have the channel id, channel name, the name of the user who sent the request, and the command, along with a few other details:

We can make this cleaner using the “response” configuration option within the Tines webhook agent. You can enter text like:

(note, the json path is simply the underlying key, excluding the agent name).

Now when you submit a command you’ll get a more contextual response:

We can now use Tines to perform an analysis of the domains submitted. Before we do, however, let’s get our Slack Credential so we can post a message back to Slack using the Tines Chatbot.

Creating a Slack Credential

The easiest method to posting a message to Slack is to use the OAuth token found in the OAuth & Permissions section. Copy this access token to create a credential called “slack_chatbot” in Tines.

In the same section, you will also need to add in the scopes “chat:write:bot” and “chat:write:user”. You will likely need to install the application again.

Now that we have our Slack configuration created we can build an Automation Story using the following agents in Tines:

  1. A Webhook agent, with the title “receive commands”, as created above
  2. A Trigger agent – trigger on the the Slash Command for the value /analyzedomain

3. An Event Transformation Agent to extract out all the domains sent for analysis

4. Another Event Transformation Agent to explode all the domains found

5. Three HTTP Request Agents to search the domain in Virustotal, URLHaus and Get the Domain Age in ipty.de.
Note, you will need to have added a Virustotal credential containing your Virustotal API Key for this agent to run successfully.

6. An Event Transformation Agent to transform these results into a cleaner format. We can use “If Widget” for this:

7. Lastly, another HTTP Request Agent to respond to the user

We can respond to the user using the channel_id received in the webhook event, and include the slack_chatbot credential we created earlier.

Rich Notifications

Earlier this year, Slack debuted the ability to build “Blocks”. They include templates of how you can create rich notifications like the below, and a “Block Builder” where you can build out your own notification templates.

Using these rich notifications you can create cleaner notifications to send to your users using the Tines Chatbot. These can also include prompts for a user to take additional action. The example included in our story is above. 

If you want you can take the “whitelist” and “block” prompts and create trigger agents to take actions based on these prompts. The completed story looks like the below

Proactive Slack Chatbot Notifications

As we noted in Part 2, a Chatbot is useful not just for responding to user requests, but it’s useful for sending messages to users within Slack proactively. There are many reasons why you might want to proactively contact a user, for example:

  • Crowdsourcing suspicious activity with users e.g. logins from unusual IPs
  • Informing a security team of a high priority incident, or an IT team of a new ticket or request
  • Confirming validity of sudo commands
  • Validating change to user permissions
  • Confirming installation of unusual software
  • Processing approval permissions from managers and service owners for access requests
  • Prompting users to take action before escalation of a ticket; to manage evidence etc.

Searching for a User Within Slack

In order to send a message to a user proactively it is necessary to find their Channel ID using their email address. You can search for a Slack user using their email address with the following agent template in Tines:

Note, to do this you will need to add the users:read and users:read.email scopes to your application. You can do this in the OAuth & Permissions section of your application. 

Once successful you’ll receive a result similar to the below in Tines:

You can now send a message to the user directly in their Private Channel using a message like this:

Congratulations! You’ve now created a Chatbot that can proactively alert users in dozens of Automation Stories. 

You can download the complete story for all the above Slack actions here.

To learn more about how Chatbots work or about other ways Tines can help your automation journey, book a demo, start a free trial, or contact us hello@tines.io. 

Updated – Microsoft Graph Security Automation

August 2, 2019 in Blog

If your organisation leverages Office 365, Microsoft Graph provides programmatic access to a wealth of data which can be used to better inform decision making during threat detection and response. In this post, we explore how to enable Tines for Microsoft Graph security automation. So that you can use information such as Outlook emails, organisational structure, advanced threat analytics and more in your security automation program.

Step 1 – Getting an app ID and secret for use in Microsoft Graph

Authenticating for Microsoft Graph security automation

We will authenticate to Microsoft Graph using an app ID and secret. To get these, we need to register a new application in the Microsoft Azure App Registrations Portal. Sign in with your Microsoft credentials. Note, you will need to be working with an administrator of your Microsoft account.

Click “New Registration”

Enter an application name and select “Accounts in this organizational directory only (yourorganization.com)”.

Then enter your callback URL. You can find your callback url in your Tines tenant by creating a new OAuth 2.0 credential. We’ll return to this page in Tines shortly.

Now click “Register”

You should be redirected to a page like this:

Now create a new application secret using the “Certificates and Secrets” tab.

Take note of the generated secret (you only see it once) and the application id, we will need these when creating a Tines credential later.

Step 2 – Selecting Scopes

Finally, we need to define the permissions this application should have, this is also referred to as the OAuth2.0 scopes. Permissions include everything from creating tasks to sending emails. A full list of permissions is available in the Microsoft Graph docs.

It is best security practice to provide the application with the minimum amount of permissions necessary to perform its required task(s).

In our example, we want to read Outlook emails using Tines, so we’ll include the Mail.read permission. To view and edit permissions go to the API Permissions Tab, click “Add a permission” select “Microsoft Graph” and then “Delegated Permissions”. Choose the relevant permissions, including “offline_access” and click “Add Permissions”

You may need to click “Grant Consent” as an administrator for some or all permissions.

Step 3 – Adding Details to a Tines credential

Next, we now need to add these details to a the Tines credential so they correspond with the application we’ve just registered. We will use this credential in our agent’s to access Microsoft Graph security data. From your Tines tenant, choose “Credentials” and “New Credential”. From the “Type” dropdown, choose OAuth2.0. Give your credential a name, I used “msgraph”, but you can use whatever makes sense in your situation.

Under “client id” and “client secret” in the “Create credential” page, enter the “application id” and “application secret” from the application you just registered in Step 1.

Copy the Client/Application ID and return to the New Credential page and copy the Secret from the Client Secrets you just created.

Under scope, we’ll enter a space separated list of the permissions we used when registering the Graph application in Step 2. That is: Mail.read and User.read. Additionally, we will include the offline_access scope. This scope will allow Tines request fresh access tokens as necessary.

From the “Grant type” dropdown, choose “authorization_code”.

Under “Oauth url” and “Oauth token url”, we need to tell Tines where to request authorization and access tokens.

You can find these under “Quickstart” > “Endpoints”

In our example we have chosen the v2 endpoints.

Having entered all the required information into the “Create credential” page, it should look similar to the below. You can optionally choose to share the credential.

When you select “Save credential”, Tines will redirect to a Microsoft account consent page, where you will be asked to authorize the application’s access to your account.

After accepting the request, Microsoft will securely redirect you to Tines.

Tines OAuth2 consent flow for Microsoft Graph Security Automation

Credential auth flow

Step 4 – Creating a Tines agent

We now have everything we need to connect Tines and Microsoft Graph. So, we’ll now use a standard Tines HTTP Request Agent to read emails from an Outlook account.

The Graph Explorer is a very useful tool for understanding how to interact with the data in Graph. Using the Graph Explorer, we can read Microsoft Graph security data. In addition, we can see that in order to read Outlook messages, we need to send a GET request to the following URL:

As such, we will create a HTTP Request with the following Options block:

Consequently When this agent runs, Tines will replace the credential widget ({% credential msgraph %}) with a valid access token. The event emitted by this agent will contain emails from my Outlook inbox. For example:

Tines - Event generated by Microsoft Graph Security Automation

Summary

In conclusion, Microsoft Graph exposes an extraordinarily rich repository of data and capabilities. By using the Tines advanced security automation platform to automate interaction with Graph, security analysts can automate their Microsoft Graph security tasks, and perform more thorough threat detection and response. Of course, all while simultaneously freeing up analyst resources and allowing them refocus on higher-impact activities.

References

Microsoft Graph quickstart guide: https://developer.microsoft.com/en-us/graph/quick-start

Chatbots for Security and IT Teams – Part 2: Microsoft Teams

July 30, 2019 in Blog

In 2019 Security and IT Teams are finding it harder to source and retain talent which is why many teams today are embracing remote workers and distributed teams. Communicating within and between remote teams is challenging, and many organizations are using communication tools like Slack and Microsoft Teams, and with them, Chatbots, to improve communication and collaboration.

Chatbots Blog Series

In Part 1 of the series we examined how to set up a chatbot within Microsoft Teams. This Chatbot received commands from users from within Teams and replied with details collected using Tines.

This tutorial will delve deeper into Microsoft Teams chatbots and examine how to send rich notifications using Cards. It will also explain how to use the Microsoft Graph API and this Chatbot to proactively find and contact users within Microsoft Teams. You can use these proactive notifications to crowdsource and confirm frequent incidents of suspicious activity from users in your organization.

In part three we’ll examine setting up Chatbots within Slack which can both take commands and crowdsource information from users.

Microsoft Teams Advanced Chatbots

This tutorial will build upon part 1 where we set up a Chatbot within Microsoft Teams. If you haven’t followed the first tutorial, click here and follow the steps to create a working Chatbot within Tines.

Sending Cards within Microsoft Teams

In our last tutorial we learned how to send replies to users who sent messages to our chatbot. A quick way to make these notifications look more professional is to send cards. Adaptive Cards are a way for developers to exchange card content in a common and consistent way in bot communications.

Cards can come in several formats. One of the most common formats is Hero Cards which contain a large image, one or more buttons, and a small amount of text:

Another common format is Thumbnail Cards. Thumbnail Cards typically contain a single, small thumbnail image, some short text, and one or more buttons.

You can make cards as complicated or rich as you deem necessary, for example using a card like below. These cards are all available as templates within Tines.

Sending Messages Proactively to Users

The next challenge is to send messages proactively to users within Microsoft Teams. There are many reasons why you might want to proactively contact a user, for example:

  • Confirming suspicious activity with a user e.g. a login from an unusual IP
  • Informing a security team of a high priority incident, or an IT team of a new ticket or request
  • Confirming validity of sudo commands
  • Validating change to user permissions
  • Confirming installation of unrecognized software
  • Processing approval permissions from managers and service owners for access requests
  • Prompting users to take action before escalation of a ticket; to manage evidence etc.

In order to send a proactive message to a user in Microsoft Teams you need two pieces of information – the tenant ID of the Microsoft Teams tenant; and the Microsoft Teams Member ID of the individual user.

The tenant ID is easy to find as you used it in part 1 to create the Tines Bot. It is also returned in any communication sent to or by the bot. It can also be found manually in the link “Get link to team” within Microsoft Teams.

The Member ID is the the “id” field in the responses when retrieving details about a Team using the Teams API. It is not to be confused with the “objectId” which is the userId used in Microsoft Graph api calls.

All conversations initiated by a user with a Bot include the Member ID of the user. This is how it’s possible to reply to a user when they proactively send a message to the Tines Chatbot.

Finding a User ID Proactively

Unfortunately, however, Microsoft Teams does not allow you to search for a user using an email address and retrieve this Member ID. According to Microsoft “This is intentional to prevent spambots within the bot framework”.

Fortunately, there are ways around this limitation. The most simple way is to fetch the team roster. If your organization has a team that all members of your organization are automatically members of then you can return all members of that team using the below command made by the Tines Bot.

You will need the teams “teamId” or “internalId” which you can find within the Microsoft Teams UI. It is the parameter with the format guid@thread.MStool in the URL below. https://teams.microsoft.com/l/team/{{internalID}}/conversations/… The Team ID is also sent in all communications to the Bot from the Team Chat within the Teams UI.

Using the data returned from the team roster, you can then filter on the user whose email address matches the email address of the user you are searching for.

An alternative solution is to use an external tool like DynamoDB to store the team roster details and, using Tines, search the table for the user id of the individual you want to contact.

Finding a Member ID using only an Email Address

For the purpose of this blog, however, we’ll examine a worst case scenario – one where you neither have a team which all employees are members, nor do you want to use a lookup table to store this information. In this case, you can perform a following series of searches in Microsoft Graph to retrieve the user id.

Finding a User and Team in Microsoft Graph

First you can search for the user’s graph ID using the email address of the user. We’re taking the “user_email” value from a “receive_events” webhook.

Using the id returned you can search for the teams they have joined

Then you can retrieve details for one of those teams. The data returned will include the “internal ID” which acts as The Microsoft Teams ID for that team.

Fetching the Team Roster in Microsoft Teams

You can then use your Chatbot to get team roster as above. This will return the members of that Team and their id which can be used to initiate a conversation with them

As this team will likely have more than one member, you will have to filter on the member whose email address matches the user you wish to contact

You can then begin a conversation with that user, and send them cards like the below.

The last step is to send a notification to the user – this can be done very easily using templates from any of the cards above. You can include a prompt which will automatically take the next step – e.g. escalating to on call, closing an incident or locking an account. The prompt can also force a second factor confirmation through Tines using a tool like DUO or Okta.

The complete Proactive Chatbots Story looks like the above, and can be downloaded from here.

The story will need to be customized for your environment. It can be edited to include just details from below the “get bearer token from ms” agent if the team id is known, or from below the “create conversation with user” if the user id is known.

Congratulations – you’ve now setup a chatbot in Microsoft Teams that can send complex alerts to any user in your organization!


In Part 3 we’ll examine how to create a Chatbot to send similar alerts in Slack.

To learn more about how Chatbots work or about other ways Tines can help your automation journey, book a demo, start a free trial, or contact us hello@tines.io. 

Chatbots for Security and IT Teams – Part 1: Microsoft Teams

July 30, 2019 in Blog

In 2019 Security and IT organizations are finding it harder to source and retain talent which is why many companies are embracing remote workers and distributed teams. Communicating within and between remote teams is challenging, and many organizations are using communication tools like Slack and Microsoft Teams, and with them, Chatbots, to improve communication and collaboration.

Often during security incident security teams create virtual rooms are to discuss the incident, investigate IOCs and take actions. Frequently multiple teams from different disciplines are invited. On IT and Product Development teams, virtual rooms are often created on a per-project basis to discuss project specific initiatives and challenges. 

This process of collaborating within a chat tool is commonly called “chatops” – “a collaboration model that connects people, tools, process, and automation into a transparent workflow” according to Atlassian.

ChatOps can be improved significantly using Chatbots – autonomous programs that interact with users within chats. They provide the “automation” part of chatops and allow users take actions from within their chat application. ChatOps and Chatbots allow analysts maintain their focus in one location, and to operate using just one pane of glass – keeping them focused on performing more meaningful and impactful work.

In order for ChatOps to be successful it requires both the ability to kick-off automated actions from within the chat application, and for an automation solution to send alerts and data back to the Chat program either proactively or reactively.

Because most security and IT tools don’t integrate natively with Slack or Microsoft Teams, you can use Tines to connect your tools together by creating a Tines Chatbot. These chatbots can leverage the full power of the Tines Automation platform and send data back to Microsoft Teams or Slack.

The idea of Chatbots for security and IT teams is not new – security teams in Slack, Netflix and Dropbox, among others, have created open source Chatbots for alerting purposes and for indicator enrichment. Creating your own Chatbot which fits your own internal processes allows you to be more flexible in your tool and process choice, and keeps your information private, however you should check out their blogs for useful ideas!

Chatbots Blog Series

This series will examine:

  • Setting up Chatbots within Microsoft Teams which receives commands from users
  • Using this Chatbot to proactively notify and crowdsource information from users within Microsoft Teams
  • Setting up Chatbots within Slack which can both take commands and crowdsource information from users

This blog will look at the steps required to set up both a communication bot which receives questions, and a proactive notification bot, using Tines and Slack and Microsoft Teams. 

At the end of this first tutorial you will have created a bot which can receive a command to analyze a domain from any use on your team, and respond with the Virustotal and URLHaus analysis of that domain.

Microsoft Teams Communication Bot

This first tutorial will examine how to set up Chatbots within Microsoft Teams which receives commands from users and replies. Before we begin, you’ll need to be, or be working with, an admin on your Microsoft account. You can also read the getting started with Bots guide, here.

Creating an application in Microsoft Teams

First, let’s begin by Installing App Studio for Microsoft Teams https://aka.ms/InstallTeamsAppStudio

Select Install. When installation is complete you should get the following notification:

Now click “Open” on setting up a Bot within App Studio.

You can also get to this page by clicking the “more” button ( … ) on the sidebar of teams and selecting App Studio

Give you application a name (e.g. the Security Chatbot) and a GUID and version number. (Note, this is the application ID, not the Bot ID which is used later on.)

Give your application a description:

Then enter Privacy Statement URL, Terms of Use URL, and some branding for your application

Creating a Chatbot for your Application

Now go to the Bots tab on the left hand side of the page:

 Create a Bot and give it the scopes “Personal”, “Team” and “Group Chat”

You should also save the ID of the Tines Chatbot, just above this.

Once you have created your Chatbot, click “Generate new password” and keep this safe! This will be used to retrieve your bearer token later. 

Now let’s go to Tines and create a webhook using the Webhook template. Let’s call it “receive commands”

Copy the webhook URL and enter it in the “Bot Messaging Endpoint” path.

Lastly, let’s add a command to “analyzedomain”

Click “Add Command”

Fill out the details of the command text and help text. Choose the scopes “Personal”, “Team” and “Group Chat”

Now, return to the menu and click “Finish > Test & Distribute”

Then click “Install”

Click “Add for You”, and select the relevant team to add the application for your team.

Then click “Install” again

If you get an error saying “uploading of custom apps is not allowed, then follow the process here. You’ll need to allow sideloading of external apps.

Receiving Data in Tines

Congratulations! You’ve just installed a Tines Chatbot!

Now, within your team chat you should be to call the “Tines Chatbot”

When you “@” the Tines Chatbot you’ll be prompted with the command “analyzedomain” or the command we selected earlier

Let’s go ahead and enter the domain “tines.io” and click enter

This data will now have been sent by Microsoft Teams to Tines. Check your webhook agent to make sure data has been sent and received:

If you do not see any data within Tines, make sure the webhook URL specified in the “Bot Messaging Endpoint” above is correct. You should also check that Microsoft Teams is able to reach that webhook (i.e. that it’s not blocked by your Firewall). If you change the webhook address you will need to re-install the application following the instructions above.

Sending a Reply to Microsoft Teams

To reply, let’s take the password you saved earlier and save it as a credential in Tines:

Now let’s examine more closely the data that Tines receives in the webhook agent

Tines has received the text of the query (analyzedomain galaxy.com). But Tines has also received several other pieces of information we need to reply:

  • The serviceurl is the base URL that Tines must send data back to using a HTTP Request Agent
  • The “from” details tell Tines who to send this reply to
  • The recipient id is the recipient ID (note, not the Bot ID) of the Tines Bot – this is needed to send the reply as the Times Bot

Lastly, Tines has received the channelData – this is important for Part 2 when we enumerate all the members of a channel or team.

Our first step is to take the password and Bot ID we saved earlier and request a Bearer Token from Microsoft which we can use to reply. Use the below template to request this token

We use the bearer token as a credential in the reply.

The reply uses the Service URL, the Conversation ID, and sends the data back to the recipient who sent the message, from the Tines Chatbot.

Chaining These results together we can reply to every message sent by a webhook:

Congratulations! You’ve just sent a reply! You can download this basic story here.

Advanced Replies

Of course, the aim of this story is to analyze the domain in VirusTotal, URLHaus and other services and then reply with the results. In Tines this is pretty easy to do – just use your own tools or templates to perform the analysis and return the details to the requestor. The example below uses VirusTotal and URLHaus. 

The first step of this process is to Trigger on the command being used (in this case, analyzedomain):

We can create separate flows for each command as we expand the functionality of the Tines Chatbot

We then extract into an array all the domains in the text

Tines can then query these domains in virustotal and urlhaus, among others. 

A complete flow will look like the following:

This story will reply with simple results to the user like the following:

You can download this complete story here.

Additional Use Cases

Now that we have a process to receive data from Microsoft Teams and reply, there are near unlimited possibilities of what we can automate within Tines. Other automation steps you can use with the Tines Chatbot include:

  • Enrich Domains, IPs, URLs
  • Retrieve User Profile Information from Workday, MSGraph etc.
  • Escalate Tickets to On Call
  • Create Tickets, Subtickets or Increase Ticket Severity
  • Add Comments to Tickets
  • Block Domains and IP addresses
  • Query Logs
  • Kick Off Vulnerabiltiy Scans and PenTests
  • Isolate Hosts or Lock User Accounts
  • Send Push Notifications to other users
  • Post all updates for an incident into a team room
  • Post all comments from an incident room to an incident ticket
  • Sharing Incident Handover Notes
  • Proactively run any automation story

Next Up: Part 2 – How to Proactively Send Notifications in Microsoft Teams

To learn more about how Chatbots or other ways Tines can help your automation journey, book a demo, start a free trial, or contact us hello@tines.io. 

Tines Summer Release 2019

July 12, 2019 in Blog

On today’s blog we’re delighted to announce details of the latest and greatest Tines features launched in the Tines Summer 2019 release. The Tines Summer Release is jam-packed with new features including:

  • Agent Templates & Private Templates
  • Improved Searching
  • Time-based Deduplication
  • Emit and Tag Duplicate Events
  • Emit and Tag Non-Matching Trigger Events
  • Asynchronous Event Loading

Existing Cloud tenants always stay on the latest release so Tines Cloud customers do not need to take any action. Tines On-Premise and Kubernetes customers can login to the Tines customer portal and download the release and installation instructions now. 

Agent Templates

You asked, we answered! The most exciting figure of the Tines Summer Release 2019 is Agent Templates. Tines now has automation templates for nearly 1,000 security actions commonly performed by security teams for the most popular security products.

Sample templates include:

  • Create A New Issue in Jira
  • Isolate a Host in Carbon Black
  • Search for Hash in VirusTotal
  • Disable a User Account in Microsoft Graph
  • Retrieve Email Headers in Outlook
  • Upload an Attachment to Box
  • Search for Details within Tickets in Service Now 
  • Create a New Alert in The Hive
  • Upload Samples to the Hybrid Analysis Sandbox
  • Scan a DynamoDB Table
  • Retrieve Analysis Results from App.Any.Run

It’s important to note that Tines integrates automatically with any tool in your stack with any API, regardless of the templates that exist. Templates help jump-start automation stories but are just that: a springboard on which you can begin automating all your manual workflows!

To view all available templates now, simply create a new agent within Tines. You will be presented with a list of hundreds of automatically generated templates which can be filtered by vendor, agent type, and privacy level. You can also search on the right hand side for specific terms like “Carbon Black” or “MD5”.

Users can still build agents from scratch using the “Start with a Blank Agent” tab.

Got a suggestion for agent templates that we’re missing? Email hello@tines.io and we’ll add them in right away!

Private Templates

In addition to the thousand public templates that are now available, Tines has also enabled “Private Templates”. If you have a private API that you use internally, or if you have custom fields and configurations for your own tools (like Jira, Splunk, AWS etc.) you can create your own Private Agent Templates within Tines. These templates are viewable to everyone within your company, and can be shared among all your Tines production and test tenants.

Creating Private Templates

To create a Private Template, find an agent that you have saved, and in the Actions Menu click “Create Template”. (Note, only Tines admins are able to create Private Templates).

Fill the appropriate details in the “Create a New Agent Template” page

Your template will then be visible in the “Manage Templates” page in the Admin Tab in your Tines tenant.

In addition, you will be able to choose this template from within the “Create New Agent” templates page.

You can also view all your Private Agent Templates using the Visibility: “Private” filter on the left hand side of the Agent Template search page.

Retry on Status Failure in HTTP Request Agents

When trying to automate manual processes using Tines, custom scripts, or any automation platform, customers often run into a stubmling block: when an action fails or is interrupted (e.g. when sites are down, or when the receiving server detects an error, or is rate limited the script) the entire automation flow fails. Common causes of this are rate limits on the server or a simple network blip. When an error occurs in automation stories or in scripts it can be tough to detect, and in some cases the entire automation flow fails. 

To tackle this problem in Tines you can now add an optional flag to every http request agent called “fail_on_status”. With this flag enabled, if Tines receives a non-2xx http response code when an agent runs it will re-run the agent 40 times with an exponential back-off over a 30 day period, until it receives a 2xx http response code. Now when Jira is down, or when VirusTotal returns a 429 rate limit response code, Tines will auto-rerun the agent with the same incoming event. Your Tines automation story will then continue as soon as the service is back-up. A sample configuration is below

Improved Search

We’re delighted to announce that the Summer Release includes a much improved search interface within Tines. The Search bar in the top right hand corner will now search and return results for Stories, Agents and Credentials. It performs a full text search within agents configurations too, so you can find all agents which reference a particular hostname or use a particular command. Try it out now in your own tenant!

Time Based Deduplication

One of the most frequent causes of fatigue in information security teams is alert overload. That’s why in Tines we have a “deduplication” mode within Event Transformation Agents – to suppress noisy alerts and prevent analysts having to repeat the same work over and over again.

In Tines we recognize that you often need to suppress events for a set period rather than than just ignoring all duplicate events. If an alert fires, you may want to suppress that same alert for another 24 hours, or simply not see it for another 100 events, or ever again. As a result, we have enhanced our deduplication mode in the event transformation agent – you can now deduplicate based on Time Period or based as well as based on a Lookback through previous emitted events.

  • A time-based deduplication analyzes each event that is received for uniqueness, and subsequent matching events will not be emitted until this time period has elapsed. A sample time based deduplication is below.
  • A lookback deduplication will examine the previous X events for uniqueness, regardless of when the events happened. It takes a parameter “lookback” which will be the number of events to store which Tines checks against for uniqueness.

Emit Duplicate and Emit on No Match

Emit Duplicates in Event Transformation Agents

A complementary feature launched along with Time Based Deduplication is an emit_duplicate flag for deduplication events and an emit_no_match for trigger events. 

When the emit_duplicate flag is set to “true”, in deduplication mode, duplicate events are emitted by the Event Transformation Agent. Duplicate events return the value “unique_event”:”false” in the emitted event, non-duplicate events will return the value “true”. Using this flag, users can create more complex stories, e.g. adding details of duplicate events to existing tickets, creating lower priority duplicate alerts, or taking a lower-risk action based on the fact it is a duplicate event. A sample configuration is below.

Emit on No Match in Trigger Agents

Similar to the “Emit Duplicate” flag, the emit_no_match flag is also available within Trigger Agents. Events which do not match the trigger agent’s rules can now be emitted, but will have the field “rule_matched” value set to ‘false’. Events which match the rule will have the “rule_matched” value set to ‘true’. This new feature allows users build and maintain a set of trigger rules within one agent.

A sample configuration for a trigger agent with emit_no_match set to true is below.

Asynchronous Event Loading

The last major feature of the Summer Release is an under-the-hood user experience improvement. When using Tines to automate AWS workflows; collect logs; analyze malware; and other common use-cases, some events in Tines can become extremely large. Previewing these Events within Tines is now much faster thanks to our new Asynchronous Event Loading feature. Tines will now only show the event data that the user wants to see. Expanding the json in the View Events page will then dynamically pull back the relevant data from the Tines database. Asynchronous Event Loading allows users to quickly preview the relevant section of the event, without waiting for the entire event to be downloaded. Each event should now take just fractions of a second to load making for a more seamless user experience.

That’s all for this year’s Summer Release. To get on the beta to test new features as they are being developed, simply talk to your Tines account manager. 

To learn more about how these features can help your automation journey, book a demo, start a free trial, or contact us hello@tines.io. 

Processing and Enriching AWS Security Hub Findings in Tines

July 5, 2019 in Blog

With AWS Security Hub, Amazon have provided a way for AWS customers to “quickly see their entire AWS security and compliance state in one place, and so help to identify specific accounts and resources that require attention.”

Security Hub went GA in July 2019 and although there is debate around the material value the service will provide, specifically in terms of ROI (when it’s enabled, 30+ Config rules are created per account, this can quickly become expensive), the benefit for enterprise security teams of having a centralised portal for Inspector, GuardDuty and CIS benchmark findings is intriguing.

In this post we will explore how to send findings from Security Hub to Tines so they can be enriched, prioritised, deduplicated and ticketed.

How AWS Security Hub Works

When you enable Security Hub, it immediately begins consuming, aggregating, organizing, and prioritizing findings from AWS services, such as Amazon GuardDutyAmazon Inspector, and Amazon Macie, and from AWS partner security products. Security Hub generates its own findings by running continuous, automated compliance checks based on AWS best practices and supported industry standards. It then correlates and consolidates findings across providers to help you to prioritize the most significant findings.

As AWS Security Hub discovers findings, it will automatically send them to CloudWatch Events. As a result of this automated process, it’s simple to trigger notifications to Tines through SNS Topics.

Tines AWS Security Hub Automation Story

Here you will find a Tines automation story which you should download and import into your Tines tenant. The story contains five agents. including a Webhook agent which we’ll use to receive events from Security Hub.

AWS Security Hub Tines Automation Story
Tines Webhook Agent Receive SNS Notifications

Take a note of the Webhook URL from the Summary tab in the Tines agent view, we’ll need to provide this to AWS. In the above example, the Webhook URL is: https://hq.tines.io/users/1/web_requests/1162/de38b6203ae66ed5ec6b76ba419f7f8e

Using the Tines AWS Security Hub CloudFormation Template

Next you will need to configure AWS Security Hub to send CloudWatch Events to Tines. Although you can do this manually, we also provide a CloudFormation template which does the hard work for you.

Download the template from here and upload it to CloudFormation.

Once you have uploaded the file, click Next and give the stack a name, then provide the following parameters:

EventPatternParameter: { "source": [ "aws.securityhub" ] }

TinesWebhookURL: The Webhook URL taken from the Receive AWS Security Hub Notification.

After selecting Create Stack, CloudFormation will begin creating the stack. When CloudFormation is finished creating the stack, it sends a new SNS Subscription Confirmation Event to Tines (sample below).

AWS Security Hub SNS SubscribeConfirmation Event in Tines

We’ve configured the Confirm subscription HTTP Request Agent to send a GET request to the SubscribeURL defined by SNS. This confirms the SNS subscription so Security Hub so it will now send Findings to Tines.

Receiving AWS Notifications in Tines

You should now have everything needed to begin automating response to Security Hub Findings in Tines. When Security Hub triggers a Finding, it will send a notification event to the Tines Webhook agent. A sample event is shown below:

AWS Security Hub Notification Event in Tines Security Automation Platform

The important information describing the Security Hub finding is in an escaped JSON string, this makes further automation challenging. To parse this string into a “friendlier” format, we use the Liquid Filter json_parse in a message_only mode Event Transformation Agent.

Events emitted by this agent will contain the Finding’s details in a format we can easily use in Tines to further enrich, deduplicate, prioritise and even automatically remediate the Finding.

To learn more about AWS Automation in Tines, or what this might look like in your environment, you can book a demostart a free trial, or contact us hello@tines.io. 

Tines {} urlscan automation

June 14, 2019 in Blog

On this week’s blog, we are delighted to announce that Tines is sponsoring one of our favorite tools, urlscan.io. Welcome urlscan users to the Tines website! In this blog you’ll learn more about urlscan automation including how you can automate your URL analysis processes; search for IOCs within urlscan; search for leaked credentials; and share threat intelligence with the security community.

For those Tines readers unfamiliar with urlscan, you’re one of today’s lucky 10,000! urlscan.io is a website scanner built by Johannes Gilger, which scans and classifies almost 100,000 urls every day. This includes submissions from thousands of public and enterprise users and security researchers and all urls in openphish, phishtank, certstream, urlhaus and more. urlscan runs all the analysis on its own servers and records http request data; all domain interactions; all links on the scanned page; the website technologies in use; a hash of every file on the page; and ssl certificate detection, as well as related scans, IP information, google safe browsing information for the domain and more.

Even better, urlscan makes all this information available, for free, via an intuitive and well built API. This makes automating scanning, searching, and interacting with urlscan through the Tines security automation platform incredibly easy.

For those of you visiting Tines for the first time, Tines is a Security Orchestration, Automation and Response (SOAR) platform that helps security teams automate any repetitive manual task. If you are unfamiliar with Security Automation, you can check out our ‘getting started’ guide. If you are familiar with security automation, you can read about why Tines is different than all other SOAR platforms. (hint: we don’t rely on any prebuilt integrations – you can integrate easily with every tool in your technology stack!)

Why sponsor urlscan?

At Tines we’ve long been fans of URLScan. Before setting up Tines we worked as security engineers in eBay/PayPal and DocuSign, some of the most phished brands in the world. When we had to analyze thousands of phishing urls we quickly realized that manually analyzing them one-by-one was time consuming, error prone and, frankly, boring. As a result, we turned to automation and we started using urlscan.io. When we started Tines urlscan’s incredible API made it easy to showcase how to analyze urls, and to share threat intelligence back to the community.

It’s no surprise that many enterprise security teams rely on urlscan.io to analyze suspicious URLs. It’s also no surprise that urlscan has been mentioned heavily in other blog posts by Tines! At Tines we want to give help ensure urlscan continues to be an incredible resource for the security community.

Does Tines integrate with urlscan?

Yes! urlscan is a tool which exposes all its analysis information up front in a clean and simple to use API. Because of this, it’s very easy for Tines customers to search for and submit urls to urlscan. We have several out of the box stories which harness the power of urlscan. Customers can easily customize these to suit their own needs and processes.

As mentioned above, Tines does not rely on pre-built apps to integrate with external systems. Instead, the HTTP Request Agent (one of the six agents available in Tines) provides direct integration with the target tool, in this case urlscan. This means consistent integration with any tool, regardless of the vendor, regardless of whether it’s open or closed-source, and regardless of whether it’s commercial off the shelf or custom built.

Tell me about urlscan automation in Tines!

The primary purpose of urlscan is to analyze urls. Those familiar with urlscan will know that every page that is analyzed is categorized and given a malicious score verdict:

the verdict of a url analysis in urlscan

The most obvious process to automate, therefore, is the analysis of urls sent to employee or customer abuse inboxes. You can read more about Tines in depth and out-of-the-box abuse inbox processing here. If you are spending significant time analyzing urls you should consider automating that process using an automation platform like Tines.

Submitting a url to urlscan through Tines is easy:

You can also use Tines to pull suspicious URLs from other sources which can then be analyzed in urlscan.io. Common sources of malicious or suspicious urls include:

  • URLs blocked by your email security solution like Proofpoint, FireEye ETP, Barracuda, Mimecast or Microsoft APT.
  • DMARC failures or rejects
  • Suspicious uncategorized or punycode URLs from your firewall logs or DNS logs
  • New SSL Certificates registered with domains similar to your brand (e.g. from crt.sh)
  • Threat Intel sources like the Phish.ai threat intel feed which generates feeds based on the brands attacked
  • Free feeds of malicious urls like Phishtank, Openphish, phishstats.info or Urlhaus. Note, these feeds are often are high-reputation so don’t necessarily need to be further analyzed.
automate the process for scanning urls from Tines in urlscan

Using Tines’ Phishing Story it’s easy to collect suspicious urls from dozens of different sources automatically. Once these feeds are in Tines it’s easy to deduplicate and classify urls to prevent alert overload and to generate more accurate metrics.

Does urlscan detect if a site is malicious?

urlscan results in Tines

Yes! The above screenshot shows that a verdict, or overall malicious score, is returned in the urlscan UI based on an analysis of the content on the page. This verdict takes into account the classification of the domain and IP in other security tools like GSB, openphish, phishtank, urlhaus etc.

This verdict is also returned in API calls, so we can use this information to automate the url analysis process. (note, this urlscan api feature is in beta mode so may change in the future)

Using the information returned via the urlscan API we can build a trigger agent to flag urls classified as malicious. We can then take additional actions including blocking that URL; scanning for traffic to the domain in our environment; sending takedown notices for malicious content; creating tickets for analysts etc.

urlscan automation to take action on all malicious urls submitted
completing the urlscan url analysis automation process

What else can I automate with urlscan.io?

Automate IOC extraction

Urlscan.io records the hash of every file it downloads as an indicator of compromise or ‘ioc’. Using the urlscan API you can search for other pages with this same IOC. This means if you’re a highly phished brand, for example, or a researcher tracking a phishing campaign, you can search for pages with similar IOCs which may be using the same phishing kit. E.g. searching for this md5 (a PayPal logo) will return several thousand other pages impersonating PayPal.

You can automate this search using Tines to extract urls with matching IOC every hour and issue takedown notices for pages abusing your brand, for example.

Search for leaked credentials or access tokens

Unfortunately, users and analysts occasionally mistake legitimate emails as suspicious and use urlscan to analyze legitimate web pages. Occasionally sensitive information like document access links or password reset tokens are exposed. For many services the url itself can be enough to give an attacker access to an account or to sensitive information.

With a trivial amount of effort searching urlscan (no, we won’t give you the searches here!) you can find several password reset tokens for high profile enterprise services as well as access links to enterprise file sharing services like Dropbox, OneDrive etc..

For more information on how to see if your company’s accounts or website might be affected, and how you can automate the detection of these for your enterprise, you can read this Tines blog on the topic.

Share threat intelligence with the community

urlscan is valuable threat intelligence tool for researchers and security professional, however it’s only as good as the data that the community submits and shares. If you have a feed of malicious urls you have detected privately, you can give back to the security community by sharing this information automatically to urlscan using Tines. Sharing threat intelligence with urlscan means researchers and other security teams can keep their customers, companies and the wider internet community safer.

For more information you can read this Tines blog on how to share threat intelligence information using urlscan.

Conclusion

In conclusion, integrating with urlscan.io is easy with Tines. If you find yourself using urlscan frequently to analyze urls you should consider looking at an SOAR platform to help with urlscan automation and let your team focus on more impactful risk reduction efforts.

To learn more about the automating URL analysis, or what this might look like in your environment, you can book a demostart a free trial, or contact us hello@tines.io.


Malware Analysis Automation using Public and Private Sandboxes

May 31, 2019 in Blog

Performing malware analysis on suspicious files is a bread and butter activity of any security operations or incident response team. Whether submitted to an abuse inbox, caught by an email gateway, detected by anti-virus, or found during a breach investigation, the malware analysis process is time-consuming, repetitive and manual – which is why many teams are examining malware analysis automation.

There are dozens of approaches to analyzing potentially malicious files and binaries, including using static and dynamic analysis . For now at least, nothing will perform better than a sophisticated malware reverse engineer interacting with and analyzing a file manually in a secure environment. However until humans can work at the scale and speed of malware analysis engines, relying on some form of automation is necessary.

One of the most popular methods of Malware Analysis Automation to determine the maliciousness of suspicious files is using public and private sandboxes. Popular sandboxes include Any.Run, Hybrid Analysis, Joe Sandbox, Valkyrie Sandbox, Cuckoo Sandbox. In this blog we examine some private and public sandboxes that analyze suspicious files. We’ll also learn how the results of the analysis can help proactively protect our environments.

Firstly, a word of caution: at Tines we don’t want you to think that you can completely automate the process of securing your environment by analyzing suspicious files. There are dangers and pitfalls to completely automating the analysis of malware. Modern malware often requires multiple applications to be running on a box for the malware to be triggered. Other malware can detect that it’s running in a sandbox. Furthermore, in many cases different contamination levels will require different triggers. Automated sandboxes struggle to accurately simulate the activities of a real, infected end-user. However, several sandboxes like app.any.run allow interactive analysis with malware and may help in this regard.

When should I automate the analysis of malware?


Using a Sandbox is the right approach for frequent, repetitive malware analysis tasks. A good example of a process like this is analyzing files which AV software detects as suspicious. Attachments which users submit to an abuse mailbox are another source of files which frequently require non-sophisticated malware analysis.

There are several free, public sandboxes available online, however if you suspect that a suspicious file may be targeted at your organization directly you should consider using a private sandbox. This will help prevent a targeted attacker knowing you have detected their activity.

You can setup your own free private sandbox like Cuckoo Sandbox, or you can make a private submission to sandboxes like app.any.run, hybrid analysis or other commerical sandboxes like the Crowdstrike Falcon or Palo Alto Wildfire.

How can I upload files programatically?

Uploading files to app.any.run, cuckoo, or hybrid analysis from Tines is simple once the contents of the file are in Tines. Tines can read the contents of email attachments, or, in some cases, extract the contents of files in a shared drive or the contents of a quarantined file. Once the contents are base64 encoded it’s possible to upload them using any api with a file upload capability.

In the below templates can see how to upload a file to three popular online sandboxes – Cuckoo, App.Any.Run and Hybrid Analysis. In the examples we simply replace the base64 encoded contents with the contents from a previous agent and you can upload to any sandbox. Below you’ll see examples of how to upload to any.run, hybrid analysis and cuckoo sandbox. There are also templates available for uploading to VirusTotal and several other sandboxes in your own Tines tenant.

Before uploading to any sandbox, we recommend checking to see if the file has been seen before in tools like VirusTotal or Hybrid Analysis or your own threat intel platform. If VirusTotal or your threat intelligence platform has seen the file before we can avoid duplicating work and take response actions immediately. You can read more about VirusTotal automation here.

We can easily check if a file exists in VirusTotal using the below agent template

This template requires the MD5 of the file to check in VirusTotal. If you do not have the MD5 of the file, you can use the agent template below to extract the md5 of a file in Tines.

Note, you can also upload files programmatically from a desktop using curl and a command similar to:

Or

Analysis of Manually Uploaded Files

At Tines we recognize that many processes involve analyzing files found manually – for example suspicious files found during a breach investigation. This does not mean that the results of the analysis can not be extracted and automated however. Using Tines we can extract indicators from every file analyzed in your private sandbox, regardless of how it is submitted.

Analyzing the results of all files uploaded to a Malware Sandbox

The most obvious aim in analyzing malware using a sandbox is to determine its maliciousness. A secondary aim is to extract potential indicators which can be searched for across our environment.

The best way to automate this in Tines is to create a HTTP Request Agent. The HTTP Request agent periodically polls your Sandbox for any new files that have been uploaded. Tines can then extract all the relevant indicators for those reports.

In this story, we’ve used several liquid filters to extract out the relevant elements of the malware analysis report:

The end result is an event that looks like the below – replete with registry modifications, file modifications, network connections etc. The extracted data is in a format that can then be easily used by other agents.

Post Analysis Automation Actions

You can use these results of this file analysis to take other automated actions in your environment. For example, for every domain that the malware connected to you can you can search for associated traffic to the network indicators. If the file was found to be malicious, you can ban malicious hash from endpoints in your enterprise. You can also search for unique file modifications or registry modifications in an EDR tool, for example. In addition, the Tines Security Automation, Orchestration and Response platform can use these results to  

  • Isolate or quarantine infected endpoints
  • Block domains or IP addresses in Firewalls or in DNS tools
  • Document details about the file to ticketing systems like JIRA, including PCAPs for analysis etc.
  • Collate all the analysis of multiple to help build better detections on known threats for your environment
  • Use the detailed results to help prioritize detections using the Mitre Attack Framework
  • Upload artifacts like malware PCAPs into ServiceNow or Jira or The Hive
  • Perform memory dumps on infected endpoints
  • Block email addresses or domains on your email gateway

In short, automating the malware analysis process can help security operations teams react more quickly to potential threats. This allows them to focus on more impactful, risk-reduction efforts

Note – this should not be taken as a complete method for analyzing malware. The correct approach will depend on your environment and risk tolerance. However this is illustrative of some of the analysis that you can automate using Tines and malware sandboxes.

Conclusion

Malware Analysis Automation can have several benefits in allowing teams to move quickly and automatically extract the most relevant data from malware reports.
To learn more about the automating malware analysis, or what this might look like in your environment you can book a demo, start a free trial, or contact us hello@tines.io. You can also directly download the Cuckoo Sandbox Story or App.Any.Run Story directly for your own Tines tenant.

Automate the Analysis of Email Headers Using Tines.io

May 2, 2019 in Blog

Continuing our series analyzing on automating the analysis of phishing messages, this blog will look at the importance, and methods, for analyzing email headers.

It is becoming harder and harder to determine the validity of suspicious emails. Malware distributors are using unique URLs for every recipient, compromising or creating hundreds of new domains every day, developing more sophisticated malware detection evasion techniques, and even now hijacking real conversations.

To be successful, analysts should be using all the tools at their disposal. One of these tools is analyzing email headers. Often overlooked, Email headers contain important information about the route an email took before arriving in a recipient’s inbox and this information can help determine the legitimacy of a given email. Spammers frequently and easily spoof messages to make them look like they were sent from somewhere else. As such, it’s important to know how to analyze these headers correctly.

With many Tines customers running their own abuse inbox, it’s no surprise that one of the most frequent requests we hear from our customers is how they can automate the analysis of email headers.

Why Analyze Email Email Headers

According to RFC 2822 from IETF, all email messages must pass through certain characteristics to be processed by the receiving mailbox. Contained in these headers is a huge amount of information that can tell us more about the message and its authenticity. Headers can help us determine the sender’s IP, ISP, server, the tools they used to send the email, and the route the message took to arrive at its destination. Furthermore, they can even tell us the malware group that sent the message. When analyzing email headers there a few fields which are the most important to analyze.

What Email Headers should I analyze?

Originator Headers

Originator headers include common fields like ‘From’, ‘To’, ‘Subject’, ‘Originator-Date’ which are set by the sending mail server. Unfortunately, because the sending server sets these headers, a determined attacker can easily spoof them so they are not as valuable as other headers. They do, however, also include the message-id field which can be useful in determining the legitimacy of a message.

Message-ID

Perhaps the most overlooked field in message headers is the “Message-ID”. From Emotet/Geodo to Ursnif to Phorphiex/Trik (not to be confused with Trickbot!) the message-id field is often used by a botnet to track its operations and establish which ‘bot’ sent a particular message. It can also be used to detect whether your organization is being spoofed (e.g. if the message header shows it’s from @yourdomain.com, but doesn’t match your message ID pattern).

We can analyze the message-id field for certain patterns to help us identify whether the message was by a particular botnet. For example, the Emotet Group have previously used the Message-ID pattern:

For example, the Emotet Group have previously used the Message-ID pattern:

<20 numeric characters>.<16 hex characters>@<recipient domain>

Or, more literally:

11223344556677889900.0123456789ABCDEF@recipient-domain.com (see this article by Cofense for more information)

In Tines we can write a regex to catch this message header, using a Trigger Agent:

trigger agent for searching for email headers

More recently Emotet have been using patterns like a 51 character hex string followed by @recipient-domain.com, or <11.22.334455.AA55CC99@recipient-domain.com> and . We can also include regexes for these patterns in the same event transformation agent.

trigger agent to search for emotet message id patterns

Note, these are sometimes prone to false positives – for the most up to date version you should check with your threat intel vendor, or you can contact Tines and we’ll be able to assist!

Authentication-Results

The Authentication-Results header is a trace header field where a receiver can record the results of email authentication checks that it carried out. Multiple results for multiple methods can be reported in the same field, separated by semicolons and wrapped as appropriate [0].
Frequently included in the Authentication-Results header is information on whether the sender passed DMARC, DKIM and SPF.

dmarc and dkim authentication results

The best way to search for these is through regexes for the pass or fail values. In Tines we can generate a message-only mode to give the results: dkim=neutral, pass or fail, which simply extracts out the results:

If the DMARC, DKIM or SPF results returned are “Fail” then it’s possible an attacker has spoofed the message.

Trace Fields

Trace Fields are a group of header fields which provide trace information and provide an audit trail of message handling. In addition, they also indicates a route back to the sender of the message. The main Trace Fields to analyze are the ‘Return-Path’ and ‘Received’ headers.

a. Return-Path

The final transport system that delivers the message to its recipient adds a ‘return-path’ header.  This field is supposed to contain accurate data about route back to the message’s originating server.

b. Received

Every time a server or transport service relays a message it adds a new ‘Received’ header to the message. There are often three or more received headers associated with a single message. The first server that handled the message will have the ‘bottom’ received entry. Therefore, you should read the ‘Received’ Headers from the bottom up, as . This information is very useful to help us investigate phishing or spam. With this information we can find the server used to send the message and what relays delivered it. We can also use this information to determine if any open relays or relays known for sending spam sent the mail.

In the example below, we can tell that the message was sent from z17.autocontabil.com. The IP 139.99.75.19 then received the message and sent it to the recipient.

Analyzing the IP 139.99.75.19 in Talos Intelligence, for example, we can see this is a known Spam IP with a poor email reputation.

automate the analysis of email headers ips by searching in Talos Intelligence by Cisco

Using Tines and a Liquid Tag we can extract the last message header from headers Array in a Event Transformation Agent:

We can also extract all the IP addresses from ‘Received’ headers using Tines

Using Tines we can also automate the process of checking the IPs against known blacklists like Cisco Talos Intelligence (or any other Threat Intelligence Provider) which will give us an ip address reputation score for sending email.

Lines Beginning with X

Receiving Email Servers also add their own email analysis of the message which is useful when analyzing. If received by your own email servers, these are the completely trustworthy entries.

The most valuable of these are often x-originating-ip and x-php-originating-script – these will extract out information we have extracted out previously. We can then automate checking them against a blacklist.

Received-SPF

The Received-SPF header is a useful way of determining whether a message has been spoofed. For example, a ‘permanent error’ in the ‘Received-SPF’ header most likely indicates that the mail has been spoofed by an attacker and the sender policy framework has failed. Note, however, this not always included in Message Header, but it can be a

You can read more about how to manually extract email headers here https://mxtoolbox.com/Public/Content/EmailHeaders/

How to Extract Email Headers

Knowing what to analyze is often just half the problem in the analysis of email headers. The other half of the problem is extracting the message headers themselves. Fortunately, using the Tines Security Automation and Orchestration platform, it’s possible to retrieve Message Headers from any email platform. In the below two examples we’ll analyze headers in emails in Microsoft Office 365 and in GMail.

Using Microsoft Office 365

Microsoft’s default configuration for viewing emails in Office 365 using the Graph API does not automatically returning message headers, however to retrieve the message headers of a message in Office 365 is pretty simple, as Microsoft expose the endpoint:

If we do not know what the emailId is we can search for messages using different keywords or terms in the user’s mailbox. (Be careful not to confuse the emailId with the ‘message-id’ header!)

Extract Headers from Microsoft Attachment

Because this practice is prone to error and false positives, a lot of companies already automate process of sending suspicious messages as attachments to their abuse inbox. In this case, we can also use the graph api to analyze the message headers of the attachment.

(Note, in order for this to work, Microsoft must recognize the attachment as an #microsoft.graph.itemAttachment )

This request returns email headers in the format of an array in the field ‘internetMessageHeaders’

The end result is a Story which looks like this:

automate the analysis of email headers in microsoft graph

You can download this story for your own Tines tenant here.

Using GMail

Gmail makes the analysis of email headers easy as it automatically returns message headers within the message itself. In contrast to Microsoft, headers are always returned in GMail with a simple ‘read email’ request using the email’s ‘emailId’. (Again, do not confuse this with the Message-ID header!). As an example see the agent ‘retrieve email from gmail’ below:

You can find the extracted headers in the ‘payload.headers’ path of the json returned.

Consequently, every field in the array can easily be referenced in future agents using a ‘where’ liquid loop:

Extract Headers from Gmail Attachment

Gmail also make the process of analyzing mails sent as attachments to an abuse inbox very easy. The first step is to get the contents of the attachment, by attachmentId.

First, we find the attachment details in the original message:

Then, taking the attachment ID we can create a query in Tines to return the raw eml file:

This query will return data in the form of a base64url encoded version of a .eml file:

We can take this ‘data’ returned and import it directly into Gmail, which will create an email that can be recursively analyzed in gmail. The upload will return an ‘id’ and ‘threadId’.

We can read the new email using the same configuration from the ‘retrieve email from gmail’ agent above. Any attached email files will also be pulled out recursively. The end result is a flow which looks like this:

automate the analysis of email headers in GMail

You can download this story for your own Tines tenant here.

Next Steps

To learn more about the analysis of email headers, or what this might look like in your environment you can book a demo, start a free trial, or contact us hello@tines.io . You can also directly download the Microsoft Story or GMail Story directly for your own Tines tenant.

[0] https://en.wikipedia.org/wiki/Email_authentication

Useful Resources:
https://sqrrl.com/hunting-email-headers/
https://mlhale.github.io/nebraska-gencyber-modules/phishing/email-headeranalysis/
https://www.alyninc.com/2018/11/10/email-headers-what-can-they-tell-the-forensic-investigator/
https://cofense.com/category/threat-intelligence/page/4/
https://cofense.com/dark-realm-shifting-ways-geodo-malware/

DynamoDB and Tines Security Automation

April 3, 2019 in Blog

An increasingly popular database choice amongst security teams is AWS DynamoDB. The key-value storage, simplicity, scalability and security offered by DynamoDB make it suitable for the kinds of data storage tasks common in security operations and incident response, especially if they already use AWS.

In this post we’ll explore how security teams can use DynamoDB in their automation stories.

Authenticating to AWS DynamoDB from Tines

To begin integrating Tines with AWS DynamoDB, we first need to create a credential. In your AWS console, create an IAM user with the appropriate permissions to perform actions in DynamoDB. Take the access key and access secret for the user and enter them into a new Tines AWS mode credential.

Next, specify a name for the credential and choose the AWS region you will be working with. Finally, under service name enter ‘dynamodb’.

When finished your Tines AWS credential should look like the below:

Creating a Tines AWS credential

Using the AWS Credential

AWS credentials work a little differently to the other credential modes in Tines. When a HTTP Request Agent with an AWS mode credential included in a header called “Authorization” runs, Tines will use the AWS Signature Version 4 Signing Process and include the corresponding auth headers in the request before submitting it to AWS.

For example, the below HTTP Request agent uses an AWS mode credential (aws_cloudtrail) to list cloudtrails in the us-east-1 region.

When this agent runs, the request will be signed and will be converted to the following before being sent to AWS:

DynamoDB Tines Agents

Tines can perform all available DynamoDB actions. The following agent examples cover a selection of the cost common.

List Amazon AWS DynamoDB Tables

Scan an Amazon AWS DynamoDB table with a filter

Scan an Amazon AWS DynamoDB Table

Delete an Amazon AWS DynamoDB table

Create an Amazon AWS DynamoDB Table

Add an item to Amazon AWS DynamoDB table

Get an item from an Amazon AWS DynamoDB table

Delete an item from an Amazon AWS DynamoDB table

Summary

By including DynamoDB actions in Tines automation stories, security teams can quickly and reliably fetch and store important data, allowing them enrich security incidents and make better decisions around incident investigation and remediation.

For more information on how Tines can automate interaction with DynamoDB and other AWS services, contact us here.