Skip to content
Blog
Tines {} urlscan automation

On this week’s blog, we are delighted to announce that Tines is sponsoring one of our favorite tools, urlscan.io. Welcome urlscan users to the Tines website! In this blog you’ll learn more about urlscan automation including how you can automate your URL analysis processes; search for IOCs within urlscan; search for leaked credentials; and share threat intelligence with the security community.

For those Tines readers unfamiliar with urlscan, you’re one of today’s lucky 10,000! urlscan.io is a website scanner built by Johannes Gilger, which scans and classifies almost 100,000 urls every day. This includes submissions from thousands of public and enterprise users and security researchers and all urls in openphish, phishtank, certstream, urlhaus and more. urlscan runs all the analysis on its own servers and records http request data; all domain interactions; all links on the scanned page; the website technologies in use; a hash of every file on the page; and ssl certificate detection, as well as related scans, IP information, google safe browsing information for the domain and more.

Even better, urlscan makes all this information available, for free, via an intuitive and well built API. This makes automating scanning, searching, and interacting with urlscan through the Tines security automation platform incredibly easy.

For those of you visiting Tines for the first time, Tines is a Security Orchestration, Automation and Response (SOAR) platform that helps security teams automate any repetitive manual task. If you are unfamiliar with Security Automation, you can check out our ‘getting started’ guide. If you are familiar with security automation, you can read about why Tines is different than all other SOAR platforms. (hint: we don’t rely on any prebuilt integrations – you can integrate easily with every tool in your technology stack!)

Why sponsor urlscan?

At Tines we’ve long been fans of URLScan. Before setting up Tines we worked as security engineers in eBay/PayPal and DocuSign, some of the most phished brands in the world. When we had to analyze thousands of phishing urls we quickly realized that manually analyzing them one-by-one was time consuming, error prone and, frankly, boring. As a result, we turned to automation and we started using urlscan.io. When we started Tines urlscan’s incredible API made it easy to showcase how to analyze urls, and to share threat intelligence back to the community.

It’s no surprise that many enterprise security teams rely on urlscan.io to analyze suspicious URLs. It’s also no surprise that urlscan has been mentioned heavily in other blog posts by Tines! At Tines we want to give help ensure urlscan continues to be an incredible resource for the security community.

Does Tines integrate with urlscan?

Yes! urlscan is a tool which exposes all its analysis information up front in a clean and simple to use API. Because of this, it’s very easy for Tines customers to search for and submit urls to urlscan. We have several out of the box stories which harness the power of urlscan. Customers can easily customize these to suit their own needs and processes.

As mentioned above, Tines does not rely on pre-built apps to integrate with external systems. Instead, the HTTP Request Agent (one of the six agents available in Tines) provides direct integration with the target tool, in this case urlscan. This means consistent integration with any tool, regardless of the vendor, regardless of whether it’s open or closed-source, and regardless of whether it’s commercial off the shelf or custom built.

Tell me about urlscan automation in Tines!

The primary purpose of urlscan is to analyze urls. Those familiar with urlscan will know that every page that is analyzed is categorized and given a malicious score verdict:

the verdict of a url analysis in urlscan

The most obvious process to automate, therefore, is the analysis of urls sent to employee or customer abuse inboxes. You can read more about Tines in depth and out-of-the-box abuse inbox processing here. If you are spending significant time analyzing urls you should consider automating that process using an automation platform like Tines.

Submitting a url to urlscan through Tines is easy:

You can also use Tines to pull suspicious URLs from other sources which can then be analyzed in urlscan.io. Common sources of malicious or suspicious urls include:

  • URLs blocked by your email security solution like Proofpoint, FireEye ETP, Barracuda, Mimecast or Microsoft APT.
  • DMARC failures or rejects
  • Suspicious uncategorized or punycode URLs from your firewall logs or DNS logs
  • New SSL Certificates registered with domains similar to your brand (e.g. from crt.sh)
  • Threat Intel sources like the Phish.ai threat intel feed which generates feeds based on the brands attacked
  • Free feeds of malicious urls like Phishtank, Openphish, phishstats.info or Urlhaus. Note, these feeds are often are high-reputation so don’t necessarily need to be further analyzed.
automate the process for scanning urls from Tines in urlscan

Using Tines’ Phishing Story it’s easy to collect suspicious urls from dozens of different sources automatically. Once these feeds are in Tines it’s easy to deduplicate and classify urls to prevent alert overload and to generate more accurate metrics.

Does urlscan detect if a site is malicious?

urlscan results in Tines

Yes! The above screenshot shows that a verdict, or overall malicious score, is returned in the urlscan UI based on an analysis of the content on the page. This verdict takes into account the classification of the domain and IP in other security tools like GSB, openphish, phishtank, urlhaus etc.

This verdict is also returned in API calls, so we can use this information to automate the url analysis process. (note, this urlscan api feature is in beta mode so may change in the future)

Using the information returned via the urlscan API we can build a trigger agent to flag urls classified as malicious. We can then take additional actions including blocking that URL; scanning for traffic to the domain in our environment; sending takedown notices for malicious content; creating tickets for analysts etc.

urlscan automation to take action on all malicious urls submitted
completing the urlscan url analysis automation process

What else can I automate with urlscan.io?

Automate IOC extraction

Urlscan.io records the hash of every file it downloads as an indicator of compromise or ‘ioc’. Using the urlscan API you can search for other pages with this same IOC. This means if you’re a highly phished brand, for example, or a researcher tracking a phishing campaign, you can search for pages with similar IOCs which may be using the same phishing kit. E.g. searching for this md5 (a PayPal logo) will return several thousand other pages impersonating PayPal.

You can automate this search using Tines to extract urls with matching IOC every hour and issue takedown notices for pages abusing your brand, for example.

Search for leaked credentials or access tokens

Unfortunately, users and analysts occasionally mistake legitimate emails as suspicious and use urlscan to analyze legitimate web pages. Occasionally sensitive information like document access links or password reset tokens are exposed. For many services the url itself can be enough to give an attacker access to an account or to sensitive information.

With a trivial amount of effort searching urlscan (no, we won’t give you the searches here!) you can find several password reset tokens for high profile enterprise services as well as access links to enterprise file sharing services like Dropbox, OneDrive etc..

For more information on how to see if your company’s accounts or website might be affected, and how you can automate the detection of these for your enterprise, you can read this Tines blog on the topic.

Share threat intelligence with the community

urlscan is valuable threat intelligence tool for researchers and security professional, however it’s only as good as the data that the community submits and shares. If you have a feed of malicious urls you have detected privately, you can give back to the security community by sharing this information automatically to urlscan using Tines. Sharing threat intelligence with urlscan means researchers and other security teams can keep their customers, companies and the wider internet community safer.

For more information you can read this Tines blog on how to share threat intelligence information using urlscan.

Conclusion

In conclusion, integrating with urlscan.io is easy with Tines. If you find yourself using urlscan frequently to analyze urls you should consider looking at an SOAR platform to help with urlscan automation and let your team focus on more impactful risk reduction efforts.

To learn more about the automating URL analysis, or what this might look like in your environment, you can book a demostart a free trial, or contact us hello@tines.io.


Want the latest?

Sign up for our newsletter!

The Tines security automation platform helps the world's leading security teams automate any manual task.

Making them more effective and efficient.