Threat Intel RSS Feeds via Twitter Lists

The level of knowledge sharing that takes place within infosec is amazing! Many security researchers take time to publish their scripts, tips, successes, and failures on Twitter for all to see, so as a security professional, it’s important to learn how to effectively use Twitter to hone your craft.

• Red teamers can learn new tactics, techniques, and procedures (TTPs) by following other red teamers.
• Blue teamers can learn new detections or preventative controls published by other blue teamers.

To really take it to the next level, everyone can watch and learn from each other in a cycle of continuous improvement:

• Red observes blue’s defensive tips and adjusts their attacks to evade the controls
• Blue models and signatures red’s new TTPs to tweak their controls accordingly

Drinking from the fire hose

So all of this great information is flying back and forth on Twitter, but you have a job to do and can’t just spend your whole day trying to keep up with your timeline. How can you start sifting through the noise to get what you need?

Curated Twitter Lists

When I was part of an adversarial simulation team, I felt it essential to treat certain researcher’s tweets as ‘daily must reads’ so I could continue adding to the long list of attacks we needed to simulate and then signature. I initially tried to tackle this by creating Twitter Lists to help segment off those users that I cared most about.

I now have lists for:

• Red team TTPs
• Blue team defenses
• Hardware security and reverse engineering
• General infosec news
• Close friends

The point is to make lists that mean something to you and that will help you prioritize what tweets you will read every day. But while this is a necessary first step, I found it to be difficult to keep up with and experienced syncing issues across devices.

RSS Feeds

My solution to this problem was to apply an old technology (RSS) to a newer technology (Twitter). That might seem like going backwards, but it has helped me stay focused on using Twitter to keep me up to date on the topics I care most about.

If you don’t know how RSS works (or haven’t used an RSS feed since the Google Reader days), you begin by subscribing to certain feed inside a reader app (like Feedly or Inoreader). Every time a new article is posted, the feed is updated by the content provider and then your reader pulls the new article. So, much like how Twitter shows you new tweets as they are published, your feed reader will show you new articles as they are published.

Where the real advantages come into play is how a feed reader handles that differently than Twitter. Twitter is essentially a constant stream of tweets that you can jump into the middle of whenever you want to see what’s going on. RSS feed readers keep track of which articles you’ve already seen and help you manage viewing them all, kind of like a task list.

To help merge the two concepts, I created a script that will poll Twitter for tweets from predefined lists and then map that to an RSS feed. This means that even if you step away from Twitter for a week, you don’t have to worry about what you missed because your feed reader will have the important tweets waiting for you.

Implementation

First off, you can use services such as Zapier to do this for you to some extent. I tried that first, but wasn’t happy with the lack of customization and the high monthly subscription cost. So I decided to write a small python script that runs on AWS Lambda to do this instead, for pennies a month!

In short, the final solution will:

1. Run a python script to turn Twitter List Timeline’s into an RSS feed
2. Publish the RSS feed to an S3 bucket
3. Your feed reader polls the S3 bucket looking for new tweets

At this point, you might be thinking this is way too much work. It might be! If that’s the case for you, I recommend ponying up the money for Zapier and being done with it. In my opinion though, the value you get from setting this up far outweighs the one-time investment of your time.

Prerequisites

1. Install Python 3 and pip3 (refer to your OS-specific installation instructions)
2. Install the AWS CLI and configure your credentials
3. Install git or just download the packaged zip file manually
4. Install jq

Initial Configuration

Twitter Setup

At this point, let’s say you already have your Twitter lists created for the different types of tweets you care about. You can learn more about Twitter Lists here.

For this demo, I created a red-oriented list named offensive and a blue-oriented list named defensive. Now we need to enable our developer account by following Twitter’s instructions. Be warned, this goes through a manual review process by Twitter so it will take a few hours!

Once you are accepted as a developer, you just need to create an app to receive your API keys.

1. Go here and click on Create an app
2. Fill in your details, then click Create and Create again to accept the user agreement
3. Click on Permissions, Edit, choose Read-only, and then Save
4. Click on Keys and tokens and then Create. Make a note of all four keys, as we’ll need them for our config file.

Create the S3 bucket

Now, we’ll need to create an S3 bucket to store our RSS feed, served up as a static website. For this demo, I’ll be using fracturelabs-rss-demo for the bucket name.

This is really simple using the AWS CLI:

1. aws s3api create-bucket --acl private --bucket fracturelabs-rss-demo
2. aws s3 website s3://fracturelabs-rss-demo/ --index-document index.html

You’ll also have to edit the lambda-exec-policy.json file and set the Resource by updating bucket_and_folder_path with your bucket and folder path. For example, ours now looks like this: arn:aws:s3:::fracturelabs-rss-demo/*

Building and deploying the code

Get the code

The code for this can be found on our Github repo. You need to make sure you have Python 3 installed, as this code will not work with Python 2!

To get the code and packages, you’ll need to do the following:

git clone https://github.com/fracturelabs/TwitteRSS
cd TwitteRSS
pip3 install -r requirements.txt --system -t packages

Config updates

You’ll need to edit the config.json file to meet your needs:

• Enter your Twitter API keys in the twitter section
• Enter your S3 bucket name in the s3.bucket section
• Enter the S3 folder name (or "" if none) in the s3.folder section
• Enter any random string in the s3.filename_salt section. This is only to help give the final RSS filename some uniqueness so people don’t just stumble upon your RSS feed. Not that it’s very sensitive, but there’s no reason to incur any S3 costs for someone accessing your files!
• Edit the feeds section to meet your needs. In our example, I would put something like ‘Tweets - Offensive’ in the title section and ‘offensive’ in the lists section.
• Edit the preferences section to whatever fits your needs. Usually the defaults will do just fine.

Here’s a sample config file that would result in three feeds: one for offensive, one for defensive, and one that joins both. This is just meant to show some of the options, you probably wouldn’t want all three.

{
  "twitter": {
    "consumer_key": "REDACTED",
    "consumer_secret": "REDACTED",
    "access_token": "REDACTED",
    "access_token_secret": "REDACTED"
  },

  "s3": {
    "bucket": "fracturelabs-rss-demo",
    "folder": "/",
    "filename_salt": "red leather yellow leather"
  },

  "feeds": [
    {
      "title": "Tweets - Offensive",
      "lists": [
        "offensive"
      ],
      "preferences": {
        "max_items": 50,
        "exclude_retweets": "false",
        "require_retweets": "false",
        "exclude_quotes": "false",
        "require_quotes": "false",
        "exclude_tweets_with_media": "false",
        "require_tweets_with_media": "false"
      }
    },
    {
      "title": "Tweets - Defensive",
      "lists": [
        "defensive"
      ],
      "preferences": {
        "max_items": 50,
        "exclude_retweets": "false",
        "require_retweets": "false",
        "exclude_quotes": "false",
        "require_quotes": "false",
        "exclude_tweets_with_media": "false",
        "require_tweets_with_media": "false"
      }
    },
    {
      "title": "Threat Intel",
      "lists": [
        "offensive",
        "defensive"
      ],
      "preferences": {
        "max_items": 50,
        "exclude_retweets": "false",
        "require_retweets": "false",
        "exclude_quotes": "false",
        "require_quotes": "false",
        "exclude_tweets_with_media": "false",
        "require_tweets_with_media": "false"
      }
    }
  ]
}

Build and deploy the code

We’ll build a deployable package and load it into AWS with:

# Build the package
(cd packages; zip -r9 ../build.zip .); zip -g build.zip twitterss.py config.json

# Create a Lambda role and attach a policy to it
policy_arn=$(aws iam create-policy --policy-name Twitter-Lambda-Policy --policy-document file://lambda-exec-policy.json | jq -r ".Policy.Arn")
role_arn=$(aws iam create-role --role-name TwitteRSS-Lambda-Role --assume-role-policy-document file://assume-lambda-policy.json | jq -r ".Role.Arn")
aws iam attach-role-policy --role-name TwitteRSS-Lambda-Role --policy-arn ${policy_arn}

# Create the Lambda function
function_arn=$(aws lambda create-function --function-name TwitteRSS-Demo --runtime python3.6 --handler twitterss.twitterss_handler --zip-file fileb://build.zip --role ${role_arn} --timeout 30 | jq -r ".FunctionArn")

# Create the scheduling (every five minutes)
rule_arn=$(aws events put-rule --name TwitteRSS-Lambda-Trigger --schedule-expression 'rate(5 minutes)' | jq -r ".RuleArn")  
aws lambda add-permission --function-name TwitteRSS-Demo --statement-id TwitteRSS-Lambda-Demo --action lambda:InvokeFunction --principal events.amazonaws.com --source-arn ${rule_arn}
aws events put-targets --rule TwitteRSS-Lambda-Trigger --targets "Id"="1","Arn"="${function_arn}"

# Test the new function - should return DONE
aws lambda invoke --function-name TwitteRSS-Demo output.log && cat output.log

# Look for the RSS feed filenames in S3
aws s3api list-objects --bucket fracturelabs-rss-demo | jq ".Contents[].Key"

Troubleshooting Lambda can be difficult, so if you have any problems, it’s best to go into the console and run tests from there and/or look at the Cloudwatch logs for any errors.

Making updates

To rebuild and redeploy your package after making any config adjustments:

(cd packages; zip -r9 ../build.zip .); zip -g build.zip twitterss.py config.json
aws lambda update-function-code --function-name TwitteRSS-Demo --zip-file fileb://build.zip
aws lambda invoke --function-name TwitteRSS-Demo test.log && cat test.log

Add the feeds to your RSS reader

At this point, you’ll have an RSS feed (or multiple if you configured multiple in your config file) stored in your S3 bucket. The filenames are all output to Cloudwatch, so you could go into there to get the full URLs or you can just assemble it yourself like this:

https://s3.amazonaws.com/ + <bucket_name>/ + <folder_name>/ + <object_key>

You can also get a list of the objects with this command:

aws s3api list-objects --bucket <REPLACE_WITH_BUCKET_NAME> | jq ".Contents[].Key"

For this demo, the following feeds have been created. Feel free to inspect these or use these for your own readers if you’d like (no guarantee how long I’ll keep them up for though!)

• Offensive
• Defensive
• Threat Intel (both Offensive and Defensive)

All you have to do is put that into your favorite RSS reader and it will start polling for changes. I’ve noticed InoReader polled consistently every 30 minutes, whereas Feedly took a little while to warm up to it. It would do the initial poll, but then nothing for a while. It eventually polls much quicker once it realizes it’s an active feed though, so just stick with it!

Here’s a sample of what this looks like in Feedly.