Provided by SANS

About the course

Immeasurable amounts of personal and potentially incriminating data are currently stored in the websites, apps, and social media platforms that people access and update daily via their devices. Those data can become evidence for citizens, governments, and businesses to use in solving real financial, employment, and criminal issues with the help of a professional information gatherer.

Many people think using their favorite Internet search engine is sufficient to find the data they need and do not realize that most of the Internet is not indexed by search engines. SEC487 teaches students legitimate and effective ways to find, gather, and analyze this data from the Internet. You'll learn about reliable places to harvest data using manual and automated methods and tools. Once you have the information, we'll show you how to ensure that it is sound, how to analyze what you've gathered, and how to make it is useful to your investigations.

This is an entry-level, introduction to open-source intelligence (OSINT) course and, as such, will move quickly through many areas of the field. You will learn current, real-world skills, techniques, and tools that law enforcement, private investigators, cyber attackers, and defenders use to scour the massive amount of information across the Internet, analyze the results, and pivot on interesting pieces of data to find other areas for investigation. Our goal is to provide the OSINT knowledge base for students to be successful in their fields whether they are cyber defenders, threat intelligence analysts, private investigators, insurance claims investigators, intelligence analysts, law enforcement personnel, or just someone curious about OSINT.

Throughout the course week, students will participate in numerous hands-on labs using the tools and techniques that are the basis for gathering free data from the Internet. More than 20 labs in this course use the live Internet and dark web to help students gain real-world confidence. You'll leave the course knowing not just how to use search features on a website, but all of the scenario-based requirements and OSINT techniques needed to gather truly important OSINT data.

Course Syllabus

SEC487.1: Foundations of OSINT

Overview

We begin with the basics and answer the questions "what is OSINT" and "how do people use it." This first day is about level-setting and ensuring that all students understand the background behind what we do in the OSINT field. We also establish the foundation for the rest of the week by learning how to document findings and set up an OSINT platform, and we discuss effective research habits for OSINT analysts. This day is a key component for the success of an OSINT analyst because without these concepts and processes in place, researchers can get themselves into serious trouble during assessments by inadvertently alerting their targets or improperly collecting data, making them less useful when delivered to the customer.

During the first half of the day we work through the pieces of the OSINT cycle to understand what our process might look like. Then we move into how law-abiding people use OSINT to get the data they need, be it parents trying to figure out if the person they want to hire as a child care worker is trustworthy, a person "googling" someone they are going out with on a date, businesses looking for information about a rival company, or law enforcement using social media and OSINT to capture criminals. We then move into how criminals use OSINT to target victims and conduct other attacks against people and their electronic data. We finish the morning with a review of documentation tools for creating MindMaps, compiling notes, creating timelines, and analyzing relationships within data.

The day continues by jumping into understanding threat profiles so that we can protect ourselves and infiltrate the places we need to gather the data our customers want. Recognizing that some students will be creating their own OSINT collection platforms, we move into what that may look like, covering topics such as the platform, operating system, networks, tools, and plugins. At the end of the day, students learn about the use of sock puppet (false accounts) and set up their own account for the labs in the course.

CPE/CMU Credits: 6

Topics

• Understanding OSINT
  • The OSINT cycle
• Goals of OSINT Collection
  • Law enforcement
  • Parents
  • Spouses
  • Businesses
  • Media
  • Intelligence agencies
  • Criminals
• Diving into Collecting
  • How just "diving" into an assessment can cause problems
  • Case study
• Taking Excellent Notes
  • Why good note-taking is important
  • How to document
  • Tools to document
    • Visualization
    • Note-taking applications
    • Documentation application/Hunchly
    • Word processing
  • Taking screenshots
  • Timelines
• Determining Your Threat Profile
  • How "covert" do you need to be?
  • Methods that may reveal what OSINT is doing to a target
• Setting Up an OSINT Platform
  • Types of platforms
    • Virtual machine (VM)
    • USB media
    • Cloud server
  • Networking and VPNs
  • Web browsers
    • Useful extensions and add-ons
  • Data at rest
  • Mobile OSINT
  • Password management
• Effective Research Habits
  • Engaging your target
  • Sensitive data
  • Sanitizing your platform
  • Managing your time
  • Being disciplined
• Creating Sock Puppets
  • What is a sock puppet?
  • Why do we use them?

SEC487.2: Gathering, Searching, and Analyzing OSINT

Overview

OSINT data collection begins on day two after we get a glimpse of some of the fallacies that could influence our conclusions and recommendations. From this point in the class forward, we examine distinct categories of data and think about what it could mean for our investigations. Retrieving data from the Internet could mean using a web browser to view a page or, as we learn in this section, using command line tools, scripts, and helper applications.

Our focus for day 2 begins with creating an OSINT assessment process and examining several OSINT frameworks that will help us while conducting our assessments. We then move to harvesting data from and about websites. Analyzing data such as SSL/TLS certificates and Google analytics IDs can be important to our assessments. Shifting from systems to people, we examine different methods for finding and validating basic data about people such as home addresses, phone numbers, and email addresses. These pieces of data become "pivot points" for our investigations, as we can often perform additional searches using these key data points to discover additional data that may be useful in our work. Students learn how to harvest user names and avatars and how they tie a single user to multiple user profiles across sites.

With user avatar images fresh in our minds, we pivot and consider how to perform reverse image searches. Day two concludes with a deep look at how we can execute advanced search engine queries to increase our chances of getting meaningful results.

CPE/CMU Credits: 6

Topics

• Data Analysis Challenges
  • Inaccurate data
  • Bias
  • Analysis fallacies
• Creating Your OSINT Process
  • Start with a clean system
  • Gathering requirements
  • Decide on TTPs
  • Gathering data
  • Analyzing data
  • Creating output for customer
• Harvesting Web Data
  • Proxy web applications
  • Command line tools for harvesting data
  • Scripting with Python
  • APIs
  • Cached content
  • Google Analytics
  • Encryption certificates
• OSINT Frameworks
  • PTES
  • Advanced Recon Framework (ARF)
  • OSINT Start.me pages
  • Additional OSINT web resources
• Basic Data: Street Addresses
  • Why gather street addresses?
  • Places you can find street addresses
  • Online newsletters
  • Real estate sites
• Basic Data: Phone Numbers
  • Reverse phone look-ups
  • Places you can find phone numbers
  • Removing personal data
• Basic Data: Email Addresses
  • Places you can gather email addresses
  • Email formats
  • Email validation
  • Gathering emails in bulk
• User Names
  • Understanding why we collect user names
  • Websites and tools that can be used to harvest user names
  • Social media account aggregation sites
• Avatars and Reverse Image Searches
  • Why avatars are interesting to OSINT
  • Image search engines
• Leveraging Search Engines
  • Detailed examination of how to use several search engines for OSINT
  • Advanced queries and directives
  • Google dorks and Google Hacking database

SEC487.3: Social Media and Geolocation

Overview

Finding data on people, especially basic content such as email addresses, home addresses, and phone numbers, can be made easier using online people search engines. This is how day three kicks off, examining free and paid choices in this data aggregator area and understanding how to use the data we receive from them. Some of these engines provide social media content in their results. This makes a terrific transition for us to move into social media data.

The first social media site we look at from an OSINT perspective is Facebook, with its worldwide reach. Students explore Facebook profiles, groups, events, and other Facebook objects using graph searches and Facebook query techniques. We then move to detailed examinations of LinkedIn, Twitter, and Instagram, and what OSINT data can be found in each of them.

An increasing number of social media sites allow users to geolocate themselves. The afternoon of day 3 starts with an examination of how to harvest and use this content for OSINT. Then, focusing on the "social" aspect of social media, we dive into the content on dating and adult websites. A natural progression from dating is sometimes a wedding, so we inspect wedding websites and registries for OSINT data. Next, we see how we can use web and traffic cameras for remote reconnaissance. We finish the day by examining document and image metadata to glean interesting data points from different document types.

CPE/CMU Credits: 6

Topics

• People Searching
  • Free people search engines
  • Paid consumer-level search engines
  • Commercial aggregators
  • Family trees
  • Obituaries
• Facebook Analysis
  • Facebook primer
  • Intro to graph search
  • Websites that make searching Facebook easier
  • Crafting custom graph searches
  • Finding intersections and relationships
  • Gathering business data
• LinkedIn Data
  • OSINT value of LinkedIn
  • Data that can be gleaned from LinkedIn
• Instagram
  • OSINT value of Instagram
  • Retrieving data from Instagram with custom URLs
  • Instagram API
• Twitter Data
  • OSINT value of Twitter
  • Searching Twitter
  • Tweet content analysis
  • Twitter geolocation
  • Deleted tweets
  • Gathering data from protected accounts
• Geolocation
  • OSINT value of geolocation
  • Faking GPS locations
  • Analysis of social media systems that geolocate users
  • Geolocation discovery tools
• Dating and Adult Websites
  • Harvesting data from dating and adult site user profiles
• Registries and Wish Lists
  • OSINT value of registries and wish lists
  • Finding registries and wish lists
• Web and Traffic Cameras
  • Reliable web and traffic camera sites
  • Leveraging these cameras in assessments
• File Metadata Analysis
  • What is metadata?
  • How do we use it?
  • How do we retrieve it?

SEC487.4: Imagery, Networks, Government, and Business

Overview

Day four focuses on many different but related OSINT issues. We begin by looking at how various mapping sites can assist our assessments with aerial data, distance-measuring, and "street view" imagery. Moving beyond using just one vendor's mapping system, students will work with a variety of free, online mapping resources.

We then shift from OSINT about people and locations to OSINT about networks and computers, as researching IP addresses, domain names, and related content can be important aspects of our investigations. Starting with the basics, we get comfortable retrieving information about IP addresses, network blocks, and using the whois protocol. Students then move to making advanced queries to the domain name system (DNS) to grab subdomains and other domain data. To complete our work looking at computers, we examine how we use wireless network data in our work.

The second portion of the day has three modules. The first covers OSINT framework tool suites. These tools can accelerate our OSINT research by very rapidly acquiring data about people, networks, hosts, and more. We examine three frameworks in-depth during class. The next module covers harvesting information from federal, state, and local government web pages within the United States. The public data on these sites can help us research people and businesses. Completing the day, we look at the methods that can be used to gather data about businesses.

CPE/CMU Credits: 6

Topics

• Remote Location Recon
  • Satellite and aerial imagery
  • Ground-based imagery (commercial and consumer)
  • Using mapping tools to measure data and markup maps
  • Using historical ground-based imagery
• IP Address and Whois
  • Basic introduction of computer networking concepts
  • Leveraging the Whois protocol for OSINT
• IP Address Geolocation
  • Accuracy of IP address geolocation
  • How to geolocate from an IP address
• Domain Name System (DNS)
  • Description of DNS
  • DNS data
  • Tools to harvest data from DNS servers
• Wireless Networks
  • Searching wireless network data
• Recon Tool Suites and Frameworks
  • Detailed comparison of three reconnaissance frameworks
    • SpiderFoot
    • Intrigue
    • Recon-ng
• U.S. Government Data
  • Federal resources
  • State and local government resources
• Researching Companies
  • Retrieving basic data about businesses
  • Business profile sites
  • Non-profit and charity organization OSINT
  • Business filings/EDGAR
  • International business OSINT
  • Management analysis
  • Business systems analysis
    • Censys.io
    • Shodan.io

SEC487.5: The Dark Web and International Issues

Overview

The entire morning of day five focuses on understanding and using three of the most popular dark web networks for OSINT purposes. Students will learn why people use Freenet, I2P, and Tor. Each network is discussed at length so that students don't just know how and why to use it, but also gain an understanding of how those networks work. With the Tor network being such a big player in the dark web, the course spends extra time diving into its resources.

The first module in the afternoon examines how blue teamers (cyber defenders) can use monitoring to receive alerts when data of interest appears on the Internet. We then shift our focus to data found on "paste" sites. These websites sometimes contain content such as user names and passwords of compromised user accounts, detailed network information about our target's systems, or just data that our customers need to know.

Considering that a big barrier to using non-English websites can be the language, students learn how to use techniques to translate content and search locally for relevant information in our international OSINT section. We also examine how to discover popular websites and applications used in foreign countries. Since we talk about international data and traveling around the world, our courseware finishes up with an examination of how we track transportation (planes, boats, cars, etc.).

We leave some time at the end of the day for a massive lab, the "Solo CTF," which helps students put together all that they have learned in a semi-guided walk-through that touches on many of the concepts taught throughout the week. Setting aside time to work through our OSINT process in an organized manner reinforces key concepts and allows students to practice executing OSINT process, procedures, and techniques.

CPE/CMU Credits: 6

Topics

• The Surface, Deep, and Dark Webs
  • Levels of the Internet
  • Understanding of what data are at what layer and how to access them
• The Dark Web
  • Risks in using the dark web
  • Overview of top three dark web networks
• Freenet
  • Modes of Freenet
  • Accessing Freenet
  • Services and resources in the Freenet
• I2P - Invisible Internet Project
  • What data are in I2P?
  • I2P tunnels
  • Using I2P
  • Eepsites
• Tor
  • Who uses Tor and why?
  • How Tor works
  • Cautions when using Tor
  • Accessing Tor
  • Tor hidden services
  • Scanning and monitoring Tor hidden services
  • Sharing files in Tor
• Monitoring and Alerting
  • Performing searches in HaveIBeenPwned.com using web and API
  • Setting up alerts to monitor web content
  • Creating web dashboards
  • What do people use paste sites for?
  • Harvesting content from paste sites
  • Google Custom Search Engines
• International Issues
  • Language translation tools
  • Popular websites
  • Popular mobile applications
  • Searching regionally
• Vehicle Searches
  • License plates
  • Vehicle ID Numbers (VINs)
  • Plane registrations
  • Plane tracking
  • Ships and watercraft
  • Putting it all together

SEC487.6: Capstone: Capture (and Present) the Flag

Overview

The capstone for the course is a group event that brings together everything that students learned throughout the week. This is not a "canned" Capture the Flag event where specific flags are planted and your team must find them. It is a competition where each team will collect specific OSINT data about a certain group of people. The output from this work will be turned in as a "deliverable" to the "client" (the instructor), and then the three teams with the most-complete work will present their research to the class for voting.

This multi-hour, hands-on event will reinforce what the students practiced in the Solo CTF the day before and add the complexity of performing OSINT assessments under pressure and in a group.

CPE/CMU Credits: 6

Topics

• Capstone Capture the Flag Event

Who Should Take This Course

We found that the more efficiently you find useful information on the Internet, the more successful you can be in your work. Whether you are trying to find suspects for a legal investigation or candidates to fill a job requisition, gathering hosts for a penetration test or placing honey tokens as a defender, this class will teach you techniques that will aid you in your work.

While far from complete, the Open Source Intelligence (OSINT) topics in SEC487 will be helpful to:

• Cyber Incident Responders
• Digital Forensics (DFIR) analysts
• Penetration Testers
• Social Engineers
• Law Enforcement
• Intelligence Personnel
• Recruiters/Sources
• Private Investigators
• Insurance Investigators
• Human Resources Personnel
• Researchers

What You Will Receive

• A USB storage device with a custom Linux virtual machine containing software ready for you to conduct your own investigations
• A digital wiki (inside the virtual machine) containing electronic versions of the labs, tools, and more
• Video walk-throughs, recorded by the author, for each lab to assist you in your work

This Course Will Prepare You To

• Create an OSINT process
• Conduct OSINT investigations in support of a wide range of customers
• Understand the data collection life cycle
• Create a secure platform for data collection
• Analyze customer collection requirements
• Capture and record data
• Create sock puppet accounts
• Create your own OSINT process
• Harvest web data
• Perform searches for people
• Access social media data
• Assess a remote location using online cameras and maps
• Examine geolocated social media
• Research businesses
• Use government-provided data
• Collect data from the dark web
• Leverage international sites and tools

Hands-On Labs

This is a learn it-do it course where we examine a topic and then dive into a hands-on lab to reinforce the learning. The course has over 22 labs spaced across the first five days, followed by the final hands-on Capture-the-Flag challenge on day six. Check out the lab content below to get a feel for what you will be doing within our class virtual machines.

Day 1

• Set up the course virtual machine and configure the VPN that is used to secure all web traffic
• Use a MindMap tool to document OSINT data and then analyze relationships between people using a data visualization application
• Set up a password manager to securely store all the passwords that we will need for our sock puppets and other accounts
• Create a sock puppet account with realistic user-attributes, which will be key to succeeding in some of the other labs later in the course
• Join a class Slack group to discuss OSINT and the class by way of a lab that walks you through the setup and use of the application

Day 2

• Harvest web data such as Google Analytics IDs and the information within HTTPS certificates
• Trace a home address and phone number to their owners
• Gather email addresses for a company
• Use a reconnaissance framework to rapidly scan websites looking for specific user accounts
• Search reverse images to find the identity of the person and other places that image was used

Day 3

• Execute queries on search engines to find information about someone
• Conduct Facebook queries to retrieve surface and deep data
• Analyze tweets to determine sentiment and discover where the tweets are geolocated
• Scrape metadata and map GPS coordinates

Day 4

• Use online mapping sites to recon an area
• Search for wireless network data and use it to verify an alibi
• Run an OSINT framework to discover what information can be found about a domain
• Examine various government websites to answer trivia questions
• Gather data points about the CEO and the systems used at a business

Day 5

• Dive into the deep web by using Tor to visit Internet sites and hidden services, and set up our own hidden service
• Query the HaveIBeenPwned.com website and API to find compromised user accounts
• Use translation sites to practice translating text into other languages
• Discover the popular websites and mobile apps used in several countries
• Undertake the Solo CTF that brings together many of the previous labs and helps students practice process

Day 6

• Participate in the group Capture the Flag competition