How to use Discovery?

Overview

Discovery is a threat detection system that identifies potential security risks by running automated scripts against selected assets. The system operates on a scheduled basis, processing tasks in a queue-based manner to ensure systematic and organized threat assessment.

Key Concepts

What is Discovery?

Discovery is the process of finding threats and vulnerabilities by executing specialized detection scripts against target assets. Each discovery run analyzes specific assets using predefined flows to identify potential security issues, data exposures, and other threats.

Core Components

• Assets: Target domains, applications, or resources to be analyzed
• Flows: Keywords and configurations that determine which specific scripts will be executed
• Schedule: Automated timing system for running discovery processes
• Queue System: Organized processing of discovery tasks to manage resources efficiently

How Discovery Works

1. Asset Selection

Assets must be selected before running any discovery process. These can include:

• Domain names (e.g., tikaj.com, newindiabank.in)
• Subdomains
• Web applications
• API endpoints

2. Flow Configuration

Flows are related keywords that signify which scripts will be executed during the discovery process. Different flows target different types of threats:

• Code Exposure Detection (github_d020) (Use keyword assets)
• Breach Risk Assessment (hibp_d035) (Use email assets)
• Scam Detection (job_search_d023) (Use keyword assets)
• Malware Detection (mobile_app_d026) (Use keyword assets)
• Counterfeit Detection (product_search_d025) (Use product assets)
• Data Exposure Detection (trawler_d024) (Use keyword assets)

3. Scheduling System

The discovery system operates on a scheduled basis with configurable frequency:

• ONCE: Single execution
• Recurring: Multiple scheduled runs
• Tasks are processed in queue order to prevent resource conflicts

4. Execution Queue

All scheduled discoveries are processed in a queue manner, ensuring:

• Systematic execution of tasks
• Resource management
• No overlapping processes
• Orderly completion of discovery runs

Status Indicators

Each discovery card displays a colored dot indicating the current status:

Discovery Cards Information

Each discovery card displays the following information:

• Discovery ID: Unique identifier (e.g., SCH-DISC-75E1-75C4-4113)
• Asset Type: Category of the target (e.g., subdomain)
• Status Badge: Current execution status
• Discoveries Count: Number of threats/issues found
• Detections Count: Number of potential detections identified
• Last Run: Timestamp of the most recent execution
• Assets List: Specific assets being monitored

Setting Up a Discovery

Step 1: Select Targets

1. Click "Select targets..." to choose your assets
2. Add domain names, subdomains, or other resources
3. Confirm your asset selection

Step 2: Configure Schedule

1. Set the Schedule Frequency:
   • Choose "ONCE" for single execution
   • Select recurring options for ongoing monitoring
2. Configure Flows related to:
   • Select relevant keywords/flows for your use case
   • Multiple flows can be selected for comprehensive coverage

Step 3: Execute

1. Click "Schedule" to add the discovery to the queue
2. Monitor the status dot for execution progress
3. Review results once the discovery completes

Viewing Discovery Details

Accessing Discovery Results

Once a scheduled discovery runs, you can view detailed information by:

1. Click on Discovery Card: Click on any discovery card to open the detailed view
2. Check Execution Status: View the current execution status and history
3. Review All Discoveries: Access the "All Discoveries" section to see comprehensive execution logs

Discovery Detail Views

All Discoveries Section

• Lists all discovery executions with detailed information
• Shows Discovery ID, Execution Info, and Created/Updated timestamps
• Provides filtering and pagination for easy navigation

Execution Details

• Success Cases: View the complete script response and results
• Error Cases: Display detailed error messages and troubleshooting information
• Execution History: Track all previous runs and their outcomes

Error Tracking and Debugging

When a discovery encounters an error:

• Error messages are prominently displayed in the discovery details
• Detailed error information helps identify root causes
• Error tracking allows for systematic troubleshooting
• Historical error logs help identify patterns and recurring issues

Managing Detection Results

Detection Categories

The system organizes detections into three main categories:

1. Potential Detections: Newly discovered threats awaiting review
2. Created Detections: Validated threats added to the system
3. Invalid Detections: False positives or irrelevant findings

Potential Detections Table

The Potential Detections table displays:

• Title: Description of the detected threat
• Identifier: Unique detection identifier with clickable links
• Target: The asset where the threat was found
• Category: Type of threat (e.g., subdomain, vulnerability)
• Actions: Add/Remove buttons for detection management

Detection Management Actions

Adding Valid Detections

1. Review threats in the "Potential Detections" table
2. Click the "+ Add" button next to valid threats
3. Detection moves to "Created Detections" and becomes part of the system
4. Valid detections are now tracked and monitored

Removing Invalid Detections

1. Identify false positives in the "Potential Detections" table
2. Click the "Remove" button next to invalid findings
3. Detection is marked as invalid and moved to "Invalid Detections" table
4. Helps improve detection accuracy over time

Detection Workflow

Potential Detection → Review → Decision
                              ↙        ↘
                    Add (Valid)      Remove (Invalid)
                         ↓                   ↓
               Created Detections    Invalid Detections

Best Practices

Asset Management

• Use valid key words
• Ensure assets are properly formatted (e.g., domain.com)
• Group related assets for efficient processing
• Regularly update asset lists as your infrastructure changes

Flow Selection

• Choose flows that match your security concerns
• Consider running multiple flows for comprehensive coverage
• Review flow documentation to understand detection capabilities

Scheduling Strategy

• Use "ONCE" for initial assessments
• Implement recurring schedules for ongoing monitoring
• Balance frequency with resource availability

Monitoring Results

• Regularly check discovery status
• Inform any error states immediately
• Review detection results and take appropriate action
• Actively manage potential detections by adding valid ones and removing un-related/invalid detections.

Integration

The discovery system integrates with various providers and scripts located in:

• providers: Detection scripts
• rules: YAML configuration files for detection logic
• flow: JavaScript flow definitions for different threat types

This modular approach allows for easy expansion and customization of discovery capabilities based on evolving security needs.