Base Search

The BaseSearch class provides an abstract foundation for implementing search functionality across various sources. It is designed to be flexible and extensible, allowing developers to build their own search mechanisms by subclassing and extending its methods. Subclasses can extend this class to implement specific search behaviors and capabilities.\n

Reusability: Provide a common interface for multiple search implementations.
Flexibility: Allow developers to customize source initialization and search logic.
Scalability: Efficiently handle concurrent operations using asynchronous generators.

Use Cases

The BaseSearch class is designed for applications requiring a unified approach to search across diverse data sources,

Fetching data from the multiple database or api.
Searching through documents, articles and media.\n

Methods

Constructors

The constructor initializes the class with default sources.
Subclasses can override this constructor to customize the sources according to their needs.

Search Method

Purpose: An asynchronous generator method that retrieves search results from sources.
Usage: Subclasses must override this method to implement the logic for fetching search results.

Example:

class MySearch extends BaseSearch<SearchCard> {
  async *search(request: SearchRequest): AsyncGenerator<SearchCard, void, unknown> {
    for (let source of this.sources) {
      const result = await fetchFromSource(source, request);
      yield result;
    }
  }
}

Why Generators?
- Asynchronous generators allow results to be streamed progressively as they are retrieved.
- Improves responsiveness, especially for scenarios where results from some sources may be delayed.

getCapabilities

Purpose: An abstract method that subclasses must implement to return the capabilities of their search source.

Example:

getCapabilities(): Capability {
  return {
      whois: true,
      technologies: true,
      txt: true,
      carbon: true
  };
}

RacePromises

Purpose: Execute multiple asynchronous tasks concurrently and yield results as each task completes.
How It Works:
- All tasks are initiated simultaneously.
- The generator yields results as they are resolved.
- Once a task is resolved, it is removed from the active list.
Why This Approach?
- Reduces overall latency by resolving tasks concurrently.
- Results are processed as they become available rather than waiting for all tasks to finish.

Example:

async *racePromises() {
  let activeTasks = tasks.map((task, index) => taskProcessor(task, index));
    while (activeTasks.length > 0) {
      const raceResult = await Promise.race(
        activeTasks.map((task, index) =>
          task.then((result) => ({ result, index })),
        ),
      );
      if (raceResult.result) {
        yield raceResult.result;
      }
      activeTasks.splice(raceResult.index, 1);
    }
}

How to use BaseSearch

export class IntelSearch extends BaseSearch<SearchCard> {
  intel: Intel;

  constructor() {
    super();
    this.intel = new Intel();
    // <============================ Mention your sources here ====================================>
  }

  getCapabilities(): Capability {
    return {
      takeScreenshot: true,
      // <========================== Mention your Capabilities ====================================>
    };
  }

  public async *search(
    request: SearchRequest,
  ): AsyncGenerator<SearchCard, void, unknown> {
    if (request.parsed) {
      yield* this.racePromises(request.parsed.and, (task, index) =>
        this.processTask(task, index),
      );
    }
  }

  private async processTask(task, index: number): Promise<SearchCard | null> {
    // <=========================== Logic for how to obtain your result =============================> 
    if (data) {
      return {
        cardType: cardType,
        title: "Screenshots",
        facts: {
          factId: uuidv4(),
          data: {
            json: data,
            expanded: true,
          }, 
        },
      };
    }
    return null;
  }
}