API Features and Endpoints

[tlmn]

The API runs on an AW server returns

• a list of active investigations and
• plugin configuration for a single active investigation.

It receives and processes

• scraped data sent by a plugin after completing an investigation.

Investigations are fed manually into the API as well as data export from the API is not provided by an endpoint.

Short vocabulary list for common understanding:

• investigation: an investigation is a scraping job that is executed by the plugin. The job is like to contain multiple steps that eg introduce the investigation and ask the user to input form data or perform other tasks.

The following endpoints are required:

• GET listInvestigations: returns an array of active investigations.

  • No parameters required.
  • The following attributes are returned per investigation:
    • investigationID (integer): ID of the investigation, unique among investigations
    • name (string): name of the investigation
    • owner (string): owner of the investigation eg "AW" or third parry owner like "ADFC"
    • description (string): short description of the investigation
    • startTimestamp (timestamp): timestamp when the investigation begins
    • endTimestamp (timestamp): timestamp when the investigation ends
    • startURL (string, optional): regex pattern that matches the URL of the investigated site and triggers the plugin to start
    • stats (object, optional): to be defined
    • consentRequired (string, allowed values anonymous, optIn, optOut): type of investigation defining what meta data is transmitted to the API when sending scraped data
    • type (string, allowed values scheduled, userDriven): determining if investigation should be scheduled for execution or only run manually

• GET getInvestigation: returns information (meta data of the investigation and configuration data) concerning a specific investigation.

  • The endpoint receives the following parameters:
    • investigationID (integer): ID of the investigation, unique among investigations
  • The following attributes are returned:
    • investigationID (integer): ID of the investigation, unique among investigations
    • name (string): name of the investigation
    • owner (string): owner of the investigation eg "AW" or third parry owner like "ADFC"
    • description (string): short description of the investigation
    • startTimestamp (timestamp): timestamp when the investigation begins
    • endTimestamp (timestamp): timestamp when the investigation ends
    • startURL (string, optional): regex pattern that matches the URL of the investigated site and triggers the plugin to start
    • stats (object, optional): to be defined
    • consentRequired (string, allowed values anonymous, optIn, optOut): type of investigation defining what meta data is transmitted to the API when sending scraped data
    • type (string, allowed values scheduled, userDriven): determining if investigation should be scheduled for execution or only run manually
    • steps (object): object containing configuration data for the specific investigation (eg steps and / or intervals for scheduled investigations)

• PUT scrapedData: receives scraped data from an investigation

  • No parameters required.
  • The following attributes are accepted:
    • investigationID (integer): ID of the investigation, unique among investigations
    • scrapedData (object): Object of extracted scraped data
    • clientID (string, optional): unique ID of the plugin instance, sent if investigation type is optIn or optOut and user chose to send clientID

The following features are required apart those defined by the endpoints:

• add investigations via a form builder, delete/update investigation
• manually export sent in data for an investigation
• rate limiting for PUT endpoint