RQL

RQL (Ruhis Query Language) is a powerful library designed to simplify the process of filtering, sorting, and aggregating large amounts of data. With RQL, you can effortlessly extract valuable insights from complex datasets, making data analysis and manipulation tasks more efficient. RQL was initially developed for an internal SIEM project, so it is well suited for security-related use cases, but it can be used for any type of data.

---

Documentation | npm

---

Key Features

• Simple and intuitive syntax - RQL is designed to be easy to learn and use. The syntax is similar to KQL or XQL, but with a few key differences that make it more intuitive and powerful.
• Light, type-safe and developer friendly - RQL is written in TypeScript and compiled to JavaScript. It is very lightweight and has a great documentation, making it easy to integrate into any project.
• Thoroughly tested - RQL has a comprehensive test suite with a very code coverage, ensuring that it works as expected in all scenarios.

Quick Start Guide

1. Install via your preferred package manager:

   • npm install @ruhisfi/rql
   • yarn add @ruhisfi/rql

2. Import QueryParser and QueryExecutor to your code:

   import { QueryParser, QueryExecutor } from "@ruhisfi/rql";

3. Parse query and execute it against a dataset:

   const query =
     'dataset = example_data | filter name = "John" or country = "Finland" | fields name, country, city, email, age | sort age desc | limit 10';
   const parsedQuery = QueryParser.parseQuery(query); // This will validate the query and convert it into a JS object
   const result = QueryExecutor.executeQuery(parsedQuery, data); // This will execute the query against the dataset

Syntax Guide

The query consists of multiple statements separated by the pipe (|) character. The statements are case-sensitive, and must be written in lowercase. The query lines can be commented out with #. The statements are executed in the order they are written in the query.

Operators

The following operators are supported in RQL:

Operator	Description
=, !=	Equal, Not equal
>, <	Greater than, Less than
>=, <=	Greater than or equal, Less than or equal
and	Boolean AND
or	Boolean OR
contains	Returns true if the specified value is contained in string or array
not contains	Returns true if the specified value is not contained in string or array
matches, ~=	Returns true if the regex pattern matches
incidr	Returns true if the IP address is in the CIDR range
not incidr	Returns true if the IP address is not in the CIDR range
in	Returns true if the value is in the specified list
not in	Returns true if the value is not in the specified list

Statements

alter

Syntax

alter <name> = <function>

Description

The alter statement is used to create new or overwrite existing fields in the dataset using a value functions like addition, subtraction, letter casing, etc. The alter statement can be used multiple times in a query and the fields created by it can be used in other statements.

Functions

Supported functions can be found in the documentation: RQL Docs - Functions

Examples

dataset = products
| filter ean = "6410405082657"
| alter price = multiply(cost, 1.2)
| fields ean, name, cost

comp

Syntax

comp <function> <field> [as <returnField>] [by <groupingField>][, <function> <field> [as <returnField>], ...]

Description

The comp statement is used to calculate statistics for results. This function will override other returned records. If used multiple times, the statistics will be merged on one row.

Functions

Function	Description
avg	Returns the average value of the field
count	Returns the number of records where field is not null
count_distinct	Returns the number of distinct values where field is not null
earliest	Returns the earliest timestamp
first	Returns the first value
last	Returns the last value
latest	Returns the latest timestamp
max	Returns the maximum value
median	Returns the median value
min	Returns the minimum value
sum	Returns the sum of values
to_array	Returns an array of values

Examples

// Returns the total number of users, the number of distinct users and the first login time in the USA
dataset = logins
| filter country = "USA"
| comp count username as totalUsers, count_distinct username as distinctUsers, earliest _time as firstLogin

// Returns the amount of logins per country
dataset = logins
| comp count correlationId as logins by country

config

Syntax

config <option> = <value>

Description

The config statement is used to set various options for the query execution. RQL comes with built-in options, but you can also add your own custom options for your application.

Options

Option	Description	Default
case_sensitive	Determine whether values are evaluated as case sensitive in filter statements	true
grouping	Deprecated: Group results by a field in comp statement	''

Examples

dataset = users
| config case_sensitive = false
| filter name contains "john"

dataset

Syntax

dataset = <string>

Description

The dataset statement sets the context for the query by specifying the dataset to be processed. This statement is not processed by RQL itself but is intended for use in your application to allow differentiation between multiple datasets. This can be especially handy if your application deals with multiple data sources or tables, and you want to apply RQL operations to a specific one.

Examples

dataset = transaction_logs | filter transactionID = "TX1001"

dedup

Syntax

dedup <field1>[,<field2>, ...] by asc | desc <field>

Description

The dedup statement is used to remove duplicate records based on field(s). By default it returns the first record, but you can specify the direction of the dedup using the asc (ascending) or desc (descending) keywords and some other field, such as timestamp to return chronologically latest record.

Examples

# Returns all the latest unique username + deviceName sign-in combinations
dataset = signInLogs
| filter location.country = "GB"
| dedup username, deviceName by _time desc

fields

Syntax

fields <field1> [as <alias1>], <field2> [as <alias2>], ...

Description

The fields statement enables you to cherry-pick the fields you're interested in from your dataset. This becomes useful when dealing with data structures having multiple fields, and you want to limit the output to only a few specific ones. If you don't specify any fields, all fields will be returned.

You can optionally rename the fields in the output using the as keyword, providing an alias for the original field name.

Examples

dataset = customer_records
| filter customerID = "CUST1001"
| fields firstName as Name, emailID as Email

filter

Syntax

filter <field> = <value> [and|or] <field> = <value> ...

Description

The filter statement is used to limit the dataset to records that match the specified criteria. You can compare fields to values using logical operators, and you can combine multiple criteria using the and and or keywords. For a list of supported operators, see the Operators section.

Examples

dataset = users
| filter age > 18 and email not contains "@gmail.com"
| filter country = "Canada" or country = "Spain"
| filter role in ("admin", "manager")
| fields name, age, country, email

limit

Syntax

limit <number>

Description

The limit statement is used to limit the number of records returned in the result. This is useful for paging or returning a top N list.

Examples

dataset = logins
| filter country = "USA"
| sort username desc
| limit 10
| fields country, username

search

Syntax

search <query>

Description

The search statement is used to limit the dataset to records that match the specified query. This is useful for full-text search or searching for specific patterns in the data. Compared to the filter statement, the search statement searches all fields in the dataset.

Examples

// find all users with "john" in their name or email
dataset = users
| fields name, email
| search "john"

sort

Syntax

sort <field> [asc|desc], <field> [asc|desc] ...

Description

The sort statement is used to order the results by one or more fields. You can specify the direction of the sort using the asc (ascending) or desc (descending) keywords. If no direction is specified, the data will not be sorted.

Examples

dataset = users
| filter age > 18
| sort age desc, name asc
| fields name, age

Changelog

4.1.1 (2025-10-14)

• Fixed bug with invalid return field names in comp statement

4.1.0 (2025-10-14)

• Deprecated grouping option in config statement
• Added by [field] argument to comp statement to allow grouping by a field

4.0.1 (2025-06-08)

• Fixed bug with filter statement treating field names ending with in or some other operators as list filter

4.0.0 (2025-06-02)

• Removed ElasticSearch integration
  • The ElasticSearch integration was removed due to the complexity and performance issues it introduced. The library is now focused on providing a lightweight and efficient query language for data manipulation without the overhead of ElasticSearch.
• Added functions:
  • ago()
  • fnv1a()
  • future()
  • geo_in_polygon()
  • md5()
  • now()
  • sha1()
  • sha256()
• Renamed distance() to geo_distance()
• Fixed bug with matches filter not handling regex flags properly

3.3.0 (2025-05-17)

• Unified functions in alter and filter statements
  • Both statements now support the same functions to modify fields
• Added distance() function
• Fixed bug with search statement not working correctly with non-string values

3.2.1 (2025-04-11)

• Fixed bug with dedup field parsing

3.2.0 (2025-03-10)

• Added support for in and not in operators in filter statement

3.1.8 (2025-01-17)

• Fixed small bug in ElasticSearch query execution

3.1.7 (2025-01-06)

• Optimized ElasticSearch query execution
  • Some dedup queries now use inner_hits to get the latest record, which improves performance especially in large datasets

3.1.6 (2025-01-06)

• Removed debug logging from ElasticSearch query execution

3.1.5 (2025-01-06)

• Optimized ElasticSearch query execution
  • Some filters are now converted to ElasticSearch compatible filters, which should improve performance especially in large datasets

3.1.4 (2024-11-30)

• Added to_number function to alter statement
• Improved test coverage

3.1.3 (2024-05-07)

• Fixed bug in not equals operator in filter statement

3.1.2 (2024-04-27)

• Fixed bug with inconsistent UUID filtering in filter statement

3.1.1 (2024-04-26)

• Fixed bug with inconsistent date types (string vs Date) in filter statement
  • Now all date values are converted to Date objects for consistency

3.1.0 (2024-04-26)

• Added support for relative date filtering in filter statement
  • Supported units: s (seconds), m (minutes), h (hours), d (days)
  • Example: filter date > -1h
• Added filter value alias now() for current date and time
  • Example: filter date > now()
• Improved date filtering consistency

3.0.0 (2024-04-21)

• Breaking change: Reworked query parsing and execution logic
  • Queries are now parsed in the order they are written in the query string, instead of being grouped by statement type
  • This might break existing queries that rely on the old fixed order of statements
  • This change makes the query execution more predictable and easier to understand and also allows chaining of statements in a more flexible way
  • Chaining multiple comp functions must be done on one statement seperated by commas instead of multiple comp statements
• Removed old deprecated LegacyQueryExecutor and LegacyQueryParser classes

2.0.1 (2024-04-12)

• Hotfix: Fixed exports of QueryParser and QueryExecutor

2.0.0 (2024-04-12)

• Added better functionality for search statement
• Added grouping option for comp statement via config statement
• Refactored codebase to improve maintainability and readability
• Deprecated old query execution and parsing logic and moved them to LegacyQueryExecutor and LegacyQueryParser

1.8.0 (2024-04-07)

• Added support for search statement

1.7.0 (2024-02-24)

• Added support for config statement
• Added case-sensitive option to filter statement via config statement
• Added support for single quotes in filter statement

1.6.3 (2024-02-10)

• Hotfix for ElasticSearch scroll API not working correctly

1.6.2 (2024-02-10)

• Changed ElasticSearch to use scroll API instead of search_after
• Re-enabled sorting for ElasticSearch queries

1.6.1 (2024-02-10)

• Disabled sorting for ElasticSearch queries to fix issue with Elastic pre-document sorting

1.6.0 (2024-02-10)

• Changed ElasticSearch to use search_after instead of hard coded body size

1.5.7 (2023-12-29)

• Changed size to 10k for ElasticSearch search

1.5.6 (2023-12-29)

• Removed ElasticSearch query limit, using default value instead

1.5.5 (2023-12-29)

• Added ElasticSearch sorting to prevent missing results in larger datasets
• Changed ElasticSearch body size from 10k to 20k

1.5.4 (2023-12-02)

• Added to_string function to alter statement
• Added to_date function to alter statement
• Added get function to alter statement
• Added get_array function to alter statement
• Added base64_encode and base64_decode functions to alter statement
• Added round, ceil and floor functions to alter statement
• Added extract_url_host function to alter statement
• Added json_parse and json_stringify function to alter statement
• Changed execution order of alter statement to be executed before fields statement
• Added support for dynamic fields in substring function
• Added support for line comments starting with //

1.5.3 (2023-12-02)

• Added to_array function to comp statement
• Added trim function to alter statement
• Added split function to alter statement
• Added length function to alter statement

1.5.2 (2023-12-02)

• Fixed a bug where dedup would not work correctly if the field was not present in the dataset
• Improved documentation

1.5.1 (2023-11-30)

• Added support for null values in filter statement

1.5.0 (2023-11-22)

• Added support for comp statement
  • Added support for avg, count, count_distinct, earliest, first, last, latest, max, median, min, sum functions

1.4.0 (2023-11-22)

• Added support for dedup statement
• Added support for line comments (starting a line with #)
• Added nested field support for alter statement
• Arithmetic alter functions now handle invalid fields as 0 instead of NaN
• Added coalesce function to alter statement
• Added incidr function to alter statement

1.3.2 (2023-11-21)

• Fixed a bug where nested field in filter statement was not working correctly if the field didn't exist

1.3.1 (2023-11-20)

• Fixed a bug where OR operator was not working correctly in filter statement
• Added QueryParsingOptions to QueryParser
• Added option to disable dataset requirement via strictDataset option

1.3.0 (2023-11-03)

• Added support for <= and >= operators
• Added incidr and not incidr operators
• Added alias ~= for matches operator
• Fixed a bug where query couldn't contain multiple filter statements
• Improved test coverage for QueryExecutor
• Cleaned test code
• Updated dependencies

License

This project is licensed under the GNU GPLv3 License - see the LICENSE file for details.