Omni Search
Search Types
This document outlines the types of search inputs supported and how they are identified based on user queries.
These search types would be auto detected with this system.
- First Letter of Each Word (Gurmukhi)
- Full Word (Gurmukhi)
- Full Word Translation (English)
- Romanized First Letter (English)
- Ask a Question (English)
- Ang (Numeric)
These search types won't be auto detected yet, we can include them in future scope.
- Romanized Gurmukhi (English)
- Main Letters (Gurmukhi)
The goal is to support accurate classification of user input for efficient search behavior.
Detection Rules:
- Query has no spaces.
- Query has no numbers
- Query length is mostly between 4 and 12 characters (flexible but ideal).
- It can have any of the 35 characters from ੳ-ੜ as per the following key map.
- i.e. it can only have one of the following characters
aAeshkKgG|cCjJ\tTfFxqQdDnpPbBmXrlvS^Zz&LV
Example: ਸਚਨਕ
Detection Rules:
- Query contains Gurmukhi words.
- May contain spaces.
- Words resemble valid Punjabi vocabulary (if lexicon is available).
- Does not match Numeric or question patterns.
Example: ਸਚ ਪਾਉ
Detection Rules:
- Query contains only English characters (A-Z, a-z) and spaces.
- Words are valid English (based on dictionary or NLP confidence).
- Does not form a question (i.e., no interrogative format).
- Does not contain numbers.
Example: true name
- Query has no spaces.
- Query has no numbers
- Query length is mostly between 4 and 12 characters (flexible but ideal).
- It can have any of the 26 characters from a-z.
- It doesn't matter if the query character is uppercase of lowercase.
Example: jtmvh
Detection Rules:
- Query is in English.
- Begins with or contains interrogative keywords:
who,what,when,where,why,how, etc. - May or may not end with a
?. - May include pronouns, verbs indicating question formation.
- Does not match any other format like numeric (Ang) or specific word searches.
Examples: What is meaning of life how to wake up early I need peace, how to get it?
Detection Rules:
- Query is only numeric (e.g., regex:
^\d+$). - No spaces, punctuation, or alphabetic characters.
- Typically falls within a known valid Ang range (e.g., 1–1430).
Example: 212
This is the detection logic to classify user input into a supported search type and then perform a search using Meilisearch.
- User selects the language (Gurmukhi or English) and enters a search query.
- Check if the input is numeric (
^\d+$)—if yes, classify as Ang.
- Use full-text search on the following fields:
- Gurmukhi (for full word Gurmukhi search)
- Translations (for full word English search)
- FirstLetterStr (for first letter Gurmukhi search)
- MainLetters (for main letter Gurmukhi search)
- FirstLetterEng (for first letter English search)
- Meilisearch automatically ranks results by relevance based on the query.
- Scoring is based on proximity, typo tolerance, and exactness.
- You can also customize the ranking rules based on your requirements (e.g., boosting Gurmukhi matches).
const searchResults = await index.search(searchQuery, {
attributesToRetrieve: ['ID', 'Gurmukhi', 'Translations', 'FirstLetterStr', 'MainLetters', 'FirstLetterEng'],
});
console.log(searchResults.hits); // Display results ranked by relevanceYou need to install and run Meilisearch locally or use a hosted version. Here’s how to set up Meilisearch with Docker:
docker run -d -p 7700:7700 -e MEILI_MASTER_KEY='your_master_key' getmeili/meilisearchCreate an index for storing verses in Meilisearch:
const { Client } = require('meilisearch');
const client = new Client({
host: 'http://localhost:7700', // URL for your Meilisearch instance
apiKey: 'your_master_key',
});
// Create index for verses
const index = await client.createIndex('verses', {
primaryKey: 'ID',
});Set up the fields for searching in Meilisearch:
await index.updateSettings({
searchableAttributes: [
'Gurmukhi',
'Translations',
'FirstLetterStr',
'MainLetters',
'FirstLetterEng',
],
rankingRules: [
'typo', // Typo tolerance
'words', // Exact match
'proximity', // Proximity to search term
'attribute', // Attribute-based ranking
'exactness' // Exactness in matching the query
],
});Fetch your verses from the database and add them to Meilisearch:
const documents = [
{
ID: 1,
Gurmukhi: 'ਸਤਿ ਨਾਮੁ',
Translations: 'True Name',
FirstLetterStr: 'ਸ',
MainLetters: 'ਸਤ..',
FirstLetterEng: 'T..',
// Additional columns...
},
// More verses...
];
await index.addDocuments(documents);How to create the Meilisearch index from the BaniDB Verses?
This section outlines how to create a script to add data from BaniDB database to a Meilisearch index.
First, ensure we have a running Meilisearch instance.
docker run -d -p 7700:7700 -e MEILI_MASTER_KEY='master_key' getmeili/meilisearchIn your Node.js application, initialize the Meilisearch client:
const { Client } = require('meilisearch');
// Initialize the Meilisearch client
const client = new Client({
host: 'http://localhost:7700', // URL for your Meilisearch instance
apiKey: 'your_master_key', // Optional, if you set a master key
});
// Create or get an index for verses
const index = client.index('verses'); // Index name: 'verses'Next, fetch the verses (and related fields like Gurmukhi, Translations, etc.) from database.
const mysql = require('mysql2');
// Database connection
const pool = mysql.createPool({
host: 'localhost',
user: 'root',
password: 'password',
database: 'your_database',
});
async function fetchVerses() {
return new Promise((resolve, reject) => {
pool.query('SELECT ID, Gurmukhi, Translations, FirstLetterStr, MainLetters, FirstLetterEng FROM Verse', (err, results) => {
if (err) reject(err);
resolve(results); // Returns all verses
});
});
}This function retrieves all the relevant data from the database.
Once the data is fetched from the database, format it for Meilisearch. Each verse will be added as a document with the required fields.
async function prepareDataForIndex() {
const verses = await fetchVerses();
const documents = verses.map(verse => ({
ID: verse.ID, // Unique ID
Gurmukhi: verse.Gurmukhi, // Gurmukhi text
Translations: verse.Translations, // English translation
FirstLetterStr: verse.FirstLetterStr, // First letter (Gurmukhi)
MainLetters: verse.MainLetters, // Main letters (Gurmukhi)
FirstLetterEng: verse.FirstLetterEng, // First letter (English)
}));
return documents;
}Now that the data is formatted correctly, add it to Meilisearch. Use the addDocuments method to index the documents.
async function addDocumentsToMeili() {
const documents = await prepareDataForIndex();
// Add the documents to Meilisearch
try {
const result = await index.addDocuments(documents);
console.log('Documents added:', result);
} catch (error) {
console.error('Error adding documents to Meilisearch:', error);
}
}This function will send the documents to Meilisearch for indexing.
To execute the script, simply call the addDocumentsToMeili function:
addDocumentsToMeili();This will index all the verses from the database into Meilisearch.
To keep the Meilisearch index up to date, we can periodically update or delete documents in Meilisearch based on changes to the database:
-
Updating documents: If a verse is updated, we can use
updateDocuments:
await index.updateDocuments([updatedVerse]);-
Deleting documents: If a verse is deleted, we can use
deleteDocuments:
await index.deleteDocuments([verseId]);This process can be automated with a scheduled job (e.g., using cron jobs) to sync the database with Meilisearch periodically.
Once the data is indexed, we can start querying it using Meilisearch:
const searchResults = await index.search('True Name', {
attributesToRetrieve: ['ID', 'Gurmukhi', 'Translations', 'FirstLetterStr', 'MainLetters', 'FirstLetterEng'],
});