Skip to content

Commit 70c809e

Browse files
committed
Fix migration script error handling and improve documentation
- Add proper error handling for JSON marshaling in bulk update processing - Fix scroll timeout format to use duration string instead of minutes - Add links to PR #20 and query service repository in documentation - Simplify usage examples to use go run directly instead of building - Remove redundant custom configuration section from README 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]> Signed-off-by: Andres Tobon <[email protected]>
1 parent 87d6932 commit 70c809e

File tree

2 files changed

+13
-23
lines changed

2 files changed

+13
-23
lines changed

scripts/migration/001_add_access_query_fields/README.md

Lines changed: 6 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -1,39 +1,26 @@
11
# Access Query Fields Migration Script
22

3-
This script migrates existing OpenSearch documents to add the new `access_check_query` and `history_check_query` fields that were recently added to the enricher system.
3+
This script migrates existing OpenSearch documents to add the new `access_check_query` and `history_check_query` fields that were recently added as part of [PR #20](https://github.com/linuxfoundation/lfx-v2-indexer-service/pull/20).
44

55
## Background
66

7-
The LFX indexer service was recently updated to include new query fields for Fine-Grained Authorization (FGA):
7+
The LFX indexer service was recently updated to include new document fields for Fine-Grained Authorization (FGA) of documents via the [query service](https://github.com/linuxfoundation/lfx-v2-query-service):
88

99
- `access_check_query`: Combination of `access_check_object` + "#" + `access_check_relation`
1010
- `history_check_query`: Combination of `history_check_object` + "#" + `history_check_relation`
1111

12-
These fields are automatically populated for newly indexed documents, but existing documents need to be migrated.
12+
These fields are automatically populated for newly indexed documents (implemented in [PR #20](https://github.com/linuxfoundation/lfx-v2-indexer-service/pull/20)), but existing documents need to be migrated.
1313

1414
## Usage
1515

1616
### Basic Usage
1717

1818
```bash
19-
# Build the migration script
20-
go build -o migrate main.go
21-
2219
# Run in dry-run mode to see what would be changed
23-
DRY_RUN=true ./migrate
20+
DRY_RUN=true go run scripts/migration/001_add_access_query_fields/main.go
2421

2522
# Run the actual migration
26-
./migrate
27-
```
28-
29-
### With Custom Configuration
30-
31-
```bash
32-
# Custom OpenSearch settings
33-
OPENSEARCH_URL=http://opensearch:9200 \
34-
OPENSEARCH_INDEX=resources \
35-
BATCH_SIZE=500 \
36-
./migrate
23+
go run scripts/migration/001_add_access_query_fields/main.go
3724
```
3825

3926
## Environment Variables
@@ -71,7 +58,7 @@ The script:
7158

7259
## Example Output
7360

74-
```
61+
```text
7562
Starting access query fields migration...
7663
=== Migration Configuration ===
7764
OpenSearch URL: http://opensearch:9200

scripts/migration/001_add_access_query_fields/main.go

Lines changed: 7 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,7 @@
1010
//
1111
// Usage:
1212
//
13-
// go run scripts/data-migration/add-access-query-fields/main.go
13+
// go run scripts/migration/001_add_access_query_fields/main.go
1414
//
1515
// Environment variables:
1616
// - OPENSEARCH_URL: OpenSearch cluster URL (default: http://localhost:9200)
@@ -116,7 +116,6 @@ func loadConfig() *Config {
116116
return config
117117
}
118118

119-
120119
// createOpenSearchClient creates and configures OpenSearch client
121120
func createOpenSearchClient(config *Config) (*opensearch.Client, error) {
122121
cfg := opensearch.Config{
@@ -247,7 +246,7 @@ func searchDocuments(ctx context.Context, client *opensearch.Client, config *Con
247246
func scrollDocuments(ctx context.Context, client *opensearch.Client, scrollID string, scrollTimeout time.Duration) (*SearchResponse, error) {
248247
scrollBody := map[string]interface{}{
249248
"scroll_id": scrollID,
250-
"scroll": fmt.Sprintf("%dm", int(scrollTimeout.Minutes())),
249+
"scroll": scrollTimeout.String(),
251250
}
252251

253252
scrollBodyJSON, err := json.Marshal(scrollBody)
@@ -328,7 +327,11 @@ func processBatch(ctx context.Context, client *opensearch.Client, config *Config
328327
bulkBody.WriteString(fmt.Sprintf(`{"update":{"_index":"%s","_id":"%s"}}`, config.IndexName, doc.ID))
329328
bulkBody.WriteString("\n")
330329

331-
updateJSON, _ := json.Marshal(map[string]interface{}{"doc": updateDoc})
330+
updateJSON, err := json.Marshal(map[string]interface{}{"doc": updateDoc})
331+
if err != nil {
332+
stats.ErroredDocuments++
333+
return fmt.Errorf("failed to marshal update document for %s: %w", doc.ID, err)
334+
}
332335
bulkBody.Write(updateJSON)
333336
bulkBody.WriteString("\n")
334337

0 commit comments

Comments
 (0)