You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/stream_maps.md
+156-1Lines changed: 156 additions & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -292,13 +292,168 @@ records being generated.
292
292
293
293
The following logic is applied in determining the SCHEMA of the transformed stream:
294
294
295
-
1. Calculations which begin with the text `str(`, `float(`, `int(` will be
295
+
1. Calculations which begin with the text `str(`, `float(`, `int(`, or `bool(` will be
296
296
assumed to be belonging to the specified type.
297
297
1. Otherwise, if the property already existed in the original stream, it will be assumed
298
298
to have the same data type as the original stream.
299
299
1. Otherwise, if no type is detected using the above rules, any new stream properties will
300
300
be assumed to be of type `str` .
301
301
302
+
#### Type Casting
303
+
304
+
Stream maps support explicit type casting to convert values to different data types. By wrapping expressions with type casting functions, you can ensure proper schema detection and data type conversion.
305
+
306
+
##### Supported Type Casting Functions
307
+
308
+
The following type casting functions are available in stream mapping expressions:
309
+
310
+
| Function | Target Type | JSON Schema Type | Description |
# Explicitly cast to string (useful for schema detection)
378
+
description: str('[masked]')
379
+
```
380
+
````
381
+
382
+
````{tab} JSON
383
+
```json
384
+
{
385
+
"stream_maps": {
386
+
"repositories": {
387
+
"description": "str('[masked]')"
388
+
}
389
+
}
390
+
}
391
+
```
392
+
````
393
+
394
+
**Converting values to booleans:**
395
+
396
+
````{tab} meltano.yml
397
+
```yaml
398
+
stream_maps:
399
+
mystream:
400
+
# Convert to boolean with null handling
401
+
is_active: bool(status_value) if status_value else None
402
+
```
403
+
````
404
+
405
+
````{tab} JSON
406
+
```json
407
+
{
408
+
"stream_maps": {
409
+
"mystream": {
410
+
"is_active": "bool(status_value) if status_value else None"
411
+
}
412
+
}
413
+
}
414
+
```
415
+
````
416
+
417
+
##### When to Use Type Casting
418
+
419
+
Type casting is particularly useful in the following scenarios:
420
+
421
+
1.**Schema Detection Hints**: When creating new calculated fields, wrap the expression in a type casting function to ensure the SDK correctly detects the output type in the schema.
422
+
423
+
1.**Type Conversion**: When you need to convert a value from one type to another (e.g., converting a numeric string to an actual integer).
424
+
425
+
1.**Consistent Data Types**: When working with data that may have inconsistent types in the source, you can enforce a specific type in the output.
426
+
427
+
**Example combining type casting with conditional logic:**
428
+
429
+
````{tab} meltano.yml
430
+
```yaml
431
+
stream_maps:
432
+
nested_jellybean:
433
+
# Extract custom field value and cast to integer, handling null values
434
+
custom_field_2: >-
435
+
int(dict([(x["id"], x["value"]) for x in custom_fields]).get(2))
436
+
if dict([(x["id"], x["value"]) for x in custom_fields]).get(2)
437
+
else None
438
+
```
439
+
````
440
+
441
+
````{tab} JSON
442
+
```json
443
+
{
444
+
"stream_maps": {
445
+
"nested_jellybean": {
446
+
"custom_field_2": "int(dict([(x[\"id\"], x[\"value\"]) for x in custom_fields]).get(2)) if dict([(x[\"id\"], x[\"value\"]) for x in custom_fields]).get(2) else None"
447
+
}
448
+
}
449
+
}
450
+
```
451
+
````
452
+
453
+
:::{note}
454
+
Type casting functions are standard Python built-ins that are available in the stream mapping expression evaluator. The SDK performs static type detection by examining the beginning of the expression string, so the type casting function must appear at the start of the expression for proper schema detection.
0 commit comments