Skip to content

Commit 406f02b

Browse files
aimurphyaimurphy
committed
fix 3 more things
1 parent e08b98b commit 406f02b

File tree

4 files changed

+166
-93
lines changed

4 files changed

+166
-93
lines changed

‎modules/ROOT/nav.adoc‎

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,6 @@
1010
* xref:cfgPulsarMapJson.adoc[]
1111
* xref:cfgPulsarMapAvroMessages.adoc[]
1212
* xref:cfgPulsarMapUdt.adoc[]
13-
* xref:cfgPulsarRecordHeaderToTable.adoc[]
1413
* xref:cfgPulsarCqlQuery.adoc[]
1514
* xref:cfgPulsarMapMultipleTables.adoc[]
1615
* xref:cfgPulsarMultTopicToTables.adoc[]

‎modules/ROOT/pages/cfgPulsarMapMultipleTables.adoc‎

Lines changed: 59 additions & 40 deletions
Original file line numberDiff line numberDiff line change
@@ -4,56 +4,75 @@ The DataStax Apache Pulsar(TM) connector allows for mapping a single topic to mu
44

55
In the example `stocks_topic`, the key is a basic string and the value is regular JSON.
66

7-
[cols="a,a"]
87
|===
98
|key|value
109

11-
|
12-
APPLE
13-
|
14-
[source,console]
10+
|`APPLE`
11+
a|
12+
[source,json]
13+
----
14+
{
15+
"symbol":"APPL",
16+
"value":208,
17+
"exchange":"NASDAQ",
18+
"industry":"TECH",
19+
"ts":"2018-11-26T19:26:27.483"
20+
}
1521
----
16-
{"symbol":"APPL", "value":208, "exchange":"NASDAQ", "industry":"TECH", "ts":"2018-11-26T19:26:27.483"}
17-
----
18-
|
19-
EXXON MOBIL
20-
|
21-
[source,console]
22-
----
23-
{"symbol":"M",
24-
"value":80,
25-
"exchange":"NYSE",
26-
"industry":"ENERGY",
27-
"ts":"2018-11-26T19:26:27.483"}
28-
----
29-
|
30-
GENERAL MOTORS
31-
|
32-
[source,console]
33-
----
34-
{"symbol":"GM", "value":38, "exchange":"NYSE", "industry":"AUTO", "ts":"2018-11-26T19:26:27.483"}
35-
----
36-
|
37-
AT&T
38-
|
39-
[source,console]
22+
23+
|`EXXON MOBIL`
24+
a|
25+
[source,json]
26+
----
27+
{
28+
"symbol":"M",
29+
"value":80,
30+
"exchange":"NYSE",
31+
"industry":"ENERGY",
32+
"ts":"2018-11-26T19:26:27.483"
33+
}
4034
----
41-
{"symbol":"AT&T",
42-
"value":33,
43-
"exchange":"NYSE",
44-
"industry":"TELECOM",
45-
"ts":"2018-11-26T19:26:27.483"}
35+
36+
|`GENERAL MOTORS`
37+
a|
38+
[source,json]
39+
----
40+
{
41+
"symbol":"GM",
42+
"value":38,
43+
"exchange":"NYSE",
44+
"industry":"AUTO",
45+
"ts":"2018-11-26T19:26:27.483"
46+
}
4647
----
47-
|
48-
FORD MOTOR
49-
|
50-
[source,console]
48+
49+
|`AT&T`
50+
a|
51+
[source,json]
52+
----
53+
{
54+
"symbol":"AT&T",
55+
"value":33,
56+
"exchange":"NYSE",
57+
"industry":"TELECOM",
58+
"ts":"2018-11-26T19:26:27.483"
59+
}
5160
----
52-
{"symbol":"F", "value":10, "exchange":"NYSE", "industry":"AUTO", "ts":"2018-11-26T19:26:27.483"}
61+
62+
|`FORD MOTORS`
63+
a|
64+
[source,json]
65+
----
66+
{
67+
"symbol":"F",
68+
"value":10,
69+
"exchange":"NYSE",
70+
"industry":"AUTO",
71+
"ts":"2018-11-26T19:26:27.483"
72+
}
5373
----
5474
|===
5575

56-
5776
In the DataStax keyspace `stocks_keyspace`, create three different tables that optimized with different schemas.
5877

5978
* `stocks_table_by_symbol`

‎modules/ROOT/pages/cfgPulsarMultTopicToTables.adoc‎

Lines changed: 88 additions & 49 deletions
Original file line numberDiff line numberDiff line change
@@ -6,46 +6,72 @@ NOTE: Most Apache Pulsar(TM) systems store all messages in the same format and D
66

77
In the example `stocks_topic`, the key is a basic string and the value is regular JSON.
88

9-
[cols="a,a"]
109
|===
1110
|key|value
1211

13-
|APPLE|
14-
[source,console]
15-
----
16-
{"symbol":"APPL", "value":208, "exchange":"NASDAQ", "industry":"TECH", "ts":"2018-11-26T19:26:27.483"}
17-
----
18-
19-
|EXXON MOBIL|
20-
[source,console]
21-
----
22-
{"symbol":"M",
23-
"value":80,
24-
"exchange":"NYSE",
25-
"industry":"ENERGY",
26-
"ts":"2018-11-26T19:26:27.483"}
27-
----
28-
29-
|GENERAL MOTORS|
30-
[source,console]
31-
----
32-
{"symbol":"GM", "value":38, "exchange":"NYSE", "industry":"AUTO", "ts":"2018-11-26T19:26:27.483"}
33-
----
34-
35-
|AT&T|
36-
[source,console]
37-
----
38-
{"symbol":"AT&T",
39-
"value":33,
40-
"exchange":"NYSE",
41-
"industry":"TELECOM",
42-
"ts":"2018-11-26T19:26:27.483"}
43-
----
44-
45-
|FORD MOTOR|
46-
[source,console]
47-
----
48-
{"symbol":"F", "value":10, "exchange":"NYSE", "industry":"AUTO", "ts":"2018-11-26T19:26:27.483"}
12+
|`APPLE`
13+
a|
14+
[source,json]
15+
----
16+
{
17+
"symbol":"APPL",
18+
"value":208,
19+
"exchange":"NASDAQ",
20+
"industry":"TECH",
21+
"ts":"2018-11-26T19:26:27.483"
22+
}
23+
----
24+
25+
|`EXXON MOBIL`
26+
a|
27+
[source,json]
28+
----
29+
{
30+
"symbol":"M",
31+
"value":80,
32+
"exchange":"NYSE",
33+
"industry":"ENERGY",
34+
"ts":"2018-11-26T19:26:27.483"
35+
}
36+
----
37+
38+
|`GENERAL MOTORS`
39+
a|
40+
[source,json]
41+
----
42+
{
43+
"symbol":"GM",
44+
"value":38,
45+
"exchange":"NYSE",
46+
"industry":"AUTO",
47+
"ts":"2018-11-26T19:26:27.483"
48+
}
49+
----
50+
51+
|`AT&T`
52+
a|
53+
[source,json]
54+
----
55+
{
56+
"symbol":"AT&T",
57+
"value":33,
58+
"exchange":"NYSE",
59+
"industry":"TELECOM",
60+
"ts":"2018-11-26T19:26:27.483"
61+
}
62+
----
63+
64+
|`FORD MOTORS`
65+
a|
66+
[source,json]
67+
----
68+
{
69+
"symbol":"F",
70+
"value":10,
71+
"exchange":"NYSE",
72+
"industry":"AUTO",
73+
"ts":"2018-11-26T19:26:27.483"
74+
}
4975
----
5076
|===
5177

@@ -55,8 +81,9 @@ And `baseball_topic` has the same structure, basic key and JSON value.
5581
|===
5682
|key|value
5783

58-
|redsox|
59-
[source,console]
84+
|redsox
85+
a|
86+
[source,json]
6087
----
6188
{"number":50,
6289
"firstname":"mookie",
@@ -65,14 +92,20 @@ And `baseball_topic` has the same structure, basic key and JSON value.
6592
"city":"boston"}
6693
----
6794

68-
|redsox|
69-
[source,console]
95+
|redsox
96+
a|
97+
[source,json]
7098
----
71-
{"number":28, "firstname":"jd", "lastname":"martinez", "street":"4 yawkey way", "city":"boston"}
99+
{"number":28,
100+
"firstname":"jd",
101+
"lastname":"martinez",
102+
"street":"4 yawkey way",
103+
"city":"boston"}
72104
----
73105

74-
|redsox|
75-
[source,console]
106+
|redsox
107+
a|
108+
[source,json]
76109
----
77110
{"number":16,
78111
"firstname":"andrew",
@@ -81,14 +114,20 @@ And `baseball_topic` has the same structure, basic key and JSON value.
81114
"city":"boston"}
82115
----
83116

84-
|redsox|
85-
[source,console]
117+
|redsox
118+
a|
119+
[source,json]
86120
----
87-
{"number":41, "firstname":"chris", "lastname":"sale", "street":"4 yawkey way", "city":"boston"}
121+
{"number":41,
122+
"firstname":"chris",
123+
"lastname":"sale",
124+
"street":"4 yawkey way",
125+
"city":"boston"}
88126
----
89127

90-
|redsox|
91-
[source,console]
128+
|redsox
129+
a|
130+
[source,json]
92131
----
93132
{"number":24,
94133
"firstname":"david",

‎modules/ROOT/pages/cfgRefPulsarDseTable.adoc‎

Lines changed: 19 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
= Pulsar topic-to-table parameters
2-
:page-aliases: ROOT:cfgPulsarSettingRowLevelTtl.adoc, ROOT:cfgPulsarMapTopicTable.adoc, ROOT:cfgPulsarColumnWritetimeTimestamp.adoc, ROOT:cfgRefPulsarDates.adoc, ROOT:cfgPulsarNowFunction.adoc
2+
:page-aliases: ROOT:cfgPulsarSettingRowLevelTtl.adoc, ROOT:cfgPulsarMapTopicTable.adoc, ROOT:cfgPulsarColumnWritetimeTimestamp.adoc, ROOT:cfgRefPulsarDates.adoc, ROOT:cfgPulsarNowFunction.adoc, ROOT:cfgPulsarRecordHeaderToTable.adoc
33

44
The connector writes messages from your Apache Pulsar(TM) topics to tables in your database using the mapping and topic properties set in your connector's configuration YAML file.
55

@@ -177,11 +177,27 @@ If you specify a different time unit, such as `HOURS`, the DataStax Pulsar conne
177177
[#mapping-properties]
178178
=== Mapping properties and functions
179179

180-
The `mapping` parameter accepts additional properties that you can use to modify how the connector writes records to the table:
180+
The `<<mapping,mapping>>` parameter accepts additional properties that you can use to modify how the connector writes records to the table:
181+
182+
header::
183+
In the `mapping` parameter, you can extract values from the message properties, and then write those values to a database table.
184+
+
185+
For example, the following mapping extracts the `header` property `f4` from the message properties, and writes it to the `col3` column in the specified table:
186+
+
187+
[source,yaml]
188+
----
189+
topic:
190+
topic_name:
191+
keyspace_name:
192+
table_name:
193+
mapping: 'col1=key.f1, col2=value.f1, __ttl=value.f2, __timestamp=value.f3, col3=header.f4'
194+
----
195+
+
196+
Use one or more individual mapping statements to extract the desired values and write them to table columns.
181197

182198
[#timestamp]
183199
__timestamp, timestampTimeUnit::
184-
In the `<<mapping,mapping>>` parameter, you can use the optional `\__timestamp` property to specify which column should be used for the writetime timestamp when a record is written to the database.
200+
In the `mapping` parameter, you can use the optional `\__timestamp` property to specify which column should be used for the writetime timestamp when a record is written to the database.
185201
The specified `__timestamp` column must be a number type.
186202
For example:
187203
+

0 commit comments

Comments
 (0)