Skip to content

Commit 3266e52

Browse files
committed
DOCSP-48557 spark options update
1 parent 78d1807 commit 3266e52

File tree

1 file changed

+110
-0
lines changed

1 file changed

+110
-0
lines changed

source/streaming-mode/streaming-write-config.txt

Lines changed: 110 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -56,6 +56,116 @@ You can configure the following properties when writing data to MongoDB in strea
5656
interface.
5757
|
5858
| **Default:** ``com.mongodb.spark.sql.connector.connection.DefaultMongoClientFactory``
59+
60+
* - ``convertJson``
61+
- | Specifies whether the connector parses the string and converts extended JSON
62+
into BSON.
63+
|
64+
| This setting accepts the following values:
65+
66+
- ``any``: The connector converts all JSON values to BSON.
67+
68+
- ``"{a: 1}"`` becomes ``{a: 1}``.
69+
- ``"[1, 2, 3]"`` becomes ``[1, 2, 3]``.
70+
- ``"true"`` becomes ``true``.
71+
- ``"01234"`` becomes ``1234``.
72+
- ``"{a:b:c}"`` doesn't change.
73+
74+
- ``objectOrArrayOnly``: The connector converts only JSON objects and arrays to
75+
BSON.
76+
77+
- ``"{a: 1}"`` becomes ``{a: 1}``.
78+
- ``"[1, 2, 3]"`` becomes ``[1, 2, 3]``.
79+
- ``"true"`` doesn't change.
80+
- ``"01234"`` doesn't change.
81+
- ``"{a:b:c}"`` doesn't change.
82+
83+
- ``false``: The connector leaves all values as strings.
84+
85+
| **Default:** ``false``
86+
87+
* - ``idFieldList``
88+
- | Field or list of fields by which to split the collection data. To
89+
specify more than one field, separate them using a comma as shown
90+
in the following example:
91+
92+
.. code-block:: none
93+
:copyable: false
94+
95+
"fieldName1,fieldName2"
96+
97+
| **Default:** ``_id``
98+
99+
* - ``ignoreNullValues``
100+
- | When ``true``, the connector ignores any ``null`` values when writing,
101+
including ``null`` values in arrays and nested documents.
102+
|
103+
| **Default:** ``false``
104+
105+
* - ``maxBatchSize``
106+
- | Specifies the maximum number of operations to batch in bulk
107+
operations.
108+
|
109+
| **Default:** ``512``
110+
111+
* - ``operationType``
112+
- | Specifies the type of write operation to perform. You can set
113+
this to one of the following values:
114+
115+
- ``insert``: Insert the data.
116+
- ``replace``: Replace an existing document that matches the
117+
``idFieldList`` value with the new data. If no match exists, the
118+
value of ``upsertDocument`` indicates whether the connector
119+
inserts a new document.
120+
- ``update``: Update an existing document that matches the
121+
``idFieldList`` value with the new data. If no match exists, the
122+
value of ``upsertDocument`` indicates whether the connector
123+
inserts a new document.
124+
125+
|
126+
| **Default:** ``replace``
127+
128+
* - ``ordered``
129+
- | Specifies whether to perform ordered bulk operations.
130+
|
131+
| **Default:** ``true``
132+
133+
* - ``upsertDocument``
134+
- | When ``true``, replace and update operations will insert the data
135+
if no match exists.
136+
|
137+
| For time series collections, you must set ``upsertDocument`` to
138+
``false``.
139+
|
140+
| **Default:** ``true``
141+
142+
* - ``writeConcern.journal``
143+
- | Specifies ``j``, a write-concern option to enable request for
144+
acknowledgment that the data is confirmed on on-disk journal for
145+
the criteria specified in the ``w`` option. You can specify
146+
either ``true`` or ``false``.
147+
|
148+
| For more information on ``j`` values, see the MongoDB server
149+
guide on the
150+
:manual:`WriteConcern j option </reference/write-concern/#j-option>`.
151+
152+
* - ``writeConcern.w``
153+
- | Specifies ``w``, a write-concern option to request acknowledgment
154+
that the write operation has propagated to a specified number of
155+
MongoDB nodes. For a list
156+
of allowed values for this option, see :manual:`WriteConcern
157+
</reference/write-concern/#w-option>` in the MongoDB manual.
158+
|
159+
| **Default:** ``1``
160+
161+
* - ``writeConcern.wTimeoutMS``
162+
- | Specifies ``wTimeoutMS``, a write-concern option to return an error
163+
when a write operation exceeds the number of milliseconds. If you
164+
use this optional setting, you must specify a nonnegative integer.
165+
|
166+
| For more information on ``wTimeoutMS`` values, see the MongoDB server
167+
guide on the
168+
:manual:`WriteConcern wtimeout option </reference/write-concern/#wtimeout>`.
59169

60170
* - ``checkpointLocation``
61171
- | The absolute file path of the directory to which the connector writes checkpoint

0 commit comments

Comments
 (0)