Skip to content

Commit 394b02b

Browse files
nvillahermosa-mdbnvillahermosa-mdb
authored
Docsp 32680 time series migration using out backport v7.0 (#4939)
* DOCSP-33154 Make covered query info discoverable (#4762) * Summarized covered query benefits. Opted not to use an include * Removed covered query info from wildcard indexes * Cherry pick * Manual merge conflict resolution
1 parent f4f5108 commit 394b02b

File tree

1 file changed

+140
-205
lines changed

1 file changed

+140
-205
lines changed

‎source/core/timeseries/timeseries-migrate-data-into-timeseries-collection.txt‎

Lines changed: 140 additions & 205 deletions
Original file line numberDiff line numberDiff line change
@@ -16,66 +16,155 @@ Migrate Data into a Time Series Collection
1616
:keywords: Time series, IOT
1717

1818
To migrate data from an existing collection into a :ref:`time series
19-
collection <manual-timeseries-collection>`:
20-
21-
1. :ref:`migrate-timeseries-new-collection`
22-
2. :ref:`migrate-timeseries-transform`
23-
3. :ref:`migrate-timeseries-migrate-data`
24-
25-
.. _migrate-timeseries-new-collection:
26-
27-
Create a New Time Series Collection
28-
-----------------------------------
29-
30-
To create a new :ref:`time series collection
31-
<manual-timeseries-collection>`, issue the following command in the
32-
:binary:`~bin.mongosh`:
33-
34-
.. code-block:: javascript
35-
36-
db.createCollection(
37-
"weathernew", {
38-
timeseries: {
39-
timeField: "ts",
40-
metaField: "metaData",
41-
granularity: "hours"
42-
}
19+
collection <manual-timeseries-collection>`, use an :pipeline:`$out`
20+
stage in your aggregation pipeline.
21+
22+
Migrate Data to a Time Series Collection
23+
----------------------------------------
24+
25+
.. procedure::
26+
27+
.. step:: (Optional) Transform your data to create a metadata field if one doesn't exist. This field is not required.
28+
29+
If the original collection doesn't have a metadata field, use the :pipeline:`$addFields` aggregation stage to add it.
30+
31+
Consider a collection with weather data that uses the format:
32+
33+
.. code-block:: javascript
34+
35+
{
36+
"_id" : ObjectId("5553a998e4b02cf7151190b8"),
37+
"st" : "x+47600-047900",
38+
"ts" : ISODate("1984-03-05T13:00:00Z"),
39+
"position" : {
40+
"type" : "Point",
41+
"coordinates" : [ -47.9, 47.6 ]
42+
},
43+
"elevation" : 9999,
44+
"callLetters" : "VCSZ",
45+
"qualityControlProcess" : "V020",
46+
"dataSource" : "4",
47+
"type" : "FM-13",
48+
"airTemperature" : { "value" : -3.1, "quality" : "1" },
49+
"dewPoint" : { "value" : 999.9, "quality" : "9" },
50+
"pressure" : { "value" : 1015.3, "quality" : "1" },
51+
"wind" : {
52+
"direction" : { "angle" : 999, "quality" : "9" },
53+
"type" : "9",
54+
"speed" : { "rate" : 999.9, "quality" : "9" }
55+
},
56+
"visibility" : {
57+
"distance" : { "value" : 999999, "quality" : "9" },
58+
"variability" : { "value" : "N", "quality" : "9" }
59+
},
60+
"skyCondition" : {
61+
"ceilingHeight" : { "value" : 99999, "quality" : "9", "determination" : "9" },
62+
"cavok" : "N"
63+
},
64+
"sections" : [ "AG1" ],
65+
"precipitationEstimatedObservation" : { "discrepancy" : "2",
66+
"estimatedWaterDepth" : 999 }
4367
}
44-
)
45-
46-
For more information on the preceeding command, see
47-
:ref:`manual-timeseries-collection-create`.
48-
49-
.. _migrate-timeseries-transform:
5068

51-
Transform Data (Optional)
52-
-------------------------
53-
54-
Time series collections support :ref:`secondary indexes
55-
<timeseries-add-secondary-index>` on the field specified as the
56-
``metaField``. If the data model of your time series data does not have
57-
a designated field for your metadata, you can transform your data to
58-
create one. To transform the data in your existing collection, use
59-
:pipeline:`$merge` or :pipeline:`$out` to create a temporary collection
60-
with your time series data.
61-
62-
Consider a collection with weather data of the following format:
69+
The following pipeline stages add a ``metaData`` field and use
70+
:pipeline:`$project` to include or exclude the remaining fields in
71+
the document:
72+
73+
.. code-block:: javascript
74+
75+
{ $addFields: {
76+
metaData: {
77+
"st": "$st",
78+
"position": "$position",
79+
"elevation": "$elevation",
80+
"callLetters": "$callLetters",
81+
"qualityControlProcess": "$qualityControlProcess",
82+
"type": "$type"
83+
}
84+
},
85+
},
86+
{ $project: {
87+
_id: 1,
88+
ts: 1,
89+
metaData: 1,
90+
dataSource: 1,
91+
airTemperature: 1,
92+
dewPoint: 1,
93+
pressure: 1,
94+
wind: 1,
95+
visibility: 1,
96+
skyCondition: 1,
97+
sections: 1,
98+
precipitationEstimatedObservation: 1
99+
}
100+
}
101+
102+
.. step:: Use the timeseries option with the $out aggregation stage
103+
104+
The example below uses the :method:`db.collection.aggregate` helper method. For the aggregation stage syntax, see :pipeline:`$out`. For a full explanation of the time series options, see the :ref:`Time Series Field Reference <time-series-fields>`.
105+
106+
.. code-block:: javascript
107+
108+
db.weather_data.aggregate([
109+
{
110+
$addFields: {
111+
metaData: {
112+
"st": "$st",
113+
"position": "$position",
114+
"elevation": "$elevation",
115+
"callLetters": "$callLetters",
116+
"qualityControlProcess": "$qualityControlProcess",
117+
"type": "$type"
118+
}
119+
},
120+
}, {
121+
$project: {
122+
_id: 1,
123+
ts: 1,
124+
metaData: 1,
125+
dataSource: 1,
126+
airTemperature: 1,
127+
dewPoint: 1,
128+
pressure: 1,
129+
wind: 1,
130+
visibility: 1,
131+
skyCondition: 1,
132+
sections: 1,
133+
precipitationEstimatedObservation: 1
134+
}
135+
}, {
136+
$out: {
137+
db: "mydatabase",
138+
coll: "weathernew",
139+
timeseries: {
140+
timeField: "ts",
141+
metaField: "metaData"
142+
}
143+
}
144+
}
145+
])
146+
147+
After you run this command, you have the ``weathernew``
148+
collection below:
63149

64150
.. code-block:: javascript
65151

152+
db.weathernew.findOne()
66153
{
67154
"_id" : ObjectId("5553a998e4b02cf7151190b8"),
68-
"st" : "x+47600-047900",
69155
"ts" : ISODate("1984-03-05T13:00:00Z"),
70-
"position" : {
71-
"type" : "Point",
72-
"coordinates" : [ -47.9, 47.6 ]
156+
"metaData" : {
157+
"st" : "x+47600-047900",
158+
"position" : {
159+
"type" : "Point",
160+
"coordinates" : [ -47.9, 47.6 ]
161+
},
162+
"elevation" : 9999,
163+
"callLetters" : "VCSZ",
164+
"qualityControlProcess" : "V020",
165+
"type" : "FM-13"
73166
},
74-
"elevation" : 9999,
75-
"callLetters" : "VCSZ",
76-
"qualityControlProcess" : "V020",
77167
"dataSource" : "4",
78-
"type" : "FM-13",
79168
"airTemperature" : { "value" : -3.1, "quality" : "1" },
80169
"dewPoint" : { "value" : 999.9, "quality" : "9" },
81170
"pressure" : { "value" : 1015.3, "quality" : "1" },
@@ -94,153 +183,8 @@ Consider a collection with weather data of the following format:
94183
},
95184
"sections" : [ "AG1" ],
96185
"precipitationEstimatedObservation" : { "discrepancy" : "2", "estimatedWaterDepth" : 999 }
97-
}
98-
99-
To transform this data, we issue the following command:
100-
101-
.. code-block:: javascript
102-
103-
db.weather_data.aggregate([
104-
{
105-
$addFields: {
106-
metaData: {
107-
"st": "$st",
108-
"position": "$position",
109-
"elevation": "$elevation",
110-
"callLetters": "$callLetters",
111-
"qualityControlProcess": "$qualityControlProcess",
112-
"type": "$type"
113-
}
114-
},
115-
}, {
116-
$project: {
117-
_id: 1,
118-
ts: 1,
119-
metaData: 1,
120-
dataSource: 1,
121-
airTemperature: 1,
122-
dewPoint: 1,
123-
pressure: 1,
124-
wind: 1,
125-
visibility: 1,
126-
skyCondition: 1,
127-
sections: 1,
128-
precipitationEstimatedObservation: 1
129-
}
130-
}, {
131-
$out: "temporarytimeseries"
132-
}
133-
])
134-
135-
136-
After you run this command, you have an intermediary
137-
``temporarytimeseries`` collection:
138-
139-
.. code-block:: javascript
140-
141-
db.temporarytimeseries.findOne()
142-
{
143-
"_id" : ObjectId("5553a998e4b02cf7151190b8"),
144-
"ts" : ISODate("1984-03-05T13:00:00Z"),
145-
"dataSource" : "4",
146-
"airTemperature" : { "value" : -3.1, "quality" : "1" },
147-
"dewPoint" : { "value" : 999.9, "quality" : "9" },
148-
"pressure" : { "value" : 1015.3, "quality" : "1" },
149-
"wind" : {
150-
"direction" : { "angle" : 999, "quality" : "9" },
151-
"type" : "9",
152-
"speed" : { "rate" : 999.9, "quality" : "9" }
153-
},
154-
"visibility" : {
155-
"distance" : { "value" : 999999, "quality" : "9" },
156-
"variability" : { "value" : "N", "quality" : "9" }
157-
},
158-
"skyCondition" : {
159-
"ceilingHeight" : { "value" : 99999, "quality" : "9", "determination" : "9" },
160-
"cavok" : "N"
161-
},
162-
"sections" : [ "AG1" ],
163-
"precipitationEstimatedObservation" : { "discrepancy" : "2", "estimatedWaterDepth" : 999 },
164-
"metaData" : {
165-
"st" : "x+47600-047900",
166-
"position" : {
167-
"type" : "Point",
168-
"coordinates" : [ -47.9, 47.6 ]
169-
},
170-
"elevation" : 9999,
171-
"callLetters" : "VCSZ",
172-
"qualityControlProcess" : "V020",
173-
"type" : "FM-13"
174-
}
175186
}
176187

177-
.. _migrate-timeseries-migrate-data:
178-
179-
Migrate Data into a Time Series Collection
180-
------------------------------------------
181-
182-
To migrate your data from an existing collection that is not of type
183-
``timeseries`` into a :ref:`time series collection
184-
<manual-timeseries-collection>`, use :binary:`~bin.mongodump` and
185-
:binary:`~bin.mongorestore`.
186-
187-
.. warning::
188-
189-
When migrating or backfilling into a time series collection you
190-
should always insert the documents in order, from oldest to newest.
191-
In this case :binary:`~bin.mongodump` exports documents in natural
192-
order and the ``--maintainInsertionOrder`` option for
193-
:binary:`~bin.mongorestore` guarantees the same insertion order for
194-
documents.
195-
196-
For example, to export the ``temporarytimeseries`` collection, issue the
197-
following command:
198-
199-
.. code-block:: javascript
200-
201-
mongodump
202-
--uri="mongodb://mongodb0.example.com:27017,mongodb1.example.com:27017,mongodb2.example.com:27017/weather" \
203-
--collection=temporarytimeseries --out=timeseries
204-
205-
The command returns the following output:
206-
207-
.. code-block:: javascript
208-
:copyable: false
209-
210-
2021-06-01T16:48:39.980+0200 writing weather.temporarytimeseries to timeseries/weather/temporarytimeseries.bson
211-
2021-06-01T16:48:40.056+0200 done dumping weather.temporarytimeseries (10000 documents)
212-
213-
To import ``timeseries/weather/temporarytimeseries.bson`` into the new
214-
collection ``weathernew``, issue the following command:
215-
216-
.. code-block:: javascript
217-
218-
mongorestore
219-
--uri="mongodb://mongodb0.example.com:27017,mongodb1.example.com:27017,mongodb2.example.com:27017/weather" \
220-
--collection=weathernew --noIndexRestore \
221-
--maintainInsertionOrder \
222-
timeseries/weather/temporarytimeseries.bson
223-
224-
The command returns the following output:
225-
226-
.. code-block:: javascript
227-
:copyable: false
228-
229-
2021-06-01T16:50:56.639+0200 checking for collection data in timeseries/weather/temporarytimeseries.bson
230-
2021-06-01T16:50:56.640+0200 restoring to existing collection weather.weathernew without dropping
231-
2021-06-01T16:50:56.640+0200 reading metadata for weather.weathernew from timeseries/weather/temporarytimeseries.metadata.json
232-
2021-06-01T16:50:56.640+0200 restoring weather.weathernew from timeseries/weather/temporarytimeseries.bson
233-
2021-06-01T16:51:01.229+0200 no indexes to restore
234-
2021-06-01T16:51:01.229+0200 finished restoring weather.weathernew (10000 documents, 0 failures)
235-
2021-06-01T16:51:01.229+0200 10000 document(s) restored successfully. 0 document(s) failed to restore.
236-
237-
.. note::
238-
239-
Ensure that you run the preceeding command with the
240-
:option:`--noIndexRestore <mongorestore.--noIndexRestore>` option.
241-
:binary:`~bin.mongorestore` cannot create indexes on time series
242-
collections.
243-
244188
If your original collection had secondary indexes, manually recreate
245189
them now. If your collection includes ``timeField`` values before
246190
``1970-01-01T00:00:00.000Z`` or after ``2038-01-19T03:14:07.000Z``,
@@ -251,13 +195,4 @@ performance and resolve the log warning.
251195

252196
.. seealso::
253197

254-
:ref:`timeseries-add-secondary-index`
255-
256-
257-
If you insert a document into a collection with a ``timeField``
258-
value before ``1970-01-01T00:00:00.000Z`` or after
259-
``2038-01-19T03:14:07.000Z``,
260-
MongoDB logs a warning and prevents some query optimizations from
261-
using the internal index. :ref:`Create a secondary index <timeseries-add-secondary-index>`
262-
on the ``timeField`` to regain query performance and resolve the log
263-
warning.
198+
:ref:`timeseries-add-secondary-index`

0 commit comments

Comments
 (0)