Revert "DOCSP-29500 Clarify oplog Sizing (#131)" (#134)

This reverts commit d1fb748.

Author: kennethdyer

source/faq.txt

@@ -31,11 +31,20 @@ clusters.
 Should I increase the size of the ``oplog`` in the source cluster?
 ------------------------------------------------------------------
 
-The :term:`oplog` is a special capped collection that keeps a rolling 
-record of all operations that modify the data stored in your databases. 
-
 .. include:: /includes/fact-oplog-background
 
+The proper ``oplog`` size depends on system hardware, network speed,
+and other factors including system workload. However, assuming network
+transfer speeds of 30-50GB per hour, a rough formula to estimate the
+required ``oplog`` size is:
+
+.. code-block:: shell
+
+   minimumRetentionHours = dataSizeInGB / 30
+
+To estimate the size of ``oplog`` needed for initial synchronization,
+see: :ref:`c2c-oplog-sizing`.
+
 To learn more about how to increase the size of the ``oplog``, see:
 :ref:`tutorial-change-oplog-size`. 
 

source/includes/fact-oplog-background.rst

@@ -1,15 +1,7 @@
-
-``mongosync`` applies operations in the ``oplog`` on the source cluster
-to the data on the destination cluster.  When older operations 
-that ``mongosync`` has not applied roll off the ``oplog`` 
-on the source cluster, the sync fails and ``mongosync`` exits.
-
-During initial sync, ``mongosync`` may apply operations at a slower
-rate. After ``mongosync`` completes the initial sync, it applies changes 
-faster and stays more current in the source cluster ``oplog``.
+The :term:`oplog` in the source cluster must be large enough to track
+events that happen during the time it takes to complete the initial
+sync to the destination cluster.
 
 If you anticipate syncing a large data set, or if you plan to pause
-synchronization for an extended period of time, you might exceed the
-:term:`oplog window`. Use the :setting:`~replication.oplogSizeMB` setting
-to increase the size of the ``oplog`` on the source cluster.
-
+synchronization for an extended period of time, increase the size of the
+replica set ``oplog`` in the source cluster.

source/reference/oplog-sizing.txt

@@ -16,12 +16,11 @@ The :ref:`mongosync <c2c-mongosync>` program uses :ref:`change streams
 <changeStreams>` to synchronize data between source and destination
 clusters. ``mongosync`` does not access the :term:`oplog` directly,
 but when a change stream returns events from the past, the events must
-be within the ``oplog`` time range. 
+be within the :term:`oplog` time range. 
 
+.. include:: /includes/fact-oplog-background.txt
 
-.. include:: /includes/fact-oplog-background
-
-Monitor ``oplog`` Size Needed for Initial Sync
+Estimate ``oplog`` Size Needed for Initial Sync
 -----------------------------------------------
 
 .. procedure::
@@ -41,30 +40,55 @@ Monitor ``oplog`` Size Needed for Initial Sync
       cluster. If there are multiple shards, the smallest number is the
       minimum ``oplog`` window.
 
-   .. step:: Determine mongosync replication lag
+   .. step:: Estimate Copy Rate During Synching
+
+      .. _c2c-step-est-size:
+
+      To gather performance data while synching, start the sync process
+      and monitor how fast data is transferred between clusters.
+      
+      To start syncing, run the :ref:`/start <c2c-api-start>`
+      command.
 
-      To get the ``lagTimeSeconds`` value, run the  
-      :ref:`/progress <c2c-api-progress>` command. 
-      The lag time is the time in seconds between the 
-      last event applied by ``mongosync`` and time of the current
-      latest event on the source cluster.
+      To get the ``copy_rate``:
+      
+      - run the  :ref:`/progress <c2c-api-progress>` command to get
+        ``estimatedCopiedBytes_time01``
+      - wait a second or two
+      - run the  :ref:`/progress <c2c-api-progress>` command to get
+        ``estimatedCopiedBytes_time02``
+      
+      The ``copy_rate`` is:
+
+      .. code-block:: shell
+
+         copy_rate = ( estimatedCopiedBytes_time02 - estimatedCopiedBytes_01) / time_between_requests
+
+   .. step:: Estimate Copy Time
+
+      Estimate the time needed to copy the entire collection. The
+      estimated copy time is:
+      
+      .. code-block:: shell
 
-      It is a measure of how far behind the source cluster mongosync is.
+         estimatedCopyTime = estimatedTotalBytes / copy_rate
 
    .. step:: Validate ``oplog`` Size
 
-      If the lag time approaches the minimum ``oplog`` window, you
-      should make one of the following changes:
+      If the estimated time is larger than the minimum oplog window you
+      must cancel synchronization. Before restarting, make one of the
+      following changes:
 
-      - Increase the ``oplog`` window. Use :dbcommand:`replSetResizeOplog`
-        to set ``minRetentionHours`` greater than the current ``oplog`` 
-        window.
+      - Increase the oplog window. Use :dbcommand:`replSetResizeOplog`
+        to set ``minRetentionHours`` greater than the estimated copy
+        time.
       - Scale up the ``mongosync`` instance. Add cpu or memory to scale
         up the ``mongosync`` node so that it has a higher copy rate.
 
 .. note::
 
-   The ``oplog`` window and rate of change of replication lag may vary during 
-   synchronization. Repeat these steps during a migration to monitor the 
-   progress.
+   The copy rate may vary during synchronization. To monitor progress,
+   repeat the :ref:`steps to estimate the copy rate
+   <c2c-step-est-size>` and verify that the copy rate stays about the
+   same.