Update of nbackup documentation

src/docs/asciidoc/en/firebirddocs/gfix/firebird-gfix.adoc

@@ -1,7 +1,7 @@
 [[gfix]]
 = Firebird Database Housekeeping Utility
 Norman Dunbar
-1.8, 19 June 2020
+1.9, 25 July 2020
 :doctype: book
 :sectnums:
 :sectanchors:
@@ -343,7 +343,9 @@ It is far quicker to read data from memory that it is to have to physically read
 
 The size of the database cache is dependent on the database page size and the number of buffers allocated, a buffer is the same size as a database page, and whether the installation is using Classic or Superserver versions of Firebird.
 
-In a Classic Server installation, each connection to the database gets its own relatively small cache of 75 pages while Superserver creates a much larger cache of 2,048 pages which is shared between all the connections.
+In a Classic and SuperClassic Server installation, by default, each connection to the database gets its own relatively small cache of 256 pages (was 75 before Firebird 3.0) while Superserver creates a much larger cache of 8196 pages (was 2048 before Firebird 3.0) which is shared between all the connections.
+
+Usually, page buffers value is set as parameter DefaultDbCachePages in firebird.conf (and, in databases.conf since Firebird 3.0), but it also can be set directly in the database header using gfix tool (and this setting will override firebird.conf and databases.conf, so be careful).
 
 The command to set the number of cache pages is:
 
@@ -355,11 +357,21 @@ gfix -b[uffers] _BUFFERS_ _database_name_
 This command allows you to change the number of buffers (pages) allocated in RAM to create the database cache.
 
 You cannot change the database page size in this manner, only the number of pages reserved in RAM.
-One parameter is required which must be numeric and between 50 (the minimum) and 131,072 (the maximum).
+
+One parameter is required which must be numeric and between 50 (the minimum) and 2147483646 (the maximum since Firebird 2.5, before it was 131072). 
+
+[NOTE]
+====
+Please don't consider page buffers value as an ultimate parameter to improve Firebird's performance, and don't set it too high without clear understanding what are you doing.  
+====
 
 The setting applies only to the database you specify.
 No other databases running on the same server are affected.
 
+The new value of page buffers will be allocated at the first connection to the database. 
+
+There is a difference in behaviour between Firebird architectures: for SuperServer, if number of page buffers was changed while database had active connection, the old value will be used until all connections disconnect; for Classic/SuperClassic, new connections will use the new value of page buffers immediately after the change.
+
 The following example shows the use of `gstat` to read the current number of buffers, the `gfix` utility being used to set the buffers to 4,000 pages and `gstat` being used to confirm the setting.
 The value of zero for page buffers indicates the default setting for the server type is in use.
 
@@ -771,11 +783,12 @@ Firebird will automatically sweep through the database and remove the remnants o
 
 [NOTE]
 ====
-One other method of clearing out old rolled back transactions' garbage is simply to carry out a database backup.
+Theoretically, it is possible to clear out old rolled back transactions' garbage with a database backup by gbak tool.
 `Gbak` reads every table sequentially and thus visits every row in every table.
 Applications which also visit every row in one or more tables, will also cause the garbage in those tables to be collected.
-
 Neither will affect the database's OIT (Oldest Intersting Transaction) or OST (Oldest Snapshot) settings however.
+
+In practice, better do not consider backup as an alternative for sweep, because sweep does more things than collective garbage collection.
 ====
 
 In the Super Server version of Firebird 2.0, garbage collection has been vastly improved.
@@ -1144,8 +1157,9 @@ linux>
 [[gfix-pagespace]]
 == Database Page Space Utilisation
 
-When a database page is being written to, Firebird reserves 20% of the page for future use.
-This could be used to extend `VARCHAR` columns that started off small and then were updated to a longer value, for example.
+Firebird reserves some space on the data page for the possible future updates or deletes of records on this page, to reduce possible fragmentation. 
+The amount of reserved space can be approximately considered as a number of records, multipled by the record header size.
+Therefore, for many very small records, reserve can be close to 50% of data page, and for the single huge record can be 22 bytes or so (depends on the version of On-Disk Structure of the database).
 
 If you wish to use all the available space in each database page, you may use the `-use` command to configure the database to do so.
 If you subsequently wish to return to the default behaviour, the `-use` command can be used to revert back to leaving 20% free space per page.
@@ -1316,6 +1330,65 @@ This causes the corruptions in data records to be ignored.
 While this sounds like a good thing, it is not.
 Subsequent database actions (such as taking a backup) will not include the corrupted records, leading to data loss.
 
+[[gfix-db-practice]]
+==== Practical tips for databases recovery
+
+In practice, to recover database it is necessary to use gfix and gbak tools.
+
+First of all, before the recovery, it is necessary to create a file copy of the corrupted database file, to be able to repeat recovery procedure with some variations.
+
+Then, run gfix full validation command with disabled cheksums validation:
+[listing,subs=+quotes]
+----
+gfix -v[alidate] -full -ignore _database_name_
+----
+After that, run mend, also with disabled checksum validation:
+[listing,subs=+quotes]
+----
+gfix -m[end] -ignore _database_name_
+----
+
+In case of a corruption, gfix returns the summary of errors found:
+
+* Number of record level errors - number of corrupted records found during gfix work. 
+These records are not correct - essentially, lost.
+* Number of index page errors - number of index pages in bad indices. 
+When even the only key is incorrect in index, gfix marks whole index as bad, so number of pages usually is high. 
+However, since it does not affect user's data, and due to the fact that corrupted indices will be recreated during backup/restore, this can be considered as for your information only.
+* Number of transaction page errors - number of transaction pages which were fixed by gfix. 
+Usually if you see this message it means that gfix did its job and now transactions are Ok.
+* Number of BLOB errors - number of bad BLOB pages, it indicates number of bad BLOBs.
+* Number of database page errors - this is the overall number of database pages, which were visited and changed/marked as bad by gfix. 
+Again, this is mostly for your information.
+
+[NOTE]
+====
+Gfix prints the detailed information about found errors to the firebird.log
+====
+
+After that, it is necessary to do backup and restore of the corrupted database using gbak command:
+[listing,subs=+quotes]
+----
+gbak -backup -ignore _database_name_  _backup1.fbk_
+
+gbak -create  _backup1.fbk_ _new_database_name_ 
+----
+
+If backup and restore with gbak completed successfully, database is recovered.
+
+Since Firebird 3.0, gbak tools has very useful option: -SKIP_D[ATA]. 
+It allows to specify comma-separated list of tables to be skipped from the backup:
+----
+gbak -b -skip_data country,currency  localhost/3050:C:\Data\inventory.fdb C:\data\111.gbk  -user SYSDBA -pass masterkey
+----
+For recovery purposes, it allows to exclude tables with reported corruptions from the backup.
+
+If the process above did not fix the corruption, or error still persists, consider the following alternatives:
+
+* If database is readable, pump data from the corrupted database to the empty database with the same structure
+* Use https://firebirdsql.org/en/third-party-tools/#rec[third-party Firebird recovery tools]
+* Revert to the last backup
+
 [IMPORTANT]
 ====
 The best way to avoid data loss is to make sure that you have enough regular backups of your database and to regularly carry out test restorations.
@@ -1553,6 +1626,11 @@ Courtesy of Ann Harrison.
 |19 Jun 2020
 |MR
 |Conversion to AsciiDoc, minor copy-editing
+
+|1.9
+|25 Jul 2020
+|AK
+|Changed description for gfix -buffers, gfix -reserve, gfix recovery, and some small fixes.
 |===
 
 :sectnums:
@@ -1573,5 +1651,6 @@ The Initial Writer of the Original Documentation is: Norman Dunbar.
 Copyright (C) 2007–2019.
 All Rights Reserved.
 Initial Writer contact: NormanDunbar at users dot sourceforge dot net.
+Contributor(s): Alexey Kovyazin
 
-:sectnums:
+:sectnums:

src/docs/asciidoc/en/firebirddocs/nbackup/firebird-nbackup.adoc

@@ -1,7 +1,7 @@
 [[nbackup]]
 = Firebird's nbackup tool
-Paul Vinkenoog; Mark Rotteveel
-1.5, 27 June 2020
+Paul Vinkenoog; Mark Rotteveel; Alexey Kovyazin
+1.6, 25 July 2020
 :doctype: book
 :sectnums:
 :sectanchors:
@@ -27,14 +27,16 @@ they will likely coexist for some time to come.
 With nbackup, you can perform two different kinds of tasks:
 
 . Making and restoring of both full and [term]_incremental_ backups.
-An [term]_incremental backup_ only contains the mutations since some specific previous backup.
+An [term]_incremental backup_ only contains the mutations since some specific previous backup. 
+This task can be performed only with Firebird command-line tool nbackup.
 . Locking the main database file so you can subsequently back it up yourself with copying or backup tools of your own choice.
-In this mode, nbackup doesn't back up anything;
-it just creates the conditions under which you can safely make the backup yourself.
-There's a provision for restoring here, too.
+In this mode, nbackup doesn't back up anything; it just creates the conditions under which you can safely make the backup yourself. 
+There's a provision for restoring here, too. 
+Locking is necessary to make valid backups of Firebird databases inside virtual machines: without locking a database before backup of VM, the database file in the VM backup could be inconsistent. 
+See <<nbackup-lock-virtual-machine>>.
 
 Both modes can operate on an active database, without hindering connected users.
-The backup created will always reflect the state of the database _at the beginning of the     operation_.
+The backup created will always reflect the state of the database _at the beginning of the operation_.
 Furthermore, only `SYSDBA` and the database owner (and sometimes OS admins) can make a backup, but every user can restore a backup to a new database.
 In these respects nbackup doesn't differ from gbak.
 
@@ -50,7 +52,7 @@ This can make a huge difference with databases in the gigabyte range.
 [[nbackup-limitations]]
 === Limitations of nbackup
 
-* Nbackup will not sweep and compact your database the way gbak does.
+* Nbackup will not sweep and compact your database the way gbak does (in a case of backup with subsequent restore).
 * You can't change the database owner with an nbackup backup/restore cycle, like you can with gbak.
 * Nbackup can't make [term]_transportable backups_, that is: backups you can restore on an incompatible platform or under another server version.
 * At this moment, nbackup should not be used on multi-file databases.
@@ -171,7 +173,7 @@ Access to the backup file is -- obviously -- always on the filesystem level.
 == Making and restoring backups
 
 To begin with: `nbackup.exe` is located in the `bin` subdirectory of your Firebird folder.
-Typical locations are e.g. `C:\Program Files\Firebird\Firebird_2_0\bin` (Windows) or `/opt/firebird/bin` (Linux).
+Typical locations are e.g. `C:\Program Files\Firebird\Firebird_3_0` or `C:\Program Files\Firebird\Firebird_2_0\bin` (Windows) or `/opt/firebird/bin` (Linux).
 Just like most of the tools that come with Firebird, nbackup has no graphical interface;
 you launch it from the command prompt or call it from within a batch file or application.
 
@@ -196,10 +198,24 @@ To make a full database backup, the command syntax is:
 nbackup [-U _user_ -P _password_] -B 0 _database_ [_backupfile_]
 ----
 
-For instance:
+For instance, assuming the database is located in  `C:\Data`, and  `nbackup.exe ` is in the search of path Windows:
+
+[listing,subs=+quotes]
+----
+C:\Data>nbackup -B 0 inventory.fdb inventory_1-Mar-2006.nbk
+----
+
+Or, if Firebird (from version 2.5) is running on non-standard port, in this example, 3051:
+[listing,subs=+quotes]
+----
+C:\Data>nbackup -B 0 localhost/3051:C:\Data\inventory.fdb C:\Data\inventory-level-0-Jul-2020.nbk -user SYSDBA -pass masterkey
+----
 
+In Firebird 3.0 and higher, in a case of successful completing the backup, the nbackup will print the short statistics:
 ----
-C:\Data> nbackup -B 0 inventory.fdb inventory_1-Mar-2006.nbk
+time elapsed    0 sec
+page reads      307
+page writes     307
 ----
 
 [[nbackup-backups-comments]]
@@ -255,6 +271,7 @@ It just gives a rough (and incomplete) impression of what happens under the hood
 
 . First of all, the main database file is locked by changing an internal state flag.
 From this moment on, any and all mutations in the database are written to a temporary file -- the difference file or [term]_delta file_.
+By default, the delta file is created in the same folder as a database file, with the additional extension .delta, for example: MyDatabase.fdb.delta
 . Then the actual backup is made.
 This isn't a straight file copy;
 restoring must be done by nbackup as well.
@@ -332,6 +349,11 @@ C:\Data> nbackup -B 1 inventory.fdb inventory_3-Mar-2006.nbk
 
 This one contains the mutations of the last _two_ days, since the full backup, not only those since the previous level-1 backup.
 
+[NOTE]
+====
+The previous incremental backup of any level must be completed before the start of the next incremental backup, otherwise `nbackup` execution will not do the desired backup, and return error "Database is already in the physical backup mode". 
+====
+
 A couple of hours on we go for a level-2 backup:
 
 ----
@@ -340,6 +362,14 @@ C:\Data> nbackup -B 2 inventory.fdb inventory_3-Mar-2006_2.nbk
 
 This youngest backup only contains the mutations since the most recent level-1 backup, that is: of the last few hours.
 
+[[nbackup-performance-notice]]
+==== Important performance notice
+
+Before Firebird 3.0, all incremental backups (i.e., level 1, 2, etc) will read the whole database file to locate the changes and store them into a backup file.
+It can affect the database performance in a case of big databases (100Gb+) and slow disk subsystem (usually, slowness can be noticed on non-SSD drives). 
+Starting from Firebird 3.0, only the changed portion of the database file is read, so incremental backups level 1 and higher are much faster, and give the less impact on the database performance.
+
+
 [NOTE]
 ====
 All the <<nbackup-backups-comments,comments>> that have been made about full backups also apply to incremental backups.
@@ -536,6 +566,7 @@ If you do have sufficient rights, then it's more practical to use nbackup itself
 * Specifying `-nbk_no_triggers` or `-nbk_direct` with `-action_nrest` leads to an error message.
 Nbackup itself is more lenient here: it simply ignores the `-T` and `-D` parameters if they are used in the wrong context.
 * Instead of a database filename you may also use an alias.
+* Database path (or alias) length is limited to 255 characters.
 
 [[nbackup-backups-pract]]
 === A practical application
@@ -705,6 +736,22 @@ This parameter is only valid in combination with `-L` and when it is present, nb
 Because the size is given in pages, it has to be multiplied by the database page size in order to get the actual number of bytes to be copied.
 Or, if you use the `dd` copy utility, you could specify the page size as `(i)bs` and the output of `nbackup -L -S` as `count`.
 
+[[nbackup-lock-virtual-machine]]
+=== Locking/Unlocking database in case of Virtual Machines Backup
+
+Using Virtual Machines backup tools without preparing database for such type of backup can lead to the corrupted (i.e., useless) backup copy.
+
+Firebird server intensively uses its own cache in RAM to speed up operations, and implements complex techniques to ensure database consistency at the every given moment.
+Virtual Machine backup tools are not aware about Firebird's cache, and usually they do not consider database files as random-access files.
+
+As a result, when the virtual machine backup is done, the database file inside it will have the state as after a hard reset of VM, and very often such copy is not consistent (i.e., corrupted). 
+The chance of such problem is higher when many active users are changing the database, or if there is active sweep process.
+
+Such inconsistent backups can occur in any virtualized environment, including public clouds.
+
+In order to create good Firebird database backup with VM backup tool, it is necessary to lock database file with nbackup before the VM backup, and unlock after it. 
+Usually VM backup tool allows executing custom pre- and post-backup scripts, where you can lock/unlock Firebird databases.
+
 [[nbackup-deltafile]]
 == Setting the delta file
 
@@ -977,8 +1024,13 @@ In second table: self-restore -> user restore.
 |27 Jun 2020
 |MR
 |Conversion to AsciiDoc, minor copy-editing
-|===
 
+
+|1.6
+|25 Jul 2020
+|AK
+|Paragraph about VM backups, example with non-standard port, performance notice paragraph on incremental backup
+|===
 :sectnums:
 
 :sectnums!:
@@ -998,6 +1050,6 @@ Copyright (C) 2005–2020.
 All Rights Reserved.
 Initial Writer contact: <firstname> at <lastname> dot nl.
 
-Contributor(s): Mark Rotteveel
+Contributor(s): Mark Rotteveel, Alexey Kovyazin
 
-:sectnums:
+:sectnums: