HACKING

Information for GNU Gnulib maintainers and contributors
*******************************************************

Using git
=========

* We don't use topic branches.  Changes are usually small enough that
  they can be committed directly to the master branch, after appropriate
  testing.

* We maintain stable branches, though, as described in the documentation:
  https://www.gnu.org/software/gnulib/manual/html_node/Stable-Branches.html
  When backporting a commit to a stable branch of the last year, be sure
  to update the copyright year of each modified file (since we don't run
  "make update-copyright" on the stable branches).

* We use a linear git history — no merges. To work in this setting, it's
  recommended that you configure git with 'git config pull.rebase = true'.

* Before pushing a commit, it is highly recommended that you review it in
  its entirety. The easiest ways to do so are
    * to run
        $ git format-patch -1
      and then read the patch in an editor that has syntax-colouring of patch
      files, or
    * to run
        $ gitk

* We update the ChangeLog by hand.  The commit message is usually identical
  to the ChangeLog entry, with the date and author line removed, with
  the leading tabs removed, and with a blank line after the commit's
  summary line.
  In order to work efficiently with ChangeLog files, it is recommended that
  you configure git to use the git-merge-changelog driver; see the instructions
  in the lib/git-merge-changelog.c file.
  Note: This driver reasonably keeps the ChangeLog entries together; however,
  it does not always keep them in the order you would desire. For example,
  when you had prepared a commit, you try to "git push" it but that fails due
  to someone else's commit that came earlier, what you need to do is:
    1. $ git pull
    2. Verify that your ChangeLog entry is still the top-most one.
    3. If it is not, then edit ChangeLog to move it to the top, and
       $ git commit --amend ChangeLog
    4. $ gitk
    5. $ git push

* When you commit a contributor's patch, please
  - add a reasonable ChangeLog entry in the usual style (meaningful
    summary line and detailed change list),
  - if the contribution is so small that it does not require a
    copyright assignment (cf.
    https://www.gnu.org/prep/maintain/html_node/Legally-Significant.html )
    add a line:
    Copyright-paperwork-exempt: Yes
  - use the 'git commit' option --author="Contributor Name <email@address>"


License Notices
===============

In *.m4 files, use a notice like this:
  dnl Copyright (C) YEARS Free Software Foundation, Inc.
  dnl This file is free software; the Free Software Foundation
  dnl gives unlimited permission to copy and/or distribute it,
  dnl with or without modifications, as long as this notice is preserved.

In lib/, tests/, build-aux/ files, except those that are shared with glibc,
use the license notices from etc/license-notices/ . This avoids gratuitous
differences in wording, as well misunderstandings when a license notice
would say "This program ...".


Test Suite
==========

When adding a module, add a unit test module as well.  This is our best
chance to catch portability problems.

A unit test can have many sub-tests. Try to make the sub-tests independent
of each other, so that it becomes easy to disable some sub-tests by enclosing
them in #if 0 ... #endif.

The main() function's exit code meaning is:
  - 0: PASS
  - 77: SKIP; you should print the reason why the test is skipped.
  - 99: ERROR, i.e. test framework error
  - any other exit code < 126: FAIL

In tests that #include "macros.h" and use the ASSERT macro:
The main() function should, before it returns 0 (for PASS) or 77 (for SKIP)
test the value of test_exit_status and return that instead. So:
  - not
        return 0;
    but instead
        return test_exit_status;
  - not
        return result; // where result can be 0 or 1
    but instead
        return (result ? result : test_exit_status);
  - not
        fputs ("Skipping test: <reason>\n", stderr);
        return 77;
    but instead
        if (test_exit_status != EXIT_SUCCESS)
          return test_exit_status;
        fputs ("Skipping test: <reason>\n", stderr);
        return 77;
    Only at the beginning of the main() function, when ASSERT has not yet been
    invoked, we know that test_exit_status must be zero and can therefore write
        fputs ("Skipping test: <reason>\n", stderr);
        return 77;
    directly.


Maintaining high quality
========================

It is a good idea to occasionally create a testdir of all of Gnulib:
  $ rm -rf ../testdir-all; ./gnulib-tool --create-testdir --dir=../testdir-all --with-c++-tests --without-privileged-tests --single-configure `./all-modules`
and test this directory on various platforms:
  - Linux/glibc systems,
  - Linux/musl systems,
  - macOS,
  - FreeBSD,
  - NetBSD,
  - OpenBSD,
  - AIX,
  - Solaris 10 and 11,
  - Cygwin,
  - Haiku,
  - Android,
  - and other platforms of your choice.

There are two continuous integrations that regularly perform this testing:
* On a Linux/glibc system only:
  https://gitlab.com/gnulib/gnulib-ci
  This one will catch only the most blatant mistakes.
* On many platforms:
  https://github.com/gnu-gnulib/ci-testdir-check/actions
  This one runs on many platforms, currently (as of June 2024):
  - Ubuntu GNU/Linux 22.04
  - CentOS GNU/Linux 7
  - Alpine Linux
  - macOS 11, 12, 13 (all x86_64)
  - macOS 14 (arm64)
  - FreeBSD 14.0
  - NetBSD 10.0
  - OpenBSD 7.5
  - Solaris 11.4
  - Solaris 11 OmniOS
  - Cygwin 3.3.6 (32 bit) and 3.5.3 (64 bit)
  - mingw (32 bit and 64 bit)
  - MSVC (32 bit and 64 bit)
  and also
  - on Ubuntu GNU/Linux 22.04 with clang's UBSAN and ASAN sanitizers.
  This one catches real portability problems.
  Note that the following platforms are not covered and thus still require
  occasional manual testing:
  - AIX
  - Solaris 10
  - Haiku
  - Android
  - and other platforms of your choice.


Warning Options
===============

For packages that use Gnulib, we recommend to use the 'warnings' or
'manywarnings' module, as documented in
https://www.gnu.org/software/gnulib/manual/html_node/warnings.html
https://www.gnu.org/software/gnulib/manual/html_node/manywarnings.html

When building Gnulib testdirs, e.g. when preparing a Gnulib patch,
there are three possible approaches:

* The simplest approach, which warns about the most common mistakes, is to
  use GCC's -Wall option, both for C and C++ compilation units. Just set
    $ ./configure CPPFLAGS="-Wall"
    $ make
  You should generally fix all compiler warnings that you see from this
  approach.

* If you are developing on a glibc system and have GCC version 13 binaries
  available, here's a recipe that will find more mistakes, but is nearly
  as easy to use. Here, different warning options are needed for C and
  for C++:
    $ WARN_GCC13=`echo '
      -fanalyzer
      -Wall
      -Warith-conversion
      -Wcast-align=strict
      -Wdate-time
      -Wdisabled-optimization
      -Wduplicated-cond
      -Wextra
      -Wformat-signedness
      -Winit-self
      -Winvalid-pch
      -Wlogical-op
      -Wmissing-include-dirs
      -Wopenmp-simd
      -Woverlength-strings
      -Wpacked
      -Wpointer-arith
      -Wstrict-overflow
      -Wsuggest-final-methods
      -Wsuggest-final-types
      -Wsync-nand
      -Wsystem-headers
      -Wtrampolines
      -Wuninitialized
      -Wunknown-pragmas
      -Wunsafe-loop-optimizations
      -Wvariadic-macros
      -Wvector-operation-performance
      -Wwrite-strings
      -Warray-bounds=2
      -Wattribute-alias=2
      -Wformat-overflow=2
      -Wformat-truncation=2
      -Wshift-overflow=2
      -Wunused-const-variable=2
      -Wvla-larger-than=4031
      -Wno-empty-body
      -Wno-analyzer-allocation-size
      -Wno-analyzer-fd-double-close
      -Wno-analyzer-double-fclose
      -Wno-analyzer-double-free
      -Wno-analyzer-fd-leak
      -Wno-analyzer-fd-use-after-close
      -Wno-analyzer-fd-use-without-check
      -Wno-analyzer-free-of-non-heap
      -Wno-analyzer-malloc-leak
      -Wno-analyzer-mismatching-deallocation
      -Wno-analyzer-null-argument
      -Wno-analyzer-null-dereference
      -Wno-analyzer-out-of-bounds
      -Wno-analyzer-possible-null-argument
      -Wno-analyzer-possible-null-dereference
      -Wno-analyzer-use-after-free
      -Wno-analyzer-use-of-pointer-in-stale-stack-frame
      -Wno-analyzer-use-of-uninitialized-value
      -Wno-analyzer-va-arg-type-mismatch
      -Wno-attribute-warning
      -Wno-cast-align
      -Wno-clobbered
      -Wno-dangling-pointer
      -Wno-format
      -Wno-implicit-fallthrough
      -Wno-maybe-uninitialized
      -Wno-missing-field-initializers
      -Wno-restrict
      -Wno-sign-compare
      -Wno-switch
      -Wno-type-limits
      -Wno-unused-parameter
    ' | tr -d '\n' | sed -e 's/  */ /g'`
    $ WARN_CFLAGS_GCC13="$WARN_GCC13 -Wnested-externs -Wshadow=local -Wno-discarded-qualifiers"
    $ WARN_CXXFLAGS_GCC13="$WARN_GCC13 -Wno-cpp"
    $ ./configure CFLAGS="-O2 -g $WARN_CFLAGS_GCC13" CXXFLAGS="-O2 -g $WARN_CXXFLAGS_GCC13"
    $ make
  You should generally fix all compiler warnings that you see from this
  approach, or report when this approach produced a pointless warning
  (so that we can fix the value of WARN_GCC13 above).

* If you are developing on a glibc system and have GCC version 13 binaries
  available: Here's a recipe that will find even more mistakes, but it
  requires that you are willing to filter out and ignore pointless warnings.
    $ WARN_GCC13=`echo '
      -fanalyzer
      -Wall
      -Warith-conversion
      -Wcast-align=strict
      -Wdate-time
      -Wdisabled-optimization
      -Wduplicated-cond
      -Wextra
      -Wformat-signedness
      -Winit-self
      -Winvalid-pch
      -Wlogical-op
      -Wmissing-include-dirs
      -Wnull-dereference
      -Wopenmp-simd
      -Woverlength-strings
      -Wpacked
      -Wpointer-arith
      -Wstrict-overflow
      -Wsuggest-attribute=format
      -Wsuggest-final-methods
      -Wsuggest-final-types
      -Wsync-nand
      -Wsystem-headers
      -Wtrampolines
      -Wuninitialized
      -Wunknown-pragmas
      -Wunsafe-loop-optimizations
      -Wvariadic-macros
      -Wvector-operation-performance
      -Wwrite-strings
      -Warray-bounds=2
      -Wattribute-alias=2
      -Wformat-overflow=2
      -Wformat=2
      -Wformat-truncation=2
      -Wimplicit-fallthrough=5
      -Wshift-overflow=2
      -Wunused-const-variable=2
      -Wvla-larger-than=4031
      -Wno-empty-body
      -Wno-analyzer-double-fclose
      -Wno-analyzer-double-free
      -Wno-analyzer-free-of-non-heap
      -Wno-analyzer-malloc-leak
      -Wno-analyzer-null-argument
      -Wno-analyzer-null-dereference
      -Wno-analyzer-use-after-free
      -Wno-attribute-warning
      -Wno-cast-align
      -Wno-clobbered
      -Wno-format-nonliteral
      -Wno-sign-compare
      -Wno-type-limits
      -Wno-unused-parameter
    ' | tr -d '\n' | sed -e 's/  */ /g'`
    $ WARN_CFLAGS_GCC13="$WARN_GCC13 -Wnested-externs -Wshadow=local"
    $ WARN_CXXFLAGS_GCC13="$WARN_GCC13 -Wno-cpp"
    $ ./configure CFLAGS="-O2 -g $WARN_CFLAGS_GCC13" CXXFLAGS="-O2 -g $WARN_CXXFLAGS_GCC13"
    $ make
  With this approach, use your own judgement whether to fix warnings
  arising from your new code or not.
  Do *not* submit patches to silence warnings from existing code:
    - For these warnings, often the cure will be worse than the disease.
    - Some of the warnings are false positives. Rather than silencing
      these warnings, we prefer to report them in the GCC bug tracker
      and wait until they are fixed in a future GCC release.

Similarly, for clang version 16 you can use the following recipe, that uses
selected warning options from
https://releases.llvm.org/16.0.0/tools/clang/docs/DiagnosticsReference.html :
  $ WARN_CLANG16=`echo '
    -Wall
    -Wanon-enum-enum-conversion
    -Warc-repeated-use-of-weak
    -Warray-bounds-pointer-arithmetic
    -Warray-parameter
    -Watomic-properties
    -Wbinary-literal
    -Wbit-int-extension
    -Wbitfield-enum-conversion
    -Wbitwise-op-parentheses
    -Wbool-operation
    -Wc++-compat
    -Wc2x-compat
    -Wc2x-extensions
    -Wc99-compat
    -Wc99-designator
    -Wc99-extensions
    -Wcalled-once-parameter
    -Wcast-function-type
    -Wchar-subscripts
    -Wcomment
    -Wcompletion-handler
    -Wcomplex-component-init
    -Wcompound-token-split
    -Wconsumed
    -Wconversion
    -Wcstring-format-directive
    -Wcuda-compat
    -Wdate-time
    -Wdelimited-escape-sequence-extension
    -Wdeprecated
    -Wdeprecated-dynamic-exception-spec
    -Wdeprecated-implementations
    -Wdeprecated-this-capture
    -Wdeprecated-writable-strings
    -Wdirect-ivar-access
    -Wdocumentation
    -Wdocumentation-deprecated-sync
    -Wdocumentation-html
    -Wdocumentation-pedantic
    -Wdocumentation-unknown-command
    -Wdollar-in-identifier-extension
    -Wduplicate-decl-specifier
    -Wduplicate-enum
    -Wduplicate-method-arg
    -Wduplicate-method-match
    -Wdynamic-exception-spec
    -Wembedded-directive
    -Wempty-init-stmt
    -Wempty-translation-unit
    -Wenum-compare-conditional
    -Wenum-conversion
    -Wenum-enum-conversion
    -Wenum-float-conversion
    -Wexit-time-destructors
    -Wexpansion-to-defined
    -Wexplicit-ownership-type
    -Wextra
    -Wextra-semi
    -Wfixed-enum-extension
    -Wflexible-array-extensions
    -Wfloat-overflow-conversion
    -Wfloat-zero-conversion
    -Wfor-loop-analysis
    -Wformat
    -Wformat-pedantic
    -Wformat-type-confusion
    -Wformat=2
    -Wfour-char-constants
    -Wframe-address
    -Wfuse-ld-path
    -Wfuture-attribute-extensions
    -Wgcc-compat
    -Wgnu
    -Wgnu-anonymous-struct
    -Wgnu-auto-type
    -Wgnu-case-range
    -Wgnu-complex-integer
    -Wgnu-compound-literal-initializer
    -Wgnu-conditional-omitted-operand
    -Wgnu-designator
    -Wgnu-empty-initializer
    -Wgnu-empty-struct
    -Wgnu-flexible-array-initializer
    -Wgnu-flexible-array-union-member
    -Wgnu-folding-constant
    -Wgnu-imaginary-constant
    -Wgnu-label-as-value
    -Wgnu-line-marker
    -Wgnu-null-pointer-arithmetic
    -Wgnu-offsetof-extensions
    -Wgnu-pointer-arith
    -Wgnu-redeclared-enum
    -Wgnu-statement-expression
    -Wgnu-statement-expression-from-macro-expansion
    -Wgnu-union-cast
    -Wgnu-zero-line-directive
    -Wgnu-zero-variadic-macro-arguments
    -Wheader-hygiene
    -Widiomatic-parentheses
    -Wignored-qualifiers
    -Wimplicit
    -Wimplicit-fallthrough
    -Wimplicit-fallthrough-per-function
    -Wimplicit-function-declaration
    -Wimplicit-int
    -Wimplicit-retain-self
    -Wimport-preprocessor-directive-pedantic
    -Wincomplete-module
    -Winconsistent-missing-destructor-override
    -Winfinite-recursion
    -Wint-in-bool-context
    -Winvalid-or-nonexistent-directory
    -Winvalid-utf8
    -Wkeyword-macro
    -Wlanguage-extension-token
    -Wlocal-type-template-args
    -Wlogical-op-parentheses
    -Wlong-long
    -Wloop-analysis
    -Wmain
    -Wmax-tokens
    -Wmethod-signatures
    -Wmicrosoft
    -Wmicrosoft-anon-tag
    -Wmicrosoft-charize
    -Wmicrosoft-comment-paste
    -Wmicrosoft-cpp-macro
    -Wmicrosoft-end-of-file
    -Wmicrosoft-enum-value
    -Wmicrosoft-exception-spec
    -Wmicrosoft-fixed-enum
    -Wmicrosoft-flexible-array
    -Wmicrosoft-redeclare-static
    -Wmisleading-indentation
    -Wmismatched-tags
    -Wmissing-braces
    -Wmissing-method-return-type
    -Wmost
    -Wmove
    -Wnested-anon-types
    -Wnewline-eof
    -Wnon-gcc
    -Wnon-modular-include-in-framework-module
    -Wnon-modular-include-in-module
    -Wnon-virtual-dtor
    -Wnonportable-system-include-path
    -Wnull-pointer-arithmetic
    -Wnull-pointer-subtraction
    -Wnullability-extension
    -Wnullable-to-nonnull-conversion
    -Wopenmp
    -Wover-aligned
    -Woverlength-strings
    -Woverloaded-virtual
    -Woverriding-method-mismatch
    -Wpacked
    -Wpacked-non-pod
    -Wparentheses
    -Wpedantic
    -Wpedantic-core-features
    -Wpessimizing-move
    -Wpointer-arith
    -Wpoison-system-directories
    -Wpragma-pack
    -Wpragma-pack-suspicious-include
    -Wpragmas
    -Wpre-c2x-compat
    -Wpre-openmp-51-compat
    -Wprofile-instr-missing
    -Wquoted-include-in-framework-header
    -Wrange-loop-analysis
    -Wrange-loop-bind-reference
    -Wrange-loop-construct
    -Wreceiver-forward-class
    -Wredundant-move
    -Wredundant-parens
    -Wreorder
    -Wreorder-ctor
    -Wreserved-user-defined-literal
    -Wretained-language-linkage
    -Wselector
    -Wselector-type-mismatch
    -Wself-assign
    -Wself-assign-overloaded
    -Wself-move
    -Wsemicolon-before-method-body
    -Wshadow-all
    -Wshadow-field
    -Wshadow-field-in-constructor
    -Wshadow-field-in-constructor-modified
    -Wshadow-uncaptured-local
    -Wshift-sign-overflow
    -Wsigned-enum-bitfield
    -Wsometimes-uninitialized
    -Wsource-uses-openmp
    -Wspir-compat
    -Wstatic-in-inline
    -Wstrict-potentially-direct-selector
    -Wstrict-selector-match
    -Wstring-concatenation
    -Wstring-conversion
    -Wsuggest-destructor-override
    -Wsuggest-override
    -Wsuper-class-method-mismatch
    -Wtautological-bitwise-compare
    -Wtautological-compare
    -Wtautological-constant-in-range-compare
    -Wtautological-overlap-compare
    -Wtautological-type-limit-compare
    -Wtautological-unsigned-char-zero-compare
    -Wtautological-unsigned-enum-zero-compare
    -Wtautological-unsigned-zero-compare
    -Wtautological-value-range-compare
    -Wthread-safety
    -Wthread-safety-analysis
    -Wthread-safety-attributes
    -Wthread-safety-beta
    -Wthread-safety-negative
    -Wthread-safety-precise
    -Wthread-safety-reference
    -Wthread-safety-verbose
    -Wtype-limits
    -Wunaligned-access
    -Wundeclared-selector
    -Wundef-prefix
    -Wundefined-func-template
    -Wundefined-internal-type
    -Wundefined-reinterpret-cast
    -Wunguarded-availability
    -Wuninitialized
    -Wuninitialized-const-reference
    -Wunknown-pragmas
    -Wunnamed-type-template-args
    -Wunneeded-internal-declaration
    -Wunneeded-member-function
    -Wunreachable-code-fallthrough
    -Wunreachable-code-loop-increment
    -Wunsupported-dll-base-class-template
    -Wunused
    -Wunused-but-set-parameter
    -Wunused-but-set-variable
    -Wunused-const-variable
    -Wunused-exception-parameter
    -Wunused-function
    -Wunused-label
    -Wunused-lambda-capture
    -Wunused-local-typedef
    -Wunused-member-function
    -Wunused-private-field
    -Wunused-property-ivar
    -Wunused-template
    -Wunused-variable
    -Wvariadic-macros
    -Wvector-conversion
    -Wweak-template-vtables
    -Wweak-vtables
    -Wzero-length-array
    -Wno-bitwise-instead-of-logical
    -Wno-c11-extensions
    -Wno-cast-function-type-strict
    -Wno-float-conversion
    -Wno-format-nonliteral
    -Wno-gnu-include-next
    -Wno-implicit-float-conversion
    -Wno-implicit-int-conversion
    -Wno-implicit-int-float-conversion
    -Wno-include-next-absolute-path
    -Wno-missing-field-initializers
    -Wno-reserved-macro-identifier
    -Wno-shadow
    -Wno-shorten-64-to-32
    -Wno-sign-compare
    -Wno-sign-conversion
    -Wno-strict-prototypes
    -Wno-switch
    -Wno-unused-parameter
    -Wno-tautological-constant-out-of-range-compare
    -Wno-tautological-type-limit-compare
    -Wno-tautological-unsigned-zero-compare
    -Wno-tautological-value-range-compare
    -Wno-unused-command-line-argument
    -Wno-user-defined-warnings
  ' | tr -d '\n' | sed -e 's/  */ /g'`
  $ ./configure CFLAGS="-O2 -g $WARN_CLANG16" CXXFLAGS="-O2 -g $WARN_CLANG16"
  $ make
Again, use your own judgement to determine whether to fix or ignore a
specific warning.


Information for gnulib-tool maintenance
***************************************

Running the unit tests
======================

The unit tests of gnulib-tool reside in the maint-tools repository, that is a
satellite repository of the main gnulib repository. Instructions how to obtain
it are in https://savannah.gnu.org/git/?group=gnulib .

To run the unit tests of gnulib-tool.sh:
  $ cd maint-tools/gnulib-tool-tests/
  $ GNULIB_TOOL_IMPL=sh ./test-all.sh

To run the unit tests of gnulib-tool.py:
  $ cd maint-tools/gnulib-tool-tests/
  $ GNULIB_TOOL_IMPL=py ./test-all.sh

It is *mandatory* that you run the test suite before pushing any change to
gnulib-tool.sh or gnulib-tool.py! If you fail to do so, and your change contains
a bug, it will start to affect users immediately.


Debugging the Python implementation of gnulib-tool
==================================================

With Eclipse and PyDev as IDE
-----------------------------

* Download and configuration:
  - Eclipse IDE from https://www.eclipse.org/downloads/packages/
    (either the build for Java or for C/C++ should work fine;
    either use the Eclipse Installer program at the top of the page,
    or one of the individual download links below).
  - PyDev from https://www.pydev.org/download.html
    section "Install as Plugin".
    (Don't use LiClipse, since the license costs money.)
  - Follow https://www.pydev.org/manual_101_install.html,
    replacing http://www.pydev.org/updates
    with      https://www.pydev.org/updates
  - Once installed, restart the IDE.
  - Window > Preferences > PyDev > Interpreters > Python Interpreter:
    New > path: /usr/bin/python3
  - Window > Preferences > PyDev > Editor > Code Analysis:
    Others > Redefinition of builtin symbols: Ignore

* Create a project:
  Let GNULIB_DIR be my gnulib checkout.
  - File > New > Project... > PyDev > PyDev Project. Call it 'gnulib-tool'.
    Note that this is a project inside the Eclipse workspace, so far
    unrelated to the GNULIB_DIR.
  - Popup menu > New > Link to Existing Source
    Name: gnulib
    Path: <GNULIB_DIR>

* Create a run configuration:
  - Run > Run Configurations... > Python Run
    Popup menu > New configuration
      Name: gnulib-tool.py
      Main > Project: gnulib-tool
      Main > Main Module: ${workspace_loc:gnulib-tool/gnulib/.gnulib-tool.py}
      Arguments > Program arguments: --help
  - Test it: Run this configuration.

* Create a debug configuration:
  - Run > Debug Configurations... > gnulib-tool.py
    Popup menu > Duplicate
  - In the duplicate, set
    Arguments > Working directory: /tmp/oath-toolkit/lib
    Arguments > Program arguments: --add-import

* Debug it:
  - Open GLImport.py.
  - On the left-hand border of this editor, do "Add breakpoint".
  - Run > Debug Configurations... > pick the duplicate. Press Debug.


Maintaining high quality
========================

There is a continuous integration of gnulib-tool.py that runs the unit tests,
at https://gitlab.com/gnulib/gnulib-tool-ci/ .