poky: subtree update:745e38ff0f..81f9e815d3

Adrian Bunk (6):
      openssl: Upgrade 1.1.1c -> 1.1.1d
      glib-2.0: Upgrade 2.60.6 -> 2.60.7
      lttng-modules: Upgrade 2.10.10 -> 2.10.11
      lttng-ust: Upgrade 2.10.4 -> 2.10.5
      squashfs-tools: Remove UPSTREAM_CHECK_COMMITS
      libmpc: Remove dead UPSTREAM_CHECK_URI

Alexander Kanavin (2):
      runqemu: decouple gtk and gl options
      strace: add a timeout for running ptests

Alistair Francis (1):
      gdb: Mark gdbserver as ALLOW_EMPTY for riscv32

Andre McCurdy (9):
      busybox: drop unused mount.busybox and umount.busybox wrappers
      busybox: drop inittab from SRC_URI ( now moved to busybox-inittab )
      busybox-inittab: minor formatting tweaks
      base-files: drop legacy empty file /etc/default/usbd
      busybox: rcS and rcK should not be writeable by everyone
      ffmpeg: add PACKAGECONFIG controls for alsa and zlib (enable by default)
      libwebp: apply ARM specific config options to big endian ARM
      initscripts: enable alignment.sh init script for big endian ARM
      libunwind: apply configure over-ride to both big and little endian ARM

Andrew F. Davis (4):
      libepoxy: Disable x11 when not building for x11
      cogl: Set depends to the virtual needed not explicitly on Mesa
      gtk+3: Set depends to the virtual needed not explicitly on Mesa
      weston: Set depends to the virtual needed not explicitly on Mesa

Armin Kuster (1):
      gcc: Security fix for CVE-2019-15847

Changhyeok Bae (1):
      iw: upgrade to 5.3

Changqing Li (2):
      classextend.py: don't extend file for file dependency
      report-error.bbclass: add local.conf/auto.conf into error report

Chen Qi (1):
      python-numpy: fix build for libn32

Daniel Gomez (1):
      lttng-modules: Add missing SRCREV_FORMAT

Diego Rondini (1):
      initramfs-framework: support PARTLABEL option

Dmitry Eremin-Solenikov (7):
      image-uefi.conf: add config file holding configuration for UEFI images
      grub-bootconf: switch to image-uefi.conf
      grub-efi: switch to image-uefi.conf
      grub-efi.bbclass: switch to image-uefi.conf
      systemd-boot: switch to image-uefi.conf
      systemd-boot.bbclass: switch to image-uefi.conf
      live-vm-common.bbclass: provide efi population functions for live images

Hector Palacios (1):
      udev-extraconf: skip mounting partitions already mounted by systemd

Henning Schild (6):
      oe-git-proxy: allow setting SOCAT from outside
      oeqa: add case for oe-git-proxy
      Revert "oe-git-proxy: Avoid resolving NO_PROXY against local files"
      oe-git-proxy: disable shell pathname expansion for the whole script
      oe-git-proxy: NO_PROXY suffix matching without wildcard for match_host
      oe-git-proxy: fix dash "Bad substitution"

Hongxu Jia (1):
      elfutils: 0.176 -> 0.177

Jack Mitchell (1):
      iptables: add systemd helper unit to load/restore rules

Jaewon Lee (1):
      populate_sdk_ext: Introduce mechanism to keep nativesdk* sstate in esdk

Jason Wessel (1):
      gnupg: Extend -native wrapper to fix gpgme-native's gpgconf problems

Jiang Lu (2):
      glib-networking:enable glib-networking build as native package
      libsoup:enable libsoup build as native package

Joshua Watt (4):
      sstatesig: Update server URI
      Remove SSTATE_HASHEQUIV_SERVER
      bitbake: bitbake: Rework hash equivalence
      classes/archiver: Fix WORKDIR for shared source

Kai Kang (1):
      systemd: provides ${base_sbindir}/udevadm

Khem Raj (10):
      ptrace: Drop ptrace aid for musl/ppc
      elfutils: Fix build on ppc/musl
      cogl: Do not depend PN-dev on empty PN
      musl: Update to latest master
      glibc: Move DISTRO_FEATURE specific do_install code for target recipe only
      populate_sdk_base.bbclass: nativesdk-glibc-locale is required on musl too
      nativesdk.bbclass: Clear out LIBCEXTENSION and ABIEXTENSION
      openssl: Enable os option for with-rand-seed as well
      weston-init: Add possibility to run weston as non-root user
      layer.conf: Remove weston-conf from SIGGEN_EXCLUDE_SAFE_RECIPE_DEPS

Li Zhou (1):
      qemu: Security Advisory - qemu - CVE-2019-15890

Limeng (1):
      tune-cortexa57-cortexa53: add tunes for ARM Cortex-A53-Cortex-A57

Martin Jansa (2):
      perf: fix build on kernels which don't have ${S}/tools/include/linux/bits.h
      bitbake: Revert "bitbake: cooker: Ensure bbappends are found in stable order"

Maxime Roussin-Bélanger (1):
      meta: add missing descriptions and homepage in bsp

Mikko Rapeli (2):
      busybox.inc: handle empty DEBUG_PREFIX_MAP
      bitbake: svn fetcher: allow "svn propget svn:externals" to fail

Nathan Rossi (7):
      resulttool: Handle multiple series containing ptestresults
      gcc-cross.inc: Process binaries in build dir to be relocatable
      oeqa/core/case.py: Add OEPTestResultTestCase for ptestresult helpers
      oeqa/selftest: Rework toolchain tests to use OEPTestResultTestCase
      glibc-testsuite: SkipRecipe if libc is not glibc
      cmake: 3.15.2 -> 3.15.3
      meson.bbclass: Handle microblaze* mapping to cpu family

Oleksandr Kravchuk (5):
      python3-pygobject: update to 3.34.0
      font-util: update to 1.3.2
      expat: update to 2.2.8
      curl: update to 7.66.0
      python3-dbus: update to 1.2.12

Otavio Salvador (1):
      mesa: Upgrade 19.1.1 -> 19.1.6

Peter Kjellerstedt (3):
      glibc: Make it build without ldconfig in DISTRO_FEATURES
      package_rpm.bbclass: Remove a misleading bb.note()
      tzdata: Correct the packaging of /etc/localtime and /etc/timezone

Quentin Schulz (1):
      externalsrc: stop rebuilds of 2+ externalsrc recipes sharing the same git repo

Randy MacLeod (4):
      valgrind: enable ~500 more ptests
      valgrind: make a few more ptests pass
      valgrind: ptest improvements to run-ptest and more
      valgrind: disable 256 ptests for aarch64

Richard Purdie (8):
      bitbake: runqueue/siggen: Optimise hash equiv queries
      runqemu: Mention snapshot in the help output
      initramfs-framework: support PARTLABEL option
      systemd: Handle slow to boot mips hwdb update timeouts
      meta-extsdk: Either an sstate task is a proper task or it isn't
      oeqa/concurrenttest: Use ionice to delete build directories
      bitbake: utils: Add ionice option to prunedir
      build-appliance-image: Update to master head revision

Robert Yang (2):
      conf/multilib.conf: Add ovmf to NON_MULTILIB_RECIPES
      bitbake: runqueue: validate_hashes(): currentcount should be a number

Ross Burton (16):
      libtasn1: fix build with api-documentation enabled
      gstreamer1.0-libav: enable gtk-doc again
      python3: handle STAGING_LIBDIR/INCDIR being unset
      mesa: no need to depend on target python3
      adwaita-icon-theme: fix rare install race
      oeqa/selftest/wic: improve assert messages in test_fixed_size
      oeqa/selftest/imagefeatures: dump the JSON if it can't be parsed
      libical: upgrade to 3.0.6
      acpica: upgrade 20190509 -> 20190816
      gdk-pixbuf: upgrade 2.38.1 -> 2.38.2
      piglit: upgrade to latest revision
      libinput: upgrade 1.14.0 -> 1.14.1
      rootfs-postcommands: check /etc/gconf exists before working on it
      systemd-systemctl-native: don't care about line endings
      opkg-utils: respect SOURCE_DATE_EPOCH when building ipkgs
      bitbake: fetch2/git: add git-lfs toggle option

Scott Murray (1):
      systemd: upgrade to 243

Stefan Ghinea (1):
      ghostscript: CVE-2019-14811, CVE-2019-14817

Tim Blechmann (1):
      icecc: blacklist pixman

Yeoh Ee Peng (3):
      bitbake: bitbake-layers: show-recipes: Show recipes only
      bitbake: bitbake-layers: show-recipes: Select recipes from selected layer
      bitbake: bitbake-layers: show-recipes: Enable bare output

Yi Zhao (3):
      screen: add /etc/screenrc as global config file
      nfs-utils: fix nfs mount error on 32bit nfs server
      grub: remove diffutils and freetype runtime dependencies

Zang Ruochen (2):
      btrfs-tools:upgrade 5.2.1 -> 5.2.2
      timezone:upgrade 2019b -> 2019c

Change-Id: I1ec24480a8964e474cd99d60a0cb0975e49b46b8
Signed-off-by: Brad Bishop <bradleyb@fuzziesquirrel.com>

poky/bitbake/bin/bitbake-hashclient

@@ -0,0 +1,170 @@
+#! /usr/bin/env python3
+#
+# Copyright (C) 2019 Garmin Ltd.
+#
+# SPDX-License-Identifier: GPL-2.0-only
+#
+
+import argparse
+import hashlib
+import logging
+import os
+import pprint
+import sys
+import threading
+import time
+
+try:
+    import tqdm
+    ProgressBar = tqdm.tqdm
+except ImportError:
+    class ProgressBar(object):
+        def __init__(self, *args, **kwargs):
+            pass
+
+        def __enter__(self):
+            return self
+
+        def __exit__(self, *args, **kwargs):
+            pass
+
+        def update(self):
+            pass
+
+sys.path.insert(0, os.path.join(os.path.dirname(os.path.dirname(__file__)), 'lib'))
+
+import hashserv
+
+DEFAULT_ADDRESS = 'unix://./hashserve.sock'
+METHOD = 'stress.test.method'
+
+
+def main():
+    def handle_stats(args, client):
+        if args.reset:
+            s = client.reset_stats()
+        else:
+            s = client.get_stats()
+        pprint.pprint(s)
+        return 0
+
+    def handle_stress(args, client):
+        def thread_main(pbar, lock):
+            nonlocal found_hashes
+            nonlocal missed_hashes
+            nonlocal max_time
+
+            client = hashserv.create_client(args.address)
+
+            for i in range(args.requests):
+                taskhash = hashlib.sha256()
+                taskhash.update(args.taskhash_seed.encode('utf-8'))
+                taskhash.update(str(i).encode('utf-8'))
+
+                start_time = time.perf_counter()
+                l = client.get_unihash(METHOD, taskhash.hexdigest())
+                elapsed = time.perf_counter() - start_time
+
+                with lock:
+                    if l:
+                        found_hashes += 1
+                    else:
+                        missed_hashes += 1
+
+                    max_time = max(elapsed, max_time)
+                    pbar.update()
+
+        max_time = 0
+        found_hashes = 0
+        missed_hashes = 0
+        lock = threading.Lock()
+        total_requests = args.clients * args.requests
+        start_time = time.perf_counter()
+        with ProgressBar(total=total_requests) as pbar:
+            threads = [threading.Thread(target=thread_main, args=(pbar, lock), daemon=False) for _ in range(args.clients)]
+            for t in threads:
+                t.start()
+
+            for t in threads:
+                t.join()
+
+        elapsed = time.perf_counter() - start_time
+        with lock:
+            print("%d requests in %.1fs. %.1f requests per second" % (total_requests, elapsed, total_requests / elapsed))
+            print("Average request time %.8fs" % (elapsed / total_requests))
+            print("Max request time was %.8fs" % max_time)
+            print("Found %d hashes, missed %d" % (found_hashes, missed_hashes))
+
+        if args.report:
+            with ProgressBar(total=args.requests) as pbar:
+                for i in range(args.requests):
+                    taskhash = hashlib.sha256()
+                    taskhash.update(args.taskhash_seed.encode('utf-8'))
+                    taskhash.update(str(i).encode('utf-8'))
+
+                    outhash = hashlib.sha256()
+                    outhash.update(args.outhash_seed.encode('utf-8'))
+                    outhash.update(str(i).encode('utf-8'))
+
+                    client.report_unihash(taskhash.hexdigest(), METHOD, outhash.hexdigest(), taskhash.hexdigest())
+
+                    with lock:
+                        pbar.update()
+
+    parser = argparse.ArgumentParser(description='Hash Equivalence Client')
+    parser.add_argument('--address', default=DEFAULT_ADDRESS, help='Server address (default "%(default)s")')
+    parser.add_argument('--log', default='WARNING', help='Set logging level')
+
+    subparsers = parser.add_subparsers()
+
+    stats_parser = subparsers.add_parser('stats', help='Show server stats')
+    stats_parser.add_argument('--reset', action='store_true',
+                              help='Reset server stats')
+    stats_parser.set_defaults(func=handle_stats)
+
+    stress_parser = subparsers.add_parser('stress', help='Run stress test')
+    stress_parser.add_argument('--clients', type=int, default=10,
+                               help='Number of simultaneous clients')
+    stress_parser.add_argument('--requests', type=int, default=1000,
+                               help='Number of requests each client will perform')
+    stress_parser.add_argument('--report', action='store_true',
+                               help='Report new hashes')
+    stress_parser.add_argument('--taskhash-seed', default='',
+                               help='Include string in taskhash')
+    stress_parser.add_argument('--outhash-seed', default='',
+                               help='Include string in outhash')
+    stress_parser.set_defaults(func=handle_stress)
+
+    args = parser.parse_args()
+
+    logger = logging.getLogger('hashserv')
+
+    level = getattr(logging, args.log.upper(), None)
+    if not isinstance(level, int):
+        raise ValueError('Invalid log level: %s' % args.log)
+
+    logger.setLevel(level)
+    console = logging.StreamHandler()
+    console.setLevel(level)
+    logger.addHandler(console)
+
+    func = getattr(args, 'func', None)
+    if func:
+        client = hashserv.create_client(args.address)
+        # Try to establish a connection to the server now to detect failures
+        # early
+        client.connect()
+
+        return func(args, client)
+
+    return 0
+
+
+if __name__ == '__main__':
+    try:
+        ret = main()
+    except Exception:
+        ret = 1
+        import traceback
+        traceback.print_exc()
+    sys.exit(ret)

poky/bitbake/bin/bitbake-hashserv

@@ -11,20 +11,26 @@ import logging
 import argparse
 import sqlite3
 
-sys.path.insert(0, os.path.join(os.path.dirname(os.path.dirname(__file__)),'lib'))
+sys.path.insert(0, os.path.join(os.path.dirname(os.path.dirname(__file__)), 'lib'))
 
 import hashserv
 
 VERSION = "1.0.0"
 
-DEFAULT_HOST = ''
-DEFAULT_PORT = 8686
+DEFAULT_BIND = 'unix://./hashserve.sock'
+
 
 def main():
-    parser = argparse.ArgumentParser(description='HTTP Equivalence Reference Server. Version=%s' % VERSION)
-    parser.add_argument('--address', default=DEFAULT_HOST, help='Bind address (default "%(default)s")')
-    parser.add_argument('--port', type=int, default=DEFAULT_PORT, help='Bind port (default %(default)d)')
-    parser.add_argument('--prefix', default='', help='HTTP path prefix (default "%(default)s")')
+    parser = argparse.ArgumentParser(description='Hash Equivalence Reference Server. Version=%s' % VERSION,
+                                     epilog='''The bind address is the path to a unix domain socket if it is
+                                               prefixed with "unix://". Otherwise, it is an IP address
+                                               and port in form ADDRESS:PORT. To bind to all addresses, leave
+                                               the ADDRESS empty, e.g. "--bind :8686". To bind to a specific
+                                               IPv6 address, enclose the address in "[]", e.g.
+                                               "--bind [::1]:8686"'''
+                                     )
+
+    parser.add_argument('--bind', default=DEFAULT_BIND, help='Bind address (default "%(default)s")')
     parser.add_argument('--database', default='./hashserv.db', help='Database file (default "%(default)s")')
     parser.add_argument('--log', default='WARNING', help='Set logging level')
 
@@ -41,10 +47,11 @@ def main():
     console.setLevel(level)
     logger.addHandler(console)
 
-    server = hashserv.create_server((args.address, args.port), args.database, args.prefix)
+    server = hashserv.create_server(args.bind, args.database)
     server.serve_forever()
     return 0
 
+
 if __name__ == '__main__':
     try:
         ret = main()
@@ -53,4 +60,3 @@ if __name__ == '__main__':
         import traceback
         traceback.print_exc()
     sys.exit(ret)
-

poky/bitbake/bin/bitbake-worker

@@ -418,7 +418,7 @@ class BitbakeWorker(object):
         bb.msg.loggerDefaultDomains = self.workerdata["logdefaultdomain"]
         for mc in self.databuilder.mcdata:
             self.databuilder.mcdata[mc].setVar("PRSERV_HOST", self.workerdata["prhost"])
-            self.databuilder.mcdata[mc].setVar("BB_HASHSERVE", self.workerdata["hashservport"])
+            self.databuilder.mcdata[mc].setVar("BB_HASHSERVE", self.workerdata["hashservaddr"])
 
     def handle_newtaskhashes(self, data):
         self.workerdata["newhashes"] = pickle.loads(data)

poky/bitbake/lib/bb/cooker.py

@@ -194,7 +194,7 @@ def __init__(self, configuration, featureSet=None):
 
         self.ui_cmdline = None
         self.hashserv = None
-        self.hashservport = None
+        self.hashservaddr = None
 
         self.initConfigurationData()
 
@@ -392,19 +392,20 @@ def handlePRServ(self):
         except prserv.serv.PRServiceConfigError as e:
             bb.fatal("Unable to start PR Server, exitting")
 
-        if self.data.getVar("BB_HASHSERVE") == "localhost:0":
+        if self.data.getVar("BB_HASHSERVE") == "auto":
+            # Create a new hash server bound to a unix domain socket
             if not self.hashserv:
                 dbfile = (self.data.getVar("PERSISTENT_DIR") or self.data.getVar("CACHE")) + "/hashserv.db"
-                self.hashserv = hashserv.create_server(('localhost', 0), dbfile, '')
-                self.hashservport = "localhost:" + str(self.hashserv.server_port)
+                self.hashservaddr = "unix://%s/hashserve.sock" % self.data.getVar("TOPDIR")
+                self.hashserv = hashserv.create_server(self.hashservaddr, dbfile, sync=False)
                 self.hashserv.process = multiprocessing.Process(target=self.hashserv.serve_forever)
                 self.hashserv.process.daemon = True
                 self.hashserv.process.start()
-            self.data.setVar("BB_HASHSERVE", self.hashservport)
-            self.databuilder.origdata.setVar("BB_HASHSERVE", self.hashservport)
-            self.databuilder.data.setVar("BB_HASHSERVE", self.hashservport)
+            self.data.setVar("BB_HASHSERVE", self.hashservaddr)
+            self.databuilder.origdata.setVar("BB_HASHSERVE", self.hashservaddr)
+            self.databuilder.data.setVar("BB_HASHSERVE", self.hashservaddr)
             for mc in self.databuilder.mcdata:
-                self.databuilder.mcdata[mc].setVar("BB_HASHSERVE", self.hashservport)
+                self.databuilder.mcdata[mc].setVar("BB_HASHSERVE", self.hashservaddr)
 
         bb.parse.init_parser(self.data)
 
@@ -1852,7 +1853,6 @@ def get_file_appends(self, fn):
             (bbappend, filename) = b
             if (bbappend == f) or ('%' in bbappend and bbappend.startswith(f[:bbappend.index('%')])):
                 filelist.append(filename)
-        filelist.sort()
         return filelist
 
     def collection_priorities(self, pkgfns, d):

poky/bitbake/lib/bb/fetch2/git.py

@@ -464,6 +464,8 @@ def unpack(self, ud, destdir, d):
         if os.path.exists(destdir):
             bb.utils.prunedir(destdir)
 
+        need_lfs = ud.parm.get("lfs", "1") == "1"
+
         source_found = False
         source_error = []
 
@@ -493,14 +495,16 @@ def unpack(self, ud, destdir, d):
         runfetchcmd("%s remote set-url origin %s" % (ud.basecmd, repourl), d, workdir=destdir)
 
         if self._contains_lfs(ud, d, destdir):
-            path = d.getVar('PATH')
-            if path:
-                gitlfstool = bb.utils.which(path, "git-lfs", executable=True)
-                if not gitlfstool:
-                    raise bb.fetch2.FetchError("Repository %s has lfs content, install git-lfs plugin on host to download" % (repourl))
+            if need_lfs:
+                path = d.getVar('PATH')
+                if path:
+                    gitlfstool = bb.utils.which(path, "git-lfs", executable=True)
+                    if not gitlfstool:
+                        raise bb.fetch2.FetchError("Repository %s has LFS content, install git-lfs on host to download (or set lfs=0 to ignore it)" % (repourl))
+                else:
+                    bb.note("Could not find 'PATH'")
             else:
-                bb.note("Could not find 'PATH'")
-
+                bb.note("Repository %s has LFS content but it is not being fetched" % (repourl))
 
         if not ud.nocheckout:
             if subdir != "":

poky/bitbake/lib/bb/fetch2/svn.py

@@ -145,7 +145,7 @@ def download(self, ud, d):
 
             if not ("externals" in ud.parm and ud.parm["externals"] == "nowarn"):
                 # Warn the user if this had externals (won't catch them all)
-                output = runfetchcmd("svn propget svn:externals", d, workdir=ud.moddir)
+                output = runfetchcmd("svn propget svn:externals || true", d, workdir=ud.moddir)
                 if output:
                     if "--ignore-externals" in svnfetchcmd.split():
                         bb.warn("%s contains svn:externals." % ud.url)

poky/bitbake/lib/bb/runqueue.py

@@ -1173,6 +1173,7 @@ def invalidate_task(tid, error_nostamp):
                     self.prepare_task_hash(tid)
 
         bb.parse.siggen.writeout_file_checksum_cache()
+        bb.parse.siggen.set_setscene_tasks(self.runq_setscene_tids)
 
         #self.dump_data()
         return len(self.runtaskentries)
@@ -1259,7 +1260,7 @@ def _start_worker(self, mc, fakeroot = False, rqexec = None):
             "buildname" : self.cfgData.getVar("BUILDNAME"),
             "date" : self.cfgData.getVar("DATE"),
             "time" : self.cfgData.getVar("TIME"),
-            "hashservport" : self.cooker.hashservport,
+            "hashservaddr" : self.cooker.hashservaddr,
         }
 
         worker.stdin.write(b"<cookerconfig>" + pickle.dumps(self.cooker.configuration) + b"</cookerconfig>")
@@ -1393,7 +1394,7 @@ def get_timestamp(f):
             cache[tid] = iscurrent
         return iscurrent
 
-    def validate_hashes(self, tocheck, data, currentcount=None, siginfo=False):
+    def validate_hashes(self, tocheck, data, currentcount=0, siginfo=False):
         valid = set()
         if self.hashvalidate:
             sq_data = {}
@@ -1600,7 +1601,7 @@ def print_diffscenetasks(self):
 
             tocheck.add(tid)
 
-        valid_new = self.validate_hashes(tocheck, self.cooker.data, None, True)
+        valid_new = self.validate_hashes(tocheck, self.cooker.data, 0, True)
 
         # Tasks which are both setscene and noexec never care about dependencies
         # We therefore find tasks which are setscene and noexec and mark their
@@ -1981,7 +1982,7 @@ def execute(self):
                             continue
                         logger.debug(1, "Task %s no longer deferred" % nexttask)
                         del self.sq_deferred[nexttask]
-                        valid = self.rq.validate_hashes(set([nexttask]), self.cooker.data, None, False)
+                        valid = self.rq.validate_hashes(set([nexttask]), self.cooker.data, 0, False)
                         if not valid:
                             logger.debug(1, "%s didn't become valid, skipping setscene" % nexttask)
                             self.sq_task_failoutright(nexttask)
@@ -2173,7 +2174,7 @@ def filtermcdeps(self, task, mc, deps):
             ret.add(dep)
         return ret
 
-    # We filter out multiconfig dependencies from taskdepdata we pass to the tasks 
+    # We filter out multiconfig dependencies from taskdepdata we pass to the tasks
     # as most code can't handle them
     def build_taskdepdata(self, task):
         taskdepdata = {}