Release notes from Impala

cdh6.3.3-release: IMPALA-8960: Fix test_owner_privileges::test_drop_if_exists on S3

2020-01-15T15:40:42Z

hdfs:// URIs do not work on Impala's S3 tests. The fix is to drop
the hdfs:// part of the URI. Our tests run against the default
filesystem, so dropping the hdfs:// works on both HDFS test setups
and S3 test setups.

Testing:

Ran tests on S3
Ran test_owner_privileges.py on HDFS locally

Change-Id: Id7be961094f03c5fde3f8b4c340814d26470818a
Reviewed-on: http://gerrit.cloudera.org:8080/14334
Reviewed-by: Michael Ho kwho@cloudera.com
Tested-by: Impala Public Jenkins impala-public-jenkins@cloudera.com
Reviewed-on: https://gerrit.sjc.cloudera.com/c/cdh/impala/+/66469
API-Compat: Gabor Solya gsolya@cloudera.com
CDH-Build: Gabor Solya gsolya@cloudera.com
Reviewed-by: Gabor Solya gsolya@cloudera.com
Quasar-L0: Gabor Solya gsolya@cloudera.com
Tested-by: Gabor Solya gsolya@cloudera.com

cdh6.3.2-release: CLOUDERA-BUILD. Preparing for 6.3.2 release.

2019-11-06T21:14:57Z

Commit performed on https://master-01.jenkins.cloudera.com/

Signed-off-by: cloudera cauldron-dev@cloudera.com

cdh6.3.1-release: CLOUDERA-BUILD. Preparing for 6.3.1 release.

2019-09-18T09:43:30Z

Commit performed on https://master-01.jenkins.cloudera.com/

Signed-off-by: cloudera cauldron-dev@cloudera.com

cdh6.2.1-release: CLOUDERA-BUILD. Preparing for 6.2.1 release.

2019-08-28T06:56:33Z

Commit performed on https://master-01.jenkins.cloudera.com/

Signed-off-by: cloudera cauldron-dev@cloudera.com

cdh6.3.0-release: CLOUDERA-BUILD. Preparing for 6.3.0 release.

2019-06-26T09:14:06Z

Commit performed on https://master-01.jenkins.cloudera.com/

Signed-off-by: cloudera cauldron-dev@cloudera.com

cdh5.16.2-release: IMPALA-8270: fix MemTracker teardown in FeSupport

2019-04-24T18:45:08Z

This patch tries to simplify and standardise the order
in which control structures are torn down. As a
consequence the bug is fixed. I've described the bug
below.

The changes are:

Make more control structures owned directly by
QueryState::obj_pool_, so that they are all
destroyed at the same time via ~QueryState.
Tear down local_query_state_ explicitly before
other destructors run.

Either change is sufficient to fix the bug, but
I preferred to do both to reduce the chances
of similar bugs in future.

Description of bug:

In the normal query execution flow:

RuntimeState is in QueryState::obj_pool_
RuntimeState owns RuntimeState::instance_mem_tracker_ via scoped_ptr
QueryState::query_mem_tracker_ is in QueryState::obj_pool_
QueryState::query_mem_tracker_ has a reference to
RuntimeState::instance_mem_tracker_
The tear-down works because ~QueryState unregisters query_mem_tracker_
from its parent, making the whole subtree unreachable before destroying
QueryState::obj_pool_. It is thus safe to destroy instance_mem_tracker_
along with the rest of obj_pool_.

FeSupport messes this up by having RuntimeState own the QueryState
RuntimeState::local_query_state_ via a scoped_ptr, and the implied
destructor order means that RuntimeState::instance_mem_tracker_ is
destroyed before RuntimeState::local_query_state_, which breaks the
above flow and the destroyed instance_mem_tracker_ is reachable from
the process MemTracker via QueryState::query_mem_tracker_ for a
small window until it is unregistered.

Testing:
Added a backend test that reproduced the ASAN use-after-free
failure when run against unmodified RuntimeState code. I did
not make it a unified backend test so that it would be easier
to backport this fix to older versions that don't have unified
tests.

Change-Id: If815130cd4db00917746f10b28514f779ee254f0
Reviewed-on: http://gerrit.cloudera.org:8080/13057
Reviewed-by: Tim Armstrong tarmstrong@cloudera.com
Reviewed-by: Joe McDonnell joemcdonnell@cloudera.com
Reviewed-by: Bikramjeet Vig bikramjeet.vig@cloudera.com
Tested-by: Impala Public Jenkins impala-public-jenkins@cloudera.com
Reviewed-on: https://gerrit.sjc.cloudera.com/c/Impala/+/44586
Tested-by: Tim Armstrong tarmstrong@cloudera.com
(cherry picked from commit ec0c0050b390c3e78ca0b9c7052e6e25d227509a)
Reviewed-on: https://gerrit.sjc.cloudera.com/c/Impala/+/44664

cdh6.2.0-release: IMPALA-7972 Detect self-events to avoid unnecessary invalidates

2019-03-12T04:56:42Z

This patch adds support to detect self-generated events from catalog.
This is used to avoid unnecessary invalidates to the tables from such
self-events. Currently, alter_table, alter_partition, add_partition and
drop_partition event types can invalidate the table metadata.

Originally, we planned to have a global version number support from
metastore (see HIVE-21115). But since that is still not complete, we
rely on a combination of other identifiers to determine if a event is
self-generated or not. These self-event identifiers consists of values
from the table/partition parameters. A catalog service uuid
and the catalog version number. The uuid is generated for each
catalogservice when it comes up and it adds it to the table/partition
parameters with the key "impala.CatalogServiceId". The catalog version
number is added with the key "impala.CatalogVersion".

When catalog executes a DDL operation it appends the current catalog
version to the list of version numbers for the in-flight events for the
table. Events processor clears this version when the corresponding
version number identified by serviceId is received in the event. This is
needed since it is possible that a external non-Impala system which
generates the event presents the same serviceId and version number later
on. The algorithm to detect a self-event is as below.

Add the service id and expected catalog version to table/partition
parameters when executing the DDL operation. When the HMS operation is
successful, add the version number to the list of version for in-flight
events at table level.
When the event is received, the first time you see the combination of
serviceId and version number, event processor clears the version number
from table's list and determines the event as self-generated (and hence
ignored)
If the event data presents a unknown serviceId or if the version
number is not present in the list of in-flight versions, event is not a
self-event and needs to be processed.

In order to limit the total memory footprint, only 10 version numbers
are stored at the table. Since the event processor is expected to poll
every few seconds this should be a reasonable bound which satisfies most
use-cases. Otherwise, event processor may wrongly process a self-event
to invalidate the table. In such a case, its a performance penalty not a
correctness issue.

In case of drop_partition event, the partition object is not available
in the event. Hence we cannot determine if its a self-event. In such
cases currently we always issue a invalidate command. This is a known
limitation and will be improved in IMPALA-7973

Patch adds new tests to trigger alter table/partition DDLs from impala
and makes sure that the table is not invalidated.

Change-Id: I29e487b903ba7493f90575c26c44a0019ff0362e
Reviewed-on: https://gerrit.sjc.cloudera.com/42671
Tested-by: Jenkins User jenkins@cloudera.com
CDH-Build: Jenkins User jenkins@cloudera.com
Quasar-L0: Jenkins User jenkins@cloudera.com
Reviewed-by: Vihang Karajgaonkar vihang@cloudera.com

cdh6.1.1-release: IMPALA-7943: Bump the default client timeout set on impala-shell

2019-01-29T14:05:55Z

As part of IMPALA-7555, we added a default socket timeout of 5 seconds
when connecting to an impalad. Under heavy load with kerberos and SSL
enabled, we could hit this default timeout. This change bumps up the
timeout to 60 secs to make the impala-shell more robust.

Change-Id: Ifc40069e86cbf93634320804efba003fb5551afe
Reviewed-on: http://gerrit.cloudera.org:8080/12051
Reviewed-by: Impala Public Jenkins impala-public-jenkins@cloudera.com
Reviewed-by: Lars Volker lv@cloudera.com
Tested-by: Impala Public Jenkins impala-public-jenkins@cloudera.com
(cherry picked from commit c2627ba)
Reviewed-on: https://gerrit.sjc.cloudera.com/41169
Tested-by: Jenkins User jenkins@cloudera.com
CDH-Build: Jenkins User jenkins@cloudera.com
Quasar-L0: Laszlo Gaal laszlo.gaal@cloudera.com
Reviewed-by: Laszlo Gaal laszlo.gaal@cloudera.com
(cherry picked from commit 3794d20f1ef24435b575aba56bda2023cee60fa4)
Reviewed-on: https://gerrit.sjc.cloudera.com/41201
Quasar-L0: Jenkins User jenkins@cloudera.com

cdh5.16.1-release: Updated save-version.sh for CDH5.16.1

2018-11-20T18:00:39Z

Change-Id: Iddcd9cddc6c3529ebb4442fd4cd7ff30ed512a83
Reviewed-on: https://gerrit.sjc.cloudera.com/39721
Reviewed-by: Bernadett Balogh bbalogh@cloudera.com
Tested-by: Bernadett Balogh bbalogh@cloudera.com

cdh5.15.2-release

2018-11-13T09:32:41Z

Update thirdparty dependencies