KAFKA-18773: Migrate the log4j1 config to log4j 2 for native image and README #18872

FrankYang0529 · 2025-02-12T12:21:45Z

Test steps

Build release gzipped tar ball and use a local server to host it.

> ./gradlew clean releaseTarGz
> python -m http.server

In another terminal, create native image.

> cd docker
> python docker_build_test.py kafka/test --image-tag KAFKA-18773 --image-type native --kafka-url http://<your ip>:8000/core/build/distributions/kafka_2.13-4.1.0-SNAPSHOT.tgz --build

Run the container.

> docker run -p 9092:9092 --name kafka --rm kafka/test:KAFKA-18773

Committer Checklist (excluded from commit message)

Verify design and implementation
Verify test coverage and CI build status
Verify documentation (including upgrade notes)

dajac · 2025-02-13T19:46:00Z

@FrankYang0529 How is it going with this one?

FrankYang0529 · 2025-02-14T15:34:35Z

Hi @dajac, I may need more time on this. Currently, there is some error message like this. I'm still check where the log from. Thanks.

2025-02-13T03:30:07.514257Z main ERROR Unrecognized format specifier [d]
2025-02-13T03:30:07.514271Z main ERROR Unrecognized conversion specifier [d] starting at position 16 in conversion pattern.
2025-02-13T03:30:07.514293Z main ERROR Unrecognized format specifier [thread]
2025-02-13T03:30:07.514302Z main ERROR Unrecognized conversion specifier [thread] starting at position 25 in conversion pattern.
2025-02-13T03:30:07.514318Z main ERROR Unrecognized format specifier [level]
2025-02-13T03:30:07.514331Z main ERROR Unrecognized conversion specifier [level] starting at position 35 in conversion pattern.
2025-02-13T03:30:07.514347Z main ERROR Unrecognized format specifier [logger]
2025-02-13T03:30:07.514361Z main ERROR Unrecognized conversion specifier [logger] starting at position 47 in conversion pattern.
2025-02-13T03:30:07.514376Z main ERROR Unrecognized format specifier [msg]
2025-02-13T03:30:07.514391Z main ERROR Unrecognized conversion specifier [msg] starting at position 54 in conversion pattern.
2025-02-13T03:30:07.514406Z main ERROR Unrecognized format specifier [n]
2025-02-13T03:30:07.514420Z main ERROR Unrecognized conversion specifier [n] starting at position 56 in conversion pattern.

dajac · 2025-02-17T10:26:11Z

@FrankYang0529 Thanks. Do you think that we could merge it this week?

chia7712 · 2025-02-17T18:11:10Z

the failure may be related to apache/logging-log4j2#1539 - I will test the 2.25.0-SNAPSHOT locally.

chia7712 · 2025-02-18T07:26:20Z

ok, I run the native image successfully on my local. There are many issues we need to fix.

@FrankYang0529 Could you please try following steps on your PR.

use following reflect-config.json and resource-config.json. They are generated by Tracing Agent (see https://www.graalvm.org/22.2/reference-manual/native-image/metadata/AutomaticMetadataCollection/#tracing-agent)

resource-config.json
reflect-config.json

remove jacksonModuleScala from dependency. it has issue on native image support FasterXML/jackson-module-scala#621 but we don't use it actually.
fix the bug according to my comment (https://github.com/apache/kafka/pull/18394/files#r1959192770)

Additionally, the storage command has issue and it is handled by #18844 - hence, this PR can't fix native image totally until #18844 gets fixed - however, this PR can adopt the solution temporarily.

Do you think that we could merge it this week?

@dajac yes, I think it can get fixed this week

chia7712 · 2025-02-18T07:50:33Z

docker/native/launch

@@ -42,11 +42,11 @@ result=$(/opt/kafka/kafka.Kafka setup \
      --default-configs-dir /etc/kafka/docker \
      --mounted-configs-dir /mnt/shared/config \
      --final-configs-dir /opt/kafka/config \
-      -Dlog4j.configuration=file:/opt/kafka/config/tools-log4j.properties 2>&1) || \
+      -Dlog4j2.configurationFile=file:/opt/kafka/config/tools-log4j2.yaml 2>&1) || \


/opt/kafka/config/tools-log4j2.yaml is nonexistent in running the setup. Maybe we can just remove it to align the command to jvm dockerfile

FrankYang0529 · 2025-02-18T11:22:02Z

Hi @chia7712, thanks for the review. I also update the document about generating reachability metadata. If we need new files next time, we can follow it to generate metadata.

…d README Signed-off-by: PoAn Yang <payang@apache.org>

chia7712 · 2025-02-18T14:59:46Z

build.gradle

@@ -1072,7 +1072,6 @@ project(':core') {
    implementation libs.argparse4j
    implementation libs.commonsValidator
    implementation libs.jacksonDatabind
-    implementation libs.jacksonModuleScala


for future reader, the dependency was introduced by 47a9871, and b6183a4 rewrite the tool by java so we don't need to use jackson-module-scala anymore.

for another, we will file a PR to fix the license check. The PR #18004 also introduces a license warning.

chia7712

LGTM and I run this PR locally, it pass

The follow-ups are shown below.

storage format should use static voter - will be fixed by KAFKA-18737: KafkaDockerWrapper setup functions fails due to storage format command #18844
native image does not honor the exit code of gpg key check - https://issues.apache.org/jira/browse/KAFKA-18823
add a flag to skip gpg check - will be fixed by KAFKA-18737: KafkaDockerWrapper setup functions fails due to storage format command #18844

frankvicky

I ran this PR on my local machine and also applied to test #18844

chia7712 · 2025-02-18T16:44:18Z

the flaky is traced by https://issues.apache.org/jira/browse/KAFKA-15900

… README (#18872) - update reflection-config.json and resource-config.json to include log4j2 and jackson - remove unused jackson scala library - fix the incorrect path of log4j2.yaml - adopt workaround (--standalone) to make this PR work and it will be fixed by KAFKA-18737) Reviewers: TengYao Chi <kitingiao@gmail.com>, Chia-Ping Tsai <chia7712@gmail.com>

chia7712 · 2025-02-18T16:51:50Z

cherry-pick to 4.0

… README (apache#18872) - update reflection-config.json and resource-config.json to include log4j2 and jackson - remove unused jackson scala library - fix the incorrect path of log4j2.yaml - adopt workaround (--standalone) to make this PR work and it will be fixed by KAFKA-18737) Reviewers: TengYao Chi <kitingiao@gmail.com>, Chia-Ping Tsai <chia7712@gmail.com>

github-actions bot added docker Official Docker image small Small PRs labels Feb 12, 2025

dajac added the Blocker This pull request is identified as solving a blocker for a release. label Feb 14, 2025

chia7712 mentioned this pull request Feb 17, 2025

Add GraalVM native compilation support apache/logging-log4j2#1539

Closed

chia7712 mentioned this pull request Feb 18, 2025

KAFKA-18396: Migrate log4j1 configuration to log4j2 in KafkaDockerWrapper #18394

Merged

3 tasks

chia7712 reviewed Feb 18, 2025

View reviewed changes

FrankYang0529 force-pushed the KAFKA-18773 branch from 0ebb8ff to 498deb6 Compare February 18, 2025 11:17

github-actions bot added core Kafka Broker build Gradle build or GitHub Actions and removed small Small PRs labels Feb 18, 2025

FrankYang0529 changed the title ~~KAFKA-18773: Migrate the log4j1 config to log4j 2 for native image and README (wip)~~ KAFKA-18773: Migrate the log4j1 config to log4j 2 for native image and README Feb 18, 2025

FrankYang0529 marked this pull request as ready for review February 18, 2025 11:20

frankvicky mentioned this pull request Feb 18, 2025

KAFKA-18737: KafkaDockerWrapper setup functions fails due to storage format command #18844

Merged

3 tasks

KAFKA-18773: Migrate the log4j1 config to log4j 2 for native image an…

fb5e2cb

…d README Signed-off-by: PoAn Yang <payang@apache.org>

FrankYang0529 force-pushed the KAFKA-18773 branch from 498deb6 to fb5e2cb Compare February 18, 2025 14:11

chia7712 reviewed Feb 18, 2025

View reviewed changes

chia7712 approved these changes Feb 18, 2025

View reviewed changes

frankvicky approved these changes Feb 18, 2025

View reviewed changes

chia7712 merged commit 1132f08 into apache:trunk Feb 18, 2025
8 of 9 checks passed

Parkerhiphop mentioned this pull request Feb 18, 2025

MINOR: Fix the missing and updated licenses #18950

Merged

3 tasks

FrankYang0529 deleted the KAFKA-18773 branch February 19, 2025 00:09

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

KAFKA-18773: Migrate the log4j1 config to log4j 2 for native image and README #18872

KAFKA-18773: Migrate the log4j1 config to log4j 2 for native image and README #18872

FrankYang0529 commented Feb 12, 2025 •

edited

Loading

dajac commented Feb 13, 2025

FrankYang0529 commented Feb 14, 2025

dajac commented Feb 17, 2025

chia7712 commented Feb 17, 2025

chia7712 commented Feb 18, 2025

chia7712 Feb 18, 2025

FrankYang0529 commented Feb 18, 2025

chia7712 Feb 18, 2025

chia7712 Feb 18, 2025

chia7712 Feb 18, 2025

chia7712 left a comment

frankvicky left a comment

chia7712 commented Feb 18, 2025

chia7712 commented Feb 18, 2025

KAFKA-18773: Migrate the log4j1 config to log4j 2 for native image and README #18872

KAFKA-18773: Migrate the log4j1 config to log4j 2 for native image and README #18872

Conversation

FrankYang0529 commented Feb 12, 2025 • edited Loading

Test steps

Committer Checklist (excluded from commit message)

dajac commented Feb 13, 2025

FrankYang0529 commented Feb 14, 2025

dajac commented Feb 17, 2025

chia7712 commented Feb 17, 2025

chia7712 commented Feb 18, 2025

chia7712 Feb 18, 2025

Choose a reason for hiding this comment

FrankYang0529 commented Feb 18, 2025

chia7712 Feb 18, 2025

Choose a reason for hiding this comment

chia7712 Feb 18, 2025

Choose a reason for hiding this comment

chia7712 Feb 18, 2025

Choose a reason for hiding this comment

chia7712 left a comment

Choose a reason for hiding this comment

frankvicky left a comment

Choose a reason for hiding this comment

chia7712 commented Feb 18, 2025

chia7712 commented Feb 18, 2025

FrankYang0529 commented Feb 12, 2025 •

edited

Loading