:generateProto fails with directory not found when importing other proto definitions

[anandwana001]

Topic Proto file

syntax = "proto3";

package model;

import "src/main/proto/subtitled_html.proto";
import "src/main/proto/thumbnail.proto";
import "src/main/proto/translation.proto";
import "src/main/proto/voiceover.proto";

option java_package = "org.oppia.app.model";
option java_multiple_files = true;

Build Failed

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':model:generateProto'.
> protoc: stdout: . stderr: src/main/proto/subtitled_html.proto: File not found.
  src/main/proto/thumbnail.proto: File not found.
  src/main/proto/translation.proto: File not found.
  src/main/proto/voiceover.proto: File not found.
  topic.proto: Import "src/main/proto/subtitled_html.proto" was not found or had errors.
  topic.proto: Import "src/main/proto/thumbnail.proto" was not found or had errors.
  topic.proto: Import "src/main/proto/translation.proto" was not found or had errors.
  topic.proto: Import "src/main/proto/voiceover.proto" was not found or had errors.
  topic.proto:31:3: "model.LessonThumbnail" seems to be defined in "thumbnail.proto", which is not imported by "topic.proto".  To use it here, please add the necessary import.
  topic.proto: "model.VoiceoverMapping" seems to be defined in "voiceover.proto", which is not imported by "topic.proto".  To use it here, please add the necessary import.
  topic.proto: "model.TranslationMapping" seems to be defined in "translation.proto", which is not imported by "topic.proto".  To use it here, please add the necessary import.
  topic.proto:49:3: "model.SubtitledHtml" seems to be defined in "subtitled_html.proto", which is not imported by "topic.proto".  To use it here, please add the necessary import.
  topic.proto:52:12: "model.SubtitledHtml" seems to be defined in "subtitled_html.proto", which is not imported by "topic.proto".  To use it here, please add the necessary import.
  topic.proto:72:3: "model.LessonThumbnail" seems to be defined in "thumbnail.proto", which is not imported by "topic.proto".  To use it here, please add the necessary import.
  topic.proto:93:3: "model.LessonThumbnail" seems to be defined in "thumbnail.proto", which is not imported by "topic.proto".  To use it here, please add the necessary import.
  topic.proto:109:3: "model.LessonThumbnail" seems to be defined in "thumbnail.proto", which is not imported by "topic.proto".  To use it here, please add the necessary import.
  topic.proto:152:3: "model.LessonThumbnail" seems to be defined in "thumbnail.proto", which is not imported by "topic.proto".  To use it here, please add the necessary import.
  topic.proto:193:3: "model.LessonThumbnail" seems to be defined in "thumbnail.proto", which is not imported by "topic.proto".  To use it here, please add the necessary import.
  topic.proto:226:3: "model.LessonThumbnail" seems to be defined in "thumbnail.proto", which is not imported by "topic.proto".  To use it here, please add the necessary import.
  topic.proto:308:3: "model.LessonThumbnail" seems to be defined in "thumbnail.proto", which is not imported by "topic.proto".  To use it here, please add the necessary import.
  topic.proto: "model.VoiceoverMapping" seems to be defined in "voiceover.proto", which is not imported by "topic.proto".  To use it here, please add the necessary import.
  topic.proto: "model.TranslationMapping" seems to be defined in "translation.proto", which is not imported by "topic.proto".  To use it here, please add the necessary import.
  topic.proto:317:3: "model.SubtitledHtml" seems to be defined in "subtitled_html.proto", which is not imported by "topic.proto".  To use it here, please add the necessary import.

Now if I update this to

syntax = "proto3";

package model;

import "subtitled_html.proto";
import "thumbnail.proto";
import "translation.proto";
import "voiceover.proto";

option java_package = "org.oppia.app.model";
option java_multiple_files = true;

It will not fail, but check lint fails

command - protoc -I . --buf-check-lint_out=. $(find . -name '*.proto')

subtitled_html.proto: File not found.
thumbnail.proto: File not found.
translation.proto: File not found.
voiceover.proto: File not found.
model/build/resources/main/topic.proto:5:1: Import "subtitled_html.proto" was not found or had errors.
model/build/resources/main/topic.proto:6:1: Import "thumbnail.proto" was not found or had errors.
model/build/resources/main/topic.proto:7:1: Import "translation.proto" was not found or had errors.
model/build/resources/main/topic.proto:8:1: Import "voiceover.proto" was not found or had errors.
model/build/resources/main/topic.proto:31:3: "LessonThumbnail" is not defined.
model/build/resources/main/topic.proto: "VoiceoverMapping" is not defined.
model/build/resources/main/topic.proto: "TranslationMapping" is not defined.
model/build/resources/main/topic.proto:49:3: "SubtitledHtml" is not defined.
model/build/resources/main/topic.proto:52:12: "SubtitledHtml" is not defined.
model/build/resources/main/topic.proto:72:3: "LessonThumbnail" is not defined.
model/build/resources/main/topic.proto:93:3: "LessonThumbnail" is not defined.
model/build/resources/main/topic.proto:109:3: "LessonThumbnail" is not defined.
model/build/resources/main/topic.proto:152:3: "LessonThumbnail" is not defined.
model/build/resources/main/topic.proto:193:3: "LessonThumbnail" is not defined.
model/build/resources/main/topic.proto:226:3: "LessonThumbnail" is not defined.
model/build/resources/main/topic.proto:308:3: "LessonThumbnail" is not defined.
model/build/resources/main/topic.proto: "VoiceoverMapping" is not defined.
model/build/resources/main/topic.proto: "TranslationMapping" is not defined.
model/build/resources/main/topic.proto:317:3: "SubtitledHtml" is not defined.

Folders

Gradle file

apply plugin: 'java-library'
apply plugin: 'com.google.protobuf'

protobuf {
  protoc {
    artifact = 'com.google.protobuf:protoc:3.0.0'
  }
  plugins {
    javalite {
      artifact = 'com.google.protobuf:protoc-gen-javalite:3.0.0'
    }
  }
  generateProtoTasks {
    all().each { task ->
      task.builtins {
        remove java
      }
      task.plugins {
        javalite {}
      }
    }
  }
}

dependencies {
  compile 'com.google.protobuf:protobuf-lite:3.0.0'
}

sourceSets {
  main.java.srcDirs += "${protobuf.generatedFilesBaseDir}/main/javalite"
  main.java.srcDirs += "$projectDir/src/main/proto"
}

If I try adding --proto_path so that It will know where to look for files:
command - protoc -I=model/src/main/proto -I=model/build/resources/main --buf-check-lint_out=. $(find . -name '*.proto')

Error
./model/build/resources/main/onboarding.proto: Input is shadowed in the --proto_path by "model/src/main/proto/onboarding.proto". Either use the latter file as your input or reorder the --proto_path so that the former file's location comes first.

[anandwana001]

@voidzcy

[ejona86]

import "src/main/proto/subtitled_html.proto";

Each "proto" directory is a separate "proto root", which is what imports should be relative to. (Just like Java source files are relative to the "java" folder.) So the import should be import "subtitled_html.proto", which you figured out.

Each "proto root" corresponds to a -I flag. So you'd want to use protoc -I src/main/proto when running lint, which you also figured out.

The last error is because you don't need model/build/resources/main; that's a copy from model/src/main/proto. Protoc is failing because there are duplicate proto files. You only need to include other files for imports, like build/extracted-include-protos/main/ and maybe src/main/protos from other gradle projects in a multiproject build.

[anandwana001]

@ejona86 model/build/resources/main this is something automatically generated by protoc Gradle plugin whenever the Gradle build happens

[anandwana001]

Deleting model/build/resources/main by force and then running a protoc lint check or

processResources {
    exclude('**/*.proto')
}

protoc -I . --buf-check-lint_out=. $(find . -name '*.proto')

subtitled_html.proto: File not found.
thumbnail.proto: File not found.
translation.proto: File not found.
voiceover.proto: File not found.
model/src/main/proto/topic.proto:5:1: Import "subtitled_html.proto" was not found or had errors.
model/src/main/proto/topic.proto:6:1: Import "thumbnail.proto" was not found or had errors.
model/src/main/proto/topic.proto:7:1: Import "translation.proto" was not found or had errors.
model/src/main/proto/topic.proto:8:1: Import "voiceover.proto" was not found or had errors.
model/src/main/proto/topic.proto:31:3: "LessonThumbnail" is not defined.
model/src/main/proto/topic.proto: "VoiceoverMapping" is not defined.
model/src/main/proto/topic.proto: "TranslationMapping" is not defined.
model/src/main/proto/topic.proto:49:3: "SubtitledHtml" is not defined.
model/src/main/proto/topic.proto:52:12: "SubtitledHtml" is not defined.
model/src/main/proto/topic.proto:72:3: "LessonThumbnail" is not defined.
model/src/main/proto/topic.proto:93:3: "LessonThumbnail" is not defined.
model/src/main/proto/topic.proto:109:3: "LessonThumbnail" is not defined.
model/src/main/proto/topic.proto:152:3: "LessonThumbnail" is not defined.
model/src/main/proto/topic.proto:193:3: "LessonThumbnail" is not defined.
model/src/main/proto/topic.proto:226:3: "LessonThumbnail" is not defined.
model/src/main/proto/topic.proto:308:3: "LessonThumbnail" is not defined.
model/src/main/proto/topic.proto: "VoiceoverMapping" is not defined.
model/src/main/proto/topic.proto: "TranslationMapping" is not defined.
model/src/main/proto/topic.proto:317:3: "SubtitledHtml" is not defined.

Changing imports to:

package model;

import "model/src/main/proto/subtitled_html.proto";
import "model/src/main/proto/thumbnail.proto";
import "model/src/main/proto/translation.proto";
import "model/src/main/proto/voiceover.proto";

Passes the protoc (only if I delete model/build/resources/main or model/build/extracted-include-protos/test/topic.proto or any other .proto file reference inside build directory) but generateProto fails

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':model:generateProto'.
> protoc: stdout: . stderr: model/src/main/proto/subtitled_html.proto: File not found.
  model/src/main/proto/thumbnail.proto: File not found.
  model/src/main/proto/translation.proto: File not found.
  model/src/main/proto/voiceover.proto: File not found.
  topic.proto: Import "model/src/main/proto/subtitled_html.proto" was not found or had errors.
  topic.proto: Import "model/src/main/proto/thumbnail.proto" was not found or had errors.
  topic.proto: Import "model/src/main/proto/translation.proto" was not found or had errors.
  topic.proto: Import "model/src/main/proto/voiceover.proto" was not found or had errors.
  topic.proto:31:3: "model.LessonThumbnail" seems to be defined in "thumbnail.proto", which is not imported by "topic.proto".  To use it here, please add the necessary import.
  topic.proto: "model.VoiceoverMapping" seems to be defined in "voiceover.proto", which is not imported by "topic.proto".  To use it here, please add the necessary import.
  topic.proto: "model.TranslationMapping" seems to be defined in "translation.proto", which is not imported by "topic.proto".  To use it here, please add the necessary import.
  topic.proto:49:3: "model.SubtitledHtml" seems to be defined in "subtitled_html.proto", which is not imported by "topic.proto".  To use it here, please add the necessary import.
  topic.proto:52:12: "model.SubtitledHtml" seems to be defined in "subtitled_html.proto", which is not imported by "topic.proto".  To use it here, please add the necessary import.
  topic.proto:72:3: "model.LessonThumbnail" seems to be defined in "thumbnail.proto", which is not imported by "topic.proto".  To use it here, please add the necessary import.
  topic.proto:93:3: "model.LessonThumbnail" seems to be defined in "thumbnail.proto", which is not imported by "topic.proto".  To use it here, please add the necessary import.
  topic.proto:109:3: "model.LessonThumbnail" seems to be defined in "thumbnail.proto", which is not imported by "topic.proto".  To use it here, please add the necessary import.
  topic.proto:152:3: "model.LessonThumbnail" seems to be defined in "thumbnail.proto", which is not imported by "topic.proto".  To use it here, please add the necessary import.
  topic.proto:193:3: "model.LessonThumbnail" seems to be defined in "thumbnail.proto", which is not imported by "topic.proto".  To use it here, please add the necessary import.
  topic.proto:226:3: "model.LessonThumbnail" seems to be defined in "thumbnail.proto", which is not imported by "topic.proto".  To use it here, please add the necessary import.
  topic.proto:308:3: "model.LessonThumbnail" seems to be defined in "thumbnail.proto", which is not imported by "topic.proto".  To use it here, please add the necessary import.
  topic.proto: "model.VoiceoverMapping" seems to be defined in "voiceover.proto", which is not imported by "topic.proto".  To use it here, please add the necessary import.
  topic.proto: "model.TranslationMapping" seems to be defined in "translation.proto", which is not imported by "topic.proto".  To use it here, please add the necessary import.
  topic.proto:317:3: "model.SubtitledHtml" seems to be defined in "subtitled_html.proto", which is not imported by "topic.proto".  To use it here, please add the necessary import.

[anandwana001]

I feel like the plugin is working correctly but not understanding why lint is throwing such error!!

[ejona86]

@ejona86 model/build/resources/main this is something automatically generated by protoc Gradle plugin whenever the Gradle build happens

I don't disagree with that. But you don't need to use it. It seems like you are running protoc for the lint manually. So simply run with a different command. Something closer to: protoc -I src/main/proto --buf-check-lint_out=. $(cd src/main/proto && find . -name '*.proto')

---

I think you can actually have the protoc plugin run the linter for you. It seems you are relying on protoc-gen-buf-check-lint being in your PATH, which is how protoc will find the plugin when you specify --buf-check-lint_out. I think you can just call task.builtins { "buf-check-lint" {} } and the gradle protoc plugin will run the linter plugin.

(It acts as a "builtin" plugin because the gradle protoc plugin won't need to provide the location of the executable to run. Normally with protoc plugins (like javalite) the gradle protoc plugin would need to download the executable from Maven Central. But in this case you are just dependeing on PATH.)

[anandwana001]

Thanks @ejona86 Big help, it worked out.

1. The last error is because you don't need model/build/resources/main
   For this I used:

processResources {
    exclude('**/*.proto')
}

2. passing the dir loc in command
   protoc -I=model/src/main/proto --buf-check-lint_out=. $(find . -name '*.proto')

3. similarly with buf command
   buf check lint --input=model/src/main/proto --input-config buf.yaml it also explicitly need the location of its YAML file to know the set of rules.