[Bug]: MongoDBtoBigQuery with UDF: ScriptObjectMirror cannot be cast to bson.Document

[NichaRoj]

Related Template(s)

MongoDB to BigQuery

What happened?

When creating a job with MongoDB to BigQuery template and specify a UDF javascript file and function, the job fails to start. See relevant logs below.

When UDF file and function are not specified, jobs with the same template can run normally.

Beam Version

Newer than 2.43.0

Relevant log output

com.google.cloud.teleport.v2.common.UncaughtExceptionLogger - The template launch failed.
java.lang.ClassCastException: class org.openjdk.nashorn.api.scripting.ScriptObjectMirror cannot be cast to class org.bson.Document (org.openjdk.nashorn.api.scripting.ScriptObjectMirror and org.bson.Document are in unnamed module of loader 'app')
	at com.google.cloud.teleport.v2.mongodb.templates.MongoDbUtils.getTableFieldSchemaForUDF(MongoDbUtils.java:167)
	at com.google.cloud.teleport.v2.mongodb.templates.MongoDbToBigQuery.run(MongoDbToBigQuery.java:95)
	at com.google.cloud.teleport.v2.mongodb.templates.MongoDbToBigQuery.main(MongoDbToBigQuery.java:82)

[Polber]

This is an issue with how UDF output is transformed into a Document object that is then passed into the pipeline. The output should be parsed from JSON output instead of directly casted. I can work on a fix and update this PR this week.

[Polber]

I created a PR to fix this issue: #588, but it does introduce a couple side-effects I wanted to discuss with the MongoDB team

@theshanbhag Since you added the feature, I thought I would tie you in to get your input.

Currently, MongoDB templates that use javascript UDF's pass a bson Document to the Javascript function, and expect a bson Document back. However, what is actually occurring is a bson Document is being converted to a String by the nashorn javascript engine before being passed to the function. This String is difficult to manipulate with native javascript because it cannot be easily parsed into a JSON object, and the return object cannot be easily casted into a Document.

My solution I introduced in #588 instead passes a json string representation of the Document to the javascript function, and the function is expected to return either a Document-castable object OR a JSON String that can be easily parsed into a Document.

This would allow for a UDF similar to below:

/**
 * A simple transform function.
 * @param {string} inJson
 * @return {string} outJson
 */
function transform(inJson) {
    var outJson = JSON.parse(inJson);
    outJson.key = "value";
    return JSON.stringify(outJson);
}

Thoughts on this?

[theshanbhag]

Hi Jeff. This looks great. I will test it out over the next few days. I will work on updating other documentation with the latest changes you have added in the PR. Thanks and regards, Venkatesh Shanbhag

…

On Fri, Feb 10, 2023 at 9:13 PM Jeff Kinard ***@***.***> wrote: I created a PR to fix this issue: #588 <#588>, but it does introduce a couple side-effects I wanted to discuss with the MongoDB team @theshanbhag <https://github.com/theshanbhag> Since you added the feature, I thought I would tie you in to get your input. Currently, MongoDB templates that use javascript UDF's pass a bson Document to the Javascript function, and expect a bson Document back. However, what is actually occurring is a bson Document is being converted to a String by the nashorn javascript engine before being passed to the function. This String is difficult to manipulate with native javascript because it cannot be easily parsed into a JSON object, and the return object cannot be easily casted into a Document. My solution I introduced in #588 <#588> instead passes a json string representation of the Document to the javascript function, and the function is expected to return either a Document-castable object OR a JSON String that can be easily parsed into a Document. This would allow for a UDF similar to below: /** * A simple transform function. * @param {string} inJson * @return {string} outJson */ function transform(inJson) { var outJson = JSON.parse(inJson); outJson.key = "value"; return JSON.stringify(outJson); } Thoughts on this? — Reply to this email directly, view it on GitHub <#582 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AV3XKTAYJAM6ZWGFWLZRHZLWWZOZ5ANCNFSM6AAAAAAUTUXRTY> . You are receiving this because you were mentioned.Message ID: ***@***.***>

-- { name : "Venkatesh Shanbhag", company : ["MongoDB <https://www.mongodb.com/blog/channel/culture>"], phone : "+91-9972646444", location : "India", designation : "Solutions Architect | Partner presales" linkedIn : "www.linkedin.com/in/venkatesh-shanbhag" }

[Polber]

Hi @theshanbhag, has there been any progress on this?