spark input_file_name() not working in cobrix

[kriswijnants]

Hi,

Thank you for creating and maintaining Cobrix. It's a tool we discovered recently, and plan to implement it in our cloud data platform for our Mainframe project.

Just a small question to ask. We notice the input_file_name() command in spark always returns blanks when using cobrix. This in combination with the option("is_record_sequence", "true") option.

spark.read.format("cobol").option("copybook", "/mnt/inputMDP/BIWA_GUTEX/Copybooks/"+dbutils.widgets.get("version")+"/GAGUSECO_20070115.txt").option("is_record_sequence", "true").load("/mnt/inputMDP/BIWA_GUTEX/Datafiles/"+dbutils.widgets.get("version")+"/GA-GA324001*").withColumn("ISN_Source", input_file_name).createOrReplaceTempView("vw_gutex_GA")

Do you notice the same behaviour? Is there any chance to get this working?

Keep up the good work!

Regards,

Kris

[yruslan]

Thanks for reporting the issue!

Looks interesting. Will take a look.

[yruslan]

I can confirm the issue. Indeed, for variable-record-size files input_file_name() returns an empty string. That is due to the way we handle sparse indexes creation to parallelize the reading of such files.

It will take a while to fix this properly (probably need to create a custom RDD). But we can add a workaround to generate a column with the input file name for each record. That's what we are going to do first. It would look like this:

.option("with_input_file_name_col", "ISN_Source")

[yruslan]

Just a double check. Which Spark version are you using?

We are planning to release Cobrix 2.0.0 first and all further changes will be made there. But it will support Spark 2.4 or above.

[kriswijnants]

Hi Ruslan, Thanks for your intervention. Really appreciate this! We are running on the Databricks runtime 6.2. So we use spark version 2.4.4. Regards, Kris Kris Wijnants Innovation Wizard m +32 (0)496 121 111 From: Ruslan Yushchenko <notifications@github.com> Sent: dinsdag 10 december 2019 15:05 To: AbsaOSS/cobrix <cobrix@noreply.github.com> Cc: Wijnants Kris <kris.wijnants@kohera.be>; Author <author@noreply.github.com> Subject: Re: [AbsaOSS/cobrix] spark input_file_name() not working in cobrix (#221) Just a double check. Which Spark version are you using? We are planning to release Cobrix 2.0.0 first and all further changes will be made there. But it will support Spark 2.4 or above. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub<https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FAbsaOSS%2Fcobrix%2Fissues%2F221%3Femail_source%3Dnotifications%26email_token%3DANWTSU2BVDM7LCASKSXC7PTQX6OZDA5CNFSM4JWVEGA2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEGPKZYY%23issuecomment-564047075&data=02%7C01%7Ckris.wijnants%40kohera.be%7C968d7e2082384915d9db08d77d79f628%7C49c3d703357947bfa8887c913fbdced9%7C0%7C0%7C637115835110774511&sdata=xfwMf%2F8SbQ5Xzg52TGLLde7rnu9P97uyKVZw7xMdbe4%3D&reserved=0>, or unsubscribe<https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FANWTSUZFFQBDGPXNOYL7D3DQX6OZDANCNFSM4JWVEGAQ&data=02%7C01%7Ckris.wijnants%40kohera.be%7C968d7e2082384915d9db08d77d79f628%7C49c3d703357947bfa8887c913fbdced9%7C0%7C0%7C637115835110774511&sdata=HNYtsVttf8fhLHBARa1k%2FQdfRKH18gybDHvRd291RBQ%3D&reserved=0>. This email has been scanned by BullGuard antivirus protection. For more info visit www.bullguard.com<http://www.bullguard.com/tracking.aspx?affiliate=bullguard&buyaffiliate=smtp&url=/>

[yruslan]

Great! Cobrix 2.0.0 is planned to be released this week. And the workaround for this issue can be expected sometime next week.

[kriswijnants]

Great news. Thanks Ruslan! Kris Wijnants Innovation Wizard m +32 (0)496 121 111 From: Ruslan Yushchenko <notifications@github.com> Sent: dinsdag 10 december 2019 15:46 To: AbsaOSS/cobrix <cobrix@noreply.github.com> Cc: Wijnants Kris <kris.wijnants@kohera.be>; Author <author@noreply.github.com> Subject: Re: [AbsaOSS/cobrix] spark input_file_name() not working in cobrix (#221) Great! Cobrix 2.0.0 is planned to be released this week. And the workaround for this issue can be expected sometime next week. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub<https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FAbsaOSS%2Fcobrix%2Fissues%2F221%3Femail_source%3Dnotifications%26email_token%3DANWTSU5CQLKIC2F4LC43NSLQX6TR7A5CNFSM4JWVEGA2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEGPPJ4Q%23issuecomment-564065522&data=02%7C01%7Ckris.wijnants%40kohera.be%7Cb4ec7fe5d008462962e108d77d7fa7f9%7C49c3d703357947bfa8887c913fbdced9%7C0%7C0%7C637115859539676949&sdata=Nd07Sh3DeOInfc6NjNoK2slXkGeon44mojCzT89MKho%3D&reserved=0>, or unsubscribe<https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FANWTSU65PCGSQQLHMLLTSEDQX6TR7ANCNFSM4JWVEGAQ&data=02%7C01%7Ckris.wijnants%40kohera.be%7Cb4ec7fe5d008462962e108d77d7fa7f9%7C49c3d703357947bfa8887c913fbdced9%7C0%7C0%7C637115859539686944&sdata=fgjNFw9wvc2TiPcYLH5arXNM2UuSY%2ByVh2hv1PozIkY%3D&reserved=0>. This email has been scanned by BullGuard antivirus protection. For more info visit www.bullguard.com<http://www.bullguard.com/tracking.aspx?affiliate=bullguard&buyaffiliate=smtp&url=/>