Compatibility with Spark 3.2 + Scala 2.13 #7399

emartinezs44 · 2023-02-01T17:09:57Z

I would like to know if there are any plans to compile bigdl-dllib with Scala 2.13, and publish the artifact. Currently I´m working with Scala 3 in some research and I would like to present a paper for the next Scaladays using new lang features + some bigdl examples.

Thanks!

qiuxin2012 · 2023-02-02T04:35:00Z

@emartinezs44 Bad news, I tried to upgrade dllib to scala 2.13, but only one of dllib's dependency xgboost4j didn't support scala 2.13. dmlc/xgboost#6596

jason-dai · 2023-02-02T06:55:35Z

@emartinezs44 Bad news, I tried to upgrade dllib to scala 2.13, but only one of dllib's dependency xgboost4j didn't support scala 2.13. dmlc/xgboost#6596

Maybe we can release an experimental dllib package for Spark 3.2+scala 3.13 without xgboost support

emartinezs44 · 2023-02-02T09:11:13Z

I see, this kind of dependences are a problem(having in mind that Spark 3.2 was released a year and a half ago). By now, the only solution is to create a profile and exclude the dependence and the code in Scala referencing xgboost. I will try to create a branch in my fork with this config and see if it compiles or if there are another problems.

qiuxin2012 · 2023-02-03T01:35:18Z

I have a branch https://github.com/qiuxin2012/BigDL/tree/scala3, I'm working on the compatiability with scala 2.13
100+ build errors, I'm tring to fix them.

qiuxin2012 · 2023-02-03T08:54:48Z

@emartinezs44 I'm blocking by a "ambiguous implicit values" error, could you help?
You can use ./make-dist.sh -P spark_3.x -P scala_2.13 -Dspark.version=3.2.3 to reproduce the error message, on my branch https://github.com/qiuxin2012/BigDL/tree/scala3.

[ERROR] /home/xin/IdeaProjects/BigDL2/scala/dllib/src/main/scala/com/intel/analytics/bigdl/dllib/tensor/DenseTensor.scala:239: ambiguous implicit values:
 both value evidence$1 in class DenseTensor of type scala.reflect.ClassTag[T]
 and value evidence$9 of type scala.reflect.ClassTag[T]
 match expected type scala.reflect.ClassTag[T]
[ERROR]     DenseTensor.newWithStorage(this, storage, _storageOffset, _size, _stride, ev)
[ERROR]                               ^
[ERROR] /home/xin/IdeaProjects/BigDL2/scala/dllib/src/main/scala/com/intel/analytics/bigdl/dllib/tensor/DenseTensor.scala:234: ambiguous implicit values:
 both value evidence$1 in class DenseTensor of type scala.reflect.ClassTag[T]
 and value evidence$9 of type scala.reflect.ClassTag[T]
 match expected type scala.reflect.ClassTag[T]
[ERROR]   private[tensor] def this(storage: ArrayStorage[T])(implicit ev: TensorNumeric[T]) = {
[ERROR]                                                                                       ^
[ERROR] /home/xin/IdeaProjects/BigDL2/scala/dllib/src/main/scala/com/intel/analytics/bigdl/dllib/tensor/DenseTensor.scala:249: ambiguous implicit values:
 both value evidence$1 in class DenseTensor of type scala.reflect.ClassTag[T]
 and value evidence$10 of type scala.reflect.ClassTag[T]
 match expected type scala.reflect.ClassTag[T]
[ERROR]       DenseTensor.newWithStorage[T](this, storage, _storageOffset, _size, _stride, ev)
[ERROR]                                    ^
[ERROR] /home/xin/IdeaProjects/BigDL2/scala/dllib/src/main/scala/com/intel/analytics/bigdl/dllib/tensor/DenseTensor.scala:245: ambiguous implicit values:
 both value evidence$1 in class DenseTensor of type scala.reflect.ClassTag[T]
 and value evidence$10 of type scala.reflect.ClassTag[T]
 match expected type scala.reflect.ClassTag[T]
[ERROR]     if (storage != null) {
[ERROR]                          ^
[ERROR] /home/xin/IdeaProjects/BigDL2/scala/dllib/src/main/scala/com/intel/analytics/bigdl/dllib/tensor/DenseTensor.scala:261: ambiguous implicit values:
 both value evidence$1 in class DenseTensor of type scala.reflect.ClassTag[T]
 and value evidence$11 of type scala.reflect.ClassTag[T]
 match expected type scala.reflect.ClassTag[T]
[ERROR]     DenseTensor.newWithStorage[T](this, _storage, _storageOffset, _size, _stride, ev)
[ERROR]                                  ^
[ERROR] /home/xin/IdeaProjects/BigDL2/scala/dllib/src/main/scala/com/intel/analytics/bigdl/dllib/tensor/DenseTensor.scala:253: ambiguous implicit values:
 both value evidence$1 in class DenseTensor of type scala.reflect.ClassTag[T]
 and value evidence$11 of type scala.reflect.ClassTag[T]
 match expected type scala.reflect.ClassTag[T]
[ERROR]   private[tensor] def this(other: Tensor[T])(implicit ev: TensorNumeric[T]) = {
[ERROR]                                                                               ^
[ERROR] /home/xin/IdeaProjects/BigDL2/scala/dllib/src/main/scala/com/intel/analytics/bigdl/dllib/tensor/QuantizedTensor.scala:118: ambiguous implicit values:
 both value evidence$1 in class QuantizedTensor of type scala.reflect.ClassTag[T]
 and value evidence$2 of type scala.reflect.ClassTag[T]
 match expected type scala.reflect.ClassTag[T]
[ERROR]     this.desc = Desc.get(params, null, 0, null, null)
[ERROR]                         ^
[ERROR] /home/xin/IdeaProjects/BigDL2/scala/dllib/src/main/scala/com/intel/analytics/bigdl/dllib/tensor/QuantizedTensor.scala:122: ambiguous implicit values:
 both value evidence$1 in class QuantizedTensor of type scala.reflect.ClassTag[T]
 and value evidence$3 of type scala.reflect.ClassTag[T]
 match expected type scala.reflect.ClassTag[T]
[ERROR]     implicit ev: TensorNumeric[T]) = {
[ERROR]                                      ^
[ERROR] /home/xin/IdeaProjects/BigDL2/scala/dllib/src/main/scala/com/intel/analytics/bigdl/dllib/tensor/QuantizedTensor.scala:126: ambiguous implicit values:
 both value evidence$1 in class QuantizedTensor of type scala.reflect.ClassTag[T]
 and value evidence$3 of type scala.reflect.ClassTag[T]
 match expected type scala.reflect.ClassTag[T]
[ERROR]     this.desc = Desc.get(descParams, this.internalStorage, 0, this.maxOfRow, this.minOfRow)
[ERROR]                         ^
[ERROR] /home/xin/IdeaProjects/BigDL2/scala/dllib/src/main/scala/com/intel/analytics/bigdl/dllib/tensor/QuantizedTensor.scala:130: ambiguous implicit values:
 both value evidence$1 in class QuantizedTensor of type scala.reflect.ClassTag[T]
 and value evidence$4 of type scala.reflect.ClassTag[T]
 match expected type scala.reflect.ClassTag[T]
[ERROR]     descParams: DescParams)(implicit ev: TensorNumeric[T]) = {
[ERROR]                                                              ^
[ERROR] /home/xin/IdeaProjects/BigDL2/scala/dllib/src/main/scala/com/intel/analytics/bigdl/dllib/tensor/QuantizedTensor.scala:140: ambiguous implicit values:
 both value evidence$1 in class QuantizedTensor of type scala.reflect.ClassTag[T]
 and value evidence$4 of type scala.reflect.ClassTag[T]
 match expected type scala.reflect.ClassTag[T]
[ERROR]     this.desc = Desc.get(descParams, this.internalStorage, 0, this.maxOfRow, this.minOfRow)
[ERROR]                         ^

emartinezs44 · 2023-02-03T09:07:14Z

let me check it out

emartinezs44 · 2023-02-06T10:27:52Z

Well, there is a problem related to the implicit evidences that the compiler creates.

This is with Scala 2.13

private[tensor] class DenseTensor[@specialized T: ClassTag]( ...

It includes an implicit evidence for the ClassTag[T]. Here a simplified output of the 'log-implicits' scalac compiler for the same problem but much simpler:

private class CustomArray[@specialized T] extends scala.AnyRef {
   <paramaccessor> private[this] var storage: Array[T] = _;
   <accessor> <paramaccessor> private def storage: Array[T] = CustomArray.this.storage;
   <accessor> <paramaccessor> private def storage_=(x$1: Array[T]): Unit = CustomArray.this.storage = x$1;

   // Here the evidence created with the [@specialized T: ClassTag]
   implicit <synthetic> <paramaccessor> private[this] val evidence$1: scala.reflect.ClassTag[T] = _; 
   
   def <init>(storage: Array[T])(implicit evidence$1: scala.reflect.ClassTag[T]): CustomArray[T] = {
     CustomArray.super.<init>();
     ()
   };

   /** the constructor here includes another evidence because it is required. So we have two ClassTag evidences in scope **/ 
   private def <init>(storage: Array[T], length: Int)(implicit evidence$2: scala.reflect.ClassTag[T]): CustomArray[T] = {
     CustomArray.this.<init>(null)(evidence$2);
     /** The compiler here does not know which one include and .... Error **/
     CustomArray.newWithStorage[T](this, storage)();
     ()
   }
 };
 object CustomArray extends scala.AnyRef {
   def <init>(): CustomArray.type = {
     CustomArray.super.<init>();
     ()
   };
   private def newWithStorage[@specialized(scala.Float, scala.Double) T](customArray: CustomArray[T], in: Array[T])(implicit evidence$3: scala.reflect.ClassTag[T]): CustomArray[T] = {
     customArray.storage_=(in);
     customArray
   }
 }
}

These kind of conflicts were resolved in Scala 2.12 taking the evidence created in the constructor call and passing it directly to the function:

For Scala 2.12:

private class CustomArray[@specialized T] extends scala.AnyRef {
    <paramaccessor> private[this] var storage: Array[T] = _;
    <accessor> <paramaccessor> private def storage: Array[T] = CustomArray.this.storage;
    <accessor> <paramaccessor> private def storage_=(x$1: Array[T]): Unit = CustomArray.this.storage = x$1;
    implicit <paramaccessor> private[this] val ev: scala.reflect.ClassTag[T] = _;
    def <init>(storage: Array[T])(implicit ev: scala.reflect.ClassTag[T]): CustomArray[T] = {
      CustomArray.super.<init>();
      ()
    };
    private def <init>(storage: Array[T], length: Int)(implicit ev: scala.reflect.ClassTag[T]): CustomArray[T] = {
      CustomArray.this.<init>(null)(ev);
      /** Here the dependence is resolved by the compiler including explicitly the evidence when calling the 'newWithStorage function */
      CustomArray.newWithStorage[T](this, storage)(ev);
      ()
    }
  };
  object CustomArray extends scala.AnyRef {
    def <init>(): CustomArray.type = {
      CustomArray.super.<init>();
      ()
    };
    private def newWithStorage[@specialized(scala.Float, scala.Double) T](customArray: CustomArray[T], in: Array[T])(implicit evidence$1: scala.reflect.ClassTag[T]): CustomArray[T] = {
      customArray.storage_=(in);
      customArray
    }
  }

SOLUTION:

This solution is valid for Scala 2.12, and consists in declaring the implicit in the class definition:

private[tensor] class DenseTensor[@specialized T](
  private[tensor] var _storage: ArrayStorage[T],
  private[tensor] var _storageOffset: Int,
  private[tensor] var _size: Array[Int],
  private[tensor] var _stride: Array[Int],
  var nDimension: Int)(implicit ev: TensorNumeric[T], /** HERE **/ ev100: ClassTag[T])
  extends Tensor[T] {

and in the constructors that create the conflict like:

private[tensor] def this(other: Tensor[T])(implicit ev: TensorNumeric[T])

changing the constructor definition like:

 /** including the implicit reference in the constructor definition **/
  private[tensor] def this(other: Tensor[T])(implicit ev: TensorNumeric[T], ev1: ClassTag[T]) = {
    this(null, 0, null, null, 0)
    Log4Error.unKnowExceptionError(other.isInstanceOf[DenseTensor[_]],
      "Only support dense tensor in this operation")
    val _storage = other.storage().asInstanceOf[ArrayStorage[T]]
    val _storageOffset = other.storageOffset() - 1
    val _size = other.size()
    val _stride = other.stride()
    /** pass this evidence explicitly **/
    DenseTensor.newWithStorage[T](this, _storage, _storageOffset, _size, _stride, ev)(ev1)
  }

This resolves the problems in DenseTensor definition, maybe there are other related to this ambiguity problem. Take a look and tell me if you can handle it or if you need me to work in your branch.

qiuxin2012 · 2023-02-06T11:52:14Z

Well, there is a problem related to the implicit evidences that the compiler creates.

This is with Scala 2.13

private[tensor] class DenseTensor[@specialized T: ClassTag]( ...

It includes an implicit evidence for the ClassTag[T]. Here a simplified output of the 'log-implicits' scalac compiler for the same problem but much simpler:

private class CustomArray[@specialized T] extends scala.AnyRef {
   <paramaccessor> private[this] var storage: Array[T] = _;
   <accessor> <paramaccessor> private def storage: Array[T] = CustomArray.this.storage;
   <accessor> <paramaccessor> private def storage_=(x$1: Array[T]): Unit = CustomArray.this.storage = x$1;

   // Here the evidence created with the [@specialized T: ClassTag]
   implicit <synthetic> <paramaccessor> private[this] val evidence$1: scala.reflect.ClassTag[T] = _; 
   
   def <init>(storage: Array[T])(implicit evidence$1: scala.reflect.ClassTag[T]): CustomArray[T] = {
     CustomArray.super.<init>();
     ()
   };

   /** the constructor here includes another evidence because it is required. So we have two ClassTag evidences in scope **/ 
   private def <init>(storage: Array[T], length: Int)(implicit evidence$2: scala.reflect.ClassTag[T]): CustomArray[T] = {
     CustomArray.this.<init>(null)(evidence$2);
     /** The compiler here does not know which one include and .... Error **/
     CustomArray.newWithStorage[T](this, storage)();
     ()
   }
 };
 object CustomArray extends scala.AnyRef {
   def <init>(): CustomArray.type = {
     CustomArray.super.<init>();
     ()
   };
   private def newWithStorage[@specialized(scala.Float, scala.Double) T](customArray: CustomArray[T], in: Array[T])(implicit evidence$3: scala.reflect.ClassTag[T]): CustomArray[T] = {
     customArray.storage_=(in);
     customArray
   }
 }
}

These kind of conflicts were resolved in Scala 2.12 taking the evidence created in the constructor call and passing it directly to the function:

For Scala 2.12:

private class CustomArray[@specialized T] extends scala.AnyRef {
    <paramaccessor> private[this] var storage: Array[T] = _;
    <accessor> <paramaccessor> private def storage: Array[T] = CustomArray.this.storage;
    <accessor> <paramaccessor> private def storage_=(x$1: Array[T]): Unit = CustomArray.this.storage = x$1;
    implicit <paramaccessor> private[this] val ev: scala.reflect.ClassTag[T] = _;
    def <init>(storage: Array[T])(implicit ev: scala.reflect.ClassTag[T]): CustomArray[T] = {
      CustomArray.super.<init>();
      ()
    };
    private def <init>(storage: Array[T], length: Int)(implicit ev: scala.reflect.ClassTag[T]): CustomArray[T] = {
      CustomArray.this.<init>(null)(ev);
      /** Here the dependence is resolved by the compiler including explicitly the evidence when calling the 'newWithStorage function */
      CustomArray.newWithStorage[T](this, storage)(ev);
      ()
    }
  };
  object CustomArray extends scala.AnyRef {
    def <init>(): CustomArray.type = {
      CustomArray.super.<init>();
      ()
    };
    private def newWithStorage[@specialized(scala.Float, scala.Double) T](customArray: CustomArray[T], in: Array[T])(implicit evidence$1: scala.reflect.ClassTag[T]): CustomArray[T] = {
      customArray.storage_=(in);
      customArray
    }
  }

SOLUTION:

This solution is valid for Scala 2.12, and consists in declaring the implicit in the class definition:

private[tensor] class DenseTensor[@specialized T](
  private[tensor] var _storage: ArrayStorage[T],
  private[tensor] var _storageOffset: Int,
  private[tensor] var _size: Array[Int],
  private[tensor] var _stride: Array[Int],
  var nDimension: Int)(implicit ev: TensorNumeric[T], /** HERE **/ ev100: ClassTag[T])
  extends Tensor[T] {

and in the constructors that create the conflict like:

private[tensor] def this(other: Tensor[T])(implicit ev: TensorNumeric[T])

changing the constructor definition like:

 /** including the implicit reference in the constructor definition **/
  private[tensor] def this(other: Tensor[T])(implicit ev: TensorNumeric[T], ev1: ClassTag[T]) = {
    this(null, 0, null, null, 0)
    Log4Error.unKnowExceptionError(other.isInstanceOf[DenseTensor[_]],
      "Only support dense tensor in this operation")
    val _storage = other.storage().asInstanceOf[ArrayStorage[T]]
    val _storageOffset = other.storageOffset() - 1
    val _size = other.size()
    val _stride = other.stride()
    /** pass this evidence explicitly **/
    DenseTensor.newWithStorage[T](this, _storage, _storageOffset, _size, _stride, ev)(ev1)
  }

This resolves the problems in DenseTensor definition, maybe there are other related to this ambiguity problem. Take a look and tell me if you can handle it or if you need me to work in your branch.

Thanks for you help, I will try your solution later.
I have used a workaround to solve the ambiguity problem. I moved DenseTensor.newWithStorage[T] out of the constructor, and define an apply method to instead of this constucutor.

All compile errors are all fixed now, and the unit tests is running.

qiuxin2012 · 2023-02-07T09:31:04Z

@emartinezs44 We have released a snapshot version with Spark 3.2 and Scala 2.13 to
https://oss.sonatype.org/content/repositories/snapshots/com/intel/analytics/bigdl/bigdl-dllib-spark_3.2.3/2.3.0-SNAPSHOT/
Please info us if you meet any error.

emartinezs44 · 2023-02-07T09:51:06Z

Wonderful!!!, I try as as soon as possible. Thank you very much!

qiuxin2012 · 2023-02-13T01:55:15Z

The changes of scala 2.13 support are merged to https://github.com/intel-analytics/BigDL/tree/scala-2.13

We still has some failures due to:

orca scala-2.13 build failed
spark 2.4 scala2.10 build failed due to scala-java Map conversion API missmatch.
xgboost don't support scala 2.13

Please info us, if you has more requirements.

emartinezs44 · 2023-02-13T11:27:15Z

It seems to work well. Some aspects related to implicit conversions, which is normal having in mind that the Scala 3 compiler is a new compiler. But it looks great, let me try more in other scenarios.

s0t00524 · 2024-08-21T08:00:26Z

Hi. xgboost4j now supports Scala 2.13.
I would like to hear if there are any plans to release bigdl-dllib compiled with Scala 2.13.

dmlc/xgboost#9099
https://mvnrepository.com/artifact/ml.dmlc/xgboost4j

Thank you!

s0t00524 · 2024-10-15T02:47:13Z

@qiuxin2012 @shane-huang Hi. xgboost4j now supports Scala 2.13.
I would like to hear if there are any plans to release bigdl-dllib compiled with Scala 2.13.

dmlc/xgboost#9099
https://mvnrepository.com/artifact/ml.dmlc/xgboost4j

Thank you!

qiuxin2012 · 2024-10-16T02:39:42Z

@s0t00524 No plan to support scala 2.13

s0t00524 · 2024-11-06T07:14:02Z

@qiuxin2012 Thank you for your reply. So, do you have any plan to support Scala 3?

qiuxin2012 · 2024-11-07T02:20:15Z

@s0t00524 No plan to support Scala 3 now.

emartinezs44 · 2024-11-07T09:42:49Z

You can create Scala 3 projects following the repo: https://github.com/emartinezs44/ts4s. There is a branch compiled for 2.13. See the examples that are created in Scala 3.

s0t00524 · 2024-11-11T07:03:45Z

@emartinezs44 Thank you for your information. I will check it.

helenlly added the user issue label Feb 2, 2023

qiuxin2012 self-assigned this Feb 2, 2023

shane-huang added the DLlib label Jan 22, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Compatibility with Spark 3.2 + Scala 2.13 #7399

Compatibility with Spark 3.2 + Scala 2.13 #7399

emartinezs44 commented Feb 1, 2023

qiuxin2012 commented Feb 2, 2023

jason-dai commented Feb 2, 2023

emartinezs44 commented Feb 2, 2023

qiuxin2012 commented Feb 3, 2023 •

edited

Loading

qiuxin2012 commented Feb 3, 2023

emartinezs44 commented Feb 3, 2023

emartinezs44 commented Feb 6, 2023 •

edited

Loading

qiuxin2012 commented Feb 6, 2023

qiuxin2012 commented Feb 7, 2023

emartinezs44 commented Feb 7, 2023 •

edited

Loading

qiuxin2012 commented Feb 13, 2023

emartinezs44 commented Feb 13, 2023

s0t00524 commented Aug 21, 2024

s0t00524 commented Oct 15, 2024

qiuxin2012 commented Oct 16, 2024

s0t00524 commented Nov 6, 2024

qiuxin2012 commented Nov 7, 2024

emartinezs44 commented Nov 7, 2024

s0t00524 commented Nov 11, 2024

Compatibility with Spark 3.2 + Scala 2.13 #7399

Compatibility with Spark 3.2 + Scala 2.13 #7399

Comments

emartinezs44 commented Feb 1, 2023

qiuxin2012 commented Feb 2, 2023

jason-dai commented Feb 2, 2023

emartinezs44 commented Feb 2, 2023

qiuxin2012 commented Feb 3, 2023 • edited Loading

qiuxin2012 commented Feb 3, 2023

emartinezs44 commented Feb 3, 2023

emartinezs44 commented Feb 6, 2023 • edited Loading

qiuxin2012 commented Feb 6, 2023

qiuxin2012 commented Feb 7, 2023

emartinezs44 commented Feb 7, 2023 • edited Loading

qiuxin2012 commented Feb 13, 2023

emartinezs44 commented Feb 13, 2023

s0t00524 commented Aug 21, 2024

s0t00524 commented Oct 15, 2024

qiuxin2012 commented Oct 16, 2024

s0t00524 commented Nov 6, 2024

qiuxin2012 commented Nov 7, 2024

emartinezs44 commented Nov 7, 2024

s0t00524 commented Nov 11, 2024

qiuxin2012 commented Feb 3, 2023 •

edited

Loading

emartinezs44 commented Feb 6, 2023 •

edited

Loading

emartinezs44 commented Feb 7, 2023 •

edited

Loading