Skip to content

Conversation

@josepablocam
Copy link
Contributor

The original call vec(startLoc until endLoc) on a Vector[T] returns a SliceVector[Int, T] (as shown below). SliceVector[Int, T] does not inherit from Serializable as per breeze docs.

Only relevant parts of REPL echos are kept below:

scala> import breeze.linalg._
scala> val dv: DenseVector[Double] = DenseVector.zeros[Double](5)
scala> val v: Vector[Double] = DenseVector.zeros[Double](5)
scala> dv(0 until 5)
res6: breeze.linalg.DenseVector[Double] = DenseVector(0.0, 0.0, 0.0, 0.0, 0.0)
scala> v(0 until 5)
res7: breeze.linalg.SliceVector[Int,Double] = breeze.linalg.SliceVector@7daa0333
scala> sc.parallelize(Array(dv(0 until 5))).collect() // works
scala> sc.parallelize(Array(v(0 until 5))).collect() // fails
15/08/14 11:59:09 ERROR TaskSetManager: Failed to serialize task 47, not attempting to retry it.
java.lang.reflect.InvocationTargetException
...

This means that if a user currently tries to use a function in the public API that calls this private helper, it results in an exception. For example, calling slice, on a TimeSeriesRDD with a uniform index, which calls this function. The cast to DenseVector solves this issue.

(Additionally, this issue is masked by setting the Kryo serializer, since that seems to get around this issue... I'm not clear on why that is, but it seems reasonable to think that the user shouldn't have to do that to be able to use).

jose.cambronero and others added 30 commits June 12, 2015 11:40
A couple of simple changes to get started: corrected
added EWMA functionality and tests
…evant tests), added missing type to EasyPlot functions
cdalzell and others added 29 commits July 16, 2015 19:29
Return figures in plotting functions
added option to drop intercept in AR(n) model
…clear if correctly implementing that function, added test data
… Sandy's feedback. Modified forecast to take same approach (i.e. use AR of intercept and MA of 0.0 prior to start of series). Added tests and modified leniency on intercept term
…change was breaking up (p, d, q) as separate ARIMAModel members
…ameters, fixed typo in hannan, moved around arguments in functions in ARIMA object for consistency
…cordingly. Added 2 new functions to UnivariateTimeSeries to perform differencing of order-d, rather than differencing at lag-d
…calls to differences of lag, removed unnecessary copying of vector in differences of order
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants