Skip to content

yennanliu/utility_Scala

BranchesTags

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

650 Commits
.github/workflows
.github/workflows
 
 
archived
archived
 
 
data
data
 
 
doc
doc
 
 
project
project
 
 
script
script
 
 
src
src
 
 
.gitignore
.gitignore
 
 
Dockerfile
Dockerfile
 
 
README.md
README.md
 
 
build.sbt
build.sbt
 
 
build.sbt.dev
build.sbt.dev
 
 

Repository files navigation

Utility_Scala

Introduction of Scala programming language as well as Spark Scala via couples of basic scripts in common use cases. Please check the Main files for more information.

General Type

Immutable

Mutable

Scala Projects

Note

Main Files

  1. Utility Scala - Scala scripts for utility

  2. Spark Scala Demo - Scala spark basic demo

  3. Scala basic - variable, data structure

    • Scala_basic_demo_3 - Scala basic data structrue : array, list, tuple, apply function, lambda with them basic. And map, exception demo
    • Immutable & mutable
      • UNCHANGED : Immutable
        • example : scala.collection.Immutable
        • Scala has some Immutable class that java already has
          • e.g. : Set, Map
        • however, Scala has its own unique Immutable class
          • e.g. Seq
        • immutable.Seq.LinearSeq uses a lot : Queue, Stack ...
        • string belongs to immutable.Seq.IndexedSeq
        • there is a SortedMap under Map offering "ordering" feature
      • CAN CHANGED : mutable
        • example : scala.collection.mutable
      • Scala uses "immutable" by default
      • For most of the data structure, Scala offers both immutable and mutable versions
    • Array
    • ArrayBuffer
    • Array <--> ArrayBuffer
    • ArrayBuffer <--> Java List
    • Dimension Array
    • Tuple
      • TupleDemo1
      • can be recognized as a "container"
      • can save various elements with different/same type
      • tuple can only save 22 elements as MAX
    • List
      • ListDemo_1
      • ListDemo_2
      • ListDemo_3
      • Scala_yield_to_List
      • Scala list can storage data directly (a object); while in Java, list is an interface, the implementation is ArrayList
      • scala list can save any type of elements
      • if want to get a null list -> use Nil (e.g. val list2 = Nil)
      • Scala Lists are quite similar to arrays, which means all the elements of a list have the same type - but there are two important differences.
      • list are immutable, which means elements of a list cannot be changed by assignment.
      • By default, Scala List is immutable (unchanged)
      • list is ordering (order matters)
      • list adding element :
       val list_1 = List(1,2,4,"xxx")
 val list_2 = list_1 :+ 4
      • ListDemo_4
      • ListDemo_5
      • list adding element via ::, ::::
        • :: means adding new element to a list
        • we must put the list on the right of ::. (i.e. 4 :: list_1)
        • computing ordering : from right to list
        • ::: means add EVERY element in list to a null list
        • we must put the list on the right and left of :::. (i.e. list_1 ::: list_2 ::: list_3)
      • list represents a linked list whereas arrays are flat. The type of a list that has elements of type T is written as List[T].
    • Queue
    • Map
    • Set
      • collection of unique elements, no ordering, default is hashmap's implementation
      • Set is immuatable default in Scala, if want mutable, need import scala.collection.mutable.set
      • SetDemo_1
      • SetDemo_2
      • SetDemo_3
      • JavaSetDemo_1
    • Stream
      • stream is one kind of the collection that can storgae unlimit elements. However, the unlimit elements are not showed up at once, but with dynamic space, the last element in stream following the lazy style (only computed when called)
      • StreamDemo_1

  4. Scala basic - operation

  5. Scala basic - function, class, case class, constructor...

    • Class

    • Case class

      • Case Class
      • Case Class2
      • Case Class3 - pattern match example
      • Case Class4 - copy method demo
      • Case Class5 - Pattern match : case class nested structure, @ notation (wrap values in case class to variable)
      • Case Class6 - Pattern match : case class nested structure 2
      • Case Class7 - case class enumeration
      • Case Class8 - case class enumeration
      • Case Class9 - case class with user defined class
      • case class is still class
      • decorate class via case
      • for pattern match purpose
      • it offers apply method so we don't need to new case class (instantiate), but can use it directly
      • it offers unapply method so pattern match can work
      • will implement some methods by default : e.g. apply, unapply, toString, copy, hashCode....
      • Difference between class and case class
        • => case class can initiate attr, value, structure when instantiated
      • it's not necessary that case class to inherent from other class
      • apart from above, case class is as same as class, we can extend, rewrite it
      • implements serialization by default
      • implements apply method by default

    • Sealed class

      • Seal demo 1
      • sealed class can only be extended in the same class/object

    • ClassPolymorphismDemo1 - Scala class Polymorphism Demo

    • ScalaDefaultValueDemo1 - Scala Scala Default Value

    • ScalaIfElseMatchForloop - Scala if, else, while, match basic

    • Function

    • Scala_basic_demo_4 - Scala try, exception, error handling demo

    • ScalaFileIODemo, ScalaFileIODemo2, ScalaFileIODemo3 - Scala file IO basic

    • ScalaLazyDemo - Scala lazy value demo

    • ScalaUnitDemo - Scala function with no return value called as procedure, with return type Unit

    • Override

    • Higher Order function

    • Closure

      • closure_demo1
      • Closure is a combination of a func and its relative references
      • Benefit : can reuse the argument in method, so we don't need to re import same arguments everytime when a run the method
      • pattern
       // pattern
   def minusxy(x: Int) = {
     (y: Int) => x - y // anonymous func
   }

    • Curry

    • Control Abstract

    • RecursionDemo - Scala Recursion basic op, properties

    • Constructor

      • Constructor can define value to its attribution when create the object
      • can has as many as posssible constructor methods in scala
      • "main constructor", "support constructor"
      • scala constructor format
       class ClassName{parameters}{ // main constructor
 	// code
 	//
 	def this(parameters){//support constructor
 	}
 	def this(parameters){// can have multiple support constructors ...
 	}
 }

    • Patternmatch

       def showImportantNotification(notification: Notification, importantPeopleInfo: Seq[String]): String = {
   notification match {
     case Email(sender, _, _) if importantPeopleInfo.contains(sender) =>
       "You got an email from special someone!"
     case SMS(number, _) if importantPeopleInfo.contains(number) =>
       "You got an SMS from special someone!"
     case other =>
       showNotification(other) // nothing special, delegate to our original showNotification function
   }
 }
      • pattern match variable : can get variable in pattern match -> can be used in next steps op
      • Note : if there is a case _ exists in the middle of match -> meaning the pattern match will NOT to compare the value, but still do the type compare. (this case is NOT match any case)
      • example:
       val result = obj match{
 	case a: Int => a
 	// NOTICE here
 	case _ => Int.MaxValue
 }

    • This

    • Some

      • SomeDemo
      • Scala Some basic op, properties

    • CaseHeadTailNil - Scala Case on List Head Tail Nil basic op, properties

    • TryGetOrElse, TryGetOrElse2 - Scala try GetOrElse(else) example

    • UpperCass - Scala UpperCass ( <: ) basic op, properties

    • Find - Scala Find, exists, contains, and isDefined examples

    • Partial Function

          // V1
    // Any : input type  (in this example)
    // Int : output type (in this example)
    val addOne_2 = new PartialFunction[Any, Int] {
      override def isDefinedAt(any: Any): Boolean = {
        if (any.isInstanceOf[Int]) true else false
      }
      override def apply(any: Any) = {
        any.asInstanceOf[Int] + 1
      }
    }
    val ans1 = List(1,2,3,4,"ABC").collect(addOne_2)
    // V2
    def f2:PartialFunction[Any, Int]{ // NOTE : we need this : PartialFunction
    	case i:Int => i + 1  // case can transform to partial func automatically
    }
    val ans2 = List(1,2,3,4,"ABC").collect(f2)
    // V3
    val ans3 = List(1,2,3,4,"ABC").collect{case i:Int = > i + 1}
      • can only do op in specific cases or defined variable type
      • In scala, Partial func is the subclass of trait : PartialFunction

    • Anomaly Func

    • Type Inference

      • typeInference_1
      • argument type can be neglected if type is inferenced
      • when there is only one argument, we can neglect the ()
      • if the argument only shows once after =>, we can use _ as argument

    • Bean

    • Type Convert

      • TypeConvert1
      • TypeConvert2- Scala type convert : upper transform, lower transform : asInstanceOf, check if object's class type : classOf, getClass.getName

    • example

     // *** declare a Employer_03 instance, but transform it to Person_03 class
 val emp:Person_03 = new Employer_03
 // *** transform emp to Employer_03 class (asInstanceOf)
 emp.asInstanceOf[Employer_03].sayHello() // downward transform

  6. Scala trait, abstract class..

    • Trait
      • Trait
      • TraitAbstract
      • TraitDemo2
      • TraitDemo3
      • TraitDemo4
      • TraitDemo5
      • TraitDemo6
      • TraitDemo7
      • TraitDemo8
      • TraitDemo9
      • TraitDemo10
        • Compare the ordering with trait construct ways
          • way 1) create class instance -> class hasn't been created yet when mixing trait
          • way 2) create "anonymous" sub class -> class already been created when mixing trait
      • TraitDemo11
      • TraitDemo12
      • TraitDemo13
      • TraitAsInterface
      • TraitAbstractOverwrite1
      • Scala trait : a "supplement" of scala Inheritance (scala only allows "single Inheritance", e.g. a class can only have one parent class), so trait offers more flexibility
      • Scala trait can do sth like "java interface"
      • trait can have both abstract method and regular method (a method that Not implemented in trait is the "abstract" method)
      • ALL Java interface can be used in scala as Scala trait
      • if there are "composition" traits (class className extends trait1 with trait2 with trait3 ...) when creating, then scala will
        • declare the instance from left to right
        • execute the method from right to left
      • Scala TraitDemo, Trait Abstract basic op, properties. NOTICE : Scala traits don’t allow constructor parameters (so that's why we use abstract class)
      • pattern
       // if no parent class
 class className extends trait1 with trait2 with trait3 ...
 //
 // if there is parent class
 class className extends parentclassName with trait1 with trait2 with trait3 ...
      • TraitMixin1
      • TraitMixin2
      • TraitMixin3
      • TraitMixin:
        • Can mixin trait when construct the object (class) => to extend functionality
        • also can implement on abstract class
        • TraitMixin only works in Scala (not Java)
        • Can do the extension but not change inheritance status
    • Abstract Class
      • Abstract Class
      • AbstractDemo
      • Scala also has a concept of an abstract class that is similar to Java’s abstract class. But because traits are so powerful, you rarely need to use an abstract class. In fact, you only need to use an abstract class when:
      • You want to create a base class that requires constructor arguments
      • Your Scala code will be called from Java code
    • Class VS Object 1, Class VS Object 2 - Compare Class, object difference, feature in Scala
    • Type Parameterlization Demo1 - Type Parameterlization in Scala
    • Apply
    • anonymousClass
    • Nest class 
       // java example 1
 class Outer{  // outer class
 	class inner{  // inner class
 // code
 	}
 }
 class Other{ // outer other class
 // code
 }
       // java example 2
 class OuterClass{  // outer class
 	class InnerClass{ // inner class
 		publice void test (InnerClass ic){
 			System.out.Prlintln(ic);
 		}
 	}
 	static class StaticInnerClass {  // static inner class
 		// code
 	}
 }

  7. Scala object

    • LoadPackageDemo1, LoadPackageDemo2, LoadPackageDemo3, LoadPackageDemo4 - Scala load package examples

    • pkgObject - package can have class, object, and trait... but it CAN'T HAVE function, var... In order to solve it, scala offers the package object concept

    • Singleton pattern

    • Companion

      • CompanionDemo1
      • CompanionDemo2
      • CompanionDemo3
      • CompanionDemo4
      • CompanionDemo5
      • Companion is the combinaton of class static method/value.. + class basic method/value..
      • Scala Companion demo (An object that has the same name as a class is called a companion object of the class, and it is often used to contain factory methods for the class that it complements)
      • Since there is no static class/method.. in Scala, so Scala offers the Companion for similIar functionality
      • In development, we put the basic attribution, method ... in Companion class ; and we put the static consents in the Companion object

    • VisibilityDemo1 - extend package visibility. e.g. : private[ScalaBasic] val name = "jackkkk"

    • ImportPackage1 - Scala import package demo

  8. Scala implicit

    • implicit is the way that you dont need to pass parameters explicitly in functions in Scala, but Scala will be able to find them from the implitict scope once you defined them. Use implicit can make your function more general and easy to import/deal with different cases per pattern
    • Implicit_Demo1
    • Implicit_Demo2
    • Implicit_Demo3
    • Implicit_Demo4 : implicit parameter
    • Implicit_Demo5
    • Implicit_Demo6
    • Implicit_Demo7
    • Implicit_Demo8
    • Implicit_Demo9
    • more implicit demos
    • Implicit_1
    • Implicit_2
    • Implicit_3
    • Implicit_4 - implicit class demo
      • More implicit demos from utube tutorial
      • implicit func ONLY ACCEPTS ONE ARGUMENT
      • the implicit func will BE USED AUTOMATICALLY WHEN THERE IS SUCH CASE
      • the NAME of implicit func IS NOT MATTER; INPUT AND OUTPUT DTYPE IS THE (IMPORTANT) THING
      • there can be multiple implicit func, but have to make sure THERE IS ONLY ONE implicit func CAN BE REFERRED IN CURRENT STATE
      • Note : implicit CAN'T IN "NEST" FORMAT (e.g. one implicit in the other implicit)
      • AVOID the case : "FAILED TO FIND IMPLICIT IN CURRENT FIELD, THEN SEARCH ALL POSSIBILITIES (same type) IN THE SPACE"
    • implicit_transformation_demo_1, implicit_transformation_demo_2 - implicit transformation can automatically transform "high accuracy" val to "low accuracy". e.g. : Byte -> Short, Short -> Int, Int -> Long, Long -> Float, Char -> Int ..
    • ImplicitParameters, ImplicitFunc, ImplicitClass - Scala implicit in Parameters, func, class

  9. Scala OOP

    • Features : Encapsulation, Inheritance, Polymorphism

    • AbstractDemo1

    • encapsulationDemo1, encapsulationDemo2

      • encapsulation is one of the features in OOP, abstract the data and methods and encapsulate them, so only the "defined" method can be implimented to the data
      • Pros on encapsulation
        • hide the implementation details
        • validate the data, make it safe and feat the business needs
      • Steps do encapsulation:
        • step1) "private" the method/attributions
        • step2) offer "public" methods (e.g. : getter, setter or @BeanProperty...) (getter : set attribution value, getter : get attribution value)

    • OOP hw 1

  10. Scala advance

  11. Scala Design Pattern

    • Decorator

    • Factory

      • Abstract the implemented classes, and put them into a class for better code maintenance, and management.
      • We DON'T instantiate the subclass directly, but we instantiate its factory.
      • 3 types of factory
        • simple factory
        • factory method
        • abstract factory
      • Simple Factory
      • Factory Method
        • FactoryMethodDemo1
        • Make instantiate methods as abstract methods, and implement in subclass
        • Define the abstract method for instantiating object, and let subclass decide with class need to instantiate
        • Delay class instantiation to subclass
      • Abstract Factory
        • AbstractFactoryDemo1
        • Define a trait for creating relative dependent class
        • Integrate Simple Factory and Factory method
        • an improvement on Simple Factory (design)
        • Abstract the factory to 2 layers
          • AbsFactory
          • ActualImplementSubcass
      • Singleton
        • SingletonDesignPattern1
        • Make sure THERE IS ONLY 1 INSTANCE in the software system
        • Use cases
          • SessionFactory in Hibernate
          • ActorSystem in Akka
      • Decorator
      • Observer
      • Proxy

  12. Scala Script example

  13. Scala exception, error handling

    • ExceptionDemo1
    • ExceptionDemo2 - format : try - catch - finally
      • there is NO compile exception in Scala (only java has), all exceptions in Scala happen during runtime. All exceptions inherit from the "throwable" class, which is the class with "nothing" type that is acceptable to all class/method...
    • ExceptionDemo3

  14. Backend framework - Akka

    • AkkaDemo1
    • AkkaDemo2
    • AkkaDemo3 : YellowChicken
    • AkkaDemo4 : SparkMasterWorker
    • High level
      • Akka is a framework for concurrent and distributed applications on the JVM.
      • Akka supports multiple programming models for concurrency
      • Akka offers Scala, Java API
      • All we need to implement is : Actor. Akka framework will take care rest of them
    • Main component
      • Actor
        • In Akka, everything is an Actor (like OOP, everything is an object)
        • Actor is for concurrent
        • Actor - Actor communication can only via "Mailbox"
        • messages are storaged in the message queue, and will be post into the Mailbox
        • Actor can do sync or async operation
        • Actor receives message via receive method
      • ActorSystem
        • manage/create Actor
        • ActorSystem is singleton
        • one JVM can only has one ActorSystem, but it can has multiuple Actor
      • ActorRef
        • Actor representation or reference
        • message are sent via ActorRef (not Actor)
      • Dispatcher message
        • message pool
        • Follow FIFO (first in, first out)
      • Mailbox
        • managed by Akka, not exposed to developer/user
        • implement runnable JVM object
    • Summary
      • Akka will created an Actor Mailbox instance, it's a queue, can get msg from Dispatcher message
      • Mailbox implement Runnable trait, it's a thread, will keep running and use Actor's receive method, so when Dispatcher send msg to Mailbox, Actor can get it via its receive method
      • ActorRef | "HELLO" means send hello msg to A Actor's Mailbox
    • Mechanism
    // Actor 1 -> Actor 2
Actor 1 -> ActorRef 1 -> Dispatcher message -> Mailbox -> receive (Actor 2) -> Actor 2

// Actor 2 -> Actor 1
Actor 2 -> ActorRef 2 -> Dispatcher message -> Mailbox -> receive (Actor 1) -> Actor 1

  1. Others

Quick Start

Quick start manually 
# DEMO 1) run scala hello world 
$ git clone https://github.com/yennanliu/utility_Scala.git
$ cd utility_Scala
$ scala src/main/scala/UtilityScala/HelloWorld.scala 
#$ Hello World


# DEMO 2) run scala spark hello world via sbt 
$ cd utility_Scala
$ sbt package
$ sbt
# inside sbt console
sbt:Simple Project> run 
# [warn] Multiple main classes detected.  Run 'show discoveredMainClasses' to see the list

# Multiple main classes detected, select one to run:

#  [1] AnonymousFuncDemo
#  [2] ClassDemo
#  [3] FileIODemo
#  [4] ForLoopDemo
#  [5] FunctionChangeableParameterDemo
#  [6] FunctionCompositionDemo
#  [7] HelloWorld
#  [8] IfElseDemo
#  [9] OperatorDemo
#  [10] PatterMatchDemo
#  [11] SimpleApp
#  [12] Test
#  [13] UderDefinedDefaultParamFuncDemo
#  [14] UderDefinedFuncDemo

Enter number: 11

# [info] Running SimpleApp 
# ...
#  >>>>>>>>>>>>>> OUTPUT
# Lines with a: 21, Lines with b: 9
#  >>>>>>>>>>>>>> OUTPUT
# ...

# DEMO 3) run scala spark hello world
$ cd utility_Scala
$ sbt clean compile && sbt assembly 
$ spark-submit \
  --class "SimpleApp" \
  --master local[4] \
  target/scala-2.11/simple-project_2.11-1.0.jar

# REPL via sbt console
$ sbt
console
scala> 

# ✘ yennanliu@MacBook-Pro  ~/utility_Scala   master ●  
# ✘ yennanliu@MacBook-Pro  ~/utility_Scala   master ●  sbt  
# [info] Loading settings for project utility_scala-build from plugins.sbt ...
# [info] Loading project definition from /Users/yennanliu/utility_Scala/project
# [info] Loading settings for project utility_scala from build.sbt ...
# [info] Set current project to UtilityScala (in build file:/Users/yennanliu/utility_Scala/)
# [info] sbt server started at local:///Users/yennanliu/.sbt/1.0/server/ff2f518f2235c5fb0743/sock
# sbt:UtilityScala> console
# [info] Starting scala interpreter...
# Welcome to Scala 2.11.8 (OpenJDK 64-Bit Server VM, Java 1.8.0_252).
# Type in expressions for evaluation. Or try :help.

# scala> import slick.driver.H2Driver.api._
# import slick.driver.H2Driver.api._

# scala>
Quick start via java -cp 
sbt assembly

export env=dev

# example 1
java -cp \
./target/scala-2.11/utilityscala_2-0.0.1.jar \
LoadConfigsFromEnv.runWithEnv1 \
-Dconfig.resource=application.${env}.conf

# exampl 2
java -cp target/scala-2.11/utilityscala_2.11-1.0.jar  ScalaBasic.CaseClass6
Quick start via Docker 
$ git clone https://github.com/yennanliu/utility_Scala.git
$ cd utility_Scala
$ docker build . -t spark_env
$ docker run  --mount \
type=bind,\
source="$(pwd)"/.,\
target=/utility_Scala \
-i -t spark_env \
/bin/bash
Quick start via `Spark-submit` 
# package the scala saprk scripts
$ sbt package
# list the current classes
$ ls target/scala-2.11/classes
# run ForLoopDemo
$ spark-submit \
  --class ForLoopDemo \
  target/scala-2.11/utilityscala_2.11-1.0.jar 
# run LambdaFuncDemo
$ spark-submit \
  --class LambdaFuncDemo \
  target/scala-2.11/utilityscala_2.11-1.0.jar 
# run spark_basic_demo_4
$ spark-submit \
  --class SparkBasic.spark_basic_demo_4 \
  target/scala-2.11/utilityscala_2.11-1.0.jar
# run MovieSimilarities
$ spark-submit \
  --class SparkBasic.MovieSimilarities \
  target/scala-2.11/utilityscala_2.11-1.0.jar 50

Development

  • Trouble shooting

  • Clean cache (sbt) : in case when there is issue build the project via IntelliJ sbt

ls -al
rm .idea

scala-learn-material

scala-learn-material

Ref

Ref

About

Scala programming language basic & functional programming (fp) & and other utility cases | #SE

Topics

scala sbt functional-programming jvm oop fp scala-learning

Resources

Readme
Activity

Stars

3 stars

Watchers

1 watching

Forks

2 forks
Report repository

Releases 2

V20210723 Latest
Jul 23, 2021
+ 1 release

Packages

No packages published

Languages