Skip to content

The Crawler now registers itself at the instance registry #24

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 22 commits into from
Sep 19, 2018
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
6e6a675
Implemented a Preflight check connecting to the Instance Registry
Aug 1, 2018
7edf235
ElasticSearch instance can now be set to the one responded by th IR
Aug 7, 2018
ab2bce3
Included client API code from swagger to connect to IR
Aug 21, 2018
bb3d605
Used akka http to call IR instead of outdated spray http
Aug 28, 2018
c2c9801
Removed unused swagger code, used logger where println was used
Aug 28, 2018
3439480
Crawler can now get elastic search ip from IR
Aug 28, 2018
bf368e8
Cleaned up unused dependencies, fixed some minor bugs
Sep 1, 2018
ba109f6
Crawler now reads its assigned IP after registration at the Instance …
Sep 1, 2018
c67ce57
Made posting matching-result work by storing the matched ElasticSearc…
Sep 2, 2018
51fe0b1
Crawler is now deregistering itself from the Instance Registry on shu…
Sep 4, 2018
d9111f6
Merge remote-tracking branch 'origin/develop' into feature/instancere…
Sep 4, 2018
0d6b1a9
Code style cleanup
Sep 5, 2018
18a71a0
Restored setting of Hermes config, fails on Linux and Windows
Sep 7, 2018
cc45162
CodeStyle: Replaced .get with .getOrElse
Sep 7, 2018
81e8482
Moved default host to val, removed unused resolver/unnecessary condition
Sep 8, 2018
10fb5bc
Made class 'Instance' not use Options anymore
Sep 9, 2018
1e8ec5e
Fixed shutdown hook not being triggered, fixed port of IR
Sep 9, 2018
b2458f0
Better handling of getMatchingInstance returning 404
Sep 12, 2018
c2d86be
Adapted IR communication to use new attribute names
Sep 12, 2018
3372a89
CodeStyle: Replaced if-else if with match-case
Sep 14, 2018
d66bf31
Merge branch 'develop' into feature/instanceregistry
Sep 19, 2018
44462af
Fixed merge error (missing comma)
Sep 19, 2018
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Code style cleanup
  • Loading branch information
Johannes Duesing committed Sep 5, 2018
commit 0d6b1a9a979bdba3e5021d586229fda188960cc8
11 changes: 8 additions & 3 deletions src/main/scala/de/upb/cs/swt/delphi/crawler/Configuration.scala
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,12 @@ class Configuration {

lazy val elasticsearchInstance : Instance = InstanceRegistry.retrieveElasticSearchInstance(this) match {
case Success(instance) => instance
case Failure(_) => Instance(None, Some(sys.env.getOrElse("DELPHI_ELASTIC_URI","elasticsearch://localhost:9200")), None, Some("Default ElasticSearch instance"), Some(ComponentType.ElasticSearch) )
case Failure(_) => Instance(
None,
Some(sys.env.getOrElse("DELPHI_ELASTIC_URI","elasticsearch://localhost:9200")),
None,
Some("Default ElasticSearch instance"),
Some(ComponentType.ElasticSearch))
}

val mavenRepoBase: URI = new URI("http://repo1.maven.org/maven2/") // TODO: Create a local demo server "http://localhost:8881/maven2/"
Expand All @@ -55,12 +60,12 @@ class Configuration {
val instanceName = "MyCrawlerInstance"
val instanceRegistryUri : String = sys.env.getOrElse("DELPHI_IR_URI", "http://localhost:8085")

lazy val usingInstanceRegistry = assignedID match {
lazy val usingInstanceRegistry : Boolean = assignedID match {
case Some(_) => true
case None => false
}

lazy val assignedID = InstanceRegistry.register(this) match {
lazy val assignedID : Option[Long] = InstanceRegistry.register(this) match {
case Success(id) => Some(id)
case Failure(_) => None
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -18,24 +18,26 @@ package de.upb.cs.swt.delphi.crawler.instancemanagement

import java.net.InetAddress

import akka.actor.ActorSystem
import akka.http.scaladsl.Http
import akka.http.scaladsl.marshalling.Marshal
import akka.http.scaladsl.model._
import akka.http.scaladsl.unmarshalling.Unmarshal
import akka.stream.ActorMaterializer
import de.upb.cs.swt.delphi.crawler.{AppLogging, Configuration, Crawler}
import de.upb.cs.swt.delphi.crawler.io.swagger.client.model.InstanceEnums.ComponentType
import de.upb.cs.swt.delphi.crawler.io.swagger.client.model.{Instance, JsonSupport}

import scala.concurrent.duration.Duration
import scala.concurrent.{Await, Future}
import scala.concurrent.{Await, ExecutionContext, Future}
import scala.util.{Failure, Success, Try}

object InstanceRegistry extends JsonSupport with AppLogging
{

implicit val system = Crawler.system
implicit val ec = system.dispatcher
implicit val materializer = Crawler.materializer
implicit val system : ActorSystem = Crawler.system
implicit val ec : ExecutionContext = system.dispatcher
implicit val materializer : ActorMaterializer = Crawler.materializer


def register(configuration: Configuration) : Try[Long] = {
Expand Down Expand Up @@ -102,7 +104,9 @@ object InstanceRegistry extends JsonSupport with AppLogging
Failure(new RuntimeException("Cannot post matching result to Instance Registry, assigned ElasticSearch instance has no ID."))
} else {
val IdToPost = configuration.elasticsearchInstance.iD.get
val request = HttpRequest(method = HttpMethods.POST, configuration.instanceRegistryUri + s"/matchingResult?Id=$IdToPost&MatchingSuccessful=$isElasticSearchReachable")
val request = HttpRequest(
method = HttpMethods.POST,
configuration.instanceRegistryUri + s"/matchingResult?Id=$IdToPost&MatchingSuccessful=$isElasticSearchReachable")

Await.result(Http(system).singleRequest(request) map {response =>
if(response.status == StatusCodes.OK){
Expand Down Expand Up @@ -157,5 +161,6 @@ object InstanceRegistry extends JsonSupport with AppLogging
}


private def createInstance(id: Option[Long], controlPort : Int, name : String) : Instance = Instance(id, Option(InetAddress.getLocalHost().getHostAddress()), Option(controlPort), Option(name), Option(ComponentType.Crawler))
private def createInstance(id: Option[Long], controlPort : Int, name : String) : Instance =
Instance(id, Option(InetAddress.getLocalHost.getHostAddress), Option(controlPort), Option(name), Option(ComponentType.Crawler))
}
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ import spray.json._

trait JsonSupport extends SprayJsonSupport with DefaultJsonProtocol {
implicit val componentTypeFormat = new JsonFormat[InstanceEnums.ComponentType] {
def write(compType : InstanceEnums.ComponentType) = JsString(compType.toString())
def write(compType : InstanceEnums.ComponentType) = JsString(compType.toString)

def read(value: JsValue) = value match {
case JsString(s) => s match {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -46,13 +46,13 @@ object ElasticReachablePreflightCheck extends PreflightCheck {
val f = (client.execute {
nodeInfo()
} map { i => {
if(configuration.usingInstanceRegistry) InstanceRegistry.sendMatchingResult(true, configuration)
if(configuration.usingInstanceRegistry) InstanceRegistry.sendMatchingResult(isElasticSearchReachable = true, configuration)
Success(configuration)
}
} recover { case e => {
if(configuration.usingInstanceRegistry) InstanceRegistry.sendMatchingResult(false, configuration)
} recover { case e =>
if(configuration.usingInstanceRegistry) InstanceRegistry.sendMatchingResult(isElasticSearchReachable = false, configuration)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Having the condition at all places where it is used can be dangerous.
Could we always make the call and decide this in the method? (or through inheritance)

Failure(e)
}

}).andThen {
case _ => client.close()
}
Expand Down