Skip to content

Integrate progress #40

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 106 commits into from
May 28, 2019
Merged
Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
106 commits
Select commit Hold shift + click to select a range
1591a05
Nonfunctional partial implementation of call graph generation
almacken Jun 21, 2018
63e480e
"Working" implementation of call graph generation
almacken Jul 4, 2018
02cd16d
Call graph generation much more efficient, can handle larger samples.
almacken Jul 12, 2018
d4e49eb
Merge branch 'feature/streamworkaround' of https://github.com/delphi-…
almacken Jul 17, 2018
8c27117
Merge remote-tracking branch 'origin/develop' into develop
almacken Jul 25, 2018
6f8812b
Merge remote-tracking branch 'origin/develop' into develop
almacken Jul 26, 2018
afc2e22
Implemented loading Jars from remote streams, misc. bug fixes
almacken Jul 26, 2018
6e6a675
Implemented a Preflight check connecting to the Instance Registry
Aug 1, 2018
301ca2f
Stream gracefully drops any error-causing elements
almacken Aug 2, 2018
7edf235
ElasticSearch instance can now be set to the one responded by th IR
Aug 7, 2018
738f1b9
Merge remote-tracking branch 'origin/develop' into develop
almacken Aug 7, 2018
5e3583f
Invalid version numbers are now interpreted as the latest version BEF…
almacken Aug 9, 2018
e51d865
Updated badges
bhermann Aug 11, 2018
12a3208
Fixed badge code
bhermann Aug 11, 2018
331f8e9
Fixed badge code
bhermann Aug 11, 2018
f32329e
Fixed badge code
bhermann Aug 11, 2018
a967033
Improved stream structure
almacken Aug 13, 2018
db32618
Added license information
bhermann Aug 14, 2018
68b8689
Added documentation
bhermann Aug 14, 2018
faa03c2
Removed unnecessary output
bhermann Aug 14, 2018
e5145c9
Removed magic number
bhermann Aug 14, 2018
9777aee
Refactoring for storage queries
bhermann Aug 14, 2018
596f37e
Refactoring Hermes processing strain
bhermann Aug 15, 2018
ce56267
Added OPAL configuration
bhermann Aug 15, 2018
5d40624
Code quality
bhermann Aug 15, 2018
4078f9a
Refactored HermesActor
bhermann Aug 15, 2018
ab2bce3
Included client API code from swagger to connect to IR
Aug 21, 2018
c010661
Added some comments describing implementation of call graph stream
almacken Aug 22, 2018
7da8226
Merge branch 'develop' into feature/callGraphGeneration
almacken Aug 23, 2018
d277fce
Fixed some errors with ElasticClient, still going to improve code qua…
almacken Aug 24, 2018
9665292
Migration to elastic4s 6.3.0
bhermann Aug 25, 2018
654e612
Improved code quality
almacken Aug 27, 2018
bb3d605
Used akka http to call IR instead of outdated spray http
Aug 28, 2018
c2c9801
Removed unused swagger code, used logger where println was used
Aug 28, 2018
3439480
Crawler can now get elastic search ip from IR
Aug 28, 2018
b979f5f
Further improved code quality
almacken Aug 28, 2018
3a1b456
Merge pull request #22 from almacken/feature/callGraphGeneration
bhermann Aug 29, 2018
bf368e8
Cleaned up unused dependencies, fixed some minor bugs
Sep 1, 2018
ba109f6
Crawler now reads its assigned IP after registration at the Instance …
Sep 1, 2018
c67ce57
Made posting matching-result work by storing the matched ElasticSearc…
Sep 2, 2018
51fe0b1
Crawler is now deregistering itself from the Instance Registry on shu…
Sep 4, 2018
d9111f6
Merge remote-tracking branch 'origin/develop' into feature/instancere…
Sep 4, 2018
0d6b1a9
Code style cleanup
Sep 5, 2018
18a71a0
Restored setting of Hermes config, fails on Linux and Windows
Sep 7, 2018
cc45162
CodeStyle: Replaced .get with .getOrElse
Sep 7, 2018
81e8482
Moved default host to val, removed unused resolver/unnecessary condition
Sep 8, 2018
10fb5bc
Made class 'Instance' not use Options anymore
Sep 9, 2018
1e8ec5e
Fixed shutdown hook not being triggered, fixed port of IR
Sep 9, 2018
b2458f0
Better handling of getMatchingInstance returning 404
Sep 12, 2018
c2d86be
Adapted IR communication to use new attribute names
Sep 12, 2018
3372a89
CodeStyle: Replaced if-else if with match-case
Sep 14, 2018
2cf291c
Bumped akka-http version b/c of CVE-2018-16131
bhermann Sep 17, 2018
d66bf31
Merge branch 'develop' into feature/instanceregistry
Sep 19, 2018
44462af
Fixed merge error (missing comma)
Sep 19, 2018
9a2a5b6
Merge pull request #24 from delphi-hub/feature/instanceregistry
bhermann Sep 19, 2018
c63c1c7
Implementation of Stop crawler instance refs #25
Ayybeeshafi Sep 25, 2018
5dc4761
Implementation of stop functionalty of Crawler instance refs #25
Ayybeeshafi Oct 2, 2018
3cd4fc8
Upgraded to new IR API model. Now using spray (de-)serialization.
Oct 7, 2018
858de50
Crawler is now reporting Failure when preflight checks fail, instead …
Oct 8, 2018
5523921
Cleanup of unused code and imports.
Oct 9, 2018
35b656f
Merge branch 'stop-crawler-instance' into feature/newIRApi
Oct 9, 2018
8eca4f8
Refactored Instance model to another package... cleaner like this
bhermann Oct 15, 2018
8ebba09
Orrr...
bhermann Oct 15, 2018
7a77a31
Added new process scheduler
bhermann Oct 16, 2018
72fbf3f
Stable pipeline
bhermann Oct 17, 2018
3ad504e
Adapted test for new ask-style (not pretty... but it works)
bhermann Oct 17, 2018
688c64d
Merge branch 'develop' into feature/newIRApi
Oct 17, 2018
d7a6f9d
Merge branch 'develop' into feature/newIRApi
Oct 18, 2018
eca1872
Renamed file for Actor Signals
bhermann Oct 19, 2018
60d9059
Changed base image for crawler dockerization
bhermann Oct 19, 2018
3a4e0f1
Workaround for Hermes' need for a physical configuration file
bhermann Oct 19, 2018
9e8abc6
Added more detailed error information in case the pre-flight check fa…
bhermann Oct 19, 2018
f47cd97
Merge branch 'develop' into feature/newIRApi
Oct 19, 2018
3435d2e
Codestyle cleanup
Oct 19, 2018
875e55d
Added timeout with regard to issue #27
bhermann Oct 20, 2018
6f3bca9
Added backoff restart to maven discovery source with regard to issue #27
bhermann Oct 20, 2018
553f54a
Added discovery timestamp
bhermann Oct 20, 2018
bf3edee
Added Hermes Version to the index
bhermann Oct 20, 2018
cf2e664
Added JavaFX requirement to README.
bhermann Oct 20, 2018
acba0c9
Corrected grammar mistakes
bhermann Oct 21, 2018
31989ea
Merge remote-tracking branch 'origin/develop' into feature/newIRApi
Oct 21, 2018
66353b5
Replaced usage of 'try-catch' with 'Try' (codestyle)
Oct 21, 2018
2407bc0
Merge pull request #26 from delphi-hub/feature/newIRApi
bhermann Oct 21, 2018
97888f4
Pinned jackson-databind now... *sigh*
bhermann Oct 21, 2018
dbb55ad
Activated configuration reading. Should work for all systems now.
bhermann Oct 22, 2018
fdb8c6d
Added encoding to parts of Maven URLs in order to deal with irregular…
bhermann Oct 23, 2018
fecc237
Made read operation more resilient. Closes #29.
bhermann Oct 24, 2018
bbd97ad
Adapted instance registry client to reflect the interface change in e…
Nov 4, 2018
d67329a
Adapted instance model to reflect latest changes: attribute 'labels' …
Nov 5, 2018
6e72f48
Merge pull request #30 from delphi-hub/feature/newMatchingInterface
bhermann Nov 8, 2018
abac46b
Adapt crawler to latest api changes
Nov 22, 2018
d9bdabb
Merge pull request #33 from delphi-hub/feature/linksAsAttributes
bhermann Nov 24, 2018
03fbf83
Introduced JWT based authorization for registry communication
Jan 3, 2019
9331aee
Codestyle: Inserted spaces before plus sign
Jan 4, 2019
e9b8daf
Merge pull request #35 from delphi-hub/feature/authentication
bhermann Jan 4, 2019
e668109
Adapted registry interface to newest API version
Jan 22, 2019
9c8e200
Fixed variable name being uppercase (codestyle)
Feb 7, 2019
1ac5941
Merge pull request #36 from delphi-hub/feature/newRegistryAPI
bhermann Feb 7, 2019
d23256c
Updated the jackson version to 2.9.9 to fix the dependency vulnerabil…
SViktorS May 25, 2019
1e498b5
Adding packages to travis config
bhermann May 28, 2019
3095278
Moving to Oracle JDK 8
bhermann May 28, 2019
30c39ff
bumping to 9 *sigh*
bhermann May 28, 2019
e47f243
Guessing tbh
bhermann May 28, 2019
cfdb10a
Merge branch 'develop' into feature/jacksonVulnerability
bhermann May 28, 2019
1ca63b1
Merge pull request #38 from delphi-hub/feature/jacksonVulnerability
bhermann May 28, 2019
05ee37d
Merge branch 'master' into develop
bhermann May 28, 2019
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ import java.net.URI
import akka.stream.ThrottleMode
import com.sksamuel.elastic4s.ElasticsearchClientUri
import de.upb.cs.swt.delphi.crawler.instancemanagement.InstanceEnums.{ComponentType, InstanceState}
import de.upb.cs.swt.delphi.crawler.instancemanagement.{Instance, InstanceRegistry}
import de.upb.cs.swt.delphi.crawler.instancemanagement.{Instance, InstanceLink, InstanceRegistry}

import scala.concurrent.duration._
import scala.util.{Failure, Success, Try}
Expand All @@ -43,7 +43,9 @@ class Configuration {
ComponentType.ElasticSearch,
None,
InstanceState.Running,
List.empty[String])
List.empty[String],
List.empty[InstanceLink],
List.empty[InstanceLink])
}

val mavenRepoBase: URI = new URI("http://repo1.maven.org/maven2/") // TODO: Create a local demo server "http://localhost:8881/maven2/"
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -16,61 +16,111 @@
package de.upb.cs.swt.delphi.crawler.instancemanagement

import akka.http.scaladsl.marshallers.sprayjson.SprayJsonSupport
import de.upb.cs.swt.delphi.crawler.instancemanagement.InstanceEnums.{ComponentType, InstanceState}
import spray.json.{DefaultJsonProtocol, DeserializationException, JsString, JsValue, JsonFormat}

trait JsonSupport extends SprayJsonSupport with DefaultJsonProtocol {
/**
* Trait defining the implicit JSON formats needed to work with Instances
*/
trait InstanceJsonSupport extends SprayJsonSupport with DefaultJsonProtocol with InstanceLinkJsonSupport {

implicit val componentTypeFormat : JsonFormat[InstanceEnums.ComponentType] = new JsonFormat[InstanceEnums.ComponentType] {
//Custom JSON format for an ComponentType
implicit val componentTypeFormat : JsonFormat[ComponentType] = new JsonFormat[ComponentType] {

def write(compType : InstanceEnums.ComponentType) = JsString(compType.toString)
/**
* Custom write method for serializing an ComponentType
* @param compType The ComponentType to serialize
* @return JsString containing the serialized value
*/
def write(compType : ComponentType) = JsString(compType.toString)

def read(value: JsValue) : InstanceEnums.ComponentType = value match {
/**
* Custom read method for deserialization of an ComponentType
* @param value JsValue to deserialize (must be a JsString)
* @return ComponentType that has been read
* @throws DeserializationException Exception thrown when JsValue is in incorrect format
*/
def read(value: JsValue) : ComponentType = value match {
case JsString(s) => s match {
case "Crawler" => InstanceEnums.ComponentType.Crawler
case "WebApi" => InstanceEnums.ComponentType.WebApi
case "WebApp" => InstanceEnums.ComponentType.WebApp
case "DelphiManagement" => InstanceEnums.ComponentType.DelphiManagement
case "ElasticSearch" => InstanceEnums.ComponentType.ElasticSearch
case "Crawler" => ComponentType.Crawler
case "WebApi" => ComponentType.WebApi
case "WebApp" => ComponentType.WebApp
case "DelphiManagement" => ComponentType.DelphiManagement
case "ElasticSearch" => ComponentType.ElasticSearch
case x => throw DeserializationException(s"Unexpected string value $x for component type.")
}
case y => throw DeserializationException(s"Unexpected type $y while deserializing component type.")
case y => throw DeserializationException(s"Unexpected type $y during deserialization component type.")
}
}

implicit val stateFormat : JsonFormat[InstanceEnums.State] = new JsonFormat[InstanceEnums.State] {
//Custom JSON format for an InstanceState
implicit val stateFormat : JsonFormat[InstanceState] = new JsonFormat[InstanceState] {

def write(compType : InstanceEnums.State) = JsString(compType.toString)
/**
* Custom write method for serializing an InstanceState
* @param state The InstanceState to serialize
* @return JsString containing the serialized value
*/
def write(state : InstanceState) = JsString(state.toString)

def read(value: JsValue) : InstanceEnums.State = value match {
/**
* Custom read method for deserialization of an InstanceState
* @param value JsValue to deserialize (must be a JsString)
* @return InstanceState that has been read
* @throws DeserializationException Exception thrown when JsValue is in incorrect format
*/
def read(value: JsValue) : InstanceState = value match {
case JsString(s) => s match {
case "Running" => InstanceEnums.InstanceState.Running
case "Stopped" => InstanceEnums.InstanceState.Stopped
case "Failed" => InstanceEnums.InstanceState.Failed
case "Paused" => InstanceEnums.InstanceState.Paused
case "NotReachable" => InstanceEnums.InstanceState.NotReachable
case "Running" => InstanceState.Running
case "Stopped" => InstanceState.Stopped
case "Failed" => InstanceState.Failed
case "Paused" => InstanceState.Paused
case "NotReachable" => InstanceState.NotReachable
case "Deploying" => InstanceState.Deploying
case x => throw DeserializationException(s"Unexpected string value $x for instance state.")
}
case y => throw DeserializationException(s"Unexpected type $y while deserializing instance state.")
case y => throw DeserializationException(s"Unexpected type $y during deserialization instance state.")
}
}

implicit val instanceFormat : JsonFormat[Instance] = jsonFormat8(Instance)
//JSON format for Instances
implicit val instanceFormat : JsonFormat[Instance] = jsonFormat10(Instance)
}

/**
* The instance type used for transmitting data about an instance from an to the registry
* @param id Id of the instance. This is an Option[Long], as an registering instance will not yet have an id.
* @param host Host of the instance.
* @param portNumber Port the instance is reachable at.
* @param name Name of the instance
* @param componentType ComponentType of the instance.
* @param dockerId The docker container id of the instance. This is an Option[String], as not all instance have to be docker containers.
* @param instanceState State of the instance
*/
final case class Instance (
id: Option[Long],
host: String,
portNumber: Long,
name: String,
componentType: InstanceEnums.ComponentType,
componentType: ComponentType,
dockerId: Option[String],
instanceState: InstanceEnums.State,
labels: List[String]
) {}
instanceState: InstanceState,
labels: List[String],
linksTo: List[InstanceLink],
linksFrom: List[InstanceLink]
)

/**
* Enumerations concerning instances
*/
object InstanceEnums {

//Type to use when working with component types
type ComponentType = ComponentType.Value

/**
* ComponentType enumeration defining the valid types of delphi components
*/
object ComponentType extends Enumeration {
val Crawler : Value = Value("Crawler")
val WebApi : Value = Value("WebApi")
Expand All @@ -79,8 +129,14 @@ object InstanceEnums {
val ElasticSearch : Value = Value("ElasticSearch")
}

type State = InstanceState.Value
//Type to use when working with instance states
type InstanceState = InstanceState.Value

/**
* InstanceState enumeration defining the valid states for instances of delphi components
*/
object InstanceState extends Enumeration {
val Deploying : Value = Value("Deploying")
val Running : Value = Value("Running")
val Stopped : Value = Value("Stopped")
val Failed : Value = Value("Failed")
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
// Copyright (C) 2018 The Delphi Team.
// See the LICENCE file distributed with this work for additional
// information regarding copyright ownership.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package de.upb.cs.swt.delphi.crawler.instancemanagement

import LinkEnums.LinkState
import akka.http.scaladsl.marshallers.sprayjson.SprayJsonSupport
import spray.json.{DefaultJsonProtocol, DeserializationException, JsString, JsValue, JsonFormat}

trait InstanceLinkJsonSupport extends SprayJsonSupport with DefaultJsonProtocol {

implicit val linkStateFormat: JsonFormat[LinkState] = new JsonFormat[LinkState] {
override def read(value: JsValue): LinkState = value match {
case JsString(s) => s match {
case "Assigned" => LinkState.Assigned
case "Outdated" => LinkState.Outdated
case "Failed" => LinkState.Failed
case x => throw DeserializationException(s"Unexpected string value $x for LinkState.")
}
case y => throw DeserializationException(s"Unexpected type $y during deserialization of LinkState")
}

override def write(linkState: LinkState): JsValue = JsString(linkState.toString)
}

implicit val instanceLinkFormat: JsonFormat[InstanceLink] =
jsonFormat3(InstanceLink)
}


final case class InstanceLink(idFrom: Long, idTo:Long, linkState: LinkState)

object LinkEnums {
type LinkState = LinkState.Value

object LinkState extends Enumeration {
val Assigned: Value = Value("Assigned")
val Failed: Value = Value("Failed")
val Outdated: Value = Value("Outdated")
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ import scala.concurrent.{Await, ExecutionContext, Future}
import scala.util.{Failure, Success, Try}
import spray.json._

object InstanceRegistry extends JsonSupport with AppLogging
object InstanceRegistry extends InstanceJsonSupport with AppLogging
{

implicit val system : ActorSystem = Crawler.system
Expand Down Expand Up @@ -251,7 +251,8 @@ object InstanceRegistry extends JsonSupport with AppLogging

private def createInstance(id: Option[Long], controlPort : Int, name : String) : Instance =
Instance(id, InetAddress.getLocalHost.getHostAddress,
controlPort, name, ComponentType.Crawler, None, InstanceState.Running, List.empty[String])
controlPort, name, ComponentType.Crawler, None, InstanceState.Running,
List.empty[String], List.empty[InstanceLink], List.empty[InstanceLink])


object ReportOperationType extends Enumeration {
Expand Down