Skip to content

Pre-Release Merge #19

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 93 commits into from
Aug 11, 2018
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
93 commits
Select commit Hold shift + click to select a range
a3d3e28
Updated project structure
bhermann May 10, 2018
5458035
Added repository to Maven identifier
bhermann May 18, 2018
37e5663
Refactored Maven actor
bhermann May 18, 2018
3b48cc2
Added comments
bhermann May 18, 2018
db2b22b
Added logging for non-actor classes
bhermann May 18, 2018
c941bad
Added configuration object
bhermann May 18, 2018
e8c59ac
Added new Maven-index-based stream source
bhermann May 18, 2018
5110d76
Added Maven download infrastructure
bhermann May 18, 2018
997b889
Added new starter infrastructure
bhermann May 18, 2018
5b946cd
Renamed preflight check for elasticsearch and added a close
bhermann May 19, 2018
175f0e2
Refactored to pass the ActorSystem around
bhermann May 19, 2018
709fd50
Added preflight check for index integrity
bhermann May 19, 2018
3dd5817
Closed connection for index check
bhermann May 19, 2018
9cc0e6e
Slimmed down the code a bit
bhermann May 19, 2018
8e763be
Added missing execution context and drafted index creation
bhermann May 19, 2018
9f7ac7b
First draft to index maven artifacts.
bhermann May 20, 2018
69972ee
Changed Elasticsearch mapping
almacken May 23, 2018
70ae83e
Made the elasticsearch URI configurable
bhermann May 24, 2018
fa7c372
Changed feature mapping storage format to be easier to access with ot…
almacken May 24, 2018
66cba45
Improved formatting of identifier field definitions
almacken May 24, 2018
2c7d138
Added comment describing changes to Hermes field names
almacken May 25, 2018
34df423
ElasticActor should now handle GitIdentifiers
almacken May 25, 2018
15f5867
updated sbt version to 1.1.1 to match the version of the main repository
janniclas May 28, 2018
e2a74cc
Merge pull request #10 from delphi-hub/global-sbt
bhermann May 28, 2018
687a955
Merge branch 'develop' into develop
almacken May 28, 2018
e0141d2
Merge pull request #11 from almacken/develop
bhermann Jun 3, 2018
5222aee
Fixed logging
bhermann Jun 5, 2018
8797e8e
Maven Downloader
Jun 5, 2018
331188b
Merge branch 'develop' of https://github.com/delphi-hub/delphi-crawle…
Jun 5, 2018
cad5514
Removed unused imports
Jun 5, 2018
8788ade
Added test with actor for maven downloader
Jun 5, 2018
7018a5b
Improved test cases for maven downloader and dispatch actor of prepro…
Jun 11, 2018
d7f503d
Deleted Test actor and added scala doc
Jun 11, 2018
2ef2edb
Updated mapping to support call graphs
almacken Jun 13, 2018
3867a79
Merge branch 'develop' of https://github.com/almacken/delphi-crawler …
almacken Jun 13, 2018
ac475d8
Merge branch 'develop' into feature/callGraphMapping
bhermann Jun 14, 2018
b10963d
Merge pull request #13 from almacken/feature/callGraphMapping
bhermann Jun 14, 2018
b31acfc
Merge branch 'develop' into downloader
bhermann Jun 14, 2018
5adefac
Merge pull request #12 from delphi-hub/downloader
bhermann Jun 14, 2018
0be7f79
Updated OPAL/Hermes
bhermann Jun 17, 2018
7503b51
Removed unused inputs
bhermann Jun 17, 2018
c512c89
Made index processing a bit more error resillient
bhermann Jun 17, 2018
4bae15f
Cleaned up logging
bhermann Jun 17, 2018
239b707
Cleaned up crawler
bhermann Jun 17, 2018
d3b8887
Removed the code for the legacy crawl actor
bhermann Jun 17, 2018
838d1bd
Drafted processing dispatch
bhermann Jun 17, 2018
004530b
Some refactoring
bhermann Jun 17, 2018
39ab222
Removed old implementation
bhermann Jun 17, 2018
93295e9
Reformulated maven downloader test
bhermann Jun 17, 2018
facdd9d
Added missing implicit
bhermann Jun 17, 2018
f34e852
Working on the pipelining
bhermann Jun 17, 2018
6940bfb
Moved hard coded crawling URL into the configuration
bhermann Jun 17, 2018
d05dd77
Wired pipeline
bhermann Jun 17, 2018
6379cf0
More wiring
bhermann Jun 17, 2018
23332ad
Configured chain
bhermann Jun 20, 2018
2d5de69
Reverting to old OPAL version, newer seems to be broken
bhermann Jun 20, 2018
c7c6d50
Adding scalastyle to delphi crawler
Jul 3, 2018
8c8ea42
Added classes to read classes from Jars inside HttpStreams
bhermann Jul 4, 2018
c970f36
Added author mark
bhermann Jul 5, 2018
d3a6b85
Added OPAL Project creation from InputStreams
bhermann Jul 5, 2018
fcb38e1
Removed log file writing from configuration for now
bhermann Jul 17, 2018
fa6c5d2
Fixing rapid uniqueness check
bhermann Jul 17, 2018
951d2ab
Unifying index naming
bhermann Jul 17, 2018
856065d
Fixing uniqueness check with pre-existing artifacts
bhermann Jul 17, 2018
872000d
Reconfigured class file reading
bhermann Jul 18, 2018
21c5900
Pinning security critical dependencies
bhermann Jul 25, 2018
23c8322
Attached OPAL to general logging infrastructure
bhermann Jul 25, 2018
307b3c8
Implemented a faster and tamer version of OPAL project creation for H…
bhermann Jul 25, 2018
e085acb
Added dependency plugin for snyk
bhermann Jul 25, 2018
c4a9870
Finished license application
bhermann Jul 26, 2018
c1f4dd3
Added code coverage support and support for snyk
bhermann Jul 26, 2018
1637b7e
Removed insecure (=predictable) use of RNG (here not that critical tbh)
bhermann Jul 26, 2018
9eb3dfe
Made project reification faster by removing the full reification of t…
bhermann Jul 26, 2018
6f67868
Reformatted and added a comment on library pinning
bhermann Jul 26, 2018
4de72b5
Updated dependencies b/c of vulnerable 3rd party components (clenup)
bhermann Jul 26, 2018
4ab9a39
Moved import
bhermann Jul 26, 2018
8dc0e17
Reorganized import
bhermann Jul 26, 2018
481530d
Removed ununsed brackets
bhermann Jul 26, 2018
bdbe438
Restored to ASCII characters
bhermann Jul 26, 2018
e130e4a
Converted to named parameters
bhermann Jul 26, 2018
d217727
Activated scalastyle plugin
bhermann Jul 26, 2018
7af2151
Moved scala style config to project folder
Jul 31, 2018
28ab5f0
Stabilized InputStream handling. There seem to be many flavors of ZIP…
bhermann Aug 7, 2018
0eada11
Added old version for correct zip files back to the implementation as…
bhermann Aug 7, 2018
19a2d3d
Simplified code
bhermann Aug 7, 2018
4976bd5
Added a buffer, now it performs uniformly
bhermann Aug 7, 2018
f83e2f8
Attached Hermes processing to index procedure
bhermann Aug 8, 2018
7a747c7
Removing scalastyle from test target
Aug 9, 2018
bc8d01f
Merge branch 'develop' into Feature/4/scalastyle
bhermann Aug 11, 2018
25952cd
Rename scalastyle-config.xml to scalastyle_config.xml
bhermann Aug 11, 2018
946aa1c
Update build.sbt
bhermann Aug 11, 2018
f55d837
Rename scalastyle_config.xml to scalastyle-config.xml
bhermann Aug 11, 2018
d29fae1
Merge pull request #18 from delphi-hub/Feature/4/scalastyle
bhermann Aug 11, 2018
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Stabilized InputStream handling. There seem to be many flavors of ZIP…
… files.
  • Loading branch information
bhermann committed Aug 7, 2018
commit 28ab5f021ef43121b87c972afa379132a5c1133e
Original file line number Diff line number Diff line change
Expand Up @@ -35,17 +35,15 @@ trait ClassStreamReader {

while (je != null) {
val entryName = je.getName
if (je.getSize.toInt > 0 && entryName.endsWith(".class")) {
val entryBytes = new Array[Byte](je.getSize.toInt)
if (entryName.endsWith(".class")) {
val baos = new ByteArrayOutputStream()

var remaining = entryBytes.length
var offset = 0
while (remaining > 0) {
val readBytes = in.read(entryBytes, offset, remaining)
if (readBytes < 0) throw new EOFException()
remaining -= readBytes
offset += readBytes
Stream.continually(in.read()).takeWhile(_ != -1).foreach { x =>
baos.write(x)
}

val entryBytes = baos.toByteArray

futures ::= Future[List[(ClassFile, String)]] {
val cfs = reader.ClassFile(new DataInputStream(new ByteArrayInputStream(entryBytes)))
cfs map { cf => (cf, entryName) }
Expand All @@ -54,7 +52,9 @@ trait ClassStreamReader {
je = in.getNextJarEntry()
}

futures.flatMap(f => Await.result(f, Duration.Inf))
val result = futures.flatMap(f => Await.result(f, Duration.Inf))

result
}

/**
Expand All @@ -67,11 +67,12 @@ trait ClassStreamReader {
val config : Config = org.opalj.br.BaseConfig

val projectClasses: Traversable[(ClassFile, URL)] = readClassFiles(jarInputStream).map { case (classFile, _) => (classFile, source) }
val libraryClasses: Traversable[(ClassFile, URL)] =
val libraryClasses: Traversable[(ClassFile, URL)] = Nil
/*
readClassFiles(new JarInputStream
(new FileInputStream(org.opalj.bytecode.RTJar)), Project.JavaLibraryClassFileReader)
.map{ case (classFile, _) => (classFile, org.opalj.bytecode.RTJar.toURI.toURL) }

*/

Project(projectClasses, libraryClasses, true, Traversable.empty)(config, OPALLogAdapter)
}
Expand Down
Binary file added src/test/resources/xsddoc-0.8.jar
Binary file not shown.
Original file line number Diff line number Diff line change
Expand Up @@ -71,4 +71,18 @@ class ClassStreamReaderTest extends FlatSpec with Matchers with BeforeAndAfter {
p.projectClassFilesCount should be (23)
p.allMethodsWithBody.size should not be (0)
}

it should "be able to construct classes out of a somewhat broken JAR file" in {
val target = File("src/test/resources/xsddoc-0.8.jar")
val fis = target.inputStream()


val jis = new JarInputStream(fis)

val p = new ClassStreamReader {}.createProject(target.toURL, jis)

p shouldBe a [Project[_]]
p.projectClassFilesCount should be (21)
p.allMethodsWithBody.size should not be (0)
}
}