GP-678, GP-811: Refactored fetchDependencies.gradle, now downloads FI…

…D datasets from ghidra-data repo, and revisied depencies/flatRepo layout.

.gitignore

@@ -3,7 +3,7 @@ ghidra.repos.config
 
 # Misc files produced while executing application
 repositories/
-flatRepo/
+dependencies/
 Ghidra/.ghidraSvrKeys
 wrapper.log*
 

DevGuide.md

@@ -33,7 +33,7 @@ You may not need all of these, depending on which portions you are building or d
       - https://adoptopenjdk.net/releases.html?variant=openjdk11&jvmVariant=hotspot
     - Amazon Corretto
       - https://docs.aws.amazon.com/corretto/latest/corretto-11-ug/downloads-list.html
-* Gradle 5.0 or later - We use version 5.0, and tested with up to 5.6.3.
+* Gradle 5.0 or later - We use version 5.0, and tested with up to 6.8.3.
     - https://gradle.org/next-steps/?version=5.0&format=bin
 * A C/C++ compiler - We use GCC on Linux, Xcode (Clang) on macOS, and Visual Studio (2017 or later) on Windows.
     - https://gcc.gnu.org/
@@ -61,9 +61,9 @@ You may not need all of these, depending on which portions you are building or d
     - https://sourceforge.net/projects/yajsw/files/yajsw/yajsw-stable-12.12/
 * Eclipse PDE - Environment for developing the GhidraDev plugin.
     - https://www.eclipse.org/pde/
-* Eclipse CDT. We use version 8.6.0 - Build dependency for the GhidraDev plugin.
+* Eclipse CDT. We build against version 8.6.0 - Build dependency for the GhidraDev plugin.
     - https://www.eclipse.org/cdt/
-* PyDev. We use version 6.3.1 - Build dependency for the GhidraDev plugin.
+* PyDev. We build against version 6.3.1 - Build dependency for the GhidraDev plugin.
     - https://sourceforge.net/projects/pydev/files/pydev/
 
 There are many, many others automatically downloaded by Gradle from Maven Central and Bintray JCenter when building and/or setting up the development environment.
@@ -110,64 +110,64 @@ or manually by downloading the required dependencies.  Choose one of the two fol
 The flat directory-style repository can be setup automatically by running a simple Gradle script. 
 Navigate to `~/git/ghidra` and run the following:
 ```
-gradle --init-script gradle/support/fetchDependencies.gradle init
+gradle -I gradle/support/fetchDependencies.gradle init
 ```
 The Gradle task to be executed, in this case _init_, is unimportant. The point is to have Gradle execute
-the `fetchDependencies.gradle` script. If it ran correctly you will have a new `~/git/ghidra/flatRepo/` 
-directory populated with the following jar files:
- * AXMLPrinter2
- * csframework
- * dex-ir-2.0
- * dex-reader-2.0
- * dex-reader-api-2.0
- * dex-tools-2.0
- * dex-translator-2.0
- * dex-writer-2.0
- * hfsx
- * hfsx_dmglib
- * iharder-base64
-
-There will also be a new archive files at:
- * ~/git/ghidra/Ghidra/Features/GhidraServer/build/`yajsw-stable-12.12.zip`
- * ~/git/ghidra/GhidraBuild/EclipsePlugins/GhidraDev/GhidraDevPlugin/build/`PyDev 6.3.1.zip`
- * ~/git/ghidra/GhidraBuild/EclipsePlugins/GhidraDev/GhidraDevPlugin/build/`cdt-8.6.0.zip`
+the `fetchDependencies.gradle` script. If it ran correctly you will have a new `~/git/ghidra/dependencies/` 
+directory populated with the following files:
+ * flatRepo/AXMLPrinter2.jar
+ * flatRepo/csframework.jar
+ * flatRepo/dex-ir-2.0.jar
+ * flatRepo/dex-reader-2.0.jar
+ * flatRepo/dex-reader-api-2.0.jar
+ * flatRepo/dex-tools-2.0.jar
+ * flatRepo/dex-translator-2.0.jar
+ * flatRepo/dex-writer-2.0.jar
+ * flatRepo/hfsx.jar
+ * flatRepo/hfsx_dmglib.jar
+ * flatRepo/iharder-base64.jar
+ * cdt-8.6.0.zip
+ * PyDev 6.3.1.zip
+ * yajsw-stable-12.12.zip
+ * fid/*.fidb
 
 If you see these, congrats! Skip to [building](#building-ghidra) or [developing](#developing-ghidra). If not, continue with manual download 
 instructions below...
 
 ### Manual Download Instructions
 
-Create the `~/git/ghidra/flatRepo/` directory to hold the manually-downloaded dependencies:
+Create the `~/git/ghidra/dependencies/` and `~/git/ghidra/dependencies/flatRepo` directories to hold the manually-downloaded dependencies:
 
 ```bash
-mkdir ~/git/ghidra/flatRepo
+mkdir ~/git/ghidra/dependencies
+mkdir ~/git/ghidra/dependencies/flatRepo
 ```
 
 #### Get Dependencies for FileFormats:
 
 Download `dex-tools-2.0.zip` from the dex2jar project's releases page on GitHub.
-Unpack the `dex-*.jar` files from the `lib` directory to `~/git/ghidra/flatRepo`:
+Unpack the `dex-*.jar` files from the `lib` directory to `~/git/ghidra/dependencies/flatRepo`:
 
 ```bash
 cd ~/Downloads   # Or wherever
 curl -OL https://github.com/pxb1988/dex2jar/releases/download/2.0/dex-tools-2.0.zip
 unzip dex-tools-2.0.zip
-cp dex2jar-2.0/lib/dex-*.jar ~/git/ghidra/flatRepo/
+cp dex2jar-2.0/lib/dex-*.jar ~/git/ghidra/dependencies/flatRepo/
 
 ```
 
 Download `AXMLPrinter2.jar` from the "android4me" archive on code.google.com.
-Place it in `~/git/ghidra/flatRepo`:
+Place it in `~/git/ghidra/dependencies/flatRepo`:
 
 ```bash
-cd ~/git/ghidra/flatRepo
+cd ~/git/ghidra/dependencies/flatRepo
 curl -OL https://storage.googleapis.com/google-code-archive-downloads/v2/code.google.com/android4me/AXMLPrinter2.jar
 ```
 
 #### Get Dependencies for DMG:
 
 Download `hfsexplorer-0_21-bin.zip` from www.catacombae.org.
-Unpack the `lib` directory to `~/git/ghidra/flatRepo`:
+Unpack the `lib` directory to `~/git/ghidra/dependencies/flatRepo`:
 
 ```bash
 cd ~/Downloads   # Or wherever
@@ -176,43 +176,41 @@ mkdir hfsx
 cd hfsx
 unzip ../hfsexplorer-0_21-bin.zip
 cd lib
-cp csframework.jar hfsx_dmglib.jar hfsx.jar iharder-base64.jar ~/git/ghidra/flatRepo/
+cp csframework.jar hfsx_dmglib.jar hfsx.jar iharder-base64.jar ~/git/ghidra/dependencies/flatRepo/
 ```
 
 #### Get Dependencies for GhidraServer
 
 Building the GhidraServer requires "Yet another Java service wrapper" (yajsw) version 12.12.
 Download `yajsw-stable-12.12.zip` from their project on www.sourceforge.net, and place it in:
-`~/git/ghidra/Ghidra/Features/GhidraServer/build`:
+`~/git/ghidra/dependencies/`:
 
 ```bash
 cd ~/Downloads   # Or wherever
 curl -OL https://sourceforge.net/projects/yajsw/files/yajsw/yajsw-stable-12.12/yajsw-stable-12.12.zip
-mkdir -p ~/git/ghidra/Ghidra/Features/GhidraServer/build/
-cp ~/Downloads/yajsw-stable-12.12.zip ~/git/ghidra/Ghidra/Features/GhidraServer/build/
+cp ~/Downloads/yajsw-stable-12.12.zip ~/git/ghidra/dependencies/
 ```
 
 #### Get Dependencies for GhidraDev
 
 Building the GhidraDev plugin for Eclipse requires the CDT and PyDev plugins for Eclipse.
 Download `cdt-8.6.0.zip` from The Eclipse Foundation, and place it in:
-`~/git/ghidra/GhidraBuild/EclipsePlugins/GhidraDev/GhidraDevPlugin/build/`:
+`~/git/ghidra/dependencies/`:
 
 ```bash
 cd ~/Downloads   # Or wherever
 curl -OL 'https://archive.eclipse.org/tools/cdt/releases/8.6/cdt-8.6.0.zip'
 curl -o 'cdt-8.6.0.zip.sha512' -L --retry 3 'https://www.eclipse.org/downloads/sums.php?type=sha512&file=/tools/cdt/releases/8.6/cdt-8.6.0.zip'
 shasum -a 512 -c 'cdt-8.6.0.zip.sha512'
-mkdir -p ~/git/ghidra/GhidraBuild/EclipsePlugins/GhidraDev/GhidraDevPlugin/build/
-cp ~/Downloads/cdt-8.6.0.zip ~/git/ghidra/GhidraBuild/EclipsePlugins/GhidraDev/GhidraDevPlugin/build/
+cp ~/Downloads/cdt-8.6.0.zip ~/git/ghidra/dependencies/
 ```
 
 Download `PyDev 6.3.1.zip` from www.pydev.org, and place it in the same directory:
 
 ```bash
 cd ~/Downloads   # Or wherever
 curl -L -o 'PyDev 6.3.1.zip' https://sourceforge.net/projects/pydev/files/pydev/PyDev%206.3.1/PyDev%206.3.1.zip
-cp ~/Downloads/'PyDev 6.3.1.zip' ~/git/ghidra/GhidraBuild/EclipsePlugins/GhidraDev/GhidraDevPlugin/build/
+cp ~/Downloads/'PyDev 6.3.1.zip' ~/git/ghidra/dependencies/
 ```
 
 ## Building Ghidra

Ghidra/Features/FunctionID/build.gradle

@@ -31,10 +31,11 @@ dependencies {
 
 }
 
-// All *.fidb files located in the BIN repo under src/main/fidb will be unpacked
-def fidbSrcDir = "${getProjectLocationInBinRepo(project)}/src/main/fidb"
-
-def fidDbFiles = fileTree(fidbSrcDir) {
+// All *.fidb files located in the dependencies/fid directory OR the 
+// BIN repo under src/main/fidb will be unpacked
+def depsDir = file("${DEPS_DIR}/fidb")
+def binRepoDir = "${getProjectLocationInBinRepo(project)}/src/main/fidb"
+def fidDbFiles = fileTree(depsDir.exists() ? depsDir : binRepoDir) {
 	include '**/*.fidb'
 }
 

Ghidra/Features/GhidraServer/build.gradle

@@ -40,12 +40,11 @@ addExports([
 ])
 
 CopySpec yajswCopySpec = copySpec {
-	File localFile = file("build/${yajswRelease}.zip")
-	File binFile = file("${BIN_REPO}/Ghidra/Features/GhidraServer/${yajswRelease}.zip")
+	File depsFile = file("${DEPS_DIR}/GhidraServer/${yajswRelease}.zip")
+	File binRepoFile = file("${BIN_REPO}/Ghidra/Features/GhidraServer/${yajswRelease}.zip")
 
-	// First check if the file was downloaded and dropped in locally.  If not, check in the bin
-	// repo.
-	def yajswZipTree = localFile.exists() ? zipTree(localFile) : zipTree(binFile)
+	// First check if the file is in the dependencies repo.  If not, check in the bin repo.
+	def yajswZipTree = depsFile.exists() ? zipTree(depsFile) : zipTree(binRepoFile)
 
 	from(yajswZipTree) {
 		include "${yajswRelease}/lib/core/**"

GhidraBuild/EclipsePlugins/GhidraDev/GhidraDevPlugin/build.gradle

@@ -80,12 +80,11 @@ task pyDevUnpack(type:Copy) {
 		!pyDevDestDir.exists()
 	}
 
-	File localFile = file("build/PyDev 6.3.1.zip")
-	File binFile = file("${BIN_REPO}/GhidraBuild/EclipsePlugins/GhidraDev/buildDependencies/PyDev 6.3.1.zip")
+	File depsFile = file("${DEPS_DIR}/GhidraDev/PyDev 6.3.1.zip")
+	File binRepoFile = file("${BIN_REPO}/GhidraBuild/EclipsePlugins/GhidraDev/buildDependencies/PyDev 6.3.1.zip")
 
-	// First check if the file was downloaded and dropped in locally.  If not, check in the bin
-	// repo.
-	def pyDevZipTree = localFile.exists() ? zipTree(localFile) : zipTree(binFile)
+	// First check if the file is in the dependencies repo.  If not, check in the bin repo.
+	def pyDevZipTree = depsFile.exists() ? zipTree(depsFile) : zipTree(binRepoFile)
 
 	from pyDevZipTree
 	exclude "**/.project", "**/.pydevproject"
@@ -104,12 +103,11 @@ task cdtUnpack(type:Copy) {
 		!cdtDestDir.exists()
 	}
 
-	File localFile = file("build/cdt-8.6.0.zip")
-	File binFile = file("${BIN_REPO}/GhidraBuild/EclipsePlugins/GhidraDev/buildDependencies/cdt-8.6.0.zip")
+	File depsFile = file("${DEPS_DIR}/GhidraDev/cdt-8.6.0.zip")
+	File binRepoFile = file("${BIN_REPO}/GhidraBuild/EclipsePlugins/GhidraDev/buildDependencies/cdt-8.6.0.zip")
 
-	// First check if the file was downloaded and dropped in locally.  If not, check in the bin
-	// repo.
-	def cdtZipTree = localFile.exists() ? zipTree(localFile) : zipTree(binFile)
+	// First check if the file is in the dependencies repo.  If not, check in the bin repo.
+	def cdtZipTree = depsFile.exists() ? zipTree(depsFile) : zipTree(binRepoFile)
 
 	from cdtZipTree
 

build.gradle

@@ -51,9 +51,9 @@ if ("32".equals(System.getProperty("sun.arch.data.model"))) {
  *  Define the location of bin repo
  *********************************************************************************/
 project.ext.GHIDRA_GROUP = "Z Ghidra"
-project.ext.BIN_REPO = file("${projectDir}/../ghidra.bin").absolutePath
 project.ext.ROOT_PROJECT_DIR = projectDir.absolutePath
-project.ext.BIN_REPO_PATH = BIN_REPO // TODO make path names consistent
+project.ext.BIN_REPO = file("${projectDir}/../ghidra.bin").absolutePath
+project.ext.DEPS_DIR = file("${projectDir}/dependencies")
 
 /*********************************************************************************
  *  Prevent forked Java processes from stealing focus
@@ -67,13 +67,14 @@ allprojects {
 /*********************************************************************************
  *  Use flat directory-style repository if flatRepo directory is present.
  *********************************************************************************/
-if (file("flatRepo").isDirectory()) {
+def flatRepo = file("${DEPS_DIR}/flatRepo") 
+if (flatRepo.isDirectory()) {
 	allprojects {
 		repositories {
 			mavenLocal()
 			mavenCentral()
 			jcenter()
-			flatDir name: "flat", dirs:["$rootProject.projectDir/flatRepo"]
+			flatDir name: "flat", dirs:["$flatRepo"]
 		}
 	}
 }

gradle/root/svg.gradle

@@ -43,7 +43,6 @@ task rasterizeSvg(type: JavaExec) {
 		// added these in the individual projects which use this task (eg: to the 'compile' 
 		// configuration) but since this is the only task which requires them, it seemed 
 		// appropriate to just add them here.
-		def BIN_REPO = rootProject.file(BIN_REPO_PATH).toString()
 		classpath = files (	BIN_REPO + "/ExternalLibraries/libsforBuild/batik-all-1.7.jar",
 							BIN_REPO + "/ExternalLibraries/libsforBuild/xml-apis-ext.jar")