| layout | title | excerpt | tags | Categories | filename | image | comments | |||
|---|---|---|---|---|---|---|---|---|---|---|
post |
Install, configure, and test all on a MacOS laptop |
Everything you need to be a professional developer |
|
Devops |
README.md |
true |
{{ page.excerpt }}
The tagline of this repo is "Automatically install and configure the dozens of programs that a developer needs to work offline as a developer of several stacks on a Mac."
This script was created to help you cope with the large number of apps needed to be a "full stack" developer today. This installs several stacks with sample repositories:
- MEAN (MongoDB, Express, Angular, NodeJs) with the MeanJs sample app
- JAM (Jekyll, APIs, Markup) with a sample Github.io website
- MAMP (Macintosh, Apache/Nginx, MySQL, PHP) for WordPress websites
- Elastic (ELK) stack (Elasticsearch, Logstash, Kibana, etc.)
- Serverless on Amazon Lambda, Azure Functions, Google Actions, Iron.io
- etc.
Most tutorials ask you to manually type or copy and paste strings from web pages (often with missing steps), which can take time, and be error-prone. And don't get me started on webinars with demos that brags rather than teach.
This script cuts through all that by scripts running on your Mac and displaying on your screen.
You will benefit most from this if you configure a new laptop for yourself or for other developers joining your organization. You'll skip wasted days installing everything one at a time (and doing it differently than colleagues). This repo brings DevSecOps-style "immutable architecture" to MacOS laptops. Immutability means replacing the whole machine instance instead of upgrading or repairing faulty components.
But this script helps with updates too. You can, but don't have to, start from scratch. Although you may use Apple's Time Machine app to backup everything to a USB drive or AirPort Time Capsule, you may want a way to keep up with the latest changes in apps updated to the latest version, by running a single "upgrade" command.
This bash script enables you to work offline by installing several servers.
You manage allocation of port numbers in one place:
ELASTIC_PORT="9200" # DATA_TOOLS from default 9200 GRAFANA_PORT="8089" # VIZ_TOOLS from default 8080 JEKYLL_PORT="4000" # LOCAOHOSTS from default 4000 JENKINS_PORT="8088" # LOCALHOSTS from default 8080 KIBANA_PORT="5601" # DATA_TOOLS default 5601 MYSQL_PORT="3060" # DATA_TOOLS default 3060 MEANJS_PORT="3000" # NODE_TOOLS from default 3000 MINIKUBE_PORT="8083" # LOCAOHOSTS from default 8080 NEO4J_PORT="7474" # DATA_TOOL default 7474 HTTPS: 7473 NGINX_PORT="8086" # LOCALHOSTS from default 8080 PACT_PORT="6666" # TEST_TOOLS from default 6666 POSTGRESQL_PORT="5432" # DATA_TOOLS default 5432 PROMETHEUS_PORT="9090" # MON_TOOLS default 9090 REDIS_PORT="6379" # DATA_TOOLS default 6379 SONAR_PORT="9000" # DATA_TOOLS default 9000 TOMCAT_PORT="8087" # LOCALHOSTS from default 8080
The above are fron the secrets.sh file in your $HOME folder, which you edit to specify which port numbers and the keywords for apps you want installed.
The file's name is suffixed with ".sh" because another script runs it to establish variables for the script to reference.
Each program belongs to a category.
-
MAC_TOOLS mas, Ansible, 1Password, PowerShell, Kindle, etc.
-
DATA_TOOLS mongodb, postgresql, mysql, mariadb, graphql?
-
EDITORS Atom, Code, Eclipse, Emacs, IntelliJ, Macvim, STS, Sublime, Textmate, vim
-
BROWSERS chrome, firefox, brave, phantomjs
-
GIT_CLIENTS git, cola, github, gitkraken, smartgit, sourcetree, tower, magit, gitup
-
GIT_TOOLS hooks, tig, lfs, diff-so-fancy, grip, p4merge, git-flow, signing, hub
-
JAVA_TOOLS Maven, Ant, Gradle, TestNG, Cucumber, Junit4, Junit5, Yarn, dbunit, Mockito, JMeter, GCViewer, JProfiler, etc.
-
PYTHON_TOOLS Virtualenv, jupyter, anaconda, ipython, numpy, scipy, matplotlib, pytest, robotframework, etc.
-
NODE_TOOLS Bower, gulp, gulp-cli, npm-check, jscs, less, jshint, eslint, webpack, etc.
-
LOCALHOSTS Apache (httpd, apachectl)
-
TEST_TOOLS selenium, sikulix, golum, dbunit?
-
CLOUDS icloud, aws, gcp, azure, cf, heroku, docker, vagrant, terraform, serverless
-
MON_TOOLS (for monitoring) WireShark, Prometheus, others
-
VIZ_TOOLS (for visualization) Grafana, others (Prometheus, Kibana, Graphite)
-
COLAB_TOOLS (for collaboration) google-hangouts, hipchat, joinme, keybase, microsoft-lync, skype, slack, teamviewer, whatsapp, sococo, zoom
-
MEDIA_TOOLS Camtasia, Kindle (others: Snagit, etc.)
Links for individual apps above take you to technical descriptions about that technology.
The categories are run in dependency sequence. MAC_TOOLS to provide underlying utilities. DATA_TOOLS to provide databases.
Yes, you can just run brew yourself, one at a time. But logic in the script goes beyond what Homebrew does, and configures the component just installed:
- Install dependent components where necessary
- Display the version number installed (to a log)
- Add alias and paths in .bash_profile (if needed)
- Perform configuration (such as adding a missing file needed for mariadb to start)
- Edit configuration settings (such as changing default port within Nginx within config.conf file)
- Upgrade and uninstall if that is available
- Run a demo using the component to ensure that what has been installed actually works.
We also have Docker instances:
PROMETHEUS_PORT="9090" # from default 9090
The section below explains to someone relatively new to Mac machines the steps to automate installation of additional MacOS application programs. Along the way, we explore basic skills to use a command-line Terminal and common commands.
-
Obtain the Mac's Launch bar by positioning your mouse at the bottom edge of the screen until it appears.
-
If you don't see an icon for the Terminal program, click the magnifying glass icon always at the upper-right corner and type in Term until "Terminal app" is highlighted, then press Enter to accept it.
-
Click menu Shell then click New Window for a Terminal session.
PROTIP: More experienced people hover the mouse over New Window and click on one of the options.
The Terminal program is called a "Bash" shell, which is a contraction of the term "Bourne-agan shell", which is a play on words.
-
Test if you have Bash v4 installed by typing this:
bash --version | grep 'bash'
PROTIP: The attribute to obtain the version can vary among different commands. "--version" or "-v" or "version" may be used.
Hold the Shift key to press the | (called pipe) key at the upper-right of the keyboard.
The grep 'bash' is needed to filter out lines that do not contain the word "bash" in the response such as:
GNU bash, version 4.4.19(1)-release (x86_64-apple-darwin17.3.0) Copyright (C) 2016 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html This is free software; you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law.
If you have bash v3 that comes with MacOS, the next few steps will update it to version 4.
This blog describes what is improved by version 4. Bash 4.0 was released in 2009, but Apple still ships version 3.x, which first released in 2007.-
Switch to back to this web page by holding down the command key and pressing Tab repeatedly until it rests on the browser icon.
-
Triple-click on the script line below to highlight it for copying:
sh -c "$(curl -fsSL https://raw.githubusercontent.com/wilsonmar/mac-setup/master/mac-bash-up.sh)"
-
Press Command+C to copy it to your invisible Clipboard.
-
Switch to the Terminal by holding down command and pressing Tab repeatedly until it rests on the Termial icon.
-
At the Terminal, click on a Terminal window and paste in the command by holding down command then V. It doesn't matter what folder you're on at this point.
-
Press Enter to run the command, which upgrades Bash to version 4 and copies a file to your Home folder.
The script first makes use of the Ruby program to install Homebrew which, in turn, installs Bash v4 using the brew command to download and configure packages.
PROTIP: There is a (Ruby) web page for each brew install formula, such as:
https://github.com/Homebrew/homebrew-core/blob/master/Formula/httpd.rb -
After it runs, verify the version again as described above to ensure it's version 4.
The script also copies the secrets.sh file from the public on-line repository into your laptop so that you can add your secrets in the file but have no chance the file will be uploaded from the Git repository where it came from.
The file is placed in your account Home folder.
-
The default location the Teminal command opens to by default is your "Home" folder, which you can reach anytime by:
cd
-
The "~" (tilde character) prompt represents the $HOME folder, which is equivalent to a path that contains your user account, such as (if you were me):
/Users/wilsonmar
-
You can also use this variable to reach Home:
cd $HOME
In other words these commands all achieve the same result:
cd = cd ~ = cd $HOME
-
Use a text editor to edit the secrets.sh file using a text editor that comes pre-loaded on every Mac:
textedit ~/secrets.sh
The tilde character specifies that the file is in your Home folder.
Looking in the file, consider the first line in the secrets.sh file:
#!/bin/bash
That is the "Bourne-compliant" path for the Bash v3.2 shell installed by default on MacOS up to High Sierra. BTW, other Linux flavors may alternately use this for portability:
#!/usr/bin/env
BTW, unlike Windows, which determines the program to open files based on the suffix (or extension) of the file name, Linux shell programs such as Bash reference the "shebang" on the first line inside the file.
-
Open another Terminal window.
-
View the above files to see that they are binary executable files, such as:
textedit /usr/bin/bash
-
Exit the file.
-
Press the command key with the back-tick (`) at the upper-left of the keyboard to switch among textedit windows.
If you instead see this on the first line:
#!/usr/local/bin/bash
that is the Bash program associated with Bash v4.
Version 4 is needed for "associative arrays" needed in the script.
This is why we needed to first upgrade Bash before running other scripts.
The initial secrets.sh file does not have keywords which specify additional apps to install.
-
Scroll down or press command+F to type an app keyword to find its category.
-
Scroll to the list of ports (listed above).
-
May sure that none of the ports are the same (conflicts).
-
Save the file and exit the text editor.
-
Now copy, switch, click and paste in a Terminal window to run this command:
sh -c "$(curl -fsSL https://raw.githubusercontent.com/wilsonmar/mac-setup/master/mac-setup-all.sh)"
The script referenced in the command obtains more files needed by cloning from a public GitHub repository to a folder under your home folder.
A folder is necessary to hold additional folders such as "hooks" used by Git (if marked for install.) File "mac-bash-profile.txt" contains starter entries to insert in ~/.bash_profile that is executed before MacOS opens a Terminal session. Ignore the other files.
On a 4mbps network the run takes less than 5 minutes for a minimal install.
PROTIP: A faster network or a proxy Nexus server providing installers within the firewall would speed things up and ensure that vetted installers are used.
-
Wait for the script to finish.
On a 4mbps network the run takes less than 5 minutes for a minimal install.
PROTIP: A faster network or a proxy nexus server providing installers within the firewall would speed things up a lot and ensure that vetted installers are used.
When the script ends it pops up a log file in the TextEdit program that comes with MacOS.
-
Switch to the TextEdit window by clicking it.
-
Scroll to review the log file. Press command+F to input text to search.
-
Close the log file by clicking the red button.
-
Switch to a Finder window to your account's Home folder and delete log files.
mac-setup
To update what is installed on your Mac, re-run the bash script.
-
cd into your Home folder to find the secrets.sh file.
-
Edit the file, then run again locally:
chmod +x mac-setup-all.sh ./mac-setup-all.sh
The chmod (pronounced "che-mod") changes the permissions for executing the file.
Now let's look at the Bash coding in Bash script file from https://github.com/wilsonmar/mac-setup
Apps on Apple's App Store for Mac need to be installed manually. Popular apps include:
- Office for Mac 2016
- BitDefender for OSX
- CrashPlan (for backups)
- Amazon Music
- HockeyApp RDP (Remote Desktop Protocol client for controlling Microsoft Windows servers)
- Colloquy IRC client (at https://github.com/colloquy/colloquy)
- etc.
The brew "mas" manages Apple Store apps, but it only manages apps that have already been paid for. But mas does not install apps new to your Apple Store account.
Apps added by specifying in JAVA_TOOLS are GUI apps.
Most other Java dependencies are specified by manually added in each custom app's pom.xml file to specify what Maven downloads from the Maven Central online repository of installers at
http://search.maven.org/#search%7Cga%7C1%7Cg%3A%22org.dbunit%22
Popular in the Maven Repository are:
-
yarn for code generation. JHipster uses it as an integrated tool in Java Spring development.
-
DbUnit extends the JUnit TestCase class to put databases into a known state between test runs. Written by Manuel Laflamme, DbUnit is added in the Maven pom.xml (or Ant) for download from Maven Central. See http://dbunit.wikidot.com/
-
mockito enables calls to be mocked as if they have been creted. Insert file java-mockito-maven.xml as a dependency to maven pom.xml See https://www.youtube.com/watch?v=GKUlQMrbtHE - May 28, 2016 and https://zeroturnaround.com/rebellabs/rebel-labs-report-go-away-bugs-keeping-your-code-safe-with-junit-testng-and-mockito/9/
-
TestNG See http://testng.org/doc/download.html and https://docs.mendix.com/howto/testing/create-automated-tests-with-testng
When using Gradle, insert file java-testng-gradle as a dependency to gradle working within Eclipse plug-in Build from source git://github.com/cbeust/testng.git using ./build-with-gradle
TODO: The Python edition of this will insert specs such as this in pom.xml files.
The script outputs logs to a file.
This is so that during runs, what appears on the command console are only what is relevant to debugging the current issue.
At the end of the script, the log is shown in an editor to enable search through the whole log.
To start the Jenkins server to a specified port:
<pre>jenkins --httpPort=$JENKINS_PORT &</pre>
The "&" puts the process in the background so that the script can continue running.
The response is a bunch of lines ending with "INFO: Jenkins is fully up and running".
Several other methods (which don't work now) are presented on the internet:
This blog, on Dec 29, 2014 recommends
sudo defaults write /Library/Preferences/org.jenkins-ci httpPort "$JENKINS_PORT" sudo launchctl unload /Library/LaunchDaemons/org.jenkins-ci.plist sudo launchctl load /Library/LaunchDaemons/org.jenkins-ci.plist
The command "jenkins" above is actually a bash script that invokes Java:
#!/bin/bash JAVA_HOME="$(/usr/libexec/java_home --version 1.8)" \ exec java -jar /usr/local/Cellar/jenkins/2.113/libexec/jenkins.war "$@"
The code within "$(...)" is run to obtain the value. In this case, it's:
<pre>/Library/Java/JavaVirtualMachines/jdk1.8.0_162.jdk/Contents/Home
</pre>
The link above is the folder where MacOS keeps the Java SDK. Java executables (java, javac, etc.) are in the bin folder below that location.
The path to jenkins.war and jenkins-cli.war executable files are physcally at:
ls /usr/local/opt/jenkins/libexec
Instead of specifying the port in the command, change the configuration file.
On MacOS, services are defined by plist files containing XML, such as this for Jenkins server:
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>Label</key>
<string>homebrew.mxcl.jenkins</string>
<key>ProgramArguments</key>
<array>
<string>/usr/libexec/java_home</string>
<string>-v</string>
<string>1.8</string>
<string>--exec</string>
<string>java</string>
<string>-Dmail.smtp.starttls.enable=true</string>
<string>-jar</string>
<string>/usr/local/opt/jenkins/libexec/jenkins.war</string>
<string>--httpListenAddress=127.0.0.1</string>
<string>--httpPort=8080</string>
</array>
<key>RunAtLoad</key>
<true/>
</dict>
</plist>
The "1.8" is the version of Java, described below.
The "httpPort=8080" default is customized using this variable in secrets.sh:
JENKINS_PORT="8082" # default 8080
The above is file homebrew.mxcl.jenkins.plist within folder /usr/local/opt/jenkins installed by brew. The folder is a symlink created by brew to the physical path where brew installed it:
/usr/local/Cellar/Jenkins/2.113/homebrew.mxcl.jenkins.plist
The "2.113" means that several versions of Jenkins can be installed side-by-side. This version number changes over time. So it is captured by command:
JENKINS_VERSION=$(jenkins --version) # 2.113
The folder is actually a symlnk which points to the physical folder defined by: JENKINS_CONF="/usr/local/Cellar/Jenkins/$JENKINS_VERSION/homebrew.mxcl.jenkins.plist"
The path is defined in a variable so simplify the sed command to make the change:
sed -i "s/httpPort=8080/httpPort=$JENKINS_PORT/g" $JENKINS_CONF
# --httpPort=8080 is default.
The command to view the server in the default internet browser (such as Safari, Chrome, etc.) is:
open "http://localhost:$JENKINS_PORT"
It's "http" and not "https" because a certificate has not been established yet.
When executed the first time, Jenkins displays this screen:
However, we don't want to open it from the command line script, but from a GUI automation script.
The script invokes a GUI automation script that opens the file mentioned on the web page above:
/Users/wilsonmar/.jenkins/secrets/initialAdminPassword
"/Users/wilsonmar" is represented by the environment variable named $HOME or ~ symbol, which would be different for you, with your own MacOS account name. Thus, the generic coding is:
JENKINS_SECRET=$(<$HOME/.jenkins/secrets/initialAdminPassword)
The file (and now $JENKINS_SECRET) contains a string in clear-text like "851ed535fd3249ab95a274d23242655c".
We then call a GUI automation script to get that string to paste it in the box labeled "Administrator Password" based on the id "security-token" defined in this HTML:
<input id="security-token" class="form-control" type="password" name="j_password">
This was determined by obtaining the outer HTML from Chrome Developer Tools.
The call is:
python tests/jenkins_secret_chrome.py chrome $JENKINS_PORT $JENKINS_SECRET
We use Selenium Python because it reads and writes system environment variables.
Use of Selenium and Python this way requires them to be installed before Jenkins and other web servers.
To shut down Jenkins,
PID="ps -A | grep -m1 'jenkins' | awk '{print $1}'"
fancy_echo "Shutting downn jenkins $PID ..."
kill $PID
The above is the automated approach to the manual on recommended by many blogs on the internet:
Some say in Finder look for Applications -> Utilities -> Activity Monitor
Others say use command:
ps -el | grep jenkins
Two lines would appear. One is the bash command to do the ps command.
The PID desired is the one that lists the path used to invoke Jenkins, described above:
/usr/bin/java -jar /usr/local/Cellar/jenkins/2.113/libexec/jenkins.war
kill 2134
That is the equivalent of Windows command "taskkill /F /PID XXXX"
There is also:
sudo service jenkins stop
Either way, the response expected is:
INFO: JVM is terminating. Shutting down Winstone
If the title is not found an error message like this appears on the console:
File "tests/jenkins_secret_chrome.py", line 30, in
assert "Jenkins [Jenkins]" in driver.title # bail out if not found.
AssertionError
Some put in a 5 second delay:
time.sleep(5)
Use of this feature requires a library to be specified at the top of the file:
import sys
Some also take a photo to "prove" that the result was achieved:
driver.save_screenshot('jenkins_secret_chrome.py' +utc_offset_sec+ '.png')
We put the name of the script file in the picture name to trace back to its origin. We put a time stamp in ISO 8601 format so that several png files sort by date.
utc_offset_sec = time.altzone if time.localtime().tm_isdst else time.timezone datetime.datetime.now().replace(tzinfo=datetime.timezone(offset=utc_offset_sec)).isoformat()
The long explanation is https://docs.python.org/2/library/datetime.html
NOTE:- webDriver.Close() - Close the browser window that currently has focus
- webDriver.Quit() - Calls Dispose()
- webDriver.Dispose() Closes all browser windows and safely ends the session
driver.quit() means that someone watching the script execute would only see the web app's screen for a split second.
We prefer to use id rather than name fields because the HTML standard states that id's are supposed to be unique in each web page.
Other similar scripts (listed in "References" below) run
http://groovy-lang.org/install.html
Dropbox, OneDrive, Google Drive, Amazon Drive
Some developers have not put their stuff from GitHub into Homebrew. So we need to read (scrape) the website and see what is listed, then grab the text and URL to download.
Such is the situation with font files at https://github.com/adobe-fonts/source-code-pro/releases/tag/variable-fonts The two files desired downloaded using the curl command are:
- https://github.com/adobe-fonts/source-code-pro/releases/download/variable-fonts/SourceCodeVariable-Italic.ttf
- https://github.com/adobe-fonts/source-code-pro/releases/download/variable-fonts/SourceCodeVariable-Roman.ttf
The files are downloaded into where MacOS holds fonts available to all users: /Library/Fonts/
ITerm2 can make use of these font files.
Lists of Mac programs:
-
https://github.com/paulirish/dotfiles/blob/master/brew-cask.sh (one of the earliest ones by a legend at Google)
-
https://github.com/andrewconnell/osx-install described at http://www.andrewconnell.com/blog/rapid-complete-install-reinstall-os-x-like-a-champ-in-three-ish-hours separates coreinstall.sh from myinstall.sh for personal preferences.
-
https://www.reddit.com/r/osx/comments/3u6mob/what_are_the_top_10_osx_applications_you_use/
-
https://github.com/jaywcjlove/awesome-mac/blob/master/README.md
-
https://medium.com/@ankushagarwal/maximize-developer-productivity-on-a-mac-a9ae6fbaedab