A small go app to crawl a web site and print out the links to the pages it finds. These are separated into off site links, resources and pages.
sudo yum install git sudo yum install golang mkdir go
Create env file
sudo vi /etc/profile.d/goenv.sh
export GOROOT=/usr/lib/golang export GOPATH=$HOME/go export PATH=$PATH:$GOROOT/bin:$GOPATH/bin
source /etc/profile.d/goenv.sh
to test installation create src/github.com/user_name/hello directory, where user_name is your github user
inside hello directory create a hello.go file and insert the following text
package main
import "fmt"
func main(){
fmt.Println("Hello world")
}
save this file and then run
go install github.com/user_name/hello
This will create the binary in your GOPATH/bin
run this "hello" binary
For more info please see https://golang.org/doc/install
from your /go/src/github.com/user_name directory
type
git clone https://github.com/mjdilworth/links.git
get a couple of third party packages used in links - good example of how to reuse code - many solved problems.
go get "github.com/asaskevich/govalidator"
go get "golang.org/x/net/html"
To run the code
go run links.go
and to build the binary in the src dir
go build links.go
and to install the binary into the project bin directory. From the project src directory
go install github.com/user_name/links
and to run the binary (-url switch is optional and can be used to change the seeding page)
links -url http://wiprodigital.com
I really like go and especially how its channels and goroutines make it fairly easy to implement concurrency. this is really useful when you want to simultaneously crawl many web pages. Using threads in C would be more complicated as there is nothing "out of the box".
Go also has great packages for testing and benchmarking. Unfortunately I didnt get the time to use any of these features.
I spent a lot more than 2 hours on this exercise and the result is pretty.. er basic.. and not really production quality. I am not really happy with it. I have not written any code for ages and certainly never in go. It took a long time to set up the environment and decide on editor, to do real basic stuff etc. Its a great language and pretty simple, but to be quick at stuff you do need to use it fairly frequently, or at least i do.
anyway, I like go and will endeavour to become more accomplished with it.