Skip to content

Latest commit

 

History

History
1126 lines (749 loc) · 28.3 KB

go_tips.md

File metadata and controls

1126 lines (749 loc) · 28.3 KB

...menustart

...menuend

Go Tips

Common

multiple characters replacement

r := strings.NewReplacer("(", "", ")", "")
fmt.Println( r.Replace( "a(b)c)d" )  )

Ternary expression

func If(condition bool, trueVal, falseVal interface{}) interface{} {  
    if condition {
        return trueVal
    }
    return falseVal
}

// If(i==0,"A","B").(string)

How to find out which types implement ByteReader interface in golang pkg ?

https://golang.org/search?q=ReadByte

don't use alias for enums 'cause this breaks type safety

package main
type Status = int
type Format = int // remove `=` to have type safety

const A Status = 1
const B Format = 1

func main() {
    println(A == B)   // true
}

the short form for slice initialization is a := []T{}

the zero value of a slice is nil

var s []int
fmt.Println(s, len(s), cap(s))
if s == nil {
    fmt.Println(s, "is nil!")
}
// Output:
// [] 0 0
// [] is nil!

var a []string
b := []string{}
fmt.Println( a==nil, b==nil )
fmt.Println(reflect.DeepEqual(a, []string{}))
fmt.Println(reflect.DeepEqual(b, []string{}))
// Output:
// true false
// false
// true

use %+v to print data with sufficient details

using range loop to iterate over array or slice

for _, c := range a[3:7] {...}

comparing timestamps by using time.Before or time.After.

be careful with empty struct struct{}

func f1() {
    var a, b struct{}
    print(&a, "\n", &b, "\n") // Prints same address
    fmt.Println(&a == &b)     // Comparison returns false
}

func f2() {
    var a, b struct{}
    fmt.Printf("%p\n%p\n", &a, &b) // Again, same address
    fmt.Println(&a == &b)          // ...but the comparison returns true
}

be careful with range in Go

  • for i := range a and for i, v := range &a doesn't make a copy of a
  • but for i, v := range a does

reading nonexistent key from map will not panic

  • value := map["no_key"] will be zero value
  • value, ok := map["no_key"] is much better

simple random element from a slice

[]string{"one", "two", "three"}[rand.Intn(3)]

Concurrency

T12 inherits the mutex lock , YES anonymous structs are cool

type words struct {
    sync.Mutex
    found map[string]int
}

func (w *words) add(word string, n int) {
    w.Lock()
    defer w.Unlock()
    ...
}

T14 Closing channels

  • write(send) to a closed channel will cause panic
    • the close function should be called only by a sender
  • when receiver has done, it should notify the sender that sender can safely close the channel now
  • implementation:
    • sender should take 2 channel, for example:
      • msg channel: for messages
      • done channel: the other is for notification
    • when receiver is done, nofity the sender, and now sender can close the channle safely

use chan struct{} to pass signal, chan bool makes it less clear

T15 Locking with buffered channels (size of 1, to replace Lock)

best candidate to make something once in a thread-safe way is sync.Once

  • don't use flags, mutexes, channels or atomics

concurrency-safe

  • In general the rule is this:
    • top-level functions like strings.Split or fmt.Printf or rand.Int63 may be called from any goroutine at any time
      • (otherwise programming with them would be too restrictive)
    • but objects you create (like a new bytes.Buffer or rand.Rand) must only be used by one goroutine at a time unless otherwise noted
      • (like net.Conn's documentation does).

Error

T17 Use custom error types to return additional information

Or use to wrap an error

$ go get github.com/pkg/errors
errors.Wrap(err, "additional message to a given error")

T18 Error variables , create errors as package-scoped variables and reference those variables

var ErrTimeout = errors.New("The request timed out") 

return "", ErrTimeout

Panics and goroutines

  • If a panic on a goroutine goes unhandled on that goroutine’s call stack, it crashes the entire program
  • github.com/Masterminds/cookoo/safely
    • safely.Go( xxxx )

use panic only in very specific situations, you have to handle error

Debug & Test

Accessing stack traces

import (
    "runtime/debug"
)

func bar() {
    debug.PrintStack()
}
import (
    "runtime"
)
func bar() {
    // Makes a buffer
    buf := make([]byte, 1024)
    // Writes the stack into the buffer
    runtime.Stack(buf, false)
    // Prints the results
    fmt.Printf(“Trace:\n %s\n", buf)
}

dump goroutines

  • To mimic the Java behaviour of stack-dump on SIGQUIT but still leaving the program running:
go func() {
sigs := make(chan os.Signal, 1)
      signal.Notify(sigs, syscall.SIGQUIT)
      buf := make([]byte, 1<<20)
      for {
          <-sigs
          stacklen := runtime.Stack(buf, true)
          log.Printf("=== received SIGQUIT ===\n*** goroutine dump...\n%s\n*** end\n", buf[:stacklen])
      }
}()

to get call stack we've runtime.Caller

func Caller(skip int) (pc uintptr, file string, line int, ok bool)

Generative testing , random test edge cases

  • testing/quick
// assume your Pad function always return 
// a string with given length
func Pad(s string, max uint) string {
    ...
}

func TestPadGenerative(t *testing.T) {
    // fn takes a string and a uint8,
    // runs Pad(), and checks that
    // the returned length is right.
    fn := func(s string, max uint8) bool {
        p := Pad(s, uint(max))
        return len(p) == int(max)
    }
    // Using testing/quick, you tell it to
    // run no more than 200
    // randomly generated tests of fn
    if err := quick.Check(fn, &quick.Config{MaxCount: 200}); err != nil {
        // You report any errors throug
        // the normal testing package
        t.Error(err)
    }
}

30 Parallel benchmarks and race detection

$ go test -bench . -cpu=1,2,4
$ go test -bench . -race -cpu=1,2,4

easy way to split test into different builds

  • use // +build integration and run them with go test -v --tags integration .

Web Cloud and Micro Service

T9 URL Faster routing

github.com/gorilla/mux

T49 Detecting timeouts

T50 resuming download with HTTP

T58 detect IP addr on the host

func main() {
    name, err := os.Hostname()
    if err != nil {
        fmt.Println(err)
        return
    }
    // Looks up the IP addresses
    // associated with the hostname
    addrs, err := net.LookupHost(name)
    if err != nil {
        fmt.Println(err)
        return
    }
    // Prints each of the IP addresses,
    // as there can be more than one
    for _, a := range addrs {
        fmt.Println(a)
    }
}

T59 Detecting where depend command exists on host

func checkDep(name string) error {
    // Checks whether the passed-in dependency 
    // is in one of the PATHs. 
    if _, err := exec.LookPath(name); err != nil {
        es := "Could not find '%s' in PATH: %s"
        return fmt.Errorf(es, name, err)
    }
    return nil
}

T60 Batch Cross-compiling

$ go get -u github.com/mitchellh/gox
$ gox \
  -os="linux darwin windows " \
  -arch="amd64 386" \
  -output="dist/{{.OS}}-{{.Arch}}/{{.Dir}}" .

T61 Monitoring the Go runtime

// Listing 9.9 Monitor an application’s runtime
func monitorRuntime() {
    log.Println("Number of CPUs:", runtime.NumCPU())
    m := &runtime.MemStats{}
    for {
        r := runtime.NumGoroutine()
        log.Println("Number of goroutines", r)
        runtime.ReadMemStats(m)
        log.Println("Allocated memory", m.Alloc)
        time.Sleep(10 * time.Second)
    }
}

T62 Reusing connections

  • Go tries to reuse connections out of the box.
  • It’s the patterns in an application’s code that can cause this to not happen., for example:
    • one response body not being closed before another HTTP request is made.
      • ALWAYS close http body
    • custom http.Transport without Dial property in which the KeepAlive is set.
tr := &http.Transport{
     TLSClientConfig:    &tls.Config{RootCAs: pool},
     DisableCompression: true,
     Dial: (&net.Dialer{
            Timeout:   30 * time.Second,
            KeepAlive: 30 * time.Second,
     }).Dial,
}

T63 Faster JSON marshal and unmarshal

$ go get -u github.com/ugorji/go/codec/codecgen

httputil.DumpRequest is very useful thing, don't create your own

func DumpRequest(req *http.Request, body bool) ([]byte, error)

Reflection

T70 Generating code with go generate

Performance

Do NOT overuse fmt.Sprintf in your hot path.

  • It is costly due to maintaining the buffer pool and dynamic dispatches for interfaces.
  • if you are doing fmt.Sprintf("%s%s", var1, var2), consider simple string concatenation.
  • if you are doing fmt.Sprintf("%x", var), consider using hex.EncodeToString or strconv.FormatInt(var, 16)

always discard body e.g. io.Copy(ioutil.Discard, resp.Body) if you don't use it

  • HTTP client's Transport will not reuse connections unless the body is read to completion and closed

don't use defer in a loop or you'll get a small memory leak

  • cause defers will grow your stack without the reason

don't forget to stop ticker, unless you need a leaked channel

ticker := time.NewTicker(1 * time.Second)
defer ticker.Stop()

sync.Map isn't a silver bullet, do not use it without a strong reasons

to hide a pointer from escape analysis you might carefully(!!!) use this func:

// noescape hides a pointer from escape analysis.  noescape is
// the identity function but escape analysis doesn't think the
// output depends on the input. noescape is inlined and currently
// compiles down to zero instructions.
func noescape(p unsafe.Pointer) unsafe.Pointer {
x := uintptr(p)
       return unsafe.Pointer(x ^ 0)
}

for fastest atomic swap you might use this m := (*map[int]int)(atomic.LoadPointer(&ptr))

use buffered I/O if you do many sequential reads or writes

  • to reduce number of syscalls

there are 2 ways to clear a map:

reuse map memory

for k := range m {
    delete(m, k)
}

allocate new

m = make(map[int]int)

Build

strip your binaries with this command go build -ldflags="-s -w" ...

tiniest Go docker image

CGO_ENABLED=0 go build -ldflags="-s -w" app.go && tar C app | docker import - myimage:latest

check if there are mistakes in code formatting diff -u <(echo -n) <(gofmt -d .)

check interface implementation during compilation

var _ io.Reader = (*MyFastReader)(nil)

sql

where in

var openids []interface{}
... // init openids

// expand multiple `?` in `where in ()`
table:= "tbl_user"
query := fmt.Sprintf(" select DISTINCT id from %s where id in (?" , table ) + strings.Repeat(",?", len(openid_list)-1) + ")" 

rows, err := db.Query( query ,   openid_list...  )
if err != nil {
    log.Fatalln( err )    
} 
defer rows.Close() 

for rows.Next() {
    var (
        openid string
    )
    if err := rows.Scan(&openid); err != nil {
        ... 
    }
    // DO something
}

Scan rows to struct

    for rows.Next() {
        var match Match
        // match == Person{0, ""}
        pointers := make([]interface{}, len(columnNames))
        // pointers == `[]interface{}{nil, nil}`
        structVal := reflect.ValueOf(&match).Elem()
        for i, colName := range columnNames {
            fieldVal := structVal.FieldByName(strings.Title(colName))
            // log.Println( i, colName, fieldVal )
            if !fieldVal.IsValid() {
                log.Println("field not valid")
                return nil, errors.New( "field not valid " +  colName )
            }
            pointers[i] = fieldVal.Addr().Interface()
        }
        // pointers == `[]interface{}{&int, &string}`
        err := rows.Scan(pointers...)
        if err != nil {
            // handle err
            log.Println(err)
            return nil , err
        }

        // log.Print( "%+v" , match )
    }

Parameter Placeholder Syntax

mysql postgreSQL Oracle
WHERE col = ? WHERE col = $1 WHERE col = :col
VALUES(?, ?, ?) VALUES($1, $2, $3) VALUES(:val1, :val2, :val3)

数据库操作

多行查询

  • db.Query , 返回 Rows, error
  • rows.Next() 迭代
  • rows.Scan() 读取行
  • 每次db.Query操作后, 都建议调用rows.Close().
	dbw.QueryDataPre()
	rows, err := dbw.Db.Query(`SELECT * From user where age >= 20 AND age < 30`)
	defer rows.Close()
	if err != nil {
		fmt.Printf("insert data error: %v\n", err)
		return
	}
	for rows.Next() {
		rows.Scan(&dbw.UserInfo.Id, &dbw.UserInfo.Name, &dbw.UserInfo.Age)
		if err != nil {
			fmt.Printf(err.Error())
			continue
		}
		if !dbw.UserInfo.Name.Valid {
			dbw.UserInfo.Name.String = ""
		}
		if !dbw.UserInfo.Age.Valid {
			dbw.UserInfo.Age.Int64 = 0
		}
		fmt.Println("get data, id: ", dbw.UserInfo.Id, " name: ", dbw.UserInfo.Name.String, " age: ", int(dbw.UserInfo.Age.Int64))
	}

	err = rows.Err()
	if err != nil {
		fmt.Printf(err.Error())
	}

单行查询

  • db.QueryRow , 返回 Row
  • rows.Scan() 读取行
  • err在Scan后才产生
var name string
err = db.QueryRow("select name from user where id = ?", 1).Scan(&name)

插入数据

	ret, err := dbw.Db.Exec(`INSERT INTO user (name, age) VALUES ("xys", 23)`)
	if err != nil {
		fmt.Printf("insert data error: %v\n", err)
		return
	}
	if LastInsertId, err := ret.LastInsertId(); nil == err {
		fmt.Println("LastInsertId:", LastInsertId)
	}
	if RowsAffected, err := ret.RowsAffected(); nil == err {
		fmt.Println("RowsAffected:", RowsAffected)
	}

其他

  • db.Prepare()返回的statement使用完之后需要手动关闭,即defer stmt.Close()

事务

CREATE PROCEDURE PRO2()
BEGIN
    DECLARE t_error INTEGER;
    DECLARE    CONTINUE HANDLER FOR SQLEXCEPTION SET t_error = 1;

    START TRANSACTION;
        INSERT INTO test_tab VALUES    (1, '2');
        INSERT INTO test_tab VALUES    (1, '3');

        IF t_error = 1 THEN
            ROLLBACK;
        ELSE
            COMMIT;
        END IF;
END
  • Go uses sql.DB for autocommit, and sql.Tx for manual commit.
  • 事务场景
func clearTransaction(tx *sql.Tx){
    err := tx.Rollback()
    if err != sql.ErrTxDone && err != nil{
        log.Println(err)
    }
}
func DoSomething() error {
    tx , err := db.Begin()
    if err != nil {
        log.Println( err )    
        return err
    }
    defer clearTransaction( tx )


    if _, err = tx.Exec(...); err != nil {
        return err
    }
    if _, err = tx.Exec(...); err != nil {
        return err
    }
    // ...

    if err := tx.Commit(); err != nil {
        log.Println( err )    
        return err
    }
    return nil
}
  • 事务中,如果有 Query 命令, 执行后续命令之间,必需调用 rows.Close() 关闭连接

Misc

regexp , reference captured group

  • use $1 instead of \1

CGO_ENALBE=1 情况下, 实现纯静态连接

cmd/link的两种工作模式:internal linking和external linking。

1、internal linking

internal linking的大致意思是若用户代码中仅仅使用了net、os/user等几个标准库中的依赖cgo的包时,cmd/link默认使用internal linking,而无需启动外部external linker(如:gcc、clang等)

因此如果标准库是在CGO_ENABLED=1情况下编译的,那么编译出来的最终二进制文件依旧是动态链接的,即便在go build时传入 -ldflags '-extldflags "-static"'亦无用,因为根本没有使用external linker

2、external linking

而external linking机制则是cmd/link将所有生成的.o都打到一个.o文件中,再将其交给外部的链接器,比如gcc或clang去做最终链接处理。

如果此时,我们在cmd/link的参数中传入 -ldflags '-linkmode "external" -extldflags "-static"',那么gcc/clang将会去做静态链接,将.o中undefined的符号都替换为真正的代码。

因为 MacOSX 上只有 libc的dylib, 没有 .a , 所以一般会失败。

BASH: auto add current path to GOPATH when opening a new terminal window

if [ -d "./src" ] && [[ ! $GOPATH =~ $curDir ]]
then
    export GOPATH=$curDir:$GOPATH
fi

go time format

time format example

time parse , pls use the std format

const (
    stdLongMonth      = "January"
    stdMonth          = "Jan"
    stdNumMonth       = "1"
    stdZeroMonth      = "01"
    stdLongWeekDay    = "Monday"
    stdWeekDay        = "Mon"
    stdDay            = "2"
    stdUnderDay       = "_2"
    stdZeroDay        = "02"
    stdHour           = "15"
    stdHour12         = "3"
    stdZeroHour12     = "03"
    stdMinute         = "4"
    stdZeroMinute     = "04"
    stdSecond         = "5"
    stdZeroSecond     = "05"
    stdLongYear       = "2006"
    stdYear           = "06"
    stdPM             = "PM"
    stdpm             = "pm"
    stdTZ             = "MST"
    stdISO8601TZ      = "Z0700"  // prints Z for UTC
    stdISO8601ColonTZ = "Z07:00" // prints Z for UTC
    stdNumTZ          = "-0700"  // always numeric
    stdNumShortTZ     = "-07"    // always numeric
    stdNumColonTZ     = "-07:00" // always numeric
)
  • a common used format : "20060102150405"

Forward request

    if REQ_NEED_PROXY {
        url, _ := url.Parse( "https://xxxxxx" )
        proxy := httputil.NewSingleHostReverseProxy(url)

        r.URL.Host = url.Host
        r.URL.Scheme = "https"
        r.Header.Set("X-Forwarded-Host", r.Header.Get("Host"))
        r.Host = url.Host
        // the body has been read ?
        // r.Body = ioutil.NopCloser(bytes.NewBuffer([]byte(body)))
        // if different uri path
        // r.URL.Path = "/new/uri/path"

        // Note that ServeHttp is non blocking and uses a go routine under the hood
        proxy.ServeHTTP(w, r)
        return
    }