...menustart
- Common
- multiple characters replacement
- Ternary expression
- How to find out which types implement ByteReader interface in golang pkg ?
- don't use alias for enums 'cause this breaks type safety
- the short form for slice initialization is
a := \[\]T{}
- use
%+v
to print data with sufficient details - using range loop to iterate over array or slice
- comparing timestamps by using
time.Before
ortime.After
. - be careful with empty struct
struct{}
- be careful with range in Go
- reading nonexistent key from map will not panic
- simple random element from a slice
- Concurrency
- Error
- Debug & Test
- Web Cloud and Micro Service
- T9 URL Faster routing
- T49 Detecting timeouts
- T50 resuming download with HTTP
- T58 detect IP addr on the host
- T59 Detecting where depend command exists on host
- T60 Batch Cross-compiling
- T61 Monitoring the Go runtime
- T62 Reusing connections
- T63 Faster JSON marshal and unmarshal
- httputil.DumpRequest is very useful thing, don't create your own
- Reflection
- Performance
- Do NOT overuse fmt.Sprintf in your hot path.
- always discard body e.g.
io.Copy(ioutil.Discard, resp.Body)
if you don't use it - don't use defer in a loop or you'll get a small memory leak
- don't forget to stop ticker, unless you need a leaked channel
sync.Map
isn't a silver bullet, do not use it without a strong reasons- to hide a pointer from escape analysis you might carefully(!!!) use this func:
- for fastest atomic swap you might use this
m := (*map\[int\]int)(atomic.LoadPointer(&ptr))
- use buffered I/O if you do many sequential reads or writes
- there are 2 ways to clear a map:
- Build
- sql
- Misc
...menuend
Go Tips
r := strings.NewReplacer("(", "", ")", "")
fmt.Println( r.Replace( "a(b)c)d" ) )
func If(condition bool, trueVal, falseVal interface{}) interface{} {
if condition {
return trueVal
}
return falseVal
}
// If(i==0,"A","B").(string)
https://golang.org/search?q=ReadByte
package main
type Status = int
type Format = int // remove `=` to have type safety
const A Status = 1
const B Format = 1
func main() {
println(A == B) // true
}
var s []int
fmt.Println(s, len(s), cap(s))
if s == nil {
fmt.Println(s, "is nil!")
}
// Output:
// [] 0 0
// [] is nil!
var a []string
b := []string{}
fmt.Println( a==nil, b==nil )
fmt.Println(reflect.DeepEqual(a, []string{}))
fmt.Println(reflect.DeepEqual(b, []string{}))
// Output:
// true false
// false
// true
for _, c := range a[3:7] {...}
func f1() {
var a, b struct{}
print(&a, "\n", &b, "\n") // Prints same address
fmt.Println(&a == &b) // Comparison returns false
}
func f2() {
var a, b struct{}
fmt.Printf("%p\n%p\n", &a, &b) // Again, same address
fmt.Println(&a == &b) // ...but the comparison returns true
}
for i := range a
andfor i, v := range &a
doesn't make a copy of a- but
for i, v := range
a does
value := map["no_key"]
will be zero valuevalue, ok := map["no_key"]
is much better
[]string{"one", "two", "three"}[rand.Intn(3)]
type words struct {
sync.Mutex
found map[string]int
}
func (w *words) add(word string, n int) {
w.Lock()
defer w.Unlock()
...
}
- write(send) to a closed channel will cause panic
- the close function should be called only by a sender
- when receiver has done, it should notify the sender that sender can safely close the channel now
- implementation:
- sender should take 2 channel, for example:
msg
channel: for messagesdone
channel: the other is for notification
- when receiver is done, nofity the sender, and now sender can close the channle safely
- sender should take 2 channel, for example:
- don't use flags, mutexes, channels or atomics
- In general the rule is this:
- top-level functions like strings.Split or fmt.Printf or rand.Int63 may be called from any goroutine at any time
- (otherwise programming with them would be too restrictive)
- but objects you create (like a new bytes.Buffer or rand.Rand) must only be used by one goroutine at a time unless otherwise noted
- (like net.Conn's documentation does).
- top-level functions like strings.Split or fmt.Printf or rand.Int63 may be called from any goroutine at any time
$ go get github.com/pkg/errors
errors.Wrap(err, "additional message to a given error")
var ErrTimeout = errors.New("The request timed out")
return "", ErrTimeout
- If a panic on a goroutine goes unhandled on that goroutine’s call stack, it crashes the entire program
github.com/Masterminds/cookoo/safely
safely.Go( xxxx )
import (
"runtime/debug"
)
func bar() {
debug.PrintStack()
}
import (
"runtime"
)
func bar() {
// Makes a buffer
buf := make([]byte, 1024)
// Writes the stack into the buffer
runtime.Stack(buf, false)
// Prints the results
fmt.Printf(“Trace:\n %s\n", buf)
}
- To mimic the Java behaviour of stack-dump on SIGQUIT but still leaving the program running:
go func() {
sigs := make(chan os.Signal, 1)
signal.Notify(sigs, syscall.SIGQUIT)
buf := make([]byte, 1<<20)
for {
<-sigs
stacklen := runtime.Stack(buf, true)
log.Printf("=== received SIGQUIT ===\n*** goroutine dump...\n%s\n*** end\n", buf[:stacklen])
}
}()
- On
*NIX
systems (including OSX) send a signal abort SIGQUIT:pkill -SIGQUIT program_name
- stackoverflow discussion
func Caller(skip int) (pc uintptr, file string, line int, ok bool)
testing/quick
// assume your Pad function always return
// a string with given length
func Pad(s string, max uint) string {
...
}
func TestPadGenerative(t *testing.T) {
// fn takes a string and a uint8,
// runs Pad(), and checks that
// the returned length is right.
fn := func(s string, max uint8) bool {
p := Pad(s, uint(max))
return len(p) == int(max)
}
// Using testing/quick, you tell it to
// run no more than 200
// randomly generated tests of fn
if err := quick.Check(fn, &quick.Config{MaxCount: 200}); err != nil {
// You report any errors throug
// the normal testing package
t.Error(err)
}
}
$ go test -bench . -cpu=1,2,4
$ go test -bench . -race -cpu=1,2,4
- use
// +build integration
and run them withgo test -v --tags integration
.
github.com/gorilla/mux
func main() {
name, err := os.Hostname()
if err != nil {
fmt.Println(err)
return
}
// Looks up the IP addresses
// associated with the hostname
addrs, err := net.LookupHost(name)
if err != nil {
fmt.Println(err)
return
}
// Prints each of the IP addresses,
// as there can be more than one
for _, a := range addrs {
fmt.Println(a)
}
}
func checkDep(name string) error {
// Checks whether the passed-in dependency
// is in one of the PATHs.
if _, err := exec.LookPath(name); err != nil {
es := "Could not find '%s' in PATH: %s"
return fmt.Errorf(es, name, err)
}
return nil
}
$ go get -u github.com/mitchellh/gox
$ gox \
-os="linux darwin windows " \
-arch="amd64 386" \
-output="dist/{{.OS}}-{{.Arch}}/{{.Dir}}" .
// Listing 9.9 Monitor an application’s runtime
func monitorRuntime() {
log.Println("Number of CPUs:", runtime.NumCPU())
m := &runtime.MemStats{}
for {
r := runtime.NumGoroutine()
log.Println("Number of goroutines", r)
runtime.ReadMemStats(m)
log.Println("Allocated memory", m.Alloc)
time.Sleep(10 * time.Second)
}
}
- Go tries to reuse connections out of the box.
- It’s the patterns in an application’s code that can cause this to not happen., for example:
- one response body not being closed before another HTTP request is made.
- ALWAYS close http body
- custom
http.Transport
withoutDial
property in which theKeepAlive
is set.
- one response body not being closed before another HTTP request is made.
tr := &http.Transport{
TLSClientConfig: &tls.Config{RootCAs: pool},
DisableCompression: true,
Dial: (&net.Dialer{
Timeout: 30 * time.Second,
KeepAlive: 30 * time.Second,
}).Dial,
}
$ go get -u github.com/ugorji/go/codec/codecgen
func DumpRequest(req *http.Request, body bool) ([]byte, error)
- It is costly due to maintaining the buffer pool and dynamic dispatches for interfaces.
- if you are doing fmt.Sprintf("%s%s", var1, var2), consider simple string concatenation.
- if you are doing fmt.Sprintf("%x", var), consider using hex.EncodeToString or strconv.FormatInt(var, 16)
- HTTP client's Transport will not reuse connections unless the body is read to completion and closed
- cause defers will grow your stack without the reason
ticker := time.NewTicker(1 * time.Second)
defer ticker.Stop()
// noescape hides a pointer from escape analysis. noescape is
// the identity function but escape analysis doesn't think the
// output depends on the input. noescape is inlined and currently
// compiles down to zero instructions.
func noescape(p unsafe.Pointer) unsafe.Pointer {
x := uintptr(p)
return unsafe.Pointer(x ^ 0)
}
- to reduce number of syscalls
reuse map memory
for k := range m {
delete(m, k)
}
allocate new
m = make(map[int]int)
CGO_ENABLED=0 go build -ldflags="-s -w" app.go && tar C app | docker import - myimage:latest
var _ io.Reader = (*MyFastReader)(nil)
var openids []interface{}
... // init openids
// expand multiple `?` in `where in ()`
table:= "tbl_user"
query := fmt.Sprintf(" select DISTINCT id from %s where id in (?" , table ) + strings.Repeat(",?", len(openid_list)-1) + ")"
rows, err := db.Query( query , openid_list... )
if err != nil {
log.Fatalln( err )
}
defer rows.Close()
for rows.Next() {
var (
openid string
)
if err := rows.Scan(&openid); err != nil {
...
}
// DO something
}
for rows.Next() {
var match Match
// match == Person{0, ""}
pointers := make([]interface{}, len(columnNames))
// pointers == `[]interface{}{nil, nil}`
structVal := reflect.ValueOf(&match).Elem()
for i, colName := range columnNames {
fieldVal := structVal.FieldByName(strings.Title(colName))
// log.Println( i, colName, fieldVal )
if !fieldVal.IsValid() {
log.Println("field not valid")
return nil, errors.New( "field not valid " + colName )
}
pointers[i] = fieldVal.Addr().Interface()
}
// pointers == `[]interface{}{&int, &string}`
err := rows.Scan(pointers...)
if err != nil {
// handle err
log.Println(err)
return nil , err
}
// log.Print( "%+v" , match )
}
mysql | postgreSQL | Oracle |
---|---|---|
WHERE col = ? | WHERE col = $1 | WHERE col = :col |
VALUES(?, ?, ?) | VALUES($1, $2, $3) | VALUES(:val1, :val2, :val3) |
- db.Query , 返回 Rows, error
- rows.Next() 迭代
- rows.Scan() 读取行
- 每次db.Query操作后, 都建议调用rows.Close().
dbw.QueryDataPre()
rows, err := dbw.Db.Query(`SELECT * From user where age >= 20 AND age < 30`)
defer rows.Close()
if err != nil {
fmt.Printf("insert data error: %v\n", err)
return
}
for rows.Next() {
rows.Scan(&dbw.UserInfo.Id, &dbw.UserInfo.Name, &dbw.UserInfo.Age)
if err != nil {
fmt.Printf(err.Error())
continue
}
if !dbw.UserInfo.Name.Valid {
dbw.UserInfo.Name.String = ""
}
if !dbw.UserInfo.Age.Valid {
dbw.UserInfo.Age.Int64 = 0
}
fmt.Println("get data, id: ", dbw.UserInfo.Id, " name: ", dbw.UserInfo.Name.String, " age: ", int(dbw.UserInfo.Age.Int64))
}
err = rows.Err()
if err != nil {
fmt.Printf(err.Error())
}
- db.QueryRow , 返回 Row
- rows.Scan() 读取行
- err在Scan后才产生
var name string
err = db.QueryRow("select name from user where id = ?", 1).Scan(&name)
ret, err := dbw.Db.Exec(`INSERT INTO user (name, age) VALUES ("xys", 23)`)
if err != nil {
fmt.Printf("insert data error: %v\n", err)
return
}
if LastInsertId, err := ret.LastInsertId(); nil == err {
fmt.Println("LastInsertId:", LastInsertId)
}
if RowsAffected, err := ret.RowsAffected(); nil == err {
fmt.Println("RowsAffected:", RowsAffected)
}
- db.Prepare()返回的statement使用完之后需要手动关闭,即defer stmt.Close()
CREATE PROCEDURE PRO2()
BEGIN
DECLARE t_error INTEGER;
DECLARE CONTINUE HANDLER FOR SQLEXCEPTION SET t_error = 1;
START TRANSACTION;
INSERT INTO test_tab VALUES (1, '2');
INSERT INTO test_tab VALUES (1, '3');
IF t_error = 1 THEN
ROLLBACK;
ELSE
COMMIT;
END IF;
END
- Go uses sql.DB for autocommit, and sql.Tx for manual commit.
- 事务场景
func clearTransaction(tx *sql.Tx){
err := tx.Rollback()
if err != sql.ErrTxDone && err != nil{
log.Println(err)
}
}
func DoSomething() error {
tx , err := db.Begin()
if err != nil {
log.Println( err )
return err
}
defer clearTransaction( tx )
if _, err = tx.Exec(...); err != nil {
return err
}
if _, err = tx.Exec(...); err != nil {
return err
}
// ...
if err := tx.Commit(); err != nil {
log.Println( err )
return err
}
return nil
}
- 事务中,如果有 Query 命令, 执行后续命令之间,必需调用 rows.Close() 关闭连接
- use
$1
instead of\1
cmd/link的两种工作模式:internal linking和external linking。
1、internal linking
internal linking的大致意思是若用户代码中仅仅使用了net、os/user等几个标准库中的依赖cgo的包时,cmd/link默认使用internal linking,而无需启动外部external linker(如:gcc、clang等)
因此如果标准库是在CGO_ENABLED=1情况下编译的,那么编译出来的最终二进制文件依旧是动态链接的,即便在go build时传入 -ldflags '-extldflags "-static"'亦无用,因为根本没有使用external linker
2、external linking
而external linking机制则是cmd/link将所有生成的.o都打到一个.o文件中,再将其交给外部的链接器,比如gcc或clang去做最终链接处理。
如果此时,我们在cmd/link的参数中传入 -ldflags '-linkmode "external" -extldflags "-static"',那么gcc/clang将会去做静态链接,将.o中undefined的符号都替换为真正的代码。
因为 MacOSX 上只有 libc的dylib, 没有 .a , 所以一般会失败。
if [ -d "./src" ] && [[ ! $GOPATH =~ $curDir ]]
then
export GOPATH=$curDir:$GOPATH
fi
time parse , pls use the std format
const (
stdLongMonth = "January"
stdMonth = "Jan"
stdNumMonth = "1"
stdZeroMonth = "01"
stdLongWeekDay = "Monday"
stdWeekDay = "Mon"
stdDay = "2"
stdUnderDay = "_2"
stdZeroDay = "02"
stdHour = "15"
stdHour12 = "3"
stdZeroHour12 = "03"
stdMinute = "4"
stdZeroMinute = "04"
stdSecond = "5"
stdZeroSecond = "05"
stdLongYear = "2006"
stdYear = "06"
stdPM = "PM"
stdpm = "pm"
stdTZ = "MST"
stdISO8601TZ = "Z0700" // prints Z for UTC
stdISO8601ColonTZ = "Z07:00" // prints Z for UTC
stdNumTZ = "-0700" // always numeric
stdNumShortTZ = "-07" // always numeric
stdNumColonTZ = "-07:00" // always numeric
)
- a common used format : "20060102150405"
if REQ_NEED_PROXY {
url, _ := url.Parse( "https://xxxxxx" )
proxy := httputil.NewSingleHostReverseProxy(url)
r.URL.Host = url.Host
r.URL.Scheme = "https"
r.Header.Set("X-Forwarded-Host", r.Header.Get("Host"))
r.Host = url.Host
// the body has been read ?
// r.Body = ioutil.NopCloser(bytes.NewBuffer([]byte(body)))
// if different uri path
// r.URL.Path = "/new/uri/path"
// Note that ServeHttp is non blocking and uses a go routine under the hood
proxy.ServeHTTP(w, r)
return
}