Panic on encoding struct in a loop

[philband]

I am currently experiencing a panic when encoding a struct during some JSON-testdata generation.
The testdata in this case is a repetition of the same struct being encoded 50k times.
The first 3-5k encodings work, after that I always get a panic.
Encoding with stdlib encoding/json works without a problem.
If you need any additional information to debug this I am happy to assist.

unexpected fault address 0xc000714000
fatal error: fault
[signal 0xc0000005 code=0x0 addr=0xc000714000 pc=0x705ea9]

goroutine 27 [running]:
runtime.throw({0xf9f839, 0x400})
	C:/Program Files/Go/src/runtime/panic.go:1198 +0x76 fp=0xc00015c8a8 sp=0xc00015c878 pc=0x2e6f76
runtime.sigpanic()
	C:/Program Files/Go/src/runtime/signal_windows.go:260 +0x10c fp=0xc00015c8f0 sp=0xc00015c8a8 pc=0x2f9e6c
github.com/goccy/go-json/internal/encoder/vm.ptrToUint64(...)
	C:/Users/philband/go/pkg/mod/github.com/goccy/go-json@v0.7.9/internal/encoder/vm/util.go:71
github.com/goccy/go-json/internal/encoder/vm.Run(0xc0008da9c0, {0xc0000f5c00, 0xc00015fdf0, 0x2bd234}, 0xc00015fda0)
	C:/Users/philband/go/pkg/mod/github.com/goccy/go-json@v0.7.9/internal/encoder/vm/vm.go:4360 +0x1df29 fp=0xc00015fd60 sp=0xc00015c8f0 pc=0x705ea9
github.com/goccy/go-json/internal/encoder/vm.DebugRun(0x8, {0xc0000f5c00, 0x0, 0x40}, 0x64184e)
	C:/Users/philband/go/pkg/mod/github.com/goccy/go-json@v0.7.9/internal/encoder/vm/debug_vm.go:33 +0x7f fp=0xc00015fde8 sp=0xc00015fd60 pc=0x6e785f
github.com/goccy/go-json.encodeRunCode(0xf04720, {0xc0000f5c00, 0xc0000f5c00, 0xc00093a29a}, 0x2030000)
	C:/Users/philband/go/pkg/mod/github.com/goccy/go-json@v0.7.9/encode.go:302 +0x45 fp=0xc00015fe20 sp=0xc00015fde8 pc=0x7196a5
github.com/goccy/go-json.encode(0xc0008da9c0, {0xf04720, 0xc000713ff0})
	C:/Users/philband/go/pkg/mod/github.com/goccy/go-json@v0.7.9/encode.go:232 +0x21c fp=0xc00015fea0 sp=0xc00015fe20 pc=0x7191dc
github.com/goccy/go-json.(*Encoder).encodeWithOption(0xc00015ffa8, 0xc0008da9c0, {0xf04720, 0xc000713ff0}, {0xc00015ff98, 0x1, 0xc00015ff58})
	C:/Users/philband/go/pkg/mod/github.com/goccy/go-json@v0.7.9/encode.go:74 +0xeb fp=0xc00015ff00 sp=0xc00015fea0 pc=0x718cab
github.com/goccy/go-json.(*Encoder).EncodeWithOption(0xc0000f5c00, {0xf04720, 0xc000713ff0}, {0xc00015ff98, 0x1, 0x1})
	C:/Users/philband/go/pkg/mod/github.com/goccy/go-json@v0.7.9/encode.go:41 +0x92 fp=0xc00015ff68 sp=0xc00015ff00 pc=0x718b32
gitlab.example.org/fsw-dl/dl/pkg/envelope.Envelope.MarshalJSON({{0xf98217, 0x2}, 0xcaa, {0xc04fcbe570f13814, 0x3475845, 0x193a680}, {0x0, 0x0}, {0xf04720, 0xc000713ff0}})
	C:/Users/philband/go/src/gitlab.example.org/fsw-dl/dl/pkg/envelope/envelope.go:143 +0xb0 fp=0xc0001600a0 sp=0xc00015ff68 pc=0xd83830
gitlab.example.org/fsw-dl/dl/pkg/envelope.(*Envelope).MarshalJSON(0xc00014f6d0)
	<autogenerated>:1 +0x78 fp=0xc000160150 sp=0xc0001600a0 pc=0xd87758
github.com/goccy/go-json/internal/encoder.AppendMarshalJSON(0xc0005bc000, 0xc00019e400, {0xc000baa000, 0x0, 0x400}, {0xf28140, 0xc00014f6d0})
	C:/Users/philband/go/pkg/mod/github.com/goccy/go-json@v0.7.9/internal/encoder/encoder.go:387 +0x41c fp=0xc0001602c8 sp=0xc000160150 pc=0x64cf1c
github.com/goccy/go-json/internal/encoder/vm.appendMarshalJSON(0x0, 0x0, {0xc000baa000, 0x0, 0x0}, {0xf28140, 0xc00014f6d0})
	C:/Users/philband/go/pkg/mod/github.com/goccy/go-json@v0.7.9/internal/encoder/vm/util.go:136 +0x28 fp=0xc000160310 sp=0xc0001602c8 pc=0x6e7d88
github.com/goccy/go-json/internal/encoder/vm.Run(0xc0005bc000, {0xc000baa000, 0x1b29d620108, 0x70}, 0xc000680000)
	C:/Users/philband/go/pkg/mod/github.com/goccy/go-json@v0.7.9/internal/encoder/vm/vm.go:280 +0x17f3 fp=0xc000163780 sp=0xc000160310 pc=0x6e9773
github.com/goccy/go-json/internal/encoder/vm.DebugRun(0x0, {0xc000baa000, 0x9, 0x70}, 0x1b2c2a5a7b6)
	C:/Users/philband/go/pkg/mod/github.com/goccy/go-json@v0.7.9/internal/encoder/vm/debug_vm.go:33 +0x7f fp=0xc000163808 sp=0xc000163780 pc=0x6e785f
github.com/goccy/go-json.encodeRunCode(0xf28140, {0xc000baa000, 0x2030000, 0x20}, 0x2030000)
	C:/Users/philband/go/pkg/mod/github.com/goccy/go-json@v0.7.9/encode.go:302 +0x45 fp=0xc000163840 sp=0xc000163808 pc=0x7196a5
github.com/goccy/go-json.encode(0xc0005bc000, {0xf28140, 0xc00014f6d0})
	C:/Users/philband/go/pkg/mod/github.com/goccy/go-json@v0.7.9/encode.go:232 +0x21c fp=0xc0001638c0 sp=0xc000163840 pc=0x7191dc
github.com/goccy/go-json.(*Encoder).encodeWithOption(0xc000163aa0, 0xc0005bc000, {0xf28140, 0xc00014f6d0}, {0xc000163a40, 0x1, 0xf28140})
	C:/Users/philband/go/pkg/mod/github.com/goccy/go-json@v0.7.9/encode.go:74 +0xeb fp=0xc000163920 sp=0xc0001638c0 pc=0x718cab
github.com/goccy/go-json.(*Encoder).EncodeWithOption(0xf28140, {0xf28140, 0xc00014f6d0}, {0xc000163a40, 0x1, 0x1})
	C:/Users/philband/go/pkg/mod/github.com/goccy/go-json@v0.7.9/encode.go:41 +0x92 fp=0xc000163988 sp=0xc000163920 pc=0x718b32
gitlab.example.org/fsw-dl/dl/pkg/envelope.(*Wal).Write(0xc00059e540, {0xc00013d800, 0x15, 0xc04fcbe570f13814})
	C:/Users/philband/go/src/gitlab.example.org/fsw-dl/dl/pkg/envelope/wal.go:224 +0x43f fp=0xc000163af0 sp=0xc000163988 pc=0xd861df
gitlab.example.org/fsw-dl/dl/pkg/envelope.(*EnvelopeLogger).RunLogger(0xc000834af0, {0x1184d58, 0xc00045c600}, 0xc0000a36e0)
	C:/Users/philband/go/src/gitlab.example.org/fsw-dl/dl/pkg/envelope/envelope_logger.go:111 +0xc28 fp=0xc000163f48 sp=0xc000163af0 pc=0xd84788
main.main.func1()
	C:/Users/philband/go/src/gitlab.example.org/fsw-dl/dl/cmd/client/main.go:74 +0x29 fp=0xc000163f78 sp=0xc000163f48 pc=0xd889c9
golang.org/x/sync/errgroup.(*Group).Go.func1()
	C:/Users/philband/go/pkg/mod/golang.org/x/sync@v0.0.0-20210220032951-036812b2e83c/errgroup/errgroup.go:57 +0x67 fp=0xc000163fe0 sp=0xc000163f78 pc=0xabf9a7
runtime.goexit()
	C:/Program Files/Go/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc000163fe8 sp=0xc000163fe0 pc=0x314ce1
created by golang.org/x/sync/errgroup.(*Group).Go
	C:/Users/philband/go/pkg/mod/golang.org/x/sync@v0.0.0-20210220032951-036812b2e83c/errgroup/errgroup.go:54 +0x92

The data being encoded (static only for testing)

Envelope{
	Type: LogMessageIvtName,
	Data: LogMessageIvt{
		Timestamp:   1456465,
		Voltage:     600000,
		Current:     50000,
		Temperature: 544,
	},
}

Relevant Code extracts

type EnvelopeName string

const (
	LogMessageIvtName  EnvelopeName = "lm"
)

type Envelope struct {
	Type      EnvelopeName `json:"t"`                   
	Index     uint64       `json:"i,omitempty"`         
	Timestamp time.Time    `json:"ts,omitempty"`        
	Signature string       `json:"s,omitempty" hash:"-"`
	Data      interface{}  `json:"d"`                   
}

type EnvelopeWithRawData struct {
	Type      EnvelopeName    `json:"t"`                    
	Index     uint64          `json:"i,omitempty"`          
	Timestamp time.Time       `json:"ts,omitempty"`         
	Signature string          `json:"s,omitempty" hash:"-"` 
	Data      json.RawMessage `json:"d"`                    
}

type LogMessageIvt struct {
	Timestamp   uint32 `json:"ts"`
	Voltage     int32  `json:"u"`
	Current     int32  `json:"i"`
	Temperature int32  `json:"temp"`
}

func (t Envelope) MarshalJSON() ([]byte, error) {
	jsonData, err := json.Marshal(t.Data)
	if err != nil {
		return nil, err
	}
	e := EnvelopeWithRawData{
		Type:      t.Type,
		Index:     t.Index,
		Timestamp: t.Timestamp,
		Signature: t.Signature,
		Data:      jsonData,
	}
	if !e.Timestamp.IsZero() {
		e.Timestamp = time.Now()
	}
	return json.Marshal(e)
}

[goccy]

Thank you for the reporting !

Looking at the stacktrace, it looks like you have the Debug option enabled.
I'd like to investigate this problem, but it seems difficult to deal with from this information, so could you show me something close to the code you were actually executing ?

[philband]

Thanks for getting back to me so quickly.
You are correct, I have the Debug option enabled, I browsed a little bit through previous issues before posting this.
I have created a full reproduction of the issue here: https://github.com/philband/go-json-panic-repro

During working on the reproduction, I found out that this issue only happens on windows-amd64.
linux-amd64 works without problems. Maybe that has something todo with memory protections that are working differently on linux?

[goccy]

Thank you for providing me with more detailed information.

During working on the reproduction, I found out that this issue only happens on windows-amd64.

Oh..., If the Windows environment is the cause of this problem, it can be difficult to investigate ( currently, I don't have a Windows )

For the time being, I will check if it can be reproduced on my local environment ( macOS ) .

[philband]

I think I finally have a clue to what is happening here:

go-json/internal/encoder/vm/vm.go

Line 4360 in d1195df

b = appendInt(ctx, b, ptrToUint64(p+uintptr(code.Offset)), code)

The above line is trying to create a ptr to an uint64, however, my underlying datatype is only an int32 or uint32.
Therefore, in some cases near the edges of the buffers, the pointer might point half to the int32, whilst the second 32 bits referenced by the pointer are unallocated memory.

I suppose the same problem would occur with all int types of less than 64 bits.

[twocs]

I was running code at this issue, so if it helps, here is code to reproduce with magic numbers from a non-panic and panic:

package main

import (
	"fmt"
	"unsafe"
)

func ptrToUint64(p uintptr) uint64 { return **(**uint64)(unsafe.Pointer(&p)) }

func main() {
	fmt.Println("won't panic:", ptrToUint64(824633827884))
	fmt.Println("will panic:", ptrToUint64(824650285056))
}

$ go run main.go
won't panic: 0
unexpected fault address 0xc000fcc000
fatal error: fault
[signal SIGSEGV: segmentation violation code=0x2 addr=0xc000fcc000 pc=0x47e2cf]

goroutine 1 [running]:
runtime.throw({0x4948ee, 0xc000136000})
        /usr/local/go/src/runtime/panic.go:1198 +0x71 fp=0xc000114eb8 sp=0xc000114e88 pc=0x42f971
runtime.sigpanic()
        /usr/local/go/src/runtime/signal_unix.go:742 +0x2f6 fp=0xc000114f08 sp=0xc000114eb8 pc=0x443476
main.ptrToUint64(...)
        /mnt/c/Users/twocs/github/go-json-panic-repro/main.go:8
main.main()
        /mnt/c/Users/twocs/github/go-json-panic-repro/main.go:13 +0x8f fp=0xc000114f80 sp=0xc000114f08 pc=0x47e2cf
runtime.main()
        /usr/local/go/src/runtime/proc.go:255 +0x227 fp=0xc000114fe0 sp=0xc000114f80 pc=0x432047
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc000114fe8 sp=0xc000114fe0 pc=0x45ab61
exit status 2

[goccy]

That makes sense. Thank you for providing the information !
When encoding integer, go-json converts pointer to uint64 and then masks it to convert it to a value such as int32.
Therefore, depending on the state of memory, it seems that an error occurs when converting from pointer to uint64.

[goccy]

@philband
I was probably able to fix this problem with #294 .
Could you please try to test with the feature branch ? ( go get github.com/goccy/go-json@feature/fix-uint64-conversion )

[philband]

@goccy I can confirm it is fixed with commit d7372a4 github.com/goccy/go-json v0.7.10-0.20211014060028-d7372a47cd1a