Skip to content

decodeChar CharsetToUTF8 return error data for SQL_Latin1_General_CP1_CI_AS chinese code #213

Open

Description

Describe the bug
insert into some chinese code return data error

To Reproduce

SELECT
SERVERPROPERTY('ProductVersion ') AS ProductVersion
,SERVERPROPERTY('ProductLevel') AS ProductLevel
,SERVERPROPERTY('ResourceVersion') AS ResourceVersion
,SERVERPROPERTY('ResourceLastUpdateDateTime') AS ResourceLastUpdateDateTime
,SERVERPROPERTY('Collation') AS Collation
,@@version as Version
,@@LANGUAGE AS Language
,collation_name from sys.databases where name = db_name()
ProductVersion|ProductLevel|ResourceVersion|ResourceLastUpdateDateTime|Collation                   |Version                                                                                                                                                                                              |Language  |collation_name              |
--------------+------------+---------------+--------------------------+----------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------+----------------------------+
16.0.4075.1   |RTM         |16.00.4075     |2023-08-23 14:45:27.7     |SQL_Latin1_General_CP1_CI_AS|Microsoft SQL Server 2022 (RTM-CU8) (KB5029666) - 16.0.4075.1 (X64) ¶ Aug 23 2023 14:04:50 ¶ Copyright (C) 2022 Microsoft Corporation¶ Developer Edition (64-bit) on Linux (Ubuntu 20.04.6 LTS) <X64>|us_english|SQL_Latin1_General_CP1_CI_AS|
			s := "福建九州通譚嵄医疗器械有限公司泉州分公司"
			gbkEnc, err := ianaindex.MIB.Encoding("GBK")
			if err != nil {
				return
			}
			tmp, err := ioutil.ReadAll(
				transform.NewReader(bytes.NewReader([]byte(s)), gbkEnc.NewEncoder()),
			)
			fmt.Println("GBK  ",hex.EncodeToString(tmp))
			iso88591, err := ianaindex.MIB.Encoding("ISO-8859-1")
			if err != nil {
				return
			}
			tmp, err = ioutil.ReadAll(
				transform.NewReader(bytes.NewReader(tmp), iso88591.NewDecoder()),
			)
			fmt.Println("ISO-8859-1  ",hex.EncodeToString(tmp))

			_,err = m.DB.Exec("drop table table dbo.yp_supplyer")
			_,err = m.DB.Exec("create table dbo.yp_supplyer(supplyer_code bigint,supplyer_name char(100))")
			_,err = m.DB.Exec("insert into dbo.yp_supplyer(supplyer_code,supplyer_name ) values(7,@p1)",string(tmp))
			
			rows, err := m.DB.Query(
				"SELECT supplyer_code,supplyer_name from dbo.yp_supplyer",
			)
			if err != nil {
				t.Error(err)
				return
			}
			defer rows.Close()
			for rows.Next() {
				var (
					ColTime string
					supplyer_name []byte
				)
				if err = rows.Scan(
					&ColTime, &supplyer_name,
				); err != nil {
					t.Error(err)
					return
				}
				fmt.Println("R1 ISO-8859-1  ",hex.EncodeToString(supplyer_name))
				tmp, err = ioutil.ReadAll(
					transform.NewReader(bytes.NewReader(supplyer_name), iso88591.NewEncoder()),
				)
				fmt.Println("R2 ISO-8859-1  ",hex.EncodeToString(tmp))
			}


GBK   b8a3bda8bec5d6ddcda8d7548db1d2bdc1c6c6f7d0b5d3d0cfdeb9abcbbec8aad6ddb7d6b9abcbbe
ISO-8859-1   c2b8c2a3c2bdc2a8c2bec385c396c39dc38dc2a8c39754c28dc2b1c392c2bdc381c386c386c3b7c390c2b5c393c390c38fc39ec2b9c2abc38bc2bec388c2aac396c39dc2b7c396c2b9c2abc38bc2be
a1 372020202020
a1 b8a3bda8bec5d6ddcda8d7548db1d2bdc1c6c6f7d0b5d3d0cfdeb9abcbbec8aad6ddb7d6b9abcbbe20202020202020202020 --> database return client gbk code
R1 ISO-8859-1   c2b8c2a3c2bdc2a8c2bec385c396c39dc38dc2a8c39754efbfbdc2b1c392c2bdc381c386c386c3b7c390c2b5c393c390c38fc39ec2b9c2abc38bc2bec388c2aac396c39dc2b7c396c2b9c2abc38bc2be20202020202020202020 -- CharsetToUTF8  return 
R2 ISO-8859-1   b8a3bda8bec5d6ddcda8d754

Expected behavior
A clear and concise description of what you expected to happen.

Further technical details

SQL Server version: (e.g. SQL Server 2017)
Operating system: (e.g. Windows 2019, Ubuntu 18.04, macOS 10.13, Docker container)
Table schema

Additional context
Add any other context about the problem here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions