Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] Mapping does not present error when data is incorrect #1110

Open
MitchellGale opened this issue Nov 25, 2022 · 0 comments
Open

[BUG] Mapping does not present error when data is incorrect #1110

MitchellGale opened this issue Nov 25, 2022 · 0 comments
Labels
bug Something isn't working infrastructure Changes to infrastructure, testing, CI/CD, pipelines, etc.

Comments

@MitchellGale
Copy link
Contributor

What is the bug?
Invalid data format for mapping does not cause error in integration tests. It does cause an error when loading with curl manually.

Error when loading through curl.
 Bits-MacBook-Pro:resources mitchellg$ curl -H 'Content-Type: application/x-ndjson' -XPOST 'http://localhost:9200/text_keyword/_bulk?pretty' -u admin:admin --insecure --data-binary @text_keyword_index.json
{
  "took" : 231,
  "errors" : true,
  "items" : [
    {
      "index" : {
        "_index" : "text_keyword",
        "_id" : "1",
        "_version" : 1,
        "result" : "created",
        "_shards" : {
          "total" : 2,
          "successful" : 1,
          "failed" : 0
        },
        "_seq_no" : 0,
        "_primary_term" : 1,
        "status" : 201
      }
    },
    {
      "index" : {
        "_index" : "text_keyword",
        "_id" : "2",
        "status" : 400,
        "error" : {
          "type" : "mapper_parsing_exception",
          "reason" : "failed to parse",
          "caused_by" : {
            "type" : "json_parse_exception",
            "reason" : "Duplicate field 'typeTextFieldData'\n at [Source: (byte[])\"{\"typeKeyword\": \"key01\", \"typeText\": \"text01\", \"typeKeywordFieldNoFieldData\": \"keyword01\", \"typeTextFieldData\": \"keyFD01\", \"typeTextFieldData\":  \"textFD01OverTen\", \"textDataFieldNoFields\": \"textFDNF01\", \"int0\": 1}\"; line: 1, column: 143]"
          }
        }
      }
    },
    {
      "index" : {
        "_index" : "text_keyword",
        "_id" : "3",
        "_version" : 1,
        "result" : "created",
        "_shards" : {
          "total" : 2,
          "successful" : 1,
          "failed" : 0
        },
        "_seq_no" : 1,
        "_primary_term" : 1,
        "status" : 201
      }
    },
    {
      "index" : {
        "_index" : "text_keyword",
        "_id" : "4",
        "_version" : 1,
        "result" : "created",
        "_shards" : {
          "total" : 2,
          "successful" : 1,
          "failed" : 0
        },
        "_seq_no" : 2,
        "_primary_term" : 1,
        "status" : 201
      }
    },
    {
      "index" : {
        "_index" : "text_keyword",
        "_id" : "5",
        "_version" : 1,
        "result" : "created",
        "_shards" : {
          "total" : 2,
          "successful" : 1,
          "failed" : 0
        },
        "_seq_no" : 3,
        "_primary_term" : 1,
        "status" : 201
      }
    },
    {
      "index" : {
        "_index" : "text_keyword",
        "_id" : "6",
        "_version" : 1,
        "result" : "created",
        "_shards" : {
          "total" : 2,
          "successful" : 1,
          "failed" : 0
        },
        "_seq_no" : 4,
        "_primary_term" : 1,
        "status" : 201
      }
    },
    {
      "index" : {
        "_index" : "text_keyword",
        "_id" : "7",
        "_version" : 1,
        "result" : "created",
        "_shards" : {
          "total" : 2,
          "successful" : 1,
          "failed" : 0
        },
        "_seq_no" : 5,
        "_primary_term" : 1,
        "status" : 201
      }
    },
    {
      "index" : {
        "_index" : "text_keyword",
        "_id" : "8",
        "_version" : 1,
        "result" : "created",
        "_shards" : {
          "total" : 2,
          "successful" : 1,
          "failed" : 0
        },
        "_seq_no" : 6,
        "_primary_term" : 1,
        "status" : 201
      }
    }
  ]
}

How can one reproduce the bug?
Steps to reproduce the behavior:
Create mapping and load data that is incorrectly formatted.

Load
Data](https://github.com/opensearch-project/sql/blob/081882ce140666f577f1e6e7359fa2675a5269df/integ-test/src/test/resources/datatypes.json)
Mappings

in IT tests and observe there is no error from the invalid data..

What is the expected behavior?
IT tests should produce error instead of ignoring data.

What is your host/environment?

  • OS: MacOS
  • Version 12.6.1

Do you have any additional context?
This file, datatypes.json, was incorrectly formatted with brackets. Issue was not caught until PR 867.

Old data issue

Fix

@MitchellGale MitchellGale added bug Something isn't working untriaged labels Nov 25, 2022
@dai-chen dai-chen added infrastructure Changes to infrastructure, testing, CI/CD, pipelines, etc. and removed untriaged labels Dec 14, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working infrastructure Changes to infrastructure, testing, CI/CD, pipelines, etc.
Projects
None yet
Development

No branches or pull requests

2 participants