Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BugFix] Fix the bug where the es catalog cannot view the es6 table #30608

Open
wants to merge 3 commits into
base: main
Choose a base branch
from

Conversation

yeyudefeng
Copy link

@yeyudefeng yeyudefeng commented Sep 8, 2023

Fixes #issue

What type of PR is this:

  • BugFix
  • Feature
  • Enhancement
  • Refactor
  • UT
  • Doc
  • Tool

Does this PR entail a change in behavior?

  • [] Yes, this PR will result in a change in behavior.
  • No, this PR will not result in a change in behavior.

If yes, please specify the type of change:

  • Interface/UI changes: syntax, type conversion, expression evaluation, display information
  • Parameter changes: default values, similar parameters but with different default values
  • Policy changes: use new policy to replace old one, functionality automatically enabled
  • Feature removed
  • Miscellaneous: upgrade & downgrade compatibility, etc.

Checklist:

  • I have added test cases for my bug fix or my new feature
  • This pr needs user documentation (for new or modified features or behaviors)
    • I have added documentation for my new feature or new function

Bugfix cherry-pick branch check:

  • I have checked the version labels which the pr will be auto-backported to the target branch
    • 3.1
    • 3.0
    • 2.5
    • 2.4

Please refer to the following website for detailed feedback

https://forum.mirrorship.cn/t/topic/8756

We have created a global dynamic template in the ES 6.4.3 cluster. Each time we add an idnex, there are two JSON keys in the mappings of each index, which can cause parsing exceptions and prevent the table from being read. Any extra keys are called_ Default_.
So in response to this situation_ Default_ I made a judgment.
The code submitted this time can address this situation.
It will not affect the original logic.

{
"test2222": {
"mappings": {
"_doc": {
"_all": {
"enabled": false
},
"dynamic_templates": [

            ],
            "properties": {
                "Name": {
                    "type": "keyword",
                    "ignore_above": 2048
                },
                "age": {
                    "type": "long"
                },
                "value2": {
                    "type": "long"
                }
            }
        },
        "_default_": {
            "_all": {
                "enabled": false
            },
            "dynamic_templates": [
                {
                    "message_full": {
                        "match": "message_full",
                        "mapping": {
                            "fields": {
                                "keyword": {
                                    "ignore_above": 2048,
                                    "type": "keyword"
                                }
                            },
                            "type": "text"
                        }
                    }
                },
                {
                    "message": {
                        "match": "message",
                        "mapping": {
                            "type": "text"
                        }
                    }
                }
            ]
        }
    }
}

}

end

There are other questions, such as the index mapping format in the ES8 system
I'm not sure how to handle it. Should I not read this table, or should I code it and read this table
{
"metrics-endpoint.metadata_current_default": {
"mappings": {
"dynamic": "false",
"_meta": {
"created_by": "transform",
"_transform": {
"transform": "endpoint.metadata_current-default-8.6.1",
"version": {
"created": "8.6.0"
},
"creation_date_in_millis": 1688377236210
}
},
"dynamic_templates": [
{
"strings_as_keyword": {
"match_mapping_type": "string",
"mapping": {
"ignore_above": 1024,
"type": "keyword"
}
}
}
],
"date_detection": false,
"properties": {
"@timestamp": {
"type": "date"
},
"Endpoint": {
"properties": {
"capabilities": {
"type": "keyword",
"doc_values": false,
"ignore_above": 128
},
"configuration": {
"properties": {
"isolation": {
"type": "boolean",
"null_value": false
}
}
},
"policy": {
"properties": {
"applied": {
"properties": {
"id": {
"type": "keyword",
"ignore_above": 1024
},
"name": {
"type": "keyword",
"ignore_above": 1024
},
"status": {
"type": "keyword",
"ignore_above": 1024
}
}
}
}
},
"state": {
"properties": {
"isolation": {
"type": "boolean",
"null_value": false
}
}
},
"status": {
"type": "keyword",
"ignore_above": 1024
}
}
},
"agent": {
"properties": {
"id": {
"type": "keyword",
"ignore_above": 1024
},
"name": {
"type": "keyword",
"ignore_above": 1024
},
"type": {
"type": "keyword",
"ignore_above": 1024
},
"version": {
"type": "keyword",
"ignore_above": 1024
}
}
},
"data_stream": {
"properties": {
"dataset": {
"type": "constant_keyword",
"value": "endpoint.metadata"
},
"namespace": {
"type": "keyword"
},
"type": {
"type": "constant_keyword",
"value": "metrics"
}
}
},
"ecs": {
"properties": {
"version": {
"type": "keyword",
"ignore_above": 1024
}
}
},
"elastic": {
"properties": {
"agent": {
"properties": {
"id": {
"type": "keyword",
"ignore_above": 1024
}
}
}
}
},
"event": {
"properties": {
"action": {
"type": "keyword",
"ignore_above": 1024
},
"category": {
"type": "keyword",
"ignore_above": 1024
},
"code": {
"type": "keyword",
"ignore_above": 1024
},
"created": {
"type": "date"
},
"dataset": {
"type": "keyword",
"ignore_above": 1024
},
"hash": {
"type": "keyword",
"ignore_above": 1024
},
"id": {
"type": "keyword",
"ignore_above": 1024
},
"ingested": {
"type": "date"
},
"kind": {
"type": "keyword",
"ignore_above": 1024
},
"module": {
"type": "keyword",
"ignore_above": 1024
},
"outcome": {
"type": "keyword",
"ignore_above": 1024
},
"provider": {
"type": "keyword",
"ignore_above": 1024
},
"sequence": {
"type": "long"
},
"severity": {
"type": "long"
},
"type": {
"type": "keyword",
"ignore_above": 1024
}
}
},
"host": {
"properties": {
"architecture": {
"type": "keyword",
"ignore_above": 1024
},
"domain": {
"type": "keyword",
"ignore_above": 1024
},
"hostname": {
"type": "keyword",
"ignore_above": 1024
},
"id": {
"type": "keyword",
"ignore_above": 1024
},
"ip": {
"type": "ip"
},
"mac": {
"type": "keyword",
"ignore_above": 1024
},
"name": {
"type": "keyword",
"ignore_above": 1024
},
"os": {
"properties": {
"Ext": {
"properties": {
"variant": {
"type": "keyword",
"ignore_above": 1024
}
}
},
"family": {
"type": "keyword",
"ignore_above": 1024
},
"full": {
"type": "keyword",
"ignore_above": 1024,
"fields": {
"caseless": {
"type": "keyword",
"ignore_above": 1024,
"normalizer": "lowercase"
},
"text": {
"type": "text",
"norms": false
}
}
},
"kernel": {
"type": "keyword",
"ignore_above": 1024
},
"name": {
"type": "keyword",
"ignore_above": 1024,
"fields": {
"caseless": {
"type": "keyword",
"ignore_above": 1024,
"normalizer": "lowercase"
},
"text": {
"type": "text",
"norms": false
}
}
},
"platform": {
"type": "keyword",
"ignore_above": 1024
},
"version": {
"type": "keyword",
"ignore_above": 1024
}
}
},
"type": {
"type": "keyword",
"ignore_above": 1024
},
"uptime": {
"type": "long"
}
}
},
"updated_at": {
"type": "alias",
"path": "event.ingested"
}
}
}
}
}

@CLAassistant
Copy link

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.


yeyudefeng seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
You have signed the CLA already but the status is still pending? Let us recheck it.

@yeyudefeng yeyudefeng removed their assignment Sep 11, 2023
@yeyudefeng yeyudefeng changed the title 修改代码,判断es6的index的mappings有多个key的情况 [BugFix] Fix the bug where the es catalog cannot view the es6 table Sep 11, 2023
@yeyudefeng yeyudefeng force-pushed the starrocks-fix-es6catalog branch from e29af86 to fff08e9 Compare September 11, 2023 06:03
yeyudefeng added 3 commits September 11, 2023 14:10
Signed-off-by: yeyudefeng <yu.qiu@clife.cn>
Signed-off-by: yeyudefeng <yu.qiu@clife.cn>
Signed-off-by: yeyudefeng <yu.qiu@clife.cn>
@yeyudefeng yeyudefeng force-pushed the starrocks-fix-es6catalog branch from fff08e9 to 1426bdc Compare September 11, 2023 06:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants