Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Re-add Index Mappings #173

Merged
merged 2 commits into from
Mar 17, 2020
Merged

Re-add Index Mappings #173

merged 2 commits into from
Mar 17, 2020

Conversation

mymasse
Copy link
Contributor

@mymasse mymasse commented Mar 1, 2020

Biggest impact of not having this is the dynamic_template for String
type.

Fixes cloudfoundry-community/logsearch-for-cloudfoundry#345

Biggest impact of not having this is the dynamic_template for String 
type.
@TheMoves
Copy link

TheMoves commented Mar 3, 2020

What OPs files are you testing with? For some reason i'm getting this error:

++ curl -s -H 'Content-Type: application/json' -w '%{http_code}' -o /var/vcap/data/elasticsearch_config/elasticsearch_config-response.txt -X PUT --data-binary @/var/vcap/jobs/elasticsearch_config/index-templates/index-mappings.json http://INTERNALIP:9200/_template/index-mappings
+ HTTP_CODE=400
+ grep --quiet errors:true /var/vcap/data/elasticsearch_config/elasticsearch_config-response.txt
+ '[' 400 -le 300 ']'
+ cat /var/vcap/data/elasticsearch_config/elasticsearch_config-response.txt
{"error":{"root_cause":[{"type":"mapper_parsing_exception","reason":"Root mapping definition has unsupported parameters:  [doc : {dynamic_templates=[{string_fields={mapping={index=true, omit_norms=true, type=keyword}, match_mapping_type=string, match=*}}], _all={enabled=false}, properties={geoip={dynamic=true, type=object, properties={location={type=geo_point}}}, @raw={norms={enabled=false}, index=true, type=text}, @version={index=true, type=keyword}}}]"}],"type":"mapper_parsing_exception","reason":"Failed to parse mapping [_doc]: Root mapping definition has unsupported parameters:  [doc : {dynamic_templates=[{string_fields={mapping={index=true, omit_norms=true, type=keyword}, match_mapping_type=string, match=*}}], _all={enabled=false}, properties={geoip={dynamic=true, type=object, properties={location={type=geo_point}}}, @raw={norms={enabled=false}, index=true, type=text}, @version={index=true, type=keyword}}}]","caused_by":{"type":"mapper_parsing_exception","reason":"Root mapping definition has unsupported parameters:  [doc : {dynamic_templates=[{string_fields={mapping={index=true, omit_norms=true, type=keyword}, match_mapping_type=string, match=*}}], _all={enabled=false}, properties={geoip={dynamic=true, type=object, properties={location={type=geo_point}}}, @raw={norms={enabled=false}, index=true, type=text}, @version={index=true, type=keyword}}}]"}},"status":400}+ exit 400

piped the above file to jq, cat /var/vcap/data/elasticsearch_config/elasticsearch_config-response.txt | jq ..

{
  "error": {
    "root_cause": [
      {
        "type": "mapper_parsing_exception",
        "reason": "Root mapping definition has unsupported parameters:  [doc : {dynamic_templates=[{string_fields={mapping={index=true, omit_norms=true, type=keyword}, match_mapping_type=string, match=*}}], _all={enabled=false}, properties={geoip={dynamic=true, type=object, properties={location={type=geo_point}}}, @raw={norms={enabled=false}, index=true, type=text}, @version={index=true, type=keyword}}}]"
      }
    ],
    "type": "mapper_parsing_exception",
    "reason": "Failed to parse mapping [_doc]: Root mapping definition has unsupported parameters:  [doc : {dynamic_templates=[{string_fields={mapping={index=true, omit_norms=true, type=keyword}, match_mapping_type=string, match=*}}], _all={enabled=false}, properties={geoip={dynamic=true, type=object, properties={location={type=geo_point}}}, @raw={norms={enabled=false}, index=true, type=text}, @version={index=true, type=keyword}}}]",
    "caused_by": {
      "type": "mapper_parsing_exception",
      "reason": "Root mapping definition has unsupported parameters:  [doc : {dynamic_templates=[{string_fields={mapping={index=true, omit_norms=true, type=keyword}, match_mapping_type=string, match=*}}], _all={enabled=false}, properties={geoip={dynamic=true, type=object, properties={location={type=geo_point}}}, @raw={norms={enabled=false}, index=true, type=text}, @version={index=true, type=keyword}}}]"
    }
  },
  "status": 400
}

I pulled down the latest of this and then cherry-picked your commit/PR to test. What you are doing makes sense, but I don't have enough experience with ELK to troubleshoot or make sense of the error

@mymasse
Copy link
Contributor Author

mymasse commented Mar 3, 2020

All I did was put back what was there in the previous release. I tested part of it by adding the index mappings directly in Kibana. But these files used to work in ELK 6.8

@mymasse
Copy link
Contributor Author

mymasse commented Mar 3, 2020

I'm away until Thursday I'll try to debug it then.

@TheMoves
Copy link

TheMoves commented Mar 4, 2020

I believe the template needs to look like this...i've tested it my lab and it works. Kinda annoying that KIBANA freaks out spewing errors until it starts to index data. Please note I have almost 0 experience with ELK and 0 experience outside of the below at working with indexes/mappings/etc... Just spend 3 hours this morning reading the ELK doc for 7.6 and reading blog posts...

The stuff I removed, per my understanding, is deprecated in 7.6.

My understanding is the

{
  "template": "<%= p('elasticsearch_config.index_prefix') %>*",
  "order": 102,
  "settings": {
    "index": {
      "query": {
        "default_field": "@raw"
      }
    }
  },
  "mappings": {
    "dynamic_templates": [
      {
        "string_fields": {
          "match": "*",
          "match_mapping_type": "string",
          "mapping": {
            "type": "keyword",
            "index": true
          }
        }
      }
    ],
    "properties": {
      "@version": {
        "type": "keyword",
        "index": true
      },
      "@raw": {
        "type": "text",
        "index": true
      },
      "geoip": {
        "type": "object",
        "dynamic": true,
        "properties": {
          "location": {
            "type": "geo_point"
          }
        }
      }
    }
  }
}

@TheMoves
Copy link

TheMoves commented Mar 5, 2020

Also need to update the logsearch-for-cloudfoundry templates for ELK 7.6 i think

@mymasse
Copy link
Contributor Author

mymasse commented Mar 6, 2020

I made the fixes to the mappings, kept the norms fields but adapted to the correct syntax. Also made a fix in the logsearch-for-cloudfoundry mappings. Can somebody give it a test I'm not setup at all to test this (currently at home for the next few days)

Copy link

@TheMoves TheMoves left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 from me for these particular changes. You addition to jobs/elasticsearch_config/templates/index-templates/index-mappings.json.erb matches what I have and pasted before.

I still need to review logsearch-for-cloudfoundry

@axelaris
Copy link
Member

LGTM

@axelaris axelaris merged commit 54c5ed0 into cloudfoundry-community:develop Mar 17, 2020
@axelaris
Copy link
Member

Thank you @mymasse !

@akhettal
Copy link

akhettal commented May 18, 2020

I'm trying to upgrade logsearch from v210.5.0 to v210.5.2 and using elasticsearch_config.templates with those custom template logservice template

Here is my elasticsearch_config maintenace job manifest :

properties:
      elasticsearch_config:
        index_prefix: logs-
        templates:
        - shards-and-replicas: /var/vcap/jobs/elasticsearch_config/index-templates/shards-and-replicas.json
        - index-settings: /var/vcap/jobs/elasticsearch_config/index-templates/index-settings.json
        - index-mappings: /var/vcap/jobs/elasticsearch_config/index-templates/index-mappings.json
        - index-mappings-logservice: /var/vcap/jobs/logservice-logsearch-filters/index-mappings.json
        - index-mappings-app-logservice: /var/vcap/jobs/logservice-logsearch-filters/index-mappings-app.json
        - index-mappings-platform-logservice: /var/vcap/jobs/logservice-logsearch-filters/index-mappings-platform.json

The upgrade seems no te be working and the bosh logs are :

Error: 'maintenance/768ccaf8-0f81-466e-9164-bc486f9e64c3 (0)' is not running after update. Review logs for failed jobs: elasticsearch_config
Task 524125 | 12:32:39 | Error: 'maintenance/768ccaf8-0f81-466e-9164-bc486f9e64c3 (0)' is not running after update. Review logs for failed jobs: elasticsearch_config

The errors jobs logs are :

+ URL=http://172.17.8.37:9200/_template/shards-and-replicas
+ CONTENT_PATH=/var/vcap/jobs/elasticsearch_config/index-templates/shards-and-replicas.json
++ curl -s -H 'Content-Type: application/json' -w '%{http_code}' -o /var/vcap/data/elasticsearch_config/elasticsearch_config-response.txt -X PUT --data-binary @/var/vcap/jobs/elasticsearch_config/index-templates/shards-and-replicas.json http://172.17.8.37:9200/_template/shards-and-replicas
+ HTTP_CODE=200
+ grep --quiet errors:true /var/vcap/data/elasticsearch_config/elasticsearch_config-response.txt
+ '[' 200 -le 300 ']'
+ cat /var/vcap/data/elasticsearch_config/elasticsearch_config-response.txt
+ URL=http://172.17.8.37:9200/_template/index-settings
+ CONTENT_PATH=/var/vcap/jobs/elasticsearch_config/index-templates/index-settings.json
++ curl -s -H 'Content-Type: application/json' -w '%{http_code}' -o /var/vcap/data/elasticsearch_config/elasticsearch_config-response.txt -X PUT --data-binary @/var/vcap/jobs/elasticsearch_config/index-templates/index-settings.json http://172.17.8.37:9200/_template/index-settings
+ HTTP_CODE=200
+ grep --quiet errors:true /var/vcap/data/elasticsearch_config/elasticsearch_config-response.txt
+ '[' 200 -le 300 ']'
+ cat /var/vcap/data/elasticsearch_config/elasticsearch_config-response.txt
+ URL=http://172.17.8.37:9200/_template/index-mappings
+ CONTENT_PATH=/var/vcap/jobs/elasticsearch_config/index-templates/index-mappings.json
++ curl -s -H 'Content-Type: application/json' -w '%{http_code}' -o /var/vcap/data/elasticsearch_config/elasticsearch_config-response.txt -X PUT --data-binary @/var/vcap/jobs/elasticsearch_config/index-templates/index-mappings.json http://172.17.8.37:9200/_template/index-mappings
+ HTTP_CODE=200
+ grep --quiet errors:true /var/vcap/data/elasticsearch_config/elasticsearch_config-response.txt
+ '[' 200 -le 300 ']'
+ cat /var/vcap/data/elasticsearch_config/elasticsearch_config-response.txt
+ URL=http://172.17.8.37:9200/_template/index-mappings-logservice
+ CONTENT_PATH=/var/vcap/jobs/logservice-logsearch-filters/index-mappings.json
++ curl -s -H 'Content-Type: application/json' -w '%{http_code}' -o /var/vcap/data/elasticsearch_config/elasticsearch_config-response.txt -X PUT --data-binary @/var/vcap/jobs/logservice-logsearch-filters/index-mappings.json http://172.17.8.37:9200/_template/index-mappings-logservice
+ HTTP_CODE=400
+ grep --quiet errors:true /var/vcap/data/elasticsearch_config/elasticsearch_config-response.txt
+ '[' 400 -le 300 ']'
+ cat /var/vcap/data/elasticsearch_config/elasticsearch_config-response.txt
tail: elasticsearch_config.stderr.log: file truncated
{"error":{"root_cause":[{"type":"parse_exception","reason":"request body is required"}],"type":"parse_exception","reason":"request body is required"},"status":400}+ exit 400

It's not taking the custom templates.

Note that when i'm exuciting the script manually it seems to be good :

/var/vcap/jobs/elasticsearch_config/bin
+ export DATA_DIR=/var/vcap/data/elasticsearch_config
+ DATA_DIR=/var/vcap/data/elasticsearch_config
+ URL=http://172.17.8.37:9200
++ curl -s http://172.17.8.37:9200/_cat/templates
++ awk '{print $1}'
+ for template in '$(curl -s ${URL}/_cat/templates | awk '\''{print $1}'\'')'
+ grep -qw security-index-template /dev/fd/63
++ echo 'shards-and-replicas index-settings index-mappings index-mappings-logservice index-mappings-app-logservice index-mappings-platform-logservice'
+ curl -XDELETE http://172.17.8.37:9200/_template/security-index-template
{"acknowledged":true}+ for template in '$(curl -s ${URL}/_cat/templates | awk '\''{print $1}'\'')'
+ grep -qw .ml-config /dev/fd/63
++ echo 'shards-and-replicas index-settings index-mappings index-mappings-logservice index-mappings-app-logservice index-mappings-platform-logservice'
+ curl -XDELETE http://172.17.8.37:9200/_template/.ml-config
{"acknowledged":true}+ for template in '$(curl -s ${URL}/_cat/templates | awk '\''{print $1}'\'')'
+ grep -qw index-mappings-logservice /dev/fd/63
++ echo 'shards-and-replicas index-settings index-mappings index-mappings-logservice index-mappings-app-logservice index-mappings-platform-logservice'
+ for template in '$(curl -s ${URL}/_cat/templates | awk '\''{print $1}'\'')'
+ grep -qw shards-and-replicas /dev/fd/63
++ echo 'shards-and-replicas index-settings index-mappings index-mappings-logservice index-mappings-app-logservice index-mappings-platform-logservice'
+ for template in '$(curl -s ${URL}/_cat/templates | awk '\''{print $1}'\'')'
+ grep -qw index-mappings-app-logservice /dev/fd/63
++ echo 'shards-and-replicas index-settings index-mappings index-mappings-logservice index-mappings-app-logservice index-mappings-platform-logservice'
+ for template in '$(curl -s ${URL}/_cat/templates | awk '\''{print $1}'\'')'
+ grep -qw index-settings /dev/fd/63
++ echo 'shards-and-replicas index-settings index-mappings index-mappings-logservice index-mappings-app-logservice index-mappings-platform-logservice'
+ for template in '$(curl -s ${URL}/_cat/templates | awk '\''{print $1}'\'')'
+ grep -qw index-mappings /dev/fd/63
++ echo 'shards-and-replicas index-settings index-mappings index-mappings-logservice index-mappings-app-logservice index-mappings-platform-logservice'
+ for template in '$(curl -s ${URL}/_cat/templates | awk '\''{print $1}'\'')'
+ grep -qw .ml-anomalies- /dev/fd/63
++ echo 'shards-and-replicas index-settings index-mappings index-mappings-logservice index-mappings-app-logservice index-mappings-platform-logservice'
+ curl -XDELETE http://172.17.8.37:9200/_template/.ml-anomalies-
{"acknowledged":true}+ for template in '$(curl -s ${URL}/_cat/templates | awk '\''{print $1}'\'')'
+ grep -qw security_audit_log /dev/fd/63
++ echo 'shards-and-replicas index-settings index-mappings index-mappings-logservice index-mappings-app-logservice index-mappings-platform-logservice'
+ curl -XDELETE http://172.17.8.37:9200/_template/security_audit_log
{"acknowledged":true}+ for template in '$(curl -s ${URL}/_cat/templates | awk '\''{print $1}'\'')'
+ grep -qw .ml-notifications /dev/fd/63
++ echo 'shards-and-replicas index-settings index-mappings index-mappings-logservice index-mappings-app-logservice index-mappings-platform-logservice'
+ curl -XDELETE http://172.17.8.37:9200/_template/.ml-notifications
{"acknowledged":true}+ for template in '$(curl -s ${URL}/_cat/templates | awk '\''{print $1}'\'')'
+ grep -qw index-mappings-platform-logservice /dev/fd/63
++ echo 'shards-and-replicas index-settings index-mappings index-mappings-logservice index-mappings-app-logservice index-mappings-platform-logservice'
+ for template in '$(curl -s ${URL}/_cat/templates | awk '\''{print $1}'\'')'
+ grep -qw .ml-meta /dev/fd/63
++ echo 'shards-and-replicas index-settings index-mappings index-mappings-logservice index-mappings-app-logservice index-mappings-platform-logservice'
+ curl -XDELETE http://172.17.8.37:9200/_template/.ml-meta
{"acknowledged":true}+ for template in '$(curl -s ${URL}/_cat/templates | awk '\''{print $1}'\'')'
+ grep -qw logstash-index-template /dev/fd/63
++ echo 'shards-and-replicas index-settings index-mappings index-mappings-logservice index-mappings-app-logservice index-mappings-platform-logservice'
+ curl -XDELETE http://172.17.8.37:9200/_template/logstash-index-template
{"acknowledged":true}+ for template in '$(curl -s ${URL}/_cat/templates | awk '\''{print $1}'\'')'
+ grep -qw .ml-state /dev/fd/63
++ echo 'shards-and-replicas index-settings index-mappings index-mappings-logservice index-mappings-app-logservice index-mappings-platform-logservice'
+ curl -XDELETE http://172.17.8.37:9200/_template/.ml-state
{"acknowledged":true}+ curl -f 'http://172.17.8.37:9200/_cluster/health?wait_for_status=yellow&timeout=10m'
{"cluster_name":"logsearch","status":"green","timed_out":false,"number_of_nodes":7,"number_of_data_nodes":3,"active_primary_shards":854,"active_shards":1708,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0,"delayed_unassigned_shards":0,"number_of_pending_tasks":3,"number_of_in_flight_fetch":0,"task_max_waiting_in_queue_millis":15,"active_shards_percent_as_number":100.0}+ URL=http://172.17.8.37:9200/_template/shards-and-replicas
+ CONTENT_PATH=/var/vcap/jobs/elasticsearch_config/index-templates/shards-and-replicas.json
++ curl -s -H 'Content-Type: application/json' -w '%{http_code}' -o /var/vcap/data/elasticsearch_config/elasticsearch_config-response.txt -X PUT --data-binary @/var/vcap/jobs/elasticsearch_config/index-templates/shards-and-replicas.json http://172.17.8.37:9200/_template/shards-and-replicas
+ HTTP_CODE=200
+ grep --quiet errors:true /var/vcap/data/elasticsearch_config/elasticsearch_config-response.txt
+ '[' 200 -le 300 ']'
+ cat /var/vcap/data/elasticsearch_config/elasticsearch_config-response.txt
{"acknowledged":true}+ URL=http://172.17.8.37:9200/_template/index-settings
+ CONTENT_PATH=/var/vcap/jobs/elasticsearch_config/index-templates/index-settings.json
++ curl -s -H 'Content-Type: application/json' -w '%{http_code}' -o /var/vcap/data/elasticsearch_config/elasticsearch_config-response.txt -X PUT --data-binary @/var/vcap/jobs/elasticsearch_config/index-templates/index-settings.json http://172.17.8.37:9200/_template/index-settings
+ HTTP_CODE=200
+ grep --quiet errors:true /var/vcap/data/elasticsearch_config/elasticsearch_config-response.txt
+ '[' 200 -le 300 ']'
+ cat /var/vcap/data/elasticsearch_config/elasticsearch_config-response.txt
{"acknowledged":true}+ URL=http://172.17.8.37:9200/_template/index-mappings
+ CONTENT_PATH=/var/vcap/jobs/elasticsearch_config/index-templates/index-mappings.json
++ curl -s -H 'Content-Type: application/json' -w '%{http_code}' -o /var/vcap/data/elasticsearch_config/elasticsearch_config-response.txt -X PUT --data-binary @/var/vcap/jobs/elasticsearch_config/index-templates/index-mappings.json http://172.17.8.37:9200/_template/index-mappings
+ HTTP_CODE=200
+ grep --quiet errors:true /var/vcap/data/elasticsearch_config/elasticsearch_config-response.txt
+ '[' 200 -le 300 ']'
+ cat /var/vcap/data/elasticsearch_config/elasticsearch_config-response.txt
{"acknowledged":true}+ URL=http://172.17.8.37:9200/_template/index-mappings-logservice
+ CONTENT_PATH=/var/vcap/jobs/logservice-logsearch-filters/index-mappings.json
++ curl -s -H 'Content-Type: application/json' -w '%{http_code}' -o /var/vcap/data/elasticsearch_config/elasticsearch_config-response.txt -X PUT --data-binary @/var/vcap/jobs/logservice-logsearch-filters/index-mappings.json http://172.17.8.37:9200/_template/index-mappings-logservice
+ HTTP_CODE=200
+ grep --quiet errors:true /var/vcap/data/elasticsearch_config/elasticsearch_config-response.txt
+ '[' 200 -le 300 ']'
+ cat /var/vcap/data/elasticsearch_config/elasticsearch_config-response.txt
{"acknowledged":true}+ URL=http://172.17.8.37:9200/_template/index-mappings-app-logservice
+ CONTENT_PATH=/var/vcap/jobs/logservice-logsearch-filters/index-mappings-app.json
++ curl -s -H 'Content-Type: application/json' -w '%{http_code}' -o /var/vcap/data/elasticsearch_config/elasticsearch_config-response.txt -X PUT --data-binary @/var/vcap/jobs/logservice-logsearch-filters/index-mappings-app.json http://172.17.8.37:9200/_template/index-mappings-app-logservice
+ HTTP_CODE=200
+ grep --quiet errors:true /var/vcap/data/elasticsearch_config/elasticsearch_config-response.txt
+ '[' 200 -le 300 ']'
+ cat /var/vcap/data/elasticsearch_config/elasticsearch_config-response.txt
{"acknowledged":true}+ URL=http://172.17.8.37:9200/_template/index-mappings-platform-logservice
+ CONTENT_PATH=/var/vcap/jobs/logservice-logsearch-filters/index-mappings-platform.json
++ curl -s -H 'Content-Type: application/json' -w '%{http_code}' -o /var/vcap/data/elasticsearch_config/elasticsearch_config-response.txt -X PUT --data-binary @/var/vcap/jobs/logservice-logsearch-filters/index-mappings-platform.json http://172.17.8.37:9200/_template/index-mappings-platform-logservice
+ HTTP_CODE=200
+ grep --quiet errors:true /var/vcap/data/elasticsearch_config/elasticsearch_config-response.txt
+ '[' 200 -le 300 ']'
+ cat /var/vcap/data/elasticsearch_config/elasticsearch_config-response.txt
{"acknowledged":true}


Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Dashboards and Visualizations don't work in a fresh install of 211.0.1
4 participants