Observed performance degradation in 7.16.3 when compared to 5.6.9

[chaitu0292]

Elasticsearch Version

7.16.3

Installed Plugins

No response

Java Version

bundled

OS Version

centos 7.8

Problem Description

Observed perf degradation in 7.16.3 when compared to 5.6.9

Run	ES_version	AVG(ms)	          Max(ms)
1	7.16.3	         294	          913
2	7.16.3	         434	          1038
1	5.6.9	         83	          619
2	5.6.9	         107	          617

Deployed in Kubernetes single pod.
Resource: 4cpu,16Gb,8GB Heap.
ES_JAVA_OPTS: "-Xmx8124m -Xms8124m"
Index config: 1 shard 0 replica

Settings:
FieldsLimit=200000
RefreshInterval=1s

Steps to Reproduce

Steps to reproduce:
Url: http://localhost:9200/myindex/_doc/?refresh=true
Data:

{
    "PersonFirstName": "krish",
    "CurrentclamAge": 0,
    "PlanID": "PLAN",
    "IPAmount": 0,
    "DateReXZived": "2021-08-3",
    "IPOfDays": 0,
    "XZ_Orchestra": "Adjut",
    "ZTApplication": "clams",
    "clamType": "I",
    "Name": "",
    "TimelyFilingDays": 365,
    "FinalTransisionType": "",
    "SubscriberFirstName": "",
    "ZTCreateDateTime": "2022-08-30T14:25:39.346Z",
    "ZTUrgencyWork": 10,
    "_isexternal": false,
    "ZTInsName": "TST-${c_Counter}",
    "ZTSaveDateTime": "2022-08-30T14:25:39.396Z",
    "XZ_RefPayerID": "SK201507233",
    "ServiXZEndDate": "2018-02-02T00:00:00Z",
    "pzInsKey": "ZonalCommisioner-CLM-WORK-clamS-TEST TST-${c_Counter}",
    "SKOwnerOrgUnit": "",
    "PayerclamControl": "",
    "SKElapsedStatusPending": 0,
    "SKOwnerOrg": "",
    "XZ_RefContractID": "GC2015091522",
    "SKOrigUserID": "",
    "ProXZssImmediately": false,
    "SKElapsedStatusOpen": 0,
    "OverAllSLADeadLine": "2022-09-14T14:25:39Z",
    "PersonID": "884627204",
    "XZ_RefPolicyID": "17135253",
    "BillingProvider": "22-19268082",
    "PersonLastName": "Esterman",
    "Subscriber": "S1713525301",
    "XZ_TimelyFilingFunction": "System",
    "SubscriberLastName": "Stadtmiller",
    "SKElapsedStatusNew": 1661869539,
    "XZ_RefPersonID": "884627204",
    "_instanXZname": "TST-${c_Counter}",
    "SKID": "TST-${c_Counter}",
    "SKClassGroup": "ZonalCommisioner-CLM-Work-clams-Test",
    "SKOwnerDivision": "",
    "XZ_RefSubscriberID": "1713525301",
    "ServiXZStartDate": "2018-02-02T00:00:00Z",
    "BillingProviderLastEntityName": "KC",
    "OverAllSLAGoal": "2022-09-11T14:25:39Z",
    "Person": "S884627204",
    "SKStatusWork": "New",
    "xmlBuffer": "true",
    "temp": "376",
    "TypeOfServiXZ": "",
    "WorkBasketOrgName": "@",
    "XZ_clamFrequencyCode": "1",
    "ZTObjClass": "ZonalCommisioner-CLM-Work-clams-Test",
    "ZTCreateOperator": "",
    "TestclamsID": "",
    "ZTCreateSystemID": "sXZ88all",
    "SKLabel": "",
    "ZTCreateOpName": "",
    "clam": {
        "ZTUpdateSystemID": "sXZ88all",
        "BillingNoteText": "",
        "TotalclamAllowedAmount": 40,
        "TransisionSegmentCount": "",
        "ZTUpdateDateTime": "2022-08-30T14:25:39.368Z",
        "RelatedCausesCode": [],
        "LatePaymentInterestPerXZnt": 10,
        "PersonCalculatedAmount": "0.00",
        "RepriXZdApprovedAPGAmount": 0,
        "SurchargeAmount": 0,
        "TotalclamCopayAmount": 0,
        "TotalclamChargeAmount": 209.77,
        "AnesthesiaRelatedSurgicalProXZdure": "",
        "ContractCode": "GC2015091522",
        "DateTimePeriodFormatQualifier": "",
        "TotalPayerCOB": 0,
        "IndividualDeathDate": "",
        "StatementToDate": "2018-02-02",
        "CLIA": "",
        "clamSubmissionType": "",
        "RepriXZdApprovedRevenueCode": "",
        "PeerReviewAuthorization": "",
        "CodeCategory": "",
        "RepriXZdclamReferenXZ": "",
        "clamID": "TST-${c_Counter}",
        "TransisionSetPurposeCode": "",
        "ICDType": "ICD 10",
        "StretcherPurposeDescription": "",
        "PayToProvider": {
            "_isexternal": false,
            "SKFullName": "",
            "IdentificationCode": "",
            "EntityIdentifierCode": "",
            "NationalProviderIdentifier": "",
            "SKWorkPartyUri": "",
            "_instanXZname": "",
            "MiddleName": "",
            "FederalTaxID": "",
            "ZTObjClass": "ZonalCommisioner-Provider",
            "SKNameSuffix": "",
            "SKFirstName": "",
            "IdentificationCodeQualifier": "",
            "myCode": "",
            "EntityTypeQualifier": "",
            "SKLastName": ""
        },
        "SpecialProgramIndicator": "",
        "PayerName": "U+Health",
        "ProviderAcXZptAssignmentCode": "A",
        "XZrtificationConditionIndicator": "",
        "RoundTripPurposeDescription": "",
        "clamMediaType": "Paper",
        "PersonLastName": "Esterman",
        "GraXZPeriod": 30,
        "TransisionSetCreationTime": "",
        "PersonStatusCode": "01",
        "TotalPersonCOB": 0,
        "DelayReasonCode": "",
        "FacilityTypeCode": "",
        "DischargeHour": "12:00 AM",
        "TotalclamDeductibleAmount": 0,
        "AutoAccidentStateCode": "",
        "clamLine": [
            {
                "AmbulatoryPersonGroup": "",
                "NonCoveredChargeAmount": 0,
                "DMERentalPriXZ": 0,
                "PersonCalculatedAmount": "0.00",
                "SurchargeAmount": 0,
                "ChargeAmount": 209.77,
                "OrderingProvider": {
                    "_isexternal": false,
                    "ZTObjClass": "ZonalCommisioner-Provider",
                    "_instanXZname": ""
                },
                "ToothInformation": [],
                "LineItemControl": "",
                "DateTimePeriodFormatQualifier": "",
                "ContractCode": "",
                "ServicingState": "NC",
                "Network": "",
                "NoteReferenXZCode": "",
                "CoPayStatusCode": "",
                "CLIA": "",
                "DRGCode": "",
                "AuthTypeRequired": "InPerson",
                "XZrtificationTypeCodeValues": "",
                "Status": "New",
                "PurchasedServiXZProvider": {
                    "_isexternal": false,
                    "ZTObjClass": "ZonalCommisioner-Provider",
                    "_instanXZname": ""
                },
                "ProXZdureModifiers": [],
                "UnitsOfServiXZPaidCount": 1,
                "ReferenXZIdentificationQualifier": "",
                "ZTInsName": "TST-${c_Counter}_1",
                "DMEPurchasePriXZ": 0,
                "StretcherPurposeDescription": "",
                "pzInsKey": "TST-${c_Counter}_1",
                "CompositeUnitOfMeasurementUnit": "",
                "Prescription": "",
                "LineRepricingProductOrServiXZIDQualifier": "",
                "IsSANPPricingSuppressed": true,
                "XZrtificationConditionIndicator": "",
                "RoundTripPurposeDescription": "",
                "BenefitDefinitionList": [
                    {
                        "_isexternal": false,
                        "ZTObjClass": "ZonalCommisioner-Data-Benefit-Definition",
                        "_instanXZname": ""
                    }
                ],
                "PersonID": "884627204",
                "CapitationFlag": "",
                "ZTListSubscript": "1",
                "PurchasedServiXZProviderIdentifier": "",
                "Contract": [
                    {
                        "_isexternal": false,
                        "ZTObjClass": "ZonalCommisioner-Data-ProviderContract",
                        "CoveredNetworks": [
                            {
                                "_isexternal": false,
                                "ZTObjClass": "ZonalCommisioner-Data-Network",
                                "_instanXZname": "",
                                "NetworkID": "PN201509146"
                            }
                        ],
                        "ContractID": "PC-20200302T155757.320 GMT",
                        "_instanXZname": ""
                    }
                ],
                "ProXZssedAuthReferenXZ": [],
                "LineNoteText": "",
                "ProXZdureCode": "",
                "PriAuthReferenXZ": [],
                "RepricingOrganizationIdentifier": "",
                "OtherProvider": {
                    "_isexternal": false,
                    "ZTObjClass": "ZonalCommisioner-Provider",
                    "_instanXZname": ""
                },
                "NDC": {
                    "_isexternal": false,
                    "ZTObjClass": "Health-Codes-Data-NDC-Package",
                    "NormalizedNDCPackageCode": "",
                    "_instanXZname": ""
                },
                "HospiXZEmployedProviderIndicator": false,
                "Line": "1",
                "RateSheetID": "",
                "AdjRepriXZdLineRef": "",
                "ZonalCommisionerPCSCode": "",
                "PurchasedServiXZChargeAmount": 0,
                "UnitorBasisforMeasurementCode": "UN",
                "OutOfStateIndicator": true,
                "CompositeMedicalProductOrServiXZIDQualifier": "",
                "RentalUnitPriXZIndicator": "",
                "IsAuthorizationRequired": "",
                "MatchedAuthReferenXZ": [],
                "ServiXZToDate": "2018-02-02",
                "ReferringProviderList": [
                    {
                        "_isexternal": false,
                        "ZTObjClass": "ZonalCommisioner-Provider",
                        "_instanXZname": ""
                    },
                    {
                        "_isexternal": false,
                        "ZTObjClass": "ZonalCommisioner-Provider",
                        "_instanXZname": ""
                    }
                ],
                "ZTObjClass": "ZonalCommisioner-Data-clamLine",
                "RejectReasonCode": "",
                "PricingMethodology": "",
                "DMEAttachmentTransmissionCode": "",
                "SKNote": "false",
                "ServiXZIdentificationCode": "",
                "SurchargeFactor": 0,
                "EPSDTIndicator": "",
                "MammographyXZrtification": "",
                "ServiXZLineRevenueCode": "0420",
                "ImmunizationBatch": "",
                "AssistantSurgeon": {
                    "_isexternal": false,
                    "ZTObjClass": "ZonalCommisioner-Provider",
                    "_instanXZname": ""
                },
                "OralCavityDesignationCode": [],
                "RepriXZdLineItemReferenXZ": "",
                "PlaXZofServiXZCode": "22",
                "SKHistoryMemo": "",
                "AmbulanXZTransportReasonCode": "",
                "FamilyPlanningIndicator": "",
                "ServiXZFacility": {
                    "_isexternal": false,
                    "Address": [
                        {
                            "ZTObjClass": "Embed-ZonalCommisioner-Address",
                            "AddressLine2": "",
                            "AddressLine1": "Mulberry des",
                            "SKPostalCode": "",
                            "SKState": ""
                        }
                    ],
                    "SKFullName": "",
                    "IdentificationCode": "",
                    "EntityIdentifierCode": "",
                    "NationalProviderIdentifier": "",
                    "SKWorkPartyUri": "",
                    "_instanXZname": "",
                    "MiddleName": "",
                    "FederalTaxID": "",
                    "ZTObjClass": "ZonalCommisioner-Provider",
                    "SKNameSuffix": "",
                    "SKFirstName": "",
                    "IdentificationCodeQualifier": "",
                    "myCode": "",
                    "EntityTypeQualifier": "",
                    "SKLastName": ""
                },
                "CompositeMedicalProXZdureCode": "",
                "ServiXZFromDate": "2018-02-02",
                "OperatingPhysician": {
                    "_isexternal": false,
                    "ZTObjClass": "ZonalCommisioner-Provider",
                    "_instanXZname": ""
                },
                "AttendingPhysician": {
                    "_isexternal": false,
                    "ZTObjClass": "ZonalCommisioner-Provider",
                    "_instanXZname": ""
                },
                "DMEUnitorBasisforMeasurementCode": "",
                "ReferringProvider": {
                    "_isexternal": false,
                    "ZTObjClass": "ZonalCommisioner-Provider",
                    "_instanXZname": ""
                },
                "PaidServiXZUnitCount": 0,
                "_isexternal": false,
                "XZ_DetailStatus": "New",
                "ReferringCLIA": "",
                "AllowedBasis": 40,
                "LengthOfMedicalNeXZssity": 0,
                "OtherOperatingPhysician": {
                    "_isexternal": false,
                    "ZTObjClass": "ZonalCommisioner-Provider",
                    "_instanXZname": ""
                },
                "Weight": 0,
                "ObstetricAdditionalUnits": 0,
                "Payment": {
                    "DeductibleAmount": 0,
                    "RepricingPerDiemOrFlatRateAmount": 0,
                    "PPSCapitalFSPDRGAmount": 0,
                    "LineItemDeniedNonCoveredChargeAmount": 0,
                    "OldCapitalAmount": 0,
                    "_instanXZname": "",
                    "RepriXZdApprovedAPGAmount": 0,
                    "PPSOperatingFederalSpecificDRGAmount": 0,
                    "Name": "Fee-for-ServiXZ",
                    "ContractAmount": 0,
                    "clamESRDPaymentAmount": 0,
                    "RepriXZdAllowedAmount": 0,
                    "AliasName": "Rate Sheet",
                    "PostageclamedAmount": 0,
                    "clamPaymentRemarkCode": [],
                    "PPSCapitalDSHDRGAmount": 0,
                    "CheckOrEFTTraXZ": "",
                    "clamPPSCapitalAmount": 0,
                    "clamDisproportionateShareAmount": 0,
                    "ReimbursementRate": 0,
                    "SalesTaxAmount": 0,
                    "_isexternal": false,
                    "LineItemChargeAmount": 209.77,
                    "LifetimePsychiatricDaysCount": 0,
                    "CoveredDaysorVisitsCount": 0,
                    "clamPPSCapitalOutlierAmount": 0,
                    "NonPayableProfComponentBillAmount": 0,
                    "AmountQualifierCode": "",
                    "PPSOperatingHospitalSpecificDRGAmount": 0,
                    "PPSCapitalHSPDRGAmount": 0,
                    "PPSCapitalIMEAmount": 0,
                    "clamDRGAmount": 0,
                    "SelectedPricingModel": "",
                    "ZonalCommisionerPCSPayableAmount": 0,
                    "CoinsuranXZAmount": 0,
                    "PaidAmount": 0,
                    "ZTObjClass": "ZonalCommisioner-Data-Payment",
                    "CopayAmount": 0,
                    "clamIndirectTeachingAmount": 0,
                    "RepriXZdSavingAmount": 0,
                    "PPSCapitalExXZptionAmount": 0,
                    "ApprovedAmount": 0,
                    "PersonPaidAmount": 0,
                    "CostReportDayCount": 0,
                    "AllowedAmount": 40,
                    "clamMSPPassThroughAmount": 0
                },
                "FacilityTaxAmount": 0,
                "ContractType": "04",
                "ExXZptionCode": "",
                "TransportDistanXZ": 0,
                "NationalDrugUnitCount": 0,
                "TermsDiscountPerXZntage": 0,
                "RenderingProviderID": "2500467644",
                "MedicalProductOrServiXZIDQualifier": "",
                "SupervisingProvider": {
                    "_isexternal": false,
                    "ZTObjClass": "ZonalCommisioner-Provider",
                    "_instanXZname": ""
                },
                "_instanXZname": "",
                "EmergencyIndicator": "",
                "SKID": "TST-${c_Counter}*2022242P0000037600",
                "ServiXZTaxAmount": 0,
                "ServicingCounty": "Gaston",
                "AmbulanXZPersonCount": 0,
                "PolicyComplianXZCode": "",
                "ServiXZUnitCount": 1,
                "ReferralList": [],
                "ThirdPartyOrganizationNotes": "",
                "PriorPlaXZmentDateTimeQualifier": "",
                "ServicingZipCode": "28033",
                "ContractPerXZntage": 0,
                "Assigned": "1",
                "ProXZdureCodeDescription": "",
                "RenderingProvider": {
                    "_isexternal": false,
                    "SKFullName": "KC",
                    "IdentificationCode": "",
                    "EntityIdentifierCode": "",
                    "NationalProviderIdentifier": "",
                    "SKWorkPartyUri": "",
                    "_instanXZname": "",
                    "MiddleName": "",
                    "FederalTaxID": "",
                    "ZTObjClass": "ZonalCommisioner-Provider",
                    "SKNameSuffix": "",
                    "SKFirstName": "",
                    "IdentificationCodeQualifier": "",
                    "myCode": "",
                    "EntityTypeQualifier": "",
                    "SKLastName": "KC"
                },
                "ProsthesisCrownOrInlayCode": "",
                "IsParticipatingNetwork": true,
                "CalculatedAllowedAmount": 40,
                "DMEDuration": 0,
                "ServiXZPredeterminationIdentification": [],
                "ContractVersion": ""
            }
        ],
        "MedicalRecord": "",
        "ServiXZAuthorizationExXZptionCode": "",
        "RepricingOrganizationIdentifier": "",
        "ZTUpdateOpName": "",
        "ZTUpdateOperator": "",
        "clamFrequencyCode": "1",
        "DiagnosisType": "",
        "COB": [],
        "AdmissionSourXZCode": "1",
        "DischargedDate": "2018-02-02",
        "OutOfStateIndicator": true,
        "LifetimePsychiatricDaysCount": 0,
        "RepriXZdApprovedAPGGroupCode": "",
        "Policy": "17135253",
        "clamFormType": "I",
        "ServiXZToDate": "2018-02-02",
        "ConditionAdditionalDescriptionPerson": "",
        "ReleaseofInformationCode": "Y",
        "ZTObjClass": "ZonalCommisioner-Data-clam",
        "RejectReasonCode": "",
        "TotalclamCoinsuranXZAmount": 0,
        "PricingMethodology": "",
        "HierarchicalStructureCode": "",
        "StatementFromDate": "2018-02-02",
        "MedicareSection4081Indicator": "",
        "clamStatus": "",
        "FacilityCodeQualifier": "",
        "SurchargeFactor": 0,
        "Referral": "",
        "PredeterminationOfBenefitsIdentifier": "",
        "FacilityType": "Specialty Facility, Hospital ASC Surgery",
        "MammographyXZrtification": "",
        "RepricingPerDiemOrFlatRateAmount": 0,
        "PrincipalDiagnosis": "S83509A",
        "DemonstrationProjectIdentifier": "",
        "CarePlanOversight": "",
        "RepriXZdApprovedServiXZUnitCount": 0,
        "PlaXZofServiXZCode": "22",
        "BillingEntity": {
            "HierarchicalID": "",
            "TaxIdentifierQualifer": "",
            "SKPostalCode": "",
            "SKFullName": "KC",
            "HierarchicalLevelCode": "",
            "SKWorkPartyUri": "BE-O-19268082",
            "_instanXZname": "",
            "CurrencyCode": "",
            "FederalTaxID": "2219268082",
            "PractiXZAddress": [
                {
                    "ZTObjClass": "Embed-ZonalCommisioner-Address",
                    "AddressLine2": "",
                    "AddressLine1": "1953"
                }
            ],
            "ProviderCode": "BI",
            "myCode": "",
            "Identification": "98082",
            "_isexternal": false,
            "SKCountry": "",
            "IdentificationCode": "",
            "EntityIdentifierCode": "",
            "NationalProviderIdentifier": "2500467644",
            "MiddleName": "",
            "HierarchicalChildCode": "",
            "Contact": [
                {
                    "_isexternal": false,
                    "ZTObjClass": "ZonalCommisioner-Data-Contact",
                    "_instanXZname": ""
                }
            ],
            "ZTObjClass": "ZonalCommisioner-Private-Party-BusinessEntity",
            "SKNameSuffix": "",
            "SKCity": "",
            "SKState": "",
            "IdentificationCodeQualifier": "",
            "EntityTypeQualifier": "XX",
            "SKLastName": "KC",
            "EffectiveDate": "2014-01-01"
        },
        "PriorAuthorization": "",
        "AmbulanXZTransportReasonCode": "",
        "InterChangeControl": "2022242600",
        "RepriXZdAllowedAmount": 0,
        "ServiXZFacility": {
            "_isexternal": false,
            "Address": [
                {
                    "ZTObjClass": "Embed-ZonalCommisioner-Address",
                    "AddressLine2": "",
                    "AddressLine1": "193  Rd",
                    "SKPostalCode": "23",
                    "SKState": "DC"
                }
            ],
            "SKFullName": "",
            "IdentificationCode": "",
            "EntityIdentifierCode": "",
            "NationalProviderIdentifier": "",
            "SKWorkPartyUri": "",
            "_instanXZname": "",
            "MiddleName": "",
            "FederalTaxID": "",
            "ZTObjClass": "ZonalCommisioner-Provider",
            "SKNameSuffix": "",
            "SKFirstName": "",
            "IdentificationCodeQualifier": "",
            "myCode": "",
            "EntityTypeQualifier": "",
            "SKLastName": ""
        },
        "BenefitAssignmentXZrtificationIndicator": "Y",
        "TotalclamApprovedAmount": 0,
        "Subscriber": {
            "SKFullName": "Sasubilli",
            "Address": [
                {
                    "ZTObjClass": "Embed-ZonalCommisioner-Address",
                    "AddressLine1": "721-24-271",
                    "State": "DCAZ",
                    "Country": "INJP",
                    "City": "Narsipatnam",
                    "AddressType": "Home",
                    "EffectiveDate": "2011-01-01T00:00:00Z"
                }
            ],
            "ZTUpdateDateTime": "2020-07-29T21",
            "OtherInformation": [
                {
                    "ZTObjClass": "Embed-ZonalCommisioner-OtherInformation",
                    "Type": "Member",
                    "ESID": "1301"
                },
                {
                    "ZTObjClass": "Embed-ZonalCommisioner-OtherInformation",
                    "Type": "Subscriber",
                    "ESID": "S301"
                }
            ],
            "SKWorkPartyUri": "301",
            "UCASELastName": "",
            "VIPIndicator": false,
            "_instanXZname": "",
            "UCASEFirstName": "",
            "EmployeeIndicator": false,
            "PreferredLanguage": "English",
            "IndividualRelationshipCode": "00",
            "_isexternal": false,
            "GenderCode": "M",
            "DateOfBirth": "1969-02-21",
            "ZTName": "17",
            "ZTSaveDateTime": "2020-07-29T21:54:30.879Z",
            "OtherName": [
                {
                    "ZTObjClass": "Embed-ZonalCommisioner-OtherName",
                    "FirstName": "",
                    "LastName": ""
                }
            ],
            "pzInsKey": "ZonalCommisioner-Private-Party-MEMBER 1713525301",
            "Weight": 0,
            "Contact": [],
            "ZTObjClass": "ZonalCommisioner-Private-Party-Member",
            "SitusState": "SC",
            "AddressLine1": "721  ",
            "SKFirstName": "",
            "SKNote": "37",
            "ZTCommitDateTime": "2022-08-12T06:54:32.828Z",
            "SKLastName": "",
            "EffectiveDate": "2011-01-01"
        },
        "LatePaymentInterest": 0,
        "ServiXZFromDate": "2018-02-02",
        "OperatingPhysician": {
            "_isexternal": false,
            "ZTObjClass": "ZonalCommisioner-Provider",
            "_instanXZname": ""
        },
        "TraXZ": "",
        "AttendingPhysician": {
            "_isexternal": false,
            "ZTObjClass": "ZonalCommisioner-Provider",
            "_instanXZname": ""
        },
        "ReferringProvider": {
            "_isexternal": false,
            "SKFullName": "",
            "IdentificationCode": "",
            "EntityIdentifierCode": "",
            "NationalProviderIdentifier": "",
            "SKWorkPartyUri": "",
            "_instanXZname": "",
            "MiddleName": "",
            "FederalTaxID": "",
            "ZTObjClass": "ZonalCommisioner-Provider",
            "SKNameSuffix": "",
            "SKFirstName": "",
            "IdentificationCodeQualifier": "",
            "myCode": "",
            "EntityTypeQualifier": "",
            "SKLastName": ""
        },
        "ValueAddedNetworkTraXZ": "",
        "BillClassification": "1",
        "XZ_HeaderStatus": "New",
        "_isexternal": false,
        "Payer": [
            {
                "_isexternal": false,
                "FederalTaxID": "32-98765432",
                "ZTObjClass": "ZonalCommisioner-Private-Party-Payer",
                "Address": [
                    {
                        "ZTObjClass": "Embed-ZonalCommisioner-Address",
                        "AddressLine2": "",
                        "Type": "",
                        "AddressLine1": "Street",
                        "Email": "",
                        "State": "DXA",
                        "PostalCode": "531111111115",
                        "Country": "",
                        "City": "Narsipatnam",
                        "AddressType": "",
                        "Fax": "",
                        "County": ""
                    }
                ],
                "IdentificationCodeQualifier": "",
                "EntityTypeQualifier": "PI",
                "EntityIdentifierCode": "",
                "SKWorkPartyUri": "",
                "PayerName": "U",
                "_instanXZname": ""
            }
        ],
        "IndustryCode": "",
        "DisabilityDateQualifier": "",
        "ReXZiver": {
            "_isexternal": false,
            "ZTObjClass": "ZonalCommisioner-Private-Party",
            "SKFullName": "",
            "IdentificationCodeQualifier": "",
            "EntityTypeQualifier": "",
            "EntityIdentifierCode": "",
            "IndividualIdentifier": "",
            "_instanXZname": "",
            "SKLastName": ""
        },
        "PayerclamControl": "",
        "OtherOperatingPhysician": {
            "_isexternal": false,
            "ZTObjClass": "ZonalCommisioner-Provider",
            "_instanXZname": ""
        },
        "ConditionCodePerson": "",
        "clamOrEncounterIdentifier": "CH",
        "SubscriberPrimaryIdentifier": "S5301",
        "DischargeDateTime": "2018-02-02T00:00:00Z",
        "ReXZivedDate": "2022-08-30",
        "Payment": {
            "DeductibleAmount": 0,
            "PPSCapitalFSPDRGAmount": 0,
            "LineItemDeniedNonCoveredChargeAmount": 0,
            "OldCapitalAmount": 0,
            "_instanXZname": "",
            "PPSOperatingFederalSpecificDRGAmount": 0,
            "ContractAmount": 0,
            "clamESRDPaymentAmount": 0,
            "clamPaymentRemarkCode": [],
            "PPSCapitalDSHDRGAmount": 0,
            "CheckOrEFTTraXZ": "",
            "clamPPSCapitalAmount": 0,
            "clamDisproportionateShareAmount": 0,
            "ReimbursementRate": 0,
            "_isexternal": false,
            "LineItemChargeAmount": 202329.77,
            "LifetimePsychiatricDaysCount": 0,
            "CoveredDaysorVisitsCount": 0,
            "clamPPSCapitalOutlierAmount": 0,
            "NonPayableProfComponentBillAmount": 0,
            "AmountQualifierCode": "",
            "PPSOperatingHospitalSpecificDRGAmount": 0,
            "PPSCapitalHSPDRGAmount": 0,
            "PPSCapitalIMEAmount": 0,
            "clamDRGAmount": 0,
            "SelectedPricingModel": "",
            "CoinsuranXZAmount": 0,
            "ZonalCommisionerPCSPayableAmount": 0,
            "PaidAmount": 0,
            "ZTObjClass": "ZonalCommisioner-Data-Payment",
            "CopayAmount": 0,
            "clamIndirectTeachingAmount": 0,
            "PPSCapitalExXZptionAmount": 0,
            "ApprovedAmount": 0,
            "PersonPaidAmount": 0,
            "CostReportDayCount": 0,
            "AllowedAmount": 40,
            "clamMSPPassThroughAmount": 0
        },
        "ConditionDescriptionPerson": "",
        "PersonPaidAmount": 0,
        "ContractType": "012234",
        "SubmitterName": "",
        "ProviderOrSupplierSignatureIndicator": "Y",
        "ExXZptionCode": "",
        "TransportDistanXZ": 0,
        "TransisionType": "",
        "SubscriberLastName": "",
        "InsuranXZTypeCode": "",
        "PersonDateOfBirth": "1991-08-11",
        "TermsDiscountPerXZntage": 0,
        "PersonSignatureSourXZCode": "P",
        "CheckStatus": "Pending",
        "HomeboundIndicator": "",
        "SupervisingProvider": {
            "_isexternal": false,
            "ZTObjClass": "ZonalCommisioner-Provider",
            "_instanXZname": ""
        },
        "_instanXZname": "",
        "SubmittedDate": "2022-08-30",
        "Submitter": "",
        "ServicingCounty": "",
        "DisabilityDateFormatQualifier": "",
        "PolicyComplianXZCode": "",
        "BillType": "01",
        "SKCountry": "",
        "CoveredDaysorVisitsCount": 0,
        "ServicingZipCode": "28033",
        "ContractPerXZntage": 0,
        "AdjRepriXZdclamRef": "",
        "Person": {
            "SKFullName": "Krish Sasubilli",
            "Address": [
                {
                    "ZTObjClass": "Embed-ZonalCommisioner-Address",
                    "AddressLine1": "32323282 Rd",
                    "State": "DXA",
                    "Country": "INJP",
                    "City": "Narsipatnam",
                    "AddressType": "Home",
                    "EffectiveDate": "2011-01-01T00"
                }
            ],
            "ZTUpdateDateTime": "2020-07-29T23",
            "OtherInformation": [
                {
                    "ZTObjClass": "Embed-ZonalCommisioner-OtherInformation",
                    "Type": "Member",
                    "ESID": "7212321104"
                },
                {
                    "ZTObjClass": "Embed-ZonalCommisioner-OtherInformation",
                    "Type": "Subscriber",
                    "ESID": "S2123wesweer3204"
                }
            ],
            "SKWorkPartyUri": "",
            "UCASELastName": "",
            "VIPIndicator": false,
            "_instanXZname": "",
            "UCASEFirstName": "",
            "EmployeeIndicator": false,
            "PreferredLanguage": "English",
            "IndividualRelationshipCode": "19",
            "Age": 26,
            "_isexternal": false,
            "GenderCode": "X",
            "DateOfBirth": "1991-08-11",
            "ZTInsName": "88462753663737383838833204",
            "ZTSaveDateTime": "2020-07-29T23:59:51.761Z",
            "OtherName": [
                {
                    "ZTObjClass": "Embed-ZonalCommisioner-OtherName",
                    "FirstName": "",
                    "LastName": ""
                }
            ],
            "pzInsKey": "ZonalCommisioner-Private-Party-MEMBER 884627204",
            "Weight": 0,
            "Contact": [],
            "ZTObjClass": "ZonalCommisioner-Private-Party-Member",
            "SitusState": "SC",
            "AddressLine1": "382 Rd",
            "SKFirstName": "",
            "SKNote": "37",
            "ZTCommitDateTime": "2022-08-12T06:56:02.408Z",
            "SKLastName": "Estersasubilli",
            "EffectiveDate": "2011-01-01"
        },
        "AdmissionTypeCode": "9",
        "RenderingProvider": {
            "_isexternal": false,
            "SKFullName": "KC",
            "IdentificationCode": "",
            "EntityIdentifierCode": "O-19268082",
            "NationalProviderIdentifier": "2500467644",
            "SKWorkPartyUri": "",
            "_instanXZname": "",
            "MiddleName": "",
            "FederalTaxID": "",
            "ZTObjClass": "ZonalCommisioner-Provider",
            "SKNameSuffix": "",
            "SKFirstName": "",
            "IdentificationCodeQualifier": "",
            "myCode": "",
            "EntityTypeQualifier": "",
            "SKLastName": "KC"
        },
        "SubscriberDateOfBirth": "1969-02-21",
        "Image": "TST-${c_Counter}",
        "RepriXZdSavingAmount": 0,
        "PayToPlan": {
            "_isexternal": false,
            "ZTObjClass": "ZonalCommisioner-Private-Party-Payer",
            "_instanXZname": ""
        },
        "TotalclamPaidAmount": 0,
        "CostReportDayCount": 0,
        "ContractVersion": ""
    }
}

Logs (if relevant)

No response

[kingherc]

Thank you @chaitu0292 for the issue. Please note that version 5.6.9 is old & unsupported (it is 3 major versions below the current one). However, in order to see if your issue can be reproduced, I would like to ask you for more details on how you run the benchmark. Please provide as much detail as possible. For example:

• Please write the exact HTTP you tried, whether it was PUT/POST, how you created the index with the settings, etc. for each version of Elasticsearch you tried. Because the versions have some changes in how the REST requests are formulated, that is why I would like to ask you to write exactly what you run.
• I see some avg and max statistics per run. How many iterations do they refer to?
• Deployed in Kubernetes single pod. How did you deploy Elasticsearch with Kubernetes? Was the host machine the same for all iterations? And what were the details of the host machine (e.g., processor type)?

[elasticsearchmachine]

Pinging @elastic/es-distributed (Team:Distributed)

[chaitu0292]

It's a Post Request.
http://localhost:9200/perfindex/_doc/<unique_id>?refresh=true {json document pasted above}
Index settings creation:
1- Index creation:

curl --location --request PUT 'http://localhost:9200/perfindex' \
--header 'Content-Type: application/json' \
--data-raw '{
  "settings": {
    "index": {
      "number_of_shards": 1,  
      "number_of_replicas": 0 
    }
  }
}'

2 - Settings

curl --location --request PUT 'http://localhost:9200/perfindex/_settings' \
--header 'Content-Type: application/json' \
--data-raw '{"index.mapping.total_fields.limit": 200000}'

curl --location --request PUT 'http://localhost:9200/perfindex/_settings' \
--header 'Content-Type: application/json' \
--data-raw '{ "index" : { "refresh_interval" : "1s"  }}'

• Same commands were used to create and apply settings for both 5.6.9 and 7.16.3
• No of Iterations 1000
• Used the default deployment yaml generated by ES helm charts with pod config - 4CPU/16GB RAM and 8GB Heap
• Yes the machine is the same for all Iterations.

Machine Details:

Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Thread(s) per core: 2
Core(s) per socket: 4
Socket(s): 2
NUMA node(s): 2
Vendor ID: GenuineIntel
CPU family: 6
Model: 63
Model name: Intel(R) Xeon(R) CPU E5-2637 v3 @ 3.50GHz
Stepping: 2
CPU MHz: 1802.337
CPU max MHz: 3700.0000
CPU min MHz: 1200.0000
BogoMIPS: 6999.80
Virtualization: VT-x
L1d cache: 32K
L1i cache: 32K
L2 cache: 256K
L3 cache: 15360K
RAM - 125GB

[chaitu0292]

@kingherc, @kimchy, @elastic/es-distributed any update on this?

[kingherc]

Hi @chaitu0292 , I tried the POST request for putting the first document on the 5.6.9 elasticsearch binary (not helm chart) and I receive:

http://localhost:9200/perfindex/_doc/1?refresh=true

HTTP/1.1 400 Bad Request
content-type: application/json; charset=UTF-8

{
  "error": {
    "root_cause": [
      {
        "type": "invalid_type_name_exception",
        "reason": "Document mapping type name can't start with '_', found: [_doc]"
      }
    ],
    "type": "invalid_type_name_exception",
    "reason": "Document mapping type name can't start with '_', found: [_doc]"
  },
  "status": 400
}

Response code: 400 (Bad Request); Time: 263ms; Content length: 263 bytes

Can you confirm your POST request works for 5.6.9?

[chaitu0292]

Hi @kingherc . That request is for 7.16. For 5.6.9 we used the below request url.

http://localhost:9200/perfindex/def/1?refresh=true

[kingherc]

Hi @chaitu0292 , thank you for the detail. I think we can use the same request for both versions then. I made a crude script to re-play your scenario. You can find it here:
89972-script-and-results.zip

It also has the results from both versions. At the bottom of each file is the average of the 1000 iterations (excluding the first iteration). Results are:

• 5.6.9 has an average of 50ms
• 7.16.3 has an average of 65ms

So I do not see such a big difference as you observed, certainly not triple or quadruple factor. Please note that I run the script on my laptop for the standalone versions (just downloaded the .zip and started Elasticsearch). So not the Helm charts.

I am a bit concerned now that there may be more at play here than just the CRUD operation you are performing (it can be the Helm charts, the system you use, and/or many other factors). For this reason, I am re-labeling this for the performance team who could advise more methodologically (if needed at all).