Project is not reindexed after changes on the scope page (mapper_parsing_exception)

[maxceem]

Issue report from the Rishi on PROD:

isn't the project getting indexed in ES when changing the scope ? I mean, I just tried changing the title of the project but on listing page its still the older one

adding/removing member triggers ES indexing though

The project with this issue https://connect.topcoder.com/projects/12310

Have to reproduce it on DEV, find the cause and fix.

[maxceem]

@RishiRajSahu I can reproduce this issue on DEV with the next project for example https://connect.topcoder-dev.com/projects/8369

But I cannot reproduce it locally.
Is it possible to get the log on what it's going on when we change the name of the project on DEV and click save? Maybe it could give me some hint.

[RishiRajSahu]

@maxceem shared the recent error log with you.

[maxceem]

Thanks, @RishiRajSahu it helps a lot. As it's from dev I can share it here, right?

[RishiRajSahu]

yep, you can.

[maxceem]

@RishiRajSahu I've just created a new project on DEV, but it doesn't appear on the project list also.

Could you please have a look at the log, if there are any errors.
Project id 8401, name ES issues test 7 (no addons) https://connect.topcoder-dev.com/projects/8401.

If I create the same project on the local setup it works: I can see it on the project list.

[maxceem]

Error which causes the issue

This is the issue which happens at the Project Service, which prevents the project to reindex in ES:

object mapping for [details.appDefinition.addons.design] tried to parse field [design] as object, but found a concrete value

You may check the full log for reference:

{
    "name": "tc-projects-service-dev",
    "hostname": "ip-10-15-50-137.ec2.internal",
    "pid": 34,
    "requestId": "95d04179-456e-42ad-897b-91f9b0e68594",
    "level": 50,
    "msg": "failed to get project document, (projectId: 8369) { [mapper_parsing_exception] object mapping for [details.appDefinition.addons.design] tried to parse field [design] as object, but found a concrete value :: {\"path\":\"/projects-2019-05-29/projectV4/8369/_update\",\"query\":{},\"body\":\"{\\\"doc\\\":{\\\"id\\\":8369,\\\"directProjectId\\\":18999,\\\"billingAccountId\\\":null,\\\"name\\\":\\\"Invitation test max 2 changed 1\\\",\\\"description\\\":\\\"We will ask you several questions in order to determine your project's scope. All estimates are based on our 15 years of experience and thousands of projects. Final prices will be determined after our team completes a final scope review.\\\\n\\\",\\\"external\\\":null,\\\"bookmarks\\\":[],\\\"estimatedPrice\\\":null,\\\"actualPrice\\\":null,\\\"terms\\\":[],\\\"type\\\":\\\"scoped-solutions\\\",\\\"status\\\":\\\"draft\\\",\\\"details\\\":{\\\"utm\\\":{\\\"code\\\":\\\"\\\"},\\\"appDefinition\\\":{\\\"deliverables\\\":[\\\"design\\\"],\\\"designGoal\\\":\\\"concept-designs\\\",\\\"quickTurnaround\\\":\\\"under-3-days\\\",\\\"targetDevices\\\":[\\\"mobile\\\"],\\\"numberScreens\\\":\\\"2-4\\\",\\\"caNeeded\\\":\\\"yes\\\",\\\"addons\\\":{\\\"design\\\":\\\"\\\"},\\\"mobilePlatforms\\\":\\\"\\\",\\\"features\\\":{}},\\\"apiDefinition\\\":{\\\"notes\\\":\\\"\\\"},\\\"hideDiscussions\\\":true},\\\"challengeEligibility\\\":[],\\\"cancelReason\\\":null,\\\"templateId\\\":104,\\\"createdAt\\\":\\\"2019-09-22T08:54:57.000Z\\\",\\\"updatedAt\\\":\\\"2019-10-10T04:59:06.000Z\\\",\\\"deletedBy\\\":null,\\\"createdBy\\\":40152856,\\\"updatedBy\\\":40152856,\\\"version\\\":\\\"v3\\\",\\\"lastActivityAt\\\":\\\"2019-10-10T04:38:06.895Z\\\",\\\"lastActivityUserId\\\":\\\"40152856\\\",\\\"members\\\":[{\\\"lastName\\\":\\\"Dev\\\",\\\"role\\\":\\\"manager\\\",\\\"updatedBy\\\":40153891,\\\"handle\\\":\\\"maxceem-dev\\\",\\\"userId\\\":40153891,\\\"deletedBy\\\":null,\\\"createdAt\\\":\\\"2019-10-03T07:58:04.000Z\\\",\\\"firstName\\\":\\\"Maksym1\\\",\\\"deletedAt\\\":null,\\\"createdBy\\\":40153891,\\\"isPrimary\\\":false,\\\"id\\\":11485,\\\"projectId\\\":8369,\\\"email\\\":\\\"maxceem.dev@gmail.com\\\",\\\"updatedAt\\\":\\\"2019-10-03T07:58:04.000Z\\\"},{\\\"lastName\\\":\\\"ProgramManager\\\",\\\"role\\\":\\\"manager\\\",\\\"updatedBy\\\":40158598,\\\"handle\\\":\\\"pshahprojmanag\\\",\\\"userId\\\":40158598,\\\"deletedBy\\\":null,\\\"createdAt\\\":\\\"2019-10-09T07:27:26.000Z\\\",\\\"firstName\\\":\\\"Pshah\\\",\\\"deletedAt\\\":null,\\\"createdBy\\\":40158598,\\\"isPrimary\\\":false,\\\"id\\\":11522,\\\"projectId\\\":8369,\\\"email\\\":\\\"maxceem+tc+pshahprojmanag@gmail.com\\\",\\\"updatedAt\\\":\\\"2019-10-09T07:27:26.000Z\\\"},{\\\"lastName\\\":\\\"Customer\\\",\\\"role\\\":\\\"customer\\\",\\\"updatedBy\\\":40152922,\\\"handle\\\":\\\"pshah_customer\\\",\\\"userId\\\":40152922,\\\"deletedBy\\\":null,\\\"createdAt\\\":\\\"2019-10-10T04:30:36.000Z\\\",\\\"firstName\\\":\\\"Parth\\\",\\\"deletedAt\\\":null,\\\"createdBy\\\":40152922,\\\"isPrimary\\\":false,\\\"id\\\":11533,\\\"projectId\\\":8369,\\\"email\\\":\\\"vikas.agarwal+pshah_customer@topcoder.com\\\",\\\"updatedAt\\\":\\\"2019-10-10T04:30:36.000Z\\\"},{\\\"lastName\\\":\\\"Manager1\\\",\\\"role\\\":\\\"manager\\\",\\\"updatedBy\\\":40152856,\\\"handle\\\":\\\"pshah_manager\\\",\\\"userId\\\":40152856,\\\"deletedBy\\\":null,\\\"createdAt\\\":\\\"2019-10-10T04:38:06.000Z\\\",\\\"firstName\\\":\\\"Parth121\\\",\\\"deletedAt\\\":null,\\\"createdBy\\\":40152856,\\\"isPrimary\\\":false,\\\"id\\\":11535,\\\"projectId\\\":8369,\\\"email\\\":\\\"vikas.agarwal+pshah_manager@topcoder.com\\\",\\\"updatedAt\\\":\\\"2019-10-10T04:38:06.000Z\\\"}],\\\"phases\\\":[],\\\"estimations\\\":[{\\\"maxTime\\\":3,\\\"createdAt\\\":\\\"2019-09-22T08:54:57.000Z\\\",\\\"metadata\\\":{\\\"deliverable\\\":\\\"design\\\"},\\\"updatedBy\\\":40152856,\\\"buildingBlockKey\\\":\\\"FREE_SIZE_CONC_DESIGN_ASAP_CA\\\",\\\"createdBy\\\":40152856,\\\"price\\\":16000,\\\"minTime\\\":3,\\\"id\\\":578,\\\"conditions\\\":\\\"( HAS_DESIGN_DELIVERABLE && CONCEPT_DESIGN && QUICK_DESIGN_3_DAYS && CA_NEEDED )\\\",\\\"projectId\\\":8369,\\\"updatedAt\\\":\\\"2019-09-22T08:54:57.000Z\\\"}],\\\"invites\\\":[{\\\"createdAt\\\":\\\"2019-09-23T08:17:13.000Z\\\",\\\"deletedAt\\\":null,\\\"role\\\":\\\"customer\\\",\\\"updatedBy\\\":40152856,\\\"createdBy\\\":40152856,\\\"id\\\":729,\\\"projectId\\\":8369,\\\"userId\\\":40135460,\\\"email\\\":\\\"test@test.com\\\",\\\"deletedBy\\\":null,\\\"updatedAt\\\":\\\"2019-09-23T08:17:13.000Z\\\",\\\"status\\\":\\\"pending\\\"},{\\\"createdAt\\\":\\\"2019-10-03T03:09:26.000Z\\\",\\\"deletedAt\\\":null,\\\"role\\\":\\\"customer\\\",\\\"updatedBy\\\":40152856,\\\"createdBy\\\":40152856,\\\"id\\\":779,\\\"projectId\\\":8369,\\\"userId\\\":40035291,\\\"email\\\":null,\\\"deletedBy\\\":null,\\\"updatedAt\\\":\\\"2019-10-03T03:09:26.000Z\\\",\\\"status\\\":\\\"pending\\\"}],\\\"utm\\\":null,\\\"deletedAt\\\":null,\\\"attachments\\\":[]}}\",\"statusCode\":400,\"response\":\"{\\\"error\\\":{\\\"root_cause\\\":[{\\\"type\\\":\\\"mapper_parsing_exception\\\",\\\"reason\\\":\\\"object mapping for [details.appDefinition.addons.design] tried to parse field [design] as object, but found a concrete value\\\"}],\\\"type\\\":\\\"mapper_parsing_exception\\\",\\\"reason\\\":\\\"object mapping for [details.appDefinition.addons.design] tried to parse field [design] as object, but found a concrete value\\\"},\\\"status\\\":400}\"}\n    at respond (/usr/src/app/node_modules/elasticsearch/src/lib/transport.js:289:15)\n    at checkRespForFailure (/usr/src/app/node_modules/elasticsearch/src/lib/transport.js:248:7)\n    at done (/usr/src/app/node_modules/http-aws-es/connector.js:48:7)\n    at IncomingMessage.cleanUp (/usr/src/app/node_modules/http-aws-es/src/node.js:20:7)\n    at emitNone (events.js:110:20)\n    at IncomingMessage.emit (events.js:207:7)\n    at endReadableNT (_stream_readable.js:1045:12)\n    at /usr/src/app/node_modules/async-listener/glue.js:188:31\n    at _combinedTickCallback (internal/process/next_tick.js:138:11)\n    at process._tickDomainCallback (internal/process/next_tick.js:218:9)\n    at process.fallback (/usr/src/app/node_modules/async-listener/index.js:539:15)\nFrom previous event:\n    at runCallback (timers.js:781:20)\n    at tryOnImmediate (timers.js:743:5)\n    at processImmediate [as _immediateCallback] (timers.js:714:5)\nFrom previous event:\n    at /usr/src/app/dist/services/rabbitmq.js:130:13\n    at Channel.BaseChannel.dispatchMessage (/usr/src/app/node_modules/amqplib/lib/channel.js:466:12)\n    at Channel.BaseChannel.handleDelivery (/usr/src/app/node_modules/amqplib/lib/channel.js:475:15)\n    at emitOne (events.js:115:13)\n    at Channel.emit (events.js:210:7)\n    at /usr/src/app/node_modules/amqplib/lib/channel.js:263:10\n    at Channel.content [as handleMessage] (/usr/src/app/node_modules/amqplib/lib/channel.js:316:9)\n    at Channel.C.acceptMessageFrame (/usr/src/app/node_modules/amqplib/lib/channel.js:231:31)\n    at Channel.C.accept (/usr/src/app/node_modules/amqplib/lib/channel.js:384:17)\n    at Connection.mainAccept [as accept] (/usr/src/app/node_modules/amqplib/lib/connection.js:63:33)\n    at Socket.go (/usr/src/app/node_modules/amqplib/lib/connection.js:476:48)\n    at emitNone (events.js:105:13)\n    at Socket.emit (events.js:207:7)\n    at emitReadable_ (_stream_readable.js:502:10)\n    at emitReadable (_stream_readable.js:496:7)\n    at addChunk (_stream_readable.js:263:7)\n    at readableAddChunk (_stream_readable.js:239:11)\n    at Socket.Readable.push (_stream_readable.js:197:10)\n    at TCP.onread [as _originalOnread] (net.js:589:20)\n  status: 400,\n  displayName: 'BadRequest',\n  message: '[mapper_parsing_exception] object mapping for [details.appDefinition.addons.design] tried to parse field [design] as object, but found a concrete value',\n  path: '/projects-2019-05-29/projectV4/8369/_update',\n  query: {},\n  body: \n   { error: \n      { root_cause: [Array],\n        type: 'mapper_parsing_exception',\n        reason: 'object mapping for [details.appDefinition.addons.design] tried to parse field [design] as object, but found a concrete value' },\n     status: 400 },\n  statusCode: 400,\n  response: '{\"error\":{\"root_cause\":[{\"type\":\"mapper_parsing_exception\",\"reason\":\"object mapping for [details.appDefinition.addons.design] tried to parse field [design] as object, but found a concrete value\"}],\"type\":\"mapper_parsing_exception\",\"reason\":\"object mapping for [details.appDefinition.addons.design] tried to parse field [design] as object, but found a concrete value\"},\"status\":400}',\n  toString: [Function],\n  toJSON: [Function] }",
    "time": "2019-10-10T04:59:06.687Z",
    "v": 0
}

Steps to reproduce

1. Create a project using a form with add-ons, for example https://connect.topcoder-dev.com/new-project/app_new. During creation check some deliverable which would let you choose some add-ons, but DON'T choose any add-ons, even don't touch add-ons questions.

2. As a results a project would be created the next details:

    {
       "utm": {},
       "appDefinition": {
         "deliverables": [
           "design"
         ],
         "designGoal": "concept-designs",
         "quickTurnaround": "under-3-days",
         "targetDevices": [
           "mobile"
         ],
         "numberScreens": "2-4",
         "caNeeded": "yes",
         "addons": {}
       },
       "apiDefinition": {},
       "hideDiscussions": true
     }

   • everything is good so far, and "addons": {}

3. Now open created project for editing, got to the Scope tab, and change something small not connected with add-ons or event details field, for example, project name.
   Now, the saved details would be updated, even we didn't change them, and would look like this:

   {
      "utm": {
        "code": ""
      },
      "appDefinition": {
        "deliverables": [
          "design"
        ],
        "designGoal": "concept-designs",
        "quickTurnaround": "under-3-days",
        "targetDevices": [
          "mobile"
        ],
        "numberScreens": "2-4",
        "caNeeded": "yes",
        "addons": {
          "design": ""
        },
        "mobilePlatforms": "",
        "features": {}
      },
      "apiDefinition": {
        "notes": ""
      },
      "hideDiscussions": true
    }

   • note, that some default values have been changed, even though we didn't touch them. In particular, the value of addons becomes { "design": "" } when such value is saved, it would trigger that error during ES indexing, as it expects this value to be an array.

Solution (in process)

I've fixed this inconsistency, described above (localy), so all new projects would have a good default value "addons": { "design": [] }. But after testing it looks like there might be other issues of the same type. As if I create a project after the fix, it's not shown on the Project List. So I keep checking...

[RishiRajSahu]

@maxceem great ! let me check the logs for you.

[vikasrohit]

@maxceem you made changes for this default value logic in connect-app, right? Not in this repo?

[maxceem]

Yes, @vikasrohit. It's only client-side changes. I didn't push them yet, as I wasn't sure, that all the cases are solved.

Though, now I'm started thinking that other issues that I've faced were caused by this:

Today we faced a issue in dev env because v5 deployment has rabbitmq consumer up and was consuming the messages before v4 consumer and hence, we were missing many object from being indexed in the database.

So I'll test once more my solution.

Anyway, it only client-side.

[vikasrohit]

Thanks for the info. Yes, in last couple of weeks the ES indexing issues were mainly because of v5 deployment. However, the issue which you mentioned originally here is also a legitimate issue because I have faced this issue on production few months back and I fixed that indexing issue by going to the particular project and just check/uncheck the milestones.

[maxceem]

This has been fixed client-side via appirio-tech/connect-app#3381