Update MongoDB protocol with new opcodes

[pooqadmin]

Please post all questions and issues on https://discuss.elastic.co/c/beats
before opening a Github Issue. Your questions will reach a wider audience there,
and if we confirm that there is a bug, then you can open a new issue.

For security vulnerabilities please only send reports to security@elastic.co.
See https://www.elastic.co/community/security for more information.

For confirmed bugs, please report:

• Version: packetbeat-6.1.2-linux-x86_64
• Operating System: Linux contents-mongo-dev-01 4.9.20-11.31.amzn1.x86_64 I can't install packetbeat #1 SMP Thu Apr 13 01:53:57 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
• Mongodb Version: db version v3.2.13
• Steps to Reproduce:

[root@contents-mongo-dev-01 packetbeat-6.1.2-linux-x86_64]# /beat/packetbeat-6.1.2-linux-x86_64/packetbeat -c /beat/packetbeat-6.1.2-linux-x86_64/packetbeat.yml -e

Q) When packetbeat is executed, the following error occurs.

2018/01/26 06:30:01.311175 mongodb_parser.go:42: ERR Unknown operation code:
2018/01/26 06:30:01.311872 mongodb_parser.go:42: ERR Unknown operation code:
2018/01/26 06:30:01.311902 mongodb_parser.go:42: ERR Unknown operation code:
2018/01/26 06:30:01.311928 mongodb_parser.go:42: ERR Unknown operation code:
2018/01/26 06:30:01.403019 mongodb_parser.go:42: ERR Unknown operation code:
2018/01/26 06:30:01.403631 mongodb_parser.go:42: ERR Unknown operation code:

Q) My setup is as below and I simply linked it to the logstash. Please check the cause of the above error.

[root@contents-mongo-dev-01 packetbeat-6.1.2-linux-x86_64]# cat packetbeat.yml | grep -v '#'

packetbeat.interfaces.device: any

packetbeat.flows:
timeout: 30s
period: 10s

packetbeat.protocols:

• type: mongodb
  ports: [27017]
  send_request: true
  send_response: true
  max_docs: 0
  max_doc_length: 0

setup.template.settings:
index.number_of_shards: 3
setup.kibana:

output.elasticsearch:
hosts: ["internal-bh-elasticsearch-lb-1602908268.ap-northeast-2.elb.amazonaws.com:9200"]

logging.level: error

[adriansr]

It seems the mongodb parser needs to be updated with new operations added to the protocol.

Btw I will push a quick fix for that error message to print the actual operation code

[dolftax]

This missing ones are op_command and op_commandreply. As per docs, both these opcodes are

• Cluster internal and should not be implemented by clients or drivers.
• Changed in version 3.6: Deprecated and may be removed in a future release of MongoDB.

Should we really implement this? @adriansr

[adriansr]

@jaipradeesh I see, if they are deprecated then implementing might be a waste of time. However, we should at least expect to see those messages and don't log an error every time.

[dolftax]

@adriansr #6440 works?

[dolftax]

freenode/#mongodb logs - 2018-02-27

15:41 < dolftax> In need of pcap files to test OP_COMMAND and OP_COMMANDREPLY opcodes // How to trigger them?
15:46 <@Derick> dolftax: let me check for you
15:46 <@Derick> dolftax: what do you need the pcap files for?
15:47 <@Derick> dolftax: I believe these are used by the mongos balancer in a sharded environment
15:48 <@Derick> and only with MongoDB 3.4 (although 3.2 and 3.6 support it too for up/downgrade)
15:49 < dolftax> Just need to know it some calls to DB from the client returns such opcodes. Since what queries is made by the client is not in my control, just need to run a quick test on the response opcodes and see if it sends any invalid opcodes.
15:49 < dolftax> invalid -> not whitelisted by us.
15:49 <@Derick> I would strongly suggest you do not white or blacklist any opcodes that MongoDB uses, as this can change in the future without warning
15:51 <@Derick> But: mongos 3.4 balancer commands use it
15:52 <@Derick> and to illustrate that op codes get added all the time: https://derickrethans.nl/wireshark-mongo-36.html
15:53 < dolftax> So, https://docs.mongodb.com/manual/reference/mongodb-wire-protocol/ are not really a whitelist of opcodes?
15:54 <@Derick> no
15:54 <@Derick> it misses things
15:55 <@Derick> do not restrict what MongoDB can send/receive

Derick is with the MongoDB team.

@ruflin In such case, I don't think validating across the whitelist of opcodes is necessary. What do you think?

// @adriansr

[ruflin]

I leave it to @adriansr to comment :-)

[dolftax]

@adriansr What do you think?

[adriansr]

Hi @jaipradeesh

Sorry about taking so long to respond to this, I screwed up my github notifications.

So I understand the whole point in here is to have packetbeat ignore these opcodes and not log an error every time it receives one. I don't think this is a problem, maybe Derick from MongoDB is understanding something different as to what whitelist and blacklist mean. We are not blocking this opcodes from being sent, we just don't want to store an event in Elasticsearch nor log an error.

Have you been able to test it and confirm that no more ERR Unknown operation code: are printed with the code in #6440 ?

[dolftax]

Okay. So, if we don't maintain a list of supported opcodes, then in those cases, Err Unknown operation code would be printed. PR #6440 adds OP_COMMAND and OP_COMMANDREPLY. But, say OP_COMPRESSED would still return ERR Unknown operation code:.