From b988c7f758512880ebfe2bd4b594a7c62cb1f210 Mon Sep 17 00:00:00 2001 From: Marcin Tojek Date: Wed, 12 Feb 2020 21:59:24 +0100 Subject: [PATCH] Cherry-pick #16204 to 7.x: Filebeat: Merge "mqtt" input to master (#16284) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * Filebeat: Merge "mqtt" input to master (#16204) * Add MQTT input to Filebeat (#15287) * Inital commit for MQTT input * Improved naming and error handling * Improved naming and connection procedure * Merge "master" branch into "feature-mqtt-input" (#15745) * [Filebeat] Fixes for NetFlow v9 devices from various vendors (#15449) - Allow for zero scope fields in options template NetFlow v9 spec allows for options templates that contain no scope fields. The netflow input was treating this case as an error and discarding the template, but that is only applicable to IPFIX. - Use additional fields to populate bytes/pkt counters Some devices out there (Cisco NSEL) use fields 231/232 as bytes counters, when those are supposed to be layer 4 payload counters. This updates the ECS fields populator to use those fields when the expected ones are not found. - Support a classId of 32 bits While the spec mandates a classId of 8 bits, some Cisco ASA devices actually use a 32 bit version of this field. This patches the field to allow up to 32-bit integers and updates the index pattern to use `long` for the `netflow.class_id` field. - Add more fields from v9 Cisco devices Fixes #14212 * update settings for `decode_csv_fields` (#15249) (#15550) Co-authored-by: DeDe Morton Co-authored-by: Sophia Xu * docs: updates to output config (#15272) * [Filebeat] Handle error message in handleS3Objects function (#15545) * Handle error message in handleS3Objects function * remove s3Context.Fail and use setError and done instead * Add changelog * Fix use of wrong fields in Cisco ASA dashboard (#15553) This dashboard wasn't updated after a couple of fields were renamed. Fixes: #15420 * Add test for publisher spool encode and decode. (#15534) * Add test for publisher queue encode and decode. * Run mage fmt. * Fixes from code review. * Fix convert processor conversion of string with leading zeros to integer (#15557) The conversion failed when for strings with leading zeroes and a decimal digit 8 or 9, as the underlying runtime function would try to parse that as an octal number. This is fixed by only allowing decimal and hex, which in turns makes the processor more aligned to its Elasticsearch counterpart. Fixes #15513 * New mage target: generate pkg file to test the manager (#15580) This PR adds a new mage target to Functionbeat named `buildPkgForFunction`. It generates the folder `pkg` with the functions to make testing the manager more comfortable during development. * Packetbeat TLS: Replace array fields with keyword (#15597) Use of `type: array` in some fields (which was inconsistent) causes those fields to be excluded from the template. This prevents pointing aliases to those fields, which we need in 7.6+. Setting those fields to `keyword` explicitly so that they are included in the template. Fixes #15588 * Add a pull request template providing valuable information when reviewing a PR (#15388) * Add a PR template that provides valuable information when reviewing a PR * Add CLA check * Fix typo * Address comments during review * SF: Fix typo * Add deprecation as PR type * Make it clear how to strike through in markdown * Add default configuration files to the checklist * [Metricbeat] Implement IBM MQ module (#15301) * Modify cockroachdb source * Define testdata * Do not publish ports * Update docs * mage fmt update * Describe containerized environment * Update CHANGELOG.next.asciidoc Co-Authored-By: Chris Mark * Update data.json * Rename image * Update source after review * Filter ibmmq_ metrics * mage check * Fix: mage check * Don't expose port * Rename status to qmgr * Add subscriptions overview dashboard for IBM MQ module * Add calls, messages overview dashboard for IBM MQ module * Add screenshots * Fix: mage check * Fix: CHANGELOG * Add explanation * Fix: mage check Co-authored-by: Chris Mark * Cleanup changelogs for master (#15617) * Cleanup changelogs for master * Remove extra header in CHANGELOG.asciidoc * [Metricbeat] Add lambda metricset in aws module (#15260) * Add lambda metricset * Adds missing imports (#15624) * [docs] Clarify privileges required for the writer role (#15604) * Mask password discovered via module autodiscover hint (#15616) * Mask password is string representation of config * Rename method * Adding unit test * Use const for module config password setting name * Using common.DebugString * Simplifying * Removing now-invalid unit test * Removing now-unnecessary const * Refactoring: moving debug-related var and func to common file * Refactoring: rename from black list to mask list * Implement fmt.Formatter for common.MapStr * Reintroduce debug statement * Make MarshalLogObject always filter MapStr object for logging purposes * Refactoring: renaming to be bit more generic * Forgot to add license header to new file * Fixing verb syntax * Update KQL to get estimated cost without dimension ServiceName (#15640) * Adding monitoring.cloud.* settings to reference files (#15648) * Adding monitoring.cloud.* settings to reference files * Missed winlogbeat somehow * Missed x-pack/winlogbeat * remove lablels (#15644) * Fix panic: don't send events if client is nil (#15568) * Fix panic: don't send events if client is nil * Use mutex * Add CHANGELOG entry * Rename changelog entry * Fix: changelog * Temporarily use specific logstash release * [Metricbeat] Add Istio mesh metricset (#15535) * [Metricbeat] Fix changelog (#15681) * Fix changelog * ci: use APM pipeline library (#15636) it uses APM pipeline library configured in the instance * AWS Lambda: downgrade Kibana dashboard (#15682) * AWS Lambda: downgrade Kibana dashboard * Downgrade other AWS dashboards * Log command error if setup dashboards fails * Another downgrade * Use github.com/godror/godror instead of goracle.v2 (#15683) From the README of goracle: > Goracle is deprecated because of naming (trademark) issues. From now on we are using github.com/godror/godror instead. * Move pdh query to shared location in order for new modules/metricsets to reuse (#15503) * Move pdh query to shared location * Update changelog * Fix make update * mage fmt * fix changelog * Remove datasource option from SQL module and add tests (#15686) Remove datasource option from SQL module. This option was intended to set the DSN of a database connection, and we were ignoring the hosts setting. In other SQL modules we are using the values in hosts as DSNs, do here the same for consistency. Host is redacted when we cannot parse it as it can contain passwords. StandardizeEvent is exposed in mbtest.Fetcher interface so we can more easily check contents of events in tests. Add integration tests of the module with MySQL and PostgreSQL. Add real data.json with data from MySQL and PostgreSQL. * [metricbeat] add service metricset to reference documentation (#15643) * add service metricset to ref docs * update xpack docs * [metricbeat] Add divide by zero check to docker/diskio (#15649) * add NaN check to docker/diskio * return 0 * Change joda style pattern to java style (#15695) since 7.0 elasticsearch is using java.time style patterns. YYYY becomes yyyy * [DOCS] Add missing config options to shared file (#15136) * [DOCS] Add missing config options to shared file * Add fixes from review * Run mage fmt update to fix build error * [Heartbeat] Support for multiple status codes #13595 (#15587) Allow for multiple status codes in config. Fixes https://github.com/elastic/beats/issues/13595 * Add missing changelog entry for #15587 (#15721) * Update github.com/godror/godror to v0.10.4 (#15737) ## What does this PR do? This PR updates the dependency `github.com/godror/godror` to v0.10.4. ## Why is it important? Packaging of Metricbeat fails due to the issue in the `godror` version we are currently using. See more about the problem: https://github.com/godror/godror/issues/8 * Collect normalized CPU percentages by default (#15729) * Collect normalized CPU percentages by default * Adding CHANGELOG entry * Updating x-pack/metricbeat * Fix: mage check * Detect Eclipse Public License Co-authored-by: Adrian Serrano Co-authored-by: DeDe Morton Co-authored-by: Sophia Xu Co-authored-by: Brandon Morelli Co-authored-by: kaiyan-sheng Co-authored-by: Blake Rouse Co-authored-by: Noémi Ványi Co-authored-by: Manuel de la Peña Co-authored-by: Chris Mark Co-authored-by: Michael Madden Co-authored-by: Shaunak Kashyap Co-authored-by: Pablo Mercado Co-authored-by: Ivan Fernandez Calvo Co-authored-by: Mariana Dima Co-authored-by: Jaime Soriano Pastor Co-authored-by: Alex K. <8418476+fearful-symmetry@users.noreply.github.com> Co-authored-by: Przemyslaw Gomulka Co-authored-by: Amanda H. L. de Andrade Katz Co-authored-by: Andrew Cholakian * [Filebeat] Refactor mqtt input (#16014) * Refactor mqtt input * Fix: comment * Add unit tests * Test: input run * Fix Test: run and stop * Test: backoff * Adjust code after review * MQTT: update docs (#16152) * MQTT: add integration test (#16143) * Create mosquitto image * MQTT input: add integration test * Fix * Verify connectivity * Fix * Fix: mage check * Fix * Fix * Fix: remove global var * Update changelog * Fix: regenerate notice file * Remove unused dependency * Fix: zero qos * Wait asynchronously for client being disconnected Co-authored-by: Felix Co-authored-by: Adrian Serrano Co-authored-by: DeDe Morton Co-authored-by: Sophia Xu Co-authored-by: Brandon Morelli Co-authored-by: kaiyan-sheng Co-authored-by: Blake Rouse Co-authored-by: Noémi Ványi Co-authored-by: Manuel de la Peña Co-authored-by: Chris Mark Co-authored-by: Michael Madden Co-authored-by: Shaunak Kashyap Co-authored-by: Pablo Mercado Co-authored-by: Ivan Fernandez Calvo Co-authored-by: Mariana Dima Co-authored-by: Jaime Soriano Pastor Co-authored-by: Alex K. <8418476+fearful-symmetry@users.noreply.github.com> Co-authored-by: Przemyslaw Gomulka Co-authored-by: Amanda H. L. de Andrade Katz Co-authored-by: Andrew Cholakian (cherry picked from commit 56c93e536c14abccbd14c64427e6023a865833b5) * Update CHANGELOG --- CHANGELOG.next.asciidoc | 1 + NOTICE.txt | 122 ++ dev-tools/generate_notice.py | 10 +- filebeat/docker-compose.yml | 8 + filebeat/docs/filebeat-options.asciidoc | 3 + filebeat/docs/inputs/input-mqtt.asciidoc | 81 ++ filebeat/include/list.go | 1 + filebeat/input/mqtt/client.go | 54 + filebeat/input/mqtt/client_mocked.go | 220 +++ filebeat/input/mqtt/config.go | 52 + filebeat/input/mqtt/input.go | 204 +++ filebeat/input/mqtt/input_test.go | 349 +++++ filebeat/input/mqtt/logging.go | 79 ++ filebeat/input/mqtt/mqtt_integration_test.go | 170 +++ .../environments/docker/mosquitto/Dockerfile | 2 + .../eclipse/paho.mqtt.golang/CONTRIBUTING.md | 56 + .../eclipse/paho.mqtt.golang/DISTRIBUTION | 15 + .../eclipse/paho.mqtt.golang/LICENSE | 87 ++ .../eclipse/paho.mqtt.golang/README.md | 71 + .../eclipse/paho.mqtt.golang/about.html | 41 + .../eclipse/paho.mqtt.golang/client.go | 965 +++++++++++++ .../eclipse/paho.mqtt.golang/components.go | 31 + .../eclipse/paho.mqtt.golang/edl-v10 | 15 + .../eclipse/paho.mqtt.golang/epl-v10 | 70 + .../eclipse/paho.mqtt.golang/filestore.go | 255 ++++ .../eclipse/paho.mqtt.golang/memstore.go | 138 ++ .../eclipse/paho.mqtt.golang/message.go | 127 ++ .../eclipse/paho.mqtt.golang/messageids.go | 141 ++ .../eclipse/paho.mqtt.golang/net.go | 330 +++++ .../eclipse/paho.mqtt.golang/notice.html | 108 ++ .../eclipse/paho.mqtt.golang/oops.go | 21 + .../eclipse/paho.mqtt.golang/options.go | 374 +++++ .../paho.mqtt.golang/options_reader.go | 161 +++ .../paho.mqtt.golang/packets/connack.go | 52 + .../paho.mqtt.golang/packets/connect.go | 151 +++ .../paho.mqtt.golang/packets/disconnect.go | 34 + .../paho.mqtt.golang/packets/packets.go | 346 +++++ .../paho.mqtt.golang/packets/pingreq.go | 34 + .../paho.mqtt.golang/packets/pingresp.go | 34 + .../paho.mqtt.golang/packets/puback.go | 42 + .../paho.mqtt.golang/packets/pubcomp.go | 42 + .../paho.mqtt.golang/packets/publish.go | 83 ++ .../paho.mqtt.golang/packets/pubrec.go | 42 + .../paho.mqtt.golang/packets/pubrel.go | 42 + .../paho.mqtt.golang/packets/suback.go | 57 + .../paho.mqtt.golang/packets/subscribe.go | 69 + .../paho.mqtt.golang/packets/unsuback.go | 42 + .../paho.mqtt.golang/packets/unsubscribe.go | 56 + .../eclipse/paho.mqtt.golang/ping.go | 69 + .../eclipse/paho.mqtt.golang/router.go | 181 +++ .../eclipse/paho.mqtt.golang/store.go | 136 ++ .../eclipse/paho.mqtt.golang/token.go | 183 +++ .../eclipse/paho.mqtt.golang/topic.go | 86 ++ .../eclipse/paho.mqtt.golang/trace.go | 40 + .../eclipse/paho.mqtt.golang/websocket.go | 95 ++ vendor/github.com/gorilla/websocket/AUTHORS | 9 + vendor/github.com/gorilla/websocket/LICENSE | 22 + vendor/github.com/gorilla/websocket/README.md | 64 + vendor/github.com/gorilla/websocket/client.go | 395 ++++++ .../gorilla/websocket/client_clone.go | 16 + .../gorilla/websocket/client_clone_legacy.go | 38 + .../gorilla/websocket/compression.go | 148 ++ vendor/github.com/gorilla/websocket/conn.go | 1201 +++++++++++++++++ .../gorilla/websocket/conn_write.go | 15 + .../gorilla/websocket/conn_write_legacy.go | 18 + vendor/github.com/gorilla/websocket/doc.go | 227 ++++ vendor/github.com/gorilla/websocket/go.mod | 3 + vendor/github.com/gorilla/websocket/go.sum | 2 + vendor/github.com/gorilla/websocket/join.go | 42 + vendor/github.com/gorilla/websocket/json.go | 60 + vendor/github.com/gorilla/websocket/mask.go | 54 + .../github.com/gorilla/websocket/mask_safe.go | 15 + .../github.com/gorilla/websocket/prepared.go | 102 ++ vendor/github.com/gorilla/websocket/proxy.go | 77 ++ vendor/github.com/gorilla/websocket/server.go | 363 +++++ vendor/github.com/gorilla/websocket/trace.go | 19 + .../github.com/gorilla/websocket/trace_17.go | 12 + vendor/github.com/gorilla/websocket/util.go | 283 ++++ .../gorilla/websocket/x_net_proxy.go | 473 +++++++ .../appengine/datastore/datastore.go | 407 ++++++ .../appengine/datastore/doc.go | 361 +++++ .../datastore/internal/cloudkey/cloudkey.go | 120 ++ .../datastore/internal/cloudpb/entity.pb.go | 344 +++++ .../appengine/datastore/key.go | 400 ++++++ .../appengine/datastore/keycompat.go | 89 ++ .../appengine/datastore/load.go | 429 ++++++ .../appengine/datastore/metadata.go | 78 ++ .../appengine/datastore/prop.go | 330 +++++ .../appengine/datastore/query.go | 774 +++++++++++ .../appengine/datastore/save.go | 333 +++++ .../appengine/datastore/transaction.go | 96 ++ vendor/vendor.json | 48 + .../metricbeat/module/istio/_meta/Dockerfile | 0 .../module/istio/docker-compose.yml | 0 94 files changed, 13743 insertions(+), 2 deletions(-) create mode 100644 filebeat/docs/inputs/input-mqtt.asciidoc create mode 100644 filebeat/input/mqtt/client.go create mode 100644 filebeat/input/mqtt/client_mocked.go create mode 100644 filebeat/input/mqtt/config.go create mode 100644 filebeat/input/mqtt/input.go create mode 100644 filebeat/input/mqtt/input_test.go create mode 100644 filebeat/input/mqtt/logging.go create mode 100644 filebeat/input/mqtt/mqtt_integration_test.go create mode 100644 testing/environments/docker/mosquitto/Dockerfile create mode 100644 vendor/github.com/eclipse/paho.mqtt.golang/CONTRIBUTING.md create mode 100644 vendor/github.com/eclipse/paho.mqtt.golang/DISTRIBUTION create mode 100644 vendor/github.com/eclipse/paho.mqtt.golang/LICENSE create mode 100644 vendor/github.com/eclipse/paho.mqtt.golang/README.md create mode 100644 vendor/github.com/eclipse/paho.mqtt.golang/about.html create mode 100644 vendor/github.com/eclipse/paho.mqtt.golang/client.go create mode 100644 vendor/github.com/eclipse/paho.mqtt.golang/components.go create mode 100644 vendor/github.com/eclipse/paho.mqtt.golang/edl-v10 create mode 100644 vendor/github.com/eclipse/paho.mqtt.golang/epl-v10 create mode 100644 vendor/github.com/eclipse/paho.mqtt.golang/filestore.go create mode 100644 vendor/github.com/eclipse/paho.mqtt.golang/memstore.go create mode 100644 vendor/github.com/eclipse/paho.mqtt.golang/message.go create mode 100644 vendor/github.com/eclipse/paho.mqtt.golang/messageids.go create mode 100644 vendor/github.com/eclipse/paho.mqtt.golang/net.go create mode 100644 vendor/github.com/eclipse/paho.mqtt.golang/notice.html create mode 100644 vendor/github.com/eclipse/paho.mqtt.golang/oops.go create mode 100644 vendor/github.com/eclipse/paho.mqtt.golang/options.go create mode 100644 vendor/github.com/eclipse/paho.mqtt.golang/options_reader.go create mode 100644 vendor/github.com/eclipse/paho.mqtt.golang/packets/connack.go create mode 100644 vendor/github.com/eclipse/paho.mqtt.golang/packets/connect.go create mode 100644 vendor/github.com/eclipse/paho.mqtt.golang/packets/disconnect.go create mode 100644 vendor/github.com/eclipse/paho.mqtt.golang/packets/packets.go create mode 100644 vendor/github.com/eclipse/paho.mqtt.golang/packets/pingreq.go create mode 100644 vendor/github.com/eclipse/paho.mqtt.golang/packets/pingresp.go create mode 100644 vendor/github.com/eclipse/paho.mqtt.golang/packets/puback.go create mode 100644 vendor/github.com/eclipse/paho.mqtt.golang/packets/pubcomp.go create mode 100644 vendor/github.com/eclipse/paho.mqtt.golang/packets/publish.go create mode 100644 vendor/github.com/eclipse/paho.mqtt.golang/packets/pubrec.go create mode 100644 vendor/github.com/eclipse/paho.mqtt.golang/packets/pubrel.go create mode 100644 vendor/github.com/eclipse/paho.mqtt.golang/packets/suback.go create mode 100644 vendor/github.com/eclipse/paho.mqtt.golang/packets/subscribe.go create mode 100644 vendor/github.com/eclipse/paho.mqtt.golang/packets/unsuback.go create mode 100644 vendor/github.com/eclipse/paho.mqtt.golang/packets/unsubscribe.go create mode 100644 vendor/github.com/eclipse/paho.mqtt.golang/ping.go create mode 100644 vendor/github.com/eclipse/paho.mqtt.golang/router.go create mode 100644 vendor/github.com/eclipse/paho.mqtt.golang/store.go create mode 100644 vendor/github.com/eclipse/paho.mqtt.golang/token.go create mode 100644 vendor/github.com/eclipse/paho.mqtt.golang/topic.go create mode 100644 vendor/github.com/eclipse/paho.mqtt.golang/trace.go create mode 100644 vendor/github.com/eclipse/paho.mqtt.golang/websocket.go create mode 100644 vendor/github.com/gorilla/websocket/AUTHORS create mode 100644 vendor/github.com/gorilla/websocket/LICENSE create mode 100644 vendor/github.com/gorilla/websocket/README.md create mode 100644 vendor/github.com/gorilla/websocket/client.go create mode 100644 vendor/github.com/gorilla/websocket/client_clone.go create mode 100644 vendor/github.com/gorilla/websocket/client_clone_legacy.go create mode 100644 vendor/github.com/gorilla/websocket/compression.go create mode 100644 vendor/github.com/gorilla/websocket/conn.go create mode 100644 vendor/github.com/gorilla/websocket/conn_write.go create mode 100644 vendor/github.com/gorilla/websocket/conn_write_legacy.go create mode 100644 vendor/github.com/gorilla/websocket/doc.go create mode 100644 vendor/github.com/gorilla/websocket/go.mod create mode 100644 vendor/github.com/gorilla/websocket/go.sum create mode 100644 vendor/github.com/gorilla/websocket/join.go create mode 100644 vendor/github.com/gorilla/websocket/json.go create mode 100644 vendor/github.com/gorilla/websocket/mask.go create mode 100644 vendor/github.com/gorilla/websocket/mask_safe.go create mode 100644 vendor/github.com/gorilla/websocket/prepared.go create mode 100644 vendor/github.com/gorilla/websocket/proxy.go create mode 100644 vendor/github.com/gorilla/websocket/server.go create mode 100644 vendor/github.com/gorilla/websocket/trace.go create mode 100644 vendor/github.com/gorilla/websocket/trace_17.go create mode 100644 vendor/github.com/gorilla/websocket/util.go create mode 100644 vendor/github.com/gorilla/websocket/x_net_proxy.go create mode 100644 vendor/google.golang.org/appengine/datastore/datastore.go create mode 100644 vendor/google.golang.org/appengine/datastore/doc.go create mode 100644 vendor/google.golang.org/appengine/datastore/internal/cloudkey/cloudkey.go create mode 100644 vendor/google.golang.org/appengine/datastore/internal/cloudpb/entity.pb.go create mode 100644 vendor/google.golang.org/appengine/datastore/key.go create mode 100644 vendor/google.golang.org/appengine/datastore/keycompat.go create mode 100644 vendor/google.golang.org/appengine/datastore/load.go create mode 100644 vendor/google.golang.org/appengine/datastore/metadata.go create mode 100644 vendor/google.golang.org/appengine/datastore/prop.go create mode 100644 vendor/google.golang.org/appengine/datastore/query.go create mode 100644 vendor/google.golang.org/appengine/datastore/save.go create mode 100644 vendor/google.golang.org/appengine/datastore/transaction.go create mode 100644 x-pack/metricbeat/module/istio/_meta/Dockerfile create mode 100644 x-pack/metricbeat/module/istio/docker-compose.yml diff --git a/CHANGELOG.next.asciidoc b/CHANGELOG.next.asciidoc index 2d2e1c5646e..9c0629728e5 100644 --- a/CHANGELOG.next.asciidoc +++ b/CHANGELOG.next.asciidoc @@ -190,6 +190,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d - Add ECS tls fields to zeek:smtp,rdp,ssl and aws:s3access,elb {issue}15757[15757] {pull}15935[15936] - move create-[module,fileset,fields] to mage and enable in x-pack/filebeat {pull}15836[15836] - Add ECS tls and categorization fields to apache module. {issue}16032[16032] {pull}16121[16121] +- Add MQTT input. {issue}15602[15602] {pull}16204[16204] *Heartbeat* diff --git a/NOTICE.txt b/NOTICE.txt index d4413b086bb..bfe230e4640 100644 --- a/NOTICE.txt +++ b/NOTICE.txt @@ -1289,6 +1289,99 @@ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. -------------------------------------------------------------------- +Dependency: github.com/eclipse/paho.mqtt.golang +Revision: 0d940dd29fd24f905cd16b28b1209b4977b97e1a +License type (autodetected): EPL-1.0 +./vendor/github.com/eclipse/paho.mqtt.golang/LICENSE: +-------------------------------------------------------------------- +Eclipse Public License - v 1.0 + +THE ACCOMPANYING PROGRAM IS PROVIDED UNDER THE TERMS OF THIS ECLIPSE PUBLIC LICENSE ("AGREEMENT"). ANY USE, REPRODUCTION OR DISTRIBUTION OF THE PROGRAM CONSTITUTES RECIPIENT'S ACCEPTANCE OF THIS AGREEMENT. + +1. DEFINITIONS + +"Contribution" means: + +a) in the case of the initial Contributor, the initial code and documentation distributed under this Agreement, and + +b) in the case of each subsequent Contributor: + +i) changes to the Program, and + +ii) additions to the Program; + +where such changes and/or additions to the Program originate from and are distributed by that particular Contributor. A Contribution 'originates' from a Contributor if it was added to the Program by such Contributor itself or anyone acting on such Contributor's behalf. Contributions do not include additions to the Program which: (i) are separate modules of software distributed in conjunction with the Program under their own license agreement, and (ii) are not derivative works of the Program. + +"Contributor" means any person or entity that distributes the Program. + +"Licensed Patents" mean patent claims licensable by a Contributor which are necessarily infringed by the use or sale of its Contribution alone or when combined with the Program. + +"Program" means the Contributions distributed in accordance with this Agreement. + +"Recipient" means anyone who receives the Program under this Agreement, including all Contributors. + +2. GRANT OF RIGHTS + +a) Subject to the terms of this Agreement, each Contributor hereby grants Recipient a non-exclusive, worldwide, royalty-free copyright license to reproduce, prepare derivative works of, publicly display, publicly perform, distribute and sublicense the Contribution of such Contributor, if any, and such derivative works, in source code and object code form. + +b) Subject to the terms of this Agreement, each Contributor hereby grants Recipient a non-exclusive, worldwide, royalty-free patent license under Licensed Patents to make, use, sell, offer to sell, import and otherwise transfer the Contribution of such Contributor, if any, in source code and object code form. This patent license shall apply to the combination of the Contribution and the Program if, at the time the Contribution is added by the Contributor, such addition of the Contribution causes such combination to be covered by the Licensed Patents. The patent license shall not apply to any other combinations which include the Contribution. No hardware per se is licensed hereunder. + +c) Recipient understands that although each Contributor grants the licenses to its Contributions set forth herein, no assurances are provided by any Contributor that the Program does not infringe the patent or other intellectual property rights of any other entity. Each Contributor disclaims any liability to Recipient for claims brought by any other entity based on infringement of intellectual property rights or otherwise. As a condition to exercising the rights and licenses granted hereunder, each Recipient hereby assumes sole responsibility to secure any other intellectual property rights needed, if any. For example, if a third party patent license is required to allow Recipient to distribute the Program, it is Recipient's responsibility to acquire that license before distributing the Program. + +d) Each Contributor represents that to its knowledge it has sufficient copyright rights in its Contribution, if any, to grant the copyright license set forth in this Agreement. + +3. REQUIREMENTS + +A Contributor may choose to distribute the Program in object code form under its own license agreement, provided that: + +a) it complies with the terms and conditions of this Agreement; and + +b) its license agreement: + +i) effectively disclaims on behalf of all Contributors all warranties and conditions, express and implied, including warranties or conditions of title and non-infringement, and implied warranties or conditions of merchantability and fitness for a particular purpose; + +ii) effectively excludes on behalf of all Contributors all liability for damages, including direct, indirect, special, incidental and consequential damages, such as lost profits; + +iii) states that any provisions which differ from this Agreement are offered by that Contributor alone and not by any other party; and + +iv) states that source code for the Program is available from such Contributor, and informs licensees how to obtain it in a reasonable manner on or through a medium customarily used for software exchange. + +When the Program is made available in source code form: + +a) it must be made available under this Agreement; and + +b) a copy of this Agreement must be included with each copy of the Program. + +Contributors may not remove or alter any copyright notices contained within the Program. + +Each Contributor must identify itself as the originator of its Contribution, if any, in a manner that reasonably allows subsequent Recipients to identify the originator of the Contribution. + +4. COMMERCIAL DISTRIBUTION + +Commercial distributors of software may accept certain responsibilities with respect to end users, business partners and the like. While this license is intended to facilitate the commercial use of the Program, the Contributor who includes the Program in a commercial product offering should do so in a manner which does not create potential liability for other Contributors. Therefore, if a Contributor includes the Program in a commercial product offering, such Contributor ("Commercial Contributor") hereby agrees to defend and indemnify every other Contributor ("Indemnified Contributor") against any losses, damages and costs (collectively "Losses") arising from claims, lawsuits and other legal actions brought by a third party against the Indemnified Contributor to the extent caused by the acts or omissions of such Commercial Contributor in connection with its distribution of the Program in a commercial product offering. The obligations in this section do not apply to any claims or Losses relating to any actual or alleged intellectual property infringement. In order to qualify, an Indemnified Contributor must: a) promptly notify the Commercial Contributor in writing of such claim, and b) allow the Commercial Contributor to control, and cooperate with the Commercial Contributor in, the defense and any related settlement negotiations. The Indemnified Contributor may participate in any such claim at its own expense. + +For example, a Contributor might include the Program in a commercial product offering, Product X. That Contributor is then a Commercial Contributor. If that Commercial Contributor then makes performance claims, or offers warranties related to Product X, those performance claims and warranties are such Commercial Contributor's responsibility alone. Under this section, the Commercial Contributor would have to defend claims against the other Contributors related to those performance claims and warranties, and if a court requires any other Contributor to pay any damages as a result, the Commercial Contributor must pay those damages. + +5. NO WARRANTY + +EXCEPT AS EXPRESSLY SET FORTH IN THIS AGREEMENT, THE PROGRAM IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, EITHER EXPRESS OR IMPLIED INCLUDING, WITHOUT LIMITATION, ANY WARRANTIES OR CONDITIONS OF TITLE, NON-INFRINGEMENT, MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Each Recipient is solely responsible for determining the appropriateness of using and distributing the Program and assumes all risks associated with its exercise of rights under this Agreement , including but not limited to the risks and costs of program errors, compliance with applicable laws, damage to or loss of data, programs or equipment, and unavailability or interruption of operations. + +6. DISCLAIMER OF LIABILITY + +EXCEPT AS EXPRESSLY SET FORTH IN THIS AGREEMENT, NEITHER RECIPIENT NOR ANY CONTRIBUTORS SHALL HAVE ANY LIABILITY FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING WITHOUT LIMITATION LOST PROFITS), HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OR DISTRIBUTION OF THE PROGRAM OR THE EXERCISE OF ANY RIGHTS GRANTED HEREUNDER, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. + +7. GENERAL + +If any provision of this Agreement is invalid or unenforceable under applicable law, it shall not affect the validity or enforceability of the remainder of the terms of this Agreement, and without further action by the parties hereto, such provision shall be reformed to the minimum extent necessary to make such provision valid and enforceable. + +If Recipient institutes patent litigation against any entity (including a cross-claim or counterclaim in a lawsuit) alleging that the Program itself (excluding combinations of the Program with other software or hardware) infringes such Recipient's patent(s), then such Recipient's rights granted under Section 2(b) shall terminate as of the date such litigation is filed. + +All Recipient's rights under this Agreement shall terminate if it fails to comply with any of the material terms or conditions of this Agreement and does not cure such failure in a reasonable period of time after becoming aware of such noncompliance. If all Recipient's rights under this Agreement terminate, Recipient agrees to cease use and distribution of the Program as soon as reasonably practicable. However, Recipient's obligations under this Agreement and any licenses granted by Recipient relating to the Program shall continue and survive. + +Everyone is permitted to copy and distribute copies of this Agreement, but in order to avoid inconsistency the Agreement is copyrighted and may only be modified in the following manner. The Agreement Steward reserves the right to publish new versions (including revisions) of this Agreement from time to time. No one other than the Agreement Steward has the right to modify this Agreement. The Eclipse Foundation is the initial Agreement Steward. The Eclipse Foundation may assign the responsibility to serve as the Agreement Steward to a suitable separate entity. Each new version of the Agreement will be given a distinguishing version number. The Program (including Contributions) may always be distributed subject to the version of the Agreement under which it was received. In addition, after a new version of the Agreement is published, Contributor may elect to distribute the Program (including its Contributions) under the new version. Except as expressly stated in Sections 2(a) and 2(b) above, Recipient receives no rights or licenses to the intellectual property of any Contributor under this Agreement, whether expressly, by implication, estoppel or otherwise. All rights in the Program not expressly granted under this Agreement are reserved. + +This Agreement is governed by the laws of the State of New York and the intellectual property laws of the United States of America. No party to this Agreement will bring a legal action under this Agreement more than one year after the cause of action arose. Each party waives its rights to a jury trial in any resulting litigation. +-------------------------------------------------------------------- Dependency: github.com/elastic/ecs Version: v1.4.0 Revision: cc4b36eebec29975f57cd0475c3987c9bde5c15a @@ -2731,6 +2824,35 @@ License type (autodetected): Apache-2.0 Apache License 2.0 +-------------------------------------------------------------------- +Dependency: github.com/gorilla/websocket +Revision: c3e18be99d19e6b3e8f1559eea2c161a665c4b6b +License type (autodetected): BSD-2-Clause +./vendor/github.com/gorilla/websocket/LICENSE: +-------------------------------------------------------------------- +Copyright (c) 2013 The Gorilla WebSocket Authors. All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are met: + + Redistributions of source code must retain the above copyright notice, this + list of conditions and the following disclaimer. + + Redistributions in binary form must reproduce the above copyright notice, + this list of conditions and the following disclaimer in the documentation + and/or other materials provided with the distribution. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND +ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED +WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE +FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR +SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER +CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, +OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + -------------------------------------------------------------------- Dependency: github.com/grpc-ecosystem/go-grpc-prometheus Revision: ae0d8660c5f2108ca70a3776dbe0fb53cf79f1da diff --git a/dev-tools/generate_notice.py b/dev-tools/generate_notice.py index 645b7ad65ff..ba6d65f0d0c 100644 --- a/dev-tools/generate_notice.py +++ b/dev-tools/generate_notice.py @@ -309,6 +309,10 @@ def create_notice(filename, beat, copyright, vendor_dirs, csvfile, overrides=Non "Creative Commons Attribution-ShareAlike 4.0 International" ] +ECLIPSE_PUBLIC_LICENSE_TITLES = [ + "Eclipse Public License - v 1.0" +] + LGPL_3_LICENSE_TITLE = [ "GNU LESSER GENERAL PUBLIC LICENSE Version 3" ] @@ -348,16 +352,18 @@ def detect_license_summary(content): return "LGPL-3.0" if any(sentence in content[0:1500] for sentence in UNIVERSAL_PERMISSIVE_LICENSE_TITLES): return "UPL-1.0" - + if any(sentence in content[0:1500] for sentence in ECLIPSE_PUBLIC_LICENSE_TITLES): + return "EPL-1.0" return "UNKNOWN" ACCEPTED_LICENSES = [ "Apache-2.0", - "MIT", "BSD-4-Clause", "BSD-3-Clause", "BSD-2-Clause", + "EPL-1.0", + "MIT", "MPL-2.0", "UPL-1.0", ] diff --git a/filebeat/docker-compose.yml b/filebeat/docker-compose.yml index 52437a78a8d..19302ae1e6f 100644 --- a/filebeat/docker-compose.yml +++ b/filebeat/docker-compose.yml @@ -16,6 +16,8 @@ services: - KAFKA_PORT=9092 - KIBANA_HOST=kibana - KIBANA_PORT=5601 + - MOSQUITTO_HOST=mosquitto + - MOSQUITTO_PORT=1883 working_dir: /go/src/github.com/elastic/beats/filebeat volumes: - ${PWD}/..:/go/src/github.com/elastic/beats/ @@ -31,6 +33,7 @@ services: elasticsearch: { condition: service_healthy } kafka: { condition: service_healthy } kibana: { condition: service_healthy } + mosquitto: { condition: service_healthy } redis: { condition: service_healthy } elasticsearch: @@ -51,5 +54,10 @@ services: file: ${ES_BEATS}/testing/environments/${TESTING_ENVIRONMENT}.yml service: kibana + mosquitto: + build: ${ES_BEATS}/testing/environments/docker/mosquitto + expose: + - 1883 + redis: build: ${PWD}/input/redis/_meta diff --git a/filebeat/docs/filebeat-options.asciidoc b/filebeat/docs/filebeat-options.asciidoc index 1611c6b50bb..ca54a8a4ff1 100644 --- a/filebeat/docs/filebeat-options.asciidoc +++ b/filebeat/docs/filebeat-options.asciidoc @@ -51,6 +51,7 @@ You can configure {beatname_uc} to use the following inputs: * <<{beatname_lc}-input-docker>> * <<{beatname_lc}-input-tcp>> * <<{beatname_lc}-input-syslog>> +* <<{beatname_lc}-input-mqtt>> * <<{beatname_lc}-input-s3>> * <<{beatname_lc}-input-netflow>> * <<{beatname_lc}-input-google-pubsub>> @@ -75,6 +76,8 @@ include::inputs/input-tcp.asciidoc[] include::inputs/input-syslog.asciidoc[] +include::inputs/input-mqtt.asciidoc[] + include::../../x-pack/filebeat/docs/inputs/input-aws-s3.asciidoc[] include::../../x-pack/filebeat/docs/inputs/input-netflow.asciidoc[] diff --git a/filebeat/docs/inputs/input-mqtt.asciidoc b/filebeat/docs/inputs/input-mqtt.asciidoc new file mode 100644 index 00000000000..563682abf3c --- /dev/null +++ b/filebeat/docs/inputs/input-mqtt.asciidoc @@ -0,0 +1,81 @@ +:type: mqtt + +[id="{beatname_lc}-input-{type}"] +=== MQTT input + +++++ +MQTT +++++ + +Use the `MQTT` input to read data transmitted using lightweight messaging protocol +for small and mobile devices, optimized for high-latency or unreliable networks. + +This input connects to the MQTT broker, subscribes to selected topics and parses data +into common message lines. Everything happens before line filtering, multiline, and JSON decoding, +so this input can be used in combination with those settings. + +Example configuration: + +["source","yaml",subs="attributes"] +---- +{beatname_lc}.inputs: +- type: mqtt + hosts: <1> + - tcp://broker:1883 + - ssl://secure_broker:8883 + topics: <2> + - sample_topic +---- + +<1> `hosts` are required. + +<2> `paths` are required. + +All other settings are optional. + +==== Configuration options + +The `mqtt` input supports the following configuration options plus the +<<{beatname_lc}-input-{type}-common-options>> described later. + +===== `hosts` + +A list of MQTT brokers to connect to. + +===== `topics` + +A list of topics to subscribe to and read from. + +===== `qos` + +An agreement level between the sender of a message and the receiver of a message that defines the guarantee of delivery. + +There are 3 QoS levels in MQTT: + +* At most once (`0`), +* At least once (`1`), +* Exactly once (`2`). + +===== `client_id` + +A unique identifier of each MQTT client connecting to a MQTT broker. + +===== `username` + +A client username used for authentication provided on the application level by the MQTT protocol. + +===== `password` + +A client password used for authentication provided on the application level by the MQTT protocol. + +===== `ssl` + +Configuration options for SSL parameters like the certificate, key and the certificate authorities +to use. + +See <> for more information. + +[id="{beatname_lc}-input-{type}-common-options"] +include::../inputs/input-common-options.asciidoc[] + +:type!: diff --git a/filebeat/include/list.go b/filebeat/include/list.go index 72d70d9ae24..acaa460cfdf 100644 --- a/filebeat/include/list.go +++ b/filebeat/include/list.go @@ -25,6 +25,7 @@ import ( _ "github.com/elastic/beats/filebeat/input/docker" _ "github.com/elastic/beats/filebeat/input/kafka" _ "github.com/elastic/beats/filebeat/input/log" + _ "github.com/elastic/beats/filebeat/input/mqtt" _ "github.com/elastic/beats/filebeat/input/redis" _ "github.com/elastic/beats/filebeat/input/stdin" _ "github.com/elastic/beats/filebeat/input/syslog" diff --git a/filebeat/input/mqtt/client.go b/filebeat/input/mqtt/client.go new file mode 100644 index 00000000000..73e8d2f5344 --- /dev/null +++ b/filebeat/input/mqtt/client.go @@ -0,0 +1,54 @@ +// Licensed to Elasticsearch B.V. under one or more contributor +// license agreements. See the NOTICE file distributed with +// this work for additional information regarding copyright +// ownership. Elasticsearch B.V. licenses this file to you under +// the Apache License, Version 2.0 (the "License"); you may +// not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package mqtt + +import ( + libmqtt "github.com/eclipse/paho.mqtt.golang" + + "github.com/elastic/beats/libbeat/outputs" +) + +func createClientOptions(config mqttInputConfig, onConnectHandler func(client libmqtt.Client)) (*libmqtt.ClientOptions, error) { + clientOptions := libmqtt.NewClientOptions(). + SetClientID(config.ClientID). + SetUsername(config.Username). + SetPassword(config.Password). + SetConnectRetry(true). + SetOnConnectHandler(onConnectHandler) + + for _, host := range config.Hosts { + clientOptions.AddBroker(host) + } + + if config.TLS != nil { + tlsConfig, err := outputs.LoadTLSConfig(config.TLS) + if err != nil { + return nil, err + } + clientOptions.SetTLSConfig(tlsConfig.BuildModuleConfig("")) + } + return clientOptions, nil +} + +func createClientSubscriptions(config mqttInputConfig) map[string]byte { + subscriptions := map[string]byte{} + for _, topic := range config.Topics { + subscriptions[topic] = byte(config.QoS) + } + return subscriptions +} diff --git a/filebeat/input/mqtt/client_mocked.go b/filebeat/input/mqtt/client_mocked.go new file mode 100644 index 00000000000..dadccdf9356 --- /dev/null +++ b/filebeat/input/mqtt/client_mocked.go @@ -0,0 +1,220 @@ +// Licensed to Elasticsearch B.V. under one or more contributor +// license agreements. See the NOTICE file distributed with +// this work for additional information regarding copyright +// ownership. Elasticsearch B.V. licenses this file to you under +// the Apache License, Version 2.0 (the "License"); you may +// not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package mqtt + +import ( + "time" + + libmqtt "github.com/eclipse/paho.mqtt.golang" + + "github.com/elastic/beats/filebeat/channel" + "github.com/elastic/beats/libbeat/beat" + "github.com/elastic/beats/libbeat/common" + "github.com/elastic/beats/libbeat/common/backoff" +) + +type mockedMessage struct { + duplicate bool + messageID uint16 + qos byte + retained bool + topic string + payload []byte + ack func() +} + +var _ libmqtt.Message = new(mockedMessage) + +func (m *mockedMessage) Duplicate() bool { + return m.duplicate +} + +func (m *mockedMessage) Qos() byte { + return m.qos +} + +func (m *mockedMessage) Retained() bool { + return m.retained +} + +func (m *mockedMessage) Topic() string { + return m.topic +} + +func (m *mockedMessage) MessageID() uint16 { + return m.messageID +} + +func (m *mockedMessage) Payload() []byte { + return m.payload +} + +func (m *mockedMessage) Ack() { + panic("implement me") +} + +type mockedBackoff struct { + resetCount int + + waits []bool + waitIndex int +} + +var _ backoff.Backoff = new(mockedBackoff) + +func (m *mockedBackoff) Wait() bool { + wait := m.waits[m.waitIndex] + m.waitIndex++ + return wait +} + +func (m *mockedBackoff) Reset() { + m.resetCount++ +} + +type mockedToken struct { + timeout bool +} + +var _ libmqtt.Token = new(mockedToken) + +func (m *mockedToken) Wait() bool { + panic("implement me") +} + +func (m *mockedToken) WaitTimeout(time.Duration) bool { + return m.timeout +} + +func (m *mockedToken) Error() error { + return nil +} + +type mockedClient struct { + connectCount int + disconnectCount int + subscribeMultipleCount int + + subscriptions []string + messages []mockedMessage + + tokens []libmqtt.Token + tokenIndex int + + onConnectHandler func(client libmqtt.Client) + onMessageHandler func(client libmqtt.Client, message libmqtt.Message) +} + +var _ libmqtt.Client = new(mockedClient) + +func (m *mockedClient) IsConnected() bool { + panic("implement me") +} + +func (m *mockedClient) IsConnectionOpen() bool { + panic("implement me") +} + +func (m *mockedClient) Connect() libmqtt.Token { + m.connectCount++ + + if m.onConnectHandler != nil { + m.onConnectHandler(m) + } + return nil +} + +func (m *mockedClient) Disconnect(quiesce uint) { + m.disconnectCount++ +} + +func (m *mockedClient) Publish(topic string, qos byte, retained bool, payload interface{}) libmqtt.Token { + panic("implement me") +} + +func (m *mockedClient) Subscribe(topic string, qos byte, callback libmqtt.MessageHandler) libmqtt.Token { + panic("implement me") +} + +func (m *mockedClient) SubscribeMultiple(filters map[string]byte, callback libmqtt.MessageHandler) libmqtt.Token { + m.subscribeMultipleCount++ + + for filter := range filters { + m.subscriptions = append(m.subscriptions, filter) + } + m.onMessageHandler = callback + + for _, msg := range m.messages { + var thatMsg = msg + go func() { + m.onMessageHandler(m, &thatMsg) + }() + } + + token := m.tokens[m.tokenIndex] + m.tokenIndex++ + return token +} + +func (m *mockedClient) Unsubscribe(topics ...string) libmqtt.Token { + panic("implement me") +} + +func (m *mockedClient) AddRoute(topic string, callback libmqtt.MessageHandler) { + panic("implement me") +} + +func (m *mockedClient) OptionsReader() libmqtt.ClientOptionsReader { + panic("implement me") +} + +type mockedConnector struct { + connectWithError error + outlet channel.Outleter +} + +var _ channel.Connector = new(mockedConnector) + +func (m *mockedConnector) Connect(*common.Config) (channel.Outleter, error) { + panic("implement me") +} + +func (m *mockedConnector) ConnectWith(*common.Config, beat.ClientConfig) (channel.Outleter, error) { + if m.connectWithError != nil { + return nil, m.connectWithError + } + return m.outlet, nil +} + +type mockedOutleter struct { + onEventHandler func(event beat.Event) bool +} + +var _ channel.Outleter = new(mockedOutleter) + +func (m mockedOutleter) Close() error { + panic("implement me") +} + +func (m mockedOutleter) Done() <-chan struct{} { + panic("implement me") +} + +func (m mockedOutleter) OnEvent(event beat.Event) bool { + return m.onEventHandler(event) +} diff --git a/filebeat/input/mqtt/config.go b/filebeat/input/mqtt/config.go new file mode 100644 index 00000000000..6a1ed8ca1e9 --- /dev/null +++ b/filebeat/input/mqtt/config.go @@ -0,0 +1,52 @@ +// Licensed to Elasticsearch B.V. under one or more contributor +// license agreements. See the NOTICE file distributed with +// this work for additional information regarding copyright +// ownership. Elasticsearch B.V. licenses this file to you under +// the Apache License, Version 2.0 (the "License"); you may +// not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package mqtt + +import ( + "errors" + + "github.com/elastic/beats/libbeat/common/transport/tlscommon" +) + +type mqttInputConfig struct { + Hosts []string `config:"hosts" validate:"required,min=1"` + Topics []string `config:"topics" validate:"required,min=1"` + QoS int `config:"qos" validate:"min=0,max=2"` + + ClientID string `config:"client_id" validate:"nonzero"` + Username string `config:"username"` + Password string `config:"password"` + + TLS *tlscommon.Config `config:"ssl"` +} + +// The default config for the mqtt input. +func defaultConfig() mqttInputConfig { + return mqttInputConfig{ + ClientID: "filebeat", + Topics: []string{"#"}, + } +} + +// Validate validates the config. +func (mic *mqttInputConfig) Validate() error { + if len(mic.ClientID) < 1 || len(mic.ClientID) > 23 { + return errors.New("ClientID must be between 1 and 23 characters long") + } + return nil +} diff --git a/filebeat/input/mqtt/input.go b/filebeat/input/mqtt/input.go new file mode 100644 index 00000000000..f2edab19b6f --- /dev/null +++ b/filebeat/input/mqtt/input.go @@ -0,0 +1,204 @@ +// Licensed to Elasticsearch B.V. under one or more contributor +// license agreements. See the NOTICE file distributed with +// this work for additional information regarding copyright +// ownership. Elasticsearch B.V. licenses this file to you under +// the Apache License, Version 2.0 (the "License"); you may +// not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package mqtt + +import ( + "strings" + "sync" + "time" + + libmqtt "github.com/eclipse/paho.mqtt.golang" + "github.com/pkg/errors" + + "github.com/elastic/beats/filebeat/channel" + "github.com/elastic/beats/filebeat/input" + "github.com/elastic/beats/libbeat/beat" + "github.com/elastic/beats/libbeat/common" + "github.com/elastic/beats/libbeat/common/backoff" + "github.com/elastic/beats/libbeat/logp" +) + +const ( + disconnectTimeout = 3 * time.Second + + subscribeTimeout = 35 * time.Second // in client: subscribeWaitTimeout = 30s + subscribeRetryInterval = 1 * time.Second +) + +// Input contains the input and its config +type mqttInput struct { + once sync.Once + + logger *logp.Logger + + client libmqtt.Client + clientDisconnected *sync.WaitGroup + inflightMessages *sync.WaitGroup +} + +func init() { + err := input.Register("mqtt", NewInput) + if err != nil { + panic(err) + } +} + +// NewInput method creates a new mqtt input, +func NewInput( + cfg *common.Config, + connector channel.Connector, + inputContext input.Context, +) (input.Input, error) { + return newInput(cfg, connector, inputContext, libmqtt.NewClient, backoff.NewEqualJitterBackoff) +} + +func newInput( + cfg *common.Config, + connector channel.Connector, + inputContext input.Context, + newMqttClient func(options *libmqtt.ClientOptions) libmqtt.Client, + newBackoff func(done <-chan struct{}, init, max time.Duration) backoff.Backoff, +) (input.Input, error) { + config := defaultConfig() + if err := cfg.Unpack(&config); err != nil { + return nil, errors.Wrap(err, "reading mqtt input config") + } + + out, err := connector.ConnectWith(cfg, beat.ClientConfig{ + Processing: beat.ProcessingConfig{ + DynamicFields: inputContext.DynamicFields, + }, + }) + if err != nil { + return nil, err + } + + logger := logp.NewLogger("mqtt input").With("hosts", config.Hosts) + setupLibraryLogging() + + clientDisconnected := new(sync.WaitGroup) + inflightMessages := new(sync.WaitGroup) + clientSubscriptions := createClientSubscriptions(config) + onMessageHandler := createOnMessageHandler(logger, out, inflightMessages) + onConnectHandler := createOnConnectHandler(logger, &inputContext, onMessageHandler, clientSubscriptions, newBackoff) + clientOptions, err := createClientOptions(config, onConnectHandler) + if err != nil { + return nil, err + } + + return &mqttInput{ + client: newMqttClient(clientOptions), + clientDisconnected: clientDisconnected, + inflightMessages: inflightMessages, + logger: logp.NewLogger("mqtt input").With("hosts", config.Hosts), + }, nil +} + +func createOnMessageHandler(logger *logp.Logger, outlet channel.Outleter, inflightMessages *sync.WaitGroup) func(client libmqtt.Client, message libmqtt.Message) { + return func(client libmqtt.Client, message libmqtt.Message) { + inflightMessages.Add(1) + + logger.Debugf("Received message on topic '%s', messageID: %d, size: %d", message.Topic(), + message.MessageID(), len(message.Payload())) + + mqttFields := common.MapStr{ + "duplicate": message.Duplicate(), + "message_id": message.MessageID(), + "qos": message.Qos(), + "retained": message.Retained(), + "topic": message.Topic(), + } + outlet.OnEvent(beat.Event{ + Timestamp: time.Now(), + Fields: common.MapStr{ + "message": string(message.Payload()), + "mqtt": mqttFields, + }, + }) + + inflightMessages.Done() + } +} + +func createOnConnectHandler(logger *logp.Logger, + inputContext *input.Context, + onMessageHandler func(client libmqtt.Client, message libmqtt.Message), + clientSubscriptions map[string]byte, + newBackoff func(done <-chan struct{}, init, max time.Duration) backoff.Backoff) func(client libmqtt.Client) { + // The function subscribes the client to the specific topics (with retry backoff in case of failure). + return func(client libmqtt.Client) { + backoff := newBackoff( + inputContext.Done, + subscribeRetryInterval, + 8*subscribeRetryInterval) + + var topics []string + for topic := range clientSubscriptions { + topics = append(topics, topic) + } + + var success bool + for !success { + logger.Debugf("Try subscribe to topics: %v", strings.Join(topics, ", ")) + + token := client.SubscribeMultiple(clientSubscriptions, onMessageHandler) + if !token.WaitTimeout(subscribeTimeout) { + if token.Error() != nil { + logger.Warnf("Subscribing to topics failed due to error: %v", token.Error()) + } + + if !backoff.Wait() { + backoff.Reset() + success = true + } + } else { + backoff.Reset() + success = true + } + } + } +} + +// Run method starts the mqtt input and processing. +// The mqtt client starts in auto-connect mode (with connection retries and resuming topic subscriptions). +func (mi *mqttInput) Run() { + mi.once.Do(func() { + mi.logger.Debug("Run the input once.") + mi.client.Connect() + }) +} + +// Stop method stops the input. +func (mi *mqttInput) Stop() { + mi.logger.Debug("Stop the input.") + + mi.clientDisconnected.Add(1) + go func() { + mi.client.Disconnect(uint(disconnectTimeout.Milliseconds())) + mi.clientDisconnected.Done() + }() +} + +// Wait method stops the input and waits until event processing is finished. +func (mi *mqttInput) Wait() { + mi.logger.Debug("Wait for the input to finish processing.") + + mi.Stop() + mi.clientDisconnected.Wait() + mi.inflightMessages.Wait() +} diff --git a/filebeat/input/mqtt/input_test.go b/filebeat/input/mqtt/input_test.go new file mode 100644 index 00000000000..3e54e17f118 --- /dev/null +++ b/filebeat/input/mqtt/input_test.go @@ -0,0 +1,349 @@ +// Licensed to Elasticsearch B.V. under one or more contributor +// license agreements. See the NOTICE file distributed with +// this work for additional information regarding copyright +// ownership. Elasticsearch B.V. licenses this file to you under +// the Apache License, Version 2.0 (the "License"); you may +// not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package mqtt + +import ( + "errors" + "sync" + "testing" + "time" + + libmqtt "github.com/eclipse/paho.mqtt.golang" + "github.com/stretchr/testify/require" + + finput "github.com/elastic/beats/filebeat/input" + "github.com/elastic/beats/libbeat/beat" + "github.com/elastic/beats/libbeat/common" + "github.com/elastic/beats/libbeat/common/backoff" + "github.com/elastic/beats/libbeat/logp" +) + +var logger = logp.NewLogger("test") + +func TestNewInput_MissingConfigField(t *testing.T) { + config := common.MustNewConfigFrom(common.MapStr{ + "topics": "#", + }) + connector := new(mockedConnector) + var inputContext finput.Context + + input, err := NewInput(config, connector, inputContext) + + require.Error(t, err) + require.Nil(t, input) +} + +func TestNewInput_ConnectWithFailed(t *testing.T) { + connectWithError := errors.New("failure") + config := common.MustNewConfigFrom(common.MapStr{ + "hosts": "tcp://mocked:1234", + "topics": "#", + }) + connector := &mockedConnector{ + connectWithError: connectWithError, + } + var inputContext finput.Context + + input, err := NewInput(config, connector, inputContext) + + require.Equal(t, connectWithError, err) + require.Nil(t, input) +} + +func TestNewInput_Run(t *testing.T) { + config := common.MustNewConfigFrom(common.MapStr{ + "hosts": "tcp://mocked:1234", + "topics": []string{"first", "second"}, + "qos": 2, + }) + + eventsCh := make(chan beat.Event) + defer close(eventsCh) + + outlet := &mockedOutleter{ + onEventHandler: func(event beat.Event) bool { + eventsCh <- event + return true + }, + } + connector := &mockedConnector{ + outlet: outlet, + } + var inputContext finput.Context + + firstMessage := mockedMessage{ + duplicate: false, + messageID: 1, + qos: 2, + retained: false, + topic: "first", + payload: []byte("first-message"), + } + secondMessage := mockedMessage{ + duplicate: false, + messageID: 2, + qos: 2, + retained: false, + topic: "second", + payload: []byte("second-message"), + } + + var client *mockedClient + newMqttClient := func(o *libmqtt.ClientOptions) libmqtt.Client { + client = &mockedClient{ + onConnectHandler: o.OnConnect, + messages: []mockedMessage{firstMessage, secondMessage}, + tokens: []libmqtt.Token{&mockedToken{ + timeout: true, + }}, + } + return client + } + + input, err := newInput(config, connector, inputContext, newMqttClient, backoff.NewEqualJitterBackoff) + require.NoError(t, err) + require.NotNil(t, input) + + input.Run() + + require.Equal(t, 1, client.connectCount) + require.Equal(t, 1, client.subscribeMultipleCount) + require.ElementsMatch(t, []string{"first", "second"}, client.subscriptions) + + for _, event := range []beat.Event{<-eventsCh, <-eventsCh} { + topic, err := event.GetValue("mqtt.topic") + require.NoError(t, err) + + if topic == "first" { + assertEventMatches(t, firstMessage, event) + } else { + assertEventMatches(t, secondMessage, event) + } + } +} + +func TestNewInput_Run_Wait(t *testing.T) { + config := common.MustNewConfigFrom(common.MapStr{ + "hosts": "tcp://mocked:1234", + "topics": []string{"first", "second"}, + "qos": 2, + }) + + const numMessages = 5 + + var eventProcessing sync.WaitGroup + eventProcessing.Add(numMessages) + + eventsCh := make(chan beat.Event) + defer close(eventsCh) + + outlet := &mockedOutleter{ + onEventHandler: func(event beat.Event) bool { + eventProcessing.Done() + eventsCh <- event + return true + }, + } + connector := &mockedConnector{ + outlet: outlet, + } + var inputContext finput.Context + + var messages []mockedMessage + for i := 0; i < numMessages; i++ { + messages = append(messages, mockedMessage{ + duplicate: false, + messageID: 1, + qos: 2, + retained: false, + topic: "topic", + payload: []byte("a-message"), + }) + } + + var client *mockedClient + newMqttClient := func(o *libmqtt.ClientOptions) libmqtt.Client { + client = &mockedClient{ + onConnectHandler: o.OnConnect, + messages: messages, + tokens: []libmqtt.Token{&mockedToken{ + timeout: true, + }}, + } + return client + } + + input, err := newInput(config, connector, inputContext, newMqttClient, backoff.NewEqualJitterBackoff) + require.NoError(t, err) + require.NotNil(t, input) + + input.Run() + eventProcessing.Wait() + + go func() { + time.Sleep(100 * time.Millisecond) // let input.Stop() be executed. + for range eventsCh { + } + }() + + input.Wait() +} + +func TestRun_Once(t *testing.T) { + client := new(mockedClient) + input := &mqttInput{ + client: client, + logger: logger, + } + + input.Run() + + require.Equal(t, 1, client.connectCount) +} + +func TestRun_Twice(t *testing.T) { + client := new(mockedClient) + input := &mqttInput{ + client: client, + logger: logger, + } + + input.Run() + input.Run() + + require.Equal(t, 1, client.connectCount) +} + +func TestWait(t *testing.T) { + clientDisconnected := new(sync.WaitGroup) + inflightMessages := new(sync.WaitGroup) + client := new(mockedClient) + input := &mqttInput{ + client: client, + clientDisconnected: clientDisconnected, + logger: logger, + inflightMessages: inflightMessages, + } + + input.Wait() + + require.Equal(t, 1, client.disconnectCount) +} + +func TestStop(t *testing.T) { + client := new(mockedClient) + clientDisconnected := new(sync.WaitGroup) + input := &mqttInput{ + client: client, + clientDisconnected: clientDisconnected, + logger: logger, + } + + input.Stop() +} + +func TestOnCreateHandler_SubscribeMultiple_Succeeded(t *testing.T) { + inputContext := new(finput.Context) + onMessageHandler := func(client libmqtt.Client, message libmqtt.Message) {} + var clientSubscriptions map[string]byte + newBackoff := func(done <-chan struct{}, init, max time.Duration) backoff.Backoff { + return backoff.NewEqualJitterBackoff(inputContext.Done, time.Nanosecond, 2*time.Nanosecond) + } + handler := createOnConnectHandler(logger, inputContext, onMessageHandler, clientSubscriptions, newBackoff) + + client := &mockedClient{ + tokens: []libmqtt.Token{&mockedToken{ + timeout: true, + }}, + } + handler(client) + + require.Equal(t, 1, client.subscribeMultipleCount) +} + +func TestOnCreateHandler_SubscribeMultiple_BackoffSucceeded(t *testing.T) { + inputContext := new(finput.Context) + onMessageHandler := func(client libmqtt.Client, message libmqtt.Message) {} + var clientSubscriptions map[string]byte + newBackoff := func(done <-chan struct{}, init, max time.Duration) backoff.Backoff { + return backoff.NewEqualJitterBackoff(inputContext.Done, time.Nanosecond, 2*time.Nanosecond) + } + handler := createOnConnectHandler(logger, inputContext, onMessageHandler, clientSubscriptions, newBackoff) + + client := &mockedClient{ + tokens: []libmqtt.Token{&mockedToken{ + timeout: false, + }, &mockedToken{ + timeout: true, + }}, + } + handler(client) + + require.Equal(t, 2, client.subscribeMultipleCount) +} + +func TestOnCreateHandler_SubscribeMultiple_BackoffSignalDone(t *testing.T) { + inputContext := new(finput.Context) + onMessageHandler := func(client libmqtt.Client, message libmqtt.Message) {} + var clientSubscriptions map[string]byte + mockedBackoff := &mockedBackoff{ + waits: []bool{true, false}, + } + newBackoff := func(done <-chan struct{}, init, max time.Duration) backoff.Backoff { + return mockedBackoff + } + handler := createOnConnectHandler(logger, inputContext, onMessageHandler, clientSubscriptions, newBackoff) + + client := &mockedClient{ + tokens: []libmqtt.Token{&mockedToken{ + timeout: false, + }, &mockedToken{ + timeout: false, + }}, + } + handler(client) + + require.Equal(t, 2, client.subscribeMultipleCount) + require.Equal(t, 1, mockedBackoff.resetCount) +} + +func assertEventMatches(t *testing.T, expected mockedMessage, got beat.Event) { + topic, err := got.GetValue("mqtt.topic") + require.NoError(t, err) + require.Equal(t, expected.topic, topic) + + duplicate, err := got.GetValue("mqtt.duplicate") + require.NoError(t, err) + require.Equal(t, expected.duplicate, duplicate) + + messageID, err := got.GetValue("mqtt.message_id") + require.NoError(t, err) + require.Equal(t, expected.messageID, messageID) + + qos, err := got.GetValue("mqtt.qos") + require.NoError(t, err) + require.Equal(t, expected.qos, qos) + + retained, err := got.GetValue("mqtt.retained") + require.NoError(t, err) + require.Equal(t, expected.retained, retained) + + message, err := got.GetValue("message") + require.NoError(t, err) + require.Equal(t, string(expected.payload), message) +} diff --git a/filebeat/input/mqtt/logging.go b/filebeat/input/mqtt/logging.go new file mode 100644 index 00000000000..35510c20b34 --- /dev/null +++ b/filebeat/input/mqtt/logging.go @@ -0,0 +1,79 @@ +// Licensed to Elasticsearch B.V. under one or more contributor +// license agreements. See the NOTICE file distributed with +// this work for additional information regarding copyright +// ownership. Elasticsearch B.V. licenses this file to you under +// the Apache License, Version 2.0 (the "License"); you may +// not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package mqtt + +import ( + "sync" + + libmqtt "github.com/eclipse/paho.mqtt.golang" + "go.uber.org/zap" + + "github.com/elastic/beats/libbeat/logp" +) + +var setupLoggingOnce sync.Once + +type loggerWrapper struct { + log *logp.Logger +} + +type ( + debugLogger loggerWrapper + errorLogger loggerWrapper + warnLogger loggerWrapper +) + +var ( + _ libmqtt.Logger = new(debugLogger) + _ libmqtt.Logger = new(errorLogger) + _ libmqtt.Logger = new(warnLogger) +) + +func setupLibraryLogging() { + setupLoggingOnce.Do(func() { + logger := logp.NewLogger("libmqtt", zap.AddCallerSkip(1)) + libmqtt.CRITICAL = &errorLogger{log: logger} + libmqtt.DEBUG = &debugLogger{log: logger} + libmqtt.ERROR = &errorLogger{log: logger} + libmqtt.WARN = &warnLogger{log: logger} + }) +} + +func (l *debugLogger) Println(v ...interface{}) { + l.log.Debug(v...) +} + +func (l *debugLogger) Printf(format string, v ...interface{}) { + l.log.Debugf(format, v...) +} + +func (l *errorLogger) Println(v ...interface{}) { + l.log.Error(v...) +} + +func (l *errorLogger) Printf(format string, v ...interface{}) { + l.log.Errorf(format, v...) +} + +func (l *warnLogger) Println(v ...interface{}) { + l.log.Warn(v...) +} + +func (l *warnLogger) Printf(format string, v ...interface{}) { + l.log.Warnf(format, v...) +} diff --git a/filebeat/input/mqtt/mqtt_integration_test.go b/filebeat/input/mqtt/mqtt_integration_test.go new file mode 100644 index 00000000000..3fd3506cbf4 --- /dev/null +++ b/filebeat/input/mqtt/mqtt_integration_test.go @@ -0,0 +1,170 @@ +// Licensed to Elasticsearch B.V. under one or more contributor +// license agreements. See the NOTICE file distributed with +// this work for additional information regarding copyright +// ownership. Elasticsearch B.V. licenses this file to you under +// the Apache License, Version 2.0 (the "License"); you may +// not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +// +build integration + +package mqtt + +import ( + "fmt" + "os" + "sync" + "testing" + "time" + + libmqtt "github.com/eclipse/paho.mqtt.golang" + "github.com/stretchr/testify/require" + + "github.com/elastic/beats/filebeat/channel" + "github.com/elastic/beats/filebeat/input" + "github.com/elastic/beats/libbeat/beat" + "github.com/elastic/beats/libbeat/common" + "github.com/elastic/beats/libbeat/logp" +) + +const ( + message = "hello-world" + + waitTimeout = 30 * time.Second +) + +var ( + hostPort = fmt.Sprintf("tcp://%s:%s", + getOrDefault(os.Getenv("MOSQUITTO_HOST"), "mosquitto"), + getOrDefault(os.Getenv("MOSQUITTO_PORT"), "1883")) + topic = fmt.Sprintf("topic-%d", time.Now().UnixNano()) +) + +type eventCaptor struct { + c chan struct{} + closeOnce sync.Once + closed bool + events chan beat.Event +} + +func newEventCaptor(events chan beat.Event) channel.Outleter { + return &eventCaptor{ + c: make(chan struct{}), + events: events, + } +} + +func (ec *eventCaptor) OnEvent(event beat.Event) bool { + ec.events <- event + return true +} + +func (ec *eventCaptor) Close() error { + ec.closeOnce.Do(func() { + ec.closed = true + close(ec.c) + }) + return nil +} + +func (ec *eventCaptor) Done() <-chan struct{} { + return ec.c +} + +func TestInput(t *testing.T) { + logp.TestingSetup(logp.WithSelectors("mqtt input", "libmqtt")) + + // Setup the input config. + config := common.MustNewConfigFrom(common.MapStr{ + "hosts": []string{hostPort}, + "topics": []string{topic}, + }) + + // Route input events through our captor instead of sending through ES. + eventsCh := make(chan beat.Event) + defer close(eventsCh) + + captor := newEventCaptor(eventsCh) + defer captor.Close() + + connector := channel.ConnectorFunc(func(_ *common.Config, _ beat.ClientConfig) (channel.Outleter, error) { + return channel.SubOutlet(captor), nil + }) + + // Mock the context. + inputContext := input.Context{ + Done: make(chan struct{}), + BeatDone: make(chan struct{}), + } + + // Setup the input + input, err := NewInput(config, connector, inputContext) + require.NoError(t, err) + require.NotNil(t, input) + + // Run the input. + input.Run() + + // Create Publisher + publisher := createPublisher(t) + + // Verify that event has been received + verifiedCh := make(chan struct{}) + defer close(verifiedCh) + + emitInputData(t, verifiedCh, publisher) + + event := <-eventsCh + verifiedCh <- struct{}{} + + val, err := event.GetValue("message") + require.NoError(t, err) + require.Equal(t, message, val) +} + +func createPublisher(t *testing.T) libmqtt.Client { + clientOptions := libmqtt.NewClientOptions(). + SetClientID("emitter"). + SetAutoReconnect(false). + SetConnectRetry(false). + AddBroker(hostPort) + client := libmqtt.NewClient(clientOptions) + token := client.Connect() + require.True(t, token.WaitTimeout(waitTimeout)) + require.NoError(t, token.Error()) + return client +} + +func emitInputData(t *testing.T, verifiedCh <-chan struct{}, publisher libmqtt.Client) { + go func() { + ticker := time.NewTicker(time.Second) + defer ticker.Stop() + + for { + select { + case <-verifiedCh: + return + case <-ticker.C: + token := publisher.Publish(topic, 1, false, []byte(message)) + require.True(t, token.WaitTimeout(waitTimeout)) + require.NoError(t, token.Error()) + } + } + }() +} + +func getOrDefault(s, defaultString string) string { + if s == "" { + return defaultString + } + return s +} diff --git a/testing/environments/docker/mosquitto/Dockerfile b/testing/environments/docker/mosquitto/Dockerfile new file mode 100644 index 00000000000..eac5d1e0d6c --- /dev/null +++ b/testing/environments/docker/mosquitto/Dockerfile @@ -0,0 +1,2 @@ +FROM eclipse-mosquitto:1.6.8 +HEALTHCHECK --interval=1s --retries=600 CMD nc -z localhost 1883 diff --git a/vendor/github.com/eclipse/paho.mqtt.golang/CONTRIBUTING.md b/vendor/github.com/eclipse/paho.mqtt.golang/CONTRIBUTING.md new file mode 100644 index 00000000000..9791dc60318 --- /dev/null +++ b/vendor/github.com/eclipse/paho.mqtt.golang/CONTRIBUTING.md @@ -0,0 +1,56 @@ +Contributing to Paho +==================== + +Thanks for your interest in this project. + +Project description: +-------------------- + +The Paho project has been created to provide scalable open-source implementations of open and standard messaging protocols aimed at new, existing, and emerging applications for Machine-to-Machine (M2M) and Internet of Things (IoT). +Paho reflects the inherent physical and cost constraints of device connectivity. Its objectives include effective levels of decoupling between devices and applications, designed to keep markets open and encourage the rapid growth of scalable Web and Enterprise middleware and applications. Paho is being kicked off with MQTT publish/subscribe client implementations for use on embedded platforms, along with corresponding server support as determined by the community. + +- https://projects.eclipse.org/projects/technology.paho + +Developer resources: +-------------------- + +Information regarding source code management, builds, coding standards, and more. + +- https://projects.eclipse.org/projects/technology.paho/developer + +Contributor License Agreement: +------------------------------ + +Before your contribution can be accepted by the project, you need to create and electronically sign the Eclipse Foundation Contributor License Agreement (CLA). + +- http://www.eclipse.org/legal/CLA.php + +Contributing Code: +------------------ + +The Go client is developed in Github, see their documentation on the process of forking and pull requests; https://help.github.com/categories/collaborating-on-projects-using-pull-requests/ + +Git commit messages should follow the style described here; + +http://tbaggery.com/2008/04/19/a-note-about-git-commit-messages.html + +Contact: +-------- + +Contact the project developers via the project's "dev" list. + +- https://dev.eclipse.org/mailman/listinfo/paho-dev + +Search for bugs: +---------------- + +This project uses Github issues to track ongoing development and issues. + +- https://github.com/eclipse/paho.mqtt.golang/issues + +Create a new bug: +----------------- + +Be sure to search for existing bugs before you create another one. Remember that contributions are always welcome! + +- https://github.com/eclipse/paho.mqtt.golang/issues diff --git a/vendor/github.com/eclipse/paho.mqtt.golang/DISTRIBUTION b/vendor/github.com/eclipse/paho.mqtt.golang/DISTRIBUTION new file mode 100644 index 00000000000..34e49731daa --- /dev/null +++ b/vendor/github.com/eclipse/paho.mqtt.golang/DISTRIBUTION @@ -0,0 +1,15 @@ + + +Eclipse Distribution License - v 1.0 + +Copyright (c) 2007, Eclipse Foundation, Inc. and its licensors. + +All rights reserved. + +Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: + + Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. + Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. + Neither the name of the Eclipse Foundation, Inc. nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. diff --git a/vendor/github.com/eclipse/paho.mqtt.golang/LICENSE b/vendor/github.com/eclipse/paho.mqtt.golang/LICENSE new file mode 100644 index 00000000000..aa7cc810fa1 --- /dev/null +++ b/vendor/github.com/eclipse/paho.mqtt.golang/LICENSE @@ -0,0 +1,87 @@ +Eclipse Public License - v 1.0 + +THE ACCOMPANYING PROGRAM IS PROVIDED UNDER THE TERMS OF THIS ECLIPSE PUBLIC LICENSE ("AGREEMENT"). ANY USE, REPRODUCTION OR DISTRIBUTION OF THE PROGRAM CONSTITUTES RECIPIENT'S ACCEPTANCE OF THIS AGREEMENT. + +1. DEFINITIONS + +"Contribution" means: + +a) in the case of the initial Contributor, the initial code and documentation distributed under this Agreement, and + +b) in the case of each subsequent Contributor: + +i) changes to the Program, and + +ii) additions to the Program; + +where such changes and/or additions to the Program originate from and are distributed by that particular Contributor. A Contribution 'originates' from a Contributor if it was added to the Program by such Contributor itself or anyone acting on such Contributor's behalf. Contributions do not include additions to the Program which: (i) are separate modules of software distributed in conjunction with the Program under their own license agreement, and (ii) are not derivative works of the Program. + +"Contributor" means any person or entity that distributes the Program. + +"Licensed Patents" mean patent claims licensable by a Contributor which are necessarily infringed by the use or sale of its Contribution alone or when combined with the Program. + +"Program" means the Contributions distributed in accordance with this Agreement. + +"Recipient" means anyone who receives the Program under this Agreement, including all Contributors. + +2. GRANT OF RIGHTS + +a) Subject to the terms of this Agreement, each Contributor hereby grants Recipient a non-exclusive, worldwide, royalty-free copyright license to reproduce, prepare derivative works of, publicly display, publicly perform, distribute and sublicense the Contribution of such Contributor, if any, and such derivative works, in source code and object code form. + +b) Subject to the terms of this Agreement, each Contributor hereby grants Recipient a non-exclusive, worldwide, royalty-free patent license under Licensed Patents to make, use, sell, offer to sell, import and otherwise transfer the Contribution of such Contributor, if any, in source code and object code form. This patent license shall apply to the combination of the Contribution and the Program if, at the time the Contribution is added by the Contributor, such addition of the Contribution causes such combination to be covered by the Licensed Patents. The patent license shall not apply to any other combinations which include the Contribution. No hardware per se is licensed hereunder. + +c) Recipient understands that although each Contributor grants the licenses to its Contributions set forth herein, no assurances are provided by any Contributor that the Program does not infringe the patent or other intellectual property rights of any other entity. Each Contributor disclaims any liability to Recipient for claims brought by any other entity based on infringement of intellectual property rights or otherwise. As a condition to exercising the rights and licenses granted hereunder, each Recipient hereby assumes sole responsibility to secure any other intellectual property rights needed, if any. For example, if a third party patent license is required to allow Recipient to distribute the Program, it is Recipient's responsibility to acquire that license before distributing the Program. + +d) Each Contributor represents that to its knowledge it has sufficient copyright rights in its Contribution, if any, to grant the copyright license set forth in this Agreement. + +3. REQUIREMENTS + +A Contributor may choose to distribute the Program in object code form under its own license agreement, provided that: + +a) it complies with the terms and conditions of this Agreement; and + +b) its license agreement: + +i) effectively disclaims on behalf of all Contributors all warranties and conditions, express and implied, including warranties or conditions of title and non-infringement, and implied warranties or conditions of merchantability and fitness for a particular purpose; + +ii) effectively excludes on behalf of all Contributors all liability for damages, including direct, indirect, special, incidental and consequential damages, such as lost profits; + +iii) states that any provisions which differ from this Agreement are offered by that Contributor alone and not by any other party; and + +iv) states that source code for the Program is available from such Contributor, and informs licensees how to obtain it in a reasonable manner on or through a medium customarily used for software exchange. + +When the Program is made available in source code form: + +a) it must be made available under this Agreement; and + +b) a copy of this Agreement must be included with each copy of the Program. + +Contributors may not remove or alter any copyright notices contained within the Program. + +Each Contributor must identify itself as the originator of its Contribution, if any, in a manner that reasonably allows subsequent Recipients to identify the originator of the Contribution. + +4. COMMERCIAL DISTRIBUTION + +Commercial distributors of software may accept certain responsibilities with respect to end users, business partners and the like. While this license is intended to facilitate the commercial use of the Program, the Contributor who includes the Program in a commercial product offering should do so in a manner which does not create potential liability for other Contributors. Therefore, if a Contributor includes the Program in a commercial product offering, such Contributor ("Commercial Contributor") hereby agrees to defend and indemnify every other Contributor ("Indemnified Contributor") against any losses, damages and costs (collectively "Losses") arising from claims, lawsuits and other legal actions brought by a third party against the Indemnified Contributor to the extent caused by the acts or omissions of such Commercial Contributor in connection with its distribution of the Program in a commercial product offering. The obligations in this section do not apply to any claims or Losses relating to any actual or alleged intellectual property infringement. In order to qualify, an Indemnified Contributor must: a) promptly notify the Commercial Contributor in writing of such claim, and b) allow the Commercial Contributor to control, and cooperate with the Commercial Contributor in, the defense and any related settlement negotiations. The Indemnified Contributor may participate in any such claim at its own expense. + +For example, a Contributor might include the Program in a commercial product offering, Product X. That Contributor is then a Commercial Contributor. If that Commercial Contributor then makes performance claims, or offers warranties related to Product X, those performance claims and warranties are such Commercial Contributor's responsibility alone. Under this section, the Commercial Contributor would have to defend claims against the other Contributors related to those performance claims and warranties, and if a court requires any other Contributor to pay any damages as a result, the Commercial Contributor must pay those damages. + +5. NO WARRANTY + +EXCEPT AS EXPRESSLY SET FORTH IN THIS AGREEMENT, THE PROGRAM IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, EITHER EXPRESS OR IMPLIED INCLUDING, WITHOUT LIMITATION, ANY WARRANTIES OR CONDITIONS OF TITLE, NON-INFRINGEMENT, MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Each Recipient is solely responsible for determining the appropriateness of using and distributing the Program and assumes all risks associated with its exercise of rights under this Agreement , including but not limited to the risks and costs of program errors, compliance with applicable laws, damage to or loss of data, programs or equipment, and unavailability or interruption of operations. + +6. DISCLAIMER OF LIABILITY + +EXCEPT AS EXPRESSLY SET FORTH IN THIS AGREEMENT, NEITHER RECIPIENT NOR ANY CONTRIBUTORS SHALL HAVE ANY LIABILITY FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING WITHOUT LIMITATION LOST PROFITS), HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OR DISTRIBUTION OF THE PROGRAM OR THE EXERCISE OF ANY RIGHTS GRANTED HEREUNDER, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. + +7. GENERAL + +If any provision of this Agreement is invalid or unenforceable under applicable law, it shall not affect the validity or enforceability of the remainder of the terms of this Agreement, and without further action by the parties hereto, such provision shall be reformed to the minimum extent necessary to make such provision valid and enforceable. + +If Recipient institutes patent litigation against any entity (including a cross-claim or counterclaim in a lawsuit) alleging that the Program itself (excluding combinations of the Program with other software or hardware) infringes such Recipient's patent(s), then such Recipient's rights granted under Section 2(b) shall terminate as of the date such litigation is filed. + +All Recipient's rights under this Agreement shall terminate if it fails to comply with any of the material terms or conditions of this Agreement and does not cure such failure in a reasonable period of time after becoming aware of such noncompliance. If all Recipient's rights under this Agreement terminate, Recipient agrees to cease use and distribution of the Program as soon as reasonably practicable. However, Recipient's obligations under this Agreement and any licenses granted by Recipient relating to the Program shall continue and survive. + +Everyone is permitted to copy and distribute copies of this Agreement, but in order to avoid inconsistency the Agreement is copyrighted and may only be modified in the following manner. The Agreement Steward reserves the right to publish new versions (including revisions) of this Agreement from time to time. No one other than the Agreement Steward has the right to modify this Agreement. The Eclipse Foundation is the initial Agreement Steward. The Eclipse Foundation may assign the responsibility to serve as the Agreement Steward to a suitable separate entity. Each new version of the Agreement will be given a distinguishing version number. The Program (including Contributions) may always be distributed subject to the version of the Agreement under which it was received. In addition, after a new version of the Agreement is published, Contributor may elect to distribute the Program (including its Contributions) under the new version. Except as expressly stated in Sections 2(a) and 2(b) above, Recipient receives no rights or licenses to the intellectual property of any Contributor under this Agreement, whether expressly, by implication, estoppel or otherwise. All rights in the Program not expressly granted under this Agreement are reserved. + +This Agreement is governed by the laws of the State of New York and the intellectual property laws of the United States of America. No party to this Agreement will bring a legal action under this Agreement more than one year after the cause of action arose. Each party waives its rights to a jury trial in any resulting litigation. \ No newline at end of file diff --git a/vendor/github.com/eclipse/paho.mqtt.golang/README.md b/vendor/github.com/eclipse/paho.mqtt.golang/README.md new file mode 100644 index 00000000000..8ba3f2039f8 --- /dev/null +++ b/vendor/github.com/eclipse/paho.mqtt.golang/README.md @@ -0,0 +1,71 @@ + +[![GoDoc](https://godoc.org/github.com/eclipse/paho.mqtt.golang?status.svg)](https://godoc.org/github.com/eclipse/paho.mqtt.golang) +[![Go Report Card](https://goreportcard.com/badge/github.com/eclipse/paho.mqtt.golang)](https://goreportcard.com/report/github.com/eclipse/paho.mqtt.golang) + +Eclipse Paho MQTT Go client +=========================== + + +This repository contains the source code for the [Eclipse Paho](http://eclipse.org/paho) MQTT Go client library. + +This code builds a library which enable applications to connect to an [MQTT](http://mqtt.org) broker to publish messages, and to subscribe to topics and receive published messages. + +This library supports a fully asynchronous mode of operation. + + +Installation and Build +---------------------- + +This client is designed to work with the standard Go tools, so installation is as easy as: + +``` +go get github.com/eclipse/paho.mqtt.golang +``` + +The client depends on Google's [proxy](https://godoc.org/golang.org/x/net/proxy) package and the [websockets](https://godoc.org/github.com/gorilla/websocket) package, +also easily installed with the commands: + +``` +go get github.com/gorilla/websocket +go get golang.org/x/net/proxy +``` + + +Usage and API +------------- + +Detailed API documentation is available by using to godoc tool, or can be browsed online +using the [godoc.org](http://godoc.org/github.com/eclipse/paho.mqtt.golang) service. + +Make use of the library by importing it in your Go client source code. For example, +``` +import "github.com/eclipse/paho.mqtt.golang" +``` + +Samples are available in the `cmd` directory for reference. + +Note: + +The library also supports using MQTT over websockets by using the `ws://` (unsecure) or `wss://` (secure) prefix in the URI. If the client is running behind a corporate http/https proxy then the following environment variables `HTTP_PROXY`, `HTTPS_PROXY` and `NO_PROXY` are taken into account when establishing the connection. + + +Runtime tracing +--------------- + +Tracing is enabled by assigning logs (from the Go log package) to the logging endpoints, ERROR, CRITICAL, WARN and DEBUG + + +Reporting bugs +-------------- + +Please report bugs by raising issues for this project in github https://github.com/eclipse/paho.mqtt.golang/issues + + +More information +---------------- + +Discussion of the Paho clients takes place on the [Eclipse paho-dev mailing list](https://dev.eclipse.org/mailman/listinfo/paho-dev). + +General questions about the MQTT protocol are discussed in the [MQTT Google Group](https://groups.google.com/forum/?hl=en-US&fromgroups#!forum/mqtt). + +There is much more information available via the [MQTT community site](http://mqtt.org). diff --git a/vendor/github.com/eclipse/paho.mqtt.golang/about.html b/vendor/github.com/eclipse/paho.mqtt.golang/about.html new file mode 100644 index 00000000000..b183f417abb --- /dev/null +++ b/vendor/github.com/eclipse/paho.mqtt.golang/about.html @@ -0,0 +1,41 @@ + + + +About + + +

About This Content

+ +

December 9, 2013

+

License

+ +

The Eclipse Foundation makes available all content in this plug-in ("Content"). Unless otherwise +indicated below, the Content is provided to you under the terms and conditions of the +Eclipse Public License Version 1.0 ("EPL") and Eclipse Distribution License Version 1.0 ("EDL"). +A copy of the EPL is available at +http://www.eclipse.org/legal/epl-v10.html +and a copy of the EDL is available at +http://www.eclipse.org/org/documents/edl-v10.php. +For purposes of the EPL, "Program" will mean the Content.

+ +

If you did not receive this Content directly from the Eclipse Foundation, the Content is +being redistributed by another party ("Redistributor") and different terms and conditions may +apply to your use of any object code in the Content. Check the Redistributor's license that was +provided with the Content. If no such license exists, contact the Redistributor. Unless otherwise +indicated below, the terms and conditions of the EPL still apply to any source code in the Content +and such source code may be obtained at http://www.eclipse.org.

+ + +

Third Party Content

+

The Content includes items that have been sourced from third parties as set out below. If you + did not receive this Content directly from the Eclipse Foundation, the following is provided + for informational purposes only, and you should look to the Redistributor's license for + terms and conditions of use.

+

+ None

+

+

+ + + + diff --git a/vendor/github.com/eclipse/paho.mqtt.golang/client.go b/vendor/github.com/eclipse/paho.mqtt.golang/client.go new file mode 100644 index 00000000000..a8743a23845 --- /dev/null +++ b/vendor/github.com/eclipse/paho.mqtt.golang/client.go @@ -0,0 +1,965 @@ +/* + * Copyright (c) 2013 IBM Corp. + * + * All rights reserved. This program and the accompanying materials + * are made available under the terms of the Eclipse Public License v1.0 + * which accompanies this distribution, and is available at + * http://www.eclipse.org/legal/epl-v10.html + * + * Contributors: + * Seth Hoenig + * Allan Stockdill-Mander + * Mike Robertson + */ + +// Portions copyright © 2018 TIBCO Software Inc. + +// Package mqtt provides an MQTT v3.1.1 client library. +package mqtt + +import ( + "bytes" + "errors" + "fmt" + "net" + "strings" + "sync" + "sync/atomic" + "time" + + "github.com/eclipse/paho.mqtt.golang/packets" +) + +const ( + disconnected uint32 = iota + connecting + reconnecting + connected +) + +// Client is the interface definition for a Client as used by this +// library, the interface is primarily to allow mocking tests. +// +// It is an MQTT v3.1.1 client for communicating +// with an MQTT server using non-blocking methods that allow work +// to be done in the background. +// An application may connect to an MQTT server using: +// A plain TCP socket +// A secure SSL/TLS socket +// A websocket +// To enable ensured message delivery at Quality of Service (QoS) levels +// described in the MQTT spec, a message persistence mechanism must be +// used. This is done by providing a type which implements the Store +// interface. For convenience, FileStore and MemoryStore are provided +// implementations that should be sufficient for most use cases. More +// information can be found in their respective documentation. +// Numerous connection options may be specified by configuring a +// and then supplying a ClientOptions type. +type Client interface { + // IsConnected returns a bool signifying whether + // the client is connected or not. + IsConnected() bool + // IsConnectionOpen return a bool signifying whether the client has an active + // connection to mqtt broker, i.e not in disconnected or reconnect mode + IsConnectionOpen() bool + // Connect will create a connection to the message broker, by default + // it will attempt to connect at v3.1.1 and auto retry at v3.1 if that + // fails + Connect() Token + // Disconnect will end the connection with the server, but not before waiting + // the specified number of milliseconds to wait for existing work to be + // completed. + Disconnect(quiesce uint) + // Publish will publish a message with the specified QoS and content + // to the specified topic. + // Returns a token to track delivery of the message to the broker + Publish(topic string, qos byte, retained bool, payload interface{}) Token + // Subscribe starts a new subscription. Provide a MessageHandler to be executed when + // a message is published on the topic provided, or nil for the default handler + Subscribe(topic string, qos byte, callback MessageHandler) Token + // SubscribeMultiple starts a new subscription for multiple topics. Provide a MessageHandler to + // be executed when a message is published on one of the topics provided, or nil for the + // default handler + SubscribeMultiple(filters map[string]byte, callback MessageHandler) Token + // Unsubscribe will end the subscription from each of the topics provided. + // Messages published to those topics from other clients will no longer be + // received. + Unsubscribe(topics ...string) Token + // AddRoute allows you to add a handler for messages on a specific topic + // without making a subscription. For example having a different handler + // for parts of a wildcard subscription + AddRoute(topic string, callback MessageHandler) + // OptionsReader returns a ClientOptionsReader which is a copy of the clientoptions + // in use by the client. + OptionsReader() ClientOptionsReader +} + +// client implements the Client interface +type client struct { + lastSent atomic.Value + lastReceived atomic.Value + pingOutstanding int32 + status uint32 + sync.RWMutex + messageIds + conn net.Conn + ibound chan packets.ControlPacket + obound chan *PacketAndToken + oboundP chan *PacketAndToken + msgRouter *router + stopRouter chan bool + incomingPubChan chan *packets.PublishPacket + errors chan error + stop chan struct{} + persist Store + options ClientOptions + optionsMu sync.Mutex // Protects the options in a few limited cases where needed for testing + workers sync.WaitGroup +} + +// NewClient will create an MQTT v3.1.1 client with all of the options specified +// in the provided ClientOptions. The client must have the Connect method called +// on it before it may be used. This is to make sure resources (such as a net +// connection) are created before the application is actually ready. +func NewClient(o *ClientOptions) Client { + c := &client{} + c.options = *o + + if c.options.Store == nil { + c.options.Store = NewMemoryStore() + } + switch c.options.ProtocolVersion { + case 3, 4: + c.options.protocolVersionExplicit = true + case 0x83, 0x84: + c.options.protocolVersionExplicit = true + default: + c.options.ProtocolVersion = 4 + c.options.protocolVersionExplicit = false + } + c.persist = c.options.Store + c.status = disconnected + c.messageIds = messageIds{index: make(map[uint16]tokenCompletor)} + c.msgRouter, c.stopRouter = newRouter() + c.msgRouter.setDefaultHandler(c.options.DefaultPublishHandler) + return c +} + +// AddRoute allows you to add a handler for messages on a specific topic +// without making a subscription. For example having a different handler +// for parts of a wildcard subscription +func (c *client) AddRoute(topic string, callback MessageHandler) { + if callback != nil { + c.msgRouter.addRoute(topic, callback) + } +} + +// IsConnected returns a bool signifying whether +// the client is connected or not. +// connected means that the connection is up now OR it will +// be established/reestablished automatically when possible +func (c *client) IsConnected() bool { + c.RLock() + defer c.RUnlock() + status := atomic.LoadUint32(&c.status) + switch { + case status == connected: + return true + case c.options.AutoReconnect && status > connecting: + return true + case c.options.ConnectRetry && status == connecting: + return true + default: + return false + } +} + +// IsConnectionOpen return a bool signifying whether the client has an active +// connection to mqtt broker, i.e not in disconnected or reconnect mode +func (c *client) IsConnectionOpen() bool { + c.RLock() + defer c.RUnlock() + status := atomic.LoadUint32(&c.status) + switch { + case status == connected: + return true + default: + return false + } +} + +func (c *client) connectionStatus() uint32 { + c.RLock() + defer c.RUnlock() + status := atomic.LoadUint32(&c.status) + return status +} + +func (c *client) setConnected(status uint32) { + c.Lock() + defer c.Unlock() + atomic.StoreUint32(&c.status, uint32(status)) +} + +//ErrNotConnected is the error returned from function calls that are +//made when the client is not connected to a broker +var ErrNotConnected = errors.New("Not Connected") + +// Connect will create a connection to the message broker, by default +// it will attempt to connect at v3.1.1 and auto retry at v3.1 if that +// fails +func (c *client) Connect() Token { + var err error + t := newToken(packets.Connect).(*ConnectToken) + DEBUG.Println(CLI, "Connect()") + + if c.options.ConnectRetry && atomic.LoadUint32(&c.status) != disconnected { + // if in any state other than disconnected and ConnectRetry is + // enabled then the connection will come up automatically + // client can assume connection is up + WARN.Println(CLI, "Connect() called but not disconnected") + t.returnCode = packets.Accepted + t.flowComplete() + return t + } + + c.obound = make(chan *PacketAndToken) + c.oboundP = make(chan *PacketAndToken) + c.ibound = make(chan packets.ControlPacket) + + c.persist.Open() + if c.options.ConnectRetry { + c.reserveStoredPublishIDs() // Reserve IDs to allow publish before connect complete + } + c.setConnected(connecting) + + go func() { + c.errors = make(chan error, 1) + c.stop = make(chan struct{}) + + var rc byte + protocolVersion := c.options.ProtocolVersion + + if len(c.options.Servers) == 0 { + t.setError(fmt.Errorf("No servers defined to connect to")) + return + } + + RETRYCONN: + c.optionsMu.Lock() // Protect c.options.Servers so that servers can be added in test cases + brokers := c.options.Servers + c.optionsMu.Unlock() + + for _, broker := range brokers { + cm := newConnectMsgFromOptions(&c.options, broker) + c.options.ProtocolVersion = protocolVersion + CONN: + DEBUG.Println(CLI, "about to write new connect msg") + c.Lock() + c.conn, err = openConnection(broker, c.options.TLSConfig, c.options.ConnectTimeout, + c.options.HTTPHeaders) + c.Unlock() + if err == nil { + DEBUG.Println(CLI, "socket connected to broker") + switch c.options.ProtocolVersion { + case 3: + DEBUG.Println(CLI, "Using MQTT 3.1 protocol") + cm.ProtocolName = "MQIsdp" + cm.ProtocolVersion = 3 + case 0x83: + DEBUG.Println(CLI, "Using MQTT 3.1b protocol") + cm.ProtocolName = "MQIsdp" + cm.ProtocolVersion = 0x83 + case 0x84: + DEBUG.Println(CLI, "Using MQTT 3.1.1b protocol") + cm.ProtocolName = "MQTT" + cm.ProtocolVersion = 0x84 + default: + DEBUG.Println(CLI, "Using MQTT 3.1.1 protocol") + c.options.ProtocolVersion = 4 + cm.ProtocolName = "MQTT" + cm.ProtocolVersion = 4 + } + cm.Write(c.conn) + + rc, t.sessionPresent = c.connect() + if rc != packets.Accepted { + c.Lock() + if c.conn != nil { + c.conn.Close() + c.conn = nil + } + c.Unlock() + //if the protocol version was explicitly set don't do any fallback + if c.options.protocolVersionExplicit { + ERROR.Println(CLI, "Connecting to", broker, "CONNACK was not CONN_ACCEPTED, but rather", packets.ConnackReturnCodes[rc]) + continue + } + if c.options.ProtocolVersion == 4 { + DEBUG.Println(CLI, "Trying reconnect using MQTT 3.1 protocol") + c.options.ProtocolVersion = 3 + goto CONN + } + } + break + } else { + ERROR.Println(CLI, err.Error()) + WARN.Println(CLI, "failed to connect to broker, trying next") + rc = packets.ErrNetworkError + } + } + + if c.conn == nil { + if c.options.ConnectRetry { + DEBUG.Println(CLI, "Connect failed, sleeping for", int(c.options.ConnectRetryInterval.Seconds()), "seconds and will then retry") + time.Sleep(c.options.ConnectRetryInterval) + + if atomic.LoadUint32(&c.status) == connecting { + goto RETRYCONN + } + } + ERROR.Println(CLI, "Failed to connect to a broker") + c.setConnected(disconnected) + c.persist.Close() + t.returnCode = rc + if rc != packets.ErrNetworkError { + t.setError(packets.ConnErrors[rc]) + } else { + t.setError(fmt.Errorf("%s : %s", packets.ConnErrors[rc], err)) + } + return + } + + c.options.protocolVersionExplicit = true + + if c.options.KeepAlive != 0 { + atomic.StoreInt32(&c.pingOutstanding, 0) + c.lastReceived.Store(time.Now()) + c.lastSent.Store(time.Now()) + c.workers.Add(1) + go keepalive(c) + } + + c.incomingPubChan = make(chan *packets.PublishPacket) + c.msgRouter.matchAndDispatch(c.incomingPubChan, c.options.Order, c) + + c.setConnected(connected) + DEBUG.Println(CLI, "client is connected") + if c.options.OnConnect != nil { + go c.options.OnConnect(c) + } + + c.workers.Add(4) + go errorWatch(c) + go alllogic(c) + go outgoing(c) + go incoming(c) + + // Take care of any messages in the store + if !c.options.CleanSession { + c.workers.Add(1) // disconnect during resume can lead to reconnect being called before resume completes + c.resume(c.options.ResumeSubs) + } else { + c.persist.Reset() + } + + DEBUG.Println(CLI, "exit startClient") + t.flowComplete() + }() + return t +} + +// internal function used to reconnect the client when it loses its connection +func (c *client) reconnect() { + DEBUG.Println(CLI, "enter reconnect") + var ( + err error + + rc = byte(1) + sleep = time.Duration(1 * time.Second) + ) + + for rc != 0 && atomic.LoadUint32(&c.status) != disconnected { + if nil != c.options.OnReconnecting { + c.options.OnReconnecting(c, &c.options) + } + c.optionsMu.Lock() // Protect c.options.Servers so that servers can be added in test cases + brokers := c.options.Servers + c.optionsMu.Unlock() + for _, broker := range brokers { + cm := newConnectMsgFromOptions(&c.options, broker) + DEBUG.Println(CLI, "about to write new connect msg") + c.Lock() + c.conn, err = openConnection(broker, c.options.TLSConfig, c.options.ConnectTimeout, c.options.HTTPHeaders) + c.Unlock() + if err == nil { + DEBUG.Println(CLI, "socket connected to broker") + switch c.options.ProtocolVersion { + case 0x83: + DEBUG.Println(CLI, "Using MQTT 3.1b protocol") + cm.ProtocolName = "MQIsdp" + cm.ProtocolVersion = 0x83 + case 0x84: + DEBUG.Println(CLI, "Using MQTT 3.1.1b protocol") + cm.ProtocolName = "MQTT" + cm.ProtocolVersion = 0x84 + case 3: + DEBUG.Println(CLI, "Using MQTT 3.1 protocol") + cm.ProtocolName = "MQIsdp" + cm.ProtocolVersion = 3 + default: + DEBUG.Println(CLI, "Using MQTT 3.1.1 protocol") + cm.ProtocolName = "MQTT" + cm.ProtocolVersion = 4 + } + cm.Write(c.conn) + + rc, _ = c.connect() + if rc != packets.Accepted { + if c.conn != nil { + c.conn.Close() + c.conn = nil + } + //if the protocol version was explicitly set don't do any fallback + if c.options.protocolVersionExplicit { + ERROR.Println(CLI, "Connecting to", broker, "CONNACK was not Accepted, but rather", packets.ConnackReturnCodes[rc]) + continue + } + } + break + } else { + ERROR.Println(CLI, err.Error()) + WARN.Println(CLI, "failed to connect to broker, trying next") + rc = packets.ErrNetworkError + } + } + if rc != 0 { + DEBUG.Println(CLI, "Reconnect failed, sleeping for", int(sleep.Seconds()), "seconds") + time.Sleep(sleep) + if sleep < c.options.MaxReconnectInterval { + sleep *= 2 + } + + if sleep > c.options.MaxReconnectInterval { + sleep = c.options.MaxReconnectInterval + } + } + } + // Disconnect() must have been called while we were trying to reconnect. + if c.connectionStatus() == disconnected { + DEBUG.Println(CLI, "Client moved to disconnected state while reconnecting, abandoning reconnect") + return + } + + c.stop = make(chan struct{}) + + if c.options.KeepAlive != 0 { + atomic.StoreInt32(&c.pingOutstanding, 0) + c.lastReceived.Store(time.Now()) + c.lastSent.Store(time.Now()) + c.workers.Add(1) + go keepalive(c) + } + + c.setConnected(connected) + DEBUG.Println(CLI, "client is reconnected") + if c.options.OnConnect != nil { + go c.options.OnConnect(c) + } + + c.workers.Add(4) + go errorWatch(c) + go alllogic(c) + go outgoing(c) + go incoming(c) + + c.workers.Add(1) // disconnect during resume can lead to reconnect being called before resume completes + c.resume(c.options.ResumeSubs) +} + +// This function is only used for receiving a connack +// when the connection is first started. +// This prevents receiving incoming data while resume +// is in progress if clean session is false. +func (c *client) connect() (byte, bool) { + DEBUG.Println(NET, "connect started") + + ca, err := packets.ReadPacket(c.conn) + if err != nil { + ERROR.Println(NET, "connect got error", err) + return packets.ErrNetworkError, false + } + if ca == nil { + ERROR.Println(NET, "received nil packet") + return packets.ErrNetworkError, false + } + + msg, ok := ca.(*packets.ConnackPacket) + if !ok { + ERROR.Println(NET, "received msg that was not CONNACK") + return packets.ErrNetworkError, false + } + + DEBUG.Println(NET, "received connack") + return msg.ReturnCode, msg.SessionPresent +} + +// Disconnect will end the connection with the server, but not before waiting +// the specified number of milliseconds to wait for existing work to be +// completed. +func (c *client) Disconnect(quiesce uint) { + status := atomic.LoadUint32(&c.status) + if status == connected { + DEBUG.Println(CLI, "disconnecting") + c.setConnected(disconnected) + + dm := packets.NewControlPacket(packets.Disconnect).(*packets.DisconnectPacket) + dt := newToken(packets.Disconnect) + c.oboundP <- &PacketAndToken{p: dm, t: dt} + + // wait for work to finish, or quiesce time consumed + dt.WaitTimeout(time.Duration(quiesce) * time.Millisecond) + } else { + WARN.Println(CLI, "Disconnect() called but not connected (disconnected/reconnecting)") + c.setConnected(disconnected) + } + + c.disconnect() +} + +// ForceDisconnect will end the connection with the mqtt broker immediately. +func (c *client) forceDisconnect() { + if !c.IsConnected() { + WARN.Println(CLI, "already disconnected") + return + } + c.setConnected(disconnected) + c.conn.Close() + DEBUG.Println(CLI, "forcefully disconnecting") + c.disconnect() +} + +func (c *client) internalConnLost(err error) { + // Only do anything if this was called and we are still "connected" + // forceDisconnect can cause incoming/outgoing/alllogic to end with + // error from closing the socket but state will be "disconnected" + if c.IsConnected() { + c.closeStop() + c.conn.Close() + c.workers.Wait() + if c.options.CleanSession && !c.options.AutoReconnect { + c.messageIds.cleanUp() + } + if c.options.AutoReconnect { + c.setConnected(reconnecting) + go c.reconnect() + } else { + c.setConnected(disconnected) + } + if c.options.OnConnectionLost != nil { + go c.options.OnConnectionLost(c, err) + } + } +} + +func (c *client) closeStop() { + c.Lock() + defer c.Unlock() + select { + case <-c.stop: + DEBUG.Println("In disconnect and stop channel is already closed") + default: + if c.stop != nil { + close(c.stop) + } + } +} + +func (c *client) closeStopRouter() { + c.Lock() + defer c.Unlock() + select { + case <-c.stopRouter: + DEBUG.Println("In disconnect and stop channel is already closed") + default: + if c.stopRouter != nil { + close(c.stopRouter) + } + } +} + +func (c *client) closeConn() { + c.Lock() + defer c.Unlock() + if c.conn != nil { + c.conn.Close() + } +} + +func (c *client) disconnect() { + c.closeStop() + c.closeConn() + c.workers.Wait() + c.messageIds.cleanUp() + c.closeStopRouter() + DEBUG.Println(CLI, "disconnected") + c.persist.Close() +} + +// Publish will publish a message with the specified QoS and content +// to the specified topic. +// Returns a token to track delivery of the message to the broker +func (c *client) Publish(topic string, qos byte, retained bool, payload interface{}) Token { + token := newToken(packets.Publish).(*PublishToken) + DEBUG.Println(CLI, "enter Publish") + switch { + case !c.IsConnected(): + token.setError(ErrNotConnected) + return token + case c.connectionStatus() == reconnecting && qos == 0: + token.flowComplete() + return token + } + pub := packets.NewControlPacket(packets.Publish).(*packets.PublishPacket) + pub.Qos = qos + pub.TopicName = topic + pub.Retain = retained + switch p := payload.(type) { + case string: + pub.Payload = []byte(p) + case []byte: + pub.Payload = p + case bytes.Buffer: + pub.Payload = p.Bytes() + default: + token.setError(fmt.Errorf("Unknown payload type")) + return token + } + + if pub.Qos != 0 && pub.MessageID == 0 { + pub.MessageID = c.getID(token) + token.messageID = pub.MessageID + } + persistOutbound(c.persist, pub) + switch c.connectionStatus() { + case connecting: + DEBUG.Println(CLI, "storing publish message (connecting), topic:", topic) + case reconnecting: + DEBUG.Println(CLI, "storing publish message (reconnecting), topic:", topic) + default: + DEBUG.Println(CLI, "sending publish message, topic:", topic) + publishWaitTimeout := c.options.WriteTimeout + if publishWaitTimeout == 0 { + publishWaitTimeout = time.Second * 30 + } + select { + case c.obound <- &PacketAndToken{p: pub, t: token}: + case <-time.After(publishWaitTimeout): + token.setError(errors.New("publish was broken by timeout")) + } + } + return token +} + +// Subscribe starts a new subscription. Provide a MessageHandler to be executed when +// a message is published on the topic provided. +func (c *client) Subscribe(topic string, qos byte, callback MessageHandler) Token { + token := newToken(packets.Subscribe).(*SubscribeToken) + DEBUG.Println(CLI, "enter Subscribe") + if !c.IsConnected() { + token.setError(ErrNotConnected) + return token + } + if !c.IsConnectionOpen() { + switch { + case !c.options.ResumeSubs: + // if not connected and resumesubs not set this sub will be thrown away + token.setError(fmt.Errorf("not currently connected and ResumeSubs not set")) + return token + case c.options.CleanSession && c.connectionStatus() == reconnecting: + // if reconnecting and cleansession is true this sub will be thrown away + token.setError(fmt.Errorf("reconnecting state and cleansession is true")) + return token + } + } + sub := packets.NewControlPacket(packets.Subscribe).(*packets.SubscribePacket) + if err := validateTopicAndQos(topic, qos); err != nil { + token.setError(err) + return token + } + sub.Topics = append(sub.Topics, topic) + sub.Qoss = append(sub.Qoss, qos) + + if strings.HasPrefix(topic, "$share/") { + topic = strings.Join(strings.Split(topic, "/")[2:], "/") + } + + if strings.HasPrefix(topic, "$queue/") { + topic = strings.TrimPrefix(topic, "$queue/") + } + + if callback != nil { + c.msgRouter.addRoute(topic, callback) + } + + token.subs = append(token.subs, topic) + + if sub.MessageID == 0 { + sub.MessageID = c.getID(token) + token.messageID = sub.MessageID + } + DEBUG.Println(CLI, sub.String()) + + persistOutbound(c.persist, sub) + switch c.connectionStatus() { + case connecting: + DEBUG.Println(CLI, "storing subscribe message (connecting), topic:", topic) + case reconnecting: + DEBUG.Println(CLI, "storing subscribe message (reconnecting), topic:", topic) + default: + DEBUG.Println(CLI, "sending subscribe message, topic:", topic) + subscribeWaitTimeout := c.options.WriteTimeout + if subscribeWaitTimeout == 0 { + subscribeWaitTimeout = time.Second * 30 + } + select { + case c.oboundP <- &PacketAndToken{p: sub, t: token}: + case <-time.After(subscribeWaitTimeout): + token.setError(errors.New("subscribe was broken by timeout")) + } + } + DEBUG.Println(CLI, "exit Subscribe") + return token +} + +// SubscribeMultiple starts a new subscription for multiple topics. Provide a MessageHandler to +// be executed when a message is published on one of the topics provided. +func (c *client) SubscribeMultiple(filters map[string]byte, callback MessageHandler) Token { + var err error + token := newToken(packets.Subscribe).(*SubscribeToken) + DEBUG.Println(CLI, "enter SubscribeMultiple") + if !c.IsConnected() { + token.setError(ErrNotConnected) + return token + } + if !c.IsConnectionOpen() { + switch { + case !c.options.ResumeSubs: + // if not connected and resumesubs not set this sub will be thrown away + token.setError(fmt.Errorf("not currently connected and ResumeSubs not set")) + return token + case c.options.CleanSession && c.connectionStatus() == reconnecting: + // if reconnecting and cleansession is true this sub will be thrown away + token.setError(fmt.Errorf("reconnecting state and cleansession is true")) + return token + } + } + sub := packets.NewControlPacket(packets.Subscribe).(*packets.SubscribePacket) + if sub.Topics, sub.Qoss, err = validateSubscribeMap(filters); err != nil { + token.setError(err) + return token + } + + if callback != nil { + for topic := range filters { + c.msgRouter.addRoute(topic, callback) + } + } + token.subs = make([]string, len(sub.Topics)) + copy(token.subs, sub.Topics) + + if sub.MessageID == 0 { + sub.MessageID = c.getID(token) + token.messageID = sub.MessageID + } + persistOutbound(c.persist, sub) + switch c.connectionStatus() { + case connecting: + DEBUG.Println(CLI, "storing subscribe message (connecting), topics:", sub.Topics) + case reconnecting: + DEBUG.Println(CLI, "storing subscribe message (reconnecting), topics:", sub.Topics) + default: + DEBUG.Println(CLI, "sending subscribe message, topics:", sub.Topics) + subscribeWaitTimeout := c.options.WriteTimeout + if subscribeWaitTimeout == 0 { + subscribeWaitTimeout = time.Second * 30 + } + select { + case c.oboundP <- &PacketAndToken{p: sub, t: token}: + case <-time.After(subscribeWaitTimeout): + token.setError(errors.New("subscribe was broken by timeout")) + } + } + DEBUG.Println(CLI, "exit SubscribeMultiple") + return token +} + +// reserveStoredPublishIDs reserves the ids for publish packets in the persistent store to ensure these are not duplicated +func (c *client) reserveStoredPublishIDs() { + // The resume function sets the stored id for publish packets only (some other packets + // will get new ids in net code). This means that the only keys we need to ensure are + // unique are the publish ones (and these will completed/replaced in resume() ) + if !c.options.CleanSession { + storedKeys := c.persist.All() + for _, key := range storedKeys { + packet := c.persist.Get(key) + if packet == nil { + continue + } + switch packet.(type) { + case *packets.PublishPacket: + details := packet.Details() + token := &PlaceHolderToken{id: details.MessageID} + c.claimID(token, details.MessageID) + } + } + } +} + +// Load all stored messages and resend them +// Call this to ensure QOS > 1,2 even after an application crash +func (c *client) resume(subscription bool) { + defer c.workers.Done() // resume must complete before any attempt to reconnect is made + + storedKeys := c.persist.All() + for _, key := range storedKeys { + packet := c.persist.Get(key) + if packet == nil { + continue + } + details := packet.Details() + if isKeyOutbound(key) { + switch packet.(type) { + case *packets.SubscribePacket: + if subscription { + DEBUG.Println(STR, fmt.Sprintf("loaded pending subscribe (%d)", details.MessageID)) + subPacket := packet.(*packets.SubscribePacket) + token := newToken(packets.Subscribe).(*SubscribeToken) + token.messageID = details.MessageID + token.subs = append(token.subs, subPacket.Topics...) + c.claimID(token, details.MessageID) + select { + case c.oboundP <- &PacketAndToken{p: packet, t: token}: + case <-c.stop: + return + } + } + case *packets.UnsubscribePacket: + if subscription { + DEBUG.Println(STR, fmt.Sprintf("loaded pending unsubscribe (%d)", details.MessageID)) + token := newToken(packets.Unsubscribe).(*UnsubscribeToken) + select { + case c.oboundP <- &PacketAndToken{p: packet, t: token}: + case <-c.stop: + return + } + } + case *packets.PubrelPacket: + DEBUG.Println(STR, fmt.Sprintf("loaded pending pubrel (%d)", details.MessageID)) + select { + case c.oboundP <- &PacketAndToken{p: packet, t: nil}: + case <-c.stop: + return + } + case *packets.PublishPacket: + token := newToken(packets.Publish).(*PublishToken) + token.messageID = details.MessageID + c.claimID(token, details.MessageID) + DEBUG.Println(STR, fmt.Sprintf("loaded pending publish (%d)", details.MessageID)) + DEBUG.Println(STR, details) + select { + case c.obound <- &PacketAndToken{p: packet, t: token}: + case <-c.stop: + return + } + default: + ERROR.Println(STR, "invalid message type in store (discarded)") + c.persist.Del(key) + } + } else { + switch packet.(type) { + case *packets.PubrelPacket: + DEBUG.Println(STR, fmt.Sprintf("loaded pending incomming (%d)", details.MessageID)) + select { + case c.ibound <- packet: + case <-c.stop: + return + } + default: + ERROR.Println(STR, "invalid message type in store (discarded)") + c.persist.Del(key) + } + } + } +} + +// Unsubscribe will end the subscription from each of the topics provided. +// Messages published to those topics from other clients will no longer be +// received. +func (c *client) Unsubscribe(topics ...string) Token { + token := newToken(packets.Unsubscribe).(*UnsubscribeToken) + DEBUG.Println(CLI, "enter Unsubscribe") + if !c.IsConnected() { + token.setError(ErrNotConnected) + return token + } + if !c.IsConnectionOpen() { + switch { + case !c.options.ResumeSubs: + // if not connected and resumesubs not set this unsub will be thrown away + token.setError(fmt.Errorf("not currently connected and ResumeSubs not set")) + return token + case c.options.CleanSession && c.connectionStatus() == reconnecting: + // if reconnecting and cleansession is true this unsub will be thrown away + token.setError(fmt.Errorf("reconnecting state and cleansession is true")) + return token + } + } + unsub := packets.NewControlPacket(packets.Unsubscribe).(*packets.UnsubscribePacket) + unsub.Topics = make([]string, len(topics)) + copy(unsub.Topics, topics) + + if unsub.MessageID == 0 { + unsub.MessageID = c.getID(token) + token.messageID = unsub.MessageID + } + + persistOutbound(c.persist, unsub) + + switch c.connectionStatus() { + case connecting: + DEBUG.Println(CLI, "storing unsubscribe message (connecting), topics:", topics) + case reconnecting: + DEBUG.Println(CLI, "storing unsubscribe message (reconnecting), topics:", topics) + default: + DEBUG.Println(CLI, "sending unsubscribe message, topics:", topics) + subscribeWaitTimeout := c.options.WriteTimeout + if subscribeWaitTimeout == 0 { + subscribeWaitTimeout = time.Second * 30 + } + select { + case c.oboundP <- &PacketAndToken{p: unsub, t: token}: + for _, topic := range topics { + c.msgRouter.deleteRoute(topic) + } + case <-time.After(subscribeWaitTimeout): + token.setError(errors.New("unsubscribe was broken by timeout")) + } + } + + DEBUG.Println(CLI, "exit Unsubscribe") + return token +} + +// OptionsReader returns a ClientOptionsReader which is a copy of the clientoptions +// in use by the client. +func (c *client) OptionsReader() ClientOptionsReader { + r := ClientOptionsReader{options: &c.options} + return r +} + +//DefaultConnectionLostHandler is a definition of a function that simply +//reports to the DEBUG log the reason for the client losing a connection. +func DefaultConnectionLostHandler(client Client, reason error) { + DEBUG.Println("Connection lost:", reason.Error()) +} diff --git a/vendor/github.com/eclipse/paho.mqtt.golang/components.go b/vendor/github.com/eclipse/paho.mqtt.golang/components.go new file mode 100644 index 00000000000..01f5fafdf8f --- /dev/null +++ b/vendor/github.com/eclipse/paho.mqtt.golang/components.go @@ -0,0 +1,31 @@ +/* + * Copyright (c) 2013 IBM Corp. + * + * All rights reserved. This program and the accompanying materials + * are made available under the terms of the Eclipse Public License v1.0 + * which accompanies this distribution, and is available at + * http://www.eclipse.org/legal/epl-v10.html + * + * Contributors: + * Seth Hoenig + * Allan Stockdill-Mander + * Mike Robertson + */ + +package mqtt + +type component string + +// Component names for debug output +const ( + NET component = "[net] " + PNG component = "[pinger] " + CLI component = "[client] " + DEC component = "[decode] " + MES component = "[message] " + STR component = "[store] " + MID component = "[msgids] " + TST component = "[test] " + STA component = "[state] " + ERR component = "[error] " +) diff --git a/vendor/github.com/eclipse/paho.mqtt.golang/edl-v10 b/vendor/github.com/eclipse/paho.mqtt.golang/edl-v10 new file mode 100644 index 00000000000..cf989f1456b --- /dev/null +++ b/vendor/github.com/eclipse/paho.mqtt.golang/edl-v10 @@ -0,0 +1,15 @@ + +Eclipse Distribution License - v 1.0 + +Copyright (c) 2007, Eclipse Foundation, Inc. and its licensors. + +All rights reserved. + +Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: + + Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. + Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. + Neither the name of the Eclipse Foundation, Inc. nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + diff --git a/vendor/github.com/eclipse/paho.mqtt.golang/epl-v10 b/vendor/github.com/eclipse/paho.mqtt.golang/epl-v10 new file mode 100644 index 00000000000..79e486c3d2c --- /dev/null +++ b/vendor/github.com/eclipse/paho.mqtt.golang/epl-v10 @@ -0,0 +1,70 @@ +Eclipse Public License - v 1.0 + +THE ACCOMPANYING PROGRAM IS PROVIDED UNDER THE TERMS OF THIS ECLIPSE PUBLIC LICENSE ("AGREEMENT"). ANY USE, REPRODUCTION OR DISTRIBUTION OF THE PROGRAM CONSTITUTES RECIPIENT'S ACCEPTANCE OF THIS AGREEMENT. + +1. DEFINITIONS + +"Contribution" means: + +a) in the case of the initial Contributor, the initial code and documentation distributed under this Agreement, and +b) in the case of each subsequent Contributor: +i) changes to the Program, and +ii) additions to the Program; +where such changes and/or additions to the Program originate from and are distributed by that particular Contributor. A Contribution 'originates' from a Contributor if it was added to the Program by such Contributor itself or anyone acting on such Contributor's behalf. Contributions do not include additions to the Program which: (i) are separate modules of software distributed in conjunction with the Program under their own license agreement, and (ii) are not derivative works of the Program. +"Contributor" means any person or entity that distributes the Program. + +"Licensed Patents" mean patent claims licensable by a Contributor which are necessarily infringed by the use or sale of its Contribution alone or when combined with the Program. + +"Program" means the Contributions distributed in accordance with this Agreement. + +"Recipient" means anyone who receives the Program under this Agreement, including all Contributors. + +2. GRANT OF RIGHTS + +a) Subject to the terms of this Agreement, each Contributor hereby grants Recipient a non-exclusive, worldwide, royalty-free copyright license to reproduce, prepare derivative works of, publicly display, publicly perform, distribute and sublicense the Contribution of such Contributor, if any, and such derivative works, in source code and object code form. +b) Subject to the terms of this Agreement, each Contributor hereby grants Recipient a non-exclusive, worldwide, royalty-free patent license under Licensed Patents to make, use, sell, offer to sell, import and otherwise transfer the Contribution of such Contributor, if any, in source code and object code form. This patent license shall apply to the combination of the Contribution and the Program if, at the time the Contribution is added by the Contributor, such addition of the Contribution causes such combination to be covered by the Licensed Patents. The patent license shall not apply to any other combinations which include the Contribution. No hardware per se is licensed hereunder. +c) Recipient understands that although each Contributor grants the licenses to its Contributions set forth herein, no assurances are provided by any Contributor that the Program does not infringe the patent or other intellectual property rights of any other entity. Each Contributor disclaims any liability to Recipient for claims brought by any other entity based on infringement of intellectual property rights or otherwise. As a condition to exercising the rights and licenses granted hereunder, each Recipient hereby assumes sole responsibility to secure any other intellectual property rights needed, if any. For example, if a third party patent license is required to allow Recipient to distribute the Program, it is Recipient's responsibility to acquire that license before distributing the Program. +d) Each Contributor represents that to its knowledge it has sufficient copyright rights in its Contribution, if any, to grant the copyright license set forth in this Agreement. +3. REQUIREMENTS + +A Contributor may choose to distribute the Program in object code form under its own license agreement, provided that: + +a) it complies with the terms and conditions of this Agreement; and +b) its license agreement: +i) effectively disclaims on behalf of all Contributors all warranties and conditions, express and implied, including warranties or conditions of title and non-infringement, and implied warranties or conditions of merchantability and fitness for a particular purpose; +ii) effectively excludes on behalf of all Contributors all liability for damages, including direct, indirect, special, incidental and consequential damages, such as lost profits; +iii) states that any provisions which differ from this Agreement are offered by that Contributor alone and not by any other party; and +iv) states that source code for the Program is available from such Contributor, and informs licensees how to obtain it in a reasonable manner on or through a medium customarily used for software exchange. +When the Program is made available in source code form: + +a) it must be made available under this Agreement; and +b) a copy of this Agreement must be included with each copy of the Program. +Contributors may not remove or alter any copyright notices contained within the Program. + +Each Contributor must identify itself as the originator of its Contribution, if any, in a manner that reasonably allows subsequent Recipients to identify the originator of the Contribution. + +4. COMMERCIAL DISTRIBUTION + +Commercial distributors of software may accept certain responsibilities with respect to end users, business partners and the like. While this license is intended to facilitate the commercial use of the Program, the Contributor who includes the Program in a commercial product offering should do so in a manner which does not create potential liability for other Contributors. Therefore, if a Contributor includes the Program in a commercial product offering, such Contributor ("Commercial Contributor") hereby agrees to defend and indemnify every other Contributor ("Indemnified Contributor") against any losses, damages and costs (collectively "Losses") arising from claims, lawsuits and other legal actions brought by a third party against the Indemnified Contributor to the extent caused by the acts or omissions of such Commercial Contributor in connection with its distribution of the Program in a commercial product offering. The obligations in this section do not apply to any claims or Losses relating to any actual or alleged intellectual property infringement. In order to qualify, an Indemnified Contributor must: a) promptly notify the Commercial Contributor in writing of such claim, and b) allow the Commercial Contributor to control, and cooperate with the Commercial Contributor in, the defense and any related settlement negotiations. The Indemnified Contributor may participate in any such claim at its own expense. + +For example, a Contributor might include the Program in a commercial product offering, Product X. That Contributor is then a Commercial Contributor. If that Commercial Contributor then makes performance claims, or offers warranties related to Product X, those performance claims and warranties are such Commercial Contributor's responsibility alone. Under this section, the Commercial Contributor would have to defend claims against the other Contributors related to those performance claims and warranties, and if a court requires any other Contributor to pay any damages as a result, the Commercial Contributor must pay those damages. + +5. NO WARRANTY + +EXCEPT AS EXPRESSLY SET FORTH IN THIS AGREEMENT, THE PROGRAM IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, EITHER EXPRESS OR IMPLIED INCLUDING, WITHOUT LIMITATION, ANY WARRANTIES OR CONDITIONS OF TITLE, NON-INFRINGEMENT, MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Each Recipient is solely responsible for determining the appropriateness of using and distributing the Program and assumes all risks associated with its exercise of rights under this Agreement , including but not limited to the risks and costs of program errors, compliance with applicable laws, damage to or loss of data, programs or equipment, and unavailability or interruption of operations. + +6. DISCLAIMER OF LIABILITY + +EXCEPT AS EXPRESSLY SET FORTH IN THIS AGREEMENT, NEITHER RECIPIENT NOR ANY CONTRIBUTORS SHALL HAVE ANY LIABILITY FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING WITHOUT LIMITATION LOST PROFITS), HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OR DISTRIBUTION OF THE PROGRAM OR THE EXERCISE OF ANY RIGHTS GRANTED HEREUNDER, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. + +7. GENERAL + +If any provision of this Agreement is invalid or unenforceable under applicable law, it shall not affect the validity or enforceability of the remainder of the terms of this Agreement, and without further action by the parties hereto, such provision shall be reformed to the minimum extent necessary to make such provision valid and enforceable. + +If Recipient institutes patent litigation against any entity (including a cross-claim or counterclaim in a lawsuit) alleging that the Program itself (excluding combinations of the Program with other software or hardware) infringes such Recipient's patent(s), then such Recipient's rights granted under Section 2(b) shall terminate as of the date such litigation is filed. + +All Recipient's rights under this Agreement shall terminate if it fails to comply with any of the material terms or conditions of this Agreement and does not cure such failure in a reasonable period of time after becoming aware of such noncompliance. If all Recipient's rights under this Agreement terminate, Recipient agrees to cease use and distribution of the Program as soon as reasonably practicable. However, Recipient's obligations under this Agreement and any licenses granted by Recipient relating to the Program shall continue and survive. + +Everyone is permitted to copy and distribute copies of this Agreement, but in order to avoid inconsistency the Agreement is copyrighted and may only be modified in the following manner. The Agreement Steward reserves the right to publish new versions (including revisions) of this Agreement from time to time. No one other than the Agreement Steward has the right to modify this Agreement. The Eclipse Foundation is the initial Agreement Steward. The Eclipse Foundation may assign the responsibility to serve as the Agreement Steward to a suitable separate entity. Each new version of the Agreement will be given a distinguishing version number. The Program (including Contributions) may always be distributed subject to the version of the Agreement under which it was received. In addition, after a new version of the Agreement is published, Contributor may elect to distribute the Program (including its Contributions) under the new version. Except as expressly stated in Sections 2(a) and 2(b) above, Recipient receives no rights or licenses to the intellectual property of any Contributor under this Agreement, whether expressly, by implication, estoppel or otherwise. All rights in the Program not expressly granted under this Agreement are reserved. + +This Agreement is governed by the laws of the State of New York and the intellectual property laws of the United States of America. No party to this Agreement will bring a legal action under this Agreement more than one year after the cause of action arose. Each party waives its rights to a jury trial in any resulting litigation. diff --git a/vendor/github.com/eclipse/paho.mqtt.golang/filestore.go b/vendor/github.com/eclipse/paho.mqtt.golang/filestore.go new file mode 100644 index 00000000000..d15f6146c2e --- /dev/null +++ b/vendor/github.com/eclipse/paho.mqtt.golang/filestore.go @@ -0,0 +1,255 @@ +/* + * Copyright (c) 2013 IBM Corp. + * + * All rights reserved. This program and the accompanying materials + * are made available under the terms of the Eclipse Public License v1.0 + * which accompanies this distribution, and is available at + * http://www.eclipse.org/legal/epl-v10.html + * + * Contributors: + * Seth Hoenig + * Allan Stockdill-Mander + * Mike Robertson + */ + +package mqtt + +import ( + "io/ioutil" + "os" + "path" + "sort" + "sync" + + "github.com/eclipse/paho.mqtt.golang/packets" +) + +const ( + msgExt = ".msg" + tmpExt = ".tmp" + corruptExt = ".CORRUPT" +) + +// FileStore implements the store interface using the filesystem to provide +// true persistence, even across client failure. This is designed to use a +// single directory per running client. If you are running multiple clients +// on the same filesystem, you will need to be careful to specify unique +// store directories for each. +type FileStore struct { + sync.RWMutex + directory string + opened bool +} + +// NewFileStore will create a new FileStore which stores its messages in the +// directory provided. +func NewFileStore(directory string) *FileStore { + store := &FileStore{ + directory: directory, + opened: false, + } + return store +} + +// Open will allow the FileStore to be used. +func (store *FileStore) Open() { + store.Lock() + defer store.Unlock() + // if no store directory was specified in ClientOpts, by default use the + // current working directory + if store.directory == "" { + store.directory, _ = os.Getwd() + } + + // if store dir exists, great, otherwise, create it + if !exists(store.directory) { + perms := os.FileMode(0770) + merr := os.MkdirAll(store.directory, perms) + chkerr(merr) + } + store.opened = true + DEBUG.Println(STR, "store is opened at", store.directory) +} + +// Close will disallow the FileStore from being used. +func (store *FileStore) Close() { + store.Lock() + defer store.Unlock() + store.opened = false + DEBUG.Println(STR, "store is closed") +} + +// Put will put a message into the store, associated with the provided +// key value. +func (store *FileStore) Put(key string, m packets.ControlPacket) { + store.Lock() + defer store.Unlock() + if !store.opened { + ERROR.Println(STR, "Trying to use file store, but not open") + return + } + full := fullpath(store.directory, key) + write(store.directory, key, m) + if !exists(full) { + ERROR.Println(STR, "file not created:", full) + } +} + +// Get will retrieve a message from the store, the one associated with +// the provided key value. +func (store *FileStore) Get(key string) packets.ControlPacket { + store.RLock() + defer store.RUnlock() + if !store.opened { + ERROR.Println(STR, "Trying to use file store, but not open") + return nil + } + filepath := fullpath(store.directory, key) + if !exists(filepath) { + return nil + } + mfile, oerr := os.Open(filepath) + chkerr(oerr) + msg, rerr := packets.ReadPacket(mfile) + chkerr(mfile.Close()) + + // Message was unreadable, return nil + if rerr != nil { + newpath := corruptpath(store.directory, key) + WARN.Println(STR, "corrupted file detected:", rerr.Error(), "archived at:", newpath) + os.Rename(filepath, newpath) + return nil + } + return msg +} + +// All will provide a list of all of the keys associated with messages +// currently residing in the FileStore. +func (store *FileStore) All() []string { + store.RLock() + defer store.RUnlock() + return store.all() +} + +// Del will remove the persisted message associated with the provided +// key from the FileStore. +func (store *FileStore) Del(key string) { + store.Lock() + defer store.Unlock() + store.del(key) +} + +// Reset will remove all persisted messages from the FileStore. +func (store *FileStore) Reset() { + store.Lock() + defer store.Unlock() + WARN.Println(STR, "FileStore Reset") + for _, key := range store.all() { + store.del(key) + } +} + +// lockless +func (store *FileStore) all() []string { + var err error + var keys []string + var files fileInfos + + if !store.opened { + ERROR.Println(STR, "Trying to use file store, but not open") + return nil + } + + files, err = ioutil.ReadDir(store.directory) + chkerr(err) + sort.Sort(files) + for _, f := range files { + DEBUG.Println(STR, "file in All():", f.Name()) + name := f.Name() + if name[len(name)-4:] != msgExt { + DEBUG.Println(STR, "skipping file, doesn't have right extension: ", name) + continue + } + key := name[0 : len(name)-4] // remove file extension + keys = append(keys, key) + } + return keys +} + +// lockless +func (store *FileStore) del(key string) { + if !store.opened { + ERROR.Println(STR, "Trying to use file store, but not open") + return + } + DEBUG.Println(STR, "store del filepath:", store.directory) + DEBUG.Println(STR, "store delete key:", key) + filepath := fullpath(store.directory, key) + DEBUG.Println(STR, "path of deletion:", filepath) + if !exists(filepath) { + WARN.Println(STR, "store could not delete key:", key) + return + } + rerr := os.Remove(filepath) + chkerr(rerr) + DEBUG.Println(STR, "del msg:", key) + if exists(filepath) { + ERROR.Println(STR, "file not deleted:", filepath) + } +} + +func fullpath(store string, key string) string { + p := path.Join(store, key+msgExt) + return p +} + +func tmppath(store string, key string) string { + p := path.Join(store, key+tmpExt) + return p +} + +func corruptpath(store string, key string) string { + p := path.Join(store, key+corruptExt) + return p +} + +// create file called "X.[messageid].tmp" located in the store +// the contents of the file is the bytes of the message, then +// rename it to "X.[messageid].msg", overwriting any existing +// message with the same id +// X will be 'i' for inbound messages, and O for outbound messages +func write(store, key string, m packets.ControlPacket) { + temppath := tmppath(store, key) + f, err := os.Create(temppath) + chkerr(err) + werr := m.Write(f) + chkerr(werr) + cerr := f.Close() + chkerr(cerr) + rerr := os.Rename(temppath, fullpath(store, key)) + chkerr(rerr) +} + +func exists(file string) bool { + if _, err := os.Stat(file); err != nil { + if os.IsNotExist(err) { + return false + } + chkerr(err) + } + return true +} + +type fileInfos []os.FileInfo + +func (f fileInfos) Len() int { + return len(f) +} + +func (f fileInfos) Swap(i, j int) { + f[i], f[j] = f[j], f[i] +} + +func (f fileInfos) Less(i, j int) bool { + return f[i].ModTime().Before(f[j].ModTime()) +} diff --git a/vendor/github.com/eclipse/paho.mqtt.golang/memstore.go b/vendor/github.com/eclipse/paho.mqtt.golang/memstore.go new file mode 100644 index 00000000000..499c490bdbb --- /dev/null +++ b/vendor/github.com/eclipse/paho.mqtt.golang/memstore.go @@ -0,0 +1,138 @@ +/* + * Copyright (c) 2013 IBM Corp. + * + * All rights reserved. This program and the accompanying materials + * are made available under the terms of the Eclipse Public License v1.0 + * which accompanies this distribution, and is available at + * http://www.eclipse.org/legal/epl-v10.html + * + * Contributors: + * Seth Hoenig + * Allan Stockdill-Mander + * Mike Robertson + */ + +package mqtt + +import ( + "sync" + + "github.com/eclipse/paho.mqtt.golang/packets" +) + +// MemoryStore implements the store interface to provide a "persistence" +// mechanism wholly stored in memory. This is only useful for +// as long as the client instance exists. +type MemoryStore struct { + sync.RWMutex + messages map[string]packets.ControlPacket + opened bool +} + +// NewMemoryStore returns a pointer to a new instance of +// MemoryStore, the instance is not initialized and ready to +// use until Open() has been called on it. +func NewMemoryStore() *MemoryStore { + store := &MemoryStore{ + messages: make(map[string]packets.ControlPacket), + opened: false, + } + return store +} + +// Open initializes a MemoryStore instance. +func (store *MemoryStore) Open() { + store.Lock() + defer store.Unlock() + store.opened = true + DEBUG.Println(STR, "memorystore initialized") +} + +// Put takes a key and a pointer to a Message and stores the +// message. +func (store *MemoryStore) Put(key string, message packets.ControlPacket) { + store.Lock() + defer store.Unlock() + if !store.opened { + ERROR.Println(STR, "Trying to use memory store, but not open") + return + } + store.messages[key] = message +} + +// Get takes a key and looks in the store for a matching Message +// returning either the Message pointer or nil. +func (store *MemoryStore) Get(key string) packets.ControlPacket { + store.RLock() + defer store.RUnlock() + if !store.opened { + ERROR.Println(STR, "Trying to use memory store, but not open") + return nil + } + mid := mIDFromKey(key) + m := store.messages[key] + if m == nil { + CRITICAL.Println(STR, "memorystore get: message", mid, "not found") + } else { + DEBUG.Println(STR, "memorystore get: message", mid, "found") + } + return m +} + +// All returns a slice of strings containing all the keys currently +// in the MemoryStore. +func (store *MemoryStore) All() []string { + store.RLock() + defer store.RUnlock() + if !store.opened { + ERROR.Println(STR, "Trying to use memory store, but not open") + return nil + } + keys := []string{} + for k := range store.messages { + keys = append(keys, k) + } + return keys +} + +// Del takes a key, searches the MemoryStore and if the key is found +// deletes the Message pointer associated with it. +func (store *MemoryStore) Del(key string) { + store.Lock() + defer store.Unlock() + if !store.opened { + ERROR.Println(STR, "Trying to use memory store, but not open") + return + } + mid := mIDFromKey(key) + m := store.messages[key] + if m == nil { + WARN.Println(STR, "memorystore del: message", mid, "not found") + } else { + delete(store.messages, key) + DEBUG.Println(STR, "memorystore del: message", mid, "was deleted") + } +} + +// Close will disallow modifications to the state of the store. +func (store *MemoryStore) Close() { + store.Lock() + defer store.Unlock() + if !store.opened { + ERROR.Println(STR, "Trying to close memory store, but not open") + return + } + store.opened = false + DEBUG.Println(STR, "memorystore closed") +} + +// Reset eliminates all persisted message data in the store. +func (store *MemoryStore) Reset() { + store.Lock() + defer store.Unlock() + if !store.opened { + ERROR.Println(STR, "Trying to reset memory store, but not open") + } + store.messages = make(map[string]packets.ControlPacket) + WARN.Println(STR, "memorystore wiped") +} diff --git a/vendor/github.com/eclipse/paho.mqtt.golang/message.go b/vendor/github.com/eclipse/paho.mqtt.golang/message.go new file mode 100644 index 00000000000..903e5dcf5e7 --- /dev/null +++ b/vendor/github.com/eclipse/paho.mqtt.golang/message.go @@ -0,0 +1,127 @@ +/* + * Copyright (c) 2013 IBM Corp. + * + * All rights reserved. This program and the accompanying materials + * are made available under the terms of the Eclipse Public License v1.0 + * which accompanies this distribution, and is available at + * http://www.eclipse.org/legal/epl-v10.html + * + * Contributors: + * Seth Hoenig + * Allan Stockdill-Mander + * Mike Robertson + */ + +package mqtt + +import ( + "net/url" + + "github.com/eclipse/paho.mqtt.golang/packets" + "sync" +) + +// Message defines the externals that a message implementation must support +// these are received messages that are passed to the callbacks, not internal +// messages +type Message interface { + Duplicate() bool + Qos() byte + Retained() bool + Topic() string + MessageID() uint16 + Payload() []byte + Ack() +} + +type message struct { + duplicate bool + qos byte + retained bool + topic string + messageID uint16 + payload []byte + once sync.Once + ack func() +} + +func (m *message) Duplicate() bool { + return m.duplicate +} + +func (m *message) Qos() byte { + return m.qos +} + +func (m *message) Retained() bool { + return m.retained +} + +func (m *message) Topic() string { + return m.topic +} + +func (m *message) MessageID() uint16 { + return m.messageID +} + +func (m *message) Payload() []byte { + return m.payload +} + +func (m *message) Ack() { + m.once.Do(m.ack) +} + +func messageFromPublish(p *packets.PublishPacket, ack func()) Message { + return &message{ + duplicate: p.Dup, + qos: p.Qos, + retained: p.Retain, + topic: p.TopicName, + messageID: p.MessageID, + payload: p.Payload, + ack: ack, + } +} + +func newConnectMsgFromOptions(options *ClientOptions, broker *url.URL) *packets.ConnectPacket { + m := packets.NewControlPacket(packets.Connect).(*packets.ConnectPacket) + + m.CleanSession = options.CleanSession + m.WillFlag = options.WillEnabled + m.WillRetain = options.WillRetained + m.ClientIdentifier = options.ClientID + + if options.WillEnabled { + m.WillQos = options.WillQos + m.WillTopic = options.WillTopic + m.WillMessage = options.WillPayload + } + + username := options.Username + password := options.Password + if broker.User != nil { + username = broker.User.Username() + if pwd, ok := broker.User.Password(); ok { + password = pwd + } + } + if options.CredentialsProvider != nil { + username, password = options.CredentialsProvider() + } + + if username != "" { + m.UsernameFlag = true + m.Username = username + //mustn't have password without user as well + if password != "" { + m.PasswordFlag = true + m.Password = []byte(password) + } + } + + m.Keepalive = uint16(options.KeepAlive) + + return m +} diff --git a/vendor/github.com/eclipse/paho.mqtt.golang/messageids.go b/vendor/github.com/eclipse/paho.mqtt.golang/messageids.go new file mode 100644 index 00000000000..e98cc24f846 --- /dev/null +++ b/vendor/github.com/eclipse/paho.mqtt.golang/messageids.go @@ -0,0 +1,141 @@ +/* + * Copyright (c) 2013 IBM Corp. + * + * All rights reserved. This program and the accompanying materials + * are made available under the terms of the Eclipse Public License v1.0 + * which accompanies this distribution, and is available at + * http://www.eclipse.org/legal/epl-v10.html + * + * Contributors: + * Seth Hoenig + * Allan Stockdill-Mander + * Mike Robertson + */ + +package mqtt + +import ( + "fmt" + "sync" + "time" +) + +// MId is 16 bit message id as specified by the MQTT spec. +// In general, these values should not be depended upon by +// the client application. +type MId uint16 + +type messageIds struct { + sync.RWMutex + index map[uint16]tokenCompletor +} + +const ( + midMin uint16 = 1 + midMax uint16 = 65535 +) + +func (mids *messageIds) cleanUp() { + mids.Lock() + for _, token := range mids.index { + switch token.(type) { + case *PublishToken: + token.setError(fmt.Errorf("Connection lost before Publish completed")) + case *SubscribeToken: + token.setError(fmt.Errorf("Connection lost before Subscribe completed")) + case *UnsubscribeToken: + token.setError(fmt.Errorf("Connection lost before Unsubscribe completed")) + case nil: + continue + } + token.flowComplete() + } + mids.index = make(map[uint16]tokenCompletor) + mids.Unlock() + DEBUG.Println(MID, "cleaned up") +} + +func (mids *messageIds) freeID(id uint16) { + mids.Lock() + delete(mids.index, id) + mids.Unlock() +} + +func (mids *messageIds) claimID(token tokenCompletor, id uint16) { + mids.Lock() + defer mids.Unlock() + if _, ok := mids.index[id]; !ok { + mids.index[id] = token + } else { + old := mids.index[id] + old.flowComplete() + mids.index[id] = token + } +} + +func (mids *messageIds) getID(t tokenCompletor) uint16 { + mids.Lock() + defer mids.Unlock() + for i := midMin; i < midMax; i++ { + if _, ok := mids.index[i]; !ok { + mids.index[i] = t + return i + } + } + return 0 +} + +func (mids *messageIds) getToken(id uint16) tokenCompletor { + mids.RLock() + defer mids.RUnlock() + if token, ok := mids.index[id]; ok { + return token + } + return &DummyToken{id: id} +} + +type DummyToken struct { + id uint16 +} + +func (d *DummyToken) Wait() bool { + return true +} + +func (d *DummyToken) WaitTimeout(t time.Duration) bool { + return true +} + +func (d *DummyToken) flowComplete() { + ERROR.Printf("A lookup for token %d returned nil\n", d.id) +} + +func (d *DummyToken) Error() error { + return nil +} + +func (p *DummyToken) setError(e error) {} + +// PlaceHolderToken does nothing and was implemented to allow a messageid to be reserved +// it differs from DummyToken in that calling flowComplete does not generate an error (it +// is expected that flowComplete will be called when the token is overwritten with a real token) +type PlaceHolderToken struct { + id uint16 +} + +func (p *PlaceHolderToken) Wait() bool { + return true +} + +func (p *PlaceHolderToken) WaitTimeout(t time.Duration) bool { + return true +} + +func (p *PlaceHolderToken) flowComplete() { +} + +func (p *PlaceHolderToken) Error() error { + return nil +} + +func (p *PlaceHolderToken) setError(e error) {} diff --git a/vendor/github.com/eclipse/paho.mqtt.golang/net.go b/vendor/github.com/eclipse/paho.mqtt.golang/net.go new file mode 100644 index 00000000000..804cb3fc485 --- /dev/null +++ b/vendor/github.com/eclipse/paho.mqtt.golang/net.go @@ -0,0 +1,330 @@ +/* + * Copyright (c) 2013 IBM Corp. + * + * All rights reserved. This program and the accompanying materials + * are made available under the terms of the Eclipse Public License v1.0 + * which accompanies this distribution, and is available at + * http://www.eclipse.org/legal/epl-v10.html + * + * Contributors: + * Seth Hoenig + * Allan Stockdill-Mander + * Mike Robertson + */ + +package mqtt + +import ( + "crypto/tls" + "errors" + "net" + "net/http" + "net/url" + "os" + "reflect" + "sync/atomic" + "time" + + "github.com/eclipse/paho.mqtt.golang/packets" + "golang.org/x/net/proxy" +) + +func signalError(c chan<- error, err error) { + select { + case c <- err: + default: + } +} + +func openConnection(uri *url.URL, tlsc *tls.Config, timeout time.Duration, headers http.Header) (net.Conn, error) { + switch uri.Scheme { + case "ws": + conn, err := NewWebsocket(uri.String(), nil, timeout, headers) + return conn, err + case "wss": + conn, err := NewWebsocket(uri.String(), tlsc, timeout, headers) + return conn, err + case "tcp": + allProxy := os.Getenv("all_proxy") + if len(allProxy) == 0 { + conn, err := net.DialTimeout("tcp", uri.Host, timeout) + if err != nil { + return nil, err + } + return conn, nil + } + proxyDialer := proxy.FromEnvironment() + + conn, err := proxyDialer.Dial("tcp", uri.Host) + if err != nil { + return nil, err + } + return conn, nil + case "unix": + conn, err := net.DialTimeout("unix", uri.Host, timeout) + if err != nil { + return nil, err + } + return conn, nil + case "ssl": + fallthrough + case "tls": + fallthrough + case "tcps": + allProxy := os.Getenv("all_proxy") + if len(allProxy) == 0 { + conn, err := tls.DialWithDialer(&net.Dialer{Timeout: timeout}, "tcp", uri.Host, tlsc) + if err != nil { + return nil, err + } + return conn, nil + } + proxyDialer := proxy.FromEnvironment() + + conn, err := proxyDialer.Dial("tcp", uri.Host) + if err != nil { + return nil, err + } + + tlsConn := tls.Client(conn, tlsc) + + err = tlsConn.Handshake() + if err != nil { + conn.Close() + return nil, err + } + + return tlsConn, nil + } + return nil, errors.New("Unknown protocol") +} + +// actually read incoming messages off the wire +// send Message object into ibound channel +func incoming(c *client) { + var err error + var cp packets.ControlPacket + + defer c.workers.Done() + + DEBUG.Println(NET, "incoming started") + + for { + if cp, err = packets.ReadPacket(c.conn); err != nil { + break + } + DEBUG.Println(NET, "Received Message") + select { + case c.ibound <- cp: + // Notify keepalive logic that we recently received a packet + if c.options.KeepAlive != 0 { + c.lastReceived.Store(time.Now()) + } + case <-c.stop: + // This avoids a deadlock should a message arrive while shutting down. + // In that case the "reader" of c.ibound might already be gone + WARN.Println(NET, "incoming dropped a received message during shutdown") + break + } + } + // We received an error on read. + // If disconnect is in progress, swallow error and return + select { + case <-c.stop: + DEBUG.Println(NET, "incoming stopped") + return + // Not trying to disconnect, send the error to the errors channel + default: + ERROR.Println(NET, "incoming stopped with error", err) + signalError(c.errors, err) + return + } +} + +// receive a Message object on obound, and then +// actually send outgoing message to the wire +func outgoing(c *client) { + defer c.workers.Done() + DEBUG.Println(NET, "outgoing started") + + for { + DEBUG.Println(NET, "outgoing waiting for an outbound message") + select { + case <-c.stop: + DEBUG.Println(NET, "outgoing stopped") + return + case pub := <-c.obound: + msg := pub.p.(*packets.PublishPacket) + + if c.options.WriteTimeout > 0 { + c.conn.SetWriteDeadline(time.Now().Add(c.options.WriteTimeout)) + } + + if err := msg.Write(c.conn); err != nil { + ERROR.Println(NET, "outgoing stopped with error", err) + pub.t.setError(err) + signalError(c.errors, err) + return + } + + if c.options.WriteTimeout > 0 { + // If we successfully wrote, we don't want the timeout to happen during an idle period + // so we reset it to infinite. + c.conn.SetWriteDeadline(time.Time{}) + } + + if msg.Qos == 0 { + pub.t.flowComplete() + } + DEBUG.Println(NET, "obound wrote msg, id:", msg.MessageID) + case msg := <-c.oboundP: + DEBUG.Println(NET, "obound priority msg to write, type", reflect.TypeOf(msg.p)) + if err := msg.p.Write(c.conn); err != nil { + ERROR.Println(NET, "outgoing stopped with error", err) + if msg.t != nil { + msg.t.setError(err) + } + signalError(c.errors, err) + return + } + switch msg.p.(type) { + case *packets.DisconnectPacket: + msg.t.(*DisconnectToken).flowComplete() + DEBUG.Println(NET, "outbound wrote disconnect, stopping") + return + } + } + // Reset ping timer after sending control packet. + if c.options.KeepAlive != 0 { + c.lastSent.Store(time.Now()) + } + } +} + +// receive Message objects on ibound +// store messages if necessary +// send replies on obound +// delete messages from store if necessary +func alllogic(c *client) { + defer c.workers.Done() + DEBUG.Println(NET, "logic started") + + for { + DEBUG.Println(NET, "logic waiting for msg on ibound") + + select { + case msg := <-c.ibound: + DEBUG.Println(NET, "logic got msg on ibound") + persistInbound(c.persist, msg) + switch m := msg.(type) { + case *packets.PingrespPacket: + DEBUG.Println(NET, "received pingresp") + atomic.StoreInt32(&c.pingOutstanding, 0) + case *packets.SubackPacket: + DEBUG.Println(NET, "received suback, id:", m.MessageID) + token := c.getToken(m.MessageID) + switch t := token.(type) { + case *SubscribeToken: + DEBUG.Println(NET, "granted qoss", m.ReturnCodes) + for i, qos := range m.ReturnCodes { + t.subResult[t.subs[i]] = qos + } + } + token.flowComplete() + c.freeID(m.MessageID) + case *packets.UnsubackPacket: + DEBUG.Println(NET, "received unsuback, id:", m.MessageID) + c.getToken(m.MessageID).flowComplete() + c.freeID(m.MessageID) + case *packets.PublishPacket: + DEBUG.Println(NET, "received publish, msgId:", m.MessageID) + DEBUG.Println(NET, "putting msg on onPubChan") + switch m.Qos { + case 2: + c.incomingPubChan <- m + DEBUG.Println(NET, "done putting msg on incomingPubChan") + case 1: + c.incomingPubChan <- m + DEBUG.Println(NET, "done putting msg on incomingPubChan") + case 0: + select { + case c.incomingPubChan <- m: + case <-c.stop: + } + DEBUG.Println(NET, "done putting msg on incomingPubChan") + } + case *packets.PubackPacket: + DEBUG.Println(NET, "received puback, id:", m.MessageID) + // c.receipts.get(msg.MsgId()) <- Receipt{} + // c.receipts.end(msg.MsgId()) + c.getToken(m.MessageID).flowComplete() + c.freeID(m.MessageID) + case *packets.PubrecPacket: + DEBUG.Println(NET, "received pubrec, id:", m.MessageID) + prel := packets.NewControlPacket(packets.Pubrel).(*packets.PubrelPacket) + prel.MessageID = m.MessageID + select { + case c.oboundP <- &PacketAndToken{p: prel, t: nil}: + case <-c.stop: + } + case *packets.PubrelPacket: + DEBUG.Println(NET, "received pubrel, id:", m.MessageID) + pc := packets.NewControlPacket(packets.Pubcomp).(*packets.PubcompPacket) + pc.MessageID = m.MessageID + persistOutbound(c.persist, pc) + select { + case c.oboundP <- &PacketAndToken{p: pc, t: nil}: + case <-c.stop: + } + case *packets.PubcompPacket: + DEBUG.Println(NET, "received pubcomp, id:", m.MessageID) + c.getToken(m.MessageID).flowComplete() + c.freeID(m.MessageID) + } + case <-c.stop: + WARN.Println(NET, "logic stopped") + return + } + } +} + +func (c *client) ackFunc(packet *packets.PublishPacket) func() { + return func() { + switch packet.Qos { + case 2: + pr := packets.NewControlPacket(packets.Pubrec).(*packets.PubrecPacket) + pr.MessageID = packet.MessageID + DEBUG.Println(NET, "putting pubrec msg on obound") + select { + case c.oboundP <- &PacketAndToken{p: pr, t: nil}: + case <-c.stop: + } + DEBUG.Println(NET, "done putting pubrec msg on obound") + case 1: + pa := packets.NewControlPacket(packets.Puback).(*packets.PubackPacket) + pa.MessageID = packet.MessageID + DEBUG.Println(NET, "putting puback msg on obound") + persistOutbound(c.persist, pa) + select { + case c.oboundP <- &PacketAndToken{p: pa, t: nil}: + case <-c.stop: + } + DEBUG.Println(NET, "done putting puback msg on obound") + case 0: + // do nothing, since there is no need to send an ack packet back + } + } +} + +func errorWatch(c *client) { + defer c.workers.Done() + select { + case <-c.stop: + WARN.Println(NET, "errorWatch stopped") + return + case err := <-c.errors: + ERROR.Println(NET, "error triggered, stopping") + go c.internalConnLost(err) + return + } +} diff --git a/vendor/github.com/eclipse/paho.mqtt.golang/notice.html b/vendor/github.com/eclipse/paho.mqtt.golang/notice.html new file mode 100644 index 00000000000..f19c483b9c8 --- /dev/null +++ b/vendor/github.com/eclipse/paho.mqtt.golang/notice.html @@ -0,0 +1,108 @@ + + + + + +Eclipse Foundation Software User Agreement + + + +

Eclipse Foundation Software User Agreement

+

February 1, 2011

+ +

Usage Of Content

+ +

THE ECLIPSE FOUNDATION MAKES AVAILABLE SOFTWARE, DOCUMENTATION, INFORMATION AND/OR OTHER MATERIALS FOR OPEN SOURCE PROJECTS + (COLLECTIVELY "CONTENT"). USE OF THE CONTENT IS GOVERNED BY THE TERMS AND CONDITIONS OF THIS AGREEMENT AND/OR THE TERMS AND + CONDITIONS OF LICENSE AGREEMENTS OR NOTICES INDICATED OR REFERENCED BELOW. BY USING THE CONTENT, YOU AGREE THAT YOUR USE + OF THE CONTENT IS GOVERNED BY THIS AGREEMENT AND/OR THE TERMS AND CONDITIONS OF ANY APPLICABLE LICENSE AGREEMENTS OR + NOTICES INDICATED OR REFERENCED BELOW. IF YOU DO NOT AGREE TO THE TERMS AND CONDITIONS OF THIS AGREEMENT AND THE TERMS AND + CONDITIONS OF ANY APPLICABLE LICENSE AGREEMENTS OR NOTICES INDICATED OR REFERENCED BELOW, THEN YOU MAY NOT USE THE CONTENT.

+ +

Applicable Licenses

+ +

Unless otherwise indicated, all Content made available by the Eclipse Foundation is provided to you under the terms and conditions of the Eclipse Public License Version 1.0 + ("EPL"). A copy of the EPL is provided with this Content and is also available at http://www.eclipse.org/legal/epl-v10.html. + For purposes of the EPL, "Program" will mean the Content.

+ +

Content includes, but is not limited to, source code, object code, documentation and other files maintained in the Eclipse Foundation source code + repository ("Repository") in software modules ("Modules") and made available as downloadable archives ("Downloads").

+ +
    +
  • Content may be structured and packaged into modules to facilitate delivering, extending, and upgrading the Content. Typical modules may include plug-ins ("Plug-ins"), plug-in fragments ("Fragments"), and features ("Features").
    • +
  • Each Plug-in or Fragment may be packaged as a sub-directory or JAR (Java™ ARchive) in a directory named "plugins".
    • +
  • A Feature is a bundle of one or more Plug-ins and/or Fragments and associated material. Each Feature may be packaged as a sub-directory in a directory named "features". Within a Feature, files named "feature.xml" may contain a list of the names and version numbers of the Plug-ins + and/or Fragments associated with that Feature.
    • +
  • Features may also include other Features ("Included Features"). Within a Feature, files named "feature.xml" may contain a list of the names and version numbers of Included Features.
    • +
+ +

The terms and conditions governing Plug-ins and Fragments should be contained in files named "about.html" ("Abouts"). The terms and conditions governing Features and +Included Features should be contained in files named "license.html" ("Feature Licenses"). Abouts and Feature Licenses may be located in any directory of a Download or Module +including, but not limited to the following locations:

+ +
    +
  • The top-level (root) directory
    • +
  • Plug-in and Fragment directories
    • +
  • Inside Plug-ins and Fragments packaged as JARs
    • +
  • Sub-directories of the directory named "src" of certain Plug-ins
    • +
  • Feature directories
    • +
+ +

Note: if a Feature made available by the Eclipse Foundation is installed using the Provisioning Technology (as defined below), you must agree to a license ("Feature Update License") during the +installation process. If the Feature contains Included Features, the Feature Update License should either provide you with the terms and conditions governing the Included Features or +inform you where you can locate them. Feature Update Licenses may be found in the "license" property of files named "feature.properties" found within a Feature. +Such Abouts, Feature Licenses, and Feature Update Licenses contain the terms and conditions (or references to such terms and conditions) that govern your use of the associated Content in +that directory.

+ +

THE ABOUTS, FEATURE LICENSES, AND FEATURE UPDATE LICENSES MAY REFER TO THE EPL OR OTHER LICENSE AGREEMENTS, NOTICES OR TERMS AND CONDITIONS. SOME OF THESE +OTHER LICENSE AGREEMENTS MAY INCLUDE (BUT ARE NOT LIMITED TO):

+ + + +

IT IS YOUR OBLIGATION TO READ AND ACCEPT ALL SUCH TERMS AND CONDITIONS PRIOR TO USE OF THE CONTENT. If no About, Feature License, or Feature Update License is provided, please +contact the Eclipse Foundation to determine what terms and conditions govern that particular Content.

+ + +

Use of Provisioning Technology

+ +

The Eclipse Foundation makes available provisioning software, examples of which include, but are not limited to, p2 and the Eclipse + Update Manager ("Provisioning Technology") for the purpose of allowing users to install software, documentation, information and/or + other materials (collectively "Installable Software"). This capability is provided with the intent of allowing such users to + install, extend and update Eclipse-based products. Information about packaging Installable Software is available at http://eclipse.org/equinox/p2/repository_packaging.html + ("Specification").

+ +

You may use Provisioning Technology to allow other parties to install Installable Software. You shall be responsible for enabling the + applicable license agreements relating to the Installable Software to be presented to, and accepted by, the users of the Provisioning Technology + in accordance with the Specification. By using Provisioning Technology in such a manner and making it available in accordance with the + Specification, you further acknowledge your agreement to, and the acquisition of all necessary rights to permit the following:

+ +
    +
  1. A series of actions may occur ("Provisioning Process") in which a user may execute the Provisioning Technology + on a machine ("Target Machine") with the intent of installing, extending or updating the functionality of an Eclipse-based + product.
    2. +
  2. During the Provisioning Process, the Provisioning Technology may cause third party Installable Software or a portion thereof to be + accessed and copied to the Target Machine.
    3. +
  3. Pursuant to the Specification, you will provide to the user the terms and conditions that govern the use of the Installable + Software ("Installable Software Agreement") and such Installable Software Agreement shall be accessed from the Target + Machine in accordance with the Specification. Such Installable Software Agreement must inform the user of the terms and conditions that govern + the Installable Software and must solicit acceptance by the end user in the manner prescribed in such Installable Software Agreement. Upon such + indication of agreement by the user, the provisioning Technology will complete installation of the Installable Software.
    4. +
+ +

Cryptography

+ +

Content may contain encryption software. The country in which you are currently may have restrictions on the import, possession, and use, and/or re-export to + another country, of encryption software. BEFORE using any encryption software, please check the country's laws, regulations and policies concerning the import, + possession, or use, and re-export of encryption software, to see if this is permitted.

+ +

Java and all Java-based trademarks are trademarks of Oracle Corporation in the United States, other countries, or both.

+ + diff --git a/vendor/github.com/eclipse/paho.mqtt.golang/oops.go b/vendor/github.com/eclipse/paho.mqtt.golang/oops.go new file mode 100644 index 00000000000..39630d7f28a --- /dev/null +++ b/vendor/github.com/eclipse/paho.mqtt.golang/oops.go @@ -0,0 +1,21 @@ +/* + * Copyright (c) 2013 IBM Corp. + * + * All rights reserved. This program and the accompanying materials + * are made available under the terms of the Eclipse Public License v1.0 + * which accompanies this distribution, and is available at + * http://www.eclipse.org/legal/epl-v10.html + * + * Contributors: + * Seth Hoenig + * Allan Stockdill-Mander + * Mike Robertson + */ + +package mqtt + +func chkerr(e error) { + if e != nil { + panic(e) + } +} diff --git a/vendor/github.com/eclipse/paho.mqtt.golang/options.go b/vendor/github.com/eclipse/paho.mqtt.golang/options.go new file mode 100644 index 00000000000..4a391192d43 --- /dev/null +++ b/vendor/github.com/eclipse/paho.mqtt.golang/options.go @@ -0,0 +1,374 @@ +/* + * Copyright (c) 2013 IBM Corp. + * + * All rights reserved. This program and the accompanying materials + * are made available under the terms of the Eclipse Public License v1.0 + * which accompanies this distribution, and is available at + * http://www.eclipse.org/legal/epl-v10.html + * + * Contributors: + * Seth Hoenig + * Allan Stockdill-Mander + * Mike Robertson + * Måns Ansgariusson + */ + +// Portions copyright © 2018 TIBCO Software Inc. + +package mqtt + +import ( + "crypto/tls" + "net/http" + "net/url" + "regexp" + "strings" + "time" +) + +// CredentialsProvider allows the username and password to be updated +// before reconnecting. It should return the current username and password. +type CredentialsProvider func() (username string, password string) + +// MessageHandler is a callback type which can be set to be +// executed upon the arrival of messages published to topics +// to which the client is subscribed. +type MessageHandler func(Client, Message) + +// ConnectionLostHandler is a callback type which can be set to be +// executed upon an unintended disconnection from the MQTT broker. +// Disconnects caused by calling Disconnect or ForceDisconnect will +// not cause an OnConnectionLost callback to execute. +type ConnectionLostHandler func(Client, error) + +// OnConnectHandler is a callback that is called when the client +// state changes from unconnected/disconnected to connected. Both +// at initial connection and on reconnection +type OnConnectHandler func(Client) + +// ReconnectHandler is invoked prior to reconnecting after +// the initial connection is lost +type ReconnectHandler func(Client, *ClientOptions) + +// ClientOptions contains configurable options for an Client. +type ClientOptions struct { + Servers []*url.URL + ClientID string + Username string + Password string + CredentialsProvider CredentialsProvider + CleanSession bool + Order bool + WillEnabled bool + WillTopic string + WillPayload []byte + WillQos byte + WillRetained bool + ProtocolVersion uint + protocolVersionExplicit bool + TLSConfig *tls.Config + KeepAlive int64 + PingTimeout time.Duration + ConnectTimeout time.Duration + MaxReconnectInterval time.Duration + AutoReconnect bool + ConnectRetryInterval time.Duration + ConnectRetry bool + Store Store + DefaultPublishHandler MessageHandler + OnConnect OnConnectHandler + OnConnectionLost ConnectionLostHandler + OnReconnecting ReconnectHandler + WriteTimeout time.Duration + MessageChannelDepth uint + ResumeSubs bool + HTTPHeaders http.Header +} + +// NewClientOptions will create a new ClientClientOptions type with some +// default values. +// Port: 1883 +// CleanSession: True +// Order: True +// KeepAlive: 30 (seconds) +// ConnectTimeout: 30 (seconds) +// MaxReconnectInterval 10 (minutes) +// AutoReconnect: True +func NewClientOptions() *ClientOptions { + o := &ClientOptions{ + Servers: nil, + ClientID: "", + Username: "", + Password: "", + CleanSession: true, + Order: true, + WillEnabled: false, + WillTopic: "", + WillPayload: nil, + WillQos: 0, + WillRetained: false, + ProtocolVersion: 0, + protocolVersionExplicit: false, + KeepAlive: 30, + PingTimeout: 10 * time.Second, + ConnectTimeout: 30 * time.Second, + MaxReconnectInterval: 10 * time.Minute, + AutoReconnect: true, + ConnectRetryInterval: 30 * time.Second, + ConnectRetry: false, + Store: nil, + OnConnect: nil, + OnConnectionLost: DefaultConnectionLostHandler, + WriteTimeout: 0, // 0 represents timeout disabled + ResumeSubs: false, + HTTPHeaders: make(map[string][]string), + } + return o +} + +// AddBroker adds a broker URI to the list of brokers to be used. The format should be +// scheme://host:port +// Where "scheme" is one of "tcp", "ssl", or "ws", "host" is the ip-address (or hostname) +// and "port" is the port on which the broker is accepting connections. +// +// Default values for hostname is "127.0.0.1", for schema is "tcp://". +// +// An example broker URI would look like: tcp://foobar.com:1883 +func (o *ClientOptions) AddBroker(server string) *ClientOptions { + re := regexp.MustCompile(`%(25)?`) + if len(server) > 0 && server[0] == ':' { + server = "127.0.0.1" + server + } + if !strings.Contains(server, "://") { + server = "tcp://" + server + } + server = re.ReplaceAllLiteralString(server, "%25") + brokerURI, err := url.Parse(server) + if err != nil { + ERROR.Println(CLI, "Failed to parse %q broker address: %s", server, err) + return o + } + o.Servers = append(o.Servers, brokerURI) + return o +} + +// SetResumeSubs will enable resuming of stored (un)subscribe messages when connecting +// but not reconnecting if CleanSession is false. Otherwise these messages are discarded. +func (o *ClientOptions) SetResumeSubs(resume bool) *ClientOptions { + o.ResumeSubs = resume + return o +} + +// SetClientID will set the client id to be used by this client when +// connecting to the MQTT broker. According to the MQTT v3.1 specification, +// a client id must be no longer than 23 characters. +func (o *ClientOptions) SetClientID(id string) *ClientOptions { + o.ClientID = id + return o +} + +// SetUsername will set the username to be used by this client when connecting +// to the MQTT broker. Note: without the use of SSL/TLS, this information will +// be sent in plaintext across the wire. +func (o *ClientOptions) SetUsername(u string) *ClientOptions { + o.Username = u + return o +} + +// SetPassword will set the password to be used by this client when connecting +// to the MQTT broker. Note: without the use of SSL/TLS, this information will +// be sent in plaintext across the wire. +func (o *ClientOptions) SetPassword(p string) *ClientOptions { + o.Password = p + return o +} + +// SetCredentialsProvider will set a method to be called by this client when +// connecting to the MQTT broker that provide the current username and password. +// Note: without the use of SSL/TLS, this information will be sent +// in plaintext across the wire. +func (o *ClientOptions) SetCredentialsProvider(p CredentialsProvider) *ClientOptions { + o.CredentialsProvider = p + return o +} + +// SetCleanSession will set the "clean session" flag in the connect message +// when this client connects to an MQTT broker. By setting this flag, you are +// indicating that no messages saved by the broker for this client should be +// delivered. Any messages that were going to be sent by this client before +// diconnecting previously but didn't will not be sent upon connecting to the +// broker. +func (o *ClientOptions) SetCleanSession(clean bool) *ClientOptions { + o.CleanSession = clean + return o +} + +// SetOrderMatters will set the message routing to guarantee order within +// each QoS level. By default, this value is true. If set to false, +// this flag indicates that messages can be delivered asynchronously +// from the client to the application and possibly arrive out of order. +func (o *ClientOptions) SetOrderMatters(order bool) *ClientOptions { + o.Order = order + return o +} + +// SetTLSConfig will set an SSL/TLS configuration to be used when connecting +// to an MQTT broker. Please read the official Go documentation for more +// information. +func (o *ClientOptions) SetTLSConfig(t *tls.Config) *ClientOptions { + o.TLSConfig = t + return o +} + +// SetStore will set the implementation of the Store interface +// used to provide message persistence in cases where QoS levels +// QoS_ONE or QoS_TWO are used. If no store is provided, then the +// client will use MemoryStore by default. +func (o *ClientOptions) SetStore(s Store) *ClientOptions { + o.Store = s + return o +} + +// SetKeepAlive will set the amount of time (in seconds) that the client +// should wait before sending a PING request to the broker. This will +// allow the client to know that a connection has not been lost with the +// server. +func (o *ClientOptions) SetKeepAlive(k time.Duration) *ClientOptions { + o.KeepAlive = int64(k / time.Second) + return o +} + +// SetPingTimeout will set the amount of time (in seconds) that the client +// will wait after sending a PING request to the broker, before deciding +// that the connection has been lost. Default is 10 seconds. +func (o *ClientOptions) SetPingTimeout(k time.Duration) *ClientOptions { + o.PingTimeout = k + return o +} + +// SetProtocolVersion sets the MQTT version to be used to connect to the +// broker. Legitimate values are currently 3 - MQTT 3.1 or 4 - MQTT 3.1.1 +func (o *ClientOptions) SetProtocolVersion(pv uint) *ClientOptions { + if (pv >= 3 && pv <= 4) || (pv > 0x80) { + o.ProtocolVersion = pv + o.protocolVersionExplicit = true + } + return o +} + +// UnsetWill will cause any set will message to be disregarded. +func (o *ClientOptions) UnsetWill() *ClientOptions { + o.WillEnabled = false + return o +} + +// SetWill accepts a string will message to be set. When the client connects, +// it will give this will message to the broker, which will then publish the +// provided payload (the will) to any clients that are subscribed to the provided +// topic. +func (o *ClientOptions) SetWill(topic string, payload string, qos byte, retained bool) *ClientOptions { + o.SetBinaryWill(topic, []byte(payload), qos, retained) + return o +} + +// SetBinaryWill accepts a []byte will message to be set. When the client connects, +// it will give this will message to the broker, which will then publish the +// provided payload (the will) to any clients that are subscribed to the provided +// topic. +func (o *ClientOptions) SetBinaryWill(topic string, payload []byte, qos byte, retained bool) *ClientOptions { + o.WillEnabled = true + o.WillTopic = topic + o.WillPayload = payload + o.WillQos = qos + o.WillRetained = retained + return o +} + +// SetDefaultPublishHandler sets the MessageHandler that will be called when a message +// is received that does not match any known subscriptions. +func (o *ClientOptions) SetDefaultPublishHandler(defaultHandler MessageHandler) *ClientOptions { + o.DefaultPublishHandler = defaultHandler + return o +} + +// SetOnConnectHandler sets the function to be called when the client is connected. Both +// at initial connection time and upon automatic reconnect. +func (o *ClientOptions) SetOnConnectHandler(onConn OnConnectHandler) *ClientOptions { + o.OnConnect = onConn + return o +} + +// SetConnectionLostHandler will set the OnConnectionLost callback to be executed +// in the case where the client unexpectedly loses connection with the MQTT broker. +func (o *ClientOptions) SetConnectionLostHandler(onLost ConnectionLostHandler) *ClientOptions { + o.OnConnectionLost = onLost + return o +} + +// SetReconnectingHandler sets the OnReconnecting callback to be executed prior +// to the client attempting a reconnect to the MQTT broker. +func (o *ClientOptions) SetReconnectingHandler(cb ReconnectHandler) *ClientOptions { + o.OnReconnecting = cb + return o +} + +// SetWriteTimeout puts a limit on how long a mqtt publish should block until it unblocks with a +// timeout error. A duration of 0 never times out. Default 30 seconds +func (o *ClientOptions) SetWriteTimeout(t time.Duration) *ClientOptions { + o.WriteTimeout = t + return o +} + +// SetConnectTimeout limits how long the client will wait when trying to open a connection +// to an MQTT server before timing out and erroring the attempt. A duration of 0 never times out. +// Default 30 seconds. Currently only operational on TCP/TLS connections. +func (o *ClientOptions) SetConnectTimeout(t time.Duration) *ClientOptions { + o.ConnectTimeout = t + return o +} + +// SetMaxReconnectInterval sets the maximum time that will be waited between reconnection attempts +// when connection is lost +func (o *ClientOptions) SetMaxReconnectInterval(t time.Duration) *ClientOptions { + o.MaxReconnectInterval = t + return o +} + +// SetAutoReconnect sets whether the automatic reconnection logic should be used +// when the connection is lost, even if disabled the ConnectionLostHandler is still +// called +func (o *ClientOptions) SetAutoReconnect(a bool) *ClientOptions { + o.AutoReconnect = a + return o +} + +// SetConnectRetryInterval sets the time that will be waited between connection attempts +// when initially connecting if ConnectRetry is TRUE +func (o *ClientOptions) SetConnectRetryInterval(t time.Duration) *ClientOptions { + o.ConnectRetryInterval = t + return o +} + +// SetConnectRetry sets whether the connect function will automatically retry the connection +// in the event of a failure (when true the token returned by the Connect function will +// not complete until the connection is up or it is cancelled) +// If ConnectRetry is true then subscriptions should be requested in OnConnect handler +// Setting this to TRUE permits mesages to be published before the connection is established +func (o *ClientOptions) SetConnectRetry(a bool) *ClientOptions { + o.ConnectRetry = a + return o +} + +// SetMessageChannelDepth DEPRECATED The value set here no longer has any effect, this function +// remains so the API is not altered. +func (o *ClientOptions) SetMessageChannelDepth(s uint) *ClientOptions { + o.MessageChannelDepth = s + return o +} + +// SetHTTPHeaders sets the additional HTTP headers that will be sent in the WebSocket +// opening handshake. +func (o *ClientOptions) SetHTTPHeaders(h http.Header) *ClientOptions { + o.HTTPHeaders = h + return o +} diff --git a/vendor/github.com/eclipse/paho.mqtt.golang/options_reader.go b/vendor/github.com/eclipse/paho.mqtt.golang/options_reader.go new file mode 100644 index 00000000000..0f252e88395 --- /dev/null +++ b/vendor/github.com/eclipse/paho.mqtt.golang/options_reader.go @@ -0,0 +1,161 @@ +/* + * Copyright (c) 2013 IBM Corp. + * + * All rights reserved. This program and the accompanying materials + * are made available under the terms of the Eclipse Public License v1.0 + * which accompanies this distribution, and is available at + * http://www.eclipse.org/legal/epl-v10.html + * + * Contributors: + * Seth Hoenig + * Allan Stockdill-Mander + * Mike Robertson + */ + +package mqtt + +import ( + "crypto/tls" + "net/http" + "net/url" + "time" +) + +// ClientOptionsReader provides an interface for reading ClientOptions after the client has been initialized. +type ClientOptionsReader struct { + options *ClientOptions +} + +//Servers returns a slice of the servers defined in the clientoptions +func (r *ClientOptionsReader) Servers() []*url.URL { + s := make([]*url.URL, len(r.options.Servers)) + + for i, u := range r.options.Servers { + nu := *u + s[i] = &nu + } + + return s +} + +//ResumeSubs returns true if resuming stored (un)sub is enabled +func (r *ClientOptionsReader) ResumeSubs() bool { + s := r.options.ResumeSubs + return s +} + +//ClientID returns the set client id +func (r *ClientOptionsReader) ClientID() string { + s := r.options.ClientID + return s +} + +//Username returns the set username +func (r *ClientOptionsReader) Username() string { + s := r.options.Username + return s +} + +//Password returns the set password +func (r *ClientOptionsReader) Password() string { + s := r.options.Password + return s +} + +//CleanSession returns whether Cleansession is set +func (r *ClientOptionsReader) CleanSession() bool { + s := r.options.CleanSession + return s +} + +func (r *ClientOptionsReader) Order() bool { + s := r.options.Order + return s +} + +func (r *ClientOptionsReader) WillEnabled() bool { + s := r.options.WillEnabled + return s +} + +func (r *ClientOptionsReader) WillTopic() string { + s := r.options.WillTopic + return s +} + +func (r *ClientOptionsReader) WillPayload() []byte { + s := r.options.WillPayload + return s +} + +func (r *ClientOptionsReader) WillQos() byte { + s := r.options.WillQos + return s +} + +func (r *ClientOptionsReader) WillRetained() bool { + s := r.options.WillRetained + return s +} + +func (r *ClientOptionsReader) ProtocolVersion() uint { + s := r.options.ProtocolVersion + return s +} + +func (r *ClientOptionsReader) TLSConfig() *tls.Config { + s := r.options.TLSConfig + return s +} + +func (r *ClientOptionsReader) KeepAlive() time.Duration { + s := time.Duration(r.options.KeepAlive * int64(time.Second)) + return s +} + +func (r *ClientOptionsReader) PingTimeout() time.Duration { + s := r.options.PingTimeout + return s +} + +func (r *ClientOptionsReader) ConnectTimeout() time.Duration { + s := r.options.ConnectTimeout + return s +} + +func (r *ClientOptionsReader) MaxReconnectInterval() time.Duration { + s := r.options.MaxReconnectInterval + return s +} + +func (r *ClientOptionsReader) AutoReconnect() bool { + s := r.options.AutoReconnect + return s +} + +//ConnectRetryInterval returns the delay between retries on the initial connection (if ConnectRetry true) +func (r *ClientOptionsReader) ConnectRetryInterval() time.Duration { + s := r.options.ConnectRetryInterval + return s +} + +//ConnectRetry returns whether the initial connection request will be retried until connection established +func (r *ClientOptionsReader) ConnectRetry() bool { + s := r.options.ConnectRetry + return s +} + +func (r *ClientOptionsReader) WriteTimeout() time.Duration { + s := r.options.WriteTimeout + return s +} + +func (r *ClientOptionsReader) MessageChannelDepth() uint { + s := r.options.MessageChannelDepth + return s +} + +func (r *ClientOptionsReader) HTTPHeaders() http.Header { + h := r.options.HTTPHeaders + return h +} diff --git a/vendor/github.com/eclipse/paho.mqtt.golang/packets/connack.go b/vendor/github.com/eclipse/paho.mqtt.golang/packets/connack.go new file mode 100644 index 00000000000..de58c81e9ee --- /dev/null +++ b/vendor/github.com/eclipse/paho.mqtt.golang/packets/connack.go @@ -0,0 +1,52 @@ +package packets + +import ( + "bytes" + "fmt" + "io" +) + +//ConnackPacket is an internal representation of the fields of the +//Connack MQTT packet +type ConnackPacket struct { + FixedHeader + SessionPresent bool + ReturnCode byte +} + +func (ca *ConnackPacket) String() string { + return fmt.Sprintf("%s sessionpresent: %t returncode: %d", ca.FixedHeader, ca.SessionPresent, ca.ReturnCode) +} + +func (ca *ConnackPacket) Write(w io.Writer) error { + var body bytes.Buffer + var err error + + body.WriteByte(boolToByte(ca.SessionPresent)) + body.WriteByte(ca.ReturnCode) + ca.FixedHeader.RemainingLength = 2 + packet := ca.FixedHeader.pack() + packet.Write(body.Bytes()) + _, err = packet.WriteTo(w) + + return err +} + +//Unpack decodes the details of a ControlPacket after the fixed +//header has been read +func (ca *ConnackPacket) Unpack(b io.Reader) error { + flags, err := decodeByte(b) + if err != nil { + return err + } + ca.SessionPresent = 1&flags > 0 + ca.ReturnCode, err = decodeByte(b) + + return err +} + +//Details returns a Details struct containing the Qos and +//MessageID of this ControlPacket +func (ca *ConnackPacket) Details() Details { + return Details{Qos: 0, MessageID: 0} +} diff --git a/vendor/github.com/eclipse/paho.mqtt.golang/packets/connect.go b/vendor/github.com/eclipse/paho.mqtt.golang/packets/connect.go new file mode 100644 index 00000000000..da0fc05226e --- /dev/null +++ b/vendor/github.com/eclipse/paho.mqtt.golang/packets/connect.go @@ -0,0 +1,151 @@ +package packets + +import ( + "bytes" + "fmt" + "io" +) + +//ConnectPacket is an internal representation of the fields of the +//Connect MQTT packet +type ConnectPacket struct { + FixedHeader + ProtocolName string + ProtocolVersion byte + CleanSession bool + WillFlag bool + WillQos byte + WillRetain bool + UsernameFlag bool + PasswordFlag bool + ReservedBit byte + Keepalive uint16 + + ClientIdentifier string + WillTopic string + WillMessage []byte + Username string + Password []byte +} + +func (c *ConnectPacket) String() string { + return fmt.Sprintf("%s protocolversion: %d protocolname: %s cleansession: %t willflag: %t WillQos: %d WillRetain: %t Usernameflag: %t Passwordflag: %t keepalive: %d clientId: %s willtopic: %s willmessage: %s Username: %s Password: %s", c.FixedHeader, c.ProtocolVersion, c.ProtocolName, c.CleanSession, c.WillFlag, c.WillQos, c.WillRetain, c.UsernameFlag, c.PasswordFlag, c.Keepalive, c.ClientIdentifier, c.WillTopic, c.WillMessage, c.Username, c.Password) +} + +func (c *ConnectPacket) Write(w io.Writer) error { + var body bytes.Buffer + var err error + + body.Write(encodeString(c.ProtocolName)) + body.WriteByte(c.ProtocolVersion) + body.WriteByte(boolToByte(c.CleanSession)<<1 | boolToByte(c.WillFlag)<<2 | c.WillQos<<3 | boolToByte(c.WillRetain)<<5 | boolToByte(c.PasswordFlag)<<6 | boolToByte(c.UsernameFlag)<<7) + body.Write(encodeUint16(c.Keepalive)) + body.Write(encodeString(c.ClientIdentifier)) + if c.WillFlag { + body.Write(encodeString(c.WillTopic)) + body.Write(encodeBytes(c.WillMessage)) + } + if c.UsernameFlag { + body.Write(encodeString(c.Username)) + } + if c.PasswordFlag { + body.Write(encodeBytes(c.Password)) + } + c.FixedHeader.RemainingLength = body.Len() + packet := c.FixedHeader.pack() + packet.Write(body.Bytes()) + _, err = packet.WriteTo(w) + + return err +} + +//Unpack decodes the details of a ControlPacket after the fixed +//header has been read +func (c *ConnectPacket) Unpack(b io.Reader) error { + var err error + c.ProtocolName, err = decodeString(b) + if err != nil { + return err + } + c.ProtocolVersion, err = decodeByte(b) + if err != nil { + return err + } + options, err := decodeByte(b) + if err != nil { + return err + } + c.ReservedBit = 1 & options + c.CleanSession = 1&(options>>1) > 0 + c.WillFlag = 1&(options>>2) > 0 + c.WillQos = 3 & (options >> 3) + c.WillRetain = 1&(options>>5) > 0 + c.PasswordFlag = 1&(options>>6) > 0 + c.UsernameFlag = 1&(options>>7) > 0 + c.Keepalive, err = decodeUint16(b) + if err != nil { + return err + } + c.ClientIdentifier, err = decodeString(b) + if err != nil { + return err + } + if c.WillFlag { + c.WillTopic, err = decodeString(b) + if err != nil { + return err + } + c.WillMessage, err = decodeBytes(b) + if err != nil { + return err + } + } + if c.UsernameFlag { + c.Username, err = decodeString(b) + if err != nil { + return err + } + } + if c.PasswordFlag { + c.Password, err = decodeBytes(b) + if err != nil { + return err + } + } + + return nil +} + +//Validate performs validation of the fields of a Connect packet +func (c *ConnectPacket) Validate() byte { + if c.PasswordFlag && !c.UsernameFlag { + return ErrRefusedBadUsernameOrPassword + } + if c.ReservedBit != 0 { + //Bad reserved bit + return ErrProtocolViolation + } + if (c.ProtocolName == "MQIsdp" && c.ProtocolVersion != 3) || (c.ProtocolName == "MQTT" && c.ProtocolVersion != 4) { + //Mismatched or unsupported protocol version + return ErrRefusedBadProtocolVersion + } + if c.ProtocolName != "MQIsdp" && c.ProtocolName != "MQTT" { + //Bad protocol name + return ErrProtocolViolation + } + if len(c.ClientIdentifier) > 65535 || len(c.Username) > 65535 || len(c.Password) > 65535 { + //Bad size field + return ErrProtocolViolation + } + if len(c.ClientIdentifier) == 0 && !c.CleanSession { + //Bad client identifier + return ErrRefusedIDRejected + } + return Accepted +} + +//Details returns a Details struct containing the Qos and +//MessageID of this ControlPacket +func (c *ConnectPacket) Details() Details { + return Details{Qos: 0, MessageID: 0} +} diff --git a/vendor/github.com/eclipse/paho.mqtt.golang/packets/disconnect.go b/vendor/github.com/eclipse/paho.mqtt.golang/packets/disconnect.go new file mode 100644 index 00000000000..c8d374508cf --- /dev/null +++ b/vendor/github.com/eclipse/paho.mqtt.golang/packets/disconnect.go @@ -0,0 +1,34 @@ +package packets + +import ( + "io" +) + +//DisconnectPacket is an internal representation of the fields of the +//Disconnect MQTT packet +type DisconnectPacket struct { + FixedHeader +} + +func (d *DisconnectPacket) String() string { + return d.FixedHeader.String() +} + +func (d *DisconnectPacket) Write(w io.Writer) error { + packet := d.FixedHeader.pack() + _, err := packet.WriteTo(w) + + return err +} + +//Unpack decodes the details of a ControlPacket after the fixed +//header has been read +func (d *DisconnectPacket) Unpack(b io.Reader) error { + return nil +} + +//Details returns a Details struct containing the Qos and +//MessageID of this ControlPacket +func (d *DisconnectPacket) Details() Details { + return Details{Qos: 0, MessageID: 0} +} diff --git a/vendor/github.com/eclipse/paho.mqtt.golang/packets/packets.go b/vendor/github.com/eclipse/paho.mqtt.golang/packets/packets.go new file mode 100644 index 00000000000..42eeb46d39c --- /dev/null +++ b/vendor/github.com/eclipse/paho.mqtt.golang/packets/packets.go @@ -0,0 +1,346 @@ +package packets + +import ( + "bytes" + "encoding/binary" + "errors" + "fmt" + "io" +) + +//ControlPacket defines the interface for structs intended to hold +//decoded MQTT packets, either from being read or before being +//written +type ControlPacket interface { + Write(io.Writer) error + Unpack(io.Reader) error + String() string + Details() Details +} + +//PacketNames maps the constants for each of the MQTT packet types +//to a string representation of their name. +var PacketNames = map[uint8]string{ + 1: "CONNECT", + 2: "CONNACK", + 3: "PUBLISH", + 4: "PUBACK", + 5: "PUBREC", + 6: "PUBREL", + 7: "PUBCOMP", + 8: "SUBSCRIBE", + 9: "SUBACK", + 10: "UNSUBSCRIBE", + 11: "UNSUBACK", + 12: "PINGREQ", + 13: "PINGRESP", + 14: "DISCONNECT", +} + +//Below are the constants assigned to each of the MQTT packet types +const ( + Connect = 1 + Connack = 2 + Publish = 3 + Puback = 4 + Pubrec = 5 + Pubrel = 6 + Pubcomp = 7 + Subscribe = 8 + Suback = 9 + Unsubscribe = 10 + Unsuback = 11 + Pingreq = 12 + Pingresp = 13 + Disconnect = 14 +) + +//Below are the const definitions for error codes returned by +//Connect() +const ( + Accepted = 0x00 + ErrRefusedBadProtocolVersion = 0x01 + ErrRefusedIDRejected = 0x02 + ErrRefusedServerUnavailable = 0x03 + ErrRefusedBadUsernameOrPassword = 0x04 + ErrRefusedNotAuthorised = 0x05 + ErrNetworkError = 0xFE + ErrProtocolViolation = 0xFF +) + +//ConnackReturnCodes is a map of the error codes constants for Connect() +//to a string representation of the error +var ConnackReturnCodes = map[uint8]string{ + 0: "Connection Accepted", + 1: "Connection Refused: Bad Protocol Version", + 2: "Connection Refused: Client Identifier Rejected", + 3: "Connection Refused: Server Unavailable", + 4: "Connection Refused: Username or Password in unknown format", + 5: "Connection Refused: Not Authorised", + 254: "Connection Error", + 255: "Connection Refused: Protocol Violation", +} + +//ConnErrors is a map of the errors codes constants for Connect() +//to a Go error +var ConnErrors = map[byte]error{ + Accepted: nil, + ErrRefusedBadProtocolVersion: errors.New("Unnacceptable protocol version"), + ErrRefusedIDRejected: errors.New("Identifier rejected"), + ErrRefusedServerUnavailable: errors.New("Server Unavailable"), + ErrRefusedBadUsernameOrPassword: errors.New("Bad user name or password"), + ErrRefusedNotAuthorised: errors.New("Not Authorized"), + ErrNetworkError: errors.New("Network Error"), + ErrProtocolViolation: errors.New("Protocol Violation"), +} + +//ReadPacket takes an instance of an io.Reader (such as net.Conn) and attempts +//to read an MQTT packet from the stream. It returns a ControlPacket +//representing the decoded MQTT packet and an error. One of these returns will +//always be nil, a nil ControlPacket indicating an error occurred. +func ReadPacket(r io.Reader) (ControlPacket, error) { + var fh FixedHeader + b := make([]byte, 1) + + _, err := io.ReadFull(r, b) + if err != nil { + return nil, err + } + + err = fh.unpack(b[0], r) + if err != nil { + return nil, err + } + + cp, err := NewControlPacketWithHeader(fh) + if err != nil { + return nil, err + } + + packetBytes := make([]byte, fh.RemainingLength) + n, err := io.ReadFull(r, packetBytes) + if err != nil { + return nil, err + } + if n != fh.RemainingLength { + return nil, errors.New("Failed to read expected data") + } + + err = cp.Unpack(bytes.NewBuffer(packetBytes)) + return cp, err +} + +//NewControlPacket is used to create a new ControlPacket of the type specified +//by packetType, this is usually done by reference to the packet type constants +//defined in packets.go. The newly created ControlPacket is empty and a pointer +//is returned. +func NewControlPacket(packetType byte) ControlPacket { + switch packetType { + case Connect: + return &ConnectPacket{FixedHeader: FixedHeader{MessageType: Connect}} + case Connack: + return &ConnackPacket{FixedHeader: FixedHeader{MessageType: Connack}} + case Disconnect: + return &DisconnectPacket{FixedHeader: FixedHeader{MessageType: Disconnect}} + case Publish: + return &PublishPacket{FixedHeader: FixedHeader{MessageType: Publish}} + case Puback: + return &PubackPacket{FixedHeader: FixedHeader{MessageType: Puback}} + case Pubrec: + return &PubrecPacket{FixedHeader: FixedHeader{MessageType: Pubrec}} + case Pubrel: + return &PubrelPacket{FixedHeader: FixedHeader{MessageType: Pubrel, Qos: 1}} + case Pubcomp: + return &PubcompPacket{FixedHeader: FixedHeader{MessageType: Pubcomp}} + case Subscribe: + return &SubscribePacket{FixedHeader: FixedHeader{MessageType: Subscribe, Qos: 1}} + case Suback: + return &SubackPacket{FixedHeader: FixedHeader{MessageType: Suback}} + case Unsubscribe: + return &UnsubscribePacket{FixedHeader: FixedHeader{MessageType: Unsubscribe, Qos: 1}} + case Unsuback: + return &UnsubackPacket{FixedHeader: FixedHeader{MessageType: Unsuback}} + case Pingreq: + return &PingreqPacket{FixedHeader: FixedHeader{MessageType: Pingreq}} + case Pingresp: + return &PingrespPacket{FixedHeader: FixedHeader{MessageType: Pingresp}} + } + return nil +} + +//NewControlPacketWithHeader is used to create a new ControlPacket of the type +//specified within the FixedHeader that is passed to the function. +//The newly created ControlPacket is empty and a pointer is returned. +func NewControlPacketWithHeader(fh FixedHeader) (ControlPacket, error) { + switch fh.MessageType { + case Connect: + return &ConnectPacket{FixedHeader: fh}, nil + case Connack: + return &ConnackPacket{FixedHeader: fh}, nil + case Disconnect: + return &DisconnectPacket{FixedHeader: fh}, nil + case Publish: + return &PublishPacket{FixedHeader: fh}, nil + case Puback: + return &PubackPacket{FixedHeader: fh}, nil + case Pubrec: + return &PubrecPacket{FixedHeader: fh}, nil + case Pubrel: + return &PubrelPacket{FixedHeader: fh}, nil + case Pubcomp: + return &PubcompPacket{FixedHeader: fh}, nil + case Subscribe: + return &SubscribePacket{FixedHeader: fh}, nil + case Suback: + return &SubackPacket{FixedHeader: fh}, nil + case Unsubscribe: + return &UnsubscribePacket{FixedHeader: fh}, nil + case Unsuback: + return &UnsubackPacket{FixedHeader: fh}, nil + case Pingreq: + return &PingreqPacket{FixedHeader: fh}, nil + case Pingresp: + return &PingrespPacket{FixedHeader: fh}, nil + } + return nil, fmt.Errorf("unsupported packet type 0x%x", fh.MessageType) +} + +//Details struct returned by the Details() function called on +//ControlPackets to present details of the Qos and MessageID +//of the ControlPacket +type Details struct { + Qos byte + MessageID uint16 +} + +//FixedHeader is a struct to hold the decoded information from +//the fixed header of an MQTT ControlPacket +type FixedHeader struct { + MessageType byte + Dup bool + Qos byte + Retain bool + RemainingLength int +} + +func (fh FixedHeader) String() string { + return fmt.Sprintf("%s: dup: %t qos: %d retain: %t rLength: %d", PacketNames[fh.MessageType], fh.Dup, fh.Qos, fh.Retain, fh.RemainingLength) +} + +func boolToByte(b bool) byte { + switch b { + case true: + return 1 + default: + return 0 + } +} + +func (fh *FixedHeader) pack() bytes.Buffer { + var header bytes.Buffer + header.WriteByte(fh.MessageType<<4 | boolToByte(fh.Dup)<<3 | fh.Qos<<1 | boolToByte(fh.Retain)) + header.Write(encodeLength(fh.RemainingLength)) + return header +} + +func (fh *FixedHeader) unpack(typeAndFlags byte, r io.Reader) error { + fh.MessageType = typeAndFlags >> 4 + fh.Dup = (typeAndFlags>>3)&0x01 > 0 + fh.Qos = (typeAndFlags >> 1) & 0x03 + fh.Retain = typeAndFlags&0x01 > 0 + + var err error + fh.RemainingLength, err = decodeLength(r) + return err +} + +func decodeByte(b io.Reader) (byte, error) { + num := make([]byte, 1) + _, err := b.Read(num) + if err != nil { + return 0, err + } + + return num[0], nil +} + +func decodeUint16(b io.Reader) (uint16, error) { + num := make([]byte, 2) + _, err := b.Read(num) + if err != nil { + return 0, err + } + return binary.BigEndian.Uint16(num), nil +} + +func encodeUint16(num uint16) []byte { + bytes := make([]byte, 2) + binary.BigEndian.PutUint16(bytes, num) + return bytes +} + +func encodeString(field string) []byte { + return encodeBytes([]byte(field)) +} + +func decodeString(b io.Reader) (string, error) { + buf, err := decodeBytes(b) + return string(buf), err +} + +func decodeBytes(b io.Reader) ([]byte, error) { + fieldLength, err := decodeUint16(b) + if err != nil { + return nil, err + } + + field := make([]byte, fieldLength) + _, err = b.Read(field) + if err != nil { + return nil, err + } + + return field, nil +} + +func encodeBytes(field []byte) []byte { + fieldLength := make([]byte, 2) + binary.BigEndian.PutUint16(fieldLength, uint16(len(field))) + return append(fieldLength, field...) +} + +func encodeLength(length int) []byte { + var encLength []byte + for { + digit := byte(length % 128) + length /= 128 + if length > 0 { + digit |= 0x80 + } + encLength = append(encLength, digit) + if length == 0 { + break + } + } + return encLength +} + +func decodeLength(r io.Reader) (int, error) { + var rLength uint32 + var multiplier uint32 + b := make([]byte, 1) + for multiplier < 27 { //fix: Infinite '(digit & 128) == 1' will cause the dead loop + _, err := io.ReadFull(r, b) + if err != nil { + return 0, err + } + + digit := b[0] + rLength |= uint32(digit&127) << multiplier + if (digit & 128) == 0 { + break + } + multiplier += 7 + } + return int(rLength), nil +} diff --git a/vendor/github.com/eclipse/paho.mqtt.golang/packets/pingreq.go b/vendor/github.com/eclipse/paho.mqtt.golang/packets/pingreq.go new file mode 100644 index 00000000000..27e49fedadd --- /dev/null +++ b/vendor/github.com/eclipse/paho.mqtt.golang/packets/pingreq.go @@ -0,0 +1,34 @@ +package packets + +import ( + "io" +) + +//PingreqPacket is an internal representation of the fields of the +//Pingreq MQTT packet +type PingreqPacket struct { + FixedHeader +} + +func (pr *PingreqPacket) String() string { + return pr.FixedHeader.String() +} + +func (pr *PingreqPacket) Write(w io.Writer) error { + packet := pr.FixedHeader.pack() + _, err := packet.WriteTo(w) + + return err +} + +//Unpack decodes the details of a ControlPacket after the fixed +//header has been read +func (pr *PingreqPacket) Unpack(b io.Reader) error { + return nil +} + +//Details returns a Details struct containing the Qos and +//MessageID of this ControlPacket +func (pr *PingreqPacket) Details() Details { + return Details{Qos: 0, MessageID: 0} +} diff --git a/vendor/github.com/eclipse/paho.mqtt.golang/packets/pingresp.go b/vendor/github.com/eclipse/paho.mqtt.golang/packets/pingresp.go new file mode 100644 index 00000000000..86f6d926123 --- /dev/null +++ b/vendor/github.com/eclipse/paho.mqtt.golang/packets/pingresp.go @@ -0,0 +1,34 @@ +package packets + +import ( + "io" +) + +//PingrespPacket is an internal representation of the fields of the +//Pingresp MQTT packet +type PingrespPacket struct { + FixedHeader +} + +func (pr *PingrespPacket) String() string { + return pr.FixedHeader.String() +} + +func (pr *PingrespPacket) Write(w io.Writer) error { + packet := pr.FixedHeader.pack() + _, err := packet.WriteTo(w) + + return err +} + +//Unpack decodes the details of a ControlPacket after the fixed +//header has been read +func (pr *PingrespPacket) Unpack(b io.Reader) error { + return nil +} + +//Details returns a Details struct containing the Qos and +//MessageID of this ControlPacket +func (pr *PingrespPacket) Details() Details { + return Details{Qos: 0, MessageID: 0} +} diff --git a/vendor/github.com/eclipse/paho.mqtt.golang/packets/puback.go b/vendor/github.com/eclipse/paho.mqtt.golang/packets/puback.go new file mode 100644 index 00000000000..4f2ee0bbd1c --- /dev/null +++ b/vendor/github.com/eclipse/paho.mqtt.golang/packets/puback.go @@ -0,0 +1,42 @@ +package packets + +import ( + "fmt" + "io" +) + +//PubackPacket is an internal representation of the fields of the +//Puback MQTT packet +type PubackPacket struct { + FixedHeader + MessageID uint16 +} + +func (pa *PubackPacket) String() string { + return fmt.Sprintf("%s MessageID: %d", pa.FixedHeader, pa.MessageID) +} + +func (pa *PubackPacket) Write(w io.Writer) error { + var err error + pa.FixedHeader.RemainingLength = 2 + packet := pa.FixedHeader.pack() + packet.Write(encodeUint16(pa.MessageID)) + _, err = packet.WriteTo(w) + + return err +} + +//Unpack decodes the details of a ControlPacket after the fixed +//header has been read +func (pa *PubackPacket) Unpack(b io.Reader) error { + var err error + pa.MessageID, err = decodeUint16(b) + + return err +} + +//Details returns a Details struct containing the Qos and +//MessageID of this ControlPacket +func (pa *PubackPacket) Details() Details { + return Details{Qos: pa.Qos, MessageID: pa.MessageID} +} diff --git a/vendor/github.com/eclipse/paho.mqtt.golang/packets/pubcomp.go b/vendor/github.com/eclipse/paho.mqtt.golang/packets/pubcomp.go new file mode 100644 index 00000000000..494e2d52192 --- /dev/null +++ b/vendor/github.com/eclipse/paho.mqtt.golang/packets/pubcomp.go @@ -0,0 +1,42 @@ +package packets + +import ( + "fmt" + "io" +) + +//PubcompPacket is an internal representation of the fields of the +//Pubcomp MQTT packet +type PubcompPacket struct { + FixedHeader + MessageID uint16 +} + +func (pc *PubcompPacket) String() string { + return fmt.Sprintf("%s MessageID: %d", pc.FixedHeader, pc.MessageID) +} + +func (pc *PubcompPacket) Write(w io.Writer) error { + var err error + pc.FixedHeader.RemainingLength = 2 + packet := pc.FixedHeader.pack() + packet.Write(encodeUint16(pc.MessageID)) + _, err = packet.WriteTo(w) + + return err +} + +//Unpack decodes the details of a ControlPacket after the fixed +//header has been read +func (pc *PubcompPacket) Unpack(b io.Reader) error { + var err error + pc.MessageID, err = decodeUint16(b) + + return err +} + +//Details returns a Details struct containing the Qos and +//MessageID of this ControlPacket +func (pc *PubcompPacket) Details() Details { + return Details{Qos: pc.Qos, MessageID: pc.MessageID} +} diff --git a/vendor/github.com/eclipse/paho.mqtt.golang/packets/publish.go b/vendor/github.com/eclipse/paho.mqtt.golang/packets/publish.go new file mode 100644 index 00000000000..7d425780c0a --- /dev/null +++ b/vendor/github.com/eclipse/paho.mqtt.golang/packets/publish.go @@ -0,0 +1,83 @@ +package packets + +import ( + "bytes" + "fmt" + "io" +) + +//PublishPacket is an internal representation of the fields of the +//Publish MQTT packet +type PublishPacket struct { + FixedHeader + TopicName string + MessageID uint16 + Payload []byte +} + +func (p *PublishPacket) String() string { + return fmt.Sprintf("%s topicName: %s MessageID: %d payload: %s", p.FixedHeader, p.TopicName, p.MessageID, string(p.Payload)) +} + +func (p *PublishPacket) Write(w io.Writer) error { + var body bytes.Buffer + var err error + + body.Write(encodeString(p.TopicName)) + if p.Qos > 0 { + body.Write(encodeUint16(p.MessageID)) + } + p.FixedHeader.RemainingLength = body.Len() + len(p.Payload) + packet := p.FixedHeader.pack() + packet.Write(body.Bytes()) + packet.Write(p.Payload) + _, err = w.Write(packet.Bytes()) + + return err +} + +//Unpack decodes the details of a ControlPacket after the fixed +//header has been read +func (p *PublishPacket) Unpack(b io.Reader) error { + var payloadLength = p.FixedHeader.RemainingLength + var err error + p.TopicName, err = decodeString(b) + if err != nil { + return err + } + + if p.Qos > 0 { + p.MessageID, err = decodeUint16(b) + if err != nil { + return err + } + payloadLength -= len(p.TopicName) + 4 + } else { + payloadLength -= len(p.TopicName) + 2 + } + if payloadLength < 0 { + return fmt.Errorf("Error unpacking publish, payload length < 0") + } + p.Payload = make([]byte, payloadLength) + _, err = b.Read(p.Payload) + + return err +} + +//Copy creates a new PublishPacket with the same topic and payload +//but an empty fixed header, useful for when you want to deliver +//a message with different properties such as Qos but the same +//content +func (p *PublishPacket) Copy() *PublishPacket { + newP := NewControlPacket(Publish).(*PublishPacket) + newP.TopicName = p.TopicName + newP.Payload = p.Payload + + return newP +} + +//Details returns a Details struct containing the Qos and +//MessageID of this ControlPacket +func (p *PublishPacket) Details() Details { + return Details{Qos: p.Qos, MessageID: p.MessageID} +} diff --git a/vendor/github.com/eclipse/paho.mqtt.golang/packets/pubrec.go b/vendor/github.com/eclipse/paho.mqtt.golang/packets/pubrec.go new file mode 100644 index 00000000000..7056089f9d6 --- /dev/null +++ b/vendor/github.com/eclipse/paho.mqtt.golang/packets/pubrec.go @@ -0,0 +1,42 @@ +package packets + +import ( + "fmt" + "io" +) + +//PubrecPacket is an internal representation of the fields of the +//Pubrec MQTT packet +type PubrecPacket struct { + FixedHeader + MessageID uint16 +} + +func (pr *PubrecPacket) String() string { + return fmt.Sprintf("%s MessageID: %d", pr.FixedHeader, pr.MessageID) +} + +func (pr *PubrecPacket) Write(w io.Writer) error { + var err error + pr.FixedHeader.RemainingLength = 2 + packet := pr.FixedHeader.pack() + packet.Write(encodeUint16(pr.MessageID)) + _, err = packet.WriteTo(w) + + return err +} + +//Unpack decodes the details of a ControlPacket after the fixed +//header has been read +func (pr *PubrecPacket) Unpack(b io.Reader) error { + var err error + pr.MessageID, err = decodeUint16(b) + + return err +} + +//Details returns a Details struct containing the Qos and +//MessageID of this ControlPacket +func (pr *PubrecPacket) Details() Details { + return Details{Qos: pr.Qos, MessageID: pr.MessageID} +} diff --git a/vendor/github.com/eclipse/paho.mqtt.golang/packets/pubrel.go b/vendor/github.com/eclipse/paho.mqtt.golang/packets/pubrel.go new file mode 100644 index 00000000000..27d7d32d81c --- /dev/null +++ b/vendor/github.com/eclipse/paho.mqtt.golang/packets/pubrel.go @@ -0,0 +1,42 @@ +package packets + +import ( + "fmt" + "io" +) + +//PubrelPacket is an internal representation of the fields of the +//Pubrel MQTT packet +type PubrelPacket struct { + FixedHeader + MessageID uint16 +} + +func (pr *PubrelPacket) String() string { + return fmt.Sprintf("%s MessageID: %d", pr.FixedHeader, pr.MessageID) +} + +func (pr *PubrelPacket) Write(w io.Writer) error { + var err error + pr.FixedHeader.RemainingLength = 2 + packet := pr.FixedHeader.pack() + packet.Write(encodeUint16(pr.MessageID)) + _, err = packet.WriteTo(w) + + return err +} + +//Unpack decodes the details of a ControlPacket after the fixed +//header has been read +func (pr *PubrelPacket) Unpack(b io.Reader) error { + var err error + pr.MessageID, err = decodeUint16(b) + + return err +} + +//Details returns a Details struct containing the Qos and +//MessageID of this ControlPacket +func (pr *PubrelPacket) Details() Details { + return Details{Qos: pr.Qos, MessageID: pr.MessageID} +} diff --git a/vendor/github.com/eclipse/paho.mqtt.golang/packets/suback.go b/vendor/github.com/eclipse/paho.mqtt.golang/packets/suback.go new file mode 100644 index 00000000000..263e19c6580 --- /dev/null +++ b/vendor/github.com/eclipse/paho.mqtt.golang/packets/suback.go @@ -0,0 +1,57 @@ +package packets + +import ( + "bytes" + "fmt" + "io" +) + +//SubackPacket is an internal representation of the fields of the +//Suback MQTT packet +type SubackPacket struct { + FixedHeader + MessageID uint16 + ReturnCodes []byte +} + +func (sa *SubackPacket) String() string { + return fmt.Sprintf("%s MessageID: %d", sa.FixedHeader, sa.MessageID) +} + +func (sa *SubackPacket) Write(w io.Writer) error { + var body bytes.Buffer + var err error + body.Write(encodeUint16(sa.MessageID)) + body.Write(sa.ReturnCodes) + sa.FixedHeader.RemainingLength = body.Len() + packet := sa.FixedHeader.pack() + packet.Write(body.Bytes()) + _, err = packet.WriteTo(w) + + return err +} + +//Unpack decodes the details of a ControlPacket after the fixed +//header has been read +func (sa *SubackPacket) Unpack(b io.Reader) error { + var qosBuffer bytes.Buffer + var err error + sa.MessageID, err = decodeUint16(b) + if err != nil { + return err + } + + _, err = qosBuffer.ReadFrom(b) + if err != nil { + return err + } + sa.ReturnCodes = qosBuffer.Bytes() + + return nil +} + +//Details returns a Details struct containing the Qos and +//MessageID of this ControlPacket +func (sa *SubackPacket) Details() Details { + return Details{Qos: 0, MessageID: sa.MessageID} +} diff --git a/vendor/github.com/eclipse/paho.mqtt.golang/packets/subscribe.go b/vendor/github.com/eclipse/paho.mqtt.golang/packets/subscribe.go new file mode 100644 index 00000000000..ab0e9734e42 --- /dev/null +++ b/vendor/github.com/eclipse/paho.mqtt.golang/packets/subscribe.go @@ -0,0 +1,69 @@ +package packets + +import ( + "bytes" + "fmt" + "io" +) + +//SubscribePacket is an internal representation of the fields of the +//Subscribe MQTT packet +type SubscribePacket struct { + FixedHeader + MessageID uint16 + Topics []string + Qoss []byte +} + +func (s *SubscribePacket) String() string { + return fmt.Sprintf("%s MessageID: %d topics: %s", s.FixedHeader, s.MessageID, s.Topics) +} + +func (s *SubscribePacket) Write(w io.Writer) error { + var body bytes.Buffer + var err error + + body.Write(encodeUint16(s.MessageID)) + for i, topic := range s.Topics { + body.Write(encodeString(topic)) + body.WriteByte(s.Qoss[i]) + } + s.FixedHeader.RemainingLength = body.Len() + packet := s.FixedHeader.pack() + packet.Write(body.Bytes()) + _, err = packet.WriteTo(w) + + return err +} + +//Unpack decodes the details of a ControlPacket after the fixed +//header has been read +func (s *SubscribePacket) Unpack(b io.Reader) error { + var err error + s.MessageID, err = decodeUint16(b) + if err != nil { + return err + } + payloadLength := s.FixedHeader.RemainingLength - 2 + for payloadLength > 0 { + topic, err := decodeString(b) + if err != nil { + return err + } + s.Topics = append(s.Topics, topic) + qos, err := decodeByte(b) + if err != nil { + return err + } + s.Qoss = append(s.Qoss, qos) + payloadLength -= 2 + len(topic) + 1 //2 bytes of string length, plus string, plus 1 byte for Qos + } + + return nil +} + +//Details returns a Details struct containing the Qos and +//MessageID of this ControlPacket +func (s *SubscribePacket) Details() Details { + return Details{Qos: 1, MessageID: s.MessageID} +} diff --git a/vendor/github.com/eclipse/paho.mqtt.golang/packets/unsuback.go b/vendor/github.com/eclipse/paho.mqtt.golang/packets/unsuback.go new file mode 100644 index 00000000000..c6b2591d1e5 --- /dev/null +++ b/vendor/github.com/eclipse/paho.mqtt.golang/packets/unsuback.go @@ -0,0 +1,42 @@ +package packets + +import ( + "fmt" + "io" +) + +//UnsubackPacket is an internal representation of the fields of the +//Unsuback MQTT packet +type UnsubackPacket struct { + FixedHeader + MessageID uint16 +} + +func (ua *UnsubackPacket) String() string { + return fmt.Sprintf("%s MessageID: %d", ua.FixedHeader, ua.MessageID) +} + +func (ua *UnsubackPacket) Write(w io.Writer) error { + var err error + ua.FixedHeader.RemainingLength = 2 + packet := ua.FixedHeader.pack() + packet.Write(encodeUint16(ua.MessageID)) + _, err = packet.WriteTo(w) + + return err +} + +//Unpack decodes the details of a ControlPacket after the fixed +//header has been read +func (ua *UnsubackPacket) Unpack(b io.Reader) error { + var err error + ua.MessageID, err = decodeUint16(b) + + return err +} + +//Details returns a Details struct containing the Qos and +//MessageID of this ControlPacket +func (ua *UnsubackPacket) Details() Details { + return Details{Qos: 0, MessageID: ua.MessageID} +} diff --git a/vendor/github.com/eclipse/paho.mqtt.golang/packets/unsubscribe.go b/vendor/github.com/eclipse/paho.mqtt.golang/packets/unsubscribe.go new file mode 100644 index 00000000000..e7a53bdecaf --- /dev/null +++ b/vendor/github.com/eclipse/paho.mqtt.golang/packets/unsubscribe.go @@ -0,0 +1,56 @@ +package packets + +import ( + "bytes" + "fmt" + "io" +) + +//UnsubscribePacket is an internal representation of the fields of the +//Unsubscribe MQTT packet +type UnsubscribePacket struct { + FixedHeader + MessageID uint16 + Topics []string +} + +func (u *UnsubscribePacket) String() string { + return fmt.Sprintf("%s MessageID: %d", u.FixedHeader, u.MessageID) +} + +func (u *UnsubscribePacket) Write(w io.Writer) error { + var body bytes.Buffer + var err error + body.Write(encodeUint16(u.MessageID)) + for _, topic := range u.Topics { + body.Write(encodeString(topic)) + } + u.FixedHeader.RemainingLength = body.Len() + packet := u.FixedHeader.pack() + packet.Write(body.Bytes()) + _, err = packet.WriteTo(w) + + return err +} + +//Unpack decodes the details of a ControlPacket after the fixed +//header has been read +func (u *UnsubscribePacket) Unpack(b io.Reader) error { + var err error + u.MessageID, err = decodeUint16(b) + if err != nil { + return err + } + + for topic, err := decodeString(b); err == nil && topic != ""; topic, err = decodeString(b) { + u.Topics = append(u.Topics, topic) + } + + return err +} + +//Details returns a Details struct containing the Qos and +//MessageID of this ControlPacket +func (u *UnsubscribePacket) Details() Details { + return Details{Qos: 1, MessageID: u.MessageID} +} diff --git a/vendor/github.com/eclipse/paho.mqtt.golang/ping.go b/vendor/github.com/eclipse/paho.mqtt.golang/ping.go new file mode 100644 index 00000000000..dbc1ff454b3 --- /dev/null +++ b/vendor/github.com/eclipse/paho.mqtt.golang/ping.go @@ -0,0 +1,69 @@ +/* + * Copyright (c) 2013 IBM Corp. + * + * All rights reserved. This program and the accompanying materials + * are made available under the terms of the Eclipse Public License v1.0 + * which accompanies this distribution, and is available at + * http://www.eclipse.org/legal/epl-v10.html + * + * Contributors: + * Seth Hoenig + * Allan Stockdill-Mander + * Mike Robertson + */ + +package mqtt + +import ( + "errors" + "sync/atomic" + "time" + + "github.com/eclipse/paho.mqtt.golang/packets" +) + +func keepalive(c *client) { + defer c.workers.Done() + DEBUG.Println(PNG, "keepalive starting") + var checkInterval int64 + var pingSent time.Time + + if c.options.KeepAlive > 10 { + checkInterval = 5 + } else { + checkInterval = c.options.KeepAlive / 2 + } + + intervalTicker := time.NewTicker(time.Duration(checkInterval * int64(time.Second))) + defer intervalTicker.Stop() + + for { + select { + case <-c.stop: + DEBUG.Println(PNG, "keepalive stopped") + return + case <-intervalTicker.C: + lastSent := c.lastSent.Load().(time.Time) + lastReceived := c.lastReceived.Load().(time.Time) + + DEBUG.Println(PNG, "ping check", time.Since(lastSent).Seconds()) + if time.Since(lastSent) >= time.Duration(c.options.KeepAlive*int64(time.Second)) || time.Since(lastReceived) >= time.Duration(c.options.KeepAlive*int64(time.Second)) { + if atomic.LoadInt32(&c.pingOutstanding) == 0 { + DEBUG.Println(PNG, "keepalive sending ping") + ping := packets.NewControlPacket(packets.Pingreq).(*packets.PingreqPacket) + //We don't want to wait behind large messages being sent, the Write call + //will block until it it able to send the packet. + atomic.StoreInt32(&c.pingOutstanding, 1) + ping.Write(c.conn) + c.lastSent.Store(time.Now()) + pingSent = time.Now() + } + } + if atomic.LoadInt32(&c.pingOutstanding) > 0 && time.Since(pingSent) >= c.options.PingTimeout { + CRITICAL.Println(PNG, "pingresp not received, disconnecting") + c.errors <- errors.New("pingresp not received, disconnecting") + return + } + } + } +} diff --git a/vendor/github.com/eclipse/paho.mqtt.golang/router.go b/vendor/github.com/eclipse/paho.mqtt.golang/router.go new file mode 100644 index 00000000000..dd55e0ddc3d --- /dev/null +++ b/vendor/github.com/eclipse/paho.mqtt.golang/router.go @@ -0,0 +1,181 @@ +/* + * Copyright (c) 2013 IBM Corp. + * + * All rights reserved. This program and the accompanying materials + * are made available under the terms of the Eclipse Public License v1.0 + * which accompanies this distribution, and is available at + * http://www.eclipse.org/legal/epl-v10.html + * + * Contributors: + * Seth Hoenig + * Allan Stockdill-Mander + * Mike Robertson + */ + +package mqtt + +import ( + "container/list" + "strings" + "sync" + + "github.com/eclipse/paho.mqtt.golang/packets" +) + +// route is a type which associates MQTT Topic strings with a +// callback to be executed upon the arrival of a message associated +// with a subscription to that topic. +type route struct { + topic string + callback MessageHandler +} + +// match takes a slice of strings which represent the route being tested having been split on '/' +// separators, and a slice of strings representing the topic string in the published message, similarly +// split. +// The function determines if the topic string matches the route according to the MQTT topic rules +// and returns a boolean of the outcome +func match(route []string, topic []string) bool { + if len(route) == 0 { + return len(topic) == 0 + } + + if len(topic) == 0 { + return route[0] == "#" + } + + if route[0] == "#" { + return true + } + + if (route[0] == "+") || (route[0] == topic[0]) { + return match(route[1:], topic[1:]) + } + return false +} + +func routeIncludesTopic(route, topic string) bool { + return match(routeSplit(route), strings.Split(topic, "/")) +} + +// removes $share and sharename when splitting the route to allow +// shared subscription routes to correctly match the topic +func routeSplit(route string) []string { + var result []string + if strings.HasPrefix(route, "$share") { + result = strings.Split(route, "/")[2:] + } else { + result = strings.Split(route, "/") + } + return result +} + +// match takes the topic string of the published message and does a basic compare to the +// string of the current Route, if they match it returns true +func (r *route) match(topic string) bool { + return r.topic == topic || routeIncludesTopic(r.topic, topic) +} + +type router struct { + sync.RWMutex + routes *list.List + defaultHandler MessageHandler + messages chan *packets.PublishPacket + stop chan bool +} + +// newRouter returns a new instance of a Router and channel which can be used to tell the Router +// to stop +func newRouter() (*router, chan bool) { + router := &router{routes: list.New(), messages: make(chan *packets.PublishPacket), stop: make(chan bool)} + stop := router.stop + return router, stop +} + +// addRoute takes a topic string and MessageHandler callback. It looks in the current list of +// routes to see if there is already a matching Route. If there is it replaces the current +// callback with the new one. If not it add a new entry to the list of Routes. +func (r *router) addRoute(topic string, callback MessageHandler) { + r.Lock() + defer r.Unlock() + for e := r.routes.Front(); e != nil; e = e.Next() { + if e.Value.(*route).topic == topic { + r := e.Value.(*route) + r.callback = callback + return + } + } + r.routes.PushBack(&route{topic: topic, callback: callback}) +} + +// deleteRoute takes a route string, looks for a matching Route in the list of Routes. If +// found it removes the Route from the list. +func (r *router) deleteRoute(topic string) { + r.Lock() + defer r.Unlock() + for e := r.routes.Front(); e != nil; e = e.Next() { + if e.Value.(*route).topic == topic { + r.routes.Remove(e) + return + } + } +} + +// setDefaultHandler assigns a default callback that will be called if no matching Route +// is found for an incoming Publish. +func (r *router) setDefaultHandler(handler MessageHandler) { + r.Lock() + defer r.Unlock() + r.defaultHandler = handler +} + +// matchAndDispatch takes a channel of Message pointers as input and starts a go routine that +// takes messages off the channel, matches them against the internal route list and calls the +// associated callback (or the defaultHandler, if one exists and no other route matched). If +// anything is sent down the stop channel the function will end. +func (r *router) matchAndDispatch(messages <-chan *packets.PublishPacket, order bool, client *client) { + go func() { + for { + select { + case message := <-messages: + sent := false + r.RLock() + m := messageFromPublish(message, client.ackFunc(message)) + handlers := []MessageHandler{} + for e := r.routes.Front(); e != nil; e = e.Next() { + if e.Value.(*route).match(message.TopicName) { + if order { + handlers = append(handlers, e.Value.(*route).callback) + } else { + hd := e.Value.(*route).callback + go func() { + hd(client, m) + m.Ack() + }() + } + sent = true + } + } + if !sent && r.defaultHandler != nil { + if order { + handlers = append(handlers, r.defaultHandler) + } else { + go func() { + r.defaultHandler(client, m) + m.Ack() + }() + } + } + r.RUnlock() + for _, handler := range handlers { + func() { + handler(client, m) + m.Ack() + }() + } + case <-r.stop: + return + } + } + }() +} diff --git a/vendor/github.com/eclipse/paho.mqtt.golang/store.go b/vendor/github.com/eclipse/paho.mqtt.golang/store.go new file mode 100644 index 00000000000..24a76b7df3c --- /dev/null +++ b/vendor/github.com/eclipse/paho.mqtt.golang/store.go @@ -0,0 +1,136 @@ +/* + * Copyright (c) 2013 IBM Corp. + * + * All rights reserved. This program and the accompanying materials + * are made available under the terms of the Eclipse Public License v1.0 + * which accompanies this distribution, and is available at + * http://www.eclipse.org/legal/epl-v10.html + * + * Contributors: + * Seth Hoenig + * Allan Stockdill-Mander + * Mike Robertson + */ + +package mqtt + +import ( + "fmt" + "strconv" + + "github.com/eclipse/paho.mqtt.golang/packets" +) + +const ( + inboundPrefix = "i." + outboundPrefix = "o." +) + +// Store is an interface which can be used to provide implementations +// for message persistence. +// Because we may have to store distinct messages with the same +// message ID, we need a unique key for each message. This is +// possible by prepending "i." or "o." to each message id +type Store interface { + Open() + Put(key string, message packets.ControlPacket) + Get(key string) packets.ControlPacket + All() []string + Del(key string) + Close() + Reset() +} + +// A key MUST have the form "X.[messageid]" +// where X is 'i' or 'o' +func mIDFromKey(key string) uint16 { + s := key[2:] + i, err := strconv.Atoi(s) + chkerr(err) + return uint16(i) +} + +// Return true if key prefix is outbound +func isKeyOutbound(key string) bool { + return key[:2] == outboundPrefix +} + +// Return true if key prefix is inbound +func isKeyInbound(key string) bool { + return key[:2] == inboundPrefix +} + +// Return a string of the form "i.[id]" +func inboundKeyFromMID(id uint16) string { + return fmt.Sprintf("%s%d", inboundPrefix, id) +} + +// Return a string of the form "o.[id]" +func outboundKeyFromMID(id uint16) string { + return fmt.Sprintf("%s%d", outboundPrefix, id) +} + +// govern which outgoing messages are persisted +func persistOutbound(s Store, m packets.ControlPacket) { + switch m.Details().Qos { + case 0: + switch m.(type) { + case *packets.PubackPacket, *packets.PubcompPacket: + // Sending puback. delete matching publish + // from ibound + s.Del(inboundKeyFromMID(m.Details().MessageID)) + } + case 1: + switch m.(type) { + case *packets.PublishPacket, *packets.PubrelPacket, *packets.SubscribePacket, *packets.UnsubscribePacket: + // Sending publish. store in obound + // until puback received + s.Put(outboundKeyFromMID(m.Details().MessageID), m) + default: + ERROR.Println(STR, "Asked to persist an invalid message type") + } + case 2: + switch m.(type) { + case *packets.PublishPacket: + // Sending publish. store in obound + // until pubrel received + s.Put(outboundKeyFromMID(m.Details().MessageID), m) + default: + ERROR.Println(STR, "Asked to persist an invalid message type") + } + } +} + +// govern which incoming messages are persisted +func persistInbound(s Store, m packets.ControlPacket) { + switch m.Details().Qos { + case 0: + switch m.(type) { + case *packets.PubackPacket, *packets.SubackPacket, *packets.UnsubackPacket, *packets.PubcompPacket: + // Received a puback. delete matching publish + // from obound + s.Del(outboundKeyFromMID(m.Details().MessageID)) + case *packets.PublishPacket, *packets.PubrecPacket, *packets.PingrespPacket, *packets.ConnackPacket: + default: + ERROR.Println(STR, "Asked to persist an invalid messages type") + } + case 1: + switch m.(type) { + case *packets.PublishPacket, *packets.PubrelPacket: + // Received a publish. store it in ibound + // until puback sent + s.Put(inboundKeyFromMID(m.Details().MessageID), m) + default: + ERROR.Println(STR, "Asked to persist an invalid messages type") + } + case 2: + switch m.(type) { + case *packets.PublishPacket: + // Received a publish. store it in ibound + // until pubrel received + s.Put(inboundKeyFromMID(m.Details().MessageID), m) + default: + ERROR.Println(STR, "Asked to persist an invalid messages type") + } + } +} diff --git a/vendor/github.com/eclipse/paho.mqtt.golang/token.go b/vendor/github.com/eclipse/paho.mqtt.golang/token.go new file mode 100644 index 00000000000..6085546ff6c --- /dev/null +++ b/vendor/github.com/eclipse/paho.mqtt.golang/token.go @@ -0,0 +1,183 @@ +/* + * Copyright (c) 2014 IBM Corp. + * + * All rights reserved. This program and the accompanying materials + * are made available under the terms of the Eclipse Public License v1.0 + * which accompanies this distribution, and is available at + * http://www.eclipse.org/legal/epl-v10.html + * + * Contributors: + * Allan Stockdill-Mander + */ + +package mqtt + +import ( + "sync" + "time" + + "github.com/eclipse/paho.mqtt.golang/packets" +) + +// PacketAndToken is a struct that contains both a ControlPacket and a +// Token. This struct is passed via channels between the client interface +// code and the underlying code responsible for sending and receiving +// MQTT messages. +type PacketAndToken struct { + p packets.ControlPacket + t tokenCompletor +} + +// Token defines the interface for the tokens used to indicate when +// actions have completed. +type Token interface { + Wait() bool + WaitTimeout(time.Duration) bool + Error() error +} + +type TokenErrorSetter interface { + setError(error) +} + +type tokenCompletor interface { + Token + TokenErrorSetter + flowComplete() +} + +type baseToken struct { + m sync.RWMutex + complete chan struct{} + err error +} + +// Wait will wait indefinitely for the Token to complete, ie the Publish +// to be sent and confirmed receipt from the broker +func (b *baseToken) Wait() bool { + <-b.complete + return true +} + +// WaitTimeout takes a time.Duration to wait for the flow associated with the +// Token to complete, returns true if it returned before the timeout or +// returns false if the timeout occurred. In the case of a timeout the Token +// does not have an error set in case the caller wishes to wait again +func (b *baseToken) WaitTimeout(d time.Duration) bool { + timer := time.NewTimer(d) + select { + case <-b.complete: + if !timer.Stop() { + <-timer.C + } + return true + case <-timer.C: + } + + return false +} + +func (b *baseToken) flowComplete() { + select { + case <-b.complete: + default: + close(b.complete) + } +} + +func (b *baseToken) Error() error { + b.m.RLock() + defer b.m.RUnlock() + return b.err +} + +func (b *baseToken) setError(e error) { + b.m.Lock() + b.err = e + b.flowComplete() + b.m.Unlock() +} + +func newToken(tType byte) tokenCompletor { + switch tType { + case packets.Connect: + return &ConnectToken{baseToken: baseToken{complete: make(chan struct{})}} + case packets.Subscribe: + return &SubscribeToken{baseToken: baseToken{complete: make(chan struct{})}, subResult: make(map[string]byte)} + case packets.Publish: + return &PublishToken{baseToken: baseToken{complete: make(chan struct{})}} + case packets.Unsubscribe: + return &UnsubscribeToken{baseToken: baseToken{complete: make(chan struct{})}} + case packets.Disconnect: + return &DisconnectToken{baseToken: baseToken{complete: make(chan struct{})}} + } + return nil +} + +// ConnectToken is an extension of Token containing the extra fields +// required to provide information about calls to Connect() +type ConnectToken struct { + baseToken + returnCode byte + sessionPresent bool +} + +// ReturnCode returns the acknowledgement code in the connack sent +// in response to a Connect() +func (c *ConnectToken) ReturnCode() byte { + c.m.RLock() + defer c.m.RUnlock() + return c.returnCode +} + +// SessionPresent returns a bool representing the value of the +// session present field in the connack sent in response to a Connect() +func (c *ConnectToken) SessionPresent() bool { + c.m.RLock() + defer c.m.RUnlock() + return c.sessionPresent +} + +// PublishToken is an extension of Token containing the extra fields +// required to provide information about calls to Publish() +type PublishToken struct { + baseToken + messageID uint16 +} + +// MessageID returns the MQTT message ID that was assigned to the +// Publish packet when it was sent to the broker +func (p *PublishToken) MessageID() uint16 { + return p.messageID +} + +// SubscribeToken is an extension of Token containing the extra fields +// required to provide information about calls to Subscribe() +type SubscribeToken struct { + baseToken + subs []string + subResult map[string]byte + messageID uint16 +} + +// Result returns a map of topics that were subscribed to along with +// the matching return code from the broker. This is either the Qos +// value of the subscription or an error code. +func (s *SubscribeToken) Result() map[string]byte { + s.m.RLock() + defer s.m.RUnlock() + return s.subResult +} + +// UnsubscribeToken is an extension of Token containing the extra fields +// required to provide information about calls to Unsubscribe() +type UnsubscribeToken struct { + baseToken + messageID uint16 +} + +// DisconnectToken is an extension of Token containing the extra fields +// required to provide information about calls to Disconnect() +type DisconnectToken struct { + baseToken +} diff --git a/vendor/github.com/eclipse/paho.mqtt.golang/topic.go b/vendor/github.com/eclipse/paho.mqtt.golang/topic.go new file mode 100644 index 00000000000..01b536d73c5 --- /dev/null +++ b/vendor/github.com/eclipse/paho.mqtt.golang/topic.go @@ -0,0 +1,86 @@ +/* + * Copyright (c) 2014 IBM Corp. + * + * All rights reserved. This program and the accompanying materials + * are made available under the terms of the Eclipse Public License v1.0 + * which accompanies this distribution, and is available at + * http://www.eclipse.org/legal/epl-v10.html + * + * Contributors: + * Seth Hoenig + * Allan Stockdill-Mander + * Mike Robertson + */ + +package mqtt + +import ( + "errors" + "strings" +) + +//ErrInvalidQos is the error returned when an packet is to be sent +//with an invalid Qos value +var ErrInvalidQos = errors.New("Invalid QoS") + +//ErrInvalidTopicEmptyString is the error returned when a topic string +//is passed in that is 0 length +var ErrInvalidTopicEmptyString = errors.New("Invalid Topic; empty string") + +//ErrInvalidTopicMultilevel is the error returned when a topic string +//is passed in that has the multi level wildcard in any position but +//the last +var ErrInvalidTopicMultilevel = errors.New("Invalid Topic; multi-level wildcard must be last level") + +// Topic Names and Topic Filters +// The MQTT v3.1.1 spec clarifies a number of ambiguities with regard +// to the validity of Topic strings. +// - A Topic must be between 1 and 65535 bytes. +// - A Topic is case sensitive. +// - A Topic may contain whitespace. +// - A Topic containing a leading forward slash is different than a Topic without. +// - A Topic may be "/" (two levels, both empty string). +// - A Topic must be UTF-8 encoded. +// - A Topic may contain any number of levels. +// - A Topic may contain an empty level (two forward slashes in a row). +// - A TopicName may not contain a wildcard. +// - A TopicFilter may only have a # (multi-level) wildcard as the last level. +// - A TopicFilter may contain any number of + (single-level) wildcards. +// - A TopicFilter with a # will match the absence of a level +// Example: a subscription to "foo/#" will match messages published to "foo". + +func validateSubscribeMap(subs map[string]byte) ([]string, []byte, error) { + if len(subs) == 0 { + return nil, nil, errors.New("Invalid subscription; subscribe map must not be empty") + } + + var topics []string + var qoss []byte + for topic, qos := range subs { + if err := validateTopicAndQos(topic, qos); err != nil { + return nil, nil, err + } + topics = append(topics, topic) + qoss = append(qoss, qos) + } + + return topics, qoss, nil +} + +func validateTopicAndQos(topic string, qos byte) error { + if len(topic) == 0 { + return ErrInvalidTopicEmptyString + } + + levels := strings.Split(topic, "/") + for i, level := range levels { + if level == "#" && i != len(levels)-1 { + return ErrInvalidTopicMultilevel + } + } + + if qos > 2 { + return ErrInvalidQos + } + return nil +} diff --git a/vendor/github.com/eclipse/paho.mqtt.golang/trace.go b/vendor/github.com/eclipse/paho.mqtt.golang/trace.go new file mode 100644 index 00000000000..195c8173dcf --- /dev/null +++ b/vendor/github.com/eclipse/paho.mqtt.golang/trace.go @@ -0,0 +1,40 @@ +/* + * Copyright (c) 2013 IBM Corp. + * + * All rights reserved. This program and the accompanying materials + * are made available under the terms of the Eclipse Public License v1.0 + * which accompanies this distribution, and is available at + * http://www.eclipse.org/legal/epl-v10.html + * + * Contributors: + * Seth Hoenig + * Allan Stockdill-Mander + * Mike Robertson + */ + +package mqtt + +type ( + // Logger interface allows implementations to provide to this package any + // object that implements the methods defined in it. + Logger interface { + Println(v ...interface{}) + Printf(format string, v ...interface{}) + } + + // NOOPLogger implements the logger that does not perform any operation + // by default. This allows us to efficiently discard the unwanted messages. + NOOPLogger struct{} +) + +func (NOOPLogger) Println(v ...interface{}) {} +func (NOOPLogger) Printf(format string, v ...interface{}) {} + +// Internal levels of library output that are initialised to not print +// anything but can be overridden by programmer +var ( + ERROR Logger = NOOPLogger{} + CRITICAL Logger = NOOPLogger{} + WARN Logger = NOOPLogger{} + DEBUG Logger = NOOPLogger{} +) diff --git a/vendor/github.com/eclipse/paho.mqtt.golang/websocket.go b/vendor/github.com/eclipse/paho.mqtt.golang/websocket.go new file mode 100644 index 00000000000..5e8511e2241 --- /dev/null +++ b/vendor/github.com/eclipse/paho.mqtt.golang/websocket.go @@ -0,0 +1,95 @@ +package mqtt + +import ( + "crypto/tls" + "io" + "net" + "net/http" + "sync" + "time" + + "github.com/gorilla/websocket" +) + +// NewWebsocket returns a new websocket and returns a net.Conn compatible interface using the gorilla/websocket package +func NewWebsocket(host string, tlsc *tls.Config, timeout time.Duration, requestHeader http.Header) (net.Conn, error) { + if timeout == 0 { + timeout = 10 * time.Second + } + + dialer := &websocket.Dialer{ + Proxy: http.ProxyFromEnvironment, + HandshakeTimeout: timeout, + EnableCompression: false, + TLSClientConfig: tlsc, + Subprotocols: []string{"mqtt"}, + } + ws, _, err := dialer.Dial(host, requestHeader) + + if err != nil { + return nil, err + } + + wrapper := &websocketConnector{ + Conn: ws, + } + return wrapper, err +} + +// websocketConnector is a websocket wrapper so it satisfies the net.Conn interface so it is a +// drop in replacement of the golang.org/x/net/websocket package. +// Implementation guide taken from https://github.com/gorilla/websocket/issues/282 +type websocketConnector struct { + *websocket.Conn + r io.Reader + rio sync.Mutex + wio sync.Mutex +} + +// SetDeadline sets both the read and write deadlines +func (c *websocketConnector) SetDeadline(t time.Time) error { + if err := c.SetReadDeadline(t); err != nil { + return err + } + err := c.SetWriteDeadline(t) + return err +} + +// Write writes data to the websocket +func (c *websocketConnector) Write(p []byte) (int, error) { + c.wio.Lock() + defer c.wio.Unlock() + + err := c.WriteMessage(websocket.BinaryMessage, p) + if err != nil { + return 0, err + } + return len(p), nil +} + +// Read reads the current websocket frame +func (c *websocketConnector) Read(p []byte) (int, error) { + c.rio.Lock() + defer c.rio.Unlock() + for { + if c.r == nil { + // Advance to next message. + var err error + _, c.r, err = c.NextReader() + if err != nil { + return 0, err + } + } + n, err := c.r.Read(p) + if err == io.EOF { + // At end of message. + c.r = nil + if n > 0 { + return n, nil + } + // No data read, continue to next message. + continue + } + return n, err + } +} diff --git a/vendor/github.com/gorilla/websocket/AUTHORS b/vendor/github.com/gorilla/websocket/AUTHORS new file mode 100644 index 00000000000..1931f400682 --- /dev/null +++ b/vendor/github.com/gorilla/websocket/AUTHORS @@ -0,0 +1,9 @@ +# This is the official list of Gorilla WebSocket authors for copyright +# purposes. +# +# Please keep the list sorted. + +Gary Burd +Google LLC (https://opensource.google.com/) +Joachim Bauch + diff --git a/vendor/github.com/gorilla/websocket/LICENSE b/vendor/github.com/gorilla/websocket/LICENSE new file mode 100644 index 00000000000..9171c972252 --- /dev/null +++ b/vendor/github.com/gorilla/websocket/LICENSE @@ -0,0 +1,22 @@ +Copyright (c) 2013 The Gorilla WebSocket Authors. All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are met: + + Redistributions of source code must retain the above copyright notice, this + list of conditions and the following disclaimer. + + Redistributions in binary form must reproduce the above copyright notice, + this list of conditions and the following disclaimer in the documentation + and/or other materials provided with the distribution. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND +ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED +WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE +FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR +SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER +CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, +OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. diff --git a/vendor/github.com/gorilla/websocket/README.md b/vendor/github.com/gorilla/websocket/README.md new file mode 100644 index 00000000000..0827d059c11 --- /dev/null +++ b/vendor/github.com/gorilla/websocket/README.md @@ -0,0 +1,64 @@ +# Gorilla WebSocket + +[![GoDoc](https://godoc.org/github.com/gorilla/websocket?status.svg)](https://godoc.org/github.com/gorilla/websocket) +[![CircleCI](https://circleci.com/gh/gorilla/websocket.svg?style=svg)](https://circleci.com/gh/gorilla/websocket) + +Gorilla WebSocket is a [Go](http://golang.org/) implementation of the +[WebSocket](http://www.rfc-editor.org/rfc/rfc6455.txt) protocol. + +### Documentation + +* [API Reference](http://godoc.org/github.com/gorilla/websocket) +* [Chat example](https://github.com/gorilla/websocket/tree/master/examples/chat) +* [Command example](https://github.com/gorilla/websocket/tree/master/examples/command) +* [Client and server example](https://github.com/gorilla/websocket/tree/master/examples/echo) +* [File watch example](https://github.com/gorilla/websocket/tree/master/examples/filewatch) + +### Status + +The Gorilla WebSocket package provides a complete and tested implementation of +the [WebSocket](http://www.rfc-editor.org/rfc/rfc6455.txt) protocol. The +package API is stable. + +### Installation + + go get github.com/gorilla/websocket + +### Protocol Compliance + +The Gorilla WebSocket package passes the server tests in the [Autobahn Test +Suite](https://github.com/crossbario/autobahn-testsuite) using the application in the [examples/autobahn +subdirectory](https://github.com/gorilla/websocket/tree/master/examples/autobahn). + +### Gorilla WebSocket compared with other packages + + + + + + + + + + + + + + + + + + +
github.com/gorillagolang.org/x/net
RFC 6455 Features
Passes Autobahn Test SuiteYesNo
Receive fragmented messageYesNo, see note 1
Send close messageYesNo
Send pings and receive pongsYesNo
Get the type of a received data messageYesYes, see note 2
Other Features
Compression ExtensionsExperimentalNo
Read message using io.ReaderYesNo, see note 3
Write message using io.WriteCloserYesNo, see note 3
+ +Notes: + +1. Large messages are fragmented in [Chrome's new WebSocket implementation](http://www.ietf.org/mail-archive/web/hybi/current/msg10503.html). +2. The application can get the type of a received data message by implementing + a [Codec marshal](http://godoc.org/golang.org/x/net/websocket#Codec.Marshal) + function. +3. The go.net io.Reader and io.Writer operate across WebSocket frame boundaries. + Read returns when the input buffer is full or a frame boundary is + encountered. Each call to Write sends a single frame message. The Gorilla + io.Reader and io.WriteCloser operate on a single WebSocket message. + diff --git a/vendor/github.com/gorilla/websocket/client.go b/vendor/github.com/gorilla/websocket/client.go new file mode 100644 index 00000000000..962c06a391c --- /dev/null +++ b/vendor/github.com/gorilla/websocket/client.go @@ -0,0 +1,395 @@ +// Copyright 2013 The Gorilla WebSocket Authors. All rights reserved. +// Use of this source code is governed by a BSD-style +// license that can be found in the LICENSE file. + +package websocket + +import ( + "bytes" + "context" + "crypto/tls" + "errors" + "io" + "io/ioutil" + "net" + "net/http" + "net/http/httptrace" + "net/url" + "strings" + "time" +) + +// ErrBadHandshake is returned when the server response to opening handshake is +// invalid. +var ErrBadHandshake = errors.New("websocket: bad handshake") + +var errInvalidCompression = errors.New("websocket: invalid compression negotiation") + +// NewClient creates a new client connection using the given net connection. +// The URL u specifies the host and request URI. Use requestHeader to specify +// the origin (Origin), subprotocols (Sec-WebSocket-Protocol) and cookies +// (Cookie). Use the response.Header to get the selected subprotocol +// (Sec-WebSocket-Protocol) and cookies (Set-Cookie). +// +// If the WebSocket handshake fails, ErrBadHandshake is returned along with a +// non-nil *http.Response so that callers can handle redirects, authentication, +// etc. +// +// Deprecated: Use Dialer instead. +func NewClient(netConn net.Conn, u *url.URL, requestHeader http.Header, readBufSize, writeBufSize int) (c *Conn, response *http.Response, err error) { + d := Dialer{ + ReadBufferSize: readBufSize, + WriteBufferSize: writeBufSize, + NetDial: func(net, addr string) (net.Conn, error) { + return netConn, nil + }, + } + return d.Dial(u.String(), requestHeader) +} + +// A Dialer contains options for connecting to WebSocket server. +type Dialer struct { + // NetDial specifies the dial function for creating TCP connections. If + // NetDial is nil, net.Dial is used. + NetDial func(network, addr string) (net.Conn, error) + + // NetDialContext specifies the dial function for creating TCP connections. If + // NetDialContext is nil, net.DialContext is used. + NetDialContext func(ctx context.Context, network, addr string) (net.Conn, error) + + // Proxy specifies a function to return a proxy for a given + // Request. If the function returns a non-nil error, the + // request is aborted with the provided error. + // If Proxy is nil or returns a nil *URL, no proxy is used. + Proxy func(*http.Request) (*url.URL, error) + + // TLSClientConfig specifies the TLS configuration to use with tls.Client. + // If nil, the default configuration is used. + TLSClientConfig *tls.Config + + // HandshakeTimeout specifies the duration for the handshake to complete. + HandshakeTimeout time.Duration + + // ReadBufferSize and WriteBufferSize specify I/O buffer sizes in bytes. If a buffer + // size is zero, then a useful default size is used. The I/O buffer sizes + // do not limit the size of the messages that can be sent or received. + ReadBufferSize, WriteBufferSize int + + // WriteBufferPool is a pool of buffers for write operations. If the value + // is not set, then write buffers are allocated to the connection for the + // lifetime of the connection. + // + // A pool is most useful when the application has a modest volume of writes + // across a large number of connections. + // + // Applications should use a single pool for each unique value of + // WriteBufferSize. + WriteBufferPool BufferPool + + // Subprotocols specifies the client's requested subprotocols. + Subprotocols []string + + // EnableCompression specifies if the client should attempt to negotiate + // per message compression (RFC 7692). Setting this value to true does not + // guarantee that compression will be supported. Currently only "no context + // takeover" modes are supported. + EnableCompression bool + + // Jar specifies the cookie jar. + // If Jar is nil, cookies are not sent in requests and ignored + // in responses. + Jar http.CookieJar +} + +// Dial creates a new client connection by calling DialContext with a background context. +func (d *Dialer) Dial(urlStr string, requestHeader http.Header) (*Conn, *http.Response, error) { + return d.DialContext(context.Background(), urlStr, requestHeader) +} + +var errMalformedURL = errors.New("malformed ws or wss URL") + +func hostPortNoPort(u *url.URL) (hostPort, hostNoPort string) { + hostPort = u.Host + hostNoPort = u.Host + if i := strings.LastIndex(u.Host, ":"); i > strings.LastIndex(u.Host, "]") { + hostNoPort = hostNoPort[:i] + } else { + switch u.Scheme { + case "wss": + hostPort += ":443" + case "https": + hostPort += ":443" + default: + hostPort += ":80" + } + } + return hostPort, hostNoPort +} + +// DefaultDialer is a dialer with all fields set to the default values. +var DefaultDialer = &Dialer{ + Proxy: http.ProxyFromEnvironment, + HandshakeTimeout: 45 * time.Second, +} + +// nilDialer is dialer to use when receiver is nil. +var nilDialer = *DefaultDialer + +// DialContext creates a new client connection. Use requestHeader to specify the +// origin (Origin), subprotocols (Sec-WebSocket-Protocol) and cookies (Cookie). +// Use the response.Header to get the selected subprotocol +// (Sec-WebSocket-Protocol) and cookies (Set-Cookie). +// +// The context will be used in the request and in the Dialer. +// +// If the WebSocket handshake fails, ErrBadHandshake is returned along with a +// non-nil *http.Response so that callers can handle redirects, authentication, +// etcetera. The response body may not contain the entire response and does not +// need to be closed by the application. +func (d *Dialer) DialContext(ctx context.Context, urlStr string, requestHeader http.Header) (*Conn, *http.Response, error) { + if d == nil { + d = &nilDialer + } + + challengeKey, err := generateChallengeKey() + if err != nil { + return nil, nil, err + } + + u, err := url.Parse(urlStr) + if err != nil { + return nil, nil, err + } + + switch u.Scheme { + case "ws": + u.Scheme = "http" + case "wss": + u.Scheme = "https" + default: + return nil, nil, errMalformedURL + } + + if u.User != nil { + // User name and password are not allowed in websocket URIs. + return nil, nil, errMalformedURL + } + + req := &http.Request{ + Method: "GET", + URL: u, + Proto: "HTTP/1.1", + ProtoMajor: 1, + ProtoMinor: 1, + Header: make(http.Header), + Host: u.Host, + } + req = req.WithContext(ctx) + + // Set the cookies present in the cookie jar of the dialer + if d.Jar != nil { + for _, cookie := range d.Jar.Cookies(u) { + req.AddCookie(cookie) + } + } + + // Set the request headers using the capitalization for names and values in + // RFC examples. Although the capitalization shouldn't matter, there are + // servers that depend on it. The Header.Set method is not used because the + // method canonicalizes the header names. + req.Header["Upgrade"] = []string{"websocket"} + req.Header["Connection"] = []string{"Upgrade"} + req.Header["Sec-WebSocket-Key"] = []string{challengeKey} + req.Header["Sec-WebSocket-Version"] = []string{"13"} + if len(d.Subprotocols) > 0 { + req.Header["Sec-WebSocket-Protocol"] = []string{strings.Join(d.Subprotocols, ", ")} + } + for k, vs := range requestHeader { + switch { + case k == "Host": + if len(vs) > 0 { + req.Host = vs[0] + } + case k == "Upgrade" || + k == "Connection" || + k == "Sec-Websocket-Key" || + k == "Sec-Websocket-Version" || + k == "Sec-Websocket-Extensions" || + (k == "Sec-Websocket-Protocol" && len(d.Subprotocols) > 0): + return nil, nil, errors.New("websocket: duplicate header not allowed: " + k) + case k == "Sec-Websocket-Protocol": + req.Header["Sec-WebSocket-Protocol"] = vs + default: + req.Header[k] = vs + } + } + + if d.EnableCompression { + req.Header["Sec-WebSocket-Extensions"] = []string{"permessage-deflate; server_no_context_takeover; client_no_context_takeover"} + } + + if d.HandshakeTimeout != 0 { + var cancel func() + ctx, cancel = context.WithTimeout(ctx, d.HandshakeTimeout) + defer cancel() + } + + // Get network dial function. + var netDial func(network, add string) (net.Conn, error) + + if d.NetDialContext != nil { + netDial = func(network, addr string) (net.Conn, error) { + return d.NetDialContext(ctx, network, addr) + } + } else if d.NetDial != nil { + netDial = d.NetDial + } else { + netDialer := &net.Dialer{} + netDial = func(network, addr string) (net.Conn, error) { + return netDialer.DialContext(ctx, network, addr) + } + } + + // If needed, wrap the dial function to set the connection deadline. + if deadline, ok := ctx.Deadline(); ok { + forwardDial := netDial + netDial = func(network, addr string) (net.Conn, error) { + c, err := forwardDial(network, addr) + if err != nil { + return nil, err + } + err = c.SetDeadline(deadline) + if err != nil { + c.Close() + return nil, err + } + return c, nil + } + } + + // If needed, wrap the dial function to connect through a proxy. + if d.Proxy != nil { + proxyURL, err := d.Proxy(req) + if err != nil { + return nil, nil, err + } + if proxyURL != nil { + dialer, err := proxy_FromURL(proxyURL, netDialerFunc(netDial)) + if err != nil { + return nil, nil, err + } + netDial = dialer.Dial + } + } + + hostPort, hostNoPort := hostPortNoPort(u) + trace := httptrace.ContextClientTrace(ctx) + if trace != nil && trace.GetConn != nil { + trace.GetConn(hostPort) + } + + netConn, err := netDial("tcp", hostPort) + if trace != nil && trace.GotConn != nil { + trace.GotConn(httptrace.GotConnInfo{ + Conn: netConn, + }) + } + if err != nil { + return nil, nil, err + } + + defer func() { + if netConn != nil { + netConn.Close() + } + }() + + if u.Scheme == "https" { + cfg := cloneTLSConfig(d.TLSClientConfig) + if cfg.ServerName == "" { + cfg.ServerName = hostNoPort + } + tlsConn := tls.Client(netConn, cfg) + netConn = tlsConn + + var err error + if trace != nil { + err = doHandshakeWithTrace(trace, tlsConn, cfg) + } else { + err = doHandshake(tlsConn, cfg) + } + + if err != nil { + return nil, nil, err + } + } + + conn := newConn(netConn, false, d.ReadBufferSize, d.WriteBufferSize, d.WriteBufferPool, nil, nil) + + if err := req.Write(netConn); err != nil { + return nil, nil, err + } + + if trace != nil && trace.GotFirstResponseByte != nil { + if peek, err := conn.br.Peek(1); err == nil && len(peek) == 1 { + trace.GotFirstResponseByte() + } + } + + resp, err := http.ReadResponse(conn.br, req) + if err != nil { + return nil, nil, err + } + + if d.Jar != nil { + if rc := resp.Cookies(); len(rc) > 0 { + d.Jar.SetCookies(u, rc) + } + } + + if resp.StatusCode != 101 || + !strings.EqualFold(resp.Header.Get("Upgrade"), "websocket") || + !strings.EqualFold(resp.Header.Get("Connection"), "upgrade") || + resp.Header.Get("Sec-Websocket-Accept") != computeAcceptKey(challengeKey) { + // Before closing the network connection on return from this + // function, slurp up some of the response to aid application + // debugging. + buf := make([]byte, 1024) + n, _ := io.ReadFull(resp.Body, buf) + resp.Body = ioutil.NopCloser(bytes.NewReader(buf[:n])) + return nil, resp, ErrBadHandshake + } + + for _, ext := range parseExtensions(resp.Header) { + if ext[""] != "permessage-deflate" { + continue + } + _, snct := ext["server_no_context_takeover"] + _, cnct := ext["client_no_context_takeover"] + if !snct || !cnct { + return nil, resp, errInvalidCompression + } + conn.newCompressionWriter = compressNoContextTakeover + conn.newDecompressionReader = decompressNoContextTakeover + break + } + + resp.Body = ioutil.NopCloser(bytes.NewReader([]byte{})) + conn.subprotocol = resp.Header.Get("Sec-Websocket-Protocol") + + netConn.SetDeadline(time.Time{}) + netConn = nil // to avoid close in defer. + return conn, resp, nil +} + +func doHandshake(tlsConn *tls.Conn, cfg *tls.Config) error { + if err := tlsConn.Handshake(); err != nil { + return err + } + if !cfg.InsecureSkipVerify { + if err := tlsConn.VerifyHostname(cfg.ServerName); err != nil { + return err + } + } + return nil +} diff --git a/vendor/github.com/gorilla/websocket/client_clone.go b/vendor/github.com/gorilla/websocket/client_clone.go new file mode 100644 index 00000000000..4f0d943723a --- /dev/null +++ b/vendor/github.com/gorilla/websocket/client_clone.go @@ -0,0 +1,16 @@ +// Copyright 2013 The Gorilla WebSocket Authors. All rights reserved. +// Use of this source code is governed by a BSD-style +// license that can be found in the LICENSE file. + +// +build go1.8 + +package websocket + +import "crypto/tls" + +func cloneTLSConfig(cfg *tls.Config) *tls.Config { + if cfg == nil { + return &tls.Config{} + } + return cfg.Clone() +} diff --git a/vendor/github.com/gorilla/websocket/client_clone_legacy.go b/vendor/github.com/gorilla/websocket/client_clone_legacy.go new file mode 100644 index 00000000000..babb007fb41 --- /dev/null +++ b/vendor/github.com/gorilla/websocket/client_clone_legacy.go @@ -0,0 +1,38 @@ +// Copyright 2013 The Gorilla WebSocket Authors. All rights reserved. +// Use of this source code is governed by a BSD-style +// license that can be found in the LICENSE file. + +// +build !go1.8 + +package websocket + +import "crypto/tls" + +// cloneTLSConfig clones all public fields except the fields +// SessionTicketsDisabled and SessionTicketKey. This avoids copying the +// sync.Mutex in the sync.Once and makes it safe to call cloneTLSConfig on a +// config in active use. +func cloneTLSConfig(cfg *tls.Config) *tls.Config { + if cfg == nil { + return &tls.Config{} + } + return &tls.Config{ + Rand: cfg.Rand, + Time: cfg.Time, + Certificates: cfg.Certificates, + NameToCertificate: cfg.NameToCertificate, + GetCertificate: cfg.GetCertificate, + RootCAs: cfg.RootCAs, + NextProtos: cfg.NextProtos, + ServerName: cfg.ServerName, + ClientAuth: cfg.ClientAuth, + ClientCAs: cfg.ClientCAs, + InsecureSkipVerify: cfg.InsecureSkipVerify, + CipherSuites: cfg.CipherSuites, + PreferServerCipherSuites: cfg.PreferServerCipherSuites, + ClientSessionCache: cfg.ClientSessionCache, + MinVersion: cfg.MinVersion, + MaxVersion: cfg.MaxVersion, + CurvePreferences: cfg.CurvePreferences, + } +} diff --git a/vendor/github.com/gorilla/websocket/compression.go b/vendor/github.com/gorilla/websocket/compression.go new file mode 100644 index 00000000000..813ffb1e843 --- /dev/null +++ b/vendor/github.com/gorilla/websocket/compression.go @@ -0,0 +1,148 @@ +// Copyright 2017 The Gorilla WebSocket Authors. All rights reserved. +// Use of this source code is governed by a BSD-style +// license that can be found in the LICENSE file. + +package websocket + +import ( + "compress/flate" + "errors" + "io" + "strings" + "sync" +) + +const ( + minCompressionLevel = -2 // flate.HuffmanOnly not defined in Go < 1.6 + maxCompressionLevel = flate.BestCompression + defaultCompressionLevel = 1 +) + +var ( + flateWriterPools [maxCompressionLevel - minCompressionLevel + 1]sync.Pool + flateReaderPool = sync.Pool{New: func() interface{} { + return flate.NewReader(nil) + }} +) + +func decompressNoContextTakeover(r io.Reader) io.ReadCloser { + const tail = + // Add four bytes as specified in RFC + "\x00\x00\xff\xff" + + // Add final block to squelch unexpected EOF error from flate reader. + "\x01\x00\x00\xff\xff" + + fr, _ := flateReaderPool.Get().(io.ReadCloser) + fr.(flate.Resetter).Reset(io.MultiReader(r, strings.NewReader(tail)), nil) + return &flateReadWrapper{fr} +} + +func isValidCompressionLevel(level int) bool { + return minCompressionLevel <= level && level <= maxCompressionLevel +} + +func compressNoContextTakeover(w io.WriteCloser, level int) io.WriteCloser { + p := &flateWriterPools[level-minCompressionLevel] + tw := &truncWriter{w: w} + fw, _ := p.Get().(*flate.Writer) + if fw == nil { + fw, _ = flate.NewWriter(tw, level) + } else { + fw.Reset(tw) + } + return &flateWriteWrapper{fw: fw, tw: tw, p: p} +} + +// truncWriter is an io.Writer that writes all but the last four bytes of the +// stream to another io.Writer. +type truncWriter struct { + w io.WriteCloser + n int + p [4]byte +} + +func (w *truncWriter) Write(p []byte) (int, error) { + n := 0 + + // fill buffer first for simplicity. + if w.n < len(w.p) { + n = copy(w.p[w.n:], p) + p = p[n:] + w.n += n + if len(p) == 0 { + return n, nil + } + } + + m := len(p) + if m > len(w.p) { + m = len(w.p) + } + + if nn, err := w.w.Write(w.p[:m]); err != nil { + return n + nn, err + } + + copy(w.p[:], w.p[m:]) + copy(w.p[len(w.p)-m:], p[len(p)-m:]) + nn, err := w.w.Write(p[:len(p)-m]) + return n + nn, err +} + +type flateWriteWrapper struct { + fw *flate.Writer + tw *truncWriter + p *sync.Pool +} + +func (w *flateWriteWrapper) Write(p []byte) (int, error) { + if w.fw == nil { + return 0, errWriteClosed + } + return w.fw.Write(p) +} + +func (w *flateWriteWrapper) Close() error { + if w.fw == nil { + return errWriteClosed + } + err1 := w.fw.Flush() + w.p.Put(w.fw) + w.fw = nil + if w.tw.p != [4]byte{0, 0, 0xff, 0xff} { + return errors.New("websocket: internal error, unexpected bytes at end of flate stream") + } + err2 := w.tw.w.Close() + if err1 != nil { + return err1 + } + return err2 +} + +type flateReadWrapper struct { + fr io.ReadCloser +} + +func (r *flateReadWrapper) Read(p []byte) (int, error) { + if r.fr == nil { + return 0, io.ErrClosedPipe + } + n, err := r.fr.Read(p) + if err == io.EOF { + // Preemptively place the reader back in the pool. This helps with + // scenarios where the application does not call NextReader() soon after + // this final read. + r.Close() + } + return n, err +} + +func (r *flateReadWrapper) Close() error { + if r.fr == nil { + return io.ErrClosedPipe + } + err := r.fr.Close() + flateReaderPool.Put(r.fr) + r.fr = nil + return err +} diff --git a/vendor/github.com/gorilla/websocket/conn.go b/vendor/github.com/gorilla/websocket/conn.go new file mode 100644 index 00000000000..6f17cd29982 --- /dev/null +++ b/vendor/github.com/gorilla/websocket/conn.go @@ -0,0 +1,1201 @@ +// Copyright 2013 The Gorilla WebSocket Authors. All rights reserved. +// Use of this source code is governed by a BSD-style +// license that can be found in the LICENSE file. + +package websocket + +import ( + "bufio" + "encoding/binary" + "errors" + "io" + "io/ioutil" + "math/rand" + "net" + "strconv" + "sync" + "time" + "unicode/utf8" +) + +const ( + // Frame header byte 0 bits from Section 5.2 of RFC 6455 + finalBit = 1 << 7 + rsv1Bit = 1 << 6 + rsv2Bit = 1 << 5 + rsv3Bit = 1 << 4 + + // Frame header byte 1 bits from Section 5.2 of RFC 6455 + maskBit = 1 << 7 + + maxFrameHeaderSize = 2 + 8 + 4 // Fixed header + length + mask + maxControlFramePayloadSize = 125 + + writeWait = time.Second + + defaultReadBufferSize = 4096 + defaultWriteBufferSize = 4096 + + continuationFrame = 0 + noFrame = -1 +) + +// Close codes defined in RFC 6455, section 11.7. +const ( + CloseNormalClosure = 1000 + CloseGoingAway = 1001 + CloseProtocolError = 1002 + CloseUnsupportedData = 1003 + CloseNoStatusReceived = 1005 + CloseAbnormalClosure = 1006 + CloseInvalidFramePayloadData = 1007 + ClosePolicyViolation = 1008 + CloseMessageTooBig = 1009 + CloseMandatoryExtension = 1010 + CloseInternalServerErr = 1011 + CloseServiceRestart = 1012 + CloseTryAgainLater = 1013 + CloseTLSHandshake = 1015 +) + +// The message types are defined in RFC 6455, section 11.8. +const ( + // TextMessage denotes a text data message. The text message payload is + // interpreted as UTF-8 encoded text data. + TextMessage = 1 + + // BinaryMessage denotes a binary data message. + BinaryMessage = 2 + + // CloseMessage denotes a close control message. The optional message + // payload contains a numeric code and text. Use the FormatCloseMessage + // function to format a close message payload. + CloseMessage = 8 + + // PingMessage denotes a ping control message. The optional message payload + // is UTF-8 encoded text. + PingMessage = 9 + + // PongMessage denotes a pong control message. The optional message payload + // is UTF-8 encoded text. + PongMessage = 10 +) + +// ErrCloseSent is returned when the application writes a message to the +// connection after sending a close message. +var ErrCloseSent = errors.New("websocket: close sent") + +// ErrReadLimit is returned when reading a message that is larger than the +// read limit set for the connection. +var ErrReadLimit = errors.New("websocket: read limit exceeded") + +// netError satisfies the net Error interface. +type netError struct { + msg string + temporary bool + timeout bool +} + +func (e *netError) Error() string { return e.msg } +func (e *netError) Temporary() bool { return e.temporary } +func (e *netError) Timeout() bool { return e.timeout } + +// CloseError represents a close message. +type CloseError struct { + // Code is defined in RFC 6455, section 11.7. + Code int + + // Text is the optional text payload. + Text string +} + +func (e *CloseError) Error() string { + s := []byte("websocket: close ") + s = strconv.AppendInt(s, int64(e.Code), 10) + switch e.Code { + case CloseNormalClosure: + s = append(s, " (normal)"...) + case CloseGoingAway: + s = append(s, " (going away)"...) + case CloseProtocolError: + s = append(s, " (protocol error)"...) + case CloseUnsupportedData: + s = append(s, " (unsupported data)"...) + case CloseNoStatusReceived: + s = append(s, " (no status)"...) + case CloseAbnormalClosure: + s = append(s, " (abnormal closure)"...) + case CloseInvalidFramePayloadData: + s = append(s, " (invalid payload data)"...) + case ClosePolicyViolation: + s = append(s, " (policy violation)"...) + case CloseMessageTooBig: + s = append(s, " (message too big)"...) + case CloseMandatoryExtension: + s = append(s, " (mandatory extension missing)"...) + case CloseInternalServerErr: + s = append(s, " (internal server error)"...) + case CloseTLSHandshake: + s = append(s, " (TLS handshake error)"...) + } + if e.Text != "" { + s = append(s, ": "...) + s = append(s, e.Text...) + } + return string(s) +} + +// IsCloseError returns boolean indicating whether the error is a *CloseError +// with one of the specified codes. +func IsCloseError(err error, codes ...int) bool { + if e, ok := err.(*CloseError); ok { + for _, code := range codes { + if e.Code == code { + return true + } + } + } + return false +} + +// IsUnexpectedCloseError returns boolean indicating whether the error is a +// *CloseError with a code not in the list of expected codes. +func IsUnexpectedCloseError(err error, expectedCodes ...int) bool { + if e, ok := err.(*CloseError); ok { + for _, code := range expectedCodes { + if e.Code == code { + return false + } + } + return true + } + return false +} + +var ( + errWriteTimeout = &netError{msg: "websocket: write timeout", timeout: true, temporary: true} + errUnexpectedEOF = &CloseError{Code: CloseAbnormalClosure, Text: io.ErrUnexpectedEOF.Error()} + errBadWriteOpCode = errors.New("websocket: bad write message type") + errWriteClosed = errors.New("websocket: write closed") + errInvalidControlFrame = errors.New("websocket: invalid control frame") +) + +func newMaskKey() [4]byte { + n := rand.Uint32() + return [4]byte{byte(n), byte(n >> 8), byte(n >> 16), byte(n >> 24)} +} + +func hideTempErr(err error) error { + if e, ok := err.(net.Error); ok && e.Temporary() { + err = &netError{msg: e.Error(), timeout: e.Timeout()} + } + return err +} + +func isControl(frameType int) bool { + return frameType == CloseMessage || frameType == PingMessage || frameType == PongMessage +} + +func isData(frameType int) bool { + return frameType == TextMessage || frameType == BinaryMessage +} + +var validReceivedCloseCodes = map[int]bool{ + // see http://www.iana.org/assignments/websocket/websocket.xhtml#close-code-number + + CloseNormalClosure: true, + CloseGoingAway: true, + CloseProtocolError: true, + CloseUnsupportedData: true, + CloseNoStatusReceived: false, + CloseAbnormalClosure: false, + CloseInvalidFramePayloadData: true, + ClosePolicyViolation: true, + CloseMessageTooBig: true, + CloseMandatoryExtension: true, + CloseInternalServerErr: true, + CloseServiceRestart: true, + CloseTryAgainLater: true, + CloseTLSHandshake: false, +} + +func isValidReceivedCloseCode(code int) bool { + return validReceivedCloseCodes[code] || (code >= 3000 && code <= 4999) +} + +// BufferPool represents a pool of buffers. The *sync.Pool type satisfies this +// interface. The type of the value stored in a pool is not specified. +type BufferPool interface { + // Get gets a value from the pool or returns nil if the pool is empty. + Get() interface{} + // Put adds a value to the pool. + Put(interface{}) +} + +// writePoolData is the type added to the write buffer pool. This wrapper is +// used to prevent applications from peeking at and depending on the values +// added to the pool. +type writePoolData struct{ buf []byte } + +// The Conn type represents a WebSocket connection. +type Conn struct { + conn net.Conn + isServer bool + subprotocol string + + // Write fields + mu chan bool // used as mutex to protect write to conn + writeBuf []byte // frame is constructed in this buffer. + writePool BufferPool + writeBufSize int + writeDeadline time.Time + writer io.WriteCloser // the current writer returned to the application + isWriting bool // for best-effort concurrent write detection + + writeErrMu sync.Mutex + writeErr error + + enableWriteCompression bool + compressionLevel int + newCompressionWriter func(io.WriteCloser, int) io.WriteCloser + + // Read fields + reader io.ReadCloser // the current reader returned to the application + readErr error + br *bufio.Reader + // bytes remaining in current frame. + // set setReadRemaining to safely update this value and prevent overflow + readRemaining int64 + readFinal bool // true the current message has more frames. + readLength int64 // Message size. + readLimit int64 // Maximum message size. + readMaskPos int + readMaskKey [4]byte + handlePong func(string) error + handlePing func(string) error + handleClose func(int, string) error + readErrCount int + messageReader *messageReader // the current low-level reader + + readDecompress bool // whether last read frame had RSV1 set + newDecompressionReader func(io.Reader) io.ReadCloser +} + +func newConn(conn net.Conn, isServer bool, readBufferSize, writeBufferSize int, writeBufferPool BufferPool, br *bufio.Reader, writeBuf []byte) *Conn { + + if br == nil { + if readBufferSize == 0 { + readBufferSize = defaultReadBufferSize + } else if readBufferSize < maxControlFramePayloadSize { + // must be large enough for control frame + readBufferSize = maxControlFramePayloadSize + } + br = bufio.NewReaderSize(conn, readBufferSize) + } + + if writeBufferSize <= 0 { + writeBufferSize = defaultWriteBufferSize + } + writeBufferSize += maxFrameHeaderSize + + if writeBuf == nil && writeBufferPool == nil { + writeBuf = make([]byte, writeBufferSize) + } + + mu := make(chan bool, 1) + mu <- true + c := &Conn{ + isServer: isServer, + br: br, + conn: conn, + mu: mu, + readFinal: true, + writeBuf: writeBuf, + writePool: writeBufferPool, + writeBufSize: writeBufferSize, + enableWriteCompression: true, + compressionLevel: defaultCompressionLevel, + } + c.SetCloseHandler(nil) + c.SetPingHandler(nil) + c.SetPongHandler(nil) + return c +} + +// setReadRemaining tracks the number of bytes remaining on the connection. If n +// overflows, an ErrReadLimit is returned. +func (c *Conn) setReadRemaining(n int64) error { + if n < 0 { + return ErrReadLimit + } + + c.readRemaining = n + return nil +} + +// Subprotocol returns the negotiated protocol for the connection. +func (c *Conn) Subprotocol() string { + return c.subprotocol +} + +// Close closes the underlying network connection without sending or waiting +// for a close message. +func (c *Conn) Close() error { + return c.conn.Close() +} + +// LocalAddr returns the local network address. +func (c *Conn) LocalAddr() net.Addr { + return c.conn.LocalAddr() +} + +// RemoteAddr returns the remote network address. +func (c *Conn) RemoteAddr() net.Addr { + return c.conn.RemoteAddr() +} + +// Write methods + +func (c *Conn) writeFatal(err error) error { + err = hideTempErr(err) + c.writeErrMu.Lock() + if c.writeErr == nil { + c.writeErr = err + } + c.writeErrMu.Unlock() + return err +} + +func (c *Conn) read(n int) ([]byte, error) { + p, err := c.br.Peek(n) + if err == io.EOF { + err = errUnexpectedEOF + } + c.br.Discard(len(p)) + return p, err +} + +func (c *Conn) write(frameType int, deadline time.Time, buf0, buf1 []byte) error { + <-c.mu + defer func() { c.mu <- true }() + + c.writeErrMu.Lock() + err := c.writeErr + c.writeErrMu.Unlock() + if err != nil { + return err + } + + c.conn.SetWriteDeadline(deadline) + if len(buf1) == 0 { + _, err = c.conn.Write(buf0) + } else { + err = c.writeBufs(buf0, buf1) + } + if err != nil { + return c.writeFatal(err) + } + if frameType == CloseMessage { + c.writeFatal(ErrCloseSent) + } + return nil +} + +// WriteControl writes a control message with the given deadline. The allowed +// message types are CloseMessage, PingMessage and PongMessage. +func (c *Conn) WriteControl(messageType int, data []byte, deadline time.Time) error { + if !isControl(messageType) { + return errBadWriteOpCode + } + if len(data) > maxControlFramePayloadSize { + return errInvalidControlFrame + } + + b0 := byte(messageType) | finalBit + b1 := byte(len(data)) + if !c.isServer { + b1 |= maskBit + } + + buf := make([]byte, 0, maxFrameHeaderSize+maxControlFramePayloadSize) + buf = append(buf, b0, b1) + + if c.isServer { + buf = append(buf, data...) + } else { + key := newMaskKey() + buf = append(buf, key[:]...) + buf = append(buf, data...) + maskBytes(key, 0, buf[6:]) + } + + d := time.Hour * 1000 + if !deadline.IsZero() { + d = deadline.Sub(time.Now()) + if d < 0 { + return errWriteTimeout + } + } + + timer := time.NewTimer(d) + select { + case <-c.mu: + timer.Stop() + case <-timer.C: + return errWriteTimeout + } + defer func() { c.mu <- true }() + + c.writeErrMu.Lock() + err := c.writeErr + c.writeErrMu.Unlock() + if err != nil { + return err + } + + c.conn.SetWriteDeadline(deadline) + _, err = c.conn.Write(buf) + if err != nil { + return c.writeFatal(err) + } + if messageType == CloseMessage { + c.writeFatal(ErrCloseSent) + } + return err +} + +// beginMessage prepares a connection and message writer for a new message. +func (c *Conn) beginMessage(mw *messageWriter, messageType int) error { + // Close previous writer if not already closed by the application. It's + // probably better to return an error in this situation, but we cannot + // change this without breaking existing applications. + if c.writer != nil { + c.writer.Close() + c.writer = nil + } + + if !isControl(messageType) && !isData(messageType) { + return errBadWriteOpCode + } + + c.writeErrMu.Lock() + err := c.writeErr + c.writeErrMu.Unlock() + if err != nil { + return err + } + + mw.c = c + mw.frameType = messageType + mw.pos = maxFrameHeaderSize + + if c.writeBuf == nil { + wpd, ok := c.writePool.Get().(writePoolData) + if ok { + c.writeBuf = wpd.buf + } else { + c.writeBuf = make([]byte, c.writeBufSize) + } + } + return nil +} + +// NextWriter returns a writer for the next message to send. The writer's Close +// method flushes the complete message to the network. +// +// There can be at most one open writer on a connection. NextWriter closes the +// previous writer if the application has not already done so. +// +// All message types (TextMessage, BinaryMessage, CloseMessage, PingMessage and +// PongMessage) are supported. +func (c *Conn) NextWriter(messageType int) (io.WriteCloser, error) { + var mw messageWriter + if err := c.beginMessage(&mw, messageType); err != nil { + return nil, err + } + c.writer = &mw + if c.newCompressionWriter != nil && c.enableWriteCompression && isData(messageType) { + w := c.newCompressionWriter(c.writer, c.compressionLevel) + mw.compress = true + c.writer = w + } + return c.writer, nil +} + +type messageWriter struct { + c *Conn + compress bool // whether next call to flushFrame should set RSV1 + pos int // end of data in writeBuf. + frameType int // type of the current frame. + err error +} + +func (w *messageWriter) endMessage(err error) error { + if w.err != nil { + return err + } + c := w.c + w.err = err + c.writer = nil + if c.writePool != nil { + c.writePool.Put(writePoolData{buf: c.writeBuf}) + c.writeBuf = nil + } + return err +} + +// flushFrame writes buffered data and extra as a frame to the network. The +// final argument indicates that this is the last frame in the message. +func (w *messageWriter) flushFrame(final bool, extra []byte) error { + c := w.c + length := w.pos - maxFrameHeaderSize + len(extra) + + // Check for invalid control frames. + if isControl(w.frameType) && + (!final || length > maxControlFramePayloadSize) { + return w.endMessage(errInvalidControlFrame) + } + + b0 := byte(w.frameType) + if final { + b0 |= finalBit + } + if w.compress { + b0 |= rsv1Bit + } + w.compress = false + + b1 := byte(0) + if !c.isServer { + b1 |= maskBit + } + + // Assume that the frame starts at beginning of c.writeBuf. + framePos := 0 + if c.isServer { + // Adjust up if mask not included in the header. + framePos = 4 + } + + switch { + case length >= 65536: + c.writeBuf[framePos] = b0 + c.writeBuf[framePos+1] = b1 | 127 + binary.BigEndian.PutUint64(c.writeBuf[framePos+2:], uint64(length)) + case length > 125: + framePos += 6 + c.writeBuf[framePos] = b0 + c.writeBuf[framePos+1] = b1 | 126 + binary.BigEndian.PutUint16(c.writeBuf[framePos+2:], uint16(length)) + default: + framePos += 8 + c.writeBuf[framePos] = b0 + c.writeBuf[framePos+1] = b1 | byte(length) + } + + if !c.isServer { + key := newMaskKey() + copy(c.writeBuf[maxFrameHeaderSize-4:], key[:]) + maskBytes(key, 0, c.writeBuf[maxFrameHeaderSize:w.pos]) + if len(extra) > 0 { + return w.endMessage(c.writeFatal(errors.New("websocket: internal error, extra used in client mode"))) + } + } + + // Write the buffers to the connection with best-effort detection of + // concurrent writes. See the concurrency section in the package + // documentation for more info. + + if c.isWriting { + panic("concurrent write to websocket connection") + } + c.isWriting = true + + err := c.write(w.frameType, c.writeDeadline, c.writeBuf[framePos:w.pos], extra) + + if !c.isWriting { + panic("concurrent write to websocket connection") + } + c.isWriting = false + + if err != nil { + return w.endMessage(err) + } + + if final { + w.endMessage(errWriteClosed) + return nil + } + + // Setup for next frame. + w.pos = maxFrameHeaderSize + w.frameType = continuationFrame + return nil +} + +func (w *messageWriter) ncopy(max int) (int, error) { + n := len(w.c.writeBuf) - w.pos + if n <= 0 { + if err := w.flushFrame(false, nil); err != nil { + return 0, err + } + n = len(w.c.writeBuf) - w.pos + } + if n > max { + n = max + } + return n, nil +} + +func (w *messageWriter) Write(p []byte) (int, error) { + if w.err != nil { + return 0, w.err + } + + if len(p) > 2*len(w.c.writeBuf) && w.c.isServer { + // Don't buffer large messages. + err := w.flushFrame(false, p) + if err != nil { + return 0, err + } + return len(p), nil + } + + nn := len(p) + for len(p) > 0 { + n, err := w.ncopy(len(p)) + if err != nil { + return 0, err + } + copy(w.c.writeBuf[w.pos:], p[:n]) + w.pos += n + p = p[n:] + } + return nn, nil +} + +func (w *messageWriter) WriteString(p string) (int, error) { + if w.err != nil { + return 0, w.err + } + + nn := len(p) + for len(p) > 0 { + n, err := w.ncopy(len(p)) + if err != nil { + return 0, err + } + copy(w.c.writeBuf[w.pos:], p[:n]) + w.pos += n + p = p[n:] + } + return nn, nil +} + +func (w *messageWriter) ReadFrom(r io.Reader) (nn int64, err error) { + if w.err != nil { + return 0, w.err + } + for { + if w.pos == len(w.c.writeBuf) { + err = w.flushFrame(false, nil) + if err != nil { + break + } + } + var n int + n, err = r.Read(w.c.writeBuf[w.pos:]) + w.pos += n + nn += int64(n) + if err != nil { + if err == io.EOF { + err = nil + } + break + } + } + return nn, err +} + +func (w *messageWriter) Close() error { + if w.err != nil { + return w.err + } + return w.flushFrame(true, nil) +} + +// WritePreparedMessage writes prepared message into connection. +func (c *Conn) WritePreparedMessage(pm *PreparedMessage) error { + frameType, frameData, err := pm.frame(prepareKey{ + isServer: c.isServer, + compress: c.newCompressionWriter != nil && c.enableWriteCompression && isData(pm.messageType), + compressionLevel: c.compressionLevel, + }) + if err != nil { + return err + } + if c.isWriting { + panic("concurrent write to websocket connection") + } + c.isWriting = true + err = c.write(frameType, c.writeDeadline, frameData, nil) + if !c.isWriting { + panic("concurrent write to websocket connection") + } + c.isWriting = false + return err +} + +// WriteMessage is a helper method for getting a writer using NextWriter, +// writing the message and closing the writer. +func (c *Conn) WriteMessage(messageType int, data []byte) error { + + if c.isServer && (c.newCompressionWriter == nil || !c.enableWriteCompression) { + // Fast path with no allocations and single frame. + + var mw messageWriter + if err := c.beginMessage(&mw, messageType); err != nil { + return err + } + n := copy(c.writeBuf[mw.pos:], data) + mw.pos += n + data = data[n:] + return mw.flushFrame(true, data) + } + + w, err := c.NextWriter(messageType) + if err != nil { + return err + } + if _, err = w.Write(data); err != nil { + return err + } + return w.Close() +} + +// SetWriteDeadline sets the write deadline on the underlying network +// connection. After a write has timed out, the websocket state is corrupt and +// all future writes will return an error. A zero value for t means writes will +// not time out. +func (c *Conn) SetWriteDeadline(t time.Time) error { + c.writeDeadline = t + return nil +} + +// Read methods + +func (c *Conn) advanceFrame() (int, error) { + // 1. Skip remainder of previous frame. + + if c.readRemaining > 0 { + if _, err := io.CopyN(ioutil.Discard, c.br, c.readRemaining); err != nil { + return noFrame, err + } + } + + // 2. Read and parse first two bytes of frame header. + + p, err := c.read(2) + if err != nil { + return noFrame, err + } + + final := p[0]&finalBit != 0 + frameType := int(p[0] & 0xf) + mask := p[1]&maskBit != 0 + c.setReadRemaining(int64(p[1] & 0x7f)) + + c.readDecompress = false + if c.newDecompressionReader != nil && (p[0]&rsv1Bit) != 0 { + c.readDecompress = true + p[0] &^= rsv1Bit + } + + if rsv := p[0] & (rsv1Bit | rsv2Bit | rsv3Bit); rsv != 0 { + return noFrame, c.handleProtocolError("unexpected reserved bits 0x" + strconv.FormatInt(int64(rsv), 16)) + } + + switch frameType { + case CloseMessage, PingMessage, PongMessage: + if c.readRemaining > maxControlFramePayloadSize { + return noFrame, c.handleProtocolError("control frame length > 125") + } + if !final { + return noFrame, c.handleProtocolError("control frame not final") + } + case TextMessage, BinaryMessage: + if !c.readFinal { + return noFrame, c.handleProtocolError("message start before final message frame") + } + c.readFinal = final + case continuationFrame: + if c.readFinal { + return noFrame, c.handleProtocolError("continuation after final message frame") + } + c.readFinal = final + default: + return noFrame, c.handleProtocolError("unknown opcode " + strconv.Itoa(frameType)) + } + + // 3. Read and parse frame length as per + // https://tools.ietf.org/html/rfc6455#section-5.2 + // + // The length of the "Payload data", in bytes: if 0-125, that is the payload + // length. + // - If 126, the following 2 bytes interpreted as a 16-bit unsigned + // integer are the payload length. + // - If 127, the following 8 bytes interpreted as + // a 64-bit unsigned integer (the most significant bit MUST be 0) are the + // payload length. Multibyte length quantities are expressed in network byte + // order. + + switch c.readRemaining { + case 126: + p, err := c.read(2) + if err != nil { + return noFrame, err + } + + if err := c.setReadRemaining(int64(binary.BigEndian.Uint16(p))); err != nil { + return noFrame, err + } + case 127: + p, err := c.read(8) + if err != nil { + return noFrame, err + } + + if err := c.setReadRemaining(int64(binary.BigEndian.Uint64(p))); err != nil { + return noFrame, err + } + } + + // 4. Handle frame masking. + + if mask != c.isServer { + return noFrame, c.handleProtocolError("incorrect mask flag") + } + + if mask { + c.readMaskPos = 0 + p, err := c.read(len(c.readMaskKey)) + if err != nil { + return noFrame, err + } + copy(c.readMaskKey[:], p) + } + + // 5. For text and binary messages, enforce read limit and return. + + if frameType == continuationFrame || frameType == TextMessage || frameType == BinaryMessage { + + c.readLength += c.readRemaining + // Don't allow readLength to overflow in the presence of a large readRemaining + // counter. + if c.readLength < 0 { + return noFrame, ErrReadLimit + } + + if c.readLimit > 0 && c.readLength > c.readLimit { + c.WriteControl(CloseMessage, FormatCloseMessage(CloseMessageTooBig, ""), time.Now().Add(writeWait)) + return noFrame, ErrReadLimit + } + + return frameType, nil + } + + // 6. Read control frame payload. + + var payload []byte + if c.readRemaining > 0 { + payload, err = c.read(int(c.readRemaining)) + c.setReadRemaining(0) + if err != nil { + return noFrame, err + } + if c.isServer { + maskBytes(c.readMaskKey, 0, payload) + } + } + + // 7. Process control frame payload. + + switch frameType { + case PongMessage: + if err := c.handlePong(string(payload)); err != nil { + return noFrame, err + } + case PingMessage: + if err := c.handlePing(string(payload)); err != nil { + return noFrame, err + } + case CloseMessage: + closeCode := CloseNoStatusReceived + closeText := "" + if len(payload) >= 2 { + closeCode = int(binary.BigEndian.Uint16(payload)) + if !isValidReceivedCloseCode(closeCode) { + return noFrame, c.handleProtocolError("invalid close code") + } + closeText = string(payload[2:]) + if !utf8.ValidString(closeText) { + return noFrame, c.handleProtocolError("invalid utf8 payload in close frame") + } + } + if err := c.handleClose(closeCode, closeText); err != nil { + return noFrame, err + } + return noFrame, &CloseError{Code: closeCode, Text: closeText} + } + + return frameType, nil +} + +func (c *Conn) handleProtocolError(message string) error { + c.WriteControl(CloseMessage, FormatCloseMessage(CloseProtocolError, message), time.Now().Add(writeWait)) + return errors.New("websocket: " + message) +} + +// NextReader returns the next data message received from the peer. The +// returned messageType is either TextMessage or BinaryMessage. +// +// There can be at most one open reader on a connection. NextReader discards +// the previous message if the application has not already consumed it. +// +// Applications must break out of the application's read loop when this method +// returns a non-nil error value. Errors returned from this method are +// permanent. Once this method returns a non-nil error, all subsequent calls to +// this method return the same error. +func (c *Conn) NextReader() (messageType int, r io.Reader, err error) { + // Close previous reader, only relevant for decompression. + if c.reader != nil { + c.reader.Close() + c.reader = nil + } + + c.messageReader = nil + c.readLength = 0 + + for c.readErr == nil { + frameType, err := c.advanceFrame() + if err != nil { + c.readErr = hideTempErr(err) + break + } + + if frameType == TextMessage || frameType == BinaryMessage { + c.messageReader = &messageReader{c} + c.reader = c.messageReader + if c.readDecompress { + c.reader = c.newDecompressionReader(c.reader) + } + return frameType, c.reader, nil + } + } + + // Applications that do handle the error returned from this method spin in + // tight loop on connection failure. To help application developers detect + // this error, panic on repeated reads to the failed connection. + c.readErrCount++ + if c.readErrCount >= 1000 { + panic("repeated read on failed websocket connection") + } + + return noFrame, nil, c.readErr +} + +type messageReader struct{ c *Conn } + +func (r *messageReader) Read(b []byte) (int, error) { + c := r.c + if c.messageReader != r { + return 0, io.EOF + } + + for c.readErr == nil { + + if c.readRemaining > 0 { + if int64(len(b)) > c.readRemaining { + b = b[:c.readRemaining] + } + n, err := c.br.Read(b) + c.readErr = hideTempErr(err) + if c.isServer { + c.readMaskPos = maskBytes(c.readMaskKey, c.readMaskPos, b[:n]) + } + rem := c.readRemaining + rem -= int64(n) + c.setReadRemaining(rem) + if c.readRemaining > 0 && c.readErr == io.EOF { + c.readErr = errUnexpectedEOF + } + return n, c.readErr + } + + if c.readFinal { + c.messageReader = nil + return 0, io.EOF + } + + frameType, err := c.advanceFrame() + switch { + case err != nil: + c.readErr = hideTempErr(err) + case frameType == TextMessage || frameType == BinaryMessage: + c.readErr = errors.New("websocket: internal error, unexpected text or binary in Reader") + } + } + + err := c.readErr + if err == io.EOF && c.messageReader == r { + err = errUnexpectedEOF + } + return 0, err +} + +func (r *messageReader) Close() error { + return nil +} + +// ReadMessage is a helper method for getting a reader using NextReader and +// reading from that reader to a buffer. +func (c *Conn) ReadMessage() (messageType int, p []byte, err error) { + var r io.Reader + messageType, r, err = c.NextReader() + if err != nil { + return messageType, nil, err + } + p, err = ioutil.ReadAll(r) + return messageType, p, err +} + +// SetReadDeadline sets the read deadline on the underlying network connection. +// After a read has timed out, the websocket connection state is corrupt and +// all future reads will return an error. A zero value for t means reads will +// not time out. +func (c *Conn) SetReadDeadline(t time.Time) error { + return c.conn.SetReadDeadline(t) +} + +// SetReadLimit sets the maximum size in bytes for a message read from the peer. If a +// message exceeds the limit, the connection sends a close message to the peer +// and returns ErrReadLimit to the application. +func (c *Conn) SetReadLimit(limit int64) { + c.readLimit = limit +} + +// CloseHandler returns the current close handler +func (c *Conn) CloseHandler() func(code int, text string) error { + return c.handleClose +} + +// SetCloseHandler sets the handler for close messages received from the peer. +// The code argument to h is the received close code or CloseNoStatusReceived +// if the close message is empty. The default close handler sends a close +// message back to the peer. +// +// The handler function is called from the NextReader, ReadMessage and message +// reader Read methods. The application must read the connection to process +// close messages as described in the section on Control Messages above. +// +// The connection read methods return a CloseError when a close message is +// received. Most applications should handle close messages as part of their +// normal error handling. Applications should only set a close handler when the +// application must perform some action before sending a close message back to +// the peer. +func (c *Conn) SetCloseHandler(h func(code int, text string) error) { + if h == nil { + h = func(code int, text string) error { + message := FormatCloseMessage(code, "") + c.WriteControl(CloseMessage, message, time.Now().Add(writeWait)) + return nil + } + } + c.handleClose = h +} + +// PingHandler returns the current ping handler +func (c *Conn) PingHandler() func(appData string) error { + return c.handlePing +} + +// SetPingHandler sets the handler for ping messages received from the peer. +// The appData argument to h is the PING message application data. The default +// ping handler sends a pong to the peer. +// +// The handler function is called from the NextReader, ReadMessage and message +// reader Read methods. The application must read the connection to process +// ping messages as described in the section on Control Messages above. +func (c *Conn) SetPingHandler(h func(appData string) error) { + if h == nil { + h = func(message string) error { + err := c.WriteControl(PongMessage, []byte(message), time.Now().Add(writeWait)) + if err == ErrCloseSent { + return nil + } else if e, ok := err.(net.Error); ok && e.Temporary() { + return nil + } + return err + } + } + c.handlePing = h +} + +// PongHandler returns the current pong handler +func (c *Conn) PongHandler() func(appData string) error { + return c.handlePong +} + +// SetPongHandler sets the handler for pong messages received from the peer. +// The appData argument to h is the PONG message application data. The default +// pong handler does nothing. +// +// The handler function is called from the NextReader, ReadMessage and message +// reader Read methods. The application must read the connection to process +// pong messages as described in the section on Control Messages above. +func (c *Conn) SetPongHandler(h func(appData string) error) { + if h == nil { + h = func(string) error { return nil } + } + c.handlePong = h +} + +// UnderlyingConn returns the internal net.Conn. This can be used to further +// modifications to connection specific flags. +func (c *Conn) UnderlyingConn() net.Conn { + return c.conn +} + +// EnableWriteCompression enables and disables write compression of +// subsequent text and binary messages. This function is a noop if +// compression was not negotiated with the peer. +func (c *Conn) EnableWriteCompression(enable bool) { + c.enableWriteCompression = enable +} + +// SetCompressionLevel sets the flate compression level for subsequent text and +// binary messages. This function is a noop if compression was not negotiated +// with the peer. See the compress/flate package for a description of +// compression levels. +func (c *Conn) SetCompressionLevel(level int) error { + if !isValidCompressionLevel(level) { + return errors.New("websocket: invalid compression level") + } + c.compressionLevel = level + return nil +} + +// FormatCloseMessage formats closeCode and text as a WebSocket close message. +// An empty message is returned for code CloseNoStatusReceived. +func FormatCloseMessage(closeCode int, text string) []byte { + if closeCode == CloseNoStatusReceived { + // Return empty message because it's illegal to send + // CloseNoStatusReceived. Return non-nil value in case application + // checks for nil. + return []byte{} + } + buf := make([]byte, 2+len(text)) + binary.BigEndian.PutUint16(buf, uint16(closeCode)) + copy(buf[2:], text) + return buf +} diff --git a/vendor/github.com/gorilla/websocket/conn_write.go b/vendor/github.com/gorilla/websocket/conn_write.go new file mode 100644 index 00000000000..a509a21f87a --- /dev/null +++ b/vendor/github.com/gorilla/websocket/conn_write.go @@ -0,0 +1,15 @@ +// Copyright 2016 The Gorilla WebSocket Authors. All rights reserved. +// Use of this source code is governed by a BSD-style +// license that can be found in the LICENSE file. + +// +build go1.8 + +package websocket + +import "net" + +func (c *Conn) writeBufs(bufs ...[]byte) error { + b := net.Buffers(bufs) + _, err := b.WriteTo(c.conn) + return err +} diff --git a/vendor/github.com/gorilla/websocket/conn_write_legacy.go b/vendor/github.com/gorilla/websocket/conn_write_legacy.go new file mode 100644 index 00000000000..37edaff5a57 --- /dev/null +++ b/vendor/github.com/gorilla/websocket/conn_write_legacy.go @@ -0,0 +1,18 @@ +// Copyright 2016 The Gorilla WebSocket Authors. All rights reserved. +// Use of this source code is governed by a BSD-style +// license that can be found in the LICENSE file. + +// +build !go1.8 + +package websocket + +func (c *Conn) writeBufs(bufs ...[]byte) error { + for _, buf := range bufs { + if len(buf) > 0 { + if _, err := c.conn.Write(buf); err != nil { + return err + } + } + } + return nil +} diff --git a/vendor/github.com/gorilla/websocket/doc.go b/vendor/github.com/gorilla/websocket/doc.go new file mode 100644 index 00000000000..c6f4df8960f --- /dev/null +++ b/vendor/github.com/gorilla/websocket/doc.go @@ -0,0 +1,227 @@ +// Copyright 2013 The Gorilla WebSocket Authors. All rights reserved. +// Use of this source code is governed by a BSD-style +// license that can be found in the LICENSE file. + +// Package websocket implements the WebSocket protocol defined in RFC 6455. +// +// Overview +// +// The Conn type represents a WebSocket connection. A server application calls +// the Upgrader.Upgrade method from an HTTP request handler to get a *Conn: +// +// var upgrader = websocket.Upgrader{ +// ReadBufferSize: 1024, +// WriteBufferSize: 1024, +// } +// +// func handler(w http.ResponseWriter, r *http.Request) { +// conn, err := upgrader.Upgrade(w, r, nil) +// if err != nil { +// log.Println(err) +// return +// } +// ... Use conn to send and receive messages. +// } +// +// Call the connection's WriteMessage and ReadMessage methods to send and +// receive messages as a slice of bytes. This snippet of code shows how to echo +// messages using these methods: +// +// for { +// messageType, p, err := conn.ReadMessage() +// if err != nil { +// log.Println(err) +// return +// } +// if err := conn.WriteMessage(messageType, p); err != nil { +// log.Println(err) +// return +// } +// } +// +// In above snippet of code, p is a []byte and messageType is an int with value +// websocket.BinaryMessage or websocket.TextMessage. +// +// An application can also send and receive messages using the io.WriteCloser +// and io.Reader interfaces. To send a message, call the connection NextWriter +// method to get an io.WriteCloser, write the message to the writer and close +// the writer when done. To receive a message, call the connection NextReader +// method to get an io.Reader and read until io.EOF is returned. This snippet +// shows how to echo messages using the NextWriter and NextReader methods: +// +// for { +// messageType, r, err := conn.NextReader() +// if err != nil { +// return +// } +// w, err := conn.NextWriter(messageType) +// if err != nil { +// return err +// } +// if _, err := io.Copy(w, r); err != nil { +// return err +// } +// if err := w.Close(); err != nil { +// return err +// } +// } +// +// Data Messages +// +// The WebSocket protocol distinguishes between text and binary data messages. +// Text messages are interpreted as UTF-8 encoded text. The interpretation of +// binary messages is left to the application. +// +// This package uses the TextMessage and BinaryMessage integer constants to +// identify the two data message types. The ReadMessage and NextReader methods +// return the type of the received message. The messageType argument to the +// WriteMessage and NextWriter methods specifies the type of a sent message. +// +// It is the application's responsibility to ensure that text messages are +// valid UTF-8 encoded text. +// +// Control Messages +// +// The WebSocket protocol defines three types of control messages: close, ping +// and pong. Call the connection WriteControl, WriteMessage or NextWriter +// methods to send a control message to the peer. +// +// Connections handle received close messages by calling the handler function +// set with the SetCloseHandler method and by returning a *CloseError from the +// NextReader, ReadMessage or the message Read method. The default close +// handler sends a close message to the peer. +// +// Connections handle received ping messages by calling the handler function +// set with the SetPingHandler method. The default ping handler sends a pong +// message to the peer. +// +// Connections handle received pong messages by calling the handler function +// set with the SetPongHandler method. The default pong handler does nothing. +// If an application sends ping messages, then the application should set a +// pong handler to receive the corresponding pong. +// +// The control message handler functions are called from the NextReader, +// ReadMessage and message reader Read methods. The default close and ping +// handlers can block these methods for a short time when the handler writes to +// the connection. +// +// The application must read the connection to process close, ping and pong +// messages sent from the peer. If the application is not otherwise interested +// in messages from the peer, then the application should start a goroutine to +// read and discard messages from the peer. A simple example is: +// +// func readLoop(c *websocket.Conn) { +// for { +// if _, _, err := c.NextReader(); err != nil { +// c.Close() +// break +// } +// } +// } +// +// Concurrency +// +// Connections support one concurrent reader and one concurrent writer. +// +// Applications are responsible for ensuring that no more than one goroutine +// calls the write methods (NextWriter, SetWriteDeadline, WriteMessage, +// WriteJSON, EnableWriteCompression, SetCompressionLevel) concurrently and +// that no more than one goroutine calls the read methods (NextReader, +// SetReadDeadline, ReadMessage, ReadJSON, SetPongHandler, SetPingHandler) +// concurrently. +// +// The Close and WriteControl methods can be called concurrently with all other +// methods. +// +// Origin Considerations +// +// Web browsers allow Javascript applications to open a WebSocket connection to +// any host. It's up to the server to enforce an origin policy using the Origin +// request header sent by the browser. +// +// The Upgrader calls the function specified in the CheckOrigin field to check +// the origin. If the CheckOrigin function returns false, then the Upgrade +// method fails the WebSocket handshake with HTTP status 403. +// +// If the CheckOrigin field is nil, then the Upgrader uses a safe default: fail +// the handshake if the Origin request header is present and the Origin host is +// not equal to the Host request header. +// +// The deprecated package-level Upgrade function does not perform origin +// checking. The application is responsible for checking the Origin header +// before calling the Upgrade function. +// +// Buffers +// +// Connections buffer network input and output to reduce the number +// of system calls when reading or writing messages. +// +// Write buffers are also used for constructing WebSocket frames. See RFC 6455, +// Section 5 for a discussion of message framing. A WebSocket frame header is +// written to the network each time a write buffer is flushed to the network. +// Decreasing the size of the write buffer can increase the amount of framing +// overhead on the connection. +// +// The buffer sizes in bytes are specified by the ReadBufferSize and +// WriteBufferSize fields in the Dialer and Upgrader. The Dialer uses a default +// size of 4096 when a buffer size field is set to zero. The Upgrader reuses +// buffers created by the HTTP server when a buffer size field is set to zero. +// The HTTP server buffers have a size of 4096 at the time of this writing. +// +// The buffer sizes do not limit the size of a message that can be read or +// written by a connection. +// +// Buffers are held for the lifetime of the connection by default. If the +// Dialer or Upgrader WriteBufferPool field is set, then a connection holds the +// write buffer only when writing a message. +// +// Applications should tune the buffer sizes to balance memory use and +// performance. Increasing the buffer size uses more memory, but can reduce the +// number of system calls to read or write the network. In the case of writing, +// increasing the buffer size can reduce the number of frame headers written to +// the network. +// +// Some guidelines for setting buffer parameters are: +// +// Limit the buffer sizes to the maximum expected message size. Buffers larger +// than the largest message do not provide any benefit. +// +// Depending on the distribution of message sizes, setting the buffer size to +// to a value less than the maximum expected message size can greatly reduce +// memory use with a small impact on performance. Here's an example: If 99% of +// the messages are smaller than 256 bytes and the maximum message size is 512 +// bytes, then a buffer size of 256 bytes will result in 1.01 more system calls +// than a buffer size of 512 bytes. The memory savings is 50%. +// +// A write buffer pool is useful when the application has a modest number +// writes over a large number of connections. when buffers are pooled, a larger +// buffer size has a reduced impact on total memory use and has the benefit of +// reducing system calls and frame overhead. +// +// Compression EXPERIMENTAL +// +// Per message compression extensions (RFC 7692) are experimentally supported +// by this package in a limited capacity. Setting the EnableCompression option +// to true in Dialer or Upgrader will attempt to negotiate per message deflate +// support. +// +// var upgrader = websocket.Upgrader{ +// EnableCompression: true, +// } +// +// If compression was successfully negotiated with the connection's peer, any +// message received in compressed form will be automatically decompressed. +// All Read methods will return uncompressed bytes. +// +// Per message compression of messages written to a connection can be enabled +// or disabled by calling the corresponding Conn method: +// +// conn.EnableWriteCompression(false) +// +// Currently this package does not support compression with "context takeover". +// This means that messages must be compressed and decompressed in isolation, +// without retaining sliding window or dictionary state across messages. For +// more details refer to RFC 7692. +// +// Use of compression is experimental and may result in decreased performance. +package websocket diff --git a/vendor/github.com/gorilla/websocket/go.mod b/vendor/github.com/gorilla/websocket/go.mod new file mode 100644 index 00000000000..1a7afd5028a --- /dev/null +++ b/vendor/github.com/gorilla/websocket/go.mod @@ -0,0 +1,3 @@ +module github.com/gorilla/websocket + +go 1.12 diff --git a/vendor/github.com/gorilla/websocket/go.sum b/vendor/github.com/gorilla/websocket/go.sum new file mode 100644 index 00000000000..cf4fbbaa07a --- /dev/null +++ b/vendor/github.com/gorilla/websocket/go.sum @@ -0,0 +1,2 @@ +github.com/gorilla/websocket v1.4.0 h1:WDFjx/TMzVgy9VdMMQi2K2Emtwi2QcUQsztZ/zLaH/Q= +github.com/gorilla/websocket v1.4.0/go.mod h1:E7qHFY5m1UJ88s3WnNqhKjPHQ0heANvMoAMk2YaljkQ= diff --git a/vendor/github.com/gorilla/websocket/join.go b/vendor/github.com/gorilla/websocket/join.go new file mode 100644 index 00000000000..c64f8c82901 --- /dev/null +++ b/vendor/github.com/gorilla/websocket/join.go @@ -0,0 +1,42 @@ +// Copyright 2019 The Gorilla WebSocket Authors. All rights reserved. +// Use of this source code is governed by a BSD-style +// license that can be found in the LICENSE file. + +package websocket + +import ( + "io" + "strings" +) + +// JoinMessages concatenates received messages to create a single io.Reader. +// The string term is appended to each message. The returned reader does not +// support concurrent calls to the Read method. +func JoinMessages(c *Conn, term string) io.Reader { + return &joinReader{c: c, term: term} +} + +type joinReader struct { + c *Conn + term string + r io.Reader +} + +func (r *joinReader) Read(p []byte) (int, error) { + if r.r == nil { + var err error + _, r.r, err = r.c.NextReader() + if err != nil { + return 0, err + } + if r.term != "" { + r.r = io.MultiReader(r.r, strings.NewReader(r.term)) + } + } + n, err := r.r.Read(p) + if err == io.EOF { + err = nil + r.r = nil + } + return n, err +} diff --git a/vendor/github.com/gorilla/websocket/json.go b/vendor/github.com/gorilla/websocket/json.go new file mode 100644 index 00000000000..dc2c1f6415f --- /dev/null +++ b/vendor/github.com/gorilla/websocket/json.go @@ -0,0 +1,60 @@ +// Copyright 2013 The Gorilla WebSocket Authors. All rights reserved. +// Use of this source code is governed by a BSD-style +// license that can be found in the LICENSE file. + +package websocket + +import ( + "encoding/json" + "io" +) + +// WriteJSON writes the JSON encoding of v as a message. +// +// Deprecated: Use c.WriteJSON instead. +func WriteJSON(c *Conn, v interface{}) error { + return c.WriteJSON(v) +} + +// WriteJSON writes the JSON encoding of v as a message. +// +// See the documentation for encoding/json Marshal for details about the +// conversion of Go values to JSON. +func (c *Conn) WriteJSON(v interface{}) error { + w, err := c.NextWriter(TextMessage) + if err != nil { + return err + } + err1 := json.NewEncoder(w).Encode(v) + err2 := w.Close() + if err1 != nil { + return err1 + } + return err2 +} + +// ReadJSON reads the next JSON-encoded message from the connection and stores +// it in the value pointed to by v. +// +// Deprecated: Use c.ReadJSON instead. +func ReadJSON(c *Conn, v interface{}) error { + return c.ReadJSON(v) +} + +// ReadJSON reads the next JSON-encoded message from the connection and stores +// it in the value pointed to by v. +// +// See the documentation for the encoding/json Unmarshal function for details +// about the conversion of JSON to a Go value. +func (c *Conn) ReadJSON(v interface{}) error { + _, r, err := c.NextReader() + if err != nil { + return err + } + err = json.NewDecoder(r).Decode(v) + if err == io.EOF { + // One value is expected in the message. + err = io.ErrUnexpectedEOF + } + return err +} diff --git a/vendor/github.com/gorilla/websocket/mask.go b/vendor/github.com/gorilla/websocket/mask.go new file mode 100644 index 00000000000..577fce9efd7 --- /dev/null +++ b/vendor/github.com/gorilla/websocket/mask.go @@ -0,0 +1,54 @@ +// Copyright 2016 The Gorilla WebSocket Authors. All rights reserved. Use of +// this source code is governed by a BSD-style license that can be found in the +// LICENSE file. + +// +build !appengine + +package websocket + +import "unsafe" + +const wordSize = int(unsafe.Sizeof(uintptr(0))) + +func maskBytes(key [4]byte, pos int, b []byte) int { + // Mask one byte at a time for small buffers. + if len(b) < 2*wordSize { + for i := range b { + b[i] ^= key[pos&3] + pos++ + } + return pos & 3 + } + + // Mask one byte at a time to word boundary. + if n := int(uintptr(unsafe.Pointer(&b[0]))) % wordSize; n != 0 { + n = wordSize - n + for i := range b[:n] { + b[i] ^= key[pos&3] + pos++ + } + b = b[n:] + } + + // Create aligned word size key. + var k [wordSize]byte + for i := range k { + k[i] = key[(pos+i)&3] + } + kw := *(*uintptr)(unsafe.Pointer(&k)) + + // Mask one word at a time. + n := (len(b) / wordSize) * wordSize + for i := 0; i < n; i += wordSize { + *(*uintptr)(unsafe.Pointer(uintptr(unsafe.Pointer(&b[0])) + uintptr(i))) ^= kw + } + + // Mask one byte at a time for remaining bytes. + b = b[n:] + for i := range b { + b[i] ^= key[pos&3] + pos++ + } + + return pos & 3 +} diff --git a/vendor/github.com/gorilla/websocket/mask_safe.go b/vendor/github.com/gorilla/websocket/mask_safe.go new file mode 100644 index 00000000000..2aac060e52e --- /dev/null +++ b/vendor/github.com/gorilla/websocket/mask_safe.go @@ -0,0 +1,15 @@ +// Copyright 2016 The Gorilla WebSocket Authors. All rights reserved. Use of +// this source code is governed by a BSD-style license that can be found in the +// LICENSE file. + +// +build appengine + +package websocket + +func maskBytes(key [4]byte, pos int, b []byte) int { + for i := range b { + b[i] ^= key[pos&3] + pos++ + } + return pos & 3 +} diff --git a/vendor/github.com/gorilla/websocket/prepared.go b/vendor/github.com/gorilla/websocket/prepared.go new file mode 100644 index 00000000000..74ec565d2c3 --- /dev/null +++ b/vendor/github.com/gorilla/websocket/prepared.go @@ -0,0 +1,102 @@ +// Copyright 2017 The Gorilla WebSocket Authors. All rights reserved. +// Use of this source code is governed by a BSD-style +// license that can be found in the LICENSE file. + +package websocket + +import ( + "bytes" + "net" + "sync" + "time" +) + +// PreparedMessage caches on the wire representations of a message payload. +// Use PreparedMessage to efficiently send a message payload to multiple +// connections. PreparedMessage is especially useful when compression is used +// because the CPU and memory expensive compression operation can be executed +// once for a given set of compression options. +type PreparedMessage struct { + messageType int + data []byte + mu sync.Mutex + frames map[prepareKey]*preparedFrame +} + +// prepareKey defines a unique set of options to cache prepared frames in PreparedMessage. +type prepareKey struct { + isServer bool + compress bool + compressionLevel int +} + +// preparedFrame contains data in wire representation. +type preparedFrame struct { + once sync.Once + data []byte +} + +// NewPreparedMessage returns an initialized PreparedMessage. You can then send +// it to connection using WritePreparedMessage method. Valid wire +// representation will be calculated lazily only once for a set of current +// connection options. +func NewPreparedMessage(messageType int, data []byte) (*PreparedMessage, error) { + pm := &PreparedMessage{ + messageType: messageType, + frames: make(map[prepareKey]*preparedFrame), + data: data, + } + + // Prepare a plain server frame. + _, frameData, err := pm.frame(prepareKey{isServer: true, compress: false}) + if err != nil { + return nil, err + } + + // To protect against caller modifying the data argument, remember the data + // copied to the plain server frame. + pm.data = frameData[len(frameData)-len(data):] + return pm, nil +} + +func (pm *PreparedMessage) frame(key prepareKey) (int, []byte, error) { + pm.mu.Lock() + frame, ok := pm.frames[key] + if !ok { + frame = &preparedFrame{} + pm.frames[key] = frame + } + pm.mu.Unlock() + + var err error + frame.once.Do(func() { + // Prepare a frame using a 'fake' connection. + // TODO: Refactor code in conn.go to allow more direct construction of + // the frame. + mu := make(chan bool, 1) + mu <- true + var nc prepareConn + c := &Conn{ + conn: &nc, + mu: mu, + isServer: key.isServer, + compressionLevel: key.compressionLevel, + enableWriteCompression: true, + writeBuf: make([]byte, defaultWriteBufferSize+maxFrameHeaderSize), + } + if key.compress { + c.newCompressionWriter = compressNoContextTakeover + } + err = c.WriteMessage(pm.messageType, pm.data) + frame.data = nc.buf.Bytes() + }) + return pm.messageType, frame.data, err +} + +type prepareConn struct { + buf bytes.Buffer + net.Conn +} + +func (pc *prepareConn) Write(p []byte) (int, error) { return pc.buf.Write(p) } +func (pc *prepareConn) SetWriteDeadline(t time.Time) error { return nil } diff --git a/vendor/github.com/gorilla/websocket/proxy.go b/vendor/github.com/gorilla/websocket/proxy.go new file mode 100644 index 00000000000..e87a8c9f0c9 --- /dev/null +++ b/vendor/github.com/gorilla/websocket/proxy.go @@ -0,0 +1,77 @@ +// Copyright 2017 The Gorilla WebSocket Authors. All rights reserved. +// Use of this source code is governed by a BSD-style +// license that can be found in the LICENSE file. + +package websocket + +import ( + "bufio" + "encoding/base64" + "errors" + "net" + "net/http" + "net/url" + "strings" +) + +type netDialerFunc func(network, addr string) (net.Conn, error) + +func (fn netDialerFunc) Dial(network, addr string) (net.Conn, error) { + return fn(network, addr) +} + +func init() { + proxy_RegisterDialerType("http", func(proxyURL *url.URL, forwardDialer proxy_Dialer) (proxy_Dialer, error) { + return &httpProxyDialer{proxyURL: proxyURL, forwardDial: forwardDialer.Dial}, nil + }) +} + +type httpProxyDialer struct { + proxyURL *url.URL + forwardDial func(network, addr string) (net.Conn, error) +} + +func (hpd *httpProxyDialer) Dial(network string, addr string) (net.Conn, error) { + hostPort, _ := hostPortNoPort(hpd.proxyURL) + conn, err := hpd.forwardDial(network, hostPort) + if err != nil { + return nil, err + } + + connectHeader := make(http.Header) + if user := hpd.proxyURL.User; user != nil { + proxyUser := user.Username() + if proxyPassword, passwordSet := user.Password(); passwordSet { + credential := base64.StdEncoding.EncodeToString([]byte(proxyUser + ":" + proxyPassword)) + connectHeader.Set("Proxy-Authorization", "Basic "+credential) + } + } + + connectReq := &http.Request{ + Method: "CONNECT", + URL: &url.URL{Opaque: addr}, + Host: addr, + Header: connectHeader, + } + + if err := connectReq.Write(conn); err != nil { + conn.Close() + return nil, err + } + + // Read response. It's OK to use and discard buffered reader here becaue + // the remote server does not speak until spoken to. + br := bufio.NewReader(conn) + resp, err := http.ReadResponse(br, connectReq) + if err != nil { + conn.Close() + return nil, err + } + + if resp.StatusCode != 200 { + conn.Close() + f := strings.SplitN(resp.Status, " ", 2) + return nil, errors.New(f[1]) + } + return conn, nil +} diff --git a/vendor/github.com/gorilla/websocket/server.go b/vendor/github.com/gorilla/websocket/server.go new file mode 100644 index 00000000000..887d558918c --- /dev/null +++ b/vendor/github.com/gorilla/websocket/server.go @@ -0,0 +1,363 @@ +// Copyright 2013 The Gorilla WebSocket Authors. All rights reserved. +// Use of this source code is governed by a BSD-style +// license that can be found in the LICENSE file. + +package websocket + +import ( + "bufio" + "errors" + "io" + "net/http" + "net/url" + "strings" + "time" +) + +// HandshakeError describes an error with the handshake from the peer. +type HandshakeError struct { + message string +} + +func (e HandshakeError) Error() string { return e.message } + +// Upgrader specifies parameters for upgrading an HTTP connection to a +// WebSocket connection. +type Upgrader struct { + // HandshakeTimeout specifies the duration for the handshake to complete. + HandshakeTimeout time.Duration + + // ReadBufferSize and WriteBufferSize specify I/O buffer sizes in bytes. If a buffer + // size is zero, then buffers allocated by the HTTP server are used. The + // I/O buffer sizes do not limit the size of the messages that can be sent + // or received. + ReadBufferSize, WriteBufferSize int + + // WriteBufferPool is a pool of buffers for write operations. If the value + // is not set, then write buffers are allocated to the connection for the + // lifetime of the connection. + // + // A pool is most useful when the application has a modest volume of writes + // across a large number of connections. + // + // Applications should use a single pool for each unique value of + // WriteBufferSize. + WriteBufferPool BufferPool + + // Subprotocols specifies the server's supported protocols in order of + // preference. If this field is not nil, then the Upgrade method negotiates a + // subprotocol by selecting the first match in this list with a protocol + // requested by the client. If there's no match, then no protocol is + // negotiated (the Sec-Websocket-Protocol header is not included in the + // handshake response). + Subprotocols []string + + // Error specifies the function for generating HTTP error responses. If Error + // is nil, then http.Error is used to generate the HTTP response. + Error func(w http.ResponseWriter, r *http.Request, status int, reason error) + + // CheckOrigin returns true if the request Origin header is acceptable. If + // CheckOrigin is nil, then a safe default is used: return false if the + // Origin request header is present and the origin host is not equal to + // request Host header. + // + // A CheckOrigin function should carefully validate the request origin to + // prevent cross-site request forgery. + CheckOrigin func(r *http.Request) bool + + // EnableCompression specify if the server should attempt to negotiate per + // message compression (RFC 7692). Setting this value to true does not + // guarantee that compression will be supported. Currently only "no context + // takeover" modes are supported. + EnableCompression bool +} + +func (u *Upgrader) returnError(w http.ResponseWriter, r *http.Request, status int, reason string) (*Conn, error) { + err := HandshakeError{reason} + if u.Error != nil { + u.Error(w, r, status, err) + } else { + w.Header().Set("Sec-Websocket-Version", "13") + http.Error(w, http.StatusText(status), status) + } + return nil, err +} + +// checkSameOrigin returns true if the origin is not set or is equal to the request host. +func checkSameOrigin(r *http.Request) bool { + origin := r.Header["Origin"] + if len(origin) == 0 { + return true + } + u, err := url.Parse(origin[0]) + if err != nil { + return false + } + return equalASCIIFold(u.Host, r.Host) +} + +func (u *Upgrader) selectSubprotocol(r *http.Request, responseHeader http.Header) string { + if u.Subprotocols != nil { + clientProtocols := Subprotocols(r) + for _, serverProtocol := range u.Subprotocols { + for _, clientProtocol := range clientProtocols { + if clientProtocol == serverProtocol { + return clientProtocol + } + } + } + } else if responseHeader != nil { + return responseHeader.Get("Sec-Websocket-Protocol") + } + return "" +} + +// Upgrade upgrades the HTTP server connection to the WebSocket protocol. +// +// The responseHeader is included in the response to the client's upgrade +// request. Use the responseHeader to specify cookies (Set-Cookie) and the +// application negotiated subprotocol (Sec-WebSocket-Protocol). +// +// If the upgrade fails, then Upgrade replies to the client with an HTTP error +// response. +func (u *Upgrader) Upgrade(w http.ResponseWriter, r *http.Request, responseHeader http.Header) (*Conn, error) { + const badHandshake = "websocket: the client is not using the websocket protocol: " + + if !tokenListContainsValue(r.Header, "Connection", "upgrade") { + return u.returnError(w, r, http.StatusBadRequest, badHandshake+"'upgrade' token not found in 'Connection' header") + } + + if !tokenListContainsValue(r.Header, "Upgrade", "websocket") { + return u.returnError(w, r, http.StatusBadRequest, badHandshake+"'websocket' token not found in 'Upgrade' header") + } + + if r.Method != "GET" { + return u.returnError(w, r, http.StatusMethodNotAllowed, badHandshake+"request method is not GET") + } + + if !tokenListContainsValue(r.Header, "Sec-Websocket-Version", "13") { + return u.returnError(w, r, http.StatusBadRequest, "websocket: unsupported version: 13 not found in 'Sec-Websocket-Version' header") + } + + if _, ok := responseHeader["Sec-Websocket-Extensions"]; ok { + return u.returnError(w, r, http.StatusInternalServerError, "websocket: application specific 'Sec-WebSocket-Extensions' headers are unsupported") + } + + checkOrigin := u.CheckOrigin + if checkOrigin == nil { + checkOrigin = checkSameOrigin + } + if !checkOrigin(r) { + return u.returnError(w, r, http.StatusForbidden, "websocket: request origin not allowed by Upgrader.CheckOrigin") + } + + challengeKey := r.Header.Get("Sec-Websocket-Key") + if challengeKey == "" { + return u.returnError(w, r, http.StatusBadRequest, "websocket: not a websocket handshake: 'Sec-WebSocket-Key' header is missing or blank") + } + + subprotocol := u.selectSubprotocol(r, responseHeader) + + // Negotiate PMCE + var compress bool + if u.EnableCompression { + for _, ext := range parseExtensions(r.Header) { + if ext[""] != "permessage-deflate" { + continue + } + compress = true + break + } + } + + h, ok := w.(http.Hijacker) + if !ok { + return u.returnError(w, r, http.StatusInternalServerError, "websocket: response does not implement http.Hijacker") + } + var brw *bufio.ReadWriter + netConn, brw, err := h.Hijack() + if err != nil { + return u.returnError(w, r, http.StatusInternalServerError, err.Error()) + } + + if brw.Reader.Buffered() > 0 { + netConn.Close() + return nil, errors.New("websocket: client sent data before handshake is complete") + } + + var br *bufio.Reader + if u.ReadBufferSize == 0 && bufioReaderSize(netConn, brw.Reader) > 256 { + // Reuse hijacked buffered reader as connection reader. + br = brw.Reader + } + + buf := bufioWriterBuffer(netConn, brw.Writer) + + var writeBuf []byte + if u.WriteBufferPool == nil && u.WriteBufferSize == 0 && len(buf) >= maxFrameHeaderSize+256 { + // Reuse hijacked write buffer as connection buffer. + writeBuf = buf + } + + c := newConn(netConn, true, u.ReadBufferSize, u.WriteBufferSize, u.WriteBufferPool, br, writeBuf) + c.subprotocol = subprotocol + + if compress { + c.newCompressionWriter = compressNoContextTakeover + c.newDecompressionReader = decompressNoContextTakeover + } + + // Use larger of hijacked buffer and connection write buffer for header. + p := buf + if len(c.writeBuf) > len(p) { + p = c.writeBuf + } + p = p[:0] + + p = append(p, "HTTP/1.1 101 Switching Protocols\r\nUpgrade: websocket\r\nConnection: Upgrade\r\nSec-WebSocket-Accept: "...) + p = append(p, computeAcceptKey(challengeKey)...) + p = append(p, "\r\n"...) + if c.subprotocol != "" { + p = append(p, "Sec-WebSocket-Protocol: "...) + p = append(p, c.subprotocol...) + p = append(p, "\r\n"...) + } + if compress { + p = append(p, "Sec-WebSocket-Extensions: permessage-deflate; server_no_context_takeover; client_no_context_takeover\r\n"...) + } + for k, vs := range responseHeader { + if k == "Sec-Websocket-Protocol" { + continue + } + for _, v := range vs { + p = append(p, k...) + p = append(p, ": "...) + for i := 0; i < len(v); i++ { + b := v[i] + if b <= 31 { + // prevent response splitting. + b = ' ' + } + p = append(p, b) + } + p = append(p, "\r\n"...) + } + } + p = append(p, "\r\n"...) + + // Clear deadlines set by HTTP server. + netConn.SetDeadline(time.Time{}) + + if u.HandshakeTimeout > 0 { + netConn.SetWriteDeadline(time.Now().Add(u.HandshakeTimeout)) + } + if _, err = netConn.Write(p); err != nil { + netConn.Close() + return nil, err + } + if u.HandshakeTimeout > 0 { + netConn.SetWriteDeadline(time.Time{}) + } + + return c, nil +} + +// Upgrade upgrades the HTTP server connection to the WebSocket protocol. +// +// Deprecated: Use websocket.Upgrader instead. +// +// Upgrade does not perform origin checking. The application is responsible for +// checking the Origin header before calling Upgrade. An example implementation +// of the same origin policy check is: +// +// if req.Header.Get("Origin") != "http://"+req.Host { +// http.Error(w, "Origin not allowed", http.StatusForbidden) +// return +// } +// +// If the endpoint supports subprotocols, then the application is responsible +// for negotiating the protocol used on the connection. Use the Subprotocols() +// function to get the subprotocols requested by the client. Use the +// Sec-Websocket-Protocol response header to specify the subprotocol selected +// by the application. +// +// The responseHeader is included in the response to the client's upgrade +// request. Use the responseHeader to specify cookies (Set-Cookie) and the +// negotiated subprotocol (Sec-Websocket-Protocol). +// +// The connection buffers IO to the underlying network connection. The +// readBufSize and writeBufSize parameters specify the size of the buffers to +// use. Messages can be larger than the buffers. +// +// If the request is not a valid WebSocket handshake, then Upgrade returns an +// error of type HandshakeError. Applications should handle this error by +// replying to the client with an HTTP error response. +func Upgrade(w http.ResponseWriter, r *http.Request, responseHeader http.Header, readBufSize, writeBufSize int) (*Conn, error) { + u := Upgrader{ReadBufferSize: readBufSize, WriteBufferSize: writeBufSize} + u.Error = func(w http.ResponseWriter, r *http.Request, status int, reason error) { + // don't return errors to maintain backwards compatibility + } + u.CheckOrigin = func(r *http.Request) bool { + // allow all connections by default + return true + } + return u.Upgrade(w, r, responseHeader) +} + +// Subprotocols returns the subprotocols requested by the client in the +// Sec-Websocket-Protocol header. +func Subprotocols(r *http.Request) []string { + h := strings.TrimSpace(r.Header.Get("Sec-Websocket-Protocol")) + if h == "" { + return nil + } + protocols := strings.Split(h, ",") + for i := range protocols { + protocols[i] = strings.TrimSpace(protocols[i]) + } + return protocols +} + +// IsWebSocketUpgrade returns true if the client requested upgrade to the +// WebSocket protocol. +func IsWebSocketUpgrade(r *http.Request) bool { + return tokenListContainsValue(r.Header, "Connection", "upgrade") && + tokenListContainsValue(r.Header, "Upgrade", "websocket") +} + +// bufioReaderSize size returns the size of a bufio.Reader. +func bufioReaderSize(originalReader io.Reader, br *bufio.Reader) int { + // This code assumes that peek on a reset reader returns + // bufio.Reader.buf[:0]. + // TODO: Use bufio.Reader.Size() after Go 1.10 + br.Reset(originalReader) + if p, err := br.Peek(0); err == nil { + return cap(p) + } + return 0 +} + +// writeHook is an io.Writer that records the last slice passed to it vio +// io.Writer.Write. +type writeHook struct { + p []byte +} + +func (wh *writeHook) Write(p []byte) (int, error) { + wh.p = p + return len(p), nil +} + +// bufioWriterBuffer grabs the buffer from a bufio.Writer. +func bufioWriterBuffer(originalWriter io.Writer, bw *bufio.Writer) []byte { + // This code assumes that bufio.Writer.buf[:1] is passed to the + // bufio.Writer's underlying writer. + var wh writeHook + bw.Reset(&wh) + bw.WriteByte(0) + bw.Flush() + + bw.Reset(originalWriter) + + return wh.p[:cap(wh.p)] +} diff --git a/vendor/github.com/gorilla/websocket/trace.go b/vendor/github.com/gorilla/websocket/trace.go new file mode 100644 index 00000000000..834f122a00d --- /dev/null +++ b/vendor/github.com/gorilla/websocket/trace.go @@ -0,0 +1,19 @@ +// +build go1.8 + +package websocket + +import ( + "crypto/tls" + "net/http/httptrace" +) + +func doHandshakeWithTrace(trace *httptrace.ClientTrace, tlsConn *tls.Conn, cfg *tls.Config) error { + if trace.TLSHandshakeStart != nil { + trace.TLSHandshakeStart() + } + err := doHandshake(tlsConn, cfg) + if trace.TLSHandshakeDone != nil { + trace.TLSHandshakeDone(tlsConn.ConnectionState(), err) + } + return err +} diff --git a/vendor/github.com/gorilla/websocket/trace_17.go b/vendor/github.com/gorilla/websocket/trace_17.go new file mode 100644 index 00000000000..77d05a0b574 --- /dev/null +++ b/vendor/github.com/gorilla/websocket/trace_17.go @@ -0,0 +1,12 @@ +// +build !go1.8 + +package websocket + +import ( + "crypto/tls" + "net/http/httptrace" +) + +func doHandshakeWithTrace(trace *httptrace.ClientTrace, tlsConn *tls.Conn, cfg *tls.Config) error { + return doHandshake(tlsConn, cfg) +} diff --git a/vendor/github.com/gorilla/websocket/util.go b/vendor/github.com/gorilla/websocket/util.go new file mode 100644 index 00000000000..7bf2f66c674 --- /dev/null +++ b/vendor/github.com/gorilla/websocket/util.go @@ -0,0 +1,283 @@ +// Copyright 2013 The Gorilla WebSocket Authors. All rights reserved. +// Use of this source code is governed by a BSD-style +// license that can be found in the LICENSE file. + +package websocket + +import ( + "crypto/rand" + "crypto/sha1" + "encoding/base64" + "io" + "net/http" + "strings" + "unicode/utf8" +) + +var keyGUID = []byte("258EAFA5-E914-47DA-95CA-C5AB0DC85B11") + +func computeAcceptKey(challengeKey string) string { + h := sha1.New() + h.Write([]byte(challengeKey)) + h.Write(keyGUID) + return base64.StdEncoding.EncodeToString(h.Sum(nil)) +} + +func generateChallengeKey() (string, error) { + p := make([]byte, 16) + if _, err := io.ReadFull(rand.Reader, p); err != nil { + return "", err + } + return base64.StdEncoding.EncodeToString(p), nil +} + +// Token octets per RFC 2616. +var isTokenOctet = [256]bool{ + '!': true, + '#': true, + '$': true, + '%': true, + '&': true, + '\'': true, + '*': true, + '+': true, + '-': true, + '.': true, + '0': true, + '1': true, + '2': true, + '3': true, + '4': true, + '5': true, + '6': true, + '7': true, + '8': true, + '9': true, + 'A': true, + 'B': true, + 'C': true, + 'D': true, + 'E': true, + 'F': true, + 'G': true, + 'H': true, + 'I': true, + 'J': true, + 'K': true, + 'L': true, + 'M': true, + 'N': true, + 'O': true, + 'P': true, + 'Q': true, + 'R': true, + 'S': true, + 'T': true, + 'U': true, + 'W': true, + 'V': true, + 'X': true, + 'Y': true, + 'Z': true, + '^': true, + '_': true, + '`': true, + 'a': true, + 'b': true, + 'c': true, + 'd': true, + 'e': true, + 'f': true, + 'g': true, + 'h': true, + 'i': true, + 'j': true, + 'k': true, + 'l': true, + 'm': true, + 'n': true, + 'o': true, + 'p': true, + 'q': true, + 'r': true, + 's': true, + 't': true, + 'u': true, + 'v': true, + 'w': true, + 'x': true, + 'y': true, + 'z': true, + '|': true, + '~': true, +} + +// skipSpace returns a slice of the string s with all leading RFC 2616 linear +// whitespace removed. +func skipSpace(s string) (rest string) { + i := 0 + for ; i < len(s); i++ { + if b := s[i]; b != ' ' && b != '\t' { + break + } + } + return s[i:] +} + +// nextToken returns the leading RFC 2616 token of s and the string following +// the token. +func nextToken(s string) (token, rest string) { + i := 0 + for ; i < len(s); i++ { + if !isTokenOctet[s[i]] { + break + } + } + return s[:i], s[i:] +} + +// nextTokenOrQuoted returns the leading token or quoted string per RFC 2616 +// and the string following the token or quoted string. +func nextTokenOrQuoted(s string) (value string, rest string) { + if !strings.HasPrefix(s, "\"") { + return nextToken(s) + } + s = s[1:] + for i := 0; i < len(s); i++ { + switch s[i] { + case '"': + return s[:i], s[i+1:] + case '\\': + p := make([]byte, len(s)-1) + j := copy(p, s[:i]) + escape := true + for i = i + 1; i < len(s); i++ { + b := s[i] + switch { + case escape: + escape = false + p[j] = b + j++ + case b == '\\': + escape = true + case b == '"': + return string(p[:j]), s[i+1:] + default: + p[j] = b + j++ + } + } + return "", "" + } + } + return "", "" +} + +// equalASCIIFold returns true if s is equal to t with ASCII case folding as +// defined in RFC 4790. +func equalASCIIFold(s, t string) bool { + for s != "" && t != "" { + sr, size := utf8.DecodeRuneInString(s) + s = s[size:] + tr, size := utf8.DecodeRuneInString(t) + t = t[size:] + if sr == tr { + continue + } + if 'A' <= sr && sr <= 'Z' { + sr = sr + 'a' - 'A' + } + if 'A' <= tr && tr <= 'Z' { + tr = tr + 'a' - 'A' + } + if sr != tr { + return false + } + } + return s == t +} + +// tokenListContainsValue returns true if the 1#token header with the given +// name contains a token equal to value with ASCII case folding. +func tokenListContainsValue(header http.Header, name string, value string) bool { +headers: + for _, s := range header[name] { + for { + var t string + t, s = nextToken(skipSpace(s)) + if t == "" { + continue headers + } + s = skipSpace(s) + if s != "" && s[0] != ',' { + continue headers + } + if equalASCIIFold(t, value) { + return true + } + if s == "" { + continue headers + } + s = s[1:] + } + } + return false +} + +// parseExtensions parses WebSocket extensions from a header. +func parseExtensions(header http.Header) []map[string]string { + // From RFC 6455: + // + // Sec-WebSocket-Extensions = extension-list + // extension-list = 1#extension + // extension = extension-token *( ";" extension-param ) + // extension-token = registered-token + // registered-token = token + // extension-param = token [ "=" (token | quoted-string) ] + // ;When using the quoted-string syntax variant, the value + // ;after quoted-string unescaping MUST conform to the + // ;'token' ABNF. + + var result []map[string]string +headers: + for _, s := range header["Sec-Websocket-Extensions"] { + for { + var t string + t, s = nextToken(skipSpace(s)) + if t == "" { + continue headers + } + ext := map[string]string{"": t} + for { + s = skipSpace(s) + if !strings.HasPrefix(s, ";") { + break + } + var k string + k, s = nextToken(skipSpace(s[1:])) + if k == "" { + continue headers + } + s = skipSpace(s) + var v string + if strings.HasPrefix(s, "=") { + v, s = nextTokenOrQuoted(skipSpace(s[1:])) + s = skipSpace(s) + } + if s != "" && s[0] != ',' && s[0] != ';' { + continue headers + } + ext[k] = v + } + if s != "" && s[0] != ',' { + continue headers + } + result = append(result, ext) + if s == "" { + continue headers + } + s = s[1:] + } + } + return result +} diff --git a/vendor/github.com/gorilla/websocket/x_net_proxy.go b/vendor/github.com/gorilla/websocket/x_net_proxy.go new file mode 100644 index 00000000000..2e668f6b882 --- /dev/null +++ b/vendor/github.com/gorilla/websocket/x_net_proxy.go @@ -0,0 +1,473 @@ +// Code generated by golang.org/x/tools/cmd/bundle. DO NOT EDIT. +//go:generate bundle -o x_net_proxy.go golang.org/x/net/proxy + +// Package proxy provides support for a variety of protocols to proxy network +// data. +// + +package websocket + +import ( + "errors" + "io" + "net" + "net/url" + "os" + "strconv" + "strings" + "sync" +) + +type proxy_direct struct{} + +// Direct is a direct proxy: one that makes network connections directly. +var proxy_Direct = proxy_direct{} + +func (proxy_direct) Dial(network, addr string) (net.Conn, error) { + return net.Dial(network, addr) +} + +// A PerHost directs connections to a default Dialer unless the host name +// requested matches one of a number of exceptions. +type proxy_PerHost struct { + def, bypass proxy_Dialer + + bypassNetworks []*net.IPNet + bypassIPs []net.IP + bypassZones []string + bypassHosts []string +} + +// NewPerHost returns a PerHost Dialer that directs connections to either +// defaultDialer or bypass, depending on whether the connection matches one of +// the configured rules. +func proxy_NewPerHost(defaultDialer, bypass proxy_Dialer) *proxy_PerHost { + return &proxy_PerHost{ + def: defaultDialer, + bypass: bypass, + } +} + +// Dial connects to the address addr on the given network through either +// defaultDialer or bypass. +func (p *proxy_PerHost) Dial(network, addr string) (c net.Conn, err error) { + host, _, err := net.SplitHostPort(addr) + if err != nil { + return nil, err + } + + return p.dialerForRequest(host).Dial(network, addr) +} + +func (p *proxy_PerHost) dialerForRequest(host string) proxy_Dialer { + if ip := net.ParseIP(host); ip != nil { + for _, net := range p.bypassNetworks { + if net.Contains(ip) { + return p.bypass + } + } + for _, bypassIP := range p.bypassIPs { + if bypassIP.Equal(ip) { + return p.bypass + } + } + return p.def + } + + for _, zone := range p.bypassZones { + if strings.HasSuffix(host, zone) { + return p.bypass + } + if host == zone[1:] { + // For a zone ".example.com", we match "example.com" + // too. + return p.bypass + } + } + for _, bypassHost := range p.bypassHosts { + if bypassHost == host { + return p.bypass + } + } + return p.def +} + +// AddFromString parses a string that contains comma-separated values +// specifying hosts that should use the bypass proxy. Each value is either an +// IP address, a CIDR range, a zone (*.example.com) or a host name +// (localhost). A best effort is made to parse the string and errors are +// ignored. +func (p *proxy_PerHost) AddFromString(s string) { + hosts := strings.Split(s, ",") + for _, host := range hosts { + host = strings.TrimSpace(host) + if len(host) == 0 { + continue + } + if strings.Contains(host, "/") { + // We assume that it's a CIDR address like 127.0.0.0/8 + if _, net, err := net.ParseCIDR(host); err == nil { + p.AddNetwork(net) + } + continue + } + if ip := net.ParseIP(host); ip != nil { + p.AddIP(ip) + continue + } + if strings.HasPrefix(host, "*.") { + p.AddZone(host[1:]) + continue + } + p.AddHost(host) + } +} + +// AddIP specifies an IP address that will use the bypass proxy. Note that +// this will only take effect if a literal IP address is dialed. A connection +// to a named host will never match an IP. +func (p *proxy_PerHost) AddIP(ip net.IP) { + p.bypassIPs = append(p.bypassIPs, ip) +} + +// AddNetwork specifies an IP range that will use the bypass proxy. Note that +// this will only take effect if a literal IP address is dialed. A connection +// to a named host will never match. +func (p *proxy_PerHost) AddNetwork(net *net.IPNet) { + p.bypassNetworks = append(p.bypassNetworks, net) +} + +// AddZone specifies a DNS suffix that will use the bypass proxy. A zone of +// "example.com" matches "example.com" and all of its subdomains. +func (p *proxy_PerHost) AddZone(zone string) { + if strings.HasSuffix(zone, ".") { + zone = zone[:len(zone)-1] + } + if !strings.HasPrefix(zone, ".") { + zone = "." + zone + } + p.bypassZones = append(p.bypassZones, zone) +} + +// AddHost specifies a host name that will use the bypass proxy. +func (p *proxy_PerHost) AddHost(host string) { + if strings.HasSuffix(host, ".") { + host = host[:len(host)-1] + } + p.bypassHosts = append(p.bypassHosts, host) +} + +// A Dialer is a means to establish a connection. +type proxy_Dialer interface { + // Dial connects to the given address via the proxy. + Dial(network, addr string) (c net.Conn, err error) +} + +// Auth contains authentication parameters that specific Dialers may require. +type proxy_Auth struct { + User, Password string +} + +// FromEnvironment returns the dialer specified by the proxy related variables in +// the environment. +func proxy_FromEnvironment() proxy_Dialer { + allProxy := proxy_allProxyEnv.Get() + if len(allProxy) == 0 { + return proxy_Direct + } + + proxyURL, err := url.Parse(allProxy) + if err != nil { + return proxy_Direct + } + proxy, err := proxy_FromURL(proxyURL, proxy_Direct) + if err != nil { + return proxy_Direct + } + + noProxy := proxy_noProxyEnv.Get() + if len(noProxy) == 0 { + return proxy + } + + perHost := proxy_NewPerHost(proxy, proxy_Direct) + perHost.AddFromString(noProxy) + return perHost +} + +// proxySchemes is a map from URL schemes to a function that creates a Dialer +// from a URL with such a scheme. +var proxy_proxySchemes map[string]func(*url.URL, proxy_Dialer) (proxy_Dialer, error) + +// RegisterDialerType takes a URL scheme and a function to generate Dialers from +// a URL with that scheme and a forwarding Dialer. Registered schemes are used +// by FromURL. +func proxy_RegisterDialerType(scheme string, f func(*url.URL, proxy_Dialer) (proxy_Dialer, error)) { + if proxy_proxySchemes == nil { + proxy_proxySchemes = make(map[string]func(*url.URL, proxy_Dialer) (proxy_Dialer, error)) + } + proxy_proxySchemes[scheme] = f +} + +// FromURL returns a Dialer given a URL specification and an underlying +// Dialer for it to make network requests. +func proxy_FromURL(u *url.URL, forward proxy_Dialer) (proxy_Dialer, error) { + var auth *proxy_Auth + if u.User != nil { + auth = new(proxy_Auth) + auth.User = u.User.Username() + if p, ok := u.User.Password(); ok { + auth.Password = p + } + } + + switch u.Scheme { + case "socks5": + return proxy_SOCKS5("tcp", u.Host, auth, forward) + } + + // If the scheme doesn't match any of the built-in schemes, see if it + // was registered by another package. + if proxy_proxySchemes != nil { + if f, ok := proxy_proxySchemes[u.Scheme]; ok { + return f(u, forward) + } + } + + return nil, errors.New("proxy: unknown scheme: " + u.Scheme) +} + +var ( + proxy_allProxyEnv = &proxy_envOnce{ + names: []string{"ALL_PROXY", "all_proxy"}, + } + proxy_noProxyEnv = &proxy_envOnce{ + names: []string{"NO_PROXY", "no_proxy"}, + } +) + +// envOnce looks up an environment variable (optionally by multiple +// names) once. It mitigates expensive lookups on some platforms +// (e.g. Windows). +// (Borrowed from net/http/transport.go) +type proxy_envOnce struct { + names []string + once sync.Once + val string +} + +func (e *proxy_envOnce) Get() string { + e.once.Do(e.init) + return e.val +} + +func (e *proxy_envOnce) init() { + for _, n := range e.names { + e.val = os.Getenv(n) + if e.val != "" { + return + } + } +} + +// SOCKS5 returns a Dialer that makes SOCKSv5 connections to the given address +// with an optional username and password. See RFC 1928 and RFC 1929. +func proxy_SOCKS5(network, addr string, auth *proxy_Auth, forward proxy_Dialer) (proxy_Dialer, error) { + s := &proxy_socks5{ + network: network, + addr: addr, + forward: forward, + } + if auth != nil { + s.user = auth.User + s.password = auth.Password + } + + return s, nil +} + +type proxy_socks5 struct { + user, password string + network, addr string + forward proxy_Dialer +} + +const proxy_socks5Version = 5 + +const ( + proxy_socks5AuthNone = 0 + proxy_socks5AuthPassword = 2 +) + +const proxy_socks5Connect = 1 + +const ( + proxy_socks5IP4 = 1 + proxy_socks5Domain = 3 + proxy_socks5IP6 = 4 +) + +var proxy_socks5Errors = []string{ + "", + "general failure", + "connection forbidden", + "network unreachable", + "host unreachable", + "connection refused", + "TTL expired", + "command not supported", + "address type not supported", +} + +// Dial connects to the address addr on the given network via the SOCKS5 proxy. +func (s *proxy_socks5) Dial(network, addr string) (net.Conn, error) { + switch network { + case "tcp", "tcp6", "tcp4": + default: + return nil, errors.New("proxy: no support for SOCKS5 proxy connections of type " + network) + } + + conn, err := s.forward.Dial(s.network, s.addr) + if err != nil { + return nil, err + } + if err := s.connect(conn, addr); err != nil { + conn.Close() + return nil, err + } + return conn, nil +} + +// connect takes an existing connection to a socks5 proxy server, +// and commands the server to extend that connection to target, +// which must be a canonical address with a host and port. +func (s *proxy_socks5) connect(conn net.Conn, target string) error { + host, portStr, err := net.SplitHostPort(target) + if err != nil { + return err + } + + port, err := strconv.Atoi(portStr) + if err != nil { + return errors.New("proxy: failed to parse port number: " + portStr) + } + if port < 1 || port > 0xffff { + return errors.New("proxy: port number out of range: " + portStr) + } + + // the size here is just an estimate + buf := make([]byte, 0, 6+len(host)) + + buf = append(buf, proxy_socks5Version) + if len(s.user) > 0 && len(s.user) < 256 && len(s.password) < 256 { + buf = append(buf, 2 /* num auth methods */, proxy_socks5AuthNone, proxy_socks5AuthPassword) + } else { + buf = append(buf, 1 /* num auth methods */, proxy_socks5AuthNone) + } + + if _, err := conn.Write(buf); err != nil { + return errors.New("proxy: failed to write greeting to SOCKS5 proxy at " + s.addr + ": " + err.Error()) + } + + if _, err := io.ReadFull(conn, buf[:2]); err != nil { + return errors.New("proxy: failed to read greeting from SOCKS5 proxy at " + s.addr + ": " + err.Error()) + } + if buf[0] != 5 { + return errors.New("proxy: SOCKS5 proxy at " + s.addr + " has unexpected version " + strconv.Itoa(int(buf[0]))) + } + if buf[1] == 0xff { + return errors.New("proxy: SOCKS5 proxy at " + s.addr + " requires authentication") + } + + // See RFC 1929 + if buf[1] == proxy_socks5AuthPassword { + buf = buf[:0] + buf = append(buf, 1 /* password protocol version */) + buf = append(buf, uint8(len(s.user))) + buf = append(buf, s.user...) + buf = append(buf, uint8(len(s.password))) + buf = append(buf, s.password...) + + if _, err := conn.Write(buf); err != nil { + return errors.New("proxy: failed to write authentication request to SOCKS5 proxy at " + s.addr + ": " + err.Error()) + } + + if _, err := io.ReadFull(conn, buf[:2]); err != nil { + return errors.New("proxy: failed to read authentication reply from SOCKS5 proxy at " + s.addr + ": " + err.Error()) + } + + if buf[1] != 0 { + return errors.New("proxy: SOCKS5 proxy at " + s.addr + " rejected username/password") + } + } + + buf = buf[:0] + buf = append(buf, proxy_socks5Version, proxy_socks5Connect, 0 /* reserved */) + + if ip := net.ParseIP(host); ip != nil { + if ip4 := ip.To4(); ip4 != nil { + buf = append(buf, proxy_socks5IP4) + ip = ip4 + } else { + buf = append(buf, proxy_socks5IP6) + } + buf = append(buf, ip...) + } else { + if len(host) > 255 { + return errors.New("proxy: destination host name too long: " + host) + } + buf = append(buf, proxy_socks5Domain) + buf = append(buf, byte(len(host))) + buf = append(buf, host...) + } + buf = append(buf, byte(port>>8), byte(port)) + + if _, err := conn.Write(buf); err != nil { + return errors.New("proxy: failed to write connect request to SOCKS5 proxy at " + s.addr + ": " + err.Error()) + } + + if _, err := io.ReadFull(conn, buf[:4]); err != nil { + return errors.New("proxy: failed to read connect reply from SOCKS5 proxy at " + s.addr + ": " + err.Error()) + } + + failure := "unknown error" + if int(buf[1]) < len(proxy_socks5Errors) { + failure = proxy_socks5Errors[buf[1]] + } + + if len(failure) > 0 { + return errors.New("proxy: SOCKS5 proxy at " + s.addr + " failed to connect: " + failure) + } + + bytesToDiscard := 0 + switch buf[3] { + case proxy_socks5IP4: + bytesToDiscard = net.IPv4len + case proxy_socks5IP6: + bytesToDiscard = net.IPv6len + case proxy_socks5Domain: + _, err := io.ReadFull(conn, buf[:1]) + if err != nil { + return errors.New("proxy: failed to read domain length from SOCKS5 proxy at " + s.addr + ": " + err.Error()) + } + bytesToDiscard = int(buf[0]) + default: + return errors.New("proxy: got unknown address type " + strconv.Itoa(int(buf[3])) + " from SOCKS5 proxy at " + s.addr) + } + + if cap(buf) < bytesToDiscard { + buf = make([]byte, bytesToDiscard) + } else { + buf = buf[:bytesToDiscard] + } + if _, err := io.ReadFull(conn, buf); err != nil { + return errors.New("proxy: failed to read address from SOCKS5 proxy at " + s.addr + ": " + err.Error()) + } + + // Also need to discard the port number + if _, err := io.ReadFull(conn, buf[:2]); err != nil { + return errors.New("proxy: failed to read port from SOCKS5 proxy at " + s.addr + ": " + err.Error()) + } + + return nil +} diff --git a/vendor/google.golang.org/appengine/datastore/datastore.go b/vendor/google.golang.org/appengine/datastore/datastore.go new file mode 100644 index 00000000000..576bc50132a --- /dev/null +++ b/vendor/google.golang.org/appengine/datastore/datastore.go @@ -0,0 +1,407 @@ +// Copyright 2011 Google Inc. All rights reserved. +// Use of this source code is governed by the Apache 2.0 +// license that can be found in the LICENSE file. + +package datastore + +import ( + "errors" + "fmt" + "reflect" + + "github.com/golang/protobuf/proto" + "golang.org/x/net/context" + + "google.golang.org/appengine" + "google.golang.org/appengine/internal" + pb "google.golang.org/appengine/internal/datastore" +) + +var ( + // ErrInvalidEntityType is returned when functions like Get or Next are + // passed a dst or src argument of invalid type. + ErrInvalidEntityType = errors.New("datastore: invalid entity type") + // ErrInvalidKey is returned when an invalid key is presented. + ErrInvalidKey = errors.New("datastore: invalid key") + // ErrNoSuchEntity is returned when no entity was found for a given key. + ErrNoSuchEntity = errors.New("datastore: no such entity") +) + +// ErrFieldMismatch is returned when a field is to be loaded into a different +// type than the one it was stored from, or when a field is missing or +// unexported in the destination struct. +// StructType is the type of the struct pointed to by the destination argument +// passed to Get or to Iterator.Next. +type ErrFieldMismatch struct { + StructType reflect.Type + FieldName string + Reason string +} + +func (e *ErrFieldMismatch) Error() string { + return fmt.Sprintf("datastore: cannot load field %q into a %q: %s", + e.FieldName, e.StructType, e.Reason) +} + +// protoToKey converts a Reference proto to a *Key. If the key is invalid, +// protoToKey will return the invalid key along with ErrInvalidKey. +func protoToKey(r *pb.Reference) (k *Key, err error) { + appID := r.GetApp() + namespace := r.GetNameSpace() + for _, e := range r.Path.Element { + k = &Key{ + kind: e.GetType(), + stringID: e.GetName(), + intID: e.GetId(), + parent: k, + appID: appID, + namespace: namespace, + } + if !k.valid() { + return k, ErrInvalidKey + } + } + return +} + +// keyToProto converts a *Key to a Reference proto. +func keyToProto(defaultAppID string, k *Key) *pb.Reference { + appID := k.appID + if appID == "" { + appID = defaultAppID + } + n := 0 + for i := k; i != nil; i = i.parent { + n++ + } + e := make([]*pb.Path_Element, n) + for i := k; i != nil; i = i.parent { + n-- + e[n] = &pb.Path_Element{ + Type: &i.kind, + } + // At most one of {Name,Id} should be set. + // Neither will be set for incomplete keys. + if i.stringID != "" { + e[n].Name = &i.stringID + } else if i.intID != 0 { + e[n].Id = &i.intID + } + } + var namespace *string + if k.namespace != "" { + namespace = proto.String(k.namespace) + } + return &pb.Reference{ + App: proto.String(appID), + NameSpace: namespace, + Path: &pb.Path{ + Element: e, + }, + } +} + +// multiKeyToProto is a batch version of keyToProto. +func multiKeyToProto(appID string, key []*Key) []*pb.Reference { + ret := make([]*pb.Reference, len(key)) + for i, k := range key { + ret[i] = keyToProto(appID, k) + } + return ret +} + +// multiValid is a batch version of Key.valid. It returns an error, not a +// []bool. +func multiValid(key []*Key) error { + invalid := false + for _, k := range key { + if !k.valid() { + invalid = true + break + } + } + if !invalid { + return nil + } + err := make(appengine.MultiError, len(key)) + for i, k := range key { + if !k.valid() { + err[i] = ErrInvalidKey + } + } + return err +} + +// It's unfortunate that the two semantically equivalent concepts pb.Reference +// and pb.PropertyValue_ReferenceValue aren't the same type. For example, the +// two have different protobuf field numbers. + +// referenceValueToKey is the same as protoToKey except the input is a +// PropertyValue_ReferenceValue instead of a Reference. +func referenceValueToKey(r *pb.PropertyValue_ReferenceValue) (k *Key, err error) { + appID := r.GetApp() + namespace := r.GetNameSpace() + for _, e := range r.Pathelement { + k = &Key{ + kind: e.GetType(), + stringID: e.GetName(), + intID: e.GetId(), + parent: k, + appID: appID, + namespace: namespace, + } + if !k.valid() { + return nil, ErrInvalidKey + } + } + return +} + +// keyToReferenceValue is the same as keyToProto except the output is a +// PropertyValue_ReferenceValue instead of a Reference. +func keyToReferenceValue(defaultAppID string, k *Key) *pb.PropertyValue_ReferenceValue { + ref := keyToProto(defaultAppID, k) + pe := make([]*pb.PropertyValue_ReferenceValue_PathElement, len(ref.Path.Element)) + for i, e := range ref.Path.Element { + pe[i] = &pb.PropertyValue_ReferenceValue_PathElement{ + Type: e.Type, + Id: e.Id, + Name: e.Name, + } + } + return &pb.PropertyValue_ReferenceValue{ + App: ref.App, + NameSpace: ref.NameSpace, + Pathelement: pe, + } +} + +type multiArgType int + +const ( + multiArgTypeInvalid multiArgType = iota + multiArgTypePropertyLoadSaver + multiArgTypeStruct + multiArgTypeStructPtr + multiArgTypeInterface +) + +// checkMultiArg checks that v has type []S, []*S, []I, or []P, for some struct +// type S, for some interface type I, or some non-interface non-pointer type P +// such that P or *P implements PropertyLoadSaver. +// +// It returns what category the slice's elements are, and the reflect.Type +// that represents S, I or P. +// +// As a special case, PropertyList is an invalid type for v. +func checkMultiArg(v reflect.Value) (m multiArgType, elemType reflect.Type) { + if v.Kind() != reflect.Slice { + return multiArgTypeInvalid, nil + } + if v.Type() == typeOfPropertyList { + return multiArgTypeInvalid, nil + } + elemType = v.Type().Elem() + if reflect.PtrTo(elemType).Implements(typeOfPropertyLoadSaver) { + return multiArgTypePropertyLoadSaver, elemType + } + switch elemType.Kind() { + case reflect.Struct: + return multiArgTypeStruct, elemType + case reflect.Interface: + return multiArgTypeInterface, elemType + case reflect.Ptr: + elemType = elemType.Elem() + if elemType.Kind() == reflect.Struct { + return multiArgTypeStructPtr, elemType + } + } + return multiArgTypeInvalid, nil +} + +// Get loads the entity stored for k into dst, which must be a struct pointer +// or implement PropertyLoadSaver. If there is no such entity for the key, Get +// returns ErrNoSuchEntity. +// +// The values of dst's unmatched struct fields are not modified, and matching +// slice-typed fields are not reset before appending to them. In particular, it +// is recommended to pass a pointer to a zero valued struct on each Get call. +// +// ErrFieldMismatch is returned when a field is to be loaded into a different +// type than the one it was stored from, or when a field is missing or +// unexported in the destination struct. ErrFieldMismatch is only returned if +// dst is a struct pointer. +func Get(c context.Context, key *Key, dst interface{}) error { + if dst == nil { // GetMulti catches nil interface; we need to catch nil ptr here + return ErrInvalidEntityType + } + err := GetMulti(c, []*Key{key}, []interface{}{dst}) + if me, ok := err.(appengine.MultiError); ok { + return me[0] + } + return err +} + +// GetMulti is a batch version of Get. +// +// dst must be a []S, []*S, []I or []P, for some struct type S, some interface +// type I, or some non-interface non-pointer type P such that P or *P +// implements PropertyLoadSaver. If an []I, each element must be a valid dst +// for Get: it must be a struct pointer or implement PropertyLoadSaver. +// +// As a special case, PropertyList is an invalid type for dst, even though a +// PropertyList is a slice of structs. It is treated as invalid to avoid being +// mistakenly passed when []PropertyList was intended. +func GetMulti(c context.Context, key []*Key, dst interface{}) error { + v := reflect.ValueOf(dst) + multiArgType, _ := checkMultiArg(v) + if multiArgType == multiArgTypeInvalid { + return errors.New("datastore: dst has invalid type") + } + if len(key) != v.Len() { + return errors.New("datastore: key and dst slices have different length") + } + if len(key) == 0 { + return nil + } + if err := multiValid(key); err != nil { + return err + } + req := &pb.GetRequest{ + Key: multiKeyToProto(internal.FullyQualifiedAppID(c), key), + } + res := &pb.GetResponse{} + if err := internal.Call(c, "datastore_v3", "Get", req, res); err != nil { + return err + } + if len(key) != len(res.Entity) { + return errors.New("datastore: internal error: server returned the wrong number of entities") + } + multiErr, any := make(appengine.MultiError, len(key)), false + for i, e := range res.Entity { + if e.Entity == nil { + multiErr[i] = ErrNoSuchEntity + } else { + elem := v.Index(i) + if multiArgType == multiArgTypePropertyLoadSaver || multiArgType == multiArgTypeStruct { + elem = elem.Addr() + } + if multiArgType == multiArgTypeStructPtr && elem.IsNil() { + elem.Set(reflect.New(elem.Type().Elem())) + } + multiErr[i] = loadEntity(elem.Interface(), e.Entity) + } + if multiErr[i] != nil { + any = true + } + } + if any { + return multiErr + } + return nil +} + +// Put saves the entity src into the datastore with key k. src must be a struct +// pointer or implement PropertyLoadSaver; if a struct pointer then any +// unexported fields of that struct will be skipped. If k is an incomplete key, +// the returned key will be a unique key generated by the datastore. +func Put(c context.Context, key *Key, src interface{}) (*Key, error) { + k, err := PutMulti(c, []*Key{key}, []interface{}{src}) + if err != nil { + if me, ok := err.(appengine.MultiError); ok { + return nil, me[0] + } + return nil, err + } + return k[0], nil +} + +// PutMulti is a batch version of Put. +// +// src must satisfy the same conditions as the dst argument to GetMulti. +func PutMulti(c context.Context, key []*Key, src interface{}) ([]*Key, error) { + v := reflect.ValueOf(src) + multiArgType, _ := checkMultiArg(v) + if multiArgType == multiArgTypeInvalid { + return nil, errors.New("datastore: src has invalid type") + } + if len(key) != v.Len() { + return nil, errors.New("datastore: key and src slices have different length") + } + if len(key) == 0 { + return nil, nil + } + appID := internal.FullyQualifiedAppID(c) + if err := multiValid(key); err != nil { + return nil, err + } + req := &pb.PutRequest{} + for i := range key { + elem := v.Index(i) + if multiArgType == multiArgTypePropertyLoadSaver || multiArgType == multiArgTypeStruct { + elem = elem.Addr() + } + sProto, err := saveEntity(appID, key[i], elem.Interface()) + if err != nil { + return nil, err + } + req.Entity = append(req.Entity, sProto) + } + res := &pb.PutResponse{} + if err := internal.Call(c, "datastore_v3", "Put", req, res); err != nil { + return nil, err + } + if len(key) != len(res.Key) { + return nil, errors.New("datastore: internal error: server returned the wrong number of keys") + } + ret := make([]*Key, len(key)) + for i := range ret { + var err error + ret[i], err = protoToKey(res.Key[i]) + if err != nil || ret[i].Incomplete() { + return nil, errors.New("datastore: internal error: server returned an invalid key") + } + } + return ret, nil +} + +// Delete deletes the entity for the given key. +func Delete(c context.Context, key *Key) error { + err := DeleteMulti(c, []*Key{key}) + if me, ok := err.(appengine.MultiError); ok { + return me[0] + } + return err +} + +// DeleteMulti is a batch version of Delete. +func DeleteMulti(c context.Context, key []*Key) error { + if len(key) == 0 { + return nil + } + if err := multiValid(key); err != nil { + return err + } + req := &pb.DeleteRequest{ + Key: multiKeyToProto(internal.FullyQualifiedAppID(c), key), + } + res := &pb.DeleteResponse{} + return internal.Call(c, "datastore_v3", "Delete", req, res) +} + +func namespaceMod(m proto.Message, namespace string) { + // pb.Query is the only type that has a name_space field. + // All other namespace support in datastore is in the keys. + switch m := m.(type) { + case *pb.Query: + if m.NameSpace == nil { + m.NameSpace = &namespace + } + } +} + +func init() { + internal.NamespaceMods["datastore_v3"] = namespaceMod + internal.RegisterErrorCodeMap("datastore_v3", pb.Error_ErrorCode_name) + internal.RegisterTimeoutErrorCode("datastore_v3", int32(pb.Error_TIMEOUT)) +} diff --git a/vendor/google.golang.org/appengine/datastore/doc.go b/vendor/google.golang.org/appengine/datastore/doc.go new file mode 100644 index 00000000000..85616cf2741 --- /dev/null +++ b/vendor/google.golang.org/appengine/datastore/doc.go @@ -0,0 +1,361 @@ +// Copyright 2011 Google Inc. All rights reserved. +// Use of this source code is governed by the Apache 2.0 +// license that can be found in the LICENSE file. + +/* +Package datastore provides a client for App Engine's datastore service. + + +Basic Operations + +Entities are the unit of storage and are associated with a key. A key +consists of an optional parent key, a string application ID, a string kind +(also known as an entity type), and either a StringID or an IntID. A +StringID is also known as an entity name or key name. + +It is valid to create a key with a zero StringID and a zero IntID; this is +called an incomplete key, and does not refer to any saved entity. Putting an +entity into the datastore under an incomplete key will cause a unique key +to be generated for that entity, with a non-zero IntID. + +An entity's contents are a mapping from case-sensitive field names to values. +Valid value types are: + - signed integers (int, int8, int16, int32 and int64), + - bool, + - string, + - float32 and float64, + - []byte (up to 1 megabyte in length), + - any type whose underlying type is one of the above predeclared types, + - ByteString, + - *Key, + - time.Time (stored with microsecond precision), + - appengine.BlobKey, + - appengine.GeoPoint, + - structs whose fields are all valid value types, + - slices of any of the above. + +Slices of structs are valid, as are structs that contain slices. However, if +one struct contains another, then at most one of those can be repeated. This +disqualifies recursively defined struct types: any struct T that (directly or +indirectly) contains a []T. + +The Get and Put functions load and save an entity's contents. An entity's +contents are typically represented by a struct pointer. + +Example code: + + type Entity struct { + Value string + } + + func handle(w http.ResponseWriter, r *http.Request) { + ctx := appengine.NewContext(r) + + k := datastore.NewKey(ctx, "Entity", "stringID", 0, nil) + e := new(Entity) + if err := datastore.Get(ctx, k, e); err != nil { + http.Error(w, err.Error(), 500) + return + } + + old := e.Value + e.Value = r.URL.Path + + if _, err := datastore.Put(ctx, k, e); err != nil { + http.Error(w, err.Error(), 500) + return + } + + w.Header().Set("Content-Type", "text/plain; charset=utf-8") + fmt.Fprintf(w, "old=%q\nnew=%q\n", old, e.Value) + } + +GetMulti, PutMulti and DeleteMulti are batch versions of the Get, Put and +Delete functions. They take a []*Key instead of a *Key, and may return an +appengine.MultiError when encountering partial failure. + + +Properties + +An entity's contents can be represented by a variety of types. These are +typically struct pointers, but can also be any type that implements the +PropertyLoadSaver interface. If using a struct pointer, you do not have to +explicitly implement the PropertyLoadSaver interface; the datastore will +automatically convert via reflection. If a struct pointer does implement that +interface then those methods will be used in preference to the default +behavior for struct pointers. Struct pointers are more strongly typed and are +easier to use; PropertyLoadSavers are more flexible. + +The actual types passed do not have to match between Get and Put calls or even +across different calls to datastore. It is valid to put a *PropertyList and +get that same entity as a *myStruct, or put a *myStruct0 and get a *myStruct1. +Conceptually, any entity is saved as a sequence of properties, and is loaded +into the destination value on a property-by-property basis. When loading into +a struct pointer, an entity that cannot be completely represented (such as a +missing field) will result in an ErrFieldMismatch error but it is up to the +caller whether this error is fatal, recoverable or ignorable. + +By default, for struct pointers, all properties are potentially indexed, and +the property name is the same as the field name (and hence must start with an +upper case letter). + +Fields may have a `datastore:"name,options"` tag. The tag name is the +property name, which must be one or more valid Go identifiers joined by ".", +but may start with a lower case letter. An empty tag name means to just use the +field name. A "-" tag name means that the datastore will ignore that field. + +The only valid options are "omitempty" and "noindex". + +If the options include "omitempty" and the value of the field is empty, then the field will be omitted on Save. +The empty values are false, 0, any nil interface value, and any array, slice, map, or string of length zero. +Struct field values will never be empty. + +If options include "noindex" then the field will not be indexed. All fields are indexed +by default. Strings or byte slices longer than 1500 bytes cannot be indexed; +fields used to store long strings and byte slices must be tagged with "noindex" +or they will cause Put operations to fail. + +To use multiple options together, separate them by a comma. +The order does not matter. + +If the options is "" then the comma may be omitted. + +Example code: + + // A and B are renamed to a and b. + // A, C and J are not indexed. + // D's tag is equivalent to having no tag at all (E). + // I is ignored entirely by the datastore. + // J has tag information for both the datastore and json packages. + type TaggedStruct struct { + A int `datastore:"a,noindex"` + B int `datastore:"b"` + C int `datastore:",noindex"` + D int `datastore:""` + E int + I int `datastore:"-"` + J int `datastore:",noindex" json:"j"` + } + + +Structured Properties + +If the struct pointed to contains other structs, then the nested or embedded +structs are flattened. For example, given these definitions: + + type Inner1 struct { + W int32 + X string + } + + type Inner2 struct { + Y float64 + } + + type Inner3 struct { + Z bool + } + + type Outer struct { + A int16 + I []Inner1 + J Inner2 + Inner3 + } + +then an Outer's properties would be equivalent to those of: + + type OuterEquivalent struct { + A int16 + IDotW []int32 `datastore:"I.W"` + IDotX []string `datastore:"I.X"` + JDotY float64 `datastore:"J.Y"` + Z bool + } + +If Outer's embedded Inner3 field was tagged as `datastore:"Foo"` then the +equivalent field would instead be: FooDotZ bool `datastore:"Foo.Z"`. + +If an outer struct is tagged "noindex" then all of its implicit flattened +fields are effectively "noindex". + + +The PropertyLoadSaver Interface + +An entity's contents can also be represented by any type that implements the +PropertyLoadSaver interface. This type may be a struct pointer, but it does +not have to be. The datastore package will call Load when getting the entity's +contents, and Save when putting the entity's contents. +Possible uses include deriving non-stored fields, verifying fields, or indexing +a field only if its value is positive. + +Example code: + + type CustomPropsExample struct { + I, J int + // Sum is not stored, but should always be equal to I + J. + Sum int `datastore:"-"` + } + + func (x *CustomPropsExample) Load(ps []datastore.Property) error { + // Load I and J as usual. + if err := datastore.LoadStruct(x, ps); err != nil { + return err + } + // Derive the Sum field. + x.Sum = x.I + x.J + return nil + } + + func (x *CustomPropsExample) Save() ([]datastore.Property, error) { + // Validate the Sum field. + if x.Sum != x.I + x.J { + return nil, errors.New("CustomPropsExample has inconsistent sum") + } + // Save I and J as usual. The code below is equivalent to calling + // "return datastore.SaveStruct(x)", but is done manually for + // demonstration purposes. + return []datastore.Property{ + { + Name: "I", + Value: int64(x.I), + }, + { + Name: "J", + Value: int64(x.J), + }, + }, nil + } + +The *PropertyList type implements PropertyLoadSaver, and can therefore hold an +arbitrary entity's contents. + + +Queries + +Queries retrieve entities based on their properties or key's ancestry. Running +a query yields an iterator of results: either keys or (key, entity) pairs. +Queries are re-usable and it is safe to call Query.Run from concurrent +goroutines. Iterators are not safe for concurrent use. + +Queries are immutable, and are either created by calling NewQuery, or derived +from an existing query by calling a method like Filter or Order that returns a +new query value. A query is typically constructed by calling NewQuery followed +by a chain of zero or more such methods. These methods are: + - Ancestor and Filter constrain the entities returned by running a query. + - Order affects the order in which they are returned. + - Project constrains the fields returned. + - Distinct de-duplicates projected entities. + - KeysOnly makes the iterator return only keys, not (key, entity) pairs. + - Start, End, Offset and Limit define which sub-sequence of matching entities + to return. Start and End take cursors, Offset and Limit take integers. Start + and Offset affect the first result, End and Limit affect the last result. + If both Start and Offset are set, then the offset is relative to Start. + If both End and Limit are set, then the earliest constraint wins. Limit is + relative to Start+Offset, not relative to End. As a special case, a + negative limit means unlimited. + +Example code: + + type Widget struct { + Description string + Price int + } + + func handle(w http.ResponseWriter, r *http.Request) { + ctx := appengine.NewContext(r) + q := datastore.NewQuery("Widget"). + Filter("Price <", 1000). + Order("-Price") + b := new(bytes.Buffer) + for t := q.Run(ctx); ; { + var x Widget + key, err := t.Next(&x) + if err == datastore.Done { + break + } + if err != nil { + serveError(ctx, w, err) + return + } + fmt.Fprintf(b, "Key=%v\nWidget=%#v\n\n", key, x) + } + w.Header().Set("Content-Type", "text/plain; charset=utf-8") + io.Copy(w, b) + } + + +Transactions + +RunInTransaction runs a function in a transaction. + +Example code: + + type Counter struct { + Count int + } + + func inc(ctx context.Context, key *datastore.Key) (int, error) { + var x Counter + if err := datastore.Get(ctx, key, &x); err != nil && err != datastore.ErrNoSuchEntity { + return 0, err + } + x.Count++ + if _, err := datastore.Put(ctx, key, &x); err != nil { + return 0, err + } + return x.Count, nil + } + + func handle(w http.ResponseWriter, r *http.Request) { + ctx := appengine.NewContext(r) + var count int + err := datastore.RunInTransaction(ctx, func(ctx context.Context) error { + var err1 error + count, err1 = inc(ctx, datastore.NewKey(ctx, "Counter", "singleton", 0, nil)) + return err1 + }, nil) + if err != nil { + serveError(ctx, w, err) + return + } + w.Header().Set("Content-Type", "text/plain; charset=utf-8") + fmt.Fprintf(w, "Count=%d", count) + } + + +Metadata + +The datastore package provides access to some of App Engine's datastore +metadata. This metadata includes information about the entity groups, +namespaces, entity kinds, and properties in the datastore, as well as the +property representations for each property. + +Example code: + + func handle(w http.ResponseWriter, r *http.Request) { + // Print all the kinds in the datastore, with all the indexed + // properties (and their representations) for each. + ctx := appengine.NewContext(r) + + kinds, err := datastore.Kinds(ctx) + if err != nil { + serveError(ctx, w, err) + return + } + + w.Header().Set("Content-Type", "text/plain; charset=utf-8") + for _, kind := range kinds { + fmt.Fprintf(w, "%s:\n", kind) + props, err := datastore.KindProperties(ctx, kind) + if err != nil { + fmt.Fprintln(w, "\t(unable to retrieve properties)") + continue + } + for p, rep := range props { + fmt.Fprintf(w, "\t-%s (%s)\n", p, strings.Join(rep, ", ")) + } + } + } +*/ +package datastore // import "google.golang.org/appengine/datastore" diff --git a/vendor/google.golang.org/appengine/datastore/internal/cloudkey/cloudkey.go b/vendor/google.golang.org/appengine/datastore/internal/cloudkey/cloudkey.go new file mode 100644 index 00000000000..643d4049c6b --- /dev/null +++ b/vendor/google.golang.org/appengine/datastore/internal/cloudkey/cloudkey.go @@ -0,0 +1,120 @@ +// Copyright 2019 Google Inc. All rights reserved. +// Use of this source code is governed by the Apache 2.0 +// license that can be found in the LICENSE file. + +// Package cloudpb is a subset of types and functions, copied from cloud.google.com/go/datastore. +// +// They are copied here to provide compatibility to decode keys generated by the cloud.google.com/go/datastore package. +package cloudkey + +import ( + "encoding/base64" + "errors" + "strings" + + "github.com/golang/protobuf/proto" + cloudpb "google.golang.org/appengine/datastore/internal/cloudpb" +) + +///////////////////////////////////////////////////////////////////// +// Code below is copied from https://github.com/googleapis/google-cloud-go/blob/master/datastore/datastore.go +///////////////////////////////////////////////////////////////////// + +var ( + // ErrInvalidKey is returned when an invalid key is presented. + ErrInvalidKey = errors.New("datastore: invalid key") +) + +///////////////////////////////////////////////////////////////////// +// Code below is copied from https://github.com/googleapis/google-cloud-go/blob/master/datastore/key.go +///////////////////////////////////////////////////////////////////// + +// Key represents the datastore key for a stored entity. +type Key struct { + // Kind cannot be empty. + Kind string + // Either ID or Name must be zero for the Key to be valid. + // If both are zero, the Key is incomplete. + ID int64 + Name string + // Parent must either be a complete Key or nil. + Parent *Key + + // Namespace provides the ability to partition your data for multiple + // tenants. In most cases, it is not necessary to specify a namespace. + // See docs on datastore multitenancy for details: + // https://cloud.google.com/datastore/docs/concepts/multitenancy + Namespace string +} + +// DecodeKey decodes a key from the opaque representation returned by Encode. +func DecodeKey(encoded string) (*Key, error) { + // Re-add padding. + if m := len(encoded) % 4; m != 0 { + encoded += strings.Repeat("=", 4-m) + } + + b, err := base64.URLEncoding.DecodeString(encoded) + if err != nil { + return nil, err + } + + pKey := new(cloudpb.Key) + if err := proto.Unmarshal(b, pKey); err != nil { + return nil, err + } + return protoToKey(pKey) +} + +// valid returns whether the key is valid. +func (k *Key) valid() bool { + if k == nil { + return false + } + for ; k != nil; k = k.Parent { + if k.Kind == "" { + return false + } + if k.Name != "" && k.ID != 0 { + return false + } + if k.Parent != nil { + if k.Parent.Incomplete() { + return false + } + if k.Parent.Namespace != k.Namespace { + return false + } + } + } + return true +} + +// Incomplete reports whether the key does not refer to a stored entity. +func (k *Key) Incomplete() bool { + return k.Name == "" && k.ID == 0 +} + +// protoToKey decodes a protocol buffer representation of a key into an +// equivalent *Key object. If the key is invalid, protoToKey will return the +// invalid key along with ErrInvalidKey. +func protoToKey(p *cloudpb.Key) (*Key, error) { + var key *Key + var namespace string + if partition := p.PartitionId; partition != nil { + namespace = partition.NamespaceId + } + for _, el := range p.Path { + key = &Key{ + Namespace: namespace, + Kind: el.Kind, + ID: el.GetId(), + Name: el.GetName(), + Parent: key, + } + } + if !key.valid() { // Also detects key == nil. + return key, ErrInvalidKey + } + return key, nil +} diff --git a/vendor/google.golang.org/appengine/datastore/internal/cloudpb/entity.pb.go b/vendor/google.golang.org/appengine/datastore/internal/cloudpb/entity.pb.go new file mode 100644 index 00000000000..af8195f3f8d --- /dev/null +++ b/vendor/google.golang.org/appengine/datastore/internal/cloudpb/entity.pb.go @@ -0,0 +1,344 @@ +// Copyright 2019 Google Inc. All rights reserved. +// Use of this source code is governed by the Apache 2.0 +// license that can be found in the LICENSE file. + +// Package cloudpb is a subset of protobufs, copied from google.golang.org/genproto/googleapis/datastore/v1. +// +// They are copied here to provide compatibility to decode keys generated by the cloud.google.com/go/datastore package. +package cloudpb + +import ( + "fmt" + + "github.com/golang/protobuf/proto" +) + +// A partition ID identifies a grouping of entities. The grouping is always +// by project and namespace, however the namespace ID may be empty. +// +// A partition ID contains several dimensions: +// project ID and namespace ID. +// +// Partition dimensions: +// +// - May be `""`. +// - Must be valid UTF-8 bytes. +// - Must have values that match regex `[A-Za-z\d\.\-_]{1,100}` +// If the value of any dimension matches regex `__.*__`, the partition is +// reserved/read-only. +// A reserved/read-only partition ID is forbidden in certain documented +// contexts. +// +// Foreign partition IDs (in which the project ID does +// not match the context project ID ) are discouraged. +// Reads and writes of foreign partition IDs may fail if the project is not in +// an active state. +type PartitionId struct { + // The ID of the project to which the entities belong. + ProjectId string `protobuf:"bytes,2,opt,name=project_id,json=projectId,proto3" json:"project_id,omitempty"` + // If not empty, the ID of the namespace to which the entities belong. + NamespaceId string `protobuf:"bytes,4,opt,name=namespace_id,json=namespaceId,proto3" json:"namespace_id,omitempty"` + XXX_NoUnkeyedLiteral struct{} `json:"-"` + XXX_unrecognized []byte `json:"-"` + XXX_sizecache int32 `json:"-"` +} + +func (m *PartitionId) Reset() { *m = PartitionId{} } +func (m *PartitionId) String() string { return proto.CompactTextString(m) } +func (*PartitionId) ProtoMessage() {} +func (*PartitionId) Descriptor() ([]byte, []int) { + return fileDescriptor_entity_096a297364b049a5, []int{0} +} +func (m *PartitionId) XXX_Unmarshal(b []byte) error { + return xxx_messageInfo_PartitionId.Unmarshal(m, b) +} +func (m *PartitionId) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) { + return xxx_messageInfo_PartitionId.Marshal(b, m, deterministic) +} +func (dst *PartitionId) XXX_Merge(src proto.Message) { + xxx_messageInfo_PartitionId.Merge(dst, src) +} +func (m *PartitionId) XXX_Size() int { + return xxx_messageInfo_PartitionId.Size(m) +} +func (m *PartitionId) XXX_DiscardUnknown() { + xxx_messageInfo_PartitionId.DiscardUnknown(m) +} + +var xxx_messageInfo_PartitionId proto.InternalMessageInfo + +func (m *PartitionId) GetProjectId() string { + if m != nil { + return m.ProjectId + } + return "" +} + +func (m *PartitionId) GetNamespaceId() string { + if m != nil { + return m.NamespaceId + } + return "" +} + +// A unique identifier for an entity. +// If a key's partition ID or any of its path kinds or names are +// reserved/read-only, the key is reserved/read-only. +// A reserved/read-only key is forbidden in certain documented contexts. +type Key struct { + // Entities are partitioned into subsets, currently identified by a project + // ID and namespace ID. + // Queries are scoped to a single partition. + PartitionId *PartitionId `protobuf:"bytes,1,opt,name=partition_id,json=partitionId,proto3" json:"partition_id,omitempty"` + // The entity path. + // An entity path consists of one or more elements composed of a kind and a + // string or numerical identifier, which identify entities. The first + // element identifies a _root entity_, the second element identifies + // a _child_ of the root entity, the third element identifies a child of the + // second entity, and so forth. The entities identified by all prefixes of + // the path are called the element's _ancestors_. + // + // An entity path is always fully complete: *all* of the entity's ancestors + // are required to be in the path along with the entity identifier itself. + // The only exception is that in some documented cases, the identifier in the + // last path element (for the entity) itself may be omitted. For example, + // the last path element of the key of `Mutation.insert` may have no + // identifier. + // + // A path can never be empty, and a path can have at most 100 elements. + Path []*Key_PathElement `protobuf:"bytes,2,rep,name=path,proto3" json:"path,omitempty"` + XXX_NoUnkeyedLiteral struct{} `json:"-"` + XXX_unrecognized []byte `json:"-"` + XXX_sizecache int32 `json:"-"` +} + +func (m *Key) Reset() { *m = Key{} } +func (m *Key) String() string { return proto.CompactTextString(m) } +func (*Key) ProtoMessage() {} +func (*Key) Descriptor() ([]byte, []int) { + return fileDescriptor_entity_096a297364b049a5, []int{1} +} +func (m *Key) XXX_Unmarshal(b []byte) error { + return xxx_messageInfo_Key.Unmarshal(m, b) +} +func (m *Key) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) { + return xxx_messageInfo_Key.Marshal(b, m, deterministic) +} +func (dst *Key) XXX_Merge(src proto.Message) { + xxx_messageInfo_Key.Merge(dst, src) +} +func (m *Key) XXX_Size() int { + return xxx_messageInfo_Key.Size(m) +} +func (m *Key) XXX_DiscardUnknown() { + xxx_messageInfo_Key.DiscardUnknown(m) +} + +// A (kind, ID/name) pair used to construct a key path. +// +// If either name or ID is set, the element is complete. +// If neither is set, the element is incomplete. +type Key_PathElement struct { + // The kind of the entity. + // A kind matching regex `__.*__` is reserved/read-only. + // A kind must not contain more than 1500 bytes when UTF-8 encoded. + // Cannot be `""`. + Kind string `protobuf:"bytes,1,opt,name=kind,proto3" json:"kind,omitempty"` + // The type of ID. + // + // Types that are valid to be assigned to IdType: + // *Key_PathElement_Id + // *Key_PathElement_Name + IdType isKey_PathElement_IdType `protobuf_oneof:"id_type"` + XXX_NoUnkeyedLiteral struct{} `json:"-"` + XXX_unrecognized []byte `json:"-"` + XXX_sizecache int32 `json:"-"` +} + +func (m *Key_PathElement) Reset() { *m = Key_PathElement{} } +func (m *Key_PathElement) String() string { return proto.CompactTextString(m) } +func (*Key_PathElement) ProtoMessage() {} +func (*Key_PathElement) Descriptor() ([]byte, []int) { + return fileDescriptor_entity_096a297364b049a5, []int{1, 0} +} +func (m *Key_PathElement) XXX_Unmarshal(b []byte) error { + return xxx_messageInfo_Key_PathElement.Unmarshal(m, b) +} +func (m *Key_PathElement) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) { + return xxx_messageInfo_Key_PathElement.Marshal(b, m, deterministic) +} +func (dst *Key_PathElement) XXX_Merge(src proto.Message) { + xxx_messageInfo_Key_PathElement.Merge(dst, src) +} +func (m *Key_PathElement) XXX_Size() int { + return xxx_messageInfo_Key_PathElement.Size(m) +} +func (m *Key_PathElement) XXX_DiscardUnknown() { + xxx_messageInfo_Key_PathElement.DiscardUnknown(m) +} + +var xxx_messageInfo_Key_PathElement proto.InternalMessageInfo + +func (m *Key_PathElement) GetKind() string { + if m != nil { + return m.Kind + } + return "" +} + +type isKey_PathElement_IdType interface { + isKey_PathElement_IdType() +} + +type Key_PathElement_Id struct { + Id int64 `protobuf:"varint,2,opt,name=id,proto3,oneof"` +} + +type Key_PathElement_Name struct { + Name string `protobuf:"bytes,3,opt,name=name,proto3,oneof"` +} + +func (*Key_PathElement_Id) isKey_PathElement_IdType() {} + +func (*Key_PathElement_Name) isKey_PathElement_IdType() {} + +func (m *Key_PathElement) GetIdType() isKey_PathElement_IdType { + if m != nil { + return m.IdType + } + return nil +} + +func (m *Key_PathElement) GetId() int64 { + if x, ok := m.GetIdType().(*Key_PathElement_Id); ok { + return x.Id + } + return 0 +} + +func (m *Key_PathElement) GetName() string { + if x, ok := m.GetIdType().(*Key_PathElement_Name); ok { + return x.Name + } + return "" +} + +// XXX_OneofFuncs is for the internal use of the proto package. +func (*Key_PathElement) XXX_OneofFuncs() (func(msg proto.Message, b *proto.Buffer) error, func(msg proto.Message, tag, wire int, b *proto.Buffer) (bool, error), func(msg proto.Message) (n int), []interface{}) { + return _Key_PathElement_OneofMarshaler, _Key_PathElement_OneofUnmarshaler, _Key_PathElement_OneofSizer, []interface{}{ + (*Key_PathElement_Id)(nil), + (*Key_PathElement_Name)(nil), + } +} + +func _Key_PathElement_OneofMarshaler(msg proto.Message, b *proto.Buffer) error { + m := msg.(*Key_PathElement) + // id_type + switch x := m.IdType.(type) { + case *Key_PathElement_Id: + b.EncodeVarint(2<<3 | proto.WireVarint) + b.EncodeVarint(uint64(x.Id)) + case *Key_PathElement_Name: + b.EncodeVarint(3<<3 | proto.WireBytes) + b.EncodeStringBytes(x.Name) + case nil: + default: + return fmt.Errorf("Key_PathElement.IdType has unexpected type %T", x) + } + return nil +} + +func _Key_PathElement_OneofUnmarshaler(msg proto.Message, tag, wire int, b *proto.Buffer) (bool, error) { + m := msg.(*Key_PathElement) + switch tag { + case 2: // id_type.id + if wire != proto.WireVarint { + return true, proto.ErrInternalBadWireType + } + x, err := b.DecodeVarint() + m.IdType = &Key_PathElement_Id{int64(x)} + return true, err + case 3: // id_type.name + if wire != proto.WireBytes { + return true, proto.ErrInternalBadWireType + } + x, err := b.DecodeStringBytes() + m.IdType = &Key_PathElement_Name{x} + return true, err + default: + return false, nil + } +} + +func _Key_PathElement_OneofSizer(msg proto.Message) (n int) { + m := msg.(*Key_PathElement) + // id_type + switch x := m.IdType.(type) { + case *Key_PathElement_Id: + n += 1 // tag and wire + n += proto.SizeVarint(uint64(x.Id)) + case *Key_PathElement_Name: + n += 1 // tag and wire + n += proto.SizeVarint(uint64(len(x.Name))) + n += len(x.Name) + case nil: + default: + panic(fmt.Sprintf("proto: unexpected type %T in oneof", x)) + } + return n +} + +var fileDescriptor_entity_096a297364b049a5 = []byte{ + // 780 bytes of a gzipped FileDescriptorProto + 0x1f, 0x8b, 0x08, 0x00, 0x00, 0x00, 0x00, 0x00, 0x02, 0xff, 0x7c, 0x94, 0xff, 0x6e, 0xdc, 0x44, + 0x10, 0xc7, 0xed, 0xbb, 0x5c, 0x1a, 0x8f, 0xdd, 0xa4, 0x6c, 0x2a, 0x61, 0x02, 0x28, 0x26, 0x80, + 0x74, 0x02, 0xc9, 0x6e, 0xc2, 0x1f, 0x54, 0x14, 0xa4, 0x72, 0x25, 0xe0, 0x28, 0x15, 0x9c, 0x56, + 0x55, 0x24, 0x50, 0xa4, 0xd3, 0xde, 0x79, 0xeb, 0x2e, 0x67, 0xef, 0x5a, 0xf6, 0x3a, 0xaa, 0xdf, + 0x05, 0xf1, 0x00, 0x3c, 0x0a, 0x8f, 0x80, 0x78, 0x18, 0xb4, 0x3f, 0xec, 0x0b, 0xed, 0x35, 0xff, + 0x79, 0x67, 0x3e, 0xdf, 0xd9, 0xef, 0xec, 0xce, 0x1a, 0xa2, 0x5c, 0x88, 0xbc, 0xa0, 0x49, 0x46, + 0x24, 0x69, 0xa4, 0xa8, 0x69, 0x72, 0x73, 0x9a, 0x50, 0x2e, 0x99, 0xec, 0xe2, 0xaa, 0x16, 0x52, + 0xa0, 0x43, 0x43, 0xc4, 0x03, 0x11, 0xdf, 0x9c, 0x1e, 0x7d, 0x64, 0x65, 0xa4, 0x62, 0x09, 0xe1, + 0x5c, 0x48, 0x22, 0x99, 0xe0, 0x8d, 0x91, 0x0c, 0x59, 0xbd, 0x5a, 0xb6, 0x2f, 0x93, 0x46, 0xd6, + 0xed, 0x4a, 0xda, 0xec, 0xf1, 0x9b, 0x59, 0xc9, 0x4a, 0xda, 0x48, 0x52, 0x56, 0x16, 0x08, 0x2d, + 0x20, 0xbb, 0x8a, 0x26, 0x05, 0x91, 0x05, 0xcf, 0x4d, 0xe6, 0xe4, 0x17, 0xf0, 0xe7, 0xa4, 0x96, + 0x4c, 0x6d, 0x76, 0x91, 0xa1, 0x8f, 0x01, 0xaa, 0x5a, 0xfc, 0x4e, 0x57, 0x72, 0xc1, 0xb2, 0x70, + 0x14, 0xb9, 0x53, 0x0f, 0x7b, 0x36, 0x72, 0x91, 0xa1, 0x4f, 0x20, 0xe0, 0xa4, 0xa4, 0x4d, 0x45, + 0x56, 0x54, 0x01, 0x3b, 0x1a, 0xf0, 0x87, 0xd8, 0x45, 0x76, 0xf2, 0x8f, 0x0b, 0xe3, 0x4b, 0xda, + 0xa1, 0x67, 0x10, 0x54, 0x7d, 0x61, 0x85, 0xba, 0x91, 0x3b, 0xf5, 0xcf, 0xa2, 0x78, 0x4b, 0xef, + 0xf1, 0x2d, 0x07, 0xd8, 0xaf, 0x6e, 0xd9, 0x79, 0x0c, 0x3b, 0x15, 0x91, 0xaf, 0xc2, 0x51, 0x34, + 0x9e, 0xfa, 0x67, 0x9f, 0x6d, 0x15, 0x5f, 0xd2, 0x2e, 0x9e, 0x13, 0xf9, 0xea, 0xbc, 0xa0, 0x25, + 0xe5, 0x12, 0x6b, 0xc5, 0xd1, 0x0b, 0xd5, 0xd7, 0x10, 0x44, 0x08, 0x76, 0xd6, 0x8c, 0x1b, 0x17, + 0x1e, 0xd6, 0xdf, 0xe8, 0x01, 0x8c, 0x6c, 0x8f, 0xe3, 0xd4, 0xc1, 0x23, 0x96, 0xa1, 0x87, 0xb0, + 0xa3, 0x5a, 0x09, 0xc7, 0x8a, 0x4a, 0x1d, 0xac, 0x57, 0x33, 0x0f, 0xee, 0xb1, 0x6c, 0xa1, 0x8e, + 0xee, 0xe4, 0x29, 0xc0, 0xf7, 0x75, 0x4d, 0xba, 0x2b, 0x52, 0xb4, 0x14, 0x9d, 0xc1, 0xee, 0x8d, + 0xfa, 0x68, 0x42, 0x57, 0xfb, 0x3b, 0xda, 0xea, 0x4f, 0xb3, 0xd8, 0x92, 0x27, 0x7f, 0x4c, 0x60, + 0x62, 0xd4, 0x4f, 0x00, 0x78, 0x5b, 0x14, 0x0b, 0x9d, 0x08, 0xfd, 0xc8, 0x9d, 0xee, 0x6f, 0x2a, + 0xf4, 0x37, 0x19, 0xff, 0xdc, 0x16, 0x85, 0xe6, 0x53, 0x07, 0x7b, 0xbc, 0x5f, 0xa0, 0xcf, 0xe1, + 0xfe, 0x52, 0x88, 0x82, 0x12, 0x6e, 0xf5, 0xaa, 0xb1, 0xbd, 0xd4, 0xc1, 0x81, 0x0d, 0x0f, 0x18, + 0xe3, 0x92, 0xe6, 0xb4, 0xb6, 0x58, 0xdf, 0x6d, 0x60, 0xc3, 0x06, 0xfb, 0x14, 0x82, 0x4c, 0xb4, + 0xcb, 0x82, 0x5a, 0x4a, 0xf5, 0xef, 0xa6, 0x0e, 0xf6, 0x4d, 0xd4, 0x40, 0xe7, 0x70, 0x30, 0x8c, + 0x95, 0xe5, 0x40, 0xdf, 0xe9, 0xdb, 0xa6, 0x5f, 0xf4, 0x5c, 0xea, 0xe0, 0xfd, 0x41, 0x64, 0xca, + 0x7c, 0x0d, 0xde, 0x9a, 0x76, 0xb6, 0xc0, 0x44, 0x17, 0x08, 0xdf, 0x75, 0xaf, 0xa9, 0x83, 0xf7, + 0xd6, 0xb4, 0x1b, 0x4c, 0x36, 0xb2, 0x66, 0x3c, 0xb7, 0xda, 0xf7, 0xec, 0x25, 0xf9, 0x26, 0x6a, + 0xa0, 0x63, 0x80, 0x65, 0x21, 0x96, 0x16, 0x41, 0x91, 0x3b, 0x0d, 0xd4, 0xc1, 0xa9, 0x98, 0x01, + 0xbe, 0x83, 0x83, 0x9c, 0x8a, 0x45, 0x25, 0x18, 0x97, 0x96, 0xda, 0xd3, 0x26, 0x0e, 0x7b, 0x13, + 0xea, 0xa2, 0xe3, 0xe7, 0x44, 0x3e, 0xe7, 0x79, 0xea, 0xe0, 0xfb, 0x39, 0x15, 0x73, 0x05, 0x1b, + 0xf9, 0x53, 0x08, 0xcc, 0x53, 0xb6, 0xda, 0x5d, 0xad, 0xfd, 0x70, 0x6b, 0x03, 0xe7, 0x1a, 0x54, + 0x0e, 0x8d, 0xc4, 0x54, 0x98, 0x81, 0x4f, 0xd4, 0x08, 0xd9, 0x02, 0x9e, 0x2e, 0x70, 0xbc, 0xb5, + 0xc0, 0x66, 0xd4, 0x52, 0x07, 0x03, 0xd9, 0x0c, 0x5e, 0x08, 0xf7, 0x4a, 0x4a, 0x38, 0xe3, 0x79, + 0xb8, 0x1f, 0xb9, 0xd3, 0x09, 0xee, 0x97, 0xe8, 0x11, 0x3c, 0xa4, 0xaf, 0x57, 0x45, 0x9b, 0xd1, + 0xc5, 0xcb, 0x5a, 0x94, 0x0b, 0xc6, 0x33, 0xfa, 0x9a, 0x36, 0xe1, 0xa1, 0x1a, 0x0f, 0x8c, 0x6c, + 0xee, 0xc7, 0x5a, 0x94, 0x17, 0x26, 0x33, 0x0b, 0x00, 0xb4, 0x13, 0x33, 0xe0, 0xff, 0xba, 0xb0, + 0x6b, 0x7c, 0xa3, 0x2f, 0x60, 0xbc, 0xa6, 0x9d, 0x7d, 0xb7, 0xef, 0xbc, 0x22, 0xac, 0x20, 0x74, + 0xa9, 0x7f, 0x1b, 0x15, 0xad, 0x25, 0xa3, 0x4d, 0x38, 0xd6, 0xaf, 0xe1, 0xcb, 0x3b, 0x0e, 0x25, + 0x9e, 0x0f, 0xf4, 0x39, 0x97, 0x75, 0x87, 0x6f, 0xc9, 0x8f, 0x7e, 0x85, 0x83, 0x37, 0xd2, 0xe8, + 0xc1, 0xc6, 0x8b, 0x67, 0x76, 0x7c, 0x04, 0x93, 0xcd, 0x44, 0xdf, 0xfd, 0xf4, 0x0c, 0xf8, 0xcd, + 0xe8, 0xb1, 0x3b, 0xfb, 0xd3, 0x85, 0xf7, 0x57, 0xa2, 0xdc, 0x06, 0xcf, 0x7c, 0x63, 0x6d, 0xae, + 0x86, 0x78, 0xee, 0xfe, 0xf6, 0xad, 0x65, 0x72, 0x51, 0x10, 0x9e, 0xc7, 0xa2, 0xce, 0x93, 0x9c, + 0x72, 0x3d, 0xe2, 0x89, 0x49, 0x91, 0x8a, 0x35, 0xff, 0xfb, 0xcb, 0x3f, 0x19, 0x16, 0x7f, 0x8d, + 0x3e, 0xf8, 0xc9, 0xc8, 0x9f, 0x15, 0xa2, 0xcd, 0xe2, 0x1f, 0x86, 0x8d, 0xae, 0x4e, 0xff, 0xee, + 0x73, 0xd7, 0x3a, 0x77, 0x3d, 0xe4, 0xae, 0xaf, 0x4e, 0x97, 0xbb, 0x7a, 0x83, 0xaf, 0xfe, 0x0b, + 0x00, 0x00, 0xff, 0xff, 0xf3, 0xdd, 0x11, 0x96, 0x45, 0x06, 0x00, 0x00, +} + +var xxx_messageInfo_Key proto.InternalMessageInfo diff --git a/vendor/google.golang.org/appengine/datastore/key.go b/vendor/google.golang.org/appengine/datastore/key.go new file mode 100644 index 00000000000..fd598dc9657 --- /dev/null +++ b/vendor/google.golang.org/appengine/datastore/key.go @@ -0,0 +1,400 @@ +// Copyright 2011 Google Inc. All rights reserved. +// Use of this source code is governed by the Apache 2.0 +// license that can be found in the LICENSE file. + +package datastore + +import ( + "bytes" + "encoding/base64" + "encoding/gob" + "errors" + "fmt" + "strconv" + "strings" + + "github.com/golang/protobuf/proto" + "golang.org/x/net/context" + + "google.golang.org/appengine/internal" + pb "google.golang.org/appengine/internal/datastore" +) + +type KeyRangeCollisionError struct { + start int64 + end int64 +} + +func (e *KeyRangeCollisionError) Error() string { + return fmt.Sprintf("datastore: Collision when attempting to allocate range [%d, %d]", + e.start, e.end) +} + +type KeyRangeContentionError struct { + start int64 + end int64 +} + +func (e *KeyRangeContentionError) Error() string { + return fmt.Sprintf("datastore: Contention when attempting to allocate range [%d, %d]", + e.start, e.end) +} + +// Key represents the datastore key for a stored entity, and is immutable. +type Key struct { + kind string + stringID string + intID int64 + parent *Key + appID string + namespace string +} + +// Kind returns the key's kind (also known as entity type). +func (k *Key) Kind() string { + return k.kind +} + +// StringID returns the key's string ID (also known as an entity name or key +// name), which may be "". +func (k *Key) StringID() string { + return k.stringID +} + +// IntID returns the key's integer ID, which may be 0. +func (k *Key) IntID() int64 { + return k.intID +} + +// Parent returns the key's parent key, which may be nil. +func (k *Key) Parent() *Key { + return k.parent +} + +// AppID returns the key's application ID. +func (k *Key) AppID() string { + return k.appID +} + +// Namespace returns the key's namespace. +func (k *Key) Namespace() string { + return k.namespace +} + +// Incomplete returns whether the key does not refer to a stored entity. +// In particular, whether the key has a zero StringID and a zero IntID. +func (k *Key) Incomplete() bool { + return k.stringID == "" && k.intID == 0 +} + +// valid returns whether the key is valid. +func (k *Key) valid() bool { + if k == nil { + return false + } + for ; k != nil; k = k.parent { + if k.kind == "" || k.appID == "" { + return false + } + if k.stringID != "" && k.intID != 0 { + return false + } + if k.parent != nil { + if k.parent.Incomplete() { + return false + } + if k.parent.appID != k.appID || k.parent.namespace != k.namespace { + return false + } + } + } + return true +} + +// Equal returns whether two keys are equal. +func (k *Key) Equal(o *Key) bool { + for k != nil && o != nil { + if k.kind != o.kind || k.stringID != o.stringID || k.intID != o.intID || k.appID != o.appID || k.namespace != o.namespace { + return false + } + k, o = k.parent, o.parent + } + return k == o +} + +// root returns the furthest ancestor of a key, which may be itself. +func (k *Key) root() *Key { + for k.parent != nil { + k = k.parent + } + return k +} + +// marshal marshals the key's string representation to the buffer. +func (k *Key) marshal(b *bytes.Buffer) { + if k.parent != nil { + k.parent.marshal(b) + } + b.WriteByte('/') + b.WriteString(k.kind) + b.WriteByte(',') + if k.stringID != "" { + b.WriteString(k.stringID) + } else { + b.WriteString(strconv.FormatInt(k.intID, 10)) + } +} + +// String returns a string representation of the key. +func (k *Key) String() string { + if k == nil { + return "" + } + b := bytes.NewBuffer(make([]byte, 0, 512)) + k.marshal(b) + return b.String() +} + +type gobKey struct { + Kind string + StringID string + IntID int64 + Parent *gobKey + AppID string + Namespace string +} + +func keyToGobKey(k *Key) *gobKey { + if k == nil { + return nil + } + return &gobKey{ + Kind: k.kind, + StringID: k.stringID, + IntID: k.intID, + Parent: keyToGobKey(k.parent), + AppID: k.appID, + Namespace: k.namespace, + } +} + +func gobKeyToKey(gk *gobKey) *Key { + if gk == nil { + return nil + } + return &Key{ + kind: gk.Kind, + stringID: gk.StringID, + intID: gk.IntID, + parent: gobKeyToKey(gk.Parent), + appID: gk.AppID, + namespace: gk.Namespace, + } +} + +func (k *Key) GobEncode() ([]byte, error) { + buf := new(bytes.Buffer) + if err := gob.NewEncoder(buf).Encode(keyToGobKey(k)); err != nil { + return nil, err + } + return buf.Bytes(), nil +} + +func (k *Key) GobDecode(buf []byte) error { + gk := new(gobKey) + if err := gob.NewDecoder(bytes.NewBuffer(buf)).Decode(gk); err != nil { + return err + } + *k = *gobKeyToKey(gk) + return nil +} + +func (k *Key) MarshalJSON() ([]byte, error) { + return []byte(`"` + k.Encode() + `"`), nil +} + +func (k *Key) UnmarshalJSON(buf []byte) error { + if len(buf) < 2 || buf[0] != '"' || buf[len(buf)-1] != '"' { + return errors.New("datastore: bad JSON key") + } + k2, err := DecodeKey(string(buf[1 : len(buf)-1])) + if err != nil { + return err + } + *k = *k2 + return nil +} + +// Encode returns an opaque representation of the key +// suitable for use in HTML and URLs. +// This is compatible with the Python and Java runtimes. +func (k *Key) Encode() string { + ref := keyToProto("", k) + + b, err := proto.Marshal(ref) + if err != nil { + panic(err) + } + + // Trailing padding is stripped. + return strings.TrimRight(base64.URLEncoding.EncodeToString(b), "=") +} + +// DecodeKey decodes a key from the opaque representation returned by Encode. +func DecodeKey(encoded string) (*Key, error) { + // Re-add padding. + if m := len(encoded) % 4; m != 0 { + encoded += strings.Repeat("=", 4-m) + } + + b, err := base64.URLEncoding.DecodeString(encoded) + if err != nil { + return nil, err + } + + ref := new(pb.Reference) + if err := proto.Unmarshal(b, ref); err != nil { + // Couldn't decode it as an App Engine key, try decoding it as a key encoded by cloud.google.com/go/datastore. + if k := decodeCloudKey(encoded); k != nil { + return k, nil + } + return nil, err + } + + return protoToKey(ref) +} + +// NewIncompleteKey creates a new incomplete key. +// kind cannot be empty. +func NewIncompleteKey(c context.Context, kind string, parent *Key) *Key { + return NewKey(c, kind, "", 0, parent) +} + +// NewKey creates a new key. +// kind cannot be empty. +// Either one or both of stringID and intID must be zero. If both are zero, +// the key returned is incomplete. +// parent must either be a complete key or nil. +func NewKey(c context.Context, kind, stringID string, intID int64, parent *Key) *Key { + // If there's a parent key, use its namespace. + // Otherwise, use any namespace attached to the context. + var namespace string + if parent != nil { + namespace = parent.namespace + } else { + namespace = internal.NamespaceFromContext(c) + } + + return &Key{ + kind: kind, + stringID: stringID, + intID: intID, + parent: parent, + appID: internal.FullyQualifiedAppID(c), + namespace: namespace, + } +} + +// AllocateIDs returns a range of n integer IDs with the given kind and parent +// combination. kind cannot be empty; parent may be nil. The IDs in the range +// returned will not be used by the datastore's automatic ID sequence generator +// and may be used with NewKey without conflict. +// +// The range is inclusive at the low end and exclusive at the high end. In +// other words, valid intIDs x satisfy low <= x && x < high. +// +// If no error is returned, low + n == high. +func AllocateIDs(c context.Context, kind string, parent *Key, n int) (low, high int64, err error) { + if kind == "" { + return 0, 0, errors.New("datastore: AllocateIDs given an empty kind") + } + if n < 0 { + return 0, 0, fmt.Errorf("datastore: AllocateIDs given a negative count: %d", n) + } + if n == 0 { + return 0, 0, nil + } + req := &pb.AllocateIdsRequest{ + ModelKey: keyToProto("", NewIncompleteKey(c, kind, parent)), + Size: proto.Int64(int64(n)), + } + res := &pb.AllocateIdsResponse{} + if err := internal.Call(c, "datastore_v3", "AllocateIds", req, res); err != nil { + return 0, 0, err + } + // The protobuf is inclusive at both ends. Idiomatic Go (e.g. slices, for loops) + // is inclusive at the low end and exclusive at the high end, so we add 1. + low = res.GetStart() + high = res.GetEnd() + 1 + if low+int64(n) != high { + return 0, 0, fmt.Errorf("datastore: internal error: could not allocate %d IDs", n) + } + return low, high, nil +} + +// AllocateIDRange allocates a range of IDs with specific endpoints. +// The range is inclusive at both the low and high end. Once these IDs have been +// allocated, you can manually assign them to newly created entities. +// +// The Datastore's automatic ID allocator never assigns a key that has already +// been allocated (either through automatic ID allocation or through an explicit +// AllocateIDs call). As a result, entities written to the given key range will +// never be overwritten. However, writing entities with manually assigned keys in +// this range may overwrite existing entities (or new entities written by a separate +// request), depending on the error returned. +// +// Use this only if you have an existing numeric ID range that you want to reserve +// (for example, bulk loading entities that already have IDs). If you don't care +// about which IDs you receive, use AllocateIDs instead. +// +// AllocateIDRange returns nil if the range is successfully allocated. If one or more +// entities with an ID in the given range already exist, it returns a KeyRangeCollisionError. +// If the Datastore has already cached IDs in this range (e.g. from a previous call to +// AllocateIDRange), it returns a KeyRangeContentionError. Errors of other types indicate +// problems with arguments or an error returned directly from the Datastore. +func AllocateIDRange(c context.Context, kind string, parent *Key, start, end int64) (err error) { + if kind == "" { + return errors.New("datastore: AllocateIDRange given an empty kind") + } + + if start < 1 || end < 1 { + return errors.New("datastore: AllocateIDRange start and end must both be greater than 0") + } + + if start > end { + return errors.New("datastore: AllocateIDRange start must be before end") + } + + req := &pb.AllocateIdsRequest{ + ModelKey: keyToProto("", NewIncompleteKey(c, kind, parent)), + Max: proto.Int64(end), + } + res := &pb.AllocateIdsResponse{} + if err := internal.Call(c, "datastore_v3", "AllocateIds", req, res); err != nil { + return err + } + + // Check for collisions, i.e. existing entities with IDs in this range. + // We could do this before the allocation, but we'd still have to do it + // afterward as well to catch the race condition where an entity is inserted + // after that initial check but before the allocation. Skip the up-front check + // and just do it once. + q := NewQuery(kind).Filter("__key__ >=", NewKey(c, kind, "", start, parent)). + Filter("__key__ <=", NewKey(c, kind, "", end, parent)).KeysOnly().Limit(1) + + keys, err := q.GetAll(c, nil) + if err != nil { + return err + } + if len(keys) != 0 { + return &KeyRangeCollisionError{start: start, end: end} + } + + // Check for a race condition, i.e. cases where the datastore may have + // cached ID batches that contain IDs in this range. + if start < res.GetStart() { + return &KeyRangeContentionError{start: start, end: end} + } + + return nil +} diff --git a/vendor/google.golang.org/appengine/datastore/keycompat.go b/vendor/google.golang.org/appengine/datastore/keycompat.go new file mode 100644 index 00000000000..371a64eeefe --- /dev/null +++ b/vendor/google.golang.org/appengine/datastore/keycompat.go @@ -0,0 +1,89 @@ +// Copyright 2019 Google Inc. All rights reserved. +// Use of this source code is governed by the Apache 2.0 +// license that can be found in the LICENSE file. + +package datastore + +import ( + "sync" + + "golang.org/x/net/context" + + "google.golang.org/appengine/datastore/internal/cloudkey" + "google.golang.org/appengine/internal" +) + +var keyConversion struct { + mu sync.RWMutex + appID string // read using getKeyConversionAppID +} + +// EnableKeyConversion enables encoded key compatibility with the Cloud +// Datastore client library (cloud.google.com/go/datastore). Encoded keys +// generated by the Cloud Datastore client library will be decoded into App +// Engine datastore keys. +// +// The context provided must be an App Engine context if running in App Engine +// first generation runtime. This can be called in the /_ah/start handler. It is +// safe to call multiple times, and is cheap to call, so can also be inserted as +// middleware. +// +// Enabling key compatibility does not affect the encoding format used by +// Key.Encode, it only expands the type of keys that are able to be decoded with +// DecodeKey. +func EnableKeyConversion(ctx context.Context) { + // Only attempt to set appID if it's unset. + // If already set, ignore. + if getKeyConversionAppID() != "" { + return + } + + keyConversion.mu.Lock() + // Check again to avoid race where another goroutine set appID between the call + // to getKeyConversionAppID above and taking the write lock. + if keyConversion.appID == "" { + keyConversion.appID = internal.FullyQualifiedAppID(ctx) + } + keyConversion.mu.Unlock() +} + +func getKeyConversionAppID() string { + keyConversion.mu.RLock() + appID := keyConversion.appID + keyConversion.mu.RUnlock() + return appID +} + +// decodeCloudKey attempts to decode the given encoded key generated by the +// Cloud Datastore client library (cloud.google.com/go/datastore), returning nil +// if the key couldn't be decoded. +func decodeCloudKey(encoded string) *Key { + appID := getKeyConversionAppID() + if appID == "" { + return nil + } + + k, err := cloudkey.DecodeKey(encoded) + if err != nil { + return nil + } + return convertCloudKey(k, appID) +} + +// convertCloudKey converts a Cloud Datastore key and converts it to an App +// Engine Datastore key. Cloud Datastore keys don't include the project/app ID, +// so we must add it back in. +func convertCloudKey(key *cloudkey.Key, appID string) *Key { + if key == nil { + return nil + } + k := &Key{ + intID: key.ID, + kind: key.Kind, + namespace: key.Namespace, + parent: convertCloudKey(key.Parent, appID), + stringID: key.Name, + appID: appID, + } + return k +} diff --git a/vendor/google.golang.org/appengine/datastore/load.go b/vendor/google.golang.org/appengine/datastore/load.go new file mode 100644 index 00000000000..38a63653979 --- /dev/null +++ b/vendor/google.golang.org/appengine/datastore/load.go @@ -0,0 +1,429 @@ +// Copyright 2011 Google Inc. All rights reserved. +// Use of this source code is governed by the Apache 2.0 +// license that can be found in the LICENSE file. + +package datastore + +import ( + "fmt" + "reflect" + "strings" + "time" + + "github.com/golang/protobuf/proto" + "google.golang.org/appengine" + pb "google.golang.org/appengine/internal/datastore" +) + +var ( + typeOfBlobKey = reflect.TypeOf(appengine.BlobKey("")) + typeOfByteSlice = reflect.TypeOf([]byte(nil)) + typeOfByteString = reflect.TypeOf(ByteString(nil)) + typeOfGeoPoint = reflect.TypeOf(appengine.GeoPoint{}) + typeOfTime = reflect.TypeOf(time.Time{}) + typeOfKeyPtr = reflect.TypeOf(&Key{}) + typeOfEntityPtr = reflect.TypeOf(&Entity{}) +) + +// typeMismatchReason returns a string explaining why the property p could not +// be stored in an entity field of type v.Type(). +func typeMismatchReason(pValue interface{}, v reflect.Value) string { + entityType := "empty" + switch pValue.(type) { + case int64: + entityType = "int" + case bool: + entityType = "bool" + case string: + entityType = "string" + case float64: + entityType = "float" + case *Key: + entityType = "*datastore.Key" + case time.Time: + entityType = "time.Time" + case appengine.BlobKey: + entityType = "appengine.BlobKey" + case appengine.GeoPoint: + entityType = "appengine.GeoPoint" + case ByteString: + entityType = "datastore.ByteString" + case []byte: + entityType = "[]byte" + } + return fmt.Sprintf("type mismatch: %s versus %v", entityType, v.Type()) +} + +type propertyLoader struct { + // m holds the number of times a substruct field like "Foo.Bar.Baz" has + // been seen so far. The map is constructed lazily. + m map[string]int +} + +func (l *propertyLoader) load(codec *structCodec, structValue reflect.Value, p Property, requireSlice bool) string { + var v reflect.Value + var sliceIndex int + + name := p.Name + + // If name ends with a '.', the last field is anonymous. + // In this case, strings.Split will give us "" as the + // last element of our fields slice, which will match the "" + // field name in the substruct codec. + fields := strings.Split(name, ".") + + for len(fields) > 0 { + var decoder fieldCodec + var ok bool + + // Cut off the last field (delimited by ".") and find its parent + // in the codec. + // eg. for name "A.B.C.D", split off "A.B.C" and try to + // find a field in the codec with this name. + // Loop again with "A.B", etc. + for i := len(fields); i > 0; i-- { + parent := strings.Join(fields[:i], ".") + decoder, ok = codec.fields[parent] + if ok { + fields = fields[i:] + break + } + } + + // If we never found a matching field in the codec, return + // error message. + if !ok { + return "no such struct field" + } + + v = initField(structValue, decoder.path) + if !v.IsValid() { + return "no such struct field" + } + if !v.CanSet() { + return "cannot set struct field" + } + + if decoder.structCodec != nil { + codec = decoder.structCodec + structValue = v + } + + if v.Kind() == reflect.Slice && v.Type() != typeOfByteSlice { + if l.m == nil { + l.m = make(map[string]int) + } + sliceIndex = l.m[p.Name] + l.m[p.Name] = sliceIndex + 1 + for v.Len() <= sliceIndex { + v.Set(reflect.Append(v, reflect.New(v.Type().Elem()).Elem())) + } + structValue = v.Index(sliceIndex) + requireSlice = false + } + } + + var slice reflect.Value + if v.Kind() == reflect.Slice && v.Type().Elem().Kind() != reflect.Uint8 { + slice = v + v = reflect.New(v.Type().Elem()).Elem() + } else if requireSlice { + return "multiple-valued property requires a slice field type" + } + + // Convert indexValues to a Go value with a meaning derived from the + // destination type. + pValue := p.Value + if iv, ok := pValue.(indexValue); ok { + meaning := pb.Property_NO_MEANING + switch v.Type() { + case typeOfBlobKey: + meaning = pb.Property_BLOBKEY + case typeOfByteSlice: + meaning = pb.Property_BLOB + case typeOfByteString: + meaning = pb.Property_BYTESTRING + case typeOfGeoPoint: + meaning = pb.Property_GEORSS_POINT + case typeOfTime: + meaning = pb.Property_GD_WHEN + case typeOfEntityPtr: + meaning = pb.Property_ENTITY_PROTO + } + var err error + pValue, err = propValue(iv.value, meaning) + if err != nil { + return err.Error() + } + } + + if errReason := setVal(v, pValue); errReason != "" { + // Set the slice back to its zero value. + if slice.IsValid() { + slice.Set(reflect.Zero(slice.Type())) + } + return errReason + } + + if slice.IsValid() { + slice.Index(sliceIndex).Set(v) + } + + return "" +} + +// setVal sets v to the value pValue. +func setVal(v reflect.Value, pValue interface{}) string { + switch v.Kind() { + case reflect.Int, reflect.Int8, reflect.Int16, reflect.Int32, reflect.Int64: + x, ok := pValue.(int64) + if !ok && pValue != nil { + return typeMismatchReason(pValue, v) + } + if v.OverflowInt(x) { + return fmt.Sprintf("value %v overflows struct field of type %v", x, v.Type()) + } + v.SetInt(x) + case reflect.Bool: + x, ok := pValue.(bool) + if !ok && pValue != nil { + return typeMismatchReason(pValue, v) + } + v.SetBool(x) + case reflect.String: + switch x := pValue.(type) { + case appengine.BlobKey: + v.SetString(string(x)) + case ByteString: + v.SetString(string(x)) + case string: + v.SetString(x) + default: + if pValue != nil { + return typeMismatchReason(pValue, v) + } + } + case reflect.Float32, reflect.Float64: + x, ok := pValue.(float64) + if !ok && pValue != nil { + return typeMismatchReason(pValue, v) + } + if v.OverflowFloat(x) { + return fmt.Sprintf("value %v overflows struct field of type %v", x, v.Type()) + } + v.SetFloat(x) + case reflect.Ptr: + x, ok := pValue.(*Key) + if !ok && pValue != nil { + return typeMismatchReason(pValue, v) + } + if _, ok := v.Interface().(*Key); !ok { + return typeMismatchReason(pValue, v) + } + v.Set(reflect.ValueOf(x)) + case reflect.Struct: + switch v.Type() { + case typeOfTime: + x, ok := pValue.(time.Time) + if !ok && pValue != nil { + return typeMismatchReason(pValue, v) + } + v.Set(reflect.ValueOf(x)) + case typeOfGeoPoint: + x, ok := pValue.(appengine.GeoPoint) + if !ok && pValue != nil { + return typeMismatchReason(pValue, v) + } + v.Set(reflect.ValueOf(x)) + default: + ent, ok := pValue.(*Entity) + if !ok { + return typeMismatchReason(pValue, v) + } + + // Recursively load nested struct + pls, err := newStructPLS(v.Addr().Interface()) + if err != nil { + return err.Error() + } + + // if ent has a Key value and our struct has a Key field, + // load the Entity's Key value into the Key field on the struct. + if ent.Key != nil && pls.codec.keyField != -1 { + + pls.v.Field(pls.codec.keyField).Set(reflect.ValueOf(ent.Key)) + } + + err = pls.Load(ent.Properties) + if err != nil { + return err.Error() + } + } + case reflect.Slice: + x, ok := pValue.([]byte) + if !ok { + if y, yok := pValue.(ByteString); yok { + x, ok = []byte(y), true + } + } + if !ok && pValue != nil { + return typeMismatchReason(pValue, v) + } + if v.Type().Elem().Kind() != reflect.Uint8 { + return typeMismatchReason(pValue, v) + } + v.SetBytes(x) + default: + return typeMismatchReason(pValue, v) + } + return "" +} + +// initField is similar to reflect's Value.FieldByIndex, in that it +// returns the nested struct field corresponding to index, but it +// initialises any nil pointers encountered when traversing the structure. +func initField(val reflect.Value, index []int) reflect.Value { + for _, i := range index[:len(index)-1] { + val = val.Field(i) + if val.Kind() == reflect.Ptr { + if val.IsNil() { + val.Set(reflect.New(val.Type().Elem())) + } + val = val.Elem() + } + } + return val.Field(index[len(index)-1]) +} + +// loadEntity loads an EntityProto into PropertyLoadSaver or struct pointer. +func loadEntity(dst interface{}, src *pb.EntityProto) (err error) { + ent, err := protoToEntity(src) + if err != nil { + return err + } + if e, ok := dst.(PropertyLoadSaver); ok { + return e.Load(ent.Properties) + } + return LoadStruct(dst, ent.Properties) +} + +func (s structPLS) Load(props []Property) error { + var fieldName, reason string + var l propertyLoader + for _, p := range props { + if errStr := l.load(s.codec, s.v, p, p.Multiple); errStr != "" { + // We don't return early, as we try to load as many properties as possible. + // It is valid to load an entity into a struct that cannot fully represent it. + // That case returns an error, but the caller is free to ignore it. + fieldName, reason = p.Name, errStr + } + } + if reason != "" { + return &ErrFieldMismatch{ + StructType: s.v.Type(), + FieldName: fieldName, + Reason: reason, + } + } + return nil +} + +func protoToEntity(src *pb.EntityProto) (*Entity, error) { + props, rawProps := src.Property, src.RawProperty + outProps := make([]Property, 0, len(props)+len(rawProps)) + for { + var ( + x *pb.Property + noIndex bool + ) + if len(props) > 0 { + x, props = props[0], props[1:] + } else if len(rawProps) > 0 { + x, rawProps = rawProps[0], rawProps[1:] + noIndex = true + } else { + break + } + + var value interface{} + if x.Meaning != nil && *x.Meaning == pb.Property_INDEX_VALUE { + value = indexValue{x.Value} + } else { + var err error + value, err = propValue(x.Value, x.GetMeaning()) + if err != nil { + return nil, err + } + } + outProps = append(outProps, Property{ + Name: x.GetName(), + Value: value, + NoIndex: noIndex, + Multiple: x.GetMultiple(), + }) + } + + var key *Key + if src.Key != nil { + // Ignore any error, since nested entity values + // are allowed to have an invalid key. + key, _ = protoToKey(src.Key) + } + return &Entity{key, outProps}, nil +} + +// propValue returns a Go value that combines the raw PropertyValue with a +// meaning. For example, an Int64Value with GD_WHEN becomes a time.Time. +func propValue(v *pb.PropertyValue, m pb.Property_Meaning) (interface{}, error) { + switch { + case v.Int64Value != nil: + if m == pb.Property_GD_WHEN { + return fromUnixMicro(*v.Int64Value), nil + } else { + return *v.Int64Value, nil + } + case v.BooleanValue != nil: + return *v.BooleanValue, nil + case v.StringValue != nil: + if m == pb.Property_BLOB { + return []byte(*v.StringValue), nil + } else if m == pb.Property_BLOBKEY { + return appengine.BlobKey(*v.StringValue), nil + } else if m == pb.Property_BYTESTRING { + return ByteString(*v.StringValue), nil + } else if m == pb.Property_ENTITY_PROTO { + var ent pb.EntityProto + err := proto.Unmarshal([]byte(*v.StringValue), &ent) + if err != nil { + return nil, err + } + return protoToEntity(&ent) + } else { + return *v.StringValue, nil + } + case v.DoubleValue != nil: + return *v.DoubleValue, nil + case v.Referencevalue != nil: + key, err := referenceValueToKey(v.Referencevalue) + if err != nil { + return nil, err + } + return key, nil + case v.Pointvalue != nil: + // NOTE: Strangely, latitude maps to X, longitude to Y. + return appengine.GeoPoint{Lat: v.Pointvalue.GetX(), Lng: v.Pointvalue.GetY()}, nil + } + return nil, nil +} + +// indexValue is a Property value that is created when entities are loaded from +// an index, such as from a projection query. +// +// Such Property values do not contain all of the metadata required to be +// faithfully represented as a Go value, and are instead represented as an +// opaque indexValue. Load the properties into a concrete struct type (e.g. by +// passing a struct pointer to Iterator.Next) to reconstruct actual Go values +// of type int, string, time.Time, etc. +type indexValue struct { + value *pb.PropertyValue +} diff --git a/vendor/google.golang.org/appengine/datastore/metadata.go b/vendor/google.golang.org/appengine/datastore/metadata.go new file mode 100644 index 00000000000..6acacc3db9a --- /dev/null +++ b/vendor/google.golang.org/appengine/datastore/metadata.go @@ -0,0 +1,78 @@ +// Copyright 2016 Google Inc. All rights reserved. +// Use of this source code is governed by the Apache 2.0 +// license that can be found in the LICENSE file. + +package datastore + +import "golang.org/x/net/context" + +// Datastore kinds for the metadata entities. +const ( + namespaceKind = "__namespace__" + kindKind = "__kind__" + propertyKind = "__property__" +) + +// Namespaces returns all the datastore namespaces. +func Namespaces(ctx context.Context) ([]string, error) { + // TODO(djd): Support range queries. + q := NewQuery(namespaceKind).KeysOnly() + keys, err := q.GetAll(ctx, nil) + if err != nil { + return nil, err + } + // The empty namespace key uses a numeric ID (==1), but luckily + // the string ID defaults to "" for numeric IDs anyway. + return keyNames(keys), nil +} + +// Kinds returns the names of all the kinds in the current namespace. +func Kinds(ctx context.Context) ([]string, error) { + // TODO(djd): Support range queries. + q := NewQuery(kindKind).KeysOnly() + keys, err := q.GetAll(ctx, nil) + if err != nil { + return nil, err + } + return keyNames(keys), nil +} + +// keyNames returns a slice of the provided keys' names (string IDs). +func keyNames(keys []*Key) []string { + n := make([]string, 0, len(keys)) + for _, k := range keys { + n = append(n, k.StringID()) + } + return n +} + +// KindProperties returns all the indexed properties for the given kind. +// The properties are returned as a map of property names to a slice of the +// representation types. The representation types for the supported Go property +// types are: +// "INT64": signed integers and time.Time +// "DOUBLE": float32 and float64 +// "BOOLEAN": bool +// "STRING": string, []byte and ByteString +// "POINT": appengine.GeoPoint +// "REFERENCE": *Key +// "USER": (not used in the Go runtime) +func KindProperties(ctx context.Context, kind string) (map[string][]string, error) { + // TODO(djd): Support range queries. + kindKey := NewKey(ctx, kindKind, kind, 0, nil) + q := NewQuery(propertyKind).Ancestor(kindKey) + + propMap := map[string][]string{} + props := []struct { + Repr []string `datastore:"property_representation"` + }{} + + keys, err := q.GetAll(ctx, &props) + if err != nil { + return nil, err + } + for i, p := range props { + propMap[keys[i].StringID()] = p.Repr + } + return propMap, nil +} diff --git a/vendor/google.golang.org/appengine/datastore/prop.go b/vendor/google.golang.org/appengine/datastore/prop.go new file mode 100644 index 00000000000..5cb2079d887 --- /dev/null +++ b/vendor/google.golang.org/appengine/datastore/prop.go @@ -0,0 +1,330 @@ +// Copyright 2011 Google Inc. All rights reserved. +// Use of this source code is governed by the Apache 2.0 +// license that can be found in the LICENSE file. + +package datastore + +import ( + "fmt" + "reflect" + "strings" + "sync" + "unicode" +) + +// Entities with more than this many indexed properties will not be saved. +const maxIndexedProperties = 20000 + +// []byte fields more than 1 megabyte long will not be loaded or saved. +const maxBlobLen = 1 << 20 + +// Property is a name/value pair plus some metadata. A datastore entity's +// contents are loaded and saved as a sequence of Properties. An entity can +// have multiple Properties with the same name, provided that p.Multiple is +// true on all of that entity's Properties with that name. +type Property struct { + // Name is the property name. + Name string + // Value is the property value. The valid types are: + // - int64 + // - bool + // - string + // - float64 + // - ByteString + // - *Key + // - time.Time + // - appengine.BlobKey + // - appengine.GeoPoint + // - []byte (up to 1 megabyte in length) + // - *Entity (representing a nested struct) + // This set is smaller than the set of valid struct field types that the + // datastore can load and save. A Property Value cannot be a slice (apart + // from []byte); use multiple Properties instead. Also, a Value's type + // must be explicitly on the list above; it is not sufficient for the + // underlying type to be on that list. For example, a Value of "type + // myInt64 int64" is invalid. Smaller-width integers and floats are also + // invalid. Again, this is more restrictive than the set of valid struct + // field types. + // + // A Value will have an opaque type when loading entities from an index, + // such as via a projection query. Load entities into a struct instead + // of a PropertyLoadSaver when using a projection query. + // + // A Value may also be the nil interface value; this is equivalent to + // Python's None but not directly representable by a Go struct. Loading + // a nil-valued property into a struct will set that field to the zero + // value. + Value interface{} + // NoIndex is whether the datastore cannot index this property. + NoIndex bool + // Multiple is whether the entity can have multiple properties with + // the same name. Even if a particular instance only has one property with + // a certain name, Multiple should be true if a struct would best represent + // it as a field of type []T instead of type T. + Multiple bool +} + +// An Entity is the value type for a nested struct. +// This type is only used for a Property's Value. +type Entity struct { + Key *Key + Properties []Property +} + +// ByteString is a short byte slice (up to 1500 bytes) that can be indexed. +type ByteString []byte + +// PropertyLoadSaver can be converted from and to a slice of Properties. +type PropertyLoadSaver interface { + Load([]Property) error + Save() ([]Property, error) +} + +// PropertyList converts a []Property to implement PropertyLoadSaver. +type PropertyList []Property + +var ( + typeOfPropertyLoadSaver = reflect.TypeOf((*PropertyLoadSaver)(nil)).Elem() + typeOfPropertyList = reflect.TypeOf(PropertyList(nil)) +) + +// Load loads all of the provided properties into l. +// It does not first reset *l to an empty slice. +func (l *PropertyList) Load(p []Property) error { + *l = append(*l, p...) + return nil +} + +// Save saves all of l's properties as a slice or Properties. +func (l *PropertyList) Save() ([]Property, error) { + return *l, nil +} + +// validPropertyName returns whether name consists of one or more valid Go +// identifiers joined by ".". +func validPropertyName(name string) bool { + if name == "" { + return false + } + for _, s := range strings.Split(name, ".") { + if s == "" { + return false + } + first := true + for _, c := range s { + if first { + first = false + if c != '_' && !unicode.IsLetter(c) { + return false + } + } else { + if c != '_' && !unicode.IsLetter(c) && !unicode.IsDigit(c) { + return false + } + } + } + } + return true +} + +// structCodec describes how to convert a struct to and from a sequence of +// properties. +type structCodec struct { + // fields gives the field codec for the structTag with the given name. + fields map[string]fieldCodec + // hasSlice is whether a struct or any of its nested or embedded structs + // has a slice-typed field (other than []byte). + hasSlice bool + // keyField is the index of a *Key field with structTag __key__. + // This field is not relevant for the top level struct, only for + // nested structs. + keyField int + // complete is whether the structCodec is complete. An incomplete + // structCodec may be encountered when walking a recursive struct. + complete bool +} + +// fieldCodec is a struct field's index and, if that struct field's type is +// itself a struct, that substruct's structCodec. +type fieldCodec struct { + // path is the index path to the field + path []int + noIndex bool + // omitEmpty indicates that the field should be omitted on save + // if empty. + omitEmpty bool + // structCodec is the codec fot the struct field at index 'path', + // or nil if the field is not a struct. + structCodec *structCodec +} + +// structCodecs collects the structCodecs that have already been calculated. +var ( + structCodecsMutex sync.Mutex + structCodecs = make(map[reflect.Type]*structCodec) +) + +// getStructCodec returns the structCodec for the given struct type. +func getStructCodec(t reflect.Type) (*structCodec, error) { + structCodecsMutex.Lock() + defer structCodecsMutex.Unlock() + return getStructCodecLocked(t) +} + +// getStructCodecLocked implements getStructCodec. The structCodecsMutex must +// be held when calling this function. +func getStructCodecLocked(t reflect.Type) (ret *structCodec, retErr error) { + c, ok := structCodecs[t] + if ok { + return c, nil + } + c = &structCodec{ + fields: make(map[string]fieldCodec), + // We initialize keyField to -1 so that the zero-value is not + // misinterpreted as index 0. + keyField: -1, + } + + // Add c to the structCodecs map before we are sure it is good. If t is + // a recursive type, it needs to find the incomplete entry for itself in + // the map. + structCodecs[t] = c + defer func() { + if retErr != nil { + delete(structCodecs, t) + } + }() + + for i := 0; i < t.NumField(); i++ { + f := t.Field(i) + // Skip unexported fields. + // Note that if f is an anonymous, unexported struct field, + // we will promote its fields. + if f.PkgPath != "" && !f.Anonymous { + continue + } + + tags := strings.Split(f.Tag.Get("datastore"), ",") + name := tags[0] + opts := make(map[string]bool) + for _, t := range tags[1:] { + opts[t] = true + } + switch { + case name == "": + if !f.Anonymous { + name = f.Name + } + case name == "-": + continue + case name == "__key__": + if f.Type != typeOfKeyPtr { + return nil, fmt.Errorf("datastore: __key__ field on struct %v is not a *datastore.Key", t) + } + c.keyField = i + case !validPropertyName(name): + return nil, fmt.Errorf("datastore: struct tag has invalid property name: %q", name) + } + + substructType, fIsSlice := reflect.Type(nil), false + switch f.Type.Kind() { + case reflect.Struct: + substructType = f.Type + case reflect.Slice: + if f.Type.Elem().Kind() == reflect.Struct { + substructType = f.Type.Elem() + } + fIsSlice = f.Type != typeOfByteSlice + c.hasSlice = c.hasSlice || fIsSlice + } + + var sub *structCodec + if substructType != nil && substructType != typeOfTime && substructType != typeOfGeoPoint { + var err error + sub, err = getStructCodecLocked(substructType) + if err != nil { + return nil, err + } + if !sub.complete { + return nil, fmt.Errorf("datastore: recursive struct: field %q", f.Name) + } + if fIsSlice && sub.hasSlice { + return nil, fmt.Errorf( + "datastore: flattening nested structs leads to a slice of slices: field %q", f.Name) + } + c.hasSlice = c.hasSlice || sub.hasSlice + // If f is an anonymous struct field, we promote the substruct's fields up to this level + // in the linked list of struct codecs. + if f.Anonymous { + for subname, subfield := range sub.fields { + if name != "" { + subname = name + "." + subname + } + if _, ok := c.fields[subname]; ok { + return nil, fmt.Errorf("datastore: struct tag has repeated property name: %q", subname) + } + c.fields[subname] = fieldCodec{ + path: append([]int{i}, subfield.path...), + noIndex: subfield.noIndex || opts["noindex"], + omitEmpty: subfield.omitEmpty, + structCodec: subfield.structCodec, + } + } + continue + } + } + + if _, ok := c.fields[name]; ok { + return nil, fmt.Errorf("datastore: struct tag has repeated property name: %q", name) + } + c.fields[name] = fieldCodec{ + path: []int{i}, + noIndex: opts["noindex"], + omitEmpty: opts["omitempty"], + structCodec: sub, + } + } + c.complete = true + return c, nil +} + +// structPLS adapts a struct to be a PropertyLoadSaver. +type structPLS struct { + v reflect.Value + codec *structCodec +} + +// newStructPLS returns a structPLS, which implements the +// PropertyLoadSaver interface, for the struct pointer p. +func newStructPLS(p interface{}) (*structPLS, error) { + v := reflect.ValueOf(p) + if v.Kind() != reflect.Ptr || v.Elem().Kind() != reflect.Struct { + return nil, ErrInvalidEntityType + } + v = v.Elem() + codec, err := getStructCodec(v.Type()) + if err != nil { + return nil, err + } + return &structPLS{v, codec}, nil +} + +// LoadStruct loads the properties from p to dst. +// dst must be a struct pointer. +func LoadStruct(dst interface{}, p []Property) error { + x, err := newStructPLS(dst) + if err != nil { + return err + } + return x.Load(p) +} + +// SaveStruct returns the properties from src as a slice of Properties. +// src must be a struct pointer. +func SaveStruct(src interface{}) ([]Property, error) { + x, err := newStructPLS(src) + if err != nil { + return nil, err + } + return x.Save() +} diff --git a/vendor/google.golang.org/appengine/datastore/query.go b/vendor/google.golang.org/appengine/datastore/query.go new file mode 100644 index 00000000000..4124534b22f --- /dev/null +++ b/vendor/google.golang.org/appengine/datastore/query.go @@ -0,0 +1,774 @@ +// Copyright 2011 Google Inc. All rights reserved. +// Use of this source code is governed by the Apache 2.0 +// license that can be found in the LICENSE file. + +package datastore + +import ( + "encoding/base64" + "errors" + "fmt" + "math" + "reflect" + "strings" + + "github.com/golang/protobuf/proto" + "golang.org/x/net/context" + + "google.golang.org/appengine/internal" + pb "google.golang.org/appengine/internal/datastore" +) + +type operator int + +const ( + lessThan operator = iota + lessEq + equal + greaterEq + greaterThan +) + +var operatorToProto = map[operator]*pb.Query_Filter_Operator{ + lessThan: pb.Query_Filter_LESS_THAN.Enum(), + lessEq: pb.Query_Filter_LESS_THAN_OR_EQUAL.Enum(), + equal: pb.Query_Filter_EQUAL.Enum(), + greaterEq: pb.Query_Filter_GREATER_THAN_OR_EQUAL.Enum(), + greaterThan: pb.Query_Filter_GREATER_THAN.Enum(), +} + +// filter is a conditional filter on query results. +type filter struct { + FieldName string + Op operator + Value interface{} +} + +type sortDirection int + +const ( + ascending sortDirection = iota + descending +) + +var sortDirectionToProto = map[sortDirection]*pb.Query_Order_Direction{ + ascending: pb.Query_Order_ASCENDING.Enum(), + descending: pb.Query_Order_DESCENDING.Enum(), +} + +// order is a sort order on query results. +type order struct { + FieldName string + Direction sortDirection +} + +// NewQuery creates a new Query for a specific entity kind. +// +// An empty kind means to return all entities, including entities created and +// managed by other App Engine features, and is called a kindless query. +// Kindless queries cannot include filters or sort orders on property values. +func NewQuery(kind string) *Query { + return &Query{ + kind: kind, + limit: -1, + } +} + +// Query represents a datastore query. +type Query struct { + kind string + ancestor *Key + filter []filter + order []order + projection []string + + distinct bool + distinctOn []string + keysOnly bool + eventual bool + limit int32 + offset int32 + count int32 + start *pb.CompiledCursor + end *pb.CompiledCursor + + err error +} + +func (q *Query) clone() *Query { + x := *q + // Copy the contents of the slice-typed fields to a new backing store. + if len(q.filter) > 0 { + x.filter = make([]filter, len(q.filter)) + copy(x.filter, q.filter) + } + if len(q.order) > 0 { + x.order = make([]order, len(q.order)) + copy(x.order, q.order) + } + return &x +} + +// Ancestor returns a derivative query with an ancestor filter. +// The ancestor should not be nil. +func (q *Query) Ancestor(ancestor *Key) *Query { + q = q.clone() + if ancestor == nil { + q.err = errors.New("datastore: nil query ancestor") + return q + } + q.ancestor = ancestor + return q +} + +// EventualConsistency returns a derivative query that returns eventually +// consistent results. +// It only has an effect on ancestor queries. +func (q *Query) EventualConsistency() *Query { + q = q.clone() + q.eventual = true + return q +} + +// Filter returns a derivative query with a field-based filter. +// The filterStr argument must be a field name followed by optional space, +// followed by an operator, one of ">", "<", ">=", "<=", or "=". +// Fields are compared against the provided value using the operator. +// Multiple filters are AND'ed together. +func (q *Query) Filter(filterStr string, value interface{}) *Query { + q = q.clone() + filterStr = strings.TrimSpace(filterStr) + if len(filterStr) < 1 { + q.err = errors.New("datastore: invalid filter: " + filterStr) + return q + } + f := filter{ + FieldName: strings.TrimRight(filterStr, " ><=!"), + Value: value, + } + switch op := strings.TrimSpace(filterStr[len(f.FieldName):]); op { + case "<=": + f.Op = lessEq + case ">=": + f.Op = greaterEq + case "<": + f.Op = lessThan + case ">": + f.Op = greaterThan + case "=": + f.Op = equal + default: + q.err = fmt.Errorf("datastore: invalid operator %q in filter %q", op, filterStr) + return q + } + q.filter = append(q.filter, f) + return q +} + +// Order returns a derivative query with a field-based sort order. Orders are +// applied in the order they are added. The default order is ascending; to sort +// in descending order prefix the fieldName with a minus sign (-). +func (q *Query) Order(fieldName string) *Query { + q = q.clone() + fieldName = strings.TrimSpace(fieldName) + o := order{ + Direction: ascending, + FieldName: fieldName, + } + if strings.HasPrefix(fieldName, "-") { + o.Direction = descending + o.FieldName = strings.TrimSpace(fieldName[1:]) + } else if strings.HasPrefix(fieldName, "+") { + q.err = fmt.Errorf("datastore: invalid order: %q", fieldName) + return q + } + if len(o.FieldName) == 0 { + q.err = errors.New("datastore: empty order") + return q + } + q.order = append(q.order, o) + return q +} + +// Project returns a derivative query that yields only the given fields. It +// cannot be used with KeysOnly. +func (q *Query) Project(fieldNames ...string) *Query { + q = q.clone() + q.projection = append([]string(nil), fieldNames...) + return q +} + +// Distinct returns a derivative query that yields de-duplicated entities with +// respect to the set of projected fields. It is only used for projection +// queries. Distinct cannot be used with DistinctOn. +func (q *Query) Distinct() *Query { + q = q.clone() + q.distinct = true + return q +} + +// DistinctOn returns a derivative query that yields de-duplicated entities with +// respect to the set of the specified fields. It is only used for projection +// queries. The field list should be a subset of the projected field list. +// DistinctOn cannot be used with Distinct. +func (q *Query) DistinctOn(fieldNames ...string) *Query { + q = q.clone() + q.distinctOn = fieldNames + return q +} + +// KeysOnly returns a derivative query that yields only keys, not keys and +// entities. It cannot be used with projection queries. +func (q *Query) KeysOnly() *Query { + q = q.clone() + q.keysOnly = true + return q +} + +// Limit returns a derivative query that has a limit on the number of results +// returned. A negative value means unlimited. +func (q *Query) Limit(limit int) *Query { + q = q.clone() + if limit < math.MinInt32 || limit > math.MaxInt32 { + q.err = errors.New("datastore: query limit overflow") + return q + } + q.limit = int32(limit) + return q +} + +// Offset returns a derivative query that has an offset of how many keys to +// skip over before returning results. A negative value is invalid. +func (q *Query) Offset(offset int) *Query { + q = q.clone() + if offset < 0 { + q.err = errors.New("datastore: negative query offset") + return q + } + if offset > math.MaxInt32 { + q.err = errors.New("datastore: query offset overflow") + return q + } + q.offset = int32(offset) + return q +} + +// BatchSize returns a derivative query to fetch the supplied number of results +// at once. This value should be greater than zero, and equal to or less than +// the Limit. +func (q *Query) BatchSize(size int) *Query { + q = q.clone() + if size <= 0 || size > math.MaxInt32 { + q.err = errors.New("datastore: query batch size overflow") + return q + } + q.count = int32(size) + return q +} + +// Start returns a derivative query with the given start point. +func (q *Query) Start(c Cursor) *Query { + q = q.clone() + if c.cc == nil { + q.err = errors.New("datastore: invalid cursor") + return q + } + q.start = c.cc + return q +} + +// End returns a derivative query with the given end point. +func (q *Query) End(c Cursor) *Query { + q = q.clone() + if c.cc == nil { + q.err = errors.New("datastore: invalid cursor") + return q + } + q.end = c.cc + return q +} + +// toProto converts the query to a protocol buffer. +func (q *Query) toProto(dst *pb.Query, appID string) error { + if len(q.projection) != 0 && q.keysOnly { + return errors.New("datastore: query cannot both project and be keys-only") + } + if len(q.distinctOn) != 0 && q.distinct { + return errors.New("datastore: query cannot be both distinct and distinct-on") + } + dst.Reset() + dst.App = proto.String(appID) + if q.kind != "" { + dst.Kind = proto.String(q.kind) + } + if q.ancestor != nil { + dst.Ancestor = keyToProto(appID, q.ancestor) + if q.eventual { + dst.Strong = proto.Bool(false) + } + } + if q.projection != nil { + dst.PropertyName = q.projection + if len(q.distinctOn) != 0 { + dst.GroupByPropertyName = q.distinctOn + } + if q.distinct { + dst.GroupByPropertyName = q.projection + } + } + if q.keysOnly { + dst.KeysOnly = proto.Bool(true) + dst.RequirePerfectPlan = proto.Bool(true) + } + for _, qf := range q.filter { + if qf.FieldName == "" { + return errors.New("datastore: empty query filter field name") + } + p, errStr := valueToProto(appID, qf.FieldName, reflect.ValueOf(qf.Value), false) + if errStr != "" { + return errors.New("datastore: bad query filter value type: " + errStr) + } + xf := &pb.Query_Filter{ + Op: operatorToProto[qf.Op], + Property: []*pb.Property{p}, + } + if xf.Op == nil { + return errors.New("datastore: unknown query filter operator") + } + dst.Filter = append(dst.Filter, xf) + } + for _, qo := range q.order { + if qo.FieldName == "" { + return errors.New("datastore: empty query order field name") + } + xo := &pb.Query_Order{ + Property: proto.String(qo.FieldName), + Direction: sortDirectionToProto[qo.Direction], + } + if xo.Direction == nil { + return errors.New("datastore: unknown query order direction") + } + dst.Order = append(dst.Order, xo) + } + if q.limit >= 0 { + dst.Limit = proto.Int32(q.limit) + } + if q.offset != 0 { + dst.Offset = proto.Int32(q.offset) + } + if q.count != 0 { + dst.Count = proto.Int32(q.count) + } + dst.CompiledCursor = q.start + dst.EndCompiledCursor = q.end + dst.Compile = proto.Bool(true) + return nil +} + +// Count returns the number of results for the query. +// +// The running time and number of API calls made by Count scale linearly with +// the sum of the query's offset and limit. Unless the result count is +// expected to be small, it is best to specify a limit; otherwise Count will +// continue until it finishes counting or the provided context expires. +func (q *Query) Count(c context.Context) (int, error) { + // Check that the query is well-formed. + if q.err != nil { + return 0, q.err + } + + // Run a copy of the query, with keysOnly true (if we're not a projection, + // since the two are incompatible), and an adjusted offset. We also set the + // limit to zero, as we don't want any actual entity data, just the number + // of skipped results. + newQ := q.clone() + newQ.keysOnly = len(newQ.projection) == 0 + newQ.limit = 0 + if q.limit < 0 { + // If the original query was unlimited, set the new query's offset to maximum. + newQ.offset = math.MaxInt32 + } else { + newQ.offset = q.offset + q.limit + if newQ.offset < 0 { + // Do the best we can, in the presence of overflow. + newQ.offset = math.MaxInt32 + } + } + req := &pb.Query{} + if err := newQ.toProto(req, internal.FullyQualifiedAppID(c)); err != nil { + return 0, err + } + res := &pb.QueryResult{} + if err := internal.Call(c, "datastore_v3", "RunQuery", req, res); err != nil { + return 0, err + } + + // n is the count we will return. For example, suppose that our original + // query had an offset of 4 and a limit of 2008: the count will be 2008, + // provided that there are at least 2012 matching entities. However, the + // RPCs will only skip 1000 results at a time. The RPC sequence is: + // call RunQuery with (offset, limit) = (2012, 0) // 2012 == newQ.offset + // response has (skippedResults, moreResults) = (1000, true) + // n += 1000 // n == 1000 + // call Next with (offset, limit) = (1012, 0) // 1012 == newQ.offset - n + // response has (skippedResults, moreResults) = (1000, true) + // n += 1000 // n == 2000 + // call Next with (offset, limit) = (12, 0) // 12 == newQ.offset - n + // response has (skippedResults, moreResults) = (12, false) + // n += 12 // n == 2012 + // // exit the loop + // n -= 4 // n == 2008 + var n int32 + for { + // The QueryResult should have no actual entity data, just skipped results. + if len(res.Result) != 0 { + return 0, errors.New("datastore: internal error: Count request returned too much data") + } + n += res.GetSkippedResults() + if !res.GetMoreResults() { + break + } + if err := callNext(c, res, newQ.offset-n, q.count); err != nil { + return 0, err + } + } + n -= q.offset + if n < 0 { + // If the offset was greater than the number of matching entities, + // return 0 instead of negative. + n = 0 + } + return int(n), nil +} + +// callNext issues a datastore_v3/Next RPC to advance a cursor, such as that +// returned by a query with more results. +func callNext(c context.Context, res *pb.QueryResult, offset, count int32) error { + if res.Cursor == nil { + return errors.New("datastore: internal error: server did not return a cursor") + } + req := &pb.NextRequest{ + Cursor: res.Cursor, + } + if count >= 0 { + req.Count = proto.Int32(count) + } + if offset != 0 { + req.Offset = proto.Int32(offset) + } + if res.CompiledCursor != nil { + req.Compile = proto.Bool(true) + } + res.Reset() + return internal.Call(c, "datastore_v3", "Next", req, res) +} + +// GetAll runs the query in the given context and returns all keys that match +// that query, as well as appending the values to dst. +// +// dst must have type *[]S or *[]*S or *[]P, for some struct type S or some non- +// interface, non-pointer type P such that P or *P implements PropertyLoadSaver. +// +// As a special case, *PropertyList is an invalid type for dst, even though a +// PropertyList is a slice of structs. It is treated as invalid to avoid being +// mistakenly passed when *[]PropertyList was intended. +// +// The keys returned by GetAll will be in a 1-1 correspondence with the entities +// added to dst. +// +// If q is a ``keys-only'' query, GetAll ignores dst and only returns the keys. +// +// The running time and number of API calls made by GetAll scale linearly with +// the sum of the query's offset and limit. Unless the result count is +// expected to be small, it is best to specify a limit; otherwise GetAll will +// continue until it finishes collecting results or the provided context +// expires. +func (q *Query) GetAll(c context.Context, dst interface{}) ([]*Key, error) { + var ( + dv reflect.Value + mat multiArgType + elemType reflect.Type + errFieldMismatch error + ) + if !q.keysOnly { + dv = reflect.ValueOf(dst) + if dv.Kind() != reflect.Ptr || dv.IsNil() { + return nil, ErrInvalidEntityType + } + dv = dv.Elem() + mat, elemType = checkMultiArg(dv) + if mat == multiArgTypeInvalid || mat == multiArgTypeInterface { + return nil, ErrInvalidEntityType + } + } + + var keys []*Key + for t := q.Run(c); ; { + k, e, err := t.next() + if err == Done { + break + } + if err != nil { + return keys, err + } + if !q.keysOnly { + ev := reflect.New(elemType) + if elemType.Kind() == reflect.Map { + // This is a special case. The zero values of a map type are + // not immediately useful; they have to be make'd. + // + // Funcs and channels are similar, in that a zero value is not useful, + // but even a freshly make'd channel isn't useful: there's no fixed + // channel buffer size that is always going to be large enough, and + // there's no goroutine to drain the other end. Theoretically, these + // types could be supported, for example by sniffing for a constructor + // method or requiring prior registration, but for now it's not a + // frequent enough concern to be worth it. Programmers can work around + // it by explicitly using Iterator.Next instead of the Query.GetAll + // convenience method. + x := reflect.MakeMap(elemType) + ev.Elem().Set(x) + } + if err = loadEntity(ev.Interface(), e); err != nil { + if _, ok := err.(*ErrFieldMismatch); ok { + // We continue loading entities even in the face of field mismatch errors. + // If we encounter any other error, that other error is returned. Otherwise, + // an ErrFieldMismatch is returned. + errFieldMismatch = err + } else { + return keys, err + } + } + if mat != multiArgTypeStructPtr { + ev = ev.Elem() + } + dv.Set(reflect.Append(dv, ev)) + } + keys = append(keys, k) + } + return keys, errFieldMismatch +} + +// Run runs the query in the given context. +func (q *Query) Run(c context.Context) *Iterator { + if q.err != nil { + return &Iterator{err: q.err} + } + t := &Iterator{ + c: c, + limit: q.limit, + count: q.count, + q: q, + prevCC: q.start, + } + var req pb.Query + if err := q.toProto(&req, internal.FullyQualifiedAppID(c)); err != nil { + t.err = err + return t + } + if err := internal.Call(c, "datastore_v3", "RunQuery", &req, &t.res); err != nil { + t.err = err + return t + } + offset := q.offset - t.res.GetSkippedResults() + var count int32 + if t.count > 0 && (t.limit < 0 || t.count < t.limit) { + count = t.count + } else { + count = t.limit + } + for offset > 0 && t.res.GetMoreResults() { + t.prevCC = t.res.CompiledCursor + if err := callNext(t.c, &t.res, offset, count); err != nil { + t.err = err + break + } + skip := t.res.GetSkippedResults() + if skip < 0 { + t.err = errors.New("datastore: internal error: negative number of skipped_results") + break + } + offset -= skip + } + if offset < 0 { + t.err = errors.New("datastore: internal error: query offset was overshot") + } + return t +} + +// Iterator is the result of running a query. +type Iterator struct { + c context.Context + err error + // res is the result of the most recent RunQuery or Next API call. + res pb.QueryResult + // i is how many elements of res.Result we have iterated over. + i int + // limit is the limit on the number of results this iterator should return. + // A negative value means unlimited. + limit int32 + // count is the number of results this iterator should fetch at once. This + // should be equal to or greater than zero. + count int32 + // q is the original query which yielded this iterator. + q *Query + // prevCC is the compiled cursor that marks the end of the previous batch + // of results. + prevCC *pb.CompiledCursor +} + +// Done is returned when a query iteration has completed. +var Done = errors.New("datastore: query has no more results") + +// Next returns the key of the next result. When there are no more results, +// Done is returned as the error. +// +// If the query is not keys only and dst is non-nil, it also loads the entity +// stored for that key into the struct pointer or PropertyLoadSaver dst, with +// the same semantics and possible errors as for the Get function. +func (t *Iterator) Next(dst interface{}) (*Key, error) { + k, e, err := t.next() + if err != nil { + return nil, err + } + if dst != nil && !t.q.keysOnly { + err = loadEntity(dst, e) + } + return k, err +} + +func (t *Iterator) next() (*Key, *pb.EntityProto, error) { + if t.err != nil { + return nil, nil, t.err + } + + // Issue datastore_v3/Next RPCs as necessary. + for t.i == len(t.res.Result) { + if !t.res.GetMoreResults() { + t.err = Done + return nil, nil, t.err + } + t.prevCC = t.res.CompiledCursor + var count int32 + if t.count > 0 && (t.limit < 0 || t.count < t.limit) { + count = t.count + } else { + count = t.limit + } + if err := callNext(t.c, &t.res, 0, count); err != nil { + t.err = err + return nil, nil, t.err + } + if t.res.GetSkippedResults() != 0 { + t.err = errors.New("datastore: internal error: iterator has skipped results") + return nil, nil, t.err + } + t.i = 0 + if t.limit >= 0 { + t.limit -= int32(len(t.res.Result)) + if t.limit < 0 { + t.err = errors.New("datastore: internal error: query returned more results than the limit") + return nil, nil, t.err + } + } + } + + // Extract the key from the t.i'th element of t.res.Result. + e := t.res.Result[t.i] + t.i++ + if e.Key == nil { + return nil, nil, errors.New("datastore: internal error: server did not return a key") + } + k, err := protoToKey(e.Key) + if err != nil || k.Incomplete() { + return nil, nil, errors.New("datastore: internal error: server returned an invalid key") + } + return k, e, nil +} + +// Cursor returns a cursor for the iterator's current location. +func (t *Iterator) Cursor() (Cursor, error) { + if t.err != nil && t.err != Done { + return Cursor{}, t.err + } + // If we are at either end of the current batch of results, + // return the compiled cursor at that end. + skipped := t.res.GetSkippedResults() + if t.i == 0 && skipped == 0 { + if t.prevCC == nil { + // A nil pointer (of type *pb.CompiledCursor) means no constraint: + // passing it as the end cursor of a new query means unlimited results + // (glossing over the integer limit parameter for now). + // A non-nil pointer to an empty pb.CompiledCursor means the start: + // passing it as the end cursor of a new query means 0 results. + // If prevCC was nil, then the original query had no start cursor, but + // Iterator.Cursor should return "the start" instead of unlimited. + return Cursor{&zeroCC}, nil + } + return Cursor{t.prevCC}, nil + } + if t.i == len(t.res.Result) { + return Cursor{t.res.CompiledCursor}, nil + } + // Otherwise, re-run the query offset to this iterator's position, starting from + // the most recent compiled cursor. This is done on a best-effort basis, as it + // is racy; if a concurrent process has added or removed entities, then the + // cursor returned may be inconsistent. + q := t.q.clone() + q.start = t.prevCC + q.offset = skipped + int32(t.i) + q.limit = 0 + q.keysOnly = len(q.projection) == 0 + t1 := q.Run(t.c) + _, _, err := t1.next() + if err != Done { + if err == nil { + err = fmt.Errorf("datastore: internal error: zero-limit query did not have zero results") + } + return Cursor{}, err + } + return Cursor{t1.res.CompiledCursor}, nil +} + +var zeroCC pb.CompiledCursor + +// Cursor is an iterator's position. It can be converted to and from an opaque +// string. A cursor can be used from different HTTP requests, but only with a +// query with the same kind, ancestor, filter and order constraints. +type Cursor struct { + cc *pb.CompiledCursor +} + +// String returns a base-64 string representation of a cursor. +func (c Cursor) String() string { + if c.cc == nil { + return "" + } + b, err := proto.Marshal(c.cc) + if err != nil { + // The only way to construct a Cursor with a non-nil cc field is to + // unmarshal from the byte representation. We panic if the unmarshal + // succeeds but the marshaling of the unchanged protobuf value fails. + panic(fmt.Sprintf("datastore: internal error: malformed cursor: %v", err)) + } + return strings.TrimRight(base64.URLEncoding.EncodeToString(b), "=") +} + +// Decode decodes a cursor from its base-64 string representation. +func DecodeCursor(s string) (Cursor, error) { + if s == "" { + return Cursor{&zeroCC}, nil + } + if n := len(s) % 4; n != 0 { + s += strings.Repeat("=", 4-n) + } + b, err := base64.URLEncoding.DecodeString(s) + if err != nil { + return Cursor{}, err + } + cc := &pb.CompiledCursor{} + if err := proto.Unmarshal(b, cc); err != nil { + return Cursor{}, err + } + return Cursor{cc}, nil +} diff --git a/vendor/google.golang.org/appengine/datastore/save.go b/vendor/google.golang.org/appengine/datastore/save.go new file mode 100644 index 00000000000..7b045a59556 --- /dev/null +++ b/vendor/google.golang.org/appengine/datastore/save.go @@ -0,0 +1,333 @@ +// Copyright 2011 Google Inc. All rights reserved. +// Use of this source code is governed by the Apache 2.0 +// license that can be found in the LICENSE file. + +package datastore + +import ( + "errors" + "fmt" + "math" + "reflect" + "time" + + "github.com/golang/protobuf/proto" + + "google.golang.org/appengine" + pb "google.golang.org/appengine/internal/datastore" +) + +func toUnixMicro(t time.Time) int64 { + // We cannot use t.UnixNano() / 1e3 because we want to handle times more than + // 2^63 nanoseconds (which is about 292 years) away from 1970, and those cannot + // be represented in the numerator of a single int64 divide. + return t.Unix()*1e6 + int64(t.Nanosecond()/1e3) +} + +func fromUnixMicro(t int64) time.Time { + return time.Unix(t/1e6, (t%1e6)*1e3).UTC() +} + +var ( + minTime = time.Unix(int64(math.MinInt64)/1e6, (int64(math.MinInt64)%1e6)*1e3) + maxTime = time.Unix(int64(math.MaxInt64)/1e6, (int64(math.MaxInt64)%1e6)*1e3) +) + +// valueToProto converts a named value to a newly allocated Property. +// The returned error string is empty on success. +func valueToProto(defaultAppID, name string, v reflect.Value, multiple bool) (p *pb.Property, errStr string) { + var ( + pv pb.PropertyValue + unsupported bool + ) + switch v.Kind() { + case reflect.Invalid: + // No-op. + case reflect.Int, reflect.Int8, reflect.Int16, reflect.Int32, reflect.Int64: + pv.Int64Value = proto.Int64(v.Int()) + case reflect.Bool: + pv.BooleanValue = proto.Bool(v.Bool()) + case reflect.String: + pv.StringValue = proto.String(v.String()) + case reflect.Float32, reflect.Float64: + pv.DoubleValue = proto.Float64(v.Float()) + case reflect.Ptr: + if k, ok := v.Interface().(*Key); ok { + if k != nil { + pv.Referencevalue = keyToReferenceValue(defaultAppID, k) + } + } else { + unsupported = true + } + case reflect.Struct: + switch t := v.Interface().(type) { + case time.Time: + if t.Before(minTime) || t.After(maxTime) { + return nil, "time value out of range" + } + pv.Int64Value = proto.Int64(toUnixMicro(t)) + case appengine.GeoPoint: + if !t.Valid() { + return nil, "invalid GeoPoint value" + } + // NOTE: Strangely, latitude maps to X, longitude to Y. + pv.Pointvalue = &pb.PropertyValue_PointValue{X: &t.Lat, Y: &t.Lng} + default: + unsupported = true + } + case reflect.Slice: + if b, ok := v.Interface().([]byte); ok { + pv.StringValue = proto.String(string(b)) + } else { + // nvToProto should already catch slice values. + // If we get here, we have a slice of slice values. + unsupported = true + } + default: + unsupported = true + } + if unsupported { + return nil, "unsupported datastore value type: " + v.Type().String() + } + p = &pb.Property{ + Name: proto.String(name), + Value: &pv, + Multiple: proto.Bool(multiple), + } + if v.IsValid() { + switch v.Interface().(type) { + case []byte: + p.Meaning = pb.Property_BLOB.Enum() + case ByteString: + p.Meaning = pb.Property_BYTESTRING.Enum() + case appengine.BlobKey: + p.Meaning = pb.Property_BLOBKEY.Enum() + case time.Time: + p.Meaning = pb.Property_GD_WHEN.Enum() + case appengine.GeoPoint: + p.Meaning = pb.Property_GEORSS_POINT.Enum() + } + } + return p, "" +} + +type saveOpts struct { + noIndex bool + multiple bool + omitEmpty bool +} + +// saveEntity saves an EntityProto into a PropertyLoadSaver or struct pointer. +func saveEntity(defaultAppID string, key *Key, src interface{}) (*pb.EntityProto, error) { + var err error + var props []Property + if e, ok := src.(PropertyLoadSaver); ok { + props, err = e.Save() + } else { + props, err = SaveStruct(src) + } + if err != nil { + return nil, err + } + return propertiesToProto(defaultAppID, key, props) +} + +func saveStructProperty(props *[]Property, name string, opts saveOpts, v reflect.Value) error { + if opts.omitEmpty && isEmptyValue(v) { + return nil + } + p := Property{ + Name: name, + NoIndex: opts.noIndex, + Multiple: opts.multiple, + } + switch x := v.Interface().(type) { + case *Key: + p.Value = x + case time.Time: + p.Value = x + case appengine.BlobKey: + p.Value = x + case appengine.GeoPoint: + p.Value = x + case ByteString: + p.Value = x + default: + switch v.Kind() { + case reflect.Int, reflect.Int8, reflect.Int16, reflect.Int32, reflect.Int64: + p.Value = v.Int() + case reflect.Bool: + p.Value = v.Bool() + case reflect.String: + p.Value = v.String() + case reflect.Float32, reflect.Float64: + p.Value = v.Float() + case reflect.Slice: + if v.Type().Elem().Kind() == reflect.Uint8 { + p.NoIndex = true + p.Value = v.Bytes() + } + case reflect.Struct: + if !v.CanAddr() { + return fmt.Errorf("datastore: unsupported struct field: value is unaddressable") + } + sub, err := newStructPLS(v.Addr().Interface()) + if err != nil { + return fmt.Errorf("datastore: unsupported struct field: %v", err) + } + return sub.save(props, name+".", opts) + } + } + if p.Value == nil { + return fmt.Errorf("datastore: unsupported struct field type: %v", v.Type()) + } + *props = append(*props, p) + return nil +} + +func (s structPLS) Save() ([]Property, error) { + var props []Property + if err := s.save(&props, "", saveOpts{}); err != nil { + return nil, err + } + return props, nil +} + +func (s structPLS) save(props *[]Property, prefix string, opts saveOpts) error { + for name, f := range s.codec.fields { + name = prefix + name + v := s.v.FieldByIndex(f.path) + if !v.IsValid() || !v.CanSet() { + continue + } + var opts1 saveOpts + opts1.noIndex = opts.noIndex || f.noIndex + opts1.multiple = opts.multiple + opts1.omitEmpty = f.omitEmpty // don't propagate + // For slice fields that aren't []byte, save each element. + if v.Kind() == reflect.Slice && v.Type().Elem().Kind() != reflect.Uint8 { + opts1.multiple = true + for j := 0; j < v.Len(); j++ { + if err := saveStructProperty(props, name, opts1, v.Index(j)); err != nil { + return err + } + } + continue + } + // Otherwise, save the field itself. + if err := saveStructProperty(props, name, opts1, v); err != nil { + return err + } + } + return nil +} + +func propertiesToProto(defaultAppID string, key *Key, props []Property) (*pb.EntityProto, error) { + e := &pb.EntityProto{ + Key: keyToProto(defaultAppID, key), + } + if key.parent == nil { + e.EntityGroup = &pb.Path{} + } else { + e.EntityGroup = keyToProto(defaultAppID, key.root()).Path + } + prevMultiple := make(map[string]bool) + + for _, p := range props { + if pm, ok := prevMultiple[p.Name]; ok { + if !pm || !p.Multiple { + return nil, fmt.Errorf("datastore: multiple Properties with Name %q, but Multiple is false", p.Name) + } + } else { + prevMultiple[p.Name] = p.Multiple + } + + x := &pb.Property{ + Name: proto.String(p.Name), + Value: new(pb.PropertyValue), + Multiple: proto.Bool(p.Multiple), + } + switch v := p.Value.(type) { + case int64: + x.Value.Int64Value = proto.Int64(v) + case bool: + x.Value.BooleanValue = proto.Bool(v) + case string: + x.Value.StringValue = proto.String(v) + if p.NoIndex { + x.Meaning = pb.Property_TEXT.Enum() + } + case float64: + x.Value.DoubleValue = proto.Float64(v) + case *Key: + if v != nil { + x.Value.Referencevalue = keyToReferenceValue(defaultAppID, v) + } + case time.Time: + if v.Before(minTime) || v.After(maxTime) { + return nil, fmt.Errorf("datastore: time value out of range") + } + x.Value.Int64Value = proto.Int64(toUnixMicro(v)) + x.Meaning = pb.Property_GD_WHEN.Enum() + case appengine.BlobKey: + x.Value.StringValue = proto.String(string(v)) + x.Meaning = pb.Property_BLOBKEY.Enum() + case appengine.GeoPoint: + if !v.Valid() { + return nil, fmt.Errorf("datastore: invalid GeoPoint value") + } + // NOTE: Strangely, latitude maps to X, longitude to Y. + x.Value.Pointvalue = &pb.PropertyValue_PointValue{X: &v.Lat, Y: &v.Lng} + x.Meaning = pb.Property_GEORSS_POINT.Enum() + case []byte: + x.Value.StringValue = proto.String(string(v)) + x.Meaning = pb.Property_BLOB.Enum() + if !p.NoIndex { + return nil, fmt.Errorf("datastore: cannot index a []byte valued Property with Name %q", p.Name) + } + case ByteString: + x.Value.StringValue = proto.String(string(v)) + x.Meaning = pb.Property_BYTESTRING.Enum() + default: + if p.Value != nil { + return nil, fmt.Errorf("datastore: invalid Value type for a Property with Name %q", p.Name) + } + } + + if p.NoIndex { + e.RawProperty = append(e.RawProperty, x) + } else { + e.Property = append(e.Property, x) + if len(e.Property) > maxIndexedProperties { + return nil, errors.New("datastore: too many indexed properties") + } + } + } + return e, nil +} + +// isEmptyValue is taken from the encoding/json package in the standard library. +func isEmptyValue(v reflect.Value) bool { + switch v.Kind() { + case reflect.Array, reflect.Map, reflect.Slice, reflect.String: + // TODO(perfomance): Only reflect.String needed, other property types are not supported (copy/paste from json package) + return v.Len() == 0 + case reflect.Bool: + return !v.Bool() + case reflect.Int, reflect.Int8, reflect.Int16, reflect.Int32, reflect.Int64: + return v.Int() == 0 + case reflect.Uint, reflect.Uint8, reflect.Uint16, reflect.Uint32, reflect.Uint64, reflect.Uintptr: + // TODO(perfomance): Uint* are unsupported property types - should be removed (copy/paste from json package) + return v.Uint() == 0 + case reflect.Float32, reflect.Float64: + return v.Float() == 0 + case reflect.Interface, reflect.Ptr: + return v.IsNil() + case reflect.Struct: + switch x := v.Interface().(type) { + case time.Time: + return x.IsZero() + } + } + return false +} diff --git a/vendor/google.golang.org/appengine/datastore/transaction.go b/vendor/google.golang.org/appengine/datastore/transaction.go new file mode 100644 index 00000000000..2ae8428f856 --- /dev/null +++ b/vendor/google.golang.org/appengine/datastore/transaction.go @@ -0,0 +1,96 @@ +// Copyright 2011 Google Inc. All rights reserved. +// Use of this source code is governed by the Apache 2.0 +// license that can be found in the LICENSE file. + +package datastore + +import ( + "errors" + + "golang.org/x/net/context" + + "google.golang.org/appengine/internal" + pb "google.golang.org/appengine/internal/datastore" +) + +func init() { + internal.RegisterTransactionSetter(func(x *pb.Query, t *pb.Transaction) { + x.Transaction = t + }) + internal.RegisterTransactionSetter(func(x *pb.GetRequest, t *pb.Transaction) { + x.Transaction = t + }) + internal.RegisterTransactionSetter(func(x *pb.PutRequest, t *pb.Transaction) { + x.Transaction = t + }) + internal.RegisterTransactionSetter(func(x *pb.DeleteRequest, t *pb.Transaction) { + x.Transaction = t + }) +} + +// ErrConcurrentTransaction is returned when a transaction is rolled back due +// to a conflict with a concurrent transaction. +var ErrConcurrentTransaction = errors.New("datastore: concurrent transaction") + +// RunInTransaction runs f in a transaction. It calls f with a transaction +// context tc that f should use for all App Engine operations. +// +// If f returns nil, RunInTransaction attempts to commit the transaction, +// returning nil if it succeeds. If the commit fails due to a conflicting +// transaction, RunInTransaction retries f, each time with a new transaction +// context. It gives up and returns ErrConcurrentTransaction after three +// failed attempts. The number of attempts can be configured by specifying +// TransactionOptions.Attempts. +// +// If f returns non-nil, then any datastore changes will not be applied and +// RunInTransaction returns that same error. The function f is not retried. +// +// Note that when f returns, the transaction is not yet committed. Calling code +// must be careful not to assume that any of f's changes have been committed +// until RunInTransaction returns nil. +// +// Since f may be called multiple times, f should usually be idempotent. +// datastore.Get is not idempotent when unmarshaling slice fields. +// +// Nested transactions are not supported; c may not be a transaction context. +func RunInTransaction(c context.Context, f func(tc context.Context) error, opts *TransactionOptions) error { + xg := false + if opts != nil { + xg = opts.XG + } + readOnly := false + if opts != nil { + readOnly = opts.ReadOnly + } + attempts := 3 + if opts != nil && opts.Attempts > 0 { + attempts = opts.Attempts + } + var t *pb.Transaction + var err error + for i := 0; i < attempts; i++ { + if t, err = internal.RunTransactionOnce(c, f, xg, readOnly, t); err != internal.ErrConcurrentTransaction { + return err + } + } + return ErrConcurrentTransaction +} + +// TransactionOptions are the options for running a transaction. +type TransactionOptions struct { + // XG is whether the transaction can cross multiple entity groups. In + // comparison, a single group transaction is one where all datastore keys + // used have the same root key. Note that cross group transactions do not + // have the same behavior as single group transactions. In particular, it + // is much more likely to see partially applied transactions in different + // entity groups, in global queries. + // It is valid to set XG to true even if the transaction is within a + // single entity group. + XG bool + // Attempts controls the number of retries to perform when commits fail + // due to a conflicting transaction. If omitted, it defaults to 3. + Attempts int + // ReadOnly controls whether the transaction is a read only transaction. + // Read only transactions are potentially more efficient. + ReadOnly bool +} diff --git a/vendor/vendor.json b/vendor/vendor.json index f6550e2a8f1..25c81f3df6f 100644 --- a/vendor/vendor.json +++ b/vendor/vendor.json @@ -2869,6 +2869,18 @@ "revision": "44cc805cf13205b55f69e14bcb69867d1ae92f98", "revisionTime": "2016-08-05T00:47:13Z" }, + { + "checksumSHA1": "OSx+nvOfDkLfsBTZmsDUD2xw1gw=", + "path": "github.com/eclipse/paho.mqtt.golang", + "revision": "0d940dd29fd24f905cd16b28b1209b4977b97e1a", + "revisionTime": "2020-01-21T10:57:43Z" + }, + { + "checksumSHA1": "59nucqU3g1fgFTJiaIoxtKHYGmE=", + "path": "github.com/eclipse/paho.mqtt.golang/packets", + "revision": "0d940dd29fd24f905cd16b28b1209b4977b97e1a", + "revisionTime": "2020-01-21T10:57:43Z" + }, { "checksumSHA1": "Tt1KFInyaxjtvhGI75+crc75n/Q=", "path": "github.com/elastic/ecs/code/go/ecs", @@ -3760,6 +3772,12 @@ "revision": "d520615e531a6bf3fb69406b9eba718261285ec8", "revisionTime": "2016-12-05T14:13:22Z" }, + { + "checksumSHA1": "ajAqUByI39Sfm99F/ZNOguPP3Mk=", + "path": "github.com/gorilla/websocket", + "revision": "c3e18be99d19e6b3e8f1559eea2c161a665c4b6b", + "revisionTime": "2019-08-25T01:20:11Z" + }, { "checksumSHA1": "dF75743hHL364Dx3HKdZbBBFrpE=", "path": "github.com/grpc-ecosystem/go-grpc-prometheus", @@ -5731,6 +5749,12 @@ "version": "v0.14.0", "versionExact": "v0.14.0" }, + { + "path": "google.golang.org/api/internal/gensupport", + "revision": "02490b97dff7cfde1995bd77de808fd27053bc87", + "version": "v0.7.0", + "versionExact": "v0.7.0" + }, { "checksumSHA1": "nN+zggDyWr8HPYzwltMkzJJr1Jc=", "path": "google.golang.org/api/internal/third_party/uritemplates", @@ -5819,6 +5843,30 @@ "version": "v1.6.5", "versionExact": "v1.6.5" }, + { + "checksumSHA1": "5XkGWfndn+Th+AOFw9u7xLO8Qp8=", + "path": "google.golang.org/appengine/datastore", + "revision": "b2f4a3cf3c67576a2ee09e1fe62656a5086ce880", + "revisionTime": "2019-06-06T17:30:15Z", + "version": "v1.6.1", + "versionExact": "v1.6.1" + }, + { + "checksumSHA1": "BmZtAIDebUf0ivXIeQ33/3H4094=", + "path": "google.golang.org/appengine/datastore/internal/cloudkey", + "revision": "b2f4a3cf3c67576a2ee09e1fe62656a5086ce880", + "revisionTime": "2019-06-06T17:30:15Z", + "version": "v1.6.1", + "versionExact": "v1.6.1" + }, + { + "checksumSHA1": "mqxlap2M+qVsEvWvXZ4hOfwLVvw=", + "path": "google.golang.org/appengine/datastore/internal/cloudpb", + "revision": "b2f4a3cf3c67576a2ee09e1fe62656a5086ce880", + "revisionTime": "2019-06-06T17:30:15Z", + "version": "v1.6.1", + "versionExact": "v1.6.1" + }, { "checksumSHA1": "/R9+Y0jX9ijye8Ea6oh/RBwOOg4=", "path": "google.golang.org/appengine/internal", diff --git a/x-pack/metricbeat/module/istio/_meta/Dockerfile b/x-pack/metricbeat/module/istio/_meta/Dockerfile new file mode 100644 index 00000000000..e69de29bb2d diff --git a/x-pack/metricbeat/module/istio/docker-compose.yml b/x-pack/metricbeat/module/istio/docker-compose.yml new file mode 100644 index 00000000000..e69de29bb2d