[request] Additional Metadata stored w/ the Consumer

[harlow]

We have a few additional attributes outside of custom_id we'd like to store w/ the consumer record and then expose through headers to the downstream services.

This could be achieved w/ a custom_data map column on the consumers table:

CREATE TABLE IF NOT EXISTS consumers(
  id uuid,
  username text,
  custom_id text,
  custom_data map<text, text>, # or `text` for compatibility w/ other data stores
  created_at timestamp,
  PRIMARY KEY (id)
);

This would allow users to add a tiny bit of extra free-form metadata in a JSON-style map.

An alternative could be a separate table consumer_data where additional key-value pairs could be stored. The upside of the alternate table is it could be managed as a plugin w/ it's own migrations.

[ahmadnassri]

👍

[tyiss]

@thibaultcha does current lua-cassandra support inserting rows map data type ?
i was trying to add it as lua table or as json string with errors.

if not, this could be implemented using json string (text data type) in the custom_data field.

[thibaultcha]

Yes it does. However I think one can relatively safely assume two things from this feature: users will want to store arbitrary values (strings, numbers, arrays, other maps...) and users will want to query their consumers based on those metadata values.

Unfortunately for this use case Cassandra is not schema-less, which makes this feature nearly impossible to be implemented in an efficient way.

Doing this with a map<text, text> allows for (limited) querying support, but enforces the values type. And if we want the other piece of the cake so that the values types are flexible enough, we'll have to use a JSON field stored as text, which makes the querying part impossible.

Happy to hear about a solution.

[harlow]

Yeah that seems reasonable.

In my particular use-case I wouldn't need to query the field, so a text
field that stored raw JSON would do the trick.

With that said I think my use case may be specific enough to warrant a
customized meta-data plugin which can be compiled into Kong and not
necessarily part of the Auth plugin.
On Sun, Nov 8, 2015 at 12:12 PM yahel bahat notifications@github.com
wrote:

@thibaultcha https://github.com/thibaultCha does current lua-cassandra
support inserting rows map data type ?
i was trying to add it as lua table or as json string with errors.

if not, this could be implemented using json string (text data type) in
the custom_data field.

—
Reply to this email directly or view it on GitHub
#550 (comment).

[JnMik]

+1

That would also be really nice if we could use these metadatas in the api plugins.

something like:

name: "request-transformer"
    ensure: "present"
    attributes:
        config.add.querystring: "api-key:{remote-api-key-from-metadata}"

[schellkenig]

+1, and as mentioned it would be really nice if we can use these values in the request transformer. I don't want to query my consumers based on this values as mentioned @thibaultcha and I don't need specific types. If it is possible to use these values to transform requests that would be more than enough from my point of view.

[JnMik]

If it is possible to use these values to transform requests that would be more than enough from my point of view.

Totally agree with that, this is basic usage. We need meta on our end as well.
Is this planned for a future release or still waiting on the bench to get picked ?

[JnMik]

I need a proof of concept of a Metadata plugin that can alter request or emit values usable by the request transformer plugin. I will start working on it in the next few days because our company need it for January.

Does anyone have already a proof of concept of this or a good idea how it should be builded ?
Is there any guidelines I should follow if I would like to submit that plugin back to mashape for everyone's usage ?

I've never done Lua before but I have a good knowledge of nginx.
I'll follow documentation and try to do my best, but any input is appreciated !

Useful doc :
https://getkong.org/docs/0.9.x/plugin-development/
https://github.com/openresty/lua-nginx-module
http://streamdata.io/blog/developing-an-helloworld-kong-plugin/

Anything else ?

[Tieske]

Haven't given this a lot of thought, but how about combining it with regex support. Basically 2 types of plugins.

1. data provider;
   provides information based on user metadata for example, or has configuration options to extract data from headers, body, etc. using regex for example.
   name = "username", type = "header", id = "x-custom-username", regex = "xyz"
   This would store field "username" with a value derived from excuting regex "xyz" on header "x-custom-username".
   This could also work on the request url, query parameters, etc. anywhere you'd want to get data from.
   store all results in the ngx.ctx table (in a sub table).

2. data insertion;
   some sort of templating thing that will add/update url, headers, body, based on the values in the previously created table in ngx.ctx. This should probably work both on the request as well as on the response.

This would allow any future plugin to add stuff to that table and be able to insert/re-use that in the data-insertion part.

does that make sense, or am I talking rubbish?

[JnMik]

The #2 seems pretty much like something needed, but the #1 is not exactly how I imagined things.
Seems you want to be able to collect data from a request, if it's matching a regex, and store that data for later use. I have no scenario in mind that this feature could help me with, mind sharing yours ?

What I need to do for my needs is to be able to persist additionnal data in database for the consumers (permanently), and re-use that data in request transformation.

I'll provide an example of how I see the configuration taking form when we request the plugins configuration of an API on http://X.X.X.X:8001/apis/id/plugins in few minutes

[JnMik]

Sorry it took time, just got out of a meeting

So from my point of view, I think the json representation of the consumer-metadata plugin could look like this (a bunch of key-value data linked to consumers) :

So this could be a configuration of plugin linked to a consumer
http://dk1.jmcyr.dev.lan:8001/consumers/consumer-id/plugins

{
	"data": [{
		"api_id": "some-id",
		"id": "some-id",
		"created_at": 1477937320000,
		"enabled": true,
		"name": "consumer-metadata",
		"config": [{
			"consumer_username": "me@gmail.com", // Act as a foreign key to the consumers representation ?
			"metadata": [{
				"location": "europe",
				"third-party-api-key": "myapikey"
			}]
		}],
		"total": 1
	}]
}

And the maybe we could access that data from the request-transformer,
the configuration could look like this

And this could be the configuration of an API request-transformer plugin
http://dk1.jmcyr.dev.lan:8001/apis/some-api/plugins
{
	"data": [{
		"api_id": "some-id",
		"id": "some-id",
		"created_at": 1477937310000,
		"enabled": true,
		"name": "request-transformer",
		"config": {
			"add": {
				"querystring": [
				    "location:%consumer_metadata.location%",
				    "third-party-api-key:%consumer_metadata.third-party-api-key%"
				],
				"body": {},
				"headers": {}
			}
		}
	}],
	"total": 1
}

Again, this is the way I see it at the moment, so I can accomplish what I need to do for my employer needs. I'm open to new suggestions and comments if you think you have a better approach.

[Tieske]

@JnMik the request-transformer-plugin won't have access to the config of the metadata-plugin.

Hence you must use the metadata-plugin to store the relevant data in the ngx.ctx table. This is exactly what I meant by nr. 1 above, but just generic. There could be multiple nr. 1's and multiple nr. 2's for my part. Your consumer-metadata plugin would just be another one for nr. 1 (just my examples were about extracting data from the request, where yours is just static configured data for the consumer)

[JnMik]

I see, so plugins cannot interact with other plugins.

So let's consider the plugin config should have the two configuration blocs you mentionned (data-provisioning and data-insertion).

We should also consider data-provisioning can be feed from regex extraction or static provided values.

You mentionned having your Regex extraction storing a value derived from excuting the regex, is that a requirement for you ? Could the extraction result only be passed through the data_insertion procedure on the fly and not stored in database ? Since the extraction is mostly to get a value from the current volatile request, what's the +value of storing that data ?

Let me try another example of json and see if it is more valuable to you :

{
	"data": [{
		"api_id": "some-id",
		"id": "some-id",
		"created_at": 1477937320000,
		"enabled": true,
		"name": "metadata",
		"config": [{
			"data_provisioning": [{
				"type": "static-consumer-metadata",
				"config": [{
					"consumer_username": "me@gmail.com",
					"metadata": [{
						"location": "europe",
						"third-party-api-key": "myapikey"
					}]
				}]
			}, {
				"type": "regex-extraction-metadata",
				"config": [{
					"name": "value_extracted_from_username",
					"target_type": "header",
					"target_name": "x-custom-username",
					"regex": "[some-regex-extracting-a-value]"
				}]
			}],
			"data_insertion": {
				"add": {
					"querystring": [
						"location: %static-consumer-metadata.location%",
						"third-party-api-key: %static-consumer-metadata.third-party-api-key%",
						"value-extracted: %regex-extraction-metadata.value_extracted_from_username%"
					],
					"body": {},
					"headers": {}
				}
			}

		}],
		"total": 1
	}]
}

Waiting for your input on this one :)

[JnMik]

@Tieske Hello Again!
I've been working on my metadata plugin POC a bit.

So far I managed to make kong-vagrant works, create a plugin with some dummy data and build unit test for it.

So I'm starting working on the real deal, and I realize there is stuff that I could re-use from the request-transformer plugin. Functions like transform_querystrings() at least.
Should I extract it from the request-transformer plugin and move it to another folder in a separate file to avoid code duplication ? Or Maybe you want to keep the core of Kong simple without too much utilities ?

If you want me to move it, do you have an idea for naming the thing ? (request_transform.lua, querystring_transform.lua, or create some kind of querystring_factory.lua handling array of args in constructor ?)

Being new to Lua and to this project, I'm not confortable moving stuff around without asking lol

Edit: Maybe moving it around would imply having to do some refactor to for a better decoupling.
For my POC anyway I think I'll just push it in something like plugin/metadata/querystring_factory.lua and change the code a bit for my needs.

[Tieske]

Could the extraction result only be passed through the data_insertion procedure on the fly and not stored in database ?

I mentioned the ngx.ctx table. I think there is some confusion on your part. As a 'table' is a Lua datastructure (the only complex type Lua has), so that is what I was referring to, not to a database table.

The specific table ngx.ctx is a table that OpenResty creates and passes along to all request phases. So in OpenResty, if you want to pass data from the access phase to the log phase, you'd store it in the ngx.ctx table. And all of this is in memory, without databases.

see also; https://github.com/openresty/lua-nginx-module#ngxctx

hth

[Tieske]

Should I extract it from the request-transformer plugin and move it to another folder in a separate file to avoid code duplication ?

For now, just copy it, but mark it with a comment that is is a copy and we'll see later about refactoring it.

Additionally on your json snippet; see my remark above on the ngx.ctx table. It is just in memory, and per request. You'll have to use it anyway, so I'd still be in favour of two plugins, one for extraction, another for insertion. It would also make each plugin simpler on its own, instead of one very complex plugin.

[JnMik]

I mentioned the ngx.ctx table. I think there is some confusion on your part. As a 'table' is a Lua datastructure (the only complex type Lua has), so that is what I was referring to, not to a database table.

Oh yes, I have discovered about Lua table lol xD
Sorry I didn't knew at that time.

I'll see what I can do, inspiring myself from other plugins.
I'll probably get back to you with further questions

thx !

[JnMik]

@Tieske I would like your advise regarding some business logic,

let's say I have 2 consumers,
the first one have some metadata country stored in database and the second doesn't have it.

Both users try to query some API route which try to forward that metadata field to the micro service behind.

What should happend to the second user ?

#1 Request is forwarded with an empty value in the metadata field
#2 Request is blocked, returning a 401 bad request "Required metadata field : Country"
#3 ?

Thx buddy !

P.S I'm currently building all the api.lua / Daos / Hooks facets, starting to enjoy it ;)

[JnMik]

I have chosen the scenario #2 for now

[DWboutin]

Yeah, it would be useful

[danilobreda]

Today theres no way to pass a metadata when creating a consumer?

[Tieske]

@danilobreda only the custom_id field, tags have been planned but not here yet.

[guanlan]

It's supported in Kong 1.1

[IzioDev]

any doc référence would be usefull

[Tieske]

@izio38 use tags, see https://docs.konghq.com/1.2.x/admin-api/#tags

[chienhsingwu]

@guanlan, what happened to this metadata support feature? Was it completed?

[Tieske]

@chienhsingwu see the tags above

[chienhsingwu]

@Tieske, I am not sure how tags can be use for meta data, which are name-value pairs, while tags are just names.

[Tieske]

it's a different form, but both are metadata. If you want name-value pairs; don't hold your breath, I haven't seen anything like that in the works.

[chienhsingwu]

Thanks @Tieske. I guess the only option is to implement one that just fit my purpose specifically?