Convert JSON or Dict to client objects

[remram44]

See also #340 (comment) (was closed for inactivity, NOT resolved)
See also #63

Motivation
It is sometimes useful to be able to load Python objects from JSON, for example to read it from a config file. It should be possible to get Python objects from JSON, to pass around the code in a type-safe manner, for modification and ultimately sending through the client.

Feature request
There should be functions to convert JSON to objects and objects to JSON. This code already exists (it's required to talk to the API server), it should be exposed.

Workaround?
Right now, you can put this raw JSON into Python objects' fields, and the API client will apparently accept this, however if you use such mixed objects your code will probably break since their attribute names are different (API uses PascalCase, Python objects use snake_case). Even if taking care to handle that, you will reach the point where you have to handle things coming from the API client (Python objects using snake_case) differently from what you load from config files (JSON using PascalCase). Really messy.

Right now I am left considering dropping this lib altogether and issuing requests directly, since the point of the wrapper is to expose safe typed Python classes instead of the raw JSON, but this doesn't work if you use any kind of config file.

Current status
It is somewhat possible to convert from a Python object to JSON using client.api_client.sanitize_for_serialization(), however converting from JSON to Python is completely impossible; the deserialize() method that exists expects a requests.Response, not JSON. Calling client.api_client._ApiClient__deserialize() works, but that is a private method, and you need to pass it the kind read from the JSON yourself.

Apologies for duplicating #340 but I cannot re-open it myself.

[yliaog]

/assign @roycaihw

[chekolyn]

I was just trying to do this. This would be a useful function.

[yliaog]

@chekolyn could you take this issue, and work on a pr?

[chekolyn]

@yliaog sure, I'll give it a shot. i think the deserializer() might have been auto generated by swagger codegen. So where would this be appropriate? utils?

[micw523]

Yes that looks like generated code. I think utils should be fine, but somewhere in the base repo may work as well since we don’t necessarily require e2e tests on this one (do we?)

[chekolyn]

I've been doing some tests and using a similar approach to utils.create_from_yaml to infer the object response_type to use in the already generated codegen deserialize function.

Using this approach is possible to deserialize single objects and a list of multiple items to a list of individual namespaced objects; however this not covered all the different deserialize options we have.
For example we have:

• list_xxx_for_all_namespaces
• list_namespaced_xxx

My question is is there a need to deserialize to these type of objects? Or deserializing to single objects covers the use case? One complication of this is when the is a List with multiple kind of objects (something like json output of kubectl get all -o json)

@remram44 will the proposed solution work for you?
Anyone else has any opinions or suggestions ?

[remram44]

I'm not sure what you mean, what kind of lists wouldn't be deserializable?

[chekolyn]

Is really more about the object we output and not the input.
And almost every object has a list object as well.

For example:

pods = core_v1.list_pod_for_all_namespaces()
type(pods)
kubernetes.client.models.v1_pod_list.V1PodList

type(pods.items[0])
kubernetes.client.models.v1_pod.V1Pod

So in the proposed solution we would output a native python list/array that has individually deserialize objects.

Something like this:

objs = deserialize.load_from_json('/Users/user/all.ns.data.json')

for obj in objs:
  print(type(obj))
<class 'kubernetes.client.models.v1_pod.V1Pod'>
<class 'kubernetes.client.models.v1_pod.V1Pod'>
<class 'kubernetes.client.models.v1_pod.V1Pod'>
<class 'kubernetes.client.models.v1_pod.V1Pod'>
<class 'kubernetes.client.models.v1_pod.V1Pod'>
<class 'kubernetes.client.models.v1_pod.V1Pod'>
<class 'kubernetes.client.models.v1_service.V1Service'>
<class 'kubernetes.client.models.v1_service.V1Service'>
<class 'kubernetes.client.models.v1_service.V1Service'>
<class 'kubernetes.client.models.v1_service.V1Service'>
<class 'kubernetes.client.models.v1_daemon_set.V1DaemonSet'>
<class 'kubernetes.client.models.v1_deployment.V1Deployment'>
<class 'kubernetes.client.models.v1_deployment.V1Deployment'>
<class 'kubernetes.client.models.v1_replica_set.V1ReplicaSet'>
<class 'kubernetes.client.models.v1_replica_set.V1ReplicaSet'>
<class 'kubernetes.client.models.v1_replica_set.V1ReplicaSet'>

type(objs)
list

The alternative is to separate the list items by kind and return another
native python list but instead of individual objects return k8s list objects that would be accesible via the each obj.items

kubernetes.client.models.v1_pod_list.V1PodList
kubernetes.client.models.v1_service_list.V1ServiceList
etc

The reason I mentioned this is that they way we would iterate over the items would be different.
I'm inclined to take the first approach since it's less complicated, but was wondering if anyone needs to deserialize to these other k8s List objects.

[davlum]

I see there is a PR already open for this functionality. I'm not sure if the discussion is better had here or on the PR directly. There has been talk over on the apache-airflow repo of how useful the private deserialize methods would be exposed as public. It seems to me that the way the private method approaches this is the simplest, leaving it to the caller to also pass in the kind. Are there not ambiguities when trying to infer the kind? For example if I have a json { "name": "source" } this could be inferred as a V1SecretEnvSource or a V1ConfigMapEnvSource

[chekolyn]

@davlum if I understand correctly, you would like to pass the object type directly; perhaps we can pass it along and not infer it, basically skip this line: https://github.com/chekolyn/python/blob/json_deserialize/kubernetes/utils/deserialize.py#L252
and pass the value directly.

Additionally, we look for more than just the kind to infer an object. We look for the api version and api group.

Do you have any json/yaml examples I can test with?

[davlum]

Yeah, I'm essentially looking for the functionality of __deserialize_model(self, data, klass): but made public. Because you have to pass the model and the models include the API version number, the API version is explicit as well.

__deserialize_model({ "name": "source" }, V1ConfigMapEnvSource)

[chekolyn]

@davlum i think a klaas/response_type str parameter in here https://github.com/chekolyn/python/blob/json_deserialize/kubernetes/utils/deserialize.py#L238 would do pass the klass to the __deserialize_model via this line:

python/kubernetes/client/api_client.py

Line 289 in ca4f311

return self.__deserialize_model(data, klass)

I will amend/add this tonight. This is still work in progress and I thank you for the suggestion.

[remram44]

Passing in the kind is acceptable for my use case, I always know what kind of object I am trying to deserialize.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[remram44]

/remove-lifecycle stale

[davlum]

Any mouvement on this?

[marcellodesales]

Alright, I saw that #574 (comment) works... It's awkward, but works...

[marcellodesales]

@davlum try the method at #574 (comment) works... I needed to clone objects and change the names and properties. So I had it transformed into a dict, worked on that lavel, and then converted to yaml... That's how far I got...

https://stackoverflow.com/questions/59977058/clone-kuberenetes-objects-programmatically-python/59977059#59977059

[davlum]

That's not quite what I'm looking for. I would like to go from JSON/YAML to client objects defined within this library. If I'm not mistaken your example returns JSON from the client, modifies it and writes back YAML, never making the conversion from YAML -> client object.

[roycaihw]

the deserialize() method that exists expects a requests.Response, not JSON

Reading deserialize, it mostly needs the data from a RESTResponse. The method could be reused for JSON by mocking a RESTResponse object, like what @chekolyn did in https://github.com/kubernetes-client/python/pull/989/files#diff-0204f9aa3f2d80c3fe32d62f37882dfaR256-R263

[piroszhog]

That's how I have to do it now:

class FakeKubeResponse:
    def __init__(self, obj):
        import json
        self.data = json.dumps(obj)

fake_kube_response = FakeKubeResponse(initial_object_json)
v1pod = api_client.deserialize(fake_kube_response, 'V1Pod')

Will be glad to remove this hack after this issue close.

[roycaihw]

@piroszhog I think that is the right workaround at the moment, and this issue should be closed once #989 is finished.

[postmaxin]

That's how I have to do it now:

class FakeKubeResponse:
    def __init__(self, obj):
        import json
        self.data = json.dumps(obj)

fake_kube_response = FakeKubeResponse(initial_object_json)
v1pod = api_client.deserialize(fake_kube_response, 'V1Pod')

Will be glad to remove this hack after this issue close.

To make this hack work for me, I had to add explicit datetime deserialization. I'm not sure but I think this may be because I'm also doing funky stuff like deepcopies in my code;

[...]
            def _time_dumps(val):
                if isinstance(val, datetime.datetime):
                    return iso8601time(val)
                else:
                    raise TypeError(
                        "Don't know how to dump {}".format(type(val)))
            self.data = json.dumps(body, default=_time_dumps)
[...]

[pigletfly]

the origin PR #989 is closed, so can we reopen this issue?

[pigletfly]

/reopen

[k8s-ci-robot]

@pigletfly: You can't reopen an issue/PR unless you authored it or you are a collaborator.

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[pigletfly]

@roycaihw

[Don-Burns]

👍 if this could be reopened. Would be very useful

[remram44]

Open another duplicate, it's what I did. If they'd rather create all this noise and duplication by using stupid bots, we must play along.

[yliaog]

/reopen

[k8s-ci-robot]

@yliaog: Reopened this issue.

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[msw-kialo]

/remove-lifecycle rotten

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle rotten
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[postmaxin]

/remove-lifecycle rotten

[shouhanzen]

Hi, I'm interested in contributing to this issue. How can I get started?

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle rotten
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[remram44]

The project might be rotten but the issue is fresh

/remove-lifecycle stale

[remram44]

Sorry if I didn't use the right incantation to scare away the bots

/remove-lifecycle rotten

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale