In-Place Update of Pod Resources #1287

vinaykul · 2019-10-08T04:46:18Z

Enhancement Description

Please to keep this description up to date. This will help the Enhancement Team track efficiently the evolution of the enhancement

~~Identify CRI changes needed for UpdateContainerResources API, define response message for UpdateContainerResources~~
- ~~Extend UpdateContainerResources API to return info such as ‘not supported’, ‘not enough memory’, ‘successful’, ‘pending page evictions’ etc.~~
- ~~Define expected behavior for runtime when UpdateContainerResources is invoked. Define timeout duration of the CRI call.~~
  - Resolution: Separate KEP for CRI changes.
    - Discussed draft CRI changes with SIG-Node on Oct 22, and we agreed to do this as an incremental change outside the scope of this KEP, in a new mini-KEP. It does not block implementation of this KEP.
Define behavior when multiple containers are being resized, and UpdateContainerResources fails for one or more containers.
- One Possible solution:
  - Do not update Status.Resources.Limits if UpdateContainerResources API fails, and keep retrying until it succeeds.
~~Check with API reviewers if we can keep maps instead list of named sub-objects for ResizePolicy.~~
- After discussion with @liggitt , we are going to use list of named subobjects for extensibility.
Can we find a more intuitive name for ResizePolicy?
Can we use ResourceVersion to figure out the ordering of Pod resize requests?
~~Do we need to add back the ‘RestartPod’ resize policy? Is there a strong use-case for it?~~
- Resolution: No.
  - Discussed with SIG-Node on Oct 15th, not adding RestartPod policy for simplicity, will revisit if we encounter problems.

Alpha Feature Code Issues:
These are Items and issues discovered during code review that need further discussion and need to be addressed before Beta.

~~Can we figure out GetPodQOS differently once it is determined on pod create? See In-place Pod Vertical Scaling feature kubernetes#102884 (comment)~~
- Fixed by Perf optimization: GetPodQOS() returns persisted value of PodStatus.QOSClass, if set. kubernetes#119665
How do we deal with a pod that requests 1m/1m cpu requests/limits. See In-place Pod Vertical Scaling feature kubernetes#102884 (comment)
Add internal representation of ContainerStatus.Resources in kubeContainer. Convert it to ContainerStatus.Resources in kubelet_pods generate functions. See In-place Pod Vertical Scaling feature kubernetes#102884 (comment) and In-place Pod Vertical Scaling feature kubernetes#102884 (comment) and In-place Pod Vertical Scaling feature kubernetes#102884 (comment)
Can we get rid of resize mutex? Is there a better way to handle resize retries? See In-place Pod Vertical Scaling feature kubernetes#102884 (comment)
Can we recover from resize checkpoint store failures? See In-place Pod Vertical Scaling feature kubernetes#102884 (comment)
CRI clarification for ContainerStatus.Resources and how to handle runtimes that don't support it. See In-place Pod Vertical Scaling feature kubernetes#102884 (comment)
~~Add real values to dockershim test for ContainerStatus.Resources In-place Pod Vertical Scaling feature kubernetes#102884 (comment)~~
- Resolution: Not required due to dockershim deprecation.
~~Change PodStatus.Resources from v1.ResourceRequirements to *v1.ResourceRequirements~~
- Resolution: Fixed
Address all places in the code that has 'TODO(vinaykul)'
Current implementation does not work with node toploogy manager enabled. This limitation is not capturedi in the KEP. Add this to the release documentation for alpha, we will address this in beta. See In-place Pod Vertical Scaling feature kubernetes#102884 (comment)

vinaykul · 2019-10-08T04:47:10Z

/assign @vinaykul

jeremyrickard · 2019-10-09T14:26:31Z

👋 Hey there @vinaykul. I'm a shadow on the 1.17 Release Team, working on Enhancements. We're tracking issues for the 1.17 release and I wanted to reach out and ask we should track this (or more specifically I guess the In-Place Update of Pod Resources feature) for 1.17?

The current release schedule is:

Monday, September 23 - Release Cycle Begins
Tuesday, October 15, EOD PST - Enhancements Freeze
Thursday, November 14, EOD PST - Code Freeze
Tuesday, November 22 - Docs must be completed and reviewed
Monday, December 9 - Kubernetes 1.17.0 Released

We're only 5 days away from the Enhancements Freeze, so if you intend to graduate this capability in the 1.17 release, here are the requirements that you'll need to satisfy:

KEP must be merged in implementable state
KEP must define graduation criteria
KEP must have a test plan defined

Thanks @vinaykul

vinaykul · 2019-10-10T21:43:23Z

KEP must be merged in implementable state

KEP must define graduation criteria

KEP must have a test plan defined

Hi @jeremyrickard I'll do my best to get this KEP to implementable state by next Tuesday, but it looks like a stretch at this point - the major item is to complete API review with @thockin , and that depends on his availability.

The actual code changes are not that big. Nevertheless, the safe option would be to track this for 1.18.0 release, I'll update you by next Monday.

CC: @dashpole @derekwaynecarr @dchen1107

vinaykul · 2019-10-14T16:38:37Z

@jeremyrickard @mrbobbytables This KEP will take some more discussion - key thing is API review. It does not look like @thockin or another API reviewer is available soon. Could we please track this KEP for v1.18?
Thanks,

jeremyrickard · 2019-10-14T16:49:39Z

/milestone v1.18

vinaykul · 2019-10-22T05:25:59Z

@PatrickLang Here's a first stab at the proposed CRI change to allow UpdateContainerResources to work with Windows. Please take a look.. let's discuss in tomorrow's sig meeting

root@skibum:~/km16/staging/src/k8s.io/cri-api# git diff --cached .
diff --git a/staging/src/k8s.io/cri-api/pkg/apis/runtime/v1alpha2/api.proto b/staging/src/k8s.io/cri-api/pkg/apis/runtime/v1alpha2/api.proto
index 0290d0f..b05bb56 100644
--- a/staging/src/k8s.io/cri-api/pkg/apis/runtime/v1alpha2/api.proto
+++ b/staging/src/k8s.io/cri-api/pkg/apis/runtime/v1alpha2/api.proto
@@ -924,14 +924,33 @@ message ContainerStatusResponse {
     map<string, string> info = 2;
 }
 
+// ContainerResources holds the fields representing a container's resource limits
+message ContainerResources {
+    // Resource configuration specific to Linux container.
+    LinuxContainerResources linux = 1;
+    // Resource configuration specific to Windows container.
+    WindowsContainerResources windows = 2;
+}
+
 message UpdateContainerResourcesRequest {
     // ID of the container to update.
     string container_id = 1;
-    // Resource configuration specific to Linux containers.
+    // Resource configuration specific to Linux container.
     LinuxContainerResources linux = 2;
+    // Resource configuration specific to Windows container.
+    WindowsContainerResources windows = 3;
 }
 
-message UpdateContainerResourcesResponse {}
+message UpdateContainerResourcesResponse {
+    // ID of the container that was updated.
+    string container_id = 1;
+    // Resource configuration currently applied to the Linux container.
+    LinuxContainerResources linux = 2;
+    // Resource configuration currently applied to the Windows container.
+    WindowsContainerResources windows = 3;
+    // Error message if UpdateContainerResources fails in the runtime.
+    string error_message = 4;
+}
 
 message ExecSyncRequest {
     // ID of the container.
diff --git a/staging/src/k8s.io/cri-api/pkg/apis/services.go b/staging/src/k8s.io/cri-api/pkg/apis/services.go
index 9a22ecb..9f1d893 100644
--- a/staging/src/k8s.io/cri-api/pkg/apis/services.go
+++ b/staging/src/k8s.io/cri-api/pkg/apis/services.go
@@ -44,7 +44,7 @@ type ContainerManager interface {
        // ContainerStatus returns the status of the container.
        ContainerStatus(containerID string) (*runtimeapi.ContainerStatus, error)
        // UpdateContainerResources updates the cgroup resources for the container.
-       UpdateContainerResources(containerID string, resources *runtimeapi.LinuxContainerResources) error
+       UpdateContainerResources(containerID string, resources *runtimeapi.ContainerResources) error
        // ExecSync executes a command in the container, and returns the stdout output.
        // If command exits with a non-zero exit code, an error is returned.
        ExecSync(containerID string, cmd []string, timeout time.Duration) (stdout []byte, stderr []byte, err error)

dashpole · 2019-10-24T20:23:04Z

@vinaykul It looks like since the above PR was merged, this was removed from the API review queue. I believe you need to open a new PR that moves the state to implementable, and then add the API-review label to get it back in the queue and get a reviewer.

Edit: you should also include any other changes (e.g. windows CRI changes) required to move the feature to implementable in the PR as well.

vinaykul · 2019-10-29T00:38:18Z

@vinaykul It looks like since the above PR was merged, this was removed from the API review queue. I believe you need to open a new PR that moves the state to implementable, and then add the API-review label to get it back in the queue and get a reviewer.

Edit: you should also include any other changes (e.g. windows CRI changes) required to move the feature to implementable in the PR as well.

@dashpole Thanks!

I've started a provisional mini-KEP per our discussion last week for the CRI changes (Dawn mentioned last week that we should take that up separately). imho the CRI changes does not block the implementation of this KEP, as it is between Kubelet and runtime, and user is not affected by it.

In a second commit to the same PR, I've addressed another key issue (update api failure handling), and requested change to move primary KEP to implementable.

With this, everything is in one place, and we can use it for API review.

palnabarun · 2020-01-13T07:04:31Z

Hey there @vinaykul -- 1.18 Enhancements shadow here. I wanted to check in and see if you think this Enhancement will be graduating to alpha in 1.18?

The current release schedule is:

Monday, January 6th - Release Cycle Begins
Tuesday, January 28th EOD PST - Enhancements Freeze
Thursday, March 5th, EOD PST - Code Freeze
Monday, March 16th - Docs must be completed and reviewed
Tuesday, March 24th - Kubernetes 1.18.0 Released

To be included in the release,

The KEP PR must be merged
The KEP must be in an implementable state
The KEP must have test plans and graduation criteria.

If you would like to include this enhancement, once coding begins please list all relevant k/k PRs in this issue so they can be tracked properly. 👍

We'll be tracking enhancements here: http://bit.ly/k8s-1-18-enhancements

Thanks! :)

vinaykul · 2020-01-14T16:47:56Z

Hey there @vinaykul -- 1.18 Enhancements shadow here. I wanted to check in and see if you think this Enhancement will be graduating to alpha in 1.18?

The current release schedule is:

Monday, January 6th - Release Cycle Begins

Tuesday, January 28th EOD PST - Enhancements Freeze

Thursday, March 5th, EOD PST - Code Freeze

Monday, March 16th - Docs must be completed and reviewed

Tuesday, March 24th - Kubernetes 1.18.0 Released

To be included in the release,

The KEP PR must be merged

The KEP must be in an implementable state

The KEP must have test plans and graduation criteria.

If you would like to include this enhancement, once coding begins please list all relevant k/k PRs in this issue so they can be tracked properly. 👍

We'll be tracking enhancements here: http://bit.ly/k8s-1-18-enhancements

Thanks! :)

@palnabarun Yes, I'm planning to work towards alpha code targets for this feature in 1.18. I've updated the KEP adding test plan and graduation criteria sections that I will be reviewing with SIG-Node this week and hope to get it implementable before Jan 28. I'll update this thread if anything changes.

palnabarun · 2020-01-14T17:00:19Z

Thank you @vinaykul for the updates. :)

palnabarun · 2020-01-14T17:00:27Z

/stage alpha

palnabarun · 2020-01-14T17:00:36Z

/milestone v1.18

Jamstah · 2024-09-13T14:44:27Z

I'm interested in this feature mainly for startup resource usage reasons - one of the uses listed in the proposal.

I'm putting a comment here because I'm not sure the right place to propose such a change. Would this make sense as a PR to the KEP? Would I be raising a follow on KEP? Happy to put some work into thinking it through and making a decent proposal, but not sure of the right process :)

The proposal I have is, it would be really good if k8s handled the downsize for me automatically when the pod becomes ready - I imagine this could look something like:

spec:
  containers:
  - name: cpu-demo-ctr
    image: vish/stress
    resources:
      limits:
        cpu: "1"
      requests:
        cpu: "0.5"
    startupResources:
      limits:
        cpu: "3"
      requests:
        cpu: "1"

NicklasWallgren · 2024-09-13T14:51:27Z

I'm interested in this feature mainly for startup resource usage reasons - one of the uses listed in the proposal.

I'm putting a comment here because I'm not sure the right place to propose such a change. Would this make sense as a PR to the KEP? Would I be raising a follow on KEP? Happy to put some work into thinking it through and making a decent proposal, but not sure of the right process :)

The proposal I have is, it would be really good if k8s handled the downsize for me automatically when the pod becomes ready - I imagine this could look something like:
spec:
  containers:
  - name: cpu-demo-ctr
    image: vish/stress
    resources:
      limits:
        cpu: "1"
      requests:
        cpu: "0.5"
    startupResources:
      limits:
        cpu: "3"
      requests:
        cpu: "1"

Take a look at https://github.com/google/kube-startup-cpu-boost

haircommander · 2024-09-17T18:09:54Z

/milestone v1.32
/label lead-opted-in

impact-maker · 2024-10-01T12:20:15Z

Hello @vinaykul @haircommander 👋, Enhancements team here.

Just checking in as we approach enhancements freeze on 02:00 UTC Friday 11th October 2024 / 19:00 PDT Thursday 10th October 2024.

This enhancement is targeting for stage beta for v1.32 (correct me, if otherwise).

Here's where this enhancement currently stands:

KEP readme using the latest template has been merged into the k/enhancements repo.
KEP status is marked as implementable for latest-milestone: v1.32.
KEP readme has up-to-date graduation criteria
KEP has a production readiness review that has been completed and merged into k/enhancements. (For more information on the PRR process, check here). If your production readiness review is not completed yet, please make sure to fill the production readiness questionnaire in your KEP by the PRR Freeze deadline on Thursday, October 3rd, 2024 so that the PRR team has enough time to review your KEP.

For this KEP, we would just need to update the following:

KEP has a production readiness review that has been completed and merged into k/enhancements.

The status of this enhancement is marked as at risk for enhancement freeze. Please keep the issue description up-to-date with appropriate stages as well.

If you anticipate missing enhancements freeze, you can file an exception request in advance. Thank you!

vinaykul · 2024-10-06T23:31:39Z

This enhancement is targeting for stage beta for v1.32 (correct me, if otherwise).

@impact-maker Yes. @tallclair is driving this effort now.

dipesh-rawat · 2024-10-10T23:13:49Z

Hello @vinaykul @haircommander👋, v1.32 Enhancements team here,

Now that PR #4704 has been merged, all the KEP requirements are in place and merged into k/enhancements.

The status of this enhancement is now marked as tracked for enhancement freeze. Please keep the issue description up-to-date with appropriate stages as well. Thank you!

SergeyKanzhelev · 2024-10-10T23:32:55Z

@tallclair we will need to include handling of swap limit updates in this KEP: https://github.com/kubernetes/enhancements/tree/master/keps/sig-node/2400-node-swap#steps-to-calculate-swap-limit

If we are not handling it, we may set higher sum of swap limits than swap available if many Pods starting big and becoming smaller over time

mbianchidev · 2024-10-14T17:37:16Z

Hey hey @vinaykul @haircommander 👋 from the v1.32 Communications Team!

We'd love for you to consider writing a feature blog about your enhancement.
Some reasons why you might want to write a blog for this feature include (but are not limited to) if this introduces breaking changes, is important to our users, or has been in progress for a long time and it is graduating.

To opt-in, let us know by opening a Feature Blog placeholder PR against the website repository by 30th Oct 2024. For more information about writing a blog see the blog contribution guidelines.

Note: In your placeholder PR, use XX characters for the blog date in the front matter and file name. We will work with you on updating the PR with the publication date once we finalize the blog schedule.

hacktivist123 · 2024-10-15T15:30:30Z

Hello @vinaykul @haircommander 👋 1.32 Docs Shadow here.

Does this enhancement work planned for 1.32 require any new docs or modifications to existing docs?
If so, please follow the steps here to open a PR against the dev-1.32 branch in the k/website repo. This PR can be just a placeholder at this time and must be created before Thursday, October 24th 2024 18:00 PDT.

Also, take a look at Documenting for a release to get yourself familiarize with the docs requirement for the release.
Thank you!

hacktivist123 · 2024-10-18T12:17:27Z

Hello @vinaykul @haircommander 👋 1.32 Docs Shadow here.

This is just a reminder to open a placeholder PR against dev-1.32 branch in the k/website repo for this (steps available here) for this KEP if it requires new or modifications to existing docs:

The deadline for this is Thursday, Oct 24 at 18:00 PDT.
Thanks! 🚀

mbianchidev · 2024-10-28T09:48:32Z

Hey hey @vinaykul @haircommander 👋 from the v1.32 Communications Team!

We'd love for you to consider writing a feature blog about your enhancement. Some reasons why you might want to write a blog for this feature include (but are not limited to) if this introduces breaking changes, is important to our users, or has been in progress for a long time and it is graduating.

To opt-in, let us know by opening a Feature Blog placeholder PR against the website repository by 30th Oct 2024. For more information about writing a blog see the blog contribution guidelines.

Note: In your placeholder PR, use XX characters for the blog date in the front matter and file name. We will work with you on updating the PR with the publication date once we finalize the blog schedule.

Just a reminder since the blog opt-in deadline is so close!

tallclair · 2024-10-28T16:17:19Z

Blog placeholder: kubernetes/website#48576

tjons · 2024-11-04T01:18:52Z

Hey again @tallclair 👋 v1.32 Enhancements team here,

Just checking in as we approach code freeze at 02:00 UTC Friday 8th November 2024 / 19:00 PDT Thursday 7th November 2024 .

Here's where this enhancement currently stands:

All PRs to the Kubernetes repo that are related to your enhancement are linked in the above issue description (for tracking purposes).
All PR/s are ready to be merged (they have approved and lgtm labels applied) by the code freeze deadline. This includes tests.

For this enhancement, it looks like the following PRs are open and need to be merged before code freeze (and we need to update the Issue description to include all the related PRs of this KEP):

Additionally, please let me know if there are any other PRs in k/k not listed in the description or not linked with this GitHub issue that we should track for this KEP, so that we can maintain accurate status.

The status of this enhancement is marked as at risk for code freeze.

If you anticipate missing code freeze, you can file an exception request in advance. Thank you!

tallclair · 2024-11-04T18:00:27Z

Thanks @tjons. Yes, I agree this is at risk for code freeze. We have a separate tracking board for this feature here: https://github.com/orgs/kubernetes/projects/178/views/2, and there are quite a few more PRs that need to merge by Thursday. I will continue to push on these, but there's a good chance we miss the deadline. With so many PRs, it will be easier for me to add them after the fact to the PR description.

tjons · 2024-11-08T02:27:41Z

Hello @tallclair 👋 Enhancements team here,

Unfortunately, the implementation (code related) PR(s) associated with this enhancement is not in the merge-ready state by code-freeze and hence this enhancement is now removed from the 1.32 milestone.

If you still wish to progress this enhancement in 1.32, please file an exception request as soon as possible, within three days. If you have any questions, you can reach out in the #release-enhancements channel on Slack and we'll be happy to help. Thanks!

/milestone clear

sreeram-venkitesh · 2024-11-08T04:47:06Z

Exception has been filed.

fsmunoz · 2024-11-11T18:00:32Z

The v1.32 Release Team is APPROVING this Code Freeze exception request. The updated deadline is 19:00 PDT Tuesday, 12th November 2024.
cc @tjons

/milestone v1.32

k8s-ci-robot added the needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. label Oct 8, 2019

k8s-ci-robot assigned vinaykul Oct 8, 2019

vinaykul mentioned this issue Oct 8, 2019

KEP: in-place update of pod resources #686

Merged

mrbobbytables added sig/autoscaling Categorizes an issue or PR as relevant to SIG Autoscaling. sig/node Categorizes an issue or PR as relevant to SIG Node. labels Oct 14, 2019

k8s-ci-robot removed needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Oct 14, 2019

mrbobbytables added this to the v1.17 milestone Oct 14, 2019

k8s-ci-robot modified the milestones: v1.17, v1.18 Oct 14, 2019

jeremyrickard added tracked/no Denotes an enhancement issue is NOT actively being tracked by the Release Team and removed tracked/yes Denotes an enhancement issue is actively being tracked by the Release Team labels Oct 14, 2019

k8s-ci-robot added the stage/alpha Denotes an issue tracking an enhancement targeted for Alpha status label Jan 14, 2020

palnabarun removed the tracked/no Denotes an enhancement issue is NOT actively being tracked by the Release Team label Jan 14, 2020

Shubham82 mentioned this issue Jul 29, 2024

Request for release timeline of "Resize CPU and Memory Resources assigned to Containers". kubernetes/kubernetes#126433

Closed

tallclair self-assigned this Sep 4, 2024

This was referenced Sep 9, 2024

[FG:InPlacePodVerticalScaling] K8s fails to throttle pod so that it follows resized CPU limit kubernetes/kubernetes#118371

Closed

[FG:InPlacePodVerticalScaling] Fix order of resizing pod cgroups in doPodResizeAction() kubernetes/kubernetes#125708

Merged

esotsal mentioned this issue Sep 10, 2024

[FG:InPlacePodVerticalScaling] Static CPU management policy alongside InPlacePodVerticalScaling kubernetes/kubernetes#127262

Open

k8s-ci-robot added the lead-opted-in Denotes that an issue has been opted in to a release label Sep 17, 2024

pacoxu mentioned this issue Sep 20, 2024

[FG:InPlacePodVerticalScaling] kubelet: Add VPA start, finished, and failure events, while optimizing vpa errors kubernetes/kubernetes#127479

Open

shanshanying mentioned this issue Sep 26, 2024

[Improvement] add tests for InPlacePodVerticalScaling to test pipline. apecloud/kubeblocks#8216

Open

tallclair mentioned this issue Oct 22, 2024

[PLACEHOLDER] In-Place Pod Resize v1.32 (beta) kubernetes/website#48503

Draft

tallclair mentioned this issue Oct 28, 2024

[PLACEHOLDER] In-Place pod resize beta blog post kubernetes/website#48576

Draft

tallclair mentioned this issue Nov 7, 2024

[FG:InPlacePodVerticalScaling] Graduate to Beta kubernetes/kubernetes#128682

Merged

8 tasks

k8s-ci-robot removed this from the v1.32 milestone Nov 8, 2024

k8s-ci-robot added this to the v1.32 milestone Nov 11, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

In-Place Update of Pod Resources #1287

In-Place Update of Pod Resources #1287

vinaykul commented Oct 8, 2019 •

edited by pacoxu

Loading

vinaykul commented Oct 8, 2019

jeremyrickard commented Oct 9, 2019 •

edited

Loading

vinaykul commented Oct 10, 2019

vinaykul commented Oct 14, 2019

jeremyrickard commented Oct 14, 2019

vinaykul commented Oct 22, 2019

dashpole commented Oct 24, 2019 •

edited

Loading

vinaykul commented Oct 29, 2019

palnabarun commented Jan 13, 2020 •

edited

Loading

vinaykul commented Jan 14, 2020

palnabarun commented Jan 14, 2020

palnabarun commented Jan 14, 2020

palnabarun commented Jan 14, 2020

Jamstah commented Sep 13, 2024

NicklasWallgren commented Sep 13, 2024

haircommander commented Sep 17, 2024

impact-maker commented Oct 1, 2024 •

edited

Loading

vinaykul commented Oct 6, 2024

dipesh-rawat commented Oct 10, 2024

SergeyKanzhelev commented Oct 10, 2024 •

edited

Loading

mbianchidev commented Oct 14, 2024

hacktivist123 commented Oct 15, 2024

hacktivist123 commented Oct 18, 2024

mbianchidev commented Oct 28, 2024

tallclair commented Oct 28, 2024

tjons commented Nov 4, 2024

tallclair commented Nov 4, 2024

tjons commented Nov 8, 2024

sreeram-venkitesh commented Nov 8, 2024

fsmunoz commented Nov 11, 2024

In-Place Update of Pod Resources #1287

In-Place Update of Pod Resources #1287

Comments

vinaykul commented Oct 8, 2019 • edited by pacoxu Loading

Enhancement Description

Please to keep this description up to date. This will help the Enhancement Team track efficiently the evolution of the enhancement

vinaykul commented Oct 8, 2019

jeremyrickard commented Oct 9, 2019 • edited Loading

vinaykul commented Oct 10, 2019

vinaykul commented Oct 14, 2019

jeremyrickard commented Oct 14, 2019

vinaykul commented Oct 22, 2019

dashpole commented Oct 24, 2019 • edited Loading

vinaykul commented Oct 29, 2019

palnabarun commented Jan 13, 2020 • edited Loading

vinaykul commented Jan 14, 2020

palnabarun commented Jan 14, 2020

palnabarun commented Jan 14, 2020

palnabarun commented Jan 14, 2020

Jamstah commented Sep 13, 2024

NicklasWallgren commented Sep 13, 2024

haircommander commented Sep 17, 2024

impact-maker commented Oct 1, 2024 • edited Loading

vinaykul commented Oct 6, 2024

dipesh-rawat commented Oct 10, 2024

SergeyKanzhelev commented Oct 10, 2024 • edited Loading

mbianchidev commented Oct 14, 2024

hacktivist123 commented Oct 15, 2024

hacktivist123 commented Oct 18, 2024

mbianchidev commented Oct 28, 2024

tallclair commented Oct 28, 2024

tjons commented Nov 4, 2024

tallclair commented Nov 4, 2024

tjons commented Nov 8, 2024

sreeram-venkitesh commented Nov 8, 2024

fsmunoz commented Nov 11, 2024

vinaykul commented Oct 8, 2019 •

edited by pacoxu

Loading

jeremyrickard commented Oct 9, 2019 •

edited

Loading

dashpole commented Oct 24, 2019 •

edited

Loading

palnabarun commented Jan 13, 2020 •

edited

Loading

impact-maker commented Oct 1, 2024 •

edited

Loading

SergeyKanzhelev commented Oct 10, 2024 •

edited

Loading