Sample scripts to create an image with an applicaiton running on a ma… #401

aseembajaj · 2017-05-31T22:11:55Z

…naged server running in MSI mode (Managed Server Independence mode)

Djelibeybi

There are some changes that need to be made to make the resulting image more efficient.

Djelibeybi · 2017-06-01T21:01:14Z

OracleWebLogic/samples/12212-msiserver/Dockerfile

+
+# Environment variables required for this build (do NOT change)
+# -------------------------------------------------------------
+ENV MW_HOME="$ORACLE_HOME"


These three ENV directives should be combined into a single directive/layer.

Djelibeybi · 2017-06-01T21:01:31Z

OracleWebLogic/samples/12212-msiserver/Dockerfile

+# Copy scripts
+# --------------------------------
+USER oracle
+COPY container-scripts/provision-domain-for-msi.py /u01/oracle/


These three COPY directives should be combined into a single directive/layer.

Djelibeybi · 2017-06-01T21:02:07Z

OracleWebLogic/samples/12212-msiserver/Dockerfile

+
+# Scripts and parameters for launching
+# -----------------------------------
+COPY container-scripts/launcher.sh /u01/oracle/


This COPY directive should be included in the previous COPY so that it's contained in the same layer.

Djelibeybi · 2017-06-01T21:03:54Z

OracleWebLogic/samples/12212-msiserver/Dockerfile

+
+EXPOSE 8011
+WORKDIR $ORACLE_HOME/wlserver/samples/domains/msi-sample
+ENTRYPOINT "/usr/bin/bash" "/u01/oracle/launcher.sh" "$ORACLE_HOME/wlserver/samples/domains/msi-sample" "ms1" "weblogic" "weblogic1" $number_of_ms


Please do not use bash as the ENTRYPOINT. You should be calling launcher.sh and providing the command-line parameters via a CMD directive. You should also switch this to use the JSON representation.

https://docs.docker.com/engine/reference/builder/#understand-how-cmd-and-entrypoint-interact

Hi Avi. I have removed the use of bash as entry point. But I need to use shell form of Entrypoint because I need variable replacement that is otherwise unavailable in exec form. So, I can't split this into ENTRYPOINT and CMD. Nor can I switch to JSON form. Using either of these causes variable replacement to stop working.

Your launcher.sh should just be able to read the environment variable as you're setting it higher in the Dockerfile. You don't need to pass it as a parameter here.

Yes, you are right. So, I'll be making these changes for this particular suggestion that you made:

No use of bash, invoke launcher.sh directly

Use JSON format

No parameter needed, so no need to specify CMD

Here is what the updated directive will look like:

ENTRYPOINT ["/u01/oracle/launcher.sh"]

Djelibeybi · 2017-06-01T21:04:21Z

OracleWebLogic/samples/12212-msiserver/Dockerfile.addapp

+
+# Arguments
+# ---------
+ARG name


These three ARG directives should be combined into a single directive/layer.

Hi Avi. I can't seem to find an ARG directive that can be used to declare multiple arguments:
https://docs.docker.com/engine/reference/builder/#arg
Can you please point me to it? Thanks

Sorry, you're right. Was getting confused with ENV directives.

Should these ARG directives not get default values then?

Yes, that's right Avi. These are arguments that specify what application to deploy and from where. So there can't be any defaults for these.

How can there not be defaults? What happens if these are not provided? Your launcher/entrypoint will need to handle the case of being run without any ARG values gracefully.

There are not options, these are more like required arguments. How can we default on the application directory to be added?

I can make them default to the application bundled in the sample directory (like the way it was before these arguments were introduced). What do you say?

Arguments

---------

ARG name=summercamps
ARG source=apps/summercamps.ear
ARG simple_filename=summercamps.ear

That seems fine. Or your container should throw an error and refuse to start if the required arguments are not provided. Whatever you feel is the best outcome.

Hi Avi,

I'll go with default. These are sample scripts. Customers might have to change the script anyways to add more parameters to deployment, they can choose their own defaults at that time. We can use the default based on samples bundled in this directory.

Djelibeybi · 2017-06-01T21:04:36Z

OracleWebLogic/samples/12212-msiserver/Dockerfile.addapp

+# Copy scripts
+# --------------------------------
+USER oracle
+COPY container-scripts/add-app-to-domain.py /u01/oracle/


Please combine the two COPY directives into a single directive/layer.

Djelibeybi · 2017-06-01T21:05:45Z

OracleWebLogic/samples/12212-msiserver/container-scripts/launcher.sh

@@ -0,0 +1,19 @@
+#! /usr/bin/bash


This launcher.sh script has no handlers for stop/start. You need to add handlers so that users can use docker stop effectively.

Hi Avi,

By handlers for stop, did you mean something like trapping (using trap) SIGKILL and SIGTERM and passing them on to server process?

What do you mean by a handler for start? Isn't launcher the script that will run when we start the container?

Yes, sorry. I meant handlers for SIGKILL and SIGTERM (which are what docker stop calls).

Hi Avi,

launcher.sh invokes startManagedWebLogic.sh. This script (startManagedWebLogic.sh) like startWebLogic.sh used by other docker samples is created during WebLogic domain creation. I think these scripts communicate these signals to the underlying JVM (or not at the very least not inhibit the transmission of signal to the underlying JVM). WLS running in JVM handles SIGTERM and SIGKILL correctly.

Have you tested and confirmed that the signal handlers still work as expected?

Hi Avi,

Following up on your question about behavior of docker stop with WLS, I experimented and found the following.

About docker stop

docker stop sends a SIGTERM to the main process initially and then sends a SIGKILL after some time. This is a configurable value, defaults to 10 seconds.

WebLogic java process handles SIGTERM to perform a forceShutdown() of the server, cleaning up a few things.

When we launch WebLogic using generated shell scripts, sending SIGTERM to the pid of shell script doesn’t invoke the hook mentioned in Add Coherence Dockerfiles and scripts #2

However if we find the pid of the java process (JVM that is running the server) and send a SIGTERM to it, shutdown hook mentioned in Add Coherence Dockerfiles and scripts #2 is invoked

Trying to replicate Project description #3 in Docker image is equally ineffective. More specifically we can try and capture the pid of the process for the WLS startup script being launched as a part of docker container, but trapping SIGTERM (on docker stop) and forwarding to that captured pid is ineffective (like Project description #3)

Perhaps we can replicate Reestruction of folders and scripts #4 in docker image i.e. try to trap SIGTERM and translate to a SIGTERM to all running java processes. But I am having trouble with this because we use oracle slim image in PS2 docker image and “ps” is not available.

For WLS MSI docker image, I have been using the generated WLS script called startManagedWebLogic.sh. But I believe that observations in Project description #3 and Revised work #5 will be the same if we use startWebLogic.sh for admin server in a regular WLS docker image

About docker kill

docker kill results in a SIGKILL signal. There is no way to intercept or trap it. That is also the case when we send a SIGKILL signal to a WLS server started with shell scripts (outside docker)

Is this behaviour replicated in the base WebLogic image or introduced by your addition of the launcher.sh script? If the former, please open another issue so that @mriccell can look into that. If the latter, you'll need to reimplement the handlers so that the SIGTERM is sent to the right pid.

Hi Avi,

As suspected, docker stop isn't converted to a forceShutdown request for WLS server. Here is what I tried:
docker pull container-registry.oracle.com/middleware/weblogic:12.2.1.1
docker run container-registry.oracle.com/middleware/weblogic:12.2.1.1

And then on the side I issued a docker stop request using a generated container name

docker stop epic_meitner

I can see that shutdown hook is not invoked, and the container stopped after 10 seconds (the default difference between the time SIGTERM and SIGKILL are sent to the container process)

Had it worked, I would have expected to see something like the following on stdout of the container:

<Jun 5, 2017 3:18:30,533 PM PDT>
<Jun 5, 2017 3:18:30,534 PM PDT> <Server shutdown has been requested by .>
<Jun 5, 2017 3:18:30,538 PM PDT>
<Jun 5, 2017 3:18:30,549 PM PDT>
<Jun 5, 2017 3:18:30,549 PM PDT>
<Jun 5, 2017 3:18:30,561 PM PDT>
<Jun 5, 2017 3:18:30,601 PM PDT> <JMX Connector Server stopped at service:jmx:iiop://10.147.117.136:7001/jndi/weblogic.management.mbeanservers.edit.>
<Jun 5, 2017 3:18:30,602 PM PDT> <JMX Connector Server stopped at service:jmx:iiop://10.147.117.136:7001/jndi/weblogic.management.mbeanservers.domainruntime.>
<Jun 5, 2017 3:18:30,666 PM PDT> <JMX Connector Server stopped at service:jmx:iiop://10.147.117.136:7001/jndi/weblogic.management.mbeanservers.runtime.>
Stopping Derby server...
Derby server stopped.

Can you please open a new issue for this, so that @mriccell can take a look?

…naged server running in MSI mode (Managed Server Independence mode)

…er-images into msi-mode-samples

aseembajaj · 2017-06-02T18:17:55Z

Hi Avi,

Other than the issue about stop / start, I have incorporated all your feedback and updated the pull request.

mriccell · 2017-06-02T19:59:01Z

Aseem Sorry I did not have an opportunity to review, will do that now. Thanks Monica

…

On 6/2/2017 2:18 PM, Aseem Bajaj wrote: Hi Avi, Other than the issue about stop / start, I have incorporated all your feedback and updated the pull request. — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <#401 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AKKX3NTfFoFjCpE-4YhivhyNgOK3BLJHks5sAFHYgaJpZM4NsQrs>.

Djelibeybi · 2017-06-05T20:07:20Z

I have approved this PR. Just waiting on @brunoborges and @mriccell to complete their review.

Djelibeybi · 2017-06-06T20:46:25Z

@aseembajaj I see the new commit you just added, but wouldn't this make more sense as a seperate PR for the upstream createAndStartEmptyDomain.sh script instead?

aseembajaj · 2017-06-06T21:05:13Z

Hi Avi,

This change only handles MSI server image. The changes to be made for non-MSI setups could be similar, but different.

The managed servers in that setup must have been started using NodeManager, so we have to find another way to initiate forceShutdown on those servers
The admin server in that setup could be forceShutdown in the same manner. But I would assume that we would want to make a coherent change for both cases that setup

Djelibeybi · 2017-06-06T21:06:26Z

@aseembajaj it would be great if you could work with @mriccell to see if you can come up with a more global solution that would work across all downstream images that extend the base WLS image.

aseembajaj · 2017-06-06T21:21:08Z

Sure, I'll work with Monica and give her inputs about how we can implement this so that all images benefit.

mriccell · 2017-06-14T03:55:47Z

OracleWebLogic/samples/12212-msiserver/Dockerfile

+
+EXPOSE 8011
+WORKDIR $ORACLE_HOME/wlserver/samples/domains/msi-sample
+ENTRYPOINT "/usr/bin/bash" "/u01/oracle/launcher.sh" "$ORACLE_HOME/wlserver/samples/domains/msi-sample" "ms1" "weblogic" "weblogic1" $number_of_ms


Tge RUN command provision-domain-for-msi.sh parameters such as weblogic weblogic1 8001 ms1 8011 dev $number_of_ms should be set with ENV directive in the dockerfile.

brunoborges · 2017-06-15T21:17:33Z

Can launcher.sh live on the top level image? If yes, maybe that should be a script for ENTRYPOINT

aseembajaj · 2017-06-15T22:37:08Z

Hi Bruno,

You are right in that there could be come collaboration in the startup scripts, one for the base image (that helps all extending images) and the ones specific for scenarios (like MSI). Monica and I had a meeting yesterday. Among other things, we decided that I will try to reuse code around container stop. This will involve slight change in base image too (to allow for reusability). But we still need a separate script for extending images (like MSI) because there are some other tasks that are done specific to the image.

brunoborges · 2017-07-10T22:48:07Z

@mriccell Monica, mind to follow up on this when possible?

Djelibeybi · 2017-07-13T07:01:32Z

Can you please rebase this pull request from master? Some of the files have changed since it was submitted.

aseembajaj · 2017-07-13T18:19:41Z

Hi Avi, Bruno & Monica,

Based on a meeting with Monica, we had identified 5 changes that I had to make to this. I have been unable to work on those changes as I have been involved with other work. I'll re-submit this request after I make changes.

Djelibeybi · 2017-07-13T18:49:17Z

You don't need to resubmit, just push the required changes to your branch and this PR will automatically update and notify us.

aseembajaj · 2017-07-18T02:16:11Z

Hi Avi, Bruno & Monica,

I have updated the pull request, to incorporate Monica's feedback. Here are my notes from the review:

Add a sentence or two about MSI in introduction, running in Docker
Environment variable declaration using ENV, move to all arguments
managed name prefix as argument
Readme.md : A little segment about parameters like default name and default port
Try to re-use trap command from base image

Note that for the last item, Monica asked me to comment out my "trap" logic since it is being built into the base image.

Can you please re-review and accept the pull request if you are okay with it? Thanks.

Djelibeybi

Some minor changes required, but some great improvements.

Djelibeybi · 2017-07-18T04:15:21Z

OracleWebLogic/samples/12212-msiserver/README.md

+CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                    NAMES
+cf579fd131fc        12212-msiserver     "/u01/oracle/launc..."   2 minutes ago       Up 2 minutes        0.0.0.0:8011->8011/tcp   vigilant_volhard
+
+$ docker exec -it cf579fd131fc bash


You should never need to exec bash into a container. If you need the user to know something, that log file should be sent to stdout at runtime so that the user can use docker logs (or any of the logging drivers) to get the output.

Djelibeybi · 2017-07-18T04:16:14Z

OracleWebLogic/samples/12212-msiserver/README.md

+
+$ docker exec -it cf579fd131fc bash
+
+$ cat servers/ms3/logs/ms3.log | grep BEA-150018


As above. You should be tailing this log to stdout by default. Take a look at the Database and other Dockerfiles for examples of this.

Djelibeybi · 2017-07-18T04:17:17Z

OracleWebLogic/samples/12212-msiserver/container-scripts/launcher.sh

@@ -0,0 +1,37 @@
+#! /usr/bin/bash


This should be #!/bin/bash

Djelibeybi · 2017-07-18T04:17:57Z

OracleWebLogic/samples/12212-msiserver/container-scripts/launcher.sh

+fi
+
+bin/startManagedWebLogic.sh $ms_name
+


If the rest of this script is not needed (it's all commented out) then just remove those lines.

Djelibeybi · 2017-07-18T04:19:34Z

OracleWebLogic/samples/12212-msiserver/README.md

+This Dockerfile extends the Oracle WebLogic image by creating a domain that configures a managed
+server in MSI mode (or Managed Server Independence mode). In this mode, a managed server can run
+without the need of an admin server. Such a managed server is not driven by admin server for
+configuration or deployment changes. However, it can handle all configuration and deploymnets


Typo: should be "deployments"

Djelibeybi · 2017-07-18T04:21:41Z

OracleWebLogic/samples/12212-msiserver/README.md

+Using swarm service creation with this image
+--------------------------------------------
+The image called 12212-summercamps-msiserver can be used for Docker service creation
+to scale out to multiple replicas. docker service creation can only be done


Typo: "docker" should be "Docker"

Djelibeybi · 2017-07-18T04:23:22Z

OracleWebLogic/samples/12212-msiserver/README.md

+registry you can start one locally. More details can be found here
+https://docs.docker.com/registry/deploying/
+
+Here are the three steps you'll need to start, use and push to your local registry.


Don't provide documentation on how to run a registry, because this will get out of date. You already point to the Docker docs, that's sufficient.

Also, a registry is not required for a service, if you have the same image on all the hosts in the Swarm. It's just that the simplest way for that to happen is to use a local registry. :)

Djelibeybi · 2017-07-18T04:23:56Z

OracleWebLogic/samples/12212-msiserver/README.md

+docker tag 12212-summercamps-msiserver localhost:5000/12212-summercamps-msiserver
+docker push localhost:5000/12212-summercamps-msiserver
+
+Now that your image is published to the registry, start by joining swarm, either a swarm


Please do not provide instructions on how to use Swarm here either. Rather just point to the Swarm documentation.

Djelibeybi · 2017-07-18T04:24:39Z

OracleWebLogic/samples/12212-msiserver/README.md

+argument helps identify the source of the application. The source is copied into the image.
+So "simple_filename" helps identify the name of the file where the source is copied to
+
+Using swarm service creation with this image


Your instructions should assume that user has a local registry and Swarm running. You don't have to provide documentation on how to do that. You can provide links to the registry and Swarm documentation though.

Hi @Djelibeybi

Thanks for the inputs. I have incorporated all your feedback. Please take a look.

mriccell

@aseembajaj thank you so much for following up with requested changes. I am happy with the changes to address my concerns.

aseembajaj · 2017-07-18T18:10:35Z

you are welcome @mriccell. And thank you for your patience I am not sure why I see a comment like "aseembajaj dismissed mriccell's stale review". Perhaps because I responded to this comment over email. I surely did not dismiss you comment, I appreciate it :)

…

On Tue, Jul 18, 2017 at 8:29 AM, Monica Riccelli ***@***.***> wrote: ***@***.**** approved this pull request. @aseembajaj <https://github.com/aseembajaj> thank you so much for following up with requested changes. I am happy with the changes to address my concerns. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#401 (review)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AXKFAJSYefcmmBTYxdbEaO_uCIOn3Layks5sPM9JgaJpZM4NsQrs> .

Djelibeybi

There are some minor fixes that are required, but the overall Markdown formatting needs to be reviewed. When I view the entire file, it seems like a lot of the code examples are not formatted correctly.

Djelibeybi · 2017-07-18T18:13:24Z

OracleWebLogic/samples/12212-msiserver/README.md

@@ -18,21 +18,14 @@ Next, to build the base msi image, run:

 Finally, to start the Managed Server in MSI mode, run:

-        $ docker run -d -p 8011:8011 12212-msiserver
+        $ docker run -p 8011:8011 12212-msiserver


Why did you remove the -d here? The image needs to be run in daemon mode, i.e. the background You should then use docker logs to retrieve the stdout from the running container to determine things like the MS name.

Hi @Djelibeybi I removed -d so that stdout can be shown on the console for further investigation. You had mentioned earlier that "exec bash" wasn't a best practice. Anyways, I'll put it back and user "docker logs" as you suggested. Thanks.

Djelibeybi · 2017-07-18T18:14:11Z

OracleWebLogic/samples/12212-msiserver/README.md

-}
-
-3. Push to the local registry
+Here is an example of how this image can be pushed to the local registry


Please remove lines 95-97. We shouldn't be documenting how to use a local registry. That's handled by our own documentation as well as the upstream documentation and is out of scope here.

Alright @Djelibeybi, I'll remove those 3 lines.

Djelibeybi · 2017-07-18T18:14:41Z

OracleWebLogic/samples/12212-msiserver/README.md

@@ -94,35 +87,18 @@ So "simple_filename" helps identify the name of the file where the source is cop
 Using swarm service creation with this image
 --------------------------------------------
 The image called 12212-summercamps-msiserver can be used for Docker service creation
-to scale out to multiple replicas. docker service creation can only be done
-using an image from a docker registry. If you don't have publishing rights to a docker
+to scale out to multiple replicas. Docker service creation can only be done


This is not accurate: Docker service creation can be done either using a local registry or by ensuring the same image is available to all hosts. Using a local registry is just easier, but it's not mandatory.

Hi @Djelibeybi I have removed that line altogether now

Djelibeybi

Great work, thanks.

Djelibeybi · 2017-07-18T19:06:37Z

I want @mriccell to give her final approval too, then I'll merge.

mriccell

I approve these changes

Sample scripts to create an image with an applicaiton running on a ma…

27aede9

…naged server running in MSI mode (Managed Server Independence mode)

brunoborges requested a review from mriccell May 31, 2017 23:26

brunoborges assigned mriccell May 31, 2017

brunoborges requested review from Djelibeybi and brunoborges May 31, 2017 23:26

brunoborges added the weblogic label May 31, 2017

Djelibeybi requested changes Jun 1, 2017

View reviewed changes

aseembajaj added 3 commits June 2, 2017 10:29

Sample scripts to create an image with an applicaiton running on a ma…

ca9f703

…naged server running in MSI mode (Managed Server Independence mode)

Incorporating feedback from Avi

c6fc9a6

Merge branch 'msi-mode-samples' of https://github.com/aseembajaj/dock…

2d11650

…er-images into msi-mode-samples

Djelibeybi previously approved these changes Jun 5, 2017

View reviewed changes

Handling docker stop to cause MSI server shutdown

5d2b3f4

mriccell reviewed Jun 14, 2017

View reviewed changes

brunoborges previously approved these changes Jun 19, 2017

View reviewed changes

aseembajaj added 2 commits July 17, 2017 19:10

Merge remote-tracking branch 'upstream/master' into msi-mode-samples

6f7f485

Incorporating Monica's feedback comments

e32e988

aseembajaj dismissed brunoborges’s stale review via e32e988 July 18, 2017 02:13

aseembajaj dismissed Djelibeybi’s stale review via e32e988 July 18, 2017 02:13

Djelibeybi requested changes Jul 18, 2017

View reviewed changes

mriccell previously approved these changes Jul 18, 2017

View reviewed changes

Incorporating feedback comments from Avi

ad10d7d

aseembajaj dismissed mriccell’s stale review via ad10d7d July 18, 2017 18:08

Djelibeybi requested changes Jul 18, 2017

View reviewed changes

Incorporating documentation and formatting related feedback from Avi

39fb783

Djelibeybi approved these changes Jul 18, 2017

View reviewed changes

mriccell approved these changes Jul 20, 2017

View reviewed changes

Djelibeybi merged commit cbf0aed into oracle:master Jul 20, 2017


		$ docker exec -it cf579fd131fc bash

		$ cat servers/ms3/logs/ms3.log \| grep BEA-150018

		fi

		bin/startManagedWebLogic.sh $ms_name

Sample scripts to create an image with an applicaiton running on a ma… #401

Sample scripts to create an image with an applicaiton running on a ma… #401

Conversation

aseembajaj commented May 31, 2017

Djelibeybi left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Arguments

---------

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Djelibeybi Jun 6, 2017 • edited Loading

Choose a reason for hiding this comment

aseembajaj commented Jun 2, 2017

mriccell commented Jun 2, 2017 via email

Djelibeybi commented Jun 5, 2017

Djelibeybi commented Jun 6, 2017

aseembajaj commented Jun 6, 2017

Djelibeybi commented Jun 6, 2017

aseembajaj commented Jun 6, 2017

Choose a reason for hiding this comment

brunoborges commented Jun 15, 2017

aseembajaj commented Jun 15, 2017

brunoborges commented Jul 10, 2017

Djelibeybi commented Jul 13, 2017

aseembajaj commented Jul 13, 2017

Djelibeybi commented Jul 13, 2017

aseembajaj commented Jul 18, 2017

Djelibeybi left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mriccell left a comment

Choose a reason for hiding this comment

aseembajaj commented Jul 18, 2017 via email • edited Loading

Djelibeybi left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Djelibeybi left a comment

Choose a reason for hiding this comment

Djelibeybi commented Jul 18, 2017

mriccell left a comment

Choose a reason for hiding this comment

Djelibeybi Jun 6, 2017 •

edited

Loading

aseembajaj commented Jul 18, 2017 via email •

edited

Loading