Making Pulsar Proxy more secure by jai1 · Pull Request #1002 · apache/pulsar

jai1 · 2017-12-21T17:13:46Z

Following has been done to make the Pulsar Proxy more secure:-
a. Removed access to zoo keeper
b. Authorization is no longer done by proxy, it is now done by broker. Proxy only authenticates the client and forwards the authRoleNam
c. An option is added to disable discovery service in proxy
d. A new auth action is added which allows admin to restrict whether a topic is proxyable or not

merlimat · 2017-12-21T22:02:09Z

d. A new auth action is added which allows admin to restrict whether a topic is proxyable or not

I don't think "proxy" it's a good match for an action.
An action in the ACLs is associated with a "role" while for proxy is just a flag

jai1 · 2017-12-22T00:37:04Z

I don't think "proxy" it's a good match for an action. An action in the ACLs is associated with a "role" while for proxy is just a flag

Should we rename the action to lookup and add another or condition to
https://github.com/apache/incubator-pulsar/blob/master/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/authorization/AuthorizationManager.java?utf8=%E2%9C%93#L109

OR

Not add any Action and enforce that proxy has same AuthAcion permission as the client (produce and consume) - finer control

jai1 · 2017-12-29T06:21:39Z

retest this please

merlimat · 2018-01-02T08:40:30Z

@jai1 This change is adding or changing the behavior of a big component and adding new metadata options. Can you create a PIP wiki page and start the discussion on the mailing list to review the design doc?

jai1 · 2018-01-02T20:04:39Z

@merlimat
Created a PIP
https://github.com/apache/incubator-pulsar/wiki/PIP-9:-Adding-more-Security-checks-to-Pulsar-Proxy

merlimat

Added few comments

merlimat · 2018-01-26T22:54:17Z

        None;
    }
+
+    public static ByteBuf newPartitionMetadataRequest(String topic, long requestId, String clientAuthRole) {


We should reuse the same implementation, either passing null or having an ovverride method and passing null

merlimat · 2018-01-26T22:54:40Z

+        return res;
+    }
+
+    public static ByteBuf newLookup(String topic, boolean authoritative, String clientAuthRole, long requestId) {


Same here, most of method is identical to current newLookup()

merlimat · 2018-01-26T22:56:52Z

 	v8 = 8;  // Added CommandConsumerStats - Client fetches broker side consumer stats
 	v9 = 9;  // Added end of topic notification
 	v10 = 10;// Added proxy to broker
+	v11 = 11;// Added more security to proxy lookup requests


Do we need to increase version? If the broker doesn't know about original_principal it will just ignore the field.
The version should only be incremented when we add new Command or features that require checking that the other side supports them.

merlimat · 2018-01-26T22:58:36Z

@@ -0,0 +1,72 @@
+Certificate:


Make sure these files are ignored by license:check and apache-rat:check

jai1 · 2018-01-28T02:59:20Z

@merlimat @rdhabalia @saandrews - thanks for the feedback - have addressed all your comments
Just need a +1 to merge

jai1 · 2018-01-28T22:12:59Z

retest this please

jai1 · 2018-01-29T12:17:24Z

retest this please

jai1 · 2018-01-29T22:26:38Z

retest this please

rdhabalia

LGTM.. just a minor comments.

rdhabalia · 2018-01-30T01:42:19Z

        this.proxyConnection = proxyConnection;
        this.clientAddress = proxyConnection.clientAddress();
        this.connectWithTLS = proxy.getConfiguration().isTlsEnabledWithBroker();
+        this.brokerServiceURL = this.connectWithTLS ? proxy.getConfiguration().getBrokerServiceURLTLS()


should we also add another check
this.connectWithTLS && isNotBlank(proxy.getConfiguration().getBrokerServiceURLTLS()) ? proxy.getConfiguration().getBrokerServiceURLTLS() : proxy.getConfiguration().getBrokerServiceURL() ?

rdhabalia · 2018-01-30T01:47:03Z

+                    finalResult.complete(produceAuthorized);
+                    return;
+                }
+            } else if (log.isDebugEnabled()) {


else { if (log.isDebugEnabled()) { : } }

rdhabalia · 2018-01-30T01:53:00Z

+                        return;
+                    }
+                } else if (log.isDebugEnabled()) {
+                    log.debug(


canProduceAsync() and canConsumeAsync() fails the future when it has zk-connection issue and in that case, we should give 5XX error instead 403. So, I think we should fail finalResult future here instead returning completing it with false.

rdhabalia

👍 LGTM.. added last few minor comments..

saandrews

Looks good.

jai1 · 2018-01-30T22:26:41Z

retest this please

ivankelly · 2018-06-25T14:32:03Z

+            CompletableFuture<Boolean> isProxyAuthorizedFuture;
+            if (service.isAuthorizationEnabled() && originalPrincipal != null) {
+                isProxyAuthorizedFuture = service.getAuthorizationManager()
+                        .canLookupAsync(DestinationName.get(topicName), authRole);


@jai1 @merlimat is authRole correct here? Surely if we are coming from the proxy and originalPrincipal is set, we should be checking if the original principle which can access the resource?

ivankelly · 2018-08-20T12:45:52Z

@jai1 was the proxy auth action ever added?

jai1 · 2018-08-21T06:49:12Z

was the proxy auth action ever added?

No, and we don't need to

@jai1 <https://github.com/jai1> @merlimat <https://github.com/merlimat> is

authRole correct here? Surely if we are coming from the proxy and originalPrincipal is set, we should be checking if the original principle which can access the resource? The code looks convoluted but I think it's correct so basically if, ' originalPrincipal != null' then check if both proxy role (authRole) and client role (original Principal) can lookup on the topic. So on line 251 (first call to canLookupAsync) we check whether the proxy (authRole) has permission to lookup then on line 261 (call to lookupTopicAsync) we check if client role (original Principal) has permission to lookup. Basic reasoning for this was that we don't want the proxy to have access to all Pulsar namespaces - we want to have a configuration where proxy X can only access namespace Y.

…

On Mon, Aug 20, 2018 at 5:45 AM, Ivan Kelly ***@***.***> wrote: @jai1 <https://github.com/jai1> was the proxy auth action ever added? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1002 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AIQh_Miv5a2zuv0lQVmXYzdDyyADNbGtks5uSq-GgaJpZM4RKG7r> .

…ort multi uri yet (#21972) ### Motivation At the beginning of the design, these two configurations(`brokerServiceURL & brokerServiceURLTLS`) do not support setting multiple broker addresses, which should instead be set to a “discovery service provider.” see: #1002 and #14682 Users will get the below error if they set A to a multi-broker URLs ``` "2024-01-09 00:20:10,261 -0800 [pulsar-proxy-io-4-7] WARN io.netty.channel.DefaultChannelPipeline - An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception. java.lang.IllegalArgumentException: port out of range:-1 at java.net.InetSocketAddress.checkPort(InetSocketAddress.java:143) ~[?:?] at java.net.InetSocketAddress.createUnresolved(InetSocketAddress.java:254) ~[?:?] at org.apache.pulsar.proxy.server.LookupProxyHandler.getAddr(LookupProxyHandler.java:432) ~[org.apache.pulsar-pulsar-proxy-2.9.0.jar:2.9.0] at org.apache.pulsar.proxy.server.LookupProxyHandler.handleGetSchema(LookupProxyHandler.java:357) ~[org.apache.pulsar-pulsar-proxy-2.9.0.jar:2.9.0] at org.apache.pulsar.proxy.server.ProxyConnection.handleGetSchema(ProxyConnection.java:463) ~[org.apache.pulsar-pulsar-proxy-2.9.0.jar:2.9.0] at org.apache.pulsar.common.protocol.PulsarDecoder.channelRead(PulsarDecoder.java:326) ~[io.streamnative-pulsar-common-2.9.2.12.jar:2.9.2.12] at org.apache.pulsar.proxy.server.ProxyConnection.channelRead(ProxyConnection.java:221) ~[org.apache.pulsar-pulsar-proxy-2.9.0.jar:2.9.0] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[io.netty-netty-transport-4.1.74.Final.jar:4.1.74.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[io.netty-netty-transport-4.1.74.Final.jar:4.1.74.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[io.netty-netty-transport-4.1.74.Final.jar:4.1.74.Final] at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:327) ~[io.netty-netty-codec-4.1.74.Final.jar:4.1.74.Final] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:299) ~[io.netty-netty-codec-4.1.74.Final.jar:4.1.74.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[io.netty-netty-transport-4.1.74.Final.jar:4.1.74.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[io.netty-netty-transport-4.1.74.Final.jar:4.1.74.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[io.netty-netty-transport-4.1.74.Final.jar:4.1.74.Final] at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1372) ~[io.netty-netty-handler-4.1.74.Final.jar:4.1.74.Final] at io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1246) ~[io.netty-netty-handler-4.1.74.Final.jar:4.1.74.Final] at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1286) ~[io.netty-netty-handler-4.1.74.Final.jar:4.1.74.Final] at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:510) ~[io.netty-netty-codec-4.1.74.Final.jar:4.1.74.Final] at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:449) ~[io.netty-netty-codec-4.1.74.Final.jar:4.1.74.Final] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:279) ~[io.netty-netty-codec-4.1.74.Final.jar:4.1.74.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[io.netty-netty-transport-4.1.74.Final.jar:4.1.74.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[io.netty-netty-transport-4.1.74.Final.jar:4.1.74.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[io.netty-netty-transport-4.1.74.Final.jar:4.1.74.Final] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[io.netty-netty-transport-4.1.74.Final.jar:4.1.74.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[io.netty-netty-transport-4.1.74.Final.jar:4.1.74.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[io.netty-netty-transport-4.1.74.Final.jar:4.1.74.Final] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[io.netty-netty-transport-4.1.74.Final.jar:4.1.74.Final] at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:795) ~[io.netty-netty-transport-classes-epoll-4.1.74.Final.jar:4.1.74.Final] at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:480) ~[io.netty-netty-transport-classes-epoll-4.1.74.Final.jar:4.1.74.Final] at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) ~[io.netty-netty-transport-classes-epoll-4.1.74.Final.jar:4.1.74.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) ~[io.netty-netty-common-4.1.74.Final.jar:4.1.74.Final] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[io.netty-netty-common-4.1.74.Final.jar:4.1.74.Final] at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[io.netty-netty-common-4.1.74.Final.jar:4.1.74.Final] ``` ### Modifications - Improve the description - Add a check to prevent wrong settings

…ort multi uri yet (apache#21972) At the beginning of the design, these two configurations(`brokerServiceURL & brokerServiceURLTLS`) do not support setting multiple broker addresses, which should instead be set to a “discovery service provider.” see: apache#1002 and apache#14682 Users will get the below error if they set A to a multi-broker URLs ``` "2024-01-09 00:20:10,261 -0800 [pulsar-proxy-io-4-7] WARN io.netty.channel.DefaultChannelPipeline - An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception. java.lang.IllegalArgumentException: port out of range:-1 at java.net.InetSocketAddress.checkPort(InetSocketAddress.java:143) ~[?:?] at java.net.InetSocketAddress.createUnresolved(InetSocketAddress.java:254) ~[?:?] at org.apache.pulsar.proxy.server.LookupProxyHandler.getAddr(LookupProxyHandler.java:432) ~[org.apache.pulsar-pulsar-proxy-2.9.0.jar:2.9.0] at org.apache.pulsar.proxy.server.LookupProxyHandler.handleGetSchema(LookupProxyHandler.java:357) ~[org.apache.pulsar-pulsar-proxy-2.9.0.jar:2.9.0] at org.apache.pulsar.proxy.server.ProxyConnection.handleGetSchema(ProxyConnection.java:463) ~[org.apache.pulsar-pulsar-proxy-2.9.0.jar:2.9.0] at org.apache.pulsar.common.protocol.PulsarDecoder.channelRead(PulsarDecoder.java:326) ~[io.streamnative-pulsar-common-2.9.2.12.jar:2.9.2.12] at org.apache.pulsar.proxy.server.ProxyConnection.channelRead(ProxyConnection.java:221) ~[org.apache.pulsar-pulsar-proxy-2.9.0.jar:2.9.0] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[io.netty-netty-transport-4.1.74.Final.jar:4.1.74.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[io.netty-netty-transport-4.1.74.Final.jar:4.1.74.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[io.netty-netty-transport-4.1.74.Final.jar:4.1.74.Final] at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:327) ~[io.netty-netty-codec-4.1.74.Final.jar:4.1.74.Final] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:299) ~[io.netty-netty-codec-4.1.74.Final.jar:4.1.74.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[io.netty-netty-transport-4.1.74.Final.jar:4.1.74.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[io.netty-netty-transport-4.1.74.Final.jar:4.1.74.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[io.netty-netty-transport-4.1.74.Final.jar:4.1.74.Final] at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1372) ~[io.netty-netty-handler-4.1.74.Final.jar:4.1.74.Final] at io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1246) ~[io.netty-netty-handler-4.1.74.Final.jar:4.1.74.Final] at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1286) ~[io.netty-netty-handler-4.1.74.Final.jar:4.1.74.Final] at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:510) ~[io.netty-netty-codec-4.1.74.Final.jar:4.1.74.Final] at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:449) ~[io.netty-netty-codec-4.1.74.Final.jar:4.1.74.Final] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:279) ~[io.netty-netty-codec-4.1.74.Final.jar:4.1.74.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[io.netty-netty-transport-4.1.74.Final.jar:4.1.74.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[io.netty-netty-transport-4.1.74.Final.jar:4.1.74.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[io.netty-netty-transport-4.1.74.Final.jar:4.1.74.Final] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[io.netty-netty-transport-4.1.74.Final.jar:4.1.74.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[io.netty-netty-transport-4.1.74.Final.jar:4.1.74.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[io.netty-netty-transport-4.1.74.Final.jar:4.1.74.Final] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[io.netty-netty-transport-4.1.74.Final.jar:4.1.74.Final] at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:795) ~[io.netty-netty-transport-classes-epoll-4.1.74.Final.jar:4.1.74.Final] at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:480) ~[io.netty-netty-transport-classes-epoll-4.1.74.Final.jar:4.1.74.Final] at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) ~[io.netty-netty-transport-classes-epoll-4.1.74.Final.jar:4.1.74.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) ~[io.netty-netty-common-4.1.74.Final.jar:4.1.74.Final] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[io.netty-netty-common-4.1.74.Final.jar:4.1.74.Final] at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[io.netty-netty-common-4.1.74.Final.jar:4.1.74.Final] ``` - Improve the description - Add a check to prevent wrong settings (cherry picked from commit 8a56873)

Jai ASher and others added 14 commits November 22, 2017 16:02

Made service discovery optional for proxy

f956e8d

Creating a proto file remains

878266a

Updated proto file for Pulsar Proxy

85206bf

Fixed compilation error

d982ec7

Merge branch 'Proxy2' of https://github.com/jai1/pulsar into Proxy2

cee5751

Added tests and fixed some logic which causes existing test to break

09d5bcf

Fixed tests

0f77bfe

Added certificates

069003c

Tests

76497c7

Tests

2516038

Readded Server connection code

578fdb6

Corrected and fixed some test cases

9939742

Removed sending of original auth data

b716ae3

Handled merge conflict

bc95361

jai1 added the type/enhancement The enhancements for the existing features or docs. e.g. reduce memory usage of the delayed messages label Dec 21, 2017

jai1 added this to the 1.22.0-incubating milestone Dec 21, 2017

jai1 self-assigned this Dec 21, 2017

jai1 requested review from merlimat, rdhabalia and saandrews December 21, 2017 17:13

Fixed some spaces in license header and Commands.java

25fe708

Jai ASher added 2 commits December 28, 2017 18:10

Fixed Redirect logic

bc54ffd

Handled merge conflict

2071d13

Added licene headers

9ee13dd

Resolved Merge Conflict

f5f7a2e

merlimat reviewed Jan 26, 2018

View reviewed changes

Jai ASher added 3 commits January 26, 2018 15:39

Addressed Rajans changes

a5c9b6a

Addressed Matteos comments

d50a8bf

Fixed compilation errors

75958af

Changd cluster name to get tests to work

006e8c6

Jai ASher added 2 commits January 29, 2018 09:11

Merge branch 'master' of https://github.com/yahoo/pulsar into Proxy2

2bbac2f

Fixed test cases to not create clusters since broker already does

5e5a2a2

rdhabalia reviewed Jan 30, 2018

View reviewed changes

rdhabalia approved these changes Jan 30, 2018

View reviewed changes

saandrews approved these changes Jan 30, 2018

View reviewed changes

jai1 merged commit dde5865 into apache:master Jan 31, 2018

jai1 deleted the Proxy2 branch January 31, 2018 00:09

jai1 mentioned this pull request Feb 13, 2018

Documentation required for Pulsar Proxy and TLS changes #1229

Closed

ivankelly reviewed Jun 25, 2018

View reviewed changes

ivankelly mentioned this pull request Aug 20, 2018

Document configuring the proxy without service discovery #2405

Closed

nodece mentioned this pull request Mar 13, 2023

[cleanup][proxy] Remove authorization config #19797

Closed

4 tasks

poorbarcode mentioned this pull request Jan 25, 2024

[improve] [proxy] Add a check for brokerServiceURL that does not support multi uri yet #21972

Merged

4 tasks

Uh oh!

Conversation

jai1 commented Dec 21, 2017

Uh oh!

merlimat commented Dec 21, 2017

Uh oh!

jai1 commented Dec 22, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jai1 commented Dec 29, 2017

Uh oh!

merlimat commented Jan 2, 2018

Uh oh!

jai1 commented Jan 2, 2018

Uh oh!

merlimat left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jai1 commented Jan 28, 2018

Uh oh!

jai1 commented Jan 28, 2018

Uh oh!

jai1 commented Jan 29, 2018

Uh oh!

jai1 commented Jan 29, 2018

Uh oh!

rdhabalia left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rdhabalia left a comment

Choose a reason for hiding this comment

Uh oh!

saandrews left a comment

Choose a reason for hiding this comment

Uh oh!

jai1 commented Jan 30, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ivankelly commented Aug 20, 2018

Uh oh!

jai1 commented Aug 21, 2018 via email

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

jai1 commented Dec 22, 2017 •

edited

Loading