Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Authorize other run api #2735

Merged
merged 114 commits into from
Dec 18, 2019
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
114 commits
Select commit Hold shift + click to select a range
75c309d
add namespace to some run APIs
gaoning777 Nov 26, 2019
566cc10
update only the create run api
gaoning777 Nov 26, 2019
1060cbb
add resourcereference for namespace runs
gaoning777 Nov 26, 2019
05151b6
pass user identity header from the gRPC server to KFP service
gaoning777 Nov 26, 2019
e6c1443
add variables in const
gaoning777 Nov 26, 2019
fdb260b
Merge branch 'add-ns-in-API' into get-user-identity-from-header
gaoning777 Nov 26, 2019
be5a167
declare a flag and fill in the authorizations
gaoning777 Nov 26, 2019
659165e
add types to toModel func
gaoning777 Nov 26, 2019
f92c90f
Merge branch 'add-ns-in-API' into get-user-identity-from-header
gaoning777 Nov 26, 2019
5016f3a
Merge branch 'get-user-identity-from-header' into authorize-requests
gaoning777 Nov 26, 2019
755f542
bug fix
gaoning777 Nov 27, 2019
ee45be5
strip the namespace resource reference when mapping to the db model
gaoning777 Nov 27, 2019
2ace3fc
Merge branch 'add-ns-in-API' into get-user-identity-from-header
gaoning777 Nov 27, 2019
aea1924
Merge branch 'get-user-identity-from-header' into authorize-requests
gaoning777 Nov 27, 2019
bdd7dac
Merge branch 'master' into add-ns-in-API
gaoning777 Nov 27, 2019
204824b
add unit tests
gaoning777 Nov 27, 2019
b0aaf18
Merge branch 'add-ns-in-API' into get-user-identity-from-header
gaoning777 Nov 27, 2019
50eb63c
Merge branch 'get-user-identity-from-header' into authorize-requests
gaoning777 Nov 27, 2019
a21b60a
add authorization
gaoning777 Nov 28, 2019
a7d3db3
interpret json response
gaoning777 Nov 28, 2019
8085cbf
use gofmt
gaoning777 Dec 2, 2019
c5db317
Merge branch 'add-ns-in-API' into get-user-identity-from-header
gaoning777 Dec 2, 2019
91963c0
Merge branch 'get-user-identity-from-header' into authorize-requests
gaoning777 Dec 2, 2019
cefc903
add more meaningful error message; format
gaoning777 Dec 2, 2019
a48a554
refactoring codes
gaoning777 Dec 2, 2019
28dab1c
separate workflow client
gaoning777 Dec 3, 2019
836573a
replace belonging relationshipreference to owner
gaoning777 Dec 3, 2019
80ab78d
put a todo for further investigation of using namespace or uuid
gaoning777 Dec 3, 2019
7cc2397
apply gofmt
gaoning777 Dec 3, 2019
882872d
revert minor change
gaoning777 Dec 3, 2019
4c21962
Merge branch 'add-ns-in-API' into authorize-requests
gaoning777 Dec 3, 2019
42fa5c5
refactor codes
gaoning777 Dec 3, 2019
cd9967e
minor change
gaoning777 Dec 3, 2019
65ea8e5
Merge branch 'authorize-requests' into separate-run-resources-in-name…
gaoning777 Dec 3, 2019
7d273e5
Merge branch 'master' into authorize-requests
gaoning777 Dec 4, 2019
7c5cdc1
use internal server error in kfam client
gaoning777 Dec 4, 2019
bf30a21
minor change
gaoning777 Dec 4, 2019
a497f79
use timeout in kfam client
gaoning777 Dec 4, 2019
d24542a
make kfam service host/port configurable
gaoning777 Dec 4, 2019
f3a303d
minor changes
gaoning777 Dec 4, 2019
d4fb0f5
update name
gaoning777 Dec 4, 2019
67f414a
rename
gaoning777 Dec 4, 2019
24e5994
update the util function to accept a list of resourcereferences
gaoning777 Dec 4, 2019
bbe6f56
better error message
gaoning777 Dec 4, 2019
96aa2e3
reformat
gaoning777 Dec 4, 2019
1815bf9
remove IsRequestAuthorized func
gaoning777 Dec 4, 2019
5dc4263
Merge branch 'authorize-requests' into separate-run-resources-in-name…
gaoning777 Dec 4, 2019
0a54b7e
add multi-user mode flag
gaoning777 Dec 5, 2019
eea973a
apply different service accounts based on the multi-user mode flag
gaoning777 Dec 5, 2019
70bffca
apply service account only when it is not set
gaoning777 Dec 5, 2019
4aa1064
add kfam host and port in config.json
gaoning777 Dec 5, 2019
e1dbb41
Merge branch 'authorize-requests' into separate-run-resources-in-name…
gaoning777 Dec 5, 2019
aa08a9a
generalize the auth code
gaoning777 Dec 5, 2019
66a1150
rename KFAMInterface to KFAMClientInterface
gaoning777 Dec 5, 2019
008e7a6
add kfam fake for tests
gaoning777 Dec 5, 2019
39df35b
add build bazel
gaoning777 Dec 5, 2019
752c8de
add unit tests for util func
gaoning777 Dec 5, 2019
d68ad13
remove the config
gaoning777 Dec 5, 2019
50b1d19
add unit test for authorization with httptest
gaoning777 Dec 6, 2019
60ded01
only intialize the kfam client when kubeflow deployment
gaoning777 Dec 6, 2019
2e5e4a1
Merge branch 'authorize-requests' into separate-run-resources-in-name…
gaoning777 Dec 6, 2019
20d70c7
minor change
gaoning777 Dec 9, 2019
01a0cd4
fix typo
gaoning777 Dec 9, 2019
15cfb0e
wrap the whole auth func
gaoning777 Dec 9, 2019
1aea670
update authz logic to be enabled when it is kubeflow deployment
gaoning777 Dec 9, 2019
7158548
change flag from kubeflow deployment to multiuser mode
gaoning777 Dec 10, 2019
20de11a
gofmt
gaoning777 Dec 10, 2019
cbf8490
Merge branch 'authorize-requests' into separate-run-resources-in-name…
gaoning777 Dec 10, 2019
e07755a
minor change
gaoning777 Dec 10, 2019
4b158c4
combine getnamespace func
gaoning777 Dec 10, 2019
aee108e
insert annotation to disable istio injection
gaoning777 Dec 10, 2019
86e7b31
move unit tests
gaoning777 Dec 10, 2019
c4fcb9c
move fake kfam to the original kfam; create multiple fake kfam clients
gaoning777 Dec 10, 2019
cca49c4
combine authorize func, add unit tests for util_test
gaoning777 Dec 10, 2019
a69d965
Merge branch 'authorize-requests' into separate-run-resources-in-name…
gaoning777 Dec 10, 2019
f2e75eb
Merge branch 'master' into authorize-requests
gaoning777 Dec 10, 2019
cf95a94
Merge branch 'authorize-requests' into separate-run-resources-in-name…
gaoning777 Dec 10, 2019
f649b6a
wrap errors
gaoning777 Dec 10, 2019
2128b27
Merge branch 'authorize-requests' into separate-run-resources-in-name…
gaoning777 Dec 10, 2019
18864db
fix unit test
gaoning777 Dec 10, 2019
0d403a7
Merge branch 'authorize-requests' into separate-run-resources-in-name…
gaoning777 Dec 10, 2019
673db26
service unauthorized info to user
gaoning777 Dec 10, 2019
1cfe9a5
Merge branch 'authorize-requests' into separate-run-resources-in-name…
gaoning777 Dec 10, 2019
8744848
better user errors
gaoning777 Dec 10, 2019
1b83f94
Merge branch 'authorize-requests' into separate-run-resources-in-name…
gaoning777 Dec 10, 2019
464c7a9
inject default sa when it is empty or injected by the SDK in multi-us…
gaoning777 Dec 10, 2019
52eab3e
revert some accidental change
gaoning777 Dec 10, 2019
97535ba
Merge branch 'authorize-requests' into separate-run-resources-in-name…
gaoning777 Dec 10, 2019
684e725
revert some accidental change
gaoning777 Dec 10, 2019
df09ce1
Update util.go
IronPan Dec 10, 2019
be2806b
make functions local
gaoning777 Dec 10, 2019
3ebff5a
Merge branch 'authorize-requests' of https://github.com/gaoning777/pi…
gaoning777 Dec 10, 2019
c371e20
deduplicate return values from isauthorized
gaoning777 Dec 10, 2019
c47ec14
Merge branch 'authorize-requests' into separate-run-resources-in-name…
gaoning777 Dec 10, 2019
60714bb
Merge branch 'master' into separate-run-resources-in-namespaces
gaoning777 Dec 11, 2019
dbbb34c
update kfam service host env variable
gaoning777 Dec 11, 2019
8f1e4ee
Merge branch 'update_kfam_env_variable' into separate-run-resources-i…
gaoning777 Dec 11, 2019
472c1e5
disable istio injection
gaoning777 Dec 11, 2019
f1a2bdc
set annotations to template instead of the workflow
gaoning777 Dec 11, 2019
9757420
fix reference/value bug
gaoning777 Dec 11, 2019
adaea40
addressing comments
gaoning777 Dec 12, 2019
46b9075
Create an argoclient class
gaoning777 Dec 13, 2019
96a0c3f
move podnamespace to argo client
gaoning777 Dec 13, 2019
bf59b0e
addressing comments
gaoning777 Dec 13, 2019
fe636c7
add authorization for other run modifier
gaoning777 Dec 13, 2019
80a5e7c
add unit tests to GetNamespaceFromResourceReferencesModel; add author…
gaoning777 Dec 16, 2019
e8b0bdf
resolve circular dependency
gaoning777 Dec 16, 2019
3d10899
gofmt
gaoning777 Dec 16, 2019
b98ea26
add unit tests for IsAuthorizedRunID
gaoning777 Dec 16, 2019
d1c9547
addressing comments
gaoning777 Dec 16, 2019
c3bea17
Merge branch 'separate-run-resources-in-namespaces' into authorize-ot…
gaoning777 Dec 16, 2019
2128963
addressing comments
gaoning777 Dec 16, 2019
25ba93c
Merge branch 'master' into authorize-other-run-api
gaoning777 Dec 16, 2019
51c64c3
addressing comments
gaoning777 Dec 18, 2019
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
addressing comments
  • Loading branch information
gaoning777 committed Dec 18, 2019
commit 51c64c3fa2b8b9486b2e46b0c797f6a86f2c85d3
32 changes: 27 additions & 5 deletions backend/src/apiserver/server/run_server.go
Original file line number Diff line number Diff line change
Expand Up @@ -19,9 +19,11 @@ import (

"github.com/golang/protobuf/ptypes/empty"
api "github.com/kubeflow/pipelines/backend/api/go_client"
"github.com/kubeflow/pipelines/backend/src/apiserver/common"
"github.com/kubeflow/pipelines/backend/src/apiserver/model"
"github.com/kubeflow/pipelines/backend/src/apiserver/resource"
"github.com/kubeflow/pipelines/backend/src/common/util"
"github.com/pkg/errors"
)

type RunServer struct {
Expand Down Expand Up @@ -72,7 +74,7 @@ func (s *RunServer) ListRuns(ctx context.Context, request *api.ListRunsRequest)
}

func (s *RunServer) ArchiveRun(ctx context.Context, request *api.ArchiveRunRequest) (*empty.Empty, error) {
err := CanAccessRun(s.resourceManager, ctx, request.Id)
err := s.canAccessRun(ctx, request.Id)
if err != nil {
return nil, util.Wrap(err, "Failed to authorize the requests.")
}
Expand All @@ -84,7 +86,7 @@ func (s *RunServer) ArchiveRun(ctx context.Context, request *api.ArchiveRunReque
}

func (s *RunServer) UnarchiveRun(ctx context.Context, request *api.UnarchiveRunRequest) (*empty.Empty, error) {
err := CanAccessRun(s.resourceManager, ctx, request.Id)
err := s.canAccessRun(ctx, request.Id)
if err != nil {
return nil, util.Wrap(err, "Failed to authorize the requests.")
}
Expand All @@ -96,7 +98,7 @@ func (s *RunServer) UnarchiveRun(ctx context.Context, request *api.UnarchiveRunR
}

func (s *RunServer) DeleteRun(ctx context.Context, request *api.DeleteRunRequest) (*empty.Empty, error) {
err := CanAccessRun(s.resourceManager, ctx, request.Id)
err := s.canAccessRun(ctx, request.Id)
if err != nil {
return nil, util.Wrap(err, "Failed to authorize the requests.")
}
Expand Down Expand Up @@ -155,7 +157,7 @@ func (s *RunServer) validateCreateRunRequest(request *api.CreateRunRequest) erro
}

func (s *RunServer) TerminateRun(ctx context.Context, request *api.TerminateRunRequest) (*empty.Empty, error) {
err := CanAccessRun(s.resourceManager, ctx, request.RunId)
err := s.canAccessRun(ctx, request.RunId)
if err != nil {
return nil, util.Wrap(err, "Failed to authorize the requests.")
}
Expand All @@ -167,7 +169,7 @@ func (s *RunServer) TerminateRun(ctx context.Context, request *api.TerminateRunR
}

func (s *RunServer) RetryRun(ctx context.Context, request *api.RetryRunRequest) (*empty.Empty, error) {
err := CanAccessRun(s.resourceManager, ctx, request.RunId)
err := s.canAccessRun(ctx, request.RunId)
if err != nil {
return nil, util.Wrap(err, "Failed to authorize the requests.")
}
Expand All @@ -179,6 +181,26 @@ func (s *RunServer) RetryRun(ctx context.Context, request *api.RetryRunRequest)

}

func (s *RunServer) canAccessRun(ctx context.Context, runId string) error {
if common.IsMultiUserMode() == false {
// Skip authz if not multi-user mode.
return nil
}
runDetail, err := s.resourceManager.GetRun(runId)
if err != nil {
return util.Wrap(err, "Failed to authorize with the run Id.")
}
namespace := model.GetNamespaceFromModelResourceReferences(runDetail.ResourceReferences)
if len(namespace) == 0 {
return util.NewInternalServerError(errors.New("There is no namespace in the ResourceReferences"), "There is no namespace in the ResourceReferences")
}
err = isAuthorized(s.resourceManager, ctx, namespace)
if err != nil {
return util.Wrap(err, "Failed to authorize with API resource references")
}
return nil
}

func NewRunServer(resourceManager *resource.ResourceManager) *RunServer {
return &RunServer{resourceManager: resourceManager}
}
69 changes: 69 additions & 0 deletions backend/src/apiserver/server/run_server_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -7,9 +7,12 @@ import (
"github.com/argoproj/argo/pkg/apis/workflow/v1alpha1"
"github.com/golang/protobuf/ptypes/timestamp"
api "github.com/kubeflow/pipelines/backend/api/go_client"
"github.com/kubeflow/pipelines/backend/src/apiserver/common"
"github.com/kubeflow/pipelines/backend/src/common/util"
"github.com/spf13/viper"
"github.com/stretchr/testify/assert"
"google.golang.org/grpc/codes"
"google.golang.org/grpc/metadata"
)

func TestCreateRun(t *testing.T) {
Expand Down Expand Up @@ -318,3 +321,69 @@ func TestReportRunMetrics_PartialFailures(t *testing.T) {
}
assert.Equal(t, expectedResponse, response)
}

func TestCanAccessRun_Unauthorized(t *testing.T) {
clients, manager, experiment := initWithExperiment_KFAM_Unauthorized(t)
defer clients.Close()
runServer := RunServer{resourceManager: manager}
viper.Set(common.MultiUserMode, "true")
md := metadata.New(map[string]string{common.GoogleIAPUserIdentityHeader: "accounts.google.com:user@google.com"})
ctx := metadata.NewIncomingContext(context.Background(), md)

apiRun := &api.Run{
Name: "run1",
PipelineSpec: &api.PipelineSpec{
WorkflowManifest: testWorkflow.ToStringForStore(),
Parameters: []*api.Parameter{
{Name: "param1", Value: "world"},
},
},
ResourceReferences: []*api.ResourceReference{
{
Key: &api.ResourceKey{Type: api.ResourceType_NAMESPACE, Id: "ns"},
Relationship: api.Relationship_OWNER,
},
{
Key: &api.ResourceKey{Type: api.ResourceType_EXPERIMENT, Id: experiment.UUID},
Relationship: api.Relationship_OWNER,
},
},
}
runDetail, _ := manager.CreateRun(apiRun)

err := runServer.canAccessRun(ctx, runDetail.UUID)
assert.NotNil(t, err)
}

func TestCanAccessRun_Authorized(t *testing.T) {
clients, manager, experiment := initWithExperiment(t)
defer clients.Close()
runServer := RunServer{resourceManager: manager}
viper.Set(common.MultiUserMode, "true")
md := metadata.New(map[string]string{common.GoogleIAPUserIdentityHeader: "accounts.google.com:user@google.com"})
ctx := metadata.NewIncomingContext(context.Background(), md)

apiRun := &api.Run{
Name: "run1",
PipelineSpec: &api.PipelineSpec{
WorkflowManifest: testWorkflow.ToStringForStore(),
Parameters: []*api.Parameter{
{Name: "param1", Value: "world"},
},
},
ResourceReferences: []*api.ResourceReference{
{
Key: &api.ResourceKey{Type: api.ResourceType_NAMESPACE, Id: "ns"},
Relationship: api.Relationship_OWNER,
},
{
Key: &api.ResourceKey{Type: api.ResourceType_EXPERIMENT, Id: experiment.UUID},
Relationship: api.Relationship_OWNER,
},
},
}
runDetail, _ := manager.CreateRun(apiRun)

err := runServer.canAccessRun(ctx, runDetail.UUID)
assert.Nil(t, err)
}
21 changes: 0 additions & 21 deletions backend/src/apiserver/server/util.go
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,6 @@ import (
"github.com/golang/glog"
api "github.com/kubeflow/pipelines/backend/api/go_client"
"github.com/kubeflow/pipelines/backend/src/apiserver/common"
"github.com/kubeflow/pipelines/backend/src/apiserver/model"
"github.com/kubeflow/pipelines/backend/src/apiserver/resource"
"github.com/kubeflow/pipelines/backend/src/common/util"
"github.com/pkg/errors"
Expand Down Expand Up @@ -298,26 +297,6 @@ func getUserIdentity(ctx context.Context) (string, error) {
return "", util.NewBadRequestError(errors.New("Request header error: there is no user identity header."), "Request header error: there is no user identity header.")
}

func CanAccessRun(resourceManager *resource.ResourceManager, ctx context.Context, runId string) error {
if common.IsMultiUserMode() == false {
// Skip authz if not multi-user mode.
return nil
}
runDetail, err := resourceManager.GetRun(runId)
if err != nil {
return util.Wrap(err, "Failed to authorize with the run Id.")
}
namespace := model.GetNamespaceFromModelResourceReferences(runDetail.ResourceReferences)
if len(namespace) == 0 {
return util.NewInternalServerError(errors.New("There is no namespace in the ResourceReferences"), "There is no namespace in the ResourceReferences")
}
err = isAuthorized(resourceManager, ctx, namespace)
if err != nil {
return util.Wrap(err, "Failed to authorize with API resource references")
}
return nil
}

func CanAccessNamespaceInResourceReferences(resourceManager *resource.ResourceManager, ctx context.Context, resourceRefs []*api.ResourceReference) error {
if common.IsMultiUserMode() == false {
// Skip authz if not multi-user mode.
Expand Down
64 changes: 0 additions & 64 deletions backend/src/apiserver/server/util_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -362,38 +362,6 @@ func TestCanAccessNamespaceInResourceReferencesUnauthorized(t *testing.T) {
assert.NotNil(t, err)
}

func TestCanAccessRun_Unauthorized(t *testing.T) {
clients, manager, experiment := initWithExperiment_KFAM_Unauthorized(t)
defer clients.Close()
viper.Set(common.MultiUserMode, "true")
md := metadata.New(map[string]string{common.GoogleIAPUserIdentityHeader: "accounts.google.com:user@google.com"})
ctx := metadata.NewIncomingContext(context.Background(), md)

apiRun := &api.Run{
Name: "run1",
PipelineSpec: &api.PipelineSpec{
WorkflowManifest: testWorkflow.ToStringForStore(),
Parameters: []*api.Parameter{
{Name: "param1", Value: "world"},
},
},
ResourceReferences: []*api.ResourceReference{
{
Key: &api.ResourceKey{Type: api.ResourceType_NAMESPACE, Id: "ns"},
Relationship: api.Relationship_OWNER,
},
{
Key: &api.ResourceKey{Type: api.ResourceType_EXPERIMENT, Id: experiment.UUID},
Relationship: api.Relationship_OWNER,
},
},
}
runDetail, _ := manager.CreateRun(apiRun)

err := CanAccessRun(manager, ctx, runDetail.UUID)
assert.NotNil(t, err)
}

func TestCanAccessNamespaceInResourceReferences_Authorized(t *testing.T) {
clients, manager, _ := initWithExperiment(t)
defer clients.Close()
Expand All @@ -410,35 +378,3 @@ func TestCanAccessNamespaceInResourceReferences_Authorized(t *testing.T) {
err := CanAccessNamespaceInResourceReferences(manager, ctx, references)
assert.Nil(t, err)
}

func TestCanAccessRun_Authorized(t *testing.T) {
clients, manager, experiment := initWithExperiment(t)
defer clients.Close()
viper.Set(common.MultiUserMode, "true")
md := metadata.New(map[string]string{common.GoogleIAPUserIdentityHeader: "accounts.google.com:user@google.com"})
ctx := metadata.NewIncomingContext(context.Background(), md)

apiRun := &api.Run{
Name: "run1",
PipelineSpec: &api.PipelineSpec{
WorkflowManifest: testWorkflow.ToStringForStore(),
Parameters: []*api.Parameter{
{Name: "param1", Value: "world"},
},
},
ResourceReferences: []*api.ResourceReference{
{
Key: &api.ResourceKey{Type: api.ResourceType_NAMESPACE, Id: "ns"},
Relationship: api.Relationship_OWNER,
},
{
Key: &api.ResourceKey{Type: api.ResourceType_EXPERIMENT, Id: experiment.UUID},
Relationship: api.Relationship_OWNER,
},
},
}
runDetail, _ := manager.CreateRun(apiRun)

err := CanAccessRun(manager, ctx, runDetail.UUID)
assert.Nil(t, err)
}