Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Add DQM Logging on GRPC Server with FileLogStorage for Testing #2403

Merged
merged 97 commits into from
Apr 13, 2022
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
97 commits
Select commit Hold shift + click to select a range
65880a9
Make a proof of concept
kevjumba Mar 9, 2022
b32b157
Update
kevjumba Mar 11, 2022
1996fd9
revert feature store
kevjumba Mar 17, 2022
c6656d4
refactor
kevjumba Mar 17, 2022
17c8168
Add time
kevjumba Mar 17, 2022
a4efbce
Add time
kevjumba Mar 17, 2022
79d8ba7
clean up
kevjumba Mar 17, 2022
cd652bc
Add comment
kevjumba Mar 17, 2022
4a13e8e
Add pseudocode
kevjumba Mar 17, 2022
a3f8384
Refactor logging functionality to hide internals
kevjumba Mar 20, 2022
706ee80
Refactor
kevjumba Mar 20, 2022
1831003
Revert changes
kevjumba Mar 20, 2022
9e669c3
Add tests
kevjumba Mar 20, 2022
b972634
Add new timeout test
kevjumba Mar 21, 2022
42bdc8b
Fix python ci for m1 mac
kevjumba Mar 25, 2022
ab8d1e0
Fix lint
kevjumba Mar 29, 2022
f6c2c87
Working state
kevjumba Mar 30, 2022
b6b412d
Move offline log store
kevjumba Mar 30, 2022
cb1da99
refactor
kevjumba Mar 30, 2022
155fc77
Update logs
kevjumba Mar 30, 2022
86f2208
Update log storage
kevjumba Mar 30, 2022
746dae9
WOrking state
kevjumba Mar 30, 2022
514c2f6
Work
kevjumba Mar 31, 2022
161d81d
Add tests for filestorage
kevjumba Mar 31, 2022
beec6b2
Fix logging
kevjumba Mar 31, 2022
2a1d2ec
Add more tests
kevjumba Mar 31, 2022
9446e05
Fix
kevjumba Mar 31, 2022
31e0de8
Fix
kevjumba Mar 31, 2022
137d700
Clean up
kevjumba Mar 31, 2022
ee847cd
Update error
kevjumba Mar 31, 2022
ce12eee
semi working state
kevjumba Apr 1, 2022
79c700c
b state
kevjumba Apr 4, 2022
6cbf2c2
Update types to be public
kevjumba Apr 6, 2022
aefda92
Update structs to make fields public
kevjumba Apr 6, 2022
29f72b0
Fix
kevjumba Apr 6, 2022
c4dbb84
clean up
kevjumba Apr 6, 2022
5d7978f
Fix
kevjumba Apr 7, 2022
686c583
Fix go
kevjumba Apr 7, 2022
884b014
Fix issues
kevjumba Apr 7, 2022
c02cf83
Fix
kevjumba Apr 7, 2022
e89abc1
Fix tests
kevjumba Apr 7, 2022
559bb83
Fix
kevjumba Apr 7, 2022
0f2aca9
Working state
kevjumba Apr 8, 2022
c631ef8
Fix
kevjumba Apr 8, 2022
b0a2166
fix
kevjumba Apr 8, 2022
9ed79e8
fix
kevjumba Apr 8, 2022
e472104
Clean up code a bit
kevjumba Apr 8, 2022
629be18
Fixes
kevjumba Apr 8, 2022
8a789fb
Fix
kevjumba Apr 8, 2022
6357efd
Fix tests
kevjumba Apr 8, 2022
3168473
Fix
kevjumba Apr 8, 2022
00228f7
Clean up
kevjumba Apr 8, 2022
b692e81
Update schema functionality
kevjumba Apr 9, 2022
49cd91a
Remove xitongsys parquet reader
kevjumba Apr 9, 2022
4890116
Clean up
kevjumba Apr 11, 2022
71e1e56
Fix go mode
kevjumba Apr 11, 2022
050db82
Fix tests and errors and everything
kevjumba Apr 11, 2022
dd235ca
Fix tests
kevjumba Apr 11, 2022
7c92d93
Fix
kevjumba Apr 11, 2022
4265efd
Remove unused code
kevjumba Apr 11, 2022
79cbe42
Fix
kevjumba Apr 11, 2022
21e99bd
Last working commit
kevjumba Apr 11, 2022
1b173da
work
kevjumba Apr 11, 2022
b5484c3
Address some changes
kevjumba Apr 11, 2022
5458034
More addresses.
kevjumba Apr 11, 2022
17b2bf4
Fix more review comments
kevjumba Apr 11, 2022
0a1802d
Fix
kevjumba Apr 11, 2022
18c7d60
Fix
kevjumba Apr 12, 2022
a39c9b5
Rename
kevjumba Apr 12, 2022
a4587c3
Fix
kevjumba Apr 12, 2022
83c5f89
Add request id
kevjumba Apr 12, 2022
86e605d
More fixes
kevjumba Apr 12, 2022
fa14dae
Fix odfv
kevjumba Apr 12, 2022
9f81d04
Fix
kevjumba Apr 12, 2022
e87fe22
Fix
kevjumba Apr 12, 2022
bdf0c2b
Address other changes
kevjumba Apr 12, 2022
f59f98d
Reorder for optimization
kevjumba Apr 12, 2022
d185ccd
Fix
kevjumba Apr 12, 2022
3747a5d
Add more shcema tests
kevjumba Apr 12, 2022
e9bd35b
Fix tests
kevjumba Apr 12, 2022
89974b3
refactor to clean
kevjumba Apr 12, 2022
d51544b
Add initialized repo for testing
kevjumba Apr 13, 2022
e0a4ec6
Fix
kevjumba Apr 13, 2022
5f9a50e
Remove
kevjumba Apr 13, 2022
1a5d98e
Fix tests
kevjumba Apr 13, 2022
b4d94dc
Fix
kevjumba Apr 13, 2022
8d9d0f9
Fix tests
kevjumba Apr 13, 2022
4e65987
Fix
kevjumba Apr 13, 2022
9d4effa
Text
kevjumba Apr 13, 2022
e604750
Fix
kevjumba Apr 13, 2022
49fb8bc
Fix?
kevjumba Apr 13, 2022
72b1700
Fix
kevjumba Apr 13, 2022
b4f7f41
Fix
kevjumba Apr 13, 2022
f19d19d
remove entity map
pyalex Apr 13, 2022
0145e3d
remove Cache method from registry
pyalex Apr 13, 2022
cad1156
clean up pre-initialized repo
pyalex Apr 13, 2022
89960b2
git ignore full data directory in tests
pyalex Apr 13, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Fix
Signed-off-by: Kevin Zhang <kzhang@tecton.ai>
  • Loading branch information
kevjumba committed Apr 13, 2022
commit 4e65987508973eda08623c88de9bcbe816b80940
Binary file not shown.
6 changes: 3 additions & 3 deletions go/cmd/server/server_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -100,7 +100,7 @@ func TestGetFeastServingInfo(t *testing.T) {
ctx := context.Background()
// Pregenerated using `feast init`.
dir := "logging/"
err := test.MaterializeInInitializedRepo(dir)
err := test.SetupInitializedRepo(dir)
assert.Nil(t, err)
client, closer := getClient(ctx, "", dir, false)
defer closer()
Expand All @@ -113,7 +113,7 @@ func TestGetOnlineFeaturesSqlite(t *testing.T) {
ctx := context.Background()
// Pregenerated using `feast init`.
dir := "logging/"
err := test.MaterializeInInitializedRepo(dir)
err := test.SetupInitializedRepo(dir)
assert.Nil(t, err)
client, closer := getClient(ctx, "", dir, false)
defer closer()
Expand Down Expand Up @@ -170,7 +170,7 @@ func TestGetOnlineFeaturesSqliteWithLogging(t *testing.T) {
ctx := context.Background()
// Pregenerated using `feast init`.
dir := "logging/"
err := test.MaterializeInInitializedRepo(dir)
err := test.SetupInitializedRepo(dir)
assert.Nil(t, err)
client, closer := getClient(ctx, "file", dir, true)
defer closer()
Expand Down
16 changes: 14 additions & 2 deletions go/internal/test/go_integration_test_utils.go
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
package test

import (
"bytes"
"context"
"fmt"

Expand Down Expand Up @@ -121,12 +122,22 @@ func SetupCleanFeatureRepo(basePath string) error {
return nil
}

func MaterializeInInitializedRepo(basePath string) error {
func SetupInitializedRepo(basePath string) error {
path, err := filepath.Abs(basePath)
if err != nil {
return err
}
feature_repo_path, err := filepath.Abs(filepath.Join(path, "feature_repo/"))
applyCommand := exec.Command("feast", "apply")
applyCommand.Env = os.Environ()
feature_repo_path, err := filepath.Abs(filepath.Join(path, "feature_repo"))
if err != nil {
return err
}
var stderr bytes.Buffer

applyCommand.Dir = feature_repo_path
applyCommand.Stderr = &stderr
err = applyCommand.Run()
if err != nil {
return err
}
Expand All @@ -135,6 +146,7 @@ func MaterializeInInitializedRepo(basePath string) error {
formattedTime := fmt.Sprintf("%d-%02d-%02dT%02d:%02d:%02d",
t.Year(), t.Month(), t.Day(),
t.Hour(), t.Minute(), t.Second())

materializeCommand := exec.Command("feast", "materialize-incremental", formattedTime)
materializeCommand.Env = os.Environ()
materializeCommand.Dir = feature_repo_path
Expand Down