Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

CMR-9502: Has-granules-map cache to redis #2058

Open
wants to merge 9 commits into
base: master
Choose a base branch
from
Open

CMR-9502: Has-granules-map cache to redis #2058

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
f23d5a4
move job to bootstrap, switch to redis
jmaeng72 Feb 13, 2024
837cd9a
move has-granules-map scheduled job to bootstrap
jmaeng72 Feb 14, 2024
496845c
add cache back into search system
jmaeng72 Feb 14, 2024
4b3ec04
update int test
jmaeng72 Feb 15, 2024
240a64f
Merge branch 'master' into CMR-9502-has-granules-map-cache-to-redis
jmaeng72 Feb 15, 2024
1dec454
fix compile error
jmaeng72 Feb 15, 2024
5c96683
fix No method in multimethod 'concept-type+result-format->fields' error
jmaeng72 Feb 16, 2024
b89b104
add logs
jmaeng72 Feb 16, 2024
a3c1a46
remove log
jmaeng72 Feb 16, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions bootstrap-app/src/cmr/bootstrap/api/routes.clj
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@
[cmr.common-app.services.search.elastic-search-index-names-cache :as elastic-search-index-names-cache]
[cmr.common.log :refer [info]]
[cmr.common.generics :as common-generic]
[cmr.search.services.query-execution.has-granules-results-feature :as has-granules-results-feature]
[compojure.core :refer :all]
[compojure.route :as route]
[drift.execute :as drift]
Expand Down Expand Up @@ -143,6 +144,9 @@
(= keyword-cache-name humanizer-alias-cache/humanizer-alias-cache-key)
(humanizer-alias-cache/refresh-entire-cache request-context)

(= keyword-cache-name has-granules-results-feature/has-granule-cache-key)
(has-granules-results-feature/refresh-has-granules-map request-context)

:else
(route/not-found "Not Found")))
{:status 200}))
Expand Down
5 changes: 4 additions & 1 deletion bootstrap-app/src/cmr/bootstrap/system.clj
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,7 @@
[cmr.metadata-db.config :as mdb-config]
[cmr.metadata-db.system :as mdb-system]
[cmr.metadata-db.services.util :as mdb-util]
[cmr.search.services.query-execution.has-granules-results-feature :as has-granules-results-feature]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you could say has-gran-rslt-feat, it's a bit long, but I don't mind either way. I love how people use longer names these days, but still remember fondly when you could shorting things up

[cmr.redis-utils.redis-cache]
[cmr.search.services.humanizers.humanizer-report-service :as humanizer-report-service]
[cmr.transmit.config :as transmit-config]))
Expand Down Expand Up @@ -96,6 +97,7 @@
coll-gran-acls-caches/coll-by-concept-id-cache-key (coll-gran-acls-caches/create-coll-by-concept-id-cache-client)
elastic-search-index-names-cache/index-names-cache-key (elastic-search-index-names-cache/create-index-cache)
humanizer-alias-cache/humanizer-alias-cache-key (humanizer-alias-cache/create-cache-client)
has-granules-results-feature/has-granule-cache-key (has-granules-results-feature/create-has-granules-map-cache)
humanizer-report-service/humanizer-report-cache-key (humanizer-report-service/create-humanizer-report-cache-client)}
:scheduler (jobs/create-scheduler `system-holder [jvm-info/log-jvm-statistics-job
(kf/refresh-kms-cache-job "bootstrap-kms-cache-refresh")
Expand All @@ -104,7 +106,8 @@
(b-coll-metadata-cache/update-collections-metadata-cache-job "bootstrap-collections-metadata-cache-update")
(coll-gran-acls-caches/refresh-collections-cache-for-granule-acls-job "bootstrap-collections-for-gran-acls-cache-refresh")
(elastic-search-index-names-cache/refresh-index-names-cache-job "bootstrap-elastic-search-index-names-cache")
(humanizer-alias-cache/refresh-humanizer-alias-cache-job "bootstrap-humanizer-alias-cache-refresh")
(humanizer-alias-cache/refresh-humanizer-alias-cache-job "boostrap-humanizer-alias-cache-refresh")
(has-granules-results-feature/refresh-has-granules-map-job "bootstrap-has-granules-map-refresh")
(humanizer-report-service/refresh-humanizer-report-cache-job "bootstrap-humanizer-report-cache-refresh")])
:queue-broker queue-broker}]
(transmit-config/system-with-connections sys [:metadata-db :echo-rest :kms :indexer :access-control :search])))
Expand Down
6 changes: 6 additions & 0 deletions common-app-lib/src/cmr/common_app/services/search/elastic_search_index.clj
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,12 @@
(fn [concept-type query]
[concept-type (qm/base-result-format query)]))

(defmethod concept-type+result-format->fields [:granule :xml]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have not seen a + sign in a function name in our code base, have you seen another function do this? Obviously you can do it, but it stands out and I wonder what other developers think.

[concept-type query]
["granule-ur"
"provider-id"
"concept-id"])

(defn context->search-index
"Returns the search index given a context. This assumes that the search index is always located in a
system using the :search-index key."
Expand Down
18 changes: 11 additions & 7 deletions search-app/src/cmr/search/services/query_execution/has_granules_results_feature.clj
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,8 @@
(:require [cmr.common-app.services.search.query-execution :as query-execution]
[cmr.common.jobs :refer [defjob]]
[cmr.common.cache :as cache]
[cmr.search.data.elastic-search-index :as idx]
[cmr.common.cache.in-memory-cache :as mem-cache]))
[cmr.redis-utils.redis-cache :as redis-cache]
[cmr.search.data.elastic-search-index :as idx]))

(def REFRESH_HAS_GRANULES_MAP_JOB_INTERVAL
"The frequency in seconds of the refresh-has-granules-map-job"
Expand All @@ -19,7 +19,7 @@
(defn create-has-granules-map-cache
"Returns a 'cache' which will contain the cached has granules map."
[]
(mem-cache/create-in-memory-cache))
(redis-cache/create-redis-cache))

(defn- collection-granule-counts->has-granules-map
"Converts a map of collection ids to granule counts to a map of collection ids to true or false
Expand All @@ -34,16 +34,18 @@
(let [has-granules-map (collection-granule-counts->has-granules-map
(idx/get-collection-granule-counts context nil))]
(cache/set-value (cache/context->cache context has-granule-cache-key)
:has-granules has-granules-map)))
has-granule-cache-key
has-granules-map)))

(defn get-has-granules-map
"Gets the cached has granules map from the context which contains collection ids to true or false
of whether the collections have granules or not. If the has-granules-map has not yet been cached
it will retrieve it and cache it."
it will retrieve it and cache it.
Example) {\"C0000000002-PROV1\" true, \"C0000000003-PROV1\" false}"
[context]
(let [has-granules-map-cache (cache/context->cache context has-granule-cache-key)]
(cache/get-value has-granules-map-cache
:has-granules
has-granule-cache-key
(fn []
(collection-granule-counts->has-granules-map
(idx/get-collection-granule-counts context nil))))))
Expand All @@ -57,6 +59,8 @@
[ctx system]
(refresh-has-granules-map {:system system}))

(def refresh-has-granules-map-job
(defn refresh-has-granules-map-job
[job-key]
{:job-type RefreshHasGranulesMapJob
:job-key job-key
:interval REFRESH_HAS_GRANULES_MAP_JOB_INTERVAL})
3 changes: 1 addition & 2 deletions search-app/src/cmr/search/system.clj
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -117,7 +117,7 @@
;; Caches a map of tokens to the security identifiers
context-augmenter/token-sid-cache-name (context-augmenter/create-token-sid-cache)
context-augmenter/token-user-id-cache-name (context-augmenter/create-token-user-id-cache)
:has-granules-map (hgrf/create-has-granules-map-cache)
hgrf/has-granule-cache-key (hgrf/create-has-granules-map-cache)
:has-granules-or-cwic-map (hgocrf/create-has-granules-or-cwic-map-cache)
:has-granules-or-opensearch-map (hgocrf/create-has-granules-or-opensearch-map-cache)
metadata-transformer/xsl-transformer-cache-name (mem-cache/create-in-memory-cache)
Expand Down Expand Up @@ -145,7 +145,6 @@
:scheduler (jobs/create-scheduler
`system-holder
[(af/refresh-acl-cache-job "search-acl-cache-refresh")
hgrf/refresh-has-granules-map-job
hgocrf/refresh-has-granules-or-cwic-map-job
hgocrf/refresh-has-granules-or-opensearch-map-job
(metadata-cache/refresh-collections-metadata-cache-job)
Expand Down
5 changes: 2 additions & 3 deletions system-int-test/test/cmr/system_int_test/admin/cache_api_test.clj
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -180,9 +180,8 @@
["ABC-1" "read"]
["ABC-2" "read"]]
"search 1" (url/search-read-caches-url) "acls" ["acls"]
;; This is now a shared redis cache server, so many keys exist
;;"search 2" (url/search-read-caches-url) "collections-for-gran-acls" []
"search 3" (url/search-read-caches-url) "has-granules-map" []
;; collections-for-gran-acls-by-concept-id and has-granules-map are purposefully left out because they are
;; in a shared redis cache server and will have multiple keys dependent on the data now
"search 5" (url/search-read-caches-url) "token-imp" [["ABC-1" "read"] ["ABC-2" "read"]]
"search 6" (url/search-read-caches-url) "token-sid" ["ABC-2" "ABC-1"]
"search 7" (url/search-read-caches-url) "xsl-transformer-templates" []
Expand Down
Loading