Skip to content

Commit

Permalink
Lots of code comments, some refactoring and enhancements to /q endpoint
Browse files Browse the repository at this point in the history
  • Loading branch information
atdixon committed Dec 23, 2022
1 parent a1817bd commit 878de6a
Show file tree
Hide file tree
Showing 8 changed files with 346 additions and 102 deletions.
17 changes: 11 additions & 6 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ conf/logback.xml
```

Unpack on server, update config files to your personal liking (note: leave
the "supported_nips" and "version" as-is in the nip11.json file), and run
the "supported_nips" and "version" as-is in the `nip11.json` file), and run
(using java 11+):

```
Expand All @@ -53,9 +53,17 @@ $ java -Xms1g -Xmx1g \
This runs the relay on the port specified in `conf/relay.yaml` (default 9090).

You'll want your users to hit a reverse proxy, configured to serve SSL traffic
(wss://...) and proxy to the relay.
(wss://...) and proxy to the relay server.

If you're developing you can build a jar or deployment archive from latest
See [Deploy](./doc/deploy.md) for more information on how to run a real
deployment.

### Develop

The best place to start reading the code is from the `-main` method in the
well-documented [me.untethr.nostr.app](./src/me/untethr/nostr/app.clj) namespace.

If you're developing you can build a jar or deployment archive from latest
source, like so:


Expand All @@ -68,6 +76,3 @@ or
```
$ make deploy-archive
```

See [Deploy](./doc/deploy.md) for more information on how to run a real
deployment.
36 changes: 29 additions & 7 deletions doc/deploy.md
Original file line number Diff line number Diff line change
Expand Up @@ -68,23 +68,45 @@ WantedBy=multi-user.target

And ran:

```
sudo systemctl daemon-reload
sudo systemctl enable nostr-relay.service
sudo systemctl start nostr-relay
sudo systemctl status nostr-relay
```shell
$ sudo systemctl daemon-reload
$ sudo systemctl enable nostr-relay.service
$ sudo systemctl start nostr-relay
$ sudo systemctl status nostr-relay
```

Now, 2 ways to look at logs.

* Relay server logs:

```
```shell
/home/aaron$ tail -f logs/nostr-relay-<latest>.log
```

* systemd service logs &mdash; this will show uncaught errors, due to bad requests or other issues, etc:

```shell
$ journalctl -u nostr-relay -f -o cat
```
journalctl -u nostr-relay -f -o cat

You can query for server metrics:

```shell
$ curl https://<relay-host>/metrics
```

And issue basic non-websocket queries over your relay's data:

```shell
$ curl https://<your-relay-host>/q

# basic query param filters like these are convenient from a browser
$ curl <your-relay-host>/q?until=now

$ curl <your-relay-host>/q?author=aff9a9f017f32b2e8b60754a4102db9d9cf9ff2b967804b50e070780aa45c9a8

# using more complex nostr filters in the request body
$ curl -H 'Content-Type: application/json' \
-XGET <your-relay-host>/q \
--data '[{"authors":["aff9a9f017f32b2e8b60754a4102db9d9cf9ff2b967804b50e070780aa45c9a8"]}]'
```
230 changes: 160 additions & 70 deletions src/me/untethr/nostr/app.clj

Large diffs are not rendered by default.

105 changes: 105 additions & 0 deletions src/me/untethr/nostr/extra.clj
Original file line number Diff line number Diff line change
@@ -0,0 +1,105 @@
(ns me.untethr.nostr.extra
(:require
[clojure.pprint :as pprint]
[clojure.set :as set]
[clojure.walk :as walk]
[me.untethr.nostr.conf]
[me.untethr.nostr.json-facade :as json-facade]
[me.untethr.nostr.query :as query]
[me.untethr.nostr.util :as util]
[me.untethr.nostr.validation :as validation]
[next.jdbc :as jdbc]
[next.jdbc.result-set :as rs])
(:import (me.untethr.nostr.conf Conf)))

(defn- coerce-num
[x]
(cond
(= "now" x) (util/current-system-epoch-seconds)
(string? x) (Long/parseLong x)
:else (long x)))

(defn- query-params->filter
[query-params]
(let [prepared (some-> query-params
walk/keywordize-keys
(select-keys [:since :until :limit :kind :id :author])
not-empty
(set/rename-keys {:id :ids :author :authors :kind :kinds}))]
(cond-> prepared
(contains? prepared :authors) (update :authors vector)
(contains? prepared :ids) (update :ids vector)
(contains? prepared :kinds) (update :kinds (comp vector coerce-num))
(contains? prepared :since) (update :since coerce-num)
(contains? prepared :until) (update :until coerce-num)
(contains? prepared :limit) (update :limit coerce-num))))

(defn- validate-filters!
[filters]
(when (> (count filters) 5)
(throw (ex-info "too many filters" {:filters filters})))
(when-not (every? map? filters)
(throw (ex-info "bad filters" {:filters filters})))
(when-let [req-err (validation/req-err "dummy-id" filters)]
(throw (ex-info "bad request" {:err req-err :filters filters}))))

(defn handler-q
"A ring hander that supports ad hoc queries over relay data. Primarily for
admin purposes, i.e., should not be used by any clients.
This handler supports querying using both URL parameters (especially
useful from a browser) and the full query filter forms in the HTTP request
body.
Examples (as if using curl):
curl https://<relay-host>/q
curl https://<relay-host>/q?since=1671816629&until=now
curl https://<relay-host>/q?author=<pubkey>
curl https://<relay-host>/q?id=<sha256>
When using filters in body request, you'll need to specify
`Content-Type` to something other than \"x-www-form-urlencoded\"
(otherwise ring middleware may consume the request :body before our handler
here can):
curl -H 'Content-Type: application/json' \\
-XGET <your-relay-host>/q \\
--data '[{\"authors\":[\"<pubkey>\"]}]'
"
[^Conf conf db prepare-req-filters-fn req]
(let [query-params-as-filter (some-> req :query-params query-params->filter)
body-as-filters (some->> req :body slurp not-empty json-facade/parse)
use-filters (or (some-> query-params-as-filter vector) body-as-filters [{}])
_ (validate-filters! use-filters)
prepared-filters (prepare-req-filters-fn conf use-filters)
;; default limit to 25 if unspecified, but don't let limit exceed 100:
modified-filters (mapv #(update % :limit (fn [a b] (min (or a b) 100)) 25) prepared-filters)
as-query (query/filters->query modified-filters)]
(let [rows (jdbc/execute! db as-query
{:builder-fn rs/as-unqualified-lower-maps})
rows' (mapv
(fn [row]
(let [parsed-event (-> row :raw_event json-facade/parse)]
(-> row
(dissoc :raw_event)
(merge
(select-keys parsed-event [:kind :pubkey]))
(assoc :content
(let [max-summary-len 75
the-content (:content parsed-event)
the-content-len (count the-content)
needs-summary? (> the-content-len max-summary-len)
the-summary (if needs-summary?
(subs the-content 0 max-summary-len) the-content)
suffix (if needs-summary? "..." "")]
(str the-summary suffix)))))) rows)]
{:status 200
:headers {"Content-Type" "text/plain"}
:body (format "filters: %s%n%s"
(json-facade/write-str* use-filters)
(if (empty? rows')
"No results."
(with-out-str
(pprint/print-table
[:rowid :kind :pubkey :content] rows'))))})))
5 changes: 3 additions & 2 deletions src/me/untethr/nostr/store.clj
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,8 @@
[clojure.tools.logging :as log]
[next.jdbc :as jdbc]
[next.jdbc.result-set :as rs])
(:import (org.sqlite SQLiteException)))
(:import (javax.sql DataSource)
(org.sqlite SQLiteException)))

(def get-datasource*
(memoize
Expand Down Expand Up @@ -37,7 +38,7 @@
(throw e))))))

(defn init!
[path]
^DataSource [path]
(doto (get-datasource* path)
apply-schema!))

Expand Down
4 changes: 4 additions & 0 deletions src/me/untethr/nostr/util.clj
Original file line number Diff line number Diff line change
@@ -1,5 +1,9 @@
(ns me.untethr.nostr.util)

(defn current-system-epoch-seconds
[]
(long (/ (System/currentTimeMillis) 1000)))

(defn dissoc-in-if-empty
[m ks]
(let [v (get-in m ks)]
Expand Down
34 changes: 17 additions & 17 deletions test/test/app_test.clj
Original file line number Diff line number Diff line change
Expand Up @@ -39,38 +39,38 @@

(deftest prepare-req-filters*-test
(let [conf (make-test-conf)]
(is (= [] (#'app/prepare-req-filters* conf [])))
(is (= [] (#'app/prepare-req-filters* conf [{:authors []}])))
(is (= [{}] (#'app/prepare-req-filters* conf [{}])))
(is (= [{}] (#'app/prepare-req-filters* conf [{} {}])))
(is (= [{}] (#'app/prepare-req-filters* conf [{} {} {:authors []}])))
(is (= [] (#'app/prepare-req-filters conf [])))
(is (= [] (#'app/prepare-req-filters conf [{:authors []}])))
(is (= [{}] (#'app/prepare-req-filters conf [{}])))
(is (= [{}] (#'app/prepare-req-filters conf [{} {}])))
(is (= [{}] (#'app/prepare-req-filters conf [{} {} {:authors []}])))
(is (= [{} {:authors [support/fake-hex-str]}]
(#'app/prepare-req-filters* conf [{} {} {:authors [support/fake-hex-str]}])))
(#'app/prepare-req-filters conf [{} {} {:authors [support/fake-hex-str]}])))
(is (= [{} {:authors [support/fake-hex-str]}]
(#'app/prepare-req-filters* conf [{} {} {:authors [support/fake-hex-str]}
(#'app/prepare-req-filters conf [{} {} {:authors [support/fake-hex-str]}
{:authors [support/fake-hex-str]}])))
(is (= [{} {:authors [support/fake-hex-str]} {:authors [support/fake-hex-str] :kinds [1 2 3]}]
(#'app/prepare-req-filters* conf [{} {} {:authors [support/fake-hex-str]}
(#'app/prepare-req-filters conf [{} {} {:authors [support/fake-hex-str]}
{:authors [support/fake-hex-str]
:kinds [1 2 3]}]))))
(let [conf (make-test-conf ["1-2"] nil)]
(is (= [] (#'app/prepare-req-filters* conf [])))
(is (= [] (#'app/prepare-req-filters* conf [{:authors []}])))
(is (= [{}] (#'app/prepare-req-filters* conf [{}])))
(is (= [{}] (#'app/prepare-req-filters* conf [{} {}])))
(is (= [{}] (#'app/prepare-req-filters* conf [{} {} {:authors []}])))
(is (= [] (#'app/prepare-req-filters conf [])))
(is (= [] (#'app/prepare-req-filters conf [{:authors []}])))
(is (= [{}] (#'app/prepare-req-filters conf [{}])))
(is (= [{}] (#'app/prepare-req-filters conf [{} {}])))
(is (= [{}] (#'app/prepare-req-filters conf [{} {} {:authors []}])))
(is (= [{} {:authors [support/fake-hex-str]}]
(#'app/prepare-req-filters* conf [{} {} {:authors [support/fake-hex-str]}])))
(#'app/prepare-req-filters conf [{} {} {:authors [support/fake-hex-str]}])))
(is (= [{} {:authors [support/fake-hex-str]}]
(#'app/prepare-req-filters* conf [{} {} {:authors [support/fake-hex-str]}
(#'app/prepare-req-filters conf [{} {} {:authors [support/fake-hex-str]}
{:authors [support/fake-hex-str]}])))
(is (= [{} {:authors [support/fake-hex-str]} {:authors [support/fake-hex-str] :kinds [1 2 3]}]
(#'app/prepare-req-filters* conf [{} {} {:authors [support/fake-hex-str]}
(#'app/prepare-req-filters conf [{} {} {:authors [support/fake-hex-str]}
{:authors [support/fake-hex-str]
:kinds [1 2 3]}])))
;; note: here if none of the kinds a filter references supports, the filter is removed:
(is (= [{} {:authors [support/fake-hex-str]}]
(#'app/prepare-req-filters* conf [{} {} {:authors [support/fake-hex-str]}
(#'app/prepare-req-filters conf [{} {} {:authors [support/fake-hex-str]}
{:authors [support/fake-hex-str]
:kinds [3 4]}])))))

Expand Down
17 changes: 17 additions & 0 deletions test/test/extra_test.clj
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
(ns test.extra-test
(:require [clojure.test :refer :all]
[me.untethr.nostr.extra :as extra]
[me.untethr.nostr.util :as util]))

(deftest query-params->filter
(is (nil? (#'extra/query-params->filter nil)))
(is (nil? (#'extra/query-params->filter {})))
(is (nil? (#'extra/query-params->filter {"unsupported-key" 100})))
(is (= {:limit 10} (#'extra/query-params->filter {"limit" "10"})))
(is (= {:limit 10} (#'extra/query-params->filter {"limit" "10" "unsupported-key" 100})))
(is (= {:limit 10 :authors ["xyz"]} (#'extra/query-params->filter {"limit" "10" "author" "xyz"})))
(with-redefs [util/current-system-epoch-seconds (constantly 1234)]
(is (= {:since 0 :until 1234
:authors ["xyz"] :ids ["abc"]}
(#'extra/query-params->filter
{"since" 0 "until" "now" "author" "xyz" "id" "abc"})))))

0 comments on commit 878de6a

Please sign in to comment.