support seen endpoint

[sssoleileraaa]

Description

Fixes #132

Support for using the updated SDK and endpoints has partial support in the client in a working branch. We are going to use this client to test that the SDK changes in this PR are working. Remember that seen/unseen is still not working completely in the client and that there will be things to iron out, so we just want to pay attention in the test plan to the SDK working as far as sending seen data back and forth between client and server as a pass-through.

Test Plan

1. Test updated /replies and /submissions endpoints

Test that the existing endpoints are retrieving seen data

1. from develop, run dev server: NUM_SOURCES=0 make dev
2. in a different tab, checkout add-seen-tables branch in the client repo
3. pip uninstall securedrop-sdk
4. pip install git+https://github.com/freedomofpress/securedrop-sdk@support-seen-endpoint#egg=securedrop-sdk (sdk from this pr branch that you are testing)
5. rm -rf ~/.securedrop_client
6. LOGLEVEL=debug ./run.sh --sdc-home ~/.securedrop_client
7. log in as journalist and don't click anything yet!
8. in a different tab, run: sqlite3 ~/.securedrop_client/svs.sqlite
9. run select * from seen_replies;

• confirm that there are no results

9. run select * from seen_messages;

• confirm that there are no results

10. log in a new source, add a file and message submission
11. log in as a journalist, add a reply to that source
12. download the file and message (to mark them as read) from the JI
13. wait for the client sync to pull in the new reply, message, and file
14. back to your sqlite3 tab, run select * from seen_replies; (sometimes you need to wait a minute for the db to be updated)

• confirm that the new reply is now marked as seen (since the sender of the reply automatically sees it)

15. run select * from seen_messages;

• confirm that the new message is now marked as seen

16. run select * from seen_files;

• confirm that the new file is now marked as seen
  (continue on to test 2 below)

2. Test new endpoint /seen

1. as the same source, send another file and message submission
2. in the client, click on the source in the menu to trigger the call to the SDK's seen api
3. in a new tab, run docker exec -it securedrop-dev-0 bash to connect to the dev server
4. run: sqlite3 /var/lib/securedrop/db.sqlite
5. run select * from seen_messages;

• confirm that you see all the source messages in the results (since you clicked on that source)

6. run select * from seen_files;

• confirm that you see all the source files in the results (since you clicked on that source)

[sssoleileraaa]

Many, many tests use the submissions and replies endpoints so it was easier to update all of the cassettes rather than manually do it one by one (there are some manual steps in sdk cassette generation that make it cumbersome), so focus the review on the following files, and make sure make test passes with the new cassette data:

• sdclientapi/__init__.py
• sdclientapi/sdlocalobjects.py
• tests/test_api.py
• tests/test_apiproxy.py

[sssoleileraaa]

I think the test-against-latest-api will fail until freedomofpress/securedrop#5578 is complete. Will look into tomorrow.

[emkll]

Restarted CI after freedomofpress/securedrop#5578 was merged, tests are now passing.

[kushaldas]

Step number 15 and 16 are not showing anything in the seen_messages and seen_files tables.

I can see the file and message in the files and messages tables.

sqlite> select * from files;
1|b94e41d8-150c-4fd2-b0ad-4c7b0bc48e6b|deb_raji.png|2|1269143|/api/v1/sources/e6197e5d-187c-43a6-9771-0e8af638e29f/submissions/b94e41d8-150c-4fd2-b0ad-4c7b0bc48e6b/download|1|1||0|1|2020-10-16 05:50:15.422781
sqlite> select * from messages;
1|05413808-a743-4435-a785-75e83707b7c6|1-whole_tannery-msg.gpg|1|596|/api/v1/sources/e6197e5d-187c-43a6-9771-0e8af638e29f/submissions/05413808-a743-4435-a785-75e83707b7c6/download|1|1||0|Deb Raji|1|2020-10-16 05:49:45.598556
sqlite> select * from seen_messages;
sqlite> select * from seen_files;
sqlite> select * from seen_replies;
1|1|2302a718-592e-429b-921c-3c6d9ca68c3f

From the client log:

2020-10-16 11:44:15,409 - urllib3.connectionpool:205(_new_conn) DEBUG: Starting new HTTP connection (1): localhost:8081
2020-10-16 11:44:15,421 - urllib3.connectionpool:393(_make_request) DEBUG: http://localhost:8081 "GET /api/v1/replies HTTP/1.1" 200 518
2020-10-16 11:44:15,422 - securedrop_client.storage:108(get_remote_data) INFO: Fetched 1 remote sources.
2020-10-16 11:44:15,422 - securedrop_client.storage:109(get_remote_data) INFO: Fetched 2 remote submissions.
2020-10-16 11:44:15,422 - securedrop_client.storage:110(get_remote_data) INFO: Fetched 1 remote replies.
2020-10-16 11:44:15,426 - securedrop_client.storage:189(update_sources) DEBUG: Updated source e6197e5d-187c-43a6-9771-0e8af638e29f
2020-10-16 11:44:15,428 - securedrop_client.storage:89(chronometer) INFO: update_sources duration: 0.0052s
2020-10-16 11:44:15,435 - securedrop_client.storage:288(__update_submissions) DEBUG: Updated File b94e41d8-150c-4fd2-b0ad-4c7b0bc48e6b
2020-10-16 11:44:15,436 - securedrop_client.storage:89(chronometer) INFO: update_files duration: 0.0075s
2020-10-16 11:44:15,442 - securedrop_client.storage:288(__update_submissions) DEBUG: Updated Message 05413808-a743-4435-a785-75e83707b7c6
2020-10-16 11:44:15,443 - securedrop_client.storage:89(chronometer) INFO: update_messages duration: 0.0071s
2020-10-16 11:44:15,447 - securedrop_client.storage:374(update_replies) DEBUG: Updated reply a62e8ec3-7cb5-4794-82e7-39b6bb5cfbba
2020-10-16 11:44:15,448 - securedrop_client.storage:89(chronometer) INFO: update_replies duration: 0.0041s
2020-10-16 11:44:15,449 - urllib3.connectionpool:205(_new_conn) DEBUG: Starting new HTTP connection (1): localhost:8081
2020-10-16 11:44:15,513 - urllib3.connectionpool:393(_make_request) DEBUG: http://localhost:8081 "GET /api/v1/user HTTP/1.1" 200 162
2020-10-16 11:44:15,532 - securedrop_client.gui.widgets:633(on_source_changed) DEBUG: Drawing source conversation for e6197e5d-187c-43a6-9771-0e8af638e29f
2020-10-16 11:44:15,545 - securedrop_client.queue:265(resume_queues) DEBUG: Resuming main queue
2020-10-16 11:44:15,546 - securedrop_client.queue:268(resume_queues) DEBUG: Resuming download queue

[sssoleileraaa]

my best guess is that the records showed up after you checked (sometimes you have to wait a good solid minute). i just ran through the test plan again and see records there, granted the uuid is getting stored in the client table instead of the id, but that is something client-specific to that client test branch add-seen-tables (need to update for journalist_id in submission.seen_by to for journalist_uuid in submission.seen_by and then get the associated id)

could you or someone else run through it again? looking at the test plan, it says:

12. download the file and message (to mark them as read) from the JI
13. wait for the client sync to pull in the new reply, message, and file
14. back to your sqlite3 tab, run select * from seen_replies; (sometimes you need to wait a minute for the db to be updated)

• confirm that the new reply is now marked as seen (since the sender of the reply automatically sees it)

15. run select * from seen_messages;

• confirm that the new message is now marked as seen

16. run select * from seen_files;

• confirm that the new file is now marked as seen

I should have also said "sometimes you need to wait a minute for the db to be updated" for step 15/16.

[sssoleileraaa]

meanwhile, i'll try to think what else could be wrong. maybe there's an extra step in the test plan that i'm doing but not realizing i'm doing.

[sssoleileraaa]

another debugging step:

run select * from files; and if you don't see the correct number of files from your download then exit out of your sqlite3 session and rerun:

• sqlite3 ~/.securedrop_client/svs.sqlite
• select * from files;

make sure you see that the file exists there -- if it doesn't then there won't be a seen_file record for it

[kushaldas]

another debugging step:

run select * from files; and if you don't see the correct number of files from your download then exit out of your sqlite3 session and rerun:

* `sqlite3 ~/.securedrop_client/svs.sqlite`

* `select * from files;`

make sure you see that the file exists there -- if it doesn't then there won't be a seen_file record for it

I saw the file in the files table.

sqlite> select * from files;
1|b94e41d8-150c-4fd2-b0ad-4c7b0bc48e6b|deb_raji.png|2|1269143|/api/v1/sources/e6197e5d-187c-43a6-9771-0e8af638e29f/submissions/b94e41d8-150c-4fd2-b0ad-4c7b0bc48e6b/download|1|1||0|1|2020-10-16 05:50:15.422781

[sssoleileraaa]

Update: per conversation with kushal outside of github, this review will be picked back up on monday. we discovered the issue was in the test plan process on step 12: download the file and message (to mark them as read) from the JI - this needs to be done on the web journalist interface, not the desktop interface.

[kushaldas]

The code change looks good and has tests.

Followed the following test procedure.

1. Test updated /replies and /submissions endpoints

Test that the existing endpoints are retrieving seen data

1. from develop, run dev server: NUM_SOURCES=0 make dev
2. in a different tab, checkout add-seen-tables branch in the client repo
3. pip uninstall securedrop-sdk
4. pip install git+https://github.com/freedomofpress/securedrop-sdk@support-seen-endpoint#egg=securedrop-sdk (sdk from this pr branch that you are testing)
5. rm -rf ~/.securedrop_client
6. LOGLEVEL=debug ./run.sh --sdc-home ~/.securedrop_client
7. log in as journalist and don't click anything yet!
8. in a different tab, run: sqlite3 ~/.securedrop_client/svs.sqlite
9. run select * from seen_replies;

• confirm that there are no results

9. run select * from seen_messages;

• confirm that there are no results

10. log in a new source, add a file and message submission
11. log in as a journalist, add a reply to that source
12. download the file and message (to mark them as read) from the JI
13. wait for the client sync to pull in the new reply, message, and file
14. back to your sqlite3 tab, run select * from seen_replies; (sometimes you need to wait a minute for the db to be updated)

• confirm that the new reply is now marked as seen (since the sender of the reply automatically sees it)

15. run select * from seen_messages;

• confirm that the new message is now marked as seen

16. run select * from seen_files;

• confirm that the new file is now marked as seen
  (continue on to test 2 below)

2. Test new endpoint /seen

1. as the same source, send another file and message submission
2. in the client, click on the source in the menu to trigger the call to the SDK's seen api
3. in a new tab, run docker exec -it securedrop-dev-0 bash to connect to the dev server
4. run: sqlite3 /var/lib/securedrop/db.sqlite
5. run select * from seen_messages;

• confirm that you see all the source messages in the results (since you clicked on that source)

6. run select * from seen_files;

• confirm that you see all the source files in the results (since you clicked on that source)