Harvesting document links from Slack

[bill-anderson]

We post a lot of useful links into Slack. Going back and finding them is difficult. I propose building a DDW table to allow better access to this resource.

Slack has an API but for the purpose of this proposal I have worked with its export facility.

1. All data from all public channels can be exported in a single pull. Export instructions are available here - https://slack.com/intl/en-gb/help/articles/201658943-Export-your-workspace-data

2. A zip file is exported containing a folder for each channel and within each folder a json file for each day.

3. Links that Slack has been able to read and render an image and summary can be identified by an "attachments" array in the file. This block contains:
   "from_url" - NB "" is represented as "/"
   "title" - title of the article
   "service_name" - the name of the publication or site

4. Links that Slack cannot render can be identified in the "blocks" array.

5. DDW table should contain the following fields:
   channel
   date
   url
   title
   publication ("service_name")
   saved_by ("real_name" in "user_profile")

6. The above provides a basic load. We could then add some ETL logic to build a key-word index.