Skip to content

Commit e4f228d

Browse files
author
boonhapus
committed
add process-searchable
1 parent 15f7dc9 commit e4f228d

File tree

1 file changed

+286
-0
lines changed

1 file changed

+286
-0
lines changed

‎docs/guides/process-searchable.md‎

Lines changed: 286 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,286 @@
1+
---
2+
hide:
3+
- navigation
4+
---
5+
6+
# Searchable
7+
8+
Use __ThoughtSpot__, on __ThoughtSpot__!
9+
10+
??? tinm "There is No Magic!"
11+
12+
Remember, __CS Tools__ wraps the __ThoughtSpot__ [__REST APIs__][ts-rest-v2]. This tool uses the following endpoints.
13+
14+
- [`/orgs/search`][ts-rest-orgs-search] *for fetching Org info*{ .fc-gray }
15+
- [`/groups/search`][ts-rest-groups-search] *for fetching Group info, assigned Privileges, as well as their Group Memberships*{ .fc-gray }
16+
- [`/users/search`][ts-rest-users-search] *for fetching User info, as well as their Org and Group Memberships*{ .fc-gray }
17+
- [`/tags/search`][ts-rest-tags-search] *for fetching Tag info*{ .fc-gray }
18+
- [`/metadata/search`][ts-rest-metadata-search] *for fetching Connection, Table, Model, Column, Answer, and Liveboard info, as well as their tags*{ .fc-gray }
19+
- [`/metadata/search`][ts-rest-metadata-search] *for fetching Column details, Synonyms, even on hidden columns*{ .fc-gray }
20+
- [`/metadata/search`][ts-rest-metadata-search] *for fetching Dependents based on all available columns*{ .fc-gray }
21+
- [`/security/metadata/fetch-permissions`][ts-rest-metadata-security] *for fetching Sharing access controls on all available metadata*{ .fc-gray }
22+
- [`/searchdata`][ts-rest-searchdata] *for extracting the __TS: BI Server__ Model*{ .fc-gray }
23+
- [`/logs/fetch`][ts-rest-audit-logs] *for extracting the __Audit Logs__ data feed*{ .fc-gray }
24+
- [`/metadata/tml/export`][ts-rest-metadata-tml-export] *for extracting the TML representation of metadata*{ .fc-gray }
25+
- [`/metadata/tml/import`][ts-rest-metadata-tml-import] *for deploying the Searchable SpotApp to your __ThoughtSpot__ cluster*{ .fc-gray }
26+
27+
!!! tip ""
28+
29+
=== "--help"
30+
??? abstract "Get the Command"
31+
```shell
32+
cs_tools tools searchable --help
33+
```
34+
~cs~tools tools searchable --help
35+
=== "metadata"
36+
??? abstract "Get the Command"
37+
```shell
38+
cs_tools tools searchable metadata --help
39+
```
40+
~cs~tools tools searchable metadata --help
41+
=== "bi-server"
42+
??? abstract "Get the Command"
43+
```shell
44+
cs_tools tools searchable bi-server --help
45+
```
46+
~cs~tools tools searchable bi-server --help
47+
=== "audit-logs"
48+
??? abstract "Get the Command"
49+
```shell
50+
cs_tools tools searchable audit-logs --help
51+
```
52+
~cs~tools tools searchable audit-logs --help
53+
=== "tml"
54+
??? abstract "Get the Command"
55+
```shell
56+
cs_tools tools searchable tml --help
57+
```
58+
~cs~tools tools searchable tml --help
59+
60+
---
61+
62+
### What's in the SpotApp?
63+
64+
You can find the [__raw TML in GitHub__][gh-searchable-tml], the entire package contains..
65+
66+
- All Tables which are [__Modeled for Search__][tsa-mfs]
67+
- A starter Model focused on __User Adoption__{ .fc-green }
68+
- A starter Model focused on __Metadata Lineage__{ .fc-purple }
69+
- A __BETA__{ .fc-blue } starter Model for __Audit Events__
70+
- A __BETA__{ .fc-blue } starter Model for __Metadata Snapshots__
71+
72+
!!! tip ""
73+
74+
=== "Worksheet Column Utilization"
75+
__Learn about how your Worksheets are being used! Here's some questions you can answer with this new Worksheet and Liveboard combo.__{ .fc-purple }
76+
77+
- What's the breakdown of columns in your Worksheet?
78+
- Are there any columns which aren't seeing any use?
79+
- Have you exposed any Hidden columns, which are unintentionally locking dependencies?
80+
- Come to think of it, _just how many dependencies_ are there on this Worksheet?
81+
- Who is creating content that's being heavily used by others?
82+
83+
=== "Overview"
84+
<img src="../../changelog/v1_5_0/worksheet_column_utilization_overview.png">
85+
=== "Dependencies"
86+
<img src="../../changelog/v1_5_0/worksheet_column_utilization_dependencies.png">
87+
=== "Influencers"
88+
<img src="../../changelog/v1_5_0/worksheet_column_utilization_influencers.png">
89+
90+
=== "TS BI Server Advanced"
91+
__Learn about how well your Cluster is being adopted! Here's some questions you can answer with this new Worksheet and Liveboard combo.__{ .fc-purple }
92+
93+
- When do Users log in and interact with ThoughtSpot?
94+
- What does my Month Active Users look like on Mobile? How about for Search?
95+
- How many Users are losing or re-engaging on the platform?
96+
- What does query latency look like in ThoughtSpot? Is anyone having a poor experience?
97+
- Which Groups contribute the most activity in ThoughtSpot?
98+
99+
=== "Overview"
100+
<img src="../../changelog/v1_5_0/thoughtspot_adoption_overview.png">
101+
=== "Adoption"
102+
<img src="../../changelog/v1_5_0/thoughtspot_adoption_adoption.png">
103+
=== "Health"
104+
<img src="../../changelog/v1_5_0/thoughtspot_adoption_health.png">
105+
=== "Archiver"
106+
<img src="../../changelog/v1_5_0/thoughtspot_adoption_archiver.png">
107+
=== "Groups"
108+
<img src="../../changelog/v1_5_0/thoughtspot_adoption_groups.png">
109+
110+
---
111+
112+
### Gather Metadata
113+
114+
Most of the "data" that lives in your __ThoughtSpot__ cluster is data *__about your data__*. This changes every day as
115+
users build, edit, and delete Models, Answers, and Liveboards in the system.
116+
117+
To get a complete view of your __Metadata__, you will need to run this command __on a regular basis__{ .fc-green }.
118+
119+
!!! tip "Syncer Load Strategy: TRUNCATE"
120+
Since this metadata is a snapshot at the time of running, it is best to use the __Syncer__ `load_strategy=TRUNCATE` for this command.
121+
122+
<sub>
123+
<b class=fc-purple>Haven't learned about Syncers yet?</b> <span class=fc-gray>Head on over to the
124+
[Syncer docs][syncer-base] to figure out how to store your __ThoughtSpot__ metadata in an external data store.
125+
</sub>
126+
127+
!!! tip ""
128+
??? abstract "Get the Command"
129+
```shell
130+
cs_tools tools searchable metadata --help
131+
```
132+
~cs~tools tools searchable metadata --help
133+
134+
??? tinm "There is No Magic!"
135+
136+
The __ThoughtSpot__ team runs this command on a daily cadence. You can find it here in our [__GitHub Actions__](https://github.com/thoughtspot/cs_tools/actions/workflows/fetch-metdata.yaml) and [__Workflow File__](https://github.com/thoughtspot/cs_tools/blob/6a8e43bba3140371109e79ef2de7309b7d2a21e5/.github/workflows/fetch-metdata.yaml#L63-L68).
137+
138+
---
139+
140+
### Mirror TS: BI Server
141+
142+
The central `FACT` table in the __Searcahble SpotApp__ is actually one that lives inside your __Thoughtspot__ platform
143+
already! However, since __Thoughtspot__ does not allow relationships across Connections, we will need to copy that data
144+
into our own database first.
145+
146+
__TS: BI Server__ is a User Activity history table, telling Administrators about actions a user takes while logged in to the system.
147+
148+
To get a complete view of your __User Activity__, you will need to run this command __on a regular basis__{ .fc-green }.
149+
150+
!!! tip "Syncer Load Strategy: UPSERT"
151+
Since this is actual activity data, it is best to use the __Syncer__ `load_strategy=UPSERT` for this command.
152+
153+
<sub>
154+
<b class=fc-purple>Customers often schedule this command more often than their data pull window.</b>
155+
<span class=fc-gray>(eg. if you schedule daily, run for the last 7 days!)</span>
156+
</sub>
157+
158+
!!! tip ""
159+
??? abstract "Get the Command"
160+
```shell
161+
cs_tools tools searchable bi-server --help
162+
```
163+
~cs~tools tools searchable bi-server --help
164+
165+
??? tinm "There is No Magic!"
166+
167+
The __ThoughtSpot__ team runs this command on a daily cadence. You can find it here in our [__GitHub Actions__](https://github.com/thoughtspot/cs_tools/actions/workflows/fetch-bi-server.yaml) and [__Workflow File__](https://github.com/thoughtspot/cs_tools/blob/6a8e43bba3140371109e79ef2de7309b7d2a21e5/.github/workflows/fetch-bi-data.yaml#L70-L77).
168+
169+
---
170+
171+
### Deploy Searchable SpotApp
172+
173+
Once you're regularly extracting the __Metadata__ and __BI Server__, you are ready to deploy the __Searchable SpotApp__!
174+
175+
??? note "Should I use a new connection?"
176+
It's recommended to __use a separate Connection__{ .fc-purple } for the __CS Tools__ Searchable tables, even if
177+
you're combining them with your own business data. This is primarily so the __ThoughtSpot__ metadata is scoped
178+
separately from data your users would actually interact with.
179+
180+
This command will ask for some details about which Connection can access the __CS Tools__ tables created by your Syncer.
181+
This are normal __ThoughtSpot__ fundamentals -- you just need to know your Connection GUID, and the name of the external
182+
database (or catalog) and schema of where the Searchable data lives.
183+
184+
__You should only need to run this command once!__{ .fc-purple }
185+
186+
!!! tip ""
187+
??? abstract "Get the Command"
188+
```shell
189+
cs_tools tools searchable deploy --help
190+
```
191+
~cs~tools tools searchable deploy --help
192+
193+
---
194+
195+
### Optional: Extract Audit Logs
196+
197+
While __TS: BI Server__ is all about how Users interact with the system, it is also possible to extract __Audit Logs__.
198+
These logs record all the security actions and changes made to objects on your __ThoughtSpot__ platform. Any time an
199+
object is created, updated, deleted, or their access controls change, the Audit Logs will capture and record the event.
200+
201+
__This dataset is very "noisy" and produces a lot of data!__{ .fc-red }
202+
203+
<sup class=fc-gray>The lifetime of this dataset is only 30-45 rolling days of data, depending on the size of your cluster.</sup>
204+
205+
To get a complete view of your __Logs__, you will need to run this command __on a regular basis__{ .fc-green }.
206+
207+
!!! tip "Syncer Load Strategy: UPSERT"
208+
Since this actual log data, it is best to use the __Syncer__ `load_strategy=UPSERT` for this command.
209+
210+
<sub>
211+
<b class=fc-purple>Customers often schedule this command more often than their data pull window.</b>
212+
<span class=fc-gray>(eg. if you schedule daily, run for the last 7 days!)</span>
213+
</sub>
214+
215+
!!! tip ""
216+
??? abstract "Get the Command"
217+
```shell
218+
cs_tools tools searchable audit-logs --help
219+
```
220+
~cs~tools tools searchable audit-logs --help
221+
222+
??? tinm "There is No Magic!"
223+
224+
The __ThoughtSpot__ team runs this command on a daily cadence. You can find it here in our [__GitHub Actions__](https://github.com/thoughtspot/cs_tools/actions/workflows/fetch-audit-logs.yaml) and [__Workflow File__](https://github.com/thoughtspot/cs_tools/blob/6a8e43bba3140371109e79ef2de7309b7d2a21e5/.github/workflows/fetch-audit-logs.yaml#L64-L70).
225+
226+
---
227+
228+
### Optional: Extract TML Snapshots
229+
230+
The `metadata` commands provides a __LOT__ of data, and not all of it is used by the __Searchable SpotApp__ yet. Still,
231+
many customers have asked about other properties of objects over the years. __TML__ is a the derived representation of
232+
all user-facing objects in the system. Nearly every setting available on objects is captured within the __ThoughtSpot
233+
Modeling Language__.
234+
235+
??? bug "With great power comes great ... work effort"
236+
237+
__This is for advanced users only!__{ .fc-red }
238+
239+
This command gives you the ability to export the TML of your objects as JSON, and store it in your database. From
240+
here, you can use JSON queries to pull out the data you need and join it back to the larger data models.
241+
242+
- What Tables have RLS defined on them?
243+
- How many Vizualization on Liveboards have user-defined Formulas?
244+
- What is the formula expression used on my Models?
245+
- How many Users are engaging `vs` or `in` subqueries on their Searches?
246+
- What's the percentage of visualizations that are in `display_mode: TABLE_MODE`?
247+
248+
All of these can be answered with the __TML__!
249+
250+
To get a complete view of __TML snapshots__, you will need to run this command __on a regular basis__{ .fc-green }.
251+
252+
!!! tip "Syncer Load Strategy: UPSERT"
253+
This table has a `snapshot_date` and defaults to tracking edits, it is best to use the __Syncer__ `load_strategy=UPSERT` for this command.
254+
255+
<sub>
256+
<b class=fc-purple>Customers often schedule this command more often than their data pull window.</b>
257+
<span class=fc-gray>(eg. if you schedule daily, run for the last 7 days!)</span>
258+
</sub>
259+
260+
!!! tip ""
261+
??? abstract "Get the Command"
262+
```shell
263+
cs_tools tools searchable tml --help
264+
```
265+
~cs~tools tools searchable tml --help
266+
267+
??? tinm "There is No Magic!"
268+
269+
The __ThoughtSpot__ team runs this command on a daily cadence. You can find it here in our [__GitHub Actions__](https://github.com/thoughtspot/cs_tools/actions/workflows/fetch-tml.yaml) and [__Workflow File__](https://github.com/thoughtspot/cs_tools/blob/6a8e43bba3140371109e79ef2de7309b7d2a21e5/.github/workflows/fetch-tml.yaml#L64-L72).
270+
271+
---
272+
273+
[gh-searchable-tml]: https://github.com/thoughtspot/cs_tools/tree/master/cs_tools/cli/tools/searchable/static
274+
[tsa-mfs]: https://docs.thoughtspot.com/cloud/latest/data-modeling-settings
275+
[syncer-base]: ../../syncer/what-is
276+
[ts-rest-v2]: https://developers.thoughtspot.com/docs/rest-apiv2-reference
277+
[ts-rest-orgs-search]: https://developers.thoughtspot.com/docs/restV2-playground?apiResourceId=http%2Fapi-endpoints%2Forgs%2Fsearch-orgs
278+
[ts-rest-groups-search]: https://developers.thoughtspot.com/docs/restV2-playground?apiResourceId=http%2Fapi-endpoints%2Fgroups%2Fsearch-groups
279+
[ts-rest-users-search]: https://developers.thoughtspot.com/docs/restV2-playground?apiResourceId=http%2Fapi-endpoints%2Fusers%2Fsearch-users
280+
[ts-rest-tags-search]: https://developers.thoughtspot.com/docs/restV2-playground?apiResourceId=http%2Fapi-endpoints%2Ftags%2Fsearch-tags
281+
[ts-rest-metadata-search]: https://developers.thoughtspot.com/docs/restV2-playground?apiResourceId=http%2Fapi-endpoints%2Fmetadata%2Fsearch-metadata
282+
[ts-rest-metadata-security]: https://developers.thoughtspot.com/docs/restV2-playground?apiResourceId=http%2Fapi-endpoints%2Fsecurity%2Ffetch-permissions-on-metadata
283+
[ts-rest-searchdata]: https://developers.thoughtspot.com/docs/restV2-playground?apiResourceId=http%2Fapi-endpoints%2Fdata%2Fsearch-data
284+
[ts-rest-audit-logs]: https://developers.thoughtspot.com/docs/restV2-playground?apiResourceId=http%2Fapi-endpoints%2Flog%2Ffetch-logs
285+
[ts-rest-metadata-tml-export]: https://developers.thoughtspot.com/docs/restV2-playground?apiResourceId=http%2Fapi-endpoints%2Fmetadata%2Fexport-metadata-tml
286+
[ts-rest-metadata-tml-import]: https://developers.thoughtspot.com/docs/restV2-playground?apiResourceId=http%2Fapi-endpoints%2Fmetadata%2Fimport-metadata-tml

0 commit comments

Comments
 (0)