feat(frontend): update datahub-web client UI code #1806

cptran777 · 2020-08-13T01:01:45Z

This is the current state of open-sourcable UI code in datahub-web. Large scale migration of code, removal of internal only logic that pollutes craftsmanship of UI code, additional features and entities support.

Some things that are included now are:

More extensible UI models as well as better integration with the API
Lots of internal code that was leaked and non-functional in the open source package has been removed to reduce confusion and make a cleaner experience
Better usage of configuration to setup UI rendering logic
More generic use of components and unifying the user experience throughout the application
Better pipeline and methodology for more consistent rollout of changes to UI code

Note that the above are all work in progress, but that's the main reason for the massive file change

Testing Done:

./gradlew datahub-web:emberWorkspaceTest results:

Done in 601.91s.

BUILD SUCCESSFUL in 10m 13s

Checklist

The PR conforms to DataHub's Contributing Guideline (particularly Commit Message Format)
Links to related issues (if applicable)
Tests for the changes have been added/updated (if applicable)
Docs related to the changes have been added/updated (if applicable)

cobolbaby · 2020-08-13T01:30:18Z

What specific content has been changed in web-ui? Why are so many files changed?

mars-lan · 2020-08-13T02:22:57Z

Have you tested this end-to-end using the docker images? @jplaisted I assume we don't have a dev image for frontend yet?

cptran777 · 2020-08-13T03:24:34Z

What specific content has been changed in web-ui? Why are so many files changed?

Hey @cobolbaby , effectively the UI code on github has been stale/behind for about 6+ months. In that time, internally we made a lot of changes for a friendlier open source experience. Lots of code was cleaned up and migrated.

Some things that are included now are:

More extensible UI models as well as better integration with the API
Lots of internal code that was leaked and non-functional in the open source package has been removed to reduce confusion and make a cleaner experience
Better usage of configuration to setup UI rendering logic
More generic use of components and unifying the user experience throughout the application
Better pipeline and methodology for more consistent rollout of changes to UI code

Note that the above are all work in progress, but that's the main reason for the massive file change

cobolbaby · 2020-08-13T03:33:12Z

Does this pull request involve the following features?

cptran777 · 2020-08-13T05:03:31Z

Does this pull request involve the following features?

UI support for customProperties in DatasetProperties aspect #1730

Dataset schema informance show nothing #1728

I'll have to get back to you on * #1728 , as I'm not sure I understand the issue. Will follow up with @mars-lan to get more context. As for * #1730 unfortunately this PR only catches up to state on internal code changes, but that is a new feature we still have in our backlog.

datahub-web/@datahub/data-models/LICENSE.md

jplaisted · 2020-08-13T17:55:55Z

Have you tested this end-to-end using the docker images? @jplaisted I assume we don't have a dev image for frontend yet?

Correct. I was not familiar enough with our frontend stack to make one in that initial PR; it will require a little more dedicated time from me and/or Charlie. Let me make an issue to track it.

jplaisted · 2020-08-13T19:19:14Z

2,600 files changed is... a lot. Are any of these auto generated?

cptran777 · 2020-08-18T18:24:33Z

2,600 files changed is... a lot. Are any of these auto generated?

They're only auto generated in the sense that our script took this directly from the current internal state.

jplaisted · 2020-08-18T18:40:52Z

2,600 files changed is... a lot. Are any of these auto generated?

They're only auto generated in the sense that our script took this directly from the current internal state.

Yeah I meant generated on every build.

You're saying we have 2,600 hand written files?

jplaisted · 2020-08-18T18:46:58Z

datahub-web/@nacho-ui

Why is this here? Is this experimental code (nacho referring to @igbopie ?)? It shouldn't be checked in internally, much less externally...

Also I see a lot of JS files. Are we not fully on TS yet? If those JS files are generated from TS we can delete them, right?

cptran777 · 2020-08-18T20:00:33Z

2,600 files changed is... a lot. Are any of these auto generated?

They're only auto generated in the sense that our script took this directly from the current internal state.

Yeah I meant generated on every build.

You're saying we have 2,600 hand written files?

So I can't give an exact ratio, but there's a large chunk of code that gets produced by script when initializing a new module (i.e. configuration files and such). They're not produced on every build, but they can be thought of as an MP skeleton framework of sorts. But other than that, yeah, this is what happens when the UI code isn't touched for 6 months haha.

cptran777 · 2020-08-18T20:04:16Z

datahub-web/@nacho-ui

Why is this here? Is this experimental code (nacho referring to @igbopie ?)? It shouldn't be checked in internally, much less externally...

Also I see a lot of JS files. Are we not fully on TS yet? If those JS files are generated from TS we can delete them, right?

Because of the way the Ember build works under the hood, not all files can be in JS. (Ember pipeline works some voodoo magic and builds a working tree from TS and configuration type files from all the modules and the main host application and does a lot of merging to get you the final output. A lot of these configuration type files cannot be in TS or else they aren't recognized by the voodoo magic of Ember).

However, all code that we ourselves should be modifying (i.e. not generated boiler plate) should be in TS.

Also, @nacho-ui is actually a library for shared components across the internal data apps that we are currently housing in our own repo due to logistics reasons. The name is definitely a joke on our actual teammate, but the code in there is very real and un-experimental, and necessary for the application to actually run.

cptran777 · 2020-08-19T02:26:43Z

With the latest commit, the UI build is currently in a working state, confirmed end to end using docker images.

igbopie · 2020-08-19T17:43:31Z

datahub-web/@datahub/data-models/addon/constants/entity/person/tabs.ts

 }

 /**
 * Tab properties available for Person entity
 */
-export const personTabProperties: Array<ITabProperties> = [TabProperties.userlists, TabProperties.userownership];
+export const personTabProperties: Array<ITabProperties> = [


Doesnt this mean, we are exposing UMP flows as well as JIT? can you double check?

We don't as the actual render props only take in UserOwnership. That being said, this is a minor leak and we'll have to figure out a better way to do tab properties in the future to address this.

…asets

… changes

cptran777 · 2020-08-26T18:41:11Z

Will be merging this in if there are no further comments.

* Releases updated version of datahub-web client UI code * Fix typo in yarn lock * Change yarn lock to match yarn registry directories * Previous commit missed some paths * Even more changes to yarnlock missing in previous commit * Include codegen file for typings * Add files to get parity for datahub-web and current OS datahub-midtier * Add in typo fix from previous commit - change to proper license * Implement proper OS fix for person entity picture url * Workarounds for open source DH issues * Fixes institutional memory api and removes unopensourced tabs for datasets * Fixes search dataset deprecation and user search issue as a result of changes * Remove internal only options in the avatar menu

* ML Model Schema Initial Version for feedback * Added Deprecation Model * Remove lock files * Committing yarn lock file * Fix Review Comments * Using Common VersionTag Entity * PR Review Comments Round-2 * Updated all model and feature references to MLModel and MLFeature * Addressing PR Comments (Round 3) * Updating Hyperparameter to a Map type * Update to Dataset * Review comments based on RFC * ML Model Schema Initial Version for feedback * Added Deprecation Model * Remove lock files * Committing yarn lock file * Fix Review Comments * Using Common VersionTag Entity * PR Review Comments Round-2 * Updated all model and feature references to MLModel and MLFeature * Addressing PR Comments (Round 3) * Updating Hyperparameter to a Map type * Update to Dataset * fix: modify the etl script dependency (#1726) Co-authored-by: Cobolbaby <Zhang.Xing-Long@inventec.com> * fix: correct the way to catch the exception (#1727) * fix: modify the etl script dependency * fix: Correct the way to catch the exception * fix: Compatible with the following kafka cluster when the Kafka Topic message Key cannot be empty * fix: Adjust the kafka message key; Improve the comment of field * fix: Avro schema required for key Co-authored-by: Cobolbaby <Zhang.Xing-Long@inventec.com> * refactor(models): remove internal cluster model (#1733) * refactor(models): remove internal cluster model Remove internal model which is not used in open source * build(deps): bump lodash from 4.17.15 to 4.17.19 in /datahub-web (#1738) Bumps [lodash](https://github.com/lodash/lodash) from 4.17.15 to 4.17.19. - [Release notes](https://github.com/lodash/lodash/releases) - [Commits](lodash/lodash@4.17.15...4.17.19) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Update README.md * Update README.md * Update README.md * Update the roadmap (#1740) * Update the roadmap - Make short term more like what we're doing this quarter - Medium term is next quarter - Long term is 2 or 3 quarters from now - Visionary is even beyond that Making this PR mostly to discuss the roadmap. I've moved a few items down to "unprioritized"; before merging this we should put these in a category. Mostly saving the state of what I've done so far. * Update roadmap.md Co-authored-by: Mars Lan <mars.th.lan@gmail.com> * Update roadmap.md * Update README.md * doc: add a separate doc to keep track of the full list or links (#1744) * Update README.md * Create links.md * Update README.md * Update links.md * Update README.md * Update README.md * Update features.md * Update faq.md * Update README.md * Update README.md * feat(gms): add postgres & mariadb supports to GMS (#1742) * feat(gms): add postgres & mariadb supports to GMS Also add corresponding docker-compose files * Update README.md * build(frontend): Drop unnecessary DB-related dependencies (#1741) * refactor(frontend): Drop unnecessary DB-related dependencies * Drop unused dependencies from top-level build script * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Update links.md * Update README.md * Doc fixes * Update roadmap.md * Update faq.md * Set theme jekyll-theme-cayman * Create _config.yml * Delete _config.yml * Set theme jekyll-theme-cayman * Update _config.yml * Update _config.yml * build: build GitHub Page from /docs directory (#1750) - Move top-level MD files to /docs and symlink them back - Update all absolute links to files in /docs to relative links * Revert "build: build GitHub Page from /docs directory (#1750)" (#1751) This reverts commit b0f56de. * build: build GitHub Pages from /docs directory (#1752) - Move non-README top-level MD files to /docs - Update all absolute links to files in /docs to relative links - Add a placeholder front page for GitHub Pages * Update README.md * Update README.md * Update README.md * feat(kafka-config): Add ability to configure other Kafka props (#1745) * Integarte spring-kafka & spring-boot for security props - Upgrade spring-kafka to 2.1.14 - Use KafkaListener and KafkaTemplates to enable KafkaAutoConfiguration - Integrates spring-boot's KafkaProperties into spring-kafka's config * Cleanup imports * Add DataHub kafka env vars * Remove kafka-streams dependency * Add KafkaProperties to gms; Add docs * Add to Adoption * Remove KAFKA_BOOTSTRAP_SERVER default Co-authored-by: jsotelo <javier.sotelo@viasat.com> Co-authored-by: Kerem Sahin <ksahin@linkedin.com> * Agenda for next town hall * Update townhalls.md * Update README.md * Update README.md * Add documentation around the DataHub RFC process. (#1754) Other repos have similar RFC processes (though they seem to have a separate repo for their RFC docs). This provides a more structured way for contributors to make siginficant design contributions. #1692 * metadata-models 72.0.8 -> 80.0.0 (#1756) * <refactor>[ingestions]: align the default kafka topics with PR #1756 (#1758) * docs: add a sequence diagram and a description (#1757) * add a sequence diagram and a description * update descrpition based on feedback * Update README.md * Update README.md Co-authored-by: Mars Lan <mars.th.lan@gmail.com> * Update README.md * Fix reflinks in PR template (#1764) * Update kafka-config.md (#1763) Fix name of spring-kafka property to pass SASL_JAAS config * Update entity.md * Update README.md * Update faq.md * Update townhalls.md * Update README.md * Update townhalls.md * Update townhalls.md * docs: move quickstart guide to a separate file under docs (#1765) docs: move quickstart guide to a separate doc under docs directory * Update slack.md * Update README.md * Update slack.md * Update metadata-ingestion.md * Add workflow to check build and tests on PRs + releases. (#1769) PRs are setup to skip docs. Also, only run docker actions on linkedin/datahub (i.e. disable on forks; makes forks nicer since you don't have failing actions). * Update developers.md * Update developers.md * Update README.md * fix(models): remove unused model (#1748) * fix(models): remove unused model Fixes #1719 * Drop DeploymentInfo from Dataset's value model & rebuild snapshot * Update README.md * Add a separate page for previous townhalls * Update for August invite; link to history * Update README.md * build: remove travis (we're using GitHub actions). (#1770) Remove travis (we're using GitHub actions). Also ignore markdown in our current workflows. Also update the README.md badge. * update townhall date * Update README.md * Update townhalls.md * build(docker): build & publish GitHub Package (#1771) * build(docker): build & publish docker images to GitHub Packages Will kepp publishing to Docker Hub meanwhile until all Dockerfiles have been updated to point to GitHub. Fixes #1548 * Rebase & fix dockerfile locations * Update README.md * Fix README.md * docs: add placeholders for advanced topics (#1780) * Create high-cardinality.md * Create pdl-best-practices * Create partial-update.md * Rename pdl-best-practices to pdl-best-practices.md * Create entity-hierarchy.md * docs: more placeholders for advance topics (#1781) * Create aspect-versioning.md * Create derived-aspects.md * Create backfilling.md * Update README.md * Update aspect-versioning.md * Update aspect.md * Update README.md * Update townhall-history.md * Update townhall-history.md * Update rfc.md * refactor(docker): make docker files easier to use during development. (#1777) * Make docker files easier to use during development. During development it quite nice to have docker work with locally built code. This allows you to launch all services very quickly, with your changes, and optionally with debugging support. Changes made to docker files: - Removed all redundant docker-compose files. We now have 1 giant file, and smaller files to use as overrides. - Remove redundant README files that provided little information. - Rename docker/<dir> to match the service name in the docker-compose file for clarity. - Move environment variables to .env files. We only provide dev / the default environment for quickstart. - Add debug options to docker files using multistage build to build minimal images with the idea that built files will be mounted instead. - Add a docker/dev.sh script + compose file to easily use the dev override images (separate tag; images never published; uses debug docker files; mounts binaries to image). - Added docs/docker documentation for this. * build: fix docker actions. (#1787) * bug: Fix docker actions. We renamed directories in docker/ which broke the actions. Also try to refactor the action files a little so that we can run (but not publish) these images on pull requests that change the docker/ dir as an extra check. Note this only seems to be supported by the dockerhub plugin; the github plugin doesn't support this (so that will be an issue when we move to it only). * Drop extra pipes * Update README.md * refactor: remove unused model (#1788) * refactor: remove unused internal models (#1789) * docs: create search-over-new-field.md (#1790) Add a doc on searching over a new field * Update search-onboarding.md * add description field for dataset index mapping (#1791) * docs: how to customize the search experience (#1795) * add description field for dataset index mapping * documentation on how to customize the search experience * feat(ingest): add example crawler for MS SQL (#1803) Also fix the incorrect assumption on column comments & add sample docker-compose file * Add log documentation we didn't end up mounting logs; docker desktop is a better experience * Update townhall-history.md * Update quickstart.md * fix(search): clear description from dataset index when it's cleared (#1808) Fixes #1798 * Update README.md * Revert "Update README.md" This reverts commit 74a0d7b. * Update README.md * Update README.md * Update high-cardinality.md * Update README.md * Update relationship.md * Update high-cardinality.md * Update metadata-models to head! (#1811) metadata-models 80.0.0 -> 90.0.13: 90.0.13: Roll forward: Fix the open source build by avoiding URN method that isn't part of the open source URN. 90.0.2: Refactor listUrnsFromIndex method 90.0.0: Start distinguishing between [] aspects vs null aspects input param 89.0.4: Fix the open source build by avoiding URN method that isn't part of the open source URN. 89.0.2: fix some test case name 89.0.0: META-12686: Made the MXE_v5 topics become strictly ACL'ed to avoid the wildcard write ACL as "MetadataXEvent.+" 88.0.6: change DAO to take Storage Config as input 88.0.3: Add a comment on lack of avro generation for MXEv5 + add MXEv5 to the pegasus validation task. 87.0.15: META-12651: Integrate the metadata-models-ext with metadata-models 87.0.13: add StorageConfig to Local DAO 87.0.3: Treat empty aspect vs optional aspect same until all clients are migrated 87.0.2: Treat empty aspect vs optional aspect differently 87.0.1: META-12533: Skip processing unregistered aspect specific MAE. 83.0.6: action method to return list of urns from strong consistent index 83.0.4: Change input param type for batch backfill 83.0.3: Implement batch backfill 83.0.1: Implement support for OR filter in browse query 82.0.10: Throw UnsupportedOperationException for unsupported condition types in search filter 82.0.6: Implement local secondary backfilling index as part of backfill method 82.0.5: [strongly consistent index] implement getUrns method 82.0.4: Add indexing urn fields to the local secondary index 82.0.0: Render Delta fiels in the MCE_v5. 81.0.1: Add pegasus to avro conversion for FMCE 80.0.4: add get all support for BaseSingleAspectEntitySimpleKeyResource 80.0.2: Add a BaseSearchWriterDAO with an ESBulkWriterDAO implementation. 80.0.1: META-12254: Produce aspect specific MAE with always emit option 80.0.0: Convert getNodesInTraversedPath to getSubgraph to return complete view of the subgraph (nodes+edges) * Update townhalls.md * Update townhalls.md * fix: drop the commits badge as it's flakey * Update README.md * fix: update defaults of aspectNames params (#1815) fix: Update defaults of aspectNames params. The last PR to sync internal code broke the external GMS, as code was now expected aspectNames to be null rather than empty by default. This preventing me logging into DataHub as the corp user request would fail (it assumed I asked for no aspects rather than all aspects). TESTED: Built locally, launched with docker/dev.sh (so used latest frontend, but whatever). Verified I can now log into DataHub, browse and search for datasets, and view my profile. * Update README.md * Update README.md * feat(kubernetes): Improve the security of the kubernetes/helm charts (#1782) * 1747 | remove obsolete yaml files * 1747 | remove configmap and its hardcoded references * 1747 | add missing input parameter of neo4j.host * 1747 | remove obsolete secrets and parameterize the rest * 1747 | auto-generate gms secret * 1747 | remove fullName overrides * 1747 | fix parameters in subchart's values.yaml * 1747 | remove hardcoding from parameters for gms host and port * 1747 | upgrade chart version * 1747 | update helm docs * 1747 | add extraEnv, extraVolume and extraMounts * 1747 | Alters pull policy of images to 'always' for ldh Co-authored-by: shakti-garg <shakti.garg@gmail.com> * Update README.md * feat(data-platforms): adding rest resource for /dataPlatforms and mid-tier support (#1817) * feat(data-platforms): Adding rest resource for /dataPlatforms and mid-tier support * Removed data platforms which are Linkedin internal * docs: add NOTICE (#1810) * Copy NOTICE from wherehows Copies the file from the wherehows branch. * Update notice. * Update links.md * Update links.md * Update README.md * feat(dashboards): RFC for dashboards (#1778) * feature(dashboards): RFC for dashboards * Change directory structure * Create goals & non-goals sections * Removing alternatives section * Update README.md * Update links.md * Update townhalls.md * Update notice to include embedded licenses Also list apache projects specifically. * feat(frontend): update datahub-web client UI code (#1806) * Releases updated version of datahub-web client UI code * Fix typo in yarn lock * Change yarn lock to match yarn registry directories * Previous commit missed some paths * Even more changes to yarnlock missing in previous commit * Include codegen file for typings * Add files to get parity for datahub-web and current OS datahub-midtier * Add in typo fix from previous commit - change to proper license * Implement proper OS fix for person entity picture url * Workarounds for open source DH issues * Fixes institutional memory api and removes unopensourced tabs for datasets * Fixes search dataset deprecation and user search issue as a result of changes * Remove internal only options in the avatar menu * Update search-over-new-field.md * docs: add external link (#1828) * Update README.md * Update links.md * Review comments based on RFC Co-authored-by: cobolbaby <cobolbaby@qq.com> Co-authored-by: Cobolbaby <Zhang.Xing-Long@inventec.com> Co-authored-by: Harsh Shah <hrshah@linkedin.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Mars Lan <mars.th.lan@gmail.com> Co-authored-by: John Plaisted <jplaisted@linkedin.com> Co-authored-by: Kerem Sahin <ksahin@linkedin.com> Co-authored-by: Javier Sotelo <javier.a.sotelo@gmail.com> Co-authored-by: jsotelo <javier.sotelo@viasat.com> Co-authored-by: Jyoti Wadhwani <jywadhwani@linkedin.com> Co-authored-by: Chris Lee <wlee@linkedin.com> Co-authored-by: Liangjun Jiang <ljiang510@gmail.com> Co-authored-by: shakti-garg-saxo <68685481+shakti-garg-saxo@users.noreply.github.com> Co-authored-by: na zhang <nazhang@linkedin.com> Co-authored-by: shakti-garg <shakti.garg@gmail.com> Co-authored-by: Charlie Tran <catran@linkedin.com>

cptran777 requested review from jplaisted, keremsahin1 and jywadhwani August 13, 2020 01:01

Releases updated version of datahub-web client UI code

94173ce

cptran777 force-pushed the update-datahub-web branch from 59aaacf to 94173ce Compare August 13, 2020 01:15

mars-lan changed the title ~~Releases updated version of datahub-web client UI code~~ feat(frontend): update datahub-web client UI code Aug 13, 2020

Fix typo in yarn lock

1e3596a

mars-lan requested a review from igbopie August 13, 2020 02:21

cptran777 added 3 commits August 12, 2020 20:12

Change yarn lock to match yarn registry directories

df3e8e3

Previous commit missed some paths

4a41b97

Even more changes to yarnlock missing in previous commit

8d5dd87

Include codegen file for typings

dfb2285

mars-lan reviewed Aug 13, 2020

View reviewed changes

datahub-web/@datahub/data-models/LICENSE.md Outdated Show resolved Hide resolved

cptran777 added 2 commits August 18, 2020 19:14

Add files to get parity for datahub-web and current OS datahub-midtier

a609fdb

Add in typo fix from previous commit - change to proper license

63394e8

Implement proper OS fix for person entity picture url

33b14ea

igbopie reviewed Aug 19, 2020

View reviewed changes

cptran777 added 4 commits August 20, 2020 12:58

Workarounds for open source DH issues

243bdd9

Fixes institutional memory api and removes unopensourced tabs for dat…

ca0160f

…asets

Fixes search dataset deprecation and user search issue as a result of…

23ecf48

… changes

Remove internal only options in the avatar menu

ac58393

cptran777 merged commit 843a6c5 into master Aug 26, 2020

cptran777 deleted the update-datahub-web branch August 26, 2020 22:44

This was referenced Aug 27, 2020

Pagination in userownership-datasets view doesn't work #1563

Closed

View as JSON doesn't show the schema #1671

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(frontend): update datahub-web client UI code #1806

feat(frontend): update datahub-web client UI code #1806

cptran777 commented Aug 13, 2020 •

edited by mars-lan

Loading

cobolbaby commented Aug 13, 2020

mars-lan commented Aug 13, 2020

cptran777 commented Aug 13, 2020

cobolbaby commented Aug 13, 2020

cptran777 commented Aug 13, 2020

jplaisted commented Aug 13, 2020

jplaisted commented Aug 13, 2020

cptran777 commented Aug 18, 2020

jplaisted commented Aug 18, 2020

jplaisted commented Aug 18, 2020

cptran777 commented Aug 18, 2020

cptran777 commented Aug 18, 2020

cptran777 commented Aug 19, 2020

igbopie Aug 19, 2020

cptran777 Aug 19, 2020

cptran777 commented Aug 26, 2020

feat(frontend): update datahub-web client UI code #1806

feat(frontend): update datahub-web client UI code #1806

Conversation

cptran777 commented Aug 13, 2020 • edited by mars-lan Loading

Checklist

cobolbaby commented Aug 13, 2020

mars-lan commented Aug 13, 2020

cptran777 commented Aug 13, 2020

cobolbaby commented Aug 13, 2020

cptran777 commented Aug 13, 2020

jplaisted commented Aug 13, 2020

jplaisted commented Aug 13, 2020

cptran777 commented Aug 18, 2020

jplaisted commented Aug 18, 2020

jplaisted commented Aug 18, 2020

cptran777 commented Aug 18, 2020

cptran777 commented Aug 18, 2020

cptran777 commented Aug 19, 2020

igbopie Aug 19, 2020

Choose a reason for hiding this comment

cptran777 Aug 19, 2020

Choose a reason for hiding this comment

cptran777 commented Aug 26, 2020

cptran777 commented Aug 13, 2020 •

edited by mars-lan

Loading