Would like a guide for How-To deploy Amundsen in production

[jornh]

Please add points on what you expect from such a guide in a comment below. I will then try to consolidate input and draft up an outline in this comment.

The guide can end up as /docs/deployment.md is /docs/owners_manual.md better?

Initial outline:

• Basic install of services (in different environments)

  • Docker-compose “vanilla”, but with Gunicorn (WIP optimize docker images #109) data in volumes etc.
  • AWS ECS. original PR: Feature/Instructions to install Amundsen using AWS ECS amundsenfrontendlibrary#216 (or EC2): https://github.com/lyft/amundsenfrontendlibrary/issues/186
  • Kubernetes helm chart install (convert from Compose using https://kompose.io?) (upcoming PR see https://github.com/lyft/amundsen/issues/53#issuecomment-538575978 below)

• Setting up ingest (with or without Airflow, see https://github.com/lyft/amundsen/issues/53#issuecomment-617370073)

  Figure out which parts of this belongs with Architecture.md and which in Databuilder repo?

  • Compared to Quickstart ingest (https://github.com/lyft/amundsen/issues/75)
  • Then mention source by source; Extractor(s), Model, Metadata
    - Table Metadata:
    - Users
    - Table Usage: (How it works and why in https://github.com/lyft/amundsen/issues/381#issuecomment-613387814)
    - ...

• Configuration - custom build of frontend (to not have to maintain a fork we need to get https://github.com/lyft/amundsen/issues/408 transmogrified into proper documentation/tooling)

  • Small tweaks to turn on/off features, adding logo etc. (mostly Done) amundsen-io/amundsenfrontendlibrary@c256115 in PR Update ResourceListItem UI amundsenfrontendlibrary#255
  • Config of email notification/feedback Done in PR Notifications Configs + Doc amundsenfrontendlibrary#291
  • Data preview (integration to SuperSet) - https://github.com/lyft/amundsen/issues/27#issuecomment-517477074 has some draft contextual lead in and reasoning and a link to example setup. But ultimately what ticks off the box for this is Taos Guide in https://github.com/lyft/amundsen/blob/master/docs/tutorials/data-preview-with-superset.md (or on the https://lyft.github.io/amundsen/ site, search for SuperSet!)

• Security

  • Auth - passwords etc.
  • secure communication
  • production grade docker as per Production-ready Docker images (via https://www.youtube.com/watch?v=cDzFm68aMao)

• Backup - initial WiP in https://github.com/lyft/amundsen/issues/53#issuecomment-516159598 below ... current result in https://github.com/lyft/amundsen/issues/381#issuecomment-614534794 - and restore (on K8s) implemented in https://github.com/lyft/amundsen/pull/394

• Monitoring (statsd etc.?)

• Handling upgrades

• ....

[joaolcorreia]

AWS could be common for deployment, possibly using https://aws.amazon.com/ecs/?

[jornh]

Neo4j backup and restore

Install the Neo4j APOC plugin (in a folder next to your example/docker/neo4j/conf/)

	mkdir example/docker/neo4j/plugins
	pushd example/docker/neo4j/plugins
	wget https://github.com/neo4j-contrib/neo4j-apoc-procedures/releases/download/3.3.0.4/apoc-3.3.0.4-all.jar
	popd
	mkdir example/backup

Add volumes for plugins + backup in amundsen-docker.yml:

	      volumes:
	          - ./example/docker/neo4j/conf:/conf
	          - ./example/docker/neo4j/plugins:/plugins
	          - ./example/backup:/backup  
	

Start containers,

Docker-compose -f docker-amundsen.yml up

ingest data via Databuilder

In the Amundsen frontend web, change descriptions. Maybe add owners…

In the Neo4j web console

CALL apoc.export.cypher.schema('/backup/amundsen_schema.cypher')
CALL apoc.export.graphml.all('/backup/amundsen_data.graphml', {useTypes: true, readLabels: true})

Delete the Neo4j graph (still in the Neo4j web console):

MATCH (n)
DETACH DELETE n

Restore the backup (yep, you guessed it, still in the Neo4j console) :

CALL apoc.import.graphml('/backup/amundsen_data.graphml', {useTypes: true, readLabels: true})

ToDo:

• Figure out where CLI/cron job should live: as part of metadata - as shell/cron (wrap in airflow) - as Databuilder - as Airflow Operator
• Test volume add works - does not break for non-existing plugin/backup in repo (or add KeepFolder file)
• Check under what circumstances restore of Schema is needed

---

Related: #196 and slack thread with some script snippets etc

[jornh]

@ttannis we lost access to the useful content of former FE issue https://github.com/lyft/amundsenfrontendlibrary/issues/186 referenced in the snippet shown below.

Can that content be salvaged somehow? E.g. will transferring https://github.com/lyft/amundsenfrontendlibrary/issues/186 in a closed state to here do it?

Basic install of services (in different environments)
Docker-compose “vanilla”, but with Gunicorn, data in volumes etc.
AWS (ECS PR): lyft/amundsenfrontendlibrary#216 (or EC2): lyft/amundsenfrontendlibrary#186
Kubernetes (convert from Compose using https://kompose.io?)

[ttannis]

Transferred that closed issue over: https://github.com/lyft/amundsen/issues/77

[jornh]

Thanks for the quick turnaround on this @ttannis - seems to work nicely!

Also please extend my thanks to other Lyft team members on the recent even higher systematic focus on grooming PRs etc. I think going forward that will really encourage more to hopefully contribute even more!

[javamonkey79]

@jornh I have amundsen on aws eks + k8s + helm now; I will put up a PR next week with docs; I'm not sure if it will fully fulfill this story, or, if I should put up another one. wdyt?

[jornh]

Great @javamonkey79! I think it should definitely tick the Kubernetes box above (I edited a bit above).

Just push what you think is suitable to cover Kubernetes on it's own and we'll figure the rest out later, when there's some good pieces of content it's easy to shuffle around afterwards if needed.

Right now I'm thinking the list above should end up as just a jump list or "annotated ToC" for what the sys-admin would like/need to know. Haven't really figured out how much or little prose will be needed to glue it together... Thoughts are welcome! 😜

[javamonkey79]

Ok @jornh I've got the PR up here; I've opted to not include the aws setup at this point, as it is tied to our org a bit. I might add it later, if there is enough interest. cc @markgrover @feng-tao