Sync tracks and exercises from git (exercism#291)

* Make git_head_sha method explicit

* Add git_sha property to tracks and exercises

* Rename parameter to sha

* Allow retrieving diff of commits

* Add blurb to track

* Add F# track to v3-monorepo test repo

* Add F# commits to v3-monorepo

* Allow syncing of track information

* Use exercise SHA for Git reference

* Remove unused require

* Simplify track sync tests

* Remove unneeded guard

* Naming refactoring in track sync

* Add script to recreate DB

* Add deprecated field to exercise

* Add sync to git field

* Revert exercise Git HEAD change

* Use update! method

* Use correct name for file path

* Delegate update! to repo in track

* Don't hardcode head SHA in tests

* Minor refactoring to sync track

* Delegate more calls to git for track

* [F#] Fix filenames in repo

* Add tests for git SHA syncing for exercise

* Update repo

* Sync exercise config updates

* Address review comments

* Add tests for non-ignored exercise files changing

* Add test to sync concept

* Skip exercise sync tests that are failing

* Add synced_to_git_sha field to concept

* Work on adding tests for git synced SHA for concept

* Store concept blurb in DB

* Rename track syncing

* Use Git::Track instance to sync metadata

* Rewrite syncing code to use git better

* Add concept config.json tests

* Add test to verify concept blurb is updated

* Add test for change concept document

* Replace sync attr_reader with instance methods

* Improve readability

* Introduce base sync class

* Add exercise file tests

* Move method

* Update app/commands/git/sync_concept.rb

Co-authored-by: Jeremy Walker <jez.walker@gmail.com>

* Update app/commands/git/sync.rb

Co-authored-by: Jeremy Walker <jez.walker@gmail.com>

* Fix stylecop issue

* Update app/models/git/exercise.rb

Co-authored-by: Jeremy Walker <jez.walker@gmail.com>

* Treat bare repositories as binary files

* Fix seeds

* Update test/factories/track/concepts.rb

* Sync taught concepts

* Fix test

* Test for prerequisites

* Fix naming

* Rename files

* Sync track

* Make exercise sync not add concepts

* Make exercise uuid unique

* Working on importing exercises

* Update seeds to use track

* Update script a bit

* Remove unneeded namespacing

* Add test for command create

* Use attributes for track create

* Introduce attributes parameter for create command

* Add syncing tests

* Fix track sync tests

* Remove unused namespacing

* Temporarily allow importing of concept exercises without title

* Allow skipping of exercises with null as the uuid

* Support unique slug in tracks (exercism#327)

* Revert refactor

* Simplify tests

* Rename exercise sync tests to include type

* Working on practice exercise sync tests

* Disable practice exercise syncing

* Update

* Add F# into v3-monorepo

* Commit F# basics concept blurb change

* Fix concept sync tests to work with new monorepo

* Split concept and practice exercise syncing

* Simplify null check

* Add sync practice exercise tests

* Add concept exercise tests

* Fix practice exercise tests

* Fix concept exercise tests

* Fix sync tests

* Skip importing of practice exercises

* Move synced to head logic

* Move update_git_repo to sync_track

* Renamed git update! to fetch!

* Update synced git sha in before_create

* Make config suffix instead of prefix

* Renamed sync track to current track

* Make sync methods private where possible

* Add comment on what syncing means

* Fix duplicate track slug error in system test

Co-authored-by: Jeremy Walker <jez.walker@gmail.com>

.gitattributes

@@ -2,5 +2,9 @@
 # to be used when mapped in a UNIX docker container
 * text eol=lf
 
+# The bare repositories used for testing purposes
+# should be treated as binary files
+test/repos/** binary
+
 config/credentials/*.yml.enc diff=rails_credentials
 config/credentials.yml.enc diff=rails_credentials

app/commands/concept_exercise/create.rb

@@ -0,0 +1,13 @@
+class ConceptExercise
+  class Create
+    include Mandate
+
+    initialize_with :uuid, :track, :attributes
+
+    def call
+      ConceptExercise.create_or_find_by!(uuid: uuid, track: track) do |ce|
+        ce.attributes = attributes
+      end
+    end
+  end
+end

app/commands/git/sync.rb

@@ -0,0 +1,66 @@
+module Git
+  class Sync
+    include Mandate
+
+    initialize_with :track, :synced_to_git_sha
+
+    def call
+      raise NotImplementedError
+    end
+
+    def filepath_in_diff?(filepath)
+      diff.each_delta.any? do |delta|
+        [delta.old_file[:path], delta.new_file[:path]].include?(filepath)
+      end
+    end
+
+    memoize
+    def git_repo
+      Git::Repository.new(track.slug, repo_url: track.repo_url)
+    end
+
+    memoize
+    def head_git_track
+      Git::Track.new(track.slug, git_repo.head_sha, repo: git_repo)
+    end
+
+    memoize
+    def synced_to_head?
+      current_git_track.commit.oid == head_git_track.commit.oid
+    end
+
+    memoize
+    def track_config_modified?
+      filepath_in_diff?(head_git_track.config_filepath)
+    end
+
+    memoize
+    def concept_exercises_config
+      config[:exercises][:concept]
+    end
+
+    memoize
+    def practice_exercises_config
+      config[:exercises][:practice]
+    end
+
+    memoize
+    def concepts_config
+      config[:concepts]
+    end
+
+    private
+    memoize
+    delegate :config, to: :head_git_track
+
+    memoize
+    def current_git_track
+      Git::Track.new(track.slug, synced_to_git_sha, repo: git_repo)
+    end
+
+    memoize
+    def diff
+      head_git_track.commit.diff(current_git_track.commit)
+    end
+  end
+end

app/commands/git/sync_concept.rb

@@ -0,0 +1,45 @@
+module Git
+  class SyncConcept < Sync
+    include Mandate
+
+    def initialize(concept)
+      super(concept.track, concept.synced_to_git_sha)
+
+      @concept = concept
+    end
+
+    def call
+      return concept.update!(synced_to_git_sha: head_git_concept.commit.oid) unless concept_needs_updating?
+
+      concept.update!(
+        slug: concept_config[:slug],
+        name: concept_config[:name],
+        blurb: concept_config[:blurb],
+        synced_to_git_sha: head_git_concept.commit.oid
+      )
+    end
+
+    private
+    attr_reader :concept
+
+    def concept_needs_updating?
+      return false if synced_to_head?
+      return false unless track_config_modified?
+
+      concept_config[:slug] != concept.slug ||
+        concept_config[:name] != concept.name ||
+        concept_config[:blurb] != concept.blurb
+    end
+
+    memoize
+    def concept_config
+      # TODO: determine what to do when the concept could not be found
+      concepts_config.find { |e| e[:uuid] == concept.uuid }
+    end
+
+    memoize
+    def head_git_concept
+      Git::Concept.new(concept.track.slug, concept.slug, git_repo.head_sha, repo: git_repo)
+    end
+  end
+end

app/commands/git/sync_concept_exercise.rb

@@ -0,0 +1,65 @@
+module Git
+  class SyncConceptExercise < Sync
+    include Mandate
+
+    def initialize(exercise)
+      super(exercise.track, exercise.synced_to_git_sha)
+      @exercise = exercise
+    end
+
+    def call
+      return exercise.update!(synced_to_git_sha: head_git_exercise.commit.oid) unless exercise_needs_updating?
+
+      exercise.update!(
+        slug: exercise_config[:slug],
+        title: exercise_config[:name],
+        deprecated: exercise_config[:deprecated] || false,
+        git_sha: head_git_exercise.commit.oid,
+        synced_to_git_sha: head_git_exercise.commit.oid,
+        taught_concepts: find_concepts(exercise_config[:concepts]),
+        prerequisites: find_concepts(exercise_config[:prerequisites])
+      )
+    end
+
+    private
+    attr_reader :exercise
+
+    def exercise_needs_updating?
+      return false if synced_to_head?
+
+      exercise_config_modified? || exercise_files_modified?
+    end
+
+    def exercise_config_modified?
+      return false unless track_config_modified?
+
+      exercise_config[:slug] != exercise.slug ||
+        exercise_config[:name] != exercise.title ||
+        !!exercise_config[:deprecated] != exercise.deprecated ||
+        exercise_config[:concepts].sort != exercise.taught_concepts.map(&:slug).sort ||
+        exercise_config[:prerequisites].sort != exercise.prerequisites.map(&:slug).sort
+    end
+
+    def exercise_files_modified?
+      head_git_exercise.non_ignored_absolute_filepaths.any? { |filepath| filepath_in_diff?(filepath) }
+    end
+
+    def find_concepts(slugs)
+      slugs.map do |slug|
+        concept_config = concepts_config.find { |e| e[:slug] == slug }
+        ::Track::Concept.find_by!(uuid: concept_config[:uuid])
+      end
+    end
+
+    memoize
+    def exercise_config
+      # TODO: determine what to do when the exercise could not be found
+      concept_exercises_config.find { |e| e[:uuid] == exercise.uuid }
+    end
+
+    memoize
+    def head_git_exercise
+      Git::Exercise.new(exercise.track.slug, exercise.slug, exercise.git_type, git_repo.head_sha, repo: git_repo)
+    end
+  end
+end

app/commands/git/sync_practice_exercise.rb

@@ -0,0 +1,64 @@
+module Git
+  class SyncPracticeExercise < Sync
+    include Mandate
+
+    def initialize(exercise)
+      super(exercise.track, exercise.synced_to_git_sha)
+      @exercise = exercise
+    end
+
+    def call
+      return exercise.update!(synced_to_git_sha: head_git_exercise.commit.oid) unless exercise_needs_updating?
+
+      exercise.update!(
+        slug: exercise_config[:slug],
+        title: exercise_config[:name],
+        deprecated: exercise_config[:deprecated] || false,
+        git_sha: head_git_exercise.commit.oid,
+        synced_to_git_sha: head_git_exercise.commit.oid,
+        prerequisites: find_concepts(exercise_config[:prerequisites])
+      )
+    end
+
+    private
+    attr_reader :exercise
+
+    def exercise_needs_updating?
+      return false if synced_to_head?
+
+      exercise_config_modified? || exercise_files_modified?
+    end
+
+    def exercise_config_modified?
+      return false unless track_config_modified?
+
+      exercise_config[:slug] != exercise.slug ||
+        # TODO: enable the line underneath when (if?) practice exercises have names
+        # exercise_config[:name] != exercise.title ||
+        !!exercise_config[:deprecated] != exercise.deprecated ||
+        exercise_config[:prerequisites].sort != exercise.prerequisites.map(&:slug).sort
+    end
+
+    def exercise_files_modified?
+      head_git_exercise.non_ignored_absolute_filepaths.any? { |filepath| filepath_in_diff?(filepath) }
+    end
+
+    def find_concepts(slugs)
+      slugs.map do |slug|
+        concept_config = concepts_config.find { |e| e[:slug] == slug }
+        ::Track::Concept.find_by!(uuid: concept_config[:uuid])
+      end
+    end
+
+    memoize
+    def exercise_config
+      # TODO: determine what to do when the exercise could not be found
+      practice_exercises_config.find { |e| e[:uuid] == exercise.uuid }
+    end
+
+    memoize
+    def head_git_exercise
+      Git::Exercise.new(exercise.track.slug, exercise.slug, exercise.git_type, git_repo.head_sha, repo: git_repo)
+    end
+  end
+end

app/commands/git/sync_track.rb

@@ -0,0 +1,113 @@
+# Syncing a track involves the following steps:
+#
+# 1. Fetch the latest data of the Git repo
+# 2. Stop syncing if the track is already synced to the Git repo's HEAD commit
+# 3. Update the track's metadata (title, blurb, etc.) if the track data in the
+#    config.json file has changed after last syncing the track.
+# 4. Update the track's concepts if the concept data in the config.json file or
+#    one of the concept files (about.md, links.json, etc.) has changed after
+#    last syncing the track.
+# 5. Update the track's exercises if the exercise data in the config.json file
+#    or one of the exercise files (instructions.md, test suite, etc.) has
+#    changed after last syncing the track.
+module Git
+  class SyncTrack < Sync
+    include Mandate
+
+    def initialize(track)
+      super(track, track.synced_to_git_sha)
+      @track = track
+    end
+
+    def call
+      fetch_git_repo!
+
+      return track.update!(synced_to_git_sha: head_git_track.commit.oid) unless track_needs_updating?
+
+      # TODO: consider raising error when slug in config is different from track slug
+      # TODO: validate track to prevent invalid track data
+      track.update!(
+        blurb: head_git_track.config[:blurb],
+        active: head_git_track.config[:active],
+        title: head_git_track.config[:language],
+        synced_to_git_sha: head_git_track.commit.oid,
+        concepts: concepts,
+        concept_exercises: concept_exercises
+        # TODO: re-enable once we import practice exercises
+        # practice_exercises: practice_exercises
+      )
+
+      track.concepts.each { |concept| Git::SyncConcept.(concept) }
+      track.concept_exercises.each { |concept_exercise| Git::SyncConceptExercise.(concept_exercise) }
+
+      # TODO: re-enable once we import practice exercises
+      # track.practice_exercises.each { |practice_exercise| Git::SyncPracticeExercise.(practice_exercise) }
+    end
+
+    private
+    attr_reader :track
+
+    def concepts
+      # TODO: verify that all exercise concepts and prerequisites are in the concepts section
+      concepts_config.map do |concept_config|
+        ::Track::Concept::Create.(
+          concept_config[:uuid],
+          track,
+          slug: concept_config[:slug],
+          name: concept_config[:name],
+          blurb: concept_config[:blurb],
+          synced_to_git_sha: head_git_track.commit.oid
+        )
+      end
+    end
+
+    def concept_exercises
+      concept_exercises_config.map do |exercise_config|
+        next if exercise_config[:uuid].blank? # TODO: decide if we want to allow null as the uuid
+
+        ::ConceptExercise::Create.(
+          exercise_config[:uuid],
+          track,
+          slug: exercise_config[:slug],
+          # TODO: the DB used title, config.json used name. Consider if we want this
+          # TODO: remove title option once tracks have all updated the config.json
+          title: exercise_config[:name] || exercise_config[:slug].titleize,
+          taught_concepts: find_concepts(exercise_config[:concepts]),
+          prerequisites: find_concepts(exercise_config[:prerequisites]),
+          deprecated: exercise_config[:deprecated] || false,
+          git_sha: head_git_track.commit.oid
+        )
+      end
+    end
+
+    def practice_exercises
+      practice_exercises_config.map do |exercise_config|
+        next if exercise_config[:uuid].blank? # TODO: decide if we want to allow null as the uuid
+
+        ::PracticeExercise::Create.(
+          exercise_config[:uuid],
+          track,
+          slug: exercise_config[:slug],
+          title: exercise_config[:slug].titleize, # TODO: what to do with practice exercise names?
+          prerequisites: find_concepts(exercise_config[:prerequisites]),
+          deprecated: exercise_config[:deprecated] || false,
+          git_sha: head_git_track.commit.oid
+        )
+      end
+    end
+
+    def track_needs_updating?
+      return false if synced_to_head?
+
+      track_config_modified?
+    end
+
+    def find_concepts(concept_slugs)
+      concept_slugs.map { |concept_slug| ::Track::Concept.find_by!(slug: concept_slug) }
+    end
+
+    def fetch_git_repo!
+      git_repo.fetch!
+    end
+  end
+end

app/commands/practice_exercise/create.rb

@@ -0,0 +1,13 @@
+class PracticeExercise
+  class Create
+    include Mandate
+
+    initialize_with :uuid, :track, :attributes
+
+    def call
+      PracticeExercise.create_or_find_by!(uuid: uuid, track: track) do |pe|
+        pe.attributes = attributes
+      end
+    end
+  end
+end