Add upstream patches

.github/workflows/ci.yml

@@ -35,7 +35,7 @@ jobs:
       - name: Set up tree-sitter
         uses: tree-sitter/setup-action/cli@v1
         with:
-          tree-sitter-ref: v0.24.7
+          tree-sitter-ref: v0.25.6
       - name: Run tests
         uses: tree-sitter/parser-test-action@v2
         with:

Cargo.toml

@@ -21,4 +21,4 @@ tree-sitter-language = "0.1.0"
 cc = "1.0"
 
 [dev-dependencies]
-tree-sitter = "0.24"
+tree-sitter = "0.25.6"

codee/HowToMergeUpstream.md

@@ -0,0 +1,153 @@
+# How to merge `tree-sitter-fortran`
+
+This document explains the process for updating our downstream copy of
+`tree-sitter-fortran` repository with the latest changes from the upstream
+repository, ensuring Codee remains up-to-date with new features, fixes, and
+improvements.
+
+## 0. Prerequisites
+
+Before merging upstream changes, ensure that the official `tree-sitter-fortran`
+repository is added as a Git remote and updated. If it is not already
+configured, you can set it up with the following commands:
+
+```bash
+$ git remote add tree-sitter-fortran git@github.com:stadelmanma/tree-sitter-fortran.git
+$ git fetch tree-sitter-fortran master
+```
+
+Additionally, confirm that your local working directory is clean, with no
+uncommitted changes, to avoid conflicts during the merge process.
+
+## 1. Run the update script
+
+Run the [`codee/merge-upstream.sh`](/codee/merge-upstream.bash) script to check
+whether our [`codee/patches`](/codee/patches) are synchronized with our
+downstream repository. If the patches are outdated, you must update and merge
+them to ensure compatibility before proceeding with the main merge process.
+
+Once the patches are verified and up-to-date, the script transitions to a second
+stage. In this stage, the script applies the patches to our downstream project,
+incorporating the latest upstream changes. This process involves using `git am`
+to apply each patch from the `codee/patches` directory. During this step,
+conflicts may arise that you will need to resolve manually.
+
+To identify the source of conflicts, use `git am --show-current-patch=diff`.
+This command highlights the specific patch causing the issue, helping you assess
+and resolve the conflict efficiently. After addressing the conflict, continue
+applying the remaining patches until all patches have been successfully applied.
+
+After all patches are applied, your current branch should resemble the
+following:
+
+```bash
+$ git log --oneline
+92081e66c3b5 Baz
+4b5ccc6f50e4 Bar
+6858ad4b7960 Foo
+# ...
+dbdb4564d47c Merge tree-sitter-fortran/master
+# ...
+```
+
+The commits following `Merge tree-sitter-fortran/master` represent the updated
+versions of the patches with conflicts resolved and context synchronized with
+the latest upstream changes.
+
+At this point, regenerate the patches to reflect the latest changes by running
+the appropriate script. Once regenerated, commit these changes as a fixup commit
+to keep the process organized. After updating the patches, you can squash all
+the commits created by `git am`, as the relevant changes are now captured within
+the patches themselves. There is no need to revisit all the upstream changes;
+instead, focus the review on the updates made to the patches in the fixup
+commit, ensuring that only the necessary adjustments are highlighted.
+
+## 2. Prepare the changes for review
+
+At this stage, it is time to prepare your branch for review. Take into
+consideration that a clean and well-organized commit history will significantly
+improve the review process and make it easier to identify meaningful changes.
+
+For example, the commit that updates the conflicting patches can be separated.
+This commit often contains updates to the context of the patches, but we are
+only interested in those where conflicts actually occurred. You can either
+squash these updates into the merge commit or leave them as a separate fixup
+commit. If you choose the latter, make sure to clearly label these commits with
+a message like "Do not review" so reviewers can easily skip them.
+
+Additionally, ensure that each commit has a clear and concise message, ideally
+with references to relevant upstream changes. Once you have organized and
+squashed any unnecessary commits, your branch should be ready for review.
+Reviewers can then focus solely on the key changes that align with the upstream
+updates, without being bogged down by irrelevant or redundant commits.
+
+Furthermore, during the process, you may identify commits that can be merged
+into the downstream project prior to the actual upstream merge. These commits
+could include improvements that help streamline the review process, such as
+migrating away from deprecated APIs that will be removed in the merge. In such
+cases, it is best to separate these changes and submit them as a distinct
+pull request, to be merged before the main upstream merge. This approach keeps
+the integration process clean, ensures clarity, and avoids potential
+complications by addressing non-merge-related improvements ahead of time.
+
+To further ease the review process, consider submitting a branch with just the
+merge commit. Then, create a separate branch that targets this merge branch,
+containing only the fixup commits. This will allow reviewers to focus
+exclusively on the necessary adjustments, without having to look through the
+upstream commit history.
+
+## 3. Cleanup and pushing the final changes
+
+Once the pull request has been fully reviewed and you have addressed all the
+review comments, it is time to finalize the changes and prepare them for
+merging. The goal here is to ensure that all necessary changes are
+well-organized and that the commit history is clean and concise.
+
+First, take a look at any post-merge commits and decide how to handle them. You
+may place them before the merge commit if applicable, or squash them into the
+merge commit, making sure to retain the squashed commit's message in the merge
+commit's message.
+
+Assuming the merge is already passing tests in the
+`feature/UpgradeTreeSitterFortranAuto` branch, the workflow for this step should
+look something like this:
+
+```bash
+# Branch off feature/UpgradeTreeSitterFortranAuto to update it
+$ git switch -c feature/UpdatedUpgradeTreeSitterFortranAuto feature/UpgradeTreeSitterFortranAuto
+# Merge changes from origin/codee and solve any potential conflicts with newer
+# codee changes
+$ git merge origin/codee
+# Create a final branch to clean things up
+$ git switch -c feature/FinalUpgradeTreeSitterFortranAuto origin/codee
+# Cherry-pick commits to place before the merge commit
+$ git cherry-pick <commit to place before merge>
+# Re-run the intended merge
+$ git merge --no-commit <merge commit>
+# If merging unrelated histories, you may need to remove deleted files
+$ for f in $(git status --porcelain | grep "UD" | cut -d " " -f 2); do git rm $f; done
+# Don't worry about conflicts; simply restore the original branch
+$ git restore --source feature/UpgradeTreeSitterFortranAuto .
+# Stage everything
+$ git add .
+# Commit to finish the merge, including messages for the remaining post-merge
+# commits that were not moved
+$ git commit
+# Optionally add commits that need to be placed after the merge
+$ git cherry-pick <commit to place after merge>
+# Optionally check that the new branch matches the updated branch
+$ git diff feature/FinalUpgradeTreeSitterFortranAuto feature/UpdatedUpgradeTreeSitterFortranAuto
+# Push the final branch
+$ git push feature/FinalUpgradeTreeSitterFortranAuto origin/feature/FinalUpgradeTreeSitterFortranAuto
+```
+
+Once you have pushed your final branch, open a pull request targeting `codee`
+with `feature/FinalUpgradeTreeSitterFortranAuto` and request any necessary
+approvals. This step should be a formality as everything has already been
+reviewed in the previous pull request.
+
+After the pull request is approved and tests pass, push it to `codee`:
+
+```bash
+$ git push origin feature/FinalUpgradeTreeSitterFortranAuto:codee
+```

codee/merge-upstream.bash

@@ -0,0 +1,197 @@
+#!/usr/bin/env bash
+
+set -e
+
+################################################################################
+# Description
+################################################################################
+# This script is here to assist the maintenance of the upstream patches that are
+# required to support this project.
+#
+# The script is divided in 2 stages:
+#  - Stage 1 is to ensure that the current stored patches are exactly the same
+#    to what is already commited in the repo.
+#  - Stage 2 is to re-apply the patches over a merge-commit on the target branch
+#    and ensure that they are still valid, update them if necessary and change
+#    the stored patches accordingly.
+
+################################################################################
+# Functions
+################################################################################
+function _info() {
+  >&2 echo "## ${@}"
+}
+
+function _error() {
+  >&2 echo "error: ${@}"
+}
+
+function _git() {
+  >&2 echo "+ git ${@}"
+  git -C "${REPO_DIR}" "${@}"
+}
+
+function cleanup() {
+  rm -r "${TMP_DIR}"
+  _git switch "${STARTING_BRANCH}"
+  _git branch -D "${TESTING_BRANCH}"
+}
+
+function format_patch_help() {
+  echo "rm '${PATCHES_DIR}'/* && git format-patch --no-signature --keep-subject --zero-commit --output-directory '${PATCHES_DIR}' '${INITIAL_REF}'..HEAD"
+}
+
+################################################################################
+# Globals
+################################################################################
+# The path to this script parent folder
+SCRIPT_DIR=$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" &>/dev/null && pwd)
+# The path to the root of the git repo
+REPO_DIR="${SCRIPT_DIR}/.."
+# The path to the dir with all the upstream patches
+PATCHES_DIR="${REPO_DIR}/codee/patches"
+# The starting branch in case we want to go back
+STARTING_BRANCH="$(_git branch --show-current)"
+# The branch that we will use to detect that the first stage was done correctly
+TARGET_BRANCH="feature/UpgradeTreeSitterFortranAuto"
+# Name of a temporary branch that we can delete later
+TESTING_BRANCH="merge-upstream-testing-${RANDOM}"
+# The name of the upstream git remote repo
+MERGE_WITH="${MERGE_WITH:-tree-sitter-fortran/master}"
+# List of git refspecs for the upstream code excluding downstream
+DOWNSTREAM_PATHS=(:/ :^codee :^src/tree_sitter :^src/grammar.json :^src/node-types.json :^src/parser.c)
+
+################################################################################
+# Script
+################################################################################
+if [[ $# != 0 ]]; then
+  >&2 echo "usage: ${0}
+
+  Helper script to help merge (and verify our changes in) upstream code.
+
+  This script parameters aren't supposed to be modified, so they are coded as
+  variables inside the script."
+  exit 1
+fi
+
+if [[ "${STARTING_BRANCH}" != "${TARGET_BRANCH}" ]]; then
+  _info "Stage 1: Validate that the existing patches represent the current state
+  - First, create a new '${TESTING_BRANCH}' branch to perform the validation
+  - On that branch, reset the upstream folders to the upstream version
+  - Apply all the stored patches and check if they match the original branch
+    - If so, proceed to the second stage
+    - If not, give instructions on how to fix the issue
+"
+  STAGE2=false
+else
+  _info "Stage 2: Finish the upstream update
+  - It is assumed that the first stage finished successfully.
+  - We will merge the '${MERGE_WITH}' branch discarding all our changes
+  - To then apply all the stored patches one by one
+  - After solving all conflicts, the merge should be ready to:
+    - Pass tests, prepare for review and finally recreate the patches
+"
+  STAGE2=true
+fi
+
+_info "Do you want to proceed (Enter to continue, Ctrl+C to exit)?"
+read
+
+# Always ensure that the working tree and the stage are clean
+if [ "$(_git status --porcelain)" != "" ]; then
+  _error "Your working tree and stage must be clean"
+  exit 1
+fi
+
+# In the first stage, sync up the remote branch
+if ! _git describe "${MERGE_WITH}" &>/dev/null; then
+  _error "there is no upstream to merge with ('${MERGE_WITH}'). You can add it
+with:
+
+  git remote add 'tree-sitter-fortran' git@github.com:stadelmanma/tree-sitter-fortran.git
+  git fetch tree-sitter-fortran master"
+  exit 1
+fi
+
+if [ "${STAGE2}" == false ]; then
+  # The target branch shouldn't exist on the first stage
+  if git show-ref --quiet --verify "refs/heads/${TARGET_BRANCH}"; then
+    _error "target branch '${TARGET_BRANCH}' must not exist on stage1"
+    exit 1
+  fi
+
+  UPSTREAM_TARGET=$(_git merge-base HEAD "${MERGE_WITH}")
+  echo "# Stage 1: Verify that the patches are up to date (on current base)"
+  _git switch --create "${TESTING_BRANCH}"
+else
+  UPSTREAM_TARGET="${MERGE_WITH}"
+
+  echo "# Stage 2: Apply the already verified patches (on '${UPSTREAM_TARGET}')"
+  _git merge --strategy=ours --no-commit "${UPSTREAM_TARGET}"
+fi
+
+# Copy the patches to a place outside git (in case you're changing them in the process)
+TMP_DIR="$(mktemp -d)"
+cp "${PATCHES_DIR}"/* "${TMP_DIR}"
+
+# Discard all the upstream changes already committed
+_git restore --source="${UPSTREAM_TARGET}" --worktree --staged -- "${DOWNSTREAM_PATHS[@]}"
+
+# Create a commit to then, apply the patches stored
+if [ "${STAGE2}" == false ]; then
+  _git commit -m "[merge-upstream] Reverted upstream chances since '${UPSTREAM_TARGET}'"
+else
+  _git commit -m "Merge tree-sitter-fortran/master (to be finished)"
+fi
+
+# Store the reference to the start of the format-patch spec
+INITIAL_REF="$(_git rev-parse HEAD)"
+
+# Apply all the patches
+if ! _git am --3way -k "${TMP_DIR}/"*; then
+  _error "Patches are not up to date. You'll need to address the issues
+and redo the patches before trying again.
+
+    $(format_patch_help)
+"
+  exit 3
+fi
+
+if [ "${STAGE2}" == false ]; then
+  # Check if there are any other changes that weren't in the patches
+  if ! _git diff --quiet HEAD.."${STARTING_BRANCH}"; then
+    >&2 echo "There are changes that are still not part of the patches:
+
+    git diff HEAD..'${STARTING_BRANCH}'
+
+  If the changes are new, just commit them as normal. If the changes are
+  modifications of previous commits, you can try to ammend them automatically:
+
+    git diff HEAD..'${STARTING_BRANCH}' | git apply --index
+    git absorb --base '${INITIAL_REF}'
+
+  Review carefully the changes you need to do, and then recreate the patches.
+
+    $(format_patch_help)
+"
+    exit 2
+  fi
+
+  cleanup
+  # Create the target branch for the second stage
+  _git switch --create "${TARGET_BRANCH}"
+
+  echo "
+# Stage 1 completed succesfully. You are ready to jump to stage 2:
+  $0"
+else
+  echo "
+# Stage 2 completed succesfully. You're now on your own. Don't forget to:
+  - Ensure that the patches compile successfully.
+  - Fix the failing tests.
+  - Update the final patches to the new version and open a PR to review them.
+    $(format_patch_help)
+  - Squash all the patches in the final merge commit!
+
+  Good luck!"
+fi

codee/patches/0001-Add-TypeScript-source-annotation.patch

@@ -0,0 +1,26 @@
+From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001
+From: Diego Alonso <diego.alonso@appentra.com>
+Date: Wed, 19 Mar 2025 13:42:50 +0100
+Subject: Add TypeScript source annotation
+
+---
+ grammar.js | 6 ++++++
+ 1 file changed, 6 insertions(+)
+
+diff --git a/grammar.js b/grammar.js
+index bcae961..6c99d98 100644
+--- a/grammar.js
++++ b/grammar.js
+@@ -14,6 +14,12 @@
+ // since the parser doesn't support the ^ regex token, a using a seq
+ // might work as long as the label is optional.
+ //
++
++// This annotation tells typescript where to find the DSL. Useful for LSP
++// navigation and hover documentation
++/// <reference types="tree-sitter-cli/dsl" />
++// @ts-check
++
+ const PREC = {
+   ASSIGNMENT: -10,
+   DEFAULT: 0,