[DEV-2583] Add optimized GitBook sync workflow and metadata synchronization script #1711

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Open

tommaso1 wants to merge 14 commits into main from optimize-gitbook-sync

Collaborator

tommaso1 commented Sep 30, 2025 •

edited

Loading

List of Changes

Optimize GitBook sync workflow to reduce API calls and improve performance

Add unified metadata generation script that reduces Strapi API calls from 8 to 5
Implement S3 operations caching to avoid duplicate file downloads
Parallelize all Strapi fetches using Promise.all()
Create optimized workflow that uses the unified script
Maintain backward compatibility with existing workflow

Motivation and Context

How Has This Been Tested?

Frist go to the gitbook docs folder
cd packages/gitbook-docs
compile
npm run compile
set env variables

export GENERATE_URL_METADATA=false
export METADATA_TYPE=guides
export S3_BUCKET_NAME=devportal-d-website-static-content
export ENVIRONMENT=dev

or 

export GENERATE_URL_METADATA=false
export METADATA_TYPE=all
export S3_BUCKET_NAME=devportal-d-website-static-content
export ENVIRONMENT=dev

run the script
npm run sync-all-metadata

Screenshots (if appropriate):

Types of changes

Chore (nothing changes by a user perspective)
Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to not work as expected)

Checklist:

My change requires a change to the documentation.
I have updated the documentation accordingly.

github-user-name added 2 commits

September 30, 2025 14:58


          Add optimized GitBook sync workflow and metadata synchronization script

1b8977e


          autolaunch

e6a384e

tommaso1 requested review from a team, MarcoPonchia and marcobottaro as code owners

September 30, 2025 13:11

changeset-bot bot commented Sep 30, 2025 •

edited

Loading

🦋 Changeset detected

Latest commit: 332a95a

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package

Name	Type
gitbook-docs	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

github-actions bot added changeset packages workflows size/large labels


          remove trigger

5f02245

marcobottaro requested changes

View reviewed changes

packages/gitbook-docs/src/scripts/syncAllMetadata.ts Outdated

+                guides: StrapiGuide[];
+                solutions: StrapiSolution[];
+                releaseNotes: StrapiReleaseNote[];
+                guidesResponse?: any;

Collaborator

marcobottaro Oct 7, 2025

Maybe you should find a more explicit name for *Response attributes

packages/gitbook-docs/src/scripts/syncAllMetadata.ts Outdated

+                }[];
+              }
+              let s3Client: ReturnType<typeof makeS3Client> | undefined;

Collaborator

marcobottaro Oct 7, 2025

Why don't you use S3Client?

Suggested change

      
            let s3Client: ReturnType<typeof makeS3Client> | undefined;
          
            let s3Client: S3Client | undefined;

packages/gitbook-docs/src/scripts/syncAllMetadata.ts Outdated


		let s3Client: ReturnType<typeof makeS3Client> \| undefined;

		function getS3Client(): ReturnType<typeof makeS3Client> {

Collaborator

marcobottaro Oct 7, 2025

Same here

packages/gitbook-docs/src/scripts/syncAllMetadata.ts Outdated

+                    'api/release-notes/?populate[bannerLinks][populate][0]=icon&populate[product][populate][0]=logo&populate[product][populate][1]=bannerLinks.icon&populate[product][populate][2]=overview&populate[product][populate][3]=quickstart_guide&populate[product][populate][4]=release_note&populate[product][populate][5]=api_data_list_page&populate[product][populate][6]=api_data_list_page.apiData.*&populate[product][populate][7]=api_data_list_page.apiData.apiRestDetail.slug&populate[product][populate][8]=api_data_list_page.apiData.apiRestDetail.specUrls&populate[product][populate][9]=api_data_list_page.apiData.apiSoapDetail.*&populate[product][populate][10]=guide_list_page&populate[product][populate][11]=tutorial_list_page&populate[seo][populate]=*,metaImage,metaSocial.image&pagination[pageSize]=1000&pagination[page]=1'
+                  ),
+                  // Guide list pages
+                  getResponseFromStrapi(

Collaborator

marcobottaro Oct 7, 2025

You should use the typed function fetchFromStrapi

packages/gitbook-docs/src/scripts/syncAllMetadata.ts Outdated

Comment on lines 161 to 168

+                  guides: guidesResult.data,
+                  solutions: solutionsResult.data,
+                  releaseNotes: releaseNotesResult.data,
+                  guidesResponse: guidesResult.responseJson,
+                  solutionsResponse: solutionsResult.responseJson,
+                  releaseNotesResponse: releaseNotesResult.responseJson,
+                  guideListPagesResponse,
+                  solutionListPageResponse,

Collaborator

marcobottaro Oct 7, 2025

Is it necessary to distinguish between data, responseJson and full Response?

packages/gitbook-docs/src/scripts/syncAllMetadata.ts Outdated

Comment on lines 101 to 121

+              async function cachedListS3Files(prefix: string): Promise<string[]> {
+                if (s3ListCache.has(prefix)) {
+                  return s3ListCache.get(prefix)!;
+                }
+                const files = await listS3Files(prefix, S3_BUCKET_NAME!, getS3Client());
+                s3ListCache.set(prefix, files);
+                return files;
+              }
+              async function cachedDownloadS3File(filePath: string): Promise<string> {
+                if (s3FileCache.has(filePath)) {
+                  return s3FileCache.get(filePath)!;
+                }
+                const content = await downloadS3File(
+                  filePath,
+                  S3_BUCKET_NAME!,
+                  getS3Client()
+                );
+                s3FileCache.set(filePath, content);
+                return content;
+              }

Collaborator

marcobottaro Oct 7, 2025

You should read files from file system

github-user-name added 2 commits

October 8, 2025 21:36


          Add Strapi response interfaces for guide and solution list pages

3ccea38


          Refactor metadata synchronization script: remove unused S3 caching, i…

c20a346

…mprove directory checks, and streamline file reading

marcobottaro requested changes

View reviewed changes

packages/gitbook-docs/src/scripts/syncAllMetadata.ts Outdated

+                ] = await Promise.all([
+                  // Guides with full populate
+                  fetchFromStrapi<StrapiGuide>(
+                    'api/guides?populate[0]=product&populate[1]=versions&populate[2]=image&populate[3]=mobileImage&populate[4]=bannerLinks&populate[5]=bannerLinks.icon&populate[6]=listItems&populate[7]=seo&populate[8]=seo.metaImage&populate[9]=seo.metaSocial.image&pagination[pageSize]=1000&pagination[page]=1'

Collaborator

marcobottaro Oct 10, 2025

Here we should use Query String's stringify function like in apps/nextjs-website/src/lib/strapi/fetches/fetchGuides.ts.
By doing so, we could manage this query params by declaring a const like in this example:

  populate: {
    image: { populate: '*' },
    mobileImage: { populate: '*' },
    listItems: { populate: '*' },
    versions: { populate: '*' },
    bannerLinks: { populate: ['icon'] },
    seo: { populate: 'metaSocial.image' },
    product: {
      ...productRelationsPopulate,
    },
  },
};```

and then converting it to a string:
`qs.stringify({
    ...guidesPopulate,
  });`

packages/gitbook-docs/src/scripts/syncAllMetadata.ts Outdated

+                ] = await Promise.all([
+                  // Guides with full populate
+                  fetchFromStrapi<StrapiGuide>(
+                    'api/guides?populate[0]=product&populate[1]=versions&populate[2]=image&populate[3]=mobileImage&populate[4]=bannerLinks&populate[5]=bannerLinks.icon&populate[6]=listItems&populate[7]=seo&populate[8]=seo.metaImage&populate[9]=seo.metaSocial.image&pagination[pageSize]=1000&pagination[page]=1'

Collaborator

marcobottaro Oct 10, 2025

You should add &populate[product][populate][10]=use_case_list_page

Collaborator

Sebastiano-Bertolin Oct 10, 2025

This is the current and updated query:

api/guides/?populate[image][populate]=*&populate[mobileImage][populate]=*&populate[listItems][populate]=*&populate[versions][populate]=*&populate[bannerLinks][populate][0]=icon&populate[seo][populate]=metaSocial.image&populate[product][populate][0]=logo&populate[product][populate][1]=bannerLinks.icon&populate[product][populate][2]=overview&populate[product][populate][3]=quickstart_guide&populate[product][populate][4]=release_note&populate[product][populate][5]=api_data_list_page&populate[product][populate][6]=api_data_list_page.apiData.*&populate[product][populate][7]=api_data_list_page.apiData.apiRestDetail.*&populate[product][populate][8]=guide_list_page&populate[product][populate][9]=tutorial_list_page&populate[product][populate][10]=use_case_list_page

Sebastiano-Bertolin requested changes

View reviewed changes

packages/gitbook-docs/src/scripts/syncAllMetadata.ts Outdated

+                  ),
+                  // Guide list pages
+                  fetchFromStrapi<StrapiGuideListPageResponse>(
+                    'api/guide-list-pages/?populate%5Bproduct%5D%5Bpopulate%5D%5B0%5D=logo&populate%5Bproduct%5D%5Bpopulate%5D%5B1%5D=bannerLinks.icon&populate%5Bproduct%5D%5Bpopulate%5D%5B2%5D=overview&populate%5Bproduct%5D%5Bpopulate%5D%5B3%5D=quickstart_guide&populate%5Bproduct%5D%5Bpopulate%5D%5B4%5D=release_note&populate%5Bproduct%5D%5Bpopulate%5D%5B5%5D=api_data_list_page&populate%5Bproduct%5D%5Bpopulate%5D%5B6%5D=api_data_list_page.apiData.%2A&populate%5Bproduct%5D%5Bpopulate%5D%5B7%5D=api_data_list_page.apiData.apiRestDetail.%2A&populate%5Bproduct%5D%5Bpopulate%5D%5B8%5D=guide_list_page&populate%5Bproduct%5D%5Bpopulate%5D%5B9%5D=tutorial_list_page&populate%5BguidesByCategory%5D%5Bpopulate%5D%5B0%5D=guides.mobileImage&populate%5BguidesByCategory%5D%5Bpopulate%5D%5B1%5D=guides.image&populate%5BguidesByCategory%5D%5Bpopulate%5D%5B2%5D=guides.listItems&populate%5BbannerLinks%5D%5Bpopulate%5D%5B0%5D=icon&populate%5Bseo%5D%5Bpopulate%5D=%2A%2CmetaImage%2CmetaSocial.image'

Collaborator

Sebastiano-Bertolin Oct 10, 2025

Same for StrapiGuideListPage:

api/guide-list-pages/?populate[product][populate][0]=logo&populate[product][populate][1]=bannerLinks.icon&populate[product][populate][2]=overview&populate[product][populate][3]=quickstart_guide&populate[product][populate][4]=release_note&populate[product][populate][5]=api_data_list_page&populate[product][populate][6]=api_data_list_page.apiData.*&populate[product][populate][7]=api_data_list_page.apiData.apiRestDetail.*&populate[product][populate][8]=guide_list_page&populate[product][populate][9]=tutorial_list_page&populate[product][populate][10]=use_case_list_page&populate[guidesByCategory][populate][0]=guides.mobileImage&populate[guidesByCategory][populate][1]=guides.image&populate[guidesByCategory][populate][2]=guides.listItems&populate[bannerLinks][populate][0]=icon&populate[seo][populate]=*,metaImage,metaSocial.image

packages/gitbook-docs/src/scripts/syncAllMetadata.ts Outdated

+                  ),
+                  // Release notes with full populate
+                  fetchFromStrapi<StrapiReleaseNote>(
+                    'api/release-notes/?populate[bannerLinks][populate][0]=icon&populate[product][populate][0]=logo&populate[product][populate][1]=bannerLinks.icon&populate[product][populate][2]=overview&populate[product][populate][3]=quickstart_guide&populate[product][populate][4]=release_note&populate[product][populate][5]=api_data_list_page&populate[product][populate][6]=api_data_list_page.apiData.*&populate[product][populate][7]=api_data_list_page.apiData.apiRestDetail.slug&populate[product][populate][8]=api_data_list_page.apiData.apiRestDetail.specUrls&populate[product][populate][9]=api_data_list_page.apiData.apiSoapDetail.*&populate[product][populate][10]=guide_list_page&populate[product][populate][11]=tutorial_list_page&populate[seo][populate]=*,metaImage,metaSocial.image&pagination[pageSize]=1000&pagination[page]=1'

Collaborator

Sebastiano-Bertolin Oct 10, 2025

This is the updated query with use_case_list_page:

api/release-notes/?populate[bannerLinks][populate][0]=icon&populate[product][populate][0]=logo&populate[product][populate][1]=bannerLinks.icon&populate[product][populate][2]=overview&populate[product][populate][3]=quickstart_guide&populate[product][populate][4]=release_note&populate[product][populate][5]=api_data_list_page&populate[product][populate][6]=api_data_list_page.apiData.*&populate[product][populate][7]=api_data_list_page.apiData.apiRestDetail.slug&populate[product][populate][8]=api_data_list_page.apiData.apiRestDetail.specUrls&populate[product][populate][9]=api_data_list_page.apiData.apiSoapDetail.*&populate[product][populate][10]=guide_list_page&populate[product][populate][11]=tutorial_list_page&populate[product][populate][12]=use_case_list_page&populate[seo][populate]=*,metaImage,metaSocial.image&pagination[pageSize]=1000&pagination[page]=1

MarBert requested changes

View reviewed changes

.github/workflows/sync_gitbook_docs_optimized.yaml

Collaborator

MarBert Oct 20, 2025

@tommaso1 i added a step in [CAI-612] which generates and saves on the s3 bucket two files (response from fetchFromStrapi for products and api-datas) and the sitemap.xml. Please don't forget to add the same step here (or in sync all metadata, where is more fitting)

github-user-name added 2 commits

October 20, 2025 23:37


          Add query parameters for Strapi data fetching: enhance population for…

7a8f642

… guides, solutions, and release notes


          Add Strapi product and API data interfaces: enhance data fetching and…

f6751a3

… synchronization

marcobottaro requested changes

View reviewed changes

.github/workflows/sync_gitbook_docs_optimized.yaml Outdated Show resolved Hide resolved

tommaso1 and others added 3 commits

October 22, 2025 23:16


          Update .github/workflows/sync_gitbook_docs_optimized.yaml

f7a1c04

Co-authored-by: marcobottaro <39835990+marcobottaro@users.noreply.github.com>


          Refactor S3 sync step in GitBook docs workflow: move sync command to …

2a7deeb

…a new position to sync metadata before markdown docs


          Refactor S3_PRODUCTS_METADATA_JSON_PATH declaration: streamline forma…

34a62d7

…tting for improved readability

MarBert requested review from MarBert, Sebastiano-Bertolin and marcobottaro

October 28, 2025 10:39

MarBert requested changes

View reviewed changes

.github/workflows/sync_gitbook_docs_optimized.yaml Outdated

+                        GENERATE_URL_METADATA: 'false'
+                        GENERATE_SITEMAP_METADATA: 'true'
+                        SAVE_STRAPI_RESPONSES: 'true'
+                      run: npm run sync-all-metadata -w gitbook-docs

Collaborator

MarBert Oct 28, 2025

sync-all-metadata is called two times in this action. Can we optimize it?

Collaborator

MarBert Oct 28, 2025

We should be able to generate all metadatas before the url parsing step! Move the AWS cretential step up in order to sync the metadatas on the S3 bucket

github-user-name added 3 commits

October 28, 2025 14:58


          Resolve merge conflicts in package.json and guidesMetadataHelper.ts

cef00e0


          Refactor GitBook sync workflow to enhance metadata generation and AWS…

1ede24d

… configuration


          Fix import issues in syncAllMetadata.ts and optimize workflow

08f01ee

- Replace SitemapItem with MetadataItem (sitemapItem.ts was removed)
- Replace writeSitemapJson with writeMetadataJson (function was renamed)
- Optimize sync_gitbook_docs_optimized.yaml workflow:
  * Move AWS credentials before metadata generation
  * Combine two metadata generation steps into one
  * Eliminate duplicate sync-all-metadata calls

Contributor

github-actions bot commented Oct 28, 2025

Branch is not up to date with base branch

@tommaso1 it seems this Pull Request is not updated with base branch.
Please proceed with a merge or rebase to solve this.

github-actions bot added size/small and removed size/large labels

github-actions bot assigned tommaso1

MarBert approved these changes

View reviewed changes


          Merge branch 'main' into optimize-gitbook-sync

332a95a

Contributor

github-actions bot commented Oct 28, 2025 •

edited by atlassian bot

Loading

Jira Pull Request Link

This Pull Request refers to the following Jira issue DEV-2583

Contributor

github-actions bot commented Oct 28, 2025

This PR exceeds the recommended size of 800 lines. Please make sure you are NOT addressing multiple issues with one PR. Note this PR might be rejected due to its size.

github-actions bot added size/large and removed size/small labels

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

MarBert MarBert approved these changes

MarcoPonchia Awaiting requested review from MarcoPonchia MarcoPonchia is a code owner

marcobottaro Awaiting requested review from marcobottaro marcobottaro is a code owner

Sebastiano-Bertolin Awaiting requested review from Sebastiano-Bertolin

Requested changes must be addressed to merge this pull request.

Labels

changeset packages size/large workflows