Merge pull request #42232 from github/repo-sync

Repo sync

Author: docs-bot

content/copilot/tutorials/copilot-chat-cookbook/analyze-security/manage-dependency-updates.md

@@ -90,8 +90,6 @@ Once you've committed the `dependabot.yml` file to your repository, {% data vari
 
 ### Example scenario
 
-### Example scenario
-
 Let's say your team has been using the basic {% data variables.product.prodname_dependabot %} configuration for a few months, and you're getting lots of dependency update pull requests. However, you're finding that:
 
 * Pull requests are sitting un-reviewed because no one is automatically assigned.

package-lock.json

@@ -195,7 +195,7 @@
     "helmet": "^8.0.0",
     "highlight.js": "^11.11.1",
     "highlightjs-curl": "^1.3.0",
-    "hot-shots": "^11.1.0",
+    "hot-shots": "^12.0.0",
     "html-entities": "^2.5.6",
     "http-proxy-middleware": "^3.0.5",
     "imurmurhash": "^0.1.4",

package.json

@@ -168,3 +168,42 @@ A list of pages available for a fully qualified path containing the target langu
 
 ---
 
+
+## Local Development
+
+### Bootstrap steps
+
+1. Start the dev server:
+   ```bash
+   npm run dev
+   ```
+
+2. Test API endpoints locally:
+   ```bash
+   curl http://localhost:4000/api/article/meta?pathname=/en/get-started
+   curl http://localhost:4000/api/article/body?pathname=/en/get-started
+   curl http://localhost:4000/api/pagelist/en/free-pro-team@latest
+   ```
+
+### Running tests
+
+```bash
+npm run test -- src/article-api/tests
+```
+
+## Ownership
+
+- Team: Docs Engineering
+
+## Transformers
+
+Currently implemented transformers:
+- **REST API transformer** (`rest-transformer.ts`) - Converts REST API autogenerated content
+- **GraphQL transformer** (`graphql-transformer.ts`) - Converts GraphQL API autogenerated content
+- **Audit logs transformer** (`audit-logs-transformer.ts`) - Converts audit log tables to markdown
+
+### Known limitations
+- Some autogenerated content types don't have transformers yet
+- Cache invalidation is manual
+- No built-in rate limiting
+- Limited API versioning

src/article-api/README.md

@@ -53,3 +53,150 @@ Slack: `#docs-engineering`
 Repo: `github/docs-engineering`
 
 If you have a question about automation pipelines, you can ask in the `#docs-engineering` Slack channel. If you notice a problem with one of the automation pipelines, you can open an issue in the `github/docs-engineering` repository.
+
+## Sample Pipeline Template
+
+### Basic pipeline structure
+
+```
+src/<pipeline-name>/
+├── README.md                 # Pipeline documentation
+├── scripts/
+│   └── sync.ts              # Main sync script
+├── data/                    # Generated structured data (optional)
+│   ├── fpt/
+│   ├── ghec/
+│   └── ghes-*/
+├── lib/                     # Utilities and helpers
+├── components/              # React components (if needed)
+├── pages/                   # Next.js pages (if needed)
+└── tests/                   # Pipeline-specific tests
+### Minimal sync script example
+
+```typescript
+// scripts/sync.ts
+import { Command } from 'commander'
+
+const program = new Command()
+  .description('Sync <pipeline-name> data')
+  .option('--source-branch <branch>', 'Source repo branch', 'main')
+  .parse()
+
+const opts = program.opts()
+
+async function main() {
+  // 1. Fetch data from external source
+  // 2. Transform data
+  // 3. Write to data/ directory (or generate Markdown)
+  // 4. Validate output
+}
+
+main()
+```
+
+### Workflow example
+
+```yaml
+# .github/workflows/sync-<pipeline-name>.yml
+name: Sync <pipeline-name>
+
+on:
+  workflow_dispatch:
+    inputs:
+      SOURCE_BRANCH:
+        description: 'Branch to sync from'
+        default: 'main'
+  schedule:
+    - cron: '16 20 * * *'  # Daily
+
+jobs:
+  sync:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - run: npm ci
+      - run: npm run sync-<pipeline-name>
+```
+
+## Ownership Table
+
+| Pipeline | Owning Team | Source Data Owner | Sync Frequency |
+|----------|-------------|-------------------|----------------|
+| REST | Docs Engineering | API Platform | Daily |
+| GraphQL | Docs Engineering | API Platform | Daily |
+| Webhooks | Docs Engineering | API Platform | Daily |
+| CodeQL CLI | Docs Engineering | Code Scanning | Per release |
+| GitHub Apps | Docs Engineering | Integrations | Daily |
+| Audit Logs | Docs Engineering | Enterprise | Daily |
+| Secret Scanning | Docs Engineering | Security | Daily |
+
+## Migration Status
+
+### Active pipelines (✅ Production)
+- REST API - Fully automated, daily sync
+- GraphQL API - Fully automated, daily sync
+- Webhooks - Fully automated, daily sync
+- GitHub Apps - Fully automated, daily sync
+- Secret Scanning - Fully automated, daily sync
+- Audit Logs - Fully automated, daily sync
+
+### Manual sync per release
+- CodeQL CLI - Manual sync per release
+
+### Legacy pipelines (📦 To migrate)
+- None currently identified
+
+### Migration patterns
+
+When migrating manual content to automated pipelines:
+1. **Audit existing content** - Document current structure
+2. **Source data analysis** - Identify gaps between source and docs
+3. **Data enrichment** - Work with source team to add missing fields
+4. **Scraping phase** - Temporarily scrape existing content to preserve prose
+5. **Gradual migration** - Migrate section by section
+6. **Validation** - Compare old vs new output
+7. **Deprecate manual** - Remove manual content once automated is stable
+
+## Shared Components
+
+Components in `src/automated-pipelines/components/` available for reuse:
+- Parameter tables
+- Response schemas
+- Code example formatting
+- Common layout patterns
+
+## Testing Strategy
+
+### Automated pipeline tests
+
+All autogenerated pages tested in `src/automated-pipelines/tests/rendering.ts`:
+- Page renders without errors
+- Required sections present
+- Links are valid
+- Schema validation
+
+### Pipeline-specific tests
+
+Each pipeline in `src/<pipeline-name>/tests/` should test:
+- Data transformation logic
+- Schema validation
+- Version handling
+- Edge cases specific to that pipeline
+
+### Testing locally
+
+```bash
+# Run all automated pipeline tests
+npm run test -- src/automated-pipelines/tests
+
+# Run specific pipeline tests
+npm run test -- src/<pipeline-name>/tests
+```
+
+We are not expecting significant investment here, but we will add and support pipelines as needed to meet business needs.
+
+### Developer experience
+- Pipeline scaffolding tool
+- Validation helpers
+- Testing fixtures
+- Documentation generator

src/automated-pipelines/README.md

@@ -46,3 +46,79 @@ Slack: `#docs-engineering`
 Repo: `github/docs-engineering`
 
 If you have a question about the CodeQL CLI pipeline, you can ask in the `#docs-engineering` Slack channel. If you notice a problem with the CodeQL CLI pipeline, you can open an issue in the `github/docs-engineering` repository.
+
+## Dependency Matrix
+
+### Required dependencies
+
+| Dependency | Purpose | Installation | Notes |
+|------------|---------|--------------|-------|
+| **Pandoc** | Convert ReStructuredText to Markdown | `brew install pandoc` (macOS) | Version 2.x+ required |
+| **semmle-code repo** | Source ReStructuredText files | Clone at root of docs-internal | Private GitHub repo |
+| **Node.js** | Run sync script | Already in project | Version specified in `.nvmrc` |
+
+### Source data location
+- **Repository**: `github/semmle-code` (private)
+- **Branch**: Configurable (default: `main`)
+- **Source files**: ReStructuredText documentation in semmle-code
+- **Output location**: `content/code-security/codeql-cli/codeql-cli-manual/`
+
+### Version compatibility
+- CodeQL CLI releases approximately every 2 weeks
+- Pipeline should be run to align with CLI releases
+- Multiple CLI versions may be documented simultaneously
+
+## Escalation Policy
+
+### Ownership
+- **Team**: Docs Engineering
+- **Source team**: Code Security (CodeQL team)
+
+### Escalation path
+1. **Pipeline failures** → #docs-engineering Slack
+2. **Source data issues** → #code-security-docs Slack
+3. **Pandoc conversion issues** → Check Pandoc version and file format
+4. **Repo access issues** → Request access to `github/semmle-code`
+
+### On-call procedures
+If the pipeline fails:
+1. Check workflow logs for errors
+2. Verify Pandoc is installed and working
+3. Confirm access to `semmle-code` repo
+4. Check for ReStructuredText format changes
+5. Test conversion locally before re-running
+6. Escalate to Code Security team if source data issue
+
+### Common failure modes
+- **Pandoc not found** - Install Pandoc locally or in CI
+- **semmle-code not cloned** - Clone repo at docs-internal root
+- **Conversion errors** - Check for unsupported RST syntax
+- **Branch not found** - Verify branch name in workflow input
+
+## Known Constraints
+
+### Pipeline limitations
+- **Manual trigger only** - No automated schedule (runs on-demand)
+- **Pandoc dependency** - Requires external tool installation
+- **Repo access** - Requires access to private `semmle-code` repo
+- **Local testing** - Must clone semmle-code locally to test
+
+### Content constraints
+- **Frontmatter preservation** - Only first sync sets frontmatter
+- **Introduction placement** - Must be above auto-generation comment
+- **No inline editing** - Cannot edit autogenerated sections
+- **Version handling** - Limited liquid versioning support
+
+### Technical constraints
+- **ReStructuredText format** - Source must be valid RST
+- **Pandoc conversion** - Limited control over conversion output
+- **File naming** - Determined by source file structure
+- **Link rewriting** - May need manual adjustment after generation
+
+### Process constraints
+- **Release alignment** - Sync timing depends on CodeQL releases
+- **PR review required** - Changes must be reviewed before merge
+- **Content coordination** - Writers must coordinate intro additions
+- **Multiple versions** - May need separate syncs for version branches
+
+We are not expecting significant new work to this pipeline, but we will support incoming changes from semmle-code.