Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Service definition export format #2397

Merged
merged 8 commits into from
Dec 2, 2018
Merged

Service definition export format #2397

merged 8 commits into from
Dec 2, 2018

Conversation

paulmelnikow
Copy link
Member

@paulmelnikow paulmelnikow commented Nov 26, 2018

Three main goals:

  1. In the front end:
    a. Show form fields and automatically assemble badge URLs (Add Username and Repo Name to Shield Creation fields #701)
    c. Group together examples for the same service
    b. Show deprecated services
  2. Make it easy to changing the schema of examples, thanks to 100% validation. One challenge with frameworks is that when there are typos things fail silently which is pretty unfriendly to developers. The validation should really help with that. (This caught one bug in AUR, though I fixed it in Tweak [AUR] and add static examples #2405 which landed first.)
  3. Produce a service definition export for external tool builders. (Shields data format? #776)
  4. Build toward harmony between the front-end data structure and the examples key in the service classes. I aliased staticPreview to staticExample which starts this process.

The old format:

  • Lacked a consistent machine-readable representation of the fields.
  • Flattened multiple examples for the same service were flattened.
  • Excluded deprecated services.

The new format improves a few things, too:

  • It cleans up the naming. Since this file evolved over time, the names were a bit muddled (i.e. what was an example vs a preview).
  • It duplicated information (like .svg). (I can imagine dropping the .svg from our badge URLs someday, which would make the URLs easier to read and maintain.)
  • For a human reading the YAML file, providing the static example as a deconstructed object is more readable.

Here are a couple snippets:

  - category: build
    name: AppVeyorCi
    isDeprecated: false
    route:
      format: '([^/]+/[^/]+)(?:/(.+))?'
      queryParams: []
    examples:
      - title: AppVeyor
        example: {path: /appveyor/ci/gruntjs/grunt, queryParams: {}}
        preview: {label: build, message: passing, color: brightgreen}
        keywords: []
      - title: AppVeyor branch
        example: {path: /appveyor/ci/gruntjs/grunt/master, queryParams: {}}
        preview: {label: build, message: passing, color: brightgreen}
        keywords: []
  - category: downloads
    name: AmoDownloads
    isDeprecated: false
    examples:
      - title: Mozilla Add-on
        example: {path: /amo/d/dustman, queryParams: {}}
        preview: {path: /amo/d/dustman, queryParams: {}}
        keywords: [amo, firefox]

If you run npm run defs you can see the whole thing in service-definitions.yml. You can also see the schema in services/service-definitions.js.

Though this format is redundant with badge-examples.json, it’s initially being added in parallel. That's to make this change more manageable to review.

After the frontend is modified to consume the new format, the old code (prepareExamples(), export-badge-examples-cli.js, and friends) will be removed.

BaseService.examples has its own schema. This adds new functions for validating and transforming the content of examples, which has its own schema). These are covered by new tests in base.spec.js.

To harmonize BaseService.examples with the new format, this renames two keys:

  • query -> queryParams
  • staticExample -> staticPreview

The old names are aliased, though the aliases needn't be in place for long. Especially since examples is now fully validated, they can be removed with a search and replace soon after this is merged (like #2341). (We could also go with preview instead of staticPreview, which might be even better.)

I would say this closes #776, though there is a bit of housekeeping to do, like formalizing the schema and publishing an initial copy to npm.

Blocks the frontend refactor and the follow-on cleanup work.

@paulmelnikow paulmelnikow added the core Server, BaseService, GitHub auth, Shared helpers label Nov 26, 2018
@shields-ci
Copy link

shields-ci commented Nov 26, 2018

Warnings
⚠️

This PR modified package.json, but not package-lock.json - Perhaps you need to run npm install?

⚠️ This PR modified service code for gemnasium but not its test code.
That's okay so long as it's refactoring existing code.
⚠️ This PR modified service code for gratipay but not its test code.
That's okay so long as it's refactoring existing code.
⚠️ This PR modified service code for imagelayers but not its test code.
That's okay so long as it's refactoring existing code.
⚠️ This PR modified service code for dotnetstatus but not its test code.
That's okay so long as it's refactoring existing code.
⚠️ This PR modified service code for libscore but not its test code.
That's okay so long as it's refactoring existing code.
⚠️ This PR modified service code for magnumci but not its test code.
That's okay so long as it's refactoring existing code.
⚠️ This PR modified service code for snap-ci but not its test code.
That's okay so long as it's refactoring existing code.
⚠️ This PR modified service code for travis but not its test code.
That's okay so long as it's refactoring existing code.
⚠️ This PR modified service code for versioneye but not its test code.
That's okay so long as it's refactoring existing code.
⚠️ This PR modified service code for cauditor but not its test code.
That's okay so long as it's refactoring existing code.
⚠️

Found 'assert' statement added in services/categories.js.
Please ensure tests are written using Chai expect syntax

⚠️ This PR modified service code for bithound but not its test code.
That's okay so long as it's refactoring existing code.
⚠️

Found 'assert' statement added in services/service-definitions.js.
Please ensure tests are written using Chai expect syntax

⚠️ This PR modified service code for issuestats but not its test code.
That's okay so long as it's refactoring existing code.
Messages
📖 ✨ Thanks for your contribution to Shields, @paulmelnikow!

Generated by 🚫 dangerJS

@paulmelnikow paulmelnikow added the blocker PRs and epics which block other work label Nov 26, 2018
Copy link
Member

@chris48s chris48s left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've had a go at reviewing this commenting on the "how", but could you also provide a bit more info on the "why". i.e: what are the problems with badge-examples.json that service-definitions.yml is solving (in terms of data structure rather than code that generates it). Having a slightly clearer idea of the high-level objective and plan for future work here would be useful.

services/transform-example.js Show resolved Hide resolved
services/categories.js Show resolved Hide resolved
.gitignore Show resolved Hide resolved
@paulmelnikow
Copy link
Member Author

I significantly expanded on the why in the top comment. Does that make sense?

@paulmelnikow paulmelnikow temporarily deployed to shields-staging-pr-2397 December 1, 2018 21:12 Inactive
@chris48s
Copy link
Member

chris48s commented Dec 1, 2018

Yeah that extra info makes sense. Thanks.

One other thing I've thought about is performance. I might be wrong on this because I have not tried benchmarking it, but I've got a hunch that the way we're working with the badge examples data structure in https://github.com/badges/shields/blob/master/frontend/lib/prepare-examples.js is one of the things that makes the front end laggy on mobile devices. We certainly spend more time iterating over that data structure than we need to. Maybe its a red herring, but while we're redesigning this data structure, have you given any thought to whether this is more or less efficient to group/search?

@paulmelnikow
Copy link
Member Author

Yea, it would be good to improve the frontend performance. I haven't dug into it that much either. My instinct is it's related to shipping too much JavaScript, DOM manipulation, or requesting all the SVGs.

Dropping Next would probably have a big impact.

We could also inline the SVGs. That should be easy to do for the static examples.

I doubt string matching or data-structure manipulation is what's making the frontend slow, though if we needed to group the examples for performance reasons, we could do it once when they're first loaded.

@paulmelnikow paulmelnikow requested a deployment to shields-staging-pr-2397 December 2, 2018 16:20 Abandoned
@paulmelnikow paulmelnikow merged commit 2045489 into master Dec 2, 2018
@shields-deployment
Copy link

This pull request was merged to master branch. This change is now waiting for deployment, which will usually happen within a few days. Stay tuned by joining our #ops channel on Discord!

After deployment, changes are copied to gh-pages branch:

@paulmelnikow paulmelnikow deleted the service-defs branch December 2, 2018 16:21
paulmelnikow added a commit that referenced this pull request Dec 8, 2018
- The goal of this PR is:
    - Consume the new service-definition format. (#2397)
    - Make the frontend more readable.
- Behavior changes:
    - I changed the **Image** field in the markup modal to show only the path.
    - I added another click-to-select field below that shows the complete URL.
    - This made it easier to suppress the live badge preview while it contains placeholders like `:user` or `:gem`, a minor tweak discussed at #2427 (comment).
    - The search box now searches all categories, regardless of the current page. (This is an improvement, I would say.)
- I did not deliberately address performance, though I ripped out a bunch of anonymous functions and avoided re-filtering all the examples by category on every render, which I expect will not hurt. I haven't really tested this on a mobile connection and it'd be worth doing that.
- It would be great to have some tests of the components, though getting started with that seemed like a big project and I did not want to make this any larger than it already is.

It's a medium-sized refactor:

1. Replace `BadgeExamples`, `Category` and `Badge` component with a completely rewritten `BadgeExamples` component which renders a table of badges, and `CategoryHeading` and `CategoryHeadings` components.
2. Refactor `ExamplesPage` and `SearchResults` components into a new `Main` component.
3. Rewrite the data flow for `MarkupModal`. Rather than rely on unmounting and remounting the component to copy the badge URL into state, employ the `getDerivedStateFromProps` lifecycle method.
4. Remove `prepareExamples` and `all-badge-examples`.
5. Rewrite the `$suggest` schema to harmonize with the service definition format. It's not backward-compatible which means at deploy time there probably will be 10–20 minutes of downtime on that feature, between the first server deploy and the final gh-pages deploy.  🤷‍♂️ (We could leave the old version in place if it seems worth it.)
6. Added two new functions in `make-badge-url` with tests. I removed _most_ of the uses of the old functions, but there are some in parts of the frontend I didn't touch like the static and dynamic badge generators, and again I didn't want to make this any larger than it already is.
7. Fix a couple bugs in the service-definition export.
@paulmelnikow paulmelnikow mentioned this pull request Dec 10, 2018
12 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
blocker PRs and epics which block other work core Server, BaseService, GitHub auth, Shared helpers
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Shields data format?
3 participants