Skip to content

Commit

Permalink
Add Lychee Link Checker into OSD (#938)
Browse files Browse the repository at this point in the history
1. Fix broken links in OSD
2. Generate lycheeexcude list to filter out false negative warnings from test files or external links
3. Add TODO items for internal unavaiable links
4. Integrate with doc link service change.
5. Standardize all opensearch url with `https://opensearch.org/` and add unavilable urls into noDocument list

Signed-off-by: Zuocheng Ding <zding817@gmail.com>
  • Loading branch information
zuochengding authored Dec 8, 2021
1 parent b8c5453 commit 8c11eff
Show file tree
Hide file tree
Showing 187 changed files with 805 additions and 320 deletions.
31 changes: 31 additions & 0 deletions .github/workflows/links_checker.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
# Copyright OpenSearch Contributors
# SPDX-License-Identifier: Apache-2.0

name: Link Checker

on:
push:
branches: [ main ]
pull_request:
branches: [ main ]

jobs:
linkchecker:

runs-on: ubuntu-latest

steps:
- uses: actions/checkout@v2
- name: Load Excludes
run: |
LYCHEE_EXCLUDE=$(sed -e :a -e 'N;s/\n/ --exclude /;ta' .lycheeexclude)
echo "LYCHEE_EXCLUDE=$LYCHEE_EXCLUDE" >> $GITHUB_ENV
- name: Lychee Link Checker
id: lychee
uses: lycheeverse/lychee-action@master
with:
args: --accept=200,403,429 --exclude ${{ env.LYCHEE_EXCLUDE }} --exclude-mail "**/*.html" "**/*.md" "**/*.txt" "**/*.json" "**/*.js" "**/*.ts" "**/*.tsx"
env:
GITHUB_TOKEN: ${{secrets.GITHUB_TOKEN}}
- name: Fail if there were link errors
run: exit ${{ steps.lychee.outputs.exit_code }}
114 changes: 114 additions & 0 deletions .lycheeexclude
Original file line number Diff line number Diff line change
@@ -0,0 +1,114 @@
# Copyright OpenSearch Contributors
# SPDX-License-Identifier: Apache-2.0

# Local or predefined end points
http://localhost
https://localhost
http://127.0.0.1/
https://127.0.0.1/
http://127.0.0.1:10002/bar
http://127.0.0.1:10002/
http://opensearch
https://opensearch
https://opensearch-dashboards
http://opensearch-dashboards
https://opensearch.internal.org/
https://maps.search-services.aws.a2z.com/
https://tiles.maps.search-services.aws.a2z.com/
https://telemetry.opensearch.org/
https://telemetry-staging.opensearch.org/
https://api.github.com/repos/opensearch-project/OpenSearch-Dashboards/
file:///*
git://*

# Dummy urls in tests
http://domain.
http://www.domain.
http://somehost
https://somehost
http://some.host
https://some-host.com/
https://other.some-host.com/
http://test:user@somehost/
https://some.another.host/
http://noone.nowhere.none/
http://bar
http://foo
http://test.com/
https://files.foobar/
https://tiles.foobar/
https://1.1.1.1:9200/
http://192.168.1.1:1234/
http://9.8.7.6/
http://1.2.3.4/
http://8.8.8.8/
https://path.to/
https://example.com/
http://example.com/
https://example.org/
http://some-url/
http://buildurl/
https://dryrun/
https://url/
http://url/
http://notfound.svg/
https://validurl/
https://myopensearch-dashboardsdomain.com
http://myopensearch-dashboardsdomain.com
https://other-opensearch-dashboards.external:8080/
http://myotherdomain.com:5601/
http://myopensearch-dashboardsdomain.com:5601/
https://your-cdn-host.com/
http://not-your-opensearch-dashboards.com/
http://www.mysite.com/
http://myserver.mydomain.com:5601/
https://myexternaldep.com/
http://notlocalhost/
http://site.com/
http://node-b/
http://node-a:9200/
https://node-c/
https://elsewhere
https://opensearch:changeme@example.com:9200
http://test:user@somehost/
https://mycloudinstance:9200/
https://dev-url.co/
https://extenal.org/_search
http://plugins.example.com/
https://build-url/
http://test.com/
https://path.to/
http://site.com/
http://not-your-opensearch-dashboards.com/
http://evil.com/
https://opensearch.org/cool/path
https://opensearch.org/redirect
http://www.opensearch.org/painlessDocs
https://opensearch.org/subscriptions
https://www.hostedgraphite.com/

# External urls
https://www.zeek.org/
http://google.com/
https://api.worldbank.org/
https://vega.github.io/
http://twitter.com/
https://twitter.com/
https://storage.googleapis.com/
http://tools.ietf.org/
https://github.com/
http://github.com/
http://jsperf
https://jsperf
https://gist.githubusercontent.com/
https://nodejs.org/
https://www.npmjs.com/
https://microsoft.github.io/
http://api.worldbank.org/
https://f1542b814f674090afd914960583265f.apm.us-central1.gcp.cloud.es.io/
http://www.matthewcopeland.me/
http://threedubmedia.googlecode.com/*
https://developer.mozilla.org/
https://a.tile.openstreetmap.org/
http://www.creedthoughts.gov
https://media-for-the-masses.theacademyofperformingartsandscience.org/
6 changes: 6 additions & 0 deletions TESTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,12 @@ Say that you would want to debug a test in CI group 1, you can run the following

This will print of an address, to which you could open your chrome browser on your instance and navigate to `chrome://inspect/#devices` and inspect the functional test runner `scripts/functional_tests.js`.

### Additional checks
Make sure you run lint checker before submitting a pull request. To run lint checker:
`node scripts/precommit_hook.js --fix`

Please ensure that you don't introduce any broken links accidently. For any intentional broken link (e.g. dummy url in unit test), you can add it to [lycheeexclude](https://github.com/opensearch-project/OpenSearch-Dashboards/blob/main/.lycheeexclude) allow list specifically.

# Continuous Integration
Automated testing is provided with Jenkins for Continuous Integration. Jenkins enables developers to build, deploy, and automate projects and provides us to run groups of tests quickly. CI groups are ran from the [Jenkinsfile](https://github.com/opensearch-project/OpenSearch-Dashboards/blob/main/Jenkinsfile).

Expand Down
2 changes: 1 addition & 1 deletion package.json
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@
"number": 8467,
"sha": "6cb7fec4e154faa0a4a3fee4b33dfef91b9870d9"
},
"homepage": "https://www.opensearch.org",
"homepage": "https://opensearch.org",
"bugs": {
"url": "http://github.com/opensearch-project/OpenSearch-Dashboards/issues"
},
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -35,14 +35,14 @@ const { TextHighlightRules } = ace.acequire('ace/mode/text_highlight_rules');
const oop = ace.acequire('ace/lib/oop');

export const OpenSearchSqlHighlightRules = function (this: any) {
// See https://www.opensearch.org/guide/en/elasticsearch/reference/current/sql-commands.html
// See https://opensearch.org/docs/latest/search-plugins/sql/index/
const keywords =
'describe|between|in|like|not|and|or|desc|select|from|where|having|group|by|order' +
'asc|desc|pivot|for|in|as|show|columns|include|frozen|tables|escape|limit|rlike|all|distinct|is';

const builtinConstants = 'true|false';

// See https://www.opensearch.org/guide/en/elasticsearch/reference/current/sql-syntax-show-functions.html
// See https://opensearch.org/docs/latest/search-plugins/sql/functions/
const builtinFunctions =
'avg|count|first|first_value|last|last_value|max|min|sum|kurtosis|mad|percentile|percentile_rank|skewness' +
'|stddev_pop|sum_of_squares|var_pop|histogram|case|coalesce|greatest|ifnull|iif|isnull|least|nullif|nvl' +
Expand All @@ -56,7 +56,7 @@ export const OpenSearchSqlHighlightRules = function (this: any) {
'|ltrim|octet_length|position|repeat|replace|right|rtrim|space|substring|ucase|cast|convert|database|user|st_astext|st_aswkt' +
'|st_distance|st_geometrytype|st_geomfromtext|st_wkttosql|st_x|st_y|st_z|score';

// See https://www.opensearch.org/guide/en/elasticsearch/reference/current/sql-data-types.html
// See https://opensearch.org/docs/latest/search-plugins/sql/datatypes/
const dataTypes =
'null|boolean|byte|short|integer|long|double|float|half_float|scaled_float|keyword|text|binary|date|ip|object|nested|time' +
'|interval_year|interval_month|interval_day|interval_hour|interval_minute|interval_second|interval_year_to_month' +
Expand Down
2 changes: 1 addition & 1 deletion packages/osd-babel-preset/README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# @osd/babel-preset

This package contains the shared bits of babel config that we use for transpiling our source code to code compatible with Node.JS and the various [browsers we support](https://www.opensearch.org/support/matrix#matrix_browsers).
This package contains the shared bits of babel config that we use for transpiling our source code to code compatible with Node.JS and the various [browsers we support](https://opensearch.org/docs/latest/dashboards/browser-compatibility/).

## usage

Expand Down
2 changes: 1 addition & 1 deletion packages/osd-dev-utils/certs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ The password used for both of these is "storepass". Other copies are also provid

## Certificate generation

[OpenSearch cert-util](https://www.opensearch.org/guide/en/elasticsearch/reference/current/certutil.html) and [OpenSSL](https://www.openssl.org/) were used to generate these certificates. The following commands were used from the root directory of OpenSearch:
[OpenSearch Self-signed Certificates](https://opensearch.org/docs/latest/security-plugin/configuration/generate-certificates/) and [OpenSSL](https://www.openssl.org/) were used to generate these certificates. The following commands were used from the root directory of OpenSearch:

```
# Generate the PKCS #12 keystore for a CA, valid for 50 years
Expand Down
2 changes: 1 addition & 1 deletion packages/osd-logging/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ is still in `core` for now.
- [Log level](#log-level)
- [Layouts](#layouts)

The way logging works in OpenSearch Dashboards is inspired by `log4j 2` logging framework used by [Elasticsearch](https://www.opensearch.org/guide/en/elasticsearch/reference/current/settings.html#logging).
The way logging works in OpenSearch Dashboards is inspired by `log4j 2` logging framework used by [OpenSearch](https://opensearch.org/docs/latest/opensearch/logs/).
The main idea is to have consistent logging behaviour (configuration, log format etc.) across the entire Elastic Stack
where possible.

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,6 @@
"_cluster/health",
"_cluster/health/{indices}"
],
"documentation": "https://www.opensearch.org/guide/en/elasticsearch/reference/master/cluster-health.html"
"documentation": "https://opensearch.org/docs/latest/opensearch/rest-api/cluster-health/"
}
}
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
{
"cluster.health":{
"documentation":{
"url":"https://www.opensearch.org/guide/en/elasticsearch/reference/master/cluster-health.html",
"url":"https://opensearch.org/docs/latest/opensearch/rest-api/cluster-health/",
"description":"Returns basic information about the health of the cluster."
},
"stability":"stable",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -29,22 +29,22 @@ You can fix this problem by:
https://on.cypress.io/element-is-animating
Because this error occurred during a `after each` hook we are skipping the remaining tests in the current suite: `timeline flyout button`
at cypressErr (http://elastic:changeme@localhost:61141/__cypress/runner/cypress_runner.js:146621:16)
at cypressErrByPath (http://elastic:changeme@localhost:61141/__cypress/runner/cypress_runner.js:146630:10)
at Object.throwErrByPath (http://elastic:changeme@localhost:61141/__cypress/runner/cypress_runner.js:146593:11)
at Object.ensureElementIsNotAnimating (http://elastic:changeme@localhost:61141/__cypress/runner/cypress_runner.js:137560:24)
at ensureNotAnimating (http://elastic:changeme@localhost:61141/__cypress/runner/cypress_runner.js:127434:13)
at runAllChecks (http://elastic:changeme@localhost:61141/__cypress/runner/cypress_runner.js:127522:9)
at retryActionability (http://elastic:changeme@localhost:61141/__cypress/runner/cypress_runner.js:127542:16)
at tryCatcher (http://elastic:changeme@localhost:61141/__cypress/runner/cypress_runner.js:9065:23)
at Function.Promise.attempt.Promise.try (http://elastic:changeme@localhost:61141/__cypress/runner/cypress_runner.js:6339:29)
at tryFn (http://elastic:changeme@localhost:61141/__cypress/runner/cypress_runner.js:140680:21)
at whenStable (http://elastic:changeme@localhost:61141/__cypress/runner/cypress_runner.js:140715:12)
at http://elastic:changeme@localhost:61141/__cypress/runner/cypress_runner.js:140259:16
at tryCatcher (http://elastic:changeme@localhost:61141/__cypress/runner/cypress_runner.js:9065:23)
at Promise._settlePromiseFromHandler (http://elastic:changeme@localhost:61141/__cypress/runner/cypress_runner.js:7000:31)
at Promise._settlePromise (http://elastic:changeme@localhost:61141/__cypress/runner/cypress_runner.js:7057:18)
at Promise._settlePromise0 (http://elastic:changeme@localhost:61141/__cypress/runner/cypress_runner.js:7102:10)]]></failure>
at cypressErr (https://example.com/__cypress/runner/cypress_runner.js:146621:16)
at cypressErrByPath (https://example.com/__cypress/runner/cypress_runner.js:146630:10)
at Object.throwErrByPath (https://example.com/__cypress/runner/cypress_runner.js:146593:11)
at Object.ensureElementIsNotAnimating (https://example.com/__cypress/runner/cypress_runner.js:137560:24)
at ensureNotAnimating (https://example.com/__cypress/runner/cypress_runner.js:127434:13)
at runAllChecks (https://example.com/__cypress/runner/cypress_runner.js:127522:9)
at retryActionability (https://example.com/__cypress/runner/cypress_runner.js:127542:16)
at tryCatcher (https://example.com/__cypress/runner/cypress_runner.js:9065:23)
at Function.Promise.attempt.Promise.try (https://example.com/__cypress/runner/cypress_runner.js:6339:29)
at tryFn (https://example.com/__cypress/runner/cypress_runner.js:140680:21)
at whenStable (https://example.com/__cypress/runner/cypress_runner.js:140715:12)
at https://example.com/__cypress/runner/cypress_runner.js:140259:16
at tryCatcher (https://example.com/__cypress/runner/cypress_runner.js:9065:23)
at Promise._settlePromiseFromHandler (https://example.com/__cypress/runner/cypress_runner.js:7000:31)
at Promise._settlePromise (https://example.com/__cypress/runner/cypress_runner.js:7057:18)
at Promise._settlePromise0 (https://example.com/__cypress/runner/cypress_runner.js:7102:10)]]></failure>
</testcase>
</testsuite>
</testsuites>
Original file line number Diff line number Diff line change
Expand Up @@ -335,8 +335,8 @@ it('rewrites cypress reports with minimal changes', async () => {
You can fix this problem by:
- Passing \`{force: true}\` which disables all error checking
@@ -46,5 +37,5 @@
at Promise._settlePromise (http://elastic:changeme@localhost:61141/__cypress/runner/cypress_runner.js:7057:18)
at Promise._settlePromise0 (http://elastic:changeme@localhost:61141/__cypress/runner/cypress_runner.js:7102:10)]]›‹/failure›
at Promise._settlePromise (https://example.com/__cypress/runner/cypress_runner.js:7057:18)
at Promise._settlePromise0 (https://example.com/__cypress/runner/cypress_runner.js:7102:10)]]›‹/failure›
‹/testcase›
‹/testsuite›
-‹/testsuites›
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,7 @@ export const opensearchTestConfig = new (class OpenSearchTestConfig {
}

getUrlParts() {
// Allow setting one complete TEST_OPENSEARCH_URL for opensearch like https://elastic:changeme@myCloudInstance:9200
// Allow setting one complete TEST_OPENSEARCH_URL for opensearch like https://opensearch:changeme@example.com:9200
if (process.env.TEST_OPENSEARCH_URL) {
const testOpenSearchUrl = url.parse(process.env.TEST_OPENSEARCH_URL);
return {
Expand Down
6 changes: 5 additions & 1 deletion src/core/public/chrome/chrome_service.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -215,7 +215,11 @@ export class ChromeService {
defaultMessage="Support for Internet Explorer will be dropped in future versions of this software, please check {link}."
values={{
link: (
<EuiLink target="_blank" href="https://www.opensearch.org/support/matrix" external>
<EuiLink
target="_blank"
href={docLinks.links.opensearchDashboards.browser}
external
>
<FormattedMessage
id="core.chrome.browserDeprecationLink"
defaultMessage="the support matrix on our website"
Expand Down
6 changes: 0 additions & 6 deletions src/core/public/chrome/constants.ts
Original file line number Diff line number Diff line change
Expand Up @@ -30,12 +30,6 @@
* GitHub history for details.
*/

// export const OPENSEARCH_DASHBOARDS_FEEDBACK_LINK =
// 'https://www.opensearch.org/products/kibana/feedback?blade=kibanafeedback';
// export const OPENSEARCH_DASHBOARDS_ASK_OPENSEARCH_LINK =
// 'https://www.opensearch.org/products/kibana/ask-elastic?blade=kibanaaskelastic';
// export const GITHUB_CREATE_ISSUE_LINK = 'https://github.com/opensearch-project/OpenSearch-Dashboards/issues/new/choose';

export const OPENSEARCH_DASHBOARDS_FEEDBACK_LINK = 'https://github.com/opensearch-project';
export const OPENSEARCH_DASHBOARDS_ASK_OPENSEARCH_LINK = 'https://github.com/opensearch-project';
export const GITHUB_CREATE_ISSUE_LINK =
Expand Down
2 changes: 1 addition & 1 deletion src/core/public/chrome/ui/header/header_help_menu.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -78,7 +78,7 @@ export type ChromeHelpExtensionMenuDiscussLink = EuiButtonEmptyProps & {
linkType: 'discuss';
/**
* URL to discuss page.
* i.e. `https://discuss.opensearch.org/c/${appName}`
* i.e. https://discuss.opendistrocommunity.dev/
*/
href: string;
};
Expand Down
6 changes: 3 additions & 3 deletions src/core/public/doc_links/doc_links_service.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ describe('DocLinksService#start()', () => {
const api = service.start({ injectedMetadata });
expect(api.DOC_LINK_VERSION).toEqual('test-branch');
expect(api.links.opensearchDashboards.introduction).toEqual(
'https://www.opensearch.org/docs/test-branch/dashboards/index/'
'https://opensearch.org/docs/test-branch/dashboards/index/'
);
});

Expand All @@ -52,7 +52,7 @@ describe('DocLinksService#start()', () => {
const api = service.start({ injectedMetadata });
expect(api.DOC_LINK_VERSION).toEqual('latest');
expect(api.links.opensearchDashboards.introduction).toEqual(
'https://www.opensearch.org/docs/latest/dashboards/index/'
'https://opensearch.org/docs/latest/dashboards/index/'
);
});

Expand All @@ -63,7 +63,7 @@ describe('DocLinksService#start()', () => {
const api = service.start({ injectedMetadata });
expect(api.DOC_LINK_VERSION).toEqual('1.1');
expect(api.links.opensearchDashboards.introduction).toEqual(
'https://www.opensearch.org/docs/1.1/dashboards/index/'
'https://opensearch.org/docs/1.1/dashboards/index/'
);
});
});
Loading

0 comments on commit 8c11eff

Please sign in to comment.