Skip to content

Commit

Permalink
Env: Only perform expensive install work when required (#23809)
Browse files Browse the repository at this point in the history
  • Loading branch information
noahtallen authored Jul 14, 2020
1 parent 160ed96 commit 87be0aa
Show file tree
Hide file tree
Showing 10 changed files with 404 additions and 76 deletions.
9 changes: 5 additions & 4 deletions packages/env/CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,17 +6,18 @@

- The `config` and `mappings` options in `.wp-env.json` are now merged with any overrides instead of being overwritten.
- The first listed theme is no longer activated when running wp-env start, since this overwrote whatever theme the user manually activated.
- `wp-env start` no longer stops the WordPress instance if it was already started unless it needs to configure WordPress.
- `wp-env start` no longer updates remote sources after first install if the configuration is the same. Use `wp-env start --update` to update sources.

### New Feature

- You may now specify specific configuration for different environments using `env.tests` or `env.development` in `.wp-env.json`.

### Bug Fixes

- `wp-env start` performance is now 2x faster after first install.
- `wp-env start` is significantly faster after first install.

## 1.6.0-rc.0 (2020-06-24)

### Bug Fixes

- `wp-env destroy` now removes dangling docker volumes and networks associated with the WordPress environment.

## 1.4.0 (2020-05-28)
Expand Down
34 changes: 18 additions & 16 deletions packages/env/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -172,20 +172,22 @@ $ wp-env start

`wp-env` creates generated files in the `wp-env` home directory. By default, this is `~/.wp-env`. The exception is Linux, where files are placed at `~/wp-env` [for compatibility with Snap Packages](https://github.com/WordPress/gutenberg/issues/20180#issuecomment-587046325). The `wp-env` home directory contains a subdirectory for each project named `/$md5_of_project_path`. To change the `wp-env` home directory, set the `WP_ENV_HOME` environment variable. For example, running `WP_ENV_HOME="something" wp-env start` will download the project files to the directory `./something/$md5_of_project_path` (relative to the current directory).

### `wp-env start [ref]`
### `wp-env start`

The start command installs and initalizes the WordPress environment, which includes downloading any specified remote sources. By default, `wp-env` will not update or re-configure the environment except when the configuration file changes. Tell `wp-env` to update sources and apply the configuration options again with `wp-env start --update`. This will not overrwrite any existing content.

```sh
wp-env start

Starts WordPress for development on port 8888 (​http://localhost:8888​)
(override with WP_ENV_PORT) and tests on port 8889 (​http://localhost:8889​)
(override with WP_ENV_TESTS_PORT). The current working directory must be a
WordPress installation, a plugin, a theme, or contain a .wp-env.json file.

Starts WordPress for development on port 8888 (override with WP_ENV_PORT) and
tests on port 8889 (override with WP_ENV_TESTS_PORT). The current working
directory must be a WordPress installation, a plugin, a theme, or contain a
.wp-env.json file. After first insall, use the '--update' flag to download updates
to mapped sources and to re-apply WordPress configuration options.

Positionals:
ref A `https://github.com/WordPress/WordPress` git repo branch or commit for
choosing a specific version. [string] [default: "master"]
Options:
--update Download source updates and apply WordPress configuration.
[boolean] [default: false]
```

### `wp-env stop`
Expand Down Expand Up @@ -286,14 +288,14 @@ You can customize the WordPress installation, plugins and themes that the develo
`.wp-env.json` supports six fields for options applicable to both the tests and development instances.
| Field | Type | Default | Description |
| ------------ | -------------- | -------------------------------------- | ------------------------------------------------------------------------------------------------------------------------- |
| `"core"` | `string\|null` | `null` | The WordPress installation to use. If `null` is specified, `wp-env` will use the latest production release of WordPress. |
| `"plugins"` | `string[]` | `[]` | A list of plugins to install and activate in the environment. |
| `"themes"` | `string[]` | `[]` | A list of themes to install in the environment. The first theme in the list will be activated. |
| Field | Type | Default | Description |
| ------------ | -------------- | -------------------------------------- | -------------------------------------------------------------------------------------------------------------------------- |
| `"core"` | `string\|null` | `null` | The WordPress installation to use. If `null` is specified, `wp-env` will use the latest production release of WordPress. |
| `"plugins"` | `string[]` | `[]` | A list of plugins to install and activate in the environment. |
| `"themes"` | `string[]` | `[]` | A list of themes to install in the environment. The first theme in the list will be activated. |
| `"port"` | `integer` | `8888` (`8889` for the tests instance) | The primary port number to use for the installation. You'll access the instance through the port: 'http://localhost:8888'. |
| `"config"` | `Object` | See below. | Mapping of wp-config.php constants to their desired values. |
| `"mappings"` | `Object` | `"{}"` | Mapping of WordPress directories to local directories to be mounted in the WordPress instance. |
| `"config"` | `Object` | See below. | Mapping of wp-config.php constants to their desired values. |
| `"mappings"` | `Object` | `"{}"` | Mapping of WordPress directories to local directories to be mounted in the WordPress instance. |

_Note: the port number environment variables (`WP_ENV_PORT` and `WP_ENV_TESTS_PORT`) take precedent over the .wp-env.json values._

Expand Down
89 changes: 89 additions & 0 deletions packages/env/lib/cache.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,89 @@
/**
* External dependencies
*/
const path = require( 'path' );
const fs = require( 'fs' ).promises;

/**
* Options for cache parsing.
*
* @typedef WPEnvCacheOptions
* @property {string} workDirectoryPath Path to the work directory located in ~/.wp-env.
*/

const CACHE_FILE_NAME = 'wp-env-cache.json';

/**
* This function can be used to compare a possible new cache value against an
* existing cache value. For example, we can use this to check if the configuration
* has changed in a new run of wp-env start.
*
* @param {string} key A unique identifier for the cache.
* @param {any} value The value to check against the existing cache.
* @param {WPEnvCacheOptions} options Parsing options
*
* @return {boolean} If true, the value is different from the cache which exists.
*/
async function didCacheChange( key, value, options ) {
const existingValue = await getCache( key, options );
return value !== existingValue;
}

/**
* This function persists the given cache value to the cache file. It creates the
* file if it does not exist yet, and overwrites the existing cache value for the
* given key if it already exists.
*
* @param {string} key A unique identifier for the cache.
* @param {any} value The value to persist.
* @param {WPEnvCacheOptions} options Parsing options
*/
async function setCache( key, value, options ) {
const existingCache = await getCacheFile( options );
existingCache[ key ] = value;

await fs.writeFile(
getPathToCacheFile( options.workDirectoryPath ),
JSON.stringify( existingCache )
);
}

/**
* This function retrieves the cache associated with the given key from the file.
* Returns undefined if the key does not exist or if the cache file has not been
* created yet.
*
* @param {string} key The unique identifier for the cache value.
* @param {WPEnvCacheOptions} options Parsing options
*
* @return {any?} The cache value. Undefined if it has not been set or if the cache
* file has not been created.
*/
async function getCache( key, options ) {
const cache = await getCacheFile( options );
return cache[ key ];
}

/**
* Returns the data stored in the cache file as a JS object. Instead of throwing
* an error, simply returns an empty object if the file cannot be retrieved.
*
* @param {WPEnvCacheOptions} options Parsing options
*
* @return {Object} The data from the cache file. Empty if the file does not exist.
*/
async function getCacheFile( { workDirectoryPath } ) {
const filename = getPathToCacheFile( workDirectoryPath );
try {
const rawCache = await fs.readFile( filename );
return JSON.parse( rawCache );
} catch {
return {};
}
}

function getPathToCacheFile( workDirectoryPath ) {
return path.resolve( workDirectoryPath, CACHE_FILE_NAME );
}

module.exports = { didCacheChange, setCache, getCache, getCacheFile };
11 changes: 9 additions & 2 deletions packages/env/lib/cli.js
Original file line number Diff line number Diff line change
Expand Up @@ -90,9 +90,16 @@ module.exports = function cli() {
) }} (override with WP_ENV_PORT) and tests on port {bold.underline ${ terminalLink(
'8889',
'http://localhost:8889'
) }} (override with WP_ENV_TESTS_PORT). The current working directory must be a WordPress installation, a plugin, a theme, or contain a .wp-env.json file.`
) }} (override with WP_ENV_TESTS_PORT). The current working directory must be a WordPress installation, a plugin, a theme, or contain a .wp-env.json file. After first insall, use the '--update' flag to download updates to mapped sources and to re-apply WordPress configuration options.`
),
() => {},
( args ) => {
args.option( 'update', {
type: 'boolean',
describe:
'Download source updates and apply WordPress configuration.',
default: false,
} );
},
withSpinner( env.start )
);
yargs.command(
Expand Down
112 changes: 69 additions & 43 deletions packages/env/lib/commands/start.js
Original file line number Diff line number Diff line change
Expand Up @@ -26,32 +26,24 @@ const {
configureWordPress,
setupWordPressDirectories,
} = require( '../wordpress' );
const { didCacheChange, setCache } = require( '../cache' );
const md5 = require( '../md5' );

/**
* @typedef {import('../config').Config} Config
*/
const CONFIG_CACHE_KEY = 'config_checksum';

/**
* Starts the development server.
*
* @param {Object} options
* @param {Object} options.spinner A CLI spinner which indicates progress.
* @param {boolean} options.debug True if debug mode is enabled.
* @param {boolean} options.update If true, update sources.
*/
module.exports = async function start( { spinner, debug } ) {
/**
* If the Docker image is already running and the `wp-env` files have been
* deleted, the start command will not complete successfully. Stopping
* the container before continuing allows the docker entrypoint script,
* which restores the files, to run again when we start the containers.
*
* Additionally, this serves as a way to restart the container entirely
* should the need arise.
*
* @see https://github.com/WordPress/gutenberg/pull/20253#issuecomment-587228440
*/
await stop( { spinner, debug } );

module.exports = async function start( { spinner, debug, update } ) {
spinner.text = 'Reading configuration.';
await checkForLegacyInstall( spinner );

const config = await initConfig( { spinner, debug } );
Expand All @@ -64,18 +56,42 @@ module.exports = async function start( { spinner, debug } ) {
spinner.start();
}

spinner.text = 'Downloading WordPress.';
// Check if the hash of the config has changed. If so, run configuration.
const configHash = md5( config );
const workDirectoryPath = config.workDirectoryPath;
const shouldConfigureWp =
update ||
( await didCacheChange( CONFIG_CACHE_KEY, configHash, {
workDirectoryPath,
} ) );

/**
* If the Docker image is already running and the `wp-env` files have been
* deleted, the start command will not complete successfully. Stopping
* the container before continuing allows the docker entrypoint script,
* which restores the files, to run again when we start the containers.
*
* Additionally, this serves as a way to restart the container entirely
* should the need arise.
*
* @see https://github.com/WordPress/gutenberg/pull/20253#issuecomment-587228440
*/
if ( shouldConfigureWp ) {
await stop( { spinner, debug } );
spinner.text = 'Downloading sources.';
}

await Promise.all( [
// Preemptively start the database while we wait for sources to download.
dockerCompose.upOne( 'mysql', {
config: config.dockerComposeConfigPath,
log: config.debug,
} ),
downloadSources( config, spinner ),
shouldConfigureWp && downloadSources( config, spinner ),
] );

await setupWordPressDirectories( config );
if ( shouldConfigureWp ) {
await setupWordPressDirectories( config );
}

spinner.text = 'Starting WordPress.';

Expand All @@ -84,37 +100,47 @@ module.exports = async function start( { spinner, debug } ) {
log: config.debug,
} );

if ( config.coreSource === null ) {
// Don't chown wp-content when it exists on the user's local filesystem.
// Only run WordPress install/configuration when config has changed.
if ( shouldConfigureWp ) {
spinner.text = 'Configuring WordPress.';

if ( config.coreSource === null ) {
// Don't chown wp-content when it exists on the user's local filesystem.
await Promise.all( [
makeContentDirectoriesWritable( 'development', config ),
makeContentDirectoriesWritable( 'tests', config ),
] );
}

try {
await checkDatabaseConnection( config );
} catch ( error ) {
// Wait 30 seconds for MySQL to accept connections.
await retry( () => checkDatabaseConnection( config ), {
times: 30,
delay: 1000,
} );

// It takes 3-4 seconds for MySQL to be ready after it starts accepting connections.
await sleep( 4000 );
}

// Retry WordPress installation in case MySQL *still* wasn't ready.
await Promise.all( [
makeContentDirectoriesWritable( 'development', config ),
makeContentDirectoriesWritable( 'tests', config ),
retry( () => configureWordPress( 'development', config, spinner ), {
times: 2,
} ),
retry( () => configureWordPress( 'tests', config, spinner ), {
times: 2,
} ),
] );
}

try {
await checkDatabaseConnection( config );
} catch ( error ) {
// Wait 30 seconds for MySQL to accept connections.
await retry( () => checkDatabaseConnection( config ), {
times: 30,
delay: 1000,
// Set the cache key once everything has been configured.
await setCache( CONFIG_CACHE_KEY, configHash, {
workDirectoryPath,
} );

// It takes 3-4 seconds for MySQL to be ready after it starts accepting connections.
await sleep( 4000 );
}

// Retry WordPress installation in case MySQL *still* wasn't ready.
await Promise.all( [
retry( () => configureWordPress( 'development', config, spinner ), {
times: 2,
} ),
retry( () => configureWordPress( 'tests', config, spinner ), {
times: 2,
} ),
] );

spinner.text = 'WordPress started.';
};

Expand Down
12 changes: 1 addition & 11 deletions packages/env/lib/config/config.js
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,6 @@
const fs = require( 'fs' ).promises;
const path = require( 'path' );
const os = require( 'os' );
const crypto = require( 'crypto' );

/**
* Internal dependencies
Expand All @@ -14,6 +13,7 @@ const detectDirectoryType = require( './detect-directory-type' );
const { validateConfig, ValidationError } = require( './validate-config' );
const readRawConfigFile = require( './read-raw-config-file' );
const parseConfig = require( './parse-config' );
const md5 = require( '../md5' );

/**
* wp-env configuration.
Expand Down Expand Up @@ -332,13 +332,3 @@ async function getHomeDirectory() {
: '.wp-env'
);
}

/**
* Hashes the given string using the MD5 algorithm.
*
* @param {string} data The string to hash.
* @return {string} An MD5 hash string.
*/
function md5( data ) {
return crypto.createHash( 'md5' ).update( data ).digest( 'hex' );
}
17 changes: 17 additions & 0 deletions packages/env/lib/md5.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
/**
* External dependencies
*/
const crypto = require( 'crypto' );

/**
* Hashes the given string using the MD5 algorithm.
*
* @param {any} data The data to hash. If not a string, converted with JSON.stringify.
* @return {string} An MD5 hash string.
*/
module.exports = function md5( data ) {
const convertedData =
typeof data === 'string' ? data : JSON.stringify( data );

return crypto.createHash( 'md5' ).update( convertedData ).digest( 'hex' );
};
Loading

0 comments on commit 87be0aa

Please sign in to comment.