Skip to content
This repository was archived by the owner on Jun 4, 2023. It is now read-only.

Skip writing unchanged files #5

Merged
davbree merged 1 commit into from
May 11, 2021
Merged

Skip writing unchanged files #5

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 13 additions & 5 deletions packages/core/src/index.js
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
#!/usr/bin/env node

const fs = require('fs');
const path = require('path');
const _ = require('lodash');
const fse = require('fs-extra');
const yaml = require('js-yaml');
const toml = require('@iarna/toml');
Expand Down Expand Up @@ -77,11 +77,19 @@ function writeFiles(encodedFiles) {
for (let i = 0; i < encodedFiles.length; i++) {
const fullPath = path.join(process.cwd(), encodedFiles[i].filePath);
fse.ensureDirSync(path.dirname(fullPath));
if (fs.existsSync(fullPath) && ['yml', 'yaml', 'toml', 'json'].includes(path.extname(fullPath).substring(1))) {
encodedFiles[i].data = mergeFile(fullPath, encodedFiles[i].data);
let fileContentChanged = true;
if (fs.existsSync(fullPath)) {
if (['yml', 'yaml', 'toml', 'json'].includes(path.extname(fullPath).substring(1))) {
encodedFiles[i].data = mergeFile(fullPath, encodedFiles[i].data);
}
fileContentChanged = !_.isEqual(encodedFiles[i].data, fs.readFileSync(fullPath, 'utf8'));
}
Comment on lines +81 to +86
Copy link
Member

@smnh smnh May 10, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚨 @rodikh @davbree an important notes about merging data here...

Originally the reason for this kind of merging in stackbit-pull, and also in container here, was done because we didn't model the full structure of Jekyll/Hugo config files.

For example, while Jekyll config could be something like this:

baseurl: "/"
title: "Site Title"
include: [".htaccess"]
ollections:
  posts:
    output: true
kramdown:
  auto_ids: true

We only modeled the baseurl and title. Meaning that data stored in Contentful had only the baseurl and title fields. The rest of the fields were left in the config.yaml.

That means that when stackbit-pull fetches the content, it should not override everything but assign the additional values stored in CMS.

I admit, this was a huge mistake :)

  • First of all, this should only happen to config models because only they are modeled partially, other files should be always modeled fully
  • Second, this presents a huge bug - if user deletes a field in Contentful, and we fetch the data and merge it with older file that has the field, then we didn't actually delete the field.

One of the possible solution was this task:
https://www.notion.so/stackbit/Themes-Move-all-config-yaml-params-to-a-separate-file-91517dbceafb40a4a21c2fc01de4319d#764577d127764bfd925d9d157449ad98

Copy link
Member Author

@davbree davbree May 10, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds like we should fix it! :)
Created this task https://www.notion.so/stackbit/Fix-stackbit-pull-merging-behavior-4c99c343ed704a3097ca4784bdedf9f5

Copy link
Member

@smnh smnh May 10, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, this should be fixed, but only after we update our themes according to the previous task, and also we need to make sure it is backward compatible with existing projects.

Basically, the merge should be made only if the model is of type config - meaning the object in CMS has stackbit_model_type=config.

Copy link
Contributor

@rodikh rodikh May 10, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@smnh I actually mentioned this exact issue to David on slack just before you commented here.
In any case, let's release this totally unrelated PR and deal with this new issue separately.

if (fileContentChanged) {
console.log('creating file', fullPath);
fs.writeFileSync(fullPath, encodedFiles[i].data);
} else {
console.log('content unchanged, skipping write', fullPath);
}
console.log('creating file', fullPath);
fs.writeFileSync(fullPath, encodedFiles[i].data);
}
}

Expand Down