Allow including generic asset files

[EmericW]

I'm currently creating a small registry to share AWS CDK stacks.
Often these stacks might depend on some non JS files/assets. In my case it is a lambda layer ZIP file.

If I understand the docs correctly, I don't think this is possible right now.

I'm guessing this would be somewhat related to the sidecar files referenced in this issue.

It would be sufficient for my use case that these assets are copied without any diffing for changes. Just a hard copy.

The hard part would be that these files are not referenced by any imports. Only by a string parameter containing the path in my case.

    const mysqldumpLayer = new lambda.LayerVersion(
      this,
      `${ENVIRONMENT}-mysqldump-layer`,
      {
        code: lambda.Code.fromAsset(
          path.join(__dirname, './mysqldump.layer.zip'),
        ),
        compatibleRuntimes: [lambda.Runtime.NODEJS_22_X],
      },
    );

I would be open to create a PR given some pointers.

[ieedan]

I think configFiles should cover this...

These are files that are included when you initialize the registry. They can be optional or required.

Take a look at the docs here and lmk if that works for you use case

https://jsrepo.dev/docs/registry/jsrepo-build-config-json

[EmericW]

I thought that the config files are copied over in your project when initialising the JSRepo?

The files that I would like to copy over are block specific.
My current example where "stacks" are my blocks:

src/
├─ stacks/
│  ├─ rds-backup/
│  │  ├─ backup.lambda.ts
│  │  ├─ index.stack.ts
│  │  ├─ mysqldump.layer.zip
│  ├─ some-other-stack/

I guess it would be like a block specific config file.

[EmericW]

Just for completeness, I would place the .jsrepokeep file inside the block directory. It would also be completely optional only used to add "extra" files.
In my case this would be.

src/
├─ stacks/
│  ├─ rds-backup/
│  │  ├─ backup.lambda.ts
│  │  ├─ index.stack.ts
│  │  ├─ mysqldump.layer.zip
│  │  ├─ .jsrepokeep             <----- The content of this file would be "mysqldump.layer.zip"
│  ├─ some-other-stack/

My reasoning behind this is to colocate the "keep config" with the block. Otherwise it would be separated from the block, which would make it harder to reason about.

If you use the feature frequently it could also result in the root config file containing a lot off seemingly unrelated files.

---

That being said, I'll follow whatever vision you have for the tool. Having the names of the files to include in the root config would also fix the problem for me. I just think it is a bit less maintainable in the long run.

I would be happy to implement either short term solution.

[ieedan]

I would be happy to implement either short term solution.

Go for it but lets not add any config files for now! I will probably start working on a more semi auto build that would likely replace that long term.

[EmericW]

(I started writing this before you edited you comment, so I'll post it anyway for completeness. 😄)

This is my reasoning on why it would be less maintainable.
If we add this option to the root config. It would probably look something like this in my case.

{
  "$schema": "https://unpkg.com/jsrepo@1.41.2/schemas/registry-config.json",
  "dirs": [
    "./src"
  ],
  "preview": true,
  "allowSubdirectories": true,
  "includeFiles": [
    "./src/stacks/rds-backup/mysqldump.layer.zip"
  ]
}

Which is fine, but for a large registry it might look something like this.

{
  "$schema": "https://unpkg.com/jsrepo@1.41.2/schemas/registry-config.json",
  "dirs": [
    "./src"
  ],
  "preview": true,
  "allowSubdirectories": true,
  "includeFiles": [
    "./src/stacks/rds-backup/mysqldump.layer.zip",
    "./src/stacks/a/a.layer.zip",
    "./src/stacks/b/b.layer.zip",
    "./src/stacks/c/c.layer.zip",
    "./src/stacks/d/d.layer.zip",
    ...perhaps many more
  ]
}

• When modifying a block, it would not be obvious that you also have to edit the config file in the root config.
• When deleting a block, it would not be obvious that you also have to remove the reference in the root config
• Since these are not hard "import" references, it would be harder to figure out which of these references are still relevant.

[EmericW]

Anyway, I'll probably start implementing your preferred method tonight.

[ieedan]

(I started writing this before you edited you comment, so I'll post it anyway for completeness. 😄)

If we add this option to the root config. It would probably look something like this in my case.

{
  "$schema": "https://unpkg.com/jsrepo@1.41.2/schemas/registry-config.json",
  "dirs": [
    "./src"
  ],
  "preview": true,
  "allowSubdirectories": true,
  "includeFiles": [
    "./src/stacks/rds-backup/mysqldump.layer.zip"
  ]
}

Which is fine, but for a large registry it might look something like this.

{
  "$schema": "https://unpkg.com/jsrepo@1.41.2/schemas/registry-config.json",
  "dirs": [
    "./src"
  ],
  "preview": true,
  "allowSubdirectories": true,
  "includeFiles": [
    "./src/stacks/rds-backup/mysqldump.layer.zip",
    "./src/stacks/a/a.layer.zip",
    "./src/stacks/b/b.layer.zip",
    "./src/stacks/c/c.layer.zip",
    "./src/stacks/d/d.layer.zip",
    ...perhaps many more
  ]
}

• When modifying a block, it would not be obvious that you also have to edit the config file in the root config.
• When deleting a block, it would not be obvious that you also have to remove the reference in the root config
• Since these are not hard "import" references, it would be harder to figure out which of these references are still relevant.

Since these are not hard "import" references, it would be harder to figure out which of these references are still relevant.

We can always just check for existence and warn when they no longer exist.

It can be a bit tedious to add all those files so I think allowing glob patterns could be a nice feature:

{
  "$schema": "https://unpkg.com/jsrepo@1.41.2/schemas/registry-config.json",
  "dirs": [
    "./src"
  ],
  "preview": true,
  "allowSubdirectories": true,
  "includeFiles": [
    "src/**/*.layer.zip"
  ]
}

[EmericW]

This has been implemented in #623

[EmericW]

I'm afraid I have encountered a new issue.
The feature is working great for text based files, but not for binaries, like the zip file that was the whole reason for creating this feature.

I've done some digging and this line seems to be causing the issue:

jsrepo/src/utils/registry-providers/index.ts

Line 65 in 4de29fc

return Ok(await response.text());

Every file response is parsed as a text file. This results in the zip file being corrupted when added to you project via the add command.

As a quick POC I added the following code locally:

if (url.toString().includes('.layer.zip')) {
	// @ts-ignore
	return Ok(Buffer.from(await response.arrayBuffer()));
}

return Ok(await response.text());

This works nicely at runtime but causes a lot of cascading typescript problems since most of the codebase makes the assumption that all files are text based.

I don't think this will be a quick fix, this is the only solution I can think of right now.
Treat all files as binary throughout the codebase and only convert them to text for the supported languages.
But It will be a big job and has some risks.