S3 File adapter - accessing previous, non-s3 files

[Samigos]

We have a mobile app that runs on Parse Server 2.8.4, hosted at DigitalOcean. We were using the default file adapter (GridStoreAdapter) and now we want to use Spaces by DigitalOcean. This is our updated index file:

var express = require('express');
var ParseServer = require('parse-server').ParseServer;
var ParseDashboard = require('parse-dashboard');
var path = require('path');
var S3Adapter = require('parse-server').S3Adapter;
var AWS = require("aws-sdk");

var databaseUri = process.env.DATABASE_URI || process.env.MONGODB_URI;

//Set Digital Ocean Spaces EndPoint
const spacesEndpoint = new AWS.Endpoint('fra1.digitaloceanspaces.com');
//Define S3 options
var s3Options = {
  bucket: 'bucketName',
  baseUrl: 'https://domain.com',
  region: 'fra1',
  directAccess: true,
  globalCacheControl: "public, max-age=31536000",
  s3overrides: {
    accessKeyId: 'accessKey',
    secretAccessKey: ' secretAccessKey',
    endpoint: spacesEndpoint
  }
};

var s3Adapter = new S3Adapter(s3Options);

var api = new ParseServer({
  databaseURI: databaseUri,
  appId: process.env.APP_ID,
  allowClientClassCreation: false,
  masterKey: process.env.MASTER_KEY,
  fileKey: 'fileKey',  
  serverURL: process.env.SERVER_URL,
  publicServerURL: 'http://ip/parse',
  filesAdapter: s3Adapter
});

Everything works great! The files are stored and retrieved properly, except of the old ones! Every image that was stored with the GridStoreAdapter cannot be fetched anymore! Is there anything that can be done in order to be able to view both old and new files?

[dplewis]

A PR was submitted just for this purpose.

parse-community/parse-server-s3-adapter#79

I'll do a release right now so that you use it without having to update your server.

You could use parse-files-utils to transfer the files.

[Samigos]

Thank you very much @dplewis ! So what should I do now? I'm kinda lost!

[dplewis]

Wait for the 1.4.0 release of the adapter.

Remove var S3Adapter = require('parse-server').S3Adapter;

npm install --save-exact @parse/s3-files-adapter

Add var S3Adapter = require("@parse/s3-files-adapter");

https://github.com/parse-community/parse-server-s3-adapter#usage-with-digital-ocean-spaces

[Samigos]

And after that, it will know where each file is? MongoDB or DO Space?

[dplewis]

Parse Server doesn't currently allow multiple file adapters. Therefore, I recommended the migration tool.

[Samigos]

So wait, I'm confused! Do I have to transfer all the files from Mongo to the DO Space? Wouldn't that PR you mentioned take care of it? (sorry for bothering you, but I really have no idea of how all these work)

Wouldn't that PR you mentioned take care of it?

I mean wouldn't it separate the MongoDB files and the DO files?

[dplewis]

A fileAdapter is used to access the files. Since you changed GridStoreAdapter, you can't use it to fetch files anymore from Mongo.

I mean wouldn't it separate the MongoDB files and the DO files?

Yes they will be separated. You could use a workaround to access the mongo files.

[Samigos]

You could use a workaround to access the mongo files.

And that would be the PR you're going to release?

[dplewis]

@Samigos Sorry I reread your post and misunderstood. The PR #6274 was to support s3 overrides and endpoints but looks like you have that working.

You have to migrate your files if you want to view old and new files.

[Samigos]

Ok! Thank you very much!

[Samigos]

Hey @dplewis once more 😄I'm sorry for bothering you again, but I'm desperate! I'm stuck in the migration process and there's really not much I could find!

The repo you sent me above, parse-files-utils can it be used to transfer my files to an S3 bucket? If so, how exactly? I started following the guide, but as soon as I hit npm start config.js it failed with the error logs below:

Cannot load /root/parse-files-utils/config.js
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! parse-files-utils@0.0.1 start: `node index.js "config.js"`
npm ERR! Exit status 1
npm ERR! 
npm ERR! Failed at the parse-files-utils@0.0.1 start script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /root/.npm/_logs/2019-12-12T22_35_35_482Z-debug.log

0 info it worked if it ends with ok
1 verbose cli [ '/usr/bin/node', '/usr/bin/npm', 'start', 'config.js' ]
2 info using npm@6.4.1
3 info using node@v8.12.0
4 verbose run-script [ 'prestart', 'start', 'poststart' ]
5 info lifecycle parse-files-utils@0.0.1~prestart: parse-files-utils@0.0.1
6 info lifecycle parse-files-utils@0.0.1~start: parse-files-utils@0.0.1
7 verbose lifecycle parse-files-utils@0.0.1~start: unsafe-perm in lifecycle true
8 verbose lifecycle parse-files-utils@0.0.1~start: PATH: /usr/lib/node_modules/npm/node_modules/npm-lifecycle/node-gyp-bin:/root/parse-files-utils/node_modules/.bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
9 verbose lifecycle parse-files-utils@0.0.1~start: CWD: /root/parse-files-utils
10 silly lifecycle parse-files-utils@0.0.1~start: Args: [ '-c', 'node index.js "config.js"' ]
11 silly lifecycle parse-files-utils@0.0.1~start: Returned: code: 1  signal: null
12 info lifecycle parse-files-utils@0.0.1~start: Failed to exec start script
13 verbose stack Error: parse-files-utils@0.0.1 start: `node index.js "config.js"`
13 verbose stack Exit status 1
13 verbose stack     at EventEmitter.<anonymous> (/usr/lib/node_modules/npm/node_modules/npm-lifecycle/index.js:301:16)
13 verbose stack     at emitTwo (events.js:126:13)
13 verbose stack     at EventEmitter.emit (events.js:214:7)
13 verbose stack     at ChildProcess.<anonymous> (/usr/lib/node_modules/npm/node_modules/npm-lifecycle/lib/spawn.js:55:14)
13 verbose stack     at emitTwo (events.js:126:13)
13 verbose stack     at ChildProcess.emit (events.js:214:7)
13 verbose stack     at maybeClose (internal/child_process.js:915:16)
13 verbose stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:209:5)
14 verbose pkgid parse-files-utils@0.0.1
15 verbose cwd /root/parse-files-utils
16 verbose Linux 4.15.0-34-generic
17 verbose argv "/usr/bin/node" "/usr/bin/npm" "start" "config.js"
18 verbose node v8.12.0
19 verbose npm  v6.4.1
20 error code ELIFECYCLE
21 error errno 1
22 error parse-files-utils@0.0.1 start: `node index.js "config.js"`
22 error Exit status 1
23 error Failed at the parse-files-utils@0.0.1 start script.
23 error This is probably not a problem with npm. There is likely additional logging output above.
24 verbose exit [ 1, true ]

Any ideas?

[dplewis]

That is an old repo. I’d recommend forking it and make changes you need and run it.

The core logic is there on how to get the all files from the Schema and transfer them.