Enable AWS S3 image storage : Issue #208

[suratdas]

This is based on the issue #208. Integrated AWS S3 bucket for image storage. To use this feature, change the backend .env file to include below details

USE_AWS_S3_BUCKET=true
AWS_ACCESS_KEY_ID=<your_key>
AWS_SECRET_ACCESS_KEY=<your_secret_key>
AWS_REGION=<your_region>
AWS_S3_BUCKET_NAME=<your_bucket>
MAX_TEMP_STORAGE_FOR_S3_DOWNLOAD=1024

Also, add the line in the front end (another pull request will be added to the front end to support it once this pull request is merged.) .env file
USE_AWS_S3_BUCKET=true
Add the line to env.config.ts file also to support reading of this new env entry.
Change the frontend static.service.ts getImage function as below

function getImage(name: string): string {
  if (name) {
    if (`${USE_AWS_S3_BUCKET}`) {
      console.log('AWS found.');
      return `${API_URL}/images/${name}`;
    } else {
      console.log('local file found.');
      return `${API_URL}/${name}`;
    }
  }
  return noImage;
}

The image will be stored like below.

A few important notes on the implementation:

1. We can decide whether to run as a local file or AWS S3 depending on the .env file configuration. Current functionality will not get impacted in any way if choose to use only local file.
2. If we switch between running from local vs AWS or vice-versa, there is a chance of images in imageUploads folder. Keep a backup of the folder in case you want to switch between two methods.
3. There is a small amount of data needed to temporarily download images to the local impageUploads folder. This memory size can be configured via .env file. This is needed to ensure that major code changes in backend and fronend are not required immediately in how the static contents are served.

[pashidlos]

let's move clean up logic to TasksService
mb make sense to extract this to another PR? guess this could be great project setting

[suratdas]

Yeah, let's do it via another PR.

[pashidlos]

not sure why we need controller for this purpose
could frontend get the image by direct url?
think I don't understand why we need to save file locally and than delete it

[pashidlos]

something like this
https://stackoverflow.com/questions/73302353/using-getobject-from-aws-to-display-an-image-in-the-browser

[pashidlos]

sorry for merge conflicts
I did prepare code for another static provider
could you move this implementation here?
existing code ideally would not be changed and we could write new e2e tests
https://github.com/Visual-Regression-Tracker/backend/blob/master/src/static/aws/s3.service.ts

[suratdas]

Thanks for incorporating changes to support multiple image storage. I have pushed code which does the following

1. Static controller is added so that the frontend will be storage (hdd, s3) agnostic. This will need just one line change in the frontend. We will not have to expose S3 keys and libraries to the frontend. This will increase security as well as prevent bundle size increase because of S3 storage.
2. The condition to check hdd storage in main.ts is removed as this is controlled/redirected by the new controller.
3. odiff.service.ts - The throw error part is removed because it does not boot up the application.

[pashidlos]

let's move this logic to service layer like getImageUrl for both hdd and s3

[suratdas]

Thanks. Done.

[pashidlos]

LGTM!