New Feature: AEM Remote Assets

[HitmanInWis]

This is a PR for the AEM Remote Assets feature, to be presented at Adobe SUMMIT AEM Rockstar on Wednesday Mar 28.

The purpose of AEM Remote Assets is to allow an AEM server to be fully functional with assets from a remote AEM instance without sync'ing the entire set of DAM asset file binaries (the bulk of the disk space) up front. Asset file binaries are sync'd into the server the first time they are requested by a user. To the end user it appears all assets are present, but in actuality some assets may be getting fetched "just in time" from the source AEM server as the user accesses them. Once an asset file has been pulled in from the source server the asset is now a standard AEM asset. The bulk of rarely-used assets, however, remain un-synced.

I would consider this feature "beta" level. It's pretty solid in our local testing, but I encourage anyone looking to use this feature to do so with caution in non-production environments until the feature is further battle-tested. Though this feature ultimately supports use of a remote AEM Assets instance from an AEM Sites instance, which is a very important business use case, it is also extremely useful for developers working on local AEM instances of production websites, where pulling the entire DAM from production to the local server is simply not an option due to size.

I understand that unit testing will need to be added to this PR before it is accepted, but wanted to get feedback on the code before diving into that effort.

Other "nice to have" features that I can see for the future of AEM Remote Assets include:

• Function to purge remote asset binaries (to manually maintain a small footprint)
• Expire image binaries after awhile (to automatically maintain a small footprint)
• Protection mechanisms against fetching assets > specified size
• Protection mechanisms against fetching too many assets at once
• Option to delete assets that are no longer in the source system when sync'ing nodes
• When syncing nodes, keep any renditions that are smaller than the temporary binaries.
• Sync content fragments and experiences fully (so they can be indexed and since they are just content rather than large binaries)
• Determine a way to sync Binary properties (currently ignored b/c JSON fetch simply gives the binary property size rather than anything useable)

[HitmanInWis]

No idea if this logout() is necessary - my gut says it isnt but I dont know enough of the internals to be sure. The reason I'm concerned in the first place is b/c the getResourceResolver() function uses the session to setUserData() on the workspace observation manager so as to prevent triggering workflows.

[HitmanInWis]

If theres a simple way to do this w/o Node I'm happy to update this line.

[davidjgonzalez]

I dont think there is... best i know is RR.create(..) which i believe only does single nodes at a time..

[davidjgonzalez]

@HitmanInWis @kaushalmall - so i was playing around this past 2 nights... and it seems that the sync times are very high (i.e. very noticeable as a user of the system).

For the scheduled sync this is transparent since its an async job (though watching the logs it takes 2 mins or so to pull my 50 assets down from my remote AEM instance -- which is remote, it's not running on my local - im using a hosted Asset Share Commons author and syncing the ASC folder).

The more concerning part is the renditions (binary) sync .. when i search and get a pull page of results, it tends to hang for quite a while (pulling in 10 asset x 5 renditions = 50 requests, some large)...

I also noticed that certain requests, especially when accessing AEM Assets folders, tend to hit many assets in a folder.. so browsing, to a folder that contains other folders which in turn contain assets, will invoke the thumbnail folder servlet which seems to walk many assets in that folder (even though i never clicked in).

I also noticed that occasionally the following...

1. Request A references DAM Asset X (perhaps a request for the assets cards servlet in folder view)
2. Request A blocks and downloads Asset X (along w a many other assets, making this a long request)
3. Request B is initiated for an asset itself.
4. Request B references DAM Asset X, but DAM Asset X is already in the list of being sync'd .. so it doesnt sync.
5. Asset details page looks "broken" (ie still shows the placeholder).
6. Request A completes after the renditions are downloaded.
7. Refresh the details page and everything looks great.

Am I doing something wrong here? I watched your video and you didn't have delays but it sounded like you were running both Src and Dest locally removing the network time.

[HitmanInWis]

Hi David. I think all of your concerns are imperfections in the system as it stands today. However, I think we should look at it from a right perspective of "is it useful for some use cases, even if not all use cases yet?".

I think its valid to say that there will be a delay when the remote AEM server is truly remote. However, that's going to largely depend on connection quality between AEM instances. Two servers on the same network, or in the AWS cloud, may perform significantly better than what you're seeing (maybe not as fast as in my demo, but much closer).

Regardless of the delay, keep in mind that this is a one-time "tax" you are paying for auto-sync of just the assets you want. For example, a developer working on a client website locally would get "taxed" the first time they go to a certain page on the site, but after that page loads the first time it will load w/o any delay in the future (the assets are now in the local DAM). The same can be said about a true Assets -> Sites scenario - the first time a search is performed it will delay in retrieving binaries, but the second time the assets will already be there.

Down another thought path, the core tenet of this feature is pulling only the assets that are actually in use - hence why doing the "just in time" sync. I think for assets used in a site we could add a future enhancement to pre-sync binaries, thus eliminating the delay for that particular use case. For use cases like asset search and DAM traversal, however, we'd still be doing "just in time".

Lastly, good catch on the race condition where the 2nd browser sees temporary binaries that the first browser is currently sync'ing. We could definitely have the 2nd request sleep while the first thread is pulling the asset - we already have logic in there that's preventing that 2nd request from fetching the same binary that the first request is pulling, so the change would be relatively trivial (sleep 100ms in a loop until the asset is sync'd).

[HitmanInWis]

Also regarding "large" assets I think a potential future enhancement is having a config that sync's assets greater than the configured size in an asynchronous rather than synchronous fashion.

I think this feature has huge potential for enhancement, but I also dont want to hold off the initial push of this feature until we've covered every single scenario. As it stands right now I think this feature is already sufficient for sync'ing assets to local and non-production server instances. A TON of interest has been shown in this feature, and I'm hoping that getting in an initial cut may set the stage for other companies to contribute the one or two things they need to make the feature work for them.

[HitmanInWis]

@davidjgonzalez another thing I thought of - please make sure all of your asset workflow launchers have event-user-data:changedByWorkflowProcess in the "Exclude List" so that the sync does not trigger workflows. Unfortunately I couldn't find an easier way to avoid triggering workflows, so this will be part of configuration instructions.

[HitmanInWis]

Fixed the race condition you found @davidjgonzalez, where Thread 2 does not wait for Thread 1's binary sync to complete. With the new impl, both Thread 1 and Thread 2 will correctly see the real asset when the page finishes loading.

[HitmanInWis]

@davidjgonzalez is there anything remaining on this PR other than writing unit tests? I do plan on getting around to doing tests (especially since a couple companies have reached out to me directly about this feature) but just want to make sure that otherwise things are good.

[davidjgonzalez]

@HitmanInWis TBH I'm still not sold on this; when I use it it makes AEM feel broken/not working properly. That being said, I cant think of a better way to do this... I spent a bit of time trying out different methods to make the content syncing more robust [1] using packages/batching of request/async requests ... and a combo of them all... but nothing felt good.

I'm also leery on the JSON -> Node transformation since it makes assumptions wrt to the content; which could be ok, but IDK. I'd rather it be an exact copy (which is where packages/XML help out).

@badvision / @kaushalmall - have you tried this out and reviewed the approach? Would be good to get your (experience/review-based ;)) thoughts.

[1] https://github.com/davidjgonzalez/acs-aem-commons/tree/feature/remote-assets-packages-poc

[HitmanInWis]

Ok. I do want to get confirmation on whether or not this is a "go" before I invest time into a bunch of unit tests, so best we work through these questions now rather than later.

I can say the I've had two separate companies reach out to me specifically about the feature, and have heard from someone at Adobe they were getting all sorts of requests for it, so there's definitely a high level of interest in the feature.

Even though the feature is not yet currently perfect, I believe it provides a TON of developer value if nothing else, as it can be used to sync assets from a non-prod server to their local instance to have a full working local site. Getting the feature in at its current state will then allow for future iterations to make it even better.

I'm less concerned about the JSON node transformation issue because a) it seems to be working well even if not 100% perfect and b) the server that is using remote assets is not the system of record, so a 99% perfect copy of meta data is probably going to work just fine nearly 100% of the time.

[kaushalmall]

I had some time to play around as well. Not as much as @davidjgonzalez though. My setup was running two AEM instances locally, so, I didn't really see any network latency issues.

My opinion is to merge this, but document all the drawbacks and implementation details that we have discussed here. That ways anyone who decides to use this feature knows what they are getting and if they see issues, hopefully there will be more PRs in the future.

[badvision]

@HitmanInWis The config should clarify if URL scheme is required or not.

[badvision]

Also, this doesn't synchronize even the smaller 140x140 thumbnail images?

[badvision]

Also, remote assets should not be editable because that would lead to a lot of confusion.

[HitmanInWis]

Just a FYI, we are looking to get someone internally assigned to banging out these tests in the (hopefully) near future.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[HitmanInWis]

It's fine if this rolls off as "stale" - we can resurrect it when we finally have the time to write some tests. At this point, I'm unsure when that will be.