Performance Optimization for AEM Assets
High performance requirement for digital asset processing
Working closely with partner Adobe, pro!vision - a subsidiary of diva-e - has developed various product enhancements for AEM end customers over the years. In 2016 for example, diva-e achieved a drastic increase in capacity for processing digital assets in AEM 6.3 and 6.2 on behalf of Adobe Custom Engineering.
With the so-called Performance Feature Pack (PFP), AEM and the content repository were made fit for extremely large data volumes. One customer requirement was to store 600 TB of asset data (InDesign and PDF) on globally distributed instances and to perform extensive post-processing. Increasing DAM asset capacity was a major technical challenge. In solving the task, Adobe was able to rely on the technical know-how of diva-e's experts to meet the requirements for performance optimization, scalability and reliability. As the project opened up new customer segments, it was of great strategic importance to Adobe.
Objective
Significant capacity increase
The task for the diva-e experts was to meet the Adobe customer's target metrics. Asset processing, upload speed and other areas should not be affected, even with large data volumes. The first step was to set up a scalable test environment. The second step should be to create the ability to successfully upload and post-process the required data volumes. Any performance problems that might arise in the process were to be resolved to such an extent that in future it would be possible to ensure high-performance work with the required asset types and quantities.
Implementation
Performance, scalability, availability
Various implementation and testing activities were needed to meet the customer requirement.
Upload and process 150TB of assets into AEM assets with workflows and automated InDesign file extraction at top speed.
Fully automated setup of AEM and Oak clusters in the AWS cloud using the open source automation tool Ansible and storage of binary data in S3 Cloud Store
Set up multiple clusters in globally distributed AWS regions with cross-region replication
Generation of test data batches
Automating and monitoring asset uploads with a technology stack consisting of ElasticSearch, LogStash, and Kibana, and measuring KPIs
Determine optimized repository statistics using BigData approaches (Apache Spark)
Determine optimal configuration for AWS, AEM, and repository.
Bottlenecks in the AEM product and platform (Jackrabbit Oak, Apache Sling) were analyzed and resolved
Result
Strong performance of AEM Assets
AEM version 6.3 has been significantly improved in terms of scalability when uploading and processing assets. In the form of a custom feature pack, these improvements were also made available for the previous version AEM 6.2 (backporting). Many bottlenecks in Apache Jackrabbit Oak and Sling as well as in the AEM product have been fixed.
To meet the customer requirements, diva-e built a suitable test scenario. The AEM environments consisting of AEM clusters and MongoDB clusters were hosted in the AWS Cloud. The cluster setup was highly automated in order to be able to set up new test instances very quickly. Extensive InDesign test files were generated for testing purposes. Here, the processing of the InDesign files went beyond simply uploading large amounts of data. When the InDesign files were added to the data repository, they were also analyzed and the assets they contained were stored in a structured manner. diva-e succeeded in achieving an acceptable speed for the user, even when a large number of users were working on the system in parallel worldwide.