SETLIST ORGANIZER

_{Built by Maksim Meliashchuk}

Table of contents

• Project Information
• Running example
• Applications Demo
• Technology Stack
• Requirements
• Installation Information
• Microservices
• Rest app configure
• Documenting a REST API
• Web app configure
• Security
• Run with docker-compose
• Run mutation testing
• Run load testing
• Local tests with Postman
• OpenAPI generated server
• Excel Import and Export
• Generate test data
• Create and restore database archive

Project Information

🎙️ 'Setlist Organizer' is a web application for organizing repertoires of musical bands. 🎙️

Mind Map

• Software requirements specification
• Available REST endpoints
• Presentation

⚠️ _{Note: the Mind map, Software requirements specification, Available REST endpoints and Presentation describe the basic version of the project.The project is constantly being improved and developed. The features of the current version of the project may differ from the information in this section. See the following sections in this README for up-to-date project information.} ⚠️

Running example

The gif below shows a real-time updating of bands page when the repertoire changes.

Applications Demo

Applications deployed on Heroku

The web application will be accessible at:

https://setlist-organizer-web.herokuapp.com/

The rest application will be accessible at:

https://setlist-organizer-rest.herokuapp.com/

API documentation with Swagger UI:

https://setlist-organizer-rest.herokuapp.com/swagger-ui/index.html

Technology Stack

• Programming Language: Java
• Core Framework: Spring boot
• Tools for distributed systems:
  • Spring Cloud
    • Spring Cloud Config
    • Spring Cloud
    • Spring Cloud Stream
• Data Access:
  • Spring JDBC
  • Spring Data JPA
• Security: Spring Security
• Message broker: Apache Kafka
• Identity Management: Keycloak
• Validation Framework: Hibernate Validator
• Annotation processor:
  • Lombok
  • MapStruct
• Build System: Maven
• Control System: Git
• License: Apache license, version 2.0
• Code Style: Codacy
• Automated Testing:
  • JUnit5
  • Mockito
• Mutation Testing: PIT
• Load Testing: Gatling
• Log:
  • Log4j 2
  • Logback
  • SLF4J
• Monitoring:
  • Micrometer
  • Prometheus
  • Grafana
• Database:
  • H2
  • PostgreSQL
• Database migration tool: Flyway
• JSON library:
  • Jackson
  • GSON
• Generate test data: Java Faker
• Java API for Microsoft Excel: Apache POI
• API documentation generation:
  • Springdoc-openapi
  • Swagger UI
• Code generation:
  • OpenAPI Generator
  • Swagger Codegen
• Template Engine: Thymeleaf
• CSS Framework: Bootstrap
• App containerization: Docker

Requirements

The following software is required for the complete workflow (from git clone to the running Docker Container). The specified versions are the tested ones.

• JDK 11+
• Git 2.25.1+
• Apache Maven 3.6.3+
• Docker 20.10.12+
• Docker-compose 1.29.2+
• Scala-sdk 2.13.6+ (for load testing)

Installation Information

  $ git clone https://github.com/Brest-Java-Course-2021-2/Maksim-Meliashchuk-Setlist-Organizer.git
  $ cd Maksim-Meliashchuk-Setlist-Organizer
  $ mvn clean install

Microservices

Config server

• Stores configs of all microservices. A Spring Cloud Config server can manage a services configuration information using a file system/classpath or GitHub-based repository.

Service discovery server

• Performs Service registry function. A Eureka server running as a Spring-Cloud based service. This service will allow multiple service instances to register with it. Clients that need to call a service will use Eureka to lookup the physical location of the target service.

REST service

• A Spring Boot app (REST API).

WEB service

• A Spring Boot app (UI) that uses REST API. Uses Thymeleaf to render HTML pages and Bootstrap CSS framework.

Notification service

• A Spring Boot app that reads messages from Kafka and sends notifications to a WEB application using a WebSocket.

Rest app configure

Profile	Description
dev	Run application with embedded H2 database in memory
postgres	Run application with PostgreSQL database
test	Run application with embedded H2 database in memory, Flyway migration disabled
jdbc	Interact with a database by using Spring JDBC
jpa	Interact with a database by using Spring Data JPA
nokafka	Creating an empty ProducerService to run the application without using Apache Kafka

Documenting a REST API

Using OpenAPI 3.0

The OpenAPI descriptions in JSON format will be available at the path: http://localhost:8088/v3/api-docs

The OpenAPI spec will be available at the path: http://localhost:8088/v3/api-docs.yaml

API documentation with Swagger UI: http://localhost:8088/swagger-ui/index.html

Click the Authorize button and log in as an example user (available usernames admin1, user1, password 123) of the setlist_organizer_client client to test secured endpoints, the client is public so you don't need to fill the client_secret field.

Web app configure

The web application has three web client implementations for making HTTP calls to REST application services - RestTemplate, a WebClient alternative and automatically generated ApiClient.

Using RestTemplate (is deprecated since Spring 5):

app.httpClient = RestTemplate

Using WebClient (exists since Spring 5):

app.httpClient = WebClient

Using ApiClient (OkHttpClient and GSON based):

app.httpClient = ApiClient

ApiClient is a preferable choice. This client API is automatically generated by the Swagger Codegen by using the swagger-codegen-maven-plugin.

Refer to the module service-swagger-client for more information.

⚠️ _{Note: this Web application has endpoints for using create and import Excel, XML, ZIP files, Security support with the 'ApiClient' property only.} ⚠️

The web application has the following properties to check if the Keycloak container is ready:

• pre-connection-check boolean
• number-of-connection-attempts integer
• timeout-between-connection-attempts integer seconds

These properties are useful to wait until the Keycloak container becomes available.

Security

This project uses the OpenID Connect support in Spring Security 5 and Keycloak as the OpenID Connect Identity Provider without using the Keycloak adapter.
Only authenticated users can call secured endpoints available through Swagger UI and the web-app UI.

You can run the Keycloak container with the following commands in the root directory of the project:

$ sudo docker-compose -f keycloak/keycloak.yml up

Keycloak is assumed to run on port 8484 on localhost. The Keycloak service can be started with a default realm by importing it from the keycloak/realm-export.json file which lists default users. For example users of this project:

• available usernames: admin1, user1
• password: 123

admin1 has the admin realm role that is required to call the POST and DELETE.

Run with docker-compose

In the root directory of the project, run all microservices in one go:

$ sudo docker-compose up --build

The WEB application will be accessible at http://localhost:8080

The REST application will be accessible at http://localhost:8088

The Notification application will be accessible at http://localhost:8099

The Config Serverwill be accessible at http://localhost:8071

The Service Discovery Serverwill be accessible at http://localhost:8761

The PostgreSQL database can be accessed in docker at: http://localhost:5431

You can run the PostgreSQL container with the following commands in the root directory of the project:

$ sudo docker-compose -f prod-db/postgresql.yml up

PostgreSQL require(can be customized in application-postgres.yaml file in prod-db module):

• driver - org.postgresql.Driver
• url - jdbc:postgresql://localhost:5433/setlistOrganizer
• user - postgres
• password - password

Access the Prometheus webUI on http://localhost:9090

Micrometer by default shows jvm metrics at http://localhost:8088/actuator/prometheus

Access the Grafana webUI (log in as admin:admin) with configured Prometheus datasource and added jvm-micrometer dashboard on http://localhost:3000

• Admin panel: http://localhost:8484/auth (log in as the Keycloak admin admin:admin_password)
• As an admin you can see a list of users associated with the setlist_organizer_realm realm by clicking

The Kafka is running at default http://localhost:9090, Zookeeper is running at http://localhost:2181

To stop the containers:

$ sudo docker-compose down

Run mutation testing

PIT runs unit tests against automatically modified versions of the application code, the most effective way to use mutation testing is to run it frequently against only the code that has been changed. TargetClasses and TargetTests to limit the tests available to be run added to parent project POM:

<configuration>
  <targetClasses>
      <param>com.epam.brest.service.impl.jpa.*</param>
  </targetClasses>
  <targetTests>
      <param>com.epam.brest.service.impl.jpa.*</param>
  </targetTests>
</configuration>

Run directly from the commandline:

$ mvn test-compile org.pitest:pitest-maven:mutationCoverage

PIT Test Coverage Report will be accessible: /service/target/pit-reports/index.html)

Run load testing

Testing starts with 10 users per second making two requests to the repertoire, holding at that concurrency for five seconds. Then RestSimulationLocal test scenario increases the number of users per second by 5, 5 times, holding for 5 seconds each time. Every request must return an HTTP 200 status code to pass the test. Run the test in the root directory of the project:

$ sh ./gatling-test/gatling-test.sh

Make sure the rest-app at http://localhost:8088 is running. Gatling reports generated in: /gatling-test/target/gatling/

Local tests with Postman

To test the REST API, you can use the Postman collection to access the API endpoints:

postman_collection.json

See the Postman online documentation.

OpenAPI generated server

⚠️ _{Note: This application is deprecated. New features not included.} ⚠️

Automatically generated by the OpenAPI Generator project by using the OpenAPI-Spec and the openapi-generator-maven-plugin. API Controllers generated automatically and implementation moved to delegates.

An OpenAPI document that conforms to the OpenAPI Specification openapi.yaml has been generated automatically by using springdoc-openapi-maven-plugin in the module rest-app.

Refer to the module rest-app-openapi for more information.

Excel Import and Export

These REST applications have several endpoints (/export/excel and /import/excel) to create and import Excel files using Apache POI.

Please, after running the REST application, refer to the API documentation with Swagger UI for more information.

Following are samples of Excel files that can be imported:

Band.xlsx Track.xlsx

Generate test data

These REST applications have several endpoints /fill to generate fake data for showcase with uses Java Faker. It is possible specify language for generated data (EN, DE, FR), default EN. It is possible specify count fakes data, use parameter to multiply default size (default value 1).

Please, after running the REST application, refer to the API documentation with Swagger UI for more information.

Create and restore database archive

The database archive can be created and exported in XML format and saved as a ZIP archive. These REST applications have a /downloadZipFile endpoint with the ability to create a database archive and a /uploadZipFile endpoint with the ability to restore the database from the archive. The Simple API for XML (SAX) is used to parse XML documents.

The following is an example of a ZIP file that is the result of an export that can be imported:

database.zip