Get best of the week

Front work on the backend development of the largest portals on open source software: sharing experience

One of the areas of the Digital Design group of companies is the creation of portal solutions that provide end users with a large set of custom services within a single site. Services can be different: news, forums, document repositories, statistics, events, etc. —– this is what we call the portal. In this article I would like to tell how the team and I create portal solutions: what we use, what advantages and disadvantages of our implementation are possible.




In general, the idea of ​​a portal with your own hands did not arise by chance. Recently, Russia has introduced restrictions on the use of proprietary software, which primarily affected large IT companies working in tight symbiosis with the public sector. Our experience working on portals in the external and internal environment of the largest employers in Russia and abroad allowed us to evaluate the best practices and standardize many things.

For example, the site of RosstandartIt is in the public domain, it has a large flow of visitors, so we searched and created solutions that will work stably under conditions of instant high load. It is also necessary to consider that many portal elements need to be indexed in search engines (from the point of view of a self-written portal solution it is not as simple and obvious as it seems), but we did not use ready-made solutions and CMS (since not everyone could handle tasks set by the customer).

The main interest of the implementation for us as developers is that each portal service is a self-contained unit that can exist without knowing anything about neighboring services. Due to this, they can be developed, updated and scaled independently of each other.

This article is the first part of a series of articles: it describes how everything looks from the backend side, what technologies we use, how the architecture works, what difficulties and advantages the approach we have implemented has. The following articles will also describe the frontend development side.

So, let's begin.

Architecture


The main goal was to develop a platform based on open-source solutions, which is easy to scale and maintain.

This platform has two deployment options at the customer’s facilities:

1) The first option requires more capacities, but allows you to easily scale and support multiple systems at the same time due to microservice architecture. There are many ways to implement it, but we chose the Spring Cloud (Gateway, Discovery) framework along with Spring Boot (based on experience, our Java architect decided that this technology stack would be best suited for these tasks).



2) The second option is modular. It is considered if it is planned to support only one system (portal). This option is good because the entire infrastructure can be deployed on one or two servers.



Used technologies


Open JDK 8

Wildfly -

PostgreSQL application server - DBMS. All modules are implemented on the basis of ORM technologies, therefore, in theory, it will make it easy to migrate to another DBMS.

ExoPlatform is a portal platform that allows you to combine various modules in a single web application, manage their location on the page, administer access rights and much more. Community version can be downloaded on the official website , but only for Tomcat application server. You can take the sources and modify the assembly of the platform for Wildfly. Holivar in this article on the topic: "why did not choose Liferay" will not arrange.

Modules- these are the so-called portlets, they are executed in the portlet container and are responsible for drawing their area on the page. Portlets are being developed according to the JSR specification ( JSR 362 ).

Spring Framework is the basis of all applications developed in Digital Design from the point of view of Java development ( https://spring.io/ ). The framework has many modules for implementing various tasks (Spring MVC, Spring Mail, Spring Data (JPA), Spring JMS and much more).

Spring Cloud - the basis for the development of microservices (Discovery, Gateway, Ribbon, Config Server).

Keycloak- authorization and access control server (IDM \ IAM). The server supports several authorization protocols (OpenId, SAML, Oauth, Kerberos). This allows you to easily manage authorization strategies, configure access rights and ensure security of access to system resources. (https://www.keycloak.org/).

Kurento is a media server. It was used to develop video and audio broadcasts between users using the WebRTC protocol (https://www.kurento.org/).

Elasticsearch is a full-text search server. To search the portal, its own crawler was developed, which collected data from the portal for indexing, as well as a parser for extracting the contents of files.

Swagger is a self-documenting guide for interacting with the Rest-API.

Jasperepors- Reporting framework. JasperStudio is used to build report templates.

Graylog is a server for collecting logs from applications.

All modules are built using Maven and CI \ CD (Jenkins).

Modules and Services


Microservices that ensure the functioning of modules include:

  • File management service - a microservice that allows you to store data in a single storage, analyze the contents, convert files to other formats for playback or display on the web interface, download and attach to other systems. As well as co-editing files and their versioning.
  • – , . , , Elasticseach ( , ..). Apache Tika, .
  • – , - \- WebRTC - Kurento.
  • – , . ElasticSearch, JasperReports.
  • \ – , .
  • – , , , .
  • - – , Workflow BPMN. JBPM.



  • WCM – .
  • – , , .
  • – .
  • – .
  • – .
  • – .
  • β€” - c .
  • – -. , , .
  • Organizational structure - various directories and reports.
  • Administrative module - Management of access rights to the portal, sites, pages, portraits and display settings for portlets, pages and virtual portals.
  • Navigation - allows you to create custom navigation on the portal and portal nodes.
  • Surveys and tests - a large module with the ability to create tests and surveys of various types (from a single choice, to questions-loops and differentials). Designed for testing, training and conducting surveys among portal users.

And there are many other specific and not very modules with business processes for specific customers. There are about 30 of them.

Difficulties and advantages.

Benefits of this solution.


  • , .
  • , , , .
  • JNDI, . .
  • , .
  • , , .
  • .
  • , , open-source , .
  • .
  • open-source , .

,


  • API – .
  • – ABAC .
  • – - .
  • Centralized data caching - a centralized cache server is planned to be introduced (this is necessary for centralized collection of logs for the work of dev test prod stands to correctly monitor the stability and correctness of our ecosystem).

Total


This set of technologies, coupled with the chosen architectural solution, allows us to raise the issue of developing portal, intranet and external solutions "on the conveyor": do it quickly, efficiently, with different settings and use the best practices.

And in the next article we will tell how our frontend is arranged: about the assembly and how we achieved great reuse.

Continued very soon ...

More articles:


All Articles