Softonic is the world's largest app for apk download.
Softonic is the world's largest software and app discovery destination and one of the world's most highly-trafficked websites. You have probably landed on their website when you need to download something - and you are not the only one. Over 100 million users are reaching Softonic - per month. It’s an app guide that helps you discover the best applications for your device, offering you reviews, news, articles and free downloads.
CloudAMQP provides hosted RabbitMQ clusters, in all the biggest data centers all over the world and Softonic is one of our many customers. We met up with Riccardo Piccoli, a developer at Softonic, at the RabbitMQ Summit 2018 in London where he kindly shared Softonic’s customer story with us.
This article is broken down into two pieces; the first part is an overview of the system, which shows a simple RabbitMQ use cases of an event-based architecture. The second part is a deep-dive into the internal architecture in Softonic - plugins they are using and examples of events they are sending.
A simple RabbitMQ use case
Users can upload files to Softonic. All uploaded files are scanned for virus and information about the file is collected, before the file is distributed to other users. The new binary data is, first of all, persisted within a dedicated service, and a notification about the upload is sent to an event bus. Other services collect this information which in the end will be added to the website. In this case, the user gets notified straight after the upload has succeeded and a scanning event is simply placed on an event-bus for other services to handle. An event bus, also called message queue, allows web servers to respond to requests quickly instead of being forced to perform a resource-heavy process on the spot, and instead of keeping the user waiting.
The scanning process is one of those services. The virus scanning application takes a message of the event bus, such as a “ScanFile” command and starts the processing of the file. At the same time, other users are able to upload new files to Softonic and processing tasks are just piling up in the queue. The event “FileScanned” is added back to the event-bus, once the consuming application has handled the event.
The software and app discovery portal Softonic is reached by over 100 million users per month, delivers more than 2 million downloads per day and has a constant flow of events and commands between their services. CloudAMQP met up with Softonic to hear their story on how RabbitMQ, as an event bus between microservices, contributes to a reliable, fast and effective architecture perfect for their purpose.
An architecture like this creates two simple applications and low coupling between the sender and the receiver. Users can still upload files, even if the scanning application is busy or is under maintenance.
Different events or commands are published to the event bus, e.g., a “ScanFile” command.
Softonic is using RabbitMQ as an event bus, events or commands are simply added to the queue.
The consuming application retrieves the event and starts to process the event. Some data is stored to the database, and more events might be published back to another event queue (more about this in “Internal Structure of RabbtiMQ”).
The consuming application stores lot’s of information in a database (MySQL).
When a microservice receives an event, it can update its own business entities, which might lead to more events being published, and that is exactly the case here.
Internal Structure of RabbitMQ
It’s time for a deep-dive into the internal architecture of RabbitMQ, and into the Softonic Application. Two RabbitMQ concepts need to be described if you are not already familiar with them. Softonic is using the consistent hash exchange plugin and RabbitMQ sharding.
asynchronously Event 0 will be processed straight away, event 1,2 and 3 will be re-published to the queue so that other sharded consumers can process them.