Background Introduction
The deputy manager of the PSS department of Eastern Airlines’ Information Technology Department, Tong Shuaihua, stated that the choice of MongoDB was to build a new generation shopping system for Eastern Airlines, hoping that the backend data storage would not only have all the query functions of relational databases but also support variable data models and structures.
So, how did Eastern Airlines specifically carry out the technology selection? What were the results? To this end, InfoQ conducted an exclusive interview with Huang Chong, the technical leader of the related project at Eastern Airlines, to answer these questions one by one.
Huang Chong, architect of the PSS department at Eastern Airlines. As an engineer, he has worked at PCCW, Sinolife, and other companies, with over 10 years of development experience, familiar with open-source products such as Spark, MongoDB, and Drools.
Eastern Airlines’ Practice with MongoDB
Huang Chong: Currently, Eastern Airlines is using MongoDB in only one project, which is the Eastern Airlines Shopping project. It aims to provide users with personalized flight search services, supporting multi-destination searches, budget-based searches, city-themed searches, inspiration semantic searches, and real-time low-price calendar searches, among others. At the same time, it flexibly combines transit paths to improve OD route coverage.
The technical challenge posed by the above project background is: the complexity of backend inventory and fare searches brought by a single frontend search is dozens or even hundreds of times greater than before.
Project Configuration: Using 3 machines with 64 cores and 128G memory SSD hard drives.
Production Environment: A MongoDB Replica-set cluster consisting of 3 servers.
Access Volume: Currently, part of the channels have been promoted online, with passenger query volumes exceeding 6 million times daily, translating into over 45 million database queries daily.
Total Number of Database Records: 72 billion records, with over 26 million updates daily, and over 99% of queries completed in less than 200ms.
Why Choose MongoDB?
Why choose MongoDB for this project? What aspects did Eastern Airlines evaluate to ensure normal or better service? What was the specific evaluation process, data, and results?
Huang Chong: The project characteristics are: high load and high concurrency. Therefore, when selecting a database, we mainly referred to the following technical standards:
-
Support for in-memory queries to reduce disk I/O interactions.
-
Support for complex and variable data storage structures and types.
-
Support for cluster architecture to ensure high availability.
-
Support for complex SQL queries, such as searching for flights that meet conditions based on budget; or searching for flights based on flight schedules (morning and afternoon), etc.
Based on the above standards, we ultimately chose MongoDB.
At the beginning of the project, we evaluated technologies such as Greenplum, Oracle, MySQL, and MongoDB, considering RDBMS (ACID) and NoSQL (CAP); looking at both key-value types and the need for complex range searches. We considered the impact of total data volume on performance, as well as our high query performance requirements, while the write performance requirements were not as high but secondary. We also took into account the actual development difficulty and learning curve, as well as operational costs.
Finally, after comparing performance, development difficulty, and many other factors, we chose MongoDB.
Combining Spark and MongoDB
Does the combination of Spark and MongoDB perfectly match Eastern Airlines’ needs? What advantages does this combination have in terms of real-time performance? How has it performed in current usage, and has it shown advantages or exposed shortcomings?
Huang Chong: Spark, as an in-memory computing engine, has now entered Eastern Airlines’ field of vision. We conducted POC tests and usage scenario validations, proving that it helps improve our performance. Next, we will consider applying Spark + MongoDB technology in production.
We validated the advantages of this technology in fare calculation scenarios. Originally, each request required fare calculation, but now we store pre-calculated results in MongoDB, transforming the calculation performance issue into a query performance issue. Based on our experience in optimizing queries in MongoDB, we simplified the problem itself.
The advantages are obvious, but the downside is that Spark’s map method does not adapt to some custom map programming, making it difficult to implement some complex transit fare calculations in Spark.
Considering Migration, Maintenance, and Costs
How was the migration, deployment, and maintenance of this project considered? How were costs estimated? Is the project officially online and in use now?
Huang Chong: Currently, there is no migration process for the project in use because it is a new project that selected MongoDB as the database. At present, MongoDB is still a new thing for Eastern Airlines, so operations and maintenance still rely on the project team and MongoDB’s original manufacturer for support.
The cost of using MongoDB mainly involves purchasing operational support from the original manufacturer, as the shopping project using MongoDB is a core component of Eastern Airlines’ e-commerce project, and the level of operational maintenance is very high, so it is essential to procure the original manufacturer to support the database’s operations and maintenance.
The project has officially gone live and is in production use.
Technical Implementation and Experience Sharing
How was the technical solution of this project specifically implemented? Were there any obstacles? What experiences can be shared for companies looking to adopt similar technical solutions?
Huang Chong: As early as the beginning of 2014, we had already implemented all the functions previously using Oracle as the database with MongoDB Version 2.8. This switch was very convenient and quick, allowing for much code reuse. Since MongoDB is schema-less and does not have to comply with normalization, there were debates when designing the document structure and choosing chunk keys, and MongoDB consultants provided us with significant help.
As the project progressed, MongoDB itself also updated and iterated, and we grew together. At the same time, we found that our new application is at least 5 man-days faster than before. I believe that MongoDB is a very worthwhile database to adopt in scenarios where strong transaction support is not required.
MongoDB is a complete data storage solution with operational capabilities that can cover many scenarios. Deploying in the cloud, combined with Docker and other operational tools, will improve operational efficiency and reduce operational costs.
How to View the Heated Discussions?
Regarding Eastern Airlines’ adoption of MongoDB, there have been many comments online, including many misunderstandings, with supporters and skeptics. How do you view these opinions? If the project’s performance meets Eastern Airlines’ satisfaction, will it be promoted to other projects in the future?
Huang Chong: For us, using MongoDB is not about seeking attention or using open-source technology for the sake of it, but rather conducting multiple rounds of technical validation and several database POC tests based on the usage scenarios to select products that meet our requirements, while also ensuring the costs are acceptable.
Therefore, this selection is beneficial for Eastern Airlines. At the same time, in today’s internet era, traditional enterprises like Eastern Airlines are continuously opening themselves up to embrace internet technology and occupy a place in the civil aviation IT ecosystem.
In future projects, we will continue to consider using MongoDB for business scenarios suitable for it.
The author of this article will share online on September 22 in “Big Data Talk”. Reply with the keyword “Eastern Airlines” to register for the group and listen to more comprehensive and detailed technical details, and communicate with the author without barriers online.
▽
Further Reading (Click the title):
Technology Returns Business! Performance Optimization Practices of the World’s Third Largest Cross-Border E-Commerce Platform
What Functions Should a Complete Microservices System Include?
IT Veteran: What Can You Learn After Ten Years at Google?
If you like us, please give us a thumbs up, and if you love us, please share!
