Even Technology: In-depth understanding of the "integration of lakes and warehouses" to avoid missing the best strategic opportunity for transformation

This month, Alibaba Cloud held the "2022 Alibaba Cloud Data Storage Ecosystem Conference" in Beijing. Even Technology, as a pioneer in the field of domestic cloud-native data warehouse technology, received invited to attend this conference.

As a guest speaker, Tao Zhenglin, Chief Architect of Even Technology, reviewed the evolution of analytical databases with industry technology experts at the meeting, as well as the current use of Even Technology in the lake. Cutting-edge concepts and practices in warehouse integration.

In this regard, Tao Zhenglin focused on the six major features of the lake-warehouse integrated ANCHOR at the meeting: Real-time T+0 , one copy of data, ultra-high concurrency, data consistency, cloud native, and multi-type data support. With the support of the latest version and architecture of OushuDB, the even-numbered lake and warehouse integrated solution will help customers unleash the value of their data on cloud facilities.

Why the split model of "lake" + "warehouse" is not the best choice

With the gradual promotion of Hadoop big data platform construction in recent years, enterprises I began to try to use Hadoop for some non-core scenarios, but Hadoop has limited performance and concurrency support, weak transaction support, high delivery and operation and maintenance costs, and cannot replace the core data warehouse. It can basically only be used as a "data lake". In order to meet user requirements in terms of performance, transactions, etc., many companies have begun to consider the complementary approach of data lakes and data warehouses. While building the data lake, MPP is also used. The lake warehouses are deployed independently, and the data is connected through ETL.

This is the "Lake and warehouse split" model of Hadoop+MPP that is often referred to in the industry.

Although this model allows lakes and warehouses to have very complementary technical features, it also creates serious problems that often confuse enterprises, including:

These common situations give practitioners even more headaches. To solve these problems, it is necessary to form an integrated architecture at the data and query levels and completely get rid of these bottlenecks encountered by the big data platform. This can greatly reduce IT operation and maintenance costs and the technical threshold of data management.

What is the difference between the integrated lake and warehouse model of OushuDB that separates storage and computing?

So, what is the difference between the integrated lake and warehouse model based on OushuDB's separation of storage and computing and the "split lake and warehouse" model of Hadoop+MPP What's the difference?

OushuDB, the world's fastest new-generation analytical database engine developed by Even Technology, innovatively adopts a cloud-native architecture that separates storage and computing. As a new data platform architecture, the storage and computing separation architecture can ensure that storage and computing can be elastically expanded and scaled independently.

Neither traditional MPP nor Hadoop adapt to such requirements:

In addition, in order to meet the needs of real-time stream processing, real-time on-demand analysis and offline analysis at the same time , Odd Technology has creatively explored the Omega full real-time data processing architecture, which has obvious advantages compared with the traditional Kappa architecture and Lambda architecture.

It can be said that OushuDB has basically solved the technical bottleneck of "Lake warehouse split", and its technical advantages are quite obvious:

When selecting Hucang, ANCHOR goes first

Odd Technology believes that to truly solve the pain points of the business, choose the right company For Hucang products, we can select them according to the ANCHOR standards mentioned earlier. The six initials of ANCHOR respectively represent six major characteristics:

Industry recognition and even number’s continuous breakthroughs and innovations

Since the birth of even number technology, even number technology’s products and solutions have also It has been widely deployed and applied in non-banking finance, telecommunications, government, energy, manufacturing, Internet and other industries, helping companies in multiple industries to take small steps and carry out digital and intelligent transformation. At the same time, as a leading startup in the database field, the feasibility and growth of its business model have also been recognized by capital, and it has successively received four rounds of investment from top domestic investment institutions Sequoia China, Tencent, Redpoint China and Kingsoft Cloud.

Among the common customer industries of big data, the banking industry is one of the fields with the highest requirements for autonomous controllability, high availability, and high reliability of applications. Even Technology solves this problem The implementation of the solution in the banking industry is a testament to its technical strength and understanding of user pain points.

As early as 2020, Odd Technology established a high-performance big data joint laboratory with China Construction Bank to jointly explore the implementation path of lake-warehouse integration. After continuous technical discussions and application verification, the two parties jointly developed a fully real-time integrated lake and warehouse solution based on cloud native database technology. It uses a set of technology stacks and unified storage to build dual lake and warehouse capabilities. It has extremely fast performance and elastic scalability. , on-demand allocation of computing resources, single storage of all data, no need for frequent derivatives, mixed loads and other related capabilities, can fully build real-time application scenarios for banks and their customers, helping CCB improve real-time demand response performance, enhance system flexibility, and save money at the same time Operation and maintenance costs.

Recently, Even Technology was officially selected into the list of national-level specialized and special new (specialized, refined, distinctive, novel) "little giant" companies. As a start-up that helps the country break through the "stuck" problem in key technology fields, Even Technology's efforts in database localization and technological independence and security are being gradually verified and recognized at the national level.

With the gradual establishment of the Internet of Things and the Industrial Internet in the future, the field of big data will face an increasingly wide range of data sources, an increasing amount of data, and an increasing amount of unstructured data. With increasingly rich application scenarios and increasingly complex technology stacks, the difficulty of big data processing and analysis will further increase. From databases in the 1960s, to data warehouses, data lakes, and now to integrated lakes and warehouses, new products always solve the business pain points of previous practitioners in terms of performance and functionality. We can say that integrated lakes and warehouses are the development of databases. It is an inevitable product of the cloud native era.

Through virtual computing cluster technology, we can achieve high concurrency on ultra-large-scale clusters with hundreds of thousands of nodes, ensure transaction support, and provide real-time capabilities. There will be no data islands for one piece of data. The new generation of lake-warehouse integrated architecture will be future development trends. As a leader in the field of lake-warehouse integration, Even Technology will continue to optimize technology to bring users higher performance and more robust solutions, supporting more industry users to transform data into productivity.