When to use the in-memory database

Compared with disk, the speed of reading and writing data in memory is several orders of magnitude higher, and saving data in memory can greatly improve the performance of application programs compared with accessing from disk. At the same time, the memory database abandons the traditional way of managing data on disk, redesigns its architecture based on all the data in memory, and makes corresponding improvements in data cache, fast algorithm and parallel operation, so the data processing speed is much faster than that of the traditional database, generally more than 10 times. The main feature of the main memory database is that its "master copy" or "working version" memory resides, that is, active transactions only deal with the memory copy of the real-time main memory database. Obviously, it needs a lot of memory, but the whole database is not always stored in memory, that is, the main memory database system still has to handle I/O.

Memory database is exchanged for real-time data processing at the expense of memory resources. Memory database and disk database are both essential relational database products for every enterprise in today's information society. Disk database solves the problems of mass storage and data analysis, while memory database solves the problems of real-time processing and high concurrency. Their existence is complementary, and the real-time transaction processing performance of main memory database is far better than that of disk database. But relatively speaking, its data security has not reached the level of disk database.

In-memory database uses physical memory as the first storage medium of data and disk as backup. With the development of telecom services, the system has very high requirements for real-time and flexible modification of services. In this case, the demand for in-memory database is also increasing. The method of disk database is to store data in memory for processing, which is not guaranteed in manageability and data security and reliability. The main memory database has been improved for this weakness.

In fact, in-memory database is not a fashionable technology. It appeared in the late 1960s, but it began to develop in the late 1990s due to market demand. As a new generation database, Altibase products have moved towards hybrid database, and its version Altibase 4.0 already has its own disk database. Once users buy the main memory database of Altibase, they don't need to buy the disk database. It puts hot data (frequently used, frequently accessed and frequently calculated data) in memory database and historical data in disk database, which can further reduce users' investment.

For in-memory databases, one part of the same database can be stored on disk, while the other part can be stored in memory. Users can choose to store data in a memory table to provide instant data access. If the access time is not urgent or the data stored in the memory takes up too much space, users can store these data in the disk table.

For example, when a mobile phone user starts to make a call, if a hybrid data management engine based on memory database technology is applied, its service options will be retrieved through the memory table and the user's identity will be verified immediately, while the call list and billing list will be archived in the disk table. Therefore, the balance between speed and resource use is achieved.

A very important feature of in-memory database technology is that it can handle all data in memory, which is completely different from just putting data in memory in the form of arrays. Moreover, the main memory database is application-independent, so obviously this architecture is reasonable. The memory engine can use exactly the same database to realize query and archive functions, while the memory table and the disk table also use exactly the same access method. The choice of storage is completely transparent to application developers.

For in-memory database, it realizes the management of data in memory, not just as a cache of the database. Different from other databases that cache disk data blocks into main memory, the memory engine of main memory database uses data structures and algorithms specially designed for random access to memory, which avoids the problem that sorting commands often destroy the performance of cache database. By reducing disk I/O through the main memory database, the processing speed that the traditional disk-based I/O database can not compare with can be achieved.

Therefore, the application of in-memory database technology can greatly improve the speed of database and provide strong support for database applications that need high-speed response, such as telecommunications and finance.

Because most data is operated in memory, the performance of memory database is much higher than that of disk database, which is very suitable for the real-time requirements of telecom enterprise operation support system.

The all-round development of competition in the telecommunications industry will inevitably bring new value chain models and new charging methods. These changes pose a challenge to the existing telecom operation support system. For example, the charging link of various services will no longer be based on duration or communication distance, but may be based on the combination of various parameters such as duration, content and usage. In order to meet these challenges, telecom enterprises have introduced memory database to improve the real-time, accuracy and flexibility of background data management.

Although the main memory database is no longer the concept of traditional disk database, it is still a database in essence, and it also has the basic functions of a general database:

■ Management of permanent data, including definition, storage and maintenance of database;

■ Complete various data operations, such as query processing, access and integrity check;

■ Transaction management, including scheduling and concurrency control;

■ Access control and security inspection;

■ Have a database reliability recovery mechanism.

Compared with calling memory processing through program development, memory database has its own advantages. First of all, the main memory database is a product-based database management software, which greatly shortens the development cycle; Secondly, the main memory database has an open platform and interface, which makes the program development and transplantation more flexible and convenient, and is also convenient for maintenance and secondary development; Third, you can use the unified SQL language to query the data in memory conveniently; Finally, the security and integrity of the data in the database can be guaranteed. These advantages are conducive to rapid deployment and simplified maintenance.

However, the main memory database also has its inevitable shortcomings, such as: it is not easy to recover, the data in the main memory database is not always permanent, and in order to ensure real-time, it is not necessarily consistent and absolutely correct, some are short-lived, some are temporarily inconsistent or not absolutely correct.

Telecom enterprises have always been the main users of main memory databases. In recent years, with the rapid development of computer hardware technology, the improvement of memory capacity, the decline of price and the fact that computers can support larger addresses after entering the 64-bit era operating system, it is possible to realize the main memory database. At present, the application of main memory database in telecom industry is becoming more and more mature. In the case of telecommunication system above 90G, it can automatically expand memory space, provide ESOL custom stored procedures, support multithreading without restarting the database, and has high development efficiency and convenient program transplantation.

Here are two examples to introduce the application of main memory database.

Loading of telecom billing data

Telecom's second price approval and real-time account accumulation are two essential functions in the billing system.

The so-called second batch price is relative to the first batch price.

A price verification is to calculate the price according to the national standard tariff. For example, GSM local calls, 0.4 yuan per minute. After the first price approval is completed, it will be calculated again according to the user's package. Take a 4-minute call from a Beijing GSM user as an example. After the first price approval, the price of this call charge is 1.6 yuan. If this user participates in the monthly package of 10 yuan, then after the second price approval, the cost of this call will be 0 yuan.

The first price approval is used for settlement between major operators, and the second price approval is aimed at individual users.

Real-time phone bill accumulation is to accumulate all the expenses of users from 1 to the present, that is, users can query the real-time phone bill as of the previous day through 10086. Accumulated account value can help users control high telephone bills or provide users with instant consumption information.

The process of secondary price approval and real-time account accumulation involves user-related information such as user data and user packages, which must be loaded when the telecom support system starts price approval. The data of slightly larger provincial operators will exceed 6.5438+million. Due to the combination of packages, products and different preferential rules, the billing processing model is quite complicated. Loading these data is a big expense for the system, which makes the current billing processing speed very slow and it is difficult to update the data in real time. The introduction of main memory database solves this problem to some extent.

The largest amount of data is the subsidiary ledger data in the second charging process. This part of the data does not need to be put in the main memory database. When processing a bill file or reaching the set number of submitted records, directly operating the disk database will not affect the system performance. It is urgent to store user data, packages, data corresponding to business packages and billing packages, model data of billing packages and accumulated user data in the memory database. This part of data query operation is far more frequent than data addition and update operation. Of course, in addition to these data, other data required by the application can also be loaded into the main memory database.

After using the in-memory database, users can query the real-time phone bill through the sales department or customers, which is a qualitative leap in business compared with the current real-time phone bill that can only be queried the day before. Because the query process of the system is exactly the same as before when processing this part of data, the system saves the synchronous link between the data in memory and the data in the disk database, so it can realize real-time query. The same is true of credit control. In the past, the system had to refresh the credit control data according to a certain period after the account was accumulated, so there was a time difference and it could not be completely real-time.

After using the memory database, the signal control can directly obtain the data in the real-time telephone bill accumulation table in the memory database, and completely realize real-time early warning and shutdown. After using the main memory database in the secondary price approval and accumulated accounts, it is also of great benefit to the anti-fraud and income security system, which can fully protect the vital interests of operators.

In addition, after using the in-memory database, the processing speed of system price approval and accumulated accounts is improved as a whole, the pressure of accessing the disk database is greatly alleviated, the efficiency of data query, modification and deletion is improved, and the integration of post-payment and pre-payment can be realized.

Synchronization of telecom billing data

The data of telecom service and billing system are always changing, which involves the synchronization of data in main memory database and data in disk database (for the sake of clarity, the disk database here is illustrated by taking Oracle DB as an example). Data synchronization includes two parts: data synchronization from main memory database to Oracle DB and data synchronization from Oracle DB to main memory database.

1. Synchronization from Oracle database to main memory database

This part of data synchronization adopts the way of incremental table, and the data added or updated by business system or CRM will be generated into the incremental table of Oracle, and the charging background program will query the data in these incremental tables first. If the data can be found in these incremental tables, the data will be updated to the corresponding table of the main memory database, if not, the data will be directly queried from the main memory database, thus ensuring the integrity and real-time performance of the data. Because the data volume of the incremental table is generally small, this part of the operation will not affect the performance of the system.

2. Synchronization from main memory database to Oracle database

Because almost all the batch price and accumulated account data of Oracle's billing background are loaded into the main memory database, the data table corresponding to Oracle database will be mainly used for data backup of the main memory database.

Users' latest real-time telephone charges and other information are stored in the in-memory database, and the real-time telephone charges query will be directly connected to the in-memory database to ensure that users get the latest fee information. Information control is also to query data directly from the main memory database, so there is no real-time requirement for this part of data in Oracle. At this time, the application can generate the synchronization between the main memory database and Oracle, and back it up to the Oracle database at regular intervals, or use the Oracle stored procedure to import data when the system is relatively idle.

Generally speaking, due to the rapid development of market and technology, the telecom business has been continuously expanded and the operation and management have been continuously optimized. Some traditional support systems have gradually failed to meet the growing business requirements and customer needs, so it is inevitable to introduce some new technologies to solve the problems we encounter in production. For example, the memory database is used to replace the previous memory sharing technology, which makes the nonstandard things in memory, including interfaces, formats and management, standardized.

Main memory database is just one of many new technologies. As long as we emancipate our minds and make proper choices, we can completely overcome the bottleneck in the system with little investment and get the maximum return at the lowest cost.

We have seen many general databases, such as Oracle, Db2, Sqlserver, Sybase, Informix, Mysql and Pqllite, which are popular recently. Of course, we can't forget the open source PostgreSQL. Usually, these databases can undertake important business, but they are still slightly insufficient in demanding high performance. In the billing system, if the user information changes frequently, the delay will have a great impact, and even affect the normal operation of the billing system.

The only in-memory database I came into contact with was used by AsiaInfo in the audit system of China Mobile Billing Center. Because the audit system needs to synchronize user status information and subscription information, and then audit the generated bills, if the response speed is slow, it will produce wrong results. When there was no audit system at first, the charging standards were basically issued by sp. However, users often find that when they don't actually use the service or cancel the service, they still charge the bill. Therefore, China Mobile is determined to audit sp's bill and completely cut off sp's charging method based on its own data.

If you want to obtain user status information and subscription information, you need to synchronize from multiple systems, and at the same time, you should review the dialogue sheet. The middle processing time is strict (users may check their phone bill information in a short time), and the response time to the system should be as short as possible.

Universal database is at a disadvantage in this respect. AsiaInfo takes three rx8420 as database hosts, and shares the user information of 3 1 province to the three hosts according to the number. Each province has at least one warehousing process, and for those with more users, multiple processes are adopted for warehousing. The source of data collection is mainly through BOSS and primary billing system.

Because data is stored in memory, the data structure stored is different from that of general databases. At the same time, in order to ensure the safety of data, there is a mirror image of memory data on the disk, and the data in the memory is synchronized to the disk every once in a while, and the data can be recovered through the disk when the host fails. When the host fails, the standby host will take over through HA. However, the logging and rollback of data operations are not as good as Oracle, and only provide a simple recovery mechanism.

In the billing system, the bills sent by sp should be audited first, and the main criteria are user status and subscription information. For example, the user has been turned off for 7 days, and if the new subscription information appears in the bill of the service provider, the bill will be regarded as an error bill. In this way, mobile takes the initiative in the game with sp. After the audit system went online, users' complaints about sp decreased obviously.

Link 1: Similarities and differences between main memory database and traditional database

The traditional database system is a relational database, and the purpose of developing this database is to deal with permanent and stable data. Relational database emphasizes maintaining the integrity and consistency of data, but it is difficult to take into account the time limit of related data and its processing, which can not meet the needs of real-time application of industrial production management, because real-time transactions require the system to accurately predict the running time of transactions.

For the disk database, the actual average execution time of transactions is quite different from the estimated worst-case execution time due to disk access, data in and out of memory, buffer management, queue waiting and lock delay. If the whole database or its main "working" part is put into memory, each transaction will have no I/O during execution, which will provide strong support for the system to accurately estimate and arrange the running time of the transaction, and make it more dynamic and predictable. This is the main reason for the emergence of main memory database.

The data processed by main memory database is usually "transient", that is, there is a certain effective time, when it is out of date, new data will be generated, and the current decision-making deduction will be invalid. Therefore, in practical application, the main memory database is used to process real-time business logic processing data. However, the traditional database aims at processing permanent and stable data, and its performance goal is high system throughput and low cost, so the real-time processing of data should be considered relatively little. In practical applications, traditional databases are used to store data with relatively low real-time requirements.

In practical application, these two kinds of databases are often used in combination, instead of replacing the traditional database with the main memory database.

Link 2: Several in-memory database products

■ Oracle Bone Inscriptions Times

Oracle TimesTen is a memory-optimized relational database acquired by Oracle from Timeten Company, which provides applications with instant response and extremely high throughput required by real-time enterprises and industries (such as telecommunications, capital markets and national defense). Oracle TimesTen can be deployed in the application layer as a cache or embedded database, using standard SQL interface to operate the data storage area completely in physical memory.

■ Altibase

Altibase is a software solution that provides high performance and high availability in a transaction-first environment. It provides high performance, fault tolerance and transaction management capabilities, and is especially suitable for communication, online banking, securities trading, real-time applications and embedded systems. Altibase can maximize the potential of database service system and enhance the processing capacity of data server. Altibase supports client/server architecture or embedded architecture. Client/server architecture is very suitable for general applications. Embedded architecture embeds applications into database server, which is suitable for real-time systems with high real-time requirements.

■ limit b

The eXtremeDB real-time database is designed by McObject company for real-time and embedded system data management, with an overhead of 50K to 130K and a speed of microsecond. EXtremeDB completely resides in main memory and does not use file systems (including memory disks). EXtremeDB adopts a brand-new disk fusion technology, which extends the memory to the disk and uses the disk as virtual memory. The real-time performance is microsecond, and the data management capacity can reach 20G at 32BIT.