What are the mainstream data analysis platform architectures?

Hadoop adopts Map Reduce distributed computing framework, develops HDFS distributed file system according to GFS, and develops HBase data storage system according to large tables. Hadoop's open source characteristics make it the de facto international standard for distributed computing systems. Yahoo, Facebook, Amazon, Baidu, Alibaba and many other Internet companies in China have built their own distributions based on Hadoop.

2. Spark

Spark has made some architectural improvements based on Hadoop. The biggest difference between Spark and Hadoop is that Hadoop uses hard disk to store data, and Spark uses memory to store data, so what Spark can provide is better than Ha? Doop 100 times the operation speed. Spark can't be used to deal with data that needs to be saved for a long time, because the memory will lose data after power failure.

3. Storm

Storm is a distributed computing system promoted by Twitter. Based on Hadoop, it provides the characteristics of real-time operation and can handle large data streams in real time. Unlike Hadoop and Spark, Storm does not collect and store data. It directly receives and processes data in real time through the network, and then directly returns the results in real time through the network.

4、Samza

Linked In's open technology Samza is a distributed stream processing framework, which is dedicated to real-time data processing, much like Twitter's stream processing system Storm. The difference is Sam? Za is based on Hadoop and uses Linked In's own Kafka distributed message system.

Samza is very suitable for real-time streaming data processing, such as data tracking, log service, real-time service and other applications. It can help developers deal with high-speed messages and has good fault tolerance.