本文選題：海量小文件存儲 + 分布式文件系統　； 參考：《國防科學技術大學》2012年碩士論文

【摘要】：近年來，企業和個人數據都呈現爆炸性增長的趨勢。谷歌首席執行官EricSchmidt表示，現在全球每兩天所創造的數據量等同于從人類文明至2003年間產生的數據量的總和。如何存儲海量的數據，成為當前存儲系統所面臨的巨大挑戰。傳統集中存儲方式已經滿足不了數據存儲的需求，，于是出現了用于大規模數據存儲的分布式文件系統，如Google File System（GFS）、Hadoop File System（HDFS）、PVFS、Luster等。 這些分布式文件系統具有良好的可擴展性和容錯特性，能夠滿足海量數據存儲的需求。但是在很多應用場合除了要求支持海量大文件的存儲，還需要支持海量小文件的存儲。雖然GFS、HDFS等分布式文件系統能夠滿足大文件的高效存儲，但在存儲海量小文件時，效率卻很低。針對此問題，工業界和學術界提出了很多方法，但普遍存在性能低，系統可靠性不高，不能高效存儲小文件元數據等問題。針對這些挑戰，本文設計實現了一種基于HDFS的海量小文件存儲系統。 該系統的主要設計思想是，在HDFS現有的目錄樹結構下，將一個文件夾內的小文件，打包成一個大文件進行存儲，該文件稱為小文件數據文件。同時生成小文件索引，記錄小文件在對應數據文件中的位置。 本文設計和實現的基于HDFS的海量小文件存儲系統是可擴展、高容錯、分布式的海量小文件存儲集群系統。本文提出小文件聚合存儲技術通過將小文件數據存儲在HDFS數據文件中，實現數據的分布式存儲和容錯；同時提出小文件分布索引管理技術將索引分布到各個數據節點管理，解決了單一元數據節點在存儲海量小文件成為瓶頸的缺點；設計的海量小文件存儲系統索引容錯機制通過對索引進行容錯，降低小文件丟失的風險；通過在單個目錄下創建多個多數據文件，解決訪問同一目錄下小文件沖突的問題。在以上基礎上，系統在客戶端緩存用戶常用到的小文件索引位置及數據文件流的信息，提高系統的文件訪問的效率。 通過實驗表明，該系統小文件讀寫延遲、吞吐率與不增加小文件支持的原生HDFS相比有了很大的提高。并且，該系統能夠有效解決海量小文件存儲元數據過于龐大的問題，且通過索引容錯機制，提高了該系統的可靠性。
[Abstract]:In recent years, both corporate and personal data have shown an explosive growth trend. Google CEO Eric Schmidt said the amount of data created every two days in the world is now equivalent to the amount of data generated between human civilization and 2003. How to store huge amounts of data has become a great challenge to the current storage system. The traditional centralized storage method can no longer meet the requirement of data storage, so distributed file systems for large-scale data storage, such as Google File system / GFSU / Hadoop File system HDFSU / PVFS Luster, etc., have good extensibility and fault tolerance. It can meet the demand of massive data storage. However, in many applications, it is necessary to support the storage of large files as well as large files. Although distributed file systems such as GFSU HDFS can satisfy the efficient storage of large files, the efficiency of storing large numbers of small files is very low. In order to solve this problem, many methods have been put forward by industry and academic circles. However, there are many problems such as low performance, low reliability of system and low efficient storage of small file metadata. Aiming at these challenges, this paper designs and implements a large amount of small file storage system based on HDFS. The main idea of this system is that, under the existing directory tree structure of HDFS, a small file in a folder is designed. Packaged into a large file for storage, this file is called a small file data file. At the same time, the index of small files is generated, and the location of small files in the corresponding data files is recorded. This paper designs and implements a large amount of small file storage system based on HDFS, which is an extensible, highly fault-tolerant and distributed large size small file storage cluster system. In this paper, we propose a small file aggregation storage technology to realize distributed data storage and fault tolerance by storing small file data in HDFS data file, at the same time, we propose a small file distributed index management technology to distribute the index to each data node management. It solves the problem that the single metadata node becomes the bottleneck in storing the large amount of small files, and the fault-tolerant mechanism of the index of the mass small file storage system can reduce the risk of small file loss by fault-tolerant of the index. By creating multiple data files in a single directory, the problem of accessing small files in the same directory is solved. On the basis of the above, the system caches the information of small file index position and data file flow, which is commonly used by users in the client side, and improves the efficiency of file access of the system. The experiment shows that the system has delayed reading and writing of small files. Throughput is much higher than native HDFS without small file support. Moreover, the system can effectively solve the problem that the large amount of metadata stored in small files is too large, and the reliability of the system is improved by index fault-tolerant mechanism.
【學位授予單位】：國防科學技術大學
【學位級別】：碩士
【學位授予年份】：2012
【分類號】：TP333

【參考文獻】

相關期刊論文 前2條

1 楊德志,黃華,張建剛,許魯;大容量、高性能、高擴展能力的藍鯨分布式文件系統[J];計算機研究與發展;2005年06期

2 余思;桂小林;黃汝維;莊威;;一種提高云存儲中小文件存儲效率的方案[J];西安交通大學學報;2011年06期

本文編號：2001333