Sunday, August 27, 2017

filesystems - How many files can I have directly under a directory in ext3?



I have a root directory 'data_0'.

Under this directory are about 15,000 directories ('a', 'b', 'c', ... 'aa', 'ab' ...).
Under each of theses directories there are thousands of very small files (4~10kB), something between 1,000 and 2,000 files each.



All this leads to 30 million files. I need to move these from 'data_0' to a 'data_1' folder, but without the "level 2" folders (a, b, c etc), so:



/data_0/a/1.txt --> /data_1/a_1.txt
/data_0/a/2.txt --> /data_1/a_2.txt
...
/data_0/ccc/989.txt --> /data_1/ccc_989.txt
...



How far can I go with this? Performance is unimportant here. Is there a logical limit or just a performance limit?


Answer



If strangers on the internet are to be trusted, there is no limit to the number of files that an ext3 folder can contain. So says the ext3-users RedHat mailing list. The 2.6 kernel supposedly allows for the theoretical allowance of "billions" of files in one directory. You may want to tweak dir_index a bit to make it run smoothly if you'll be doing in searching on the files. There are also some other side effects of massive amounts of files in one directory that you might want to read through in this StackOverflow thread.



So the answer is most likely: "Yes, but..."


No comments:

Post a Comment

linux - How to SSH to ec2 instance in VPC private subnet via NAT server

I have created a VPC in aws with a public subnet and a private subnet. The private subnet does not have direct access to external network. S...