> ... I have about 2 million images ...
> folder structure
The "Who's On First" gazetteer with ~ 26M geojson records - using 3-number chunks subfolder structure.
> folder structure
"Given a Who's On First ID its (relative) URI can be derived by splitting the ID in to 3-number chunks representing nested subdirectories, followed by filename consisting of the ID followed by .geojson. For example the ID for Montréal is 101736545 which becomes: 101/736/545/101736545.geojson"
it is working .. but this is also not optimal
"As of this writing it remains clear that this approach (lots of tiny files parented by lots of nested directories) can be problematic. We may be forced to choose another approach, like fewer subdirectories but nothing has been decided and anything we do will be backwards compatible." ( from https://whosonfirst.org/data/principles/ )
Now the structure have been migrated to per-country repositories ( https://whosonfirst.org/blog/2019/05/09/changes/ )
so the US structure is: https://github.com/whosonfirst-data/whosonfirst-data-admin-us/tree/master/data
maybe you can adopt some ideas.
imho: with 3-number chunks representing nested subdirectories - you can choose more file systems / hosting solutions ..
regards,
Imre
Estevan Rech <softrech@xxxxxxxxx> ezt írta (időpont: 2021. dec. 20., H, 11:30):
How is this folder structure like 10,000 folders? and the backup of it, how long does it take?
