IPFS is a powerful file storage system, but it is not without its limitations. In recent months, many individuals and projects have begun storing folders with thousands (sometimes tens of thousands) of files in them. When trying to view the contents of these folders through a gateway, they are confused about why the content won't load.
The simple answer is IPFS Gateways do not have built-in pagination. This means when you try to view a folder with, say, 10,000 files in it, the gateway and the browser are trying to load all of those files at once. As you can imagine, this can be overwhelming for a browser.
How do you fix it?
Well, you can't. But you can get around it. Every one of the files within your folder is still accessible, so you should use the link to your folder directory with the filename appended at the end. Let's say your folder CID is "MY_FOLDER_CID". And let's say you have a file named "1.json". You would load that file by going to:
โ
โGATEWAY_URL/ipfs/MY_FOLDER_CID/1.json
This is still a content addressable way to access files within folders on IPFS. If the contents of the folder ever change, the folder CID would change and thus the path to the file would change. You have the same verifiability assurances with this method.
What If I Need The CIDs Inside The Folder?
In order to get all of the child file CIDs, you're going to need to be running your own local IPFS node. You will need to add the folder to your IPFS node and then run the following command from the command line:
ipfs dag get <FOLDER CID>