How to view over 10,000 files? (photos, etc)

Hi all,

I have just been onto my shared hosting provider (Arvixe) as I cannot view the photos files listing in Cpanel or via FTP. They state that it is a security concern to raise the file view limit above 10K, as it must be done server wide, not by folders. This will obviously create a major problem for any D7 user with over 2000 members with photos and avatars, and in reality not many members with images to create such a problem.

What happened to the individual image folder for each member as in D6? It would take 10K members with images to fill up the folder with D6, now it is a fifth of that with D7. At least D6 could be managed reasonably well and not have ALL the members images in the one folder aka  /modules/boonex/photos/data/files/ folder. One folder for so many files?

How can we possibly manage this properly? What will happen on a site with 50K members with images? How could anyone manage 250K files in ONE folder?

Seems odd that Boonex have chosen such a method, considering D6 member image folders beforehand.

What do you other fellow D7 users do to manage this?

Would someone from Boonex care to offer a solution to my 'small' problem with one of your recommended hosting providers?

More than that...  some file systems, like EXT3, which is commonly used on Linux, has a 32K file limit per folder.

On our large site(s), including one with 1 million users, and 8.5 million images, we have split the folders up into sub-folders.

Using this kind of layout

/media/images/profile/0/1

where 0 folder contains folders for users 0 to 9999...

and the 1, well that is user 1's folder.

/media/images/profile/23/823

would be for user 230823

for example.

This is a real concern, what can be done about it?

Thanks for the input Smoge. I do take it that you can 'mod' for this problem in D7. Have you done it for others? Sorry to ask, I have not been to MMS for a while, guess I better get my butt over there now huh?

This seems so crazy that Boonex would allow such an oversight in folder and file management, surely they know how servers work and our potential problem associated with file limits, especially ALL the profile images in one folder. They need to implement some form of folder and file management for the general 'noob' like me, so we don't have this concern down the track, once things grow a bit.

Does VPS eliminate this problem?

I will have a free site anyway, so little use me paying big for hosting.

Yes... done before...

The mod can be done from the start, or it can be done later... when needed.. using some scripts to move stuff around.

As for getting around it without code changes, you can look into other file systems other than EXT3, which don't have the 32K limit.

But EXT3 is the file system most often used these days.

Can't get MMS Smoge to contact you.

Does anyone know how we can get around this problem? Is it a major hack to bring back individual user ID folders as in D6?

Are we expected to have ALL the images in ONE folder without any possibility to view or edit them all?

this is really serious issue. I would welcome boonex staff reply on this....

Yes! This is really serious issue!  Upload a photo will become the five files!!! (See boonex-photo.jpg)

If 32K file limit per folder:

32K / 5 = 6400 photos

Limit Only "6400 photos" ??? Set up a website for fun, of course, no problem at all, but in fact used in photography site,  this PHOTO mod would be crash soon!

Is there a better photo module?

Not really a serious issue.

32K is not the limit. The information is inaccurate. Many of the utilities in linux cannot handle listing directories that contain over 32K files, however that is not the limit. The limit is based on how the drive is formated, the size of the drive and number of free inodes. You can put millions of files into a folder. I have a feeling that by time you have enough members to actually get that many images, you will be running into other issues.

I manage a site that has somewhere around 7000 images in the photos module...data...files directory. Not huge and unheard of, but a decent amount. Ftp client will not allow me to see them all. Cpanel filemanager will, however it freezes and hangs like you wouldn't believe every time I try accessing the directory.

What I usually do is zip up the entire directory, then download the .zip. Extract it to my local pc, make necessary changes/edits/deletes, and if necessary zip it back up (or the changes anyway)...upload it, and extracted on the server.

It sounds like a pain, and it is to some extent but it usually won't freeze up on me this way.

The good news is I don't need to do this very often.

Ideally it would be nice to split them into several directories, or reduce the number of images that get created for 1 upload would be helpful. Uploading 1 image and resize it on the fly if we could do this without wasting resources. I don't see the point of turning 1 image into 5.

We have some members, of which a few members there are more than 100,000 photos, so this is definitely the problems encountered

Single folder, there are too many files in the server performance will be very slow

http://www.modmysite.com/dolphin-general-discussion-v6-00-above/9081.htm

Linux does have unlimited files, I know people storing billions in one directory without any problems, yet...

And if your on windows don't think you'll get away with it either....windows has a few dirty little secrets when it comes to directories as well but it's up in 1000K range....Windows claims no limits but some testing as shown other wise.

http://www.boonex.com/forums/topic/Speaking-of-photo-albums-.htm

Whose bright idea was it to dump ALL the photos in the same folder on the server? Has anyone undertaken the task of creating a hierarchical folder photo storage setup? (like storing photos by either owner or better yet, owner/album?)

Well, How do I change the the photo modules folder structure, to become "photos/data/files/username/albums-nane/photos.jpg" like this structure?