On March 3rd, 2006 Stewart Smith (not verified) says:
If your servers crunch under only 2 million files, you're probably using the wrong file system. Remember, on ext3, things get sad with more than a few hundred files in a directory. XFS (and reiser) do a *lot* better.
One way to "replicate" between filesystems on different machines could be to have a central NFS share for "incoming" with a directory for each slave. Copy the file into an "incoming" directory, hard link into directory for each slave. As a script running on the slaves finds files in its directory, it copies it to its local storage and unlinks (rm) it from its incoming tree.
If you need to wait until a file is copied to a slave - just check for it in the incoming directory of that slave!
To resync/add a new slave - add a directory in the incoming tree (as your scripts are smart about this and look at what directories are in the incoming tree) and rsync the data across!
What's extra tricky dicky is to serve the images directly off these machines - i.e. run an unlta-light web server! This way your application (running in, for example PHP) runs on one box just generating URLs to another - load balancing!
If your servers crunch under
If your servers crunch under only 2 million files, you're probably using the wrong file system. Remember, on ext3, things get sad with more than a few hundred files in a directory. XFS (and reiser) do a *lot* better.
One way to "replicate" between filesystems on different machines could be to have a central NFS share for "incoming" with a directory for each slave. Copy the file into an "incoming" directory, hard link into directory for each slave. As a script running on the slaves finds files in its directory, it copies it to its local storage and unlinks (rm) it from its incoming tree.
If you need to wait until a file is copied to a slave - just check for it in the incoming directory of that slave!
To resync/add a new slave - add a directory in the incoming tree (as your scripts are smart about this and look at what directories are in the incoming tree) and rsync the data across!
What's extra tricky dicky is to serve the images directly off these machines - i.e. run an unlta-light web server! This way your application (running in, for example PHP) runs on one box just generating URLs to another - load balancing!