When Ruby doesn't clean up after itself: tempfile leftovers
Today, Planetaki went down for a few minutes as the master reaper server ran out of hard drive space. The first thing you do when you run out of space is remain calm and try to find out whats using it all up. In this particular case where the database is the only real space user (and its not that bad) everything was fine and the log files where all within limits. It didn’t take long to figure out that the temporary directory had turned into a 30GB monster and was locking up the system.
We use the ruby open-uri library to download content from the web which stores everything it downloads in temporary files. Normally, these are supposed to be destroyed as soon as the object that is using the temp file is removed. It turns out however that these files are not destroyed correctly if the Garbage collector does not get chance to run before the program terminates.
In the case of our Planetaki backend, we are opening new sites constantly and downloading files and use forking to download from lots of feeds at the same time. The tempfile library it seems has a problem with forking, and will not correctly remove the tempory files when each instance terminates.
The solution, take the garbage out before going to bed:
spawn do
open('http://website.com') do | f |
# do stuff with data
end
GC.start
end
The GC.start at the end there is vital it seems to ensure your temp files don’t cause you system to run out of hard disk space.
No comments yet