We live in a world with lots of data. I mean lots and lots of data! Documents, pictures, scans, databases, logs, programs, emails, videos, libraries, records and more. It is absolutely stunning the amount of data each and every person generates. So what do you do when the inevitable happens and you fill up your server?
Storage is cheap! Or is it?
The good news is storage is cheap and getting cheaper. Hard drives and other forms of storage technology keep getting better and better and the price per gigabyte, terabyte, petabyte just keep falling. But there is more to it than just raw storage.
For server drives, you have to backup all that data. And best practices dictate you keep multiple, redundant backups. For example, we keep hourly snapshots for all our managed backup clients. We keep this data both locally (in their very own office for speed) and replicated offsite into the cloud. We also replicate it across different local media so that if any one backup storage drive, appliance or system fails, we have redundant physical storage. Then on each backup storage system, we keep multiple independent backups. Finally, we keep historical backups.
If you have servers and storage in the cloud, it’s almost worse as they charge you more every single month forever as you add storage and backup copies.
And data growth is never linear. If your data grew from 1TB to 2TB in a year, it probably won’t grow to 3TB next year, it probably means it will grow to 4TB next year and 8TB the year after that.
The long story short is that for every 1 Terabyte of live data storage, we may have 4 – 10 Terabytes of total storage space in use!
That doesn’t just dramatically increase costs, it greatly increases backup windows, recovery time in a disaster, and management headaches.
Cool Tricks of the Trade
So what can you do if you are running out of space or would like to shrink the complexity and consequences of so much data? Here are some of the tricks we use.
- Utilities: There are some great utilities out there that help you find out where all that data use is going to! Tree File Size (there is a cool pro trial version and a limited but free version) lets you rank files and folders from the largest to the smallest. It often can point out an area you had no idea was growing out of control! Some utilities have graphical illustrations of folders by the sizes they take up.
- The Largest Files: Windows Explorer has a built-in feature where you can search for files over a certain size. A trick you can do is to search for large files, say over 100MB. If you get no files found, lower the size you search for. If you get too many to go thru, raise the size. Ideally, try to have it show you the top 100 or so files in size you have. Are they expected and reasonable? Did you find files that surprised you? It’s not unusual to find the receptionist has personal pictures, or maybe Jimmy in accounting is scanning massive files with the copy machine that could be purged or reduced in size dramatically? Maybe you find old and unneeded data?
- Temporary files: Computers love to make temporary files all over. They are supposed to automatically delete them in most cases but they often leave them behind. There are free utilities that can help you find and delete temporary files. You can also manually search for many types in Windows such as *.tmp, *.temp, *.$$$, *.~*, and other known suspects.
- Backup Files: This is another common culprit. Some programs make backup copies of their data automatically and those can easily get out of control. QuickBooks for example can flood a drive with backup copies over the years. Microsoft Office applications also frequently keep temporary backups that it forgets to purge.
- Archive: Chances are your data has been moved from server to server over the years and very little has been tossed out. Those projects from 10 years ago? Probably still there. That old program's database? Yup, also probably still there. One easy trick is to take all of the old data off that drive and archive it someplace else. That can be offline storage or a different drive.
- Deleting Data: The best option of all! If you use the methods above to find old data, decide what is no longer needed. Often this is the biggest chunk of data that research finds. It can be silly things like multiple copies of the same databases or folders. Maybe a user from a decade ago copied their entire computer’s hard drive to the server. Maybe you find old stuff that isn’t needed any longer.
- Old User Data: Administrators are often paranoid to delete data that was used by former employees. But it might be time to take a hard look at that data. Particularly employees that haven’t been with the firm in years and the data may have even gone untouched since they left.
- Pictures: Pictures can take up a surprisingly large amount of space. There is a lot of data stored in every picture file and with modern cameras and phones taking such high-resolution pictures, the sizes have exploded. Are there any large folders full of pictures? Is anyone keeping personal pictures on the server?
- Scanning! This one is a frequent culprit! You want to set your scanner to scan at the lowest acceptable resolution. You would think that if you doubled the resolution of a scanner, say from 150 dpi (dots per inch) to 300 dpi you are doubling the file size but in actuality, file size grows exponentially and may be 4X larger or even larger than the lower resolution. And it never fails! We often find firms that set their scanners to the highest setting and have rapidly filled up their server's hard drives! So what was scanning at 150dpi and is now scanning at 2,400 dpi takes up 100 times as much storage per page than it was before!
I’ve done everything above to clean, but I still need more space!
If you have done everything above and still need more space, you have a few options.
- You can grow the drive: If you have a virtual server and the engineer that built your server left free/safety space “unassigned”, it may be possible to grow the existing drive by assigning it more of the unallocated space. If that isn’t possible, some of the advanced hardware setups in servers (we like to set this up in the servers we design for our clients) allow you to add a physical hard drive or drives to the server and expand the existing “drive” on the fly. It’s tricky but pretty cool and a pretty smooth way to grow the existing storage.
- Add another drive: Another option is to add one or more physical drives and set up a new, additional virtual hard drive. So if before you had the X drive that was full, you now have a Y drive in addition that is empty. The downside is you have to move some of the existing data from X over to Y. Often times the data can be split logically. Maybe client data is on X and administrative data is on Y. Or maybe you archive data to the new drive.
- Is it time for a new server: A bigger question may be to ask if this is just the tip of the iceberg? Is the server old? Is it running out of other resources like memory? Is it out of warranty or running slow? It might be time to get a new server which almost always comes with a lot more storage.
Some “Gotchas”
Some advice from engineers that have been around the block for a while. Stay way ahead of storage limitations. It is much easier to keep a close eye on storage and deal with it well before the space gets low than it is to try and deal with it when it’s getting full.
Another whopper is file permissions. If you have to add in more storage or move data to a new drive, it’s fairly common to have issues with permissions. For example, if a folder is locked down, it might not be movable to the other drive. And it can be very hard to figure out what did and didn’t move when you are talking about moving millions of files.
So, while something as fundamental as storage is often overlooked, it can be a huge advantage to review it regularly, clean it and stay well ahead of any limits. A little maintenance and attention now can save a lot of money and even more headaches! If you have any questions about storage or need any help, just give us a call!