|
Audio Asylum Thread Printer Get a view of an entire thread on one page |
For Sale Ads |
68.3.91.159
Backing up my 1/2 TB music library takes 15 hours, the effective transfer rate is slightly below 10MB/s. My disks are rated at around 150 MB/s. I did some research and it appears that 60 MB/s transfer rates via USB are possible but 20/30 MB/s are more realistic. When I start these transfers the rates are well above 20 MB/s but within minutes drop down to 10 MB/s.
I'm using Windows 7, a Sata connected drive and a USB connected drive.
I suspect the issue is with how cache is being used. Does anyone have an idea on how to optimize for maximum data transfer rate?
Follow Ups:
MB = Megabyte
Mb = MegabitA huge difference.
15 hours is a LONG time.
Edits: 08/03/13
sounds like you are transferring the entire file structure.
While that does need to be done once and only once, when making routine backups, its much handier if you use sync toy 2.1. This is free backup software available from microsoft. the software compares the files on drive 1 to the back up drive and only adds new files.
I believe you can control caching using the Windows Device Manager. You go to the disk drive you want to set and you open the Policies tab. However, you may already have this set up to work in the "fast" mode if transfers start out at high speed and then slow down.
Don't expect to get anything like the "rated" speed of a device in a file copy. If there is a bottleneck elsewhere, then it will be impossible. This may be the processor or some I/O channel limitation. Details will depend on your particular configuration. As a simple test of whether or not the problem is USB related, you might try doing an internal file copy and see what speed you get. If it is much faster, then you will want to go the ESATA route. For this test to be valid you will need to use two separate drives, otherwise you will probably get degraded performance due to disk seeks. (You will probably be able to hear these seeks.) If you are using a desktop adding an internal SATA drive won't be too hard. Good luck if you are using a laptop, but in that case that's probably your problem in the first place.
So if you want more speed there are really only three possibilities:
1. faster computer
2. faster devices (e.g. ESATA instead of USB 2.0)
3. get the job done by moving less data.
Several months ago, I backed up 4 TB of data, moving it from one computer through a second computer to an external drive connected by ESATA. This took about 24 hours, but it was no big deal as this was a one-time event associated with purchase of a new off-site backup drive.
I used SyncBackPro software. This allows recovery from a partially completed transfer where it left off, which means that the fear / frustration factor of these long transfers is largely gone. Once this was set up, subsequent monthly backups to this disaster recovery drive only involve those files that have changed and typically involve a few hundred GB of data, mostly Windows backup files, plus additions to my music library. (Before the transfers start it takes a few minutes to scan the two file systems and figure out which files need to be copied.)
Tony Lauck
"Diversity is the law of nature; no two entities in this universe are uniform." - P.R. Sarkar
There are lots of variables. System loads will affect transfer rates and most copy or backup programs do verification, so after the file is written, it's read from the backup and compared against a check sum of the original.
Other than USB 3.0 which is quite a bit faster, I don't see any way that's safe to speed it up. Music files are already compressed data and turning off verification isn't a great idea.
Once you have a copy though, it shouldn't take long at all to update the backup. I use a simple batch file as a scheduled task with xcopy:
xcopy M:\Music\*.* r:\Music\ /D /S /E /C /I /H /R /K /Y
I just ran it for grins and it took 2 minutes to compare my local USB drive over the network to my wife's system with another copy on a 1TB USB drive. If I'd added a few new CDs, it'd take another couple minutes per CD.
-Rod
I get about nearly 100 MB/s speeds over a Gigabit Ethernet to/from my NAS. However, this is basically irrelevant for backing up my music library. I use SyncBackPro software which works on the file directories on the main volume and the backup volume. I have it configured so that it automatically backs up all new files on the audio PC to the NAS, but flags any other differences for my review. This way "cockpit error" such as accidentally deleting the main copy of a file doesn't propagate to the backup. Another problem with traditional backup software that is avoided with properly set up file sync software is that "bit rot" caused by disk hardware glitches won't propagate to backup copies. My backup software runs automatically every night, with the NAS only powered up for the duration of the backup. During the same period the NAS is powered up the usual operating system file backup software also runs on my other files.
I recently made a new and complete backup of my entire set of digital files, just about filling a 4 TB drive that was for offsite backup. That took over 24 hours, but the speed was good, as the external drive was connected via ESATA.
Tony Lauck
"Diversity is the law of nature; no two entities in this universe are uniform." - P.R. Sarkar
Now that I've decided to do manual backups and have established/verified all the directory trees involved I can use winmerge to find tree differences and update Albums accordingly manually using Explorer. What I need now is a way to check Songs, I'm thinking comparing filenames ignoring the extension - something I can write myself if I can't find it in Beyond Compare or WinMerge. Hopefully I can just use it as a plugin.
The hardest part of this whole ordeal is verifying what's on my shelves, ie. CDs and records is actually represented by digital libraries. And that in itself was no easy task even when I was just trying to keep a list of my records before I ever had a computer.
I also get about 10MB/sec sustained disk write speed on my USB drives, too.
While my library is only about a third of your size, my backups only take minutes because I do as Andy does and do only incremental changes. Which aren't that significant in any given week. I keep four additional copies to the main server.
Are you completely erasing and backing up the entire data set each and every time?
Yes these are complete copies to backup. I keep my library in alac and flac on internal drives and every now and then I delete backups and do clean copies to the external drives. Takes about 30 hours to copy both libraries to a single external drive and I've got two external drives. Just BTW - it takes about as long (15 hours) for DBPowerAmp to create an alac library from the flac library on the local drives, and thanks to "the" and "unknown" I explicity define an identical tree structure and file placement for both libraries relying on DbPowerAmp for conversion and file naming only. Kind of important, not so much when using WinAmp (flac) and iTunes (alac), but things like my OPPO actually rely on directory structure to provide a user interface. It was a real pisser seeing the Beatles alphabetized in "T" and how often meta-data led to the generation of unknown artist or unknown album directories.
I think most of the issue is the efficiency (or lack of same) of the algorithms of the backup software. The initial backup will be slow, but subsequent ones should be fast, as only a fairly small number of files need to be copied. I tried a number of backup solutions and finally settled on one that, for me at least, was way faster than the others. It is the freeware Microsoft RichCopy . The UI is a bit rough, but for me, the backup speed makes up for that.
With my system, an incremental backup of a 2TB internal SATA drive to an external USB drive using RichCopy takes about 15 minutes.
Post a Followup:
FAQ |
Post a Message! |
Forgot Password? |
|
||||||||||||||
|
This post is made possible by the generous support of people like you and our sponsors: