Multi-threaded file copy, RichCopy


Keith Combs convinced me to start blog (http://blogs.technet.com/keithcombs/). What for? I often receive e-mails asking about RichCopy, but it changed when Joshua Hoffman wrote an article about RichCopy in TechNet Magazine, April, 2009 issue. (http://technet.microsoft.com/en-us/magazine/2009.04.utilityspotlight.aspx) Keith covered this in his article, and consequently got tons of question on his blog (http://blogs.technet.com/keithcombs/archive/2009/03/22/richcopy-bulk-file-copy-tool-released-get-it-here.aspx). If I understand correctly, he screamed for help “I am not a support contact of RichCopy”, and asked me to start bloging on the TechNet.


What is RichCopy? Please visit Keith’s blog and 2009.4 issue of TechNet magazine. RichCopy was originally developed more than 10 years ago when I was working on Exchange Server 4.0 (a.k.a Touchdown project) and some other projects as a development lead. One of painful job was to copying latest build to my test machines over slow and large latency network. 10M bps was still majority of edge switches and NetBEUI as protocol at that time. RichCopy was developed to address those issues, especially to put another layer of packet transaction window. In my blogs, I will cover best practice of RichCopy configuration, QA, and I will also write about implementation for example how RichCopy supports over 64 threads while WaitForMultipleObjectEx() does not support more than 64 threads (MAXIMUM_WAIT_OBJECTS).


 

Comments (47)
  1. correnta1_1 says:

    Ken,

       I am using your RichCopy tool and have a few questions

    1)Can I use MS Premier support if I have a problem with the tool?

    2)We have data on the NAS and the NAS rounds down timestamp values to seconds

              a.Robocopy uses the millisecond level  

              timestamp and this causes files to always

              be copied, regardless of timestamp

              b.In Robocopy, if I use the /FFT switch  

                (assume FAT File Times, 2-second  

                granularity), we work around this problem

                  ii. Does something like this exist in

                      RichCopy?

  2. Anonymous says:

    I see many folks are looking for the newest version that fixes the ACL issue.  I would like to add  my voice to that choirs.  This an OK tool to me but will be awesome if I brings the security over as it is supposed to.  Anyone know if there is a fixed version and if so, where to get it?

  3. Anonymous says:

    Hello,

    I am trying to copy files from our production line pc’s.  I cannot copy a file from the current day, or risk shutting down our production lines.

    I tried to set the exclude filters but the settings do not stay checked.  I can set up what I want, hit apply, and if I come back to the screen everything is unchecked and my settings are gone.  

    What am I doing wrong.  Any help would be greatly appreciated.

    Thank You

  4. Anonymous says:

    Ken,

    First off – Excellent job!  

    Second – I can’t seem to get the ACL’s to copy, like others I’ve read about – I am using the latest version, all NTFS – any ideas?  Is there workaround for it?

    Thanks,

    Mark

  5. Anonymous says:

    any idea when the internal version where the ACL bug is fixed will be released? It would be rather cool to use RichCopy as opposed to Robocopy

  6. Anonymous says:

    Hi, I have been copying large number of files during a data migration an noticed tha target volume is heavily fragmented. I suppose this is to be expected with a multithreaded file copy, but is there any way to minimising it? – Ta

  7. Anonymous says:

    Using the latest v4.0.217.0 on VistaSP2. UAC disabled.

    I still don’t seem to be able to copy over the ACLs.

    I’ve tried copying always and disabling copy always. Copy if Security Setting is different.

    Changing options under Security by selecting and unselecting options.

    Disk to disk on the same PC.

    No success in any way. What am I doing wrong?

    Cheers

    Tropolite

    (PS Can you please at some stage fix the spelling error on the Overview page "Create directo’r’y Only when it has child")  🙂 Ta

  8. Anonymous says:

    I discover something new. If I copy the files for the first time, the ACL is not copied. However, if I repeat the copy, ACL will be copied.

  9. Anonymous says:

    has anybody successfully executed this via a batch file?  could you post an example?  i need to run this over a WAN link so any advise parameter wise would be appreciated.

    i need to run this as a scheduled task so i could run it on a regular basis.  unfortunately the timer only facilitates 1 event.

    will this work on a 64bit 2008 server?  i saw that the program directory included richCopy64.exe.  i wanted to try it out on a 32bit 2k3 server first, but would prefer to run it on their 2k8 server if possible.

    TIA ~Jim Rock

  10. Anonymous says:

    I’m currently looking at v4.0.217.0.  In help – copy option – method – consolidate multiple sources, I think the top graphic belongs with the bottom text and vice versa.

    I am trying to copy folders from a 2003R2 share via mapped drive M: on a 2008SP1 server. RichCopy is running on the 2008SP1 server.

    If I select a single folder as source e.g M:data and specify a drive as destination e.g E: the copy shows "copy start" and never completes or copies anything. If I select multiple folders from M: and destination E: I get error 87, The parameter is incorrect. If I create the destination folder first I can successfully copy single folders.  I cannot see at all how to correctly specify the destination for multiple folder copy (not consolidated). Love the speed though.

  11. Anonymous says:

    You say above that 216 fixed the permissions copy issue and I have even found 217 on a site and neither have fixed the issue.

    What is the trick to getting this to work?

  12. Anonymous says:

    KEN, CAN WE AT LEAST GET AN ACKNOWLEDGMENT THAT YOUR READING THIS BLOG????

  13. Anonymous says:

    Hi,

    Anyone know how to do a multiple source copy using the command line?  I don’t see anything in the help, and can’t find any info elsewhere.  It appears to work fine via the GUI by separating the source paths with a semicolon, however, this doesn’t work from the command line.

    For example, this doesn’t work:

    RichCopy.exe "I:Backups;J:Backups" "K:Backups"

    (params have been left off for brevity)

    The app just quits without any error or other logging.

    Any help much appreciated!!

  14. Anonymous says:

    The Exclude filter checkbox for both files and directories are not staying checked. I have checked them and then added a file extension to exclude "*.pdb" and as soon as I hit Apply and Ok, and then look at the Eclude settings again and the check boxes are empty and the file extension to exclude is grayed out. The command line also cleared out the exclusion, in the 'Other' settings screen. I see a few other people have complained about this, but I see no fix?? I have the latest version loaded. .217

    When you run the saved RichCopy it copies everything! I'd like to have a automated and repeatable process that my team can run, but if the excluded files are included, it can cause to many issues.

    Thanks,

    Waldo

    Build Master

  15. Anonymous says:

    Can anyone tell me how to copy the source folder.

    users$

    -name1

    –folder1

    –files1

    -name2

    –folder2

    –files2

    -name3

    –folder3

    –files3

    I would like to copy name1 and all the files and folder under it.

    When I try to copy name1 and files1 and folder1 under it to the destination location I only manage to copy the files1 and folders1 but not the source folder which shold be name1.

    Thx alot

  16. Anonymous says:

    Hello, I need help…  I think this App will do what I want, but I can’t get it to work.

    I have one Source folder and many Destination folders.

    Each Destination folder needs to contain the entire contents of the Source or just a few files. I only want to copy files from the Source to the Destination, if the file already exists in the Destination.

    Also, there are files in the Destination folders that Don’t exist in the Source, and they need to stay intact.

    Thanks in advance for your help.

    Steve

  17. Anonymous says:

    Ken,

    First off – Excellent job!  

    Second – I can’t seem to get the ACL’s to copy, like others I’ve read about – I am using the latest version, all NTFS – any ideas?  Is there workaround for it?

    Thanks,

    Mark

  18. Anonymous says:

    I can’t find RichCopy on Microsoft’s download site. The only version I find is linked to the technet article. I’d like an updated one that fixes the ACL/DACL issue. Any idea when it’ll be released? Can someone post a link to the updated RichCopy? The buggy version I currently have installed is 4.0.211

  19. Anonymous says:

    Hey Tamaru san, domo arigatou gozaimasu for this great tool!

    Sumimasen, but I am having trouble getting a command line option to do this simple thing: copy from a list of folders from my main server, to the same set of folders on my backup server. Changes only, with no purge. I will manually handle the problem when I reorganize source directories. For now I don’t mind some duplicate files.

    When I get this specified correctly, I will set up a system task to copy the updates every night.

    Can you (or anybody else) help me to create this command? I have tried using the Copy Options but somehow I messed up. I’m sure that it is my fault.

  20. Anonymous says:

    Same problem with ACL on Windows 2003 R2. Tried all suggestion above, no avail.

    Two other problems:

    1.The log says that it copies more files than the source, however, checking on the "Properties" of the root folder, the number is the same on both destination and source.

    2.Setting file copy thread to 2 sometimes would cause problem (Error code 32) on the destination file.

  21. Anonymous says:

    How can I use RichCOpy to monitor Files & Directory change?

    I need this to poprompt a file copy process once there is a certain change in a folder or file.

  22. Anonymous says:

    Noticed the Exclude filter checkboxes do get saved when a profile bag based profile is used (This is the Copy Option profile that can be created and then selected in the Option drop down box. The Exclude filter checkboxes are getting dropped when you do a Save As, and a filed based copy option profile is used (This is the profile that is saved in a .rcx file). This needs to be fixed because the .rcx copy profiles can be kept on a share for multiple people to use. Where the other type profile must be kept locally in the Registry? So it can't be shared to a team.

  23. Anonymous says:

    I am try to backup roaming profiles and I can’t do it using RichCopy. I currently use Xcopy which allows you to ignore permisions when copying.

    Can anyone help with this.

  24. Anonymous says:

    Hello Ken,

    great tool.

    But there is one problem by copy a bunch of files from a NTFS filesystem to a NAS filesystem like EXT3.

    Does richcopy considers the "2-seconds-granularity" btw the files on different filesystem like NTFS and EXT3? Richcopy means during an update copy !all! files from the source are newer than the files within the destination.

  25. Anonymous says:

    I just recently discovered your utility and it looks like itwill make my life a lot easier, however I am running into a problem.

    I’m trying to use RichCopy to migrate about 400GB of data made up of small files (mostly pdf and tiff) from a 2003 sp2 machne to a network share on a NAS box.

    I keep getting the following error after the copy runs for awhile:

    Event Type: Error

    Event Source: Application Error

    Event Category: (100)

    Event ID: 1000

    Date: 7/30/2009

    Time: 12:43:05 PM

    User: N/A

    Computer: CRBCP-NAS2

    Description:

    Faulting application RichCopy.exe, version 4.0.217.0, faulting module comctl32.dll, version 6.0.3790.3959, fault address 0x0005d84b.

    Am I missing something stupid?

    Thanks,

    Bill

  26. Anonymous says:

    GGonzalez, any follow up on your problem?

    We’ve successfully used RICHCOPY to copy from multiple sources to a single destination using the method you described.  Here is a command line that worked:

    c:richcopy64 "\CO01E$WORKGROUPFINSERVShared;\CO01E$WORKGROUPBUDGETShared"  "D:A720-OBFRCOMMON" /P /C /PRP /TSD /CLA /CLW /FSD /QA /QP "d:richcopy.log" /UE /US /UD /UPF /UPC /UPS /UFC /USC /USD /UPR /UET

    However, we’ve run into a situation where it will not work with the purge /p option from the command line.  Notice the /p is present is my example and it worked, however we have another case where removing the /p makes it work but RICHCOPY bombs with the /p present.

    Thanks,

  27. Anonymous says:

    I’m using the logging option via the command line /QA and /QP "my log path.log".  The log is never appended to or re-created using the /QO option.

    If I run the gui it logs correctly to the file.

    Is this is issue with 4.0.217.0?

    Thanks!

  28. Anonymous says:

    Hi, I just stumbled upon your application and I love it!  However, I might have discovered a (somewhat) major bug in the program.

    I was trying to mirror the contents from C:ABC (source) to H:XYZ (destination), therefore I selected the "purge" option.  I deleted some files off of C:ABC before running the app, and when I ran it, in the status window I got messages saying "purging: C:ABC"!!

    I was horrified as I thought the program was deleting files off my source folder, however when I checked the folders after the process had finished, I realised the app functioned as intended, deleting files off of H:XYZ.

    Even though the function worked, I still think the message in the status window needs to be fixed, as it has to relay accurate messages or else it’d give its users a big scare!

  29. Anonymous says:

    Hi,

    is it possible to do incremental backups with RichCopy? If yes, please let me know how.

    Many thanks.

    Beto.

  30. Ken Tamaru says:

    Sorry for being slow update..

    I have fixed ACL related issues and posted new build 4.0.216 onto download center. URL is same as before http://download.microsoft.com/download/f/d/0/fd05def7-68a1-4f71-8546-25c359cc0842/HoffmanUtilitySpotlight2009_04.exe

    Here is list of bugs fixed in this build.

    1. Cannot copy security information when /CSA /CSD /CSG /CSO /CSS options are specified or equivalent options are set in option dialog.

    2. Logging information is truncated when RichCopy is executed from command line.

    3. Destination path got corrupted when RichCopy is executed from command line. (rarely happen..)

    4. There is no description of file and directory filtering command parameters in helpfile.

  31. Anonymous says:

    This is a great utility but for one small problem… the timer doesn’t work which is a pain ’cause I need to move 55gb from one server to another and it can only start after 7.00pm 🙁   Any fix for this? thanks Tracy.

  32. Anonymous says:

    I cannot get it to copy from a ftp site.  When I choose source path it will find my ftp site and login and even show me all the directories.  I choose the one I want and click ok.  When I go to run it though, it says error 3 – cannot find the path specified.  I’ve been trying off and on for 3 days to get this to work.  No luck.  I do have my username and password in the options section.  One thing I noticed is that the help says to put ftp://<site.com/path.  When I click ok it shortens it to //<site.com/path.  I have put ftp in front of the path also and it still does not work.  I am able to ftp to my site using ftp command line and filezilla.  Both work fine.  Just not richcopy.  No firewall is enabled.  Any suggestions?

  33. Anonymous says:

    Hi Ken,

    Any luck with fixing the wildcards when executed from a batch file?  Thanks for the great tool.

  34. Anonymous says:

    I have got the timer to work, but there is a bug in it – it saves the time in GMT, whereas in the UK we are now on BST. Therefore to make it work I had to add one hour to the intended time.

    Like other posters, I have found the include and exclude option flags do not work.

  35. Anonymous says:

    For info, I had to disable User Account Control (UAC) for the ACL ‘s to copy. I hope this can help.

    Thanks to Ken for a very nice tool.

    Fongen

  36. Anonymous says:

    Hey there,

    if you are still reading here …

    Great tool, you know that by now.

    A thing that gives me a headache is the help file included in the package.

    Could you convince somebody at M$ to proofread it and make it a bit more understandable?

    This tool really deserves a good documentation.

    Thanks

  37. Bridget says:

    Is there a way to minimize the Splash screen?  The machine this I'm running this on has multiple users that are using it and I'm trying to get this to run in the background executing a shell command from a vb.net application.  Any help would be greatly appreciated!

  38. Frank W. Vans Evers says:

    I am a new user. Is there an examples document, manual? Like I want to copy the folders on my V: drive to D:c-11242010… all folders and files on "V:" to the target, keep dates and whatever the same. what are my desired options? Thanks in advance /s/ Frank

  39. Hopeful says:

    Latest version 4.0.217.0 x64,x86 does not seem to work from command line on Windows 7 x64.  I see a splash screen but no update to the log file.  I have tried two methods

    "pathtoRichCopy.exe"  sourcedir destdir params

    and

    "pathtoRichCopy.exe" pathtoOptionsFile.rcx

    Files are not copied, log files are not updated, not output error messages

    I few people have claimed to have gotten it working (or at least a previous version working) but frankly there are literally hundreds of posts all over the net asking the same question:  does Richcopy work from the command line?  It would be really nice to hear a definitive answer to this.  Frankly, the consensus is that it does not work at the command line but may have once before.  

    Thank-you.

  40. Hopeful Aha says:

    Okay, RichCopy does work from the command line.  It opens RichCopy in GUI mode and runs the command line parameters.  Then it closes the GUI.

    Try something like this in a DOS cmd file (example: doit.cmd)

    set RC="c:program files (x86)Microsoft Rich ToolsRichCopy 4.0RichCopy64.exe"  

    set parms="D:usersadministratorDocumentsGroovefiles"  "N:OneNoteShareGroovefiles" /P /QO /QP "D:usersadministratorDocumentsRichCopyLogsgroovefiles.log" /UE /US /UC /UPF /UPC /UFC /USC

    %RC% %parms% > NUL 2>1&

    I should qualify that RichCopy does work from the command line at least when it wants too.  Either that or its messing with me

    The parameters come from the GUI copy options | others | command line parameters box.

    Good luck.

  41. WinTakeAll says:

    The bug that Europa2010AD mentions also shows in the log files:

    The log entries for purged files incorrectly show them as if RichCopy purged them from the source directory. I agree that looks scary, please fix for next release.

  42. vl says:

    I agree with that. At home I use Roadkil's Unstoppable Copier which is an other copy manager that copies, moves and recovers files from damaged disks.

    bstdownload.com/…/roadkils-unstoppable-copier-5

  43. RCampbell says:

    I tried richcopy using the GUI interface, selected the source folder of d:mainsubfoldersubfolder and the destination of e:targetsubfoldersubfolder and noticed that upon completion that inside the second subfolder was the name of the first sub folder of the d: rather than all the folder in the second folder.  What switch should I have added to avoid this? Sorry is I am so unfamiliar with rich copy.  I almost prefer robocopy as it works more like xcopy and does exactly what I expect it to do.

  44. JRS says:

    I have been trying to find a list of error codes and definitions as they relate to the entries in the RichCopy log.  I have not been able to find anything that explains the error codes.

  45. Ponvel Shanmuganathan says:

    I am using Richopy exe to transfer files from local share to the central share in a Data center. Issue here is i l see lots of files are not playable. When i try to play the same in Windows Media player it show me a message saying that "Cannot play the file".

    Centeral Server might be in hung state during the process. i am not sure this might me the problem because all problem wav files are not 0 KB files. Files are in different file size and not playing at all.

    Any Suggestion to recover the file is highly appreciated.

    Thanks

  46. tvineet says:

    Hi,

    anyone can help me how to create batch file for copy the files from richcopy

Comments are closed.

Skip to main content