Resources
-
Article about data migration using fpart and rsync (GNU Linux Magazine #164 - October 2013, french) : Parallélisez vos transferts de fichiers
-
Steve French mentioned fpart and fpsync at 2019 Linux Storage, Filesystem, and Memory-Management Summit (LSFMM)
-
The partition problem and bin packing problem descriptions on Wikipedia
-
I am sure you will also be interested in Packo which was developed by Jean-Baptiste Denis as the original proof of concept. See also his newer tool, Msrsync
Projects using fpart
-
Harry Mangalam, from UCI, has an excellent article about data transfer here. Check out his parsyncfp and parsyncfp2 tools.
-
Dave Altschuler wrote dsync, a tool using fpart + rsync or rclone that can sync to the cloud
-
K-rsync uses fpart and the kubernetes Job scheduler to transfer files between PVCs
Fpsync users (Research / Education)
-
Bioteam used fpart + fpsync + rsync to migrate 2 PB of data
-
FAS RC (Harvard University) writes about fpsync to move data on Harvard's Odyssey cluster
-
Standford University's Sherlock HPC cluster offers fpart as a file management tool
-
Nantes University's BiRD cluster provides a fpart module
-
Sweden's NSC (National Supercomputer Centre)'s Centre Storage suggests using fpsync
-
National Energy Research Scientific Computing Center (NERSC) writes about fpsync
-
Utah's CHPC suggests fpart to transfer data in and out of CHPC resources
-
Fpart is proudly referenced in the French Government's 'SILL'
Fpsync users (Storage / Cloud providers)
-
Intel has written a white paper about data migration, presenting fpart and fpsync
-
Amazon uses fpart and fpsync in their EFS-to-EFS backup solution. See also their Amazon Elastic File System (Amazon EFS) for File Storage presentation (AWS Storage Days, New York, September 6-8, 2017) and the Amazon EFS performance tutorial, both presenting fpart and fpsync capabilities
-
Microsoft suggests using fpart and fpsync to speed-up file transfers
