Resumable Transfers in Cloud Storage

Large uploads and downloads can be interrupted for many reasons, such as network failures, application crashes, or a user cancelling the transfer. The components in Cloud Storage can resume interrupted transfers instead of starting over, and the information needed to resume a transfer can be saved so that it can be resumed by a different component instance, or even after the application restarts. This entry covers resuming uploads and downloads, tracking progress, cancelling transfers, and downloading part of a file.

Contents

• Resuming Uploads with CloudFiles
• Resuming Uploads with the Service-Specific Components
• Resuming Downloads
• Tracking Progress
• Cancelling a Transfer
• Downloading Part of a File

Resuming Uploads with CloudFiles

The CloudFiles component provides a single interface for resumable uploads across all supported service providers. To enable resumable uploads, set the UseResumableUpload property to true before calling Upload. The file is then uploaded in fragments (as a multipart or session-based upload, depending on the provider), so that an interrupted upload can be continued instead of restarted.

While the upload is in progress, the ResumableUploadState property holds an opaque value that identifies the upload. To be able to resume the upload after the application restarts, read this value during the upload (for example, in the Progress event) and save it. To resume, assign the saved value to ResumableUploadState and call Upload again with the same LocalFile and RemoteFile.

// First run: start the upload, saving the resumable upload state as soon as it is available. CloudFiles cloudfiles = new CloudFiles(); cloudfiles.ServiceProvider = CloudFilesServiceProviders.cspOneDrive; cloudfiles.Authorization = "Bearer ACCESS_TOKEN"; cloudfiles.OnProgress += (s, e) => { if (cloudfiles.ResumableUploadState.Length > 0) SaveUploadState(cloudfiles.ResumableUploadState); }; cloudfiles.UseResumableUpload = true; cloudfiles.LocalFile = "C:\\data\\large-file.zip"; cloudfiles.RemoteFile = "/Backups/large-file.zip"; cloudfiles.Upload();

// Later run (for example, after a restart): resume the upload using the saved state. CloudFiles cloudfiles = new CloudFiles(); cloudfiles.ServiceProvider = CloudFilesServiceProviders.cspOneDrive; cloudfiles.Authorization = "Bearer ACCESS_TOKEN"; cloudfiles.UseResumableUpload = true; cloudfiles.ResumableUploadState = LoadUploadState(); cloudfiles.LocalFile = "C:\\data\\large-file.zip"; cloudfiles.RemoteFile = "/Backups/large-file.zip"; // The upload continues from where it left off. cloudfiles.Upload();

In the example above, SaveUploadState and LoadUploadState are placeholders for application code that stores the value. Once an upload completes successfully, ResumableUploadState is cleared, so the saved value should be discarded.

For Amazon S3 and other S3-compatible providers, Azure Blob Storage, Azure Files, and Backblaze B2, the component can also find the partially uploaded data on the server automatically, so calling Upload again with the same LocalFile and RemoteFile resumes the upload even if ResumableUploadState was not saved. For Box, Dropbox, Google Drive, OneDrive, and SharePoint, the upload session cannot be found again from the server, so ResumableUploadState must be saved to resume the upload from a different instance.

Assigning an empty string to ResumableUploadState discards any pending upload, so the next call to Upload starts a new upload from the beginning. Note that the discarded upload session is not cancelled on the server, and expires after a provider-defined period of time.

Resuming Uploads with the Service-Specific Components

The service-specific components (such as OneDrive, Box, and AmazonS3) also support resumable uploads. When the same component instance is used, resuming an upload only requires calling the upload method again, since the component still holds the necessary information. To resume from a different instance, save the values listed in the table below while the upload is in progress, and restore them before calling the upload method again.

For Box, Dropbox, Google Drive, OneDrive, SharePoint, and ShareFile, the UseResumableUpload property must be set to true before the upload is started. For the S3-compatible components, AzureBlob, and BackblazeB2, uploads larger than the SimpleUploadLimit configuration setting automatically use a multipart (or block) upload that can be resumed. The AzureFile component always uploads files in ranges, so its uploads can always be resumed. Refer to the documentation for each component for additional details.

// Resume an interrupted OneDrive upload from a new component instance. OneDrive onedrive = new OneDrive(); onedrive.Authorization = "Bearer ACCESS_TOKEN"; onedrive.UseResumableUpload = true; onedrive.ResumeURL = LoadUploadState(); // Retrieve the current position of the upload from the server. onedrive.PollUploadStatus(); // Continue the upload from StartByte. onedrive.RemotePath = "/Backups"; onedrive.LocalFile = "C:\\data\\large-file.zip"; onedrive.UploadFile("large-file.zip");

When uploading from a stream with SetUploadStream, the stream must be positioned at the same place it was when the upload was first started (typically the beginning) before resuming. Upload sessions expire if they are not resumed within a provider-defined period of time. For Box, OneDrive, and SharePoint, an interrupted upload session can also be cancelled explicitly with the AbortUpload method.

Resuming Downloads

Downloads are resumed using the StartByte property. If a download to LocalFile is interrupted, set StartByte to the size of the partially downloaded file, and download the file again. The component then requests only the remaining data and appends it to the file. Since the partial file itself records how much has been downloaded, no other information needs to be saved to resume a download after a restart.

CloudFiles cloudfiles = new CloudFiles(); cloudfiles.ServiceProvider = CloudFilesServiceProviders.cspOneDrive; cloudfiles.Authorization = "Bearer ACCESS_TOKEN"; string downloadFile = "C:\\data\\large-file.zip"; cloudfiles.LocalFile = downloadFile; cloudfiles.RemoteFile = "/Backups/large-file.zip"; // If a previous download was interrupted, resume it from the end of the partial file. if (File.Exists(downloadFile)) cloudfiles.StartByte = new FileInfo(downloadFile).Length; cloudfiles.Download();

When downloading an encrypted file, the encrypted data is held in a temporary file until the download is complete, so resuming requires the path of that file as well. In that case, the DownloadTempFile configuration setting must be saved, and StartByte must be set to the size of the temporary file instead. Refer to the component documentation for more information.

Tracking Progress

The Progress event fires during uploads and downloads to report the progress of the transfer. Its Direction parameter indicates whether the transfer is an upload (0) or a download (1), and its BytesTransferred, TotalBytes, and PercentDone parameters report how much of the transfer is complete. TotalBytes and PercentDone are -1 if the total size of the transfer is unknown.

cloudfiles.OnProgress += (s, e) => { string direction = e.Direction == 0 ? "Uploading" : "Downloading"; Console.WriteLine(direction + ": " + e.PercentDone + "% (" + e.BytesTransferred + " of " + e.TotalBytes + " bytes)"); };

By default, the event fires each time PercentDone increases by 1 percent; the ProgressStep configuration setting can be used to change this. By default, progress is also tracked absolutely, meaning that when a transfer is resumed, the reported values include the data transferred before the interruption. To report progress for only the resumed portion of a transfer, disable the ProgressAbsolute configuration setting.

The service-specific components also fire the FragmentComplete event after each fragment of a resumable or multipart upload is uploaded, which reports the fragment number and the total number of fragments.

Cancelling a Transfer

To cancel an upload or download that is in progress, call the Interrupt method. It can be called from within an event handler (such as the Progress event) or from another thread. The interrupted method then fails with error 301 ("Interrupted."), which the application can catch.

bool cancelRequested = false; cloudfiles.OnProgress += (s, e) => { // Cancel the transfer if the user has requested it. if (cancelRequested) cloudfiles.Interrupt(); }; try { cloudfiles.Upload(); } catch (CloudStorageException ex) { if (ex.Code == 301) Console.WriteLine("The upload was cancelled."); else throw; }

An interrupted resumable upload can later be resumed as described above. In the service-specific components, setting the Interrupt parameter of the FragmentComplete event to true also interrupts the upload after the current fragment.

Downloading Part of a File

Several components can download only part of a file using the Range property, which is useful for reading the header of a large file or splitting a download across multiple connections. The Range property accepts a value in the format StartByte-EndByte (for example, 0-1023 for the first 1024 bytes), or StartByte- to download everything from the specified byte to the end of the file. Both values are zero-based and inclusive.

// Download 8 bytes from the middle of the file. onedrive.RemotePath = "/Backups/large-file.zip"; onedrive.LocalFile = "C:\\data\\partial.bin"; onedrive.Range = "2-9"; onedrive.DownloadFile(); // Clear the range so that later downloads retrieve the whole file. onedrive.Range = "";

The Range property is supported by the AmazonS3, S3, AzureBlob, BackblazeB2, GoogleStorage, OneDrive, SharePoint, and Wasabi components. If StartByte is also set when the download is resumed, the component automatically advances the start of the range by StartByte bytes, so previously downloaded data is not downloaded again.