Getting Started with S3 Component

The easy-to-use S3 component is part of the IPWorks and Cloud Storage toolkits, and allows you to access many different S3-compatible storage services in a consistent manner. Supported service providers include Amazon S3, Cloudflare R2, Digital Ocean Spaces, Google Cloud Storage, IBM Cloud Object Storage, Oracle Cloud Object Storage, Wasabi, and more! A custom provider option is also included, giving you the freedom and flexibility to use any S3-compatible service.

First Steps
Managing Buckets
Working with Objects
Performing Multipart Uploads
Additional Features

First Steps

To begin, use the ServiceProvider property to select the S3 service provider that you wish to use. Then, use the AccessKey and SecretKey properties to specify the access key and secret key that the component should use to sign requests. (These keys can be obtained after signing up for an account with the selected service provider.)


S3 s3 = new S3();
s3.ServiceProvider = S3ServiceProviders.spAmazonS3;
s3.AccessKey = "MY_ACCESS_KEY";
s3.SecretKey = "MY_SECRET_KEY";

Some providers, including Cloudflare R2 and Oracle Cloud Object Storage, require additional information to authenticate. Note that, for the custom service provider, you'll also need to use the URL configuration setting to specify a base URL; this must be done before any requests can be made. If your service's URLs include a region string, you can include a %region% token in the URL setting's value, and the component will automatically replace it with the current value of the Region property anytime it sends a request to the server.


s3.ServiceProvider = S3ServiceProviders.spCustom;
// The "%region%" part of the URL will be replaced with the current value
// of the Region property anytime the S3 component sends a request.
s3.Config("Region=region_1");
s3.Config("URL=https://s3.%region%.my-custom-provider.com");

At this point, you should be ready to start using the component!

Managing Buckets

Calling the ListBuckets method will populate the Buckets collection with a list of all buckets in your account. To select a bucket for the component to operate on, set the Bucket property to its name.


s3.ListBuckets();
for (int i = 0; i < s3.Buckets.Count; i++) {
  Console.WriteLine(s3.Buckets[i].Name);
  Console.WriteLine(s3.Buckets[i].CreationDate);
  Console.WriteLine(s3.Buckets[i].OwnerDisplayName);
}

s3.Bucket = "my-favorite-bucket";

If you need to create a new bucket, or delete an existing on, set the Bucket property to the name of the bucket, and then call CreateBucket or DeleteBucket. Keep in mind that buckets must be empty to be deleted!


s3.Bucket = "my-new-bucket";
s3.CreateBucket();

s3.Bucket = "my-old-bucket";
s3.DeleteBucket();

The S3 component also provides a handful of other methods for managing buckets; refer to the documentation for more information.

Working with Objects

The ListObjects method will populate the Objects collection with a list of objects in the currently-selected bucket. The ObjectPrefix and/or ObjectDelimiter properties may optionally be set prior to calling it to filter the results.

For buckets that contain many objects, there may be multiple pages of results. Each call to ListObjects will retrieve one page of results; if there are more pages available for retrieval, the ObjectMarker property will be non-empty after ListObjects returns. Continue calling ListObjects until the ObjectMarker property is empty to accumulate all pages of results in the Objects collection.


s3.Bucket = "my-bucket";
s3.ObjectPrefix = "photos/2016/";

do {
  // Continue calling until ObjectMarker is empty to accumulate all pages of results.
  s3.ListObjects();
} while (!string.IsNullOrEmpty(s3.ObjectMarker));

for (int i = 0; i < s3.Objects.Count; i++) {
  Console.WriteLine(s3.Objects[i].Name);
  Console.WriteLine(s3.Objects[i].LastModified);
  Console.WriteLine(s3.Objects[i].Size);
}

You can create new objects or delete existing objects using the CreateObject and DeleteObject methods. Existing objects can also be downloaded using the GetObject method.


// Upload an object
s3.LocalFile = "C:\test\important-stuff.zip";
s3.CreateObject("my-new-object");

// Delete an object
s3.DeleteObject("my-old-object");

// Download an object
s3.LocalFile = "C:\test\cat-pics.zip";
s3.GetObject("my-favorite-stuff");

The S3 component also provides numerous other methods for working with objects; refer to the documentation for more information.

Performing Multipart Uploads

Multipart uploads, where data is uploaded in two or more parts and joined together later, are useful for uploading large objects. The S3 component's CreateObject method will automatically perform a multipart upload internally for objects above a certain size; but in certain situations, it can still be beneficial to do it yourself (for example, if you wanted to upload multiple parts in parallel).

Performing a multipart upload is a fairly simple process:

Call StartMultipartUpload to inform the server that you're starting a new multipart upload. This method returns the upload Id generated by the server.
Call UploadPart for each part you need to upload, passing it the upload Id and a part number each time. Part numbers determine the order in which the parts are joined when the multipart upload finishes; you're free to upload the parts themselves in any order.
When all of the parts have been uploaded, call CompleteMultipartUpload to have the server join all of the parts together into the final object.


string uploadId = s3.StartMultipartUpload("my-new-big-object");

// This is just an example, so we'll upload 5 parts filled with data from a fake "GetRandomBytes()" method.
// Our loop starts at 1 since part numbers must be in the range 1-10000 (inclusive).
for (int i = 1; i <= 5; i++) {
  s3.ObjectDataB = GetRandomBytes();
  s3.UploadPart("my-new-big-object", i, uploadId);
}

s3.CompleteMultipartUpload("my-new-big-object", uploadId);

The following multipart-upload-related methods are also available:

AbortMultipartUpload will abort a multipart upload, causing the server to discard any parts that have been uploaded for it.
CopyPart can be used in place of UploadPart to create a part using data copied from an existing object on the server.
ListMultipartUploads will retrieve a list of all pending multipart uploads in the currently-selected bucket.
ListParts will retrieve a list of all parts uploaded so far for a particular multipart upload.

Additional Features

In addition to the functionality described above, the S3 component also supports the following features:

Client-Side Encryption
Bucket versioning
Object metadata
Canned ACLs
Pre-signed link generation
And more!

Keep in mind that some S3 service providers do not support certain features; when in doubt, refer to your service provider's documentation to determine whether a feature is supported.

We appreciate your feedback. If you have any questions, comments, or suggestions about this article please contact our support team at kb@nsoftware.com.