Getting Started with the S3 Component

Requirements: IPWorks or Cloud Storage

The easy-to-use S3 component is part of the IPWorks and Cloud Storage toolkits, and provides access to many different S3-compatible storage services in a consistent manner. Supported service providers include Amazon S3, Akamai (Linode) Object Storage, Alibaba Cloud Object Storage, Backblaze B2, Cloudflare R2, DigitalOcean Spaces, Google Cloud Storage, Huawei Cloud Object Storage, IBM Cloud Object Storage, Oort DSS, Oracle Cloud Object Storage, Seagate Lyve Cloud, Supabase Storage, Tencent Cloud Object Storage, and Wasabi. A custom provider option is also included, which allows the component to be used with any S3-compatible service (such as MinIO).

Contents

• First Steps
• Managing Buckets
• Working with Objects
• Performing Multipart Uploads
• Additional Features

First Steps

To begin, use the ServiceProvider property to select the S3 service provider. Then, use the AccessKey and SecretKey properties to specify the access key and secret key that the component should use to sign requests. These keys can be obtained after signing up for an account with the selected service provider.

S3 s3 = new S3(); s3.ServiceProvider = S3ServiceProviders.spAmazonS3; s3.AccessKey = "MY_ACCESS_KEY"; s3.SecretKey = "MY_SECRET_KEY";

Other authentication options are also available. For Amazon S3, temporary credentials may be supplied with the SessionToken configuration setting, EC2 instance role credentials may be used by enabling the UseEC2RoleCredentials configuration setting, and credentials may be loaded from an AWS profile with the AWSProfile configuration setting. For Google Cloud Storage, an OAuth token may be supplied with the OAuthAuthorization configuration setting.

Some providers require additional information to determine the endpoint to which requests are sent. Cloudflare R2 requires the CloudflareAccountID configuration setting, and Oracle Cloud Object Storage requires the OracleNamespace configuration setting. Supabase Storage requires either the SupabaseProjectId or the ServiceProviderURL configuration setting.

For the custom service provider, the ServiceProviderURL configuration setting must be set to the service's base URL before any requests can be made. If the service's URLs include a region string, a %region% token may be included in the ServiceProviderURL setting's value. The component automatically replaces this token with the current value of the Region property each time it sends a request to the server.

s3.ServiceProvider = S3ServiceProviders.spCustom; // The "%region%" part of the URL will be replaced with the current value of the Region property. s3.Region = "region_1"; s3.Config("ServiceProviderURL=https://s3.%region%.my-custom-provider.com");

At this point, the component is ready to use.

Managing Buckets

Calling the ListBuckets method will populate the Buckets collection with a list of all buckets in the account. To select a bucket for the component to operate on, set the Bucket property to its name.

s3.ListBuckets(); for (int i = 0; i < s3.Buckets.Count; i++) { Console.WriteLine(s3.Buckets[i].Name); Console.WriteLine(s3.Buckets[i].CreationDate); Console.WriteLine(s3.Buckets[i].OwnerDisplayName); } s3.Bucket = "my-favorite-bucket";

To create a new bucket or delete an existing one, set the Bucket property to the name of the bucket, and then call the CreateBucket or DeleteBucket method, respectively. Most service providers require a bucket to be empty before it can be deleted. For Wasabi, the ForceDeleteNonEmptyBuckets configuration setting (enabled by default) allows non-empty buckets to be deleted.

// Create a bucket. s3.Bucket = "my-new-bucket"; s3.CreateBucket(); // Delete a bucket. s3.Bucket = "my-old-bucket"; s3.DeleteBucket();

The S3 component also provides a handful of other methods for managing buckets. Refer to the documentation for more information.

Working with Objects

The ListObjects method will populate the Objects collection with a list of objects in the currently selected bucket. The ObjectPrefix and ObjectDelimiter properties may optionally be set prior to calling it to filter the results.

For buckets that contain many objects, there may be multiple pages of results. Each call to ListObjects will retrieve one page of results. If more pages are available, the ObjectMarker property will be non-empty after ListObjects returns. Continue calling ListObjects until the ObjectMarker property is empty to accumulate all pages of results in the Objects collection.

s3.Bucket = "my-bucket"; s3.ObjectPrefix = "photos/2026/"; // Continue calling until ObjectMarker is empty to accumulate all pages of results. do { s3.ListObjects(); } while (!string.IsNullOrEmpty(s3.ObjectMarker)); for (int i = 0; i < s3.Objects.Count; i++) { Console.WriteLine(s3.Objects[i].Name); Console.WriteLine(s3.Objects[i].LastModified); Console.WriteLine(s3.Objects[i].Size); }

New objects can be created and existing objects deleted using the CreateObject and DeleteObject methods, respectively. Existing objects can also be downloaded using the GetObject method.

// Upload an object. s3.LocalFile = "C:\\test\\important-stuff.zip"; s3.CreateObject("my-new-object"); // Delete an object. s3.DeleteObject("my-old-object"); // Download an object. s3.LocalFile = "C:\\test\\cat-pics.zip"; s3.GetObject("my-favorite-stuff");

The S3 component also provides numerous other methods for working with objects. Refer to the documentation for more information.

Performing Multipart Uploads

Multipart uploads, where data is uploaded in two or more parts and joined together later, are useful for uploading large objects. The CreateObject method automatically performs a multipart upload internally for objects above a certain size. However, in certain situations it can still be beneficial to manage the multipart upload manually (for example, to upload multiple parts in parallel).

Performing a multipart upload is a fairly simple process:

1. Call the StartMultipartUpload method to inform the server that a new multipart upload is starting. This method returns the upload ID generated by the server.
2. Call UploadPart for each part to be uploaded, passing it the upload ID and a part number each time. Part numbers determine the order in which the parts are joined when the multipart upload finishes, but the parts themselves may be uploaded in any order.
3. When all of the parts have been uploaded, call the CompleteMultipartUpload method to have the server join all of the parts together into the final object.

Each part must be at least 5 MB in size, except for the last part, which can be any non-zero size.

string uploadId = s3.StartMultipartUpload("my-new-big-object"); // Upload 5 parts filled with data from a placeholder "GetRandomBytes()" method. // Each part except the last must be at least 5 MB. // The loop starts at 1 since part numbers must be in the range 1-10000 (inclusive). for (int i = 1; i <= 5; i++) { s3.ObjectDataB = GetRandomBytes(); s3.UploadPart("my-new-big-object", i, uploadId); } s3.CompleteMultipartUpload("my-new-big-object", uploadId);

The following methods related to multipart uploads are also available:

• AbortMultipartUpload will abort a multipart upload, causing the server to discard any parts that have been uploaded for it.
• CopyPart can be used in place of UploadPart to create a part using data copied from an existing object on the server.
• ListMultipartUploads will retrieve a list of all pending multipart uploads in the currently selected bucket.
• ListParts will retrieve a list of all parts uploaded so far for a particular multipart upload.

Additional Features

In addition to the functionality described above, the S3 component also supports the following features:

• Client-side encryption
• Bucket versioning
• Object metadata
• Canned ACLs
• Pre-signed link generation
• And more!

Some S3 service providers do not support certain features. When in doubt, refer to the service provider's documentation to determine whether a feature is supported.