ViPR S3 Object Service API Support

Table of Contents

Back to Top

Amazon S3 API

This article describes ViPR support for the Amazon S3 API.

This article applies to EMC ViPR 2.0.

The following sections describe the support for the S3 API and describe how to authenticate with the service and how to use SDKs to develop clients to access the service:
Some aspects of bucket addressing and authentication are specific to ViPR. If you want to configure an existing application to talk to ViPR, or develop a new application that uses the S3 API to talk to ViPR, you should refer to the following article:
Back to Top

S3 API Supported and Unsupported Features

ViPR supports a subset of the Amazon S3 REST API.

The following sections detail the supported and unsupported APIs:

Supported S3 APIs

The following table lists the supported S3 API methods.

Unsupported S3 APIs

The following table lists the unsupported S3 API methods.

Back to Top

Authenticating with the S3 service

Authenticating with the Amazon S3 API is described in the Amazon S3 documentation referenced below. This topic identifies any ViPR-specific aspects of the authentication process.

Amazon S3 uses an authorization header that must be present in all requests to identify the user and provide a signature for the request. When calling Amazon the header has the following format:

Authorization: AWS <AWSAccessKeyId>:<Signature>

In ViPR, the AWSAccessKeyId maps to the ViPR user id (UID). An AWS access key ID has 20 characters (some S3 clients, such as the S3 Browser, check this), but ViPR data service does not have this limitation.

The signature is calculated from elements of the request and the user's Secret Key as detailed in the Amazon S3 documentation:

The following notes apply:

  • In the ViPR object data service, the UID can be configured (through the ViPR API or the ViPR UI) with 2 secret keys. The ViPR data service will try to use the first secret key, and if the calculated signature does not match, it will try to use the second secret key. If the second key fails, it will reject the request. When users add or change the secret key, they should wait 2 minutes so that all data service nodes can be refreshed with the new secret key before using the new secret key.
  • In the ViPR data service, namespace is also taken into HMAC signature calculation.
Back to Top

Using SDKs to access the S3 service

When developing applications that talk to the ViPR S3 service, there are a number of SDKs that will support your development activity.

The EMC Community provides information on the various clients that are available and provides guidance on their use:

The following topics describe the use of the Amazon S3 SDK and the use of the EMC ViPR Java SDK.

Note Image

If you want to make use of the ViPR API Extensions (see Use the ViPR Object Services), support for these extensions is provided in the EMC ViPR Java SDK. If you do not need support for the ViPR extensions, or you have existing applications that use it, you can use the Amazon Java SDK.

Back to Top

Using the Java Amazon SDK

You can access ViPR object storage using the Java S3 SDK.

By default the AmazonS3Client client object is coded to work directly against amazon.com. This section shows how to set up the AmazonS3Client to work against ViPR.

In order to create an instance of the AmazonS3Client object, you need to pass it credentials. This is achieved through creating an AWSCredentials object and passing it the AWS Access Key (your ViPR username) and your generated secret key for ViPR.

The following code snippet shows how to set this up.

AmazonS3Client client = new AmazonS3Client(new BasicAWSCredentials(uid, secret));

By default the Amazon client will attempt to contact Amazon WebServices. In order to override this behavior and contact ViPR you need to set a specific endpoint.

You can set the endpoint using the setEndpoint method. The protocol specified on the endpoint dictates whether the client should be directed at the HTTP port (9020) or the HTTPS port (9021).
Note Image

If you intend to use the HTTPS port, the JDK of your application must be set up to validate the ViPR certificate successfully; otherwise the client will throw SSL verification errors and fail to connect.

In the snippet below, the client is being used to access ViPR over HTTP:

AmazonS3Client client = new AmazonS3Client(new BasicAWSCredentials(uid, secret)); client.setEndpoint("http://vipr1.emc.com:9020");

When using path-style addressing ( virp1.emc.com/mybucket ), you will need to set the setPathStyleAccess option, as shown below:

S3ClientOptions options = new S3ClientOptions(); options.setPathStyleAccess(true); AmazonS3Client client = new AmazonS3Client(new BasicAWSCredentials(uid, secret)); client.setEndpoint("http://vipr1.emc.com:9020"); client.setS3ClientOptions(options);

The following code shows how to list objects in a bucket.

ObjectListing objects = client.listObjects("mybucket"); for (S3ObjectSummary summary : objects.getObjectSummaries()) { System.out.println(summary.getKey()+ " "+summary.getOwner()); }

The CreateBucket operation differs from other operations in that it expects a region to be specified. Against S3 this would indicate the datacenter in which the bucket should be created. However, ViPR does not support regions. For this reason, when calling the CreateBucket operation, we specify the standard region, which stops the AWS client from downloading the Amazon Region configuration file from Amazon CloudFront.

client.createBucket("mybucket", "Standard");

The complete example for communicating with the ViPR S3 data service, creating a bucket, and then manipulating an object is provided below:

public class Test { public static String uid = "root"; public static String secret = "KHBkaH0Xd7YKF43ZPFbWMBT9OP0vIcFAMkD/9dwj"; public static String viprDataNode = "http://vipr.yourco.com:9020"; public static String bucketName = "myBucket"; public static File objectFile = new File("/photos/cat1.jpg"); public static void main(String[] args) throws Exception { AmazonS3Client client = new AmazonS3Client(new BasicAWSCredentials(uid, secret)); S3ClientOptions options = new S3ClientOptions(); options.setPathStyleAccess(true); AmazonS3Client client = new AmazonS3Client(credentials); client.setEndpoint(viprDataNode); client.setS3ClientOptions(options); client.createBucket(bucketName, "Standard"); listObjects(client); client.putObject(bucketName, objectFile.getName(), objectFile); listObjects(client); client.copyObject(bucketName,objectFile.getName(),bucketName, "copy-" + objectFile.getName()); listObjects(client); } public static void listObjects(AmazonS3Client client) { ObjectListing objects = client.listObjects(bucketName); for (S3ObjectSummary summary : objects.getObjectSummaries()) { System.out.println(summary.getKey()+ " "+summary.getOwner()); } } }

Back to Top

Java SDK client for ViPR

The ViPR Java SDK builds on the Amazon S3 Java SDK and supports the VIPR API extensions.

An example of using the ViPRS3client is shown below.

package com.emc.vipr.sample; import com.amazonaws.util.StringInputStream; import com.emc.vipr.services.s3.ViPRS3Client; public class BucketCreate { private ViPRS3Client s3; public BucketCreate() { URI endpoint = new URI(“http://vipr.yourco.com:9020”); String accessKey = “fred@yourco.com”; String secretKey = “pcQQ20rDI2DHZOIWNkAug3wK4XJP9sQnZqbQJev3”; BasicAWSCredentials creds = new BasicAWSCredentials(accessKey, secretKey); ViPRS3Client client = new ViPRS3Client(endpoint, creds); } public static void main(String[] args) throws Exception { BucketCreate instance = new BucketCreate(); instance.runSample(); } public void runSample() { String bucketName="mybucket"; String key1 = "test1.txt"; String content = "Hello World!"; try { s3.createBucket(bucketName); s3.putObject(bucketName, key1, new StringInputStream(content), null); } catch (Exception e) { } } }