ECS 2.0 – ECS Amazon S3 Object Service API support

Table of Contents

Amazon S3 API

This article describes ECS support for the Amazon S3 API.

The Amazon S3 Object Service is made available on the following ports.
The following sections describe the support for the S3 API, the extension provided by ECS, and describe how to authenticate with the service and how to use SDKs to develop clients to access the service:
Some aspects of bucket addressing and authentication are specific to ECS. If you want to configure an existing application to talk to ECS, or develop a new application that uses the S3 API to talk to ECS, you should refer to the following article:
Back to Top

S3 API Supported and Unsupported Features

ECS supports a subset of the Amazon S3 REST API.

The following sections detail the supported and unsupported APIs:

Supported S3 APIs

The following table lists the supported S3 API methods.

Unsupported S3 APIs

The following table lists the unsupported S3 API methods.

Back to Top

API Extensions

A number of extensions to the object APIs are supported.

The extensions and the APIs that support them are listed in the following table.

Back to Top

Updating a byte range within an object

An example of using the ECS API extensions to update a byte range of an object is provided below.

First do a GET request on the object named object1 located in bucket1 to review the object. object1 has the value The quick brown fox jumps over the lazy dog.

GET /bucket1/object1 HTTP/1.1
Date: Mon, 17 Jun 2013 20:04:40 -0000
x-emc-namespace: emc
Content-Type: application/octet-stream
Authorization: AWS wuser1:9qxKiHt2H7upUDPF86dvGp8VdvI=
Accept-Encoding: gzip, deflate, compress
 
HTTP/1.1 200 OK
Date: Mon, 17 Jun 2013 20:04:40 GMT
Content-Type: application/octet-stream
Last-Modified: Mon, 17 Jun 2013 20:04:28 GMT
ETag: 6
Content-Type: application/json
Content-Length: 43
 
The quick brown fox jumps over the lazy dog.

Now you want to update a specific byte range within this object. To do this, the Range header in the object data request must include the start and end offsets of the object that you want to update. 
The format is: Range: bytes=<startOffset>-<endOffset>

In the example below, the PUT request includes the Range header with the value bytes=10-14 indicating that bytes 10,11,12,13,14 are to be replaced by the value sent in the request. Here, the new value green is being sent.

PUT /bucket1/object1 HTTP/1.1
Content-Length: 5
Range: bytes=10-14
ACCEPT: application/json,application/xml,text/html,application/octet-stream
Date: Mon, 17 Jun 2013 20:15:16 -0000
x-emc-namespace: emc
Content-Type: application/octet-stream
Authorization: AWS wuser1:xHJcAYAEQansKLaF+/4PdLBHyaM=
Accept-Encoding: gzip, deflate, compress
 
green
 
HTTP/1.1 204 No Content
ETag: 10
x-amz-id-2: object1
x-amz-request-id: 027f037c-29ea-4670-8670-de82d0e9f52a
Content-Length: 0
Date: Mon, 17 Jun 2013 20:15:16 GMT

When reading the object again, the new value is now The quick green fox jumps over the lazy dog. (The word brown has been replaced with green.) You have updated a specific byte range within this object.

GET /bucket1/object1 HTTP/1.1
Cookie: JSESSIONID=wdit99359t8rnvipinz4tbtu
ACCEPT: application/json,application/xml,text/html,application/octet-stream
Date: Mon, 17 Jun 2013 20:16:00 -0000
x-emc-namespace: emc
Content-Type: application/octet-stream
Authorization: AWS wuser1:OGVN4z8NV5vnSAilQTdpv/fcQzU=
Accept-Encoding: gzip, deflate, compress
 
HTTP/1.1 200 OK
Date: Mon, 17 Jun 2013 20:16:00 GMT
Content-Type: application/octet-stream
Last-Modified: Mon, 17 Jun 2013 20:15:16 GMT
ETag: 10
Content-Type: application/json
Content-Length: 43
 
The quick green fox jumps over the lazy dog.
Back to Top

Overwriting part of an object

An example of using the ECS API extensions to overwrite part of an object is provided below.

You can overwrite part of an object by providing only the starting offset in the data request. The data in the request will be written starting at the provided offset. The format is: Range: <startingOffset>-

For example, to write the data brown cat starting at offset 10, you would issue this PUT request:

PUT /bucket1/object1 HTTP/1.1
Content-Length: 9
Range: bytes=10-
ACCEPT: application/json,application/xml,text/html,application/octet-stream
Date: Mon, 17 Jun 2013 20:51:41 -0000
x-emc-namespace: emc
Content-Type: application/octet-stream
Authorization: AWS wuser1:uwPjDAgmazCP5lu77Zvbo+CiT4Q=
Accept-Encoding: gzip, deflate, compress
 
brown cat
 
HTTP/1.1 204 No Content
ETag: 25
x-amz-id-2: object1
x-amz-request-id: 65be45c2-0ee8-448a-a5a0-fff82573aa3b
Content-Length: 0
Date: Mon, 17 Jun 2013 20:51:41 GMT

When retrieving the object, you can see the final value The quick brown cat jumps over the lazy dog and cat. (green fox has been replaced with brown cat). You have overwritten part of the data in this object at the provided starting offset.

GET /bucket1/object1 HTTP/1.1
Date: Mon, 17 Jun 2013 20:51:55 -0000
x-emc-namespace: emc
Content-Type: application/octet-stream
Authorization: AWS wuser1:/UQpdxNqZtyDkzGbK169GzhZmt4=
Accept-Encoding: gzip, deflate, compress
 
HTTP/1.1 200 OK
Date: Mon, 17 Jun 2013 20:51:55 GMT
Content-Type: application/octet-stream
Last-Modified: Mon, 17 Jun 2013 20:51:41 GMT
ETag: 25
Content-Type: application/json
Content-Length: 51
 
The quick brown cat jumps over the lazy dog and cat.
Back to Top

Appending data to an object

An example of using the ECS API extensions to append data to an object is provided below.

There may be cases where you need to append to an object, but determining the exact byte offset is not efficient or useful. For this scenario, ECS provides the ability to atomically append data to the object without specifying an offset (the correct offset is returned to you in the response).

A Range header with the special value bytes=-1- can be used to append data to an object. In this way, the object can be extended without knowing the existing object size.

The format is: Range: bytes=-1-

A sample request showing appending to an existing object using a Range value of bytes=-1-. Here the value and cat is sent in the request.

PUT /bucket1/object1 HTTP/1.1
Content-Length: 8
Range: bytes=-1-
ACCEPT: application/json,application/xml,text/html,application/octet-stream
Date: Mon, 17 Jun 2013 20:46:01 -0000
x-emc-namespace: emc
Content-Type: application/octet-stream
Authorization: AWS wuser1:/sqOFL65riEBSWLg6t8hL0DFW4c=
Accept-Encoding: gzip, deflate, compress
 
and cat
 
HTTP/1.1 204 No Content
ETag: 24
x-amz-id-2: object1
x-amz-request-id: 087ac237-6ff5-43e3-b587-0c8fe5c08732
Content-Length: 0
Date: Mon, 17 Jun 2013 20:46:01 GMT

When retrieving the object again, you can see the full value The quick green fox jumps over the lazy dog and cat. You have appended data to this object.

GET /bucket1/object1 HTTP/1.1
ACCEPT: application/json,application/xml,text/html,application/octet-stream
Date: Mon, 17 Jun 2013 20:46:56 -0000
x-emc-namespace: emc
Content-Type: application/octet-stream
Authorization: AWS wuser1:D8FSE8JoLl0MTQcFmd4nG1gMDTg=
Accept-Encoding: gzip, deflate, compress
 
HTTP/1.1 200 OK
Date: Mon, 17 Jun 2013 20:46:56 GMT
Content-Type: application/octet-stream
Last-Modified: Mon, 17 Jun 2013 20:46:01 GMT
ETag: 24
Content-Type: application/json
Content-Length: 51
 
The quick green fox jumps over the lazy dog and cat.
Back to Top

Reading multiple byte ranges within an object

An example of using the ECS API extensions to read multiple byte ranges within an object is provided below.

To read two specific byte ranges within the object named object1, you would issue the following GET request for Range: bytes==4-8,41-44. The read response would be for the words quick and lazy.
Note Image
The Amazon S3 API only supports one range when using the HTTP header Range for reading; ECS supports multiple byte ranges.

GET /bucket1/object1 HTTP/1.1
Date: Mon, 17 Jun 2013 20:51:55 -0000
x-emc-namespace: emc
Range: bytes==4-8,41-44
Content-Type: application/octet-stream
Authorization: AWS wuser1:/UQpdxNqZtyDkzGbK169GzhZmt4=
Accept-Encoding: gzip, deflate, compress
 
HTTP/1.1 206 Partial Content
Date: Mon, 17 Jun 2013 20:51:55 GMT
Content-Type: multipart/byteranges;boundary=bound04acf7f0ae3ccc
Last-Modified: Mon, 17 Jun 2013 20:51:41 GMT
Content-Length: 230
 
--bound04acf7f0ae3ccc
Content-Type: application/octet-stream
Content-Range: bytes 4-8/50
quick
--bound04acf7f0ae3ccc
Content-Type: application/octet-stream
Content-Range: bytes 41-44/50
lazy
--bound04acf7f0ae3ccc--
Back to Top

Authenticating with the S3 service

Authenticating with the Amazon S3 API is described in the Amazon S3 documentation referenced below. This topic identifies any ECS-specific aspects of the authentication process.

Amazon S3 uses an authorization header that must be present in all requests to identify the user and provide a signature for the request. When calling Amazon the header has the following format:
Authorization: AWS <AWSAccessKeyId>:<Signature>

In ECS, the AWSAccessKeyId maps to the ECS user id (UID). An AWS access key ID has 20 characters (some S3 clients, such as the S3 Browser, check this), but ECS data service does not have this limitation.

The signature is calculated from elements of the request and the user's Secret Key as detailed in the Amazon S3 documentation:

The following notes apply:

  • In the ECS object data service, the UID can be configured (through the ECS API or the ECS UI) with 2 secret keys. The ECS data service will try to use the first secret key, and if the calculated signature does not match, it will try to use the second secret key. If the second key fails, it will reject the request. When users add or change the secret key, they should wait 2 minutes so that all data service nodes can be refreshed with the new secret key before using the new secret key.
  • In the ECS data service, namespace is also taken into HMAC signature calculation.
Back to Top

Using SDKs to access the S3 service

When developing applications that talk to the ECS S3 service, there are a number of SDKs that will support your development activity.

The EMC Community provides information on the various clients that are available and provides guidance on their use: EMC ViPR Developer Resources.

The following topics describe the use of the Amazon S3 SDK and the use of the EMC ECS Java SDK.

Note Image

If you want to make use of the ECS API Extensions (see ), support for these extensions is provided in the EMC ECS Java SDK. If you do not need support for the ECS extensions, or you have existing applications that use it, you can use the Amazon Java SDK.


Back to Top

Using the Java Amazon SDK

You can access ECS object storage using the Java S3 SDK.

By default the AmazonS3Client client object is coded to work directly against amazon.com. This section shows how to set up the AmazonS3Client to work against ECS.

In order to create an instance of the AmazonS3Client object, you need to pass it credentials. This is achieved through creating an AWSCredentials object and passing it the AWS Access Key (your ECS username) and your generated secret key for ECS.

The following code snippet shows how to set this up.

AmazonS3Client client = new AmazonS3Client(new BasicAWSCredentials(uid, secret));

By default the Amazon client will attempt to contact Amazon WebServices. In order to override this behavior and contact ECS you need to set a specific endpoint.

You can set the endpoint using the setEndpoint method. The protocol specified on the endpoint dictates whether the client should be directed at the HTTP port (9020) or the HTTPS port (9021).
Note Image

If you intend to use the HTTPS port, the JDK of your application must be set up to validate the ECS certificate successfully; otherwise the client will throw SSL verification errors and fail to connect.


In the snippet below, the client is being used to access ECS over HTTP:

AmazonS3Client client = new AmazonS3Client(new BasicAWSCredentials(uid, secret));
client.setEndpoint("http://ecs1.emc.com:9020");

When using path-style addressing ( ecs1.emc.com/mybucket ), you will need to set the setPathStyleAccess option, as shown below:

S3ClientOptions options = new S3ClientOptions();
options.setPathStyleAccess(true);

AmazonS3Client client = new AmazonS3Client(new BasicAWSCredentials(uid, secret));
client.setEndpoint("http://ecs1.emc.com:9020");
client.setS3ClientOptions(options);
The following code shows how to list objects in a bucket.
ObjectListing objects = client.listObjects("mybucket");
for (S3ObjectSummary summary : objects.getObjectSummaries()) {
    System.out.println(summary.getKey()+ "   "+summary.getOwner());
}
The CreateBucket operation differs from other operations in that it expects a region to be specified. Against S3 this would indicate the datacenter in which the bucket should be created. However, ECS does not support regions. For this reason, when calling the CreateBucket operation, we specify the standard region, which stops the AWS client from downloading the Amazon Region configuration file from Amazon CloudFront.
client.createBucket("mybucket", "Standard");

The complete example for communicating with the ECS S3 data service, creating a bucket, and then manipulating an object is provided below:

public class Test {
    public static String uid = "root";
    public static String secret = "KHBkaH0Xd7YKF43ZPFbWMBT9OP0vIcFAMkD/9dwj";
    public static String viprDataNode = "http://ecs.yourco.com:9020";

    public static String bucketName = "myBucket";
    public static File objectFile = new File("/photos/cat1.jpg");

    public static void main(String[] args) throws Exception {

        AmazonS3Client client = new AmazonS3Client(new BasicAWSCredentials(uid, secret));

        S3ClientOptions options = new S3ClientOptions();
        options.setPathStyleAccess(true);

        AmazonS3Client client = new AmazonS3Client(credentials);
        client.setEndpoint(viprDataNode);
        client.setS3ClientOptions(options);

        client.createBucket(bucketName, "Standard");
        listObjects(client);

        client.putObject(bucketName, objectFile.getName(), objectFile);
        listObjects(client);

        client.copyObject(bucketName,objectFile.getName(),bucketName, "copy-" + objectFile.getName());
        listObjects(client);
    }

    public static void listObjects(AmazonS3Client client) {
        ObjectListing objects = client.listObjects(bucketName);
        for (S3ObjectSummary summary : objects.getObjectSummaries()) {
            System.out.println(summary.getKey()+ "   "+summary.getOwner());
        }
    }
}
Back to Top

Java SDK client for ECS

The ECS Java SDK builds on the Amazon S3 Java SDK and supports the ECS API extensions.

An example of using the ViPRS3client is shown below.
package com.emc.ecs.sample;

import com.amazonaws.util.StringInputStream;
import com.emc.vipr.services.s3.ViPRS3Client;

public class BucketCreate {

	private ViPRS3Client s3;
	


 public BucketCreate() {

    URI endpoint = new URI(“http://ecs.yourco.com:9020”);
    String accessKey = “fred@yourco.com”;
    String secretKey = “pcQQ20rDI2DHZOIWNkAug3wK4XJP9sQnZqbQJev3”;
    BasicAWSCredentials creds = new BasicAWSCredentials(accessKey, secretKey);
    ViPRS3Client client = new ViPRS3Client(endpoint, creds);

  }

    public static void main(String[] args) throws Exception {
            BucketCreate instance = new BucketCreate();
            instance.runSample();
    }
    
    public void runSample() {
    	
    	String bucketName="mybucket";
    	String key1 = "test1.txt";
        String content = "Hello World!";
        
    	try {
    		s3.createBucket(bucketName);
    	    s3.putObject(bucketName, key1, new StringInputStream(content), null);
    	}
    	
    	catch (Exception e) {
    		
    	}
    	
    }
}
Back to Top