ECS 2.0 – Create and configure buckets

Table of Contents

Introduction

Containers are required to store object data. In S3 these containers are called buckets and this term has been adopted as a general term in ECS. In Atmos, the equivalent of a bucket is a subtenant, in Swift, the equivalent of a bucket is a container, and for CAS, a bucket is a CAS pool.

In ECS, buckets are assigned a type which can be S3, Swift, Atmos, or CAS. In addition, S3, Atmos, or Swift buckets can be configured to support HDFS, and a bucket configured for HDFS access can be read and written using its object protocol and using the HDFS protocol.

Buckets can be created for each object protocol using its API, usually using a client that supports the appropriate protocol. Additional support for creating S3, HDFS, and CAS buckets is provided by the ECS portal and the ECS Management API. The ability to create buckets from the portal makes it easy to create buckets for HDFS and CAS and makes it easy to take advantage of some of the more advanced bucket configuration options provided by ECS, such as quotas and retention periods.

Where you want to create buckets using the object protocols, you can use special x-emc headers to control bucket configuration.

This article describes how to create and edit buckets, and set ACLs for a bucket, using the ECS portal and also describes the additional x-emc headers that you can be used to control bucket configuration when using the supported object protocols.

Back to Top

Bucket concepts and attributes

Buckets are object containers and can be used to control access to objects and to set properties that define attributes for all contained objects, such as retention periods and quotas.

Bucket access

Buckets are associated with a replication group. Where the replication group spans multiple VDCs, the bucket contents are similarly replicated across the VDCs. Objects in a bucket that belongs to a replication group which spans two VDCs, VDC1 and VDC2, for example, can be accessed from either VDC1 or VDC2. Objects in a bucket that belongs to a replication group that is only associated with VDC1, can only be accessed from VDC1, they cannot be accessed from other VDCs in a federated ECS system.

The identity of a bucket and its metadata, such as its ACL, are global management information in ECS, which means that they are replicated across the system storage pools and can be seen from all VDCs in the federation. However, the bucket can only be listed from a VDC that is part of the replication group to which the bucket belongs.

Bucket ownership

A bucket belongs to a namespace and object users are also assigned to a namespace. Each object user can create buckets only in the namespace to which they belong, however, any ECS object user can be assigned as the owner of a bucket or object, or a grantee in a bucket ACL, even if the user does not belong to the same namespace as the bucket or object. This enables buckets and objects to be shared between users in different namespaces. For example, in an enterprise where a namespace is a department, a bucket or object can be shared between users in different departments.

When an object user wants to access a bucket in a namespace that they don't belong to, the namespace must be specified using the x-emc-namespace header.

Access to a bucket during temporary site outage

ECS provides a temporary site outage mechanism that enables objects to be retrieved even if the primary copy of the object is not available due to the site that hosts the primary being unavailable.

Because there is a risk that object data retrieved during a temporary site outage is not the most recent, the user must indicate that they are prepared to accept this by marking the bucket as available during an outage.

Bucket attributes

The ECS Portal enables buckets to be created and managed at the Manage > Buckets page.

The Bucket Management page provides a bucket table which displays the buckets for a selected namespace. The table displays bucket attributes and provides Edit Bucket, Edit ACL, and Delete actions for each bucket.

The attributes associated with a bucket are described in the following table. To view and change attributes that are not displayed on the Bucket Management page, you can select Edit Bucket.

Back to Top

Bucket ACLs

The privileges a user has when accessing a bucket are set using an ACLs.

When you create a bucket and assign an owner to it, an ACL is created that assigns a default set of permissions to the bucket owner - the owner is, by default, assigned full control.

You can modify the permissions assigned to the owner or you can add new permissions for a user by selecting the Edit ACL operation for the bucket.

At the ECS Portal, the Bucket ACLs Management page provides User ACLs and Group ACLs panels to manage the ACLs associated with individual users and with pre-defined groups.

User ACLs

The User ACL panel show the ACLs that have been applied to users and enables ACLs to be assigned to a user using the Add operation.

The ACL meanings are provided in the following table.

Note Image

Because the ECS Portal supports S3, HDFS, and CAS buckets, the range of permissions that can be set are not applicable to all bucket types.


Group ACLs

You can set permissions for a set of pre-defined groups. The following groups are supported:
public
All users authenticated or not.
all users
All authenticated users.
other
Authenticated users but not the bucket owner.
log delivery
Not supported.

The permissions that can be assigned are listed in Bucket ACLs.

Back to Top

Create a bucket using the ECS Portal

The ECS portal enables the creation of buckets and provides the ability to specify the configuration of the bucket. Buckets created at the portal can be either S3, S3+HDFS, or CAS buckets.

Before you begin

  • You must be a Namespace Admin or a System Admin to create a bucket at the ECS portal.
  • If you are a Namespace Admin you can create buckets in your namespace.
  • If you are System Admin you can create a bucket belonging to any namespace.

Procedure

  1. At the ECS portal, select Manage > Buckets.
  2. Select New Bucket.
  3. Select the replication group that the bucket associated with.
  4. Select the namespace that the bucket and its objects will belong to.
    If you are a System Admin and the ECS system has more than one namespace, select the namespace to which the bucket will belong.
    If you are a Namespace Admin, you will only be able to select your own namespace.
  5. Specify a bucket owner.
    The bucket owner should be an ECS object user for the namespace. If you don't specify a user, you will be assigned as the owner, however, you will not be able to access the bucket unless your username is also assigned as an object user.
    The user that you specify will be given Full Control.
  6. If you want the bucket to be a CAS bucket, set the CAS control to ON.
    By default, the bucket will be marked as an S3 bucket.
  7. If you want the bucket to support HDFS operation, set the File System Enabled control to ON.
    The bucket will be an S3 bucket that supports HDFS.
  8. If the bucket is a CAS bucket or an S3 bucket on which you have not set File System Enabled, you can set Access During outage to ON.
  9. If required, specify a quota for the bucket
    The settings that you can apply are described in: Bucket concepts and attributes.
  10. If required, set a bucket retention period for the bucket.
    You can read more about retention periods in: Bucket concepts and attributes.
  11. Select Save to create the bucket.

Results

You can assign users to the bucket and set permissions for users (or pre-defined groups) from the buckets table Actions menu.

Back to Top

Edit a bucket

You can edit some bucket settings after the bucket has been created and after it has had objects written to it.

Before you begin

  • You must be a Namespace Admin or a System Admin to edit a bucket.
  • If you are a Namespace Admin you can edit the setting for buckets belonging to your namespace.
  • If you are System Admin you can edit the settings for a bucket belonging to any namespace.

Procedure

  1. At the ECS portal, select Manage > Buckets.
  2. In the Buckets table, select the Edit action for the bucket for which you want to change the settings.
  3. You can edit the following bucket attributes:
    • Quota
    • Bucket Owner
    • Access During Outage
      Note Image
      Not available if File System Enabled was set on the bucket.

    You cannot change the following attributes of the bucket:
    • Replication Group
    • File Access Enabled
    • CAS enabled
    You can find out more information about these settings in: Bucket concepts and attributes.
  4. Select Save.
Back to Top

Set the bucket ACL permissions for a user

The ECS portal enables the ACL for a bucket to be set for a user or for a pre-defined group.

Before you begin

  • You must be a Namespace Admin or a System Admin to edit the ACL for a bucket.
  • If you are a Namespace Admin you can edit the ACL settings for buckets belonging to your namespace.
  • If you are System Admin you can edit the ACL settings for a bucket belonging to any namespace.

Procedure

  1. At the ECS portal, select Manage > Buckets.
  2. In the Buckets table, select the Edit ACL action for the bucket for which you want to change the settings.
  3. To set the ACL permissions for a user, select the User ACLs button.
    To select the ACL for a group, select Group ACLs. You can refer to Set the bucket ACL permissions for a pre-defined group for more information on setting group ACLs.
  4. You can edit the permissions for a user that already has permissions assigned, or you can add a user that you want to assign permissions for.
    • To set (or remove) the ACL permissions for a user that already has permissions, select Edit (or Remove) from the Action column in the ACL table.
    • To add a user to which you want to assign permissions, select Add.
    The user that you have set as the bucket owner will have already have default permissions assigned.
  5. If you have added an ACL, enter the username of the user that the permissions will apply to.
  6. Specify the permissions that you want to apply to the user.
    More information on ACL privileges is provided in Bucket concepts and attributes.
  7. Select Save.
Back to Top

Set the bucket ACL permissions for a pre-defined group

The ECS portal enables the ACL for a bucket to be set for a user or for a pre-defined group.

Before you begin

  • You must be a Namespace Admin or a System Admin to edit the group ACL for a bucket.
  • If you are a Namespace Admin you can edit the group ACL settings for buckets belonging to your namespace.
  • If you are System Admin you can edit the group ACL settings for a bucket belonging to any namespace.

Procedure

  1. At the ECS portal, select Manage > Buckets.
  2. In the Buckets table, select the Edit ACL action for the bucket for which you want to change the settings.
  3. To set the ACL permissions for a group, select the Group ACLs button.
    To select the ACL for a user, select User ACLs. You can refer to Set the bucket ACL permissions for a user for more information on setting user ACLs.
  4. Select the group that you want to assign ACL privileges for.
    You can read more about the pre-defined privileges in: Bucket concepts and attributes
  5. Select the privileges that you want to assign to the group.
  6. Select Save.
Back to Top

Create bucket using the object APIs

When creating buckets using the object APIs or using tools that call the object APIs, there are a number of headers that determine the behavior.

The following x-emc headers are provided:

x-emc-dataservice-vpool
Determines the replication group that will be used to store the objects associated with this bucket. If you do not specify a replication group using the x-emc-dataservice-vpool header, ECS will choose the default replication group associated with the namespace.
x-emc-file-system-access-enabled
Configures the bucket for HDFS access. The header must not conflict with the interface that is being used. That is, a create bucket request from HDFS cannot specify x-emc-file-system-access-enabled=false.
x-emc-namespace
Specifies the namespace to be used for this bucket. If the namespace is not specified using the S3 convention of host/path style request, then it can be specified using the x-emc-namespace header. If the namespace is not specified as this header, the namespace associated with the user is used.
x-emc-retention-period
Specifies the retention period that will be applied to objects in a bucket. Each time a request is made to modify an object in a bucket, the expiration of the retention period for the object is calculated based on the retention period associated with the bucket.
x-emc-is-stale-enabled
Flag that indicates whether the bucket can be accesses during a temporary VDC outage in a federated configuration.
An example of using the S3curl tool to create a bucket is provided:
Back to Top

Create a bucket using the S3 API (with s3curl)

You can use the S3 API to create a bucket in an replication group. Because ECS uses custom headers (x-emc), the string to sign must be constructed to include these headers. In this procedure the s3curl tool is used; there are also a number of programmatic clients you can use, for example, the S3 Java client.

Before you begin

  • ECS must have at least one replication group configured.
  • Perl must be installed on the Linux machine on which you will run s3curl.
  • You will need to have curl installed and you will need the s3curl module, which acts as a wrapper around curl.

To use s3curl with x-emc headers, minor modifications must be made to the s3curl script. These modifications are described in the procedure.

Procedure

  1. Obtain a secret key for the user who will create the bucket.
    Refer to the article: Obtain secret key to access object storage for details.
  2. Obtain the identity of the replication group in which you want the bucket to be created.
    You can obtain the replication group identity by using the ECS REST API:
    GET https://<ECS IP Address>:4443/vdc/data-service/vpools
    
    The response provides the name and identity of all data services virtual pools. For example:
    <data_service_vpools>
    <data_service_vpool>
        <creation_time>1403519186936</creation_time>
        <id>urn:storageos:ReplicationGroupInfo:8fc8e19b-edf0-4e81-bee8-79accc867f64:global</id>
        <inactive>false</inactive>
        <tags/>
        <description>IsilonVPool1</description>
        <name>IsilonVPool1</name>
        <varrayMappings>
            <name>urn:storageos:VirtualDataCenter:1de0bbc2-907c-4ede-b133-f5331e03e6fa:vdc1</name>
            <value>urn:storageos:VirtualArray:793757ab-ad51-4038-b80a-682e124eb25e:vdc1</value>
        </varrayMappings>
    </data_service_vpool>
    </data_service_vpools>

    Here the ID is urn:storageos:ReplicationGroupInfo:8fc8e19b-edf0-4e81-bee8-79accc867f64:global.

  3. Set up s3curl by creating a .s3curl file in which to enter the user credentials.
    The .s3curl file must have permissions 0600 (rw-/---/---) when s3curl.pl is run.
    In the example below, the profile "my_profile" is used to reference the user credentials for the "user@yourco.com" account, and "root_profile" references the credentials for the root account.
    %awsSecretAccessKeys = (
        my_profile => {
            id  => 'user@yourco.com',
            key => 'sZRCTZyk93IWukHEGQ3evPJEvPUq4ASL8Nre0awN'
        },
       root_profile => {
            id  => 'root',
            key => 'sZRCTZyk93IWukHEGQ3evPJEvPUq4ASL8Nre0awN'
        },
    );
    
  4. Add the endpoint that you want to use s3curl against to the .s3curl file.
    This will be the address of your data node or the load balancer that sits in front of your data node.
    For example:
    push @endpoints , (
        '203.0.113.10',  'lglw3183.lss.emc.com',
    );
    
  5. Modify the s3curl.pl script so that it includes the x-emc headers in its "string to sign".
    Replace the following lines:
    elsif ($header =~ /^(?'header'[Xx]-(([Aa][Mm][Zz])|([Ee][Mm][Cc]))-[^:]+): *(?'val'.+)$/) {
    
        my $name = lc $+{header};
        my $value = $+{val};
    
    
    with:
     
    
    elsif ($header =~ /^([Xx]-(?:(?:[Aa][Mm][Zz])|(?:[Ee][Mm][Cc]))-[^:]+): *(.+)$/) {
       
        my $name = lc $1;
        my $value = $2;
    
    
  6. Create the bucket using s3curl.pl.
    Specify the following:
    • Profile of the user.
    • Identity of the replication group in which to create the bucket (<vpool_id>). This must be set using the x-emc-dataservice-vpool header.
    • x-emc-file-system-access-enabled header to enable file system access.
    • Name of the bucket (<BucketName>).
    The fully specified command looks like this:
    ./s3curl.pl --debug --id=my_profile --acl public-read-write 
    --createBucket -- -H 'x-emc-file-system-access-enabled:true' 
    -H 'x-emc-dataservice-vpool:<vpool_id>' http://<DataNodeIP>:9020/<BucketName>
    
    
    Note that the -acl public-read-write argument is optional, but is needed if you plan to access the bucket in an anonymous environment. For example, if you intend to access to bucket as HDFS from an environment that is not secured using Kerberos.
    If successful (with --debug on) you should see output similar to the following:
    s3curl: Found the url: host=203.0.113.10; port=9020; uri=/S3B4; query=;
    s3curl: ordinary endpoint signing case
    s3curl: StringToSign='PUT\n\n\nThu, 12 Dec 2013 07:58:39 +0000\nx-amz-acl:public-read-write
    \nx-emc-file-system-access-enabled:true\nx-emc-dataservice-vpool:
    urn:storageos:ReplicationGroupInfo:8fc8e19b-edf0-4e81-bee8-79accc867f64:global:\n/S3B4'
    s3curl: exec curl -H Date: Thu, 12 Dec 2013 07:58:39 +0000 -H Authorization: AWS 
    root:AiTcfMDhsi6iSq2rIbHEZon0WNo= -H x-amz-acl: public-read-write -L -H content-type:  
    --data-binary  -X PUT -H x-emc-file-system-access-enabled:true 
    -H x-emc-dataservice-vpool:urn:storageos:ObjectStore:e0506a04-340b-4e78-a694-4c389ce14dc8: http://203.0.113.10:9020/S3B4
    
    You can list the buckets using the S3 interface, using:
    ./s3curl.pl --debug --id=my_profile http://<DataNodeIP>:9020/
Back to Top

Bucket and key naming conventions

Bucket and object/key names must conform to the specification presented here.

Note Image

If you want to use a bucket for HDFS, you should not use underscores in the bucket name as they are not supported by the URI Java class. For example, viprfs://my_bucket.ns.site/ will not work as this is an invalid URI and is thus not understood by Hadoop.


Namespace name

The following rules apply to the naming of ECS namespaces:
  • Cannot be null or an empty string
  • Length range is 1..255 (Unicode char)
  • Valid characters are defined by regex /[a-zA-Z0-9-_]+/. Hence:
    • Alphanumeric characters
    • Special characters: hyphen (-) and underscore (_).

Back to Top

S3 bucket and object naming in ECS

This topic details the rules that apply to the naming of buckets and objects when using the ECS S3 Object API.

Bucket name

The following rules apply to the naming of S3 buckets in ECS:

  • Names must be between one and 255 characters in length. (S3 requires bucket names to be from 1 to 255 characters long).
  • Names can include dot (.), hyphen (-), and underscore (_) characters and alphanumeric characters ([a-zA-Z0-9]).

  • Names can start with a hyphen (-) or alphanumeric character.
  • The name does not support:
    • Starting with a dot (.)
    • Containing a double dot (..)
    • Ending with a dot (.)
    • Name must not be formatted as IPv4 address.

You can compare this with naming restriction specified by the S3 specification: http://docs.aws.amazon.com/AmazonS3/latest/dev/BucketRestrictions.html.

Object Name

The following rules apply to the naming of ECS S3 objects:
  • Cannot be null or an empty string
  • Length range is 1..255 (Unicode char)
  • No validation on characters.

Back to Top

OpenStack Swift container and object naming in ECS

This topic details the rules that apply to the naming of buckets and objects when using the ECS OpenStack Swift Object API.

Container Name

The following rules apply to the naming of Swift containers:
  • Cannot be null or an empty string
  • Length range is 1..255 (Unicode char)
  • Valid characters are defined by regex /[a-zA-Z0-9\\.\\-_]+/
    • Alphanumeric characters
    • Special characters: dot (.), hyphen (-), and underscore (_).

Object Name

The following rules apply to the naming of Swift objects:
  • Cannot be null or an empty string
  • Length range is 1..255 (Unicode char)
  • No validation on characters.

Back to Top

Atmos bucket and object naming in ECS

This topic details the rules that apply to the naming of buckets and objects when using the ECS Atmos Object API.

Subtenant (bucket)

This is created by the server, so the client does not need to know the naming scheme.

Object name

The following rules apply to the naming of Atmos objects:
  • Cannot be null or an empty string
  • Length range is 1..255 (Unicode char)
  • No validation on characters.

Name should be percent-encoded UTF-8.

Back to Top

CAS pool and object naming in ECS

This topic details the rules that apply to the naming of CAS pools and objects ('clips' in CAS terminology) when using the CAS API.

CAS pool naming

The following rules apply to the naming of CAS pools in ECS:
  • a maximum of 255 characters
  • cannot contain: ' " / & ? * < > <tab> <newline> or <space>

Clip naming

There are no user defined keys in the CAS API. When an application using CAS API creates a clip, it opens a pool, creates a new clip, and adds tags, attributes, streams etc. After a clip is complete it is written to a device.

A corresponding clip ID is returned by CAS engine and can be referred to using <pool name>/<clip id>.

Back to Top