Administration Guide

• 1: Management Overview
• 1.1: Overview of Eucalyptus
• 1.2: Command Line Interface
• 2: Manage Your Cloud
• 2.1: Cloud Overview
• 2.2: Cloud Best Practices
• 2.2.1: Synchronize Clocks
• 2.2.2: Configure SSL
• 2.2.3: Storage Volumes
• 2.3: Cloud Tasks
• 2.3.1: Inspect System Health
• 2.3.2: View User Resources
• 2.3.3: Change Network Configuration
• 2.3.3.1: Networking Configuration Options
• 2.3.4: Add a Node Controller
• 2.3.5: Migrate Instances Between Node Controllers
• 2.3.6: Remove a Node Controller
• 2.3.7: Restart Eucalyptus
• 2.3.7.1: Shut Down All Instances
• 2.3.7.2: Restart the CLC
• 2.3.7.3: Restart Walrus
• 2.3.7.4: Restart the CC
• 2.3.7.5: Restart the SC
• 2.3.7.6: Restart an NC
• 2.3.8: Shut Down Eucalyptus
• 2.3.8.1: Shut Down All Instances
• 2.3.8.2: Shut Down the NCs
• 2.3.8.3: Shut Down the CCs
• 2.3.8.4: Shut Down the SCs
• 2.3.8.5: Shut Down Walrus
• 2.3.8.6: Shut Down the CLC
• 2.3.9: Disable CloudWatch
• 3: Operations
• 3.1: Operations Overview
• 3.2: Planning Your Deployment
• 3.3: Testing Your Deployment
• 3.4: Customizing Your Deployment
• 3.5: Managing Policies
• 3.6: Networking
• 3.7: Monitoring
• 3.8: Backup and Recovery
• 3.8.1: Back Up Eucalyptus Cloud Data
• 3.8.1.1: Back Up the Database
• 3.8.2: Recover Eucalyptus Cloud Data
• 3.8.2.1: Restore the Database
• 3.9: Troubleshooting
• 3.9.1: Eucalyptus Log Files
• 3.9.2: Network Information
• 3.9.3: Common Problems
• 3.9.3.1: Problem: can't communicate with instance
• 3.9.3.2: Problem: install-time checks
• 3.9.3.3: Problem: instance runs but fails
• 3.9.3.4: Problem: snapshot creation failed
• 3.9.3.5: Problem: volume creation failed
• 3.9.4: Component Workarounds
• 3.9.4.1: Access and Identities
• 3.9.4.2: Elastic Load Balancing
• 3.9.4.3: Imaging Worker
• 3.9.4.4: Instances
• 3.9.4.5: Walrus and Storage
• 4: Manage Resources
• 4.1: Manage Auto Scaling Resources
• 4.2: Manage CloudWatch Resources
• 4.3: Manage Compute Resources
• 4.4: Manage ELB Resources
• 4.5: Manage IAM Resources
• 4.6: Manage Walrus Resources
• 5: Manage Regions
• 5.1: Regions Overview
• 5.2: Region Configuration File Format
• 5.3: Examples
• 5.4: Federation Differences Between AWS and Eucalyptus
• 5.5: Troubleshooting
• 6: Manage Security
• 6.1: Security Overview
• 6.2: Best Practices
• 6.2.1: Authentication and Access Control Best Practices
• 6.2.2: Hosts
• 6.2.3: Images and Instances
• 6.2.4: Management Console
• 6.2.5: Message Security
• 6.2.6: Networking Modes
• 6.3: Security Tasks
• 6.3.1: Configure SSL
• 6.3.1.1: Configure and Enable SSL for the Management Console
• 6.3.1.2: Configure and Enable SSL for the UFS
• 6.3.2: Change Multicast Address
• 6.3.3: Configure Replay Protection
• 6.3.4: Configure Session Timeouts
• 6.3.5: Configure STS Actions
• 6.3.6: Configure the Firewall
• 6.3.7: Reserve Ports
• 6.3.8: Synchronize Components
• 7: Eucalyptus Commands
• 7.1: Eucalyptus Administration Commands
• 7.1.1: euctl
• 7.1.2: euserv-deregister-service
• 7.1.3: euserv-describe-events
• 7.1.4: euserv-describe-node-controllers
• 7.1.5: euserv-describe-service-types
• 7.1.6: euserv-describe-services
• 7.1.7: euserv-migrate-instances
• 7.1.8: euserv-modify-service
• 7.1.9: euserv-register-service
• 7.2: Eucalyptus Configuration Variables
• 8: Advanced Storage Configuration
• 8.1: OSG Advanced Configuration

1 - Management Overview

Management Overview

The section shows you how to access Eucalyptus with a web-based console and with command line tools. This section also describes how to perform common management tasks. This document is intended to be a reference. You do not need to read it in order, unless you are following the directions for a particular task.

1.1 - Overview of Eucalyptus

Eucalyptus is a Linux-based software architecture that implements scalable, efficiency-enhancing private and hybrid clouds within an enterprise’s existing IT infrastructure. Because Eucalyptus provides Infrastructure as a Service (IaaS), you can provision your own resources (hardware, storage, and network) through Eucalyptus on an as-needed basis.

A Eucalyptus cloud is deployed across your enterprise’s on-premise data center. As a result, your organization has a full control of the cloud infrastructure. You can implement and enforce various level of security. Sensitive data managed by the cloud does not have to leave your enterprise boundaries, keeping data completely protected from external access by your enterprise firewall.

Eucalyptus was designed from the ground up to be easy to install and non-intrusive. The software framework is modular, with industry-standard, language-agnostic communication. Eucalyptus is also unique in that it provides a virtual network overlay that isolates network traffic of different users as well as allows two or more clusters to appear to belong to the same Local Area Network (LAN).

Eucalyptus also is compatible with Amazon’s EC2, S3, and IAM services. This offers you hybrid cloud capability.

1.2 - Command Line Interface

Eucalyptus supports two command line interfaces (CLIs): the administration CLI and the user CLI.The administration CLI is installed when you install Eucalyptus server-side components. The administration CLI is for maintaining and modifying Eucalyptus.

The other user CLI, called Euca2ools, can be downloaded and installed on clients. Euca2ools is a set of commands for end users and can be used with both Eucalyptus and Amazon Web Services (AWS).

2 - Manage Your Cloud

Manage Your Cloud

After you install and initially configure Eucalyptus, there are some common administration tasks you can perform. This section describes these tasks and associated concepts.

2.1 - Cloud Overview

This topic presents an overview of the components in Eucalyptus. Eucalyptus is comprised of several components: Cloud Controller, Walrus, Cluster Controller, Storage Controller, and Node Controller. Each component is a stand-alone web service. This architecture allows Eucalyptus both to expose each web service as a well-defined, language-agnostic API, and to support existing web service standards for secure communication between its components.

Cloud Controller

The Cloud Controller (CLC) is the entry-point into the cloud for administrators, developers, project managers, and end-users. The CLC queries other components for information about resources, makes high-level scheduling decisions, and makes requests to the Cluster Controllers (CCs). As the interface to the management platform, the CLC is responsible for exposing and managing the underlying virtualized resources (servers, network, and storage). You can access the CLC through command line tools that are compatible with Amazon’s Elastic Compute Cloud (EC2).

Walrus

Walrus allows users to store persistent data, organized as buckets and objects. You can use Walrus to create, delete, and list buckets, or to put, get, and delete objects, or to set access control policies. Walrus is interface compatible with Amazon’s Simple Storage Service (S3). It provides a mechanism for storing and accessing virtual machine images and user data. Walrus can be accessed by end-users, whether the user is running a client from outside the cloud or from a virtual machine instance running inside the cloud.

Cluster Controller

The Cluster Controller (CC) generally executes on a machine that has network connectivity to both the machines running the Node Controller (NC) and to the machine running the CLC. CCs gather information about a set of NCs and schedules virtual machine (VM) execution on specific NCs. The CC also manages the virtual machine networks. All NCs associated with a single CC must be in the same subnet.

Storage Controller

The Storage Controller (SC) provides functionality similar to the Amazon Elastic Block Store (Amazon EBS). Elastic block storage exports storage volumes that can be attached by a VM and mounted or accessed as a raw block device. EBS volumes persist past VM termination and are commonly used to store persistent data. An EBS volume cannot be shared between VMs and can only be accessed within the same availability zone in which the VM is running. Users can create snapshots from EBS volumes. Snapshots may be stored in Walrus and made available across availability zones.

Node Controller

The Node Controller (NC) executes on any machine that hosts VM instances. The NC controls VM activities, including the execution, inspection, and termination of VM instances. It also fetches and maintains a local cache of instance images, and it queries and controls the system software (host OS and the hypervisor) in response to queries and control requests from the CC. The NC is also responsible for the management of the virtual network endpoint.

2.2 - Cloud Best Practices

Cloud Best Practices

This section details Eucalyptus best practices for your private cloud.

2.2.1 - Synchronize Clocks

Eucalyptus checks message timestamps across components in the cloud infrastructure. This assures command integrity and provides better security.Eucalyptus components receive and exchange messages using either Query or SOAP interfaces (or both). Messages received over these interfaces are required to have some form of a time stamp (as defined by AWS specification) to prevent message replay attacks. Because Eucalyptus enforces strict policies when checking timestamps in the received messages, for the correct functioning of the cloud infrastructure, it is crucial to have clocks constantly synchronized (for example, with ntpd) on all machines hosting Eucalyptus components. To prevent user command failures, it is also important to have clocks synchronized on the client machines.

Following the AWS specification, all Query interface requests containing the Timestamp element are rejected as expired after 15 minutes of the timestamp. Requests containing the Expires element expire at the time specified by the element. SOAP interface requests using WS-Security expire as specified by the WS-Security Timestamp element.

When checking the timestamps for expiration, Eucalyptus allows up to 20 seconds of clock drift between the machines. This is a default setting. You can change this value for the CLC at runtime by setting the bootstrap.webservices.clock_skew_sec property as follows:

euctl bootstrap.webservices.clock_skew_sec=<new_value_in_seconds>

For additional protection from the message replay attacks, the CLC implements a replay detection algorithm and rejects messages with the same signatures received within 15 minutes. Replay detection parameters can be tuned as described in Configure Replay Protection .

2.2.2 - Configure SSL

In order to connect to Eucalyptus using SSL/TLS, you must have a valid certificate for the Cloud Controller (CLC)

If you have more than one host (other than node controllers), note the following:

• The keystore must be updated on each host running the eucalyptus-cloud service
• The [key_alias] must be the same on each host
• Use a wildcard certificate (i.e. *.<system.dns.dnsdomain>), since UFS is responsible for all service API endpoints

Create a keystore

Eucalyptus uses a PKCS12-format keystore. If you are using a certificate signed by a trusted root CA, use the following command to convert your trusted certificate and key into an appropriate format:

openssl pkcs12 -export -in [YOURCERT.crt] -inkey [YOURKEY.key] \
  -out tmp.p12 -name [key_alias]

Save a backup of the Eucalyptus keystore, at /var/lib/eucalyptus/keys/euca.p12 , and then import your keystore into the Eucalyptus keystore as follows:

keytool -importkeystore \
  -srckeystore tmp.p12 -srcstoretype pkcs12 -srcstorepass [export_password] \
  -destkeystore /var/lib/eucalyptus/keys/euca.p12 -deststoretype pkcs12 \
  -deststorepass eucalyptus -alias [key_alias] \
  -srckeypass [export_password]

Enable the Cloud Controller to use this keystore

Run the following commands on the Cloud Controller (CLC):

euctl bootstrap.webservices.ssl.server_alias=[key_alias]

Optional: Redirect Requests to use Port 443

To allow user facing services requests on port 443 instead of the default 8773, run the following commands on the CLC:

euctl bootstrap.webservices.port=443

2.2.3 - Storage Volumes

Eucalyptus manages storage volumes for your private cloud. Volume management strategies are application specific, but this topic includes some general guidelines.When setting up your Storage Controller, consider whether performance (bandwidth and latency of read/write operations) or availability is more important for your application. For example, using several smaller volumes will allow snapshots to be taken on a rolling basis, decreasing each snapshot creation time and potentially making restore operations faster if the restore can be isolated to a single volume. However, a single larger volume allows for faster read/write operations from the VM to the storage volume.

An appropriate network configuration is an important part of optimizing the performance of your storage volumes. For best performance, each Node Controller should be connected to a distinct storage network that enables the NC to communicate with the SC or Ceph, without interfering with normal NC/VM-instance network traffic.

Eucalyptus includes configurable limits on the size of a single volume, as well as the aggregate size of all volumes on an SC. The SC can push snapshots from Ceph, where the volumes reside, to object storage, where the snapshots become available across multiple clusters. Smaller volumes will be much faster to snapshot and transfer, whereas large volumes will take longer. However, if many concurrent snapshot requests are sent to the SC, operations may take longer to complete.

EBS volumes are created from snapshots on the SC or Ceph, after the snapshot has been downloaded from object storage to the device. Creating an EBS volume from a snapshot on the same cluster as the source volume of the snapshot will reduce delays caused by having to transfer snapshots from object storage.

2.3 - Cloud Tasks

Cloud Tasks

This section contains a listing of your Eucalyptus cloud-related tasks.

2.3.1 - Inspect System Health

Eucalyptus provides access to the current view of service state and the ability to manipulate the state. You can inspect the service state to either ensure system health or to identify faulty services. You can modify a service state to maintain activities and apply external service placement policies.

View Service State

Use the euserv-describe-services command to view the service state. The output indicates:

• Component type of the service
• Partition in which the service is registered
• Unique name of the service
• Current view of service state
• Last reported epoch (this can be safely ignored)
• Service URI
• Fully qualified name of the service (This is needed for manipulating services that did not get unique names during registration. For example: internal services like DNS) The default output includes the services that are registered during configuration, as well as information about the DNS service, if present. You can obtain additional service state information, such as internal services, by providing the -a flag.

You can also make requests to retrieve service information that is filtered by either:

• current state (for example, )
• host where service is registered
• partition where service is registered
• type of service (for example, CC or Walrus) When you investigate service failures, you can specify -events to return a summary of the last fault. You can retrieve extended information (primarily useful for debugging) by specifying -events -events-verbose .

Heartbeat Service

http://CLCIPADDRESS:8773/services/Heartbeat provides a list of components and their respective statuses. This allows you to find out if a service is enabled without requiring cloud credentials.

Modify Service State

To modify a service:

Enter the following command on the CLC, Walrus, or SC machines:

systemctl stop eucalyptus-cloud.service

On the CC, use the following command:

systemctl stop eucalyptus-cluster.service

If you want to shut down the SC for maintenance. The SC is SC00 is ENABLED and needs to be DISABLED for maintenance.

To stop SC00 first verify that no volumes or snapshots are being created and that no volumes are being attached or detached, and then enter the following command on SC00:

systemctl stop eucalyptus-cloud.service

To check status of services, you would enter:

euserv-describe-services

When maintenance is complete, you can start the eucalyptus-cloud process on SC00 , which will enter the DISABLED state by default.

systemctl start eucalyptus-cloud.service

Monitor the state of services using euserv-describe-services until SC00 is ENABLED .

2.3.2 - View User Resources

To see resource use by your cloud users, Eucalyptus provides the following commands with the flag.

• : Returns information about security groups in your account, including output type identifier, security group ID, security group name, security group description, output type identifier, account ID of the group owner, name of group granting permission, type of rule, protocol to allow, start of port range, end of port range, source (for ingress rules) or destination (for egress rules), and any tags assigned to the security group.
• : Returns information about your instances, including output type identifier, reservation ID, name of each security group the instance is in, output type identifier, instance ID for each running instance, EMI ID of the image on which the instance is based, public DNS name associated with the instance (for instances in the running state), private DNS name associated with the instance (for instances in running state), instance state, key name, launch index, instance type, launch time, availability zone, kernel ID, ramdisk ID, monitoring state, public IP address, private IP address, type of root device (ebs or instance-store), placement group the cluster instance is in, virtualization type (paravirtual or hvm), any tags assigned to the instance, hypervisor type, block device identifier for each EBS volume the instance is using, along with the device name, the volume ID, and the timestamp.
• : Returns information about key pairs available to you, including keypair identifier, keypair name, and private key fingerprint.
• : Returns information about EBS snapshots available to you, including snapshot identifier, ID of the snapshot, ID of the volume, snapshot state (pending, completed, error), timestamp when snapshot initiated, percentage of completion, ID of the owner, volume sized, description, and any tags assigned to the snapshot.
• : Describes your EBS volumes, including volume identifier, volume ID, size of the volume in GiBs, snapshot from which the volume was created, availability zone, volume state (creating, available, in-use, deleting, deleted, error), timestamp of the volume creation, and any tags assigned to the volume.

2.3.3 - Change Network Configuration

Change Network Configuration

You might want to change the original network configuration of your cloud. To change your network configuration, perform the tasks listed in this topic.Log in to the CLC and open the /etc/eucalyptus/eucalyptus.conf file. Navigate to the Networking Configuration section and make your edits. Save the file. Restart the Cluster Controller.

systemctl restart eucalyptus-cluster.service

2.3.3.1 - Networking Configuration Options

All network-related options specified in /etc/eucalyptus/eucalyptus.conf use the prefix VNET_. The most commonly used VNET options are described in the following table.

2.3.4 - Add a Node Controller

If you want to increase your system’s capacity, you’ll want to add more Node Controllers (NCs).To add an NC, perform the following tasks:

Log in to the CLC and enter the following command:

clusteradmin-register-nodes node0_IP_address ... [nodeN_IP_address]

When prompted, enter the password to log into each node. Eucalyptus requires this password to propagate the cryptographic keys.

2.3.5 - Migrate Instances Between Node Controllers

In order to ensure optimal system performance, or to perform system maintenance, it is sometimes necessary to move running instances between Node Controllers (NCs). You can migrate instances individually, or migrate all instances from a given NC.

euserv-migrate-instances -i INSTANCE_ID

You can also optionally specify --include-dest HOST_NC_IP or --exclude-dest HOST_NC_IP , to ensure that the instance is migrated to one of the specified NCs, or to avoid migrating the instance to any of the specified NCs. These flags may be used more than once to specify multiple NCs.

To migrate all instances away from an NC, enter the following command:

euserv-migrate-instances --source HOST_NC_IP

You can also optionally specify euserv-modify-service -s stop HOST_NC_IP , to stop the specified NC and ensure that no new instances are started on that NC while the migration occurs. This allows you to safely remove the NC without interrupting running instances. The NC will remain in the DISABLED state until it is explicitly enabled using euserv-modify-service -s start HOST_NC_IP .

In some cases, timeouts may cause a migration to initially fail. Run the command again to complete the migration.

If the migration fails, check the nc.log file on the source and destination NCs. If you see an error similar to:

libvirt: Cannot get interface MTU on 'br0': No such device (code=38)

… then ensure the NCs have the same interface and bridge device names, as described in .

2.3.6 - Remove a Node Controller

Describes how to delete NCs in your system.If you want to decrease your system’s capacity, you’ll need to decrease NC servers. To delete an NC, perform the following tasks.

Log in to the CC and enter the following command:

clusteradmin-deregister-nodes node0_IP_address ... [nodeN_IP_address]

2.3.7 - Restart Eucalyptus

Restart Eucalyptus

Describes the recommended processes to restart Eucalyptus, including terminating instances and restarting Eucalyptus components.You must restart Eucalyptus whenever you make a physical change (e.g., switch out routers), or edit the eucalyptus.conf file. To restart Eucalyptus, perform the following tasks in the order presented.

2.3.7.1 - Shut Down All Instances

To terminate all instances on all NCs perform the steps listed in this topic. To terminate all instances on all NCs:

Enter the following command:

euca-terminate-instances <instance_id>

2.3.7.2 - Restart the CLC

Log in to the CLC and enter the following command:

systemctl restart eucalyptus-cloud.service

All Eucalyptus components on this server will restart.

2.3.7.3 - Restart Walrus

Log in to Walrus and enter the following command:

systemctl restart eucalyptus-cloud.service

2.3.7.4 - Restart the CC

Log in to the CC and enter the following command:

systemctl restart eucalyptus-cluster.service

2.3.7.5 - Restart the SC

Log in to the SC and enter the following command:

systemctl restart eucalyptus-cloud.service

2.3.7.6 - Restart an NC

To restart an NC perform the steps listed in this topic.Log in to the NC and enter the following command:

systemctl restart eucalyptus-node.service

Repeat for each NC. Verify that the following is even needed. If so, replicate for other NC-tasks. You can automate the restart command for all of your NCs. Store a list of your NCs in a file called nc-hosts that looks like:

nc-host-00
nc-host-01
...
nc-host-nn

To restart all of your NCs, run the following command:

cat nc-hosts | xargs -i ssh root@{} systemctl restart eucalyptus-node.service

2.3.8 - Shut Down Eucalyptus

Shut Down Eucalyptus

Describes the recommended processes to shut down Eucalyptus.There may be times when you need to shut down Eucalyptus. This might be because of a physical failure, topological change, backing up, or making an upgrade. We recommend that you shut down Eucalyptus components in the reverse order of how you started them. To stop the system, shut down the components in the order listed.

2.3.8.1 - Shut Down All Instances

To terminate all instances on all NCs perform the steps listed in this topic.To terminate all instances on all NCs:

Enter the following command:

euca-terminate-instances <instance_id>

2.3.8.2 - Shut Down the NCs

To shut down the NCs perform the steps listed in this topic.To shut down the NCs:

Log in as root to a machine hosting an NC. Enter the following command:

systemctl stop eucalyptus-node.service

Repeat for each machine hosting an NC.

2.3.8.3 - Shut Down the CCs

To shut down the CCs:

Log in as root to a machine hosting a CC. Enter the following command:

systemctl stop eucalyptus-cluster.service

Repeat for each machine hosting a CC.

2.3.8.4 - Shut Down the SCs

To shut down the SC:

Log in as root to the physical machine that hosts the SC. Enter the following command:

systemctl stop eucalyptus-cloud.service

Repeat for any other machine hosting an SC.

2.3.8.5 - Shut Down Walrus

To shut down Walrus:

Log in as root to the physical machine that hosts Walrus. Enter the following command:

systemctl stop eucalyptus-cloud.service

2.3.8.6 - Shut Down the CLC

To shut down the CLC:

Log in as root to the physical machine that hosts the CLC. Enter the following command:

systemctl stop eucalyptus-cloud.service

2.3.9 - Disable CloudWatch

To disable CloudWatch, run the following command.

euctl cloudwatch.enable_cloudwatch_service=true

3 - Operations

Operations

This section contains concepts and tasks associated with operating your Eucalyptus cloud.

3.1 - Operations Overview

This section is for architects and cloud administrators who plan to deploy Eucalyptus in a production environment. It is not intended for end users or proof-of-concept installations.To run Eucalyptus in a production environment, you must be aware of your hardware and network resources. This guide is to help you make decisions about deploying Eucalyptus. It is also meant to help you keep Eucalyptus running smoothly.

3.2 - Planning Your Deployment

To decide on your deployment’s scope, determine the use case for your cloud. For example, will this be a small dev-test environment, or a large and scalable web services environment?To help with scoping your deployment, see Plan Your Installation in the Installation Guide . There you will find solution examples and physical resource information.

3.3 - Testing Your Deployment

This topic details what you should test when you want to make sure your deployment is working. The following suggested test plan contains tasks that ensure DNS, imaging, and storage are working.

DNS

• Verify that instances can ping their:
• Verify that instances are pingable on their public DNS names from:

Imaging

• Verify that an EBS-backed image boots successfully
• Verify that you can create an image from a running EBS-backed instance
• Verify that you can install a new Ubuntu image
• Verify that you can deregister an image
• Verify that you can import an instance
• Verify that you can import a volume

Walrus

• Verify that you can make a basic s3cmd request
• Verify that you can successfully perform a multi-part upload (use a 1G+ file)

3.4 - Customizing Your Deployment

This section describes the most commonly applied post-install customizations and the issues they pose:

• Over-subscription
• Networking changes (EDGE mode)
• CloudWatch tweaks/customizations
• Capacity changes

Over-subscription

Over-subscription refers to the practice of expanding your computer beyond its limits. Over-subscription applies only to node controllers. You may modify disks and cores to allow enough usage buffer for your instance. Navigate to /etc/eucalyptus/ and locate the eucalyptus.conf file. Edit the following values to define the appropriate size buffers for your instances: NC_WORK_SIZE Defines the amount of disk space available for instances to be run. Defaults to 1/3 of the currently available disk space on the NC, and NC_CACHE_SIZE defaults to the other 2/3.

NC_CACHE_SIZE Defines how much disk space is needed for images to be cached. MAX_CORES Defines the maximum number of cores that can be provided to VMs on each NC. If it is 0 or not present, then the only limit on the number of instances is the number of cores available on the NC. If it is present, any value greater than 256 is treated as 256. In order for these changes to take effect, you must restart the NC.

Networking Changes (EDGE modes)

You can modify the default by adding network IPs to your cloud. Adding public IPs does not require shutting down the whole system.

To add network IPs:In EDGE mode, adding or changing the IP involves creating a JSON file and uploading it the Cloud Controller (CLC). See Configure for Edge Mode for more details. No restart needed, changes apply automatically.

Change CloudWatch Properties

You can change the following CloudWatch properties:

Change Capacity

Capacity changes refer to adding another zone or more nodes. To add another zone, install , start , and register . To add more nodes, see Add a Node Controller .

3.5 - Managing Policies

This topic details best practices for managing your cloud policies.

• Establish a workflow for account creation, including the initial request for a cloud account and the email containing credentials.
• Limit your use of individual policies. Focus your policies on groups and add individuals to the group.
• Use groups to assign permissions to individual users. Limit the use of policies for individual users. For more information about policy best practices, see IAM Best Practices .

3.6 - Networking

This topic addresses networking in the Eucalyptus cloud.

Networking Modes

Eucalyptus offers different modes to provide you with a cloud that will fit in your current network. For information what each networking mode has to offer, see Plan Networking Modes .

EC2-Classic Networking

Eucalyptus EDGE networking mode supports EC2-Classic networking. Your instances run in a single, flat network that you share with others. For more information about EC2-Classic networking, see EC2 Supported Platforms .

EC2-VPC Networking

Eucalyptus VPCMIDO networking mode resembles the Amazon Virtual Private Cloud (VPC) product wherein the network is fully configurable by users. For more information about EC2-VPC networking, see Differences Between Instances in EC2-Classic and EC2-VPC .

3.7 - Monitoring

This topic includes details about which resources you should monitor.

3.8 - Backup and Recovery

Backup and Recovery

This section provides details on important files to back up and recover.

3.8.1 - Back Up Eucalyptus Cloud Data

Back Up Cloud Data

This section explains what you need to back up and protect your cloud data.We recommend that you back up the following data:

• The cloud database: see
• Object storage. For objects in Walrus, the frequency depends on current load. Use your own discretion to determine the backup plan and strategy. You must have Walrus running.
• EBS volumes in each cluster (DAS and Overlay)
• The configuration file for the cloud is stored on the CLC: .
• Any configuration file for the cloud stored on any other host (UFS, CC, etc.): .
• The cloud security credentials on all hosts (you already backed up the CLC keys as part of the database backup). Use the tar command: .
• The CC and NC configuration files, stored on every CC and NC: .
• Any Euca2ools (.ini) configuration files, which reside on any Euca2ools host machine. Files can be found in:
• Management Console config files in should be backed up. Typical files:
• Ensure you have your instances’ so you can access the instances later.
• and LVM snapshots Users are responsible for volume backups using EBS snapshots on their defined schedules.

3.8.1.1 - Back Up the Database

To back up the cloud database follow the steps listed in this topic.Bucket and object metadata are stored in the Eucalyptus cloud database. To back up the database

Log in to the CLC. The cloud database is on the CLC. Extract the Eucalyptus PostgreSQL database cluster into a script file.

pg_dumpall --oids -c -h/var/lib/eucalyptus/db/data -p8777 -U root -f/root/eucalyptus_pg_dumpall-backup.sql

Back up the cloud security credentials in the keys directory.

tar -czvf ~/eucalyptus-keydir.tgz /var/lib/eucalyptus/keys

3.8.2 - Recover Eucalyptus Cloud Data

Recover Cloud Data

This topic explains what to include when you recover your cloud.Recovering Your Cloud Data

• The cloud database: see
• Object storage. For objects in Walrus, the frequency depends on current load. Use your own discretion to determine the restore plan and strategy.
• EBS volumes in each cluster (DAS and Overlay)
• The configuration file for the cloud is stored on the CLC: .
• Any configuration file for the cloud stored on any other host (UFS, CC, etc.): .
• The cloud security credentials on all hosts (you already restored the CLC keys as part of the database restore). Use the tar command: .
• The CC and NC configuration files, stored on every CC and NC: .
• Any Euca2ools (.ini) configuration files, which reside on any Euca2ools host machine. Files in these directories:
• Management Console config files you backed up from should be restored. Typical files:
• Ensure you have your instances’ so you can access the instances.
• and LVM snapshots Users are responsible for volume restore using EBS snapshots.

3.8.2.1 - Restore the Database

To restore the cloud database follow the steps listed in this topic.

Stop the CLC service.

systemctl stop eucalyptus-cloud.service

Remove traces of the old database.

rm -rf /var/lib/eucalyptus/db

Restore the cloud security credentials in the keys directory.

tar -xvf ~/eucalyptus-keydir.tgz -C /

Re-initialize the database structure.

clcadmin-initialize-cloud

Start the database manually.

su eucalyptus -s /bin/bash -c "/usr/bin/pg_ctl start -w \
-s -D/var/lib/eucalyptus/db/data -o '-h0.0.0.0/0 -p8777 -i'"

Restore the backup.

psql -U root -d postgres -p 8777 -h /var/lib/eucalyptus/db/data -f/root/eucalyptus_pg_dumpall-backup.sql

Stop the database manually.

su eucalyptus -s /bin/bash -c "/usr/bin/pg_ctl stop -D/var/lib/eucalyptus/db/data"

Start CLC service

systemctl start eucalyptus-cloud.service

3.9 - Troubleshooting

Troubleshooting

This topic details how to find information you need to troubleshoot most problems in your cloud. To troubleshoot Eucalyptus, you must have the following:

• a knowledge about which machines each Eucalyptus component is installed on
• root access to each machine hosting Eucalyptus components
• an understanding of the network mode (EDGE, VPCMIDO)
• an understanding of eucanetd and the configuration connecting the Eucalyptus components

For most problems, the procedure for tracing problems is the same: start at the bottom to verify the bottom-most component, and then work your way up. If you do this, you can be assured that the base is solid. This applies to virtually all Eucalyptus components and also works for proactive, targeted monitoring.

3.9.1 - Eucalyptus Log Files

Usually when an issue arises in Eucalyptus, you can find information that points to the nature of the problem either in the Eucalyptus log files or in the system log files. This topic details log file message meanings, location, configuration, and fault log information.

3.9.2 - Network Information

When you have to troubleshoot, it’s important to understand the elements of the network on your system.Here are some ideas for finding out information about your network:

• It is also important to understand the elements of the network on your system. For example, you might want to list bridges to see which devices are enslaved by the bridge. To do this, use the command.
• You might also want to list network devices and evaluate existing configurations. To do this, use these commands: , , and .
• You can use to check status, or to force eucanetd to run in the foreground, sending log messages to the terminal.
• You can get further information if you use the commands with the options. For example, returns all instances running by all users on the system. Other describe commands are:

3.9.3 - Common Problems

Common Problems

This section describes common problems and workarounds.

3.9.3.1 - Problem: can't communicate with instance

Use ping from a client (not the CLC). Can you ping it?

Yes: Check the open ports on security groups and retry connection using SSH or HTTP. Can you connect now? Yes. Okay, then. You’re work is done. No: Try the same procedure as if you can’t ping it up front. No: Is your cloud running in Edge networking mode?

• Yes: Run euca-describe-nodes . Is your instance there?

• No, it is not in Edge networking mode:

3.9.3.2 - Problem: install-time checks

Eucalyptus offers installation checks for any Eucalyptus component or service (CLC, Walrus, SC, NC, SC, services, and more). When Eucalyptus encounters an error, it presents the problem to the operator. These checks are used for install-time problems. They provide resolutions to some of the fault conditions.

Each problematic condition contains the following information:

For more information about all the faults we support, go to https://github.com/eucalyptus/eucalyptus/tree/master/util/faults/en_US .

3.9.3.3 - Problem: instance runs but fails

Run euca-describe-nodes to verify if instance is there. Is the instance there?

Yes: Go to the NC log for that NC and grep your instance ID. Did you find the instance?

• Yes: Is there an error message?

No: Go to the CC log and grep the instance ID. Is it there error message?

• Yes: The error message should give you some helpful information.

• No: grep the instance ID in cloud-output.log . Is there error message?

No: Log in as admin and run euca-describe-instance . Is the instance there?

• Yes:
• No: Start over and run a new instance, recreate failure, and start these steps over.

3.9.3.4 - Problem: snapshot creation failed

On the SC, depending on the backend used for storage:

• For Overlay, use the command to check the disk space in .
• For DAS, use the command to check the disk space in the DAS volumes.
• For any other backend, use its specific commands to check the free space for storage allocated for volumes. Is there enough space?

Yes: On the OSG host, depending on the backend used for object storage:

• For Walrus, use the command to check the disk space in .

• For RiakCS or Ceph-RGW, use its specific commands to check the free space for storage allocated for buckets and objects. Is there enough space? Yes.

• Use and note the IP addresses for the OSG and SC.

• SSH to SC and ping the OSG. Are there error messages?

No: Delete volumes or add disk space. No: Delete volumes or add disk space.

3.9.3.5 - Problem: volume creation failed

Symptom: Went from available to fail. This is typically caused by the CLC and the SC.On the SC, use df or lvdisplay to check the disk space. Is there enough space?

Yes: Check the SC log and grep the volume ID. Is there error message? Yes. This provides clues to helpful information. No: Check cloud-output.log for a volume ID error. No: Delete volumes or add disk space.

3.9.4 - Component Workarounds

Component Workarounds

This section contains troubleshooting information for Eucalyptus components and services.

3.9.4.1 - Access and Identities

This topic contains information about access-related problems and solutions. Need to verify an existing LIC file.

1. Enter the following command: The output from the example above shows the name of the LIC file and status of the synchronization (set to false).

3.9.4.2 - Elastic Load Balancing

This topic explains suggestions for problems you might have with Elastic Load Balancing (ELB). Can’t synchronize with time server Eucalyptus sets up NTP automatically for any instance that has an internet connection to a public network. If an instance doesn’t have such a connection, set the cloud property loadbalancing.loadbalancer_vm_ntp_server to a valid NTP server IP address. For example:

euctl loadbalancing.loadbalancer_vm_ntp_server=169.254.169.254
PROPERTY	loadbalancing.loadbalancer_vm_ntp_server	169.254.169.254 was {}

Need to debug an ELB instance To debug an ELB instance, set the loadbalancing.loadbalancer_vm_keyname cloud property to the keypair of the instance you want to debug. For example:

# euctl loadbalancing.loadbalancer_vm_keyname=sshlogin
PROPERTY	loadbalancing.loadbalancer_vm_keyname	sshlogin was {}

3.9.4.3 - Imaging Worker

This topic contains troubleshooting tips for the Imaging Worker.Some requests that require the Imaging Worker might remain in pending for a long time. For example: an import task or a paravirtual instance run. If request remains in pending, the Imaging Worker instance might not able to run because of a lack of resources (for example, instance slots or IP addresses).

You can check for this scenario by listing latest AutoScaling activities:

euscale-describe-scaling-activities -g asg-euca-internal-imaging-worker-01

Check for failures that indicate inadequate resources such as:

ACTIVITY        1950c4e5-0db9-4b80-ad3b-5c7c59d9c82e    2014-08-12T21:05:32.699Z        asg-euca-internal-imaging-worker-01    Failed   Not enough resources available: addresses; please stop or terminate unwanted instances or release unassociated elastic IPs and try again, or run with private addressing only

3.9.4.4 - Instances

This topic contains information to help you troubleshoot your instances. Inaccurate IP addresses display in the output of euca-describe-addresses. This can occur if you add IPs from the wrong subnet into your public IP pool, do a restart on the CC, swap out the wrong ones for the right ones, and do another restart on the CC. To resolve this issue, run the following commands.

systemctl stop eucalyptus-cloud.service
systemctl stop eucalyptus-cluster.service
iptables -F
systemctl restart eucalyptus-cluster.service
systemctl start eucalyptus-cloud.service

NC does not recalculate disk size correctly This can occur when trying to add extra disk space for instance ephemeral storage. To resolve this, you need to delete the instance cache and restart the NC.

For example:

rm -rf /var/lib/eucalyptus/instances/* 
systemctl restart eucalyptus-node.service               				

3.9.4.5 - Walrus and Storage

This topic contains information about Walrus-related problems and solutions. Walrus decryption failed. On Ubuntu 10.04 LTS, kernel version 2.6.32-31 includes a bug that prevents Walrus from decrypting images. This can be determined from the following line in cloud-output.log

javax.crypto.
BadPaddingException: pad block corrupted

If you are running this kernel:

1. Update to kernel version 2.6.32-33 or higher.
2. De-register the failed image ( ).
3. Re-register the bundle that you uploaded ( ).

Walrus physical disk is not large enough.

1. Stop the CLC.
2. Add a disk.
3. Migrate your data. Make sure you use LVM with your new disk drive(s).

4 - Manage Resources

Manage Resources

This section includes tasks to help you manage your users’ cloud resources.

4.1 - Manage Auto Scaling Resources

You can list, delete, update, and suspend your Eucalyptus cloud’s Autoscaling resources by passing the option with the keyword with the appropriate command.The followings are some examples you can use to act on your Auto Scaling resources.

To show all launch configurations in your cloud, run the following command:

euscale-describe-launch-configs --show-long verbose

To show all Auto Scaling instances in your cloud, run the following command:

euscale-describe-auto-scaling-groups --show-long verbose

To show all Auto Scaling instances in your cloud, run the following command:

euscale-describe-auto-scaling-groups --show-long verbose

To delete an Auto Scaling resource in your cloud, first get the ARN of the resource, as in this example:

$ euscale-describe-launch-configs --show-long verbose
LAUNCH-CONFIG  TestLaunchConfig  emi-06663A57  m1.medium  2013-10-30T22:52:39.392Z  true
arn:aws:autoscaling::961915002812:launchConfiguration:5ac29caf-9aad-4bdb-b228-5f
ce841dc062:launchConfigurationName/TestLaunchConfig

Then run the following command with the ARN:

euscale-delete-launch-config
arn:aws:autoscaling::961915002812:launchConfiguration:5ac29caf-9aad-4bdb-b228-5f
ce841dc062:launchConfigurationName/TestLaunchConfig

4.2 - Manage CloudWatch Resources

To manage CloudWatch resources on a Eucalyptus cloud, use the option in any command that lists, deletes, modifies, or sets a CloudWatch resource.The following are examples of what you can do with your CloudWatch resources.

To list all alarms for the cloud, run the following command:

euwatch-describe-alarms verbose

4.3 - Manage Compute Resources

To manage compute resources on a Eucalyptus cloud, use the option in any command.The following are some examples you can use to view various compute resources.

To see all instances running on your cloud, enter the following command:

euca-describe-instances verbose

To see all volumes in your cloud, enter the following command:

euca-describe-volumes verbose

To see all keypairs in your cloud, enter the following command:

euca-describe-keypairs verbose

4.4 - Manage ELB Resources

To list and delete ELB resources on a Eucalyptus cloud, use the option with any command.The following are some examples.

To list all detailed configuration information for the load balancers in your cloud, run the following command:

eulb-describe-lbs verbose

To list the details of policies for all load balancers in your cloud, run the following command:

eulb-describe-lb-policies verbose

To list meta information for all load balancer policies in your cloud, run the following command:

eulb-describe-lb-policy-types verbose

To delete any load balancer or any load balancer resource on the cloud, instead of using the ELB name, use the DNS name. For example:

$ eulb-describe-lbs verbose
LOAD_BALANCER	MyLoadBalancer	MyLoadBalancer-961915002812.lb.foobar.eucalyptus-systems.com	2013-10-30T03:02:53.39Z

$ eulb-delete-lb MyLoadBalancer-961915002812.lb.foobar.eucalyptus-systems.com 
$ eulb-describe-lbs verbose 

4.5 - Manage IAM Resources

To manage Euare (IAM) resources on your Eucalyptus cloud, use the option with any command that describes, adds, deletes, or modifies resources. This option allows you to assume the role of the admin user for a given account. You can also use a policy to control and limit instances to specific availability zones. The following are some examples.

To list all groups in an account, enter the following command:

euare-grouplistbypath --as-account <account-name>

To list all users in an account, enter the following command:

euare-userslistbypath --as-account <account-name>

To delete the login profile of a user in an account, enter the following command:

euare-userdelloginprofile --as-account <account-name> -u <user_name>

To modify the login profile of a user in an account, enter the following command:

euare-usermod --as-account <account-name> -u <user_name> -n
<new_user_name>

To restrict an image to a specific availability zone, edit and attach this sample policy to a user:

{
    "Statement":[
      {
        "Effect":"Allow",
        "Action":"ec2:*",
         "Resource":"*"
      },
      {
        "Effect": "Deny",
        "Action": [ "ec2:*" ],
        "Resource": "arn:aws:ec2:::availabilityzone/PARTI00",
        "Condition": {
          "ArnLike": {
            "ec2:TargetImage": "arn:aws:ec2:*:*:image/emi-239D37F2"
          }
        }
      }
    ]
  }

To restrict a user to actions only within a specific availability zone, edit and attach this sample policy to a user:

{
    "Version": "2012-10-17",
    "Statement": [{
      "Effect": "Allow",
      "Action": [ "ec2:TerminateInstances" ],
      "Resource": "*",
      "Condition": {
        "StringEquals": {
          "ec2:AvailabilityZone": "PARTI00"
        }
      }
    }]
  }

To deny actions at the account level, edit and attach this example policy to an account:

{
    "Statement": [ {
      "Effect": "Deny",
      "Action": [ "ec2:RunInstances" ],
      "Resource": "arn:aws:ec2:::availabilityzone/PARTI00",
      "Condition": {
          "ArnLike": {
              "ec2:TargetImage": "arn:aws:ec2:*:*:image/emi-239D37F2"
          }
      }
    } ]
  }

4.6 - Manage Walrus Resources

This topic explains Walrus resources.

• Access Control Lists (ACLs) allow an account to explicitly grant access to a bucket or object to another account. ACLs only work between accounts, not IAM users. You specify accounts with the CanonicalID or the email address associated with the account (for Eucalyptus this is the email of the account admin).
• These are set by the admin of an account to control the access of users within that specific account. This is how an admin controls what users in that specific account are allowed to do. Policies can specify allow/deny on specific S3 operations (e.g. s3:GetObject, or s3:PutObject). IAM policies are set by sending the policy to the IAM (Euare) endpoint, not S3 (Walrus).
• These are IAM-like policies set by the bucket owner are not supported in Eucalyptus. For more information about bucket ACLs, go to Access Control List (ACL) Overview and Managing ACLs Using the REST API .

For more information about IAM policies, go to Using IAM Policies .

5 - Manage Regions

Manage Regions

This section provides information about regions and identity federation.

5.1 - Regions Overview

Eucalytpus provides support for the notion of federation of identity.Federation of identity information means that a Cloud Administrator can create a federation of (otherwise independent) Eucalyptus “clouds” where a Cloud User, using the same credentials as always, can use any of these federated Eucalyptus cloud regions. For the parts of Identify Access Management (IAM) and Security Token Service (STS) that Eucalyptus implements, the experience exposed to the Cloud User is the same as that seen by an AWS user working across AWS regions.

A user can interact with any region using the same credentials, subjected to the same policies, and having uniformly accessible and structured principals (Accounts, Users, Groups, Roles, etc.). The globality also includes the STS service functionality, the temporary credentials produced by the STS service also work globally.

Notably, this feature is restricted to IAM/STS and does not include other services which have pseudo-global characteristics, such as global bucket name space for S3. The following are general principles associated with regions:

• A region needs to be Registered as a federated region
• Registered regions should be discoverable via the EC2 DescribeRegions response
• A cloud user’s credentials should be accepted by any federated cloud
• There is a global IAM service (identities and policies are global for all registered regions)

5.2 - Region Configuration File Format

This section describes the necessary configuration properties that need to be addressed.For federation to be successfully configured, each cloud (i.e. region) that will be part of the federated cloud needs to have the following properties set (at a minimum):

5.3 - Examples

In this example, there will be two clouds used (10.111.5.32 and 10.111.1.1). Before setting up federation, the clouds must meet the following requirements:

• Eucalyptus installed
• Eucalyptus DNS enabled

5.4 - Federation Differences Between AWS and Eucalyptus

This section outlines the differences between AWS and Eucalyptus with respect to federation in the following platforms:

• Euca2ools vs. AWS EC2 API Tools
• Eucalyptus OSG vs. AWS S3
• Eucalyptus Resource-Level vs. AWS Resource-Level Permissions
• Global Cloud Administration (Local vs. Federated)

5.5 - Troubleshooting

This section is presented in a Q&A format to provide a quick reference to the most frequently asked questions.

1. Can Cloud Administrators federate existing clouds (i.e. clouds that already have non-system Eucalyptus accounts)? A. No, this is currently not supported. If a cloud administrator wants to federate an Eucalyptus clouds, this must be done prior to any non-system Eucalyptus account/user/group creation.

2. Is Eucalyptus DNS required for federating Eucalyptus clouds? A. No, however its highly recommended to enable it.

3. Are supported for more granular IAM access policies per region? A. As of 4.2, no. IAM policies apply globally (for all regions). In order to get more granular IAM access, use availability zone restrictions under each region. For more information, see Restrict Image to Availability Zone .

4. What services/resources span globally? Which span regionally? A. Currently, only Eucalyptus IAM and STS are global services/resources. All other services/resources are region-based (i.e. Eucalyptus cloud-specific). The only resource that can be either global or regional are keypairs. This is because users can import the same keypair to each region, therefore, the keypair is globally accessible. For additional information, please refer to the AWS EC2 Documentation regarding Resource Locations .

5. Are Eucalyptus system accounts global in a federated setup? A. No. Any Eucalyptus system account is limited to that region. Examples of Eucalyptus system accounts are as follows:

• eucalyptus
• (eucalyptus)blockstorage
• (eucalyptus)aws-exec-read
• (eucalyptus)cloudformation

1. Is and supported? A. No. There have been no improvements associated with Object Storage Gateway (OSG) regarding cross-regional behavior similar to AWS.

2. If a user uploads an object to an Object Storage Gateway in one region, will copies show up in other regions (similar to the behavior on AWS)? A. No, this is currently unsupported.

3. Do federated Eucalyptus clouds follow the same globally? A. No, Eucalyptus IAM limitations are regionally scoped.

6 - Manage Security

Manage Security

This section details concepts and tasks required to secure your cloud.

6.1 - Security Overview

This topic is intended for people who are currently using Eucalyptus and who want to harden the cloud and underlying configuration.

This topic covers available controls and best practices for securing your Eucalyptus cloud. Cloud security depends on security across many layers of infrastructure and technology:

• Security of the physical infrastructure and hosts
• Security of the virtual infrastructure
• Security of instances
• Security of storage and data
• Security of users and accounts

6.2 - Best Practices

Best Practices

This topic contains recommendations for hardening your Eucalyptus cloud.

6.2.1 - Authentication and Access Control Best Practices

This topic describes best practices for Identity and Access Management and the account.

Identity and Access Management

Eucalyptus manages access control through an authentication, authorization, and accounting system. This system manages user identities, enforces access controls over resources, and provides reporting on resource usage as a basis for auditing and managing cloud activities. The user identity organizational model and the scheme of authorizations used to access resources are based on and compatible with the AWS Identity and Access Management (IAM) system, with some Eucalyptus extensions provided that support ease-of-use in a private cloud environment.

For a general introduction to IAM in Eucalyptus, see Access Concepts in the IAM Guide. For information about using IAM quotas to enforce limits on resource usage by users and accounts in Eucalyptus, see the Quotas section in the IAM Guide.

The Amazon Web Services IAM Best Practices are also generally applicable to Eucalyptus.

Credential Management

Protection and careful management of user credentials (passwords, access keys, X.509 certificates, and key pairs) is critical to cloud security. When dealing with credentials, we recommend:

• Limit the number of active credentials and do not create more credentials than needed.
• Only create users and credentials for the interfaces that you will actually use. For example, if a user is only going to use the Management Console, do not create credentials access keys for that user.
• Use and or to get a specific set of credentials if needed.
• Regularly check for active credentials using commands and remove unused credentials. Ideally, only one pair of active credentials should be available at any time.
• Rotate credentials regularly and delete old credentials as soon as possible. Credentials can be created and deleted using commands, such as and .
• When rotating credentials, there is an option to deactivate, instead of removing, existing access/secret keys and X.509 certificates. Requests made using deactivated credentials will not be accepted, but the credentials remain in the Eucalyptus database and can be restored if needed. You can deactivate credentials using and .

Privileged Roles

The eucalyptus account is a super-privileged account in Eucalyptus. It has access to all cloud resources, cloud setup, and management. The users within this account do not obey IAM policies and compromised credentials can result in a complete cloud compromisation that is not easy to contain. We recommend limiting the use of this account and associated users’ credentials as much as possible.

For all unprivileged operations, use regular accounts. If you require super-privileged access (for example, management of resources across accounts and cloud setup administration), we recommend that you use one of the predefined privileged roles.

The Account, Infrastructure, and Resource Administrator roles provide a more secure way to gain super privileges in the cloud. Credentials returned by an assume-role operation are short-lived (unlike regular user credentials). Privileges available to each role are limited in scope and can be revoked easily by modifying the trust or access policy for the role.

6.2.2 - Hosts

This topic describes best practices for machines that host a Eucalyptus component.Eucalyptus recommends restricting physical and network access to all hosts comprising the Eucalyptus cloud, and disabling unused applications and ports on all machines used in your cloud.

After installation, no local access to Eucalyptus component hosts is required for normal cloud operations and all normal cloud operations can be done over remote web service APIs.

The user-facing services (UFS) and object storage gateway (OSG) are the only two components that generally expect remote connections from end users. Each Eucalyptus component can be put behind a firewall following the list of open ports and connectivity requirements described in the Configure the Firewall section.

For more information on securing Red Hat hosts, see the Red Hat Enterprise Linux Security Guide .

6.2.3 - Images and Instances

Because all instances are based on images, creating a secure image helps to create secure instances. This topic lists best practices that will add additional security during image creation. As a general rule, harden your images similar to how you would harden your physical servers.

• Turn off password-based authentication by specifying the following option in :
• Encourage non-root access by providing an unprivileged user account. If necessary, use sudo to allow access to privileged commands
• Always delete the shell history and any other potentially sensitive information before bundling. If you attempt more than one bundle upload in the same image, the shell history contains your secret access key.
• Bundling a running instance requires your private key and X.509 certificate. Put these and other credentials in a location that is not bundled (e.g. when using , pass the folder location where the certificates are stored as part of the values for the option). AWS provides more in-depth information on .
• Consider installing in the image to help control root and non-root access. If cloud-init isn’t available, a custom script can be used.
• Consider using a tool such as zerofree to zero-out any unused space on the image.
• Consider editing to clear out the swap every time the instance is booted. This can be done using the following command:
• Consider enabling or for your images
• Disable all unused services and ports on the image.
• By default, all images registered have private launch permissions. Consider using to limit the accounts that can access the image. After locking down the image using the steps above, additional steps can be done to further secure instances started from that image. For example, restrict access to the instance by allowing only trusted hosts or networks to access ports on your instances. You can control access to instances using euca-authorize and euca-revoke .

Consider creating one security group that allows external logins and keep the remainder of your instances in a group that does not allow external logins. Review the rules in your security groups regularly, and ensure that you apply the principle of least privilege: only open up permissions as they are required. Use different security groups to deal with instances that have different security requirements.

6.2.4 - Management Console

This topic describes things you can do to secure the Eucalyptus Management Console.

• Enable HTTPS for communications with the console and configure the console to use a CA-signed certificate.
• We do not recommend the “Remember my keys” option for “Login to AWS” because it stores AWS credentials in your browser’s local storage and increases the security risk of AWS credentials being compromised.
• Change the default session timeouts if needed. For more information, see .
• If you don’t use the Management Console, we recommend that you disable (using ). For more information, see .
• Turn off password autocomplete for the console by setting the configuration option to false in the console’s configuration file.
• If memcached is configured to be used by the console, make sure it’s not exposed publicly because there is no authentication mechanism enabled out of the box. If the default Eucalyptus-provided configuration is used, it accepts connections only from localhost.

6.2.5 - Message Security

This topic describes which networking mode is the most secure, and describes how to enforce message security.

Replay Detection

Eucalyptus components receive and exchange messages using either Query or SOAP interfaces (or both). Messages received over these interfaces are required to have a time stamp (as defined by AWS specification) to prevent message replay attacks. Because Eucalyptus enforces strict policies when checking timestamps in the received messages, for the correct functioning of the cloud infrastructure, it is crucial to have clocks constantly synchronized (for example, with ntpd) on all machines hosting Eucalyptus components. To prevent user commands failures, it is also important to have clocks synchronized on the client machines.

Following the AWS specification, all Query interface requests containing the Timestamp element are rejected as expired after 15 minutes of the timestamp. Requests containing the Expires element expire at the time specified by the element. SOAP interface requests using WS-Security expire as specified by the WS-Security Timestamp element.

Replay detection parameters can be tuned as described in Configure Replay Protection .

Endpoints

Eucalyptus requires that all user requests (SOAP with WS-Security and Query) are signed, and that their content is properly hashed, to ensure integrity and non-repudiation of messages. For stronger security, and to ensure message confidentiality and server authenticity, client tools and applications should always use SSL/TLS protocols with server certification verification enabled for communications with Eucalyptus components.

By default, Eucalyptus components are installed with self-signed certificates. For public Eucalyptus endpoints, certificates signed by a trusted CA provider should be installed.

6.2.6 - Networking Modes

This topic describes the recommendations for networking modes.A Eucalyptus deployment can be configured in EDGE (AWS EC2 Classic compatible) or VPCMIDO (AWS VPC compatible) networking modes. In both modes, by default, instances are not allowed to send traffic with spoofed IP and/or MAC addresses and receive traffic that are not destined to their own IP and/or MAC addresses. Security groups should be used to control the ingress traffic to instances (EDGE and VPCMIDO modes) and to control the egress traffic from instances (VPCMIDO mode).

VPCMIDO mode offers many security features not present in EDGE mode. Instances of different accounts are deployed in user-defined isolated networks within a Eucalyptus cloud. A combination of security features including VPC, VPC subnets, security groups, source/destination check configuration, route tables, internet gateways, and NAT gateways can be used to selectively enable and configure network access to/from instances or group of instances.

For more information about choosing a networking modes, see Plan Networking Modes .

6.3 - Security Tasks

Tasks

This section details the tasks needed to make your cloud secure.

6.3.1 - Configure SSL

Configure SSL

In order to connect to Eucalyptus using SSL, you must have a valid certificate for the User-Facing Services (UFS).

6.3.1.1 - Configure and Enable SSL for the Management Console

You can use secure HTTP for your console.To run your console over Secure HTTP:

Install nginx on your console server with the following command: yum install nginx Overwrite the default nginx.conf file with the template provided in /usr/share/doc/eucaconsole-/nginx.conf. cp /usr/share/doc/eucaconsole-/nginx.conf /etc/nginx/nginx.conf Uncomment the ’listen’ directive and uncomment/modify the SSL certificate paths in /etc/nginx/nginx.conf (search for “SSL configuration”). For example:

# SSL configuration
listen 443 ssl;
# ssl_certificate /path/to/ssl/pem_file;
# EXAMPLE:
ssl_certificate /etc/eucaconsole/console.crt;
# ssl_certificate_key /path/to/ssl/certificate_key;
# EXAMPLE: 
ssl_certificate_key /etc/eucaconsole/console.key;
# end of SSL configuration

session.secure = true
sslcert=/etc/eucaconsole/eucalyptus.com.chained.crt
sslkey=/etc/eucaconsole/eucalyptus.com.key

6.3.1.2 - Configure and Enable SSL for the UFS

This topic details tasks to configure SSL/TLS for the User-Facing Services (UFS)

If you have more than one host (other than node controllers), note the following:

• The keystore must be updated on each host running the eucalyptus-cloud service
• The [key_alias] must be the same on each host
• Use a wildcard certificate (i.e. *.<system.dns.dnsdomain>), since UFS is responsible for all service API endpoints

Create a Keystore

Eucalyptus uses a PKCS12-format keystore. If you are using a certificate signed by a trusted root CA, perform the following steps.

Enter the following command to convert your trusted certificate and key into an appropriate format:

openssl pkcs12 -export -in [YOURCERT.crt] -inkey [YOURKEY.key] \
  -out tmp.p12 -name [key_alias]

Save a backup of the Eucalyptus keystore, at /var/lib/eucalyptus/keys/euca.p12 . Import your keystore into the Eucalyptus keystore on the UFS:

keytool -importkeystore -srckeystore tmp.p12 -srcstoretype pkcs12 \
  -srcstorepass [export_password] -destkeystore /var/lib/eucalyptus/keys/euca.p12 \
  -deststoretype pkcs12 -deststorepass eucalyptus -alias [key_alias] -destkeypass eucalyptus

Enable the UFS to Use the Keystore

To enable the UFS to use the keystore, perform the following steps in the CLC because the UFS gets all its configuration information from the CLC. Run the following commands on the CLC:

euctl bootstrap.webservices.ssl.server_alias=[key_alias]

Optional: Redirect Requests to use Port 443

To allow user facing services requests on port 443 instead of the default 8773, run the following commands on the CLC:

euctl bootstrap.webservices.port=443

6.3.2 - Change Multicast Address

This topic describes how to change your multicast address for group membership.By default, Eucalyptus uses the multicast address 239.193.7.3 for group membership. Most data centers limit multicast address communication for security measures. We recommend that you use addresses in the administratively-scoped multicast address range.

To change the multicast address for group membership Stop all services, starting from the CC, SC, Walrus, then CLC. For example:

systemctl stop eucalyptus-cluster.service
systemctl stop eucalyptus-cloud.service

Change the eucalyptus.conf on the CC, modifying the CLOUD_OPTS parameter to the new IP address:

CLOUD_OPTS="--mcast-addr=228.7.7.3"

systemctl start eucalyptus-cloud.service
systemctl start eucalyptus-cluster.service

Verify that the configured multicast address is being used via netstat:

netstat -nulp

Postrequisites

• Check the firewall after changing the multicast address. See for more information.

6.3.3 - Configure Replay Protection

You can configure replay detection in Java components (which includes the CLC, UFS, OSG, Walrus, and SC) to allow replays of the same message for a set time period.

euctl bootstrap.webservices.replay_skew_window_sec=[new_value_in_seconds]

If you set this property to 0 , Eucalyptus will not allow any message replays. This setting provides the best protection against message replay attacks.

If you set this property to any value greater than 15 minutes plus the values of ws.clock_skew_sec (that is, to a value >= 920 sec in the default installation), Eucalyptus disables replay detection completely.

When checking message timestamps for expiration, Eucalyptus allows up to 20 seconds of clock drift between the machines. This is a default setting. You can change this value for the Java components at runtime by setting the bootstrap.webservices.clock_skew_sec property as follows:

euctl bootstrap.webservices.clock_skew_sec=[new_value_in_seconds]

6.3.4 - Configure Session Timeouts

To set the session timeouts in the Management Console:

Modify the session.timeout and session.cookie_expires entries in the [app:main] section of the configuration file. The session.timeout value defines the number of seconds before an idle session is timed out. The session.cookie_expires is the maximum length that any session can be active before being timed out. All values are in seconds:

session.timeout=1800



session.cookie_expires=43200

6.3.5 - Configure STS Actions

The Security Token Service (STS) allows you to enable or disable specific token actions.By default, the enabled actions list is empty. However, this means that all actions are enabled. To disable actions, list each action in the disabledactions property. To enable specific actions, list them in the enabledactions property.

# euctl tokens
PROPERTY	tokens.disabledactions	{}
PROPERTY	tokens.enabledactions	{}

The values for each property are case-insensitive, space or comma-separated lists of token service actions. If an action is in the disable list it will not be permitted. Eucalyptus returns an HTTP status 503 and the code ServiceUnavailable .

If the enable list is not empty, Eucalyptus only permits the actions specifically listed.

For more information about STS, go to STS section of the AWS CLI Reference .

6.3.6 - Configure the Firewall

Restricting Network Access

This section provides basic guidance on setting up a firewall around your Eucalyptus components. It is not intended to be exhaustive.

On the Cloud Controller (CLC), Walrus, and Storage Controller (SC), allow for the following jGroups traffic:

• TCP connections between CLC, user-facing services (UFS), object storage gateway (OSG), Walrus, and SC on port 8779 (or the first available port in range 8779-8849)

• UDP connections between CLC, UFS, OSG, Walrus, and SC on port 7500

• Multicast connections between CLC and UFS, OSG, Walrus, and SC to IP 239.193.7.3 on UDP port 8773 On the UFS, allow the following connections:

• TCP connections from end-users and instances on ports 8773

• End-user and instance connections to DNS ports On the CLC, allow the following connections:

• TCP connections from UFS, CC and Eucalyptus instances (public IPs) on port 8773 (for metadata service)

• TCP connections from UFS, OSG, Walrus, and SC on port 8777 On the CC, make sure that all firewall rules are compatible with the dynamic changes performed by Eucalyptus, described in the section below. Also allow the following connections:

• TCP connections from CLC on port 8774 On OSG, allow the following connections:

• TCP connections from end-users and instances on port 8773

• TCP connections from SC and NC on port 8773 On Walrus, allow the following connections:

• TCP connections from OSG on port 8773 On the SC, allow the following connections:

• TCP connections from CLC and NC on TCP port 8773

• TCP connections from NC on TCP port 3260, if tgt (iSCSI open source target) is used for EBS in DAS or Overlay modes On the NC, allow the following connections:

• TCP connections from CC on port 8775

• TCP connections from other NCs on port 16514

• DHCP traffic forwarding to VMs

• Traffic forwarding to and from instances’ private IP addresses

6.3.7 - Reserve Ports

6.3.8 - Synchronize Components

To synchronize your Eucalyptus component machines with an NTP server, perform the following tasks.

Enter the following command on a machine hosting a Eucalyptus component:

# ntpdate pool.ntp.org
# systemctl start ntpd.service
# systemctl enable ntpd.service
# ps ax | grep ntp
# hwclock --systohc  

Repeat for each machine hosting a Eucalyptus component.

7 - Eucalyptus Commands

Eucalyptus Commands

This section contains reference information for Eucalyptus administration commands.

7.1 - Eucalyptus Administration Commands

Eucalyptus Administration Commands

Eucalyptus offers commands for common administration tasks and inquiries. This section provides a reference for these commands.

7.1.1 - euctl

Syntax

euctl [-Anr] [-d | -s] NAME ...
euctl [-nq] NAME=VALUE ...
euctl [-nq] NAME=@FILE ...
euctl --dump [--format {raw,json,yaml}] NAME
euctl --edit [--format {raw,json,yaml}] NAME

Positional Arguments

Options

Examples

When retrieving a variable, a subset of the MIB name may be specified to retrieve a list of variables in that subset. For example, to list all the dns variables:

euctl dns

This replaces euca-describe-properties .

When setting a variable, the MIB name should be followed by an equal sign and the new value:

euctl dns.enabled=true

This replaces euca-modify-property -p .

To write variables using the contents of the files as their new values rather than typing them into the command line, follow them with =@ and those file names:

euctl cloud.network.network_configuration=@/etc/eucalyptus/network.yaml

This replaces euca-modify-property -f .

Specify a filename to read the values from a file:

myproperty=@myvaluefile

It is possible to read or write more than one variable in a single invocation of euctl . Just separate them with spaces:

euctl one=1 two=2 three four=@4.txt five

In all of these cases, euctl will generally output each variable named on its command line, along with its current (and potentially just-changed) value. For example, the output of the command above could be:

one = 1 
two = 2
three = 3
four = 4
five = 5

To reset a variable to its default value, specify the -r option:

euctl -r dns.enabled

The information available from euctl consists of integers, strings, and structures. The structured information can only be retrieved by specialized programs and, in some cases, this program’s --edit and --dump options.

7.1.2 - euserv-deregister-service

Syntax

euserv-deregister-service [-U URL] [--region USER@REGION] [-I KEY_ID]

       [-S KEY] [--security-token TOKEN] [--debug]
              [--debugger] [--version] [-h] SVCINSTANCE

Positional Arguments

Output

Eucalyptus returns a message stating that service instance was successfully de-registered.

Example

To de-register the dns service named “API_10.111.1.44.dns”:

euserv-deregister-service API_10.111.1.44.dns

7.1.3 - euserv-describe-events

Syntax

euserv-describe-events [-s] [-f FORMAT]

Description

Events come in the form of a list, where each event contains one or more of the following tags:

Environment

Options

Output

There are several built-in formats, and you can define additional formats using a format: string , as described below. Here are the details of the built-in formats:

yaml This outputs block-style YAML designed to be easily readable. Tags that are empty or not defined do not appear in this output at all.

events:
            - timestamp: {timestamp}
            severity: {severity}
            id: {id}
            subject-type: {subject-type}
            subject-name: {subject-name}
            subject-host: {subject-host}
            subject-arn: {subject-arn}
            message: |-
            {message}
            stack-trace: |-
            {stack-trace}

oneline This output is designed to be as compact as possible.

{timestamp} {severity} {subject-type} {subject-name} {message}

format:string The format: string format allows you to specify which information you want to show using placeholders enclosed in curly braces to indicate where to show the tags for each event. For example:

euserv-describe-events -f "format:{timestamp} {subject-name} {message}"

Example

To output a list of service-affecting events in the oneline format:

euserv-describe-events --format oneline
2016-06-20 16:16:08 INFO node 10.111.1.15 the node is operating normally\nFound service status for 10.111.1.15: ENABLED
2016-06-27 17:37:57 ERROR node 10.111.5.50 Error occurred in transport
2016-06-28 07:00:17 ERROR node 10.111.5.50

7.1.4 - euserv-describe-node-controllers

Syntax

euserv-describe-node-controllers [--ec2-url URL] [--show-headers]

   [--show-empty-fields] [-U URL]
          [--region USER@REGION] [-I KEY_ID]  [-S  KEY]  [--security-token
          TOKEN] [--debug] [--debugger] [--version] [-h]

Options

Output

Eucalyptus returns information about the node controller and its instances, for example:

NODE  one  10.111.1.53  enabled    
INSTANCE  i-162a8f09      
INSTANCE  i-2b6cdd10      
NODE  one  10.111.5.132  enabled    
INSTANCE  i-ba9307d7

Example

euserv-describe-node-controllers --region localhost

7.1.5 - euserv-describe-service-types

Syntax

euserv-describe-service-types [-a] [--show-headers]

[--show-empty-fields] [-U URL]
   [--region  USER@REGION]  [-I  KEY_ID]  [-S  KEY] [--security-token TOKEN]
   [--debug] [--debugger] [--version] [-h]

Options

Output

Eucalyptus returns a list of service types.

Example

euserv-describe-service-types 
SVCTYPE  arbitrator                The Arbitrator service                                      
SVCTYPE  autoscaling     user-api  Auto Scaling API service                                    
SVCTYPE  cloudformation  user-api  Cloudformation API service                                  
SVCTYPE  cloudwatch      user-api  CloudWatch API service                                      
SVCTYPE  cluster                   The Cluster Controller service                              
SVCTYPE  compute         user-api  the Eucalyptus EC2 API service                              
SVCTYPE  dns             user-api  Eucalyptus DNS server                                       
SVCTYPE  euare           user-api  IAM API service                                             
SVCTYPE  eucalyptus                eucalyptus service implementation                           
SVCTYPE  identity        user-api  Eucalyptus identity service                                 
SVCTYPE  imaging         user-api  Eucalyptus imaging service                                  
SVCTYPE  loadbalancing   user-api  ELB API service                                             
SVCTYPE  objectstorage   user-api  S3 API service                                              
SVCTYPE  simpleworkflow  user-api  Simple Workflow API service                                 
SVCTYPE  storage                   The Storage Controller service                              
SVCTYPE  tokens          user-api  STS API service                                             
SVCTYPE  user-api                  The service group of all user-facing API endpoint services  
SVCTYPE  walrusbackend             The legacy Walrus Backend service

7.1.6 - euserv-describe-services

Syntax

euserv-describe-services [-a]

       [--group-by-type | --group-by-zone | --group-by-host | --expert]
         [--show-headers]   [--show-empty-fields]   [-U   URL]  [--region
         USER@REGION]  [-I  KEY_ID]  [-S  KEY]  [--security-token  TOKEN]
         [--filter  NAME=VALUE]  [--debug]  [--debugger] [--version] [-h]
         [SVCINSTANCE [SVCINSTANCE ...]]

Positional Arguments

Options

Output

Eucalyptus returns information about the services you specified.

Example

Verify that you are looking at the cloud controllers view of the service state by explicitly running against that host:

euserv-describe-services --filter service-type=storage -U http://localhost:8773/services/Empyrean
SERVICE  storage  one  one-sc-1  enabled

7.1.7 - euserv-migrate-instances

Syntax

euserv-migrate-instances (-s HOST | -i INSTANCE)

  [--include-dest HOST | --exclude-dest HOST]
  [-U URL] [--region USER@REGION] [-I KEY_ID]  [-S  KEY]  [--security-token
  TOKEN] [--debug] [--debugger] [--version] [-h]

Options

Output

Unless requested, no output is given. You can run the euserv-describe-* command to verify that the migration activity completed successfully, as shown in the example following.

Example

To migrate an instance from its current host:

euserv-migrate-instances -i i-8eacd211 
euserv-describe-node-controllers 
NODE  zone-555  10.104.1.200  enabled    

NODE  zone-555  10.104.1.201  enabled    
INSTANCE  i-8eacd211      

7.1.8 - euserv-modify-service

Syntax

euserv-modify-service -s STATE [-U URL] [--region USER@REGION]

   [-I KEY_ID] [-S KEY] [--security-token TOKEN]
   [--debug] [--debugger] [--version] [-h] SVCINSTANCE

Positional Arguments

Options

Output

No output is given. You can run the euserv-describe-services command to verify that the modification completed successfully, as shown in the example following.

Example

To modify the state of a storage controller service named “two-sc-1” to stopped:

euserv-modify-service -s stopped two-sc-1
euserv-describe-services two-sc-1
SERVICE  storage  two  two-sc-1  stopped  

7.1.9 - euserv-register-service

Syntax

euserv-register-service -t TYPE -h IP [--port PORT] [-z ZONE] [-U URL]

       [--region USER@REGION] [-I KEY_ID] [-S KEY]
           [--security-token   TOKEN]  [--debug]  [--debugger]  [--version]
           [--help] SVCINSTANCE

Positional Arguments

Options

Output

No output is given when it succeeds.

Example

To register the ufs service named “user-api-5”:

euserv-register-service -t user-api -h 10.0.0.15 user-api-5

7.2 - Eucalyptus Configuration Variables

Eucalyptus exposes a number of variables that can be configured using the command. This topic explains what types of variables Eucalyptus uses, and lists the most common configurable variables.

Eucalyptus Variable Types

Eucalyptus uses two types of variables: ones that can be changed (as configuration options), and ones that cannot be changed (they are displayed as variables, but configured by modifying the eucalyptus.conf file on the CC).

Eucalyptus Variables

The following table contains a list of common Eucalyptus cloud variables.

8 - Advanced Storage Configuration

Advanced Storage Configuration

This section covers advanced storage provider configuration options.

8.1 - OSG Advanced Configuration

The following properties are for tuning the behavior of the Object Storage service and Gateways; the defaults are reasonable and changing is not necessary, but they are available for unexpected situations.