After you install and initially configure Eucalyptus, there are some common administration tasks you can perform. This section describes these tasks and associated concepts.

1 - Cloud Overview

This topic presents an overview of the components in Eucalyptus. Eucalyptus is comprised of several components: Cloud Controller, Walrus, Cluster Controller, Storage Controller, and Node Controller. Each component is a stand-alone web service. This architecture allows Eucalyptus both to expose each web service as a well-defined, language-agnostic API, and to support existing web service standards for secure communication between its components.

Cloud Controller

The Cloud Controller (CLC) is the entry-point into the cloud for administrators, developers, project managers, and end-users. The CLC queries other components for information about resources, makes high-level scheduling decisions, and makes requests to the Cluster Controllers (CCs). As the interface to the management platform, the CLC is responsible for exposing and managing the underlying virtualized resources (servers, network, and storage). You can access the CLC through command line tools that are compatible with Amazon’s Elastic Compute Cloud (EC2).

Walrus

Walrus allows users to store persistent data, organized as buckets and objects. You can use Walrus to create, delete, and list buckets, or to put, get, and delete objects, or to set access control policies. Walrus is interface compatible with Amazon’s Simple Storage Service (S3). It provides a mechanism for storing and accessing virtual machine images and user data. Walrus can be accessed by end-users, whether the user is running a client from outside the cloud or from a virtual machine instance running inside the cloud.

Cluster Controller

The Cluster Controller (CC) generally executes on a machine that has network connectivity to both the machines running the Node Controller (NC) and to the machine running the CLC. CCs gather information about a set of NCs and schedules virtual machine (VM) execution on specific NCs. The CC also manages the virtual machine networks. All NCs associated with a single CC must be in the same subnet.

Storage Controller

The Storage Controller (SC) provides functionality similar to the Amazon Elastic Block Store (Amazon EBS). Elastic block storage exports storage volumes that can be attached by a VM and mounted or accessed as a raw block device. EBS volumes persist past VM termination and are commonly used to store persistent data. An EBS volume cannot be shared between VMs and can only be accessed within the same availability zone in which the VM is running. Users can create snapshots from EBS volumes. Snapshots may be stored in Walrus and made available across availability zones.

Node Controller

The Node Controller (NC) executes on any machine that hosts VM instances. The NC controls VM activities, including the execution, inspection, and termination of VM instances. It also fetches and maintains a local cache of instance images, and it queries and controls the system software (host OS and the hypervisor) in response to queries and control requests from the CC. The NC is also responsible for the management of the virtual network endpoint.

2 - Cloud Best Practices

Cloud Best Practices

This section details Eucalyptus best practices for your private cloud.

2.1 - Synchronize Clocks

Eucalyptus checks message timestamps across components in the cloud infrastructure. This assures command integrity and provides better security.Eucalyptus components receive and exchange messages using either Query or SOAP interfaces (or both). Messages received over these interfaces are required to have some form of a time stamp (as defined by AWS specification) to prevent message replay attacks. Because Eucalyptus enforces strict policies when checking timestamps in the received messages, for the correct functioning of the cloud infrastructure, it is crucial to have clocks constantly synchronized (for example, with ntpd) on all machines hosting Eucalyptus components. To prevent user command failures, it is also important to have clocks synchronized on the client machines.

Following the AWS specification, all Query interface requests containing the Timestamp element are rejected as expired after 15 minutes of the timestamp. Requests containing the Expires element expire at the time specified by the element. SOAP interface requests using WS-Security expire as specified by the WS-Security Timestamp element.

When checking the timestamps for expiration, Eucalyptus allows up to 20 seconds of clock drift between the machines. This is a default setting. You can change this value for the CLC at runtime by setting the bootstrap.webservices.clock_skew_sec property as follows:

euctl bootstrap.webservices.clock_skew_sec=<new_value_in_seconds>

For additional protection from the message replay attacks, the CLC implements a replay detection algorithm and rejects messages with the same signatures received within 15 minutes. Replay detection parameters can be tuned as described in Configure Replay Protection .

2.2 - Configure SSL

In order to connect to Eucalyptus using SSL/TLS, you must have a valid certificate for the Cloud Controller (CLC)

If you have more than one host (other than node controllers), note the following:

The keystore must be updated on each host running the eucalyptus-cloud service
The [key_alias] must be the same on each host
Use a wildcard certificate (i.e. *.<system.dns.dnsdomain>), since UFS is responsible for all service API endpoints

Create a keystore

Eucalyptus uses a PKCS12-format keystore. If you are using a certificate signed by a trusted root CA, use the following command to convert your trusted certificate and key into an appropriate format:

openssl pkcs12 -export -in [YOURCERT.crt] -inkey [YOURKEY.key] \
  -out tmp.p12 -name [key_alias]

Note

The above command will request an export password, which is used in the following steps.

Save a backup of the Eucalyptus keystore, at /var/lib/eucalyptus/keys/euca.p12 , and then import your keystore into the Eucalyptus keystore as follows:

keytool -importkeystore \
  -srckeystore tmp.p12 -srcstoretype pkcs12 -srcstorepass [export_password] \
  -destkeystore /var/lib/eucalyptus/keys/euca.p12 -deststoretype pkcs12 \
  -deststorepass eucalyptus -alias [key_alias] \
  -srckeypass [export_password]

Enable the Cloud Controller to use this keystore

Run the following commands on the Cloud Controller (CLC):

euctl bootstrap.webservices.ssl.server_alias=[key_alias]

Optional: Redirect Requests to use Port 443

To allow user facing services requests on port 443 instead of the default 8773, run the following commands on the CLC:

euctl bootstrap.webservices.port=443

2.3 - Storage Volumes

Eucalyptus manages storage volumes for your private cloud. Volume management strategies are application specific, but this topic includes some general guidelines.When setting up your Storage Controller, consider whether performance (bandwidth and latency of read/write operations) or availability is more important for your application. For example, using several smaller volumes will allow snapshots to be taken on a rolling basis, decreasing each snapshot creation time and potentially making restore operations faster if the restore can be isolated to a single volume. However, a single larger volume allows for faster read/write operations from the VM to the storage volume.

An appropriate network configuration is an important part of optimizing the performance of your storage volumes. For best performance, each Node Controller should be connected to a distinct storage network that enables the NC to communicate with the SC or Ceph, without interfering with normal NC/VM-instance network traffic.

Eucalyptus includes configurable limits on the size of a single volume, as well as the aggregate size of all volumes on an SC. The SC can push snapshots from Ceph, where the volumes reside, to object storage, where the snapshots become available across multiple clusters. Smaller volumes will be much faster to snapshot and transfer, whereas large volumes will take longer. However, if many concurrent snapshot requests are sent to the SC, operations may take longer to complete.

EBS volumes are created from snapshots on the SC or Ceph, after the snapshot has been downloaded from object storage to the device. Creating an EBS volume from a snapshot on the same cluster as the source volume of the snapshot will reduce delays caused by having to transfer snapshots from object storage.

3 - Cloud Tasks

Cloud Tasks

This section contains a listing of your Eucalyptus cloud-related tasks.

3.1 - Inspect System Health

Eucalyptus provides access to the current view of service state and the ability to manipulate the state. You can inspect the service state to either ensure system health or to identify faulty services. You can modify a service state to maintain activities and apply external service placement policies.

View Service State

Use the euserv-describe-services command to view the service state. The output indicates:

Component type of the service
Partition in which the service is registered
Unique name of the service
Current view of service state
Last reported epoch (this can be safely ignored)
Service URI
Fully qualified name of the service (This is needed for manipulating services that did not get unique names during registration. For example: internal services like DNS) The default output includes the services that are registered during configuration, as well as information about the DNS service, if present. You can obtain additional service state information, such as internal services, by providing the -a flag.

You can also make requests to retrieve service information that is filtered by either:

current state (for example, )
host where service is registered
partition where service is registered
type of service (for example, CC or Walrus) When you investigate service failures, you can specify -events to return a summary of the last fault. You can retrieve extended information (primarily useful for debugging) by specifying -events -events-verbose .

Heartbeat Service

http://CLCIPADDRESS:8773/services/Heartbeat provides a list of components and their respective statuses. This allows you to find out if a service is enabled without requiring cloud credentials.

Modify Service State

To modify a service:

Enter the following command on the CLC, Walrus, or SC machines:

systemctl stop eucalyptus-cloud.service

On the CC, use the following command:

systemctl stop eucalyptus-cluster.service

If you want to shut down the SC for maintenance. The SC is SC00 is ENABLED and needs to be DISABLED for maintenance.

To stop SC00 first verify that no volumes or snapshots are being created and that no volumes are being attached or detached, and then enter the following command on SC00:

systemctl stop eucalyptus-cloud.service

To check status of services, you would enter:

euserv-describe-services

When maintenance is complete, you can start the eucalyptus-cloud process on SC00 , which will enter the DISABLED state by default.

systemctl start eucalyptus-cloud.service

Monitor the state of services using euserv-describe-services until SC00 is ENABLED .

3.2 - View User Resources

To see resource use by your cloud users, Eucalyptus provides the following commands with the flag.

: Returns information about security groups in your account, including output type identifier, security group ID, security group name, security group description, output type identifier, account ID of the group owner, name of group granting permission, type of rule, protocol to allow, start of port range, end of port range, source (for ingress rules) or destination (for egress rules), and any tags assigned to the security group.
: Returns information about your instances, including output type identifier, reservation ID, name of each security group the instance is in, output type identifier, instance ID for each running instance, EMI ID of the image on which the instance is based, public DNS name associated with the instance (for instances in the running state), private DNS name associated with the instance (for instances in running state), instance state, key name, launch index, instance type, launch time, availability zone, kernel ID, ramdisk ID, monitoring state, public IP address, private IP address, type of root device (ebs or instance-store), placement group the cluster instance is in, virtualization type (paravirtual or hvm), any tags assigned to the instance, hypervisor type, block device identifier for each EBS volume the instance is using, along with the device name, the volume ID, and the timestamp.
: Returns information about key pairs available to you, including keypair identifier, keypair name, and private key fingerprint.
: Returns information about EBS snapshots available to you, including snapshot identifier, ID of the snapshot, ID of the volume, snapshot state (pending, completed, error), timestamp when snapshot initiated, percentage of completion, ID of the owner, volume sized, description, and any tags assigned to the snapshot.
: Describes your EBS volumes, including volume identifier, volume ID, size of the volume in GiBs, snapshot from which the volume was created, availability zone, volume state (creating, available, in-use, deleting, deleted, error), timestamp of the volume creation, and any tags assigned to the volume.

3.3 - Change Network Configuration

Change Network Configuration

You might want to change the original network configuration of your cloud. To change your network configuration, perform the tasks listed in this topic.Log in to the CLC and open the /etc/eucalyptus/eucalyptus.conf file. Navigate to the Networking Configuration section and make your edits. Save the file. Restart the Cluster Controller.

systemctl restart eucalyptus-cluster.service

3.3.1 - Networking Configuration Options

All network-related options specified in /etc/eucalyptus/eucalyptus.conf use the prefix VNET_. The most commonly used VNET options are described in the following table.

Note

If you change the value of in the file, you must restart the Cluster Controller.

Option	Description	Component
VNET_BRIDGE	This is the name of the bridge interface to which instances’ network interfaces should attach. A physical interface that can reach the CC must be attached to this bridge. Common setting for KVM is br0.	Node Controller
VNET_DHCPDAEMON	The ISC DHCP executable to use. This is set to a distro-dependent value by packaging. The internal default is /usr/sbin/dhcpd3.	Node Controller
VNET_MODE	The networking mode in which to run. The same mode must be specified on all CCs and NCs in your cloud. Valid values: EDGE	All CCs and NCs
VNET_PRIVINTERFACE	The name of the network interface that is on the same network as the NCs. Default: eth0	Node Controller
VNET_PUBINTERFACE	This is the name of the network interface that is connected to the same network as the CC. Depending on the hypervisor’s configuration this may be a bridge or a physical interface that is attached to the bridge. Default: eth0	Node Controller

3.4 - Add a Node Controller

If you want to increase your system’s capacity, you’ll want to add more Node Controllers (NCs).To add an NC, perform the following tasks:

clusteradmin-register-nodes node0_IP_address ... [nodeN_IP_address]

When prompted, enter the password to log into each node. Eucalyptus requires this password to propagate the cryptographic keys.

3.5 - Migrate Instances Between Node Controllers

In order to ensure optimal system performance, or to perform system maintenance, it is sometimes necessary to move running instances between Node Controllers (NCs). You can migrate instances individually, or migrate all instances from a given NC.

Note

For migrations to succeed, you must have set to the same value in the file on each NC.

To migrate a single instance to another NC, enter the following command:

euserv-migrate-instances -i INSTANCE_ID

You can also optionally specify --include-dest HOST_NC_IP or --exclude-dest HOST_NC_IP , to ensure that the instance is migrated to one of the specified NCs, or to avoid migrating the instance to any of the specified NCs. These flags may be used more than once to specify multiple NCs.

To migrate all instances away from an NC, enter the following command:

euserv-migrate-instances --source HOST_NC_IP

You can also optionally specify euserv-modify-service -s stop HOST_NC_IP , to stop the specified NC and ensure that no new instances are started on that NC while the migration occurs. This allows you to safely remove the NC without interrupting running instances. The NC will remain in the DISABLED state until it is explicitly enabled using euserv-modify-service -s start HOST_NC_IP .

In some cases, timeouts may cause a migration to initially fail. Run the command again to complete the migration.

If the migration fails, check the nc.log file on the source and destination NCs. If you see an error similar to:

libvirt: Cannot get interface MTU on 'br0': No such device (code=38)

… then ensure the NCs have the same interface and bridge device names, as described in .

3.6 - Remove a Node Controller

Describes how to delete NCs in your system.If you want to decrease your system’s capacity, you’ll need to decrease NC servers. To delete an NC, perform the following tasks.

clusteradmin-deregister-nodes node0_IP_address ... [nodeN_IP_address]

3.7 - Restart Eucalyptus

Restart Eucalyptus

Describes the recommended processes to restart Eucalyptus, including terminating instances and restarting Eucalyptus components.You must restart Eucalyptus whenever you make a physical change (e.g., switch out routers), or edit the eucalyptus.conf file. To restart Eucalyptus, perform the following tasks in the order presented.

Note

Before you restart Eucalyptus, we recommend that you notify all users that you are terminating all instances.

3.7.1 - Shut Down All Instances

To terminate all instances on all NCs perform the steps listed in this topic. To terminate all instances on all NCs:

Enter the following command:

euca-terminate-instances <instance_id>

3.7.2 - Restart the CLC

systemctl restart eucalyptus-cloud.service

All Eucalyptus components on this server will restart.

3.7.3 - Restart Walrus

systemctl restart eucalyptus-cloud.service

3.7.4 - Restart the CC

systemctl restart eucalyptus-cluster.service

3.7.5 - Restart the SC

systemctl restart eucalyptus-cloud.service

3.7.6 - Restart an NC

To restart an NC perform the steps listed in this topic.Log in to the NC and enter the following command:

systemctl restart eucalyptus-node.service

Repeat for each NC. Verify that the following is even needed. If so, replicate for other NC-tasks. You can automate the restart command for all of your NCs. Store a list of your NCs in a file called nc-hosts that looks like:

nc-host-00
nc-host-01
...
nc-host-nn

To restart all of your NCs, run the following command:

cat nc-hosts | xargs -i ssh root@{} systemctl restart eucalyptus-node.service

3.8 - Shut Down Eucalyptus

Shut Down Eucalyptus

Describes the recommended processes to shut down Eucalyptus.There may be times when you need to shut down Eucalyptus. This might be because of a physical failure, topological change, backing up, or making an upgrade. We recommend that you shut down Eucalyptus components in the reverse order of how you started them. To stop the system, shut down the components in the order listed.

Note

Before you shut Eucalyptus down, we recommend that you notify all users that you are terminating all instances.

3.8.1 - Shut Down All Instances

To terminate all instances on all NCs perform the steps listed in this topic.To terminate all instances on all NCs:

Enter the following command:

euca-terminate-instances <instance_id>

3.8.2 - Shut Down the NCs

To shut down the NCs perform the steps listed in this topic.To shut down the NCs:

systemctl stop eucalyptus-node.service

Repeat for each machine hosting an NC.

3.8.3 - Shut Down the CCs

To shut down the CCs:

systemctl stop eucalyptus-cluster.service

Repeat for each machine hosting a CC.

3.8.4 - Shut Down the SCs

To shut down the SC:

systemctl stop eucalyptus-cloud.service

Repeat for any other machine hosting an SC.

3.8.5 - Shut Down Walrus

To shut down Walrus:

systemctl stop eucalyptus-cloud.service

3.8.6 - Shut Down the CLC

To shut down the CLC:

systemctl stop eucalyptus-cloud.service

3.9 - Disable CloudWatch

To disable CloudWatch, run the following command.

euctl cloudwatch.enable_cloudwatch_service=true