I’m happy to announce that MAAS2.4.0beta 2 is now released and is available for Ubuntu Bionic.
MAAS2.4.0beta 2 is currently available in Bionic’s Archive or in the following PPA:
MAAS 2.4.0 (beta2)
New Features & Improvements
MAAS Internals optimisation
Continuing with MAAS’ internal surgery, a few more improvements have been made:
Improve the image download process, to ensure rack controllers immediately start image download after the region has finished downloading images.
Reduce the service monitor interval to 30 seconds. The monitor tracks the status of the various services provided alongside MAAS (DNS, NTP, Proxy).
UI Performance optimizations for machines, pods, and zones, including better filtering of node types.
KVM pod improvements
Continuing with the improvements for KVM pods, beta 2 adds the ability to:
Define a default storage pool
This feature allows users to select the default storage pool to use when composing machines, in case multiple pools have been defined. Otherwise, MAAS will pick the storage pool automatically depending which pool has the most available space.
API – Allow allocating machines with different storage pools
Allows users to request a machine with multiple storage devices from different storage pools. This feature uses storage tags to automatically map a storage pool in libvirt with a storage tag in MAAS.
Remove remaining YUI in favor of AngularJS.
As of beta 2, MAAS has now fully dropped the use of YUI for the Web Interface. The last section using YUI was the Settings page and the login page. Both sections have now been transitioned to use AngularJS instead.
Re-organize Settings page
The MAAS settings have now been reorganized into multiple tabs.
API for default DNS domain selection
Adds the ability to define the default DNS domain. This is currently only available via the API.
Vanilla framework upgrade
We would like to thank the Ubuntu web team for their hard work upgrading MAAS to the latest version of the Vanilla framework. MAAS is looking better and more consistent every day!
Please refer to the following for all 37 bug fixes in this release, which address issues with MAAS across the board:
I’m happy to announce that MAAS2.4.0 beta 1 and python-libmaas 0.6.0 have now been released and are available for Ubuntu Bionic.
MAAS2.4.0 beta 1 is currently available in Bionic -proposed waiting to be published into Ubuntu, or in the following PPA:
MAAS 2.4.0 (beta1)
Debian package maas-dns no longer needed
The Debian package ‘maas-dns’ has now been made a transitional package. This package provided some post-installation configuration to prepare bind to be managed by MAAS, but it required maas-region-api to be installed first.
In order to streamline the installation and make it easier for users to install HA environments, the configuration of bind has now been integrated to the ‘maas-region-api’ package itself, which and we have made ‘maas-dns’ a dummy transitional package that can now be removed.
New Features & Improvements
MAAS Internals optimization
Major internal surgery to MAAS 2.4 continues improve various areas not visible to the user. These updates will advance the overall performance of MAAS in larger environments. These improvements include:
Database query optimizations
Further reductions in the number of database queries, significantly cutting the queries made by the boot source cache image import process from over 100 to just under 5.
MAAS is being optimized to reduce the amount of data using the websocket API to render the UI. This is targeted at only processing data only for viewable information, improving various legacy areas. Currently, the work done for this release includes:
Only load historic script results (e.g. old commissioning/testing results) when requested / accessed by the user, instead of always making them available over the websocket.
Only load node objects in listing pages when the specific object type is requested. For instance, only load machines when accessing the machines tab instead of also loading devices and controllers.
Change the UI mechanism to only request OS Information only on initial page load rather than every 10 seconds.
KVM pod improvements
Continuing with the improvements from alpha 2, this new release provides more updates to KVM pods:
Added overcommit ratios for CPU and memory.
When composing or allocating machines, previous versions of MAAS would allow the user to request as many resources as the user wanted regardless of the available resources. This created issues when dynamically allocating machines as it could allow users to create an infinite number of machines even when the physical host was already over committed. Adding this feature allows administrators to control the amount of resources they want to over commit.
Added ability to filter which pods or pod types to avoid when allocating machines
Provides users with the ability to select which pods or pod types not to allocate resources from. This makes it particularly useful when dynamically allocating machines when MAAS has a large number of pods.
DNS UI Improvements
MAAS 2.0 introduced the ability to manage DNS, not only to allow the creation of new domains, but also to the creation of resources records such as A, AAA, CNAME, etc. However, most of this functionality has only been available over the API, as the UI only allowed to add and remove new domains.
As of 2.4, MAAS now adds the ability to manage not only DNS domains but also the following resource records:
Added ability to edit domains (e.g. TTL, name, authoritative).
Added ability to create and delete resource records (A, AAA, CNAME, TXT, etc).
Added ability to edit resource records.
Navigation UI improvements
MAAS 2.4 beta 1 is changing the top-level navigation:
Rename ‘Zones’ for ‘AZs’
Add ‘Machines, Devices, Controllers’ to the top-level navigation instead of ‘Hardware’.
A few notable improvements being made available in MAAS 2.4 include:
Add ability to force the boot type for IPMI machines.
Hardware manufactures have been upgrading their BMC firmware versions to be more compliant with the Intel IPMI 2.0 spec. Unfortunately, the IPMI 2.0 spec has made changes that provide a non-backward compatible user experience. For example, if the administrator configures their machine to always PXE boot over EFI, and the user executed an IPMI command without specifying the boot type, the machine would use the value of the configured BIOS. However, with these new changes, the user is required to always specify a boot type, avoiding a fallback to the BIOS.
As such, MAAS now allows the selection of a boot type (auto, legacy, efi) to force the machine to always PXE with the desired type (on the next boot only) .
Add ability, via the API, to skip the BMC configuration on commissioning.
Provides an API option to skip the BMC auto configuration during commissioning for IPMI systems. This option helps admins keep credentials provided over the API when adding new nodes.
Please refer to the following for all 32 bug fixes in this release.
Starting from MAAS 2.3, the way run ephemeral environments and perform deployments was changed away from using iSCSI. Instead, we introduced the ability to do the same using a squashfs image. With that, we completely removed the requirement for having tgt at all, but we didn’t drop the dependency in 2.3. As of 2.4, however, tgt has now been completely removed.
Dependency on apache2 has now been dropped in the debian packages
Starting from MAAS 2.0, MAAS now made the UI available in port 5240 and deprecated the use of port 80. However, as a mechanism to not break users when upgrading from the previous LTS, MAAS continued to have apache2 as a dependency to provide a reverse proxy to allow users to connect via port 80.
However, the MAAS snap changed that behavior no longer providing access to MAAS via port 80. In order to keep MAAS consistent with the snap, starting from MAAS 2.4, the debian package no longer depends on apache2 to provide a reverse proxy capability from port 80.
Python libmaas (0.6.0) now available in the Ubuntu Archive
I’m happy to announce that the new MAAS Client Library is now available in the Ubuntu Archives for Bionic. Libmaas is an asyncio based client library that provides a nice interface to interact with MAAS. More details below.
New Features & Improvements
MAAS now adds the ability to lock machines, which prevents the user from performing actions on machines that could change their state. This gives MAAS a prevention mechanism of potentially catastrophic actions. For example, it will prevent mistakenly powering off machines or mistanly releasing machines that could bring workloads down.
MAAS 2.4 now allows the administrators to audit the user’s actions, with the introduction of audit logging. The audit logs are available to administrators via the MAAS CLI/API, giving administrators a centralized location to access these logs.
Documentation is in the process of being published. For raw access please refer to the following link:
Commissioning Harness – Supporting firmware upgrade and hardware specific scripts
The commissioning harness has been expanded with various improvements to help administrators write their own firmware upgrade and hardware specific scripts. These improvements addresses various of the challenges administrators face when performing such tasks at scale. The improvements include:
Ability to auto-select all the firmware upgrade/storage hardware changes (API only, UI will be available soon)
Ability to run scripts only for the hardware they are intended to run on.
Ability to reboot the machine while on the commissioning environment without disrupting the commissioning process.
This allows administrators to:
Create a hardware specific by declaring in which machine it needs to be run, by specifying the hardware specific PCI ID, modalias, vendor or model of the machine or device.
Create firmware upgrade scripts that require a reboot before the machine finishes the commissioning process, by allowing to describe this in the script’s metadata.
Allows administrators to define where the script can obtain proprietary firmware and/or proprietary tools to perform any of the operations required.
Minor improvements – Gather information about BIOS & firmware
MAAS now gathers more information about the underlying system, such as the Model, Serial, BIOS and firmware information of a machine (where available). It also gathers the information for storage devices as well as network interfaces.
MAAS Client Library (python-libmaas)
New upstream release – 0.6.0
A new upstream release is now available in the Ubuntu Archive for Bionic. The new release includes the following changes:
Add/read/update/delete storage devices attached to machines.
Configure partitions and mount points
Known issues & work arounds
LP: #1748712 – 2.4.0a1 upgrade failed with old node event data
It has been reported that an upgrade to MAAS 2.4.0a1 failed due to having old data from a non-existent know stored in the database. This could have been due to a older devel version of MAAS which would have left an entry in the node event table. A work around is provided in the bug report.
If you hit this issue, please update the bug report immediately so MAAS developers.
Please refer to the following for all bug fixes in this release.
For a while, I have been wanting to write about MAAS and how it can easily deploy workloads (specially OpenStack) with Juju, and the time has finally come. This will be the first of a series of posts where I’ll provide an Overview of how to quickly get started with MAAS and Juju.
What is MAAS?
I think that MAAS does not require introduction, but if people really need to know, this awesome video will provide a far better explanation than the one I can give in this blog post.
MAAS have been designed in such a way that it can be deployed in different architectures and network environments. MAAS can be deployed as both, a Single-Node or Multi-Node Architecture. This allows MAAS to be a scalable deployment system to meet your needs. It has two basic components, the MAAS Region Controller and the MAAS Cluster Controller.
The MAAS Region Controller is the component the users interface with, and is the one that controls the Cluster Controllers. It is the place of the WebUI and API. The Region Controller is also the place for the MAAS meta-data server for cloud-init, as well as the place where the DNS server runs. The region controller also configures a rsyslogd server to log the installation process, as well as a proxy (squid-deb-proxy) that is used to cache the debian packages. The preseeds used for the different stages of the process are also being stored here.
The MAAS Cluster Controller only interfaces with the Region controller and is the one in charge of provisioning in general. The Cluster Controller is the place the TFTP and DHCP server(s) are located. This is the place where both the PXE files and ephemeral images are being stored. It is also the Cluster Controller’s job to power on/off the managed nodes (if configured).
As you can see in the image above, MAAS can be deployed in both a single node or multi-node. The way MAAS has being designed makes MAAS highly scalable allowing to add more Cluster Controllers that will manage a different pool of machines. A single-node scenario can become in a multi-node scenario by simply adding more Cluster Controllers. Each Cluster Controller has to register with the Region Controller, and each can be configured to manage a different Network. The way has this is intended to work is that each Cluster Controller will manage a different pool of machines in different networks (for provisioning), allowing MAAS to manage hundreds of machines. This is completely transparent to users because MAAS makes the machines available to them as a single pool of machines, which can all be used for deploying/orchestrating your services with juju.
How Does It Work?
MAAS has 3 basic stages. These are Enlistment, Commissioning and Deployment which are explained below:
The enlistment process is the process on which a new machine is registered to MAAS. When a new machine is started, it will obtain an IP address and PXE boot from the MAAS Cluster Controller. The PXE boot process will instruct the machine to load an ephemeral image that will run and perform an initial discovery process (via a preseed fed to cloud-init). This discovery process will obtain basic information such as network interfaces, MAC addresses and the machine’s architecture. Once this information is gathered, a request to register the machine is made to the MAAS Region Controller. Once this happens, the machine will appear in MAAS with a Declared state.
The commissioning process is the process where MAAS collects hardware information, such as the number of CPU cores, RAM memory, disk size, etc, which can be later used as constraints. Once the machine has been enlisted (Declared State), the machine must be accepted into the MAAS in order for the commissioning processes to begin and for it to be ready for deployment. For example, in the WebUI, an “Accept & Commission” button will be present. Once the machine gets accepted into MAAS, the machine will PXE boot from the MAAS Cluster Controller and will be instructed to run the same ephemeral image (again). This time, however, the commissioning process will be instructed to gather more information about the machine, which will be sent back to the MAAS region controller (via cloud-init from MAAS meta-data server). Once this process has finished, the machine information will be updated it will change to Ready state. This status means that the machine is ready for deployment.
Once the machines are in Ready state, they can be used for deployment. Deployment can happen with both juju or the maas-cli (or even the WebUI). The maas-cli will only allow you to install Ubuntu on the machine, while juju will not only allow you to deploy Ubuntu on them, but will allow you to orchestrate services. When a machine has been deployed, its state will change to Allocated to <user>. This state means that the machine is in use by the user who requested its deployment.
Once a user doesn’t need the machine anymore, it can be released and its status will change from Allocated to <user> back to Ready. This means that the machine will be turned off and will be made available for later use.
But… How do Machines Turn On/Off?
Now, you might be wondering how are the machines being turned on/off or who is the one in charge of that. MAAS can manage power devices, such as IPMI/iLO, Sentry Switch CDU’s, or even virsh. By default, we expect that all the machines being controlled by MAAS have IPMI/iLO cards. So if your machines do, MAAS will attempt to auto-detect and auto-configure your IPMI/iLO cards during the Enlistment and Commissioning processes. Once the machines are Accepted into MAAS (after enlistment) they will be turned on automatically and they will be Commissioned (that is if IPMI was discovered and configured correctly)..This also means that every time a machine is being deployed, they will be turned on automatically.
Note that MAAS not only handles physical machines, it can also handle Virtual Machines, hence the virsh power management type. However, you will have to manually configure the details in order for MAAS to manage these virtual machines and turn them on/off automatically.
For all of those who don’t know, “PowerNap is a screen saver for servers except it doesn’t save your screen, it saves the environment and lowers your energy bill.” Dustin Kirkland :). PowerNap was originally created by Dustin to be integrated with (UEC), but it has been extended for Home use. Originally, it put to sleep machines (suspend, hibernate, poweroff) when a list of Processes were not found in the process table for a determined period of time. However, during the Natty cycle improvements were made. So, PowerNap now puts to sleep (suspend, poweroff, powersave) machines that are tagged as underutilized by a set if Monitors.
PowerNap, has a set of Monitors to be able to detect activity within the server and determine if it is idled or not. If it is, PowerNap will execute an ACTION. Administrators can chose what monitors to enable/disable. These are:
ProcessMonitor: Looks for a process in the process table.
IOMonitor: Monitors IO activity by process name.
InputMonitor: Monitors Mouse/Keyboard input activity connected to USB.
LoadMonitor: Monitors a serverload threshold.
TCPMonitor: Monitors active TCP connections (i.e. SSH).
UDPMonitor: Monitors activity received in any user defined UDP port.
WoLMonitor: Monitors WoL packets on ports 7 and/or 9.
ConsoleMonitor: Monitors console activity.
The process starts when PowerNap begins monitoring for an ABSENT_PERIOD (i.e. 300secs). If within that period no activity has been detected, then PowerNap executes an ACTION.
Before the ACTION is taken, PowerNap enters to the GRACE_PERIOD (I.e 30 seconds), notifying the user that the ACTION will be taken in GRACE_PERIOD amount of seconds. (i.e. On second 270 PowerNap will notifies its users and the period between 270 and 300 seconds is known as GRACE_PERIOD).
The possible ACTIONS are:
Best-effort – Automatically decide between a user defined action or any of the other methods listed below (these methods rely on pm-utils)
Suspend (Command: pm-suspend)
Hibernate (Command: pm-hibernate)
Poweroff (Command: poweroff)
Powersave – Newly added method that reduces the Power Consumption (Command: pm-powersave)
The PowerSave method executes a set of scripts both provided by pm-utils and PowerNap. These scripts have the objective to reduce the power consumption of the machine by turning off hardware capabilities or tuning the OS. It is possible to provide any custom script as well as chose which to enable or disable. Examples of these scripts are:
Turn off all the CPU cores except of one.
Reduce the cores frequency to the lowest possible.
Disable WoL from Network Cards.
Change the NIC speed from 1Gbps to 100Mbps.
Turn off USB ports.
Disable HAL polling.
Now, when the PowerSave ACTION is taken, the machine keeps running in a lower power state. PowerNap keeps Monitoring until activity is detected. Once any of the Monitors detects activity, the PowerSave action is reverted.
PowerWake is simply a tool that sends WoL packets to an specified IP/Broadcast address to be able to wakeup a server.
powernap-now: Sends a signal to the PowerNap daemon to execute the ACTION regardless of the state of the monitors.
powerwake-now: Sends a signal to the PowerNap daemon to wakeup during the PowerSave mode.
Note that these commands have to be executed in the machine running PowerNap. If this needs to be done through the network, then the command will have to be sent remotely to be executed in the machine.
Second Stage Action: Second Stage Action when entered into PowerSave mode. (i.e. Suspend after 2 hours after running in PowerSave mode).
Client/Server Model: The main Idea is to create a powerwaked Server that tracks all the machines using PowerNap in the network and is able to schedule wakeups, upstates, etc, etc.
So, last month I was reading the “Unity Desktop and maverick backport” thread at the ubuntu-devel list. The discussion at some point became about How to Test Natty (Unity Compiz specifically) in real hardware from early stages in the development cycle. So, Dustin recommended the use of TestDrive to do the testing. However, he also mention that 3D acceleration was not available in the VM’s, and his recommendation was more related to 2D testing.
So, that discussion reminded me of a proposed branch to TestDrive that was outdated, on which an option was added to be able to Launch an ISO from GRUB, by placing the ISO in an special folder, and creating an entry for GRUB’s boot menu. So, today I decided to test that feature! It works, but the code needs improvements. So, before actually working on them, I was wondering what ya’ll think?
So my question is, would it be a good idea to add that option to TestDrive to make an ISO available for booting directly from GRUB for testing in real hardware?, or not? Pros/Cons, Comments, Suggestions?