Add Redfish support to Maintenance¶
Storyboard: https://storyboard.openstack.org/#!/story/2005861
This story adds Redfish Platform Management
support to Starling-X
Maintenance as a prioritized alternative to the existing less secure
IPMI support for the following board management functions
Reset and Power On/Off Control
Network Boot Override
Sensor Monitoring
Problem description¶
Starling-X Maintenance currently uses ipmitool
to invoke board management
functions. Unfortunately however, IPMI is aged and not evolving with the server
market.
Redfish
is a new and emerging well-defined Platform Management Application
Programming Interface (API) standard that leverages modern software, is more
secure and is easier to use and understand compared to IPMI.
Redfish API uses the HTTP protocol over a TCP/IP network using either JSON or XML data schemas to leverage common Internet and web services standards and modern tool chains to add new board management services for modern host servers to meet today’s system administrator demands.
Redfish offers a single root endpoint that expands to reveal a well-structured hierarchy of service, system, chassis and management endpoints accessed in user sessions and or single shot command operations to manage and monitor the hardware in polled and event driven models.
Use Cases¶
System developers, testers, operators, administrators and auto provisioning tools need the ability to power on, power off and reset hosts as well as force hosts to boot from the network during installation activities.
High availability products such as Starling-X also need the ability to monitor the health of its host server pool so that it can notify system administrators or system orchestrators of pending or immediate service affecting hardware failures for proactive action and service migrations.
Proposed change¶
Maintenance shall continue with the existing centralized power/reset control and sensor monitoring model.
Integrate BSD licenced Redfish tool into the load and use it similar to how
ipmitool is used today which launches a thread that runs ipmitool
as a
system command with hidden credentials and reports execution status to the
main process as a json string.
Maintain the existing ipmitool solution for hosts that do not support redfish.
A common redfish root query will be implemented and called upon BMC provisioning notification to Maintenance (mtcAgent) and the Hardware Monitor (hwmond).
If that query indicates support for Redfish
then all BMC access to that
host will be done using the new Redfish tool and managed by the associated
content added by this feature. Otherwise, current ipmitool method will be used.
This way Redfish management takes priority over IPMI.
Aside from work to integrate Redfish tool into the load, all changes for this
feature update are restricted to two maintenance daemons ; mtcAgent
and
hwmond
.
The implementation model for this Redfish support follows what is currently done for ipmitool. For each request, launch the tool thread to run the system command that makes the Redfish request followed by interpreting the response and passing pertinent data back to the main process in a formatted json string.
There are very little change to the main mtcAgent and hwmond processes. There are no changes to Starling-X System Inventory (sysinv). There are no changes to BMC provisioning.
Alternatives¶
An alternative to using the opensource Redfishtool is to implement an HTTP agent that conforms to the DMTF Redfish Scalable Platforms Management API Specification (DSP0266) with the ability to initiate and handle success and failure responses for System Reset, System setBootOverride as well as Chassis Power and Thermal targets for sensor monitoring.
Such agent would require a back-end interface that the Starling-X Maintenance and Hardware Monitor processes could bind into for orchestration purposes.
The work involved to implement this alternative is extensive and could require ongoing updates as the Redfish API evolves.
Data model impact¶
If a host represents its sensors differently in name or type between its ipmi and redfish services then the sensor model for that host may have to be relearned.
Fortunately the Hardware Monitor already supports a sensor model relearn function in support of BMC and SDR firmware upgrade but also serves feature patch cases as well.
The sensor model relearn is
automatic over a
hwmond
process restart if the detected model differs from the model stored in system inventory.manual using the
system host-sensorgroup-relearn
CLI command or by pressing the relearn button on the Host’s Sensor tab in Horizon.
REST API impact¶
None. This story does not change any existing REST APIs.
Security impact¶
A primary design goal in the development of Redfish was to offer improved platform management security compared to existing solutions such as IPMI.
Redfish API supports two authentication methods
Basic Authentication
Token Authentication
This feature makes its sparse and infrequent requests using Basic authentication. Token authentication adds complexity with no justification.
Security features built into Redfish are described in the Redfish Scalable Platforms Management API Specification ; https://www.dmtf.org/sites/default/files/standards/documents/DSP0266_1.6.0.pdf
American Department of Homeland Security warns of the security vulnerabilities of IPMI ; https://www.us-cert.gov/ncas/alerts/TA13-207A
Other end user impact¶
None.
Performance Impact¶
Any performance impact by the introduction of this feature is negligible for the following reasons:
the current method uses ipmitool while this feature uses redfishtool in a very similar way.
both methods invoke the tool as a thread to avoid blocking the main process.
maintenance actions are rare, on-demand only and while the host is locked.
sensor monitoring is periodic with a cadence in minutes not seconds.
only impact would be in the difference between the individual two open source tools and prototype testing demonstrated comparable performances.
measured both ipmitool and redfishtool command execution with
time
and found them to be comparable.
Other deployer impact¶
This feature introduces a new RPM ; redfishtool. If this feature were to be patched back to an earlier release then that redfishtool RPM would also have to be patched back.
If this feature is patched back to an earlier release or patched into a current release then * the mtcAgent process would have to be restarted. * the hwmond process would have to be restarted.
Developer impact¶
This feature has no impact to other developers working on StarlingX.
Upgrade impact¶
None currently as this is the initial implementation of Redfish support.
Newer versions of Redfishtool can be introduced if integration testing of that
newer version verifies that the currently used command line options and relied
upon underlying behavior passes the test cases listed in the Testing
section below.
If a newer version of redfishtool is required and has functionally impacting changes then maintenance will have to query the redfishtool version and behave as required by the detected version. ‘redfishtool -V’ prints the redfish tool version.
Implementation¶
Assignee(s)¶
- Primary assignee:
Eric MacDonald
- Other contributors:
Zhipeng Liu
Repos Impacted¶
stx-integ - adding redfishtool
stx-metal - updating mainteance with redfish support
Work Items¶
redfish - stx-integ/bmc/Redfishtool
create patched RPM package and include on controllers
create patch that adds unimplemented cfgFile support for hiding credentials.
push cfgFile support upstream.
create patch that makes redfishtool support python-2 and then removed once Starling-X supports python-3
Maintenance Common - stx-metal/mtce-common/src/common
create common redfishUtil.cpp/.h for similar purpose/function to the existing ipmiUtil.cpp/h for use with both hwmond and mtcAgent.
Maintenance - stx-metal/mtce/src/maintenance - mtcAgent process
create mtcRedFishUtil.cpp/h for similar purpose/function to the existing mtcIpmiUtil.cpp/h for sending and receiving RedFishTool requests for maintenance power reset and control, power status and hw/fw version query.
enhance mtcThread.cpp/h with mtcThread_redfishtool request support similar to the existing mtcThread_ipmitool thread used to handle redfish tool requests and responses as a thread.
Hardware Monitor - stx-metal/mtce/src/hwmon - hwmond process
create hwmonRedFish.cpp/h for similar purpose/function to the existing hwmonIpmi.cpp/h for parsing sensor query responses into a common format for the hardware monitor sensor manager engine.
enhance hwmonThreads.cpp/h with new hwmonThread_redfishtool request support similar to the existing mtcThread_ipmitool pthread.
Dependencies¶
This specification depends upon the open source Redfishtool.
Testing¶
This feature can be tested in a fully provisioned duplex Starling-X system with Redfish supported hosts that have their BMC provisioned through system inventory.
With a host’s BMC provisioned, verify that the mtcAgent and hwmond processes on the active controller each report a log stating that the UUT host is being managed by Redfish ; rather than IPMI.
With UUT host locked, perform Reset action and verify the host experiences a graceful shutdown followed by a reboot.
With UUT host locked and online, perform Power-Off action and verify the host experiences a graceful shutdown followed by a power-off.
With UUT host locked and powered off, perform power-on action and verify the host powers on and starts to boot.
With UUT host locked and powered off with a bootable image on disk, perform a ReInstall action and verify that the host gets powered on and reinstalls a new image from the controller.
With UUT verify sensor monitoring by viewing the sensor groups and sensors list from Horizon with CLI commands.
Documentation Impact¶
This feature change has no customer visible impact. This feature change requires no customer documentation update.
References¶
Redfish was developed by DTMF (Distributed Management Task Force), lead by a diverse board of directors and contributors from many of the major technology companies like Intel, Dell, HP, Hitachi, Lenovo, Vmware, etc.
Redfish Platform Management Application Programming Interface (API) standard and supporting specifications can be found at the following URL.
History¶
Release Name |
Description |
---|---|
2019.11 |
Introduced |