Add Redfish support to Maintenance

Storyboard: https://storyboard.openstack.org/#!/story/2005861

This story adds Redfish Platform Management support to Starling-X Maintenance as a prioritized alternative to the existing less secure IPMI support for the following board management functions

  • Reset and Power On/Off Control

  • Network Boot Override

  • Sensor Monitoring

Problem description

Starling-X Maintenance currently uses ipmitool to invoke board management functions. Unfortunately however, IPMI is aged and not evolving with the server market.

Redfish is a new and emerging well-defined Platform Management Application Programming Interface (API) standard that leverages modern software, is more secure and is easier to use and understand compared to IPMI.

Redfish API uses the HTTP protocol over a TCP/IP network using either JSON or XML data schemas to leverage common Internet and web services standards and modern tool chains to add new board management services for modern host servers to meet today’s system administrator demands.

Redfish offers a single root endpoint that expands to reveal a well-structured hierarchy of service, system, chassis and management endpoints accessed in user sessions and or single shot command operations to manage and monitor the hardware in polled and event driven models.

Use Cases

System developers, testers, operators, administrators and auto provisioning tools need the ability to power on, power off and reset hosts as well as force hosts to boot from the network during installation activities.

High availability products such as Starling-X also need the ability to monitor the health of its host server pool so that it can notify system administrators or system orchestrators of pending or immediate service affecting hardware failures for proactive action and service migrations.

Proposed change

Maintenance shall continue with the existing centralized power/reset control and sensor monitoring model.

Integrate BSD licenced Redfish tool into the load and use it similar to how ipmitool is used today which launches a thread that runs ipmitool as a system command with hidden credentials and reports execution status to the main process as a json string.

Maintain the existing ipmitool solution for hosts that do not support redfish.

A common redfish root query will be implemented and called upon BMC provisioning notification to Maintenance (mtcAgent) and the Hardware Monitor (hwmond).

If that query indicates support for Redfish then all BMC access to that host will be done using the new Redfish tool and managed by the associated content added by this feature. Otherwise, current ipmitool method will be used. This way Redfish management takes priority over IPMI.

Aside from work to integrate Redfish tool into the load, all changes for this feature update are restricted to two maintenance daemons ; mtcAgent and hwmond.

The implementation model for this Redfish support follows what is currently done for ipmitool. For each request, launch the tool thread to run the system command that makes the Redfish request followed by interpreting the response and passing pertinent data back to the main process in a formatted json string.

There are very little change to the main mtcAgent and hwmond processes. There are no changes to Starling-X System Inventory (sysinv). There are no changes to BMC provisioning.

Alternatives

An alternative to using the opensource Redfishtool is to implement an HTTP agent that conforms to the DMTF Redfish Scalable Platforms Management API Specification (DSP0266) with the ability to initiate and handle success and failure responses for System Reset, System setBootOverride as well as Chassis Power and Thermal targets for sensor monitoring.

Such agent would require a back-end interface that the Starling-X Maintenance and Hardware Monitor processes could bind into for orchestration purposes.

The work involved to implement this alternative is extensive and could require ongoing updates as the Redfish API evolves.

Data model impact

If a host represents its sensors differently in name or type between its ipmi and redfish services then the sensor model for that host may have to be relearned.

Fortunately the Hardware Monitor already supports a sensor model relearn function in support of BMC and SDR firmware upgrade but also serves feature patch cases as well.

The sensor model relearn is

  • automatic over a hwmond process restart if the detected model differs from the model stored in system inventory.

  • manual using the system host-sensorgroup-relearn CLI command or by pressing the relearn button on the Host’s Sensor tab in Horizon.

REST API impact

None. This story does not change any existing REST APIs.

Security impact

A primary design goal in the development of Redfish was to offer improved platform management security compared to existing solutions such as IPMI.

Redfish API supports two authentication methods

  • Basic Authentication

  • Token Authentication

This feature makes its sparse and infrequent requests using Basic authentication. Token authentication adds complexity with no justification.

Security features built into Redfish are described in the Redfish Scalable Platforms Management API Specification ; https://www.dmtf.org/sites/default/files/standards/documents/DSP0266_1.6.0.pdf

American Department of Homeland Security warns of the security vulnerabilities of IPMI ; https://www.us-cert.gov/ncas/alerts/TA13-207A

Other end user impact

None.

Performance Impact

Any performance impact by the introduction of this feature is negligible for the following reasons:

  • the current method uses ipmitool while this feature uses redfishtool in a very similar way.

  • both methods invoke the tool as a thread to avoid blocking the main process.

  • maintenance actions are rare, on-demand only and while the host is locked.

  • sensor monitoring is periodic with a cadence in minutes not seconds.

  • only impact would be in the difference between the individual two open source tools and prototype testing demonstrated comparable performances.

  • measured both ipmitool and redfishtool command execution with time and found them to be comparable.

Other deployer impact

This feature introduces a new RPM ; redfishtool. If this feature were to be patched back to an earlier release then that redfishtool RPM would also have to be patched back.

If this feature is patched back to an earlier release or patched into a current release then * the mtcAgent process would have to be restarted. * the hwmond process would have to be restarted.

Developer impact

This feature has no impact to other developers working on StarlingX.

Upgrade impact

None currently as this is the initial implementation of Redfish support.

Newer versions of Redfishtool can be introduced if integration testing of that newer version verifies that the currently used command line options and relied upon underlying behavior passes the test cases listed in the Testing section below.

If a newer version of redfishtool is required and has functionally impacting changes then maintenance will have to query the redfishtool version and behave as required by the detected version. ‘redfishtool -V’ prints the redfish tool version.

Implementation

Assignee(s)

Primary assignee:

Eric MacDonald

Other contributors:

Zhipeng Liu

Repos Impacted

  • stx-integ - adding redfishtool

  • stx-metal - updating mainteance with redfish support

Work Items

redfish - stx-integ/bmc/Redfishtool

  • create patched RPM package and include on controllers

  • create patch that adds unimplemented cfgFile support for hiding credentials.

  • push cfgFile support upstream.

  • create patch that makes redfishtool support python-2 and then removed once Starling-X supports python-3

Maintenance Common - stx-metal/mtce-common/src/common

  • create common redfishUtil.cpp/.h for similar purpose/function to the existing ipmiUtil.cpp/h for use with both hwmond and mtcAgent.

Maintenance - stx-metal/mtce/src/maintenance - mtcAgent process

  • create mtcRedFishUtil.cpp/h for similar purpose/function to the existing mtcIpmiUtil.cpp/h for sending and receiving RedFishTool requests for maintenance power reset and control, power status and hw/fw version query.

  • enhance mtcThread.cpp/h with mtcThread_redfishtool request support similar to the existing mtcThread_ipmitool thread used to handle redfish tool requests and responses as a thread.

Hardware Monitor - stx-metal/mtce/src/hwmon - hwmond process

  • create hwmonRedFish.cpp/h for similar purpose/function to the existing hwmonIpmi.cpp/h for parsing sensor query responses into a common format for the hardware monitor sensor manager engine.

  • enhance hwmonThreads.cpp/h with new hwmonThread_redfishtool request support similar to the existing mtcThread_ipmitool pthread.

Dependencies

This specification depends upon the open source Redfishtool.

https://github.com/DMTF/Redfishtool

Testing

This feature can be tested in a fully provisioned duplex Starling-X system with Redfish supported hosts that have their BMC provisioned through system inventory.

  • With a host’s BMC provisioned, verify that the mtcAgent and hwmond processes on the active controller each report a log stating that the UUT host is being managed by Redfish ; rather than IPMI.

  • With UUT host locked, perform Reset action and verify the host experiences a graceful shutdown followed by a reboot.

  • With UUT host locked and online, perform Power-Off action and verify the host experiences a graceful shutdown followed by a power-off.

  • With UUT host locked and powered off, perform power-on action and verify the host powers on and starts to boot.

  • With UUT host locked and powered off with a bootable image on disk, perform a ReInstall action and verify that the host gets powered on and reinstalls a new image from the controller.

  • With UUT verify sensor monitoring by viewing the sensor groups and sensors list from Horizon with CLI commands.

Documentation Impact

This feature change has no customer visible impact. This feature change requires no customer documentation update.

References

Redfish was developed by DTMF (Distributed Management Task Force), lead by a diverse board of directors and contributors from many of the major technology companies like Intel, Dell, HP, Hitachi, Lenovo, Vmware, etc.

Redfish Platform Management Application Programming Interface (API) standard and supporting specifications can be found at the following URL.

https://www.dmtf.org/standards/redfish

History

Revisions

Release Name

Description

2019.11

Introduced