Sbe Alternatives

sbe reviews and mentions

• Modern CPUs have a backstage cast
  5 projects | news.ycombinator.com | 30 May 2023

  Since all POWER9 firmware is open, I can actually answer this:

  The very first instructions executed by the SBE are from OTPROM (one-time programmable ROM, set at the factory). This is just a few instructions: https://github.com/open-power/sbe/blob/master/src/boot/otpro...

  The first SBE code embodied in a mutable nonvolatile storage device executed is here: https://github.com/open-power/sbe/blob/master/src/boot/loade...

  Interestingly the SBE code is actually stored on a serial I2C EEPROM on the CPU module itself, not on the host. This is quite unusual from an x86 perspective where there is usually not any mutable nonvolatile state on the CPU itself.

  POWER9 is also a little unusual in that just powering on the CPU doesn't cause it to do anything by itself. You have to talk to the CPU over FSI (Flexible Service Interface), a custom IBM clock+data control protocol which is mastered by the BMC and used by the BMC to send a 'Go' signal after powering on the system's power rails. This FSI interface is basically the CPU's "front port" - think of an operator front panel on an old mainframe. It's kind of fascinating how IBM's system designs still reflect this kind of past. Indeed, until very recently (I think this only changed with POWER8, but it may have been POWER7) IBM's own POWER server CPUs weren't designed to boot, in the sense of being brought up in a self-sufficient way - they were externally IPLed by the FSP (IBM's version of a BMC).

  Basically, what we think of as boot firmware would actually execute on the FSP - which is a PPC476 running Linux - so you have the boot firmware executing as a userspace Linux program on the FSP, doing register pokes to get the CPU initialised, do memory training, etc. all remotely via the FSI master, rather than it being done on the CPU itself. It would even load the hypervisor, PowerVM. The CPU itself traditionally was essentially held in reset until the service processor had completed all this init.

  So the POWER CPUs have traditionally all been initialised externally, rather than via 'bootstrapping' in the literal sense. Even memory training was done this way, all via the CPU's front port. What's really amazing is that when IBM wanted to switch to a self-boot model around the time of POWER8 (I think), they looked at their boot/IPL code, which was designed under the assumption it would be running on an external processor and initialise the CPU by probing it. Their response was to write a C++-level emulation environment for all the internal API calls this firmware uses to access the hardware. Basically, you have hardware initialisation procedures written as though running on a separate machine, and then a C++ framework all of this is hosted in which pretends that this is still the case, but allows all of these hardware initialisation procedures to be reused without having to rewrite them all. Amazingly it works. Though the size of this boot firmware is so large, it even has to have a paging mechanism for its own code while running in CAR mode(!). http://github.com/open-power/hostboot

  With POWER8/9 since IBM moved to a self-boot model, the FSI interface is often just used to send the 'Go' signal, and I believe by the BMC to query temperature information after boot. But the CPU won't do anything until you send that signal. It's kind of cute, really. Sending the 'Go' signal starts the SBE running. And yes, this means that all of the hardware initialisation procedures which were written to be used from a separate service processor are now never used that way.

  This kind of 'front port' based design to system 'IPL' in which a service processor just pokes the CPU remotely to initialise it is pretty fascinating. It's like the idea of automating what a human operator would have done to initialise a mainframe long, long, ago (makes me think of the autopilot in Airplane). Though of course the amount of stuff you'd have to do flipping switches on an operator panel to initialise one of these CPUs may take a lifetime if done manually...

  So this is an example of how IBM's technical history very definitely lives on and is reflected in their systems engineering today.

  Worth noting an advantage of this FSI interface is that it's also effectively a hardware debugger interface. In fact I believe the Talos/Blackbird systems ship with the 'pdbg' hardware debugger tool accessible via BMC SSH shell. So these systems effectively have a hardware debugger built in, which is just a Linux userspace tool which pokes the CPU's debug registers over FSI. https://github.com/open-power/pdbg

• What Happens When a CPU Starts
  3 projects | news.ycombinator.com | 10 Jan 2023

  Something fun mentioned in a comment on that article, this the majority of this "pre-boot" code is actually FOSS in POWER9. There are a set of "auxiliary processors" called "PPE"s, among which there is one, the "SBE", or "self boot engine", which is a very small and simple PowerPC core that IPLs the big POWER9 cores [0]. These big processors with tons of cache and interconnects need a lot of help to get to executing PC 0x00.

  I suspect that almost all the big "application processors" from Intel and AMD, and the exotic ARM/SPARC server chips, have equivalent embedded ICs to jump-start the "big cores".

  [0] https://github.com/open-power/sbe/blob/master/src/sbefw/app/...