Hacking around Fujitsu-Siemens RX300 S3 S4 S5 servers

I have experimented a little bit with Fujitsu RX300 servers. I have in my possession a S3, a S4 and two S5 servers. I have gathered some interesting factoids and ideas I wish to share.

Snippet 1: troubleshooting RX300 S3 – I am using the particular machine for a small ProxMox installation.
At some random time the server started to hang during the boot. The last BIOS code seen was “90”. In a random manual I found the information that “BIOS code 90” means disk controllers. Sometimes the machine succeeded booting but sometimes not. At some point it did not boot any more – it was hanging every boot.Well, thinking a little bit, I decided to open the cover, took the fan block off and looked at the disk controller chip. Everything seemed to be OK but there was too much of dust there. “Too much” in case of a a server means that it was possible to spot tiny dust collections under the fan block (my server room is not a conditioned cleanroom). The solution: I tested the big SATA connector by disconnecting and then conneting it again. I took a brush and cleaned the motherboard. I changed the CR2450 battery (albeit it surely wasn’t the culprit) because the voltage had been dropped onto 2.9V level).

After these manipulations the motherboard started to boot correctly. Thus the dust seems to have been the main reason.

RX300 S3 is extremely picky about the memories. Somehow I tried to add the wrong memory (a surely working one, but not for that particular server). After the memory error LEDs start flashing, it is not anymore enough to “enable” memories from the BIOS menu. However, I found a special procedure to start applying memory DIMMs again:

1. take all DIMMS out and put only one DIMM in, into the position 1A.
2. If you can, do use original Siemens-Fujitsu approved memory for the first DIMM. S3 is much more pickier about the very first memory slot.
3. It is a fun – with some DIMMs the motherboard says that the memory is faulty and tells: “there is no memory available to the operating system”. Believe you or not, but should you hesitate and still push <F1> to resume, the OS will boot. However, only the first DIMM will be available to the OS, in my case a 2GB one. The verdict: the BIOS will claim some memories faulty while they actually aren’t.
4. Do add the second DIMM into slot 1B (leaving slots 1C and 1D empty) and reclaim the new DIMM by “enabling” it from the BIOS
5. Only after step 4 succeeded, make any attempt to add DIMMs into slots 1C and 1D (also “enabling” these via BIOS)

This sequential procedure seems to be the only way to clear from the situation where you have fulfilled all the slots and the BIOS claims memory faulty.

The most embezzling issue for me is that S3 is not taking the M395T5750FZ4-CE66 memories while M395T5750FZ4-CE61 memories work well.

Snippet 2: exchanging RAID controllers under a FreeNAS installation. I set up a FreeNAS server on top of Fujitsu RX300 S4 hardware. The original controller was SAS 8708ELP (SAS1078). Because I was lacking the proper RAID controller, I used the original one to test the concept of the particular NAS. My previous blogpost was about changing the aged batteries on that SAS 8708ELP RAID controller . And then … I bought my M1015 that seems to be the de-facto standard for the lower end FreeNAS servers. I reflashed the M1015 into IT mode and everything went as expected. I changed the controller and connected my disks to the new controller.

To my great surprise, M1015 in IT mode was able to recognize the disks together with the volumes and datasets on those. This way I hadn’t to copy the data again. I have not tried, whether the opposite is true too – i.e. whether  a “normal” RAID controller will permit to import disks that were previously connected directly (JBOD). Probably it does not – generic RAID cards tend to “initialize” the disks. I hope the piece of information will be useful for someone – that it is possible to read your “HW RAID” disks via IT mode JBOD controller.

Snippet 3. Pwning the IPMI. As one might know, S3, S4 and S5 servers are from 7+ years generation. It means – you cannot obtain these as new. And, it seems that corporate administrators mostly forget to erase the authentication information from IPMI chips. Leaving aside the obvious risks, for me it  meant I could not use IPMI functionality because I did not know the proper password – admin/admin does not work.

I used different hacking approaches on the S3, S4 and S5. The Internet offers numerous possibilities to “pwn” the insecure IPMI section of a rack server. Here are the ways the ways to do that:

1. The obvious one – enter the BIOS menu (see the pic above) and load iRMC default values. Unsuccessfully this method is available neither for  S3  nor S4.
2. Obtain the iRMC2 update diskette image from Fujitsu, burn the disk (Well, I am kidding, you are not from the floppy generation),
   then do interrupt the boot and make the deal via local ipmiview utility – see instructions at https://forum.ts.fujitsu.com/forum/viewtopic.php?t=32597
3. Use the Linux IPMI tool as indicated by Thomas Krenn . However, it is extremely unbelieveable you have got the enterprise class server with the disks and that means – no operating system, no IPMI drivers. But keep reading, there is a more elegant solution than to use floppy with ipmiview utility.
4. Hack it over the ethernet wire. The IPMI module for RX300 S3 was insecure enough, thus I followed the Rapid7 FluffyWabbit instructions . The Pwn was a perfect one.
5. However, S4 did not open its back door using the last method. Luckily I had a FreeNAS =>9.3 installation media underway. I needed two USB sticks – one to boot off and another (better a 8GB not a 4GB one) to put the installation onto. It takes some time to proceed with the FreeNAS installation, but then do open the web console “Shell” and use ipmitool in a full accordance with Thomas Krenn’s instructions –

After that, open the ServerView WWW page and you are done. I never had an idea that a FreeNAS installation can be useful as an instrumental tool.

GUI fans could try the new functionality available at FreeNAS 9.3 GUI – a special menu item under “Network” menu called “IPMI”. To tell the truth, on the S4 I had a little trouble using this menu. The result was shown “IPMI failed” while the utility was still able to overwrite the netmask value at IPMI settings (a bad choice for the default value of the netmask). Thus I would recommend to use that new FreeNAS functionality somewhat cautiously.

Of course, one can set the IPMI address from the BIOS menu, but that part of the deal should be relatively evident even for wannabe administrators. If not, do D/L and RTFM the iRMC2 manual from FSC . Last but not least, there exist yet another way of hacking of those poor IPMI devices:

1. Obtaining the hashes via wire and cracking these with standard tools – http://fish2.com/ipmi/remote-pw-cracking.html

It is very evident that IPMI 2 devices cannot be trusted due to their orif1ces widely open. Please hide those old servers into a separate Ethernet segment behind a firewall and do use VLAN tags for the God’s sake. However – I like IPMI. I like to look at the temperature and amperage values 😉 .

Snippet 4: Reflashing RX300 S5 disk controllers. Together with my RX300 S5 servers did come the original Fujitsu RAID controllers ICT-1607:

Ot course I was eager to reflash these cards into IT mode. The excellent blog post at zorinaq.com cites certain limitations (a 2GB limit) and a somewhat slower speed than M1015 but otherwise I met no hassle while reflashing the Fujitsu RAID card into SAS1068E IT mode. The biggest obstacle was the necessity to create a DOS bootable USB stick. After that’s done, LSI utilities worked from any subdirectory like a charm. There is a forum useful for the mental preparations phase – https://lime-technology.com/forum/index.php?topic=12767.0

Snippet 5: I am currently playing with the idea that I should reflash the SAS1068E clone on the RX100 S3 motherboard. As with any experiment, there is the risk to brick thingz, this time the whole motherboard. After I’ll get done, I write about the outcome.