Megaraid SAS 9341-8i on Linux - Cooling and initialization issues

Not long ago I got some nice second-hand SAS drives that I could use with my home server. I didn’t have a SAS controller that I could use with these drives at that time, so I started looking for one. I didn’t want to spend too much money and the specifications I wanted were not very advanced. I wanted a controller with these caracteristics:

• PCIe 3.0 or 4.0.
• Support for 6Gb/s SATA and 12Gb/s SAS disks.
• Support for JBOD, RAID-0,1,5. RAID-10,50 would be nice to have.
• Support for at least 8 disk (Disk slots in my home server)
• Support for Linux
• Working with my “ASUS Pro WS X570-ACE” motherboard running an “AMD Ryzen9 3900X” CPU.
• Stable and reliable.

Controllers with battery backup and extra memory were to expensive for my budget and needs.

After some investigation I decided to purchase a Broadcom MegaRaid SAS 9341-8i controller. The SAS 9341-8i is aim to medium business (SMB) owners and gaming enthusiasts, and it had all the caracteristics in my list. This controller is a bit old but it had the functionality I was looking for at an affordable price and it could be delivered in a reasonable time in these days of delivery problems and disruptions in global supply chains.

Cooling the controller

I had read some reports of the controller getting very hot sometimes as this controller is aim for servers where the airflow is usually higher than on a home system. The documentation¹ of the controller says that the minimum airflow must be at least 75 linear feet per minute (LFPM) at an ambient/operational temperature range of 0C to +55C. It doesn’t say if the 75 LFPM is:

1. Through the cross sectional area of the server case² (W:44cm H:17cm)
2. Through the chip heat sink (ca 40x40mm)
3. Through the cross sectional area of the PCIe slot used by the controller.

It doesn’t say either at which temperature you need at least 75 FPM. I am inclined to think that we are talking about an inlet air temperature of around 20-25C and either alternative [2] or [3].

I decided to contact Broadcom support to get some more information about the temperature and airflow requirements. I didn’t get all the information I asked for but I got enough after some enquiries. They said that the LSISAS3008 SAS controller used by the SAS 9341-8i has a maximum junction temperature of 115C (+/- 5°C) and that it will start throttling the performance of the chip and will slow down between 105C and 110C, producing a significant performance loss when this occurs. In the logs from the controller it will be noted that the controller is too hot.

So the first thing I did when I received the controller was to install an extra fan³ on top of the heat sink of the LSISAS3008 chip. I installed a Noctua NF-A4x10 FLX 40x40mm fan⁴. This fan could deliver 8,2m³/h at top speed right into the heat sink. This gives us a value of 280.23 LFM for the rectangular area of the heat sink.

In addition, I took into consideration the airflow going through the case. My home server has 4x Noctua NF-A8 ULN 80mm⁵ on the front of the case (They deliver an intake airflow of around 101m³/h at the lowest speed and around 138m³/h at full speed) and 2x Noctua NF-A8 PWM 80mm⁶ on the back of the case (They deliver an outtake airflow of around 88m³/h at the lowest speed and around 111m³/h at full speed.) This configuration gives between 73,8 and 100,8 LFM through the cross sectional area of the case. The idea was also to have a server case with positive air pressure, this prevents dust from penetrating into the chassis by using filters on the intake fans and forces air out of the server case through unfiltered vents and gaps.

With this configuration I am geting a ROC temperature (LSISAS3008 junction temperature) of 56C at idle and around 58C under heavy load with an ambient temperature of 22C. According to Broadcom support this temperature is optimal for the LSISAS3008 SAS controller.

root@server:~$ /opt/MegaRAID/storcli/storcli64 /c0 show temperature 
.......
Controller Properties :
=====================
--------------------------------------
Ctrl_Prop                       Value 
--------------------------------------
ROC temperature(Degree Celsius) 56    
--------------------------------------

Problems initializing the controller

After installing the cooling fan on the controller, mounting the card into one of the PCIe 16x slots on my motherboard and connecting the two Mini-SAS-HD (SFF-8643) to Mini-SAS (SFF-8087) cables between the controller and the backplane for the 8 drive slots, I was ready to start the server and configurate the controller and my virtual drives. The server was already installed with an OS and was running at that time Ubuntu-server 21.10 with a kernel-5.13.

It was a surprise when the Linux kernel discovered the controller but could not initialize it. I thought this controller was 100% supported by Linux, but after some output about the megaraid card I got these two errors and the controller was definitely not operational:

megaraid_sas 0000:06:00.0: Failed to transition controller to ready from megasas_init_fw!
megaraid_sas 0000:06:00.0: Failed from megasas_init_fw 639

It was time to start searching the internet, I didn’t get smarter from the results and I found very lite information about what I could do with this problem. I few posts about it, but without a clear solution, only people talking about BIOS, UEFI boot and different kernel versions.

I tried different parameters in my BIOS and some combinations with different kernels, but I didn’t get anywhere. Then, to activate UEFI I had to reinstall the server, what was my surprise to see the installation crashing because it could not initialize the raid controller.

I came across an article in the Linux kernel list from an user with the same symptoms I had with my system https://lkml.kernel.org/linux-scsi/ce228d78-8651-d958-d1e5-2f82cbb8113d@qubes-os.org/ I contacted him to find out if he got his system up and running but he confirmed that he never got any answer from Broadcom even after several requests.

I was going to give up, but in a last desperate attempt before uninstalling the card, I came across this page https://forum.proxmox.com/threads/raid-controller-9341-8i.88667/ and this gave me the clue to solve the problem I was having with my Megaraid SAS 9341-8i, I have to use iommu=soft when booting the kernel.

What I did then was this:

• In BIOS: Just in case, I defined GEN3 (instead of auto) for the PCIe slot where I had the card installed (My motherboard is an “ASUS Pro WS X570-ACE”⁷ with PCIe-4.0 and the SAS 9341-8i uses PCIe-3.0)
• In BIOS: “IOMMU is enable” was enable.
• In BIOS: Under boot configuration, CSM for storage devices was “Only uefi”.
• Installed the OS with UEFI activated.
• In OS: Defined this line in /etc/default/grub: GRUB_CMDLINE_LINUX_DEFAULT="iommu=soft" and ran update-grub to install the changes.

After these changes, a reboot of the server got everything recognized and initialized.

NOTE If you are installing the OS from scratch with the controller already inserted into a PCIe slot, you will have to pass the option iommu=soft to the kernel when starting the installation program to avoid a crash and be able to install the OS. An alternative will be to do all the changes before inserting the controller into a PCIe slot.

When booting with “Storage devices in Only uefi modus” you will not get the option to start the bios from the card to configurate it. I have used storcli⁸ to do this and it works without problems.

# /opt/MegaRAID/storcli/storcli64 -v
StorCli SAS Customization Utility Ver 007.1912.0000.0000 Nov 23, 2021

Updating the firmware

The first thing I did when I got the SAS 9341-8i working was to upgrade the firmware⁹ to the newest version I could find, and reboot the server to verify that everything was working.

root@server:~/megaraid/firmware-4.680.01-8554# unzip 24.21.0-0148_SAS_iMR_FW_IMAGE_APP_4.680.01-8554.zip 

Archive:  24.21.0-0148_SAS_iMR_FW_IMAGE_APP_4.680.01-8554.zip
  inflating: 24.21.0-0148_SAS_iMR_FW_IMAGE_APP_4.680.01-8554.txt  
  inflating: 240_VD_Feature_Limitations_Known_Issues_Addendum_v1.0.pdf  
  inflating: iMR_4MB.rom             

root@server:~/megaraid/firmware-4.680.01-8554# /opt/MegaRAID/storcli/storcli64 /c0 download file=./iMR_4MB.rom

Download Completed.     
Flashing image to adapter...
CLI Version = 007.1912.0000.0000 Nov 23, 2021
Operating system = Linux 5.13.0-39-generic
Controller = 0
Status = Success
Description = F/W Flash Completed. Please reboot the system for the changes to take effect

Current package version = 24.21.0-0132
New package version = 24.21.0-0148

Interacting with the controller

After a reboot to get the last firmware in production, I could begin interacting with the 9341-8i via the storcli software. The amount of commands and options available is enormous, check the StorCLI documentation¹⁰ for full and detail information.

Here are some examples of how to interact with the controller. The first thing we want to do is get some information about the controller ID and status:

root@server:~# /opt/MegaRAID/storcli/storcli64 show 
.......
Number of Controllers = 1
Host Name = server
Operating System  = Linux 5.13.0-39-generic

System Overview :
===============
------------------------------------------------------------------------------------
Ctl Model                   Ports PDs DGs DNOpt VDs VNOpt BBU sPR DS  EHS ASOs Hlth 
------------------------------------------------------------------------------------
  0 AVAGOMegaRAIDSAS9341-8i     8   8   0     0   0     0 N/A On  1&2 Y      2 Opt  
------------------------------------------------------------------------------------
.......

Now that we know that our controller has an ID 0. We could get more detailed information about our system with these two commands: /opt/MegaRAID/storcli/storcli64 /c0 show and /opt/MegaRAID/storcli/storcli64 /c0 show all.

To get a list of all the disks attached to the controller we can run this:

root@server:~# /opt/MegaRAID/storcli/storcli64 /c0 /eall /sall show  
.......
Drive Information :
=================
--------------------------------------------------------------------------------
EID:Slt DID State DG     Size Intf Med SED PI SeSz Model                Sp Type 
--------------------------------------------------------------------------------
62:0      7 JBOD  -  5.458 TB SATA HDD N   N  512B HGST HUS726T6TALE6L4 U  -    
62:1      2 UGood -  1.745 TB SAS  SSD N   N  512B VO1920JEUQQ          U  -    
62:2      1 UGood -  1.745 TB SAS  SSD N   N  512B VO1920JEUQQ          U  -    
62:3      0 UGood -  1.745 TB SAS  SSD N   N  512B VO1920JEUQQ          U  -    
62:4      5 UGood -  1.745 TB SAS  SSD N   N  512B VO1920JEUQQ          U  -    
62:5      6 UGood -  1.745 TB SAS  SSD N   N  512B VO1920JEUQQ          U  -    
62:6      4 UGood -  1.745 TB SAS  SSD N   N  512B VO1920JEUQQ          U  -    
62:7      3 JBOD  -  1.746 TB SAS  SSD N   N  512B VO1920JEUQQ          U  -    
--------------------------------------------------------------------------------

We could define a RAID-5 virtual disk with the disks 62:1 to 62:5 with:

root@server:~# /opt/MegaRAID/storcli/storcli64 /c0 add vd type=raid5 name=RAID-5 drives=62:1-5

And a HotSpare disk 62:6 with:

root@server:~# /opt/MegaRAID/storcli/storcli64 /c0/e62/s6 add hotsparedrive

Our system would look like this after these changes:

root@server:~# /opt/MegaRAID/storcli/storcli64 /c0 /eall /sall show 
.......
Drive Information :
=================
--------------------------------------------------------------------------------
EID:Slt DID State DG     Size Intf Med SED PI SeSz Model                Sp Type 
--------------------------------------------------------------------------------
62:0      7 JBOD  -  5.458 TB SATA HDD N   N  512B HGST HUS726T6TALE6L4 U  -    
62:1      2 Onln  0  1.745 TB SAS  SSD N   N  512B VO1920JEUQQ          U  -    
62:2      1 Onln  0  1.745 TB SAS  SSD N   N  512B VO1920JEUQQ          U  -    
62:3      0 Onln  0  1.745 TB SAS  SSD N   N  512B VO1920JEUQQ          U  -    
62:4      5 Onln  0  1.745 TB SAS  SSD N   N  512B VO1920JEUQQ          U  -    
62:5      6 Onln  0  1.745 TB SAS  SSD N   N  512B VO1920JEUQQ          U  -    
62:6      4 DHS   0  1.745 TB SAS  SSD N   N  512B VO1920JEUQQ          U  -    
62:7      3 JBOD  -  1.746 TB SAS  SSD N   N  512B VO1920JEUQQ          U  -    
--------------------------------------------------------------------------------
.......

root@server~# /opt/MegaRAID/storcli/storcli64 /c0 /dall show
.......
TOPOLOGY :
========
---------------------------------------------------------------------------
DG Arr Row EID:Slot DID Type  State BT     Size PDC  PI SED DS3  FSpace TR 
---------------------------------------------------------------------------
 0 -   -   -        -   RAID5 Optl  N  6.983 TB dflt N  N   dflt N      N  
 0 0   -   -        -   RAID5 Optl  N  6.983 TB dflt N  N   dflt N      N  
 0 0   0   62:1     2   DRIVE Onln  N  1.745 TB dflt N  N   dflt -      N  
 0 0   1   62:2     1   DRIVE Onln  N  1.745 TB dflt N  N   dflt -      N  
 0 0   2   62:3     0   DRIVE Onln  N  1.745 TB dflt N  N   dflt -      N  
 0 0   3   62:4     5   DRIVE Onln  N  1.745 TB dflt N  N   dflt -      N  
 0 0   4   62:5     6   DRIVE Onln  N  1.745 TB dflt N  N   dflt -      N  
 0 -   -   62:6     4   DRIVE DHS   -  1.745 TB -    -  -   -    -      N  
---------------------------------------------------------------------------

This is all for today, I hope this information will help you to use your Megaraid controller with new versions of Linux. I recommend that you familiarize yourself with StorCLI and all the possibilities you have to interact with the Megaraid SAS 9341-8i before you start using it in production.

Footnotes