AMCC 3ware 9500s RAID controller

See http://www.3ware.com/products/pdf/HWvsSW_111804.pdf for their overview.

Overview

The 3ware 9500s series of RAID cards use SATA drives to produce RAID 0, 1, 5, 10, and 50. The card can be used in 32 and 64 bit PCI slots. The card supports up to 64-bit PCI-X-133. They do not work with the new PCIe busses though, since they were designed before PCIe came out.

SATA II is not supported. You will need to set the jumpers on your drive so that they do not cause problems with the card.

There are only 3 stripe sizes allowed, 16, 64, and 256K. The board comes with 128MBs of RAM which is apparently expandable with ECC or DRAM. The maximum allowed memory is not clear. Official claims state 256 or 512MB. However reviewers claim they can put in 2GBs of RAM.

Performance tuning this card consists of upgrading the firmware, drivers, and then issuing "blockdev --setra 16384 /dev/sda". 3ware recommends XFS as the file system and to play with the stripe size to find what works best in that configuration. If the drive isn't set aside for a particular application then run the utility "bonnie" to determine various throughputs to get the best configuration.

Installation

Pick the PCI slot nearest the CPU is an often used rule of thumb to get maximum performance. SATA cables can be installed in any port and in any order. So if you have 4 drives you can stick them on ports 0, 1, 2, and 3 or 11, 10, 9, and 8. And if you discover it would be more convenient due to physical cable issues to mount them in another order after you have made your RAID volumes just do it. The card automatically handles this.

The Battery Backup Unit, (BBU), is a bit difficult to get on. I had to actually remove the metal clip on the side that you screw down to get it on. Then I read the manual and discovered my extreme approach is exactly the right way to do it.

Before connecting the drives you can update the firmware on the drive. Go to 3ware's site and download their utility. It can be run directly from your operating system. No messy floppy drives! The firmware upgrade also updates the BIOS on the board that you will probably use to create your RAID drives.

Once this is done then follow the procedures or whatever makes sense. No instructions are needed, everything is self explanatory from this point forwards.

Benchmarking

For AMCC recommendations see: http://www.3ware.com/LInuxSell_0629.pdf

Based on http://www.3ware.com/products/serial_ata9000.asp stated benchmarks:

"Tested using Intel’s Iometer(R) benchmark program on a dual processor Intel(R) Xeon (2.4GHz) system with PhoenixBIOS(R), 512MBytes system RAM, Microsoft(R) Windows(R) 2000 Advanced Server, the 3ware 9500S-8 66MHz SATA RAID controller and Maxtor(R) 7Y250MQ, 250GB, 7200 RPM drives"

They must be using one of the following cards

Manufacturer	Model	Chipset	BIOS	BIOS Version	PCI Slot	CPU	St Price (07/04/07)
Intel	SE7501CW2	Intel E7501	Phoenix	86B.0022.P04	64-bit PCI-X-133	Dual Xeon	$320
Intel	SE7505VB2	Intel E7505	Phoenix	blank	64-bit PCI-X-133	Dual Xeon	$380
SuperMicro	X5DA8	Intel E7505	Phoenix	Rev 1.0	64-bit PCI-X-133	Dual Xeon	$490
SuperMicro	X5DPE-G2	Intel E7501	Phoenix	Rev 1.0	64-bit PCI-X-133	Dual Xeon	$440
SuperMicro	X6DAE-G	Intel E7525	Phoenix	REV 1.0a	64-bit PCI-X-133	Dual Xeon	$400
SuperMicro	X6DAi-G	Intel E7525	Phoenix	1.2c	64-bit PCI-X-133	Dual Xeon	$380
SuperMicro	X6DH8-G	Intel E7520	Phoenix	blank	64-bit PCI-X-133	Dual Xeon	$480
SuperMicro	X6DH8-G2	Intel E7520	Phoenix	1.2c/6.1	PCI-x 133 & PCI-E x8	Dual Xeon EM64T	$500
SuperMicro	X6DH8-Xb	Intel E7520	Phoenix	blank	64-bit PCI-X-133	Dual Xeon	$520
SuperMicro	X6DH8-XG2	E7520	Phoenix	blank	64-bit PCI-X-133	Dual Xeon	$1300
Tyan	S2603	Intel i860	Phoenix	1.07	64-bit PCI-66	Dual Xeon	$???
Tyan	S2665ANF	Intel i7505	Phoenix	1.05.2665	64-bit PCI-X-133	Dual Xeon	$410
Tyan	S2668AN	Intel i7505	Phoenix	1.03	32-bit PCI	Dual Xeon	$250
Tyan	S2676	Intel i7525	Phoenix	2	64-bit PCI-X-133	Dual Xeon	$???
Tyan	S5360G2NR	Intel i7520	Phoenix	V1.00	64-bit PCI-X-133	Dual Xeon	$390

The hard drive they quoted as using in the above quote, copied and pasted directly from their website gotten to by URL included above, does not exist. However if you look at their supported hardware PDF they must be referring to the following drive: http://castle.pricewatch.com/s/search.asp?s=7Y250M0 which as of 07/04/07 goes for about $69. (The highest rated retailers were three 4.5 star out of 5 star dealers. Two of those three 4.5 star dealers sell the 7Y250M0 at $69. The cheapest but with only a two star rating was $46.)

Benchmarking Overview

Run a benchmark test on one of the drives you are going to use to create a RAID device with. Do it using the motherboards normal SATA connection. On Ubuntu install bonnie, sudo apt-get install bonnie++, and then run it. I recommend the -f option otherwise you may be waiting a couple of hours for the results. So type something like "script; date; time bonnie -f; date; exit". Before you do that though open another terminal and enter "tail -f /var/log/messages" so you can immediately see if the drive is giving you any problems. Now kick off bonnie. You'll want to store the results in a spreadsheet so you can compare it with the RAID results.

Now make a RAID 1 configuration and repeat the test. Now do the tuning they recommend which consists of entering "blockdev --setra 16384 /dev/sda". Repeat the performance test and record the results once again into the spreadsheet. Now pull one of the sata cables off the RAID drive and see what happens. Does the system crash or continue? It's suppose to continue on.

Before doing the rest of the tests make sure you are running with your motherboards latest BIOS, your OS's latest and greatest, and call the HD manufacturer to determine how to get the latest firmware on your hard drive. You don't want to have to repeat the time consuming tests.

Benchmarking Results

To understand the results from the utility bonnie see the man page. Online man pages can be found here: http://man-wiki.net/index.php/1:bonnie

Please contribute your results to this wiki page so that others can gain from your experiences. Include the motherboard, bios version, 3ware card, it's bios version, hard disks, their bios version, the RAID configuration, stripe size, and any tuning parameters used to achieve the bonnie results shown.

I wrote this wiki because the performance of the card was far far below what I expected. As a result I kept reading in an attempt to figure out where the problems are on my system. Basically I went from a single disk reading and writing at about 40MB/s to a RAID5 system that writes at about 6MB/s and reads at about 80MB/s. Using the same disks. I now suspect the problem is with the motherboard. I had assumed the RAID card would give any motherboard the ability to read at 400MB/s and write at 100MB/s. So my 1900MHz CPU on a A7N8X Deluxe 2.0 ASUS motherboard, probably a 32 bit pci slot running at 33MHz, just can't handle it. Why the card slows down the writes is confusing. But oh well.

Trouble Shooting

Ok, first off, you want to only buy disks that are supported and found on their supported webpage at http://www.3ware.com/products/sys_compatibility.asp otherwise when you call them they will claim the problem is with your hard disk. Yes, they are very very quick to blame any problems on somebody else and dismiss you. Expect that and you won't have to get your expectations constantly lowered. Think that's bad? That's not even the half of it. You've got to have one of their approved motherboards too! Finally you've got to have an OS that's supported on their web page too! Would you believe Ubuntu is not supported?!?!

Secondly, buy a speaker phone. Getting through to their support takes lots of time. Getting through to any support takes lots of time. You can't hold the phone up to your head that long! It's much more relaxing to listen to whatever music they may play when you are busy doing other things. So go out and buy a cheap AT&T 1855 for $40 at the local Staples if you don't have a speaker phone yet!

Now make sure you do everything while monitoring via the "tail -f /var/log/message" file so you can see any problems that crops up.

Also download the CLI interface and get use to it. You can install it in any directory and then type "tw_cli" to get it going. All the commands are fairly obvious and soon you'll be using it without any trouble. (Now don't expect their support to know the commands, believe it or not they are supporting so many products they don't get down to the CLI interface very often.)

Remember the 9500s card is only SATA-1 and if your hard disks are using SATA-2 speeds you'll get into trouble. So set the jumpers on the drives to use only SATA-1 speeds. This has been confirmed to fix issues of arrays failing to initilise and/or degrading prematurely, when using the 9500S with SATA-2 drives.

When all else fails, turn off the caching on your drives, any queuing your MB may be doing, and the APICs. You do the latter two from your mother boards BIOS and the first from the 3ware BIOS.

From personal experience I know that unless you pick motherboards and hard disks from their recommended list you can expect your multi disk RAID 0 configuration to run SLOWER than one of the same disks via your motherboards SATA connection.

Useful Contacts

3ware support: 858-535-6517 Monday thru Friday, 7:00am to 5:00pm, Pacific Time (excluding US holidays).

Maxtor support: 1-8002maxtor (First level support seems to be from India.) The lady asked for telephone number, name, and email address, model, serial number. They must use VOIP for their phone connection and it was awful. The lady didn't know how to spell out phonetically and she wrote things down wrong repeatedly. Once you make it clear firmware updates are needed you get transferred to 2nd level support where somebody who can actually understand English will ask you all the details again. They must have a high turnover rate because it took them days to send the right upgrades. It seemed like it was the first time they were doing it. And they don't have the patches on the web freely available for us competent people. With many emails between us I was finally able to upgrade the drives several days later.

Conclusions

When I looked into first getting a RAID card I was fairly excited. I thought I would be getting 400MB/s reads and 100MB/s writes with 3% CPU utilization. And if I were not then it would only be a matter of adding more drives to the mix. Seeing what looked like incredible prices for 200MB disks and great prices for 12 SATA port cards on Ebay I proceeded to spend my money. That I could basically extrapolate the performance I got on one disk to multiple disks to the point I was happy with the speed.

This hasn't been the case at all. There have been all sorts of unexpected hiccups. I'll name a few here:

• 3Ware passing the buck. Who would have known "This information is not intended to imply that AMCC Storage endorses, or chooses not to endorse, any particular vendor or product." verbage found on their system compatibility page, http://www.3ware.com/products/sys_compatibility.asp, meant the support center will not help unless you are using hardware from that page. (I'm probably making a false assumption in assuming they would help if the product were on that page.)
• That the 3ware card would disable the motherboards SATA and EIDE controllers.
• That performance would be worse or about the same as using just one of the drives connected to the Motherboard's SATA.
• That they do not support Ubuntu despite it being on their list of supported operating systems. Apparently there are different levels of "support"...
• That the CPU usage is huge, no where near the 3% claimed, when tests are done via bonnie. (I added bonnie results here but some art major deleted them mistakenly thinking somebody was trying to ruin the site because he didn't understand the lines of numbers. Jeesh.)

I suspect this RAID controller would give performance as stated on the 3ware site if you were using the right hardware. I have now got an RSS feed into Craigs list so if anybody sales that hardware near me I can contact them and hopefully give it a try before I spend more money. It might be most economical to buy the hardware used as a complete system. After all, this card is meant to work with PCI-X busses which are no longer being produced in favor of the newer bus style PCI-E.

The final conclusion... It might be far better to buy 3Wares latest card, the 9650SE, which gives you RAID6. They claim you can read from it at 800MB/s and write at 600MB/s. Lets assume again that they have overstated themselves by a little more than a magnitude, like they have done with this card, which would leave you at read and write speeds of nearly 80MB/s and 60MB/s. That would not be too bad. The card is more pricey but the motherboards required are much cheaper. So for about the same price as getting this old technology off Ebay you can get a faster more robust solution with the latest hardware.

I'll probably always use this card now. Being able to chain multiple disks together is very nice. Even if they do not supply a working GUI to control the RAID configurations. It's liveable. I will however buy their latest card when I need to upgrade...