Finally got some parts in for the 520 that allow me to start getting it put together. I have half of the SAS hard drives, the SD card connector(don’t want to say SD card card..<G>), an extra 64 gigs of ram, all of the drive caddies, and of course the SD cards themselves.
- Read the previous post to see what I had done before this post..:)
- Installed the extra 64GB of ram. Server total is now 104GB.
- Installed the SD boot card. Also installed the two SD cards so I now have a boot mirror(though not ZFS managed..but it will do)
- I have received 4 of the 6TB SAS hard drives and the caddies. Installed 4 of the drives into the new server to begin testing
- setup the IDRAC remote console and setup the internal nic to be on the internal VLAN.
Here’s where the adventure truly began. The IDRAC virtual console and virtual media abilities make this really easy. At first the IDRAC was terribly slow in data transfers. Turns out not only does the IDRAC send everything over https…but by default it also encrypts the video feed as well. This absolutely crushes the poor CPU in the IDRAC. Turning off the video encryption brought transfer speeds to something decent instead of 10 Mbps over a gigabit connection. Once I got that figured out I could begin installing FreeNAS…or I thought I could. My first time trying to install to the SD card was met with read and write errors and the installer aborted and started over. Multiple times. I tried swapping the cards..different settings…all kinds of stuff..the system simply refused to work correctly. I then decided for giggles to try a USB install. It was terribly slow. What’s weird is it was more than 60% slower the 310!. Why was the 310 USB usable but the 420 and 520 not? The boot was so slow the system would fail and not boot..or boot into an unusable state. Again, the 310…no problems. The 520..USB boot is useless. I finally said enough of this (after screwing around for 12 hours) and tore the server apart all the way down to the mainboard. I yanked everything out. I then, one by one, started putting the machine back together. Now mind you I have not installed any hard drives. FreeNAS does not require the storage array to install and boot. Once I had spent an hour going piece by piece by piece…I fired the machine up. It definitely seemed to boot a bit faster (a machine like this still take 5+ minutes to do all of the internal checks and stuff no matter what)…so I tried the install again. USB was trash so no-go there. Why was this not working. The exact same USB sticks work fine on the 310 but not the 420 or 520…what is the commonality? It’s the IDRAC. Both of these machines have extensive IRDRAC capabilities with the Enterprise License..which all of my machines have. The capabilities though on the 3xx are much less than the 4xx and above. The IDRAC does occupy the USB bus both virtually and physically. It seems the IDRAC is causing severe signaling issues with high speed transfers in the 420 and 520. I did some research and others have tried using USB boot on 4xx server and higher with FreeNAS and hit the exact same problems. It’s anecdotal…but enough folks have hit these performance issues enough times it seems that’s the answer. Booting to SSD or something else other than the USB bus the IDRAC also occupies gets around the issue.
I then switched to trying to install to the SD card…only to get an error about existing partitions not being able to be deleted. NO worries. switched from booting from the FreeNAS ISO to the GParted ISO and whacked all partitions on the SD card. Switched back to booting from FreeNAS and tried again. For whatever reason the install was speedy fast. Not SSD fast but very quick..like 10 minutes quick. I thought I had it in the bag and then my remote console said no signal. Somehow the DRAC had locked up and now I could not log in. Because the IDRAC was also presenting the ISO as a cdrom drive the install failed. SERIOUSLY? Luckily “resetting(aka rebooting) the DRAC is easy without having to powercycle the entire server. I found the IDRAC reset button on the 520 and held it in. The fans spun up which told me the IDRAC had reset. After 2 minutes I was able to log in. I had to boot into GParted to nuke the partitions on the SD card and then switched to the FreeNAS ISO. This time the install went perfectly without a hitch.
Now it is time for some operational tests. I logged into the web interface of the 520 and setup the first testing pool. 4 x 6TB SAS drives in two mirrored VDEVS. This gives me @10 TB of usable place to play with. I built the pool and then setup ZFS replication from the 310 and pointed it at the 520. While the concept on replication has not changed in a long time the implementation suing the new web interface has. It took a few times but i now have my nearly 2TB copying from the 310 to the 520. Once that completes, I can then tear the pool down and redo it once the last 4 hard drives arrive tomorrow. Then I will have @20TB of usable storage. Over the weekend I will clear out the rack, rack the 520, re-rack the rest and then start the replication again. Once it completes i then shutdown all machine that access the 310. i then launch a final manual replication to get any changes that were made in the interim. I then take a config backup from the 310 and import it into the 520. With that the 520 literally turns into the exact same config as the 310 and then I SHOULND’T have too much left to get things back online. We shall see. The project is nearing completion. There’s still quite a bit of work to do..but there’s a light at the end of the tunnel..and it’s not a train..:)