Subscribe to News feed

Building a high performance Dual Xeon E5 Server / Workstation from scratch

Posted at: 15:56 on 29 October 2012 by Muhimbi

CaseUnlike 90% of the other posts on this blog, this post is not about SharePoint or our range of server based PDF Conversion products. Unless you have a (casual) interest in building high performance systems, you may want to skip straight to the PDF Converter Scalability tests we executed on this new machine.

So, if you are still with me, let’s talk about building some nice hardware. At Muhimbi we are big fans of virtualisation so we already have some nice hardware to run a large number of Test and Development environments on. However, the Dell T605 Servers we bought a couple of years ago are becoming a bit long in the tooth, are difficult to upgrade and are extremely ‘jet-engine’ loud (Dell sells them as ’silent servers’, right!).

So we decided to get some new kit. There always is the option to buy something off-the-shelf from HP, but for something with these specifications that would be extremely expensive. It wouldn’t give us a high level of control either and would probably be very loud as well, so we decided to go for a custom build. We approached Boston, Broadberry and Armari, who all provided competitive quotes for custom builds of silent servers. If you have no desire to build your own system, or if you are not a complete control freak, then by all means contact these guys. In the end we decided to completely build our own system, not because it would be cheaper (it was), but because it would give us ultimate control and be a lot of fun as well.

 

Buying the hardware

So we freshened up on the latest Intel Xeon processors and bought the following hardware. Most of it was ordered from Scan, but they let us down with the delivery because they couldn’t source the Xeons quickly enough.

  • Dual Xeon E5-2670: Purchased from Ballicom, one of the very few suppliers with stock. These processors are extremely pricey (£1080 / $1740 each, before tax), but the E5-2670 offer the best performance ratio without paying an additional premium.
  • Asus Z9PE-D8 WS: The choice of dual Xeon boards is extremely limited. We decided to go with Asus because they have a good track record for both compatibility and extensibility.
  • 8 x 8GB DDR3 Hynix: The maximum memory supported by the motherboard for ECC Registered memory is 64GB (256GB for non-ECC). We decided to go for the Hynix 1600MHz ECC Registered memory because it it very affordable and available from Scan.
  • 2 x OCZ RevoDrive3 X2: We don’t need crazy amounts of storage, but as the system will run a number of operating systems in parallel, including databases, we need storage to be fast with high levels of concurrent IO. The logical choice is SSD drives and the fastest, of the affordable, SSD Drives are these PCI-Express based cards that bypass the SATA bus all together. Both 480GB cards will be installed in a RAID-1 (mirrored) configuration. We bought these cards from Kikatek who had them in stock at a great price.
  • 2 x Samsung 830 SSD: We need something to boot from and they almost give these drives away. They perform brilliantly on SATA 6Gbps so we installed two 128GB units in a RAID-1 configuration. Purchased from Scan, but they were cheaper on Amazon.
  • 2 x Zalman CNPS9900-MAX: CPUs need to be cooled and these CPU coolers work very well. They are quiet and keep the cores below 50 degrees centigrade under load. We bought these from Scan and purchased the Socket 2011 support clips for this cooler from Quiet PC (unnecessary as the clips were included in the box, but that was not clear from the description). The CPU coolers were purchased from Scan as well.
  • Corsair AX850 Power supply: This professional series power supply is highly efficient and very quiet. Purchased from Scan as well.
  • Fractal Design Case: We need to put everything in a box so we bought this stylish looking Define XL computer case. It supports the E-ATX form factor and comes with quiet fans. Purchased from Box.co.uk who had it in stock.

That is all, you don’t need anything else other than perhaps a phillips screw driver. All cables, including plenty of SATA cables, are included with the various components.

All componentsThe motherboard in detail

 

Building the System

If you have never put a PC together than you may feel a bit discouraged by the sheer amount of cables, connectors, little screws and components littering your floor after opening the boxes (The picture below was taken after cleaning most of it away). However, with the help for someone experienced, or strangers on the internet, it is not too difficult to put together. There is no ‘Ikea like’ step-by-step guide, but after reading the documentation (and Googling like crazy) things quickly fall into place.

I won’t bore you with the details, but  a few issues deserve mentioning:

  1. EEB Form Factor: The Asus motherboard uses the EEB form factor, which is usually reserved for servers. It is not easy to find a case that uses this exact same form factor, but you can buy a regular E-ATX case. The only real difference is that not all of the holes in the motherboard match the case, only 7 out of 11. This is good enough, just don’t blindly screw all 11 risers into the case as the 4 that do not have a corresponding hole in the motherboard will most likely cause a short circuit on the back of the board.
  2. Direction of CPU Coolers: Each CPU cooler has a big fan that can be pointed either left or right. My concern was that if I both point them in the same direction then the ‘hot air’ from fan 1 would be blown into CPU / Fan 2. I went ahead with this anyway as both fans blow towards the nearest and most powerful case fan.
  3. I/O Shield: The motherboard comes with a nice ‘plate’ to cover the block of ports at the back of the motherboard (USB, LAN, Keyboard etc). I was unable to make this fit, there simply was no room in the case. Not the end of the world, but I would have preferred less open air holes in the back.
  4. Fan connectors: The Fractal Design case fans have 3 pins while the motherboard has 4-pin connectors. Don’t worry about it, just plug them in, they only fit in one way. The industry cannot decide on a single standard apparently.
  5. Power plug connectors: Similar to the Fan connectors, there doesn’t appear to be an agreed standard for power connectors to the CPU. The Corsair power supply comes with 2 ‘EPS’ connectors while the motherboard requires 2 ATX12V connectors. Turns out these are the same, but as the pattern on the plugs (defined by square and round pins) doesn’t match I hesitated for a full 24 hours before just forcing the plugs in. I wasn’t too keen to fry £2500 worth of motherboard and CPUs. After contacting Corsair they told me they get that question a lot and that you can push the 8 pin EPS Connector into the ATX12V slot. It worked, pfew.
  6. System panel connectors: Various little wires come out of the case to deal with the reset and power button as well as the case light. They are clearly labeled, but have a good look at your motherboard manual and remember that white & black both mean ‘ground’. (A racial equality decision, hurrah!)
  7. Raid select: The documentation for the Z9PE-D8 WS motherboard is incorrect. On page 2-28 it shows the jumper for the RAID select at the bottom, but it is located just right of the Intel C602 chip (level with the top most SATA port on the right-hand side). A digital copy of the documentation can be found here and allows you to zoom all the way in.

 
These were the only real stumbling blocks encountered (well, and one of the RevoDrives being DOA). It was our expectation to put it all together in 4 hours, but the problem with the power plug connectors delayed successful completion by 24 hours.

The office floor as it looked AFTER cleaning up

 

Installing the OS / Software

With the system built we were keen to get the OS installed to carry out a burn-in test and run some benchmarks. We decided to install Windows Server 2012. As we don’t have a DVD drive in the server (it is 2012 for god’s sake) we used the Windows 7 USB Download tool to create a bootable Win2K12 installer on a memory stick. Works like a charm!

Unfortunately the Windows 2012 installer crashed spectacularly. It turns out that the default BIOS of the Z9PE-D8 WS motherboard is not compatible with Windows 8 / Win2K12. Downloading and installing the latest BIOS solved the problem. Just make sure you write it to a FAT/FAT32 memory stick or you get the 'Rom file is not an EFI BIOS’ message. (The Win2K12 USB Stick uses NTFS, so don’t use that to upgrade the BIOS.)

As the boot volume is mirrored (in our case) we downloaded and unpacked the latest drivers for the motherboard and placed them on the same USB stick as the Windows Installer. Depending on which RAID solution you pick (the motherboard has 3) you may be asked to load a driver during installation. A separate blog post will go deeper into the topic of RAID and SSD and why not to use the on-board LSI RAID solution (seriously, don’t do it, it is painfully slow, pick the Intel or Marvell based one).

Even though all storage drivers were present on the USB stick Win2K12 installation failed as we received the “we couldn't create a partition or locate an existing one” error when selecting the installation disk. It turns out that the RAID1 boot disks could not be selected when the RevoDrive is also present in the system, so we temporarily disabled the PCIE Option Rom in the system BIOS for the slot that contains the RevoDrive.

With that change made Windows installed smoothly and quickly. That was until we enabled ‘Event Logging’ in the BIOS, which caused the Win2K12 boot process to slow down to a crawl (Another 24 hours lost before we figured out this was the cause).

Finally, we ran a Burn in test for 24 hours to check the system does not overheat under stress. This test was executed by running Prime 95 for several hours and measuring the temperature for each CPU core using CPU Thermometer 1.2. The results are impressive, CPU Core temperatures are between 18 and 30 degrees (centigrade) when idle and between 39 and 50 degrees under maximum load.

The ambient noise level in the office is 40dB with the server off (same as a Library) and, measured at 1 meter distance, 45dB with the server on (same as a Library, just including the background bird calls).

 

Any questions or remarks, did we do anything stupid or wrong? Let us know in the comments below.

.

Labels:

29 Comments:

  • May I know what graphic card that you used ?

    By Anonymous Anonymous, At 21 February, 2013 21:36  

  • As this is not used as a graphics workstation we used a cheap graphics card. I had a look at the parts list and it is a '512MB EVGA 8400 GS, 40nm, Mem 1200M'

    By Blogger Muhimbi, At 22 February, 2013 10:11  

  • Thank you for sharing the information. This is an awesome article !

    By Anonymous Anonymous, At 22 February, 2013 15:38  

  • Can you please help to explain why you choose Xeon E5-2670 ? I don't understand what you mean by " it offer the best performance ratio without paying an additional premium." ?

    By Anonymous Anonymous, At 22 February, 2013 18:36  

  • Well, it is "the best price performance ratio", "bang for your buck" etc.

    Basically what Intel does is that hey charge a premium for the very fastest processor in a range. Only a small increase in performance, but an enormous price difference. Their thinking is that if you really need speed you are almost willing to pay anything for it.

    Now, we like a speedy system, but if we have to pay a large sum extra for perhaps 5% extra performance then it is not worth it. So we looked at all prices and clock speeds and decided that the E5-2670 is fast enough for our needs and fits our budget.

    That's it really.

    By Blogger Muhimbi, At 22 February, 2013 18:51  

  • Thanks for the detail explanation. I saw there are processors listed for ES version, is that ok to buy those ? What is the different between the one with the retail package ?

    By Anonymous Anonymous, At 22 February, 2013 19:49  

  • 'ES' processors are a lot more affordable, but as these are 'Engineering Samples' I would never put these in a production system. However, if you buy 2 of them from the exact same source then you should be OK if you use the system for personal use.

    We briefly considered buying ES versions due to the considerable price benefit, but decided against it after reading up on it.

    By Blogger Muhimbi, At 25 February, 2013 10:41  

  • Thanks for the advice.

    By Anonymous Anonymous, At 25 February, 2013 19:20  

  • Do you think is it ok to put in the processor on a separate time ? Buying two processors at once cost a lot of money. Is there any consequences if I do so ?

    By Anonymous Anonymous, At 25 February, 2013 20:49  

  • You can start with one processor to save cost, but if you don't see yourself upgrading to a second one then you might as well save yourself a whole lot of money and go with a Core i7 processor.

    If you end up going for Engineering Samples then I would not buy them one at a time as the first ES may not match the ES you end up buying later.

    By Blogger Muhimbi, At 26 February, 2013 10:26  

  • May I know if I use single Core i7 and later on down the road I changed the processor to Dual Xeon E5-2670 when they are cheaper, what will happen ?

    By Anonymous Anonymous, At 05 March, 2013 19:52  

  • An i7 will not work in this motherboard. If you want to go down that road you will need to purchase an i7 compatible board, which will save you even more money. There are no dual i7 boards.

    By Blogger Muhimbi, At 06 March, 2013 11:21  

  • Hi, I am building similar system, it would be very nice if you can help
    1. Why do you need 2 x Samsung 830 SSD? I've bought one OCZ RevoDrive3 and on it it says it's bootable, can I boot directly from it on Asus Z9PE-D8 WS?
    2. I bought Engineering Sample ES Xeon 2690, should it work with it? (at least with single configuration)
    3. I didn't buy matherboard still, is there now any better than Asus Z9PE-D8 WS dual mb?

    Thank you very much, any other suggestions are very welcome,
    Max

    By Anonymous Anonymous, At 12 May, 2013 16:51  

  • Hi Max,

    I answer to your questions:

    1. In case one dies. It is a boot drive and it is essential that this machine boots. If you don't care about uptime then use a single one.

    2. See comments above about engineering samples.

    3. Don't know. This Mobo is great and will most likely have come down in price since last year.

    Good luck.

    By Blogger Muhimbi, At 13 May, 2013 10:15  

  • Hi, any recommendation for redundant power supply?
    Thx
    Kevin

    By Anonymous Anonymous, At 12 June, 2013 03:31  

  • Hi Kevin,

    I am afraid I have no recommendations regarding a redundant power supply.

    By Blogger Muhimbi, At 12 June, 2013 09:41  

  • HOW MUCH?

    By Anonymous Anonymous, At 02 September, 2013 22:43  

  • I assume you are asking about pricing. This system was put together exactly a year ago and if I remember correctly it was roughly £4000 at the time. A similarly specced system should be about £1000 less right now. (This is all guesswork)

    By Blogger Muhimbi, At 03 September, 2013 09:27  

  • Hi, thanks for sharing.
    I have a similar task and this is very helpful.
    As I was looking at the places now (as if I would buy the parts now) I see that it would be about 2,967.75 excl. VAT, so 3,561.26 incl. VAT.

    How much was it back when you bought it?

    By Anonymous Anmatr, At 03 September, 2013 14:02  

  • Hi Anmatr,

    What I mentioned in my previous reply is the best answer I can give. It looks like my estimate of £3000 (ex VAT) for the current price of the parts was spot on and sounds very reasonable.

    By Blogger Muhimbi, At 03 September, 2013 14:07  

  • Yes, spot on estimate - funny as our messages crossed.

    By Anonymous anmatr, At 03 September, 2013 14:22  

  • Correction: There was only one CPU counted on the price of 3,561.26 so add anothere 1244.88 to it which makes it 4806.14 GBP by current prices.

    Anmatr.

    By Anonymous Anmatr, At 07 September, 2013 09:40  

  • Hi Muhimbi, I'm also building a workstation at the moment and would like to ask for your advice.

    Processor: Intel Xeon E5-1660 @ 3.30GHz
    Motherboard: Z9PE-D8 WS
    Case: Cosmos II
    RAM: Crossair vengeance 8x4 (32)
    Hard Drives: 2/1 250GB Samsung 840 EVO series
    Power Supply: Corsair ax860

    OS: Windows 8
    Usage: Personal and Work related like running VM for SharePoint servers.

    Need your advice if this is enough for my basic usage and if you have any recommendation. Since this is my first time and want to built my own workstation with my own money.

    SharePoint fanatic.


    By Anonymous Anonymous, At 17 October, 2013 05:00  

  • Hi SharePoint fanatic,

    If you are only going for a single Xeon then this motherboard is overkill. In that case I would actually go for a high-end i7 and invest the money saved (a lot) in some extra Samsung EVOs and Stripe / Mirror them or buy a Revodrive in addition to the 2 EVOs on your current list.

    I believe you need WIndows 8 Pro in order to run Hyper-v locally, so make sure you get the right version.

    Good luck. Sounds like a cool project.

    By Blogger Muhimbi, At 17 October, 2013 09:21  

  • Hi Muhimbi,

    Oh that's great. I'm having a second thought in regards to the specs in my list. Thanks for the advice. It's a good thing that I asked you for an advice before buying the parts. This post is awesome! Can you please give me a hand in building a high-end i7? If you don't mind. I'm not a hardware person.

    By Anonymous Anonymous, At 17 October, 2013 22:11  

  • Just Google for "i7 build log" and you should find some results (including many YouTube videos). Buy your parts from http://www.scan.co.uk/ to make things easier. Decent prices with pretty good support as well and I believe even some kind of 'system builder wizard'.

    Alternatively contact Boston, Broadberry or Armari (links in the introduction of this post) as they have wizard like web pages as well. This will be more expensive than building your own and less fun (and less frustration :-)

    By Blogger Muhimbi, At 18 October, 2013 10:01  

  • Hi Muhimbi,

    Can you please give an advice to this new specs for my machine.

    Processor: I7 4930k
    Motherboard: Z9PE-D8 WS / P9X79 WS / Rampage IV extreme
    Case: Cosmos II
    RAM: GSkill - ripjaw (32GB)
    Hard Drives: 2/1 250GB Samsung 840 EVO series
    Power Supply: Corsair ax860
    GPU: No Idea
    Cooling: No Idea
    ODD: No Idea


    SharePoint Fanatic

    Thanks in advance!

    By Anonymous Anonymous, At 24 October, 2013 03:07  

  • Hi,

    The Z9PE-D8 WS is not compatible with the i7. You have to make sure you purchase parts that are compatible.

    It is probably best if you seek external advice / purchase from systems builder.

    By Blogger Muhimbi, At 24 October, 2013 07:41  

  • Ok. thank you very much for the help Muhimbi.

    By Anonymous Anonymous, At 24 October, 2013 23:37  

Post a Comment

Subscribe to Post Comments [Atom]

Links to this post:

Create a Link

Need support from experts?

Access our Forum

Download Free Trials