Xeon Phi Co-processor Testing, Part 1 - Rocket Science

Trang chủ » Phi 71s1p » Xeon Phi Co-processor Testing, Part 1 - Rocket Science

Có thể bạn quan tâm

Search This Blog

Wednesday, March 21, 2018

Xeon Phi Co-processor Testing, part 1

I won an auction for 12 71S1p xeon phi coprocessor cards. These are awesome...basically a 61 cores @ ~1.1GHz linux server with on-chip ultra high bandwidth memory in a double wide pcie card package. They're very picky about motherboards and thermals though. The motherboard must have "above 4G decoding" or "large memory BAR support" or something like that. Most supermicro's do, and all of the ASUS WS's seem to. Even then, it's not guaranteed all phis will work. The "p" phis are passively cooled, which means they're really meant for server applications. You can create some cooling fan ducts for them, or buy or 3D print them. I tested them with another DL380p G8 (with 2x E5-2690's I later harvested) I had purchased. I bought the complete GPU package (see this post) for it, but the DL380p G8 only supports the 5110p Phi, which has a lower wattage rating than the 71S1p. I hacked (shorted the sense pins) an extra ATX PSU to power them, with the cables coming out of the other PCI riser's slots so I could close the lid. The testing and firmware update procedure was fairly straightforward once I figured it out. This can probably be adapted for your own system. Most of this follows the readme text file and user guide that comes with the MPSS software.
  1. Update DL380p's firmware
  2. Install CentOS 7
  3. Install a Phi
  4. Change bios settings to enable large BAR support (in advanced menu) and set fans to max.
  5. Disable SELinux (re-enable after done testing Phi's.
  6. login as root, create RSA key so can use SSH later. ssh-keygen . You want to do this before configuring MPSS for the first time, otherwise you have to manually load the key (see readme text file)
  7. Download the MPSS software, readme, and user guide. If your firmware is older than that in the readme, try starting with an older MPSS. If you're using a kernel that isn't listed, then you can recompile the rpms using the instructions in the readme.
  8. Install MPSS (see the readme and user guide). I suggest rebooting.
On the host, from a terminal run:
lspci | grep -i Co-processor
That will tell you which PCI port/slot thing its in. Mine was 24:00.0, so I did:
lspci -s 24:00.0 -vv
If lspci doesn't recognize it, then there's a problem with your card (assuming your motherboard is compatible). A likely culprit is thermal overload, especially if you're trying to use a passive "P" card without a cooling system. I actually went back to bios and enabled maximum cooling to help with this. If you have a desktop, you'll need to construct a custom cooling system (see above). Another possibility is that the card isn't seated well. Try reseating it. When none of that worked, I gave up on the card. I'm sure there is more advanced troubleshooting you could do, but I just don't know how to do it. Intel tech support seems to be pretty good, so it might be worth asking them. Next, type:
modprobe mic
This starts the mic process. If you have just installed or reinstalled MPSS, then you need to do:
micctrl --initdefaults
Then:
micflash -getversion
This must be 375 for the latest MPSS release. Mine were 390. Then:
micctrl -s
This should return "ready". I'm not sure what to do if it does not. Run:
micinfo -group Board
This should return a bunch of information about your Phi, though not all of it will be available because MPSS isn't running yet. Next:
micflash -update -device all -smcbootloader
Then restart the host, and:
modprobe mic
micflash -getversion
This should show the new firmware version. Next, start MPSS:
systemctl start mpss
Now you should be able to ssh into the Phi's filesystem:
ssh mic0
If that didn't work, you need to see the readme section about ssh keys and loading them. Now, from the host, run:
miccheck
This should show all passes. Then run:
micinfo
This will show a lot of information about your Phi. You can launch a monitoring gui with:
micsmc
That's it. If your Phi passed all of that, you should be able to install software on it. I haven't done this yet...that will be the topic of another post. You should also go to /etc/sysconfig/network-scripts/micX and change "onboot" to "no". I can't remember the exact reason for this, but it's in my notes. For my lot, after all was said and done, 8/12 were recognized by lspci and tested to be working. The lspci recognition was spotty, though...probably because these weren't 5110p's. I managed to sell all of them for a profit. I kept one, the 7110p that was in the lot. While probably not useful for conventional CFD, it might be useful for something like OpenLB or anything super vectorizable that needs more umph per core than a GPU can provide. They're also supposedly really good for mining some cryptocurrencies, though I haven't tried. Posted by at 1:22 PM Labels: cluster, homelab

No comments:

Post a Comment

Newer Post Older Post Home Subscribe to: Post Comments (Atom)

About Me

Jed Storey I like designing and building things. I need more space and machine tools... View my complete profile

Pages

  • Home

Projects

  • 3D printer (37)
  • cluster (33)
  • Cubex (10)
  • Cubexy (6)
  • DBF (1)
  • Edgerton Class 2010 (5)
  • EHB (7)
  • ELB (40)
  • garage shop (14)
  • home improvement (6)
  • homelab (48)
  • L3 Rocket (4)
  • Life (1)
  • LITE (3)
  • maglev (4)
  • Mechatronics (7)
  • Meteorites (1)
  • MHD (1)
  • night light joule thief (3)
  • openbsd (6)
  • python (2)
  • Rockets (5)
  • Sailboats (2)
  • table saw (3)
  • tools (1)
  • TV (1)
  • wanhao i3 (13)
  • woodworking (7)

Links

  • ELB Design Review

Subscribe To

Posts Atom Posts Comments Atom Comments

Followers

Blog Archive

  • ▼  2018 (62)
    • ▼  March (9)
      • Wanhao i3, first good prints
      • Wanhao i3: Thingiverse Things Reviewed
      • Wanhao i3, Part 2
      • 3D Printer Wanhao i3, Part 1
      • Homelab cabinet thermal managment
      • Building a small HPC cluster, part 1
      • Xeon Phi Co-processor Testing, part 1
      • Homelab Infiniband part 1
      • HP DL380p G8 GPU Options Explained

Từ khóa » Phi 71s1p