169 lines
4.2 KiB
ReStructuredText
169 lines
4.2 KiB
ReStructuredText
|
Rack 1:
|
||
|
|
*******
|
||
|
|
|
||
|
|
|
||
|
|
|
||
|
|
device classes:
|
||
|
|
===============
|
||
|
|
|
||
|
|
- fspineX: supermicro fiber switch
|
||
|
|
- cspineX: edgecore copper switch
|
||
|
|
- edgeX: border/network 1U server (heavy on CPU)
|
||
|
|
- hX...: hypervisor
|
||
|
|
- dX...: data node
|
||
|
|
- ipmiX: cheap switch
|
||
|
|
- bastion: cheap server with extra nics
|
||
|
|
|
||
|
|
|
||
|
|
|
||
|
|
|
||
|
|
server NIC naming and requirements:
|
||
|
|
===================================
|
||
|
|
|
||
|
|
- regular server:
|
||
|
|
* ipmi: ipmi port of the server
|
||
|
|
* mgmt: 1gig pxeboot capable copper nic for mgmt (ssh, etc) and pxeboot
|
||
|
|
* peth: 10gig frontend fiber (2 per server)
|
||
|
|
* ceph: 10gig backend copper (2 per server)
|
||
|
|
|
||
|
|
|
||
|
|
- edge server (special case):
|
||
|
|
* same links as regular server
|
||
|
|
* xlink: 10gig fiber iBGP cross link (ideally 2 per server, could be mix and match with fiber and copper as they just go direclty to the peer edge server)
|
||
|
|
* up: 10gig fiber DIA uplink
|
||
|
|
|
||
|
|
|
||
|
|
- bastion server (special case):
|
||
|
|
* ipmigw: 1gig copper nic for ipmi network
|
||
|
|
* mgmtgw: 1gig copper nic for mgmt network
|
||
|
|
* peth1: 10gig fiber nic to uplink to fspine1
|
||
|
|
* peth2: 10gig fiber nic to uplink to fspine2
|
||
|
|
* ceph1: 10gig copper to cspine1
|
||
|
|
* ceph2: 10gig copper to cspine2
|
||
|
|
|
||
|
|
|
||
|
|
|
||
|
|
|
||
|
|
node id:
|
||
|
|
========
|
||
|
|
- each chassis (regardless of switch or server) will need to get an ID assinged.
|
||
|
|
- the id should be a number from 10-255
|
||
|
|
- this ID will be used in several locations:
|
||
|
|
* BGP asn final digits (i.e. ID 121 -> ASN: 4200002121)
|
||
|
|
* loopback IP final digits (i.e. ID 121 -> IP: 10.1.2.121)
|
||
|
|
* mgmt IP final digits (i.e. ID 121 -> 10.0.1.121)
|
||
|
|
* ipmi IP final digits (i.e. ID 121 -> 10.2.2.121
|
||
|
|
|
||
|
|
|
||
|
|
|
||
|
|
|
||
|
|
IP allocations:
|
||
|
|
===============
|
||
|
|
|
||
|
|
|
||
|
|
- Factory (usw1):
|
||
|
|
* 10.1.0.0/24: loopback tier spine
|
||
|
|
* 10.1.1.0/24: loopback tier border
|
||
|
|
* 10.1.2.0/24: loopback tier 2
|
||
|
|
* 10.1.3.0/24: loopback tier 3 (arm tier)
|
||
|
|
* 10.1.4.0/24: loopback ceph
|
||
|
|
|
||
|
|
* 10.0.0.0/24: mgmt/ipmi
|
||
|
|
* 10.2.0.0/24: ipmi to be used
|
||
|
|
|
||
|
|
|
||
|
|
|
||
|
|
- DC (usw2):
|
||
|
|
* 10.1.16.0/24: loopback tier spine
|
||
|
|
* 10.1.17.0/24: loopback tier border
|
||
|
|
* 10.1.18.0/24: loopback tier 2
|
||
|
|
* 10.1.19.0/24: loopback tier 3 (arm tier)
|
||
|
|
* 10.1.20.0/24: loopback ceph
|
||
|
|
|
||
|
|
* 10.0.1.0/24: mgmt
|
||
|
|
* 10.2.1.0/24: ipmi
|
||
|
|
|
||
|
|
|
||
|
|
|
||
|
|
|
||
|
|
|
||
|
|
|
||
|
|
ipmi network:
|
||
|
|
=============
|
||
|
|
|
||
|
|
|
||
|
|
::
|
||
|
|
|
||
|
|
bastion:ipmigw +---+ eth48:ipmi1:eth1-20 +---+
|
||
|
|
+---- edge1:ipmi
|
||
|
|
+---- edge2:ipmi
|
||
|
|
+---- h1:ipmi
|
||
|
|
+---- h2:ipmi
|
||
|
|
+---- h3:ipmi
|
||
|
|
...
|
||
|
|
|
||
|
|
|
||
|
|
mgmt network:
|
||
|
|
=============
|
||
|
|
|
||
|
|
::
|
||
|
|
|
||
|
|
bastion:mgmtgw +---+ eth48:mgmt1:eth1-20 +---+
|
||
|
|
+---- edge1:mgmt
|
||
|
|
+---- edge2:mgmt
|
||
|
|
+---- h1:mgmt
|
||
|
|
+---- h2:mgmt
|
||
|
|
+---- h3:mgmt
|
||
|
|
...
|
||
|
|
|
||
|
|
|
||
|
|
|
||
|
|
|
||
|
|
|
||
|
|
frontend network:
|
||
|
|
=================
|
||
|
|
|
||
|
|
::
|
||
|
|
|
||
|
|
fspine1:swp48 +-----+ swp48:fspine2
|
||
|
|
swpX swpX
|
||
|
|
| |
|
||
|
|
+-+ peth1:edge1:peth2 +-+
|
||
|
|
| xlink up1 +---|-------+ patch1: evocative-uplink
|
||
|
|
| + |
|
||
|
|
| | |
|
||
|
|
| + |
|
||
|
|
| xlink up2 +---|-------+ patch2: TBD
|
||
|
|
+-+ peth1:edge2:peth2 --+
|
||
|
|
| |
|
||
|
|
++ peth1:bastion:peth2 ++
|
||
|
|
| |
|
||
|
|
| |
|
||
|
|
| |
|
||
|
|
| |
|
||
|
|
+--+ peth1:h 1:peth2 +--+
|
||
|
|
+--+ peth1:h 1:peth2 +--+
|
||
|
|
+--+ peth1:h 1:peth2 +--+
|
||
|
|
+--------- .. ----------+
|
||
|
|
|
||
|
|
|
||
|
|
|
||
|
|
backend network:
|
||
|
|
================
|
||
|
|
|
||
|
|
::
|
||
|
|
|
||
|
|
cspine1:swp48 +-----+ swp48:cspine2
|
||
|
|
swpX swpX
|
||
|
|
| |
|
||
|
|
++ ceph1:bastion:ceph2 ++
|
||
|
|
| |
|
||
|
|
| |
|
||
|
|
| |
|
||
|
|
+--+ ceph1:h 1:ceph2 +--+
|
||
|
|
+--+ ceph1:h 1:ceph2 +--+
|
||
|
|
+--+ ceph1:h 1:ceph2 +--+
|
||
|
|
+--------- .. ----------+
|
||
|
|
|
||
|
|
|