So You Want to Build Your Own Data Center

1. Installation

   1 166 18,067 9.9 Python

   The premier source of truth powering network automation. Open source under Apache 2. Try NetBox Cloud free: https://netboxlabs.com/products/free-netbox-cloud/

   

   Look at the issue list...that is why.

   https://github.com/netbox-community/netbox/issues?q=is%3Aiss...

   Note how they want to be "NetBox functions as the source of truth for your network infrastructure."

   Your individual situation dictates what is important, but had netbox targeted being a central repository vs insisting on not allow other systems to be truthful for certain items it could be a different story.

   We have learned that trying to centralize complexity and control doesn't work, heck we knew that almost immediately after the Clinger Cohen Act passed and even ITIL and TOGAF fully call this out now and I expect this to be targeted by consultants over the next few years.

   You need a central constant way to find state, to remove any questions or doubt regarding where to find the authoritative information, but generally if you aspire to scale and grow or adapt to new changes you really need to avoid having some centralized, god-box, and prescriptive system like this.

2. Stream

   getstream.io featured

   Stream - Scalable APIs for Chat, Feeds, Moderation, & Video. Stream helps developers build engaging apps that scale to millions with performant and flexible Chat, Feeds, Moderation, and Video APIs and SDKs powered by a global edge network and enterprise-grade infrastructure.

   

3. netboot

   2 4 1,543 3.8 Go

   Packages and utilities for network booting

   

   > It would be nice to have a lot more detail

   I'm going to save this for when I'm asked to cut the three paras on power circuit types.

   Re: standardising layout at the rack level; we do now! we only figured this out after site #2. It makes everything so much easier to verify. And yeah, validation is hard - manually doing it thus far; want to play around with scraping LLDP data but out switch software stack has a bug :/. It's an evolving process, the more we work with different contractors, the more edge cases we unearth and account for. The biggest improvement is that we have built a internal DCIM that templates a rack design and exports a interactive "cabling explorer" for the site techs - including detailed annotated diagrams of equipment showing port names, etc... The screenshot of the elevation is a screenshot of part of that tool.

   > What does your metal->boot stack look like?

   We've hacked together something on top of https://github.com/danderson/netboot/tree/main/pixiecore that serves a debian netboot + preseed file. We have some custom temporal workers to connect to Redfish APIs on the BMCs to puppeteer the contraption. The a custom hot agent to provision QEMU VMs and advertise assigned IPs via BGP (using FRR) from the host.

   Re: new DCs for failure scenarios, yeah we've already blown breakers etc... testing stuff (that's how we figured out our phase balancing was off). Went in with a thermal camera on another. A site in AMS is coming up next week and the goal for that is to see how far we can push a fully loaded switch fabric.

4. pcm

   3 5 3,040 9.6 C++

   Intel® Performance Counter Monitor (Intel® PCM)

   

   We've already seen some perf improvements somewhere between 6.0 and 6.5 (I forget the exact reason/version) - but it was some fix specific to the Sapphire Rapids cpus we had. But I wish we had more time to science on it, it's really fun playing with all the knobs and benchmarking stuff. Some of the telemetry on the new CPUs is also crazy - there's stuff like Intel PCM that can pull super fine-grained telemetry direct from the CPU/chipset https://github.com/intel/pcm. Only used it to confirm that we got NUMA affinity right so far - nothing crazy.

5. ptm

   4 1 83 - C

   Prescriptive Topology Daemon (by CumulusNetworks)

   

   https://github.com/CumulusNetworks/ptm

   You give it a graphviz dot file, and it uses LLDP to ensure that reality matches that file.

Suggest a related project