San Francisco CCP : Network Operations Group

The Network Operations group is a small, focused sub-group of SFCCP. This group is responsible only for the switch that sits between our private network and our uplink. This responsibility is separated from the [Sysadmins] group so that we can feel comfortable allowing many people to participate in the [Sysadmins] group. The jurisdiction of the Network Operations group involves systems which control bandwidth usage and, therefore, client billing.

The Network Operations group is coordinated on the operations mailing list.

The following are the original ideas that we had when forming the operations group in December 2006: It has been informally discussed about adding to the CCCP organizational model by creating a Networking/Infrastructure group. This group would be responsible for the core, fundamental infrastructure of all SFCCP racks/cabinets.

This includes IP address assignment, bandwidth monitoring, traffic shaping. This would be a "trusted group" - meaning that, in some ways, membership would be restricted and membership would begin with a vouching process. This has not been formally decided.

Bill Fumerola (worked at Yahoo for many years and also volunteers a lot with FreeBSD development) has years of networking experience and has volunteered to take a lead on the difficult task of putting together a solid network in a short timeframe. Most of these tasks originated with him and he has committed to work on them.

• "Evaluate and implement as decided" -> purchase a cheap DSL line and run it into the cabinet. This would be a private link to be used in the event of a DOS attack or another situation where we cannot get in through the front-door link. For bandwidth-sucking attacks, the DSL link could be used to get into the network and document the attack (see task #2). Additionally, depending on the capabilities of the DSL it could be used for additional bandwidth.

• A DOS escape clause should be built into our contract with the colo. The vulnerability here is that a DOS attack that massively increases our bandwidth could hit us when that month's bill arrives. In order to use this escape clause, though, we must be able to document the DOS (see task #1). One of the points of consideration for bandwidth provider is their ability to identify, mitigate, and respond to DOS.

• "Evaluate and implement as decided" -> obtain a managed switch to build our network where the colo's network ends. Bill will look around for the best price on a managed switch. model of switch desired: HP procurve, Cisco Catalyst, any Foundry or Extreme. advantages of managed v. unmanaged include but not limited to: bandwidth statistics, vlan segmentation, read-only/limited access to gear to customers for self-service, trunking multiple switches, rate limiting, etc.

• "Evaluate and implement as decided" -> determine router/firewall setup. this may be a simple solution at first (just routing, no/minimal firewall) and expanded as equipment becomes available. the aforementioned switches can perform none, some, or all of these routing/firewall functions.

• "Evaluate and implement as decided" -> obtain some kind of solution for remote console access. Bill will look around for the best price. possible vendors to look for donations/used are: cyclades, mrv, lucent portmaster, digi, avocent.

• "Evaluate and implement as decided" -> we're gonna need stuff like ethernet cable, db9/console cable, crimpers, etc. Bill's gonna look around for this stuff too. we can also require members provide for each of their machines.

• "Evaluate and implement as decided" -> Bill has ideas about having a managed power strip (allows detailed monitoring, ability to turn off any given outlet, etc) and using this to build a pricing model. possible vendors: APC, BayTech???? RPC, etc. Also, figure out how Seattle CCP really figures out who owes what each month based on power usage.

• "Evaluate and implement as decided" -> develop a policy for dealing with IP address space. This includes IP address assignment, developing a policy, creating a workable scheme for subnets/VLANs for client segmentation. members SHALL justify/explain address usage for any requests beyond one address per machine.

• "Evaluate and implement as decided" -> Write firewall policy for network-at-large, infrastructure machines, and member vlans. members MAY provide firewall requirements (allowed connections from the internet at large, between members, to SFCCP infrastructure). by default members SHOULD use SFCCP infrastructure machines (SMTP smarthost, recursive DNS servers, NTP, etc) whenever possible and this policy MAY be enforced through firewall rules. exceptions CAN be made as needed.

• Help evaluate and negotiate bandwidth contracts. Determine providers in the same location, solicit donations of bandwidth from them once bandwidth requirements are further understood.