Introduction to the use of routing protocols
Flexiant Cloud Orchestrator supports three routing protocols:
- STATIC (using Automatic Router Configuration, or ARC);
- OSPF; and
Both router nodes and compute nodes (on KVM, Virtuozzo (formerly PCS), and Xen4 hypervisors) have an inbuilt router and packet filtering firewall (together referred to as a 'router' below). The router in the router nodes is used in Public VLAN mode, whereas the router in the compute nodes is used in Public Virtual IP (PVIP) mode. Private VLANs and Interworking VLANs do no use Flexiant Cloud Orchestrator managed routers. The router consists of two parts, evr which controls the router's configuration, and bird which runs the routing protocol between the node and the upstream router(s). The routing protocol is necessary for two reasons:
- To ensure that the upstream router(s) have a route to the appropriate Virtual Machines (in PVIP mode) or VLAN (in Public VLAN mode)
- To ensure that the router has a default route to a functional upstream router, as where more than one upstream router is supplied, this default route may change if the upstream router(s) malfunction or lose connectivity.
IP numbering of routers
By default Flexiant Cloud Orchestrator assumes that nodes are connected to routers using a single subnet for each of IPv4 and IPv6. These are:
- For IPv4,
NETWORK_PVIP_RANGE, defaulting to
- For IPv6,
Each node will have an offset into that range equal to the offset of the node with the node network range. So the node with node address
10.157.128.129 will have external IP addresses
fc00:41c8:10:1dd::81 (remember that
81 is hex for 129).
If static routing is used, the router should have the IP address with offset 1 into that range, i.e.
fc00:41c8:10:1dd::1. If redundancy is required, use a protocol such as VRRP or HSRP, and make these
.1 addresses the floating address, using
.3 for the static IP addresses if necessary.
If OSPF or BGP is in use, number the first router with offset 1 (i.e.
fc00:41c8:10:1dd::1) , and the second (if present) with offset 2 (i.e.
Choice of routing protocol
The default routing protocol used is STATIC, which is the easiest to set up but the most limited in terms of scalability. OSPF is the next easiest, and has wide support, but still has some scalability limitations. BGP takes a little longer to set up, but is the most scalable.
The default routing protocol used is STATIC configuration using ARC. Here the network to which the nodes attach must be flat, i.e. there must be a single upstream router or pair of upstream routers shared between all the nodes. BGP and OSPF allow a more typical hierarchical model.
OSPF is easy to set up, and has wide support. However, there are some scalability limitations.
- As the PVIP network within a cluster is normally a single broadcast domain, each of the nodes must be in the same OSPF area. This means that with a single broadcast domain, you may experience scaling problems going beyond approximately 50 nodes.
- Whilst it is possible to use a hierarchical OSPF configuration, beyond 50 nodes it is in general better to use a BGP configuration instead, as increased network size will still result in increased per-node CPU consumption.
- Whilst BGP is marginally more difficult to set up, there is no workload-based disadvantage of using it. Therefore, we recommend that for deployments that might scale beyond 50 nodes, BGP is used.
You can control the routing protocol used by changing
NODE_ROUTINGPROTOCOL to either
/etc/extility/local.cfg. We do not recommend this setting is changed whilst in production. All nodes will need to have networking restarted for the change to take place; currently this is most easily achieved through a reboot.
Both BGP and OSPF support authentication. By default, authentication is turned off for ease of installation. We strongly suggest authentication is turned on. This is achieved by setting
1 in local.cfg.
By default, Flexiant Cloud Orchestrator will automatically generate a password (or shared secret) for the routing protocol. This can be found using the command
fgrep NODE_ROUTINGPROTOCOL_PASSWORD /etc/extility/config/vars
Alternatively, you can specify your own password by setting
If you change from a STATIC routing protocol to an OSPF or BGP routing protocol, we recommend you turn ARC off. It produces unnecessary traffic in such an environment, and complicates debugging. If you use the STATIC routing protocol, ARC must be turned on (as it is by default).
Flexiant Cloud Orchestrator by default supports static routing through the use of Automatic Router Configuration (or ARC). This is suitable for small environments. The scalability limitation of the ARC is that the ARP cache within the upstream router carries one entry per VM NIC.
The major advantage of static routing is that the configuration on your upstream router is very simple indeed.
For each IP address pool you have, you must create an 'interface route' or 'device route' on your upstream router pointing at the interface to which the nodes are attached. This will be the interface carrying the PVIP network and will have the IP address set out in the previous section (normally
In the example below, let's assume the IP pools to be added are
22.214.171.124/24 (you will need to perform similar configuration for your IPv6 pools).
On a Cisco router, if this interface is
ge0/0/0, you might achieve this by:
On a Linux based router, assuming the interface is
eth1.23, you might achieve this by:
Do not add routes to a next hop. The routes added must be interface routes
From the point of view of your upstream router(s) the VM's will be directly connected at the IP level.
If you add further IP pools, it will be necessary to configure these on each of your upstream routers.
A typical static routing configuration with ARC
OSPF specific instructions
If using OSPF, we strongly recommend you supply two upstream routers that both speak OSPF. It is possible to use only a single router, but this will provide no redundancy in case of failure. Further, OSPF requires a "Designated Router" (DR) and a "Backup Designated Router" (BDR) in each OSPF area; these use more CPU resources than other routers. Supplying two routers ensures that no node is used as a DR or BDR.
We strongly recommend that the OSPF process used is reserved solely for Flexiant Cloud Orchestrator, which means it is not the same OSPF used as your IGP. If you do share the same OSPF process, starting and stopping servers may cause route propagation throughout your IGP, which will be undesirable. If you use OSPF as your internal routing protocol, you will either need to run two OSPF processes to comply with this recommendation, or you will need to avoid these two routers becoming part of your IGP.
The upstream routers should be numbered with the
.2 address in the PVIP network.
The PVIP interfaces of the upstream routers should put in area
0.0.0.0 (sometimes called area 0), and the OSPF priority of these interfaces set to 100.
You do not need to configure the upstream routers when additional nodes are added. OSPF will take care of this.
Each router should:
- Originate a default route which should be advertised
- Accept all routes from its OSPF peers
- Advertise an aggregate route upstream.
For example, if your IP allocations were in the block
126.96.36.199/20, it would be appropriate to advertise that block as an aggregate into your IGP.
Remember to configure OSPF authentication appropriately if you have enabled authentication (see above). Flexiant Cloud Orchestrator uses digest authentication.
OSPFv3 is used for IPv6 routers. Ensure that your two upstream routers have neighbour adjacency between them, and between each of them and each node.
Starting servers, for PVIP networks, and configuring VLANs, for Public VLAN mode, should result in OSPF routes appearing.
If you add further IP pools, it will not normally be necessary to change the configuration of your upstream routers unless you have configured them to filter OSPF LSAs, in which case you will need to adjust those filters.
A typical OSPF configuration
BGP specific instructions
If using BGP, we strongly recommend you supply two upstream routers that speak BGP. It is possible to use only a single router, but this will provide no redundancy in case of failure.
We recommend that you no do not use the same BGP process, and thus do not use the same AS number, as your external AS number. BGP is normally used as an exterior gateway protocol; here we are using it as an interior gateway protocol. We use AS65000 by default, and you should configure your upstream routers to use that. If your upstream routers need to speak eBGP too, that should be run in a separate routing process.
If your upstream routers do not support multiple routing processes, and you require them to speak BGP to other routers within your AS, you can configure Extility to use your normal AS number by altering the BGP files within
Flexiant Cloud Orchestrator advertises routes with the 'no-export' and 'no-advertise' well-known communities set. However, you should not rely on this, and should employ appropriate route filtering to ensure prefixes do not leak.
The upstream routers should be numbered with the .1 and .2 address in the PVIP network.
A peering should be established between each upstream router and each node. Contrary to the normal situation with iBGP, it is not necessary to establish a full mesh.
There is no need for the following:
- iBGP peerings between nodes - these should not be added
- An iBGP peering between the two upstream nodes
- The nodes to peer with any other iBGP speakers in your network
When you add additional nodes, you will need to configure the upstream routers appropriately. Using peer groups will simplify the configuration. Some router vendors support "dynamic peering", which can be used to avoid this step. If your router does not support dynamic peering, it is possible to 'preconfigure' peerings with nodes that do not (yet) exist. These may be left administratively shut down to prevent network management alerts.
Each router should:
- Originate a default route which should be advertised to each node.
- Accept all routes from its BGP peers. However, these routes should not be propagated into the rest of your network.
- Advertise an aggregate route upstream. For instance, if your IP allocations were in the block 188.8.131.52/20, it would be appropriate to advertise that block as an aggregate into your IGP.
Remember to configure BGP authentication appropriately if you have enabled authentication (see above).
The same technique is used for IPv6 routers. IPv4 are advertised over IPv4 sessions. IPv6 routes are advertised over IPv6 sessions.
Ensure that each of your two upstream routers has a peering with each node.
Starting servers (in the case of PVIP networks) and configuring VLANs (in the case of Public VLAN mode) should result in BGP routes appearing.
If you add further IP pools, you will not normally need to adjust the configuration on your upstream routers, unless you have configured BGP prefix filtering on them.
A typical BGP configuration
Automatic Router Configuration (ARC)
When using the static routing protocol, Flexiant Cloud Orchestrator ensures the upstream router knows which node to send packets to by responding to ARP requests for the IP address concerned, and also (under some circumstances) by use of promiscuous or gratuitous ARP. This is fully automatic and requires no configuration. However, this ARC function may be undesirable when a dynamic routing protocol is used. The configuration file controlling the operation of ARC is
/etc/extility/nodeconfig/evrtemplate.xml. The sample configuration file
/etc/extility/nodeconfig/evrtemplate.xml.sample has ARC switched on; this is copied to the running configuration file on install. In order to switch ARC off, copy over
You will then need to restart networking on the nodes, e.g. by rebooting them.
If you are using the STATIC protocol, you must have ARC switched on, or your traffic will not route.
If you are using BGP or OSPF, we advise you to switch ARC off to avoid unnecessary ARP traffic and facilitate debugging.
Whether the router is running on a compute node (PVIP mode) or a router node (Public VLAN mode), its configuration is managed by the following files:
/etc/extility/nodeconfig/evrtemplate.xml(Flexiant Cloud Orchestrator's Routing Config):
/etc/extility/nodeconfig/bird.conf-BGP(Bird's ipv4 BGP Config)
/etc/extility/nodeconfig/bird6.conf-BGP(Bird's ipv6 BGP Config)
/etc/extility/nodeconfig/bird.conf-OSPF(Bird's ipv4 OSPF Config)
/etc/extility/nodeconfig/bird6.conf-OSPF(Bird's ipv6 OSPF Config)
These can be overridden on a per node or per configuration group basis, in a similar way to the
nodetemplate.xmlfile. When the node boots, it will try the files in the following order (using bird.conf-BGP as the example, but the others will work the same).
Thus, if the MAC address of the node is
aa:bb:cc:dd:ee:ff, and the IP address is
10.157.128.10, and the node is in config group 3, then the following files will be tried in order:
Letters in MAC addresses must be in lower case.
An explanation of config groups is given here. If the node configuration for any given node is changed, the node will need to be rebooted for the change to take effect.
evr.conf file should not in general need altering.
bird.conf files determine the IPv4 configuration for the router, and the
bird6.conf files the IPv6 configuration. The BGP files are used when BGP is in use as internal routing protocol, and the OSPF files are used when OSPF is used. For more documentation about the format of the bird configuration files, see the Bird documentation.