This section applies to router nodes on all hypervisors, and to compute nodes on KVM and Xen only. It does not apply to Virtuozzo (formerly PCS), VMware, or Hyper-V compute nodes.
Naming of network interfaces
Nodes will typically have multiple network interfaces, for instance for management, storage, and public traffic. Using the
nodetemplate.xml file detailed here, you can control what functions are associated with what interface names. For instance, you could require that the interface used for storage is
eth3. However, this leaves an obvious question open. How do you know what the Linux interface name is of a particular network port on the node?
By way of background, it is useful to explain what Linux does by default, and then explain how Flexiant Cloud Orchestrator handles the problem. When the machine boots up, the kernel assigns names to network interfaces in an unpredictable order. Normally the ordering of interfaces within a particular NIC will be well known, and also normally between NICs handled by the same driver. However, ordering between NICs handled by different drivers is unpredictable and depends in part on initialisation timings. To make sure interface naming is consistent across reboots, Ubuntu writes a 'persistent interface' file, recording the name it gave (perhaps arbitrarily) to each network interface and the MAC address of this interface, and on future boots renames the interfaces appropriately if they do not match. This strategy does not work for Flexiant Cloud Orchestrator because:
- there is no disk in the node to record the data on;
- we wish to ensure that multiple nodes in with the same hardware configuration always boot with consistent device numbering; and
- we wish to ensure that interface naming is unchanged if a faulty NIC is replaced with a new one with different MAC addresses.
What we do therefore is to name the network interfaces in order of their PCI bus ID, then their PCI slot ID, then the on-card interface number. This ensures that for a given hardware configuration, the network configuration is always consistent. However, just like under Linux, the naming may not match the naming in the BIOS.
If the hardware configuration is changed (for instance swapping slots of network interfaces, or adding additional network interfaces), network interface naming may change
Network network interfaces and pxeboot
Nodes boot using pxeboot - in other words, they boot using the network to provide the operating system using the PXE protocol. The boot process is as follows:
- The BIOS determines that it should boot using pxeboot
- For each network interface configured for pxeboot in the BIOS, the BIOS requests a dhcp address. The node network should provide one.
- If a dhcp address is successfully acquired, the BIOS makes a tftp request for the pxeboot boot loader
- The pxeboot process loads the operating system using tftp and http
- Interface naming is set using the scheme set out above.
- The operating system launches a dhcp process to acquire the IP address (which should be the same as in step 2).
- The operating system loads Flexiant Cloud Orchestrator's node agent software, which retrieves its configuration from the node configuration server (see Nodeconfig).
- Flexiant Cloud Orchestrator configures the remaining network interfaces according to the relevant stanzas of the retrieved configuration.
- Flexiant Cloud Orchestrator starts the node agent software
This means that the interface attached to the node network should be configured for pxeboot in the BIOS. No other interfaces should be configured to pxeboot.
The naming of the interface in the BIOS may not correspond to the naming under Linux. You should check the MAC addresses match.
When the node agent boots, it will determine at step 5 above which of the interfaces was used to pxeboot. As this must be on the node network, it will use this network to retrieve its configuration at step 7. You can determine which interface the node is trying to use using the command:
That command will also detail MAC address assignments of the interfaces and any interface renaming that took place (this is useful for translating kernel boot messages).
You must ensure that the boot interface used is specified as the node interface in the
nodetemplate.xml (see Nodeconfig).
If your boot interface is not listed as the management interface in
nodetemplate.xml, your node will not boot properly and you will not be able to access it via the network.
You must also ensure that the remaining interfaces in
nodetemplate.xml match what is actually connected to the node or your node will not function correctly.
Using multiple hardware configurations
As interface naming may be different on different hardware configurations, you may need a different
nodetemplate.xml for different hardware configurations. As hardware configurations are likely to be shared between multiple nodes, you may want to use Config Groups to achieve this.
Troubleshooting node network interface problems
If your node starts to boot but does not come up fully, or does not function, it is likely you have a networking problem of some sort. A detailed tutorial on troubleshooting node networking problems can be found here. The following is a brief summary.
First, ensure the node is booting using PXE. You should see the node software loading. This may take as long as five minutes - be patient! If this process fails, it is likely that either:
- The node is not connected to the node network; or
- The network interface to which the node network is connected is not enabled for PXE; or
- You have not added the node to the system (see Adding Router Nodes and Adding Compute Nodes).
Next gain access to the node. From the cluster management box type:
If the ping command shows packets echoing back, you have connectivity to the node. In this case the command:
should gain you access.
If you do not have access to the node, it is likely that your network cables are not correctly connected, or that the
nodetemplate.xml file for the node does not specify the correct interface for the node interface - see Nodeconfig. You can gain access to the node over the console using Gaining console access to nodes.
Once you are logged into the node, you can see the configuration of network interfaces and MAC addresses, and the status of the interfaces using the commands:
You can then address and resolve any connectivity problems in the normal way.
If, having read this section, you still have not resolved your problem, we suggest you read our tutorial on Troubleshooting node networking problems.