LxCenter HyperVM & Kloxo Support

Forum



Members   Search      Help    Register    Login    Home
Home » LxCenter » Contributed HOWTOs » Xen Network Bridges and VLANs
Xen Network Bridges and VLANs [message #41613] Wed, 15 October 2008 06:10 Go to next message
pace is currently offline pace  United States
Messages: 248
Registered: May 2008
Senior Member
If you have a network with multiple VLANS and want to be able to have a single HyperVM host provide virtual machines on these VLANs, here are some instructions for making it happen.

Thanks to lxHelp and the guys at renial.net for some of the fixes and scripts required here.

First you need to create some scripts (and set them to executable):

/etc/xen/scripts/network-bridge-vlan
#!/bin/sh
#============================================================================
# Xen vlan bridge start/stop script.
# Xend calls a network script when it starts.
# The script name to use is defined in /etc/xen/xend-config.sxp
# in the network-script field.
#
# This script creates a bridge (default vlanbr${vlan}), creates a device
# (default eth0.${vlan}), and adds it to the bridge. This scrip assumes
# the Dom0 does not have an active interface on the selected vlan; if
# it does the network-bridge script should be used instead.
#
# To use this script, vconfig must be installed.
#
# Usage:
#
# network-bridge-vlan (start|stop|status) {VAR=VAL}*
#
# Vars:
#
# vlan       The vlan to bridge (default 2)
# bridge     The bridge to use (default vlanbr${vlan}).
# netdev     The interface to add to the bridge (default eth0}).
#
# Internal Vars:
# vlandev="${netdev}.${vlan}"
#
# start:
# Creates the bridge
# Adds vlandev to netdev
# Enslaves vlandev to bridge
#
# stop:
# Removes vlandev from the bridge
# Removes vlandev from netdev
# Deletes bridge
#
# status:
# Print vlan, bridge
#
#============================================================================


dir=$(dirname "$0")
. "$dir/xen-script-common.sh"

findCommand "$@"
evalVariables "$@"

vlan=${vlan:-2}
bridge=${bridge:-vlanbr${vlan}}
netdev=${netdev:-eth0}

vlandev="${netdev}.${vlan}"

##
# link_exists interface
#
# Returns 0 if the interface named exists (whether up or down), 1 otherwise.
#
link_exists()
{
    if ip link show "$1" >/dev/null 2>/dev/null
    then
        return 0
    else
        return 1
    fi
}


# Usage: create_bridge bridge
create_bridge () {
    local bridge=$1

    # Don't create the bridge if it already exists.
    if ! brctl show | grep -q ${bridge} ; then
        brctl addbr ${bridge}
        brctl stp ${bridge} off
        brctl setfd ${bridge} 0
    fi
    ip link set ${bridge} up
}

# Usage: add_to_bridge bridge dev
add_to_bridge () {
    local bridge=$1
    local dev=$2
    # Don't add $dev to $bridge if it's already on a bridge.
    if ! brctl show | grep -q ${dev} ; then
        brctl addif ${bridge} ${dev}
    fi
}

# Usage: show_status vlandev bridge
# Print vlan and bridge
show_status () {
    local vlandev=$1
    local bridge=$2

    echo '============================================================'
    cat /proc/net/vlan/${vlandev}
    echo ' '
    brctl show ${bridge}
    echo '============================================================'
}

op_start () {
    if [ "${bridge}" = "null" ] ; then
        return
    fi

    if ! link_exists "$netdev"; then
        return
    fi

    if link_exists "$vlandev"; then
        # The device is already up.
        return
    fi

    create_bridge ${bridge}

    ip link set ${netdev} up

    vconfig set_name_type DEV_PLUS_VID_NO_PAD
    vconfig add ${netdev} ${vlan}
    ip link set ${vlandev} address fe:ff:ff:ff:ff:ff
    ip link set ${vlandev} up
    ip link set ${bridge} up

    add_to_bridge2 ${bridge} ${vlandev}
}

op_stop () {
    if [ "${bridge}" = "null" ]; then
        return
    fi
    if ! link_exists "$bridge"; then
        return
    fi

    if link_exists "$vlandev"; then
        ip link set ${vlandev} down

        brctl delif ${bridge} ${vlandev}
        ip link set ${bridge} down

        vconfig rem ${vlandev}
    fi
    brctl delbr ${bridge}
}

# adds $dev to $bridge but waits for $dev to be in running state first
add_to_bridge2() {
    local bridge=$1
    local dev=$2
    local maxtries=10

    echo -n "Waiting for ${dev} to negotiate link."
    for i in `seq ${maxtries}` ; do
        if ifconfig ${dev} | grep -q RUNNING ; then
            break
        else
            echo -n '.'
            sleep 1
        fi
    done

    if [ ${i} -eq ${maxtries} ] ; then echo '(link isnt in running state)' ; fi

    add_to_bridge ${bridge} ${dev}
}

case "$command" in
    start)
        op_start
        ;;

    stop)
        op_stop
        ;;

    status)
        show_status ${vlandev} ${bridge}
        ;;

    *)
        echo "Unknown command: $command" >&2
        echo 'Valid commands are: start, stop, status' >&2
        exit 1
esac


/etc/xen/scripts/network-multi-vlan
#!/bin/sh
#============================================================================
# Xen vlan bridge start/stop script.
# Xend calls a network script when it starts.
# The script name to use is defined in /etc/xen/xend-config.sxp
# in the network-script field.
#
# This script creates multiple bridges to segregate individual domUs to
# separate VLANs. Customize to fit your needs.
#
# Usage:
#
# network-multi-vlan (start|stop|status)
#
#============================================================================

dir=$(dirname "$0")

##
# To make the tagged interface available to some DomUs, create the default
# bridge. Comment this out to only make vlan-based bridges available.
"$dir/network-bridge" "$@"

##
# Once all normal bridges are active, create any vlan-based briges.

# VLAN ID2 10.1.2.0/24
"$dir/network-bridge-vlan" "$@" vlan=2 bridge=xenbr2 netdev=eth0

# VLAN ID3 10.1.3.0/24
"$dir/network-bridge-vlan" "$@" vlan=3 bridge=xenbr3 netdev=eth0


Then edit /etc/xen/xend-config.sxp

replace:
(network-script network-bridge)

with:
(network-script network-multi-vlan)


Reboot.

Create the appropriate IP address pools for VLANs 2 and 3 so you can assign IP addresses from them. Then go to a virtual, assign a VLAN IP, and then go to Network and assing the appropriate bridge. For example:

I want a virtual with IP address 10.1.3.5 so I assign it to xenbr3.

lxHelp has given us xenbr0-xenbr4 so you can have 5 VLANs assigned. xenbr0 is usually your default bridge and will be untagged unless you go about changing the way that works (you'll have to modify network-multi-vlan to change this behavior). NOTE: in Xen 3.2 and later, xenbr0 is replaced with eth0 as the default bridge.

I can probably help out if you try this and have any problems...


Cheers,
pace

[Updated on: Wed, 15 October 2008 06:33]

Report message to a moderator

Re: Xen Network Bridges and VLANs [message #41801 is a reply to message #41613] Thu, 16 October 2008 19:32 Go to previous messageGo to next message
pace is currently offline pace  United States
Messages: 248
Registered: May 2008
Senior Member
It appears that Xen 3.3 doesn't have this issue, but in older Xen's you may need to do the following to prevent TCP and UDP connection issues:

ethtool -K eth0 tx off


Where eth0 is the interface you are running your VLANs on.

Also, if your network card driver doesn't support VLANing you'll need to adjust your MTU:

ifconfig eth0 mtu 1496


I did not need either of these. I'm running Xen 3.3 with Intel Gigabit adapters built into my motherboards...


pace
Re: Xen Network Bridges and VLANs [message #70086 is a reply to message #41613] Tue, 18 August 2009 05:45 Go to previous messageGo to next message
ensermo is currently offline ensermo  Netherlands
Messages: 100
Registered: October 2006
Location: Delft
Valuable Member
Hi,

Can someone maybe help me with the following (Xen Server)?

I have one(1) HyperVM server and connected to a switch.
The switch has untagged IP addresses (84.x.y.z.) and tagged IP addresses on VLAN 1234 (213.x.y.z)

Our datacenter has setup this for us.

Copied the scripts to /etc/xen/scripts

I changed /etc/xen/scripts/network-multi-vlan
to have this two lines.

#
"$dir/network-bridge" "$@" vifnum=0

# VLAN ID1234
"$dir/network-bridge-vlan" "$@" vlan=1234 bridge=xenbr4 netdev=eth0



I went into HyperVM, went to my virtual machine, gave my Xen VM an IP from the 213.x.y.z range, went to 'Network' and changed the bridge to xenbr4.

I rebooted my Xen VM.

Still can't reach the tagged range.

Any ideas?

[Updated on: Tue, 18 August 2009 05:46]

Report message to a moderator

Re: Xen Network Bridges and VLANs [message #70087 is a reply to message #41613] Tue, 18 August 2009 05:50 Go to previous messageGo to next message
ensermo is currently offline ensermo  Netherlands
Messages: 100
Registered: October 2006
Location: Delft
Valuable Member
ifconfig on my Master Xen Server shows this.

#ifconfig
eth0 Link encap:Ethernet HWaddr 00:12:79:AD:AA:B4
inet addr:84.x.y.z Bcast:84.x.y.255 Mask:255.255.255.0
inet6 addr: fe80::232:76ff:fecd:5b73/64 Scope:Link
UP BROADCAST RUNNING PROMISC MULTICAST MTU:1496 Metric:1
RX packets:45214 errors:0 dropped:0 overruns:0 frame:0
TX packets:11094 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:3842480 (3.6 MiB) TX bytes:5159002 (4.9 MiB)

eth0.1220 Link encap:Ethernet HWaddr FE:FF:FF:FF:FF:FF
inet6 addr: fe80::ffef:ffff:ffef:ffff/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:260 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 b) TX bytes:20596 (20.1 KiB)

lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:1664 errors:0 dropped:0 overruns:0 frame:0
TX packets:1664 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:297399 (290.4 KiB) TX bytes:297399 (290.4 KiB)

peth0 Link encap:Ethernet HWaddr FE:FF:FF:FF:FF:FF
inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link
UP BROADCAST RUNNING NOARP MTU:1500 Metric:1
RX packets:192556 errors:0 dropped:0 overruns:0 frame:0
TX packets:210467 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:26807688 (25.5 MiB) TX bytes:202416319 (193.0 MiB)
Interrupt:21


viftest0 Link encap:Ethernet HWaddr FE:FF:FF:FF:FF:FF
inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link
UP BROADCAST RUNNING NOARP MTU:1500 Metric:1
RX packets:12 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:32
RX bytes:552 (552.0 b) TX bytes:0 (0.0 b)

xenbr0 Link encap:Ethernet HWaddr FE:FF:FF:FF:FF:FF
UP BROADCAST RUNNING NOARP MTU:1500 Metric:1
RX packets:32772 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:1610843 (1.5 MiB) TX bytes:0 (0.0 b)

xenbr4 Link encap:Ethernet HWaddr FE:FF:FF:FF:FF:FF
inet6 addr: fe80::200:ff:fe00:0/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:223 errors:0 dropped:0 overruns:0 frame:0
TX packets:20 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:7108 (6.9 KiB) TX bytes:4768 (4.6 KiB)

Re: Xen Network Bridges and VLANs [message #70117 is a reply to message #41613] Tue, 18 August 2009 15:47 Go to previous messageGo to next message
pace is currently offline pace  United States
Messages: 248
Registered: May 2008
Senior Member
First thing to check is to make sure your bridges are up and that the appropriate cards and vms are ending up in the right bridge.

What does this show you:

brctl show



pace
Re: Xen Network Bridges and VLANs [message #70119 is a reply to message #41613] Tue, 18 August 2009 15:53 Go to previous messageGo to next message
ensermo is currently offline ensermo  Netherlands
Messages: 100
Registered: October 2006
Location: Delft
Valuable Member
[root@SERVER001 ~]# brctl show
bridge name bridge id STP enabled interfaces
xenbr0 8000.feffffffffff no vifvm10
vifvm20
vifvm30
peth0
vif0.0
xenbr4 8000.001278cd5bf4 no viftest0
eth0.1234
Re: Xen Network Bridges and VLANs [message #70120 is a reply to message #41613] Tue, 18 August 2009 16:06 Go to previous messageGo to next message
pace is currently offline pace  United States
Messages: 248
Registered: May 2008
Senior Member
That looks okay. In your ifconfig you have an eth0.1220, in the rest of your stuff you state that you are setting up to be in 1234.


pace
Re: Xen Network Bridges and VLANs [message #70123 is a reply to message #41613] Tue, 18 August 2009 16:44 Go to previous messageGo to next message
ensermo is currently offline ensermo  Netherlands
Messages: 100
Registered: October 2006
Location: Delft
Valuable Member
sorry should be 1234...
that's the thing it all seems fine but cant' get it to work.
I even tried it mannually like this site says.
http://www.felipe-alfaro.org/blog/2006/07/21/xen-network-con figuration-and-multiple-vlans/
Re: Xen Network Bridges and VLANs [message #70124 is a reply to message #41613] Tue, 18 August 2009 16:51 Go to previous messageGo to next message
pace is currently offline pace  United States
Messages: 248
Registered: May 2008
Senior Member
Your switch port has been configured to have VLAN 1234 on it tagged?

If you are unsure about that, add another VM and put it in xenbr4 and see if the two VMs can talk to each other. If they can then perhaps the ISP hasn't completed the config on their side?


pace
Re: Xen Network Bridges and VLANs [message #70134 is a reply to message #41613] Wed, 19 August 2009 02:52 Go to previous messageGo to next message
ensermo is currently offline ensermo  Netherlands
Messages: 100
Registered: October 2006
Location: Delft
Valuable Member
On the same switch we have other servers(dedicated with one OS) that have already been migrated to the new VLAN.
So I assume the VLAN settings are correct.

Is there a way to test if the interface eth0.1234 is really sending tagged packages out?

So I can test if the master server itself is really using VLAN before trying with the VM's?

(Also should the eth0.1234 IP configuration be empty, see ifconfig)

[Updated on: Wed, 19 August 2009 02:53]

Report message to a moderator

Re: Xen Network Bridges and VLANs [message #70136 is a reply to message #41613] Wed, 19 August 2009 04:19 Go to previous messageGo to next message
ensermo is currently offline ensermo  Netherlands
Messages: 100
Registered: October 2006
Location: Delft
Valuable Member
What about the route on the master server?

Right now my #route shows only a default route to the gateway of the untagged network via eth0

Should I add another route? (even though VLAN should be layer 2 right?)
Re: Xen Network Bridges and VLANs [message #79771 is a reply to message #70136] Tue, 15 February 2011 05:49 Go to previous messageGo to next message
lenin122 is currently offline lenin122  India
Messages: 5
Registered: February 2011
Member
With regard to the problem mentioned by ensermo I have a hunch that they are related to VLAN settings. Most probably they are not correct. I would request him to check it and make the necessary corrections. I am pretty sure that it will solve the issue. I hope that he will let us know whether this solved the problem or not.

'ALL IS WELL"
Re: Xen Network Bridges and VLANs [message #80438 is a reply to message #79771] Thu, 03 March 2011 19:13 Go to previous message
milkyflava  United States
Messages: 1
Registered: March 2011
Member
What should the IP address be for the physical device eth0?

And should the bridge have the IP address as well?

So should say xenbr2 and xenbr3 have IP addresses or should the IP addresses be on eth0.2 and eth0.3?

Thanks for the write up!
Previous Topic:AccountLab Plus with Kloxo - rewrite of LxAdmin module
Next Topic:Alp/kloxo tweak
Goto Forum:
  


Current Time: Fri May 24 06:20:30 EDT 2013

Total time taken to generate the page: 0.01274 seconds
.:: Contact :: Home :: Privacy ::.

Click here to lend your support to: LxCenter and make a donation at www.pledgie.com !

Powered by: FUDforum 3.0.2.
Copyright ©2001-2010 FUDforum Bulletin Board Software