Open vSwitch GRE Tunnel on Proxmox

使用VMware Workstation 建立實驗環境，建立 2台Proxmox 2.3 Host 並安裝Open vSwitch 和 1台POX controller 連接Open vSwitch，2台Proxmox 組成pve cluster並各建立1台VM，使用Open vSwitch GRE Tunnel 讓VM之間形成區網，架構如下圖所示。

[root@pveNode ~]$ ovs-vsctl add-br vmbr1
[root@pveNode ~]$ ovs-vsctl add-port vmbr1 gre0 -- set interface gre0 type=gre options:remote_ip=192.168.88.141
[root@pveNode ~]$ ovs-vsctl set-controller vmbr1 tcp:192.168.88.140:6633

[root@pveNode2 ~]$ ovs-vsctl add-br vmbr1
[root@pveNode2 ~]$ ovs-vsctl add-port vmbr1 gre0 -- set interface gre0 type=gre options:remote_ip=192.168.88.142
[root@pveNode2 ~]$ ovs-vsctl set-controller vmbr1 tcp:192.168.88.140:6633

Open vSwitch 設定結果如下，tap 是VM使用自動建立的:

[root@proxNode2 ~]# ovs-vsctl show
81be08ee-471b-45a0-acb7-f2e30782eeca
    Bridge "vmbr0"
        Port "eth0"
            Interface "eth0"
        Port "vmbr0"
            Interface "vmbr0"
                type: internal
    Bridge "vmbr1"
        Controller "tcp:192.168.88.140:6633"
            is_connected: true
        Port "tap100i0"
            Interface "tap100i0"
        Port "gre0"
            Interface "gre0"
                type: gre
                options: {remote_ip="192.168.88.141"}
        Port "vmbr1"
            Interface "vmbr1"
                type: internal

接下來安裝POX controller，需要python 2.7以上:

[root@ubuntu ~]$ wget http://python.org/ftp/python/2.7.3/Python-2.7.3.tgz
[root@ubuntu ~]$ tar -xvf Python-2.7.3.tgz 
[root@ubuntu ~]$ cd Python-2.7.3/
[root@ubuntu ~]$ ./configure 
[root@ubuntu ~]$ make
[root@ubuntu ~]$ sudo make altinstall
[root@ubuntu ~]$ sudo apt-get install python-pastescript
[root@ubuntu ~]$ git clone https://github.com/MurphyMc/poxdesk
[root@ubuntu ~]$ cd poxdesk/
[root@ubuntu ~]$ wget http://downloads.sourceforge.net/qooxdoo/qooxdoo-2.0.2-sdk.zip
[root@ubuntu ~]$ unzip qooxdoo-2.0.2-sdk.zip
[root@ubuntu ~]$ mv qooxdoo-2.0.2-sdk qx
[root@ubuntu ~]$ cd poxdesk
[root@ubuntu ~]$ ./generate.py
[root@ubuntu ~]$ cd ../../..
[root@ubuntu ~]$ ./pox.py forwarding.l2_learning samples.pretty_log web messenger messenger.log_service /
                    messenger.ajax_transport openflow.of_service poxdesk

網頁連接http://192.168.88.140:8000/ 選poxdesk 結果如下圖。

VM網路測試結果如下圖:

參考Mixed Hypervisor with Openvswitch and Openflow Network virtualisation

參考open-vswitch-gre-tunnel-configuration

configure OpenvSwitch on Proxmox VE

使用 Proxmox VE 2.3 來安裝 OpenvSwitch

• Build kermel module "bridge"

[root@pveNode ~]$ apt-get install build-essential autoconf libtool pkg-config openssl libssl-dev bridge-utils
[root@pveNode ~]$ apt-get install git-core rpm2cpio lintian ncurses-dev
[root@pveNode ~]$ git clone git://git.proxmox.com/git/pve-kernel-2.6.32.git pve-kernel
[root@pveNode ~]$ mv pve-kernel /usr/src
[root@pveNode ~]$ cd /usr/src/pve-kernel
##修改Makefile  gcc --version|grep "<b>4.4.5</b>" || false
[root@pveNode /usr/src/pve-kernel]$ make
[root@pveNode /usr/src/pve-kernel/linux-2.6-2.6.32]$ cd linux-2.6-2.6.32
[root@pveNode /usr/src/pve-kernel/linux-2.6-2.6.32]$ make mrproper
[root@pveNode /usr/src/pve-kernel/linux-2.6-2.6.32]$ make clean
[root@pveNode /usr/src/pve-kernel/linux-2.6-2.6.32]$ make menuconfig

Networking support ->  bridge  改為 M 退出儲存 

[root@pveNode /usr/src/pve-kernel/linux-2.6-2.6.32]$ make bzImage
[root@pveNode /usr/src/pve-kernel/linux-2.6-2.6.32]$ make modules
[root@pveNode /usr/src/pve-kernel/linux-2.6-2.6.32]$ make modules_install
[root@pveNode ~]$ cp /usr/src/pve-kernel/linux-2.6-2.6.32/arch/x86_64/boot/bzImage  /boot/vmlinuz-2.6.32-20-pve
[root@pveNode ~]$ cp /usr/src/pve-kernel/linux-2.6-2.6.32/.config /boot/config-2.6.32-20-pve
[root@pveNode ~]$ mkinitramfs -o /boot/initrd.img-2.6.32-20-pve 2.6.32-20-pve
[root@pveNode ~]$ cp /usr/src/pve-kernel/linux-2.6-2.6.32/arch/x86_64/boot/bzImage  /boot/vmlinuz-2.6.32-20-pve
[root@pveNode ~]$ cp /usr/src/pve-kernel/linux-2.6-2.6.32/.config /boot/config-2.6.32-20-pve
[root@pveNode ~]$ mkinitramfs -o /boot/initrd.img-2.6.32-20-pve 2.6.32-20-pve

在 /boot/grub/grub.cfg 改開機選單 範例如下

menuentry 'Proxmox Virtual Environment GNU/Linux, with Linux 2.6.32-20-pve' --class proxmox --class gnu-linux --class gnu --class os {
        insmod part_msdos
        insmod ext2
        set root='(hd0,msdos1)'
        search --no-floppy --fs-uuid --set 9cc709ff-7275-461e-8e91-505d0b1c158f
        echo 'Loading Linux 2.6.32-20-pve ...'
        linux /vmlinuz-2.6.32-20-pve root=/dev/mapper/pve-root ro quiet
        echo 'Loading initial ramdisk ...'
        initrd /initrd.img-2.6.32-20-pve
}

• Open vSwitch install

[root@pveNode ~]$ wget http://openvswitch.org/releases/openvswitch-1.4.0.tar.gz
[root@pveNode ~]$ tar zxvf openvswitch-1.4.0.tar.gz
[root@pveNode ~]$ cd openvswitch-1.4.0
[root@pveNode openvswitch-1.4.0]$ ./boot.sh
[root@pveNode openvswitch-1.4.0]$ ./configure --with-linux=/lib/modules/`uname -r`/build CC=gcc-4.4

修改 datapath/linux/compat/include/linux/skbuff.h 的函式 skb_frag_page 跟 skb_reset_mac_len
修改 datapath/linux/compat/include/linux/if.h 的變數 IFF_OVS_DATAPATH
修改 datapath/linux/compat/include/linux/if_vlan.h 的函式 vlan_set_encap_proto
將上述更名或註解

[root@pveNode openvswitch-1.4.0]$ make 
[root@pveNode openvswitch-1.4.0]$ make install
[root@pveNode openvswitch-1.4.0]$ rmmod bridge
[root@pveNode openvswitch-1.4.0]$ insmod ./datapath/linux/openvswitch_mod.ko
[root@pveNode openvswitch-1.4.0]$ insmod ./datapath/linux/brcompat_mod.ko

• Open vSwitch Setup

[root@pveNode ~]$ mkdir -p /usr/local/etc/openvswitch
[root@pveNode ~]$ ovsdb-tool create /usr/local/etc/openvswitch/conf.db ./vswitchd/vswitch.ovsschema
[root@pveNode ~]$ ovsdb-server ovsdb-server --remote=punix:/usr/local/var/run/openvswitch/db.sock \
        --remote=db:Open_vSwitch,manager_options \
        --private-key=db:SSL,private_key \
        --certificate=db:SSL,certificate \
        --bootstrap-ca-cert=db:SSL,ca_cert \
        --detach
[root@pveNode ~]$ ovs-vswitchd unix:/usr/local/var/run/openvswitch/db.sock --detach
[root@pveNode ~]$ ovs-brcompatd --pidfile --detach 
#set Network
[root@pveNode ~]$ ovs-vsctl add-br vmbr0
[root@pveNode ~]$ ovs-vsctl add-port vmbr0 eth0
[root@pveNode ~]$ ovs-vsctl show
[root@pveNode ~]$ ifconfig eth0 0
[root@pveNode ~]$ ifconfig vmbr0 192.168.88.100 netmask 255.255.255.0
[root@pveNode ~]$ route add default gw 192.168.88.2

參考 configure Openvswitch on Proxmox VE