👩🏿‍🎨 📣 👏🏽 基于QEMU的系统测试自动化（第2/2部分） 🚶🏿 👩🏿‍🤝‍👨🏽 💪🏼

这是有关基于VM的系统测试自动化的文章的第二部分。第一部分可以在这里找到。

在本文的这一部分中，我们将使用在第1部分中学到的技能来实际自动化系统测试。在本文的最后，我们将收到一个脚本，任何人都可以在其计算机上运行，并且完全可以从头开始获得由三台机器组成的部署立场，由受测应用程序安装，并通过了真实的系统测试（我们将编写三个测试）。

免责声明

, , , . " " , . , .

因此，在最后一部分中，我们从命令行中积累了令人印象深刻的使用虚拟机的技能库：我们学习了如何安装虚拟机，在其上部署OS（例如Ubuntu Server 18.04），如何通过网络将虚拟机连接到主机，甚至通过SSH来组织控制通道。所有这些对于本文而言对我们都是有用的，但是在继续实践之前，有几个问题需要讨论。

我们想得到什么？

要回答的最重要的问题是“我们想要得到什么结果？” 是的，上次我们谈论了很多关于自动化安装，部署和配置虚拟机的问题，但是除了最终目标之外，这一切都没有多大意义。

就我个人而言，多合一系统测试如下所示：我从VCS下载了几个小文件（带有启动脚本本身，可能还有一些辅助工件），将程序放在我需要的地方进行测试（例如，以安装程序或软件包的形式），我按一个按钮去喝咖啡。当我回来时，我想看看所有测试都通过了，或者某些测试已经损坏了。我不想参与任何展位设置，也不想在其中部署虚拟机或进行任何设置。我想下载一个脚本并使用它。

此外，该脚本应该是可重用的：如果我放了程序的新程序集，我想再次运行该脚本，并且应该毫无问题地再次运行。理想情况下，脚本应该缓存中间结果，这样我就不必每次都等待创建和配置虚拟机（毕竟，这些操作只应该执行一次）。

获得另一个脚本，如果有必要的话，该脚本将允许清除所有创建的实体并释放磁盘空间，这也是很好的。

我们将测试什么

. -, Data Plane Development Kit (DPDK). DPDK — , , , DPDK. DPDK , , end-to-end .

DPDK

DPDK (Data Plane Development Kit) — , C. , . , . , . DPDK . ? . , , Linux, , , , . , Linux — , . , DPDK .

;
;
DROP — ;
ACCEPT — ;

, , :

;
, ;

, , , :

(client, middlebox, server), Ubuntu Server 18.04 ();
: client middlebox ( net_1) middlebox server ( net_2);
;
, ;
- middlebox;
.

. :

, SSH- , , , ;
SSH net_1 net_2, ( net_for_ssh). :

.. , , - ;
, - (, ), .

!

- :

, , :

run_tests.sh

#!/bin/bash
set -euo pipefail

# =======================================
#   net_for_ssh
# =======================================
virsh net-define net_for_ssh.xml
virsh net-start net_for_ssh

# =======================================
#   net_1
# =======================================
virsh net-define net_1.xml
virsh net-start net_1

# =======================================
#   net_2
# =======================================
virsh net-define net_2.xml
virsh net-start net_2

# =======================================
#   client
# =======================================
virt-builder ubuntu-18.04 \
    --format qcow2 \
    --output client.qcow2 \
    --install wget \
    --root-password password:1111 \
    --run-command "ssh-keygen -A" \
    --run-command "sed -i \"s/.*PermitRootLogin.*/PermitRootLogin yes/g\" /etc/ssh/sshd_config" \
    --copy-in netcfg_client.yaml:/etc/netplan/

virt-install \
    --import \
    --name client \
    --ram 1024 \
    --disk client.qcow2 \
    --network network=net_for_ssh \
    --network network=net_1,mac=52:54:56:11:00:00 \
    --noautoconsole

# =======================================
#   middlebox
# =======================================
virt-builder ubuntu-18.04 \
    --format qcow2 \
    --output middlebox.qcow2 \
    --install python,daemon,libnuma1 \
    --root-password password:1111 \
    --run-command "ssh-keygen -A" \
    --run-command "sed -i \"s/.*PermitRootLogin.*/PermitRootLogin yes/g\" /etc/ssh/sshd_config" \
    --copy-in netcfg_middlebox.yaml:/etc/netplan/

virt-install \
    --import \
    --name middlebox \
    --vcpus=2,sockets=1,cores=2,threads=1 \
    --cpu host \
    --ram 2048 \
    --disk middlebox.qcow2 \
    --network network=net_for_ssh \
    --network network=net_1,model=e1000 \
    --network network=net_2,model=e1000 \
    --noautoconsole

# =======================================
#   server
# =======================================
virt-builder ubuntu-18.04 \
    --format qcow2 \
    --output server.qcow2 \
    --install nginx \
    --root-password password:1111 \
    --run-command "ssh-keygen -A" \
    --run-command "sed -i \"s/.*PermitRootLogin.*/PermitRootLogin yes/g\" /etc/ssh/sshd_config" \
    --copy-in netcfg_server.yaml:/etc/netplan/

virt-install \
    --import \
    --name server \
    --ram 1024 \
    --disk server.qcow2 \
    --network network=net_for_ssh \
    --network network=net_2,mac=52:54:56:00:00:00 \
    --noautoconsole

# =======================================
# ,    
#     
# =======================================

SSH_CMD="sshpass -p 1111 ssh -o StrictHostKeyChecking=no"

while ! SSH_CMD root@192.168.100.2 "echo Hello world from client!" echo
do
    echo "Waiting for client VM ..."
    sleep 1
done

while ! SSH_CMD root@192.168.100.3 "echo Hello world from middlebox!" echo
do
    echo "Waiting for middlebox VM ..."
    sleep 1
done

while ! SSH_CMD root@192.168.100.4 "echo Hello world from server!" echo
do
    echo "Waiting for server VM ..."
    sleep 1
done

net_for_ssh.xml

<network>
    <name>net_for_ssh</name>
    <bridge name='net_for_ssh'/>
    <ip address='192.168.100.1' netmask='255.255.255.0'/>
</network>

net_1.xml

<network>
    <name>net_1</name>
    <bridge name='net_1'/>
    <ip address='192.168.101.1' netmask='255.255.255.0'/>
</network>

net_2.xml

<network>
    <name>net_2</name>
    <bridge name='net_2'/>
    <ip address='192.168.102.1' netmask='255.255.255.0'/>
</network>

netcfg_client.yaml

network:
  version: 2
  renderer: networkd
  ethernets:
    ens3:
      addresses:
        - 192.168.100.2/24
    ens4:
      addresses:
        - 192.168.101.2/24
      gateway4: 192.168.101.3

netcfg_middlebox.yaml

network:
  version: 2
  renderer: networkd
  ethernets:
    ens3:
      addresses:
        - 192.168.100.3/24

netcfg_server.yaml

network:
  version: 2
  renderer: networkd
  ethernets:
    ens3:
      addresses:
        - 192.168.100.4/24
    ens4:
      addresses:
        - 192.168.102.2/24
      gateway4: 192.168.102.3

, :

--install virt-builder, . --run-command "apt install ...". , , — virt-builder . client wget, server — nginx ( http- ). middlebox , DPDK-;
client server , - , . ;
middlebox ( --vcpus): CPU c hyperthreading. — , DPDK . --cpu host, , , . , - QEMU , SSE3 . , , DPDK .
middlebox , : e1000. , - DPDK.

run_tests.sh ( , ). , :

;
.

, run_tests.sh, , . :

run_clean.sh

#!/bin/bash
set -euo pipefail

# =======================================
#   client
# =======================================
if virsh list --all | grep -q " client "; then
    if virsh domstate client | grep -q "running"; then
        virsh destroy client
    fi
    virsh undefine client --snapshots-metadata --remove-all-storage
fi

# =======================================
#   middlebox
# =======================================
if virsh list --all | grep -q " middlebox "; then
    if virsh domstate middlebox | grep -q "running"; then
        virsh destroy middlebox
    fi
    virsh undefine middlebox --snapshots-metadata --remove-all-storage
fi

# =======================================
#   server
# =======================================
if virsh list --all | grep -q " server "; then
    if virsh domstate server | grep -q "running"; then
        virsh destroy server
    fi
    virsh undefine server --snapshots-metadata --remove-all-storage
fi

# =======================================
#   net_for_ssh
# =======================================
if virsh net-list --all | grep -q " net_for_ssh "; then
    if virsh net-list --all | grep " net_for_ssh " | grep -q " active "; then
        virsh net-destroy net_for_ssh
    fi
    virsh net-undefine net_for_ssh
fi

# =======================================
#   net_1
# =======================================
if virsh net-list --all | grep -q " net_1 "; then
    if virsh net-list --all | grep " net_1 " | grep -q " active "; then
        virsh net-destroy net_1
    fi
    virsh net-undefine net_1
fi

# =======================================
#   net_2
# =======================================
if virsh net-list --all | grep -q " net_2 "; then
    if virsh net-list --all | grep " net_2 " | grep -q " active "; then
        virsh net-destroy net_2
    fi
    virsh net-undefine net_2
fi

, run_tests.sh, . :

( virsh list --all, — virsh net-list -all);
/, , / , ;
( --snapshots-metadata) ( --remove-all-storage).

run_tests.sh run_clean.sh. run_tests.sh , run_clean.sh.

run_clean.sh . . , , , .

. SSH- — SSH ~/.ssh/known_hosts. , SSH — , , - . SSH_CMD:

SSH_CMD="sshpass -p 1111 ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=ERROR"

-o UserKnownHostsFile=/dev/null, .

, — . Ubuntu Server ( GUI) bash-, .

bash-, . , .

-, , :

EXEC_CLIENT="$SSH_CMD root@192.168.100.2"
EXEC_MIDDLEBOX="$SSH_CMD root@192.168.100.3"
EXEC_SERVER="$SSH_CMD root@192.168.100.4"

. . :

$EXEC_CLIENT echo hello from client
$EXEC_SERVER echo hello from server

-, ? bash- Heredoc. Heredoc :

$EXEC_CLIENT << EOF
    echo hello from client
    ls
    pwd
EOF

$EXEC_MIDDLEBOX << EOF
    echo hello from middlebox
    some_another_command
EOF

EOF — , . EOF , , .

, set -xeuo pipefail. :

$EXEC_MIDDLEBOX << EOF
    set -xeuo pipefail

    command1
    command2 | command3
EOF

, :

-x bash- ;
-e , ;
-u , ;
-o pipeline , - .

, - — .

, . bash- , - . :

$EXEC_MIDDLEBOX << EOF
    set -xeuo pipefail

    command1
    ! command2
EOF

: command1 0, command2 , .

$SCP_CMD l3fwd-acl-1.0.0.deb root@192.168.100.3:~

$EXEC_MIDDLEBOX << EOF
    set -xeuo pipefail

    dpkg -i l3fwd-acl-1.0.0.deb

    echo 256 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages
    mkdir -p /mnt/huge
    mount -t hugetlbfs nodev /mnt/huge

    modprobe uio_pci_generic
    dpdk-devbind --bind=uio_pci_generic ens4 ens5

    echo "R0.0.0.0/0 192.168.102.0/24 0 : 65535 0 : 65535 0x0/0x0 1" > /etc/rule_ipv4.db
    echo "R0.0.0.0/0 192.168.101.0/24 0 : 65535 0 : 65535 0x0/0x0 0" >> /etc/rule_ipv4.db

    echo "R0:0:0:0:0:0:0:0/0 0:0:0:0:0:0:0:0/0 0 : 65535 0 : 65535 0x0/0x0 0" > /etc/rule_ipv6.db

    daemon --name l3fwd --unsafe --output /var/log/l3fwd -- l3fwd-acl \
        -l 1 \
        -n 4 \
        -- \
        -p 0x3 \
        -P \
        --config="(0,0,1),(1,0,1)" \
        --rule_ipv4="/etc/rule_ipv4.db" \
        --rule_ipv6="/etc/rule_ipv6.db"
EOF

, ( ):

deb-;
256 2 (DPDK- - );
poll-mode uio_pci_generic ( Ubuntu Server). , DPDK ;
ens4 ( ) ens5 ( ) uio_pci_generic;
rule_ipv4.db IPv4 : 192.168.102.0/24 1 ( , ), 192.168.101.0/24 0 ( );
rule_ipv6.db, " 0". IPv6 , DPDK ;
l3fwd daemon. , l3fwd: https://doc.dpdk.org/guides/sample_app_ug/l3_forward.html

, DPDK :

.deb ;
;
;
;
, , uio_pci_generic;
.

: " ", - . , . .

: , DPDK- ( unit-, ), , : - :

$EXEC_CLIENT arp -s 192.168.101.3 52:54:56:00:00:00
$EXEC_SERVER arp -s 192.168.102.3 52:54:56:11:00:00

$EXEC_CLIENT << EOF
    set -xeuo pipefail

    ping -c 5 192.168.102.2
    wget --timeout=5 --tries=1 http://192.168.102.2
EOF

ARP-

, , ARP-.

? , l3fwd,

DPDK, , ARP.

l3fwd ,

rule_ipv4.db rule_ipv6.db,

: , / , --.

:

, .

, middlebox

, MAC-

Ethernet- ( client server ).

:

destination MAC-.

ARP-.

# =======================================
#  ,  tcp 
# =======================================

$EXEC_MIDDLEBOX << EOF
    set -xeuo pipefail

    daemon --name l3fwd --stop

    #     
    echo "@0.0.0.0/0 0.0.0.0/0 0 : 65535 0 : 65535 0x06/0xff" > /etc/rule_ipv4.db

    echo "R0.0.0.0/0 192.168.102.0/24 0 : 65535 0 : 65535 0x0/0x0 1" >> /etc/rule_ipv4.db
    echo "R0.0.0.0/0 192.168.101.0/24 0 : 65535 0 : 65535 0x0/0x0 0" >> /etc/rule_ipv4.db

    daemon --name l3fwd --unsafe --output /var/log/l3fwd -- l3fwd-acl \
        -l 1 \
        -n 4 \
        -- \
        -p 0x3 \
        -P \
        --config="(0,0,1),(1,0,1)" \
        --rule_ipv4="/etc/rule_ipv4.db" \
        --rule_ipv6="/etc/rule_ipv6.db"
EOF

# =======================================
# ,  ping  ,
#  http  - 
# =======================================

$EXEC_CLIENT << EOF
    set -xeuo pipefail

    ping -c 5 192.168.102.2
    ! wget --timeout=5 --tries=1 http://192.168.102.2
EOF

, wget : , , wget .

`run_tests.sh`

, , run_clean.sh.

: : DPDK- middlebox . ? , ( ) DPDK- . , , . ?

. , ? , . : , init, .

# =======================================
#   client
# =======================================
if ! virsh list --all | grep -q " client "
then
    virt-builder ubuntu-18.04 \
        --format qcow2 \
        --output client.qcow2 \
        --hostname client \
        --install wget,net-tools \
        --root-password password:1111 \
        --run-command "ssh-keygen -A" \
        --run-command "sed -i \"s/.*PermitRootLogin.*/PermitRootLogin yes/g\" /etc/ssh/sshd_config" \
        --copy-in netcfg_client.yaml:/etc/netplan/

    virt-install \
        --import \
        --name client \
        --ram 1024 \
        --disk client.qcow2 \
        --network network=net_for_ssh \
        --network network=net_1,mac=52:54:56:11:00:00 \
        --noautoconsole

    virsh snapshot-create-as client --name init
else
    virsh snapshot-revert client --snapshotname init
fi

, init.

, , : , .deb- DPDK, . .

, , :

# =======================================
#   net_1
# =======================================
if ! virsh net-list --all | grep -q " net_1 "
then
    virsh net-define net_1.xml
    virsh net-start net_1
fi

, , , . : !

, , , , . : , , , (end-to-end) .

, , : DPDK- ( Ubuntu Server 18.04) ( ping wget). , .deb . , , ( , ). .

: (run_tests.sh run_clean.sh), xml- yaml-. VCS. .

, , " — ". , . , " ", .

"", "" "" , , , , . Proof Of Concept . - … . , , , , . ...

→

基于QEMU的系统测试自动化（第2/2部分）

我们想得到什么？

我们将测试什么

!

run_tests.sh

More articles:

`run_tests.sh`