NetworkManager-ci
NetworkManager-ci
This repo contains a set of integration tests for NetworkManager and CentOS 8 Stream based VM test instructions
Nightly status (CentOS CI)
Code Branch | Build Status |
---|---|
main | |
1.48.x | |
1.46.x |
Howto execute basic test suite manually on localhost
-
Prerequisites
- CentOS qcow2 image, running libvirtd
Name resolution of or port forwarding to a given VM are described in Networking tips for local VMs section below.dnf -y install virt-install /usr/bin/virt-sysprep virt-viewer libvirtsystemctl start libvirtdSSH_KEY=/home/vbenes/.ssh/id_rsa.pubIMG=CentOS-Stream-GenericCloud-8-20210603.0.x86_64.qcow2wget https://cloud.centos.org/centos/8-stream/x86_64/images/$IMGvirt-sysprep -root-password password:centos -a $IMG# We need sudo to access default bridged networkingsudo virt-install --name CentOS_8_Stream --memory 4096 --vcpus 4 --disk $IMG,bus=sata --import --os-variant centos-stream8 --network default# Virsh doesn't sort machines so you may need to use different VM# than the first. Alternatives are name resolution or port forwarding.IP=$(sudo virsh net-dhcp-leases default | grep ipv4 | awk '{print $5}' |head -1 | awk -F '/' '{print $1}')ssh-copy-id root@$IP
- CentOS qcow2 image, running libvirtd
-
Running Tests
-
with NM compilation
# NMCI test code. NMCI main should work everywhereTEST_BRANCH='main'# REFSPEC of your NM code change, work with repo belowREFSPEC='main'# Change to whatever repo you want to compile NM fromNM_REPO='https://gitlab.freedesktop.org/vbenes/NetworkManager'# Choose a list of features you want to test, you can have 'all' to test everythingFEATURES='adsl, bond'INSTALL1="dnf install -y git python3 wget"INSTALL2="python3 -m pip install python-gitlab pyyaml"NMCI_URL="https://gitlab.freedesktop.org/NetworkManager/NetworkManager-ci.git"CLONE="git clone $NMCI_URL; cd NetworkManager-ci; git checkout $TEST_BRANCH"TEST="cd NetworkManager-ci; python3 run/centos-ci/node_runner.py -t $TEST_BRANCH -c $REFSPEC -f \"$FEATURES\" -r $NM_REPO"ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no root@$IP $INSTALL1 && \ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no root@$IP $INSTALL2 && \ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no root@$IP $CLONE && \ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no root@$IP $TEST -
you can avoid compilation and use already installed packages
# NMCI test code. NMCI main should work everywhereTEST_BRANCH='vb/nmtest'# Choose a list of features you want to test, you can have 'all' to test everythingFEATURES='all'INSTALL1="dnf install -y git python3 wget"INSTALL2="python3 -m pip install python-gitlab pyyaml"NMCI_URL="https://gitlab.freedesktop.org/NetworkManager/NetworkManager-ci.git"CLONE="git clone $NMCI_URL;cd NetworkManager-ci; git checkout $TEST_BRANCH"TEST="cd NetworkManager-ci; python3 run/centos-ci/node_runner.py -t $TEST_BRANCH -f \"$FEATURES\" -D"ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no root@$IP $INSTALL1 && \ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no root@$IP $INSTALL2 && \ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no root@$IP $CLONE && \ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no root@$IP $TEST -
or you can just ssh into the machine and run
cd /root/NetworkManager-ci# Test by test as defined in mapper.txtrun/runtest.sh your_desired_test# Feature by feature as listed too in mapper.txtrun/runfeature.sh your_desired_feature
-
-
Results check
- you will see execution progress as it goes ( tests do have 10m timeout to prevent lockup )
- there is a summary at the end
How to write a NMCI test
-
We use slightly modified python-behave framework to execute tests
-
It's quite readable and easy to learn
-
Let's describe directory structure and files of NMCI first
-
filemapper.yaml- use for driving tests
- all tests names are written there (together with features)
- all dependencies and basically all metadata is there
dirnmci- various scripts used for driving tests
- the most interesting are tags that are used for preparing and cleaning environment
- we have unit tests for version_control here, libs for tags, run for running commands, etc
scriptnmci/helpers/version_control.py- we have just one NMCI branch for all NM versions, RHELs, Fedora, CentOSes
- this is the control mechanism if we need to skip test here and there
-
dirfeatures
dirscenarios- sets of features in
files.feature - test itself will be described deeper later on
- sets of features in
scriptenvironment.py- script driving setup and teardown of each test
- includes
to be more readablenmci/tags.py - collects all logs and creates html log
dirsteps- all steps (aka test lines) are defined here
- strictly python, with paramterized decorators used in features itself
- categorized into various functional areas like bond/bridge/team/connection/commands, etc
-
dirprepare- various scripts for preparing environment
- vethsetup.sh
- this one creates whole 11 device wide test bed
- executed at first test execution if needed
- some CI machines have equivalent environment set up by physical switches or virt/cloud
VM settings. If you need to tweak the environment for your scenario, use
steps instead, please.Prepare simulated test...
- envsetup.sh
- installs various packages and tunes environment
- executed at first test execution
-
- various executors for various envs
- runtest.sh
- the main driver of tests
- execution of test looks like: run/runtest.sh test_name
- to embed everything to HTML use: NMCI_DEBUG=yes run/runtest.sh test_name
- runfeature.sh
- doing the same as
but for whole featuresruntest.sh
for examplerun/runfeature.sh bond
just lists testsrun/runfeature.sh bond --dry
- doing the same as
-
- various files and reproducers needed for tests
-
Let's use an example test to describe how we do that *
# Reference to bugzilla, doing nothing@rhbz1915457# Version control, test will run on F32+ or RHEL8.4+# with NM of 1.30+ only. Test will be skipped in CentOS@ver+=1.30 @rhelver+=8.4 @fedver+=32 @skip_in_centos# Bond and slaves residuals will be deleted after test# Test name as stated in mapper.txt. Must be last tag and sole tag on the line@bond_8023ad_with_vlan_srcmac# Human readable test name as stated in HTML reportScenario: nmcli - bond - options - mode set to 802.3ad with vlan+srcmax# Step for creation a NM profile with options* Add "bond" connection named "bond0" for device "nm-bond" and options"""bond.options 'mode=802.3ad,miimon=100,xmit_hash_policy=vlan+srcmac'"""# Step for creation slave connection, similar to above can be used too, we have two* Add slave connection for master "nm-bond" on device "eth1" named "bond0.0"# Bring up the connection, you can use down too* Bring "up" connection "bond0.0"# You can execute various commands too* Execute "echo 'Hello world'"# Check various value via period of time (once a second). Then keyword is useles, just marking results more visibleThen "Bonding Mode: IEEE 802.3ad Dynamic link aggregation" is visible with command "cat /proc/net/bonding/nm-bond" in "5" seconds# Check various value twice (default)Then "Transmit Hash Policy:\s+vlan\+srcmac" is visible with command "cat /proc/net/bonding/nm-bond"# Check various value immediatelyThen "Transmit Hash Policy:\s+vlan\+srcmac" is visible with command "cat /proc/net/bonding/nm-bond" in "1" seconds# Very bond specific step checking if bond device is up.Then Check bond "nm-bond" link state is "up" -
Reports
- You can see stdout output when running from command line
- You can see nice HTML reports too stored in /tmp
- PASS report has just after cleanup info in it
- FAIL report has all commands listed and logs added from before, after, after cleanup
How to test your changes
- Run your new or updated scenario by hand:
as already mentioned. Therun/runtest.sh my_new_scenario
script comes with some handy features:run/runtest.sh- bash completion is available so you can type just
and it will complete the command torun/runtest.sh my_new<tab>run/runtest.sh my_new_scenario - also, the
script is forgiving of @-sign at the start of test_tag for mouse-friendly tag pastingrun/runtest.sh - when you need to pretend that NM version is different from one actually installed, nmci will honor version
set in environment variable such as NM_VERSION=1.39.1
produces terse report for PASSing scenarios by default to save space required by CI. To get full output always, set (as already mentioned)run/runtest.sh
environment variable toNMCI_DEBUGyes
- bash completion is available so you can type just
- Unit tests
- you can run them yourself before submitting or updating a MR:
NetworkManager-ci$ python3 -m pytest nmci/test_nmci.py
- you can run them yourself before submitting or updating a MR:
- or you can create a Merge Request in Freedesktop Gitlab right away and CI tooling will run both of these for you (changed scenarios only if unit tests pass successfully).
@ver
Version Tags
Tests have tags (with an
). The @ver tag checks the version of NetworkManager to run or skip the test.
See also
script. Use following best practice for choosing a version tag.
-
prefer
over@ver+=NUMBER
because it is nicer to state the version when the feature starts working, and not the version before.@ver+NUMBER -
similarly, prefer
over@ver-NUMBER
to mirror a@ver-=NUMBER
. We can write two variants of a test with different version tags, and the version selection must only match for one variants of the test. Hence there should always be separate ranges. The use of@ver+=NUMBER
to mirror@ver-NUMBER
makes that clear.@ver+=NUMBER -
in NetworkManager, stable releases have an even second version number and an even third number (e.g. 1.42.2). A patch/feature/bugfix can only be added in a development version that leads up to a stable release. In the example, between the tag 1.42.1 and 1.42.2. The right way for a
tag is thus always the development version, like@ver+=
, because the patch is already upstream in@ver+=1.42.1
branch, which is currently between 1.42.1 and 1.42.2. The test should start working with 1.42.1+ already. Yes, early versions of 1.42.1+ don't have the patch yet, but it's more important to test the later versions of the development branch and run the test. The rightnm-1-42
is what@ver+=NUMBER
gives for the commit with the patch.git describe- on main branch, the second number is odd and the third number can be odd or even. For example, 1.43.6 is a
development version leading towards the next major release 1.44.0. Also there, a patch is always introduced
between two development snapshots, like between 1.43.5 and 1.43.6. Here too, we shall use
, so that testing current@ver+=1.43.5
branch (when 1.43.6 is not yet tagged) also covers the test. Note that we may take a devel snapshot (1.43.5) and package in RHEL. But that version doesn't have the patch yet, so for that RHEL package the testmain
will run but fail. There are are three possibilities. First, don't care. These are all just development versions and the situation resolves itself in a few days. Second, choose@ver+=1.43.5
, which would match for upstream builds but not for RHEL (note the package version from upstream builds adds a large 4th number). Third, add a RHEL specific override like@ver+=1.43.5.2000
.@ver+=1.43.5 @ver/rhel+=1.43.6
- on main branch, the second number is odd and the third number can be odd or even. For example, 1.43.6 is a
development version leading towards the next major release 1.44.0. Also there, a patch is always introduced
between two development snapshots, like between 1.43.5 and 1.43.6. Here too, we shall use
The tested NetworkManager version is parsed with a "stream" along the version
number. The stream is a variant of the NetworkManager package or a specific
build configuration. For example, we can have upstream release 1.44.2, which we
can build in copr, as a Fedora package or as a CentOS Stream 9 package. The
build configurations may slightly differ for the same tarball. For example, a
RHEL 8.7 package will be detected to have "rhel/8/7" stream. For most cases,
downstream RHEL/Fedora is very close to upstream so there is no difference
between streams. However, when having a stream like rhel/8/7, then the runner
will first search for version tags
, then
, then
and finally
. If any stream specific version tag is found, it
is evaluated and more general tags (like
) are ignored. For example,
means that RHEL 9 packages require a version
1.43.6 or newer, but all other packages require at least version 1.43.5.
Gitlab merge request pipelines (CI/CD)
Another possibility how to test the changes is to open a merge request in Gitlab. Pipeline first executes UnitTests (checks that the tests are consistent, well defined). Independently, remote jenkins triggers executes the tests, when a maintaner reviews and approves the code. The following apply for RHEL and CentOS trigger:
-
The tests can be retriggered by sending
message to merge request discussion.rebuild -
Only tests in changed
files are executed (whole features), if not overridden..feature- To override, use
in@RunTests:test1,test2,...
message, or in commit message or in merge request description.rebuild
forces to execute all tests.@RunTests:*
override, which executes only specified features. It can be in either in the latest commit message, in the merge request description or in the@RunFeatures:feature1,feature2,...
comment.rebuild
- To override, use
-
Latest NetworkManager main copr build (CentOS) or stock NetworkManager RPM (internal) are tested.
- To test on specific NetworkManager build, use
or@Build:main
, in@Build:0e5a4638807dc34c517988432120e3a5
message, or in commit message or in merge request description.rebuild - In CentOS, if specified
branch is found in COPR, COPR build will be used instead of build.@Build
- To test on specific NetworkManager build, use
-
By default, builds on CentOS 9 stream and RHEL8.X (latest release) are tested. These OSs can be overriden:
- by
GitLab comment. A build is going to be started just for the OS specified by therebuild OS_STRINGOS_STRING - by
lines in the latest commit message or in the MR description. Push events or@os:OS_STRING
comment in gitlab will trigger builds on these OSs only.rebuild
The format of
for respective OSs is:OS_STRING- for Centos X Stream:
or short:centosX-stream
. So for 9 stream, it'scXs
orcentos9-streamc9s - for RHEL:
, so for RHEL 9.3, the string isrhelX.Yrhel9.3 - for Fedora:
, thusFedora-release
orFedora-39
. JustFedora-Rawhide
is also recognized.rawhide
All the strings mentioned are case insensitive so
will still get recognized.@oS:fEDoRa-rAWhIdE - by
-
The priority of overrides is
message, commit message, merge request description.rebuild -
The tests can be skipped either by pushing with
, or "Rebase without pipeline button" in WebUI.git push -o ci.skip -
If you interlink merge requests (mention counterpart merge request in description), corresponding branch will be used for testing/build:
-
In a NetworkManager merge request description mention
orNetworkManager-ci!ABC
and it will use test from merge request numberedhttps://gitlab.freedesktop.org/NetworkManager/NetworkManager-ci/-/merge_requests/ABCABC -
In a NetworkManager-ci merge request description mention
orNetworkManager!XYZ
and it will build NetworkManager from merge request numberedhttps://gitlab.freedesktop.org/NetworkManager/NetworkManager/-/merge_requests/XYZXYZ -
Example: NetworkManager!1536 and NetworkManager-ci!1317
-
-
New build can be triggered by:
- commenting to open merge request:
message will re-trigger build with no additional overrides,rebuild- message containing overrides starting with
will automatically execute new builds,@
- pushing to open merge request - overrides defined in the commit message and merge-request description will be honored.
- commenting to open merge request:
In CentOS, older builds running on the same CentOS release from the same merge request are stopped to save resources. So, if you push to the merge request before the tests are finished, you may get "Aborted" message in the merge request discussion.
Gitlab common use-case examples:
-
When creating merge request, add overrides to description, one override per line:
This is example Merge Request ... @Build:main @RunTests:my_new_regression_test @OS:rhel8.5 -
Edit merge request description and add
comment to the discussion, which will force the new run of the tests with overrides. The overrides will apply for further pushes to the merge request.rebuild -
For one-time overrides, you can either do rebuild comment in merge request discussion:
@Build:nm-1-38 @RunTests:my_test1,my_test2 rebuild c8s # this line can be ommited, works only in CentOS, rhel will be executed too.or specify overrides in the last commit before pushing to the merge request:
This adds feature XYZ to the testsuite # this feature is not yet merged in NetworkManager @Build:my_supporting_branch_in_NM_repository # We want to execute all tests, not only changed files @RunTests:*
Accessing reports over HTTP
When testing at different machine than that running your browser, it is handy to have HTTP server on the test
machine and some means to auto-archive your test report from /tmp somewhere else. NetworkManager-ci come
with script that will configure
on your machine to serve the copied reports on port 8080 and a
little system service that watches /tmp. When new report is created, the service copies it with a timestamp
to
's DocumentRoot so you can compare multiple runs of the same scenario.
Service can be set up by script
or manually by copying service and its file
to appropriate location and configuring web server as you need. CI machines install these by default so you can access
their test reports this way when the test run is still underway and CI didn't collect them yet.
Networking tips for local VMs
Default libvirt networking works just OK but leaves a bit of convenience or features for regular use or when one has multiple VMs. Following sections feature tips to make local VM network more convenient or more accessible from outside.
Name-based networking for local libvirt networks
For local libvirt VMs with floating bridges or NAT'd networks, there are several good ways how to access VMs by name:
- libvirt-nss which when set up according to their docs allows you to access
VMs by their ‘libvirt domain’ name (
nsswitch module) or by guest's hostname (libvirt_guest
nsswitch module). Prerequisite for that is however that libvirt gets IP information from within the guest system via agent, you can verify if this is the case that you can see the IP when you run:libvirt
names exposed this way are always without a domainvirsh domifaddr DOMAIN_NAME - by hooking up libvirt-run dnsmasq that works as DHCP and DNS server for
libvirt networks to the system resolver. This is achiveved by:type="nat"-
setting
and<domain name="DOMAIN"/>
for the network<forward mode='nat'/> -
pointing system DNS resolver to libvirt-run
for this network. This is pretty easy to configure statically fordnsmasq
ordnsmasq
however if you rununbound
, it's only possible to configure specific DNS server for a given domain by changing runtime configuration.systemd-resolvedIt's possible to automate this using libvirt hooks mechanism and
's commandssystemd
orresolvectl
but neither project ships necessary glue script itself. Therefore I (@djasa) created this script that is fired by libvirt network events and when relevant, updatessystemd-resolve
about it. This configuration allows me to use gnome-boxes VMs with bridge/NAT network managed by system-level libvirt (or better saysystemd-resolved
these days) and access VMs by their hostname suffixed by custom domain, e.g. http://rhel9-nightly.vm:8080/ orvirtnetworkdssh [root@]rhel9-nightly.vm
-
NAT or isolated network however don't allow easy access from outside network to VM's ssh or http ports.
Host port forwarding to the guest
There is also option to forward given port on host system to another port in the VM when using qemu's
user mode networking and it's
option. This is not integrated to libvirt, so in order to use this option, you need to add
namespace URI to libvit domain xml:
<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'> <name>nmci-vm</name> …
that allows specifying custom arguments (env vars and others, see libvirt docs for more information). Options that create user-mode network interface for the guest with port forwarding from the host can be:
… <qemu:commandline> <qemu:arg value='-netdev'/> <qemu:arg value='user,id=mynet.9,net=192.168.120.0/24,hostfwd=tcp::4922-:22,hostfwd=::4980-:8080'/> <qemu:arg value='-device'/> <qemu:arg value='virtio-net-pci,netdev=mynet.9'/> </qemu:commandline></domain>
Notes:
specifies network range used by qemu. You may want to choose range that won't collide with private networks you need to reach from the guest. Guest by default gets .15 address of the range (can be changed bynet=192.168.120.0/24
according to Qemu docs), to this addres (192.168.120.15 for this configuration, 10.0.2.15 with nodhcpstart=
specification) qemu redirects host ports configured bynetwork=
option unles overriden therehostfwd
specifies host forwarding itself:hostfwd=::4922-:22- empty source address means v4 wildcard (
). If you wonder why qemu doesn't support dual-stack wildcard0.0.0.0
in 2022, you're not alone.:: - empty guest address (between
and-
) means .15 address of range specified by:
or address specified bynet=dhcpstart=
- empty source address means v4 wildcard (
- Other arguments are well described in
manual page or online docs.qemu
Please note that NetworkManager-ci expects just one interface with outside connectivity so this is either-or situation, there is either bridged networking or qemu-provided port forwarding. Having port forwarding with bridge networking would need port forwarding configured on system level (doable using already mentioned libvirt hooks mechanism but not used by anyone in the team right now. Feel free to contribute a working solution if you have one).
Distribution dependence
nmci is primarily developed and maintained on RHEL/CentOS/Fedora systems so there are distro-specific parts of the code. They are primarily in
. Distro-specific parts are already largely or completely taken out to specific files inenvsetup
directory. Adding more of them for other distros should be straightforwardprepare/envsetup- other scripts in
and some tag implementations inprepare
(and few cases intags.py
) use hardwiredprepare.py
/rpm
/yum
invocations. These would need to be generalized or where suitable, package instalation moved todnfenvsetup - version detection code in
. Analogous may be needed for finer-grained decisions of what scenarios should run on given systemnmci/misc.py - CI runners in in run
. You may try ansible versionsetup_pbl.py
which at least abstracts away package installation.setup_pbl.yml