ovs-2.9.0/000077500000000000000000000000001324262074100123325ustar00rootroot00000000000000ovs-2.9.0/.gitignore000066400000000000000000000014311324262074100143210ustar00rootroot00000000000000#*# *.a *.d *.gcno *.gcda *.ko *.la *.lo *.loT *.mod.c *.o *.obj *.exe *.exp *.ilk *.lib *.pdb *.pyc *.so *.suo **/*.sym *~ *,cover .#* .*.cmd .*.swp .coverage .deps .dirstamp .libs .tmp_versions .gitattributes /Makefile /Makefile.in /aclocal.m4 /all-distfiles /all-gitfiles /autom4te.cache /build-arch-stamp /build-indep-stamp /compile /config.guess /config.h /config.h.in /config.log /config.status /config.sub /configure /configure-stamp /depcomp /distfiles /dist-docs /flake8-check /docs-check /install-sh /libtool /manpage-check /missing /missing-distfiles /package.m4 /stamp-h1 /_build-gcc /_build-clang Module.symvers TAGS cscope.* tags _debian odp-netlink.h OvsDpInterface.h /.vagrant/ testsuite.tmp.orig /rpm/ /openvswitch*.tar.gz /tests/lcov/ /Documentation/_build /.venv /cxx-check ovs-2.9.0/.mailmap000066400000000000000000000105051324262074100137540ustar00rootroot00000000000000# See git-shortlog(1) for official documentation of this file format. # # Name # Use "Name" for any commit with address "". This is useful when a person # has commits using different spellings of their name, but with the same email # address. # # Name # Use "Name" and "" for any commit with the address "". This is useful # when a person has used more than one email address. Aaron Conole Aaron Conole Aaron Rosen Alex Wang Alexey I. Froloff Alin Serdean Alin Serdean Andy Zhou Andy Zhou Andy Zhou Ansis Atteka Ansis Atteka Anupam Chanda Ariel Tubaltsev Babu Shanmugam Ben Pfaff Bruce Davie Bruce Davie Chandra Sekhar Vejendla Daniele Di Proietto Daniele Di Proietto Ed Maste Ethan J. Jackson Fischetti, Antonio Flavio Fernandes Flavio Leitner Gal Sagie Gurucharan Shetty Gurucharan Shetty Henry Mai Hui Kang Ian Campbell James Page Jarno Rajahalme Jarno Rajahalme Jean Tourrilhes Jean Tourrihles Jean Tourrilhes Jesse Gross Joe Stringer Joe Stringer Justin Pettit Kmindg Kyle Mestery Mauricio Vasquez Miguel Angel Ajo Neil McKee Ofer Ben-Yacov Polehn, Mike A Pravin B Shelar Raju Subramanian Ralf Spenneberg Rami Rosen Ramu Ramamurthy Robert Åkerblom-Andersson Romain Lenglet Romain Lenglet Russell Bryant Ryan Moats Sabyasachi Sengupta Saurabh Shah Shad Ansari Shih-Hao Li Simon Horman Stephen Finucane Thomas F. Herbert Thomas Graf Thomas Graf Wei Li YAMAMOTO Takashi YAMAMOTO Takashi YAMAMOTO Takashi Zhi Yong Wu Zoltan Kiss Zong Kai LI ovs-2.9.0/.travis.yml000066400000000000000000000016371324262074100144520ustar00rootroot00000000000000language: c compiler: - gcc - clang os: - linux addons: apt: packages: - bc - gcc-multilib - libssl-dev - llvm-dev - libjemalloc1 - libjemalloc-dev - libnuma-dev - python-sphinx before_install: ./.travis/${TRAVIS_OS_NAME}-prepare.sh before_script: export PATH=$PATH:$HOME/bin sudo: false env: - OPTS="--disable-ssl" - TESTSUITE=1 KERNEL=3.16.47 - TESTSUITE=1 OPTS="--enable-shared" - BUILD_ENV="-m32" OPTS="--disable-ssl" - KERNEL=3.16.47 DPDK=1 - KERNEL=3.16.47 DPDK=1 OPTS="--enable-shared" - KERNEL=4.13 - KERNEL=4.12.11 - KERNEL=4.9.48 - KERNEL=4.4.87 - KERNEL=4.1.43 - KERNEL=3.10.107 - TESTSUITE=1 LIBS=-ljemalloc matrix: include: - os: osx compiler: clang env: OPTS="--disable-ssl" script: ./.travis/${TRAVIS_OS_NAME}-build.sh $OPTS notifications: email: recipients: - ovs-build@openvswitch.org ovs-2.9.0/.travis/000077500000000000000000000000001324262074100137205ustar00rootroot00000000000000ovs-2.9.0/.travis/linux-build.sh000077500000000000000000000067301324262074100165210ustar00rootroot00000000000000#!/bin/bash set -o errexit KERNELSRC="" CFLAGS="-Werror" SPARSE_FLAGS="" EXTRA_OPTS="" function install_kernel() { if [[ "$1" =~ ^4.* ]]; then PREFIX="v4.x" elif [[ "$1" =~ ^3.* ]]; then PREFIX="v3.x" else PREFIX="v2.6/longterm/v2.6.32" fi wget https://www.kernel.org/pub/linux/kernel/${PREFIX}/linux-${1}.tar.gz tar xzvf linux-${1}.tar.gz > /dev/null cd linux-${1} make allmodconfig # Cannot use CONFIG_KCOV: -fsanitize-coverage=trace-pc is not supported by compiler sed -i 's/CONFIG_KCOV=y/CONFIG_KCOV=n/' .config # stack validation depends on tools/objtool, but objtool does not compile on travis. # It is giving following error. # >>> GEN arch/x86/insn/inat-tables.c # >>> Semantic error at 40: Unknown imm opnd: AL # So for now disable stack-validation for the build. sed -i 's/CONFIG_STACK_VALIDATION=y/CONFIG_STACK_VALIDATION=n/' .config make oldconfig # Older kernels do not include openvswitch if [ -d "net/openvswitch" ]; then make net/openvswitch/ else make net/bridge/ fi KERNELSRC=$(pwd) if [ ! "$DPDK" ]; then EXTRA_OPTS="--with-linux=$(pwd)" fi echo "Installed kernel source in $(pwd)" cd .. } function install_dpdk() { if [ -n "$DPDK_GIT" ]; then git clone $DPDK_GIT dpdk-$1 cd dpdk-$1 git checkout tags/v$1 else wget http://fast.dpdk.org/rel/dpdk-$1.tar.gz tar xzvf dpdk-$1.tar.gz > /dev/null DIR_NAME=$(tar -tf dpdk-$1.tar.gz | head -1 | cut -f1 -d"/") if [ $DIR_NAME != "dpdk-$1" ]; then mv $DIR_NAME dpdk-$1; fi cd dpdk-$1 fi find ./ -type f | xargs sed -i 's/max-inline-insns-single=100/max-inline-insns-single=400/' find ./ -type f | xargs sed -i 's/-Werror/-Werror -Wno-error=inline/' echo 'CONFIG_RTE_BUILD_FPIC=y' >>config/common_linuxapp sed -ri '/EXECENV_CFLAGS = -pthread -fPIC/{s/$/\nelse ifeq ($(CONFIG_RTE_BUILD_FPIC),y)/;s/$/\nEXECENV_CFLAGS = -pthread -fPIC/}' mk/exec-env/linuxapp/rte.vars.mk make config CC=gcc T=x86_64-native-linuxapp-gcc make CC=gcc RTE_KERNELDIR=$KERNELSRC echo "Installed DPDK source in $(pwd)" cd .. } function configure_ovs() { ./boot.sh && ./configure $* } if [ "$KERNEL" ] || [ "$DPDK" ]; then install_kernel $KERNEL fi if [ "$DPDK" ]; then if [ -z "$DPDK_VER" ]; then DPDK_VER="17.11" fi install_dpdk $DPDK_VER if [ "$CC" = "clang" ]; then # Disregard cast alignment errors until DPDK is fixed CFLAGS="$CFLAGS -Wno-cast-align" fi EXTRA_OPTS="$EXTRA_OPTS --with-dpdk=./dpdk-$DPDK_VER/build" elif [ "$CC" != "clang" ]; then # DPDK headers currently trigger sparse errors SPARSE_FLAGS="$SPARSE_FLAGS -Wsparse-error" fi configure_ovs $EXTRA_OPTS $* # Only build datapath if we are testing kernel w/o running testsuite if [ "$KERNEL" ] && [ ! "$TESTSUITE" ] && [ ! "$DPDK" ]; then cd datapath fi if [ "$CC" = "clang" ]; then make -j2 CFLAGS="$CFLAGS -Wno-error=unused-command-line-argument" elif [[ $BUILD_ENV =~ "-m32" ]]; then # Disable sparse for 32bit builds on 64bit machine make -j2 CFLAGS="$CFLAGS $BUILD_ENV" else make -j2 CFLAGS="$CFLAGS $BUILD_ENV $SPARSE_FLAGS" C=1 fi if [ "$TESTSUITE" ] && [ "$CC" != "clang" ]; then if ! make distcheck TESTSUITEFLAGS=-j4 RECHECK=yes; then # testsuite.log is necessary for debugging. cat */_build/tests/testsuite.log exit 1 fi fi exit 0 ovs-2.9.0/.travis/linux-prepare.sh000077500000000000000000000006631324262074100170570ustar00rootroot00000000000000#!/bin/bash set -ev # Build and install sparse. # # Explicitly disable sparse support for llvm because some travis # environments claim to have LLVM (llvm-config exists and works) but # linking against it fails. git clone git://git.kernel.org/pub/scm/devel/sparse/chrisl/sparse.git cd sparse && make HAVE_LLVM= install && cd .. pip install --disable-pip-version-check --user six flake8 hacking pip install --user --upgrade docutils ovs-2.9.0/.travis/osx-build.sh000077500000000000000000000010071324262074100161630ustar00rootroot00000000000000#!/bin/bash set -o errexit CFLAGS="-Werror $CFLAGS" EXTRA_OPTS="" function configure_ovs() { ./boot.sh && ./configure $* } configure_ovs $EXTRA_OPTS $* if [ "$CC" = "clang" ]; then make CFLAGS="$CFLAGS -Wno-error=unused-command-line-argument" else make CFLAGS="$CFLAGS $BUILD_ENV" fi if [ "$TESTSUITE" ] && [ "$CC" != "clang" ]; then if ! make distcheck RECHECK=yes; then # testsuite.log is necessary for debugging. cat */_build/tests/testsuite.log exit 1 fi fi exit 0 ovs-2.9.0/.travis/osx-prepare.sh000077500000000000000000000002371324262074100165260ustar00rootroot00000000000000#!/bin/bash set -ev pip2 install --user six pip2 install --user --upgrade docutils brew update || true brew uninstall libtool && brew install libtool || true ovs-2.9.0/AUTHORS.rst000066400000000000000000000760271324262074100142250ustar00rootroot00000000000000.. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. Convention for heading levels in Open vSwitch documentation: ======= Heading 0 (reserved for the title in a document) ------- Heading 1 ~~~~~~~ Heading 2 +++++++ Heading 3 ''''''' Heading 4 Avoid deeper levels because they do not render well. ======= Authors ======= The following people authored or signed off on commits in the Open vSwitch source code or webpage version control repository. =============================== =============================================== Name Email =============================== =============================================== Aaron Conole aconole@redhat.com Aaron Rosen arosen@clemson.edu Alan Pevec alan.pevec@redhat.com Alexander Duyck alexander.h.duyck@redhat.com Alexandru Copot alex.mihai.c@gmail.com Alexei Starovoitov ast@plumgrid.com Alexey I. Froloff raorn@raorn.name Alex Wang ee07b291@gmail.com Alfredo Finelli alf@computationes.de Alin Balutoiu abalutoiu@cloudbasesolutions.com Alin Serdean aserdean@cloudbasesolutions.com Ambika Arora ambika.arora@tcs.com Amit Bose bose@noironetworks.com Amitabha Biswas azbiswas@gmail.com Anand Kumar kumaranand@vmware.com Andrew Evans aevans@nicira.com Andrew Beekhof abeekhof@redhat.com Andrew Kampjes a.kampjes@gmail.com Andrew Lambeth wal@nicira.com Andy Hill hillad@gmail.com Andy Southgate andy.southgate@citrix.com Andy Zhou azhou@ovn.org Ankur Sharma ankursharma@vmware.com Anoob Soman anoob.soman@citrix.com Ansis Atteka aatteka@nicira.com Antonio Fischetti antonio.fischetti@intel.com Anupam Chanda achanda@nicira.com Ariel Tubaltsev atubaltsev@vmware.com Arnoldo Lutz arnoldo.lutz.guevara@hpe.com Arun Sharma arun.sharma@calsoftinc.com Aryan TaheriMonfared aryan.taherimonfared@uis.no Ashish Varma ashishvarma.ovs@gmail.com Ashwin Swaminathan ashwinds@arista.com Babu Shanmugam bschanmu@redhat.com Ben Pfaff blp@ovn.org Ben Warren ben@skyportsystems.com Benli Ye daniely@vmware.com Bert Vermeulen bert@biot.com Bhanuprakash Bodireddy bhanuprakash.bodireddy@intel.com Billy O'Mahony billy.o.mahony@intel.com Binbin Xu xu.binbin1@zte.com.cn Brian Kruger bkruger+ovsdev@gmail.com Bruce Davie bsd@nicira.com Bryan Phillippe bp@toroki.com Carlo Andreotti c.andreotti@m3s.it Casey Barker crbarker@google.com Chandra Sekhar Vejendla csvejend@us.ibm.com Christoph Jaeger cj@linux.com Chris Wright chrisw@sous-sol.org Chuck Short zulcss@ubuntu.com Ciara Loftus ciara.loftus@intel.com Clint Byrum clint@fewbar.com Cong Wang amwang@redhat.com Conner Herriges conner.herriges@ibm.com Damien Millescamps damien.millescamps@6wind.com Dan Carpenter dan.carpenter@oracle.com Dan McGregor dan.mcgregor@usask.ca Dan Wendlandt dan@nicira.com Daniel Alvarez dalvarez@redhat.com Daniel Borkmann dborkman@redhat.com Daniel Hiltgen daniel@netkine.com Daniel Roman droman@nicira.com Daniele Di Proietto daniele.di.proietto@gmail.com Daniele Venturino venturino.daniele+ovs@gmail.com Danny Kukawka danny.kukawka@bisect.de Darrell Ball dlu998@gmail.com Dave Tucker dave@dtucker.co.uk David Erickson derickso@stanford.edu David Hill dhill@redhat.com David S. Miller davem@davemloft.net David Yang davidy@vmware.com Dennis Sam dsam@arista.com Devendra Naga devendra.aaru@gmail.com Dmitry Krivenok krivenok.dmitry@gmail.com Dominic Curran dominic.curran@citrix.com Dongdong dongdong1@huawei.com Dongjun dongj@dtdream.com Duan Jiong djduanjiong@gmail.com Duffie Cooley dcooley@nicira.com Dustin Lundquist dustin@null-ptr.net Ed Maste emaste@freebsd.org Ed Swierk eswierk@skyportsystems.com Edouard Bourguignon madko@linuxed.net Eelco Chaudron echaudro@redhat.com Esteban Rodriguez Betancourt estebarb@hpe.com Aymerich Edward edward.aymerich@hpe.com Edward Tomasz Napierała trasz@freebsd.org Eitan Eliahu eliahue@vmware.com Eohyung Lee liquidnuker@gmail.com Eric Dumazet edumazet@google.com Eric Garver e@erig.me Eric Sesterhenn eric.sesterhenn@lsexperts.de Ethan J. Jackson ejj@eecs.berkeley.edu Ethan Rahn erahn@arista.com Eziz Durdyyev ezizdurdy@gmail.com Flavio Fernandes flavio@flaviof.com Flavio Leitner fbl@redhat.com Francesco Fusco ffusco@redhat.com FUJITA Tomonori fujita.tomonori@lab.ntt.co.jp Gabe Beged-Dov gabe@begeddov.com Gaetano Catalli gaetano.catalli@gmail.com Gal Sagie gal.sagie@gmail.com Genevieve LEsperance glesperance@pivotal.io Geoffrey Wossum gwossum@acm.org Gianluca Merlo gianluca.merlo@gmail.com Giuseppe Lettieri g.lettieri@iet.unipi.it Glen Gibb grg@stanford.edu Guoshuai Li ligs@dtdream.com Guolin Yang gyang@nicira.com Guru Chaitanya Perakam gperakam@Brocade.com Gurucharan Shetty guru@ovn.org Han Zhou zhouhan@gmail.com Henry Mai hmai@nicira.com Hao Zheng hzheng@nicira.com Helmut Schaa helmut.schaa@googlemail.com Hiteshi Kalra hiteshi.kalra@tcs.com Huanle Han hanxueluo@gmail.com Hui Kang kangh@us.ibm.com Ian Campbell Ian.Campbell@citrix.com Ian Stokes ian.stokes@intel.com Ilya Maximets i.maximets@samsung.com Iman Tabrizian tabrizian@outlook.com Isaku Yamahata yamahata@valinux.co.jp IWASE Yusuke iwase.yusuke@gmail.com Jakub Libosvar libosvar@redhat.com Jakub Sitnicki jkbs@redhat.com James P. roampune@gmail.com James Page james.page@ubuntu.com Jamie Lennox jamielennox@gmail.com Jan Scheurich jan.scheurich@ericsson.com Jan Vansteenkiste jan@vstone.eu Jarno Rajahalme jarno@ovn.org Jason Kölker jason@koelker.net Jason Wessel jason.wessel@windriver.com Jasper Capel jasper@capel.tv Jean Tourrilhes jt@hpl.hp.com Jeremy Stribling strib@nicira.com Jeroen van Bemmel jvb127@gmail.com Jesse Gross jesse@kernel.org Jian Li lijian@ooclab.com Jing Ai jinga@google.com Jiri Benc jbenc@redhat.com Joe Perches joe@perches.com Joe Stringer joe@ovn.org Jonathan Vestin jonavest@kau.se Jorge Arturo Sauma Vargas jorge.sauma@hpe.com Jun Nakajima jun.nakajima@intel.com JunoZhu zhunatuzi@gmail.com Justin Pettit jpettit@ovn.org Kaige Fu fukaige@huawei.com Keith Amidon keith@nicira.com Ken Ajiro ajiro@mxw.nes.nec.co.jp Kenneth Duda kduda@arista.com Kentaro Ebisawa ebiken.g@gmail.com Kevin Lo kevlo@FreeBSD.org Kevin Traynor kevin.traynor@intel.com Khem Raj raj.khem@gmail.com Kmindg G kmindg@gmail.com Kris Murphy kriskend@linux.vnet.ibm.com Krishna Kondaka kkondaka@vmware.com Kyle Mestery mestery@mestery.com Kyle Upton kupton@baymicrosystems.com Lance Richardson lrichard@redhat.com Lars Kellogg-Stedman lars@redhat.com Lei Huang huang.f.lei@gmail.com Leif Madsen lmadsen@redhat.com Leo Alterman lalterman@nicira.com Lilijun jerry.lilijun@huawei.com Lili Huang huanglili.huang@huawei.com Linda Sun lsun@vmware.com Lior Neudorfer lior@guardicore.com Lorand Jakab lojakab@cisco.com Lorenzo Bianconi lorenzo.bianconi@redhat.com Luca Giraudo lgiraudo@nicira.com Lucas Alvares Gomes lucasagomes@gmail.com Lucian Petrut lpetrut@cloudbasesolutions.com Luigi Rizzo rizzo@iet.unipi.it Luis E. P. l31g@hotmail.com Lukasz Rzasik lukasz.rzasik@gmail.com Madhu Challa challa@noironetworks.com Marcin Mirecki mmirecki@redhat.com Mario Cabrera mario.cabrera@hpe.com Mark D. Gray mark.d.gray@intel.com Mark Hamilton mhamilton@nicira.com Mark Kavanagh mark.b.kavanagh@intel.com Mark Maglana mmaglana@gmail.com Mark Michelson mmichels@redhat.com Markos Chandras mchandras@suse.de Martin Casado casado@nicira.com Martino Fornasa mf@fornasa.it Maryam Tahhan maryam.tahhan@intel.com Matteo Croce mcroce@redhat.com Mauricio Vásquez mauricio.vasquezbernal@studenti.polito.it Maxime Coquelin maxime.coquelin@redhat.com Mehak Mahajan mmahajan@nicira.com Michael Arnaldi arnaldimichael@gmail.com Michal Weglicki michalx.weglicki@intel.com Mickey Spiegel mickeys.dev@gmail.com Miguel Angel Ajo majopela@redhat.com Mijo Safradin mijo@linux.vnet.ibm.com Mika Vaisanen mika.vaisanen@gmail.com Minoru TAKAHASHI takahashi.minoru7@gmail.com Murphy McCauley murphy.mccauley@gmail.com Natasha Gude natasha@nicira.com Neil McKee neil.mckee@inmon.com Neil Zhu zhuj@centecnetworks.com Nimay Desai nimaydesai1@gmail.com Nithin Raju nithin@vmware.com Niti Rohilla niti.rohilla@tcs.com Numan Siddique nusiddiq@redhat.com Ofer Ben-Yacov ofer.benyacov@gmail.com Or Gerlitz ogerlitz@mellanox.com Ori Shoshan ori.shoshan@guardicore.com Padmanabhan Krishnan kprad1@yahoo.com Panu Matilainen pmatilai@redhat.com Paraneetharan Chandrasekaran paraneetharanc@gmail.com Paul Boca pboca@cloudbasesolutions.com Paul Fazzone pfazzone@nicira.com Paul Ingram paul@nicira.com Paul-Emmanuel Raoul skyper@skyplabs.net Pavithra Ramesh paramesh@vmware.com Peter Downs padowns@gmail.com Philippe Jung phil.jung@free.fr Pim van den Berg pim@nethuis.nl pritesh pritesh.kothari@cisco.com Pravin B Shelar pshelar@nicira.com Przemyslaw Szczerbik przemyslawx.szczerbik@intel.com Quentin Monnet quentin.monnet@6wind.com Raju Subramanian rsubramanian@nicira.com Rami Rosen ramirose@gmail.com Ramu Ramamurthy ramu.ramamurthy@us.ibm.com Randall Sharo andall.sharo@navy.mil Ravi Kerur Ravi.Kerur@telekom.com Raymond Burkholder ray@oneunified.net Reid Price reid@nicira.com Remko Tronçon git@el-tramo.be Rich Lane rlane@bigswitch.com Richard Oliver richard@richard-oliver.co.uk Rishi Bamba rishi.bamba@tcs.com Rob Adams readams@readams.net Robert Åkerblom-Andersson Robert.nr1@gmail.com Robert Wojciechowicz robertx.wojciechowicz@intel.com Rob Hoes rob.hoes@citrix.com Roi Dayan roid@mellanox.com Romain Lenglet romain.lenglet@berabera.info Russell Bryant russell@ovn.org RYAN D. MOATS rmoats@us.ibm.com Ryan Wilson wryan@nicira.com Sairam Venugopal vsairam@vmware.com Sajjad Lateef slateef@nicira.com Saloni Jain saloni.jain@tcs.com Samuel Ghinet sghinet@cloudbasesolutions.com Sanjay Sane ssane@nicira.com Saurabh Mohan saurabh@cplanenetworks.com Saurabh Shah ssaurabh@nicira.com Saurabh Shrivastava saurabh.shrivastava@nuagenetworks.net Scott Lowe scott.lowe@scottlowe.org Scott Mann sdmnix@gmail.com Selvamuthukumar smkumar@merunetworks.com Sha Zhang zhangsha.zhang@huawei.com Shad Ansari shad.ansari@hpe.com Shan Wei davidshan@tencent.com Shashank Ram rams@vmware.com Shashwat Srivastava shashwat.srivastava@tcs.com Shih-Hao Li shli@nicira.com Shu Shen shu.shen@radisys.com Simon Horman horms@verge.net.au Simon Horman simon.horman@netronome.com Sorin Vinturis svinturis@cloudbasesolutions.com Steffen Gebert steffen.gebert@informatik.uni-wuerzburg.de Sten Spans sten@blinkenlights.nl Stephane A. Sezer sas@cd80.net Stephen Finucane stephen@that.guru Steve Ruan ruansx@cn.ibm.com Stuart Cardall developer@it-offshore.co.uk Sugesh Chandran sugesh.chandran@intel.com SUGYO Kazushi sugyo.org@gmail.com Tadaaki Nagao nagao@stratosphere.co.jp Terry Wilson twilson@redhat.com Tetsuo NAKAGAWA nakagawa@mxc.nes.nec.co.jp Thadeu Lima de Souza Cascardo cascardo@cascardo.eti.br Thomas F. Herbert thomasfherbert@gmail.com Thomas Goirand zigo@debian.org Thomas Graf tgraf@noironetworks.com Thomas Lacroix thomas.lacroix@citrix.com Timo Puha timox.puha@intel.com Timothy Redaelli tredaelli@redhat.com Todd Deshane deshantm@gmail.com Tom Everman teverman@google.com Torgny Lindberg torgny.lindberg@ericsson.com Tsvi Slonim tsvi@toroki.com Tuan Nguyen tuan.nguyen@veriksystems.com Tyler Coumbes coumbes@gmail.com Tony van der Peet tony.vanderpeet@alliedtelesis.co.nz Tonghao Zhang xiangxia.m.yue@gmail.com Valient Gough vgough@pobox.com Venkata Anil Kommaddi vkommadi@redhat.com Vishal Deep Ajmera vishal.deep.ajmera@ericsson.com Vivien Bernet-Rollande vbr@soprive.net wangqianyu wang.qianyu@zte.com.cn Wang Sheng-Hui shhuiw@gmail.com Wang Zhike wangzhike@jd.com Wei Li liw@dtdream.com Wei Yongjun yjwei@cn.fujitsu.com Wenyu Zhang wenyuz@vmware.com William Fulton William Tu u9012063@gmail.com Xiao Liang shaw.leon@gmail.com xu rong xu.rong@zte.com.cn YAMAMOTO Takashi yamamoto@midokura.com Yasuhito Takamiya yasuhito@gmail.com Yi-Hung Wei yihung.wei@gmail.com Yifeng Sun pkusunyifeng@gmail.com Yin Lin linyi@vmware.com Yu Zhiguo yuzg@cn.fujitsu.com Yuanhan Liu yuanhan.liu@linux.intel.com Yunjian Wang wangyunjian@huawei.com ZhengLingyun konghuarukhr@163.com Zoltán Balogh zoltan.balogh@ericsson.com Zoltan Kiss zoltan.kiss@citrix.com Zongkai LI zealokii@gmail.com Zhi Yong Wu zwu.kernel@gmail.com Zang MingJie zealot0630@gmail.com Zhenyu Gao sysugaozhenyu@gmail.com ZhiPeng Lu lu.zhipeng@zte.com.cn Zhou Yangchao 1028519445@qq.com wenxu wenxu@ucloud.cn wisd0me ak47izatool@gmail.com xushengping shengping.xu@huawei.com yinpeijun yinpeijun@huawei.com zangchuanqiang zangchuanqiang@huawei.com zhaojingjing zhao.jingjing1@zte.com.cn zhongbaisong zhongbaisong@huawei.com zhaozhanxu zhaozhanxu@163.com =============================== =============================================== The following additional people are mentioned in commit logs as having provided helpful bug reports or suggestions. =============================== =============================================== Name Email =============================== =============================================== Aaron M. Ucko ucko@debian.org Abhinav Singhal Abhinav.Singhal@spirent.com Adam Heath doogie@brainfood.com Ahmed Bilal numan252@gmail.com Alan Kayahan hsykay@gmail.com Alan Shieh ashieh@nicira.com Alban Browaeys prahal@yahoo.com Alex Yip alex@nicira.com Alexey I. Froloff raorn@altlinux.org Amar Padmanabhan amar@nicira.com Amey Bhide abhide@nicira.com Amre Shakimov ashakimov@vmware.com André Ruß andre.russ@hybris.com Andreas Beckmann debian@abeckmann.de Andrei Andone andrei.andone@softvision.ro Andrey Korolyov andrey@xdel.ru Anshuman Manral anshuman.manral@outlook.com Anton Matsiuk anton.matsiuk@gmail.com Anup Khadka khadka.py@gmail.com Anuprem Chalvadi achalvadi@vmware.com Ariel Tubaltsev atubaltsev@vmware.com Arkajit Ghosh arkajit.ghosh@tcs.com Atzm Watanabe atzm@stratosphere.co.jp Aurélien Poulain aurepoulain@viacesi.fr Bastian Blank waldi@debian.org Ben Basler bbasler@nicira.com Bhargava Shastry bshastry@sec.t-labs.tu-berlin.de Bob Ball bob.ball@citrix.com Brad Hall brad@nicira.com Brad Cowie brad@wand.net.nz Brailey Josh josh@faucet.nz Brandon Heller brandonh@stanford.edu Brendan Kelley bkelley@nicira.com Brent Salisbury brent.salisbury@gmail.com Brian Field Brian_Field@cable.comcast.com Bryan Fulton bryan@nicira.com Bryan Osoro bosoro@nicira.com Cedric Hobbs cedric@nicira.com Chris Hydon chydon@aristanetworks.com Christian Stigen Larsen cslarsen@gmail.com Christopher Paggen cpaggen@cisco.com Chunhe Li lichunhe@huawei.com Daniel Badea daniel.badea@windriver.com Darragh O'Reilly darragh.oreilly@hpe.com Dave Walker DaveWalker@ubuntu.com David Evans davidjoshuaevans@gmail.com David Palma palma@onesource.pt Derek Cormier derek.cormier@lab.ntt.co.jp Dhaval Badiani dbadiani@vmware.com DK Moon dkmoon@nicira.com Ding Zhi zhi.ding@6wind.com Dong Jun dongj@dtdream.com Dustin Spinhirne dspinhirne@vmware.com Edwin Chiu echiu@vmware.com Eivind Bulie Haanaes Enas Ahmad enas.ahmad@kaust.edu.sa Eric Lopez elopez@nicira.com Frido Roose fr.roose@gmail.com Gaetano Catalli gaetano.catalli@gmail.com Gavin Remaley gavin_remaley@selinc.com Georg Schmuecking georg.schmuecking@ericsson.com George Shuklin amarao@desunote.ru Gerald Rogers gerald.rogers@intel.com Ghanem Bahri bahri.ghanem@gmail.com Giuseppe de Candia giuseppe.decandia@gmail.com Gordon Good ggood@nicira.com Greg Dahlman gdahlman@hotmail.com Greg Rose gvrose8192@gmail.com Gregor Schaffrath grsch@net.t-labs.tu-berlin.de Gregory Smith gasmith@nutanix.com Guolin Yang gyang@vmware.com Gur Stavi gstavi@mrv.com Harish Kanakaraju hkanakaraju@vmware.com Hari Sasank Bhamidipalli hbhamidi@cisco.com Hassan Khan hassan.khan@seecs.edu.pk Hector Oron hector.oron@gmail.com Hemanth Kumar Mantri mantri@nutanix.com Henrik Amren henrik@nicira.com Hiroshi Tanaka htanaka@nicira.com Hiroshi Miyata miyahiro.dazu@gmail.com Hsin-Yi Shen shenh@vmware.com Hui Xiang xianghuir@gmail.com Hyojoon Kim joonk@gatech.edu Igor Ganichev iganichev@nicira.com Igor Sever igor@xorops.com Jacob Cherkas jcherkas@nicira.com Jad Naous jnaous@gmail.com Jamal Hadi Salim hadi@cyberus.ca James Schmidt jschmidt@nicira.com Jan Medved jmedved@juniper.net Janis Hamme janis.hamme@student.kit.edu Jari Sundell sundell.software@gmail.com Javier Albornoz javier.albornoz@hpe.com Jed Daniels openvswitch@jeddaniels.com Jeff Merrick jmerrick@vmware.com Jeongkeun Lee jklee@hp.com Jian Qiu swordqiu@gmail.com Joan Cirer joan@ev0.net John Darrington john@darrington.wattle.id.au John Galgay john@galgay.net John Hurley john.hurley@netronome.com John Reumann nofutznetworks@gmail.com Karthik Sundaravel ksundara@redhat.com Kashyap Thimmaraju kashyap.thimmaraju@sec.t-labs.tu-berlin.de Keith Holleman hollemanietf@gmail.com Kevin Lin kevinlin@berkeley.edu K 華 k940545@hotmail.com Kevin Mancuso kevin.mancuso@rackspace.com Kiran Shanbhog kiran@vmware.com Kirill Kabardin Kirkland Spector kspector@salesforce.com Koichi Yagishita yagishita.koichi@jrc.co.jp Konstantin Khorenko khorenko@openvz.org Kris zhang zhang.kris@gmail.com Krishna Miriyala krishna@nicira.com Krishna Mohan Elluru elluru.kri.mohan@hpe.com László Sürü laszlo.suru@ericsson.com Len Gao leng@vmware.com Logan Rosen logatronico@gmail.com Luca Falavigna dktrkranz@debian.org Luiz Henrique Ozaki luiz.ozaki@gmail.com Manpreet Singh er.manpreet25@gmail.com Marco d'Itri md@Linux.IT Martin Vizvary vizvary@ics.muni.cz Marvin Pascual marvin@pascual.com.ph Maxime Brun m.brun@alphalink.fr Madhu Venugopal mavenugo@gmail.com Michael A. Collins mike.a.collins@ark-net.org Michael Hu mhu@nicira.com Michael J. Smalley michaeljsmalley@gmail.com Michael Mao mmao@nicira.com Michael Shigorin mike@osdn.org.ua Michael Stapelberg stapelberg@debian.org Mihir Gangar gangarm@vmware.com Mike Bursell mike.bursell@citrix.com Mike Kruze mkruze@nicira.com Mike Qing mqing@vmware.com Min Chen ustcer.tonychan@gmail.com Mikael Doverhag mdoverhag@nicira.com Mircea Ulinic ping@mirceaulinic.net Mrinmoy Das mrdas@ixiacom.com Muhammad Shahbaz mshahbaz@cs.princeton.edu Murali R muralirdev@gmail.com Nagi Reddy Jonnala njonnala@Brocade.com Niels van Adrichem N.L.M.vanAdrichem@tudelft.nl Niklas Andersson nandersson@nicira.com Pankaj Thakkar thakkar@nicira.com Pasi Kärkkäinen pasik@iki.fi Patrik Andersson R patrik.r.andersson@ericsson.com Paulo Cravero pcravero@as2594.net Pawan Shukla shuklap@vmware.com Periyasamy Palanisamy periyasamy.palanisamy@ericsson.com Peter Amidon peter@picnicpark.org Peter Balland peter@nicira.com Peter Phaal peter.phaal@inmon.com Prabina Pattnaik Prabina.Pattnaik@nechclst.in Pratap Reddy preddy@nicira.com Ralf Heiringhoff ralf@frosty-geek.net Ram Jothikumar rjothikumar@nicira.com Ramana Reddy gtvrreddy@gmail.com Ray Li rayli1107@gmail.com Richard Theis rtheis@us.ibm.com RishiRaj Maulick rishi.raj2509@gmail.com Rob Sherwood rob.sherwood@bigswitch.com Robert Strickler anomalyst@gmail.com Roger Leigh rleigh@codelibre.net Rogério Vinhal Nunes Roman Sokolkov rsokolkov@gmail.com Ronaldo A. Ferreira ronaldof@CS.Princeton.EDU Ronny L. Bull bullrl@clarkson.edu Sandeep Kumar sandeep.kumar16@tcs.com Sander Eikelenboom linux@eikelenboom.it Saul St. John sstjohn@cs.wisc.edu Scott Hendricks shendricks@nicira.com Sean Brady sbrady@gtfservices.com Sebastian Andrzej Siewior sebastian@breakpoint.cc Sébastien RICCIO sr@swisscenter.com Simon Jouet simon.jouet@gmail.com Spiro Kourtessis spiro@vmware.com Sridhar Samudrala samudrala.sridhar@gmail.com Srini Seetharaman seethara@stanford.edu Sabyasachi Sengupta Sabyasachi.Sengupta@alcatel-lucent.com Salvatore Cambria salvatore.cambria@citrix.com Soner Sevinc sevincs@vmware.com Stepan Andrushko stepanx.andrushko@intel.com Stephen Hemminger shemminger@vyatta.com Stuart Cardall developer@it-offshore.co.uk Suganya Ramachandran suganyar@vmware.com Sundar Nadathur undar.nadathur@intel.com Taekho Nam thnam@smartx.kr Takayuki HAMA t-hama@cb.jp.nec.com Teemu Koponen koponen@nicira.com Thomas Morin thomas.morin@orange.com Timothy Chen tchen@nicira.com Torbjorn Tornkvist kruskakli@gmail.com Tulio Ribeiro tribeiro@lasige.di.fc.ul.pt Tytus Kurek Tytus.Kurek@pega.com Valentin Bud valentin@hackaserver.com Vasiliy Tolstov v.tolstov@selfip.ru Vasu Dasari vdasari@gmail.com Vinllen Chen cvinllen@gmail.com Vishal Swarankar vishal.swarnkar@gmail.com Vjekoslav Brajkovic balkan@cs.washington.edu Voravit T. voravit@kth.se Yeming Zhao zhaoyeming@gmail.com Yi Ba yby.developer@yahoo.com Ying Chen yingchen@vmware.com Yongqiang Liu liuyq7809@gmail.com ZHANG Zhiming zhangzhiming@yunshan.net.cn Zhangguanghui zhang.guanghui@h3c.com Ziyou Wang ziyouw@vmware.com Zoltán Balogh zoltan.balogh@ericsson.com ankur dwivedi ankurengg2003@gmail.com chen zhang 3zhangchen9211@gmail.com james hopper jameshopper@email.com kk yap yapkke@stanford.edu likunyun kunyunli@hotmail.com meishengxin meishengxin@huawei.com neeraj mehta mehtaneeraj07@gmail.com rahim entezari rahim.entezari@gmail.com shivani dommeti shivani.dommeti@gmail.com weizj 34965317@qq.com 俊 赵 zhaojun12@outlook.com 冯全树(Crab) fqs888@126.com 张东亚 fortitude.zhang@gmail.com 胡靖飞 hujingfei914@msn.com 张伟 zhangwqh@126.com 张强 zhangqiang@meizu.com =============================== =============================================== Thanks to all Open vSwitch contributors. If you are not listed above but believe that you should be, please write to dev@openvswitch.org. ovs-2.9.0/CONTRIBUTING.rst000066400000000000000000000024201324262074100147710ustar00rootroot00000000000000.. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. Convention for heading levels in Open vSwitch documentation: ======= Heading 0 (reserved for the title in a document) ------- Heading 1 ~~~~~~~ Heading 2 +++++++ Heading 3 ''''''' Heading 4 Avoid deeper levels because they do not render well. ============================ Contributing to Open vSwitch ============================ As an open source project, we welcome contributions of any kind. These can range from bug reports and code reviews, to signficant code or documentation features. Extensive guidelines are provided in the docs at ``Documentation/internals/contributing``, or `online `__. ovs-2.9.0/COPYING000066400000000000000000000032351324262074100133700ustar00rootroot00000000000000This file is a summary of the licensing of files in this distribution. Some files may be marked specifically with a different license, in which case that license applies to the file in question. Most files are licensed under the Apache License, Version 2.0: Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at: http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. Files under the datapath directory are licensed under the GNU General Public License, version 2. File build-aux/cccl is licensed under the GNU General Public License, version 2. The following files are licensed under the 2-clause BSD license. include/windows/getopt.h lib/getopt_long.c lib/conntrack-tcp.c The following files are licensed under the 3-clause BSD-license include/windows/netinet/icmp6.h include/windows/netinet/ip6.h lib/strsep.c Files under the xenserver directory are licensed on a file-by-file basis. Refer to each file for details. Files lib/sflow*.[ch] are licensed under the terms of either the Sun Industry Standards Source License 1.1, that is available at: http://host-sflow.sourceforge.net/sissl.html or the InMon sFlow License, that is available at: http://www.inmon.com/technology/sflowlicense.txt ovs-2.9.0/Documentation/000077500000000000000000000000001324262074100151435ustar00rootroot00000000000000ovs-2.9.0/Documentation/_static/000077500000000000000000000000001324262074100165715ustar00rootroot00000000000000ovs-2.9.0/Documentation/_static/logo.png000066400000000000000000024407271324262074100202570ustar00rootroot00000000000000PNG  IHDRNtr?gAMA abKGD pHYsaa?itIME  $3M^` IDATxiz_ IDAT<Ǽ IDAT;<<;<<,--,--,--:;;,-,;<<݊ IDAT,-,;<;,--+,,,-,JKK,--;<<,--;<;,--IKJJKK,--;<<,--IKJIKJJKK,--:<;YZZ,--giiXZYIKJXZZU8 IDATgiiXZYiz:;;JKKXZYgiiXZYWYXJKK;<<giiXZY+,,YZZXZY:;;giiXZYax IDAT|yzJKKgiiXZYIJJXZZgih|yzJKKvxxXZYWYX:;;vxw,--Y IDAT|yzWYXfhgvxx^[\:;;uxwJKK^[\+,,:;;^[\+,,uwwgihH9 IDAT^[\,--mjk:;;^[\PLM2./^[\OLMnkl_\]^[\{yymjk:;;9S IDATPLMfhgvxxPLM2./|yziznkl_\]{yyWYX2./mjk{yy- IDATmjk^[\IJJnkl;<<mjk+,,A=>PLMOLM^[\,--_\] IDAT;<<^[\^[\|yz^[\mjk^[\OLM^[\OLM IDAT^[\,--IJJ:;;,-,^[\,--,-,,-,,--WYXOLM:;;,-,;<<;<<fhgvxx{yyIKJ;<<;<<WYXB>?XZYXZZ+,,;<;JKK,--uwwgihgiiXZY m IDATXZYvxx_\]IJJ,--IKJ,--A=>fhgvxxmjkIJJuxwPLMIJJ;<<XZY_\]+,,,--{yy;<<izOLM:;;IJJ{yyWYXXZYvxx+,,vxw;<<uwwgih2./WYXIJJuxw6 IDATnklgiiXZY,--PMNWYXPMN;<<|yzfhgXZYgihfhgmjkIJJ,--uwwfhg;<<mjkfhgvxxvxxWYXmjkgihWYXIJJvxxIJJvxx:;;+,,B>? IDATvxxXZY|zz^[\IJJPLMXZYvxx|yzXZYXZY{yyIKJvxxIJJvxx+,,,--;<<JKKXZY:;;|yz:;;,--+,,|yzIKJgihfhg;<<WYX p IDATWYX;<<;<<WYXIJJvxxPMNuwwgih_\]2./;<<{yzgihvxxgiifhg;<<JKKXZYgih+,,IJJvxx,--T IDAT;<<,--vxx,--;<<fhgizWYXXZYvxxIJJ+,,- IDAT+,,;<<,--vxxuww;<<XZYfhg,--WYXgih[vxx:;;c2 IDAT|yz+,,?m:jXZY,--Kwյ,--WYXIKJ|zz{yyvxxlkgii9 IDAT+,,{yyXZYֶTWYXXZY,--JKKJKK,--3/0{ƛϪ G}2&B>?_\]B>?;<<PMNgiigihXZYKwյWYX;<<+,,:;;;<<,--OLMvxxl{ƛgiiYZZB IDATXZYIKJ_\]{yyXZYϪ[WYXXZYJKKJKKB>?{yyvxx?mgiigiigii,--B>?{yyXZY:jWYXXZYXZYIKJB>?^[\izvxxKwյgiigii IDATgiigiinklA=>XZYlkfhgXZYXZYXZYB>?^[\vxxֶ:jXZZgiigiivxx|zz2./IJJ.`+,,XZYXZYvxxPMNOLM,--Kw9dc+K;<;XZZ;<< Z IDATgiiXZY|zzIKJJKK;<<nkl{ƛϪr1V:e_\],--2./;<<_\]_\]XZZgihϪ[;<<XZYXZZ;<<:;;$ !mjkIKJ?m;<<vxxgih_\]A=>;<<:j;<<XZYXZYvxxgH IDAT|zz2./_\]{ƛϪ@pc+K$ !2./B>?WYXgii;<<lk+,,XZYXZYgihWYXB>?{yy;<<ֶ:j;<;vxxvxx:;;B>?{yy:<;.`JKKXZYXZY+,,!@ IDATB>?+,,Kw,--XZZvxx2./;<; G}!IKJWYXXZY,--2./;<<1%NB2giivxx,--+,,OLMA1T2&,-,WYXXZY,--+,,PMN< IDATiz;<<b*JOgiiXZZWYX^[\{ƛ9dAq,--WYXWYXfhgPLM:;;JvϪ G|s2Wgiigiivxx^[\?m NS$?,--WYXWYXXZY+,,PLM'A3 IDAT,-,Kw1%[2&giigii,--fhg|zzΩ\b*JV:;;WYXvxx+,,|zz{yy;<<k|ǜ9dAqgihgii,--fhg{yy?md+LC3ֶ @pS$?;<<;<<:;;vxxWYXvxxJKK{yz IDAT;<<Kw!! G}2&gihXZYgii,--OLMյ;k1%[2&;<<:;;WYXXZYfhgOLM{ƛlb*JVgihJKKvxxfhg2./JvϪ9dAq;<<:;;:;;;<<vxxWYXOLM?m/H IDAT G|s2WgihgihWYXKw NS$?;<<+,,:;;;<<fhgΩ\1%[2&YZZgihvxxfhgk|ǜb*JVXZY:;;;<<vxx+,,{yz?mֶ: IDAT9dAq,--giigihXZYWYXmjkizT G|s2WXZY:;;;<<;<<fhgյl{ƛ"c+K G}2&gii|zzvxxgihvxxvxx;<;,--{ƛϪϪA1c+K1%TXZYgih+,,;<<XZYXZYJvϪb*JVM% IDATgiiYZZ,--fhg,-,?m9dAqXZYXZYvxx:;;KwG|s2WIKJgiiJKKWYXΩ\s2WS$?;<<vxxXZYfhg,-,k|ǜ>73 IDATXZYgiivxxfhg;<<?mֶvxxXZYvxxfhgT+,,XZYgiivxxfhg:;;|ǜvxxXZYvxxIJJOb*J,-,] IDATXZYgiiAq@pvxxXZY;<<fhg,--c+KN XZYIKJvxxB2N1%;<<XZZvxx;<<+,,;<<2&TA1WYXﶃ IDATXZYWYX+,,Ob*J:;;giivxxWYXiz;<;Aq@p,-,WYXXZYIJJ,--c+KN JKKgiivxx+,,JKKB2N1%,--WYX IDATXZY+,,2&TA1;<<giivxxuxw;<;,--Okֶk,--WYXXZYJKK,--Aq@p;<;YZZvxxJKKc+KN ,--,--:;;IKJ_- IDATfhg+,,B2N1%;<<gihgiiJKK;<;2&TA1,--;<<:;;WYX,--JKKOb*JIKJgihgiiIKJ,--Aq@p:;;;<<:;;WYXp IDAT^[\$ !PMNJKKIKJ;<<c+K![JKKgihgii,--XZZ<B2NlXZY;<<:;;WYX:;;+,,XZY2&TOyWYXgihgiiYZZ|zzXZYOkֶ;<<;<<WYXJ IDAT,-,XZY|zzvxxAq2&lJKKgii|zzgiiIJJJKK,--c+K![XZYXZY:;;izIKJ{yzB2N1%^[\JKKgiigih$ !_\]JKK^[\2&TA1mjk,--XZY:;;;<<,-,P IDATXZYOr1V{yygiigiigihvxx:eXZYXZY:;;;<<;<<XZY|zzgiigiigihvxx^[\XZYXZY:;;;<<,--De IDATXZYmjk^[\giiIKJgihvxxmjkOLMXZYvxx:;;;<<WYX{yyXZYXZYgihgii{yyvxxvxx;<<,-,+,,WYXp[ IDAT{yymjkXZYXZYgii;<<OLMOLMvxxvxxXZY:;;,-,gii2./^[\XZYgii,-,gih:;;^[\_\]IJJ;<<vxxXZY;<<;<<vxw|zz IDAT{yy|zzuxwIKJgii;<<;<<,--fhgmjkOLMXZYgiiXZY:;;|yzIKJvxxOLMA=>+,,vxxWYXgii,-,;<<izuxwPLM^[\vxwgiiXZY;<<JKKJKK:;;nkl^[\4k# IDATA=>|zzWYX;<<WYXgii;<<^[\mjkIJJ_\]mjk^[\_\]giiXZYXZYmjk^[\;<<B>?{yy{yy_\]WYXXZYXZZ^[\{yy:<;XZYvxx{yyuwwgihB>?^[\+,,{yy+,,vxx+,,2./2./bY IDATIJJgihWYXXZYXZY2./OLMJKK2./giiXZY_\]A=>uwwgihOLM,--|zz$ !mjkmjk$ !|zz;<<:;;XZYIKJ^[\2./,--+,,JKKIKJXZY_\]B>?WYXgii|yzfhgvxx^[\IKJJKK,--WYX<@ IDAT^[\fhgvxxmjkWYXYZZ,--OLM:<;;<<;<<+,,_\]_\]:<;;<<JKKXZY2./,--+,,^[\fhgvxxA=>:;;,-,,-,,--^[\+,,_\]nkl;<<,--d` IDAT;<<:;;WYXJKK:;;:;;;<<,--,--XZYgiigih,--,--JKKWYXXZYWYX;<<XZY,--giigiivxx+,,giiizXZYJKKrpo IDATXZYWYXvxwfhgXZYXZYgiigii;<<fhgvxxXZY,--XZYWYXvxxvxx,--+,,,--giigiiuxwfhgXZY,--+,,:;;XZYWYX IDATJKKIJJWYXvxx:;;WYXgiiYZZ+,,+,,WYXfhgXZY:;;,--+,,vxxXZYgih{yz:;;2./:;;y IDAT^[\JKK,--fhgfhgXZYgih,--JKK|yz;<<2./:;;|zzWYXJKK:;;IJJ;<<XZYgihXZYWYXgiigih2./:;;O˅1 IDATfhgvxxvxx:;;WYXgihXZYgihgihfhgvxx|yz+,,vxw;<<2./:;;mjk+,,;<<fhguww;<<IKJXZY;<<:;;WYXfhgvxxB>?H( IDATuwwgihiz:;;:;;vxxWYXgiivxxgihWYXJKK;<<B>?mjk+,,;<<WYXWYX;<<WYXgiifhgfhgXZYXZZB>?XZYvxx@< IDATXZYvxx{yyXZYXZYvxxIKJ;<<JKKB>?XZYvxxIJJ;<<XZY{yyXZYXZYvxx:;;JKKXZY_\]|yzXZYvxx Q IDAT:<;--.:<;^[\uwwgihIKJ,--JKK:<;--.:<;_\]IJJ,--IKJ:;;--.:<;^[\fhgvxxIKJ,--JKK:;;;<<_\]:;;,-,IKJX IDAT,-,,-,;<<^[\fhgvxx;<<;<<mjkJKKvv IDATiz2 IDATs4 IDAT1ѩ IDAT)# IDATiz( IDATXZY$ !:;;]y IDAT;<<:;;--.:<;:<;JKK;<< IDAT+,,JKKIKJgiiXZYJKK,--$ !:;;WYXgii{yz+,,,--:;;$ !:;;WYX|zzJKK+,,$ !:;;PLM2./|zznkly IDAT^[\mjk|zz^[\,-,;<<A=>,--|zz:<;;<<,-,,--izIKJJKK,--JKK^[\,--+,,gihJKKJKKmjk;<<|zz:;;vxw,--;<<nklfhgvxxmjkIJJOLM,--,--<~ IDAT$ !:;;_\];<<,--{yyXZYA=>;<<mjkXZY$ !:;;$ !:;;mjkfhg;<<JKKWYX^[\IJJXZYfhg;<<,--uww:;;:;;WYXXZYOLMXZY|zz+,,_\],--$ !:;;$ !:;;XZYXZY+,,JKKk; IDAT,--JKK;<;|yz,--;<<fhgWYXXZYvxxIJJ,-,[ IDATgiiXZY;<<XZYvxxIJJgiiXZY;<<|zzXZYvxxXZY:;;A=> IDATIKJ:;;JKK+,,,-,;<<JKK+,,;<<XZY+,,;<;JKK,--vxw^[\_\]:;;JKKXZY,--JKK:;;,-,,--JKK+,,;<;JKK,--2./XZY_\]fhgvxxXZYvxxIJJuxw$ !:;;|yz:;;vxw,--^[\B>?fhgvxx|zzXZYXZZA=>+,,IJJ^[\_\]fhgvxx^[\;<<IKJgii$ !:;;|yz:;;vxw,--XZYIKJPMNA=>A=>B>?_\]PMNgihvxxXZYB>?^[\giiXZY^[\_\]B>?_\]PMNiz+,,mjknkl{yy{yynklOLM^[\|zz;<<:;;IKJXZY|zz2./XZYvxx_\]^[\|zzt IDAT,--^[\{yy{yyOLM{yy|zzgihgii,--WYX2./giiXZY^[\{yy|zz,-,{yy:;;:;;+,,^[\;<<:;;WYX:;;2./XZYvxxA=>IKJJKK,--;<<;<<XZYJKK,--JKKWYXJKK;<<giiXZYgii;<<;<<nklgii|zzgiiXZYXZYOLMIKJ,--WYXJKK;<<giifhgvxxfhgvxx_\]fhgvxxA=>IJJmjk$ !+,,{yyuwwgihA=>WYXA=>WYX+,,A=>fhgvxxB>?;<<_\]fhgvxx_\]+,,vxw;<<{yy;<<$ !+,,{yyuwwgihvxxs IDATXZY+,,WYX;<<+,,IJJgiigii,--uwwOLMgiiXZY,--;<<;<<;<<;<<;<<;<<;<<;<<;<<fhg;<<gihIJJIJJ,--+,,IJJvxwXZYXZYWYXgih:;;OLMXZYvxxvxxfhgXZYfhg,--,--fhg+,,;<<;<<giigiiXZYXZYXZYfhgXZYvxx+,,;<<;<<,--,-,vxx,--fhgXZYXZZJKK,--XZYWYX;<<fhgvxxvxx:;;vxxXZYJKK:<;giiXZZXZZ\웣 IDAT:;;;<;giigiivxxXZYXZY;<;;<<XZYIJJ+,,,-,,--XZY:;;;<<vxxvxxWYX,--+,,JKK:;;;<<,--JKK^[\;<<JKKgii|zzXZYgih,--fhgJKKXZY;<<|zz,--XZY;<<JKK;<;,-,,--;<;XZYJKK:;;;<<vxxJKKvxx,--+,,XZYXZZJKK,-,,--JKK,-,,--k IDATXZYgihIKJXZYYZZJKKWYX+,,JKK,--vxx:;;;<<;<<vxxXZY;<<+,,giiIKJXZZXZYgihXZYXZYYZZWYXgiiXZYIKJuww,-,vxx:;;;<<IKJvxxvxxvxxXZYJKKgiiiz:;;vxx{yyWYX;<<9\ IDATXZYgih,-,XZYvxxWYXIKJJKKWYXgihgii,-,vxx:;;;<<PMN|zz,--vxxgiivxxgii,--;<<WYX|zzfhg,--XZYXZY:;;XZYmjk:;;WYX_[\WYX^[\mjkvxx,--vxxJKK,-,,--XZZgihgiigii;<<vxx^[\^[\:<;mjk^[\IKJgiiJKK IDATIKJWYX;<<:;;WYXJKKXZYfhg_\]|zzfhg;<<IKJ--._\]{yygiiXZY,--+,,+,,giigiiXZYgiiIKJvxx+,,|zzmjk^[\WYXvxx{yzXZZXZYmjkPLMWYXgiiYZZOLMmjkJKKWYXXZYJKKWYXgiiOLMOLM+,,B>?{yy{yyPMN+,,OLMIJJXZY{yy^[\giiXZYXZYOLMOLMgihgiiXZYgiigiiWYXOLM2./+,,2./{yy{yyB>?+,,^[\{yygihvxx{yy^[\IJJ;<<WYXgiiuxw2./A=>gii` IDATWYXXZZXZYWYXA=>PMN:;;nkl2./mjkmjk$ !nklWYXPLMA=>fhg,--A=>^[\_\]+,,giiXZYJKKIJJ|zzA=>OLMnklIJJXZYgiiWYXXZYgii|zzA=>2./fhgvxxIJJ+,,;<;vxxJKK|zz;<<WYXgiiOLM+,,WYXYZZXZZIJJ;<<|zz:;;vxx,--|yz;<<vxx|zz,--giiXZYOLMgii:;;,--WYXXZYXZYJKKfhgfhg;<<XZY,--WYX,--WYXgii+,,:;;WYXXZYJKK:;;;<<Sf IDATvxxvxxWYXWYXvxxXZY:;;,--fhgfhg,--JKKXZYfhgfhggiiJKK;<<+,,gihJKK:;;fhguww,--IJJvxx:;;:;;vxx;<<:;;XZYvxx:;;WYXvxxWYXgiiYZZ:;;;<<,--fhg:;;,-,JKK,--_\]fhgvxxIJJ;<<XZYIKJJKK,--XZY;<<;<<mjk+,,;<;XZZPLMA=>;<<;<<;<<XZYIJJ;<<XZYXZY;<<;<<;<<_\]fhgvxx,-,;<<,--,--;<<,-,;<<,--;<< IDAT;<<iz_\]fhgvxx<@f IDAT& IDAT IDATR IDATSVi IDATq"IDATmVrkIENDB`ovs-2.9.0/Documentation/_static/overview.png000066400000000000000000006145071324262074100211620ustar00rootroot00000000000000PNG  IHDRRUt9IDATxpօf63333=33; ff3ef;fz)ww^mv$y,\Sծ$O9{1YB!B!$/B!y!B !B !BH^!BB!BB!B!B!B!B!$/B!y!B!y!B !寿pGtǿ B_K=-$/49 Ӆm:|I0]Ba^8"a|lJ# !y!H_X>aZa5kL0?۩S6!\ÕyN4i [wIS:cNZg^D!ovÆ +WQGuaqAe!xYrq'/_cĉ}/xjk㺈B T.p RF~m4| lq+ҥK/_t@-[ !-C=tG3D|KM)RV!Cr-HfΜ9[dV̽<3<78|p* ^mq#}Qw"$/L[݇ͥM0s w {ݺuSD|{h > ^h^TgM:\B ͚531i<ߢ^zO?4 t"F4BA7ԃիWuM߳`UH^IwP~ -*Bp]~ay#ľR)~ Dv{/w0RQ<9+xU !,˵ ݻwRJ+WdO!r ޼Y#1˝!1YzuŊqC'o|kEL$/ȵ @8@޼y^9w֭;wRԵdL^MS/}"0Sn~D飢 !r0PoM7݄fh (\ 8_p U謳Z`$2E{1gP^B 4n۶ɓ`.zރ@[ŋߵk%WH^/ P?[!3/9hI1Lna8O>>3[ /8#G:?݀BB܁mVhQb J 90D &'[nٲ|F,aec.4OJq?EX+EH^{.M0!FRi|ȣ95kYw·~KOFzPˌ2)y!D.oڵkro&$I"f3͂m4M+!> tDaÆN8( رc;_xn?\Ĵ}RSsaŒ56o~ΝvfT\> !{ q8S׮]f`a5.]xeaѣg̘r bǏ1n1 ySIPL3Jaϫ& `8(ӧ#DN !r,"P &AKhK*y桦/(mYZGhעE ?I{z5 Jxhq" ` ? 0) !y#49{.]Hs”! ݇9j&8+ COnvj0N䫮 ~j 0 a5lYF !r}WnEs]Pj9_~_|Au_ !brQf((PuT"WCuv& .3զJ^ć$̏>L)*=Fڳg5L7y:Ofa>E{\qA #I41-2(;{])@A//?Cg_}~BBd ^bE2e`t0a9U!"b{,_L`I68i(na}YgBU /d˗/_#X !f2{11>#,P_^C^}xL[@n̙3g͚BмswΝ;mڴg}[16QF0 SWa9>i;N@ɨlRoɟ R u>6`٥-BTh1N8' fSNS]j1pj<,д`P#u 7kJ зo_kr^l'l"SuŅ^ROFKB&`O?]BH^_ ,N˖-iAx_O?t~oܙFv>.O>$8y;$e.A+!#,vװ`F͸wG8ɘ|=ѡ3QD?"B@Uv]6m7b7Pа2L|p_"׸Bfo>T҄3,@|0Lkt 5kd&/n݊]FDEH^Y[k8!0M,hLC <]ӻ5"ǡzHrC\/^3A FhJסjR {֙ Bjҷ18#ŋ߼ys !y!4N7oΏJnҽ/I9 +&bIp?"-qlޚh4F};b1ɷγrc !|S5>!-I))+VJq0: 8H.Pn Snm֭vHeo.]ԄfP{^1򂯃J^H^fa.ڝߥýC LZofl :[PKK*5\yz>ʕ+3^8yJ/&$/R5y2r ˆa.'A)c$u0a!,le`-\)fćAСCGlBBÖYůil2DO>dӍz/ăikӦ qt'"RLׯUqhDM#IR)yﾛfaQ T^<ܤӧOwߞH@ -GW^ !8/Dݻ1<bz]0g7udc%E8l,kpWSuxfذai^ĮS 7ɶmm3䅐Q ߟSu#7qh20x/0"13A)Мm۶'ND"uvԛps9l:(a_N:{뭷PBYBH^/}^e4"353fZgdF!Q,0)=`Qڒ"I'Uɝ֬Yqe0 CsN^ !A&MTJcp BȠHNG;rFwuڵYi NbSt&^pѧOİ@"R;E;  F@hC3r!Xr ݱcwFFx~ر"&{6&ɋH^؉ʓy8\8!mN00Gel P,Ό(hV_'n6ydfRjKnRСC1dxH^DEBXaϟߘ]c%KOdޏfF4xO/nbệ+VpoM :u`;N>K=y7g$! /\;UH^eZS7.be$ț7/?b6>^z 蝾:Çgݬ[8kHvOy ɀ%3"ёvvkiРA8[Kۇ44ظq#_3Ejg+ M,Yr뭷/l Ra7Tx\P\ P^k .y!y!D`|`/SL :nZP*Þ!m+>c(WÞwbpë!'&|@I"E p٣n"+aKo|rB`U?GQn!Dy*ΰsvڵ<3,([XKBUa2=_|EiX_>< "mM , t6İqXO!D|lxgi * Rpу! ?  n1'ުx(wyJ%/ľBKz.[!8[ wwٸ-Ayˏ^~7]tq|} /*(~{'3=BHa 6l۳'x"n"XD)'O8=nݺ1ɋX y! 1t'YZXq] ' <QW_};/N*W$/ľB0, xyE(]8[p .NzW*xt ]>`{(AE}AP\9t]\1BBH^8s?QdSJLA},ݧ-Kk׆AѢE?SJ^ !#vFXPq_AhEmkGj^DQ0 }'/J^ !a=(gmw5B |8x`]yMuVwBqƙh.CI s D]Tn"/_>Ԙ #myIwJ^ ZJZ)u I3B$On:uJsa7i*/T $/#8-F$k|4!D^[3QL -[Dz[Gܧj$/h,Xlذ!T!(50.{ ᎅKDkz]^ Sދ!y!jԩSB^4!$/ZP03aM2#i `"vsF /;$olwPwW4R@!]m-֤I?uYM5KPp$FH^ DF,Y0` #B"jS<|5PEDBH^p>*\d>cV^mŋ#A8FB" y!$/\dseNaÆ\H^)Pb+P?"yېӘࡅm۶0y$^sm!?,N)[q`GfEABj"er˗5k3҅䅺ؽ{w_pqUW!Aɋ܉䅐ꫯϟ>Vs$,H[T.-p1bݻщ_\Fl't͊}EBp_yQܹ36~!y!$/D5yq'LtLsZ"X`1Gʝ?+QTn~upLdCITfO24<_0TH[) oۘ<$^Y"w"y!$/6nȔ  ~߿SRo X~k5luxدC8WH} L4)TYf_|nے !BD `La׮]# ,Ѻر'B6ogpr}/7;l_:d&^w /y5klڴ#ؿ|Lo o'Oo+6m#ܶ%H^ He!_ s[vayEum}-{Z[2 m( CjFSL`W*D{o8wx}_Yxf>{>~g0) 'i{*++{; 3˄` 0(DVK9k֬V¥whчIGhD-̥w}/śoA~)^:uJŋox8-lQQ:D*j%vx<c w0s?qX&ny)RD` LƐxՄO6N0.ݹs2g̷ IpxXE/z=3Nl|3wSUPďv1xfeQo1 8^U?1:ll]y%K%: }Q"J^&I PV/8^?x q6WS`"Kjb) ^+#Ԝ l 0V4Ċ1b}!Vo'Y 1:Po"ε5ɍ̖Ps⏓|q?Zjc W83owishfcpaxU_<39[";Cף39۲yfs*cin BXǀzW(t3ꋦ׹g- /1; I?X垝Q4; G&Z̥ v<ĘD2 /VSdVl^2^.fjxcVn@H?OkRb&թZb `GB*9| ./p+Vo OӰ{}4A+,= O4h !NK0{x;ͻSDOP3I1oO1oM%I⽐ͮDYmd*UܹP|kmڜ+X/:Vxམʲ.ҝahY_0䞕嚳 6˩1Epsqzpqs2iǃűW 2ethL}<}la.C) L# c^ J_I^X}@ gu}GTբ[?lцdK=i6ּ}@tGg)I~1VH=:{bkCba6kTV4So ? /pB#~CQ{O媽W;'HqߙdO"Z M{l>A:]*W,yO)GڔWG%[H 6piLUTIr&I.[4®^ܧ^,zxÁkhzlef^yiC>x^+Ms3\ѼAS4 $B%,i(\Xřw\HBQ""ZQګ{y+Kko@ɼbmOGWڕw[V,ę1FD0jS>>7E؆ R K|BtBkL&۫Y|D:J!r ]R[d[8*Ѵ0VFj<.\8^$ /f U2x a&56YҦvBWPPL*^~3;uV?L3fhmCWY30xdvX8̺^D1C:YZSaELϋ40taHW%"XE9Dv5]~_yrnG=*Z_͙k3vL6@΍_GKbmMqr3I) 5_^'L?D&M[yEN69\ ҏ+lI %VI 0^Ù q/p½ / ?SIL{F~]gM,؀| nkedcRbP龦񤙫t4$׿vn|_ݾg hCދ8XuQj܂LgLz͟6+=@^T8>wt4Vg/ʍ!ep`2tAܟW`dyH֖ t[J7+}tk۞7LxiiN%+ q7J͈pȯ93?l5߸KCWF.7VPP8 B-{"86{wb]gЈTP1l/8^prwϽㆃ#$ O\!|b^D9Y9zI!B4Eo=pkh&7Yf*Չ "Iٞ.jLkkRqwH" (pOI/L^_йc bL/~}ܚK?kpɥܬ%վԣ (7*$Ǔ!2?ُ}}]}}J]i&gPս S*Uk;bw-ރ4QyrRyi"U2}@20x{`V5֦KV]bN ,2‚g. <8.wGYz,؄t݇:OǧY&$DQ.Ȉ~bMm=tZ~poqF\%Tm[v ov3'/.\b|E[ŋL{Fz̢)ESH/Ԥ;u?4CI&TGWگYk^` z~e^9wcw1NsF_̹J8#ͧ/^h/<#xJ]mEzu&V$❶&[YR%ϑ.W|/KVTo,G!RZ80GE]xH[ $(\HDOx [)+ ֘7xN Px}{+k#&Wl +BNj Ap$m *ϸKSͻ46d𕿼:B_qs8²C+zW"C EGET/P"CUMsvX0o]Ck5Sc/o] 䆧/{UEWVwudsTӖ;4/d9ۅڱse,TqBgoYLA}+'[*0M78zѬSPVq |t*-IcWX^MV9_Di竕8w_)( v w(> ydp[O:noK;sRR1&7zUs}ʵ!/tm"r!oFսkX~l|pe? _6]dӄr\-IO'I/ E^&-*\Q.Do(f[uPV^'M$xA#]a(}>V5^p 8`{1&— bP9E9t(ܸ4_\dޞ,LHw~A[w8U4O^`7M\k}q]xHAH%%q"F -}A.fxm`Y΅{EF.瘞}$>b9KX.Xrb]d+Hu%Z6ˍRA髧z1ZFK"麮dI/~ -[Ru:^ermtTfƒ"/^xeK 5f{b%RQnNV] {wX.u߀Ct*}Q[QkOQd7-_"_zDޚ,m=txHBX8LK-1 xU;B DFp \Hw'_κVO[W=v":<, }ڹ|ݴdϟj]y\/l!3>}]̋ wAp[vWhս*QQ-5szw\PwbNSRmH鹪}\~SNwh$wepz M(IPW" \Q>lfs5 9u #8/Βוęk^!8H"}l/JQt!w %) (6E^DQ+Nl1ͺ@dx,[Dup. Rἧž!,'Y 7:cV.R\?ËLy4CWTznuj;T ގ7+N_'o#d{, &_/>8xg =*IbekA"ΗXJgy3kx!rʉ+4vcb k%kȥ(׳ֲKhs _:ZKsF՝d?Nl  іO.=b~k*Qb2kɁiΚèɘ"U>  P~}mi[ݧKQ'+GFK ( -tGdSSW6ޫ.% \9|.F*w{¹\˞yqܴ5deLO} L.Y6Gg/=ukx/0՞?5Bu`xldk\yZ;lhcޱ\x#%ۋda+/t֏pfhG{V%WKoV-Қ+w~ZPxIx ./Tg@YnG#R 5P%j)!ozTTًXkhv|pQ!o^= sN3eT,CͮEP%*'Nj'nud6g'[~)^Z4" 쩾pzF~A@lhQy\[*M];n[ ^t46hysO^qS@kդˎpxM2N\mcu/lyS /hmS,9tU菱vR+%I{.LWVɯJe їnogIVxyP W-X,Xzt?c ./p`X TR}r6t>Zڇ +D[ee'\'za!0p;UT>]$$YY+w|G0ѻU In5V'WIW,*1Wb5Ѣw6dPo*maʌ=Xj#>Dŀl/*k"=%kVZ.nK B78|r -*RE hnha 6cHT|z5uHiW䎵𻦤;Vq^mzW etwF;vWs7/>Rl f9f8u'{*_^`2vjv="Q`I<9xI6M M޻zr0eum{5?6-[ *\%a~pPU(8 ./ uR `bBN-l<p4qʍ%+7J|}>~ a!tm릺ޅ=IV=#0Vd?j۝jܖ&~JI1lku /в1 U,dKgˏBNstSaasDF EI41?Vt S1Ẑ$i+ސa)E\uDW.N6Y O6JŅ*||Ӟ'6Nѿ@nH$?Ql_㱼a|{si-{ދ&@ i'Rpt6_UP8Ig ./p2p/FZ+p %a)x)eu lJ cHĕ+Ҁ=RdNk!eţER;w6/B-1Lf QcoYPrcT퇫/b:#Y\99[p2c3{u@o¹x5g]2к-9st^;茔 pdX,נ z&m\$Rb 4^ *E9^px 9jx, lPMAJmP0UH V?^H4"v,nc/ ZĤojoFD&]' 60CўQ=MC1 +B 3p:* = '9\n, (o}165մyp641EP{N^=L>#XZqEυe%dG dRry4DY: psp72^=G?^pxa ( (^BH{c[Ey0A52]j#Jd# ^nÊv6>UF.r/Ӌht1ePtX.WS/^nTD(lFe>⥨ lE8Ncj' ./p:i tLVO1*`H{"JCaeI fز44>M s⅋DFba0P-P2*!p.:X 4R;&T_(cہEuW I0@'eU#bWxlg)/:#3X?<8`J'j 3`"#ी[a5.C'P^a-ު-n((m(aDd *#ځLJ1~ ١OU4}p؊$6q#8Kr4^eh/8^prx/ ̴^$/xcf7hbBaB(pL# Hp ؏U1cxno /x#]_@^= /x+AUs) 1b#3Gl~2P#lL ./Qxzi#1-fH^0"PgpO60*Q3(Ǩ ŋ@J2[02)Up팗F{/T/S`Ë''3Un/ z/zƁ$cGv0&XI/pWj`H]a Mf0F"^2Ō '^ ,}kp4/htcTΙ^XT A94Ob ̽*KQN׀f PʑȺ` .\ЌzqJF)aC3 c)'p%p%?`/8bWr. 7e^!^ |&LsËD5fch|]#}I ^`CTEFvU N$`U`o4h6ɞx̌+ ./ËgW\5X!~];d>bc"t{yĊxA̿Z碅z8T  MHWGBR\}":;>BPǀ1=@G 5% ag hE8sW -I(kd~*}jiL ?xK"+'gHK!_yA&4ƋXLAGQQx {D \8^px!s駞xb2N zn]'VDsܪ J`~oT0 L*hhQggH/=QV5B<YPn+Yy2B&M$%O7+I&XZ/oS)I j#^ؚR H5O=A+TcTB '8;Vф-NQcd܄B0CCAH 0]]]r7Uz) .9+Ҥ"Ӫ{۩B$pPB 8`dx~ Hkr9:m.WU#(FVEe${9kx̙W⋳9x ./ËsXא{#KoΗvrObvh\4]in$,V(imLz7<2L8q_; lҰbib@,m~y3| ιF!jRK;xzbB(?aj1p惬y\eKE?C4 ߝLD2/`PMxK_<&yͦua] OA8~`Rd.;'/_)1f8`o3N[٘x)aMn\g#+C&OO'O aN/8^pxc,8-2$ev? 33 03L1dƉQ`]m[؞Yժ4cS#8khn-ͯ ڐ:Wxz^aWL|FǢgJ 7(yb{(/8smَA"LPhaϟOK܉tp:u;~T)+]٭w}ݻ;ѣn۶05FۘʐJz{v~=wB*ɢ={ճ9S^؛SJKI{=_0TXԿoniapA))塲={է{t/&wWo޼$x3Ny2Aо*۫sw P/)..zY=n"p=#AԪ<j4~V09= /^N@IXD"V~s,>bLY!e4hXYiYIiIY)!d0οA%|{.+܉2w?,L͑2:8 򲁥%ϔ ,))5zWڳgWsT"&`&h^ y…'>|py3Z>l\Z^^>x0%Jm2t`8)-H ?g ;X22ɓ'O0aر`!7G=qD@8"GdapL8k nܸ),`,SYc?ΊPp-*Nb:dTnIcǼo9{=eÆN4X4sy+V ab^xz/'^dM#BөG_qTv k֬ZVU\SeU*W_`O|StQ'*r[K͵et1E˳|劋.XcO;Kˠ /Zt ],)<'^|f^xzIceCﯭW[_ǚǺ QB~6Ib^X|̙3Gw3Ψ$wAΩ=:7j71.B֋)F?T_wp]uUɂ*Iʡ4#go/1e淼-]Of7mu55UH4ԴHA(wСjY"tUihj-Jv{5ֵ瞔ZG k \a32~"_S[n̉R]BB:\9"MКiEo8 ‹^7GD,2^)hnl@&'R͉x9;xg%c5CS=;b\p!5Aַڱ4!-[FKZҩ$~4〬[3kZU^ HUI!cGmM D)h#tR-Pri;dSK3; }aQg,4tcs´v U U&F A[U6NB@GJG+GjKZxUdMvg\4` kWQګ5c܃Js uzHÁdR酧^xr*$k'[?2q\rndx[8j< 8oD1jt,E<9t,L`qq$U <#I Q&*D3(]kpRo{6;;ZFKbr`Ԋ9)[ݼiiuT?,[{nthN?ޏLrmν=PD=7v O/<‹@/fsbx\+ )vo_XpɠTs,XkpFC11@J*;z4N>|zP~C$cJ!lz 2t"D5հYt]UtUݚPK-/ӫEgY|տtahs"!'^xz /^!S0j(b=dP`JuXG2 QrV}׽{w Cw:c\DfY 2ځtP5L/J ۶l? U , 3S\ u] I|[ֶ*A fW..1mgլ;vmuwf#kr4xz酧^dza~^fU?PrZӸ<|hO^ܵ 08ɧbű٦͐FB]a;L/$֚V V'@/.E˖,e-F/LBw͜sЋ<9+Guhs^>v.ZxӖ{kkX9fmܶ͡#~ȉ^@^P;_~~S P^xy /^!h(ku\& o:l2jmD$4GcO4&.frf#2Uv{%͛G5 D5-p6^"2薅v1Աs( N zF2vmp Ћ/dtBU *"R˕}xGÇYg|#1녧^<?z#8us/"UPC0w]hȹA"eas[LqCRG>7+|LjCِlP.fc3@D}"Zrcɉ !Ghũ< KC->-"GM|Ȑ!fbO ?9 /^Z5}Nb1 pEz_@0'),}dWa" !`-׾\4 N5"e.E,fSx: oxäI8|ncT!zc*nYJnr%/r ea#/֨zzO/xzώ{ҜĆ{Lмy*yX3 0\j!%2ɶi$B\bz oڋ|;겡CΡd-V0bakK, *oQ$*Xs nDi\@,U` X,"$iT Fé)18KtjakA :(L%LM d|m̘1|fȑ|r!`>SO=‚Ӈz|~a׮M4on׿'ag>BؘJ1ykQ<!`& DT 1;GEgh>^F2; !UY$! @8!" Y| uȁm H ('A1g/ ui<|9#kwƸVL>=3 O< FK(~#U`@-rwznsuf >~oM ~_g?/~! ]4w]ͷꪫJ<ʤY܈!v85*ӯ?O 駟DvD˃`p'/lërW3eY'reЪc-Оc,;ZfQ$6A\Ӌ^^ht'71"eeec:.rK3pJ{w3To7 C~`Kӟ !6<\sp"GxS3~{à ^]{>YXAoUY2 6̶}Z+?* \C5.f$p_/^RJ_?3, 1:Iр B9@åR?Uުկe(DQvFd!Gh[RlF)PSThrTU(]v/wn;w,5THlKD%$@uMY#zaD܎M0r`rUF7~p}O" B應bʕLw,jﯽZX f@k0K =]„`" iU.vI-;Vf`uaa G8)yVX(vt Uoɚ3fnMe*L]fO/ + |kv6ܕ0Ń[,W5X/\Z7@ oFL kcl fݰa`>!,`*̀@^pTY3|q.'UD c0b1CHLA(O<$P]OrLjmؐhn\t$@ D8<sm '%fy8 Ks:W!5N)'@K.4VUUAl}F2Xxl }{ =ɑ"Q;*V "8ĥxdDԎw3JL,㡡pBI4K.ĢxeGgoh"jsgas6bQCxzUK/mі=z`Eqq1@5`g`exoy!])0|3 ->`FK)}NM0"cgg8JH{ ~0OB GHGW>JI￟k044SM l:Ԕb3‚*'U<)`ψQB4V* p]%hLLn9 Gp##?IZgX-@(,$$ᅠ-; :-CEQEѱ~Tx GD* 0CKqA .e(3Nʔ"]AY0$LqAxN 5f%PN:rBI?XjMItи;"uҤ 0tEcȎǀt=aପ%3]29B4^< _N32"^{xc` rCЂVjLCp P|eM!Ҕ?[#<%69Ym.~`*zYg5BH٭MqŖ NSm4T㍟+c.IA#kHLBDbђ\@ 7!טBV\!.ȎP*2 I4#E)\(`@/(9;)x@&AЗ " t=X߿?ؖh[Rt+xhRRk/TvE%.%"߄FF+PZMBDI&!ך0`&= ‹J;Gl]kE@5 `leLd1BvnB,@CP/ -H LF7_mJ4[0l(e/Ȏ@zx 1c>9mUp_dmB[3ŅVLC\" ht2SAH ?]\Z z4%BD4q`T(SY*9?Iv]!|%BDtDDH]hb =v?' '_5WHktWD+ x/ }D-=~!e ڕ_N^JJJI@>NO/xzUK/x M1co /* @L<Bx A%CЈ`:vdƈQRQ,"}6 "h'<IY>\Bh+هZ5%y- e(BT,HQBJBwoD-S]%Zk)(˜ X =f;wf LIGiXY{ns}~< l7%U7#7Xh;գUc_op`Sx胋."5J%0:b_HP lHjM01‰a UIB ]}2fiZW:.z,c=oXD|dFYr}:\I]88;op3sj1Ūve(Pwy͔ l(=NoFes[kd0kx"]^@:)k^x‹IgGk  Aq^WPŊtp{Ι5+k׮GHSDK-M؊pe*>JrPMo.uhiǻ䒴 K#ˇ_̙UY9dԼ׍;nْ[6m>wz*G gS+2j讧^x'K$dF(TahY=ʓδcBeʊe+MwSa~х翮f?9P)$s24O<` @_g%A/PxQ-~B,h1|K¡ 7މT5~WT$^M1yzC<oȗB/M1 pTe>q`˖m ,4Χ-:w*|;p0x"Յ8ck9>3G?޽s:!'-fnnl;Ur 0v$Pǚh Vش$XQ#-^xz /^dZzO822ɦ]{voٶ(Oxh_gĶ"E"-/-w5#q/_7=F_,cQp aezeu{=1Mai} QcsSe_XӿYKvӾu" ;=na]=_77wsJ~~G7k>͹VgN˿~g`jyxha<yF\/c3z$`%U".G3/>cK~8KbϺ+I -*xqS#HX, {:O'<[׋W ՒV/WUEmaml3&ŗ ʊZݦKÏ4qXg<' ̆4WT(qaPC6˼55ZU[ Ail6^Fy "$eq"t:7 oo.\ PDV&)x<ǁQxu$>11AoBW% , <1Z+ yVѺ^0uԍ:QE5e"./qp bH3TnM ,ԘàY60H9d$[1H_q#)+`"d:ٔ{7Hȃ6kÛ& ȦW6kp_6!7gkmʄ_ΆHIeR w|FFnVH ^( p. ^$~%VPG0݈DC'?+ g)l^B 'Xb'w[Vʘ)cP >NdIhn،iٱa$9Mf,,4:-+/u4ЍtX 8<'\bbܬ+P9!1{"}_ CK[ŵ'/c?asO,v.AdlV#{L\&1BLsXTfZaI`F$I$IvIHX!$Ym6J"Rrbaβi35RH4,t֒ҐbF, nq Cgn.sdEv=mi&acͯ>: /Hshʓ#t__g q<+ 'Q' ^(x8y?8*м[7"/ @*pI?L_o\z׫~Rp6~0u%vdDrk`R~;=<9'--ot&nxIx" VaʎYZ@MxsaMLȤ3FA?2_J.H +װt! JnELt-ܔcCQs5 by&]E*1mslz$a . q dCTz%!#L%aOT%>SZ~+x,E%/yLR'AK/UIUBI+e(rgCã>bja6;G2G< ΄ Q$!cb'[?z QiB:&N! *+P.r "#&#H)UE(Ʋ7ӻǂǟO M$n7ىx^H:D@˾3@9G|L/4 b /4rH*gx9x+wNxVK|4 T$#ݩi[d[ܼ4]3~Z* XL et^.)9{ٹrħuuumZq L %)xE}EpԱltn&<=05fWעÓ33ѹuυO_'(A 3ף3 WQThr:|m:2=tʱ~"?) _NOF&~ ߸ NGõ&Ggb4ZޟSq/u ^؂wkH1G8,ʲ)md QlJR} xa[+Fm@v~=v][8r%֤b-FhG`$1b$Dw'Cfa3ϞA<lٚYJ<vw8,Ů^(I %!,/ }{׿## _^ };7|q|ȅq5G/}46zi|މ@ȮO{?_kKq×F/1)[g΍ ^pp?jΌN   @3MߘaS/e̥+:O;BbxAl{@hMf_B $ ˪)^VQp%0J7nmB|4!r&fAՐ_Hp)62U麢&b|D{w/**;g''آ)K?S?؜AFyy9ԂW %)xE//rwޞC;}ݽ]]GÀojܽ:K]n 56F|#ucmuM m퍁@kp/V78!;:l^Ar)춵ڽ #jB@oWl:Mξw_w}yH2C>:go]~9"E(a /N_ |3^$/;${42`{qb/, d%񂔈)fVn&J͘BQn[< 1TIxAlaZTB7u@8p ,ᅂt*EG@0Ǟ={.^C/PBI ^(ubzwd _NL|zjЎͯ5}{TZ͇E!~26׺ve z6E_^$RHw\lC¢H5坳:K3}^X!mkhXW9b4L6XNP'1eʱ>Bn5 %H {xiq$EM $|KW|0W-"1Bg.:طcg9w{ Dnҹ^zhu7;w4vyXx۾)u>[Auuoӎƶ탢@[,L#ƺڎ~sshi}EZ~~ްzsDx \o;{;<kSݸ{[V}?%]eW-D1- /p@x[^@sTABxM@MАx KVd~ȣxb|"dD xG!=C=bGN0(%?hX1Z!j s$^ux{ 卩.‹wy?_ؾǏ??plc=ࢺ|fQHPDl({QP`zg{TT{FĞEA@:7̙} gzֳ]~٫ޣ~j/n9{kNY8Zȋ=z]D|?_-gx q1aQ7*/(; >P9;%<)*$!,41< :8&;dISE}%ݥKV|xdUO_PlhP|XpB8+7(|A.޷4tlmcFCԩQTMr/9[B /V^/ĢS;鍠3h`_xȘ@߿ejt_ 9@?s~ AkvMx1GB&PF2g+9KWH'p /J?\P^S`k5yoC -S /_ /] iٻ7?~gHz6-O|jrgKI&S'O65oRD E5=% na7~Ęד뎥 ,f)a=^E%Kkɓ'O:nI.6veo:YP\=wݱmGϏo %dF =+de)Kŧx{'K4sJCk֬ U9#(ŋ^M۴BAim~ȹs^twIl`G6{R '8&bjee4|x__WnݵWdΞkSQ6BO^9t{݅;3q4I8m̭ǙΞangc9ڂ⭙%[ 3728r$ksW73iuL~y:f&іRz mjo] Pf6-ж]9#m/{o߁'im{@/Z.Ѯ%;mܓ7oꛏ56kbd<ά&j_梚y[.+.ɬؓ|{Z|Bhq&cMꍜS, ;{ailmelxĨiQ}bp$,*mb"x !aa-qB^-G BR<"e@~@(R‚s/F|駟g~ g(FKߐW)au{Od}I:;N-V썛>cE 6Kv? "?|cG}g|<_}Hpƒu9`Sg2.~ ?`kbn>ۧ{nIa8qnDgvM?:p!'L*J>84ԥy+ɅɑKOn=~\p eIX!x$p yuu[Ia}ݴyyb%ϥRA~Rz|p@Y&] / z/MѣIP B{{0Պwy=z:ދm !K55[wKn^-|5r$Oޚsm(ߟ%ZW;~[ooo}6U[ cRm?~wȯpt_Y,[qxWw|?- \Ts(qO;qJ|띷w^R[~kvaaX(^,,x5T)iBn+6>YMM&.浟iR veW C`h NX&  { ѽdTJ ``d(T́E \a+2^ .ǵ aB.. ƴLƘ`8 BւêL7PO&NX7 2Z,xtcBrDyaX#+DPΔK-\ISAH? hF̏ٶG'sx ?IE 4?K)\ @"GU؝X )O NQWW70D [J@(H'_x1,2Be7bXIZ.Dͺ%S3FMLJ~}v]`=iᅖ s{Бʪ{;w7570܋s|ref*I]6~~E@)N3{;LwO[ZPP2VIc7,ݔ5kK 'i[s7'-K_<)gMj,7[ɞ֎v1dtG'di)S;L 5nf9~܄&xzf\``<:HW'(b\0'I,%TB3y "GAvP7)Xth-# اjj?/u E)?Q̍ ¸겸JI(8$`Q b%8 I)Isp+d``HB$`1;2a8eH7n" ܀N\$0Ҥ,Pa `@ )AbNTE<_}wO <|~\js *K+Ř0^]0+8ѕtIiaKr!eSYƍ "9dɺe%D9;\ʏeڟ< >sD *>rB-BQ.⸊yavs8׼jA9jy*C|22GKaiiyaދT.ܹM!_۴BTߟ[rǏ;|sW\ٛu l$i 7{b,͔d6r/+Qd(*ڸ$L eȨIp<,E_^t7VS$ SJC##ƒ! E59 秉C|l"fnan98XwlUSl\w^Y>SçMcnnQAcBM LJw=~Q 7:SOJt`a2ֶ!, kgO.ᥭPʙܹyB(sUݝl(S|Wu^ܷckm>'LՃC=P = [  ~8[?FTgV2k@f&lN[y\$p0ߟwLvzx_:U*%2^UƏrAn9\wmscoM6p ||5e]/.ȫ+?~㕰Uگ^~wd(fo-p =Q۰{oJB^S/Ԋ'pHwI5UPXl(jر hP'>C~ UPhU5b(g0bX0x9 5S̙-X A yGB4'[n/ #ؚ^^G'=o Nmw[Tc%%/JȌ'%Te#gH%+22V,9,O,ړ+'* 7 36yqY(/A/?O Jp''Nl`k4qVzc MŒLJI2sY 6:FL9{|?]]=?}?YO(8[(?^wwtqqRZ~+/SԒ<ǂ?C*Ys`Q3kv㌴LVCa7$G/"dm0Nb8_bL2~(xOQ@\KfpX끻ՂJ~EօNB|ˑ*EuZ^nՂ>b1mKAˁnZq=7:{dͪTYg5/\ri |J^zNt>v}8ݴu.arrcUC܇ GsۣVRLY@.!.[*h,ɼו^8r}ppx1\\Ͱcy/xN8$jfif)F{QXALLbw/4kE^87*Ƌf]pM>&9k 8t"Y04% EòW/*X\-/<ؾϲIl4}Uٶ [L3^IAx zzz&&flsff88c0 "&k'X훮p W/H&Hcx1ܵKVd(r+7biҠв貘HI ik\zq9{uO=!FP]B7(Nv=0x^n~8>uȞ99mE&8n9{k(_]^>|7PLĥO?{U7 {xGF؛׵0"5:͖I(0( wx1X3$Br+ȓùœ55)MI.*j 0[aBx:ZV7lQ7ߪװ~Cm9{@\ŏhkoaQ) y%/3|w\]p}%r)D;w0`=>fY;) qƇ}~D.h-OC(.~;ܺ41? k~캐W}j{[NL 7z: _K==X.ݰ)w)E ~>l'x]s "*fҔictLJ;:rE K ̣&9d16K /KWL<-0#KV岋Œ8$C6S^20 TvL0Mw[P%pV\W\"3n#9< v4Gd(~^d= ['ۻ}9X0C H'zUV(-LA1M  1 wUa;>"7 r 0/\{~R rAع[Ph"7jBŸn؍do*nwo'P<lS?W*=8Υ-ߵkF`+nQNg}$vwJ.K fh{DRxT ORZ% ö7Ddi q˲R1'g:^q{2{r  5#<^Sz;p# ڭ9QpDIPUcAϸ3@&ƎM /dh=rx_jr)V6ż4ƘH K%MEs셛 {ĔFH= % جr,Dg̱77Z:1eV-7jlW?dDgeۛZ:͚0!.m"fac [a9cR 5Ltz //>Ɍ W%/4|~R;iHMEׄBL4be`=: Pls/vڟ:淼"qeT$@w!J!ڴo ƻ0LzU*m#o*ĉOZ0`(51ֻuDCSG(HZl(48DpQWGBVX_dCc;r#ڵ$য հd|w] ぞf.Bwi[_ ڹsS?R[\=~y1bp;B ۚݤYrydPJE[Ver.bJgRgZYLl?)fyаv]Bgg˂eYM`:6,l}_cdAV ECv{7{jCƒa4dCVK^"i)J; nw?A(5rD7ġ"O3?nݰxގ3PvdQXYU|ߎTUU$ZHVfV %=Bp YNsrh$ɴ; ep]4DV[V_)B%?@ʎڽ1.]VJ2J}ߟ:E*)@;lg}-V_€RAÎ'jٻuKԖ@t_S°(| ( 96o0mʄbO'uǮZ\@EͲd$Zx1߀xAow֭_MpՔs92 &4ui-‹E‹~JS߻w9N3${utL) .u -O+ 3̯Jr Ҡ0n<IVӍtLGWE gY`s. d}WY\SY͔#,mmS}lW}~HQ#L3挏&ap/f:ןU\Zr' %T* :bB_ "īVZz5gӼiӦZdؙ3_!8GTsp$,(k[c JKAL| n(D:V;^QAsW1狳;$\x}R~أ#_6T4ɝ3Cxarib2c%咴g/qWP"(+W4W9M0iJ@JJ^Z\#9 XC/Ktեu N(1v=[m Ic^c"nk7ȏڊ^HZ'x+cJS}y渹8yJ" , JBya,1KwsٓL GO011koo[{~#?е. 9+mYɩV)SGi3z/&tחFGO0 M x1G da谌2\ӗd1__5|_;;[G^hp/d-Y/觴ؗ*IlTƄzx{m!oO'o/DQ\E$MR}|M$Ll~m;߸|;[բD*;ZǓۻPR*yD1/-gϫՂxA+ɤGat4s tljYPY{f"5kM:%#Wj c9u 7b2Ek0 9N(߾5hJRCئj4'wbeJRxo'kڊ;:-Q[PwԯGKtksuFN /??lpE(@H)iT{jj]hP+~{x|p9-@:&mZxmת#Dݻ;53oĈOPuF5<:#gkBTef'rVlD*eϞ)svs37jb:c x20 uv4iab0rNDPfдmm3& /M 40ؙ LY_F$9x :+7ԩvMMmmT#I~8߂96Ba`z|ŀN[70'+ 0||'>:c$DlCZ0o (JAl?3ɽ;CWHъ\ T<#LQWT-ab  )qZ**eKH{LBz$5_:֚7X54yѼjn}C?E!FҦ+?v۶m= {T2Q0^*~͓G 흘 %?7߽{+'u+ʹ \EEc)JRĝ)FpdݻVtرTks1  °f~hpm޼9ro md'#}޵o.Ohqisx8%-NL;8;z- d{8ZEO7eejj! Q 8eB"𸁈 iQ9])ɱ2.JyEr&'K əm#ʅID8wÚJEZ( )A Vm$BB\drA@#&b5[ȸ`!'NM(UB*J'DDp:ՋDQ("H'"JI3k6 ׮} c'*EyJQJ'f I}8TƗQfT?8PW'nh"p)ݻxs k5W5X= j B [[@Sb -жuY&t4w\öceǚWkm\}b{ $,ZD͞!J8%&~Ƅ8,#'[@(tiTh776 4 d6,,|Ei5rJͦeW,= Y 晆Z@S2~38`63Ǧp&al>5cyկ P;K/>gϞ=Ѿhx{i1 oQիwjiyU*.B72G[*@(ioW(L.JAba0?LzBQ !Ƀ? O"bKER?Q#.p@Ta7ΠC, RX.1?7]USCHb:Pْ`!~\"!: 1 ۩8 tJԫZP)2$ a*6$@RU.|@1hVrKTy-&a,}۷õ2QOHn'' 82|JR։"؄@nQ< I$Vv+:":M$z/6w\MnK>|&~5;5]Csa?8$$dƍW&PzڦnTS/O;ąͮ;kַǯI'qU3}"]2f;eΚ:gr꜉qӦ$O@ަ:MNs26}M̉ixk:>y]2ONe 6r]¥@q3[z7#oizG=zi8o1mxxŘ==]T7jࠂ /p"Eyl*/#2fF}DƞdmSZt"nB-WBD#F-.!b(_ /^^`E,OǷvcqG 'a)ErrSs H5τ)BfI`>ĭB58$r֤r'@ y8rg7a :Rde\w+\ (F [}q_ Vp<@9dQ5DHk鑶3&p'W PI2G!De:_zݺ^nu2M#DlJ\G@Dp~߀jNtc 9@ ,R ;`6}Ya+>p$0Dƫo/.s*_eWkke*D޹nQ_?6F# 2 ob:6 pb"Fܸ S#?c3@s #&zz&*pwS\@ H%.q\͌ ~m"f3" /B͙}HT tTLu(8#5O*2m@3 {trkr%|n$hh [5 qBP #A *@%5% (,,,//_f 8Ax"AH}XcjYZxie/FGO>}-{̀-RfwL?2]%3#GX;h^Gל'%,әsO fYD\/zMKg3 " օA)qSXs@10 3LڇyTAOD{h_`';?zVT"H*HÀ# jKڟAi'5P =и6p'Ah=x1^Gc6]_!cWQXKֿjJ5L> +Ggƿ^jo$t uQ_OSJA1KAW?R]IP# b*%G K.^ډ#^h<2lBO $(P*(| tQQ;x,@M4 u[/mZx=tmkPd~רOc=9^%d7YS'G9e̥Q C ['㰍:)6hqDM6"4 IoT;1c[̛in=般 QS}L-cå"v2j½)nKg]̚:ӭRgNLw. } P^jkIuw;4zcH}H[@[Z\< >- (n]]3?a`r^Gǘud:2#xiαEo 8XqXnƇ(oZL?_جڻLBeY a#+ V;2”S|Q ߻mȲWde3&FN?k9Zy񯉁щ/^]W@,Td j-= d@" DrȮalg!@OMfam0 X BJ9Ov g_LL1qÄN 6@>%$@f င<bC3FLI/Ps '@>6g4*OvʮlN3Aa2Rg %$}BrZ:BEAk?LS :&#vEK֌)P>̄E1%H0MтWw>{264eNt%~ d!:爋@Bqw !SD?z Hb7'pҋ1<(wzVIg{:FZou[;{Y/G gx1C"(ҏ?{($ekIcv*[ô;XHݻ0Hƺ#krK)釖v/z?"=9~'jT41v\PUxzFch}܆-4(60Yǵ˅5 N&j)x2.IҢ,jI+331&R*OL/PVST lH_}t=ԮSB 3Qa ()F ,r  ? A2E2[G@Hxb&_` l#qB$N&E\YT$ȒX|-HaLYB!sd$h=o Fi EEjA4O,Obb5O]~xx~E}knv=a#3K/HF (vB> QUi|*322HA@NS*z{B.N(Y N@ ,ٜ?Fa{wsO645NKBɁ U}ٽ+]a0D)Ut&T_E˿y\;3o{}k?PH|zVCyR"@gh ]B0p:ldu#  =%/$Y"3$׿Ao,ASTx#5 G:ȅٙ ڦ=29kjIhdxP7l?E_=~SCT9ǎ=cAppl44<ab8fbxz*c&&gf/n/ Bw,/goj(ɥ 3FÑXGlC[]Ss]c}ݹKùzF]&D7754547k:\SjϞ;sƺ3 ukkU57V4UuVT8[W-Zْfs9ڋ;[]e-eͥe- eMkO՟g@C-Wmӹ:|suO(Άs @Zp?AcmSSc{S͋[}cS#76 Zà}J"!˿DwSп5܁>(,8--7LS-l~ [N^HHlm$;ʣSͨ,i#Jn#~7%y[0S,@@QϛnT?,@meb,rooLlnl޼v}~vvzppXa={;wz0G==P_w'錇ZvVy;ʽO{{: #Ӟjs$RF;<^W 5pux\5o]4uu!4{}} ~x7~*x==ATuc==ݽqIo4674??w>g`kW׮nb} #+N,yA駟|2BTo[0l|~ :ɪי%2Pg~g ,UH QV|;J@UoeH%' !) J~63ϫcU Tjeiy||'8c@y Z~,9&o #MPQ# }Ljcm*nn\y+/,.\^_x cb,,\QB+sWf/O/^\87=81w@WE*t\H~Wq~v^}t`-&v1܎<~wY.{'F{f&&,^00?pin W^^2xi~av~~Ne^u\z՛W_[^0  'yB2//q70$/L AlcmR#YTTUD)ODZh |/yy ,/8sqIyV! i"!۔c P"-I%(746=5o{[1wKz[~A5#ykIa3%ƍRj!\sELK[;`3ۦ!Z^`|Lذ=<eA_i[ "A@FoGA auaN`vԌW4 8 |4/+@_T|xw{v&|clkc~c2j[; fs໮Hn%RɝT2K1LD'>CD@L׋|6̱߮'tATM ?4;q8(ko^ߋ {`y)!Joa  deP$܆4яcoPX;Sը\|ݜ6(IB D`.yLD ,/hPrDɋ ܒ dE}tנ7I@{2f5Ûa$0w.njH2+y9^[SwB+*P@":ha<{>΀U`" ЙAAdWۆ.fnLw1{^z[ާ7ǕE\ KHcM9J^(c\Z lOOADJN0"ȗ[eRtzڬ䓯JIIٳg!)~?NRAV04'z}@c?WWH1Q :&;5[pja7Ԍa`ȬO\+%HIj{܍+CX\SUGՐ!oYJb1; R;*A>Qa3+oAR#:_SxR2n#8?Y3v((X$1g5KGhx%kA*%_% Jm?b-`~*E Ї>T\\ sH#)gU?P=0CM6,P&PX bpp__wz*kkjW,0W *d`f'<ATlR<r(4]mp?Oij P4MÝ+SVLpRo,ROk3ׯ;s8@bE$P XeBL~MIP@*!8f=bTŊwUxRh*á˩:k@EKӽ̔eH@JURêu_ TFXօV?]OCy,'_5_%-P$v B[PyC=p' ĢcG=n^U-fyOl0vN,#Xldi3%wUA'pSŬ%"1U(Yc`'Uq6x Ba}5 HQIrBBKs؆P{7 u  # Wzeu ٟRT@)iwQ@!ƒb?@t?3f-=>ڿ?5&B/8? .zaNWܑغ40s~t(mB30J!E8A!(2;mhJ|t ~;  "jD!u[ OISJLEQx3 o<ڽyc wݙ"5jN)Tqh7ˏk*bN 0磚ka?s@ '4= +,/`6ldn˾h4o?nցYǧ5oI>vs Ђ1(IFbukͫSӓmG%{ayr8!)JAwqtyӅ]8obAI^X yaM! CU ’y VЛq z׳|}zkmQ"a'y*!a}5#-ڽY&<NBtyjK//҂Pk$΢TM%t=;%3m߆F,,/xA `Egt4PQΔ;4ldؑMpZώ͝KGgc]z\'PJR.AE-?gRF1o.fuNadҬA¡8,,B(9utR~IWk }Qur` 7J_ >*Пyt g 2SW/+JgtJj .M5;E8qaSSI٧\/ˋ#EVe6ˆUEUfJ#+zaShȠ 3Uj||˿+KO=;;^}U 8A(+aP>99?pM"B-ȥ,/x^.bCp\$?.)O<>SIF NoRkW]0쎶C(W Ar9jSN+RPhyx)gD=3 )/kyA-BR@#f$;HS*\ 0 ,)qE{:諍G6S}W&7C~oOs>c8%[,f d@z:ى3 \ q!g'Hm[c|W};?c87Ui|'6{ùzLL@R\_qc T) 2s院Nna´,/I=(yq{qϴ>{r4oljyBY/x[L7jg~㧪J<ต3SZ1lT`[$Kf%#`cF:>0Ҡ{-g !/)l}lyb{_~?g~{??폿ػߪ*Kܝ-Cg^YXYZbgkSVVH|sM9~L}5y }/k*ZxRL쭭nMM^gO4[?{˱ۿ;߮xF"~" w,=(cƭo"Wx8 )-zi{w35vJ֎L4psbˣ+Wrjʺh=S0kL'/ mcXÃQ,2rȼP9xj5oGO`AN=nGI51OXO,qc΂^rTlY$y(~T%6}~i*玓@5򆼰!mGB|(q;OzGE/۞ʜ/؟l;nqڏZNvm!Z9;D sv]\[Aɸ+*|w9(sZpr@ wEƒ#3rPI;@FgTK[_v4\7]rW 1g*?Rb{fgp8>9lϔ:Sl(zlKm CWkBOe%Ze7F-X\} >^Y3CY,) 63ϟ;WT6<{kQS<Ap<5Ex|ULKl3T֮MȤAeiQлhfP3#4VM8aiDZG="LސeF/wcBj h19ZUho|􇱇OlI3T8_,?{h Æʊ.w>]\5~eRa.UaLy˝'>^Urhh$&Oh'˜Bdȋ3sNveDbaC=٘OTC<u D8gG|5[ 552?陲NNdGavr_#r`y q]ZҒ@o,˞U,(c:km& Oa["q8 5|{heϑKc,)Px3iy 5;SPmrl9u}5e/ORppdzS %5[OUj˹O@X|U-3!*‘T2`3 }=[5Ӂ3ŋ+˓ɝ07)fy`yLt'uj_K`e pvНZ:yc{Ɖ腁N_ Q(THMGOW{16b"؊5% f!8 yП'MB@ ͯup$ U"mY͆g J|FP s=NRq*a_D%7pF yf&c7mex9I SR@j0aCMٰ$k,/x1ЅT35#(Y9WkVbT>ilz]2S>)Ѕ#83 A桥X/..@[*<-8-Y^_ |BKDzx'8Uest- z+LF2%)(g'ªy#;=}k_ |$bHbǃZ(8q.3!dx*CBPJk`ii\_]9q00o T9tPkrONFbOvDF{ֆWj?|}}I0;tY/` Z ƚ|ե/« a!N4@)UsP6'0+͊7w: D ۊeਤ) e)=!5䂑yCdê*Sf8"C^53D#Ns)@W3%gz-#ީ鍵# 0"E/.2 a!U"X^x#Aer]I'hPc<=Q +=E6a -5"`;;:R%mlk?N4׾<#IYr`P D-a8AFP&bq>R*>`Q.ߌh҆ 9wQT 7(\*уX"mΠl,8 ÙN,z{UpȑԇsN -3!!d=u}&h8튇#:Nql%(Ic3]i2ɪEAk(q5Cat4ȠyEXFKV%a!ga,Rv`PP 'Qg~_6QN\$,Ja_Y Nv:t_:C7dfJ 7|vTp)ZJ<Y5Tv3=IdmpZ?^IȄhM~cnNΆנ3&'N sUKE/0GX0G}ɒHeKۄ\hn= UMZH-ȡ)vuX+ črzbb;Lb&3'HOh/G6zE/HþK7摙"¼Q GIPc g?TJKk ǡ*2 T7̒R^}ANN@$,ԑ:ʞ @wdP:2{hlyc?EQڅ#AjAEE&|!+F*夀Bp[RՄ벵PY_JϦ JAřC )JJBOw׻S~Rv ZV3J0Ã>] =@+U3z,RySjZc(՟q_"}sZsfʭmX?uCQ#:GraSZ4}`j #y[0nm\yurfB5}cQrI5%AȃBӛjbVJd: 0G$PQkQ:  ?`C*y[6F6'V)6EWf1=姘JふGN\>uCQSӯ;hHb$+59;TW/߸6YP@u4XT!w)q6$yAbB/I!KjL`$m=q %8(tC""F5ipQBF3) (FZ:6Uʒ ShUU!ZzZ?Ƒ,/,6}fև T ^!$$ۀDr2;f/z#L %qzlRbx1F I$ (AOPhcթa8O‚lds)D-6Wi T))CK|ؾ\#"EF Pbj'hV%wmɑ#./X^܅Vw.Pr}y0n>; K/NuHLj2Rف ׃LtN*Ufq$3 uBYў-ӡ:) 1dn 36df#虲z]έ<"vQ0a}I8h &/WM=mF"lP~qb8^O}'dXSΓHGٶ#{v2`AH:}0 \(; jHQA$iI)A*sJ/<99:wa{uV66 FQEw=SZHrˋGbvݓ#ٖ k0Э-O-ݡzԖAV, m M"(]PHz $,u,,90 ՇB} v: 2uU?d,HUŪD2ǑQ[qg  aXz!o.K^xl2Uj@>Qx_t/w~O)(MqрHh\:̔-'TRY8eC/G81Uzv%섔[X2VQ5Gݭpv|)J!Ao(-HGl"n' paM%*0k[i;(]vC` apqVNAmWpPzJJDA`${ZF s2Y3 2OU /0X^zVz)!Esreg 2;гcϋ8aԱGZyMyDVçDZ)òw {aS(=N:+ @:O |X a8̓ɷ W[yy XjmdEQ<EBT͡[ܣz1CsFxl" F  P $Thr&9;G艈BIdC=Q* &JPqn ՠ\WZ]xDm*Iq=TJ|rʹT& R') ( >9O1I üVj6c+M zj : "_veT U7jjv Gf/ .-]ٹa딴 WR^ͯx(G&M4pBA*c!0 s߰+I7JuX: z-CK}+˓6%*}eHK܋lJ[$լe`Q%DB Zm4(9B(;V}5UNPmT"^bw}??2K0 0'L:"V@Ea'aCAqq E1xȠ)7&7WһToC\jaH%uȌ$M" \^K*)s 1>VPU;MU][tő{$NH 2b"d@=WJ@4 jB( 0 %*詈<*OAg q51v$8Sp}P@{rΔU]nޚK$RSyˋ>&K p"c݅E֡gv,ֵ 3ј aفh,*mBX&84nGuHuJ)a*cE ;z<9&𹋼]Nˉ~m}Ed$pZ \Yt6u6P/СZ80 0Cu |9~ ovt&w Ns F:y"޸yub_wn3?>+wN'ߪ-h 0 3,l[*wCkeu 76B.BIYn<7ya(#Dôt 9AZ. q[kaaY.22 \-׹mA `ɤ8B#k~vtksz"ewkUJoWlf =B0(ry $%)`a7 ,.w)/|P+u,eyfF6 w!Aq+sC"U$'׋[?0 0~]!͌mo .Ezcs_^ I%waa|@B)ɋ$ aaX^0 0 H0 0 aaX^0 0 0 0,/aayqaa,/aay0 0 aaX^`a0 0 ";Z^eM ü읅WnK޲]{ IBFIݽݽ4o9ܡG-ڳwC2!s>>3U,~XJѠҺ炌^{F99_gGޟƈnsTTT<9?  {4v /`I vQ:7D{,$m坂dD7/cPH]x{HmշDZiwr+u [Ea q_0^qx%/ĔQLZ88IWqi £Ogy; ; 548 '#FVkVoKn]]Ow-0Y;r5[==>_$&DJ%Ӭs#+x='z)A9@s܀gs:A+ /ZEkN@< $&<Bd|-]Car9̉k72'n|:qk~{liK[/y=cS޺281fF/fi 99,s{J "6j!ٻe}9&#LúJjǒ,HTTTeU#龬E;ǓПb<0d[I^f$*q <>`RʑNyTALxyf8N晚'=DYC4'3\ 8%kG?ִľ?oinXIӂ b |*av3='w hrlx2A}GhF @ SWQQ6+)\/4 Oʚc$z{d0$ os7WN U^sCy_HL[]sב/KGCIG * 1GepgݩZ7st1c\D&H5b>W^8rIg*q|-/c9. yjS$)E -}-3}kBJ1eUD_7z~f1/n(@L0 \Mlu=מEEU42a (01ꛄ3W^h8)FS5ԨQSO5I^%=W;}uM? F9'ZHA89[a4@Ul?`׵kT'xæ $ 55j `;>s@Ǡյ NO1yf^xX湢%򽹌]HaA"7:\S=xⱾwR/MUti5!C15Dc~" r@&}8I4Y6&W# 'u3.e]KZQ I$ ="(JBы5jԨySTnנ%S#wO0 8/^2 E"S;haSsfIv-~%54h?4'#﫨)et52ۻBš b  qKÕ{˚VaEF$ a\?]CV>% ± R1sIjxZыtYTƱnϐT|ڢAkFԨ-.Gύeczm,vtdR}TIW(4:?!O$jԨ{d( >&` > ȏd98kE(:a GƢ?rsg0=fKٔjIWV=~rCsf(lE ZzQGg]k=x/.ܔM* ҌNd&?8:8u9.ppΆ ;_\Mkn؊[ FDg_dEG.+X<_UhK]4q-$ пsqG% W@1H'.szP0($ dbNp:Z0RmZS|^=c)i9&МN bצoc=)6K{@ TPe_a /k @U`])kjI^,i'ڟ75#8&4 b0^I/VX23d)FWtJbVvRfdmY._ A|qu6`)tO<$Wz xZxlmd;|a_[x؎'o˒&QF-Np_ z= j`$ܤ{ tʬVWġ*HslUn@bSȲZ--/XLo -sIETR.j'i~UiOyEƑaz!b/@6Ba<C[[2OxS0F%Ir[Ɨ-Ԩ9  :jNoi 3 [N(kxQkG&T݈-@%o4bYH%.Mݪ>*>#i$FE_5|O曒on!cJ-n.'Dž??R nΪZ"pKkhxHޙ61-V^ #^p3e f"!/HF503aַt QOq"/ |Ɉ汮mzZ/ ml ry#<#ȸpƔ6R/A&{z35^,I]`a ~OF-'S\Ammݾd{4D/0@,,V^LF1co,/LCRd8b~PF:^|sMc'0t(#vb 8tUŋFXW 4oś"2DDܷ=0e3fD` MWy㤰J @S "//H̩*!J?^`-9dx akS!u"<^%2p;% Uu31?HAZ*fvZ;vt𽎫AJ=h[a i Ex`b!%^h#/𖗆-*x!͏,/ዺ #^AËCg/mjp xtN@HNE/$YPt3Z٧O>x޽{=*'[j!^;voF1^tX1^HBvuN2oelѳهO_g=RRyL̉6LZz0Ҍ$7dxa nU}# ةsx{$ x2f^0,^Ym ㅢjY8!/D/Ą=9ȉJeeA2f@|VO>˗/g 0s٨Qxsũ+ 9by x!ĝ/7:i B2]Xӗ?vəܾc|`27.f¡\}ыE o4%?gSkᅱߴ^-Ȁd5&e5`AXzPL8kܟ7 eTA. ^ ^Ԯ `U@/>DƲŋϞ= WyLMMMNNr! c jj'G`9}ڦWv^,89B)fy"#[0$>||3nG;LOS$=#l_o2 1aׯyJrNr.uHOf0Q f9Ca}̋\iڗHm5o2D~,Xxq mazג#9yvR%רx% eoܸQ,!q͛7'?~ LOO߹s^}[8s%jԨQ?K*J@ ل90^/&Ъ-kl7+i7>>p|&#x\zqvz&;D!XT?bmo"QL /&MU-Ax33,֌kP (/D!Ԅ^Wbelcc;$GB zJ;ECƋXIF+{h͛7eYI$_~?~o?׬Y)={`ȀB0^srDhi9zBrY9bl( S{g t"w8{|޳p4.=3v5r֑D@g`sCƺvh𩋏e _աK0KYAZ󺱘TVpC[ AG.[x|]m~zL&uRT7\a ;`gV5 ƫ+oo}l +e m /*^ ' Hz|Ϟ=P(@XBEӹk׮ 6/~K_׾ 񌆆V'Nd\v-_pD\6Ͼ<[`y{Kަ"qfrm/ʥkGBt"w4JUHHbcL:S-HCo'_3hmc[&cJs# )Z:nK52P%22N:(2EwUyUGq e*tJPbIz% \ucp@(^Jeu>,W`,K!.'/@tY UU1e@#= )df}[cЊRtЎ:"±{N^̜C"m1{ rh8vl(z7u:d)9Mmë7mm`㚝]>0qbڬDKjطTd`Yu2)zdB$KXԵ q,Zps}CZ$} (1}px^J TOl@ѣT#qFXBюf2,RU'KR(^TlC.6wlZZDsgBoN9DFЃܑǹcOrG=?=ޏoK-M:?\ӵwOZ݇O](*Li PJc/'RqUE+]ʢI A|97Z}j-bsSIEh4VB\)-EB-CK,dM7qjz?)UIEt6/zLv @^Q[X,Xz 0u{]UG޹WE(4122Gλwoߺu#j?0[ЋP[ U&Kھ3,?}bU(ބjM?m񉃑잫{N=9p3犇/]PHN=y:8c*s}=5xT>|2x+.E UJiR(HS,i4Kqw{2LdLoOW.og?p9{vfwk34o]3^&\Ca:3K1^ج聯'a1WRNT͕ c6fIpJv{ L.mS %!4L6Qh2p#)v+%|f Ġ#P. W Dݤ%CBX%[1sZ>Tڀ,@XbƌBBW1jZ=Vhmڿz}`s<9Eg"R @i{UtE?C7wyk 7{ŶɳWny(9IR dtŵW@E NӐWl}sC2D#,3֟_05k&S(GMfq\Pa>{̙+Fa&o(21cV^FpZ r4Ί# A6LPGSEfj̸1#7n԰Y~#:Z:{Ϣ >h$j<8p~5jԀ֯_oJ],^hҤ C 3]v;_t -UT ?#ii'ּkKϲUX gN=y}wl$UJ 8HI$L߮N[OG3[wo-^\;b\ /^7Q_𧑙x{$eF*B @l y<Egřn=M^snM[=+UYؗOkj:0D 'C 0 d^ho\$a'AmQ\_Ir0wk:Y7߶FQV<,wo+-[ǮCef8 n6)12{|1|+ 8*`}foNfZ~1MŌat_N4`C0KV {Q.l8sR3M !6 tW1qn΍oH|wPǻwUM^ {.Vh._ʉի!t_|)!,#F@ [j´?^M'+_I S k׼)fM[ 1ȩjxϮk,~jUQsHUUðuWmf3-s@AR[ꭏ~tޓaavEKMN6q<48̊E*5 rO/7&yqЭQ)&he4zfk27v {>, נ#\d0b؏|͕|e6r_2k\%T|~o;vX%;+o>ط Ly8Ë}vxVLo}2Wɷ%dkO35rK(HWr:ʷe*F4nBi<6!E_hngҢ^sGfMYgF|DtLQ,4tuN{Fi hU1q]g8 &IZ ̘ي݇fEw}ҮЄwVc Έ_`<c&ws‹p;P}[ o8krJÛj@eEq|+Ry߁.^/(Ak+hdA>ˆ]SZXxjo٪ $z7m0 􁑸>0q"Y(RP#֬VAE,QʈO|] fܲ##n6rhڊb_(#esVJH( ]~Q`I2,ϣqΆL 5l^ ^?VUZbѢ+V,9|I_zMB9{,$Bɓ'C*j.Df\K7rbtJh8Bm:x>MHF}U6|63 D:i/N:eILj$$'16 q`4]WxB&W'$7 *B &mN=wnSoҲ(.R@3(i#jMvUQXi?Qs _$td܎yFEyCLG~ZWf FPc@46Q%o~vxXqSyxb; LȸpG_y3d$Ƥؠe+L%Y$"[Ah5N&-%Kʳx^qr#g]v/̥|f=|);B {ZW:15e5AWJ& [r: & #(Minc6+d.rzfs{FHxi6 J| RV5VMNO,.RYe*Gjތ.MpS1afN]9lMkE*)&0 TѬZB.Y`Ͷ*%kL#.^ڍQSw oQEL(^d( #iIx %j8^~Bv,'u|j`h:}3AZPmJ0k#Q`3Vס(@PI;B~T {"$y" *`߶-pĈa^@RG/nj3z(%Yl e/]R>B(p`Uu ñ%+q}% {)Dr𨁀T6=v&D2F=Hye6r>ÂEN9~] }q]F̠L[lPӀS\ǀ24{+ -&!b.;*[EֶXi Bo<𷘫j!9M@| G6}KNڂqH`0⺌qB'Q}*3cΚ J se bۭ\"6>fpxA'u<}}:/W` ;!7-4b>:΢:w(._ Ep-rBmR1Zx͚~O_*$'>xr=#/G|ۧsωZ2^|׵}NmV^0wCXZ{ݶZ9xe,h[(Aa# ԭwd'ٰjݦ!f;PMwsѧ+kNJ ʈ"@ {|eڶhѪy]:;L' R>/3osΟIobׅ^Sf̟߅W5./3lW!NfL͚O__Ə7|^ry㘐xF}aԱ#0JWË B3ܺq/ ڵKnݞ?N%X*tAq 0Y$2vR7Ncc7zVUYuk'.UC1P`ಌ^jA fx!ݕPګ_,Tg̘п35AT^!>9zmGn=VJ)䀥H5dsm"' Zf>C+f,`X#C$s3\H\+x½ ˜p;ZicScaQPqڼ2LbT]!j(V%s!M!mn20|SeC/"(JI,"tfD^kv!XZ  "D .$먰ߏ[]M)β:m0+zWvvܽQ9XG$'4$.`\PnݭDB-6r{_DɅ}ě4Ac:8ı ̃t]t]EBl(*; @cH!2qwoW-9po1¦M߫ ȽMag `b8Wn51ٳUO31c n18_ppMk ָݼho^O~J#vʹvlݲWO!t.vbWmtۓg#bbGD5g.]u3JSVp#zJ3`9_|睻EM-[|y 6jVrC,a]p_f*Al,zQG O:i285%/)%A'qn'FCg!6^r.h1юLT IƝ535&2I!(D}r }`5.ލƘX5YNܵGW|+gn%暁x)&gq S| ROs0%E2aU`*$ 9 |||=qG_]8׬3gՂBbf3aGf!RNh-s$Wr@SwBz(-kE) @ڙ!S0QFjTvmh .NM3îj-7`9x᧘6e&L r^,빸+9E`lI۶f@NX\虎nC|4qp|9{'<-3h߀zq#J0w"rkgF%Ҿ ^ N \߁Wˆ]5~KDn? ̠)`ߐ/4l%豚 =wm 4ȈNQԓK totjzQ~I2+KGwD':kOzGgO?P}Y~+/w'F2}qP4eϐ & {~ul^șs![5`Hg]|=lpCl!$Vjx?V{ شw`/.]j^FPVgϞ:vɓ'Bѭ';y^h1cì!i{,J0t%iCe_qY@H`G?jӡ!Z8ԟῖ(GwmZvTxwm{M݀͸Yd;dDnG "߹Cs' ;7oĩȵ|񥎒̟{F[VS NMձ|&PPY?~x,M-041+Bn=|/Hĝ7S6GǺ`hMmqͽܠ3-S0rvKxI Q5=}k#3L}[]gE5g*+(?}/ u B]d EJ11iJ̲&rq֘,kl6(*l ' #YE@ *e(2(KEYUvjƚ]|tu,cҌgUxa9b+ڹ;{,%>|i0X߷K+Gnv'&?u O'if:yjbol9BEuprBB7z;;3εN_SZޫ#9V5N#mԻ#2"%[E/W<0lShBը\%."h![3ȊiQSH3" Y>u`;nÎn=<~PVFtBK!7I'څY<Ӄx~,oxOa!qhYfȺ9po[ojЯĭg:A^tǿ z7?>ry/ҙ: vЍ}IRG)~2BƝzD[;'^NCui Ѕax'Bߚ\ WkH#_q\ct3-ڭ%hT׶26XPchQ xޏ^嶯w87VCWQ7T*-++[3v`/vI?a36lX}|&KOxGf2cT[7N$ ?`:Y^n(9yA~qPQt".0M:j㒉NHD7B$iԢ Z7پ;mr迧 [Ğm=M٢ԷSm"P 8'.H[2Y܌VhI#>j b?fؤp/Sˣ7外N]mʲ_۷3/O07Łe &%礄]u+YJK$Q3/IxA%{OaH6B zrTRQ"Cqdh0 ;a0JAJ XQ^@cϫO7> &䘲B.@\+0s Մg,PJ+9U +)$-5X)̨$lG <Ʊ Oh﹤k.Ls} ?}񖠣"D].Byv +{OBS®y" d>{,0%MG'&<d56$ia|^d///J E8b0|!&B D]e{zكLyYLoM;*&/j0iV-Wr$iSǽ Ni1op2-38|R6( ul*,!x~E2 J%) #>6y$U1iCẗ>%z|Mʽpȑc3Z=H;}oxS);plvn~zd+Uc/NlAߘ)ڒ¢\:52Y'i, =EpKsNon>:nKM,L[(WK?"wJBr]:5xiq :(8 @@Ia:#4ws:#N$IMm6^@b4}=Q:#Y|)ȏI2G@ Hȭ22T)>%{kWï'MV smƝer&6iT=-hDޏ .R%DjSdoDΫȷo{A6%*ػC^n=qZJ^~B5*͗૧edKj+AyY-1N)p*LGn]+f+s@M#W^K+x!(KJ֔J4*s;fg=fucF5@W9{žVo9zyjJ/I7mɹg$ehθL^$9g& d@˰W*"q'ܵg7ޝ h [nq@`Xlab"N#HO@Pܾ3s@|i0=co$1UYa:pcIj<yj5k`LKb1/`ׯ#ժ1e1Z!69bǶ gYVyv4tI0טGV,=y$^*rh-SC(Q*#Ф3NvPÍ ל9SHj}ƪ;'v\UjghӞ3{_UfSlʋbN^boxҖ|5%ۚMȄJ$nEdP< ξNrwCgK43l>]rvC@q]u3Xm~H\HX8[lH3Xecw$DXk¬MZ oL 8zn;E!)wlV"X$u%ZV7qؓKv!ץG^sBqBK%^r""׆N advxqqI}[b,⿇X<79Fj@@8-2:rL}}D-F<.1*7c0mלpgGVuq]a$HDʽobj5㏍^{Կ/N/0Ԃ hBON5 R!wIÐ$y\]Jxcgh< nuPURJh) ISlo Rp|J5,ۃӸį_PňݥtbU.FrhR 7";x Hbv!MMŪNwpuNx/p r1U5s_)#"K_@UNhW9^RgEd茉'jK׺ܽrKY+M[7 ؐ~ *~Bm6MT1\b~i %oX=ߥ懲t߉Aǯ?}gc_L+*g6-JN)Jl 6) j?-(ΰ$#PBpBiIcCؗܣHcI3W}=m%7bcR%uBr$k0o8Q Gd`HΠ@؞Zt꯵ҍʀZѡF%MYnf <{꼗*SJao |ZUWbn$斛\DF.#gi) 24E7=,&MيeB.2f S(ped-╜HʵoRWCؔ \Tb|)ުSx"b"٬_}٫;U"K}IZ#2nO|=Ib+@Ģj+bVѢ8i5DvTdfX8Ug5=]y; jF6n~?羙ܪ}g/jb:o*qf^q1Iǃ#M=K*Z"^- y xK'$ dTє=nAgPGQ<"p XA$&LOﻓ&;| ^2`>>P&thGίr߾xpN=A%Su -'`/Sx!4,phz ҏ 4j{~5iN"r]ƺƩYU ,0) >P,""vSrڀ9y6fbң.iȃDWE>x;5#Xs}T!Ą jB35W:}΃qK G5(j]N͐o?t~+][I-H,l* @ 0D ɨ@f$ l;M^tB8’$0np%EeW *[;zj$&ԃ:q@Ѷȋ/gfa]IHT?P=q}7x/^8 |yZ%|dmĈN$d JM{u-8Gϸ i;FN$Mk^~nN %"T;5C5?n򋮚EڌS\JTsmaЧM \"eOLt >!)995OQ fEǑFa{9za#_4>bkDw doTY~ce2 T)Bӱ6ѰHu,7q48bM:2Vah|A.0W>},vzNH9Z+1ILEK/[fUrRx!K#BƩM)1E @K/D^hQM{[aAp.b  TËuAGr2<xJKVm;yqҘ_R`c+"" :HV2I>kϩ޴84&J!*!)~sh.Ld-<70 as|܀b TJ[j QͬEۯYR9%敠q Ux\{ݸx<^yj$Gn8q*C|#~3h/WQƳYjLZ9i춨nRش^`a:ݺ}! ڋ){a-nSFރ_E&x@_O/>ˈrSM_U,LsÅ8H]B kȓg~իsO5Ցd(^J)rKcZ@Qc"MJQ/8ޜA[S<@0zu[)Q {e9̍#qMqV^Ɓ>g޿`[^D_48%")ua5(:|ޒR+wlXӝ!>FIvcy轄Gb6=sv [<ЬP٠)K2rJ՜ q6d:rhxa2heۈ#>#,Qc# Kx=2Xj-^ڻ`˂SB>(GVb_/M[\ -!%)4U~: s !Tl@ c]G@~ȭrZ1!B O3'TeyMYO8KQ @?<5%L&"O0k4L#+.h@5qLkyzM8q`#Ui GrJ7]"qP9Ӽ^̋A$RWP8Չ-Fgx6ȑFn{il/dabj>D:}Q`1:H$Id*_,"%ZHgLAF0|>'sֲcIppSf-xq|JB X,]-Ij(4 M(ԑwauY:Ql#2'+%qAעp6L{8}h[kG5x?xn](0>ǏN{9pt؂6ri nظ\~aܴem%_uM1M>-%gM΃ZcР^RԞ2c(6.˿+D ܕ'NDy/};xaቋw2W65 X^$P]_L4nIOz M{BТz5bt`N޾tƗ=~#:Pfp ^4kYӒAʈIIi]wSbԦXͤhqc'o8yL`:  ?R'?qv=G [ /SΪEDloN78FCIpiT^@p6e1ʁpIno&f#2 #pFEh=^} }g+.\gUQjNk)ZpG$!䕑Im󯥕LJ{0KeDMe)6/w+B%E٬V%Fq [*`%WKM~F S,4BB'T!OyH3zQ<1d@.hyU:{T>T97`7IN@r]i ` :V!s$)_HQHT?S35)pF>عvqe;.OY4?N ֌ا l2ʿrF>O+4"QR3\Qb߱" [s~hv5)d8sÁk aE'^ѤPzZWM?jJn~_3Y™{בXE|nR;)r(*/7{ւ}CiًvY T ,ބXOr9/Ta+Tg98Wp?n bBR-ͳ:+/V _ ۍF^ & ; $8iӖ0ʯ>SV4zE~&H(@񣱸S=`ANs -W4Kh+!ҩ D+y(㚑"|d ;S5oYq'6=s^\z0%"_voI('DZ;-qM jA oB rBbҔFz /**3r>+v+=ECX"R]?'/_UT\WC4iu#JL1f,"}<I4Zy%/%etsd29b1OhqX62ې=k&3j Ѵ04ㄎ N.J~~AnF^yFU[EyI1DTzriM `#;NXWH`FL(Ȅ,ye9F#ܴ=iMxk_mdH{ ڋjxϛ7 ^u‹at[8c|0#D cacBSAHMCF G@ HT6& ,w?o Bΐ u BƔi? &E@^Wm9 g@5Jt/Ҳ`׏l^^^J\k j3Ȓ9 'qv Sܸ?>8%xSLb;d3V}3339""6MZ1eʔE{`mV| /pfA3YY BLI)Wk5c,㸵pnǰm34&)'iɮaffha433f4ҵvgv@3;;>WW&"pSCca~Azz:76n9TIKh,*,kYazkss lj؂\eL3$@!ܶYKEeqaaam]SQ+CX* RHG;Ȳ%mɘK1; ,TlLizGgw7{Y(¬Q/21 ՜ttLtv:CQ;fѧ#)x4ŽU Jkz+ C^Kf-*C SHH _#7)c}[ D@$ rPFJ(S/zB?999!6zyA:ek?+*kM9 d N`F&bɾxXsF1^xN䦷dLe`Ƥ1JNP|Tbcaq'j;s"Jm e(|r\FΣ}`~ 5kZQ\cw`EW 2s?We XGbUd6(Sd[)uܜj~|ec^a+pŧOvON?}( A8wxlS냁BUuM:QqoxQpEEu=kU$a$%!([YWvϴ^lDlwxx[_E aq1SOK%1[4M@6AnP0$^Qf0_< ˸#|:n3W+; nRS}%`Q1»;h%%tM<}WF[h ֲ?cl44Kmq%#/;Ǻm)kLYᶩVjR['/ &|0-5 y0G_KknDOweڝIG53ٲ3g#8z=v:u JáC|Mm۶9z(0LLu*`T쌌/>}bJwBF!.\طoƍ8N' Gf|q 2FrW2n2 dzT+V,itK Zob/80)N膹u|GepX+{kyv(˼lop-j%&F9"^ {?ӳMmB`)nfGI0;{Ý-" ό3{A#f 49U> "²` >COfR8gFĵX ~*/r8-ّܸLY19iGO8~겯%uV/b$yԙ( *cx $^"E$2Ssqz`T,J 455de斕Wҫ2O>y۬ &Q݃۰q x cezB//`oF~'ڌI:{熮Yf]8ٳM9s,Y2 p vs?ɓX neH˗/OII3f' h` Ԁ|ei#?zYЇLm~@+R5t=MדX$Ou!5Ѹ~klro;Ʀᅩ*36wpUtY0v7;6p4G9)vՅ0 pۖ pDb\&/4a $)y6g%^IgW>?v nx_G܆ /=407zӜKPf_ ]ͽxa{( Ĺ6+>[UQ H;&jHX,p/ W7)&'a6V„Ymp6d/>Q ^PE4U=צ4۶a:/.;WTT0VÊ\ Ĺs`  ΰ,\$ )iiiPG02e挩d6x~m wYPN:fMԬEQ"/*8\iAY}4'dM s Iڹm;m2&lv)^ &Lx\khlɺ!=bd ܰy  t':NiFm f|m&&F;hKc+CQaBV8|fliE0' %޹D:*hD$*^xtndI^z嫷a؈^xaXƾ_/:G)w/bly曌vVBO{UF %NUB5+Apɰ6kfN_;u>Ygikfz]h?3fڦza.Jm/~쫜^8z8?u[5gLX@`P4 R&^rA%z` [ Q4 Z*`@ R/}$G6d%![rp7qҌ g\@J jr.cV h4 Tura&VAE2´&1c^so۞_c7uXQd9ȴQ0Դ dCWBh0>GؼX]bm,F5 2HI ㈔.rqrJKrV l[&&mI\}#0#_q]I)ֹ`Yt$tAb9gisTքax5 JxQPApF# ޕ>zi);RV`I [wmۋxa*jI^yB41OưI۬,mff:z8=vǍk-R0nZb%<Mk`2 qxiwӶ)x;@dl/gQ 33x&#G /eeeA'`Xx1'NLlp@OS rTAn\7P|/ 4"a`h onǵ&}QwN#tY4|xu@pRE7I?\^aJ0G^hM؇1*DHW#LqiNb~Z`3Z,017^[;Y;s:7ey RDTH?GDƻ- 5W o؄_x 5)]T\Cx!3 %A~EU,\30 ց5NN:ݑ{LMpͥ?h^ ך _,]];T"*8"FļjA;x=2{G|Ci0=sp7\wZˣ"% yq}ƌ]A5!&ә ӣA N;82 < ETAn_N1u7. ̢<:4 k9ںfB\xsA̢X.@]%)щe!F %L!J-pr(cLP>. Bb?:Ȯx\\}P; j B[\8m܍cLG$o)g0$ќum^( Sl4gAՄVbj1.n,Z,zlS%R~PT MGhv\S vE^u,vl3w *eL:#f?2x杸Tr  5 B%լBENRY9tf⽓mВ$<{+*IX$ߖDЍ.^8"vMONHu]V 8ta.MJ5F{\"oHs67A FF3Bk]E:"L#*%H8YL7&a;d>ςlCmZ!풇^ _RB\ +R̤,,_)q0kZkWýC rǢCvgB7C[n-"a] {^8w |9)9R (< R;[*aƤ" k'SX$o 'ԽP 'w~Viis$o>6Q`|&Z#솬yr1;5ֲ5Ή.3Dl_ m玢oҭX"x+һTGJLOb9(O C-]ţSw7 _P-V@;VΧ8zꀗ'FW?ze粊Ya?ᔣ!sȆzAxQVˋXA5ϩՑ";9Ѵ`qɫ\ܕ^`PAQk}32yqc8L30[939݂t@KO_(o&I:$LVH+DžlD([XpU"9uNT9/1y3 ԸtL r!T#5%2N! 7YIX8,t]O͍֠wXETȈ b4RPsG6.ۖٱilNd0b1=ۡFFF \MO$aNEkx!Kg'p@c^4Z8_3 la:u^ cJo 9h`j^ۨs}`Α E2~]O/d5E[ucf#ޔ4ӼTɅ>z6xs.dӌOG }xj.kڴᴶ>)ԋt1OM;L'pZg؄m >olGK,Fƿ^h aK^e׀ zeְ1 0Iߟ~m̬g2snޗ;A;0avD6w_{=a ga˺cU|c{=J[.O?K^ɯz K,O=}='?~ĉO^I\p̑=}~?~篎SP-;Sw>O=߷S?2d3o$$]X(PUV\g ~w/~~SM#F0_~U0t։[-soƒ-N54^ "Q^`Xl'x[0`O :%M"j\927Ve@_t؝T/ͩj+Jj=4*iʾo{?muSW&,T}4Wm B@¸TgzyB_slr tz.%f"-?̀kSHbg>"咒 G?]wX'v ebGXeg؇?dm(b@lox_OZ[.8bR&)q_C R`[.2{GP2Nu}e}qg>zQ|mؘRV5Bq2SqQ^I_~z?U"-OxN5L% QNFۇ߲:q{spY Y}|(b7=zsJs_wۻ՚f:Ʈ05Q](9{ Vhgn|jГt}oE6ztjt nC:O'BbxѴ7$7ޓ]tbi/,*<.7W$I?25 fævVP7>-0VV\:mtDMF?| q/>lLj"MS_fFٺ5kۘ2wU7ٺ9y~L4Y+?}=G>ç2?v"q^h]jc7|v?jٙnY2zMԔGj z}h!=9C4VB>Wᆈ/"5H>'q9s-RxQK យ<' %+22OL=t`[tǷ=Om;vGigf̺tEgVrT?vGl8g]]^dʴǏY8%:%{ %F O>3"j~VU 0/:|5gV-qýiga{B-v^3~ظ:6a-̰h {R/b M[-?L]>UD8,B.A4=shz P8VYϯߺTVݲes~?>;`R@XNba. GE M~Ÿƿ ͕~{뿦W%xlhb@uN]%ݶxGߴ@ټ$o\ױ\[/Uh`z7ifI=LE"iċCT-(^xGz9K6 ~mZSL@|b'¨I `iu*Wڒ^x&pUe_&X茭18؝`vaK v-**&-ywu6[s˹'gﵿkZ_cw]yy>dǞ3C?lAKF$>1-:q`|P3@I=:kU;6oо[_Z: |9ߝGxav{142#m HWiYdĎ}6k>Ёr˛P^|z CxǏ [1B5TgOǸ˵:@B0@D=bjѩ@|AoŔ n,FqlGLM{ְ2{ Sz~/WVV}SF}ͺPSԭL}cacXNCmLְxC z:OQLj E b%'j%C;7W?ń4QBlU5FC>i,1 Cez+Ho%c6i$YnB;kpZȃ‹l÷;w5:pᵒV>yPgӃw^܏r}7୻@X8@LEB:aq]w=޿pLDt捛W[׎<$bàMvo![Ƭ]3yڀuBB Z,d钐 .v՜M#6 =1h!~tym}e㶍sֲ :~@3FHוhZ 'RF*u鉋ɥ1ԒGi1W 󟩙o=1{&dl ڴ/hS||ۂ;Vlۺ̮WN>tOͰWŝ߱ž]y~KO]yv؃# 3jލ;6tΝ~700IpРH^N ӄWgb9 %2貖g~c0lՉkbp!SdkQ%g e8n^}c wKtxU"8׳cM|ys=Xe[ qaċgv!9tZkA2\ a2V(9H27qq<D1i{Y:oOK!k=HDz\v b213dJv^SD&:w`Sۡbmxhs8!fs񷏚2ײ0ӕKLbru.m-тr@`ϢePP 'waӂKΑ-3EdBy:{t󖃽u3Oבƌ?h 5uӪmwo?dDӑ.~k{[h;cY˦ L) ƒϘ L=;f`a#D';8pקþ܉7SS-ŨE] 尨Z4oK7EH+z1liS_}ԲgF?|\[6mQ1OKTJɨ IzFNܩҮcf-zt}KTB4剘- a2.EA,^8c-;K6FNCBm}ܛtl;o~O)_.jE JOM>..έtZV95bWK/+u MSh㄀ f. ^"Wb`(_&hjhX|r޿Ǝk߾[ݻm)ªzle71e,/ >w; 0+` ͚=7`QD쏮]=xO{v>mǎiBu-I6.@ɘ +j(\E2`C ER*fҲ(Ii pmGlV^9ۗKkNm6\~^ ~^` /DMDFjwQΜԿ3u\WayW-286L V9Wf5#`\"-E(@Ț8HZt!Ë́jiܡ 9c4X/.5?S祾7n&+-I*X$gr鞃'Gާcvmڶll@bFu1 Z/0_ m6cVM``&Գb(%-]zݿWv6z| G,#ЁQL^ߗ'W ` BN@j+۲LCgT=0qdTch5nkUڛ2T>pm~dY~Ъ}wZ09[OV0]$JQ8}o&aR렝[U"̛jBwplAӜJs4wј{h/Kh.ZcQ 1EUi7 {{vT2PA#!>laNS'i5ߦuܽ{)4ĪYcc9-pdGs!57s||)+A|"95=dY8_1W^ w*$> EV1D[8C?{|E;)cM?{6 :epr̘̹0'o</U/̴/Bݻ&/;]l<7؎!2E"qsfFglehU$[Wkq\7UW2֫ (x7\Qp-%Qrz&K]c>^|JK-k&O-Bb鐮@]5CBn3l"G4v.0^,ŜXu4-5'kC1CCdi'O=٫Wύl5PFP3f$ Ξ af "P,wkk.wF޺sKHRr WnY Q߆MmՌގT9!˘,%G6C >dPݪ|| S7^u\'M~{ ^"2\Kf0(`C1pL XKe,da#Q!~gy?3RÄEHBur-e肬W|[@'F1BKf"6B*o\;W4]pP/`6lAv|{rCӬ=(+jTٍAs8$Aj0%a^SW>ߏM-oG G޵҆,<Ο4mDTԻ+dq psJXQ16ʼnuZp"*H;nأ{qFYOiߊR)`A-JOYzy;N=+&j#vjծ];@[hxI /p]$s H( Ҟm{u=D,hEk}|QG=|W;Z,<86Z 56xuV3&$^dv>=x][5φp+n#S/be9]QCG>UjWi9WR)/ܼlZn1( V@><*khX*2dV y Eeܤu훀یgQ8~@rgŵ8܋TڠxɅl\׾Og, 3 S妛e3SE@0YKΣ2X;6V,fEJ:+$I$ܸ:~cxACX\#leűax3( /m$4-7z#sH3aD"UpF!1`?| S; 2T늑~I~c[8MY1(jƁܽ:⒃ awv*Qނav ZojvDc (e6BkXƒAs^UauJN]Z*)e~Rڇ>*E/y~,66ff鈱AlGl 1r7φ$%U+ݿmC^Ic Ē^XOڤu_^&XÖg h.K&/m}2e^8d)VC;.mZ#¨*47'9psC_T-sB矲2˕IiQyFbGkah٭mKFk%b-2mm) $Z=Й6x!++=P;2wa$%%'2VO‹^r{[B}}'#AӪt,-3SR껏_޼ۄO_R3s#IrJ WY٬PWY 0 ,"ԩqы:ķS^^^m۶޽; a@:0$?/OiPk⟜74MOMZۚN_ػJ0@=cC66L%%܄gl}c֢6]rysΝ;x%tחa@偺v0 dkﱷRa@xkk u*zk%g#G" \Ǽ9 2h_q /*, 9lۥp ~#klf8n|(/FRteL( u*й']t9S{{d B8G}OE$S&V{ =?tF/wsFIU*3E@\ܡ=h=iIx%VԒmݐ3L: QJB ^kCfekRZD< ݀|t !M#r?aEv%4r+xÑyȫ×f0ȟ|$N8-xXJK Ud-쉄X#XX2uK%&d7tP Hd֋RQ UT--Ma >3yeWG\ T.yt+b)^l>i"U8IE,tT(Fy\1'k NK548&EPA9)K |XIRnpQhဳқ;qF [B(| ߇ ^a@>D-4Z+a ơ>#]K>DlPN,o! =`R 0GNY-wc%5BO˅feQ8XS^˒#oɫ%`o$\j@@G)zu=DDWMolڷP zQ޻]SX}:RE ~k%6.;zt'ESG P;Y !N/*z| y<&! W*˪$Ơ'o:|w$".XqbU)_;3 6 4h;\\m,CƷ>ML6\`\EN$oGAy/xKc0GEM ɌZ/~^Y|Vr/plkiV'*W|ڙΛl`uۖDWM*z{VQ߆!y#Cŝ1{G j P%X^Z4)v|e#?֭qC+0TnHFx͓zQ c2sJZ{INzcpm0&#WZV:YώX3^|d:Iuq,m ?$R"9;ë?1(R.lɭA~RP<| j *$1`)[ ѠelpwO;@^Wb} \%CYIӈwqVpGSq>QRr:jaa1 -ZYzfGBS1cPGpKwJh<%!z]NPB.mb42"ӂsbK([<hØR Lk)@!-&L~- b-`WSZڂc{{̭} -Y d~SK덣elFZ |k#X/TBbÆ:o+ [I XYAa0X *XG3UגaL}T{5fxĉ k_xAˏQGW6@XrTTIȭw/ #FtX2ĒANOhŦb0j_ +5W^Hb5He8a.PBZ;D [G`wRG*ȩk_DVԊk"ab)~TN s=J,AO]yRXڶWz(eJچ I9RYUfY0]۬P)/>#˵e_d>ጦ2<@p{i&FK!}&T҇/'T_4:8Ē3uŁrDpL5U) Ƭx챃ڨ0ȕ-0ȍ_ dm9rQyv C@jQN PkA)(t ^Q ܳr̡^Me*Y&2y-1F_&p  /~^ܧFCxQA A̤{{\m}y赣4FS囎wZ ٚg1 |0o׀i,Lʞ:UHl% m z{_߻3=؎kNΎNWc38/!u^ˀoԌGTt?! JڃX(Lӱ[(7Px/<ěe`߹H#v{=Y?TuhXNf*"lih41%жƞ"@STW&=݀ٗlO222|"C6ӦNHiAp"vRU*,ͣdw(s:R^*ps$Ur0\?tB8Y"9۪ \aVN#5 :I.xZ?PrDng 幛sb# (9iEj̝l[8ɓwd-Ƃ`$eX@4IjBGZ !6\Pf ,ɕxmދmQ|~``ˏTS.4do;毈-wGAvp<;J#"79D- 5)8\vp8t;$.7~y%U,<߇23k}/a`Ë K@U0 >u?>$[0|||al $"0#zyqH4ߡТ7xD3$0]C HKd .0YӤ\ Ck)M򎗅%(h 5:UXy{Ęp+&dYxc,to-Ho.Y1ChӬgR2lijM2Т@/UnUM-\>T/4ilHXr-ĕ3ǩ@^.Ȇշ]:g@_ XϹp.d|=;Fmk%W+Ny)lG?z` QbcبsOQO 3GPI3@l š}63K l?3Y5 U S")M_ЌKsxͺ?Y92%Xȉi<|ؠE7lvg6-vxwK_x*~%zٱEo<{+a'bpQ6̕]-rt,|QڽMd+%⯩ib-5B\ Kt$@k5ff`>OT|xSLuy& s/`~ <=d/9%qvH[Qh>eɫ/*^ik*"O+[Ȃ6;86̲4$$SW|%`fHd%US[X6^/I~{ٜT6ho^62D[ ,XZ*V$Uq{#E[XDUhR(RpDW,HEVyF|z[-:ȑhjܹ|Ubc )Y0q QBRfJj4ESѺ*,ie@5Ze۰iB!*X2bn p(-nACxGB꠶]#[]dfx؊B%(*0V%3J "Vm;oAXxbKDrV)eϱwKI|n 0x)cէo>.-10SgB31& Rp,fۺo\> 3sK3>@ L2ތ,wcc–7-e3Y450نU ۷֡s֭(Bӗ> u|@ ZivSSJXA/PDaRWt7 (9&D3v\酬+(`N-9XC͓&ͱj ܢٴT_8q@m,7s)|u0j%۷2@34r6#BhiNfMe)ß fe +v#;xjiZHAƌ=x⌧p?ӈ1)9DsࣛED*\1]Civ ={h?VS\QR<)dz ď{8<1ibmʒĆB"Sira0wEL%JC1JV@`e=zAH8. N> o iy2=!q(dcBțUl&H jLBLzavY" PT%E_[XrD. S݈ۗY"pUeiM-&e@Ċ>g'[o~HA/|X7eL{Áhp~ok8A@Fxq/~- آn ]Bg&*-€L0 HѣgϞ*&BNZMĨ*KrrrJҟ16ZePNM%5RҹkJaM*)-ρX=0rѭ ) `8ZNE0 6RObbZ"tQ=5¦2`Dpr'+.ih2t HiM% MJ\#f "C DIg#(U(C1f)NJH4J CIfC;v]d[bBgjG5.ύ+ר_=}X[)kb1,"Sq<džc,L//?Xz1 #oq@e+l1A_.)IMBKPCk]=xwSwTV6aBSDɱNڟRS 1$mm/~j0s G tVE%c yifuŁM]!$^0d$ZN\S')XkLJ7%]Hh MHu~Ȗn-)%D{&>yU&\ ~Vm:"bU=7BCܡ]$Ĵan%z-Ypow ){/?lV@`y(X[ѧɟZܖlDcԏ/rHR>p B?uRJ ̙q̙ݻwg[n%>}:0=J?cwYqA{:.;EUɏ~!ӵh8֖ftЫV摖1r}jJNGHim[4\IGolAv;ܥdw&LVFpP 5pり5p8SFb:k&ψ<$-GÚGA4GrZ~#iY0oH'u^Ϣy[5DB7j1SB!3\X>-ӎQl?ӦSr/Jo6ZOŚ슌CNfjEi,33}gO_W/&~/_?|Jx6%%O>~xwĿzeG/:&!㧤ʲRQQaYrJ;A'wlZ^i`ad6T@r[Nnni5-BtD$~Wɠ|zfa~^\s9"2)\G,-U\XR]$̀XS N_ʪMSX 'tRI42 ˰t҂ܬܲR9 Dq[UJӚ$U*o"qKu䇶J.J D˗䕬HT))(̀*R!Z#J*z?x3"TZc, y ;jirPDU UЋhjaS)u ^W@Lr0jt`ZJTnTw9BJV Zfm4BTz:{A|+.*(+)*jZQ)AX盜V+pt%PE ehqTு 7Z 6)E *TTۃQ0n2%1^\~?^gY?&`@cLiϛ6m9s&0g u$Z/ d iOx/~5i*Op.ar}uk,NJuz &fxv$4jc=}X.2iT $GЄ)sUHc1w6l ἣk.<={7.:V2JQScۨ 'ecDGÉpDVt8yvd:IHzpjrC8+5<@hL6#.QXQDdjAI x_-ka$~q%։F7~MΑ'_x\G.0]fww1g  1Jbl5Ųb[dٞarбcǶ{CjiWvg۾OƓk}*[j{UjUOdʛ=أO?h,w] ͭvыx,YXXX\\|G{c?aq;GdY?0>>\#M覘>k;1B-K8 ֘).0X:#W`4be* ~\dz'%o*KJ1F*ld7ش&@u0̰lHnKD 7ħSG5's)SPu>T'Nm*W1Rp E ޓB y@,EsHSa$$)4[?f80ob "vxO'Nf3 b[; כW]SAc}yh,? 5ƚ 4ZGtFd# O {;YV?r(ވ?}?p{i~U7*B/&&;cAzO ~zA\e Uĸ>`0v;\.O#`g#G\WeX2Bj`s5)FhAh"'"1zpR%J4 X%"R !K|GfCU~I N& 'ȂkEѳh3gJ }0 ENānԓJb,|gB(>]}t40 GV*ZiƜCDTw,*4F#PXq ָSsy54մe΢TodX152{/L U^«ִGp-z'7o78 G9p' ޞ.|^78]^oGWWW!D0?|СC fp@oo7؃niܞNDM" \x~~.jju,ӼB+а {R&gZ1QP $: 3o!#XSiWUzݔ+S /#1< U5:m'D\J%«zz@I/B-rJO{՘a`ϐk%6Μ9^VX+wߍkx~^(^I ,zj:D?bF#Hv)1=,ۺrBghhl06B/'{zђFB60 i`3t#X9'㪗&ѣG|'NN:Xs=3%  tIHÒ?kb5 ébMhB/Ҝ^~S>XEe?f)|z1"zAN^$ ;z:2kz[QV.^xky!o?qS'CA ZS3(x- @DZw{饗3^}\gςUփABfB]h%"VU#v@1ahT~zNeq4n8%kRnza z%N"PB/&z+`y/;XRNEElBGqd,//SdK&o !w PRY c"f~? @'ajQQơI6E,B8 ٰ(H ҋNN:N/1SC.X/\=d"}yrY;]sOj^ia! 1 诂^DV3ӟ^\0sX/Hzq9zecz [v*yk)t㓨{CI"&OcK}`fр0h6yRHzqrmhe_2k? Ą.~Sk".Id$E tzGc򮻗[EIkyaW)Ѱ^ͱx''PD/1jzvVT?u6b&Ef*Hާҋsarc{N8RyAlF{W)McDz++6 VO4:^IJ6Y}/ED/HO> tF^<astOk> zЄ>^"EZ/^ζZsA/M"]];|;=k)+cїō)R2zEg;{J/F A/x_{\x<׸N/*"#G|>3斗 ׭zB @l,EB/ZŞO_lmkko\T{|5ŵ}h"Eҋ!_A]a׆H`P' i`^RWUڷN/ "E&~35A/D zak)/ z ^L9ޖM?y^[)BIֹJm탍> =^Ds6p47RZ)J!d)R$0 z_/)͹5mCM 7-Nȵ/ScvgGڡ# BzH.J&EɷW[s+[4 b!(4p!ᠫY>phk8HJ{7Hz%Ud"UOIh+|U$[ZCc@^s 33>,fƛ;{[sZvV"zL*T ŕ"E.qfTL2[IZ2wXs0 0;!ap#_hmXC:'+ ,("E B[(kڊ,˛2:G&Svѹzp&Z[-%?tȯvF ETTza+j\_S}i3i#EЋ@7<ӝ^8>@+Jzv>r.tEЉy41A1 NnFHH \~ (:רWbDP_>g]79] ~HB;862lom,+wbل@wL,Hӯ%$[GxBOhlP}7#!DSc@tK,Ժn[ tZ/s- E-hj\]PWsNd%41/藒CVJrSn&h&(qu P!+#T/3/bpv|f6Y?N, N_vl5%k ?sr"UM i !t|"EӜAÈmV6I9V: kZ]=}33͘ @/ D/L"_NaglQ\T{_Q‹lDYИaGF Zɼq,eyD Eg4UMhf ^zbt1s,F܋a0Edl<ѐ?:WBNMuǖcE:KM K$}_YRt鼰m`wbߎ⦲j[Gg4:к7غx?CjGыVO癙; wT˩~3CbHtLA3ыbӢU`QOtU=w17 zf#'w5le[*K]ρ^u߄n][ -R%$6 /n1)_A"5kl렰1A/b("\ETم?2:RWrs K~yF]I-RܴWp LkzCŎ,tla61`-βYQaS%N w~kdZTRPq]}[dK%C)R\h \Hts'V/RTTV_Y=60[v9A/8`XlvTgoʵ=X|k \1T0Oo^HHHNf GѨ؅4N{Pm` K``^A/E,yA/ƇF{*2K7u^ғqȫ!l) Cf_$FJ乢1K5]%ûb6nf1E&1b/1ЋP|cM=%7Lxb2LFnd$!qU1dZ^(\ݠt_BO=j*7Gn*h4W죈T:h=ƂȔ}zUb,Q\Wm?zJ](K EUƞ:_UTBB9Io|S zr0l?4k%@Q& 2Ca}"WsaiiCJ /YgR/$MRWS HpE{jA}ǖҦ-eeeyP=6xtBIa3Mn l8g}mg*;(K~5X ˹<RHeWִx'sX6poֶo+o/*)lo+(h4 1WP 2zaDžuj\XPu[m2kk]Ç^:w %vkqJobHHP8HwVc+oW;wlڙ=[a{sq|j!0@ϋ+rrRx>dXpTVe=[@~%R;v!IV/X#!!aq]~+u٥7/{ұ.^?8sR7cy6 %zkptfPuʛr+Q]Ǚ?W96Tza ^x8hOmG/怨 .<xE: ׺lu٥UYy[3AKm+VXzgIͶꭅE%,o’؆t+4Vh@vo-ܜ9g֬YۡnU9¡~06f.a`2@UQ,`$’^QR*Nl3T@}^T^[`Vy[dn۴#'~{nm ȬzE5We-46l/ޚWvfٲzECcuGm16dK [c6 -wLe|"|Cg*MiosZdžlʺ]9E;vfnF >3gKv'[GM \Ҏ92f@vdej媴eTtK:>8:;M;gqC4=30E:梞Ql`#vߞ=5X7oolZVv~梒-;wm.ټ]g>|S  ;wn(ؐvv tDCAʠ,Z>PsZ]H_zz 1% J<f [!Aid50ـvt9% UY噶,[E6 (Ͼ !!!T 5Y j:K{lMMV.$ +V -D`Gܦ:Cܘ0&Z&)8 -yl5嶝eU;mY@YefY.@*5A;M2[f*쪂F&CH%oZ:N ƀ@H b_k8Ԝv&G?7E=\ zjekH5EgcjFϞ.wI:;W@M *C.i/B TBn!'ԓ腈 cҢo "][о$ w 7<3418W踮΢<7۞ .VNW_SBBU.n/pwO垁ک=MD0kƤ3yN!% 웈s  ,nN -& ;Z |,9#378=ubOpHĞ+&DU09R@fƛ0H1u9:H{3 ;xý+67$0 $׊DaX+Fє}Yq] DXx=ʉQÍC( Ph#Gk}3vX=m ;m!"|z4 CI)X3O>.Y@ _XmQ/ A!oLQLk*ڨOR7cIa~:EC`xH*pR3D 4p&#bG {8q?7D:ȘA3iE18 8lZR$Xc.!ע37c ClN @$0CH;zele6q uG0ypl1A(lBaIm|C!$H%si0@P;>F sp42ƒ @$ϱ{#քcP_0ĆX6؃Bw>.b|!-~碯7 2R% G0-@VΞno\0 fgǢs@LTofr*73R7[~rLwzgbP <FfǣӁ;6ہm2%2!-*J9>v1WYH , QcwcCӑP`(z}sX?|AHHH`{Bё`p" c@/\.``s rd6zp{k|S1 Hvы`7 2x!/ v[;m Wn*Qހe[Jq V޲EJ'[KjfV6\X@A9\@tt`& ?k_@%aG:wHBބZuVY|=W' EƧ흎U?Ppa#aSQx8T\+KYr#ª lTn-6[{430ua/6'j nAHczAoev-1=X ]@c{Q'hvOl)R7ey%xI绍h -nQT_|߳0R UZ_&G^O!O ughƱ-= /=S H,$YҒg/)ϼT㡒ڎ3@t0}}0VFβH 2N z7=a_?T_VŔhw?445YоtyJfL'5%& %8HHHNȪq3ZӉnҦf;U4'(9ad zf t̗3m`v8o ffv\E+zrII%c"D贏SKJL 3:z"A,YINlH5z1ךv}VOD$׌֚(`5="E#M5JoQmT 87embr52FyuӁ(Q{!Qig#n~1˙"#*ze܇{\ bvE,lMe)KY JamĂ\>=IVɺqb@F]jzmЋv@lgٖf~ǸL aBCK%V-$$nZ\mBUV3!1VG:޶|A/z",S{2mS.P*)nZsh,7Pb(A0bs:AO*-͞=3 /˟f bQ褘sgzmiƜaq-zT-(%: ER E,ơ3[SY_$vᒑ|#y50 k'mkL A} MKΞftd+)JR7R'dZ%D5 X)RU6H_Q*Jd) DB}@@ޚ0@9\"P}?WV;3#LNeH|͖5vRlT$sz!>qR eu4yg,>Įi4"}ÁCO$:7qd*  g9 {{IORO +!XϽڥVt#g\e+R%Z4!pȾV5`ix.M{-BaDH ҍ^D3m3ՕSc_xu@QFVS$c)E Ӎ "%:cu# gR<>R/BU#LZw"x\htg3 ubW8:Z AYP&?V0 q꟞:0!ZǀG `Ta /OGۮaf:My+&@<@)|3fJ x\@gUc"\Ӓqg{z- F_x Ny㣩),\Y')tEŴ$UzSTy%Ar,>WmDáj6w|0)8)Bή6K?MGP3/+Rh)tZw(w7.T) bQa"ŭ@RB,ւXdG0oMoeTITU3*gxw\E]>k.UWcf›c,I{zNjFpd\~NiOSH"5awe"ld=n b_łw{onY%zT9*f* EaT+ɰ,L1XyL'z):A/ tA/L ̚үj*DJ3HZD 2=H1=qex6HʰVt&V@4s>@̼Da Sb1DJZ|q%8RVPZٺjXd4x];@VϞ`uSԤ7pc"Q cxLVf{zUce~DI/ )L}pIM.X)bful.4 _mS !^\1!qV f7@}ܧTI@ELshJ tS .)Nb59U@4q@aAteh9o$83^8)Zܴ'Y,jb,d^ SWLKHSGo`n:|Q2 Y,RĚ)f1Ak 8=lQEȨ 9L/M^|>oߞ|[_tHI/AT]ڣ/+Uv;'BOH15pxrj$4]%!hBbj'v #(oXf,Io4C Z@[mH!1-,, fʅj_A +q°EΰG' [,,l/qy#iģM!^$F/3rVza^*:-S℄$OT^aS/T ug\sғ^NO_ovEǎKR6z"EY3[$c's39A#و^L;M &ͺ":۳4*m:b0hb 0WoTev 5U Tf7VU}J䇞N]YfJ7`4+z߰骚( {+5N&Dk8ڻEBb3Q +. +D Mxb5F%q|¸>C &.g_.m虚` '"L3zMM{w,}~Ƌ)^PGc+=ΘecƶڂFXo bڙo&VW TO:GK:n#EnP'YD +J_\P#NuzvT%FIuo8ǚvU ɢ"&z";1n&ȼ= {-ʸ?:ntմBGau6rhqpq=Y Lӄa&|}9|ޖq˝ƨƭdrǺ/s87*{e?a1 (КdYOf@> -itI!ւCjíGkO#qNqI#k:=j^xY5D/.bM0ы$sH`==I/ah|x&||GNN};ձE(?{١G|;5 x.=ܽx&Kɋ >kWSnLҒfs,Ka㋃OwGOe|^8TOEƆҺʇ[e8HV8hf/\@sǠĹ\^:tjIUT23!|U1t VzCF hike! G5'wŇSjPz%K.] KrЅgprqqecy_hZ_(X 4~H/?9-(2nW<̱#/[ͧ ۻ[pP嵵(4xS@~VXiDMEgyrsoV- 4і.T/߾?֤]?U ؝[?y,z}Eo~~_y _,+;?rk~׿}Voi>[2! /,P}o.ƥ2>뇫JKj$9l\Y>*k;ƧA/,^ Ӌhd]ӏ^OzUtsZMc[mJ1otyzqx}jыuЋ.%s'WVݳ"3x/mj.}O^:N{ ~uU۫_,|s?{oEO2qB!;@N!;qwqT4}gw߽:O,~ayޗ[}~%'&SZ,e9ꯃ,rb.%ʼolX8޸vrԵء 9ûֵ$Fi‚m4 *Fu0!;/ޒZ٬Y *ҫ|@@S;n kzGrT%cF`lD#a3V#>b2/`a7m" XįvHյ{@RL*ʳvjwmkHd"k (5_]r'g’rNoO[{IT)Q1%a-ia ©t cML`DnnA`d @F}ݔ-+am)@?\z!m_FA`ٰ|B Ԕ̄9;|w #uwZKR~- 3Rg[7Sd"</xFT)8 = /@#/޼3GG?]oXZM7xpp`ŨxBEt/^l9pTïfPG ;7pB|9|_V<{D1P}¬}]7™"CeܗvӏVuZ1v糭7~}MΕJ*z1 )' iU/&FԶjb6w'L*}" 14DYXf4,_B@8K+[PSlhlMFOAőIgE5CCPdg6bL?;jB=W!. ӭwf`,lU)tG6ZU-`~#02NXZWi^JȺEsGuN'8THpGe{C{~Y.06 -H/ !PUm̅m[32> RJ>N G2Fn?tqq[y\!Ęb~ҥ>vXRNpjԶS;8 "'pRNWmKZ|ha˜7 e7:Ţ[_>~G^@C 0L$9_ /{/Ћ'D@B@L$*...(((,,7"«]J2 P~漟&tޟGH}sI;_AQOu;_~'}_ N;_^h7=u,Td Q.</4ʷc%/%Z"_P1ҍ6l "![D6\ oz(&Y\~ޔv~\YXXP u/^NrK i¡`#>8O ,v_xfCF>T۳L8B Hҧ6uXxJs^`)+KDYٹii陙Y9yyZPPTTTR\ C,ʕ PxT;*^?Wso4g9Y_>aN2pR5ƯŹfjf7u8%*u0VuAd*rZK`a.N <`~.^]VNrڅ?y.xux%X#װ[4P_z'g ){q ^<}tÂ_vp]k(.BSak=*:t^tܾkEh6.lmև+m |$[Xxyx]qm|hi7\x53%Ty ~P3q԰;!%o8k51Xhw PNM'.>JB7O= /̧얐R]5Ah^[2/4a8Z (wi%j9F(4Q0Q`4pȚV$?\+M~[Ng!V>N 5 BZ&#Ís^XzfU.1؆FfR1cywӚz\MlKWO߼-oi*$||y v_8ݳI% r!Z dW9|㏡)zw̯!J.QDzVD@`=9_́3-՘R;&-##sDˤ&|cNvDY,!V.υ*HJ:g@p<*ǫ)UL[|G¸sgO9%!HXe '$4CJxq,^pBފ8!JJJRh)iiiYY999y2()A!KE"Ll"+eʄ~6p5Ԙm$xYXC'/?-RU|/+P蕘!=|xQa^j4J&a5M4,K SƁaA%+?C48_}2z,R(U|>;V?ï"+ 4lr|Bw:^`aq>B@Xۏ8/+r]yR{‹ggx0*4[SD_{]qu;}7~~*cmњoxP>`uJ6ƓM,3QZ^~'gV[Svd$Hko*/ixLsםeB~qgλ6<$UimkMW$fEotl .5+ q)콇u""DV&fq)|| $ \ 5"3 ~F  ` AvwÞF&ƥ GU^0Ocjȝ|P1\9v57?ֱP`]49N/ʅ(2BbQoUC 0TN)|*c/و ]*MԞ ߀}.r#'}?1ݦ`G !7?&1QM] /&$V5!r hB'CC~~Ux'?~LO>}/[$&&xM7Q*X,l 2a7A 6γ(LUEH`@4Aū6&;9d& '^LiὈBGx߄|; .)aڡvMz1# I+kG ^eNX[b=74؅US#d*Fjx!-p x—\~esxAV!/Z1.laZ̿?PUJV\v3m彂UE)^P&L:LMfuO[h?%mYm'ŏJ%Ko)ƞ/=Q@x%w?8ݷ ͍~$37 ̋~u ]B[-୙1̦ya\F B%sLj^F\K#ZɏaܨsT.sn@! k~OJ&5vVX7:lN6-Emj{`D@*ܳ!iԼZ"FWʀfX_ *Ӈt tfh^~fx ejjh*wLA)Ocf|6D^$ E%,ɝAǯ=,ׁYpHUڢf LUp=[5U咒l"`4ǕR6= Mlnљ"vNlleL;ŔލȈU]zNy0TԻ)A&ꢪ 0MN ԤE W JWBJ?->hԧZF>岜%EA? /~ߔo޼yǏ 1ё1Qq1I Yrsrr KK$"1n, B.€^y?BlA3dռAf2 cQ :pO5UhmPfq^^E sQyWyq/dÉuwYxޜ :@S<:'Ӂ!Wߒ+*͞^5a8ylO%ɡ+>o/J{R2Pi|Tw"] Zf/~+CDI<~,[&'≆-T*V=cSղО>TV R͸)y" h]gO(%Qv+=Ɓ]>`J߳Z'v{O8pƒE!,%dAÈMIȢh#;L?\B__;|>ڣe!ńwbD9S,,0uK-iu>-mͰp #K%1l(ҽ`-9o#s \B cpEcҥ= pUpbqiv++g YYtJؓ oF+ F[px%0滈 0b3, Bb4ޕɽg%F- i,{cc}E8I't_ֵSrOрr/QCܫ#^LG4zvCQ^ q> )vUAD@M$ mtBEג-|x52ȯeJVt&.KkFxjݪ /AlpW<{߿o߾"!>61!EzZJfFZnNVA~na~`"1䎔@ YU v"€ #&a֝:sίy5#E WM(X\CsT)*1kGݒ` *t< LhH<1asԊ*4H-fZ <3 m|~  q,QH$`3zc"bPltLR0?#vlBZiq oƽ@7iݐ9=^{7o BXR\Zv= ;YKX@2TNPcpP+  mShRh#EOK" ~aay5)Ƭa'ro$‹߹);@uxQ9ѭck^`eR.I%W`Mjƅ!^ 7)K{eEhdImXþ{Zhe9:2aOXΥ֢sM,}U{)twt=5`ε(t>E c%RJ^X bw=$\+ٙMB[]nؼ b4:xa$K9##u0rtּAI.uu ]~;۲ms߾0;vL0=; ,*447"&)jaoQxʗgE7:#y123&CG%..ܽG֦2sT21=6Jۧ9ةyl`K#@vvb iF;g?|} '#NH*xvGzG\]\j.= ]AFJyu11ur!Iư簩Ҷ?/~ }Y*[Ͽ1`O䭞ʼn&n[Ϟt q46| 񋯥\Xz3{y`β9-)]*[Pjp2ҴKi#gjh aDg憩>T~~K2+P1;H{p¹rZ/oD[$M.1f?Ӝ饩:R إ01>juĩS|ϝvgFDDD (0`[5'M5+ !^Xj^9/g{By+ca=]L'Vy'-)Qq9x钤o/lel@NLl~bTtN:S8Ek,G|M*<=`R0xJωs%✨#Jn{(~yĻg? pE4 ]4)U͠oh:o}v1]V?m؎BYW[>ޟ#f =<}4$$$44ƳODzͫ޼\෈~{׎ƴkۺV.+:9577sttYvݺ;[ܠā x $"x,f)'Z0LDT5mHd̍ʅGI1y06U/D~Jum>9azÙ(b58ײ4B.;$?fۖCǎB1 S IA!ЃbfA6FO\vm6J*ِ yo=ѥ9˷rK.ժp_0xnU?Z?m3U'n*,p)3l(h^~/UE@SQ%I3:@a(-2*EAbBtTd\qLżUJ]< >B(#/).%!-WPĘ_2SC<hX?<*9*#GrDol ́ $璢 =1+5l=$TIQqvFBBRvLbO>ظ4qa)sN>)ܗ(%)9:&!+'WJ*b#R ^hinfVltLffz:Ы*Y/.)1M&U34\c(hS(R:=- Kb2!J%{An2.ܲ;%1IYE2 rtS%r/j] e Ń=z?}嫷߼ǷZK玕+U4bA Ašcut<$Op]@ `J_q09_6݀-S*WGp:U5~̔cRQПo"Sf?4!zr]̅z$~  z)B*N OWQ+ɲPɓZAtȈ ݳsƂDooC24!sw8}޺|xE:՜.[`= 6Rr EYJQ!W8*=~$̲e$ 9DHi~S)2(g1:x#I1"nQmI tå,RДd[&jVyX ]Vp./7s!Ɵ7j-`ҵ݌zU'NZdoޭ>wc##ŌլUHg*tx‰Ǟ;ͲOf\}t>斊Jb/U=RUWO~VQQ ǤbCeF&|5J<`F7pZkİWR 8r5Ŭ|54FÂoI3:Cneճ!@ t>\p.49Uc+N|jJ z@ h/ O|t>oW8sYXRv?+N?^J u5ˁ nfS !\g(쟁~onyæp\PRn'>AȐP8B27JPi)z;[v8?1ːK'S G rP^'NɆ@2VRfqyvR2*cjR[@nj>2vF93R~{Q+ ΟHRuY/NlvB@Dnٳ52GYsjٹDZ Vqu)jxZ{ދf>sor9.J^,R2JԃO{\ǐկg̝内+6{~ۦl%zy(AB=`&ߑܖM N.ދc9jdžú1qIiY9g[N8ߊ?"uIs8r4m^/FJ,? c\SŠ~UHƑ+oDG)؃2e83gq]le;.ׇÀ,+, Od~|%' ~r?劳I3  jpnA.Ą_R Wu,^ܧۣЇA{x/ <ŭj5 **leecii ҮBJCy5WQf5oݴY+u9U414jҴ32`v8-U@΀W_ #rW뼑ߒ=γ)Q3&:^F8EoU:~Ͽeo^w)̍ )HЮg_}VU4} g5sO\x&|߻uӦ|Kb(AT'/|:eϴs1H3[1ߧ__~$?ۣV owL)dO6dh&MkTsJ riϺ}]M9<^p\">jڌ9Kg =oL9f9~V,(uiXHЁ5kѨqӡc' cb Ef&֮^ݭjaFZ}n$L7 @/<ի+߬'G VFݶ8)aX¿Z&]f-E}1SJU|n^#B>~6-(4kԸz5}>xAS&(RwЫIӖ&L7o) ,#P4?8ĚGV뺶\{)@wN scz_ʨFbm7kj\M}L;{/ }wo_rBh[/>v_lz7D;.}[5!H]:r賹#Og;m{\#7-g/9\B={͛?~Gb|HրL GC->/f8q6^Ɛw8Y[1ϙGrbs$!NEh 4"/>_q+qag! ͊p4edҌkXGCUSЁ!+[/N8%|ՠ4ɧu#HXj6Dx^29{Zw~h=zjg/xJP`q;w 80^ʥuw}@`fnmnaѽF/A/ [qmwڵ[nݽtټE W2eԸt`-ne> t }s)ZQt [8qkҥm?By燶13 : `w9^? %S @]]ڙ^0v2%)O&beXLAݿ%[4R\$#Τ||j&/L3"&\Rn:5th%e-+֤҈-`&f4 %ЦJR2E 6ISuBdGn6%P'An̫;昋gl 1r`"'lQbU+c /̑{[} 9BއQ:h+E_K3ak4Kd~{yO(074-[jGJ? BZa)JSք}EENEc4G\wӽPS<2wS-WwӠ+#nV6ojR{Qt֦4ޔbKRoMUÈfƍmo%&))'/[TZI*  sSoK&Qye&QT!wtags[ 8wrk5nVZj6عc/^{ޓ'Omd_rr2<0 ο4q"V|[_l "]4x .Q|*aS1wI* |5c9a.ѓGǂ5˄4NkL p'pKIbS|BGdl0WxiCV j89`'zj!&AR;¾@1 ^/nܸx]x8xΝڹ, D=feZwHJ΀3gN4գfM`Qتc0{V[89 :$a ز ?!bs/r i!(-hp;qitM(Z*@ 4Ӗch72AE mxl^g`o.r~Գ3^V: } /Q]h7aReKx]~RFRIzy[(22sB|i1dC$E'̷H`]]{gj:Nt'fF>H[ Q>h_ig.&He,$aX7Vk}ˍv:4'ܮ:Ǭ׵%bYbiM6$uJF6p֌g]fVH0 MMUj`ès @애 '/9qٸ1sݒSxsM 1 )|2gɌ>xf.sĥϟ:qC۶m[~6ߴgϾc'ߺu ( "eXSy-<1rVb(:hа,pλΑ?>&6(/[hi"FiHXA%% 8[,қN1| Nǹ pUHuW#H=,ϝ.N!;iP}:kn(UFQ{z̷p.@EV/' ҞjA5q,>{|R*tS4%0f3o~.ڹn>-jlJ%47v_sىܲ*vgSZq jt9KFG@] e&rh qOn_V3_ MJfZΖRB!kљslhAcg2MK`] KC6s,gk5![+! qR;*2!uJxjXk֧ԓn]43ol@~ AG ;`"=K7$o*ӰQF ;Fy<~%4d8։ ( [{.[ʕ+ O|v`ch=#c;m۶}f+YX8XZ`ae[Ү<&ۤg~5rku4jԦivԱ{n'N9|-<=nޜ %Ѐ^ ^ ^00‡IlUx|7o^s'> P . \<b?6W?M$ƃ(lLeU 39|O\_P0_+Q@X*4lb8(@xj{4qutꃠaxb;^a^Y-]ǿچ?g 790KLc8zJi ^Dd]He … ຸv < - A@@4ioQ_>ڪUy3rfm,[딙_TK5btry4SuƝ[4жuc'@kN^nBJohab^u] l\0=$O FSar 30>js(KYf9&C_ lTY=-jݻ6넒Nd.T7k6;JiIFh^2eA5:JUIA!Bv1k8qgݻuӦ&TJ5`Kލ (ID?ZLB'Y]1.RaBj~~m[X_(WT)Ъ%/Z8)2!N FJ*`p0~{SwJxC@!PASaF$++|src6BR0O]`@~ ôs C8FX-:GAXqLM;N즂!4u D6 |JDj_Q+Ȝ%/?Yͨ TVox:i^;Xxs5?>K0tYVmvW)]^:o̧ xET՟{h;π;w+ ^XfP J4|"wz3|cv AA઼qޝ'ת! L7T pd,엯X[90r2p4)oaTA_!ѳGo\poVV Z2"ALFEw.bY^{7EtʍARL16.Vv42I*a0N~VS&Ēϱo)SM,SBR b x\FxU#eG δ%T5SGr6%Bhn\q2UHBI p4)o, .̳4< vzre0Bz\ }Bw5erRƝ2P@Bwd/"l!-Z[/}[]OY@ &6^ݣg}z߫g]8@B٭~pG>Y:/F/Fd{l)٢Wz ;Ժy˂n׸Ȥ贸)qQ?3T2'. b*erQb6ߣy` l 'E*'?0k\!==PPTc 9h84|C_6z%{ 0e6 CPlI*p0"艡pЂ +@ϑA=l@ |Fir8B, â$Q#͖h`4X$|^\X,S[\g1OA6kknGx dWo] >|N롳\4sùduߓm[B ?:rŧ&oyz8Y۔ {mlřu5"ʽ*D4YUB̩4gJ%MUVXH^>SsEiH@4Jb2aUЎ. >k":~Pzk;G ?9};sZr}WEw> A6$r)-PGGcwN9V *Z~=Vq :2.E䊈{Y Ey$UrE$ְA d`(iԊ94/` O>}IH~K~q̩FPTU ٽ@*PՋX|lEo#IbniҴY5kVmܸyV;Lm,AZ},1lڬŇ2PTT ~@80~/"=K|c Ew N" 2=zș7)V_̱cnjt:ͪWGWJi~!_$mx:ic)K&"[IU씘Dbҵ n6nxoϮZ#N2ۨ)a,uY>wj0*irx2#<VJɓ vz"k)Y!Q|xc.Kp Z)u !aBC\ZF㾋A\(~\116_ Pd=bct5< 0qb5]98>OK<72VD.V 9͟vSZPcK\(#;dӠd)C W L*0LCau` ^6uc^ci-?&ZkSGܨ2>3_t䆖}'.۰W )5K:sS/_qjiny٣^pCg5[:SZ˞{3Ha}dU-}`K܇yI@cWeF.'TbE+&oظ^j;394'o h?Ml9 0l!U E^< ҾSgN;gDY&>ucN9Ko+@N:+zfEm8wk̪)@QxkV/ڽ[U.Unf洹a^NDP~-&`wE>kJORjy{ҥ{Zj73zҥ!jJAQ2`$)SfO>mTY>pee*6qI>&O;cɾ7ly#HՍB_܋^Kׯ7] xn޼FXFI`~'ٱ\ņ T{t6VVQ oI hsqT h2RaDH@[NN`g€a dh<غ::ZUء^hͅF @7;(9u e͡нA!XjY·7cc3t] CJjԵb]vy>C rnME$ ݪuZsGl-I̚pDdl/%Wk(rm< P>„rYD[LySWXU_#L֩}3U{JMpYFA,ܸ_f&woe}tů3_a;ȼr=p@Y˷Kغy'цrs56$mJٱ~)+6:ű!#,q9[_?ʢR|5^ M9$f<>o-m`ߒlDԤrTZk\'3ǽ.X%رi)-YXjӉ800bw&=95MFo?xm;}$H֥Q0 >y3'O;nbV}O?w}R W qծT>+ߚqgZm6ĶYjգcUxԣa_r===ln ) 0C@#[P ՘p<U{$ADCF v8<g<y)}sIC%fZU*74% u84p,Dx:xS\\l$q4PG/$\ QZ?%;w,>ҍLc'I3!K\0Ű<]Cj:1q7[s9m:w/ bGhM-pUveRvW3X)b -,mfFB#^١R9yne rk/x ́d{w^d˗/Ch3grn|F3ahJɹ6@h2AA`:#X Sҥh/L̛͖ { s>QçJ,G#r]_ GFD%$d1aW PK M2xdi)IAS"BW*kպ` -6!s\Rr.=5)E2 <\:# V[HiavjfV)tBe"(%YI I)&4BRVRCZ*cyaJ_L2vHUCRBZAN`Rr*P--ƮE An HFVZptE5闗W(AqoaJ+LLMRӎH 6B({,U jayրĺ8`vRH>z\ S%,˜7Kv;ut˺۴[ey>* V2T}6`Ė#n }V ([3g7mԪMÇ9q=~Iē^(|ӶM:z6kIj}|-)75_Œ@leᮣTУÄ㧝t$Ƅou1\ΜRE[C!Khs/_pD5"PXS\V1A,ϒ49.9PpUK]=Aθ;ڹy֭{z4 "_Pʌx/v{\(=H;i/lyD< wf(^ ,{6 x͚ŴW-33@3IԩUtzU\);o9Q䋨f֛69] 6! _ V#f⇒M::qxӽ!Möw"?$M[.w1#l=t& DtIW?0H,)΂6@ $n^^^ AĭKn}&V6ʺAOMj>02"€ċo/\|l@$/r38ez 9*S_faTqA"(`;the[wg|zVë5ާ >mG1m7_hٗ&/x^ovDu.îzpo;zWnB 3FWe{89Fb$`44f7w>mE,43znSQk*eN~9|T s<3oNAÿ;,}#e_`12)U|QNQZ kS94g%l8`mF!ӰXp$4\FX4!'B>B+ҿ8H͇/AA8 *xv/1!VܽR7**=}!5?I;[^C"2~OJSQuc6 *~TتNBzJ8&AV0UZ$8彻5;5 FEKEqf B cuqL ɿ2q(ߤ{y[V\wۊmgi뻥%=}3R`]]`\2p8-{ Z|58-> 7HJJg dQw3g[SNlmSsng~r0dtJ>ZG Cd" L!S8;Tq}K5DKj$VB1ppS*PH8c8 b0~9,Bʸ~rЇ0AnV$z)HgJ!oڳ\<-t%f}ģ)Ǻ%z ̪.~rm6KXyQ 'YԞf߮57jcw<.Ġϵqmo^?| eT\w8|'.^̘1 AfjHC/^xh{ytYuo5Ŋ~ZdPh0y9OjU{e*=l3bo9F"ߎ揠5M,4_;RC~i'Kt۹5sG>uiJe?eӦ?#)gCW{g.^ɳz1Cjԩ^5nyXN@/vYacDlfAgAOc3J,M8&K(T&ղ7!`o[NLxa1ezVܪy@uߩ3K.E#z uLPƎ3kw1D*dL^\Qf uʲiD:7nҗg✕IIwI 7ka!(nQlyP:tCŹ!;Z@IJMВYK}Ǚt?m'Jz+:U˽ԪVX>JW-`~%"?x &u^jР" нB7gg *!Ըk5$(-L41w}vvycH:,&js Mf?q_!k*tfSYthΫYφ:l؂dԫWӧ9ŔT- ]xRW8'9:%>DJOϾcM?rU4YG+T[q)xlo ZRѸCW0!NP*|C.P8JfzZ<PbX\?oc#C  cv;kxQd'xu98^'? *6q æ|&Kl۬kӠ#G^:fW`y8<  :y~瞃 ߮M{as.^<^}Gп kݥ_|jA: ^ߢ\_q{rDžu w<Ʌ2e= ~mb|1on[r3> MYxJ RhPI(LfiAة'~OFIY޳zYN'q UA>31.ܱ 456t As$z{xQW1`oj"Y_R3 8@r8=WZ4}-lg]+H*l -N"nh{!uT /.CG g`'l 7=ai!9[^mmxYP0-fgf$0;i})z>5n ^Eી? 5DCL̟??""RI䃫ѣGK.N!}n/)Lm[{seZka5^t޵&34M7reٰϢ}=u{߾#ٸ_\oؓƍ:%͏@3sw6~M $o=]F?_'v66&)+~Img7OqҸe]CC6mYFWb5vi.4ѻGs#S@E-9 (<`=hw u>gDqמ$D&ۿV>}r>@$f|)Ak 7:ڏʄ\b ז -I / 3&B;O>M@8f'ʵZH_Mǰ⍰$VH֤سhgHJӛHC=ˏd W6wׁb[%7J b'vo ˘ z}ٓ<úJ}<`am3, 5`%4>Яq̨՘N`*պ{c3c?6^uc g|3bz椑|<#JND 60A'w]pd{qy\('̞P0XW7DSF26j_1a BO?hVu(C̫XgUeV9vT@ŁXgxEׯݿgۮݻ{ߙ;t_ڵ7n }:zxH5u?kEjߛzsמ&sqJ.]W׶k7ze*^?o`Wjtz-Mj_E0b pi$qaB[Ȕ71w 7¾H NִELBHGv!"lUvÌS8ex7|6#KTd])B0.9R('q2LV$9kѲy6:kճr">Ӗ$ֱ3nt?AaA4JbX{G Mp9Ыhϼ zjb>WRrMܦ+p|$lbz`-ѫW,J/W^S|`8N o'^]El,/j«G75Юuñ8^04¸Z Iղ"K;s궳bCU>D{=Rwnmݺy}ش}#q-n>V裇O_?w}ρ5跲+૨sՙuf<Rn RJmTi2{;w-9x,.c RkHjdH:` R9O֖Kj)52=a(D{Z"y$!Ż{!jldh`odY+WLޜ:z@FEǶk-"߼ i1o`y.%*FA Lg+У*&C^pRsp48/*raڃEEť&{/ y}Xdq㦓&M:u*O>Wl. B#rU[w ]<3ݶvi}x5je0KZ|Ê - pkc߲SLB2 9tx-(=RhW Hu]GV@)c8Z>RX ?(/*k7| \9 tlr^yq^;Yqp$@x θX_.AxQ! Xʍ 㞿荻X,<ʰ0C* 3shRgi6,/as*=PyБÏ5s`⊳~ǷmzCGO-]Ƽ'ۿo߼uį0evnӭBmi4#+-fET13'෨1v.9qS՛ 9vgSRӕ*X^IΗ>VCl91W)]re奐g ȝ&51u^^>p 06䈨"I^ks|b[cf=iYV25##|}4!!Ӑ'!O-΄VS׎g!6%ɴ\ vH3F=&{TjSsV. aPy8?:m!cRV돭ԮѽP)Jmel {`H񞭀Ap\>B "LuvJݾҸ,{%($j%{U>UL. z>3%W!oe3*=`Jf ް{>`]t-|~վ>qVtVC0_ /JEHL{DF^@ KO~(d!ַ IE?@KCXvdž6\L%'S-k:`i{N^F+.uۦA_mhۭozv [+P;m n y_ [J#S2HԯgxAx ?UoxNH{Ϧ`/N$^y_Z4 L36@1_fXM# \ hS!> >Pa_ & 80_BC7:XñRcyB]WAB^ ;㏬a-xa4Ըp afWѩ6>]p|}s7]wڎ{o߹kמ- 'O~:޽Nx9K+igh=9}Vcjo17\inG۬xl6g SPQIR;GQQLdCL^ud}, t&$ u-MlF O7O$Y/)%P|1} wg!; 3ٿ,v-jBp ^=+0-QtJ:͇Ba=5& ul" (HI0ݵ1yM \')L)~xE1 ٱU0-Mߣ%%n\;8xmR ^TȧM+|pJ>>vQfMDJj3va >P>DNJ f iN`!(UqQid K*h%2 ݡfvO_Gژ8B+^X-#=ؔd #?0Px^JIsIQ bYYdL 98])հyLV4WkHMὂu( ^T;A^(z.Pd ۭPQ`PN#^_QIRM)iUhN1]FQp!' pzÆuPRᅞȒJ>lD>*ԨɣkY4&k +x?-e^mP2C=~jx]ATr@Jj5_}+*iKi8:L5)_; [E26#W7b0qߎF#ϯ㎨y&FqdU|^]x=OΤPaQ!ƕTaZcy_s-oұn+xhA{ܽd㡍[vص EsR3gBacU|nT~Peܽc d 7܆]2\^wtc-{} +p_۲Z T\)J9[R=4ߒ)8ooz/-%SAFPB{&gK r8 eF٨ENӽ9 FKm&co dȢ(IqoM$ N.6iQg3kGw66:V,++eHXǪzr)UJ\E"Z0Zڶr ԪۘBV vINVBjoNzH>-8RM܀!,do=ȜOH`ݙٞ}30Ao=aO`m"jfUc{|=4Q"Ziܭ%ŽKA9PA:  U/0  ѕ$"\B#30 iֲ3‘...puHyAV*M_QC4!p'ڥ<%o㬲uC$"5kU+iR fKSI$B\KB%ɵpHi4 n4H& Z1Ddžn"8[ Fy_:3h#prJ&+hXqȁˣCW|5@%mzjʍ&U.p(pO^+6rmZۼm-kҶr6ﵣjMmxiܵBggN_tի}ɰq3lwi:h_q7*Ui-9JW܆_nC~dR5qǂ4-2irߩ[lVH$I١1YP9Ƞ_x/P7?JLlETL_`h_iϲ&VXM<=, +=R"In1ȽL"L+Vwz0ypyX$u]*C @2&W(#xaR!1CB@1 ϑPj7;Qe'/[ganKEWYV^. 3$QUm#W g '4̓[7"LV })ct% -f:TԵom{eВ[cYJ J.ǎ {Z%P?/*{j@d#308_n?:Y̭]CG1sbĸeGz)"C{y 1Ua70J"hѢٗO rr{N@~ ~*#4aS\K^~3յuN1dYWU(A:9ټcϗLYx'4 %>h"cflwݣl޸M[[Ŗ<^#&}KLx$#ƫc{,gYa-UQiiFpy,_7+PY]ZblKM `cbV%%W\ n6^ w-6v7Æb͗i86t)(?80%000&[ԙ+s*oڽe k5gv/g<~s/\v CH9uF~f5صm#TduBrF,A L וdE)i IY%6 aqP`g(4jp 6t1bj|%9a3 p^.SGcW1MMI_aX Tj*C ,RS<`JJ󀎓+S0LQ橎ώ F%B-7W!D*>Qu tNAv'` 8tTasFʢ!Se[ uaβd;dUE z͜R̘F?4Ch[oش8!Ȁ}{}۔<*@Vrɮ=[*Qs槉;nHЊY mH5=X*B8@:'rE?/d uo}͑$oGd闰^ry ODtӀћ70N4RAaT`TmYf:6e+۰g;wX`)S':ts/\p*yjzOz-վr]:w_&X:TS~CGl۽rӊ !ǁK pA?U#h\0*))Pq@V'eaK렕{0{sx.0*,Jj*&w0`TTR+u?6!h^1'7Q!UM0arx^C+"dȁfPfmw{țf+R,O[hf KبB"0Ae(HeʎQԋPȷ>䂀* o~0u)!* +@>ZpC Tr{?q܅g/^z80@޴[_3l[mٱvǖ.9mm!jume+BCdede( G] ?! gDD+[jO&{؃TѤy[ X[˦zAdU0p!Q074zI>a"g I4%UJ* =J if4~2K¢<>S8*|9^h(n}EUQs*X)>{YTqӣpQ_U#<XCgX5BCeA HB 2HǎyAH99 4Yh,#L LCUPm̷xve6#lQ&BT$+35##-nX rF~^^ NgZ[N(! =bn]ZԪ^^1cF< "-Ϟ9OC{^ݵaÆK+]+.ւsgtޣZ͖ ޶McpSx]+Acm>HW}&ig1#t޺~}LTl?^UhyHz1|$k؊y3:tѰQ!'?|]s;-0Дкg>ky3{vkވ[6{yΜ3gAl6-kڬAvflVO;j%6fl9oïUNuz {nD)88޸~o{UínfVlF ˓U~{vhYݭQL {CDFKfN;h@rZ쳗{jQafSf$e)Y jII#{<@Eg&Eҹ3GaXg.U5if]9%4q-G kjҤy&^0 ,زkWHH:c Pĵ']nӾ vlݲgӦ]@ۺegx敫n܀ipQaV}YMW5YkQcUQ֕{նC߹sܺ?:%wL%nFcɗ/!ъ0*E9,Iynbtc<\*poKx`l)}\*Mn3ݩ/9]{(JƒC`QqH΋}#'{CIg8?%KĦ;ϲpT3*<ϓo,NJ =_ /~ H}" Ϊia^ l,G rR4{(3T Y44QfbLM.(1;RH^>fX';Aw<…WBz& EB49c2Vn>"tȰ@N5㟚m'[`ᢴU$kr#^d%u^ ׯ!'^5ɺiQֵB@ļR;Ujum ;s6{P.U-8xfO "gиk_p,J.#? 7^O[aռ+ϒYȇ<Ɗ!W,ʨ"ϝWil}2#WwbWjw4pP]vD<0>tP_^t[rr2P.^ @_ D<-[*T)$Ʀ0.A6!CqVQz3IFcif#yfʌ򴦢:mXQJyb 0?U2vdܜ|h #:HQ\9$n ԸwPnivC%հڑXްeÇwZ7ՐϔUv}- Tz.9YH`;,/:o)XLf୳gF&dǶ/&v[[>hRJxzcgG]8ߌzpZ:Em=9v?{k4x~ed<|WN\?2r@O%;.2d*"4 XؽvFIQJ$"~`BP(oi/n߹M`U[&xDy=9 [> iݰܩ_E4oNJ90cF$ g {n8y0YQe|ƈG!Ftu[*3m0ΐN#I'6MlmڼQݶMqBK" g@3^8Tj~w2DƯ/XW1)1l~cAӰ87pF k/]{m6nݶ}5z,_'OxϫW߼yx!\hcUӰBo =vrӾ] >|wŅ"LY&Z 6;ќsx悇)8 iO24OWXEE@zYlDwC.1U2{M4y_ ؿyэs9@9 o(0<<ǫxU xsG9ip@FL&s0{GЇ6l\2A8녰0y=&O,&X8.&X m`DOA+ŊI"w8`E\\B&`‚ԬLge"Xg*Sɥc0jiFBv'-dۆqi"dF[ϖU }'%߃4+N"Uhįor"퇻 ,<*WhNwj$#T?{=ꍺɵ@Ԥr|tZ}B S,+g.هU)J:kP(ǀVpfza G}sva% (Y ӭN WQ |Y1u"5DZ 2(4#/4 qVb:<ȬZiJuIR*gcF#(RvlZ,jgJɄza`\9.CCEq̬oޏU+'9)=Ͱ+Wm:G_ٓD_C (~3hytfLC7z3g"q[Ol:+ŦW3`GGvgؑCAće<" _/~ >2ׯ1>ųgn;vhܸ1Q GMN*__Zl'p8 l,f[JuBBx~FXceTXHNJpA^KR1qRNRB邒7A| BUa\=Kfy wi };YU`7}Jڦ*ۓ$j`GWpf{&JriDGE&G^WR]$aU[zJ)3SSCĤ-2J-S4ѽ'zEJMK#{y3~_1}: ymB3BgڪĨ-GKKs1o(8kP^5H'ҏbN7uk>ՆY8iHC(TIKWmسn nܶas'΂8{ $^zͻAvr)t۴C>.8o޾[xJ 3}M2_I?*: (0( U7kў"^0J׽ia&3ݡR1E%1)7<[H~5m:̼PˊM2to^٫toڹv [7ۼvek׮lرc@8ArH]v[7n޺w7-;N6s….{wo=0}x~Wq/zxX@p$n/޼ysNкڵm`爅B T;W<s _ڵkM"--EjrJ:MMNNJNfldT?G~_fI=`2E7@N6-5w4tQ˕EVF@'%iʸ-dF+Q%ֶ US$3##kk)p 7` "ۜ}I=S[g;08M)jTcK:4mҲ5EW\Itk*4= {͡!+R\9U+Q m Z Q m3bHD`h@"T5\9?8XXAQ)]]Ի)Ip8Z@}\FF-4Q!)d>svb$ I>H^X@ȴ0гR(৔%Zim~,KDUp:[.u-HTW_Dya7T;5KJ w)SӲ_|e+qm2}Ma.Ì+wo8f̄V_aM[o ˖wرG3pWo3{wr?yӗo?}+(SЈZH!T;"/{]]DzoICOj i$!!@H-{yg1rdvvvnf3ϳm@.G&1I%! BVj2X;UfU$$!o Qq呬Aj/5 .oi.j1 íwPLDodȨMWfɧPźEfʪb&]N'6E-x2gir k]IA€*v `l.6\t!ßx.p֑S S[4jsw۫偼h|U^L( }‹onN4s&-n26MߵϤ/f_SO>!SO9n(1b3gB#qvrR)Lhk_-[ղcYnk7K_7m6eΕk/ZxӖSs].l saI"Zm)md>A8ݦ"/\S8ogJxI>V>9!`~J1#4cfmTLKf$NkO*R6rg2WX #a۬}tFxQ+X_^x5x@x‹C&N;[HyKQdx uĽ_OX@IBB;w ZF+/N%M7 gWCC駟߿\B<t <'''OpmKJT=SйrFÕԍWb|_Pku]>bK+Ϲ1OL-0 ?}Υ L.irۓ` 8IܕS71IF>42H:bot>2!! YiR oi_YUr$Fo lSk"*>xF^ x׬!NDQ6Wu|RN~ꃦ Q1#w^Zu]@<ʑ*n&xCCUe30pS.O݊@GR4Sp2!x]U1OO<>xnfޒJܰ ^a 7g`d)I? @- YMl=U G~Hw?/q \I 7&~.qȸc&U1yyIu;OE%`c^-:>ew}3owT:vbȸ'M:mѳf́gl\R VK2w^ h%+VNEY;H׭Ojچ͘om5,J߼ `*\b$'!kx&#rIBUN)*{&qگ)U{*uMj^A SX~>yA*Z/"E3OQSa@1DIt奔ƚ&#5s|Dn =yQ +6?pJFCMiR=1+ ы[J[*?2S jg#x|r04uuY$/+/Gv$~6PGkhq):"ȕ=w' GjE(:6Ebw_;QV@{ fȽ>s^v?}A&9k)cllG={i?!sа!SQhFd3gϛxx1V]t9UcD^xvi@X|K7o1[xj5lhcb WL#3I6UT BZ3o!lTM!b~nB!f,M$:9q''d, %$\&4@ ZU30f6ϊ #VUR/,X27۬o*fn=ʤD:UUqIMS oCȑ<&L\)9DI.I3EְɑISW޲! 1ߤG{Ŝmis.dx!vx^ȡ :@ ` ` dI{E]>'?!0e&F|u&C8n*U08qi!PUlʸ6ІP?` c^6htժ>cEYy./BU5ʓp&~O'X"H4g~RsRnkiW#rB[ŇOwg0Ҍ^#~dFxZj?%sjo-.qm_VդpWǧ5-iC{2ߦ}#+zwV{y\ ոUv6JͯÑq*25c*d@O_r (5sa tF);ƟރXrEZ;Ox| [g˭7_G kf^ڠ-sNT8tQpxDM;oƗtlElF@'[!Zű./=!E-+[ϙ5e-mAH吥peng { \nP8ށll BF2W=zOmzX)G64*)(".g6E黑V~njDF(xV_yU  jSE 5mϿӳ'+*Gsksf| y'b [[w}{=|dpf׼50a:qhC?5XEL$GMe vN^\\q(rb$ nAshXhW]kQDj܎,bb%r HH_hAr`T;S.3- ˘=qD)!M Y)S#cbgMNZؔ!d¥/G*d x[*SJOTrs^G6>sp j,PGIc WZ#$UTRESxfJbT m80w5Uzk7F5i^nj8% jdKR.ra彄u4Êrtl*hM$'C4oق -RCf¦52C K&JEX*n~.Es4~^TzE!IE/{A"q{-mօ/6Z,Ꝑ(c {]/$ț߿w@ d ?$74e켡N!gU,jHӥ2PdbY ; YSy SM$3n(T z$tT(cۡ;8$t4r|d7j 8a g+Buд]QOAI&l2G{z$G hXi3򿤙Q6puH*aSFaV>-nd0C[R/^uE/<%Ő>=;x.aS;z8K%@Ȉ4;"2_A.VBsg`0\%x+cbfΜ;FIIɹH,L>3xY>Gu}*er"'=]NYm) 7zKoڵ <{/<-sВ/|ڱCo]>)|Tvx]g9n͉w5q{{q| {H[j|ӻM{#m\umw4dd5&V[Lz4c_gh{y.syp>[no߮mw׽w݇\K7@Mı8LC"êM5."a" c&Hا@!:.°LuHR* <^#* U2Zu&%Dd EtʑkO,馄49R8m!Moc #4T-mu.g ̪Qqc‹ c=!AQ%,jWݘvX\7I&RͷK:7ɩtj=ϋp?EyZ\8Y+SWa^dO00-PEH4ʰ?eϾg+0/VKɲa6|?ux9sڐ"&*5]m7/ 9xY[a)o<Қ`2b5fO)[xmcW1A^?H,4 YL3 .;SzH94ėĚ>g.ڬ][yAr|78Ħ@Q٧ؑ[txt%U:<Γ{-}cz.oK켢>b\}U꣇IsW:(Zњ9?bs0Մi2#J[0x!4 $P@N!'s#v6ؙ `#çrcB5)DNJotrQ;/x1[ I†bWJ"x?A<~VEEXб@})Ls[x|*<dzX&\}BĬj%-hb҅RMQcM(+%t1U{ĨlJ,_+ZI:zR27  YPMG`t3+NJ)Zw.4t Q$@Mw d'77JH-Y=GBcLJ&Οr ^DN\̉8mş;:# 'O6|$"&NF@ȞSE0 OP>gbL4FGNҷo_znݺ9rܹw _,& ^4}GʳfJ%x :(0l6$FU8>UJjޝ}17)gAդ . ؚ9WEepRjkoޅ5d#;Jl]P x6ѵ_oN{[Z1Ƞ6Y 9Q4xb{M"gV8W+ `bHd1_?(xϕLB@cQVGFE[` ԥKG}DGfp`oxrpMMx&vUc]Pf 00uOz2s ():^hbX㗵Ƅ?}. ~5?|zsqp +0}bܗP}տ8zygA<Ŭv:C@G*;FKw E\,YCEEow/Cj~F#ٰ}<'*ۡ/|Tk/§JqTYGjLxJi^RiDUx„ϑ^BOy+থ1xlf;+ϴn 2H+Ṉ{Qn>V?7T=*ՕmmPvӃ]< H_bwFЩIѰpa $G/f&:~ %%sx9Gpc 3d[ $~k,@d8D8:z׳OWH_H>Ĝ@bdSbu͢L\-1[BJZ͵|@&14' MDCw[=I PaU%Z 1 /V+֜Z)f[8l)3SW$ /d,BRDCG@D E4dpt}ǻ!3Pfϟ6/ګW'p9H"(&9TUAjyN4; 3ңc8kx͏3'0>>SV]wn#lf]aJ ^+Δ!ƧN.\43{[fm{u|AS*Z@Oʺ]tЅ`QK;.`H #ZW8Yx`zH_qiܞ*6#pQڏƬؿ_NDov|MRe͋Zv wY1 @#j~|N8s~- p/F'{$]E$a ;ٻi>׹ /sƃ/*E=8{?ңWTwNOV#+p;! M6JaCXծ1#+,X``ыz2TذHK0joCƦ.o^s% fI.ԫ/ܙQ#m6sk$Uv}FB+I: ? VQJbJ 7 *5⸓pO¦hw}< umtLM tmC\xf A)Cis}RCXJpd|ND'sҗO ;/6kشd5@x:_VgBgZ"7+io~ȋ^|/sDؔ/Y ؼ>vujBҼؗtgxaXŅ>I`B` B\W҇f? bpLu0x%tV=CdeDc /'?iw"EפRASE_L JlUM$@2INћ/#y@,*`yRJapy"ؤ 3RRW$^lBIbzBxq^:I$ (~@9}x…h̘1C„hŊ`WUXX<)+0;~ { OיSa%9'b8/{E!9xզa!!3&^bY FO2-qAB2tnNy 1 qe\RxceE`WeOPT^Htz%q>T4I RfHq,&?p Vx9|Bt[#V@͒jTrTC,-{}XY4׳H"&a *qIcgj ,gj Hj¬ /C*!!5/DI۲. z7'xaxWAߞųcQaBEMMؚ6nCA8ش6I1&Gfjx@F I۹D/ٷФ0,V[7D-Cσ#MSvMSx5v)fk^y o@Ðez^1831zW/yQPuѲ+լLcEjq';`o PƕDѷ:lN)QQL>rjgҝ N)R*F_7c xQϯmf݅j*@;]״1N}30 8dL 1Gwx;70=rB[㺞) 4` /zau1IFA8}@X5e9e4re:ocrࡶ-~gDz"rue?з* 8)eQ JF$E.Z4mÚxv oߚGls9)CBFLaxpNjGnz-Oլ.EFz]Z6@rdE.taĝd9'~}/zG޾uGl+O$*ãZʬlxTauĩmK:rY{x܋qtЉUƪԊoIYMQ!hYkP zt {ܶ} #8U`ʟQ5GuqvA)v~-[Z4E|}aq:Av*e 5Ș~Qũ{CwdA}l<Wа1evzUnE?Z^y^Cm8vB4݆քNKܫ0T/n羓C&DN_,f:&^'̶rv'^R%= m/%|lz|RDi! aN]8t~-E9?׫ptW]1$\ OM [tƵ['$x3-a+bSƆ9)3Hc>SR%A`i{˽yh}\J{"I9HWU<5nEېH?2wOxFaRETsUuxK՚B2 tGDqT7T֋NbKU' w$Z_Ǫ$k+HsE5ytdN`VѦ )^N7=.l-Թ4?!U]{pDD4TFNE\D e(HVÿhR8QJ͒F?D ;}M=wATʨ/vޚ6 qDK8.,xͺ[7ĦF̝?%bТNȐ՜9]}~e\:qh)vbwz5HO sY1Kۚ@PA2M9<-0*y{Et'M7sBA<FqEc:,STOꌀAtE@'\a^(fD[ʾкVd\޹Uo&-W:/Z6_{gȲo{ " DHp,.cmwFѐUI<#{ثdw5Ezf{=cՕ!!  hŀr/F0X as͍lc 1vwWY}Gr,I֬_W}1bZ gBIV(%"wcw^Bt<3,[}c .t$Qd1I8V b,8:Onu{&k٢TO÷?@,Rq*H% T9eTnM5.YE_~ }*ȕܣؾFMC/twb{0 IK 3~zu/IwBj29ԇΝVF{ߟK#̂nkrmn^X>:jD-8f4\[KvݤC3p,䯱Lbdc3ۜ|=՟*i4MIQ%XA?=Uvh4&М ԫf(zƙ5~iҁ_ڣEGE h[kj5[De;"f[8TvpH=5zV˓*|LSԜHDeKn!GD]EҪ#P| hJI7619[÷aXA'A 7"-%&sz`m{nj5wuXG=]@!>1?d;if> kjSJM`.κg(`s Fkf/_c5NJ/yghwa4֚HI[d6kpCABԳ&anzĨ#^hrkpswja1ZݺviܕC{extX>оpG>GLmrU5]Lvo_(IYBTcCTUOh/^jw R8.jU (5)SRWͿ.qVAx5{1FS/?ROv܋{< -+IXL{s Է:]whgQe2-iF10:|ASgt:]e~[{]RcunZtZEŇ7mrhYH| B&m^r5UA!NvXʪpP_37NePo3g|$j\V)eS k]<$)S+I-J~rpC W,֟{]ȩㅤPV^(GFiQ9)lVxke g6X-zGڸS}{ghq>yR)su @ 5VԦ$x 4a%ۥ13jWZO$T$pL>kҠ4D]s_)Pr}yRQ=Y>[ ggd44w #A|g}'IH!{{M.E(x|2Q.9JcVXOyۺ{tm-:F/kt`f?.Q@kY^6e2S2taVx|4XЁӫ-8EXv) mtozg{&oDG =Yag,Z_\As8dE1`^C]_1%->)^I`5/N}F/&g=aq2 >nGdko߫)=> ɀȧ" 6h\ Rğ,BEY2F/ltn|}&ȓEQE(go/ρAչ^[(׋{AP`T,I{/ -%g)qD9265^l,lL+Cclv( s[C/18^̈́V3%xa者{q/kB-_fs0X_[Y9ӹ01?yhTV#Sf]ey3u`J \[nLzcX濝C~ˉMvXJDDh_?hPyk X.Vx*FRij).~' 6 ܊W{ov-̺gP$ v!uP_tRPL-w8 1'rCsqPJYI01xf)bԍS03g=HIg>MҹTrTN\.$ (ƭb'ݛFJd{wPM&xΆt* @փ" k^{/^9RXWi)D,N5k֬ 2L)$oIh֡)E-Bk݄|jMD2pj; @\p+C5 "=E" D6)\u:Q{!gyRgN {/aᩝ 0 ð{0 0^0 0 0 0 0 ð{0 0 r/w_x$܋줸0 0WٛX!;~'x5T.Ç˷sy\&3.7ba*Q +8|{{q|B}{Q:>z/bB69y;3u+3?#e͚5k֬Y_M$' I%Go.-Z䃇7P.w/Jj)18Go߻gVR|j2֬Yf͚ĭY3 Lrw~^{Q>yUs/^|c!ZZ\V'֖&W'Yf͚5kWFeትJ46ˬ/>{owi  \9W_w~G/Kaa _}6 gϞKuJ݋"GŃ}a͚5k֬Y_ Ak988*KGýPPh f͚5k֬&9y(K|ar ?{=+',,,,,,,mGw/),,,,,,a?{Q%^\uaaaaaa@BYÏ^^Yaaaaaaa)܋+ K%JZau/.s/XXXXXXXX뫸/gHIENDB`ovs-2.9.0/Documentation/automake.mk000066400000000000000000000170371324262074100173120ustar00rootroot00000000000000DOC_SOURCE = \ Documentation/group-selection-method-property.txt \ Documentation/_static/logo.png \ Documentation/_static/overview.png \ Documentation/conf.py \ Documentation/index.rst \ Documentation/contents.rst \ Documentation/intro/index.rst \ Documentation/intro/what-is-ovs.rst \ Documentation/intro/why-ovs.rst \ Documentation/intro/install/index.rst \ Documentation/intro/install/bash-completion.rst \ Documentation/intro/install/debian.rst \ Documentation/intro/install/documentation.rst \ Documentation/intro/install/distributions.rst \ Documentation/intro/install/dpdk.rst \ Documentation/intro/install/fedora.rst \ Documentation/intro/install/general.rst \ Documentation/intro/install/netbsd.rst \ Documentation/intro/install/ovn-upgrades.rst \ Documentation/intro/install/rhel.rst \ Documentation/intro/install/userspace.rst \ Documentation/intro/install/windows.rst \ Documentation/intro/install/xenserver.rst \ Documentation/tutorials/index.rst \ Documentation/tutorials/faucet.rst \ Documentation/tutorials/ovs-advanced.rst \ Documentation/tutorials/ovn-openstack.rst \ Documentation/tutorials/ovn-sandbox.rst \ Documentation/topics/index.rst \ Documentation/topics/bonding.rst \ Documentation/topics/idl-compound-indexes.rst \ Documentation/topics/datapath.rst \ Documentation/topics/design.rst \ Documentation/topics/dpdk/index.rst \ Documentation/topics/dpdk/ring.rst \ Documentation/topics/dpdk/vhost-user.rst \ Documentation/topics/testing.rst \ Documentation/topics/high-availability.rst \ Documentation/topics/integration.rst \ Documentation/topics/language-bindings.rst \ Documentation/topics/openflow.rst \ Documentation/topics/ovn-news-2.8.rst \ Documentation/topics/ovsdb-replication.rst \ Documentation/topics/porting.rst \ Documentation/topics/role-based-access-control.rst \ Documentation/topics/tracing.rst \ Documentation/topics/windows.rst \ Documentation/howto/index.rst \ Documentation/howto/docker.rst \ Documentation/howto/dpdk.rst \ Documentation/howto/firewalld.rst \ Documentation/howto/kvm.rst \ Documentation/howto/libvirt.rst \ Documentation/howto/selinux.rst \ Documentation/howto/ssl.rst \ Documentation/howto/lisp.rst \ Documentation/howto/openstack-containers.rst \ Documentation/howto/qos.png \ Documentation/howto/qos.rst \ Documentation/howto/sflow.png \ Documentation/howto/sflow.rst \ Documentation/howto/tunneling.png \ Documentation/howto/tunneling.rst \ Documentation/howto/userspace-tunneling.rst \ Documentation/howto/vlan.png \ Documentation/howto/vlan.rst \ Documentation/howto/vtep.rst \ Documentation/ref/index.rst \ Documentation/faq/index.rst \ Documentation/faq/configuration.rst \ Documentation/faq/contributing.rst \ Documentation/faq/design.rst \ Documentation/faq/general.rst \ Documentation/faq/issues.rst \ Documentation/faq/openflow.rst \ Documentation/faq/qos.rst \ Documentation/faq/releases.rst \ Documentation/faq/terminology.rst \ Documentation/faq/vlan.rst \ Documentation/faq/vxlan.rst \ Documentation/internals/index.rst \ Documentation/internals/authors.rst \ Documentation/internals/bugs.rst \ Documentation/internals/charter.rst \ Documentation/internals/committer-emeritus-status.rst \ Documentation/internals/committer-grant-revocation.rst \ Documentation/internals/committer-responsibilities.rst \ Documentation/internals/documentation.rst \ Documentation/internals/mailing-lists.rst \ Documentation/internals/maintainers.rst \ Documentation/internals/patchwork.rst \ Documentation/internals/release-process.rst \ Documentation/internals/security.rst \ Documentation/internals/contributing/index.rst \ Documentation/internals/contributing/backporting-patches.rst \ Documentation/internals/contributing/coding-style.rst \ Documentation/internals/contributing/coding-style-windows.rst \ Documentation/internals/contributing/documentation-style.rst \ Documentation/internals/contributing/libopenvswitch-abi.rst \ Documentation/internals/contributing/submitting-patches.rst \ Documentation/requirements.txt \ $(addprefix Documentation/ref/,$(RST_MANPAGES)) FLAKE8_PYFILES += Documentation/conf.py EXTRA_DIST += $(DOC_SOURCE) # You can set these variables from the command line. SPHINXOPTS = SPHINXBUILD = sphinx-build SPHINXSRCDIR = $(srcdir)/Documentation SPHINXBUILDDIR = $(builddir)/Documentation/_build # Internal variables. ALLSPHINXOPTS = -W -n -d $(SPHINXBUILDDIR)/doctrees $(SPHINXOPTS) $(SPHINXSRCDIR) sphinx_verbose = $(sphinx_verbose_@AM_V@) sphinx_verbose_ = $(sphinx_verbose_@AM_DEFAULT_V@) sphinx_verbose_0 = -q if HAVE_SPHINX docs-check: $(DOC_SOURCE) $(AM_V_GEN)$(SPHINXBUILD) $(sphinx_verbose) -b html $(ALLSPHINXOPTS) $(SPHINXBUILDDIR)/html && touch $@ $(AM_V_GEN)$(SPHINXBUILD) $(sphinx_verbose) -b man $(ALLSPHINXOPTS) $(SPHINXBUILDDIR)/man && touch $@ ALL_LOCAL += docs-check CLEANFILES += docs-check check-docs: $(SPHINXBUILD) -b linkcheck $(ALLSPHINXOPTS) $(SPHINXBUILDDIR)/linkcheck clean-docs: rm -rf $(SPHINXBUILDDIR) rm -f docs-check CLEAN_LOCAL += clean-docs endif .PHONY: check-docs .PHONY: clean-docs # Installing manpages based on rST. # # The docs-check target converts the rST files listed in RST_MANPAGES # into nroff manpages in Documentation/_build/man. The easiest way to # get these installed by "make install" is to write our own helper # rules. # rST formatted manpages under Documentation/ref. RST_MANPAGES = \ ovs-test.8.rst \ ovs-vlan-test.8.rst \ ovsdb-server.7.rst \ ovsdb.5.rst \ ovsdb.7.rst # The GNU standards say that these variables should control # installation directories for manpages in each section. Automake # will define them for us only if it sees that a manpage in the # appropriate section is to be installed through its built-in feature. # Since we're working independently, for best safety, we need to # define them ourselves. man1dir = $(mandir)/man1 man2dir = $(mandir)/man2 man3dir = $(mandir)/man3 man4dir = $(mandir)/man4 man5dir = $(mandir)/man5 man6dir = $(mandir)/man6 man7dir = $(mandir)/man7 man8dir = $(mandir)/man8 man9dir = $(mandir)/man9 # Set a shell variable for each manpage directory. set_mandirs = \ man1dir='$(man1dir)' \ man2dir='$(man2dir)' \ man3dir='$(man3dir)' \ man4dir='$(man4dir)' \ man5dir='$(man5dir)' \ man6dir='$(man6dir)' \ man7dir='$(man7dir)' \ man8dir='$(man8dir)' \ man9dir='$(man9dir)' # Given an $rst of "ovs-vlan-test.8.rst", sets $stem to # "ovs-vlan-test", $section to "8", and $mandir to $man8dir. extract_stem_and_section = \ stem=`echo "$$rst" | sed -n 's/^\(.*\)\.\([0-9]\).rst$$/\1/p'`; \ section=`echo "$$rst" | sed -n 's/^\(.*\)\.\([0-9]\).rst$$/\2/p'`; \ test -n "$$section" || { echo "$$rst: cannot infer manpage section from filename" 2>&1; continue; }; \ eval "mandir=\$$man$${section}dir"; \ test -n "$$mandir" || { echo "unknown directory for manpage section $$section"; continue; } INSTALL_DATA_LOCAL += install-man-rst if HAVE_SPHINX install-man-rst: docs-check @$(set_mandirs); \ for rst in $(RST_MANPAGES); do \ $(extract_stem_and_section); \ echo " $(MKDIR_P) '$(DESTDIR)'\"$$mandir\""; \ $(MKDIR_P) '$(DESTDIR)'"$$mandir"; \ echo " $(INSTALL_DATA) $(SPHINXBUILDDIR)/man/$$stem.$$section '$(DESTDIR)'\"$$mandir/$$stem.$$section\""; \ $(INSTALL_DATA) $(SPHINXBUILDDIR)/man/$$stem.$$section '$(DESTDIR)'"$$mandir/$$stem.$$section"; \ done else install-man-rst: @: endif UNINSTALL_LOCAL += uninstall-man-rst uninstall-man-rst: @$(set_mandirs); \ for rst in $(RST_MANPAGES); do \ $(extract_stem_and_section); \ echo "rm -f '$(DESTDIR)'\"$$mandir/$$stem.$$section\""; \ rm -f '$(DESTDIR)'"$$mandir/$$stem.$$section"; \ done ovs-2.9.0/Documentation/conf.py000066400000000000000000000104411324262074100164420ustar00rootroot00000000000000# -*- coding: utf-8 -*- # # Open vSwitch documentation build configuration file, created by # sphinx-quickstart on Fri Sep 30 09:57:36 2016. # # This file is execfile()d with the current directory set to its # containing dir. # # Note that not all possible configuration values are present in this # autogenerated file. # # All configuration values have a default; values that are commented out # serve to show the default. import string import sys try: import ovs_sphinx_theme use_ovs_theme = True except ImportError: print("Cannot find 'ovs-sphinx-theme' package. " "Falling back to default theme.") use_ovs_theme = False # -- General configuration ------------------------------------------------ # If your documentation needs a minimal Sphinx version, state it here. # needs_sphinx = '1.1' # Add any Sphinx extension module names here, as strings. They can be # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom # ones. extensions = [] # Add any paths that contain templates here, relative to this directory. templates_path = ['_templates'] # The suffix(es) of source filenames. # You can specify multiple suffix as a list of string: # # source_suffix = ['.rst', '.md'] source_suffix = '.rst' # The master toctree document. master_doc = 'contents' # General information about the project. project = u'Open vSwitch' copyright = u'2016, The Open vSwitch Development Community' author = u'The Open vSwitch Development Community' # The version info for the project you're documenting, acts as replacement for # |version| and |release|, also used in various other places throughout the # built documents. # # The full version, including alpha/beta/rc tags. release = None filename = "../configure.ac" with open(filename, 'rU') as f: for line in f: if 'AC_INIT' in line: # Parse "AC_INIT(openvswitch, 2.7.90, bugs@openvswitch.org)": release = line.split(',')[1].strip(string.whitespace + '[]') break if release is None: sys.stderr.write('%s: failed to determine Open vSwitch version\n' % filename) sys.exit(1) # The short X.Y version. # # However, it's important to know the difference between, e.g., 2.7 # and 2.7.90, which can be very different versions (2.7.90 may be much # closer to 2.8 than to 2.7), so check for that. version = release if '.90' in release else '.'.join(release.split('.')[0:2]) # List of patterns, relative to source directory, that match files and # directories to ignore when looking for source files. # This patterns also effect to html_static_path and html_extra_path exclude_patterns = ['_build', 'Thumbs.db', '.DS_Store'] # If true, check the validity of #anchors in links. linkcheck_anchors = False # -- Options for HTML output ---------------------------------------------- # The theme to use for HTML and HTML Help pages. See the documentation for # a list of builtin themes. # if use_ovs_theme: html_theme = 'ovs' # Add any paths that contain custom themes here, relative to this directory. if use_ovs_theme: html_theme_path = [ovs_sphinx_theme.get_theme_dir()] else: html_theme_path = [] # The name of an image file (relative to this directory) to place at the top # of the sidebar. # html_logo = '_static/logo.png' # Add any paths that contain custom static files (such as style sheets) here, # relative to this directory. They are copied after the builtin static files, # so a file named "default.css" will overwrite the builtin "default.css". html_static_path = ['_static'] # -- Options for manual page output --------------------------------------- # One entry per manual page. List of tuples # (source start file, name, description, authors, manual section). _man_pages = [ ('ovs-test.8', u'Check Linux drivers for performance, vlan and L3 tunneling problems'), ('ovs-vlan-test.8', u'Check Linux drivers for problems with vlan traffic'), ('ovsdb-server.7', u'Open vSwitch Database Server Protocol'), ('ovsdb.5', u'Open vSwitch Database (File Formats)'), ('ovsdb.7', u'Open vSwitch Database (Overview)'), ] # Generate list of (path, name, description, [author, ...], section) man_pages = [ ('ref/%s' % file_name, file_name.split('.', 1)[0], description, [author], file_name.split('.', 1)[1]) for file_name, description in _man_pages] ovs-2.9.0/Documentation/contents.rst000066400000000000000000000022141324262074100175310ustar00rootroot00000000000000.. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. Convention for heading levels in Open vSwitch documentation: ======= Heading 0 (reserved for the title in a document) ------- Heading 1 ~~~~~~~ Heading 2 +++++++ Heading 3 ''''''' Heading 4 Avoid deeper levels because they do not render well. =================================== Open vSwitch Documentation Contents =================================== .. toctree:: :maxdepth: 3 index .. toctree:: :maxdepth: 3 intro/index tutorials/index topics/index howto/index ref/index faq/index internals/index ovs-2.9.0/Documentation/faq/000077500000000000000000000000001324262074100157125ustar00rootroot00000000000000ovs-2.9.0/Documentation/faq/configuration.rst000066400000000000000000000232621324262074100213200ustar00rootroot00000000000000.. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. Convention for heading levels in Open vSwitch documentation: ======= Heading 0 (reserved for the title in a document) ------- Heading 1 ~~~~~~~ Heading 2 +++++++ Heading 3 ''''''' Heading 4 Avoid deeper levels because they do not render well. =================== Basic Configuration =================== Q: How do I configure a port as an access port? A. Add ``tag=VLAN`` to your ``ovs-vsctl add-port`` command. For example, the following commands configure br0 with eth0 as a trunk port (the default) and tap0 as an access port for VLAN 9: :: $ ovs-vsctl add-br br0 $ ovs-vsctl add-port br0 eth0 $ ovs-vsctl add-port br0 tap0 tag=9 If you want to configure an already added port as an access port, use ``ovs-vsctl set``, e.g.: :: $ ovs-vsctl set port tap0 tag=9 Q: How do I configure a port as a SPAN port, that is, enable mirroring of all traffic to that port? A. The following commands configure br0 with eth0 and tap0 as trunk ports. All traffic coming in or going out on eth0 or tap0 is also mirrored to tap1; any traffic arriving on tap1 is dropped: :: $ ovs-vsctl add-br br0 $ ovs-vsctl add-port br0 eth0 $ ovs-vsctl add-port br0 tap0 $ ovs-vsctl add-port br0 tap1 \ -- --id=@p get port tap1 \ -- --id=@m create mirror name=m0 select-all=true output-port=@p \ -- set bridge br0 mirrors=@m To later disable mirroring, run: :: $ ovs-vsctl clear bridge br0 mirrors Q: Does Open vSwitch support configuring a port in promiscuous mode? A: Yes. How you configure it depends on what you mean by "promiscuous mode": - Conventionally, "promiscuous mode" is a feature of a network interface card. Ordinarily, a NIC passes to the CPU only the packets actually destined to its host machine. It discards the rest to avoid wasting memory and CPU cycles. When promiscuous mode is enabled, however, it passes every packet to the CPU. On an old-style shared-media or hub-based network, this allows the host to spy on all packets on the network. But in the switched networks that are almost everywhere these days, promiscuous mode doesn't have much effect, because few packets not destined to a host are delivered to the host's NIC. This form of promiscuous mode is configured in the guest OS of the VMs on your bridge, e.g. with "ip link set promisc". - The VMware vSwitch uses a different definition of "promiscuous mode". When you configure promiscuous mode on a VMware vNIC, the vSwitch sends a copy of every packet received by the vSwitch to that vNIC. That has a much bigger effect than just enabling promiscuous mode in a guest OS. Rather than getting a few stray packets for which the switch does not yet know the correct destination, the vNIC gets every packet. The effect is similar to replacing the vSwitch by a virtual hub. This "promiscuous mode" is what switches normally call "port mirroring" or "SPAN". For information on how to configure SPAN, see "How do I configure a port as a SPAN port, that is, enable mirroring of all traffic to that port?" Q: How do I configure a DPDK port as an access port? A: Firstly, you must have a DPDK-enabled version of Open vSwitch. If your version is DPDK-enabled it will support the other-config:dpdk-init configuration in the database and will display lines with "EAL:..." during startup when other_config:dpdk-init is set to 'true'. Secondly, when adding a DPDK port, unlike a system port, the type for the interface and valid dpdk-devargs must be specified. For example:: $ ovs-vsctl add-br br0 $ ovs-vsctl add-port br0 myportname -- set Interface myportname \ type=dpdk options:dpdk-devargs=0000:06:00.0 Refer to :doc:`/intro/install/dpdk` for more information on enabling and using DPDK with Open vSwitch. Q: How do I configure a VLAN as an RSPAN VLAN, that is, enable mirroring of all traffic to that VLAN? A: The following commands configure br0 with eth0 as a trunk port and tap0 as an access port for VLAN 10. All traffic coming in or going out on tap0, as well as traffic coming in or going out on eth0 in VLAN 10, is also mirrored to VLAN 15 on eth0. The original tag for VLAN 10, in cases where one is present, is dropped as part of mirroring: :: $ ovs-vsctl add-br br0 $ ovs-vsctl add-port br0 eth0 $ ovs-vsctl add-port br0 tap0 tag=10 $ ovs-vsctl \ -- --id=@m create mirror name=m0 select-all=true select-vlan=10 \ output-vlan=15 \ -- set bridge br0 mirrors=@m To later disable mirroring, run: :: $ ovs-vsctl clear bridge br0 mirrors Mirroring to a VLAN can disrupt a network that contains unmanaged switches. See ovs-vswitchd.conf.db(5) for details. Mirroring to a GRE tunnel has fewer caveats than mirroring to a VLAN and should generally be preferred. Q: Can I mirror more than one input VLAN to an RSPAN VLAN? A: Yes, but mirroring to a VLAN strips the original VLAN tag in favor of the specified output-vlan. This loss of information may make the mirrored traffic too hard to interpret. To mirror multiple VLANs, use the commands above, but specify a comma-separated list of VLANs as the value for select-vlan. To mirror every VLAN, use the commands above, but omit select-vlan and its value entirely. When a packet arrives on a VLAN that is used as a mirror output VLAN, the mirror is disregarded. Instead, in standalone mode, OVS floods the packet across all the ports for which the mirror output VLAN is configured. (If an OpenFlow controller is in use, then it can override this behavior through the flow table.) If OVS is used as an intermediate switch, rather than an edge switch, this ensures that the RSPAN traffic is distributed through the network. Mirroring to a VLAN can disrupt a network that contains unmanaged switches. See ovs-vswitchd.conf.db(5) for details. Mirroring to a GRE tunnel has fewer caveats than mirroring to a VLAN and should generally be preferred. Q: How do I configure mirroring of all traffic to a GRE tunnel? A: The following commands configure br0 with eth0 and tap0 as trunk ports. All traffic coming in or going out on eth0 or tap0 is also mirrored to gre0, a GRE tunnel to the remote host 192.168.1.10; any traffic arriving on gre0 is dropped:: $ ovs-vsctl add-br br0 $ ovs-vsctl add-port br0 eth0 $ ovs-vsctl add-port br0 tap0 $ ovs-vsctl add-port br0 gre0 \ -- set interface gre0 type=gre options:remote_ip=192.168.1.10 \ -- --id=@p get port gre0 \ -- --id=@m create mirror name=m0 select-all=true output-port=@p \ -- set bridge br0 mirrors=@m To later disable mirroring and destroy the GRE tunnel:: $ ovs-vsctl clear bridge br0 mirrors $ ovs-vsctl del-port br0 gre0 Q: Does Open vSwitch support ERSPAN? A: No. As an alternative, Open vSwitch supports mirroring to a GRE tunnel (see above). Q: How do I connect two bridges? A: First, why do you want to do this? Two connected bridges are not much different from a single bridge, so you might as well just have a single bridge with all your ports on it. If you still want to connect two bridges, you can use a pair of patch ports. The following example creates bridges br0 and br1, adds eth0 and tap0 to br0, adds tap1 to br1, and then connects br0 and br1 with a pair of patch ports. :: $ ovs-vsctl add-br br0 $ ovs-vsctl add-port br0 eth0 $ ovs-vsctl add-port br0 tap0 $ ovs-vsctl add-br br1 $ ovs-vsctl add-port br1 tap1 $ ovs-vsctl \ -- add-port br0 patch0 \ -- set interface patch0 type=patch options:peer=patch1 \ -- add-port br1 patch1 \ -- set interface patch1 type=patch options:peer=patch0 Bridges connected with patch ports are much like a single bridge. For instance, if the example above also added eth1 to br1, and both eth0 and eth1 happened to be connected to the same next-hop switch, then you could loop your network just as you would if you added eth0 and eth1 to the same bridge (see the "Configuration Problems" section below for more information). If you are using Open vSwitch 1.9 or an earlier version, then you need to be using the kernel module bundled with Open vSwitch rather than the one that is integrated into Linux 3.3 and later, because Open vSwitch 1.9 and earlier versions need kernel support for patch ports. This also means that in Open vSwitch 1.9 and earlier, patch ports will not work with the userspace datapath, only with the kernel module. Q: How do I configure a bridge without an OpenFlow local port? (Local port in the sense of OFPP_LOCAL) A: Open vSwitch does not support such a configuration. Bridges always have their local ports. ovs-2.9.0/Documentation/faq/contributing.rst000066400000000000000000000114421324262074100211550ustar00rootroot00000000000000.. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. Convention for heading levels in Open vSwitch documentation: ======= Heading 0 (reserved for the title in a document) ------- Heading 1 ~~~~~~~ Heading 2 +++++++ Heading 3 ''''''' Heading 4 Avoid deeper levels because they do not render well. =========== Development =========== Q: How do I implement a new OpenFlow message? A: Add your new message to ``enum ofpraw`` and ``enum ofptype`` in ``include/openvswitch/ofp-msgs.h``, following the existing pattern. Then recompile and fix all of the new warnings, implementing new functionality for the new message as needed. (If you configure with ``--enable-Werror``, as described in :doc:`/intro/install/general`, then it is impossible to miss any warnings.) To add an OpenFlow vendor extension message (aka experimenter message) for a vendor that doesn't yet have any extension messages, you will also need to edit ``build-aux/extract-ofp-msgs`` and at least ``ofphdrs_decode()`` and ``ofpraw_put__()`` in ``lib/ofp-msgs.c``. OpenFlow doesn't standardize vendor extensions very well, so it's hard to make the process simpler than that. (If you have a choice of how to design your vendor extension messages, it will be easier if you make them resemble the ONF and OVS extension messages.) Q: How do I add support for a new field or header? A: Add new members for your field to ``struct flow`` in ``include/openvswitch/flow.h``, and add new enumerations for your new field to ``enum mf_field_id`` in ``include/openvswitch/meta-flow.h``, following the existing pattern. If the field uses a new OXM class, add it to OXM_CLASSES in ``build-aux/extract-ofp-fields``. Also, add support to ``miniflow_extract()`` in ``lib/flow.c`` for extracting your new field from a packet into struct miniflow, and to ``nx_put_raw()`` in ``lib/nx-match.c`` to output your new field in OXM matches. Then recompile and fix all of the new warnings, implementing new functionality for the new field or header as needed. (If you configure with ``--enable-Werror``, as described in :doc:`/intro/install/general`, then it is impossible to miss any warnings.) If you want kernel datapath support for your new field, you also need to modify the kernel module for the operating systems you are interested in. This isn't mandatory, since fields understood only by userspace work too (with a performance penalty), so it's reasonable to start development without it. If you implement kernel module support for Linux, then the Linux kernel "netdev" mailing list is the place to submit that support first; please read up on the Linux kernel development process separately. The Windows datapath kernel module support, on the other hand, is maintained within the OVS tree, so patches for that can go directly to ovs-dev. Q: How do I add support for a new OpenFlow action? A: Add your new action to ``enum ofp_raw_action_type`` in ``lib/ofp-actions.c``, following the existing pattern. Then recompile and fix all of the new warnings, implementing new functionality for the new action as needed. (If you configure with ``--enable-Werror``, as described in the :doc:`/intro/install/general`, then it is impossible to miss any warnings.) If you need to add an OpenFlow vendor extension action for a vendor that doesn't yet have any extension actions, then you will also need to add the vendor to ``vendor_map`` in ``build-aux/extract-ofp-actions``. Also, you will need to add support for the vendor to ``ofpact_decode_raw()`` and ``ofpact_put_raw()`` in ``lib/ofp-actions.c``. (If you have a choice of how to design your vendor extension actions, it will be easier if you make them resemble the ONF and OVS extension actions.) Q: How do I add support for a new OpenFlow error message? A: Add your new error to ``enum ofperr`` in ``include/openvswitch/ofp-errors.h``. Read the large comment at the top of the file for details. If you need to add an OpenFlow vendor extension error for a vendor that doesn't yet have any, first add the vendor ID to the ``_VENDOR_ID`` list in ``include/openflow/openflow-common.h``. ovs-2.9.0/Documentation/faq/design.rst000066400000000000000000000155311324262074100177220ustar00rootroot00000000000000.. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. Convention for heading levels in Open vSwitch documentation: ======= Heading 0 (reserved for the title in a document) ------- Heading 1 ~~~~~~~ Heading 2 +++++++ Heading 3 ''''''' Heading 4 Avoid deeper levels because they do not render well. ====================== Implementation Details ====================== Q: I hear OVS has a couple of kinds of flows. Can you tell me about them? A: Open vSwitch uses different kinds of flows for different purposes: - OpenFlow flows are the most important kind of flow. OpenFlow controllers use these flows to define a switch's policy. OpenFlow flows support wildcards, priorities, and multiple tables. When in-band control is in use, Open vSwitch sets up a few "hidden" flows, with priority higher than a controller or the user can configure, that are not visible via OpenFlow. (See the "Controller" section of the FAQ for more information about hidden flows.) - The Open vSwitch software switch implementation uses a second kind of flow internally. These flows, called "datapath" or "kernel" flows, do not support priorities and comprise only a single table, which makes them suitable for caching. (Like OpenFlow flows, datapath flows do support wildcarding, in Open vSwitch 1.11 and later.) OpenFlow flows and datapath flows also support different actions and number ports differently. Datapath flows are an implementation detail that is subject to change in future versions of Open vSwitch. Even with the current version of Open vSwitch, hardware switch implementations do not necessarily use this architecture. Users and controllers directly control only the OpenFlow flow table. Open vSwitch manages the datapath flow table itself, so users should not normally be concerned with it. Q: Why are there so many different ways to dump flows? A: Open vSwitch has two kinds of flows (see the previous question), so it has commands with different purposes for dumping each kind of flow: - ``ovs-ofctl dump-flows
`` dumps OpenFlow flows, excluding hidden flows. This is the most commonly useful form of flow dump. (Unlike the other commands, this should work with any OpenFlow switch, not just Open vSwitch.) - ``ovs-appctl bridge/dump-flows
`` dumps OpenFlow flows, including hidden flows. This is occasionally useful for troubleshooting suspected issues with in-band control. - ``ovs-dpctl dump-flows [dp]`` dumps the datapath flow table entries for a Linux kernel-based datapath. In Open vSwitch 1.10 and later, ovs-vswitchd merges multiple switches into a single datapath, so it will show all the flows on all your kernel-based switches. This command can occasionally be useful for debugging. - ``ovs-appctl dpif/dump-flows
``, new in Open vSwitch 1.10, dumps datapath flows for only the specified bridge, regardless of the type. Q: How does multicast snooping works with VLANs? A: Open vSwitch maintains snooping tables for each VLAN. Q: Can OVS populate the kernel flow table in advance instead of in reaction to packets? A: No. There are several reasons: - Kernel flows are not as sophisticated as OpenFlow flows, which means that some OpenFlow policies could require a large number of kernel flows. The "conjunctive match" feature is an extreme example: the number of kernel flows it requires is the product of the number of flows in each dimension. - With multiple OpenFlow flow tables and simple sets of actions, the number of kernel flows required can be as large as the product of the number of flows in each dimension. With more sophisticated actions, the number of kernel flows could be even larger. - Open vSwitch is designed so that any version of OVS userspace interoperates with any version of the OVS kernel module. This forward and backward compatibility requires that userspace observe how the kernel module parses received packets. This is only possible in a straightforward way when userspace adds kernel flows in reaction to received packets. For more relevant information on the architecture of Open vSwitch, please read "The Design and Implementation of Open vSwitch", published in USENIX NSDI 2015. Q: How many packets does OVS buffer? A: Open vSwitch fast path packet processing uses a "run to completion" model in which every packet is completely handled in a single pass. Therefore, in the common case where a packet just passes through the fast path, Open vSwitch does not buffer packets itself. The operating system and the network drivers involved in receiving and later in transmitting the packet do often include buffering. Open vSwitch is only a middleman between these and does not have direct access or influence over their buffers. Outside the common case, Open vSwitch does sometimes buffer packets. When the OVS fast path processes a packet that does not match any of the flows in its megaflow cache, it passes that packet to the Open vSwitch slow path. This procedure queues a copy of the packet to the Open vSwitch userspace which processes it and, if necessary, passes it back to the kernel module. Queuing the packet to userspace as part of this process involves buffering. (Going the opposite direction does not, because the kernel actually processes the request synchronously.) A few other exceptional cases also queue packets to userspace for processing; most of these are due to OpenFlow actions that the fast path cannot handle and that must therefore be handled by the slow path instead. OpenFlow also has a concept of packet buffering. When an OpenFlow switch sends a packet to a controller, it may opt to retain a copy of the packet in an OpenFlow "packet buffer". Later, if the controller wants to tell the switch to forward a copy of that packet, it can refer to the packet through its assigned buffer, instead of sending the whole packet back to the switch, thereby saving bandwidth in the OpenFlow control channel. Before Open vSwitch 2.7, OVS implemented such buffering; Open vSwitch 2.7 and later do not. ovs-2.9.0/Documentation/faq/general.rst000066400000000000000000000145561324262074100200740ustar00rootroot00000000000000.. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. Convention for heading levels in Open vSwitch documentation: ======= Heading 0 (reserved for the title in a document) ------- Heading 1 ~~~~~~~ Heading 2 +++++++ Heading 3 ''''''' Heading 4 Avoid deeper levels because they do not render well. ======= General ======= Q: What is Open vSwitch? A: Open vSwitch is a production quality open source software switch designed to be used as a vswitch in virtualized server environments. A vswitch forwards traffic between different VMs on the same physical host and also forwards traffic between VMs and the physical network. Open vSwitch supports standard management interfaces (e.g. sFlow, NetFlow, IPFIX, RSPAN, CLI), and is open to programmatic extension and control using OpenFlow and the OVSDB management protocol. Open vSwitch as designed to be compatible with modern switching chipsets. This means that it can be ported to existing high-fanout switches allowing the same flexible control of the physical infrastructure as the virtual infrastructure. It also means that Open vSwitch will be able to take advantage of on-NIC switching chipsets as their functionality matures. Q: What virtualization platforms can use Open vSwitch? A: Open vSwitch can currently run on any Linux-based virtualization platform (kernel 3.10 and newer), including: KVM, VirtualBox, Xen, Xen Cloud Platform, XenServer. As of Linux 3.3 it is part of the mainline kernel. The bulk of the code is written in platform- independent C and is easily ported to other environments. We welcome inquires about integrating Open vSwitch with other virtualization platforms. Q: How can I try Open vSwitch? A: The Open vSwitch source code can be built on a Linux system. You can build and experiment with Open vSwitch on any Linux machine. Packages for various Linux distributions are available on many platforms, including: Debian, Ubuntu, Fedora. You may also download and run a virtualization platform that already has Open vSwitch integrated. For example, download a recent ISO for XenServer or Xen Cloud Platform. Be aware that the version integrated with a particular platform may not be the most recent Open vSwitch release. Q: Does Open vSwitch only work on Linux? A: No, Open vSwitch has been ported to a number of different operating systems and hardware platforms. Most of the development work occurs on Linux, but the code should be portable to any POSIX system. We've seen Open vSwitch ported to a number of different platforms, including FreeBSD, Windows, and even non-POSIX embedded systems. By definition, the Open vSwitch Linux kernel module only works on Linux and will provide the highest performance. However, a userspace datapath is available that should be very portable. Q: What's involved with porting Open vSwitch to a new platform or switching ASIC? A: :doc:`/topics/porting` describes how one would go about porting Open vSwitch to a new operating system or hardware platform. Q: Why would I use Open vSwitch instead of the Linux bridge? A: Open vSwitch is specially designed to make it easier to manage VM network configuration and monitor state spread across many physical hosts in dynamic virtualized environments. Refer to :doc:`/intro/why-ovs` for a more detailed description of how Open vSwitch relates to the Linux Bridge. Q: How is Open vSwitch related to distributed virtual switches like the VMware vNetwork distributed switch or the Cisco Nexus 1000V? A: Distributed vswitch applications (e.g., VMware vNetwork distributed switch, Cisco Nexus 1000V) provide a centralized way to configure and monitor the network state of VMs that are spread across many physical hosts. Open vSwitch is not a distributed vswitch itself, rather it runs on each physical host and supports remote management in a way that makes it easier for developers of virtualization/cloud management platforms to offer distributed vswitch capabilities. To aid in distribution, Open vSwitch provides two open protocols that are specially designed for remote management in virtualized network environments: OpenFlow, which exposes flow-based forwarding state, and the OVSDB management protocol, which exposes switch port state. In addition to the switch implementation itself, Open vSwitch includes tools (ovs-ofctl, ovs-vsctl) that developers can script and extend to provide distributed vswitch capabilities that are closely integrated with their virtualization management platform. Q: Why doesn't Open vSwitch support distribution? A: Open vSwitch is intended to be a useful component for building flexible network infrastructure. There are many different approaches to distribution which balance trade-offs between simplicity, scalability, hardware compatibility, convergence times, logical forwarding model, etc. The goal of Open vSwitch is to be able to support all as a primitive building block rather than choose a particular point in the distributed design space. Q: How can I contribute to the Open vSwitch Community? A: You can start by joining the mailing lists and helping to answer questions. You can also suggest improvements to documentation. If you have a feature or bug you would like to work on, send a mail to one of the :doc:`mailing lists `. Q: Why can I no longer connect to my OpenFlow controller or OVSDB manager? A: Starting in OVS 2.4, we switched the default ports to the IANA-specified port numbers for OpenFlow (6633->6653) and OVSDB (6632->6640). We recommend using these port numbers, but if you cannot, all the programs allow overriding the default port. See the appropriate man page. ovs-2.9.0/Documentation/faq/index.rst000066400000000000000000000021741324262074100175570ustar00rootroot00000000000000.. Copyright (c) 2016, Stephen Finucane Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. Convention for heading levels in Open vSwitch documentation: ======= Heading 0 (reserved for the title in a document) ------- Heading 1 ~~~~~~~ Heading 2 +++++++ Heading 3 ''''''' Heading 4 Avoid deeper levels because they do not render well. ================ Open vSwitch FAQ ================ .. toctree:: :maxdepth: 2 configuration contributing design general issues openflow qos releases terminology vlan vxlan ovs-2.9.0/Documentation/faq/issues.rst000066400000000000000000000476331324262074100177740ustar00rootroot00000000000000.. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. Convention for heading levels in Open vSwitch documentation: ======= Heading 0 (reserved for the title in a document) ------- Heading 1 ~~~~~~~ Heading 2 +++++++ Heading 3 ''''''' Heading 4 Avoid deeper levels because they do not render well. =========================== Common Configuration Issues =========================== Q: I created a bridge and added my Ethernet port to it, using commands like these:: $ ovs-vsctl add-br br0 $ ovs-vsctl add-port br0 eth0 and as soon as I ran the "add-port" command I lost all connectivity through eth0. Help! A: A physical Ethernet device that is part of an Open vSwitch bridge should not have an IP address. If one does, then that IP address will not be fully functional. You can restore functionality by moving the IP address to an Open vSwitch "internal" device, such as the network device named after the bridge itself. For example, assuming that eth0's IP address is 192.168.128.5, you could run the commands below to fix up the situation:: $ ip addr flush dev eth0 $ ip addr add 192.168.128.5/24 dev br0 $ ip link set br0 up (If your only connection to the machine running OVS is through the IP address in question, then you would want to run all of these commands on a single command line, or put them into a script.) If there were any additional routes assigned to eth0, then you would also want to use commands to adjust these routes to go through br0. If you use DHCP to obtain an IP address, then you should kill the DHCP client that was listening on the physical Ethernet interface (e.g. eth0) and start one listening on the internal interface (e.g. br0). You might still need to manually clear the IP address from the physical interface (e.g. with "ip addr flush dev eth0"). There is no compelling reason why Open vSwitch must work this way. However, this is the way that the Linux kernel bridge module has always worked, so it's a model that those accustomed to Linux bridging are already used to. Also, the model that most people expect is not implementable without kernel changes on all the versions of Linux that Open vSwitch supports. By the way, this issue is not specific to physical Ethernet devices. It applies to all network devices except Open vSwitch "internal" devices. Q: I created a bridge and added a couple of Ethernet ports to it, using commands like these:: $ ovs-vsctl add-br br0 $ ovs-vsctl add-port br0 eth0 $ ovs-vsctl add-port br0 eth1 and now my network seems to have melted: connectivity is unreliable (even connectivity that doesn't go through Open vSwitch), all the LEDs on my physical switches are blinking, wireshark shows duplicated packets, and CPU usage is very high. A: More than likely, you've looped your network. Probably, eth0 and eth1 are connected to the same physical Ethernet switch. This yields a scenario where OVS receives a broadcast packet on eth0 and sends it out on eth1, then the physical switch connected to eth1 sends the packet back on eth0, and so on forever. More complicated scenarios, involving a loop through multiple switches, are possible too. The solution depends on what you are trying to do: - If you added eth0 and eth1 to get higher bandwidth or higher reliability between OVS and your physical Ethernet switch, use a bond. The following commands create br0 and then add eth0 and eth1 as a bond:: $ ovs-vsctl add-br br0 $ ovs-vsctl add-bond br0 bond0 eth0 eth1 Bonds have tons of configuration options. Please read the documentation on the Port table in ovs-vswitchd.conf.db(5) for all the details. Configuration for DPDK-enabled interfaces is slightly less straightforward. Refer to :doc:`/intro/install/dpdk` for more information. - Perhaps you don't actually need eth0 and eth1 to be on the same bridge. For example, if you simply want to be able to connect each of them to virtual machines, then you can put each of them on a bridge of its own:: $ ovs-vsctl add-br br0 $ ovs-vsctl add-port br0 eth0 $ ovs-vsctl add-br br1 $ ovs-vsctl add-port br1 eth1 and then connect VMs to br0 and br1. (A potential disadvantage is that traffic cannot directly pass between br0 and br1. Instead, it will go out eth0 and come back in eth1, or vice versa.) - If you have a redundant or complex network topology and you want to prevent loops, turn on spanning tree protocol (STP). The following commands create br0, enable STP, and add eth0 and eth1 to the bridge. The order is important because you don't want have to have a loop in your network even transiently:: $ ovs-vsctl add-br br0 $ ovs-vsctl set bridge br0 stp_enable=true $ ovs-vsctl add-port br0 eth0 $ ovs-vsctl add-port br0 eth1 The Open vSwitch implementation of STP is not well tested. Report any bugs you observe, but if you'd rather avoid acting as a beta tester then another option might be your best shot. Q: I can't seem to use Open vSwitch in a wireless network. A: Wireless base stations generally only allow packets with the source MAC address of NIC that completed the initial handshake. Therefore, without MAC rewriting, only a single device can communicate over a single wireless link. This isn't specific to Open vSwitch, it's enforced by the access point, so the same problems will show up with the Linux bridge or any other way to do bridging. Q: I can't seem to add my PPP interface to an Open vSwitch bridge. A: PPP most commonly carries IP packets, but Open vSwitch works only with Ethernet frames. The correct way to interface PPP to an Ethernet network is usually to use routing instead of switching. Q: Is there any documentation on the database tables and fields? A: Yes. ovs-vswitchd.conf.db(5) is a comprehensive reference. Q: When I run ovs-dpctl I no longer see the bridges I created. Instead, I only see a datapath called "ovs-system". How can I see datapath information about a particular bridge? A: In version 1.9.0, OVS switched to using a single datapath that is shared by all bridges of that type. The ``ovs-appctl dpif/*`` commands provide similar functionality that is scoped by the bridge. Q: I created a GRE port using ovs-vsctl so why can't I send traffic or see the port in the datapath? A: On Linux kernels before 3.11, the OVS GRE module and Linux GRE module cannot be loaded at the same time. It is likely that on your system the Linux GRE module is already loaded and blocking OVS (to confirm, check dmesg for errors regarding GRE registration). To fix this, unload all GRE modules that appear in lsmod as well as the OVS kernel module. You can then reload the OVS module following the directions in :doc:`/intro/install/general` , which will ensure that dependencies are satisfied. Q: Open vSwitch does not seem to obey my packet filter rules. A: It depends on mechanisms and configurations you want to use. You cannot usefully use typical packet filters, like iptables, on physical Ethernet ports that you add to an Open vSwitch bridge. This is because Open vSwitch captures packets from the interface at a layer lower below where typical packet-filter implementations install their hooks. (This actually applies to any interface of type "system" that you might add to an Open vSwitch bridge.) You can usefully use typical packet filters on Open vSwitch internal ports as they are mostly ordinary interfaces from the point of view of packet filters. For example, suppose you create a bridge br0 and add Ethernet port eth0 to it. Then you can usefully add iptables rules to affect the internal interface br0, but not the physical interface eth0. (br0 is also where you would add an IP address, as discussed elsewhere in the FAQ.) For simple filtering rules, it might be possible to achieve similar results by installing appropriate OpenFlow flows instead. The OVS conntrack feature (see the "ct" action in ovs-ofctl(8)) can implement a stateful firewall. If the use of a particular packet filter setup is essential, Open vSwitch might not be the best choice for you. On Linux, you might want to consider using the Linux Bridge. (This is the only choice if you want to use ebtables rules.) On NetBSD, you might want to consider using the bridge(4) with BRIDGE_IPF option. Q: It seems that Open vSwitch does nothing when I removed a port and then immediately put it back. For example, consider that p1 is a port of ``type=internal``:: $ ovs-vsctl del-port br0 p1 -- \ add-port br0 p1 -- \ set interface p1 type=internal Any other type of port gets the same effect. A: It's an expected behaviour. If del-port and add-port happen in a single OVSDB transaction as your example, Open vSwitch always "skips" the intermediate steps. Even if they are done in multiple transactions, it's still allowed for Open vSwitch to skip the intermediate steps and just implement the overall effect. In both cases, your example would be turned into a no-op. If you want to make Open vSwitch actually destroy and then re-create the port for some side effects like resetting kernel setting for the corresponding interface, you need to separate operations into multiple OVSDB transactions and ensure that at least the first one does not have ``--no-wait``. In the following example, the first ovs-vsctl will block until Open vSwitch reloads the new configuration and removes the port:: $ ovs-vsctl del-port br0 p1 $ ovs-vsctl add-port br0 p1 -- \ set interface p1 type=internal Q: I want to add thousands of ports to an Open vSwitch bridge, but it takes too long (minutes or hours) to do it with ovs-vsctl. How can I do it faster? A: If you add them one at a time with ovs-vsctl, it can take a long time to add thousands of ports to an Open vSwitch bridge. This is because every invocation of ovs-vsctl first reads the current configuration from OVSDB. As the number of ports grows, this starts to take an appreciable amount of time, and when it is repeated thousands of times the total time becomes significant. The solution is to add the ports in one invocation of ovs-vsctl (or a small number of them). For example, using bash:: $ ovs-vsctl add-br br0 $ cmds=; for i in {1..5000}; do cmds+=" -- add-port br0 p$i"; done $ ovs-vsctl $cmds takes seconds, not minutes or hours, in the OVS sandbox environment. Q: I created a bridge named br0. My bridge shows up in "ovs-vsctl show", but "ovs-ofctl show br0" just prints "br0 is not a bridge or a socket". A: Open vSwitch wasn't able to create the bridge. Check the ovs-vswitchd log for details (Debian and Red Hat packaging for Open vSwitch put it in /var/log/openvswitch/ovs-vswitchd.log). In general, the Open vSwitch database reflects the desired configuration state. ovs-vswitchd monitors the database and, when it changes, reconfigures the system to reflect the new desired state. This normally happens very quickly. Thus, a discrepancy between the database and the actual state indicates that ovs-vswitchd could not implement the configuration, and so one should check the log to find out why. (Another possible cause is that ovs-vswitchd is not running. This will make ovs-vsctl commands hang, if they change the configuration, unless one specifies ``--no-wait``.) Q: I have a bridge br0. I added a new port vif1.0, and it shows up in "ovs-vsctl show", but "ovs-vsctl list port" says that it has OpenFlow port ("ofport") -1, and "ovs-ofctl show br0" doesn't show vif1.0 at all. A: Open vSwitch wasn't able to create the port. Check the ovs-vswitchd log for details (Debian and Red Hat packaging for Open vSwitch put it in /var/log/openvswitch/ovs-vswitchd.log). Please see the previous question for more information. You may want to upgrade to Open vSwitch 2.3 (or later), in which ovs-vsctl will immediately report when there is an issue creating a port. Q: I created a tap device tap0, configured an IP address on it, and added it to a bridge, like this:: $ tunctl -t tap0 $ ip addr add 192.168.0.123/24 dev tap0 $ ip link set tap0 up $ ovs-vsctl add-br br0 $ ovs-vsctl add-port br0 tap0 I expected that I could then use this IP address to contact other hosts on the network, but it doesn't work. Why not? A: The short answer is that this is a misuse of a "tap" device. Use an "internal" device implemented by Open vSwitch, which works differently and is designed for this use. To solve this problem with an internal device, instead run:: $ ovs-vsctl add-br br0 $ ovs-vsctl add-port br0 int0 -- set Interface int0 type=internal $ ip addr add 192.168.0.123/24 dev int0 $ ip link set int0 up Even more simply, you can take advantage of the internal port that every bridge has under the name of the bridge:: $ ovs-vsctl add-br br0 $ ip addr add 192.168.0.123/24 dev br0 $ ip link set br0 up In more detail, a "tap" device is an interface between the Linux (or BSD) network stack and a user program that opens it as a socket. When the "tap" device transmits a packet, it appears in the socket opened by the userspace program. Conversely, when the userspace program writes to the "tap" socket, the kernel TCP/IP stack processes the packet as if it had been received by the "tap" device. Consider the configuration above. Given this configuration, if you "ping" an IP address in the 192.168.0.x subnet, the Linux kernel routing stack will transmit an ARP on the tap0 device. Open vSwitch userspace treats "tap" devices just like any other network device; that is, it doesn't open them as "tap" sockets. That means that the ARP packet will simply get dropped. You might wonder why the Open vSwitch kernel module doesn't intercept the ARP packet and bridge it. After all, Open vSwitch intercepts packets on other devices. The answer is that Open vSwitch only intercepts *received* packets, but this is a packet being transmitted. The same thing happens for all other types of network devices, except for Open vSwitch "internal" ports. If you, for example, add a physical Ethernet port to an OVS bridge, configure an IP address on a physical Ethernet port, and then issue a "ping" to an address in that subnet, the same thing happens: an ARP gets transmitted on the physical Ethernet port and Open vSwitch never sees it. (You should not do that, as documented at the beginning of this section.) It can make sense to add a "tap" device to an Open vSwitch bridge, if some userspace program (other than Open vSwitch) has opened the tap socket. This is the case, for example, if the "tap" device was created by KVM (or QEMU) to simulate a virtual NIC. In such a case, when OVS bridges a packet to the "tap" device, the kernel forwards that packet to KVM in userspace, which passes it along to the VM, and in the other direction, when the VM sends a packet, KVM writes it to the "tap" socket, which causes OVS to receive it and bridge it to the other OVS ports. Please note that in such a case no IP address is configured on the "tap" device (there is normally an IP address configured in the virtual NIC inside the VM, but this is not visible to the host Linux kernel or to Open vSwitch). There is one special case in which Open vSwitch does directly read and write "tap" sockets. This is an implementation detail of the Open vSwitch userspace switch, which implements its "internal" ports as Linux (or BSD) "tap" sockets. In such a userspace switch, OVS receives packets sent on the "tap" device used to implement an "internal" port by reading the associated "tap" socket, and bridges them to the rest of the switch. In the other direction, OVS transmits packets bridged to the "internal" port by writing them to the "tap" socket, causing them to be processed by the kernel TCP/IP stack as if they had been received on the "tap" device. Users should not need to be concerned with this implementation detail. Open vSwitch has a network device type called "tap". This is intended only for implementing "internal" ports in the OVS userspace switch and should not be used otherwise. In particular, users should not configure KVM "tap" devices as type "tap" (use type "system", the default, instead). Q: I observe packet loss at the beginning of RFC2544 tests on a server running few hundred container apps bridged to OVS with traffic generated by HW traffic generator. How can I fix this? A: This is expected behavior on virtual switches. RFC2544 tests were designed for hardware switches, which don't have caches on the fastpath that need to be heated. Traffic generators in order to prime the switch use learning phase to heat the caches before sending the actual traffic in test phase. In case of OVS the cache is flushed quickly and to accommodate the traffic generator's delay between learning and test phase, the max-idle timeout settings should be changed to 50000 ms.:: $ ovs-vsctl --no-wait set Open_vSwitch . other_config:max-idle=50000 Q: How can I configure the bridge internal interface MTU? Why does Open vSwitch keep changing internal ports MTU? A: By default Open vSwitch overrides the internal interfaces (e.g. br0) MTU. If you have just an internal interface (e.g. br0) and a physical interface (e.g. eth0), then every change in MTU to eth0 will be reflected to br0. Any manual MTU configuration using `ip` on internal interfaces is going to be overridden by Open vSwitch to match the current bridge minimum. Sometimes this behavior is not desirable, for example with tunnels. The MTU of an internal interface can be explicitly set using the following command:: $ ovs-vsctl set int br0 mtu_request=1450 After this, Open vSwitch will configure br0 MTU to 1450. Since this setting is in the database it will be persistent (compared to what happens with `ip`). The MTU configuration can be removed to restore the default behavior with:: $ ovs-vsctl set int br0 mtu_request=[] The ``mtu_request`` column can be used to configure MTU even for physical interfaces (e.g. eth0). Q: I just upgraded and I see a performance drop. Why? A: The OVS kernel datapath may have been updated to a newer version than the OVS userspace components. Sometimes new versions of OVS kernel module add functionality that is backwards compatible with older userspace components but may cause a drop in performance with them. Especially, if a kernel module from OVS 2.1 or newer is paired with OVS userspace 1.10 or older, there will be a performance drop for TCP traffic. Updating the OVS userspace components to the latest released version should fix the performance degradation. To get the best possible performance and functionality, it is recommended to pair the same versions of the kernel module and OVS userspace. ovs-2.9.0/Documentation/faq/openflow.rst000066400000000000000000000617221324262074100203050ustar00rootroot00000000000000.. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. Convention for heading levels in Open vSwitch documentation: ======= Heading 0 (reserved for the title in a document) ------- Heading 1 ~~~~~~~ Heading 2 +++++++ Heading 3 ''''''' Heading 4 Avoid deeper levels because they do not render well. ============== Using OpenFlow ============== Q: What versions of OpenFlow does Open vSwitch support? A: The following table lists the versions of OpenFlow supported by each version of Open vSwitch: =============== ===== ===== ===== ===== ===== ===== ===== Open vSwitch OF1.0 OF1.1 OF1.2 OF1.3 OF1.4 OF1.5 OF1.6 =============== ===== ===== ===== ===== ===== ===== ===== 1.9 and earlier yes --- --- --- --- --- --- 1.10, 1.11 yes --- (*) (*) --- --- --- 2.0, 2.1 yes (*) (*) (*) --- --- --- 2.2 yes (*) (*) (*) (%) (*) --- 2.3, 2.4 yes yes yes yes (*) (*) --- 2.5, 2.6, 2.7 yes yes yes yes (*) (*) (*) 2.8 yes yes yes yes yes (*) (*) =============== ===== ===== ===== ===== ===== ===== ===== --- Not supported. yes Supported and enabled by default (*) Supported, but missing features, and must be enabled by user. (%) Experimental, unsafe implementation. In any case, the user may override the default: - To enable OpenFlow 1.0, 1.1, 1.2, and 1.3 on bridge br0:: $ ovs-vsctl set bridge br0 \ protocols=OpenFlow10,OpenFlow11,OpenFlow12,OpenFlow13 - To enable OpenFlow 1.0, 1.1, 1.2, 1.3, 1.4, and 1.5 on bridge br0:: $ ovs-vsctl set bridge br0 \ protocols=OpenFlow10,OpenFlow11,OpenFlow12,OpenFlow13,OpenFlow14,OpenFlow15 - To enable only OpenFlow 1.0 on bridge br0:: $ ovs-vsctl set bridge br0 protocols=OpenFlow10 All current versions of ovs-ofctl enable only OpenFlow 1.0 by default. Use the -O option to enable support for later versions of OpenFlow in ovs-ofctl. For example:: $ ovs-ofctl -O OpenFlow13 dump-flows br0 (Open vSwitch 2.2 had an experimental implementation of OpenFlow 1.4 that could cause crashes. We don't recommend enabling it.) :doc:`/topics/openflow` tracks support for OpenFlow 1.1 and later features. When support for OpenFlow 1.5 and 1.6 is solidly implemented, Open vSwitch will enable those version by default. Q: Does Open vSwitch support MPLS? A: Before version 1.11, Open vSwitch did not support MPLS. That is, these versions can match on MPLS Ethernet types, but they cannot match, push, or pop MPLS labels, nor can they look past MPLS labels into the encapsulated packet. Open vSwitch versions 1.11, 2.0, and 2.1 have very minimal support for MPLS. With the userspace datapath only, these versions can match, push, or pop a single MPLS label, but they still cannot look past MPLS labels (even after popping them) into the encapsulated packet. Kernel datapath support is unchanged from earlier versions. Open vSwitch version 2.3 can match, push, or pop a single MPLS label and look past the MPLS label into the encapsulated packet. Both userspace and kernel datapaths will be supported, but MPLS processing always happens in userspace either way, so kernel datapath performance will be disappointing. Open vSwitch version 2.4 can match, push, or pop up to 3 MPLS labels and look past the MPLS label into the encapsulated packet. It will have kernel support for MPLS, yielding improved performance. Q: I'm getting "error type 45250 code 0". What's that? A: This is a Open vSwitch extension to OpenFlow error codes. Open vSwitch uses this extension when it must report an error to an OpenFlow controller but no standard OpenFlow error code is suitable. Open vSwitch logs the errors that it sends to controllers, so the easiest thing to do is probably to look at the ovs-vswitchd log to find out what the error was. If you want to dissect the extended error message yourself, the format is documented in include/openflow/nicira-ext.h in the Open vSwitch source distribution. The extended error codes are documented in include/openvswitch/ofp-errors.h. Q: Some of the traffic that I'd expect my OpenFlow controller to see doesn't actually appear through the OpenFlow connection, even though I know that it's going through. A: By default, Open vSwitch assumes that OpenFlow controllers are connected "in-band", that is, that the controllers are actually part of the network that is being controlled. In in-band mode, Open vSwitch sets up special "hidden" flows to make sure that traffic can make it back and forth between OVS and the controllers. These hidden flows are higher priority than any flows that can be set up through OpenFlow, and they are not visible through normal OpenFlow flow table dumps. Usually, the hidden flows are desirable and helpful, but occasionally they can cause unexpected behavior. You can view the full OpenFlow flow table, including hidden flows, on bridge br0 with the command:: $ ovs-appctl bridge/dump-flows br0 to help you debug. The hidden flows are those with priorities greater than 65535 (the maximum priority that can be set with OpenFlow). The ``Documentation/topics/design`` doc describes the in-band model in detail. If your controllers are not actually in-band (e.g. they are on localhost via 127.0.0.1, or on a separate network), then you should configure your controllers in "out-of-band" mode. If you have one controller on bridge br0, then you can configure out-of-band mode on it with:: $ ovs-vsctl set controller br0 connection-mode=out-of-band Q: Some of the OpenFlow flows that my controller sets up don't seem to apply to certain traffic, especially traffic between OVS and the controller itself. A: See above. Q: I configured all my controllers for out-of-band control mode but "ovs-appctl bridge/dump-flows" still shows some hidden flows. A: You probably have a remote manager configured (e.g. with "ovs-vsctl set-manager"). By default, Open vSwitch assumes that managers need in-band rules set up on every bridge. You can disable these rules on bridge br0 with:: $ ovs-vsctl set bridge br0 other-config:disable-in-band=true This actually disables in-band control entirely for the bridge, as if all the bridge's controllers were configured for out-of-band control. Q: My OpenFlow controller doesn't see the VLANs that I expect. A: See answer under "VLANs", above. Q: I ran ``ovs-ofctl add-flow br0 nw_dst=192.168.0.1,actions=drop`` but I got a funny message like this:: ofp_util|INFO|normalization changed ofp_match, details: ofp_util|INFO| pre: nw_dst=192.168.0.1 ofp_util|INFO|post: and when I ran ``ovs-ofctl dump-flows br0`` I saw that my nw_dst match had disappeared, so that the flow ends up matching every packet. A: The term "normalization" in the log message means that a flow cannot match on an L3 field without saying what L3 protocol is in use. The "ovs-ofctl" command above didn't specify an L3 protocol, so the L3 field match was dropped. In this case, the L3 protocol could be IP or ARP. A correct command for each possibility is, respectively:: $ ovs-ofctl add-flow br0 ip,nw_dst=192.168.0.1,actions=drop and:: $ ovs-ofctl add-flow br0 arp,nw_dst=192.168.0.1,actions=drop Similarly, a flow cannot match on an L4 field without saying what L4 protocol is in use. For example, the flow match ``tp_src=1234`` is, by itself, meaningless and will be ignored. Instead, to match TCP source port 1234, write ``tcp,tp_src=1234``, or to match UDP source port 1234, write ``udp,tp_src=1234``. Q: How can I figure out the OpenFlow port number for a given port? A: The ``OFPT_FEATURES_REQUEST`` message requests an OpenFlow switch to respond with an ``OFPT_FEATURES_REPLY`` that, among other information, includes a mapping between OpenFlow port names and numbers. From a command prompt, ``ovs-ofctl show br0`` makes such a request and prints the response for switch br0. The Interface table in the Open vSwitch database also maps OpenFlow port names to numbers. To print the OpenFlow port number associated with interface eth0, run:: $ ovs-vsctl get Interface eth0 ofport You can print the entire mapping with:: $ ovs-vsctl -- --columns=name,ofport list Interface but the output mixes together interfaces from all bridges in the database, so it may be confusing if more than one bridge exists. In the Open vSwitch database, ofport value ``-1`` means that the interface could not be created due to an error. (The Open vSwitch log should indicate the reason.) ofport value ``[]`` (the empty set) means that the interface hasn't been created yet. The latter is normally an intermittent condition (unless ovs-vswitchd is not running). Q: I added some flows with my controller or with ovs-ofctl, but when I run "ovs-dpctl dump-flows" I don't see them. A: ovs-dpctl queries a kernel datapath, not an OpenFlow switch. It won't display the information that you want. You want to use ``ovs-ofctl dump-flows`` instead. Q: It looks like each of the interfaces in my bonded port shows up as an individual OpenFlow port. Is that right? A: Yes, Open vSwitch makes individual bond interfaces visible as OpenFlow ports, rather than the bond as a whole. The interfaces are treated together as a bond for only a few purposes: - Sending a packet to the OFPP_NORMAL port. (When an OpenFlow controller is not configured, this happens implicitly to every packet.) - Mirrors configured for output to a bonded port. It would make a lot of sense for Open vSwitch to present a bond as a single OpenFlow port. If you want to contribute an implementation of such a feature, please bring it up on the Open vSwitch development mailing list at dev@openvswitch.org. Q: I have a sophisticated network setup involving Open vSwitch, VMs or multiple hosts, and other components. The behavior isn't what I expect. Help! A: To debug network behavior problems, trace the path of a packet, hop-by-hop, from its origin in one host to a remote host. If that's correct, then trace the path of the response packet back to the origin. The open source tool called ``plotnetcfg`` can help to understand the relationship between the networking devices on a single host. Usually a simple ICMP echo request and reply (``ping``) packet is good enough. Start by initiating an ongoing ``ping`` from the origin host to a remote host. If you are tracking down a connectivity problem, the "ping" will not display any successful output, but packets are still being sent. (In this case the packets being sent are likely ARP rather than ICMP.) Tools available for tracing include the following: - ``tcpdump`` and ``wireshark`` for observing hops across network devices, such as Open vSwitch internal devices and physical wires. - ``ovs-appctl dpif/dump-flows
`` in Open vSwitch 1.10 and later or ``ovs-dpctl dump-flows
`` in earlier versions. These tools allow one to observe the actions being taken on packets in ongoing flows. See ovs-vswitchd(8) for ``ovs-appctl dpif/dump-flows`` documentation, ovs-dpctl(8) for ``ovs-dpctl dump-flows`` documentation, and "Why are there so many different ways to dump flows?" above for some background. - ``ovs-appctl ofproto/trace`` to observe the logic behind how ovs-vswitchd treats packets. See ovs-vswitchd(8) for documentation. You can out more details about a given flow that ``ovs-dpctl dump-flows`` displays, by cutting and pasting a flow from the output into an ``ovs-appctl ofproto/trace`` command. - SPAN, RSPAN, and ERSPAN features of physical switches, to observe what goes on at these physical hops. Starting at the origin of a given packet, observe the packet at each hop in turn. For example, in one plausible scenario, you might: 1. ``tcpdump`` the ``eth`` interface through which an ARP egresses a VM, from inside the VM. 2. ``tcpdump`` the ``vif`` or ``tap`` interface through which the ARP ingresses the host machine. 3. Use ``ovs-dpctl dump-flows`` to spot the ARP flow and observe the host interface through which the ARP egresses the physical machine. You may need to use ``ovs-dpctl show`` to interpret the port numbers. If the output seems surprising, you can use ``ovs-appctl ofproto/trace`` to observe details of how ovs-vswitchd determined the actions in the ``ovs-dpctl dump-flows`` output. 4. ``tcpdump`` the ``eth`` interface through which the ARP egresses the physical machine. 5. ``tcpdump`` the ``eth`` interface through which the ARP ingresses the physical machine, at the remote host that receives the ARP. 6. Use ``ovs-dpctl dump-flows`` to spot the ARP flow on the remote host remote host that receives the ARP and observe the VM ``vif`` or ``tap`` interface to which the flow is directed. Again, ``ovs-dpctl show`` and ``ovs-appctl ofproto/trace`` might help. 7. ``tcpdump`` the ``vif`` or ``tap`` interface to which the ARP is directed. 8. ``tcpdump`` the ``eth`` interface through which the ARP ingresses a VM, from inside the VM. It is likely that during one of these steps you will figure out the problem. If not, then follow the ARP reply back to the origin, in reverse. Q: How do I make a flow drop packets? A: To drop a packet is to receive it without forwarding it. OpenFlow explicitly specifies forwarding actions. Thus, a flow with an empty set of actions does not forward packets anywhere, causing them to be dropped. You can specify an empty set of actions with ``actions=`` on the ovs-ofctl command line. For example:: $ ovs-ofctl add-flow br0 priority=65535,actions= would cause every packet entering switch br0 to be dropped. You can write "drop" explicitly if you like. The effect is the same. Thus, the following command also causes every packet entering switch br0 to be dropped:: $ ovs-ofctl add-flow br0 priority=65535,actions=drop ``drop`` is not an action, either in OpenFlow or Open vSwitch. Rather, it is only a way to say that there are no actions. Q: I added a flow to send packets out the ingress port, like this:: $ ovs-ofctl add-flow br0 in_port=2,actions=2 but OVS drops the packets instead. A: Yes, OpenFlow requires a switch to ignore attempts to send a packet out its ingress port. The rationale is that dropping these packets makes it harder to loop the network. Sometimes this behavior can even be convenient, e.g. it is often the desired behavior in a flow that forwards a packet to several ports ("floods" the packet). Sometimes one really needs to send a packet out its ingress port ("hairpin"). In this case, output to ``OFPP_IN_PORT``, which in ovs-ofctl syntax is expressed as just ``in_port``, e.g.:: $ ovs-ofctl add-flow br0 in_port=2,actions=in_port This also works in some circumstances where the flow doesn't match on the input port. For example, if you know that your switch has five ports numbered 2 through 6, then the following will send every received packet out every port, even its ingress port:: $ ovs-ofctl add-flow br0 actions=2,3,4,5,6,in_port or, equivalently:: $ ovs-ofctl add-flow br0 actions=all,in_port Sometimes, in complicated flow tables with multiple levels of ``resubmit`` actions, a flow needs to output to a particular port that may or may not be the ingress port. It's difficult to take advantage of ``OFPP_IN_PORT`` in this situation. To help, Open vSwitch provides, as an OpenFlow extension, the ability to modify the in_port field. Whatever value is currently in the in_port field is the port to which outputs will be dropped, as well as the destination for ``OFPP_IN_PORT``. This means that the following will reliably output to port 2 or to ports 2 through 6, respectively:: $ ovs-ofctl add-flow br0 in_port=2,actions=load:0->NXM_OF_IN_PORT[],2 $ ovs-ofctl add-flow br0 actions=load:0->NXM_OF_IN_PORT[],2,3,4,5,6 If the input port is important, then one may save and restore it on the stack: $ ovs-ofctl add-flow br0 actions=push:NXM_OF_IN_PORT[],\ load:0->NXM_OF_IN_PORT[],\ 2,3,4,5,6,\ pop:NXM_OF_IN_PORT[] Q: My bridge br0 has host 192.168.0.1 on port 1 and host 192.168.0.2 on port 2. I set up flows to forward only traffic destined to the other host and drop other traffic, like this:: priority=5,in_port=1,ip,nw_dst=192.168.0.2,actions=2 priority=5,in_port=2,ip,nw_dst=192.168.0.1,actions=1 priority=0,actions=drop But it doesn't work--I don't get any connectivity when I do this. Why? A: These flows drop the ARP packets that IP hosts use to establish IP connectivity over Ethernet. To solve the problem, add flows to allow ARP to pass between the hosts:: priority=5,in_port=1,arp,actions=2 priority=5,in_port=2,arp,actions=1 This issue can manifest other ways, too. The following flows that match on Ethernet addresses instead of IP addresses will also drop ARP packets, because ARP requests are broadcast instead of being directed to a specific host:: priority=5,in_port=1,dl_dst=54:00:00:00:00:02,actions=2 priority=5,in_port=2,dl_dst=54:00:00:00:00:01,actions=1 priority=0,actions=drop The solution already described above will also work in this case. It may be better to add flows to allow all multicast and broadcast traffic:: priority=5,in_port=1,dl_dst=01:00:00:00:00:00/01:00:00:00:00:00,actions=2 priority=5,in_port=2,dl_dst=01:00:00:00:00:00/01:00:00:00:00:00,actions=1 Q: My bridge disconnects from my controller on add-port/del-port. A: Reconfiguring your bridge can change your bridge's datapath-id because Open vSwitch generates datapath-id from the MAC address of one of its ports. In that case, Open vSwitch disconnects from controllers because there's no graceful way to notify controllers about the change of datapath-id. To avoid the behaviour, you can configure datapath-id manually.:: $ ovs-vsctl set bridge br0 other-config:datapath-id=0123456789abcdef Q: My controller complains that OVS is not buffering packets. What's going on? A: "Packet buffering" is an optional OpenFlow feature, and controllers should detect how many "buffers" an OpenFlow switch implements. It was recently noticed that OVS implementation of the buffering feature was not compliant to OpenFlow specifications. Rather than fix it and risk controller incompatibility, the buffering feature is removed as of OVS 2.7. Controllers are already expected to work properly in cases where the switch can not buffer packets, but sends full packets in "packet-in" messages instead, so this change should not affect existing users. After the change OVS always sends the ``buffer_id`` as ``0xffffffff`` in "packet-in" messages and will send an error response if any other value of this field is included in a "packet-out" or a "flow mod" sent by a controller. Packet buffers have limited usefulness in any case. Table-miss packet-in messages most commonly pass the first packet in a microflow to the OpenFlow controller, which then sets up an OpenFlow flow that handles remaining traffic in the microflow without further controller intervention. In such a case, the packet that initiates the microflow is in practice usually small (certainly for TCP), which means that the switch sends the entire packet to the controller and the buffer only saves a small number of bytes in the reverse direction. Q: How does OVS divide flows among buckets in an OpenFlow "select" group? A: In Open vSwitch 2.3 and earlier, Open vSwitch used the destination Ethernet address to choose a bucket in a select group. Open vSwitch 2.4 and later by default hashes the source and destination Ethernet address, VLAN ID, Ethernet type, IPv4/v6 source and destination address and protocol, and for TCP and SCTP only, the source and destination ports. The hash is "symmetric", meaning that exchanging source and destination addresses does not change the bucket selection. Select groups in Open vSwitch 2.4 and later can be configured to use a different hash function, using a Netronome extension to the OpenFlow 1.5+ group_mod message. For more information, see Documentation/group-selection-method-property.txt in the Open vSwitch source tree. (OpenFlow 1.5 support in Open vSwitch is still experimental.) Q: I added a flow to accept packets on VLAN 123 and output them on VLAN 456, like so:: $ ovs-ofctl add-flow br0 dl_vlan=123,actions=output:1,mod_vlan_vid:456 but the packets are actually being output in VLAN 123. Why? A: OpenFlow actions are executed in the order specified. Thus, the actions above first output the packet, then change its VLAN. Since the output occurs before changing the VLAN, the change in VLAN will have no visible effect. To solve this and similar problems, order actions so that changes to headers happen before output, e.g.:: $ ovs-ofctl add-flow br0 dl_vlan=123,actions=mod_vlan_vid:456,output:1 See also the following question. Q: I added a flow to a redirect packets for TCP port 80 to port 443, like so:: $ ovs-ofctl add-flow br0 tcp,tcp_dst=123,actions=mod_tp_dst:443 but the packets are getting dropped instead. Why? A: This set of actions does change the TCP destination port to 443, but then it does nothing more. It doesn't, for example, say to continue to another flow table or to output the packet. Therefore, the packet is dropped. To solve the problem, add an action that does something with the modified packet. For example:: $ ovs-ofctl add-flow br0 tcp,tcp_dst=123,actions=mod_tp_dst:443,normal See also the preceding question. Q: The "learn" action can't learn the action I want, can you improve it? A: By itself, the "learn" action can only put two kinds of actions into the flows that it creates: "load" and "output" actions. If "learn" is used in isolation, these are severe limits. However, "learn" is not meant to be used in isolation. It is a primitive meant to be used together with other Open vSwitch features to accomplish a task. Its existing features are enough to accomplish most tasks. Here is an outline of a typical pipeline structure that allows for versatile behavior using "learn": - Flows in table A contain a "learn" action, that populates flows in table L, that use a "load" action to populate register R with information about what was learned. - Flows in table B contain two sequential resubmit actions: one to table L and another one to table B+1. - Flows in table B+1 match on register R and act differently depending on what the flows in table L loaded into it. This approach can be used to implement many "learn"-based features. For example: - Resubmit to a table selected based on learned information, e.g. see: https://mail.openvswitch.org/pipermail/ovs-discuss/2016-June/021694.html - MAC learning in the middle of a pipeline, as described in :doc:`/tutorials/ovs-advanced` - TCP state based firewalling, by learning outgoing connections based on SYN packets and matching them up with incoming packets. - At least some of the features described in T. A. Hoff, "Extending Open vSwitch to Facilitate Creation of Stateful SDN Applications". Q: When using the "ct" action with FTP connections, it doesn't seem to matter if I set the "alg=ftp" parameter in the action. Is this required? A: It is advisable to use this option. Some platforms may automatically detect and apply ALGs in the "ct" action regardless of the parameters you provide, however this is not consistent across all implementations. The `ovs-ofctl(8) `_ man pages contain further details in the description of the ALG parameter. ovs-2.9.0/Documentation/faq/qos.rst000066400000000000000000000177451324262074100172640ustar00rootroot00000000000000.. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. Convention for heading levels in Open vSwitch documentation: ======= Heading 0 (reserved for the title in a document) ------- Heading 1 ~~~~~~~ Heading 2 +++++++ Heading 3 ''''''' Heading 4 Avoid deeper levels because they do not render well. ======================== Quality of Service (QoS) ======================== Q: Does OVS support Quality of Service (QoS)? A: Yes. For traffic that egresses from a switch, OVS supports traffic shaping; for traffic that ingresses into a switch, OVS support policing. Policing is a simple form of quality-of-service that simply drops packets received in excess of the configured rate. Due to its simplicity, policing is usually less accurate and less effective than egress traffic shaping, which queues packets. Keep in mind that ingress and egress are from the perspective of the switch. That means that egress shaping limits the rate at which traffic is allowed to transmit from a physical interface, but not the rate at which traffic will be received on a virtual machine's VIF. For ingress policing, the behavior is the opposite. Q: How do I configure egress traffic shaping? A: Suppose that you want to set up bridge br0 connected to physical Ethernet port eth0 (a 1 Gbps device) and virtual machine interfaces vif1.0 and vif2.0, and that you want to limit traffic from vif1.0 to eth0 to 10 Mbps and from vif2.0 to eth0 to 20 Mbps. Then, you could configure the bridge this way:: $ ovs-vsctl -- \ add-br br0 -- \ add-port br0 eth0 -- \ add-port br0 vif1.0 -- set interface vif1.0 ofport_request=5 -- \ add-port br0 vif2.0 -- set interface vif2.0 ofport_request=6 -- \ set port eth0 qos=@newqos -- \ --id=@newqos create qos type=linux-htb \ other-config:max-rate=1000000000 \ queues:123=@vif10queue \ queues:234=@vif20queue -- \ --id=@vif10queue create queue other-config:max-rate=10000000 -- \ --id=@vif20queue create queue other-config:max-rate=20000000 At this point, bridge br0 is configured with the ports and eth0 is configured with the queues that you need for QoS, but nothing is actually directing packets from vif1.0 or vif2.0 to the queues that we have set up for them. That means that all of the packets to eth0 are going to the "default queue", which is not what we want. We use OpenFlow to direct packets from vif1.0 and vif2.0 to the queues reserved for them:: $ ovs-ofctl add-flow br0 in_port=5,actions=set_queue:123,normal $ ovs-ofctl add-flow br0 in_port=6,actions=set_queue:234,normal Each of the above flows matches on the input port, sets up the appropriate queue (123 for vif1.0, 234 for vif2.0), and then executes the "normal" action, which performs the same switching that Open vSwitch would have done without any OpenFlow flows being present. (We know that vif1.0 and vif2.0 have OpenFlow port numbers 5 and 6, respectively, because we set their ofport_request columns above. If we had not done that, then we would have needed to find out their port numbers before setting up these flows.) Now traffic going from vif1.0 or vif2.0 to eth0 should be rate-limited. By the way, if you delete the bridge created by the above commands, with:: $ ovs-vsctl del-br br0 then that will leave one unreferenced QoS record and two unreferenced Queue records in the Open vSwich database. One way to clear them out, assuming you don't have other QoS or Queue records that you want to keep, is:: $ ovs-vsctl -- --all destroy QoS -- --all destroy Queue If you do want to keep some QoS or Queue records, or the Open vSwitch you are using is older than version 1.8 (which added the ``--all`` option), then you will have to destroy QoS and Queue records individually. Q: How do I configure ingress policing? A: A policing policy can be configured on an interface to drop packets that arrive at a higher rate than the configured value. For example, the following commands will rate-limit traffic that vif1.0 may generate to 10Mbps: $ ovs-vsctl set interface vif1.0 ingress_policing_rate=10000 $ ovs-vsctl set interface vif1.0 ingress_policing_burst=8000 Traffic policing can interact poorly with some network protocols and can have surprising results. The "Ingress Policing" section of ovs-vswitchd.conf.db(5) discusses the issues in greater detail. Q: I configured Quality of Service (QoS) in my OpenFlow network by adding records to the QoS and Queue table, but the results aren't what I expect. A: Did you install OpenFlow flows that use your queues? This is the primary way to tell Open vSwitch which queues you want to use. If you don't do this, then the default queue will be used, which will probably not have the effect you want. Refer to the previous question for an example. Q: I'd like to take advantage of some QoS feature that Open vSwitch doesn't yet support. How do I do that? A: Open vSwitch does not implement QoS itself. Instead, it can configure some, but not all, of the QoS features built into the Linux kernel. If you need some QoS feature that OVS cannot configure itself, then the first step is to figure out whether Linux QoS supports that feature. If it does, then you can submit a patch to support Open vSwitch configuration for that feature, or you can use "tc" directly to configure the feature in Linux. (If Linux QoS doesn't support the feature you want, then first you have to add that support to Linux.) Q: I configured QoS, correctly, but my measurements show that it isn't working as well as I expect. A: With the Linux kernel, the Open vSwitch implementation of QoS has two aspects: - Open vSwitch configures a subset of Linux kernel QoS features, according to what is in OVSDB. It is possible that this code has bugs. If you believe that this is so, then you can configure the Linux traffic control (QoS) stack directly with the "tc" program. If you get better results that way, you can send a detailed bug report to bugs@openvswitch.org. It is certain that Open vSwitch cannot configure every Linux kernel QoS feature. If you need some feature that OVS cannot configure, then you can also use "tc" directly (or add that feature to OVS). - The Open vSwitch implementation of OpenFlow allows flows to be directed to particular queues. This is pretty simple and unlikely to have serious bugs at this point. However, most problems with QoS on Linux are not bugs in Open vSwitch at all. They tend to be either configuration errors (please see the earlier questions in this section) or issues with the traffic control (QoS) stack in Linux. The Open vSwitch developers are not experts on Linux traffic control. We suggest that, if you believe you are encountering a problem with Linux traffic control, that you consult the tc manpages (e.g. tc(8), tc-htb(8), tc-hfsc(8)), web resources (e.g. http://lartc.org/), or mailing lists (e.g. http://vger.kernel.org/vger-lists.html#netdev). Q: Does Open vSwitch support OpenFlow meters? A: Since version 2.0, Open vSwitch has OpenFlow protocol support for OpenFlow meters. Currently, only the userspace datapath implements meters. ovs-2.9.0/Documentation/faq/releases.rst000066400000000000000000000273531324262074100202610ustar00rootroot00000000000000.. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. Convention for heading levels in Open vSwitch documentation: ======= Heading 0 (reserved for the title in a document) ------- Heading 1 ~~~~~~~ Heading 2 +++++++ Heading 3 ''''''' Heading 4 Avoid deeper levels because they do not render well. ======== Releases ======== Q: What does it mean for an Open vSwitch release to be LTS (long-term support)? A: All official releases have been through a comprehensive testing process and are suitable for production use. Planned releases occur twice a year. If a significant bug is identified in an LTS release, we will provide an updated release that includes the fix. Releases that are not LTS may not be fixed and may just be supplanted by the next major release. The current LTS release is 2.5.x. For more information on the Open vSwitch release process, refer to :doc:`/internals/release-process`. Q: What Linux kernel versions does each Open vSwitch release work with? A: The following table lists the Linux kernel versions against which the given versions of the Open vSwitch kernel module will successfully build. The Linux kernel versions are upstream kernel versions, so Linux kernels modified from the upstream sources may not build in some cases even if they are based on a supported version. This is most notably true of Red Hat Enterprise Linux (RHEL) kernels, which are extensively modified from upstream. ============ ============== Open vSwitch Linux kernel ============ ============== 1.4.x 2.6.18 to 3.2 1.5.x 2.6.18 to 3.2 1.6.x 2.6.18 to 3.2 1.7.x 2.6.18 to 3.3 1.8.x 2.6.18 to 3.4 1.9.x 2.6.18 to 3.8 1.10.x 2.6.18 to 3.8 1.11.x 2.6.18 to 3.8 2.0.x 2.6.32 to 3.10 2.1.x 2.6.32 to 3.11 2.3.x 2.6.32 to 3.14 2.4.x 2.6.32 to 4.0 2.5.x 2.6.32 to 4.3 2.6.x 3.10 to 4.7 2.7.x 3.10 to 4.9 2.8.x 3.10 to 4.12 2.9.x 3.10 to 4.13 ============ ============== Open vSwitch userspace should also work with the Linux kernel module built into Linux 3.3 and later. Open vSwitch userspace is not sensitive to the Linux kernel version. It should build against almost any kernel, certainly against 2.6.32 and later. Q: Are all features available with all datapaths? A: Open vSwitch supports different datapaths on different platforms. Each datapath has a different feature set: the following tables try to summarize the status. Supported datapaths: Linux upstream The datapath implemented by the kernel module shipped with Linux upstream. Since features have been gradually introduced into the kernel, the table mentions the first Linux release whose OVS module supports the feature. Linux OVS tree The datapath implemented by the Linux kernel module distributed with the OVS source tree. Userspace Also known as DPDK, dpif-netdev or dummy datapath. It is the only datapath that works on NetBSD, FreeBSD and Mac OSX. Hyper-V Also known as the Windows datapath. The following table lists the datapath supported features from an Open vSwitch user's perspective. ===================== ============== ============== ========= ======= Feature Linux upstream Linux OVS tree Userspace Hyper-V ===================== ============== ============== ========= ======= NAT 4.6 YES Yes NO Connection tracking 4.3 YES PARTIAL PARTIAL Tunnel - LISP NO YES NO NO Tunnel - STT NO YES NO YES Tunnel - GRE 3.11 YES YES YES Tunnel - VXLAN 3.12 YES YES YES Tunnel - Geneve 3.18 YES YES YES Tunnel - GRE-IPv6 NO NO YES NO Tunnel - VXLAN-IPv6 4.3 YES YES NO Tunnel - Geneve-IPv6 4.4 YES YES NO QoS - Policing YES YES YES NO QoS - Shaping YES YES NO NO sFlow YES YES YES NO IPFIX 3.10 YES YES NO Set action YES YES YES PARTIAL NIC Bonding YES YES YES YES Multiple VTEPs YES YES YES YES ===================== ============== ============== ========= ======= Do note, however: * Only a limited set of flow fields is modifiable via the set action by the Hyper-V datapath. The following table lists features that do not *directly* impact an Open vSwitch user, e.g. because their absence can be hidden by the ofproto layer (usually this comes with a performance penalty). ===================== ============== ============== ========= ======= Feature Linux upstream Linux OVS tree Userspace Hyper-V ===================== ============== ============== ========= ======= SCTP flows 3.12 YES YES YES MPLS 3.19 YES YES YES UFID 4.0 YES YES NO Megaflows 3.12 YES YES NO Masked set action 4.0 YES YES NO Recirculation 3.19 YES YES YES TCP flags matching 3.13 YES YES NO Validate flow actions YES YES N/A NO Multiple datapaths YES YES YES NO Tunnel TSO - STT N/A YES NO YES ===================== ============== ============== ========= ======= Q: What DPDK version does each Open vSwitch release work with? A: The following table lists the DPDK version against which the given versions of Open vSwitch will successfully build. ============ ======= Open vSwitch DPDK ============ ======= 2.2.x 1.6 2.3.x 1.6 2.4.x 2.0 2.5.x 2.2 2.6.x 16.07.2 2.7.x 16.11.4 2.8.x 17.05.2 2.9.x 17.11 ============ ======= Q: I get an error like this when I configure Open vSwitch: configure: error: Linux kernel in is version , but version newer than is not supported (please refer to the FAQ for advice) What should I do? A: You have the following options: - Use the Linux kernel module supplied with the kernel that you are using. (See also the following FAQ.) - If there is a newer released version of Open vSwitch, consider building that one, because it may support the kernel that you are building against. (To find out, consult the table in the previous FAQ.) - The Open vSwitch "master" branch may support the kernel that you are using, so consider building the kernel module from "master". All versions of Open vSwitch userspace are compatible with all versions of the Open vSwitch kernel module, so you do not have to use the kernel module from one source along with the userspace programs from the same source. Q: What features are not available in the Open vSwitch kernel datapath that ships as part of the upstream Linux kernel? A: The kernel module in upstream Linux does not include support for LISP. Work is in progress to add support for LISP to the upstream Linux version of the Open vSwitch kernel module. For now, if you need this feature, use the kernel module from the Open vSwitch distribution instead of the upstream Linux kernel module. Certain features require kernel support to function or to have reasonable performance. If the ovs-vswitchd log file indicates that a feature is not supported, consider upgrading to a newer upstream Linux release or using the kernel module paired with the userspace distribution. Q: Why do tunnels not work when using a kernel module other than the one packaged with Open vSwitch? A: Support for tunnels was added to the upstream Linux kernel module after the rest of Open vSwitch. As a result, some kernels may contain support for Open vSwitch but not tunnels. The minimum kernel version that supports each tunnel protocol is: ======== ============ Protocol Linux Kernel ======== ============ GRE 3.11 VXLAN 3.12 Geneve 3.18 LISP not upstream STT not upstream ======== ============ If you are using a version of the kernel that is older than the one listed above, it is still possible to use that tunnel protocol. However, you must compile and install the kernel module included with the Open vSwitch distribution rather than the one on your machine. If problems persist after doing this, check to make sure that the module that is loaded is the one you expect. Q: Why are UDP tunnel checksums not computed for VXLAN or Geneve? A: Generating outer UDP checksums requires kernel support that was not part of the initial implementation of these protocols. If using the upstream Linux Open vSwitch module, you must use kernel 4.0 or newer. The out-of-tree modules from Open vSwitch release 2.4 and later support UDP checksums. Q: What features are not available when using the userspace datapath? A: Tunnel virtual ports are not supported, as described in the previous answer. It is also not possible to use queue-related actions. On Linux kernels before 2.6.39, maximum-sized VLAN packets may not be transmitted. Q: Should userspace or kernel be upgraded first to minimize downtime? A. In general, the Open vSwitch userspace should be used with the kernel version included in the same release or with the version from upstream Linux. However, when upgrading between two releases of Open vSwitch it is best to migrate userspace first to reduce the possibility of incompatibilities. Q: What happened to the bridge compatibility feature? A: Bridge compatibility was a feature of Open vSwitch 1.9 and earlier. When it was enabled, Open vSwitch imitated the interface of the Linux kernel "bridge" module. This allowed users to drop Open vSwitch into environments designed to use the Linux kernel bridge module without adapting the environment to use Open vSwitch. Open vSwitch 1.10 and later do not support bridge compatibility. The feature was dropped because version 1.10 adopted a new internal architecture that made bridge compatibility difficult to maintain. Now that many environments use OVS directly, it would be rarely useful in any case. To use bridge compatibility, install OVS 1.9 or earlier, including the accompanying kernel modules (both the main and bridge compatibility modules), following the instructions that come with the release. Be sure to start the ovs-brcompatd daemon. ovs-2.9.0/Documentation/faq/terminology.rst000066400000000000000000000023441324262074100210170ustar00rootroot00000000000000.. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. Convention for heading levels in Open vSwitch documentation: ======= Heading 0 (reserved for the title in a document) ------- Heading 1 ~~~~~~~ Heading 2 +++++++ Heading 3 ''''''' Heading 4 Avoid deeper levels because they do not render well. =========== Terminology =========== Q: I thought Open vSwitch was a virtual Ethernet switch, but the documentation keeps talking about bridges. What's a bridge? A: In networking, the terms "bridge" and "switch" are synonyms. Open vSwitch implements an Ethernet switch, which means that it is also an Ethernet bridge. Q: What's a VLAN? A: See :doc:`vlan`. ovs-2.9.0/Documentation/faq/vlan.rst000066400000000000000000000316501324262074100174110ustar00rootroot00000000000000.. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. Convention for heading levels in Open vSwitch documentation: ======= Heading 0 (reserved for the title in a document) ------- Heading 1 ~~~~~~~ Heading 2 +++++++ Heading 3 ''''''' Heading 4 Avoid deeper levels because they do not render well. ===== VLANs ===== Q: What's a VLAN? A: At the simplest level, a VLAN (short for "virtual LAN") is a way to partition a single switch into multiple switches. Suppose, for example, that you have two groups of machines, group A and group B. You want the machines in group A to be able to talk to each other, and you want the machine in group B to be able to talk to each other, but you don't want the machines in group A to be able to talk to the machines in group B. You can do this with two switches, by plugging the machines in group A into one switch and the machines in group B into the other switch. If you only have one switch, then you can use VLANs to do the same thing, by configuring the ports for machines in group A as VLAN "access ports" for one VLAN and the ports for group B as "access ports" for a different VLAN. The switch will only forward packets between ports that are assigned to the same VLAN, so this effectively subdivides your single switch into two independent switches, one for each group of machines. So far we haven't said anything about VLAN headers. With access ports, like we've described so far, no VLAN header is present in the Ethernet frame. This means that the machines (or switches) connected to access ports need not be aware that VLANs are involved, just like in the case where we use two different physical switches. Now suppose that you have a whole bunch of switches in your network, instead of just one, and that some machines in group A are connected directly to both switches 1 and 2. To allow these machines to talk to each other, you could add an access port for group A's VLAN to switch 1 and another to switch 2, and then connect an Ethernet cable between those ports. That works fine, but it doesn't scale well as the number of switches and the number of VLANs increases, because you use up a lot of valuable switch ports just connecting together your VLANs. This is where VLAN headers come in. Instead of using one cable and two ports per VLAN to connect a pair of switches, we configure a port on each switch as a VLAN "trunk port". Packets sent and received on a trunk port carry a VLAN header that says what VLAN the packet belongs to, so that only two ports total are required to connect the switches, regardless of the number of VLANs in use. Normally, only switches (either physical or virtual) are connected to a trunk port, not individual hosts, because individual hosts don't expect to see a VLAN header in the traffic that they receive. None of the above discussion says anything about particular VLAN numbers. This is because VLAN numbers are completely arbitrary. One must only ensure that a given VLAN is numbered consistently throughout a network and that different VLANs are given different numbers. (That said, VLAN 0 is usually synonymous with a packet that has no VLAN header, and VLAN 4095 is reserved.) Q: VLANs don't work. A: Many drivers in Linux kernels before version 3.3 had VLAN-related bugs. If you are having problems with VLANs that you suspect to be driver related, then you have several options: - Upgrade to Linux 3.3 or later. - Build and install a fixed version of the particular driver that is causing trouble, if one is available. - Use a NIC whose driver does not have VLAN problems. - Use "VLAN splinters", a feature in Open vSwitch 1.4 upto 2.5 that works around bugs in kernel drivers. To enable VLAN splinters on interface eth0, use the command:: $ ovs-vsctl set interface eth0 other-config:enable-vlan-splinters=true For VLAN splinters to be effective, Open vSwitch must know which VLANs are in use. See the "VLAN splinters" section in the Interface table in ovs-vswitchd.conf.db(5) for details on how Open vSwitch infers in-use VLANs. VLAN splinters increase memory use and reduce performance, so use them only if needed. - Apply the "vlan workaround" patch from the XenServer kernel patch queue, build Open vSwitch against this patched kernel, and then use ovs-vlan-bug-workaround(8) to enable the VLAN workaround for each interface whose driver is buggy. (This is a nontrivial exercise, so this option is included only for completeness.) It is not always easy to tell whether a Linux kernel driver has buggy VLAN support. The ovs-vlan-test(8) and ovs-test(8) utilities can help you test. See their manpages for details. Of the two utilities, ovs-test(8) is newer and more thorough, but ovs-vlan-test(8) may be easier to use. Q: VLANs still don't work. I've tested the driver so I know that it's OK. A: Do you have VLANs enabled on the physical switch that OVS is attached to? Make sure that the port is configured to trunk the VLAN or VLANs that you are using with OVS. Q: Outgoing VLAN-tagged traffic goes through OVS to my physical switch and to its destination host, but OVS seems to drop incoming return traffic. A: It's possible that you have the VLAN configured on your physical switch as the "native" VLAN. In this mode, the switch treats incoming packets either tagged with the native VLAN or untagged as part of the native VLAN. It may also send outgoing packets in the native VLAN without a VLAN tag. If this is the case, you have two choices: - Change the physical switch port configuration to tag packets it forwards to OVS with the native VLAN instead of forwarding them untagged. - Change the OVS configuration for the physical port to a native VLAN mode. For example, the following sets up a bridge with port eth0 in "native-tagged" mode in VLAN 9:: $ ovs-vsctl add-br br0 $ ovs-vsctl add-port br0 eth0 tag=9 vlan_mode=native-tagged In this situation, "native-untagged" mode will probably work equally well. Refer to the documentation for the Port table in ovs-vswitchd.conf.db(5) for more information. Q: I added a pair of VMs on different VLANs, like this:: $ ovs-vsctl add-br br0 $ ovs-vsctl add-port br0 eth0 $ ovs-vsctl add-port br0 tap0 tag=9 $ ovs-vsctl add-port br0 tap1 tag=10 but the VMs can't access each other, the external network, or the Internet. A: It is to be expected that the VMs can't access each other. VLANs are a means to partition a network. When you configured tap0 and tap1 as access ports for different VLANs, you indicated that they should be isolated from each other. As for the external network and the Internet, it seems likely that the machines you are trying to access are not on VLAN 9 (or 10) and that the Internet is not available on VLAN 9 (or 10). Q: I added a pair of VMs on the same VLAN, like this:: $ ovs-vsctl add-br br0 $ ovs-vsctl add-port br0 eth0 $ ovs-vsctl add-port br0 tap0 tag=9 $ ovs-vsctl add-port br0 tap1 tag=9 The VMs can access each other, but not the external network or the Internet. A: It seems likely that the machines you are trying to access in the external network are not on VLAN 9 and that the Internet is not available on VLAN 9. Also, ensure VLAN 9 is set up as an allowed trunk VLAN on the upstream switch port to which eth0 is connected. Q: Can I configure an IP address on a VLAN? A: Yes. Use an "internal port" configured as an access port. For example, the following configures IP address 192.168.0.7 on VLAN 9. That is, OVS will forward packets from eth0 to 192.168.0.7 only if they have an 802.1Q header with VLAN 9. Conversely, traffic forwarded from 192.168.0.7 to eth0 will be tagged with an 802.1Q header with VLAN 9:: $ ovs-vsctl add-br br0 $ ovs-vsctl add-port br0 eth0 $ ovs-vsctl add-port br0 vlan9 tag=9 \ -- set interface vlan9 type=internal $ ip addr add 192.168.0.7/24 dev vlan9 $ ip link set vlan0 up See also the following question. Q: I configured one IP address on VLAN 0 and another on VLAN 9, like this:: $ ovs-vsctl add-br br0 $ ovs-vsctl add-port br0 eth0 $ ip addr add 192.168.0.5/24 dev br0 $ ip link set br0 up $ ovs-vsctl add-port br0 vlan9 tag=9 -- set interface vlan9 type=internal $ ip addr add 192.168.0.9/24 dev vlan9 $ ip link set vlan0 up but other hosts that are only on VLAN 0 can reach the IP address configured on VLAN 9. What's going on? A: `RFC 1122 section 3.3.4.2 "Multihoming Requirements" `__ describes two approaches to IP address handling in Internet hosts: - In the "Strong ES Model", where an ES is a host ("End System"), an IP address is primarily associated with a particular interface. The host discards packets that arrive on interface A if they are destined for an IP address that is configured on interface B. The host never sends packets from interface A using a source address configured on interface B. - In the "Weak ES Model", an IP address is primarily associated with a host. The host accepts packets that arrive on any interface if they are destined for any of the host's IP addresses, even if the address is configured on some interface other than the one on which it arrived. The host does not restrict itself to sending packets from an IP address associated with the originating interface. Linux uses the weak ES model. That means that when packets destined to the VLAN 9 IP address arrive on eth0 and are bridged to br0, the kernel IP stack accepts them there for the VLAN 9 IP address, even though they were not received on vlan9, the network device for vlan9. To simulate the strong ES model on Linux, one may add iptables rule to filter packets based on source and destination address and adjust ARP configuration with sysctls. BSD uses the strong ES model. Q: My OpenFlow controller doesn't see the VLANs that I expect. A: The configuration for VLANs in the Open vSwitch database (e.g. via ovs-vsctl) only affects traffic that goes through Open vSwitch's implementation of the OpenFlow "normal switching" action. By default, when Open vSwitch isn't connected to a controller and nothing has been manually configured in the flow table, all traffic goes through the "normal switching" action. But, if you set up OpenFlow flows on your own, through a controller or using ovs-ofctl or through other means, then you have to implement VLAN handling yourself. You can use "normal switching" as a component of your OpenFlow actions, e.g. by putting "normal" into the lists of actions on ovs-ofctl or by outputting to OFPP_NORMAL from an OpenFlow controller. In situations where this is not suitable, you can implement VLAN handling yourself, e.g.: - If a packet comes in on an access port, and the flow table needs to send it out on a trunk port, then the flow can add the appropriate VLAN tag with the "mod_vlan_vid" action. - If a packet comes in on a trunk port, and the flow table needs to send it out on an access port, then the flow can strip the VLAN tag with the "strip_vlan" action. Q: I configured ports on a bridge as access ports with different VLAN tags, like this:: $ ovs-vsctl add-br br0 $ ovs-vsctl set-controller br0 tcp:192.168.0.10:6653 $ ovs-vsctl add-port br0 eth0 $ ovs-vsctl add-port br0 tap0 tag=9 $ ovs-vsctl add-port br0 tap1 tag=10 but the VMs running behind tap0 and tap1 can still communicate, that is, they are not isolated from each other even though they are on different VLANs. A: Do you have a controller configured on br0 (as the commands above do)? If so, then this is a variant on the previous question, "My OpenFlow controller doesn't see the VLANs that I expect," and you can refer to the answer there for more information. Q: How MAC learning works with VLANs? A: Open vSwitch implements Independent VLAN Learning (IVL) for ``OFPP_NORMAL`` action, e.g. it logically has separate learning tables for each VLANs. ovs-2.9.0/Documentation/faq/vxlan.rst000066400000000000000000000042021324262074100175720ustar00rootroot00000000000000.. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. Convention for heading levels in Open vSwitch documentation: ======= Heading 0 (reserved for the title in a document) ------- Heading 1 ~~~~~~~ Heading 2 +++++++ Heading 3 ''''''' Heading 4 Avoid deeper levels because they do not render well. ====== VXLANs ====== Q: What's a VXLAN? A: VXLAN stands for Virtual eXtensible Local Area Network, and is a means to solve the scaling challenges of VLAN networks in a multi-tenant environment. VXLAN is an overlay network which transports an L2 network over an existing L3 network. For more information on VXLAN, please see `RFC 7348 `__. Q: How much of the VXLAN protocol does Open vSwitch currently support? A: Open vSwitch currently supports the framing format for packets on the wire. There is currently no support for the multicast aspects of VXLAN. To get around the lack of multicast support, it is possible to pre-provision MAC to IP address mappings either manually or from a controller. Q: What destination UDP port does the VXLAN implementation in Open vSwitch use? A: By default, Open vSwitch will use the assigned IANA port for VXLAN, which is 4789. However, it is possible to configure the destination UDP port manually on a per-VXLAN tunnel basis. An example of this configuration is provided below.:: $ ovs-vsctl add-br br0 $ ovs-vsctl add-port br0 vxlan1 -- set interface vxlan1 type=vxlan \ options:remote_ip=192.168.1.2 options:key=flow options:dst_port=8472 ovs-2.9.0/Documentation/group-selection-method-property.txt000066400000000000000000000114271324262074100241700ustar00rootroot00000000000000Proposal for Group Selection Method Property Version: 0.0.3 Author: Simon Horman , et al. Initial Public Revision: September 2014 Contents ======== 1. Introduction 2. How it Works 3. Experimenter Id 4. Experimenter Messages 5. History 1. Introduction =============== This text describes a Netronome Extension to (draft) OpenFlow 1.5 that allows a controller to provide more information on the selection method for select groups. This proposal is in the form of an enhanced select group type. This may subsequently be proposed as an extension or update to the OpenFlow specification. 2. How it works =============== A new Netronome group experimenter property is defined which provides compatibility with the group mod message defined in draft Open Flow 1.5 (also known as ONF EXT-350) and allows parameters for the selection method of select groups to be passed by the controller. In particular it allows controllers to: * Specify the fields used for bucket selection by the select group. * Designate the selection method used. * Provide a non-field parameter to the selection method. 3. Experimenter ID ================== The Experimenter ID of this extension is: NTR_VENDOR_ID = 0x0000154d 4. Group Experimenter Property ============================== The following group property experimenter type defined by this extension. enum ntr_group_mod_subtype { NTRT_SELECTION_METHOD = 1, }; Modifications to the group table from the controller may be done with a OFPT_GROUP_MOD message described (draft) Open Flow 1.5. Group Entry Message. Of relevance here is that (draft) Open Flow 1.5 group messages have properties. This proposal is defined in terms of an implementation of struct ofp_group_prop_experimenter which is described in (draft) Open Flow 1.5. The implementation is: struct ntr_group_prop_selection_method { ovs_be16 type; /* OFPGPT_EXPERIMENTER. */ ovs_be16 length; /* Length in bytes of this property. */ ovs_be32 experimenter; /* NTR_VENDOR_ID. */ ovs_be32 exp_type; /* NTRT_SELECTION_METHOD. */ ovs_be32 pad; char selection_method[NTR_MAX_SELECTION_METHOD_LEN]; /* Null-terminated */ ovs_be64 selection_method_param; /* Non-Field parameter for * bucket selection. */ /* Followed by: * - Exactly (length - 40) (possibly 0) bytes containing OXM TLVs, then * - Exactly ((length + 7)/8*8 - length) (between 0 and 7) bytes of * all-zero bytes * In summary, ntr_group_prop_selection_method is padded as needed, * to make its overall size a multiple of 8, to preserve alignment * in structures using it. */ /* uint8_t field_array[0]; */ /* Zero or more fields encoded as * OXM TLVs where the has_mask bit must * be zero and the value it specifies is * a mask to apply to packet fields and * then input them to the selection * method of a select group. */ /* uint8_t pad2[0]; */ }; OFP_ASSERT(sizeof(struct ntr_group_mod) == 40); This property may only be used with group mod messages whose: * command is OFPGC_ADD or OFPGC_MODIFY; and * type is OFPGT_SELECT The type field is the OFPGPT_EXPERIMENTER which is defined in EXT-350 as 0xffff. The experimenter field is the Experimenter ID (see 3). The exp_type field is NTRT_SELECTION_METHOD. The group selection_method is a null-terminated string which if non-zero length specifies a selection method known to an underlying layer of the switch. The value of NTR_MAX_SELECTION_METHOD_LEN is 16. The group selection_method may be zero-length to request compatibility with Open Flow 1.4. The selection_method_param provides a non-field parameter for the group selection_method. It must be all-zeros unless the group selection_method is non-zero length. The selection_method_param may for example be used as an initial value for the hash of a hash group selection method. The fields field is an ofp_match structure which includes the fields which should be used as inputs to bucket selection. ofp_match is described in Open Flow 1.4 section 7.2.2 Flow Match Structures. Fields must not be specified unless the group selection_method is non-zero length. The pre-requisites for fields specified must be satisfied in the match for any flow that uses the group. Masking is allowed but not required for fields whose TLVs allow masking. The fields may for example be used as the fields that are hashed by a hash group selection method. 5. History ========== This proposal has been developed independently of any similar work in this area. No such work is known. ovs-2.9.0/Documentation/howto/000077500000000000000000000000001324262074100163035ustar00rootroot00000000000000ovs-2.9.0/Documentation/howto/docker.rst000066400000000000000000000271331324262074100203120ustar00rootroot00000000000000.. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. Convention for heading levels in Open vSwitch documentation: ======= Heading 0 (reserved for the title in a document) ------- Heading 1 ~~~~~~~ Heading 2 +++++++ Heading 3 ''''''' Heading 4 Avoid deeper levels because they do not render well. =================================== Open Virtual Networking With Docker =================================== This document describes how to use Open Virtual Networking with Docker 1.9.0 or later. .. important:: Requires Docker version 1.9.0 or later. Only Docker 1.9.0+ comes with support for multi-host networking. Consult www.docker.com for instructions on how to install Docker. .. note:: You must build and install Open vSwitch before proceeding with the below guide. Refer to :doc:`/intro/install/index` for more information. Setup ----- For multi-host networking with OVN and Docker, Docker has to be started with a destributed key-value store. For example, if you decide to use consul as your distributed key-value store and your host IP address is ``$HOST_IP``, start your Docker daemon with:: $ docker daemon --cluster-store=consul://127.0.0.1:8500 \ --cluster-advertise=$HOST_IP:0 OVN provides network virtualization to containers. OVN's integration with Docker currently works in two modes - the "underlay" mode or the "overlay" mode. In the "underlay" mode, OVN requires a OpenStack setup to provide container networking. In this mode, one can create logical networks and can have containers running inside VMs, standalone VMs (without having any containers running inside them) and physical machines connected to the same logical network. This is a multi-tenant, multi-host solution. In the "overlay" mode, OVN can create a logical network amongst containers running on multiple hosts. This is a single-tenant (extendable to multi-tenants depending on the security characteristics of the workloads), multi-host solution. In this mode, you do not need a pre-created OpenStack setup. For both the modes to work, a user has to install and start Open vSwitch in each VM/host that they plan to run their containers on. .. _docker-overlay: The "overlay" mode ------------------ .. note:: OVN in "overlay" mode needs a minimum Open vSwitch version of 2.5. 1. Start the central components. OVN architecture has a central component which stores your networking intent in a database. On one of your machines, with an IP Address of ``$CENTRAL_IP``, where you have installed and started Open vSwitch, you will need to start some central components. Start ovn-northd daemon. This daemon translates networking intent from Docker stored in the OVN\_Northbound database to logical flows in ``OVN_Southbound`` database. For example:: $ /usr/share/openvswitch/scripts/ovn-ctl start_northd With Open vSwitch version of 2.7 or greater, you need to run the following additional commands (Please read the manpages of ovn-nb for more control on the types of connection allowed.) :: $ ovn-nbctl set-connection ptcp:6641 $ ovn-sbctl set-connection ptcp:6642 2. One time setup On each host, where you plan to spawn your containers, you will need to run the below command once. You may need to run it again if your OVS database gets cleared. It is harmless to run it again in any case:: $ ovs-vsctl set Open_vSwitch . \ external_ids:ovn-remote="tcp:$CENTRAL_IP:6642" \ external_ids:ovn-nb="tcp:$CENTRAL_IP:6641" \ external_ids:ovn-encap-ip=$LOCAL_IP \ external_ids:ovn-encap-type="$ENCAP_TYPE" where: ``$LOCAL_IP`` is the IP address via which other hosts can reach this host. This acts as your local tunnel endpoint. ``$ENCAP_TYPE`` is the type of tunnel that you would like to use for overlay networking. The options are ``geneve`` or ``stt``. Your kernel must have support for your chosen ``$ENCAP_TYPE``. Both ``geneve`` and ``stt`` are part of the Open vSwitch kernel module that is compiled from this repo. If you use the Open vSwitch kernel module from upstream Linux, you will need a minumum kernel version of 3.18 for ``geneve``. There is no ``stt`` support in upstream Linux. You can verify whether you have the support in your kernel as follows:: $ lsmod | grep $ENCAP_TYPE In addition, each Open vSwitch instance in an OVN deployment needs a unique, persistent identifier, called the ``system-id``. If you install OVS from distribution packaging for Open vSwitch (e.g. .deb or .rpm packages), or if you use the ovs-ctl utility included with Open vSwitch, it automatically configures a system-id. If you start Open vSwitch manually, you should set one up yourself. For example:: $ id_file=/etc/openvswitch/system-id.conf $ test -e $id_file || uuidgen > $id_file $ ovs-vsctl set Open_vSwitch . external_ids:system-id=$(cat $id_file) 3. Start the ``ovn-controller``. You need to run the below command on every boot:: $ /usr/share/openvswitch/scripts/ovn-ctl start_controller 4. Start the Open vSwitch network driver. By default Docker uses Linux bridge for networking. But it has support for external drivers. To use Open vSwitch instead of the Linux bridge, you will need to start the Open vSwitch driver. The Open vSwitch driver uses the Python's flask module to listen to Docker's networking api calls. So, if your host does not have Python's flask module, install it:: $ sudo pip install Flask Start the Open vSwitch driver on every host where you plan to create your containers. Refer to the note on ``$OVS_PYTHON_LIBS_PATH`` that is used below at the end of this document:: $ PYTHONPATH=$OVS_PYTHON_LIBS_PATH ovn-docker-overlay-driver --detach .. note:: The ``$OVS_PYTHON_LIBS_PATH`` variable should point to the directory where Open vSwitch Python modules are installed. If you installed Open vSwitch Python modules via the Debian package of ``python-openvswitch`` or via pip by running ``pip install ovs``, you do not need to specify the PATH. If you installed it by following the instructions in :doc:`/intro/install/general`, then you should specify the PATH. In this case, the PATH depends on the options passed to ``./configure``. It is usually either ``/usr/share/openvswitch/python`` or ``/usr/local/share/openvswitch/python`` Docker has inbuilt primitives that closely match OVN's logical switches and logical port concepts. Consult Docker's documentation for all the possible commands. Here are some examples. Create a logical switch ~~~~~~~~~~~~~~~~~~~~~~~ To create a logical switch with name 'foo', on subnet '192.168.1.0/24', run:: $ NID=`docker network create -d openvswitch --subnet=192.168.1.0/24 foo` List all logical switches ~~~~~~~~~~~~~~~~~~~~~~~~~ :: $ docker network ls You can also look at this logical switch in OVN's northbound database by running the following command:: $ ovn-nbctl --db=tcp:$CENTRAL_IP:6640 ls-list Delete a logical switch ~~~~~~~~~~~~~~~~~~~~~~~ :: $ docker network rm bar Create a logical port ~~~~~~~~~~~~~~~~~~~~~ Docker creates your logical port and attaches it to the logical network in a single step. For example, to attach a logical port to network ``foo`` inside container busybox, run:: $ docker run -itd --net=foo --name=busybox busybox List all logical ports ~~~~~~~~~~~~~~~~~~~~~~ Docker does not currently have a CLI command to list all logical ports but you can look at them in the OVN database by running:: $ ovn-nbctl --db=tcp:$CENTRAL_IP:6640 lsp-list $NID Create and attach a logical port to a running container ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ :: $ docker network create -d openvswitch --subnet=192.168.2.0/24 bar $ docker network connect bar busybox Detach and delete a logical port from a running container ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ You can delete your logical port and detach it from a running container by running: :: $ docker network disconnect bar busybox .. _docker-underlay: The "underlay" mode ------------------- .. note:: This mode requires that you have a OpenStack setup pre-installed with OVN providing the underlay networking. 1. One time setup A OpenStack tenant creates a VM with a single network interface (or multiple) that belongs to management logical networks. The tenant needs to fetch the port-id associated with the interface via which he plans to send the container traffic inside the spawned VM. This can be obtained by running the below command to fetch the 'id' associated with the VM:: $ nova list and then by running:: $ neutron port-list --device_id=$id Inside the VM, download the OpenStack RC file that contains the tenant information (henceforth referred to as ``openrc.sh``). Edit the file and add the previously obtained port-id information to the file by appending the following line:: $ export OS_VIF_ID=$port_id After this edit, the file will look something like:: #!/bin/bash export OS_AUTH_URL=http://10.33.75.122:5000/v2.0 export OS_TENANT_ID=fab106b215d943c3bad519492278443d export OS_TENANT_NAME="demo" export OS_USERNAME="demo" export OS_VIF_ID=e798c371-85f4-4f2d-ad65-d09dd1d3c1c9 2. Create the Open vSwitch bridge If your VM has one ethernet interface (e.g.: 'eth0'), you will need to add that device as a port to an Open vSwitch bridge 'breth0' and move its IP address and route related information to that bridge. (If it has multiple network interfaces, you will need to create and attach an Open vSwitch bridge for the interface via which you plan to send your container traffic.) If you use DHCP to obtain an IP address, then you should kill the DHCP client that was listening on the physical Ethernet interface (e.g. eth0) and start one listening on the Open vSwitch bridge (e.g. breth0). Depending on your VM, you can make the above step persistent across reboots. For example, if your VM is Debian/Ubuntu-based, read `openvswitch-switch.README.Debian` found in `debian` folder. If your VM is RHEL-based, refer to :doc:`/intro/install/rhel`. 3. Start the Open vSwitch network driver The Open vSwitch driver uses the Python's flask module to listen to Docker's networking api calls. The driver also uses OpenStack's ``python-neutronclient`` libraries. If your host does not have Python's ``flask`` module or ``python-neutronclient`` you must install them. For example:: $ pip install python-neutronclient $ pip install Flask Once installed, source the ``openrc`` file:: $ . ./openrc.sh Start the network driver and provide your OpenStack tenant password when prompted:: $ PYTHONPATH=$OVS_PYTHON_LIBS_PATH ovn-docker-underlay-driver \ --bridge breth0 --detach From here-on you can use the same Docker commands as described in `docker-overlay`_. Refer the the ovs-architecture man pages (``man ovn-architecture``) to understand OVN's architecture in detail. ovs-2.9.0/Documentation/howto/dpdk.rst000066400000000000000000000640671324262074100177740ustar00rootroot00000000000000.. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. Convention for heading levels in Open vSwitch documentation: ======= Heading 0 (reserved for the title in a document) ------- Heading 1 ~~~~~~~ Heading 2 +++++++ Heading 3 ''''''' Heading 4 Avoid deeper levels because they do not render well. ============================ Using Open vSwitch with DPDK ============================ This document describes how to use Open vSwitch with DPDK datapath. .. important:: Using the DPDK datapath requires building OVS with DPDK support. Refer to :doc:`/intro/install/dpdk` for more information. Ports and Bridges ----------------- ovs-vsctl can be used to set up bridges and other Open vSwitch features. Bridges should be created with a ``datapath_type=netdev``:: $ ovs-vsctl add-br br0 -- set bridge br0 datapath_type=netdev ovs-vsctl can also be used to add DPDK devices. ovs-vswitchd should print the number of dpdk devices found in the log file:: $ ovs-vsctl add-port br0 dpdk-p0 -- set Interface dpdk-p0 type=dpdk \ options:dpdk-devargs=0000:01:00.0 $ ovs-vsctl add-port br0 dpdk-p1 -- set Interface dpdk-p1 type=dpdk \ options:dpdk-devargs=0000:01:00.1 Some NICs (i.e. Mellanox ConnectX-3) have only one PCI address associated with multiple ports. Using a PCI device like above won't work. Instead, below usage is suggested:: $ ovs-vsctl add-port br0 dpdk-p0 -- set Interface dpdk-p0 type=dpdk \ options:dpdk-devargs="class=eth,mac=00:11:22:33:44:55:01" $ ovs-vsctl add-port br0 dpdk-p1 -- set Interface dpdk-p1 type=dpdk \ options:dpdk-devargs="class=eth,mac=00:11:22:33:44:55:02" Note: such syntax won't support hotplug. The hotplug is supposed to work with future DPDK release, v18.05. After the DPDK ports get added to switch, a polling thread continuously polls DPDK devices and consumes 100% of the core, as can be checked from ``top`` and ``ps`` commands:: $ top -H $ ps -eLo pid,psr,comm | grep pmd Creating bonds of DPDK interfaces is slightly different to creating bonds of system interfaces. For DPDK, the interface type and devargs must be explicitly set. For example:: $ ovs-vsctl add-bond br0 dpdkbond p0 p1 \ -- set Interface p0 type=dpdk options:dpdk-devargs=0000:01:00.0 \ -- set Interface p1 type=dpdk options:dpdk-devargs=0000:01:00.1 To stop ovs-vswitchd & delete bridge, run:: $ ovs-appctl -t ovs-vswitchd exit $ ovs-appctl -t ovsdb-server exit $ ovs-vsctl del-br br0 PMD Thread Statistics --------------------- To show current stats:: $ ovs-appctl dpif-netdev/pmd-stats-show To clear previous stats:: $ ovs-appctl dpif-netdev/pmd-stats-clear Port/RXQ Assigment to PMD Threads --------------------------------- To show port/rxq assignment:: $ ovs-appctl dpif-netdev/pmd-rxq-show To change default rxq assignment to pmd threads, rxqs may be manually pinned to desired cores using:: $ ovs-vsctl set Interface \ other_config:pmd-rxq-affinity= where: - ```` is a CSV list of ``:`` values For example:: $ ovs-vsctl set interface dpdk-p0 options:n_rxq=4 \ other_config:pmd-rxq-affinity="0:3,1:7,3:8" This will ensure: - Queue #0 pinned to core 3 - Queue #1 pinned to core 7 - Queue #2 not pinned - Queue #3 pinned to core 8 After that PMD threads on cores where RX queues was pinned will become ``isolated``. This means that this thread will poll only pinned RX queues. .. warning:: If there are no ``non-isolated`` PMD threads, ``non-pinned`` RX queues will not be polled. Also, if provided ``core_id`` is not available (ex. this ``core_id`` not in ``pmd-cpu-mask``), RX queue will not be polled by any PMD thread. If pmd-rxq-affinity is not set for rxqs, they will be assigned to pmds (cores) automatically. The processing cycles that have been stored for each rxq will be used where known to assign rxqs to pmd based on a round robin of the sorted rxqs. For example, in the case where here there are 5 rxqs and 3 cores (e.g. 3,7,8) available, and the measured usage of core cycles per rxq over the last interval is seen to be: - Queue #0: 30% - Queue #1: 80% - Queue #3: 60% - Queue #4: 70% - Queue #5: 10% The rxqs will be assigned to cores 3,7,8 in the following order: Core 3: Q1 (80%) | Core 7: Q4 (70%) | Q5 (10%) core 8: Q3 (60%) | Q0 (30%) To see the current measured usage history of pmd core cycles for each rxq:: $ ovs-appctl dpif-netdev/pmd-rxq-show .. note:: A history of one minute is recorded and shown for each rxq to allow for traffic pattern spikes. An rxq's pmd core cycles usage changes due to traffic pattern or reconfig changes will take one minute before they are fully reflected in the stats. Rxq to pmds assignment takes place whenever there are configuration changes or can be triggered by using:: $ ovs-appctl dpif-netdev/pmd-rxq-rebalance QoS --- Assuming you have a vhost-user port transmitting traffic consisting of packets of size 64 bytes, the following command would limit the egress transmission rate of the port to ~1,000,000 packets per second:: $ ovs-vsctl set port vhost-user0 qos=@newqos -- \ --id=@newqos create qos type=egress-policer other-config:cir=46000000 \ other-config:cbs=2048` To examine the QoS configuration of the port, run:: $ ovs-appctl -t ovs-vswitchd qos/show vhost-user0 To clear the QoS configuration from the port and ovsdb, run:: $ ovs-vsctl destroy QoS vhost-user0 -- clear Port vhost-user0 qos Refer to vswitch.xml for more details on egress-policer. Rate Limiting -------------- Here is an example on Ingress Policing usage. Assuming you have a vhost-user port receiving traffic consisting of packets of size 64 bytes, the following command would limit the reception rate of the port to ~1,000,000 packets per second:: $ ovs-vsctl set interface vhost-user0 ingress_policing_rate=368000 \ ingress_policing_burst=1000` To examine the ingress policer configuration of the port:: $ ovs-vsctl list interface vhost-user0 To clear the ingress policer configuration from the port:: $ ovs-vsctl set interface vhost-user0 ingress_policing_rate=0 Refer to vswitch.xml for more details on ingress-policer. Flow Control ------------ Flow control can be enabled only on DPDK physical ports. To enable flow control support at tx side while adding a port, run:: $ ovs-vsctl add-port br0 dpdk-p0 -- set Interface dpdk-p0 type=dpdk \ options:dpdk-devargs=0000:01:00.0 options:tx-flow-ctrl=true Similarly, to enable rx flow control, run:: $ ovs-vsctl add-port br0 dpdk-p0 -- set Interface dpdk-p0 type=dpdk \ options:dpdk-devargs=0000:01:00.0 options:rx-flow-ctrl=true To enable flow control auto-negotiation, run:: $ ovs-vsctl add-port br0 dpdk-p0 -- set Interface dpdk-p0 type=dpdk \ options:dpdk-devargs=0000:01:00.0 options:flow-ctrl-autoneg=true To turn ON the tx flow control at run time for an existing port, run:: $ ovs-vsctl set Interface dpdk-p0 options:tx-flow-ctrl=true The flow control parameters can be turned off by setting ``false`` to the respective parameter. To disable the flow control at tx side, run:: $ ovs-vsctl set Interface dpdk-p0 options:tx-flow-ctrl=false pdump ----- pdump allows you to listen on DPDK ports and view the traffic that is passing on them. To use this utility, one must have libpcap installed on the system. Furthermore, DPDK must be built with ``CONFIG_RTE_LIBRTE_PDUMP=y`` and ``CONFIG_RTE_LIBRTE_PMD_PCAP=y``. .. warning:: A performance decrease is expected when using a monitoring application like the DPDK pdump app. To use pdump, simply launch OVS as usual, then navigate to the ``app/pdump`` directory in DPDK, ``make`` the application and run like so:: $ sudo ./build/app/dpdk-pdump -- \ --pdump port=0,queue=0,rx-dev=/tmp/pkts.pcap \ --server-socket-path=/usr/local/var/run/openvswitch The above command captures traffic received on queue 0 of port 0 and stores it in ``/tmp/pkts.pcap``. Other combinations of port numbers, queues numbers and pcap locations are of course also available to use. For example, to capture all packets that traverse port 0 in a single pcap file:: $ sudo ./build/app/dpdk-pdump -- \ --pdump 'port=0,queue=*,rx-dev=/tmp/pkts.pcap,tx-dev=/tmp/pkts.pcap' \ --server-socket-path=/usr/local/var/run/openvswitch ``server-socket-path`` must be set to the value of ``ovs_rundir()`` which typically resolves to ``/usr/local/var/run/openvswitch``. Many tools are available to view the contents of the pcap file. Once example is tcpdump. Issue the following command to view the contents of ``pkts.pcap``:: $ tcpdump -r pkts.pcap More information on the pdump app and its usage can be found in the `DPDK docs `__. Jumbo Frames ------------ By default, DPDK ports are configured with standard Ethernet MTU (1500B). To enable Jumbo Frames support for a DPDK port, change the Interface's ``mtu_request`` attribute to a sufficiently large value. For example, to add a DPDK Phy port with MTU of 9000:: $ ovs-vsctl add-port br0 dpdk-p0 -- set Interface dpdk-p0 type=dpdk \ options:dpdk-devargs=0000:01:00.0 mtu_request=9000 Similarly, to change the MTU of an existing port to 6200:: $ ovs-vsctl set Interface dpdk-p0 mtu_request=6200 Some additional configuration is needed to take advantage of jumbo frames with vHost ports: 1. *mergeable buffers* must be enabled for vHost ports, as demonstrated in the QEMU command line snippet below:: -netdev type=vhost-user,id=mynet1,chardev=char0,vhostforce \ -device virtio-net-pci,mac=00:00:00:00:00:01,netdev=mynet1,mrg_rxbuf=on 2. Where virtio devices are bound to the Linux kernel driver in a guest environment (i.e. interfaces are not bound to an in-guest DPDK driver), the MTU of those logical network interfaces must also be increased to a sufficiently large value. This avoids segmentation of Jumbo Frames received in the guest. Note that 'MTU' refers to the length of the IP packet only, and not that of the entire frame. To calculate the exact MTU of a standard IPv4 frame, subtract the L2 header and CRC lengths (i.e. 18B) from the max supported frame size. So, to set the MTU for a 9018B Jumbo Frame:: $ ip link set eth1 mtu 9000 When Jumbo Frames are enabled, the size of a DPDK port's mbuf segments are increased, such that a full Jumbo Frame of a specific size may be accommodated within a single mbuf segment. Jumbo frame support has been validated against 9728B frames, which is the largest frame size supported by Fortville NIC using the DPDK i40e driver, but larger frames and other DPDK NIC drivers may be supported. These cases are common for use cases involving East-West traffic only. Rx Checksum Offload ------------------- By default, DPDK physical ports are enabled with Rx checksum offload. Rx checksum offload can offer performance improvement only for tunneling traffic in OVS-DPDK because the checksum validation of tunnel packets is offloaded to the NIC. Also enabling Rx checksum may slightly reduce the performance of non-tunnel traffic, specifically for smaller size packet. .. _extended-statistics: Extended & Custom Statistics ---------------------------- DPDK Extended Statistics API allows PMD to expose unique set of statistics. The Extended statistics are implemented and supported only for DPDK physical and vHost ports. Custom statistics are dynamic set of counters which can vary depenend on a driver. Those statistics are implemented for DPDK physical ports and contain all "dropped", "error" and "management" counters from XSTATS. XSTATS counters list can be found here: `__. To enable statistics, you have to enable OpenFlow 1.4 support for OVS. Configure bridge br0 to support OpenFlow version 1.4:: $ ovs-vsctl set bridge br0 datapath_type=netdev \ protocols=OpenFlow10,OpenFlow11,OpenFlow12,OpenFlow13,OpenFlow14 Check the OVSDB protocols column in the bridge table if OpenFlow 1.4 support is enabled for OVS:: $ ovsdb-client dump Bridge protocols Query the port statistics by explicitly specifying -O OpenFlow14 option:: $ ovs-ofctl -O OpenFlow14 dump-ports br0 Note about "Extended Statistics": vHost ports supports only partial statistics. RX packet size based counter are only supported and doesn't include TX packet size counters. .. _port-hotplug: Port Hotplug ------------ OVS supports port hotplugging, allowing the use of ports that were not bound to DPDK when vswitchd was started. In order to attach a port, it has to be bound to DPDK using the ``dpdk_nic_bind.py`` script:: $ $DPDK_DIR/tools/dpdk_nic_bind.py --bind=igb_uio 0000:01:00.0 Then it can be attached to OVS:: $ ovs-vsctl add-port br0 dpdkx -- set Interface dpdkx type=dpdk \ options:dpdk-devargs=0000:01:00.0 Detaching will be performed while processing del-port command:: $ ovs-vsctl del-port dpdkx Sometimes, the del-port command may not detach the device. Detaching can be confirmed by the appearance of an INFO log. For example:: INFO|Device '0000:04:00.1' has been detached If the log is not seen, then the port can be detached using:: $ ovs-appctl netdev-dpdk/detach 0000:01:00.0 Detaching can be confirmed by console output:: Device '0000:04:00.1' has been detached .. warning:: Detaching should not be done if a device is known to be non-detachable, as this may cause the device to behave improperly when added back with add-port. The Chelsio Terminator adapters which use the cxgbe driver seem to be an example of this behavior; check the driver documentation if this is suspected. This feature does not work with some NICs. For more information please refer to the `DPDK Port Hotplug Framework `__. .. _vdev-support: Vdev Support ------------ DPDK provides drivers for both physical and virtual devices. Physical DPDK devices are added to OVS by specifying a valid PCI address in 'dpdk-devargs'. Virtual DPDK devices which do not have PCI addresses can be added using a different format for 'dpdk-devargs'. Typically, the format expected is 'eth_' where 'x' is a unique identifier of your choice for the given port. For example to add a dpdk port that uses the 'null' DPDK PMD driver:: $ ovs-vsctl add-port br0 null0 -- set Interface null0 type=dpdk \ options:dpdk-devargs=eth_null0 Similarly, to add a dpdk port that uses the 'af_packet' DPDK PMD driver:: $ ovs-vsctl add-port br0 myeth0 -- set Interface myeth0 type=dpdk \ options:dpdk-devargs=eth_af_packet0,iface=eth0 More information on the different types of virtual DPDK PMDs can be found in the `DPDK documentation `__. Note: Not all DPDK virtual PMD drivers have been tested and verified to work. EMC Insertion Probability ------------------------- By default 1 in every 100 flows are inserted into the Exact Match Cache (EMC). It is possible to change this insertion probability by setting the ``emc-insert-inv-prob`` option:: $ ovs-vsctl --no-wait set Open_vSwitch . other_config:emc-insert-inv-prob=N where: ``N`` is a positive integer representing the inverse probability of insertion ie. on average 1 in every N packets with a unique flow will generate an EMC insertion. If ``N`` is set to 1, an insertion will be performed for every flow. If set to 0, no insertions will be performed and the EMC will effectively be disabled. With default ``N`` set to 100, higher megaflow hits will occur initially as observed with pmd stats:: $ ovs-appctl dpif-netdev/pmd-stats-show For certain traffic profiles with many parallel flows, it's recommended to set ``N`` to '0' to achieve higher forwarding performance. For more information on the EMC refer to :doc:`/intro/install/dpdk` . .. _dpdk-ovs-in-guest: OVS with DPDK Inside VMs ------------------------ Additional configuration is required if you want to run ovs-vswitchd with DPDK backend inside a QEMU virtual machine. ovs-vswitchd creates separate DPDK TX queues for each CPU core available. This operation fails inside QEMU virtual machine because, by default, VirtIO NIC provided to the guest is configured to support only single TX queue and single RX queue. To change this behavior, you need to turn on ``mq`` (multiqueue) property of all ``virtio-net-pci`` devices emulated by QEMU and used by DPDK. You may do it manually (by changing QEMU command line) or, if you use Libvirt, by adding the following string to ```` sections of all network devices used by DPDK:: where: ``N`` determines how many queues can be used by the guest. This requires QEMU >= 2.2. .. _dpdk-phy-phy: PHY-PHY ------- Add a userspace bridge and two ``dpdk`` (PHY) ports:: # Add userspace bridge $ ovs-vsctl add-br br0 -- set bridge br0 datapath_type=netdev # Add two dpdk ports $ ovs-vsctl add-port br0 phy0 -- set Interface phy0 type=dpdk \ options:dpdk-devargs=0000:01:00.0 ofport_request=1 $ ovs-vsctl add-port br0 phy1 -- set Interface phy1 type=dpdk options:dpdk-devargs=0000:01:00.1 ofport_request=2 Add test flows to forward packets betwen DPDK port 0 and port 1:: # Clear current flows $ ovs-ofctl del-flows br0 # Add flows between port 1 (phy0) to port 2 (phy1) $ ovs-ofctl add-flow br0 in_port=1,action=output:2 $ ovs-ofctl add-flow br0 in_port=2,action=output:1 Transmit traffic into either port. You should see it returned via the other. .. _dpdk-vhost-loopback: PHY-VM-PHY (vHost Loopback) --------------------------- Add a userspace bridge, two ``dpdk`` (PHY) ports, and two ``dpdkvhostuser`` ports:: # Add userspace bridge $ ovs-vsctl add-br br0 -- set bridge br0 datapath_type=netdev # Add two dpdk ports $ ovs-vsctl add-port br0 phy0 -- set Interface phy0 type=dpdk \ options:dpdk-devargs=0000:01:00.0 ofport_request=1 $ ovs-vsctl add-port br0 phy1 -- set Interface phy1 type=dpdk options:dpdk-devargs=0000:01:00.1 ofport_request=2 # Add two dpdkvhostuser ports $ ovs-vsctl add-port br0 dpdkvhostuser0 \ -- set Interface dpdkvhostuser0 type=dpdkvhostuser ofport_request=3 $ ovs-vsctl add-port br0 dpdkvhostuser1 \ -- set Interface dpdkvhostuser1 type=dpdkvhostuser ofport_request=4 Add test flows to forward packets betwen DPDK devices and VM ports:: # Clear current flows $ ovs-ofctl del-flows br0 # Add flows $ ovs-ofctl add-flow br0 in_port=1,action=output:3 $ ovs-ofctl add-flow br0 in_port=3,action=output:1 $ ovs-ofctl add-flow br0 in_port=4,action=output:2 $ ovs-ofctl add-flow br0 in_port=2,action=output:4 # Dump flows $ ovs-ofctl dump-flows br0 Create a VM using the following configuration: .. table:: ===================== ======== ============ Configuration Values Comments ===================== ======== ============ QEMU version 2.2.0 n/a QEMU thread affinity core 5 taskset 0x20 Memory 4GB n/a Cores 2 n/a Qcow2 image CentOS7 n/a mrg_rxbuf off n/a ===================== ======== ============ You can do this directly with QEMU via the ``qemu-system-x86_64`` application:: $ export VM_NAME=vhost-vm $ export GUEST_MEM=3072M $ export QCOW2_IMAGE=/root/CentOS7_x86_64.qcow2 $ export VHOST_SOCK_DIR=/usr/local/var/run/openvswitch $ taskset 0x20 qemu-system-x86_64 -name $VM_NAME -cpu host -enable-kvm \ -m $GUEST_MEM -drive file=$QCOW2_IMAGE --nographic -snapshot \ -numa node,memdev=mem -mem-prealloc -smp sockets=1,cores=2 \ -object memory-backend-file,id=mem,size=$GUEST_MEM,mem-path=/dev/hugepages,share=on \ -chardev socket,id=char0,path=$VHOST_SOCK_DIR/dpdkvhostuser0 \ -netdev type=vhost-user,id=mynet1,chardev=char0,vhostforce \ -device virtio-net-pci,mac=00:00:00:00:00:01,netdev=mynet1,mrg_rxbuf=off \ -chardev socket,id=char1,path=$VHOST_SOCK_DIR/dpdkvhostuser1 \ -netdev type=vhost-user,id=mynet2,chardev=char1,vhostforce \ -device virtio-net-pci,mac=00:00:00:00:00:02,netdev=mynet2,mrg_rxbuf=off For a explanation of this command, along with alternative approaches such as booting the VM via libvirt, refer to :doc:`/topics/dpdk/vhost-user`. Once the guest is configured and booted, configure DPDK packet forwarding within the guest. To accomplish this, build the ``testpmd`` application as described in :ref:`dpdk-testpmd`. Once compiled, run the application:: $ cd $DPDK_DIR/app/test-pmd; $ ./testpmd -c 0x3 -n 4 --socket-mem 1024 -- \ --burst=64 -i --txqflags=0xf00 --disable-hw-vlan $ set fwd mac retry $ start When you finish testing, bind the vNICs back to kernel:: $ $DPDK_DIR/usertools/dpdk-devbind.py --bind=virtio-pci 0000:00:03.0 $ $DPDK_DIR/usertools/dpdk-devbind.py --bind=virtio-pci 0000:00:04.0 .. note:: Valid PCI IDs must be passed in above example. The PCI IDs can be retrieved like so:: $ $DPDK_DIR/usertools/dpdk-devbind.py --status More information on the dpdkvhostuser ports can be found in :doc:`/topics/dpdk/vhost-user`. PHY-VM-PHY (vHost Loopback) (Kernel Forwarding) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ :ref:`dpdk-vhost-loopback` details steps for PHY-VM-PHY loopback testcase and packet forwarding using DPDK testpmd application in the Guest VM. For users wishing to do packet forwarding using kernel stack below, you need to run the below commands on the guest:: $ ip addr add 1.1.1.2/24 dev eth1 $ ip addr add 1.1.2.2/24 dev eth2 $ ip link set eth1 up $ ip link set eth2 up $ systemctl stop firewalld.service $ systemctl stop iptables.service $ sysctl -w net.ipv4.ip_forward=1 $ sysctl -w net.ipv4.conf.all.rp_filter=0 $ sysctl -w net.ipv4.conf.eth1.rp_filter=0 $ sysctl -w net.ipv4.conf.eth2.rp_filter=0 $ route add -net 1.1.2.0/24 eth2 $ route add -net 1.1.1.0/24 eth1 $ arp -s 1.1.2.99 DE:AD:BE:EF:CA:FE $ arp -s 1.1.1.99 DE:AD:BE:EF:CA:EE PHY-VM-PHY (vHost Multiqueue) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ vHost Multiqueue functionality can also be validated using the PHY-VM-PHY configuration. To begin, follow the steps described in :ref:`dpdk-phy-phy` to create and initialize the database, start ovs-vswitchd and add ``dpdk``-type devices to bridge ``br0``. Once complete, follow the below steps: 1. Configure PMD and RXQs. For example, set the number of dpdk port rx queues to at least 2 The number of rx queues at vhost-user interface gets automatically configured after virtio device connection and doesn't need manual configuration:: $ ovs-vsctl set Open_vSwitch . other_config:pmd-cpu-mask=0xc $ ovs-vsctl set Interface phy0 options:n_rxq=2 $ ovs-vsctl set Interface phy1 options:n_rxq=2 2. Instantiate Guest VM using QEMU cmdline We must configure with appropriate software versions to ensure this feature is supported. .. list-table:: Recommended BIOS Settings :header-rows: 1 * - Setting - Value * - QEMU version - 2.5.0 * - QEMU thread affinity - 2 cores (taskset 0x30) * - Memory - 4 GB * - Cores - 2 * - Distro - Fedora 22 * - Multiqueue - Enabled To do this, instantiate the guest as follows:: $ export VM_NAME=vhost-vm $ export GUEST_MEM=4096M $ export QCOW2_IMAGE=/root/Fedora22_x86_64.qcow2 $ export VHOST_SOCK_DIR=/usr/local/var/run/openvswitch $ taskset 0x30 qemu-system-x86_64 -cpu host -smp 2,cores=2 -m 4096M \ -drive file=$QCOW2_IMAGE --enable-kvm -name $VM_NAME \ -nographic -numa node,memdev=mem -mem-prealloc \ -object memory-backend-file,id=mem,size=$GUEST_MEM,mem-path=/dev/hugepages,share=on \ -chardev socket,id=char1,path=$VHOST_SOCK_DIR/dpdkvhostuser0 \ -netdev type=vhost-user,id=mynet1,chardev=char1,vhostforce,queues=2 \ -device virtio-net-pci,mac=00:00:00:00:00:01,netdev=mynet1,mq=on,vectors=6 \ -chardev socket,id=char2,path=$VHOST_SOCK_DIR/dpdkvhostuser1 \ -netdev type=vhost-user,id=mynet2,chardev=char2,vhostforce,queues=2 \ -device virtio-net-pci,mac=00:00:00:00:00:02,netdev=mynet2,mq=on,vectors=6 .. note:: Queue value above should match the queues configured in OVS, The vector value should be set to "number of queues x 2 + 2" 3. Configure the guest interface Assuming there are 2 interfaces in the guest named eth0, eth1 check the channel configuration and set the number of combined channels to 2 for virtio devices:: $ ethtool -l eth0 $ ethtool -L eth0 combined 2 $ ethtool -L eth1 combined 2 More information can be found in vHost walkthrough section. 4. Configure kernel packet forwarding Configure IP and enable interfaces:: $ ip addr add 5.5.5.1/24 dev eth0 $ ip addr add 90.90.90.1/24 dev eth1 $ ip link set eth0 up $ ip link set eth1 up Configure IP forwarding and add route entries:: $ sysctl -w net.ipv4.ip_forward=1 $ sysctl -w net.ipv4.conf.all.rp_filter=0 $ sysctl -w net.ipv4.conf.eth0.rp_filter=0 $ sysctl -w net.ipv4.conf.eth1.rp_filter=0 $ ip route add 2.1.1.0/24 dev eth1 $ route add default gw 2.1.1.2 eth1 $ route add default gw 90.90.90.90 eth1 $ arp -s 90.90.90.90 DE:AD:BE:EF:CA:FE $ arp -s 2.1.1.2 DE:AD:BE:EF:CA:FA Check traffic on multiple queues:: $ cat /proc/interrupts | grep virtio ovs-2.9.0/Documentation/howto/firewalld.rst000066400000000000000000000101151324262074100210040ustar00rootroot00000000000000.. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. Convention for heading levels in Open vSwitch documentation: ======= Heading 0 (reserved for the title in a document) ------- Heading 1 ~~~~~~~ Heading 2 +++++++ Heading 3 ''''''' Heading 4 Avoid deeper levels because they do not render well. =================================== Open Virtual Network With firewalld =================================== firewalld is a service that allows for easy administration of firewalls. OVN ships with a set of service files that can be used with firewalld to allow for remote connections to the northbound and southbound databases. This guide will describe how you can use these files with your existing firewalld setup. Setup and administration of firewalld is outside the scope of this document. Installation ------------ If you have installed OVN from an RPM, then the service files for firewalld will automatically be installed in ``/usr/lib/firewalld/services``. Installation from RPM includes installation from the yum or dnf package managers. If you have installed OVN from source, then from the top level source directory, issue the following commands to copy the firewalld service files: :: $ cp rhel/usr_lib_firewalld_services_ovn-central-firewall-service.xml \ /etc/firewalld/services/ $ cp rhel/usr_lib_firewalld_services_ovn-host-firewall-service.xml \ /etc/firewalld/services/ Activation ---------- Assuming you are already running firewalld, you can issue the following commands to enable the OVN services. On the central server (the one running ``ovn-northd``), issue the following:: $ firewall-cmd --zone=public --add-service=ovn-central-firewall-service This will open TCP ports 6641 and 6642, allowing for remote connections to the northbound and southbound databases. On the OVN hosts (the ones running ``ovn-controller``), issue the following:: $ firewall-cmd --zone=public --add-service=ovn-host-firewall-service This will open UDP port 6081, allowing for geneve traffic to flow between the controllers. Variations ---------- When installing the XML service files, you have the choice of copying them to ``/etc/firewalld/services`` or ``/usr/lib/firewalld/services``. The former is recommened since the latter can be overwritten if firewalld is upgraded. The above commands assumed your underlay network interfaces are in the "public" firewalld zone. If your underlay network interfaces are in a separate zone, then adjust the above commands accordingly. The ``--permanent`` option may be passed to the above firewall-cmd invocations in order for the services to be permanently added to the firewalld configuration. This way it is not necessary to re-issue the commands each time the firewalld service restarts. The ovn-host-firewall-service only opens port 6081. This is because the default protocol for OVN tunnels is geneve. If you are using a different encapsulation protocol, you will need to modify the XML service file to open the appropriate port(s). For VXLAN, open port 4789. For STT, open port 7471. Recommendations --------------- The firewalld service files included with the OVS repo are meant as a convenience for firewalld users. All that the service files do is to open the common ports used by OVN. No additional security is provided. To ensure a more secure environment, it is a good idea to do the following * Use tools such as iptables or nftables to restrict access to known hosts. * Use SSL for all remote connections to OVN databases. * Use role-based access control for connections to the OVN southbound database. ovs-2.9.0/Documentation/howto/index.rst000066400000000000000000000025161324262074100201500ustar00rootroot00000000000000.. Copyright (c) 2016, Stephen Finucane Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. Convention for heading levels in Open vSwitch documentation: ======= Heading 0 (reserved for the title in a document) ------- Heading 1 ~~~~~~~ Heading 2 +++++++ Heading 3 ''''''' Heading 4 Avoid deeper levels because they do not render well. ============= How-to Guides ============= Answers to common "How do I?"-style questions. For more information on the topics covered herein, refer to :doc:`/topics/index`. OVS --- .. toctree:: :maxdepth: 2 kvm selinux libvirt ssl lisp tunneling userspace-tunneling vlan qos vtep sflow dpdk OVN --- .. toctree:: :maxdepth: 1 docker openstack-containers firewalld ovs-2.9.0/Documentation/howto/kvm.rst000066400000000000000000000060651324262074100176410ustar00rootroot00000000000000.. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. Convention for heading levels in Open vSwitch documentation: ======= Heading 0 (reserved for the title in a document) ------- Heading 1 ~~~~~~~ Heading 2 +++++++ Heading 3 ''''''' Heading 4 Avoid deeper levels because they do not render well. ===================== Open vSwitch with KVM ===================== This document describes how to use Open vSwitch with the Kernel-based Virtual Machine (KVM). .. note:: This document assumes that you have Open vSwitch set up on a Linux system. Setup ----- KVM uses tunctl to handle various bridging modes, which you can install with the Debian/Ubuntu package ``uml-utilities``:: $ apt-get install uml-utilities Next, you will need to modify or create custom versions of the ``qemu-ifup`` and ``qemu-ifdown`` scripts. In this guide, we'll create custom versions that make use of example Open vSwitch bridges that we'll describe in this guide. Create the following two files and store them in known locations. For example:: $ cat << 'EOF' > /etc/ovs-ifup #!/bin/sh switch='br0' ip link set $1 up ovs-vsctl add-port ${switch} $1 EOF :: $ cat << 'EOF' > /etc/ovs-ifdown #!/bin/sh switch='br0' ip addr flush dev $1 ip link set $1 down ovs-vsctl del-port ${switch} $1 EOF The basic usage of Open vSwitch is described at the end of :doc:`/intro/install/general`. If you haven't already, create a bridge named ``br0`` with the following command:: $ ovs-vsctl add-br br0 Then, add a port to the bridge for the NIC that you want your guests to communicate over (e.g. ``eth0``):: $ ovs-vsctl add-port br0 eth0 Refer to ovs-vsctl(8) for more details. Next, we'll start a guest that will use our ifup and ifdown scripts:: $ kvm -m 512 -net nic,macaddr=00:11:22:EE:EE:EE -net \ tap,script=/etc/ovs-ifup,downscript=/etc/ovs-ifdown -drive \ file=/path/to/disk-image,boot=on This will start the guest and associate a tap device with it. The ``ovs-ifup`` script will add a port on the br0 bridge so that the guest will be able to communicate over that bridge. To get some more information and for debugging you can use Open vSwitch utilities such as ovs-dpctl and ovs-ofctl, For example:: $ ovs-dpctl show $ ovs-ofctl show br0 You should see tap devices for each KVM guest added as ports to the bridge (e.g. tap0) Refer to ovs-dpctl(8) and ovs-ofctl(8) for more details. Bug Reporting ------------- Please report problems to bugs@openvswitch.org. ovs-2.9.0/Documentation/howto/libvirt.rst000066400000000000000000000061551324262074100205170ustar00rootroot00000000000000.. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. Convention for heading levels in Open vSwitch documentation: ======= Heading 0 (reserved for the title in a document) ------- Heading 1 ~~~~~~~ Heading 2 +++++++ Heading 3 ''''''' Heading 4 Avoid deeper levels because they do not render well. ========================= Open vSwitch with Libvirt ========================= This document describes how to use Open vSwitch with Libvirt 0.9.11 or later. This document assumes that you followed :doc:`/intro/install/general` or installed Open vSwitch from distribution packaging such as a .deb or .rpm. The Open vSwitch support is included by default in Libvirt 0.9.11. Consult www.libvirt.org for instructions on how to build the latest Libvirt, if your Linux distribution by default comes with an older Libvirt release. Limitations ----------- Currently there is no Open vSwitch support for networks that are managed by libvirt (e.g. NAT). As of now, only bridged networks are supported (those where the user has to manually create the bridge). Setup ----- First, create the Open vSwitch bridge by using the ovs-vsctl utility (this must be done with administrative privileges):: $ ovs-vsctl add-br ovsbr Once that is done, create a VM, if necessary, and edit its Domain XML file:: $ virsh edit Lookup in the Domain XML file the ```` section. There should be one such XML section for each interface the VM has::
And change it to something like this::
The interface type must be set to ``bridge``. The ```` XML element specifies to which bridge this interface will be attached to. The ```` element indicates that the bridge in ```` element is an Open vSwitch bridge. Then (re)start the VM and verify if the guest's vnet interface is attached to the ovsbr bridge:: $ ovs-vsctl show Troubleshooting --------------- If the VM does not want to start, then try to run the libvirtd process either from the terminal, so that all errors are printed in console, or inspect Libvirt/Open vSwitch log files for possible root cause. Bug Reporting ------------- Report problems to bugs@openvswitch.org. ovs-2.9.0/Documentation/howto/lisp.rst000066400000000000000000000114111324262074100200020ustar00rootroot00000000000000.. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. Convention for heading levels in Open vSwitch documentation: ======= Heading 0 (reserved for the title in a document) ------- Heading 1 ~~~~~~~ Heading 2 +++++++ Heading 3 ''''''' Heading 4 Avoid deeper levels because they do not render well. ==================== Using LISP tunneling ==================== LISP is a layer 3 tunneling mechanism, meaning that encapsulated packets do not carry Ethernet headers, and ARP requests shouldn't be sent over the tunnel. Because of this, there are some additional steps required for setting up LISP tunnels in Open vSwitch, until support for L3 tunnels will improve. This guide assumes tunneling between two VMs connected to OVS bridges on different hypervisors reachable over IPv4. Of course, more than one VM may be connected to any of the hypervisors, and a hypervisor may communicate with several different hypervisors over the same lisp tunneling interface. A LISP "map-cache" can be implemented using flows, see example at the bottom of this file. There are several scenarios: 1) the VMs have IP addresses in the same subnet and the hypervisors are also in a single subnet (although one different from the VM's); 2) the VMs have IP addresses in the same subnet but the hypervisors are separated by a router; 3) the VMs are in different subnets. In cases 1) and 3) ARP resolution can work as normal: ARP traffic is configured not to go through the LISP tunnel. For case 1) ARP is able to reach the other VM, if both OVS instances default to MAC address learning. Case 3) requires the hypervisor be configured as the default router for the VMs. In case 2) the VMs expect ARP replies from each other, but this is not possible over a layer 3 tunnel. One solution is to have static MAC address entries preconfigured on the VMs (e.g., ``arp -f /etc/ethers`` on startup on Unix based VMs), or have the hypervisor do proxy ARP. In this scenario, the eth0 interfaces need not be added to the br0 bridge in the examples below. On the receiving side, the packet arrives without the original MAC header. The LISP tunneling code attaches a header with harcoded source and destination MAC address ``02:00:00:00:00:00``. This address has all bits set to 0, except the locally administered bit, in order to avoid potential collisions with existing allocations. In order for packets to reach their intended destination, the destination MAC address needs to be rewritten. This can be done using the flow table. See below for an example setup, and the associated flow rules to enable LISP tunneling. :: Diagram +---+ +---+ |VM1| |VM2| +---+ +---+ | | +--[tap0]--+ +--[tap0]---+ | | | | [lisp0] OVS1 [eth0]-----------------[eth0] OVS2 [lisp0] | | | | +----------+ +-----------+ On each hypervisor, interfaces tap0, eth0, and lisp0 are added to a single bridge instance, and become numbered 1, 2, and 3 respectively: :: $ ovs-vsctl add-br br0 $ ovs-vsctl add-port br0 tap0 $ ovs-vsctl add-port br0 eth0 $ ovs-vsctl add-port br0 lisp0 \ -- set Interface lisp0 type=lisp options:remote_ip=flow options:key=flow The last command sets up flow based tunneling on the lisp0 interface. From the LISP point of view, this is like having the Tunnel Router map cache implemented as flow rules. Flows on br0 should be configured as follows: :: priority=3,dl_dst=02:00:00:00:00:00,action=mod_dl_dst:,output:1 priority=2,in_port=1,dl_type=0x0806,action=NORMAL priority=1,in_port=1,dl_type=0x0800,vlan_tci=0,nw_src=,action=set_field:->tun_dst,output:3 priority=0,action=NORMAL The third rule is like a map cache entry: the ```` specified by the ``nw_src`` match field is mapped to the RLOC ````, which is set as the tunnel destination for this particular flow. Optionally, if you want to use Instance ID in a flow, you can add ``set_tunnel:`` to the action list. ovs-2.9.0/Documentation/howto/openstack-containers.rst000066400000000000000000000156111324262074100231730ustar00rootroot00000000000000.. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. Convention for heading levels in Open vSwitch documentation: ======= Heading 0 (reserved for the title in a document) ------- Heading 1 ~~~~~~~ Heading 2 +++++++ Heading 3 ''''''' Heading 4 Avoid deeper levels because they do not render well. ================================================ Integration of Containers with OVN and OpenStack ================================================ Isolation between containers is weaker than isolation between VMs, so some environments deploy containers for different tenants in separate VMs as an additional security measure. This document describes creation of containers inside VMs and how they can be made part of the logical networks securely. The created logical network can include VMs, containers and physical machines as endpoints. To better understand the proposed integration of containers with OVN and OpenStack, this document describes the end to end workflow with an example. * A OpenStack tenant creates a VM (say VM-A) with a single network interface that belongs to a management logical network. The VM is meant to host containers. OpenStack Nova chooses the hypervisor on which VM-A is created. * A Neutron port may have been created in advance and passed in to Nova with the request to create a new VM. If not, Nova will issue a request to Neutron to create a new port. The ID of the logical port from Neutron will also be used as the vif-id for the virtual network interface (VIF) of VM-A. * When VM-A is created on a hypervisor, its VIF gets added to the Open vSwitch integration bridge. This creates a row in the Interface table of the ``Open_vSwitch`` database. As explained in the :doc:`integration guide `, the vif-id associated with the VM network interface gets added in the ``external_ids:iface-id`` column of the newly created row in the Interface table. * Since VM-A belongs to a logical network, it gets an IP address. This IP address is used to spawn containers (either manually or through container orchestration systems) inside that VM and to monitor the health of the created containers. * The vif-id associated with the VM's network interface can be obtained by making a call to Neutron using tenant credentials. * This flow assumes a component called a "container network plugin". If you take Docker as an example for containers, you could envision the plugin to be either a wrapper around Docker or a feature of Docker itself that understands how to perform part of this workflow to get a container connected to a logical network managed by Neutron. The rest of the flow refers to this logical component that does not yet exist as the "container network plugin". * All the calls to Neutron will need tenant credentials. These calls can either be made from inside the tenant VM as part of a container network plugin or from outside the tenant VM (if the tenant is not comfortable using temporary Keystone tokens from inside the tenant VMs). For simplicity, this document explains the work flow using the former method. * The container hosting VM will need Open vSwitch installed in it. The only work for Open vSwitch inside the VM is to tag network traffic coming from containers. * When a container needs to be created inside the VM with a container network interface that is expected to be attached to a particular logical switch, the network plugin in that VM chooses any unused VLAN (This VLAN tag only needs to be unique inside that VM. This limits the number of container interfaces to 4096 inside a single VM). This VLAN tag is stripped out in the hypervisor by OVN and is only useful as a context (or metadata) for OVN. * The container network plugin then makes a call to Neutron to create a logical port. In addition to all the inputs that a call to create a port in Neutron that are currently needed, it sends the vif-id and the VLAN tag as inputs. * Neutron in turn will verify that the vif-id belongs to the tenant in question and then uses the OVN specific plugin to create a new row in the Logical_Switch_Port table of the OVN Northbound Database. Neutron responds back with an IP address and MAC address for that network interface. So Neutron becomes the IPAM system and provides unique IP and MAC addresses across VMs and containers in the same logical network. * The Neutron API call above to create a logical port for the container could add a relatively significant amount of time to container creation. However, an optimization is possible here. Logical ports could be created in advance and reused by the container system doing container orchestration. Additional Neutron API calls would only be needed if the port needs to be attached to a different logical network. * When a container is eventually deleted, the network plugin in that VM may make a call to Neutron to delete that port. Neutron in turn will delete the entry in the ``Logical_Switch_Port`` table of the OVN Northbound Database. As an example, consider Docker containers. Since Docker currently does not have a network plugin feature, this example uses a hypothetical wrapper around Docker to make calls to Neutron. * Create a Logical switch:: $ ovn-docker --cred=cca86bd13a564ac2a63ddf14bf45d37f create network LS1 The above command will make a call to Neutron with the credentials to create a logical switch. The above is optional if the logical switch has already been created from outside the VM. * List networks available to the tenant:: $ ovn-docker --cred=cca86bd13a564ac2a63ddf14bf45d37f list networks * Create a container and attach a interface to the previously created switch as a logical port:: $ ovn-docker --cred=cca86bd13a564ac2a63ddf14bf45d37f --vif-id=$VIF_ID \ --network=LS1 run -d --net=none ubuntu:14.04 /bin/sh -c \ "while true; do echo hello world; sleep 1; done" The above command will make a call to Neutron with all the inputs it currently needs to create a logical port. In addition, it passes the $VIF_ID and a unused VLAN. Neutron will add that information in OVN and return back a MAC address and IP address for that interface. ovn-docker will then create a veth pair, insert one end inside the container as 'eth0' and the other end as a port of a local OVS bridge as an access port of the chosen VLAN. ovs-2.9.0/Documentation/howto/qos.png000066400000000000000000002103511324262074100176150ustar00rootroot00000000000000PNG  IHDRZ)LiCCPICC ProfilexYy8߷{&spdvyscL)JATH*dhPJE$L}5|s~Zkλ:QÃaFBB"l(.;+2\ ϥub3d=#B|X+<" *B`}3G "6/A J()1^~XPPz^Hm.zT?:T߂D&6 مG#WcBzBhoXό7yr!_>Cd P;%C=-`-#[#c!(m37<=)_iWOi lg1"n3D0QDo>0`d[f 2ݞ9P \0DR?T"ABp2" `9燌) g߳Q9 K"sl󶭋tHל%FVvV_hy"ZB ;*h]6Z#0hkFߘx5G=r/ \_ Ca~Q]dHRLC%)r7m{mbOE$DE 8-$qEF⎾y^1dd@" gyԁ0f ď? ) iPJ@9.F @S`| ;؀ !2BB$@Z!dB.B( : P5t u@gkh B0 &07, *.l{a?x9p\Mp~ Ÿe@ѡXQ|() JerE"PI,T> UjEFQs54MFSRH^}$t6} ݄B?GЛ" #QØb1~XL&SƼLbcXVVkubc v38vñ$p8+pU:^y#WPT|;444BjV޴񴹴x^Sz|7~NΆ.. }15AOC&ׄ%"(L!9j=qL/MoJM~~AA!!C?#-0>#1&2L#YBH٤ Iɐɛ)8E 듽i r7y,l|2&,,q,E,YFYQ¬;wqdG+ll:l>lYl l/)A'؛ه9699899889p\\\ʹyùra 3K BѥS (]>.>h |O6ES*y  R:#+","$),<#&b* R+N(-OLVLE,HSqX\Q_H_P8'L#**Y&9$EҕfNnq9!+)(,[!VIL.UU| Aŝ;}vJK1SS񧒲RRҬr J}UA655%(F/RA53">T Z-RQm>mvoJi]1@:y=Yz+j(c,'L#FF~FF ƊM0&&'LLMLM̔ͺ v,-",ZwvR2Բ XZg}kcmSd3e+g{׎lnWc^>CC#j'N2ΉΏ\8\\Z\q˻ w=GqOƞ"{>pp vNupAQ=M==x}93{wO߬\~@abI`IJUPUVSpCMGPРЮ0g޷aQ ElbF.}ѢчbbbVccőBO'%\܏ﵿ߁c$Ϥ΃N&'_J^:hyMZu77fg7nv޸%}6;;w&]not|{ދ.'{z޽yڃU6?RzԧwO4+ KOM@!:22zBݬ*952_R^H }."/;&oШȥE^M_ּ }68*&gL-jw[umBluO9Ĝ\\#v9{%jgWyB,5$9CBCg^k̏:k/c?PbGR3))!i҇820?r=hvc999sN;yӉQg <:K:Ow~dɅke*Yp/K{i=}LxɻuN$={aRmP`8jηʆO$;fAq2:ztLl]sCK烾(/`뾦/y|]ڇ?~~.J5dbp4,4Ӵ=JlB ѝބADG$0ؕ99/p pPrB҈H!R8s\IJ9VSNNHsL[L'DQo@0ͨɬ9ւRJZFVN^AȑӉ8qq}uυn=u'nPKq=.ngwᇃ<{gyЇWLnnxadѵиЄdTYGO'^ϋI[jtkY{êڭ C/ }ϡ4蛘vBDk/u | $,+YYŐqG [2i˜\#Y>K0BKlb~ْW+dS|4BPU3Q8YU]s^7W/I?PHؘxƤY+%u6IjvW;:;;.㷗wqwSȣ#/?PD . q % 7هw/"%R+GŘx$RX $$DĊPbC_2oܿ@ AWo:8%5 >-}]]]ߵ3@r !7FΏ>ثw!!"4< &o",8, :~a)xï(~)*U@}B"=t(E,L V`7qʸeG_Jd Jg8|*6x^̐э9ɜԃ2=d Kf*"K+;;VJW88]\]I}~+JX!3aAHdTS $} edv(J(+ȩ hjjhы570Z76u42ii`uzVr:8oRwram}zto ]?! z_,o܋VIߧ\I<䙩s5]v\$gr韣*+뮨,t&ε^L#u&| F.ѩEɸc~g/8^+56zm2sG%ʲJٍ[?G@!'2F ;JRg@RCK#$ A{8(jD|&Ê3| (6*uՃZB H~>d6H.~ +Öcq{pŸa 7M5 m m F$? zWBI~yx2|Y>7Vp`óc?ȡɱɕmdzM)d.4*XMARDXHtLlCeE%Ies7h\Zs$5]< #yPZqحىvsqnuޝ>O _L f 3HHxqpCڙ=Y9Y,O_K[.}!"oܥ[5G, W0ilYl}VyPOiL}ܣ7~vyKWoᇛ{~0Y:<8dJXxSqр'A*MFi4^2E*BH ZY` >(JYP8:]p#=3eaOc_xq:REɣU=Bҕ@Rߘac`& 2yȭa,,.Aϱ9S[B);_.P/&@xHdV Pt:,}CfJKIηJ*wHZGt< LLͽ,,C&mmGœ7]AE{z;bn"ws;^5Y_I?-C]=1rtҳNjRŔ_)^VM,pscV6ۣw;iv?T|qY3#_x I ۏľx|p䳩ƙď/J_Vn.F}:Tj^} ~5uՍs?LRDDCʏ#[[KH$?Olmmmm,G ?Wl c{q6OO~{/ pHYs   IDATx `ս'%H  T/%(hA-hJ(p+>@U(TAKKl TP噄lMvNv7!@v(3gw>gfw9s&NF$@$@$@N <$@$@$@$y@$@$@$ iY   j$@$@$@@&je   9@$@$@$ iY   j$@$@$@@&je   9@$@$@$ iY   j$@$@$@@&je   9@$@$@$ iY   j$@$@$@@&je   9@$@$@$ iY   j$@$@$@@&je   $ %PqoWL{OPŴ:﹬3؏8wԮꈏ[_[r7V~gF!>.7# :T[u礘AY 鋶m5XYSsinƶwǩ#ۺv;}IUysҗm]o%  `?MYjhK=-Qj} NnhE6ilW<2dJ䑖:d OK($@$@fOcANf7[#0lz[1mccQik' l{i'M4f̤uyVs\b]7l8HHHҸTzCL1'U[v&l_nJpƚ}/_7ާÑ5=GҬN> T&F|W{$E[_[$L$  @Ms$j̙GqX{oLIKOrrjc<<Ihd=g᭥W 4G/#F&uan$  Xb>wBBd앪eܗ&Ǥ+&ɂ-g>ۙRG_5;_ַONJKsIMO`*GK$@$@|Fȡ7 Sp\,qL>u=Gmq)Djq;:ǷmGſ$@$@ @M -G [6m0MYbz;L tLMكqc["欜>"eԍyFHHZ[̸wȡlKf;:fRb-3yجjDM:kQ2dxC^ $@$@-B4-ݷV֛[Q]钽ٸRRx:RvWذ*qm~N/z7c0 qɎRLcg]11s|[ݹ}R-2a_}|C֤QM w?Փ5jGS[O,/ \ jaGL]J6mP⤙ NsBgƣ>ME!};Ǐk׭X0[)cpL~3=1lV{!Xoijl+gZ"<†HqSHHJiJ*™F]N^=e}Byx9 K\ڴwً3ީ76.Ƹc7R[YQG?RBE4 OJ% 짹XԢRS!Y9droOY#] wZxʞXZm_h,C2rN DNj?w\{͡LGtt 4@X]]]1H a "=11-Vh\$.P' m& %@MuGIHH8dA @ ܺ$@$@$@N4Nt .j;ZN$@$@$$@MdA @ ܺ$@$@$@N4Nt .j;ZN$@$@$$@MdA @ ܺ$@$@$@N4Nt .j;ZN$@$@$$@MdA @ ܺ$@$@$@N4Nt .j;ZN$@$@$$@MdA @ ܺ$@$@$@N4Nt .j;ZN$@$@$$@MdA @ ܺ$@$@$@N4Nt .j;ZN$@$@$$@MdA @ ܺ$@$@$@N4Nt .j;ZN$@$@$$@MdA @ ܺ$@$@$@N4Nt .j;ZN$@$@$$@MdA @ ܺ$@$@$@N4Nt .j;ZN$@$@$$@MdA @[`4@XX "Q"H  qƇz ZaI+#G1|fG$\ +JG4ͅ @EDDsHH ƂZ~i 4tϔDGGCְ92  _'i$x5Ӡ4⚑0"4(9i*++^yU@ f!5qތ %kpB4KLH|L4 n k p.q3IrUV]]mNםq;t @CNPUhsk]f,t%UAS^^nN>t @GM/..]vQQQ|#}H@` }QHH? iݺ5~i4UG&22aoΞ2eJaa׌$&&{"LH|F 2kJF555hϺw.!4111MIbl+"`A1= ~7 Э['N\l%\xȚ'OFFFHEȅpuDIߵj儦$ DMJE6X9|;wUQ0G h{I2&,UDj1(tĚЬhUE ,4z}ӢCL\XyAkW?| X?Tl3|ƣJ_4M$@$p8Ӡ)EW%C 2oC[H2( kUW푦ԫY\Zviq1VD ¤$@$@MsHHH PC- $@$@$@4<HHH5M0"@$@$@$@MsHHH PC- $@$@$@4<HHH5M0"@$@$@$@MsHHH PC- $@$@$@4<HHH5M0"@$@$@$@MsHHH PC- $@$@$@4<HHH5M0"@$@$@$@MsHHH PC- $@$@$@4<HHH5M0"@$@$@$@MsHHH PC- $@$@$@4<HHH5M0"@$@$@$@MsHHH PC- $@$@$@4<HHH5M0"@$@$@$@MsHHH PC- $@$@$@4<HHH5M0"@$@$@$IJ k~cĴ{flNu[8\u^zj4vvz}b4\5ey/K`=Sv5GNxN#|}ZIO)%"B+++}d% ?$@Ms6mkm+F{5\ߎ9{brBC%S?,ݮ[GF^#ۣۧ)mhD{|V~'k@ط:4_S= ^SviJ jaIHi.xEbCapG:ž= SSSSi(: tjTѲ[s߁BwW_?!g X7|I[,9 ,j_>驉4a}dx2meg?RߏOd״CM+uTZ.9U/zh"('ϸ j˪C_>c$ܳcJ)C!iʎBy*0a"\yВ8:V;Ef8ua-Tу;5tjb *j[]ƨCtGxڳ5'_)jץSijVx6-Q}Zfєl( A$@$Ul>/_pvFZzoԜteb~;f 0"AKd4u;m0­2b4*&|}bM\>7Ȭ |Vs319ءٟ@m B$@$M4@2Q8k:h-xd>%xG Q;Ѧ#'d%f0(TIEq_Ze9 U,7 jfç#=}tc*:|ksmx}0x:azW5rS=ЏHZJ(-cEZɑ=^査xWu,Wncs2»'E#JcT'!-ސ_'a^N[nhy |[5Z9Z8׹8HZ5MP߼?{t,+^4ZBv7:iwh?S{*Ť[O-kۛn_lR`*ެ* *Q?QZ}<{sۛn^? HHHcO䩽e}]ThQفSi[Sn#m"yj+ċQ[XXن+! 4תX j˫DVϟ'ƄUվf;Ra3뒢2aP\}WWLߍj.3U^Ɍ" @MӜX^a=A}Ac֬rT t4gȬ % "NHHB5MV= N$@$@AE&!  %@MUς @P daHHH d Pӄlճ$@$@$Ti:Y  Y4![,8 jNHHB5MV= N$@$@AE&!  %@MUς @P daHHH d Pӄlճ$@$@$Ti:Y  Y4![,8 jNHHB5MV= N$@$@AE&!  %@MUς @P daHHH d Pӄlճ$@$@$Ti:Y  Y4![,8 jNHHB5MV= N$@$@AE&!  %@MUς @P daHHH d Dl)x]mon$+g :z ~   iAVڵkw٠?/ԩS~n囇-**a ~Nzbeobt;c 񃻺\g㪿lq܀:ǓOٝo|7.bot *Wۉ:؁sNxSO-ZtVWW1}ѿ=Ν K0[*bΜ9/BAA%pQQZ"ڷo?{lj sa^sbKcX޴Q@ڶN?@g,@ֵwдcGOj0IfUȊS\if<04MS Svj1bu]W^^^SSPkƋ󮭭Eo;＃?я&M]\Br!222&&)v1 x=ۆ vWWTTl6To?O~⃊@( Wv(e`Rܗ=~Ѽ4 Ļn3(^zBimLӘ򊏶4[^ܹ.2P NM4ژ6m@j`ʫ4/,,\~e|SRR`6,ĐsV68i"N> 8ƍ@\~xnIif(H7~1[1t)fٴhgTڅ!M5M6DoK*z8`ųg#fOBjX{kȸ:B# Ú~z<4hciڶmkXiЎW^`lSbB(j}iRޮ_{,PP6~XV}uAc[~Ǽgi>n؃ )o~mҗe;eLY܇vgj+oykqI˜'4Myy9D2g< s'?C#_}sVcƟO~82Ogg rO/ӳV+_OB+޿yKKԻ<--c8TL8Hm 췏\,I\lCSn}_e>e˿97Nn[ϭzR۸i 9"i'\zZ{ٗk=}eЀ"rm*ۼ9HK|pSSaZ@վKf ޽؆СC:tҼ` B7dw}wf^Ç`ٿ 6,lpW+?0a0(Z꘣tfw\hfBǽgŤiY(oZƔ!J{?#LCfZ$mkbƒ|fwe* ֵSc& SUka2V{=+ӦQ3;ؐ!;W2{?͡XT>KAb1euɊHϞZ]Pii创) l#)/s IDATB1YiKv=6,H֝buisߝp?YN6v3v;`ubæ0fnj5^>o|pg{hanB@)ϐەA;{)EH3 .t4qwqG֭6Ў"URUR|oW"܅h~"\pY&߸ؕ0xMzh'`1]ޚ?Lvҧ̿GѴF 69vJ*pDJϘ2 ymKb|k)ѓ9gd={'##t=ITh4po4[ִaW2o54ɞ,SM~7M y͑Ν{ M a=-sѻ0gpF(ޚi4iZRh'<|wIֿME#OUhp V.؅A^`?V^-W^x1CjJŗX0GPb]ߖ/_.3mIst.K6.ϔJX] ]p*ڲᑙ[Q:þ~ZƢ-¡drGm-KfdYH@)NI3Jv>/HI5vLG`Oͳd6=)شf͎}6n'^ MYMWm;Rt|206smiٖ6e}X-]Y]Qrp#J*JKlSCɰ㏝UڌչF*ۜ2ódƞS}ǫ/0tz$m< G|FFwy2dz~O^Brpz~gϝ;`p{ɚ!A4&醇e]Cmr L?RUDeީ]Z߳gZ{>8WeƆ>뿞:k\iq' .ƻ~z^F/|TKӿkܥugnx?0bۻN[&松g`y[wMH:u#U#{$$ hzĦ0%S{$%踎-Z|vqq X~S~-;QF"O}v#s庳ı3aԥť5/.R@xٲesBv[Ν%o7{p`C`l>,IJXLQߌ3sÅlzV*1<,w Tkq==`eմʆ_|ߐ}4uztSY;vֹcyGy߭(&otci5r$c̡nSP쩛سo-bu4#F88QoUD,8{8ۄjǨ)>ݷ(ӑdQ/n9==R1'q^vȺz;ӁJG&qf;f#mcFh(k͚cӴ0VrW꙱^0\&jip]`UGy+k\Mi\\V{ hؔ6Ň^K,%q ;9s&~G-^]|ArqLo-^-SBmSǕ̞ɘzߑ#4-Ϭwٱj!gz J0flqu9EHr7K߽x7$9nCER[W*~]&%~e>#g.ڱaI[`$pq_Z3xx i̼P[f--5fV6cbIqq3*ݾ}?Z܄MSNխP5{֎61W:Z {M&&:DR4tIG#X-EzX';bLc%o 59-r:sZd|8y0U!.L F{B OLX B,7Y*-4XR;P/y= ^'u(xs_Fdסjk&} Xto[7NE^kf_rQC0f7֭]Ljٳ6ȉ#!#Nz=KWkKfع+;ͼ#m&ӵ90)e ijGL] J4sܩ{NXxt֧Ȟ* |ɞWռUc^)<DŽȵOq͞rɫL>9wtYϔo}i4yG<#ӆݥ9>bg1޷r֠K/ةs%6Nٲ>#̩w9"g̥y}{%Lt`pA7EVʇ6+t4yZ!O\C9KBhqNVo?<@&9A\%71LX}^qn0M@ 9N=XN\L7Y'z 5;rW -G_&~-qA8#M#M\Qig12f,md8 GMSS:|M쓋/LmrX08e2i;5:]GYԔCxt쪂4P)X癑Wz evUYɆ92 L/^><&`F0ԍ_ A߀V V0I"[u2&h/O%@Ȩ|"0Wo#$!{G͏1:>؈ AMG$@$iZuPZfL$@$@AAJ~S+M7Fѐ"T6 B: E js>$@$@Ө;8\({#K0&FT`0'SQQl\MC%yIDl$"5Sw ]r4gK2r^~8SݳCx)3=$lVkeIH#kZcӨk "n9{r|\S"J` §`sDIołrv£X]Uxr# &࿚Fn={fvEk~q޽;n.q4% _|96!!%ر9JxqX[nA,b*.c.f(Мt^utcӍ?>KR\CF%Z8T|QF{=)imCF/!es wqUW]Ol*" 7|#/q!DFxeMA{/tTDLz|NڶmrT%J~afdǎb$;v'6sjsڴiiMa$X{ｷuVsH >b#C..N)Sf:Dv_|Utqè^{ꫯ\o}J}4!  #৚F^G0M#فm3~mQ/Dc""6D )2d,.v:P@(Hd'6 R{שS'heCv(> 2ѰBHHv(8 6|/++[Rܑ, _(:t(Td$C]8d†(a-\>L̡Hjy|7VŸqG!{=E@C#WPwx$b8\""}4QH !$R&MDDv#+a V!e _v8Qff&x) " m28;1ʀ(R0„%hݕ?!"[ "FP JV16 =4Q4v"%1Nd]svcvA(JёN,s 2}?.;~һwo9D`vXN4B: vII!""` 9YhNqE2.$̳q>d 0bIH QX17qCGFMp+pG[ ,/T^p AAccno= G"("64*;EHyP l$>pr )#Yd'$ QY'a48*ƘIDKA3aC$b(ᐌ`AxKp# (;q|$"HH$CtE`Xc(E@xI  D\J8^@!;dKT^!MXtrTQ;]JƬ Q"P%`!#Ba! BC/ (?42ʸe.=7kܲeK†8 A8D7t)BFb! Br!;il9Yy7) " 6)M)(9kIH S}w^pk8PՀCl(JGrT-\Sb! b1J,hes)D&=*CM #tODiʙ &%.=GGT*s@ u>[K  iDsUKorn$!_FTya@Uw=uO>$@$@$$ OØg,G3cOޫAy srr\`B o@HH PӸ3H|C˦y睘ʠLq}뭷w:kB8QlF_,k>C?q>!E' #@M @A@۾}'ZpO?4I00+CPK;V6SS:/M TzXb 04>0+7glٲ'OR5mR(CՂc41d^{ ,b>cf'.Z꫑]K.ϴg1>O2AhJ`$@$4^(^5~ӟC>6v+=\&PEEwƌBX}S}H=cRY a.3SF' j@׆A%v0C!W5q 즦;{lJQ1 U㓥̙3:7n\e=Ԛ7`$@$@ [7zH)VCS뮻%У>咱0t]m".idfLoBS'O:BK3IH PD@6}tH7xM7Aw t4#tàt2qDs?:tCet 3jZGl+W{]wIP0< 6 )j!\Pw^4ʻCj-0q gW7RgfhMe2SNݺukQQ n.t6 5kX|BgΜ9x5WL' ୷@>ܹs甔PQ;}5d;vb T&p3TWWeg:Lox~j ! ל"t,|"c̘1ϟoYK; \X~Μ9lHOmB{"- *СC^Bl_u9_g$+/nذzdv<_zL & ο6/,K"pցjA*AF&;36&dddlذoa$pA XspMy#iTѶ.v$O67X[᭢G\%c2RT:q/[ '9_!Uq9;XEl'm# 8{,d^̯\0V@ѫObhݫm5UϖЕD-)g|nj{lXq%ԄX' 'G}O^Z"֘79@Sअ)++3 5r׺׆MWHKPŅx,J s% ?Q\ A>hL~V5 -#h*0 5T p!38EOゅ$@$@@ǘzǸCA@aNX«L+ @S@UizƔV{sVNn@4Ʀgd>KAS, @ oHP|bQCeUD|C4ͺgEL4gHH.Ǟ.VSߢ^#)&}S ZuRѦBoG Xެ51g A!t>,* @i;HN_px\a11o)?='MwpsHk cǛ2Nt @ iZJ|qc?f͚ՋH3eoڒێ-3Z$cy kZ5var7Y`3HxG3R˹/m }>A -sapHӾj鴌Ep^5T'WYn9`kˍHHp>M3I[8TfD& ?^Of\Ω0Z>wI~ kOqXMo д'椤DFk+&#(_w}ۆtDCAO?Lntك$ּd HiFڴq (޹y֭׭eAc;gk+XӪ9rӎ(1B4yX$i<2{]K07Jϑ`:oZq{5u'ԁ!hlG7{m ]G>7_]dIH +,wQi5X+#Tʲys{Jk>׮sKe0Be(yp[Mފ,Qmꄘ5{T-E[vx#;ܣ `ZjޜE$@$iRREXM g-^){*cJxe#'}Z:WfGD6gL,t] \(^ZҧܕޣS[!.IH[{ Mz{:4hI|uJCfr*ziՕ$P[cbE }նU^t_A M޿cX,(+Rnڧ@[QK8eJ&LVuL^RQr9ks6d?Xkk`HdzqL;&&_:IH7wD $xQ6wȰa#F [\cKS{;N=V3%\ss:pڃ7E_WQ0YhYC&ꇊ7t;lذQ%/GK$@$i?ݣu ߶RI OFo3l!YwQ=|З ʕd/6|dKC'c[0th2(82!MmmE6-0cMSώ%  @&~殽9ǃ*+ٿXL5oyhz⍹/6 [7,/CK2,-, 棺  /X"I8{ pc`K_\kak{Q30 G4v,hW"##X:EZl02KmmX)QQQ_ZkYo9>>J كlڂ Y4{MtԪUqb4KUW4Hʿk׮=y$,;Ć  (l|s @cit:ƠIkݺ5vcbbjjj5F宪*//OJJ*|4?o#F@Hl͔6Uʋu[MEĞXf5Do;SR| 9m] K rC5K( i3VnCҟH5^Ҧ≹M 4&|C\$ ,j{K mL4~nAXVu֩ݻwGf1?CQ3gΔgzrҬZM(B VTT4iҤ?Xx-++SSNR񙦁%~_H5ȆU5u>WOWCedd̙3=5[l9dߌEF6Uxx3GRu裏IMM5jԚ5kTك0Yl0@{W]XO$@&@MSp1z@Sگ_?8woE/BvvvΝǏ=`>MBBZhizNd[oѣG_s5Ǭݻ7|W\KMB #sEuN>㏣?˗! V):Mqe˖aZҠA6lg!ǔ>}Hɐ2K^ &K$@> @M(z b*h}Çcn ƧwfaAà^~~>d"J/ӈ B$2ԃ],(pAIx5PӴp]4 :qĝw9yd+**Nt,8p"hDΛ73Qx4x n EVcL_a: JKK\~ D`x D}_hp>M^KPӄO:ᒟ2~aa!` (  r7E߿zjl}Rs[cP͝;o90.\+6C5)) 2 2 .vetha5iuz .%\\xNPw)YWp_."]=5RM/O3X4hh"NjN4qU&=^?ƊW^y%;PM;<׈KJv[n4V7ܦ$0$:RH_ |P|ܠ5VH\;a(ؠohq:F{iB`.rx"Ջ94+o!O*15>(fսKX! h!BaR0A`#PCAR&6 GB# c ?V*Dt Ra=jWґ+5d >p܁'o}™(3i\xAo[FAx%/XW5{'֯_|  Yaf 0dկ~5MA E߀5Hh& x."ts3g?xS-@vš⟣ob!!~:+)j%r1Cp҅ʁ1]l7tT_5 w&0͘1gϞ2؀Fo(5@tgϞuTZRwkjf0NC>2bDNbLFol_qm>!wl7_P8P|U5y X1c5?xjرRIMM{~)u 9vw A "jFP*[{:!$nn6f"#JUqB2f>dnR9})f D _<`=9"!E4~]ݸV՘F0V2eʔoFC#x¥.m_ćtP~P%7f͚t4|0u^l>4Y@L~a~qq1fM[(]|t1_Y 6=;&;v' P{u5x뮃â5Q{h`m=:'dv0VՃ4:[K `p‚oE1oO 3AxO?lTr Ӡ ;6a{j A4P鸆6c 1c /Э[7' U"h[~c50<~n$.< Vx8a cJ38A%#,:^'Bຓ ~¢TDrdd .8&0*6M0t"6^YcOo^~{pxJ{gvڅK@`:F㝜* 6\;x):[TUY4_Lv5zxCH,| '}VkUuUue%@K+ fG۝q!{̢,qv7 sT +\d{ZQ^z ?H$L6mڵm߾CǎIW\ѩ]tB%?fq! YcBwHp=BUX\h65;-- O? 7V׋XE|k?8͞=K.x^P\>d\ϡ)( F k (?{cyy;|7*Q,ORneIl!CN:}[=z\s5rƗz*_`WXq+:DGH ꖋzxcyŸ'NiӦk6z_q Ka9jtܸqrRhp $+CICclqz6 ׻C_#%^ (-.:;c?8ps171l}wC[lxB֭ .p'F6UV(Fpc!:B5MU5hE` ?ƺ|7o}7y ^Ÿᆸc̒F<Ȟ}Y44aɰE@ ѠF4~تNv֭~}yLsI#-ԛ ou%^ƻXU q9l|MNNC4,?*5KKM.W\_U-/޴w={`wߕ 6h)>6 #q赖Q'1c( ཱྀ]̡.>utbbJVWZt/?ّԌ ''njj fwjՃJqΟ/mY8pwuf vM`D(\8<IhK1j@\\!kDOw}!ܼ+xw@-aFX`އÆ Ll|y $t @ʜ?]T\q ePon]P3 b§ ܒ"£:]ӄ˝G/ƄȨػF9ڷwg0Wk"ZEFhJ۹ **Gmݺuڵ[OLLRY~]T1:>i '@Md o\4h?'WZյk9s .dæK~_:v_}x /oCwӠt hfbMɹ]SdUg H4Ȕ*k϶Χw}j7DEGAR5Zmzz޽zC{UIIUµ\"DAF$&/-9'NB|gpcl}ɸE06evS`St7t/nVx h_׸Q*/r|:h ʕc!Ƙ0@ujC.:&0:ŹkpTQZG?odY ,MciSsl5u74{jjll wֵx C_uuU5u6(\銁7 a={] [>bŊ%K0h)]p}a` {.OQ mEzkp cçA_W(koV@(NSDp¢F4r}p rB? $<VS5E脱';g8/~Arw+#",Ȋ%׿.2;<iixWf/kӠw! !Sila#/8iPJN))  |/СC·i `7>.slƞp7@yȧ# 8O.y_\VE^%֪zc~)cVe%Ui#n?\ң<~/}Th WWV*UT6P4(_|Bҽːe[XcqFtёcvO!51.Ƃ~x!Ф @ r@1 +}3>l[v4q;AL|dD1س_(!{v?}Ά 4Xz=VtAvXXq IDAT1V׭k|BB6qw7a{=OJrr'_CCWO;Oj%U??Ӊu +WR@n(Yc* 6(>ϗW~dOm8ܾ~?aNSO]xhx󛯏tKWW6x׆7a9>3IڄhQKcT?/WemUQÓWi[t0[ F,Waze0FSTW{2 |bjgr0үjfTj7+ð$04hh*18_cbbw2wO<%Q%ƭÆ*h7w3;AD 5 NtlyK~orMވHH@BHZjXL~9x̙s_ӪUdC }~߁=b5=QSx|7oU=Sb ,=w"u8c+[GAT/\k\ҟ_h*__;PwIԟ). \,)>[< C N1\FDa޵>{]1!E*1!9&6o\C8 aD>5[ְ Z/P+X8I]m]:`Maaqdx$tv (z @O5dbf0-;-g09M۷ϯ˽ǂ:<]pw''NnOj/=t\Qi1p)>[uhKLT+ @  7*zx]0vHt H3EJ%n8 CDC c(L;[\ k4"94n~=0CＳnV7M8AU55u,)c *i0fomMeM+gVizs{uCڵKsU8c/ \rE6dbvHlAI 4 u<֦:?%pP(u|Zm|.k۵q7}9[Jͯ1ª*2}uo] n(wul4/,=EhӿO\CƉލz9}c),ooDAQ뇏gMLH+x!7^K#=22̺Pa0`/>kO$⦶ZUswa7i'6sN[!9>/Z\(I0Y4 v뗈{F 4 lܸTV >NLttNM}|\ŵݾZiՋ, lcqM IB!H!(^MlM \1reVˮofVWj\s|3s7眙Y6$Q>{x `;ݙT ܃UYcSR/Wfţ:l̉&p xXWCi@;Gf$4l=zqqfcv `Upbb[7ƶ%wd%J|$%'(}~"qb<)y;ݒZBoxӠ9٤Z#lQ> ZG|F. //)XM?H~>1p"^xSf5NB{/j96?f,0+k, -Yl5$*zda g0aVQ040vHI?rtɉXR܅s /p;Y >}͎_zK.D>k81qٙ] 7|s޼yl:58?3G*~iS,C 04hP;eJTt!V8'mHO8Հu!D*xgy @* >Zǰɮ$'aIYYUZ\{^S(mj;Nk^3l?^λ8p9J \!pϞ=6l;h2'Wܰ,@Jq iD[ofIуN:#}S|_hnnEYMf+)} WBҩg [ $(i)Y(wMD\do zprha&3N\gĔ4k {w:jD!|'+VHHHF WƒY(o7= q K].t,]49E'a;AOv[}DQ 1J\}aC/Sd7 ELFϝS|]/$caY%^SI@,#*320}=.ڷ\z,dw kP4'^>!)l.pDVe F F[[O8^t.i`~CtmZx!f'CLlOF-l<KZZ%1%6MC` Ji0T84ګkJ$۷dV$0^D]?Az(z AFЫEAOY삛MqI“+ჂG@fƈ)^#`W]2f8&!dXJʃ9&䑪j:8?L:ccb :%|y`FPEe(A k['@s⌁-fBkT cQ}}͉ZתnVJ{9L8X|(i\֫T&F\6.ps1) ݞ=i*o6 NtNyeo=gNwqbg+:@?Cp4X[B;_w0&#@0Υ `,܋YiԺRK8{˕DB`;[U!ߙzVU:Ib6LP =yN5M9+ayѣsFf9|vRUjkfM:MB6p>= >'1)fݧ# N˄N S2UӀ H\zKĮBuNF+<5O- `)k`zTl-j7[:op!1f3οRĖtIsbi-o*UU9M')//R邲pL+pp0^\in3BV+H=+NWT"* cޖ*M?O01%)WOK3R{,MG~(|*%fn18{26i"wLxQB<ɝXnP:X㘑sqln:z)薰Pt|qkS=K(MΘsQF΂-O.b%":ɀ"猏 Cl(@4HR%`CN8뼌;_踗$_FhnEZLh2ω{fɓSjɘpΒv5Z%EM.@Rմ669a'F/փ݄nϞ4i,8wZyM<;!3.%^baXu b2[ 7 {T,/I@eт33%ɐ=%'W~ǴuMmR1jYJ)RJ؃A>Y n4ѩ{ t{8.DliWpGM39! O9k,EH+4?n ]H7t†Mحɭ2g@(nrR`1g. s 'uX4@!Icy1>qNT[4d 2EU*ܰA!B$+-SOPi4$ⴴ`sΟ?_=E8PXXػ6Sqުqsnw\t+\X;c]1`0l9P "۳uq*WarzO}2N,jw01 kib QLh~Xj 'D;ּɱ*)rXPc{jf Ƀp{*;Sk 2^ 2Z MM>~h6h(?hgO_}OJfRq՟zi/WQtt:4q 'jn)yJwOj%ĂBuƨ(ASlxg[@6&;]IgdrYV`;7Rp݋WAuazb CNqK2YWn|祚]:挍;R! ŝ{R=溤 ]cӢZn# 1쯹=:,00r.f?; CokR3飌FDjxdX[O)4 48Dz 75x*&:a♒1¦!ï:\bzcj\A:9jDa~sPDCჽܒlԕTkmnHIKro5fg= + IDAT݊>&m4%oӾp՗_,ɄH{𸬚bՖE!),5SJY# v8mW\厞&FP?.OAPQ{<6)seN<{yUvlV j}iw¹ E<KXfwZ;dkS+:!Y1x3 2zG5{v.##Ǧ'O-%{Ҭ@bX’gZ ^[wWg' ވ:e>yxLH6 e- B2T[ǒE^/ ͥyLh|2wZm*+&s(&NA?hR*QBr'&%% M=ꆪ!}3x$;ԾⷐkαƍV 4ޥI[ PIZ5)Rϟ[wv`6jl OMd{IXKck`?%hFO]ȷC^/pغ)jWFu:nkC)8^Gdx,y@C!D>p1aj`puYP2h0G.w[ǧy_=Mm;L]93&\-%,iѓO.;V):ytc:_1er)N~ѨMp׳l:f'Em{9L"Y7Gt8'iXjTNo۳M!B @`'͜I( &x"#'ND1,pYx)YEQAZ|ʶ@8M`+4ɋN쬪TAvW)a,䱊kFbVw|I揊dʊv}lGL,7cdYyr_%}yoqi䘆Y1Go5DQ}QIYisyP ;ۧBDQt4f"4Ɛy5:6CoP ]L;v,JJnn.kuiT~7]>//fA،*{ilL9celaC~ǏRn[8 ˇq %;k4M?8nae-pVcԗ4e`j9 $gtop-8':^ק)]w^Yb \g oָ raL!)ۣtf=匮a8J>Bb>v v|hFú$xF;\ث/^  PAbd*#0;yvxrtS۞9dLϖT.Z4ᢎ\%>-l^/==vWrgMaDc* 5OIش{˶W_wNwh+תԨNbR^TΝ+8 %Jq3b]:\H {|A8 ՗`_c^$7&=@$p061@!xwn5L60܌7vqi0Z|~ekj\ * qkӪbW>*]NLL\`/# 3 OȄ^8Mu=?kkh.Kݱ}$'PeY,wՑ/;:ht8/aOOޱ?ZS@TJF:]'3JɈ Z=5s-ӊ .{f_~'->cI8ǿ?|'o πƪ5zTpE'B=Eomb_> Fn" >9b18qIIB>+HaE`FkNd͚Ke]H>cƌiӦaNPg'2hB 4=gw/~'2Qr6Ԗwd*vc$+ǰn=ci:VBj嶞'+lZϖfre{*[li3p ~'U>NQr8,-є{/pԒ}"X[/?K6ՠTAl2u^CLB=ߝl*o4 Z=Sp- ^ \L&ϸ 44Lg<'ef͚w]z,w&44?mO~<>=뀼ѸvjwqDUwݙS'a,k1o /r2HHӴC!4Ugsy}v/]ҭ-2.nˎiPxTAF˘jwT/ζefSK[~6Ōxh-Β]f AIaJ'M(31-&]Idk) J㞘3=m,KRG%Ie/qa73<׏MOh̙l2,*(fq #0Nu]  l9uq@s/A?GDž0cKہ]]N穃56:,##uC~4 DKFb?|A:w<Y[N@9vo:ZlXI@G͜hɉ!ZT,B.* *fW-Akܹ٧2n_9݆vghرa]%Z0*ӧ/_d2q "}@QӫQ'L۾ii䷿-N0I|>̘lͨO@4&)@¯ S/;ݠZVxXarLeC=VTFu3LgiG|=DLUo:N8 1?cH -yN VaJ &F+paDS>D)&$Vg^.;) 4w}7T/FkDWp @hԹBwq|1 dmT4l ˎќ`>t0QNlfi$#U6b/,c1XMVSUՍ,1*L$s 4ʨUj02fl5}Q0!0x&6l6éml Ȼ6m}\r5T/~뭷i3[M"QزI4/nUڛ ^Ϙ|~W o9̄`g4(ѫu0]Gҗـ( SԻ댺6oXTan,31B%!V5tQXX5hP0 R)T4 *kr9hԽq0's"B1;/SjȳpÙm-D`Dn3[˥9.:K;Wa`r@XaBa=1aaAiDȄ}*bbK {<(8PeQ&LpM7Z ',3%^,G>L󷇮^6p4h8IR3JjMŠwV غh'K vD^}2\m}!3y1|eF=$#~B6:v/|H#ˌ ͙8}JP"f2\w's͛D+#*6:%e֡/^: j`4N5Xj0+- aˌ3rٽ{ƍpII (_ʞz96]p1Gf!LfD ]2N~ %twFVL\gT+v,J$(kSm߈fM:jE1dY79c{&9w)[RչR9RTx CLRIaB`pCCt05lF,Px@qWLb TFRX[j.sd~a׮]} g}"38 x=fQvd3،kJEC)`I08 :% bD2Yj[4YhUF]:G|B`"TkKūL>1d·I{ Se$5tt<lխJѻ3"],ёz̛Xb!GNF=HCCAрp_)mvVyR5V0'?~ɒ%O?|\YYyet s3.{\RR ,M=6^#ޞT@C ]yˆӠ.y@oDyjYm.եr~YB^@ nm6!3Ϟb6W\cV롭\̬1aiSal`ƈQ&hh[WP n1*eK]]csS0kaV,&A1qc) v0-15i3ZvvT-+/@hhhK2ڹhpIJNȄ f`a[pv[^Ae|\H/ 4/\a;N:]|9, :$zz&3a^d@rIuu_}14hHaTle^ﺜέ{n{ 2k!ar™M)):rZZdMM &CA033|> AeF(aW8 Kddf8sR`IO-[ 䃓#&'Ifpfww<70d[i`Lwȅ׵4['Dt|<Бp`|N#ZA?K lxdddNyA!\/t) 3BC>H !BF(Ў1ËG. F!9 rRԇL1(YEv/P*-糲F.WD!\" E AU΃΀&T5`380|MoA@ wA _,<peVo=`FQ8 *n@ hM(VS]Sn(88@tG8 ?0-Yor0b 3W1)+$k-HA4v*# o~Ty-"H ] 9iP9a㊶wYG]]x̎οPψ'`:G074@"Zeoو8#TA#lr*ʕ6;ˣ"ְe`7`,RPvA#Jk 犗t"Y/<sm{ {”ӈztWYo@Ɖ q jjģU nUQ0BXv(22r Q2qq6(ͅb(…03KpN""Ú4zEoڮ{â"p` x%FJ-CDPPEc'P8e}!o tpF>&NB @4ZfeC# ^/ ˠwyR*P-F"6#3 @\?[2^e4tA8 5~#   +AISkDBV5B "a883üY\oh'aFZ@P><3DĂH :1tD4#q2Aݡ@t~AS"FX #pл |1f&| x 7D1 V:+60s8ڴaxۆϖ P+Ԇ4=BC7zG@&᫧AKAX $8t%eaa'Pڪ8e6΀p^$x H_ ^:(Ïq<($# =n!UN* !@p+b, ?)9%zFQ[ IDAT&\ft V=BR+&keRn<  aO4}[ 8M` B $0 7i<(-,PP0Q7rFPXÉP@c`pbFF~!@_pQ4Xt]%B mp$&&Bv6c1w =`@pP?8&J3|>CM]4aV]j*!@`{IX[k((s / $ #@fR l Ѹ B"?³TDf !jzhM. A8MU`Tм(!@!q,X'= )i|z!@@8 "MO󞠧ADk +=O!O0? ]&BZ -a:T5XXgi¤ҩ!@iF/?| 7aa{ bP B 4A^#}A.a{"z?!@aq!*/WP!@@/WF=Q?-Y|FM8"Ee&B.HO3\H{i@\L2j(_޷o4裏"T*zXHUZO!B  Ny=Z~[z!97?_!@;iG2`*\py#G8qBW_E.ëWFN:!@ )iLEPT*x{"""PÇW_70BЄJE! z\NM njsM7a4oiP B` N3Dx@ւ p;vod0phtCaB @8Pi@T2Ѱ@%..nʕ: ¢AP! Biӌt pYf-X@n!%M-e D jL;v%X@_{bbb~ B#{6h0 г!@qЫK$X'mNz1H m-fs[tzK1"ҥ4*7ILk榢DniJtjsiXJ4w|Ry#V b4ÉH t͉_| evN;wo4:J-)=[t5''߱+R#|`}~Cq|oW8~x̩6њ઻8p /vF}:r!3\`b#/{u~~@@2zUޗ~HtXǵ6lk &G5XF1DAh ~Gq=OՆ q Ìqa|d^ MӌLV6+!Ў,.

ՇTh ^Ņ$\qq2ϟ%s"b##B ;1Cr /!lngG+LAGf3M{'/_Vs+~W2~$/3iPzXylncZĞj-zt&::b=o4ҢOn8 :!@NK藼Xc)"":B\`$-zA!N_$40Sxu;9wڧ%ꑵ4~e.|huׯ]#aO=CeWv=s[yO얓,4y1 eۼu#y;7G>gJ1u\^m\Ņ$ݴ?հKސ,ҴKnS! # ;5W(@V}]2mk.wыW\e4mX7f᎛}ז$̜_ycker"T&hxwâ4ID=B B Xׯ5]^L} lc?~Q'-ZlJ $ؓec9]^H|0-MәE4hPw>#Mі7~wϣ~_YubMpӶP-۶mՉgyșmÆ]E<kY˶68\Rp|D ~FsO70aWIu6o:p+0ص؝$ToO伂23K:c2 X(EU$@;n6Xlg%|FSrtKqKn5kߏ׉]|C&Ѱw.{]%t4>}*IJLdLIx#NJJNoB B xO1y,]X{x+n',[b%۹ْŋȋ)eFջ2B8ᑯR`vz~٩=kۥL.J;Y#eky!"_Ͻ-fJVQݼI4ZxAB HC+kL`HL?c&3khZ{RnTS ȮI4FaA&3,/2m/ҸE׏QsME+],8;\,.*bx+tڳSFB_$,0x(= )pUg6flH&0% miC `n6EԾV2:$n|&͘ ϙmNek6nK!qk/}%E I2m~~M{g/2 >}湒-1 L`XMH^aB Nӎdn=l|dO!`yQY_p?>b<y!;gqS+/roF_DWSW0G]u.D&)ӧɕ2O{=gHv65-Z{tD)!@4+uڕG=(e#/_ 9V WgxO K-nɮڸa5_ϼu=  :"@#ˋSYsc7Y=[^cjګκrów ç YoaK!@0/$,RcC`XpǕ/ʋr+˿xRV<[GA9WcZ=_FîW@Wtń0ܚ{ϫ@1(1k~gkM) ш_܄fGj'6}Fntj(z%w =gJyy\2)w9m:F6t 󔔻p՜ ֳ+ַ̉?lùVa-qs}'ߍ0q5m p#@fm]ߛYfMu9>|FFIAFBv*B_#n> sr :"@x/B BNgQ B B#a{x"zjmcMN4 | h h&鮻j<>]h 4(Wpѥsi`%ˤC.k`~GՁ]uC߮ǿ!|%=DDD׿NvCm?҆--m D8EA\.bXNum۶//W7|gN6hQ###qVTH٧5qF!]rrrll,k g`ђv;00͢ M[ٿhː(h'}y=b"|l5j… SSSu:zdB/YBAuMP53 pԁ 1** gO|QO4b0 -~O{]/8ҥKQM[nszɓ'^ yt QH+j)9o4qa9NxШ y- > mC QУqJޛxc5kٳGDFZd̙3Tvx:/"h ¨wQ h(//:uiPB6 CGFR@HCP]BA`d;Ｓi&~OR/諐MW9)>68h',rcinnzVaH Y\g~_Mğ7o5\ M 8ƆGO#2 >ӪN-0Np } c#_J=&u4}߯8HYY51`?IkJ8@rHEH j1:-uYAN#\LnQ^Qdq))qRzݿUfee?8q"2B +0_`SCG%wjj0Į"',~!YTh 48@nlܸS'3Ъ^z?̘e?/~ M: CCmlx[ӌ84 $5P`?W_!  Mf#@ >l@)wyE9s# 3`3PG}_-i­Ƈ_Akp~c;;;va B o9_z)^6Hʕ+aoI&4p-Ӊ( N:to`@ Ak` @kӉ ^z~A_<̚5 1cƀl'al>>5o$N `/1V{ciʔ)}$(]B\\\ "&3"m8cfE]44FF668_%PJ}B8M`H@@AaG{H3Ov-њ`KÃP`02jy{go=&&Ix tB8M'@?i;v ={6VOX$ZO)-B"LGb" &026uJc,|9u]w=6<7l5Fu4IK b;~L?񏯺* ؄bfa zQO 7 Ū.9Pp/Y^^w\x$# D!N E3/2+-Zm DCgl0M?i)HI!`Ddee%"_~oJ ]!tRlf߾}蛻vD: %fpc-<3 6њKAlKFG44/&CK i hz/F(,ć"7h @@gZ[[YV޾tm&y]`PGLJȴw lp} ]yR`_̶̇ Acg(ܺqWy!pzD3? /[t+,rJ@bhH14ARQMAV`Q⋱O<ӟbphM*l7߄X9s10-- ;h8v]YRw q+Ak|IXmn-[|k G@?9z(M[nRRR3ݣFW : @Í5XrxجY(ҥKA}*Nzbv酽" h3k7n<묳" #`J PxK..qqqG`__`b3aD©i©F`ԇ~$$$`=ӉI}R6 $ ̎;}ě`fiڴib(8f0g' U(]B`D N3{@@5pu]hͧ~'0}/!Z~t9@{衇N IDATa[n;0t3ʀlF, Lo]B4]\z_Z#58cal\c>GpRֆ5@b^ulZSS#^3|,<B#،llI=3$Ai6(/ @[#Ӛ۷oGDm ΤK ezСC06m۶M555Ν+|&g`{U fAekC8M7 .Âee7vA~'LZ њ=ʔ?3):( Vپ馛Jw26}J+H N$ٔi 9558^9y?v5aFB7y8y U%e|ML7*^0Z#Pk$ÇGHv  c~__^?.nou@ i;2Кj;v s9X+ fRP~GEGT .3.fBPaqEqG_ZÕ5NDb >w$:(z`@SG06 )N8ښI&u,| @&Tj2<YŅL(,ZSVV.{=քGsR  6A_ bfl T~"@QF@(䡊ښ_ k߆DHWmz;>" l mMRRXxHw/05~M.iBnCdր@[Z ]{k֬'YM J&"`6ߝw޹sNQŒ_Wg}63xP@OL5*? N)aG@&kk, Zi 7܀2Ifk^oИq`I@}(N'k2ML!4aS!WP|5ڍP eO~sa,K&j>t 36ps(3hɣG DBPIq#J #mA.?Af+^zf`6Mēv?%K@_:z@J*l N奲Ba+e2e֭[߽{7|n.q!U\o( Z)5a, 6ƌEhM\C8aMo Y6eof޼y` )~!Plٲeٲe3_1ksd60ٛ/E&Dm  |h #\jXva4#*gʕE&ěB`OpkL{BO|}ZCkdl"G7E#@&+x!D&y'Ep몫"#TWlL޺Hܹ./<#[=4(0D|ðCepڵkyߒu]|Š>0%A ,(( I7oYHNNiAשADu;C 4!UT%XV + X6b769VkMv=.xź=DoЪ/&B:˯'%lsZVot{#[CyT\\RRdm:.JSjTF{iumNdې"5I1G"Vxs/u/k񯢮Zbk:jA7ztjވ@\p#BA`ᥣ#2"DY{yih%2 g}RQQ7m4#0~xNf`=JPWӄk͇A%jF~i,tcƍNQL]n]N=ǣbz V2TLUҁ@*v{5~W\duF d"4hHgmKVOSӪ,59#i1O!76{Y`ҌO^+ct "+Ok[S`WiY3ħF>oY{eSB[z5ѓM`3bQ`al{}@aBi2F߰au>ӧAy,5'eu-5&k}pݑj@t13z 7TRVNҥEdGGM!js道cYDiD_V]f5J9榎IbJi2Nړ9\WJ6ȊrҨLIPWo ,;9X͈љ }Tzf׮]wy? ^]Tsssa]ٌ06꙾d N3 Cm ӟ'J5xLZ#2_Rײ-zga-cbT¨1,<'rRO05.Pda$qޓ>{bTJח<]C]YexUz_A|anGpIalgR <i!߃F5 aK~̘158g^> M|_i9&MMVg.hs '%R4[e -A+sTs3CM|A >:X3Fu $<@uuHԮ&gM& @/0R"AnY[aLo+&##wZ#sG5۪M-vsw.QFFҠ 2c\Ax׊I,ߘ pvco >fM^67'p\#1؆6 $iyc35jFB/ͳ:VuwUuկ__}UC3$N{.Rp\_b\1/ǐkXLba3GuSr[ R~-d|cUD.PHh!Ϻf'ćj.l^cǎaVTii)&:a1`3{bMHl% Z @F Lm`$Hˬ5X9C̙3qK{?p@G{ۏi8&X2̂aOe*'$se ȊO (Vfp*-SBgݾo/O9(z/JV=M$ߵ;c^>^|`C!,7׬Y) @a,#kҦD` @TT l v(8u=z0CSç{{ p9pgo3BJ_@q=qy=FaR#P!ISxv{~saT:;:::X^*Xx#hkD qH$-TYZkGbͫҿsp[%]^ݤ4cꓠo2?W!\*j\gœ|4yw `?Tl.< f6]i ."hiMOGs.Xy+n/SͲ XM.U ('ґn) qD}W9֝/ezϏʙ|/KfCxp,2*αFlI^ГHӌ.g;U`h3?1sN-IDk}rmyʆYìLO;J`;sѫ'yRUbT;cRtצ~YɁE2i ,6ӇN8tezH&oVt\ne?ȭ|rI&Hd@'QAs#Dϯ>M)V>?N#:@H=53I>%Ԍ$ɻzӅ9$d+=Yӊ`!M23pi3 S0,=7]Sw:W7J**gU1žuK[z]?,i iǦ,4XٽGMJp)Ba_unam/4uO9_Y ;%ex/ ( JKKt$r~)9X6+'#,"6nzN6tH.\jmCϾVZIVxۢ*:2P1 Ynӿ|sRiB֌In 7@"Ӓ F>;D}32Y]ҥKWXfSMغ.zꗺ]#j:)\T.W)PofQPÉ@ &q!>-,YJ$O͐Komjf]o|pe܌,$`3r F݉9^a㞴\ D N`i2֌æ/d/U Hg45rNFYҐ t11 7Ŧat ;MR&L8zC҅>EO.{Ű$d𓩤`uzcR]jveb=t71կ@~@*~Rx4f*c8p]O4*|g&cf_'9"BWX5cdMt #_YץY X}л_ffi7gi9m_],ajV M.=A4{1$Dnpt?]BD +з!+Q`FCٮI&' ;niV"Diu h…F'ʞ EIRD@98 C5fs*B%L4M6n6 h留o5w@.MS#@*Iuz``*2%ˠ"˘w쨡ΠdeJ!7F@4!/)W;rW9qc$sTnxiIv^l)a˭Zd>MnxIzlKyR&+K_:٧2H*cZuIѠ뿐7iҤ|M$$Z @F m#\,n7c45ORJ jCWn^ M/ 3[0҈KEEEVUד&AMji|̃ }}}5>IU7L:d~f- DzA\5M9Nl*pi%$l  B`4~BB0w,FA<f⍘JK!=:&\9pú:E7Ð MťeŅ0 we G|JcdWD, d $9gp/ PAsF>Ga1 RT@~Ƨ4)¯? Ӱ;qrnuq\kϓZҮSoU> \n+|R,6K[3yyyi*PO.8Ԛt$AN+`j_?`THₒnN<w5qv=$Π;׶i7խz+fӶdaWWSicB&ŌWLqoCi@~5 'AvBY{Rt["01o  &h_e;'1wItrHTDiٲEFodOni帍-݀,(7na-gn^Y@@0AwEi9CVM@?s= ڏ5qr\=힡i:Y+fJ>AP3bfV KT+{IDAT!>콿o^a؞)N0`IIh}vҪÉH' 2MkhnqeOMk#[9B>>q-ڊ0Զ7>r9~;g|O YzB:ѱ_*p,{|uDɳV,P-@~/]5l|;Ҏ ~떌2\~L_~l\y&n10裝  V9&*s99L4Z[x:ک*ֵt >t<s}ҐE]p S3l|hFAI@b/ĴJ6l nXٯ/jXS%;v=_ZN:s71AS*7y{gwp-[`ˊ)nߺMw SƧWٳE z{>Lݮ?ze vE/"D cL1wz["xw"HЫ\o#bb_hLl󳚔:/->5ydsD^r{};̹}dI9UW 0`1 @@ n8 #qӜDHِz!2 ͛1ѩږUΫ2xܵ-rv-k ?..5y+f7YCZ¿>~|R "eEŸZO689nB۠VaiFv5f5Txlm4C9}cJճr>,6 J5-] F3l 윕RzHl"[a7plaww=(*XxMw>J&h"'j>磂?~l֞t4tXV xҞzG 7)?prضG"V 44'I&@v$M0 /~U(3]>}mwMODݮvnk9 D]w:[#6O-{B[QѸ&"fL(DO.߳oHg kc,# `a0&aqeɆÞBG"@G4MRɉ%>6 W ߘm3sbjw<ʵn%kAFAlhG+/r^,fEVڟZ?<ٲehu}-j !PQ3\rz ;{mYˊqk9"KrO/ Ai]90Y$ t xiM.zXD M;?w:Ba饣3צYǮ[`y疂t狞߾}#Fnmaʗ0qss*mΦ&סEjaଌ+9biq8vm"^Qo`\MhjAΆt潭jϯ3f;VMv7tx3ig4Ju@dyLxx:HlVIz C4P4iGi!xhδ/ţOD0*}]2-]l\ K⒒0|:a PSt 1Iz)in eJ5 ? a8 Oj )C< ;J£\XC6M)zWJSMѼ/hll`۲mbUa o4oYx}6x;v2.:G*zv_ uq.f76;\Y`~"F5>Llk,{İJcn3|$:jXԜtK/|3lmVxK .oD)l)=0Yj0f=o^Qqo^9&Sj,4Xf0$cߟ~nݮхX%u\fZ1fT)2~+S-{jp_aÌ KgqC,ew ydJ[xƶ-65+Y_Ј2laceC*7JN>W(NIw. 2S1S*@ D xODT&HX& zq`q+(=aWLaxK gxQguxR'(*|\YWlfUY0m0 G+X$oI4"q'YFIe<)f1 2 vT/5rp|^қlv^4R />ڍ5}\{aBuwr5g q8;?<;/g1 c5x¨cCq"@2iL)aKa(H&kp7H!YP+>iZ'ul<#r UW&|_X>ѫ|rY Jsr,|oq6DM*4#@&㺌*<:UD 4Td Nq&ngPO}yʄ;Jt.׼Vz\|Hr(2h+>Uz}2ܜ+S} ~3` CАf» $I&@&q% CC-"3 ^W ˂j!@\ z,ݢbNz"*]s+TwHhy2+g7]9!Ip~RdI #Jfd f%83A)ی: *VP^"@ni%F3 j 3ɨGHHHA#VA@R>s1[;I2?-<'m(Hz%  E;'; (VB)AYCP85YB2vh64RÊ; dnV M2ԌI'E'ec숚AaRqfp&ig{Du'N*n.&_Fujbvl]ޜnf,)5|x^ $ADI=#DxE22Y/3' "l-(+L@ͨN,0dQL*{H- Ms){&`5V#j|P!kyZk0 Ŝ3~s/flRkj,NK\%S;8'K3_wR4=Ռ (&GYE`u`u)H25W S*4T'}G1&O1Y8@`zxs>h{JUǗr])c0;/cFjna.#k8s%΂ Qc5ṕ8B lgGWpaSQ hi&(؇ʆ}e!aTqNEwuƞa@1^pI)y,oTl;)@ x^K&]QBc8:1_iK%y%9]'9G/K9=G`͠`R88e*vDz\aGDZQ# ,=˒o(U1  2U 77Lʆ+B0 꼒挃anPaEޤMowW92'ɼWE4rXRo1pVAe%AYau ZNsޑFN6>GRE;L2D A@F4i`fةzw DE##GdS"@iSjф7eO,3@@0A*ȁday=$ NcQtC;PE\cW^2AfDQVXi,X0.6\^)"~]$p l|Dд@Qa hFSiLvpmĕqpud\t#@&Z4U,82} aG&ea8~*/1C@!B݇<̺2?PWa'%&VBp0^&UH͡R31XL>B[0*M MnF,U0J&bpdv&bT5ԠV!*i"U0 ;eq!8n!; bB٠|+/#WALE9 >0>>1ׯ\grP)6EPY,|l  t P3lb aV!P"pH:C*A Ml,#8UKr i7̒YDU@H6K GlTQ%,w8L^` L:A{%x'cXNh, }X-FT b7<]$s0 &A05MS3hk_j3  &@&x P 8T HT"#NY\DբP F 5,0XعsP/* "\CG,(Y,z%W&P A 0A"82X~%ر||(6͈#d S6LȲ /JfDZji2b ,^Ͱ!'pA2>Vt"0!HLF**MTq1#L0}4ڀH|Kəaza"RpDifR M8T2H KLa&>jFVifwQ ")J` "Ii4hm`meEFՌ+^DtʐRa6ތ -dL$D IPŊԸ2E_:#]!D$idҦg4%0a5VL6Eˈ D"@ҟ,'"@ YO4Mw15"@&D7S# Dd=4Y@"@  @FL$D"Hd}S DhiMt35"@@ M]L $D" i4H"@ YO4Mw15"@&D7S# Dd=4Y@"@  @FL$D"Hd}S DhiMt35"@@ M]L $D" i4H"@ YO4Mw15"@&D7S# Dd=4Y@"@  @FL$D"Hd}S DhiMt35"@@ M]L $D" i4H"@ YO4Mw15"@&D7S# Dd=4Y@"@  @FL$D"Hd}S DhiMt35"@@ M]L $D" i4H"@ YO4Mw15"@&D7S# Dd=~CIENDB`ovs-2.9.0/Documentation/howto/qos.rst000066400000000000000000000135551324262074100176500ustar00rootroot00000000000000.. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. Convention for heading levels in Open vSwitch documentation: ======= Heading 0 (reserved for the title in a document) ------- Heading 1 ~~~~~~~ Heading 2 +++++++ Heading 3 ''''''' Heading 4 Avoid deeper levels because they do not render well. ====================================== Quality of Service (QoS) Rate Limiting ====================================== This document explains how to use Open vSwitch to rate-limit traffic by a VM to either 1 Mbps or 10 Mbps. .. image:: qos.png :align: center Setup ----- This guide assumes the environment is configured as described below. One Physical Network ~~~~~~~~~~~~~~~~~~~~ - Data Network Ethernet network for VM data traffic. This network is used to send traffic to and from an external host used for measuring the rate at which a VM is sending. For experimentation, this physical network is optional; you can instead connect all VMs to a bridge that is not connected to a physical interface and use a VM as the measurement host. There may be other networks (for example, a network for management traffic), but this guide is only concerned with the Data Network. Two Physical Hosts ~~~~~~~~~~~~~~~~~~ The first host, named `host1`, is a hypervisor that runs Open vSwitch and has one NIC. This single NIC, `eth0`, is connected to the Data Network. Because it is participating in an OVS bridge, no IP address can be assigned on `eth0`. The second host, named Measurement Host, can be any host capable of measuring throughput from a VM. For this guide, we use `netperf `__, a free tool for testing the rate at which one host can send to another. The Measurement Host has only a single NIC, `eth0`, which is connected to the Data Network. `eth0` has an IP address that can reach any VM on `host1`. Two VMs ~~~~~~~ Both VMs (`vm1` and `vm2`) run on `host1`. Each VM has a single interface that appears as a Linux device (e.g., ``tap0``) on the physical host. .. note:: For Xen/XenServer, VM interfaces appears as Linux devices with names like ``vif1.0``. Other Linux systems may present these interfaces as ``vnet0``, ``vnet1``, etc. Configuration Steps ------------------- For both VMs, we modify the Interface table to configure an ingress policing rule. There are two values to set: ``ingress_policing_rate`` the maximum rate (in Kbps) that this VM should be allowed to send ``ingress_policing_burst`` a parameter to the policing algorithm to indicate the maximum amount of data (in Kb) that this interface can send beyond the policing rate. To rate limit VM1 to 1 Mbps, use these commands:: $ ovs-vsctl set interface tap0 ingress_policing_rate=1000 $ ovs-vsctl set interface tap0 ingress_policing_burst=100 Similarly, to limit `vm2` to 10 Mbps, enter these commands on `host1`:: $ ovs-vsctl set interface tap1 ingress_policing_rate=10000 $ ovs-vsctl set interface tap1 ingress_policing_burst=1000 To see the current limits applied to VM1, run this command:: $ ovs-vsctl list interface tap0 Testing ------- To test the configuration, make sure `netperf` is installed and running on both VMs and on the Measurement Host. `netperf` consists of a client (``netperf``) and a server (``netserver``). In this example, we run ``netserver`` on the Measurement Host (installing Netperf usually starts ``netserver`` as a daemon, meaning this is running by default). For this example, we assume that the Measurement Host has an IP of 10.0.0.100 and is reachable from both VMs. From `vm1`, run this command:: $ netperf -H 10.0.0.100 This will cause VM1 to send TCP traffic as quickly as it can to the Measurement Host. After 10 seconds, this will output a series of values. We are interested in the "Throughput" value, which is measured in Mbps (10^6 bits/sec). For VM1 this value should be near 1. Running the same command on VM2 should give a result near 10. Troubleshooting --------------- Open vSwitch uses the Linux `traffic-control `__ capability for rate-limiting. If you are not seeing the configured rate-limit have any effect, make sure that your kernel is built with "ingress qdisc" enabled, and that the user-space utilities (e.g., ``/sbin/tc``) are installed. Additional Information ---------------------- Open vSwitch's rate-limiting uses policing, which does not queue packets. It drops any packets beyond the specified rate. Specifying a larger burst size lets the algorithm be more forgiving, which is important for protocols like TCP that react severely to dropped packets. Setting a burst size of less than than the MTU (e.g., 10 kb) should be avoided. For TCP traffic, setting a burst size to be a sizeable fraction (e.g., > 10%) of the overall policy rate helps a flow come closer to achieving the full rate. If a burst size is set to be a large fraction of the overall rate, the client will actually experience an average rate slightly higher than the specific policing rate. For UDP traffic, set the burst size to be slightly greater than the MTU and make sure that your performance tool does not send packets that are larger than your MTU (otherwise these packets will be fragmented, causing poor performance). For example, you can force netperf to send UDP traffic as 1000 byte packets by running:: $ netperf -H 10.0.0.100 -t UDP_STREAM -- -m 1000 ovs-2.9.0/Documentation/howto/selinux.rst000066400000000000000000000165621324262074100205360ustar00rootroot00000000000000.. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. Convention for heading levels in Open vSwitch documentation: ======= Heading 0 (reserved for the title in a document) ------- Heading 1 ~~~~~~~ Heading 2 +++++++ Heading 3 ''''''' Heading 4 Avoid deeper levels because they do not render well. ========================= Open vSwitch with SELinux ========================= Security-Enhanced Linux (SELinux) is a Linux kernel security module that limits "the malicious things" that certain processes, including OVS, can do to the system in case they get compromised. In our case SELinux basically serves as the "second line of defense" that limits the things that OVS processes are allowed to do. The "first line of defense" is proper input validation that eliminates code paths that could be used by attacker to do any sort of "escape attacks", such as file name escape, shell escape, command line argument escape, buffer escape. Since developers don't always implement proper input validation, then SELinux Access Control's goal is to confine damage of such attacks, if they turned out to be possible. Besides Type Enforcement there are other SELinux features, but they are out of scope for this document. Currently there are two SELinux policies for Open vSwitch: - the one that ships with your Linux distribution (i.e. selinux-policy-targeted package) - the one that ships with OVS (i.e. openvswitch-selinux-policy package) Limitations ----------- If Open vSwitch is directly started from command line, then it will run under ``unconfined_t`` SELinux domain that basically lets daemon to do whatever it likes. This is very important for developers to understand, because they might introduced code in OVS that invokes new system calls that SELinux policy did not anticipate. This means that their feature may have worked out just fine for them. However, if someone else would try to run the same code when Open vSwitch is started through systemctl, then Open vSwitch would get Permission Denied errors. Currently the only distributions that enforce SELinux on OVS by default are RHEL, CentOS and Fedora. While Ubuntu and Debian also have some SELinux support, they run Open vSwitch under the unrestricted ``unconfined`` domain. Also, it seems that Ubuntu is leaning towards Apparmor that works slightly differently than SELinux. SELinux and Open vSwitch are moving targets. What this means is that, if you solely rely on your Linux distribution's SELinux policy, then this policy might not have correctly anticipated that a newer Open vSwitch version needs extra white list rules. However, if you solely rely on SELinux policy that ships with Open vSwitch, then Open vSwitch developers might not have correctly anticipated the feature set that your SELinux implementation supports. Installation ------------ Refer to :doc:`/intro/install/fedora` for instructions on how to build all Open vSwitch rpm packages. Once the package is built, install it on your Linux distribution:: $ dnf install openvswitch-selinux-policy-2.4.1-1.el7.centos.noarch.rpm Restart Open vSwitch:: $ systemctl restart openvswitch Troubleshooting --------------- When SELinux was implemented some of the standard system utilities acquired ``-Z`` flag (e.g. ``ps -Z``, ``ls -Z``). For example, to find out under which SELinux security domain process runs, use:: $ ps -AZ | grep ovs-vswitchd system_u:system_r:openvswitch_t:s0 854 ? ovs-vswitchd To find out the SELinux label of file or directory, use:: $ ls -Z /etc/openvswitch/conf.db system_u:object_r:openvswitch_rw_t:s0 /etc/openvswitch/conf.db If, for example, SELinux policy for Open vSwitch is too strict, then you might see in Open vSwitch log files "Permission Denied" errors:: $ cat /var/log/openvswitch/ovs-vswitchd.log vlog|INFO|opened log file /var/log/openvswitch/ovs-vswitchd.log ovs_numa|INFO|Discovered 2 CPU cores on NUMA node 0 ovs_numa|INFO|Discovered 1 NUMA nodes and 2 CPU cores reconnect|INFO|unix:/var/run/openvswitch/db.sock: connecting... reconnect|INFO|unix:/var/run/openvswitch/db.sock: connected netlink_socket|ERR|fcntl: Permission denied dpif_netlink|ERR|Generic Netlink family 'ovs_datapath' does not exist. The Open vSwitch kernel module is probably not loaded. dpif|WARN|failed to enumerate system datapaths: Permission denied dpif|WARN|failed to create datapath ovs-system: Permission denied However, not all "Permission denied" errors are caused by SELinux. So, before blaming too strict SELinux policy, make sure that indeed SELinux was the one that denied OVS access to certain resources, for example, run: $ grep "openvswitch_t" /var/log/audit/audit.log | tail type=AVC msg=audit(1453235431.640:114671): avc: denied { getopt } for pid=4583 comm="ovs-vswitchd" scontext=system_u:system_r:openvswitch_t:s0 tcontext=system_u:system_r:openvswitch_t:s0 tclass=netlink_generic_socket permissive=0 If SELinux denied OVS access to certain resources, then make sure that you have installed our SELinux policy package that "loosens" up distribution's SELinux policy:: $ rpm -qa | grep openvswitch-selinux openvswitch-selinux-policy-2.4.1-1.el7.centos.noarch Then verify that this module was indeed loaded:: # semodule -l | grep openvswitch openvswitch-custom 1.0 openvswitch 1.1.1 If you still see Permission denied errors, then take a look into ``selinux/openvswitch.te.in`` file in the OVS source tree and try to add white list rules. This is really simple, just run SELinux audit2allow tool:: $ grep "openvswitch_t" /var/log/audit/audit.log | audit2allow -M ovslocal Contributing SELinux policy patches ----------------------------------- Here are few things to consider before proposing SELinux policy patches to Open vSwitch developer mailing list: 1. The SELinux policy that resides in Open vSwitch source tree amends SELinux policy that ships with your distributions. Implications of this are that it is assumed that the distribution's Open vSwitch SELinux module must be already loaded to satisfy dependencies. 2. The SELinux policy that resides in Open vSwitch source tree must work on all currently relevant Linux distributions. Implications of this are that you should use only those SELinux policy features that are supported by the lowest SELinux version out there. Typically this means that you should test your SELinux policy changes on the oldest RHEL or CentOS version that this OVS version supports. Refer to :doc:`/intro/install/fedora` to find out this. 3. The SELinux policy is enforced only when state transition to ``openvswitch_t`` domain happens. Implications of this are that perhaps instead of loosening SELinux policy you can do certain things at the time rpm package is installed. Reporting Bugs -------------- Report problems to bugs@openvswitch.org. ovs-2.9.0/Documentation/howto/sflow.png000066400000000000000000002473731324262074100201630ustar00rootroot00000000000000PNG  IHDRitLiCCPICC ProfilexYy8߷{&spdvyscL)JATH*dhPJE$L}5|s~Zkλ:QÃaFBB"l(.;+2\ ϥub3d=#B|X+<" *B`}3G "6/A J()1^~XPPz^Hm.zT?:T߂D&6 مG#WcBzBhoXό7yr!_>Cd P;%C=-`-#[#c!(m37<=)_iWOi lg1"n3D0QDo>0`d[f 2ݞ9P \0DR?T"ABp2" `9燌) g߳Q9 K"sl󶭋tHל%FVvV_hy"ZB ;*h]6Z#0hkFߘx5G=r/ \_ Ca~Q]dHRLC%)r7m{mbOE$DE 8-$qEF⎾y^1dd@" gyԁ0f ď? ) iPJ@9.F @S`| ;؀ !2BB$@Z!dB.B( : P5t u@gkh B0 &07, *.l{a?x9p\Mp~ Ÿe@ѡXQ|() JerE"PI,T> UjEFQs54MFSRH^}$t6} ݄B?GЛ" #QØb1~XL&SƼLbcXVVkubc v38vñ$p8+pU:^y#WPT|;444BjV޴񴹴x^Sz|7~NΆ.. }15AOC&ׄ%"(L!9j=qL/MoJM~~AA!!C?#-0>#1&2L#YBH٤ Iɐɛ)8E 듽i r7y,l|2&,,q,E,YFYQ¬;wqdG+ll:l>lYl l/)A'؛ه9699899889p\\\ʹyùra 3K BѥS (]>.>h |O6ES*y  R:#+","$),<#&b* R+N(-OLVLE,HSqX\Q_H_P8'L#**Y&9$EҕfNnq9!+)(,[!VIL.UU| Aŝ;}vJK1SS񧒲RRҬr J}UA655%(F/RA53">T Z-RQm>mvoJi]1@:y=Yz+j(c,'L#FF~FF ƊM0&&'LLMLM̔ͺ v,-",ZwvR2Բ XZg}kcmSd3e+g{׎lnWc^>CC#j'N2ΉΏ\8\\Z\q˻ w=GqOƞ"{>pp vNupAQ=M==x}93{wO߬\~@abI`IJUPUVSpCMGPРЮ0g޷aQ ElbF.}ѢчbbbVccőBO'%\܏ﵿ߁c$Ϥ΃N&'_J^:hyMZu77fg7nv޸%}6;;w&]not|{ދ.'{z޽yڃU6?RzԧwO4+ KOM@!:22zBݬ*952_R^H }."/;&oШȥE^M_ּ }68*&gL-jw[umBluO9Ĝ\\#v9{%jgWyB,5$9CBCg^k̏:k/c?PbGR3))!i҇820?r=hvc999sN;yӉQg <:K:Ow~dɅke*Yp/K{i=}LxɻuN$={aRmP`8jηʆO$;fAq2:ztLl]sCK烾(/`뾦/y|]ڇ?~~.J5dbp4,4Ӵ=JlB ѝބADG$0ؕ99/p pPrB҈H!R8s\IJ9VSNNHsL[L'DQo@0ͨɬ9ւRJZFVN^AȑӉ8qq}uυn=u'nPKq=.ngwᇃ<{gyЇWLnnxadѵиЄdTYGO'^ϋI[jtkY{êڭ C/ }ϡ4蛘vBDk/u | $,+YYŐqG [2i˜\#Y>K0BKlb~ْW+dS|4BPU3Q8YU]s^7W/I?PHؘxƤY+%u6IjvW;:;;.㷗wqwSȣ#/?PD . q % 7هw/"%R+GŘx$RX $$DĊPbC_2oܿ@ AWo:8%5 >-}]]]ߵ3@r !7FΏ>ثw!!"4< &o",8, :~a)xï(~)*U@}B"=t(E,L V`7qʸeG_Jd Jg8|*6x^̐э9ɜԃ2=d Kf*"K+;;VJW88]\]I}~+JX!3aAHdTS $} edv(J(+ȩ hjjhы570Z76u42ii`uzVr:8oRwram}zto ]?! z_,o܋VIߧ\I<䙩s5]v\$gr韣*+뮨,t&ε^L#u&| F.ѩEɸc~g/8^+56zm2sG%ʲJٍ[?G@!'2F ;JRg@RCK#$ A{8(jD|&Ê3| (6*uՃZB H~>d6H.~ +Öcq{pŸa 7M5 m m F$? zWBI~yx2|Y>7Vp`óc?ȡɱɕmdzM)d.4*XMARDXHtLlCeE%Ies7h\Zs$5]< #yPZqحىvsqnuޝ>O _L f 3HHxqpCڙ=Y9Y,O_K[.}!"oܥ[5G, W0ilYl}VyPOiL}ܣ7~vyKWoᇛ{~0Y:<8dJXxSqр'A*MFi4^2E*BH ZY` >(JYP8:]p#=3eaOc_xq:REɣU=Bҕ@Rߘac`& 2yȭa,,.Aϱ9S[B);_.P/&@xHdV Pt:,}CfJKIηJ*wHZGt< LLͽ,,C&mmGœ7]AE{z;bn"ws;^5Y_I?-C]=1rtҳNjRŔ_)^VM,pscV6ۣw;iv?T|qY3#_x I ۏľx|p䳩ƙď/J_Vn.F}:Tj^} ~5uՍs?LRDDCʏ#[[KH$?Olmmmm,G ?Wl c{q6OO~{/ pHYs   IDATx |T՝$3$d^!-P U؊n[/jlb_D>}tkgUA u%"o{ 3If<{LLwəs{sι<ƒHHHH  MHHHHH@%@ @%E;    jG~HHHH%@,)ڑ P;3@$@$@$@$,j`IюHHHHڑ    ` P;Kv$@$@$@$@Ԏ K1XR#   vgHHHH XԎ #?$@$@$@$@v HHHH    chG$@$@$@$@ @%E;    jG~HHHH%@,)ڑ P;3@$@$@$@$,j`IюHHHHڑ    ` P;Kv$@$@$@$@Ԏ K1XR#   vgHHHH X` id~NZLlafۮ߮ްSE)W=3HՑ @0xcJKXxմW]WOPߨ +R=0HHHP;xvJM?07i!!o/|}YxrQ|%Z%۫ˮXt Z%  /w /$)ޫB#UT8qӋ}N^lQVSٱR, 4vvhEr&[-䁭ԴUav}u$  x!qx驾])vg_}ͶuV:7GM{r~֍s|IX W>+f_vf   =j3LL|Y -Wn9xʦO{}'+}կqwqVҪth\xW   v0,[^[,h2iɚEqaE #6pd/s5 a0LY"3.#  zu.&4уچ0ZBr v\J]zIj}w5rұ:b4UbUjoL_=ϱ @d p1|t.)Vax<#|Oͻ|7^{ +}m-@L(-Wݯҥ w$@$@$@!`x<)#<8mڂCSf~~% d ͎TK"nL$@$TX   Y 3 @RvLfcIHHHW{IHHH P;&Uw$@$@$@$+Ԏ$@$@$@$TX   j^cf   H*ԎIl, c13 $jǤn6HHHzEڱWHHHcRu7K$@$@$@"@+|L$@$@$@IE1%   ^v>f&   "@Tƒ @P; 3 @RvLfcIHHHW{IHHH P;&Uw$@$@$@$+Ԏ$@$@$@$TX   j^cf   H*ԎIl, c13 $jǤn6HHHzEڱWHHHcRu7K$@$@$@"@+|L$@$@$@IE1%   ^v>f&   "@Tƒ @P; 3 @RvLfcIHHHW{IHHH P;&Uw$@$@$@$+Ԏ$@$@$@$TZǃDjB1H`0+%  Hc۶mַb$@СC_M6 RbaH+vKKK)7لx!xԔ~N$@$[!!GvvvbH*@pj5͐z䬄HO 欥vLOa#/4f3Lw7.Y 2DЎqV j^$H ,>BG|233!R> ! }qǍN0L! eNS_w/M0 @wA;mza8DH}!JcaB&ocKK|aHH $v $Œ;@0B j凐YHH@jnj~1a>lHt @;FIHB a#FGǨe$`rq1! %qPhO$H_@&! $$A$a$@$@$@$"j1 $!j$t6HHHB$@"8f#   $$@현& @Cl$@$@$@$d   cHHHcv:L$@$@$@!v  @vLNgIHHH D <y\f6ǃ2œlxq,;͎ޡo$@$@D1ބX+"]̘4@[ڴ< iSߝ'Yx0  SԎ}qG>>{Aseetl>qo2R\uڰb ;MS*ĩbeGC&0O(ΏOrꥻ$@$@Fڱ{Y_gJsT8QCƳu>dFNic답>hd?cht ׈Uֺ $@$@$k9#o,b jէ2R~ twc<mqasY鐺;2/ o} >Cj3 @_vK޺; J֘S;&Sbc&d:^{2`?/wGo;"Gxrs$@$@$=zB֜75O)SI zc?KtgMH]زW2 ":"s?j748=C iB$@$@1F1: 3)][8mEuѰtO1ػ'u]cnR7;%vN>鈴]>+Hi=b$96HH I P;F[903ګ P{ѳ=?lOʜlM v+a\0{ j?ϛ?q$@$@qC[q<mcb4}OX{g-Wm|՞̥(2VR4*Ϝ;] {i@d;޻\(~%i>g6=>K$@$'Q;co4;`|ti=BtT>w,KSTmdj_z?آX8aJ{2sݯa ߑ @luӁB.fI۹wk?r,t;5D#FxN ;F+~21r8ğҶwgI6]zWМc1"l?دtŎb+uډq. @F{˪: GotYn%0$2#|$HH p:?l? O1xV$   d'@HHH'@<+Z @vLOO$@$@$@v -IHHH P;&''   P;ϊ$@$@$@$ @gEK   HvԎ `IHHH xԎ% $;jd$@$@$@$<jYђHHHc~   chI$@$@$@N1?l? O1xV$   d'@HHH'@<+Z @vLOO$@$@$@v -IHHH P;&''   P;ϊ$@$@$@$ @gEK   HvԎ `IHHH xԎ% $;jd$@$@$@$<c쁀ݶ/`>#3ԬHH P;&a$@$@Q%~G8I_l_{-o,͋hUlˎ%ԎFZZZnnncccEIĸ̙31d-==HNS{֬C6?#wϚ8,Emwftl^ {5zQ {o t#L1 ^TVu߷#W4l[M{!S+f͓Mpڵ♪ x9~è`% B`0+EG,]?q]]]%\V(vDAA%Kb#. #I *>޲tʚW,|>O_6]gߑ+,&;舗_~m}"333??1"=I Lyf=!qz?#bț+zEa٪¦Ǖ(6X&jlsgj0_"C7%M^jrٜIjG!7xC[o͚5nDNCdLLaE2o#vq7|S {7///!c#",J̫Ph͒?Ҏ+a/FDgplo}r8`i6 -cx\<7>#'Jr|C%je`FPѪ<<@U]jvcKKޙwP"qr,'7j=DЎ999&-rzNW'O ILL2xf!Q IDATG_uDkkk;}_3<*P1G 8 y]ߞ]:QGI'KgƵ/kU/b xdZr˳Q =ʶ/\)LTKkvS;{tYEZn|l7}7Yp*3ilX_WVu:·boju|ۼyϝTq`MF0l~Vc:a(E3PmԖ ?}_WW 3+-2L%ۅmw~G+|t>\a`0R>؁hG{g#Z}|0L$ܲU(b$e1i !<7eUͯղe&*^w53/N5_F^ص R5iET춾j8sBW徰NZ3aWxgRk`}RYF͠1Z&Jo;:e"{-z;WN[/[y E^^uڱ}ͫ:FOEr3C5M-n`NE|KԎ}I2^h"$ƍ 7>CE`{2dn޼c!/2vO@tnRC,?nX9Fx %(!_7K\=l W3nR%7Zd6R)NV cC|gu4r5w~nZ WnLX77T?s)Ӵcʔ3gLvM~3nn~eŮr/ P;%˫KhG|c=6h 6.dX,x}Hryp/x! 8~W`@l`x Θ1C}9eӖH z0m['(Tjãܡ|y+J-l|+?k}mTxݟSt]LgUj ]=Yw9&c_Ο"*eP;ˬcW8/^cl@ ׎x}a˗ ˬAlx!f///y9,%:m۶=B#KPEӈH ,<:SurΔӡ =_k E\_)_}W-+Rj[֯R=8;gGzKܼg84 d\?s KNVcIꮈ;q)Ԏ"tqqOLvv6ƺ*GA$a:RFGD`NL,:#R;bne[%2 V-6C#j..wQ叛˦dn~/,1lO,732^>u ą>,vr߫ØnRY1{+q ᔹ: bspWϛ4zp'.Y?Kb^y_j8E Z&:1R )H< fnhuv>^&%>♀wbuY|cZy/nTê+Vv9,xQ2R&Íe)bWu\T以1UWQm&h]_iȢӼTe#*{˥Wƒˊ+|KR9 r&0Fc$L1܅a-H 1@İ,Z<C Le \:%j`G ""rH LD=DZͫu,P '0=t_WSM7oڸnizw>@!N(]l n1]Rn 7Tk7vx!CQ_-~r( û){'MT%HCwtFwIڱK,)4L ŢHH d )vu% ,qn7wLd\Oq/.ctdxE7HH7ީݺ8`e;/=go*f8!q8(M>:}% yc=괟bM[cǛek؎6ӈv,G#P<9,;;'eAy{ȅre.D#Ɛ $K4LEHL(p"%,Q%e@_N"K`F#B'"(j.Qo{ȅ(du=d@~5%*{^xZ"XIcZE!G`Fڪn3a)jfE)((̅gϢ̈xam---gΜ|EIQ[V1Gnė=z1]G,"EH$/oCfr}"#E^KT= ts='ˑ$*.??{n(텱,}QM'իO8!2 'a "Du8υ "3"I 'o6Wb?ڵKo~gr3ϟН~->,}'NHz뮻OTo~ǁ"_ ĺ3c< @BH(/cӦe?BZC+]fEEEˌȅ#1=#7|3ƨP_CuЅ#GDF@\{ȸo߾.Cхz?!+amm-*2 "!+QQ?F9@~"]Ǐf~~KsQ3ֿx"l֡^D@_T7d¢"`06OD@98 aB#",T #2j^7n1" 1Z! GXD:/}KDxE(6WdPIb /jgDXY` DĈ0Չ$z(\VH1lȅ(ՉH+~ae q,ۂS4Yֈ࠯N"^EY@3J,Q_< q fԈ%^VT$Y$S BBՉoqH'̄%.|0ɎQ'A.#3" .dXP 7PDxE.d??':E$ fK?Ei $^[ ʶK  H* v$qU61#ć(%2LWY*'Y`x G@ JqEǁH yChF.@Ȃ"4:J*QjQO[' `C6 AJë~^ɼeEEw~ʌ~~" ~X0:)"t(8  A`,@^HgwWVA]"\2>"a?=e  $!82(jp})/zcUb)3$!pA:@V$"^š.sLC s![$A` U ,EF*,EiR]uYcwP22(Y֫<m9+3" bC.}"Ȃ\ˌCDF ~y12,jG]8Dh#'̄0`J}9"R,ˌ.sH ̈qG a& H6sV+p%Pt*R$.2ˀ,A#jJ"\A.qti/ Lշ*D~d"&P s!UDW KK}Q"c0f~BΨ3bQ$P^(apI8PK f"^~e @bH(( 6yEG@L`ԗe]=aW] 20{o   8%Ӷ%͑:8 }] vzx.ٳ7TH(~$ 3mI%:w@S֬YtKHţ_z%$alXli.)ΛNIHH Ԏ]@(! 1VO~#ڵkO>0OzĨ$UKH!pç~ZjG`_|b $  !@;}  U u4 6#al.#`qTB#bJkA'|"z#ux44K# !Ԏ1~fq7"76+xR{~m *:/ìQoWo[t Me $@$@$cST?EE%&o!RN:u oKJJ<-FNɢ  ]wm݆.\ 0{#pHHH :C T vdcn믿sKT !Ń|=/`޼y#P0L$@$c[v..h"HIM7t7 I1*S#VD/ 2OOH< @v ]-2z;S cA8X%KD{ xqc+ HcD{ qqn @vB,X`׮]X8rHf\ c!qo;&K.E/ R,`G< L`ؔСC*{9QСC'N&ᙟڱ>Q\~=nd ħܐb{".PX~ TWWcqjTW_$/TI ^oʖ-[D/Ox≟gxldCD/uoD8^W#G 5@xEvqz(BUbE_8=5FGyOmQ߀Ѻ>W#z #7@L?=oӓMÉA1왊E8rh[l0큥e8!999:|h\Mnl׫%Ck IDAT?9SӞX1'5S[gNl[F{\hZҢοV5aņ$'<{-l5PP;t55N fH1_دw7ʻ{_>K t%'٘Qv Oxl}UOE%߯b2C9zK1ĈAGǩcϖ @d.س1F{n) ##W#?$@$@Q&dp}CJ usnғ(1:M% X&(Xv%-|8~ }~- @,c,N8|+?$SQQUSY(>z/IHbcIDvvC]XyM8e=}(>>l* $jtqÿ>__X9H-g )/oPM25@uc4Xa쁊fEIt@eHH A P;Fc*mZ{Չ?ؼysuUOWUoY[{ޝՕsb]*67 \+a7Jyl)-4v|@ٜMό -& e6V[RTw=1:-  H\n,,V$̚zsMK6l[ʨQ'ƫnF)rߌ)yK X5F)箟jl>/ZKVԞZZ\l4|1J(;偱j7hR$*ƎZ"XTOB?~˼ HH p1xӚ رkמ/E=u|߮_~M㴅?, Uh;-L&|:l)V}{L{ !c|5xtkvvavG!] w5kM{Rdv $>Dk͞^j''*O7u&\bo{^UXrwv++49{/AzLjgJὖ6: V-^ig>}{Se3W5]gF%膿췎"  @~,1 gRz%ޕϙ_>4Gi!ՑÎ7pDTՖha,+c/6kk~: KPI˔ӅU+QY;K(Œ.v?yI$@$@Oa7:ZXxk ee ]33)+Ijh \ֳG- @LvoHe @ĕ1 4t:yY)Q'Kv.fܴ3,^ح;IztkCp|yojGF_ $jtMe`[!+kQٶltuTQ*w{O 5cؘ2+g_) ,Jzx1ڛpgw_#7ih ('#,& w* :;0w2 '1RQVC{=_h>,CRxŐb@e %E1(i@H/cP\P ƚnYZreAE%'M+ftaye*8_ީΘWz ;IÆ}V)1Tɛbu l8gkm5ի**j8~eIev@mqi\*/I)++M72I5{ׄH"Bs6nۣl[K{rN$ O4ڐRl6 n'5-eO9[q8݆ 'MH7k|PjXhV>Ͽh>uߞXg<<"o?uX.Y4r6^yxt԰@_<k+2g:]jE3ɺʉ/"y&{<>Vl7=Xq#A(HDK2GH‘_XX6:^vC}m2 FJn]*.E(6ӱ>?v%%$jKMO3K3`Bp;ۍ(18ct >Tz q:ڀu1Y; @ q?=kqaLU&O=']wCcc#6  ӎ%tXfOC֑+-5u̔{L#*!ZjkgJrߜPU F<SJJ怒~'𥁼ZtLVᠾGa0ev _X $4ns8l܌TLOJzQ|1$Nْ v9ةG'R;?hRyEfksr(ˉZp;}Һ^5nxU?jGp\frYnG3l.Gk+]}V>+8y*'4c@YRQoh0 $ lhs`I1\ѿXx5`Ea|L&|$| dddbˤЋS;g wge瘚8"]O*,-NOOosFlx-|_9Q 7=ݘY KoKHoIPLmwI)![[[?c˖-OFfAƢ"46; /| c$}q2Oiokn=pˋGGz79 Ćo})[*mWm̻h: aB O>sj 8Bv"$h~)QO^5jM ?v>3_:K1vdHg(x@Lڱg>ݦk9vsʈn3/ +uZZZ ю__җnSBDIEͣ].Off&-V(hß'l&G999w~'2>B'Uq?* cFmK"Bxy#!Mؗ;c;t\_pۄ=ۜV'\v7n0gIn6bWLMVO3fO݅zܠ3N.OK7A晌6҂V 6iCFg3Zڌ[g23asњ&kii ņ~YYfT+3&17|PhԙcgA#@XlW| 10l6v (FiO~{Rirଊs(N\N26hGB<F( b; 6c':9^ZZkX@#9 MN hÃcΜ>[Ui8-9ؐZ1Geeè#KKljۃAeC}@рî8p59[gh6g[q%W_vaO?9 /݉] ~+yyÇw5AzBf31cŜuV'+4Ĉ#$/ϻjoEmN`RJnnգF]lj`[ _x P;لvn' ̜8a Ԉi^{M1b to)'xBQI|AqD]4MhQ8s;~$``Y nτ`ciGxS%H  @v:nCAW@~FG#~nD%ÀG;nP2 ]aܳG1M) R0O68a<ÑMWOjOJ*xg,\R );B*j^ p=FSVn6CJ9+CvSZ]CYyXfbzV@FsKγ_u ZMTTs`J4Er"z+dݹ) 6iA)?\j;rq@HIzqA)U (2a SGXt)Fw܉'qa"#Uazh2*A)CX<>|_)KJJOyf -.T#~62>ҁƲ|(:4%5H: u Y joTlIh5F}5ݓbN5fC4Ԉ_{ DVN920cu:b8Ỉ ÇggϞwrJLU[R_*%l0%!q@;b>zڵ6 KK_}տ𨸸xԨQB;0@(rR_ZAiWs6vT67bo<_g66jeh65ٮ,-mnl9w:oĤXCyj;9wVN4k/KE:/6~AC8XYjfb!zԈ՜PKC?W' O :ګ;m!;'Bܙ?pIOLjL;ʼn,Kv~nn}t:ζEEvRkD; ' СLcpb7 zE|᯾jhG,W_O_y7tqR'v{ i޺ގ #H~;y7XHupC%';zAPKF mC N1̙-wJ6{)cSY Nd/qsKcKՙ¶:u- 5nEyCϜ9UCˢ蜼|yZy`jlnkl3d7|vľ(<#3M<lhwac@brI6f](%rK9imu];mwjK1r /#3- @Q!GGՎ#;M-2})MW]3׷nwCcТLR;pԩSwqGCCƍ]`6#qTUoƉ'2\iVXKZc=H]2Ԏӱ48ڍd[eb h|qVTgJE Ғa.7TEZyi^/H]NWnpVjmun>YW ^x1NhaqƐ1Vjp iXW4A1H1i #&L0z'N`K1wu|ĴUN}s8lݺ7 27x#1:GJ @|q mlXdmQVB:w$'/7GZWLGꖖP t^ë8Dw^g'.) H}a]F@fcqvFQ hX[N*.y8# \S27˜O AÐC,McШbA|=0%$##r6g}vɒ%:&.∱Ċ;< [?jvp -w'Xص 2q̶F@d 'p2W"98pA*v"lߴ8A|;Z-V.!/ MJ]<:(,E.Ge"/jDX*$r ¶>B'Z b=ŀY@|Bo\i<fΜ{~{~=`*6~%$gX=c@nqVŁӨ8"CV|s!bmqbx $x*4PfH^y   DxsAQ(U@Yb1${>\/pC(y6EUz9;+lx-dX- Gv\SR\T+S :BDVTT;8wTB$O޼y3"qr|'q?qMBcH \pbP2 Gl쀳7NMPl8c:b` KH F L qÒ$X$B b(6(YWH+YU XZZHQrhD80c Z[-V]f08p>8H=k?⌉=*Ug>N88QWH7p~M>΋5NDYTe!p>+fqv;?#Fd7ϝosUjPH @$ *lDxڨ"Eilֿ-`?nB J!TjϮ:5[u+uu{9~g[y~:# \[&wC 7/IwU{a /ZA`8\:d'Hogk.kvwq;sN\F ڇ对n ChDo0Kdb8!2iP Vچy95Q;"i䷃K=h ='RjP4u*b\R x5G繇ߔTp4fxy%6ҥKG_ r83_y啈q3 Ǐ0"YSSqi瘁  z{Rnu?=Rrˍe^ЩeYMvt'';nvhL &'T?A|ĸ=M@p4PWN^d}A3JЉ0dQǓH*a%NOHbbQl-l0VsłG+88R) sUV#<[ׯ_l2J9&Qv7^j\n+Dr  0"8Yy$ ; ƠFNe<m|jyO;y5חdM='awg7d։r) g!5}D}2ڈ& EEwCI30~39-L~;&=_/c8|7^z 70{5LɏåBYYfӟ4Qs qL;)f # ܍hוwA;?mNj \N8Fj4_8z8ph"> Mg\g8>[p ImFl,?;R肴,&2,NOQ4$%zh LiWCx4X.4UGj,#J8PNh$&1Hr(%) 71*VnCy9G̶ԧ>FtwUc Qq|ݻ2ȁ~ 11Y3 hOγf9%Nk=GqJX%HҼ  =a=G@{{!BŌRW])[.QEonS h:B!XC!yx0B֞\̏iO4f8qb|xH|&kSt]V1NAϚұMG6 ;M x%|^l^u۞~_{S>vmhL?|3TG}EE.]80 >ј?dGtzݳ2 |6O;%\1J@R ȥΈ~T5$ b"a3"N^i:ЀfI\bہ6ΤZ U{z1N;8qLln-4תU@, M T 7g1RG) /ZGR붦 Jm-@?]y'&_߼RΫ&>e:siiUdTfE#[XSƯܑ}#g\D-^IAi-IǛ9>pgPA_.knhlRr[#*ac0j*^Eu7NIu޷FW÷%Z)>Ԓ@Hmx5)]sw742^4?k k^R$!&SoGp6Ge~_谶uLY5˙,m;Zg$S, ,'dF9WЭnۺQ_>:PL=2qu=Vۛݾ?z̭3Nn~K_|b,?L` Uc=U[r;DAA7y\{v=E!]}{:rr^*la+gx|Ӏ+ LF8]88tj&p,E=Ņ9ݚ/7UQ^@NjI\S,]='{X/BkߕP#I'Lpw]'yVybi $8JGaX&;Nͳ1h*lP)]gXu=Ao mP~-km bܚ[SUL= RS }ndsۏ8`pԞ'1I}AJ{ej seƀOyZ0Z[ں Xz_yWතm`JỄV`.8_c/]Y?3=i7pkf'=*%&48vK[sf;h'֬ lr,4IO[sCa4#!q4tޓ" x5L/?/ɖPG7!h 6p"nϥ='۫satRb ?fi2j路=Q83ٯ+VK'{`5tS}13? Q8Vd}ȴ wg/s;Iì)+ F-^s_c_Gg/um_tmH+:̲-n첅톍G,x$'# q?o2J47RX1X}sK^NnZ5 d_/!Jrc(tb12JZ?{՚{w>B]jƉud$튼/us!v`VMTlÃ]K33=PRNS/a_82:BnVTf{`¼>kHM \=-5,g)HՀ<'jfLRJb 0^BQpAMGgf.0^C =~8|>6,׸/Z==TMT%۽}sc_#*V;41ևTO2I$3kKhvu{ dNj{)Vr@78vsnQFp'q J4IEX$<q3 AW1ne̲$h3fB|~B՜s,Onl^"&CA+\f:8tْ3].nq)rC*5M*Ƒ82 wbQQohb$"CKj`{B=/ɦ/ ̏vqs:́}P2/lN[(pO?N8ӧN ϺLUmexs"N/)ē%F^&1T%s \첚@s0זYїV$O1 a aw,tw.FDTPne[[#؊ISoh)R+WeCL%ްpxK[yE Lr2v.p}XV#q#G#{X.2G(jmTf HFh&A %gp }fR8t0Yfi. =qbCI*cEwWkN8ew[KSC/eeUZ{='7y~w<#db̠ )CAm%c;i3EdM0f2ZZZf̘A2{Kf,v4Kf Nsa~F. uԂ˼<qщ~Ј#B5VUUqIv.ttt@(xTa9S_.kjjH57@0aWpm1gx`<%bγ'ޝGj Qճ=p##ܙPG뉷< ¬Ӯ =Rv;ºαwۋ[mCݨ b?M4JuDڡKxya{6R!PP2O_"FYNtRYpI.C!>u9N唁ZfjT*p++.9uj*Ҟ[cjv3@{M|W/01I@ث=7i5MV+IqUrdx^Vaɠ=>71EJ p"Wb,G(9Ktzvt ;[3n "epx 5R'/IEE|01:  o""U  Kґb*IJaNWyeZԠ#P ju-$BH >fү3K.Dꌜ$;ùao0NbDV ema4"gn jUg0F;N4ka+pګi/;?W'0(HiCFWLzJ!r1ja$>u8+ i@Ơ/XT9j ^u'Ƃj׹xk{[&gK^vaW]-1SE*H7"H+:,K谾D?:m7bʢ h[Y"l+:l՝: XYkK' l-C:Ibe[OaEE%%eN'Mg%}h 40!c `o8e0Y`2hcS0D6Bn8:iOzk6ҕ)3D Ic^1[,=F<]&= h#$R3-ڼS Z>T8hXB?+/:CJ>_Ri0 uk%j.gNd֊͙v`.1.JT}I*ѯub1:ni@HT}&^g΢!8 a9sXFbC$qE8oF'c HnR%>'s4h->Β :%;c*%w2;-)KGCб2D| }-I2 hrցC}t@+k4ljaT's%f|i/XbRs*JDDL&.S"j a:¨KfىlDP"Vr֩ s̱`AX Y(,qfZumstH4D˲Τ" Dp#Hgf#kUO aKI@MD]V)C@c|x4˱; Sp_,z'x7~TfL΄t&*36 xA@HQehdbaѹ6/d@tb8f*œ0?8P4AJ\t.IիE""n蹭654H+\z3Yj䵉 W Č}(0B^JrѢEC(ZItxPV"F!()e_SI8[<$5WYd;c;{6x`/..s%wn>:Es.l'a^Ro3fѡ}êȞ yΛ]xdH)9## NxEճMon;fk.X9K;'!M6֠u(w 3 d3ۨƠá$Fwqd<]7zP2s0YbdWJbcL锨T3Yc,$%VDϳFf$d,Cd!",1ő %6F_Z Pq-ZXc4 ~`~ ҷqSu?-_u[KΛW^a,[hj=ShέIJkݽ3Y&%͞*X |gysfn-Na~g< ;eT~pQ O'Lk٤)ǯQmy5E%L/ÇXO@dG>V1̈́,e;CM$ N% %_F+<ZآҦg3LE%rPAچ "PEAEX3{ j˭G:!U_Z\~8b47PR@C7:{Bc-*53ju/ 9xW,:];O7GFvwjʘ[Æ WFϭշJTăE,bpYV֯)ZlgWҥqgj$ǫ3`c#~p h:˕ d۲ 5Wif34(R)iLm@f$#e7Bqt6m$ܣi&L^02 (qFB8(#՚ #$[@ P5h%cbz5e^?$lH=}CUD_}s.2ĘK־c[Ovm=s⮇W_p qjyjM@㠬=~as[suAmuGa`iͣ 3|N iVǡ#1^:TpGf'Ť(N!jgH1eH2ҦvtؕD%p%0FQ2vYhC4fMi040!eizV:MO+{*Ϲ~_ //C|t )21H~7c7sW,m?诸~r`={E}jxK>A@C{8Ɨ-Ҕ;li1~Fؕ" )Ҿ8A(z 1I#h>Z德E+̀vn ;|fƮ=i'yM/O""ݧ|tosc=G `{?}{H!?eK vGO_>'WD`P+AnzquɬwUNT-H|љxXg۳s;kem]tX3{nï44mz0\/{޲5_̓q _o0s8t%Wk 3\SXR_psty.҂m&bP- Z )^)ч]301Qz1[cLh1si2Ţ3=%I vH*G6mm&%w]#+4zƚs4l9tJ=[\l/uF޽t߬%5 =Zz[-:@" jo9m̖85,?щYy`C3Ѩ׌\/R ;Sn=Q 'dR١őz09V= Oׅ^'SCg=R.v5NjnwF4Te*k i8kLQ$q4o- Ayu_~M/JO~$Tosbvhk3͹tZCy '>t&cw>oAnHW[ʜ]_.=3teTjޢʙfc-g<|HE\ @Jh_䈗!R%p)uz v nnPt^vߕ>0-+]VvzyP䁽$JЎm>{kn5W~xƴ9JۚշUdN3*W|A>/ix;zܮS͇[Y?>#KO㚹k̟o :rF4 p^uεENa\Szp n*@ }~]7"IA@H7dYO 7[_^\Nktޕk]ٟƁq1k{x}}Q{ F̋39CG_ސ2Z'82wb{[XPq*Sک=*Ff^rKZ^.2p[㞖\aAQ2 0SG]8deme.[P'mQVS/t\v/ ذO｢(q8yWybwY^7{zb5l`We=uNmִi? D;^c 3._pEoXT/Ѱ~gCNӏWaA@Hrdw, "xGFsbY3SΓ4z` gw~vk.a`2Bz jHb㏇A96#G!4lnힵqٝVaE ;ڻÃsF>Iui p to= @GK5:G`LZƁgcL:M)));2C9J dT`5Ϛ+9ln$2Tm|,VQ<'Kk{=s$ǫp#ذ`=͎F$U0WuЧ͓* xhkk;um4WfxgYQ>Fz"J8 $9T㹦d`ghA!nmNtP+iw,*r$r #[4,z]j@#kjmFyA`dZZZҚLeFI@8[5=ˠT;ƒ 00 Y'^ w\h GIw6 @OԮu,{AuAqU4 1^r99a>a(m7zx{^0AG֨O FyA`jMER4z& pcfnkmE$gMC > <$G&Lr} u7 YH<CCmM[vfzs˲<FD*i6so1/{zzz;aLR5!2:NMqFL( 4yx@%VFD`L3bΌLޛ[A`:FϸopDZ:MY-v GID3HY^AgvA- ڏwmWďpvםkӝvy.7+z9"Hwo#UKf+3hQN8@}=@w?l9`?EዞEU/ҨI8EU%AGיYUN*5IA߇)j5)U @,ǹQLicÐ6!|1ABՉsPJsD {"]=:! Oj9talK7w^Vi2E`n&2:}T4ќcERuS27H~Y,zy7^j," }"6P^@<^AA4gT-,.ɯ@C3H&H3$tL#Hi4c,=ؾ}m`UM],/W&9U T7#B-z-ʈf3UBz#]NGU^*L Y-!:O%;;{b?j P+ + =,<#(0(qE w. irk mvQ:cruȈSxqK T41ڻ~6 EcYI~VG.ͦS(/ANj՚fžhYbڪ TPdRz`a(A`tZ[[^yK,Q_q.:3KC,ep|Q8S#9LZᎣea"~tS;VFf&ugьI4D$y3lgJ.7lۖPܬvZG}|j[9?U4+e̳I>wD DfYj$0~q#.|>g^G6XHAp&/?_ġYţ ǘlnj.)*#hĄ;%+wMV%]6@qiT{N/1gVŦ̞UQ;녗bcNm/#]/WTG ]Jtz_ ~O(0= 7bG`LL4 TEj=+VY<*|G{2fYywR7ؚG˖iPC3x37# ]>VĪ:xG5łMxŘpw[f_\j);8&n" @IQfPGMsG~sssi_YޔmiX3^_SboC7 e~a;RD^z"ݑ=sz^[GP1Vb#//T;MA@HSp (3w~5tg).hVݯڳ!}8G+ & z;_| G\U BW ]3/yyז49M&Hx3\aG7\4 IDAT;?P=?T񠘻 3IS\\,Y rla>e X:(0Ý(p1nkea Qwp|='c1teL2X‡^y}X 4˯_k.Y2ovv9 l&TQ ]`=ul 9J̙;Y;v)Ë-`D̝;i<}ytX#-2ELW18֜g"/v-q t9xg{zzp4<CK% 䥗^7)_v%\?~Kβ3 =99nl#"m$tUs8~L%g#`ͳS2 3fYJE/Ř1ڭYF ;dKkS {cm} 3#F_|ߗЇ*I"FQQQVV4 ~ eWHjDR4u@8k:ZQ~0Ոcw 83?ۺz71i!M7l/} |mwd 04g7pCUUQe0 掚>@rXqLʢH& W~~> 9' c&Xcwf}mdG*J_9Mׯ_O~W`qGTS<\j_CW7''a9{"?+IA@LfΜSDfMSB9|@A+͛ov8đE8^K/?0H| ѳGPP"YGF=hA/X.Wҍ3%aqVa67E>l^yla8)K/k=V?%Cf_~oDp-|Sw eS!zHUě`T[|L@ 9;ＣGH\-^wtEt(WQ!#K JT.YNEK*>SAxMdOF"ikkˑTY"ىG Q.1Z3„2QE`P*b'ѩ:;g+UTH.9aE*Tmh*1JQemsaEӧOP;K<6*VG3P`X$+cEJFB@{F12M7O:8Yxϵ5RJ Db |Dہi_k 0E4+hذ>|, #Gs9L'NIC&ٳgϙ3G2@PC,|j %&j!CP.1G|Pe˖ajy~gΘAX+U&\ρǷ4TD( c-a*ZW_8b 3![[ΏKQQ! ;*WDvVooOEy9zYR\JfP(.ZQ:(9Ǐp1&{OqW_}?y85ɗ!Ch8d?At1lp xqY+}ay 2&jIB\-DV&QtUeG(*cc.PR>GFL`jVXF9 Ch:YW0ݦ/Qz#>'1#Ą;D.cB7GF\s5 j!W\q}ihGI  PKsNW QɒA?;''77ץ浰t ;vG(|Zt0>zD` e7eX`G{{VNF.B̓K;:h!60kDNtP$ {h\{}>NjkM(6S ̹F@of7ܣinrϱk4c?%0Ģ\š-1hgTrE.iʰk VG:1 ƺZXۖp+ilnhlj?AWg۱G+:~rjGxdjgpէOAAfuw5`;\NrWVU1g1ѷpcǏege:qiӦ7{Y]]y Rk wv߷fF Ғ "`â\5;4FyqB={IQi$UG$j5qܷo :/[?+])@j#`e}ݸT@{U?t$DZ*Px$p7$KYQAPzm8^U qPZ\܎(. LvG>&Dvˣ>4{U@ڔKkupЧJ尹sgM+Gy!0hވZ,~ơ\wg V4qd 0Yol^x??YzrX9A@4nZ|tG"0=N8<,-˗߀5r@tTKj  +35qd##=z!8Jcf>RkA@ D@c#;Js|~dV##!q`iA@A 5lY#+8}۵kŲ?̾XVWdzu7A@A`8 i-đ=c ۷o?I qLڻ&  pI7kRlQ͎dΜ9\^^tUA@" qȥu>Mۯ^{Κ56JcZy  0(!_+3fđxrVKM!ׯ'See#<2}t☛+-a)R @z" 1=8jcoo- Ǚ3gBit8T%  fwL:hn{-,CW5Sdt#[J8l @! 1nxG?3Ϡ=c ,cuU{<8^%  >wL{cM`)>}cO?4J語7oU-1FlELA@H{;->SAhb8nhhA_O|⩧BYc8NB  Џp~$2/1 >쳧OfhΝ;a<ŋ-(c3๐*  q 1RZT7:7lؠ+r{?a6?Ld4qt:xJw1^A@H,gRk{Ɨ_~+Yew{K6ַuyA9X1Bvq T B)@vP(t#GPehe-^Z|9BIϵ)CR  Hc*ܥDب͛2 .2Z-O"  @" ixSVIvdK/Su0<Ӯ7nܸk׮UVqB(1  )O^g֭Vw'?!Ї>$-$  8և>v옮!!uW\A<,n $HA@ " qFvgi&\%S%r.lLHRei +  pG X|.֬WA91L^b :22=: y}kFMjoJ>L*pA IHi?#N&I3C2z~D/5 '<=]w_5g/=ߛt\7v{ OVOYT+ʙ*UCqLxz]P)s%Jlx18ܼR`6~n]*;/Ww ~w^dXH85{[q aw*htUn!+6U|QٔOf&ž7O8 Hώ΅>jaY5qvLRDywTb0]W/=wz!U?[.c>Dغ_=: -.lpGK>hܐ0f(<^ߝb(F>_ȥTeA@!6N& gƑ@KW~M)w`N;!b;Q|8bP>7Č)`TMu8MdF{$I3&li>q=A@85pJcb &1nTQA@! 1A@A@A@2p  $ 2,]zi 2& )sPA 5'-3&U|t Y^ƿ毸g:Zn'Fk\Q'F.ĉ@Nfڻϫ6?'">;Tbݏckӯ\t#dQT=fNatOt c8{h͜"KNl5=""z@jm9͚ [!\Oh5歼\^&E5/>g÷M~HA aɬ^}ı7ɫZ?t;#G@re˔xj]{u+??{j13|dm"G}֕^jn CY7֘~_egY]|r?,0O?nݓwG`oݏnڦ2n۴A%m:lTNPي2v^Evݖ>پ*/}'{bC/?mͻեyg_zZ- d&v2w6]S濺+&o[]kK-31Zmg8)%!ܾVwK7_~-e-Ys~?hg6a6OH]OkYjΆ7)?Max g.=%+v}k~X[/CeknIpNm_Fb$+QNnN,o%iıcSߺ?xA!wWhzs;N!j^5Ni%;:A ='09+Q 9e^Q5T pe*J2G`7?R~޹ט3o߹0oޥu5OVߗ~oyNwڻnܲ+=sC'' q2٦vyN#4ƃXZpDy6tՖ3#'tMgYFۼ.0㍟iHظ%ˍ@YJ頣M7&.Q2O!jwwrx^]CG3]z@'${.m;sDT&=zM#?0ј9h8>(feRa}E[2i R-)0׷ Ք.1k/4 9~b0&6Vtx-+cU3F)Iqю5ey=w-: @W$ L;\,:byxr@g g}֠NPAwWYW?Xwf×w?Gr|¨>Nʜ<x$3joϬT<qu￸Ӝ68g^=/v3=ޅik)$ ?ϑ.?&74uj=u7]hݱUY_lϯX-3w1֟H&Y;#EFH&7^w'g*ݲqVC<;N-+**S΢a_/~w<ڻ: ma|<[V lL1S9-+Ikwas-KBK7h( GSl޸a%JIr2̽_wn57`E-R@@Q9RGz#w{zu6FE_Y{4w{B'Pl IDATݳbOVuQr輟,5ꬬ_f}Kt}š5rkp#tVU?Чtlq\2&﹨o_xyA@H?&(ݳEmsAQ! 1nFl\7X q*Qђ{Њː@'bdսnV1i嗀 d<d;Ylz  ^JMA@A@lҶϚal Dk#9cD$C^ s3U+ߩz4iZ5@ (l۝N?‹ۋFLT^gO]G#?xB!YrK܀䎼s _nʱKl;naR(̞=[e\>ZBA =#/~`Ko499%Ux"b0!_z wZW9܏]yE3Tw8c}' W^S1Tx#J NI&'Eul&H}pH8tw>-N:IU{D=1];Avad9]EmG-**/[P9͊Dx|qqB67nj S?ϩ[=/h"\vr`ܕ97MABFrltAcC}3|k_khhH;Tyt a/eee_җ%fH/D#0AFB/۷zm!6 [VBhU0G4[ "[7:w}3w#DOVʙCy1}`0hNxQDgD/ q\YYY999(OxhxSM >fgg.y8$mA>Wt4vݜK;a]0,'h9_}֭=㣕wPHE :܎_u}CO01{.] %[wLE]o֎; >sIK-Zdl >!x0sB<`BL qW-ZL‚lk཰&C*Xxrsft>ӺV z\Ƒ(PdLLs >30]thw}EM/st=9>32;!nHWYa>9#%QO;%qh34盜@91#t_|^:~xtjkk?]奀FKNyXǓCh4N9,b@"K{r&hL/>Or-xinwX$o>ǒKd@82 w^O<0;3!(1W"鮻"LQ5kj燐ʮI$?чE"y睍7#)U dP"zdDoluiJ{p0( M7ywEw`n]̯FGxR\`׮]Ee˖=4:aMOݛ wL:Lt ZL;޽og:k,DGioݗ 7CAZ$Hd`eC"Yц qQ} 9tbW՛o2(%nb)18 RBnzя~cQ뮻=BpDH$ˆDjəM"O8?,m$3o6&ְҸ!xdA +sf}1UFGX I^ 1/Q[M9x(6oG>rA /y&dtµ}2DٴD2jƛLͦ 69{" noş'MwyY22 ),f; UZ?hON8f^~e#>?O;%[@9VlI*4 Dnڴe;-zjfp X^l}d$ I~%8fk<34iثT8B܂(s cA_ # ~Ks%.$b!xn7al&c۶m0$j $0>vA I] >%G3D7ܱo،'IGU@0`; jq"(+f H!%IU;x+אDDJDDJ0i8 `"2dhB׊TMo/A8Ӫ`Zw0nJJJ6S10&K̦$]6ԘJP#s# Ο8 Y5g0]>|cƯP  A#-^xPɺu`ZpwBA5rф?,d`h pSNmp}"+"2HI #hgH"`ܽ'bI@0"o^)(|rA5` KwC&-POGI @SD |C=D8ќ +dZ!e$ 3KĐa-`1$h Q~_~'@<(N u tU z*hA Nw޽{z-O~Bdk,Js~ *wiNk.kmm象z.\^k8-w6abF\I$q$ we$0~q…09DJ !@>ĵwڷ~owEx`ޑ~M>Pf}:Hf>B[3p'66GQ|?II,S;>?DCC :Q40sh!xI ^!7o ਧW^y6c#n_Aj:|`+ 0s ;lnpl1HvéF2@`Y# #H.^}텅xq„ 5{J ٰ#y'a+;%S;zOl./Ϻ"%d=|s'A~%D ٫$7H`S>s! URHhB`B>"#(#ЀzɌUr@5/Ew@g8voV"8"}W5wrs1$HR|SAPXk!"jrj6Lp0:B렎111lM)m!LAv3 ч :Jb{: 4RXAO $Sp ?iifH#H, $HU! !&觟~ Hqq?ϖ/_uĞy |)^B(8h؀0';}w p݈暄{ .;c@D!a="&$a"{u>u8HXC$dp8lmŊ<IpIy;C_v_0/Z d|F @ꬽ׀U!L$6pSFp )Cx>>6?K^5JUC6߂#‚œ TVVb sՏ>_R~!tLo[!b D؄#r8IǚA־}.] . ñZ$C#!e.B+~B6 M7+I$XF`>B$#e"%؃>|8KO7dDw'=áKƎ`LwCX,Db ]6"hXL@f H†L-BdOȱ^زeկ֮] g2SH#Ou8i2 8Flv? ނ"!l&Zl5773/ȭJ!c7ӜfC I 9h Rl_43w::BWP`IT920FG0,uۉU \! +Jjq͆lDw=.0J==u/@|`󃵍p|:7"q;dB"{w?ADo>òݙH& N> Χ\ϟʃ>:lfH~,u 7ހK=:u|{= p!q5ॗ_~y͚5D9P8Db!I$/Hei#x H">~j6ۋ/fٲeAAA}L8]R9l|X!̘1^#y<0#]AX{>nL$)OD7|&$H k6G0p͖| ()f }򗿠tѝ@ I+?;O{E!(dv,7|~CHuB<ܙHvc"GbaC[(DBM^$ AE6\q8xG>{BKR`c DzLm7:!-ás8U7ֆM5J $E$bH\q0𿆕׈7A5qYxb +I`"Q ?k׮73"h; IϪqsBi "\Isz6خ$W袢"r͛GyDZ7:Y*g ~!$ p 8{Μ9p2  %ɠ̈́  t_?BwĶbM@D]CVi?5G *wy^  g?@Ow (`dIHK Ilb= i5C,EiXA|{˔ C3 zh∍OdhmA:f+*kD 1%K#ى1XxHIHM$XLǎ5wuJ%=|hƫjt6f9..ntĹJCNWO+*ګSӑQA3i${g1--b15>LZ]6`5y =wܑduu5}ٷ~._ $iXY !] V+jTj*[,Z`KH+5P㨁z,4D0HV(ĉ:HIXiDb! Đ =B-qQِD¸}( eC9sL-u%4O!`b)b3w܋/BGMSnɋd Pggff)0w*&=8*i(g '-zy IDAT愶yPa:j$=,p%< }@Lf"v<,8!Fg|ܟ2e @s=*"w1"pRq4R%PL7bI

qŶ[bBRt"M׵؎]J]QŠdZ!B/3z%C*w=ZLo VԙɑYN Ѡ3!-V4X/i8}b٠EYZȃEi[閇$.ؙ6Զ[pbFR،Fb`ZխNsxE\" HHH" #GyEر-]L$$ }H%g䱿2[B9`汃l = I-m.|uJ&@zYJW4JP!4[ *3dLiRXb=q #՚l%6\lѰLL<\#DX` |j_͑#LUVjrwLO1!$#X)(@1.3[[pfG3F^bq`@$h(F# bZ `k''XkUͶ&>Xea wUp\hC-H" hmš*K '3 KAvģ"6-V~fbgN0uJ0 ÊB Ō 读1(C y$KM &nT$X (m0[**[kL6ͩe іhA` e<L'\rPH >A71BgԹ0G_8qSp+.YRlrȷIsWlK^dMmZtef~usUI1' !n Noe&Evٙ|1s'E@%2S7П~;D^7Llyө҆seM-vH"0EM_#v6M67YيC`&fL%;A:xھˣ̌HYrP|1R)ocN knl>Bv]8|sR?ƺ$j[vߍOk^"͸Cjp]6$`"%HMx\pu;H>ru/6t'ZKVcԌQC-G?c`hGM6&ʵ: F-pӓBS .:j;vU+gF0),1;駉=yNJ6y<~ dqSJdy6 T+zNʘ.R~)*9ID f #I$L$#bHkƁ"5a I1d];zTEs$Qz> C}\nb.':5ܚ*j-L`Ѣ;Ww x"9iNZVE!0$PqH ǃ!i^kjDw BFQ)WW!,n3jq2'sW|JxgI_>N8JMB$eliR_.k>-}c_}qH%L$8Hg2@0{:!kvLLHtϽKHASPz{8ˍzḙ11i-wX-h:kv,<BHn⩡vÞ ރQY" ȴڜ0˫iX;,rZv0F&YDgƀiJiAA4U`uL:)P[^>wevlzlrE)@gj+MܒJosQE,囜Q[K?F f0oHE"X4L*6 *-" aelA>;v H@ʂ*-!锑MxZU0GK7̊͌bգ'Gڄ=g ѢSi3w[y3p ~Woo:1)Ҽ .-@!)PSz$[*1E)ԛkZ0 UCՈJhen R.TYB@P 6D8+tl93xnRTp2_e679[VѶdh2g]+*\g  W!$BG$H"%H"Sdmf#/0eee=#pF,a/6 !),i)E*sI] F3CJɨY^i xD8>?إXZ`Z2ifWć铣>&:K귎cN $Q}==>j67H{mL5wVYrxzޤwDRP?Uxj}QeKU=!X@5p])Wlاc4pgQȸe p獯Zv?,%?8wq  qubSp`ܙHGDxH08?n5?)ِGuZI`hxh.oE1zY1ba䁡ڄ6”:b\-ͱ,ECs]ۍqS#E:"q |5d; mPy5$"c QTU>^/O {&6`A@˹ (BцIA1!J6|L wl"tA'8ƽñ$IŐ'ayJªh=;1w M)"_i棴"BGkx?|:x FGFMÑ2"K3aB8j㨁z H h>p|eP65;, }zC7)(/:dvԵ*[8 KJO 19,Ҍ@x$WH$RCT%VTTlr.})p\D\=LU"PUzcaFVXp|CFo7se w N ne@IsԘiQpt)43DI "%Hdpf_xL$R7obHĉH Dzb%.R̑KxP#|%yYzb)nۯ6t?"K*O3Ã7oE7[dQzG>wQ8WO iF(WNQmdHhp0xIɓ'A5qSjZ]ojȕga4Bw|h υzt?#izb(V]<@><(8xXy$Lٶb9AN7^XlR*.12 &s×3-1&u'H>"A]6)2}qSZ;_NyIVjPZhD1%h^L c5^Zu-FL Pq4<`D8qDže7G}^ +%$ѓpa*1Y#Y$$'nB0̘8U884OAb=E7gRتA긭򎾟zP^/Nv?/UwVG'M:%R7z1<>ՎS(xJ뻯ӫi\qLDGݙH!ڮ[0%*Ӈx׻lQO_4cޱL^В>c81 'O}]5 S1Pg' Ʊr+_?wZ=gCk 7)_y"{V'Wk7O8券n4xv~=9\UѸCwK' |$A^֜s(zx=fn֞ÛRz;I^^ ;4+B4CeZZ@r*R;Qd奟N \1cs=yggٳ/ e̖-^>i[lOba"Fmxhvs*[_~dQimSSSh3=teړoͩ</Zm:푵`Xr>?YXT[BLl^r_\RG{2y?VޒΦwך%kgx 3gdf0[߽(^w58 %0ws9]>xS?eY=TjL%?R%>2ibR$zi1i7߿(SEM ?#ά{ґάYzhO9HGK13vXtߣ. f5QVGxj͊ j賙S+47cGu&026濜-tҙ#/|2)eہ7E=,rܞxlu ÔWD|gzyoîC+ =^7@?؟_~]dHtnO!0=~(?_Vx0Hm|6^yy,c95dW>uÇa+~K+>wNsD=\]dAgsp'~걭=,GwLJ7^~3-<HZK|w-wAj5$ )dnؼcZO[>+( ;m]׮޼mǶm[e~*d:>cu/[ٌm}bMOZr\ Pm%;Tᲂ~vKd1' {@N5kFZ2j,gbVn\1yxzId޺3+7E{# .tצlsScE #U_ F}7/9gG7\>O"F4-L;/dflik~c]b~"] ;*Y8R<3=:O;sP)@j8V4-k\Rl͙Q+Q1za#e'['0GbW|Zu3&D6,'$OI m*YrOw1|:v?iJfD(J>%?5h8W|RuKyU-g,B+(WeN(=,` {EZ:{nnV+;p=wf3Le >`@?-uyt֘=7c0L3=5wV{mpRy  `w8FŚjk[C8+TIBy\-5F+ bDB՛w p϶ D7n]k+ a3Vڴ&eh+]yxFONI#jpb4a-o^)m$(H ʞMV*^qƃYL$yobqI3kCB{}p%{:#'#2V㳈;7ku;v;zrw5XJ!2UT$@ W|4EkWr/OAM?> ZI2 g hw諉|}~za1.'-<ϿKo2]dmpGm_zH{sC{^P֐;8~{jTeV~_rn>rJ#E^1 EU{_]f NG]QΣKs*kþ߬ĥ#܆3S22 Ka-knqv:!C192sR2R-$]?ZΓu3T9f"u¶O/)3'$P%>zofEyZ6a=$vƆ[; ROR!'t(NwC֕OmbӔGu+wTͣ܃1tW9?[t&)]E#t\9;Wsv.@?0}}9՟MS̈dLW )I\nöoZ~!}ޭS\% 㘵\q{uz mnRuz㊭{gud "ovMkNٝd8~nN طwmXdg6t}谖ngS2?u$3tQv*2RT"mzI{h*+ H?dQ-Yu&tWPqhI@oH]jQMPu{,ud,6ŬEM,{ #e@x+(|8bT]Yavw//6N:TQkp=_꺝11˷ݽ?u8lo#owCGVHRo(K]Ca1Ac1-HO)(85R@J F&hA`n,ٚ{gƁ@Ro/D] ˿rǼǖeFt\@溻 \-!dcCK[DX".Cp ք K>>@NjK.Ff%!x K㉜v .v1#{e)5RGh }LQ.kBJT?|| nY}>kD+ͩ1bF$t)4tx~ ьctǰnE<iB \×7LBnD?+͎ԈCߤ CtB̀LT#"w fO{8nj:wq3p("rN!P;? YsRTY1,gaVD_FFUT?PI"V{ZZ AƄEqQk60'i93:̺k?m 6>FO1gDBtG%Lմ1 ӏX44*8aQ MjHS4EaRb<|7h핓PlA*ѫZH N:Qb@Be0{YDQm*` ;hWB"B"Wѫc;W!* 6D+`kldG!}`1Lffx秥2hl3G9CMql|0ku CW5f"DRg(u=EM#\!!@f_Ӧ;?+!:uL:~`w@ fN" xb`-Db /;6!T*:&w. @윂eXBmxY(`ݦ%GX*z,.X~]${C4mZR)k*Jx1/VOV%4>!L$8HcWYJZe)Fk>q"Tay5ziNȽL3 EͤՎ PqDI HJ2*Z_&+G!X҈FE_>rY^d"ɫZ8vۊlsjsfh-p 2>FiQ;26%71 :!BK"L+ΦGMVG[1QN` l᪨̈& _8:l6k,)c`6/U+XVB*BJaɆ$9A e4ڏw{ƨ'.2Ň^ߛ_11F҈O޽; SPGdG](&#\r\jgMV☇.%Fj#4b7EZ'BBbf҉ۄ#\#A *dQ:8yn4XVVЉfF+ Kt}ok|sZ4eb1 fO+)6G.5u\~ap.=Ts7U~V:n C[m0PKGBxWEyǑN>/K CW%ʧD9\@GIC@V`dV'oA|Q@IaHó\9䕜:_-t;tLu7ZO8券n4چPG*T ?xVe=MS8rc;gaU#NWV@Gpg_mua,Y=*.,;TT[vg1)2ѓ#b,[z%}Jb*Nnhy ]@ KP!Ў\ )1_LSAN`#Ww8oMK$(!QuoUFm@#>bRTTg- Z'@D A-9A 2j 㷗/T4?8Rf_>2yN:n=c !rx\oխp>7uQdsR0 h)lLV[>>SP8:G$1#=C("pvn,{hSƽD3-|WTӤ9PT[=٠eE iHAH6cMUDb_3,sey[?yYڴU}?>&&4tFzf}JA\7TNz^6l6u#NWW_W>>CX_qo1k ./󏜪h/~ʔ[2߹(]hݞy9ηy)4>7_s0ڽv޶y1~ޚdpiXA\kKĊ01Am]eZe+qWbH$`nzBrW<أj ^iS0[]{V!dwPkSOs #@xVTρ̬AX*!5f/8UCϡa0e(ɕ?XOxsC+|Mkm˒=_\\Pu+V̊tڪt<,س)e-[n]v)P{\N1Lݳ%͔kpk<ŏXQ)..۳%cݾMpa+_aSn$s7.]U$#q ht3o76BEV"ֹFgZ Q~PRlq#Դ8|r{BͺkӛݔoH"{m4Bn)Ƈ>Gۛ'_k"\nBҞwDZᶅՊr'H6Di&@$O 9`SJ: WzEd??uJ|q\uCڜ*WKlcuyێm۶n۱yZĬl\cUUsl^iKG\7SDId㛿lgm޳o+/Nng7dg 5ÕZh;uߟgNW%\UŸ-w|T圲|M9WjdM4/ o1ܞh7GEEEGGGFF=5eGg !wб[eFyrw9+!W/#c^5&N/,ki?$ AarCKN;o쪣d aZG7dY'a"!a$qGOnu\Ò~J)m\6&}?goʃ/oL~Xu;#!K~ױ`&tߒSKڙd>S 4Ln\7ϖ{ԞqՅZ/JhgБ|Bܸ{Sv 2Ley%tÌ_^ebюÝe'i 0 8Hx\|);J^3* iS*E^DB I7(jP@$ q5x?Rη9}30v'äbsv3@ {zMRA1ck~z&w\c {nOR+VNr0 O0p])㓇A rG3W IF|60;zlA-@*&*.4:I_U=[ 6yծsw_!~wEo]wRO[L$Y8}"N\(!_]Ty *:I܈4ҝ'z Qyg]BG3&2kjW.teRm8Hk᥿36FCӒƇ3a4hM  lXX(1xp"F W;w@*1=փo!@#k]\_u’5]m3b`"&wӚ5ͦ3V~FiqDc 9c431 ‡"@LL pJ"k$8_;R@ZHasu?/)9o}̽TvnwDL,~rYZ}8502;R.1o%7ESg1ys}Svּ+'`ۛ˼%k7pW1d3& 8qH-H9/!3@r_H1sru |ϧ yGrG_`f73! Af&w9+?VO"nq,/wnfD%v5+;X,̝sS7l&?c{t;+;眨}scƺ;+\k1sw/;BA<+L򅔨2U@,<[t)\b 0Ssc{ O<1c`gre²慏7.|;p̓[P@K>INO\ U""w`!0$z;¶ŧO:T\ؖW7X[0R ܑݜW˷8Qb\Fh83$ߓ6P 1ZS_;!H^`\a'rGO^yfg6/Eڥ@DL;斾4sy.=g!GY3W=2 A -dIzM[j|P fETKfoi+pu?s"S a;5:,uap]O/\s7xd̴uP! b2-a"SW:;cAԮ=zDn Eʚ;Aq?mvy3+tRǝ 1H3b]~ Lz#t'k ~0YpnqY-:8UkW:ys8 FL;"='6-G; -B 8Atp#rG@XT;zlRzƈРms.[ݒMXcu,+a2esGJ gh3՞9r%7ĵ%Xj:xJfg$ĘHvVΟak6*E%ۿڥ3KY\{. K~E\W*_rܑ5v<@Na bufP f sIB n=\XgT22Ek}]z[&W{7ly3~=b'yIl7#]%P/0h&{5~>c cRXc`%;w@ ykulONqR”D@)B&х;zA>bwHFd&Zk ^hp朆ekΫ\V~&З9]qY,W%*PfvzRN߹HYqߎ"1#sR;jbcnK}T'ްAƼi6)D5 M{npK/g~v_IBYu&?o+b_F/6rm%1PT!tADڌ !I!y{c-odUw7S@Qk͍.TC艭](!ӫ&rOB_tVr\iGuFlW[@6@G,@@7f+YVQHMΦhfδ5+)z\a`!> ms<TQ# U& )m&ņdj^ʵ,BHbvq6'Ϥ IDAT"T!`V=5~܉`η'w?Vn1ǪAJfF%V'`"b):@k6QB@'@@5mfNaMv+/ Ru pF%'0lq LZeRkz!p62v7 B…DE4:|wZY* E Jqg%8 !`3 Fb6ieC @+D H#qOG(&n!<( w>0|CSaQޙ,)Lha@~?Us0-|#.h HH}ů o ;>||.|Ў$@HpmV^n\\P^4k2̯μy~SMYtdj|a= S Am Z;q6 GБjGfg @yGg-  @8HDl IԂ(@ #_uj tzxyP,jMUj^=+me.6r%Nۂ!kZ yX>P3GͫT$}&! \@78U'p l"/F9{Gcu-|YpWB>9o$;IbH$qdӪ;ZC/+ dH$&2wDap4fVdhph͜X [lN^:l s*-BM4"52BatkF+_oahuBu(p>D#E2"qF+tw`Pɐ2+%eIeog}z Pc!$78 * y 'ȃUÌcC_&Zdfp=VSx 8!6{Ҕj) ;b,ߍH=GF@UZ V5 0|Cz 'wv~c4;+6UpPxcZe&F gXjAMpp21zW2j D  F`dNcMkq2:,U$q $ِhJ؅`KH% !be4iԦ+6kTr pN"x=6Hp(Y {iX\ʚiB&AD-+lMJ5*sXB6 9Ա=:KQ/tC6ΫX&T-V8p5*X( 5' C[KL6(8l EP)]%"AG$3@ 6[]Wb6ؙiኩFY\fFZ]x}RGx 73BS vpo $+)`-AƂH@ Z@3XS\*J D y O @> 0 tː&V#,% Bzk6e"K5i܇_"BM-O&)`0a'{9Ab!9p$6! ʱ{dd,<ޑ! j cy  !+PA"fw"X}9l;Z@j"$Jf'[bQ]i[")I |qgϭ^C5TD%]c^=%!J"j H$r`RH\;wj""A$$5q} ;XWH&Yu}(adB(mTtV~RbA %{8 S,lWMWi|$0#s>;~}dJ'>;61BNl;Z XOQ^B&}B!<%j@!ũ)knˁ(JWuXRgµVAce&B8"v /K9EpfK#F@ܠZH4dH2lIFkH2WzP !fްH@'!E\kq* %*83:H[`'T WM ݖN й@f1A,Be!%0 x@F"?h_=$#qx$,rܺ(Fwm=$AĐw@$9v9Num2!*0Q v(F?Ё6'+ok2&dU:xAe %#BaW3*.Ibo%/3C&9 nDB*c8I@wts*ۙǮrGFvU@)DO0;C@,E,HAe$Wp8[umTMECDE7Lf\#JLе%Z()l(qxZR[HKsI!0 DS>Rz!qȝ[GP !D+W3#ֱgr X %*Y[**zMyU0DdM¹:>X'jڒ6ZF !$Hm6Ky)+&C~>Aw`+ IHB$:G'qE"?I%hAj")B+ho{۶8˲8m"b? H6R7~c_9qQ%E>!9(C3B^.y"R?8?lׯ?<|ηo?77C.̿L|}֟O8ZI *T́ݺphKxZΥ].VL=^ǀ[F4X_ć PjDĄD˟^\}/_M8MfqUlχ]kdbBs@6m*ϼqHRArFz!W[hǽ'C3TY*ӮFUk%=$I wﯿY)$bx$~|4}Wv~N_y򟧓''E=% QQ$!hn uWAY1[cBO@R[4H#Cv@Ȝ g˟8<[=;ΓQ{S>mx}ͽ'V2n|R5V45;P;~L}Q ĴFɍmڿ c&LkvS=kl9"or 'SuBrbrth7_wzw0|xXZ!SL>.;V 3O[sIrӫ4R>w.-/@zG$5ۍj2#rPC1& BA[8K WGurnEm~uq֪PRVaX{urr*7\>:Wy ! L;))s&&d0Z\ l&&] bǻ/]ߟ]!.fh!cL& 1 كqH{L*di5Ɠ5Vil~ˍP1afo@E?=_NvvbȰJ@XĄ< l<|Ã5b!n Y* Կ1cҗ'A||E jvx rqx35]K3N^ɸeJ9[nsjr@'*[dE@i6"v٘^4bPPA jT8 k@Ȟ   Y* lCjwXzwJgבCD"EYiqGe6NPaeqdۺw?0k p'!q@_UcEq4H;CH`m\?Ӯ˼* vy7逬D*vn5Qej;fxjM:_eU-y>{-6՝ug܆ !=Vjr ƹ.laBAHBmИNc'nF ?j 9Z{uX_?sZH#aq8dk?3[{1^EP;uQ?zɖqlOF@@d)I@ +0Ļ @@ڱGI @A@< @P;JG vt @ {#@P;: @zԎR @ @j^) @Ԏ @^c~ @jG @@ڱWJ? @s @W@+ @9@ +vҏ @@ @P;JG vt @ {#@P;: @zԎR @ @j^) @Ԏ @^c~ @jG @@ڱWJ? @s @W@+ @9@ +vҏ @@ @P;JG vt @ {#@P;: @zԎR @ @j^) @Ԏ @^c~ @jG @@ڱWJ? @s @W@+ @9@ +vҏ @@ @P;JG vt @ {#@7MUWIENDB`ovs-2.9.0/Documentation/howto/sflow.rst000066400000000000000000000142761324262074100202010ustar00rootroot00000000000000.. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. Convention for heading levels in Open vSwitch documentation: ======= Heading 0 (reserved for the title in a document) ------- Heading 1 ~~~~~~~ Heading 2 +++++++ Heading 3 ''''''' Heading 4 Avoid deeper levels because they do not render well. ================================ Monitoring VM Trafic Using sFlow ================================ This document describes how to use Open vSwitch is to monitor traffic sent between two VMs on the same host using an sFlow collector. VLANs. .. image:: sflow.png :align: center Setup ----- This guide assumes the environment is configured as described below. Two Physical Networks ~~~~~~~~~~~~~~~~~~~~~ - Data Network Ethernet network for VM data traffic. For experimentation, this physical network is optional. You can instead connect all VMs to a bridge that is not connected to a physical interface. - Management Network This network must exist, as it is used to send sFlow data from the agent to the remote collector. Two Physical Hosts ~~~~~~~~~~~~~~~~~~ The environment assumes the use of two hosts: `host1` and `hostMon`. `host` is a hypervisor that run Open vSwitch and has two NICs: - eth0 is connected to the Data Network. No IP address can be assigned on eth0 because it is part of an OVS bridge. - eth1 is connected to the Management Network. eth1 has an IP address for management traffic, including sFlow. `hostMon` can be any computer that can run the sFlow collector. For this cookbook entry, we use `sFlowTrend `__, a free sFlow collector that is a simple cross-platform Java download. Other sFlow collectors should work equally well. `hostMon` has a single NIC, `eth0`, that is connected to the Management Network. `eth0` has an IP adress that can reach `eth1` on `host1`. Two Virtual Machines ~~~~~~~~~~~~~~~~~~~~ This guide uses two virtual machines - `vm1` and `vm2`- running on `host1`. .. note:: For Xen/XenServer, VM interfaces appears as Linux devices with names like ``vif1.0``. Other Linux systems may present these interfaces as ``vnet0``, ``vnet1``, etc. Configuration Steps ------------------- On `host1`, define the following configuration values in your shell environment:: COLLECTOR_IP=10.0.0.1 COLLECTOR_PORT=6343 AGENT_IP=eth1 HEADER_BYTES=128 SAMPLING_N=64 POLLING_SECS=10 Port 6343 (``COLLECTOR_PORT``) is the default port number for sFlowTrend. If you are using an sFlow collector other than sFlowTrend, set this value to the appropriate port for your particular collector. Set your own IP address for the collector in the place of 10.0.0.1 (``COLLECTOR_IP``). Setting the ``AGENT_IP`` value to eth1 indicates that the sFlow agent should send traffic from `eth1`'s IP address. The other values indicate settings regarding the frequency and type of packet sampling that sFlow should perform. Still on `host1`, run the following command to create an sFlow configuration and attach it to bridge br0:: $ ovs-vsctl -- --id=@sflow create sflow agent=${AGENT_IP} \ target="\"${COLLECTOR_IP}:${COLLECTOR_PORT}\"" header=${HEADER_BYTES} \ sampling=${SAMPLING_N} polling=${POLLING_SECS} \ -- set bridge br0 sflow=@sflow Make note of the UUID that is returned by this command; this value is necessary to remove the sFlow configuration. On `hostMon`, go to the `sFlowTrend `__ and click "Install" in the upper right-hand corner. If you have Java installed, this will download and start the sFlowTrend application. Once sFlowTrend is running, the light in the lower right-hand corner of the sFlowTrend application should blink green to indicate that the collector is receiving traffic. The sFlow configuration is now complete, and sFlowTrend on `hostMon` should be receiving sFlow data from OVS on `host1`. To configure sFlow on additional bridges, just replace ``br0`` in the above command with a different bridge name. To remove sFlow configuration from a bridge (in this case, ``br0``), run this command, where "sFlow UUID" is the UUID returned by the command used to set the sFlow configuration initially:: $ ovs-vsctl remove bridge br0 sflow To see all current sets of sFlow configuration parameters, run:: $ ovs-vsctl list sflow Troubleshooting --------------- If sFlow data isn't being collected and displayed by sFlowTrend, check the following items: - Make sure the VMs are sending/receiving network traffic over bridge br0, preferably to multiple other hosts and using a variety of protocols. - To confirm that the agent is sending traffic, check that running the following command shows that the agent on the physical server is sending traffic to the collector IP address (change the port below to match the port your collector is using):: $ tcpdump -ni eth1 udp port 6343 If no traffic is being sent, there is a problem with the configuration of OVS. If traffic is being sent but nothing is visible in the sFlowTrend user interface, this may indicate a configuration problem with the collector. Check to make sure the host running the collector (`hostMon`) does not have a firewall that would prevent UDP port 6343 from reaching the collector. Credit ------ This document is heavily based on content from Neil McKee at InMon: - `https://mail.openvswitch.org/pipermail/ovs-dev/2010-July/165245.html `__ - `http://blog.sflow.com/2010/01/open-vswitch.html `__ (note: the configuration syntax is out of date, but the high-level descriptions are correct) ovs-2.9.0/Documentation/howto/ssl.rst000066400000000000000000000327101324262074100176410ustar00rootroot00000000000000.. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. Convention for heading levels in Open vSwitch documentation: ======= Heading 0 (reserved for the title in a document) ------- Heading 1 ~~~~~~~ Heading 2 +++++++ Heading 3 ''''''' Heading 4 Avoid deeper levels because they do not render well. ===================== Open vSwitch with SSL ===================== If you plan to configure Open vSwitch to connect across the network to an OpenFlow controller, then we recommend that you build Open vSwitch with OpenSSL. SSL support ensures integrity and confidentiality of the OpenFlow connections, increasing network security. This document describes how to configure an Open vSwitch to connect to an OpenFlow controller over SSL. Refer to :doc:`/intro/install/general`. for instructions on building Open vSwitch with SSL support. Open vSwitch uses TLS version 1.0 or later (TLSv1), as specified by RFC 2246, which is very similar to SSL version 3.0. TLSv1 was released in January 1999, so all current software and hardware should implement it. This document assumes basic familiarity with public-key cryptography and public-key infrastructure. SSL Concepts for OpenFlow ------------------------- This section is an introduction to the public-key infrastructure architectures that Open vSwitch supports for SSL authentication. To connect over SSL, every Open vSwitch must have a unique private/public key pair and a certificate that signs that public key. Typically, the Open vSwitch generates its own public/private key pair. There are two common ways to obtain a certificate for a switch: * Self-signed certificates: The Open vSwitch signs its certificate with its own private key. In this case, each switch must be individually approved by the OpenFlow controller(s), since there is no central authority. This is the only switch PKI model currently supported by NOX (http://noxrepo.org). * Switch certificate authority: A certificate authority (the "switch CA") signs each Open vSwitch's public key. The OpenFlow controllers then check that any connecting switches' certificates are signed by that certificate authority. This is the only switch PKI model supported by the simple OpenFlow controller included with Open vSwitch. Each Open vSwitch must also have a copy of the CA certificate for the certificate authority that signs OpenFlow controllers' keys (the "controller CA" certificate). Typically, the same controller CA certificate is installed on all of the switches within a given administrative unit. There are two common ways for a switch to obtain the controller CA certificate: * Manually copy the certificate to the switch through some secure means, e.g. using a USB flash drive, or over the network with "scp", or even FTP or HTTP followed by manual verification. * Open vSwitch "bootstrap" mode, in which Open vSwitch accepts and saves the controller CA certificate that it obtains from the OpenFlow controller on its first connection. Thereafter the switch will only connect to controllers signed by the same CA certificate. Establishing a Public Key Infrastructure ---------------------------------------- Open vSwitch can make use of your existing public key infrastructure. If you already have a PKI, you may skip forward to the next section. Otherwise, if you do not have a PKI, the ovs-pki script included with Open vSwitch can help. To create an initial PKI structure, invoke it as: :: $ ovs-pki init This will create and populate a new PKI directory. The default location for the PKI directory depends on how the Open vSwitch tree was configured (to see the configured default, look for the ``--dir`` option description in the output of ``ovs-pki --help``). The pki directory contains two important subdirectories. The `controllerca` subdirectory contains controller CA files, including the following: `cacert.pem` Root certificate for the controller certificate authority. Each Open vSwitch must have a copy of this file to allow it to authenticate valid controllers. `private/cakey.pem` Private signing key for the controller certificate authority. This file must be kept secret. There is no need for switches or controllers to have a copy of it. The `switchca` subdirectory contains switch CA files, analogous to those in the `controllerca` subdirectory: `cacert.pem` Root certificate for the switch certificate authority. The OpenFlow controller must have this file to enable it to authenticate valid switches. `private/cakey.pem` Private signing key for the switch certificate authority. This file must be kept secret. There is no need for switches or controllers to have a copy of it. After you create the initial structure, you can create keys and certificates for switches and controllers with ovs-pki. Refer to the ovs-pki(8) manage for complete details. A few examples of its use follow: Controller Key Generation ~~~~~~~~~~~~~~~~~~~~~~~~~ To create a controller private key and certificate in files named ctl-privkey.pem and ctl-cert.pem, run the following on the machine that contains the PKI structure: :: $ ovs-pki req+sign ctl controller ctl-privkey.pem and ctl-cert.pem would need to be copied to the controller for its use at runtime. If, for testing purposes, you were to use ovs-testcontroller, the simple OpenFlow controller included with Open vSwitch, then the --private-key and --certificate options, respectively, would point to these files. It is very important to make sure that no stray copies of ctl-privkey.pem are created, because they could be used to impersonate the controller. Switch Key Generation with Self-Signed Certificates ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ If you are using self-signed certificates (see "SSL Concepts for OpenFlow"), this is one way to create an acceptable certificate for your controller to approve. 1. Run the following command on the Open vSwitch itself:: $ ovs-pki self-sign sc .. note:: This command does not require a copy of any of the PKI files generated by ``ovs-pki init``, and you should not copy them to the switch because some of them have contents that must remain secret for security.) The ``ovs-pki self-sign`` command has the following output: sc-privkey.pem the switch private key file. For security, the contents of this file must remain secret. There is ordinarily no need to copy this file off the Open vSwitch. sc-cert.pem the switch certificate, signed by the switch's own private key. Its contents are not a secret. 2. Optionally, copy `controllerca/cacert.pem` from the machine that has the OpenFlow PKI structure and verify that it is correct. (Otherwise, you will have to use CA certificate bootstrapping when you configure Open vSwitch in the next step.) 3. Configure Open vSwitch to use the keys and certificates (see "Configuring SSL Support", below). Switch Key Generation with a Switch PKI (Easy Method) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ If you are using a switch PKI (see "SSL Concepts for OpenFlow", above), this method of switch key generation is a little easier than the alternate method described below, but it is also a little less secure because it requires copying a sensitive private key from file from the machine hosting the PKI to the switch. 1. Run the following on the machine that contains the PKI structure:: $ ovs-pki req+sign sc switch This command has the following output: sc-privkey.pem the switch private key file. For security, the contents of this file must remain secret. sc-cert.pem the switch certificate. Its contents are not a secret. 2. Copy sc-privkey.pem and sc-cert.pem, plus controllerca/cacert.pem, to the Open vSwitch. 3. Delete the copies of sc-privkey.pem and sc-cert.pem on the PKI machine and any other copies that may have been made in transit. It is very important to make sure that there are no stray copies of sc-privkey.pem, because they could be used to impersonate the switch. .. warning:: Don't delete controllerca/cacert.pem! It is not security-sensitive and you will need it to configure additional switches. 4. Configure Open vSwitch to use the keys and certificates (see "Configuring SSL Support", below). Switch Key Generation with a Switch PKI (More Secure) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ If you are using a switch PKI (see "SSL Concepts for OpenFlow", above), then, compared to the previous method, the method described here takes a little more work, but it does not involve copying the private key from one machine to another, so it may also be a little more secure. 1. Run the following command on the Open vSwitch itself:: $ ovs-pki req sc .. note:: This command does not require a copy of any of the PKI files generated by "ovs-pki init", and you should not copy them to the switch because some of them have contents that must remain secret for security. The "ovs-pki req" command has the following output: sc-privkey.pem the switch private key file. For security, the contents of this file must remain secret. There is ordinarily no need to copy this file off the Open vSwitch. sc-req.pem the switch "certificate request", which is essentially the switch's public key. Its contents are not a secret. a fingerprint this is output on stdout. 2. Write the fingerprint down on a slip of paper and copy `sc-req.pem` to the machine that contains the PKI structure. 3. On the machine that contains the PKI structure, run:: $ ovs-pki sign sc switch This command will output a fingerprint to stdout and request that you verify it. Check that it is the same as the fingerprint that you wrote down on the slip of paper before you answer "yes". ``ovs-pki sign`` creates a file named `sc-cert.pem`, which is the switch certificate. Its contents are not a secret. 4. Copy the generated `sc-cert.pem`, plus `controllerca/cacert.pem` from the PKI structure, to the Open vSwitch, and verify that they were copied correctly. You may delete `sc-cert.pem` from the machine that hosts the PKI structure now, although it is not important that you do so. .. warning:: Don't delete `controllerca/cacert.pem`! It is not security-sensitive and you will need it to configure additional switches. 5. Configure Open vSwitch to use the keys and certificates (see "Configuring SSL Support", below). Configuring SSL Support ----------------------- SSL configuration requires three additional configuration files. The first two of these are unique to each Open vSwitch. If you used the instructions above to build your PKI, then these files will be named `sc-privkey.pem` and `sc-cert.pem`, respectively: - A private key file, which contains the private half of an RSA or DSA key. This file can be generated on the Open vSwitch itself, for the greatest security, or it can be generated elsewhere and copied to the Open vSwitch. The contents of the private key file are secret and must not be exposed. - A certificate file, which certifies that the private key is that of a trustworthy Open vSwitch. This file has to be generated on a machine that has the private key for the switch certification authority, which should not be an Open vSwitch; ideally, it should be a machine that is not networked at all. The certificate file itself is not a secret. The third configuration file is typically the same across all the switches in a given administrative unit. If you used the instructions above to build your PKI, then this file will be named `cacert.pem`: - The root certificate for the controller certificate authority. The Open vSwitch verifies it that is authorized to connect to an OpenFlow controller by verifying a signature against this CA certificate. Once you have these files, configure ovs-vswitchd to use them using the ``ovs-vsctl set-ssl`` command, e.g.:: $ ovs-vsctl set-ssl /etc/openvswitch/sc-privkey.pem \ /etc/openvswitch/sc-cert.pem /etc/openvswitch/cacert.pem Substitute the correct file names, of course, if they differ from the ones used above. You should use absolute file names (ones that begin with ``/``), because ovs-vswitchd's current directory is unrelated to the one from which you run ovs-vsctl. If you are using self-signed certificates (see "SSL Concepts for OpenFlow") and you did not copy controllerca/cacert.pem from the PKI machine to the Open vSwitch, then add the ``--bootstrap`` option, e.g.:: $ ovs-vsctl -- --bootstrap set-ssl /etc/openvswitch/sc-privkey.pem \ /etc/openvswitch/sc-cert.pem /etc/openvswitch/cacert.pem After you have added all of these configuration keys, you may specify ``ssl:`` connection methods elsewhere in the configuration database. ``tcp:`` connection methods are still allowed even after SSL has been configured, so for security you should use only ``ssl:`` connections. Reporting Bugs -------------- Report problems to bugs@openvswitch.org. ovs-2.9.0/Documentation/howto/tunneling.png000066400000000000000000002150051324262074100210170ustar00rootroot00000000000000PNG  IHDR  0iCCPICCProfilexwTSϽ7PkhRH H.*1 J"6DTpDQ2(C"QDqpId߼y͛~kg}ֺLX Xňg` lpBF|،l *?Y"1P\8=W%Oɘ4M0J"Y2Vs,[|e92<se'9`2&ctI@o|N6(.sSdl-c(2-yH_/XZ.$&\SM07#1ؙYrfYym";8980m-m(]v^DW~ emi]P`/u}q|^R,g+\Kk)/C_|Rax8t1C^7nfzDp 柇u$/ED˦L L[B@ٹЖX!@~(* {d+} G͋љς}WL$cGD2QZ4 E@@A(q`1D `'u46ptc48.`R0) @Rt CXCP%CBH@Rf[(t CQhz#0 Zl`O828.p|O×X ?:0FBx$ !i@ڐH[EE1PL ⢖V6QP>U(j MFkt,:.FW8c1L&ӎ9ƌaX: rbl1 {{{;}#tp8_\8"Ey.,X%%Gщ1-9ҀKl.oo/O$&'=JvMޞxǥ{=Vs\x ‰N柜>ucKz=s/ol|ϝ?y ^d]ps~:;/;]7|WpQoH!ɻVsnYs}ҽ~4] =>=:`;cܱ'?e~!ańD#G&}'/?^xI֓?+\wx20;5\ӯ_etWf^Qs-mw3+?~O~ iTXtXML:com.adobe.xmp 5 2 1 :w pHYsgRQIDATx[TU_.d2 ,Chda].CCIA)$KЅ,nq!* =EEfVZgqiì3{{Gk,G[fY[t8)9ik2rЀtb%3u7[h>B],Yoo_1V7gd#tɖ-suy~ϯ,R }.Vw.r_?tb̭k,G\+;r}/~Q`Xo4/"3,wZwX/Z[ndmcyv|}{S٣' :0H-Ř{YY/S{X]+w[-YNQ|pGv FRX˶lV 0|,ˍAm~SEԾڜXc+o DAs;>lWU}E΋ٶ2r'{nxJa{չ ŝ=o(?b@M)_e y)'+zכ-ck]UW3Z=c(`cnM __p+uZվxŅ5'=fJ@c90e$'mCRU-|oy=c(`cUs UUu|zײ%UܔdЯ>'ݳpCq~^ ߑOQ{9ߡ+߿UVT ^ |U}-u~(]^.}{Ll_Zk;;_w{"x-AGFpb1ndh44 j2L~fAggؽ΅{|Lٝy۞>[\〞 ugQkl䜼|1[ _]}AJpO(Q/k-Z0}h?zn;m oS ؟\3KS^KmMd: 5"!)K jMCGjzA+:t}N   -z A A   -:9,:AZ0trX0t``i!a!҂CÂCՖ.?wߐo$w2%`tُU 1gC_1kHՐ\JUzij f?TP4~ڬxZRWҳEv ^f3ijХ3>wd?>Y=PU3tYS:n"0xtWLqJNɣ>Hva:\w+.M͗ 2^W&^GgN"ύ{[X ޜx= n$7f:Iv."cY9'zz҈;|9_0/FgYּ˗> n_ެu ut1lݓf?x1tm,Wo F_:m;>s-{y#/.7 }QUeЗ?}z:2baKb#5r ,k:/?^ϯrτ7!okuN0y?ow>[_=ݱ2 =SCyʇ D:$R#/Չ=M`ݤ䗒_?hmǮ޸o2t[^t?XI!C?b={ KR IZQK7?Jvܥ}5Gv[ }78?.?ǂyi?o|\?z4=몍C CqsC18ϛ2F#z|+ʑpulwŠos cѼ5-St*N+b浺6q/?7As3Q eIOP0I5V"C5nUCOt =&~ ͚.[)zCr>֦WǼCX1tg&g+zma=[wciu?I[4BzM7F t㗴2#g1%\:WS`ڼmCĩ&.c.ב2Е+7q[y-JmLG*[ceٳ9kZ/6Ri#rO/9@_!rO/9@wMqi%^r0rMq\S\zKҐr|00~=-@,Zظn7-IzQ|/3[Pa(6S4oNI.z*O$kՔF7A-ϸr~c Sy*'v`9g̡\C=-@mA" 5ۆE^]-Ɯ2].Ԯ='3UWߛQo75 IR>)Lam̯c_@W2u^Mђs YΧLEͳݏhlM`_}Uos[=b*'u-,pSҼ޸b2#%ˡ0\PR su݂;@O Cۦr%d16&yP\YF%$ˋWmBw>O|*Tw :m6U\+S/Ss@/_BȥKF.pȥKУ^gUI}ot쇦P#ty3 @~,zc<zm<\n Nx8BkMvDl."B݌]]f=[<=-"􅑋=߶xg6 }~EڏM$ FF &&t>#1q i*sScFُՖvF=gu~F9,c LGCLt3BVubY,-Hu܂@He܂8M9a^Pb,2&<-Pz__c\KEvΆw.)z(VWU<{eþkLr6)) n ^#o첂02IPR]k-B7zbzeo(Z<rt;su/ה _7ȔXE,KcW'xOO=P7FԌE|o}ݜ|w)6nkB_Ów024E|5m{Xd؝X&؁U2y8c>8u_ֳ\B^?4){BEOggb"l;JCcɱ 3;e_>y38ΔUWVl?uQ+L1t>&Rσ#N]q=qrZ[yЏD_CŽE9O%((3wc=l0m -b@7$WIn24\9>kՋ;ģ?&?.St'zw8K#v%3"kGu9mw#s;R0 o6w_6^o8,~ah:"a#؏VvA=}5daaw%=vEb.>hY;Enٞh?96MUm >!^\N薔{PC_n[I%pWSsslJ}8)۪| #x!=Xg iN8 P蟭.Cl8\p54Rl]Z)2Gv9tu3o_0K~Ae렸6tS2ZF}b@{}~DNk1E3o=sH_8"t߹As0g*Pʐ7lHp pu^k/б^Z^;NNV&CeKWEȡ"` `#AAZ.(!ZO:Gc^t; F#F7e@Q2pHN #WpSIz\bcerf|@70MK[Vހ0 З} {ϵQJߨXZ=7E|;[ ௦tb,?TѼd2udk+#c5 Yc_!$dU7fv`>;|ui+Zn;.=L R2Z|BtmԊ_. $ >0H !*DZ8qǯ_'^ZR%]=[y j ] F ۘTT@yM\,&7 c{"J55%P ۺ:Tnj]n08$>M,JEkxcͩP׿(-:4#.: }`t#_)8F>5tzK'PND {#e]NeԌrRlVCbnSZJ:qgH@KY7y$FZ`I߁::A j5,0϶񸕟Y="E_[{JɳbO|2˭i*yhl;{j^Et f=bG.f)wY˼NWUqg|QeS_ZD&)]tiCyKdY)ٻEk6+á {f /)3t-EIGYV ]8l}CJ1[3,kks@mPH4ePWrm止t֚,@pIbL# E $ Ցn=WY7Tb7Qg?00 u!el eD} П a;Z;80X;a:b=O).dz9{]>yL,o(@PO7&g-dysKԈ v! 3)w{0Ǩ%*i:xffEuCF3bv>N`]* #MAߥ#)6 cє̠!Fٚ^~ukUEw:{9YC&@G&7\`&VYf<@΍p䉃_8Ny .: wa шX1|G @ovq}iS8hta6^jiX]};((e)ٶ%Jl|nיq*Z񐱿SĈpW]䌢v&ř6 h)Ȥ@ nǠ? y -̧aM;fw\fa;`q-ٕ?m`@C6NP9@OQ=l[ = >![ۆ\> / NXK}Htϛ!)~c4ger ;ÙAD-kXIxm ?(Mq!iQ>ȍFx7{>sm@g(>ȠmK+z R>epS|;>)~ \#ylM3yGG5G鏳52-=zdkz]x{r?h#gK w~OWл) z5][ k]O\SkEEXJkB"$.w>$Q4+ ba؈ + + ""2ӰJ e;9ܙfλ|'3ߛ7ssY_2Gܶ w$q+ <]?6A -p?R~*K"tΐ:Kڥإ!sn:ڟ @2ћwq |0\ܠq)nBJRCR:e28G8XLཧF`J̭+ q'SNd =$C;CؙI;~>&&z(30܇~]:=[Ims16 #BiJC  )93pjV yat `*E!aݹp-7S\ '`M!ЦP|nqpܢOt+/<ӕMqD]]}2BOЫ!"NKX¯h::?zBmwLi< cLjßC!ؒ cԿQ,B g'Ïu't{!R wg3f ?k4 ]}ʒ cQ e?l@m} .㱣R1x[I_[exbsۃ#=RUf谭::>qds/Sb`}e<-rzIyߧ}Ȋ#YL&-לЃaGc*9F a /;c!z!i:r0BE')gPQBA%߯H:w2{e71 ]}ʒ2 7/mz(~Vۏ-{@/!q} 7ÖB-: Ё'!\K 75t8hw}Ysw(E58< G0?Eީ݌jxgc ɕb*:c]>`)'ïo[י;NKpxq(x]$B>9|{ mB{KmoԞWr֛йF/6LۢX|{]kBV*&k{wjB7]CϾTKBr&L4zBoQCB.nZB_o\Rg_zcM苵!77c79BSc7&`cc٥hg-5"}B7xv }Ӷ {n=ۻTo'/rwҖsjClWo7?,>] p5wæjCv\S{ܯ\:~ݲ[oͯ1}za]8MSVkW2ϿzBD }ZodTA+[v?ƹ?ZwS;apOY[Y?/kHY jqCF 'k=WٌqQtUϏ\V NT7)ZWv8vQe%_Qyw޶q,)Rzk'Wk\X1aebH\7_S*3gTqީ, z5{<g߬,gEP_ߪt`e3}_qޔfJcߴ^,>okޮɀ6j !eԷvi*u,z.a7۪%ٲ"wْ ƽ`x`B&'ciPBxt> 6 N(13ʒͮvwu;gΜ9gܙn -Ve9|*VoCunvG5vpHa$_U}K,wF1`vtXzOX#$?7z{XA*X[#V0cl?.]% M\B>lpr;gw|&]L:L\-+p؀I? t3Qr#A14k39`%ꑁ3LSot=s>KED $\{,9umt VOW[.ڦ~s26'HFR8se@&Vp::hԋc#bmHs9\,[XT9eep@N{j X2׉U]|J!q9qfl1fsrr?tYT(0/qXOOٹmCgY%:lИĚ}up@^b@]5X:4AW,_0;8oX@w<U>"9sŵdـpSTP3@mFLS}U\NNp@^b]X|};*(Pd>Aq;VviO{t6IUFZ.Dʍh@;;K 쉇=xI}8.+P@ֻ^z7T~ ;?zwf/J[zFlbp(zK>s+g3w:JD{u@+S{W(@:kJ8wiǩkQt}f@DYuԕg^M)"؛|79/m;}D\ h,mtR;Z-2!v4Ou,O^F:o[K%;*cgaKg]9#^NEGww©zkRw%dt mhbb)--L㵞={ѫW/ݻw3zъ'..Ρ#|rssa}90rO]GP\߈#wSBBJFF{H^^HvvF62 1޽;_;јh0n%x)))9qӘƨ;MNoy DH4Rذ9.>H@ lذg̜93U8cƌ'Oq 'SL):ujYZ9666BiӦ TL>=#--iz5@` T̞2O!:={l<# loĉIV4`2~O!p~رcK gO<-|HNH< 82d jV,B xpp#@=q%y_  ;WP;q3Oq$߇#p!Q&׺uEy7lߏQg߭7o'_شiv9qooM_l&rG |=xA;I4Xu!dqH[l9.6Ge]>kp-$͆/gEF^X( u\|KEjO{SM,c|xW7h2&()z/瑶|>">2y|"CQ q̓my[2Ӛ<<04g2E@ʳ ߛ2/2{~>'x'ڒ.:Z|o= }n9/ApϟڊCZ+|_A7ų-z|t!5):qeCr@}v?DP`IY|:^P6o">*- 0:2.q.t0x ϟ3Ƿ jŶ!Cϣ]j WRZև(/^^Tx?I'݄iG~kߢSϡ Gߣp 6l?v;)~G^e/d5H8:ģ~p ǃ"\ߋt{Q ~%o0j\;px~ z:ѣGYBІ4nnMT\xjP 6 xPt3:|WW"sxj|/@̸<zZn@cp Yq&p::f9jiD_nљ;2n*l&P`.n9_uw:ڊ|tn/Be} #} xG])-= q=ݍ8w@ip/`Xq7/GnF`87*ú2ӸY/իכCjp5C6#У,30K` b"yxY'si΁KviT8@P͖ g1i<q: ߁ƏKL\zx6Р7l6U%kNsIHHH/<{2a*^l 1~YK.b AD<\_Brr򢊊 ʶyEd xN‘iµJf&"piωzc=\c%h3 H c C#Sf .QxfDDzvQ$֤jCa9J|MQ7FKNYYl||MZ2=-7+PiJ*SʮsDX(_w%_ )_*So[>U|JgndQ}XCV` (_!_ujuα]'|G>1GWiȦWdOoO+kJ'IJ2W'ﻐ'0;tNéժT`JSVR5? 3*Y0*OݿȋƔt  A˗CP:I[+-c&\*+6ӨOc#+x Д .U e4.~OE*i||J'lXݔ~5F%{!uw+QeP U$غw-2dtA J^} V'nOLJL4U&o%gW/?&14.`F*InM](> lnڍoz#?/K:Kx-I3{H49ijQ305͡P~NKˤN#^ɠds:4mboǛ|~q^yK5cT~&7ےzc9.7R6~|eҺzh//;7=8G$4tYyvcF'{oFіƊky3@mXtWT8"I1xXde7C|нQ(rh-%;ghX:ZNv~^`qoayـCu:?@Z?ǣxڂ _{<(?N>+Oox2q<&_[<ڞeKgz{grcsWaCSK\zb~mѱz{]w4=N[b[  o,5Hw9e:Y?{mt2u@uto,!^ ݎ 79zW ]9#юħq3aW '#) |΀Nw`E#f@glѯBD*pXzޡ5L%6Zô'IGI/ 5 &v+Fb. VNH;q^_=!V'+(}HfsYWph`G*oWtVk<z=8I%gs ;ߡF5n =F~c#G~lV.Qu@5I(VYxl p\6'Pd'=QNKnB=8 +VY߮V_HzWם\X8:XGD9kfiD,7qNZ7@#-FL?&֓{]ѮFgSvnd]>inN}C:SPF򫗬=b ?t^pܾRbM` F Qt`Ĺ ىt։m)(Ѓ0jqXͯɅIwكRD:U>$*\V~Nj><[OlSjZ﷈51)T;Q72[bMӯ;2Ax~XUzPW@씑-3ps*\m::#ǜ h7EUD1.:mHFw'P8 #:>\+'q4o3pqu!u\/- gu>W@'u+  k'dٸwqä́š?C^o$+ǵ< OzP3!#Ĵ"ÔYYP=6h,?"TӊԿZg^C2[}?RUu)~l=N=*G&gK򈂚qE|j<(?@fw;,6"_~|~U |=ɬs-~lOvY9{ov=xO]i+msک4a}!G |ѰCs{K< ^z7 icS8@mXsOvЌh_?8)>'nod _;qsFͻh@^_+.|0#){gЇ.8 ]ٽj\eA6G^-UU6ɅƼn S|mrWfq[>gU䕸guM?GVSi8@8{ː<֥ڥn:T_{^1CH w#߄HgI[\s~-i@Ӏ]TW =k1E뺱AbXcﱷZb[{Qc챬kܽ{;>033˝yYӛ~eM46yiHԋu;jҸyMf񸌿u]y P#Ȗ7>"PN,eQvDnUD?ytEǬП2r-k՘N&-{PE$|-Ҙ&-=Qr agmęЫ5+m6t>>cn+oO4L1qa-h;׀w=[SjzEꪅye ?c'2%Л\zֽR/qK+0j5Er #iB;&qzuU81[ U\L{ЉEB#\[Y"t`'tg-BW UikE"tgggNNNپ_3]/_>E4-oeyU?Ui$tNe^53۷]my...omaAp~"EL̻wN/^L7nH'LYݧYJ!ONNs̡AAAڧE4CzBmzA-[Fׯ_OǎnLduf!{>zGғ&M˧MFϞ=׹g-3|{:7_u]Fj߾^z|'|B {?>|/o߾4**{.]rgϞMm ۤח)S" GϹ~BϐI*QB?ϟ?OGerʕt˖-^||HӧO>2辒s4 ׮]:ux<\7lAB|ͣ;w'˗pfЕ5C:={pFSn{UTǏDh߁ 'iz|;w73R!Xhs4 px@0~"iӦMycǎhl߾#zj":,B1͐,bÆ \?A:K;oŏ?HO>mÇyo ޽{ܞ@ݺuoF^K BJ;vA򂽅`|ЃrC'7}gtttæX5[rfroܸ1'qe:t(`62d3 j8%8%%l G }̘1E%P̆kpVa6BS".\ā#&&!%M1`QDzJ%J_Ta*Ͱ݄>Ey "Q~ {:u}8G%K}(=?3:95^bޡ~,"t DC8N?k,1{^鋽,B7c OQD ۿ?O#:Rj &@.bxB堈/@Q q/Ű`[@+@.uƤDxb(PGx J.XXo^#FLxaRxs7!/يgJ1C/\e4s e `?)9OUi%t B3ӡ{ Axf, "ClKzxޣ$dB7nb܊wL 6rH>}dx#ƽ[lɗ U{`Ad[Sr3~7/Rn{ixuJ=Y,"yջ<1q2J^i*͐~`6پ`2{)- p^d:5t 7"t`GДLx'"tU! ] dzLS?%]oBA1* a$zLMYk]oO BOZph^N@Bo^!1jSJ?zDcViОƍZy1GzA ͆.`_z1kE=٧©{`j m>bt"tAާQ WHR-=xqpB4{|*Z_AP-]5\7{dLC ,uuk:јEUj1z0&܈,R"狱ǍZq!^&Y{7,EL R/aؤ@\regXr6ߌ;'>ywKLX1H^~E?cNN$ՓL-BW *QBm8HV1=h:dލh4`IK/&=O6.7:ޟ<ʖHop e>ǎVBF~G͓ik2ڃ%=hDNN8W:~ eGaVkxT Ž{RO֟T?[גSsa£RPvRNy\5q 'oFxU5HdURKgӒ~E=-BW |zxWӛC?ǘ-ȐȐ= +Y6:dǑ^5~u1'0=ehxfcӿ.[ZS{X&0i7JEXN!tb. et A#6^a`Hahڮ͇]df<31Texv~San4ٮfA%Ĺvv[HKfz6sv16`PC=^Fq Wjك L? cT%,6v.dʒ4yjD I 4&T`psvq-@ЉʥZ'`3L`/9=HEc v^7#i.oa GIS k(J|2Y1؎=@`:a73SxB? Sp\S$s{L$=aAzFhq؎pP 2LrQV b6}*(J|pm|,B~@]}D[3TB?sN-o>$Mg!"!dەHNnnP&3"֮zW0!RΝH=' D&"0|z$)]"1!,B7^$k-ARõE FiCJCDv$Ę֎o7FAQAwQU<ԨQDW"F41=kϬc5Y̚3ךs9u=VQE^Uŭ{O߷ϳFJ.ĽXW$pT4pUL$DC0{ kp 1"4RxA8:z{|=BxpS8Ďu4!#H>sfF[.^i`7I/~yWees繄G롰A'a3Νw^'Q-Z"io #u''i=BQ'1i ;neC#xC?W"s;ƒ#)t{ ІBuB<# X\2!;纸ԛd a`>1md:Hs=8sh؃S<$eoRA.vAG6PtBLaP8Ch0'΃1p=M?x (:H=S qqᎮ  ߞ@SIۢz`J,Ix  o',?d8G[P7  :cCĹ $):K)HpK ă [܉N'j VoW< ЇAK ܮCG"91669ЉcH hh#xl§(ByhMЃW0Xŝ18Մx1\o<  _x6"N~&#H Z +A0N#8q#DtBnNE6=꽄_C v`O\:n)Jϼ uƋ+QGCr KOqlc,: ȷ~Ens{kǹDN~(v6t4!=!| [Rpt`.\tB H:@8^Hv,"+ypx- OE:Fڣ ?E`\$mI&BƑN!R@_oo=|W1n'u4hX{RqJM \OH!u܊#^qq C:)ΖAJ|X.o =%6!o5T'K8{ 0o?gxH?c5. }Bee+vF ?S<^0&(KaaޚIFtms|gq*!Ŕ͋*9te6e|E[Ys11yg K'ߗ_+ay+*rw+~?2l\􆃖mnl/yݤϲcOF/1xcr *iYXINQy? l#9iޖCk N.}bgLfmj*vvWge~UƹSGO^uB{r jՎӧq^d NB$ď&W=8Ǣ{Pbʖʛ:E5_R qM2:y*Z:TC;|P2׷C9 7w߀jafVVh(, Pz_ek춍Zx]06=Cϛ֥UGN\[Sʦ&!OŴ *$< Gցd9$D3}j*-gYaMr$Vg*TtBYS'3W`XvI3pJRigg8p+k>ЮudVJ[zrX c͆,~5Ԥ@]K}*ЖS4צg|]nwuJ寫N HMZp|x'\7f͒uJ#pү>kVƎ=G \X0Oխ7h@b!Ї]IOd"D-a:{Ū _I%_Ҳ8;bKhB< z`_w&ɕaZ|>.W\5X4}5lY՚ mW2ҹ}ԉwM7.XB^z=r+R/`e=CbEڐupF&KmQee 1u]:{(oZY0< -^dRZVazFq꽕3a~5{ےw$Ǵf+KC-9S#~m%޶,8jgۅٔ$ݞ/**.^Jpu};Hk&\2z3Ė4M G;uakP<ۤ'gM]tA:%&тm0k܆4XIal1#WS_Mv0r^Y]\~dgU[vaI't$l[ɭ3e}]v\A9}&_*,~FMeWJl#uJvE; b حKx#NϊU~o8;yL`WYr622gJpVϢ>ո+T~".dx [nö=A:y;"zrj7Y"4XpF̉]MzS6cPV,}9װ{]$YŞZG,HZwݠ3%p+ J)4P9hGjն&__NTѹK }x ~\G4 iة[?(ɉю)U C$kWԱ88Z 9^}g9 :rd=\7x`yA<I1pфL} |Ξʜ:`0ZNtSa}VMl| AXZӗnƭx %w-6]y.kH[=imrGi6ݑN4y>jRlK/yV.ӎe!\Ihꤟu{nt3ޣ<{@C>S3 3-[p# {*l|T=$y7Bl\o'GeX!94Aa$. iɢipl t0unTSGCGER{7ɇn^0̫ N;)<|I8:s# (=`T~e%Jv*Ї3lʽ'9ܮlD+a&8Mi`H_Oă5ԧZ/aV;`{)Sx FZr#zHq݌a&LwSgi@^ooB`g^4ksjAP:;"qq‘#R2 ]+vD{}ZS. )N`s$g(ipV k 4PiPAWW_&Nm=@szcҁ)wD{8xOAtޒGm4ڶ90zI-H-79y_2AǕxخ}"X./MR<1go+yS5w֋l kC.A "|=گf)"[7.Mp4Ɂz=JS!YOf⦻Y}1p S+2. #ݯ5>gay!?)G<@һX[_ 2RAŌ>I/<eCv ˯+5`Z2LwDb8; hF"_tBJ0 ;V'Y2N9iQla|KG»v5 Zp/j*/UPY-4H۷fJtFikE ]uĎōGF8t: Pr,A8t |:[<O蹄;N[>zf8tۿZ.qh8+Ն\)%o,1=}C9u-{#ɮyCWη,W\h׿0ύi!vvqvWkYԷS'е|\֛m.og?U];|^e$t-Ύ:W; \; IiT=t<6A#)t6;D>t}EZāU=t.D BWE2ôbr^#s^%;RN{T -3 *}O>wLϠ gt$׏.]p߈]Yzp&tB5z綇.)){C7Gz>ܿ_n]ƒ^~s(m8 G$Z>mkfzM] pǙ]S&[88ImƉ7wSĩTvbalčq }ߺzB-@@!qCO !87u;$90SzOozzfzꞖ=7Bw#cKaJ zPzFME{_$omCǏ8W8AOصms8F'enfVUx`pPn`xLɏ [9(3QmnKx`zt?,^pӖ3JH @zy*K:^O0,}U:9.g3Y >~̲alC }^w ̒@* ` EF+=$͘H9}Lyf g'C;}Ţcrx *iIx [- P;B-F%0EX\קlw;p 28<{r]ɐp~Wd+u&=ȡQ.->t)$ַBw1nNE6X +0GD<=V/`0"Ne3<KaM$lߔVc*IHn4=b<%%9\"n]Z/W/$^ Wn `IݹP (( 'HCI V>/\R҂^=Co-t7hצPsFI-ڞ&[3 5w.3*S&JñK[xtל= . Ám[chs7W2[6,U*7-j떝٨%vMfM'32H}v$RF0bgBaFfر%/\8w-e#oNj'tx=|M.ٮ'Cvc%.w_++*HƼ0kuP='sf8[v-i>f",ղ_=Ap 7߽pcz6J ?w*ܿ_kö *{[9N-#<8RJX9 {JLɠ~˺xl)%˼^\FNJn@EB΃GWY+7 I^u<@OZ##U!yqo; `{™{ +VLE_I߭9_]{}d$`mSbx!31_j~Ԯfؑɹع䝓W~)1AQ;OYikx~\TN*(9ߺs }^ڶpb142R/("D<ai?Me-f$.igK1KJr 70>۲ !_J¹cK*ud@z42h|6z9xb\R.\ ̵=p~ \9un@jF LsH`ۙk5']y 7 71wgZC&6<{]'VH#[VW~Ǧ ZqdC5 E[:r[ނ! 1lN!BG;C8{"m #P_WExuGK'ؾa31&H»@J^T䤴/Vj^{5r#!KAM a#AXFbN+9-FeQ8bRL"&H觮"W" iޮQjXF砡6>iqr(WeV<"}W3B@B"xlh1E{Cq) .bBwc0>:3X5 [:mA#]IѾ"Ι7=kHQ_:NwƐo*\D=z* <&ط;rEViuB[s =#JC~ -xݹҞw.wk-'!3νUb%r.;Y@Z1+%jq5&a$7`~ԊV:xݛ*r7l&{Ѧ[Bg[:#سcɺH " ѪG<@ښII}+o8ODy̯A% Nio\a$':kBvIQu h?)g^$t^B5+м1͂NL,FMP^G-ݚNjopy_u;!lXyNa;3:fR'CMeJӦXz׉ H-4PPH]Pnlx>T];,MQK&|]Rp/JlQֳ$ckuӾhdz6sS :sa,RwntWD p8 &speX?-Οܬw89;{@׮1JpC L3+fƔ1c cWkxČRZrq[ra̟ރ^x2g;MBaښg ]=UM-&?3.lwjƵ ]=Y߻Jr(f ҾDM[]{1F1$y E~| Zr8E%lFOm$V^"Z**5eBe'jX^6B`O1Kɪ H7V,xCE/Ga+t+7нh} MWmt0##$t(QݙPDڼق;d(/:' CD]%GL76=~Pƒ1trRORBw 3{TVgt+e C6QMBK^f(*O+;rp+1d% &:)#`uD.&l^8K4HW#@YЗ^5tU%r<y髻)@`vjZ/bL;2eBׅzi&Wie)/x~y*Cai^̜9+]RlE˖*.:?Np#;Hex<ϙkD99vx[sf?TW]})nz 13?\wپq" 6&R 3&u5xSiku搻]'.@\@ H蚥ɲgౙ%~p}(rU-G++\i?[ ]OtyHڨ1=*ʋHД )gsɂy7nUhsþ]P#%ܪLdO{-tQ+W. &wADܲ,rJعp\LL^/8j٧Zk@q4iMp=G>j黎g:ZNJ4#+r,M 6нhoUaMn8楣OcXrrݹyMuexBe_dd~ 3uVKUZrGx9t~#+K3c4T ǖK%#2_ n Pv,B !?N gRQ֙j{O^DPo=ܼq˴ʬU@owcKK4Gcca2 1vMPnmgнy/Rذ~0s0ebH,4s1IsR<K;`l̈́^Yъ4T\(c9bBBá'/%ʼnPHb2yBg=;ݣ.d罛Iq$t/mk禌C6P AsL~ >ϟ9H;9*+CbNxp $X. Œn0uj}R\0a-|M8vq+,8SYrKjIADU#8͵h==-p[y0)3=;, 맽fL3l/˚S3z E6_4p1M[S_03%hu ] $8/f~1I0ڮɿE[X*HY2Z@Om(VȜ״ۓybÄOa'(G] Fͤ8ݵva^bhetoK;֏VyvgƖ HF<v >~^N qʣC-DZF;<2g<߃DxP RBOpA*ġ#;ܨ/!m]!va?MS , ګq+JB&O_=&]/+ pk\ėyyӾe.+`Rs8'R 9ޢ9>U| ,>I~3 cB!6~lZcQ2M茧k:Z~Ax0>SEږx dV⃇Uo"_ƍGFSb!d.br)>oS+zMm|Ռ])u,;wָySa|)lVO>+ ƍczP|W0ˉk¹#F]ixMc|a,qZvk_Op LgeMBwצ7.wo `U8 [ac.w4z1$l/Lc2#l=!i@|a]7(fƁ@޺ Z||O/`Zs}IDV bv/O+^ob7`:7]۞!x yyzjyp'L PtOa6(${Ȉ0ezJCDm0%8Z;usb[dr&ZѢlv-E>\3<y5ޢ {n6/H+ǐLkOmy8'x0Yt<`FWX֗ڨA\ZԩŃ4."7LB3||7X8SÃdSuK/_*Bޚ6;{`C?U!%ձr=f2<3$I't"GtÇ s&wc>H'OL5L@K^g!Wj$zۋ ^-l3y];G?Ҽ'Oe$UG&Lqpf{AY,Vc OD}|߫jl$N4 |f1<_2ȨDr]\IkLvbG5433c uy= 6ێm.]d djn/5aLh&У} >œ/`a|ON ~dc>m/:X7:0E6 ΁88{Pkcc$ ]x ~mT=mϊ;|c2 CO/`;>{ă/ɵ=qԧ0)ڣF2;>S:S{ 'RK?Zhuۖ3<`B#)3п L`^o D.9>GDn]ic$t/|v݀7l#y ;F|44Vù m^;ׅVϴtԀqj 0 j0a&hA1L|ݑ ~FhE;uOX]([6˗2ox kJUXswC<Go~emϊ}d(K w C#H죛B "na)AGM`VCljǃ= ubo!N-l# |]{/6._1 ݫ6xe-j" *.Y8\ wTBWmIB,$Z횦HZ0&͠0ᛚ0^=aDs|&}NB-LÂZLVpVA%Y-\Zx.]a5Z}ݐ,.ruXWU>2b fx~$tm V[ωS y<߭Ʊ#+2L"ၝy<@/Z$/Yшsx͈7 xy1GL!+:\q6^%$e0ScE)1Z~h/ [Ю~38K}MH\>?vik2ܖ>*f 2っx:M."OvtoN_3Ram pǘMI5~JlS`v\X"<$6ͷ~G6yo}p3 ֮BUBMKTQ;33 .Z-eĢ,+L1jFA/DѢ!Z&Qh1I}@vΙg4ל9L=ۼxGO._#~[63 FӖ{oVsWꗈSxS]c{>\AUk #cϗ=g)A.q0tttttttttttttttttttttttt ]?šp9B:grzT׏wUr?8}5j ^t}xnuy֘yhfceJv_ܫkhrBᕘǚZ՝W;^cFW+ ,\SbMט箏ZA_'ot&-Ϗ,}1ys6q6<=>)iMv Mq[H^y>rz Ln7kmվ'}w&ٳ+6vDj)cId\5hu=RN=81sVto\.qXăoF.Cحw\A//^sX߭䇟a60 7z9t&9L#$yhuk]x4 tiJHMEC $޻*%ޫ"UJ? T:Ix9o;7{{{7;{9sf?O%tGڔHRǀ.)3ⷸZ@vgj1ﵜvFB%m?F'l[M$#QzpU'y.ڽfatM5j*m*elޙga^٤-Eh aitBwq 0qMID , _C:OIQ8.Uj1wJUR\"xxul93~dL /3׿7U*6J$W^DN@p5e,gaO?d m?&&jF$tC4d|~&-D'tW65zݧ\BBDz9HH`&ћe~НlBT'7jxŪDf~K'T78Gx8NԯBP'tNN6StBwm MT'N: sYZ<ۡmuBϖ-|C m-]UÒ~:j:K2+ oGN,./Y93f4Yϕ+q9C v%B !C 8t18FGNnk't\τΝ.]n92e2,XЦ.[Bz%tlr3f ?~wdɏ^֠z:&tJ*lٲ&~wvZJUX1c+W={Ѐ^$%Bij}v?Z<>K,UVѫW.]8|?4O%Rlћ7oҩSZ%0zqnիMH+sxxɶГ'OZ<^dH/N zE? ]ۄn)ڃ~~k֬zM۷aLݻG+^H/No۶m:Gd77nLkN_Çw`uBw[ {T˗ӍK+2e9~ҥK[ntڴi, ^˄n,~'}'NǎOx% ]Ä.7qѕ+W޽{!5|z)7ie8isÇi\\qM+UODr"QA~&L|^ 7̙CgΜi24ѷo_.ך5kr鄮 B$ 5kO7mڔo˟??OB_}l7nЄڵkWb ڹsg5  \Szv]'t ] viGTڧ~:u&&&ҿ/t|;uDȱ7x`+Zvm$<9<֭[v]c;R dx--[s(FwJu###'OtBWGxB=͛~~716ydےxϱM6<؄_p=|Z%YpZj[B_EFѣ<0~$ۇI:|0ߎ1| gx0cMӄ.a/z0hڼy90n!bcp XF=t۶mˍ{РA#=z+@6``?lrΦMàޫW/:37Ld@tBWMxB@xA:qGΜ9:͛g܏?-j#\=kG` ] |'E`&Mݻwi~:2tw ݻ=| ΝˉQF: B?{,7Fߠ|0n"z؏H~֭& DS=GJ%^q?z葊-x#%%d1Co|?j {;*˗/σ$VU4AʕO>vD1|~1@9qD6?l4a]~̂A LZ$tw+EXG@~R rG鄞.]͑Bj2 tyQרPwGbgϞahBsqFn0@CE{9BWd^t&rŋT! A't5M:;Ę%7b9:[r3O$eBG47%tB]kθ|t5Nb1s(]*CcN$ՙ۷۠lh0d}||FF:uwh"^l}a8>F Eb… 9Uר/-[ Ř?Zb#4Xt+Гq_x/yy0D0!;d4 y Y(,zMHӋ~ o8Id6PDuk m ]^0*|f8݋ dRؿ aď쓘c/D4NIo@2l%:Ph,crEYo@G YglA't:`9#&:%pD|P?L#ߟ#G}|ғRe|!O->9 ?w`\sTb,yZYbY.?ޑokm_,Twaɮ-{϶U>5LĆX.a{ dz$Bw\~wvNnk tb?1wJ dj`:qlDyGi>Q.hf>u= Vi;'i8$>-޳|0n*/#I=ϧ.e˙H o~"BeإbxáGT@΢L.w(%wJVE%#"xESyM dci^OnMJ$@%6vWWiٹS~!L7a @o C1-zH= )^E)M*\2MT܆wAf1 o1#GI'yd@{u_i٩?Y75 %ћYC4$ R4 c "©b9$"E DzQ_'Ro N:{>TԠ+Hʇ} GTgSH>d8"PE$A==Ԇ-"eQ;GQқvz(-`C$Gq ER2DjRoA esoAD  1v)m_o 42taÅ\eKAO@€m&[O"SD o(ѕ(KE#1Άzi̗8 8{#HsU'F^ ]Ec9yW[qG>dzogS@= F.]!{=)x3Fܢ^4m,R:3E2%ƽ.tH&25缀9@!jp"4(.F&ѹOzȝ 8w] WDL{ec-Z]h ᎇ׵ET=#6ٸΨ5%}w9u1vdu7euR#ŌxGd˛l9h^d-q<4nAl믑0rʈ]r])v {c３HL%0FBjR>cʔGRUA*2R"kf ]?ӭ3g g2#nTUiwPnN|Y_Jeb ĹC ǰ&KDs.LtQҲҳE`i-B[G(hBib$c_!jG$*v3Rh|٨=#j !֭jc{/`m^;G)^d1/Gf\˹~+q AG*2rʼnlUYNrQYNjEd$1 j:;osQ6LDZKbʛ:8"52՛+X&IDwFwHʾ:2r7*M/TMƂN \Le6'~#ҰRkAI78FFHG7G L<=)Arϔo4N$yƐ53D55|Qd#MQ@]@;y)',~rzJpSWv֟h.aejc3\-|Bt5wzg(E[wGQ5;OԹifc$#~n] nݬL %Lm~3hg(M.MJ}XUI!Ӷ9`V̂̍9c@0!(Q̊ KL<;៝靿{gޝCsRMJ۷S~W_U_b@b|m޲;oK?Kk1ǂ캄Wg+-J٦(j7v,,xl  wD#R~\-Ȗk/9S_-ْ>V,}g 5[0- M_'ؒrěl-rD~/Nn5Vpc[ځ7EDXsògk3rǪrXS>NozQً`%ڧܗlp& 1 K,vzca&`Msna$x2AvcyL) e+!)Wk%n:A m(pb'* 7=tiR&{&ĸ`_6B M)lɖ^GbsR:N] Kvu; |G1AOܚ:+4mghN)AmBCCӂ&lO&O!ԦGf׫ {8`w>;Ϧ|CS'-i\hnmR ڰ'4mohx5\&سՌVbK >!nxE0-; EJ' nJ"Q LD0U߹pPo^#w[D WAB%>Ip7>4uyx^֥cݼh/Cy:(5ޮ(v]| j즄mi;%2)ɓVvqiْ5*>uEQ[:rA*;%W6>"[&,]To.*]0jzԝ>m,zlE2 S)܌Yk76/;˂A\˟,WFÚ5&;U+U0 ;҃/=jl?dꊖBNm6/UaļbrplQFP>sڎ\7A/˚/H/ݓl=!Iu$.0'~@!IU_ǕkŖߞcU{r_ukƑEg_ 4߰N6cݻ檎-lClkzE-z ؽ5G8Ah~L%guCbݘ\dR5oL|ْ_/) 8Yhɧi3iӱ]go6eϖjdH'[ZRBK*J%ZHt:|TS7HDQ$IYHh"Yڤ|2{v /î}V"WhIs-yRВj,^\R}Wi ͟bJ&gq[ұe/eťYXdPGE) pAġԱ(t76΂5npS,w<GuהGj !zmޞL aҽ-:0rʓ='a6=Ӈ cW=(SdMx3ş?ULܓEg%pAYCTw5^ל/V}y[)0ÜVU")B-(LŽL}5}VHOWcVY"wJ/EZRR!J U Q>NJHs57 jՂW#ODLJd-e M$KE \-9%ܾLOf։֌: sx]bqwETEJNU7k)Thq Պr2%](~zSRէuUH%m,4h}_jb-9}K$w(аY -y/ H<h0a7,&e0"48l" XH75Lojfm2ygYQ7קh{ SFSYpiE-k1.cHy%"~%sDɚ3esZm})1e@RƣodXx@+ uA6^KkJH$3u_<+\w֯I_mJx,oBP zZ\_WĐ"r/lH5dH@"U $Ѭ ih9OErHuYt070%g ~jr ZeED04.e{\K,~iA܄ ap1º|x:"xp-Xۼ9X0)x RkȳHoaar(] ',bo}N'CP9|Jک<)3ݛTGz$q.~^ \N{KU5R:ƴZ55@F%5k-ecnjC5C5<3"'"\yUe?bHc[p`2` Vhɝ"-Iӑ"ٜREN\&kȳ u͸p뚀ЍY ^e;>8yEIU呃OYV/E81v/CT:,JV nXBkԕ+ڡ@  e0l#)PK[Sw KgŅ?|UuRn 1X,PX Z[W7` eR|"(<1Ccj#$Tܧn(qZrN cpBW)ں{̋)-{IOHaKN,¹quOlt?$Omu%|Ҳ}IIAq@m<ԽɗWJ~ܐhn)'b枷!«!BڟYMhITܪ!2t_Ɩb 33ѩ^x oPK4䝣>f+0f[xrg0 )Hy#R߈Ax$]u8RcOb6&U{MXr -{ƹ884$Bw{%^5qk0Qa),tkԄ_8HFV})UټS%O>t}C.cK8;eW/M9RSZ%[4 \R~BdeBڎnno+?ꈴ;Dud_EFy86 osya= Y(Ŝk> cl$sHyFP wp»W)4eO_$}ƴ*C(*-DCf<S͈ ̃Cc: ?.y``!x*aQa)?< ) s ;{> s r[75ްF0[SFAU~N7a$8&=3Sؒc0PŎs}D[ sҺV~& "YWQ3e["(>ᱭ/pԨή{%05MVBC]dokj@. sBZ}}?Gz_noK*sG (f7]RNeg9 eTjRTޟ"jUl*VÀ)YP\s#tVR&jBWb%9@tqٛ!Y7Mܚ8{s7DY:ºu눫+3f !'N$Inn.1Z97&,Lג9M=qq15X؜{:|WCc-) \d}m$aJ ҡ1.ߺΰ(R4dNG"""ѣGUz E-[FJKK}X_7ӖؘL~s$3 ˲XŎ*-{:cCڮfK&1}5>{T$s1"IKK#O&d#'O$gΜAxLk<18 "ke~cڳY9G|*_N~ c%,)r ==|#+ϗߤ~Rhg-JmnΌ#Q7ttlwZWw,Y†R_UUE.\HϷmۖ<~RP4OXB7i?c_?~ߟZ8]^^N>}JpѣG$//Kaa!-.‡|P܂}'dʼ546h{lڴFQH߾}y`w~(v" "&ŋ͛Tat2e -nݢ}86{l*cxFݣGuB8p*(mσ6 zcC 2d=֮];}vÐ95%]>.)>Yˆ7T{ ڰ( rjnjW#tSN \r <))DEEQ @h>|^/As9r}{n1@7XBqN>ݻwuk㩅Eh Ca'N;wP2{-ݣi1~6u/;/v.c0vn }I]-fD\˯k[SF!!!̤)1oJ0. ?9INN}D ZxL =ݳg+r; Ba`ߙ\o߾Mq02`ߡ`H!p q>^("nPQ7+ ²ի=Y狌ߡD@sԐ@K`ݜT+S#/<#òhQsGnP7E{вàRưiuZ X<#30 .&媵+Pw܌ vk:9B o4a}`2 az8͛{Imhl }6tP(3X ZA0L0E عs'u.CXX5 'wF^~;fEQAf#W[:v!]ՈãIaHN߬&s4]wZ6r)pX''[z<J]t vAyD5v^@W~H ׃1$t\m!T D@pvq`<7wИ:o^Tiɛ2=7]x`Ah\RO݇x nsw+ktQKUgg^/)&CUt1 e't6. sf-~:<;XYf3Bر#%tcYB1@9c2'ӟİ0sr$q #t 10aǎK#ԡ4ratmgc1EMӆ+֒(4tpzF";d9Bgtή~=L^Jo8[;Y䌝k-:2I&m۶#tc߼;٘<2'ڀkҜu^\ t@|jV爣c\ 1VʹyQcD2A4ԏq8:a#!l gzi4l 46m ʇ (V0dZs t~IcSMKDWmX]ybז,O[X]&Zl]\ !`J7s1pi`p*^a)(۷DSR1 {l3 `H .uxPG=L^?D Ƶ3NII8;c043/,m  k@yC[Y]P0/j*{,Ԅ/Ւ=GdUӝ٘RkqظkDVنך$Qwk7+/ݪq7fT`uvڣ96A8,v CC6/Жqr59V/ځOv #G 3K֭7[Ի^=yŚڍ=Tg<ɼ=zuU[*̮0VхS#"H8{ [np°f-^Q E[[cnd(~ ^Xp(L&A^yߓv [KnN^X*u%Ş ٗ)~7 *ٹxt-%[]ocHM9_/ߨ_mYGeYe]Y `q .c1/ ԿqF / hJwq:ε]]v!,M=iעR G _FKtgtS4RxX+0~]{.:z9!͡[x>;GYzyߋ@/~oe9 m#6u6m;,r;޿]j*q&AZAOMhةV?ZqIAY gt8_®_2iiyoصnyH<[ = :b_y k[dRi7蔬l'zfYv( hA蜛$0(Wx7ym~wЅ)7/Mq@mEkX+)&q*Z+k&!-ŶЈ y_{5-&&L3qd0S),qYn`4Vcf'Iͥ^ָSǮtèN63[C'`˱i³}m{^< 2'˓Z D[ ǫqqqZiB,nHU8r<4p'/;6'Jw۸qd# 4(1v,P\<kEpi#RRRS2ivhĂ#j:+ǥ{.5JDzu/N"vEb'S([Y-f]^˧YjG22Vfg+0e3E/i M7xbMF.HkZ(X`ru Ғ1rÁS Tv8K,APA#ss^{GS.ڵRvx[Xᗄ!sܡ{/C.5swz~ӗ=pm z5#U70i4n5zEyV}s{}v|yqo^+Ăr6n:ndviIiM_۠s,/wQA(M,k=M5*714-sbg|Au#(@59K!@GVtH+XD T4MsjՆh17r\FB9*U8/-tj=S{{.?пm;W ;uA&)Au]m7WhX8n7\ނSc\ע#^-4 |ߗб ַA1Q}:uݻk'Oj05:isUb5Y}v:V9 dઔ(.s86.w{c b-""B"exq瑄7l5|c9?hʞy[89HSSq+DDwgXoMjwt9|9.7H2ghcꊁХEzb) me, ohƈx,xm!vR`HBWt ]ee'kaÆZ}u\{zҟ=1ۭ.uRҭ?VoDCހN)ncK7AEKzha}&$n mP1Htڴij5G(X،K|y#t%  9˦O)Ud$ne. 1= W=?qw_v;4ŭq(LsQfR`KY4#d֖SNCc\vU}= ]F5: FГ\iA̘1cԨ(MSg92s,E`^q,薅K ]XNR>Ftɇ76Amذ7..Io>ßev-48,{VԹ= ]`1(t1oO&_ƥ=X ɢ%Y 5V:h 29]jc|p"GBeEo2N{{Zv2j\Ԋh˭>7%qs} 35TNYu6oG6Efĸ[=SBo_:QDKRg=z WWpC(lw: asa p.0-]xUV-B7b%m_[A\s+C(ވic]5=0ZLfkUr1: -{۷/Y y0DN[qyTӡC_~ hc# 4! 7a[~KVpK}6A`T$#JDS KCr|@'AS7ӿ-b>%HRYOeǟv-N'JSF0r(oW<>KJ2#xriX8 \Vg  o }N;Hw3y.%x~D=V6!k62cc[=`ς)=aʚ=Kt= Z 5V`M#uamЀ ٹNqƬJ o~&Pg䪧_Ro ~SY\N<$M65󾣀_^K2tpy~(ن, q9#^ "Oycڔ":B\ҳ~┭)O4-ލȠk+n$O\q Z~w jwwsǮn!}(0E||S4E#Xو;وoĥ;/{.v~)KV$w)3Jh;5үU/|ӉxƛN:ԡ2~մߴ[ iŜ7mZgO6APDBی;evx׏d?ec6_h̖|щlSOٵ$Q'(,ސgߋx-ox{uСdWf!?V{QNKoo:? ЍqrT0K> y c\zVΌ&6?Pc7]mڝO7$S;HwfA˔Mg;W\j\`.9o]!݇۵]q^vE.RNZMM~УS;ma1rIҘoMYC;d)=b1p]q4Ub+Rq;J93T-aTmtHD8 !ͮfqEqo1oX#{/z|+ow[m ;ˡ>$,L6i$co%oŸ?W4O:Pg{7ƬZ3UxVUt bbhk{}7ۙO3S$G78xR/d%ΗEYܰk1z~u'C8aaxʡTti}0NwYFeֳX&%9_OvWi?m;&.|Zo;)z'a=PȈ@ygF@M"&{N~#]PWލ76q#T!5e- C~ nٙIEwxxx<5Q sێOwiq[o~Ѓ/}KM͐afL;jq$@9S]ySw|D…3vea0<*3R67%}d y pϏWuPP#KB08!9! ln+36Џ7KܘѴWĎ i/Kwf!zMl6d궓2w+x7Sgk|7O|]]yڮ}) M'l*,n 4Vx'o94& @KFPH8 P|z ؁>giڽ&'Sd= 1t2Pqs2$p3mơ8wbmV;ݾOo6Qq[O|q[vY{PW:!5[|F`\ƉT6uo{ūI>!A=dha׺'k$ӸGܼ^KR88@r~k65G+xr8f3[8rXKr $߳RjGvҪZt1?͆-^r욤 cWnn6|! ꙗϭݪĐfcT|:Fɯ [Gh`*.y̮T0 ;Yv![1B5on5Ri 8(=-EM|\1-b)n'r^Fv_{q=.$`]˵~_;)H9skNV?ʯoy $܀+k5 G QObߌ.yw iCB1e`94V6~R b\ ^"kY е?t٬ZҜg%}Y#nl2>L@6 #L}vuv6hy ɐ~J~ty2.یZ'd%#exoo_Oդ.u\5%MRjcd l_HywMg<{EQv;HyE±3e,W7dȵq  RÀ;P قi!dN(f~d[t-ǒleRy.9+z/ ^'tհ۸Ya13?7djێ_mu5-jԨ)^|mcV%7t}V4ڼ:N(T e5+6@CyJɏ.7rZQۨK2@Dh` @;9%}@HVG2oyKֵäG:Nݞ^]Ũ[]&;]DO !3,hy?Z*[rA esIX0ϹOv w;YU+ L?L;7 d|΀f<%omp!A\ .`ZKSkpoqg`N9qY@zKg2a2mޘe7NWON[K ׄs>8\U `ּ~܏we#sY&1s9hoggƛ0+v3>W=y._e2kTP:iEw`IENDB`ovs-2.9.0/Documentation/howto/tunneling.rst000066400000000000000000000131151324262074100210410ustar00rootroot00000000000000.. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. Convention for heading levels in Open vSwitch documentation: ======= Heading 0 (reserved for the title in a document) ------- Heading 1 ~~~~~~~ Heading 2 +++++++ Heading 3 ''''''' Heading 4 Avoid deeper levels because they do not render well. ============================ Connecting VMs Using Tunnels ============================ This document describes how to use Open vSwitch to allow VMs on two different hosts to communicate over port-based GRE tunnels. .. note:: This guide covers the steps required to configure GRE tunneling. The same approach can be used for any of the other tunneling protocols supported by Open vSwitch. .. image:: tunneling.png :align: center Setup ----- This guide assumes the environment is configured as described below. Two Physical Networks ~~~~~~~~~~~~~~~~~~~~~ - Transport Network Ethernet network for tunnel traffic between hosts running OVS. Depending on the tunneling protocol being used (this cookbook uses GRE), some configuration of the physical switches may be required (for example, it may be necessary to adjust the MTU). Configuration of the physical switching hardware is outside the scope of this cookbook entry. - Management Network Strictly speaking this network is not required, but it is a simple way to give the physical host an IP address for remote access since an IP address cannot be assigned directly to a physical interface that is part of an OVS bridge. Two Physical Hosts ~~~~~~~~~~~~~~~~~~ The environment assumes the use of two hosts, named `host1` and `host2`. Both hosts are hypervisors running Open vSwitch. Each host has two NICs, `eth0` and `eth1`, which are configured as follows: - `eth0` is connected to the Transport Network. `eth0` has an IP address that is used to communicate with Host2 over the Transport Network. - `eth1` is connected to the Management Network. `eth1` has an IP address that is used to reach the physical host for management. Four Virtual Machines ~~~~~~~~~~~~~~~~~~~~~ Each host will run two virtual machines (VMs). `vm1` and `vm2` are running on `host1`, while `vm3` and `vm4` are running on `host2`. Each VM has a single interface that appears as a Linux device (e.g., ``tap0``) on the physical host. .. note:: For Xen/XenServer, VM interfaces appears as Linux devices with names like ``vif1.0``. Other Linux systems may present these interfaces as ``vnet0``, ``vnet1``, etc. Configuration Steps ------------------- Before you begin, you'll want to ensure that you know the IP addresses assigned to `eth0` on both `host1` and `host2`, as they will be needed during the configuration. Perform the following configuration on `host1`. #. Create an OVS bridge:: $ ovs-vsctl add-br br0 .. note:: You will *not* add `eth0` to the OVS bridge. #. Boot `vm1` and `vm2` on `host1`. If the VMs are not automatically attached to OVS, add them to the OVS bridge you just created (the commands below assume ``tap0`` is for `vm1` and ``tap1`` is for `vm2`):: $ ovs-vsctl add-port br0 tap0 $ ovs-vsctl add-port br0 tap1 #. Add a port for the GRE tunnel:: $ ovs-vsctl add-port br0 gre0 \ -- set interface gre0 type=gre options:remote_ip= Create a mirrored configuration on `host2` using the same basic steps: #. Create an OVS bridge, but do not add any physical interfaces to the bridge:: $ ovs-vsctl add-br br0 #. Launch `vm3` and `vm4` on `host2`, adding them to the OVS bridge if needed (again, ``tap0`` is assumed to be for `vm3` and ``tap1`` is assumed to be for `vm4`):: $ ovs-vsctl add-port br0 tap0 $ ovs-vsctl add-port br0 tap1 #. Create the GRE tunnel on `host2`, this time using the IP address for ``eth0`` on `host1` when specifying the ``remote_ip`` option: $ ovs-vsctl add-port br0 gre0 \ -- set interface gre0 type=gre options:remote_ip= Testing ------- Pings between any of the VMs should work, regardless of whether the VMs are running on the same host or different hosts. Using ``ip route show`` (or equivalent command), the routing table of the operating system running inside the VM should show no knowledge of the IP subnets used by the hosts, only the IP subnet(s) configured within the VM's operating system. To help illustrate this point, it may be preferable to use very different IP subnet assignments within the guest VMs than what is used on the hosts. Troubleshooting --------------- If connectivity between VMs on different hosts isn't working, check the following items: - Make sure that `host1` and `host2` have full network connectivity over ``eth0`` (the NIC attached to the Transport Network). This may necessitate the use of additional IP routes or IP routing rules. - Make sure that ``gre0`` on `host1` points to ``eth0`` on `host2`, and that ``gre0`` on `host2` points to ``eth0`` on `host1`. - Ensure that all the VMs are assigned IP addresses on the same subnet; there is no IP routing functionality in this configuration. ovs-2.9.0/Documentation/howto/userspace-tunneling.rst000066400000000000000000000170411324262074100230330ustar00rootroot00000000000000.. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. Convention for heading levels in Open vSwitch documentation: ======= Heading 0 (reserved for the title in a document) ------- Heading 1 ~~~~~~~ Heading 2 +++++++ Heading 3 ''''''' Heading 4 Avoid deeper levels because they do not render well. ======================================== Connecting VMs Using Tunnels (Userspace) ======================================== This document describes how to use Open vSwitch to allow VMs on two different hosts to communicate over VXLAN tunnels. Unlike :doc:`tunneling`, this configuration works entirely in userspace. .. note:: This guide covers the steps required to configure VXLAN tunneling. The same approach can be used for any of the other tunneling protocols supported by Open vSwitch. .. TODO(stephenfin): Convert this to a (prettier) PNG with same styling as the rest of the document :: +--------------+ | vm0 | 192.168.1.1/24 +--------------+ (vm_port0) | | | +--------------+ | br-int | 192.168.1.2/24 +--------------+ +--------------+ | vxlan0 | | vxlan0 | +--------------+ +--------------+ | | | | | | 172.168.1.1/24 | +--------------+ | | br-phy | 172.168.1.2/24 +--------------+ +---------------+ | dpdk0/eth1 |----------------------------------| eth1 | +--------------+ +---------------+ Host A with OVS. Remote host. Setup ----- This guide assumes the environment is configured as described below. Two Physical Hosts ~~~~~~~~~~~~~~~~~~ The environment assumes the use of two hosts, named `host1` and `host2`. We only detail the configuration of `host1` but a similar configuration can be used for `host2`. Both hosts should be configured with Open vSwitch (with or without the DPDK datapath), QEMU/KVM and suitable VM images. Open vSwitch should be running before proceeding. Configuration Steps ------------------- Perform the folowing configuration on `host1`: #. Create a ``br-int`` bridge:: $ ovs-vsctl --may-exist add-br br-int \ -- set Bridge br-int datapath_type=netdev \ -- br-set-external-id br-int bridge-id br-int \ -- set bridge br-int fail-mode=standalone #. Add a port to this bridge. If using tap ports, first boot a VM and then add the port to the bridge:: $ ovs-vsctl add-port br-int tap0 If using DPDK vhost-user ports, add the port and then boot the VM accordingly, using ``vm_port0`` as the interface name:: $ ovs-vsctl add-port br-int vm_port0 \ -- set Interface vm_port0 type=dpdkvhostuser #. Configure the IP address of the VM interface *in the VM itself*:: $ ip addr add 192.168.1.1/24 dev eth0 $ ip link set eth0 up #. On `host1`, add a port for the VXLAN tunnel:: $ ovs-vsctl add-port br-int vxlan0 \ -- set interface vxlan0 type=vxlan options:remote_ip=172.168.1.2 .. note:: ``172.168.1.2`` is the remote tunnel end point address. On the remote host this will be ``172.168.1.1`` #. Create a ``br-phy`` bridge:: $ ovs-vsctl --may-exist add-br br-phy \ -- set Bridge br-phy datapath_type=netdev \ -- br-set-external-id br-phy bridge-id br-phy \ -- set bridge br-phy fail-mode=standalone \ other_config:hwaddr= .. note:: This additional bridge is required when running Open vSwitch in userspace rather than kernel-based Open vSwitch. The purpose of this bridge is to allow use of the kernel network stack for routing and ARP resolution. The datapath needs to look-up the routing table and ARP table to prepare the tunnel header and transmit data to the output port. .. note:: ``eth1`` is used rather than ``eth0``. This is to ensure network connectivity is retained. #. Attach ``eth1``/``dpdk0`` to the ``br-phy`` bridge. If the physical port ``eth1`` is operating as a kernel network interface, run:: $ ovs-vsctl --timeout 10 add-port br-phy eth1 $ ip addr add 172.168.1.1/24 dev br-phy $ ip link set br-phy up $ ip addr flush dev eth1 2>/dev/null $ ip link set eth1 up $ iptables -F If instead the interface is a DPDK interface and bound to the ``igb_uio`` or ``vfio`` driver, run:: $ ovs-vsctl --timeout 10 add-port br-phy dpdk0 \ -- set Interface dpdk0 type=dpdk options:dpdk-devargs=0000:06:00.0 $ ip addr add 172.168.1.1/24 dev br-phy $ ip link set br-phy up $ iptables -F The commands are different as DPDK interfaces are not managed by the kernel, thus, the port details are not visible to any ``ip`` commands. .. important:: Attempting to use the kernel network commands for a DPDK interface will result in a loss of connectivity through ``eth1``. Refer to :doc:`/faq/configuration` for more details. Once complete, check the cached routes using ovs-appctl command:: $ ovs-appctl ovs/route/show If the tunnel route is missing, adding it now:: $ ovs-appctl ovs/route/add 172.168.1.1/24 br-eth1 Repeat these steps if necessary for `host2`, but using ``192.168.1.1`` and ``172.168.1.2`` for the VM and tunnel interface IP addresses, respectively. Testing ------- With this setup, ping to VXLAN target device (``192.168.1.2``) should work. Traffic will be VXLAN encapsulated and sent over the ``eth1``/``dpdk0`` interface. Tunneling-related Commands -------------------------- Tunnel routing table ~~~~~~~~~~~~~~~~~~~~ To add route:: $ ovs-appctl ovs/route/add / To see all routes configured:: $ ovs-appctl ovs/route/show To delete route:: $ ovs-appctl ovs/route/del / To look up and display the route for a destination:: $ ovs-appctl ovs/route/lookup ARP ~~~ To see arp cache content:: $ ovs-appctl tnl/arp/show To flush arp cache:: $ ovs-appctl tnl/arp/flush To set a specific arp entry:: $ ovs-appctl tnl/arp/set Ports ~~~~~ To check tunnel ports listening in ovs-vswitchd:: $ ovs-appctl tnl/ports/show To set range for VxLan UDP source port:: $ ovs-appctl tnl/egress_port_range To show current range:: $ ovs-appctl tnl/egress_port_range Datapath ~~~~~~~~ To check datapath ports:: $ ovs-appctl dpif/show To check datapath flows:: $ ovs-appctl dpif/dump-flows ovs-2.9.0/Documentation/howto/vlan.png000066400000000000000000002432161324262074100177610ustar00rootroot00000000000000PNG  IHDRorLiCCPICC ProfilexYy8߷{&spdvyscL)JATH*dhPJE$L}5|s~Zkλ:QÃaFBB"l(.;+2\ ϥub3d=#B|X+<" *B`}3G "6/A J()1^~XPPz^Hm.zT?:T߂D&6 مG#WcBzBhoXό7yr!_>Cd P;%C=-`-#[#c!(m37<=)_iWOi lg1"n3D0QDo>0`d[f 2ݞ9P \0DR?T"ABp2" `9燌) g߳Q9 K"sl󶭋tHל%FVvV_hy"ZB ;*h]6Z#0hkFߘx5G=r/ \_ Ca~Q]dHRLC%)r7m{mbOE$DE 8-$qEF⎾y^1dd@" gyԁ0f ď? ) iPJ@9.F @S`| ;؀ !2BB$@Z!dB.B( : P5t u@gkh B0 &07, *.l{a?x9p\Mp~ Ÿe@ѡXQ|() JerE"PI,T> UjEFQs54MFSRH^}$t6} ݄B?GЛ" #QØb1~XL&SƼLbcXVVkubc v38vñ$p8+pU:^y#WPT|;444BjV޴񴹴x^Sz|7~NΆ.. }15AOC&ׄ%"(L!9j=qL/MoJM~~AA!!C?#-0>#1&2L#YBH٤ Iɐɛ)8E 듽i r7y,l|2&,,q,E,YFYQ¬;wqdG+ll:l>lYl l/)A'؛ه9699899889p\\\ʹyùra 3K BѥS (]>.>h |O6ES*y  R:#+","$),<#&b* R+N(-OLVLE,HSqX\Q_H_P8'L#**Y&9$EҕfNnq9!+)(,[!VIL.UU| Aŝ;}vJK1SS񧒲RRҬr J}UA655%(F/RA53">T Z-RQm>mvoJi]1@:y=Yz+j(c,'L#FF~FF ƊM0&&'LLMLM̔ͺ v,-",ZwvR2Բ XZg}kcmSd3e+g{׎lnWc^>CC#j'N2ΉΏ\8\\Z\q˻ w=GqOƞ"{>pp vNupAQ=M==x}93{wO߬\~@abI`IJUPUVSpCMGPРЮ0g޷aQ ElbF.}ѢчbbbVccőBO'%\܏ﵿ߁c$Ϥ΃N&'_J^:hyMZu77fg7nv޸%}6;;w&]not|{ދ.'{z޽yڃU6?RzԧwO4+ KOM@!:22zBݬ*952_R^H }."/;&oШȥE^M_ּ }68*&gL-jw[umBluO9Ĝ\\#v9{%jgWyB,5$9CBCg^k̏:k/c?PbGR3))!i҇820?r=hvc999sN;yӉQg <:K:Ow~dɅke*Yp/K{i=}LxɻuN$={aRmP`8jηʆO$;fAq2:ztLl]sCK烾(/`뾦/y|]ڇ?~~.J5dbp4,4Ӵ=JlB ѝބADG$0ؕ99/p pPrB҈H!R8s\IJ9VSNNHsL[L'DQo@0ͨɬ9ւRJZFVN^AȑӉ8qq}uυn=u'nPKq=.ngwᇃ<{gyЇWLnnxadѵиЄdTYGO'^ϋI[jtkY{êڭ C/ }ϡ4蛘vBDk/u | $,+YYŐqG [2i˜\#Y>K0BKlb~ْW+dS|4BPU3Q8YU]s^7W/I?PHؘxƤY+%u6IjvW;:;;.㷗wqwSȣ#/?PD . q % 7هw/"%R+GŘx$RX $$DĊPbC_2oܿ@ AWo:8%5 >-}]]]ߵ3@r !7FΏ>ثw!!"4< &o",8, :~a)xï(~)*U@}B"=t(E,L V`7qʸeG_Jd Jg8|*6x^̐э9ɜԃ2=d Kf*"K+;;VJW88]\]I}~+JX!3aAHdTS $} edv(J(+ȩ hjjhы570Z76u42ii`uzVr:8oRwram}zto ]?! z_,o܋VIߧ\I<䙩s5]v\$gr韣*+뮨,t&ε^L#u&| F.ѩEɸc~g/8^+56zm2sG%ʲJٍ[?G@!'2F ;JRg@RCK#$ A{8(jD|&Ê3| (6*uՃZB H~>d6H.~ +Öcq{pŸa 7M5 m m F$? zWBI~yx2|Y>7Vp`óc?ȡɱɕmdzM)d.4*XMARDXHtLlCeE%Ies7h\Zs$5]< #yPZqحىvsqnuޝ>O _L f 3HHxqpCڙ=Y9Y,O_K[.}!"oܥ[5G, W0ilYl}VyPOiL}ܣ7~vyKWoᇛ{~0Y:<8dJXxSqр'A*MFi4^2E*BH ZY` >(JYP8:]p#=3eaOc_xq:REɣU=Bҕ@Rߘac`& 2yȭa,,.Aϱ9S[B);_.P/&@xHdV Pt:,}CfJKIηJ*wHZGt< LLͽ,,C&mmGœ7]AE{z;bn"ws;^5Y_I?-C]=1rtҳNjRŔ_)^VM,pscV6ۣw;iv?T|qY3#_x I ۏľx|p䳩ƙď/J_Vn.F}:Tj^} ~5uՍs?LRDDCʏ#[[KH$?Olmmmm,G ?Wl c{q6OO~{/ pHYs   IDATx |UտH 8cFbU vZQjJ[lW}m+,Bj-AA&eI  w%;{{Ͻ{>^{}Yg)6ED@D@D@R@fvM=0;" " " )N@NXP1 " " "=;" " " )N@NXP1 " " "=;" " " )N@NXP1 " " "=;" " " )N@NXP1 " " "=;" " " )N@NXP1 " " "=,!H![E@C z獝:1&/+835Fy2I샺ɭe!k D—lSc׏q92T\hDD @W.n4ý?W^Iq&RI#C+Ku35UTT"`py͈Ju(bk@R81qpZ@*ׄ 'W`g驥7J=J@34'7U{;^3F]|3nPw_yNFvH6%ι9+?9`ѵSʋ/r l]:; *|t۽H-p sFQWPkHn!qβF[gt)w"E{wœ;bo=,.۲ao /{&z[x}9Ѓwzuk?ZZNzE\)%D@GHɍ O/~9 4@Pv.Af3aEZ@@O48*v.ȭҎg'Z8ly930~&^wyϮ(q=6-7C~Z(T?S_lj6@83K41|^x)trΙ:}~[@oos>yTv48D*$<'gM[}?fB7=cSAUmȜ*yԝΨPxنڪUisKKv,:6@))~9֫ ˊ׆DHm: 4>(j7t*QƋLDU{!k5?Æ=edI\мt\PVN~>*t햟߽{f y0;zs֕whu][_eh'ҫ,%E@Ρ%i̝Y]0㾆i{0}Uz^gpa]/K~p|SƎ=90D --#$ p(4cd_М6wS 4yzRiO@g9&|qg5m 璊}ꊃp[}imw(㇭\sΫ<ܔ"NXKMoO⠗\q+:Lw8JfwZB) TtSpyUeT9xNJ*/ʳoﲕU?64l@APzo.X~o3yoŘ@'ȴF>#ZSmExAMG}AP8Qq5z\wE6G>)QpG@hgrMz[Džw8ҵ)R\tQ0ʌ&ٝh"ɼP O"4}4g,z׸`Zu0mц'N|SۋMp ^m?KR-$'՚@ ?'G۟YDG=[@GqohG<+5 F2x+Pt֓@5K(RkΒ2ezCu]@Ac yki9[֋wxHu5&rR]QkgiYUѿ{EDC/H$]J$}-" " " wYM$DW" " " q p'Մ@" (I$}-" " " wYM$DW" " " q p'Մ@" (I$}-" " " wYM$DW" " " q p'Մ@" (I$}-" " " wYM$DW" " " q p'Մ@" (I$}-" " " wYM$DW" " " q p'Մ@" (I$}-" " " wYM$DW" " " q p'Մ@" (I$}-" " " wYM$DW" " " q p'Մ@" (I$}-" " " wYM$DW" " " q p'Մ@" (I$}-" " " wYM$DW" " " q p'Մ@" (I$}-" " "YqhMR~[." m A<"r 4TJUI@X̙o?N~@puQ<ȸqp[湂[k51RTT|gƍWy4hO|ҥ@rTTT$J0"ఒwȞ={zhjBDty@q1˅;1 E 'fdgg3{ݩ.//Nx{s $ȁ"TVV:⪸ .E }&L]^^Opl&tjN{NQBڅ~q饗ZmqN pd, |,I="pUUUy5aϫXOWDKCi8|@׼U% D;sX^c'bQZLX[΋|oN2c vA&^\Nc I@8??[n:u9" -!cb"ӖN2#tU$(E:J+p'77Urs4C?IɾI$-(F޽{X nVwQ1fp\avr ͌m6 m HpRA@ZP$p_@SAP :$Ta;fqWD@D@Ҏ3:," " F@NY\#p'L@PnWE@D@D (I;" " "n_H; w;fqWD@D@Ҏ3:," " F ̊%jkkiǧCr Fal/qHp={S7tӞ={R}o;nܸ`[3k׮M!LUUeKJJ!ցG\3]Bt뵋Лi3x"U/̇mۖ 9 'iXiz1 o A-;S̺5]M*0s2HOCp,PHHH.,IuIa fV2P6=H='ANiIؿ! l6urR$0yI=C<{C =,ᎷJ',@2I4큪?Lt.4/g-,&xlmPW҉U˫V@[x! `xZTb'?pUiCOjQ6]6@|R))YʇXhʹ@<7i32 $;e/i+" " "j wZL" " " E@NrKڊV#@rP\" " " &p$ " " "\$@ (i52 $;e/i+" " "j wZL" " " E@NrKڊV#@rP\" " " &p$ " " "\$@ (i52 $;e/i+" " "j wZL" " " E@NrKڊV#@rP\" " " &p$ " " "\$@ (i52 $;e/i+" " "j wZL" " " E@NrKڊV#@rP\" " " &p$ " " "\$@ dZ"5Ӻvص宗֗Ţv%Cu-^ck|1W[} Td.|5!u`_Zb9i:nٌ#rj{|ݡCv߾}K>t6#z}~ͩ]=݋ ʹaqt 2||/ˇxT:]vu|/Fˎ8әS۽CP.7\p:zre9(qܠ w:C"-q3ɟ8{Yw-(8묑]>enDIq'~] 9cz2w7?=q3?{ܘyȼ/Ӻ*]^I]b$3clC̾]#N\G~"F'uXt3~blEc5e^ qt_3uk7uuj xcrdN@kh=.^q}?﮸y+>\μ]FҤ~@k]%p˷/6ѹy܏8]X3W΁gyLfG"=*wWpa5Z6ϤMD [p AX6͍@-:\}g^5+1̒# f~ox[ǮƑ}ka׼w:q̳+Y{4R_ݯI.}ig]4t; v?ldžkbJ CGhDێcddՊ;6u#i Y^r@N-;"B}s{D;o~eZc?-2ya)6f{dY~ \ZJ!rFߍ UA7SX*qf??}n┓;/fUorKZ'fu[>&q6\OO뵎mat8n̳ GmNC9 ,^ʷ/K`flvdւ q3D_94r^oްص 4B N㶬zj&'ֽgd!?8#t4'u]WG$ptݯy RkΦ^;zspyhoi b7]1K3*/I .`7m\j3<gφ;fu/ثˀ^]ʫ7{@A=V`="-ټ@B}O<6ah˚-jMT"wjV -!xaU}wf|M6sHU4Vs{dhWC|Ð|6-U#GDT]޲/r%6-ʡCd*9%M_ɪk w߷aP{eEnLu3\vp O9UhӱomҳJ6/֩lQ+vׯU++n)^GC^Q79}6>煴Q;m:o鶯 =슮݁5F \ }dcs7)G򩑌L_RﶴNic#/<t_1v?Dq~ :ıc>'_n~:|iXz7#l|ΩGs3Y_ _ަإ0˫}Jc{$@KK&*S]Ӱ9_As[oGCԔ77wFJ?3ǭ$߸-y.Uٱq/W![,$5s87T~!PsPy1ݸ:1%ZOq$?XUdE IDATիw5砂"kox5;+3]H4;zDPh } (H4;zDPh } (H4;zDPh } (H4;zDPh } (H4;zDPh } (H4;zDPh } (H4;zDPh } (H4D+kk?sݧMz!C5e4n`9y X!:hv'vV" " " F Ý;v֭X}Xjװ]2:u.UܹsՈ=0R w222233ՋKScvO>wq%ݔ G+t34if1k(?,m0xbu84ݯ_n-&XC45DҡCcǞr)eee05554sϽ /|+DƮMlQQ _CD^^^=9,b )Ǐ?zhm|8qe]6ńظ SȔ'aٓjT4L;`7ҥ ~P_o߾}Y_VG {l;  m55Vf kcزe35K/t Ax "c fbq,dqp 6mzWK/Dqu 'O.W)MV$]v޿ފC,G1*'j/;grNy:NBNe!*++68'q0M`"P"ظŔq _WnI``ҺH<=yQu`=yY4LpaG8vQ ,Y2sL?e9vNp6؍N!x9sx W_}gjzJXZaիEx:E1%T-.:n@0H !#34 w|G qN^-w}ypOS]OyE+߆?yѴi^y&QX<[gg?҇qμy8OHggCȁD$ZMp#qbW!fR?3pB8ꨣ>csʕ+WXOr-xL'567>ӧ2dիI  .&ZxϞ=#oek)Z~dii)8(@-,,4 \!ӟt۶mq0HkOpW{kf=X\җD7Fz}L1sl:P/uӜwRқ7oI))́pƍ߻wo5'Hs &bx[ēLA9`oF;Һ CJ֓O>Iw`^6sv%y䑬Θ1Q'J LDS1WFH${O?M1#GZ@@ 6Cu8Ηx3'i-Ըm 8Cȁ XpK)67\GqOZ{eyc*%˹VGY,^N9Kɫnf~/"QbY!`d7HnW0́k67Ɂ> Ԑַ*j ছn7I4p OVnlo3f#Sq9nH[bxq`ܸr ԧ:,{ա׸mw9Cȁ;8Wp/v'aX3䌹h4}_ڤq3hp|%؋¶qˀ|ѹiqK1s > !xVpGڸ$^ Ŵ(.d$A&(*]fntpIp^S&C0 \v/ݻCpoX(ou UI K*ˎvN{bW/"\͵&9g۪D{p5+jv͑! qKnKN#E[X i!k8VU~De%Yw8WcKޞ\s3V`18m+` ,"C82J$[82)pSU37zjr H w;7$JxȖp P\ ,8KVQˁ$A zuL5D@R 9:W w;r&ʽrEEB Wn UEth.f5ϧz/f[3\U'K΁{)9}oF)iJQ)2mU )rLnlq= 1[θM5JhF)iJQ)2C*nZ"kBdT ZUKx+t͹w pčDyF۲vlF YM֫!kF5Fm.J:dsR^F丅Oq=UebNm(=i_~ Ul?8άnS&/I>2E/ h_8^r ]!0PԨerdtHArب,i3fK(ڵo[":tkΕׯc?&4i:͌@:RXXhVoltq- (A4ܶm<6QZZQh+h)g+f2.=cV h4-ʘi.iRNO]I(am[zN +W>sq "ˆ̙.;3.Vފ!{1 hž}X]dIrxtުHA?Ys{hh1Ќ~hѼ⤖2Qz7;C(=m'$]pMϰxrb+aqz: EFmNFAwhh:4Z>3!֒UYD͑Yod L2QlK,>h㖪XlM%'DOC*!3B I,("Ma״X6XrP15A^Ϋ'm਩[\TNO*'B7(ֹj!Q9^=裏&:hŐb2Τ9/W)I'gi[<!e"Xs[IPHդȤ9V1A3XL~iPI& bΏXo yӓhoE/rB!O|zVm_ZZz+kVcZDXA3me4t:5R#A#t/ uQ07%:,90a&xk,A"Ȫ bG-FsX!K(JO'|"{J\4J4oT2i\u:$Q"@cjoVC&`Jāˑ`LX:,V@y/ƣ*&E%vab@``FbO[ej B_=㏋nhUiW;02ы9@9 CeAmL"!R,x7JU2The,hżM(EzlsHYhŨA~IC1o[f+ղЖJR'lW4mŏ) ZsHіkiB> T%[nEC(AS)4D%'Xq 96ᗼ![BjT0[* 06̓[1Ra.PؚszR'ZfM JQȤ獵mڶR搥k6yZuq%E"@ㅷ"S1l0 =;[rŠ,?ׄ7A8dQm"dž;:yN2P|J4רTY]cqۆ;9$yZ$M&mQ3:aeȧ~MЫ l)O؄tRP/ PpV'Z Nipz-6);#&ӓ Z(oUQ )\,BAt&wx4DζZl AuF*!lᰰɼU S#/7[S x! 1vvy-&j@F ,"齨)`[NҔ7)АMvVȷ X#ϹP"VOc 6Q  zAljeLO60E xQNOj3Y*$Q V2TH&NOWD0JO:`-V6M,n2G䘱'RCyJK>[|x Xs&e4; PuIHC漂VD*JO = |Pvq Pc6p6y= ,+ ,&~&(,IcZ(`lI mVV3ҦV9^'[[*Y%kz"EP9S>7AE 3*os3}LO_96y8XЇM$o[YGTӠaM Ԡ_,ìU4kYVpf09=GIDڔRz;h5\ߝp ;:H(b# /Ueqo+fYbT{'9 X$,*$Q',j(@MVJha+dPb,mF#tM )Za+֜ZmVl#HBa'ekXȧQ F5G>UxNWOT]WO걾g Xw\UC8eΙMdKyߊKu$ǰՆ2Q͑I<|QR֜ w+l;2SإPA wd h~Ic /IUQq6d+c"v&흔ckB4 _9H9AILDL6[AޕK'{"/ 1Ad-L2l[1W,V6k5d ʘ o٨Q)Q :ɤ9[%Xu &n)D Z]7Q/IՌ&%*&(oˬۢ͡'KSzR3D5g9Ha!Q'e%Br0h9R!4 cK<ت$ ,IPmV1{ 0v``@xDŽ++hCex&ů-O0yh Z1~iUjhjFO*ūj P҉ 5cXWLD)'ml¯%4Qa{gY=ر f`6gb cKSXiMH ⭘(J^o%v4ݴVQ &ʌ!9v9pip}!tf(vĎW-X)tnxT2-)%b֦r6ֲ1M52]8d()V,4t SXmpnjvzg* +96\N(AVY(U[D.]mt/*<%ӼU6oE&6WiI(6 Z[mh6HyՆsX ~KYiVm Ag֨EjXl~<bՖX͡p-<ٖԟ\e#,Id`;c('gk f-/[֦p$EHj+$uO<'*][ς?om5]H k^ϟ{$xtjWH 68x )@9\.X@$H8ڄ?c )w3\~GP˸>c MWmyCرcy#*t糉4Fx7ӟhmN>nܸ1?@|1іn+UlsUwj.aR( IDATa.;diqK {X C-0G駟f3pvDg&'[у⧞zʍL߇:u*@|BW\ᎏ 1sSc4wF,!L?GIvfH|T+ͪ) ފ u?]˰3G1f݀0rH&*s=8++cɊr > w|d~44Cc&gG~ߑ/W]uP-N p 'PkƗƭZt)i.8Y+ dv7*l5"_qر憙!p uZ 5r ܲ w| 2N-XMWqhʳPV U M7dY5k=xf, Xt~7>} Wvq2ʁEĕ,o۴䮡pk!Kv;bĈ .# \ 3 :hg4gb \> /+IS;ZAރqk"m| ćy܀/P!`Qmz>-o+}jªtS tM9?鑖?5KYQ`:fwt͉)Axií Ʋe˸ =_xsO]j@teǁTrK|WPy@H쬟;&ā$dGk/˦Gv랷>Å/ ד1gVCNSa>Q %;7FڱW3oͻ=h?^Nϻ8.W^˞-R8}ٵknxANp]۪.]<32NE&5|&mؾ IFښt\l%@Ŕ%;ā 9dS&G#! x؁ Jq պՃ.V~[qmZ&.& @u?j_[Dt{teN bSN 8 S˥R$og^$P-FpNp3|I"f2p;-w8:!1@Mq4XPA #v l ֊v[X~ 1T`J-p?;\C|;ifnuWD@D@ґt," " iE@NZ[t$p'>@ZH{Ҋu;[8ay%/nc%;,&YST,Q‰ )r0)I5;y??gh57@GLPbc#CjBډH;>-0a?'Jw6pU*.@₹F@:-GN{/=>\K;g̘QT} 7W4wAq2 M)-;iP3.);A7sH^=~jW|xc$u+~B`ix6BGL =Y wY'y pB<*«ZD сLoƂFN~paxɁVr - Wa5kdddS>:y_.l= vƨűɖmsIIA[DLX\޺ྺ#go.)~ԓٺ$_Do]WރΗwv9k놧'ŶDj:P)KW޽Rϕh~vnURR8ι~rs 6mE? t+mt ;_>j=hsr yݿfے1þׅwP<FJaKpKhUٝqߦԎwYѮwz{޼ ߜ9럡Єpq~C=vEqYtB<#Ez!Swްb?J<5f9 `U g}Mh2Skݼ{ތ8'u!+N;@*Ê>W99aBoMȁАr -dax#[ѡ;v~}}OK}~3>|n U3' )5iq~K?egw\q"P2։鷍̂KZ3{cݱ.{3 4-٪8LR: 㧆GxͩK>0H m'BP)PLN;qp&ʦ?p/k&4kom 4,^mYӟFk2{Ef1:{j,}S;Fpew2/(گkWn#H$h)]:(]tל? @ZgF9Jҍ'AS{M̜v@&_<#ԡPCCCUEG];k.ٲ:5LEON=h@ ԁ49ծw4@9%ft<ı ZS.1>_L-_ Y|e*JCvj)grV_p䠞"%K>{7F-Pm-0z׻[<+m" 68DDΆBy+r mH%nUncuܮgNuBs=bgk)w#]anO6ܕp0<~_W`ѤSvx4;;0WѥhxkfD=Gt9XE5ldIZ탢xzn ;ҤoqtPE]}䳩^~uVܽSćo5Ow]߯ɉC|x3-غ'<_RUʪ~vYC/ڹuWVMD8tYў_9}cvxZ68 ^eչ7o&ʁr •5+ *CŷмœXblMLk떯@]Tl|K돤 "/㩋uW-|c(rVKBSϾe޺pNUviube&-h)6;u:~ Ri-jIҪŧN1uI\vWVrz‘aKEczdrJ@JW ;mʄI[=^ڨ+rɁUҖT~+.nBӽYQMguW*z{o(=5K@JQ:*܉U.N A$"'M=U@ЦF" " " !ٝP?BknJjjt?=z%O#u(]ggf|Y jkܥG۱:L3HSQ9x(N1+#c@6u?(2l)VZ;̳W,T]U]w,ټRSSiC]߽'|OUXP5DСCNNzJG!=:9U* v[Yʬ۷GPCx̌PFݺu>"?|h*@r.S܁Xg{ `Sixo))޿:8:%W)Ý.fʮuoq^Cwo'y)vԧ>UP&nwBөSV,ApXt:v'cn({'@_r*++_}~ӦM=ze f:[Ѕ"Y9*& #s>w(!pNNv.r xѾ2vB~;W^yy:l9aˎ۹sg8p nފ]w1''>s=wر=(6JXyKsˣVfMܶm^xA8Zڵ+vee,v̉X2]" qcӭ*xtBƩ@l_zʾJFWuVȟ'&ʁ47~w8MpǼkmphPLÞ={N9 w Y~D5'׼k4Q u :K1ܤ**͎?_*N^ڭ[|h!ܡqkͼi-5ǁ_̃,$n-ۛwPN݁pO@?C If( wp>۾ g1MEE-r=e̙0G_87nhjcIނbumΝW^ykro>GY4 GFr mC.R>9\ˎ;⊅ :Uq {uGqtwala|`Z1!IcrµիWO0og~gZ\".a>K.7j#vOUosǏ?c bڌ@'rҰſ{VbJ[qCh}~.D,۷3DR$:1 Ð߸ٌB< 裏{(**0`e]vsNqKYøhw X>OE]t'N6;xԱb f v-Q[Qjw̾@xp hя~|ÇHȁ8+$E"~'I#>Jr|eWA-[2CG׿&!R?0@vig~];8& +WSiFO{P97swXr΢E\pǚjOVZ;>=D8,olF.|;A=?5k ^{%jf;@Eo;AZil9)w}˗/ѣ'N2oիW/[5 RK@WX(OrJ'?y饗@vjϞ=@|5+jX,1b' N38*͈x[ zZ4R#斝M?awrN:$\yү$9K/O gGwq΁|9Wms5!Ӊfؗ8)-6qJqu׭Z7޸ꪫyJR:q04h*{I^^F@ RiOKf3n:/qwrÒ%Kx&f{Z͗p Ϝgt!:* />}x4sxI#wp)/˼r|'yg&P*4r >Y7p]h 3aKˁdUCԌ0*;GMC׿~Gx8Dio(xO#07uHm3~$19?guf֩{+j:½r ~:5ِf!=f}{fW썇77>:+0@9]+$ı)4VA45VHc`ziwL+x@qdUMk w|\x+`+9Ny晼Z P4\tMÆ 2o6@ ȁ́ufW0!Mˁ/8צp'3eg Yx861Þvk_CG.^?'bd+}D9]vZ2= /0<y[#~ pEw4PIG,r m39}5Bq=,1Q?n[CJ;"dE^6y}sn!{&"MV~BSOPz"nV᭘g(=ES7DҦr m09ޞ}h:hrGAߵywX=S fi cqޣ<_W*tIWFNpMrX|57pGia8Ϛ 0gΜ74+1宸3 9. }BՋ@ DALr Qv'xMߕ(sp'ꖝ(A&#;6mh:j*zp J>eΆ |& n#d2L r"  is k3<`D'K͠KM wn88ۣgc.2 IDAT\0U1[۲e V{Ņ1=ODDA ȁ4 /k3@uRQtɛA۽#ĉ|_裏-[q<ꫯaС_=~ 8$0T ȁ26ݐQ_q+&p'9밸4R,^E{y4WExx' ms`N1--H@ _ź{VfoJ)$aGәqO<,]Z|7"\ܳgOeϥÇe,B=84cZƑXgǎ|mb2yk3r qqmJN\qfc\Kx>R-taeg>V.nV\g}6wN%́-׭[yw=~po\?hD<?s_~\~&'|臈[ LutX=O?R?xBJR$@p ^H o9\I#p]o)W9A-aEt^@]ZjՍ7Hgsr^߾}ݓ皅xTIF@'l7rqOˁ$hnY,:œ;E#a |T]N^]KY[޹ewW\qg0޲Cfr5 2#Iթu |,۷ W]uըQڑv9]ju۶T?~H-vU+_wK7I\.oz~!KUe9BY;%?oEz91[y[vN^l:,U{ꐾg MwE`JȎoQnJ9aR]?>\\\})aN炤s 2u;mpSwϑiLpWUR7>|;lQ-rဇWeTWdߗY>贂̝e<8z׮lݺ?-o֭\Zu4x@^^_yg0BN/ޙt u>YKf*kE.l:q}&ԹZ ~0,9Szf6<>@AzuC0tJNAGxs&_I2£(UayΆ[k .9c`^ǔoaPY;,rֻ}ۡ8(nG2B5ښyB{|Ynڱc7_ܱc+l\ZO?>by >\{Oz&HSաs^eN*nfKd[ |ɷ~ۅ'٣s8d i rv+6wԬCJ][eX S_+F8.{>;%kct[$tdhPU~p~]s@CiqrS~3~cX8s!N GםέhTVVY`}]n S&F#75OCYZU}'ߵ;|t99ٹ:fgp4X2 |sڟl]p4ߗjvY便X̷;d$NRҁԄWLgy giF;j&H̅*O} (jK ]zH;y;u%gvGז/޹s'vlV?x{N{KDkovދ]r**weW;t ] L fhM/^~ׂQ%PeO w@vʤ]R>yw`}t\RN؁d#Lp Ge\JJ>^|S?U1fg};i/p=C ^N=uʥ/}vr޸yTN}pN{/u\D ?9/nvꔍS@Cf.)9Mՠui˪h5eɒY2HR7&wEM&óL؁՟e{w-\ˁ4R<o5gc2nBNџ]8k/6w_ep&۽gD6WU@gxdtS'AVmXYK>rXf盀9澿mXH$)HsW7 EOqǝ3 y ĝ-EJϰwGr 2oUty=B3/kIS]{W&?~Sf^LxzVۏ>Wfwpyb{,H:ӇqCҿ^UQYuU')R0zdn+ڸk>)%m8wUS[wM4y 8s%캏,Kh(v_Enx[CHB0 =I=]mvg#R{p5m~nlXc4֤.C7~\szÝ܍{>2622ؚuY˲iX;RVE5._[Ǩs>O^fA(+9{mNhy$pMCnXe~FP;̪j]tcl. i%1brx#;:Z1ތ.VYtVXԤ*"@+~%@bi[ؐNht@g41hYх&@l~+fA{&=#Gw-} ,@J胗Q#Bbᚠ %@Z+إzH+kfk1&sfO8Q;wٰeTTjC\)8(N3to|1&tiA hD>tx`Q"Ur5`VVK."y Yr8cNԓJ%$:yWD]R! a$ӮYŴa˖@oD:Wr&P5Dbn"f2dܘgN9Bٜ4 J8Xh2^ %tL<&yivJОeyABӳ'/뎃;vf|cXNWO-C.mMgn-gIbqxApiqVe@N9iMOGZAn&<ԉCӓ<젮 Č:X ";/ ,Hw]bk"GX|~k=1o+K}LkDc;úQءc8M`vBxvV3dC,iXM˥`0=uΑ Uٗ2k2J"h@*"S$g0 %(8%M8t,7 Q aA}YQzjͶn\"x)8D3#5HdM_ $Wx~[aDSn r @ƌO"@b`bNW@OErAzdGۉoeqcdyYiNWP)nV>@A5qܠ!-Cc9 ,uk⎶A&BH'56ҩPG NJg dzLruQ$*s<;Mw2ml#9=1KRRE Y]U!x@BMpk&>B<# ^8Ch61!@< E 9Kj;DG)iaH lc &h \3 BhևqV8br;= ^7bɑpC>_3NgŤ4)b {3r<;դO&@\u%XRi/hyO&n V5ya b͝N}VHZNS(‰VM:zPDa['[ݖȫUƾg8FԇN9${,QEI'd蛮=v rz5fA%K!\fqd85%pCNKh\atW/@  *PWѝ)sqikD hOca0ˆB:pbb޲}OCÑB\u1f<' 34=|@nkL:]c>_ xE:m)^&\GǗfKߣ†@mcA}MiQ*,Yu$Ԛ 9SP:?mj e% JtGOa,9{v7_BZltZm][voyoz;$:S<&,'ie7eXS9C݇M麷ǩ# ۓ9d5Rnо)=<9־; ={&$ wSP^'kUa+eˑzWn|bִ_6x _?m{Փ O<|~SK7(+.*7kUꗞ _ #Zq!QQ>F~`JۤXJ0fCBmHհ/_mkV<3ØڷiucK]Eges *Et*- ]~2+\s Y9fv` `WCmeA' mu;\hʝ^}nLspC[opm@<1㫎;5>5RjZUc;+^_ccؼ]3dd2ّw|Om/,v5 =W^qt].ZwՔh#ua`J=|Q>`f1kH@oH67e1ClZW,"@H-Sɛ *!@B.4; [PtGK=GVf 1Ik\ :F I[1^!4ڢpgհA+b1WtG+⒢Bj3sz}Xxc`>+EZ5Y=׶dy JHI7')K;ò=e:]y^['l %k3xFwXZ+o1)YL]SU-ϓ3e<015AFʘ*m}o),YBŵj,h ++e G:nXy# *޳7x{,(w3}ˤV#O٦WwxM+r* 0J wspC25vr>& 5au.h"A!C \j>ijr͵"&bѮ@vͳЩ; ue9YND|:UQ;Z쬟rgsȦ |Wk"3yDեrtJ8f ޻~XgTA{3O/t#Cڬ[v0j|WSm5YDv1Ԕnå)&U>"/%pd N1T;03o5UL9u] +ZCiJv@ۜ*@Ck$hPeo(#嘛E6!3:oŽ3{Y=O?cSV15?yqڞgfzci?SOq+k6{7'k~iq%h']y($)_0|fÓ~#dwvPlr9s̱pyl>cFʬRTST"fղP /OH8?;r|7=9cSo+eZMdbGɑvz<B㱢n,HT&52jMtԣt`6gL6ݲ3syG8KO7VNwvXsmtYhvq IDATǥ/YN~0@x\޺ >)];?ެD̹0gDh045x6l`9X2%eL3eZmg'q|qj+wȬ0Bo34m 33Ik[daK^#N35'9FH`gIҔDƇ ڝ&E~,Jc-E/9TWNm-+޸M3|?0}udGZW`aq%C'=,m|JĽw]Ԋ9vͶ}̉/-\֧W%+ѩ߹4j5q(wZw41128cT*oP@Xǂ/mmN_]+iYk u=/;Td{ HrHX5[a%_UUUPP0tPIخ]233 {vAv ---}y徊:ori}* 7&1-||k44z˛MZs lY`Mm]eeͱj=;1Ӧ-.WWԑ#a[i>Zhl5ǗS?NIvjkZϚ֪D %bGLwbJ58MꗖvkGnu!E|UVV:t褓N7oիWHII |>ȹ}շښ²cwjs+_[뫨HUM9'#@+z[K|-%@؂oCIdoR܉=)8a.'55BG7btfZdV /C"Zx-_udWϻ~Z4࠷.,*D@#7 -$R$@*_wĕѣ_JrD(hF AW?CZMb!+zݡC||f 44ڡ4kYڭFɞ\i)ޔT‹U0P jЈfdkemMζDeUUUeGQnsbMmtiD5o o@;x46@ف}eT8l%MHxR w%$%Nw+k6w@A#lm,0*jj=VsL^$kF5P$!!1d¥DHϙuW@:<.gcQISeD& '\H m 4.E7Ie Y$[.WK+S0AeLkvE2XMMZ ÓL"4ĞZ rٖW ViIVk{V@]fiv0}P@Q.\}bLgF* VVT>jf:pp[v,/׎ ¨4%D#g.)^m(4l:aFr_F,~<%d ܇cT =  D\[ _~AY~Q9ׂBZU@J{u ,h[\&FLDn$p"]w>/#Cl';Cg%>i_4ou lV I=cZSL$HVq ]^w6 j{4: RIĀQS#H(/uϿqhIaiA2Ғ (HBLbJ4C*ZtKaLwƩѝǐ {^٨ إ14[AF6`W0 M|Ux [>S#WZby,ZL5(5LWh4!UUE'8fT&fKڀUa@APuSӜ{+F oH" mM=yy'Um98jDߤdM# MP(e;yl6"L aeC)xpXڙS>^$Y6^ `. ]u O/]vևQmJ*bF=`Jߌ6PR"<~މeC)Rt^7))ibvey '9ۋ⧌;S Ό;;dEϷfnΩ~o,ԇji'yrcLZjH~^o %99ԜʊZǥLv,KtCQ|iN6Ze5U ~{o>?f ][ LrVΜm?S]-BtiRSS3%0e_?H^|;Kkf1 [925h@, uf"9 9Jl=iKMMRW =Y݅M@KMuuE.ǧE_<׶=iJK4ţbPwa᱿т2REMB̓UF <}K&N4e5Biۜ>l':2D@ʱP2}CWL `:Z1ުͥ=wVY#8zE4q}j_z熥g,;cn6iM5kOsټ!;CSw x]pxS*Ӓcp펐&_An u9qҁ2UeT9~S FS/@C{}#5@!>矮ZT .;{[И6n^䔀U{AԹm%:|^Fē"RKe޽sIKK#|κW[H2 b 2yg_~uBgɻ'O<6ӑjx`-n6Av~ˮpV`6t2hJ=CRYYYUU%5Z&?=pgܹӦM,94#ed`;M{ *DJ\NgȀ [Jy!7нQp2v`nKよ%bkJm|$ugqe]&j=;rr?m،D&oRңC(zTKCp -^}CqJw1J;Ҹ*62 t'xBDUW]5gILvܸq999=z)D`6y8h &G@L:l;StG t4 "WvW":8 Wڷo_Hjlt8G1uBqhD4H ѣGo5kpOGt ׯÇ3Rxʯ}+UppcdQGEU'=ԡ\+ Yx-R[NtDY3̙3N#V2[ 1L2H $(HOP ip72.ڻW|FFLhÐ$z :Qң mTE3o[4ڧ43r6躖/_>o<4G|HkSĀC<N>~i feeڂ#G>ChbBܐhz2, 2MlO7;RF1<\ ;2 u'|rc͝ځ@PxxSPa'oy eDwXB+ĘB\ @5HsD0qJw"9,b )+nV3AT)~rY0G*I @X@Pg?{'% AodP xa-x(",qt֪U.rmC0V~^a#zφx2 8IeU!h@Rرc]w[om dsܢaJ5|li $'(ЭpVR\|%%1@x~1,k(]uND!۷噬*' O2E ;pY4$FDHE"K\{Eig?O #2(* @}kEL}Ç#7 :8&:2zHB8fU|E@֦M.ru]uzW\'"WX&^h9#\F~#FrXi u,Wx*A6;X lݺu;0:2>Szx* U2`+ ~={T\%= !&,Tݓ:;v`sCYg_Rl NE!`q8?IREXXĔϗJ^!N|8bQݻ:fd?Emu.:ʭBDb_s5|o ?H D\ \+ #NˠR +//9,Nկ~̒uXbR)PTD " cΝ,Ώt"(|SN9o:2^G ˖J N\gXk.=iҤGyCr1DS)6|*NO?SoРAH a[-:깅PG F"\pvh„ ,5|B `ʛ_yc5ɓ? :2LU Ewb9rkYEO>{ a#SJAP!&".<_}UUO<`9BCuDQ="j2ˊ4bg /pƍc u&/";.E/‹PDs* 69BVD".6o ," z!&͆J]ŪRkF@3UVVş9c=VkcUNEqΒ%Kj!bV~X~G,;K L-Q M W^^@u>p@:H.l -*4@4" K.DδiӞz6a)N4J1 ~:W\qŲeC4W:P* . w}7&}r[oe dXGtliPt'eDf-^é~ D#KqbRD#"+8nx7bS@pC N"Vin;M"INN\P:v׉UIWdsXw ` .;쬣arRPt'eשkydasXX(^)tHW(:k׮Eo>rѻwo ٥d]pU@@*N<S[[o~s…C+ύ9,ǣ氢DUBA~Yf 3fO?=j(d;&sUGʄ{jPt'Љ!7̸y}^`"1bK@"`F.g܇̟?o߾ 9Bq.1݉E! jժUu]6}G^׉UiUeB}DPq?Qۼy~ubڎ+tEBvBq¤ X5o=z7l0~xɓY`$Xʶ/D(X`ZLt' uPAuy~" & p2uiFuS!޿ҥK㩧 Ybz#Gp-}!C\G-ŠʠfF@Ѱ_tԻw޻w/ 8q":g4iDS\eځdAs=\9BBC"("0 (펥ǜ8;˗/,9,?,8Cf_!'|믿N~9|0 :y'?Ad!(D⊓Tt':*1n%cև~qƙ3g^DGT*Bvܹff/x`3ַ:)arHK@fi|5 IDATt'㶕+W7~=NpUW]Lv G!' pa[Z&9X}uVlPtnju"0BT2bcPM1R6떥JB <YŘ:ydܜ2e  DGACwX˖-332"lƌ'tO83R@# #\V3E?ߟ}&JDaQx@@ѝPAAB&e :Ceag+8EG/ɰC"͂޷'>*MLc!"^}U3@v`&wiÈ5Y(͎VюoNa¿۲wP-FNt4;xǗ!5Y櫫Xe3gzR#I.YsƕLH$xmĠ[N85 ]rZ{θ>x:eY^HQE P'Oy%xDuU !Қj*~ɐNN+π>zxX|nמSʐv5StFRWN㶢c5ݏV,l ^:-;!(ڒǻ~,_꒼jhC ;lHg~Y_؅_>ڶy[Fo̯u+mE҂j}/}{G qzp`W4lWWlG6v&+4ɠrٝ.Pv'3zƥFe"":P&`Ksw|vgr/|9?%"rд/Aߢ-71k~/Zd?PeOd\@meBpFϸHV16AADPQ-"XFQjo|Kۃ>JfO' Q }"mݍ(zLE*DaɈX BjND inT`A DmB#(":Tm1&<ւ/9B1`NWa$ڝe-EPUBw"U\'2ځ`qQ (7 сH'M<5pfݱOj?{Y-0#fV""\7dP&x]m h=;Y(MKңm47Yqa z`7 WS2މOϬi7VkeΗVT-uLwW!il M; fӎjͽsŁ\WbarjXXfM%#w>8۝8/Ͼq4-$Hu .  DDd -[]Sj%݉X9͛7^;$=ҢǠ̺ -z&[U̺iڇ-zW̽LZy>˚|m|S/f揰{~gVO>fb6\,kߴ,ٳ' $/u$$y54XN9l;'ضlg0 Od]eeed][*n'FiCk)[n5OLLxM6,۶3_jD+ND' @ C ~D;,MRI&NOYnǎgD=rlV 턹Wڝ&;~_}5K^^caW~o|:wYdo.$U-|sNn{IȌqOuՎ|bH3DY,]gkoy|| K?VwGTt|8-i~Cs`_!)l!wwkIw䐥P 5e7K~'q7IRܰ|vVhۍr;h #DD!n30faw Vfuʍp͠uǷ;u3 ͯ`=^߄l*\Or0Oә?H[)v;LNhqR{wg-/-xSkwQɼ@G2 %dF+JDDƒi~PǭI>YٝOmm P^`l[44x{ΛG#8<}frM-}rsOvϻi*Ta9s<{Me&IȽsG;~\b"X⋍6{Duws_Y]b#`ډ(%"u_C@MfY@Tr B@!5QB@!P(CYs-8&b'Gɕ[3OyaU<)d6*j{Q-)cT?X0h ;aÚpˆ y8ίvIDHСC%Hu'iBUWWw)U֯ˈnQlܩ2jrs$h;>f|>X62eTf86ތG}o5S[ U02j#1Tk%Qv`ӛܫ/}]7ٴ/^߰ξ CH쯫ΟI=J7“؝j TQfPBѐ`]UE-E{6OJfw9z9Jm".߉o&edr`t6Ю=Vrd %7J],%ݹ鞯H'%B~54HR?-Z|w }:~ @HFhH".t_t-';p'@b!FdHF.SNM6_;֎8OM6zhZj!xRFO3n$Z%Z4uF9\H?Q!uo޼>Zr%#&L8q"9i!8A,"" A? @w5':b?)Θ1I4C xo)V)x c r'!1{ףv-dT]dC~3 8tC~_̙3G}3H%T\b\ TIT?ЖGIFYYk2 ;x S8㌓O>Qq(MRm4"@=*!|չwMy]}xOpDt 's頺!DMQQ%,Aw@ZBu^Rv@j%rm?Q^^^LG[ />P3i?\4wsNM*&"pCzM`+pw_t|sPC ׾6waÆ$H &s $`*dRpRLYsړ_]Z>W*ꫯ~ 4{3gסȄ3f8Zi*vض!)P)'|wzf|tP N{;$@ʵZO+d4 +Ma7׿EnF9jijr!@Xt/3x֭[鐠>MlcO=ԫ QH&E#B&&/D \s/+(FP4QX(rhLp姦'8b4GBDb;v Xy7vj *Rb!TP1# =o`h[d $PluG_@&.s"aՋ/FCD9v,}X Ȃd$v]DRKf_W*9˘\x@nDW?_ฏ3XN,f,zד`<OKKAiU9kѝI$N^D7KI ^VaU`؉HCj{PPPrCp=OaĤGFtH:jP4g'Ť '=(T ?oCA;a0CQyhx`9r C]R@/R !$ {iao~Re1Utǀz\єk\I>z#VCԦYFڥ@tpG>S.1xϗ_~O# .k *Jļ#kw&PȲ83ϼ N?t2$rsS5GUJ qF(sYT;dQpR^Im|Dѝ)wK;[l7o-vT/I)G&ࠈ91e CaTHnzU2 RgR=HCz>|8,+_ ,9bcVP[8Pt=+x(\ evG#I> dff"ʈ4(chW2hT(D6V<~饗8Ucv( q9xN&=B}rGx Sbh=Yub׌ՍvC [/lAfι a H 9YN=ёI+Qa@A)\{R, Dg5TVB)Y4IP1 Ew^;fsE#UQ4]'P^$zUusQ3egnBBӃ}C~vM8ŪF>EL1Gyܔ1 2f:t{:G`q+DWp#[]8ছnbϟ#e ])wϰNX W1#e w'VL^#HY a;V+hi,x:,>' |#l/`9Mq~=,|ls2Y+P*cF&~)2 p<Ո% 3ay k2i%e|HU+grc`1ҳbd _1(: s*ܔF `*3ˁr r'O4UYCQAQP+(}5Zi*cXz66SNQ;YcA~!ENTp>#X/K@h04a<UJ >=iGoC)[J(cXhtutlrt(LX'I8-Fcڱpуu@hcK=`EIMN, tXkWH1`"qAv ݁#OY܇q:&`{"KCR:Dl۶#XW=?,rE T9Z oU O?47uHJ(2}nL9z( <ow&fk{)8Z׌8Y>w\{ D肑x;r}]s˜tdLGa9Tu3Q6F *OGQNʴ 8>5:f665"0dmxK R"}FO SYEA 2 ̍r,%6涮*xkQRu",6l7x֬YPy"blpـHyuVyG~_CTF8EY:ߕo#NsLb퓎 : ن痿%+r |A'ÑɪFI.ڵ-1`g~s~Ň\[ aDsPSr:0H<&3/V<2ߤ+N5%"K@&Ld(xHYO^Pt 4pc},LM30ꤓ`~pu£]9yPrwXa svѯ{PԳN #1̦r1"bT- <NcX{,'TH?m2ζ ,cTTEw:H: :-YF=aMo$Ն, Pxm4Yug° zC%veLfXRΉ0vg]̔)S :bcQ#N:4IX34;ݏyHAGͲ OiҘt t ݚ Y ̃+'H#k*<Ŝ}TsHP0&KX"91"pE{"&I+ Y&> $+0ƼVaIg}6݆j/J} =+wp"RsNs.‹ 8uN~1TN9PYUW^= 9O<6d=~Qoe29 PѝakT0&`A%}f*e"B)VI C"j ̑mz꩘vibrsǛ8rDw2Ergdv,8h(!YQW/Qd+W۷4a7.tR +=:>CYnRGR,Nh6 : qXHȪ+;QwEj S~K85)JqT&v'LZ.XڭheeN>bBe2|pt-Wlі If({0g.;W`N91A d4sLa 9 B蔪D H5f!X@>bO&FBw!*P!NphTh:,~a6=T=ܘ,9(ȉJס YƜ^L>1Rƍ.TQe(ة*H"=>V$1vƲe߾}z-vL%- ~8Wt'Z7Lz dfZ_~L:X7*eFGt-c`A3gιwxSr}Glۃ]3^漲 <;dV+@JlYres.ǰΡHb9chxU@ztKy:1{07B3j(Yr&G55@D#/*KۡQR&^ѝ(-$[6`; hUV1Q7I2(e_?`Ӷd =.GR+NMt{6Rhx!@5RV][U ?]Ht{oq ۛZbb[`6A8c%vL1s?<srL7Ɠcb$# o$cl Ю]ޒZRk[j\U׭[ꩪ[Ul@4]GKdyLAlO< ;Ls.n$Ma@p 022}MAKf?q Μ9jm8q{{1!F  z2_[\>74z݄՟GA{Iw}W@Ʃ&h$x/kx%Zu` h2,d6X͆g! $ۉ2>_g}[M O~>`. c4LYT6v\wU54]BrtV^e1NONsneH1g[&Eߔ#P']ofˍW]ל޶NGb<,ʨSqL'@]~'Ɏ(c͔o43 7\!L8A8x[ ZiiilټƻlҊ)>LyW2=PAar=.[u\,|l!LŭfkFWS#xE9ք΀'w$wNAI>%ڔ5-ٞ fU>=IΊr$ :f[ KJZd,y[|"kVS#*3hK1.r1ĝq?"^;HfÅ%Ú\_tyD(;|Ax nxhYL^XfvM_\j,wT.)JN&ȱ #Z܃Ç1Ʌ+Zf {pEY l6Za 3?.+t(1@ ZH` Y.(pZI'G .ڤu` mpA!Թle^NlI`f-l:qv ϥF,rĘt3nA!@٣r w2N5B6 25rؤu쫩sX"ZXcF{@&= i;jb Zj9..rchM תfWMiGo'tctJS.fL ;6JqVnè~8k9B_89Z[`vFr:Ur: IDAT^=69~5{\5PhB=V5|O+ Bك\̣\#[= LE +i6L751/n|E:$7tJ'U?/u,Ƨ5—ҩ@5@KgٺfLEĝ94+Ĥ>.wx|eU4…` 32F[va3%Ęḇ&b̓\<}Tݜ̋5NrƂkNՋZe7af?K}`V<{TxO`kyy9ܵkQ[ CaE Py b@=Avf=z`C->{`Yz:=,,XL+}N7D3|al`uQ&-$%!ڬG*{V~Z'퉑 ^8c84*UƤdd욃sNNd,.)X6ZBnH WUQv.PaF͐# rϱc_;sA3je/ ͛bӧtf,[B`\HYC)7vr nC)G]qudX=kHNB*`г&!/2;j1§]XʶuX"4`&%WK6}n~VBh`:3cz\TFmj}ZRtj1"ч5*|w]+읃5\]p!Af+ؒrf m[i 7ɩV6B{ 0?܃8b@sIBWtjSĮ.Tus2L,գjVHMq`5+6GA1(ĝ^C{2w=E}0p$=ؿխo}v{[$}Hi]etwFਟ {Z ظ Us-26 ڌnNM ^O#@VLFǫZ~%^Kae"E*\|QwY7% MШ>'3O> ~^^SWP`>$ L!qh\ 3jf\=ǏЃ\S{4>>+l2KC5I'x#sry3j<fluywS4T  togϮ YI1O|mʥ_@+ rvv6o CAt94Xr(=l @\0A"g\ Mw.+=+q,v$to"9&Hl3V3mFr-,8gĝTs[;2B42):@Իo|od SXCswb ?٤U6lV=PiFLÌ!A<=pX}إ3l;pPc&oii,"m$_ Fu^h>,_>m A{'p|߮V{$3$̚>{$;SWOzK ;H9BJuk=LAM{ɳ7˘7mĆwvy˙9%ZU4G wRNo^xJ/|QK+}[GJVtyd -6?slKSҶh#ѽ]Q\rsm [Y;~*;2&e}Q'K?}dnr,m !uN Q `W8e~.ys w]na~(`L ʺ5(r|{Wn sT19"WVT}hS*XȒ%K-[Qj㋻۽aF g-؈tN1zE]6vBG4F&=ֱ7r7ŇL帝W<~p=Lʺ=͋o^T|ēِuVR424WZ;g'ǥl{9G? <;EkL }#˫Z[[2^[a%vN]냖ϖ7ⱸX3]IdV8ˀI9+?mܐQk~A~YiWh̓K2{ -"[9S@2peqOr1౵K?~~˘~C87?R..=\)){jvrr |dIqك'[9..cn-i̬Yw`ǮN}> z('XnT)ݝt骵n0t Ԇ.nŒ@ s{ 6aE ~e;!nEIh*7hq l$:V]/>>K0he=+P muMA#?/'[}'SCll4pf qW)uB-GeTMNYZ=hAR^P {vCS8dΗ7߿iuDz~]c, [<!g|#'!8T8{ϗ({`:CacjCZm%У,{]3 [?i2hבxL#W)86nWf0@jpnWm_U\lh1QXg3=IwNg ӊҚOU?%EVy+_QgFxnn~\\k0/=Ls{l"C~UŹ=kHq*l=ӵmkZ/\ki\!Dݖ;X-x|h<Ϲ{E>WM[-_m*Z{r$zNzE\i1We,Jx F=)M͹97z{]5ă6B/4KZ3k[QLLwI R#7{?:_YyCCoj,n pb_wAKA+`XxeI_,ߵ@wVvq]Jg5{zǦ5yct#^'ܕD2l;~+~ݪnEAA |w鈕{ZV`Wn`"Ǧm۶vAp٬m#)k9'T bu6rj8aHo/ tN_iѪ[uwbkn)2)N྽eݨDrϯ*:"߸v,hUrWQr箈\.{=5.6S6P\KUkP]Rfnu*w'=t>#a!T7Dux~^<A$WwfW [ϕU0ߙ5jd:Hn J[|Ctyf5ْ&`ŀ x!M<e +c#{Ao1~T-8}2s8wXL]0`Uozݳ? EXBk=8 xd-ZPSYSYy;ႥՑd/aYⅲ?QdYW0 ʒpSe8 **Do%5w?SE!k+ީ=B@<~e/J7,jJ30-F~ UEBw?;kzMCf}Jq5xp@N!1ǧeO܋/_//}kbt n񪁢Ծ_׍ăPֈ3=zE)ߌxC3QgKG.! pBȊèu{^;/; B'[=}<&6^huIc?$|Cn__tV1P8v=8t:9r#zNQ,J5=hJ(J^~ʩ܊u :3,>% a({OK_5HG 93ZzVî/ySweo>Xrm:9 #3 ԥ5_r݆xlYdN^P,,>]Eseoٽ9hzr }G9$n}9FMɊ5rI P5XBa/SZ^43Y7tYzĚKϺUK?޳D+eK!|,:2uVݳg6T3Y{=kҽW]NnA}sȌzfdQp%{,i^;kH6uX4*~ ;V{̕o40N?qd2[/s=[^G- 0 fî<&b /[ohNeV4ro1A1Y}|hSjm *eo* HM_:p:gf(uV?u8˪s%PpE/ڸGX1m  ҇STx"`r8.Wk~9P;uHo+/9J:j*Pn( Ψn132P9Klu 8YFnv󰾯 |yϦrB!<|t\8:(Uo;LСzh읃cjm&u?h8UZOշU 'XfP>+܇LOFFvF %Z_䚑j7&ZI69qng Ga)L*R/$԰SE "մ FƍܤC?#Ff* v̡%iwbS4)XkRIS*(vB U3y'#FewbX8F*Ӎ16 R;M_g+b1\g^ZRoC&z4l^- !`FeZ'(|c#Vm IDAT! BЎI,UVhl>FI3{ϵ3V<ͤKߜߚC2> }X%9!JS!j{ G &**;A#SH`r+0+;HCN a8s٣oV"mof鯕BOrXO SqzhyaC9B?azMv;33t+(\hX#9=x8 'n$97uI6E߹wKpp9[煙¼d>...&&b`AI|rNJ5S܁j_-y(z2I)G&h8|f̷z.\RM7Y_SoLAlCI-/6LÁ欼ntOx[W^˟bBUqQϱǃJ!G[w?oE5lJ?/I_߹Y'g B[럗9S\)Rk ]8W^L5vdM^_s^bf BUJG~yzdL"aiO}\?HEk*R-;\tt4d TϓTn!:h(+[Veq=zƢtʃݮvH的fXCxWswf1 6aisYx9^]q=S x|Qfs\e8|Sꗍ<=yO>/_MFlr:d|õ"Tw$ybcfnV+QۀY!@aBFQm%[;qfB+:\Qa(h2SĆSi}׷VO>}Wm-Sp>|uՂ劀 Ce͉Jćf8uҧW0eƒOvqhHSxCP~yu_;Y䝍O8`w牷^..o~ELO&Tě/RITKƉ&}jaicM0cF`?*=k3aMc]?AS`SJv[3d Q\q* U7n:SbX1"^/W_*} ;$e0KLLamt)~Tcd91 Gy/ٽgCD D# 4~( -}\-5aVɭbԽ45X?'p'y걙7 +\1qU,?޺^+(|Xl87?>Dyt+]{^>1ݗv]4[mG<+{8{מ .OeW`ކ|XzDm+f2OZiz#/s m(V/vvJyjQj cdv|됆OGKm2&Y셾Z5&S6ׁFWf:BB#@KDŽ"Tf:E+]Ż8nOU1! o/é5ru 6rlIx[(&iќap`I)ʇjWc#aխk ,=v C>菖4 cee;lxuo;wv,۽l {׷RV\Nd߳x܁Ű฼ ݗµk"'U**JKvo_f$p,)JJvvEZ*9/_Y_={Rd:6撮vun]e?ޢK4x:D"r0{cJPpp'tqy>xÅO-J Z D(\HuXn0L*--bYN6l8ᢢ 4Bd+ BHrm;1*NB|M|P}b5<򻧻d[vwQ܆YgXDe;5ZNNvt%f~]: ~RK׈M$Զ% ͷ'ʆ: )))IIIl 36UT;5ͮ0obcOB8! A _Zd[~j~{ n: b\]NJ.9sNO,ױ4Ź49?35Vl@ cF'p*7;27lMqmI^qx\,TJ3=1x}o/W(nU ў#o~t:V/9 e&5n/- [ԣ{jx'{|U6UYplR)-\NKB ]Իuwm*geJ*cWbR+V(QSOgԟuG | >WmFE"N'p)q4:! NNЙL`4M!^SZC#hwz3Ne}J%6"ەTO7TYgSQ]ON~ u(#6/ۍGR];|FΎ.HpFaIڝ@p`)qs-:D)b,fj5ſ{'jH-Y69hbCs,W¤[WKbW;՟,Ӌ-xOYm*WN˘ WV}'vk߹[7)N?WQ"tY+no]ƇRHh^p̈3|||7 0z!±ڂ nlb_{c4m0i\fM<oZKG1>lY8&w't*[Cn*fɦ]U>\llJxENW_"jnB];ѥc0[͔6]+Zvl j4e8_1Hr`("bJMC NhhpoHiჺTÇoNݶnCnA&(*F^\ll2p6ƺ9xnQpF>b} m4,g1f9q /?ֹZnq{K =0O?=Pĕs\>Xd7]xRc#ĝ`JļW~1n_˴ua=Ekr]][=<"8Dj^4BiP0@m߾+{ ^?FmyҍPٸ7 Hl# Lim9T+*1* Bs}aFy*68qyVgI a: <[vl}lhɏ6 N'&ӵdͿK#}xBzT3 Oɫ&Õ1IaE^'=I"}AjKr0-m<%0:#R*=m5/t_cS9cw`=?znbW[z3 +znª`/IUQ&ߝ BPvkl?5?~|gCVhUbu3H{\7a5!{%d?-YtbeH?43-=w%x@AYi4cV7SYESJ\3N^GBW8;m8&ǜe0+^}e]Mw\.dIv{ s`7욃v[40 퍍4чm\+KQ^Pq vTYq⚈uvA朥/#LLBIK%VDJ>*(F'(%0Z ܄hT"$Tc7EL@$EߕԹ Uvůr䙘 4%VjKg.^̍7|pe fΆSRq&fY\|K ubOg):ԻֱTkj=d]۲fӁ HDϞQӲe?`|͕lzVؑ6J.&|)h26S -jP j uJ!.Lݗ:{쪈wZ 1/YU%yf/0?L}18e~dVhdhs,auhFcGJBKԇUqi\N;_b?d/m|JeajӏbiC.d"ur^QM|T57m.PU6S*q8n]3 4vmHȭgNϴcl,|BYCQ]OOh[etq}o {V?iO~{ Olw a5rU@oJr3[ZDbM !@?q2 ^BcMBN-LҢ_ T&!\/FYӏ<J(Y^,FZDxO77TliB6ߤ1 Lx #&q];5 ;Obx{ppMlHPꑱp=`F dz#.۸wƂ^_NOws+{r.ێǜufwا32*ڴ{ˆp֨'Xz%%Zp>穽[XR>yM .Kue)C܁jKkB@B`{d%JmN7_hqڷ/BUlɯh8]RxN[-|͕`(M B6obd.yjjJ߿daZzW$0<~'Duvv\.<B*\hn%K0vghfcysk%Y$.ڠ8EgMu8 w s^>ff54 㡱-XxBE+p !9k1>)` WݗkmXPS VGh#77֝f!+ `0m PF x4B0N`p&ܒէsz%JMzM8TD9zEl1mY\YPv, #!^ԫvCsuNBJ*#(3DZY6gGf!@-o\Uj{nlgU0s$ DD\p } =:Ge_ {ZM=joFǯ=2ӘQW3z47N2PH mugU>N>IW@t:NNRϙ\s ;#L3hztI:(ƛѝQ>f6`zj7$ 2 @[LarN#8& s  |A5[m2V@UEzhbp֩5mbUwETgcAdxn9\FN ZFGgrO !H*(ч,F{2 ;B+mbäflQ{rmT!kf5s˕9 $넱Jĝ!8&0Õ9nDf!EGnor݂-Eq@LXdQYv]Ep)(JJ4eGjrs:e쒌n2r5`wT[q)y853Yg/G"4 dWt뻙@4*`Q"TݖSΨ(t}zhgaG`7IfOx+8Ac~64wJMr[W<"MO0Ң&@2*'E^6|a2Jv838}8 9&u1dGGr:fpܭ)0Qvn~~Ğk}[S0rrQX>M0~Qg7qvfQـi )32's:A`WumlBjLcy_I`48M1Lq1J5"! XB|ksZᲘ[S3x"9Wh)cdb3ҊW8eEK2zX> *2AՓ_ïdL<g1z_Yy):8dp<0 a%KΈR"X8`#9;p 8;;>k;N0tc'<1a< &·˅qvc+s>wKtn %xFyJe4sI'8XEΏAUp,& }S0Fcq X{; V?FA{k莪sf8Y L~{"SxQקO@Oya?/F 4cNwip:kaN!`଼B0饁qpF nxeĝQȐJ% x Nӹ\vtx:t!;ˎ[ p>(@ 8(*ccW|^@mwYX/ eg21?Gƀ,Q1lT~V:@l>Kcx0!:-u^kGkbWf,/J8,c#]O{^kx-ghA!+/! F@ ڳqAĝ®: 8EQ DŽp_`56kduaUncAl[sW:j +[2׳3]_0JlQRg͈XM2N9aA@kt uȣ(؀9āCKvFhru^آɆv?`xEuX|=1 sZtr>/69RvFYyc11_僮/:J7"=sNO-Hg&#gz VJ'xg<1Qʡ`;!P( "kHIidH8] #;v]/>,"@S [,1xl䪓:‹fNxk jW 9QMV1_^uq0ObX㥾NBwMvfE(OL ܕMTo0rXxXRǘޓ(:HAI!,8#*5w N`IqFjk#8 'AtP[8 @8>W2B@?'Y5>~žҨAKB8&%~B]8Fx%;$[I5FMQƙ*S2p@dXEeA⿌-ػ Tx7DGk$nXYGzuF?/lgg5BuvwP#\I<0RB@Nлɠ/T]"UF! q?&҅8\E6iħyp 1Y|ve2bծ)8&0R ktRc1lW.!@ .Ø! <£]Y33!B_ A "+t=U7V{&*$LԚ|k$rad+]UZ $d $&,yw tX cYvz!@A@gW1*C4Ժ3슎DhW-2R` ҕwnAPN"=1B\ڕ=NWB(Ύ\2e𳎯y4vWC$.Fv 8{:6640 Where ```` is your controller's IP address that is accessible via the Host Machine's eth0 interface. Simulating an NVC ----------------- A VTEP implementation expects to be driven by a Network Virtualization Controller (NVC), such as NSX. If one does not exist, it's possible to use vtep-ctl to simulate one: 1. Create a logical switch: :: $ vtep-ctl add-ls ls0 2. Bind the logical switch to a port: :: $ vtep-ctl bind-ls br0 p0 0 ls0 $ vtep-ctl set Logical_Switch ls0 tunnel_key=33 3. Direct unknown destinations out a tunnel. For handling L2 broadcast, multicast and unknown unicast traffic, packets can be sent to all members of a logical switch referenced by a physical switch. The "unknown-dst" address below is used to represent these packets. There are different modes to replicate the packets. The default mode of replication is to send the traffic to a service node, which can be a hypervisor, server or appliance, and let the service node handle replication to other transport nodes (hypervisors or other VTEP physical switches). This mode is called *service node* replication. An alternate mode of replication, called *source node* replication, involves the source node sending to all other transport nodes. Hypervisors are always responsible for doing their own replication for locally attached VMs in both modes. Service node mode is the default. Service node replication mode is considered a basic requirement because it only requires sending the packet to a single transport node. The following configuration is for service node replication mode as only a single transport node destination is specified for the unknown-dst address: :: $ vtep-ctl add-mcast-remote ls0 unknown-dst 10.2.2.2 4. Optionally, change the replication mode from a default of ``service_node`` to ``source_node``, which can be done at the logical switch level: :: $ vtep-ctl set-replication-mode ls0 source_node 5. Direct unicast destinations out a different tunnel: :: $ vtep-ctl add-ucast-remote ls0 00:11:22:33:44:55 10.2.2.3 ovs-2.9.0/Documentation/index.rst000066400000000000000000000100371324262074100170050ustar00rootroot00000000000000.. Copyright (c) 2016, Stephen Finucane Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. Convention for heading levels in Open vSwitch documentation: ======= Heading 0 (reserved for the title in a document) ------- Heading 1 ~~~~~~~ Heading 2 +++++++ Heading 3 ''''''' Heading 4 Avoid deeper levels because they do not render well. ========================== Open vSwitch Documentation ========================== How the Documentation is Organised ---------------------------------- The Open vSwitch documentation is organised into multiple sections: - :doc:`Installation guides ` guide you through installing Open vSwitch (OVS) and Open Virtual Network (OVN) on a variety of different platforms - :doc:`Tutorials ` take you through a series of steps to configure OVS and OVN in sandboxed environments - :doc:`Topic guides ` provide a high level overview of OVS and OVN internals and operation - :doc:`How-to guides ` are recipes or use-cases for OVS and OVN. They are more advanced than the tutorials. - :doc:`Frequently Asked Questions ` provide general insight into a variety of topics related to configuration and operation of OVS and OVN. First Steps ----------- Getting started with Open vSwitch (OVS) or Open Virtual Network (OVN) for Open vSwitch? Start here. - **Overview:** :doc:`intro/what-is-ovs` | :doc:`intro/why-ovs` - **Install:** :doc:`intro/install/general` | :doc:`intro/install/userspace` | :doc:`intro/install/netbsd` | :doc:`intro/install/windows` | :doc:`intro/install/xenserver` | :doc:`intro/install/dpdk` | :doc:`Installation FAQs ` - **Tutorials:** :doc:`tutorials/faucet` | :doc:`tutorials/ovs-advanced` | :doc:`tutorials/ovn-sandbox` | :doc:`tutorials/ovn-openstack` Deeper Dive ----------- - **Architecture** :doc:`topics/design` | :doc:`topics/openflow` | :doc:`topics/integration` | :doc:`topics/porting` - **DPDK** :doc:`howto/dpdk` | :doc:`topics/dpdk/vhost-user` - **Windows** :doc:`topics/windows` - **Integrations:** :doc:`topics/language-bindings` - **Reference Guides:** :doc:`ref/index` - **Testing** :doc:`topics/testing` - **Packaging:** :doc:`intro/install/debian` | :doc:`intro/install/rhel` | :doc:`intro/install/fedora` The Open vSwitch Project ------------------------ Learn more about the Open vSwitch project and about how you can contribute: - **Community:** :doc:`internals/release-process` | :doc:`internals/authors` | :doc:`internals/mailing-lists` | :doc:`internals/patchwork` | :doc:`internals/bugs` | :doc:`internals/security` - **Contributing:** :doc:`internals/contributing/submitting-patches` | :doc:`internals/contributing/backporting-patches` | :doc:`internals/contributing/coding-style` | :doc:`internals/contributing/coding-style-windows` - **Maintaining:** :doc:`internals/charter` | :doc:`internals/maintainers` | :doc:`internals/committer-responsibilities` | :doc:`internals/committer-grant-revocation` | :doc:`internals/committer-emeritus-status` - **Documentation:** :doc:`internals/contributing/documentation-style` | :doc:`Building Open vSwitch Documentation ` | :doc:`internals/documentation` Getting Help ------------- - Seeing an issue of potential bug? Report problems to bugs@openvswitch.org - Looking for specific information? Try the :ref:`genindex`, :ref:`modindex` or the :doc:`detailed table of contents `. ovs-2.9.0/Documentation/internals/000077500000000000000000000000001324262074100171425ustar00rootroot00000000000000ovs-2.9.0/Documentation/internals/authors.rst000066400000000000000000000016121324262074100213610ustar00rootroot00000000000000.. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. Convention for heading levels in Open vSwitch documentation: ======= Heading 0 (reserved for the title in a document) ------- Heading 1 ~~~~~~~ Heading 2 +++++++ Heading 3 ''''''' Heading 4 Avoid deeper levels because they do not render well. .. include:: ../../AUTHORS.rst ovs-2.9.0/Documentation/internals/bugs.rst000066400000000000000000000047551324262074100206470ustar00rootroot00000000000000.. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. Convention for heading levels in Open vSwitch documentation: ======= Heading 0 (reserved for the title in a document) ------- Heading 1 ~~~~~~~ Heading 2 +++++++ Heading 3 ''''''' Heading 4 Avoid deeper levels because they do not render well. ============================== Reporting Bugs in Open vSwitch ============================== We are eager to hear from users about problems that they have encountered with Open vSwitch. This file documents how best to report bugs so as to ensure that they can be fixed as quickly as possible. Please report bugs by sending email to bugs@openvswitch.org. For reporting security vulnerabilities, please read :doc:`security`. The most important parts of your bug report are the following: - What you did that make the problem appear. - What you expected to happen. - What actually happened. Please also include the following information: - The Open vSwitch version number (as output by ``ovs-vswitchd --version``). - The Git commit number (as output by ``git rev-parse HEAD``), if you built from a Git snapshot. - Any local patches or changes you have applied (if any). The following are also handy sometimes: - The kernel version on which Open vSwitch is running (from ``/proc/version``) and the distribution and version number of your OS (e.g. "Centos 5.0"). - The contents of the vswitchd configuration database (usually ``/etc/openvswitch/conf.db``). - The output of ``ovs-dpctl show``. - If you have Open vSwitch configured to connect to an OpenFlow controller, the output of ``ovs-ofctl show `` for each ```` configured in the vswitchd configuration database. - A fix or workaround, if you have one. - Any other information that you think might be relevant. .. important:: bugs@openvswitch.org is a public mailing list, to which anyone can subscribe, so do not include confidential information in your bug report. ovs-2.9.0/Documentation/internals/charter.rst000066400000000000000000000206571324262074100213360ustar00rootroot00000000000000The Linux Foundation Open vSwitch Project Charter ================================================= Effective August 9, 2016 1. Mission of Open vSwitch Project (“OVS”). The mission of OVS is to: a. create an open source, production quality virtual networking platform, including a software switch, control plane, and related components, that supports standard management interfaces and opens the forwarding functions to programmatic extension and control; and b. host the infrastructure for an OVS community, establishing a neutral home for community assets, infrastructure, meetings, events and collaborative discussions. 2. Technical Steering Committee (“TSC”) a. A TSC shall be composed of the Committers for OVS. The list of Committers on the TSC are available at :doc:`/internals/maintainers`. b. TSC projects generally will involve Committers and Contributors: i. Contributors: anyone in the technical community that contributes code, documentation or other technical artifacts to the OVS codebase. ii. Committers: Contributors who have the ability to commit directly to a project’s main branch or repository on an OVS project. c. Participation in as a Contributor and/or Committer is open to anyone under the terms of this Charter. The TSC may: i. establish work flows and procedures for the submission, approval and closure or archiving of projects, ii. establish criteria and processes for the promotion of Contributors to Committer status, available at :doc:`/internals/committer-grant-revocation`. and iii. amend, adjust and refine the roles of Contributors and Committers listed in Section 2.b., create new roles and publicly document responsibilities and expectations for such roles, as it sees fit, available at :doc:`/internals/committer-responsibilities`. d. Responsibilities: The TSC is responsible for overseeing OVS activities and making decisions that impact the mission of OVS, including: i. coordinating the technical direction of OVS; ii. approving project proposals (including, but not limited to, incubation, deprecation and changes to a project’s charter or scope); iii. creating sub-committees or working groups to focus on cross-project technical issues and requirements; iv. communicating with external and industry organizations concerning OVS technical matters; v. appointing representatives to work with other open source or standards communities; vi. establishing community norms, workflows or policies including processes for contributing (available at :doc:`/internals/contributing/index`), issuing releases, and security issue reporting policies; vii. discussing, seeking consensus, and where necessary, voting on technical matters relating to the code base that affect multiple projects; and viii. coordinate any marketing, events or communications with The Linux Foundation. 3. TSC Voting a. While it is the goal of OVS to operate as a consensus based community, if any TSC decision requires a vote to move forward, the Committers shall vote on a one vote per Committer basis. b. TSC votes should be conducted by email. In the case of a TSC meeting where a valid vote is taken, the details of the vote and any discussion should be subsequently documented for the community (e.g. to the appropriate email mailing list). c. Quorum for TSC meetings shall require two-thirds of the TSC representatives. The TSC may continue to meet if quorum is not met, but shall be prevented from making any decisions requiring a vote at the meeting. d. Except as provided in Section 8.d. and 9.a., decisions by electronic vote (e.g. email) shall require a majority of all voting TSC representatives. Decisions by electronic vote shall be made timely, and unless specified otherwise, within three (3) business days. Except as provided in Section 8.d. and 9.a., decisions by vote at a meeting shall require a majority vote, provided quorum is met. e. In the event of a tied vote with respect to an action that cannot be resolved by the TSC, any TSC representative shall be entitled to refer the matter to the Linux Foundation for assistance in reaching a decision. 4. Antitrust Guidelines a. All participants in OVS shall abide by The Linux Foundation Antitrust Policy available at http://www.linuxfoundation.org/antitrust-policy. b. All members shall encourage open participation from any organization able to meet the participation requirements, regardless of competitive interests. Put another way, the community shall not seek to exclude any participant based on any criteria, requirements or reasons other than those that are reasonable and applied on a non-discriminatory basis to all participants. 5. Code of Conduct a. The TSC may adopt a specific OVS Project code of conduct, with approval from the LF. 6. Budget and Funding a. The TSC shall coordinate any budget or funding needs with The Linux Foundation. Companies participating may be solicited to sponsor OVS activities and infrastructure needs on a voluntary basis. b. The Linux Foundation shall have custody of and final authority over the usage of any fees, funds and other cash receipts. c. A General & Administrative (G&A) fee will be applied by the Linux Foundation to funds raised to cover Finance, Accounting, and operations. The G&A fee shall equal 9% of OVS’s first $1,000,000 of gross receipts and 6% of OVS’s gross receipts over $1,000,000. d. Under no circumstances shall The Linux Foundation be expected or required to undertake any action on behalf of OVS that is inconsistent with the tax exempt purpose of The Linux Foundation. 7. General Rules and Operations. The OVS project shall be conducted so as to: a. engage in the work of the project in a professional manner consistent with maintaining a cohesive community, while also maintaining the goodwill and esteem of The Linux Foundation in the open source software community; b. respect the rights of all trademark owners, including any branding and usage guidelines; c. engage The Linux Foundation for all OVS press and analyst relations activities; d. upon request, provide information regarding Project participation, including information regarding attendance at Project-sponsored events, to The Linux Foundation; and e. coordinate with The Linux Foundation in relation to any websites created directly for OVS. 8. Intellectual Property Policy a. Members agree that all new inbound code contributions to OVS shall be made under the Apache License, Version 2.0 (available at http://www.apache.org/licenses/LICENSE-2.0). All contributions shall be accompanied by a Developer Certificate of Origin sign-off (http://developercertificate.org) that is submitted through a TSC and LF-approved contribution process. b. All outbound code will be made available under the Apache License, Version 2.0. c. All documentation will be contributed to and made available by OVS under the Apache License, Version 2.0. d. For any new project source code, if an alternative inbound or outbound license is required for compliance with the license for a leveraged open source project (e.g. GPLv2 for Linux kernel) or is otherwise required to achieve OVS’s mission, the TSC may approve the use of an alternative license for specific inbound or outbound contributions on an exception basis. Any exceptions must be approved by a majority vote of the entire TSC and must be limited in scope to what is required for such purpose. Please email tsc@openvswitch.org to obtain exception approval. e. Subject to available funds, OVS may engage The Linux Foundation to determine the availability of, and register, trademarks, service marks, which shall be owned by the LF. 9. Amendments a. This charter may be amended by a two-thirds vote of the entire TSC, subject to approval by The Linux Foundation. ovs-2.9.0/Documentation/internals/committer-emeritus-status.rst000066400000000000000000000054001324262074100250520ustar00rootroot00000000000000.. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. Convention for heading levels in Open vSwitch documentation: ======= Heading 0 (reserved for the title in a document) ------- Heading 1 ~~~~~~~ Heading 2 +++++++ Heading 3 ''''''' Heading 4 Avoid deeper levels because they do not render well. ================================== Emeritus Status for OVS Committers ================================== OVS committers are nominated and elected based on their impact on the Open vSwitch project. Over time, as committers' responsibilities change, some may become unable or uninterested in actively participating in project governance. Committer "emeritus" status provides a way for committers to take a leave of absence from OVS governance responsibilities. The following guidelines clarify the process around the emeritus status for committers: * A committer may choose to transition from active to emeritus, or from emeritus to active, by sending an email to the committers mailing list. * If a committer hasn't been heard from in 6 months, and does not respond to reasonable attempts to contact him or her, the other committers can vote as a majority to transition the committer from active to emeritus. (If the committer resurfaces, he or she can transition back to active by sending an email to the committers mailing list.) * Emeritus committers may stay on the committers mailing list to continue to follow any discussions there. * Emeritus committers do not nominate or vote in committer elections. From a governance perspective, they are equivalent to a non-committer. * Emeritus committers cannot merge patches to the OVS repository. * Emeritus committers will be listed in a separate section in the MAINTAINERS.rst file to continue to recognize their contributions to the project. Emeritus status does not replace the procedures for forcibly removing a committer. Note that just because a committer is not able to work on the project on a day-to-day basis, we feel they are still capable of providing input on the direction of the project. No committer should feel pressured to move themselves to this status. Again, it's just an option for those that do not currently have the time or interest. ovs-2.9.0/Documentation/internals/committer-grant-revocation.rst000066400000000000000000000332771324262074100251730ustar00rootroot00000000000000.. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. Convention for heading levels in Open vSwitch documentation: ======= Heading 0 (reserved for the title in a document) ------- Heading 1 ~~~~~~~ Heading 2 +++++++ Heading 3 ''''''' Heading 4 Avoid deeper levels because they do not render well. ===================================== OVS Committer Grant/Revocation Policy ===================================== An OVS committer is a participant in the project with the ability to commit code directly to the master repository. Commit access grants a broad ability to affect the progress of the project as presented by its most important artifact, the code and related resources that produce working binaries of Open vSwitch. As such it represents a significant level of trust in an individual's commitment to working with other committers and the community at large for the benefit of the project. It can not be granted lightly and, in the worst case, must be revocable if the trust placed in an individual was inappropriate. This document suggests guidelines for granting and revoking commit access. It is intended to provide a framework for evaluation of such decisions without specifying deterministic rules that wouldn't be sensitive to the nuance of specific situations. In the end the decision to grant or revoke committer privileges is a judgment call made by the existing set of committers. Granting Commit Access ---------------------- Granting commit access should be considered when a candidate has demonstrated the following in their interaction with the project: - Contribution of significant new features through the patch submission process where: - Submissions are free of obvious critical defects - Submissions do not typically require many iterations of improvement to be accepted - Consistent participation in code review of other's patches, including existing committers, with comments consistent with the overall project standards - Assistance to those in the community who are less knowledgeable through active participation in project forums such as the ovs-discuss mailing list. - Plans for sustained contribution to the project compatible with the project's direction as viewed by current committers. - Commitment to meet the expectations described in the "Expectations of Developer's with Open vSwitch Access" The process to grant commit access to a candidate is simple: - An existing committer nominates the candidate by sending an email to all existing committers with information substantiating the contributions of the candidate in the areas described above. - All existing committers discuss the pros and cons of granting commit access to the candidate in the email thread. - When the discussion has converged or a reasonable time has elapsed without discussion developing (e.g. a few business days) the nominator calls for a final decision on the candidate with a followup email to the thread. - Each committer may vote yes, no, or abstain by replying to the email thread. A failure to reply is an implicit abstention. - After votes from all existing committers have been collected or a reasonable time has elapsed for them to be provided (e.g. a couple of business days) the votes are evaluated. To be granted commit access the candidate must receive yes votes from a majority of the existing committers and zero no votes. Since a no vote is effectively a veto of the candidate it should be accompanied by a reason for the vote. - The nominator summarizes the result of the vote in an email to all existing committers. - If the vote to grant commit access passed, the candidate is contacted with an invitation to become a committer to the project which asks them to agree to the committer expectations documented on the project web site. - If the candidate agrees access is granted by setting up commit access to the repos on github. Revoking Commit Access ---------------------- When a committer behaves in a manner that other committers view as detrimental to the future of the project, it raises a delicate situation with the potential for the creation of division within the greater community. These situations should be handled with care. The process in this case is: - Discuss the behavior of concern with the individual privately and explain why you believe it is detrimental to the project. Stick to the facts and keep the email professional. Avoid personal attacks and the temptation to hypothesize about unknowable information such as the other's motivations. Make it clear that you would prefer not to discuss the behavior more widely but will have to raise it with other contributors if it does not change. Ideally the behavior is eliminated and no further action is required. If not, - Start an email thread with all committers, including the source of the behavior, describing the behavior and the reason it is detrimental to the project. The message should have the same tone as the private discussion and should generally repeat the same points covered in that discussion. The person whose behavior is being questioned should not be surprised by anything presented in this discussion. Ideally the wider discussion provides more perspective to all participants and the issue is resolved. If not, - Start an email thread with all committers except the source of the detrimental behavior requesting a vote on revocation of commit rights. Cite the discussion among all committers and describe all the reasons why it was not resolved satisfactorily. This email should be carefully written with the knowledge that the reasoning it contains may be published to the larger community to justify the decision. - Each committer may vote yes, no, or abstain by replying to the email thread. A failure to reply is an implicit abstention. - After all votes have been collected or a reasonable time has elapsed for them to be provided (e.g. a couple of business days) the votes are evaluated. For the request to revoke commit access for the candidate to pass it must receive yes votes from two thirds of the existing committers. - anyone that votes no must provide their reasoning, and - if the proposal passes then counter-arguments for the reasoning in no votes should also be documented along with the initial reasons the revocation was proposed. Ideally there should be no new counter-arguments supplied in a no vote as all concerns should have surfaced in the discussion before the vote. - The original person to propose revocation summarizes the result of the vote in an email to all existing committers excepting the candidate for removal. - If the vote to revoke commit access passes, access is removed and the candidate for revocation is informed of that fact and the reasons for it as documented in the email requesting the revocation vote. - Ideally the revoked committer peacefully leaves the community and no further action is required. However, there is a distinct possibility that he/she will try to generate support for his/her point of view within the larger community. In this case the reasoning for removing commit access as described in the request for a vote will be published to the community. Changing the Policy ------------------- The process for changing the policy is: - Propose the changes to the policy in an email to all current committers and request discussion. - After an appropriate period of discussion (a few days) update the proposal based on feedback if required and resend it to all current committers with a request for a formal vote. - After all votes have been collected or a reasonable time has elapsed for them to be provided (e.g. a couple of business days) the votes are evaluated. For the request to modify the policy to pass it must receive yes votes from two thirds of the existing committers. Template Emails =============== Nomination to Grant Commit Access --------------------------------- I would like to nominate *[candidate]* for commit access. I believe *[he/she]* has met the conditions for commit access described in the committer grant policy on the project web site in the following ways: *[list of requirements & evidence]* Please reply to all in this message thread with your comments and questions. If that discussion concludes favorably I will request a formal vote on the nomination in a few days. Vote to Grant Commit Access --------------------------- I nominated *[candidate]* for commit access on *[date]*. Having allowed sufficient time for discussion it's now time to formally vote on the proposal. Please reply to all in this thread with your vote of: YES, NO, or ABSTAIN. A failure to reply will be counted as an abstention. If you vote NO, by our policy you must include the reasons for that vote in your reply. The deadline for votes is *[date and time]*. If a majority of committers vote YES and there are zero NO votes commit access will be granted. Vote Results for Grant of Commit Access --------------------------------------- The voting period for granting to commit access to *[candidate]* initiated at *[date and time]* is now closed with the following results: YES: *[count of yes votes]* (*[% of voters]*) NO: *[count of no votes]* (*[% of voters]*) ABSTAIN: *[count of abstentions]* (*[% of voters]*) Based on these results commit access *[is/is NOT]* granted. Invitation to Accepted Committer -------------------------------- Due to your sustained contributions to the Open vSwitch (OVS) project we would like to provide you with commit access to the project repository. Developers with commit access must agree to fulfill specific responsibilities described in the source repository: /Documentation/internals/committer-responsibilities.rst Please let us know if you would like to accept commit access and if so that you agree to fulfill these responsibilities. Once we receive your response we'll set up access. We're looking forward continuing to work together to advance the Open vSwitch project. Proposal to Revoke Commit Access for Detrimental Behavior --------------------------------------------------------- I regret that I feel compelled to propose revocation of commit access for *[candidate]*. I have privately discussed with *[him/her]* the following reasons I believe *[his/her]* actions are detrimental to the project and we have failed to come to a mutual understanding: *[List of reasons and supporting evidence]* Please reply to all in this thread with your thoughts on this proposal. I plan to formally propose a vote on the proposal on or after *[date and time]*. It is important to get all discussion points both for and against the proposal on the table during the discussion period prior to the vote. Please make it a high priority to respond to this proposal with your thoughts. Vote to Revoke Commit Access ---------------------------- I nominated *[candidate]* for revocation of commit access on *[date]*. Having allowed sufficient time for discussion it's now time to formally vote on the proposal. Please reply to all in this thread with your vote of: YES, NO, or ABSTAIN. A failure to reply will be counted as an abstention. If you vote NO, by our policy you must include the reasons for that vote in your reply. The deadline for votes is *[date and time]*. If 2/3rds of committers vote YES commit access will be revoked. The following reasons for revocation have been given in the original proposal or during discussion: *[list of reasons to remove access]* The following reasons for retaining access were discussed: *[list of reasons to retain access]* The counter-argument for each reason for retaining access is: *[list of counter-arguments for retaining access]* Vote Results for Revocation of Commit Access -------------------------------------------- The voting period for revoking the commit access of *[candidate]* initiated at *[date and time]* is now closed with the following results: - YES: *[count of yes votes]* (*[% of voters]*) - NO: *[count of no votes]* (*[% of voters]*) - ABSTAIN: *[count of abstentions]* (*[% of voters]*) Based on these results commit access *[is/is NOT]* revoked. The following reasons for retaining commit access were proposed in NO votes: *[list of reasons]* The counter-arguments for each of these reasons are: *[list of counter-arguments]* Notification of Commit Revocation for Detrimental Behavior ---------------------------------------------------------- After private discussion with you and careful consideration of the situation, the other committers to the Open vSwitch (OVS) project have concluded that it is in the best interest of the project that your commit access to the project repositories be revoked and this has now occurred. The reasons for this decision are: *[list of reasons for removing access]* While your goals and those of the project no longer appear to be aligned we greatly appreciate all the work you have done for the project and wish you continued success in your future work. ovs-2.9.0/Documentation/internals/committer-responsibilities.rst000066400000000000000000000101441324262074100252640ustar00rootroot00000000000000.. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. Convention for heading levels in Open vSwitch documentation: ======= Heading 0 (reserved for the title in a document) ------- Heading 1 ~~~~~~~ Heading 2 +++++++ Heading 3 ''''''' Heading 4 Avoid deeper levels because they do not render well. ========================================================= Expectations for Developers with Open vSwitch Repo Access ========================================================= Pre-requisites -------------- Be familar with the guidlines and standards defined in :doc:`contributing/index`. Review ------ Code (yours or others') must be reviewed publicly (by you or others) before you push it to the repository. With one exception (see below), every change needs at least one review. If one or more people know an area of code particularly well, code that affects that area should ordinarily get a review from one of them. The riskier, more subtle, or more complicated the change, the more careful the review required. When a change needs careful review, use good judgment regarding the quality of reviews. If a change adds 1000 lines of new code, and a review posted 5 minutes later says just "Looks good," then this is probably not a quality review. (The size of a change is correlated with the amount of care needed in review, but it is not strictly tied to it. A search and replace across many files may not need much review, but one-line optimization changes can have widespread implications.) Your own small changes to fix a recently broken build ("make") or tests ("make check"), that you believe to be visible to a large number of developers, may be checked in without review. If you are not sure, ask for review. If you do push a build fix without review, send the patch to ovs-dev afterward as usual, indicating in the email that you have already pushed it. Regularly review submitted code in areas where you have expertise. Consider reviewing other code as well. Git conventions --------------- Do not push merge commits to the Git repository without prior discussion on ovs-dev. If you apply a change (yours or another's) then it is your responsibility to handle any resulting problems, especially broken builds and other regressions. If it is someone else's change, then you can ask the original submitter to address it. Regardless, you need to ensure that the problem is fixed in a timely way. The definition of "timely" depends on the severity of the problem. If a bug is present on master and other branches, fix it on master first, then backport the fix to other branches. Straightforward backports do not require additional review (beyond that for the fix on master). Feature development should be done only on master. Occasionally it makes sense to add a feature to the most recent release branch, before the first actual release of that branch. These should be handled in the same way as bug fixes, that is, first implemented on master and then backported. Keep the authorship of a commit clear by maintaining a correct list of "Signed-off-by:"s. If a confusing situation comes up, as it occasionally does, bring it up on the mailing list. If you explain the use of "Signed-off-by:" to a new developer, explain not just how but why, since the intended meaning of "Signed-off-by:" is more important than the syntax. As part of your explanation, quote or provide a URL to the Developer's Certificate of Origin in :doc:`contributing/submitting-patches`. Use Reported-by: and Tested-by: tags in commit messages to indicate the source of a bug report. Keep the ``AUTHORS.rst`` file up to date. ovs-2.9.0/Documentation/internals/contributing/000077500000000000000000000000001324262074100216515ustar00rootroot00000000000000ovs-2.9.0/Documentation/internals/contributing/backporting-patches.rst000066400000000000000000000256111324262074100263400ustar00rootroot00000000000000.. Copyright (c) 2017 Nicira, Inc. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. Convention for heading levels in Open vSwitch documentation: ======= Heading 0 (reserved for the title in a document) ------- Heading 1 ~~~~~~~ Heading 2 +++++++ Heading 3 ''''''' Heading 4 Avoid deeper levels because they do not render well. =================== Backporting patches =================== .. note:: This is an advanced topic for developers and maintainers. Readers should familiarize themselves with building and running Open vSwitch, with the git tool, and with the Open vSwitch patch submission process. The backporting of patches from one git tree to another takes multiple forms within Open vSwitch, but is broadly applied in the following fashion: - Contributors submit their proposed changes to the latest development branch - Contributors and maintainers provide feedback on the patches - When the change is satisfactory, maintainers apply the patch to the development branch. - Maintainers backport changes from a development branch to release branches. With regards to Open vSwitch user space code and code that does not comprise the Linux datapath and compat code, the development branch is `master` in the Open vSwitch repository. Patches are applied first to this branch, then to the most recent `branch-X.Y`, then earlier `branch-X.Z`, and so on. The most common kind of patch in this category is a bugfix which affects master and other branches. For Linux datapath code, the primary development branch is in the `net-next`_ tree as described in the section below, and patch discussion occurs on the `netdev`__ mailing list. Patches are first applied to the upstream branch by the networking maintainer, then the contributor backports the patch to the Open vSwitch `master` development branch. Patches in this category may include features which have been applied upstream, or bugfixes to the Open vSwitch datapath code. For bugfixes, the patches subsequently follow the regular Open vSwitch process as described above to reach older branches. __ http://vger.kernel.org/vger-lists.html#netdev Changes to userspace components ------------------------------- Patches which are fixing bugs should be considered for backporting from `master` to release branches. Open vSwitch contributors submit their patches targeted to the `master` branch, using the ``Fixes`` tag described in :doc:`submitting-patches`. The maintainer first applies the patch to `master`, then backports the patch to each older affected tree, as far back as it goes or at least to all currently supported branches. This is usually each branch back to the most recent LTS release branch. If the fix only affects a particular branch and not `master`, contributors should submit the change with the target branch listed in the subject line of the patch. Contributors should list all versions that the bug affects. The ``git format-patch`` argument ``--subject-prefix`` may be used when posting the patch, for example: :: $ git format-patch HEAD --subject-prefix="PATCH branch-2.7" If a maintainer is backporting a change to older branches and the backport is not a trivial cherry-pick, then the maintainer may opt to submit the backport for the older branch on the mailinglist for further review. This should be done in the same manner as described above. Changes to Linux kernel components ---------------------------------- The Linux kernel components in Open vSwitch go through initial review in the upstream Linux netdev community before they go into the Open vSwitch tree. As such, backports from upstream to the Open vSwitch tree may include bugfixes or new features. The `netdev-FAQ`_ describes the general process for merging patches to the upstream Linux tree. To keep track of the changes which are made upstream against the changes which have been backported to the Open vSwitch tree, backports should be done in the order that they are applied to the upstream `net-next`_ tree. For example, if the git history in ``linux/net/openvswitch/`` in the `net-next` tree lists patches A, B and C that were applied (in that order), then the backports of these patches to ``openvswitch/datapath/`` should be done submitted in the order A, B, then C. Patches that are proposed against the Open vSwitch tree, including backports, should follow the guidelines described in :doc:`submitting-patches`. Ideally, a series which backports new functionality would also include a series of patches for the userspace components which show how to use the new functionality, and include tests to validate the behaviour. However, in the interests of keeping the Open vSwitch tree in sync with upstream `net-next`, contributors may send Open vSwitch kernel module changes independently of userspace changes. .. _netdev-faq: https://www.kernel.org/doc/Documentation/networking/netdev-FAQ.txt .. _net-next: http://git.kernel.org/cgit/linux/kernel/git/davem/net-next.git How to backport kernel patches ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ First, the patch should be submitted upstream to `netdev`. When the patch has been applied to `net-next`, it is ready to be backported. Starting from the Linux tree, use ``git format-patch`` to format each patch that should be backported. For each of these patches, they may only include changes to ``linux/net/openvswitch/``, or they may include changes to other directories. Depending on which files the patch touches, the backport may be easier or more difficult to undertake. Start by formatting the relevant patches from the Linux tree. For example, to format the last 5 patches to ``net/openvswitch``, going back from OVS commit ``1234c0ffee5``, placing them into ``/tmp/``: :: $ git format-patch -5 1234c0ffee5 net/openvswitch/ -o /tmp Next, change into the Open vSwitch directory and apply the patch: :: $ git am -p3 --reject --directory=datapath/ If this is successful, proceed to the next patch: :: $ git am --continue If this is unsuccessful, the above command applies all changes that it can to the working tree, and leaves rejected hunks in corresponding \*.rej files. Proceed by using ``git diff`` to identify the changes, and edit the files so that the hunk matches what the file looks like when the corresponding commit is checked out in the linux tree. When all hunks are fixed, add the files to the index using ``git add``. If the patch only changes filepaths under ``linux/net/openvswitch``, then most likely the patch is fully backported. At this point, review the patch's changes and compare with the latest upstream code for the modified functions. Occasionally, there may be bugs introduced in a particular patch which were fixed in a later patch upstream. To prevent breakage in the OVS tree, consider rolling later bugfixes into the current patch - particularly if they are small, clear bugfixes in the logic of this patch. Then proceed to the next patch using ``git am --continue``. If you made any changes to the patch compared with the original version, describe the changes in the commit message. If the changes affects other paths, then you may also need to backport function definitions from the upstream tree into the ``datapath/linux/compat`` directory. First, attempt to compile the datapath. If this is successful, then most likely there is no further work required. As per the previous paragraph, consider reviewing and backporting any minor fixes to this code if applicable, then proceed to the next patch using ``git am --continue``. If compilation fails, the compiler will show which functions are missing or broken. Typically this should match with some function definitions provided in the patch file. The following command will attempt to apply all such changes from the patch into the ``openvswitch/datapath/linux/compat`` directory; Like the previous ``git am`` command above, it may succeed or fail. If it succeeds, review the patch and proceed to the next patch using ``git am --continue``. :: $ git am -p3 --reject --directory='datapath/linux/compat/' For each conflicting hunk, attempt to resolve the change so that the function reflects what the function looks like in the upstream Linux tree. After resolving these changes, compile the changes, add the modified files to the index using ``git add``, review the patch, and proceed to the next patch using ``git am --continue``. Submission ~~~~~~~~~~ Once the patches are all assembled and working on the Open vSwitch tree, they need to be formatted again using ``git format-patch``. The common format for commit messages for Linux backport patches is as follows: :: datapath: Remove incorrect WARN_ONCE(). Upstream commit: commit c6b2aafffc6934be72d96855c9a1d88970597fbc Author: Jarno Rajahalme Date: Mon Aug 1 19:08:29 2016 -0700 openvswitch: Remove incorrect WARN_ONCE(). ovs_ct_find_existing() issues a warning if an existing conntrack entry classified as IP_CT_NEW is found, with the premise that this should not happen. However, a newly confirmed, non-expected conntrack entry remains IP_CT_NEW as long as no reply direction traffic is seen. This has resulted into somewhat confusing kernel log messages. This patch removes this check and warning. Fixes: 289f2253 ("openvswitch: Find existing conntrack entry after upcall.") Suggested-by: Joe Stringer Signed-off-by: Jarno Rajahalme Acked-by: Joe Stringer Signed-off-by: Jarno Rajahalme The upstream commit SHA should be the one that appears in Linus' tree so that reviewers can compare the backported patch with the one upstream. Note that the subject line for the backported patch replaces the original patch's ``openvswitch`` prefix with ``datapath``. Patches which only affect the ``datapath/linux/compat`` directory should be prefixed with ``compat``. The contents of a backport should be equivalent to the changes made by the original patch; explain any variations from the original patch in the commit message - For instance if you rolled in a bugfix. Reviewers will verify that the changes made by the backport patch are the same as the changes made in the original commit which the backport is based upon. Patch submission should otherwise follow the regular steps described in :doc:`submitting-patches`. In particular, if performing kernel patch backports, pay attention to :ref:`datapath-testing`. ovs-2.9.0/Documentation/internals/contributing/coding-style-windows.rst000066400000000000000000000154301324262074100264770ustar00rootroot00000000000000.. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. Convention for heading levels in Open vSwitch documentation: ======= Heading 0 (reserved for the title in a document) ------- Heading 1 ~~~~~~~ Heading 2 +++++++ Heading 3 ''''''' Heading 4 Avoid deeper levels because they do not render well. ========================================== Open vSwitch Windows Datapath Coding Style ========================================== The :doc:`coding style ` guide gives the flexiblity for each platform to use its own coding style for the kernel datapath. This file describes the specific coding style used in most of the C files in the Windows kernel datapath of the Open vSwitch distribution. Most of the coding conventions applicable for the Open vSwitch distribution are applicable to the Windows kernel datapath as well. There are some exceptions and new guidlines owing to the commonly followed practices in Windows kernel/driver code. They are noted as follows: Basics ------ - Limit lines to 79 characters. Many times, this is not possible due to long names of functions and it is fine to go beyond the characters limit. One common example is when calling into NDIS functions. Types ----- Use data types defined by Windows for most of the code. This is a common practice in Windows driver code, and it makes integrating with the data structures and functions defined by Windows easier. Example: ``DWORD`` and ``BOOLEAN``. Use caution in portions of the code that interface with the OVS userspace. OVS userspace does not use Windows specific data types, and when copying data back and forth between kernel and userspace, care should be exercised. Naming ------ It is common practice to use camel casing for naming variables, functions and files in Windows. For types, especially structures, unions and enums, using all upper case letters with words seprated by '_' is common. These practices can be used for OVS Windows datapath. However, use the following guidelines: - Use lower case to begin the name of a variable. - Do not use '_' to begin the name of the variable. '_' is to be used to begin the parameters of a pre-processor macro. - Use upper case to begin the name of a function, enum, file name etc. - Static functions whose scope is limited to the file they are defined in can be prefixed with '_'. This is not mandatory though. - For types, use all upper case for all letters with words separated by '_'. If camel casing is preferred, use upper case for the first letter. - It is a common practice to define a pointer type by prefixing the letter 'P' to a data type. The same practice can be followed here as well. For example:: static __inline BOOLEAN OvsDetectTunnelRxPkt(POVS_FORWARDING_CONTEXT ovsFwdCtx, POVS_FLOW_KEY flowKey) { POVS_VPORT_ENTRY tunnelVport = NULL; if (!flowKey->ipKey.nwFrag && flowKey->ipKey.nwProto == IPPROTO_UDP && flowKey->ipKey.l4.tpDst == VXLAN_UDP_PORT_NBO) { tunnelVport = OvsGetTunnelVport(OVSWIN_VPORT_TYPE_VXLAN); ovsActionStats.rxVxlan++; } else { return FALSE; } if (tunnelVport) { ASSERT(ovsFwdCtx->tunnelRxNic == NULL); ovsFwdCtx->tunnelRxNic = tunnelVport; return TRUE; } return FALSE; } For declaring variables of pointer type, use of the pointer data type prefixed with 'P' is preferred over using '*'. This is not mandatory though, and is only prescribed since it is a common practice in Windows. Example, #1 is preferred over #2 though #2 is also equally correct: 1. ``PNET_BUFFER_LIST curNbl;`` 2. ``NET_BUFFER_LIST *curNbl;`` Comments -------- Comments should be written as full sentences that start with a capital letter and end with a period. Putting two spaces between sentances is not necessary. ``//`` can be used for comments as long as the comment is a single line comment. For block comments, use ``/* */`` comments Functions --------- Put the return type, function name, and the braces that surround the function's code on separate lines, all starting in column 0. Before each function definition, write a comment that describes the function's purpose, including each parameter, the return value, and side effects. References to argument names should be given in single-quotes, e.g. 'arg'. The comment should not include the function name, nor need it follow any formal structure. The comment does not need to describe how a function does its work, unless this information is needed to use the function correctly (this is often better done with comments *inside* the function). Mention any side effects that the function has that are not obvious based on the name of the function or based on the workflow it is called from. In the interest of keeping comments describing functions similar in structure, use the following template. :: /* *---------------------------------------------------------------------------- * Any description of the function, arguments, return types, assumptions and * side effects. *---------------------------------------------------------------------------- */ Source Files ------------ Each source file should state its license in a comment at the very top, followed by a comment explaining the purpose of the code that is in that file. The comment should explain how the code in the file relates to code in other files. The goal is to allow a programmer to quickly figure out where a given module fits into the larger system. The first non-comment line in a .c source file should be:: #include ``#include`` directives should appear in the following order: 1. ``#include `` 2. The module's own headers, if any. Including this before any other header (besides ````) ensures that the module's header file is self-contained (see *Header Files*) below. 3. Standard C library headers and other system headers, preferably in alphabetical order. (Occasionally one encounters a set of system headers that must be included in a particular order, in which case that order must take precedence.) 4. Open vSwitch headers, in alphabetical order. Use ``""``, not ``<>``, to specify Open vSwitch header names. ovs-2.9.0/Documentation/internals/contributing/coding-style.rst000066400000000000000000000524701324262074100250140ustar00rootroot00000000000000.. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. Convention for heading levels in Open vSwitch documentation: ======= Heading 0 (reserved for the title in a document) ------- Heading 1 ~~~~~~~ Heading 2 +++++++ Heading 3 ''''''' Heading 4 Avoid deeper levels because they do not render well. ========================= Open vSwitch Coding Style ========================= This file describes the coding style used in most C files in the Open vSwitch distribution. However, Linux kernel code datapath directory follows the Linux kernel's established coding conventions. For the Windows kernel datapath code, use the coding style described in :doc:`coding-style-windows`. The following GNU indent options approximate this style. :: -npro -bad -bap -bbb -br -blf -brs -cdw -ce -fca -cli0 -npcs -i4 -l79 \ -lc79 -nbfda -nut -saf -sai -saw -sbi4 -sc -sob -st -ncdb -pi4 -cs -bs \ -di1 -lp -il0 -hnl .. _basics: Basics ------ - Limit lines to 79 characters. - Use form feeds (control+L) to divide long source files into logical pieces. A form feed should appear as the only character on a line. - Do not use tabs for indentation. - Avoid trailing spaces on lines. .. _naming: Naming ------ - Use names that explain the purpose of a function or object. - Use underscores to separate words in an identifier: ``multi_word_name``. - Use lowercase for most names. Use uppercase for macros, macro parameters, and members of enumerations. - Give arrays names that are plural. - Pick a unique name prefix (ending with an underscore) for each module, and apply that prefix to all of that module's externally visible names. Names of macro parameters, struct and union members, and parameters in function prototypes are not considered externally visible for this purpose. - Do not use names that begin with ``_``. If you need a name for "internal use only", use ``__`` as a suffix instead of a prefix. - Avoid negative names: ``found`` is a better name than ``not_found``. - In names, a ``size`` is a count of bytes, a ``length`` is a count of characters. A buffer has size, but a string has length. The length of a string does not include the null terminator, but the size of the buffer that contains the string does. .. _comments: Comments -------- Comments should be written as full sentences that start with a capital letter and end with a period. Put two spaces between sentences. Write block comments as shown below. You may put the ``/*`` and ``*/`` on the same line as comment text if you prefer. :: /* * We redirect stderr to /dev/null because we often want to remove all * traffic control configuration on a port so its in a known state. If * this done when there is no such configuration, tc complains, so we just * always ignore it. */ Each function and each variable declared outside a function, and each struct, union, and typedef declaration should be preceded by a comment. See functions_ below for function comment guidelines. Each struct and union member should each have an inline comment that explains its meaning. structs and unions with many members should be additionally divided into logical groups of members by block comments, e.g.: :: /* An event that will wake the following call to poll_block(). */ struct poll_waiter { /* Set when the waiter is created. */ struct ovs_list node; /* Element in global waiters list. */ int fd; /* File descriptor. */ short int events; /* Events to wait for (POLLIN, POLLOUT). */ poll_fd_func *function; /* Callback function, if any, or null. */ void *aux; /* Argument to callback function. */ struct backtrace *backtrace; /* Event that created waiter, or null. */ /* Set only when poll_block() is called. */ struct pollfd *pollfd; /* Pointer to element of the pollfds array (null if added from a callback). */ }; Use ``XXX`` or ``FIXME`` comments to mark code that needs work. Don't use ``//`` comments. Don't comment out or #if 0 out code. Just remove it. The code that was there will still be in version control history. .. _functions: Functions --------- Put the return type, function name, and the braces that surround the function's code on separate lines, all starting in column 0. Before each function definition, write a comment that describes the function's purpose, including each parameter, the return value, and side effects. References to argument names should be given in single-quotes, e.g. 'arg'. The comment should not include the function name, nor need it follow any formal structure. The comment does not need to describe how a function does its work, unless this information is needed to use the function correctly (this is often better done with comments *inside* the function). Simple static functions do not need a comment. Within a file, non-static functions should come first, in the order that they are declared in the header file, followed by static functions. Static functions should be in one or more separate pages (separated by form feed characters) in logical groups. A commonly useful way to divide groups is by "level", with high-level functions first, followed by groups of progressively lower-level functions. This makes it easy for the program's reader to see the top-down structure by reading from top to bottom. All function declarations and definitions should include a prototype. Empty parentheses, e.g. ``int foo();``, do not include a prototype (they state that the function's parameters are unknown); write ``void`` in parentheses instead, e.g. ``int foo(void);``. Prototypes for static functions should either all go at the top of the file, separated into groups by blank lines, or they should appear at the top of each page of functions. Don't comment individual prototypes, but a comment on each group of prototypes is often appropriate. In the absence of good reasons for another order, the following parameter order is preferred. One notable exception is that data parameters and their corresponding size parameters should be paired. 1. The primary object being manipulated, if any (equivalent to the "this" pointer in C++). 2. Input-only parameters. 3. Input/output parameters. 4. Output-only parameters. 5. Status parameter. Example: :: ``` /* Stores the features supported by 'netdev' into each of '*current', * '*advertised', '*supported', and '*peer' that are non-null. Each value * is a bitmap of "enum ofp_port_features" bits, in host byte order. * Returns 0 if successful, otherwise a positive errno value. On failure, * all of the passed-in values are set to 0. */ int netdev_get_features(struct netdev *netdev, uint32_t *current, uint32_t *advertised, uint32_t *supported, uint32_t *peer) { ... } ``` Functions that destroy an instance of a dynamically-allocated type should accept and ignore a null pointer argument. Code that calls such a function (including the C standard library function ``free()``) should omit a null-pointer check. We find that this usually makes code easier to read. Functions in ``.c`` files should not normally be marked ``inline``, because it does not usually help code generation and it does suppress compiler warnings about unused functions. (Functions defined in .h usually should be marked inline.) .. _function prototypes: Function Prototypes ------------------- Put the return type and function name on the same line in a function prototype: :: static const struct option_class *get_option_class(int code); Omit parameter names from function prototypes when the names do not give useful information, e.g.: :: int netdev_get_mtu(const struct netdev *, int *mtup); Statements ---------- Indent each level of code with 4 spaces. Use BSD-style brace placement: :: if (a()) { b(); d(); } Put a space between ``if``, ``while``, ``for``, etc. and the expressions that follow them. Enclose single statements in braces: :: if (a > b) { return a; } else { return b; } Use comments and blank lines to divide long functions into logical groups of statements. Avoid assignments inside ``if`` and ``while`` conditions. Do not put gratuitous parentheses around the expression in a return statement, that is, write ``return 0;`` and not ``return(0);`` Write only one statement per line. Indent ``switch`` statements like this: :: switch (conn->state) { case S_RECV: error = run_connection_input(conn); break; case S_PROCESS: error = 0; break; case S_SEND: error = run_connection_output(conn); break; default: OVS_NOT_REACHED(); } ``switch`` statements with very short, uniform cases may use an abbreviated style: :: switch (code) { case 200: return "OK"; case 201: return "Created"; case 202: return "Accepted"; case 204: return "No Content"; default: return "Unknown"; } Use ``for (;;)`` to write an infinite loop. In an if/else construct where one branch is the "normal" or "common" case and the other branch is the "uncommon" or "error" case, put the common case after the "if", not the "else". This is a form of documentation. It also places the most important code in sequential order without forcing the reader to visually skip past less important details. (Some compilers also assume that the "if" branch is the more common case, so this can be a real form of optimization as well.) Return Values ------------- For functions that return a success or failure indication, prefer one of the following return value conventions: - An ``int`` where 0 indicates success and a positive errno value indicates a reason for failure. - A ``bool`` where true indicates success and false indicates failure. Macros ------ Don't define an object-like macro if an enum can be used instead. Don't define a function-like macro if a "static inline" function can be used instead. If a macro's definition contains multiple statements, enclose them with ``do { ... } while (0)`` to allow them to work properly in all syntactic circumstances. Do use macros to eliminate the need to update different parts of a single file in parallel, e.g. a list of enums and an array that gives the name of each enum. For example: :: /* Logging importance levels. */ #define VLOG_LEVELS \ VLOG_LEVEL(EMER, LOG_ALERT) \ VLOG_LEVEL(ERR, LOG_ERR) \ VLOG_LEVEL(WARN, LOG_WARNING) \ VLOG_LEVEL(INFO, LOG_NOTICE) \ VLOG_LEVEL(DBG, LOG_DEBUG) enum vlog_level { #define VLOG_LEVEL(NAME, SYSLOG_LEVEL) VLL_##NAME, VLOG_LEVELS #undef VLOG_LEVEL VLL_N_LEVELS }; /* Name for each logging level. */ static const char *level_names[VLL_N_LEVELS] = { #define VLOG_LEVEL(NAME, SYSLOG_LEVEL) #NAME, VLOG_LEVELS #undef VLOG_LEVEL }; Thread Safety Annotations ------------------------- Use the macros in ``lib/compiler.h`` to annotate locking requirements. For example: :: static struct ovs_mutex mutex = OVS_MUTEX_INITIALIZER; static struct ovs_rwlock rwlock = OVS_RWLOCK_INITIALIZER; void function_require_plain_mutex(void) OVS_REQUIRES(mutex); void function_require_rwlock(void) OVS_REQ_RDLOCK(rwlock); Pass lock objects, not their addresses, to the annotation macros. (Thus we have ``OVS_REQUIRES(mutex)`` above, not ``OVS_REQUIRES(&mutex)``.) .. _source files: Source Files ------------ Each source file should state its license in a comment at the very top, followed by a comment explaining the purpose of the code that is in that file. The comment should explain how the code in the file relates to code in other files. The goal is to allow a programmer to quickly figure out where a given module fits into the larger system. The first non-comment line in a .c source file should be: :: #include ``#include`` directives should appear in the following order: 1. ``#include `` 2. The module's own headers, if any. Including this before any other header (besides ) ensures that the module's header file is self-contained (see `header files`_ below). 3. Standard C library headers and other system headers, preferably in alphabetical order. (Occasionally one encounters a set of system headers that must be included in a particular order, in which case that order must take precedence.) 4. Open vSwitch headers, in alphabetical order. Use ``""``, not ``<>``, to specify Open vSwitch header names. .. _header files: Header Files ------------ Each header file should start with its license, as described under `source files`_ above, followed by a "header guard" to make the header file idempotent, like so: :: #ifndef NETDEV_H #define NETDEV_H 1 ... #endif /* netdev.h */ Header files should be self-contained; that is, they should ``#include`` whatever additional headers are required, without requiring the client to ``#include`` them for it. Don't define the members of a struct or union in a header file, unless client code is actually intended to access them directly or if the definition is otherwise actually needed (e.g. inline functions defined in the header need them). Similarly, don't ``#include`` a header file just for the declaration of a struct or union tag (e.g. just for ``struct ;``). Just declare the tag yourself. This reduces the number of header file dependencies. Types ----- Use typedefs sparingly. Code is clearer if the actual type is visible at the point of declaration. Do not, in general, declare a typedef for a struct, union, or enum. Do not declare a typedef for a pointer type, because this can be very confusing to the reader. A function type is a good use for a typedef because it can clarify code. The type should be a function type, not a pointer-to-function type. That way, the typedef name can be used to declare function prototypes. (It cannot be used for function definitions, because that is explicitly prohibited by C89 and C99.) You may assume that ``char`` is exactly 8 bits and that ``int`` and ``long`` are at least 32 bits. Don't assume that ``long`` is big enough to hold a pointer. If you need to cast a pointer to an integer, use ``intptr_t`` or ``uintptr_t`` from . Use the ``int_t`` and ``uint_t`` types from for exact-width integer types. Use the ``PRId``, ``PRIu``, and ``PRIx`` macros from for formatting them with ``printf()`` and related functions. For compatibility with antique ``printf()`` implementations: - Instead of ``"%zu"``, use ``"%"PRIuSIZE``. - Instead of ``"%td"``, use ``"%"PRIdPTR``. - Instead of ``"%ju"``, use ``"%"PRIuMAX``. Other variants exist for different radixes. For example, use ``"%"PRIxSIZE`` instead of ``"%zx"`` or ``"%x"`` instead of ``"%hhx"``. Also, instead of ``"%hhd"``, use ``"%d"``. Be cautious substituting ``"%u"``, ``"%x"``, and ``"%o"`` for the corresponding versions with ``"hh"``: cast the argument to unsigned char if necessary, because ``printf("%hhu", -1)`` prints 255 but ``printf("%u", -1)`` prints 4294967295. Use bit-fields sparingly. Do not use bit-fields for layout of network protocol fields or in other circumstances where the exact format is important. Declare bit-fields to be signed or unsigned integer types or \_Bool (aka bool). Do *not* declare bit-fields of type ``int``: C99 allows these to be either signed or unsigned according to the compiler's whim. (A 1-bit bit-field of type ``int`` may have a range of -1...0!) Try to order structure members such that they pack well on a system with 2-byte ``short``, 4-byte ``int``, and 4- or 8-byte ``long`` and pointer types. Prefer clear organization over size optimization unless you are convinced there is a size or speed benefit. Pointer declarators bind to the variable name, not the type name. Write ``int *x``, not ``int* x`` and definitely not ``int * x``. Expressions ----------- Put one space on each side of infix binary and ternary operators: :: * / % + - << >> < <= > >= == != & ^ | && || ?: = += -= *= /= %= &= ^= |= <<= >>= Avoid comma operators. Do not put any white space around postfix, prefix, or grouping operators: :: () [] -> . ! ~ ++ -- + - * & Exception 1: Put a space after (but not before) the "sizeof" keyword. Exception 2: Put a space between the () used in a cast and the expression whose type is cast: ``(void \*) 0``. Break long lines before the ternary operators ? and :, rather than after them, e.g. :: return (out_port != VIGP_CONTROL_PATH ? alpheus_output_port(dp, skb, out_port) : alpheus_output_control(dp, skb, fwd_save_skb(skb), VIGR_ACTION)); Parenthesize the operands of ``&&`` and ``||`` if operator precedence makes it necessary, or if the operands are themselves expressions that use ``&&`` and ``||``, but not otherwise. Thus:: if (rule && (!best || rule->priority > best->priority)) { best = rule; } but:: if (!isdigit((unsigned char)s[0]) || !isdigit((unsigned char)s[1]) || !isdigit((unsigned char)s[2])) { printf("string %s does not start with 3-digit code\n", s); } Do parenthesize a subexpression that must be split across more than one line, e.g.:: *idxp = ((l1_idx << PORT_ARRAY_L1_SHIFT) | (l2_idx << PORT_ARRAY_L2_SHIFT) | (l3_idx << PORT_ARRAY_L3_SHIFT)); Breaking a long line after a binary operator gives its operands a more consistent look, since each operand has the same horizontal position. This makes the end-of-line position a good choice when the operands naturally resemble each other, as in the previous two examples. On the other hand, breaking before a binary operator better draws the eye to the operator, which can help clarify code by making it more obvious what's happening, such as in the following example:: if (!ctx.freezing && xbridge->has_in_band && in_band_must_output_to_local_port(flow) && !actions_output_to_local_port(&ctx)) { Thus, decide whether to break before or after a binary operator separately in each situation, based on which of these factors appear to be more important. Try to avoid casts. Don't cast the return value of malloc(). The "sizeof" operator is unique among C operators in that it accepts two very different kinds of operands: an expression or a type. In general, prefer to specify an expression, e.g. ``int *x = xmalloc(sizeof *\ x);``. When the operand of sizeof is an expression, there is no need to parenthesize that operand, and please don't. Use the ``ARRAY_SIZE`` macro from ``lib/util.h`` to calculate the number of elements in an array. When using a relational operator like ``<`` or ``==``, put an expression or variable argument on the left and a constant argument on the right, e.g. ``x == 0``, *not* ``0 == x``. Blank Lines ----------- Put one blank line between top-level definitions of functions and global variables. C DIALECT --------- Most C99 features are OK because they are widely implemented: - Flexible array members (e.g. ``struct { int foo[]; }``). - ``static inline`` functions (but no other forms of ``inline``, for which GCC and C99 have differing interpretations). - ``long long`` - ``bool`` and ````, but don't assume that bool or \_Bool can only take on the values 0 or 1, because this behavior can't be simulated on C89 compilers. Also, don't assume that a conversion to ``bool`` or ``_Bool`` follows C99 semantics, i.e. use ``(bool)(some_value != 0)`` rather than ``(bool)some_value``. The latter might produce unexpected results on non-C99 environments. For example, if bool is implemented as a typedef of char and ``some_value = 0x10000000``. - Designated initializers (e.g. ``struct foo foo = {.a = 1};`` and ``int a[] = {[2] = 5};``). - Mixing of declarations and code within a block. Favor positioning that allows variables to be initialized at their point of declaration. - Use of declarations in iteration statements (e.g. ``for (int i = 0; i < 10; i++)``). - Use of a trailing comma in an enum declaration (e.g. ``enum { x = 1, };``). As a matter of style, avoid ``//`` comments. Avoid using GCC or Clang extensions unless you also add a fallback for other compilers. You can, however, use C99 features or GCC extensions also supported by Clang in code that compiles only on GNU/Linux (such as ``lib/netdev-linux.c``), because GCC is the system compiler there. Python ------ When introducing new Python code, try to follow Python's `PEP 8 `__ style. Consider running the ``pep8`` or ``flake8`` tool against your code to find issues. Libraries --------- When introducing a new library, follow :doc:`Open vSwitch Library ABI guide ` ovs-2.9.0/Documentation/internals/contributing/documentation-style.rst000066400000000000000000000262511324262074100264200ustar00rootroot00000000000000.. Copyright (c) 2016 Stephen Finucane Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. Convention for heading levels in Open vSwitch documentation: ======= Heading 0 (reserved for the title in a document) ------- Heading 1 ~~~~~~~ Heading 2 +++++++ Heading 3 ''''''' Heading 4 Avoid deeper levels because they do not render well. ================================ Open vSwitch Documentation Style ================================ This file describes the documentation style used in all documentation found in Open vSwitch. Documentation includes any documents found in ``Documentation`` along with any ``README``, ``MAINTAINERS``, or generally ``rst`` suffixed documents found in the project tree. .. note:: This guide only applies to documentation for Open vSwitch v2.7. or greater. Previous versions of Open vSwitch used a combination of Markdown and raw plain text, and guidelines for these are not detailed here. reStructuredText vs. Sphinx --------------------------- `reStructuredText (rST)`__ is the syntax, while `Sphinx`__ is a documentation generator. Sphinx introduces a number of extensions to rST, like the ``:ref:`` role, which can and should be used in documentation, but these will not work correctly on GitHub. As such, these extensions should not be used in any documentation in the root level, such as the ``README``. __ http://docutils.sourceforge.net/rst.html __ http://www.sphinx-doc.org/ rST Conventions --------------- Basics ~~~~~~ Many of the basic documentation guidelines match those of the :doc:`coding-style`. - Use reStructuredText (rST) for all documentation. Sphinx extensions can be used, but only for documentation in the ``Documentation`` folder. - Limit lines at 79 characters. .. note:: An exception to this rule is text within code-block elements that cannot be wrapped and links within references. - Use spaces for indentation. - Match indentation levels. A change in indentation level usually signifies a change in content nesting, by either closing the existing level or introducing a new level. - Avoid trailing spaces on lines. - Include a license (see this file) in all docs. - Most importantly, always build and display documentation before submitting changes! Docs aren't unit testable, so visible inspection is necessary. File Names ~~~~~~~~~~ - Use hyphens as space delimiters. For example: ``my-readme-document.rst`` .. note:: An exception to this rule is any man pages, which take an trailing number corresponding to the number of arguments required. This number is preceded by an underscore. - Use lowercase filenames. .. note:: An exception to this rule is any documents found in the root-level of the project. Titles ~~~~~~ - Use the following headers levels. | ``=======`` Heading 0 (reserved for the title in a document) | ``-------`` Heading 1 | ``~~~~~~~`` Heading 2 | ``+++++++`` Heading 3 | ``'''''''`` Heading 4 .. note:: Avoid using lower heading levels by rewriting and reorganizing the information. - Under- and overlines should be of the same length as that of the heading text. - Use "title case" for headers. Code ~~~~ - Use ``::`` to prefix code. - Don't use syntax highlighting such as ``.. highlight:: `` or ``code-block:: `` because it depends on external ``pygments`` library. - Prefix commands with ``$``. - Where possible, include fully-working snippets of code. If there pre-requisites, explain what they are and how to achieve them. Admonitions ~~~~~~~~~~~ - Use admonitions to call attention to important information.:: .. note:: This is a sample callout for some useful tip or trick. Example admonitions include: ``warning``, ``important``, ``note``, ``tip`` or ``seealso``. - Use notes sparingly. Avoid having more than one per subsection. Tables ~~~~~~ - Use either graphic tables, list tables or CSV tables. Graphic tables ++++++++++++++ :: .. table:: OVS-Linux kernel compatibility ============ ============== Open vSwitch Linux kernel ============ ============== 1.4.x 2.6.18 to 3.2 1.5.x 2.6.18 to 3.2 1.6.x 2.6.18 to 3.2 ============ ============== :: .. table:: OVS-Linux kernel compatibility +--------------+---------------+ | Open vSwitch | Linux kernel | +==============+===============+ | 1.4.x | 2.6.18 to 3.2 | +--------------+---------------+ | 1.5.x | 2.6.18 to 3.2 | +--------------+---------------+ | 1.6.x | 2.6.18 to 3.2 | +--------------+---------------+ .. note:: The ``table`` role - ``.. table:: `` - can be safely omitted. List tables +++++++++++ :: .. list-table:: OVS-Linux kernel compatibility :widths: 10 15 :header-rows: 1 * - Open vSwitch - Linux kernel * - 1.4.x - 2.6.18 to 3.2 * - 1.5.x - 2.6.18 to 3.2 * - 1.6.x - 2.6.18 to 3.2 CSV tables ++++++++++ :: .. csv-table:: OVS-Linux kernel compatibility :header: Open vSwitch, Linux kernel :widths: 10 15 1.4.x, 2.6.18 to 3.2 1.5.x, 2.6.18 to 3.2 1.6.x, 2.6.18 to 3.2 Cross-referencing ~~~~~~~~~~~~~~~~~ - To link to an external file or document, include as a link.:: Here's a `link `__ to the Open vSwitch website. Here's a `link`_ in reference style. .. _link: http://openvswitch.org - You can also use citations.:: Refer to the Open vSwitch documentation [1]_. References ---------- .. [1]: http://openvswitch.org - To cross-reference another doc, use the ``doc`` role.:: Here is a link to the :doc:`/README.rst` .. note:: This is a Sphinx extension. Do not use this in any top-level documents. - To cross-reference an arbitrary location in a doc, use the ``ref`` role.:: .. _sample-crossref Title ~~~~~ Hello, world. Another Title ~~~~~~~~~~~~~ Here is a cross-reference to :ref:`sample-crossref`. .. note:: This is a Sphinx extension. Do not use this in any top-level documents. Figures and Other Media ~~~~~~~~~~~~~~~~~~~~~~~ - All images should be in PNG format and compressed where possible. For PNG files, use OptiPNG and AdvanceCOMP's ``advpng``: :: $ optipng -o7 -zm1-9 -i0 -strip all $ advpng -z4 - Any ASCII text "images" should be included in code-blocks to preserve formatting - Include other reStructuredText verbatim in a current document Comments ~~~~~~~~ - Comments are indicated by means of the ``..`` marker.:: .. TODO(stephenfin) This section needs some work. This TODO will not appear in the final generated document, however. Man Pages --------- In addition to the above, man pages have some specific requirements: - You **must** define the following sections: - Synopsis - Description - Options Note that `NAME` is not included - this is automatically generated by Sphinx and should not be manually defined. Also note that these do not need to be uppercase - Sphinx will do this automatically. Additional sections are allowed. Refer to `man-pages(8)` for information on the sections generally allowed. - You **must not** define a `NAME` section. See above. - The `OPTIONS` section must describe arguments and options using the `program`__ and `option`__ directives. This ensures the output is formatted correctly and that you can cross-reference various programs and commands from the documentation. For example:: .. program:: ovs-do-something .. option:: -f, --force Force the operation .. option:: -b , --bridge Name or ID of bridge .. important:: Option argument names should be enclosed in angle brackets, as above. - Any references to the application or any other Open vSwitch application must be marked up using the `program` role. This allows for easy linking in the HTML output and correct formatting in the man page output. For example:: To do something, run :program:`ovs-do-something`. - The man page must be included in the list of man page documents found in `conf.py`__ Refer to existing man pages, such as :doc:`/ref/ovs-vlan-test.8` for a worked example. __ http://www.sphinx-doc.org/en/stable/domains.html#directive-program __ http://www.sphinx-doc.org/en/stable/domains.html#directive-option __ http://www.sphinx-doc.org/en/stable/config.html#confval-man_pages Writing Style ------------- Follow these guidelines to ensure readability and consistency of the Open vSwitch documentation. These guidelines are based on the `IBM Style Guide `__. - Use standard US English Use a spelling and grammar checking tool as necessary. - Expand initialisms and acronyms on first usage. Commonly used terms like CPU or RAM are allowed. .. list-table:: :header-rows: 1 * - Do not use - Do use * - OVS is a virtual switch. OVS has... - Open vSwitch (OVS) is a virtual switch. OVS has... * - The VTEP emulator is... - The Virtual Tunnel Endpoint (VTEP) emulator is... - Write in the active voice The subject should do the verb's action, rather than be acted upon. .. list-table:: :header-rows: 1 * - Do not use - Do use * - A bridge is created by you - Create a bridge - Write in the present tense .. list-table:: :header-rows: 1 * - Do not use - Do use * - Once the bridge is created, you can create a port - Once the bridge is created, create a port - Write in second person .. list-table:: :header-rows: 1 * - Do not use - Do use * - To create a bridge, the user runs: - To create a bridge, run: - Keep sentences short and consise - Eliminate needless politeness Avoid "please" and "thank you" Helpful Tools ------------- There are a number of tools, online and offline, which can be used to preview documents are you edit them: - `rst.ninjs.org `__ An online rST editor/previewer - `ReText `__ A simple but powerful editor for Markdown and reStructuredText. ReText is written in Python. - `restview `__ A viewer for ReStructuredText documents that renders them on the fly. Useful Links ------------ - `Quick reStructuredText `__ - `Sphinx Documentation `__ ovs-2.9.0/Documentation/internals/contributing/index.rst000066400000000000000000000022611324262074100235130ustar00rootroot00000000000000.. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. Convention for heading levels in Open vSwitch documentation: ======= Heading 0 (reserved for the title in a document) ------- Heading 1 ~~~~~~~ Heading 2 +++++++ Heading 3 ''''''' Heading 4 Avoid deeper levels because they do not render well. ============================ Contributing to Open vSwitch ============================ The below guides provide information on contributing to Open vSwitch itself. .. toctree:: :maxdepth: 2 submitting-patches backporting-patches coding-style coding-style-windows documentation-style libopenvswitch-abi ovs-2.9.0/Documentation/internals/contributing/libopenvswitch-abi.rst000066400000000000000000000120171324262074100261750ustar00rootroot00000000000000.. Copyright (c) 2017 Red Hat, Inc. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. Convention for heading levels in Open vSwitch documentation: ======= Heading 0 (reserved for the title in a document) ------- Heading 1 ~~~~~~~ Heading 2 +++++++ Heading 3 ''''''' Heading 4 Avoid deeper levels because they do not render well. =================================== Open vSwitch Library ABI Updates =================================== This file describes the manner in which the Open vSwitch shared library manages different ABI and API revisions. This document aims to describe the background, goals, and concrete mechanisms used to export code-space functionality so that it may be shared between multiple applications. .. _definitions: Definitions ----------- .. csv-table:: Definitions for terms appearing in this document :header: "Term", "Definition" "ABI", "Abbreviation of Application Binary Interface" "API", "Abbreviation of Application Programming Interface" "Application Binary Interface", "The low-level runtime interface exposed by an object file." "Application Programming Interface", "The source-code interface descriptions intended for use in multiple translation units when compiling." "Code library", "A collection of function implementations and definitions intended to be exported and called through a well-defined interface." "Shared Library", "A code library which is imported at run time." .. _overview: Overview ---------- C and C++ applications often use 'external' functionality, such as printing specialized data types or parsing messages, which has been exported for common use. There are many possible ways for applications to call such external functionality, for instance by including an appropriate inline definition which the compiler can emit as code in each function it appears. One such way of exporting and importing such functionality is through the use of a library of code. When a compiler builds object code from source files to produce object code, the results are binary data arranged with specific calling conventions, alignments, and order suitable for a run-time environment or linker. This result defines a specific ABI. As library of code develops and its exported interfaces change over time, the resulting ABI may change as well. Therefore, care must be taken to ensure the changes made to libraries of code are effectively communicated to applications which use them. This includes informing the applications when incompatible changes are made. The Open vSwitch project exports much of its functionality through multiple such libraries of code. These libraries are intended for multiple applications to import and use. As the Open vSwitch project continues to evolve and change, its exported code will evolve as well. To ensure that applications linking to these libraries are aware of these changes, Open vSwitch employs libtool version stamps. .. _policies: ABI Policy ---------- Open vSwitch will export the ABI version at the time of release, such that the library name will be the major.minor version, and the rest of the release version information will be conveyed with a libtool interface version. The intent is for Open vSwitch to maintain an ABI stability for each minor revision only (so that Open vSwitch release 2.5 carries a guarantee for all 2.5.ZZ micro-releases). This means that any porting effort to stable branches must take not to disrupt the existing ABI. In the event that a bug must be fixed in a backwards-incompatible way, developers must bump the libtool 'current' version to inform the linker of the ABI breakage. This will signal that libraries exposed by the subsequent release will not maintain ABI stability with the previous version. Coding ------- At build time, if building shared libraries by passing the `--enable-shared` arguments to `./configure`, version information is extracted from the ``$PACKAGE_VERSION`` automake variable and formatted into the appropriate arguments. These get exported for use in Makefiles as ``$OVS_LTINFO``, and passed to each exported library along with other ``LDFLAGS``. Therefore, when adding a new library to the build system, these version flags should be included with the ``$LDFLAGS`` variable. Nothing else needs to be done. Changing an exported function definition (from a file in, for instance `lib/*.h`) is only permitted from minor release to minor release. Likewise changes to library data structures should only occur from minor release to minor release. ovs-2.9.0/Documentation/internals/contributing/submitting-patches.rst000066400000000000000000000421431324262074100262210ustar00rootroot00000000000000.. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. Convention for heading levels in Open vSwitch documentation: ======= Heading 0 (reserved for the title in a document) ------- Heading 1 ~~~~~~~ Heading 2 +++++++ Heading 3 ''''''' Heading 4 Avoid deeper levels because they do not render well. ================== Submitting Patches ================== Send changes to Open vSwitch as patches to dev@openvswitch.org. One patch per email. More details are included below. If you are using Git, then `git format-patch` takes care of most of the mechanics described below for you. Before You Start ---------------- Before you send patches at all, make sure that each patch makes sense. In particular: - A given patch should not break anything, even if later patches fix the problems that it causes. The source tree should still build and work after each patch is applied. (This enables `git bisect` to work best.) - A patch should make one logical change. Don't make multiple, logically unconnected changes to disparate subsystems in a single patch. - A patch that adds or removes user-visible features should also update the appropriate user documentation or manpages. Consider adding an item to NEWS for nontrivial changes. Check "Feature Deprecation Guidelines" section in this document if you intend to remove user-visible feature. Testing is also important: - Test a patch that modifies existing code with ``make check`` before submission. Refer to the "Unit Tests" in :doc:`/topics/testing`, for more information. We also encourage running the kernel and userspace system tests. - Consider testing a patch that adds or deletes files with ``make distcheck`` before submission. - A patch that modifies Linux kernel code should be at least build-tested on various Linux kernel versions before submission. I suggest versions 3.10 and whatever the current latest release version is at the time. - A patch that adds a new feature should add appropriate tests for the feature. A bug fix patch should preferably add a test that would fail if the bug recurs. If you are using GitHub, then you may utilize the travis-ci.org CI build system by linking your GitHub repository to it. This will run some of the above tests automatically when you push changes to your repository. See the "Continuous Integration with Travis-CI" in :doc:`/topics/testing` for details on how to set it up. Email Subject ------------- The subject line of your email should be in the following format: [PATCH /] :

Where: ``[PATCH /]``: indicates that this is the nth of a series of m patches. It helps reviewers to read patches in the correct order. You may omit this prefix if you are sending only one patch. ````: indicates the area of the Open vSwitch to which the change applies (often the name of a source file or a directory). You may omit it if the change crosses multiple distinct pieces of code. ````: briefly describes the change. Use the the imperative form, e.g. "Force SNAT for multiple gateway routers." or "Fix daemon exit for bad datapaths or flows." Try to keep the summary short, about 50 characters wide. The subject, minus the ``[PATCH /]`` prefix, becomes the first line of the commit's change log message. Description ----------- The body of the email should start with a more thorough description of the change. This becomes the body of the commit message, following the subject. There is no need to duplicate the summary given in the subject. Please limit lines in the description to 75 characters in width. That allows the description to format properly even when indented (e.g. by "git log" or in email quotations). The description should include: - The rationale for the change. - Design description and rationale (but this might be better added as code comments). - Testing that you performed (or testing that should be done but you could not for whatever reason). - Tags (see below). There is no need to describe what the patch actually changed, if the reader can see it for himself. If the patch refers to a commit already in the Open vSwitch repository, please include both the commit number and the subject of the patch, e.g. 'commit 632d136c (vswitch: Remove restriction on datapath names.)'. If you, the person sending the patch, did not write the patch yourself, then the very first line of the body should take the form ``From: ``, followed by a blank line. This will automatically cause the named author to be credited with authorship in the repository. Tags ---- The description ends with a series of tags, written one to a line as the last paragraph of the email. Each tag indicates some property of the patch in an easily machine-parseable manner. Examples of common tags follow. ``Signed-off-by: Author Name `` Informally, this indicates that Author Name is the author or submitter of a patch and has the authority to submit it under the terms of the license. The formal meaning is to agree to the Developer's Certificate of Origin (see below). If the author and submitter are different, each must sign off. If the patch has more than one author, all must sign off. :: Signed-off-by: Author Name Signed-off-by: Submitter Name ``Co-authored-by: Author Name `` Git can only record a single person as the author of a given patch. In the rare event that a patch has multiple authors, one must be given the credit in Git and the others must be credited via Co-authored-by: tags. (All co-authors must also sign off.) ``Acked-by: Reviewer Name `` Reviewers will often give an ``Acked-by:`` tag to code of which they approve. It is polite for the submitter to add the tag before posting the next version of the patch or applying the patch to the repository. Quality reviewing is hard work, so this gives a small amount of credit to the reviewer. Not all reviewers give ``Acked-by:`` tags when they provide positive reviews. It's customary only to add tags from reviewers who actually provide them explicitly. ``Tested-by: Tester Name `` When someone tests a patch, it is customary to add a Tested-by: tag indicating that. It's rare for a tester to actually provide the tag; usually the patch submitter makes the tag himself in response to an email indicating successful testing results. ``Tested-at: `` When a test report is publicly available, this provides a way to reference it. Typical s would be build logs from autobuilders or references to mailing list archives. Some autobuilders only retain their logs for a limited amount of time. It is less useful to cite these because they may be dead links for a developer reading the commit message months or years later. ``Reported-by: Reporter Name `` When a patch fixes a bug reported by some person, please credit the reporter in the commit log in this fashion. Please also add the reporter's name and email address to the list of people who provided helpful bug reports in the AUTHORS file at the top of the source tree. Fairly often, the reporter of a bug also tests the fix. Occasionally one sees a combined "Reported-and-tested-by:" tag used to indicate this. It is also acceptable, and more common, to include both tags separately. (If a bug report is received privately, it might not always be appropriate to publicly credit the reporter. If in doubt, please ask the reporter.) ``Requested-by: Requester Name `` When a patch implements a request or a suggestion made by some person, please credit that person in the commit log in this fashion. For a helpful suggestion, please also add the person's name and email address to the list of people who provided suggestions in the AUTHORS file at the top of the source tree. (If a suggestion or a request is received privately, it might not always be appropriate to publicly give credit. If in doubt, please ask.) ``Suggested-by: Suggester Name `` See ``Requested-by:``. ``CC: Person `` This is a way to tag a patch for the attention of a person when no more specific tag is appropriate. One use is to request a review from a particular person. It doesn't make sense to include the same person in CC and another tag, so e.g. if someone who is CCed later provides an Acked-by, add the Acked-by and remove the CC at the same time. ``Reported-at: `` If a patch fixes or is otherwise related to a bug reported in a public bug tracker, please include a reference to the bug in the form of a URL to the specific bug, e.g.: :: Reported-at: https://bugs.debian.org/743635 This is also an appropriate way to refer to bug report emails in public email archives, e.g.: :: Reported-at: https://mail.openvswitch.org/pipermail/ovs-dev/2014-June/284495.html ``Submitted-at: `` If a patch was submitted somewhere other than the Open vSwitch development mailing list, such as a GitHub pull request, this header can be used to reference the source. :: Submitted-at: https://github.com/openvswitch/ovs/pull/92 ``VMware-BZ: #1234567`` If a patch fixes or is otherwise related to a bug reported in a private bug tracker, you may include some tracking ID for the bug for your own reference. Please include some identifier to make the origin clear, e.g. "VMware-BZ" refers to VMware's internal Bugzilla instance and "ONF-JIRA" refers to the Open Networking Foundation's JIRA bug tracker. ``ONF-JIRA: EXT-12345`` See ``VMware-BZ:``. ``Bug #1234567.`` These are obsolete forms of VMware-BZ: that can still be seen in old change log entries. (They are obsolete because they do not tell the reader what bug tracker is referred to.) ``Issue: 1234567`` See ``Bug:``. ``Fixes: 63bc9fb1c69f (“packets: Reorder CS_* flags to remove gap.”)`` If you would like to record which commit introduced a bug being fixed, you may do that with a “Fixes” header. This assists in determining which OVS releases have the bug, so the patch can be applied to all affected versions. The easiest way to generate the header in the proper format is with this git command. This command also CCs the author of the commit being fixed, which makes sense unless the author also made the fix or is already named in another tag: :: $ git log -1 --pretty=format:"CC: %an <%ae>%nFixes: %h (\"%s\")" \ --abbrev=12 COMMIT_REF ``Vulnerability: CVE-2016-2074`` Specifies that the patch fixes or is otherwise related to a security vulnerability with the given CVE identifier. Other identifiers in public vulnerability databases are also suitable. If the vulnerability was reported publicly, then it is also appropriate to cite the URL to the report in a Reported-at tag. Use a Reported-by tag to acknowledge the reporters. Developer's Certificate of Origin --------------------------------- To help track the author of a patch as well as the submission chain, and be clear that the developer has authority to submit a patch for inclusion in openvswitch please sign off your work. The sign off certifies the following: :: Developer's Certificate of Origin 1.1 By making a contribution to this project, I certify that: (a) The contribution was created in whole or in part by me and I have the right to submit it under the open source license indicated in the file; or (b) The contribution is based upon previous work that, to the best of my knowledge, is covered under an appropriate open source license and I have the right under that license to submit that work with modifications, whether created in whole or in part by me, under the same open source license (unless I am permitted to submit under a different license), as indicated in the file; or (c) The contribution was provided directly to me by some other person who certified (a), (b) or (c) and I have not modified it. (d) I understand and agree that this project and the contribution are public and that a record of the contribution (including all personal information I submit with it, including my sign-off) is maintained indefinitely and may be redistributed consistent with this project or the open source license(s) involved. See also http://developercertificate.org/. Feature Deprecation Guidelines ------------------------------ Open vSwitch is intended to be user friendly. This means that under normal circumstances we don't abruptly remove features from OVS that some users might still be using. Otherwise, if we would, then we would possibly break our user setup when they upgrade and would receive bug reports. Typical process to deprecate a feature in Open vSwitch is to: (a) Mention deprecation of a feature in the NEWS file. Also, mention expected release or absolute time when this feature would be removed from OVS altogether. Don't use relative time (e.g. "in 6 months") because that is not clearly interpretable. (b) If Open vSwitch is configured to use deprecated feature it should print a warning message to the log files clearly indicating that feature is deprecated and that use of it should be avoided. (c) If this feature is mentioned in man pages, then add "Deprecated" keyword to it. Also, if there is alternative feature to the one that is about to be marked as deprecated, then mention it in (a), (b) and (c) as well. Remember to follow-up and actually remove the feature from OVS codebase once deprecation grace period has expired and users had opportunity to use at least one OVS release that would have informed them about feature deprecation! Comments -------- If you want to include any comments in your email that should not be part of the commit's change log message, put them after the description, separated by a line that contains just ``---``. It may be helpful to include a diffstat here for changes that touch multiple files. Patch ----- The patch should be in the body of the email following the description, separated by a blank line. Patches should be in ``diff -up`` format. We recommend that you use Git to produce your patches, in which case you should use the ``-M -C`` options to ``git diff`` (or other Git tools) if your patch renames or copies files. `Quilt `__ might be useful if you do not want to use Git. Patches should be inline in the email message. Some email clients corrupt white space or wrap lines in patches. There are hints on how to configure many email clients to avoid this problem on `kernel.org `__. If you cannot convince your email client not to mangle patches, then sending the patch as an attachment is a second choice. Follow the style used in the code that you are modifying. :doc:`coding-style` file describes the coding style used in most of Open vSwitch. Use Linux kernel coding style for Linux kernel code. If your code is non-datapath code, you may use the ``utilities/checkpatch.py`` utility as a quick check for certain commonly occuring mistakes (improper leading/trailing whitespace, missing signoffs, some improper formatted patch files). For linux datapath code, it is a good idea to use the linux script ``checkpatch.pl``. Example ------- :: From fa29a1c2c17682879e79a21bb0cdd5bbe67fa7c0 Mon Sep 17 00:00:00 2001 From: Jesse Gross Date: Thu, 8 Dec 2011 13:17:24 -0800 Subject: [PATCH] datapath: Alphabetize include/net/ipv6.h compat header. Signed-off-by: Jesse Gross --- datapath/linux/Modules.mk | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/datapath/linux/Modules.mk b/datapath/linux/Modules.mk index fdd952e..f6cb88e 100644 --- a/datapath/linux/Modules.mk +++ b/datapath/linux/Modules.mk @@ -56,11 +56,11 @@ openvswitch_headers += \ linux/compat/include/net/dst.h \ linux/compat/include/net/genetlink.h \ linux/compat/include/net/ip.h \ + linux/compat/include/net/ipv6.h \ linux/compat/include/net/net_namespace.h \ linux/compat/include/net/netlink.h \ linux/compat/include/net/protocol.h \ linux/compat/include/net/route.h \ - linux/compat/include/net/ipv6.h \ linux/compat/genetlink.inc both_modules += brcompat -- 1.7.7.3 ovs-2.9.0/Documentation/internals/documentation.rst000066400000000000000000000060431324262074100225500ustar00rootroot00000000000000.. Copyright (c) 2017 Stephen Finucane Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. Convention for heading levels in Open vSwitch documentation: ======= Heading 0 (reserved for the title in a document) ------- Heading 1 ~~~~~~~ Heading 2 +++++++ Heading 3 ''''''' Heading 4 Avoid deeper levels because they do not render well. ====================================== How Open vSwitch's Documentation Works ====================================== This document provides a brief overview on how the documentation build system within Open vSwitch works. This is intended to maximize the "bus factor" and share best practices with other projects. reStructuredText and Sphinx --------------------------- Nearly all of Open vSwitch's documentation is written in `reStructuredText`__, with man pages being the sole exception. Of this documentation, most of it is fed into `Sphinx`__, which provides not only the ability to convert rST to a variety of other output formats but also allows for things like cross-referencing and indexing. for more information on the two, refer to the :doc:`contributing/documentation-style`. ovs-sphinx-theme ---------------- The documentation uses its own theme, `ovs-sphinx-theme`, which can be found on GitHub__ and is published on pypi__. This is packaged separately from Open vSwitch itself to ensure all documentation gets the latest version of the theme (assuming there are no major version bumps in that package). If building locally and the package is installed, it will be used. If the package is not installed, Sphinx will fallback to the default theme. The package is currently maintained by Stephen Finucane and Russell Bryant. Read the Docs ------------- The documentation is hosted on readthedocs.org and a CNAME redirect is in place to allow access from docs.openvswitch.org. *Read the Docs* provides a couple of nifty features for us, such as automatic building of docs whenever there are changes and versioning of documentation. The *Read the Docs* project is currently maintained by Stephen Finucane, Russell Bryant and Ben Pfaff. openvswitch.org --------------- The sources for openvswitch.org are maintained separately from docs.openvswitch.org. For modifications to this site, refer to the `GitHub project`__. __ http://docutils.sourceforge.net/rst.html __ http://www.sphinx-doc.org/ __ https://github.com/openvswitch/ovs-sphinx-theme __ https://pypi.python.org/pypi/ovs-sphinx-theme __ https://github.com/openvswitch/openvswitch.github.io ovs-2.9.0/Documentation/internals/index.rst000066400000000000000000000025521324262074100210070ustar00rootroot00000000000000.. Copyright (c) 2016, Stephen Finucane Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. Convention for heading levels in Open vSwitch documentation: ======= Heading 0 (reserved for the title in a document) ------- Heading 1 ~~~~~~~ Heading 2 +++++++ Heading 3 ''''''' Heading 4 Avoid deeper levels because they do not render well. ====================== Open vSwitch Internals ====================== Information for people who want to know more about the Open vSwitch project itself and how they might involved. .. toctree:: :maxdepth: 2 contributing/index mailing-lists patchwork release-process bugs security charter committer-emeritus-status committer-responsibilities committer-grant-revocation authors maintainers documentation ovs-2.9.0/Documentation/internals/mailing-lists.rst000066400000000000000000000054671324262074100224640ustar00rootroot00000000000000.. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. Convention for heading levels in Open vSwitch documentation: ======= Heading 0 (reserved for the title in a document) ------- Heading 1 ~~~~~~~ Heading 2 +++++++ Heading 3 ''''''' Heading 4 Avoid deeper levels because they do not render well. ============= Mailing Lists ============= .. important:: Report security issues **only** to security@openvswitch.org. For more information, refer to our :doc:`security policies `. ovs-announce ------------ The `ovs-announce`__ mailing list is used to announce new versions of Open vSwitch and is extremely low-volume. `(subscribe)`__ `(archives)`__ __ ovs-announce@openvswitch.org __ https://mail.openvswitch.org/mailman/listinfo/ovs-announce/ __ https://mail.openvswitch.org/pipermail/ovs-announce/ ovs-discuss ----------- The `ovs-discuss`__ mailing list is used to discuss plans and design decisions for Open vSwitch. It is also an appropriate place for user questions. `(subscribe)`__ `(archives)`__ __ ovs-discuss@openvswitch.org __ https://mail.openvswitch.org/mailman/listinfo/ovs-discuss/ __ https://mail.openvswitch.org/pipermail/ovs-discuss/ ovs-dev ------- The `ovs-dev`__ mailing list is used to discuss development and review code before being committed. `(subscribe)`__ `(archives)`__ __ ovs-dev@openvswitch.org __ https://mail.openvswitch.org/mailman/listinfo/ovs-dev/ __ https://mail.openvswitch.org/pipermail/ovs-dev/ ovs-git ------- The `ovs-git`__ mailing list hooks into Open vSwitch's version control system to receive commits. `(subscribe)`__ `(archives)`__ __ ovs-git@openvswitch.org __ https://mail.openvswitch.org/mailman/listinfo/ovs-git/ __ https://mail.openvswitch.org/pipermail/ovs-git/ ovs-build --------- The `ovs-build`__ mailing list hooks into Open vSwitch's continuous integration system to receive build reports. `(subscribe)`__ `(archives)`__ __ ovs-build@openvswitch.org __ https://mail.openvswitch.org/mailman/listinfo/ovs-build/ __ https://mail.openvswitch.org/pipermail/ovs-build/ bugs ----- The `bugs`__ mailing list is an alias for the discuss mailing list. __ bugs@openvswitch.org security -------- The `security`__ mailing list is for submitting security vulnerabilities to the security team. __ security@ovs.org ovs-2.9.0/Documentation/internals/maintainers.rst000066400000000000000000000016161324262074100222120ustar00rootroot00000000000000.. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. Convention for heading levels in Open vSwitch documentation: ======= Heading 0 (reserved for the title in a document) ------- Heading 1 ~~~~~~~ Heading 2 +++++++ Heading 3 ''''''' Heading 4 Avoid deeper levels because they do not render well. .. include:: ../../MAINTAINERS.rst ovs-2.9.0/Documentation/internals/patchwork.rst000066400000000000000000000055151324262074100217040ustar00rootroot00000000000000.. Copyright (C) 2016, Stephen Finucane Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. Convention for heading levels in Open vSwitch documentation: ======= Heading 0 (reserved for the title in a document) ------- Heading 1 ~~~~~~~ Heading 2 +++++++ Heading 3 ''''''' Heading 4 Avoid deeper levels because they do not render well. ========= Patchwork ========= Open vSwitch uses `Patchwork`__ to track the status of patches sent to the :doc:`ovs-dev mailing list `. The Open vSwitch Patchwork instance can be found on `ozlabs.org`__. Patchwork provides a number of useful features for developers working on Open vSwitch: - Tracking the lifecycle of patches (accepted, rejected, under-review, ...) - Assigning reviewers (delegates) to patches - Downloading/applying patches, series, and bundles via the web UI or the REST API (see :ref:`git-pw`) - A usable UI for viewing patch discussions __ https://github.com/getpatchwork/patchwork __ https://patchwork.ozlabs.org/project/openvswitch/list/ .. _git-pw: git-pw ------ The *git-pw* tool provides a way to download and apply patches, series, and bundles. You can install *git-pw* from `PyPi`__ like so:: $ pip install --user git-pw To actually use *git-pw*, you must configure it with the Patchwork instance URL, Patchwork project, and your Patchwork user authentication token. The URL and project are provided below, but you must obtain your authentication token from your `Patchwork User Profile`__ page. If you do not already have a Patchwork user account, you should create one now. Once your token is obtained, configure *git-pw* as below. Note that this must be run from within the Open vSwitch Git repository:: $ git config pw.server https://patchwork.ozlabs.org/ $ git config pw.project openvswitch $ git config pw.token $PW_TOKEN # using the token obtained earlier Once configured, run the following to get information about available commands:: $ git pw --help __ https://pypi.python.org/pypi/git-pw __ https://patchwork.ozlabs.org/user/ .. _pwclient: pwclient -------- The *pwclient* is a legacy tool that provides some of the functionality of *git-pw* but uses the legacy XML-RPC API. It is considered deprecated in its current form and *git-pw* should be used instead. ovs-2.9.0/Documentation/internals/release-process.rst000066400000000000000000000135331324262074100227750ustar00rootroot00000000000000.. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. Convention for heading levels in Open vSwitch documentation: ======= Heading 0 (reserved for the title in a document) ------- Heading 1 ~~~~~~~ Heading 2 +++++++ Heading 3 ''''''' Heading 4 Avoid deeper levels because they do not render well. ============================ Open vSwitch Release Process ============================ This document describes the process ordinarily used for Open vSwitch development and release. Exceptions are sometimes necessary, so all of the statements here should be taken as subject to change through rough consensus of Open vSwitch contributors, obtained through public discussion on, e.g., ovs-dev or the #openvswitch IRC channel. Release Strategy ---------------- Open vSwitch feature development takes place on the "master" branch. Ordinarily, new features are rebased against master and applied directly. For features that take significant development, sometimes it is more appropriate to merge a separate branch into master; please discuss this on ovs-dev in advance. Periodically, the OVS developers fork a branch from master to become an official release. These release branches are named for expected release number, e.g. "branch-2.3" for the branch that will yield Open vSwitch 2.3.x. Release branches should receive only bug fixes, not new features. Bug fixes applied to release branches should be backports of corresponding bug fixes to the master branch, except for bugs present only on release branches (which are rare in practice). Sometimes there can be exceptions to the rule that a release branch receives only bug fixes. In particular, after a release branch is created, but before the first actual release from that branch, it can be appropriate to add features. Like bug fixes, new features on release branches should be backports of the corresponding commits on the master branch. Features to be added to release branches should be limited in scope and risk and discussed on ovs-dev before creating the branch. After a period of testing and stabilization, and rough consensus obtained from contributors that the release is ready, the developers release the .0 release on its branch, e.g. 2.3.0 for branch-2.3. To make the actual release, a developer pushes a signed tag named, e.g. v2.3.0, to the Open vSwitch repository, makes a release tarball available on openvswitch.org, and posts a release announcement to ovs-announce. As a number of bug fixes accumulate, or after important bugs or vulnerabilities are fixed, the OVS developers may make additional releases from a branch: 2.3.1, 2.3.2, and so on. The process is the same for these additional release as for a .0 release. At most two release branches are formally maintained at any given time: the latest release and the latest release designed as LTS. An LTS release is one that the OVS project has designated as being maintained for a longer period of time. Currently, an LTS release is maintained until the next LTS is chosen. There is not currently a strict guideline on how often a new LTS release is chosen, but so far it has been about every 2 years. That could change based on the current state of OVS development. For example, we do not want to designate a new release as LTS that includes disruptive internal changes, as that may make it harder to support for a longer period of time. Discussion about choosing the next LTS release occurs on the OVS development mailing list. Release Numbering ----------------- The version number on master should normally end in .90. This indicates that the Open vSwitch version is "almost" the next version to branch. Forking master into branch-x.y requires two commits to master. The first is titled "Prepare for x.y.0" and increments the version number to x.y. This is the initial commit on branch-x.y. The second is titled "Prepare for post-x.y.0 (x.y.90)" and increments the version number to x.y.90. The version number on a release branch is x.y.z, where z is initially 0. Making a release requires two commits. The first is titled *Set release dates for x.y.z.* and updates NEWS and debian/changelog to specify the release date of the new release. This commit is the one made into a tarball and tagged. The second is titled *Prepare for x.y.(z+1).* and increments the version number and adds a blank item to NEWS with an unspecified date. Release Scheduling ------------------ Open vSwitch makes releases at the following six-month cadence, which of course is subject to change. +-------------------+-----------------------+--------------------------------------+ | **Time (months)** | **Approximate Dates** | **Event** | +-------------------+-----------------------+--------------------------------------+ | T | Mar 1, Sep 1 | Release cycle for version x.y begins | +-------------------+-----------------------+--------------------------------------+ | T + 4 | Jul 1, Jan 1 | branch-x.y forks from master | +-------------------+-----------------------+--------------------------------------+ | T + 5.5 | Aug 15, Feb 15 | branch-x.y released as version x.y.0 | +-------------------+-----------------------+--------------------------------------+ Contact ------- Use dev@openvswitch.org to discuss the Open vSwitch development and release process. ovs-2.9.0/Documentation/internals/security.rst000066400000000000000000000270141324262074100215470ustar00rootroot00000000000000.. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. Convention for heading levels in Open vSwitch documentation: ======= Heading 0 (reserved for the title in a document) ------- Heading 1 ~~~~~~~ Heading 2 +++++++ Heading 3 ''''''' Heading 4 Avoid deeper levels because they do not render well. =============================== Open vSwitch's Security Process =============================== This is a proposed security vulnerability reporting and handling process for Open vSwitch. It is based on the OpenStack vulnerability management process described at https://wiki.openstack.org/wiki/Vulnerability\_Management. The OVS security team coordinates vulnerability management using the ovs-security mailing list. Membership in the security team and subscription to its mailing list consists of a small number of trustworthy people, as determined by rough consensus of the Open vSwitch committers on the ovs-committers mailing list. The Open vSwitch security team should include Open vSwitch committers, to ensure prompt and accurate vulnerability assessments and patch review. We encourage everyone involved in the security process to GPG-sign their emails. We additionally encourage GPG-encrypting one-on-one conversations as part of the security process. What is a vulnerability? ------------------------ All vulnerabilities are bugs, but not every bug is a vulnerability. Vulnerabilities compromise one or more of: * Confidentiality (personal or corporate confidential data). * Integrity (trustworthiness and correctness). * Availability (uptime and service). Here are some examples of vulnerabilities to which one would expect to apply this process: * A crafted packet that causes a kernel or userspace crash (Availability). * A flow translation bug that misforwards traffic in a way likely to hop over security boundaries (Integrity). * An OpenFlow protocol bug that allows a controller to read arbitrary files from the file system (Confidentiality). * Misuse of the OpenSSL library that allows bypassing certificate checks (Integrity). * A bug (memory corruption, overflow, ...) that allows one to modify the behaviour of OVS through external configuration interfaces such as OVSDB (Integrity). * Privileged information is exposed to unprivileged users (Confidentiality). If in doubt, please do use the vulnerability management process. At worst, the response will be to report the bug through the usual channels. Step 1: Reception ----------------- To report an Open vSwitch vulnerability, send an email to the ovs-security mailing list (see contact_ at the end of this document). A security team member should reply to the reporter acknowledging that the report has been received. Consider reporting the information mentioned in :doc:`bugs`, where relevant. Reporters may ask for a GPG key while initiating contact with the security team to deliver more sensitive reports. The Linux kernel has `its own vulnerability management process `__. Handling of vulnerabilities that affect both the Open vSwitch tree and the upstream Linux kernel should be reported through both processes. Send your report as a single email to both the kernel and OVS security teams to allow those teams to most easily coordinate among themselves. Step 2: Assessment ------------------ The security team should discuss the vulnerability. The reporter should be included in the discussion (via "CC") to an appropriate degree. The assessment should determine which Open vSwitch versions are affected (e.g. every version, only the latest release, only unreleased versions), the privilege required to take advantage of the vulnerability (e.g. any network user, any local L2 network user, any local system user, connected OpenFlow controllers), the severity of the vulnerability, and how the vulnerability may be mitigated (e.g. by disabling a feature). The treatment of the vulnerability could end here if the team determines that it is not a realistic vulnerability. Step 3a: Document ----------------- The security team develops a security advisory document. The security team may, at its discretion, include the reporter (via "CC") in developing the security advisory document, but in any case should accept feedback from the reporter before finalizing the document. When the document is final, the security team should obtain a CVE for the vulnerability from a CNA (https://cve.mitre.org/cve/cna.html). The document credits the reporter and describes the vulnerability, including all of the relevant information from the assessment in step 2. Suitable sections for the document include: :: * Title: The CVE identifier, a short description of the vulnerability. The title should mention Open vSwitch. In email, the title becomes the subject. Pre-release advisories are often passed around in encrypted email, which have plaintext subjects, so the title should not be too specific. * Description: A few paragraphs describing the general characteristics of the vulnerability, including the versions of Open vSwitch that are vulnerable, the kind of attack that exposes the vulnerability, and potential consequences of the attack. The description should re-state the CVE identifier, in case the subject is lost when an advisory is sent over email. * Mitigation: How an Open vSwitch administrator can minimize the potential for exploitation of the vulnerability, before applying a fix. If no mitigation is possible or recommended, explain why, to reduce the chance that at-risk users believe they are not at risk. * Fix: Describe how to fix the vulnerability, perhaps in terms of applying a source patch. The patch or patches themselves, if included in the email, should be at the very end of the advisory to reduce the risk that a reader would stop reading at this point. * Recommendation: A concise description of the security team's recommendation to users. * Acknowledgments: Thank the reporters. * Vulnerability Check: A step-by-step procedure by which a user can determine whether an installed copy of Open vSwitch is vulnerable. The procedure should clearly describe how to interpret the results, including expected results in vulnerable and not-vulnerable cases. Thus, procedures that produce clear and easily distinguished results are preferred. The procedure should assume as little understanding of Open vSwitch as possible, to make it more likely that a competent administrator who does not specialize in Open vSwitch can perform it successfully. The procedure should have minimal dependencies on tools that are not widely installed. Given a choice, the procedure should be one that takes at least some work to turn into a useful exploit. For example, a procedure based on "ovs-appctl" commands, which require local administrator access, is preferred to one that sends test packets to a machine, which only requires network connectivity. The section should say which operating systems it is designed for. If the procedure is likely to be specific to particular architectures (e.g. x86-64, i386), it should state on which ones it has been tested. This section should state the risks of the procedure. For example, if it can crash Open vSwitch or disrupt packet forwarding, say so. It is more useful to explain how to check an installed and running Open vSwitch than one built locally from source, but if it is easy to use the procedure from a sandbox environment, it can be helpful to explain how to do so. * Patch: If a patch or patches are available, and it is practical to include them in the email, put them at the end. Format them as described in :doc:`contributing/submitting-patches`, that is, as output by "git format-patch". The patch subjects should include the version for which they are suited, e.g. "[PATCH branch-2.3]" for a patch against Open vSwitch 2.3.x. If there are multiple patches for multiple versions of Open vSwitch, put them in separate sections with clear titles. Multiple patches for a single version of Open vSwitch, that must be stacked on top of each other to fix a single vulnerability, are undesirable because users are less likely to apply all of them correctly and in the correct order. Each patch should include a Vulnerability tag with the CVE identifier, a Reported-by tag or tags to credit the reporters, and a Signed-off-by tag to acknowledge the Developer's Certificate of Origin. It should also include other appropriate tags, such as Acked-by tags obtained during review. `CVE-2016-2074 `__ is an example advisory document. Step 3b: Fix ------------ Steps 3a and 3b may proceed in parallel. The security team develops and obtains (private) reviews for patches that fix the vulnerability. If necessary, the security team pulls in additional developers, who must agree to maintain confidentiality. Step 4: Embargoed Disclosure ---------------------------- The security advisory and patches are sent to downstream stakeholders, with an embargo date and time set from the time sent. Downstream stakeholders are expected not to deploy or disclose patches until the embargo is passed. A disclosure date is negotiated by the security team working with the bug submitter as well as vendors. However, the Open vSwitch security team holds the final say when setting a disclosure date. The timeframe for disclosure is from immediate (esp. if it's already publicly known) to a few weeks. As a basic default policy, we expect report date to disclosure date to be 10 to 15 business days. Operating system vendors are obvious downstream stakeholders. It may not be necessary to be too choosy about who to include: any major Open vSwitch user who is interested and can be considered trustworthy enough could be included. To become a downstream stakeholder, email the ovs-security mailing list. If the vulnerability is already public, skip this step. Step 5: Public Disclosure ------------------------- When the embargo expires, push the (reviewed) patches to appropriate branches, post the patches to the ovs-dev mailing list (noting that they have already been reviewed and applied), post the security advisory to appropriate mailing lists (ovs-announce, ovs-discuss), and post the security advisory on the Open vSwitch webpage. When the patch is applied to LTS (long-term support) branches, a new version should be released. The security advisory should be GPG-signed by a security team member with a key that is in a public web of trust. .. _contact: Contact ======= Report security vulnerabilities to the ovs-security mailing list: security@openvswitch.org Report problems with this document to the ovs-bugs mailing list: bugs@openvswitch.org ovs-2.9.0/Documentation/intro/000077500000000000000000000000001324262074100162765ustar00rootroot00000000000000ovs-2.9.0/Documentation/intro/index.rst000066400000000000000000000021141324262074100201350ustar00rootroot00000000000000.. Copyright (c) 2016, Stephen Finucane Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. Convention for heading levels in Open vSwitch documentation: ======= Heading 0 (reserved for the title in a document) ------- Heading 1 ~~~~~~~ Heading 2 +++++++ Heading 3 ''''''' Heading 4 Avoid deeper levels because they do not render well. =============== Getting Started =============== How to get started with Open vSwitch. .. toctree:: :maxdepth: 2 what-is-ovs why-ovs install/index ovs-2.9.0/Documentation/intro/install/000077500000000000000000000000001324262074100177445ustar00rootroot00000000000000ovs-2.9.0/Documentation/intro/install/bash-completion.rst000066400000000000000000000064771324262074100236000ustar00rootroot00000000000000.. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. Convention for heading levels in Open vSwitch documentation: ======= Heading 0 (reserved for the title in a document) ------- Heading 1 ~~~~~~~ Heading 2 +++++++ Heading 3 ''''''' Heading 4 Avoid deeper levels because they do not render well. ==================================== Bash command-line completion scripts ==================================== There are two completion scripts available: ``ovs-appctl-bashcomp.bash`` and ``ovs-vsctl-bashcomp.bash``. ovs-appctl-bashcomp ------------------- ``ovs-appctl-bashcomp.bash`` adds bash command-line completion support for ``ovs-appctl``, ``ovs-dpctl``, ``ovs-ofctl`` and ``ovsdb-tool`` commands. Features ~~~~~~~~ - Display available completion or complete on unfinished user input (long option, subcommand, and argument). - Subcommand hints - Convert between keywords like ``bridge``, ``port``, ``interface``, or ``dp`` and the available record in ovsdb. Limitations ~~~~~~~~~~~ - Only supports a small set of important keywords (``dp``, ``datapath``, ``bridge``, ``switch``, ``port``, ``interface``, ``iface``). - Does not support parsing of nested options. For example:: $ ovsdb-tool create [db [schema]] - Does not support expansion on repeated argument. For example:: $ ovs-dpctl show [dp...]). - Only supports matching on long options, and only in the format ``--option [arg]``. Do not use ``--option=[arg]``. ovs-vsctl-bashcomp ------------------- ``ovs-vsctl-bashcomp.bash`` adds Bash command-line completion support for ``ovs-vsctl`` command. Features ~~~~~~~~ - Display available completion and complete on user input for global/local options, command, and argument. - Query database and expand keywords like ``table``, ``record``, ``column``, or ``key``, to available completions. - Deal with argument relations like 'one and more', 'zero or one'. - Complete multiple ``ovs-vsctl`` commands cascaded via ``--``. Limitations ~~~~~~~~~~~ Completion of very long ``ovs-vsctl`` commands can take up to several seconds. Usage ----- The bashcomp scripts should be placed at ``/etc/bash_completion.d/`` to be available for all bash sessions. Running ``make install`` will place the scripts to ``$(sysconfdir)/bash_completion.d/``, thus, the user should specify ``--sysconfdir=/etc`` at configuration. If OVS is installed from packages, the scripts will automatically be placed inside ``/etc/bash_completion.d/``. If you just want to run the scripts in one bash, you can remove them from ``/etc/bash_completion.d/`` and run the scripts via ``. ovs-appctl-bashcomp.bash`` or ``. ovs-vsctl-bashcomp.bash``. Tests ----- Unit tests are added in ``tests/completion.at`` and integrated into autotest framework. To run the tests, just run ``make check``. ovs-2.9.0/Documentation/intro/install/debian.rst000066400000000000000000000111701324262074100217200ustar00rootroot00000000000000.. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. Convention for heading levels in Open vSwitch documentation: ======= Heading 0 (reserved for the title in a document) ------- Heading 1 ~~~~~~~ Heading 2 +++++++ Heading 3 ''''''' Heading 4 Avoid deeper levels because they do not render well. ================================= Debian Packaging for Open vSwitch ================================= This document describes how to build Debian packages for Open vSwitch. To install Open vSwitch on Debian without building Debian packages, refer to :doc:`general` instead. .. note:: These instructions should also work on Ubuntu and other Debian derivative distributions. Before You Begin ---------------- Before you begin, consider whether you really need to build packages yourself. Debian "wheezy" and "sid", as well as recent versions of Ubuntu, contain pre-built Debian packages for Open vSwitch. It is easier to install these than to build your own. To use packages from your distribution, skip ahead to "Installing .deb Packages", below. Building Open vSwitch Debian packages ------------------------------------- You may build from an Open vSwitch distribution tarball or from an Open vSwitch Git tree with these instructions. You do not need to be the superuser to build the Debian packages. 1. Install the "build-essential" and "fakeroot" packages. For example:: $ apt-get install build-essential fakeroot 2. Obtain and unpack an Open vSwitch source distribution and ``cd`` into its top level directory. 3. Install the build dependencies listed under "Build-Depends:" near the top of ``debian/control``. You can install these any way you like, e.g. with ``apt-get install``. Check your work by running ``dpkg-checkbuilddeps`` in the top level of your ovs directory. If you've installed all the dependencies properly, ``dpkg-checkbuilddeps`` will exit without printing anything. If you forgot to install some dependencies, it will tell you which ones. 4. Build the package:: $ fakeroot debian/rules binary This will do a serial build that runs the unit tests. This will take approximately 8 to 10 minutes. If you prefer, you can run a faster parallel build:: $ DEB_BUILD_OPTIONS='parallel=8' fakeroot debian/rules binary If you are in a big hurry, you can even skip the unit tests:: $ DEB_BUILD_OPTIONS='parallel=8 nocheck' fakeroot debian/rules binary .. note:: There are a few pitfalls in the Debian packaging building system so that, occasionally, you may find that in a tree that you have using for a while, the build command above exits immediately without actually building anything. To fix the problem, run:: $ fakeroot debian/rules clean or start over from a fresh copy of the source tree. 5. The generated .deb files will be in the parent directory of the Open vSwitch source distribution. Installing .deb Packages ------------------------ These instructions apply to installing from Debian packages that you built yourself, as described in the previous section. In this case, use a command such as ``dpkg -i`` to install the .deb files that you build. You will have to manually install any missing dependencies. You can also use these instruction to install from packages provided by Debian or a Debian derivative distribution such as Ubuntu. In this case, use a program such as ``apt-get`` or ``aptitude`` to download and install the provided packages. These programs will also automatically download and install any missing dependencies. .. important:: You must be superuser to install Debian packages. 1. Start by installing an Open vSwitch kernel module. See ``debian/openvswitch-switch.README.Debian`` for the available options. 2. Install the ``openvswitch-switch`` and ``openvswitch-common`` packages. These packages include the core userspace components of the switch. Open vSwitch ``.deb`` packages not mentioned above are rarely useful. Refer to their individual package descriptions to find out whether any of them are useful to you. Reporting Bugs -------------- Report problems to bugs@openvswitch.org. ovs-2.9.0/Documentation/intro/install/distributions.rst000066400000000000000000000046131324262074100234040ustar00rootroot00000000000000.. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. Convention for heading levels in Open vSwitch documentation: ======= Heading 0 (reserved for the title in a document) ------- Heading 1 ~~~~~~~ Heading 2 +++++++ Heading 3 ''''''' Heading 4 Avoid deeper levels because they do not render well. ==================================== Distributions packaging Open vSwitch ==================================== This document lists various popular distributions packaging Open vSwitch. Open vSwitch is packaged by various distributions for multiple platforms and architectures. .. note:: The packaged version available with distributions may not be latest Open vSwitch release. Debian ------- You can use ``apt-get`` or ``aptitude`` to install the .deb packages and must be superuser. 1. Debian has ``openvswitch-switch`` and ``openvswitch-common`` .deb packages that includes the core userspace components of the switch. 2. For kernel datapath, ``openvswitch-datapath-dkms`` can be installed to automatically build and install Open vSwitch kernel module for your running kernel. 3. For DPDK datapath, Open vSwitch with DPDK support is bundled in the package ``openvswitch-switch-dpdk``. Fedora ------ Fedora provides ``openvswitch``, ``openvswitch-devel``, ``openvswitch-test`` and ``openvswitch-debuginfo`` rpm packages. You can install ``openvswitch`` package in minimum installation. Use ``yum`` or ``dnf`` to install the rpm packages and must be superuser. Red Hat ------- RHEL distributes ``openvswitch`` rpm package that supports kernel datapath. DPDK accelerated Open vSwitch can be installed using ``openvswitch-dpdk`` package. OpenSuSE -------- OpenSUSE provides ``openvswitch``, ``openvswitch-switch`` rpm packages. Also ``openvswitch-dpdk`` and ``openvswitch-dpdk-switch`` can be installed for Open vSwitch using DPDK accelerated datapath. ovs-2.9.0/Documentation/intro/install/documentation.rst000066400000000000000000000057501324262074100233560ustar00rootroot00000000000000.. Copyright (c) 2016 Stephen Finucane Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. Convention for heading levels in Open vSwitch documentation: ======= Heading 0 (reserved for the title in a document) ------- Heading 1 ~~~~~~~ Heading 2 +++++++ Heading 3 ''''''' Heading 4 Avoid deeper levels because they do not render well. ========================== Open vSwitch Documentation ========================== This document describes how to build the OVS documentation for use offline. A continuously updated, online version can be found at `docs.openvswitch.org `__. .. note:: These instructions provide information on building the documentation locally. For information on writing documentation, refer to :doc:`/internals/contributing/documentation-style` Build Requirements ------------------ As described in the :doc:`/internals/contributing/documentation-style`, the Open vSwitch documentation is written in reStructuredText and built with Sphinx. A detailed guide on installing Sphinx in many environments is available on the `Sphinx website`__ but, for most Linux distributions, you can install with your package manager. For example, on Debian/Ubuntu run:: $ sudo apt-get install python-sphinx Similarly, on RHEL/Fedora run:: $ sudo dnf install python-sphinx A ``requirements.txt`` is also provided in the ``/Documentation``, should you wish to install using ``pip``:: $ virtualenv .venv $ source .venv/bin/activate $ pip install -r Documentation/requirements.txt __ http://www.sphinx-doc.org/install.html Configuring ----------- It's unlikely that you'll need to customize any aspect of the configuration. However, the ``Documentation/conf.py`` is the go-to place for all configuration. This file is well documented and further information is available on the `Sphinx website`__. Building -------- Once Sphinx installed, the documentation can be built using the provided Makefile targets:: $ make docs-check .. important:: The ``docs-check`` target will fail if there are any syntax errors. However, it won't catch more succint issues such as style or grammar issues. As a result, you should always inspect changes visually to ensure the result is as intended. Once built, documentation is available in the ``/Documentation/_build`` folder. Open the root ``index.html`` to browse the documentation. __ http://www.sphinx-doc.org/config.html ovs-2.9.0/Documentation/intro/install/dpdk.rst000066400000000000000000000612771324262074100214350ustar00rootroot00000000000000.. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. Convention for heading levels in Open vSwitch documentation: ======= Heading 0 (reserved for the title in a document) ------- Heading 1 ~~~~~~~ Heading 2 +++++++ Heading 3 ''''''' Heading 4 Avoid deeper levels because they do not render well. ====================== Open vSwitch with DPDK ====================== This document describes how to build and install Open vSwitch using a DPDK datapath. Open vSwitch can use the DPDK library to operate entirely in userspace. .. seealso:: The :doc:`releases FAQ ` lists support for the required versions of DPDK for each version of Open vSwitch. Build requirements ------------------ In addition to the requirements described in :doc:`general`, building Open vSwitch with DPDK will require the following: - DPDK 17.11 - A `DPDK supported NIC`_ Only required when physical ports are in use - A suitable kernel On Linux Distros running kernel version >= 3.0, only `IOMMU` needs to enabled via the grub cmdline, assuming you are using **VFIO**. For older kernels, ensure the kernel is built with ``UIO``, ``HUGETLBFS``, ``PROC_PAGE_MONITOR``, ``HPET``, ``HPET_MMAP`` support. If these are not present, it will be necessary to upgrade your kernel or build a custom kernel with these flags enabled. Detailed system requirements can be found at `DPDK requirements`_. .. _DPDK supported NIC: http://dpdk.org/doc/nics .. _DPDK requirements: http://dpdk.org/doc/guides/linux_gsg/sys_reqs.html Installing ---------- Install DPDK ~~~~~~~~~~~~ #. Download the `DPDK sources`_, extract the file and set ``DPDK_DIR``:: $ cd /usr/src/ $ wget http://fast.dpdk.org/rel/dpdk-17.11.tar.xz $ tar xf dpdk-17.11.tar.xz $ export DPDK_DIR=/usr/src/dpdk-17.11 $ cd $DPDK_DIR #. (Optional) Configure DPDK as a shared library DPDK can be built as either a static library or a shared library. By default, it is configured for the former. If you wish to use the latter, set ``CONFIG_RTE_BUILD_SHARED_LIB=y`` in ``$DPDK_DIR/config/common_base``. .. note:: Minor performance loss is expected when using OVS with a shared DPDK library compared to a static DPDK library. #. Configure and install DPDK Build and install the DPDK library:: $ export DPDK_TARGET=x86_64-native-linuxapp-gcc $ export DPDK_BUILD=$DPDK_DIR/$DPDK_TARGET $ make install T=$DPDK_TARGET DESTDIR=install #. (Optional) Export the DPDK shared library location If DPDK was built as a shared library, export the path to this library for use when building OVS:: $ export LD_LIBRARY_PATH=$DPDK_DIR/x86_64-native-linuxapp-gcc/lib .. _DPDK sources: http://dpdk.org/rel Install OVS ~~~~~~~~~~~ OVS can be installed using different methods. For OVS to use DPDK datapath, it has to be configured with DPDK support (``--with-dpdk``). .. note:: This section focuses on generic recipe that suits most cases. For distribution specific instructions, refer to one of the more relevant guides. .. _OVS sources: http://openvswitch.org/releases/ #. Ensure the standard OVS requirements, described in :ref:`general-build-reqs`, are installed #. Bootstrap, if required, as described in :ref:`general-bootstrapping` #. Configure the package using the ``--with-dpdk`` flag:: $ ./configure --with-dpdk=$DPDK_BUILD where ``DPDK_BUILD`` is the path to the built DPDK library. This can be skipped if DPDK library is installed in its default location. If no path is provided to ``--with-dpdk``, but a pkg-config configuration for libdpdk is available the include paths will be generated via an equivalent ``pkg-config --cflags libdpdk``. .. note:: While ``--with-dpdk`` is required, you can pass any other configuration option described in :ref:`general-configuring`. #. Build and install OVS, as described in :ref:`general-building` Additional information can be found in :doc:`general`. .. note:: If you are running using the Fedora or Red Hat package, the Open vSwitch daemon will run as a non-root user. This implies that you must have a working IOMMU. Visit the `RHEL README`__ for additional information. __ https://github.com/openvswitch/ovs/blob/master/rhel/README.RHEL.rst Setup ----- Setup Hugepages ~~~~~~~~~~~~~~~ Allocate a number of 2M Huge pages: - For persistent allocation of huge pages, write to hugepages.conf file in `/etc/sysctl.d`:: $ echo 'vm.nr_hugepages=2048' > /etc/sysctl.d/hugepages.conf - For run-time allocation of huge pages, use the ``sysctl`` utility:: $ sysctl -w vm.nr_hugepages=N # where N = No. of 2M huge pages To verify hugepage configuration:: $ grep HugePages_ /proc/meminfo Mount the hugepages, if not already mounted by default:: $ mount -t hugetlbfs none /dev/hugepages`` .. _dpdk-vfio: Setup DPDK devices using VFIO ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ VFIO is prefered to the UIO driver when using recent versions of DPDK. VFIO support required support from both the kernel and BIOS. For the former, kernel version > 3.6 must be used. For the latter, you must enable VT-d in the BIOS and ensure this is configured via grub. To ensure VT-d is enabled via the BIOS, run:: $ dmesg | grep -e DMAR -e IOMMU If VT-d is not enabled in the BIOS, enable it now. To ensure VT-d is enabled in the kernel, run:: $ cat /proc/cmdline | grep iommu=pt $ cat /proc/cmdline | grep intel_iommu=on If VT-d is not enabled in the kernel, enable it now. Once VT-d is correctly configured, load the required modules and bind the NIC to the VFIO driver:: $ modprobe vfio-pci $ /usr/bin/chmod a+x /dev/vfio $ /usr/bin/chmod 0666 /dev/vfio/* $ $DPDK_DIR/usertools/dpdk-devbind.py --bind=vfio-pci eth1 $ $DPDK_DIR/usertools/dpdk-devbind.py --status Setup OVS ~~~~~~~~~ Open vSwitch should be started as described in :doc:`general` with the exception of ovs-vswitchd, which requires some special configuration to enable DPDK functionality. DPDK configuration arguments can be passed to ovs-vswitchd via the ``other_config`` column of the ``Open_vSwitch`` table. At a minimum, the ``dpdk-init`` option must be set to ``true``. For example:: $ export PATH=$PATH:/usr/local/share/openvswitch/scripts $ export DB_SOCK=/usr/local/var/run/openvswitch/db.sock $ ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-init=true $ ovs-ctl --no-ovsdb-server --db-sock="$DB_SOCK" start There are many other configuration options, the most important of which are listed below. Defaults will be provided for all values not explicitly set. ``dpdk-init`` Specifies whether OVS should initialize and support DPDK ports. This is a boolean, and defaults to false. ``dpdk-lcore-mask`` Specifies the CPU cores on which dpdk lcore threads should be spawned and expects hex string (eg '0x123'). ``dpdk-socket-mem`` Comma separated list of memory to pre-allocate from hugepages on specific sockets. ``dpdk-hugepage-dir`` Directory where hugetlbfs is mounted ``vhost-sock-dir`` Option to set the path to the vhost-user unix socket files. If allocating more than one GB hugepage, you can configure the amount of memory used from any given NUMA nodes. For example, to use 1GB from NUMA node 0 and 0GB for all other NUMA nodes, run:: $ ovs-vsctl --no-wait set Open_vSwitch . \ other_config:dpdk-socket-mem="1024,0" or:: $ ovs-vsctl --no-wait set Open_vSwitch . \ other_config:dpdk-socket-mem="1024" .. note:: Changing any of these options requires restarting the ovs-vswitchd application See the section ``Performance Tuning`` for important DPDK customizations. Validating ---------- At this point you can use ovs-vsctl to set up bridges and other Open vSwitch features. Seeing as we've configured the DPDK datapath, we will use DPDK-type ports. For example, to create a userspace bridge named ``br0`` and add two ``dpdk`` ports to it, run:: $ ovs-vsctl add-br br0 -- set bridge br0 datapath_type=netdev $ ovs-vsctl add-port br0 myportnameone -- set Interface myportnameone \ type=dpdk options:dpdk-devargs=0000:06:00.0 $ ovs-vsctl add-port br0 myportnametwo -- set Interface myportnametwo \ type=dpdk options:dpdk-devargs=0000:06:00.1 DPDK devices will not be available for use until a valid dpdk-devargs is specified. Refer to ovs-vsctl(8) and :doc:`/howto/dpdk` for more details. Performance Tuning ------------------ To achieve optimal OVS performance, the system can be configured and that includes BIOS tweaks, Grub cmdline additions, better understanding of NUMA nodes and apt selection of PCIe slots for NIC placement. .. note:: This section is optional. Once installed as described above, OVS with DPDK will work out of the box. Recommended BIOS Settings ~~~~~~~~~~~~~~~~~~~~~~~~~ .. list-table:: Recommended BIOS Settings :header-rows: 1 * - Setting - Value * - C3 Power State - Disabled * - C6 Power State - Disabled * - MLC Streamer - Enabled * - MLC Spacial Prefetcher - Enabled * - DCU Data Prefetcher - Enabled * - DCA - Enabled * - CPU Power and Performance - Performance * - Memeory RAS and Performance Config -> NUMA optimized - Enabled PCIe Slot Selection ~~~~~~~~~~~~~~~~~~~ The fastpath performance can be affected by factors related to the placement of the NIC, such as channel speeds between PCIe slot and CPU or the proximity of PCIe slot to the CPU cores running the DPDK application. Listed below are the steps to identify right PCIe slot. #. Retrieve host details using ``dmidecode``. For example:: $ dmidecode -t baseboard | grep "Product Name" #. Download the technical specification for product listed, e.g: S2600WT2 #. Check the Product Architecture Overview on the Riser slot placement, CPU sharing info and also PCIe channel speeds For example: On S2600WT, CPU1 and CPU2 share Riser Slot 1 with Channel speed between CPU1 and Riser Slot1 at 32GB/s, CPU2 and Riser Slot1 at 16GB/s. Running DPDK app on CPU1 cores and NIC inserted in to Riser card Slots will optimize OVS performance in this case. #. Check the Riser Card #1 - Root Port mapping information, on the available slots and individual bus speeds. In S2600WT slot 1, slot 2 has high bus speeds and are potential slots for NIC placement. Advanced Hugepage Setup ~~~~~~~~~~~~~~~~~~~~~~~ Allocate and mount 1 GB hugepages. - For persistent allocation of huge pages, add the following options to the kernel bootline:: default_hugepagesz=1GB hugepagesz=1G hugepages=N For platforms supporting multiple huge page sizes, add multiple options:: default_hugepagesz= hugepagesz= hugepages=N where: ``N`` number of huge pages requested ``size`` huge page size with an optional suffix ``[kKmMgG]`` - For run-time allocation of huge pages:: $ echo N > /sys/devices/system/node/nodeX/hugepages/hugepages-1048576kB/nr_hugepages where: ``N`` number of huge pages requested ``X`` NUMA Node .. note:: For run-time allocation of 1G huge pages, Contiguous Memory Allocator (``CONFIG_CMA``) has to be supported by kernel, check your Linux distro. Now mount the huge pages, if not already done so:: $ mount -t hugetlbfs -o pagesize=1G none /dev/hugepages Isolate Cores ~~~~~~~~~~~~~ The ``isolcpus`` option can be used to isolate cores from the Linux scheduler. The isolated cores can then be used to dedicatedly run HPC applications or threads. This helps in better application performance due to zero context switching and minimal cache thrashing. To run platform logic on core 0 and isolate cores between 1 and 19 from scheduler, add ``isolcpus=1-19`` to GRUB cmdline. .. note:: It has been verified that core isolation has minimal advantage due to mature Linux scheduler in some circumstances. Compiler Optimizations ~~~~~~~~~~~~~~~~~~~~~~ The default compiler optimization level is ``-O2``. Changing this to more aggressive compiler optimization such as ``-O3 -march=native`` with gcc (verified on 5.3.1) can produce performance gains though not siginificant. ``-march=native`` will produce optimized code on local machine and should be used when software compilation is done on Testbed. Multiple Poll-Mode Driver Threads ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ With pmd multi-threading support, OVS creates one pmd thread for each NUMA node by default, if there is at least one DPDK interface from that NUMA node added to OVS. However, in cases where there are multiple ports/rxq's producing traffic, performance can be improved by creating multiple pmd threads running on separate cores. These pmd threads can share the workload by each being responsible for different ports/rxq's. Assignment of ports/rxq's to pmd threads is done automatically. A set bit in the mask means a pmd thread is created and pinned to the corresponding CPU core. For example, to run pmd threads on core 1 and 2:: $ ovs-vsctl set Open_vSwitch . other_config:pmd-cpu-mask=0x6 When using dpdk and dpdkvhostuser ports in a bi-directional VM loopback as shown below, spreading the workload over 2 or 4 pmd threads shows significant improvements as there will be more total CPU occupancy available:: NIC port0 <-> OVS <-> VM <-> OVS <-> NIC port 1 Refer to ovs-vswitchd.conf.db(5) for additional information on configuration options. Affinity ~~~~~~~~ For superior performance, DPDK pmd threads and Qemu vCPU threads needs to be affinitized accordingly. - PMD thread Affinity A poll mode driver (pmd) thread handles the I/O of all DPDK interfaces assigned to it. A pmd thread shall poll the ports for incoming packets, switch the packets and send to tx port. A pmd thread is CPU bound, and needs to be affinitized to isolated cores for optimum performance. Even though a PMD thread may exist, the thread only starts consuming CPU cycles if there is at least one receive queue assigned to the pmd. .. note:: On NUMA systems, PCI devices are also local to a NUMA node. Unbound rx queues for a PCI device will be assigned to a pmd on it's local NUMA node if a non-isolated PMD exists on that NUMA node. If not, the queue will be assigned to a non-isolated pmd on a remote NUMA node. This will result in reduced maximum throughput on that device and possibly on other devices assigned to that pmd thread. If such a queue assignment is made a warning message will be logged: "There's no available (non-isolated) pmd thread on numa node N. Queue Q on port P will be assigned to the pmd on core C (numa node N'). Expect reduced performance." Binding PMD threads to cores is described in the above section ``Multiple Poll-Mode Driver Threads``. - QEMU vCPU thread Affinity A VM performing simple packet forwarding or running complex packet pipelines has to ensure that the vCPU threads performing the work has as much CPU occupancy as possible. For example, on a multicore VM, multiple QEMU vCPU threads shall be spawned. When the DPDK ``testpmd`` application that does packet forwarding is invoked, the ``taskset`` command should be used to affinitize the vCPU threads to the dedicated isolated cores on the host system. Enable HyperThreading ~~~~~~~~~~~~~~~~~~~~~ With HyperThreading, or SMT, enabled, a physical core appears as two logical cores. SMT can be utilized to spawn worker threads on logical cores of the same physical core there by saving additional cores. With DPDK, when pinning pmd threads to logical cores, care must be taken to set the correct bits of the ``pmd-cpu-mask`` to ensure that the pmd threads are pinned to SMT siblings. Take a sample system configuration, with 2 sockets, 2 * 10 core processors, HT enabled. This gives us a total of 40 logical cores. To identify the physical core shared by two logical cores, run:: $ cat /sys/devices/system/cpu/cpuN/topology/thread_siblings_list where ``N`` is the logical core number. In this example, it would show that cores ``1`` and ``21`` share the same physical core. Logical cores can be specified in pmd-cpu-masks similarly to physical cores, as described in ``Multiple Poll-Mode Driver Threads``. NUMA/Cluster-on-Die ~~~~~~~~~~~~~~~~~~~ Ideally inter-NUMA datapaths should be avoided where possible as packets will go across QPI and there may be a slight performance penalty when compared with intra NUMA datapaths. On Intel Xeon Processor E5 v3, Cluster On Die is introduced on models that have 10 cores or more. This makes it possible to logically split a socket into two NUMA regions and again it is preferred where possible to keep critical datapaths within the one cluster. It is good practice to ensure that threads that are in the datapath are pinned to cores in the same NUMA area. e.g. pmd threads and QEMU vCPUs responsible for forwarding. If DPDK is built with ``CONFIG_RTE_LIBRTE_VHOST_NUMA=y``, vHost User ports automatically detect the NUMA socket of the QEMU vCPUs and will be serviced by a PMD from the same node provided a core on this node is enabled in the ``pmd-cpu-mask``. ``libnuma`` packages are required for this feature. Binding PMD threads is described in the above section ``Multiple Poll-Mode Driver Threads``. DPDK Physical Port Rx Queues ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ :: $ ovs-vsctl set Interface options:n_rxq= The above command sets the number of rx queues for DPDK physical interface. The rx queues are assigned to pmd threads on the same NUMA node in a round-robin fashion. .. _dpdk-queues-sizes: DPDK Physical Port Queue Sizes ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ :: $ ovs-vsctl set Interface dpdk0 options:n_rxq_desc= $ ovs-vsctl set Interface dpdk0 options:n_txq_desc= The above command sets the number of rx/tx descriptors that the NIC associated with dpdk0 will be initialised with. Different ``n_rxq_desc`` and ``n_txq_desc`` configurations yield different benefits in terms of throughput and latency for different scenarios. Generally, smaller queue sizes can have a positive impact for latency at the expense of throughput. The opposite is often true for larger queue sizes. Note: increasing the number of rx descriptors eg. to 4096 may have a negative impact on performance due to the fact that non-vectorised DPDK rx functions may be used. This is dependent on the driver in use, but is true for the commonly used i40e and ixgbe DPDK drivers. Exact Match Cache ~~~~~~~~~~~~~~~~~ Each pmd thread contains one Exact Match Cache (EMC). After initial flow setup in the datapath, the EMC contains a single table and provides the lowest level (fastest) switching for DPDK ports. If there is a miss in the EMC then the next level where switching will occur is the datapath classifier. Missing in the EMC and looking up in the datapath classifier incurs a significant performance penalty. If lookup misses occur in the EMC because it is too small to handle the number of flows, its size can be increased. The EMC size can be modified by editing the define ``EM_FLOW_HASH_SHIFT`` in ``lib/dpif-netdev.c``. As mentioned above, an EMC is per pmd thread. An alternative way of increasing the aggregate amount of possible flow entries in EMC and avoiding datapath classifier lookups is to have multiple pmd threads running. Rx Mergeable Buffers ~~~~~~~~~~~~~~~~~~~~ Rx mergeable buffers is a virtio feature that allows chaining of multiple virtio descriptors to handle large packet sizes. Large packets are handled by reserving and chaining multiple free descriptors together. Mergeable buffer support is negotiated between the virtio driver and virtio device and is supported by the DPDK vhost library. This behavior is supported and enabled by default, however in the case where the user knows that rx mergeable buffers are not needed i.e. jumbo frames are not needed, it can be forced off by adding ``mrg_rxbuf=off`` to the QEMU command line options. By not reserving multiple chains of descriptors it will make more individual virtio descriptors available for rx to the guest using dpdkvhost ports and this can improve performance. Output Packet Batching ~~~~~~~~~~~~~~~~~~~~~~ To make advantage of batched transmit functions, OVS collects packets in intermediate queues before sending when processing a batch of received packets. Even if packets are matched by different flows, OVS uses a single send operation for all packets destined to the same output port. Furthermore, OVS is able to buffer packets in these intermediate queues for a configurable amount of time to reduce the frequency of send bursts at medium load levels when the packet receive rate is high, but the receive batch size still very small. This is particularly beneficial for packets transmitted to VMs using an interrupt-driven virtio driver, where the interrupt overhead is significant for the OVS PMD, the host operating system and the guest driver. The ``tx-flush-interval`` parameter can be used to specify the time in microseconds OVS should wait between two send bursts to a given port (default is ``0``). When the intermediate queue fills up before that time is over, the buffered packet batch is sent immediately:: $ ovs-vsctl set Open_vSwitch . other_config:tx-flush-interval=50 This parameter influences both throughput and latency, depending on the traffic load on the port. In general lower values decrease latency while higher values may be useful to achieve higher throughput. Low traffic (``packet rate < 1 / tx-flush-interval``) should not experience any significant latency or throughput increase as packets are forwarded immediately. At intermediate load levels (``1 / tx-flush-interval < packet rate < 32 / tx-flush-interval``) traffic should experience an average latency increase of up to ``1 / 2 * tx-flush-interval`` and a possible throughput improvement. Very high traffic (``packet rate >> 32 / tx-flush-interval``) should experience the average latency increase equal to ``32 / (2 * packet rate)``. Most send batches in this case will contain the maximum number of packets (``32``). A ``tx-burst-interval`` value of ``50`` microseconds has shown to provide a good performance increase in a ``PHY-VM-PHY`` scenario on ``x86`` system for interrupt-driven guests while keeping the latency increase at a reasonable level: https://mail.openvswitch.org/pipermail/ovs-dev/2017-December/341628.html .. note:: Throughput impact of this option significantly depends on the scenario and the traffic patterns. For example: ``tx-burst-interval`` value of ``50`` microseconds shows performance degradation in ``PHY-VM-PHY`` with bonded PHY scenario while testing with ``256 - 1024`` packet flows: https://mail.openvswitch.org/pipermail/ovs-dev/2017-December/341700.html The average number of packets per output batch can be checked in PMD stats:: $ ovs-appctl dpif-netdev/pmd-stats-show Limitations ------------ - Currently DPDK ports does not use HW offload functionality. - Network Interface Firmware requirements: Each release of DPDK is validated against a specific firmware version for a supported Network Interface. New firmware versions introduce bug fixes, performance improvements and new functionality that DPDK leverages. The validated firmware versions are available as part of the release notes for DPDK. It is recommended that users update Network Interface firmware to match what has been validated for the DPDK release. The latest list of validated firmware versions can be found in the `DPDK release notes`_. .. _DPDK release notes: http://dpdk.org/doc/guides/rel_notes/release_17_11.html - Upper bound MTU: DPDK device drivers differ in how the L2 frame for a given MTU value is calculated e.g. i40e driver includes 2 x vlan headers in MTU overhead, em driver includes 1 x vlan header, ixgbe driver does not include a vlan header in overhead. Currently it is not possible for OVS DPDK to know what upper bound MTU value is supported for a given device. As such OVS DPDK must provision for the case where the L2 frame for a given MTU includes 2 x vlan headers. This reduces the upper bound MTU value for devices that do not include vlan headers in their L2 frames by 8 bytes e.g. ixgbe devices upper bound MTU is reduced from 9710 to 9702. This work around is temporary and is expected to be removed once a method is provided by DPDK to query the upper bound MTU value for a given device. Reporting Bugs -------------- Report problems to bugs@openvswitch.org. ovs-2.9.0/Documentation/intro/install/fedora.rst000066400000000000000000000116711324262074100217440ustar00rootroot00000000000000.. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. Convention for heading levels in Open vSwitch documentation: ======= Heading 0 (reserved for the title in a document) ------- Heading 1 ~~~~~~~ Heading 2 +++++++ Heading 3 ''''''' Heading 4 Avoid deeper levels because they do not render well. =========================================== Fedora, RHEL 7.x Packaging for Open vSwitch =========================================== This document provides instructions for building and installing Open vSwitch RPM packages on a Fedora Linux host. Instructions for the installation of Open vSwitch on a Fedora Linux host without using RPM packages can be found in the :doc:`general`. These instructions have been tested with Fedora 23, and are also applicable for RHEL 7.x and its derivatives, including CentOS 7.x and Scientific Linux 7.x. Build Requirements ------------------ You will need to install all required packages to build the RPMs. Newer distributions use ``dnf`` but if it's not available, then use ``yum`` instructions. The command below will install RPM tools and generic build dependencies. And (optionally) include these packages: libcap-ng libcap-ng-devel dpdk-devel. DNF: :: $ dnf install @'Development Tools' rpm-build dnf-plugins-core YUM: :: $ yum install @'Development Tools' rpm-build yum-utils Then it is necessary to install Open vSwitch specific build dependencies. The dependencies are listed in the SPEC file, but first it is necessary to replace the VERSION tag to be a valid SPEC. The command below will create a temporary SPEC file:: $ sed -e 's/@VERSION@/0.0.1/' rhel/openvswitch-fedora.spec.in \ > /tmp/ovs.spec And to install specific dependencies, use the corresponding tool below. For some of the dependencies on RHEL you may need to add two additional repositories to help yum-builddep, e.g.:: $ subscription-manager repos --enable=rhel-7-server-extras-rpms $ subscription-manager repos --enable=rhel-7-server-optional-rpms DNF:: $ dnf builddep /tmp/ovs.spec YUM:: $ yum-builddep /tmp/ovs.spec Once that is completed, remove the file ``/tmp/ovs.spec``. Bootstraping ------------ Refer to :ref:`general-bootstrapping`. Configuring ----------- Refer to :ref:`general-configuring`. Building -------- User Space RPMs ~~~~~~~~~~~~~~~ To build Open vSwitch user-space RPMs, execute the following from the directory in which `./configure` was executed: :: $ make rpm-fedora This will create the RPMs `openvswitch`, `python-openvswitch`, `openvswitch-test`, `openvswitch-devel`, `openvswitch-ovn-common`, `openvswitch-ovn-central`, `openvswitch-ovn-host`, `openvswitch-ovn-vtep`, `openvswitch-ovn-docker`, and `openvswitch-debuginfo`. To enable DPDK support in the openvswitch package, the ``--with dpdk`` option can be added: :: $ make rpm-fedora RPMBUILD_OPT="--with dpdk --without check" You can also have the above commands automatically run the Open vSwitch unit tests. This can take several minutes. :: $ make rpm-fedora RPMBUILD_OPT="--with check" Kernel OVS Tree Datapath RPM ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ To build the Open vSwitch kernel module for the currently running kernel version, run: :: $ make rpm-fedora-kmod To build the Open vSwitch kernel module for another kernel version, the desired kernel version can be specified via the `kversion` macro. For example: :: $ make rpm-fedora-kmod \ RPMBUILD_OPT='-D "kversion 4.3.4-300.fc23.x86_64"' Installing ---------- RPM packages can be installed by using the command ``rpm -i``. Package installation requires superuser privileges. The `openvswitch-kmod` RPM should be installed first if the Linux OVS tree datapath module is to be used. The `openvswitch-kmod` RPM should not be installed if only the in-tree Linux datapath or user-space datapath is needed. Refer to the :doc:`/faq/index` for more information about the various Open vSwitch datapath options. In most cases only the `openvswitch` RPM will need to be installed. The `python-openvswitch`, `openvswitch-test`, `openvswitch-devel`, and `openvswitch-debuginfo` RPMs are optional unless required for a specific purpose. The `openvswitch-ovn-*` packages are only needed when using OVN. Refer to the `RHEL README`__ for additional usage and configuration information. __ https://github.com/openvswitch/ovs/blob/master/rhel/README.RHEL.rst Reporting Bugs -------------- Report problems to bugs@openvswitch.org. ovs-2.9.0/Documentation/intro/install/general.rst000066400000000000000000000545471324262074100221320ustar00rootroot00000000000000.. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. Convention for heading levels in Open vSwitch documentation: ======= Heading 0 (reserved for the title in a document) ------- Heading 1 ~~~~~~~ Heading 2 +++++++ Heading 3 ''''''' Heading 4 Avoid deeper levels because they do not render well. ========================================= Open vSwitch on Linux, FreeBSD and NetBSD ========================================= This document describes how to build and install Open vSwitch on a generic Linux, FreeBSD, or NetBSD host. For specifics around installation on a specific platform, refer to one of the other installation guides listed in :doc:`index`. Obtaining Open vSwitch Sources ------------------------------ The canonical location for Open vSwitch source code is its Git repository, which you can clone into a directory named "ovs" with:: $ git clone https://github.com/openvswitch/ovs.git Cloning the repository leaves the "master" branch initially checked out. This is the right branch for general development. If, on the other hand, if you want to build a particular released version, you can check it out by running a command such as the following from the "ovs" directory:: $ git checkout v2.7.0 The repository also has a branch for each release series. For example, to obtain the latest fixes in the Open vSwitch 2.7.x release series, which might include bug fixes that have not yet been in any released version, you can check it out from the "ovs" directory with:: $ git checkout origin/branch-2.7 If you do not want to use Git, you can also obtain tarballs for Open vSwitch release versions via http://openvswitch.org/download/, or download a ZIP file for any snapshot from the web interface at https://github.com/openvswitch/ovs. .. _general-build-reqs: Build Requirements ------------------ To compile the userspace programs in the Open vSwitch distribution, you will need the following software: - GNU make - A C compiler, such as: - GCC 4.6 or later. - Clang 3.4 or later. - MSVC 2013. Refer to :doc:`windows` for additional Windows build instructions. While OVS may be compatible with other compilers, optimal support for atomic operations may be missing, making OVS very slow (see ``lib/ovs-atomic.h``). - libssl, from OpenSSL, is optional but recommended if you plan to connect the Open vSwitch to an OpenFlow controller. libssl is required to establish confidentiality and authenticity in the connections from an Open vSwitch to an OpenFlow controller. If libssl is installed, then Open vSwitch will automatically build with support for it. - libcap-ng, written by Steve Grubb, is optional but recommended. It is required to run OVS daemons as a non-root user with dropped root privileges. If libcap-ng is installed, then Open vSwitch will automatically build with support for it. - Python 2.7. You must also have the Python ``six`` library version 1.4.0 or later. On Linux, you may choose to compile the kernel module that comes with the Open vSwitch distribution or to use the kernel module built into the Linux kernel (version 3.3 or later). See the :doc:`/faq/index` question "What features are not available in the Open vSwitch kernel datapath that ships as part of the upstream Linux kernel?" for more information on this trade-off. You may also use the userspace-only implementation, at some cost in features and performance. Refer to :doc:`userspace` for details. To compile the kernel module on Linux, you must also install the following: - A supported Linux kernel version. For optional support of ingress policing, you must enable kernel configuration options ``NET_CLS_BASIC``, ``NET_SCH_INGRESS``, and ``NET_ACT_POLICE``, either built-in or as modules. ``NET_CLS_POLICE`` is obsolete and not needed.) On kernels before 3.11, the ``ip_gre`` module, for GRE tunnels over IP (``NET_IPGRE``), must not be loaded or compiled in. To configure HTB or HFSC quality of service with Open vSwitch, you must enable the respective configuration options. To use Open vSwitch support for TAP devices, you must enable ``CONFIG_TUN``. - To build a kernel module, you need the same version of GCC that was used to build that kernel. - A kernel build directory corresponding to the Linux kernel image the module is to run on. Under Debian and Ubuntu, for example, each linux-image package containing a kernel binary has a corresponding linux-headers package with the required build infrastructure. If you are working from a Git tree or snapshot (instead of from a distribution tarball), or if you modify the Open vSwitch build system or the database schema, you will also need the following software: - Autoconf version 2.63 or later. - Automake version 1.10 or later. - libtool version 2.4 or later. (Older versions might work too.) The datapath tests for userspace and Linux datapaths also rely upon: - pyftpdlib. Version 1.2.0 is known to work. Earlier versions should also work. - GNU wget. Version 1.16 is known to work. Earlier versions should also work. - netcat. Several common implementations are known to work. - curl. Version 7.47.0 is known to work. Earlier versions should also work. - tftpy. Version 0.6.2 is known to work. Earlier versions should also work. The ovs-vswitchd.conf.db(5) manpage will include an E-R diagram, in formats other than plain text, only if you have the following: - dot from graphviz (http://www.graphviz.org/). If you are going to extensively modify Open vSwitch, consider installing the following to obtain better warnings: - "sparse" version 0.5.1 or later (https://git.kernel.org/pub/scm/devel/sparse/sparse.git/). - GNU make. - clang, version 3.4 or later - flake8 along with the hacking flake8 plugin (for Python code). The automatic flake8 check that runs against Python code has some warnings enabled that come from the "hacking" flake8 plugin. If it's not installed, the warnings just won't occur until it's run on a system with "hacking" installed. You may find the ovs-dev script found in ``utilities/ovs-dev.py`` useful. .. _general-install-reqs: Installation Requirements ------------------------- The machine you build Open vSwitch on may not be the one you run it on. To simply install and run Open vSwitch you require the following software: - Shared libraries compatible with those used for the build. - On Linux, if you want to use the kernel-based datapath (which is the most common use case), then a kernel with a compatible kernel module. This can be a kernel module built with Open vSwitch (e.g. in the previous step), or the kernel module that accompanies Linux 3.3 and later. Open vSwitch features and performance can vary based on the module and the kernel. Refer to :doc:`/faq/releases` for more information. - For optional support of ingress policing on Linux, the "tc" program from iproute2 (part of all major distributions and available at https://wiki.linuxfoundation.org/networking/iproute2). - Python 2.7. You must also have the Python six library version 1.4.0 or later. On Linux you should ensure that ``/dev/urandom`` exists. To support TAP devices, you must also ensure that ``/dev/net/tun`` exists. .. _general-bootstrapping: Bootstrapping ------------- This step is not needed if you have downloaded a released tarball. If you pulled the sources directly from an Open vSwitch Git tree or got a Git tree snapshot, then run boot.sh in the top source directory to build the "configure" script:: $ ./boot.sh .. _general-configuring: Configuring ----------- Configure the package by running the configure script. You can usually invoke configure without any arguments. For example:: $ ./configure By default all files are installed under ``/usr/local``. Open vSwitch also expects to find its database in ``/usr/local/etc/openvswitch`` by default. If you want to install all files into, e.g., ``/usr`` and ``/var`` instead of ``/usr/local`` and ``/usr/local/var`` and expect to use ``/etc/openvswitch`` as the default database directory, add options as shown here:: $ ./configure --prefix=/usr --localstatedir=/var --sysconfdir=/etc .. note:: Open vSwitch installed with packages like .rpm (e.g. via ``yum install`` or ``rpm -ivh``) and .deb (e.g. via ``apt-get install`` or ``dpkg -i``) use the above configure options. By default, static libraries are built and linked against. If you want to use shared libraries instead:: $ ./configure --enable-shared To use a specific C compiler for compiling Open vSwitch user programs, also specify it on the configure command line, like so:: $ ./configure CC=gcc-4.2 To use 'clang' compiler:: $ ./configure CC=clang To supply special flags to the C compiler, specify them as ``CFLAGS`` on the configure command line. If you want the default CFLAGS, which include ``-g`` to build debug symbols and ``-O2`` to enable optimizations, you must include them yourself. For example, to build with the default CFLAGS plus ``-mssse3``, you might run configure as follows:: $ ./configure CFLAGS="-g -O2 -mssse3" For efficient hash computation special flags can be passed to leverage built-in intrinsics. For example on X86_64 with SSE4.2 instruction set support, CRC32 intrinsics can be used by passing ``-msse4.2``:: $ ./configure CFLAGS="-g -O2 -msse4.2"` Also builtin popcnt instruction can be used to speedup the counting of the bits set in an integer. For example on X86_64 with POPCNT support, it can be enabled by passing ``-mpopcnt``:: $ ./configure CFLAGS="-g -O2 -mpopcnt"` If you are on a different processor and don't know what flags to choose, it is recommended to use ``-march=native`` settings:: $ ./configure CFLAGS="-g -O2 -march=native" With this, GCC will detect the processor and automatically set appropriate flags for it. This should not be used if you are compiling OVS outside the target machine. .. note:: CFLAGS are not applied when building the Linux kernel module. Custom CFLAGS for the kernel module are supplied using the ``EXTRA_CFLAGS`` variable when running make. For example:: $ make EXTRA_CFLAGS="-Wno-error=date-time" To build the Linux kernel module, so that you can run the kernel-based switch, pass the location of the kernel build directory on ``--with-linux``. For example, to build for a running instance of Linux:: $ ./configure --with-linux=/lib/modules/$(uname -r)/build .. note:: If ``--with-linux`` requests building for an unsupported version of Linux, then ``configure`` will fail with an error message. Refer to the :doc:`/faq/index` for advice in that case. If you wish to build the kernel module for an architecture other than the architecture of the machine used for the build, you may specify the kernel architecture string using the KARCH variable when invoking the configure script. For example, to build for MIPS with Linux:: $ ./configure --with-linux=/path/to/linux KARCH=mips If you plan to do much Open vSwitch development, you might want to add ``--enable-Werror``, which adds the ``-Werror`` option to the compiler command line, turning warnings into errors. That makes it impossible to miss warnings generated by the build. For example:: $ ./configure --enable-Werror If you're building with GCC, then, for improved warnings, install ``sparse`` (see "Prerequisites") and enable it for the build by adding ``--enable-sparse``. Use this with ``--enable-Werror`` to avoid missing both compiler and ``sparse`` warnings, e.g.:: $ ./configure --enable-Werror --enable-sparse To build with gcov code coverage support, add ``--enable-coverage``:: $ ./configure --enable-coverage The configure script accepts a number of other options and honors additional environment variables. For a full list, invoke configure with the ``--help`` option:: $ ./configure --help You can also run configure from a separate build directory. This is helpful if you want to build Open vSwitch in more than one way from a single source directory, e.g. to try out both GCC and Clang builds, or to build kernel modules for more than one Linux version. For example:: $ mkdir _gcc && (cd _gcc && ./configure CC=gcc) $ mkdir _clang && (cd _clang && ./configure CC=clang) Under certains loads the ovsdb-server and other components perform better when using the jemalloc memory allocator, instead of the glibc memory allocator. If you wish to link with jemalloc add it to LIBS:: $ ./configure LIBS=-ljemalloc .. _general-building: Building -------- 1. Run GNU make in the build directory, e.g.:: $ make or if GNU make is installed as "gmake":: $ gmake If you used a separate build directory, run make or gmake from that directory, e.g.:: $ make -C _gcc $ make -C _clang .. note:: Some versions of Clang and ccache are not completely compatible. If you see unusual warnings when you use both together, consider disabling ccache. 2. Consider running the testsuite. Refer to :doc:`/topics/testing` for instructions. 3. Run ``make install`` to install the executables and manpages into the running system, by default under ``/usr/local``:: $ make install 5. If you built kernel modules, you may install them, e.g.:: $ make modules_install It is possible that you already had a Open vSwitch kernel module installed on your machine that came from upstream Linux (in a different directory). To make sure that you load the Open vSwitch kernel module you built from this repository, you should create a ``depmod.d`` file that prefers your newly installed kernel modules over the kernel modules from upstream Linux. The following snippet of code achieves the same:: $ config_file="/etc/depmod.d/openvswitch.conf" $ for module in datapath/linux/*.ko; do modname="$(basename ${module})" echo "override ${modname%.ko} * extra" >> "$config_file" echo "override ${modname%.ko} * weak-updates" >> "$config_file" done $ depmod -a Finally, load the kernel modules that you need. e.g.:: $ /sbin/modprobe openvswitch To verify that the modules have been loaded, run ``/sbin/lsmod`` and check that openvswitch is listed:: $ /sbin/lsmod | grep openvswitch .. note:: If the ``modprobe`` operation fails, look at the last few kernel log messages (e.g. with ``dmesg | tail``). Generally, issues like this occur when Open vSwitch is built for a kernel different from the one into which you are trying to load it. Run ``modinfo`` on ``openvswitch.ko`` and on a module built for the running kernel, e.g.:: $ /sbin/modinfo openvswitch.ko $ /sbin/modinfo /lib/modules/$(uname -r)/kernel/net/bridge/bridge.ko Compare the "vermagic" lines output by the two commands. If they differ, then Open vSwitch was built for the wrong kernel. If you decide to report a bug or ask a question related to module loading, include the output from the ``dmesg`` and ``modinfo`` commands mentioned above. .. _general-starting: Starting -------- On Unix-alike systems, such as BSDs and Linux, starting the Open vSwitch suite of daemons is a simple process. Open vSwitch includes a shell script, and helpers, called ovs-ctl which automates much of the tasks for starting and stopping ovsdb-server, and ovs-vswitchd. After installation, the daemons can be started by using the ovs-ctl utility. This will take care to setup initial conditions, and start the daemons in the correct order. The ovs-ctl utility is located in '$(pkgdatadir)/scripts', and defaults to '/usr/local/share/openvswitch/scripts'. An example after install might be:: $ export PATH=$PATH:/usr/local/share/openvswitch/scripts $ ovs-ctl start Additionally, the ovs-ctl script allows starting / stopping the daemons individually using specific options. To start just the ovsdb-server:: $ export PATH=$PATH:/usr/local/share/openvswitch/scripts $ ovs-ctl --no-ovs-vswitchd start Likewise, to start just the ovs-vswitchd:: $ export PATH=$PATH:/usr/local/share/openvswitch/scripts $ ovs-ctl --no-ovsdb-server start Refer to ovs-ctl(8) for more information on ovs-ctl. In addition to using the automated script to start Open vSwitch, you may wish to manually start the various daemons. Before starting ovs-vswitchd itself, you need to start its configuration database, ovsdb-server. Each machine on which Open vSwitch is installed should run its own copy of ovsdb-server. Before ovsdb-server itself can be started, configure a database that it can use:: $ mkdir -p /usr/local/etc/openvswitch $ ovsdb-tool create /usr/local/etc/openvswitch/conf.db \ vswitchd/vswitch.ovsschema Configure ovsdb-server to use database created above, to listen on a Unix domain socket, to connect to any managers specified in the database itself, and to use the SSL configuration in the database:: $ mkdir -p /usr/local/var/run/openvswitch $ ovsdb-server --remote=punix:/usr/local/var/run/openvswitch/db.sock \ --remote=db:Open_vSwitch,Open_vSwitch,manager_options \ --private-key=db:Open_vSwitch,SSL,private_key \ --certificate=db:Open_vSwitch,SSL,certificate \ --bootstrap-ca-cert=db:Open_vSwitch,SSL,ca_cert \ --pidfile --detach --log-file .. note:: If you built Open vSwitch without SSL support, then omit ``--private-key``, ``--certificate``, and ``--bootstrap-ca-cert``.) Initialize the database using ovs-vsctl. This is only necessary the first time after you create the database with ovsdb-tool, though running it at any time is harmless:: $ ovs-vsctl --no-wait init Start the main Open vSwitch daemon, telling it to connect to the same Unix domain socket:: $ ovs-vswitchd --pidfile --detach --log-file Validating ---------- At this point you can use ovs-vsctl to set up bridges and other Open vSwitch features. For example, to create a bridge named ``br0`` and add ports ``eth0`` and ``vif1.0`` to it:: $ ovs-vsctl add-br br0 $ ovs-vsctl add-port br0 eth0 $ ovs-vsctl add-port br0 vif1.0 Refer to ovs-vsctl(8) for more details. You may also wish to refer to :doc:`/topics/testing` for information on more generic testing of OVS. Upgrading --------- When you upgrade Open vSwitch from one version to another you should also upgrade the database schema: .. note:: The following manual steps may also be accomplished by using ovs-ctl to stop and start the daemons after upgrade. The ovs-ctl script will automatically upgrade the schema. 1. Stop the Open vSwitch daemons, e.g.:: $ kill `cd /usr/local/var/run/openvswitch && cat ovsdb-server.pid ovs-vswitchd.pid` 2. Install the new Open vSwitch release by using the same configure options as was used for installing the previous version. If you do not use the same configure options, you can end up with two different versions of Open vSwitch executables installed in different locations. 3. Upgrade the database, in one of the following two ways: - If there is no important data in your database, then you may delete the database file and recreate it with ovsdb-tool, following the instructions under "Building and Installing Open vSwitch for Linux, FreeBSD or NetBSD". - If you want to preserve the contents of your database, back it up first, then use ``ovsdb-tool convert`` to upgrade it, e.g.:: $ ovsdb-tool convert /usr/local/etc/openvswitch/conf.db \ vswitchd/vswitch.ovsschema 4. Start the Open vSwitch daemons as described under `Starting`_ above. Hot Upgrading ------------- Upgrading Open vSwitch from one version to the next version with minimum disruption of traffic going through the system that is using that Open vSwitch needs some considerations: 1. If the upgrade only involves upgrading the userspace utilities and daemons of Open vSwitch, make sure that the new userspace version is compatible with the previously loaded kernel module. 2. An upgrade of userspace daemons means that they have to be restarted. Restarting the daemons means that the OpenFlow flows in the ovs-vswitchd daemon will be lost. One way to restore the flows is to let the controller re-populate it. Another way is to save the previous flows using a utility like ovs-ofctl and then re-add them after the restart. Restoring the old flows is accurate only if the new Open vSwitch interfaces retain the old 'ofport' values. 3. When the new userspace daemons get restarted, they automatically flush the old flows setup in the kernel. This can be expensive if there are hundreds of new flows that are entering the kernel but userspace daemons are busy setting up new userspace flows from either the controller or an utility like ovs-ofctl. Open vSwitch database provides an option to solve this problem through the ``other_config:flow-restore-wait`` column of the ``Open_vSwitch`` table. Refer to the ovs-vswitchd.conf.db(5) manpage for details. 4. If the upgrade also involves upgrading the kernel module, the old kernel module needs to be unloaded and the new kernel module should be loaded. This means that the kernel network devices belonging to Open vSwitch is recreated and the kernel flows are lost. The downtime of the traffic can be reduced if the userspace daemons are restarted immediately and the userspace flows are restored as soon as possible. The ovs-ctl utility's ``restart`` function only restarts the userspace daemons, makes sure that the 'ofport' values remain consistent across restarts, restores userspace flows using the ovs-ofctl utility and also uses the ``other_config:flow-restore-wait`` column to keep the traffic downtime to the minimum. The ovs-ctl utility's ``force-reload-kmod`` function does all of the above, but also replaces the old kernel module with the new one. Open vSwitch startup scripts for Debian, XenServer and RHEL use ovs-ctl's functions and it is recommended that these functions be used for other software platforms too. Reporting Bugs -------------- Report problems to bugs@openvswitch.org. ovs-2.9.0/Documentation/intro/install/index.rst000066400000000000000000000035211324262074100216060ustar00rootroot00000000000000.. Copyright (c) 2016, Stephen Finucane Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. Convention for heading levels in Open vSwitch documentation: ======= Heading 0 (reserved for the title in a document) ------- Heading 1 ~~~~~~~ Heading 2 +++++++ Heading 3 ''''''' Heading 4 Avoid deeper levels because they do not render well. ======================= Installing Open vSwitch ======================= A collection of guides detailing how to install Open vSwitch in a variety of different environments and using different configurations. Installation from Source ------------------------ .. TODO(stephenfin): Based on the title alone, the NetBSD doc should probably be merged into the general install doc .. toctree:: :maxdepth: 2 general netbsd windows xenserver userspace dpdk Installation from Packages -------------------------- Open vSwitch is packaged on a variety of distributions. The tooling required to build these packages is included in the Open vSwitch tree. The instructions are provided below. .. toctree:: :maxdepth: 2 distributions debian fedora rhel Upgrades -------- .. toctree:: :maxdepth: 2 ovn-upgrades Others ------ .. toctree:: :maxdepth: 2 bash-completion documentation ovs-2.9.0/Documentation/intro/install/netbsd.rst000066400000000000000000000040761324262074100217640ustar00rootroot00000000000000.. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. Convention for heading levels in Open vSwitch documentation: ======= Heading 0 (reserved for the title in a document) ------- Heading 1 ~~~~~~~ Heading 2 +++++++ Heading 3 ''''''' Heading 4 Avoid deeper levels because they do not render well. ====================== Open vSwitch on NetBSD ====================== On NetBSD, you might want to install requirements from pkgsrc. In that case, you need at least the following packages. - automake - libtool-base - gmake - python27 - py27-six - py27-xml Some components have additional requirements. Refer to :doc:`general` for more information. Assuming you are running NetBSD/amd64 6.1.2, you can download and install pre-built binary packages as the following:: $ PKG_PATH=http://ftp.netbsd.org/pub/pkgsrc/packages/NetBSD/amd64/7.0.2/All/ $ export PKG_PATH $ pkg_add automake libtool-base gmake python27 py27-six py27-xml \ pkg_alternatives .. note:: You might get some warnings about minor version mismatch. These can be safely ignored. NetBSD's ``/usr/bin/make`` is not GNU make. GNU make is installed as ``/usr/pkg/bin/gmake`` by the above mentioned ``gmake`` package. As all executables installed with pkgsrc are placed in ``/usr/pkg/bin/`` directory, it might be a good idea to add it to your PATH. Or install ovs by ``gmake`` and ``gmake install``. Open vSwitch on NetBSD is currently "userspace switch" implementation in the sense described in :doc:`userspace` and :doc:`/topics/porting`. ovs-2.9.0/Documentation/intro/install/ovn-upgrades.rst000066400000000000000000000057331324262074100231200ustar00rootroot00000000000000.. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. Convention for heading levels in Open vSwitch documentation: ======= Heading 0 (reserved for the title in a document) ------- Heading 1 ~~~~~~~ Heading 2 +++++++ Heading 3 ''''''' Heading 4 Avoid deeper levels because they do not render well. ============ OVN Upgrades ============ Since OVN is a distributed system, special consideration must be given to the process used to upgrade OVN across a deployment. This document discusses the recommended upgrade process. Release Notes ------------- You should always check the OVS and OVN release notes (NEWS file) for any release specific notes on upgrades. OVS --- OVN depends on and is included with OVS. It's expected that OVS and OVN are upgraded together, partly for convenience. OVN is included in OVS releases so it's easiest to upgrade them together. OVN may also make use of new features of OVS only available in that release. Upgrade ovn-controller ---------------------- You should start by upgrading ovn-controller on each host it's running on. First, you upgrade the OVS and OVN packages. Then, restart the ovn-controller service. You can restart with ovn-ctl:: $ sudo /usr/share/openvswitch/scripts/ovn-ctl restart_controller or with systemd:: $ sudo systemd restart ovn-controller Upgrade OVN Databases and ovn-northd ------------------------------------ The OVN databases and ovn-northd should be upgraded next. Since ovn-controller has already been upgraded, it will be ready to operate on any new functionality specified by the database or logical flows created by ovn-northd. Upgrading the OVN packages installs everything needed for an upgrade. The only step required after upgrading the packages is to restart ovn-northd, which automatically restarts the databases and upgrades the database schema, as well. You may perform this restart using the ovn-ctl script:: $ sudo /usr/share/openvswitch/scripts/ovn-ctl restart_northd or if you're using a Linux distribution with systemd:: $ sudo systemctl restart ovn-northd Upgrading OVN Integration ------------------------- Lastly, you may also want to upgrade integration with OVN that you may be using. For example, this could be the OpenStack Neutron driver or ovn-kubernetes. OVN's northbound database schema is a backwards compatible interface, so you should be able to safely complete an OVN upgrade before upgrading any integration in use. ovs-2.9.0/Documentation/intro/install/rhel.rst000066400000000000000000000177051324262074100214420ustar00rootroot00000000000000.. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. Convention for heading levels in Open vSwitch documentation: ======= Heading 0 (reserved for the title in a document) ------- Heading 1 ~~~~~~~ Heading 2 +++++++ Heading 3 ''''''' Heading 4 Avoid deeper levels because they do not render well. ======================================== RHEL 5.6, 6.x Packaging for Open vSwitch ======================================== This document describes how to build and install Open vSwitch on a Red Hat Enterprise Linux (RHEL) host. If you want to install Open vSwitch on a generic Linux host, refer to :doc:`general` instead. We have tested these instructions with RHEL 5.6 and RHEL 6.0. For RHEL 7.x (or derivatives, such as CentOS 7.x), you should follow the instructions in the :doc:`fedora`. The Fedora spec files are used for RHEL 7.x. .. _rhel-prerequisites: Prerequisites ------------- You may build from an Open vSwitch distribution tarball or from an Open vSwitch Git tree. The default RPM build directory, ``_topdir``, has five directories in the top-level. BUILD/ where the software is unpacked and built RPMS/ where the newly created binary package files are written SOURCES/ contains the original sources, patches, and icon files SPECS/ contains the spec files for each package to be built SRPMS/ where the newly created source package files are written Before you begin, note the RPM sources directory on your version of RHEL. The command ``rpmbuild --showrc`` will show the configuration for each of those directories. Alternatively, the command ``rpm --eval '%{_topdir}'`` shows the current configuration for the top level directory and the command ``rpm --eval '%{_sourcedir}'`` does the same for the sources directory. On RHEL 5, the default RPM ``_topdir`` is ``/usr/src/redhat`` and the default RPM sources directory is ``/usr/src/redhat/SOURCES``. On RHEL 6, the default ``_topdir`` is ``$HOME/rpmbuild`` and the default RPM sources directory is ``$HOME/rpmbuild/SOURCES``. Build Requirements ------------------ You will need to install all required packages to build the RPMs. The command below will install RPM tools and generic build dependencies:: $ yum install @'Development Tools' rpm-build yum-utils Then it is necessary to install Open vSwitch specific build dependencies. The dependencies are listed in the SPEC file, but first it is necessary to replace the VERSION tag to be a valid SPEC. The command below will create a temporary SPEC file:: $ sed -e 's/@VERSION@/0.0.1/' rhel/openvswitch.spec.in > /tmp/ovs.spec And to install specific dependencies, use yum-builddep tool:: $ yum-builddep /tmp/ovs.spec Once that is completed, remove the file ``/tmp/ovs.spec``. If python-sphinx package is not available in your version of RHEL, you can install it via pip with 'pip install sphinx'. Open vSwitch requires python 2.7 or newer which is not available in older distributions. In the case of RHEL 6.x and its derivatives, one option is to install python34 and python34-six from `EPEL`_. .. _EPEL: https://fedoraproject.org/wiki/EPEL .. _rhel-bootstrapping: Bootstrapping and Configuring ----------------------------- If you are building from a distribution tarball, skip to :ref:`rhel-building`. If not, you must be building from an Open vSwitch Git tree. Determine what version of Autoconf is installed (e.g. run ``autoconf --version``). If it is not at least version 2.63, then you must upgrade or use another machine to build the packages. Assuming all requirements have been met, build the tarball by running:: $ ./boot.sh $ ./configure $ make dist You must run this on a machine that has the tools listed in :ref:`general-build-reqs` as prerequisites for building from a Git tree. Afterward, proceed with the rest of the instructions using the distribution tarball. Now you have a distribution tarball, named something like ``openvswitch-x.y.z.tar.gz``. Copy this file into the RPM sources directory, e.g.:: $ cp openvswitch-x.y.z.tar.gz $HOME/rpmbuild/SOURCES Broken ``build`` symlink ~~~~~~~~~~~~~~~~~~~~~~~~ Some versions of the RHEL 6 kernel-devel package contain a broken ``build`` symlink. If you are using such a version, you must fix the problem before continuing. To find out whether you are affected, run:: $ cd /lib/modules/ $ ls -l build/ where ```` is the version number of the RHEL 6 kernel. .. note:: The trailing slash in the final command is important. Be sure to include it. If the ``ls`` command produces a directory listing, your kernel-devel package is OK. If it produces a ``No such file or directory`` error, your kernel-devel package is buggy. If your kernel-devel package is buggy, then you can fix it with:: $ cd /lib/modules/ $ rm build $ ln -s /usr/src/kernels/ build where ```` is the name of an existing directory under ``/usr/src/kernels``, whose name should be similar to ```` but may contain some extra parts. Once you have done this, verify the fix with the same procedure you used above to check for the problem. .. _rhel-building: Building -------- You should have a distribution tarball named something like openvswitch-x.y.z.tar.gz. Copy this file into the RPM sources directory:: $ cp openvswitch-x.y.z.tar.gz $HOME/rpmbuild/SOURCES Make another copy of the distribution tarball in a temporary directory. Then unpack the tarball and ``cd`` into its root:: $ tar xzf openvswitch-x.y.z.tar.gz $ cd openvswitch-x.y.z Userspace ~~~~~~~~~ To build Open vSwitch userspace, run:: $ rpmbuild -bb rhel/openvswitch.spec This produces two RPMs: "openvswitch" and "openvswitch-debuginfo". The above command automatically runs the Open vSwitch unit tests. To disable the unit tests, run:: $ rpmbuild -bb --without check rhel/openvswitch.spec .. note:: If the build fails with ``configure: error: source dir /lib/modules/2.6.32-279.el6.x86_64/build doesn't exist`` or similar, then the kernel-devel package is missing or buggy. Kernel Module ~~~~~~~~~~~~~ On RHEL 6, to build the Open vSwitch kernel module, copy rhel/openvswitch-kmod.files into the RPM sources directory and run:: $ rpmbuild -bb rhel/openvswitch-kmod-rhel6.spec You might have to specify a kernel version and/or variants, e.g.: $ rpmbuild -bb \ -D "kversion 2.6.32-131.6.1.el6.x86_64" \ -D "kflavors default debug kdump" \ rhel/openvswitch-kmod-rhel6.spec This produces an "kmod-openvswitch" RPM for each kernel variant, in this example: "kmod-openvswitch", "kmod-openvswitch-debug", and "kmod-openvswitch-kdump". .. _rhel-script-integrations: Red Hat Network Scripts Integration ----------------------------------- A RHEL host has default firewall rules that prevent any Open vSwitch tunnel traffic from passing through. If a user configures Open vSwitch tunnels like Geneve, GRE, VXLAN, LISP etc., they will either have to manually add iptables firewall rules to allow the tunnel traffic or add it through a startup script Refer to the "enable-protocol" command in the ovs-ctl(8) manpage for more information. In addition, simple integration with Red Hat network scripts has been implemented. Refer to `README.RHEL.rst`__ in the source tree or /usr/share/doc/openvswitch/README.RHEL.rst in the installed openvswitch package for details. __ https://github.com/openvswitch/ovs/blob/master/rhel/README.RHEL.rst Reporting Bugs -------------- Report problems to bugs@openvswitch.org. ovs-2.9.0/Documentation/intro/install/userspace.rst000066400000000000000000000077341324262074100225030ustar00rootroot00000000000000.. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. Convention for heading levels in Open vSwitch documentation: ======= Heading 0 (reserved for the title in a document) ------- Heading 1 ~~~~~~~ Heading 2 +++++++ Heading 3 ''''''' Heading 4 Avoid deeper levels because they do not render well. =================================== Open vSwitch without Kernel Support =================================== Open vSwitch can operate, at a cost in performance, entirely in userspace, without assistance from a kernel module. This file explains how to install Open vSwitch in such a mode. This version of Open vSwitch should be built manually with ``configure`` and ``make``. Debian packaging for Open vSwitch is also included, but it has not been recently tested, and so Debian packages are not a recommended way to use this version of Open vSwitch. .. warning:: The userspace-only mode of Open vSwitch without DPDK is considered experimental. It has not been thoroughly tested. Building and Installing ----------------------- The requirements and procedure for building, installing, and configuring Open vSwitch are the same as those given in :doc:`general`. You may omit configuring, building, and installing the kernel module, and the related requirements. On Linux, the userspace switch additionally requires the kernel TUN/TAP driver to be available, either built into the kernel or loaded as a module. If you are not sure, check for a directory named ``/sys/class/misc/tun``. If it does not exist, then attempt to load the module with ``modprobe tun``. The tun device must also exist as ``/dev/net/tun``. If it does not exist, then create ``/dev/net`` (if necessary) with ``mkdir /dev/net``, then create ``/dev/net/tun`` with ``mknod /dev/net/tun c 10 200``. On FreeBSD and NetBSD, the userspace switch additionally requires the kernel tap(4) driver to be available, either built into the kernel or loaded as a module. Using the Userspace Datapath with ovs-vswitchd ---------------------------------------------- To use ovs-vswitchd in userspace mode, create a bridge with ``datapath_type=netdev`` in the configuration database. For example:: $ ovs-vsctl add-br br0 $ ovs-vsctl set bridge br0 datapath_type=netdev $ ovs-vsctl add-port br0 eth0 $ ovs-vsctl add-port br0 eth1 $ ovs-vsctl add-port br0 eth2 ovs-vswitchd will create a TAP device as the bridge's local interface, named the same as the bridge, as well as for each configured internal interface. Currently, on FreeBSD, the functionality required for in-band control support is not implemented. To avoid related errors, you can disable the in-band support with the following command:: $ ovs-vsctl set bridge br0 other_config:disable-in-band=true Firewall Rules -------------- On Linux, when a physical interface is in use by the userspace datapath, packets received on the interface still also pass into the kernel TCP/IP stack. This can cause surprising and incorrect behavior. You can use "iptables" to avoid this behavior, by using it to drop received packets. For example, to drop packets received on eth0:: $ iptables -A INPUT -i eth0 -j DROP $ iptables -A FORWARD -i eth0 -j DROP Other Settings -------------- On NetBSD, depending on your network topology and applications, the following configuration might help. See sysctl(7).:: $ sysctl -w net.inet.ip.checkinterface=1 Reporting Bugs -------------- Report problems to bugs@openvswitch.org. ovs-2.9.0/Documentation/intro/install/windows.rst000066400000000000000000000620171324262074100221760ustar00rootroot00000000000000.. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. Convention for heading levels in Open vSwitch documentation: ======= Heading 0 (reserved for the title in a document) ------- Heading 1 ~~~~~~~ Heading 2 +++++++ Heading 3 ''''''' Heading 4 Avoid deeper levels because they do not render well. ======================= Open vSwitch on Windows ======================= .. _windows-build-reqs: Build Requirements ------------------ Open vSwitch on Linux uses autoconf and automake for generating Makefiles. It will be useful to maintain the same build system while compiling on Windows too. One approach is to compile Open vSwitch in a MinGW environment that contains autoconf and automake utilities and then use Visual C++ as a compiler and linker. The following explains the steps in some detail. - Mingw Install Mingw on a Windows machine by following the instructions on `mingw.org `__. This should install mingw at ``C:\Mingw`` and msys at ``C:\Mingw\msys``. Add ``C:\MinGW\bin`` and ``C:\Mingw\msys\1.0\bin`` to PATH environment variable of Windows. You can either use the MinGW installer or the command line utility ``mingw-get`` to install both the base packages and additional packages like automake and autoconf(version 2.68). Also make sure that ``/mingw`` mount point exists. If its not, please add/create the following entry in ``/etc/fstab``:: 'C:/MinGW /mingw'. - Python Install the latest Python 2.x from python.org and verify that its path is part of Windows' PATH environment variable. We require that you have Python six and pypiwin32 libraries installed. The libraries can be installed via pip command: :: $ pip install six $ pip install pypiwin32 - Visual Studio You will need at least Visual Studio 2013 (update 4) to compile userspace binaries. In addition to that, if you want to compile the kernel module you will also need to install Windows Driver Kit (WDK) 8.1 Update. It is important to get the Visual Studio related environment variables and to have the $PATH inside the bash to point to the proper compiler and linker. One easy way to achieve this for VS2013 is to get into the "VS2013 x86 Native Tools Command Prompt" (in a default installation of Visual Studio 2013 this can be found under the following location: ``C:\Program Files (x86)\Microsoft Visual Studio 12.0\Common7\Tools\Shortcuts``) and through it enter into the bash shell available from msys by typing ``bash --login``. There is support for generating 64 bit binaries too. To compile under x64, open the "VS2013 x64 Native Tools Command Prompt" (if your current running OS is 64 bit) or "VS2013 x64 Cross Tools Command Prompt" (if your current running OS is not 64 bit) instead of opening its x86 variant. This will point the compiler and the linker to their 64 bit equivalent. If after the above step, a ``which link`` inside MSYS's bash says, ``/bin/link.exe``, rename ``/bin/link.exe`` to something else so that the Visual studio's linker is used. You should also see a 'which sort' report ``/bin/sort.exe``. - pthreads-win32 For pthread support, install the library, dll and includes of pthreads-win32 project from `sourceware `__ to a directory (e.g.: ``C:/pthread``). You should add the pthread-win32's dll path (e.g.: ``C:\pthread\dll\x86``) to the Windows' PATH environment variable. - OpenSSL To get SSL support for Open vSwitch on Windows, you will need to install `OpenSSL for Windows `__ Note down the directory where OpenSSL is installed (e.g.: ``C:/OpenSSL-Win32``) for later use. .. note:: Commands prefixed by ``$`` must be run in the Bash shell provided by MinGW. Open vSwitch commands, such as ``ovs-dpctl`` are shown running under the DOS shell (``cmd.exe``), as indicated by the ``>`` prefix, but will also run under Bash. The remainder, prefixed by ``>``, are PowerShell commands and must be run in PowerShell. Install Requirements -------------------- * Share network adaptors We require that you don't disable the "Allow management operating system to share this network adapter" under 'Virtual Switch Properties' > 'Connection type: External network', in the HyperV virtual network switch configuration. * Checksum Offloads While there is some support for checksum/segmentation offloads in software, this is still a work in progress. Till the support is complete we recommend disabling TX/RX offloads for both the VM's as well as the HyperV. Bootstrapping ------------- This step is not needed if you have downloaded a released tarball. If you pulled the sources directly from an Open vSwitch Git tree or got a Git tree snapshot, then run boot.sh in the top source directory to build the "configure" script: :: $ ./boot.sh .. _windows-configuring: Configuring ----------- Configure the package by running the configure script. You should provide some configure options to choose the right compiler, linker, libraries, Open vSwitch component installation directories, etc. For example: :: $ ./configure CC=./build-aux/cccl LD="$(which link)" \ LIBS="-lws2_32 -liphlpapi -lwbemuuid -lole32 -loleaut32" \ --prefix="C:/openvswitch/usr" \ --localstatedir="C:/openvswitch/var" \ --sysconfdir="C:/openvswitch/etc" \ --with-pthread="C:/pthread" .. note:: By default, the above enables compiler optimization for fast code. For default compiler optimization, pass the ``--with-debug`` configure option. To configure with SSL support, add the requisite additional options: :: $ ./configure CC=./build-aux/cccl LD="`which link`" \ LIBS="-lws2_32 -liphlpapi -lwbemuuid -lole32 -loleaut32" \ --prefix="C:/openvswitch/usr" \ --localstatedir="C:/openvswitch/var" --sysconfdir="C:/openvswitch/etc" \ --with-pthread="C:/pthread" \ --enable-ssl --with-openssl="C:/OpenSSL-Win32" Finally, to the kernel module also: :: $ ./configure CC=./build-aux/cccl LD="`which link`" \ LIBS="-lws2_32 -liphlpapi -lwbemuuid -lole32 -loleaut32" \ --prefix="C:/openvswitch/usr" \ --localstatedir="C:/openvswitch/var" \ --sysconfdir="C:/openvswitch/etc" \ --with-pthread="C:/pthread" \ --enable-ssl --with-openssl="C:/OpenSSL-Win32" \ --with-vstudiotarget="" Possible values for ```` are: ``Debug`` and ``Release`` .. note:: You can directly use the Visual Studio 2013 IDE to compile the kernel datapath. Open the ovsext.sln file in the IDE and build the solution. Refer to :doc:`general` for information on additional configuration options. .. _windows-building: Building -------- Once correctly configured, building Open vSwitch on Windows is similar to building on Linux, FreeBSD, or NetBSD. #. Run make for the ported executables in the top source directory, e.g.: :: $ make For faster compilation, you can pass the ``-j`` argument to make. For example, to run 4 jobs simultaneously, run ``make -j4``. .. note:: MSYS 1.0.18 has a bug that causes parallel make to hang. You can overcome this by downgrading to MSYS 1.0.17. A simple way to downgrade is to exit all MinGW sessions and then run the below command from MSVC developers command prompt.: :: > mingw-get upgrade msys-core-bin=1.0.17-1 #. To run all the unit tests in Open vSwitch, one at a time: :: $ make check To run all the unit tests in Open vSwitch, up to 8 in parallel: :: $ make check TESTSUITEFLAGS="-j8" #. To install all the compiled executables on the local machine, run: :: $ make install .. note:: This will install the Open vSwitch executables in ``C:/openvswitch``. You can add ``C:\openvswitch\usr\bin`` and ``C:\openvswitch\usr\sbin`` to Windows' PATH environment variable for easy access. The Kernel Module ~~~~~~~~~~~~~~~~~ If you are building the kernel module, you will need to copy the below files to the target Hyper-V machine. - ``./datapath-windows/x64/Win8.1Debug/package/ovsext.inf`` - ``./datapath-windows/x64/Win8.1Debug/package/OVSExt.sys`` - ``./datapath-windows/x64/Win8.1Debug/package/ovsext.cat`` - ``./datapath-windows/misc/install.cmd`` - ``./datapath-windows/misc/uninstall.cmd`` .. note:: The above path assumes that the kernel module has been built using Windows DDK 8.1 in Debug mode. Change the path appropriately, if a different WDK has been used. Now run ``./uninstall.cmd`` to remove the old extension. Once complete, run ``./install.cmd`` to insert the new one. For this to work you will have to turn on ``TESTSIGNING`` boot option or 'Disable Driver Signature Enforcement' during boot. The following commands can be used: :: > bcdedit /set LOADOPTIONS DISABLE_INTEGRITY_CHECKS > bcdedit /set TESTSIGNING ON > bcdedit /set nointegritychecks ON .. note:: You may have to restart the machine for the settings to take effect. In the Virtual Switch Manager configuration you can enable the Open vSwitch Extension on an existing switch or create a new switch. If you are using an existing switch, make sure to enable the "Allow Management OS" option for VXLAN to work (covered later). The command to create a new switch named 'OVS-Extended-Switch' using a physical NIC named 'Ethernet 1' is: :: PS > New-VMSwitch "OVS-Extended-Switch" -NetAdapterName "Ethernet 1" .. note:: You can obtain the list of physical NICs on the host using 'Get-NetAdapter' command. In the properties of any switch, you should should now see "Open vSwitch Extension" under 'Extensions'. Click the check box to enable the extension. An alternative way to do the same is to run the following command: :: PS > Enable-VMSwitchExtension "Open vSwitch Extension" OVS-Extended-Switch .. note:: If you enabled the extension using the command line, a delay of a few seconds has been observed for the change to be reflected in the UI. This is not a bug in Open vSwitch. Starting -------- .. important:: The following steps assume that you have installed the Open vSwitch utilities in the local machine via 'make install'. Before starting ovs-vswitchd itself, you need to start its configuration database, ovsdb-server. Each machine on which Open vSwitch is installed should run its own copy of ovsdb-server. Before ovsdb-server itself can be started, configure a database that it can use: :: > ovsdb-tool create C:\openvswitch\etc\openvswitch\conf.db \ C:\openvswitch\usr\share\openvswitch\vswitch.ovsschema Configure ovsdb-server to use database created above and to listen on a Unix domain socket: :: > ovsdb-server -vfile:info --remote=punix:db.sock --log-file \ --pidfile --detach .. note:: The logfile is created at ``C:/openvswitch/var/log/openvswitch/`` Initialize the database using ovs-vsctl. This is only necessary the first time after you create the database with ovsdb-tool, though running it at any time is harmless: :: > ovs-vsctl --no-wait init .. tip:: If you would later like to terminate the started ovsdb-server, run: :: > ovs-appctl -t ovsdb-server exit Start the main Open vSwitch daemon, telling it to connect to the same Unix domain socket: :: > ovs-vswitchd -vfile:info --log-file --pidfile --detach .. tip:: If you would like to terminate the started ovs-vswitchd, run: :: > ovs-appctl exit .. note:: The logfile is created at ``C:/openvswitch/var/log/openvswitch/`` Validating ---------- At this point you can use ovs-vsctl to set up bridges and other Open vSwitch features. Add bridges ~~~~~~~~~~~ Let's start by creating an integration bridge, ``br-int`` and a PIF bridge, ``br-pif``: :: > ovs-vsctl add-br br-int > ovs-vsctl add-br br-pif .. note:: There's a known bug that running the ovs-vsctl command does not terminate. This is generally solved by having ovs-vswitchd running. If you face the issue despite that, hit Ctrl-C to terminate ovs-vsctl and check the output to see if your command succeeded. Validate that ports are added by dumping from both ovs-dpctl and ovs-vsctl: :: > ovs-dpctl show system@ovs-system: lookups: hit:0 missed:0 lost:0 flows: 0 port 2: br-pif (internal) <<< internal port on 'br-pif' bridge port 1: br-int (internal) <<< internal port on 'br-int' bridge > ovs-vsctl show a56ec7b5-5b1f-49ec-a795-79f6eb63228b Bridge br-pif Port br-pif Interface br-pif type: internal Bridge br-int Port br-int Interface br-int type: internal .. note:: There's a known bug that the ports added to OVSDB via ovs-vsctl don't get to the kernel datapath immediately, ie. they don't show up in the output of ``ovs-dpctl show`` even though they show up in output of ``ovs-vsctl show``. In order to workaround this issue, restart ovs-vswitchd. (You can terminate ovs-vswitchd by running ``ovs-appctl exit``.) Add physicals NICs (PIF) ~~~~~~~~~~~~~~~~~~~~~~~~ Now, let's add the physical NIC and the internal port to ``br-pif``. In OVS for Hyper-V, we use the name of the adapter on top of which the Hyper-V virtual switch was created, as a special name to refer to the physical NICs connected to the Hyper-V switch, e.g. if we created the Hyper-V virtual switch on top of the adapter named ``Ethernet0``, then in OVS we use that name (``Ethernet0``) as a special name to refer to that adapter. .. note:: We assume that the OVS extension is enabled Hyper-V switch. Internal ports are the virtual adapters created on the Hyper-V switch using the ``ovs-vsctl add-br `` command. By default they are created under the following rule "" and the adapters are disabled. One needs to enable them and set the corresponding values to it to make them IP-able. As a whole example, if we issue the following in a powershell console: :: PS > Get-NetAdapter | select Name,InterfaceDescription Name InterfaceDescription ---- -------------------- Ethernet1 Intel(R) PRO/1000 MT Network Connection br-pif Hyper-V Virtual Ethernet Adapter #2 Ethernet0 Intel(R) PRO/1000 MT Network Connection #2 br-int Hyper-V Virtual Ethernet Adapter #3 PS > Get-VMSwitch Name SwitchType NetAdapterInterfaceDescription ---- ---------- ------------------------------ external External Intel(R) PRO/1000 MT Network Connection #2 We can see that we have a switch(external) created upon adapter name 'Ethernet0' with the internal ports under name 'br-pif' and 'br-int'. Thus resulting into the following ovs-vsctl commands: :: > ovs-vsctl add-port br-pif Ethernet0 Dumping the ports should show the additional ports that were just added: :: > ovs-dpctl show system@ovs-system: lookups: hit:0 missed:0 lost:0 flows: 0 port 2: br-pif (internal) <<< internal port adapter on Hyper-V switch port 1: br-int (internal) <<< internal port adapter on Hyper-V switch port 3: Ethernet0 <<< Physical NIC > ovs-vsctl show a56ec7b5-5b1f-49ec-a795-79f6eb63228b Bridge br-pif Port br-pif Interface br-pif type: internal Port "Ethernet0" Interface "Ethernet0" Bridge br-int Port br-int Interface br-int type: internal Add virtual interfaces (VIFs) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Adding VIFs to openvswitch is a two step procedure. The first step is to assign a 'OVS port name' which is a unique name across all VIFs on this Hyper-V. The next step is to add the VIF to the ovsdb using its 'OVS port name' as key. First, assign a unique 'OVS port name' to the VIF. The VIF needs to have been disconnected from the Hyper-V switch before assigning a 'OVS port name' to it. In the example below, we assign a 'OVS port name' called ``ovs-port-a`` to a VIF on a VM ``VM1``. By using index 0 for ``$vnic``, the first VIF of the VM is being addressed. After assigning the name ``ovs-port-a``, the VIF is connected back to the Hyper-V switch with name ``OVS-HV-Switch``, which is assumed to be the Hyper-V switch with OVS extension enabled.: :: PS > import-module .\datapath-windows\misc\OVS.psm1 PS > $vnic = Get-VMNetworkAdapter PS > Disconnect-VMNetworkAdapter -VMNetworkAdapter $vnic[0] PS > $vnic[0] | Set-VMNetworkAdapterOVSPort -OVSPortName ovs-port-a PS > Connect-VMNetworkAdapter -VMNetworkAdapter $vnic[0] \ -SwitchName OVS-Extended-Switch Next, add the VIFs to ``br-int``: :: > ovs-vsctl add-port br-int ovs-port-a Dumping the ports should show the additional ports that were just added: :: > ovs-dpctl show system@ovs-system: lookups: hit:0 missed:0 lost:0 flows: 0 port 4: ovs-port-a port 2: br-pif (internal) port 1: br-int (internal port 3: Ethernet0 > ovs-vsctl show 4cd86499-74df-48bd-a64d-8d115b12a9f2 Bridge br-pif Port "vEthernet (external)" Interface "vEthernet (external)" Port "Ethernet0" Interface "Ethernet0" Port br-pif Interface br-pif type: internal Bridge br-int Port br-int Interface br-int type: internal Port "ovs-port-a" Interface "ovs-port-a" Add multiple NICs to be managed by OVS ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ To leverage support of multiple NICs into OVS, we will be using the MSFT cmdlets for forwarding team extension. More documentation about them can be found at technet_. .. _technet: https://technet.microsoft.com/en-us/library/jj553812%28v=wps.630%29.aspx For example, to set up a switch team combined from ``Ethernet0 2`` and ``Ethernet1 2`` named ``external``: :: PS > Get-NetAdapter Name InterfaceDescription ---- -------------------- br-int Hyper-V Virtual Ethernet Adapter #3 br-pif Hyper-V Virtual Ethernet Adapter #2 Ethernet3 2 Intel(R) 82574L Gigabit Network Co...#3 Ethernet2 2 Intel(R) 82574L Gigabit Network Co...#4 Ethernet1 2 Intel(R) 82574L Gigabit Network Co...#2 Ethernet0 2 Intel(R) 82574L Gigabit Network Conn... PS > New-NetSwitchTeam -Name external -TeamMembers "Ethernet0 2","Ethernet1 2" PS > Get-NetSwitchTeam Name : external Members : {Ethernet1 2, Ethernet0 2} This will result in a new adapter bound to the host called ``external``: :: PS > Get-NetAdapter Name InterfaceDescription ---- -------------------- br-test Hyper-V Virtual Ethernet Adapter #4 br-pif Hyper-V Virtual Ethernet Adapter #2 external Microsoft Network Adapter Multiplexo... Ethernet3 2 Intel(R) 82574L Gigabit Network Co...#3 Ethernet2 2 Intel(R) 82574L Gigabit Network Co...#4 Ethernet1 2 Intel(R) 82574L Gigabit Network Co...#2 Ethernet0 2 Intel(R) 82574L Gigabit Network Conn... Next we will set up the Hyper-V VMSwitch on the new adapter ``external``: :: PS > New-VMSwitch -Name external -NetAdapterName external \ -AllowManagementOS $false Under OVS the adapters under the team ``external``, ``Ethernet0 2`` and ``Ethernet1 2``, can be added either under a bond device or separately. The following example shows how the bridges look with the NICs being separated: :: > ovs-vsctl show 6cd9481b-c249-4ee3-8692-97b399dd29d8 Bridge br-test Port br-test Interface br-test type: internal Port "Ethernet1 2" Interface "Ethernet1 2" Bridge br-pif Port "Ethernet0 2" Interface "Ethernet0 2" Port br-pif Interface br-pif type: internal Add patch ports and configure VLAN tagging ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The Windows Open vSwitch implementation support VLAN tagging in the switch. Switch VLAN tagging along with patch ports between ``br-int`` and ``br-pif`` is used to configure VLAN tagging functionality between two VMs on different Hyper-Vs. To start, add a patch port from ``br-int`` to ``br-pif``: :: > ovs-vsctl add-port br-int patch-to-pif > ovs-vsctl set interface patch-to-pif type=patch \ options:peer=patch-to-int Add a patch port from ``br-pif`` to ``br-int``: :: > ovs-vsctl add-port br-pif patch-to-int > ovs-vsctl set interface patch-to-int type=patch \ options:peer=patch-to-pif Re-Add the VIF ports with the VLAN tag: :: > ovs-vsctl add-port br-int ovs-port-a tag=900 > ovs-vsctl add-port br-int ovs-port-b tag=900 Add tunnels ~~~~~~~~~~~ The Windows Open vSwitch implementation support VXLAN and STT tunnels. To add tunnels. For example, first add the tunnel port between 172.168.201.101 <-> 172.168.201.102: :: > ovs-vsctl add-port br-int tun-1 > ovs-vsctl set Interface tun-1 type= > ovs-vsctl set Interface tun-1 options:local_ip=172.168.201.101 > ovs-vsctl set Interface tun-1 options:remote_ip=172.168.201.102 > ovs-vsctl set Interface tun-1 options:in_key=flow > ovs-vsctl set Interface tun-1 options:out_key=flow ...and the tunnel port between 172.168.201.101 <-> 172.168.201.105: :: > ovs-vsctl add-port br-int tun-2 > ovs-vsctl set Interface tun-2 type= > ovs-vsctl set Interface tun-2 options:local_ip=172.168.201.102 > ovs-vsctl set Interface tun-2 options:remote_ip=172.168.201.105 > ovs-vsctl set Interface tun-2 options:in_key=flow > ovs-vsctl set Interface tun-2 options:out_key=flow Where ```` is one of: ``stt`` or ``vxlan`` .. note:: Any patch ports created between br-int and br-pif MUST be be deleted prior to adding tunnels. Windows Services ---------------- Open vSwitch daemons come with support to run as a Windows service. The instructions here assume that you have installed the Open vSwitch utilities and daemons via ``make install``. To start, create the database: :: > ovsdb-tool create C:/openvswitch/etc/openvswitch/conf.db \ "C:/openvswitch/usr/share/openvswitch/vswitch.ovsschema" Create the ovsdb-server service and start it: :: > sc create ovsdb-server \ binpath="C:/openvswitch/usr/sbin/ovsdb-server.exe \ C:/openvswitch/etc/openvswitch/conf.db \ -vfile:info --log-file --pidfile \ --remote=punix:db.sock --service --service-monitor" > sc start ovsdb-server .. tip:: One of the common issues with creating a Windows service is with mungled paths. You can make sure that the correct path has been registered with the Windows services manager by running: :: > sc qc ovsdb-server Check that the service is healthy by running: :: > sc query ovsdb-server Initialize the database: :: > ovs-vsctl --no-wait init Create the ovs-vswitchd service and start it: :: > sc create ovs-vswitchd \ binpath="C:/openvswitch/usr/sbin/ovs-vswitchd.exe \ --pidfile -vfile:info --log-file --service --service-monitor" > sc start ovs-vswitchd Check that the service is healthy by running: :: > sc query ovs-vswitchd To stop and delete the services, run: :: > sc stop ovs-vswitchd > sc stop ovsdb-server > sc delete ovs-vswitchd > sc delete ovsdb-server Windows CI Service ------------------ `AppVeyor `__ provides a free Windows autobuild service for opensource projects. Open vSwitch has integration with AppVeyor for continuous build. A developer can build test his changes for Windows by logging into appveyor.com using a github account, creating a new project by linking it to his development repository in github and triggering a new build. TODO ---- * Investigate the working of sFlow on Windows and re-enable the unit tests. * Investigate and add the feature to provide QoS. * Sign the driver & create an MSI for installing the different OpenvSwitch components on Windows. ovs-2.9.0/Documentation/intro/install/xenserver.rst000066400000000000000000000217101324262074100225200ustar00rootroot00000000000000.. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. Convention for heading levels in Open vSwitch documentation: ======= Heading 0 (reserved for the title in a document) ------- Heading 1 ~~~~~~~ Heading 2 +++++++ Heading 3 ''''''' Heading 4 Avoid deeper levels because they do not render well. ================================ Open vSwitch on Citrix XenServer ================================ This document describes how to build and install Open vSwitch on a Citrix XenServer host. If you want to install Open vSwitch on a generic Linux or BSD host, refer to :doc:`general` instead. Open vSwitch should work with XenServer 5.6.100 and later. However, Open vSwitch requires Python 2.7 or later, so using Open vSwitch with XenServer 6.5 or earlier requires installing Python 2.7. Building -------- You may build from an Open vSwitch distribution tarball or from an Open vSwitch Git tree. The recommended build environment to build RPMs for Citrix XenServer is the DDK VM available from Citrix. 1. If you are building from an Open vSwitch Git tree, then you will need to first create a distribution tarball by running:: $ ./boot.sh $ ./configure $ make dist You cannot run this in the DDK VM, because it lacks tools that are necessary to bootstrap the Open vSwitch distribution. Instead, you must run this on a machine that has the tools listed in :ref:`general-install-reqs` as prerequisites for building from a Git tree. 2. Copy the distribution tarball into ``/usr/src/redhat/SOURCES`` inside the DDK VM. 3. In the DDK VM, unpack the distribution tarball into a temporary directory and "cd" into the root of the distribution tarball. 4. To build Open vSwitch userspace, run:: $ rpmbuild -bb xenserver/openvswitch-xen.spec This produces three RPMs in ``/usr/src/redhat/RPMS/i386``: - ``openvswitch`` - ``openvswitch-modules-xen`` - ``openvswitch-debuginfo`` The above command automatically runs the Open vSwitch unit tests. To disable the unit tests, run:: $ rpmbuild -bb --without check xenserver/openvswitch-xen.spec Build Parameters ---------------- ``openvswitch-xen.spec`` needs to know a number of pieces of information about the XenServer kernel. Usually, it can figure these out for itself, but if it does not do it correctly then you can specify them yourself as parameters to the build. Thus, the final ``rpmbuild`` step above can be elaborated as:: $ VERSION= $ KERNEL_NAME= $ KERNEL_VERSION= $ KERNEL_FLAVOR= $ rpmbuild \ -D "openvswitch_version $VERSION" \ -D "kernel_name $KERNEL_NAME" \ -D "kernel_version $KERNEL_VERSION" \ -D "kernel_flavor $KERNEL_FLAVOR" \ -bb xenserver/openvswitch-xen.spec where: ```` is the version number that appears in the name of the Open vSwitch tarball, e.g. 0.90.0. ```` is the name of the XenServer kernel package, e.g. ``kernel-xen`` or ``kernel-NAME-xen``, without the ``kernel-`` prefix. ```` is the output of:: $ rpm -q --queryformat "%{Version}-%{Release}" , e.g. ``2.6.32.12-0.7.1.xs5.6.100.323.170596``, where ```` is the name of the ``-devel`` package corresponding to ````. ```` is either ``xen`` or ``kdump``, where ``xen`` flavor is the main running kernel flavor and the ``kdump`` flavor is the crashdump kernel flavor. Commonly, one would specify ``xen`` here. For XenServer 6.5 or above, the kernel version naming no longer contains KERNEL_FLAVOR. In fact, only providing the ``uname -r`` output is enough. So, the final ``rpmbuild`` step changes to:: $ KERNEL_UNAME=<`uname -r` output> $ rpmbuild \ -D "kenel_uname $KERNEL_UNAME" \ -bb xenserver/openvswitch-xen.spec Installing Open vSwitch for XenServer ------------------------------------- To install Open vSwitch on a XenServer host, or to upgrade to a newer version, copy the ``openvswitch`` and ``openvswitch-modules-xen`` RPMs to that host with ``scp``, then install them with ``rpm -U``, e.g.:: $ scp openvswitch-$VERSION-1.i386.rpm \ openvswitch-modules-xen-$XEN_KERNEL_VERSION-$VERSION-1.i386.rpm \ root@: # Enter 's root password. $ ssh root@ # Enter 's root password again. $ rpm -U openvswitch-$VERSION-1.i386.rpm \ openvswitch-modules-xen-$XEN_KERNEL_VERSION-$VERSION-1.i386.rpm To uninstall Open vSwitch from a XenServer host, remove the packages:: $ ssh root@ # Enter 's root password again. $ rpm -e openvswitch openvswitch-modules-xen-$XEN_KERNEL_VERSION After installing or uninstalling Open vSwitch, the XenServer should be rebooted as soon as possible. Open vSwitch Boot Sequence on XenServer --------------------------------------- When Open vSwitch is installed on XenServer, its startup script ``/etc/init.d/openvswitch`` runs early in boot. It does roughly the following: * Loads the OVS kernel module, openvswitch. * Starts ovsdb-server, the OVS configuration database. * XenServer expects there to be no bridges configured at startup, but the OVS configuration database likely still has bridges configured from before reboot. To match XenServer expectations, the startup script deletes all configured bridges from the database. * Starts ovs-vswitchd, the OVS switching daemon. At this point in the boot process, then, there are no Open vSwitch bridges, even though all of the Open vSwitch daemons are running. Later on in boot, ``/etc/init.d/management-interface`` (part of XenServer, not Open vSwitch) creates the bridge for the XAPI management interface by invoking ``/opt/xensource/libexec/interface-reconfigure``. Normally this program consults XAPI's database to obtain information about how to configure the bridge, but XAPI is not running yet(\*) so it instead consults ``/var/xapi/network.dbcache``, which is a cached copy of the most recent network configuration. (\*) Even if XAPI were running, if this XenServer node is a pool slave then the query would have to consult the master, which requires network access, which begs the question of how to configure the management interface. XAPI starts later on in the boot process. XAPI can then create other bridges on demand using ``/opt/xensource/libexec/interface-reconfigure``. Now that XAPI is running, that program consults XAPI directly instead of reading the cache. As part of its own startup, XAPI invokes the Open vSwitch XAPI plugin script ``/etc/xapi.d/openvswitch-cfg-update`` passing the ``update`` command. The plugin script does roughly the following: * Calls ``/opt/xensource/libexec/interface-reconfigure`` with the ``rewrite`` command, to ensure that the network cache is up-to-date. * Queries the Open vSwitch manager setting (named ``vswitch_controller``) from the XAPI database for the XenServer pool. * If XAPI and OVS are configured for different managers, or if OVS is configured for a manager but XAPI is not, runs ``ovs-vsctl emer-reset`` to bring the Open vSwitch configuration to a known state. One effect of emer-reset is to deconfigure any manager from the OVS database. * If XAPI is configured for a manager, configures the OVS manager to match with ``ovs-vsctl set-manager``. Notes ----- * The Open vSwitch boot sequence only configures an OVS configuration database manager. There is no way to directly configure an OpenFlow controller on XenServer and, as a consequence of the step above that deletes all of the bridges at boot time, controller configuration only persists until XenServer reboot. The configuration database manager can, however, configure controllers for bridges. See the BUGS section of ovs-testcontroller(8) for more information on this topic. * The Open vSwitch startup script automatically adds a firewall rule to allow GRE traffic. This rule is needed for the XenServer feature called "Cross-Host Internal Networks" (CHIN) that uses GRE. If a user configures tunnels other than GRE (ex: Geneve, VXLAN, LISP), they will have to either manually add a iptables firewall rule to allow the tunnel traffic or add it through a startup script (Please refer to the "enable-protocol" command in the ovs-ctl(8) manpage). Reporting Bugs -------------- Please report problems to bugs@openvswitch.org. ovs-2.9.0/Documentation/intro/what-is-ovs.rst000066400000000000000000000023321324262074100212110ustar00rootroot00000000000000.. Copyright (c) 2016, Stephen Finucane Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. Convention for heading levels in Open vSwitch documentation: ======= Heading 0 (reserved for the title in a document) ------- Heading 1 ~~~~~~~ Heading 2 +++++++ Heading 3 ''''''' Heading 4 Avoid deeper levels because they do not render well. ===================== What Is Open vSwitch? ===================== .. image:: ../_static/overview.png :align: center Overview -------- .. NOTE(stephenfin): The below line numbers may need to be updated if the README is modified .. include:: ../../README.rst :start-line: 13 :end-line: 71 ovs-2.9.0/Documentation/intro/why-ovs.rst000066400000000000000000000144101324262074100204440ustar00rootroot00000000000000.. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. Convention for heading levels in Open vSwitch documentation: ======= Heading 0 (reserved for the title in a document) ------- Heading 1 ~~~~~~~ Heading 2 +++++++ Heading 3 ''''''' Heading 4 Avoid deeper levels because they do not render well. ================= Why Open vSwitch? ================= Hypervisors need the ability to bridge traffic between VMs and with the outside world. On Linux-based hypervisors, this used to mean using the built-in L2 switch (the Linux bridge), which is fast and reliable. So, it is reasonable to ask why Open vSwitch is used. The answer is that Open vSwitch is targeted at multi-server virtualization deployments, a landscape for which the previous stack is not well suited. These environments are often characterized by highly dynamic end-points, the maintenance of logical abstractions, and (sometimes) integration with or offloading to special purpose switching hardware. The following characteristics and design considerations help Open vSwitch cope with the above requirements. The mobility of state --------------------- All network state associated with a network entity (say a virtual machine) should be easily identifiable and migratable between different hosts. This may include traditional "soft state" (such as an entry in an L2 learning table), L3 forwarding state, policy routing state, ACLs, QoS policy, monitoring configuration (e.g. NetFlow, IPFIX, sFlow), etc. Open vSwitch has support for both configuring and migrating both slow (configuration) and fast network state between instances. For example, if a VM migrates between end-hosts, it is possible to not only migrate associated configuration (SPAN rules, ACLs, QoS) but any live network state (including, for example, existing state which may be difficult to reconstruct). Further, Open vSwitch state is typed and backed by a real data-model allowing for the development of structured automation systems. Responding to network dynamics ------------------------------ Virtual environments are often characterized by high-rates of change. VMs coming and going, VMs moving backwards and forwards in time, changes to the logical network environments, and so forth. Open vSwitch supports a number of features that allow a network control system to respond and adapt as the environment changes. This includes simple accounting and visibility support such as NetFlow, IPFIX, and sFlow. But perhaps more useful, Open vSwitch supports a network state database (OVSDB) that supports remote triggers. Therefore, a piece of orchestration software can "watch" various aspects of the network and respond if/when they change. This is used heavily today, for example, to respond to and track VM migrations. Open vSwitch also supports OpenFlow as a method of exporting remote access to control traffic. There are a number of uses for this including global network discovery through inspection of discovery or link-state traffic (e.g. LLDP, CDP, OSPF, etc.). Maintenance of logical tags ---------------------------- Distributed virtual switches (such as VMware vDS and Cisco's Nexus 1000V) often maintain logical context within the network through appending or manipulating tags in network packets. This can be used to uniquely identify a VM (in a manner resistant to hardware spoofing), or to hold some other context that is only relevant in the logical domain. Much of the problem of building a distributed virtual switch is to efficiently and correctly manage these tags. Open vSwitch includes multiple methods for specifying and maintaining tagging rules, all of which are accessible to a remote process for orchestration. Further, in many cases these tagging rules are stored in an optimized form so they don't have to be coupled with a heavyweight network device. This allows, for example, thousands of tagging or address remapping rules to be configured, changed, and migrated. In a similar vein, Open vSwitch supports a GRE implementation that can handle thousands of simultaneous GRE tunnels and supports remote configuration for tunnel creation, configuration, and tear-down. This, for example, can be used to connect private VM networks in different data centers. Hardware integration -------------------- Open vSwitch's forwarding path (the in-kernel datapath) is designed to be amenable to "offloading" packet processing to hardware chipsets, whether housed in a classic hardware switch chassis or in an end-host NIC. This allows for the Open vSwitch control path to be able to both control a pure software implementation or a hardware switch. There are many ongoing efforts to port Open vSwitch to hardware chipsets. These include multiple merchant silicon chipsets (Broadcom and Marvell), as well as a number of vendor-specific platforms. The "Porting" section in the documentation discusses how one would go about making such a port. The advantage of hardware integration is not only performance within virtualized environments. If physical switches also expose the Open vSwitch control abstractions, both bare-metal and virtualized hosting environments can be managed using the same mechanism for automated network control. Summary ------- In many ways, Open vSwitch targets a different point in the design space than previous hypervisor networking stacks, focusing on the need for automated and dynamic network control in large-scale Linux-based virtualization environments. The goal with Open vSwitch is to keep the in-kernel code as small as possible (as is necessary for performance) and to re-use existing subsystems when applicable (for example Open vSwitch uses the existing QoS stack). As of Linux 3.3, Open vSwitch is included as a part of the kernel and packaging for the userspace utilities are available on most popular distributions. ovs-2.9.0/Documentation/ref/000077500000000000000000000000001324262074100157175ustar00rootroot00000000000000ovs-2.9.0/Documentation/ref/index.rst000066400000000000000000000246421324262074100175700ustar00rootroot00000000000000.. Copyright (c) 2016, Stephen Finucane Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. Convention for heading levels in Open vSwitch documentation: ======= Heading 0 (reserved for the title in a document) ------- Heading 1 ~~~~~~~ Heading 2 +++++++ Heading 3 ''''''' Heading 4 Avoid deeper levels because they do not render well. =============== Reference Guide =============== Man Pages --------- .. TODO(stephenfin): Remove the below notice once everything is converted to rST The following man pages are written in rST and converted to roff at compile time: .. toctree:: :maxdepth: 3 ovs-test.8 ovs-vlan-test.8 ovsdb-server.7 ovsdb.5 ovsdb.7 The remainder are still in roff format can be found below: .. list-table:: * - ovn-architecture(7) - `(pdf) `__ - `(html) `__ - `(plain text) `__ * - ovn-controller(8) - `(pdf) `__ - `(html) `__ - `(plain text) `__ * - ovn-controller-vtep(8) - `(pdf) `__ - `(html) `__ - `(plain text) `__ * - ovn-ctl(8) - `(pdf) `__ - `(html) `__ - `(plain text) `__ * - ovn-nb(5) - `(pdf) `__ - `(html) `__ - `(plain text) `__ * - ovn-nbctl(8) - `(pdf) `__ - `(html) `__ - `(plain text) `__ * - ovn-northd(8) - `(pdf) `__ - `(html) `__ - `(plain text) `__ * - ovn-sb(5) - `(pdf) `__ - `(html) `__ - `(plain text) `__ * - ovn-sbctl(8) - `(pdf) `__ - `(html) `__ - `(plain text) `__ * - ovn-trace(8) - `(pdf) `__ - `(html) `__ - `(plain text) `__ * - ovs-appctl(8) - `(pdf) `__ - `(html) `__ - `(plain text) `__ * - ovs-bugtool(8) - `(pdf) `__ - `(html) `__ - `(plain text) `__ * - ovs-ctl(8) - `(pdf) `__ - `(html) `__ - `(plain text) `__ * - ovsdb-client(1) - `(pdf) `__ - `(html) `__ - `(plain text) `__ * - ovsdb-server(1) - `(pdf) `__ - `(html) `__ - `(plain text) `__ * - ovsdb-tool(1) - `(pdf) `__ - `(html) `__ - `(plain text) `__ * - ovs-dpctl(8) - `(pdf) `__ - `(html) `__ - `(plain text) `__ * - ovs-dpctl-top(8) - `(pdf) `__ - `(html) `__ - `(plain text) `__ * - ovs-fields(7) - `(pdf) `__ - `(html) `__ - `(plain text) `__ * - ovs-l3ping(8) - `(pdf) `__ - `(html) `__ - `(plain text) `__ * - ovs-ofctl(8) - `(pdf) `__ - `(html) `__ - `(plain text) `__ * - ovs-parse-backtrace(8) - `(pdf) `__ - `(html) `__ - `(plain text) `__ * - ovs-pcap(1) - `(pdf) `__ - `(html) `__ - `(plain text) `__ * - ovs-pki(8) - `(pdf) `__ - `(html) `__ - `(plain text) `__ * - ovs-tcpdump(8) - `(pdf) `__ - `(html) `__ - `(plain text) `__ * - ovs-tcpundump(1) - `(pdf) `__ - `(html) `__ - `(plain text) `__ * - ovs-test(8) - `(pdf) `__ - `(html) `__ - `(plain text) `__ * - ovs-testcontroller(8) - `(pdf) `__ - `(html) `__ - `(plain text) `__ * - ovs-vlan-bug-workaround(8) - `(pdf) `__ - `(html) `__ - `(plain text) `__ * - ovs-vlan-test(8) - `(pdf) `__ - `(html) `__ - `(plain text) `__ * - ovs-vsctl(8) - `(pdf) `__ - `(html) `__ - `(plain text) `__ * - ovs-vswitchd(8) - `(pdf) `__ - `(html) `__ - `(plain text) `__ * - ovs-vswitchd.conf.db(5) - `(pdf) `__ - `(html) `__ - `(plain text) `__ * - vtep(5) - `(pdf) `__ - `(html) `__ - `(plain text) `__ * - vtep-ctl(8) - `(pdf) `__ - `(html) `__ - `(plain text) `__ ovs-2.9.0/Documentation/ref/ovs-test.8.rst000066400000000000000000000147761324262074100204220ustar00rootroot00000000000000======== ovs-test ======== Synopsis ======== **ovs-test** -s *port* **ovs-test** -c *server1* *server2* [**-b** *targetbandwidth*] [**-i** *testinterval*] [**-d**] [**-l** *vlantag*] [**-t** *tunnelmodes*] Description =========== The :program:`ovs-test` program may be used to check for problems sending 802.1Q or GRE traffic that Open vSwitch may uncover. These problems, for example, can occur when Open vSwitch is used to send 802.1Q traffic through physical interfaces running certain drivers of certain Linux kernel versions. To run a test, configure IP addresses on `server1` and `server2` for interfaces you intended to test. These interfaces could also be already configured OVS bridges that have a physical interface attached to them. Then, on one of the nodes, run :program:`ovs-test` in server mode and on the other node run it in client mode. The client will connect to :program:`ovs-test` server and schedule tests between both of them. The :program:`ovs-test` client will perform UDP and TCP tests. UDP tests can report packet loss and achieved bandwidth for various datagram sizes. By default target bandwidth for UDP tests is 1Mbit/s. TCP tests report only achieved bandwidth, because kernel TCP stack takes care of flow control and packet loss. TCP tests are essential to detect potential TSO related issues. To determine whether Open vSwitch is encountering any problems, the user must compare packet loss and achieved bandwidth in a setup where traffic is being directly sent and in one where it is not. If in the 802.1Q or L3 tunneled tests both :program:`ovs-test` processes are unable to communicate or the achieved bandwidth is much lower compared to direct setup, then, most likely, Open vSwitch has encountered a pre-existing kernel or driver bug. Some examples of the types of problems that may be encountered are: - When NICs use VLAN stripping on receive they must pass a pointer to a `vlan_group` when reporting the stripped tag to the networking core. If no `vlan_group` is in use then some drivers just drop the extracted tag. Drivers are supposed to only enable stripping if a `vlan_group` is registered but not all of them do that. - On receive, some drivers handle priority tagged packets specially and don't pass the tag onto the network stack at all, so Open vSwitch never has a chance to see it. - Some drivers size their receive buffers based on whether a `vlan_group` is enabled, meaning that a maximum size packet with a VLAN tag will not fit if no `vlan_group` is configured. - On transmit, some drivers expect that VLAN acceleration will be used if it is available, which can only be done if a `vlan_group` is configured. In these cases, the driver may fail to parse the packet and correctly setup checksum offloading or TSO. Client Mode An :program:`ovs-test` client will connect to two :program:`ovs-test` servers and will ask them to exchange test traffic. It is also possible to spawn an :program:`ovs-test` server automatically from the client. Server Mode To conduct tests, two :program:`ovs-test` servers must be running on two different hosts where the client can connect. The actual test traffic is exchanged only between both :program:`ovs-test` servers. It is recommended that both servers have their IP addresses in the same subnet, otherwise one would have to make sure that routing is set up correctly. Options ======= .. program:: ovs-test .. option:: -s , --server Run in server mode and wait for the client to establish XML RPC Control Connection on this TCP port. It is recommended to have `ethtool(8)` installed on the server so that it could retrieve information about the NIC driver. .. option:: -c , --client Run in client mode and schedule tests between `server1` and `server2`, where each server must be given in the following format:: OuterIP[:OuterPort],InnerIP[/Mask][:InnerPort]. The `OuterIP` must be already assigned to the physical interface which is going to be tested. This is the IP address where client will try to establish XML RPC connection. If `OuterIP` is 127.0.0.1 then client will automatically spawn a local instance of :program:`ovs-test` server. OuterPort is TCP port where server is listening for incoming XML/RPC control connections to schedule tests (by default it is 15531). The :program:`ovs-test` will automatically assign `InnerIP[/Mask]` to the interfaces that will be created on the fly for testing purposes. It is important that `InnerIP[/Mask]` does not interfere with already existing IP addresses on both :program:`ovs-test` servers and client. InnerPort is port which will be used by server to listen for test traffic that will be encapsulated (by default it is 15532). .. option:: -b , --bandwidth Target bandwidth for UDP tests. The targetbandwidth must be given in bits per second. It is possible to use postfix `M` or `K` to alter the target bandwidth magnitude. .. option:: -i , --interval How long each test should run. By default 5 seconds. .. option:: -h, --help Prints a brief help message to the console. .. option:: -V, --version Prints version information to the console. The following test modes are supported by :program:`ovs-test`. It is possible to combine multiple of them in a single :program:`ovs-test` invocation. .. option:: -d, --direct Perform direct tests between both OuterIP addresses. These tests could be used as a reference to compare 802.1Q or L3 tunneling test results. .. option:: -l , --vlan-tag Perform 802.1Q tests between both servers. These tests will create a temporary OVS bridge, if necessary, and attach a VLAN tagged port to it for testing purposes. .. option:: -t , --tunnel-modes Perform L3 tunneling tests. The given argument is a comma sepa‐ rated string that specifies all the L3 tunnel modes that should be tested (e.g. gre). The L3 tunnels are terminated on interface that has the OuterIP address assigned. Examples ======== On host 1.2.3.4 start :program:`ovs-test` in server mode:: ovs-test -s 15531 On host 1.2.3.5 start :program:`ovs-test` in client mode and do direct, VLAN and GRE tests between both nodes:: ovs-test -c 127.0.0.1,1.1.1.1/30 1.2.3.4,1.1.1.2/30 -d -l 123 -t gre See Also ======== `ovs-vswitchd(8)`, `ovs-ofctl(8)`, `ovs-vsctl(8)`, :program:`ovs-vlan-test`, `ethtool(8)`, `uname(1)` ovs-2.9.0/Documentation/ref/ovs-vlan-test.8.rst000066400000000000000000000101251324262074100213400ustar00rootroot00000000000000============= ovs-vlan-test ============= Synopsis ======== **ovs-vlan-test** [**-s** | **--server**] *control_ip* *vlan_ip* Description =========== The :program:`ovs-vlan-test` utility has some limitations, for example, it does not use TCP in its tests. Also it does not take into account MTU to detect potential edge cases. To overcome those limitations a new tool was developed - :program:`ovs-test`. :program:`ovs-test` is currently supported only on Debian so, if possible, try to use that on instead of :program:`ovs-vlan-test`. The :program:`ovs-vlan-test` program may be used to check for problems sending 802.1Q traffic which may occur when running Open vSwitch. These problems can occur when Open vSwitch is used to send 802.1Q traffic through physical interfaces running certain drivers of certain Linux kernel versions. To run a test, configure Open vSwitch to tag traffic originating from `vlan_ip` and forward it out the target interface. Then run the :program:`ovs-vlan-test` in client mode connecting to an :program:`ovs-vlan-test` server. :program:`ovs-vlan-test` will display "OK" if it did not detect problems. Some examples of the types of problems that may be encountered are: - When NICs use VLAN stripping on receive they must pass a pointer to a `vlan_group` when reporting the stripped tag to the networking core. If no `vlan_group` is in use then some drivers just drop the extracted tag. Drivers are supposed to only enable stripping if a `vlan_group` is registered but not all of them do that. - On receive, some drivers handle priority tagged packets specially and don't pass the tag onto the network stack at all, so Open vSwitch never has a chance to see it. - Some drivers size their receive buffers based on whether a `vlan_group` is enabled, meaning that a maximum size packet with a VLAN tag will not fit if no `vlan_group` is configured. - On transmit, some drivers expect that VLAN acceleration will be used if it is available, which can only be done if a `vlan_group` is configured. In these cases, the driver may fail to parse the packet and correctly setup checksum offloading or TSO. Client Mode An :program:`ovs-vlan-test` client may be run on a host to check for VLAN connectivity problems. The client must be able to establish HTTP connections with an :program:`ovs-vlan-test` server located at the specified `control_ip` address. UDP traffic sourced at `vlan_ip` should be tagged and directed out the interface whose connectivity is being tested. Server Mode To conduct tests, an :program:`ovs-vlan-test` server must be running on a host known not to have VLAN connectivity problems. The server must have a `control_ip` on a non-VLAN network which clients can establish connectivity with. It must also have a `vlan_ip` address on a VLAN network which clients will use to test their VLAN connectivity. Multiple clients may test against a single :program:`ovs-vlan-test` server concurrently. Options ======= .. program:: ovs-vlan-test .. option:: -s, --server Run in server mode. .. option:: -h, --help Prints a brief help message to the console. .. option:: -V, --version Prints version information to the console. Examples ======== Display the Linux kernel version and driver of `eth1`:: uname -r ethtool -i eth1 Set up a bridge which forwards traffic originating from `1.2.3.4` out `eth1` with VLAN tag 10:: ovs-vsctl -- add-br vlan-br \ -- add-port vlan-br eth1 \ -- add-port vlan-br vlan-br-tag tag=10 \ -- set Interface vlan-br-tag type=internal ip addr add 1.2.3.4/8 dev vlan-br-tag ip link set vlan-br-tag up Run an :program:`ovs-vlan-test` server listening for client control traffic on `172.16.0.142` port `8080` and VLAN traffic on the default port of `1.2.3.3`:: ovs-vlan-test -s 172.16.0.142:8080 1.2.3.3 Run an :program:`ovs-vlan-test` client with a control server located at `172.16.0.142` port `8080` and a local VLAN IP of `1.2.3.4`:: ovs-vlan-test 172.16.0.142:8080 1.2.3.4 See Also ======== `ovs-vswitchd(8)`, `ovs-ofctl(8)`, `ovs-vsctl(8)`, :program:`ovs-test`, `ethtool(8)`, `uname(1)` ovs-2.9.0/Documentation/ref/ovsdb-server.7.rst000066400000000000000000000363661324262074100212550ustar00rootroot00000000000000.. Copyright (c) 2017 Nicira, Inc. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. Convention for heading levels in Open vSwitch documentation: ======= Heading 0 (reserved for the title in a document) ------- Heading 1 ~~~~~~~ Heading 2 +++++++ Heading 3 ''''''' Heading 4 Avoid deeper levels because they do not render well. ============ ovsdb-server ============ Description =========== ``ovsdb-server`` implements the Open vSwitch Database (OVSDB) protocol specified in RFC 7047. This document provides clarifications for how ``ovsdb-server`` implements the protocol and describes the extensions that it provides beyond RFC 7047. Numbers in section headings refer to corresponding sections in RFC 7047. 3.1 JSON Usage -------------- RFC 4627 says that names within a JSON object should be unique. The Open vSwitch JSON parser discards all but the last value for a name that is specified more than once. The definition of allows for implementation extensions. Currently ``ovsdb-server`` uses the following additional ``error`` strings (which might change in later releases): ``syntax error`` or ``unknown column`` The request could not be parsed as an OVSDB request. An additional ``syntax`` member, whose value is a string that contains JSON, may narrow down the particular syntax that could not be parsed. ``internal error`` The request triggered a bug in ``ovsdb-server``. ``ovsdb error`` A map or set contains a duplicate key. ``permission error`` The request was denied by the role-based access control extension, introduced in version 2.8. 3.2 Schema Format ----------------- RFC 7047 requires the ``version`` field in . Current versions of ``ovsdb-server`` allow it to be omitted (future versions are likely to require it). RFC 7047 allows columns that contain weak references to be immutable. This raises the issue of the behavior of the weak reference when the rows that it references are deleted. Since version 2.6, ``ovsdb-server`` forces columns that contain weak references to be mutable. Since version 2.8, the table name ``RBAC_Role`` is used internally by the role-based access control extension to ``ovsdb-server`` and should not be used for purposes other than defining mappings of role names to table access permissions. This table has one row per role name and the following columns: ``name`` The role name. ``permissions`` A map of table name to a reference to a row in a separate permission table. The separate RBAC permission table has one row per access control configuration and the following columns: ``name`` The name of the table to which the row applies. ``authorization`` The set of column names and column:key pairs to be compared with the client ID in order to determine the authorization status of the requested operation. ``insert_delete`` A boolean value, true if authorized insertions and deletions are allowed, false if no insertions or deletions are allowed. ``update`` The set of columns and column:key pairs for which authorized update and mutate operations should be permitted. 4 Wire Protocol --------------- The original OVSDB specifications included the following reasons, omitted from RFC 7047, to operate JSON-RPC directly over a stream instead of over HTTP: * JSON-RPC is a peer-to-peer protocol, but HTTP is a client-server protocol, which is a poor match. Thus, JSON-RPC over HTTP requires the client to periodically poll the server to receive server requests. * HTTP is more complicated than stream connections and doesn't provide any corresponding advantage. * The JSON-RPC specification for HTTP transport is incomplete. 4.1.3 Transact -------------- Since version 2.8, role-based access controls can be applied to operations within a transaction that would modify the contents of the database (these operations include row insert, row delete, column update, and column mutate). Role-based access controls are applied when the database schema contains a table with the name ``RBAC_Role`` and the connection on which the transaction request was received has an associated role name (from the ``role`` column in the remote connection table). When role-based access controls are enabled, transactions that are otherwise well-formed may be rejected depending on the client's role, ID, and the contents of the ``RBAC_Role`` table and associated permissions table. 4.1.5 Monitor ------------- For backward compatibility, ``ovsdb-server`` currently permits a single to be used instead of an array; it is treated as a single-element array. Future versions of ``ovsdb-server`` might remove this compatibility feature. Because the parameter is used to match subsequent update notifications (see below) to the request, it must be unique among all active monitors. ``ovsdb-server`` rejects attempt to create two monitors with the same identifier. 4.1.12 Monitor_cond ------------------- A new monitor method added in Open vSwitch version 2.6. The ``monitor_cond`` request enables a client to replicate subsets of tables within an OVSDB database by requesting notifications of changes to rows matching one of the conditions specified in ``where`` by receiving the specified contents of these rows when table updates occur. ``monitor_cond`` also allows a more efficient update notifications by receiving notifications (described below). The ``monitor`` method described in Section 4.1.5 also applies to ``monitor_cond``, with the following exceptions: * RPC request method becomes ``monitor_cond``. * Reply result follows , described in Section 4.1.14. * Subsequent changes are sent to the client using the ``update2`` monitor notification, described in Section 4.1.14 * Update notifications are being sent only for rows matching [*]. The request object has the following members:: "method": "monitor_cond" "params": [, , ] "id": The parameter is used to match subsequent update notifications (see below) to this request. The object maps the name of the table to an array of . Each is an object with the following members:: "columns": [*] optional "where": [*] optional "select": optional The ``columns``, if present, define the columns within the table to be monitored that match conditions. If not present, all columns are monitored. The ``where``, if present, is a JSON array of and boolean values. If not present or condition is an empty array, implicit True will be considered and updates on all rows will be sent. is an object with the following members:: "initial": optional "insert": optional "delete": optional "modify": optional The contents of this object specify how the columns or table are to be monitored as explained in more detail below. The response object has the following members:: "result": "error": null "id": same "id" as request The object is described in detail in Section 4.1.14. It contains the contents of the tables for which initial rows are selected. If no tables initial contents are requested, then ``result`` is an empty object. Subsequently, when changes to a specified table that match one of the conditions in are committed, the changes are automatically sent to the client using the ``update2`` monitor notification (see Section 4.1.14). This monitoring persists until the JSON-RPC session terminates or until the client sends a ``monitor_cancel`` JSON-RPC request. Each specifies one or more conditions and the manner in which the rows that match the conditions are to be monitored. The circumstances in which an ``update`` notification is sent for a row within the table are determined by : * If ``initial`` is omitted or true, every row in the original table that matches one of the conditions is sent as part of the response to the ``monitor_cond`` request. * If ``insert`` is omitted or true, update notifications are sent for rows newly inserted into the table that match conditions or for rows modified in the table so that their old version does not match the condition and new version does. * If ``delete`` is omitted or true, update notifications are sent for rows deleted from the table that match conditions or for rows modified in the table so that their old version does match the conditions and new version does not. * If ``modify`` is omitted or true, update notifications are sent whenever a row in the table that matches conditions in both old and new version is modified. Both ``monitor`` and ``monitor_cond`` sessions can exist concurrently. However, ``monitor`` and ``monitor_cond`` shares the same parameter space; it must be unique among all ``monitor`` and ``monitor_cond`` sessions. 4.1.13 Monitor_cond_change -------------------------- The ``monitor_cond_change`` request enables a client to change an existing ``monitor_cond`` replication of the database by specifying a new condition and columns for each replicated table. Currently changing the columns set is not supported. The request object has the following members:: "method": "monitor_cond_change" "params": [, , ] "id": The parameter should have a value of an existing conditional monitoring session from this client. The second in params array is the requested value for this session. This value is valid only after ``monitor_cond_change`` is committed. A user can use these values to distinguish between update messages before conditions update and after. The object maps the name of the table to an array of . Monitored tables not included in retain their current conditions. Each is an object with the following members:: "columns": [*] optional "where": [*] optional The ``columns`` specify a new array of columns to be monitored, although this feature is not yet supported. The ``where`` specify a new array of conditions to be applied to this monitoring session. The response object has the following members:: "result": null "error": null "id": same "id" as request Subsequent notifications are described in detail in Section 4.1.14 in the RFC. If insert contents are requested by original monitor_cond request, will contain rows that match the new condition and do not match the old condition. If deleted contents are requested by origin monitor request, will contain any matched rows by old condition and not matched by the new condition. Changes according to the new conditions are automatically sent to the client using the ``update2`` monitor notification. An update, if any, as a result of a condition change, will be sent to the client before the reply to the ``monitor_cond_change`` request. 4.1.14 Update2 notification --------------------------- The ``update2`` notification is sent by the server to the client to report changes in tables that are being monitored following a ``monitor_cond`` request as described above. The notification has the following members:: "method": "update2" "params": [, ] "id": null The in ``params`` is the same as the value passed as the in ``params`` for the corresponding ``monitor`` request. is an object that maps from a table name to a . A is an object that maps from row's UUID to a object. A is an object with one of the following members: ``"initial": `` present for ``initial`` updates ``"insert": `` present for ``insert`` updates ``"delete": `` present for ``delete`` updates ``"modify": "`` present for ``modify`` updates The format of is described in Section 5.1. is always a null object for a ``delete`` update. In ``initial`` and ``insert`` updates, omits columns whose values equal the default value of the column type. For a ``modify`` update, contains only the columns that are modified. stores the difference between the old and new value for those columns, as described below. For columns with single value, the difference is the value of the new column. The difference between two sets are all elements that only belong to one of the sets. The difference between two maps are all key-value pairs whose keys appears in only one of the maps, plus the key-value pairs whose keys appear in both maps but with different values. For the latter elements, includes the value from the new column. Initial views of rows are not presented in update2 notifications, but in the response object to the ``monitor_cond`` request. The formatting of the object, however, is the same in either case. 4.1.15 Get Server ID -------------------- A new RPC method added in Open vSwitch version 2.7. The request contains the following members:: "method": "get_server_id" "params": null "id": The response object contains the following members:: "result": "" "error": null "id": same "id" as request is JSON string that contains a UUID that uniquely identifies the running OVSDB server process. A fresh UUID is generated when the process restarts. 5.1 Notation ------------ For , RFC 7047 only allows the use of ``!=``, ``==``, ``includes``, and ``excludes`` operators with set types. Open vSwitch 2.4 and later extend to allow the use of ``<``, ``<=``, ``>=``, and ``>`` operators with columns with type "set of 0 or 1 integer" and "set of 0 or 1 real". These conditions evaluate to false when the column is empty, and otherwise as described in RFC 7047 for integer and real types. is specified in Section 5.1 in the RFC with the following change: A condition can be either a 3-element JSON array as described in the RFC or a boolean value. In case of an empty array an implicit true boolean value will be considered. 5.2.6 Wait, 5.2.7 Commit, 5.2.9 Comment --------------------------------------- RFC 7047 says that the ``wait``, ``commit``, and ``comment`` operations have no corresponding result object. This is not true. Instead, when such an operation is successful, it yields a result object with no members. ovs-2.9.0/Documentation/ref/ovsdb.5.rst000066400000000000000000000122341324262074100177330ustar00rootroot00000000000000.. Copyright (c) 2017 Nicira, Inc. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. Convention for heading levels in Open vSwitch documentation: ======= Heading 0 (reserved for the title in a document) ------- Heading 1 ~~~~~~~ Heading 2 +++++++ Heading 3 ''''''' Heading 4 Avoid deeper levels because they do not render well. ===== ovsdb ===== Description =========== OVSDB, the Open vSwitch Database, is a database system whose network protocol is specified by RFC 7047. The RFC does not specify an on-disk storage format. This manpage documents the format used by Open vSwitch. Most users do not need to be concerned with this specification. Instead, to manipulate OVSDB files, refer to `ovsdb-tool(1)`. For an introduction to OVSDB as a whole, read `ovsdb(7)`. OVSDB files explicitly record changes that are implied by the database schema. For example, the OVSDB "garbage collection" feature means that when a client removes the last reference to a garbage-collected row, the database server automatically removes that row. The database file explicitly records the deletion of the garbage-collected row, so that the reader does not need to infer it. OVSDB files do not include the values of ephemeral columns. Database files are text files encoded in UTF-8 with LF (U+000A) line ends, organized as append-only series of records. Each record consists of 2 lines of text. The first line in each record has the format ``OVSDB JSON`` *length* *hash*, where *length* is a positive decimal integer and *hash* is a SHA-1 checksum expressed as 40 hexadecimal digits. Words in the first line must be separated by exactly one space. The second line must be exactly *length* bytes long (including the LF) and its SHA-1 checksum (including the LF) must match *hash* exactly. The line's contents must be a valid JSON object as specified by RFC 4627. Strings in the JSON object must be valid UTF-8. To ensure that the second line is exactly one line of text, the OVSDB implementation expresses any LF characters within a JSON string as ``\n``. For the same reason, and to save space, the OVSDB implementation does not "pretty print" the JSON object with spaces and LFs. (The OVSDB implementation tolerates LFs when reading an OVSDB database file, as long as *length* and *hash* are correct.) JSON Notation ------------- We use notation from RFC 7047 here to describe the JSON data in records. In addition to the notation defined there, we add the following: A 36-character JSON string that contains a UUID in the format described by RFC 4122, e.g. ``"550e8400-e29b-41d4-a716-446655440000"`` Standalone Format ----------------- The first record in a standalone database contains the JSON schema for the database, as specified in RFC 7047. Only this record is mandatory (a standalone file that contains only a schema represents an empty database). The second and subsequent records in a standalone database are transaction records. Each record may have the following optional special members, which do not have any semantics but are often useful to administrators looking through a database log with ``ovsdb-tool show-log``: ``"_date": `` The time at which the transaction was committed, as an integer number of milliseconds since the Unix epoch. Early versions of OVSDB counted seconds instead of milliseconds; these can be detected by noticing that their values are less than 2**32. OVSDB always writes a ``_date`` member. ``"_comment": `` A JSON string that specifies the comment provided in a transaction ``comment`` operation. If a transaction has multiple ``comment`` operations, OVSDB concatenates them into a single ``_comment`` member, separated by a new-line. OVSDB only writes a ``_comment`` member if it would be a nonempty string. Each of these records also has one or more additional members, each of which maps from the name of a database table to a : A JSON object that describes the effects of a transaction on a database table. Its names are s for rows in the table and its values are s. Either ``null``, which indicates that the transaction deleted this row, or a JSON object that describes how the transaction inserted or modified the row, whose names are the names of columns and whose values are s that give the column's new value. For new rows, the OVSDB implementation omits columns whose values have the default values for their types defined in RFC 7047 section 5.2.1; for modified rows, the OVSDB implementation omits columns whose values are unchanged. ovs-2.9.0/Documentation/ref/ovsdb.7.rst000066400000000000000000000551531324262074100177440ustar00rootroot00000000000000.. Copyright (c) 2017 Nicira, Inc. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. Convention for heading levels in Open vSwitch documentation: ======= Heading 0 (reserved for the title in a document) ------- Heading 1 ~~~~~~~ Heading 2 +++++++ Heading 3 ''''''' Heading 4 Avoid deeper levels because they do not render well. ===== ovsdb ===== Description =========== OVSDB, the Open vSwitch Database, is a network database system. Schemas in OVSDB specify the tables in a database and their columns' types and can include data, uniqueness, and referential integrity constraints. OVSDB offers atomic, consistent, isolated, durable transactions. RFC 7047 specifies the JSON-RPC based protocol that OVSDB clients and servers use to communicate. The OVSDB protocol is well suited for state synchronization because it allows each client to monitor the contents of a whole database or a subset of it. Whenever a monitored portion of the database changes, the server tells the client what rows were added or modified (including the new contents) or deleted. Thus, OVSDB clients can easily keep track of the newest contents of any part of the database. While OVSDB is general-purpose and not particularly specialized for use with Open vSwitch, Open vSwitch does use it for multiple purposes. The leading use of OVSDB is for configuring and monitoring ``ovs-vswitchd(8)``, the Open vSwitch switch daemon, using the schema documented in ``ovs-vswitchd.conf.db(5)``. The Open Virtual Network (OVN) sub-project of OVS uses two OVSDB schemas, documented in ``ovn-nb(5)`` and ``ovn-sb(5)``. Finally, Open vSwitch includes the "VTEP" schema, documented in ``vtep(5)`` that many third-party hardware switches support for configuring VXLAN, although OVS itself does not directly use this schema. The OVSDB protocol specification allows independent, interoperable implementations of OVSDB to be developed. Open vSwitch includes an OVSDB server implementation named ``ovsdb-server(1)``, which supports several protocol extensions documented in its manpage, and a basic command-line OVSDB client named ``ovsdb-client(1)``, as well as OVSDB client libraries for C and for Python. Open vSwitch documentation often speaks of these OVSDB implementations in Open vSwitch as simply "OVSDB," even though that is distinct from the OVSDB protocol; we make the distinction explicit only when it might otherwise be unclear from the context. In addition to these generic OVSDB server and client tools, Open vSwitch includes tools for working with databases that have specific schemas: ``ovs-vsctl`` works with the ``ovs-vswitchd`` configuration database, ``vtep-ctl`` works with the VTEP database, ``ovn-nbctl`` works with the OVN Northbound database, and so on. RFC 7047 specifies the OVSDB protocol but it does not specify an on-disk storage format. Open vSwitch includes ``ovsdb-tool(1)`` for working with its own on-disk database formats. The most notable feature of this format is that ``ovsdb-tool(1)`` makes it easy for users to print the transactions that have changed a database since the last time it was compacted. This feature is often useful for troubleshooting. Schemas ======= Schemas in OVSDB have a JSON format that is specified in RFC 7047. They are often stored in files with an extension ``.ovsschema``. An on-disk database in OVSDB includes a schema and data, embedding both into a single file. The Open vSwitch utility ``ovsdb-tool`` has commands that work with schema files and with the schemas embedded in database files. An Open vSwitch schema has three important identifiers. The first is its name, which is also the name used in JSON-RPC calls to identify a database based on that schema. For example, the schema used to configure Open vSwitch has the name ``Open_vSwitch``. Schema names begin with a letter or an underscore, followed by any number of letters, underscores, or digits. The ``ovsdb-tool`` commands ``schema-name`` and ``db-name`` extract the schema name from a schema or database file, respectively. An OVSDB schema also has a version of the form ``x.y.z`` e.g. ``1.2.3``. Schemas managed within the Open vSwitch project manage version numbering in the following way (but OVSDB does not mandate this approach). Whenever we change the database schema in a non-backward compatible way (e.g. when we delete a column or a table), we increment and set and to 0. When we change the database schema in a backward compatible way (e.g. when we add a new column), we increment and set to 0. When we change the database schema cosmetically (e.g. we reindent its syntax), we increment . The ``ovsdb-tool`` commands ``schema-version`` and ``db-version`` extract the schema version from a schema or database file, respectively. Very old OVSDB schemas do not have a version, but RFC 7047 mandates it. An OVSDB schema optionally has a "checksum." RFC 7047 does not specify the use of the checksum and recommends that clients ignore it. Open vSwitch uses the checksum to remind developers to update the version: at build time, if the schema's embedded checksum, ignoring the checksum field itself, does not match the schema's content, then it fails the build with a recommendation to update the version and the checksum. Thus, a developer who changes the schema, but does not update the version, receives an automatic reminder. In practice this has been an effective way to ensure compliance with the version number policy. The ``ovsdb-tool`` commands ``schema-cksum`` and ``db-cksum`` extract the schema checksum from a schema or database file, respectively. Service Models ============== OVSDB supports two service models for databases: **standalone**, and **active-backup**. The service models provide different compromises among consistency and availability. RFC 7047, which specifies the OVSDB protocol, does not mandate or specify any particular service model. The following sections describe the individual service models. Standalone Database Service Model --------------------------------- A **standalone** database runs a single server. If the server stops running, the database becomes inaccessible, and if the server's storage is lost or corrupted, the database's content is lost. This service model is appropriate when the database controls a process or activity to which it is linked via "fate-sharing." For example, an OVSDB instance that controls an Open vSwitch virtual switch daemon, ``ovs-vswitchd``, is a standalone database because a server failure would take out both the database and the virtual switch. To set up a standalone database, use ``ovsdb-tool create`` to create a database file, then run ``ovsdb-server`` to start the database service. Active-Backup Database Service Model ------------------------------------ An **active-backup** database runs two servers (on different hosts). At any given time, one of the servers is designated with the **active** role and the other the **backup** role. An active server behaves just like a standalone server. A backup server makes an OVSDB connection to the active server and uses it to continuously replicate its content as it changes in real time. OVSDB clients can connect to either server but only the active server allows data modification or lock transactions. Setup for an active-backup database starts from a working standalone database service, which is initially the active server. On another node, to set up a backup server, create a database file with the same schema as the active server. The initial contents of the database file do not matter, as long as the schema is correct, so ``ovsdb-tool create`` will work, as will copying the database file from the active server. Then use ``ovsdb-server --sync-from=`` to start the backup server, where is an OVSDB connection method (see `Connection Methods`_ below) that connects to the active server. At that point, the backup server will fetch a copy of the active database and keep it up-to-date until it is killed. When the active server in an active-backup server pair fails, an administrator can switch the backup server to an active role with the ``ovs-appctl`` command ``ovsdb-server/disconnect-active-ovsdb-server``. Clients then have read/write access to the now-active server. Of course, administrators are slow to respond compared to software, so in practice external management software detects the active server's failure and changes the backup server's role. For example, the "Integration Guide for Centralized Control" in the Open vSwitch documentation describes how to use Pacemaker for this purpose in OVN. Suppose an active server fails and its backup is promoted to active. If the failed server is revived, it must be started as a backup server. Otherwise, if both servers are active, then they may start out of sync, if the database changed while the server was down, and they will continue to diverge over time. This also happens if the software managing the database servers cannot reach the active server and therefore switches the backup to active, but other hosts can reach both servers. These "split-brain" problems are unsolvable in general for server pairs. Compared to a standalone server, the active-backup service model somewhat increases availability, at a risk of split-brain. It adds generally insignificant performance overhead. Open vSwitch 2.6 introduced support for the active-backup service model. Database Replication ==================== OVSDB can layer **replication** on top of any of its service models. Replication, in this context, means to make, and keep up-to-date, a read-only copy of the contents of a database (the ``replica``). One use of replication is to keep an up-to-date backup of a database. A replica used solely for backup would not need to support clients of its own. A set of replicas that do serve clients could be used to scale out read access to the primary database. A database replica is set up in the same way as a backup server in an active-backup pair, with the difference that the replica is never promoted to an active role. A database can have multiple replicas. Open vSwitch 2.6 introduced support for database replication. Connection Methods ================== An OVSDB **connection method** is a string that specifies how to make a JSON-RPC connection between an OVSDB client and server. Connection methods are part of the Open vSwitch implementation of OVSDB and not specified by RFC 7047. ``ovsdb-server`` uses connection methods to specify how it should listen for connections from clients and ``ovsdb-client`` uses them to specify how it should connect to a server. Connections in the opposite direction, where ``ovsdb-server`` connects to a client that is configured to listen for an incoming connection, are also possible. Connection methods are classified as **active** or **passive**. An active connection method makes an outgoing connection to a remote host; a passive connection method listens for connections from remote hosts. The most common arrangement is to configure an OVSDB server with passive connection methods and clients with active ones, but the OVSDB implementation in Open vSwitch supports the opposite arrangement as well. OVSDB supports the following active connection methods: ssl:: The specified SSL or TLS on the host at the given . tcp:: The specified TCP on the host at the given . unix: On Unix-like systems, connect to the Unix domain server socket named . On Windows, connect to a local named pipe that is represented by a file created in the path to mimic the behavior of a Unix domain socket. OVSDB supports the following passive connection methods: pssl:[:] Listen on the given TCP for SSL or TLS connections. By default, connections are not bound to a particular local IP address. Specifying limits connections to those from the given IP. ptcp:[:] Listen on the given TCP . By default, connections are not bound to a particular local IP address. Specifying limits connections to those from the given IP. punix: On Unix-like systems, listens for connections on the Unix domain socket named . On Windows, listens on a local named pipe, creating a named pipe to mimic the behavior of a Unix domain socket. All IP-based connection methods accept IPv4 and IPv6 addresses. To specify an IPv6 address, wrap it in square brackets, e.g. ``ssl:[::1]:6640``. Passive IP-based connection methods by default listen for IPv4 connections only; use ``[::]`` as the address to accept both IPv4 and IPv6 connections, e.g. ``pssl:6640:[::]``. DNS names are not accepted. On Linux, use ``%`` to designate a scope for IPv6 link-level addresses, e.g. ``ssl:[fe80::1234%eth0]:6653``. The may be omitted from connection methods that use a port number. The default for TCP-based connection methods is 6640, e.g. ``pssl:`` is equivalent to ``pssl:6640``. In Open vSwitch prior to version 2.4.0, the default port was 6632. To avoid incompatibility between older and newer versions, we encourage users to specify a port number. The ``ssl`` and ``pssl`` connection methods requires additional configuration through ``--private-key``, ``--certificate``, and ``--ca-cert`` command line options. Open vSwitch can be built without SSL support, in which case these connection methods are not supported. Database Life Cycle =================== This section describes how to handle various events in the life cycle of a database using the Open vSwitch implementation of OVSDB. Creating a Database ------------------- Creating and starting up the service for a new database was covered separately for each database service model in the `Service Models`_ section, above. Backing Up and Restoring a Database ----------------------------------- OVSDB is often used in contexts where the database contents are not particularly valuable. For example, in many systems, the database for configuring ``ovs-vswitchd`` is essentially rebuilt from scratch at boot time. It is not worthwhile to back up these databases. When OVSDB is used for valuable data, a backup strategy is worth considering. One way is to use database replication, discussed above in `Database Replication`_ which keeps an online, up-to-date copy of a database, possibly on a remote system. This works with all OVSDB service models. A more common backup strategy is to periodically take and store a snapshot. For the standalone and active-backup service models, making a copy of the database file, e.g. using ``cp``, effectively makes a snapshot, and because OVSDB database files are append-only, it works even if the database is being modified when the snapshot takes place. Another way to make a backup is to use ``ovsdb-client backup``, which connects to a running database server and outputs an atomic snapshot of its schema and content, in the same format used for on-disk databases. Multiple options are also available when the time comes to restore a database from a backup. One option is to stop the database server or servers, overwrite the database file with the backup (e.g. with ``cp``), and then restart the servers. Another way is to use ``ovsdb-client restore``, which connects to a running database server and replaces the data in one of its databases by a provided snapshot. The advantage of ``ovsdb-client restore`` is that it causes zero downtime for the database and its server. It has the downside that UUIDs of rows in the restored database will differ from those in the snapshot, because the OVSDB protocol does not allow clients to specify row UUIDs. None of these approaches saves and restores data in columns that the schema designates as ephemeral. This is by design: the designer of a schema only marks a column as ephemeral if it is acceptable for its data to be lost when a database server restarts. Upgrading or Downgrading a Database ----------------------------------- The evolution of a piece of software can require changes to the schemas of the databases that it uses. For example, new features might require new tables or new columns in existing tables, or conceptual changes might require a database to be reorganized in other ways. In some cases, the easiest way to deal with a change in a database schema is to delete the existing database and start fresh with the new schema, especially if the data in the database is easy to reconstruct. But in many other cases, it is better to convert the database from one schema to another. The OVSDB implementation in Open vSwitch has built-in support for some simple cases of converting a database from one schema to another. This support can handle changes that add or remove database columns or tables or that eliminate constraints (for example, changing a column that must have exactly one value into one that has one or more values). It can also handle changes that add constraints or make them stricter, but only if the existing data in the database satisfies the new constraints (for example, changing a column that has one or more values into a column with exactly one value, if every row in the column has exactly one value). The built-in conversion can cause data loss in obvious ways, for example if the new schema removes tables or columns, or indirectly, for example by deleting unreferenced rows in tables that the new schema marks for garbage collection. Converting a database can lose data, so it is wise to make a backup beforehand. To use OVSDB's built-in support for schema conversion with a standalone or active-backup database, first stop the database server or servers, then use ``ovsdb-tool convert`` to convert it to the new schema, and then restart the database server. Schema versions and checksums (see Schemas_ above) can give hints about whether a database needs to be converted to a new schema. If there is any question, though, the ``needs-conversion`` command on ``ovsdb-tool`` can provide a definitive answer. Working with Database History ----------------------------- Both on-disk database formats that OVSDB supports are organized as a stream of transaction records. Each record describes a change to the database as a list of rows that were inserted or deleted or modified, along with the details. Therefore, in normal operation, a database file only grows, as each change causes another record to be appended at the end. Usually, a user has no need to understand this file structure. This section covers some exceptions. Compacting Databases -------------------- If OVSDB database files were truly append-only, then over time they would grow without bound. To avoid this problem, OVSDB can **compact** a database file, that is, replace it by a new version that contains only the current database contents, as if it had been inserted by a single transaction. From time to time, ``ovsdb-server`` automatically compacts a database that grows much larger than its minimum size. Because ``ovsdb-server`` automatically compacts databases, it is usually not necessary to compact them manually, but OVSDB still offers a few ways to do it. First, ``ovsdb-tool compact`` can compact a standalone or active-backup database that is not currently being served by ``ovsdb-server`` (or otherwise locked for writing by another process). To compact any database that is currently being served by ``ovsdb-server``, use ``ovs-appctl`` to send the ``ovsdb-server/compact`` command. Each server in an active-backup database maintains its database file independently, so to compact all of them, issue this command separately on each server. Viewing History --------------- The ``ovsdb-tool`` utility's ``show-log`` command displays the transaction records in an OVSDB database file in a human-readable format. By default, it shows minimal detail, but adding the option ``-m`` once or twice increases the level of detail. In addition to the transaction data, it shows the time and date of each transaction and any "comment" added to the transaction by the client. The comments can be helpful for quickly understanding a transaction; for example, ``ovs-vsctl`` adds its command line to the transactions that it makes. For active-backup databases, the sequence of transactions in each server's log will differ, even at points when they reflect the same data. Truncating History ------------------ It may occasionally be useful to "roll back" a database file to an earlier point. Because of the organization of OVSDB records, this is easy to do. Start by noting the record number of the first record to delete in ``ovsdb-tool show-log`` output. Each record is two lines of plain text, so trimming the log is as simple as running ``head -n ``, where = 2 * . Corruption ---------- When ``ovsdb-server`` opens an OVSDB database file, of any kind, it reads as many transaction records as it can from the file until it reaches the end of the file or it encounters a corrupted record. At that point it stops reading and regards the data that it has read to this point as the full contents of the database file, effectively rolling the database back to an earlier point. Each transaction record contains an embedded SHA-1 checksum, which the server verifies as it reads a database file. It detects corruption when a checksum fails to verify. Even though SHA-1 is no longer considered secure for use in cryptography, it is acceptable for this purpose because it is not used to defend against malicious attackers. The first record in a standalone or active-backup database file specifies the schema. ``ovsdb-server`` will refuse to work with a database whose first record is corrupted. Delete and recreate such a database, or restore it from a backup. When ``ovsdb-server`` adds records to a database file in which it detected corruption, it first truncates the file just after the last good record. See Also ======== RFC 7047, "The Open vSwitch Database Management Protocol." Open vSwitch implementations of generic OVSDB functionality: ``ovsdb-server(1)``, ``ovsdb-client(1)``, ``ovsdb-tool(1)``. Tools for working with databases that have specific OVSDB schemas: ``ovs-vsctl(8)``, ``vtep-ctl(8)``, ``ovn-nbctl(8)``, ``ovn-sbctl(8)``. OVSDB schemas for Open vSwitch and related functionality: ``ovs-vswitchd.conf.db(5)``, ``vtep(5)``, ``ovn-nb(5)``, ``ovn-sb(5)``. ovs-2.9.0/Documentation/requirements.txt000066400000000000000000000000541324262074100204260ustar00rootroot00000000000000sphinx>=1.1,<2.0 ovs_sphinx_theme>=1.0,<1.1 ovs-2.9.0/Documentation/topics/000077500000000000000000000000001324262074100164445ustar00rootroot00000000000000ovs-2.9.0/Documentation/topics/bonding.rst000066400000000000000000000275451324262074100206330ustar00rootroot00000000000000.. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. Convention for heading levels in Open vSwitch documentation: ======= Heading 0 (reserved for the title in a document) ------- Heading 1 ~~~~~~~ Heading 2 +++++++ Heading 3 ''''''' Heading 4 Avoid deeper levels because they do not render well. ======= Bonding ======= Bonding allows two or more interfaces (the "slaves") to share network traffic. From a high-level point of view, bonded interfaces act like a single port, but they have the bandwidth of multiple network devices, e.g. two 1 GB physical interfaces act like a single 2 GB interface. Bonds also increase robustness: the bonded port does not go down as long as at least one of its slaves is up. In vswitchd, a bond always has at least two slaves (and may have more). If a configuration error, etc. would cause a bond to have only one slave, the port becomes an ordinary port, not a bonded port, and none of the special features of bonded ports described in this section apply. There are many forms of bonding of which ovs-vswitchd implements only a few. The most complex bond ovs-vswitchd implements is called "source load balancing" or SLB bonding. SLB bonding divides traffic among the slaves based on the Ethernet source address. This is useful only if the traffic over the bond has multiple Ethernet source addresses, for example if network traffic from multiple VMs are multiplexed over the bond. .. note:: Most of the ovs-vswitchd implementation is in ``vswitchd/bridge.c``, so code references below should be assumed to refer to that file except as otherwise specified. Enabling and Disabling Slaves ----------------------------- When a bond is created, a slave is initially enabled or disabled based on whether carrier is detected on the NIC (see ``iface_create()``). After that, a slave is disabled if its carrier goes down for a period of time longer than the downdelay, and it is enabled if carrier comes up for longer than the updelay (see ``bond_link_status_update()``). There is one exception where the updelay is skipped: if no slaves at all are currently enabled, then the first slave on which carrier comes up is enabled immediately. The updelay should be set to a time longer than the STP forwarding delay of the physical switch to which the bond port is connected (if STP is enabled on that switch). Otherwise, the slave will be enabled, and load may be shifted to it, before the physical switch starts forwarding packets on that port, which can cause some data to be "blackholed" for a time. The exception for a single enabled slave does not cause any problem in this regard because when no slaves are enabled all output packets are blackholed anyway. When a slave becomes disabled, the vswitch immediately chooses a new output port for traffic that was destined for that slave (see ``bond_enable_slave()``). It also sends a "gratuitous learning packet", specifically a RARP, on the bond port (on the newly chosen slave) for each MAC address that the vswitch has learned on a port other than the bond (see ``bundle_send_learning_packets()``), to teach the physical switch that the new slave should be used in place of the one that is now disabled. (This behavior probably makes sense only for a vswitch that has only one port (the bond) connected to a physical switch; vswitchd should probably provide a way to disable or configure it in other scenarios.) Bond Packet Input ----------------- Bonding accepts unicast packets on any bond slave. This can occasionally cause packet duplication for the first few packets sent to a given MAC, if the physical switch attached to the bond is flooding packets to that MAC because it has not yet learned the correct slave for that MAC. Bonding only accepts multicast (and broadcast) packets on a single bond slave (the "active slave") at any given time. Multicast packets received on other slaves are dropped. Otherwise, every multicast packet would be duplicated, once for every bond slave, because the physical switch attached to the bond will flood those packets. Bonding also drops received packets when the vswitch has learned that the packet's MAC is on a port other than the bond port itself. This is because it is likely that the vswitch itself sent the packet out the bond port on a different slave and is now receiving the packet back. This occurs when the packet is multicast or the physical switch has not yet learned the MAC and is flooding it. However, the vswitch makes an exception to this rule for broadcast ARP replies, which indicate that the MAC has moved to another switch, probably due to VM migration. (ARP replies are normally unicast, so this exception does not match normal ARP replies. It will match the learning packets sent on bond fail-over.) The active slave is simply the first slave to be enabled after the bond is created (see ``bond_choose_active_slave()``). If the active slave is disabled, then a new active slave is chosen among the slaves that remain active. Currently due to the way that configuration works, this tends to be the remaining slave whose interface name is first alphabetically, but this is by no means guaranteed. Bond Packet Output ------------------ When a packet is sent out a bond port, the bond slave actually used is selected based on the packet's source MAC and VLAN tag (see ``bond_choose_output_slave()``). In particular, the source MAC and VLAN tag are hashed into one of 256 values, and that value is looked up in a hash table (the "bond hash") kept in the ``bond_hash`` member of struct port. The hash table entry identifies a bond slave. If no bond slave has yet been chosen for that hash table entry, vswitchd chooses one arbitrarily. Every 10 seconds, vswitchd rebalances the bond slaves (see ``bond_rebalance()``). To rebalance, vswitchd examines the statistics for the number of bytes transmitted by each slave over approximately the past minute, with data sent more recently weighted more heavily than data sent less recently. It considers each of the slaves in order from most-loaded to least-loaded. If highly loaded slave H is significantly more heavily loaded than the least-loaded slave L, and slave H carries at least two hashes, then vswitchd shifts one of H's hashes to L. However, vswitchd will only shift a hash from H to L if it will decrease the ratio of the load between H and L by at least 0.1. Currently, "significantly more loaded" means that H must carry at least 1 Mbps more traffic, and that traffic must be at least 3% greater than L's. Bond Balance Modes ------------------ Each bond balancing mode has different considerations, described below. LACP Bonding ~~~~~~~~~~~~ LACP bonding requires the remote switch to implement LACP, but it is otherwise very simple in that, after LACP negotiation is complete, there is no need for special handling of received packets. Several of the physical switches that support LACP block all traffic for ports that are configured to use LACP, until LACP is negotiated with the host. When configuring a LACP bond on a OVS host (eg: XenServer), this means that there will be an interruption of the network connectivity between the time the ports on the physical switch and the bond on the OVS host are configured. The interruption may be relatively long, if different people are responsible for managing the switches and the OVS host. Such network connectivity failure can be avoided if LACP can be configured on the OVS host before configuring the physical switch, and having the OVS host fall back to a bond mode (active-backup) till the physical switch LACP configuration is complete. An option "lacp-fallback-ab" exists to provide such behavior on openvswitch. Active Backup Bonding ~~~~~~~~~~~~~~~~~~~~~ Active Backup bonds send all traffic out one "active" slave until that slave becomes unavailable. Since they are significantly less complicated than SLB bonds, they are preferred when LACP is not an option. Additionally, they are the only bond mode which supports attaching each slave to a different upstream switch. SLB Bonding ~~~~~~~~~~~ SLB bonding allows a limited form of load balancing without the remote switch's knowledge or cooperation. The basics of SLB are simple. SLB assigns each source MAC+VLAN pair to a link and transmits all packets from that MAC+VLAN through that link. Learning in the remote switch causes it to send packets to that MAC+VLAN through the same link. SLB bonding has the following complications: 0. When the remote switch has not learned the MAC for the destination of a unicast packet and hence floods the packet to all of the links on the SLB bond, Open vSwitch will forward duplicate packets, one per link, to each other switch port. Open vSwitch does not solve this problem. 1. When the remote switch receives a multicast or broadcast packet from a port not on the SLB bond, it will forward it to all of the links in the SLB bond. This would cause packet duplication if not handled specially. Open vSwitch avoids packet duplication by accepting multicast and broadcast packets on only the active slave, and dropping multicast and broadcast packets on all other slaves. 2. When Open vSwitch forwards a multicast or broadcast packet to a link in the SLB bond other than the active slave, the remote switch will forward it to all of the other links in the SLB bond, including the active slave. Without special handling, this would mean that Open vSwitch would forward a second copy of the packet to each switch port (other than the bond), including the port that originated the packet. Open vSwitch deals with this case by dropping packets received on any SLB bonded link that have a source MAC+VLAN that has been learned on any other port. (This means that SLB as implemented in Open vSwitch relies critically on MAC learning. Notably, SLB is incompatible with the "flood_vlans" feature.) 3. Suppose that a MAC+VLAN moves to an SLB bond from another port (e.g. when a VM is migrated from this hypervisor to a different one). Without additional special handling, Open vSwitch will not notice until the MAC learning entry expires, up to 60 seconds later as a consequence of rule #2. Open vSwitch avoids a 60-second delay by listening for gratuitous ARPs, which VMs commonly emit upon migration. As an exception to rule #2, a gratuitous ARP received on an SLB bond is not dropped and updates the MAC learning table in the usual way. (If a move does not trigger a gratuitous ARP, or if the gratuitous ARP is lost in the network, then a 60-second delay still occurs.) 4. Suppose that a MAC+VLAN moves from an SLB bond to another port (e.g. when a VM is migrated from a different hypervisor to this one), that the MAC+VLAN emits a gratuitous ARP, and that Open vSwitch forwards that gratuitous ARP to a link in the SLB bond other than the active slave. The remote switch will forward the gratuitous ARP to all of the other links in the SLB bond, including the active slave. Without additional special handling, this would mean that Open vSwitch would learn that the MAC+VLAN was located on the SLB bond, as a consequence of rule #3. Open vSwitch avoids this problem by "locking" the MAC learning table entry for a MAC+VLAN from which a gratuitous ARP was received from a non-SLB bond port. For 5 seconds, a locked MAC learning table entry will not be updated based on a gratuitous ARP received on a SLB bond. ovs-2.9.0/Documentation/topics/datapath.rst000066400000000000000000000313141324262074100207660ustar00rootroot00000000000000.. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. Convention for heading levels in Open vSwitch documentation: ======= Heading 0 (reserved for the title in a document) ------- Heading 1 ~~~~~~~ Heading 2 +++++++ Heading 3 ''''''' Heading 4 Avoid deeper levels because they do not render well. ======================================= Open vSwitch Datapath Development Guide ======================================= The Open vSwitch kernel module allows flexible userspace control over flow-level packet processing on selected network devices. It can be used to implement a plain Ethernet switch, network device bonding, VLAN processing, network access control, flow-based network control, and so on. The kernel module implements multiple "datapaths" (analogous to bridges), each of which can have multiple "vports" (analogous to ports within a bridge). Each datapath also has associated with it a "flow table" that userspace populates with "flows" that map from keys based on packet headers and metadata to sets of actions. The most common action forwards the packet to another vport; other actions are also implemented. When a packet arrives on a vport, the kernel module processes it by extracting its flow key and looking it up in the flow table. If there is a matching flow, it executes the associated actions. If there is no match, it queues the packet to userspace for processing (as part of its processing, userspace will likely set up a flow to handle further packets of the same type entirely in-kernel). Flow Key Compatibility ---------------------- Network protocols evolve over time. New protocols become important and existing protocols lose their prominence. For the Open vSwitch kernel module to remain relevant, it must be possible for newer versions to parse additional protocols as part of the flow key. It might even be desirable, someday, to drop support for parsing protocols that have become obsolete. Therefore, the Netlink interface to Open vSwitch is designed to allow carefully written userspace applications to work with any version of the flow key, past or future. To support this forward and backward compatibility, whenever the kernel module passes a packet to userspace, it also passes along the flow key that it parsed from the packet. Userspace then extracts its own notion of a flow key from the packet and compares it against the kernel-provided version: - If userspace's notion of the flow key for the packet matches the kernel's, then nothing special is necessary. - If the kernel's flow key includes more fields than the userspace version of the flow key, for example if the kernel decoded IPv6 headers but userspace stopped at the Ethernet type (because it does not understand IPv6), then again nothing special is necessary. Userspace can still set up a flow in the usual way, as long as it uses the kernel-provided flow key to do it. - If the userspace flow key includes more fields than the kernel's, for example if userspace decoded an IPv6 header but the kernel stopped at the Ethernet type, then userspace can forward the packet manually, without setting up a flow in the kernel. This case is bad for performance because every packet that the kernel considers part of the flow must go to userspace, but the forwarding behavior is correct. (If userspace can determine that the values of the extra fields would not affect forwarding behavior, then it could set up a flow anyway.) How flow keys evolve over time is important to making this work, so the following sections go into detail. Flow Key Format --------------- A flow key is passed over a Netlink socket as a sequence of Netlink attributes. Some attributes represent packet metadata, defined as any information about a packet that cannot be extracted from the packet itself, e.g. the vport on which the packet was received. Most attributes, however, are extracted from headers within the packet, e.g. source and destination addresses from Ethernet, IP, or TCP headers. The ```` header file defines the exact format of the flow key attributes. For informal explanatory purposes here, we write them as comma-separated strings, with parentheses indicating arguments and nesting. For example, the following could represent a flow key corresponding to a TCP packet that arrived on vport 1:: in_port(1), eth(src=e0:91:f5:21:d0:b2, dst=00:02:e3:0f:80:a4), eth_type(0x0800), ipv4(src=172.16.0.20, dst=172.18.0.52, proto=17, tos=0, frag=no), tcp(src=49163, dst=80) Often we ellipsize arguments not important to the discussion, e.g.:: in_port(1), eth(...), eth_type(0x0800), ipv4(...), tcp(...) Wildcarded Flow Key Format -------------------------- A wildcarded flow is described with two sequences of Netlink attributes passed over the Netlink socket. A flow key, exactly as described above, and an optional corresponding flow mask. A wildcarded flow can represent a group of exact match flows. Each ``1`` bit in the mask specifies an exact match with the corresponding bit in the flow key. A ``0`` bit specifies a don't care bit, which will match either a ``1`` or ``0`` bit of an incoming packet. Using a wildcarded flow can improve the flow set up rate by reducing the number of new flows that need to be processed by the user space program. Support for the mask Netlink attribute is optional for both the kernel and user space program. The kernel can ignore the mask attribute, installing an exact match flow, or reduce the number of don't care bits in the kernel to less than what was specified by the user space program. In this case, variations in bits that the kernel does not implement will simply result in additional flow setups. The kernel module will also work with user space programs that neither support nor supply flow mask attributes. Since the kernel may ignore or modify wildcard bits, it can be difficult for the userspace program to know exactly what matches are installed. There are two possible approaches: reactively install flows as they miss the kernel flow table (and therefore not attempt to determine wildcard changes at all) or use the kernel's response messages to determine the installed wildcards. When interacting with userspace, the kernel should maintain the match portion of the key exactly as originally installed. This will provides a handle to identify the flow for all future operations. However, when reporting the mask of an installed flow, the mask should include any restrictions imposed by the kernel. The behavior when using overlapping wildcarded flows is undefined. It is the responsibility of the user space program to ensure that any incoming packet can match at most one flow, wildcarded or not. The current implementation performs best-effort detection of overlapping wildcarded flows and may reject some but not all of them. However, this behavior may change in future versions. Unique Flow Identifiers ----------------------- An alternative to using the original match portion of a key as the handle for flow identification is a unique flow identifier, or "UFID". UFIDs are optional for both the kernel and user space program. User space programs that support UFID are expected to provide it during flow setup in addition to the flow, then refer to the flow using the UFID for all future operations. The kernel is not required to index flows by the original flow key if a UFID is specified. Basic Rule for Evolving Flow Keys --------------------------------- Some care is needed to really maintain forward and backward compatibility for applications that follow the rules listed under "Flow key compatibility" above. The basic rule is obvious: New network protocol support must only supplement existing flow key attributes. It must not change the meaning of already defined flow key attributes. This rule does have less-obvious consequences so it is worth working through a few examples. Suppose, for example, that the kernel module did not already implement VLAN parsing. Instead, it just interpreted the 802.1Q TPID (``0x8100``) as the Ethertype then stopped parsing the packet. The flow key for any packet with an 802.1Q header would look essentially like this, ignoring metadata:: eth(...), eth_type(0x8100) Naively, to add VLAN support, it makes sense to add a new "vlan" flow key attribute to contain the VLAN tag, then continue to decode the encapsulated headers beyond the VLAN tag using the existing field definitions. With this change, a TCP packet in VLAN 10 would have a flow key much like this:: eth(...), vlan(vid=10, pcp=0), eth_type(0x0800), ip(proto=6, ...), tcp(...) But this change would negatively affect a userspace application that has not been updated to understand the new "vlan" flow key attribute. The application could, following the flow compatibility rules above, ignore the "vlan" attribute that it does not understand and therefore assume that the flow contained IP packets. This is a bad assumption (the flow only contains IP packets if one parses and skips over the 802.1Q header) and it could cause the application's behavior to change across kernel versions even though it follows the compatibility rules. The solution is to use a set of nested attributes. This is, for example, why 802.1Q support uses nested attributes. A TCP packet in VLAN 10 is actually expressed as:: eth(...), eth_type(0x8100), vlan(vid=10, pcp=0), encap(eth_type(0x0800), ip(proto=6, ...), tcp(...))) Notice how the ``eth_type``, ``ip``, and ``tcp`` flow key attributes are nested inside the ``encap`` attribute. Thus, an application that does not understand the ``vlan`` key will not see either of those attributes and therefore will not misinterpret them. (Also, the outer ``eth_type`` is still ``0x8100``, not changed to ``0x0800``) Handling Malformed Packets -------------------------- Don't drop packets in the kernel for malformed protocol headers, bad checksums, etc. This would prevent userspace from implementing a simple Ethernet switch that forwards every packet. Instead, in such a case, include an attribute with "empty" content. It doesn't matter if the empty content could be valid protocol values, as long as those values are rarely seen in practice, because userspace can always forward all packets with those values to userspace and handle them individually. For example, consider a packet that contains an IP header that indicates protocol 6 for TCP, but which is truncated just after the IP header, so that the TCP header is missing. The flow key for this packet would include a tcp attribute with all-zero ``src`` and ``dst``, like this:: eth(...), eth_type(0x0800), ip(proto=6, ...), tcp(src=0, dst=0) As another example, consider a packet with an Ethernet type of 0x8100, indicating that a VLAN TCI should follow, but which is truncated just after the Ethernet type. The flow key for this packet would include an all-zero-bits vlan and an empty encap attribute, like this:: eth(...), eth_type(0x8100), vlan(0), encap() Unlike a TCP packet with source and destination ports 0, an all-zero-bits VLAN TCI is not that rare, so the CFI bit (aka VLAN_TAG_PRESENT inside the kernel) is ordinarily set in a vlan attribute expressly to allow this situation to be distinguished. Thus, the flow key in this second example unambiguously indicates a missing or malformed VLAN TCI. Other Rules ----------- The other rules for flow keys are much less subtle: - Duplicate attributes are not allowed at a given nesting level. - Ordering of attributes is not significant. - When the kernel sends a given flow key to userspace, it always composes it the same way. This allows userspace to hash and compare entire flow keys that it may not be able to fully interpret. Coding Rules ------------ Implement the headers and codes for compatibility with older kernel in ``linux/compat/`` directory. All public functions should be exported using ``EXPORT_SYMBOL`` macro. Public function replacing the same-named kernel function should be prefixed with ``rpl_``. Otherwise, the function should be prefixed with ``ovs_``. For special case when it is not possible to follow this rule (e.g., the ``pskb_expand_head()`` function), the function name must be added to ``linux/compat/build-aux/export-check-whitelist``, otherwise, the compilation check ``check-export-symbol`` will fail. ovs-2.9.0/Documentation/topics/design.rst000066400000000000000000001442011324262074100204510ustar00rootroot00000000000000.. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. Convention for heading levels in Open vSwitch documentation: ======= Heading 0 (reserved for the title in a document) ------- Heading 1 ~~~~~~~ Heading 2 +++++++ Heading 3 ''''''' Heading 4 Avoid deeper levels because they do not render well. ================================ Design Decisions In Open vSwitch ================================ This document describes design decisions that went into implementing Open vSwitch. While we believe these to be reasonable decisions, it is impossible to predict how Open vSwitch will be used in all environments. Understanding assumptions made by Open vSwitch is critical to a successful deployment. The end of this document contains contact information that can be used to let us know how we can make Open vSwitch more generally useful. Asynchronous Messages --------------------- Over time, Open vSwitch has added many knobs that control whether a given controller receives OpenFlow asynchronous messages. This section describes how all of these features interact. First, a service controller never receives any asynchronous messages unless it changes its miss_send_len from the service controller default of zero in one of the following ways: - Sending an ``OFPT_SET_CONFIG`` message with nonzero ``miss_send_len``. - Sending any ``NXT_SET_ASYNC_CONFIG`` message: as a side effect, this message changes the ``miss_send_len`` to ``OFP_DEFAULT_MISS_SEND_LEN`` (128) for service controllers. Second, ``OFPT_FLOW_REMOVED`` and ``NXT_FLOW_REMOVED`` messages are generated only if the flow that was removed had the ``OFPFF_SEND_FLOW_REM`` flag set. Third, ``OFPT_PACKET_IN`` and ``NXT_PACKET_IN`` messages are sent only to OpenFlow controller connections that have the correct connection ID (see ``struct nx_controller_id`` and ``struct nx_action_controller``): - For packet-in messages generated by a ``NXAST_CONTROLLER`` action, the controller ID specified in the action. - For other packet-in messages, controller ID zero. (This is the default ID when an OpenFlow controller does not configure one.) Finally, Open vSwitch consults a per-connection table indexed by the message type, reason code, and current role. The following table shows how this table is initialized by default when an OpenFlow connection is made. An entry labeled ``yes`` means that the message is sent, an entry labeled ``---`` means that the message is suppressed. .. table:: ``OFPT_PACKET_IN`` / ``NXT_PACKET_IN`` =========================================== ======= ===== master/ message and reason code other slave =========================================== ======= ===== ``OFPR_NO_MATCH`` yes --- ``OFPR_ACTION`` yes --- ``OFPR_INVALID_TTL`` --- --- ``OFPR_ACTION_SET`` (OF1.4+) yes --- ``OFPR_GROUP`` (OF1.4+) yes --- ``OFPR_PACKET_OUT`` (OF1.4+) yes --- =========================================== ======= ===== .. table:: ``OFPT_FLOW_REMOVED`` / ``NXT_FLOW_REMOVED`` =========================================== ======= ===== master/ message and reason code other slave =========================================== ======= ===== ``OFPRR_IDLE_TIMEOUT`` yes --- ``OFPRR_HARD_TIMEOUT`` yes --- ``OFPRR_DELETE`` yes --- ``OFPRR_GROUP_DELETE`` (OF1.3+) yes --- ``OFPRR_METER_DELETE`` (OF1.4+) yes --- ``OFPRR_EVICTION`` (OF1.4+) yes --- =========================================== ======= ===== .. table:: ``OFPT_PORT_STATUS`` =========================================== ======= ===== master/ message and reason code other slave =========================================== ======= ===== ``OFPPR_ADD`` yes yes ``OFPPR_DELETE`` yes yes ``OFPPR_MODIFY`` yes yes =========================================== ======= ===== .. table:: ``OFPT_ROLE_REQUEST`` / ``OFPT_ROLE_REPLY`` (OF1.4+) =========================================== ======= ===== master/ message and reason code other slave =========================================== ======= ===== ``OFPCRR_MASTER_REQUEST`` --- --- ``OFPCRR_CONFIG`` --- --- ``OFPCRR_EXPERIMENTER`` --- --- =========================================== ======= ===== .. table:: ``OFPT_TABLE_STATUS`` (OF1.4+) =========================================== ======= ===== master/ message and reason code other slave =========================================== ======= ===== ``OFPTR_VACANCY_DOWN`` --- --- ``OFPTR_VACANCY_UP`` --- --- =========================================== ======= ===== .. table:: ``OFPT_REQUESTFORWARD`` (OF1.4+) =========================================== ======= ===== master/ message and reason code other slave =========================================== ======= ===== ``OFPRFR_GROUP_MOD`` --- --- ``OFPRFR_METER_MOD`` --- --- =========================================== ======= ===== The ``NXT_SET_ASYNC_CONFIG`` message directly sets all of the values in this table for the current connection. The ``OFPC_INVALID_TTL_TO_CONTROLLER`` bit in the ``OFPT_SET_CONFIG`` message controls the setting for ``OFPR_INVALID_TTL`` for the "master" role. ``OFPAT_ENQUEUE`` ----------------- The OpenFlow 1.0 specification requires the output port of the ``OFPAT_ENQUEUE`` action to "refer to a valid physical port (i.e. < ``OFPP_MAX``) or ``OFPP_IN_PORT``". Although ``OFPP_LOCAL`` is not less than ``OFPP_MAX``, it is an 'internal' port which can have QoS applied to it in Linux. Since we allow the ``OFPAT_ENQUEUE`` to apply to 'internal' ports whose port numbers are less than ``OFPP_MAX``, we interpret ``OFPP_LOCAL`` as a physical port and support ``OFPAT_ENQUEUE`` on it as well. ``OFPT_FLOW_MOD`` ----------------- The OpenFlow specification for the behavior of ``OFPT_FLOW_MOD`` is confusing. The following tables summarize the Open vSwitch implementation of its behavior in the following categories: "match on priority" Whether the ``flow_mod`` acts only on flows whose priority matches that included in the ``flow_mod`` message. "match on out_port" Whether the ``flow_mod`` acts only on flows that output to the out_port included in the flow_mod message (if out_port is not ``OFPP_NONE``). OpenFlow 1.1 and later have a similar feature (not listed separately here) for ``out_group``. "match on flow_cookie": Whether the ``flow_mod`` acts only on flows whose ``flow_cookie`` matches an optional controller-specified value and mask. "updates flow_cookie": Whether the ``flow_mod`` changes the ``flow_cookie`` of the flow or flows that it matches to the ``flow_cookie`` included in the flow_mod message. "updates ``OFPFF_`` flags": Whether the flow_mod changes the ``OFPFF_SEND_FLOW_REM`` flag of the flow or flows that it matches to the setting included in the flags of the flow_mod message. "honors ``OFPFF_CHECK_OVERLAP``": Whether the ``OFPFF_CHECK_OVERLAP`` flag in the flow_mod is significant. "updates ``idle_timeout``" and "updates ``hard_timeout``": Whether the ``idle_timeout`` and hard_timeout in the ``flow_mod``, respectively, have an effect on the flow or flows matched by the ``flow_mod``. "updates idle timer": Whether the ``flow_mod`` resets the per-flow timer that measures how long a flow has been idle. "updates hard timer": Whether the ``flow_mod`` resets the per-flow timer that measures how long it has been since a flow was modified. "zeros counters": Whether the ``flow_mod`` resets per-flow packet and byte counters to zero. "may add a new flow": Whether the ``flow_mod`` may add a new flow to the flow table. (Obviously this is always true for "add" commands but in some OpenFlow versions "modify" and "modify-strict" can also add new flows.) "sends ``flow_removed`` message": Whether the flow_mod generates a flow_removed message for the flow or flows that it affects. An entry labeled ``yes`` means that the flow mod type does have the indicated behavior, ``---`` means that it does not, an empty cell means that the property is not applicable, and other values are explained below the table. OpenFlow 1.0 ~~~~~~~~~~~~ ================================ === ====== ====== ====== ====== MODIFY DELETE RULE ADD MODIFY STRICT DELETE STRICT ================================ === ====== ====== ====== ====== match on ``priority`` yes --- yes --- yes match on ``out_port`` --- --- --- yes yes match on ``flow_cookie`` --- --- --- --- --- match on ``table_id`` --- --- --- --- --- controller chooses ``table_id`` --- --- --- updates ``flow_cookie`` yes yes yes updates ``OFPFF_SEND_FLOW_REM`` yes + + honors ``OFPFF_CHECK_OVERLAP`` yes + + updates ``idle_timeout`` yes + + updates ``hard_timeout`` yes + + resets idle timer yes + + resets hard timer yes yes yes zeros counters yes + + may add a new flow yes yes yes sends ``flow_removed`` message --- --- --- % % ================================ === ====== ====== ====== ====== where: ``+`` "modify" and "modify-strict" only take these actions when they create a new flow, not when they update an existing flow. ``%`` "delete" and "delete_strict" generates a flow_removed message if the deleted flow or flows have the ``OFPFF_SEND_FLOW_REM`` flag set. (Each controller can separately control whether it wants to receive the generated messages.) OpenFlow 1.1 ~~~~~~~~~~~~ OpenFlow 1.1 makes these changes: - The controller now must specify the ``table_id`` of the flow match searched and into which a flow may be inserted. Behavior for a ``table_id`` of 255 is undefined. - A ``flow_mod``, except an "add", can now match on the ``flow_cookie``. - When a ``flow_mod`` matches on the ``flow_cookie``, "modify" and "modify-strict" never insert a new flow. ================================ === ====== ====== ====== ====== MODIFY DELETE RULE ADD MODIFY STRICT DELETE STRICT ================================ === ====== ====== ====== ====== match on ``priority`` yes --- yes --- yes match on ``out_port`` --- --- --- yes yes match on ``flow_cookie`` --- yes yes yes yes match on ``table_id`` yes yes yes yes yes controller chooses ``table_id`` yes yes yes updates ``flow_cookie`` yes --- --- updates ``OFPFF_SEND_FLOW_REM`` yes + + honors ``OFPFF_CHECK_OVERLAP`` yes + + updates ``idle_timeout`` yes + + updates ``hard_timeout`` yes + + resets idle timer yes + + resets hard timer yes yes yes zeros counters yes + + may add a new flow yes # # sends ``flow_removed`` message --- --- --- % % ================================ === ====== ====== ====== ====== where: ``+`` "modify" and "modify-strict" only take these actions when they create a new flow, not when they update an existing flow. ``%`` "delete" and "delete_strict" generates a flow_removed message if the deleted flow or flows have the ``OFPFF_SEND_FLOW_REM`` flag set. (Each controller can separately control whether it wants to receive the generated messages.) ``#`` "modify" and "modify-strict" only add a new flow if the flow_mod does not match on any bits of the flow cookie OpenFlow 1.2 ~~~~~~~~~~~~ OpenFlow 1.2 makes these changes: - Only "add" commands ever add flows, "modify" and "modify-strict" never do. - A new flag ``OFPFF_RESET_COUNTS`` now controls whether "modify" and "modify-strict" reset counters, whereas previously they never reset counters (except when they inserted a new flow). ================================ === ====== ====== ====== ====== MODIFY DELETE RULE ADD MODIFY STRICT DELETE STRICT ================================ === ====== ====== ====== ====== match on ``priority`` yes --- yes --- yes match on ``out_port`` --- --- --- yes yes match on ``flow_cookie`` --- yes yes yes yes match on ``table_id`` yes yes yes yes yes controller chooses ``table_id`` yes yes yes updates ``flow_cookie`` yes --- --- updates ``OFPFF_SEND_FLOW_REM`` yes --- --- honors ``OFPFF_CHECK_OVERLAP`` yes --- --- updates ``idle_timeout`` yes --- --- updates ``hard_timeout`` yes --- --- resets idle timer yes --- --- resets hard timer yes yes yes zeros counters yes & & may add a new flow yes --- --- sends ``flow_removed`` message --- --- --- % % ================================ === ====== ====== ====== ====== ``%`` "delete" and "delete_strict" generates a flow_removed message if the deleted flow or flows have the ``OFPFF_SEND_FLOW_REM`` flag set. (Each controller can separately control whether it wants to receive the generated messages.) ``&`` "modify" and "modify-strict" reset counters if the ``OFPFF_RESET_COUNTS`` flag is specified. OpenFlow 1.3 ~~~~~~~~~~~~ OpenFlow 1.3 makes these changes: - Behavior for a table_id of 255 is now defined, for "delete" and "delete-strict" commands, as meaning to delete from all tables. A table_id of 255 is now explicitly invalid for other commands. - New flags ``OFPFF_NO_PKT_COUNTS`` and ``OFPFF_NO_BYT_COUNTS`` for "add" operations. The table for 1.3 is the same as the one shown above for 1.2. OpenFlow 1.4 ~~~~~~~~~~~~ OpenFlow 1.4 makes these changes: - Adds the "importance" field to ``flow_mods``, but it does not explicitly specify which kinds of ``flow_mods`` set the importance. For consistency, Open vSwitch uses the same rule for importance as for ``idle_timeout`` and ``hard_timeout``, that is, only an "ADD" flow_mod sets the importance. (This issue has been filed with the ONF as EXT-496.) .. TODO(stephenfin) Link to EXT-496 - Eviction Mechanism to automatically delete entries of lower importance to make space for newer entries. OpenFlow 1.4 Bundles -------------------- Open vSwitch makes all flow table modifications atomically, i.e., any datapath packet only sees flow table configurations either before or after any change made by any ``flow_mod``. For example, if a controller removes all flows with a single OpenFlow ``flow_mod``, no packet sees an intermediate version of the OpenFlow pipeline where only some of the flows have been deleted. It should be noted that Open vSwitch caches datapath flows, and that the cached flows are *NOT* flushed immediately when a flow table changes. Instead, the datapath flows are revalidated against the new flow table as soon as possible, and usually within one second of the modification. This design amortizes the cost of datapath cache flushing across multiple flow table changes, and has a significant performance effect during simultaneous heavy flow table churn and high traffic load. This means that different cached datapath flows may have been computed based on a different flow table configurations, but each of the datapath flows is guaranteed to have been computed over a coherent view of the flow tables, as described above. With OpenFlow 1.4 bundles this atomicity can be extended across an arbitrary set of ``flow_mod``. Bundles are supported for ``flow_mod`` and port_mod messages only. For ``flow_mod``, both ``atomic`` and ``ordered`` bundle flags are trivially supported, as all bundled messages are executed in the order they were added and all flow table modifications are now atomic to the datapath. Port mods may not appear in atomic bundles, as port status modifications are not atomic. To support bundles, ovs-ofctl has a ``--bundle`` option that makes the flow mod commands (``add-flow``, ``add-flows``, ``mod-flows``, ``del-flows``, and ``replace-flows``) use an OpenFlow 1.4 bundle to operate the modifications as a single atomic transaction. If any of the flow mods in a transaction fail, none of them are executed. All flow mods in a bundle appear to datapath lookups simultaneously. Furthermore, ovs-ofctl ``add-flow`` and ``add-flows`` commands now accept arbitrary flow mods as an input by allowing the flow specification to start with an explicit ``add``, ``modify``, ``modify_strict``, ``delete``, or ``delete_strict`` keyword. A missing keyword is treated as ``add``, so this is fully backwards compatible. With the new ``--bundle`` option all the flow mods are executed as a single atomic transaction using an OpenFlow 1.4 bundle. Without the ``--bundle`` option the flow mods are executed in order up to the first failing ``flow_mod``, and in case of an error the earlier successful ``flow_mod`` calls are not rolled back. ``OFPT_PACKET_IN`` ------------------ The OpenFlow 1.1 specification for ``OFPT_PACKET_IN`` is confusing. The definition in OF1.1 ``openflow.h`` is[*]: :: /* Packet received on port (datapath -> controller). */ struct ofp_packet_in { struct ofp_header header; uint32_t buffer_id; /* ID assigned by datapath. */ uint32_t in_port; /* Port on which frame was received. */ uint32_t in_phy_port; /* Physical Port on which frame was received. */ uint16_t total_len; /* Full length of frame. */ uint8_t reason; /* Reason packet is being sent (one of OFPR_*) */ uint8_t table_id; /* ID of the table that was looked up */ uint8_t data[0]; /* Ethernet frame, halfway through 32-bit word, so the IP header is 32-bit aligned. The amount of data is inferred from the length field in the header. Because of padding, offsetof(struct ofp_packet_in, data) == sizeof(struct ofp_packet_in) - 2. */ }; OFP_ASSERT(sizeof(struct ofp_packet_in) == 24); The confusing part is the comment on the ``data[]`` member. This comment is a leftover from OF1.0 ``openflow.h``, in which the comment was correct: ``sizeof(struct ofp_packet_in)`` is 20 in OF1.0 and ``ffsetof(struct ofp_packet_in, data)`` is 18. When OF1.1 was written, the structure members were changed but the comment was carelessly not updated, and the comment became wrong: ``sizeof(struct ofp_packet_in)`` and offsetof(struct ofp_packet_in, data) are both 24 in OF1.1. That leaves the question of how to implement ``ofp_packet_in`` in OF1.1. The OpenFlow reference implementation for OF1.1 does not include any padding, that is, the first byte of the encapsulated frame immediately follows the ``table_id`` member without a gap. Open vSwitch therefore implements it the same way for compatibility. For an earlier discussion, please see the thread archived at: https://mailman.stanford.edu/pipermail/openflow-discuss/2011-August/002604.html [*] The quoted definition is directly from OF1.1. Definitions used inside OVS omit the 8-byte ``ofp_header`` members, so the sizes in this discussion are 8 bytes larger than those declared in OVS header files. VLAN Matching ------------- The 802.1Q VLAN header causes more trouble than any other 4 bytes in networking. More specifically, three versions of OpenFlow and Open vSwitch have among them four different ways to match the contents and presence of the VLAN header. The following table describes how each version works. ======== ============= =============== =============== ================ Match NXM OF1.0 OF1.1 OF1.2 ======== ============= =============== =============== ================ ``[1]`` ``0000/0000`` ``????/1,??/?`` ``????/1,??/?`` ``0000/0000,--`` ``[2]`` ``0000/ffff`` ``ffff/0,??/?`` ``ffff/0,??/?`` ``0000/ffff,--`` ``[3]`` ``1xxx/1fff`` ``0xxx/0,??/1`` ``0xxx/0,??/1`` ``1xxx/ffff,--`` ``[4]`` ``z000/f000`` ``????/1,0y/0`` ``fffe/0,0y/0`` ``1000/1000,0y`` ``[5]`` ``zxxx/ffff`` ``0xxx/0,0y/0`` ``0xxx/0,0y/0`` ``1xxx/ffff,0y`` ``[6]`` ``0000/0fff`` ```` ```` ```` ``[7]`` ``0000/f000`` ```` ```` ```` ``[8]`` ``0000/efff`` ```` ```` ```` ``[9]`` ``1001/1001`` ```` ```` ``1001/1001,--`` ``[10]`` ``3000/3000`` ```` ```` ```` ``[11]`` ``1000/1000`` ```` ``fffe/0,??/1`` ``1000/1000,--`` ======== ============= =============== =============== ================ where: Match: See the list below. NXM: ``xxxx/yyyy`` means ``NXM_OF_VLAN_TCI_W`` with value ``xxxx`` and mask ``yyyy``. A mask of ``0000`` is equivalent to omitting ``NXM_OF_VLAN_TCI(_W)``, a mask of ``ffff`` is equivalent to ``NXM_OF_VLAN_TCI``. OF1.0, OF1.1: ``wwww/x,yy/z`` means ``dl_vlan`` ``wwww``, ``OFPFW_DL_VLAN`` ``x``, ``dl_vlan_pcp`` ``yy``, and ``OFPFW_DL_VLAN_PCP`` ``z``. If ``OFPFW_DL_VLAN`` or ``OFPFW_DL_VLAN_PCP`` is 1, the corresponding field value is wildcarded, otherwise it is matched. ``?`` means that the given bits are ignored (their conventional values are ``0000/x,00/0`` in OF1.0, ``0000/x,00/1`` in OF1.1; ``x`` is never ignored). ```` means that the given match is not supported. OF1.2: ``xxxx/yyyy,zz`` means ``OXM_OF_VLAN_VID_W`` with value ``xxxx`` and mask ``yyyy``, and ``OXM_OF_VLAN_PCP`` (which is not maskable) with value ``zz``. A mask of ``0000`` is equivalent to omitting ``OXM_OF_VLAN_VID(_W)``, a mask of ``ffff`` is equivalent to ``OXM_OF_VLAN_VID``. ``--`` means that ``OXM_OF_VLAN_PCP`` is omitted. ```` means that the given match is not supported. The matches are: ``[1]``: Matches any packet, that is, one without an 802.1Q header or with an 802.1Q header with any TCI value. ``[2]`` Matches only packets without an 802.1Q header. NXM: Any match with ``vlan_tci == 0`` and ``(vlan_tci_mask & 0x1000) != 0`` is equivalent to the one listed in the table. OF1.0: The spec doesn't define behavior if ``dl_vlan`` is set to ``0xffff`` and ``OFPFW_DL_VLAN_PCP`` is not set. OF1.1: The spec says explicitly to ignore ``dl_vlan_pcp`` when ``dl_vlan`` is set to ``0xffff``. OF1.2: The spec doesn't say what should happen if ``vlan_vid == 0`` and ``(vlan_vid_mask & 0x1000) != 0`` but ``vlan_vid_mask != 0x1000``, but it would be straightforward to also interpret as ``[2]``. ``[3]`` Matches only packets that have an 802.1Q header with VID ``xxx`` (and any PCP). ``[4]`` Matches only packets that have an 802.1Q header with PCP ``y`` (and any VID). NXM: ``z`` is ``(y << 1) | 1``. OF1.0: The spec isn't very clear, but OVS implements it this way. OF1.2: Presumably other masks such that ``(vlan_vid_mask & 0x1fff) == 0x1000`` would also work, but the spec doesn't define their behavior. ``[5]`` Matches only packets that have an 802.1Q header with VID ``xxx`` and PCP ``y``. NXM: ``z`` is ``((y << 1) | 1)``. OF1.2: Presumably other masks such that ``(vlan_vid_mask & 0x1fff) == 0x1fff`` would also work. ``[6]`` Matches packets with no 802.1Q header or with an 802.1Q header with a VID of 0. Only possible with NXM. ``[7]`` Matches packets with no 802.1Q header or with an 802.1Q header with a PCP of 0. Only possible with NXM. ``[8]`` Matches packets with no 802.1Q header or with an 802.1Q header with both VID and PCP of 0. Only possible with NXM. ``[9]`` Matches only packets that have an 802.1Q header with an odd-numbered VID (and any PCP). Only possible with NXM and OF1.2. (This is just an example; one can match on any desired VID bit pattern.) ``[10]`` Matches only packets that have an 802.1Q header with an odd-numbered PCP (and any VID). Only possible with NXM. (This is just an example; one can match on any desired VID bit pattern.) ``[11]`` Matches any packet with an 802.1Q header, regardless of VID or PCP. Additional notes: OF1.2: The top three bits of ``OXM_OF_VLAN_VID`` are fixed to zero, so bits 13, 14, and 15 in the masks listed in the table may be set to arbitrary values, as long as the corresponding value bits are also zero. The suggested ``ffff`` mask for [2], [3], and [5] allows a shorter OXM representation (the mask is omitted) than the minimal ``1fff`` mask. Flow Cookies ------------ OpenFlow 1.0 and later versions have the concept of a "flow cookie", which is a 64-bit integer value attached to each flow. The treatment of the flow cookie has varied greatly across OpenFlow versions, however. In OpenFlow 1.0: - ``OFPFC_ADD`` set the cookie in the flow that it added. - ``OFPFC_MODIFY`` and ``OFPFC_MODIFY_STRICT`` updated the cookie for the flow or flows that it modified. - ``OFPST_FLOW`` messages included the flow cookie. - ``OFPT_FLOW_REMOVED`` messages reported the cookie of the flow that was removed. OpenFlow 1.1 made the following changes: - Flow mod operations ``OFPFC_MODIFY``, ``OFPFC_MODIFY_STRICT``, ``OFPFC_DELETE``, and ``OFPFC_DELETE_STRICT``, plus flow stats requests and aggregate stats requests, gained the ability to match on flow cookies with an arbitrary mask. - ``OFPFC_MODIFY`` and ``OFPFC_MODIFY_STRICT`` were changed to add a new flow, in the case of no match, only if the flow table modification operation did not match on the cookie field. (In OpenFlow 1.0, modify operations always added a new flow when there was no match.) - ``OFPFC_MODIFY`` and ``OFPFC_MODIFY_STRICT`` no longer updated flow cookies. OpenFlow 1.2 made the following changes: - ``OFPC_MODIFY`` and ``OFPFC_MODIFY_STRICT`` were changed to never add a new flow, regardless of whether the flow cookie was used for matching. Open vSwitch support for OpenFlow 1.0 implements the OpenFlow 1.0 behavior with the following extensions: - An NXM extension field ``NXM_NX_COOKIE(_W)`` allows the NXM versions of ``OFPFC_MODIFY``, ``OFPFC_MODIFY_STRICT``, ``OFPFC_DELETE``, and ``OFPFC_DELETE_STRICT`` ``flow_mod`` calls, plus flow stats requests and aggregate stats requests, to match on flow cookies with arbitrary masks. This is much like the equivalent OpenFlow 1.1 feature. - Like OpenFlow 1.1, ``OFPC_MODIFY`` and ``OFPFC_MODIFY_STRICT`` add a new flow if there is no match and the mask is zero (or not given). - The ``cookie`` field in ``OFPT_FLOW_MOD`` and ``NXT_FLOW_MOD`` messages is used as the cookie value for ``OFPFC_ADD`` commands, as described in OpenFlow 1.0. For ``OFPFC_MODIFY`` and ``OFPFC_MODIFY_STRICT`` commands, the ``cookie`` field is used as a new cookie for flows that match unless it is ``UINT64_MAX``, in which case the flow's cookie is not updated. - ``NXT_PACKET_IN`` (the Nicira extended version of ``OFPT_PACKET_IN``) reports the cookie of the rule that generated the packet, or all-1-bits if no rule generated the packet. (Older versions of OVS used all-0-bits instead of all-1-bits.) The following table shows the handling of different protocols when receiving ``OFPFC_MODIFY`` and ``OFPFC_MODIFY_STRICT`` messages. A mask of 0 indicates either an explicit mask of zero or an implicit one by not specifying the ``NXM_NX_COOKIE(_W)`` field. ============== ====== ====== ============= ============= Match Update Add on miss Add on miss cookie cookie mask!=0 mask==0 ============== ====== ====== ============= ============= OpenFlow 1.0 no yes (add on miss) (add on miss) OpenFlow 1.1 yes no no yes OpenFlow 1.2 yes no no no NXM yes yes\* no yes ============== ====== ====== ============= ============= \* Updates the flow's cookie unless the ``cookie`` field is ``UINT64_MAX``. Multiple Table Support ---------------------- OpenFlow 1.0 has only rudimentary support for multiple flow tables. Notably, OpenFlow 1.0 does not allow the controller to specify the flow table to which a flow is to be added. Open vSwitch adds an extension for this purpose, which is enabled on a per-OpenFlow connection basis using the ``NXT_FLOW_MOD_TABLE_ID`` message. When the extension is enabled, the upper 8 bits of the ``command`` member in an ``OFPT_FLOW_MOD`` or ``NXT_FLOW_MOD`` message designates the table to which a flow is to be added. The Open vSwitch software switch implementation offers 255 flow tables. On packet ingress, only the first flow table (table 0) is searched, and the contents of the remaining tables are not considered in any way. Tables other than table 0 only come into play when an ``NXAST_RESUBMIT_TABLE`` action specifies another table to search. Tables 128 and above are reserved for use by the switch itself. Controllers should use only tables 0 through 127. ``OFPTC_*`` Table Configuration ------------------------------- This section covers the history of the ``OFPTC_*`` table configuration bits across OpenFlow versions. OpenFlow 1.0 flow tables had fixed configurations. OpenFlow 1.1 enabled controllers to configure behavior upon flow table miss and added the ``OFPTC_MISS_*`` constants for that purpose. ``OFPTC_*`` did not control anything else but it was nevertheless conceptualized as a set of bit-fields instead of an enum. OF1.1 added the ``OFPT_TABLE_MOD`` message to set ``OFPTC_MISS_*`` for a flow table and added the ``config`` field to the ``OFPST_TABLE`` reply to report the current setting. OpenFlow 1.2 did not change anything in this regard. OpenFlow 1.3 switched to another means to changing flow table miss behavior and deprecated ``OFPTC_MISS_*`` without adding any more ``OFPTC_*`` constants. This meant that ``OFPT_TABLE_MOD`` now had no purpose at all, but OF1.3 kept it around "for backward compatibility with older and newer versions of the specification." At the same time, OF1.3 introduced a new message OFPMP_TABLE_FEATURES that included a field ``config`` documented as reporting the ``OFPTC_*`` values set with ``OFPT_TABLE_MOD``; of course this served no real purpose because no ``OFPTC_*`` values are defined. OF1.3 did remove the ``OFPTC_*`` field from ``OFPMP_TABLE`` (previously named ``OFPST_TABLE``). OpenFlow 1.4 defined two new ``OFPTC_*`` constants, ``OFPTC_EVICTION`` and ``OFPTC_VACANCY_EVENTS``, using bits that did not overlap with ``OFPTC_MISS_*`` even though those bits had not been defined since OF1.2. ``OFPT_TABLE_MOD`` still controlled these settings. The field for ``OFPTC_*`` values in ``OFPMP_TABLE_FEATURES`` was renamed from ``config`` to ``capabilities`` and documented as reporting the flags that are supported in a ``OFPT_TABLE_MOD`` message. The ``OFPMP_TABLE_DESC`` message newly added in OF1.4 reported the ``OFPTC_*`` setting. OpenFlow 1.5 did not change anything in this regard. .. list-table:: Revisions :header-rows: 1 * - OpenFlow - ``OFPTC_*`` flags - ``TABLE_MOD`` - Statistics - ``TABLE_FEATURES`` - ``TABLE_DESC`` * - OF1.0 - none - no (\*)(+) - no (\*) - nothing (\*)(+) - no (\*)(+) * - OF1.1/1.2 - ``MISS_*`` - yes - yes - nothing (+) - no (+) * - OF1.3 - none - yes (\*) - no (\*) - config (\*) - no (\*)(+) * - OF1.4/1.5 - ``EVICTION``/``VACANCY_EVENTS`` - yes - no - capabilities - yes where: OpenFlow: The OpenFlow version(s). ``OFPTC_*`` flags: The ``OFPTC_*`` flags defined in those versions. ``TABLE_MOD``: Whether ``OFPT_TABLE_MOD`` can modify ``OFPTC_*`` flags. Statistics: Whether ``OFPST_TABLE/OFPMP_TABLE`` reports the ``OFPTC_*`` flags. ``TABLE_FEATURES``: What ``OFPMP_TABLE_FEATURES`` reports (if it exists): either the current configuration or the switch's capabilities. ``TABLE_DESC``: Whether ``OFPMP_TABLE_DESC`` reports the current configuration. (\*): Nothing to report/change anyway. (+): No such message. IPv6 ---- Open vSwitch supports stateless handling of IPv6 packets. Flows can be written to support matching TCP, UDP, and ICMPv6 headers within an IPv6 packet. Deeper matching of some Neighbor Discovery messages is also supported. IPv6 was not designed to interact well with middle-boxes. This, combined with Open vSwitch's stateless nature, have affected the processing of IPv6 traffic, which is detailed below. Extension Headers ~~~~~~~~~~~~~~~~~ The base IPv6 header is incredibly simple with the intention of only containing information relevant for routing packets between two endpoints. IPv6 relies heavily on the use of extension headers to provide any other functionality. Unfortunately, the extension headers were designed in such a way that it is impossible to move to the next header (including the layer-4 payload) unless the current header is understood. Open vSwitch will process the following extension headers and continue to the next header: - Fragment (see the next section) - AH (Authentication Header) - Hop-by-Hop Options - Routing - Destination Options When a header is encountered that is not in that list, it is considered "terminal". A terminal header's IPv6 protocol value is stored in ``nw_proto`` for matching purposes. If a terminal header is TCP, UDP, or ICMPv6, the packet will be further processed in an attempt to extract layer-4 information. Fragments ~~~~~~~~~ IPv6 requires that every link in the internet have an MTU of 1280 octets or greater (RFC 2460). As such, a terminal header (as described above in "Extension Headers") in the first fragment should generally be reachable. In this case, the terminal header's IPv6 protocol type is stored in the ``nw_proto`` field for matching purposes. If a terminal header cannot be found in the first fragment (one with a fragment offset of zero), the ``nw_proto`` field is set to 0. Subsequent fragments (those with a non-zero fragment offset) have the ``nw_proto`` field set to the IPv6 protocol type for fragments (44). Jumbograms ~~~~~~~~~~ An IPv6 jumbogram (RFC 2675) is a packet containing a payload longer than 65,535 octets. A jumbogram is only relevant in subnets with a link MTU greater than 65,575 octets, and are not required to be supported on nodes that do not connect to link with such large MTUs. Currently, Open vSwitch doesn't process jumbograms. In-Band Control --------------- Motivation ~~~~~~~~~~ An OpenFlow switch must establish and maintain a TCP network connection to its controller. There are two basic ways to categorize the network that this connection traverses: either it is completely separate from the one that the switch is otherwise controlling, or its path may overlap the network that the switch controls. We call the former case "out-of-band control", the latter case "in-band control". Out-of-band control has the following benefits: - Simplicity: Out-of-band control slightly simplifies the switch implementation. - Reliability: Excessive switch traffic volume cannot interfere with control traffic. - Integrity: Machines not on the control network cannot impersonate a switch or a controller. - Confidentiality: Machines not on the control network cannot snoop on control traffic. In-band control, on the other hand, has the following advantages: - No dedicated port: There is no need to dedicate a physical switch port to control, which is important on switches that have few ports (e.g. wireless routers, low-end embedded platforms). - No dedicated network: There is no need to build and maintain a separate control network. This is important in many environments because it reduces proliferation of switches and wiring. Open vSwitch supports both out-of-band and in-band control. This section describes the principles behind in-band control. See the description of the Controller table in ovs-vswitchd.conf.db(5) to configure OVS for in-band control. Principles ~~~~~~~~~~ The fundamental principle of in-band control is that an OpenFlow switch must recognize and switch control traffic without involving the OpenFlow controller. All the details of implementing in-band control are special cases of this principle. The rationale for this principle is simple. If the switch does not handle in-band control traffic itself, then it will be caught in a contradiction: it must contact the controller, but it cannot, because only the controller can set up the flows that are needed to contact the controller. The following points describe important special cases of this principle. - In-band control must be implemented regardless of whether the switch is connected. It is tempting to implement the in-band control rules only when the switch is not connected to the controller, using the reasoning that the controller should have complete control once it has established a connection with the switch. This does not work in practice. Consider the case where the switch is connected to the controller. Occasionally it can happen that the controller forgets or otherwise needs to obtain the MAC address of the switch. To do so, the controller sends a broadcast ARP request. A switch that implements the in-band control rules only when it is disconnected will then send an ``OFPT_PACKET_IN`` message up to the controller. The controller will be unable to respond, because it does not know the MAC address of the switch. This is a deadlock situation that can only be resolved by the switch noticing that its connection to the controller has hung and reconnecting. - In-band control must override flows set up by the controller. It is reasonable to assume that flows set up by the OpenFlow controller should take precedence over in-band control, on the basis that the controller should be in charge of the switch. Again, this does not work in practice. Reasonable controller implementations may set up a "last resort" fallback rule that wildcards every field and, e.g., sends it up to the controller or discards it. If a controller does that, then it will isolate itself from the switch. - The switch must recognize all control traffic. The fundamental principle of in-band control states, in part, that a switch must recognize control traffic without involving the OpenFlow controller. More specifically, the switch must recognize *all* control traffic. "False negatives", that is, packets that constitute control traffic but that the switch does not recognize as control traffic, lead to control traffic storms. Consider an OpenFlow switch that only recognizes control packets sent to or from that switch. Now suppose that two switches of this type, named A and B, are connected to ports on an Ethernet hub (not a switch) and that an OpenFlow controller is connected to a third hub port. In this setup, control traffic sent by switch A will be seen by switch B, which will send it to the controller as part of an OFPT_PACKET_IN message. Switch A will then see the OFPT_PACKET_IN message's packet, re-encapsulate it in another OFPT_PACKET_IN, and send it to the controller. Switch B will then see that OFPT_PACKET_IN, and so on in an infinite loop. Incidentally, the consequences of "false positives", where packets that are not control traffic are nevertheless recognized as control traffic, are much less severe. The controller will not be able to control their behavior, but the network will remain in working order. False positives do constitute a security problem. - The switch should use echo-requests to detect disconnection. TCP will notice that a connection has hung, but this can take a considerable amount of time. For example, with default settings the Linux kernel TCP implementation will retransmit for between 13 and 30 minutes, depending on the connection's retransmission timeout, according to kernel documentation. This is far too long for a switch to be disconnected, so an OpenFlow switch should implement its own connection timeout. OpenFlow ``OFPT_ECHO_REQUEST`` messages are the best way to do this, since they test the OpenFlow connection itself. Implementation ~~~~~~~~~~~~~~ This section describes how Open vSwitch implements in-band control. Correctly implementing in-band control has proven difficult due to its many subtleties, and has thus gone through many iterations. Please read through and understand the reasoning behind the chosen rules before making modifications. Open vSwitch implements in-band control as "hidden" flows, that is, flows that are not visible through OpenFlow, and at a higher priority than wildcarded flows can be set up through OpenFlow. This is done so that the OpenFlow controller cannot interfere with them and possibly break connectivity with its switches. It is possible to see all flows, including in-band ones, with the ovs-appctl "bridge/dump-flows" command. The Open vSwitch implementation of in-band control can hide traffic to arbitrary "remotes", where each remote is one TCP port on one IP address. Currently the remotes are automatically configured as the in-band OpenFlow controllers plus the OVSDB managers, if any. (The latter is a requirement because OVSDB managers are responsible for configuring OpenFlow controllers, so if the manager cannot be reached then OpenFlow cannot be reconfigured.) The following rules (with the OFPP_NORMAL action) are set up on any bridge that has any remotes: (a) DHCP requests sent from the local port. (b) ARP replies to the local port's MAC address. (c) ARP requests from the local port's MAC address. In-band also sets up the following rules for each unique next-hop MAC address for the remotes' IPs (the "next hop" is either the remote itself, if it is on a local subnet, or the gateway to reach the remote): (d) ARP replies to the next hop's MAC address. (e) ARP requests from the next hop's MAC address. In-band also sets up the following rules for each unique remote IP address: (f) ARP replies containing the remote's IP address as a target. (g) ARP requests containing the remote's IP address as a source. In-band also sets up the following rules for each unique remote (IP,port) pair: (h) TCP traffic to the remote's IP and port. (i) TCP traffic from the remote's IP and port. The goal of these rules is to be as narrow as possible to allow a switch to join a network and be able to communicate with the remotes. As mentioned earlier, these rules have higher priority than the controller's rules, so if they are too broad, they may prevent the controller from implementing its policy. As such, in-band actively monitors some aspects of flow and packet processing so that the rules can be made more precise. In-band control monitors attempts to add flows into the datapath that could interfere with its duties. The datapath only allows exact match entries, so in-band control is able to be very precise about the flows it prevents. Flows that miss in the datapath are sent to userspace to be processed, so preventing these flows from being cached in the "fast path" does not affect correctness. The only type of flow that is currently prevented is one that would prevent DHCP replies from being seen by the local port. For example, a rule that forwarded all DHCP traffic to the controller would not be allowed, but one that forwarded to all ports (including the local port) would. As mentioned earlier, packets that miss in the datapath are sent to the userspace for processing. The userspace has its own flow table, the "classifier", so in-band checks whether any special processing is needed before the classifier is consulted. If a packet is a DHCP response to a request from the local port, the packet is forwarded to the local port, regardless of the flow table. Note that this requires L7 processing of DHCP replies to determine whether the 'chaddr' field matches the MAC address of the local port. It is interesting to note that for an L3-based in-band control mechanism, the majority of rules are devoted to ARP traffic. At first glance, some of these rules appear redundant. However, each serves an important role. First, in order to determine the MAC address of the remote side (controller or gateway) for other ARP rules, we must allow ARP traffic for our local port with rules (b) and (c). If we are between a switch and its connection to the remote, we have to allow the other switch's ARP traffic to through. This is done with rules (d) and (e), since we do not know the addresses of the other switches a priori, but do know the remote's or gateway's. Finally, if the remote is running in a local guest VM that is not reached through the local port, the switch that is connected to the VM must allow ARP traffic based on the remote's IP address, since it will not know the MAC address of the local port that is sending the traffic or the MAC address of the remote in the guest VM. With a few notable exceptions below, in-band should work in most network setups. The following are considered "supported" in the current implementation: - Locally Connected. The switch and remote are on the same subnet. This uses rules (a), (b), (c), (h), and (i). - Reached through Gateway. The switch and remote are on different subnets and must go through a gateway. This uses rules (a), (b), (c), (h), and (i). - Between Switch and Remote. This switch is between another switch and the remote, and we want to allow the other switch's traffic through. This uses rules (d), (e), (h), and (i). It uses (b) and (c) indirectly in order to know the MAC address for rules (d) and (e). Note that DHCP for the other switch will not work unless an OpenFlow controller explicitly lets this switch pass the traffic. - Between Switch and Gateway. This switch is between another switch and the gateway, and we want to allow the other switch's traffic through. This uses the same rules and logic as the "Between Switch and Remote" configuration described earlier. - Remote on Local VM. The remote is a guest VM on the system running in-band control. This uses rules (a), (b), (c), (h), and (i). - Remote on Local VM with Different Networks. The remote is a guest VM on the system running in-band control, but the local port is not used to connect to the remote. For example, an IP address is configured on eth0 of the switch. The remote's VM is connected through eth1 of the switch, but an IP address has not been configured for that port on the switch. As such, the switch will use eth0 to connect to the remote, and eth1's rules about the local port will not work. In the example, the switch attached to eth0 would use rules (a), (b), (c), (h), and (i) on eth0. The switch attached to eth1 would use rules (f), (g), (h), and (i). The following are explicitly *not* supported by in-band control: - Specify Remote by Name. Currently, the remote must be identified by IP address. A naive approach would be to permit all DNS traffic. Unfortunately, this would prevent the controller from defining any policy over DNS. Since switches that are located behind us need to connect to the remote, in-band cannot simply add a rule that allows DNS traffic from the local port. The "correct" way to support this is to parse DNS requests to allow all traffic related to a request for the remote's name through. Due to the potential security problems and amount of processing, we decided to hold off for the time-being. - Differing Remotes for Switches. All switches must know the L3 addresses for all the remotes that other switches may use, since rules need to be set up to allow traffic related to those remotes through. See rules (f), (g), (h), and (i). - Differing Routes for Switches. In order for the switch to allow other switches to connect to a remote through a gateway, it allows the gateway's traffic through with rules (d) and (e). If the routes to the remote differ for the two switches, we will not know the MAC address of the alternate gateway. Action Reproduction ------------------- It seems likely that many controllers, at least at startup, use the OpenFlow "flow statistics" request to obtain existing flows, then compare the flows' actions against the actions that they expect to find. Before version 1.8.0, Open vSwitch always returned exact, byte-for-byte copies of the actions that had been added to the flow table. The current version of Open vSwitch does not always do this in some exceptional cases. This section lists the exceptions that controller authors must keep in mind if they compare actual actions against desired actions in a bytewise fashion: - Open vSwitch zeros padding bytes in action structures, regardless of their values when the flows were added. - Open vSwitch "normalizes" the instructions in OpenFlow 1.1 (and later) in the following way: * OVS sorts the instructions into the following order: Apply-Actions, Clear-Actions, Write-Actions, Write-Metadata, Goto-Table. * OVS drops Apply-Actions instructions that have empty action lists. * OVS drops Write-Actions instructions that have empty action sets. Please report other discrepancies, if you notice any, so that we can fix or document them. Suggestions ----------- Suggestions to improve Open vSwitch are welcome at discuss@openvswitch.org. ovs-2.9.0/Documentation/topics/dpdk/000077500000000000000000000000001324262074100173665ustar00rootroot00000000000000ovs-2.9.0/Documentation/topics/dpdk/index.rst000066400000000000000000000017261324262074100212350ustar00rootroot00000000000000.. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. Convention for heading levels in Open vSwitch documentation: ======= Heading 0 (reserved for the title in a document) ------- Heading 1 ~~~~~~~ Heading 2 +++++++ Heading 3 ''''''' Heading 4 Avoid deeper levels because they do not render well. ================= The DPDK Datapath ================= .. toctree:: :maxdepth: 2 vhost-user ring ovs-2.9.0/Documentation/topics/dpdk/ring.rst000066400000000000000000000055011324262074100210600ustar00rootroot00000000000000.. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. Convention for heading levels in Open vSwitch documentation: ======= Heading 0 (reserved for the title in a document) ------- Heading 1 ~~~~~~~ Heading 2 +++++++ Heading 3 ''''''' Heading 4 Avoid deeper levels because they do not render well. =============== DPDK Ring Ports =============== .. warning:: DPDK ring interfaces cannot be used for guest communication and are retained mainly for backwards compatibility purposes. In nearly all cases, :doc:`vhost-user ports ` are a better choice and should be used instead. The DPDK datapath provides DPDK-backed ring ports that are implemented using DPDK's ``librte_ring`` library. For more information on this library, refer to the `DPDK documentation`_. Quick Example ------------- This example demonstrates how to add a ``dpdkr`` port to an existing bridge called ``br0``:: $ ovs-vsctl add-port br0 dpdkr0 -- set Interface dpdkr0 type=dpdkr dpdkr ----- To use ring ports, you must first add said ports to the switch. Unlike :doc:`vhost-user ports `, ring port names must take a specific format, ``dpdkrNN``, where ``NN`` is the port ID. For example:: $ ovs-vsctl add-port br0 dpdkr0 -- set Interface dpdkr0 type=dpdkr Once the port has been added to the switch, they can be used by host processes. A sample loopback application - ``test-dpdkr`` - is included with Open vSwitch. To use this, run the following:: $ ./tests/test-dpdkr -c 1 -n 4 --proc-type=secondary -- -n 0 Further functionality would require developing your own application. Refer to the `DPDK documentation`_ for more information on how to do this. Adding dpdkr ports to the guest ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ It is **not** recommended to use ring ports from guests. Historically, this was possible using a patched version of QEMU and the IVSHMEM feature provided with DPDK. However, this functionality was removed because: - The IVSHMEM library was removed from DPDK in DPDK 16.11 - Support for IVSHMEM was never upstreamed to QEMU and has been publicly rejected by the QEMU community - :doc:`vhost-user interfaces ` are the defacto DPDK-based path to guests .. _DPDK documentation: https://dpdk.readthedocs.io/en/v17.11/prog_guide/ring_lib.html ovs-2.9.0/Documentation/topics/dpdk/vhost-user.rst000066400000000000000000000500401324262074100222360ustar00rootroot00000000000000.. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. Convention for heading levels in Open vSwitch documentation: ======= Heading 0 (reserved for the title in a document) ------- Heading 1 ~~~~~~~ Heading 2 +++++++ Heading 3 ''''''' Heading 4 Avoid deeper levels because they do not render well. ===================== DPDK vHost User Ports ===================== The DPDK datapath provides DPDK-backed vHost user ports as a primary way to interact with guests. For more information on vHost User, refer to the `QEMU documentation`_ on same. Quick Example ------------- This example demonstrates how to add two ``dpdkvhostuserclient`` ports to an existing bridge called ``br0``:: $ ovs-vsctl add-port br0 dpdkvhostclient0 \ -- set Interface dpdkvhostclient0 type=dpdkvhostuserclient \ options:vhost-server-path=/tmp/dpdkvhostclient0 $ ovs-vsctl add-port br0 dpdkvhostclient1 \ -- set Interface dpdkvhostclient1 type=dpdkvhostuserclient \ options:vhost-server-path=/tmp/dpdkvhostclient1 For the above examples to work, an appropriate server socket must be created at the paths specified (``/tmp/dpdkvhostclient0`` and ``/tmp/dpdkvhostclient1``). These sockets can be created with QEMU; see the :ref:`vhost-user client ` section for details. vhost-user vs. vhost-user-client -------------------------------- Open vSwitch provides two types of vHost User ports: - vhost-user (``dpdkvhostuser``) - vhost-user-client (``dpdkvhostuserclient``) vHost User uses a client-server model. The server creates/manages/destroys the vHost User sockets, and the client connects to the server. Depending on which port type you use, ``dpdkvhostuser`` or ``dpdkvhostuserclient``, a different configuration of the client-server model is used. For vhost-user ports, Open vSwitch acts as the server and QEMU the client. This means if OVS dies, all VMs **must** be restarted. On the other hand, for vhost-user-client ports, OVS acts as the client and QEMU the server. This means OVS can die and be restarted without issue, and it is also possible to restart an instance itself. For this reason, vhost-user-client ports are the preferred type for all known use cases; the only limitation is that vhost-user client mode ports require QEMU version 2.7. Ports of type vhost-user are currently deprecated and will be removed in a future release. .. _dpdk-vhost-user: vhost-user ---------- .. important:: Use of vhost-user ports requires QEMU >= 2.2; vhost-user ports are *deprecated*. To use vhost-user ports, you must first add said ports to the switch. DPDK vhost-user ports can have arbitrary names with the exception of forward and backward slashes, which are prohibited. For vhost-user, the port type is ``dpdkvhostuser``:: $ ovs-vsctl add-port br0 vhost-user-1 -- set Interface vhost-user-1 \ type=dpdkvhostuser This action creates a socket located at ``/usr/local/var/run/openvswitch/vhost-user-1``, which you must provide to your VM on the QEMU command line. .. note:: If you wish for the vhost-user sockets to be created in a sub-directory of ``/usr/local/var/run/openvswitch``, you may specify this directory in the ovsdb like so:: $ ovs-vsctl --no-wait \ set Open_vSwitch . other_config:vhost-sock-dir=subdir Once the vhost-user ports have been added to the switch, they must be added to the guest. There are two ways to do this: using QEMU directly, or using libvirt. Adding vhost-user ports to the guest (QEMU) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ To begin, you must attach the vhost-user device sockets to the guest. To do this, you must pass the following parameters to QEMU:: -chardev socket,id=char1,path=/usr/local/var/run/openvswitch/vhost-user-1 -netdev type=vhost-user,id=mynet1,chardev=char1,vhostforce -device virtio-net-pci,mac=00:00:00:00:00:01,netdev=mynet1 where ``vhost-user-1`` is the name of the vhost-user port added to the switch. Repeat the above parameters for multiple devices, changing the chardev ``path`` and ``id`` as necessary. Note that a separate and different chardev ``path`` needs to be specified for each vhost-user device. For example you have a second vhost-user port named ``vhost-user-2``, you append your QEMU command line with an additional set of parameters:: -chardev socket,id=char2,path=/usr/local/var/run/openvswitch/vhost-user-2 -netdev type=vhost-user,id=mynet2,chardev=char2,vhostforce -device virtio-net-pci,mac=00:00:00:00:00:02,netdev=mynet2 In addition, QEMU must allocate the VM's memory on hugetlbfs. vhost-user ports access a virtio-net device's virtual rings and packet buffers mapping the VM's physical memory on hugetlbfs. To enable vhost-user ports to map the VM's memory into their process address space, pass the following parameters to QEMU:: -object memory-backend-file,id=mem,size=4096M,mem-path=/dev/hugepages,share=on -numa node,memdev=mem -mem-prealloc Finally, you may wish to enable multiqueue support. This is optional but, should you wish to enable it, run:: -chardev socket,id=char2,path=/usr/local/var/run/openvswitch/vhost-user-2 -netdev type=vhost-user,id=mynet2,chardev=char2,vhostforce,queues=$q -device virtio-net-pci,mac=00:00:00:00:00:02,netdev=mynet2,mq=on,vectors=$v where: ``$q`` The number of queues ``$v`` The number of vectors, which is ``$q`` * 2 + 2 The vhost-user interface will be automatically reconfigured with required number of rx and tx queues after connection of virtio device. Manual configuration of ``n_rxq`` is not supported because OVS will work properly only if ``n_rxq`` will match number of queues configured in QEMU. A least 2 PMDs should be configured for the vswitch when using multiqueue. Using a single PMD will cause traffic to be enqueued to the same vhost queue rather than being distributed among different vhost queues for a vhost-user interface. If traffic destined for a VM configured with multiqueue arrives to the vswitch via a physical DPDK port, then the number of rxqs should also be set to at least 2 for that physical DPDK port. This is required to increase the probability that a different PMD will handle the multiqueue transmission to the guest using a different vhost queue. If one wishes to use multiple queues for an interface in the guest, the driver in the guest operating system must be configured to do so. It is recommended that the number of queues configured be equal to ``$q``. For example, this can be done for the Linux kernel virtio-net driver with:: $ ethtool -L combined <$q> where: ``-L`` Changes the numbers of channels of the specified network device ``combined`` Changes the number of multi-purpose channels. Adding vhost-user ports to the guest (libvirt) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. TODO(stephenfin): This seems like something that wouldn't be acceptable in production. Is this really required? To begin, you must change the user and group that libvirt runs under, configure access control policy and restart libvirtd. - In ``/etc/libvirt/qemu.conf`` add/edit the following lines:: user = "root" group = "root" - Disable SELinux or set to permissive mode:: $ setenforce 0 - Finally, restart the libvirtd process, For example, on Fedora:: $ systemctl restart libvirtd.service Once complete, instantiate the VM. A sample XML configuration file is provided at the :ref:`end of this file `. Save this file, then create a VM using this file:: $ virsh create demovm.xml Once created, you can connect to the guest console:: $ virsh console demovm The demovm xml configuration is aimed at achieving out of box performance on VM. These enhancements include: - The vcpus are pinned to the cores of the CPU socket 0 using ``vcpupin``. - Configure NUMA cell and memory shared using ``memAccess='shared'``. - Disable ``mrg_rxbuf='off'`` Refer to the `libvirt documentation `__ for more information. .. _dpdk-vhost-user-client: vhost-user-client ----------------- .. important:: Use of vhost-user ports requires QEMU >= 2.7 To use vhost-user-client ports, you must first add said ports to the switch. Like DPDK vhost-user ports, DPDK vhost-user-client ports can have mostly arbitrary names. However, the name given to the port does not govern the name of the socket device. Instead, this must be configured by the user by way of a ``vhost-server-path`` option. For vhost-user-client, the port type is ``dpdkvhostuserclient``:: $ VHOST_USER_SOCKET_PATH=/path/to/socket $ ovs-vsctl add-port br0 vhost-client-1 \ -- set Interface vhost-client-1 type=dpdkvhostuserclient \ options:vhost-server-path=$VHOST_USER_SOCKET_PATH Once the vhost-user-client ports have been added to the switch, they must be added to the guest. Like vhost-user ports, there are two ways to do this: using QEMU directly, or using libvirt. Only the QEMU case is covered here. Adding vhost-user-client ports to the guest (QEMU) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Attach the vhost-user device sockets to the guest. To do this, you must pass the following parameters to QEMU:: -chardev socket,id=char1,path=$VHOST_USER_SOCKET_PATH,server -netdev type=vhost-user,id=mynet1,chardev=char1,vhostforce -device virtio-net-pci,mac=00:00:00:00:00:01,netdev=mynet1 where ``vhost-user-1`` is the name of the vhost-user port added to the switch. If the corresponding ``dpdkvhostuserclient`` port has not yet been configured in OVS with ``vhost-server-path=/path/to/socket``, QEMU will print a log similar to the following:: QEMU waiting for connection on: disconnected:unix:/path/to/socket,server QEMU will wait until the port is created sucessfully in OVS to boot the VM. One benefit of using this mode is the ability for vHost ports to 'reconnect' in event of the switch crashing or being brought down. Once it is brought back up, the vHost ports will reconnect automatically and normal service will resume. vhost-user-client IOMMU Support ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ vhost IOMMU is a feature which restricts the vhost memory that a virtio device can access, and as such is useful in deployments in which security is a concern. IOMMU support may be enabled via a global config value, ```vhost-iommu-support```. Setting this to true enables vhost IOMMU support for all vhost ports when/where available:: $ ovs-vsctl set Open_vSwitch . other_config:vhost-iommu-support=true The default value is false. .. important:: Changing this value requires restarting the daemon. .. important:: Enabling the IOMMU feature also enables the vhost user reply-ack protocol; this is known to work on QEMU v2.10.0, but is buggy on older versions (2.7.0 - 2.9.0, inclusive). Consequently, the IOMMU feature is disabled by default (and should remain so if using the aforementioned versions of QEMU). Starting with QEMU v2.9.1, vhost-iommu-support can safely be enabled, even without having an IOMMU device, with no performance penalty. .. _dpdk-testpmd: DPDK in the Guest ----------------- The DPDK ``testpmd`` application can be run in guest VMs for high speed packet forwarding between vhostuser ports. DPDK and testpmd application has to be compiled on the guest VM. Below are the steps for setting up the testpmd application in the VM. .. note:: Support for DPDK in the guest requires QEMU >= 2.2 To begin, instantiate a guest as described in :ref:`dpdk-vhost-user` or :ref:`dpdk-vhost-user-client`. Once started, connect to the VM, download the DPDK sources to VM and build DPDK:: $ cd /root/dpdk/ $ wget http://fast.dpdk.org/rel/dpdk-17.11.tar.xz $ tar xf dpdk-17.11.tar.xz $ export DPDK_DIR=/root/dpdk/dpdk-17.11 $ export DPDK_TARGET=x86_64-native-linuxapp-gcc $ export DPDK_BUILD=$DPDK_DIR/$DPDK_TARGET $ cd $DPDK_DIR $ make install T=$DPDK_TARGET DESTDIR=install Build the test-pmd application:: $ cd app/test-pmd $ export RTE_SDK=$DPDK_DIR $ export RTE_TARGET=$DPDK_TARGET $ make Setup huge pages and DPDK devices using UIO:: $ sysctl vm.nr_hugepages=1024 $ mkdir -p /dev/hugepages $ mount -t hugetlbfs hugetlbfs /dev/hugepages # only if not already mounted $ modprobe uio $ insmod $DPDK_BUILD/kmod/igb_uio.ko $ $DPDK_DIR/usertools/dpdk-devbind.py --status $ $DPDK_DIR/usertools/dpdk-devbind.py -b igb_uio 00:03.0 00:04.0 .. note:: vhost ports pci ids can be retrieved using:: lspci | grep Ethernet Finally, start the application:: # TODO .. important:: DPDK v17.11 virtio PMD contains a bug in the vectorized Rx function that affects testpmd/DPDK guest applications. As such, guest DPDK applications should use a non-vectorized Rx function. The DPDK v17.11 virtio net driver contains a bug that prevents guest DPDK applications from receiving packets when the vectorized Rx function is used. This only occurs when guest-bound traffic is live before a DPDK application is started within the guest, and where two or more forwarding cores are used. As such, it is not recommended for guests which execute DPDK applications to use the virtio vectorized Rx function. A simple method of ensuring that a non- vectorized Rx function is used is to enable mergeable buffers for the guest, with the following QEMU command line option:: mrg_rxbuf=on Additional details regarding the virtio driver bug are available on the `DPDK mailing list`_. .. _DPDK mailing list: http://dpdk.org/ml/archives/dev/2017-December/082801.html .. _dpdk-vhost-user-xml: Sample XML ---------- :: demovm 4a9b3f53-fa2a-47f3-a757-dd87720d9d1d 4194304 4194304 2 4096 hvm destroy restart destroy /usr/bin/qemu-kvm .. _QEMU documentation: http://git.qemu-project.org/?p=qemu.git;a=blob;f=docs/specs/vhost-user.txt;h=7890d7169;hb=HEAD vhost-user Dequeue Zero Copy (experimental) ------------------------------------------- Normally when dequeuing a packet from a vHost User device, a memcpy operation must be used to copy that packet from guest address space to host address space. This memcpy can be removed by enabling dequeue zero-copy like so:: $ ovs-vsctl add-port br0 dpdkvhostuserclient0 -- set Interface \ dpdkvhostuserclient0 type=dpdkvhostuserclient \ options:vhost-server-path=/tmp/dpdkvhostclient0 \ options:dq-zero-copy=true With this feature enabled, a reference (pointer) to the packet is passed to the host, instead of a copy of the packet. Removing this memcpy can give a performance improvement for some use cases, for example switching large packets between different VMs. However additional packet loss may be observed. Note that the feature is disabled by default and must be explicitly enabled by setting the ``dq-zero-copy`` option to ``true`` while specifying the ``vhost-server-path`` option as above. If you wish to split out the command into multiple commands as below, ensure ``dq-zero-copy`` is set before ``vhost-server-path``:: $ ovs-vsctl set Interface dpdkvhostuserclient0 options:dq-zero-copy=true $ ovs-vsctl set Interface dpdkvhostuserclient0 \ options:vhost-server-path=/tmp/dpdkvhostclient0 The feature is only available to ``dpdkvhostuserclient`` port types. A limitation exists whereby if packets from a vHost port with ``dq-zero-copy=true`` are destined for a ``dpdk`` type port, the number of tx descriptors (``n_txq_desc``) for that port must be reduced to a smaller number, 128 being the recommended value. This can be achieved by issuing the following command:: $ ovs-vsctl set Interface dpdkport options:n_txq_desc=128 Note: The sum of the tx descriptors of all ``dpdk`` ports the VM will send to should not exceed 128. For example, in case of a bond over two physical ports in balance-tcp mode, one must divide 128 by the number of links in the bond. Refer to :ref:`dpdk-queues-sizes` for more information. The reason for this limitation is due to how the zero copy functionality is implemented. The vHost device's 'tx used vring', a virtio structure used for tracking used ie. sent descriptors, will only be updated when the NIC frees the corresponding mbuf. If we don't free the mbufs frequently enough, that vring will be starved and packets will no longer be processed. One way to ensure we don't encounter this scenario, is to configure ``n_txq_desc`` to a small enough number such that the 'mbuf free threshold' for the NIC will be hit more often and thus free mbufs more frequently. The value of 128 is suggested, but values of 64 and 256 have been tested and verified to work too, with differing performance characteristics. A value of 512 can be used too, if the virtio queue size in the guest is increased to 1024 (available to configure in QEMU versions v2.10 and greater). This value can be set like so:: $ qemu-system-x86_64 ... -chardev socket,id=char1,path=,server -netdev type=vhost-user,id=mynet1,chardev=char1,vhostforce -device virtio-net-pci,mac=00:00:00:00:00:01,netdev=mynet1, tx_queue_size=1024 Because of this limitation, this feature is considered 'experimental'. The feature currently does not fully work with QEMU >= v2.7 due to a bug in DPDK which will be addressed in an upcoming release. The patch to fix this issue can be found on `Patchwork `__ Further information can be found in the `DPDK documentation `__ ovs-2.9.0/Documentation/topics/high-availability.rst000066400000000000000000000500341324262074100225670ustar00rootroot00000000000000.. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. Convention for heading levels in Open vSwitch documentation: ======= Heading 0 (reserved for the title in a document) ------- Heading 1 ~~~~~~~ Heading 2 +++++++ Heading 3 ''''''' Heading 4 Avoid deeper levels because they do not render well. ================================== OVN Gateway High Availability Plan ================================== :: OVN Gateway +---------------------------+ | | | External Network | | | +-------------^-------------+ | | +-----------+ | | | Gateway | | | +-----------+ ^ | | +-------------v-------------+ | | | OVN Virtual Network | | | +---------------------------+ The OVN gateway is responsible for shuffling traffic between the tunneled overlay network (governed by ovn-northd), and the legacy physical network. In a naive implementation, the gateway is a single x86 server, or hardware VTEP. For most deployments, a single system has enough forwarding capacity to service the entire virtualized network, however, it introduces a single point of failure. If this system dies, the entire OVN deployment becomes unavailable. To mitigate this risk, an HA solution is critical -- by spreading responsibility across multiple systems, no single server failure can take down the network. An HA solution is both critical to the manageability of the system, and extremely difficult to get right. The purpose of this document, is to propose a plan for OVN Gateway High Availability which takes into account our past experience building similar systems. It should be considered a fluid changing proposal, not a set-in-stone decree. .. note:: This document describes a range of options OVN could take to provide high availability for gateways. The current implementation provides L3 gateway high availability by the "Router Specific Active/Backup" approach described in this document. Basic Architecture ------------------ In an OVN deployment, the set of hypervisors and network elements operating under the guidance of ovn-northd are in what's called "logical space". These servers use VXLAN, STT, or Geneve to communicate, oblivious to the details of the underlying physical network. When these systems need to communicate with legacy networks, traffic must be routed through a Gateway which translates from OVN controlled tunnel traffic, to raw physical network traffic. Since the gateway is typically the only system with a connection to the physical network all traffic between logical space and the WAN must travel through it. This makes it a critical single point of failure -- if the gateway dies, communication with the WAN ceases for all systems in logical space. To mitigate this risk, multiple gateways should be run in a "High Availability Cluster" or "HA Cluster". The HA cluster will be responsible for performing the duties of a gateways, while being able to recover gracefully from individual member failures. :: OVN Gateway HA Cluster +---------------------------+ | | | External Network | | | +-------------^-------------+ | | +----------------------v----------------------+ | | | High Availability Cluster | | | | +-----------+ +-----------+ +-----------+ | | | | | | | | | | | Gateway | | Gateway | | Gateway | | | | | | | | | | | +-----------+ +-----------+ +-----------+ | +----------------------^----------------------+ | | +-------------v-------------+ | | | OVN Virtual Network | | | +---------------------------+ L2 vs L3 High Availability ~~~~~~~~~~~~~~~~~~~~~~~~~~ In order to achieve this goal, there are two broad approaches one can take. The HA cluster can appear to the network like a giant Layer 2 Ethernet Switch, or like a giant IP Router. These approaches are called L2HA, and L3HA respectively. L2HA allows ethernet broadcast domains to extend into logical space, a significant advantage, but this comes at a cost. The need to avoid transient L2 loops during failover significantly complicates their design. On the other hand, L3HA works for most use cases, is simpler, and fails more gracefully. For these reasons, it is suggested that OVN supports an L3HA model, leaving L2HA for future work (or third party VTEP providers). Both models are discussed further below. L3HA ---- In this section, we'll work through a basic simple L3HA implementation, on top of which we'll gradually build more sophisticated features explaining their motivations and implementations as we go. Naive active-backup ~~~~~~~~~~~~~~~~~~~ Let's assume that there are a collection of logical routers which a tenant has asked for, our task is to schedule these logical routers on one of N gateways, and gracefully redistribute the routers on gateways which have failed. The absolute simplest way to achieve this is what we'll call "naive-active-backup". :: Naive Active Backup HA Implementation +----------------+ +----------------+ | Leader | | Backup | | | | | | A B C | | | | | | | +----+-+-+-+----++ +-+--------------+ ^ ^ ^ ^ | | | | | | | | | | | | +-+------+---+ + + + + | ovn-northd | Traffic +------------+ In a naive active-backup, one of the Gateways is chosen (arbitrarily) as a leader. All logical routers (A, B, C in the figure), are scheduled on this leader gateway and all traffic flows through it. ovn-northd monitors this gateway via OpenFlow echo requests (or some equivalent), and if the gateway dies, it recreates the routers on one of the backups. This approach basically works in most cases and should likely be the starting point for OVN -- it's strictly better than no HA solution and is a good foundation for more sophisticated solutions. That said, it's not without it's limitations. Specifically, this approach doesn't coordinate with the physical network to minimize disruption during failures, and it tightly couples failover to ovn-northd (we'll discuss why this is bad in a bit), and wastes resources by leaving backup gateways completely unutilized. Router Failover +++++++++++++++ When ovn-northd notices the leader has died and decides to migrate routers to a backup gateway, the physical network has to be notified to direct traffic to the new gateway. Otherwise, traffic could be blackholed for longer than necessary making failovers worse than they need to be. For now, let's assume that OVN requires all gateways to be on the same IP subnet on the physical network. If this isn't the case, gateways would need to participate in routing protocols to orchestrate failovers, something which is difficult and out of scope of this document. Since all gateways are on the same IP subnet, we simply need to worry about updating the MAC learning tables of the Ethernet switches on that subnet. Presumably, they all have entries for each logical router pointing to the old leader. If these entries aren't updated, all traffic will be sent to the (now defunct) old leader, instead of the new one. In order to mitigate this issue, it's recommended that the new gateway sends a Reverse ARP (RARP) onto the physical network for each logical router it now controls. A Reverse ARP is a benign protocol used by many hypervisors when virtual machines migrate to update L2 forwarding tables. In this case, the ethernet source address of the RARP is that of the logical router it corresponds to, and its destination is the broadcast address. This causes the RARP to travel to every L2 switch in the broadcast domain, updating forwarding tables accordingly. This strategy is recommended in all failover mechanisms discussed in this document -- when a router newly boots on a new leader, it should RARP its MAC address. Controller Independent Active-backup ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ :: Controller Independent Active-Backup Implementation +----------------+ +----------------+ | Leader | | Backup | | | | | | A B C | | | | | | | +----------------+ +----------------+ ^ ^ ^ ^ | | | | | | | | + + + + Traffic The fundamental problem with naive active-backup, is it tightly couples the failover solution to ovn-northd. This can significantly increase downtime in the event of a failover as the (often already busy) ovn-northd controller has to recompute state for the new leader. Worse, if ovn-northd goes down, we can't perform gateway failover at all. This violates the principle that control plane outages should have no impact on dataplane functionality. In a controller independent active-backup configuration, ovn-northd is responsible for initial configuration while the HA cluster is responsible for monitoring the leader, and failing over to a backup if necessary. ovn-northd sets HA policy, but doesn't actively participate when failovers occur. Of course, in this model, ovn-northd is not without some responsibility. Its role is to pre-plan what should happen in the event of a failure, leaving it to the individual switches to execute this plan. It does this by assigning each gateway a unique leadership priority. Once assigned, it communicates this priority to each node it controls. Nodes use the leadership priority to determine which gateway in the cluster is the active leader by using a simple metric: the leader is the gateway that is healthy, with the highest priority. If that gateway goes down, leadership falls to the next highest priority, and conversely, if a new gateway comes up with a higher priority, it takes over leadership. Thus, in this model, leadership of the HA cluster is determined simply by the status of its members. Therefore if we can communicate the status of each gateway to each transport node, they can individually figure out which is the leader, and direct traffic accordingly. Tunnel Monitoring +++++++++++++++++ Since in this model leadership is determined exclusively by the health status of member gateways, a key problem is how do we communicate this information to the relevant transport nodes. Luckily, we can do this fairly cheaply using tunnel monitoring protocols like BFD. The basic idea is pretty straightforward. Each transport node maintains a tunnel to every gateway in the HA cluster (not just the leader). These tunnels are monitored using the BFD protocol to see which are alive. Given this information, hypervisors can trivially compute the highest priority live gateway, and thus the leader. In practice, this leadership computation can be performed trivially using the bundle or group action. Rather than using OpenFlow to simply output to the leader, all gateways could be listed in an active-backup bundle action ordered by their priority. The bundle action will automatically take into account the tunnel monitoring status to output the packet to the highest priority live gateway. Inter-Gateway Monitoring ++++++++++++++++++++++++ One somewhat subtle aspect of this model, is that failovers are not globally atomic. When a failover occurs, it will take some time for all hypervisors to notice and adjust accordingly. Similarly, if a new high priority Gateway comes up, it may take some time for all hypervisors to switch over to the new leader. In order to avoid confusing the physical network, under these circumstances it's important for the backup gateways to drop traffic they've received erroneously. In order to do this, each Gateway must know whether or not it is, in fact active. This can be achieved by creating a mesh of tunnels between gateways. Each gateway monitors the other gateways its cluster to determine which are alive, and therefore whether or not that gateway happens to be the leader. If leading, the gateway forwards traffic normally, otherwise it drops all traffic. We should note that this method works well under the assumption that there are no inter-gateway connectivity failures, in such case this method would fail to elect a single master. The simplest example is two gateways which stop seeing each other but can still reach the hypervisors. Protocols like VRRP or CARP have the same issue. A mitigation for this type of failure mode could be achieved by having all network elements (hypervisors and gateways) periodically share their link status to other endpoints. Gateway Leadership Resignation ++++++++++++++++++++++++++++++ Sometimes a gateway may be healthy, but still may not be suitable to lead the HA cluster. This could happen for several reasons including: * The physical network is unreachable * BFD (or ping) has detected the next hop router is unreachable * The Gateway recently booted and isn't fully configured In this case, the Gateway should resign leadership by holding its tunnels down using the ``other_config:cpath_down`` flag. This indicates to participating hypervisors and Gateways that this gateway should be treated as if it's down, even though its tunnels are still healthy. Router Specific Active-Backup ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ :: Router Specific Active-Backup +----------------+ +----------------+ | | | | | A C | | B D E | | | | | +----------------+ +----------------+ ^ ^ ^ ^ | | | | | | | | + + + + Traffic Controller independent active-backup is a great advance over naive active-backup, but it still has one glaring problem -- it under-utilizes the backup gateways. In ideal scenario, all traffic would split evenly among the live set of gateways. Getting all the way there is somewhat tricky, but as a step in the direction, one could use the "Router Specific Active-Backup" algorithm. This algorithm looks a lot like active-backup on a per logical router basis, with one twist. It chooses a different active Gateway for each logical router. Thus, in situations where there are several logical routers, all with somewhat balanced load, this algorithm performs better. Implementation of this strategy is quite straightforward if built on top of basic controller independent active-backup. On a per logical router basis, the algorithm is the same, leadership is determined by the liveness of the gateways. The key difference here is that the gateways must have a different leadership priority for each logical router. These leadership priorities can be computed by ovn-northd just as they had been in the controller independent active-backup model. Once we have these per logical router priorities, they simply need be communicated to the members of the gateway cluster and the hypervisors. The hypervisors in particular, need simply have an active-backup bundle action (or group action) per logical router listing the gateways in priority order for *that router*, rather than having a single bundle action shared for all the routers. Additionally, the gateways need to be updated to take into account individual router priorities. Specifically, each gateway should drop traffic of backup routers it's running, and forward traffic of active gateways, instead of simply dropping or forwarding everything. This should likely be done by having ovn-controller recompute OpenFlow for the gateway, though other options exist. The final complication is that ovn-northd's logic must be updated to choose these per logical router leadership priorities in a more sophisticated manner. It doesn't matter much exactly what algorithm it chooses to do this, beyond that it should provide good balancing in the common case. I.E. each logical routers priorities should be different enough that routers balance to different gateways even when failures occur. Preemption ++++++++++ In an active-backup setup, one issue that users will run into is that of gateway leader preemption. If a new Gateway is added to a cluster, or for some reason an existing gateway is rebooted, we could end up in a situation where the newly activated gateway has higher priority than any other in the HA cluster. In this case, as soon as that gateway appears, it will preempt leadership from the currently active leader causing an unnecessary failover. Since failover can be quite expensive, this preemption may be undesirable. The controller can optionally avoid preemption by cleverly tweaking the leadership priorities. For each router, new gateways should be assigned priorities that put them second in line or later when they eventually come up. Furthermore, if a gateway goes down for a significant period of time, its old leadership priorities should be revoked and new ones should be assigned as if it's a brand new gateway. Note that this should only happen if a gateway has been down for a while (several minutes), otherwise a flapping gateway could have wide ranging, unpredictable, consequences. Note that preemption avoidance should be optional depending on the deployment. One necessarily sacrifices optimal load balancing to satisfy these requirements as new gateways will get no traffic on boot. Thus, this feature represents a trade-off which must be made on a per installation basis. Fully Active-Active HA ~~~~~~~~~~~~~~~~~~~~~~ :: Fully Active-Active HA +----------------+ +----------------+ | | | | | A B C D E | | A B C D E | | | | | +----------------+ +----------------+ ^ ^ ^ ^ | | | | | | | | + + + + Traffic The final step in L3HA is to have true active-active HA. In this scenario each router has an instance on each Gateway, and a mechanism similar to ECMP is used to distribute traffic evenly among all instances. This mechanism would require Gateways to participate in routing protocols with the physical network to attract traffic and alert of failures. It is out of scope of this document, but may eventually be necessary. L2HA ---- L2HA is very difficult to get right. Unlike L3HA, where the consequences of problems are minor, in L2HA if two gateways are both transiently active, an L2 loop triggers and a broadcast storm results. In practice to get around this, gateways end up implementing an overly conservative "when in doubt drop all traffic" policy, or they implement something like MLAG. MLAG has multiple gateways work together to pretend to be a single L2 switch with a large LACP bond. In principle, it's the right solution to the problem as it solves the broadcast storm problem, and has been deployed successfully in other contexts. That said, it's difficult to get right and not recommended. ovs-2.9.0/Documentation/topics/idl-compound-indexes.rst000066400000000000000000000311571324262074100232340ustar00rootroot00000000000000.. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. Convention for heading levels in Open vSwitch documentation: ======= Heading 0 (reserved for the title in a document) ------- Heading 1 ~~~~~~~ Heading 2 +++++++ Heading 3 ''''''' Heading 4 Avoid deeper levels because they do not render well. ====================== C IDL Compound Indexes ====================== Introduction ------------ This document describes the design and usage of the C IDL Compound Indexes feature, which allows OVSDB client applications to efficiently search table contents using arbitrary sets of column values in a generic way. This feature is implemented entirely in the client IDL, requiring no changes to the OVSDB Server, OVSDB Protocol (OVSDB RFC (RFC 7047)) or additional interaction with the OVSDB server. Please note that in this document, the term "index" refers to the common database term defined as "a data structure that facilitates data retrieval". Unless stated otherwise, the definition for index from the OVSDB RFC (RFC 7047) is not used. Typical Use Cases ----------------- Fast lookups ~~~~~~~~~~~~ Depending on the topology, the route table of a network device could manage thousands of routes. Commands such as "show ip route <*specific route*>" would need to do a sequential lookup of the routing table to find the specific route. With an index created, the lookup time could be faster. This same scenario could be applied to other features such as Access List rules and even interfaces lists. Lexicographic order ~~~~~~~~~~~~~~~~~~~ There are a number of cases in which retrieving data in a particular lexicographic order is needed. For example, SNMP. When an administrator or even a NMS would like to retrieve data from a specific device, it's possible that they will request data from full tables instead of just specific values. Also, they would like to have this information displayed in lexicographic order. This operation could be done by the SNMP daemon or by the CLI, but it would be better if the database could provide the data ready for consumption. Also, duplicate efforts by different processes will be avoided. Another use case for requesting data in lexicographic order is for user interfaces (web or CLI) where it would be better and quicker if the DB sends the data sorted instead of letting each process to sort the data by itself. Implementation Design --------------------- This feature maintains a collection of indexes per table. The application can create any number of indexes per table. An index can be defined over any number of columns, and supports the following options: - Add a column with type string, boolean, uuid, integer or real (using default comparators). - Select ordering direction of a column (ascending or descending, must be selected when creating the index). - Use a custom ordering comparator (eg: treat a string column like a IP, or sort by the value of the "config" key in a map column). For querying the index the user needs to create a cursor. That cursor points to a position in the index. The user can then use the cursor to perform lookups (by key) and/or get the subsequent rows. The user can also compare the current value of the cursor to a record. For lookups, the user needs to provide a key to be used for locating the specific rows that meet his criteria. This key could be an IP address, a MAC address, an ACL rule, etc. When the information is found in the data structure the user's cursor is updated to point to the row. If several rows match the query then the user can easily get the next row in sequence by updating the cursor. For accessing data in lexicographic order, the user can use the ranged iterators. Those iterators need a cursor and "from" and "to" values to define a range. The indexes maintain a pointer to the row in the local replica, avoiding the need to make additional copies of the data and thereby minimizing any additional memory and CPU overhead for their maintenance. It is intended that creating and maintaining indexes should be very cheap. Another potential issue is the time needed to create the data structure and the time needed to add/remove elements. The indexes are always synchronized with the replica. For this reason is VERY IMPORTANT that the comparison functions (built-in and user provided) are FAST. Skiplists are used as the primary data structure for the implementation of indexes. Indexes therefore have an expected ``O(log(n))`` cost when inserting, deleting or modifiying a row, ``O(log(n))`` when retrieving a row by key, and O(1) when retrieving the first or next row. Indexes are maintained incrementally in the replica as notifications of database changes are received from the OVSDB server, as shown in the following diagram. :: +---------------------------------------------------------+ | | +-------------+Client changes to data IDL | | | | +---v---+ | | | OVSDB +--------->OVSDB Notification | +-------+ | + | | | +------------+ | | | | | | | | | Insert Row +----> Insert row to indexes | | | | | ^ | | +-> | Modify Row +-------------------+ | | | | v | | | Delete Row +----> Delete row from indexes | | | | | | +----+-------+ | | | | | +-> IDL Replica | | | +---------------------------------------------------------+ C IDL API --------- Index Creation ~~~~~~~~~~~~~~ Each index must be created with the function ``ovsdb_idl_create_index()``. After an index has been created the user can add one or more columns to it, using ``ovsdb_idl_index_add_column``. All indexes must be created wiith all columns added BEFORE the first call to ovsdb\_idl\_run(). Index Creation Example ^^^^^^^^^^^^^^^^^^^^^^ :: /* Define a custom comparator for the column "stringField" in table * "Test". (Note that custom comparison functions are not often * necessary.) */ int stringField_comparator(const void *a, const void *b) { struct ovsrec_test *AAA, *BBB; AAA = (struct ovsrec_test *)a; BBB = (struct ovsrec_test *)b; return strcmp(AAA->stringField, BBB->stringField); } void init_idl(struct ovsdb_idl **, char *remote) { /* Add the columns to the IDL */ *idl = ovsdb_idl_create(remote, &ovsrec_idl_class, false, true); ovsdb_idl_add_table(*idl, &ovsrec_table_test); ovsdb_idl_add_column(*idl, &ovsrec_test_col_stringField); ovsdb_idl_add_column(*idl, &ovsrec_test_col_numericField); ovsdb_idl_add_column(*idl, &ovsrec_test_col_enumField); ovsdb_idl_add_column(*idl, &ovsrec_test_col_boolField); /* Create an index. * This index is created using (stringField, numericField) as key. * Also shows the usage of some arguments of add column, although * for a string column it is unnecesary to pass a custom comparator. */ struct ovsdb_idl_index *index; index = ovsdb_idl_create_index(*idl, &ovsrec_table_test, "by_stringField"); ovsdb_idl_index_add_column(index, &ovsrec_test_col_stringField, OVSDB_INDEX_ASC, stringField_comparator); ovsdb_idl_index_add_column(index, &ovsrec_test_col_numericField, OVSDB_INDEX_DESC, NULL); /* Done. */ } Index Usage ----------- Iterators ~~~~~~~~~ The recommended way to do queries is using a "ranged foreach", an "equal foreach" or a "full foreach" over an index. The mechanism works as follows: 1. Create a cursor. 2. Create index row objects with index columns set to desired search key values (one is needed for equality iterators, two for range iterators, a search key is not needed for the full index iterator). 3. Pass the cursor, an iteration variable, and the key values to the iterator. 4. Use the values within iterator loop. To create the cursor for the example, we use the following code: :: ovsdb_idl_index_cursor my_cursor; ovsdb_idl_initialize_cursor(idl, &ovsrec_table_test, "by_stringField", &my_cursor); Now the cursor can be used to perform queries. The library implements three different iterators: a range iterator, an equality iterator and a full index iterator. The range iterator receives two values and iterates over all rows with values that are within that range (inclusive of the two values defining the range). The equality iterator iterates over all rows that exactly match the value passed. The full index iterator iterates over all rows in the index, in an order determined by the comparison function and configured direction (ascending or descending). Note that indexes are *sorted by the "concatenation" of the values in all indexed columns*, so the ranged iterator returns all the values between "from.col1 from.col2 ... from.coln" and "to.col1 to.col2 ... to.coln", *NOT the rows with a value in column 1 between from.col1 and to.col1, and so on*. The iterators are macros especific to each table. An example of the use these iterators follows: :: /* * Equality iterator; iterates over all the records equal to "value". */ ovsrec_test *value, *record; value = ovsrec_test_index_init_row(idl, &ovsrec_table_test); ovsrec_test_index_set_stringField(value, "hello world"); OVSREC_TEST_FOR_EACH_EQUAL (record, &my_cursor, value) { /* Can return zero, one or more records */ assert(strcmp(record->stringField, "hello world") == 0); printf("Found one record with %s", record->stringField); } ovsrec_test_index_destroy_row(value); /* * Range iterator; iterates over all records between two values * (inclusive). */ ovsrec_test *value_from, *value_to; value_from = ovsrec_test_index_init_row(idl, &ovsrec_table_test); value_to = ovsrec_test_index_init_row(idl, &ovsrec_table_test); ovsrec_test_index_set_stringField(value_from, "aaa"); ovsrec_test_index_set_stringField(value_to, "mmm"); OVSREC_TEST_FOR_EACH_RANGE (record, &my_cursor, value_from, value_to) { /* Can return zero, one or more records */ assert(strcmp("aaa", record->stringField) <= 0); assert(strcmp(record->stringField, "mmm") <= 0); printf("Found one record with %s", record->stringField); } ovsrec_test_index_destroy_row(value_from); ovsrec_test_index_destroy_row(value_to); /* * Index iterator; iterates over all nodes in the index, in order * determined by comparison function and configured order (ascending * or descending). */ OVSREC_TEST_FOR_EACH_BYINDEX (record, &my_cursor) { /* Can return zero, one or more records */ printf("Found one record with %s", record->stringField); } General Index Access ~~~~~~~~~~~~~~~~~~~~ While the currently defined iterators are suitable for many use cases, it is also possible to create custom iterators using the more general API on which the existing iterators have been built. This API includes the following functions, declared in "lib/ovsdb-idl.h": 1. ``ovsrec__index_compare()`` 2. ``ovsrec_
_index_next()`` 3. ``ovsrec_
_index_find()`` 4. ``ovsrec_
_index_forward_to()`` 5. ``ovsrec_
_index_get_data()`` ovs-2.9.0/Documentation/topics/index.rst000066400000000000000000000032431324262074100203070ustar00rootroot00000000000000.. Copyright (c) 2016, Stephen Finucane Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. Convention for heading levels in Open vSwitch documentation: ======= Heading 0 (reserved for the title in a document) ------- Heading 1 ~~~~~~~ Heading 2 +++++++ Heading 3 ''''''' Heading 4 Avoid deeper levels because they do not render well. ========= Deep Dive ========= How Open vSwitch and OVN are implemented and, where necessary, why it was implemented that way. OVS --- .. toctree:: :maxdepth: 2 design datapath integration porting openflow bonding ovsdb-replication dpdk/index windows language-bindings testing tracing idl-compound-indexes OVN --- .. toctree:: :maxdepth: 2 high-availability role-based-access-control ovn-news-2.8 .. list-table:: * - ovn-architecture(7) - `(pdf) `__ - `(html) `__ - `(plain text) `__ ovs-2.9.0/Documentation/topics/integration.rst000066400000000000000000000270151324262074100215260ustar00rootroot00000000000000.. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. Convention for heading levels in Open vSwitch documentation: ======= Heading 0 (reserved for the title in a document) ------- Heading 1 ~~~~~~~ Heading 2 +++++++ Heading 3 ''''''' Heading 4 Avoid deeper levels because they do not render well. ========================================= Integration Guide for Centralized Control ========================================= This document describes how to integrate Open vSwitch onto a new platform to expose the state of the switch and attached devices for centralized control. (If you are looking to port the switching components of Open vSwitch to a new platform, refer to :doc:`porting`) The focus of this guide is on hypervisors, but many of the interfaces are useful for hardware switches, as well. The XenServer integration is the most mature implementation, so most of the examples are drawn from it. The externally visible interface to this integration is platform-agnostic. We encourage anyone who integrates Open vSwitch to use the same interface, because keeping a uniform interface means that controllers require less customization for individual platforms (and perhaps no customization at all). Integration centers around the Open vSwitch database and mostly involves the ``external_ids`` columns in several of the tables. These columns are not interpreted by Open vSwitch itself. Instead, they provide information to a controller that permits it to associate a database record with a more meaningful entity. In contrast, the ``other_config`` column is used to configure behavior of the switch. The main job of the integrator, then, is to ensure that these values are correctly populated and maintained. An integrator sets the columns in the database by talking to the ovsdb-server daemon. A few of the columns can be set during startup by calling the ovs-ctl tool from inside the startup scripts. The ``xenserver/etc_init.d_openvswitch`` script provides examples of its use, and the ovs-ctl(8) manpage contains complete documentation. At runtime, ovs-vsctl can be be used to set columns in the database. The script ``xenserver/etc_xensource_scripts_vif`` contains examples of its use, and ovs-vsctl(8) manpage contains complete documentation. Python and C bindings to the database are provided if deeper integration with a program are needed. The XenServer ovs-xapi-sync daemon (``xenserver/usr_share_openvswitch_scripts_ovs-xapi-sync``) provides an example of using the Python bindings. More information on the python bindings is available at ``python/ovs/db/idl.py``. Information on the C bindings is available at ``lib/ovsdb-idl.h``. The following diagram shows how integration scripts fit into the Open vSwitch architecture: :: Diagram +----------------------------------------+ | Controller Cluster + +----------------------------------------+ | | +----------------------------------------------------------+ | | | | +--------------+---------------+ | | | | | | +-------------------+ +------------------+ | | | ovsdb-server |-----------| ovs-vswitchd | | | +-------------------+ +------------------+ | | | | | | +---------------------+ | | | | Integration scripts | | | | | (ex: ovs-xapi-sync) | | | | +---------------------+ | | | | Userspace | |----------------------------------------------------------| | | Kernel | | | | | +---------------------+ | | | OVS Kernel Module | | | +---------------------+ | +----------------------------------------------------------+ A description of the most relevant fields for integration follows. By setting these values, controllers are able to understand the network and manage it more dynamically and precisely. For more details about the database and each individual column, please refer to the ovs-vswitchd.conf.db(5) manpage. ``Open_vSwitch`` table ---------------------- The ``Open_vSwitch`` table describes the switch as a whole. The ``system_type`` and ``system_version`` columns identify the platform to the controller. The ``external_ids:system-id`` key uniquely identifies the physical host. In XenServer, the system-id will likely be the same as the UUID returned by ``xe host-list``. This key allows controllers to distinguish between multiple hypervisors. Most of this configuration can be done with the ovs-ctl command at startup. For example: :: $ ovs-ctl --system-type="XenServer" --system-version="6.0.0-50762p" \ --system-id="${UUID}" "${other_options}" start Alternatively, the ovs-vsctl command may be used to set a particular value at runtime. For example: :: $ ovs-vsctl set open_vswitch . external-ids:system-id='"${UUID}"' The ``other_config:enable-statistics`` key may be set to ``true`` to have OVS populate the database with statistics (e.g., number of CPUs, memory, system load) for the controller's use. Bridge table ------------ The Bridge table describes individual bridges within an Open vSwitch instance. The ``external-ids:bridge-id`` key uniquely identifies a particular bridge. In XenServer, this will likely be the same as the UUID returned by ``xe network-list`` for that particular bridge. For example, to set the identifier for bridge "br0", the following command can be used: :: $ ovs-vsctl set Bridge br0 external-ids:bridge-id='"${UUID}"' The MAC address of the bridge may be manually configured by setting it with the ``other_config:hwaddr`` key. For example: :: $ ovs-vsctl set Bridge br0 other_config:hwaddr="12:34:56:78:90:ab" Interface table --------------- The Interface table describes an interface under the control of Open vSwitch. The ``external_ids`` column contains keys that are used to provide additional information about the interface: attached-mac This field contains the MAC address of the device attached to the interface. On a hypervisor, this is the MAC address of the interface as seen inside a VM. It does not necessarily correlate to the host-side MAC address. For example, on XenServer, the MAC address on a VIF in the hypervisor is always FE:FF:FF:FF:FF:FF, but inside the VM a normal MAC address is seen. iface-id This field uniquely identifies the interface. In hypervisors, this allows the controller to follow VM network interfaces as VMs migrate. A well-chosen identifier should also allow an administrator or a controller to associate the interface with the corresponding object in the VM management system. For example, the Open vSwitch integration with XenServer by default uses the XenServer assigned UUID for a VIF record as the iface-id. iface-status In a hypervisor, there are situations where there are multiple interface choices for a single virtual ethernet interface inside a VM. Valid values are "active" and "inactive". A complete description is available in the ovs-vswitchd.conf.db(5) manpage. vm-id This field uniquely identifies the VM to which this interface belongs. A single VM may have multiple interfaces attached to it. As in the previous tables, the ovs-vsctl command may be used to configure the values. For example, to set the ``iface-id`` on eth0, the following command can be used: :: $ ovs-vsctl set Interface eth0 external-ids:iface-id='"${UUID}"' HA for OVN DB servers using pacemaker ------------------------------------- The ovsdb servers can work in either active or backup mode. In backup mode, db server will be connected to an active server and replicate the active servers contents. At all times, the data can be transacted only from the active server. When the active server dies for some reason, entire OVN operations will be stalled. `Pacemaker `__ is a cluster resource manager which can manage a defined set of resource across a set of clustered nodes. Pacemaker manages the resource with the help of the resource agents. One among the resource agent is `OCF `__ OCF is nothing but a shell script which accepts a set of actions and returns an appropriate status code. With the help of the OCF resource agent ovn/utilities/ovndb-servers.ocf, one can defined a resource for the pacemaker such that pacemaker will always maintain one running active server at any time. After creating a pacemaker cluster, use the following commands to create one active and multiple backup servers for OVN databases:: $ pcs resource create ovndb_servers ocf:ovn:ovndb-servers \ master_ip=x.x.x.x \ ovn_ctl= \ op monitor interval="10s" \ op monitor role=Master interval="15s" $ pcs resource master ovndb_servers-master ovndb_servers \ meta notify="true" The `master_ip` and `ovn_ctl` are the parameters that will be used by the OCF script. `ovn_ctl` is optional, if not given, it assumes a default value of /usr/share/openvswitch/scripts/ovn-ctl. `master_ip` is the IP address on which the active database server is expected to be listening, the slave node uses it to connect to the master node. You can add the optional parameters 'nb_master_port', 'nb_master_protocol', 'sb_master_port', 'sb_master_protocol' to set the protocol and port. Whenever the active server dies, pacemaker is responsible to promote one of the backup servers to be active. Both ovn-controller and ovn-northd needs the ip-address at which the active server is listening. With pacemaker changing the node at which the active server is run, it is not efficient to instruct all the ovn-controllers and the ovn-northd to listen to the latest active server's ip-address. This problem can be solved by using a native ocf resource agent ``ocf:heartbeat:IPaddr2``. The IPAddr2 resource agent is just a resource with an ip-address. When we colocate this resource with the active server, pacemaker will enable the active server to be connected with a single ip-address all the time. This is the ip-address that needs to be given as the parameter while creating the `ovndb_servers` resource. Use the following command to create the IPAddr2 resource and colocate it with the active server:: $ pcs resource create VirtualIP ocf:heartbeat:IPaddr2 ip=x.x.x.x \ op monitor interval=30s $ pcs constraint order promote ovndb_servers-master then VirtualIP $ pcs constraint colocation add VirtualIP with master ovndb_servers-master \ score=INFINITY ovs-2.9.0/Documentation/topics/language-bindings.rst000066400000000000000000000027351324262074100225630ustar00rootroot00000000000000.. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. Convention for heading levels in Open vSwitch documentation: ======= Heading 0 (reserved for the title in a document) ------- Heading 1 ~~~~~~~ Heading 2 +++++++ Heading 3 ''''''' Heading 4 Avoid deeper levels because they do not render well. ================= Language Bindings ================= Bindings exist for Open vSwitch in a variety of languages. Official Bindings ----------------- Python ~~~~~~ The Python bindings are part of the `Open vSwitch package`__. You can install the bindings using ``pip``: :: $ pip install ovs __ https://github.com/openvswitch/ovs/tree/master/python/ovs Third-Party Bindings -------------------- Lua ~~~ - `LJIT2ovs:`__ LuaJIT binding for Open vSwitch __ https://github.com/wiladams/LJIT2ovs Go ~~ - `go-odp:`__ A Go library to control the Open vSwitch in-kernel datapath __ https://github.com/weaveworks/go-odp ovs-2.9.0/Documentation/topics/openflow.rst000066400000000000000000000235431324262074100210360ustar00rootroot00000000000000.. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. Convention for heading levels in Open vSwitch documentation: ======= Heading 0 (reserved for the title in a document) ------- Heading 1 ~~~~~~~ Heading 2 +++++++ Heading 3 ''''''' Heading 4 Avoid deeper levels because they do not render well. ================================ OpenFlow Support in Open vSwitch ================================ Open vSwitch support for OpenFlow 1.1 and beyond is a work in progress. This file describes the work still to be done. The Plan -------- OpenFlow version support is not a build-time option. A single build of Open vSwitch must be able to handle all supported versions of OpenFlow. Ideally, even at runtime it should be able to support all protocol versions at the same time on different OpenFlow bridges (and perhaps even on the same bridge). At the same time, it would be a shame to litter the core of the OVS code with lots of ugly code concerned with the details of various OpenFlow protocol versions. The primary approach to compatibility is to abstract most of the details of the differences from the core code, by adding a protocol layer that translates between OF1.x and a slightly higher-level abstract representation. The core of this approach is the many ``struct ofputil_*`` structures in ``include/openvswitch/ofp-util.h``. As a consequence of this approach, OVS cannot use OpenFlow protocol definitions that closely resemble those in the OpenFlow specification, because ``openflow.h`` in different versions of the OpenFlow specification defines the same identifier with different values. Instead, ``openflow-common.h`` contains definitions that are common to all the specifications and separate protocol version-specific headers contain protocol-specific definitions renamed so as not to conflict, e.g. ``OFPAT10_ENQUEUE`` and ``OFPAT11_ENQUEUE`` for the OpenFlow 1.0 and 1.1 values for ``OFPAT_ENQUEUE``. Generally, in cases of conflict, the protocol layer will define a more abstract ``OFPUTIL_*`` or struct ``ofputil_*``. Here are the current approaches in a few tricky areas: * Port numbering. OpenFlow 1.0 has 16-bit port numbers and later OpenFlow versions have 32-bit port numbers. For now, OVS support for later protocol versions requires all port numbers to fall into the 16-bit range, translating the reserved ``OFPP_*`` port numbers. * Actions. OpenFlow 1.0 and later versions have very different ideas of actions. OVS reconciles by translating all the versions' actions (and instructions) to and from a common internal representation. OpenFlow 1.1 ------------ OpenFlow 1.1 support is complete. OpenFlow 1.2 ------------ OpenFlow 1.2 support is complete. OpenFlow 1.3 ------------ OpenFlow 1.3 support requires OpenFlow 1.2 as a prerequisite, plus the following additional work. (This is based on the change log at the end of the OF1.3 spec, reusing most of the section titles directly. I didn't compare the specs carefully yet.) * Add support for multipart requests. Currently we always report ``OFPBRC_MULTIPART_BUFFER_OVERFLOW``. (optional for OF1.3+) * IPv6 extension header handling support. Fully implementing this requires kernel support. This likely will take some careful and probably time-consuming design work. The actual coding, once that is all done, is probably 2 or 3 days work. (optional for OF1.3+) * Per-flow meters. OpenFlow protocol support is now implemented. Support for the special ``OFPM_SLOWPATH`` and ``OFPM_CONTROLLER`` meters is missing. Support for the software switch is under review. (optional for OF1.3+) * Auxiliary connections. An implementation in generic code might be a week's worth of work. The value of an implementation in generic code is questionable, though, since much of the benefit of axuiliary connections is supposed to be to take advantage of hardware support. (We could make the kernel module somehow send packets across the auxiliary connections directly, for some kind of "hardware" support, if we judged it useful enough.) (optional for OF1.3+) * Provider Backbone Bridge tagging. I don't plan to implement this (but we'd accept an implementation). (optional for OF1.3+) * On-demand flow counters. I think this might be a real optimization in some cases for the software switch. (optional for OF1.3+) OpenFlow 1.4 & ONF Extensions for 1.3.X Pack1 --------------------------------------------- The following features are both defined as a set of ONF Extensions for 1.3 and integrated in 1.4. When defined as an ONF Extension for 1.3, the feature is using the Experimenter mechanism with the ONF Experimenter ID. When defined integrated in 1.4, the feature use the standard OpenFlow structures (for example defined in openflow-1.4.h). The two definitions for each feature are independent and can exist in parallel in OVS. * Flow entry notifications This seems to be modelled after OVS's NXST_FLOW_MONITOR. (EXT-187) (optional for OF1.4+) * Role Status Already implemented as a 1.4 feature. (EXT-191) (required for OF1.4+) * Flow entry eviction OVS has flow eviction functionality. ``table_mod OFPTC_EVICTION``, ``flow_mod 'importance'``, and ``table_desc ofp_table_mod_prop_eviction`` need to be implemented. (EXT-192-e) (optional for OF1.4+) * Vacancy events (EXT-192-v) (optional for OF1.4+) * Bundle Transactional modification. OpenFlow 1.4 requires to support ``flow_mods`` and ``port_mods`` in a bundle if bundle is supported. (Not related to OVS's 'ofbundle' stuff.) Implemented as an OpenFlow 1.4 feature. Only flow_mods and port_mods are supported in a bundle. If the bundle includes port mods, it may not specify the ``OFPBF_ATOMIC`` flag. Nevertheless, port mods and flow mods in a bundle are always applied in order and consecutive flow mods between port mods are made available to lookups atomically. (EXT-230) (optional for OF1.4+) * Table synchronisation Probably not so useful to the software switch. (EXT-232) (optional for OF1.4+) * Group and Meter change notifications (EXT-235) (optional for OF1.4+) * Bad flow entry priority error Probably not so useful to the software switch. (EXT-236) (optional for OF1.4+) * Set async config error (EXT-237) (optional for OF1.4+) * PBB UCA header field See comment on Provider Backbone Bridge in section about OpenFlow 1.3. (EXT-256) (optional for OF1.4+) * Multipart timeout error (EXT-264) (required for OF1.4+) OpenFlow 1.4 only ----------------- Those features are those only available in OpenFlow 1.4, other OpenFlow 1.4 features are listed in the previous section. * More extensible wire protocol Many on-wire structures got TLVs. All required features are now supported. Remaining optional: table desc, table-status (EXT-262) (required for OF1.4+) * Optical port properties (EXT-154) (optional for OF1.4+) OpenFlow 1.5 & ONF Extensions for 1.3.X Pack2 --------------------------------------------- The following features are both defined as a set of ONF Extensions for 1.3 and integrated in 1.5. Note that this list is not definitive as those are not yet published. When defined as an ONF Extension for 1.3, the feature is using the Experimenter mechanism with the ONF Experimenter ID. When defined integrated in 1.5, the feature use the standard OpenFlow structures (for example defined in openflow-1.5.h). The two definitions for each feature are independent and can exist in parallel in OVS. * Time scheduled bundles (EXT-340) (optional for OF1.5+) OpenFlow 1.5 only ----------------- Those features are those only available in OpenFlow 1.5, other OpenFlow 1.5 features are listed in the previous section. Note that this list is not definitive as OpenFlow 1.5 is not yet published. * Egress Tables (EXT-306) (optional for OF1.5+) * Packet Type aware pipeline Prototype for OVS was done during specification. (EXT-112) (optional for OF1.5+) * Extensible Flow Entry Statistics (EXT-334) (required for OF1.5+) * Flow Entry Statistics Trigger (EXT-335) (optional for OF1.5+) * Controller connection status Prototype for OVS was done during specification. (EXT-454) (optional for OF1.5+) * Meter action (EXT-379) (required for OF1.5+ if metering is supported) * Port properties for pipeline fields Prototype for OVS was done during specification. (EXT-388) (optional for OF1.5+) * Port property for recirculation Prototype for OVS was done during specification. (EXT-399) (optional for OF1.5+) General ------- * ovs-ofctl(8) often lists as Nicira extensions features that later OpenFlow versions support in standard ways. How to contribute ----------------- If you plan to contribute code for a feature, please let everyone know on ovs-dev before you start work. This will help avoid duplicating work. Consider the following: * Testing. Please test your code. * Unit tests. Consider writing some. The tests directory has many examples that you can use as a starting point. * ovs-ofctl. If you add a feature that is useful for some ovs-ofctl command then you should add support for it there. * Documentation. If you add a user-visible feature, then you should document it in the appropriate manpage and mention it in NEWS as well. Refer to :doc:`/internals/contributing/index` for more information. ovs-2.9.0/Documentation/topics/ovn-news-2.8.rst000066400000000000000000000340351324262074100212640ustar00rootroot00000000000000.. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. Convention for heading levels in Open vSwitch documentation: ======= Heading 0 (reserved for the title in a document) ------- Heading 1 ~~~~~~~ Heading 2 +++++++ Heading 3 ''''''' Heading 4 Avoid deeper levels because they do not render well. =============================== What's New with OVS and OVN 2.8 =============================== This document is about what was added in Open vSwitch 2.8, which was released at the end of August 2017, concentrating on the new features in OVN. It also covers some of what is coming up in Open vSwitch and OVN 2.9, which is due to be released in February 2018. OVN has many features, and this document does not cover every new or enhanced feature (but contributions are welcome). This document assumes a basic familiarity with Open vSwitch, OVN, and their associated tools. For more information, please refer to the Open vSwitch and OVN documentation, such as the ``ovn-architecture``\(7) manpage. Debugging and Troubleshooting ----------------------------- Before version 2.8, Open vSwitch command-line tools were far more painful to use than they needed to be. This section covers the improvements made to the CLI in the 2.8 release. User-Hostile UUIDs ~~~~~~~~~~~~~~~~~~ The OVN CLI, through ``ovn-nbctl``, ``ovn-nbctl``, and ``ovn-trace``, used full-length UUIDs almost everywhere. It didn't even provide any assistance with completion, etc., which in practice meant always cutting and pasting UUIDs from one command or window to another. This problem wasn't limited to the places where one would expect to have to see or use a UUID, either. In many places where one would expect to be able to use a network, router, or port name, a UUID was required instead. In many places where one would want to see a name, the UUID was displayed instead. More than anything else, these shortcomings made the CLI user-hostile. There was an underlying problem that the southbound database didn't actually contain all the information needed to provide a decent user interface. In some cases, for example, the human-friendly names that one would want to use for entities simply weren't part of the database. These names weren't necessary for correctness, only for usability. OVN 2.8 eased many of these problems. Most parts of the CLI now allow the user to abbreviate UUIDs, as long as the abbreviations are unique within the database. Some parts of the CLI where full-length UUIDs make output hard to read now abbreviate them themselves. Perhaps more importantly, in many places the OVN CLI now displays and accepts human-friendly names for networks, routers, ports, and other entities. In the places where the names were not previously available, OVN (through ``ovn-northd``) now copies the names into the southbound database. The CLIs for layers below OVN, at the OpenFlow and datapath layers with ``ovs-ofctl`` and ``ovs-dpctl``, respectively, had some similar problems in which numbers were used for entities that had human-friendly names. Open vSwitch 2.8 also solves some of those problems. Other than that, the most notable enhancement in this area was the ``--no-stats`` option to ``ovs-ofctl dump-flows``, which made that command's output more readable for the cases where per-flow statistics were not interesting to the reader. Connections Between Levels ~~~~~~~~~~~~~~~~~~~~~~~~~~ OVN and Open vSwitch work almost like a stack of compilers: the OVN Neutron plugin translates Neutron configuration into OVN northbound configuration, which ``ovn-northd`` translates into logical flows, which ``ovn-controller`` translates into OpenFlow flows, which ``ovs-vswitchd`` translates into datapath flows. For debugging and troubleshooting it is often necessary to understand exactly how these translations work. The relationship from a logical flow to its OpenFlow flows, or in the other direction, from an OpenFlow flow back to the logical flow that produced it, was often of particular interest, but OVN didn't provide good tools for the job. OVN 2.8 added some new features that ease these jobs. ``ovn-sbctl lflow-list`` has a new option ``--ovs`` that lists the OpenFlow flows on a particular chassis that were generated from the logical flows that it lists. ``ovn-trace`` also added a similar ``--ovs`` option that applies to the logical flows it traces. In the other direction, OVN 2.8 added a new utility ``ovn-detrace`` that, given an Open vSwitch trace of OpenFlow flows, annotates it with the logical flows that yielded those OpenFlow flows. Distributed Firewall ~~~~~~~~~~~~~~~~~~~~ OVN supports a distributed firewall with stateful connection tracking to ensure that only packets for established connections, or those that the configuration explicitly allows, can ingress a given VM or container. Neutron uses this feature by default. Most packets in an OpenStack environment pass through it twice, once after egress from the packet's source VM and once before ingress into its destination VM. Before OVN 2.8, the ``ovn-trace`` program, which shows the path of a packet through an OVN logical network, did not support the logical firewall, which in practice made it almost useless for Neutron. In OVN 2.8, ``ovn-trace`` adds support for the logical firewall. By default it assumes that packets are part of an established connection, which is usually what the user wants as part of the trace. It also accepts command-line options to override that assumption, which allows the user to discover the treatment of packets that the firewall should drop. At the next level deeper, prior to Open vSwitch 2.8, the OpenFlow tracing command ``ofproto/trace`` also supported neither the connection tracking feature underlying the OVN distributed firewall nor the "recirculation" feature that accompanied it. This meant that, even if the user tried to look deeper into the distributed firewall mechanism, he or she would encounter a further roadblock. Open vSwitch 2.8 added support for both of these features as well. Summary Display ~~~~~~~~~~~~~~~ ``ovn-nbctl show`` and ``ovn-sbctl show``, for showing an overview of the OVN configuration, didn't show a lot of important information. OVN adds some more useful information here. DNS, and IPAM ------------- OVN 2.8 adds a built-in DNS server designed for assigning names to VMs and containers within an OVN logical network. DNS names are assigned using records in the OVN northbound database and, like other OVN features, translated into logical flows at the OVN southbound layer. DNS requests directed to the OVN DNS server never leave the hypervisor from which the request is sent; instead, OVN processes and replies to the request from its ``ovn-controller`` local agent. The OVN DNS server is not a general-purpose DNS server and cannot be used for that purpose. OVN includes simple built-in support for IP address management (IPAM), in which OVN assigns IP addresses to VMs or containers from a pool or pools of IP addresses delegated to it by the administrator. Before OVN 2.8, OVN IPAM only supported IPv4 addresses; OVN 2.8 adds support for IPv6. OVN 2.8 also enhances the address pool support to allow specific addresses to be excluded. Neutron assigns IP addresses itself and does not use OVN IPAM. High Availability ----------------- As a distributed system, in OVN a lot can go wrong. As OVN advances, it adds redundancy in places where currently a single failure could disrupt the functioning of the system as a whole. OVN 2.8 adds two new kinds of high availability. ovn-northd HA ~~~~~~~~~~~~~ The ``ovn-northd`` program sits between the OVN northbound and southbound databases and translates from a logical network configuration into logical flows. If ``ovn-northd`` itself or the host on which it runs fails, then updates to the OVN northbound configuration will not propagate to the hypervisors and the OVN configuration freezes in place until ``ovn-northd`` restarts. OVN 2.8 adds support for active-backup HA to ``ovn-northd``. When more than one ``ovn-northd`` instance runs, it uses an OVSDB locking feature to automatically choose a single active instance. When that instance dies or becomes nonresponsive, the OVSDB server automatically choose one of the remaining instance(s) to take over. L3 Gateway HA ~~~~~~~~~~~~~ In OVN 2.8, multiple chassis may now be specified for L3 gateways. When more than one chassis is specified, OVN manages high availability for that gateway. Each hypervisor uses the BFD protocol to keep track of the gateway nodes that are currently up. At any given time, a hypervisor uses the highest-priority gateway node that is currently up. OVSDB ----- The OVN architecture relies heavily on OVSDB, the Open vSwitch database, for hosting the northbound and southbound databases. OVSDB was originally selected for this purpose because it was already used in Open vSwitch for configuring OVS itself and, thus, it was well integrated with OVS and well supported in C and Python, the two languages that are used in Open vSwitch. OVSDB was well designed for its original purpose of configuring Open vSwitch. It supports ACID transactions, has a small, efficient server, a flexible schema system, and good support for troubleshooting and debugging. However, it lacked several features that are important for OVN but not for Open vSwitch. As OVN advances, these missing features have become more and more of a problem. One option would be to switch to a different database that already has many of these features, but despite a careful search, no ideal existing database was identified, so the project chose instead to improve OVSDB where necessary to bring it up to speed. The following sections talk more about recent and future improvements. High Availability ~~~~~~~~~~~~~~~~~ When ``ovsdb-server`` was only used for OVS configuration, high availability was not important. ``ovsdb-server`` was capable of restarting itself automatically if it crashed, and if the whole system went down then Open vSwitch itself was dead too, so the database server's failure was not important. In contrast, the northbound and southbound databases are centralized components of a distributed system, so it is important that they not be a single point of failure for the system as a whole. In released versions of OVN, ``ovsdb-server`` supports only "active-backup replication" across a pair of servers. This means that if one server goes down, the other can pick it back up approximately where the other one left off. The servers do not have built-in support for deciding at any given time which is the active and which the backup, so the administrator must configure an external agent to do this management. Active-backup replication is not entirely satisfactory, for multiple reasons. Replication is only approximate. Configuring the external agent requires extra work. There is no benefit from the backup server except when the active server fails. At most two servers can be used. A new form of high availability for OVSDB is under development for the OVN 2.9 release, based on the Raft algorithm for distributed consensus. Whereas replication uses two servers, clustering using Raft requires three or more (typically an odd number) and continues functioning as long as more than half of the servers are up. The clustering implementation is built into ``ovsdb-server`` and does not require an external agent. Clustering preserves the ACID properties of the database, so that a transaction that commits is guaranteed to persist. Finally, reads (which are the bulk of the OVN workload) scale with the size of the cluster, so that adding more servers should improve performance as the number of hypervisors in an OVN deployment increases. As of this writing, OVSDB support for clustering is undergoing development and early deployment testing. RBAC security ~~~~~~~~~~~~~ Until Open vSwitch 2.8, ``ovsdb-server`` had little support for access control within a database. If an OVSDB client could modify the database at all, it could make arbitrary changes. This was sufficient for most uses case to that point. Hypervisors in an OVN deployment need access to the OVN southbound database. Most of their access is reads, to find out about the OVN configuration. Hypervisors do need some write access to the southbound database, primarily to let the other hypervisors know what VMs and containers they are running and how to reach them. Thus, OVN gives all of the hypervisors in the OVN deployment write access to the OVN southbound database. This is fine when all is well, but if any of the hypervisors were compromised then they could disrupt the entire OVN deployment by corrupting the database. The OVN developers considered a few ways to solve this problem. One way would be to introduce a new central service (perhaps in ``ovn-northd``) that provided only the kinds of writes that the hypervisors legitimately need, and then grant hypervisors direct access to the southbound database only for reads. But ultimately the developers decided to introduce a new form of more access control for OVSDB, called the OVSDB RBAC (role-based access control) feature. OVSDB RBAC allows for granular enough control over access that hypervisors can be granted only the ability to add, modify, and delete the records that relate to themselves, preventing them from corrupting the database as a whole. Further Directions ------------------ For more information about new features in OVN and Open vSwitch, please refer to the NEWS file distributed with the source tree. If you have questions about Open vSwitch or OVN features, please feel free to write to the Open vSwitch discussion mailing list at ovs-discuss@openvswitch.org. ovs-2.9.0/Documentation/topics/ovsdb-replication.rst000066400000000000000000000156261324262074100226340ustar00rootroot00000000000000.. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. Convention for heading levels in Open vSwitch documentation: ======= Heading 0 (reserved for the title in a document) ------- Heading 1 ~~~~~~~ Heading 2 +++++++ Heading 3 ''''''' Heading 4 Avoid deeper levels because they do not render well. ================================ OVSDB Replication Implementation ================================ Given two Open vSwitch databases with the same schema, OVSDB replication keeps these databases in the same state, i.e. each of the databases have the same contents at any given time even if they are not running in the same host. This document elaborates on the implementation details to provide this functionality. Terminology ----------- Source of truth database database whose content will be replicated to another database. Active server ovsdb-server providing RPC interface to the source of truth database. Standby server ovsdb-server providing RPC interface to the database that is not the source of truth. Design ------ The overall design of replication consists of one ovsdb-server (active server) communicating the state of its databases to another ovsdb-server (standby server) so that the latter keep its own databases in that same state. To achieve this, the standby server acts as a client of the active server, in the sense that it sends a monitor request to keep up to date with the changes in the active server databases. When a notification from the active server arrives, the standby server executes the necessary set of operations so its databases reach the same state as the the active server databases. Below is the design represented as a diagram.:: +--------------+ replication +--------------+ | Active |<-------------------| Standby | | OVSDB-server | | OVSDB-server | +--------------+ +--------------+ | | | | +-------+ +-------+ | SoT | | | | OVSDB | | OVSDB | +-------+ +-------+ Setting Up The Replication -------------------------- To initiate the replication process, the standby server must be executed indicating the location of the active server via the command line option ``--sync-from=server``, where server can take any form described in the ovsdb-client manpage and it must specify an active connection type (tcp, unix, ssl). This option will cause the standby server to attempt to send a monitor request to the active server in every main loop iteration, until the active server responds. When sending a monitor request the standby server is doing the following: 1. Erase the content of the databases for which it is providing a RPC interface. 2. Open the jsonrpc channel to communicate with the active server. 3. Fetch all the databases located in the active server. 4. For each database with the same schema in both the active and standby servers: construct and send a monitor request message specifying the tables that will be monitored (i.e all the tables on the database except the ones blacklisted [*]). 5. Set the standby database to the current state of the active database. Once the monitor request message is sent, the standby server will continuously receive notifications of changes occurring to the tables specified in the request. The process of handling this notifications is detailed in the next section. [*] A set of tables that will be excluded from replication can be configure as a blacklist of tables via the command line option ``--sync-exclude-tables=db:table[,db:table]...``, where db corresponds to the database where the table resides. Replication Process ------------------- The replication process consists on handling the update notifications received in the standby server caused by the monitor request that was previously sent to the active server. In every loop iteration, the standby server attempts to receive a message from the active server which can be an error, an echo message (used to keep the connection alive) or an update notification. In case the message is a fatal error, the standby server will disconnect from the active without dropping the replicated data. If it is an echo message, the standby server will reply with an echo message as well. If the message is an update notification, the following process occurs: 1. Create a new transaction. 2. Get the ```` object from the ``params`` member of the notification. 3. For each ```` in the ```` object do: 1. For each ```` in ```` check what kind of operation should be executed according to the following criteria about the presence of the object members: - If ``old`` member is not present, execute an insert operation using ```` from the ``new`` member. - If ``old`` member is present and ``new`` member is not present, execute a delete operation using ```` from the ``old`` member - If both ``old`` and ``new`` members are present, execute an update operation using ```` from the ``new`` member. 4. Commit the transaction. If an error occurs during the replication process, all replication is restarted by resending a new monitor request as described in the section "Setting up the replication". Runtime Management Commands --------------------------- Runtime management commands can be sent to a running standby server via ovs-appctl in order to configure the replication functionality. The available commands are the following. ``ovsdb-server/set-remote-ovsdb-server {server}`` sets the name of the active server ``ovsdb-server/get-remote-ovsdb-server`` gets the name of the active server ``ovsdb-server/connect-remote-ovsdb-server`` causes the server to attempt to send a monitor request every main loop iteration ``ovsdb-server/disconnect-remote-ovsdb-server`` closes the jsonrpc channel between the active server and frees the memory used for the replication configuration. ``ovsdb-server/set-sync-exclude-tables {db:table,...}`` sets the tables list that will be excluded from being replicated ``ovsdb-server/get-sync-excluded-tables`` gets the tables list that is currently excluded from replication ovs-2.9.0/Documentation/topics/porting.rst000066400000000000000000000354141324262074100206670ustar00rootroot00000000000000.. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. Convention for heading levels in Open vSwitch documentation: ======= Heading 0 (reserved for the title in a document) ------- Heading 1 ~~~~~~~ Heading 2 +++++++ Heading 3 ''''''' Heading 4 Avoid deeper levels because they do not render well. ================================================ Porting Open vSwitch to New Software or Hardware ================================================ Open vSwitch (OVS) is intended to be easily ported to new software and hardware platforms. This document describes the types of changes that are most likely to be necessary in porting OVS to Unix-like platforms. (Porting OVS to other kinds of platforms is likely to be more difficult.) Vocabulary ---------- For historical reasons, different words are used for essentially the same concept in different areas of the Open vSwitch source tree. Here is a concordance, indexed by the area of the source tree: :: datapath/ vport --- vswitchd/ iface port ofproto/ port bundle ofproto/bond.c slave bond lib/lacp.c slave lacp lib/netdev.c netdev --- database Interface Port Open vSwitch Architectural Overview ----------------------------------- The following diagram shows the very high-level architecture of Open vSwitch from a porter's perspective. :: +-------------------+ | ovs-vswitchd |<-->ovsdb-server +-------------------+ | ofproto |<-->OpenFlow controllers +--------+-+--------+ | netdev | | ofproto| +--------+ |provider| | netdev | +--------+ |provider| +--------+ Some of the components are generic. Modulo bugs or inadequacies, these components should not need to be modified as part of a port: ovs-vswitchd The main Open vSwitch userspace program, in vswitchd/. It reads the desired Open vSwitch configuration from the ovsdb-server program over an IPC channel and passes this configuration down to the "ofproto" library. It also passes certain status and statistical information from ofproto back into the database. ofproto The Open vSwitch library, in ofproto/, that implements an OpenFlow switch. It talks to OpenFlow controllers over the network and to switch hardware or software through an "ofproto provider", explained further below. netdev The Open vSwitch library, in lib/netdev.c, that abstracts interacting with network devices, that is, Ethernet interfaces. The netdev library is a thin layer over "netdev provider" code, explained further below. The other components may need attention during a port. You will almost certainly have to implement a "netdev provider". Depending on the type of port you are doing and the desired performance, you may also have to implement an "ofproto provider" or a lower-level component called a "dpif" provider. The following sections talk about these components in more detail. Writing a netdev Provider ------------------------- A "netdev provider" implements an operating system and hardware specific interface to "network devices", e.g. eth0 on Linux. Open vSwitch must be able to open each port on a switch as a netdev, so you will need to implement a "netdev provider" that works with your switch hardware and software. ``struct netdev_class``, in ``lib/netdev-provider.h``, defines the interfaces required to implement a netdev. That structure contains many function pointers, each of which has a comment that is meant to describe its behavior in detail. If the requirements are unclear, report this as a bug. The netdev interface can be divided into a few rough categories: - Functions required to properly implement OpenFlow features. For example, OpenFlow requires the ability to report the Ethernet hardware address of a port. These functions must be implemented for minimally correct operation. - Functions required to implement optional Open vSwitch features. For example, the Open vSwitch support for in-band control requires netdev support for inspecting the TCP/IP stack's ARP table. These functions must be implemented if the corresponding OVS features are to work, but may be omitted initially. - Functions needed in some implementations but not in others. For example, most kinds of ports (see below) do not need functionality to receive packets from a network device. The existing netdev implementations may serve as useful examples during a port: - lib/netdev-linux.c implements netdev functionality for Linux network devices, using Linux kernel calls. It may be a good place to start for full-featured netdev implementations. - lib/netdev-vport.c provides support for "virtual ports" implemented by the Open vSwitch datapath module for the Linux kernel. This may serve as a model for minimal netdev implementations. - lib/netdev-dummy.c is a fake netdev implementation useful only for testing. .. _porting strategies: Porting Strategies ------------------ After a netdev provider has been implemented for a system's network devices, you may choose among three basic porting strategies. The lowest-effort strategy is to use the "userspace switch" implementation built into Open vSwitch. This ought to work, without writing any more code, as long as the netdev provider that you implemented supports receiving packets. It yields poor performance, however, because every packet passes through the ovs-vswitchd process. Refer to :doc:`/intro/install/userspace` for instructions on how to configure a userspace switch. If the userspace switch is not the right choice for your port, then you will have to write more code. You may implement either an "ofproto provider" or a "dpif provider". Which you should choose depends on a few different factors: * Only an ofproto provider can take full advantage of hardware with built-in support for wildcards (e.g. an ACL table or a TCAM). * A dpif provider can take advantage of the Open vSwitch built-in implementations of bonding, LACP, 802.1ag, 802.1Q VLANs, and other features. An ofproto provider has to provide its own implementations, if the hardware can support them at all. * A dpif provider is usually easier to implement, but most appropriate for software switching. It "explodes" wildcard rules into exact-match entries (with an optional wildcard mask). This allows fast hash lookups in software, but makes inefficient use of TCAMs in hardware that support wildcarding. The following sections describe how to implement each kind of port. ofproto Providers ----------------- An "ofproto provider" is what ofproto uses to directly monitor and control an OpenFlow-capable switch. ``struct ofproto_class``, in ``ofproto/ofproto-provider.h``, defines the interfaces to implement an ofproto provider for new hardware or software. That structure contains many function pointers, each of which has a comment that is meant to describe its behavior in detail. If the requirements are unclear, report this as a bug. The ofproto provider interface is preliminary. Let us know if it seems unsuitable for your purpose. We will try to improve it. Writing a dpif Provider ----------------------- Open vSwitch has a built-in ofproto provider named "ofproto-dpif", which is built on top of a library for manipulating datapaths, called "dpif". A "datapath" is a simple flow table, one that is only required to support exact-match flows, that is, flows without wildcards. When a packet arrives on a network device, the datapath looks for it in this table. If there is a match, then it performs the associated actions. If there is no match, the datapath passes the packet up to ofproto-dpif, which maintains the full OpenFlow flow table. If the packet matches in this flow table, then ofproto-dpif executes its actions and inserts a new entry into the dpif flow table. (Otherwise, ofproto-dpif passes the packet up to ofproto to send the packet to the OpenFlow controller, if one is configured.) When calculating the dpif flow, ofproto-dpif generates an exact-match flow that describes the missed packet. It makes an effort to figure out what fields can be wildcarded based on the switch's configuration and OpenFlow flow table. The dpif is free to ignore the suggested wildcards and only support the exact-match entry. However, if the dpif supports wildcarding, then it can use the masks to match multiple flows with fewer entries and potentially significantly reduce the number of flow misses handled by ofproto-dpif. The "dpif" library in turn delegates much of its functionality to a "dpif provider". The following diagram shows how dpif providers fit into the Open vSwitch architecture: :: Architecure _ | +-------------------+ | | ovs-vswitchd |<-->ovsdb-server | +-------------------+ | | ofproto |<-->OpenFlow controllers | +--------+-+--------+ _ | | netdev | |ofproto-| | userspace | +--------+ | dpif | | | | netdev | +--------+ | | |provider| | dpif | | | +---||---+ +--------+ | | || | dpif | | implementation of | || |provider| | ofproto provider |_ || +---||---+ | || || | _ +---||-----+---||---+ | | | |datapath| | kernel | | +--------+ _| | | | |_ +--------||---------+ || physical NIC struct ``dpif_class``, in ``lib/dpif-provider.h``, defines the interfaces required to implement a dpif provider for new hardware or software. That structure contains many function pointers, each of which has a comment that is meant to describe its behavior in detail. If the requirements are unclear, report this as a bug. There are two existing dpif implementations that may serve as useful examples during a port: * lib/dpif-netlink.c is a Linux-specific dpif implementation that talks to an Open vSwitch-specific kernel module (whose sources are in the "datapath" directory). The kernel module performs all of the switching work, passing packets that do not match any flow table entry up to userspace. This dpif implementation is essentially a wrapper around calls into the kernel module. * lib/dpif-netdev.c is a generic dpif implementation that performs all switching internally. This is how the Open vSwitch userspace switch is implemented. Miscellaneous Notes ------------------- Open vSwitch source code uses ``uint16_t``, ``uint32_t``, and ``uint64_t`` as fixed-width types in host byte order, and ``ovs_be16``, ``ovs_be32``, and ``ovs_be64`` as fixed-width types in network byte order. Each of the latter is equivalent to the one of the former, but the difference in name makes the intended use obvious. The default "fail-mode" for Open vSwitch bridges is "standalone", meaning that, when the OpenFlow controllers cannot be contacted, Open vSwitch acts as a regular MAC-learning switch. This works well in virtualization environments where there is normally just one uplink (either a single physical interface or a bond). In a more general environment, it can create loops. So, if you are porting to a general-purpose switch platform, you should consider changing the default "fail-mode" to "secure", which does not behave this way. See documentation for the "fail-mode" column in the Bridge table in ovs-vswitchd.conf.db(5) for more information. ``lib/entropy.c`` assumes that it can obtain high-quality random number seeds at startup by reading from /dev/urandom. You will need to modify it if this is not true on your platform. ``vswitchd/system-stats.c`` only knows how to obtain some statistics on Linux. Optionally you may implement them for your platform as well. Why OVS Does Not Support Hybrid Providers ----------------------------------------- The `porting strategies`_ section above describes the "ofproto provider" and "dpif provider" porting strategies. Only an ofproto provider can take advantage of hardware TCAM support, and only a dpif provider can take advantage of the OVS built-in implementations of various features. It is therefore tempting to suggest a hybrid approach that shares the advantages of both strategies. However, Open vSwitch does not support a hybrid approach. Doing so may be possible, with a significant amount of extra development work, but it does not yet seem worthwhile, for the reasons explained below. First, user surprise is likely when a switch supports a feature only with a high performance penalty. For example, one user questioned why adding a particular OpenFlow action to a flow caused a 1,058x slowdown on a hardware OpenFlow implementation [1]_. The action required the flow to be implemented in software. Given that implementing a flow in software on the slow management CPU of a hardware switch causes a major slowdown, software-implemented flows would only make sense for very low-volume traffic. But many of the features built into the OVS software switch implementation would need to apply to every flow to be useful. There is no value, for example, in applying bonding or 802.1Q VLAN support only to low-volume traffic. Besides supporting features of OpenFlow actions, a hybrid approach could also support forms of matching not supported by particular switching hardware, by sending all packets that might match a rule to software. But again this can cause an unacceptable slowdown by forcing bulk traffic through software in the hardware switch's slow management CPU. Consider, for example, a hardware switch that can match on the IPv6 Ethernet type but not on fields in IPv6 headers. An OpenFlow table that matched on the IPv6 Ethernet type would perform well, but adding a rule that matched only UDPv6 would force every IPv6 packet to software, slowing down not just UDPv6 but all IPv6 processing. .. [1] Aaron Rosen, "Modify packet fields extremely slow", openflow-discuss mailing list, June 26, 2011, archived at https://mailman.stanford.edu/pipermail/openflow-discuss/2011-June/002386.html. Questions --------- Direct porting questions to dev@openvswitch.org. We will try to use questions to improve this porting guide. ovs-2.9.0/Documentation/topics/role-based-access-control.rst000066400000000000000000000077311324262074100241400ustar00rootroot00000000000000.. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. Convention for heading levels in Open vSwitch documentation: ======= Heading 0 (reserved for the title in a document) ------- Heading 1 ~~~~~~~ Heading 2 +++++++ Heading 3 ''''''' Heading 4 Avoid deeper levels because they do not render well. ========================= Role Based Access Control ========================= Where SSL provides authentication when connecting to an OVS database, role based access control (RBAC) provides authorization to operations performed by clients connecting to an OVS database. RBAC allows for administrators to restrict the database operations a client may perform and thus enhance the security already provided by SSL. In theory, any OVS database could define RBAC roles and permissions, but at present only the OVN southbound database has the appropriate tables defined to facilitate RBAC. Mechanics --------- RBAC is intended to supplement SSL. In order to enable RBAC, the connection to the database must use SSL. Some permissions in RBAC are granted based on the certificate common name (CN) of the connecting client. RBAC is controlled with two database tables, RBAC_Role and RBAC_Permission. The RBAC_Permission table contains records that describe a set of permissions for a given table in the database. The RBAC_Permission table contains the following columns: table The table in the database for which permissions are being described. insert_delete Describes whether insertion and deletion of records is allowed. update A list of columns that are allowed to be updated. authorization A list of column names. One of the listed columns must match the SSL certificate CN in order for the attempted operation on the table to succeed. If a key-value pair is provided, then the key is the column name, and the value is the name of a key in that column. An empty string gives permission to all clients to perform operations. The RBAC_Role table contains the following columns: name The name of the role being defined permissions A list of key-value pairs. The key is the name of a table in the database, and the value is a UUID of a record in the RBAC_Permission table that describes the permissions the role has for that table. .. note:: All tables not explicitly referenced in an RBAC_Role record are read-only In order to enable RBAC, specify the role name as an argument to the set-connection command for the database. As an example, to enable the "ovn-controller" role on the OVN southbound database, use the following command: :: $ ovn-sbctl set-connection role=ovn-controller ssl:192.168.0.1:6642 Pre-defined Roles ----------------- This section describes roles that have been defined internally by OVS/OVN. ovn-controller ~~~~~~~~~~~~~~ The ovn-controller role is specified in the OVN southbound database and is intended for use by hypervisors running the ovn-controller daemon. ovn-controller connects to the OVN southbound database mostly to read information, but there are a few cases where ovn-controller also needs to write. The ovn-controller role was designed to allow for ovn-controllers to write to the southbound database only in places where it makes sense to do so. This way, if an intruder were to take over a hypervisor running ovn-controller, it is more difficult to compromise the entire overlay network. It is strongly recommended to set the ovn-controller role for the OVN southbound database to enhance security. ovs-2.9.0/Documentation/topics/testing.rst000066400000000000000000000343041324262074100206570ustar00rootroot00000000000000.. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. Convention for heading levels in Open vSwitch documentation: ======= Heading 0 (reserved for the title in a document) ------- Heading 1 ~~~~~~~ Heading 2 +++++++ Heading 3 ''''''' Heading 4 Avoid deeper levels because they do not render well. ======= Testing ======= It is possible to test Open vSwitch using both tooling provided with Open vSwitch and using a variety of third party tooling. Built-in Tooling ---------------- Open vSwitch provides a number of different test suites and other tooling for validating basic functionality of OVS. Before running any of the tests described here, you must bootstrap, configure and build Open vSwitch as described in :doc:`/intro/install/general`. You do not need to install Open vSwitch or to build or load the kernel module to run these test suites. You do not need supervisor privilege to run these test suites. Unit Tests ~~~~~~~~~~ Open vSwitch includes a suite of self-tests. Before you submit patches upstream, we advise that you run the tests and ensure that they pass. If you add new features to Open vSwitch, then adding tests for those features will ensure your features don't break as developers modify other areas of Open vSwitch. To run all the unit tests in Open vSwitch, one at a time, run:: $ make check This takes under 5 minutes on a modern desktop system. To run all the unit tests in Open vSwitch in parallel, run:: $ make check TESTSUITEFLAGS=-j8 You can run up to eight threads. This takes under a minute on a modern 4-core desktop system. To see a list of all the available tests, run:: $ make check TESTSUITEFLAGS=--list To run only a subset of tests, e.g. test 123 and tests 477 through 484, run:: $ make check TESTSUITEFLAGS='123 477-484' Tests do not have inter-dependencies, so you may run any subset. To run tests matching a keyword, e.g. ``ovsdb``, run:: $ make check TESTSUITEFLAGS='-k ovsdb' To see a complete list of test options, run:: $ make check TESTSUITEFLAGS=--help The results of a testing run are reported in ``tests/testsuite.log``. Report report test failures as bugs and include the ``testsuite.log`` in your report. .. note:: Sometimes a few tests may fail on some runs but not others. This is usually a bug in the testsuite, not a bug in Open vSwitch itself. If you find that a test fails intermittently, please report it, since the developers may not have noticed. You can make the testsuite automatically rerun tests that fail, by adding ``RECHECK=yes`` to the ``make`` command line, e.g.:: $ make check TESTSUITEFLAGS=-j8 RECHECK=yes .. _testing-coverage: Coverage ~~~~~~~~ If the build was configured with ``--enable-coverage`` and the ``lcov`` utility is installed, you can run the testsuite and generate a code coverage report by using the ``check-lcov`` target:: $ make check-lcov All the same options are avaiable via TESTSUITEFLAGS. For example:: $ make check-lcov TESTSUITEFLAGS='-j8 -k ovn' .. _testing-valgrind: Valgrind ~~~~~~~~ If you have ``valgrind`` installed, you can run the testsuite under valgrind by using the ``check-valgrind`` target:: $ make check-valgrind When you do this, the "valgrind" results for test ```` are reported in files named ``tests/testsuite.dir//valgrind.*``. To test the testsuite of kernel datapath under valgrind, you can use the ``check-kernel-valgrind`` target and find the "valgrind" results under directory ``tests/system-kmod-testsuite.dir/``. All the same options are available via TESTSUITEFLAGS. .. hint:: You may find that the valgrind results are easier to interpret if you put ``-q`` in ``~/.valgrindrc``, since that reduces the amount of output. OFTest ~~~~~~ OFTest is an OpenFlow protocol testing suite. Open vSwitch includes a Makefile target to run OFTest with Open vSwitch in "dummy mode". In this mode of testing, no packets travel across physical or virtual networks. Instead, Unix domain sockets stand in as simulated networks. This simulation is imperfect, but it is much easier to set up, does not require extra physical or virtual hardware, and does not require supervisor privileges. To run OFTest with Open vSwitch, you must obtain a copy of OFTest and install its prerequisites. You need a copy of OFTest that includes commit 406614846c5 (make ovs-dummy platform work again). This commit was merged into the OFTest repository on Feb 1, 2013, so any copy of OFTest more recent than that should work. Testing OVS in dummy mode does not require root privilege, so you may ignore that requirement. Optionally, add the top-level OFTest directory (containing the ``oft`` program) to your ``$PATH``. This slightly simplifies running OFTest later. To run OFTest in dummy mode, run the following command from your Open vSwitch build directory:: $ make check-oftest OFT= where ```` is the absolute path to the ``oft`` program in OFTest. If you added "oft" to your $PATH, you may omit the OFT variable assignment By default, ``check-oftest`` passes ``oft`` just enough options to enable dummy mode. You can use ``OFTFLAGS`` to pass additional options. For example, to run just the ``basic.Echo`` test instead of all tests (the default) and enable verbose logging, run:: $ make check-oftest OFT= OFTFLAGS='--verbose -T basic.Echo' If you use OFTest that does not include commit 4d1f3eb2c792 (oft: change default port to 6653), merged into the OFTest repository in October 2013, then you need to add an option to use the IETF-assigned controller port:: $ make check-oftest OFT= OFTFLAGS='--port=6653' Interpret OFTest results cautiously. Open vSwitch can fail a given test in OFTest for many reasons, including bugs in Open vSwitch, bugs in OFTest, bugs in the "dummy mode" integration, and differing interpretations of the OpenFlow standard and other standards. .. note:: Open vSwitch has not been validated against OFTest. Report test failures that you believe to represent bugs in Open vSwitch. Include the precise versions of Open vSwitch and OFTest in your bug report, plus any other information needed to reproduce the problem. Ryu ~~~ Ryu is an OpenFlow controller written in Python that includes an extensive OpenFlow testsuite. Open vSwitch includes a Makefile target to run Ryu in "dummy mode". See `OFTest`_ above for an explanation of dummy mode. To run Ryu tests with Open vSwitch, first read and follow the instructions under **Testing** above. Second, obtain a copy of Ryu, install its prerequisites, and build it. You do not need to install Ryu (some of the tests do not get installed, so it does not help). To run Ryu tests, run the following command from your Open vSwitch build directory:: $ make check-ryu RYUDIR= where ```` is the absolute path to the root of the Ryu source distribution. The default ```` is ``$srcdir/../ryu`` where ``$srcdir`` is your Open vSwitch source directory. If this is correct, omit ``RYUDIR`` .. note:: Open vSwitch has not been validated against Ryu. Report test failures that you believe to represent bugs in Open vSwitch. Include the precise versions of Open vSwitch and Ryu in your bug report, plus any other information needed to reproduce the problem. .. _datapath-testing: Datapath testing ~~~~~~~~~~~~~~~~ Open vSwitch includes a suite of tests specifically for datapath functionality, which can be run against the userspace or kernel datapaths. If you are developing datapath features, it is recommended that you use these tests and build upon them to verify your implementation. The datapath tests make some assumptions about the environment. They must be run under root privileges on a Linux system with support for network namespaces. For ease of use, the OVS source tree includes a vagrant box to invoke these tests. Running the tests inside Vagrant provides kernel isolation, protecting your development host from kernel panics or configuration conflicts in the testsuite. If you wish to run the tests without using the vagrant box, there are further instructions below. Vagrant +++++++ .. important:: Requires Vagrant (version 1.7.0 or later) and a compatible hypervisor .. note:: You must bootstrap and configure the sources (see doc:`/intro/install/general`) before you run the steps described here. A Vagrantfile is provided allowing to compile and provision the source tree as found locally in a virtual machine using the following command:: $ vagrant up This will bring up a Fedora 23 VM by default. If you wish to use a different box or a vagrant backend not supported by the default box, the ``Vagrantfile`` can be modified to use a different box as base. The VM can be reprovisioned at any time:: $ vagrant provision OVS out-of-tree compilation environment can be set up with:: $ ./boot.sh $ vagrant provision --provision-with configure_ovs,build_ovs This will set up an out-of-tree build environment inside the VM in ``/root/build``. The source code can be found in ``/vagrant``. To recompile and reinstall OVS in the VM using RPM:: $ ./boot.sh $ vagrant provision --provision-with configure_ovs,install_rpm Two provisioners are included to run system tests with the OVS kernel module or with a userspace datapath. This tests are different from the self-tests mentioned above. To run them:: $ ./boot.sh $ vagrant provision --provision-with \ configure_ovs,test_ovs_kmod,test_ovs_system_userspace The results of the testsuite reside in the VM root user's home directory:: $ vagrant ssh $ sudo -s $ cd /root/build $ ls tests/system* Native ++++++ The datapath testsuite as invoked by Vagrant above may also be run manually on a Linux system with root privileges. Make sure, no other Open vSwitch instance is running on the test suite. These tests may take several minutes to complete, and cannot be run in parallel. Userspace datapath ''''''''''''''''''' To invoke the datapath testsuite with the userspace datapath, run:: $ make check-system-userspace The results of the testsuite are in ``tests/system-userspace-testsuite.dir``. Kernel datapath ''''''''''''''' Make targets are also provided for testing the Linux kernel module. Note that these tests operate by inserting modules into the running Linux kernel, so if the tests are able to trigger a bug in the OVS kernel module or in the upstream kernel then the kernel may panic. To run the testsuite against the kernel module which is currently installed on your system, run:: $ make check-kernel To install the kernel module from the current build directory and run the testsuite against that kernel module:: $ make check-kmod The results of the testsuite are in ``tests/system-kmod-testsuite.dir``. .. _testing-static-analysis: Static Code Analysis ~~~~~~~~~~~~~~~~~~~~ Static Analysis is a method of debugging Software by examining code rather than actually executing it. This can be done through 'scan-build' commandline utility which internally uses clang (or) gcc to compile the code and also invokes a static analyzer to do the code analysis. At the end of the build, the reports are aggregated in to a common folder and can later be analyzed using 'scan-view'. Open vSwitch includes a Makefile target to trigger static code analysis:: $ ./boot.sh $ ./configure CC=clang # clang # or $ ./configure CC=gcc CFLAGS="-std=gnu99" # gcc $ make clang-analyze You should invoke scan-view to view analysis results. The last line of output from ``clang-analyze`` will list the command (containing results directory) that you should invoke to view the results on a browser. Continuous Integration with Travis CI ------------------------------------- A .travis.yml file is provided to automatically build Open vSwitch with various build configurations and run the testsuite using Travis CI. Builds will be performed with gcc, sparse and clang with the -Werror compiler flag included, therefore the build will fail if a new warning has been introduced. The CI build is triggered via git push (regardless of the specific branch) or pull request against any Open vSwitch GitHub repository that is linked to travis-ci. Instructions to setup travis-ci for your GitHub repository: 1. Go to https://travis-ci.org/ and sign in using your GitHub ID. 2. Go to the "Repositories" tab and enable the ovs repository. You may disable builds for pushes or pull requests. 3. In order to avoid forks sending build failures to the upstream mailing list, the notification email recipient is encrypted. If you want to receive email notification for build failures, replace the the encrypted string: 1. Install the travis-ci CLI (Requires ruby >=2.0): gem install travis 2. In your Open vSwitch repository: travis encrypt mylist@mydomain.org 3. Add/replace the notifications section in .travis.yml and fill in the secure string as returned by travis encrypt:: notifications: email: recipients: - secure: "....." .. note:: You may remove/omit the notifications section to fall back to default notification behaviour which is to send an email directly to the author and committer of the failing commit. Note that the email is only sent if the author/committer have commit rights for the particular GitHub repository. 4. Pushing a commit to the repository which breaks the build or the testsuite will now trigger a email sent to mylist@mydomain.org vsperf ------ The vsperf project aims to develop a vSwitch test framework that can be used to validate the suitability of different vSwitch implementations in a telco deployment environment. More information can be found on the `OPNFV wiki`_. .. _OPNFV wiki: https://wiki.opnfv.org/display/vsperf/VSperf+Home ovs-2.9.0/Documentation/topics/tracing.rst000066400000000000000000000132271324262074100206320ustar00rootroot00000000000000.. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. Convention for heading levels in Open vSwitch documentation: ======= Heading 0 (reserved for the title in a document) ------- Heading 1 ~~~~~~~ Heading 2 +++++++ Heading 3 ''''''' Heading 4 Avoid deeper levels because they do not render well. =================================== Tracing packets inside Open vSwitch =================================== Open vSwitch (OVS) is a programmable software switch that can execute actions at per packet level. This document explains how to use the tracing tool to know what is happening with packets as they go through the data plane processing. The `ovs-vswitchd(8)`_ manpage describes basic usage of the ofproto/trace command used for tracing in Open vSwitch. For a tool with a goal similar to ofproto/trace for tracing packets through OVN logical switches, see `ovn-trace(8)`_. Packet Tracing -------------- In order to understand the tool, let's use the following flows as an example: table=3,ip,tcp,tcp_dst=80,action=output:2 table=2,ip,tcp,tcp_dst=22,action=output:1 table=0,in_port=3,ip,nw_src=192.0.2.0/24,action=resubmit(,2) table=0,in_port=3,ip,nw_src=198.51.100.0/24,action=resubmit(,3) .. note:: If you can't use a "real" OVS setup you can use ``ovs-sandbox``, as described in :doc:`/tutorials/ovs-advanced`, which also provides additional tracing examples. The first line adds a rule in table 3 matching on TCP/IP packet with destination port 80 (HTTP). If a packet matches, the action is to output the packet on OpenFlow port 2. The second line is similar but matches on destination port 22. If a packet matches, the action is to output the packet on OpenFlow port 1. The next two lines matches on source IP addresses. If there is a match, the packet is submitted to table indicated as parameter to the resubmit() action. Now let's see if a packet from IP address 192.0.2.1 and destination port 22 would really go to OpenFlow port 1:: $ ovs-appctl ofproto/trace br0 in_port=3,tcp,nw_src=192.0.2.2,tcp_dst=22 Flow: tcp,in_port=3,vlan_tci=0x0000,dl_src=00:00:00:00:00:00,dl_dst=00:00:00:00:00:00,nw_src=192.0.2.2,nw_dst=0.0.0.0,nw_tos=0,nw_ecn=0,nw_ttl=0,tp_src=0,tp_dst=22,tcp_flags=0 bridge("br0") ------------- 0. ip,in_port=3,nw_src=192.0.2.0/24, priority 32768 resubmit(,2) 2. tcp,tp_dst=22, priority 32768 output:1 Final flow: unchanged Megaflow: recirc_id=0,tcp,in_port=3,nw_src=192.0.2.0/24,nw_frag=no,tp_dst=22 Datapath actions: 1 The first line is the trace command. The br0 is the bridge where the packet is going through. The next arguments describe the packet itself. For instance, the nw_src matches with the IP source address. All the packet fields are well documented in the `ovs-fields(7)`_ man-page. The second line shows the flow extracted from the packet described in the command line. Unspecified packet fields are zeroed. The second group of lines shows the packet's trip through bridge br0. We see, in table 0, the OpenFlow flow that the fields matched, along with its priority, followed by its actions, one per line. In this case, we see that this packet matches the flow that resubmit those packets to table 2. The "resubmit" causes a second lookup in OpenFlow table 2, described by the block of text that starts with "2.". In the second lookup we see that this packet matches the rule that outputs those packets to OpenFlow port #1. In summary, it is possible to follow the flow entries and actions until the final decision is made. At the end, the trace tool shows the Megaflow which matches on all relevant fields followed by the data path actions. Let's see what happens with the same packet but with another TCP destination port:: $ ovs-appctl ofproto/trace br0 in_port=3,tcp,nw_src=192.0.2.2,tcp_dst=80 Flow: tcp,in_port=3,vlan_tci=0x0000,dl_src=00:00:00:00:00:00,dl_dst=00:00:00:00:00:00,nw_src=192.0.2.2,nw_dst=0.0.0.0,nw_tos=0,nw_ecn=0,nw_ttl=0,tp_src=0,tp_dst=80,tcp_flags=0 bridge("br0") ------------- 0. ip,in_port=3,nw_src=192.0.2.0/24, priority 32768 resubmit(,2) 2. No match. drop Final flow: unchanged Megaflow: recirc_id=0,tcp,in_port=3,nw_src=192.0.2.0/24,nw_frag=no,tp_dst=0x40/0xffc0 Datapath actions: drop In the second group of lines, in table 0, you can see that the packet matches with the rule because of the source IP address, so it is resubmitted to the table 2 as before. However, it doesn't match any rule there. When the packet doesn't match any rule in the flow tables, it is called a table miss. The virtual switch table miss behavior can be configured and it depends on the OpenFlow version being used. In this example the default action was to drop the packet. Credits ------- This document is heavily based on content from Flavio Bruno Leitner at Red Hat: - https://developers.redhat.com/blog/2016/10/12/tracing-packets-inside-open-vswitch/ .. _ovs-vswitchd(8): http://openvswitch.org/support/dist-docs/ovs-vswitchd.8.html .. _ovs-fields(7): http://openvswitch.org/support/dist-docs/ovs-fields.7.pdf .. _ovn-trace(8): http://openvswitch.org/support/dist-docs/ovn-trace.8.html ovs-2.9.0/Documentation/topics/windows.rst000066400000000000000000000567511324262074100207060ustar00rootroot00000000000000.. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. Convention for heading levels in Open vSwitch documentation: ======= Heading 0 (reserved for the title in a document) ------- Heading 1 ~~~~~~~ Heading 2 +++++++ Heading 3 ''''''' Heading 4 Avoid deeper levels because they do not render well. ===================== OVS-on-Hyper-V Design ===================== This document provides details of the effort to develop Open vSwitch on Microsoft Hyper-V. This document should give enough information to understand the overall design. .. note:: The userspace portion of the OVS has been ported to Hyper-V in a separate effort, and committed to the openvswitch repo. This document will mostly emphasize on the kernel driver, though we touch upon some of the aspects of userspace as well. Background Info --------------- Microsoft’s hypervisor solution - Hyper-V [1]_ implements a virtual switch that is extensible and provides opportunities for other vendors to implement functional extensions [2]_. The extensions need to be implemented as NDIS drivers that bind within the extensible switch driver stack provided. The extensions can broadly provide the functionality of monitoring, modifying and forwarding packets to destination ports on the Hyper-V extensible switch. Correspondingly, the extensions can be categorized into the following types and provide the functionality noted: * Capturing extensions: monitoring packets * Filtering extensions: monitoring, modifying packets * Forwarding extensions: monitoring, modifying, forwarding packets As can be expected, the kernel portion (datapath) of OVS on Hyper-V solution will be implemented as a forwarding extension. In Hyper-V, the virtual machine is called the Child Partition. Each VIF or physical NIC on the Hyper-V extensible switch is attached via a port. Each port is both on the ingress path or the egress path of the switch. The ingress path is used for packets being sent out of a port, and egress is used for packet being received on a port. By design, NDIS provides a layered interface. In this layered interface, higher level layers call into lower level layers, in the ingress path. In the egress path, it is the other way round. In addition, there is a object identifier (OID) interface for control operations Eg. addition of a port. The workflow for the calls is similar in nature to the packets, where higher level layers call into the lower level layers. A good representational diagram of this architecture is in [4]_. Windows Filtering Platform (WFP) [5]_ is a platform implemented on Hyper-V that provides APIs and services for filtering packets. WFP has been utilized to filter on some of the packets that OVS is not equipped to handle directly. More details in later sections. IP Helper [6]_ is a set of API available on Hyper-V to retrieve information related to the network configuration information on the host machine. IP Helper has been used to retrieve some of the configuration information that OVS needs. Design ------ :: Various blocks of the OVS Windows implementation +-------------------------------+ | | | CHILD PARTITION | | | +------+ +--------------+ | +-----------+ +------------+ | | | | | | | | | | | | ovs- | | OVS- | | | Virtual | | Virtual | | | *ctl | | USERSPACE | | | Machine #1| | Machine #2 | | | | | DAEMON | | | | | | | +------+-++---+---------+ | +--+------+-+ +----+------++ | +--------+ | dpif- | | netdev- | | |VIF #1| |VIF #2| | |Physical| | netlink | | windows | | +------+ +------+ | | NIC | +---------+ +---------+ | || /\ | +--------+ User /\ /\ | || *#1* *#4* || | /\ =========||=========||============+------||-------------------||--+ || Kernel || || \/ || ||=====/ \/ \/ +-----+ +-----+ *#5* +-------------------------------+ | | | | | +----------------------+ | | | | | | | OVS Pseudo Device | | | | | | | +----------------------+ | | | | | | | Netlink Impl. | | | | | | | ----------------- | | I | | | | +------------+ | | N | | E | | | Flowtable | +------------+ | | G | | G | | +------------+ | Packet | |*#2*| R | | R | | +--------+ | Processing | |<=> | E | | E | | | WFP | | | | | S | | S | | | Driver | +------------+ | | S | | S | | +--------+ | | | | | | | | | | | | OVS FORWARDING EXTENSION | | | | | +-------------------------------+ +-----+-----------------+-----+ |HYPER-V Extensible Switch *#3| +-----------------------------+ NDIS STACK This diagram shows the various blocks involved in the OVS Windows implementation, along with some of the components available in the NDIS stack, and also the virtual machines. The workflow of a packet being transmitted from a VIF out and into another VIF and to a physical NIC is also shown. Later on in this section, we will discuss the flow of a packet at a high level. The figure gives a general idea of where the OVS userspace and the kernel components fit in, and how they interface with each other. The kernel portion (datapath) of OVS on Hyper-V solution has be implemented as a forwarding extension roughly implementing the following sub-modules/functionality. Details of each of these sub-components in the kernel are contained in later sections: * Interfacing with the NDIS stack * Netlink message parser * Netlink sockets * Switch/Datapath management * Interfacing with userspace portion of the OVS solution to implement the necessary functionality that userspace needs * Port management * Flowtable/Actions/packet forwarding * Tunneling * Event notifications The datapath for the OVS on Linux is a kernel module, and cannot be directly ported since there are significant differences in architecture even though the end functionality provided would be similar. Some examples of the differences are: * Interfacing with the NDIS stack to hook into the NDIS callbacks for functionality such as receiving and sending packets, packet completions, OIDs used for events such as a new port appearing on the virtual switch. * Interface between the userspace and the kernel module. * Event notifications are significantly different. * The communication interface between DPIF and the kernel module need not be implemented in the way OVS on Linux does. That said, it would be advantageous to have a similar interface to the kernel module for reasons of readability and maintainability. * Any licensing issues of using Linux kernel code directly. Due to these differences, it was a straightforward decision to develop the datapath for OVS on Hyper-V from scratch rather than porting the one on Linux. A re-development focused on the following goals: * Adhere to the existing requirements of userspace portion of OVS (such as ovs-vswitchd), to minimize changes in the userspace workflow. * Fit well into the typical workflow of a Hyper-V extensible switch forwarding extension. The userspace portion of the OVS solution is mostly POSIX code, and not very Linux specific. Majority of the userspace code does not interface directly with the kernel datapath and was ported independently of the kernel datapath effort. As explained in the OVS porting design document [7]_, DPIF is the portion of userspace that interfaces with the kernel portion of the OVS. The interface that each DPIF provider has to implement is defined in ``dpif-provider.h`` [3]_. Though each platform is allowed to have its own implementation of the DPIF provider, it was found, via community feedback, that it is desired to share code whenever possible. Thus, the DPIF provider for OVS on Hyper-V shares code with the DPIF provider on Linux. This interface is implemented in ``dpif-netlink.c``. We'll elaborate more on kernel-userspace interface in a dedicated section below. Here it suffices to say that the DPIF provider implementation for Windows is netlink-based and shares code with the Linux one. Kernel Module (Datapath) ------------------------ Interfacing with the NDIS Stack ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ For each virtual switch on Hyper-V, the OVS extensible switch extension can be enabled/disabled. We support enabling the OVS extension on only one switch. This is consistent with using a single datapath in the kernel on Linux. All the physical adapters are connected as external adapters to the extensible switch. When the OVS switch extension registers itself as a filter driver, it also registers callbacks for the switch/port management and datapath functions. In other words, when a switch is created on the Hyper-V root partition (host), the extension gets an activate callback upon which it can initialize the data structures necessary for OVS to function. Similarly, there are callbacks for when a port gets added to the Hyper-V switch, and an External Network adapter or a VM Network adapter is connected/disconnected to the port. There are also callbacks for when a VIF (NIC of a child partition) send out a packet, or a packet is received on an external NIC. As shown in the figures, an extensible switch extension gets to see a packet sent by the VM (VIF) twice - once on the ingress path and once on the egress path. Forwarding decisions are to be made on the ingress path. Correspondingly, we will be hooking onto the following interfaces: * Ingress send indication: intercept packets for performing flow based forwarding.This includes straight forwarding to output ports. Any packet modifications needed to be performed are done here either inline or by creating a new packet. A forwarding action is performed as the flow actions dictate. * Ingress completion indication: cleanup and free packets that we generated on the ingress send path, pass-through for packets that we did not generate. * Egress receive indication: pass-through. * Egress completion indication: pass-through. Interfacing with OVS Userspace ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ We have implemented a pseudo device interface for letting OVS userspace talk to the OVS kernel module. This is equivalent to the typical character device interface on POSIX platforms where we can register custom functions for read, write and ioctl functionality. The pseudo device supports a whole bunch of ioctls that netdev and DPIF on OVS userspace make use of. Netlink Message Parser ~~~~~~~~~~~~~~~~~~~~~~ The communication between OVS userspace and OVS kernel datapath is in the form of Netlink messages [1]_, [8]_. More details about this are provided below. In the kernel, a full fledged netlink message parser has been implemented along the lines of the netlink message parser in OVS userspace. In fact, a lot of the code is ported code. On the lines of ``struct ofpbuf`` in OVS userspace, a managed buffer has been implemented in the kernel datapath to make it easier to parse and construct netlink messages. Netlink Sockets ~~~~~~~~~~~~~~~ On Linux, OVS userspace utilizes netlink sockets to pass back and forth netlink messages. Since much of userspace code including DPIF provider in dpif-netlink.c (formerly dpif-linux.c) has been reused, pseudo-netlink sockets have been implemented in OVS userspace. As it is known, Windows lacks native netlink socket support, and also the socket family is not extensible either. Hence it is not possible to provide a native implementation of netlink socket. We emulate netlink sockets in lib/netlink-socket.c and support all of the nl_* APIs to higher levels. The implementation opens a handle to the pseudo device for each netlink socket. Some more details on this topic are provided in the userspace section on netlink sockets. Typical netlink semantics of read message, write message, dump, and transaction have been implemented so that higher level layers are not affected by the netlink implementation not being native. Switch/Datapath Management ~~~~~~~~~~~~~~~~~~~~~~~~~~ As explained above, we hook onto the management callback functions in the NDIS interface for when to initialize the OVS data structures, flow tables etc. Some of this code is also driven by OVS userspace code which sends down ioctls for operations like creating a tunnel port etc. Port Management ~~~~~~~~~~~~~~~ As explained above, we hook onto the management callback functions in the NDIS interface to know when a port is added/connected to the Hyper-V switch. We use these callbacks to initialize the port related data structures in OVS. Also, some of the ports are tunnel ports that don’t exist on the Hyper-V switch and get added from OVS userspace. In order to identify a Hyper-V port, we use the value of 'FriendlyName' field in each Hyper-V port. We call this the "OVS-port-name". The idea is that OVS userspace sets 'OVS-port-name' in each Hyper-V port to the same value as the 'name' field of the 'Interface' table in OVSDB. When OVS userspace calls into the kernel datapath to add a port, we match the name of the port with the 'OVS-port-name' of a Hyper-V port. We maintain separate hash tables, and separate counters for ports that have been added from the Hyper-V switch, and for ports that have been added from OVS userspace. Flowtable/Actions/Packet Forwarding ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The flowtable and flow actions based packet forwarding is the core of the OVS datapath functionality. For each packet on the ingress path, we consult the flowtable and execute the corresponding actions. The actions can be limited to simple forwarding to a particular destination port(s), or more commonly involves modifying the packet to insert a tunnel context or a VLAN ID, and thereafter forwarding to the external port to send the packet to a destination host. Tunneling ~~~~~~~~~ We make use of the Internal Port on a Hyper-V switch for implementing tunneling. The Internal Port is a virtual adapter that is exposed on the Hyper- V host, and connected to the Hyper-V switch. Basically, it is an interface between the host and the virtual switch. The Internal Port acts as the Tunnel end point for the host (aka VTEP), and holds the VTEP IP address. Tunneling ports are not actual ports on the Hyper-V switch. These are virtual ports that OVS maintains and while executing actions, if the outport is a tunnel port, we short circuit by performing the encapsulation action based on the tunnel context. The encapsulated packet gets forwarded to the external port, and appears to the outside world as though it was set from the VTEP. Similarly, when a tunneled packet enters the OVS from the external port bound to the internal port (VTEP), and if yes, we short circuit the path, and directly forward the inner packet to the destination port (mostly a VIF, but dictated by the flow). We leverage the Windows Filtering Platform (WFP) framework to be able to receive tunneled packets that cannot be decapsulated by OVS right away. Currently, fragmented IP packets fall into that category, and we leverage the code in the host IP stack to reassemble the packet, and performing decapsulation on the reassembled packet. We'll also be using the IP helper library to provide us IP address and other information corresponding to the Internal port. Event Notifications ~~~~~~~~~~~~~~~~~~~ The pseudo device interface described above is also used for providing event notifications back to OVS userspace. A shared memory/overlapped IO model is used. Userspace Components ~~~~~~~~~~~~~~~~~~~~ The userspace portion of the OVS solution is mostly POSIX code, and not very Linux specific. Majority of the userspace code does not interface directly with the kernel datapath and was ported independently of the kernel datapath effort. In this section, we cover the userspace components that interface with the kernel datapath. As explained earlier, OVS on Hyper-V shares the DPIF provider implementation with Linux. The DPIF provider on Linux uses netlink sockets and netlink messages. Netlink sockets and messages are extensively used on Linux to exchange information between userspace and kernel. In order to satisfy these dependencies, netlink socket (pseudo and non-native) and netlink messages are implemented on Hyper-V. The following are the major advantages of sharing DPIF provider code: 1. Maintenance is simpler: Any change made to the interface defined in dpif-provider.h need not be propagated to multiple implementations. Also, developers familiar with the Linux implementation of the DPIF provider can easily ramp on the Hyper-V implementation as well. 2. Netlink messages provides inherent advantages: Netlink messages are known for their extensibility. Each message is versioned, so the provided data structures offer a mechanism to perform version checking and forward/backward compatibility with the kernel module. Netlink Sockets ~~~~~~~~~~~~~~~ As explained in other sections, an emulation of netlink sockets has been implemented in ``lib/netlink-socket.c`` for Windows. The implementation creates a handle to the OVS pseudo device, and emulates netlink socket semantics of receive message, send message, dump, and transact. Most of the ``nl_*`` functions are supported. The fact that the implementation is non-native manifests in various ways. One example is that PID for the netlink socket is not automatically assigned in userspace when a handle is created to the OVS pseudo device. There's an extra command (defined in ``OvsDpInterfaceExt.h``) that is used to grab the PID generated in the kernel. DPIF Provider ~~~~~~~~~~~~~ As has been mentioned in earlier sections, the netlink socket and netlink message based DPIF provider on Linux has been ported to Windows. Most of the code is common. Some divergence is in the code to receive packets. The Linux implementation uses epoll() [9]_ which is not natively supported on Windows. netdev-windows ~~~~~~~~~~~~~~ We have a Windows implementation of the interface defined in ``lib/netdev-provider.h``. The implementation provides functionality to get extended information about an interface. It is limited in functionality compared to the Linux implementation of the netdev provider and cannot be used to add any interfaces in the kernel such as a tap interface or to send/receive packets. The netdev-windows implementation uses the datapath interface extensions defined in ``datapath-windows/include/OvsDpInterfaceExt.h``. Powershell Extensions to Set ``OVS-port-name`` ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ As explained in the section on "Port management", each Hyper-V port has a 'FriendlyName' field, which we call as the "OVS-port-name" field. We have implemented powershell command extensions to be able to set the "OVS-port-name" of a Hyper-V port. Kernel-Userspace Interface -------------------------- openvswitch.h and OvsDpInterfaceExt.h ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Since the DPIF provider is shared with Linux, the kernel datapath provides the same interface as the Linux datapath. The interface is defined in ``datapath/linux/compat/include/linux/openvswitch.h``. Derivatives of this interface file are created during OVS userspace compilation. The derivative for the kernel datapath on Hyper-V is provided in ``datapath-windows/include/OvsDpInterface.h``. That said, there are Windows specific extensions that are defined in the interface file ``datapath-windows/include/OvsDpInterfaceExt.h``. Flow of a Packet ---------------- Figure 2 shows the numbered steps in which a packets gets sent out of a VIF and is forwarded to another VIF or a physical NIC. As mentioned earlier, each VIF is attached to the switch via a port, and each port is both on the ingress and egress path of the switch, and depending on whether a packet is being transmitted or received, one of the paths gets used. In the figure, each step n is annotated as ``#n`` The steps are as follows: 1. When a packet is sent out of a VIF or an physical NIC or an internal port, the packet is part of the ingress path. 2. The OVS kernel driver gets to intercept this packet. a. OVS looks up the flows in the flowtable for this packet, and executes the corresponding action. b. If there is not action, the packet is sent up to OVS userspace to examine the packet and figure out the actions. c. Userspace executes the packet by specifying the actions, and might also insert a flow for such a packet in the future. d. The destination ports are added to the packet and sent down to the Hyper- V switch. 3. The Hyper-V forwards the packet to the destination ports specified in the packet, and sends it out on the egress path. 4. The packet gets forwarded to the destination VIF. 5. It might also get forwarded to a physical NIC as well, if the physical NIC has been added as a destination port by OVS. Build/Deployment ---------------- The userspace components added as part of OVS Windows implementation have been integrated with autoconf, and can be built using the steps mentioned in the BUILD.Windows file. Additional targets need to be specified to make. The OVS kernel code is part of a Visual Studio 2013 solution, and is compiled from the IDE. There are plans in the future to move this to a compilation mode such that we can compile it without an IDE as well. Once compiled, we have an install script that can be used to load the kernel driver. References ---------- .. [1] Hyper-V Extensible Switch https://msdn.microsoft.com/windows/hardware/drivers/network/hyper-v-extensible-switch .. [2] Hyper-V Extensible Switch Extensions https://msdn.microsoft.com/windows/hardware/drivers/network/hyper-v-extensible-switch-extensions .. [3] DPIF Provider http://openvswitch.sourcearchive.com/documentation/1.1.0-1/dpif-provider_8h_source.html .. [4] Hyper-V Extensible Switch Components https://msdn.microsoft.com/windows/hardware/drivers/network/hyper-v-extensible-switch-components .. [5] Windows Filtering Platform https://msdn.microsoft.com/en-us/library/windows/desktop/aa366510(v=vs.85).aspx .. [6] IP Helper https://msdn.microsoft.com/windows/hardware/drivers/network/ip-helper .. [7] How to Port Open vSwitch to New Software or Hardware :doc:`porting` .. [8] Netlink https://en.wikipedia.org/wiki/Netlink .. [9] epoll https://en.wikipedia.org/wiki/Epoll ovs-2.9.0/Documentation/tutorials/000077500000000000000000000000001324262074100171715ustar00rootroot00000000000000ovs-2.9.0/Documentation/tutorials/faucet.rst000066400000000000000000001724751324262074100212120ustar00rootroot00000000000000.. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. Convention for heading levels in Open vSwitch documentation: ======= Heading 0 (reserved for the title in a document) ------- Heading 1 ~~~~~~~ Heading 2 +++++++ Heading 3 ''''''' Heading 4 Avoid deeper levels because they do not render well. =================== OVS Faucet Tutorial =================== This tutorial demonstrates how Open vSwitch works with a general-purpose OpenFlow controller, using the Faucet controller as a simple way to get started. It was tested with the "master" branch of Open vSwitch and version 1.6.15 of Faucet. It does not use advanced or recently added features in OVS or Faucet, so other versions of both pieces of software are likely to work equally well. The goal of the tutorial is to demonstrate Open vSwitch and Faucet in an end-to-end way, that is, to show how it works from the Faucet controller configuration at the top, through the OpenFlow flow table, to the datapath processing. Along the way, in addition to helping to understand the architecture at each level, we discuss performance and troubleshooting issues. We hope that this demonstration makes it easier for users and potential users to understand how Open vSwitch works and how to debug and troubleshoot it. We provide enough details in the tutorial that you should be able to fully follow along by following the instructions. Setting Up OVS -------------- This section explains how to set up Open vSwitch for the purpose of using it with Faucet for the tutorial. You might already have Open vSwitch installed on one or more computers or VMs, perhaps set up to control a set of VMs or a physical network. This is admirable, but we will be using Open vSwitch in a different way to set up a simulation environment called the OVS "sandbox". The sandbox does not use virtual machines or containers, which makes it more limited, but on the other hand it is (in this writer's opinion) easier to set up. There are two ways to start a sandbox: one that uses the Open vSwitch that is already installed on a system, and another that uses a copy of Open vSwitch that has been built but not yet installed. The latter is more often used and thus better tested, but both should work. The instructions below explain both approaches: 1. Get a copy of the Open vSwitch source repository using Git, then ``cd`` into the new directory:: $ git clone https://github.com/openvswitch/ovs.git $ cd ovs The default checkout is the master branch. You can check out a tag (such as v2.8.0) or a branch (such as origin/branch-2.8), if you prefer. 2. If you do not already have an installed copy of Open vSwitch on your system, or if you do not want to use it for the sandbox (the sandbox will not disturb the functionality of any existing switches), then proceed to step 3. If you do have an installed copy and you want to use it for the sandbox, try to start the sandbox by running:: $ tutorial/ovs-sandbox If it is successful, you will find yourself in a subshell environment, which is the sandbox (you can exit with ``exit`` or Control+D). If so, you're finished and do not need to complete the rest of the steps. If it fails, you can proceed to step 3 to build Open vSwitch anyway. 3. Before you build, you might want to check that your system meets the build requirements. Read :doc:`/intro/install/general` to find out. For this tutorial, there is no need to compile the Linux kernel module, or to use any of the optional libraries such as OpenSSL, DPDK, or libcap-ng. 4. Configure and build Open vSwitch:: $ ./boot.sh $ ./configure $ make -j4 5. Try out the sandbox by running:: $ make sandbox You can exit the sandbox with ``exit`` or Control+D. Setting up Faucet ----------------- This section explains how to get a copy of Faucet and set it up appropriately for the tutorial. There are many other ways to install Faucet, but this simple approach worked well for me. It has the advantage that it does not require modifying any system-level files or directories on your machine. It does, on the other hand, require Docker, so make sure you have it installed and working. It will be a little easier to go through the rest of the tutorial if you run these instructions in a separate terminal from the one that you're using for Open vSwitch, because it's often necessary to switch between one and the other. 1. Get a copy of the Faucet source repository using Git, then ``cd`` into the new directory:: $ git clone https://github.com/faucetsdn/faucet.git $ cd faucet At this point I checked out the latest tag:: $ latest_tag=$(git describe --tags $(git rev-list --tags --max-count=1)) $ git checkout $latest_tag 2. Build a docker container image:: $ docker build -t faucet/faucet . This will take a few minutes. 3. Create an installation directory under the ``faucet`` directory for the docker image to use:: $ mkdir inst The Faucet configuration will go in ``inst/faucet.yaml`` and its main log will appear in ``inst/faucet.log``. (The official Faucet installation instructions call to put these in ``/etc/ryu/faucet`` and ``/var/log/ryu/faucet``, respectively, but we avoid modifying these system directories.) 4. Create a container and start Faucet:: $ docker run -d --name faucet -v $(pwd)/inst/:/etc/ryu/faucet/ -v $(pwd)/inst/:/var/log/ryu/faucet/ -p 6653:6653 faucet/faucet 5. Look in ``inst/faucet.log`` to verify that Faucet started. It will probably start with an exception and traceback because we have not yet created ``inst/faucet.yaml``. 6. Later on, to make a new or updated Faucet configuration take effect quickly, you can run:: $ docker exec faucet pkill -HUP -f faucet.faucet Another way is to stop and start the Faucet container:: $ docker restart faucet You can also stop and delete the container; after this, to start it again, you need to rerun the ``docker run`` command:: $ docker stop faucet $ docker rm faucet Overview -------- Now that Open vSwitch and Faucet are ready, here's an overview of what we're going to do for the remainder of the tutorial: 1. Switching: Set up an L2 network with Faucet. 2. Routing: Route between multiple L3 networks with Faucet. 3. ACLs: Add and modify access control rules. At each step, we will take a look at how the features in question work from Faucet at the top to the data plane layer at the bottom. From the highest to lowest level, these layers and the software components that connect them are: Faucet. As the top level in the system, this is the authoritative source of the network configuration. Faucet connects to a variety of monitoring and performance tools, but we won't use them in this tutorial. Our main insights into the system will be through ``faucet.yaml`` for configuration and ``faucet.log`` to observe state, such as MAC learning and ARP resolution, and to tell when we've screwed up configuration syntax or semantics. The OpenFlow subsystem in Open vSwitch. OpenFlow is the protocol, standardized by the Open Networking Foundation, that controllers like Faucet use to control how Open vSwitch and other switches treat packets in the network. We will use ``ovs-ofctl``, a utility that comes with Open vSwitch, to observe and occasionally modify Open vSwitch's OpenFlow behavior. We will also use ``ovs-appctl``, a utility for communicating with ``ovs-vswitchd`` and other Open vSwitch daemons, to ask "what-if?" type questions. In addition, the OVS sandbox by default raises the Open vSwitch logging level for OpenFlow high enough that we can learn a great deal about OpenFlow behavior simply by reading its log file. Open vSwitch datapath. This is essentially a cache designed to accelerate packet processing. Open vSwitch includes a few different datapaths, such as one based on the Linux kernel and a userspace-only datapath (sometimes called the "DPDK" datapath). The OVS sandbox uses the latter, but the principles behind it apply equally well to other datapaths. At each step, we discuss how the design of each layer influences performance. We demonstrate how Open vSwitch features can be used to debug, troubleshoot, and understand the system as a whole. Switching --------- Layer-2 (L2) switching is the basis of modern networking. It's also very simple and a good place to start, so let's set up a switch with some VLANs in Faucet and see how it works at each layer. Begin by putting the following into ``inst/faucet.yaml``:: dps: switch-1: dp_id: 0x1 timeout: 3600 arp_neighbor_timeout: 3600 interfaces: 1: native_vlan: 100 2: native_vlan: 100 3: native_vlan: 100 4: native_vlan: 200 5: native_vlan: 200 vlans: 100: 200: This configuration file defines a single switch ("datapath" or "dp") named ``switch-1``. The switch has five ports, numbered 1 through 5. Ports 1, 2, and 3 are in VLAN 100, and ports 4 and 5 are in VLAN 2. Faucet can identify the switch from its datapath ID, which is defined to be 0x1. .. note:: This also sets high MAC learning and ARP timeouts. The defaults are 5 minutes and about 8 minutes, which are fine in production but sometimes too fast for manual experimentation. (Don't use a timeout bigger than about 65000 seconds because it will crash Faucet.) Now restart Faucet so that the configuration takes effect, e.g.:: $ docker restart faucet Assuming that the configuration update is successful, you should now see a new line at the end of ``inst/faucet.log``:: Jan 06 15:14:35 faucet INFO Add new datapath DPID 1 (0x1) Faucet is now waiting for a switch with datapath ID 0x1 to connect to it over OpenFlow, so our next step is to create a switch with OVS and make it connect to Faucet. To do that, switch to the terminal where you checked out OVS and start a sandbox with ``make sandbox`` or ``tutorial/ovs-sandbox`` (as explained earlier under `Setting Up OVS`_). You should see something like this toward the end of the output:: ---------------------------------------------------------------------- You are running in a dummy Open vSwitch environment. You can use ovs-vsctl, ovs-ofctl, ovs-appctl, and other tools to work with the dummy switch. Log files, pidfiles, and the configuration database are in the "sandbox" subdirectory. Exit the shell to kill the running daemons. blp@sigabrt:~/nicira/ovs/tutorial(0)$ Inside the sandbox, create a switch ("bridge") named ``br0``, set its datapath ID to 0x1, add simulated ports to it named ``p1`` through ``p5``, and tell it to connect to the Faucet controller. To make it easier to understand, we request for port ``p1`` to be assigned OpenFlow port 1, ``p2`` port 2, and so on. As a final touch, configure the controller to be "out-of-band" (this is mainly to avoid some annoying messages in the ``ovs-vswitchd`` logs; for more information, run ``man ovs-vswitchd.conf.db`` and search for ``connection_mode``):: $ ovs-vsctl add-br br0 \ -- set bridge br0 other-config:datapath-id=0000000000000001 \ -- add-port br0 p1 -- set interface p1 ofport_request=1 \ -- add-port br0 p2 -- set interface p2 ofport_request=2 \ -- add-port br0 p3 -- set interface p3 ofport_request=3 \ -- add-port br0 p4 -- set interface p4 ofport_request=4 \ -- add-port br0 p5 -- set interface p5 ofport_request=5 \ -- set-controller br0 tcp:127.0.0.1:6653 \ -- set controller br0 connection-mode=out-of-band .. note:: You don't have to run all of these as a single ``ovs-vsctl`` invocation. It is a little more efficient, though, and since it updates the OVS configuration in a single database transaction it means that, for example, there is never a time when the controller is set but it has not yet been configured as out-of-band. Now, if you look at ``inst/faucet.log`` again, you should see that Faucet recognized and configured the new switch and its ports:: Jan 06 15:17:10 faucet INFO DPID 1 (0x1) connected Jan 06 15:17:10 faucet.valve INFO DPID 1 (0x1) Cold start configuring DP Jan 06 15:17:10 faucet.valve INFO DPID 1 (0x1) Configuring VLAN 100 vid:100 ports:Port 1,Port 2,Port 3 Jan 06 15:17:10 faucet.valve INFO DPID 1 (0x1) Configuring VLAN 200 vid:200 ports:Port 4,Port 5 Jan 06 15:17:10 faucet.valve INFO DPID 1 (0x1) Port 1 up, configuring Jan 06 15:17:10 faucet.valve INFO DPID 1 (0x1) Port 2 up, configuring Jan 06 15:17:10 faucet.valve INFO DPID 1 (0x1) Port 3 up, configuring Jan 06 15:17:10 faucet.valve INFO DPID 1 (0x1) Port 4 up, configuring Jan 06 15:17:10 faucet.valve INFO DPID 1 (0x1) Port 5 up, configuring Over on the Open vSwitch side, you can see a lot of related activity if you take a look in ``sandbox/ovs-vswitchd.log``. For example, here is the basic OpenFlow session setup and Faucet's probe of the switch's ports and capabilities:: rconn|INFO|br0<->tcp:127.0.0.1:6653: connecting... vconn|DBG|tcp:127.0.0.1:6653: sent (Success): OFPT_HELLO (OF1.4) (xid=0x1): version bitmap: 0x01, 0x02, 0x03, 0x04, 0x05 vconn|DBG|tcp:127.0.0.1:6653: received: OFPT_HELLO (OF1.3) (xid=0x2f24810a): version bitmap: 0x01, 0x02, 0x03, 0x04 vconn|DBG|tcp:127.0.0.1:6653: negotiated OpenFlow version 0x04 (we support version 0x05 and earlier, peer supports version 0x04 and earlier) rconn|INFO|br0<->tcp:127.0.0.1:6653: connected vconn|DBG|tcp:127.0.0.1:6653: received: OFPT_ECHO_REQUEST (OF1.3) (xid=0x2f24810b): 0 bytes of payload vconn|DBG|tcp:127.0.0.1:6653: sent (Success): OFPT_ECHO_REPLY (OF1.3) (xid=0x2f24810b): 0 bytes of payload vconn|DBG|tcp:127.0.0.1:6653: received: OFPT_FEATURES_REQUEST (OF1.3) (xid=0x2f24810c): vconn|DBG|tcp:127.0.0.1:6653: sent (Success): OFPT_FEATURES_REPLY (OF1.3) (xid=0x2f24810c): dpid:0000000000000001 n_tables:254, n_buffers:0 capabilities: FLOW_STATS TABLE_STATS PORT_STATS GROUP_STATS QUEUE_STATS vconn|DBG|tcp:127.0.0.1:6653: received: OFPST_PORT_DESC request (OF1.3) (xid=0x2f24810d): port=ANY vconn|DBG|tcp:127.0.0.1:6653: sent (Success): OFPST_PORT_DESC reply (OF1.3) (xid=0x2f24810d): 1(p1): addr:aa:55:aa:55:00:14 config: PORT_DOWN state: LINK_DOWN speed: 0 Mbps now, 0 Mbps max 2(p2): addr:aa:55:aa:55:00:15 config: PORT_DOWN state: LINK_DOWN speed: 0 Mbps now, 0 Mbps max 3(p3): addr:aa:55:aa:55:00:16 config: PORT_DOWN state: LINK_DOWN speed: 0 Mbps now, 0 Mbps max 4(p4): addr:aa:55:aa:55:00:17 config: PORT_DOWN state: LINK_DOWN speed: 0 Mbps now, 0 Mbps max 5(p5): addr:aa:55:aa:55:00:18 config: PORT_DOWN state: LINK_DOWN speed: 0 Mbps now, 0 Mbps max LOCAL(br0): addr:c6:64:ff:59:48:41 config: PORT_DOWN state: LINK_DOWN speed: 0 Mbps now, 0 Mbps max After that, you can see Faucet delete all existing flows and then start adding new ones:: vconn|DBG|tcp:127.0.0.1:6653: received: OFPT_FLOW_MOD (OF1.3) (xid=0x2f24810e): DEL table:255 priority=0 actions=drop vconn|DBG|tcp:127.0.0.1:6653: received: OFPT_BARRIER_REQUEST (OF1.3) (xid=0x2f24810f): vconn|DBG|tcp:127.0.0.1:6653: sent (Success): OFPT_BARRIER_REPLY (OF1.3) (xid=0x2f24810f): vconn|DBG|tcp:127.0.0.1:6653: received: OFPT_FLOW_MOD (OF1.3) (xid=0x2f248110): ADD priority=0 cookie:0x5adc15c0 out_port:0 actions=drop vconn|DBG|tcp:127.0.0.1:6653: received: OFPT_FLOW_MOD (OF1.3) (xid=0x2f248111): ADD table:1 priority=0 cookie:0x5adc15c0 out_port:0 actions=drop ... OpenFlow Layer ~~~~~~~~~~~~~~ Let's take a look at the OpenFlow tables that Faucet set up. Before we do that, it's helpful to take a look at ``docs/architecture.rst`` in the Faucet documentation to learn how Faucet structures its flow tables. In summary, this document says: Table 0 Port-based ACLs Table 1 Ingress VLAN processing Table 2 VLAN-based ACLs Table 3 Ingress L2 processing, MAC learning Table 4 L3 forwarding for IPv4 Table 5 L3 forwarding for IPv6 Table 6 Virtual IP processing, e.g. for router IP addresses implemented by Faucet Table 7 Egress L2 processing Table 8 Flooding With that in mind, let's dump the flow tables. The simplest way is to just run plain ``ovs-ofctl dump-flows``:: $ ovs-ofctl dump-flows br0 If you run that bare command, it produces a lot of extra junk that makes the output harder to read, like statistics and "cookie" values that are all the same. In addition, for historical reasons ``ovs-ofctl`` always defaults to using OpenFlow 1.0 even though Faucet and most modern controllers use OpenFlow 1.3, so it's best to force it to use OpenFlow 1.3. We could throw in a lot of options to fix these, but we'll want to do this more than once, so let's start by defining a shell function for ourselves:: $ dump-flows () { ovs-ofctl -OOpenFlow13 --names --no-stat dump-flows "$@" \ | sed 's/cookie=0x5adc15c0, //' } Let's also define ``save-flows`` and ``diff-flows`` functions for later use:: $ save-flows () { ovs-ofctl -OOpenFlow13 --no-names --sort dump-flows "$@" } $ diff-flows () { ovs-ofctl -OOpenFlow13 diff-flows "$@" | sed 's/cookie=0x5adc15c0 //' } Now let's take a look at the flows we've got and what they mean, like this:: $ dump-flows br0 First, table 0 has a flow that just jumps to table 1 for each configured port, and drops other unrecognized packets. Presumably it will do more if we configured port-based ACLs:: priority=9099,in_port=p1 actions=goto_table:1 priority=9099,in_port=p2 actions=goto_table:1 priority=9099,in_port=p3 actions=goto_table:1 priority=9099,in_port=p4 actions=goto_table:1 priority=9099,in_port=p5 actions=goto_table:1 priority=0 actions=drop Table 1, for ingress VLAN processing, has a bunch of flows that drop inappropriate packets, such as LLDP and STP:: table=1, priority=9099,dl_dst=01:80:c2:00:00:00 actions=drop table=1, priority=9099,dl_dst=01:00:0c:cc:cc:cd actions=drop table=1, priority=9099,dl_type=0x88cc actions=drop Table 1 also has some more interesting flows that recognize packets without a VLAN header on each of our ports (``vlan_tci=0x0000/0x1fff``), push on the VLAN configured for the port, and proceed to table 3. Presumably these skip table 2 because we did not configure any VLAN-based ACLs. There is also a fallback flow to drop other packets, which in practice means that if any received packet already has a VLAN header then it will be dropped:: table=1, priority=9000,in_port=p1,vlan_tci=0x0000/0x1fff actions=push_vlan:0x8100,set_field:4196->vlan_vid,goto_table:3 table=1, priority=9000,in_port=p2,vlan_tci=0x0000/0x1fff actions=push_vlan:0x8100,set_field:4196->vlan_vid,goto_table:3 table=1, priority=9000,in_port=p3,vlan_tci=0x0000/0x1fff actions=push_vlan:0x8100,set_field:4196->vlan_vid,goto_table:3 table=1, priority=9000,in_port=p4,vlan_tci=0x0000/0x1fff actions=push_vlan:0x8100,set_field:4296->vlan_vid,goto_table:3 table=1, priority=9000,in_port=p5,vlan_tci=0x0000/0x1fff actions=push_vlan:0x8100,set_field:4296->vlan_vid,goto_table:3 table=1, priority=0 actions=drop .. note:: The syntax ``set_field:4196->vlan_vid`` is curious and somewhat misleading. OpenFlow 1.3 defines the ``vlan_vid`` field as a 13-bit field where bit 12 is set to 1 if the VLAN header is present. Thus, since 4196 is 0x1064, this action sets VLAN value 0x64, which in decimal is 100. Table 2 isn't used because there are no VLAN-based ACLs. It just has a drop flow:: table=2, priority=0 actions=drop Table 3 is used for MAC learning but the controller hasn't learned any MAC yet. It also drops some inappropriate packets such as those that claim to be from a broadcast source address (why not from all multicast source addresses, though?). We'll come back here later:: table=3, priority=9099,dl_src=ff:ff:ff:ff:ff:ff actions=drop table=3, priority=9001,dl_src=0e:00:00:00:00:01 actions=drop table=3, priority=0 actions=drop table=3, priority=9000 actions=CONTROLLER:96,goto_table:7 Tables 4, 5, and 6 aren't used because we haven't configured any routing:: table=4, priority=0 actions=drop table=5, priority=0 actions=drop table=6, priority=0 actions=drop Table 7 is used to direct packets to learned MACs but Faucet hasn't learned any MACs yet, so it just sends all the packets along to table 8:: table=7, priority=0 actions=drop table=7, priority=9000 actions=goto_table:8 Table 8 implements flooding, broadcast, and multicast. The flows for broadcast and flood are easy to understand: if the packet came in on a given port and needs to be flooded or broadcast, output it to all the other ports in the same VLAN:: table=8, priority=9008,in_port=p1,dl_vlan=100,dl_dst=ff:ff:ff:ff:ff:ff actions=pop_vlan,output:p2,output:p3 table=8, priority=9008,in_port=p2,dl_vlan=100,dl_dst=ff:ff:ff:ff:ff:ff actions=pop_vlan,output:p1,output:p3 table=8, priority=9008,in_port=p3,dl_vlan=100,dl_dst=ff:ff:ff:ff:ff:ff actions=pop_vlan,output:p1,output:p2 table=8, priority=9008,in_port=p4,dl_vlan=200,dl_dst=ff:ff:ff:ff:ff:ff actions=pop_vlan,output:p5 table=8, priority=9008,in_port=p5,dl_vlan=200,dl_dst=ff:ff:ff:ff:ff:ff actions=pop_vlan,output:p4 table=8, priority=9000,in_port=p1,dl_vlan=100 actions=pop_vlan,output:p2,output:p3 table=8, priority=9000,in_port=p2,dl_vlan=100 actions=pop_vlan,output:p1,output:p3 table=8, priority=9000,in_port=p3,dl_vlan=100 actions=pop_vlan,output:p1,output:p2 table=8, priority=9000,in_port=p4,dl_vlan=200 actions=pop_vlan,output:p5 table=8, priority=9000,in_port=p5,dl_vlan=200 actions=pop_vlan,output:p4 .. note:: These flows could apparently be simpler because OpenFlow says that ``output:`` is ignored if ```` is the input port. That means that the first three flows above could apparently be collapsed into just:: table=8, priority=9008,dl_vlan=100,dl_dst=ff:ff:ff:ff:ff:ff actions=pop_vlan,output:p1,output:p2,output:p3 There might be some reason why this won't work or isn't practical, but that isn't obvious from looking at the flow table. There are also some flows for handling some standard forms of multicast, and a fallback drop flow:: table=8, priority=9006,in_port=p1,dl_vlan=100,dl_dst=33:33:00:00:00:00/ff:ff:00:00:00:00 actions=pop_vlan,output:p2,output:p3 table=8, priority=9006,in_port=p2,dl_vlan=100,dl_dst=33:33:00:00:00:00/ff:ff:00:00:00:00 actions=pop_vlan,output:p1,output:p3 table=8, priority=9006,in_port=p3,dl_vlan=100,dl_dst=33:33:00:00:00:00/ff:ff:00:00:00:00 actions=pop_vlan,output:p1,output:p2 table=8, priority=9006,in_port=p4,dl_vlan=200,dl_dst=33:33:00:00:00:00/ff:ff:00:00:00:00 actions=pop_vlan,output:p5 table=8, priority=9006,in_port=p5,dl_vlan=200,dl_dst=33:33:00:00:00:00/ff:ff:00:00:00:00 actions=pop_vlan,output:p4 table=8, priority=9002,in_port=p1,dl_vlan=100,dl_dst=01:80:c2:00:00:00/ff:ff:ff:00:00:00 actions=pop_vlan,output:p2,output:p3 table=8, priority=9002,in_port=p2,dl_vlan=100,dl_dst=01:80:c2:00:00:00/ff:ff:ff:00:00:00 actions=pop_vlan,output:p1,output:p3 table=8, priority=9002,in_port=p3,dl_vlan=100,dl_dst=01:80:c2:00:00:00/ff:ff:ff:00:00:00 actions=pop_vlan,output:p1,output:p2 table=8, priority=9004,in_port=p1,dl_vlan=100,dl_dst=01:00:5e:00:00:00/ff:ff:ff:00:00:00 actions=pop_vlan,output:p2,output:p3 table=8, priority=9004,in_port=p2,dl_vlan=100,dl_dst=01:00:5e:00:00:00/ff:ff:ff:00:00:00 actions=pop_vlan,output:p1,output:p3 table=8, priority=9004,in_port=p3,dl_vlan=100,dl_dst=01:00:5e:00:00:00/ff:ff:ff:00:00:00 actions=pop_vlan,output:p1,output:p2 table=8, priority=9002,in_port=p4,dl_vlan=200,dl_dst=01:80:c2:00:00:00/ff:ff:ff:00:00:00 actions=pop_vlan,output:p5 table=8, priority=9002,in_port=p5,dl_vlan=200,dl_dst=01:80:c2:00:00:00/ff:ff:ff:00:00:00 actions=pop_vlan,output:p4 table=8, priority=9004,in_port=p4,dl_vlan=200,dl_dst=01:00:5e:00:00:00/ff:ff:ff:00:00:00 actions=pop_vlan,output:p5 table=8, priority=9004,in_port=p5,dl_vlan=200,dl_dst=01:00:5e:00:00:00/ff:ff:ff:00:00:00 actions=pop_vlan,output:p4 table=8, priority=0 actions=drop Tracing ~~~~~~~ Let's go a level deeper. So far, everything we've done has been fairly general. We can also look at something more specific: the path that a particular packet would take through Open vSwitch. We can use OVN ``ofproto/trace`` command to play "what-if?" games. This command is one that we send directly to ``ovs-vswitchd``, using the ``ovs-appctl`` utility. .. note:: ``ovs-appctl`` is actually a very simple-minded JSON-RPC client, so you could also use some other utility that speaks JSON-RPC, or access it from a program as an API. The ``ovs-vswitchd``\(8) manpage has a lot of detail on how to use ``ofproto/trace``, but let's just start by building up from a simple example. You can start with a command that just specifies the datapath (e.g. ``br0``), an input port, and nothing else; unspecified fields default to all-zeros. Let's look at the full output for this trivial example:: $ ovs-appctl ofproto/trace br0 in_port=p1 Flow: in_port=1,vlan_tci=0x0000,dl_src=00:00:00:00:00:00,dl_dst=00:00:00:00:00:00,dl_type=0x0000 bridge("br0") ------------- 0. in_port=1, priority 9099, cookie 0x5adc15c0 goto_table:1 1. in_port=1,vlan_tci=0x0000/0x1fff, priority 9000, cookie 0x5adc15c0 push_vlan:0x8100 set_field:4196->vlan_vid goto_table:3 3. priority 9000, cookie 0x5adc15c0 CONTROLLER:96 goto_table:7 7. priority 9000, cookie 0x5adc15c0 goto_table:8 8. in_port=1,dl_vlan=100, priority 9000, cookie 0x5adc15c0 pop_vlan output:2 output:3 Final flow: unchanged Megaflow: recirc_id=0,eth,in_port=1,vlan_tci=0x0000,dl_src=00:00:00:00:00:00,dl_dst=00:00:00:00:00:00,dl_type=0x0000 Datapath actions: push_vlan(vid=100,pcp=0),userspace(pid=0,controller(reason=1,flags=1,recirc_id=1,rule_cookie=0x5adc15c0,controller_id=0,max_len=96)),pop_vlan,2,3 The first line of output, beginning with ``Flow:``, just repeats our request in a more verbose form, including the L2 fields that were zeroed. Each of the numbered items under ``bridge("br0")`` shows what would happen to our hypothetical packet in the table with the given number. For example, we see in table 1 that the packet matches a flow that push on a VLAN header, set the VLAN ID to 100, and goes on to further processing in table 3. In table 3, the packet gets sent to the controller to allow MAC learning to take place, and then table 8 floods the packet to the other ports in the same VLAN. Summary information follows the numbered tables. The packet hasn't been changed (overall, even though a VLAN was pushed and then popped back off) since ingress, hence ``Final flow: unchanged``. We'll look at the ``Megaflow`` information later. The ``Datapath actions`` summarize what would actually happen to such a packet. Triggering MAC Learning ~~~~~~~~~~~~~~~~~~~~~~~ We just saw how a packet gets sent to the controller to trigger MAC learning. Let's actually send the packet and see what happens. But before we do that, let's save a copy of the current flow tables for later comparison:: $ save-flows br0 > flows1 Now use ``ofproto/trace``, as before, with a few new twists: we specify the source and destination Ethernet addresses and append the ``-generate`` option so that side effects like sending a packet to the controller actually happen:: $ ovs-appctl ofproto/trace br0 in_port=p1,dl_src=00:11:11:00:00:00,dl_dst=00:22:22:00:00:00 -generate The output is almost identical to that before, so it is not repeated here. But, take a look at ``inst/faucet.log`` now. It should now include a line at the end that says that it learned about our MAC 00:11:11:00:00:00, like this:: Jan 06 15:56:02 faucet.valve INFO DPID 1 (0x1) L2 learned 00:11:11:00:00:00 (L2 type 0x0000, L3 src None) on Port 1 on VLAN 100 (1 hosts total Now compare the flow tables that we saved to the current ones:: diff-flows flows1 br0 The result should look like this, showing new flows for the learned MACs:: +table=3 priority=9098,in_port=1,dl_vlan=100,dl_src=00:11:11:00:00:00 hard_timeout=3601 actions=goto_table:7 +table=7 priority=9099,dl_vlan=100,dl_dst=00:11:11:00:00:00 idle_timeout=3601 actions=pop_vlan,output:1 To demonstrate the usefulness of the learned MAC, try tracing (with side effects) a packet arriving on ``p2`` (or ``p3``) and destined to the address learned on ``p1``, like this:: $ ovs-appctl ofproto/trace br0 in_port=p2,dl_src=00:22:22:00:00:00,dl_dst=00:11:11:00:00:00 -generate The first time you run this command, you will notice that it sends the packet to the controller, to learn ``p2``'s 00:22:22:00:00:00 source address:: bridge("br0") ------------- 0. in_port=2, priority 9099, cookie 0x5adc15c0 goto_table:1 1. in_port=2,vlan_tci=0x0000/0x1fff, priority 9000, cookie 0x5adc15c0 push_vlan:0x8100 set_field:4196->vlan_vid goto_table:3 3. priority 9000, cookie 0x5adc15c0 CONTROLLER:96 goto_table:7 7. dl_vlan=100,dl_dst=00:11:11:00:00:00, priority 9099, cookie 0x5adc15c0 pop_vlan output:1 If you check ``inst/faucet.log``, you can see that ``p2``'s MAC has been learned too:: Jan 06 15:58:09 faucet.valve INFO DPID 1 (0x1) L2 learned 00:22:22:00:00:00 (L2 type 0x0000, L3 src None) on Port 2 on VLAN 100 (2 hosts total) Similarly for ``diff-flows``:: $ diff-flows flows1 br0 +table=3 priority=9098,in_port=1,dl_vlan=100,dl_src=00:11:11:00:00:00 hard_timeout=3601 actions=goto_table:7 +table=3 priority=9098,in_port=2,dl_vlan=100,dl_src=00:22:22:00:00:00 hard_timeout=3604 actions=goto_table:7 +table=7 priority=9099,dl_vlan=100,dl_dst=00:11:11:00:00:00 idle_timeout=3601 actions=pop_vlan,output:1 +table=7 priority=9099,dl_vlan=100,dl_dst=00:22:22:00:00:00 idle_timeout=3604 actions=pop_vlan,output:2 Then, if you re-run either of the ``ofproto/trace`` commands (with or without ``-generate``), you can see that the packets go back and forth without any further MAC learning, e.g.:: $ ovs-appctl ofproto/trace br0 in_port=p2,dl_src=00:22:22:00:00:00,dl_dst=00:11:11:00:00:00 -generate Flow: in_port=2,vlan_tci=0x0000,dl_src=00:22:22:00:00:00,dl_dst=00:11:11:00:00:00,dl_type=0x0000 bridge("br0") ------------- 0. in_port=2, priority 9099, cookie 0x5adc15c0 goto_table:1 1. in_port=2,vlan_tci=0x0000/0x1fff, priority 9000, cookie 0x5adc15c0 push_vlan:0x8100 set_field:4196->vlan_vid goto_table:3 3. in_port=2,dl_vlan=100,dl_src=00:22:22:00:00:00, priority 9098, cookie 0x5adc15c0 goto_table:7 7. dl_vlan=100,dl_dst=00:11:11:00:00:00, priority 9099, cookie 0x5adc15c0 pop_vlan output:1 Final flow: unchanged Megaflow: recirc_id=0,eth,in_port=2,vlan_tci=0x0000/0x1fff,dl_src=00:22:22:00:00:00,dl_dst=00:11:11:00:00:00,dl_type=0x0000 Datapath actions: 1 Performance ~~~~~~~~~~~ Open vSwitch has a concept of a "fast path" and a "slow path"; ideally all packets stay in the fast path. This distinction between slow path and fast path is the key to making sure that Open vSwitch performs as fast as possible. Some factors can force a flow or a packet to take the slow path. As one example, all CFM, BFD, LACP, STP, and LLDP processing takes place in the slow path, in the cases where Open vSwitch processes these protocols itself instead of delegating to controller-written flows. As a second example, any flow that modifies ARP fields is processed in the slow path. These are corner cases that are unlikely to cause performance problems in practice because these protocols send packets at a relatively slow rate, and users and controller authors do not normally need to be concerned about them. To understand what cases users and controller authors should consider, we need to talk about how Open vSwitch optimizes for performance. The Open vSwitch code is divided into two major components which, as already mentioned, are called the "slow path" and "fast path" (aka "datapath"). The slow path is embedded in the ``ovs-vswitchd`` userspace program. It is the part of the Open vSwitch packet processing logic that understands OpenFlow. Its job is to take a packet and run it through the OpenFlow tables to determine what should happen to it. It outputs a list of actions in a form similar to OpenFlow actions but simpler, called "ODP actions" or "datapath actions". It then passes the ODP actions to the datapath, which applies them to the packet. .. note:: Open vSwitch contains a single slow path and multiple fast paths. The difference between using Open vSwitch with the Linux kernel versus with DPDK is the datapath. If every packet passed through the slow path and the fast path in this way, performance would be terrible. The key to getting high performance from this architecture is caching. Open vSwitch includes a multi-level cache. It works like this: 1. A packet initially arrives at the datapath. Some datapaths (such as DPDK and the in-tree version of the OVS kernel module) have a first-level cache called the "microflow cache". The microflow cache is the key to performance for relatively long-lived, high packet rate flows. If the datapath has a microflow cache, then it consults it and, if there is a cache hit, the datapath executes the associated actions. Otherwise, it proceeds to step 2. 2. The datapath consults its second-level cache, called the "megaflow cache". The megaflow cache is the key to performance for shorter or low packet rate flows. If there is a megaflow cache hit, the datapath executes the associated actions. Otherwise, it proceeds to step 3. 3. The datapath passes the packet to the slow path, which runs it through the OpenFlow table to yield ODP actions, a process that is often called "flow translation". It then passes the packet back to the datapath to execute the actions and to, if possible, install a megaflow cache entry so that subsequent similar packets can be handled directly by the fast path. (We already described above most of the cases where a cache entry cannot be installed.) The megaflow cache is the key cache to consider for performance tuning. Open vSwitch provides tools for understanding and optimizing its behavior. The ``ofproto/trace`` command that we have already been using is the most common tool for this use. Let's take another look at the most recent ``ofproto/trace`` output:: $ ovs-appctl ofproto/trace br0 in_port=p2,dl_src=00:22:22:00:00:00,dl_dst=00:11:11:00:00:00 -generate Flow: in_port=2,vlan_tci=0x0000,dl_src=00:22:22:00:00:00,dl_dst=00:11:11:00:00:00,dl_type=0x0000 bridge("br0") ------------- 0. in_port=2, priority 9099, cookie 0x5adc15c0 goto_table:1 1. in_port=2,vlan_tci=0x0000/0x1fff, priority 9000, cookie 0x5adc15c0 push_vlan:0x8100 set_field:4196->vlan_vid goto_table:3 3. in_port=2,dl_vlan=100,dl_src=00:22:22:00:00:00, priority 9098, cookie 0x5adc15c0 goto_table:7 7. dl_vlan=100,dl_dst=00:11:11:00:00:00, priority 9099, cookie 0x5adc15c0 pop_vlan output:1 Final flow: unchanged Megaflow: recirc_id=0,eth,in_port=2,vlan_tci=0x0000/0x1fff,dl_src=00:22:22:00:00:00,dl_dst=00:11:11:00:00:00,dl_type=0x0000 Datapath actions: 1 This time, it's the last line that we're interested in. This line shows the entry that Open vSwitch would insert into the megaflow cache given the particular packet with the current flow tables. The megaflow entry includes: * ``recirc_id``. This is an implementation detail that users don't normally need to understand. * ``eth``. This just indicates that the cache entry matches only Ethernet packets; Open vSwitch also supports other types of packets, such as IP packets not encapsulated in Ethernet. * All of the fields matched by any of the flows that the packet visited: ``in_port`` In tables 0, 1, and 3. ``vlan_tci`` In tables 1, 3, and 7 (``vlan_tci`` includes the VLAN ID and PCP fields and``dl_vlan`` is just the VLAN ID). ``dl_src`` In table 3 ``dl_dst`` In table 7. * All of the fields matched by flows that had to be ruled out to ensure that the ones that actually matched were the highest priority matching rules. The last one is important. Notice how the megaflow matches on ``dl_type=0x0000``, even though none of the tables matched on ``dl_type`` (the Ethernet type). One reason is because of this flow in OpenFlow table 1 (which shows up in ``dump-flows`` output):: table=1, priority=9099,dl_type=0x88cc actions=drop This flow has higher priority than the flow in table 1 that actually matched. This means that, to put it in the megaflow cache, ``ovs-vswitchd`` has to add a match on ``dl_type`` to ensure that the cache entry doesn't match LLDP packets (with Ethertype 0x88cc). .. note:: In fact, in some cases ``ovs-vswitchd`` matches on fields that aren't strictly required according to this description. ``dl_type`` is actually one of those, so deleting the LLDP flow probably would not have any effect on the megaflow. But the principle here is sound. So why does any of this matter? It's because, the more specific a megaflow is, that is, the more fields or bits within fields that a megaflow matches, the less valuable it is from a caching viewpoint. A very specific megaflow might match on L2 and L3 addresses and L4 port numbers. When that happens, only packets in one (half-)connection match the megaflow. If that connection has only a few packets, as many connections do, then the high cost of the slow path translation is amortized over only a few packets, so the average cost of forwarding those packets is high. On the other hand, if a megaflow only matches a relatively small number of L2 and L3 packets, then the cache entry can potentially be used by many individual connections, and the average cost is low. For more information on how Open vSwitch constructs megaflows, including about ways that it can make megaflow entries less specific than one would infer from the discussion here, please refer to the 2015 NSDI paper, "The Design and Implementation of Open vSwitch", which focuses on this algorithm. Routing ------- We've looked at how Faucet implements switching in OpenFlow, and how Open vSwitch implements OpenFlow through its datapath architecture. Now let's start over, adding L3 routing into the picture. It's remarkably easy to enable routing. We just change our ``vlans`` section in ``inst/faucet.yaml`` to specify a router IP address for each VLAN and define a router between them. The ``dps`` section is unchanged:: dps: switch-1: dp_id: 0x1 timeout: 3600 arp_neighbor_timeout: 3600 interfaces: 1: native_vlan: 100 2: native_vlan: 100 3: native_vlan: 100 4: native_vlan: 200 5: native_vlan: 200 vlans: 100: faucet_vips: ["10.100.0.254/24"] 200: faucet_vips: ["10.200.0.254/24"] routers: router-1: vlans: [100, 200] Then we restart Faucet:: $ docker restart faucet .. note:: One should be able to tell Faucet to re-read its configuration file without restarting it. I sometimes saw anomalous behavior when I did this, although I didn't characterize it well enough to make a quality bug report. I found restarting the container to be reliable. OpenFlow Layer ~~~~~~~~~~~~~~ Back in the OVS sandbox, let's see how the flow table has changed, with:: $ diff-flows flows1 br0 First, table 3 has new flows to direct ARP packets to table 6 (the virtual IP processing table), presumably to handle ARP for the router IPs. New flows also send IP packets destined to a particular Ethernet address to table 4 (the L3 forwarding table); we can make the educated guess that the Ethernet address is the one used by the Faucet router:: +table=3 priority=9131,arp,dl_vlan=100 actions=goto_table:6 +table=3 priority=9131,arp,dl_vlan=200 actions=goto_table:6 +table=3 priority=9099,ip,dl_vlan=100,dl_dst=0e:00:00:00:00:01 actions=goto_table:4 +table=3 priority=9099,ip,dl_vlan=200,dl_dst=0e:00:00:00:00:01 actions=goto_table:4 The new flows in table 4 appear to be verifying that the packets are indeed addressed to a network or IP address that Faucet knows how to route:: +table=4 priority=9131,ip,dl_vlan=100,nw_dst=10.100.0.254 actions=goto_table:6 +table=4 priority=9131,ip,dl_vlan=200,nw_dst=10.200.0.254 actions=goto_table:6 +table=4 priority=9123,ip,dl_vlan=100,nw_dst=10.100.0.0/24 actions=goto_table:6 +table=4 priority=9123,ip,dl_vlan=200,nw_dst=10.100.0.0/24 actions=goto_table:6 +table=4 priority=9123,ip,dl_vlan=100,nw_dst=10.200.0.0/24 actions=goto_table:6 +table=4 priority=9123,ip,dl_vlan=200,nw_dst=10.200.0.0/24 actions=goto_table:6 Table 6 has a few different things going on. It sends ARP requests for the router IPs to the controller; presumably the controller will generate replies and send them back to the requester. It switches other ARP packets, either broadcasting them if they have a broadcast destination or attempting to unicast them otherwise. It sends all other IP packets to the controller:: +table=6 priority=9133,arp,arp_tpa=10.100.0.254 actions=CONTROLLER:128 +table=6 priority=9133,arp,arp_tpa=10.200.0.254 actions=CONTROLLER:128 +table=6 priority=9132,arp,dl_dst=ff:ff:ff:ff:ff:ff actions=goto_table:8 +table=6 priority=9131,arp actions=goto_table:7 +table=6 priority=9130,ip actions=CONTROLLER:128 Performance is clearly going to be poor if every packet that needs to be routed has to go to the controller, but it's unlikely that's the full story. In the next section, we'll take a closer look. Tracing ~~~~~~~ As in our switching example, we can play some "what-if?" games to figure out how this works. Let's suppose that a machine with IP 10.100.0.1, on port ``p1``, wants to send a IP packet to a machine with IP 10.200.0.1 on port ``p4``. Assuming that these hosts have not been in communication recently, the steps to accomplish this are normally the following: 1. Host 10.100.0.1 sends an ARP request to router 10.100.0.254. 2. The router sends an ARP reply to the host. 3. Host 10.100.0.1 sends an IP packet to 10.200.0.1, via the router's Ethernet address. 4. The router broadcasts an ARP request to ``p4`` and ``p5``, the ports that carry the 10.200.0. network. 5. Host 10.200.0.1 sends an ARP reply to the router. 6. Either the router sends the IP packet (which it buffered) to 10.200.0.1, or eventually 10.100.0.1 times out and resends it. Let's use ``ofproto/trace`` to see whether Faucet and OVS follow this procedure. Before we start, save a new snapshot of the flow tables for later comparison:: $ save-flows br0 > flows2 Step 1: Host ARP for Router +++++++++++++++++++++++++++ Let's simulate the ARP from 10.100.0.1 to its gateway router 10.100.0.254. This requires more detail than any of the packets we've simulated previously:: $ ovs-appctl ofproto/trace br0 in_port=p1,dl_src=00:01:02:03:04:05,dl_dst=ff:ff:ff:ff:ff:ff,dl_type=0x806,arp_spa=10.100.0.1,arp_tpa=10.100.0.254,arp_sha=00:01:02:03:04:05,arp_tha=ff:ff:ff:ff:ff:ff,arp_op=1 -generate The important part of the output is where it shows that the packet was recognized as an ARP request destined to the router gateway and therefore sent to the controller:: 6. arp,arp_tpa=10.100.0.254, priority 9133, cookie 0x5adc15c0 CONTROLLER:128 The Faucet log shows that Faucet learned the host's MAC address, its MAC-to-IP mapping, and responded to the ARP request:: Jan 06 16:12:23 faucet.valve INFO DPID 1 (0x1) Adding new route 10.100.0.1/32 via 10.100.0.1 (00:01:02:03:04:05) on VLAN 100 Jan 06 16:12:23 faucet.valve INFO DPID 1 (0x1) Responded to ARP request for 10.100.0.254 from 10.100.0.1 (00:01:02:03:04:05) on VLAN 100 Jan 06 16:12:23 faucet.valve INFO DPID 1 (0x1) L2 learned 00:01:02:03:04:05 (L2 type 0x0806, L3 src 10.100.0.1) on Port 1 on VLAN 100 (1 hosts total) We can also look at the changes to the flow tables:: $ diff-flows flows2 br0 +table=3 priority=9098,in_port=1,dl_vlan=100,dl_src=00:01:02:03:04:05 hard_timeout=3600 actions=goto_table:7 +table=4 priority=9131,ip,dl_vlan=100,nw_dst=10.100.0.1 actions=set_field:4196->vlan_vid,set_field:0e:00:00:00:00:01->eth_src,set_field:00:01:02:03:04:05->eth_dst,dec_ttl,goto_table:7 +table=4 priority=9131,ip,dl_vlan=200,nw_dst=10.100.0.1 actions=set_field:4196->vlan_vid,set_field:0e:00:00:00:00:01->eth_src,set_field:00:01:02:03:04:05->eth_dst,dec_ttl,goto_table:7 +table=7 priority=9099,dl_vlan=100,dl_dst=00:01:02:03:04:05 idle_timeout=3600 actions=pop_vlan,output:1 The new flows include one in table 3 and one in table 7 for the learned MAC, which have the same forms we saw before. The new flows in table 4 are different. They matches packets directed to 10.100.0.1 (in two VLANs) and forward them to the host by updating the Ethernet source and destination addresses appropriately, decrementing the TTL, and skipping ahead to unicast output in table 7. This means that packets sent **to** 10.100.0.1 should now get to their destination. Step 2: Router Sends ARP Reply ++++++++++++++++++++++++++++++ ``inst/faucet.log`` said that the router sent an ARP reply. How can we see it? Simulated packets just get dropped by default. One way is to configure the dummy ports to write the packets they receive to a file. Let's try that. First configure the port:: $ ovs-vsctl set interface p1 options:pcap=p1.pcap Then re-run the "trace" command:: $ ovs-appctl ofproto/trace br0 in_port=p1,dl_src=00:01:02:03:04:05,dl_dst=ff:ff:ff:ff:ff:ff,dl_type=0x806,arp_spa=10.100.0.1,arp_tpa=10.100.0.254,arp_sha=00:01:02:03:04:05,arp_tha=ff:ff:ff:ff:ff:ff,arp_op=1 -generate And dump the reply packet:: $ /usr/sbin/tcpdump -evvvr sandbox/p1.pcap reading from file sandbox/p1.pcap, link-type EN10MB (Ethernet) 16:14:47.670727 0e:00:00:00:00:01 (oui Unknown) > 00:01:02:03:04:05 (oui Unknown), ethertype ARP (0x0806), length 60: Ethernet (len 6), IPv4 (len 4), Reply 10.100.0.254 is-at 0e:00:00:00:00:01 (oui Unknown), length 46 We clearly see the ARP reply, which tells us that the Faucet router's Ethernet address is 0e:00:00:00:00:01 (as we guessed before from the flow table. Let's configure the rest of our ports to log their packets, too:: $ for i in 2 3 4 5; do ovs-vsctl set interface p$i options:pcap=p$i.pcap; done Step 3: Host Sends IP Packet ++++++++++++++++++++++++++++ Now that host 10.100.0.1 has the MAC address for its router, it can send an IP packet to 10.200.0.1 via the router's MAC address, like this:: $ ovs-appctl ofproto/trace br0 in_port=p1,dl_src=00:01:02:03:04:05,dl_dst=0e:00:00:00:00:01,udp,nw_src=10.100.0.1,nw_dst=10.200.0.1,nw_ttl=64 -generate Flow: udp,in_port=1,vlan_tci=0x0000,dl_src=00:01:02:03:04:05,dl_dst=0e:00:00:00:00:01,nw_src=10.100.0.1,nw_dst=10.200.0.1,nw_tos=0,nw_ecn=0,nw_ttl=64,tp_src=0,tp_dst=0 bridge("br0") ------------- 0. in_port=1, priority 9099, cookie 0x5adc15c0 goto_table:1 1. in_port=1,vlan_tci=0x0000/0x1fff, priority 9000, cookie 0x5adc15c0 push_vlan:0x8100 set_field:4196->vlan_vid goto_table:3 3. ip,dl_vlan=100,dl_dst=0e:00:00:00:00:01, priority 9099, cookie 0x5adc15c0 goto_table:4 4. ip,dl_vlan=100,nw_dst=10.200.0.0/24, priority 9123, cookie 0x5adc15c0 goto_table:6 6. ip, priority 9130, cookie 0x5adc15c0 CONTROLLER:128 Final flow: udp,in_port=1,dl_vlan=100,dl_vlan_pcp=0,vlan_tci1=0x0000,dl_src=00:01:02:03:04:05,dl_dst=0e:00:00:00:00:01,nw_src=10.100.0.1,nw_dst=10.200.0.1,nw_tos=0,nw_ecn=0,nw_ttl=64,tp_src=0,tp_dst=0 Megaflow: recirc_id=0,eth,ip,in_port=1,vlan_tci=0x0000/0x1fff,dl_src=00:01:02:03:04:05,dl_dst=0e:00:00:00:00:01,nw_dst=10.200.0.0/25,nw_frag=no Datapath actions: push_vlan(vid=100,pcp=0),userspace(pid=0,controller(reason=1,flags=0,recirc_id=6,rule_cookie=0x5adc15c0,controller_id=0,max_len=128)) Observe that the packet gets recognized as destined to the router, in table 3, and then as properly destined to the 10.200.0.0/24 network, in table 4. In table 6, however, it gets sent to the controller. Presumably, this is because Faucet has not yet resolved an Ethernet address for the destination host 10.200.0.1. It probably sent out an ARP request. Let's take a look in the next step. Step 4: Router Broadcasts ARP Request +++++++++++++++++++++++++++++++++++++ The router needs to know the Ethernet address of 10.200.0.1. It knows that, if this machine exists, it's on port ``p4`` or ``p5``, since we configured those ports as VLAN 200. Let's make sure:: $ /usr/sbin/tcpdump -evvvr sandbox/p4.pcap reading from file sandbox/p4.pcap, link-type EN10MB (Ethernet) 16:17:43.174006 0e:00:00:00:00:01 (oui Unknown) > Broadcast, ethertype ARP (0x0806), length 60: Ethernet (len 6), IPv4 (len 4), Request who-has 10.200.0.1 tell 10.200.0.254, length 46 and:: $ /usr/sbin/tcpdump -evvvr sandbox/p5.pcap reading from file sandbox/p5.pcap, link-type EN10MB (Ethernet) 16:17:43.174268 0e:00:00:00:00:01 (oui Unknown) > Broadcast, ethertype ARP (0x0806), length 60: Ethernet (len 6), IPv4 (len 4), Request who-has 10.200.0.1 tell 10.200.0.254, length 46 For good measure, let's make sure that it wasn't sent to ``p3``:: $ /usr/sbin/tcpdump -evvvr sandbox/p3.pcap reading from file sandbox/p3.pcap, link-type EN10MB (Ethernet) Step 5: Host 2 Sends ARP Reply ++++++++++++++++++++++++++++++ The Faucet controller sent an ARP request, so we can send an ARP reply:: $ ovs-appctl ofproto/trace br0 in_port=p4,dl_src=00:10:20:30:40:50,dl_dst=0e:00:00:00:00:01,dl_type=0x806,arp_spa=10.200.0.1,arp_tpa=10.200.0.254,arp_sha=00:10:20:30:40:50,arp_tha=0e:00:00:00:00:01,arp_op=2 -generate Flow: arp,in_port=4,vlan_tci=0x0000,dl_src=00:10:20:30:40:50,dl_dst=0e:00:00:00:00:01,arp_spa=10.200.0.1,arp_tpa=10.200.0.254,arp_op=2,arp_sha=00:10:20:30:40:50,arp_tha=0e:00:00:00:00:01 bridge("br0") ------------- 0. in_port=4, priority 9099, cookie 0x5adc15c0 goto_table:1 1. in_port=4,vlan_tci=0x0000/0x1fff, priority 9000, cookie 0x5adc15c0 push_vlan:0x8100 set_field:4296->vlan_vid goto_table:3 3. arp,dl_vlan=200, priority 9131, cookie 0x5adc15c0 goto_table:6 6. arp,arp_tpa=10.200.0.254, priority 9133, cookie 0x5adc15c0 CONTROLLER:128 Final flow: arp,in_port=4,dl_vlan=200,dl_vlan_pcp=0,vlan_tci1=0x0000,dl_src=00:10:20:30:40:50,dl_dst=0e:00:00:00:00:01,arp_spa=10.200.0.1,arp_tpa=10.200.0.254,arp_op=2,arp_sha=00:10:20:30:40:50,arp_tha=0e:00:00:00:00:01 Megaflow: recirc_id=0,eth,arp,in_port=4,vlan_tci=0x0000/0x1fff,dl_dst=0e:00:00:00:00:01,arp_tpa=10.200.0.254 Datapath actions: push_vlan(vid=200,pcp=0),userspace(pid=0,controller(reason=1,flags=0,recirc_id=7,rule_cookie=0x5adc15c0,controller_id=0,max_len=128)) It shows up in ``inst/faucet.log``:: Jan 06 03:20:11 faucet.valve INFO DPID 1 (0x1) Adding new route 10.200.0.1/32 via 10.200.0.1 (00:10:20:30:40:50) on VLAN 200 Jan 06 03:20:11 faucet.valve INFO DPID 1 (0x1) ARP response 10.200.0.1 (00:10:20:30:40:50) on VLAN 200 Jan 06 03:20:11 faucet.valve INFO DPID 1 (0x1) L2 learned 00:10:20:30:40:50 (L2 type 0x0806, L3 src 10.200.0.1) on Port 4 on VLAN 200 (1 hosts total) and in the OVS flow tables:: $ diff-flows flows2 br0 +table=3 priority=9098,in_port=4,dl_vlan=200,dl_src=00:10:20:30:40:50 hard_timeout=3601 actions=goto_table:7 ... +table=4 priority=9131,ip,dl_vlan=200,nw_dst=10.200.0.1 actions=set_field:4296->vlan_vid,set_field:0e:00:00:00:00:01->eth_src,set_field:00:10:20:30:40:50->eth_dst,dec_ttl,goto_table:7 +table=4 priority=9131,ip,dl_vlan=100,nw_dst=10.200.0.1 actions=set_field:4296->vlan_vid,set_field:0e:00:00:00:00:01->eth_src,set_field:00:10:20:30:40:50->eth_dst,dec_ttl,goto_table:7 ... +table=4 priority=9123,ip,dl_vlan=100,nw_dst=10.200.0.0/24 actions=goto_table:6 +table=7 priority=9099,dl_vlan=200,dl_dst=00:10:20:30:40:50 idle_timeout=3601 actions=pop_vlan,output:4 Step 6: IP Packet Delivery ++++++++++++++++++++++++++ Now both the host and the router have everything they need to deliver the packet. There are two ways it might happen. If Faucet's router is smart enough to buffer the packet that trigger ARP resolution, then it might have delivered it already. If so, then it should show up in ``p4.pcap``. Let's take a look:: $ /usr/sbin/tcpdump -evvvr sandbox/p4.pcap ip reading from file sandbox/p4.pcap, link-type EN10MB (Ethernet) Nope. That leaves the other possibility, which is that Faucet waits for the original sending host to re-send the packet. We can do that by re-running the trace:: $ ovs-appctl ofproto/trace br0 in_port=p1,dl_src=00:01:02:03:04:05,dl_dst=0e:00:00:00:00:01,udp,nw_src=10.100.0.1,nw_dst=10.200.0.1,nw_ttl=64 -generate Flow: udp,in_port=1,vlan_tci=0x0000,dl_src=00:01:02:03:04:05,dl_dst=0e:00:00:00:00:01,nw_src=10.100.0.1,nw_dst=10.200.0.1,nw_tos=0,nw_ecn=0,nw_ttl=64,tp_src=0,tp_dst=0 bridge("br0") ------------- 0. in_port=1, priority 9099, cookie 0x5adc15c0 goto_table:1 1. in_port=1,vlan_tci=0x0000/0x1fff, priority 9000, cookie 0x5adc15c0 push_vlan:0x8100 set_field:4196->vlan_vid goto_table:3 3. ip,dl_vlan=100,dl_dst=0e:00:00:00:00:01, priority 9099, cookie 0x5adc15c0 goto_table:4 4. ip,dl_vlan=100,nw_dst=10.200.0.1, priority 9131, cookie 0x5adc15c0 set_field:4296->vlan_vid set_field:0e:00:00:00:00:01->eth_src set_field:00:10:20:30:40:50->eth_dst dec_ttl goto_table:7 7. dl_vlan=200,dl_dst=00:10:20:30:40:50, priority 9099, cookie 0x5adc15c0 pop_vlan output:4 Final flow: udp,in_port=1,vlan_tci=0x0000,dl_src=0e:00:00:00:00:01,dl_dst=00:10:20:30:40:50,nw_src=10.100.0.1,nw_dst=10.200.0.1,nw_tos=0,nw_ecn=0,nw_ttl=63,tp_src=0,tp_dst=0 Megaflow: recirc_id=0,eth,ip,in_port=1,vlan_tci=0x0000/0x1fff,dl_src=00:01:02:03:04:05,dl_dst=0e:00:00:00:00:01,nw_dst=10.200.0.1,nw_ttl=64,nw_frag=no Datapath actions: set(eth(src=0e:00:00:00:00:01,dst=00:10:20:30:40:50)),set(ipv4(dst=10.200.0.1,ttl=63)),4 Finally, we have working IP packet forwarding! Performance ~~~~~~~~~~~ Take another look at the megaflow line above:: Megaflow: recirc_id=0,eth,ip,in_port=1,vlan_tci=0x0000/0x1fff,dl_src=00:01:02:03:04:05,dl_dst=0e:00:00:00:00:01,nw_dst=10.200.0.1,nw_ttl=64,nw_frag=no This means that (almost) any packet between these Ethernet source and destination hosts, destined to the given IP host, will be handled by this single megaflow cache entry. So regardless of the number of UDP packets or TCP connections that these hosts exchange, Open vSwitch packet processing won't need to fall back to the slow path. It is quite efficient. .. note:: The exceptions are packets with a TTL other than 64, and fragmented packets. Most hosts use a constant TTL for outgoing packets, and fragments are rare. If either of those did change, then that would simply result in a new megaflow cache entry. The datapath actions might also be worth a look:: Datapath actions: set(eth(src=0e:00:00:00:00:01,dst=00:10:20:30:40:50)),set(ipv4(dst=10.200.0.1,ttl=63)),4 This just means that, to process these packets, the datapath changes the Ethernet source and destination addresses and the IP TTL, and then transmits the packet to port ``p4`` (also numbered 4). Notice in particular that, despite the OpenFlow actions that pushed, modified, and popped back off a VLAN, there is nothing in the datapath actions about VLANs. This is because the OVS flow translation code "optimizes out" redundant or unneeded actions, which saves time when the cache entry is executed later. .. note:: It's not clear why the actions also re-set the IP destination address to its original value. Perhaps this is a minor performance bug. ACLs ---- Let's try out some ACLs, since they do a good job illustrating some of the ways that OVS tries to optimize megaflows. Update ``inst/faucet.yaml`` to the following:: dps: switch-1: dp_id: 0x1 timeout: 3600 arp_neighbor_timeout: 3600 interfaces: 1: native_vlan: 100 acl_in: 1 2: native_vlan: 100 3: native_vlan: 100 4: native_vlan: 200 5: native_vlan: 200 vlans: 100: faucet_vips: ["10.100.0.254/24"] 200: faucet_vips: ["10.200.0.254/24"] routers: router-1: vlans: [100, 200] acls: 1: - rule: dl_type: 0x800 nw_proto: 6 tcp_dst: 8080 actions: allow: 0 - rule: actions: allow: 1 Then restart Faucet:: $ docker restart faucet On port 1, this new configuration blocks all traffic to TCP port 8080 and allows all other traffic. The resulting change in the flow table shows this clearly too:: $ diff-flows flows2 br0 -priority=9099,in_port=1 actions=goto_table:1 +priority=9098,in_port=1 actions=goto_table:1 +priority=9099,tcp,in_port=1,tp_dst=8080 actions=drop The most interesting question here is performance. If you recall the earlier discussion, when a packet through the flow table encounters a match on a given field, the resulting megaflow has to match on that field, even if the flow didn't actually match. This is expensive. In particular, here you can see that any TCP packet is going to encounter the ACL flow, even if it is directed to a port other than 8080. If that means that every megaflow for a TCP packet is going to have to match on the TCP destination, that's going to be bad for caching performance because there will be a need for a separate megaflow for every TCP destination port that actually appears in traffic, which means a lot more megaflows than otherwise. (Really, in practice, if such a simple ACL blew up performance, OVS wouldn't be a very good switch!) Let's see what happens, by sending a packet to port 80 (instead of 8080):: $ ovs-appctl ofproto/trace br0 in_port=p1,dl_src=00:01:02:03:04:05,dl_dst=0e:00:00:00:00:01,tcp,nw_src=10.100.0.1,nw_dst=10.200.0.1,nw_ttl=64,tp_dst=80 -generate Flow: tcp,in_port=1,vlan_tci=0x0000,dl_src=00:01:02:03:04:05,dl_dst=0e:00:00:00:00:01,nw_src=10.100.0.1,nw_dst=10.200.0.1,nw_tos=0,nw_ecn=0,nw_ttl=64,tp_src=0,tp_dst=80,tcp_flags=0 bridge("br0") ------------- 0. in_port=1, priority 9098, cookie 0x5adc15c0 goto_table:1 1. in_port=1,vlan_tci=0x0000/0x1fff, priority 9000, cookie 0x5adc15c0 push_vlan:0x8100 set_field:4196->vlan_vid goto_table:3 3. ip,dl_vlan=100,dl_dst=0e:00:00:00:00:01, priority 9099, cookie 0x5adc15c0 goto_table:4 4. ip,dl_vlan=100,nw_dst=10.200.0.0/24, priority 9123, cookie 0x5adc15c0 goto_table:6 6. ip, priority 9130, cookie 0x5adc15c0 CONTROLLER:128 Final flow: tcp,in_port=1,dl_vlan=100,dl_vlan_pcp=0,vlan_tci1=0x0000,dl_src=00:01:02:03:04:05,dl_dst=0e:00:00:00:00:01,nw_src=10.100.0.1,nw_dst=10.200.0.1,nw_tos=0,nw_ecn=0,nw_ttl=64,tp_src=0,tp_dst=80,tcp_flags=0 Megaflow: recirc_id=0,eth,tcp,in_port=1,vlan_tci=0x0000/0x1fff,dl_src=00:01:02:03:04:05,dl_dst=0e:00:00:00:00:01,nw_dst=10.200.0.1,nw_frag=no,tp_dst=0x0/0xf000 Datapath actions: push_vlan(vid=100,pcp=0) Take a look at the Megaflow line and in particular the match on ``tp_dst``, which says ``tp_dst=0x0/0xf000``. What this means is that the megaflow matches on only the top 4 bits of the TCP destination port. That works because:: 80 (base 10) == 0001,1111,1001,0000 (base 2) 8080 (base 10) == 0000,0000,0101,0000 (base 2) and so by matching on only the top 4 bits, rather than all 16, the OVS fast path can distinguish port 80 from port 8080. This allows this megaflow to match one-sixteenth of the TCP destination port address space, rather than just 1/65536th of it. .. note:: The algorithm OVS uses for this purpose isn't perfect. In this case, a single-bit match would work (e.g. tp_dst=0x0/0x1000), and would be superior since it would only match half the port address space instead of one-sixteenth. For details of this algorithm, please refer to ``lib/classifier.c`` in the Open vSwitch source tree, or our 2015 NSDI paper "The Design and Implementation of Open vSwitch". Finishing Up ------------ When you're done, you probably want to exit the sandbox session, with Control+D or ``exit``, and stop the Faucet controller with ``docker stop faucet; docker rm faucet``. Further Directions ------------------ We've looked a fair bit at how Faucet interacts with Open vSwitch. If you still have some interest, you might want to explore some of these directions: * Adding more than one switch. Faucet can control multiple switches but we've only been simulating one of them. It's easy enough to make a single OVS instance act as multiple switches (just ``ovs-vsctl add-br`` another bridge), or you could use genuinely separate OVS instances. * Additional features. Faucet has more features than we've demonstrated, such as IPv6 routing and port mirroring. These should also interact gracefully with Open vSwitch. * Real performance testing. We've looked at how flows and traces **should** demonstrate good performance, but of course there's no proof until it actually works in practice. We've also only tested with trivial configurations. Open vSwitch can scale to millions of OpenFlow flows, but the scaling in practice depends on the particular flow tables and traffic patterns, so it's valuable to test with large configurations, either in the way we've done it or with real traffic. ovs-2.9.0/Documentation/tutorials/index.rst000066400000000000000000000026721324262074100210410ustar00rootroot00000000000000.. Copyright (c) 2016, Stephen Finucane Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. Convention for heading levels in Open vSwitch documentation: ======= Heading 0 (reserved for the title in a document) ------- Heading 1 ~~~~~~~ Heading 2 +++++++ Heading 3 ''''''' Heading 4 Avoid deeper levels because they do not render well. ========= Tutorials ========= Getting started with Open vSwitch (OVS) and Open Virtual Network (OVN) for Open vSwitch. .. TODO(stephenfin): We could really do with a few basic tutorials, along with some more specialized ones (DPDK, XenServer, Windows). The latter could probably be formed from the install guides, but the former will need to be produced from scratch or reproduced from blogs (with permission of the author) .. toctree:: :maxdepth: 2 faucet ovs-advanced ovn-sandbox ovn-openstack ovs-2.9.0/Documentation/tutorials/ovn-openstack.rst000066400000000000000000002426511324262074100225240ustar00rootroot00000000000000.. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. Convention for heading levels in Open vSwitch documentation: ======= Heading 0 (reserved for the title in a document) ------- Heading 1 ~~~~~~~ Heading 2 +++++++ Heading 3 ''''''' Heading 4 Avoid deeper levels because they do not render well. ====================== OVN OpenStack Tutorial ====================== This tutorial demonstrates how OVN works in an OpenStack "DevStack" environment. It was tested with the "master" branches of DevStack and Open vSwitch near the beginning of May 2017. Anyone using an earlier version is likely to encounter some differences. In particular, we noticed some shortcomings in OVN utilities while writing the tutorial and pushed out some improvements, so it's best to use recent Open vSwitch at least from that point of view. The goal of this tutorial is to demonstrate OVN in an end-to-end way, that is, to show how it works from the cloud management system at the top (in this case, OpenStack and specifically its Neutron networking subsystem), through the OVN northbound and southbound databases, to the bottom at the OVN local controller and Open vSwitch data plane. We hope that this demonstration makes it easier for users and potential users to understand how OVN works and how to debug and troubleshoot it. In addition to new material, this tutorial incorporates content from ``testing.rst`` in OpenStack networking-ovn, by Russell Bryant and others. Without that example, this tutorial could not have been written. We provide enough details in the tutorial that you should be able to fully follow along, by creating a DevStack VM and cloning DevStack and so on. If you want to do this, start out from `Setting Up DevStack`_ below. Setting Up DevStack ------------------- This section explains how to install DevStack, a kind of OpenStack packaging for developers, in a way that allows you to follow along with the tutorial in full. Unless you have a spare computer laying about, it's easiest to install DevStacck in a virtual machine. This tutorial was built using a VM implemented by KVM and managed by virt-manager. I recommend configuring the VM configured for the x86-64 architecture, 4 GB RAM, 2 VCPUs, and a 20 GB virtual disk. .. note:: If you happen to run your Linux-based host with 32-bit userspace, then you will have some special issues, even if you use a 64-bit kernel: * You may find that you can get 32-bit DevStack VMs to work to some extent, but I personally got tired of finding workarounds. I recommend running your VMs in 64-bit mode. To get this to work, I had to go to the CPUs tab for the VM configuration in virt-manager and change the CPU model from the one originally listed to "Hypervisor Default' (it is curious that this is not the default!). * On a host with 32-bit userspace, KVM supports VMs with at most 2047 MB RAM. This is adequate, barely, to start DevStack, but it is not enough to run multiple (nested) VMs. To prevent out-of-memory failures, set up extra swap space in the guest. For example, to add 2 GB swap:: $ sudo dd if=/dev/zero of=/swapfile bs=1M count=2048 $ sudo mkswap /swapfile $ sudo swapon /swapfile and then add a line like this to ``/etc/fstab`` to add the new swap automatically upon reboot:: /swapfile swap swap defaults 0 0 Here are step-by-step instructions to get started: 1. Install a VM. I tested these instructions with Centos 7.3. Download the "minimal install" ISO and booted it. The install is straightforward. Be sure to enable networking, and set a host name, such as "ovn-devstack-1". Add a regular (non-root) user, and check the box "Make this user administrator". Also, set your time zone. 2. You can SSH into the DevStack VM, instead of running from a console. I recommend it because it's easier to cut and paste commands into a terminal than a VM console. You might also consider using a very wide terminal, perhaps 160 columns, to keep tables from wrapping. To improve convenience further, you can make it easier to log in with the following steps, which are optional: a. On your host, edit your ``~/.ssh/config``, adding lines like the following:: Host ovn-devstack-1 Hostname VMIP User VMUSER where VMIP is the VM's IP address and VMUSER is your username inside the VM. (You can omit the ``User`` line if your username is the same in the host and the VM.) After you do this, you can SSH to the VM by name, e.g. ``ssh ovn-devstack-1``, and if command-line completion is set up in your host shell, you can shorten that to something like ``ssh ovn`` followed by hitting the Tab key. b. If you have SSH public key authentication set up, with an SSH agent, run on your host:: $ ssh-copy-id ovn-devstack-1 and type your password once. Afterward, you can log in without typing your password again. (If you don't already use SSH public key authentication and an agent, consider looking into it--it will save you time in the long run.) c. Optionally, inside the VM, append the following to your ``~/.bash_profile``:: . $HOME/devstack/openrc admin It will save you running it by hand each time you log in. But it also prints garbage to the console, which can screw up services like ``ssh-copy-id``, so be careful. 2. Boot into the installed system and log in as the regular user, then install Git:: $ sudo yum install git .. note:: If you installed a 32-bit i386 guest (against the advice above), install a non-PAE kernel and reboot into it at this point:: $ sudo yum install kernel-core kernel-devel $ sudo reboot Be sure to select the non-PAE kernel from the list at boot. Without this step, DevStack will fail to install properly later. 3. Get copies of DevStack and OVN and set them up:: $ git clone http://git.openstack.org/openstack-dev/devstack.git $ git clone http://git.openstack.org/openstack/networking-ovn.git $ cd devstack $ cp ../networking-ovn/devstack/local.conf.sample local.conf .. note:: If you installed a 32-bit i386 guest (against the advice above), at this point edit ``local.conf`` to add the following line:: CIRROS_ARCH=i386 4. Initialize DevStack:: $ ./stack.sh This will spew many screenfuls of text, and the first time you run it, it will download lots of software from the Internet. The output should eventually end with something like this:: This is your host IP address: 172.16.189.6 This is your host IPv6 address: ::1 Horizon is now available at http://172.16.189.6/dashboard Keystone is serving at http://172.16.189.6/identity/ The default users are: admin and demo The password: password 2017-03-09 15:10:54.117 | stack.sh completed in 2110 seconds. If there's some kind of failure, you can restart by running ``./stack.sh`` again. It won't restart exactly where it left off, but steps up to the one where it failed will skip the download steps. (Sometimes blindly restarting after a failure will allow it to succeed.) If you reboot your VM, you need to rerun this command. (If you run into trouble with ``stack.sh`` after rebooting your VM, try running ``./unstack.sh``.) At this point you can navigate a web browser on your host to the Horizon dashboard URL. Many OpenStack operations can be initiated from this UI. Feel free to explore, but this tutorial focuses on the alternative command-line interfaces because they are easier to explain and to cut and paste. 5. As of this writing, you need to run the following to fix a problem with using VM consoles from the OpenStack web instance:: $ (cd /opt/stack/noVNC && git checkout v0.6.0) See https://serenity-networks.com/how-to-fix-setkeycodes-00-and-unknown-key-pressed-console-errors-on-openstack/ for more details. 6. The firewall in the VM by default allows SSH access but not HTTP. You will probably want HTTP access to use the OpenStack web interface. The following command enables that. (It also enables every other kind of network access, so if you're concerned about security then you might want to find a more targeted approach.) :: $ sudo iptables -F (You need to re-run this if you reboot the VM.) 7. To use OpenStack command line utilities in the tutorial, run:: $ . ~/devstack/openrc admin This needs to be re-run each time you log in (but see the following section). DevStack preliminaries ---------------------- Before we really jump in, let's set up a couple of things in DevStack. This is the first real test that DevStack is working, so if you get errors from any of these commands, it's a sign that ``stack.sh`` didn't finish properly, or perhaps that you didn't run the ``openrc admin`` command at the end of the previous instructions. If you stop and restart DevStack via ``unstack.sh`` followed by ``stack.sh``, you have to rerun these steps. 1. For SSH access to the VMs we're going to create, we'll need a SSH keypair. Later on, we'll get OpenStack to install this keypair into VMs. Create one with:: $ openstack keypair create demo > ~/id_rsa_demo $ chmod 600 ~/id_rsa_demo 2. By default, DevStack security groups drop incoming traffic, but to test networking in a reasonable way we need to enable it. You only need to actually edit one particular security group, but DevStack creates multiple and it's somewhat difficult to figure out which one is important because all of them are named "default". So, the following adds rules to allow SSH and ICMP traffic into **every** security group:: $ for group in $(openstack security group list -f value -c ID); do \ openstack security group rule create --ingress --ethertype IPv4 --dst-port 22 --protocol tcp $group; \ openstack security group rule create --ingress --ethertype IPv4 --protocol ICMP $group; \ done 3. Later on, we're going to create some VMs and we'll need an operating system image to install. DevStack comes with a very simple image built-in, called "cirros", which works fine. We need to get the UUID for this image. Our later commands assume shell variable ``IMAGE_ID`` holds this UUID. You can set this by hand, e.g.:: $ openstack image list +--------------------------------------+--------------------------+--------+ | ID | Name | Status | +--------------------------------------+--------------------------+--------+ | 77f37d2c-3d6b-4e99-a01b-1fa5d78d1fa1 | cirros-0.3.5-x86_64-disk | active | +--------------------------------------+--------------------------+--------+ $ IMAGE_ID=73ca34f3-63c4-4c10-a62f-4540afc24eaa or by parsing CLI output:: $ IMAGE_ID=$(openstack image list -f value -c ID) .. note:: Your image ID will differ from the one above, as will every UUID in this tutorial. They will also change every time you run ``stack.sh``. The UUIDs are generated randomly. Shortening UUIDs ---------------- OpenStack, OVN, and Open vSwitch all really like UUIDs. These are great for uniqueness, but 36-character strings are terrible for readability. Statistically, just the first few characters are enough for uniqueness in small environments, so let's define a helper to make things more readable:: $ abbrev() { a='[0-9a-fA-F]' b=$a$a c=$b$b; sed "s/$b-$c-$c-$c-$c$c$c//g"; } You can use this as a filter to abbreviate UUIDs. For example, use it to abbreviate the above image list:: $ openstack image list -f yaml | abbrev - ID: 77f37d Name: cirros-0.3.5-x86_64-disk Status: active The command above also adds ``-f yaml`` to switch to YAML output format, because abbreviating UUIDs screws up the default table-based formatting and because YAML output doesn't produce wrap columns across lines and therefore is easier to cut and paste. Overview -------- Now that DevStack is ready, with OVN set up as the networking back-end, here's an overview of what we're going to do in the remainder of the demo, all via OpenStack: 1. Switching: Create an OpenStack network ``n1`` and VMs ``a`` and ``b`` attached to it. An OpenStack network is a virtual switch; it corresponds to an OVN logical switch. 2. Routing: Create a second OpenStack network ``n2`` and VM ``c`` attached to it, then connect it to network ``n1`` by creating an OpenStack router and attaching ``n1`` and ``n2`` to it. 3. Gateways: Make VMs ``a`` and ``b`` available via an external network. 4. IPv6: Add IPv6 addresses to our VMs to demonstrate OVN support for IPv6 routing. 5. ACLs: Add and modify OpenStack stateless and stateful rules in security groups. 6. DHCP: How it works in OVN. 7. Further directions: Adding more compute nodes. At each step, we will take a look at how the features in question work from OpenStack's Neutron networking layer at the top to the data plane layer at the bottom. From the highest to lowest level, these layers and the software components that connect them are: * OpenStack Neutron, which as the top level in the system is the authoritative source of the virtual network configuration. We will use OpenStack's ``openstack`` utility to observe and modify Neutron and other OpenStack configuration. * networking-ovn, the Neutron driver that interfaces with OVN and translates the internal Neutron representation of the virtual network into OVN's representation and pushes that representation down the OVN northbound database. In this tutorial it's rarely worth distinguishing Neutron from networking-ovn, so we usually don't break out this layer separately. * The OVN Northbound database, aka NB DB. This is an instance of OVSDB, a simple general-purpose database that is used for multiple purposes in Open vSwitch and OVN. The NB DB's schema is in terms of networking concepts such as switches and routers. The NB DB serves the purpose that in other systems might be filled by some kind of API; for example, in place of calling an API to create or delete a logical switch, networking-ovn performs these operations by inserting or deleting a row in the NB DB's Logical_Switch table. We will use OVN's ``ovn-nbctl`` utility to observe the NB DB. (We won't directly modify data at this layer or below. Because configuration trickles down from Neutron through the stack, the right way to make changes is to use the ``openstack`` utility or another OpenStack interface and then wait for them to percolate through to lower layers.) * The ovn-northd daemon, a program that runs centrally and translates the NB DB's network representation into the lower-level representation used by the OVN Southbound database in the next layer. The details of this daemon are usually not of interest, although without it OVN will not work, so this tutorial does not often mention it. * The OVN Southbound database, aka SB DB, which is also an OVSDB database. Its schema is very different from the NB DB. Instead of familiar networking concepts, the SB DB defines the network in terms of collections of match-action rules called "logical flows", which while similar in concept to OpenFlow flows use logical concepts, such as virtual machine instances, in place of physical concepts like physical Ethernet ports. We will use OVN's ``ovn-sbctl`` utility to observe the SB DB. * The ovn-controller daemon. A copy of ovn-controller runs on each hypervisor. It reads logical flows from the SB DB, translates them into OpenFlow flows, and sends them to Open vSwitch's ovs-vswitchd daemon. Like ovn-northd, usually the details of what this daemon are not of interest, even though it's important to the operation of the system. * ovs-vswitchd. This program runs on each hypervisor. It is the core of Open vSwitch, which processes packets according to the OpenFlow flows set up by ovn-controller. * Open vSwitch datapath. This is essentially a cache designed to accelerate packet processing. Open vSwitch includes a few different datapaths but OVN installations typically use one based on the Open vSwitch Linux kernel module. Switching --------- Switching is the basis of networking in the real world and in virtual networking as well. OpenStack calls its concept of a virtual switch a "network", and OVN calls its corresponding concept a "logical switch". In this step, we'll create an OpenStack network ``n1``, then create VMs ``a`` and ``b`` and attach them to ``n1``. Creating network ``n1`` ~~~~~~~~~~~~~~~~~~~~~~~ Let's start by creating the network:: $ openstack network create --project admin --provider-network-type geneve n1 OpenStack needs to know the subnets that a network serves. We inform it by creating subnet objects. To keep it simple, let's give our network a single subnet for the 10.1.1.0/24 network. We have to give it a name, in this case ``n1subnet``:: $ openstack subnet create --subnet-range 10.1.1.0/24 --network n1 n1subnet If you ask Neutron to show us the available networks, we see ``n1`` as well as the two networks that DevStack creates by default:: $ openstack network list -f yaml | abbrev - ID: 5b6baf Name: n1 Subnets: 5e67e7 - ID: c02c4d Name: private Subnets: d88a34, fd87f9 - ID: d1ac28 Name: public Subnets: 0b1e79, c87dc1 Neutron pushes this network setup down to the OVN northbound database. We can use ``ovn-nbctl show`` to see an overview of what's in the NB DB:: $ ovn-nbctl show | abbrev switch 5b3d5f (neutron-c02c4d) (aka private) port b256dd type: router router-port: lrp-b256dd port f264e7 type: router router-port: lrp-f264e7 switch 2579f4 (neutron-d1ac28) (aka public) port provnet-d1ac28 type: localnet addresses: ["unknown"] port ae9b52 type: router router-port: lrp-ae9b52 switch 3eb263 (neutron-5b6baf) (aka n1) router c59ad2 (neutron-9b057f) (aka router1) port lrp-ae9b52 mac: "fa:16:3e:b2:d2:67" networks: ["172.24.4.9/24", "2001:db8::b/64"] port lrp-b256dd mac: "fa:16:3e:35:33:db" networks: ["fdb0:5860:4ba8::1/64"] port lrp-f264e7 mac: "fa:16:3e:fc:c8:da" networks: ["10.0.0.1/26"] nat 80914c external ip: "172.24.4.9" logical ip: "10.0.0.0/26" type: "snat" This output shows that OVN has three logical switches, each of which corresponds to a Neutron network, and a logical router that corresponds to the Neutron router that DevStack creates by default. The logical switch that corresponds to our new network ``n1`` has no ports yet, because we haven't added any. The ``public`` and ``private`` networks that DevStack creates by default have router ports that connect to the logical router. Using ovn-northd, OVN translates the NB DB's high-level switch and router concepts into lower-level concepts of "logical datapaths" and logical flows. There's one logical datapath for each logical switch or router:: $ ovn-sbctl list datapath_binding | abbrev _uuid : 0ad69d external_ids : {logical-switch="5b3d5f", name="neutron-c02c4d", "name2"=private} tunnel_key : 1 _uuid : a8a758 external_ids : {logical-switch="3eb263", name="neutron-5b6baf", "name2"="n1"} tunnel_key : 4 _uuid : 191256 external_ids : {logical-switch="2579f4", name="neutron-d1ac28", "name2"=public} tunnel_key : 3 _uuid : b87bec external_ids : {logical-router="c59ad2", name="neutron-9b057f", "name2"="router1"} tunnel_key : 2 This output lists the NB DB UUIDs in external_ids:logical-switch and Neutron UUIDs in externals_ids:uuid. We can dive in deeper by viewing the OVN logical flows that implement a logical switch. Our new logical switch is a simple and almost pathological example given that it doesn't yet have any ports attached to it. We'll look at the details a bit later:: $ ovn-sbctl lflow-list n1 | abbrev Datapath: "neutron-5b6baf" aka "n1" (a8a758) Pipeline: ingress table=0 (ls_in_port_sec_l2 ), priority=100 , match=(eth.src[40]), action=(drop;) table=0 (ls_in_port_sec_l2 ), priority=100 , match=(vlan.present), action=(drop;) ... Datapath: "neutron-5b6baf" aka "n1" (a8a758) Pipeline: egress table=0 (ls_out_pre_lb ), priority=0 , match=(1), action=(next;) table=1 (ls_out_pre_acl ), priority=0 , match=(1), action=(next;) ... We have one hypervisor (aka "compute node", in OpenStack parlance), which is the one where we're running all these commands. On this hypervisor, ovn-controller is translating OVN logical flows into OpenFlow flows ("physical flows"). It makes sense to go deeper, to see the OpenFlow flows that get generated from this datapath. By adding ``--ovs`` to the ``ovn-sbctl`` command, we can see OpenFlow flows listed just below their logical flows. We also need to use ``sudo`` because connecting to Open vSwitch is privileged. Go ahead and try it:: $ sudo ovn-sbctl --ovs lflow-list n1 | abbrev Datapath: "neutron-5b6baf" aka "n1" (a8a758) Pipeline: ingress table=0 (ls_in_port_sec_l2 ), priority=100 , match=(eth.src[40]), action=(drop;) table=0 (ls_in_port_sec_l2 ), priority=100 , match=(vlan.present), action=(drop;) ... Datapath: "neutron-5b6baf" aka "n1" (a8a758) Pipeline: egress table=0 (ls_out_pre_lb ), priority=0 , match=(1), action=(next;) table=1 (ls_out_pre_acl ), priority=0 , match=(1), action=(next;) ... You were probably disappointed: the output didn't change, and no OpenFlow flows were printed. That's because no OpenFlow flows are installed for this logical datapath, which in turn is because there are no VIFs for this logical datapath on the local hypervisor. For a better example, you can try ``ovn-sbctl --ovs`` on one of the other logical datapaths. Attaching VMs ~~~~~~~~~~~~~ A switch without any ports is not very interesting. Let's create a couple of VMs and attach them to the switch. Run the following commands, which create VMs named ``a`` and ``b`` and attaches them to our network ``n1`` with IP addresses 10.1.1.5 and 10.1.1.6, respectively. It is not actually necessary to manually assign IP address assignments, since OpenStack is perfectly happy to assign them itself from the subnet's IP address range, but predictable addresses are useful for our discussion:: $ openstack server create --nic net-id=n1,v4-fixed-ip=10.1.1.5 --flavor m1.nano --image $IMAGE_ID --key-name demo a $ openstack server create --nic net-id=n1,v4-fixed-ip=10.1.1.6 --flavor m1.nano --image $IMAGE_ID --key-name demo b These commands return before the VMs are really finished being built. You can run ``openstack server list`` a few times until each of them is shown in the state ACTIVE, which means that they're not just built but already running on the local hypervisor. These operations had the side effect of creating separate "port" objects, but without giving those ports any easy-to-read names. It'll be easier to deal with them later if we can refer to them by name, so let's name ``a``'s port ``ap`` and ``b``'s port ``bp``:: $ openstack port set --name ap $(openstack port list --server a -f value -c ID) $ openstack port set --name bp $(openstack port list --server b -f value -c ID) We'll need to refer to these ports' MAC addresses a few times, so let's put them in variables:: $ AP_MAC=$(openstack port show -f value -c mac_address ap) $ BP_MAC=$(openstack port show -f value -c mac_address bp) At this point you can log into the consoles of the VMs if you like. You can do that from the OpenStack web interface or get a direct URL to paste into a web browser using a command like:: $ openstack console url show -f yaml a (The option ``-f yaml`` keeps the URL in the output from being broken into noncontiguous pieces on a 80-column console.) The VMs don't have many tools in them but ``ping`` and ``ssh`` from one to the other should work fine. The VMs do not have any external network access or DNS configuration. Let's chase down what's changed in OVN. Start with the NB DB at the top of the system. It's clear that our logical switch now has the two logical ports attached to it:: $ ovn-nbctl show | abbrev ... switch 3eb263 (neutron-5b6baf) (aka n1) port c29d41 (aka bp) addresses: ["fa:16:3e:99:7a:17 10.1.1.6"] port 820c08 (aka ap) addresses: ["fa:16:3e:a9:4c:c7 10.1.1.5"] ... We can get some more details on each of these by looking at their NB DB records in the Logical_Switch_Port table. Each port has addressing information, port security enabled, and a pointer to DHCP configuration (which we'll look at much later in `DHCP`_):: $ ovn-nbctl list logical_switch_port ap bp | abbrev _uuid : ef17e5 addresses : ["fa:16:3e:a9:4c:c7 10.1.1.5"] dhcpv4_options : 165974 dhcpv6_options : [] dynamic_addresses : [] enabled : true external_ids : {"neutron:port_name"=ap} name : "820c08" options : {} parent_name : [] port_security : ["fa:16:3e:a9:4c:c7 10.1.1.5"] tag : [] tag_request : [] type : "" up : true _uuid : e8af12 addresses : ["fa:16:3e:99:7a:17 10.1.1.6"] dhcpv4_options : 165974 dhcpv6_options : [] dynamic_addresses : [] enabled : true external_ids : {"neutron:port_name"=bp} name : "c29d41" options : {} parent_name : [] port_security : ["fa:16:3e:99:7a:17 10.1.1.6"] tag : [] tag_request : [] type : "" up : true Now that the logical switch is less pathological, it's worth taking another look at the SB DB logical flow table. Try a command like this:: $ ovn-sbctl lflow-list n1 | abbrev | less -S and then glance through the flows. Packets that egress a VM into the logical switch travel through the flow table's ingress pipeline starting from table 0. At each table, the switch finds the highest-priority logical flow that matches and executes its actions, or if there's no matching flow then the packet is dropped. The ``ovn-sb``\(5) manpage gives all the details, but with a little thought it's possible to guess a lot without reading the manpage. For example, consider the flows in ingress pipeline table 0, which are the first flows encountered by a packet traversing the switch:: table=0 (ls_in_port_sec_l2 ), priority=100 , match=(eth.src[40]), action=(drop;) table=0 (ls_in_port_sec_l2 ), priority=100 , match=(vlan.present), action=(drop;) table=0 (ls_in_port_sec_l2 ), priority=50 , match=(inport == "820c08" && eth.src == {fa:16:3e:a9:4c:c7}), action=(next;) table=0 (ls_in_port_sec_l2 ), priority=50 , match=(inport == "c29d41" && eth.src == {fa:16:3e:99:7a:17}), action=(next;) The first two flows, with priority 100, immediately drop two kinds of invalid packets: those with a multicast or broadcast Ethernet source address (since multicast is only for packet destinations) and those with a VLAN tag (because OVN doesn't yet support VLAN tags inside logical networks). The next two flows implement L2 port security: they advance to the next table for packets with the correct Ethernet source addresses for their ingress ports. A packet that does not match any flow is implicitly dropped, so there's no need for flows to deal with mismatches. The logical flow table includes many other flows, some of which we will look at later. For now, it's most worth looking at ingress table 13:: table=13(ls_in_l2_lkup ), priority=100 , match=(eth.mcast), action=(outport = "_MC_flood"; output;) table=13(ls_in_l2_lkup ), priority=50 , match=(eth.dst == fa:16:3e:99:7a:17), action=(outport = "c29d41"; output;) table=13(ls_in_l2_lkup ), priority=50 , match=(eth.dst == fa:16:3e:a9:4c:c7), action=(outport = "820c08"; output;) The first flow in table 13 checks whether the packet is an Ethernet multicast or broadcast and, if so, outputs it to a special port that egresses to every logical port (other than the ingress port). Otherwise the packet is output to the port corresponding to its Ethernet destination address. Packets addressed to any other Ethernet destination are implicitly dropped. (It's common for an OVN logical switch to know all the MAC addresses supported by its logical ports, like this one. That's why there's no logic here for MAC learning or flooding packets to unknown MAC addresses. OVN does support unknown MAC handling but that's not in play in our example.) .. note:: If you're interested in the details for the multicast group, you can run a command like the following and then look at the row for the correct datapath:: $ ovn-sbctl find multicast_group name=_MC_flood | abbrev Now if you want to look at the OpenFlow flows, you can actually see them. For example, here's the beginning of the output that lists the first four logical flows, which we already looked at above, and their corresponding OpenFlow flows. If you want to know more about the syntax, the ``ovs-fields``\(7) manpage explains OpenFlow matches and ``ovs-ofctl``\(8) explains OpenFlow actions:: $ sudo ovn-sbctl --ovs lflow-list n1 | abbrev Datapath: "neutron-5b6baf" aka "n1" (a8a758) Pipeline: ingress table=0 (ls_in_port_sec_l2 ), priority=100 , match=(eth.src[40]), action=(drop;) table=8 metadata=0x4,dl_src=01:00:00:00:00:00/01:00:00:00:00:00 actions=drop table=0 (ls_in_port_sec_l2 ), priority=100 , match=(vlan.present), action=(drop;) table=8 metadata=0x4,vlan_tci=0x1000/0x1000 actions=drop table=0 (ls_in_port_sec_l2 ), priority=50 , match=(inport == "820c08" && eth.src == {fa:16:3e:a9:4c:c7}), action=(next;) table=8 reg14=0x1,metadata=0x4,dl_src=fa:16:3e:a9:4c:c7 actions=resubmit(,9) table=0 (ls_in_port_sec_l2 ), priority=50 , match=(inport == "c29d41" && eth.src == {fa:16:3e:99:7a:17}), action=(next;) table=8 reg14=0x2,metadata=0x4,dl_src=fa:16:3e:99:7a:17 actions=resubmit(,9) ... Logical Tracing +++++++++++++++ Let's go a level deeper. So far, everything we've done has been fairly general. We can also look at something more specific: the path that a particular packet would take through OVN, logically, and Open vSwitch, physically. Let's use OVN's ovn-trace utility to see what happens to packets from a logical point of view. The ``ovn-trace``\(8) manpage has a lot of detail on how to do that, but let's just start by building up from a simple example. You can start with a command that just specifies the logical datapath, an input port, and nothing else; unspecified fields default to all-zeros. This doesn't do much:: $ ovn-trace n1 'inport == "ap"' ... ingress(dp="n1", inport="ap") ----------------------------- 0. ls_in_port_sec_l2: no match (implicit drop) We see that the packet was dropped in logical table 0, "ls_in_port_sec_l2", the L2 port security stage (as we discussed earlier). That's because we didn't use the right Ethernet source address for ``a``. Let's see what happens if we do:: $ ovn-trace n1 'inport == "ap" && eth.src == '$AP_MAC ... ingress(dp="n1", inport="ap") ----------------------------- 0. ls_in_port_sec_l2 (ovn-northd.c:3234): inport == "ap" && eth.src == {fa:16:3e:a9:4c:c7}, priority 50, uuid 6dcc418a next; 13. ls_in_l2_lkup: no match (implicit drop) Now the packet passes through L2 port security and skips through several other tables until it gets dropped in the L2 lookup stage (because the destination is unknown). Let's add the Ethernet destination for ``b``:: $ ovn-trace n1 'inport == "ap" && eth.src == '$AP_MAC' && eth.dst == '$BP_MAC ... ingress(dp="n1", inport="ap") ----------------------------- 0. ls_in_port_sec_l2 (ovn-northd.c:3234): inport == "ap" && eth.src == {fa:16:3e:a9:4c:c7}, priority 50, uuid 6dcc418a next; 13. ls_in_l2_lkup (ovn-northd.c:3529): eth.dst == fa:16:3e:99:7a:17, priority 50, uuid 57a4c46f outport = "bp"; output; egress(dp="n1", inport="ap", outport="bp") ------------------------------------------ 8. ls_out_port_sec_l2 (ovn-northd.c:3654): outport == "bp" && eth.dst == {fa:16:3e:99:7a:17}, priority 50, uuid 8aa6426d output; /* output to "bp", type "" */ You can see that in this case the packet gets properly switched from ``a`` to ``b``. Physical Tracing for Hypothetical Packets +++++++++++++++++++++++++++++++++++++++++ ovn-trace showed us how a hypothetical packet would travel through the system in a logical fashion, that is, without regard to how VMs are distributed across the physical network. This is a convenient representation for understanding how OVN is **supposed** to work abstractly, but sometimes we might want to know more about how it actually works in the real systems where it is running. For this, we can use the tracing tool that Open vSwitch provides, which traces a hypothetical packet through the OpenFlow tables. We can actually get two levels of detail. Let's start with the version that's easier to interpret, by physically tracing a packet that looks like the one we logically traced before. One obstacle is that we need to know the OpenFlow port number of the input port. One way to do that is to look for a port whose "attached-mac" is the one we expect and print its ofport number:: $ AP_PORT=$(ovs-vsctl --bare --columns=ofport find interface external-ids:attached-mac=\"$AP_MAC\") $ echo $AP_PORT 3 (You could also just do a plain ``ovs-vsctl list interface`` and then look through for the right row and pick its ``ofport`` value.) Now we can feed this input port number into ``ovs-appctl ofproto/trace`` along with the correct Ethernet source and destination addresses and get a physical trace:: $ sudo ovs-appctl ofproto/trace br-int in_port=$AP_PORT,dl_src=$AP_MAC,dl_dst=$BP_MAC Flow: in_port=3,vlan_tci=0x0000,dl_src=fa:16:3e:a9:4c:c7,dl_dst=fa:16:3e:99:7a:17,dl_type=0x0000 bridge("br-int") ---------------- 0. in_port=3, priority 100 set_field:0x8->reg13 set_field:0x9->reg11 set_field:0xa->reg12 set_field:0x4->metadata set_field:0x1->reg14 resubmit(,8) 8. reg14=0x1,metadata=0x4,dl_src=fa:16:3e:a9:4c:c7, priority 50, cookie 0x6dcc418a resubmit(,9) 9. metadata=0x4, priority 0, cookie 0x8fe8689e resubmit(,10) 10. metadata=0x4, priority 0, cookie 0x719549d1 resubmit(,11) 11. metadata=0x4, priority 0, cookie 0x39c99e6f resubmit(,12) 12. metadata=0x4, priority 0, cookie 0x838152a3 resubmit(,13) 13. metadata=0x4, priority 0, cookie 0x918259e3 resubmit(,14) 14. metadata=0x4, priority 0, cookie 0xcad14db2 resubmit(,15) 15. metadata=0x4, priority 0, cookie 0x7834d912 resubmit(,16) 16. metadata=0x4, priority 0, cookie 0x87745210 resubmit(,17) 17. metadata=0x4, priority 0, cookie 0x34951929 resubmit(,18) 18. metadata=0x4, priority 0, cookie 0xd7a8c9fb resubmit(,19) 19. metadata=0x4, priority 0, cookie 0xd02e9578 resubmit(,20) 20. metadata=0x4, priority 0, cookie 0x42d35507 resubmit(,21) 21. metadata=0x4,dl_dst=fa:16:3e:99:7a:17, priority 50, cookie 0x57a4c46f set_field:0x2->reg15 resubmit(,32) 32. priority 0 resubmit(,33) 33. reg15=0x2,metadata=0x4, priority 100 set_field:0xb->reg13 set_field:0x9->reg11 set_field:0xa->reg12 resubmit(,34) 34. priority 0 set_field:0->reg0 set_field:0->reg1 set_field:0->reg2 set_field:0->reg3 set_field:0->reg4 set_field:0->reg5 set_field:0->reg6 set_field:0->reg7 set_field:0->reg8 set_field:0->reg9 resubmit(,40) 40. metadata=0x4, priority 0, cookie 0xde9f3899 resubmit(,41) 41. metadata=0x4, priority 0, cookie 0x74074eff resubmit(,42) 42. metadata=0x4, priority 0, cookie 0x7789c8b1 resubmit(,43) 43. metadata=0x4, priority 0, cookie 0xa6b002c0 resubmit(,44) 44. metadata=0x4, priority 0, cookie 0xaeab2b45 resubmit(,45) 45. metadata=0x4, priority 0, cookie 0x290cc4d4 resubmit(,46) 46. metadata=0x4, priority 0, cookie 0xa3223b88 resubmit(,47) 47. metadata=0x4, priority 0, cookie 0x7ac2132e resubmit(,48) 48. reg15=0x2,metadata=0x4,dl_dst=fa:16:3e:99:7a:17, priority 50, cookie 0x8aa6426d resubmit(,64) 64. priority 0 resubmit(,65) 65. reg15=0x2,metadata=0x4, priority 100 output:4 Final flow: reg11=0x9,reg12=0xa,reg13=0xb,reg14=0x1,reg15=0x2,metadata=0x4,in_port=3,vlan_tci=0x0000,dl_src=fa:16:3e:a9:4c:c7,dl_dst=fa:16:3e:99:7a:17,dl_type=0x0000 Megaflow: recirc_id=0,ct_state=-new-est-rel-rpl-inv-trk,ct_label=0/0x1,in_port=3,vlan_tci=0x0000/0x1000,dl_src=fa:16:3e:a9:4c:c7,dl_dst=fa:16:3e:99:7a:17,dl_type=0x0000 Datapath actions: 4 There's a lot there, which you can read through if you like, but the important part is:: 65. reg15=0x2,metadata=0x4, priority 100 output:4 which means that the packet is ultimately being output to OpenFlow port 4. That's port ``b``, which you can confirm with:: $ sudo ovs-vsctl find interface ofport=4 _uuid : 840a5aca-ea8d-4c16-a11b-a94e0f408091 admin_state : up bfd : {} bfd_status : {} cfm_fault : [] cfm_fault_status : [] cfm_flap_count : [] cfm_health : [] cfm_mpid : [] cfm_remote_mpids : [] cfm_remote_opstate : [] duplex : full error : [] external_ids : {attached-mac="fa:16:3e:99:7a:17", iface-id="c29d4120-20a4-4c44-bd83-8d91f5f447fd", iface-status=active, vm-id="2db969ca-ca2a-4d9a-b49e-f287d39c5645"} ifindex : 9 ingress_policing_burst: 0 ingress_policing_rate: 0 lacp_current : [] link_resets : 1 link_speed : 10000000 link_state : up lldp : {} mac : [] mac_in_use : "fe:16:3e:99:7a:17" mtu : 1500 mtu_request : [] name : "tapc29d4120-20" ofport : 4 ofport_request : [] options : {} other_config : {} statistics : {collisions=0, rx_bytes=4254, rx_crc_err=0, rx_dropped=0, rx_errors=0, rx_frame_err=0, rx_over_err=0, rx_packets=39, tx_bytes=4188, tx_dropped=0, tx_errors=0, tx_packets=39} status : {driver_name=tun, driver_version="1.6", firmware_version=""} type : "" or:: $ BP_PORT=$(ovs-vsctl --bare --columns=ofport find interface external-ids:attached-mac=\"$BP_MAC\") $ echo $BP_PORT 4 Physical Tracing for Real Packets +++++++++++++++++++++++++++++++++ In the previous sections we traced a hypothetical L2 packet, one that's honestly not very realistic: we didn't even supply an Ethernet type, so it defaulted to zero, which isn't anything one would see on a real network. We could refine our packet so that it becomes a more realistic TCP or UDP or ICMP, etc. packet, but let's try a different approach: working from a real packet. Pull up a console for VM ``a`` and start ``ping 10.1.1.6``, then leave it running for the rest of our experiment. Now go back to your DevStack session and run:: $ sudo watch ovs-dpctl dump-flows We're working with a new program. ovn-dpctl is an interface to Open vSwitch datapaths, in this case to the Linux kernel datapath. Its ``dump-flows`` command displays the contents of the in-kernel flow cache, and by running it under the ``watch`` program we see a new snapshot of the flow table every 2 seconds. Look through the output for a flow that begins with ``recirc_id(0)`` and matches the Ethernet source address for ``a``. There is one flow per line, but the lines are very long, so it's easier to read if you make the window very wide. This flow's packet counter should be increasing at a rate of 1 packet per second. It looks something like this:: recirc_id(0),in_port(3),eth(src=fa:16:3e:f5:2a:90),eth_type(0x0800),ipv4(src=10.1.1.5,frag=no), packets:388, bytes:38024, used:0.977s, actions:ct(zone=8),recirc(0x18) We can hand the first part of this (everything up to the first space) to ``ofproto/trace``, and it will tell us what happens:: $ sudo ovs-appctl ofproto/trace 'recirc_id(0),in_port(3),eth(src=fa:16:3e:a9:4c:c7),eth_type(0x0800),ipv4(src=10.1.1.5,dst=10.1.0.0/255.255.0.0,frag=no)' Flow: ip,in_port=3,vlan_tci=0x0000,dl_src=fa:16:3e:a9:4c:c7,dl_dst=00:00:00:00:00:00,nw_src=10.1.1.5,nw_dst=10.1.0.0,nw_proto=0,nw_tos=0,nw_ecn=0,nw_ttl=0 bridge("br-int") ---------------- 0. in_port=3, priority 100 set_field:0x8->reg13 set_field:0x9->reg11 set_field:0xa->reg12 set_field:0x4->metadata set_field:0x1->reg14 resubmit(,8) 8. reg14=0x1,metadata=0x4,dl_src=fa:16:3e:a9:4c:c7, priority 50, cookie 0x6dcc418a resubmit(,9) 9. ip,reg14=0x1,metadata=0x4,dl_src=fa:16:3e:a9:4c:c7,nw_src=10.1.1.5, priority 90, cookie 0x343af48c resubmit(,10) 10. metadata=0x4, priority 0, cookie 0x719549d1 resubmit(,11) 11. ip,metadata=0x4, priority 100, cookie 0x46c089e6 load:0x1->NXM_NX_XXREG0[96] resubmit(,12) 12. metadata=0x4, priority 0, cookie 0x838152a3 resubmit(,13) 13. ip,reg0=0x1/0x1,metadata=0x4, priority 100, cookie 0xd1941634 ct(table=22,zone=NXM_NX_REG13[0..15]) drop Final flow: ip,reg0=0x1,reg11=0x9,reg12=0xa,reg13=0x8,reg14=0x1,metadata=0x4,in_port=3,vlan_tci=0x0000,dl_src=fa:16:3e:a9:4c:c7,dl_dst=00:00:00:00:00:00,nw_src=10.1.1.5,nw_dst=10.1.0.0,nw_proto=0,nw_tos=0,nw_ecn=0,nw_ttl=0 Megaflow: recirc_id=0,ip,in_port=3,vlan_tci=0x0000/0x1000,dl_src=fa:16:3e:a9:4c:c7,nw_src=10.1.1.5,nw_dst=10.1.0.0/16,nw_frag=no Datapath actions: ct(zone=8),recirc(0xb) .. note:: Be careful cutting and pasting ``ovs-dpctl dump-flows`` output into ``ofproto/trace`` because the latter has terrible error reporting. If you add an extra line break, etc., it will likely give you a useless error message. There's no ``output`` action in the output, but there are ``ct`` and ``recirc`` actions (which you can see in the ``Datapath actions`` at the end). The ``ct`` action tells the kernel to pass the packet through the kernel connection tracking for firewalling purposes and the ``recirc`` says to go back to the flow cache for another pass based on the firewall results. The ``0xb`` value inside the ``recirc`` gives us a hint to look at the kernel flows for a cached flow with ``recirc_id(0xb)``. Indeed, there is one:: recirc_id(0xb),in_port(3),ct_state(-new+est-rel-rpl-inv+trk),ct_label(0/0x1),eth(src=fa:16:3e:a9:4c:c7,dst=fa:16:3e:99:7a:17),eth_type(0x0800),ipv4(dst=10.1.1.4/255.255.255.252,frag=no), packets:171, bytes:16758, used:0.271s, actions:ct(zone=11),recirc(0xc) We can then repeat our command with the match part of this kernel flow:: $ sudo ovs-appctl ofproto/trace 'recirc_id(0xb),in_port(3),ct_state(-new+est-rel-rpl-inv+trk),ct_label(0/0x1),eth(src=fa:16:3e:a9:4c:c7,dst=fa:16:3e:99:7a:17),eth_type(0x0800),ipv4(dst=10.1.1.4/255.255.255.252,frag=no)' ... Datapath actions: ct(zone=11),recirc(0xc) In other words, the flow passes through the connection tracker a second time. The first time was for ``a``'s outgoing firewall; this second time is for ``b``'s incoming firewall. Again, we continue tracing with ``recirc_id(0xc)``:: $ sudo ovs-appctl ofproto/trace 'recirc_id(0xc),in_port(3),ct_state(-new+est-rel-rpl-inv+trk),ct_label(0/0x1),eth(src=fa:16:3e:a9:4c:c7,dst=fa:16:3e:99:7a:17),eth_type(0x0800),ipv4(dst=10.1.1.6,proto=1,frag=no)' ... Datapath actions: 4 It was took multiple hops, but we finally came to the end of the line where the packet was output to ``b`` after passing through both firewalls. The port number here is a datapath port number, which is usually different from an OpenFlow port number. To check that it is ``b``'s port, we first list the datapath ports to get the name corresponding to the port number:: $ sudo ovs-dpctl show system@ovs-system: lookups: hit:1994 missed:56 lost:0 flows: 6 masks: hit:2340 total:4 hit/pkt:1.14 port 0: ovs-system (internal) port 1: br-int (internal) port 2: br-ex (internal) port 3: tap820c0888-13 port 4: tapc29d4120-20 and then confirm that this is the port we think it is with a command like this:: $ ovs-vsctl --columns=external-ids list interface tapc29d4120-20 external_ids : {attached-mac="fa:16:3e:99:7a:17", iface-id="c29d4120-20a4-4c44-bd83-8d91f5f447fd", iface-status=active, vm-id="2db969ca-ca2a-4d9a-b49e-f287d39c5645"} Finally, we can relate the OpenFlow flows from our traces back to OVN logical flows. For individual flows, cut and paste a "cookie" value from ``ofproto/trace`` output into ``ovn-sbctl lflow-list``, e.g.:: $ ovn-sbctl lflow-list 0x6dcc418a|abbrev Datapath: "neutron-5b6baf" aka "n1" (a8a758) Pipeline: ingress table=0 (ls_in_port_sec_l2 ), priority=50 , match=(inport == "820c08" && eth.src == {fa:16:3e:a9:4c:c7}), action=(next;) Or, you can pipe ``ofproto/trace`` output through ``ovn-detrace`` to annotate every flow:: $ sudo ovs-appctl ofproto/trace 'recirc_id(0xc),in_port(3),ct_state(-new+est-rel-rpl-inv+trk),ct_label(0/0x1),eth(src=fa:16:3e:a9:4c:c7,dst=fa:16:3e:99:7a:17),eth_type(0x0800),ipv4(dst=10.1.1.6,proto=1,frag=no)' | ovn-detrace ... Routing ------- Previously we set up a pair of VMs ``a`` and ``b`` on a network ``n1`` and demonstrated how packets make their way between them. In this step, we'll set up a second network ``n2`` with a new VM ``c``, connect a router ``r`` to both networks, and demonstrate how routing works in OVN. There's nothing really new for the network and the VM so let's just go ahead and create them:: $ openstack network create --project admin --provider-network-type geneve n2 $ openstack subnet create --subnet-range 10.1.2.0/24 --network n2 n2subnet $ openstack server create --nic net-id=n2,v4-fixed-ip=10.1.2.7 --flavor m1.nano --image $IMAGE_ID --key-name demo c $ openstack port set --name cp $(openstack port list --server c -f value -c ID) $ CP_MAC=$(openstack port show -f value -c mac_address cp) The new network ``n2`` is not yet connected to ``n1`` in any way. You can try tracing a broadcast packet from ``a`` to see, for example, that it doesn't make it to ``c``:: $ ovn-trace n1 'inport == "ap" && eth.src == '$AP_MAC' && eth.dst == '$CP_MAC ... Now create an OpenStack router and connect it to ``n1`` and ``n2``:: $ openstack router create r $ openstack router add subnet r n1subnet $ openstack router add subnet r n2subnet Now ``a``, ``b``, and ``c`` should all be able to reach other. You can get some verification that routing is taking place by running you ``ping`` between ``c`` and one of the other VMs: the reported TTL should be one less than between ``a`` and ``b`` (63 instead of 64). Observe via ``ovn-nbctl`` the new OVN logical switch and router and then ports that connect them together:: $ ovn-nbctl show|abbrev ... switch f51234 (neutron-332346) (aka n2) port 82b983 type: router router-port: lrp-82b983 port 2e585f (aka cp) addresses: ["fa:16:3e:89:f2:36 10.1.2.7"] switch 3eb263 (neutron-5b6baf) (aka n1) port c29d41 (aka bp) addresses: ["fa:16:3e:99:7a:17 10.1.1.6"] port 820c08 (aka ap) addresses: ["fa:16:3e:a9:4c:c7 10.1.1.5"] port 17d870 type: router router-port: lrp-17d870 ... router dde06c (neutron-f88ebc) (aka r) port lrp-82b983 mac: "fa:16:3e:19:9f:46" networks: ["10.1.2.1/24"] port lrp-17d870 mac: "fa:16:3e:f6:e2:8f" networks: ["10.1.1.1/24"] We have not yet looked at the logical flows for an OVN logical router. You might find it of interest to look at them on your own:: $ ovn-sbctl lflow-list r | abbrev | less -S ... Let's grab the ``n1subnet`` router porter MAC address to simplify later commands:: $ N1SUBNET_MAC=$(ovn-nbctl --bare --columns=mac find logical_router_port networks=10.1.1.1/24) Let's see what happens at the logical flow level for an ICMP packet from ``a`` to ``c``. This generates a long trace but an interesting one, so we'll look at it bit by bit. The first three stanzas in the output show the packet's ingress into ``n1`` and processing through the firewall on that side (via the "ct_next" connection-tracking action), and then the selection of the port that leads to router ``r`` as the output port:: $ ovn-trace n1 'inport == "ap" && eth.src == '$AP_MAC' && eth.dst == '$N1SUBNET_MAC' && ip4.src == 10.1.1.5 && ip4.dst == 10.1.2.7 && ip.ttl == 64 && icmp4.type == 8' ... ingress(dp="n1", inport="ap") ----------------------------- 0. ls_in_port_sec_l2 (ovn-northd.c:3234): inport == "ap" && eth.src == {fa:16:3e:a9:4c:c7}, priority 50, uuid 6dcc418a next; 1. ls_in_port_sec_ip (ovn-northd.c:2364): inport == "ap" && eth.src == fa:16:3e:a9:4c:c7 && ip4.src == {10.1.1.5}, priority 90, uuid 343af48c next; 3. ls_in_pre_acl (ovn-northd.c:2646): ip, priority 100, uuid 46c089e6 reg0[0] = 1; next; 5. ls_in_pre_stateful (ovn-northd.c:2764): reg0[0] == 1, priority 100, uuid d1941634 ct_next; ct_next(ct_state=est|trk /* default (use --ct to customize) */) --------------------------------------------------------------- 6. ls_in_acl (ovn-northd.c:2925): !ct.new && ct.est && !ct.rpl && ct_label.blocked == 0 && (inport == "ap" && ip4), priority 2002, uuid a12b39f0 next; 13. ls_in_l2_lkup (ovn-northd.c:3529): eth.dst == fa:16:3e:f6:e2:8f, priority 50, uuid c43ead31 outport = "17d870"; output; egress(dp="n1", inport="ap", outport="17d870") ---------------------------------------------- 1. ls_out_pre_acl (ovn-northd.c:2626): ip && outport == "17d870", priority 110, uuid 60395450 next; 8. ls_out_port_sec_l2 (ovn-northd.c:3654): outport == "17d870", priority 50, uuid 91b5cab0 output; /* output to "17d870", type "patch" */ The next two stanzas represent processing through logical router ``r``. The processing in table 5 is the core of the routing implementation: it recognizes that the packet is destined for an attached subnet, decrements the TTL and updates the Ethernet source address. Table 6 then selects the Ethernet destination address based on the IP destination. The packet then passes to switch ``n2`` via an OVN "logical patch port":: ingress(dp="r", inport="lrp-17d870") ------------------------------------ 0. lr_in_admission (ovn-northd.c:4071): eth.dst == fa:16:3e:f6:e2:8f && inport == "lrp-17d870", priority 50, uuid fa5270b0 next; 5. lr_in_ip_routing (ovn-northd.c:3782): ip4.dst == 10.1.2.0/24, priority 49, uuid 5f9d469f ip.ttl--; reg0 = ip4.dst; reg1 = 10.1.2.1; eth.src = fa:16:3e:19:9f:46; outport = "lrp-82b983"; flags.loopback = 1; next; 6. lr_in_arp_resolve (ovn-northd.c:5088): outport == "lrp-82b983" && reg0 == 10.1.2.7, priority 100, uuid 03d506d3 eth.dst = fa:16:3e:89:f2:36; next; 8. lr_in_arp_request (ovn-northd.c:5260): 1, priority 0, uuid 6dacdd82 output; egress(dp="r", inport="lrp-17d870", outport="lrp-82b983") --------------------------------------------------------- 3. lr_out_delivery (ovn-northd.c:5288): outport == "lrp-82b983", priority 100, uuid 00bea4f2 output; /* output to "lrp-82b983", type "patch" */ Finally the logical switch for ``n2`` runs through the same logic as ``n1`` and the packet is delivered to VM ``c``:: ingress(dp="n2", inport="82b983") --------------------------------- 0. ls_in_port_sec_l2 (ovn-northd.c:3234): inport == "82b983", priority 50, uuid 9a789e06 next; 3. ls_in_pre_acl (ovn-northd.c:2624): ip && inport == "82b983", priority 110, uuid ab52f21a next; 13. ls_in_l2_lkup (ovn-northd.c:3529): eth.dst == fa:16:3e:89:f2:36, priority 50, uuid dcafb3e9 outport = "cp"; output; egress(dp="n2", inport="82b983", outport="cp") ---------------------------------------------- 1. ls_out_pre_acl (ovn-northd.c:2648): ip, priority 100, uuid cd9cfa74 reg0[0] = 1; next; 2. ls_out_pre_stateful (ovn-northd.c:2766): reg0[0] == 1, priority 100, uuid 9e8e22c5 ct_next; ct_next(ct_state=est|trk /* default (use --ct to customize) */) --------------------------------------------------------------- 4. ls_out_acl (ovn-northd.c:2925): !ct.new && ct.est && !ct.rpl && ct_label.blocked == 0 && (outport == "cp" && ip4 && ip4.src == $as_ip4_0fc1b6cf_f925_49e6_8f00_6dd13beca9dc), priority 2002, uuid a746fa0d next; 7. ls_out_port_sec_ip (ovn-northd.c:2364): outport == "cp" && eth.dst == fa:16:3e:89:f2:36 && ip4.dst == {255.255.255.255, 224.0.0.0/4, 10.1.2.7}, priority 90, uuid 4d9862b5 next; 8. ls_out_port_sec_l2 (ovn-northd.c:3654): outport == "cp" && eth.dst == {fa:16:3e:89:f2:36}, priority 50, uuid 0242cdc3 output; /* output to "cp", type "" */ Physical Tracing ~~~~~~~~~~~~~~~~ It's possible to use ``ofproto/trace``, just as before, to trace a packet through OpenFlow tables, either for a hypothetical packet or one that you get from a real test case using ``ovs-dpctl``. The process is just the same as before and the output is almost the same, too. Using a router doesn't actually introduce any interesting new wrinkles, so we'll skip over this for this case and for the remainder of the tutorial, but you can follow the steps on your own if you like. Adding a Gateway ---------------- The VMs that we've created can access each other but they are isolated from the physical world. In OpenStack, the dominant way to connect a VM to external networks is by creating what is called a "floating IP address", which uses network address translation to connect an external address to an internal one. DevStack created a pair of networks named "private" and "public". To use a floating IP address from a VM, we first add a port to the VM with an IP address from the "private" network, then we create a floating IP address on the "public" network, then we associate the port with the floating IP address. Let's add a new VM ``d`` with a floating IP:: $ openstack server create --nic net-id=private --flavor m1.nano --image $IMAGE_ID --key-name demo d $ openstack port set --name dp $(openstack port list --server d -f value -c ID) $ DP_MAC=$(openstack port show -f value -c mac_address dp) $ openstack floating ip create --floating-ip-address 172.24.4.8 public $ openstack server add floating ip d 172.24.4.8 (We specified a particular floating IP address to make the examples easier to follow, but without that OpenStack will automatically allocate one.) It's also necessary to configure the "public" network because DevStack does not do it automatically:: $ sudo ip link set br-ex up $ sudo ip route add 172.24.4.0/24 dev br-ex $ sudo ip addr add 172.24.4.1/24 dev br-ex Now you should be able to "ping" VM ``d`` from the OpenStack host:: $ ping 172.24.4.8 PING 172.24.4.8 (172.24.4.8) 56(84) bytes of data. 64 bytes from 172.24.4.8: icmp_seq=1 ttl=63 time=56.0 ms 64 bytes from 172.24.4.8: icmp_seq=2 ttl=63 time=1.44 ms 64 bytes from 172.24.4.8: icmp_seq=3 ttl=63 time=1.04 ms 64 bytes from 172.24.4.8: icmp_seq=4 ttl=63 time=0.403 ms ^C --- 172.24.4.8 ping statistics --- 4 packets transmitted, 4 received, 0% packet loss, time 3003ms rtt min/avg/max/mdev = 0.403/14.731/56.028/23.845 ms You can also SSH in with the key that we created during setup:: $ ssh -i ~/id_rsa_demo cirros@172.24.4.8 Let's dive in and see how this gets implemented in OVN. First, the relevant parts of the NB DB for the "public" and "private" networks and the router between them:: $ ovn-nbctl show | abbrev switch 2579f4 (neutron-d1ac28) (aka public) port provnet-d1ac28 type: localnet addresses: ["unknown"] port ae9b52 type: router router-port: lrp-ae9b52 switch 5b3d5f (neutron-c02c4d) (aka private) port b256dd type: router router-port: lrp-b256dd port f264e7 type: router router-port: lrp-f264e7 port cae25b (aka dp) addresses: ["fa:16:3e:c1:f5:a2 10.0.0.6 fdb0:5860:4ba8:0:f816:3eff:fec1:f5a2"] ... router c59ad2 (neutron-9b057f) (aka router1) port lrp-ae9b52 mac: "fa:16:3e:b2:d2:67" networks: ["172.24.4.9/24", "2001:db8::b/64"] port lrp-b256dd mac: "fa:16:3e:35:33:db" networks: ["fdb0:5860:4ba8::1/64"] port lrp-f264e7 mac: "fa:16:3e:fc:c8:da" networks: ["10.0.0.1/26"] nat 788c6d external ip: "172.24.4.8" logical ip: "10.0.0.6" type: "dnat_and_snat" nat 80914c external ip: "172.24.4.9" logical ip: "10.0.0.0/26" type: "snat" ... What we see is: * VM ``d`` is on the "private" switch under its private IP address 10.0.0.8. The "private" switch is connected to "router1" via two router ports (one for IPv4, one for IPv6). * The "public" switch is connected to "router1" and to the physical network via a "localnet" port. * "router1" is in the middle between "private" and "public". In addition to the router ports that connect to these switches, it has "nat" entries that direct network address translation. The translation between floating IP address 172.24.4.8 and private address 10.0.0.8 makes perfect sense. When the NB DB gets translated into logical flows at the southbound layer, the "nat" entries get translated into IP matches that then invoke "ct_snat" and "ct_dnat" actions. The details are intricate, but you can get some of the idea by just looking for relevant flows:: $ ovn-sbctl lflow-list router1 | abbrev | grep nat | grep -E '172.24.4.8|10.0.0.8' table=3 (lr_in_unsnat ), priority=100 , match=(ip && ip4.dst == 172.24.4.8 && inport == "lrp-ae9b52" && is_chassis_resident("cr-lrp-ae9b52")), action=(ct_snat;) table=3 (lr_in_unsnat ), priority=50 , match=(ip && ip4.dst == 172.24.4.8), action=(reg9[0] = 1; next;) table=4 (lr_in_dnat ), priority=100 , match=(ip && ip4.dst == 172.24.4.8 && inport == "lrp-ae9b52" && is_chassis_resident("cr-lrp-ae9b52")), action=(ct_dnat(10.0.0.6);) table=4 (lr_in_dnat ), priority=50 , match=(ip && ip4.dst == 172.24.4.8), action=(reg9[0] = 1; next;) table=1 (lr_out_snat ), priority=33 , match=(ip && ip4.src == 10.0.0.6 && outport == "lrp-ae9b52" && is_chassis_resident("cr-lrp-ae9b52")), action=(ct_snat(172.24.4.8);) Let's take a look at how a packet passes through this whole gauntlet. The first two stanzas just show the packet traveling through the "public" network and being forwarded to the "router1" network:: $ ovn-trace public 'inport == "provnet-d1ac2896-18a7-4bca-8f46-b21e2370e5b1" && eth.src == 00:01:02:03:04:05 && eth.dst == fa:16:3e:b2:d2:67 && ip4.src == 172.24.4.1 && ip4.dst == 172.24.4.8 && ip.ttl == 64 && icmp4.type==8' ... ingress(dp="public", inport="provnet-d1ac28") --------------------------------------------- 0. ls_in_port_sec_l2 (ovn-northd.c:3234): inport == "provnet-d1ac28", priority 50, uuid 8d86fb06 next; 10. ls_in_arp_rsp (ovn-northd.c:3266): inport == "provnet-d1ac28", priority 100, uuid 21313eff next; 13. ls_in_l2_lkup (ovn-northd.c:3571): eth.dst == fa:16:3e:b2:d2:67 && is_chassis_resident("cr-lrp-ae9b52"), priority 50, uuid 7f28f51f outport = "ae9b52"; output; egress(dp="public", inport="provnet-d1ac28", outport="ae9b52") -------------------------------------------------------------- 8. ls_out_port_sec_l2 (ovn-northd.c:3654): outport == "ae9b52", priority 50, uuid 72fea396 output; /* output to "ae9b52", type "patch" */ In "router1", first the ``ct_snat`` action without an argument attempts to "un-SNAT" the packet. ovn-trace treats this as a no-op, because it doesn't have any state for tracking connections. As an alternative, it invokes ``ct_dnat(10.0.0.8)`` to NAT the destination IP:: ingress(dp="router1", inport="lrp-ae9b52") ------------------------------------------ 0. lr_in_admission (ovn-northd.c:4071): eth.dst == fa:16:3e:b2:d2:67 && inport == "lrp-ae9b52" && is_chassis_resident("cr-lrp-ae9b52"), priority 50, uuid 8c6945c2 next; 3. lr_in_unsnat (ovn-northd.c:4591): ip && ip4.dst == 172.24.4.8 && inport == "lrp-ae9b52" && is_chassis_resident("cr-lrp-ae9b52"), priority 100, uuid e922f541 ct_snat; ct_snat /* assuming no un-snat entry, so no change */ ----------------------------------------------------- 4. lr_in_dnat (ovn-northd.c:4649): ip && ip4.dst == 172.24.4.8 && inport == "lrp-ae9b52" && is_chassis_resident("cr-lrp-ae9b52"), priority 100, uuid 02f41b79 ct_dnat(10.0.0.6); Still in "router1", the routing and output steps transmit the packet to the "private" network:: ct_dnat(ip4.dst=10.0.0.6) ------------------------- 5. lr_in_ip_routing (ovn-northd.c:3782): ip4.dst == 10.0.0.0/26, priority 53, uuid 86e005b0 ip.ttl--; reg0 = ip4.dst; reg1 = 10.0.0.1; eth.src = fa:16:3e:fc:c8:da; outport = "lrp-f264e7"; flags.loopback = 1; next; 6. lr_in_arp_resolve (ovn-northd.c:5088): outport == "lrp-f264e7" && reg0 == 10.0.0.6, priority 100, uuid 2963d67c eth.dst = fa:16:3e:c1:f5:a2; next; 8. lr_in_arp_request (ovn-northd.c:5260): 1, priority 0, uuid eea419b7 output; egress(dp="router1", inport="lrp-ae9b52", outport="lrp-f264e7") --------------------------------------------------------------- 3. lr_out_delivery (ovn-northd.c:5288): outport == "lrp-f264e7", priority 100, uuid 42dadc23 output; /* output to "lrp-f264e7", type "patch" */ In the "private" network, the packet passes through VM ``d``'s firewall and is output to ``d``:: ingress(dp="private", inport="f264e7") -------------------------------------- 0. ls_in_port_sec_l2 (ovn-northd.c:3234): inport == "f264e7", priority 50, uuid 5b721214 next; 3. ls_in_pre_acl (ovn-northd.c:2624): ip && inport == "f264e7", priority 110, uuid 5bdc3209 next; 13. ls_in_l2_lkup (ovn-northd.c:3529): eth.dst == fa:16:3e:c1:f5:a2, priority 50, uuid 7957f80f outport = "dp"; output; egress(dp="private", inport="f264e7", outport="dp") --------------------------------------------------- 1. ls_out_pre_acl (ovn-northd.c:2648): ip, priority 100, uuid 4981c79d reg0[0] = 1; next; 2. ls_out_pre_stateful (ovn-northd.c:2766): reg0[0] == 1, priority 100, uuid 247e02eb ct_next; ct_next(ct_state=est|trk /* default (use --ct to customize) */) --------------------------------------------------------------- 4. ls_out_acl (ovn-northd.c:2925): !ct.new && ct.est && !ct.rpl && ct_label.blocked == 0 && (outport == "dp" && ip4 && ip4.src == 0.0.0.0/0 && icmp4), priority 2002, uuid b860fc9f next; 7. ls_out_port_sec_ip (ovn-northd.c:2364): outport == "dp" && eth.dst == fa:16:3e:c1:f5:a2 && ip4.dst == {255.255.255.255, 224.0.0.0/4, 10.0.0.6}, priority 90, uuid 15655a98 next; 8. ls_out_port_sec_l2 (ovn-northd.c:3654): outport == "dp" && eth.dst == {fa:16:3e:c1:f5:a2}, priority 50, uuid 5916f94b output; /* output to "dp", type "" */ IPv6 ---- OVN supports IPv6 logical routing. Let's try it out. The first step is to add an IPv6 subnet to networks ``n1`` and ``n2``, then attach those subnets to our router ``r``. As usual, though OpenStack can assign addresses itself, we use fixed ones to make the discussion easier:: $ openstack subnet create --ip-version 6 --subnet-range fc11::/64 --network n1 n1subnet6 $ openstack subnet create --ip-version 6 --subnet-range fc22::/64 --network n2 n2subnet6 $ openstack router add subnet r n1subnet6 $ openstack router add subnet r n2subnet6 Then we add an IPv6 address to each of our VMs:: $ A_PORT_ID=$(openstack port list --server a -f value -c ID) $ openstack port set --fixed-ip subnet=n1subnet6,ip-address=fc11::5 $A_PORT_ID $ B_PORT_ID=$(openstack port list --server b -f value -c ID) $ openstack port set --fixed-ip subnet=n1subnet6,ip-address=fc11::6 $B_PORT_ID $ C_PORT_ID=$(openstack port list --server c -f value -c ID) $ openstack port set --fixed-ip subnet=n2subnet6,ip-address=fc22::7 $C_PORT_ID At least for me, the new IPv6 addresses didn't automatically get propagated into the VMs. To do it by hand, pull up the console for ``a`` and run:: $ sudo ip addr add fc11::5/64 dev eth0 $ sudo ip route add via fc11::1 Then in ``b``:: $ sudo ip addr add fc11::6/64 dev eth0 $ sudo ip route add via fc11::1 Finally in ``c``:: $ sudo ip addr add fc22::7/64 dev eth0 $ sudo ip route add via fc22::1 Now you should have working IPv6 routing through router ``r``. The relevant parts of the NB DB look like the following. The interesting parts are the new ``fc11::`` and ``fc22::`` addresses on the ports in ``n1`` and ``n2`` and the new IPv6 router ports in ``r``:: $ ovn-nbctl show | abbrev ... switch f51234 (neutron-332346) (aka n2) port 1a8162 type: router router-port: lrp-1a8162 port 82b983 type: router router-port: lrp-82b983 port 2e585f (aka cp) addresses: ["fa:16:3e:89:f2:36 10.1.2.7 fc22::7"] switch 3eb263 (neutron-5b6baf) (aka n1) port ad952e type: router router-port: lrp-ad952e port c29d41 (aka bp) addresses: ["fa:16:3e:99:7a:17 10.1.1.6 fc11::6"] port 820c08 (aka ap) addresses: ["fa:16:3e:a9:4c:c7 10.1.1.5 fc11::5"] port 17d870 type: router router-port: lrp-17d870 ... router dde06c (neutron-f88ebc) (aka r) port lrp-1a8162 mac: "fa:16:3e:06:de:ad" networks: ["fc22::1/64"] port lrp-82b983 mac: "fa:16:3e:19:9f:46" networks: ["10.1.2.1/24"] port lrp-ad952e mac: "fa:16:3e:ef:2f:8b" networks: ["fc11::1/64"] port lrp-17d870 mac: "fa:16:3e:f6:e2:8f" networks: ["10.1.1.1/24"] Try tracing a packet from ``a`` to ``c``. The results correspond closely to those for IPv4 which we already discussed back under `Routing`_:: $ N1SUBNET6_MAC=$(ovn-nbctl --bare --columns=mac find logical_router_port networks=\"fc11::1/64\") $ ovn-trace n1 'inport == "ap" && eth.src == '$AP_MAC' && eth.dst == '$N1SUBNET6_MAC' && ip6.src == fc11::5 && ip6.dst == fc22::7 && ip.ttl == 64 && icmp6.type == 8' ... ingress(dp="n1", inport="ap") ----------------------------- 0. ls_in_port_sec_l2 (ovn-northd.c:3234): inport == "ap" && eth.src == {fa:16:3e:a9:4c:c7}, priority 50, uuid 6dcc418a next; 1. ls_in_port_sec_ip (ovn-northd.c:2390): inport == "ap" && eth.src == fa:16:3e:a9:4c:c7 && ip6.src == {fe80::f816:3eff:fea9:4cc7, fc11::5}, priority 90, uuid 604810ea next; 3. ls_in_pre_acl (ovn-northd.c:2646): ip, priority 100, uuid 46c089e6 reg0[0] = 1; next; 5. ls_in_pre_stateful (ovn-northd.c:2764): reg0[0] == 1, priority 100, uuid d1941634 ct_next; ct_next(ct_state=est|trk /* default (use --ct to customize) */) --------------------------------------------------------------- 6. ls_in_acl (ovn-northd.c:2925): !ct.new && ct.est && !ct.rpl && ct_label.blocked == 0 && (inport == "ap" && ip6), priority 2002, uuid 7fdd607e next; 13. ls_in_l2_lkup (ovn-northd.c:3529): eth.dst == fa:16:3e:ef:2f:8b, priority 50, uuid e1d87fc5 outport = "ad952e"; output; egress(dp="n1", inport="ap", outport="ad952e") ---------------------------------------------- 1. ls_out_pre_acl (ovn-northd.c:2626): ip && outport == "ad952e", priority 110, uuid 88f68988 next; 8. ls_out_port_sec_l2 (ovn-northd.c:3654): outport == "ad952e", priority 50, uuid 5935755e output; /* output to "ad952e", type "patch" */ ingress(dp="r", inport="lrp-ad952e") ------------------------------------ 0. lr_in_admission (ovn-northd.c:4071): eth.dst == fa:16:3e:ef:2f:8b && inport == "lrp-ad952e", priority 50, uuid ddfeb712 next; 5. lr_in_ip_routing (ovn-northd.c:3782): ip6.dst == fc22::/64, priority 129, uuid cc2130ec ip.ttl--; xxreg0 = ip6.dst; xxreg1 = fc22::1; eth.src = fa:16:3e:06:de:ad; outport = "lrp-1a8162"; flags.loopback = 1; next; 6. lr_in_arp_resolve (ovn-northd.c:5122): outport == "lrp-1a8162" && xxreg0 == fc22::7, priority 100, uuid bcf75288 eth.dst = fa:16:3e:89:f2:36; next; 8. lr_in_arp_request (ovn-northd.c:5260): 1, priority 0, uuid 6dacdd82 output; egress(dp="r", inport="lrp-ad952e", outport="lrp-1a8162") --------------------------------------------------------- 3. lr_out_delivery (ovn-northd.c:5288): outport == "lrp-1a8162", priority 100, uuid 5260dfc5 output; /* output to "lrp-1a8162", type "patch" */ ingress(dp="n2", inport="1a8162") --------------------------------- 0. ls_in_port_sec_l2 (ovn-northd.c:3234): inport == "1a8162", priority 50, uuid 10957d1b next; 3. ls_in_pre_acl (ovn-northd.c:2624): ip && inport == "1a8162", priority 110, uuid a27ebd00 next; 13. ls_in_l2_lkup (ovn-northd.c:3529): eth.dst == fa:16:3e:89:f2:36, priority 50, uuid dcafb3e9 outport = "cp"; output; egress(dp="n2", inport="1a8162", outport="cp") ---------------------------------------------- 1. ls_out_pre_acl (ovn-northd.c:2648): ip, priority 100, uuid cd9cfa74 reg0[0] = 1; next; 2. ls_out_pre_stateful (ovn-northd.c:2766): reg0[0] == 1, priority 100, uuid 9e8e22c5 ct_next; ct_next(ct_state=est|trk /* default (use --ct to customize) */) --------------------------------------------------------------- 4. ls_out_acl (ovn-northd.c:2925): !ct.new && ct.est && !ct.rpl && ct_label.blocked == 0 && (outport == "cp" && ip6 && ip6.src == $as_ip6_0fc1b6cf_f925_49e6_8f00_6dd13beca9dc), priority 2002, uuid 12fc96f9 next; 7. ls_out_port_sec_ip (ovn-northd.c:2390): outport == "cp" && eth.dst == fa:16:3e:89:f2:36 && ip6.dst == {fe80::f816:3eff:fe89:f236, ff00::/8, fc22::7}, priority 90, uuid c622596a next; 8. ls_out_port_sec_l2 (ovn-northd.c:3654): outport == "cp" && eth.dst == {fa:16:3e:89:f2:36}, priority 50, uuid 0242cdc3 output; /* output to "cp", type "" */ ACLs ---- Let's explore how ACLs work in OpenStack and OVN. In OpenStack, ACL rules are part of "security groups", which are "default deny", that is, packets are not allowed by default and the rules added to security groups serve to allow different classes of packets. The default group (named "default") that is assigned to each of our VMs so far allows all traffic from our other VMs, which isn't very interesting for testing. So, let's create a new security group, which we'll name "custom", add rules to it that allow incoming SSH and ICMP traffic, and apply this security group to VM ``c``:: $ openstack security group create custom $ openstack security group rule create --dst-port 22 custom $ openstack security group rule create --protocol icmp custom $ openstack server remove security group c default $ openstack server add security group c custom Now we can do some experiments to test security groups. From the console on ``a`` or ``b``, it should now be possible to "ping" ``c`` or to SSH to it, but attempts to initiate connections on other ports should be blocked. (You can try to connect on another port with ``ssh -p PORT IP`` or ``nc PORT IP``.) Connection attempts should time out rather than receive the "connection refused" or "connection reset" error that you would see between ``a`` and ``b``. It's also possible to test ACLs via ovn-trace, with one new wrinkle. ovn-trace can't simulate connection tracking state in the network, so by default it assumes that every packet represents an established connection. That's good enough for what we've been doing so far, but for checking properties of security groups we want to look at more detail. If you look back at the VM-to-VM traces we've done until now, you can see that they execute two ``ct_next`` actions: * The first of these is for the packet passing outward through the source VM's firewall. We can tell ovn-trace to treat the packet as starting a new connection or adding to an established connection by adding a ``--ct`` option: ``--ct new`` or ``--ct est``, respectively. The latter is the default and therefore what we've been using so far. We can also use ``--ct est,rpl``, which in addition to ``--ct est`` means that the connection was initiated by the destination VM rather than by the VM sending this packet. * The second is for the packet passing inward through the destination VM's firewall. For this one, it makes sense to tell ovn-trace that the packet is starting a new connection, with ``--ct new``, or that it is a packet sent in reply to a connection established by the destination VM, with ``--ct est,rpl``. ovn-trace uses the ``--ct`` options in order, so if we want to override the second ``ct_next`` behavior we have to specify two options. Another useful ovn-trace option for this testing is ``--minimal``, which reduces the amount of output. In this case we're really just interested in finding out whether the packet reaches the destination VM, that is, whether there's an eventual ``output`` action to ``c``, so ``--minimal`` works fine and the output is easier to read. Try a few traces. For example: * VM ``a`` initiates a new SSH connection to ``c``:: $ ovn-trace --ct new --ct new --minimal n1 'inport == "ap" && eth.src == '$AP_MAC' && eth.dst == '$N1SUBNET6_MAC' && ip4.src == 10.1.1.5 && ip4.dst == 10.1.2.7 && ip.ttl == 64 && tcp.dst == 22' ... ct_next(ct_state=new|trk) { ip.ttl--; eth.src = fa:16:3e:19:9f:46; eth.dst = fa:16:3e:89:f2:36; ct_next(ct_state=new|trk) { output("cp"); }; }; This succeeds, as you can see since there is an ``output`` action. * VM ``a`` initiates a new Telnet connection to ``c``:: $ ovn-trace --ct new --ct new --minimal n1 'inport == "ap" && eth.src == '$AP_MAC' && eth.dst == '$N1SUBNET6_MAC' && ip4.src == 10.1.1.5 && ip4.dst == 10.1.2.7 && ip.ttl == 64 && tcp.dst == 23' ct_next(ct_state=new|trk) { ip.ttl--; eth.src = fa:16:3e:19:9f:46; eth.dst = fa:16:3e:89:f2:36; ct_next(ct_state=new|trk); }; This fails, as you can see from the lack of an ``output`` action. * VM ``a`` replies to a packet that is part of a Telnet connection originally initiated by ``c``:: $ ovn-trace --ct est,rpl --ct est,rpl --minimal n1 'inport == "ap" && eth.src == '$AP_MAC' && eth.dst == '$N1SUBNET6_MAC' && ip4.src == 10.1.1.5 && ip4.dst == 10.1.2.7 && ip.ttl == 64 && tcp.dst == 23' ... ct_next(ct_state=est|rpl|trk) { ip.ttl--; eth.src = fa:16:3e:19:9f:46; eth.dst = fa:16:3e:89:f2:36; ct_next(ct_state=est|rpl|trk) { output("cp"); }; }; This succeeds, as you can see from the ``output`` action, since traffic received in reply to an outgoing connection is always allowed. DHCP ---- As a final demonstration of the OVN architecture, let's examine the DHCP implementation. Like switching, routing, and NAT, the OVN implementation of DHCP involves configuration in the NB DB and logical flows in the SB DB. Let's look at the DHCP support for ``a``'s port ``ap``. The port's Logical_Switch_Port record shows that ``ap`` has DHCPv4 options:: $ ovn-nbctl list logical_switch_port ap | abbrev _uuid : ef17e5 addresses : ["fa:16:3e:a9:4c:c7 10.1.1.5 fc11::5"] dhcpv4_options : 165974 dhcpv6_options : 26f7cd dynamic_addresses : [] enabled : true external_ids : {"neutron:port_name"=ap} name : "820c08" options : {} parent_name : [] port_security : ["fa:16:3e:a9:4c:c7 10.1.1.5 fc11::5"] tag : [] tag_request : [] type : "" up : true We can then list them either by UUID or, more easily, by port name:: $ ovn-nbctl list dhcp_options ap | abbrev _uuid : 165974 cidr : "10.1.1.0/24" external_ids : {subnet_id="5e67e7"} options : {lease_time="43200", mtu="1442", router="10.1.1.1", server_id="10.1.1.1", server_mac="fa:16:3e:bb:94:72"} These options show the basic DHCP configuration for the subnet. They do not include the IP address itself, which comes from the Logical_Switch_Port record. This allows a whole Neutron subnet to share a single DHCP_Options record. You can see this sharing in action, if you like, by listing the record for port ``bp``, which is on the same subnet as ``ap``, and see that it is the same record as before:: $ ovn-nbctl list dhcp_options bp | abbrev _uuid : 165974 cidr : "10.1.1.0/24" external_ids : {subnet_id="5e67e7"} options : {lease_time="43200", mtu="1442", router="10.1.1.1", server_id="10.1.1.1", server_mac="fa:16:3e:bb:94:72"} You can take another look at the southbound flow table if you like, but the best demonstration is to trace a DHCP packet. The following is a trace of a DHCP request inbound from ``ap``. The first part is just the usual travel through the firewall:: $ ovn-trace n1 'inport == "ap" && eth.src == '$AP_MAC' && eth.dst == ff:ff:ff:ff:ff:ff && ip4.dst == 255.255.255.255 && udp.src == 68 && udp.dst == 67 && ip.ttl == 1' ... ingress(dp="n1", inport="ap") ----------------------------- 0. ls_in_port_sec_l2 (ovn-northd.c:3234): inport == "ap" && eth.src == {fa:16:3e:a9:4c:c7}, priority 50, uuid 6dcc418a next; 1. ls_in_port_sec_ip (ovn-northd.c:2325): inport == "ap" && eth.src == fa:16:3e:a9:4c:c7 && ip4.src == 0.0.0.0 && ip4.dst == 255.255.255.255 && udp.src == 68 && udp.dst == 67, priority 90, uuid e46bed6f next; 3. ls_in_pre_acl (ovn-northd.c:2646): ip, priority 100, uuid 46c089e6 reg0[0] = 1; next; 5. ls_in_pre_stateful (ovn-northd.c:2764): reg0[0] == 1, priority 100, uuid d1941634 ct_next; The next part is the new part. First, an ACL in table 6 allows a DHCP request to pass through. In table 11, the special ``put_dhcp_opts`` action replaces a DHCPDISCOVER or DHCPREQUEST packet by a reply. Table 12 flips the packet's source and destination and sends it back the way it came in:: 6. ls_in_acl (ovn-northd.c:2925): !ct.new && ct.est && !ct.rpl && ct_label.blocked == 0 && (inport == "ap" && ip4 && ip4.dst == {255.255.255.255, 10.1.1.0/24} && udp && udp.src == 68 && udp.dst == 67), priority 2002, uuid 9c90245d next; 11. ls_in_dhcp_options (ovn-northd.c:3409): inport == "ap" && eth.src == fa:16:3e:a9:4c:c7 && ip4.src == 0.0.0.0 && ip4.dst == 255.255.255.255 && udp.src == 68 && udp.dst == 67, priority 100, uuid 8d63f29c reg0[3] = put_dhcp_opts(offerip = 10.1.1.5, lease_time = 43200, mtu = 1442, netmask = 255.255.255.0, router = 10.1.1.1, server_id = 10.1.1.1); /* We assume that this packet is DHCPDISCOVER or DHCPREQUEST. */ next; 12. ls_in_dhcp_response (ovn-northd.c:3438): inport == "ap" && eth.src == fa:16:3e:a9:4c:c7 && ip4 && udp.src == 68 && udp.dst == 67 && reg0[3], priority 100, uuid 995eeaa9 eth.dst = eth.src; eth.src = fa:16:3e:bb:94:72; ip4.dst = 10.1.1.5; ip4.src = 10.1.1.1; udp.src = 67; udp.dst = 68; outport = inport; flags.loopback = 1; output; Then the last part is just traveling back through the firewall to VM ``a``:: egress(dp="n1", inport="ap", outport="ap") ------------------------------------------ 1. ls_out_pre_acl (ovn-northd.c:2648): ip, priority 100, uuid 3752b746 reg0[0] = 1; next; 2. ls_out_pre_stateful (ovn-northd.c:2766): reg0[0] == 1, priority 100, uuid 0c066ea1 ct_next; ct_next(ct_state=est|trk /* default (use --ct to customize) */) --------------------------------------------------------------- 4. ls_out_acl (ovn-northd.c:3008): outport == "ap" && eth.src == fa:16:3e:bb:94:72 && ip4.src == 10.1.1.1 && udp && udp.src == 67 && udp.dst == 68, priority 34000, uuid 0b383e77 ct_commit; next; 7. ls_out_port_sec_ip (ovn-northd.c:2364): outport == "ap" && eth.dst == fa:16:3e:a9:4c:c7 && ip4.dst == {255.255.255.255, 224.0.0.0/4, 10.1.1.5}, priority 90, uuid 7b8cbcd5 next; 8. ls_out_port_sec_l2 (ovn-northd.c:3654): outport == "ap" && eth.dst == {fa:16:3e:a9:4c:c7}, priority 50, uuid b874ece8 output; /* output to "ap", type "" */ Further Directions ------------------ We've looked at a fair bit of how OVN works and how it interacts with OpenStack. If you still have some interest, then you might want to explore some of these directions: * Adding more than one hypervisor ("compute node", in OpenStack parlance). OVN connects compute nodes by tunneling packets with the STT or Geneve protocols. OVN scales to 1000 compute nodes or more, but two compute nodes demonstrate the principle. All of the tools and techniques we demonstrated also work with multiple compute nodes. * Container support. OVN supports seamlessly connecting VMs to containers, whether the containers are hosted on "bare metal" or nested inside VMs. OpenStack support for containers, however, is still evolving, and too difficult to incorporate into the tutorial at this point. * Other kinds of gateways. In addition to floating IPs with NAT, OVN supports directly attaching VMs to a physical network and connecting logical switches to VTEP hardware. ovs-2.9.0/Documentation/tutorials/ovn-sandbox.rst000066400000000000000000000156051324262074100221700ustar00rootroot00000000000000.. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. Convention for heading levels in Open vSwitch documentation: ======= Heading 0 (reserved for the title in a document) ------- Heading 1 ~~~~~~~ Heading 2 +++++++ Heading 3 ''''''' Heading 4 Avoid deeper levels because they do not render well. =========== OVN Sandbox =========== This tutorial shows you how to explore features using ``ovs-sandbox`` as a simulated test environment. It's assumed that you have an understanding of OVS before going through this tutorial. Detail about OVN is covered in ovn-architecture_, but this tutorial lets you quickly see it in action. Getting Started --------------- For some general information about ``ovs-sandbox``, see the "Getting Started" section of the tutorial_. ``ovs-sandbox`` does not include OVN support by default. To enable OVN, you must pass the ``--ovn`` flag. For example, if running it straight from the ovs git tree you would run:: $ make sandbox SANDBOXFLAGS="--ovn" Running the sandbox with OVN enabled does the following additional steps to the environment: 1. Creates the ``OVN_Northbound`` and ``OVN_Southbound`` databases as described in `ovn-nb(5)`_ and `ovn-sb(5)`_. 2. Creates a backup server for ``OVN_Southbond`` database. Sandbox launch screen provides the instructions on accessing the backup database. However access to the backup server is not required to go through the tutorial. 3. Creates the ``hardware_vtep`` database as described in `vtep(5)`_. 4. Runs the `ovn-northd(8)`_, `ovn-controller(8)`_, and `ovn-controller-vtep(8)`_ daemons. 5. Makes OVN and VTEP utilities available for use in the environment, including `vtep-ctl(8)`_, `ovn-nbctl(8)`_, and `ovn-sbctl(8)`_. Using GDB --------- GDB support is not required to go through the tutorial. See the "Using GDB" section of the `tutorial`_ for more info. Additional flags exist for launching the debugger for the OVN programs:: --gdb-ovn-northd --gdb-ovn-controller --gdb-ovn-controller-vtep Creating OVN Resources ---------------------- Once you have ``ovs-sandbox`` running with OVN enabled, you can start using OVN utilities to create resources in OVN. As an example, we will create an environment that has two logical switches connected by a logical router. Create the first logical switch with one port:: $ ovn-nbctl ls-add sw0 $ ovn-nbctl lsp-add sw0 sw0-port1 $ ovn-nbctl lsp-set-addresses sw0-port1 "50:54:00:00:00:01 192.168.0.2" Create the second logical switch with one port:: $ ovn-nbctl ls-add sw1 $ ovn-nbctl lsp-add sw1 sw1-port1 $ ovn-nbctl lsp-set-addresses sw1-port1 "50:54:00:00:00:03 11.0.0.2" Create the logical router and attach both logical switches:: $ ovn-nbctl lr-add lr0 $ ovn-nbctl lrp-add lr0 lrp0 00:00:00:00:ff:01 192.168.0.1/24 $ ovn-nbctl lsp-add sw0 lrp0-attachment $ ovn-nbctl lsp-set-type lrp0-attachment router $ ovn-nbctl lsp-set-addresses lrp0-attachment 00:00:00:00:ff:01 $ ovn-nbctl lsp-set-options lrp0-attachment router-port=lrp0 $ ovn-nbctl lrp-add lr0 lrp1 00:00:00:00:ff:02 11.0.0.1/24 $ ovn-nbctl lsp-add sw1 lrp1-attachment $ ovn-nbctl lsp-set-type lrp1-attachment router $ ovn-nbctl lsp-set-addresses lrp1-attachment 00:00:00:00:ff:02 $ ovn-nbctl lsp-set-options lrp1-attachment router-port=lrp1 View a summary of OVN's current logical configuration:: $ ovn-nbctl show switch 1396cf55-d176-4082-9a55-1c06cef626e4 (sw1) port lrp1-attachment addresses: ["00:00:00:00:ff:02"] port sw1-port1 addresses: ["50:54:00:00:00:03 11.0.0.2"] switch 2c9d6d03-09fc-4e32-8da6-305f129b0d53 (sw0) port lrp0-attachment addresses: ["00:00:00:00:ff:01"] port sw0-port1 addresses: ["50:54:00:00:00:01 192.168.0.2"] router f8377e8c-f75e-4fc8-8751-f3ea03c6dd98 (lr0) port lrp0 mac: "00:00:00:00:ff:01" networks: ["192.168.0.1/24"] port lrp1 mac: "00:00:00:00:ff:02" networks: ["11.0.0.1/24"] The ``tutorial`` directory of the OVS source tree includes a script that runs all of the commands for you:: $ ./ovn-setup.sh Using ovn-trace --------------- Once you have configured resources in OVN, try using ``ovn-trace`` to see how OVN would process a sample packet through its logical pipeline. For example, we can trace an IP packet from ``sw0-port1`` to ``sw1-port1``. The ``--minimal`` output shows each visible action performed on the packet, which includes: #. The logical router will decrement the IP TTL field. #. The logical router will change the source and destination MAC addresses to reflect the next hop. #. The packet will be output to ``sw1-port1``. :: $ ovn-trace --minimal sw0 'inport == "sw0-port1" \ > && eth.src == 50:54:00:00:00:01 && ip4.src == 192.168.0.2 \ > && eth.dst == 00:00:00:00:ff:01 && ip4.dst == 11.0.0.2 \ > && ip.ttl == 64' # ip,reg14=0x1,vlan_tci=0x0000,dl_src=50:54:00:00:00:01,dl_dst=00:00:00:00:ff:01,nw_src=192.168.0.2,nw_dst=11.0.0.2,nw_proto=0,nw_tos=0,nw_ecn=0,nw_ttl=64 ip.ttl--; eth.src = 00:00:00:00:ff:02; eth.dst = 50:54:00:00:00:03; output("sw1-port1"); The ``ovn-trace`` utility can also provide much more detail on how the packet would be processed through OVN's logical pipeline, as well as correlate that to OpenFlow flows programmed by ``ovn-controller``. See the `ovn-trace(8)`_ man page for more detail. .. _ovn-architecture: http://openvswitch.org/support/dist-docs/ovn-architecture.7.html .. _Tutorial: :ref:`ovs-advanced` .. _ovn-nb(5): http://openvswitch.org/support/dist-docs/ovn-nb.5.html .. _ovn-sb(5): http://openvswitch.org/support/dist-docs/ovn-sb.5.html .. _vtep(5): http://openvswitch.org/support/dist-docs/vtep.5.html .. _ovn-northd(8): http://openvswitch.org/support/dist-docs/ovn-northd.8.html .. _ovn-controller(8): http://openvswitch.org/support/dist-docs/ovn-controller.8.html .. _ovn-controller-vtep(8): http://openvswitch.org/support/dist-docs/ovn-controller-vtep.8.html .. _vtep-ctl(8): http://openvswitch.org/support/dist-docs/vtep-ctl.8.html .. _ovn-nbctl(8): http://openvswitch.org/support/dist-docs/ovn-nbctl.8.html .. _ovn-sbctl(8): http://openvswitch.org/support/dist-docs/ovn-sbctl.8.html .. _ovn-trace(8): http://openvswitch.org/support/dist-docs/ovn-trace.8.html ovs-2.9.0/Documentation/tutorials/ovs-advanced.rst000066400000000000000000001077631324262074100223130ustar00rootroot00000000000000.. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. Convention for heading levels in Open vSwitch documentation: ======= Heading 0 (reserved for the title in a document) ------- Heading 1 ~~~~~~~ Heading 2 +++++++ Heading 3 ''''''' Heading 4 Avoid deeper levels because they do not render well. ============================== Open vSwitch Advanced Features ============================== Many tutorials cover the basics of OpenFlow. This is not such a tutorial. Rather, a knowledge of the basics of OpenFlow is a prerequisite. If you do not already understand how an OpenFlow flow table works, please go read a basic tutorial and then continue reading here afterward. It is also important to understand the basics of Open vSwitch before you begin. If you have never used ovs-vsctl or ovs-ofctl before, you should learn a little about them before proceeding. Most of the features covered in this tutorial are Open vSwitch extensions to OpenFlow. Also, most of the features in this tutorial are specific to the software Open vSwitch implementation. If you are using an Open vSwitch port to an ASIC-based hardware switch, this tutorial will not help you. This tutorial does not cover every aspect of the features that it mentions. You can find the details elsewhere in the Open vSwitch documentation, especially ``ovs-ofctl(8)`` and the comments in the ``include/openflow/nicira-ext.h`` and ``include/openvswitch/meta-flow.h`` header files. Getting Started --------------- This is a hands-on tutorial. To get the most out of it, you will need Open vSwitch binaries. You do not, on the other hand, need any physical networking hardware or even supervisor privilege on your system. Instead, we will use a script called ``ovs-sandbox``, which accompanies the tutorial, that constructs a software simulated network environment based on Open vSwitch. You can use ``ovs-sandbox`` three ways: * If you have already installed Open vSwitch on your system, then you should be able to just run ``ovs-sandbox`` from this directory without any options. * If you have not installed Open vSwitch (and you do not want to install it), then you can build Open vSwitch according to the instructions in :doc:`/intro/install/general`, without installing it. Then run ``./ovs-sandbox -b DIRECTORY`` from this directory, substituting the Open vSwitch build directory for ``DIRECTORY``. * As a slight variant on the latter, you can run ``make sandbox`` from an Open vSwitch build directory. When you run ``ovs-sandbox``, it does the following: 1. **CAUTION:** Deletes any subdirectory of the current directory named "sandbox" and any files in that directory. 2. Creates a new directory "sandbox" in the current directory. 3. Sets up special environment variables that ensure that Open vSwitch programs will look inside the "sandbox" directory instead of in the Open vSwitch installation directory. 4. If you are using a built but not installed Open vSwitch, installs the Open vSwitch manpages in a subdirectory of "sandbox" and adjusts the ``MANPATH`` environment variable to point to this directory. This means that you can use, for example, ``man ovs-vsctl`` to see a manpage for the ``ovs-vsctl`` program that you built. 5. Creates an empty Open vSwitch configuration database under "sandbox". 6. Starts ``ovsdb-server`` running under "sandbox". 7. Starts ``ovs-vswitchd`` running under "sandbox", passing special options that enable a special "dummy" mode for testing. 8. Starts a nested interactive shell inside "sandbox". At this point, you can run all the usual Open vSwitch utilities from the nested shell environment. You can, for example, use ``ovs-vsctl`` to create a bridge:: $ ovs-vsctl add-br br0 From Open vSwitch's perspective, the bridge that you create this way is as real as any other. You can, for example, connect it to an OpenFlow controller or use ``ovs-ofctl`` to examine and modify it and its OpenFlow flow table. On the other hand, the bridge is not visible to the operating system's network stack, so ``ip`` cannot see it or affect it, which means that utilities like ``ping`` and ``tcpdump`` will not work either. (That has its good side, too: you can't screw up your computer's network stack by manipulating a sandboxed OVS.) When you're done using OVS from the sandbox, exit the nested shell (by entering the "exit" shell command or pressing Control+D). This will kill the daemons that ``ovs-sandbox`` started, but it leaves the "sandbox" directory and its contents in place. The sandbox directory contains log files for the Open vSwitch dameons. You can examine them while you're running in the sandboxed environment or after you exit. Using GDB --------- GDB support is not required to go through the tutorial. It is added in case user wants to explore the internals of OVS programs. GDB can already be used to debug any running process, with the usual ``gdb `` command. ``ovs-sandbox`` also has a ``-g`` option for launching ovs-vswitchd under GDB. This option can be handy for setting break points before ovs-vswitchd runs, or for catching early segfaults. Similarly, a ``-d`` option can be used to run ovsdb-server under GDB. Both options can be specified at the same time. In addition, a ``-e`` option also launches ovs-vswitchd under GDB. However, instead of displaying a ``gdb>`` prompt and waiting for user input, ovs-vswitchd will start to execute immediately. ``-r`` option is the corresponding option for running ovsdb-server under gdb with immediate execution. To avoid GDB mangling with the sandbox sub shell terminal, ``ovs-sandbox`` starts a new xterm to run each GDB session. For systems that do not support X windows, GDB support is effectively disabled. When launching sandbox through the build tree's make file, the ``-g`` option can be passed via the ``SANDBOXFLAGS`` environment variable. ``make sandbox SANDBOXFLAGS=-g`` will start the sandbox with ovs-vswitchd running under GDB in its own xterm if X is available. Motivation ---------- The goal of this tutorial is to demonstrate the power of Open vSwitch flow tables. The tutorial works through the implementation of a MAC-learning switch with VLAN trunk and access ports. Outside of the Open vSwitch features that we will discuss, OpenFlow provides at least two ways to implement such a switch: 1. An OpenFlow controller to implement MAC learning in a "reactive" fashion. Whenever a new MAC appears on the switch, or a MAC moves from one switch port to another, the controller adjusts the OpenFlow flow table to match. 2. The "normal" action. OpenFlow defines this action to submit a packet to "the traditional non-OpenFlow pipeline of the switch". That is, if a flow uses this action, then the packets in the flow go through the switch in the same way that they would if OpenFlow was not configured on the switch. Each of these approaches has unfortunate pitfalls. In the first approach, using an OpenFlow controller to implement MAC learning, has a significant cost in terms of network bandwidth and latency. It also makes the controller more difficult to scale to large numbers of switches, which is especially important in environments with thousands of hypervisors (each of which contains a virtual OpenFlow switch). MAC learning at an OpenFlow controller also behaves poorly if the OpenFlow controller fails, slows down, or becomes unavailable due to network problems. The second approach, using the "normal" action, has different problems. First, little about the "normal" action is standardized, so it behaves differently on switches from different vendors, and the available features and how those features are configured (usually not through OpenFlow) varies widely. Second, "normal" does not work well with other OpenFlow actions. It is "all-or-nothing", with little potential to adjust its behavior slightly or to compose it with other features. Scenario -------- We will construct Open vSwitch flow tables for a VLAN-capable, MAC-learning switch that has four ports: p1 a trunk port that carries all VLANs, on OpenFlow port 1. p2 an access port for VLAN 20, on OpenFlow port 2. p3, p4 both access ports for VLAN 30, on OpenFlow ports 3 and 4, respectively. .. note:: The ports' names are not significant. You could call them eth1 through eth4, or any other names you like. .. note:: An OpenFlow switch always has a "local" port as well. This scenario won't use the local port. Our switch design will consist of five main flow tables, each of which implements one stage in the switch pipeline: Table 0 Admission control. Table 1 VLAN input processing. Table 2 Learn source MAC and VLAN for ingress port. Table 3 Look up learned port for destination MAC and VLAN. Table 4 Output processing. The section below describes how to set up the scenario, followed by a section for each OpenFlow table. You can cut and paste the ``ovs-vsctl`` and ``ovs-ofctl`` commands in each of the sections below into your ``ovs-sandbox`` shell. They are also available as shell scripts in this directory, named ``t-setup``, ``t-stage0``, ``t-stage1``, ..., ``t-stage4``. The ``ovs-appctl`` test commands are intended for cutting and pasting and are not supplied separately. Setup ----- To get started, start ``ovs-sandbox``. Inside the interactive shell that it starts, run this command:: $ ovs-vsctl add-br br0 -- set Bridge br0 fail-mode=secure This command creates a new bridge "br0" and puts "br0" into so-called "fail-secure" mode. For our purpose, this just means that the OpenFlow flow table starts out empty. .. note:: If we did not do this, then the flow table would start out with a single flow that executes the "normal" action. We could use that feature to yield a switch that behaves the same as the switch we are currently building, but with the caveats described under "Motivation" above.) The new bridge has only one port on it so far, the "local port" br0. We need to add ``p1``, ``p2``, ``p3``, and ``p4``. A shell ``for`` loop is one way to do it:: for i in 1 2 3 4; do ovs-vsctl add-port br0 p$i -- set Interface p$i ofport_request=$i ovs-ofctl mod-port br0 p$i up done In addition to adding a port, the ``ovs-vsctl`` command above sets its ``ofport_request`` column to ensure that port ``p1`` is assigned OpenFlow port 1, ``p2`` is assigned OpenFlow port 2, and so on. .. note:: We could omit setting the ofport_request and let Open vSwitch choose port numbers for us, but it's convenient for the purposes of this tutorial because we can talk about OpenFlow port 1 and know that it corresponds to ``p1``. The ``ovs-ofctl`` command above brings up the simulated interfaces, which are down initially, using an OpenFlow request. The effect is similar to ``ip link up``, but the sandbox's interfaces are not visible to the operating system and therefore ``ip`` would not affect them. We have not configured anything related to VLANs or MAC learning. That's because we're going to implement those features in the flow table. To see what we've done so far to set up the scenario, you can run a command like ``ovs-vsctl show`` or ``ovs-ofctl show br0``. Implementing Table 0: Admission control --------------------------------------- Table 0 is where packets enter the switch. We use this stage to discard packets that for one reason or another are invalid. For example, packets with a multicast source address are not valid, so we can add a flow to drop them at ingress to the switch with:: $ ovs-ofctl add-flow br0 \ "table=0, dl_src=01:00:00:00:00:00/01:00:00:00:00:00, actions=drop" A switch should also not forward IEEE 802.1D Spanning Tree Protocol (STP) packets, so we can also add a flow to drop those and other packets with reserved multicast protocols:: $ ovs-ofctl add-flow br0 \ "table=0, dl_dst=01:80:c2:00:00:00/ff:ff:ff:ff:ff:f0, actions=drop" We could add flows to drop other protocols, but these demonstrate the pattern. We need one more flow, with a priority lower than the default, so that flows that don't match either of the "drop" flows we added above go on to pipeline stage 1 in OpenFlow table 1:: $ ovs-ofctl add-flow br0 "table=0, priority=0, actions=resubmit(,1)" .. note:: The "resubmit" action is an Open vSwitch extension to OpenFlow. Testing Table 0 --------------- If we were using Open vSwitch to set up a physical or a virtual switch, then we would naturally test it by sending packets through it one way or another, perhaps with common network testing tools like ``ping`` and ``tcpdump`` or more specialized tools like Scapy. That's difficult with our simulated switch, since it's not visible to the operating system. But our simulated switch has a few specialized testing tools. The most powerful of these tools is ``ofproto/trace``. Given a switch and the specification of a flow, ``ofproto/trace`` shows, step-by-step, how such a flow would be treated as it goes through the switch. Example 1 ~~~~~~~~~ Try this command:: $ ovs-appctl ofproto/trace br0 in_port=1,dl_dst=01:80:c2:00:00:05 The output should look something like this:: Flow: in_port=1,vlan_tci=0x0000,dl_src=00:00:00:00:00:00,dl_dst=01:80:c2:00:00:05,dl_type=0x0000 bridge("br0") ------------- 0. dl_dst=01:80:c2:00:00:00/ff:ff:ff:ff:ff:f0, priority 32768 drop Final flow: unchanged Megaflow: recirc_id=0,in_port=1,dl_src=00:00:00:00:00:00/01:00:00:00:00:00,dl_dst=01:80:c2:00:00:00/ff:ff:ff:ff:ff:f0,dl_type=0x0000 Datapath actions: drop The first line shows the flow being traced, in slightly greater detail than specified on the command line. It is mostly zeros because unspecified fields default to zeros. The second group of lines shows the packet's trip through bridge br0. We see, in table 0, the OpenFlow flow that the fields matched, along with its priority, followed by its actions, one per line. In this case, we see that this packet that has a reserved multicast destination address matches the flow that drops those packets. The final block of lines summarizes the results, which are not very interesting here. Example 2 ~~~~~~~~~ Try another command:: $ ovs-appctl ofproto/trace br0 in_port=1,dl_dst=01:80:c2:00:00:10 The output should be:: Flow: in_port=1,vlan_tci=0x0000,dl_src=00:00:00:00:00:00,dl_dst=01:80:c2:00:00:10,dl_type=0x0000 bridge("br0") ------------- 0. priority 0 resubmit(,1) 1. No match. drop Final flow: unchanged Megaflow: recirc_id=0,in_port=1,dl_src=00:00:00:00:00:00/01:00:00:00:00:00,dl_dst=01:80:c2:00:00:10/ff:ff:ff:ff:ff:f0,dl_type=0x0000 Datapath actions: drop This time the flow we handed to ``ofproto/trace`` doesn't match any of our "drop" flows in table 0, so it falls through to the low-priority "resubmit" flow. The "resubmit" causes a second lookup in OpenFlow table 1, described by the block of text that starts with "1." We haven't yet added any flows to OpenFlow table 1, so no flow actually matches in the second lookup. Therefore, the packet is still actually dropped, which means that the externally observable results would be identical to our first example. Implementing Table 1: VLAN Input Processing ------------------------------------------- A packet that enters table 1 has already passed basic validation in table 0. The purpose of table 1 is validate the packet's VLAN, based on the VLAN configuration of the switch port through which the packet entered the switch. We will also use it to attach a VLAN header to packets that arrive on an access port, which allows later processing stages to rely on the packet's VLAN always being part of the VLAN header, reducing special cases. Let's start by adding a low-priority flow that drops all packets, before we add flows that pass through acceptable packets. You can think of this as a "default drop" flow:: $ ovs-ofctl add-flow br0 "table=1, priority=0, actions=drop" Our trunk port ``p1``, on OpenFlow port 1, is an easy case. ``p1`` accepts any packet regardless of whether it has a VLAN header or what the VLAN was, so we can add a flow that resubmits everything on input port 1 to the next table:: $ ovs-ofctl add-flow br0 \ "table=1, priority=99, in_port=1, actions=resubmit(,2)" On the access ports, we want to accept any packet that has no VLAN header, tag it with the access port's VLAN number, and then pass it along to the next stage:: $ ovs-ofctl add-flows br0 - <<'EOF' table=1, priority=99, in_port=2, vlan_tci=0, actions=mod_vlan_vid:20, resubmit(,2) table=1, priority=99, in_port=3, vlan_tci=0, actions=mod_vlan_vid:30, resubmit(,2) table=1, priority=99, in_port=4, vlan_tci=0, actions=mod_vlan_vid:30, resubmit(,2) EOF We don't write any flows that match packets with 802.1Q that enter this stage on any of the access ports, so the "default drop" flow we added earlier causes them to be dropped, which is ordinarily what we want for access ports. .. note:: Another variation of access ports allows ingress of packets tagged with VLAN 0 (aka 802.1p priority tagged packets). To allow such packets, replace ``vlan_tci=0`` by ``vlan_tci=0/0xfff`` above. Testing Table 1 --------------- ``ofproto/trace`` allows us to test the ingress VLAN flows that we added above. Example 1: Packet on Trunk Port ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Here's a test of a packet coming in on the trunk port:: $ ovs-appctl ofproto/trace br0 in_port=1,vlan_tci=5 The output shows the lookup in table 0, the resubmit to table 1, and the resubmit to table 2 (which does nothing because we haven't put anything there yet):: Flow: in_port=1,vlan_tci=0x0005,dl_src=00:00:00:00:00:00,dl_dst=00:00:00:00:00:00,dl_type=0x0000 bridge("br0") ------------- 0. priority 0 resubmit(,1) 1. in_port=1, priority 99 resubmit(,2) 2. No match. drop Final flow: unchanged Megaflow: recirc_id=0,in_port=1,dl_src=00:00:00:00:00:00/01:00:00:00:00:00,dl_dst=00:00:00:00:00:00/ff:ff:ff:ff:ff:f0,dl_type=0x0000 Datapath actions: drop Example 2: Valid Packet on Access Port ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Here's a test of a valid packet (a packet without an 802.1Q header) coming in on access port ``p2``:: $ ovs-appctl ofproto/trace br0 in_port=2 The output is similar to that for the previous case, except that it additionally tags the packet with ``p2``'s VLAN 20 before it passes it along to table 2:: Flow: in_port=2,vlan_tci=0x0000,dl_src=00:00:00:00:00:00,dl_dst=00:00:00:00:00:00,dl_type=0x0000 bridge("br0") ------------- 0. priority 0 resubmit(,1) 1. in_port=2,vlan_tci=0x0000, priority 99 mod_vlan_vid:20 resubmit(,2) 2. No match. drop Final flow: in_port=2,dl_vlan=20,dl_vlan_pcp=0,dl_src=00:00:00:00:00:00,dl_dst=00:00:00:00:00:00,dl_type=0x0000 Megaflow: recirc_id=0,in_port=2,vlan_tci=0x0000,dl_src=00:00:00:00:00:00/01:00:00:00:00:00,dl_dst=00:00:00:00:00:00/ff:ff:ff:ff:ff:f0,dl_type=0x0000 Datapath actions: drop Example 3: Invalid Packet on Access Port ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ This tests an invalid packet (one that includes an 802.1Q header) coming in on access port ``p2``:: $ ovs-appctl ofproto/trace br0 in_port=2,vlan_tci=5 The output shows the packet matching the default drop flow:: Flow: in_port=2,vlan_tci=0x0005,dl_src=00:00:00:00:00:00,dl_dst=00:00:00:00:00:00,dl_type=0x0000 bridge("br0") ------------- 0. priority 0 resubmit(,1) 1. priority 0 drop Final flow: unchanged Megaflow: recirc_id=0,in_port=2,vlan_tci=0x0005,dl_src=00:00:00:00:00:00/01:00:00:00:00:00,dl_dst=00:00:00:00:00:00/ff:ff:ff:ff:ff:f0,dl_type=0x0000 Datapath actions: drop Implementing Table 2: MAC+VLAN Learning for Ingress Port -------------------------------------------------------- This table allows the switch we're implementing to learn that the packet's source MAC is located on the packet's ingress port in the packet's VLAN. .. note:: This table is a good example why table 1 added a VLAN tag to packets that entered the switch through an access port. We want to associate a MAC+VLAN with a port regardless of whether the VLAN in question was originally part of the packet or whether it was an assumed VLAN associated with an access port. It only takes a single flow to do this. The following command adds it:: $ ovs-ofctl add-flow br0 \ "table=2 actions=learn(table=10, NXM_OF_VLAN_TCI[0..11], \ NXM_OF_ETH_DST[]=NXM_OF_ETH_SRC[], \ load:NXM_OF_IN_PORT[]->NXM_NX_REG0[0..15]), \ resubmit(,3)" The "learn" action (an Open vSwitch extension to OpenFlow) modifies a flow table based on the content of the flow currently being processed. Here's how you can interpret each part of the "learn" action above: ``table=10`` Modify flow table 10. This will be the MAC learning table. ``NXM_OF_VLAN_TCI[0..11]`` Make the flow that we add to flow table 10 match the same VLAN ID that the packet we're currently processing contains. This effectively scopes the MAC learning entry to a single VLAN, which is the ordinary behavior for a VLAN-aware switch. ``NXM_OF_ETH_DST[]=NXM_OF_ETH_SRC[]`` Make the flow that we add to flow table 10 match, as Ethernet destination, the Ethernet source address of the packet we're currently processing. ``load:NXM_OF_IN_PORT[]->NXM_NX_REG0[0..15]`` Whereas the preceding parts specify fields for the new flow to match, this specifies an action for the flow to take when it matches. The action is for the flow to load the ingress port number of the current packet into register 0 (a special field that is an Open vSwitch extension to OpenFlow). .. note:: A real use of "learn" for MAC learning would probably involve two additional elements. First, the "learn" action would specify a hard_timeout for the new flow, to enable a learned MAC to eventually expire if no new packets were seen from a given source within a reasonable interval. Second, one would usually want to limit resource consumption by using the Flow_Table table in the Open vSwitch configuration database to specify a maximum number of flows in table 10. This definitely calls for examples. Testing Table 2 --------------- Example 1 ~~~~~~~~~ Try the following test command:: $ ovs-appctl ofproto/trace br0 \ in_port=1,vlan_tci=20,dl_src=50:00:00:00:00:01 -generate The output shows that "learn" was executed in table 2 and the particular flow that was added:: Flow: in_port=1,vlan_tci=0x0014,dl_src=50:00:00:00:00:01,dl_dst=00:00:00:00:00:00,dl_type=0x0000 bridge("br0") ------------- 0. priority 0 resubmit(,1) 1. in_port=1, priority 99 resubmit(,2) 2. priority 32768 learn(table=10,NXM_OF_VLAN_TCI[0..11],NXM_OF_ETH_DST[]=NXM_OF_ETH_SRC[],load:NXM_OF_IN_PORT[]->NXM_NX_REG0[0..15]) -> table=10 vlan_tci=0x0014/0x0fff,dl_dst=50:00:00:00:00:01 priority=32768 actions=load:0x1->NXM_NX_REG0[0..15] resubmit(,3) 3. No match. drop Final flow: unchanged Megaflow: recirc_id=0,in_port=1,vlan_tci=0x0014/0x1fff,dl_src=50:00:00:00:00:01,dl_dst=00:00:00:00:00:00/ff:ff:ff:ff:ff:f0,dl_type=0x0000 Datapath actions: drop The ``-generate`` keyword is new. Ordinarily, ``ofproto/trace`` has no side effects: "output" actions do not actually output packets, "learn" actions do not actually modify the flow table, and so on. With ``-generate``, though, ``ofproto/trace`` does execute "learn" actions. That's important now, because we want to see the effect of the "learn" action on table 10. You can see that by running:: $ ovs-ofctl dump-flows br0 table=10 which (omitting the ``duration`` and ``idle_age`` fields, which will vary based on how soon you ran this command after the previous one, as well as some other uninteresting fields) prints something like:: NXST_FLOW reply (xid=0x4): table=10, vlan_tci=0x0014/0x0fff,dl_dst=50:00:00:00:00:01 actions=load:0x1->NXM_NX_REG0[0..15] You can see that the packet coming in on VLAN ``20`` with source MAC ``50:00:00:00:00:01`` became a flow that matches VLAN ``20`` (written in hexadecimal) and destination MAC ``50:00:00:00:00:01``. The flow loads port number ``1``, the input port for the flow we tested, into register 0. Example 2 ~~~~~~~~~ Here's a second test command:: $ ovs-appctl ofproto/trace br0 \ in_port=2,dl_src=50:00:00:00:00:01 -generate The flow that this command tests has the same source MAC and VLAN as example 1, although the VLAN comes from an access port VLAN rather than an 802.1Q header. If we again dump the flows for table 10 with:: $ ovs-ofctl dump-flows br0 table=10 then we see that the flow we saw previously has changed to indicate that the learned port is port 2, as we would expect:: NXST_FLOW reply (xid=0x4): table=10, vlan_tci=0x0014/0x0fff,dl_dst=50:00:00:00:00:01 actions=load:0x2->NXM_NX_REG0[0..15] Implementing Table 3: Look Up Destination Port ---------------------------------------------- This table figures out what port we should send the packet to based on the destination MAC and VLAN. That is, if we've learned the location of the destination (from table 2 processing some previous packet with that destination as its source), then we want to send the packet there. We need only one flow to do the lookup:: $ ovs-ofctl add-flow br0 \ "table=3 priority=50 actions=resubmit(,10), resubmit(,4)" The flow's first action resubmits to table 10, the table that the "learn" action modifies. As you saw previously, the learned flows in this table write the learned port into register 0. If the destination for our packet hasn't been learned, then there will be no matching flow, and so the "resubmit" turns into a no-op. Because registers are initialized to 0, we can use a register 0 value of 0 in our next pipeline stage as a signal to flood the packet. The second action resubmits to table 4, continuing to the next pipeline stage. We can add another flow to skip the learning table lookup for multicast and broadcast packets, since those should always be flooded:: $ ovs-ofctl add-flow br0 \ "table=3 priority=99 dl_dst=01:00:00:00:00:00/01:00:00:00:00:00 \ actions=resubmit(,4)" .. note:: We don't strictly need to add this flow, because multicast addresses will never show up in our learning table. (In turn, that's because we put a flow into table 0 to drop packets that have a multicast source address.) Testing Table 3 --------------- Example ~~~~~~~ Here's a command that should cause OVS to learn that ``f0:00:00:00:00:01`` is on ``p1`` in VLAN ``20``:: $ ovs-appctl ofproto/trace br0 \ in_port=1,dl_vlan=20,dl_src=f0:00:00:00:00:01,dl_dst=90:00:00:00:00:01 \ -generate The output shows (from the "no match" looking up the resubmit to table 10) that the flow's destination was unknown:: Flow: in_port=1,dl_vlan=20,dl_vlan_pcp=0,dl_src=f0:00:00:00:00:01,dl_dst=90:00:00:00:00:01,dl_type=0x0000 bridge("br0") ------------- 0. priority 0 resubmit(,1) 1. in_port=1, priority 99 resubmit(,2) 2. priority 32768 learn(table=10,NXM_OF_VLAN_TCI[0..11],NXM_OF_ETH_DST[]=NXM_OF_ETH_SRC[],load:NXM_OF_IN_PORT[]->NXM_NX_REG0[0..15]) -> table=10 vlan_tci=0x0014/0x0fff,dl_dst=f0:00:00:00:00:01 priority=32768 actions=load:0x1->NXM_NX_REG0[0..15] resubmit(,3) 3. priority 50 resubmit(,10) 10. No match. drop resubmit(,4) 4. No match. drop Final flow: unchanged Megaflow: recirc_id=0,in_port=1,dl_vlan=20,dl_src=f0:00:00:00:00:01,dl_dst=90:00:00:00:00:01,dl_type=0x0000 Datapath actions: drop There are two ways that you can verify that the packet's source was learned. The most direct way is to dump the learning table with:: $ ovs-ofctl dump-flows br0 table=10 which ought to show roughly the following, with extraneous details removed:: table=10, vlan_tci=0x0014/0x0fff,dl_dst=f0:00:00:00:00:01 actions=load:0x1->NXM_NX_REG0[0..15] .. note:: If you tried the examples for the previous step, or if you did some of your own experiments, then you might see additional flows there. These additional flows are harmless. If they bother you, then you can remove them with `ovs-ofctl del-flows br0 table=10`. The other way is to inject a packet to take advantage of the learning entry. For example, we can inject a packet on p2 whose destination is the MAC address that we just learned on p1: $ ovs-appctl ofproto/trace br0 \ in_port=2,dl_src=90:00:00:00:00:01,dl_dst=f0:00:00:00:00:01 -generate Here is this command's output. Take a look at the lines that trace the ``resubmit(,10)``, showing that the packet matched the learned flow for the first MAC we used, loading the OpenFlow port number for the learned port ``p1`` into register ``0``:: Flow: in_port=2,vlan_tci=0x0000,dl_src=90:00:00:00:00:01,dl_dst=f0:00:00:00:00:01,dl_type=0x0000 bridge("br0") ------------- 0. priority 0 resubmit(,1) 1. in_port=2,vlan_tci=0x0000, priority 99 mod_vlan_vid:20 resubmit(,2) 2. priority 32768 learn(table=10,NXM_OF_VLAN_TCI[0..11],NXM_OF_ETH_DST[]=NXM_OF_ETH_SRC[],load:NXM_OF_IN_PORT[]->NXM_NX_REG0[0..15]) -> table=10 vlan_tci=0x0014/0x0fff,dl_dst=90:00:00:00:00:01 priority=32768 actions=load:0x2->NXM_NX_REG0[0..15] resubmit(,3) 3. priority 50 resubmit(,10) 10. vlan_tci=0x0014/0x0fff,dl_dst=f0:00:00:00:00:01, priority 32768 load:0x1->NXM_NX_REG0[0..15] resubmit(,4) 4. No match. drop Final flow: reg0=0x1,in_port=2,dl_vlan=20,dl_vlan_pcp=0,dl_src=90:00:00:00:00:01,dl_dst=f0:00:00:00:00:01,dl_type=0x0000 Megaflow: recirc_id=0,in_port=2,vlan_tci=0x0000,dl_src=90:00:00:00:00:01,dl_dst=f0:00:00:00:00:01,dl_type=0x0000 Datapath actions: drop If you read the commands above carefully, then you might have noticed that they simply have the Ethernet source and destination addresses exchanged. That means that if we now rerun the first ``ovs-appctl`` command above, e.g.: $ ovs-appctl ofproto/trace br0 \ in_port=1,dl_vlan=20,dl_src=f0:00:00:00:00:01,dl_dst=90:00:00:00:00:01 \ -generate then we see in the output, looking at the indented "load" action executed in table 10, that the destination has now been learned:: Flow: in_port=1,dl_vlan=20,dl_vlan_pcp=0,dl_src=f0:00:00:00:00:01,dl_dst=90:00:00:00:00:01,dl_type=0x0000 bridge("br0") ------------- 0. priority 0 resubmit(,1) 1. in_port=1, priority 99 resubmit(,2) 2. priority 32768 learn(table=10,NXM_OF_VLAN_TCI[0..11],NXM_OF_ETH_DST[]=NXM_OF_ETH_SRC[],load:NXM_OF_IN_PORT[]->NXM_NX_REG0[0..15]) -> table=10 vlan_tci=0x0014/0x0fff,dl_dst=f0:00:00:00:00:01 priority=32768 actions=load:0x1->NXM_NX_REG0[0..15] resubmit(,3) 3. priority 50 resubmit(,10) 10. vlan_tci=0x0014/0x0fff,dl_dst=90:00:00:00:00:01, priority 32768 load:0x2->NXM_NX_REG0[0..15] resubmit(,4) 4. No match. drop Implementing Table 4: Output Processing --------------------------------------- At entry to stage 4, we know that register 0 contains either the desired output port or is zero if the packet should be flooded. We also know that the packet's VLAN is in its 802.1Q header, even if the VLAN was implicit because the packet came in on an access port. The job of the final pipeline stage is to actually output packets. The job is trivial for output to our trunk port ``p1``:: $ ovs-ofctl add-flow br0 "table=4 reg0=1 actions=1" For output to the access ports, we just have to strip the VLAN header before outputting the packet:: $ ovs-ofctl add-flows br0 - <<'EOF' table=4 reg0=2 actions=strip_vlan,2 table=4 reg0=3 actions=strip_vlan,3 table=4 reg0=4 actions=strip_vlan,4 EOF The only slightly tricky part is flooding multicast and broadcast packets and unicast packets with unlearned destinations. For those, we need to make sure that we only output the packets to the ports that carry our packet's VLAN, and that we include the 802.1Q header in the copy output to the trunk port but not in copies output to access ports:: $ ovs-ofctl add-flows br0 - <<'EOF' table=4 reg0=0 priority=99 dl_vlan=20 actions=1,strip_vlan,2 table=4 reg0=0 priority=99 dl_vlan=30 actions=1,strip_vlan,3,4 table=4 reg0=0 priority=50 actions=1 EOF .. note:: Our flows rely on the standard OpenFlow behavior that an output action will not forward a packet back out the port it came in on. That is, if a packet comes in on p1, and we've learned that the packet's destination MAC is also on p1, so that we end up with ``actions=1`` as our actions, the switch will not forward the packet back out its input port. The multicast/broadcast/unknown destination cases above also rely on this behavior. Testing Table 4 --------------- Example 1: Broadcast, Multicast, and Unknown Destination ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Try tracing a broadcast packet arriving on ``p1`` in VLAN ``30``:: $ ovs-appctl ofproto/trace br0 \ in_port=1,dl_dst=ff:ff:ff:ff:ff:ff,dl_vlan=30 The interesting part of the output is the final line, which shows that the switch would remove the 802.1Q header and then output the packet to ``p3`` and ``p4``, which are access ports for VLAN ``30``:: Datapath actions: pop_vlan,3,4 Similarly, if we trace a broadcast packet arriving on ``p3``:: $ ovs-appctl ofproto/trace br0 in_port=3,dl_dst=ff:ff:ff:ff:ff:ff then we see that it is output to ``p1`` with an 802.1Q tag and then to ``p4`` without one:: Datapath actions: push_vlan(vid=30,pcp=0),1,pop_vlan,4 .. note:: Open vSwitch could simplify the datapath actions here to just ``4,push_vlan(vid=30,pcp=0),1`` but it is not smart enough to do so. The following are also broadcasts, but the result is to drop the packets because the VLAN only belongs to the input port:: $ ovs-appctl ofproto/trace br0 \ in_port=1,dl_dst=ff:ff:ff:ff:ff:ff $ ovs-appctl ofproto/trace br0 \ in_port=1,dl_dst=ff:ff:ff:ff:ff:ff,dl_vlan=55 Try some other broadcast cases on your own:: $ ovs-appctl ofproto/trace br0 \ in_port=1,dl_dst=ff:ff:ff:ff:ff:ff,dl_vlan=20 $ ovs-appctl ofproto/trace br0 \ in_port=2,dl_dst=ff:ff:ff:ff:ff:ff $ ovs-appctl ofproto/trace br0 \ in_port=4,dl_dst=ff:ff:ff:ff:ff:ff You can see the same behavior with multicast packets and with unicast packets whose destination has not been learned, e.g.:: $ ovs-appctl ofproto/trace br0 \ in_port=4,dl_dst=01:00:00:00:00:00 $ ovs-appctl ofproto/trace br0 \ in_port=1,dl_dst=90:12:34:56:78:90,dl_vlan=20 $ ovs-appctl ofproto/trace br0 \ in_port=1,dl_dst=90:12:34:56:78:90,dl_vlan=30 Example 2: MAC Learning ~~~~~~~~~~~~~~~~~~~~~~~ Let's follow the same pattern as we did for table 3. First learn a MAC on port ``p1`` in VLAN ``30``:: $ ovs-appctl ofproto/trace br0 \ in_port=1,dl_vlan=30,dl_src=10:00:00:00:00:01,dl_dst=20:00:00:00:00:01 \ -generate You can see from the last line of output that the packet's destination is unknown, so it gets flooded to both ``p3`` and ``p4``, the other ports in VLAN ``30``:: Datapath actions: pop_vlan,3,4 Then reverse the MACs and learn the first flow's destination on port ``p4``:: $ ovs-appctl ofproto/trace br0 \ in_port=4,dl_src=20:00:00:00:00:01,dl_dst=10:00:00:00:00:01 -generate The last line of output shows that the this packet's destination is known to be ``p1``, as learned from our previous command:: Datapath actions: push_vlan(vid=30,pcp=0),1 Now, if we rerun our first command:: $ ovs-appctl ofproto/trace br0 \ in_port=1,dl_vlan=30,dl_src=10:00:00:00:00:01,dl_dst=20:00:00:00:00:01 \ -generate ...we can see that the result is no longer a flood but to the specified learned destination port ``p4``: Datapath actions: pop_vlan,4 Contact ======= bugs@openvswitch.org http://openvswitch.org/ ovs-2.9.0/MAINTAINERS.rst000066400000000000000000000047301324262074100146420ustar00rootroot00000000000000.. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. Convention for heading levels in Open vSwitch documentation: ======= Heading 0 (reserved for the title in a document) ------- Heading 1 ~~~~~~~ Heading 2 +++++++ Heading 3 ''''''' Heading 4 Avoid deeper levels because they do not render well. ========== Committers ========== Open vSwitch committers are the people who have been granted access to push changes to to the Open vSwitch git repository. The responsibilities of an Open vSwitch committer are documented `here `__. The process for adding or removing committers is documented `here `__. This is the current list of active Open vSwitch committers: .. list-table:: OVS Maintainers :header-rows: 1 * - Name - Email * - Alex Wang - ee07b291@gmail.com * - Alin Serdean - aserdean@cloudbasesolutions.com * - Andy Zhou - azhou@ovn.org * - Ansis Atteka - aatteka@nicira.com * - Ben Pfaff - blp@ovn.org * - Daniele Di Proietto - daniele.di.proietto@gmail.com * - Ethan J. Jackson - ejj@eecs.berkeley.edu * - Gurucharan Shetty - guru@ovn.org * - Jarno Rajahalme - jarno@ovn.org * - Jesse Gross - jesse@kernel.org * - Joe Stringer - joe@ovn.org * - Justin Pettit - jpettit@ovn.org * - Pravin B Shelar - pshelar@ovn.org * - Russell Bryant - russell@ovn.org * - Simon Horman - horms@ovn.org * - Thomas Graf - tgraf@noironetworks.com * - YAMAMOTO Takashi - yamamoto@midokura.com The project also maintains a list of Emeritus Committers (or Maintainers). More information about Emeritus Committers can be found `here `__. .. list-table:: OVS Emeritus Maintainers :header-rows: 1 * - Name - Email * - - ovs-2.9.0/Makefile.am000066400000000000000000000366351324262074100144030ustar00rootroot00000000000000# Copyright (C) 2007-2017 Nicira, Inc. # # Copying and distribution of this file, with or without modification, # are permitted in any medium without royalty provided the copyright # notice and this notice are preserved. This file is offered as-is, # without warranty of any kind. AUTOMAKE_OPTIONS = foreign subdir-objects ACLOCAL_AMFLAGS = -I m4 SUBDIRS = datapath AM_CPPFLAGS = $(SSL_CFLAGS) AM_LDFLAGS = $(SSL_LDFLAGS) AM_LDFLAGS += $(OVS_LDFLAGS) if WIN32 AM_CPPFLAGS += -I $(top_srcdir)/include/windows AM_CPPFLAGS += -I $(top_srcdir)/datapath-windows/include AM_CPPFLAGS += $(PTHREAD_INCLUDES) AM_CPPFLAGS += $(MSVC_CFLAGS) AM_LDFLAGS += $(PTHREAD_LDFLAGS) AM_LDFLAGS += $(MSVC64_LDFLAGS) PLATFORM = $(MSVC_PLATFORM) endif AM_CPPFLAGS += -I $(top_srcdir)/include AM_CPPFLAGS += -I $(top_builddir)/include AM_CPPFLAGS += -I $(top_srcdir)/lib AM_CPPFLAGS += -I $(top_builddir)/lib AM_CPPFLAGS += $(SSL_INCLUDES) AM_CFLAGS = -Wstrict-prototypes AM_CFLAGS += $(WARNING_FLAGS) AM_CFLAGS += $(OVS_CFLAGS) if DPDK_NETDEV AM_CFLAGS += -D_FILE_OFFSET_BITS=64 DPDKSTRIP_FLAGS = --dpdk else DPDKSTRIP_FLAGS = --nodpdk endif if NDEBUG AM_CPPFLAGS += -DNDEBUG AM_CFLAGS += -fomit-frame-pointer endif AM_CTAGSFLAGS = $(OVS_CTAGS_IDENTIFIERS_LIST) if WIN32 psep=";" else psep=":" endif # PYTHONDONTWRITEBYTECODE=yes keeps Python from creating .pyc and .pyo # files. Creating .py[co] works OK for any given version of Open # vSwitch, but it causes trouble if you switch from a version with # foo/__init__.py into an (older) version with plain foo.py, since # foo/__init__.pyc will cause Python to ignore foo.py. run_python = \ PYTHONPATH=$(top_srcdir)/python$(psep)$$PYTHONPATH \ PYTHONDONTWRITEBYTECODE=yes $(PYTHON) ALL_LOCAL = BUILT_SOURCES = CLEANFILES = CLEAN_LOCAL = DISTCLEANFILES = PYCOV_CLEAN_FILES = build-aux/check-structs,cover EXTRA_DIST = \ AUTHORS.rst \ CONTRIBUTING.rst \ MAINTAINERS.rst \ README.rst \ NOTICE \ .travis.yml \ .travis/linux-build.sh \ .travis/linux-prepare.sh \ .travis/osx-build.sh \ .travis/osx-prepare.sh \ appveyor.yml \ boot.sh \ build-aux/cccl \ build-aux/cksum-schema-check \ build-aux/calculate-schema-cksum \ build-aux/dist-docs \ build-aux/dpdkstrip.py \ build-aux/sodepends.py \ build-aux/soexpand.py \ build-aux/xml2nroff \ $(MAN_FRAGMENTS) \ $(MAN_ROOTS) \ Vagrantfile \ Vagrantfile-FreeBSD \ .mailmap bin_PROGRAMS = sbin_PROGRAMS = bin_SCRIPTS = DIST_HOOKS = dist_man_MANS = dist_pkgdata_DATA = dist_pkgdata_SCRIPTS = dist_sbin_SCRIPTS = dist_scripts_SCRIPTS = dist_scripts_DATA = INSTALL_DATA_LOCAL = UNINSTALL_LOCAL = man_MANS = MAN_FRAGMENTS = MAN_ROOTS = noinst_DATA = noinst_HEADERS = lib_LTLIBRARIES = noinst_LTLIBRARIES = noinst_man_MANS = noinst_PROGRAMS = noinst_SCRIPTS = OVSIDL_BUILT = pkgdata_DATA = sbin_SCRIPTS = scripts_SCRIPTS = completion_SCRIPTS = scripts_DATA = SUFFIXES = check_DATA = check_SCRIPTS = pkgconfig_DATA = FLAKE8_PYFILES = if ENABLE_SPARSE_BY_DEFAULT C ?= 1 endif scriptsdir = $(pkgdatadir)/scripts completiondir = $(sysconfdir)/bash_completion.d pkgconfigdir = $(libdir)/pkgconfig # This ensures that files added to EXTRA_DIST are always distributed, # even if they are inside an Automake if...endif conditional block that is # disabled by some particular "configure" run. For more information, see: # http://article.gmane.org/gmane.comp.sysutils.automake.general/10891 noinst_HEADERS += $(EXTRA_DIST) ro_c = echo '/* -*- mode: c; buffer-read-only: t -*- */' ro_shell = printf '\043 Generated automatically -- do not modify! -*- buffer-read-only: t -*-\n' SUFFIXES += .in .in: $(AM_V_GEN)PYTHONPATH=$$PYTHONPATH$(psep)$(srcdir)/python $(PYTHON) $(srcdir)/build-aux/soexpand.py -I$(srcdir) < $< | \ $(PYTHON) $(srcdir)/build-aux/dpdkstrip.py $(DPDKSTRIP_FLAGS) | \ sed \ -e 's,[@]PKIDIR[@],$(PKIDIR),g' \ -e 's,[@]LOGDIR[@],$(LOGDIR),g' \ -e 's,[@]DBDIR[@],$(DBDIR),g' \ -e 's,[@]PYTHON[@],$(PYTHON),g' \ -e 's,[@]RUNDIR[@],$(RUNDIR),g' \ -e 's,[@]VERSION[@],$(VERSION),g' \ -e 's,[@]localstatedir[@],$(localstatedir),g' \ -e 's,[@]pkgdatadir[@],$(pkgdatadir),g' \ -e 's,[@]sysconfdir[@],$(sysconfdir),g' \ -e 's,[@]bindir[@],$(bindir),g' \ -e 's,[@]sbindir[@],$(sbindir),g' \ -e 's,[@]abs_builddir[@],$(abs_builddir),g' \ -e 's,[@]abs_top_srcdir[@],$(abs_top_srcdir),g' \ > $@.tmp @if head -n 1 $@.tmp | grep '#!' > /dev/null; then \ chmod +x $@.tmp; \ fi $(AM_V_at) mv $@.tmp $@ SUFFIXES += .xml %: %.xml $(AM_V_GEN)$(run_python) $(srcdir)/build-aux/xml2nroff $< > $@.tmp \ -I $(srcdir) \ --version=$(VERSION) \ PKIDIR='$(PKIDIR)' \ LOGDIR='$(LOGDIR)' \ DBDIR='$(DBDIR)' \ PYTHON='$(PYTHON)' \ RUNDIR='$(RUNDIR)' \ VERSION='$(VERSION)' \ localstatedir='$(localstatedir)' \ pkgdatadir='$(pkgdatadir)' \ sysconfdir='$(sysconfdir)' \ bindir='$(bindir)' \ sbindir='$(sbindir)' $(AM_v_at)mv $@.tmp $@ clean-pycov: cd $(srcdir) && rm -f $(PYCOV_CLEAN_FILES) CLEAN_LOCAL += clean-pycov .PHONY: clean-pycov # If we're checked out from a Git repository, make sure that every # file that is in Git is distributed. ALL_LOCAL += dist-hook-git dist-hook-git: distfiles @if test -e $(srcdir)/.git && (git --version) >/dev/null 2>&1; then \ (cd datapath && $(MAKE) distfiles); \ (cat distfiles; sed 's|^|datapath/|' datapath/distfiles) | \ LC_ALL=C sort -u > all-distfiles; \ (cd $(srcdir) && git ls-files) | grep -v '\.gitignore$$' | \ LC_ALL=C sort -u > all-gitfiles; \ LC_ALL=C comm -1 -3 all-distfiles all-gitfiles > missing-distfiles; \ if test -s missing-distfiles; then \ echo "The following files are in git but not the distribution:"; \ cat missing-distfiles; \ exit 1; \ fi; \ if LC_ALL=C grep '\.gitignore$$' all-distfiles; then \ echo "See above for list of files that are distributed but"; \ echo "should not be."; \ exit 1; \ fi \ fi CLEANFILES += all-distfiles all-gitfiles missing-distfiles # The following is based on commands for the Automake "distdir" target. distfiles: Makefile @srcdirstrip=`echo "$(srcdir)" | sed 's/[].[^$$\\*]/\\\\&/g'`; \ topsrcdirstrip=`echo "$(top_srcdir)" | sed 's/[].[^$$\\*]/\\\\&/g'`; \ list='$(DISTFILES)'; \ for file in $$list; do echo $$file; done | \ sed -e "s|^$$srcdirstrip/||;t" \ -e "s|^$$topsrcdirstrip/|$(top_builddir)/|;t" | \ LC_ALL=C sort -u > $@ CLEANFILES += distfiles .PHONY: dist-hook-git # Check that every .c file includes . ALL_LOCAL += config-h-check config-h-check: @cd $(srcdir); \ if test -e .git && (git --version) >/dev/null 2>&1 && \ git --no-pager grep -L '#include ' `git ls-files | grep '\.c$$' | \ grep -vE '^datapath|^lib/sflow|^third-party|^datapath-windows|^python'`; \ then \ echo "See above for list of violations of the rule that"; \ echo "every C source file must #include ."; \ exit 1; \ fi; \ if grep '#include' include/openvswitch/*.h | \ grep -vE '(<.*>)|("openvswitch)|("openflow)'; \ then \ echo "See above for list of violations of the rule that"; \ echo "public openvswitch header file should not include internal library."; \ exit 1; \ fi .PHONY: config-h-check # Check for printf() type modifiers that MSVC doesn't support. ALL_LOCAL += printf-check printf-check: @cd $(srcdir); \ if test -e .git && (git --version) >/dev/null 2>&1 && \ git --no-pager grep -n -E -e '%[-+ #0-9.*]*([ztj]|hh)' --and --not -e 'ovs_scan' `git ls-files | grep '\.[ch]$$' | \ grep -vE '^datapath|^lib/sflow|^third-party'`; \ then \ echo "See above for list of violations of the rule that"; \ echo "'z', 't', 'j', 'hh' printf() type modifiers are"; \ echo "forbidden. See coding-style.rst for replacements."; \ exit 1; \ fi .PHONY: printf-check # Check that certain data structures are always declared "static". ALL_LOCAL += static-check static-check: @if test -e $(srcdir)/.git && (git --version) >/dev/null 2>&1 && \ git --no-pager grep -n -E '^[ ]+(struct vlog_rate_limit|pthread_once_t|struct ovsthread_once).*=' $(srcdir); \ then \ echo "See above for list of violations of the rule that "; \ echo "certain data structures must always be 'static'"; \ exit 1; \ fi .PHONY: static-check # Check that assert.h is not used outside a whitelist of files. ALL_LOCAL += check-assert-h-usage check-assert-h-usage: @if test -e $(srcdir)/.git && (git --version) >/dev/null 2>&1 && \ (cd $(srcdir) && git --no-pager grep -l -E '[<]assert.h[>]') | \ $(EGREP) -v '^lib/(sflow_receiver|vlog).c$$|^tests/'; \ then \ echo "Files listed above unexpectedly #include <""assert.h"">."; \ echo "Please use ovs_assert (from util.h) instead of assert."; \ exit 1; \ fi .PHONY: check-assert-h-usage # Check that LITTLE_ENDIAN and BIG_ENDIAN are not used unless BYTE_ORDER is # also mentioned. ( always defines the former two constants. They # must be compared to BYTE_ORDER to get the machine's correct endianness. But # it is better to use WORDS_BIGENDIAN.) ALL_LOCAL += check-endian check-endian: @if test -e $(srcdir)/.git && (git --version) >/dev/null 2>&1 && \ (cd $(srcdir) && git --no-pager grep -l -E \ -e 'BIG_ENDIAN|LITTLE_ENDIAN' --and --not -e 'BYTE_ORDER' | \ $(EGREP) -v '^datapath/'); \ then \ echo "See above for list of files that misuse LITTLE""_ENDIAN"; \ echo "or BIG""_ENDIAN. Please use WORDS_BIGENDIAN instead."; \ exit 1; \ fi .PHONY: check-endian ALL_LOCAL += check-echo-n check-echo-n: @if test -e $(srcdir)/.git && (git --version) >/dev/null 2>&1 && \ git --no-pager grep -n 'echo'' -n' $(srcdir); \ then \ echo "See above for uses for \"echo"" -n\", which is non-POSIX"; \ echo "and does not work with all shells. Use \"printf\" instead."; \ exit 1; \ fi .PHONY: check-echo-n ALL_LOCAL += thread-safety-check thread-safety-check: @cd $(srcdir); \ if test -e .git && (git --version) >/dev/null 2>&1 && \ grep -n -f build-aux/thread-safety-blacklist \ `git ls-files | grep '\.[ch]$$' \ | $(EGREP) -v '^datapath|^lib/sflow|^third-party'` /dev/null \ | $(EGREP) -v ':[ ]*/?\*'; \ then \ echo "See above for list of calls to functions that are"; \ echo "blacklisted due to thread safety issues"; \ exit 1; \ fi EXTRA_DIST += build-aux/thread-safety-blacklist .PHONY: thread-safety-check # Check that "ip" is used in preference to "ifconfig", because # "ifconfig" is not installed ubiquitously anymore. ALL_LOCAL += check-ifconfig check-ifconfig: @if test -e $(srcdir)/.git && (git --version) >/dev/null 2>&1 && \ (cd $(srcdir) && git --no-pager grep -l -E -e 'ifconfig' | \ $(EGREP) -v 'Makefile.am|ovs-vsctl-bashcomp|openvswitch-custom\.te'); \ then \ echo "See above for list of files that use or reference"; \ echo "'ifconfig'. Please use 'ip' instead."; \ exit 1; \ fi .PHONY: check-ifconfig if HAVE_GROFF ALL_LOCAL += manpage-check manpage-check: $(man_MANS) $(dist_man_MANS) $(noinst_man_MANS) @error=false; \ for manpage in $?; do \ LANG=en_US.UTF-8 groff -w mac -w delim -w escape -w input -w missing -w tab -T utf8 -man -p -z $$manpage >$@.tmp 2>&1; \ if grep warning: $@.tmp; then error=:; fi; \ rm -f $@.tmp; \ done; \ if $$error; then exit 1; else touch $@; fi $(AM_V_GEN) touch -c $@ CLEANFILES += manpage-check endif if HAVE_FLAKE8 ALL_LOCAL += flake8-check # http://flake8.readthedocs.org/en/latest/user/error-codes.html # All warnings explicitly selected or ignored should be listed below. # # E***, W*** -- warnings from pep8 # E121 continuation line under-indented for hanging indent (only from flake8 v2.0) # E123 closing bracket does not match indentation of opening bracket's line # E125 continuation line with same indent as next logical line (only from flake8 v2.0) # E126 continuation line over-indented for hanging indent # E127 continuation line over-indented for visual indent # E128 continuation line under-indented for visual indent # E129 visually indented line with same indent as next logical line # E131 continuation line unaligned for hanging indent # E722 do not use bare except, specify exception instead # W503 line break before binary operator # F*** -- warnings native to flake8 # F811 redefinition of unused from line (only from flake8 v2.0) # D*** -- warnings from flake8-docstrings plugin # H*** -- warnings from flake8 hacking plugin (custom style checks beyond PEP8) # H231 Python 3.x incompatible 'except x,y:' construct # H232 Python 3.x incompatible octal 077 should be written as 0o77 # H233 Python 3.x incompatible use of print operator # H238 old style class declaration, use new style (inherit from `object`) FLAKE8_SELECT = H231,H232,H233,H238 FLAKE8_IGNORE = E121,E123,E125,E126,E127,E128,E129,E131,E722,W503,F811,D,H,I flake8-check: $(FLAKE8_PYFILES) $(FLAKE8_WERROR)$(AM_V_GEN) \ src='$^' && \ flake8 $$src --select=$(FLAKE8_SELECT) $(FLAKE8_FLAGS) && \ flake8 $$src --ignore=$(FLAKE8_IGNORE) $(FLAKE8_FLAGS) && \ touch $@ endif CLEANFILES += flake8-check include $(srcdir)/manpages.mk $(srcdir)/manpages.mk: $(MAN_ROOTS) build-aux/sodepends.py python/build/soutil.py @PYTHONPATH=$$PYTHONPATH$(psep)$(srcdir)/python $(PYTHON) $(srcdir)/build-aux/sodepends.py -I. -I$(srcdir) $(MAN_ROOTS) >$(@F).tmp @if cmp -s $(@F).tmp $@; then \ touch $@; \ rm -f $(@F).tmp; \ else \ mv $(@F).tmp $@; \ fi CLEANFILES += manpage-dep-check if VSTUDIO_DDK ALL_LOCAL += ovsext ovsext: datapath-windows/ovsext.sln $(srcdir)/datapath-windows/include/OvsDpInterface.h MSBuild.exe //nologo //maxcpucount datapath-windows/ovsext.sln /target:Build /property:Configuration="Win8$(VSTUDIO_CONFIG)" /property:Version="$(PACKAGE_VERSION)" MSBuild.exe //nologo //maxcpucount datapath-windows/ovsext.sln /target:Build /property:Configuration="Win8.1$(VSTUDIO_CONFIG)" /property:Version="$(PACKAGE_VERSION)" CLEAN_LOCAL += ovsext_clean ovsext_clean: datapath-windows/ovsext.sln MSBuild.exe //nologo //maxcpucount datapath-windows/ovsext.sln /target:Clean /property:Configuration="Win8$(VSTUDIO_CONFIG)" /property:Version="$(PACKAGE_VERSION)" MSBuild.exe //nologo //maxcpucount datapath-windows/ovsext.sln /target:Clean /property:Configuration="Win8.1$(VSTUDIO_CONFIG)" /property:Version="$(PACKAGE_VERSION)" endif .PHONY: ovsext clang-analyze: clean @which clang scan-build >/dev/null 2>&1 || \ (echo "Unable to find clang/scan-build, Install clang,clang-analyzer packages"; exit 1) @$(MKDIR_P) "$(srcdir)/tests/clang-analyzer-results" @scan-build -o $(srcdir)/tests/clang-analyzer-results --use-cc=$(CC) $(MAKE) .PHONY: clang-analyze dist-hook: $(DIST_HOOKS) all-local: $(ALL_LOCAL) clean-local: $(CLEAN_LOCAL) install-data-local: $(INSTALL_DATA_LOCAL) uninstall-local: $(UNINSTALL_LOCAL) .PHONY: $(DIST_HOOKS) $(CLEAN_LOCAL) $(INSTALL_DATA_LOCAL) $(UNINSTALL_LOCAL) modules_install: if LINUX_ENABLED cd datapath/linux && $(MAKE) modules_install endif dist-docs: VERSION=$(VERSION) MAKE='$(MAKE)' $(srcdir)/build-aux/dist-docs $(srcdir) $(docs) .PHONY: dist-docs include Documentation/automake.mk include m4/automake.mk include lib/automake.mk include ofproto/automake.mk include utilities/automake.mk include tests/automake.mk include include/automake.mk include third-party/automake.mk include debian/automake.mk include vswitchd/automake.mk include ovsdb/automake.mk include rhel/automake.mk include xenserver/automake.mk include python/automake.mk include tutorial/automake.mk include vtep/automake.mk include datapath-windows/automake.mk include datapath-windows/include/automake.mk include windows/automake.mk include ovn/automake.mk include selinux/automake.mk include build-aux/automake.mk ovs-2.9.0/NEWS000066400000000000000000001763341324262074100130470ustar00rootroot00000000000000v2.9.0 - 19 Feb 2018 -------------------- - NSH implementation now conforms to latest draft (draft-ietf-sfc-nsh-28). * Add ttl field. * Add a new action dec_nsh_ttl. * Enable NSH support in kernel datapath. - OVSDB: * New high-level documentation in ovsdb(7). * New file format documentation for developers in ovsdb(5). * Protocol documentation moved from ovsdb-server(1) to ovsdb-server(7). * ovsdb-client: New "get-schema-cksum" and "query" commands. * ovsdb-client: New "backup" and "restore" commands. * ovsdb-tool: New "db-name" and "schema-name" commands. - ovs-vsctl and other commands that display data in tables now support a --max-column-width option to limit column width. - No longer slow-path traffic that sends to a controller. Applications, such as OVN ACL logging, want to send a copy of a packet to a controller while leaving the actual packet forwarding in the datapath. - OVN: * The "requested-chassis" option for a logical switch port now accepts a chassis "hostname" in addition to a chassis "name". * IPv6 - Added support to send IPv6 Router Advertisement packets in response to the IPv6 Router Solicitation packets from the VIF ports. - Added support to generate Neighbor Solicitation packets using the OVN action 'nd_ns' to resolve unknown next hop MAC addresses for the IPv6 packets. * ovn-ctl: New commands run_nb_ovsdb and run_sb_ovsdb. - OpenFlow: * ct_clear action is now backed by kernel datapath. Support is probed for when OVS starts. - Linux kernel 4.13 * Add support for compiling OVS with the latest Linux 4.13 kernel - ovs-dpctl and related ovs-appctl commands: * "flush-conntrack" now accept a 5-tuple to delete a specific connection tracking entry. * New "ct-set-maxconns", "ct-get-maxconns", and "ct-get-nconns" commands for userspace datapath. - No longer send packets to the Linux TAP device if it's DOWN unless it is in another networking namespace. - DPDK: * Add support for DPDK v17.11 * Add support for vHost IOMMU * New debug appctl command 'netdev-dpdk/get-mempool-info'. * All the netdev-dpdk appctl commands described in ovs-vswitchd man page. * Custom statistics: - DPDK physical ports now return custom set of "dropped", "error" and "management" statistics. - ovs-ofctl dump-ports command now prints new of set custom statistics if available (for OpenFlow 1.4+). * New appctl command 'dpif-netdev/pmd-rxq-rebalance' to rebalance rxq to pmd assignments. * Add rxq utilization of pmd to appctl 'dpif-netdev/pmd-rxq-show'. * Add support for vHost dequeue zero copy (experimental) - Userspace datapath: * Output packet batching support. - vswitchd: * Datapath IDs may now be specified as 0x1 (etc.) instead of 16 digits. * Configuring a controller, or unconfiguring all controllers, now deletes all groups and meters (as well as all flows). - New --enable-sparse configure option enables "sparse" checking by default. - Added additional information to vhost-user status. v2.8.0 - 31 Aug 2017 -------------------- - ovs-ofctl: * ovs-ofctl can now accept and display port names in place of numbers. By default it always accepts names and in interactive use it displays them; use --names or --no-names to override. See ovs-ofctl(8) for details. * "ovs-ofctl dump-flows" now accepts --no-stats to omit flow statistics. - New ovs-dpctl command "ct-stats-show" to show connection tracking stats. - Tunnels: * Added support to set packet mark for tunnel endpoint using `egress_pkt_mark` OVSDB option. * When using Linux kernel datapath tunnels may be created using rtnetlink. This will allow us to take advantage of new tunnel features without having to make changes to the vport modules. - EMC insertion probability is reduced to 1% and is configurable via the new 'other_config:emc-insert-inv-prob' option. - DPDK: * DPDK log messages redirected to OVS logging subsystem. Log level can be changed in a usual OVS way using 'ovs-appctl vlog' commands for 'dpdk' module. Lower bound still can be configured via extra arguments for DPDK EAL. * dpdkvhostuser ports are marked as deprecated. They will be removed in an upcoming release. * Support for DPDK v17.05.1. - IPFIX now provides additional counters: * Total counters since metering process startup. * Per-flow TCP flag counters. * Multicast, broadcast, and unicast counters. - New support for multiple VLANs (802.1ad or "QinQ"), including a new "dot1q-tunnel" port VLAN mode. - In ovn-vsctl and vtep-ctl, record UUIDs in commands may now be abbreviated to 4 hex digits. - Userspace Datapath: * Added NAT support for userspace datapath. * Added FTP and TFTP support with NAT for userspace datapath. * Experimental NSH (Network Service Header) support in userspace datapath. - OVN: * New built-in DNS support. * IPAM for IPv4 can now exclude user-defined addresses from assignment. * IPAM can now assign IPv6 addresses. * Make the DHCPv4 router setting optional. * Gratuitous ARP for NAT addresses on a distributed logical router. * Allow ovn-controller SSL configuration to be obtained from vswitchd database. * ovn-trace now has basic support for tracing distributed firewalls. * In ovn-nbctl and ovn-sbctl, record UUIDs in commands may now be abbreviated to 4 hex digits. * "ovn-sbctl lflow-list" can now print OpenFlow flows that correspond to logical flows. * Now uses OVSDB RBAC support to reduce impact of compromised hypervisors. * Multiple chassis may now be specified for L3 gateways. When more than one chassis is specified, OVN will manage high availability for that gateway. * Add support for ACL logging. * ovn-northd now has native support for active-standby high availability. * Add support for QoS bandwidth limt with DPDK. - Tracing with ofproto/trace now traces through recirculation. - OVSDB: * New support for role-based access control (see ovsdb-server(1)). - New commands 'stp/show' and 'rstp/show' (see ovs-vswitchd(8)). - OpenFlow: * All features required by OpenFlow 1.4 are now implemented, so ovs-vswitchd now enables OpenFlow 1.4 by default (in addition to OpenFlow 1.0 to 1.3). * Increased support for OpenFlow 1.6 (draft). * Bundles now support hashing by just nw_src or nw_dst. * The "learn" action now supports a "limit" option (see ovs-ofctl(8)). * The port status bit OFPPS_LIVE now reflects link aliveness. * OpenFlow 1.5 packet-out is now supported. * Support for OpenFlow 1.5 field packet_type and packet-type-aware pipeline (PTAP). * Added generic encap and decap actions (EXT-382). First supported use case is encap/decap for Ethernet. * Added NSH (Network Service Header) support in userspace Used generic encap and decap actions to implement encapsulation and decapsulation of NSH header. IETF NSH draft - https://datatracker.ietf.org/doc/draft-ietf-sfc-nsh/ * Conntrack state is only available to the processing path that follows the "recirc_table" argument of the ct() action. Starting in OVS 2.8, this state is now cleared for the current processing path whenever ct() is called. - Fedora Packaging: * OVN services are no longer restarted automatically after upgrade. * ovs-vswitchd and ovsdb-server run as non-root users by default. - Add --cleanup option to command 'ovs-appctl exit' (see ovs-vswitchd(8)). - L3 tunneling: * Use new tunnel port option "packet_type" to configure L2 vs. L3. * In conjunction with PTAP tunnel ports can handle a mix of L2 and L3 payload. * New vxlan tunnel extension "gpe" to support VXLAN-GPE tunnels. * New support for non-Ethernet (L3) payloads in GRE and VXLAN-GPE. - The BFD detection multiplier is now user-configurable. - Add experimental support for hardware offloading * HW offloading is disabled by default. * HW offloading is done through the TC interface. - IPv6 link local addresses are now supported on Linux. Use % to designate the scope device. v2.7.0 - 21 Feb 2017 --------------------- - Utilities and daemons that support SSL now allow protocols and ciphers to be configured with --ssl-protocols and --ssl-ciphers. - OVN: * QoS is now implemented via egress shaping rather than ingress policing. * DSCP marking is now supported, via the new northbound QoS table. * IPAM now supports fixed MAC addresses. * Support for source IP address based routing. * ovn-trace: - New --ovs option to also print OpenFlow flows. - put_dhcp_opts and put_dhcp_optsv6 actions may now be traced. * Support for managing SSL and remote connection configuration in northbound and southbound databases. * TCP connections to northbound and southbound databases are no longer enabled by default and must be explicitly configured. See documentation for ovn-sbctl/ovn-nbctl "set-connection" command or the ovn-ctl "--db-sb-create-insecure-remote" and "--db-nb-create-insecure-remote" command-line options for information regarding remote connection configuration. * New appctl "inject-pkt" command in ovn-controller that allows packets to be injected into the connected OVS instance. * Distributed logical routers may now be connected directly to logical switches with localnet ports, by specifying a "redirect-chassis" on the distributed gateway port of the logical router. NAT rules may be specified directly on the distributed logical router, and are handled either centrally on the "redirect-chassis", or in many cases are handled locally on the hypervisor where the corresponding logical port resides. Gratuitous ARP for NAT addresses on a distributed logical router is not yet supported, but will be added in a future version. - Fixed regression in table stats maintenance introduced in OVS 2.3.0, wherein the number of OpenFlow table hits and misses was not accurate. - OpenFlow: * OFPT_PACKET_OUT messages are now supported in bundles. * A new "selection_method=dp_hash" type for OpenFlow select group bucket selection that uses the datapath computed 5-tuple hash without making datapath flows match the 5-tuple fields, which is useful for more efficient load balancing, for example. This uses the Netronome extension to OpenFlow 1.5+ that allows control over the OpenFlow select groups selection method. See "selection_method" and related options in ovs-ofctl(8) for details. * The "sample" action now supports "ingress" and "egress" options. * The "ct" action now supports the TFTP ALG where support is available. * New actions "clone" and "ct_clear". * The "meter" action is now supported in the userspace datapath. - ovs-ofctl: * 'bundle' command now supports packet-out messages. * New syntax for 'ovs-ofctl packet-out' command, which uses the same string parser as the 'bundle' command. The old 'packet-out' syntax is deprecated and will be removed in a later OVS release. * New unixctl "ofctl/packet-out" command, which can be used to instruct a flow monitor to issue OpenFlow packet-out messages. - ovsdb-server: * Remote connections can now be made read-only (see ovsdb-server(1)). - Tunnels: * TLV mappings for protocols such as Geneve are now segregated on a per-OpenFlow bridge basis rather than globally. (The interface has not changed.) * Removed support for IPsec tunnels. - DPDK: * New option 'n_rxq_desc' and 'n_txq_desc' fields for DPDK interfaces which set the number of rx and tx descriptors to use for the given port. * Support for DPDK v16.11. * Support for rx checksum offload. Refer DPDK HOWTO for details. * Port Hotplug is now supported. * DPDK physical ports can now have arbitrary names. The PCI address of the device must be set using the 'dpdk-devargs' option. Compatibility with the old dpdk naming scheme is broken, and as such a device will not be available for use until a valid dpdk-devargs is specified. * Virtual DPDK Poll Mode Driver (vdev PMD) support. * Removed experimental tag. - Fedora packaging: * A package upgrade does not automatically restart OVS service. - ovs-vswitchd/ovs-vsctl: * Ports now have a "protected" flag. Protected ports can not forward frames to other protected ports. Unprotected ports can receive and forward frames to protected and other unprotected ports. - ovs-vsctl, ovn-nbctl, ovn-sbctl, vtep-ctl: * Database commands now accept integer ranges, e.g. "set port eth0 trunks=1-10" to enable trunking VLANs 1 to 10. v2.6.0 - 27 Sep 2016 --------------------- - First supported release of OVN. See ovn-architecture(7) for more details. - ovsdb-server: * New "monitor_cond" "monitor_cond_update" and "update2" extensions to RFC 7047. - OpenFlow: * OpenFlow 1.3+ bundles now expire after 10 seconds since the last time the bundle was either opened, modified, or closed. * OpenFlow 1.3 Extension 230, adding OpenFlow Bundles support, is now implemented. * OpenFlow 1.3+ bundles are now supported for group mods as well as flow mods and port mods. Both 'atomic' and 'ordered' bundle flags are supported for group mods as well as flow mods. * Internal OpenFlow rule representation for load and set-field actions is now much more memory efficient. For a complex flow table this can reduce rule memory consumption by 40%. * Bundles are now much more memory efficient than in OVS 2.5. Together with memory efficiency improvements in OpenFlow rule representation, the peak OVS resident memory use during a bundle commit for large complex set of flow mods can be only 25% of that in OVS 2.5 (4x lower). * OpenFlow 1.1+ OFPT_QUEUE_GET_CONFIG_REQUEST now supports OFPP_ANY. * OpenFlow 1.4+ OFPMP_QUEUE_DESC is now supported. * OpenFlow 1.4+ OFPT_TABLE_STATUS is now supported. * New property-based packet-in message format NXT_PACKET_IN2 with support for arbitrary user-provided data and for serializing flow table traversal into a continuation for later resumption. * New extension message NXT_SET_ASYNC_CONFIG2 to allow OpenFlow 1.4-like control over asynchronous messages in earlier versions of OpenFlow. * New OpenFlow extension NXM_NX_MPLS_TTL to provide access to MPLS TTL. * New output option, output(port=N,max_len=M), to allow truncating a packet to size M bytes when outputting to port N. * New command OFPGC_ADD_OR_MOD for OFPT_GROUP_MOD message that adds a new group or modifies an existing groups * The optional OpenFlow packet buffering feature is deprecated in this release, and will be removed in the next OVS release (2.7). After the change OVS always sends the 'buffer_id' as 0xffffffff in packet-in messages and will send an error response if any other value of this field is included in packet-out and flow mod sent by a controller. Controllers are already expected to work properly in cases where the switch can not buffer packets, so this change should not affect existing users. * New OpenFlow extension NXT_CT_FLUSH_ZONE to flush conntrack zones. - Improved OpenFlow version compatibility for actions: * New OpenFlow extension to support the "group" action in OpenFlow 1.0. * OpenFlow 1.0 "enqueue" action now properly translated to OpenFlow 1.1+. * OpenFlow 1.1 "mod_nw_ecn" and OpenFlow 1.1+ "mod_nw_ttl" actions now properly translated to OpenFlow 1.0. - ovs-ofctl: * queue-get-config command now allows a queue ID to be specified. * '--bundle' option can now be used with OpenFlow 1.3 and with group mods. * New "bundle" command allows executing a mixture of flow and group mods as a single atomic transaction. * New option "--color" to produce colorized output for some commands. * New option '--may-create' to use OFPGC_ADD_OR_MOD in mod-group command. - IPFIX: * New "sampling_port" option for "sample" action to allow sampling ingress and egress tunnel metadata with IPFIX. * New ovs-ofctl commands "dump-ipfix-bridge" and "dump-ipfix-flow" to dump bridge IPFIX statistics and flow based IPFIX statistics. * New setting other-config:virtual_obs_id to add an arbitrary string to IPFIX records. - Linux: * OVS Linux datapath now implements Conntrack NAT action with all supported Linux kernels. * Support for truncate action. * New QoS type "linux-noop" that prevents Open vSwitch from trying to manage QoS for a given port (useful when other software manages QoS). - DPDK: * New option "n_rxq" for PMD interfaces. Old 'other_config:n-dpdk-rxqs' is no longer supported. Not supported by vHost interfaces. For them number of rx and tx queues is applied from connected virtio device. * New 'other_config:pmd-rxq-affinity' field for PMD interfaces, that allows to pin port's rx queues to desired cores. * New appctl command 'dpif-netdev/pmd-rxq-show' to check the port/rxq assignment. * Type of log messages from PMD threads changed from INFO to DBG. * QoS functionality with sample egress-policer implementation. * The mechanism for configuring DPDK has changed to use database * Sensible defaults have been introduced for many of the required configuration options * DB entries have been added for many of the DPDK EAL command line arguments. Additional arguments can be passed via the dpdk-extra entry. * Add ingress policing functionality. * PMD threads servicing vHost User ports can now come from the NUMA node that device memory is located on if CONFIG_RTE_LIBRTE_VHOST_NUMA is enabled in DPDK. * Basic connection tracking for the userspace datapath (no ALG, fragmentation or NAT support yet) * Support for DPDK 16.07 * Optional support for DPDK pdump enabled. * Jumbo frame support * Remove dpdkvhostcuse port type. * OVS client mode for vHost and vHost reconnect (Requires QEMU 2.7) * 'dpdkvhostuserclient' port type. - Increase number of registers to 16. - ovs-benchmark: This utility has been removed due to lack of use and bitrot. - ovs-appctl: * New "vlog/close" command. - ovs-ctl: * Added the ability to selectively start the forwarding and database functions (ovs-vswitchd and ovsdb-server, respectively). - ovsdb-server: * Remove max number of sessions limit, to enable connection scaling testing. - python: * Added support for Python 3.4+ in addition to existing support for 2.7+. - SELinux: * Introduced SELinux policy package. - Datapath Linux kernel compatibility. * Dropped support for kernel older than 3.10. * Removed VLAN splinters feature. * Datapath supports kernel upto 4.7. - Tunnels: * Flow based tunnel match and action can be used for IPv6 address using tun_ipv6_src, tun_ipv6_dst fields. * Added support for IPv6 tunnels, for details checkout FAQ. * Deprecated support for IPsec tunnels ports. - A wrapper script, 'ovs-tcpdump', to easily port-mirror an OVS port and watch with tcpdump - Introduce --no-self-confinement flag that allows daemons to work with sockets outside their run directory. - ovs-pki: Changed message digest algorithm from SHA-1 to SHA-512 because SHA-1 is no longer secure and some operating systems have started to disable it in OpenSSL. - Add 'mtu_request' column to the Interface table. It can be used to configure the MTU of the ports. Known issues: - Using openvswitch module in conjunction with upstream Linux tunnels: * When using the openvswitch module distributed with OVS against kernel versions 4.4 to 4.6, the openvswitch module cannot be loaded or used at the same time as "ip_gre". - Conntrack FTP ALGs: When using the openvswitch module distributed with OVS, particular Linux distribution kernels versions may provide diminished functionality. This typically affects active FTP data connections when using "actions=ct(alg=ftp),..." in flow tables. Specifically: * Centos 7.1 kernels (3.10.0-2xx) kernels are unable to correctly set up expectations for FTP data connections in multiple zones, eg "actions=ct(zone=1,alg=ftp),ct(zone=2,alg=ftp),...". Executing the "ct" action for subsequent data connections may fail to determine that the data connection is "related" to an existing connection. * Centos 7.2 kernels (3.10.0-3xx) kernels may not establish FTP ALG state correctly for NATed connections. As a result, flows that perform NAT, eg "actions=ct(nat,ftp=alg,table=1),..." may fail to NAT the packet, and will populate the "ct_state=inv" bit in the flow. v2.5.0 - 26 Feb 2016 --------------------- - Dropped support for Python older than version 2.7. As a consequence, using Open vSwitch 2.5 or later on XenServer 6.5 or earlier (which have Python 2.4) requires first installing Python 2.7. - OpenFlow: * Group chaining (where one OpenFlow group triggers another) is now supported. * OpenFlow 1.4+ "importance" is now considered for flow eviction. * OpenFlow 1.4+ OFPTC_EVICTION is now implemented. * OpenFlow 1.4+ OFPTC_VACANCY_EVENTS is now implemented. * OpenFlow 1.4+ OFPMP_TABLE_DESC is now implemented. * Allow modifying the ICMPv4/ICMPv6 type and code fields. * OpenFlow 1.4+ OFPT_SET_ASYNC_CONFIG and OFPT_GET_ASYNC_CONFIG are now implemented. - ovs-ofctl: * New "out_group" keyword for OpenFlow 1.1+ matching on output group. - Tunnels: * Geneve tunnels can now match and set options and the OAM bit. * The nonstandard GRE64 tunnel extension has been dropped. - Support Multicast Listener Discovery (MLDv1 and MLDv2). - Add 'symmetric_l3l4' and 'symmetric_l3l4+udp' hash functions. - sFlow agent now reports tunnel and MPLS structures. - New 'check-system-userspace', 'check-kmod' and 'check-kernel' Makefile targets to run a new system testsuite. These tests can be run inside a Vagrant box. See INSTALL.md for details - Mark --syslog-target argument as deprecated. It will be removed in the next OVS release. - Added --user option to all daemons - Add support for connection tracking through the new "ct" action and "ct_state"/"ct_zone"/"ct_mark"/"ct_label" match fields. Only available on Linux kernels with the connection tracking module loaded. - Add experimental version of OVN. OVN, the Open Virtual Network, is a system to support virtual network abstraction. OVN complements the existing capabilities of OVS to add native support for virtual network abstractions, such as virtual L2 and L3 overlays and security groups. - RHEL packaging: * DPDK ports may now be created via network scripts (see README.RHEL). - DPDK: * Requires DPDK 2.2 * Added multiqueue support to vhost-user * Note: QEMU 2.5+ required for multiqueue support v2.4.0 - 20 Aug 2015 --------------------- - Flow table modifications are now atomic, meaning that each packet now sees a coherent version of the OpenFlow pipeline. For example, if a controller removes all flows with a single OpenFlow "flow_mod", no packet sees an intermediate version of the OpenFlow pipeline where only some of the flows have been deleted. - Added support for SFQ, FQ_CoDel and CoDel qdiscs. - Add bash command-line completion support for ovs-vsctl Please check utilities/ovs-command-compgen.INSTALL.md for how to use. - The MAC learning feature now includes per-port fairness to mitigate MAC flooding attacks. - New support for a "conjunctive match" OpenFlow extension, which allows constructing OpenFlow matches of the form "field1 in {a,b,c...} AND field2 in {d,e,f...}" and generalizations. For details, see documentation for the "conjunction" action in ovs-ofctl(8). - Add bash command-line completion support for ovs-appctl/ovs-dpctl/ ovs-ofctl/ovsdb-tool commands. Please check utilities/ovs-command-compgen.INSTALL.md for how to use. - The "learn" action supports a new flag "delete_learned" that causes the learned flows to be deleted when the flow with the "learn" action is deleted. - Basic support for the Geneve tunneling protocol. It is not yet possible to generate or match options. This is planned for a future release. The protocol is documented at http://tools.ietf.org/html/draft-gross-geneve-00 - The OVS database now reports controller rate limiting statistics. - sflow now exports information about LACP-based bonds, port names, and OpenFlow port numbers, as well as datapath performance counters. - ovs-dpctl functionality is now available for datapaths integrated into ovs-vswitchd, via ovs-appctl. Some existing ovs-appctl commands are now redundant and will be removed in a future release. See ovs-vswitchd(8) for details. - OpenFlow: * OpenFlow 1.4 bundles are now supported for flow mods and port mods. For flow mods, both 'atomic' and 'ordered' bundle flags are trivially supported, as all bundled messages are executed in the order they were added and all flow table modifications are now atomic to the datapath. Port mods may not appear in atomic bundles, as port status modifications are not atomic. * IPv6 flow label and neighbor discovery fields are now modifiable. * OpenFlow 1.5 extended registers are now supported. * The OpenFlow 1.5 actset_output field is now supported. * OpenFlow 1.5 Copy-Field action is now supported. * OpenFlow 1.5 masked Set-Field action is now supported. * OpenFlow 1.3+ table features requests are now supported (read-only). * Nicira extension "move" actions may now be included in action sets. * "resubmit" actions may now be included in action sets. The resubmit is executed last, and only if the action set has no "output" or "group" action. * OpenFlow 1.4+ flow "importance" is now maintained in the flow table. * A new Netronome extension to OpenFlow 1.5+ allows control over the fields hashed for OpenFlow select groups. See "selection_method" and related options in ovs-ofctl(8) for details. - ovs-ofctl has a new '--bundle' option that makes the flow mod commands ('add-flow', 'add-flows', 'mod-flows', 'del-flows', and 'replace-flows') use an OpenFlow 1.4 bundle to operate the modifications as a single atomic transaction. If any of the flow mods in a transaction fail, none of them are executed. All flow mods in a bundle appear to datapath lookups simultaneously. - ovs-ofctl 'add-flow' and 'add-flows' commands now accept arbitrary flow mods as an input by allowing the flow specification to start with an explicit 'add', 'modify', 'modify_strict', 'delete', or 'delete_strict' keyword. A missing keyword is treated as 'add', so this is fully backwards compatible. With the new '--bundle' option all the flow mods are executed as a single atomic transaction using an OpenFlow 1.4 bundle. - ovs-pki: Changed message digest algorithm from MD5 to SHA-1 because MD5 is no longer secure and some operating systems have started to disable it in OpenSSL. - ovsdb-server: New OVSDB protocol extension allows inequality tests on "optional scalar" columns. See ovsdb-server(1) for details. - ovs-vsctl now permits immutable columns in a new row to be modified in the same transaction that creates the row. - test-controller has been renamed ovs-testcontroller at request of users who find it useful for testing basic OpenFlow setups. It is still not a necessary or desirable part of most Open vSwitch deployments. - Support for travis-ci.org based continuous integration builds has been added. Build failures are reported to build@openvswitch.org. See INSTALL.md file for additional details. - Support for the Rapid Spanning Tree Protocol (IEEE 802.1D-2004). The implementation has been tested successfully against the Ixia Automated Network Validation Library (ANVL). - Stats are no longer updated on fake bond interface. - Keep active bond slave selection across OVS restart. - A simple wrapper script, 'ovs-docker', to integrate OVS with Docker containers. If and when there is a native integration of Open vSwitch with Docker, the wrapper script will be retired. - Added support for DPDK Tunneling. VXLAN, GRE, and Geneve are supported protocols. This is generic tunneling mechanism for userspace datapath. - Support for multicast snooping (IGMPv1, IGMPv2 and IGMPv3) - Support for Linux kernels up to 4.0.x - The documentation now use the term 'destination' to mean one of syslog, console or file for vlog logging instead of the previously used term 'facility'. - Support for VXLAN Group Policy extension - Initial support for the IETF Auto-Attach SPBM draft standard. This contains rudimentary support for the LLDP protocol as needed for Auto-Attach. - The default OpenFlow and OVSDB ports are now the IANA-assigned numbers. OpenFlow is 6653 and OVSDB is 6640. - Support for DPDK vHost. - Support for outer UDP checksums in Geneve and VXLAN. - The kernel vports with dependencies are no longer part of the overall openvswitch.ko but built and loaded automatically as individual kernel modules (vport-*.ko). - Support for STT tunneling. - ovs-sim: New developer tool for simulating multiple OVS instances. See ovs-sim(1) for more information. - Support to configure method (--syslog-method argument) that determines how daemons will talk with syslog. - Support for "ovs-appctl vlog/list-pattern" command that lets to query logging message format for each destination. v2.3.0 - 14 Aug 2014 --------------------- - OpenFlow 1.1, 1.2, and 1.3 are now enabled by default in ovs-vswitchd. - Linux kernel datapath now has an exact match cache optimizing the flow matching process. - Datapath flows now have partially wildcarded tranport port field matches. This reduces userspace upcalls, but increases the number of different masks in the datapath. The kernel datapath exact match cache removes the overhead of matching the incoming packets with the larger number of masks, but when paired with an older kernel module, some workloads may perform worse with the new userspace. - Compatibility with autoconf 2.63 (previously >=2.64) v2.2.0 - Internal Release --------------------- - Internal ports are no longer brought up by default, because it should be an administrator task to bring up devices as they are configured properly. - ovs-vsctl now reports when ovs-vswitchd fails to create a new port or bridge. - Port creation and configuration errors are now stored in a new error column of the Interface table and included in 'ovs-vsctl show'. - The "ovsdbmonitor" graphical tool has been removed, because it was poorly maintained and not widely used. - New "check-ryu" Makefile target for running Ryu tests for OpenFlow controllers against Open vSwitch. See INSTALL.md for details. - Added IPFIX support for SCTP flows and templates for ICMPv4/v6 flows. - Upon the receipt of a SIGHUP signal, ovs-vswitchd no longer reopens its log file (it will terminate instead). Please use 'ovs-appctl vlog/reopen' instead. - Support for Linux kernels up to 3.14. From Kernel 3.12 onwards OVS uses tunnel API for GRE and VXLAN. - Added DPDK support. - Added support for custom vlog patterns in Python v2.1.0 - 19 Mar 2014 --------------------- - Address prefix tracking support for flow tables. New columns "prefixes" in OVS-DB table "Flow_Table" controls which packet header fields are used for address prefix tracking. Prefix tracking allows the classifier to skip rules with longer than necessary prefixes, resulting in better wildcarding for datapath flows. Default configuration is to not use any fields for prefix tracking. However, if any flow tables contain both exact matches and masked matches for IP address fields, OVS performance may be increased by using this feature. * As of now, the fields for which prefix lookup can be enabled are: 'tun_id', 'tun_src', 'tun_dst', 'nw_src', 'nw_dst' (or aliases 'ip_src' and 'ip_dst'), 'ipv6_src', and 'ipv6_dst'. (Using this feature for 'tun_id' would only make sense if the tunnel IDs have prefix structure similar to IP addresses.) * There is a maximum number of fields that can be enabled for any one flow table. Currently this limit is 3. * Examples: $ ovs-vsctl set Bridge br0 flow_tables:0=@N1 -- \ --id=@N1 create Flow_Table name=table0 $ ovs-vsctl set Bridge br0 flow_tables:1=@N1 -- \ --id=@N1 create Flow_Table name=table1 $ ovs-vsctl set Flow_Table table0 prefixes=ip_dst,ip_src $ ovs-vsctl set Flow_Table table1 prefixes=[] - TCP flags matching: OVS now supports matching of TCP flags. This has an adverse performance impact when using OVS userspace 1.10 or older (no megaflows support) together with the new OVS kernel module. It is recommended that the kernel and userspace modules both are upgraded at the same time. - The default OpenFlow and OVSDB ports will change to IANA-assigned numbers in a future release. Consider updating your installations to specify port numbers instead of using the defaults. - OpenFlow: * The OpenFlow 1.1+ "Write-Actions" instruction is now supported. * OVS limits the OpenFlow port numbers it assigns to port 32767 and below, leaving port numbers above that range free for assignment by the controller. * ovs-vswitchd now honors changes to the "ofport_request" column in the Interface table by changing the port's OpenFlow port number. * The Open vSwitch software switch now supports OpenFlow groups. - ovs-vswitchd.conf.db.5 man page will contain graphviz/dot diagram only if graphviz package was installed at the build time. - Support for Linux kernels up to 3.11 - ovs-dpctl: The "show" command also displays mega flow mask stats. - ovs-ofctl: * New command "ofp-parse-pcap" to dump OpenFlow from PCAP files. - ovs-controller has been renamed test-controller. It is no longer packaged or installed by default, because too many users assumed incorrectly that ovs-controller was a necessary or desirable part of an Open vSwitch deployment. - Added vlog option to export to a UDP syslog sink. - ovsdb-client: * The "monitor" command can now monitor all tables in a database, instead of being limited to a single table. - The flow-eviction-threshold has been replaced by the flow-limit which is a hard limit on the number of flows in the datapath. It defaults to 200,000 flows. OVS automatically adjusts this number depending on network conditions. - Added IPv6 support for active and passive socket communications. v2.0.0 - 15 Oct 2013 --------------------- - The ovs-vswitchd process is no longer single-threaded. Multiple threads are now used to handle flow set up and asynchronous logging. - OpenFlow: * Experimental support for OpenFlow 1.1 (in addition to 1.2 and 1.3, which had experimental support in 1.10). * Experimental protocol support for OpenFlow 1.1+ groups. This does not yet include an implementation in the Open vSwitch software switch. * Experimental protocol support for OpenFlow 1.2+ meters. This does not yet include an implementation in the Open vSwitch software switch. * New support for matching outer source and destination IP address of tunneled packets, for tunnel ports configured with the newly added "remote_ip=flow" and "local_ip=flow" options. * Support for matching on metadata 'pkt_mark' for interacting with other system components. On Linux this corresponds to the skb mark. * Support matching, rewriting SCTP ports - The Interface table in the database has a new "ifindex" column to report the interface's OS-assigned ifindex. - New "check-oftest" Makefile target for running OFTest against Open vSwitch. See README-OFTest for details. - The flow eviction threshold has been moved to the Open_vSwitch table. - Database names are now mandatory when specifying ovsdb-server options through database paths (e.g. Private key option with the database name should look like "--private-key=db:Open_vSwitch,SSL,private_key"). - Added ovs-dev.py, a utility script helpful for Open vSwitch developers. - Support for Linux kernels up to 3.10 - ovs-ofctl: * New "ofp-parse" for printing OpenFlow messages read from a file. * New commands for OpenFlow 1.1+ groups. - Added configurable flow caching support to IPFIX exporter. - Dropped support for Linux pre-2.6.32. - Log file timestamps and ovsdb commit timestamps are now reported with millisecond resolution. (Previous versions only reported whole seconds.) v1.11.0 - 28 Aug 2013 --------------------- - Support for megaflows, which allows wildcarding in the kernel (and any dpif implementation that supports wildcards). Depending on the flow table and switch configuration, flow set up rates are close to the Linux bridge. - The "tutorial" directory contains a new tutorial for some advanced Open vSwitch features. - Stable bond mode has been removed. - The autopath action has been removed. - New support for the data encapsulation format of the LISP tunnel protocol (RFC 6830). An external control plane or manual flow setup is required for EID-to-RLOC mapping. - OpenFlow: * The "dec_mpls_ttl" and "set_mpls_ttl" actions from OpenFlow 1.1 and later are now implemented. * New "stack" extension for use in actions, to push and pop from NXM fields. * The "load" and "set_field" actions can now modify the "in_port". (This allows one to enable output to a flow's input port by setting the in_port to some unused value, such as OFPP_NONE.) - ovs-dpctl: * New debugging commands "add-flow", "mod-flow", "del-flow". * "dump-flows" now has a -m option to increase output verbosity. - In dpif-based bridges, cache action translations, which can improve flow set up performance by 80% with a complicated flow table. - New syslog format, prefixed with "ovs|", to be easier to filter. - RHEL: Removes the default firewall rule that allowed GRE traffic to pass through. Any users that relied on this automatic firewall hole will have to manually configure it. The ovs-ctl(8) manpage documents the "enable-protocol" command that can be used as an alternative. - New CFM demand mode which uses data traffic to indicate interface liveness. v1.10.0 - 01 May 2013 --------------------- - Bridge compatibility support has been removed. Any uses that rely on ovs-brcompatd will have to stick with Open vSwitch 1.9.x or adapt to native Open vSwitch support (e.g. use ovs-vsctl instead of brctl). - The maximum size of the MAC learning table is now configurable. - With the Linux datapath, packets for new flows are now queued separately on a per-port basis, so it should no longer be possible for a large number of new flows arriving on one port to prevent new flows from being processed on other ports. - ovs-vsctl: * Previously ovs-vsctl would retry connecting to the database forever, causing it to hang if ovsdb-server was not running. Now, ovs-vsctl only tries once by default (use --retry to try forever). This change means that you may want to remove uses of --timeout to avoid hangs in ovs-vsctl calls. * Many "ovs-vsctl" database commands now accept an --if-exists option. Please refer to the ovs-vsctl manpage for details. - OpenFlow: - Experimental support for newer versions of OpenFlow. See the "What versions of OpenFlow does Open vSwitch support?" question in the FAQ for more details. - The OpenFlow "dp_desc" may now be configured by setting the value of other-config:dp-desc in the Bridge table. - It is possible to request the OpenFlow port number with the "ofport_request" column in the Interface table. - The NXM flow_removed message now reports the OpenFlow table ID from which the flow was removed. - Tunneling: - New support for the VXLAN tunnel protocol (see the IETF draft here: http://tools.ietf.org/html/draft-mahalingam-dutt-dcops-vxlan-03). - Tunneling requires the version of the kernel module paired with Open vSwitch 1.9.0 or later. - Inheritance of the Don't Fragment bit in IP tunnels (df_inherit) is no longer supported. - Path MTU discovery is no longer supported. - CAPWAP tunneling support removed. - Tunnels with multicast destination ports are no longer supported. - ovs-dpctl: - The "dump-flows" and "del-flows" no longer require an argument if only one datapath exists. - ovs-appctl: - New "vlog/disable-rate-limit" and "vlog/enable-rate-limit" commands available allow control over logging rate limits. - New "dpif/dump-dps", "dpif/show", and "dpif/dump-flows" command that mimic the equivalent ovs-dpctl commands. - The ofproto library is now responsible for assigning OpenFlow port numbers. An ofproto implementation should assign them when port_construct() is called. - All dpif-based bridges of a particular type share a common datapath called "ovs-", e.g. "ovs-system". The ovs-dpctl commands will now return information on that shared datapath. To get the equivalent bridge-specific information, use the new "ovs-appctl dpif/*" commands. - Backward-incompatible changes: - Earlier Open vSwitch versions treated ANY as a wildcard in flow syntax. OpenFlow 1.1 adds a port named ANY, which introduces a conflict. ANY was rarely used in flow syntax, so we chose to retire that meaning of ANY in favor of the OpenFlow 1.1 meaning. - Patch ports no longer require kernel support, so they now work with FreeBSD and the kernel module built into Linux 3.3 and later. - New "sample" action. v1.9.0 - 26 Feb 2013 ------------------------ - Datapath: - Support for ipv6 set action. - SKB mark matching and setting. - support for Linux kernels up to 3.8 - FreeBSD is now a supported platform, thanks to code contributions from Gaetano Catalli, Ed Maste, and Giuseppe Lettieri. - ovs-bugtool: New --ovs option to report only OVS related information. - New %t and %T log escapes to identify the subprogram within a cooperating group of processes or threads that emitted a log message. The default log patterns now include this information. - OpenFlow: - Allow bitwise masking for SHA and THA fields in ARP, SLL and TLL fields in IPv6 neighbor discovery messages, and IPv6 flow label. - Adds support for writing to the metadata field for a flow. - Tunneling: - The tunneling code no longer assumes input and output keys are symmetric. If they are not, PMTUD needs to be disabled for tunneling to work. Note this only applies to flow-based keys. - New support for a nonstandard form of GRE that supports a 64-bit key. - Tunnel Path MTU Discovery default value was set to 'disabled'. This feature is deprecated and will be removed soon. - Tunnel header caching removed. - ovs-ofctl: - Commands and actions that accept port numbers now also accept keywords that represent those ports (such as LOCAL, NONE, and ALL). This is also the recommended way to specify these ports, for compatibility with OpenFlow 1.1 and later (which use the OpenFlow 1.0 numbers for these ports for different purposes). - ovs-dpctl: - Support requesting the port number with the "port_no" option in the "add-if" command. - ovs-pki: The "online PKI" features have been removed, along with the ovs-pki-cgi program that facilitated it, because of some alarmist insecurity claims. We do not believe that these claims are true, but because we do not know of any users for this feature it seems better on balance to remove it. (The ovs-pki-cgi program was not included in distribution packaging.) - ovsdb-server now enforces the immutability of immutable columns. This was not enforced in earlier versions due to an oversight. - The following features are now deprecated. They will be removed no earlier than February 2013. Please email dev@openvswitch.org with concerns. - Bridge compatibility. - Stable bond mode. - The autopath action. - Interface type "null". - Numeric values for reserved ports (see "ovs-ofctl" note above). - Tunnel Path MTU Discovery. - CAPWAP tunnel support. - The data in the RARP packets can now be matched in the same way as the data in ARP packets. v1.8.0 - 26 Feb 2013 ------------------------ *** Internal only release *** - New FAQ. Please send updates and additions! - Authors of controllers, please read the new section titled "Action Reproduction" in DESIGN, which describes an Open vSwitch change in behavior in corner cases that may affect some controllers. - ovs-l3ping: - A new test utility that can create L3 tunnel between two Open vSwitches and detect connectivity issues. - ovs-ofctl: - New --sort and --rsort options for "dump-flows" command. - "mod-port" command can now control all OpenFlow config flags. - OpenFlow: - Allow general bitwise masking for IPv4 and IPv6 addresses in IPv4, IPv6, and ARP packets. (Previously, only CIDR masks were allowed.) - Allow support for arbitrary Ethernet masks. (Previously, only the multicast bit in the destination address could be individually masked.) - New field OXM_OF_METADATA, to align with OpenFlow 1.1. - The OFPST_QUEUE request now reports an error if a specified port or queue does not exist, or for requests for a specific queue on all ports, if the specified queue does not exist on any port. (Previous versions generally reported an empty set of results.) - New "flow monitor" feature to allow controllers to be notified of flow table changes as they happen. - Additional protocols are not mirrored and dropped when forward-bpdu is false. For a full list, see the ovs-vswitchd.conf.db man page. - Open vSwitch now sends RARP packets in situations where it previously sent a custom protocol, making it consistent with behavior of QEMU and VMware. - All Open vSwitch programs and log files now show timestamps in UTC, instead the local timezone, by default. v1.7.0 - 30 Jul 2012 ------------------------ - kernel modules are renamed. openvswitch_mod.ko is now openvswitch.ko and brcompat_mod.ko is now brcompat.ko. - Increased the number of NXM registers to 8. - Added ability to configure DSCP setting for manager and controller connections. By default, these connections have a DSCP value of Internetwork Control (0xc0). - Added the granular link health statistics, 'cfm_health', to an interface. - OpenFlow: - Added support to mask nd_target for ICMPv6 neighbor discovery flows. - Added support for OpenFlow 1.3 port description (OFPMP_PORT_DESC) multipart messages. - ovs-ofctl: - Added the "dump-ports-desc" command to retrieve port information using the new port description multipart messages. - ovs-test: - Added support for spawning ovs-test server from the client. - Now ovs-test is able to automatically create test bridges and ports. - "ovs-dpctl dump-flows" now prints observed TCP flags in TCP flows. - Tripled flow setup performance. - The "coverage/log" command previously available through ovs-appctl has been replaced by "coverage/show". The new command replies with coverage counter values, instead of logging them. v1.6.1 - 25 Jun 2012 ------------------------ - Allow OFPP_CONTROLLER as the in_port for packet-out messages. v1.6.0 - 24 Feb 2012 ------------------------ *** Internal only release *** - bonding - LACP bonds no longer fall back to balance-slb when negotiations fail. Instead they drop traffic. - The default bond_mode changed from SLB to active-backup, to protect unsuspecting users from the significant risks of SLB bonds (which are documented in vswitchd/INTERNALS). - Load balancing can be disabled by setting the bond-rebalance-interval to zero. - OpenFlow: - Added support for bitwise matching on TCP and UDP ports. See ovs-ofctl(8) for more information. - NXM flow dumps now include times elapsed toward idle and hard timeouts. - Added an OpenFlow extension NXT_SET_ASYNC_CONFIG that allows controllers more precise control over which OpenFlow messages they receive asynchronously. - New "fin_timeout" action. - Added "fin_timeout" support to "learn" action. - New Nicira action NXAST_CONTROLLER that offers additional features over output to OFPP_CONTROLLER. - When QoS settings for an interface do not configure a default queue (queue 0), Open vSwitch now uses a default configuration for that queue, instead of dropping all packets as in previous versions. - Logging: - Logging to console and file will have UTC timestamp as a default for all the daemons. An example of the default format is 2012-01-27T16:35:17Z. ovs-appctl can be used to change the default format as before. - The syntax of commands and options to set log levels was simplified, to make it easier to remember. - New support for limiting the number of flows in an OpenFlow flow table, with configurable policy for evicting flows upon overflow. See the Flow_Table table in ovs-vswitch.conf.db(5) for more information. - New "enable-async-messages" column in the Controller table. If set to false, OpenFlow connections to the controller will initially have all asynchronous messages disabled, overriding normal OpenFlow behavior. - ofproto-provider interface: - "struct rule" has a new member "used" that ofproto implementations should maintain by updating with ofproto_rule_update_used(). - ovsdb-client: - The new option --timestamp causes the "monitor" command to print a timestamp with every update. - CFM module CCM broadcasts can now be tagged with an 802.1p priority. v1.5.0 - 01 Jun 2012 ------------------------ - OpenFlow: - Added support for querying, modifying, and deleting flows based on flow cookie when using NXM. - Added new NXM_PACKET_IN format. - Added new NXAST_DEC_TTL action. - ovs-ofctl: - Added daemonization support to the monitor and snoop commands. - ovs-vsctl: - The "find" command supports new set relational operators {=}, {!=}, {<}, {>}, {<=}, and {>=}. - ovsdb-tool now uses the typical database and schema installation directories as defaults. - The default MAC learning timeout has been increased from 60 seconds to 300 seconds. The MAC learning timeout is now configurable. v1.4.0 - 30 Jan 2012 ------------------------ - Compatible with Open vSwitch kernel module included in Linux 3.3. - New "VLAN splinters" feature to work around buggy device drivers in old Linux versions. (This feature is deprecated. When broken device drivers are no longer in widespread use, we will delete this feature.) See ovs-vswitchd.conf.db(5) for more information. - OpenFlow: - Added ability to match on IPv6 flow label through NXM. - Added ability to match on ECN bits in IPv4 and IPv6 through NXM. - Added ability to match on TTL in IPv4 and IPv6 through NXM. - Added ability to modify ECN bits in IPv4. - Added ability to modify TTL in IPv4. - ovs-vswitchd: - Don't require the "normal" action to use mirrors. Traffic will now be properly mirrored for any flows, regardless of their actions. - Track packet and byte statistics sent on mirrors. - The sFlow implementation can now usually infer the correct agent device instead of having to be told explicitly. - ovs-appctl: - New "fdb/flush" command to flush bridge's MAC learning table. - ovs-test: - A new distributed testing tool that allows one to diagnose performance and connectivity issues. This tool currently is not included in RH or Xen packages. - RHEL packaging now supports integration with Red Hat network scripts. - bonding: - Post 1.4.*, OVS will be changing the default bond mode from balance-slb to active-backup. SLB bonds carry significant risks with them (documented vswitchd/INTERNALS) which we want to prevent unsuspecting users from running into. Users are advised to update any scripts or configuration which may be negatively impacted by explicitly setting the bond mode which they want to use. v1.3.0 - 09 Dec 2011 ------------------------ - OpenFlow: - Added an OpenFlow extension which allows the "output" action to accept NXM fields. - Added an OpenFlow extension for flexible learning. - Bumped number of NXM registers from four to five. - ovs-appctl: - New "version" command to determine version of running daemon. - If no argument is provided for "cfm/show", displays detailed information about all interfaces with CFM enabled. - If no argument is provided for "lacp/show", displays detailed information about all ports with LACP enabled. - ovs-dpctl: - New "set-if" command to modify a datapath port's configuration. - ovs-vswitchd: - The software switch now supports 255 OpenFlow tables, instead of just one. By default, only table 0 is consulted, but the new NXAST_RESUBMIT_TABLE action can look up in additional tables. Tables 128 and above are reserved for use by the switch itself; please use only tables 0 through 127. - Add support for 802.1D spanning tree (STP). - Fragment handling extensions: - New OFPC_FRAG_NX_MATCH fragment handling mode, in which L4 fields are made available for matching in fragments with offset 0. - New NXM_NX_IP_FRAG match field for matching IP fragments (usable via "ip_frag" in ovs-ofctl). - New ovs-ofctl "get-frags" and "set-frags" commands to get and set fragment handling policy. - CAPWAP tunneling now supports an extension to transport a 64-bit key. By default it remains compatible with the old version and other standards-based implementations. - Flow setups are now processed in a round-robin manner across ports to prevent any single client from monopolizing the CPU and conducting a denial of service attack. - Added support for native VLAN tagging. A new "vlan_mode" parameter can be set for "port". Possible values: "access", "trunk", "native-tagged" and "native-untagged". - test-openflowd has been removed. Please use ovs-vswitchd instead. v1.2.0 - 03 Aug 2011 ------------------------ - New "ofproto" abstraction layer to ease porting to hardware switching ASICs. - Packaging for Red Hat Enterprise Linux 5.6 and 6.0. - Datapath support for Linux kernels up to 3.0. - OpenFlow: - New "bundle" and "bundle_load" action extensions. - Database: - Implement table unique constraints. - Support cooperative locking between callers. - ovs-dpctl: - New "-s" option for "show" command prints packet and byte counters for each port. - ovs-ofctl: - New "--readd" option for "replace-flows". - ovs-vsctl: - New "show" command to print an overview of configuration. - New "comment" command to add remark that explains intentions. - ovs-brcompatd has been rewritten to fix long-standing bugs. - ovs-openflowd has been renamed test-openflowd and moved into the tests directory. Its presence confused too many users. Please use ovs-vswitchd instead. - New ovs-benchmark utility to test flow setup performance. - A new log level "off" has been added. Configuring a log facility "off" prevents any messages from being logged to it. Previously, "emer" was effectively "off" because no messages were ever logged at level "emer". Now, errors that cause a process to exit are logged at "emer" level. - "configure" option --with-l26 has been renamed --with-linux, and --with-l26-source has been renamed --with-linux-source. The old names will be removed after the next release, so please update your scripts. - The "-2.6" suffix has been dropped from the datapath/linux-2.6 and datapath/linux-2.6/compat-2.6 directories. - Feature removals: - Dropped support for "tun_id_from_cookie" OpenFlow extension. Please use the extensible match extensions instead. - Removed the Maintenance_Point and Monitor tables in an effort to simplify 802.1ag configuration. - Performance and scalability improvements - Bug fixes v1.1.0 - 05 Apr 2011 ------------------------ - Ability to define policies over IPv6 - LACP - 802.1ag CCM - Support for extensible match extensions to OpenFlow - QoS: - Support for HFSC qdisc. - Queue used by in-band control can now be configured. - Kernel: - Kernel<->userspace interface has been reworked and should be close to a stable ABI now. - "Port group" concept has been dropped. - GRE over IPSEC tunnels - Bonding: - New active backup bonding mode. - New L4 hashing support when LACP is enabled. - Source MAC hash now includes VLAN field also. - miimon support. - Greatly improved handling of large flow tables - ovs-dpctl: - "show" command now prints full vport configuration. - "dump-groups" command removed since kernel support for port groups was dropped. - ovs-vsctl: - New commands for working with the new Managers table. - "list" command enhanced with new formatting options and --columns option. - "get" command now accepts new --id option. - New "find" command. - ovs-ofctl: - New "queue-stats" command for printing queue stats. - New commands "replace-flows" and "diff-flows". - Commands to add and remove flows can now read from files. - New --flow-format option to enable or disable NXM. - New --more option to increase OpenFlow message verbosity. - Removed "tun-cookie" command, which is no longer useful. - ovs-controller enhancements for testing various features. - New ovs-vlan-test command for testing for Linux kernel driver VLAN bugs. New ovs-vlan-bug-workaround command for enabling and disabling a workaround for these driver bugs. - OpenFlow support: - "Resubmit" actions now update flow statistics. - New "register" extension for use in matching and actions, via NXM. - New "multipath" experimental action extension. - New support for matching multicast Ethernet frames, via NXM. - New extension for OpenFlow vendor error codes. - New extension to set the QoS output queue without actually sending to an output port. - Open vSwitch now reports a single flow table, instead of separate hash and wildcard tables. This better models the current implementation. - New experimental "note" action. - New "ofproto/trace" ovs-appctl command and associated utilities to ease debugging complex flow tables. - Database: - Schema documentation now includes an entity-relationship diagram. - The database is now garbage collected. In most tables, unreferenced rows will be deleted automatically. - Many tables now include statistics updated periodically by ovs-vswitchd or ovsdb-server. - Every table now has an "external-ids" column for use by OVS integrators. - There is no default controller anymore. Each bridge must have its controller individually specified. - The "fail-mode" is now a property of a Bridge instead of a Controller. - New versioning and checksum features. - New Managers table and manager_options column in Open_vSwitch table for specifying managers. The old "managers" column in the Open_vSwitch table has been removed. - Many "name" columns are now immutable. - Feature removals: - Dropped support for XenServer pre-5.6.100. - Dropped support for Linux pre-2.6.18. - Dropped controller discovery support. - Dropped "ovs-ofctl status" and the OpenFlow extension that it used. Statistics reporting in the database is a rough equivalent. - Dropped the "corekeeper" package (now separate, at http://openvswitch.org/cgi-bin/gitweb.cgi?p=corekeeper). - Performance and scalability improvements - Bug fixes v1.1.0pre2 - 13 Sep 2010 ------------------------ - Bug fixes v1.1.0pre1 - 31 Aug 2010 ------------------------ - OpenFlow 1.0 slicing (QoS) functionality - Python bindings for configuration database (no write support) - Performance and scalability improvements - Bug fixes v1.0.1 - 31 May 2010 -------------------- - New "patch" interface type - Bug fixes v1.0.0 - 15 May 2010 -------------------- - Configuration database with remote management - OpenFlow 1.0 - GRE tunneling - Support for XenServer 5.5 and 5.6 - Performance and scalability improvements - Bug fixes v0.99.2 - 18 Feb 2010 --------------------- - Bug fixes v0.99.1 - 25 Jan 2010 --------------------- - Add support for sFlow(R) - Make headers compatible with C++ - Bug fixes v0.99.0 - 14 Jan 2010 --------------------- - User-space forwarding engine - Bug fixes v0.90.7 - 29 Nov 2009 --------------------- - Add support for NetFlow active timeouts - Bug fixes v0.90.6 - 6 Oct 2009 -------------------- - Bug fixes v0.90.5 - 21 Sep 2009 --------------------- - Generalize in-band control to more diverse network setups - Bug fixes ovs-2.9.0/NOTICE000066400000000000000000000027151324262074100132430ustar00rootroot00000000000000This file is included in compliance with the Apache 2.0 license, available at http://www.apache.org/licenses/LICENSE-2.0.html Open vSwitch Copyright (c) 2007, 2008, 2009, 2010, 2011, 2013 Nicira, Inc. Open vSwitch BSD port Copyright (c) 2011 Gaetano Catalli Apache Portable Runtime Copyright 2008 The Apache Software Foundation. This product includes software developed by The Apache Software Foundation (http://www.apache.org/). Portions of this software were developed at the National Center for Supercomputing Applications (NCSA) at the University of Illinois at Urbana-Champaign. lib/ovs.tmac includes troff macros written by Eric S. Raymond and Werner Lemberg. m4/include_next.m4 and m4/absolute-header.m4 Copyright (C) 2006-2013 Free Software Foundation, Inc. Rapid Spanning Tree Protocol (RSTP) implementation Copyright (c) 2011-2014 M3S, Srl - Italy LLDP implementation Copyright (c) 2008, 2012 Vincent Bernat LLDP includes code used from the Net::CDP project based on the ISC license Copyright (c) 2014 Michael Chapman LLDP includes code used from the ladvd project based on the ISC license Copyright (c) 2008, 2009, 2010 Sten Spans Auto Attach implementation Copyright (c) 2014, 2015 WindRiver, Inc Copyright (c) 2014, 2015 Avaya, Inc TCP connection tracker from FreeBSD pf, BSD licensed Copyright (c) 2001 Daniel Hartmeier Copyright (c) 2002 - 2008 Henning Brauer Copyright (c) 2012 Gleb Smirnoff ovs-2.9.0/README.rst000066400000000000000000000104261324262074100140240ustar00rootroot00000000000000.. NOTE(stephenfin): If making changes to this file, ensure that the line numbers found in 'Documentation/intro/what-is-ovs' are kept up-to-date. ============ Open vSwitch ============ .. image:: https://travis-ci.org/openvswitch/ovs.png :target: https://travis-ci.org/openvswitch/ovs .. image:: https://ci.appveyor.com/api/projects/status/github/openvswitch/ovs?branch=master&svg=true&retina=true :target: https://ci.appveyor.com/project/blp/ovs/history What is Open vSwitch? --------------------- Open vSwitch is a multilayer software switch licensed under the open source Apache 2 license. Our goal is to implement a production quality switch platform that supports standard management interfaces and opens the forwarding functions to programmatic extension and control. Open vSwitch is well suited to function as a virtual switch in VM environments. In addition to exposing standard control and visibility interfaces to the virtual networking layer, it was designed to support distribution across multiple physical servers. Open vSwitch supports multiple Linux-based virtualization technologies including Xen/XenServer, KVM, and VirtualBox. The bulk of the code is written in platform-independent C and is easily ported to other environments. The current release of Open vSwitch supports the following features: - Standard 802.1Q VLAN model with trunk and access ports - NIC bonding with or without LACP on upstream switch - NetFlow, sFlow(R), and mirroring for increased visibility - QoS (Quality of Service) configuration, plus policing - Geneve, GRE, VXLAN, STT, and LISP tunneling - 802.1ag connectivity fault management - OpenFlow 1.0 plus numerous extensions - Transactional configuration database with C and Python bindings - High-performance forwarding using a Linux kernel module The included Linux kernel module supports Linux 3.10 and up. Open vSwitch can also operate entirely in userspace without assistance from a kernel module. This userspace implementation should be easier to port than the kernel-based switch. OVS in userspace can access Linux or DPDK devices. Note Open vSwitch with userspace datapath and non DPDK devices is considered experimental and comes with a cost in performance. What's here? ------------ The main components of this distribution are: - ovs-vswitchd, a daemon that implements the switch, along with a companion Linux kernel module for flow-based switching. - ovsdb-server, a lightweight database server that ovs-vswitchd queries to obtain its configuration. - ovs-dpctl, a tool for configuring the switch kernel module. - Scripts and specs for building RPMs for Citrix XenServer and Red Hat Enterprise Linux. The XenServer RPMs allow Open vSwitch to be installed on a Citrix XenServer host as a drop-in replacement for its switch, with additional functionality. - ovs-vsctl, a utility for querying and updating the configuration of ovs-vswitchd. - ovs-appctl, a utility that sends commands to running Open vSwitch daemons. Open vSwitch also provides some tools: - ovs-ofctl, a utility for querying and controlling OpenFlow switches and controllers. - ovs-pki, a utility for creating and managing the public-key infrastructure for OpenFlow switches. - ovs-testcontroller, a simple OpenFlow controller that may be useful for testing (though not for production). - A patch to tcpdump that enables it to parse OpenFlow messages. What other documentation is available? -------------------------------------- .. TODO(stephenfin): Update with a link to the hosting site of the docs, once we know where that is To install Open vSwitch on a regular Linux or FreeBSD host, please read the `installation guide `__. For specifics around installation on a specific platform, refer to one of the `other installation guides `__ For answers to common questions, refer to the `FAQ `__. To learn about some advanced features of the Open vSwitch software switch, read the `tutorial `__. Each Open vSwitch userspace program is accompanied by a manpage. Many of the manpages are customized to your configuration as part of the build process, so we recommend building Open vSwitch before reading the manpages. Contact ------- bugs@openvswitch.org ovs-2.9.0/Vagrantfile000066400000000000000000000127641324262074100145310ustar00rootroot00000000000000# -*- mode: ruby -*- # vi: set ft=ruby : # Vagrantfile API/syntax version. Don't touch unless you know what you're doing! VAGRANTFILE_API_VERSION = "2" Vagrant.require_version ">=1.7.0" $bootstrap_fedora = <