Sunday, June 26, 2016

Monitoring FAUCET with Grafana and InfluxDB

Instead of SNMP, FAUCET can push basic statistics - port input and output bytes and packets - to InfluxDB. You can then graph them using Grafana. In this post we will discuss how to set up FAUCET to do this.

snapshot11.png


  • We will indeed to install InfluxDB and Grafana on the same machine that runs FAUCET (it is possible to run them on different machines or across a network also, but this is the most simple configuration).
InfluxDB (https://docs.influxdata.com/influxdb/)
Grafana (http://docs.grafana.org/guides/gettingstarted/)
  • Create an InfluxDB database for FAUCET to populate:

$ influx
Visit https://enterprise.influxdata.com to register for updates, InfluxDB server management, and monitoring.
Connected to http://localhost:8086 version 0.13.0
InfluxDB shell version: 0.13.0
> create database faucet
>

  • Now to configure FAUCET monitoring FAUCET has a separate script, gauge, that does monitoring (monitoring therefore is completely separate from switching). gauge uses FAUCET's config file, and one of its own, gauge.yaml:
faucet_configs:
    - '/etc/ryu/faucet/faucet.yaml'
watchers:
    port_stats_poller:
        type: 'port_stats'
        dps: ['windscale-faucet-1']
        interval: 10
        db: 'influx'
dbs:
    influx:
        type: 'influx'
        influx_db: 'faucet'
        influx_host: 'localhost'
        influx_port: 8086
        influx_user: 'faucet'
        influx_pwd: ''

        influx_timeout: 10

  • Next, you will need to configure the switch to connect to gauge. gauge is, from the switch's point of view, an additional OpenFlow controller, that only requests statistics via OpenFlow. Add the necessary configuration to the switch to connect to a port on the machine where gauge will run (Eg, 6634). 

  • Next, run gauge. Eg,

/usr/bin/python /usr/local/bin/ryu-manager --ofp-listen-host=[controller IP] --ofp-tcp-listen-port=6634 /home/faucet/faucet/src/ryu_faucet/org/onfsdn/faucet/gauge.py
  • You should now start to see gauge start to poll for statistics once the switch has connected. Check /var/log/faucet/gauge.log (default log location).
  • Now check that InfluxDB is receiving the data. 

$ influx
Visit https://enterprise.influxdata.com to register for updates, InfluxDB server management, and monitoring.
Connected to http://localhost:8086 version 0.13.0
InfluxDB shell version: 0.13.0
> use faucet
Using database faucet
> show measurements
name: measurements
------------------
name
bytes_in
bytes_out
dropped_in
dropped_out
errors_in
packets_in
packets_out
port_state_reason
database
httpd
measurement
queryExecutor
runtime
shard
subscriber
tsm1_cache
tsm1_filestore
tsm1_wal
write
> show series
  • It should now be possible for you to graph FAUCET data from the Grafana UI.





Thursday, June 23, 2016

FAUCET troubleshooting/FAQ

This post documents some common troubleshooting steps for bringing up FAUCET.


How do I configure/where's the documentation for some FAUCET feature?

Look for configuration for that feature in the FAUCET unit tests (ie. read https://github.com/REANNZ/faucet/blob/master/tests/faucet_mininet_test.py and adapt a config file from that test's CONFIG attribute).


Where do I look for controller status?

By default, in /var/log/faucet/faucet.log. You should see messages as hosts are learned on VLANs, such as:

Jun 24 11:24:56 faucet DEBUG    Packet_in dp_id: eccd6df72de7 src:b8:ae:ed:73:20:90 in_port:1 vid:2003
Jun 24 11:24:56 faucet INFO     learned 5 hosts on vlan 2003

If you don't, something else is wrong - see below.


What does unknown DPID in the FAUCET log mean?

dp_id in configuration, and on the switch have to match. Make them the same. Unless they are, the switch won't come up. dp_id is a hex number.


FAUCET reports errors parsing my config file.

  • FAUCET uses YAML, which is sensitive to tabs versus spaces and correct indentation. Use spaces exclusively. Use the same configuration format as what the tests use (see above documentation FAQ).
  • Use quotes around strings (Eg, MAC addresses in ACLs)  to stop the YAML parser from misinterpreting them as numbers.


How I know the OpenFlow connection is up?

  • Check the FAUCET log (see above),
  • Check that the switch is configured to connect to port FAUCET is listening on.
  • Use standard networking troubleshooting to ensure that the switch can make a successful TCP connection to the controller (Eg, use wireshark to verify a TCP connection is attempted and completed and that OpenFlow messages are being exchanged without errors).
  • Check that FAUCET has provisioned flows on the switch - check that there flows installed in tables 0-4. If there are no flows or no flows other than in table 0, then your connection is not up (see above).

Wednesday, June 15, 2016

RAUCET - Raspberry Pi + FAUCET, controlling Zodiac FX

In this post, we will set up a Raspberry Pi 3, to control a Zodiac FX (we will cover NFV in the same combination in a future post).




Requirements

  • A Raspberry Pi 3 (a 2 might work) with power supply
  • A Zodiac FX
  • A USB Ethernet adaptor compatible with the Pi (I used a Mac 100M adaptor, but others may work)
You could also manage the Pi over WiFi, and just use the Pi's built in Ethernet to do OpenFlow, if you don't have an adaptor to spare.

Installation

  • Install at least v0.63 firmware on the ZodiacFX (see the ZodiacFX forums for firmware and user manual; you will need a x86 based computer to run the Atmel firmware updater tool, as the tool will not run on the Pi itself).
  • Configure a static IP address, on the Pi's interface that be the OpenFlow control plane network. We will use what the Zodiac FX expects for simplicity. Assuming that the adaptor is eth1, add this to your /etc/network/interfaces on the Pi. 

auto eth1
iface eth1 inet static
  address 10.0.1.8
  netmask 255.255.255.0 
  • Reboot to have the change take effect.
  • Connect cables to the ZodiacFX (as shown), and check that you can ping the Zodiac
ping 10.0.1.99
  • Install FAUCET's dependencies on the Pi.
sudo apt-get install git minicom libpython-dev python-yaml paramiko
sudo pip install ipaddr
sudo pip install --upgrade six
sudo pip install ryu
  • Install FAUCET itself, in /home/pi
git clone https://github.com/REANNZ/faucet
  • Obtain the ZodiacFX's MAC address from the label on the underside. Or, use minicom to obtain the address:
sudo minicom -s
(change serial port to /dev/ttyACM0, under serial port options - check dmesg)
(exit)

Then, via minicom:

config
show config

On my ZodiacFX, I have the following:

Zodiac_FX# config
Zodiac_FX(config)# show config

-------------------------------------------------------------------------
Configuration
 Name: Zodiac_FX
 MAC Address: 70:B3:D5:6C:D0:C0
 IP Address: 10.0.1.99
 Netmask: 255.255.255.0
 Gateway: 10.0.1.1
 OpenFlow Controller: 10.0.1.8
 OpenFlow Port: 6633
 Openflow Status: Enabled
 Failstate: Secure
 Force OpenFlow version: Disabled
 Stacking Select: MASTER
 Stacking Status: Unavailable

-------------------------------------------------------------------------
  • Back on the Pi (quit from minicom with control-A, Q), create a FAUCET config file.
mkdir /home/pi/zodiacfx
vi /home/pi/zodiacfx/faucet.yaml
  • Convert the ZodiacFX MAC address to a FAUCET DPID.  My ZodiacFX's MAC is 70:B3:D5:6C:D0:C0 which is a DPID of0x70b3d56cd0c0 (running all the digits together, removing the ":"s, and prepending 0x).
  • In the faucet.yaml file, add this (replacing dp_id as above), and save and exit.

version: 2
vlans:
    100:

        name: "100"
dps:
    zodiac-fx-1:
        dp_id: 0x70b3d56cd0c0 
        hardware: "ZodiacFX"
        interfaces:
        1:
            native_vlan: 100 
        2:
            native_vlan: 100
        3:
            native_vlan: 100

  • Now start FAUCET itself:

export FAUCET_CONFIG=~/zodiacfx/faucet.yaml
export FAUCET_LOG=~/zodiacfx/faucet.log
export FAUCET_EXCEPTION_LOG=~/zodiacfx/faucet_exception.log
/usr/local/bin/ryu-manager --ofp-listen-host=10.0.1.8 --ofp-tcp-listen-port=6633 ~/faucet/src/ryu_faucet/org/onfsdn/faucet/faucet.py --verbose

If it works, you will see the following. You now have a working FAUCET switch! If you connect hosts to ports 1, 2, or 3 they should be able to reach each other.

loading app /home/pi/faucet/src/ryu_faucet/org/onfsdn/faucet/faucet.py
loading app ryu.controller.ofp_handler
instantiating app None of DPSet
creating context dpset
instantiating app /home/pi/faucet/src/ryu_faucet/org/onfsdn/faucet/faucet.py of Faucet
instantiating app ryu.controller.ofp_handler of OFPHandler
BRICK dpset
  PROVIDES EventDPReconnected TO {'Faucet': set(['dpset'])}
  PROVIDES EventDP TO {'Faucet': set(['dpset'])}
  CONSUMES EventOFPPortStatus
  CONSUMES EventOFPSwitchFeatures
  CONSUMES EventOFPStateChange
BRICK Faucet
  CONSUMES EventOFPPortStatus
  CONSUMES EventOFPSwitchFeatures
  CONSUMES EventFaucetResolveGateways
  CONSUMES EventOFPErrorMsg
  CONSUMES EventFaucetReconfigure
  CONSUMES EventFaucetHostExpire
  CONSUMES EventOFPPacketIn
  CONSUMES EventDPReconnected
  CONSUMES EventDP
BRICK ofp_event
  PROVIDES EventOFPPortStatus TO {'dpset': set(['main']), 'Faucet': set(['main'])}
  PROVIDES EventOFPErrorMsg TO {'Faucet': set(['main'])}
  PROVIDES EventOFPSwitchFeatures TO {'dpset': set(['config']), 'Faucet': set(['config'])}
  PROVIDES EventOFPStateChange TO {'dpset': set(['main', 'dead'])}
  PROVIDES EventOFPPacketIn TO {'Faucet': set(['main'])}
  CONSUMES EventOFPEchoReply
  CONSUMES EventOFPPortStatus
  CONSUMES EventOFPErrorMsg
  CONSUMES EventOFPSwitchFeatures
  CONSUMES EventOFPPortDescStatsReply
  CONSUMES EventOFPHello
  CONSUMES EventOFPEchoRequest
EVENT Faucet->Faucet EventFaucetResolveGateways
EVENT Faucet->Faucet EventFaucetHostExpire
connected socket:<eventlet.greenio.base.GreenSocket object at 0x75cdc210> address:('10.0.1.99', 49946)
hello ev <ryu.controller.ofp_event.EventOFPHello object at 0x75cdc910>
move onto config mode
EVENT ofp_event->dpset EventOFPSwitchFeatures
EVENT ofp_event->Faucet EventOFPSwitchFeatures
switch features ev version=0x4,msg_type=0x6,msg_len=0x20,xid=0x5ce4692f,OFPSwitchFeatures(auxiliary_id=0,capabilities=7,datapath_id=123917682135232L,n_buffers=0,n_tables=10)
move onto main mode
EVENT ofp_event->dpset EventOFPStateChange
DPSET: register datapath <ryu.controller.controller.Datapath object at 0x75cdc510>
EVENT dpset->Faucet EventDP

Adding as a systemd service

Since Raspbian uses systemd, you can write a faucet.service file that causes FAUCET to be automatically run on boot.
  • Create a file /home/pi/zodiacfx/start-faucet.sh, made executable, containing:
#!/bin/bash
export FAUCET_CONFIG=~/zodiacfx/faucet.yaml
export FAUCET_LOG=~/zodiacfx/faucet.log
export FAUCET_EXCEPTION_LOG=~/zodiacfx/faucet_exception.log
/usr/local/bin/ryu-manager --ofp-listen-host=10.0.1.8 --ofp-tcp-listen-port=6633 ~/faucet/src/ryu_faucet/org/onfsdn/faucet/faucet.py --verbose


  • Create a file /etc/systemd/system/faucet.service, containing:
[Unit]
Description=FAUCET

[Service]
User=pi
TimeoutStartSec=0
ExecStart=/home/pi/zodiacfx/start-faucet.sh

[Install]
WantedBy=multi-user.target
  • Enable the service and start it:
sudo systemctl enable /etc/systemd/system/faucet.service
sudo systemctl start faucet.service








Saturday, June 4, 2016

FAUCET at the Open Network Foundation

The Open Network Foundation runs FAUCET, and challenges others to do the same.

http://us8.campaign-archive2.com/?u=394e6edee21a3eea44bef0d13&id=cc133a4d88&e=025aca6d5b

Openflow Wired/Wireless Network Deployed @ the ONF Office
Being the flag bearers of OpenFlow and SDN, ONF has started to eat its own dog food, and I’d like you to do the same. Two weeks ago, a pure OpenFlow wired/wireless network was deployed for production use at the ONF office in Menlo Park.

Wednesday, June 1, 2016

Unittesting hardware

Having unit tests for software is all very well; but how can you test hardware?

FAUCET supports running unit tests against hardware as well against software.

This means you can verify - with the exact same test code - and before you deploy a new switch software release - that your code (and your switch) works correctly.

That's a lot more fun than a time consuming lab test or an embarrassing code upgrade and rollback in production.


                     +--------------------------+
                     |                          |
                     |         FAUCET CPN       |
                     |                          |
                     |                          |
+------------------------------+     +-------------------------+
|                    |         |     |          |              |
|                    |    +--+ |     | +--+     |              |
|                    |    |  +---------+  |     |              |
|   FAUCET test host |    +--+ |     | +--+     |              |
|                    +--------------------------+              |
|                              |     |                         |
|                              |     |                         |
|                              |     |                         |
|                              |     |                         |
|          +---------------------+   |                         |
|          |   +------+   +--+ | |   | +--+                    |
|          |   |VM 1  |   |  +---------+  |                    |
|          |   +------+   +--+ | |   | +--+                    |
|          |                   | |   |                         |
|          |   +------+   +--+ | |   | +--+  OpenFlow switch   |
|          |   |VM 2  |   |  +---------+  |  under test        |
|          |   +------+   +--+ | |   | +--+                    |
|          |                   | |   |                         |
|          |   +------+   +--+ | |   | +--+                    |
|          |   |VM 3  |   |  +---------+  |                    |
|          |   +------+   +--+ | |   | +--+                    |
|          |                   | |   |                         |
|          |   +------+   +--+ | |   | +--+                    |
|          |   |VM 4  |   |  +---------+  |                    |
|          |   +------+   +--+ | |   | +--+                    |
|          |                   | |   |                         |
|          |                   | |   |                         |
+------------------------------+ |   +-------------------------+
           |                     |
           |    MININET          |
           |                     |
           |                     |
           +---------------------+

To test hardware, connect 5 ports from your hardware switch, to 5 physical Ethernet ports on your test host. 1 of those ports will have to be for your control plane, of course.

Then, modify hw_switch_config.yaml in FAUCET's test directory. This file tells FAUCET how to map the ports you connected to the mininet test hosts:

# if set to True, map a hardware OpenFlow switch to ports on this machine.
# Otherwise, run tests against OVS locally.
hw_switch: True
# Map ports on the hardware switch, to physical ports on this machine.
dp_ports:
  1: p1p1
  2: p1p2
  3: p1p3
  4: p1p4
# override test DPID if necessary
dpid: 0000eccd6d9936ed

In this example, switch port 1, is mapped to physical port p1p1 on the test host, and so on.

Then run the tests. Now, the tests will run against hardware, instead of software.