Versions Compared

Version Old Version 56 New Version 57
Changes made by

Paul Sokolovskyy

Paul Sokolovskyy

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Comment: Move "Bringing up LAVA setup after host reboot" at the end of doc after all

...

  1. Optional: Setup access for local IP/host

    If you are running lava on a machine with a fixed IP (eg: a dedicated computer on your local network) you will want to modify the  overlays/lava-server/etc/lava-server/settings.conf file to allow access.  You'll want to modify the ALLOWED_HOSTS line to add the ip or hostname of the machine.  In the example below, the ip 192.168.1.14 was added, you'll want to keep 127.0.0.1 and localhost .

    Code Block
    languageyml
    linenumberstrue
    {
    "HTTPS_XML_RPC": false,
    "MOUNT_POINT": "/",
    "STATIC_URL": "/static/",
    "ALLOWED_HOSTS": ["192.168.1.14", "127.0.0.1", "localhost"],
    "CSRF_COOKIE_SECURE": false,
    "SESSION_COOKIE_SECURE": false,
    "EVENT_NOTIFICATION": true,
    "EVENT_TOPIC": "lava-server",
    "INTERNAL_EVENT_SOCKET": "tcp://lava-publisher:5557"
}


  2. Optional: Add the proxy settings in lite-lava-dispatcher and ser2net images

    It might be required to define proxy setting in lite-lava-dispatcher/Dockerfile and ser2net/Dockerfile

    Code Block
    languagebash
    ENV http_proxy=http://<user>:<pwd>@<proxy>:<port>
ENV https_proxy=http://<user>:<pwd>@<proxy>:<port>


  3. Optional: Configure DNS in /etc/docker/daemon.json

    Get host DNS 

    Code Block
    languagebash
    nmcli dev show | grep 'IP4.DNS'

    Set reported value in /etc/docker/daemon.json

    Code Block
    languageyml
    {                                                                          
    "dns": ["w.x.y.z"]                                                                           
}


  4. Run commands to build docker images and startup lava

    Code Block
    languagebash
    docker-compose build
make

    You'll see output of docker fetching and building the images, and then all LAVA containers starting up, followed by a fair amount of logging from LAVA itself as it starts up.  This may take a few minutes to complete.

    Here are some examples of the output one can expect during this phase:

    • Docker Compose/Build Phase

      Code Block
      languagebash
      galak@ubuntu:~/lava-docker-compose$ make
docker-compose up
Creating volume "lava-server-pgdata" with default driver
Creating volume "lava-server-devices" with default driver
Creating volume "lava-server-health-checks" with default driver
Creating volume "lava-server-joboutput" with default driver
Pulling db (postgres:11.2-alpine)...
11.2-alpine: Pulling from library/postgres
bdf0201b3a05: Pull complete
365f27dc05d7: Pull complete
bf541d40dfbc: Pull complete
823ce70c3252: Extracting [========>                                          ]  4.194MB/25.04MB
a92a31ecd32a: Download complete
83cc8c6d8282: Download complete
7995b9edc9bf: Download complete
7616119153d9: Download complete
b3f69561e369: Download complete


    • Lava containers being created

      Code Block
      languagebash
      Creating lava-server-db  ... done
Creating lava-dispatcher ... done
Creating lava-ser2net    ... done
Creating lava-publisher  ... done
Creating lava-master     ... done
Creating lava-logs       ... done
Creating lava-server     ... done
Creating apache2         ...


    • Lava starting up

      Code Block
      languagebash
      lava-master        |   Applying lava_results_app.0012_namedtestattribute_metadata... OK
lava-server        | .
lava-master        |   Applying lava_results_app.0013_buglinks... OK
lava-master        |   Applying lava_results_app.0014_xaxis_maxlength_increase... OK
lava-master        |   Applying dashboard_app.0002_auto_20140917_1935... OK
lava-logs          | .
lava-master        |   Applying dashboard_app.0003_auto_20140926_1208... OK
lava-master        |   Applying dashboard_app.0004_imagereportchart_is_delta... OK
lava-dispatcher    | 2019-05-09 19:01:08,522   DEBUG [BTSP] Checking master [lava-master:5556] to create socket for lava-dispatcher
lava-dispatcher    | 2019-05-09 19:01:08,524   DEBUG [BTSP] socket IPv4 address: 172.18.0.6
lava-dispatcher    | 2019-05-09 19:01:08,525    INFO [BTSP] Greeting master => 'HELLO_RETRY' (using the same version?)
lava-master        |   Applying dashboard_app.0005_imagereportchart_chart_height... OK
lava-master        |   Applying dashboard_app.0006_auto_20141028_1146... OK
lava-master        |   Applying dashboard_app.0007_imagereportchart_chart_visibility... OK
lava-server        | .
lava-master        |   Applying dashboard_app.0008_imagechartfilter_is_all_tests_included... OK


    • Lava idle

      Code Block
      languagebash
      lava-dispatcher    | 2019-05-09 19:03:21,510   DEBUG PING => master (last message 20s ago)
lava-master        | 2019-05-09 19:03:21,513   DEBUG lava-dispatcher => PING(20)
lava-dispatcher    | 2019-05-09 19:03:21,519   DEBUG master => PONG(20)
lava-master        | 2019-05-09 19:03:36,201    INFO scheduling health checks:
lava-master        | 2019-05-09 19:03:36,205    INFO scheduling jobs:
lava-master        | 2019-05-09 19:03:38,651   DEBUG lava-logs => PING(20)
lava-logs          | 2019-05-09 19:03:38,648   DEBUG PING => master
lava-logs          | 2019-05-09 19:03:38,657   DEBUG master => PONG(20)
lava-master        | 2019-05-09 19:03:41,549   DEBUG lava-dispatcher => PING(20)
lava-dispatcher    | 2019-05-09 19:03:41,546   DEBUG PING => master (last message 20s ago)
lava-dispatcher    | 2019-05-09 19:03:41,555   DEBUG master => PONG(20)
lava-master        | 2019-05-09 19:03:56,221    INFO scheduling health checks:
lava-master        | 2019-05-09 19:03:56,225    INFO scheduling jobs:
lava-logs          | 2019-05-09 19:03:58,681   DEBUG PING => master
lava-master        | 2019-05-09 19:03:58,684   DEBUG lava-logs => PING(20)
lava-logs          | 2019-05-09 19:03:58,690   DEBUG master => PONG(20)
lava-master        | 2019-05-09 19:04:01,579   DEBUG lava-dispatcher => PING(20)
lava-dispatcher    | 2019-05-09 19:04:01,577   DEBUG PING => master (last message 20s ago)
lava-dispatcher    | 2019-05-09 19:04:01,584   DEBUG master => PONG(20)


  5. Verify lava is running

    You can verify that lava has completed startup by trying to connect to the LAVA webserver in a web browser by going to localhost.

    Here's what the startup webpage should look like in your web browser:

...

Code Block
languagebash
sudo contrib/udev-forward.py -i lava-dispatcher

Bringing up LAVA setup after host reboot

At the time of writing, the docker-compose configuration is set to automatically start on system (re)boot. However, not all containers comprising the system may start up automatically. You check the status by going to lava-docker-compose directory and running "docker-compose ps". Example output after system reboot is shown below (it may differ for your system):

...

languagebash

...

Sending the board a job

Now that we've got everything set, we can try and send an actual job to the board.  A job will try and fetch a binary to flash the board with from a URL.  How job files are composed is outside of the scope of this guide.  However, we provide an example job file to test and verify that everything is working.  In the repository example/lava.job is a YAML structured job file.  If you look at the file, you'll see a reference to  https://snapshots.linaro.org/components/kernel/zephyr/master/zephyr/frdm_k64f/3708/tests/subsys/logging/log_list/logging.log_list/zephyr/zephyr.bin this file will be fetched and flashed to the board using the boot method cmsis-dap (which is specified in this file). 


We utilize lavacli to submit the job file to the lava server.  The command will return a number to the user which is the job number (ID).  We can then utilize a web browser to see the results of the job.

  • Submit a job

    Note: the second line is the output of submitting the job, with the job ID 1

    Code Block
    languagebash
    lavacli -i dispatcher jobs submit example/lava.job
1


  • See job results

    The job results are available at  http://localhost/scheduler/job/<JOB ID> .  The case above, the job ID is 1, so the URL would be  http://localhost/scheduler/job/1 .  As with all other URLs, if you have a fixed IP/hostname you can replace localhost with that IP/hostname.

    The following screen captures try to give a sense of the output you should see, it's not a complete log of everything you will see, however we do show the final output and passing of the test:

    The start of the job:

    Image Added

You might get the following certificate error.

Code Block
Job error: Invalid job data: ["Unable to get 'https://snapshots.linaro.org/components/kernel/zephyr/master/zephyr/frdm_k64f/3708/tests/subsys/logging/log_list/logging.log_list/zephyr/zephyr.bin': HTTPSConnectionPool(host='snapshots.linaro.org', port=443): Max retries exceeded with url: /components/kernel/zephyr/master/zephyr/frdm_k64f/3708/tests/subsys/logging/log_list/logging.log_list/zephyr/zephyr.bin (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:720)'),))"]

You then need to bind your  host certificates to container by adding following line in lava-docker-compose/docker-compose.yaml:

Code Block
languageyml
  lava-master:
    container_name: lava-master
    image: ${DC_SERVER_IMAGE}
    volumes:
    ...
    - /usr/local/share/ca-certificates:/usr/local/share/ca-certificates:ro

Then, run the following command in the container and go back to job start step.

Code Block
update-ca-certificates


The job continues, we see some output result from the board in light green:
Image Added

The Final output, we see the 'result:pass'
Image Added

Bringing up LAVA setup after host reboot

At the time of writing, the docker-compose configuration is set to automatically start on system (re)boot. However, not all containers comprising the system may start up automatically. You check the status by going to lava-docker-compose directory and running "docker-compose ps". Example output after system reboot is shown below (it may differ for your system):

Code Block
languagebash
     Name                                                    
lava-server Command      /root/entrypoint.sh              Up         5500/tcp, 5555/tcp, 5556/tcp, 80/tcp  State             lava-server-db    docker-entrypoint.sh postgres    Up     Ports    5432/tcp

As can be seen, lava-dispatcher and lava-ser2net containers didn't start up successfully. This usually can be rectified by running "docker-compose up". To bring up the system into a fully functional state, udev forwarder must be started manually too.

Sending the board a job

Now that we've got everything set, we can try and send an actual job to the board.  A job will try and fetch a binary to flash the board with from a URL.  How job files are composed is outside of the scope of this guide.  However, we provide an example job file to test and verify that everything is working.  In the repository example/lava.job is a YAML structured job file.  If you look at the file, you'll see a reference to  https://snapshots.linaro.org/components/kernel/zephyr/master/zephyr/frdm_k64f/3708/tests/subsys/logging/log_list/logging.log_list/zephyr/zephyr.bin this file will be fetched and flashed to the board using the boot method cmsis-dap (which is specified in this file). 

We utilize lavacli to submit the job file to the lava server.  The command will return a number to the user which is the job number (ID).  We can then utilize a web browser to see the results of the job.

Submit a job

Note: the second line is the output of submitting the job, with the job ID 1

Code Block
languagebash
lavacli -i dispatcher jobs submit example/lava.job
1

See job results

The job results are available at  http://localhost/scheduler/job/<JOB ID> .  The case above, the job ID is 1, so the URL would be  http://localhost/scheduler/job/1 .  As with all other URLs, if you have a fixed IP/hostname you can replace localhost with that IP/hostname.

...

You might get the following certificate error.

Code Block
Job error: Invalid job data: ["Unable to get 'https://snapshots.linaro.org/components/kernel/zephyr/master/zephyr/frdm_k64f/3708/tests/subsys/logging/log_list/logging.log_list/zephyr/zephyr.bin': HTTPSConnectionPool(host='snapshots.linaro.org', port=443): Max retries exceeded with url: /components/kernel/zephyr/master/zephyr/frdm_k64f/3708/tests/subsys/logging/log_list/logging.log_list/zephyr/zephyr.bin (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:720)'),))"]

You then need to bind your  host certificates to container by adding following line in lava-docker-compose/docker-compose.yaml:

Code Block
languageyml
  lava-master:
    container_name: lava-master
    image: ${DC_SERVER_IMAGE}
    volumes:
    ...
    - /usr/local/share/ca-certificates:/usr/local/share/ca-certificates:ro

Then, run the following command in the container and go back to job start step.

Code Block
update-ca-certificates

...

----------------------------------------------------------------------------------------------------------------
apache2           /root/entrypoint.sh              Up         5500/tcp, 5555/tcp, 5556/tcp, 0.0.0.0:80->80/tcp  
lava-dispatcher   /root/entrypoint.sh              Exit 255                                                     
lava-logs         /root/entrypoint.sh              Up         5500/tcp, 0.0.0.0:5555->5555/tcp, 5556/tcp, 80/tcp
lava-master       /root/entrypoint.sh              Up         5500/tcp, 5555/tcp, 0.0.0.0:5556->5556/tcp, 80/tcp
lava-publisher    /root/entrypoint.sh              Up         0.0.0.0:5500->5500/tcp, 5555/tcp, 5556/tcp, 80/tcp
lava-ser2net      /bin/sh -c echo -n "Starti ...   Exit 255                                                     
lava-server       /root/entrypoint.sh              Up         5500/tcp, 5555/tcp, 5556/tcp, 80/tcp              
lava-server-db    docker-entrypoint.sh postgres    Up         5432/tcp

As can be seen, lava-dispatcher and lava-ser2net containers didn't start up successfully. This usually can be rectified by running "docker-compose up". To bring up the system into a fully functional state, udev forwarder must be started manually too.