Running Dreamview (Apollo 3.0) for pre-Haswell processors

Problem overview

The Apollo docker includes a version of PCL which was built with the FMA instruction set (fused multiply-add). For any Intel processor older than Haswell (and perhaps even some Broadwell processors), this causes an illegal instruction and prevents Dreaview from starting. In previous version of Apollo, this was difficult to problem solve since no error details were provided – in fact, Dreamview would be reported to be running but localhost:8888 could not be accessed. Since 3.0, however, Dreamview reports the following:
dreamview: ERROR (spawn error)
Note: this can probably be caused by many other issues. However, if you are running Haswell or earlier, the solution provided here is likely to work. You can use gdb to troubleshoot spawn errors.

The solution

You have to build PCL within the docker to overcome this problem.

cd /apollo
git clone
git checkout -b pcl-custom

Now you need to add the following to /apollo/pcl/CMakeLists.txt:

  • Below the line set(CMAKE_CONFIGURATION_TYPES "Debug;Release" CACHE STRING "possible configurations" FORCE), add the following:
    set (CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -std=gnu++11")
    message("Build with c++11 support")
else ()
endif ()

Build PCL with the default options.

cd /apollo/pcl
mkdir build
cd build
cmake ..

Replace the current PCL libraries with the new ones you just made (you can backup first).

sudo mkdir -p /usr/local/lib/pcl.backup
sudo mv /usr/local/lib/libpcl* /usr/local/lib/pcl.backup
sudo cp -a lib/* /usr/local/lib/
sudo ldconfig

Now you have to re-build Apollo.

cd /apollo
./ clean
./ build_gpu

And hopfully you can now start and access Dreamview with the usual ./scripts/

Apollo messages

This post provides an overview of the messages used by Apollo, as captured from the demo rosbag for testing the perception module.

To do:

  • Investigate protobuf message definitions for variable types
  • Investiage contents of standard ROS type messages

Standard message types


These messages use the standard ROS message format for their respective types.

Custom message types

Apollo uses a custom version of ROS which replaces the msg message description language with a Real Time Publish Subscribe and Google protobuf messaging protocol for publishing and receiving the following messages.


Messages using these types cannot be interpreted using rosbag tools as only the header is compatible. However, their content can be dumped as text using read_messages().

Message topics

The demo rosbag contains the following topics:

topics: /apollo/canbus/chassis 5851 msgs : pb_msgs/Chassis
/apollo/localization/pose 5856 msgs : pb_msgs/LocalizationEstimate
/apollo/sensor/camera/traffic/image_long 471 msgs : sensor_msgs/Image
/apollo/sensor/camera/traffic/image_short 469 msgs : sensor_msgs/Image
/apollo/sensor/conti_radar 789 msgs : pb_msgs/ContiRadar
/apollo/sensor/gnss/best_pose 59 msgs : pb_msgs/GnssBestPose
/apollo/sensor/gnss/corrected_imu 5838 msgs : pb_msgs/Imu
/apollo/sensor/gnss/gnss_status 59 msgs : pb_msgs/GnssStatus
/apollo/sensor/gnss/imu 11630 msgs : pb_msgs/Imu
/apollo/sensor/gnss/ins_stat 118 msgs : pb_msgs/InsStat
/apollo/sensor/gnss/odometry 5848 msgs : pb_msgs/Gps
/apollo/sensor/gnss/rtk_eph 49 msgs : pb_msgs/GnssEphemeris
/apollo/sensor/gnss/rtk_obs 352 msgs : pb_msgs/EpochObservation
/apollo/sensor/velodyne64/compensator/PointCloud2 587 msgs : sensor_msgs/PointCloud2
/tf 11740 msgs : tf2_msgs/TFMessage
/tf_static 1 msg : tf2_msgs/TFMessage

Example messages

All custom messages were read using the following script, within the Apollo docker:

import rosbag
import std_msgs

from std_msgs.msg import String

bag = rosbag.Bag('../apollo_data/2018-01-03-19-37-16.bag', 'r')

read_topic = '/apollo/canbus/chassis'

counter = 0

for topic, msg, t in bag.read_messages():
    if topic == read_topic:
        out_file = 'messages/' + str(counter) + '.txt'
        f = file(out_file, 'w')
        counter = counter + 1

Note that this will not work outside of the Apollo docker – the files will be empty.


Of custom protobuf type pb_msgs/Chassis.

engine_started: true
engine_rpm: 0.0
speed_mps: 0.261111110449
odometer_m: 0.0
fuel_range_m: 0
throttle_percentage: 14.9950408936
brake_percentage: 20.5966281891
steering_percentage: 4.36170196533
steering_torque_nm: -0.75
parking_brake: false
error_code: NO_ERROR
gear_location: GEAR_DRIVE
header {
    timestamp_sec: 1513807824.58
    module_name: "chassis"
    sequence_num: 96620
signal {
    turn_signal: TURN_NONE
    horn: false
chassis_gps {
    latitude: 37.416912
    longitude: -122.016053333
    gps_valid: true
    year: 17
    month: 12
    day: 20
    hours: 22
    minutes: 10
    seconds: 23
    compass_direction: 270.0
    pdop: 0.8
    is_gps_fault: false
    is_inferred: false
    altitude: -42.5
    heading: 285.75
    hdop: 0.4
    vdop: 0.6
    quality: FIX_3D
    num_satellites: 20
    gps_speed: 0.89408


Of custom protobuf type pb_msgs/LocalizationEstimate.

header {
timestamp_sec: 1513807826.05
module_name: "localization"
sequence_num: 96460
pose {
    position {
    x: 587068.814494
    y: 4141577.34872
    z: -31.0619329279
    orientation {
        qx: 0.0354450020653
        qy: 0.0137914670665
        qz: -0.608585615062
        qw: -0.792576177036
    linear_velocity {
        x: -1.08479686092
        y: 0.30964034124
        z: -0.00187507107555
    linear_acceleration {
        x: -2.10530393576
        y: 0.553837321635
        z: 0.170289232445
    angular_velocity {
        x: 0.00630864843147
        y: 0.0111583669994
        z: 0.0146261464379
    heading: 2.88124066533
    linear_acceleration_vrf {
        x: -0.0137881469155
        y: 2.15869303259
        z: 0.328470914294
    angular_velocity_vrf {
        x: 0.0120972352232
        y: -0.00428235807315
        z: 0.0146133729179
    euler_angles {
        x: 0.0213395674976
        y: 0.0730372234802
        z: -3.40194464185
measurement_time: 1513807826.04


This is a standard sensor_msgs/Image message.


This is a standard sensor_msgs/Image message.


Of custom protobuf type pb_msgs/ContiRadar.

header {
timestamp_sec: 1513807824.52
module_name: "conti_radar"
sequence_num: 12971
contiobs {
header {
timestamp_sec: 1513807824.52
module_name: "conti_radar"
sequence_num: 12971
clusterortrack: false
obstacle_id: 0
longitude_dist: 107.6
lateral_dist: -17.2
longitude_vel: -0.25
lateral_vel: -0.75
rcs: 7.5
dynprop: 1
longitude_dist_rms: 0.371
lateral_dist_rms: 0.616
longitude_vel_rms: 0.371
lateral_vel_rms: 0.616
probexist: 1.0
meas_state: 2
longitude_accel: 0.23
lateral_accel: 0.0
oritation_angle: 0.0
longitude_accel_rms: 0.794
lateral_accel_rms: 0.005
oritation_angle_rms: 1.909
length: 2.8
width: 2.4
obstacle_class: 1
...[99 more `contiobs`]...
`object_list_status {
nof_objects: 100
meas_counter: 8464
interface_version: 0

Of custom protobuf type `pb_msgs/GnssBestPose`.

`header {
timestamp_sec: 1513807825.02
measurement_time: 1197843043.0
sol_status: SOL_COMPUTED
sol_type: NARROW_INT
latitude: 37.4169108497
longitude: -122.016059063
height_msl: 2.09512365051
undulation: -32.0999984741
datum_id: WGS84
latitude_std_dev: 0.0114300707355
longitude_std_dev: 0.00970683153719
height_std_dev: 0.0248824004084
base_station_id: "0"
differential_age: 2.0
solution_age: 0.0
num_sats_tracked: 13
num_sats_in_solution: 13
num_sats_l1: 13
num_sats_multi: 11
extended_solution_status: 33
galileo_beidou_used_mask: 0
gps_glonass_used_mask: 51


Of custom protobuf type pb_msgs/Imu.

This appears to be empty, with just a timestamp header.


Of custom protobuf type pb_msgs/GnssStatus.

header {
timestamp_sec: 1513807826.02
solution_completed: true
solution_status: 0
position_type: 50
num_sats: 14


Of custom protobuf type pb_msgs/Imu.

header {
timestamp_sec: 1513807824.58
measurement_time: 1197843042.56
measurement_span: 0.00499999988824
linear_acceleration {
x: -0.172816216946
y: -0.864528119564
z: 9.75685194135
angular_velocity {
x: -0.000550057197804
y: 0.00203638196634
z: 0.00155888550527


Of custom protobuf type pb_msgs/InsStat.

header {
timestamp_sec: 1513807826.0
ins_status: 3
pos_type: 56


Of custom protobuf type pb_msgs/Gps.

header {
timestamp_sec: 1513807824.58
localization {
position {
x: 587069.287353
y: 4141577.22403
z: -31.0546750054
orientation {
qx: 0.0399296504032
qy: 0.0164343412444
qz: -0.606054063971
qw: -0.79425059458
linear_velocity {
x: -0.231635539107
y: 0.0795322332148
z: 0.00147877897123


Of custom protobuf type pb_msgs/GnssEphemeris.

gnss_type: GPS_SYS
keppler_orbit {
gnss_type: GPS_SYS
sat_prn: 23
gnss_time_type: GPS_TIME
week_num: 1980
af0: -0.000219020526856
af1: 0.0
af2: 0.0
iode: 43.0
deltan: 5.23450375238e-09
m0: -1.97123659751
e: 0.0118623136077
roota: 5153.61836624
toe: 345600.0
toc: 345600.0
cic: 1.30385160446e-07
crc: 266.5
cis: -8.56816768646e-08
crs: -9.28125
cuc: -7.13393092155e-07
cus: 5.16884028912e-06
omega0: 2.20992318653
omega: -2.40973031377
i0: 0.943533454733
omegadot: -8.16355433052e-09
idot: -2.10723063176e-11
accuracy: 2
health: 0
tgd: -2.04890966415e-08
iodc: 43.0
glonass_orbit {
gnss_type: GLO_SYS
slot_prn: 10
gnss_time_type: GLO_TIME
toe: 339318.0
frequency_no: -7
week_num: 1980
week_second_s: 339318.0
tk: 3990.0
clock_offset: 1.41123309731e-05
clock_drift: 9.09494701773e-13
health: 0
position_x: -16965363.2812
position_y: -15829665.5273
position_z: -10698784.1797
velocity_x: -994.89402771
velocity_y: -1087.91637421
velocity_z: 3184.79061127
accelerate_x: -2.79396772385e-06
accelerate_y: -3.72529029846e-06
accelerate_z: -1.86264514923e-06
infor_age: 0.0


Of custom protobuf type pb_msgs/EpochObservation.

receiver_id: 0
gnss_time_type: GPS_TIME
gnss_week: 1980
gnss_second_s: 339042.6
sat_obs_num: 13
sat_obs {
sat_prn: 31
sat_sys: GPS_SYS
band_obs_num: 2
band_obs {
band_id: GPS_L1
frequency_value: 0.0
pseudo_type: CORSE_CODE
pseudo_range: 22104170.8773
carrier_phase: 116158209.251
loss_lock_index: 173
doppler: -2880.74853516
snr: 173.0
band_obs {
band_id: GPS_L2
frequency_value: 0.0
pseudo_type: CORSE_CODE
pseudo_range: 22104174.5591
carrier_phase: 90512897.445
loss_lock_index: 140
doppler: -2244.73510742
snr: 140.0

…[13 more sat_obs]…


This is a standard sensor_msgs/PointCloud2 message.


This is a standard tf2_msgs/TFMessage message. The messages alternate between frame_id: world to child_frame_id: localization transform messages and frame_id: world to child_frame_id: novatel transform messages.


This is a standard tf2_msgs/TFMessage message, for the frame_id: novatel to velodyne64 transform.

Running Apollo 2.0 – GPU

This is a brief guide to getting Apollo 2.0 up and running. It is based on the Apollo README with additional setup for the Perception modules.


  • Ubuntu 16.04 (also works on 14.04).
  • Nvidia GPU. Install the drivers as described here. You don’t need CUDA installed (it’s included in the Apollo docker). On 16.04 you will need a new-ish version – the below is tested using 390.25. The Apollo recommended 275.39 will not work on 16.04,┬ábut will work on 14.04. However, as this requires a newer GCC version that breaks the build system, it is much easier to go straight to the 390.25 driver.

Download code and Docker image

  1. Get the code: git clone
  2. If you don’t have Docker already: ./apollo/docker/scripts/
  3. Then log out and log back in again.
  4. Pull the docker image. The script downloads the docker image (or updates it if already downloaded) and starts the container. cd apollo/ ./docker/scripts/

Install Nvidia graphics drivers in the Docker image

  1. Check which driver you are using (in host) with nvidia-smi.
  2. First off we need to enter the container with root priveledges so we can install the matching graphics drivers. docker exec -it apollo_dev /bin/bash wget***.**/NVIDIA-Linux-x86_64-***.**.run where ***.** is the driver version running on your host system. Note: Disregard the Apollo instructions to upgrade to GCC 4.9. Not only is it not necessary with newer versions of the Nvidia drivers, but it will make the build fail. Stick with the GCC version of 4.8.4 which comes in the Docker image.
  3. Now install the drivers: chmod +x NVIDIA-Linux-x86_64-***.**.run ./NVIDIA-Linux-x86_64-***.**.run -a --skip-module-unload --no-kernel-module --no-opengl-files Hit ‘enter’ to go with the default choices where prompted. Once done, check that the driver is working with nvidia-smi.
  4. To create a new image with your changes, check what the container ID of your image is (on the host): docker ps -l
  5. Use the resulting container ID with the following command to create a new image (on the host): docker commit CONTAINER_ID apolloauto/apollo:NEW_DOCKER_IMAGE_TAG where CONTAINER_ID is the container ID you found before, and NEW_DOCKER_IMAGE_TAG is the name you choose for your Apollo GPU image.

Build Apollo in your new Docker image

  1. To get into your new docker image, use the following: ./docker/scripts/ -l -t NEW_DOCKER_IMAGE_TAG ./docker/scripts/
  2. Now you should be able to build the GPU version of Apollo: ./ clean ./ build_gpu

Run Apollo!

  1. From within the docker image, start Apollo: scripts/
  2. Check that Dreamview is running at http://localhost:8888.
  3. Set up in Dreamview by selecting the setup mode, vehicle, and map in the top right. For the sample data rosbag, select “Standard”, “Mkz8” and “Sunnyvale Big Loop”.
  4. Start the rosbag in the docker container with rosbag play path/to/rosbag.bag.
  5. Once you see the vehicle moving in Dreamview, pause the rosbag with the space bar.
  6. Wait a few seconds for the perception, prediction and traffic light modules to load.
  7. Resume playing the rosbag with the spacebar.

Once the rosbag playing is complete, to play it again you have to first shutdown with scripts/ stop and then repeat the above from step 1 (otherwise the time discrepancy stops the modules from working).

Building Apollo outside of Docker

WARNING – this will probably mess up your system.
At some point I had an unrelated crash so I restarted my computer and now it is stuck in a boot loop. So don’t attempt this until I have updated with a safe version!

I will probably try this again in a VM, assuming graphics drivers will work. Apollo is extremely fussy about library versions so be warned!

Current status: Bricked my OS, but at last attempt built until hitting some ROS dependency issues, and some linker problems probably caused by Apollo wanting some archaic glibc version..

You must not have Boost or PCL installed before attempting this, as Apollo requires specific versions. You’ll also need to make sure you have the correct version of GCC (4.8) and probably glibc (TODO).

sudo apt-get install gcc-4.8 g++-4.8 g++-4.8-multilib gcc-4.8-multilib
sudo update-alternatives --install /usr/bin/g++ g++ /usr/bin/g++-4.8 50
sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-4.8 50

Now you need to build Boost.

tar -zxvf boost_1_54_0.tar.gz
cd boost_1_54_0

You will also need to modify the Boost source code to get the threading library to compile with the old GCC version – see here:
Then you can build boost as normal.

./ --prefix=/usr/local
sudo ./b2 --with=all -j 4

Now get some dependencies:

sudo apt-get install -y build-essential apt-utils curl debconf-utils doxygen lcov libcurl4-openssl-dev libfreetype6-dev lsof python-pip python-matplotlib python-scipy python-software-properties realpath software-properties-common unzip vim nano wget zip cppcheck libgtest-dev git bc apt-transport-https shellcheck cmake gdb psmisc python-empy librosconsole0d librosconsole-dev libtf-conversions0d
sudo add-apt-repository ppa:webupd8team/java
sudo apt-get install -y oracle-java8-installer

Install Bazel (again, very fussy with which version):

dpkg -i bazel_0.5.3-linux-x86_64.deb
sudo apt-get install -f

At this point I did some cleanup.

sudo apt-get clean autoclean
sudo apt-get autoremove -y
sudo rm -fr /var/lib/apt/lists/*

You have to build a specific version of protobuf. Get yourself a coffee ready:

tar xzf protobuf-cpp-3.3.0.tar.gz
cd protobuf-3.3.0/
./configure --prefix=/usr
sudo make install
sudo chmod 755 /usr/bin/protoc

Moving on to install node JS (again, a specific version required here):

tar xzf v2.1.0.tar.gz
cd n-2.1.0/
sudo make install
sudo n 8.0.0

Make up an empty file called py27_requirements.txt and insert the following:

protobuf == 3.1
requests >= 2.18

Now install the python corresponding bits and pieces for Python:

pip install -r py27_requirements.txt

I think the following might have to be done earlier, and might also get the wrong version of ROS given current build errors. Investigation needed (TODO):

curl -sS | sudo apt-key add -
echo "deb stable main" | sudo tee /etc/apt/sources.list.d/yarn.list
sudo apt-get update && sudo apt-get install -y yarn

sudo sh -c 'echo "deb $(lsb_release -sc) main" > /etc/apt/sources.list.d/ros-latest.list'
sudo apt-key adv --keyserver hkp:// --recv-key 421C365BD9FF1F717815A3895523BAEEB01FA116

And some more pesky libraries:

sudo apt-get install -y libbz2-dev libconsole-bridge-dev liblog4cxx10-dev libeigen3-dev liblz4-dev libpoco-dev libproj-dev libtinyxml-dev libyaml-cpp-dev sip-dev uuid-dev zlib1g-dev

sudo apt-get install -y libatlas-base-dev libflann-dev libhdf5-serial-dev libicu-dev libleveldb-dev liblmdb-dev libopencv-dev libopenni-dev libqhull-dev libsnappy-dev libvtk5-dev libvtk5-qt4-dev mpi-default-dev

sudo apt-get update && apt-get install -y --force-yes libglfw3 libglfw3-dev freeglut3-dev

You have to build glew to get the right version:

cd glew-2.0.0/
sudo make install

Same kind of thing for PCL:

tar xzf pcl-1.7.2.tar.gz
cd pcl-pcl-1.7.2 && mkdir build && cd build
cmake ..
sudo make install

And the big bad CUDA (you need CUDA 8.0):

sudo apt-get install -y libgflags-dev libgoogle-glog-dev
sudo dpkg -i cuda-repo-ubuntu1604-9-0-local_9.0.176-1_amd64-deb
sudo apt-key add /var/cuda-repo-9-0-local/
sudo apt-get update
sudo apt-get install -y cuda-8-0

Next up is Caffe:

cd caffe-rcg/
cp Makefile.config.example Makefile.config

In the file Makefile.config you have to uncomment the line WITH_PYTHON_LAYER := 1 before continuing.

sudo ln -s
sudo ln -s
cd python
for req in $(cat requirements.txt); do pip install --user $req; done
make all
make test
make runtest
make distribute

Now you can get to the actual Apollo part.

cd apollo-master
./ build-gpu #will fail
source bazel-apollo/external/ros/setup.bash
./ build-gpu

You might need to grab the docker image before doing the Apollo build – I will test if it works without. I think this is because it grabs a special, Apollo-specific version of ROS. You can see why they recommend you keep within the Docker!