Managing Displays Over USB

Lessons learned from sending raw bytes of data over USB to a “black box”

At Intersection, we have a variety of smart city technology and digital out-of-home products. Our fleet of 4000+ digital displays are deployed across 30+ cities, and the ability to manage and monitor these displays remotely is critical.

Intersection digital displays

When a display is undergoing hardware maintenance, we may turn the display off.

If a display is too bright, we may decrease its illuminance level.

To verify touch screen functionality, we may fetch a display’s touch logs or simulate a touch event.

These are just a few examples of the many actions we take to gain insight into the status of a display or to manage it.

Luckily for our Service Desk team, we expose such remote capabilities through one platform. However, the displays are manufactured by a variety of vendors, and different vendors provide different tools or APIs to interact with their devices.

So, what if a vendor-provided API doesn’t integrate well with our internal systems?

Decisions, decisions…

Recently, we decided not to use one of our vendor’s APIs. Without going into too much detail, we needed to have our own management board inside the structure. Because of this and our vendor’s API implementation, the integration felt too heavy as it added unnecessary layers/steps between our systems and the actual display.

Our vendor shared documentation of how their API interacted with the display’s management board directly: over a USB connection, using the Human Interface Device (HID) Protocol.

And so, our architecture looked like this, where our solution was to send raw bytes of data to a “black box.”

Though we encountered a variety of challenges, we learned a handful of lessons along the way.

Breaking the problem down to byte-sized chunks

USB 101

The documentation our vendor provided (reasonably) did not explain HID protocol basics. We found this resource Device Class Definition for Human Interface Devices (HID) was critical in familiarizing ourselves with the terminology we needed to decipher the vendor-specific protocol.

Here was our terminology “cheat sheet”:

Device / Interface / Endpoint- the different levels of descriptors, used to interact with the device

Input / Output- the direction the data flows from the perspective of the host (i.e. from Intersection’s management board to the vendor’s management board)

Endpoint In / Endpoint Out- the descriptors corresponding to the direction data is being sent/received (i.e. Input vs Output)

Get Report / Set Report- the names of the actions associated with receiving/sending data in the HID protocol

So, to summarize-

This is a simplified overview of the process we needed to implement to interact with the vendor’s management board:

1. Find and identify our device, product, and vendor IDs
2. Claim the interface of our device
3. Get the device’s endpoint addresses
4. Use our vendor’s specification to construct the commands to send or parsers for data to read
5. Write data to Endpoint Out or read data from Endpoint In

Once we found the device information specified in Step 1 (see below for the commands), we were able to implement the rest using the PyUSB, an easy-to-use Python library with solid documentation, a tutorial, and debug logging (see below).

PYUSB_DEBUG='debug' python3 [filename]

Disclaimer: Our management board runs on Linux. Many of the tools mentioned throughout this post are Linux-specific.

Step 1: Find and identify the device, product, and vendor IDs

First use dmesg to search for the device ID that corresponds to your vendor’s product, in the examples below the returned ID of the device is 1111:2222

dmesg | grep -i [device or vendor name]

Output:

usb 1–1: Product: [Name of Vendor’s Management Board]
hid-generic 0011:1111:2222.0001: hiddev0,hidraw0: USB HID v1.10 > Device [Name of Vendor’s Management Board] on usb-0000:00:14.0–1

Now, you can use lsusb to get more information about the device (the below command outputs all devices your machine is connected to, but the below snippet only shows the output for one device as an example).

lsusb -v

Output:

Bus 001 Device 011: ID 1111:2222
Device Descriptor:
bLength 18
bDescriptorType 1
bcdUSB 1.10
bDeviceClass 0 (Defined at Interface level)
bDeviceSubClass 0
bDeviceProtocol 0bMaxPacketSize0 64
idVendor 0x1111
idProduct 0x2222
bcdDevice 1.00
iManufacturer 1
iProduct 2
iSerial 3
bNumConfigurations 1
Configuration Descriptor:
  bLength 9
  bDescriptorType 2
  wTotalLength 41
  bNumInterfaces 1
  bConfigurationValue 1
  iConfiguration 0
  bmAttributes 0xc0
  Self Powered
  MaxPower 100mA
  Interface Descriptor:
    bLength 9
    bDescriptorType 4
    bInterfaceNumber 0
    bAlternateSetting 0
    bNumEndpoints 2
    bInterfaceClass 3 Human Interface Device
    bInterfaceSubClass 0 No Subclass
    bInterfaceProtocol 0 None
    iInterface 0
      HID Device Descriptor:
        bLength 9
        bDescriptorType 33
        bcdHID 1.10
        bCountryCode 0 Not supported
        bNumDescriptors 1
        bDescriptorType 34 Report
        wDescriptorLength 107
        Report Descriptors:
        ** UNAVAILABLE **
    Endpoint Descriptor:
      bLength 7
      bDescriptorType 5
      bEndpointAddress 0x81 EP 1 IN
      bmAttributes 3
        Transfer Type Interrupt
        Synch Type None
      Usage Type Data
      wMaxPacketSize 0x0040 1x 64 bytes
      bInterval 10
    Endpoint Descriptor:
      bLength 7
      bDescriptorType 5
      bEndpointAddress 0x02 EP 2 OUT
      bmAttributes 3
        Transfer Type Interrupt
        Synch Type None
        Usage Type Data
      wMaxPacketSize 0x0040 1x 64 bytes
      bInterval 10

Step 2: Claim the interface of the device

As explained in the LibUSB documentation (a library that PyUSB uses), before performing any I/O operations we need to let the operating system know we will take ownership of the interface to the device by “claiming the interface”.

The hierarchy of the device’s descriptors are: Device -> Configuration -> Interface -> Endpoints

So, you need to access the interface of your device through its parent-descriptors:

1. Find the device
2. Find the configuration
3. Find the interface
4. Claim the interface

To keep things simple, we’ve extracted out error handling in the example below. We are able to assume that the configuration and interface we need are at index 0, due to bNumDescriptors found in the lsusb output above.

Checking if the kernel driver is attached may seem like an unnecessary step, but was actually the solution to the “resource busy” errors we worked to debug for hours. If you know that your device isn’t being claimed by anything else and you see errors related to a broken pipe or resources being unavailable, explicitly detach the kernel driver. (Note: to do this step, you may have to be a root user.)

To confirm that our error was occurring due to us trying to claim the interface when it wasn’t available, we used:

1. pdb to trace/step through the PyUSB functions, and
2. strace to dig deeper into which resources were unavailable, where we searched the strace output for EBUSY.

strace [command to run your script] > outfile 2>&1

Step 3: Get the device’s endpoint addresses

You may have noticed that the endpoint addresses are listed in the lsusb output. However, we chose to get the endpoint addresses dynamically, from our interface object so our implementation was not tied to any specific device.

Note: PY_USB provides a utility function that uses a mask with an endpoint address to find the direction of the address.

Step 4: Using the vendor’s specification, construct the commands you want to send or parsers for data you want to read

Writing data

Depending on the endianness of 1) your board’s CPU architecture, 2) the device you are connected to, or 3) your protocol’s endianness, you may have to reverse the bytes you are reading from / sending to your device.

Let’s say the commands we need to send are 4 bytes long.

For example, sending bytes from a big-endian machine to a little endian machine would have to be reversed.

To construct your commands you could:

Write your commands in the format of your device’s architecture

display_off = 0x40302010 # little endian
display_on = 0x44332211 # little endian

Write your commands as lists and reverse them

display_off = [0x10, 0x20, 0x30, 0x40]
display_off.reverse()
display_on = [0x11, 0x22, 0x33, 0x44]
display_on.reverse()

Reading data

Depending on your vendor’s protocol, this step may be different for you. We needed to parse a constant stream of data being sent to Endpoint In for the data we wanted.

Let’s say our packet length is 8 bytes and is a repeated set of:

• A type that corresponds to a parameter you are interested in (as one byte), and
• A value that corresponds to the parameter (as one byte)

Our parse() function below would parse the packet of data for the type, and return the value that succeeded the type.

Example usage:

cpu_temp_header = 0x10
packet = FUNCTION_TO_READ_DATA_HERE
result = parse(packet, cpu_temp_header)

Step 5: Write data to Endpoint Out or read data from Endpoint In

Voila! At this point you’ve done 99.99% of the work and the rest is very simple.

Using the endpoints from Step 3,

Write to Endpoint Out with:

def write_command(endpoint_out, byte_array):
  endpoint_out.write(byte_array)

Read from Endpoint In with:

PACKET_LEN = 8
def read_parameter(endpoint_in, header):
  packet = endpoint_in.read(PACKET_LEN)
  return parse(packet, header)

Putting it all together we have-

This project was I/Opening

Here on the Smart Cities team, the majority of our products are applications with a frontend written in JavaScript/React, and a backend service written in one of Python, Go, Scala, or Java. Circumventing a hardware vendor’s API was a unique challenge because it exposed us to a level of the stack we don’t frequently work touch. Furthermore, this project helped us to better understand what happens under the hood when we use USB devices such as the keyboard I am typing this blog post with!