ZoomOSC Overview

Control Zoom meeting using a media server like Isadora or QLab

What is ZoomOSC?​

ZoomOSC is an expanded version of the popular real-time conference application that provides a custom OSC designed to make it compatible with media servers. This integration dramatically increases the capabilities of Zoom for live performance, and it does so by building on existing media server pipelines that digital media designers are already familiar with, creating a powerful compositing workflow that can be up and running immediately.


By using ZoomOSC, you can use your favorite media server to control Zoom user-interface functions, such as pin, spotlight, mute/unmute, video on/off, etc. This allows you to construct cues or non-linear control surfaces to operate Zoom like a virtual camera switcher. Because the OSC triggers are sent from media servers like Isadora and QLab, all of the video effects can fire in lock-step with Zoom. The media server can also react to changes within Zoom so that the video programming can react automatically.


In conjunction with creative virtual background designs, you can control the arrangement of the gallery view to create continuous images, composite multiple actors into a single frame, or seamlessly mix live and pre-recorded content. The artistic possibilities are extensive!

 

ZoomOSC is the linchpin in a video workflow that elevates your Zoom performance from a static view of your cast into a dynamic and visually complex production.

Summary of Features

Use OSC to set the spotlight, pin, video off, mute, etc.
•Use Qlab or Isadora cuelists to program your virtual show.
•Easily tools for building reactive media server patches that can respond to the live decisions of designers and directors; add interactivity and improvisation.
•Use ZoomOSC as a video switcher for your performance
•Pair ZoomOSC with the media server to create PIPs, lower thirds, and artistic graphics for your broadcast event.
•Capture ZoomOSC window and broadcast it to your audience using VMix or OBS, ensuring all your audience members have the same view.

Technical Details and Setup

ZoomOSC is a free utility created by Liminal to bring automation and enhanced control to the Zoom platform for use in live entertainment and broadcast. At its core, it is a modified version of the Zoom client for Windows that adds a bidirectional OSC engine, which allows you to send OSC commands to control different functions in the Zoom meeting, such as spotlighting or muting, while also receiving valuable information about Zoom participants, such as the order of people in Gallery View.

 

ZoomOSC has been the technical solution behind most of the critically acclaimed live virtual productions since March, including recent productions of Mad Forest and Tartuffe, and has been at the disposal of designers whose work has been featured in The New York Times and The New Yorker five times in the past two months.

 

What does ZoomOSC NOT do?

 

ZoomOSC is less of a product and more of a tool. While it adds incredible capabilities for enhanced control of Zoom, it is not an end-to-end virtual production solution (that solution is StreamWeaver by the way, go check it out here). Zoom will still compress video, gate audio, and require the technical participation of performers. On the technical side, ZoomOSC is a linchpin in a larger production workflow that requires a media server and/ or broadcast encoder.

 

The ZoomOSC Workflow:

 

ZoomOSC can be used for both serialized (cued) shows or for non-linear control surfaces. For example, Mad Forest had four instances of ZoomOSC in use for the production: one for the video operator so that they could alter the spotlight or Gallery / Active view in step with video effects from the Isadora programming, one for a virtual camera operator who used a non-linear control surface (a small Isadora patch that translated keyboard commands into OSC packets), another non-linear control surface for director Ashley Tata to live mix an improvised portion of the show, and a third instance for backup operation should any of the other operators lose power or internet.

 

From the perspective of video flow, a suggested (but not required) method of working with the software is to have ZoomOSC in OSC communication with a media server like Isadora to send and receive data, and to additionally screen-capture the ZoomOSC window (or windows, if using the multi-display mode of the Zoom interface, which we recommend as it gives a gallery view and a full-screen view simultaneously). After applying video overlays and effects within the media server, the aggregate AV feed can be sent to an encoder like Vmix, potentially using NDI, and then the show’s “program output” can be sent to a remote audience on platforms like YouTube and Vimeo via RTMP, or it can be looped back into the Zoom meeting for a program monitor within the call.

 

 

Getting Started with ZoomOSC:

 

Run the ZoomOSC installer and install the program to a location on your Windows computer, such as Documents, that does not require administrator privileges (for simplicity). When you run ZoomOSC.exe , you will see a console and the Zoom interface. The console contains helpful information about the operation of the program that you can use to debug, and the Zoom interface should operate much like the regular Zoom platform, with a few modifications. Most users can use the default domain when prompted. Please leave “Customize UI” unchecked, as this feature at present is experimental and may lead to crashes. You will also be prompted for an SDK Key and Secret, which are required to run unofficial Zoom integrations. These credentials are provided as ZoomOSC License Key.txt in the zip file you downloaded, but should you encounter any trouble with those keys, you can generate your own, so please contact us for further information on how to authorize the application. Finally, you will need to log into your Zoom account. The simplest way to accomplish this is to sign in with an account that is not associated with a “Log in with…” interface, so if you or your organization normally use Sign in with Google, for example. Please create a free Zoom account to use for running this application that does not require that sign in system. You can still use your organization’s premium account, if you have one, to create the meetings and get access to those features, but you should log into the application with this new account. When joining a meeting, always name yourself ZoomOSC, with case sensitivity.

To send commands to ZoomOSC, have your OSC packets target port 8000 on the software loopback device (127.0.0.1) or via the host PC's private IP address (if you are connecting from an external device, like QLab running on a Mac on the network).

 

To use ZoomOSC, you need to store participants into the program’s memory. There are two ways to accomplish this, /zoom/update and /zoom/load. Calling update will cause ZoomOSC to look at all the participants in the Zoom meeting and store them in memory, associating their usernames with a ZoomOSC ID, a placement in a zero-indexed list of participants. Calling load will have ZoomOSC look for performance_config.txt and load its contents into memory. This file is created by calling /zoom/save, which takes the contents of ZoomOSC’s participant memory and stores them into the text file, so if you don’t see the file, try calling update, then save. The idea of this text file is that you can just write in the Zoom usernames of the participants you want to have control of, and you can order the file so that you can have persistent bindings between ZoomOSC IDs and Zoom participants that you can use for multiple rehearsals or performances. Once you have stored the participants into memory, you can call functions on them with their ZoomOSC ID, such as /zoom/spot 1 to spotlight the second person in the ZoomOSC ID list. You can print the current contents of memory to the console with /zoom/list at any time.

 

As you can see, usernames are critical to the operation of ZoomOSC. If your talent changes their username, you will not have control over them again unless you call update or modify your performance_config.txt file and call load. That said, not all productions have the benefit of knowing the names of the Zoom participants ahead of time, particularly for productions where the audience is within the Zoom call. For these deployments, calling update and saving can be an easy way to externalize the participant data to other applications.

 

ZoomOSC automatically tracks information about users in the background, so if someone disconnects from the meeting and then rejoins, there is nothing you need to do as the operator to regain control of them for the purposes of calling commands /zoom/pin. If you want to get creative, you can have your talent swap usernames (or you can do it for them) as part of an improvisational programming scheme. There are enough building blocks to build very powerful control patches!

 

Advanced ZoomOSC Workflows – The Return Channel:

 

Once of the more recent additions to ZoomOSC is the ability to have the applications receive OSC back from ZoomOSC. Presently, ZoomOSC sends OSC packets to port 1234 on the software loopback, 127.0.0.1. It is set up like this to work with Isadora out of the box, and you can use Isadora to forward or split the OSC data to other devices.

 

The most powerful feature of the return channel is the ability to track participants throughout the Gallery View. You can build upon this framework to create an automatic switching engine to simulate iso feeds of the Zoom participants (but at low resolutions. At this point, you should consider StreamWeaver!). To use the “galtrack” system:

 

  1. Have all participants turn off video (if any were on)

  2. If you are using a program monitor, “Hide Self View” and turn off ZoomOSC’s video feed

  3. Load participants into memory with load or update

  4. Arm galtrack with /zoom/galtrack 1

  5. Have participants turn on video one at a time

  6. Listen to /zoomosc/vid/galorder to receive the ZoomOSC IDs of everyone in the gallery view

 

You can also build reactive programming off of things like /zoomosc/sound/off to fire events in your Isadora patch, for example, when such an action occurs. In combination with glatrack, you could build an effect like getting a virtual anvil to drop on whoever is the first to mute themself in the Zoom call (though I am not sure why you would want to do this specific effect!).

3.1 OSC API

(where x is the floating point index of a user in the zero-indexed list)

Commands going into ZoomOSC on port 8000---------------------------------------------------------------

Zoom Global Action Controls

/zoom/spot x --> spot user x

/zoom/unspot x --> unspot user x

/zoom/on x --> request to turn on video for user x

/zoom/off x --> turn off video for user x

/zoom/mute x --> mute audio for user x

/zoom/unmute x --> request to ummute audio for user x

 

Zoom Self Action Controls

/zoom/pin x  --> pin user x

/zoom/unpin x --> unpin user x

/zoom/me/off 1.0 --> turn off ZoomOSC video

/zoom/me/on 1.0 --> turn on ZoomOSC video

/zoom/gallery 1.0 --> switch to Gallery view

/zoom/gallery 0.0 --> switch to Speaker view

 

Memory Functions

/zoom/update 1.0 --> create a list of current participants and store it in memory

/zoom/include 1.0 --> append new participants to memory while ignoring current memory contents

/zoom/load 1.0 --> create a list in memory from the contents of performance_config.txt

/zoom/save 1.0 --> save the contents of memory to performance_config.txt

/zoom/reset 1.0 --> reset the contents of the memory containers

Zoom Gallery Tracking Controls

/zoom/galcount 1.0 --> request a reply with the count of people in call with video on

/zoom/galtrack 1 .0--> arm gallery placement tracker and/or request gallery layout

/zoom/galtrack 0.0 --> disarm gallery placement tracker and delete tracking

Console Debug Functions

/zoom/list 1.0 --> print the list in memory to the console

/zoom/survey 1.0 --> print status information

/zoom/roles 1.0 --> print the Zoom roles of each participant*

Commands coming out of ZoomOSC on port 1234-----------------------------------------------------------

 

Zoom Gallery Tracking Data

/zoomosc/vid/galorder [x1,x2,...] --> provides the gallery view order by sending a list of ids mapping from top left to bottom right

/zoomosc/gallery/count y --> provides the quantity of participants with video on

Zoom Video Tracking Data

/zoomosc/vid/on x --> provides ID of user when they turn on their video

/zoomosc/vid/off x --> provides ID of user when they turn off their video

Zoom Audio Tracking Data

/zoomosc/sound/on x --> provides the ID of a user when they turn on their sound

/zoomosc/sound/off x --> provides the ID of a user when they turn off their sound

*Zoom Role Key:

0    USERROLE_NONE,///<For initialization.
1    USERROLE_HOST,///<Role of the host.
2    USERROLE_COHOST,///<Role of co-host.
3    USERROLE_PANELIST,///<Role of the panelist, valid only in webinar.
4    USERROLE_BREAKOUTROOM_MODERATOR,///<Host role in breakout room.
5    USERROLE_ATTENDEE,///<Role of attendee.

Please remember to send floats from your control surface (we'll add other payloads in the future)!

 

 

Final Thoughts

 

In the right hands, ZoomOSC is an incredibly powerful tool to create content online. Video designers with experience in media servers like Isadora have been able to create very advanced show programming. That being said, ZoomOSC is a technical utility, and not a product, so we are distributing it for free and only monetizing consultations and training sessions designed to help get specific productions or designers who may not be familiar with these workflows out of the box comfortable with using the tool.

Head over to our Production and Technology Consultation page to schedule one-on-one training, or for custom workflow solutions.

 

©2020 by Liminal Entertainment Technologies, LLC