The human chooses how much of the programming to take part in. All programming decisions can be delegated to agents, or the human can be closely involved, by turning knobs, editing snapshots, placing events, and building and testing devices by hand in Max or Pd. Any balance in between is also possible. The human also listens, inspects the files and the experience in the real world, and tells the agents whether their decisions were correct. Not knowing in advance what an agent will do is part of the interaction model.
With agents, the human can work in programming territory that would otherwise be unknown, with assistance and guidance. The guidance can cover the technology and its context, such as the work and heritage of the human programmers. It is on by default and can be turned off in conversation.
The influence of AI on a process that was purely human is where tension lies. For that reason the relation to AI is not fixed by the design. The agent invites the human to specify it while working, the human can change it at any time, and the line is drawn by the user and not by the designer.
The conductor agent does no module-specific work and holds the context relevant to the whole system. Each module also has agents working on tasks that relate to it, including work done on the timeline. Agents cannot hear the result of what they do, so they rely on the human to confirm whether a decision was correct.
In this overview, Nexus takes on the task of whichever agent is relevant to the part being shown.
The conversational layer. Human and agent talk as they would in any chat, here through Telegram, which suits multi-agent work because it supports groups. Each module has its own group, so the context for that instrument collects in one place. Intentions are discussed here, while tasks are carried out through CUE.
Everything the system does is done through CUE, in the terminal, by the human and by agents independently of each other. Agents use the terminal in the same way as humans do, and the scope of possible tasks is kept as small as possible.
The only global object. Sixteen group tracks, one per module, each holding an open-ended number of functions. Sixteen is a deliberate constraint, modelled on the 16 channels of MIDI. A function is something built into the hardware, or built into the device itself in Pd or Max, that can be accessed with a single CC. The timeline sends bank and program changes to up to 16 modules, each addressed by its own MIDI channel, and can run for as long as the work needs.
Track 1 is mapped to the strummer of Module 01. Select an event to send its recall, or select an empty position to create one.
Each module is an independent ALSA client, addressed by MIDI channel. None of them knows about the others, and that independence keeps the system stable. All of it runs headless on a Raspberry Pi 4 under Debian, with persistent TCP connections between the components.
Only Module 01 is modelled with a working controller in this overview. Select Module 01 to see it.
A translation layer between the module and the actual hardware. The controller speaks in control changes. The device converts them into the MIDI messages the instrument needs, for example sysex, using a prepared Max or Pd patch.
A human can prototype a device by hand and inspect the patch before or while it is deployed. The human therefore knows what a device can do and how the controller relates to it, which helps the agent to operate it and helps in creating the controller mapping.
A device exposes functions. A function is something built into the hardware, or built into the device itself in Pd or Max, that can be accessed with a single CC. The CCs of the controller knobs are mapped to functions, so that turning a knob changes the function it is mapped to.
Six strings with four variables each give 24 functions, reached with CC 1 to 24. The variables of each string are the open string note (tuning), the fret number, the velocity and a time delay. Each CC has 128 values, which gives about 3.7 × 1050 possible settings. The knobs of the controller in the module set them, and the sound is synthesised in the browser.
A single place that every agent can use. It is not part of any module, because 16 workbenches would never be used at the same time. It gives agents direct control over MIDI hardware, beyond the standard CONDUCTOR conventions, so that any kind of MIDI message can be sent, including sysex.
The workbench is a more experimental, research-oriented component. It is focused on exploring the hardware and not on work on the piece itself. The exploration can still take place together with the human, and it is where the role of a piece of hardware within CONDUCTOR is decided, in preparation for authoring the device.
Instruments from the 1980s and 1990s, many of which were unused before the project began. They make the sound, and the agents operate them through the modules and devices.
MIDI is a language of small messages. A message can play a note, change a controller value, or recall a stored sound by program change, and system exclusive (sysex) messages carry longer instructions that are specific to one instrument. There are 16 channels, and the system treats this limit as a design principle, which is why the timeline has 16 tracks.
The instruments come from a period in which every device was expected to connect to complex MIDI infrastructure such as Sounddiver, a logic that has since been abandoned, so their MIDI implementations are deep.