product design
|
Engineering

Orcasound

Design and develop an application for listening to and exploring bioacoustic audio data captured from Salish Sea hydrophones.
Timeline
9 months (in progress)
Team
Scott, Product Owner / Marine Bioacoustician
Dave, Marine Ecologist
Team
Paul, Lead Developer
Skander, Senior Developer

Context

Orcas in the Salish Sea are in dangerous decline, with scientists currently tracking fewer than 100 individuals. Human-generated marine noise is a major factor.

J Pod (Southern Resident Killer Whales) vocalizations in a kelp forest off San Juan Island

Same location 12 minutes later as a powerboat passes through

Audio is one of the best ways to monitor for the presence of orcas, combined with community sightings networks. Underwater microphones called hydrophones can pick up their calls, clicks, whistles from up to 5 miles away. Every species and pod has a distinctive dialect that scientists can use to identify movements and behaviors.

Challenge

Research, strategize and design an interactive web app experience that:

  • Allows scientists and members of the public to monitor hydrophone streams, and send out real-time alerts when endangered orca species are present
  • Integrates AI audio detection models alongside community listeners to capture as many orca sounds as possible
  • Publishes orca sounds captured from the wild as a searchable and shareable library of data-rich content to train listeners, enable analysis of historical data, and drive site traffic toward conservation calls to action

Research & Insights

I began by helping the team analyze and interpret user analytics. The quantitative results surprised some team members, and shifted our collective understanding of the project. My results demonstrated the intermittent nature of traffic, with visitors peaking dramatically after a notification or social media post of live whale activity, and returning to a minimal level otherwise.

In essence, the vast majority of users do not assist in detecting orcas, but instead use the app to "tune in" when they can hear the audio. This caused new questions to arise:

  • How do we encourage more people to listen to the live streams as 'first detectors'?
  • During long quiet periods, how do we keep community scientists engaged and productive?
  • How do we leverage a growing audience of orca sound listeners to promote conservation goals?

Moderators vs. AI models

In answering some of these questions, we also needed to consider the role of our machine learning team –– a group of civic tech hackers working out of Microsoft to develop an audio interpretation model called OrcaHello. The model is trained to mimic the judgement of PhD scientists, and seeks to ease our reliance on human expertise for accurate around-the-clock whale monitoring coverage.

At the 2024 Microsoft Hackathon, we had a chance to work with the AI team for three days to work on backlog items. Project scientist Dave brought along a playlist that got us in the zone.

I took the opportunity to do a retrospective analysis comparing the performance of the AI model against Orcasound's community of listeners, with surprising results. While the human community and AI model made about the same number of detections, the humans were actually more accurate.

We hypothesized the reason for this is that humans have an unfair advantage over the AI.

Where the machine learning model is trained only on the audio signal coming from the hydrophone, which is inherently noisy and unpredictable, humans have other data sources to turn to, usually whale sightings channels on social media such as Facebook, BlueSky, and WhatsApp.

Map of an orca sighting / listening event on May 22, 2024, from a combination of social media sightings, webcams, and hydrophones as well as ship traffic data

Research Takeaways

From this process, we made some key conclusions that were impactful in how we approached the product design:

  • The audience for listening to orca vocalizations is large and enthusiastic. The audio offers a unique experience compared with other channels.‍
  • Listeners tune in not just to hear the audio but also to read reports and comments about what they are hearing.
  • Spectrograms, maps, and contextual data work well as interpretive aids.

Designs

With a firm direction and objectives, I began the design process with wireframes to capture necessary tasks and visualize opportunities.

Wireframing the moderator interface

  • Moderators have 24-7 responsibility for sending real-time alerts, so mobile access is important.
  • Web-based spectrogram editor enhances users’ ability to find and annotate whale sounds due to their distinctive shape.
  • Moderators need tools to be able to process, tag, and annotate audio events to generate a useful data set for analysis.

Wireframing the listener interface

  • 66% of listeners are on mobile, frequently coming from social media channels
  • The interface listeners need to fully experience the available data is a unique combination of interaction models, including maps, data insights, audio streaming, and user-generated content.
  • Flexible, scalable UI with modular components is needed to build iteratively and account for a growing feature set that we may not have imagined yet.

Testing

In testing the interactive wireframes, we heard positive feedback from participants but found that we needed to work with real data to properly understand how well the concepts were landing with users. To create true-data prototypes, we discovered ways to restructure the API to more easily build certain features, that would benefit both listeners and moderators.

Navigation and layout

  • Desktop provides more screen real estate, mobile has a specific types of interactions like bottom drawer navigation
  • Solving for the ability to see both visual and audio data sources, real-time and historical

Data-driven content experience

  • As listeners learn to interpret the soundscape, they become more accurate detectors and can call out egregious vessels
  • Report verification, keyword tags, and comments made by moderators add a rich layer of meta data to make the historical record searchable with the ability to compose playlists of certain species and pods
  • Recordings remain open to annotation, and users can pull out short clips to share

Outcomes

Designs are currently in development (April 2025).

  • Working alongside dev team as front-end lead building React components
  • Connecting with multiple APIs to enhance data
  • Agile approach to feature development