As seen on diginomica passim, the music industry has a complicated relationship with data and AI. While there’s justifiable concern that emerging technology can be used to shortcut the creative process and support copyright theft to train AI models, companies in the sector, just like any other, must find novel ways to exploit new data-enabled models to gain a competitive advantage.
It’s in this challenging environment that Create Music Group is using data and AI to boost its behind-the-scenes operational processes. The Los Angeles–based music technology specialist has created a data platform that helps artists and labels understand how each music stream translates into revenue.
As a key element of this platform, the company uses Astronomer’s AI and orchestration capabilities to manage over 600 data pipelines to produce real-time insight. In a complex world of data feeds and fast-changing trends.
Create began working with Astronomer in 2021. Previously, the company relied on cron jobs and Google Cloud Composer tools that struggled to keep up with the pace of change. Miko Chen, Lead Data Engineer at Create, recognized that Create needed a better way of working, and she saw an opportunity when the company updated Apache Airflow, the open-source platform for managing complex data pipelines. A move from Airflow 1 to Airflow 2 provided a window of opportunity – and into this opening came Astronomer’s managed Airflow service, Astro:
That was a fundamental change. I wanted to work with a big contributor to the Airflow community to support our future developments. After a detailed technical discussion with the Astronomer team, they resolved all my concerns.
Today, Astro sits at the heart of Create’s operational processes, with significant plans to do more, including using the technology to support a new enterprise platform and the company’s innovative use of AI models.
Delivering results
Create has used Astro to integrate its BigQuery and Google Cloud Storage technologies and APIs from Spotify, YouTube, Apple Music, and Amazon Music into a single orchestration layer that creates data pipelines for operational activities, such as analytics, financial forecasting, and acquisition modelling for label deals.
Chen estimates that this setup means her data team can now develop new data pipelines 60% faster than before. This capability allows her team to deliver iterative, scalable services. By using Astro and working with Astronomer’s technologists, Chen’s team spends about 50% less time on infrastructure management:
We can work closely with Astronomer to run a migration, such as when we recently migrated from Airflow 2 to Airflow 3. Their team helped reduce the time we spent on infrastructure management. This capability means my team can continue to focus on bringing in the business-critical data for the wider business.
The data team now has a reliable orchestration layer that powers internal decision-making processes. Chen gives the example of managing music ownership conflict issues, which could mean revenue is sent to the wrong distributor. Astro is helping the company to hone its processes effectively:
With the data we provide, we can automatically help people in the business to highlight the assets that may have ownership issues, so they can spend time on value-adding tasks instead of trying to identify those problematic assets within our system.
The high-quality data feeds mean Create can also provide timely insights to labels and artists that use its platform. Using Astro, Chen says the company gathers play-data information in a single dashboard:
When artists log in, they can check how a particular song is performing on YouTube, Spotify, or any other popular platforms. As well as performance insights, they can also get their revenue information from our system.
Supporting change
The move to Airflow 3 will help the organization to ensure that human-in-the-loop operators and real-time orchestration keep its data platform refreshed, reckons Chen.
The company is building a new enterprise platform internally, and Chen says these efforts are likely to include rebuilding the accounting system, which relies on data ingestion. She’s suggesting to stakeholders that Astronomer should play a critical role in these developments because of its data management capabilities.
Create is also exploring how AI can improve its services, such as data-enabled marketing campaigns and suggestions for the best tour locations based on streaming data. Right now, this bespoke technology is at a proof-of-concept stage, but Chen expects Astro to play a role in long-term developments:
I'm confident that, when it comes to production rollout, we will rely on Astronomer to orchestrate the approach because it will require the upstream data we collect.
Chen says her team plans to use Astro to collate all the information the company needs to power its downstream processes, including accounting, distribution, contracts, and other data-related processes:
Astronomer is moving from being a tool for data engineers to a place where the entire organization will rely on the service to power the data we need.