Conversation with Marc Planagumà

Marc Planagumà has extensive experience in scalable platforms and advanced analytics. He drives data-driven business transformations and currently focuses on Adevinta Spain's data platform and strategy. Previously, he held roles at LIDL/Schwarz Group and Zurich Insurances Corp. His background includes work at Telefónica I+D, BDigital, Eurecat, and Berlin Big Data Center.

Let's start this conversation with a trip to the year 2001, which is when you started your studies as a Telecommunications Technical Engineer. What led you to make that decision? Were you clear about your vocation as an engineer?

Since I was young, I have always been interested in technology. I had my first computer at the age of 8, and in 1996, I connected our home computer to the internet myself. This fascination with information technology and networks led me to study Telecommunications in 2000. From the beginning, I was clear about my vocation as an engineer because I have always been fascinated by applied innovation, and during my teenage years, the internet was flourishing, so I was destined to become an engineer in this field.

Once you finished your education, you began your professional career at Telefónica I+D, in the Research area. Tell me the type of projects you participated in, especially the contextual and recommendation services part

At Telefónica I+D, I started as an intern in semantic web projects, working with ontologies and semantic search engines. Later, I became a research engineer in the recommendation systems group. I worked on the design and implementation of RecSys algorithms based on user context, using semantic technologies with which I had started, and graph databases to improve, for example, the personalization of IPTV services from Movistar and other digital media. These were my beginnings as an engineer, starting in the world of data and specifically in its most innovative technologies and solutions, a path that I would never abandon.

In 2010 you started working at the Barcelona Digital Technology Center (BDigital), where you entered the field of “Data and Cloud Computing software architectures”. Nowadays, “cloud paradigm” is fully established, but I guess the situation was very different at that time. What big challenges did you have to face, both on the technical and the cultural side?

At BDigital, I started in the smart cities and mobility research department, and the fact that we were the department with the most data and the greatest need for scalability allowed me to enter the world of cloud at an early stage. One of the big technical challenges was the lack of mature standards and tools for cloud software architecture. However, being in a research center where there was no resistance to change and innovation, I was able to dive into cloud architecture when it was still a greenfield. The problems were more about budget, as research is not sufficiently supported in our country. But once again, I was facing a new technological revolution necessary for the world of data, which is the cloud, and this would be one of the main drivers of the next revolution to come: Big Data. So I had the enormous privilege once again of riding a wave of innovation in the data world of such magnitude that it has continued to the present day.

In 2014 you took the role of “Lead Data Engineer and Researcher on the Big Data Analytics Group” at EureCat & Big Data CoE Barcelona. Before getting into the work you did, I would like you to ask you for a brief description of the purpose and mission of EureCat.

EureCat is a technology center dedicated to research and development in various technological areas, including data analytics and big data. I had the honor of contributing to the birth of this research division at EureCat. I came from BDigital, which, along with other research centers in Catalunya, merged to create EureCat. At BDigital, we had already started our work on distributed computing systems for big data. In this new stage, we first established the Big Data Analytics group and then the Big Data CoE of Barcelona (now CIDAI), along with the creation of the Big Data Congress Barcelona, now known as AI & Big Data Congress BCN.

The mission of EureCat was and still is to foster innovation and transfer technological knowledge to companies to improve their competitiveness and sustainability. In the field of big data, I focused on researching new algorithms for distributed computing systems while also providing applied assistance to companies to develop new methodologies and tools that would allow them to make the most of their data.

You were there for 2 years,  so let me know the scope of the projects you were involved in.

As I mentioned, we founded a new research line on innovation in high scalability architecture, which is what big data needs. In this line, we built the data platform used by the research groups in various branches of data science and machine learning at Eurecat. Additionally, we helped companies to make decisions about the architecture and strategy of their data platforms. I also participated in research projects on distributed processing algorithms, and with this line of work, I was a visiting researcher at the Berlin Big Data Center for 6 months on a research project about the control algorithm of the distributed processing framework behind Apache Flink. All this experience also served to create the Big Data Congress Barcelona in 2015, which continues today as the AI & Big Data Congress BCN. I am very proud to have been there at the beginning. Therefore, those were 2 very productive years, and I had the honor of working from the very beginning on the research of the foundations of the data tools and solutions that are now mainstream.

Right after Eurecat, you join Zurich as Big Data Platform Manager. Which Data Architecture did you choose and what was the main driver of that decision? I am also curious to learn the biggest technical and human challenges you had to overcome 

This was my jump from the world of research to industry, and I started at Zurich Insurance as Platform Manager to build their datalake for the entire EMEA region. Zurich had already attempted to build a datalake externally, but it had not worked, so they created an internal team to do it. Thus, I led the architecture of the data platform, the creation of the engineering team to build and maintain it, and the organization of engineers, data scientists, and other profiles that would use the data platform. Here, I began a role that I have repeated in different times, contexts, and industries: directing the architecture of data platforms but also the architecture of the organization that creates and uses it to make the company more data-driven. In 2016, we created a datalake with a layered architecture (the embryonic version of the lakehouse before it was born) on an on-premise infrastructure, as in the insurance world in 2016, it was considered a significant risk to have your data on external infrastructure. However, I had to lead Zurich's first move to the cloud, convincing management despite their reservations about the cloud, as the scalability of the data platform was unfeasible without dynamic infrastructure. The journey was long and hard, but I succeeded and led the migration of the datalake to Azure and the adaptation of the architecture to a model of elastic infrastructure.

Getting closer to the present, in 2018 you moved to SCRM-Lidl. What was your role there?

At SCRM-Lidl, I was brought in to build, from scratch, the data platform and the data team that would exploit and collect data for Lidl's loyalty program. This loyalty program had started just a year before, in 2017, as a spin-off created by Lidl itself, which previously did not have a loyalty program. They wanted to create it in the most digital and purest form possible, setting it up in a new hub in Barcelona, outside of Lidl's existing technological structure and processes. It started as a pilot program in Spain and quickly expanded to cover all of Spain and then various countries such as Poland, Italy, France, England, and eventually Germany and the rest of Europe. When I joined, I was the first data engineer entering as the director of the area, and by the time I left, we had grown to over 40 data engineers and more than 100 people in total across all data-related profiles. My role, similar to my role at Zurich but with higher maturity levels, was to lead the architecture of the data platform and the organization that would build and use it. 

Tell me more about the Data Platform you were using there 

The data platform at SCRM-Lidl was an evolution of what I had built at Zurich, but already implementing what would be known as the Lakehouse architecture. This concept, originating from a paper by Stanford and the creators of Databricks, was introduced in 2017, and by late 2018, we were already implementing it. In fact, we presented it at a Databricks conference as one of the first large-scale implementations of Lakehouse. It featured a medallion architecture with different layers, both batch and streaming, built on Databricks on Azure, which allowed high levels of automation and autonomy. Data sources were integrated into the platform, and consumers could autonomously access different layers on top of this Lakehouse. Additionally, we had a data warehouse using Snowflake, and tools like Power BI and Qlik on top. This setup allowed us to process more than 2-3 GB per minute, achieving a very large scale of operation.

If I am not mistaken, during your SCRM-Lidl days you were also in charge of Data Governance. What was the main goal of this initiative? And the main challenges? 

Yes, I was also in charge of data governance at SCRM-Lidl. The main goal of this initiative was to embed governance by design, meaning to lead governance from the design of the platform itself so that it would be less dependent on manual processes and tasks, and more automated, aligning with the digital workflows and processes used by the data platform. This approach allowed for a higher degree of automation and ensured that governance was an integral part of the data platform's architecture, reducing the need for separate governance processes and making the overall system more efficient and reliable.

We finally reach the present, so let’s talk about your current role at Adevinta as Data Platform and Governance Director. Which type of Data Architecture have you implemented? 

After four years at LIDL, where I had matured the platform, the team, and the data culture, expanding it beyond the loyalty program to various parts and areas of LIDL, I embarked on a new adventure at Adevinta. My goal was to evolve their data platform, which had been operational for many years but needed an upgrade to better serve all Adevinta sites in Spain in a cross-functional manner. The platform had achieved high levels of scalability and autonomy, but it needed to scale governance. Given my specialization in architectures that scale both autonomy and automated governance, I joined Adevinta to take their organization and platform to the next level.

At Adevinta, we first updated the existing layered architecture to a lakehouse model, improving certain aspects. However, our primary focus was transitioning from a data platform to a data products platform. This meant implementing the four principles of Data Mesh [domain-oriented decentralization, data as a product, self-serve data platform, federated computational governance], particularly emphasizing data as a product. We transformed the platform into a framework for building data products, ensuring each product had common observability and access management, making the data more service-oriented. We also integrated governance by design, automating compliance, security, quality, and governance within the product itself. Thus, the main evolution we achieved was moving from a data platform to a data products platform, leveraging the principles of Data Mesh.

So help me out to understand the differences between data lake and delta lake house

What you are asking about are different names for architectures and models of data platforms. The Data Lake remains the foundation of everything, but it is an outdated model. It is outdated because a Lakehouse mainly solves what made Data Lakes turn into Data Swamps. Understanding a Data Lake as file storage with a distributed search system on top, this remains the base, but the Lakehouse takes this base and adds a structure of different layers with different purposes. These layers allow for different policies, objectives, and control styles to be applied, keeping it more organized. Therefore, the Lakehouse is a complex evolution of the Data Lake, essentially the same concept but evolved.

On the other hand, Data Warehousing is not outdated. It remains very valid, especially for aggregated data, star models, and pre-calculations that feed reports and other analyses. The combination of a Lakehouse (an evolved Data Lake) and Data Warehousing remains essential for building a robust Data Platform.

At Adevinta, we have taken it a step further, surpassing the concept of a Data Platform. A Data Platform could be compared to a monolith in the backend: a large platform that contains all the logic of an analytical application. This remains valid for many companies, but it has scalability limits. To overcome these limits, just as monoliths were broken into microservices in the operational world, in the analytical world, Data Platforms decompose into a mesh of Data Products.

The smallest unit is no longer the Data Platform but the Data Product, which is the analytical equivalent of a microservice. Data is no longer something simply passive; the results, whether KPIs, datasets, recommendation systems, weight matrices, or vectors, are delivered with services that have their own observability, governance, data quality, security, etc. These Data Products can be shared or not, but they are delivered with clear SLAs (Service Level Agreements) and service levels.

In summary, the Data Lake and its evolution, the Lakehouse, remain very valid, as does Data Warehousing. Their combination is still widely used. The next step is to use these tools to create not just a large platform but to provide constructors, frameworks, and work environments that generate small autonomous artifacts: Data Products. The Data Platform team does not build a platform where data is created and consumed internally but creates production lines for Data Products. Thus, those who use the platform produce Data Products, which are not passive entities but active ones, and connecting them forms a mesh: the Data Mesh.

Were you clear since the beginning that Data Mesh was the right solution for Adevinta or  did you evaluate other options? Why did you finally choose Data Mesh?

When I joined Adevinta, Data Mesh was relatively new, but I had already read the book by Zhamak Dehghani, which gave names to many things we had already done at LIDL, but now it was theorized into four principles. Once again, I was implementing a cutting-edge theoretical concept in reality, as I had done with cloud architectures, distributed systems, the lakehouse, and now Data Mesh. The first thing I did was talk to the leadership team to incorporate these principles into our strategy. We focused on embedding the core principles of Data Mesh, such as domain-driven design and automated governance, into the vision, mission, and OKRs of the company.

Data Mesh aligned well with Adevinta's needs. The company required a high degree of autonomy in both data ingestion and consumption, with diverse use cases ranging from BI and data analysis to machine learning and data science. This autonomy posed challenges in ensuring consistent quality, security, and governance across all data activities. Data Mesh addressed these challenges by providing common service layers for observability, security, quality, and compliance, while allowing teams to work autonomously within these frameworks.

However, Data Mesh is not a one-size-fits-all solution. It is more suitable for organizations that require extreme scalability and autonomy. For companies with centralized ingestion or fewer data use cases, Data Mesh might be unnecessarily complex. It is essential to tailor the approach to the specific needs of the organization, and for Adevinta, Data Mesh provided the scalability and governance framework we needed to succeed.

Last time we met in person you emphasized the importance of Data Products and also the value you could get from an “MVP approach”

At Adevinta, we have seen a very positive impact from implementing the concept of Data Products. To define what a Data Product means in our implementation: it is an active artifact that contains the data asset it wants to publish, which can be primary data from a source, an outcome of a model, or a specific dataset. What makes it different and active are its additional features:

• Input Port: Standardizes input data, ensuring consistency and quality from various sources such as databases, APIs, and streaming services.

• Output Ports: Provide standardized interfaces for data consumption, making the data easily accessible and usable in multiple formats like APIs, file exports, or direct database connections.

• Control Port: Manages services like access control, metadata approval, and monitoring, ensuring compliance and quality through auditing and governance mechanisms.

These ports make the data product an active artifact that delivers product-like services. This ensures the data product is discoverable through metadata catalogs and search functionalities, addressable via standard interfaces like APIs, and understandable with clear documentation and metadata. It is trustworthy due to enforced data quality, lineage, and governance policies, and secure with strict access controls and compliance measures. The data product is interoperable, providing standardized formats and interfaces for integration, and natively accessible within common environments and tools. Ultimately, it is valuable on its own, delivering actionable insights and data that meet specific business needs.

Therefore, a data product is essentially a small data platform in itself, a much smaller and more agile building block than a traditional data platform. At Adevinta, what used to be the data platform is now a collection of data product builders using standardized tools and centrally provided services.

To be honest, I love the approach…but at the same time it looks so “minimalistic” that it might create a ”blank page syndrome”. Thoughts on this? You can of course disagree

To address the "blank page syndrome" when dealing with Data Products, at Adevinta we took the existing data journey and structured it into three main categories: an experimentation and discovery framework, ingestion frameworks, and consumption frameworks.

The experimentation framework allows users to quickly prototype and test theories using notebooks and connections to all data layers. Although they can see production data, these prototypes can only be moved to development environments and not to production.

When a prototype needs to move to production, it must go through the other two frameworks: ingestion and consumption. The ingestion framework includes defining data contracts, which are configuration files where the data owner specifies the schema, PII (GDPR) information, service levels, and ownership details. This data is automatically integrated into the platform, becoming a source-aligned data product with standardized input and control ports.

The consumption framework, on the other hand, provides standardized working environments, ensuring that the resulting consumer-aligned data products are discoverable, accessible, operable, and maintained in a consistent manner.

By breaking the process into manageable steps and providing clear frameworks, teams can move quickly from idea to implementation without compromising quality or governance.

Thus, the data journey of the lakehouse and data warehouse, which handle different levels of data elaboration in a progressive manner, transforms into a product laboratory and two markets: one for primary data products and another for elaborated data products.

What profiles are relevant in a Data Platform project and which are the main responsibilities of each of them? Which role should business users have in a Data platform project?

In a Data Platform project, key profiles have evolved to include not only engineers but also governance specialists and business roles. Originally, Data Platforms primarily required Data Engineers, who could be divided into infrastructure-focused profiles and analysis-focused profiles. With the evolution towards a Data Products Platform, this division has become even more refined.

Data Platform Engineers are responsible for the infrastructure of the data platform. They design, build, and maintain systems for data ingestion, storage, and processing, ensuring reliability and scalability through automation and monitoring. They transform this infrastructure into a product that supports data operations.

Data Product Engineers use the tools and infrastructure provided by Data Platform Engineers to create specific Data Products. These products aim to meet business goals and cover business needs while complying with product quality standards, supported by the platform.

The division between Data Platform Engineers and Data Product Engineers is crucial, but so is their feedback loop. Engineers using the infrastructure provide insights into real-world challenges, helping create solutions that meet business needs. This synergy ensures the data platform is effective and adaptable.

Another key point is that as data governance has needed to scale and automate, this role has had to integrate within the scope of data engineers (both platform and product) to impact the platform’s architectures and tools, making governance more automatic and less manual. This role bridges management and business, ensuring data complies with quality, consistency, and legal standards, while managing the semantics and domain of the data, aspects that are closely related to business.

The evolution of Data Platforms has led to the inclusion of more analytical and business profiles, making the platform and the data, products that adds value to the business or even drives business evolution through a data-driven approach.

Let 's also talk about Data Governance. From your perspective, which are the main pillars of a good Data Governance initiative? What tools can be used to efficiently manage it?  
The basic pillars of a good data governance initiative are classic and common, and can be widely found documented in specialized literature such as the Data Management Body of Knowledge (DAMA-DMBOK).

However, I want to emphasize the pillars of automated data governance. To cover these classic aspects of security, compliance, etc., the key is metadata control and its automatic management. This involves:

Metadata Collection and Generation: Ensuring that metadata is collected and generated systematically and automatically.

Metadata Processing and Analysis: Automating the processing of metadata to obtain automatic lineages, access logs, and real-time quality controls.

The focus should be on automating the processing of metadata generated with the data. This includes not only using specific tools like data catalogs, data quality tools, compliance tools, and master data management tools but also impacting the design of solutions, architectures, models, and data workflows. Solutions should consider metadata from the initial design, ensuring that lineages are generated automatically, access control is automated, and quality controls are performed in real-time, etc..

In summary, the pillars of good data governance at scale should focus on metadata automation, ensuring its automatic management and integrating it into the design and tools used to manage data. This will allow for more efficient governance, less dependent on manual processes. This means that every element of the data system, from ingestion systems to final consumption points, should be designed with automated governance in mind. Real-time metadata integration ensures that data is always traceable and auditable, facilitating regulatory compliance and improving data trustworthiness. Moreover, automated governance significantly reduces the manual workload, allowing data teams to focus on higher-value activities such as advanced analytics and innovation.

What’s the role of Data Contracts in Data Governance? Why are they relevant? Who is the owner of a Data Product? 

Data contracts play a crucial role in data products by formalizing agreements between data producers and consumers. They ensure that the exchanged data meets predefined standards and expectations, which is essential for maintaining data quality, consistency, and compliance across the organization.

In our experience, we have used data contracts to define which data is taken from the operational world and exposed for analytical purposes. However, data contracts go much further. Providing a product requires contractualizing that product with your client. Similarly, a product value chain is a chain of contracts, much like logistics or supply chains. For example, a primary product is developed and transformed into other products until it reaches the end consumer. The same applies to data, which can be consumed at various stages of maturity, from raw to highly processed. Therefore, the intermediate stages between data points need to be contractualized, just as retailers contract with wholesalers and wholesalers with producers. This industrial approach to data is about industrializing existing processes for physical products in the digital world of data.

With this experience, my definition of a data contract is:

"A data contract is an automatable agreement between a data producer, data consumers, and a third party responsible for programmatic standardization. It captures expectations around schema lifecycle, semantic metadata, security and compliance requirements, quality specifications, SLOs/SLAs and other forms of data governance."

Data contracts are necessary to manage the lifecycle of data products and associated service levels. They ensure that all parties involved in the data lifecycle adhere to the same standards and expectations, preventing misunderstandings and misuse of data. Their ability to enforce data governance automatically allows for systematic metadata capture, automatic data lineage tracking, and real-time quality controls. The owner of a Data Product is usually the data producer or the entity responsible for creating and maintaining the product, ensuring that it meets contract specifications and is fit for purpose.

How LLM and Generative AI are impacting architecture decisions? 

LLMs and Generative AI are impacting architecture decisions in significant ways. Generative AI, while a major trend, has revolutionized the fields of data governance and management, areas that have traditionally been slow to evolve. This change is primarily due to the high sensitivity of these models to data quality, requiring large volumes of high-quality data. Ensuring the quality of small data volumes can still be done with traditional and manual methods, but ensuring quality in large volumes is only feasible through automation.

This has made automated computational governance essential, justifying investments in governance automation. The demand for such automation has driven the adoption of technologies that can automatically monitor, correct, and optimize data. Additionally, analytics can benefit from this automation. Much of the governance, operation, and data work can be automated using learning models that can clean, enhance availability, and prepare data more efficiently and reliably.

The automation of these tasks not only improves the quality and availability of data but also allows for greater efficiency and effectiveness in managing large data volumes, preparing the data for analytical use more quickly and reliably. This means that the approach to automating these processes is not just about coding but also about using learning models that can autonomously improve data quality and readiness, reducing the reliance on direct developer intervention

Time to discuss about the future. What big changes will we see in the next 3-5 years in the field of Data Platfoms?

In the next 3-5 years, we will see significant changes in the field of data platforms. Reflecting on my experience as a pioneer in creating a data product platform, I believe this concept will mature over the coming years. We will start to see various vendors, different cloud providers, and major players talking about data product generators rather than just data platforms. The emphasis will be on generative and automated tools to achieve this.

A key focus will be on metadata, which has often been neglected. Everyone knows the importance of metadata for governance, but now it has become critical to manage it effectively. Efforts in analytics are not just directed towards solving business problems, but also towards applying analytics to the metadata itself. This self-analytics will make data platforms more autonomous, intelligent, and scalable.

The future of data platforms will involve an evolution towards data product platforms and a product mindset. Data will increasingly become the core of many businesses, not just a service provider. Just as technology has become central to business operations, data will follow the same path, becoming the essential core of the enterprise.

What Data Architecture journey would you recommend “traditional small/mid size companies”? Let’s think about a company with an ERP, a basic CRM and a BI tool facing classical issues about data quality, load times, etc. Can they start with a Data Warehouse without a proper Data Platform? What is the right sequence / cadence?

Not all companies are at the same level of data maturity. It is crucial to recognize that data holds different levels of importance depending on the business. At one end of the spectrum, we have companies whose core business is data, and at the other end, companies from various sectors increasingly relying on data. Traditional sectors like insurance, banking, and retail may become data-centric as their primary business shifts towards data. However, this transition does not need to happen for everyone.

A Data Warehouse itself is a form of Data Platform. It provides an ecosystem for working with data. Therefore, starting with a Data Warehouse can be sufficient for many companies, especially when dealing with fundamental issues of data quality and load times. The key is scalability: understanding the level of scalability your company needs is essential.

For some companies, a Data Warehouse may be enough. For others, it may reach its limits, requiring the addition of a Data Lake (or a basic lake house) prior to the DWH to expand scalability. Some may even opt for a complete Lakehouse instead of a traditional Data Warehouse to manage everything, but this approach can face governance challenges.

Choosing between these options depends on the scale of your company and the maturity level of data importance within your organization. Classical data platforms or simpler architectures may be suitable for less mature data needs, while more complex and mature platforms are necessary for those with greater data demands.

Furthermore, there is still a clear distinction between operational data governance and analytical data governance. Analytical data governance has evolved more due to scalability needs, especially in big data environments. Operational data can be kept under control and efficiently structured. The principles of governance for operational data, as established in frameworks like DMBOK, remain valid.

However, the paths of operational and analytical data governance need to converge again. Initially, analytics was separated from operational systems to avoid clashes. This separation happened because analytical processes could overload operational systems, causing performance issues and potentially disrupting core business operations. By separating the two, businesses could ensure operational stability while still deriving insights from data. As distributed systems and automated governance have evolved, the need for integration has become more apparent. In the future, the boundaries between operational and analytical data could dissolve, allowing for a seamless data environment.

My prediction is that with the rise of autonomous data product platforms and high levels of generative services, these platforms will serve both analytical and operational needs. This would mean that the separation we saw in the 90s and 2000s between transactional and distributed systems could reverse, reuniting the two domains. As data becomes a central product, it will serve both transactional purposes and insight generation fluidly. Thus, in the future, we could see the reunification of these two worlds, creating a holistic data environment.

Final questions, more on the personal side. Your artistic name is DJ Kram. Tell more about your passion for music
I'm really excited that you asked this question because music is my other great passion. I've always loved music, especially electronic music. I listen to many genres and subgenres, and just like I started early with computing, I also got into mixing music with turntables and vinyl. When CDs came along, I started mixing with them, and then with controllers when they became popular. Mixing music has always been my great hobby. For several years, I even did it professionally, balancing it with my engineering job at Telefónica. I worked as an engineer during the week and DJed on the weekends. Eventually, I decided to pursue my engineering career because it's challenging to make a living as an artist in this country. Nonetheless, I've always kept up with DJing, buying music, and maintaining a DJ setup at home. I'm part of a DJ collective and I perform and organize a few parties throughout the year. In short, music is my other great passion. My nickname, DJ Kram, reflects my alter ego. My passions are engineering and data on one side, and music on the other, both united by technology.

And what do you think about AI tools such as Udio or Suno, which are being used to “generate music”. I believe they could boost creativity…but at the same time they are getting criticism and some record labels, including Sony and Universal have filed lawsuits against them arguing they "steal music to spit out similar work"

This field of artificial intelligence applied to artistic and musical creation represents a convergence of my two worlds: music and data. I see generative AI as a bridge between these passions. Regarding tools like Udio and Suno, although they are innovative, as you mentioned, they generate significant debate around intellectual property rights. The art world has long been commercialized through the capitalization of intellectual property, which is crucial for protecting artists and allowing them to profit from their work. However, this system can sometimes become monopolistic, hindering new artists from emerging. The conflict we are witnessing with AI-generated music is reminiscent of past disruptions in the music industry, such as the piracy era with P2P networks. These disruptions ultimately led to the creation of streaming services, which revolutionized music distribution by legalizing and democratizing access to a vast music catalog that was unimaginable in the classical music industry.

But I believe the major disruption of AI in music and art is that it facilitates and lowers the cost of artistic production, significantly lowering the barrier to entry. This democratization can lead to an explosion of content, both good and mediocre. Therefore, we face a new challenge: sifting through this abundance to find the gems. In essence, AI is breaking down barriers in music production, creating a more inclusive yet more congested creative space. Ultimately, music, thanks to AI, could be on the verge of facing a scalability crisis similar to what data experienced in the 2000s with Big Data.