The NYSE accelerated its 5x real -time streaming data with Redpanda

NYSE deployment of the Redpanda C ++-based streaming platform has made 4 to 5x performance improvements On traditional Kafka competitors, exhibiting fundamental limitations in the way most organizations manage data workloads.

This performance difference becomes essential as companies evolve AI applications which require coherent access to low latency data. Kafka -based data streaming also has a potential to allow agent agent communications, competing with other approaches like Google A2A and it can also be extended to allow the context protocol of the model (MCP).

“The market thesis is that all the major foundation models have really indexed public data sets, and the next border is private data sets, and Redpanda really unlocks private data sets for agent access,” said Alex Gallego, founder and CEO of Redpanda to Venturebeat.

What the NYSE builds in the cloud

NYSE has built its Cloud streaming platform to serve customers who cannot directly access its data centers. The exchange targets fintech companies and retail brokers who need access based on AWS to real -time market data.

“All consumers of our market data do not have the capacity to come to our data center, take the flow and use this flow,” said Vinil Bhandari, head of cloud and full engineering in Nyse in Nyse in Venturebeat. “But you know, a small shop in Hong Kong has access to the creation of their own AWS account, for example, and it is these audiences that we are trying to answer.”

NYSE diffuses its BQT (Best Quotes and Trades) flow, which brings together real -time data from the seven NYSE exchanges. Deployment required the construction of new infrastructure rather than extending existing systems.

Why Nyse chose Redpanda and how the choice of programming language is important

NYSE treats more than 500 billion messages per day over seven exchanges. During market volatility, The volume of messages can increase by 1,000x above the microsecond average.

The traditional Java implementations fight with these models because the collection of garbage creates unpredictable latency peaks.

“Kafka’s classic implementation has been written in the Java programming language, which makes this type of trafficking missed, you know, not very well with the Java waste collection that occurs in the programming language,” said Bhandari. “Redpanda has implemented Kafka by rewriting the Kafka protocol in C ++, so each time we get a gust of traffic from our market activity, volatility, we are able to better manage this streaming out of data.”

The choice of programming language is also the reason why NYSE went with Redpanda for data streaming instead of other options such as confluence or Amazon streaming for Kafka (MSK).

This technical decision led to a measurable performance improvement.

“We are safe to establish that We are at least four to five times faster in our delivery of data using Redpanda Compared to some of our personalized competitors of large tickets that use Kafka technology to disseminate similar data, “noted Bhandari.

For companies evaluating streaming platforms, this comparison highlights critical consideration: implementations based on Java for data streaming can fight during traffic peaks, while alternatives based on C ++ can maintain consistent performance.

Observability is critical for critical mission deployments

Bhandari highlighted observability as essential to production streaming deployments. Redpanda’s integrated telemetry capabilities have provided immediate operational value.

“The more deployment like this can have an observability and telemetry of what is happening under the hood, the more data producer and data consumers will be,” said Bhandari.

This observability allows proactive detection and resolution of problems before the problems have an impact on customers. Without complete supervision, companies risk discovering performance problems only after affecting production workloads and customer experience.

Shift of the philosophy of architecture: Streaming as a foundation of IA

NYSE will use streaming data capabilities in a fairly traditional way, at least initially. These are that the data of its market exchanges are made available so that users can consume.

The management that Redpanda is directing indicates a more agentic future, that of users such as Nyse will probably adopt in the years to come. The CEO of Redpanda, Gallego, maintains that companies should see architecture in streaming differently in the AI era.

“Streaming has the right architectural model, not for speed, but because it is the right architecture for reactive and agent applications,” said Gallego.

Beyond solving traditional streaming performance problems, Redpanda has repositioned himself for what Gallego calls on agency business. The company has wrapped its data connectors in MCP (Model Context Protocol), allowing AI agents to directly access business data sources.

This approach solves a problem of computer complexity that emerges while businesses deploy several AI agents.

“Without the Kafka API, you have a square communication problem where each agent must have access to all the other agents,” said Gallego. “And when you introduce the Kafka API, it goes from the computing complexity to the square to the linear.”

According to Gallego, banks are already deploying hundreds of agents. A Redpanda client plans to build 1,000 agents over the next two years. Another currently builds 130 agents for the deployment of production within 18 months. These scale requirements make architectural architecture decisions for essential agents for the success of the long -term AI strategy.

What this means for the corporate data strategy

Real -time streaming data should become an increasingly critical aspect of the operations of many organizations.

The NYSE evaluation process reveals critical decision -making criteria for business decision -makers evaluating streaming infrastructure:

Kafka, based in Java, strikes the performance walls under the gusty traffic. Organizations managing unpredictable workloads should assess the alternatives based on C ++ before scale for production deployments. The difference in performance 4-5X is not a marginal optimization but a gap of fundamental capacity.

Streaming strategies first in cloud can achieve production performance. This allows global models of access to data which were previously unexpected due to latency constraints, opening up new market opportunities for data focused on data.

The coordination of agents requires streaming architecture. As the deployments of AI develop beyond unique agents, streaming platforms become essential infrastructure rather than performance optimizations. The advantages of calculation complexity become essential on the scale.

For IA implementation planning organizations, it is essential to prioritize streaming platforms that support MCP integration and agents’ coordination. The advantages of calculation complexity become essential on the scale and the modernization of the coordination architecture after the deployment of several agents is exponentially more difficult than building it properly from the start.

Organizations waiting to adopt AI should recognize that streaming architecture decisions taken today will limit future AI capacities that most leaders think so.

Daily information on business use cases with VB daily

If you want to impress your boss, VB Daily has covered you. We give you the interior scoop on what companies do with a generative AI, from regulatory changes to practical deployments, so that you can share information for a maximum return on investment.

Read our privacy policy

Thank you for subscribing. Discover more VB newsletters here.

An error occurred.