Execution Platform Development – Part 2

If you missed Part 1, you can catch up here. In the first chapter, we followed Roger and the team through a high‑pressure market morning — and left off just as Tobias was pulled back into operational reality. Now, we zoom in on what happens next: the moments where engineering craftsmanship meets real-time market demand — and where the platform’s resilience is proven order by order.

During his operational session, Tobias observes a sudden influx of orders bombarding our SOR. Processing approximately 250,000 orders a month, the SOR is a crucial, central component of the trading platform.

Tobias knows the technical details by heart, as he spearheaded its development five years ago, when the team opted to construct their own SOR. The choice was between purchasing a pre-existing product from a vendor or investing time and resources in an in-house development. Ultimately, the benefits of complete control and ownership of the code of such a key system component outweighed the costs involved with its development. Even in this moment, Tobias does not regret their decision, as it allows him to fine-tune the order-handling configuration in real time, ensuring seamless order processing even during peak load times.

Transitioning from his operational to his development workstation, Tobias verifies the status of the nightly deployment into the test system. The whole team cares a great deal about automation and streamlined development processes, one of which is a continuous delivery pipeline that releases the latest software version every night. All checkpoints are green, indicating that the client’s change request he worked on the previous day is ready for testing.

Tobias takes pleasure in crafting tailor-made solutions for our clients, a process that begins with requirements engineering, involves in-depth discussions with the client to fully comprehend their needs, and finally also includes rigorous testing and validation of the delivery. With the trading system, we host approximately 100 highly customizable client and market connectivity plug-ins, complemented by a constant stream of real-time market data. Given the multitude of touch points, it is crucial to meticulously orchestrate changes and releases.

Tobias knows that nothing can cause more harm than an untested piece of software being released into production.

As 12:30 p.m. approaches, Cyril, another PULS contributor, asks Tobias to review the latest changes he just made to the SOR. Such pull requests serve as a means to review code modifications, a crucial tool for quality assurance, knowledge-sharing, and general discussion and reasoning about code within the team. As a Senior Software Architect, Tobias appreciates tools that facilitate a well-organized development workflow and enjoys providing insightful feedback thanks to his profound understanding of the platform. Recognizing the significant impact of the proposed code changes in the pull request, he invites Cyril to join him for lunch for a face-to-face discussion. Fostering collaboration and having regular, open discussions about daily work are vital for a high-performing team. He knows that good things take time and are best explored over a delicious meal in the Vontobel canteen; today, they’re serving his favorite, pad kra pao.

Eugene has just returned from lunch and is ready to take over operational duties for the afternoon session. After a brief handover, Eugene assumes control of the operational desk on the 2nd floor, freeing up Tobias to have his planned lunch with Cyril. During the handover, Tobias confirms that it has certainly been a busy day due to the US elections, but operationally, there are no open production incidents.

Eugene senses the charged atmosphere on the trading desk, which is expected to climax when the US markets open in a few hours. After checking the monitoring system and the operational mailbox to confirm there is nothing pending, he takes the time to switch to some development-related tasks.

This week he has two client onboarding tickets assigned to him. One client has very specific requirements, which would require some adjustments to the standard predefined FIX rules of engagement we have in place with clients. Eugene makes some comments on the ticketing system, stating what needs to be done and providing estimates for the completion of the work. He references Rainer, who heads Execution Platform Operations, as client onboarding requires a combined effort from both. The client requirements do seem quite complex, and the indication is that the client wants to test as early as next week. Eugene does not hesitate and is determined to be ready for production by the next release in two weeks. As always, he is willing to roll up his sleeves and get his hands dirty to implement client needs as soon as humanly possible. This commitment to delivering value is at the core of his approach to serving clients.

It’s 3:30 p.m. and the US markets have opened. Eugene once again turns his attention back to operational duties. He takes a quick stroll around the trading floor to get a sense of the general market mood. He glances over the operational monitoring system, which offers an overview of the hardware infrastructure. The overview includes information ranging from general server load statistics to much more technical details relating to procedures designed to free up unused memory. After analyzing the numbers, it is clear that as things stand the execution platform is capable of handling more than double the volume of orders with no noticeable impact. Eugene and the other team members have been involved in a lot of efforts over the years to ensure the platform is optimized and efficient so it can handle the kind of load it is facing today. Furthermore the Execution Platform is supported by three data centers, each housed at a different geographical location. This allows the system to operate from any of these centers and switch between them seamlessly. This setup guarantees our constant availability, regardless of any localized technical issues that might impact a single data center.

As the orders come flooding in after the US markets open, it is now clearer than ever that incoming order volumes will match or even potentially surpass previous records. Eugene reaches out to Evangelos so they can analyze the current situation and review the business continuity management (BCM) processes we have in place, should a need arise to consider them.

Evangelos starts reviewing the numbers with Eugene, and the projections indicate that today we could surpass 80,000 incoming orders. This potential new record high for the execution platform does not worry Evangelos at all, as he has proven that the system can handle more than 200,000 incoming orders, which is exactly the number achieved in the test we most recently performed less than a year ago. We conduct thorough “tsunami” load testing scenarios, where we inject extraordinarily high order volumes into the system. Simultaneously, we run software profiling on the system to ensure it always behaves as expected without any failures. We believe a system is only as strong as its weakest component, so we continuously identify and enhance these areas to improve the overall performance of the Execution Platform.

After the brief review of the order flow, and with the BCM procedure in place, Evangelos turns his attention to the current sprint board. The sprint board contains all work the team is handling during a given week. Every Monday, the team reviews this work in the planning meeting, which Evangelos personally heads. In two weeks, we are going live with the new release and everything seems to be on track, but Evangelos wants to ensure that we do not have any last-minute additions in our release scope requiring immediate attention. For this reason, Evangelos calls Roman, the Head of Trading Product Development, who is also the product owner of the Execution Platform. Roman defines the exact work priorities based on the feedback he receives from clients and traders. Roman mentions that there has in fact been a new request from the trading desk, namely, an interest in improving the 24-hour offering of our system. In conjunction with a specific client, the intention is to improve the 24-hour offering by allowing a broader coverage of the Asian markets. Evangelos acknowledges this requirement and promptly creates the tickets, which he assigns to next week’s sprint. This work will be addressed and promptly tested to go live with our next release, highlighting the team’s agility. Agile is not just a buzzword we use in our team, it’s something embedded in our DNA. We strive to deliver immediate value to our clients by completing the exact requirements they need, when they need them.

Evangelos shifts his focus to his own development tasks for the week, which include new initiatives within the TB division and Vontobel at large. Specifically he is thinking about recent advancements in artificial intelligence and cloud computing. It’s crucial for the team to keep up to date on these technological developments and to assess their potential benefits for our clients. As a team, we take pride in the innovations we’ve introduced so far, and we are committed to ensuring that the Execution Platform continues to embody this spirit of innovation in the future.

As the afternoon session continues, PULS and the algorithmic trading strategies are firing on all cylinders, executing orders with surgical precision. Occasionally, traders will reach out directly to Eugene, asking him to review and investigate instances that appear to have been mishandled. Fortunately, with the experience of the Execution Platform team and the procedures in place to intervene with order executions where necessary, all such cases are addressed seamlessly.

Eugene’s late operational duties end at 6:00 p.m., when most European exchanges close. Active trade desk support will continue, as phone calls will be forwarded to Roger, who is on call tonight. The US market trades until 10:00 p.m., and the team needs to be reachable in case of any event. On-call support will be available until the following morning, when Norbert, our seasoned operational member, takes up early duties at 7:00 a.m., thus closing the loop of continuous operational support.

In the end, today seemed to be just another day at the office for Roger and his team. As expected, the US elections did indeed put the Execution Platform to the test, but ultimately, every single one of the nearly 80,000 orders were handled flawlessly, exceeding expectations at every turn—a shining testament to an extraordinary job well done! Behind every line of code, behind every system configuration, and behind every hardware decision lies the relentless determination of our team to serve our clients and help Vontobel secure its place in a fast-paced, competitive market.

As Eugene leaves B21, snow begins to settle gently, transforming the once bustling atmosphere into a serene, tranquil scene ...