The cloud has been hailed as one of the great innovations of the last decade, offering scalability and flexibility that ensures greater speed and capacity for heavy-compute technology companies such as Numerix. We service the capital markets industry as the leading provider of innovative financial technology solutions and real-time intelligence applications for trading and risk management to help our clients reimagine operations, modernize business processes, effectively manage risk, and capture profitability.
We understood early on that moving towards a fully cloud-centered architecture and model for delivery of our analytics via SaaS would provide substantial benefits to Numerix and our clients. We needed to choose a cloud provider that supported many different technology choices and service vendors within its development ecosystem and that enabled us to deploy those efficiently in every global capital market region for our customers. To meet daily SLAs for our SaaS products, the cloud infrastructure needed to be able to quickly scale out to thousands of cores and equally scale down in cost-effective manners. AWS was a clear choice as a partner in this due to its global reach and support, reliable dynamic scaling resources and exceptional ecosystem of integrated cloud-native services, development tools and third-party vendor options.
Cloud Adoption Focus Areas
Our cloud adoption strategy has focused on three products: Migrating Numerix Oneview from on-premise delivery to SaaS, designing Numerix NxCore from the start as a cloud-native application, and migrating Numerix NxDM from on-premise delivery to a fully reengineered SaaS capability built entirely using serverless cloud technologies. Each product area has included its own set of challenges in our cloud adoption journey:
- Oneview is a comprehensive, real-time capital markets risk management system that empowers sales, traders and risk managers with a holistic, multi-dimensional view of their derivatives and structured products businesses. Our challenge here was to migrate a complete risk application with a microservices architecture and on-prem delivery to a SaaS model leveraging cloud technologies.
- NxCore is a capital markets development platform that enables institutions to rapidly build, test and deploy apps with enterprise-grade data faster than ever before, with a collaborative, cloud-based integrated developer environment that leverages the power of Numerix’s pricing and risk analytics and the strength of the Python ecosystem. As a new product, we were able to design for cloud delivery from the start, so our challenges were in how we could most efficiently take advantage of the many development options in AWS and securely make those available for app-creation use cases.
- NxDM is a robust data management solution that leverages market data from multiple vendors and exchanges to provide end-of-day, intraday, and on-demand snapshots across different time zones and among a diverse set of asset classes, including Equity, Rates, FX, Commodities, and Credit. The daily processing of this data requires efficient storage of large amounts of time-series-oriented data streams as well as dynamic compute scaling to calculate normalized and derived data values. We migrated earlier on-prem tools based on relational database schemas to a fully serverless implementation, optimized our daily operations using AWS step functions, and migrated data storage and query functionality to Snowflake.
The Stages of Oneview Migration to SaaS via AWS
There were different points in our journey to AWS adoption for all three products. For this blog post, I want to share our adoption story for Numerix Oneview, our flagship product. It took place in four stages; the backdrop for each is highlighted below and I include the benefits and concerns related to each stage.
- Stage 1: Pre-2017 - On-prem delivery focus
Oneview initially was delivered entirely as an on-prem solution using microservices with integrated grid compute components designed to run on a dedicated compute cluster within each customer data center. This model had the benefit of allowing for complete customer control of the location and access for sensitive data within their portfolio. However, it had significant cost and inefficiency, required a lengthy hardware sizing, procurement, and configuration process for each deployment, and added complexity for support and scheduling of upgrades. Compute scalability was limited to the size of the configured cluster, and often resulted in compute resources sitting idle during off-peak times.
- Stage 2: 2017-2020 - Lift-and-shift to AWS EC2 instances
Our initial customer engagements followed a fairly standard first step: to move the on-prem implementation to run roughly as-is on cloud-hosted VMs within AWS EC2, using managed relational database services and a complete single-tenant model. This was straightforward to get working and did not require significant code changes, and being single-tenant, maintained customer data security, but did not fully address all the on-prem limitations. It was clear that moving key functionality to serverless implementations or to cloud-optimized services from AWS or third-party vendors would help with system performance, reliability, and total cost of operation.
- Stage 3: 2020-2023 - Serverless compute, Snowflake storage
Oneview compute operations now run as serverless operations using AWS step functions, Lambda and AWS Batch, and for storage, we have partnered with Snowflake to move handling of market data and calculated results to cloud-native data warehouse storage with scalable query tools. Serverless compute allows us to dynamically scale up large scale calculations for derivative pricing and risk to resource amounts that were simply not possible in on-prem deployments. Using AWS batch, we regularly run calculations that scale out to tens of thousands of cores or more for multiple customers, with costs based only on the time spent in calculation. While updating to this model does have a development learning curve and adoption cost, for workloads with burst times requiring large calculation resources in limited time windows, the cloud operation tools are a game-changer.
- Stage 4: Ongoing - Automation, multi-tenant
Going forward, we will be accelerating our use of serverless and dynamically-scaled cloud services in Oneview beyond our compute and large-scale storage tools. Daily workflow and support operations can be automated using many cloud services that would have otherwise required custom integration steps to perform. Finally, centralized services are being moved to multi-tenant operation that will further reduce total operational costs and support.
About the Author
Dr. William Humphrey has been designing and building large-scale parallel and distributed applications for over 25 years, in both scientific and financial industry settings. Since joining Numerix in 2003, he has led development teams for many of the Numerix valuation and risk products, focusing on infrastructure design, risk analysis features and large-scale parallel computation capabilities. In recent years, Dr. Humphrey headed up the Numerix Architecture group, responsible for designing and coordinating the overall architecture strategy for the Numerix software platform. Most recently, in 2021 he took on the role of Chief Technology Officer and now oversees the departments responsible for engineering and development of Numerix applications, including its flagship SaaS risk application, Oneview. Prior to joining Numerix, Dr. Humphrey developed general cluster management and server deployment software for Turbolinux, and before that was a Technical Staff Member at Los Alamos National Laboratory where he was a founding member of a team creating a general purpose, scientific parallel application framework used within several key projects. Dr. Humphrey holds a Ph.D. in Physics from the University of Illinois at Urbana-Champaign and has published over 10 scientific journal articles and book chapters in his career.
