Transcription
eBookEmpowering the Quant withFaster Experimentationand Application of AI andMachine LearningCopyright 2021, Amazon Web Services, Inc. or its affiliates.
Table of contentsIntroduction.3Data acquisition and ingestion: Find and transform data quickly.4Modelling and insight: Run experiments and backtest faster.7Interpretation and push to production: Synthesize results and operationalize quickly. 10Summary. 13EMPOWERING THE QUANT WITH FASTER EXPERIMENTATION AND APPLICATION OF AI AND MACHINE LEARNING2
IntroductionAs financial markets fluctuated in 2020, updating portfolio metrics, including metrics like P&L,volatility, and value at risk (VAR), across asset classes, was challenging. Elevated levels of VARbacktesting exceptions led to higher regulatory capital multipliers. Increases of as much as 30percent were reported.1 There were also challenges with valuation adjustments, as derivativesfaced snowballing collateral calls and increasing funding costs.Rapidly changing market dynamics, the movement from actively managed to more passivefunds, and the highly competitive nature of investment management have increased thepressure on quantitative analysts (quants). They now must sift through a constant blizzard ofreal-time news and historical data to keep up with both baseline reporting and implementationof strategy adjustments. Modeling and analyzing the endless loop of asset and world eventinterdependencies requires significant compute power and scalability.Amazon Web Services (AWS) offers services like AWS Data Exchange, Amazon SageMaker,Amazon EMR, Amazon FinSpace, and infinitely scalable compute and storage infrastructure,all of which integrate with proven AWS Partner content and solutions for data acquisition,ingestion, analytics, and artificial intelligence (AI) and machine learning (ML). Using theseservices and solutions, quants can tackle their three main challenges: data acquisition andingestion, experimentation and modeling, and interpretation and production. This eBookexplores et-riskEMPOWERING THE QUANT WITH FASTER EXPERIMENTATION AND APPLICATION OF AI AND MACHINE LEARNING3
Data acquisition and ingestion:Find and transform data quicklyFor quants, researching, building, testing, implementing, and interpreting models in the context ofrisk management, portfolio creation, asset selection, rebalancing, and trading have been the naturalorder of things for years. However, recently, the proliferation of big data and alternative data hasintroduced yet another level of complexity and effort to getting quantitative analysis done.Now quants must add big, complex, and unstructured data to the mix of logging in, checking datafor accuracy and duplication, transforming, linking, and structuring data, polling for new content,and so on. It’s likely that undifferentiated tasks like infrastructure and basic data wrangling areleaving you less time to do analysis, to experiment, and to adjust strategies. AWS and AWS Partnersoffer a way out of this data preparation quagmire, so you can spend more time finding alpha,modeling, interpreting, and identifying innovative financial and trading strategies.EMPOWERING THE QUANT WITH FASTER EXPERIMENTATION AND APPLICATION OF AI AND MACHINE LEARNING4
Speed time to data acquisitionAcquiring and connecting multiple traditional and big data sources is time-consuming. Think of howmuch more analysis and modeling you could do if you could avoid this phase altogether. AWS andAWS Partner services automate data access and acquisition processes, so you can move faster.AWS Data Exchange makes it easy to find, subscribe to, and use over 3,500third-party data assets from financial data providers and other industry dataproviders such as retail, healthcare & life sciences, and manufacturing.Bloomberg Market Data Feed (B-PIPE) on AWS provides immediate andreliable access to the same normalized and consolidated real-time market dataavailable on the Bloomberg Terminal without compromise on precision, qualityor reliability. With B-PIPE accessible on-premises, hosted, and now on the cloud,clients across the globe can choose their preferred deployment method forreal-time data delivery.Xignite Cloud APIs on AWS enable you to retrieve real-time, historical,primary, and fundamental data from more than 250 data sources, such asFactSet and Morningstar, and directly from exchanges.Apptopia Mobile App Intelligence on AWS delivers daily insight into theperformance of more than 7 million apps—a proven, consistent indicator ofpublic company earnings, KPIs, and stock performance.EMPOWERING THE QUANT WITH FASTER EXPERIMENTATION AND APPLICATION OF AI AND MACHINE LEARNING5
Eliminate time-consuming, repetitivedata ingestion workCleansing, wrangling, and normalizing disparate data sets and sources can consume anywherebetween 25-50 percent of a quant’s time. Add governance, monitoring, and security in the process,and it increases to as high as 80 percent.4 AWS and AWS Partners offer managed services thatautomate repetitive data preparation, handle governance and monitoring, and simplify security.Alteryx Server on AWS complements and amplifies your AWS capabilities byproviding governed and scalable analytic automation that unifies data prep andblend, analytics, data science, and process automation.The IHS Markit Data Lake empowers you to discover business insights faster,by exploring, accessing, and coalescing our data, your data and third-party dataon a single, cloud-based platform. It is populated with 1,250 of our datasetsfrom diverse industries, including financial services, automotive, maritime,energy and natural resources to expedite time-to-value. By transforming theway you find and access data, the Data Lake supports advanced analytics anddata science at scale, helping you to gain a competitive edge.FactSet Concordance Service optimizes the organization and potential of yourproprietary and third-party data with a high-quality data model that providesseamless, instant connectivity to thousands of global market identifiers andstock exchanges. FactSet links permanent security identifiers to the entityidentifiers to provide risk insight and data dissemination on multiple levels.Financial sector quants and data analysts held back by lack of automation, Hedgeweek, 27 Jan. -lack-automation4EMPOWERING THE QUANT WITH FASTER EXPERIMENTATION AND APPLICATION OF AI AND MACHINE LEARNING6
Modelling and insight:Run experiments and backtest fasterAI and ML use new systems for spotting patterns that are beyond the capacity of the human brainand incorporate what they glean into improving and automating processes. With the exponentialgrowth of big data and as computing power becomes more affordable, AI and ML are becomingmore of a fixture in capital markets. Since finance is quantitative to start with, combining it with AImakes sense.In the last few years, there has been an increase in firms that are using AI and ML to: Improve effectiveness of trading algorithms. Backtest stressed data to aid in portfolio risk management, particularly around markets and risk. Use AI techniques to process unstructured and “big data” into investment insight. Manage surveillance and conduct transaction cost analysis (TCA).With ML, quants can combine statistics and computation to build models that use new levelsof scale and richness to generalize better, surface unseen data, and tackle new problems. AWSand AWS Partners have solutions that provide ready access to high-performance compute (HPC)capabilities for backtesting, portfolio simulations, running risk calculations, and delivering otheruseful insights.EMPOWERING THE QUANT WITH FASTER EXPERIMENTATION AND APPLICATION OF AI AND MACHINE LEARNING7
Avoid the heavy lifting of creating, training,and testing modelsML model development can be a complicated, costly, and infinitely iterative process. For quants, all this data sciencework, along with the repetitive nature of calibration and backtesting, slows their analysis. This can be frustratingwhen you want to find meaningful insights related to trading and do not want to train data. AWS and AWS Partnersoffer a rich toolkit for probabilistic, scenario-based, and risk-focused modeling that includes Jupyter, Python, and Rnotebooks and cross-asset financial model libraries.Amazon SageMaker manages all underlying infrastructure to train models at petabyte scale.Additional automation and abstraction from ML frameworks like Spark and TensorFlow, alongwith custom investment algorithms, accelerate model training and progress to your end goal.Amazon EMR lets you easily run and scale Apache Spark, Hive, Presto, and other big dataframeworks. Set up, operate, and scale your big data environments by automating timeconsuming tasks like provisioning capacity and tuning clusters.Amazon FinSpace is a data management and analytics service purpose-built for the financialservices industry, reducing the time you spend finding and preparing petabytes of financial datafor analysis from months to minutes.Accern is a no-code platform that is powered by Artificial Intelligence and Natural LanguageProcessing and designed exclusively for the financial services industry. Utilize Accern to gainunprecedented insight into structured and unstructured financial data that is generated fromhistoric and real-time sources and creates a critical competitive advantage.Databricks on AWS delivers a unified, collaborative ML platform that enables you to use yourAmazon S3 data lake as a lakehouse, providing a single platform for storing and delivering yourdata for data science as well as business analytics and other use cases.EMPOWERING THE QUANT WITH FASTER EXPERIMENTATION AND APPLICATION OF AI AND MACHINE LEARNING8
Support experimentation workloadsA machine learning experiment requires more than a single learning run; it requires multiple runscarried out under different conditions. Unlimited compute is needed to support tracking, visualizing,comparing results from different runs, quantum computing, and the graduation and promotion ofmodels that prove to be useful.Amazon Elastic Compute Cloud (EC2) P3 instances have been proven toreduce machine learning training times from days to minutes, as well as increasethe number of simulations completed for high performance computing by 3-4x.Amazon Braket, powered by IonQ and D-Wave, offers the quantum computeneeded for risk calculations such as Monte Carlo simulations. Amazon Braketis a fully managed quantum computing service that helps researchers anddevelopers get started with the technology to accelerate research and discovery.Amazon Braket provides a development environment for you to explore andbuild quantum algorithms, test them on quantum circuit simulators, and runthem on different quantum hardware technologies.EMPOWERING THE QUANT WITH FASTER EXPERIMENTATION AND APPLICATION OF AI AND MACHINE LEARNING9
Interpretation and push toproduction: Synthesize results andoperationalize quicklyInsights from ML models can lead to better predictions and improve the trustworthiness of theresults. Therefore, with the data crunched, you need to interpret your models. Yet models canbe opaque, which is why they are often referred to as black boxes. Visualizations and businessintelligence tooling can help you better identify and understand trends highlighted by the model.They make it easier to get latent insights from data. These tools can also help you identify keyfeatures and meaningful data representations in your model—and even indicate which of them canimprove results when added to systems and tools.Once you have determined which models are worth operationalizing, you need to incorporate theminto your asset management and securities platforms, which can be tricky. Training data mightnot match production data, the model can change workflows, and questions of where to host themodels can arise. Or you might have to hand the model over to an application developer, who isnot likely to know exactly how to get it into production. Operationalizing ML, often called MLOps,requires specialized knowledge and tools, usually only found in the rare combination of a developerwith data science training.AWS and AWS Partners provide the tools and services that can convert AI and ML data streamsand models into visualizations that lead you to clear information and insights. They also deliverframeworks, APIs, and connectors that remove the friction from operationalizing ML in investmentmanagement systems.EMPOWERING THE QUANT WITH FASTER EXPERIMENTATION AND APPLICATION OF AI AND MACHINE LEARNING10
Quickly understand what a model has uncoveredand make use of it in your strategyTools that paint a picture of your model’s behavior outcomes make it easier to explore and understand your data.They decompose model output into graphical representations of your data. You can spot trends, aberrations,or new trading patterns. They also offer explainability, which shows how a model arrived at an outcome. AWS andAWS Partners offer services that can transform model output into valuable insights and offer details about howthe model works, the processes it runs on input data, and the top reasons for the model’s outcome.Amazon QuickSight leverages AWS’s proven ML capabilities, making it easy for BIteams to perform advanced analytics (example: what-if analyses, anomaly detection,ML-based forecasting, and churn prediction). without prior data science experience. UseQuickSight’s pre-built models or bring your own ML models from Amazon SageMaker,which integrate with QuickSight in a few clicks. Automatically generate a summary ofyour dashboard in plain language, which interprets and describes key insights from yourdata, for consistent and shared understanding.DataRobot can export key insights and predictions from ML models into traditionalvisualization tools or Excel. A variety of built-in systems helps you explain and—ifnecessary—defend your models.InterSystems IRIS data platform provides the proven highest performance for realtime concurrent transaction, streaming and analytics processing in a single environmentwith enterprise-class reliability, durability and scalability.KX Insights an AWS-native platform for critical real-time streaming analytics andcontinuous actionable intelligence. Using complex event processing, high-speedanalytics and machine learning interfaces, it enables fast decision-making andautomated responses to events in fractions of a second.EMPOWERING THE QUANT WITH FASTER EXPERIMENTATION AND APPLICATION OF AI AND MACHINE LEARNING11
Integrate models into portfoliomanagement and trading toolsThe moment has come. It’s time to move your model into your investment or trading systems. That momentcan stretch into days and even longer, however, if you need to rely on a developer or data scientist to deploy it.MLOps is complicated. It can require specialized knowledge in container and container management and morefrequent deployments. AWS and AWS Partners offer services and API endpoints that simplify and automateMLOps. For example, the AWS MLOps Framework helps you streamline and enforce architecture best practicesfor moving ML models into production.Linedata leverages a new technology architecture with the latest cloud capabilitiesand features alongside robust core order management and software and portfoliomanagement software for a seamless, enhanced user experience. You can adopt newfeatures across the investment cycle with ease, for better integrations, performance,and scalability.Provectus offers MLOps services that help deliver ML models from research toexperimentation, through training, QA, and production to A/B testing, faster and withminimal handoff at scale.EMPOWERING THE QUANT WITH FASTER EXPERIMENTATION AND APPLICATION OF AI AND MACHINE LEARNING12
SummaryIn quant investment, AI and ML play a role in valuation, asset allocation, risk management, and compliance.AI and ML can improve accuracy in pricing, determine new investment strategies, shed light on various formsof risk, and save time and money in meeting regulatory requirements. Therefore, AI and ML are in demand ininvestment management and banking. At the same time, AI and ML require the ingestion of massive amounts ofdata in all kinds of structures and formats—and a great deal of compute power to be effective. Deep learning,quantum computing, neural networks, and more test the limits of traditional legacy infrastructure.On AWS, compute power and scalability meet deep experimentation and AI and ML compute capabilities.AWS and AWS Partner solutions automate data acquisition and ingestion, experimentation and modeling,and the push to production and interpretation of ML models. As a result, quants can skip all the mechanicsinvolved in preparing data and developing ML models and go straight to analysis. Not only does this fastertime-to-analysis benefit them, but it also benefits their investment management firms by helping them: Lower the cost and time of experimentation, allowing investors to enter (or leave) new markets efficiently. Mine decades of historical data and ever-growing alternative datasets for new signals. Increase responsiveness to changing market conditions, such as unexpected volatility, unique regulatoryrequirements, and evolving threats. Personalize investment recommendations in real time.EMPOWERING THE QUANT WITH FASTER EXPERIMENTATION AND APPLICATION OF AI AND MACHINE LEARNING13
Resources to help you get startedwith AWS and partnersAWS Partner Network (APN) and AWS MarketplaceAWS Partners are focused on helping you take full advantage of all the business and technologybenefits that AWS has to offer. With their deep knowledge of AWS, AWS Partners are uniquelypositioned to help you at any stage of your cloud journey. Work with AWS Partners that haveachieved AWS Competencies in financial services to protect customer data, support continuity ofbusiness-critical operations, and meet new regulatory standards. Discover partner solutions in AWSMarketplace, a digital catalog that makes it easy to find, test, buy, and deploy software that runson AWS.For information about the AWS services mentioned in this eBook, visit:AWS Data ExchangeAmazon SageMakerAmazon EMRAmazon FinSpaceAmazon Elastic Compute Cloud (EC2) P3 InstancesAmazon QuickSightExplore AWS Capital Markets Financial Services solutionsFind specific AWS Financial Services Partner SolutionsEMPOWERING THE QUANT WITH FASTER EXPERIMENTATION AND APPLICATION OF AI AND MACHINE LEARNING14
Copyright 2021, Amazon Web Services, Inc.or its affiliates.
FactSet Concordance Service optimizes the organization and potential of your proprietary and third-party data with a high-quality data model that provides seamless, instant connectivity to thousands of global market identifiers and stock exchanges. FactSet links permanent security identifiers to the entity
