Architectures For System Integration - EBU

Transcription

DISTRIBUTED PRODUCTIONArchitectures forSystem integrationChris ChambersBBC R&D and EBU Project Group P/MDPBroadcast production systems typically were built as a collection of “silo” or“island” devices. However, with the increasing use of IT in production, and thesharing of common resources such as storage and connectivity, the questionbecomes: how best can we glue all this equipment together into an integratedsystem? This is exactly what EBU Project Group P/MDP has been studying.It is a fact of life that there will rarely, if ever, be global agreement on common standards in somefields. Which nation will give up their national language for a universal “Esperanto”? Technology isno different. Broadcasters will remember the “camps” of PAL and NTSC.Accepting this fact leads us tolook for solutions that allowinterchange between different“standards” – what we chooseto call Middleware. In the caseof PAL/NTSC, the answer wasthe standards converter – anearly example of middleware.Figure 1A familiar problem with a well-known solutionAt a more basic level, considerthe variety of different mainspower connectors which exist throughout the world. To enable your equipment to work anywhere, itis necessary to have a different mains cable for each type of connector. And then came the middleware – the universal mains adapter.System1System . ’n’System4?System2System3Figure 2Where the universal adapteris neededIn broadcasting these days, every system tends to be a framework; with endless possibilities, but only working if tightly integrated with all of the other systems (frameworks) of theorganization. This can be very hard to achieve.Even if you do reach some level of integration, full interoperabilityis seldom achieved. It tends to be expensive, takes a lot of time toconfigure and implement, and fails to fully realize the opportunities for flexibility. On top of that, the demands to constantlychange and optimize our workflows using these systems areincreasing.We are missing the universal adapter between our systems!EBU TECHNICAL REVIEW – January 2005EBU Project Group P/MDP1 / 13

DISTRIBUTED PRODUCTIONScope of this articleFor the purpose of this article, middleware should be seen as a collection of technologies whichcombine applications, or parts of applications, in a common and structured way. Middleware doesn’tachieve this alone. The management of middleware and the actual integration are of equal importance. For that reason, the P/MDP 1 Group set the scope of its work to be: System IntegrationArchitecturesMiddleware is but one (vital) element in this environment. A collection of tools is the glue.Scope of use: the value chainObjectives, strategy, tactics & managementPlanning& tsSell &prepareprogrammesProduceprogrammesAssembleand deliverserviceDistribute& transmitservicepackagesReceive, use& interactwith servicesDistribute& deliverprogrammesReceive, use& mmes ServicesMost broadcasters have a twotier value chain consisting ofServices and Programmes,illustrated in Fig. 3. Systemintegration architectures andthe different middleware technologies affect the processes inthis chain as well as thesupporting processes.Development, service, support & operation of: ArchiveAcquire & manage:RightsMarketing & promotion of:Programmes, services & brandThe integration architectureSystem operationsensures that information andAdministrationfunctions are accessible fromall the process stages – fromFigure 3early season planning to playTypical broadcaster's value chain (source: DR)out – via production, distributionand transmission. It is evenrelevant in the delivery platform where some data reaches the set-top box (e.g. the EPG).Broadcast-specific scopeIn many parts of our operations, we can benefit from integration techniques commonly used in otherindustries. But some areas are more critical to broadcasters than to other industries. This isbecause broadcasting deals with real-time rich media and not just common data in the form of textand numbers:1) Media asset managementStorage and archiving of our products: video, audio and interactive services. This is about thegeneral integration architecture between our assets and all of our other systems.2) The production environmentWhere can benefits and new ways of working be realized using integration technologies?3) The play-out environmentWe need to ensure data and content integrity from planning and scheduling, the production,through play-out, to the end user.4) Workflow managementWe need to connect the workflows which relate to the processes in the whole value chainacross systems or modules of functionality.1. P/MDP stands for “Middleware in Distributed programme Production”. The Group was created in autumn2003 and consisted of various EBU members. The main contributions to its report and this article camefrom: the BBC, BR, DR, the IRT, NRK, RAI, RTR, SVT and VRT.EBU TECHNICAL REVIEW – January 2005EBU Project Group P/MDP2 / 13

DISTRIBUTED PRODUCTIONWhat is so great about a system integration architecture?The arguments for your organization to establish and maintain a system integration architecture canbe summarized as follows (this is not a prioritised list): Flexibility: fast changing work-processes and workflows need flexible access to functions anddata. High barriers between systems are falling or changing to small boundaries as modularityis growing. A good system integration architecture and the appropriate middleware are helpingto provide this flexibility. Speed: where functions and data have to be used across systems, middleware speeds development and changes (in the long term, but not necessarily in the short term). Cost: the costs of building and maintaining unique one-to-one integration between systems israpidly growing for larger systems. Changes are expensive as they need to be applied to morethan just the primary system. A hub-and-spoke approach is more efficient. Middleware can actas a hub. Standardization: as methods and interfaces are re-used, fewer components and skills areneeded. Having a way of handling integration makes it easier to decide what you don’t need todo. Data integrity, reliability and robustness: as more and more data are used across systemsand different work processes, a system integration architecture and some middleware solutionsensure the data integrity across systems and situations of use. For horizontal systems, effortsspent in making the evolution reliable, secure and robust can be concentrated in the servicelayers rather than distributed over single applications. New products and services: easy and secure access to information across systems opens upopportunities for new products, especially on new delivery platforms. Prevent lock-in by vendors: you can replace a sub-system from vendor A by an equivalentsystem from vendor B, without having to reconfigure the rest of the system. Overview: as common ways of achieving integration are established, the organization gets abetter overview; both due to the standardization and to the needed repository of informationabout systems, functions, data, integrations, interfaces and metadata. This opens up opportunities for stronger change management routines. More specifically, the ability to search withindata across systems is improved. System planning: standardization allows technical planning departments to use tools whichwill make the planning process much easier. The focus will no longer be fixed on specific technical problems but much more on workflows, or rather on optimizing the workflows.Planning of complex broadcast-systems in future could be much more like “Lego ”. Training: an understanding of the workflows in which the operators are involved will becomemore necessary in the future than today. Therefore, not only will dedicated technical training fora device or software application be necessary, but also education on understanding the wholeproduction chain. This can become much simpler if the system architecture is based on genericand pre-defined processes rather than on proprietary structures.ExamplesSeveral EBU members have already gained experience with system integration architectures: Bayerischer Rundfunk is implementing an IT-based play-out and production centre. It encompasses a messaging-based system for system integration. Danish Radio is in the process of moving from a vertically to a horizontally integrated systemarchitecture. Standards and policies are established and both a broker and a concept forservice-oriented integration are in operation. This has been done in close connection with theimplementation of a DR metadata specification.EBU TECHNICAL REVIEW – January 2005EBU Project Group P/MDP3 / 13

DISTRIBUTED PRODUCTION BBC Research & Development is studying a completely horizontally integrated productionarchitecture. This project can be regarded as having the luxury of a “green field” situation,trying to utilise the benefits of horizontal system integration to its fullest extent.Key conceptsWe are used to working with isolated systems, interfaced by people. These are systems in co-existence. Today this is changing. The main task of system integration today is to turn co-existingsystems into co-operating systems, to collect systems into “super-systems”, when appropriate andefficient, and to optimize workflows.Introducing layersIn the late nineties the primary view was that one had to connectall systems or modules to each other directly. However, for alarge number of modules, this becomes almost unmanageable.The alternative is to connect the modules via some common glue.But this glue needs to be well-defined and widely accepted. Thedevelopment of a layered system architecture was a good firststep.PresentationApplicationLogicA system can be seen as split into layers in a horizontal architecture where each layer is independently accessible. Systemswithout this approach are referred to as vertical systems.DataSystemHow many layers?In this horizontal approach, the number of layers can differ.Therefore it is generally called an n-tier horizontal architecture,where n indicates the number of layers.Figure 4A layered systemUsually the design starts with three layers, each with its own tasks:1) Presentation layer- provides input & output for the system;2) Application Logic layer- the “brains” of the system;3) Data- the memory or content of the system (databases and fileservers).In many situations the concept can benefit from introducing a layer under the databases and fileservers, taking into account that physical storage can differ. This then allows you to change or scalethe storage without changing the data layer. So the fourth layer would be:4) Storage- the physical place where data are put.Open or closed systemsIt is important to realize that there might be no connection between the actual construction of asystem and how it appears or reacts to the outside world. A system using the latest “horizontal”technology 2 can still appear as a closed vertical system. The constraints may be imposed byunpublished or licensed interfaces, or by proprietary additions to known application interfaces(APIs), protocols, etc.2. Such as portal servers, web-based user interfaces, modularised business logic on application servers,and XML data servers.EBU TECHNICAL REVIEW – January 2005EBU Project Group P/MDP4 / 13

DISTRIBUTED PRODUCTIONOn the other hand, some older and non-layered systems have quite open interfaces, although thereis a tendency to refer to non-layered systems as vertical and closed.In the real world, we are dealing with a complex mixture of scenarios, and the solutions tend to beequally diverse.Component-based systemsThe layers mentioned are an abstraction, introduced to illustratelogical groups within systems, which in development and dailywork are useful to treat separately.PresentationWe can go one step further and describe systems which arecomponent-based 3.Application“ A component is a “black box” with known anddescribed interfaces which can be used withoutknowing anything about the internal structure orinternal functionality of the box. ”DataThese components can be grouped inside the above-mentionedlayers, although some components work across layers as well.As long as they stick to the rules in the above definition, thatshould not be a problem.SystemFigure 5A component-based systemSome of the components' interfaces should be openly accessible,but not necessarily all. This can be achieved in many ways and is one area where the need for standardization arises. The main point to remember is that systems should be component-based.Middleware: the glueTo solve the problems mentioned at the start of this article,something was needed tocombine all these layers andcomponents in a flexible and efficient way – some kind of “glue”.This was originally referred to asmiddleware, already suggestingthat no one really knew clearlywhat it was.1Glue in form of hub & spoke(one kind of middleware)12N2NHub.3N x (N-1) / 2 connectionsCheap to do with few modules, butcomplex and expensive to maintain.One of the early techniques toact as glue was the data-brokerFigure 6(a hub and some adapters toExample of gluethe systems). Without going into technical details, the conceptand expected benefits are illustrated in Fig. 6.Spokes(adapters)3N connectionsGreater effort to start with, buteasier to maintain and scale.Cheap to connect to.Fig. 7 combines the concept of middleware (glue) with the layers introduced earlier.The way the middleware interconnects the data layer with the application (logic) and the presentation to the user, is by using communication via interfaces. These interfaces form the boundaries of3. The term component-based is preferred to object-oriented because object-oriented is just one exampleof a component-based design, e.g. a procedural design can also be component-based.EBU TECHNICAL REVIEW – January 2005EBU Project Group P/MDP5 / 13

DISTRIBUTED PRODUCTIONSystem 1Where do components fit intothis?PresentationlayerApplicationlogicAs we discussed before,systems are nowadays moreand more component-based.Fig. 8 extends our layeredmodel with this notion.ApplicationlogicData, storage & networklayerFigure 7System concept, including interfaces(based on a diagram by Bob Edge, Thomson GVG)System ationlogicSystem 2The components are drawn aslittle puzzle pieces, indicatingthey have their own interfaces.This means we can speakabout interfaces at multiplelevels: for example, the interface to the data-layer or theinterface to a component withinthat layer. Components couldbe system components ormiddlewarecomponents.Working at component level isleading us to another kind ofglue – indeed another integration technique: Web services.MiddlewareSystem 2the “puzzle” pieces in Fig. 7. Itis important that the interfacesallow for communication in acommon, flexible, structured,efficient and re-useable way.Data, storage & networklayer componentMost commonintegration typesFigure 8Systems can be built out of many small componentsThe resulting system integration architecture depends on your particular business model (the products and operations) and yourparticular IT environment. Here we describe four common integration types. Message-based integration (copying data between systems)The same data is needed in different systems and an automated exchange mechanismprovides copies of data in these systems and ensures data-integrity while doing so. It can beset up as rule-based or event-based. Functionality is made available by an integration broker.System1System2 Data-carrying integration (Different systems access the same data)Data exists as a single copy in a single location. All systems work real-time on the sameinstance of data. Integration is achieved by agreement between the systems about the datamodel they use, and about a common logic for reading and updating data. This is provided by adata-carrying integration platform, primarily consisting of an application server and a database.EBU TECHNICAL REVIEW – January 2005EBU Project Group P/MDP6 / 13

DISTRIBUTED PRODUCTIONSystem1System2 Portal integration (Wrapping of multiple user interfaces into a single one)Employed where users require a more integrated user dialogue or interface than is provided bythe separate applications. Where individual customizing of the user interface is required, orwhere certain functions should be automated, or where single logon is needed . an integrationof the user interface is necessary. Web-based user interfaces provide integration via a portal.System1System2 Workflow (Mechanisms to co-ordinate events in systems)Refers to automating or semi-automating of workflows, by integrating functionality and data. Tosupport tightly connected or integrated business processes it should be possible in advance todefine which actions need to take place when certain data are exchanged or updated. Workflow mechanisms can be found in each of the three former described integration types.Process Process12Process verThe above integration architecture concepts are related to planning, production, and play-outsystems, as well as the administrative back-office, financial and accounting systems. When itcomes to executing play-out or transfer of real-time, rich multimedia files (e.g. a 50 Mbit/s video file)other tools and technologies than described above are used. However, the management, controland reporting of this can be handled like any other kind of data in any other industry.Middleware services for broadcastersWhen setting up broadcasting facilities, this often means we have to solve recurrent problems. Wehave learned that concepts and technologies are available which enable us to reuse solutions forthose problems. An example of such a concept is the Services-Oriented Architecture.When using a Services-Oriented Architecture, one may ask the question whether there are specialmiddleware services needed for broadcasting.Let us first try to find out what middleware services for broadcasters actually are.EBU TECHNICAL REVIEW – January 2005EBU Project Group P/MDP7 / 13

DISTRIBUTED PRODUCTIONAbbreviationsAAFAdvanced Authoring FormatAPIApplication Programming InterfaceMXFMaterial eXchange FormatNTSCNational Television System Committee (USA)PALPhase Alternation LineSMPTE Society of Motion Picture and TelevisionEngineersUMLUnified Modelling LanguageVTRVideo Tape RecorderUnderstanding servicesA service is basically a special system component which can be invoked by other componentswanting a certain task to be performed on certain data, e.g. an authentication service which isgranting or denying access to resources. To be able to do this, it is necessary for the invokingcomponents to know what operations are available and how to ask for these operations. The specification of these is called a service interface specification.How should such a service interface be specified?The most efficient and reliable technique is to use modellingmethods; tools and technologies supporting the task of definingthe boundaries and behaviours of software components.A useful strategy in designing a service architecture is to distinguish services into those specific to your industry, the broadcastdomain, and those which are not specific to a particular industry,also known as pervasive services.Broadcast domain modelA domain model describes the functional elements used in orderto provide solutions specific to a business domain, in our casethe broadcast domain. It should not describe the internal workings of the (functional) elements, nor provide much detail aboutthem. Instead it must just encapsulate the essential aspectsrelated to the domain. The functional description should belimited to that of the interfaces and should include the static definition as well as the dynamic behaviour.Figure 9This figure shows differentindustry domains, including a(currently hypothetical)Broadcast Domain. The“BR”, “DR” and “BBC” slicesillustrate different broadcasters' usage of it.A broadcast domain model should define all “bread & butter” services specific to the broadcast domain, e.g.: Essence transcoding;Speech-to-text transcription;Material copies management;Material stream control.The task for the users within a certain business domain is to specify their domain model and to benefitfrom existing standards. This may be done in collaboration with vendors. The users' involvement isparticularly important since the domain model reflects the business which the users know best.Pervasive servicesPervasive services provide functionality which is not specific to the broadcasting needs. Pervasiveservices are reusable by other services and applications. The use of pervasive services allows us tobenefit from solutions worked out in other industries, for example:EBU TECHNICAL REVIEW – January 2005EBU Project Group P/MDP8 / 13

DISTRIBUTED PRODUCTION Authentication; Session management; Time references; Workflow management.For the pervasive services, the users should select available services and implementing technologies which are appropriate to support their business. The proper requirements have to be applied tothe services (e.g. the bandwidth needed and the desired network characteristics), to guarantee theymeet the users' specific operating conditions.Different viewsThere are different types of players involved in the development of IT-based TV programme production systems. These different roles require different views of the system which is modelled, forexample: Planning Engineer View – a set of service definitions and high-level descriptions of their interactions. System Integrator View – a description of how to use a service.The System Integrator View has to represent a high-level view of the interfaces of the servicesand their dependencies on other services (e.g. the pervasive services). System Developer View – all the detailed information about the components to be implemented.Modelling in the broadcast environmentModelling is a widely used method in IT system design. As the broadcasting industry is makingmore and more use of software solutions, the use of modelling techniques becomes more importantas well.The use of models to describe technical systems is not a new idea. Block diagrams and connectionschemes are models also. The difference is that, with IT modelling techniques, the diagrams represent software systems and not necessarily physical objects.Unified Modelling LanguageUML provides different types of diagrams for the static design and the dynamic behaviour of ITsystems. A brief introduction to the use of different UML diagrams is provided below, using anexample which describes the process of recording with a VTR. Arguably the best known modellingmethodology is called UML (Unified Modelling Language). In the last 10 years it has establisheditself as the predominant formal notation for the description of IT projects.Use Case diagramA Use Case Diagram gives an overview of what happens in a 'business process'. It does not explainhow the result is achieved, but only what happens, regardless of any specific product or solution.Use Case diagrams are used to describe the interaction of actors and systems in two ways:1) A drawing with formal icons for actors (representing roles, humans or systems, but never a jobdescription) and activities.EBU TECHNICAL REVIEW – January 2005EBU Project Group P/MDP9 / 13

DISTRIBUTED PRODUCTION2) A formalised text part definingthe actors, the preconditionsand the business rules relevant for the Use Case.VTRPlaySeveral Use Cases can be collectedinto logical groups to form a drawingof a Use Case scenario, includingtext parts for each Use Case.There are frameworks which givesome guidance in how to deal withsystem analysis and description.For more information look for MDAin [1] and Zachmann in [2].RecordStopPlay-out systemFigure 10Use Case scenario for use of a video systemThe vendors' perspectiveTypes of vendorsCurrent broadcast-system vendors form a heterogeneous group. Basically, there is a group of traditional broadcast vendors and a group of IT-originated companies, each approaching the customerfrom a different angle.Some traditional broadcast parties seem to have a stronger tendency to think in silo-based verticalsystems, while the other extreme is formed by some IT companies entering the market virtuallywithout knowledge of broadcasting, but equipped with a bag overflowing of IT standards nomenclature. The first can be argued as undesirable due to its inherit “lock-in” character, while the secondmay not prove any better as there still could be a “data lock-in”. That is: the system may be open,but if you don't understand the data model inside, you still cannot use it.The aforegoing sketches out the extremes but, in practice, both “camps” are moving towards abetter understanding of the customers' requirements (as is the customer himself). It seems thatsome smaller companies in particular have a good nose for understanding the customers' specificrequirements.Key observationsVendors are opening up their systemsOpenness is the norm in IT, but is only slowly becoming practice in the broadcast domain. Onereason for this is the large influence of traditional wholesale providers. The main driver to open upis, quite simply, pressure from customers demanding openness.Vendors put themselves in the centreWhen vendors claim to interoperate with others, they often put themselves in the centre of thesystem. Instead of offering a single service to the outside world (like plug-ins for your browser), theyargue that “others should interface with us”.Middleware is known, but interoperability limitedVendors are using middleware, but often only in their own product suites. For commercial reasons itis not exposed to the outside world. Another reason is that there is a lack of semantics (“What doesthe data this service provides mean?”). The net result is that interoperability between broadcastapplications using middleware is limited.EBU TECHNICAL REVIEW – January 2005EBU Project Group P/MDP10 / 13

DISTRIBUTED PRODUCTIONVendors are reactiveVendors are reactive and not proactive; they will only implement when there is a demand (customerproject driven). The broadcasting market is relatively small and each broadcaster poses differentcustomisation requirements and/or a different expression of the same requirements. Some vendorsalso remark that the broadcast industry is special in that much R&D is done by the customers themselves, which is not the case in the IT industry.Middleware reliability is ok, but at a costIn general the vendors do not see difficulties with middleware reliability, but they do warn it may notbe the best solution for each problem. The engineering effort to provide high reliability may behigher than for traditional systems. Also, reliability should not be the single reason to use middleware (but, for example, scalability reliability).Managing system integrationThe cost for system integration can differ dramatically depending on how the work is managed. Atone end of the spectrum every integration is a stand alone solution with only one person capable ofhandling development, maintenance, etc. At the other end, a corporate integration architecture isimplemented.To boost economy, quality, capability, etc there is a need for standardization and streamlining of theintegration efforts. It is important to have a strategic scope that spans more than the currentprojects. If possible this responsibility and governance should be centralized within the company –the consultant company Gartner has named such a function Integration Competency Center, ICC.As already stated, this is very much about managing and control. To deal with all this increasingcomplexity and variety is much like working with quality assurance programs:“ Describe what you do and do what you describe. ”To move away from the situation where specialists are formed around each isolated system, and intoa world with a large variety of skills across systems, components and middleware, is a huge effort.StandardizationThere are two areas where standardization efforts would be welcome:1) A common (core) metadata modelThere is a lack of a common (core) metadata model. This is the top priority issue for mostvendors. Existing metadata specifications have not been very successful and are seen as toolarge/vague to apply. There is a strong demand for a more modular and business-orientedapproach: e.g. a simple subset with a coherent structure. This must include the semantics( unambiguous meaning) of the elements it provides.2) “Bread & butter” servicesThe business services commonly used by broadcasters, such as ingest, play-out and scheduling are the broadcasters' “bread & butter” services. It seems that standardization of middleware / system integration architectures could best take place at this level. These primary services could all be broken down into non-competitive and common technical services (play,record, transfer, transcode, etc.) and all have a need to interface. Besides services, relevantbusiness objects should be specified as well. The specifications should not dictate technology,EBU TECHNICAL REVIEW – January 2005EBU Project Group P/MDP11 / 13

DISTRIBUTED PRODUCTIONbut allow for competition. Once the services are defined, it should be investigated if toolkits,APIs, etc., would be of value.Who should standardize this?A combination of an IT organization (e.g. OMG [1]), together with a broadcast organization (e.g.the EBU or SMPTE) would be preferable. Some manufacturers noted that they would like tosee a community effort using more Internet-based tools and less physical meetings, whichwould also allow smaller parties to be involved: e.g. an open forum on interoperability.When should it

Several EBU members have already gained experience with system integration architectures: Bayerischer Rundfunk is implementing an IT-based play-o ut and production centre. It encom-passes a messaging-based system for system integration. Danish Radio is in the process of moving from a vertically to a horizontally integrated system architecture.