Transcription
LLRF Embedded DevelopmentBrian Chase, Joshua Einstein-CurtisControls Modernization Group28 September 2018
Introduction LLRF is a high-speed, high-precision real time system. For the FNAL accelerator infrastructure, have relatively highdata rates– Similar to beam instrumentation– Necessitated builds of a Labview client for commissioning,debug, and diagnostics (Backdoor/custom)Example:CMTS has 4 ADC and 2 DAC channels per cavity. In addition to the raw data, there are also thebaseband IQ and amplitude/phase signals, error, and controller intermediate stages2 * (4*2 2 10) * 65 MSPS * 32-bit 82 Gbps max82/65 1.26 Gbps if 1 MSPS sampling1.26 Gbps / 8 * 360 56 TB/hour0.02 * 56 1.12 TB/hourif only 20 ms of 1MSPS data every second2LLRF Embedded Development9/27/2018
Real-time Needs Hard Realtime LLRF (timescale-dependent, of course)– Failures not allowed Bucket-resolution timestamping on data Necessary to do extensive hardware testing or simulation(Qemu, ARM simulator)– Good debugging tools a necessity Utilization and overloading– Need for optimizations and good compilers3LLRF Embedded Development9/27/2018
vxWorks MVME2400, MVME5500 with Kinetic Systems backplane VxWorks 5.5, 6.1, 6.4, 6.9– 5.5 running on MI, FAST, 6.4 on Recycler, 6.9 dev system provided to controls Power PC 603e, 750 or 7457 Standardized communication between front ends enables fasterdevelopment. Shared libraries, etc.– https://www-bd.fnal.gov/controls/micro p/mooc project/welcome.html Protocol Compiler– Custom for each device? Possibly leads to complexity and versioningissues– Each device has own receive stack, and each group develops theirown4LLRF Embedded Development9/27/2018
Current ModelLLRF Frontendsys -to-dabbelModificationScriptsAcnet and Front-end independently record and manage variables5LLRF Embedded Development9/27/2018
SoC MFC6LLRF Embedded Development9/27/2018
LLRF Control and Timing VUCD–––––Developed in mid-ninetiesNo replacement for current systems availableTiming and data decoding using backplane triggersMulti-timer systemNo planned replacement for current/deployed systems Bucket resolution and accurate timestamping needed forfuture diagnostics7LLRF Embedded Development9/27/2018
Build Environment nova build environment (esd tools) well developed– LLRF projects are all there, including configuration files for newSoC systems (spreadsheet register maps and defaults) Need for a well-developed upgrade plan across departments Need for supporting both legacy systems and new systemsusing same tools? Significant increases in performance with each controller, butprocessor-specific programming can make porting difficult8LLRF Embedded Development9/27/2018
Revision Control Currently part of build toolchains on nova One of the largest issues we’ve run in to is revision andversioning– Separate versions for firmware, software, libraries– Storing git commits in f/w and software works well– Xilinx allows for same bitfile from same files routing/placement hash/seed is based on files included in project Enforcing this across all project levels is a challenge PLC versioning? Software versioning? HDL versioning?9LLRF Embedded Development9/27/2018
Revision Control LBL/LCLS-II and CERN have self-describingfirmware/software– Register maps in control system and front-ends match gzip’d JSON in memory on devices CERN’s FESA and cheby Acnet database TFTP/remote image loading is crucial– Limit wear on flash (limited write) devices Remote/central logging is crucial– See above Verification and Continuous Integration are valuable tools10LLRF Embedded Development9/27/2018
What is plan for future? Responsibility of each department? High-speed data and diagnostics– artDAQ? FTP? Something new?– DAQ analysis and storage– High-speed means what for each system? Data lifetime? Scope capabilities Event identification Design lifetime– 20 years? 5 years? 1 year? Tool availability and automation– Build system for non-experts– Nova replacement? Command-line functionality?11LLRF Embedded Development9/27/2018
Looking Forward What will be available in 2 years? 5 years? 10 years? Need a long-term plan Due to few resources at developer level, really need processand procedure directed from managment Hardware - should there be a standard developed acrossgroups?– Shared FPGA/Processor boards?– Crate standards?– Physical interfaces?12LLRF Embedded Development9/27/2018
Xilinx Zynq Ultrascale RFSoC Designed for cell basestations, radar, military Single-chip includes allnecessary for an RF system,but might not meetperformance requirements ofmodern accelerators.Separate application andreal-time processorsShared memory makes forvery powerful, highbandwidth toolset c/rfsoc.html13LLRF Embedded Development9/27/2018
Conclusion Concerns for future– DAQ and timing/precision– Data retention and high-speed data management– Revision Control and Build Environment are significantconcerns– Ease of integrating in to control system: shared protocols,development architectures, etc Need a multi-year, division-wide path forward– While still supporting legacy systems Thanks!14LLRF Embedded Development9/27/2018
Extra Slides15LLRF Embedded Development9/27/2018
Switching to Fiber 16Fiber benefits outweigh slightly increased costDecreased powerHigher bandwidthLow cost per bandwidthRelative ease of implementation given available toolsLLRF Embedded Development9/27/2018
Communication Proposals 9.027 MHz RF synchronous frame rate– Local clock source needed on each device 1300 MHz harmonic as data rateNeed known frame size (ie 128-bit)Previous State, Events, Beam pattern for next groupSingle-source starSend as timing/events -- Return line as MPS– Data concentrators? Throw out proposal 3 links – DAQ, timing/MPS, control17LLRF Embedded Development9/27/2018
18LLRF Embedded Development9/27/2018
Intel Cyclone V and Arria 1019LLRF Embedded Development9/27/2018
SoC MFC Test DAQ SystemCourtesy Ed Cullerton20LLRF Embedded Development9/27/2018
Arm Product cation.pdf21LLRF Embedded Development9/27/2018
Microsemi SmartFusion2Single Event Upset Immune design22LLRF Embedded Development9/27/2018
NXPKinetis K60LPC17xxi.MX23LLRF Embedded Development9/27/2018
Why not Pis? The question is really: what does real-time mean?– Bucket-level resolution and repeatability?– TCLK decoding accuracy? Event decoding accuracy? What runs at each layer?– Firmware/embedded RTOS– Software/application layer Lifetime and long-term support– Open Hardware Interconnect standards– SPI I2C support? (1 SPI, 1 I2C, 1 UART)– Standard interconnects?– Bus/backplane protocols?24LLRF Embedded Development9/27/2018
DSP LLRF requires higher-performance DSP and DAQ Newer FPGAs are offering dedicated floating-point blocks– Require use of special tools or overloading standard operators Integrated simulation and verification environment beingmore important Collaboration requires the ability to transfer IP easily25LLRF Embedded Development9/27/2018
SoM s.html26LLRF Embedded Development9/27/2018
Linux (and variants) Linux w/ preemptRT RTAI (Linux as task) NI Real-Time Linux (dual-schedulers)27LLRF Embedded Development9/27/2018
RTOS FreeRTOS -- https://www.freertos.org/Contiki (IoT) -- http://www.contiki-os.org/MicroC/OS-IIxMbed (IoT)RTEMSvxWorks– Now a Linux variant MQX28LLRF Embedded Development9/27/2018
MPC603e29LLRF Embedded Development9/27/2018
MPC75030LLRF Embedded Development9/27/2018
MPC745731LLRF Embedded Development9/27/2018
Processing Models IOC Epics– “Dumb” front-end with managing IOC– Subscriber model Crate processors– Slot 0 manages all cards in crate. Advanced data processinghappens ? NADs– Multiple models available Acnet Requirements on who does each portion is not defined– Controls communication (MOOC(AN), libacnet, erlang, PC)– Debug communication (Labview, Python)– Front-end intelligence32LLRF Embedded Development9/27/2018
Real-time Needs Hard Realtime LLRF (timescale-dependent, of course) -Failures not allowed Bucket-resolution timestamping on data Necessary to do extensive hardware testing or simulation
