Transcription
jKQL User’s GuideVersion 1.3 2021 Nastel Technologies, Inc.
Document Title: jKQL User’s GuideDocument Release Date: March 2021Document Number: JKQLUG13.002Published by:Research & DevelopmentNastel Technologies, Inc.88 Sunnyside Blvd, Suite 101Plainview, NY 11803Copyright 2021. All rights reserved. No part of the contents of this document may be produced or transmitted in any form,or by any means without the written permission of Nastel Technologies.Confidentiality Statement: The information within this media is proprietary in nature and is the sole property of NastelTechnologies, Inc. All products and information developed by Nastel are intended for limited distribution to authorized Nastelemployees, licensed clients, and authorized users. This information (including software, electronic and printed media) is not tobe copied or distributed in any form without the expressed written permission from Nastel Technologies, Inc.Document HistoryRelease DateDocument NumberSummaryJuly 2019JKQLUG11.001Initial release.September 2019JKQLUG12.001Updates throughout for version 1.2. Added new sections 3.4.16,6.4 and 6.5. Machine Learning updates in section 3.3.5.November 2019JKQLUG12.002Add “Volumes” to Table 33.February 2021JKQLUG13.001Updates throughout for version 1.3. Add Chapter 8: ExtendingjKQL. Update Table 20.March 2021JKQLUG13.002Add Note box to section 4.7.
jKQL User’s GuideTable of ContentsTable of ContentsCHAPTER 1: INTRODUCTION . 71.1 HOW THIS GUIDE IS ORGANIZED .7CHAPTER 2: DATA MODEL . 92.1 DEFINITIONS .92.2 ITEM TYPE OVERVIEW.92.3 FIELDS .13CHAPTER 3: JKQL . 173.1 DATA TYPES.173.1.1 Maps .173.1.2 Variants .183.2 JKQL EXPRESSIONS .183.2.1 Literals .183.2.2 Date and Time Expressions .213.2.3 Operators .243.3 FUNCTIONS.303.3.1 Built-in Scalar Functions .303.3.2 Built-in Spanning Functions .353.3.3 Built-in Aggregate Functions .363.3.4 Built-in Analytic Functions .403.3.5 Machine Learning Functions .423.4 STATEMENT SYNTAX .463.4.1 Common Elements .463.4.2 SignIn.503.4.3 Use .513.4.4 Get .513.4.5 Find .563.4.6 Compare .573.4.7 Insert, Update, Upsert .583.4.8 Delete .593.4.9 Subscribe .593.4.10 Unsubscribe .603.4.11 Reset.613.4.12 Enable / Disable .613.4.13 Grant .613.4.14 Revoke .623.4.15 Purge .633.4.16 Compute .633.4.17 Invoke .643.4.18 Train .653.5 JKQL FIELDS .653.5.1 Primary Key Fields .653.5.2 Fully-Qualified Name (FQN) .663.5.3 Criteria.663.5.4 Objectives .663.5.5 SetSequence .683.5.6 jKQL (Generic jKQL Statement) .683.5.7 EffectiveRole .68CHAPTER 4: CONCEPTS. 694.1 SEARCHING.694.2 SET MEMBERSHIP .70JKQLUG13.0023 2021 Nastel Technologies, Inc.
jKQL User’s GuideTable of Contents4.2.1 Objectives .704.3 RELATIVES .714.3.1 Encloses .714.3.2 Send To .714.3.3 Acts On .724.3.4 Correlated .724.4 COMPUTED FIELDS .724.5 SUBSCRIPTIONS .734.6 ALERTS .734.6.1 Provider Type .734.6.2 Provider .734.6.3 Action .754.6.4 Trigger .754.6.5 Formatting .764.7 VIEWS AND VIEWTEMPLATES .794.7.1 View Queries .794.7.2 Schedule .804.7.3 Result History .804.7.4 Options .81CHAPTER 5: ACCESS CONTROL . 835.1 LEVELS .835.2 EFFECTIVE ROLES .835.3 ENTITIES .835.4 ITEMS .835.5 MEMBERSHIP.845.6 ADMINISTRATORS .845.7 OPERATION .845.8 INQUIRIES.85CHAPTER 6: ADMINISTRATION . 876.1 DATA MODEL .876.2 JKQL FIELDS .876.2.1 Admin Item Names .876.2.2 Access Token Options .876.2.3 Repository Options .886.2.2 Access Token Quotas .896.3 ADMIN STATEMENT SYNTAX .896.3.1 Common Elements .896.3.2 Create .896.3.3 Alter .896.3.4 Drop.906.4 VOLUMES .906.5 ACCESS TOKENS .91CHAPTER 7: LICENSING . 957.1 DATA MODEL .957.1.1 Features .957.1.2 Effective License .957.2 JKQL FIELDS .967.2.1 License .967.2.2 Features .967.2.3 Quotas .967.2.4 Effective Values .977.3 LOADING STATEMENT SYNTAX .98CHAPTER 8: EXTENDING JKQL . 101JKQLUG13.0024 2021 Nastel Technologies, Inc.
jKQL User’s GuideTable of Contents8.1 EXTERNAL DATA SOURCE .1018.1.1 External Data Source Definition .1018.1.2 External Field Types .1028.1.3 External Item Types .1028.1.4 External Item Fields .1038.1.5 Synonyms .1048.1.6 Configuration .1048.1.7 Example .1058.2 EXTERNAL ACTION PROVIDER TYPES .1068.2.1 Provider Type Definition .1068.2.2 Provider Type Properties .1068.2.3 Configuration .1078.2.4 Example .1078.3 EXTERNAL JKQL FUNCTIONS .1078.3.1 Function Definition .1078.3.2 Configuration .1088.3.3 Example .108INDEX . 109JKQLUG13.0025 2021 Nastel Technologies, Inc.
jKQL User’s GuideTable of ContentsThis page intentionally left blankJKQLUG13.0026 2021 Nastel Technologies, Inc.
jKQL User’s GuideChapter 1: IntroductionChapter 1: IntroductionWelcome to the jKQL User’s Guide. jKool Query Language (jKQL) defines the syntax of statements usedfor manipulating data while using Nastel XRay.1.1 How this Guide is OrganizedChapter 1:Introduction to the jKQL User’s GuideChapter 2:Data model descriptionChapter 3:Data types, jKQL expressions and functions are presentedChapter 4:Explanation of conceptsChapter 5:Information on access controlChapter 6:Administration data model is explainedChapter 7:Provides information on licensingChapter 8:Information on adding user-defined elementsIndex:Contains document indexJKQLUG13.0027 2021 Nastel Technologies, Inc.
jKQL User’s GuideChapter 1: IntroductionThis page intentionally left blankJKQLUG13.0028 2021 Nastel Technologies, Inc.
jKQL User’s GuideChapter 2: Data ModelChapter 2: Data Model2.1 DefinitionsThe Data Model contains the following terms: Items – these are what the statements act on. There are 2 classes of Items:o Physical – these items correspond to actual data store items. Physical items can beinserted/updated and deleted, in addition to queried and compared.o Logical – these Items are derived from Physical items. Logical items can only be queriedand compared.Fields – represent the properties of an item. Each item supports a defined set of fields, plus aproperties field, which is a map of {key,value} pairs, allowing for custom properties.2.2 Item Type OverviewThe data model consists of the following item types.JKQLUG13.0029 2021 Nastel Technologies, Inc.
jKQL User’s GuideChapter 2: Data ModelTable 1. Item TypesActivitiesA collection of related Events and/or sub-activities, as identified byinstrumented application.EventsAn Event represents a distinct application operation or statement,optionally containing associated message data.SnapshotsA Snapshot is a collection of information, as key/value pairs, identifiedby name and the time the information was collected.SourcesA Source represents the origin of Events, Activities, and Snapshots. ASource is identified by a string known as its Fully-Qualified Name (FQN,See Fully-Qualified Name (FQN) for details), which defines its ENCLOSESrelationships (See Relatives).A Resource represents the object that Events, Activities and Snapshotsact on, or execute within. It also can be using an FQN string (See FullyQualified Name (FQN)), which will identify the type of resource, as wellas its name. Supported resource types are:ResourcesDictionariesJKQLUG13.002 DATASTORECACHESERVICEQUEUEFILEA Dictionary entry represents a free-form record. It is essentially anamed collection of key/value pairs. The specific keys are applicationand/or user-dependent. The type of the keys is STRING. The values canbe of BOOLEAN, INTEGER, STRING or TIMESTAMP. Dictionary entriesdiffer from the others that they are not tied to a specific repository.They can be associated with several repositories, or not associated withany repositories.10 2021 Nastel Technologies, Inc.
jKQL User’s GuideChapter 2: Data ModelA Set is used to identify Activities and Events that meet specific criteria,as well as to define the objectives, or conditions, that the items thatmatch the set should meet. The critical attributes of a Set are: Sets Criteria – defines the conditions that must be met for inclusionin the set. See Criteria for specifics on format of set condition.Objectives – define the set of conditions that must be met (orshould not be met) by members of the Set. See Objectives forspecifics on defining objectives.Scope – defines how to include Activities and Events into theset, and is one of:o Singular – Only the Activities and Events that directlymatch the Set Criteria are included in the set. Thesetypes of sets are commonly referred to as“Milestones”.o Related – All Activities and Events that are “related”(stitched to) to those that directly match the Criteriaare included. These types of sets are commonlyreferred to as “Groups”.Sequence – for Related sets, defines the expected sequence ofSingular subsets.Relatives define the observed relationships between event and activitySources, as well as the relationships between Singular Sets. Thefollowing relationships are identified: Relatives JKQLUG13.002ENCLOSE – parent Source encloses, or contains, the childSource (e.g. DataCenter encloses Server indicates that thespecified Server is in the specified DataCenter)SEND TO – parent Source sends a data message to the childSource (e.g. Application A sends to Application B indicates thatApplication A has sent a message and Application B hasreceived the same message), or parent Set sends a datamessage to child Set.ACTS ON – parent Source “acts on” or “manipulates” childResource (e.g. Application A acts on Resource B). This can beone of the subtypes below:o ACTS ON WRITE – parent Source wrote to childResourceo ACTS ON READ – parent Source read from childResource11 2021 Nastel Technologies, Inc.
jKQL User’s GuideInput Data RulesChapter 2: Data ModelInput data rules allow for field value calculations at data ingest time.Both built-in fields and custom properties can be computed from otherbuilt-in fields or custom properties, and also from other computedfields. The computed value could be used to replace any value that’salready there, or appended to any existing value(s). By default, theinput data rules are applied to all incoming Activities, Events, andSnapshots. However, the rules can have an optional criteria defined, sothat the rules are only applied to specific input data.A Provider is an instance of the implementation of a type of provider,which represents definition for the particular type of action to execute,generally in response to a trigger condition. A Provider Type defines aset of supported properties to control its execution. jKQL includes thefollowing defined Provider Types: ProvidersFileWriter – defines implementation of writing information to afile Emailer – defines the implementation of sending information inan emailA Provider definition would represent an instance of one of thesetypes, optionally with the default value for one or more of the ProviderType’s properties. For example, a Provider named “FileAppender”could be defined as an instance of FileWriter, with the value of theFileWriter “Append” set to TRUE, so that, when data is written to thefile, it is appended to the current contents of the file.ActionsAn Action represents a task to execute, generally in response to atrigger condition, and is an instance of a particular Provider (NOTProvider Type), defining the values required by the specified Provider’sType. For example, an Action named “WriteToLogFile” could bedefined that would use Provider “FileAppender”, setting the FileWriterproperty “FileName” to “/tmp/trigger.log”. Triggers that reference thisaction would cause data to be appended to file “/tmp/trigger.log’.TriggersA Trigger represents a condition to test for, along with the Actions totake when the condition is met. The condition is specified using sameformat as in Subscribe (without the SUBSCRIBE TO and SHOW AS).JobsJob entries represent the state of past, current, and scheduled jobs.Log entries are records of actions occurring in system. The followinglog categories are supported: LogsJKQLUG13.002 ERROR – errors that occurred during the processing of jobs,data streaming, user queriesQUERY – user queries executedSUBSCRIBE – user subscriptions submitted and canceledTRIGGER – triggers started and stoppedGENERAL – other items not fitting into the above categories12 2021 Nastel Technologies, Inc.
jKQL User’s GuideChapter 2: Data ModelViewTemplatesA ViewTemplate defines a generic template for a View. It defines ajKQL query (optionally with substitutable parameters). See Views andView Templates for details.ViewsA View represents a named query, providing a fixed result structure.The implementation is analogous to an SQL Materialized View. Thequery is either defined explicitly in the View definition itself, or isinherited from the ViewTemplate on which it is based. In the case ofthe latter, the View definition would include bindings for the specificparameters required by ViewTemplate. Views are evaluated on adefined interval, with the results cached for quick retrieval. See Viewsand View Templates for details.MLModelsMLModels are used for Machine Learning. In order to run MachineLearning on data,
jKQL User's Guide Table of Contents JKQLUG13.002 3 2021 Nastel Technologies, Inc. Table of Contents CHAPTER 1: INTRODUCTION
