Script and Job Authoring

Runnables

Script and Jobs are Saga Runnables. A Saga Runnable has javascript that is invoked when certain conditions are met. Saga Runnables can create saga properties, query the database, communicate with external systems, and more. Each Saga Runnable invocation creates a Runnable Call that captures performance, logging, errors and calling context.

Runnable Types

Saga distinguishes between different invocation contexts depending on script type or job.

Script - Property Change

Property change scripts are called when a new property is created or a property is updated. The calling ontext is the property and parent javascript object (either bot, user or globals depending on property).

async (property, parent) => {
  //begin runnable code
  console.log(property.name,property.value)
  //end runnable code
}

Acceptable path values:

• /(bots|users|globals)/properties/PROPERTY_NAME - responds to a property with the given name for either bots,users,globals

Script - Signal Event

Signal scripts are called when a signal is emitted. The calling context is the signal and parent javascript object (either bot, user or globals depending on signal).

async (signal, parent) => {
  //begin runnable code
  console.log(signal.name,signal.value)
  //end runnable code
}

Acceptable path values:

• /(bots|users|globals)/signals/PROPERTY_NAME - responds to a signal with the given name for either bots,users,globals

Script - Message Event

When a message is created, the calling context always contains a message object. The other available variables depend on the path, and are detailed below.

async (sender, sender_type, bot, message) => {
  //begin runnable code
  const chatscript = require("chatscript");
  const response = await chatscript.send_message(
    {/**..**/}
  )
  //create outgoing message
  await Message.send({
    sender_id: bot._id,
    sender_type: 'Bot',
    user_ids: sender_type === "User" ? [sender._id] : [],
    bot_ids: sender_type === "Bot" ? [sender._id] : [],
    body: response.body ,
  });  
  //end runnable code
}

Acceptable path values:

Note that sender_type and recipient_type, if available, will always be either "User" or "Bot"

• /users/messages/sent (user, message) - triggered when a user sends a message, even if that message has no recipients
• /users/messages/incoming (sender, sender_type, user, message) - triggered when a user receives a message
• /users/messages/outgoing (user, recipient, recipient_type, message) - triggered when a user sends a message, once for each user -> recipient pair
  • For example, if user A sends a message to user B and bot B, this script will be triggered twice, for user A -> user B, and user A -> bot B
• /bots/messages/sent (bot, message) - triggered when a bot sends a message, even if that message has no recipients
• /bots/messages/incoming (sender, sender_type, bot, message) - triggered when a bot receives a message
• /bots/messages/outgoing (bot, recipient, recipient_type, message) - triggered when a bot sends a message, once for each bot -> recipient pair
  • For example, if bot A sends a message to user B and bot B, this script will be triggered twice, for bot A -> user B, and bot A -> bot B

Script - User Lifecycle Event

When a User Lifecycle Event occurs, the calling context is the user javascript object.

async (user) => {
  //begin runnable code
  const email = user.email;
  const sendgrid = require('@sendgrid/mail');
  sendgrid.setApiKey({/**...**/});
  sendgrid.send({/**...**/})
  //end runnable code
}

Acceptable path values:

• /users/events/create - user was created
• /users/events/verify - user has been verified
• /users/events/login" - user logged in
• /users/events/change_password_request - user wants to change password
• /users/events/change_password_verified - users password has been changed
• /users/events/change_email_request - user wants to change email
• /users/events/change_email_verified - users email has been changed

Script - Custom HTTP Call

When a custom http is called, the calling context is the request and response. Both are express objects and not plain javascript objects as the context's before.

async (request, response) => {
  //begin runnable code
  const count = await User.countDocuments({/**..**/});
  response.json({count})
  //end runnable code
}

Script - Socket Connection Event

When a Socket.IO connection is established between a user and Saga, the calling context is the addressed user.

async (user) => {
  //begin runnable code
  await UserProperty.add("connected_at", Date.now(), user._id);
  //end runnable code
}

Acceptable path values:

• /sockets/connections/open - triggered when a socket connection opens
• /sockets/connections/close - triggered when a socket connection closes

Job Event

A job invocation context is empty since it's time based.

async () => {
  //begin runnable code
  const bot = await Bot.find({name:"solarboy"},{_id:1},{lean:true})
  await BotProperty.add("heartbeat",1, bot._id,{inline:true})
  //end runnable code
}

Mutex

A mutex is a mutually exclusive control feature to ensure that there is only one active script call per parent. When a script has a mutex_time the script will not be called again for a parent until mutex_time milliseconds have passed. It can be set for scripts that process properties, signals and messages. The lock can be released programmatically in a script as well by calling releaseMutexLock(). The combination of releaseMutexLock() and an upper mutex_time time ensures that even when scripts or instances crash the lock can be released.

A blocked call will not appear in the statistics and does not generate a runnable call datum.

Custom Mutex

If the mutex time is not the same for parents the mutex lock functionality can be used programmatically, this however creates stats, a runnable call datum like any other invocation and are not handled differently in replay. The two functions below are available in property, signal and message scripts.

isBlockedByCustomMutexLock(name,mutex_time) this promise function returns true if there is a timed mutex with the given name. If there is none it returns false and sets the timed mutex lock for the given mutex_time. As with the regular mutex the lock automatically expires once the time has passed.

releaseCustomMutexLock(name) releases the custom mutext lock for given name.

async (parent,property) =>  {
  //begin runnable code
  const time = property.value.settings.time_to_wait
  if(!await isBlockedByCustomMutexLock("blocker",time)){
    //code to handled unblocked scenario goes here
  } else{
    //code to handled blocked scenario goes here  
  }
  
}

Filtering

Scripts have additional settings parent_name_criteria and parent_properties_criteria. They are used for filtering of scripts calls, by being able to exclude or include based parent properties and names.

Let's assume an incoming location property for user.

{
  "location":{
   "latitude": 40.75286,
   "longitude": -73.97929
  }
}

We have two scripts available whose purpose is to find close friends and ping them. one is using google and the other one apple depending on how the user signed up. When a user signed up with apple we gave the user apple_settings and for goggle google_settings.

So we have two scripts that are bound to the user location change /users/properties/location, one has the parent_properties_criteria set to apple_settings and the other one to google_settings. That way code can be kept clean and focused.

The field parent_properties_criteria let's you specify a list of properties the parent has to have for the script to apply. The parent properties becomes a positive or a negative dependency. This does not apply for custom HTTP Scripts.

• ["home"] - the script applies only if the parent user or bot has a home property
• ["!exit"] - the script applies only if the parent user or bot doesn"t have a exit property
• ["!exit","home"] - the script applies only if the parent user or bot doesn"t have a exit property and must have a home property

The field parent_name_criteria lets you specify the names of the bot or user to match or not, you can't mix in this case.

• ["rob","hans"] - script only applies to users ar bots with the name rob or hans
• ["!dominik"] - script is excluded from the user with the name dominik

Javascript Code Conventions

The javascript runs in a Node19.8 environment and as such has all the language features of that version available, such as the spread operator or optional chaining.

Errors

Errors that are thrown in a Saga Runnable context will be caught and reported.

() => {
  //begin runnable code
  const test = null;
  //will throw errors
  test.unknown_method();
  //end runnable code
}

Async / Await

Use async and await over callbacks and promises, not just for the sake of clarity and reduction of nesting but also to drastically simplify exception handling.

Property Functions

Most Saga Runnable will add properties. All the below PropertyObjects objects are available in the Runnable context without additional require.

• UserProperty
• BotProperty
• GlobalProperty

Adding

The async function PropertyObject.add adds a property value and notifies scripts and sockets of the new value.

Adds the user property with the name message with the value hi user the to the user with the identifier parent._id. This assumes that the parent is a user.

await UserProperty.add("message", "hi user" , parent._id);

Adds the bot property with the name level with the value 5 the to the bot with the identifier parent._id. This assumes that the parent is a user.

await BotProperty.add("level", 5 , bot._id);

Adds the global property with the name weather with the value 32.1. A parent object id is not needed since there is only one globals object that holds all the global properties.

await GlobalProperty.add("weather", 32.1 );

Manipulating

Beside simply overriding the property value there are multiple methods to relatively change the data. The methods return the new value.

const submissions = await UserProperty.incr("submissions", 1 , parent._id);
if(submissions.value > 100){
  await UserProperty.add("send_cake", true , parent._id);
}

Here is a list of all functions, Redis inspired. There is also a property calls jsdoc documentation with all details. Those methods are also available via the HTTP and Socket API.

• incr - Increments the value of a numeric property.
• mul - Multiplies the value of a numeric property.
• sadd - Adds a value to an array property unless the value is already present
• rem - Removes from an existing array property all instances of the given value.
• lpush - Left push to an array property.
• rpush - Right push to an array property.
• lpop - Left pop from array property.
• rpop - Right pop from array property.
• sadd - Adds a value to an array property that is treated as a set.
• rem - Removes all entries from the array property that match the value
• set - Set the key value pair in the given property.
• unset - Unset the key value pair in the given property.
• hincr - Increase the key value pair in the given property.
• hmul - Multiply the key value pair in the given property.
• hlpush - Left push to a property array entry.
• hrpush - Right push to a property array entry.
• hlpop - Left pop from property array entry.
• hrpop - Right pop from property array entry.
• hrem - Removes all entries from the property array entry that match the value.
• hsadd - Adds a value to a property array entry that is treated as a set.

Property Function Options

The fourth optional parameter of the add(name,value,parent_id,options) method of UserProperty, BotProperty and GlobalProperty is the options parameter determining property processing aspects.

inline (default:false) when a property is designated for processing by a script, a queue message containing the property_id is queued, when inline is set to true the actual property value is being queued

inline_parent (default:false) when a property is designated for processing by a script, a queue message containing the parent_id is queued, when inline_parent is set to true the actual parent value is being queued

direct (default:false) when set to true the script property processing is invoked directly in the same process rather than queuing it

passive (default:false) when set to true the property is only stored in parent object (either user, bot or global) and no script is triggered.

notify (default:true) when true it notifies all scripts and sockets of the new value, when set to false scripts and sockets are not notified.

delay and delayQueue allow you to set a time delay before listening scripts will run:

await UserProperty.add('message', 'This message will be delayed by one second', parent._id, {'notify': true, 'delay': 1000, 'delay_queue': 'my_queue'});

Note that the property value will still change immediately. Only the scripts that respond to the property change will be delayed. delay is in milliseconds. Setting a unique queue name allows you to cancel the timer by clearing the queue:

await PropertyQueue.purgeByNameAndParentId('my_queue', parent._id);

Signals

Saga Runnables can emit signals from a user, bot, or global context.

Emitting

The async function Object.emit emits a signal notifies scripts and sockets.

Emits the user signal with the name message with the value hi user to the user with the identifier parent._id. This assumes that the parent is a user.

await User.emitSignal("message", "hi user" , parent._id);

Emits the bot signal with the name level with the value 5 the to the bot with the identifier parent._id. This assumes that the parent is a bot.

await Bot.emitSignal("level", 5 , bot._id);

Emits the global signal with the name weather with the value 32.1. A parent object id is not needed since there is only one globals object that holds all the global properties.

await Global.emitSignal("weather", 32.1 );

Message

A is a single text or attachment based interaction between a user/bot and group of users/bots. It has the send method and the mongoose find, findOne and countDocuments methods. Message is directly available in each context without additional require.

  /**
   * @param {Object}              message
   * @param {ObjectId}     message.sender_id
   * @param {sender_type}  message.sender_type
   * @param {[ObjectId]}     message.user_ids
   * @param {[ObjectId]}     message.bot_ids
   * @param {String}  message.body
   * @param {Map<String>|Object} message.context
   * @param {[Object]}  message.attachments
   * @param {String}  message.attachments[].url
   * @param {String}  message.attachments[].mime_type
   * @param {Object}              [options={}]
   * @param {Boolean}         [options.notify=true]
   * @param {Boolean}         [options.direct=false]
   *
   * @returns {Promise<Object>}
   */
  Message.send = async function(message, options){...}

Example of creating a bot message in a user property script.

async (property, parent) => {
  const bot = await Bot.findOne({name: 'score_chat'});
  
  await Message.send({
    sender_id: bot._id,
    sender_type: 'Bot',
    user_ids: [parent._id], 
    body: "Highscore!"
  });
  
}

This will trigger scripts bound to /bots/messages/outgoing, /bots/messages/sent, and /users/messages/incoming. Those scripts determine how to deliver the message depending on the user, bot or application.

Examples of creating a user message in a twilio webhook custom http callback.

async (request, response) => {
    //respond to twilio
    response.set('Content-Type', 'text/plain');
    response.write('');

    //find user
    user = await User.findOne(
        {'properties.phone_number.value': request.body.From}
    );
    if (!user) {
        //let's create a basic user, we don't have a username yet, 
        //we can prefix the random name with "twilio"
        //and the last 3 digits of the number
        user = await User.generateBasicUser(
            `twilio_***${request.body.From.slice(-3)}`
        );
        await UserProperty.add("phone_number", request.body.From, user._id);
        await User.addTag(user._id, "twilio");
    }

    //find bot
    let bot = await Bot.findOne(
        {'properties.twilio_settings.value.phone_number': request.body.To}
    );

    if (!bot) {
        throw new Error("no matching bot found");
    } else {
        await Message.send({
            sender_id: user._id,
            sender_type: 'User',
            body: request.body.Body || "[blank/media]",
            bot_ids: [bot._id]
        })
    }
}

This will trigger scripts bound to the /users/messages/sent, /users/messages/outgoing, and /bots/messages/incoming paths. Those scripts determine the response by either using conversation engine services or programming logic.

Users

A User is a mongoose schema objects and directly available without additional require in each context. Usually you don't need to apply methods on the user directly.

User Registration

The user registration can be called from a Saga Runnable, usually users are created via HTTP API, before properties and messages for the user arrive.

Generate Basic User

A basic user registry is creates a user that has a generated unique identity vs one based on a name provided by the user or a system representing the user. This will trigger scripts bound to /users/events/create. See the twilio message example above for a use case.

/**
 * Register a generated user with random username with an optional username prefix
 * Triggers an  "/users/events/create" event for post processing.
 * @param {String} prefix the optional prefix
 * @return {Promise<object>} - the user
 * @throws ValidationError if the given user object doesn't validate
 */
User.generateBasicUser = async function(prefix){ }

User Registry

Registering a user let's you set a unique username, password and optionally email. This will trigger scripts bound to /users/events/create.

/**
  * Register a user.
  * Triggers an  "/users/events/create" event for post processing.
  * @param {object} fields containing the user fields
  * @param {string} fields.username the unique username, required
  * @param {string} fields.password the password, required
  * @param {string} fields.email the email, optional, needs to be unique
  * @param {object} options
  * @param {object} options.verify {boolean, default=false} skip the user verification process
  * @param {object} options.skip_password {boolean, default=false} skip the user verification process
  * @return {Promise<object>} - the user
  * @throws ValidationError if the given user object doesn't validate
*/

User.register = async function(fields, options={}) { }

MongoDB creation

Occasionally you need to create a user using an outside identifier as the username. If there is a possibility of multiple requests for the user to come in at same time you need to use the findOneAndUpdate to avoid conflicts. Note: returnOriginal:false indicates to the DB to return the new object.

Let's assume an incoming global property from a 3rd party that contains a user name and a location latitude, longitude tuple. The goal is to create/find a SAGA user and adding a location property.

{
  "name":"2131231",
  "location":{
   "latitude": 40.75286,
   "longitude": -73.97929
  }
}

Code

const user = await User.findOneAndUpdate(
  { username:name },
  { },
  {
    returnOriginal:false,
    lean:true
  }
)
await UserProperty.add("location", property.value.location, user._id);    

Storage

A storage component is an abstraction layer for cloud based file storage such as AWS S3 or Azure Cloud Storage. A storage object is a Mongoose Schema paired with a file storage methods. Storage components can be directly used in Saga Runnables. See javascript object.

Networking

Saga Runnables can communicate with other systems the regular node.js networking libraries as well as custom npm packages that can be included.

Chatscript

Saga has an API to communicate with Chatscript called chatscript. It has two functions, one to send a message to the Chatscript Host and receive a reply, the other one to rebuild/compile a bot when the host is running inside a Chatscript manager container. chatscript is directly available in each context without additional require.

/**
 * Send a message to a chatscript server
 * @param hostname the hostname
 * @param port the port
 * @param bot_name bot name
 * @param user_name user nane
 * @param message message itself
 * @param options optioms
 * @param options.networking to override the network connection for testing
 * @returns {Promise<void>}
 */
const send_message = async (
  hostname,
  port,
  bot_name,
  user_name,
  message,
  options={ networking_timeout:12000, networking:null}
  ) => { /*...*/ }

module.exports = {send_message};

Example of using it. Assume a script bound to /bots/messages/incoming. The bot has a chatscript_settings property to define the Chatscript host and bot name. It will create an outgoing message event that is processed by a scripts that knows how to deliver the message to the user given then user and bot settings.

async (sender, sender_type, bot, message) => {
  
    const chatscript_settings = bot.properties.chatscript_settings.value;
    
    const response = await chatscript.send_message(
        chatscript_settings.host_name,
        chatscript_settings.messaging_port,
        chatscript_settings.bot_name,
        user._id.toString(),
        message.body
    )

    await Message.send({
      sender_id: bot._id,
      sender_type: 'Bot',
      user_ids: [sender._id],
      body: response
    });
    
}

A more complex example is using OOB or Out of Band. A message can include a JSON island and the message. The JSON island can be used to send data other than the message to chatscript and receive data other than the message. Here is an example where the OOB from chatscript is used to create properties on the user.

async (sender, sender_type, bot, message) => {
  const Oob          = require('../../lib/oob');

  const chatscript_settings = bot.properties.chatscript_settings.value;

  const response = await chatscript.send_message(
    chatscript_settings.host_name,
    chatscript_settings.messaging_port,
    chatscript_settings.bot_name,
    user._id.toString(),
    message.body
  )

  const entries = new Oob().parse(response);
  for( const key of Object.keys(entries)){
    if(entries[key]!=""){
      await UserProperty.add(key,entries[key],message.user_id,
        {avoid_duplicate:true})
    }

  }

  await Message.send({
    sender_id: bot._id,
    sender_type: 'Bot',
    user_ids: [sender._id],
    body: new Oob().message(response) ,
  });
  
}

Included Saga Libraries

The libraries below are already included in each calling context and available directly.

• GlobalProperty

• UserProperty

• BotProperty

• Message

• User

• Bot

• chatscript

Included Third Party Packages

Saga includes the packages below, besides the packages that are included in Node16. They can be included in Saga Runnables using require.

• @breejs/later
• @mapbox/polyline
• aws4
• bluebird
• express
• jwt-simple
• lodash
• moment
• mongoose
• node-fetch
• node-input-validator
• redis
• statsd-client
• string-hash
• superagent
• underscore
• uuid
• validator

Installing packages

See npm_packages