Saga Documentation 0.9.434-4

Export and Import

Table of Contents

About

In order to transfer data from one SAGA instance to another, the user will need to export then import their SAGA data. This document provides the documentation related to the export and import functions.

Pre Condition

Export and Import require a Saga Storage resource called called import_export. The storage entity is used to store the imported and exported archives, they can't be reliably stored on ephemeral containers for processing and later retrieval

Export and Import archive are .tar.gz files. Each archive contains a manifest.json file describing the archive.

Export

The user can export data from SAGA to send to their local machine in a zip file format. Specifically, the user can export the users, bots, property definitions, jobs, scripts, npm packages, and global properties.

GET /exports

List all ongoing and finished exports. Each exports has the following

  • _id - unique idenfier
  • filter - the export filter, see below
  • user_id - the id of the user that initiated the export
  • count - how many components were exported each resource section
  • duration - how many seconds it took to generate the export
  • file_size - archive size in bytes
  • finished - is the export finished?
  • success - did it succeed?
  • error_messages - list of encountered errors during the export
  • path - the path in the import_export storage component
  • createdAt - start time
  • updatedAt - finish time

RESPONSE

[
  {
    "_id": "645f7699a1416a183c053087",
    "filter": "jobs scripts",
    "user_id": "644fde8d7a7c3a2c4ac9a1b5",
    "count": {
      "jobs": 23,
      "scripts": 87
    },
    "duration": 78,
    "file_size": 48724,
    "finished": false,
    "success": true,
    "error_messages": [],
    "path": "export-stan-local-development-0-9-1683977881486.tar.gz",
    "updatedAt": "2023-05-13T11:38:02.369Z",
    "createdAt": "2023-05-13T11:38:01.487Z",
    "id": "645f7699a1416a183c053087"
  },
  {
    "_id": "645f7411aef72a3d4c5c7592",
    "filter": "users",
    "user_id": "644fde8d7a7c3a2c4ac9a1b5",
    "count": {
      "users": 1
    },
    "duration": 37,
    "file_size": 3046,
    "finished": true,
    "success": true,
    "error_messages": [],
    "path": "export-stan-local-development-0-9-1683977233089.tar.gz",
    "updatedAt": "2023-05-13T11:27:13.497Z",
    "createdAt": "2023-05-13T11:27:13.091Z"
  }
]

POST /exports

Starts the export process, required parameters:

  • filter:
    • include all or some of the following separated by linebreak to select sections:
      • bots
      • users
      • global_properties
      • messages
      • schema_definitions
      • scripts
      • jobs
      • npm_packages
    • filtering each section with /has: to select data containing specific properties: bots/has:coordinates or multiple properties bots/has:coordinates has:location
    • filtering each section with /has: to select data containing certain property values:users/has:favorite_color='red', users/selected=true
    • filtering each section with /. This will check if their names or notes contain this value:users/james, users/nancy, bots/C-3PO
    • filtering messages with messages/bot_id or messages/bot_id,bot_id for messages involving such bots

The status of export can be retrieved by calling GET /exports/:id. Also, the /system socket rooms receives updates on ongoing exports under the /exports event.

RESPONSE

The _id can be used to track the export process

{
  "_id": "645f7411aef72a3d4c5c7592",
  "filter": "users",
  "user_id": "644fde8d7a7c3a2c4ac9a1b5",
  "count": {},
  "duration": 0,
  "finished": false,
  "success": false,
  "error_messages": [],
  "path": "export-stan-local-development-0-9-1683977233089.tar.gz",
  "updatedAt": "2023-05-13T11:27:13.091Z",
  "createdAt": "2023-05-13T11:27:13.091Z"
}

GET /exports/:id

Return the export status

RESPONSE

{
  "_id": "645f7411aef72a3d4c5c7591",
  "filter": "users scripts",
  "user_id": "644fde8d7a7c3a2c4ac9a1b5",
  "count": {
    "users": 1
  },
  "duration": 10,
  "finished": false,
  "success": false,
  "error_messages": [],
  "path": "export-stan-local-development-0-9-1683977233089.tar.gz",
  "updatedAt": "2023-05-13T11:27:13.091Z",
  "createdAt": "2023-05-13T11:27:13.091Z"
}

GET /exports/:id/download

Once the export is done use this resource to download the archive of the export with the given id

Import

Users can import a formerly exported file and deploy the contained data, items are either created or if there _id matches updated. Global properties are always newly added.

GET /imports

Lists all ongoing and finished imports.

RESPONSE

Each imports has the following.

  • _id - unique idenfier
  • user_id - the id of the user that initiated the import
  • count - how many components were import each resource section
  • duration - how many seconds it took to generate the import
  • file_size - archive size in bytes
  • finished - is the export finished?
  • success - did it succeed?
  • error_messages - list of encountered errors during the import
  • path - the path in the import_export storage component
  • createdAt - start time
  • updatedAt - finish time
[
  {
    "_id": "64527c5a07b32d06ef68f275",
    "path": "import-stan-local-development-0-9-1683127386256.tar.gz",
    "file_size": 200003,
    "user_id": "644fde8d7a7c3a2c4ac9a1b5",
    "count": {
      "global_properties": 40
    },
    "duration": 1157,
    "finished": true,
    "success": true,
    "error_messages": [],
    "updatedAt": "2023-05-03T15:23:08.076Z",
    "createdAt": "2023-05-03T15:23:06.908Z"
  },
  {
    "_id": "64527bebd620e55b95264a8f",
    "path": "import-stan-local-development-0-9-1683127272999.tar.gz",
    "file_size": 509079,
    "user_id": "644fde8d7a7c3a2c4ac9a1b5",
    "count": {
      "bots": 798,
      "jobs": 20,
      "npm_packages": 14,
      "scripts": 83
    },
    "duration": 3739,
    "finished": true,
    "success": true,
    "error_messages": [],
    "updatedAt": "2023-05-03T15:21:19.037Z",
    "createdAt": "2023-05-03T15:21:15.275Z"
  }
]

POST /imports

The upload body field for the attached file is import. Once the file is uploaded to the import_export Storage component the REST call responds and the actual import process starts.

The status of export can be retrieved by calling GET /imports/:id. Also, the /system socket rooms receives updates on ongoing exports under the /exports event.

RESPONSE

{
  "_id": "645f80faaef72a3d4c5c75d8",
  "path": "import-stan-local-development-0-9-1683980538300.tar.gz",
  "file_size": 3046,
  "user_id": "644fde8d7a7c3a2c4ac9a1b5",
  "count": {},
  "duration": 0,
  "finished": false,
  "success": false,
  "error_messages": [],
  "updatedAt": "2023-05-13T12:22:18.608Z",
  "createdAt": "2023-05-13T12:22:18.608Z"
}

GET /imports/:id

Return the import status.

RESPONSE

{
  "_id": "645f80faaef72a3d4c5c75d8",
  "path": "import-stan-local-development-0-9-1683980538300.tar.gz",
  "file_size": 3046,
  "user_id": "644fde8d7a7c3a2c4ac9a1b5",
  "count": {
    "users": 20
  },
  "duration": 10,
  "finished": false,
  "success": false,
  "error_messages": [],
  "updatedAt": "2023-05-13T12:22:25.608Z",
  "createdAt": "2023-05-13T12:22:18.608Z"
}