Integration with Engine

Integration with Rustici Engine

Generator supports integrations with Rustici Engine. After initial setup, Generator can stay in sync with one or multiple tenants in Rustici Engine by automatically importing content from Engine’s content library into Generator. For any MP3 and MP4 content in Engine’s content library, Generator can automatically create VTT files and upload those back to Engine for use during content launch. Tenant management is also handled, with Generator taking steps to keep its tenants in sync with Engine’s.

Engine integrations are supported via dedicated API endpoints, and can be configured in a single API call. Generator handles the more complex setup configuration behind the scenes.

Support: At this time, Generator supports integrations with Engine versions 22.x.x and above.

Communications between Engine and Generator

Generator strives to keep the setup and management of Engine integrations as simple as possible. This can be completed in a single API call to Generator’s PUT /api/v1/app_management/engine endpoint. Behind the scenes, Generator will:

  1. Create a unique set of credentials that will be used when Engine sends webhook messages to Generator.
  2. Use Engine’s API to create an Engine subscription for the {topic: "CourseImport", subtopic: "success"} event. This subscription is configured to use the dedicated Engine credentials created in the previous step.

And that’s it - the Engine integration has been set up. From that point on, Generator will receive a webhook message from Engine when content is successfully imported. When it receives that notification, Generator will know where to find the course files and will begin the import process. Once imported, Generator can parse the content text, generate metadata for the course, and include the content in its search feature.

NOTE: The credentials created for the Engine application’s webhook messages will be included when getting a list of all credentials from Generator’s GET /api/v1/credentials endpoint.

Configuring Your Engine Integration

NOTE: If your instances of Rustici Generator and Rustici Engine are managed by Rustici’s Hosting Team, we can set up your integrations. Please reach out to our Delight Team and let us know if you’d like us to set up an integration between your MH Generator and MH Engine instances. We’ll need to know if you want a system-to-system or tenant-to-tenant integration (see below for details). If you’re using a tenant-to-tenant integration, please also include the ID(s) of the Engine tenant(s) you’d like to integrate.

WARNING: After configuring an Engine integration, Generator will defer to Engine’s content library should it encounter any ID conflicts during the content import process. This means that if Generator receives a CONTENT_IMPORT webhook from Rustici Engine for a course, any existing course in Generator with the same ID will be overwritten. For example: if a user directly imports a course into Generator with the ID {'content_id': 'courseA', 'version': '0'} and later receives a CONTENT_IMPORT webhook from Engine for a course with the ID {'content_id': 'courseA', 'version': '0'}, Generator will delete the existing, directly-imported content and replace it with the Engine-imported content.

To begin the process of connecting an instance of Rustici Generator with an instance of Rustici Engine, users will need to provide information to Generator about the Engine instance. Generator’s /api/v1/app_management/engine endpoints should be used to configure and manage Engine integrations.

To define an integration, the following properties should be configured as part of the request schema

  • api_config: required - Information required by Generator that allows it to use Engine’s API. The engine_tenant_name is used to limit the scope of Generator’s coordination with Engine. See Tenancy Management for Engine Integrations below for more information. Because Engine treats their Engine tenant name case-insensitively, Generator will lowercase the tenant name that is provided. For this reason, you might see that the tenant name when GETing the Engine integration details is different from the uppercase name you provided.
  • s3_config: required* - For Engine installations whose content libraries are stored in Amazon S3, this schema defines access information. Generator must be able to reach the learning content so for its processing.
  • http_config: required* - For Engine installations whose content libraries are accessible via URL, this schema provides access information. Generator must be able to reach the learning content so for its processing.
  • web_path_override: optional - For Engine installations where the web path received by the webhook doesn’t line up directly with the path of the content’s storage. This property defines a regex search and replace that would be applied to the web_path defined in the COURSE_IMPORTED Engine webhook. You can see if this property will be necessary by checking the /api/v2/courses endpoint of the Engine API and reviewing the webPath property of your courses. Compare the webPath of the courses to the course file’s path within your S3 bucket or file server. If they don’t line up (e.g. the webPath contains the protocol and domain information), then this property can be used to convert the webPath to the proper path of your course files.

* mutually exclusive. Only one of these properties must be provided.

For example, the following schema could be used for the PUT /api/v1/app_management/engine endpoint to set up an integration between an Engine tenant and a Generator tenant:

{
  "api_config": {
    "base_url": "https://companyname.engine.com",
    "engine_tenant_name": "sales_department",
    "username": "user1",
    "password": "pass1"
  },
  "s3_config": {
    "s3_bucket": "company_learning_content",
    "aws_access_key_id": "abc123",
    "aws_region": "us-east-1",
    "aws_access_secret_key": "abc123",
    "aws_session_token": "abc123"
  },
}

With the above configuration, any content imported into Engine’s sales_departement tenant will be imported into Generator. Generator will use the information defined in s3_config and api_config to access the learning content files and import them into Generator, specifically in the Generator the tenant defined in the request’s tenant-id header.

Tenancy Management for Engine Integrations

It’s important to consider how tenants will be managed as part of an Engine integration. Rustici Generator supports both a tenant-to-tenant and system-to-system relationship between its own tenants and the tenants defined in Rustici Engine.

  • In a tenant-to-tenant integration, a single tenant in Generator will coordinate with a single tenant in Engine. Generator will receive a notification when content is imported into a specific Engine tenant, and that tenant’s content will be isolated into a single tenant in Generator. When setting up a tenant-to-tenant Engine integration, the PUT request must provide a tenant-id header to specify the Generator tenant, and the request schema must define a engine_tenant_name property to specify the Engine tenant.
  • In a system-to-system integration, Generator will generate multiple tenants to correspond to any tenant for which it receives a successful CONTENT_IMPORT notification. Generator will receive notifications about content imported into any Engine tenant and will create new tenants if it receives a notification that references a tenant that doesn’t have a corresponding tenant in Generator. When setting up a system-to-system Engine integration, the PUT request must not specify either the tenant-id header or the engine_tenant_name schema property.

Importing the Engine Content Library

After setting up an Engine integration, Generator will start receiving notifications anytime learning content is uploaded into the appropriate tenant. However, this won’t account for any existing Engine content. In order for Generator to import and parse existing Engine content, users must initiate an Engine Import job.

The Engine Import job will use Engine’s API to find all of its learning content and import the found content into Generator. This job can be initiated via the POST /api/v1/jobs/engine_import endpoint. Because the Engine library may contain a large number of learning content, the endpoint’s request schema can be used to filter out unwanted or outdated content. The endpoint schema accepts the following properties:

  • since_dt: optional - an ISO 8601 datetime that limits the content Generator will import to content updated at or after the specified time (e.g. 1970-01-01T00:00:00.000Z).
  • until_dt: optional - an ISO 8601 datetime that limits the content Generator will import to content updated before or at the specified time (e.g. 1970-01-01T00:00:00.000Z).
  • engine_course_ids: optional - a list of Engine content IDs that explicitly specify the content available for Generator to upload.
  • engine_tenant_id: optional - the Engine tenant ID that explicitly specify the tenant available for Generator to scrape. NOTE: this parameter should only be used when making the engine_import request for a Generator tenant that is configured with a system-level Engine Integration. See below for details.
  • latest_version: optional, default TRUE - specifies how Generator should handle multiple versions of the same Engine content. When latest_version is TRUE, Generator will import only the latest version of the content. When latest_version is FALSE, all versions of the imported content will be imported.

The POST /api/v1/jobs/engine_import endpoint requires a tenant-id header. This header specifies into which Generator tenant the Engine content should be imported. If the Generator tenant specified by the tenant-id header has a tenant-level Engine Integration configured, then the engine_tenant_id property should not be provided, as Generator will assume the Engine and Generator tenant names match. If the Generator tenant specified by the tenant-id header is inheriting the system-level Engine Integration, then the engine_tenant_id must be provided, as this will tell Generator which Engine tenant it should scrape.

Troubleshooting Engine Integrations

Monitoring Integration Status

After setting up an Engine integration, Generator will automatically start new import jobs when content is imported into Engine. To monitor the status of these jobs:

  1. Use the GET /api/v1/jobs endpoint with the appropriate tenant-id header to see all jobs for the integrated tenant.
  2. Look for any new jobs started by your Engine import, check the job’s status, and review job error messages (if present) to see if anything needs to be corrected.

Common Issues

Jobs Not Starting After Engine Imports

If content is imported into Engine but Generator is not creating corresponding import jobs, this typically indicates that Engine is unable to send webhook notifications to Generator.

Please try the following:

  1. Check Engine’s subscription status: Use Engine’s API to check the subscription that Generator created. The endpoint GET /api/v2/appManagement/subscriptions/ (for the relevant Engine tenant) will return subscription details. Look for properties named lastExceptionDate and lastExceptionReference - if these are present, they indicate that Engine encountered an error when trying to send the webhook to Generator. The error details will help identify the problem.

  2. Verify the subscription URL: Check the url property in the subscription response. It should match your Generator instance’s domain (as configured in Generator’s HOSTING__DOMAIN environment variable). If the Generator URL is incorrect or inaccessible from Engine, then Engine cannot reach Generator. For some situations, especially around DNS errors, the Engine webhook may fail without setting the lastExceptionDate and lastExceptionReference of the subscription API resource.

  3. Check logs: Review both Engine and Generator logs around the time of the import. Engine logs should show if any errors were encountered sending the request, and the Generator backend logs should show whether webhook requests were received.

“No objects found in S3 bucket” Errors

If you encounter errors like “No objects found in S3 bucket ‘bucket-name’ with prefix ‘path’”, this typically indicates that the web_path provided by Engine’s webhook doesn’t match the actual path structure of files in your S3 bucket.

Solution: Configure the web_path_override property when setting up your Engine integration. This property uses a regex search and replace to transform the web path from Engine’s webhook into the correct S3 path.

Example: If Engine’s webhook provides a path like https://engine-content.example.com/content/tenant123/courses/course-id/0/file.pdf, but your S3 bucket stores files at content/tenant123/courses/course-id/0/file.pdf, you would configure:

"web_path_override": {
  "search_regex": "^https://engine-content.example.com/",
  "replace_text": ""
}

This will strip the domain prefix from the web path, leaving only the S3 bucket path.