GAMS Engine

Introduction

GAMS Engine is our new solution for running GAMS jobs in cloud environments. It provides a REST API that you can connect to either via GAMS MIRO Desktop, via the GAMS Engine UI, via GAMS Studio or via any of the clients supported by OpenAPI (Python, Java, Javascript, C++, ...).

GAMS Engine automatically schedules your jobs and assigns them to an available GAMS worker to solve it. It comes with a powerful user management system that allows you to restrict the activities of your users according to your organizational hierarchy.

The (technical) API documentation can be found here.

System overview

GAMS Engine is a highly distributed system to solve your GAMS models. The heart of this system is a REST API - also referred to as broker - where you can submit your GAMS jobs. The system will then queue your jobs and assign them to an available worker - a GAMS process that solves your model. In case all workers are busy solving other jobs, your job is pending in the queue until a free worker is available. You can limit the amount of CPU and RAM a worker may use and workers can even run on different machines.

Components of GAMS Engine

Installation

The installation of GAMS Engine highly depends on your setup. When you purchase GAMS Engine, you will receive a ZIP file that contains everything you need to get started including installation instructions tailored to your setup. The general installation instructions are given below.

  • Install Docker from https://docs.docker.com/engine/install/. Start docker afterwards.
  • Install Docker Compose from https://docs.docker.com/compose/install/
  • Extract miro_engine.zip file into a folder
  • Open a terminal and cd into that folder
  • Run the script ./up.sh with admin privileges. The following arguments can be used:
    • -p: Engine port
      Specifies the port through which you want to reach Engine. This works only for the non-SSL (unencrypted) setup.
    • -w: Numer of workers
      Specifies how many GAMS workers should be enabled.
    • -c: Certificate file
      For the SSL setup, the path to the certificate file must be specified here.
    • -k: Certificate key file
      For the SSL setup, the path to the certificate key file must be specified here.
      Note:

      Note that when using the encrypted setup, port 80 and port 443 must be available!

    The script downloads and initializes all required Engine files. This may take a moment.

  • Once the script is finished, GAMS Engine should be running. Depending on whether an SSL or an unencrypted setup has been selected, Engine and Engine UI can be reached via:
    • SSL setup: Open http(s)://mydomain.com in your browser (replace mydomain with your URL).
    • Unencrypted setup: Engine runs on the port you selected before (default is port 80). Open http://192.168.178.184:80 in your browser (replace IP and port accordingly).
  • This will open the Engine UI. Login with the default credentials username: admin and password: admin.

  • Install Docker from https://docs.docker.com/engine/install/. Start docker afterwards.
  • Extract miro_engine.zip file into a folder
  • Open a terminal and cd into that folder
  • Run the script .\up.cmd. The following arguments can be used:
    • \p: Engine port
      Specifies the port through which you want to reach Engine. This works only for the non-SSL (unencrypted) setup.
    • \w: Numer of workers
      Specifies how many GAMS workers should be enabled.
    • \c: Certificate file
      For the SSL setup, the path to the certificate file must be specified here.
    • \k: Certificate key file
      For the SSL setup, the path to the certificate key file must be specified here.
      Note:

      Note that when using the encrypted setup, port 80 and port 443 must be available!

    The script downloads and initializes all components required for GAMS Engine. This may take a moment.

  • Once the script is finished, GAMS Engine should be running. Depending on whether the SSL or the unencrypted setup has been selected, Engine and Engine UI can be reached via:
    • SSL setup: Open http(s)://mydomain.com in your browser (replace mydomain with your URL).
    • Unencrypted setup: Engine runs on the port you selected before (default is port 80). Open http://192.168.178.184:80 in your browser (replace IP and port accordingly).
  • This will open the Engine UI. Login with the default credentials username: admin and password: admin.

Licensing

For a GAMS Engine setup, both a tailored GAMS license and an Engine license are required. The license keys can be entered directly in the Engine UI in the Users section by clicking on the corresponding buttons. For the Engine license, click on the Update Engine license button at the top of the page and enter the key.

Providing Engine and GAMS license keys

A GAMS license can be specified system-wide or user-specific.

  • System-wide means that all users of the running Engine instance use the same license when solving their GAMS models. The system-wide license can be entered at the top of the page via the Update GAMS license button, right next to the Update Engine license button.
  • A user-specific GAMS license is only valid for the individual owners of the license. If a user has such a license, this will always be used for GAMS jobs instead of the system-wide license. Also, if an inviter with such a license invites another person to use GAMS Engine, the invitee automatically inherits the license of the inviter.
    User licenses are entered in the user list under ActionsUpdate license.

User management

GAMS Engine comes with a highly flexible user management system. After first installing the system, there is a single default user available with name: admin and password: admin.

Note:

We strongly recommend that you change the default password for the admin user immediately after installing the system to prevent unauthorized users from accessing your system!

In addition, there is a single namespace available with the name: global. Namespaces are similar to directories in the UNIX file system: you can specify which users have access to which namespaces. If you want to start a new GAMS job, it always runs within a namespace. Just like the UNIX file system, GAMS Engine has three types of permissions on namespaces:

  • Read Permission
    Users can download GAMS models registered in this namespace
  • Write Permission
    Users can register new models in this namespace
  • Execute Permission
    Users can execute models in this namespace
Warning:

In order to restrict a user from accessing your model files, you have to specify a proper INEX file in addition to revoking read permission from this user!

The table below shows the mapping of these permissions to the permission codes used by GAMS Engine.

Permission Code Read Permission Write Permission Execute Permission
0 - - -
1 - -
2 - -
3 -
4 - -
5 -
6 -
7

GAMS Engine distinguishes two types of jobs:

  • Jobs with a registered model
    The GAMS model is registered inside the namespace and the user only provides the data to run the model with.
  • Jobs with an unregistered model
    The user provides both the model files as well as the data to run the model with.
To run the first type of job, users must have execute permissions for the namespace in which the model to be executed is registered. The second type of job requires additional write permissions as a new (temporary) model has to be created in order to run this job.

GAMS Engine distinguishes three types of users:

  • Users
  • Inviters
    Inviters are users with the additional privilege to invite new users to use GAMS Engine. However, inviters can only invite users who have the same or fewer permissions than themselves. For example, if an inviter has read and execute permissions on the namespace global, she is not allowed to invite a user with write permission on this namespace. Inviters can invite other inviters, but not admins.
  • Administrators
    Administrators are the most privileged users. They have full permissions for each namespace and are allowed to invite new users, even new administrators. Administrators can also add new namespaces or remove existing ones.

In order to add a new user to GAMS Engine, you need to generate an invitation code. Users can then register themselves by providing this invitation code, a username and a password. Invitation codes can be generated by administrators and inviters. When creating an invitation code, permissions to namespaces can be assigned so that the new user can start interacting with the system directly. Note that the invitee's permissions may be lower than the inviter's, but not higher. Furthermore, inviters are able to manage - i.e. modify permissions and delete - their children (invitees) as well as any grandchildren. In this way, you can set up several hierarchy levels.

Traffic reduction and access restriction

INEX file

You can use GAMS Engine to restrict the files contained in the results archive. Use cases for this function include reducing the size of the archive and restricting access to certain files. This INEX file has the following schema:

{
  "$schema": "http://json-schema.org/draft-04/schema#",
  "type": "object",
  "properties": {
    "type": {
      "type": "string",
      "enum": [
        "include",
        "exclude"
      ]
    },
    "files": {
      "type": "array",
      "items": {
        "type": "string"
      },
      "required": [
        "items"
      ]
    }
  },
  "required": [
    "type",
    "files"
  ]
}

Let's assume we register the trnsport model from the GAMS model library. We want users to be able to run this model, but not see our GAMS code. More specifically, we want to exclude all files with the extensions .gms and .inc from the results returned to the users of our model. We therefore register the model with the following INEX file:

{
  "type": "exclude",
  "files": ["*.gms", "*.inc"]
}

Note that glob patterns (sometimes also referred to as "wildcards") are supported here.

You can specify an INEX file both when registering a new model (as shown in the previous example) and when submitting a job. When you submit a new job for a registered model that already contains an INEX file, the results are first filtered against that file. Only after that, the INEX file sent with the job will be applied. This means that when you submit a new job, you cannot remove restrictions that are associated with the registered model.

Text/Stream entries

When executing a job, you can register both so-called text entries and stream entries. These can be any UTF-8 encoded text files generated during your model run. In the case of stream entries the contents of these files can be streamed while your job is running. Text entries can only be displayed after your job is complete. This feature is useful if you want to view certain files while your job is running (e.g. solvetrace files) or before downloading the (possibly extensive) result archive.

While text entries can be retrieved multiple times via the GET endpoint /jobs/{token}/text-entry/{entry_name}, the contents of stream entries are pushed into a FIFO (first-in, first-out) queue. The DELETE endpoint /jobs/{token}/stream-entry/{entry_name} can be used to empty the queue and retrieve the content.

Note:

The GAMS log file (process's standard output) is a special type of stream entry. Thus, it has its own endpoint: /jobs/{token}/unread-logs.

Hypercube jobs

In addition to a normal GAMS job, GAMS Engine also supports so-called Hypercube jobs. These are essentially a collection of normal jobs with varying command line arguments. Why is this useful? Imagine a GAMS model that runs through a number of different scenarios. The model is always initialized by the same large GDX file, but behaves differently depending on double dash parameter --scenId. In this situation you want to avoid uploading the same large GDX file for each scenario. Instead, you want to upload it once and create a number of jobs with different command line arguments: a Hypercube job.

To spawn a new Hypercube job, you need to use the POST /hypercube/ endpoint. The only difference to posting a normal job is a so-called Hypercube description file. This is a JSON file that describes your Hypercube job. This Hypercube description file has the following schema:

{
  "$schema": "http://json-schema.org/draft-04/schema#",
  "type": "object",
  "properties": {
    "jobs": {
      "type": "array",
      "items": {
        "type": "object",
        "properties": {
          "id": {
            "type": "string"
          },
          "arguments": {
            "type": "array",
            "items": {
              "type": "string"
            },
            "minItems": 1
          }
        },
        "required": [
          "id",
          "arguments"
        ]
      },
      "required": [
        "items"
      ],
      "minItems": 2
    }
  },
  "required": [
    "jobs"
  ]
}

With this in mind, our example described above could result in the following Hypercube description file:

{
  "jobs": [
    {
      "id": "first scenario",
      "arguments": ["--scenId=1"]
    },
    {
      "id": "second scenario",
      "arguments": ["--scenId=2"]
    },
    {
      "id": "third scenario",
      "arguments": ["--scenId=3"]
    }
  ]
}

This would result in the creation of three GAMS jobs with different values for --scenId. The id of a job can be any string. However, it must be unique among the different jobs of a Hypercube job and is used as the directory name where the results are stored. The content of the result archive of the Hypercube job described above would look like this (results are filtered via INEX file):

.
├── first\ scenario
│   └── results.gdx
├── second\ scenario
│   └── results.gdx
└── third\ scenario
    └── results.gdx

3 directories, 3 files

Job dependencies

Jobs can be dependent on other jobs. This means the following:

  1. A job with dependencies is not started until all its dependencies are finished.
  2. A job with dependencies has access to the results of the dependent jobs.

While the first point is fairly straightforward, the second is best explained with a small example. Suppose we have a job c that depends on jobs a and b. Before job c is started, first the result archive of job a is extracted, then that of job b and finally the model and job data of job c. If a results.gdx file exists in both a and b, the one from job b overwrites the one from job a. Thus, the order in which you specify the dependencies is important.

A common case where this feature is very useful is pre/post processing. You may have a long-running job that generates a large amount of results. However, you may not be interested in all of these results. Rather, you want to run analyses on parts of the results and then also retrieve only the part of the results you are interested in at the moment. Using a post-processing job with a dependency on the main job along with INEX files allows you to do just that.

Note:

The dependent jobs must be your own. You cannot have dependencies on other people's jobs.

GAMS Engine UI

GAMS Engine comes with a web user interface that implements most of the features. Via this interface you can manage users, models and data, but also submit new jobs and download the results. Depending on your role ("User", "Inviter" or "Administrator"), different elements of the UI are visible.

To use Engine, you must first register with the system. To register, you need an invitation code.

  1. Open Engine UI
  2. Click 'Register' below the Login button
  3. Put your registration code, username and password
  4. Click 'Register'
  5. You will automatically be logged in

Components of the GAMS Engine web user interface

The UI consists of four views (the minimum user role required to see this view is appended in brackets):

  • Jobs (User)
  • Models (User)
  • Users (Inviter)
  • Cleanup (Administrator)

The Jobs view lists all the jobs you submitted (or all jobs submitted by all users if you are an administrator). You can sort your submissions by clicking on the desired column. From this view, you can also interact with your jobs (view more details, cancel running jobs or download results) as well as submit new ones.

The Models view lists all the namespaces as well as all the models registered in each namespace. To view the models registered in a particular namespace, click on this namespace and the table that lists the models will update. Just as in the Jobs view you can sort the table by clicking on the desired column. You can add or remove namespaces from this view as well as register new models.

The Users view is only visible to inviters and administrators. If you are an administrator, you can see all users registered in the system; if you are an inviter, only users invited either by you or by one of your invitees are displayed. You can edit user roles and permissions as well as remove a user. Additionally, new invitation codes can be created from here. Note that inviters can only invite people who have the same or less privileges as themselves. To learn more about user roles and permissions click here.

The Cleanup view is only visible to administrators. Here you can clean up the database to reduce the amount of storage used by GAMS Engine. You can either remove files one by one or clean up multiple files at once by clicking the "Run housekeeping" button. This "housekeeping" dialog allows you to remove all files created more than x days ago and/or files created by users who have been removed from the system.

GAMS Studio integration

GAMS Studio has built-in functionality of some basic interactions with GAMS Engine. Starting with version 1.4.2 Studio can be used to perform GAMS jobs with Engine. The feature set is constantly being extended.

To use Engine for solving a GAMS job, select Run GAMS Engine in the Quick Access Toolbar or via the entry GAMS in the main menu.

Run GAMS Engine menu

In the dialog that appears, further settings need to be made:

  • Engine URL: URL to the GAMS Engine server.
  • Namespace: Namespace the job is to be executed in
  • Username: Username of the Engine user account
  • Password: Password of the Engine user account
  • Create GDX file: Create a GDX file containing all symbols independant from other settings.
When editing the Engine URL Studio tries to reach the server and requests version information. On success the version of GAMS and Engine on the server are displayed. If the local GAMS version is newer, the command line parameter previousWork=1 is automatically added.

With a click on OK the job is submitted. Note that the model is compiled locally before the compilation object is send to GAMS Engine. The results are written into a sub-folder named by the base name (i.e. main file name) of the model. Links in the log output are replaced by their local counterpart.

Note:

GAMS jobs processed via Engine are executed on a Linux operating system. Thus, the UNIX path separator ('/') is used. This is relevant for existing paths in the GAMS model as well as for paths on the command line when submitting a job. You should therefore ensure that the UNIX path separator is always used!

API documentation

The (technical) API documentation can be found here.

Note:

GAMS Engine is documented using OpenAPI version 2.0. You can download the definition file for the latest version of GAMS Engine here!