Records

Symbol Records. More...

Symbol Records.

Symbol records are the actual data. In GAMS Transfer Matlab they are stored in Matlab native data structures, structs, tables, dense or sparse matrices (see Records Format for more information). In GDX, a record is the combination of domain entry data and value data. Domain entries are given as UELs using Matlab categorical. Work with categorical as you would work with strings – it's just a way to save memory. The values a symbol stores per record depends on the symbol type. Sets have text, Parameters have value and Equations and Variables have level, marginal, lower, upper and scale. If some of these value fields are not provided, then default values are used. A record format is chosen for each symbol and not for each of these values independently. Hence note, a container can store different symbols with different record formats.

When working with symbol records, there are two things that can go wrong:

Note
In the very unlikely case that your Matlab (or Octave) version does NOT support categorical (Matlab earlier than R2013b or any Octave version), please read carefully about UELs. Otherwise – very likely –, you won't need the Symbol methods regarding adding or modifying UELs.

Records Format

GAMS Transfer Matlab can read and maintain the symbol records in four different formats: struct, table, dense_matrix and sparse_matrix. Both struct and table are table-like formats and the dense_matrix and sparse_matrix – obviously – matrix-like formats. The default is table as it allows for a good data display and overview. However note that table is not the most efficient format.

  • Table-Like Formats:

    The formats table and struct store the domain entries in the first dimension columns followed by value columns (text for Set, value for Parameter and level, marginal, lower, upper, scale for Variable and Equation) and the records as rows. In case of struct, the columns are given as struct fields. The column names for domain entry columns are the domain name postfixed with their dimension. These expected names are also given by Symbol.domain_labels.

    For example, x in Example as table:

    >> x.records
    ans =
    6×7 table
    i_1 j_2 level marginal lower upper scale
    _________ ________ _____ ________ _____ _____ _____
    seattle new-york 50 0 0 Inf 1
    seattle chicago 300 0 0 Inf 1
    seattle topeka 0 0.036 0 Inf 1
    san-diego new-york 275 0 0 Inf 1
    san-diego chicago 0 0.009 0 Inf 1
    san-diego topeka 275 0 0 Inf 1

    For example, x in Example as struct:

    >> x.records
    ans =
    struct with fields:
    i_1: [6×1 categorical]
    j_2: [6×1 categorical]
    level: [6×1 double]
    marginal: [6×1 double]
    lower: [6×1 double]
    upper: [6×1 double]
    scale: [6×1 double]
    >> x.records.level
    ans =
    50
    300
    0
    275
    0
    275

    Note
    Sets can only be maintained in table-like formats struct and table.
  • Matrix-Like Formats:

    The formats dense_matrix and sparse_matrix store the record values individually as matrices with dimension max(2,d), where d is the symbol dimension, and shape size. If size is undefined (see Symbol Domain), a matrix-like format is not possible. Domain entries cannot be stored in the matrix, but can be queried using the symbol method getUELs (see also Unique Elements (UELs)). Assume a symbol s has two dimensions. Then, a (row,col) matrix entry corresponds to the domain entry s.getUELs(1, [row, col]). The logic is analogue for different dimensions.

    For example, x in Example as dense_matrix:

    >> x.records
    ans =
    struct with fields:
    level: [2×3 double]
    marginal: [2×3 double]
    lower: [2×3 double]
    upper: [2×3 double]
    scale: [2×3 double]
    >> x.records.level
    ans =
    50 300 0
    275 0 275

    For example, x in Example as sparse_matrix:

    >> x.records
    ans =
    struct with fields:
    level: [2×3 double]
    marginal: [2×3 double]
    lower: [2×3 double]
    upper: [2×3 double]
    scale: [2×3 double]
    >> x.records.level
    ans =
    (1,1) 50
    (2,1) 275
    (1,2) 300
    (2,3) 275

    In order to get the domain entries for matrix elements, note that in the above examples the following holds:

    x.getUELs(1, 1:2) equals {'seattle', 'san-diego'}
    x.getUELs(2, 1:3) equals {'new-york', 'chicago', 'topeka'}

    Attention
    Matrix based formats do not store the domain UELs in its own symbol object. Instead, they are given by the elements of the domain set. This implies that changing the domain set records (e.g. in its order) will change the meaning of the matrix formats.

Each format has its advantages and disadvantages, see the following table. There, ratings are ++ (very good), +, o, -, -- (rather bad), rated relatively for each category.

Record Format Max Dimension Efficiency Memory (General) Memory (Dense Data) Display
struct 20 (GAMS limit) ++ + - -
table 20 (GAMS limit) -- o -- ++
dense_matrix 20 (GAMS limit) + -- ++ -
sparse_matrix 2 (Matlab limit) o ++ + --
Note
For scalar symbols (dimension equals 0), the formats struct and dense_matrix are equivalent. GAMS Transfer will usually prefer struct in case of ambiguity.

Domain Violations

Domain violations occur when a symbol uses other Sets as domain(s) – and is thus of domain type regular, see Symbol Domain – and uses a domain entry in its records that is not present in the corresponding referenced domain set. Such a domain violation will lead to a GDX error when writing the data!

Note
Checking for domain violations is not part of Symbol.isValid for performance reasons.

For example, altering x in Example – remember that x has domains {i,j}, where i and j are Sets and madison is not part of set i – as follows:

>> x.records.i_1(end) = "madison";

doesn't update the domain set i:

>> i.records.uni_1'
ans =
1×2 categorical array
seattle san-diego

Trying to write this to a GDX file will fail:

>> m.write();
Error using gt_gdx_write
GDX error in record x(madison,topeka): Domain violation
Error in GAMSTransfer.Container/write (line 341)
GAMSTransfer.gt_gdx_write(obj.system_directory, filename, obj.data, ...

To ask for domain violations, call the method Symbol.getDomainViolations. It returns a list of DomainViolation objects w.r.t. each dimension of the symbol which can then be used to resolve the domain violations: The GAMS Transfer Matlab methods Symbol.resolveDomainViolations and DomainViolation.resolve offer an automatic expansion of the domain sets with the violated entries in order to eliminate domain violations.

For example, continuing the example from above,

>> dv = x.getDomainViolations();

shows the domain violation:

>> dv{1}
ans =
DomainViolation with properties:
symbol: [1×1 GAMSTransfer.Variable]
dimension: 1
domain: [1×1 GAMSTransfer.Set]
violations: {'madison'}

Calling either of the following

>> x.resolveDomainViolations();
>> dv{1}.resolve();

resolves it:

>> dv = x.getDomainViolations()
dv =
0×0 empty cell array
>> i.records.uni_1'
ans =
1×3 categorical array
seattle san-diego madison

This resolving feature can further be triggered automatically by setting the symbol property Symbol.domain_forwarding to true. If records are updated by direct access (see Efficiently Assigning Symbol Records), the domain update will happen delayed for improved efficiency, but can be forced by calling isValid or the resolving methods mentioned above.

Note
The method for automatically resolving the domain violations can be convenient, but it effectively disables domain checking, which is a valuable tool for error detection. We encourage to use Symbol.resolveDomainViolations, DomainViolation.resolve or Symbol.domain_forwarding enabled as rarely as possible. The same holds for using relaxed domain information when regular domain information would be possible, see Symbol Domain.

Validate Symbol Records

GAMS Transfer Matlab requires the symbol records to be stored in one of the supported record formats in order to understand and write them to GDX. However, it can easily happen that a certain criteria of the format is not met and the symbol is marked as invalid, i.e., the symbol method Symbol.isValid returns false. In that case setting the argument verbose of Symbol.isValid to true will print the reason for invalidity and can thus help to resolve the issue.

Note
Performance hint: In case symbol records are updated within a loop, try to avoid checking the symbol validity within the loop. Note that symbol methods to query, for example, the number of records will check for a valid symbol internally. You can use the Matlab Profiler to verify that Symbol.isValid is not called within your loop.

For example, take x of Example, which is of course valid:

>> x.transformRecords('struct');
>> x.isValid(true)
ans =
logical
1

Let's invalidate this symbol by storing it in an incorrect shape:

>> x.records.level = transpose(x.records.level);
>> x.isValid(true)
Warning: Value fields must all have the same size.
> In GAMSTransfer/Symbol/isValid (line 840)
ans =
logical
0
>> x.records.marginal = transpose(x.records.marginal);
>> x.isValid(true)
Warning: Fields need to match matrix format or to be dense column vectors.
> In GAMSTransfer/Symbol/isValid (line 840)
ans =
logical
0

Let's try another example, where we use an incorrect column name for the records of x:

>> x.transformRecords('struct');
>> x.isValid(true)
ans =
logical
1
>> x.records.LEVEL = x.records.level;
>> x.isValid(true)
Warning: Field 'LEVEL' not allowed.
> In GAMSTransfer/Symbol/isValid (line 840)
ans =
logical
0

Unique Elements (UELs)

A Unique Element (UEL) is an (i,s) pair where i is an identification number (called code) for a string s. GDX uses UELs to efficiently store domain entries of a record by storing the UEL code i of a domain entry instead of the actual string s. This avoids storing the same string multiple times. The concept of UELs also exists in Matlab and is called a categorical. Therefore, GAMS Transfer Matlab uses categorical to store domain entries. It is possible to convert a categorical array to its codes by using any number conversion function like int64() in Matlab.

For example, note the categorical in x of Example

>> transpose(x.records.i_1)
ans =
1×6 categorical array
seattle seattle seattle san-diego san-diego san-diego
>> transpose(x.records.j_2)
ans =
1×6 categorical array
new-york chicago topeka new-york chicago topeka
>> transpose(int64(x.records.i_1))
ans =
1 1 1 2 2 2
>> transpose(int64(x.records.j_2))
ans =
1×6 int64 row vector
1 2 3 1 2 3
Attention
In the very unlikely case that your Matlab (or Octave) version does NOT support categorical (Matlab earlier than R2013b or any Octave version), please go on and read carefully about UELs. You must store the UEL codes in the domain columns of the Symbol.records. Storing strings is not supported. For looking up the corresponding string, use the method Symbol.getUELs. Otherwise, if you can use categorical – very likely –, you won't need the Symbol methods regarding adding or modifying UELs, explained below.

Each symbol maintains its own list of UELs per dimension, which can be accessed and modified via the methods Symbol.getUELs, Symbol.setUELs, Symbol.addUELs, Symbol.removeUELs, Symbol.renameUELs and Symbol.reorderUELs (or the Matlab functions for modifying categorical directly). The UEL codes are numbered from 1 to the number of UELs stored independently for each dimension.

Attention
The methods Symbol.setUELs and Symbol.removeUELs may reassign different UEL codes to the UEL labels. These changes are applied to the codes used in Symbol.records. This implies that removing a UEL that is used in the Symbol.records will then lead to an invalid domain entry (displayed as <undefined> in Matlab). For Symbol.setUELs, these updates can be disabled by passing the arguments ‘'rename’, true`.

For example, continuing the Example from above, the UELs are:

>> transpose(x.getUELs(1))
ans =
1×2 cell array
{'seattle'} {'san-diego'}
>> transpose(x.getUELs(2))
ans =
1×3 cell array
{'new-york'} {'chicago'} {'topeka'}

Changing these can invalidate some records:

>> x.setUELs({'madison', 'new-york', 'seattle'}, 1);
>> categories(x.records.i_1)
ans =
3×1 cell array
{'madison' }
{'new-york'}
{'seattle' }
>> transpose(x.records.i_1)
ans =
1×6 categorical array
seattle seattle seattle <undefined> <undefined> <undefined>

If categorical is supported is recommended to work directly on the categorical array, as then new UELs will be added automatically. Let's change the last domain entry of the first dimension to "houston":

>> x.records.i_1(end) = "houston";
>> categories(x.records.i_1)
ans =
4×1 cell array
{'madison' }
{'new-york'}
{'seattle' }
{'houston' }
>> transpose(x.records.i_1)
ans =
1×6 categorical array
seattle seattle seattle <undefined> <undefined> houston

Note, that this could still lead to a domain violation if "houston" is not part of the Set i, see Domain Violations.

Advanced Users Only:
Even with support of categorical it can be useful to explicitly add a UEL with Symbol.addUELs, although simply using a new UEL in Symbol.records will add it automatically to the UEL list. However, it is possible to store more UELs than those actually used in the records. Advanced users can use this fact to sort the universe set of a GDX file to their needs, see also Writing To GDX.

Classes

class  DomainViolation
 Domain Violation. More...
 
class  RecordsFormat
 GAMSTransfer Records Formats. More...
 
class  SpecialValues
 GAMS Special Values. More...