idc01.gms : Implicit Data Contract Test

Description

Contributor: M. Bussieck, December 2019


Small Model of Type : GAMS


Category : GAMS Test library


Main file : idc01.gms

$title Implicit Data Contract Test (IDC01,SEQ=807)

$onText
Contributor: M. Bussieck, December 2019
$offText


*
* GAMS Program with external input and output with data statements
* Can be configured to have uels for input symbols with text prefix (--pp=prefix)
*
$onEchoV > def1.gms
$if not set pp $set pp
set sx / h1,h2 /;
$onexternalInput
scalar p0 'p0scalar' /1/; set s1 's1set' /%pp%1/; parameter p1 'p1parameter' /%pp%1 1/, p2(*,sx) 'p2parameter' / %pp%1.h1 1 /;
table p2; variable x1(*,*) 'x1variable' / %pp%1.%pp%3.L 1 /; equation e1(*) 'e1equation' / %pp%3.L 1 /;
$offExternalInput
$onexternalOutput
scalar op0 'op0scalar' /1/; set os1 'os1set' /1.2/; parameter op1 'op1parameter' /1 1/, op2(*,sx) 'op2parameter' / 1.h1 1 /;
table op2; variable ox1(*,*) 'ox1variable' / 1.3.L 1 /; equation oe1(*) 'oe1equation' / 3.L 1 /;
$offExternalOutput
$offEcho
* Create data contract file as well as input and in/out GDX files
$call.checkErrorLevel gams def1.gms lo=%gams.lo% IDCGenerateJSON=def1.json IDCGenerateGDXInput=def1orgin.gdx IDCGenerateGDX=def1orginout.gdx

*
* GAMS Program with external input and output with data via GDX file
* Different order of symbols but otherwise identical
*
$onEchoV > def2.gms
set sx / h1,h2 /;
$onexternalInput
variable x1(*,*) 'x1variable'; equation e1(*) 'e1equation';
scalar p0 'p0scalar'; set s1(*) 's1set'; parameter p%one%(*) 'p1parameter', p2(*,sx) 'p2parameter';table p2; 
$offExternalInput
$onexternalInput
$gdxin %gdxInFile%
$load p0 s1 p%one% p2 x1 e1
$offExternalInput
$onexternalOutput
variable ox1(*,*) 'ox1variable'; equation oe1(*) 'oe1equation';
scalar op0 'op0scalar'; set os1(*,*) 'os1set'; parameter op1(*) 'op1parameter', op2(*,sx) 'op2parameter'; table op2; 
$offExternalOutput
$offEcho

$onEchoV > def3.gms
set sx / h1,h2 /;
$onexternalInput
parameter p%one%(*) 'p1parameter', p2(*,sx) 'p2parameter';table p2; 
variable x1(*,*) 'x1variable'; equation e1(*) 'e1equation';
scalar p0 'p0scalar'; set s1(*) 's1set'; 
$onEmbeddedCode Python:
gams.set('p%one%',[])
gams.set('p2',[])
gams.set('x1',[])
gams.set('e1',[])
gams.set('p0',[15])
$offEmbeddedCode p%one% p2 x1 e1 p0 s1<p2.dim2
$offExternalInput
$onexternalOutput
variable ox1(*,*) 'ox1variable'; equation oe1(*) 'oe1equation';
scalar op0 'op0scalar'; set os1(*,*) 'os1set'; parameter op1(*) 'op1parameter', op2(*,sx) 'op2parameter'; table op2; 
$offExternalOutput
$offEcho

* Run def2: Create data contract and compare with def1 data contract (IDCJSON) load GDX data traditionally save in/out symbols in def2expl.gdx
$call.checkErrorLevel gams def2.gms lo=%gams.lo% IDCGenerateJSON=def2.json IDCJSON=def1.json --one=1 --gdxInFile=def1orgin.gdx IDCGenerateGDX=def2expl.gdx
* Compare JSON text files, they shopuld be different due to different symbol order in def1 and def2
$call diff def1.json def2.json > %system.nullfile% 2>&1
$if not errorlevel 1 $abort json files def1.json def2.json should be different
$call gams def2.gms lo=%gams.lo% IDCJSON=def1.json --one=one --gdxInFile=def1orgin.gdx
$if not errorlevel 1 $abort data contract should be different
* Run def2: Load data implicitely and generate in/out symbols in def2impl.gdx
$call.checkErrorLevel gams def2.gms lo=%gams.lo% IDCGDXInput=def1orgin.gdx IDCGenerateGDX=def2impl.gdx --one=1
* Compare the different in/out files, they should be identical
$call.checkErrorLevel gdxdiff def2expl.gdx def2impl.gdx > %system.nullfile%
* Run def1: Load data implicitely and generate in/out symbols in def1impl.gdx, use different label prefix to have different data statements
$call.checkErrorLevel gams def1.gms lo=%gams.lo% IDCGDXInput=def1orgin.gdx IDCGenerateGDX=def1impl.gdx  --pp=pp 
* Compare the different in/out files, they should be identical
$call.checkErrorLevel gdxdiff def1impl.gdx def1orginout.gdx > %system.nullfile%

$ifThen setEnv GMSPYTHONHOME
* Run def3: Create data contract and compare with def1 data contract (IDCJSON)
$ call.checkErrorLevel gams def3.gms lo=%gams.lo% IDCGenerateJSON=def3.json IDCJSON=def1.json --one=1
* Compare JSON text files, they shopuld be different due to different symbol order in def1 and def3
$ call diff def1.json def3.json > %system.nullfile% 2>&1
$ if not errorlevel 1 $abort json files def1.json def3.json should be different
$ call gams def3.gms lo=%gams.lo% IDCJSON=def1.json --one=one 
$ if not errorlevel 1 $abort data contract should be different
* Run def3: Load data implicitely and generate in/out symbols in def3impl.gdx
$ call.checkErrorLevel gams def3.gms lo=%gams.lo% IDCGDXInput=def1orgin.gdx IDCGenerateGDX=def3impl.gdx --one=1
* Compare the different in/out files, they should be identical
$ call.checkErrorLevel gdxdiff def2impl.gdx def3impl.gdx > %system.nullfile%
$endIf

*
* Check nesting of $on/offExternalIn/Output
*
$onEchoV > nl.gms
$onExternalOutput
$offExternalOutput
$onExternalInput
scalar a /0/;
$offExternalInput
$onExternalInput
$offExternalInput
$onExternalOutput
scalar b /1/;
$offExternalOutput
$onExternalOutput
$offExternalOutput
$onExternalInput
$offExternalInput
$offecho
$call.checkErrorLevel gams nl.gms lo=%gams.lo%
*
* Some runs that should fail
*
$onEchoV > nl.gms
$onExternalInput
$onExternalInput
scalar a /0/;
$offExternalInput
$offecho
$call gams nl.gms lo=%gams.lo%
$if not errorlevel 1 $abort expect nl.gms to fail. 

$onEchoV > nl.gms
scalar a /0/;
$onExternalInput
$offecho
$call gams nl.gms lo=%gams.lo%
$if not errorlevel 1 $abort expect nl.gms to fail. 

$onEchoV > nl.gms
scalar a /0/;
$offExternalInput
$offecho
$call gams nl.gms lo=%gams.lo%
$if not errorlevel 1 $abort expect nl.gms to fail. 

$onEchoV > nl.gms
$onExternalInput
scalar a /0/;
$offExternalInput
$offExternalInput
$offecho
$call gams nl.gms lo=%gams.lo%
$if not errorlevel 1 $abort expect nl.gms to fail. 
*
* Same runs with Output
*
$onEchoV > nl.gms
$onExternalOutput
$onExternalOutput
scalar a /0/;
$offExternalOutput
$offecho
$call gams nl.gms lo=%gams.lo%
$if not errorlevel 1 $abort expect nl.gms to fail. 

$onEchoV > nl.gms
scalar a /0/;
$onExternalOutput
$offecho
$call gams nl.gms lo=%gams.lo%
$if not errorlevel 1 $abort expect nl.gms to fail. 

$onEchoV > nl.gms
scalar a /0/;
$offExternalInput
$offecho
$call gams nl.gms lo=%gams.lo%
$if not errorlevel 1 $abort expect nl.gms to fail. 

$onEchoV > nl.gms
$onExternalOutput
scalar a /0/;
$offExternalOutput
$offExternalOutput
$offecho
$call gams nl.gms lo=%gams.lo%
$if not errorlevel 1 $abort expect nl.gms to fail. 
*
* Overlapping input and output
*
$onEchoV > nl.gms
$onExternalOutput
scalar a /0/;
$onExternalInput
$offExternalOutput
scalar a /0/;
$offExternalInput
$offecho
$call gams nl.gms lo=%gams.lo%
$if not errorlevel 1 $abort expect nl.gms to fail.
*
* Test if $gdxIn, $[bat,lib,sys]include, $call, $hiddencall are ignored
*
$call cp -f def1orginout.gdx x.gdx
$echo $log hello > x.gms
$call cp -f x.gms "%gams.sysdir%x.gms"
$call cp -f x.gms "%gams.sysdir%inclib"
$onEchoV > ign.gms
$onExternalInput
$call.checkErrorLevel gams x.gms lo=%gams.lo%
$include x.gms
$batinclude x.gms
$libinclude x.gms
$sysinclude x.gms
scalar p0;
$gdxIn x.gdx
$load p0
$ifThen setEnv GMSPYTHONHOME
$ onEmbeddedCode Python:
  %pystatement%
$ offEmbeddedCode
$endIf
$offExternalInput
$offEcho
$call.checkErrorLevel gams ign.gms --pystatement=pass lo=%gams.lo% 
$call rm -f x.gms x.gdx "%gams.sysdir%x.gms" "%gams.sysdir%inclib%system.dirsep%x.gms"
$call gams ign.gms --pystatement=nonsense lo=%gams.lo% 
$if not errorlevel 1 $abort 'expect call to fail'
$call.checkErrorLevel gams ign.gms --pystatement=nonsense lo=%gams.lo% IDCGDXInput=def1orginout.gdx 
*
* Test that special $load instructions are ignored
*
$onEchoV > sloadgen.gms
set i /1,2/, j /a,b/;
parameter p(i,j) / 1.a 1, 2.b 1/;
$gdxout sloadqonly
$unload p=q
$offEcho
$call.checkErrorLevel gams sloadgen.gms lo=%gams.lo% gdx=sloadgen

$onEchoV > sload.gms
$onExternalInput
set i,j; parameter p(i,j);
$gdxin sloadqonly
$load i<q.dim1 j<q.dim2 p=q
$offExternalInput
$offEcho
$call.checkErrorLevel gams sload.gms lo=%gams.lo% gdx=sload1
$call.checkErrorLevel gams sload.gms lo=%gams.lo% IDCGDXInput=sloadgen.gdx gdx=sload2
$call.checkErrorLevel gdxdiff sload1.gdx sload2.gdx > %system.nullfile%

$onEchoV > sloadec.gms
$onExternalInput
set i,j; parameter p(i,j);
$onEmbeddedCode Python:
gams.set('p',[('1','a',1),('2','b',1)])
$offEmbeddedCode i<p.dim1 j<p.dim2 p
$offExternalInput
$offEcho
$ifThen setEnv GMSPYTHONHOME
$  call.checkErrorLevel gams sloadec.gms lo=%gams.lo% gdx=sloadec1
$  call.checkErrorLevel gams sloadec.gms lo=%gams.lo% IDCGDXInput=sloadgen.gdx gdx=sloadec2
$  call.checkErrorLevel gdxdiff sloadec1.gdx sloadec2.gdx > %system.nullfile%
$endIf
*
* Make sure ponly is not enough (does not contain i or j)
*
$call gams sload.gms lo=%gams.lo% IDCGDXInput=sloadqonly.gdx gdx xxx
$if not errorlevel 1 $abort 'expect sload to fail'
$call gams sloadec.gms lo=%gams.lo% IDCGDXInput=sloadqonly.gdx gdx xxx
$if not errorlevel 1 $abort 'expect sloadec to fail'
*
* Test that we have an external data statement
*
$onEchoV > havedata.gms
set i /1/;
$onExternalInput
set i;
$offExternalInput
$offEcho
$call gams havedata.gms lo=%gams.lo% 
$if not errorlevel 1 $abort 'expect havedata to fail'

$onEchoV > havedata.gms
$onExternalInput
set i;
$offExternalInput
set i /1/;
$offEcho
$call gams havedata.gms lo=%gams.lo% 
$if not errorlevel 1 $abort 'expect havedata to fail'
*
* Test that we can't overwrite IDCGDXInput while running
*
$onEchoV > overw.gms
set k / k1*k5 /;
$gdxout ext.gdx
$unload k=i
$gdxout
$onExternalInput
set i /1/;
$offExternalInput
display i;
$offEcho
* Create ext.gdx
$call rm -f ext.gdx
$call gams overw.gms lo=%gams.lo% IDCGDXInput=ext.gdx
$if not errorlevel 1 $abort 'expect havedata to fail'
*
* Test that explicit and implicit data work (data statement)
*
$echo set k / k3*k8 /, kk(k) / #k /; > gendata.gms
$call.checkErrorLevel gams gendata.gms lo=%gams.lo% gdx=gendata
$onEchoV > mr.gms
set k / k1*k5 /;
$onExternalInput
$%onxxx%
set k /k3*k%klast%/;
$offExternalInput
$eval cardk card(k)
$ife %cardk%<>%num% $abort 'wrong cardinality (%cardk%) of k expect %num%'
$offEcho
$call.checkErrorLevel gams mr.gms lo=%gams.lo% --num=8 --klast=8 --onxxx=onmulti
$call.checkErrorLevel gams mr.gms lo=%gams.lo% --num=6 --klast=8 --onxxx=onmultiR
$call.checkErrorLevel gams mr.gms lo=%gams.lo% --num=8 --klast=3 --onxxx=onmulti  IDCGDXInput=gendata.gdx
$call.checkErrorLevel gams mr.gms lo=%gams.lo% --num=6 --klast=3 --onxxx=onmultiR IDCGDXInput=gendata.gdx
*
* Test that explicit and implicit data work (GDX $load[M,R])
*
$onEchoV > mr.gms
set k / k1*k5 /;
$onExternalInput
set k;
$gdxin gendataX.gdx
$%loadxxx% k
$offExternalInput
$eval cardk card(k)
$ife %cardk%<>%num% $abort 'wrong cardinality (%cardk%) of k expect %num%'
$offEcho
$call cp -f gendata.gdx gendataX.gdx
$call.checkErrorLevel gams mr.gms lo=%gams.lo% --num=8 --loadxxx=loadM
$call.checkErrorLevel gams mr.gms lo=%gams.lo% --num=6 --loadxxx=loadR
$call rm -f gendataX.gdx
$call.checkErrorLevel gams mr.gms lo=%gams.lo% --num=8 --loadxxx=loadM IDCGDXInput=gendata.gdx
$call.checkErrorLevel gams mr.gms lo=%gams.lo% --num=6 --loadxxx=loadR IDCGDXInput=gendata.gdx
*
* Test that explicit and implicit data work (GDX $onmulti[R] and $load)
*
$onEchoV > mr.gms
set k / k1*k5 /;
$%onxxx%
$onExternalInput
set k;
$gdxin gendataX.gdx
$load k
$offExternalInput
$eval cardk card(k)
$ife %cardk%<>%num% $abort 'wrong cardinality (%cardk%) of k expect %num%'
$offEcho
$call cp -f gendata.gdx gendataX.gdx
$call.checkErrorLevel gams mr.gms lo=%gams.lo% --num=8 --onxxx=onmulti
$call.checkErrorLevel gams mr.gms lo=%gams.lo% --num=6 --onxxx=onmultiR
$call rm -f gendataX.gdx
$call.checkErrorLevel gams mr.gms lo=%gams.lo% --num=8 --onxxx=onmulti  IDCGDXInput=gendata.gdx
$call.checkErrorLevel gams mr.gms lo=%gams.lo% --num=6 --onxxx=onmultiR IDCGDXInput=gendata.gdx
*
* load obeys filtered/domain checked
*
$onEchoV > dc.gms
set k / k1*k3 /;
$onExternalInput
set kk(k);
$gdxin gendataX.gdx
$%loadxxx% kk
$offExternalInput
$eval cardkk card(kk)
$if not %cardkk%==%num% $abort 'wrong cardinality (%cardkk%) of kk expect %num%'
$offEcho
$call cp -f gendata.gdx gendataX.gdx
$call.checkErrorLevel gams dc.gms lo=%gams.lo% --num=1    --loadxxx=load
$call                 gams dc.gms lo=%gams.lo% --num=UNDF --loadxxx=loaddc
$if not errorlevel 1 $abort 'expect dc to fail'
$call rm -f gendataX.gdx
$call.checkErrorLevel gams dc.gms lo=%gams.lo% --num=1    --loadxxx=load    IDCGDXInput=gendata.gdx
$call                 gams dc.gms lo=%gams.lo% --num=UNDF --loadxxx=loaddc  IDCGDXInput=gendata.gdx
$if not errorlevel 1 $abort 'expect dc to fail'
*
* Test table layout
*
$onEchoV > table.gms
set k / kHeader1,kHeader2 /;
$onExternalInput
table t(*,k)
    kHeader1
r1         1
;
$offExternalInput
$offEcho
$call.checkErrorLevel gams table.gms lo=%gams.lo% IDCGenerateJSON=table.json
$call.checkErrorLevel grep kheader1 table.json > %system.nullfile%
* table column index also external input
$onEchoV > table.gms
$onExternalInput
set k / kHeader1,kHeader2 /;
table t(*,k)
    kHeader1
r1         1
;
$offExternalInput
$offEcho
$call.checkErrorLevel gams table.gms lo=%gams.lo% IDCGenerateJSON=table.json
$call grep kheader1 table.json > %system.nullfile%
$if not errorlevel 1 $abort 't should not be a table'
* table column index single uel
$onEchoV > table.gms
set k / kHeader1 /;
$onExternalInput
table t(*,k)
    kHeader1
r1         1
;
$offExternalInput
$offEcho
$call.checkErrorLevel gams table.gms lo=%gams.lo% IDCGenerateJSON=table.json
$call grep kheader1 table.json > %system.nullfile%
$if not errorlevel 1 $abort 't should not be a table'
* table column index also implicit external input
$onEchoV > table.gms
set k;
$onExternalInput
table t(*,k<)
    kHeader1
r1         1
;
$offExternalInput
$offEcho
$call.checkErrorLevel gams table.gms lo=%gams.lo% IDCGenerateJSON=table.json
$call grep kheader1 table.json > %system.nullfile%
$if not errorlevel 1 $abort 't should not be a table'

* table column index is universe
$onEchoV > table.gms
$onExternalInput
table t(*,*)
    kHeader1
r1         1
;
$offExternalInput
$offEcho
$call.checkErrorLevel gams table.gms lo=%gams.lo% IDCGenerateJSON=table.json
$call grep kheader1 table.json > %system.nullfile%
$if not errorlevel 1 $abort 't should not be a table'

$label cont

* Test --HCUBE stuff
$onechoV > hcube.gms
$onExternalInput
scalar f / 3 /
singleton set ss1 / 'empty' 'nothing' /
$offExternalInput
$eval f f
$if set expectf $if not %f%==%expectf% $abort expect scalar f to be %expectf%
$eval.set SS1TL ss1.tl
$eval.set SS1TE ss1.te
$if set expectsstl $if not x%SS1TL%==x%expectsstl% $abort expect singleton set ss1 tl to be %expectsstl%
$if set expectsste $if not x%SS1TE%==x%expectsste% $abort expect singleton set ss1 te to be %expectsste%
$offecho

$call.checkErrorLevel gamslib -q 1 && gams trnsport lo=0 gdx=hcube.gdx a=c
$call.checkErrorLevel gams hcube lo=%gams.lo% --HCUBE_SCALARV_f=15 --expectf=15 --HCUBE_SCALARV_ss1=leer --expectss1=leer --HCUBE_SCALART_ss1=nuescht --expectss1=nuescht idcGDXInput=hcube.gdx
$call.checkErrorLevel gams hcube lo=%gams.lo% --expectf=90 --HCUBE_SCALARV_ss1=leer --expectss1=leer --HCUBE_SCALART_ss1=nuescht --expectss1=nuescht idcGDXInput=hcube.gdx
* Need to use a pf file because empty string "" is handled funny in Unix shell
$onecho > hcube.pf
--HCUBE_SCALARV_ss1=""
--expectss1="" 
--HCUBE_SCALART_ss1=""
--expectss1=""
idcGDXInput=hcube.gdx
$offecho
$call.checkErrorLevel gams hcube lo=%gams.lo% pf=hcube.pf --HCUBE_SCALARV_f=15 --expectf=15 
$call.checkErrorLevel gams hcube lo=%gams.lo% pf=hcube.pf --expectf=90

* Check for handling of different domain elements in IDC data and declared data
$onecho > d.gms
Scalar x  / 6 /;
Set j     / j3*j5 /
    jj(j) / j3*j5 /;
Set Table jjj(j,j)
   j5
j5  1  ;
$offecho

$onecho > d2.gms
$onexternalinput
Set j     / j6*j7 /
    jj(j) / j6,j7 /;

Set Table jjj(j,j)
   j8
j6  1  ;
$offexternalinput
$offecho

$call.checkErrorlevel gams d  lo=%gams.lo% GDX d
$call.checkErrorlevel gams d2 lo=%gams.lo% IDCGDXInput d.gdx

* Make sure that uels from ignored datastatements get not stored
$onecho > d3.gms
$onexternalinput
Set j     / j5*j10 /
    jj(j) / j6,j10 /;

Set Table jjj(j,j)
    j5
j10  1  ;
$offexternalinput

Alias(u,*);
abort$(card(u)<>3) 'External data has just 3 uels';
loop(j,
  if(sameas(j,'j3'), abort$(ord(j)<>1) 'UEL 1 should be j3');
  if(sameas(j,'j4'), abort$(ord(j)<>2) 'UEL 2 should be j4');
  if(sameas(j,'j5'), abort$(ord(j)<>3) 'UEL 3 should be j5');
);
$offecho
$call.checkErrorlevel gams d3 lo=%gams.lo% IDCGDXInput d.gdx

* External input symbols cannot be changed at execution time - by default
$onEcho > idcProtect.gms
Set i / i1*i6 /;
Set j / j1*j3 /;

$onexternalInput
Scalar    x       / 5 /
          y       / 8 /;
Parameter A(*)    / system.empty 0 /
          b(i,j)  / system.empty.system.empty 0 /;
Set       ij(i,j) / system.empty.system.empty   /;
$offexternalInput
$onmulti

x=6;
$ifThenE %expectErrors%=1
$if errorfree $abort Assignment to x successful but error expected
$clearerror
$endIf

execute_load 'd.gdx', x;
$ifThenE %expectErrors%=1
$if errorfree $abort Loading x from GDX successful but error expected
$clearerror
$endIf

$ifThen setEnv GMSPYTHONHOME
EmbeddedCode Python:
gams.set("x", [1])
PauseEmbeddedCode x
$ifThenE %expectErrors%=1
$if errorfree $abort Loading x from EC successful but error expected
$clearerror
$endIf

EmbeddedCode Python:
gams.set("x", [1])
EndEmbeddedCode x
$ifThenE %expectErrors%=1
$if errorfree $abort Loading x from EC successful but error expected
$clearerror
$endIf
$endIf

Option x<y;
$ifThenE %expectErrors%=1
$if errorfree $abort Projecting to x successful but error expected
$clearerror
$endIf

Parameter c(j); c(j)=ord(j);
option b>c;
$ifThenE %expectErrors%=1
$if errorfree $abort Permuting into b successful but error expected
$clearerror
$endIf

option clear=x;
$ifThenE %expectErrors%=1
$if errorfree $abort Clearing x successful but error expected
$clearerror
$endIf

option kill=x;
$ifThenE %expectErrors%=1
$if errorfree $abort Killing x successful but error expected
$clearerror
$endIf

option shuffle = A;
$ifThenE %expectErrors%=1
$if errorfree $abort Shuffeling A successful but error expected
$clearerror
$endIf

Option ij(i:j);
$ifThenE %expectErrors%=1
$if errorfree $abort Index matching for ij successful but error expected
$clearerror
$endIf
$offEcho

* Run with a=c since we mess a lot with $clearError
$call.checkErrorlevel gams idcProtect.gms lo=%gams.lo% a=c --expectErrors=1

* With IDCProtect=0, things should work
$call.checkErrorlevel gams idcProtect.gms lo=%gams.lo% IDCProtect=0 --expectErrors=0