Missing functionality in Resolution Test framework

[alexjpwalker]

The following issues have been highlighted in the Resolution Test framework while writing resolution tests:

Matching attribute types

when {
  $x isa name;
};

will currently fail structural validation. The error is: A structural validation error has occurred. The type [name] of role player [V20672] is not allowed to play Role [instance]

Attribute re-attachment

Currently, it is not possible to complete the materialised keyspace upon defining

define
      transfer-string-attribute-to-other-people sub rule,
      when {
        $x isa person, has string-attribute $r1;
        $y isa person;
      },
      then {
        $y has string-attribute $r1;
      };

because attaching an existing attribute to a second owner is not supported. A blank AssertionError is thrown.

Re-attachment of unrelated attributes

A different error is thrown when the inferred attribute has an unrelated type to the non-inferred attribute:

transfer-attribute-value-to-unrelated-attribute sub rule,
      when {
        $x isa person, has string-attribute $r1;
      },
      then {
        $x has unrelated-attribute $r1;
      };

The error is: grakn.core.kb.graql.exception.GraqlSemanticException: Downcasting concepts from type Base Type [ATTRIBUTE_TYPE] - Id [V16560] - Label [string-attribute] to type Base Type [ATTRIBUTE_TYPE] - Id [V12464] - Label [unrelated-attribute] is not allowed.

This one might need a bit of thought, since the variable $r1 refers to two different concepts, which is a difference between rules and the rest of graql.

Materialised keyspace fails to de-duplicate attributes when counting them

Given

define
      lucky-number sub attribute, value long;
      person has lucky-number;
      rule-1337 sub rule, when { $x isa person; }, then { $x has lucky-number $n; $n 1337; };
      rule-1667 sub rule, when { $x isa person; }, then { $x has lucky-number $n; $n 1667; };
      rule-1997 sub rule, when { $x isa person; }, then { $x has lucky-number $n; $n 1997; };
insert
      $x isa person, has ref 0;
      $y isa person, has ref 1;

then the completeness test should return [3] inferred concepts in the materialised keyspace, but it in fact counts [6] as it fails to de-duplicate the inferred attributes.

Slowness of query

The scenario 3-hop transitivity takes too long to run.

Infinite materialised keyspace

From Scenario: when resolution produces an infinite stream of answers, limiting the answer size allows it to terminate in relation-inference.feature:

Given for each session, graql define
      """
      define

      dream sub relation,
        relates dreamer,
        relates dream-subject,
        plays dream-subject;

      person plays dreamer, plays dream-subject;

      inception sub rule,
      when {
        $x isa person;
        $z (dreamer: $x, dream-subject: $y) isa dream;
      }, then {
        (dreamer: $x, dream-subject: $z) isa dream;
      };
      """
    Given for each session, graql insert
      """
      insert
      $x isa person, has name "Yusuf";
      # If only Yusuf didn't dream about himself...
      (dreamer: $x, dream-subject: $x) isa dream;
      """
    When materialised keyspace is completed

It's currently unclear what this step should do - by definition, forward chaining never terminates.

Type generation

Given

define

      duelist sub person;
      poet sub person;

      romeo-is-a-duelist sub rule,
      when {
        $x isa person, has name "Romeo";
      }, then {
        $x isa duelist;
      };

the step "materialised keyspace is completed" fails with an error saying that downcasting concepts (person to duelist) is not allowed".

Schema queries (variable types)

The step all answers are correct in reasoned keyspace throws various errors, including: ["Currently we only handle Things, and not Types", "The concept RELATION_TYPE - label [friendship] is not of type concept.api.Thing"] when the query to test contains a variable type.

Unidentified issue 1

Given

define
      lucky-number sub attribute, value long;
      person has lucky-number;
      rule-1337 sub rule, when { $x isa person; }, then { $x has lucky-number $n; $n 1337; };
      rule-1667 sub rule, when { $x isa person; }, then { $x has lucky-number $n; $n 1667; };
insert
      $x isa person, has ref 0;
      $y isa person, has ref 1;

Then

      match
        $x isa person, has lucky-number $m;
        $y isa person, has lucky-number $n;
        $m >= $n;
      get;

should work fine, but in fact it throws an especially bizarre error message when verifying that answers are correct in the reasoned keyspace:

grakn.core.test.behaviour.resolution.framework.Resolution$CorrectnessException: Resolution query had 0 answers, it should have had 1. The query is:
 match { $x10 (instance: $r1-x) isa isa-property, has type-label "person"; $r1-n 1337; $x11 (owner: $r1-x) isa has-attribute-property, has lucky-number $r1-n; $r0-y has ref 0; $r1-x has ref 0; $_ (body: $x10, head: $x11) isa resolution, has rule-label "rule-1337"; $r0-x has lucky-number $r0-m; $r0-x has ref 1; $r0-n 1337 isa lucky-number; $x9 (owner: $r1-x) isa has-attribute-property, has lucky-number $r1-n; $_ (body: $x8, head: $x9) isa resolution, has rule-label "rule-1337"; $r1-x has ref 1; $r0-y has lucky-number $r0-n; $r0-m 1337 isa lucky-number; $x8 (instance: $r1-x) isa isa-property, has type-label "person"; $r1-x has lucky-number $r1-n; $r0-x isa person; $r0-y isa person; $r1-x isa person; }; get;

Unidentified issue 2

Given

define
      lucky-number sub attribute, value long;
      person has lucky-number;
      rule-1337 sub rule, when { $x isa person; }, then { $x has lucky-number $n; $n 1337; };
      rule-1667 sub rule, when { $x isa person; }, then { $x has lucky-number $n; $n 1667; };
insert
      $x isa person, has ref 0;
      $y isa person, has ref 1;

Then

      match
        $x isa person, has lucky-number $m;
        $y isa person, has lucky-number $n;
        $m > $n;
        $n > 1667;
      get;

should work fine, but in fact it throws: No resolution queries were constructed for query

Unidentified issue 3

Given for each session, graql define
      """
      define

      iceland-sells-drinks sub rule,
      when {
        $x isa soft-drink;
      },
      then {
        $x has retailer 'Iceland';
      };

      poundland-sells-drinks sub rule,
      when {
        $x isa soft-drink;
      },
      then {
        $x has retailer 'Poundland';
      };

      londis-sells-drinks sub rule,
      when {
        $x isa soft-drink;
      },
      then {
        $x has retailer 'Londis';
      };
      """
    Given for each session, graql insert
      """
      insert $x isa soft-drink, has name "Fanta", has ref 0;
      """
    When materialised keyspace is completed
    Then for graql query
      """
      match
        $x has retailer $rx;
        $rx contains "land";
      get;
      """

This fails the completeness check - The complete KB contains 0 inferred concepts, whereas the test KB contains 3 inferred concepts. It also fails to complete the materialised keyspace sometimes: Java.lang.IllegalArgumentException: Neither the sideEffects, map, nor path has a #1594749248821829-key: WherePredicateStep(eq(#1594749248821829)) at org.apache.tinkerpop.gremlin.process.traversal.step.Scoping.getScopeValue(Scoping.java:124) ...

Unidentified issue 4

Given for each session, graql define
      """
      define

      iceland-sells-drinks sub rule,
      when {
        $x isa soft-drink;
      },
      then {
        $x has retailer 'Iceland';
      };

      poundland-sells-drinks sub rule,
      when {
        $x isa soft-drink;
      },
      then {
        $x has retailer 'Poundland';
      };

      londis-sells-drinks sub rule,
      when {
        $x isa soft-drink;
      },
      then {
        $x has retailer 'Londis';
      };
      """
    Given for each session, graql insert
      """
      insert
      $x isa soft-drink, has name "Fanta", has ref 0;
      $y isa soft-drink, has name "Tango", has ref 1;
      """
    When materialised keyspace is completed
    Then for graql query
      """
      match
        $x has retailer $rx;
        $y has retailer $ry;
        $rx == $ry;
        $ry contains 'land';
      get;
      """

This fails the correctness check -

Resolution query had 0 answers, it should have had 1. The query is:
 match { $r0-rx "Iceland" isa retailer; $r1-x has retailer $r1-1594203238859810; $r0-ry "Iceland" isa retailer; $x5 (owner: $r1-x) isa has-attribute-property, has retailer $r1-1594203238859810; $r0-y has ref 0; $r1-x has ref 0; $r0-x has retailer $r0-rx; $r0-x has ref 1; $r1-x isa soft-drink; $r1-x has ref 1; $r0-ry contains "land"; $_ (body: $x2, head: $x3) isa resolution, has rule-label "iceland-sells-drinks"; $r0-y has retailer $r0-ry; $r1-x has retailer $r1-1594203238859788; $x3 (owner: $r1-x) isa has-attribute-property, has retailer $r1-1594203238859788; $_ (body: $x4, head: $x5) isa resolution, has rule-label "iceland-sells-drinks"; $x2 (instance: $r1-x) isa isa-property, has type-label "soft-drink"; $x4 (instance: $r1-x) isa isa-property, has type-label "soft-drink"; }; get;

Unidentified issue 5

Given for each session, graql define
      """
      define

      iceland-sells-drinks sub rule,
      when {
        $x isa soft-drink;
      },
      then {
        $x has retailer 'Iceland';
      };

      poundland-sells-drinks sub rule,
      when {
        $x isa soft-drink;
      },
      then {
        $x has retailer 'Poundland';
      };

      londis-sells-drinks sub rule,
      when {
        $x isa soft-drink;
      },
      then {
        $x has retailer 'Londis';
      };
      """
    Given for each session, graql insert
      """
      insert
      $x isa soft-drink, has name "Fanta", has ref 0;
      $y isa soft-drink, has name "Tango", has ref 1;
      """

then: The complete KB contains 0 inferred concepts, whereas the test KB contains 3 inferred concepts.

Unidentified issue 6

Given for each session, graql define
      """
      define
      person has age;
      age sub attribute, value long;
      not-ten sub rule,
      when {
        $x isa person;
        not { $x has age 10; };
      }, then {
        $x has name "Not Ten";
      };
      """
    Given for each session, graql insert
      """
      insert
      $x isa person, has age 10;
      $y isa person, has age 20;
      """
    When materialised keyspace is completed
    Then for graql query
      """
      match $x has name "Not Ten", has age 20; get;
      """
    Then all answers are correct in reasoned keyspace

The last step should pass, but instead it fails to find any answers.

Unidentified issue 7

Given for each session, graql define
      """
      define
      dominion sub relation, relates ruler, relates ruled-person;
      giant-turtle sub entity, plays ruler;
      person plays ruled-person;

      giant-turtles-rule-the-world sub rule,
      when {
        $r (ruled-person: $p) isa dominion;
        $gt isa giant-turtle;
      }, then {
        $r (ruler: $gt) isa dominion;
      };
      """
    Given for each session, graql insert
      """
      insert
      $x isa person;
      $y isa person;
      $z isa giant-turtle;

      (ruled-person: $x) isa dominion;
      (ruled-person: $y) isa dominion;
      """
    When materialised keyspace is completed
    Then for graql query
      """
      match
        (ruled-person: $x, ruler: $y) isa dominion;
      get;
      """
    Then all answers are correct in reasoned keyspace

The last step should find answers, but it fails to find any.

Unidentified issue 8

From Scenario: rules can divide entities into groups, linking each entity group to a specific concept by attribute value in value-predicate.feature:

    Given for each session, graql define
      """
      define

      soft-drink plays priced-item;

      price-range sub attribute, value string,
        plays price-category;

      price-classification sub relation,
        relates priced-item,
        relates price-category;

      expensive-drinks sub rule,
      when {
        $x has price >= 3.50;
        $y "expensive" isa price-range;
      }, then {
        (priced-item: $x, price-category: $y) isa price-classification;
      };

      not-expensive-drinks sub rule,
      when {
        $x has price < 3.50;
        $y "not expensive" isa price-range;
      }, then {
        (priced-item: $x, price-category: $y) isa price-classification;
      };

      low-price-drinks sub rule,
      when {
        $x has price < 1.75;
        $y "low price" isa price-range;
      }, then {
        (priced-item: $x, price-category: $y) isa price-classification;
      };

      cheap-drinks sub rule,
      when {
        (priced-item: $x, price-category: $y) isa price-classification;
        $y "not expensive" isa price-range;
        (priced-item: $x, price-category: $y2) isa price-classification;
        $y2 "low price" isa price-range;
        $y3 "cheap" isa price-range;
      }, then {
        (priced-item: $x, price-category: $y3) isa price-classification;
      };
      """
    Given for each session, graql insert
      """
      insert

      $x isa soft-drink, has name "San Pellegrino Limonata", has price 3.99;
      $y isa soft-drink, has name "Sprite", has price 2.00;
      $z isa soft-drink, has name "Tesco Value Lemonade", has price 0.39;

      $p1 "expensive" isa price-range;
      $p2 "not expensive" isa price-range;
      $p3 "low price" isa price-range;
      $p4 "cheap" isa price-range;
      """
    When materialised keyspace is completed
    Then for graql query
      """
      match
        $x "not expensive" isa price-range;
        ($x, priced-item: $y) isa price-classification;
      get;
      """
    Then all answers are correct in reasoned keyspace
    Then answer size in reasoned keyspace is: 2
    Then for graql query
      """
      match
        $x "low price" isa price-range;
        ($x, priced-item: $y) isa price-classification;
      get;
      """
    Then all answers are correct in reasoned keyspace
    Then answer size in reasoned keyspace is: 1
    Then for graql query
      """
      match
        $x "cheap" isa price-range;
        ($x, priced-item: $y) isa price-classification;
      get;
      """
#    Then all answers are correct in reasoned keyspace

The last step should succeed, but the resolution query fails to have any answers.

Unidentified issue 9

From Scenario: the relation type constraint can be excluded from a reasoned match query in relation-inference.feature

Given for each session, graql define
      """
      define
      transitive-location sub rule,
      when {
        (location-subordinate: $x, location-superior: $y) isa location-hierarchy;
        (location-subordinate: $y, location-superior: $z) isa location-hierarchy;
      }, then {
        (location-subordinate: $x, location-superior: $z) isa location-hierarchy;
      };
      """
    Given for each session, graql insert
      """
      insert
      $x isa place, has name "Turku Airport";
      $y isa place, has name "Turku";
      $z isa place, has name "Finland";

      (location-subordinate: $x, location-superior: $y) isa location-hierarchy;
      (location-subordinate: $y, location-superior: $z) isa location-hierarchy;
      """
    When materialised keyspace is completed
    Then for graql query
      """
      match
        $a isa place, has name "Turku Airport";
        ($a, $b);
        $b isa place, has name "Turku";
        ($b, $c);
      get;
      """
    Then all answers are correct in reasoned keyspace

The last step should pass, but in fact it returns the wrong number of answers: the materialised keyspace has 4 answers, but it should have had 2.

Unidentified issue 10

From Scenario: inferred relations can be filtered by shared attribute ownership in relation-inference.feature

Given for each session, graql define
      """
      define
      selection sub relation, relates choice1, relates choice2;
      person plays choice1, plays choice2;
      symmetric-selection sub rule,
      when {
        (choice1: $x, choice2: $y) isa selection;
      }, then {
        (choice1: $y, choice2: $x) isa selection;
      };
      transitive-selection sub rule,
      when {
        (choice1: $x, choice2: $y) isa selection;
        (choice1: $y, choice2: $z) isa selection;
      }, then {
        (choice1: $x, choice2: $z) isa selection;
      };
      """
    Given for each session, graql insert
      """
      insert
      $x isa person, has name "a";
      $y isa person, has name "b";
      $z isa person, has name "c";

      (choice1: $x, choice2: $y) isa selection;
      (choice1: $y, choice2: $z) isa selection;
      """
    When materialised keyspace is completed
    Then for graql query
      """
      match
        (choice1: $x, choice2: $y) isa selection;
        $x has name $n;
        $y has name $n;
      get;
      """
    Then all answers are correct in reasoned keyspace

Firstly, materialisation takes about 4 minutes, which is far too long.
Secondly, 'all answers are correct' step takes a very long time to execute.

Unidentified issue 11

Scenario: when evaluating negation blocks, global subgoals are not updated

    The test highlights a potential issue with eagerly updating global subgoals when branching out to determine whether
    negation conditions are met. When checking negation satisfiability, we are interested in a first answer that can
    prove us wrong - we are not exhaustively exploring all answer options.

    Consequently, if we use the same subgoals as for the main loop, we can end up with a query which answers weren't
    fully consumed but that was marked as visited.

    As a result, if it happens that a negated query has multiple answers and is visited more than a single time
    - because of the admissibility check, answers might be missed.

    Given for each session, graql define
      """
      define

      session sub entity,
          plays parent-session;
      fault sub entity,
          plays relevant-fault,
          plays identified-fault,
          plays diagnosed-fault;
      question sub entity,
          has response,
          plays identifying-question,
          plays question-logged,
          plays question-not-answered;

      response sub attribute, value string;

      reported-fault sub relation,
          relates relevant-fault,
          relates parent-session;

      logged-question sub relation,
          relates question-logged,
          relates parent-session;

      unanswered-question sub relation,
          relates question-not-answered,
          relates parent-session;

      fault-identification sub relation,
          relates identifying-question,
          relates identified-fault;

      diagnosis sub relation,
          relates diagnosed-fault,
          relates parent-session;


      no-response-means-unanswered-question sub rule,
      when {
          $ques isa question;
          (question-logged: $ques, parent-session: $ts) isa logged-question;
          not {
              $ques has response $r;
          };
      }, then {
          (question-not-answered: $ques, parent-session: $ts) isa unanswered-question;
      };

      determined-fault sub rule,
      when {
          (relevant-fault: $flt, parent-session: $ts) isa reported-fault;
          not {
              (question-not-answered: $ques, parent-session: $ts) isa unanswered-question;
              ($flt, $ques) isa fault-identification;
          };
      }, then {
          (diagnosed-fault: $flt, parent-session: $ts) isa diagnosis;
      };
      """
    Given for each session, graql insert
      """
      insert
      $sesh isa session;
      $q1 isa question;
      $q2 isa question;
      $f1 isa fault;
      $f2 isa fault;
      (relevant-fault: $f1, parent-session: $sesh) isa reported-fault;
      (relevant-fault: $f2, parent-session: $sesh) isa reported-fault;

      (question-logged: $q1, parent-session: $sesh) isa logged-question;
      (question-logged: $q2, parent-session: $sesh) isa logged-question;

      (identified-fault: $f1, identifying-question: $q1) isa fault-identification;
      (identified-fault: $f2, identifying-question: $q2) isa fault-identification;
      """
    When materialised keyspace is completed
    Then for graql query
      """
      match (diagnosed-fault: $flt, parent-session: $ts) isa diagnosis; get;
      """
    Then answer size in reasoned keyspace is: 0
    Then answers are consistent across 5 executions in reasoned keyspace
#    Then materialised and reasoned keyspaces are the same size

The final check fails because: The complete KB contains 15 inferred concepts, whereas the test KB contains 13 inferred concepts.