Unable to adopt example of complex with usage from docs

[spalger]

I'm attempting to insert video records into edgedb which have links to tags and actors. I'm using the bulk insert method described at https://docs.edgedb.com/libraries/js/for and trying to adapt the complex example near the end to my data model. I'm fairly confident that I've mapped things correctly, and even when I migrate to a simpler data model that more closely matches the docs I get the same problem Expr or its aliases used outside of declared 'WITH' block scope. When I do some debugging and comment out that validation I'm able to produce edgeql which fails to execute.

Error
    at new EdgeDBError (node_modules/edgedb/dist/errors/base.js:12:8)
    at new ExecutionError (:1:28)
    at new IntegrityError (:1:28)
    at new MissingRequiredError (:1:28)
    at _parseErrorMessage (node_modules/edgedb/dist/baseConn.js:123:155)
    at <anonymous> (node_modules/edgedb/dist/baseConn.js:617:44)
    at processTicksAndRejections (:12:39) {
  source: undefined,
  originalLine: 39,
  originalColumn: 12
}

Interestingly though, if I put the query directly into the edgedb UI, replace $videos with <json>$videos, then use the following value for the variable, everything works!

[
  {
    "url": "https://www.youtube.com/watch?v=1",
    "xid": "1",
    "title": "title1",
    "duration": "PT1H",
    "thumbnail": "https://www.youtube.com/watch?v=1",
    "quality": "1080P",
    "category": "category1",
    "tags": ["tag1", "tag2"],
    "actors": ["actor1", "actor2"]
  }
]

If I run it a second time I don't get any new tags, actors, or videos. Perfect!

Unfortunately, it does not work if I try to run the same string directly with client.execute() or client.query(), I get the same generic error without any message mentioned above.

The query

const INSERT_VIDEOS_QUERY = e.params(
  { videos: e.array(Video) },
  ({ videos }) => {
    const vids = e.array_unpack(videos);
    const tags = e.for(
      e.op(
        "distinct",
        e.array_unpack(e.op(vids.tags, "++", e.array_agg(vids.category))),
      ),
      (tagName) =>
        e
          .insert(e.Tag, { name: tagName })
          .unlessConflict((tag) => ({ on: tag.name, else: tag })),
    );
    const actors = e.for(
      e.op("distinct", e.array_unpack(vids.actors)),
      (actorName) =>
        e
          .insert(e.Actor, { name: actorName })
          .unlessConflict((actor) => ({ on: actor.name, else: actor })),
    );

    return e.with(
      [vids, tags, actors],
      e.for(vids, (video) =>
        e
          .insert(e.Video, {
            url: video.url,
            xid: video.xid,
            title: video.title,
            duration: video.duration,
            thumbnail: video.thumbnail,
            quality: video.quality,
            category: e.assert_single(
              e.select(tags, (t) => ({
                filter: e.op(t.name, "=", video.category),
              })),
            ),
            tags: e.select(tags, (t) => ({
              filter: e.op(t.name, "in", e.array_unpack(video.tags)),
            })),
            actors: e.select(actors, (a) => ({
              filter: e.op(a.name, "in", e.array_unpack(video.actors)),
            })),
            createdAt: e.datetime_current(),
          })
          .unlessConflict(),
      ),
    );
  },
);

Schema

Your application schema.

module default {
  scalar type VideoQuality extending enum<'SD', '720P', '1080P', '4K'>;

  type Tag {
    required name: str {
      constraint exclusive;
    }
  }

  type Actor {
    required name: str {
      constraint exclusive;
    }
  }

  type Video {
    required url: str {
      constraint exclusive;
    }
    required xid: str;
    required title: str;
    required duration: duration;
    required thumbnail: str;
    required createdAt: datetime {
      default := datetime_current();
    }
    required quality: VideoQuality;

    required category: Tag;
    required multi tags: Tag;
    required multi actors: Actor;
  }
}

Generated EdgeQL

Running .toEdgeQL() produces the following error:

127 |                 ...index_1.util.flatMap(refData.aliases, (alias) => [
128 |                     ...walkExprCtx.seen.get(alias).parentScopes,
129 |                 ]),
130 |             ]) {
131 |                 if (scope === null || !validScopes.has(scope)) {
132 |                     throw new Error(refData.boundScope
                                ^
error: Expr or its aliases used outside of declared 'WITH' block scope
      at $toEdgeQL (/dist/src/edgeql-js/toEdgeQL.js:132:27)
      at /dist/src/DB.js:74:36
      at /dist/src/index.js:17:1

Source:

edgedb-js/packages/generate/src/syntax/toEdgeQL.ts

Lines 276 to 295 in 994470f

	// check all references and aliases are within this block
	const validScopes = new Set([
	withBlock,
	...walkExprCtx.seen.get(withBlock)!.childExprs,
	]);
	for (const scope of [
	...refData.parentScopes,
	...util.flatMap(refData.aliases, (alias) => [
	...walkExprCtx.seen.get(alias)!.parentScopes,
	]),
	]) {
	if (scope === null || !validScopes.has(scope)) {
	throw new Error(
	refData.boundScope
	? `Expr or its aliases used outside of declared 'WITH' block scope`
	: `Cannot extract repeated or aliased expression into 'WITH' block, ` +
	`expression or its aliases appear outside root scope`
	);
	}
	}

If I comment out that code I get the following edgeQL:

WITH
  __param__videos := <array<tuple<url: std::str, xid: std::str, title: std::str, duration: std::duration, thumbnail: std::str, quality: default::VideoQuality, category: std::str, tags: array<std::str>, actors: array<std::str>>>>$videos
SELECT (WITH
  __withVar_0 := std::array_unpack(__param__videos),
  __withVar_2 := (
    FOR __forVar__0 IN {(distinct std::array_unpack((__withVar_0.tags ++ std::array_agg(__withVar_0.category))))}
    UNION (
      (INSERT default::Tag {
        name := __forVar__0
      }
      UNLESS CONFLICT ON default::Tag.name
      ELSE (default::Tag))
    )
  ),
  __withVar_5 := (
    FOR __forVar__1 IN {(distinct std::array_unpack(__withVar_0.actors))}
    UNION (
      (INSERT default::Actor {
        name := __forVar__1
      }
      UNLESS CONFLICT ON default::Actor.name
      ELSE (default::Actor))
    )
  )
FOR __forVar__2 IN {__withVar_0}
UNION (
  (INSERT default::Video {
    url := __forVar__2.url,
    xid := __forVar__2.xid,
    title := __forVar__2.title,
    duration := __forVar__2.duration,
    thumbnail := __forVar__2.thumbnail,
    quality := __forVar__2.quality,
    category := (
      std::assert_single((WITH
        __scope_1_defaultTag := __withVar_2
      SELECT __scope_1_defaultTag {
        id
      }
      FILTER (__scope_1_defaultTag.name = __forVar__2.category)))
    ),
    tags := (
      WITH
        __scope_3_defaultTag := __withVar_2
      SELECT __scope_3_defaultTag {
        id
      }
      FILTER (__scope_3_defaultTag.name in std::array_unpack(__forVar__2.tags))
    ),
    actors := (
      WITH
        __scope_4_defaultActor := __withVar_5
      SELECT __scope_4_defaultActor {
        id
      }
      FILTER (__scope_4_defaultActor.name in std::array_unpack(__forVar__2.actors))
    ),
    createdAt := std::datetime_current()
  }
  UNLESS CONFLICT)
))

Versions (please complete the following information):

OS: macOS Sonoma 14.4.1
EdgeDB version (e.g. 2.0): 5.2+1e07d04
EdgeDB CLI version (e.g. 2.0): 5.0.0+c9dfeec
edgedb-js version (e.g. 0.20.10;): 1.5.4
Node/Deno version: v20.12.2

[scotttrinh]

Thanks for the detailed report, I'll see if I can untangle it past the point you got. A few questions to help in my investigation:

Unfortunately, it does not work if I try to run the same string directly with client.execute() or client.query(), I get the same generic error without any message mentioned above.

Can you give some more detail about "same generic error" here since I assume you don't mean "Expr or its aliases used outside of declared 'WITH' block scope" here since that's a query builder error not an error that the server would return.

Interestingly though, if I put the query directly into the edgedb UI, replace $videos with $videos, then use the following value for the variable, everything works!

Just to be clear, if you update your code to use json instead of e.array(e.Video) (and json_array_unpack) it also works with the query builder, yeah?

[spalger]

Thank you! Happy to help however I can.

Can you give some more detail about "same generic error" here since I assume you don't mean "Expr or its aliases used outside of declared 'WITH' block scope" here since that's a query builder error not an error that the server would return.

Correct, I'm referring to the first stack track in the issue which doesn't seem to have any useful message. Just re-ran my script again and it looks like the stack trace is slightly different than the first stack track reported. Seems this might specifically be an InvalidValueError?

Error
    at new EdgeDBError (node_modules/edgedb/dist/errors/base.js:12:8)
    at new ExecutionError (:1:28)
    at new InvalidValueError (:1:28)
    at _parseErrorMessage (node_modules/edgedb/dist/baseConn.js:123:155)
    at <anonymous> (node_modules/edgedb/dist/baseConn.js:617:44)
    at processTicksAndRejections (:12:39) {
  source: undefined,
  originalLine: 39,
  originalColumn: 12
}

Just to be clear, if you update your code to use json instead of e.array(e.Video) (and json_array_unpack) it also works with the query builder, yeah?

I'm not sure what you mean regarding that edit, I took the exact edgeql produced after commenting out that validation step, and only prefixed the$videos variable in the edgeql with <json> so that the UI was able to detect the proper type for the variable. The existing type-cast in the edgeql is too complex or something and leads to this error:

By prefixing the variable with <json> the query functions great