Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] [promptflow-evals] Failed to invoke the Azure CLI when running evaluation #3438

Open
Wix97 opened this issue Jun 19, 2024 · 0 comments
Assignees
Labels
bug Something isn't working

Comments

@Wix97
Copy link

Wix97 commented Jun 19, 2024

Describe the bug
promptflow-evals version: 0.3.0

eval_result = evaluate(
    evaluation_name=f"{experiment_name}_{input_dataset.stem}",
    azure_ai_project=azure_ai_project,
    evaluators={
        "content_safety": ContentSafetyEvaluator(
            project_scope=azure_ai_project,
            credential=DefaultAzureCredential(),
            parallel=True,
        )
    },
    evaluator_config={
        "default": {
            "question": "${data.inputs.question}",
            "answer": "${data.outputs.copilot_answer}",
        }
    },
    data=str(output_dir / f"{experiment_name}.jsonl"),
)

When running the code above, the stream prompts the following error:

2024-06-19 11:33:46 +0800   28860 execution.flow     INFO     Current thread is not main thread, skip signal handler registration in AsyncNodesScheduler.
2024-06-19 11:33:46 +0800   28860 execution.flow     WARNING  Input 'project_scope' of evaluate_with_rai_service is not json serializable, use str to store it.
2024-06-19 11:33:46 +0800   28860 execution.flow     WARNING  Input 'credential' of evaluate_with_rai_service is not json serializable, use str to store it.
2024-06-19 11:33:46 +0800   28860 execution.flow     INFO     Start executing nodes in thread pool mode.
2024-06-19 11:33:46 +0800   28860 execution.flow     INFO     Current thread is not main thread, skip signal handler registration in AsyncNodesScheduler.
2024-06-19 11:33:46 +0800   28860 execution.flow     WARNING  Failed to serialize inputs or output for flow run because of cannot pickle '_thread.RLock' object.The inputs and output field in api_calls will be None.
2024-06-19 11:33:46 +0800   28860 execution          WARNING  [evaluate_with_rai_service in line None (index starts from 0)] stderr> AzureCliCredential.get_token failed: Failed to invoke the Azure CLI
2024-06-19 11:33:46 +0800   28860 execution.flow     INFO     Executing node validate_inputs. node run id: b185f8e5-f22f-4c44-8816-123f5c966134_validate_inputs_afb5ee9d-f2b6-4231-b880-046f1ed40835
2024-06-19 11:33:46 +0800   28860 execution.flow     INFO     Current thread is not main thread, skip signal handler registration in AsyncNodesScheduler.
2024-06-19 11:33:46 +0800   28860 execution          ERROR    Node evaluate_with_rai_service in line None failed. Exception: Execution failure in 'evaluate_with_rai_service': (CredentialUnavailableError) Failed to invoke the Azure CLI.
Traceback (most recent call last):
  File "c:\Users\xxxx\AppData\Local\anaconda3\envs\project\lib\site-packages\azure\identity\_credentials\azure_cli.py", line 199, in _run_command
    return subprocess.check_output(args, **kwargs)
  File "c:\Users\xxxx\AppData\Local\anaconda3\envs\project\lib\subprocess.py", line 424, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "c:\Users\xxxx\AppData\Local\anaconda3\envs\project\lib\subprocess.py", line 507, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "c:\Users\xxxx\AppData\Local\anaconda3\envs\project\lib\subprocess.py", line 1134, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "c:\Users\xxxx\AppData\Local\anaconda3\envs\project\lib\subprocess.py", line 1526, in _communicate
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['cmd', '/c', 'az account get-access-token --output json --resource https://management.azure.com']' timed out after 10 seconds

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "c:\Users\xxxx\AppData\Local\anaconda3\envs\project\lib\site-packages\promptflow\_core\flow_execution_context.py", line 182, in _invoke_tool_inner
    return f(**kwargs)
  File "c:\Users\xxxx\AppData\Local\anaconda3\envs\project\lib\site-packages\promptflow\tracing\_trace.py", line 556, in wrapped
    output = func(*args, **kwargs)
  File "C:\Users\xxxx\AppData\Local\anaconda3\envs\project\Lib\site-packages\promptflow\evals\evaluators\_content_safety\flow\evaluate_with_rai_service.py", line 189, in evaluate_with_rai_service
    rai_svc_url = get_rai_svc_url(project_scope, credential)
  File "C:\Users\xxxx\AppData\Local\anaconda3\envs\project\Lib\site-packages\promptflow\evals\evaluators\_content_safety\flow\evaluate_with_rai_service.py", line 165, in get_rai_svc_url
    discovery_url = _get_service_discovery_url(azure_ai_project=project_scope, credential=credential)
  File "C:\Users\xxxx\AppData\Local\anaconda3\envs\project\Lib\site-packages\promptflow\evals\evaluators\_content_safety\flow\evaluate_with_rai_service.py", line 148, in _get_service_discovery_url
    bearer_token = credential.get_token("https://management.azure.com/.default").token
  File "c:\Users\xxxx\AppData\Local\anaconda3\envs\project\lib\site-packages\azure\identity\_credentials\default.py", line 219, in get_token
    token = self._successful_credential.get_token(*scopes, claims=claims, tenant_id=tenant_id, **kwargs)
  File "c:\Users\xxxx\AppData\Local\anaconda3\envs\project\lib\site-packages\azure\identity\_internal\decorators.py", line 33, in wrapper
    token = fn(*args, **kwargs)
  File "c:\Users\xxxx\AppData\Local\anaconda3\envs\project\lib\site-packages\azure\identity\_credentials\azure_cli.py", line 112, in get_token
    output = _run_command(command, self._process_timeout)
  File "c:\Users\xxxx\AppData\Local\anaconda3\envs\project\lib\site-packages\azure\identity\_credentials\azure_cli.py", line 223, in _run_command
    raise error from ex
azure.identity._exceptions.CredentialUnavailableError: Failed to invoke the Azure CLI

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "c:\Users\xxxx\AppData\Local\anaconda3\envs\project\lib\site-packages\promptflow\_core\flow_execution_context.py", line 90, in invoke_tool
    result = self._invoke_tool_inner(node, f, kwargs)
  File "c:\Users\xxxx\AppData\Local\anaconda3\envs\project\lib\site-packages\promptflow\_core\flow_execution_context.py", line 206, in _invoke_tool_inner
    raise ToolExecutionError(node_name=node_name, module=module) from e
promptflow._core._errors.ToolExecutionError: Execution failure in 'evaluate_with_rai_service': (CredentialUnavailableError) Failed to invoke the Azure CLI

The bug will not be triggered if parallel parameter of content safety evaluator equals to False.

@Wix97 Wix97 added the bug Something isn't working label Jun 19, 2024
@Wix97 Wix97 changed the title [BUG] [promptflow-evals] Failed to invoke the Azure CLI when running [BUG] [promptflow-evals] Failed to invoke the Azure CLI when running evaluation Jun 19, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants