[Perf] Evaluate API: Using threads to infer signatures for eval batch runs instead of processes. #3412
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ninghu
added a commit
that referenced
this pull request
Jun 17, 2024
… pf.run (#3380) # Description The change in this PR addresses the performance issues we were seeing with the Evaluate API. This is the first step in optimizing performance. The improvements include: - Parallelizing the pf.run for evaluators. Previously, it ran sequentially, contributing to most of the latency. - Addressing the slowness of the import Evaluate API, which was due to the import of MLClient. - Using threads to infer signatures for eval batch runs instead of processes. (This change has been moved to a seperated PR: #3412) **Improvements from this change:** Windows OS, remote tracking disabled - 4 evaluators, 100 rows: - Previous (pf.run without threadpool): 320 secs - Current (pf.run with threadpool): 78 secs (~75% improvement) - 1 evaluator, 1 row: - Previous: 53 secs - Current: 17 secs (~68% improvement) Investigation details can be found [here](https://microsoft-my.sharepoint.com/:w:/p/ninhu/ETB_zdMkFrdAuf3Lcg9ssrUB6RVmyuFs5Un1G74O1HlwSA?e=23ngPm) # All Promptflow Contribution checklist: - [ ] **The pull request does not introduce [breaking changes].** - [ ] **CHANGELOG is updated for new features, bug fixes or other significant changes.** - [ ] **I have read the [contribution guidelines](../CONTRIBUTING.md).** - [ ] **Create an issue and link to the pull request to get dedicated review from promptflow team. Learn more: [suggested workflow](../CONTRIBUTING.md#suggested-workflow).** ## General Guidelines and Best Practices - [ ] Title of the pull request is clear and informative. - [ ] There are a small number of commits, each of which have an informative message. This means that previously merged commits do not appear in the history of the PR. For more information on cleaning up the commits in your PR, [see this page](https://github.com/Azure/azure-powershell/blob/master/documentation/development-docs/cleaning-up-commits.md). ### Testing Guidelines - [ ] Pull request includes test coverage for the included changes. --------- Co-authored-by: Clement Wang <[email protected]>
src/promptflow-devkit/promptflow/_proxy/_python_inspector_proxy.py
Outdated
Show resolved
Hide resolved
wangchao1230
approved these changes
Jun 18, 2024
nick863
approved these changes
Jun 18, 2024
singankit
reviewed
Jun 18, 2024
|
|
||
| def __exit__(self, exc_type, exc_val, exc_tb): | ||
| if isinstance(self.client, CodeClient): | ||
| recover_openai_api() | ||
|
|
||
| if isinstance(self.client, ProxyClient): | ||
| os.environ.pop(PF_FLOW_ENTRY_IN_TMP, None) | ||
| os.environ.pop(PF_FLOW_META_LOAD_IN_SUBPROCESS, None) |
There was a problem hiding this comment.
Should we restore it to it original value ?
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
This change is related to #3380
All Promptflow Contribution checklist:
General Guidelines and Best Practices
Testing Guidelines