-
Notifications
You must be signed in to change notification settings - Fork 1.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Bug] dbt deps automatically recognizes projects in subdirectories #9719
Comments
Thanks for raising this issue @djbelknapaw ! Does this only happen on Windows when using a PowerShell or cmd.exe terminal? Or does it also happen when using WSL (Windows Subsystem for Linux)? Suspected root causeHere's what I think is happening: 👉 When dbt installs a local package, it uses a symlink if it can. Otherwise, it makes a copy of the entire directory. My understanding is that some Windows environments don't allow creation of a symlink, so dbt installs the package via the copy approach instead. Since the install location is a subdirectory of the package being installed, it exhibits recursive behavior you observed. Potential solutionOne approach we can consider when a symlink is not possible:
Here's the relevant source code: dbt-core/core/dbt/deps/local.py Lines 65 to 70 in 6fd0a94
|
Correct - wsl successfully sets up the symlink and doesn't error out. Also when I run powershell as admin it's able to create a symlink, so it's specific to the copytree code. It seems like you could just use the
|
💡 Great idea about the No pressure, but are you interested in opening a PR that includes your solution, by any chance? |
@dbeatty10 Open! First time contributing here, so let me know if there's anything to do differently. |
Is this a new bug in dbt-core?
Current Behavior
I'm attempting to build an integration_tests sub-project similar to dbt_utils, then install the parent project when running the integration tests. The file in
my_project/integration_tests/packages.yml
file is the same as in dbt_utils:This installs the parent project as a package, but that includes the integration_tests child project which dbt is recognizing as a project, and attempts to install its dependencies again, resulting in an endless call of dependencies.
I end up with a directory structure of:
Eventually deps fails with an error:
"[WinError 206] The filename or extension is too long: 'dbt_packages\\\\my_project\\\\integration_tests\\\\dbt_packages\\\\my_project\\\\integration_tests\\\\dbt_packages\\\\my_project\\\\integration_tests\\\\dbt_packages\\\\my_project\\\\integration_tests\\\\dbt_packages\\\\my_project\\\\integration_tests'"
Expected Behavior
When running dbt deps to a local project, only recognize the project.yml and packages.yml from the directly-referenced project and not sub-project directories. In this example, dbt should only look at
../packages.yml
and not be looking at../integration_tests/packages.yml
.Steps To Reproduce
dbt deps
Relevant log output
No response
Environment
Which database adapter are you using with dbt?
No response
Additional Context
No response
The text was updated successfully, but these errors were encountered: