{"payload":{"feedbackUrl":"https://github.com/orgs/community/discussions/53140","repo":{"id":99919302,"defaultBranch":"master","name":"doris","ownerLogin":"apache","currentUserCanPush":false,"isFork":false,"isEmpty":false,"createdAt":"2017-08-10T12:13:30.000Z","ownerAvatar":"https://avatars.githubusercontent.com/u/47359?v=4","public":true,"private":false,"isOrgOwned":true},"refInfo":{"name":"","listCacheKey":"v0:1719302428.0","currentOid":""},"activityList":{"items":[{"before":"a9855ca17387369849cb489e376b11b16c0757cc","after":"e25b0d7c378a3287d5bd5ca173d3448a8d09abab","ref":"refs/heads/branch-2.1","pushedAt":"2024-06-29T10:35:41.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"morningman","name":"Mingyu Chen","path":"/morningman","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/2899462?s=80&v=4"},"commit":{"message":"[branch-2.1][improvement](mysql catalog) disable mysql AbandonedConnectionCleanupThread (#36970)\n\npick (#36655)","shortMessageHtmlLink":"[branch-2.1][improvement](mysql catalog) disable mysql AbandonedConne…"}},{"before":"e35d433629d16ab9973552c6ded69412302a17ae","after":"288fef58c0944ea967e2d0a2b6d8ac5dcb524302","ref":"refs/heads/master","pushedAt":"2024-06-29T08:06:36.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"liaoxin01","name":"Xin Liao","path":"/liaoxin01","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/19986811?s=80&v=4"},"commit":{"message":"[performance](load) do not copy input_block in memtable (#36939)\n\nPass `column_offset` to `Block::clone_without_columns()` and\r\n`MutableBlock::add_rows()`,\r\nso we do not need to copy `input_block` in `MemTable::insert()`.\r\n\r\nTo optimize performance in cases with many (e.g. 12k+) tablets:\r\n\r\nBefore:\r\n\r\n```\r\n- DeltaWriterWriteTime: 1m58s\r\n - MemTableWriterFlushTime: 0ns\r\n - MemTableWriterLockTime: 1s41ms\r\n - MemTableWriterShrinkTime: 0ns\r\n - MemTableWriterWriteTime: 1m53s\r\n - MemTableCopyCount: 12.044003M (12044003)\r\n - MemTableCopyTime: 42s857ms\r\n - MemTableCopyTime0: 11s866ms\r\n - MemTableInitTime: 13.384ms\r\n - MemTableInsertTime: 51s543ms\r\n```\r\n\r\nAfter:\r\n\r\n```\r\n- DeltaWriterWriteTime: 1m\r\n - MemTableWriterFlushTime: 0ns\r\n - MemTableWriterLockTime: 971.481ms\r\n - MemTableWriterShrinkTime: 0ns\r\n - MemTableWriterWriteTime: 55s274ms\r\n - MemTableCopyCount: 0\r\n - MemTableCopyTime: 0ns\r\n - MemTableCopyTime0: 0ns\r\n - MemTableInitTime: 18.746ms\r\n - MemTableInsertTime: 53s772ms\r\n```","shortMessageHtmlLink":"[performance](load) do not copy input_block in memtable (#36939)"}},{"before":"96178236866daa8358e06349edb366cb1cb5dca7","after":"e35d433629d16ab9973552c6ded69412302a17ae","ref":"refs/heads/master","pushedAt":"2024-06-29T01:25:41.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"gavinchou","name":"Gavin Chou","path":"/gavinchou","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6919662?s=80&v=4"},"commit":{"message":"(cloud-merge) Get fileCacheSize by RPC always (#36857)\n\nGet fileCacheSize every time when do warm up job.","shortMessageHtmlLink":"(cloud-merge) Get fileCacheSize by RPC always (#36857)"}},{"before":"d0c0a7b9ae4d4f7efff5e0fc64c3b26004d3b51f","after":"a9855ca17387369849cb489e376b11b16c0757cc","ref":"refs/heads/branch-2.1","pushedAt":"2024-06-29T01:17:38.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"morningman","name":"Mingyu Chen","path":"/morningman","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/2899462?s=80&v=4"},"commit":{"message":"[Improvement](set) enable admin_set_frontend_config can apply to all fe (#37022)\n\nbp #34685","shortMessageHtmlLink":"[Improvement](set) enable admin_set_frontend_config can apply to all …"}},{"before":"aeb89db2a5717dd2faf0bc07aaec5cd9a1668f7b","after":"96178236866daa8358e06349edb366cb1cb5dca7","ref":"refs/heads/master","pushedAt":"2024-06-28T18:13:28.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"gavinchou","name":"Gavin Chou","path":"/gavinchou","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6919662?s=80&v=4"},"commit":{"message":"Revert \"[feature](cloud) support file cache only cache index pages (#36273) (#36797)\n\nThis reverts commit f5c40a5c3bac3091707c8324c422889fa7f030c8.\r\nThis is a experimental (and buggy) commit. I hava found that caching\r\nindex only helps little. With that be the result, I think it is a good\r\ntime to\r\nrevert it.","shortMessageHtmlLink":"Revert \"[feature](cloud) support file cache only cache index pages (#…"}},{"before":"10596044c8481e94250c830268686bf7fdfa3322","after":"aeb89db2a5717dd2faf0bc07aaec5cd9a1668f7b","ref":"refs/heads/master","pushedAt":"2024-06-28T13:56:43.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"morningman","name":"Mingyu Chen","path":"/morningman","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/2899462?s=80&v=4"},"commit":{"message":"[bugfix](testcase)add java error log output (#35998)\n\nIn the testcase pipeline, this spark-connector case has a 4% chance of\r\nerror, but since there is no error log, the cause of the error cannot be\r\nlocated.\r\nTherefore, an error log is added to facilitate problem location later.","shortMessageHtmlLink":"[bugfix](testcase)add java error log output (#35998)"}},{"before":"620653f1d4d143b3e862d2810bb27a1e9105ff2f","after":"10596044c8481e94250c830268686bf7fdfa3322","ref":"refs/heads/master","pushedAt":"2024-06-28T13:12:21.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"dataroaring","name":"Yongqiang YANG","path":"/dataroaring","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/98214048?s=80&v=4"},"commit":{"message":"[Fix](autoinc) try fix concurrent load problem with auto inc column (#36421)\n\n1. increase the value of `AutoIncrementGenerator.BATCH_ID_INTERVAL` to\r\nreduce the number of writes to BDBJE. (the default value of\r\n`config::auto_inc_prefetch_size_ratio` is 10 and the default value of\r\n`AutoIncIDBuffer::_batch_size` is 4064, so mostly the request length is\r\n40960)\r\n2. only allow master fe to offer `getAutoIncrementRange` service\r\n3. write editlog before update `batchEndId` in memory in\r\n`getAutoIncrementRange `\r\n4. refactor `AutoIncIDBuffer`","shortMessageHtmlLink":"[Fix](autoinc) try fix concurrent load problem with auto inc column (#…"}},{"before":"9229dcd62ab91bf9a8939c6f58d5851cdfde0ead","after":"620653f1d4d143b3e862d2810bb27a1e9105ff2f","ref":"refs/heads/master","pushedAt":"2024-06-28T13:09:36.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"dataroaring","name":"Yongqiang YANG","path":"/dataroaring","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/98214048?s=80&v=4"},"commit":{"message":"[fix](mtmv)fix mtmv dead lock (#37009)\n\nCause of occurrence：\r\n- dropBaseTable: Holding db's writeLock ,notify mtmv alter status to\r\nschema_change,need mv's writeLock\r\n- task(insert overwrite):Holding mv's readLock,when generage plan need\r\ndb's readLock\r\n\r\nfix:\r\n- mtmv alter status to schema_change need mv's writeMvLock instead of\r\nmv's writeLock","shortMessageHtmlLink":"[fix](mtmv)fix mtmv dead lock (#37009)"}},{"before":"54592d048a0c3e0eff698d0c8a062f9e382063db","after":"f95221e34e167e5149dda9dfcc7ae5962b1582c0","ref":"refs/heads/branch-2.0","pushedAt":"2024-06-28T13:07:08.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"dataroaring","name":"Yongqiang YANG","path":"/dataroaring","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/98214048?s=80&v=4"},"commit":{"message":"[fix](oom) avoid oom when a lot of tablets fail on load (#36962)\n\nWhen a lot of tablets fail when loading, then detailed information would\r\ncause oom.\r\n\r\n## Proposed changes\r\n\r\nIssue Number: close #xxx\r\n\r\n","shortMessageHtmlLink":"[fix](oom) avoid oom when a lot of tablets fail on load (#36962)"}},{"before":"bafbb944a9afa714958297dbd24440ccfdf7dfc0","after":"9229dcd62ab91bf9a8939c6f58d5851cdfde0ead","ref":"refs/heads/master","pushedAt":"2024-06-28T12:51:52.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"Jibing-Li","name":null,"path":"/Jibing-Li","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/64681310?s=80&v=4"},"commit":{"message":"[fix](statistics)Escape fetch partition stats sql. (#36941)\n\nSql to fetch partition stats need to be escaped.","shortMessageHtmlLink":"[fix](statistics)Escape fetch partition stats sql. (#36941)"}},{"before":"584ef7d36b1bc44c88a11aa44ce0eb9373fd2689","after":"bafbb944a9afa714958297dbd24440ccfdf7dfc0","ref":"refs/heads/master","pushedAt":"2024-06-28T12:13:35.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"zy-kkk","name":null,"path":"/zy-kkk","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/70795751?s=80&v=4"},"commit":{"message":"[improvement](jdbc catalog) Modify the maximum number of connections in the connection pool to 30 by default (#36720)\n\nIn many cases, we found that users would use JDBC Catalog to perform a\r\nlarge number of queries, which resulted in the maximum of 10 connections\r\nbeing insufficient, so I adjusted it to 30, which covered most needs.","shortMessageHtmlLink":"[improvement](jdbc catalog) Modify the maximum number of connections …"}},{"before":"70b816b3060df8617458c1f7068ab3a2beaaa4d9","after":"584ef7d36b1bc44c88a11aa44ce0eb9373fd2689","ref":"refs/heads/master","pushedAt":"2024-06-28T11:22:06.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"xiaokang","name":"Kang","path":"/xiaokang","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/680838?s=80&v=4"},"commit":{"message":"[fix](regression test) fix unstable single compaction test p2 (#36881)","shortMessageHtmlLink":"[fix](regression test) fix unstable single compaction test p2 (#36881)"}},{"before":"e348d797d0050c88760d2d8fa5c84c9dbc896820","after":"70b816b3060df8617458c1f7068ab3a2beaaa4d9","ref":"refs/heads/master","pushedAt":"2024-06-28T11:05:58.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"gavinchou","name":"Gavin Chou","path":"/gavinchou","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6919662?s=80&v=4"},"commit":{"message":"[enhancement](cloud) batching get visible version from MetaService (#34615)\n\nGet visible versions one by one from MetaService is costly. Batching\r\nthem into one RPC will not only reduce the workload of RPC service but\r\nalso reduce the lag.","shortMessageHtmlLink":"[enhancement](cloud) batching get visible version from MetaService (#…"}},{"before":"8902df51ced892e2d08ddfc355e18242320a0339","after":"e348d797d0050c88760d2d8fa5c84c9dbc896820","ref":"refs/heads/master","pushedAt":"2024-06-28T10:56:32.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"morrySnow","name":null,"path":"/morrySnow","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/101034200?s=80&v=4"},"commit":{"message":"[chore](upgrade) turn off fallback_to_original_planner when upgrade (#37005)\n\nupgrade from 2.1 or ealier version to 3.0.x will\r\n\r\n- enable_nereids_planner to true\r\n- enable_nereids_dml to true\r\n- enable_fallback_to_original_planner to false","shortMessageHtmlLink":"[chore](upgrade) turn off fallback_to_original_planner when upgrade (#…"}},{"before":"12e53653eae515afa5a2810ae3f8bfb089e27e1e","after":"54592d048a0c3e0eff698d0c8a062f9e382063db","ref":"refs/heads/branch-2.0","pushedAt":"2024-06-28T10:52:43.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"lide-reed","name":"lide","path":"/lide-reed","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/2121393?s=80&v=4"},"commit":{"message":"[fix](multicatalog) fix npe issue when alter property for a non-exist catalog (#36951)","shortMessageHtmlLink":"[fix](multicatalog) fix npe issue when alter property for a non-exist…"}},{"before":"b478800da7f6b0330433ca58f3bc9f6b7f095888","after":"8902df51ced892e2d08ddfc355e18242320a0339","ref":"refs/heads/master","pushedAt":"2024-06-28T10:44:07.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"morningman","name":"Mingyu Chen","path":"/morningman","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/2899462?s=80&v=4"},"commit":{"message":"[chore](profile) rm useless code of profile (#36915)","shortMessageHtmlLink":"[chore](profile) rm useless code of profile (#36915)"}},{"before":"7c6d280def40459a9d20a2fbdcd2d66acb3263e2","after":"d0c0a7b9ae4d4f7efff5e0fc64c3b26004d3b51f","ref":"refs/heads/branch-2.1","pushedAt":"2024-06-28T10:41:40.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"morningman","name":"Mingyu Chen","path":"/morningman","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/2899462?s=80&v=4"},"commit":{"message":"[Fix](variant) ignore serialization of nothing type (#37006)\n\npicked from #36997","shortMessageHtmlLink":"[Fix](variant) ignore serialization of nothing type (#37006)"}},{"before":"cb80ae906f2a0e7e4a57b77ba645a81e238b3fcf","after":"7c6d280def40459a9d20a2fbdcd2d66acb3263e2","ref":"refs/heads/branch-2.1","pushedAt":"2024-06-28T10:38:33.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"morningman","name":"Mingyu Chen","path":"/morningman","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/2899462?s=80&v=4"},"commit":{"message":"[Test](manager_interface)append manager interface test. (#35889) (#36912)\n\nbp #35889\r\n\r\n---------\r\n\r\nCo-authored-by: daidai <2017501503@qq.com>","shortMessageHtmlLink":"[Test](manager_interface)append manager interface test. (#35889) (#36912"}},{"before":"a2ee53ad76ff651d8059b954f4fde48eaf52b9f6","after":"b478800da7f6b0330433ca58f3bc9f6b7f095888","ref":"refs/heads/master","pushedAt":"2024-06-28T10:23:36.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"eldenmoon","name":"lihangyu","path":"/eldenmoon","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/64513324?s=80&v=4"},"commit":{"message":"[Fix](variant) ignore serialization of nothing type (#36997)\n\n1. fix variant should not serialize nothing type\r\n2. fix unstable cases","shortMessageHtmlLink":"[Fix](variant) ignore serialization of nothing type (#36997)"}},{"before":"c127ed1f6588a2c22a0a78ece06a1087116f9bec","after":"a2ee53ad76ff651d8059b954f4fde48eaf52b9f6","ref":"refs/heads/master","pushedAt":"2024-06-28T09:41:20.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"morningman","name":"Mingyu Chen","path":"/morningman","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/2899462?s=80&v=4"},"commit":{"message":"[Fix](multi-catalog) Fix the transaction is not removed in abnormal situations by removing transaction in finally block. (#36705)\n\n[Fix] (multi-catalog) Fix the transaction is not removed in abnormal\r\nsituations by removing transaction in `finally` block.","shortMessageHtmlLink":"[Fix](multi-catalog) Fix the transaction is not removed in abnormal s…"}},{"before":"f4a6434da7a8db84692d922e4773d903077e0113","after":"c127ed1f6588a2c22a0a78ece06a1087116f9bec","ref":"refs/heads/master","pushedAt":"2024-06-28T09:41:02.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"morningman","name":"Mingyu Chen","path":"/morningman","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/2899462?s=80&v=4"},"commit":{"message":"[Fix](hive-writer) Fixed the issue where `uncompletedMpuPendingUploads` did not remove objects correctly. (#36905)\n\n[Fix](hive-writer) Fixed the issue where `uncompletedMpuPendingUploads`\r\ndid not remove objects correctly.","shortMessageHtmlLink":"[Fix](hive-writer) Fixed the issue where `uncompletedMpuPendingUpload…"}},{"before":"1789ec61c4046523bc81715cddb15e9d0033d0e1","after":"f4a6434da7a8db84692d922e4773d903077e0113","ref":"refs/heads/master","pushedAt":"2024-06-28T09:27:44.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"morningman","name":"Mingyu Chen","path":"/morningman","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/2899462?s=80&v=4"},"commit":{"message":"[feature](insert)support external hive truncate table DDL (#36801)\n\n## Proposed changes\r\n\r\nIssue Number: #31442\r\n\r\n1. support hive3 'truncate table'\r\n2. forbiden hive2 'truncate table' because of it is not supported by HMS\r\nClient API on hive2","shortMessageHtmlLink":"[feature](insert)support external hive truncate table DDL (#36801)"}},{"before":"4e142a303d4fcaba0ae5ea5489778f59b3cb1848","after":"1789ec61c4046523bc81715cddb15e9d0033d0e1","ref":"refs/heads/master","pushedAt":"2024-06-28T09:11:24.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"924060929","name":null,"path":"/924060929","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/8806055?s=80&v=4"},"commit":{"message":"[refactor](nereids) New distribute planner (#36531)\n\n## Proposed changes\r\nThe legacy coordinator act not only scheduler but also distribute planner. The code is so complex to understand, and hard to extend, and exist many limitations.\r\n\r\nThis pr extract and refine the computation of degree of parallel(dop) to a new DistributePlanner and resolve the limitations.\r\n\r\n\r\n## How to use this function\r\nThis function only use for nereids + pipelinex, and current only support query statement, and non cloud mode.\r\nOpen this session variables to use this function:\r\n```sql\r\nset enable_nereids_distribute_planner=true; -- default is false\r\nset enable_nereids_planner=true; -- default is true\r\n```\r\n\r\n## Core process and concepts\r\n```\r\n \r\n ┌──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐ \r\n │ │ \r\n │ ┌──────────────┐ ┌───────────────┐ ┌───────────────────┐ ┌─────────────────────────┐ │ \r\n │ Translate │ │ Typed │ │ Assign │ │ Wrap │ │ │ \r\n │ ──────────► │ PlanFragment │ ──────► │ UnassignedJob │ ───────► │ StaticAssignedJob │ ─────► │ PipelineDistributedPlan │ │ \r\n │ │ │ │ │ │ │ │ │ │ \r\n │ └──────────────┘ └───────────────┘ └───────────────────┘ └─────────────────────────┘ │ \r\n │ │ \r\n └──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘ \r\n \r\n │ │ \r\n │ │ \r\n └──────────────────────────────────────────────────┐ ┌─────────────────────────────────────────────────────┘ \r\n │ │ \r\n │ │ \r\n \r\n ┌──────────────┐ ┌─────────────────┐ ┌───────────────────┐ \r\n │ │ Distribute │ │ AdHoc │ │ \r\n │ PhysicalPlan │ ───────────► │ DistributedPlan │ ──────► │ PipelineScheduler │ \r\n │ │ │ │ │ │ \r\n └──────────────┘ └─────────────────┘ └───────────────────┘ \r\n \r\n```\r\n\r\nDistributePlanner is a new planner to compute dop and generate instances, it consume PlanFragment and do this tasks\r\n1. Use PlanFragment to generate `UnassignedJob`, it's a **Typed Fragment**, decided how to calculate dop and how to select the datasource, but this fragment not yet assigned some backends and datasources. These are some unassignedJobs: UnassignedScanSingleOlapTableJob, UnassignedScanBucketOlapTableJob, UnassignedShuffleJob, UnassignedQueryConstantJob. Keep UnassignedJob different can decoupling unrelated logic, and easy to extend: just and a new type of UnassignedJob.\r\n2. Use UnassignedJob to select datasource, compute dop, and generate `AssignedJob`, means a instance, which already assigned datasource and backend. There are StaticAssignedJob and LocalShuffleAssignedJob, we will add DynamicAssignedJob when support StageScheduler and adaptive query execution\r\n3. Wrap PlanFragment, UnassignedJob and AssignedJob to `PipelineDistributedPlan`, the coordinator will consume the DistributedPlan and translate to TPlan and schedule instances\r\n\r\n\r\n## Resolve limitations\r\n**1. left table shuffle to right table**\r\nif right table has distribution which distribute by `storage hash`, and left table has distribution which distribute by `compute hash`, we can shuffle left to right by `storage hash` to do shuffle bucket join, and keep right side not move.\r\n\r\n```sql\r\nselect *\r\nfrom\r\n(\r\n select id2\r\n from test_shuffle_left\r\n group by id2\r\n) a\r\ninner join [shuffle]\r\ntest_shuffle_left b\r\non a.id2=b.id;\r\n\r\n| PhysicalResultSink[288] ( outputExprs=[id2#1, id#2, id2#3] ) ...\r\n| +--PhysicalHashJoin[285]@4 ( type=INNER_JOIN, stats=3, hashCondition=[(id2#1 = id#2)], otherCondition=[], markCondition=[], hint=[shuffle] ) ...\r\n| |--PhysicalDistribute[281]@2 ( stats=1.5, distributionSpec=DistributionSpecHash ( orderedShuffledColumns=[1], shuffleType=STORAGE_BUCKETED, tableId=-1, selectedIndexId=-1, partitionIds=...\r\n| | +--PhysicalHashAggregate[278]@2 ( aggPhase=GLOBAL, aggMode=BUFFER_TO_RESULT, maybeUseStreaming=false, groupByExpr=[id2#1], outputExpr=[id2#1], partitionExpr=Optional[[id2#1]], requir...\r\n| | +--PhysicalDistribute[275]@7 ( stats=1.5, distributionSpec=DistributionSpecHash ( orderedShuffledColumns=[1], shuffleType=EXECUTION_BUCKETED, tableId=-1, selectedIndexId=-1, parti...\r\n| | +--PhysicalHashAggregate[272]@7 ( aggPhase=LOCAL, aggMode=INPUT_TO_BUFFER, maybeUseStreaming=true, groupByExpr=[id2#1], outputExpr=[id2#1], partitionExpr=Optional[[id2#1]], req...\r\n| | +--PhysicalProject[269]@1 ( stats=3, projects=[id2#1] ) ...\r\n| | +--PhysicalOlapScan[test_shuffle_left]@0 ( stats=3 ) ...\r\n| +--PhysicalOlapScan[test_shuffle_left]@3 ( stats=3 )\r\n```\r\n\r\n**2. support colocate union numbers function**\r\nsupport use one instance to union/join numbers, note this plan no any PhysicalDistribute plan:\r\n```sql\r\nexplain physical plan\r\nselect * from numbers('number'='3')a\r\nunion all\r\nselect * from numbers('number'='4')b\r\n\r\nPhysicalResultSink[98] ( outputExprs=[number#2] )\r\n+--PhysicalUnion@ ( qualifier=ALL, outputs=[number#2], regularChildrenOutputs=[[number#0], [number#1]], constantExprsList=[], stats=7 )\r\n |--PhysicalTVFRelation ( qualified=NumbersTableValuedFunction, output=[number#0], function=numbers('number' = '3') )\r\n +--PhysicalTVFRelation ( qualified=NumbersTableValuedFunction, output=[number#1], function=numbers('number' = '4') )\r\n```\r\n\r\n**3. support bucket prune with right outer bucket shuffle join**\r\nleft table prune some buckets, say [bucket 1, bucket 3]\r\n\r\nwe should process the right bucket shuffle join like this\r\n```\r\n[\r\n (left bucket 1) right outer join (exchange right table which should process by bucket 1),\r\n (empty bucket) right outer join (exchange right table which should process by bucket 2),\r\n (left bucket 3) right outer join (exchange right table which should process by bucket 3)\r\n]\r\n```\r\nthe left bucket 2 is pruned, so right table can not shuffle to left, because the left instance not exists, so bucket 2 will return empty rows and wrong.\r\n\r\nnew DistributePlanner can fill up this instance.\r\n\r\nthe case:\r\n```sql\r\nexplain physical plan\r\nSELECT * FROM\r\n(select * from test_outer_join1 where c0 =1)a\r\nRIGHT OUTER JOIN\r\n(select * from test_outer_join2)b\r\nON a.c0 = b.c0\r\n```\r\n\r\n### New feature\r\n\r\nadd an explain statement to show distributed plans\r\n```sql\r\nexplain distributed plan select ...\r\n```\r\n\r\nfor example, you can use this function to check how many instances generated, how many bytes the instance will scan, which backend will process the instance:\r\n```sql\r\nMySQL root@127.0.0.1:test> explain distributed plan select * from test_shuffle_left2 a join [shuffle] test_shuffle_left2 b on a.id2=b.id;\r\nExplain String(Nereids Planner)\r\n-------------------------------------------------------------------------------------------------------\r\nPipelineDistributedPlan(\r\n id: 0,\r\n parallel: 2,\r\n fragmentJob: UnassignedScanSingleOlapTableJob,\r\n fragment: {\r\n OUTPUT EXPRS:\r\n id[#8]\r\n id2[#9]\r\n id[#10]\r\n id2[#11]\r\n PARTITION: HASH_PARTITIONED: id2[#3]\r\n\r\n HAS_COLO_PLAN_NODE: false\r\n\r\n VRESULT SINK\r\n MYSQL_PROTOCAL\r\n\r\n 3:VHASH JOIN(152)\r\n | join op: INNER JOIN(PARTITIONED)[]\r\n | equal join conjunct: (id2[#3] = id[#0])\r\n | cardinality=3\r\n | vec output tuple id: 3\r\n | output tuple id: 3\r\n | vIntermediate tuple ids: 2\r\n | hash output slot ids: 0 1 2 3\r\n | isMarkJoin: false\r\n | final projections: id[#4], id2[#5], id[#6], id2[#7]\r\n | final project output tuple id: 3\r\n | distribute expr lists: id2[#3]\r\n | distribute expr lists: id[#0]\r\n | tuple ids: 1 0\r\n |\r\n |----0:VOlapScanNode(149)\r\n | TABLE: test.test_shuffle_left2(test_shuffle_left2), PREAGGREGATION: ON\r\n | partitions=1/1 (test_shuffle_left2)\r\n | tablets=10/10, tabletList=22038,22040,22042 ...\r\n | cardinality=3, avgRowSize=0.0, numNodes=1\r\n | pushAggOp=NONE\r\n | tuple ids: 0\r\n |\r\n 2:VEXCHANGE\r\n offset: 0\r\n distribute expr lists: id[#2]\r\n tuple ids: 1\r\n },\r\n instanceJobs: [\r\n LocalShuffleAssignedJob(\r\n index: 0,\r\n worker: BackendWorker(id: 10095, address: 192.168.126.1:9050),\r\n shareScanIndex: 0,\r\n scanSource: [\r\n {\r\n scanNode: OlapScanNode{id=0, tid=0, tblName=test_shuffle_left2, keyRanges=, preds= limit=-1},\r\n scanRanges: ScanRanges(bytes: 400, ranges: [\r\n tablet 22038, bytes: 0,\r\n tablet 22042, bytes: 0,\r\n tablet 22046, bytes: 0,\r\n tablet 22050, bytes: 400,\r\n tablet 22054, bytes: 0\r\n ])\r\n }\r\n ]\r\n ),\r\n LocalShuffleAssignedJob(\r\n index: 1,\r\n worker: BackendWorker(id: 10096, address: 192.168.126.2:9051),\r\n shareScanIndex: 1,\r\n scanSource: [\r\n {\r\n scanNode: OlapScanNode{id=0, tid=0, tblName=test_shuffle_left2, keyRanges=, preds= limit=-1},\r\n scanRanges: ScanRanges(bytes: 796, ranges: [\r\n tablet 22040, bytes: 397,\r\n tablet 22044, bytes: 0,\r\n tablet 22048, bytes: 399,\r\n tablet 22052, bytes: 0,\r\n tablet 22056, bytes: 0\r\n ])\r\n }\r\n ]\r\n )\r\n ]\r\n)\r\nPipelineDistributedPlan(\r\n id: 1,\r\n parallel: 2,\r\n fragmentJob: UnassignedScanSingleOlapTableJob,\r\n fragment: {\r\n PARTITION: HASH_PARTITIONED: id[#2]\r\n\r\n HAS_COLO_PLAN_NODE: false\r\n\r\n STREAM DATA SINK\r\n EXCHANGE ID: 02\r\n HASH_PARTITIONED: id2[#3]\r\n\r\n 1:VOlapScanNode(145)\r\n TABLE: test.test_shuffle_left2(test_shuffle_left2), PREAGGREGATION: ON\r\n partitions=1/1 (test_shuffle_left2)\r\n tablets=10/10, tabletList=22038,22040,22042 ...\r\n cardinality=3, avgRowSize=0.0, numNodes=1\r\n pushAggOp=NONE\r\n tuple ids: 1\r\n },\r\n instanceJobs: [\r\n LocalShuffleAssignedJob(\r\n index: 0,\r\n worker: BackendWorker(id: 10095, address: 192.168.126.1:9050),\r\n shareScanIndex: 0,\r\n scanSource: [\r\n {\r\n scanNode: OlapScanNode{id=1, tid=1, tblName=test_shuffle_left2, keyRanges=, preds= limit=-1},\r\n scanRanges: ScanRanges(bytes: 400, ranges: [\r\n tablet 22038, bytes: 0,\r\n tablet 22042, bytes: 0,\r\n tablet 22046, bytes: 0,\r\n tablet 22050, bytes: 400,\r\n tablet 22054, bytes: 0\r\n ])\r\n }\r\n ]\r\n ),\r\n LocalShuffleAssignedJob(\r\n index: 1,\r\n worker: BackendWorker(id: 10096, address: 192.168.126.2:9051),\r\n shareScanIndex: 1,\r\n scanSource: [\r\n {\r\n scanNode: OlapScanNode{id=1, tid=1, tblName=test_shuffle_left2, keyRanges=, preds= limit=-1},\r\n scanRanges: ScanRanges(bytes: 796, ranges: [\r\n tablet 22040, bytes: 397,\r\n tablet 22044, bytes: 0,\r\n tablet 22048, bytes: 399,\r\n tablet 22052, bytes: 0,\r\n tablet 22056, bytes: 0\r\n ])\r\n }\r\n ]\r\n )\r\n ]\r\n)\r\n\r\n\r\n\r\nHint log:\r\nUsed: [shuffle]_2\r\nUnUsed:\r\nSyntaxError:\r\n```\r\n\r\n## TODO\r\n1. extract PipelineScheduler from Coordinator\r\n2. move this framework into cascades and compute cost by dop\r\n3. support StageScheduler, adaptive query execution and DynamicAssignedJob","shortMessageHtmlLink":"[refactor](nereids) New distribute planner (#36531)"}},{"before":"5765ffac43098960d7136cb0b351383e850b07a7","after":"4e142a303d4fcaba0ae5ea5489778f59b3cb1848","ref":"refs/heads/master","pushedAt":"2024-06-28T09:08:25.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"dataroaring","name":"Yongqiang YANG","path":"/dataroaring","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/98214048?s=80&v=4"},"commit":{"message":"[Refactor](meta) Delete useless classes and write methods (#36894)\n\n\r\nCo-authored-by: zhangpeicheng ","shortMessageHtmlLink":"[Refactor](meta) Delete useless classes and write methods (#36894)"}},{"before":"4dcceaefea24d24ba5d30bf30e7365f8a5c32fbb","after":"cb80ae906f2a0e7e4a57b77ba645a81e238b3fcf","ref":"refs/heads/branch-2.1","pushedAt":"2024-06-28T08:59:26.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"HappenLee","name":"HappenLee","path":"/HappenLee","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/10553413?s=80&v=4"},"commit":{"message":"[Bug](runtime-filter) disable sync filter when pipeline engine is off (#36994)\n\n## Proposed changes\r\n1. disable sync filter when pipeline engine is off\r\n2. reduce some warning log","shortMessageHtmlLink":"[Bug](runtime-filter) disable sync filter when pipeline engine is off (…"}},{"before":"6b3cb56c35d3cfd270c01003b74c1e7493473727","after":"12e53653eae515afa5a2810ae3f8bfb089e27e1e","ref":"refs/heads/branch-2.0","pushedAt":"2024-06-28T08:58:21.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"xiaokang","name":"Kang","path":"/xiaokang","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/680838?s=80&v=4"},"commit":{"message":"[test](ES Catalog) Add test cases for ES 5.x (#34441) (#36996)","shortMessageHtmlLink":"[test](ES Catalog) Add test cases for ES 5.x (#34441) (#36996)"}},{"before":"4bec08c7e89cfa185f86c6d17f3f2ad7541fb274","after":"4dcceaefea24d24ba5d30bf30e7365f8a5c32fbb","ref":"refs/heads/branch-2.1","pushedAt":"2024-06-28T08:58:07.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"xiaokang","name":"Kang","path":"/xiaokang","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/680838?s=80&v=4"},"commit":{"message":"[test](ES Catalog) Add test cases for ES 5.x (#34441) (#36993)\n\nbackport #34441","shortMessageHtmlLink":"[test](ES Catalog) Add test cases for ES 5.x (#34441) (#36993)"}},{"before":"77765b23b9967924b25f68851916332b9e0d1e28","after":"6b3cb56c35d3cfd270c01003b74c1e7493473727","ref":"refs/heads/branch-2.0","pushedAt":"2024-06-28T08:57:23.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"xiaokang","name":"Kang","path":"/xiaokang","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/680838?s=80&v=4"},"commit":{"message":"[fix](inverted index)Make build index operation only affect base index (#36869) (#36991)","shortMessageHtmlLink":"[fix](inverted index)Make build index operation only affect base index ("}},{"before":"72de9c1b8fa930043d2dfb50d293447fac85a40c","after":"4bec08c7e89cfa185f86c6d17f3f2ad7541fb274","ref":"refs/heads/branch-2.1","pushedAt":"2024-06-28T08:57:07.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"xiaokang","name":"Kang","path":"/xiaokang","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/680838?s=80&v=4"},"commit":{"message":"[fix](inverted index)Make build index operation only affect base index (#36869) (#36988)\n\nbackport #36869","shortMessageHtmlLink":"[fix](inverted index)Make build index operation only affect base index ("}},{"before":"b46c86f352b1dc9b0e8cd607be616722c7f642fd","after":"72de9c1b8fa930043d2dfb50d293447fac85a40c","ref":"refs/heads/branch-2.1","pushedAt":"2024-06-28T08:54:49.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"dataroaring","name":"Yongqiang YANG","path":"/dataroaring","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/98214048?s=80&v=4"},"commit":{"message":"[fix](autoinc) avoid duplicated auto inc when role of fe changes (#36961)\n\n## Proposed changes\r\n\r\npick #36960 \r\n\r\nIssue Number: close #xxx\r\n\r\n","shortMessageHtmlLink":"[fix](autoinc) avoid duplicated auto inc when role of fe changes (#36961"}}],"hasNextPage":true,"hasPreviousPage":false,"activityType":"all","actor":null,"timePeriod":"all","sort":"DESC","perPage":30,"cursor":"djE6ks8AAAAEcoCKVQA","startCursor":null,"endCursor":null}},"title":"Activity · apache/doris"}