Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Try to reuse existing chunks #3983

Merged
merged 1 commit into from
Dec 12, 2024
Merged

Try to reuse existing chunks #3983

merged 1 commit into from
Dec 12, 2024

Conversation

yuzhichang
Copy link
Member

What problem does this PR solve?

Try to reuse existing chunks. Close #3793

Type of change

  • New Feature (non-breaking change which adds functionality)

@yuzhichang yuzhichang requested a review from KevinHuSh December 11, 2024 04:41
@yuzhichang yuzhichang added the ci Continue Integration label Dec 11, 2024
@yuzhichang yuzhichang force-pushed the reuse_chunks branch 2 times, most recently from f6d2246 to 829c389 Compare December 11, 2024 04:53
@yuzhichang yuzhichang added wip and removed ci Continue Integration labels Dec 11, 2024
@yuzhichang yuzhichang marked this pull request as draft December 11, 2024 06:29
@yuzhichang yuzhichang force-pushed the reuse_chunks branch 2 times, most recently from c4b3797 to 22e82ba Compare December 12, 2024 08:06
@yuzhichang yuzhichang marked this pull request as ready for review December 12, 2024 08:06
@yuzhichang yuzhichang added ci Continue Integration and removed wip labels Dec 12, 2024
@yuzhichang yuzhichang merged commit 301f958 into infiniflow:main Dec 12, 2024
2 checks passed
isthaison pushed a commit to isthaison/ragflow that referenced this pull request Dec 13, 2024
@yuzhichang @isthaison
Try to reuse existing chunks (infiniflow#3983)
9c2a484 
### What problem does this PR solve?

Try to reuse existing chunks. Close infiniflow#3793
### Type of change

- [x] New Feature (non-breaking change which adds functionality)
isthaison pushed a commit to isthaison/ragflow that referenced this pull request Dec 13, 2024
@yuzhichang @isthaison
Try to reuse existing chunks (infiniflow#3983)
6953019 
### What problem does this PR solve?

Try to reuse existing chunks. Close infiniflow#3793
### Type of change

- [x] New Feature (non-breaking change which adds functionality)
isthaison added a commit to isthaison/ragflow that referenced this pull request Dec 13, 2024
@isthaison
Squashed commit of the following:
0f0eb35 
commit 9b2ef62
Author: Zhichang Yu <[email protected]>
Date:   Fri Dec 13 17:31:15 2024 +0800

    Fix xinfo_groups returns unexpected result (infiniflow#4026)

    ### What problem does this PR solve?

    Fix xinfo_groups returns unexpected result. Close infiniflow#3545

    ### Type of change

    - [x] Bug Fix (non-breaking change which fixes an issue)

commit 86507af
Author: Zhichang Yu <[email protected]>
Date:   Fri Dec 13 17:15:08 2024 +0800

    Set task progress on exception (infiniflow#4025)

    ### What problem does this PR solve?

    Set task progress on exception

    ### Type of change

    - [x] Bug Fix (non-breaking change which fixes an issue)

commit 9363567
Author: balibabu <[email protected]>
Date:   Fri Dec 13 16:55:13 2024 +0800

    Feat: Reparse a file shall reuse existing chunks if possible infiniflow#3793 (infiniflow#4021)

    ### What problem does this PR solve?

    Feat: Reparse a file shall reuse existing chunks if possible infiniflow#3793

    ### Type of change

    - [x] New Feature (non-breaking change which adds functionality)

commit 1defe0b
Author: balibabu <[email protected]>
Date:   Fri Dec 13 14:43:24 2024 +0800

    Feat: Supports to debug single component in Agent. infiniflow#3993 (infiniflow#4007)

    ### What problem does this PR solve?

    Feat: Supports to debug single component in Agent. infiniflow#3993
    Fix: The github button on the login page is displayed incorrectly  infiniflow#4002

    ### Type of change

    - [x] New Feature (non-breaking change which adds functionality)

commit 0bca46a
Author: Zhichang Yu <[email protected]>
Date:   Fri Dec 13 13:43:56 2024 +0800

    Migrate infinity at startup (infiniflow#3858)

    ### What problem does this PR solve?

    Migrate infinity at startup

    infiniflow#3809
    infiniflow/infinity#2321

    ### Type of change

    - [x] New Feature (non-breaking change which adds functionality)

commit 1ecb687
Author: liuhua <[email protected]>
Date:   Fri Dec 13 10:25:52 2024 +0800

    Fix bugs in agent api and update api document (infiniflow#3996)

    ### What problem does this PR solve?

    Fix bugs in agent api and update api document

    ### Type of change

    - [x] Bug Fix (non-breaking change which fixes an issue)
    - [x] New Feature (non-breaking change which adds functionality)

    ---------

    Co-authored-by: liuhua <[email protected]>

commit 68d46b2
Author: Luo Pan <[email protected]>
Date:   Fri Dec 13 08:50:58 2024 +0800

    Fix bug in hierarchical_merge function (infiniflow#4006)

    ### What problem does this PR solve?

    Fix hierarchical_merge function. From idx vs. actual value to actual
    value vs. actual value.
    Related issue infiniflow#4003

    ### Type of change

    - [x] Bug Fix (non-breaking change which fixes an issue)

    Co-authored-by: luopan <[email protected]>

commit 7559bbd
Author: Kevin Hu <[email protected]>
Date:   Fri Dec 13 08:50:32 2024 +0800

    Component debugging funcionality. (infiniflow#4012)

    ### What problem does this PR solve?

    infiniflow#3993
    ### Type of change

    - [x] New Feature (non-breaking change which adds functionality)

commit 275b5d1
Author: Jin Hai <[email protected]>
Date:   Thu Dec 12 20:34:46 2024 +0800

    Fix json file parse (infiniflow#4004)

    ### What problem does this PR solve?

    Fix json file parsing

    ### Type of change

    - [x] Bug Fix (non-breaking change which fixes an issue)

    ---------

    Signed-off-by: jinhai <[email protected]>

commit 9ae81b4
Author: writinwaters <[email protected]>
Date:   Thu Dec 12 19:46:53 2024 +0800

    Updated UI (infiniflow#4011)

    ### What problem does this PR solve?

    ### Type of change

    - [x] Documentation Update

commit d6c74ff
Author: so95 <[email protected]>
Date:   Thu Dec 12 18:26:44 2024 +0700

    Add mssql support (infiniflow#3985)

    some thing
    -  execsql  add connection mssql
    - fix bug duckduckgo-search rate limit
    - update typo vi res

    ---------

    Co-authored-by: Kevin Hu <[email protected]>

commit e8d7410
Author: Roccat <[email protected]>
Date:   Thu Dec 12 19:00:34 2024 +0800

    Fix: Completion AttributeError: 'list' object has no attribute 'get' (infiniflow#3999)

    ### What problem does this PR solve?

    ### Type of change

    - [x] Bug Fix (non-breaking change which fixes an issue)

    Co-authored-by: [email protected] <[email protected]>

commit c8b1a56
Author: Zhichang Yu <[email protected]>
Date:   Thu Dec 12 17:47:39 2024 +0800

    Replaced md5 with xxhash64 for chunk id (infiniflow#4009)

    ### What problem does this PR solve?

    Replaced md5 with xxhash64 for chunk id

    ### Type of change

    - [x] Refactoring

commit 301f958
Author: Zhichang Yu <[email protected]>
Date:   Thu Dec 12 16:38:03 2024 +0800

    Try to reuse existing chunks (infiniflow#3983)

    ### What problem does this PR solve?

    Try to reuse existing chunks. Close infiniflow#3793
    ### Type of change

    - [x] New Feature (non-breaking change which adds functionality)

commit 835fd7a
Author: writinwaters <[email protected]>
Date:   Thu Dec 12 11:45:59 2024 +0800

    Updated RAGFlow edition descriptions (infiniflow#4001)

    ### What problem does this PR solve?

    ### Type of change

    - [x] Documentation Update

commit bb8f97c
Author: writinwaters <[email protected]>
Date:   Thu Dec 12 09:57:52 2024 +0800

    UI updates + RAGFlow image description (infiniflow#3995)

    ### What problem does this PR solve?

    ### Type of change

    - [x] Documentation Update

commit 6d19294
Author: Kevin Hu <[email protected]>
Date:   Wed Dec 11 19:23:59 2024 +0800

    Support debug components. (infiniflow#3994)

    ### What problem does this PR solve?

    infiniflow#3993

    ### Type of change

    - [x] Bug Fix (non-breaking change which fixes an issue)

commit f61c276
Author: Jin Hai <[email protected]>
Date:   Wed Dec 11 18:39:09 2024 +0800

    Update comment (infiniflow#3981)

    ### What problem does this PR solve?

    _Briefly describe what this PR aims to solve. Include background context
    that will help reviewers understand the purpose of the PR._

    ### Type of change

    - [x] Documentation Update

    ---------

    Signed-off-by: Jin Hai <[email protected]>
    Co-authored-by: writinwaters <[email protected]>

commit 409acf0
Author: balibabu <[email protected]>
Date:   Wed Dec 11 16:29:17 2024 +0800

    Fix:  Fixed the issue where two consecutive indexes were displayed incorrectly infiniflow#3839 (infiniflow#3988)

    ### What problem does this PR solve?

    Fix: Fixed the issue where two consecutive indexes were displayed
    incorrectly infiniflow#3839

    ### Type of change

    - [x] Bug Fix (non-breaking change which fixes an issue)

commit 74c6b21
Author: liuhua <[email protected]>
Date:   Wed Dec 11 12:38:57 2024 +0800

    Update api documents (infiniflow#3979)

    ### What problem does this PR solve?

    Update api documents

    ### Type of change

    - [x] Documentation Update

    Co-authored-by: liuhua <[email protected]>

commit beeacd3
Author: Kevin Hu <[email protected]>
Date:   Wed Dec 11 11:44:59 2024 +0800

    Fix exec sql exception issue. (infiniflow#3982)

    ### What problem does this PR solve?
    infiniflow#3978

    ### Type of change

    - [x] Bug Fix (non-breaking change which fixes an issue)

commit 95259af
Author: so95 <[email protected]>
Date:   Wed Dec 11 10:12:57 2024 +0700

    update typo vietnamese (infiniflow#3973)

    update typo vietnamese

    ### Type of change
    - [x] Refactoring

    ---------

    Co-authored-by: Yingfeng <[email protected]>
    Co-authored-by: Jin Hai <[email protected]>
    Co-authored-by: bill <[email protected]>

commit 8554550
Author: Scott Davidson <[email protected]>
Date:   Wed Dec 11 03:10:33 2024 +0000

    Disable SQL DB binlog in Helm chart (infiniflow#3976)

    ### What problem does this PR solve?

    The initial Helm chart implementation added in infiniflow#3815 suffers from an
    issue where the 5GB data volume for the SQL DB is filled up with
    [binlog](https://dev.mysql.com/doc/refman/8.4/en/binary-log.html) files
    after just a few days. Since the app uses a non-replicated SQL DB config
    I think it makes sense to disable the binlog in the SQL DB container.
    This is achieved by simply adding the required argument to the container
    startup command.

    ### Type of change

    - [X] Bug Fix (non-breaking change which fixes an issue)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@KevinHuSh KevinHuSh KevinHuSh approved these changes

Assignees
No one assigned
Labels
ci Continue Integration
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[Feature Request]: Reparse a file shall reuse existing chunks if possible
2 participants
@yuzhichang @KevinHuSh