diff --git a/FAQ_zh.md b/FAQ_zh.md new file mode 100644 index 00000000..b9d95ce2 --- /dev/null +++ b/FAQ_zh.md @@ -0,0 +1,4 @@ +## 运行时报错，比如大模型不回答问题，或上传文件报错，或多个文件解析失败 +- 请查看内存使用情况，目前出错大概率是因为内存不足导致的，可以先尝试释放其他程序占用内存，再尝试运行 + - 如果还是不行，可以尝试关闭部分依赖服务，比如ocr服务（只用于jpg，jpeg，png格式解析），再尝试运行 +- 请查看README中的[DEBUG]，日志中会有详细的报错信息，可以根据报错信息进行调试 \ No newline at end of file diff --git a/LICENSE b/LICENSE index 11c0f5b5..be3f7b28 100644 --- a/LICENSE +++ b/LICENSE @@ -1,201 +1,661 @@ - Apache License - Version 2.0, January 2004 - http://www.apache.org/licenses/ - - TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION - - 1. Definitions. - - "License" shall mean the terms and conditions for use, reproduction, - and distribution as defined by Sections 1 through 9 of this document. - - "Licensor" shall mean the copyright owner or entity authorized by - the copyright owner that is granting the License. - - "Legal Entity" shall mean the union of the acting entity and all - other entities that control, are controlled by, or are under common - control with that entity. For the purposes of this definition, - "control" means (i) the power, direct or indirect, to cause the - direction or management of such entity, whether by contract or - otherwise, or (ii) ownership of fifty percent (50%) or more of the - outstanding shares, or (iii) beneficial ownership of such entity. - - "You" (or "Your") shall mean an individual or Legal Entity - exercising permissions granted by this License. - - "Source" form shall mean the preferred form for making modifications, - including but not limited to software source code, documentation - source, and configuration files. - - "Object" form shall mean any form resulting from mechanical - transformation or translation of a Source form, including but - not limited to compiled object code, generated documentation, - and conversions to other media types. - - "Work" shall mean the work of authorship, whether in Source or - Object form, made available under the License, as indicated by a - copyright notice that is included in or attached to the work - (an example is provided in the Appendix below). - - "Derivative Works" shall mean any work, whether in Source or Object - form, that is based on (or derived from) the Work and for which the - editorial revisions, annotations, elaborations, or other modifications - represent, as a whole, an original work of authorship. For the purposes - of this License, Derivative Works shall not include works that remain - separable from, or merely link (or bind by name) to the interfaces of, - the Work and Derivative Works thereof. - - "Contribution" shall mean any work of authorship, including - the original version of the Work and any modifications or additions - to that Work or Derivative Works thereof, that is intentionally - submitted to Licensor for inclusion in the Work by the copyright owner - or by an individual or Legal Entity authorized to submit on behalf of - the copyright owner. For the purposes of this definition, "submitted" - means any form of electronic, verbal, or written communication sent - to the Licensor or its representatives, including but not limited to - communication on electronic mailing lists, source code control systems, - and issue tracking systems that are managed by, or on behalf of, the - Licensor for the purpose of discussing and improving the Work, but - excluding communication that is conspicuously marked or otherwise - designated in writing by the copyright owner as "Not a Contribution." - - "Contributor" shall mean Licensor and any individual or Legal Entity - on behalf of whom a Contribution has been received by Licensor and - subsequently incorporated within the Work. - - 2. Grant of Copyright License. Subject to the terms and conditions of - this License, each Contributor hereby grants to You a perpetual, - worldwide, non-exclusive, no-charge, royalty-free, irrevocable - copyright license to reproduce, prepare Derivative Works of, - publicly display, publicly perform, sublicense, and distribute the - Work and such Derivative Works in Source or Object form. - - 3. Grant of Patent License. Subject to the terms and conditions of - this License, each Contributor hereby grants to You a perpetual, - worldwide, non-exclusive, no-charge, royalty-free, irrevocable - (except as stated in this section) patent license to make, have made, - use, offer to sell, sell, import, and otherwise transfer the Work, - where such license applies only to those patent claims licensable - by such Contributor that are necessarily infringed by their - Contribution(s) alone or by combination of their Contribution(s) - with the Work to which such Contribution(s) was submitted. If You - institute patent litigation against any entity (including a - cross-claim or counterclaim in a lawsuit) alleging that the Work - or a Contribution incorporated within the Work constitutes direct - or contributory patent infringement, then any patent licenses - granted to You under this License for that Work shall terminate - as of the date such litigation is filed. - - 4. Redistribution. You may reproduce and distribute copies of the - Work or Derivative Works thereof in any medium, with or without - modifications, and in Source or Object form, provided that You - meet the following conditions: - - (a) You must give any other recipients of the Work or - Derivative Works a copy of this License; and - - (b) You must cause any modified files to carry prominent notices - stating that You changed the files; and - - (c) You must retain, in the Source form of any Derivative Works - that You distribute, all copyright, patent, trademark, and - attribution notices from the Source form of the Work, - excluding those notices that do not pertain to any part of - the Derivative Works; and - - (d) If the Work includes a "NOTICE" text file as part of its - distribution, then any Derivative Works that You distribute must - include a readable copy of the attribution notices contained - within such NOTICE file, excluding those notices that do not - pertain to any part of the Derivative Works, in at least one - of the following places: within a NOTICE text file distributed - as part of the Derivative Works; within the Source form or - documentation, if provided along with the Derivative Works; or, - within a display generated by the Derivative Works, if and - wherever such third-party notices normally appear. The contents - of the NOTICE file are for informational purposes only and - do not modify the License. You may add Your own attribution - notices within Derivative Works that You distribute, alongside - or as an addendum to the NOTICE text from the Work, provided - that such additional attribution notices cannot be construed - as modifying the License. - - You may add Your own copyright statement to Your modifications and - may provide additional or different license terms and conditions - for use, reproduction, or distribution of Your modifications, or - for any such Derivative Works as a whole, provided Your use, - reproduction, and distribution of the Work otherwise complies with - the conditions stated in this License. - - 5. Submission of Contributions. Unless You explicitly state otherwise, - any Contribution intentionally submitted for inclusion in the Work - by You to the Licensor shall be under the terms and conditions of - this License, without any additional terms or conditions. - Notwithstanding the above, nothing herein shall supersede or modify - the terms of any separate license agreement you may have executed - with Licensor regarding such Contributions. - - 6. Trademarks. This License does not grant permission to use the trade - names, trademarks, service marks, or product names of the Licensor, - except as required for reasonable and customary use in describing the - origin of the Work and reproducing the content of the NOTICE file. - - 7. Disclaimer of Warranty. Unless required by applicable law or - agreed to in writing, Licensor provides the Work (and each - Contributor provides its Contributions) on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or - implied, including, without limitation, any warranties or conditions - of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A - PARTICULAR PURPOSE. You are solely responsible for determining the - appropriateness of using or redistributing the Work and assume any - risks associated with Your exercise of permissions under this License. - - 8. Limitation of Liability. In no event and under no legal theory, - whether in tort (including negligence), contract, or otherwise, - unless required by applicable law (such as deliberate and grossly - negligent acts) or agreed to in writing, shall any Contributor be - liable to You for damages, including any direct, indirect, special, - incidental, or consequential damages of any character arising as a - result of this License or out of the use or inability to use the - Work (including but not limited to damages for loss of goodwill, - work stoppage, computer failure or malfunction, or any and all - other commercial damages or losses), even if such Contributor - has been advised of the possibility of such damages. - - 9. Accepting Warranty or Additional Liability. While redistributing - the Work or Derivative Works thereof, You may choose to offer, - and charge a fee for, acceptance of support, warranty, indemnity, - or other liability obligations and/or rights consistent with this - License. However, in accepting such obligations, You may act only - on Your own behalf and on Your sole responsibility, not on behalf - of any other Contributor, and only if You agree to indemnify, - defend, and hold each Contributor harmless for any liability - incurred by, or claims asserted against, such Contributor by reason - of your accepting any such warranty or additional liability. - - END OF TERMS AND CONDITIONS - - APPENDIX: How to apply the Apache License to your work. - - To apply the Apache License to your work, attach the following - boilerplate notice, with the fields enclosed by brackets "[]" - replaced with your own identifying information. (Don't include - the brackets!) The text should be enclosed in the appropriate - comment syntax for the file format. We also recommend that a - file or class name and description of purpose be included on the - same "printed page" as the copyright notice for easier - identification within third-party archives. - - Copyright 2023 Zihui Ren - - Licensed under the Apache License, Version 2.0 (the "License"); - you may not use this file except in compliance with the License. - You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. + GNU AFFERO GENERAL PUBLIC LICENSE + Version 3, 19 November 2007 + + Copyright (C) 2007 Free Software Foundation, Inc. + Everyone is permitted to copy and distribute verbatim copies + of this license document, but changing it is not allowed. + + Preamble + + The GNU Affero General Public License is a free, copyleft license for +software and other kinds of works, specifically designed to ensure +cooperation with the community in the case of network server software. + + The licenses for most software and other practical works are designed +to take away your freedom to share and change the works. By contrast, +our General Public Licenses are intended to guarantee your freedom to +share and change all versions of a program--to make sure it remains free +software for all its users. + + When we speak of free software, we are referring to freedom, not +price. Our General Public Licenses are designed to make sure that you +have the freedom to distribute copies of free software (and charge for +them if you wish), that you receive source code or can get it if you +want it, that you can change the software or use pieces of it in new +free programs, and that you know you can do these things. + + Developers that use our General Public Licenses protect your rights +with two steps: (1) assert copyright on the software, and (2) offer +you this License which gives you legal permission to copy, distribute +and/or modify the software. + + A secondary benefit of defending all users' freedom is that +improvements made in alternate versions of the program, if they +receive widespread use, become available for other developers to +incorporate. Many developers of free software are heartened and +encouraged by the resulting cooperation. However, in the case of +software used on network servers, this result may fail to come about. +The GNU General Public License permits making a modified version and +letting the public access it on a server without ever releasing its +source code to the public. + + The GNU Affero General Public License is designed specifically to +ensure that, in such cases, the modified source code becomes available +to the community. It requires the operator of a network server to +provide the source code of the modified version running there to the +users of that server. Therefore, public use of a modified version, on +a publicly accessible server, gives the public access to the source +code of the modified version. + + An older license, called the Affero General Public License and +published by Affero, was designed to accomplish similar goals. This is +a different license, not a version of the Affero GPL, but Affero has +released a new version of the Affero GPL which permits relicensing under +this license. + + The precise terms and conditions for copying, distribution and +modification follow. + + TERMS AND CONDITIONS + + 0. Definitions. + + "This License" refers to version 3 of the GNU Affero General Public License. + + "Copyright" also means copyright-like laws that apply to other kinds of +works, such as semiconductor masks. + + "The Program" refers to any copyrightable work licensed under this +License. Each licensee is addressed as "you". "Licensees" and +"recipients" may be individuals or organizations. + + To "modify" a work means to copy from or adapt all or part of the work +in a fashion requiring copyright permission, other than the making of an +exact copy. The resulting work is called a "modified version" of the +earlier work or a work "based on" the earlier work. + + A "covered work" means either the unmodified Program or a work based +on the Program. + + To "propagate" a work means to do anything with it that, without +permission, would make you directly or secondarily liable for +infringement under applicable copyright law, except executing it on a +computer or modifying a private copy. Propagation includes copying, +distribution (with or without modification), making available to the +public, and in some countries other activities as well. + + To "convey" a work means any kind of propagation that enables other +parties to make or receive copies. Mere interaction with a user through +a computer network, with no transfer of a copy, is not conveying. + + An interactive user interface displays "Appropriate Legal Notices" +to the extent that it includes a convenient and prominently visible +feature that (1) displays an appropriate copyright notice, and (2) +tells the user that there is no warranty for the work (except to the +extent that warranties are provided), that licensees may convey the +work under this License, and how to view a copy of this License. If +the interface presents a list of user commands or options, such as a +menu, a prominent item in the list meets this criterion. + + 1. Source Code. + + The "source code" for a work means the preferred form of the work +for making modifications to it. "Object code" means any non-source +form of a work. + + A "Standard Interface" means an interface that either is an official +standard defined by a recognized standards body, or, in the case of +interfaces specified for a particular programming language, one that +is widely used among developers working in that language. + + The "System Libraries" of an executable work include anything, other +than the work as a whole, that (a) is included in the normal form of +packaging a Major Component, but which is not part of that Major +Component, and (b) serves only to enable use of the work with that +Major Component, or to implement a Standard Interface for which an +implementation is available to the public in source code form. A +"Major Component", in this context, means a major essential component +(kernel, window system, and so on) of the specific operating system +(if any) on which the executable work runs, or a compiler used to +produce the work, or an object code interpreter used to run it. + + The "Corresponding Source" for a work in object code form means all +the source code needed to generate, install, and (for an executable +work) run the object code and to modify the work, including scripts to +control those activities. However, it does not include the work's +System Libraries, or general-purpose tools or generally available free +programs which are used unmodified in performing those activities but +which are not part of the work. For example, Corresponding Source +includes interface definition files associated with source files for +the work, and the source code for shared libraries and dynamically +linked subprograms that the work is specifically designed to require, +such as by intimate data communication or control flow between those +subprograms and other parts of the work. + + The Corresponding Source need not include anything that users +can regenerate automatically from other parts of the Corresponding +Source. + + The Corresponding Source for a work in source code form is that +same work. + + 2. Basic Permissions. + + All rights granted under this License are granted for the term of +copyright on the Program, and are irrevocable provided the stated +conditions are met. This License explicitly affirms your unlimited +permission to run the unmodified Program. The output from running a +covered work is covered by this License only if the output, given its +content, constitutes a covered work. This License acknowledges your +rights of fair use or other equivalent, as provided by copyright law. + + You may make, run and propagate covered works that you do not +convey, without conditions so long as your license otherwise remains +in force. You may convey covered works to others for the sole purpose +of having them make modifications exclusively for you, or provide you +with facilities for running those works, provided that you comply with +the terms of this License in conveying all material for which you do +not control copyright. Those thus making or running the covered works +for you must do so exclusively on your behalf, under your direction +and control, on terms that prohibit them from making any copies of +your copyrighted material outside their relationship with you. + + Conveying under any other circumstances is permitted solely under +the conditions stated below. Sublicensing is not allowed; section 10 +makes it unnecessary. + + 3. Protecting Users' Legal Rights From Anti-Circumvention Law. + + No covered work shall be deemed part of an effective technological +measure under any applicable law fulfilling obligations under article +11 of the WIPO copyright treaty adopted on 20 December 1996, or +similar laws prohibiting or restricting circumvention of such +measures. + + When you convey a covered work, you waive any legal power to forbid +circumvention of technological measures to the extent such circumvention +is effected by exercising rights under this License with respect to +the covered work, and you disclaim any intention to limit operation or +modification of the work as a means of enforcing, against the work's +users, your or third parties' legal rights to forbid circumvention of +technological measures. + + 4. Conveying Verbatim Copies. + + You may convey verbatim copies of the Program's source code as you +receive it, in any medium, provided that you conspicuously and +appropriately publish on each copy an appropriate copyright notice; +keep intact all notices stating that this License and any +non-permissive terms added in accord with section 7 apply to the code; +keep intact all notices of the absence of any warranty; and give all +recipients a copy of this License along with the Program. + + You may charge any price or no price for each copy that you convey, +and you may offer support or warranty protection for a fee. + + 5. Conveying Modified Source Versions. + + You may convey a work based on the Program, or the modifications to +produce it from the Program, in the form of source code under the +terms of section 4, provided that you also meet all of these conditions: + + a) The work must carry prominent notices stating that you modified + it, and giving a relevant date. + + b) The work must carry prominent notices stating that it is + released under this License and any conditions added under section + 7. This requirement modifies the requirement in section 4 to + "keep intact all notices". + + c) You must license the entire work, as a whole, under this + License to anyone who comes into possession of a copy. This + License will therefore apply, along with any applicable section 7 + additional terms, to the whole of the work, and all its parts, + regardless of how they are packaged. This License gives no + permission to license the work in any other way, but it does not + invalidate such permission if you have separately received it. + + d) If the work has interactive user interfaces, each must display + Appropriate Legal Notices; however, if the Program has interactive + interfaces that do not display Appropriate Legal Notices, your + work need not make them do so. + + A compilation of a covered work with other separate and independent +works, which are not by their nature extensions of the covered work, +and which are not combined with it such as to form a larger program, +in or on a volume of a storage or distribution medium, is called an +"aggregate" if the compilation and its resulting copyright are not +used to limit the access or legal rights of the compilation's users +beyond what the individual works permit. Inclusion of a covered work +in an aggregate does not cause this License to apply to the other +parts of the aggregate. + + 6. Conveying Non-Source Forms. + + You may convey a covered work in object code form under the terms +of sections 4 and 5, provided that you also convey the +machine-readable Corresponding Source under the terms of this License, +in one of these ways: + + a) Convey the object code in, or embodied in, a physical product + (including a physical distribution medium), accompanied by the + Corresponding Source fixed on a durable physical medium + customarily used for software interchange. + + b) Convey the object code in, or embodied in, a physical product + (including a physical distribution medium), accompanied by a + written offer, valid for at least three years and valid for as + long as you offer spare parts or customer support for that product + model, to give anyone who possesses the object code either (1) a + copy of the Corresponding Source for all the software in the + product that is covered by this License, on a durable physical + medium customarily used for software interchange, for a price no + more than your reasonable cost of physically performing this + conveying of source, or (2) access to copy the + Corresponding Source from a network server at no charge. + + c) Convey individual copies of the object code with a copy of the + written offer to provide the Corresponding Source. This + alternative is allowed only occasionally and noncommercially, and + only if you received the object code with such an offer, in accord + with subsection 6b. + + d) Convey the object code by offering access from a designated + place (gratis or for a charge), and offer equivalent access to the + Corresponding Source in the same way through the same place at no + further charge. You need not require recipients to copy the + Corresponding Source along with the object code. If the place to + copy the object code is a network server, the Corresponding Source + may be on a different server (operated by you or a third party) + that supports equivalent copying facilities, provided you maintain + clear directions next to the object code saying where to find the + Corresponding Source. Regardless of what server hosts the + Corresponding Source, you remain obligated to ensure that it is + available for as long as needed to satisfy these requirements. + + e) Convey the object code using peer-to-peer transmission, provided + you inform other peers where the object code and Corresponding + Source of the work are being offered to the general public at no + charge under subsection 6d. + + A separable portion of the object code, whose source code is excluded +from the Corresponding Source as a System Library, need not be +included in conveying the object code work. + + A "User Product" is either (1) a "consumer product", which means any +tangible personal property which is normally used for personal, family, +or household purposes, or (2) anything designed or sold for incorporation +into a dwelling. In determining whether a product is a consumer product, +doubtful cases shall be resolved in favor of coverage. For a particular +product received by a particular user, "normally used" refers to a +typical or common use of that class of product, regardless of the status +of the particular user or of the way in which the particular user +actually uses, or expects or is expected to use, the product. A product +is a consumer product regardless of whether the product has substantial +commercial, industrial or non-consumer uses, unless such uses represent +the only significant mode of use of the product. + + "Installation Information" for a User Product means any methods, +procedures, authorization keys, or other information required to install +and execute modified versions of a covered work in that User Product from +a modified version of its Corresponding Source. The information must +suffice to ensure that the continued functioning of the modified object +code is in no case prevented or interfered with solely because +modification has been made. + + If you convey an object code work under this section in, or with, or +specifically for use in, a User Product, and the conveying occurs as +part of a transaction in which the right of possession and use of the +User Product is transferred to the recipient in perpetuity or for a +fixed term (regardless of how the transaction is characterized), the +Corresponding Source conveyed under this section must be accompanied +by the Installation Information. But this requirement does not apply +if neither you nor any third party retains the ability to install +modified object code on the User Product (for example, the work has +been installed in ROM). + + The requirement to provide Installation Information does not include a +requirement to continue to provide support service, warranty, or updates +for a work that has been modified or installed by the recipient, or for +the User Product in which it has been modified or installed. Access to a +network may be denied when the modification itself materially and +adversely affects the operation of the network or violates the rules and +protocols for communication across the network. + + Corresponding Source conveyed, and Installation Information provided, +in accord with this section must be in a format that is publicly +documented (and with an implementation available to the public in +source code form), and must require no special password or key for +unpacking, reading or copying. + + 7. Additional Terms. + + "Additional permissions" are terms that supplement the terms of this +License by making exceptions from one or more of its conditions. +Additional permissions that are applicable to the entire Program shall +be treated as though they were included in this License, to the extent +that they are valid under applicable law. If additional permissions +apply only to part of the Program, that part may be used separately +under those permissions, but the entire Program remains governed by +this License without regard to the additional permissions. + + When you convey a copy of a covered work, you may at your option +remove any additional permissions from that copy, or from any part of +it. (Additional permissions may be written to require their own +removal in certain cases when you modify the work.) You may place +additional permissions on material, added by you to a covered work, +for which you have or can give appropriate copyright permission. + + Notwithstanding any other provision of this License, for material you +add to a covered work, you may (if authorized by the copyright holders of +that material) supplement the terms of this License with terms: + + a) Disclaiming warranty or limiting liability differently from the + terms of sections 15 and 16 of this License; or + + b) Requiring preservation of specified reasonable legal notices or + author attributions in that material or in the Appropriate Legal + Notices displayed by works containing it; or + + c) Prohibiting misrepresentation of the origin of that material, or + requiring that modified versions of such material be marked in + reasonable ways as different from the original version; or + + d) Limiting the use for publicity purposes of names of licensors or + authors of the material; or + + e) Declining to grant rights under trademark law for use of some + trade names, trademarks, or service marks; or + + f) Requiring indemnification of licensors and authors of that + material by anyone who conveys the material (or modified versions of + it) with contractual assumptions of liability to the recipient, for + any liability that these contractual assumptions directly impose on + those licensors and authors. + + All other non-permissive additional terms are considered "further +restrictions" within the meaning of section 10. If the Program as you +received it, or any part of it, contains a notice stating that it is +governed by this License along with a term that is a further +restriction, you may remove that term. If a license document contains +a further restriction but permits relicensing or conveying under this +License, you may add to a covered work material governed by the terms +of that license document, provided that the further restriction does +not survive such relicensing or conveying. + + If you add terms to a covered work in accord with this section, you +must place, in the relevant source files, a statement of the +additional terms that apply to those files, or a notice indicating +where to find the applicable terms. + + Additional terms, permissive or non-permissive, may be stated in the +form of a separately written license, or stated as exceptions; +the above requirements apply either way. + + 8. Termination. + + You may not propagate or modify a covered work except as expressly +provided under this License. Any attempt otherwise to propagate or +modify it is void, and will automatically terminate your rights under +this License (including any patent licenses granted under the third +paragraph of section 11). + + However, if you cease all violation of this License, then your +license from a particular copyright holder is reinstated (a) +provisionally, unless and until the copyright holder explicitly and +finally terminates your license, and (b) permanently, if the copyright +holder fails to notify you of the violation by some reasonable means +prior to 60 days after the cessation. + + Moreover, your license from a particular copyright holder is +reinstated permanently if the copyright holder notifies you of the +violation by some reasonable means, this is the first time you have +received notice of violation of this License (for any work) from that +copyright holder, and you cure the violation prior to 30 days after +your receipt of the notice. + + Termination of your rights under this section does not terminate the +licenses of parties who have received copies or rights from you under +this License. If your rights have been terminated and not permanently +reinstated, you do not qualify to receive new licenses for the same +material under section 10. + + 9. Acceptance Not Required for Having Copies. + + You are not required to accept this License in order to receive or +run a copy of the Program. Ancillary propagation of a covered work +occurring solely as a consequence of using peer-to-peer transmission +to receive a copy likewise does not require acceptance. However, +nothing other than this License grants you permission to propagate or +modify any covered work. These actions infringe copyright if you do +not accept this License. Therefore, by modifying or propagating a +covered work, you indicate your acceptance of this License to do so. + + 10. Automatic Licensing of Downstream Recipients. + + Each time you convey a covered work, the recipient automatically +receives a license from the original licensors, to run, modify and +propagate that work, subject to this License. You are not responsible +for enforcing compliance by third parties with this License. + + An "entity transaction" is a transaction transferring control of an +organization, or substantially all assets of one, or subdividing an +organization, or merging organizations. If propagation of a covered +work results from an entity transaction, each party to that +transaction who receives a copy of the work also receives whatever +licenses to the work the party's predecessor in interest had or could +give under the previous paragraph, plus a right to possession of the +Corresponding Source of the work from the predecessor in interest, if +the predecessor has it or can get it with reasonable efforts. + + You may not impose any further restrictions on the exercise of the +rights granted or affirmed under this License. For example, you may +not impose a license fee, royalty, or other charge for exercise of +rights granted under this License, and you may not initiate litigation +(including a cross-claim or counterclaim in a lawsuit) alleging that +any patent claim is infringed by making, using, selling, offering for +sale, or importing the Program or any portion of it. + + 11. Patents. + + A "contributor" is a copyright holder who authorizes use under this +License of the Program or a work on which the Program is based. The +work thus licensed is called the contributor's "contributor version". + + A contributor's "essential patent claims" are all patent claims +owned or controlled by the contributor, whether already acquired or +hereafter acquired, that would be infringed by some manner, permitted +by this License, of making, using, or selling its contributor version, +but do not include claims that would be infringed only as a +consequence of further modification of the contributor version. For +purposes of this definition, "control" includes the right to grant +patent sublicenses in a manner consistent with the requirements of +this License. + + Each contributor grants you a non-exclusive, worldwide, royalty-free +patent license under the contributor's essential patent claims, to +make, use, sell, offer for sale, import and otherwise run, modify and +propagate the contents of its contributor version. + + In the following three paragraphs, a "patent license" is any express +agreement or commitment, however denominated, not to enforce a patent +(such as an express permission to practice a patent or covenant not to +sue for patent infringement). To "grant" such a patent license to a +party means to make such an agreement or commitment not to enforce a +patent against the party. + + If you convey a covered work, knowingly relying on a patent license, +and the Corresponding Source of the work is not available for anyone +to copy, free of charge and under the terms of this License, through a +publicly available network server or other readily accessible means, +then you must either (1) cause the Corresponding Source to be so +available, or (2) arrange to deprive yourself of the benefit of the +patent license for this particular work, or (3) arrange, in a manner +consistent with the requirements of this License, to extend the patent +license to downstream recipients. "Knowingly relying" means you have +actual knowledge that, but for the patent license, your conveying the +covered work in a country, or your recipient's use of the covered work +in a country, would infringe one or more identifiable patents in that +country that you have reason to believe are valid. + + If, pursuant to or in connection with a single transaction or +arrangement, you convey, or propagate by procuring conveyance of, a +covered work, and grant a patent license to some of the parties +receiving the covered work authorizing them to use, propagate, modify +or convey a specific copy of the covered work, then the patent license +you grant is automatically extended to all recipients of the covered +work and works based on it. + + A patent license is "discriminatory" if it does not include within +the scope of its coverage, prohibits the exercise of, or is +conditioned on the non-exercise of one or more of the rights that are +specifically granted under this License. You may not convey a covered +work if you are a party to an arrangement with a third party that is +in the business of distributing software, under which you make payment +to the third party based on the extent of your activity of conveying +the work, and under which the third party grants, to any of the +parties who would receive the covered work from you, a discriminatory +patent license (a) in connection with copies of the covered work +conveyed by you (or copies made from those copies), or (b) primarily +for and in connection with specific products or compilations that +contain the covered work, unless you entered into that arrangement, +or that patent license was granted, prior to 28 March 2007. + + Nothing in this License shall be construed as excluding or limiting +any implied license or other defenses to infringement that may +otherwise be available to you under applicable patent law. + + 12. No Surrender of Others' Freedom. + + If conditions are imposed on you (whether by court order, agreement or +otherwise) that contradict the conditions of this License, they do not +excuse you from the conditions of this License. If you cannot convey a +covered work so as to satisfy simultaneously your obligations under this +License and any other pertinent obligations, then as a consequence you may +not convey it at all. For example, if you agree to terms that obligate you +to collect a royalty for further conveying from those to whom you convey +the Program, the only way you could satisfy both those terms and this +License would be to refrain entirely from conveying the Program. + + 13. Remote Network Interaction; Use with the GNU General Public License. + + Notwithstanding any other provision of this License, if you modify the +Program, your modified version must prominently offer all users +interacting with it remotely through a computer network (if your version +supports such interaction) an opportunity to receive the Corresponding +Source of your version by providing access to the Corresponding Source +from a network server at no charge, through some standard or customary +means of facilitating copying of software. This Corresponding Source +shall include the Corresponding Source for any work covered by version 3 +of the GNU General Public License that is incorporated pursuant to the +following paragraph. + + Notwithstanding any other provision of this License, you have +permission to link or combine any covered work with a work licensed +under version 3 of the GNU General Public License into a single +combined work, and to convey the resulting work. The terms of this +License will continue to apply to the part which is the covered work, +but the work with which it is combined will remain governed by version +3 of the GNU General Public License. + + 14. Revised Versions of this License. + + The Free Software Foundation may publish revised and/or new versions of +the GNU Affero General Public License from time to time. Such new versions +will be similar in spirit to the present version, but may differ in detail to +address new problems or concerns. + + Each version is given a distinguishing version number. If the +Program specifies that a certain numbered version of the GNU Affero General +Public License "or any later version" applies to it, you have the +option of following the terms and conditions either of that numbered +version or of any later version published by the Free Software +Foundation. If the Program does not specify a version number of the +GNU Affero General Public License, you may choose any version ever published +by the Free Software Foundation. + + If the Program specifies that a proxy can decide which future +versions of the GNU Affero General Public License can be used, that proxy's +public statement of acceptance of a version permanently authorizes you +to choose that version for the Program. + + Later license versions may give you additional or different +permissions. However, no additional obligations are imposed on any +author or copyright holder as a result of your choosing to follow a +later version. + + 15. Disclaimer of Warranty. + + THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY +APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT +HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY +OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, +THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR +PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM +IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF +ALL NECESSARY SERVICING, REPAIR OR CORRECTION. + + 16. Limitation of Liability. + + IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING +WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS +THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY +GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE +USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF +DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD +PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS), +EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF +SUCH DAMAGES. + + 17. Interpretation of Sections 15 and 16. + + If the disclaimer of warranty and limitation of liability provided +above cannot be given local legal effect according to their terms, +reviewing courts shall apply local law that most closely approximates +an absolute waiver of all civil liability in connection with the +Program, unless a warranty or assumption of liability accompanies a +copy of the Program in return for a fee. + + END OF TERMS AND CONDITIONS + + How to Apply These Terms to Your New Programs + + If you develop a new program, and you want it to be of the greatest +possible use to the public, the best way to achieve this is to make it +free software which everyone can redistribute and change under these terms. + + To do so, attach the following notices to the program. It is safest +to attach them to the start of each source file to most effectively +state the exclusion of warranty; and each file should have at least +the "copyright" line and a pointer to where the full notice is found. + + + Copyright (C) + + This program is free software: you can redistribute it and/or modify + it under the terms of the GNU Affero General Public License as published by + the Free Software Foundation, either version 3 of the License, or + (at your option) any later version. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU Affero General Public License for more details. + + You should have received a copy of the GNU Affero General Public License + along with this program. If not, see . + +Also add information on how to contact you by electronic and paper mail. + + If your software can interact with users remotely through a computer +network, you should also make sure that it provides a way for users to +get its source. For example, if your program is a web application, its +interface could display a "Source" link that leads users to an archive +of the code. There are many ways you could offer source, and different +solutions will be better for different programs; see section 13 for the +specific requirements. + + You should also get your employer (if you work as a programmer) or school, +if any, to sign a "copyright disclaimer" for the program, if necessary. +For more information on this, and how to apply and follow the GNU AGPL, see +. diff --git a/README.md b/README.md index 0fca49aa..57ecc311 100644 --- a/README.md +++ b/README.md @@ -1,119 +1,489 @@ -# Pure Python Environment Installation Guide -

+

+ + + + Logo + + +# **Q**uestion and **A**nswer based on **Anything** + +

English | 简体中文

+
+ +
+ + +     + +     + + +     + +     + +     + + +     + + + +
+ +
+Table of Contents + +- [What is QAnything](#what-is-qanything) + - [Key features](#key-features) + - [Architecture](#architecture) +- [Latest Updates](#-latest-updates) +- [Before You Start](#before-you-start) +- [Getting Started](#getting-started) + - [Latest Features Table](#latest-features-table) + - [Version 2.0.0 adds detailed optimizations:](#version-200-adds-detailed-optimizations) + - [Display of data at each stage:](#display-of-data-at-each-stage) + - [Problem fixed](#problem-fixed) + - [Comparison of New and Old Parsing Effects](#comparison-of-new-and-old-parsing-effects) + - [Installation](#installation) + - [Prerequisites](#prerequisites) + - [step1: pull qanything repository](#step1-pull-qanything-repository) + - [step2: Enter the project root directory and execute the startup command.](#step2-enter-the-project-root-directory-and-execute-the-startup-command) + - [step3: start to experience](#step3-start-to-experience) + - [API](#api) + - [DEBUG](#debug) + - [Close service](#close-service) + - [Offline Use](#offline-use) + - [FAQ](#faq) + - [Contributing](#contributing) + - [Thanks to all contributors for their efforts](#thanks-to-all-contributors-for-their-efforts) + - [Business contact information：](#business-contact-information) +- [Roadmap & Feedback](#-roadmap--feedback) +- [Community & Support](#community--support) +- [License](#license) +- [Acknowledgements](#acknowledgments) + +
+ +# 🚀 Important Updates

Important things should be said three times.

-# [2024-05-17:Latest Installation and Usage Documentation](https://github.com/netease-youdao/QAnything/blob/master/QAnything%E4%BD%BF%E7%94%A8%E8%AF%B4%E6%98%8E.md) -# [2024-05-17:Latest Installation and Usage Documentation](https://github.com/netease-youdao/QAnything/blob/master/QAnything%E4%BD%BF%E7%94%A8%E8%AF%B4%E6%98%8E.md) -# [2024-05-17:Latest Installation and Usage Documentation](https://github.com/netease-youdao/QAnything/blob/master/QAnything%E4%BD%BF%E7%94%A8%E8%AF%B4%E6%98%8E.md) +# [2024-05-17:QAnything updated to version 2.0.] +# [2024-05-17:QAnything updated to version 2.0.] +# [2024-05-17:QAnything updated to version 2.0.] + +

+ +* This update brings improvements in various aspects such as usability, resource consumption, search results, question and answer results, parsing results, front-end effects, service architecture, and usage methods. +* At the same time, the old Docker version and Python version have been merged into a new unified version, using a single-line command with Docker Compose for one-click startup, ready to use out of the box. + +

+ + +## Contributing +We appreciate your interest in contributing to our project. Whether you're fixing a bug, improving an existing feature, or adding something completely new, your contributions are welcome! +### Thanks to all contributors for their efforts + + + + ## Business contact information： ### 010-82558901 ![](docs/images/business.jpeg) +# What is QAnything? +`QAnything`(**Q**uestion and **A**nswer based on **Anything**) is a local knowledge base question-answering system designed to support a wide range of file formats and databases, allowing for offline installation and use. -## Installation +With `QAnything`, you can simply drop any locally stored file of any format and receive accurate, fast, and reliable answers. -Requirements: +Currently supported formats include: **PDF(pdf)**,**Word(docx)**,**PPT(pptx)**,**XLS(xlsx)**,**Markdown(md)**,**Email(eml)**,**TXT(txt)**,**Image(jpg，jpeg，png)**,**CSV(csv)**,**Web links(html)** and more formats coming soon… - - Python 3.10+ (suggest using conda to manage python environment) - - System - - Linux: glibc 2.28+ and Cuda 12.0+ (if using GPU) - - Windows: WSL with Ubuntu 20.04+ and GEFORCE EXPERIENCE 535.104+ (if using GPU) - - MacOS: M1/M2/M3 Mac with Xcode 15.0+ -Please create a clean Python virtual environment to avoid potential conflicts(Recommend Anaconda3). +## Key features + +- Data security, supports installation and use by unplugging the network cable throughout the process. +- Supports multiple file types, high parsing success rate, supports cross-language question and answer, freely switches between Chinese and English question and answer, regardless of the language of the file. +- Supports massive data question and answer, two-stage vector sorting, solves the problem of degradation of large-scale data retrieval, the more data, the better the effect, no limit on the number of uploaded files, fast retrieval speed. +- Hardware friendly, defaults to running in a pure CPU environment, and supports multiple platforms such as Windows, Mac, and Linux, with no dependencies other than Docker. +- User-friendly, no need for cumbersome configuration, one-click installation and deployment, ready to use, each dependent component (PDF parsing, OCR, embed, rerank, etc.) is completely independent, supports free replacement. +- Supports a quick start mode similar to Kimi, fileless chat mode, retrieval mode only, custom Bot mode. -To install the package, run: -```bash -conda create -n qanything-python python=3.10 -conda activate qanything-python -git clone -b qanything-python https://github.com/netease-youdao/QAnything.git -cd QAnything -pip install -e . -``` -## PDF parser ++ -If you want to use a more powerful PDF parsing function, please download the [corresponding checkpoints](https://www.modelscope.cn/models/netease-youdao/QAnything-pdf-parser/files) in modelscope and put it in the path qanything_kernel/utils/loader/pdf_to_markdown/checkpoints/ -## Run With 3B LLM (MiniChat-2-3B) On Windows WSL or Linux (Require GPU with >=10GB Memory) +## Architecture +
+qanything_system +
+ +### Why 2 stage retrieval? +In scenarios with a large volume of knowledge base data, the advantages of a two-stage approach are very clear. If only a first-stage embedding retrieval is used, there will be a problem of retrieval degradation as the data volume increases, as indicated by the green line in the following graph. However, after the second-stage reranking, there can be a stable increase in accuracy, **the more data, the better the performance**. +
+two stage retrievaal +
+ +QAnything uses the retrieval component [BCEmbedding](https://github.com/netease-youdao/BCEmbedding), which is distinguished for its bilingual and crosslingual proficiency. BCEmbedding excels in bridging Chinese and English linguistic gaps, which achieves +- **A high performance on Semantic Representation Evaluations in MTEB**; +- **A new benchmark in the realm of RAG Evaluations in LlamaIndex**. + + +### 1st Retrieval（embedding） +| Model | Retrieval | STS | PairClassification | Classification | Reranking | Clustering | Avg | +|:-------------------------------|:--------:|:--------:|:--------:|:--------:|:--------:|:--------:|:--------:| +| bge-base-en-v1.5 | 37.14 | 55.06 | 75.45 | 59.73 | 43.05 | 37.74 | 47.20 | +| bge-base-zh-v1.5 | 47.60 | 63.72 | 77.40 | 63.38 | 54.85 | 32.56 | 53.60 | +| bge-large-en-v1.5 | 37.15 | 54.09 | 75.00 | 59.24 | 42.68 | 37.32 | 46.82 | +| bge-large-zh-v1.5 | 47.54 | 64.73 | **79.14** | 64.19 | 55.88 | 33.26 | 54.21 | +| jina-embeddings-v2-base-en | 31.58 | 54.28 | 74.84 | 58.42 | 41.16 | 34.67 | 44.29 | +| m3e-base | 46.29 | 63.93 | 71.84 | 64.08 | 52.38 | 37.84 | 53.54 | +| m3e-large | 34.85 | 59.74 | 67.69 | 60.07 | 48.99 | 31.62 | 46.78 | +| ***bce-embedding-base_v1*** | **57.60** | **65.73** | 74.96 | **69.00** | **57.29** | **38.95** | ***59.43*** | + +- More evaluation details please check [Embedding Models Evaluation Summary](https://github.com/netease-youdao/BCEmbedding/blob/master/Docs/EvaluationSummary/embedding_eval_summary.md)。 + +### 2nd Retrieval（rerank） +| Model | Reranking | Avg | +|:-------------------------------|:--------:|:--------:| +| bge-reranker-base | 57.78 | 57.78 | +| bge-reranker-large | 59.69 | 59.69 | +| ***bce-reranker-base_v1*** | **60.06** | ***60.06*** | + +- More evaluation details please check [Reranker Models Evaluation Summary](https://github.com/netease-youdao/BCEmbedding/blob/master/Docs/EvaluationSummary/reranker_eval_summary.md) + +### RAG Evaluations in LlamaIndex（embedding and rerank） + + + +***NOTE:*** + +- In `WithoutReranker` setting, our `bce-embedding-base_v1` outperforms all the other embedding models. +- With fixing the embedding model, our `bce-reranker-base_v1` achieves the best performance. +- **The combination of `bce-embedding-base_v1` and `bce-reranker-base_v1` is SOTA**. +- If you want to use embedding and rerank separately, please refer to [BCEmbedding](https://github.com/netease-youdao/BCEmbedding) + +### LLM + +The open source version of QAnything is based on QwenLM and has been fine-tuned on a large number of professional question-answering datasets. It greatly enhances the ability of question-answering. +If you need to use it for commercial purposes, please follow the license of QwenLM. For more details, please refer to: [QwenLM](https://github.com/QwenLM/Qwen) + +# 🚀 Latest Updates + +- ***2024-08-23***: **Support quick start, front-end configuration parameters, online preview and editing of chunk blocks, greatly optimize project architecture and startup mode, greatly optimize parsing and retrieval effects.** - See More👉 [v2.0.0](https://github.com/netease-youdao/QAnything/releases/tag/v2.0.0) +- ***2024-05-20***: **Support other large model services compatible with OpenAI API, and provide an optimized powerful PDF parser.** - See More👉 [v1.4.1](https://github.com/netease-youdao/QAnything/releases/tag/v1.4.1) +- ***2024-04-26***: **Support web search, FAQ, custom bot, file traceability preview etc.** - See More👉 [v1.4.0](https://github.com/netease-youdao/QAnything/releases/tag/v1.4.0-python) +- ***2024-04-03***: **Support installation in a pure Python environment.Support hybrid search.** - See More👉 [v1.3.0](https://github.com/netease-youdao/QAnything/releases/tag/v1.3.0) +- ***2024-01-29***: **Support for custom large models, including OpenAI API and other open-source large models, with a minimum GPU requirement of GTX 1050Ti, greatly improving deployment, debugging, and user experience.** - See More👉 [v1.2.0](https://github.com/netease-youdao/QAnything/releases/tag/v1.2.0) +- ***2024-01-23***: **Enable rerank by default and fix various issues when starting on Windows.** - See More👉 [v1.1.1](https://github.com/netease-youdao/QAnything/releases/tag/v1.1.1) +- ***2024-01-18***: **Support one-click startup, support Windows deployment, improve PDF, XLSX, HTML parsing efficiency.** - See More👉 [v1.1.0](https://github.com/netease-youdao/QAnything/releases/tag/v1.1.0) + +# Before You Start +**Star us on GitHub, and be instantly notified for new release!** +![star_us](https://github.com/netease-youdao/QAnything/assets/29041332/fd5e5926-b9b2-4675-9f60-6cdcaca18e14) +* [🏄 Try QAnything Online](https://qanything.ai) +* [📚 Try read.youdao.com | 有道速读](https://read.youdao.com) +* [🛠️ Only use our BCEmbedding(embedding & rerank)](https://github.com/netease-youdao/BCEmbedding) +* [📖 FAQ](FAQ_zh.md) +* [👂️Let me hear your voice](https://qanything.canny.io/feature-requests) + + +# Getting Started +## Latest Features Table + + +| features | python （v1.4.2） | docker （v1.2.2） | QAnything v2.0.0 | Explanation | +|----------------------------------------------------------------------|-----------------|-----------------|------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| Detailed installation document | ✅ | ✅ | ✅ | | +| Support API | ✅ | ✅ | ✅ | | +| Support production environment | ❌ | ✅ | ✅ | | +| Support offline use | ❌ | ✅ | ✅ | | +| Support multiple concurrency | ❌ | ✅ | ✅ | | +| Support multi-card inference | ❌ | ✅ | ❌ | Version 2.0.0 no longer provides default local LLM. All access is through the openai interface, and users can deploy local LLM through tools such as ollama. | +| Support Mac (M series chips) | ✅ | ❌ | ✅ | | +| Support Linux | ✅ | ✅ | ✅ | The old version of Python defaults to using onnxruntime-gpu for cuda12 on Linux, and automatically switches to onnxruntime when glibc<2.28. | +| Support windows | ❌ | ❌ | ✅ | Both old versions of Python and Docker require WSL environment. Version 2.0.0 can be started directly in a non-WSL environment. | +| Support CPU only | ✅ | ❌ | ✅ | Version 2.0.0 Mac, Linux, Win unified no longer use GPU, completely migrated to CPU. | +| Support hybrid search (BM25+embedding) | ❌ | ✅ | ✅ | | +| Support web search (need VPN) | ✅ | ❌ | ✅ | | +| Support FAQ | ✅ | ❌ | ✅ | | +| Support BOT | ✅ | ❌ | ✅ | | +| Support Traceability | ✅ | ❌ | ✅ | | +| Support Log retrieval | ✅ | ❌ | ✅ | | +| Support audio file | ✅ | ❌ | ❌ | Relying on whisper, slow speed and high resource consumption, temporarily removed. | +| Support OpenCloudOS | ✅ | ❌ | ✅ | | +| Support interfaces compatible with Openaiapi (including ollama) | ✅ | ✅ | ✅ | Old versions of Python and Docker require manual modification of parameters such as api_key, base_url, model, etc. In version 2.0.0, these are all changed to be automatically saved in the front end settings. | +| PDF parsing performance improvement (including tables) | ✅ | ❌ | ✅ | Version 1.4.2 requires manual settings, version 2.0.0 does not require manual settings, and both the PDF parsing effect and performance have been improved. | +| User-defined configuration (Experimental: Improve speed) | ✅ | ❌ | ✅ | v1.4.2 needs to be set manually, v2.0.0 uses the best configuration by default. | +| Improvement in parsing performance of other file types | ❌ | ❌ | ✅ | Version 2.0.0 improves the parsing effect of URLs, Markdown, XLSX, DOCX, etc. | +| Support independent service invocation | ❌ | ❌ | ✅ | Version 2.0.0 independent dependent services, including embed, rerank, ocr, pdf parsing services, can be called independently (http) | +| Support quick start mode | ❌ | ❌ | ✅ | Quick Start: No need to create a knowledge base, support for file upload and instant questioning, support for fileless Q&A. | +| Support only retrieval mode | ❌ | ❌ | ✅ | Only return search results, do not call the large model for question answering. | +| Support parsing result chunks content visualization, manual editing. | ❌ | ❌ | ✅ | Version 2.0.0 supports manually editing the contents of chunks, which take effect in real time. | +| PDF parsing supports images, supports answering with images. | ❌ | ❌ | ✅ | | + + +## Version 2.0.0 adds detailed optimizations: + +* Support front-end configuration API_BASE, API_KEY, text chunk size, output token quantity, context message quantity, etc. +* Optimize the instruction compliance of Bot role settings, each Bot can configure model parameters separately. +* Support creating multiple dialogue windows and saving multiple sets of historical Q&A records at the same time. +* Support saving question and answer records as images +* Optimize the logic of uploading files, parse files and question-and-answer requests independently, uploading files will no longer affect question-and-answer. +* Optimize image size, the compressed size of the old version image is 18.94GB -> the compressed size of the new version image is 4.88GB, reduced to 1/4 of the original size, providing a complete Dockerfile. +* Search optimization, chunks add fragment fusion and sorting, aggregate single document or double document. +* Both the retrieval stage and the question-answering stage embed metadata information to improve the retrieval and question-answering effectiveness. + +### Display of data at each stage: +* Display the upload progress of all files in the knowledge base. +* Display the progress of uploading a single file in the knowledge base, and the time consumed in each stage of the upload. +* Question and answer information statistics, including time consumption at each stage of question and answer, token consumption, model information, etc. +* User information statistics, including total number of uploaded files, total time consumed, question and answer history records, etc. (coming soon) + +### Problem fixed +* The xlsx file format supports parsing multiple sheets. +* Optimize the problem of missing recognition of PDF tables. +* Fix some parsing errors in DOCX files. +* Optimize FAQ matching logic. +* Support for non-UTF-8 encoded txt files. + +## Comparison of New and Old Parsing Effects +* First, with regard to the parsing of large tables in documents, especially tables that span multiple pages, version 2.0 has made significant improvements. The new version's parsing logic can analyze the structure of the table, including the layout of rows and columns, and can automatically identify the table headers, placing them at the top of each table segment that is split. This improvement prevents interruptions in meaning caused by logical segmentation when parsing long tables. + +| Original image | Old version parsing effect | New version parsing effect | +|:----:|:--------------------------------------------:|:---------------------------------------------------------------------------------------------:| +| ![image.png](docs/assets/17244247170060.png) | ![image.png](docs/assets/17244247170067.png) | ![image.png](docs/assets/17244247170074.png) | + +* In addition, version 2.0 has also been optimized for handling text columnation and cross-page layout. It can recognize double-column or multi-column layouts of text and correctly divide text blocks in accordance with human reading habits. At the same time, this version can also save images in documents to ensure the integrity of content is not lost during file parsing. As shown in the figure below, the correct arrangement should be to group the text arranged in sequence as 1, 2, 3 into a large paragraph and then segment it, rather than segmenting 1, 2, 3 separately. + * In version 1.4 parsing results, the cross-page text "higher" was chunked into the next text block, which is detrimental to large model semantic understanding. In version 2.0 parsing results, it is correctly divided, and images interspersed in text paragraphs are also parsed into corresponding chunk statements. Non-main text such as "Figure 1 Identification and Authorization and Their Support Relationship 37" and "Cover Story Cover Feature" were successfully filtered out. + +| Original image | Old version parsing effect | New version parsing effect | +|:----:|:--------------------------------------------:|:---------------------------------------------------------------------------------------------:| +| ![image.png](docs/assets/17244247170088.png) | ![image.png](docs/assets/17244247170101.png)
![image.png](docs/assets/17244247170115.png) | ![image.png](docs/assets/17244247170129.png) | + +* Version 2.0 has also made corresponding optimizations for parsing tables interspersed between text columns or text blocks. The parsing of the original version could not recognize tables and would only parse them in the format of text paragraphs. This not only destroys the logical structure of the tables but also adds a section of messy and useless text for large models, which would affect the accuracy of large model responses. Version 2.0 can recognize and parse these tables embedded in the text, thereby improving the quality of parsing and the accuracy of responses from large models. + * In version 1.4 parsing results, tables interspersed in text blocks are parsed as normal text blocks. In version 2.0, this type of table can be parsed "elegantly", which not only improves the quality of parsing but also increases the accuracy of large model answers. + * In addition, in version 2.0, when processing text under specific subheadings, priority is given to ensuring that these texts are segmented into the same chunk block to maintain logical coherence. When the text is too long and needs to be split, the parsing logic of version 2.0 will repeat the same subheading before each split text block to indicate ownership. For example, in the example, the same subheading "European Conference: Legal Status of Robots" was added to all three text blocks (due to the length of the text, this title was not displayed in the original file screenshot). This processing method effectively avoids the problem of incoherent semantic logic in split text blocks caused by excessively long text. + +| Original image | Old version parsing effect | New version parsing effect | +|:----:|:--------------------------------------------:|:---------------------------------------------------------------------------------------------:| +| ![image.png](docs/assets/17244247170148.png) | ![image.png](docs/assets/17244247170166.png) | ![image.png](docs/assets/17244247171267.png) | + +* For Excel (.xlsx) documents with complex formatting, version 2.0 has undergone a series of optimization measures to accurately identify and process row and column data, including optimized handling of merged cells and text spanning across rows or columns. Specific examples can be seen below. + * In version 1.4, there may be some limitations when parsing Excel documents, especially for documents with special structures or formats. The parsing results may not be satisfactory, mainly recognizing only the plain text part. This may lead to information loss or format disorder when dealing with complex data and formats. In contrast, version 2.0 has significantly improved parsing capabilities, able to better handle various complex formats of Excel documents. Although it may not be perfect yet, it can already solve the vast majority of complex situations. + +| Original image | Old version parsing effect | New version parsing effect | +|:----:|:--------------------------------------------:|:---------------------------------------------------------------------------------------------:| +| ![image.png](docs/assets/17244247170188.png) | ![image.png](docs/assets/17244247170208.png) | ![image.png](docs/assets/17244247170228.png) | + +* Similarly, for simple formatted xlsx documents, version 2.0 of the parser has been optimized. + +| Original image | Old version parsing effect | New version parsing effect | +|:----:|:--------------------------------------------:|:---------------------------------------------------------------------------------------------:| +| ![image.png](docs/assets/17244247170272.png) | ![image.png](docs/assets/17244247170298.png) | ![image.png](docs/assets/17244247170323.png) | + +* In the latest version, we have also made significant improvements to the URL parsing function. Taking the following page as an example, the old version may miss a large amount of page information during the parsing process and cannot effectively handle more complex page elements such as tables and lists. However, the new version has been optimized for these issues and can parse these contents more accurately. + +| Original image | Old version parsing effect | New version parsing effect | +|:----:|:--------------------------------------------:|:---------------------------------------------------------------------------------------------:| +| ![image.png](docs/assets/17244247170443.png) | ![image.png](docs/assets/17244247170478.png)
![image.png](docs/assets/17244247170512.png) | ![image.png](docs/assets/17244247170546.png) | + +* In addition, for the vast majority of files, version 2.0 has also made corresponding optimizations, including but not limited to the following points. + * Improved the cutting logic of chunk blocks, avoiding semantic blocks being too short or logic interruption due to blank lines or paragraphs in the document, ensuring the coherence and integrity of text blocks. + * The new version can more accurately identify the subheadings in the document, and locate and organize the corresponding text blocks based on these subheadings, which helps optimize the parsing effect, making the parsing structure clearer and the information hierarchy more distinct. + * The comparison of the analysis results is as follows: In version 1.4, the parsing logic divides the document into 10 chunks, while in version 2.0, after parsing, there are only 4 chunks. The more reasonable and fewer chunk blocks greatly improve the coherence and integrity of the content, helping to reduce semantic breaks or logical confusion caused by improper segmentation, thereby improving the overall parsing and model answering effects. + +| Original image | Old version parsing effect | New version parsing effect | +|:----:|:--------------------------------------------:|:---------------------------------------------------------------------------------------------:| +| ![image.png](docs/assets/17244247170352.png) | ![image.png](docs/assets/17244247170380.png) | ![image.png](docs/assets/17244247170406.png) | + +### In summary, version 2.0 parsing has optimized many aspects compared to version 1.4 parsing, including but not limited to +1. By using more reasonable chunk lengths, the semantic and logical losses caused by paragraphs being too small or incomplete are reduced. +2. Improved recognition ability for columned text, able to intelligently determine reading order, even correctly handling paragraphs that span across pages. +3. The new version can recognize and save images and tables within text paragraphs, ensuring that no important text information is missed. +4. Optimized table parsing, including parsing and storage of long tables exceeding chunk limits and complex structured xlsx files. +5. Based on the identified subheadings in the document, locate and organize corresponding text blocks to make the parsing structure clearer and the information hierarchy more distinct. +6. Optimized parsing results for webpage URLs, converted to .md format. +7. Support for more encoding formats of txt files and docx files. -```bash -bash scripts/run_for_3B_in_Linux_or_WSL.sh +## Installation +### Prerequisites +| **System** | **Required item** | **Minimum Requirement** | **Note** | +|------------|-------------------------|-------------------------|----------------------------------------------------------------------------------------| +| | RAM Memory | >= 20GB | | +| Linux/Mac | Docker version | >= 20.10.5 | [Docker install](https://docs.docker.com/engine/install/) | +| Linux/Mac | docker compose version | >= 2.23.3 | [docker compose install](https://docs.docker.com/compose/install/) | +| Windows | Docker Desktop | >= 4.26.1（131620） | [Docker Desktop for Windows](https://docs.docker.com/desktop/install/windows-install/) | + +### step1: pull qanything repository +```shell +git clone https://github.com/netease-youdao/QAnything.git ``` +### step2: Enter the project root directory and execute the startup command. +* Execute the docker compose start command -## Run With 7B LLM (Qwen-7B-QAnything) On Windows WSL or Linux (Require GPU with >=24GB Memory) +```shell +cd QAnything +# Start on Linux +docker compose -f docker-compose-linux.yaml up +# Start on Mac +docker compose -f docker-compose-mac.yaml up +# Start on Windows +docker compose -f docker-compose-win.yaml up +``` -```bash -bash scripts/run_for_7B_in_Linux_or_WSL.sh +(Note) If the startup fails, you can try changing `docker compose` to `docker-compose`. + +### step3: start to experience +#### Front end +After successful installation, you can experience the application by entering the following addresses in your web browser. + +- Front end address: http://localhost:8777/qanything/ + +### API +If you want to visit API, please refer to the following address: +- API address: http://localhost:8777/qanything/ +- For detailed API documentation, please refer to [QAnything API documentation](docs/API.md) + +### DEBUG +If you want to view the relevant logs, please check the log files in the `QAnything/logs/debug_logs` directory. +- **debug.log** + - User request processing log +- **sanic_api.log** + - Backend service running log +- **llm_embed_rerank_tritonserver.log**(Single card deployment) + - LLM embedding and rerank tritonserver service startup log +- **llm_tritonserver.log**(Multi-card deployment) + - LLM tritonserver service startup log +- **embed_rerank_tritonserver.log**(Multi-card deployment or use of the OpenAI interface.) + - Embedding and rerank tritonserver service startup log +- rerank_server.log + - Rerank service running log +- ocr_server.log + - OCR service running log +- npm_server.log + - Front-end service running log +- llm_server_entrypoint.log + - LLM intermediate server running log +- fastchat_logs/*.log + - FastChat service running log + +### Close service +```shell +# Front desk service startup mode like: +docker compose -f docker-compose-xxx.yaml up # To close the service, please press Ctrl+C. +# Backend service startup mode like: +docker compose -f docker-compose-xxx.yaml up -d # To close the service, please execute the following command. +docker compose -f docker-compose-xxx.yaml down ``` -## Run With Openai API On Windows WSL or Linux，CPU Only +## Offline Use +If you want to use QAnything offline, you need to deploy the local large model (recommended to use ollama) on the offline machine in advance, and then you can start the service using the following command. +### Install offline for windows +```shell +# Download the docker image on a networked machine +docker pull quay.io/coreos/etcd:v3.5.5 +docker pull minio/minio:RELEASE.2023-03-20T20-16-18Z +docker pull milvusdb/milvus:v2.4.8 +docker pull mysql:8.4 +docker pull xixihahaliu01/qanything-win:v1.5.1 # From [https://github.com/netease-youdao/QAnything/blob/master/docker-compose-windows.yaml#L103] Get the latest version number. -Fill in the API key in scripts/run_for_openai_api_with_cpu_in_Linux_or_WSL.sh +# pack image +docker save quay.io/coreos/etcd:v3.5.5 minio/minio:RELEASE.2023-03-20T20-16-18Z milvusdb/milvus:v2.4.8 mysql:8.4 xixihahaliu01/qanything-win:v1.5.1 -o qanything_offline.tar -```bash -bash scripts/run_for_openai_api_with_cpu_in_Linux_or_WSL.sh -``` +# download QAnything code +wget https://github.com/netease-youdao/QAnything/archive/refs/heads/master.zip -## Run With Openai API On Windows WSL or Linux，GPU Only +# Copy the image qanything_offline.tar and the code qany-master.zip to the offline machine +cp QAnything-master.zip qanything_offline.tar /path/to/your/offline/machine -Fill in the API key in scripts/run_for_openai_api_with_gpu_in_Linux_or_WSL.sh +# Load image on offline machine +docker load -i qanything_offline.tar -```bash -bash scripts/run_for_openai_api_with_gpu_in_Linux_or_WSL.sh -``` +# Unzip the code and run it +unzip QAnything-master.zip +cd QAnything-master +docker compose -f docker-compose-win.yaml up +``` +Similarly for other systems, just replace the corresponding image of the system, such as replacing mac with docker-compose-mac.yaml, and linux with docker-compose-linux.yaml. -## Run With Openai API On M1 Mac -Fill in the API key in scripts/run_for_openai_api_in_M1_mac.sh +## FAQ +[FAQ](FAQ_zh.md) -```bash -bash scripts/run_for_openai_api_in_M1_mac.sh -``` -## Run With ollama API On M1 Mac +## Contributing +We appreciate your interest in contributing to our project. Whether you're fixing a bug, improving an existing feature, or adding something completely new, your contributions are welcome! +### Thanks to all contributors for their efforts + + + -```bash -bash scripts/run_for_ollama_api_in_M1_mac.sh -``` -## Run With 3B LLM (MiniChat-2-3B-INT8-GGUF) On M1 Mac -```bash -bash scripts/run_for_3B_in_M1_mac.sh -``` +# 🛣️ Roadmap & Feedback +🔎 To learn about QAnything's future plans and progress, please see here: [QAnything Roadmap](https://qanything.canny.io/) -## Run in OpenCloud -OpenCloud Need Run in Docker Container, Please Install Docker First: -Docker version >= 20.10.5 and docker-compose version >= 2.23.3 -```bash -docker-compose up -d -docker attach qanything-container -# Choose one of the 4 commands below to run: -bash scripts/run_for_3B_in_Linux_or_WSL.sh -bash scripts/run_for_7B_in_Linux_or_WSL.sh -bash scripts/run_for_openai_api_with_cpu_in_Linux_or_WSL.sh -bash scripts/run_for_openai_api_with_gpu_in_Linux_or_WSL.sh -``` +🤬To provide feedback to QAnything, please see here: [QAnything Feedbak](https://qanything.canny.io/feature-requests) +# Community & Support -## USE WITH WEB UI -Open http://0.0.0.0:8777/qanything/ in the browser to use the UI interface, +## Discord +Welcome to the QAnything [Discord](https://discord.gg/5uNpPsEJz8) community -or open http://{your host ip}:8777/qanything/ in the browser to use the UI interface -*Note that the trailing slash cannot be omitted, otherwise a 404 error will occur* +## WeChat -## API Documents -API Documentation is available at [API.md](./docs/API.md). -### Test Example -```python -python scripts/new_knowledge_base.py # print kb_id -python scripts/upload_files.py scripts/weixiaobao.jpg # print file_id -python scripts/list_files.py # print files status -python scripts/stream_file.py # print llm res -``` +Welcome to follow QAnything WeChat Official Account to get the latest information. + + + + +Welcome to scan the code to join the QAnything discussion group. + + + + +## Email +If you need to contact our team privately, please reach out to us via the following email: + +qanything@rd.netease.com + +## GitHub issues & discussions +Reach out to the maintainer at one of the following places: + +- [Github issues](https://github.com/netease-youdao/QAnything/issues) +- [Github discussions](https://github.com/netease-youdao/QAnything/discussions) +- Contact options listed on [this GitHub profile](https://github.com/netease-youdao) + + + + Logo + + + +# Star History + +[![Star History Chart](https://api.star-history.com/svg?repos=netease-youdao/QAnything,netease-youdao/BCEmbedding&type=Date)](https://star-history.com/#netease-youdao/QAnything&netease-youdao/BCEmbedding&Date) + + + +# License + +`QAnything` is licensed under [AGPL-3.0 License](./LICENSE) + +# Acknowledgments +`QAnything` adopts dependencies from the following: +- Thanks to our [BCEmbedding](https://github.com/netease-youdao/BCEmbedding) for the excellent embedding and rerank model. +- Thanks to [Qwen](https://github.com/QwenLM/Qwen) for strong base language models. +- Thanks to [Triton Inference Server](https://github.com/triton-inference-server/server) for providing great open source inference serving. +- Thanks to [FastChat](https://github.com/lm-sys/FastChat) for providing a fully OpenAI-compatible API server. +- Thanks to [FasterTransformer](https://github.com/NVIDIA/FasterTransformer) and [vllm](https://github.com/vllm-project/vllm) for highly optimized LLM inference backend. +- Thanks to [Langchain](https://github.com/langchain-ai/langchain) for the wonderful llm application framework. +- Thanks to [Langchain-Chatchat](https://github.com/chatchat-space/Langchain-Chatchat) for the inspiration provided on local knowledge base Q&A. +- Thanks to [Milvus](https://github.com/milvus-io/milvus) for the excellent semantic search library. +- Thanks to [PaddleOCR](https://github.com/PaddlePaddle/PaddleOCR) for its ease-to-use OCR library. +- Thanks to [Sanic](https://github.com/sanic-org/sanic) for the powerful web service framework. +- Thanks to [RAGFlow](https://github.com/infiniflow/ragflow) for providing some ideas for document parsing. \ No newline at end of file diff --git a/README_zh.md b/README_zh.md index 0d39814c..fe642ba1 100644 --- a/README_zh.md +++ b/README_zh.md @@ -1,121 +1,507 @@ -# 纯Python环境安装教程 -

+

+ + + + Logo + + +# **Q**uestion and **A**nswer based on **Anything** + +

English | 简体中文

- +
+ +
+ + +     + +     + + +     + +     + +     + +
+ +
+目 录 + +- [重要更新](#-重要更新) +- [什么是QAnything](#什么是qanything) + - [特点](#特点) + - [架构](#架构) +- [最近更新 ](#-最近更新) +- [开始之前](#开始之前) +- [开始](#开始) + - [最新特性表](#最新特性表) + - [V2.0.0版本新增细节优化：](#v200版本新增细节优化) + - [新旧解析效果对比](#新旧解析效果对比) + - [安装](#安装) + - [必要条件](#必要条件) + - [step1: 下载本项目](#step1-下载本项目) + - [step2: 进入项目根目录执行启动命令](#step2-进入项目根目录执行启动命令) + - [step3: 开始体验](#step3-开始体验) + - [API](#API) + - [DEBUG](#DEBUG) + - [关闭服务](#关闭服务) + - [离线使用](#离线使用) + - [常见问题](#常见问题) + - [贡献代码](#贡献代码) + - [感谢以下所有贡献者](#感谢以下所有贡献者) + - [特别鸣谢](#特别鸣谢) + - [商务问题联系方式：](#商务问题联系方式) +- [路线图&反馈](#-路线图--反馈) +- [交流&支持](#交流--支持) + - [微信](#微信) + - [邮箱](#邮箱) +- [协议](#协议) +- [Acknowledgements](#acknowledgements) + +
+ +# 🚀 重要更新

重要的事情说三遍！

-# [2024-05-17:最新的安装和使用文档](https://github.com/netease-youdao/QAnything/blob/master/QAnything%E4%BD%BF%E7%94%A8%E8%AF%B4%E6%98%8E.md) -# [2024-05-17:最新的安装和使用文档](https://github.com/netease-youdao/QAnything/blob/master/QAnything%E4%BD%BF%E7%94%A8%E8%AF%B4%E6%98%8E.md) -# [2024-05-17:最新的安装和使用文档](https://github.com/netease-youdao/QAnything/blob/master/QAnything%E4%BD%BF%E7%94%A8%E8%AF%B4%E6%98%8E.md) +# [2024-08-23: QAnything更新V2.0版本] +# [2024-08-23: QAnything更新V2.0版本] +# [2024-08-23: QAnything更新V2.0版本] + +

+ +* 此次更新带来了使用门槛，资源占用，检索效果，问答效果，解析效果，前端效果，服务架构，使用方式等多方面的改进。 +* 同时合并了旧有的docker版和python版两个版本，改为全新的统一版本，采用docker compose单行命令一键启动，开箱即用。 + +

+ + +## 欢迎贡献代码 +我们感谢您对贡献到我们项目的兴趣。无论您是修复错误、改进现有功能还是添加全新内容，我们都欢迎您的贡献！ + +### 感谢以下所有贡献者 + + + + +### 特别鸣谢！ +

请注意：我们的贡献者名单是自动更新的，因此您的贡献可能不会立即显示在此列表中。

+

特别鸣谢！：@ikun-moxiaofei

+ ## 商务问题联系方式： ### 010-82558901 ![](docs/images/business.jpeg) +# 什么是QAnything？ +**QAnything** (**Q**uestion and **A**nswer based on **Anything**) 是致力于支持任意格式文件或数据库的本地知识库问答系统，可断网安装使用。 + +您的任何格式的本地文件都可以往里扔，即可获得准确、快速、靠谱的问答体验。 + +目前已支持格式: **PDF(pdf)**，**Word(docx)**，**PPT(pptx)**，**XLS(xlsx)**，**Markdown(md)**，**电子邮件(eml)**，**TXT(txt)**，**图片(jpg，jpeg，png)**，**CSV(csv)**，**网页链接(html)**，更多格式，敬请期待... + +## 特点 +- 数据安全，支持全程拔网线安装使用。 +- 支持文件类型多，解析成功率高，支持跨语种问答，中英文问答随意切换，无所谓文件是什么语种。 +- 支持海量数据问答，两阶段向量排序，解决了大规模数据检索退化的问题，数据越多，效果越好，不限制上传文件数量，检索速度快。 +- 硬件友好，默认在纯CPU环境下运行，且win，mac，linux多端支持，除docker外无依赖项。 +- 易用性，无需繁琐的配置，一键安装部署，开箱即用，各依赖组件（pdf解析，ocr，embed，rerank等）完全独立，支持自由替换。 +- 支持类似Kimi的快速开始模式，无文件聊天模式，仅检索模式，自定义Bot模式。 + +## 架构 +
+qanything_system +
+ +### 为什么是两阶段检索? +知识库数据量大的场景下两阶段优势非常明显，如果只用一阶段embedding检索，随着数据量增大会出现检索退化的问题，如下图中绿线所示，二阶段rerank重排后能实现准确率稳定增长，即**数据越多，效果越好**。 + +
+two stage retrievaal +
+ +QAnything使用的检索组件[BCEmbedding](https://github.com/netease-youdao/BCEmbedding)有非常强悍的双语和跨语种能力，能消除语义检索里面的中英语言之间的差异，从而实现： +- **强大的双语和跨语种语义表征能力【基于MTEB的语义表征评测指标】。** +- **基于LlamaIndex的RAG评测，表现SOTA【基于LlamaIndex的RAG评测指标】。** + + +### 一阶段检索（embedding） +| 模型名称 | Retrieval | STS | PairClassification | Classification | Reranking | Clustering | 平均 | +|:-------------------------------|:--------:|:--------:|:--------:|:--------:|:--------:|:--------:|:--------:| +| bge-base-en-v1.5 | 37.14 | 55.06 | 75.45 | 59.73 | 43.05 | 37.74 | 47.20 | +| bge-base-zh-v1.5 | 47.60 | 63.72 | 77.40 | 63.38 | 54.85 | 32.56 | 53.60 | +| bge-large-en-v1.5 | 37.15 | 54.09 | 75.00 | 59.24 | 42.68 | 37.32 | 46.82 | +| bge-large-zh-v1.5 | 47.54 | 64.73 | **79.14** | 64.19 | 55.88 | 33.26 | 54.21 | +| jina-embeddings-v2-base-en | 31.58 | 54.28 | 74.84 | 58.42 | 41.16 | 34.67 | 44.29 | +| m3e-base | 46.29 | 63.93 | 71.84 | 64.08 | 52.38 | 37.84 | 53.54 | +| m3e-large | 34.85 | 59.74 | 67.69 | 60.07 | 48.99 | 31.62 | 46.78 | +| ***bce-embedding-base_v1*** | **57.60** | **65.73** | 74.96 | **69.00** | **57.29** | **38.95** | ***59.43*** | + +- 更详细的评测结果详见[Embedding模型指标汇总](https://github.com/netease-youdao/BCEmbedding/blob/master/Docs/EvaluationSummary/embedding_eval_summary.md)。 + +### 二阶段检索（rerank） +| 模型名称 | Reranking | 平均 | +|:-------------------------------|:--------:|:--------:| +| bge-reranker-base | 57.78 | 57.78 | +| bge-reranker-large | 59.69 | 59.69 | +| ***bce-reranker-base_v1*** | **60.06** | ***60.06*** | + +- 更详细的评测结果详见[Reranker模型指标汇总](https://github.com/netease-youdao/BCEmbedding/blob/master/Docs/EvaluationSummary/reranker_eval_summary.md) + +### 基于LlamaIndex的RAG评测（embedding and rerank） + + + +***NOTE:*** + +- 在WithoutReranker列中，我们的bce-embedding-base_v1模型优于所有其他embedding模型。 +- 在固定embedding模型的情况下，我们的bce-reranker-base_v1模型达到了最佳表现。 +- **bce-embedding-base_v1和bce-reranker-base_v1的组合是SOTA。** +- 如果想单独使用embedding和rerank请参阅：[BCEmbedding](https://github.com/netease-youdao/BCEmbedding) + +### LLM + +v2.0版本QAnything不再提供本地大模型，用户可自行接入OpenAI接口兼容的其他开源大模型服务，如ollama, DashScope等。 + +# 🚀 最近更新 +- ***2024-08-23***: **支持快速开始、前端配置参数、在线预览和编辑chunk块，极大优化项目架构和启动方式，极大优化解析和检索效果。** - 详见👉 [v2.0.0](https://github.com/netease-youdao/QAnything/releases/tag/v2.0.0) +- ***2024-05-20***: **支持与OpenAI API兼容的其他LLM服务，并提供优化后的PDF解析器。** - 详见👉 [v1.4.1](https://github.com/netease-youdao/QAnything/releases/tag/v1.4.1) +- ***2024-04-26***: **支持联网检索、FAQ、自定义BOT、文件溯源等。** - 详见👉 [v1.4.0](https://github.com/netease-youdao/QAnything/releases/tag/v1.4.0-python) +- ***2024-04-03***: **支持在纯Python环境中安装；支持混合检索。** - 详见👉 [v1.3.0](https://github.com/netease-youdao/QAnything/releases/tag/v1.3.0) +- ***2024-01-29***: **支持自定义大模型，包括OpenAI API和其他开源大模型，GPU需求最低降至GTX 1050Ti，极大提升部署，调试等方面的用户体验** - 详见👉 [v1.2.0](https://github.com/netease-youdao/QAnything/releases/tag/v1.2.0) +- ***2024-01-23***: **默认开启rerank，修复在windows上启动时存在的各类问题** - 详见👉 [v1.1.1](https://github.com/netease-youdao/QAnything/releases/tag/v1.1.1) +- ***2024-01-18***: **支持一键启动，支持windows部署，提升pdf，xlsx，html解析效果** - 详见👉 [v1.1.0](https://github.com/netease-youdao/QAnything/releases/tag/v1.1.0) + +# 开始之前 +**在GitHub上加星，即可立即收到新版本的通知！** +![star_us](https://github.com/netease-youdao/QAnything/assets/29041332/fd5e5926-b9b2-4675-9f60-6cdcaca18e14) +* [🏄 在线试用QAnything](https://qanything.ai) +* [📚 在线试用有道速读](https://read.youdao.com) +* [🛠️ 想只使用BCEmbedding(embedding & rerank)](https://github.com/netease-youdao/BCEmbedding) +* [📖 FAQ](FAQ_zh.md) +* [👂️需求反馈 | 让我听见你的声音](https://qanything.canny.io/feature-requests) + + +# 开始 +## 最新特性表 + +| 特性 | python （v1.4.2） | docker （v1.2.2） | 全新QAnything v2.0.0 | 说明 | +|---------------------------------|-----------------|-----------------| ---------------- |---------------------------------------------------------------------------| +| 详细安装文档 | ✅ | ✅ | ✅ | | +| API支持 | ✅ | ✅ | ✅ | | +| 生产环境（小型生产环境） | ❌ | ✅ | ✅ | | +| 离线使用 | ❌ | ✅ | ✅ | | +| 支持多并发 | ❌ | ✅ | ✅ | | +| 支持多卡推理 | ❌ | ✅ | ❌ | v2.0.0版本不再提供默认本地LLM，一律通过openai接口接入，用户可自行通过ollama等工具部署本地LLM | +| 支持Mac（M系列芯片） | ✅ | ❌ | ✅ | | +| 支持Linux | ✅ | ✅ | ✅ | python旧版本Linux下默认使用onnxruntime-gpu for cuda12，glibc<2.28时自动切换为onnxruntime | +| 支持windows （**无需WSL**） | ❌ | ❌ | ✅ | python和docker旧版本均要求WSL环境，v2.0.0可直接在非WSL环境下启动 | +| 支持纯CPU环境 | ✅ | ❌ | ✅ | v2.0.0版本Mac，Linux，Win统一不再使用GPU，完全迁移至CPU | +| 支持混合检索（BM25+embedding） | ❌ | ✅ | ✅ | | +| 支持联网检索（**需外网VPN**） | ✅ | ❌ | ✅ | | +| 支持FAQ问答 | ✅ | ❌ | ✅ | | +| 支持自定义机器人（可绑定知识库，可分享） | ✅ | ❌ | ✅ | | +| 支持文件溯源（数据来源可直接点击打开） | ✅ | ❌ | ✅ | | +| 支持问答日志检索 | ✅ | ❌ | ✅ | python和docker旧版本均只能通过API检索，v2.0.0可直接在前端检索 | +| 支持解析语音文件（依赖faster_whisper，解析速度慢） | ✅ | ❌ | ❌ | 依赖whisper，速度慢且占用资源大，暂时去除 | +| 支持OpenCloudOS | ✅ | ❌ | ✅ | | +| 支持与OpenAI接口兼容的其他开源大模型服务(包括ollama) | ✅ | ✅ | ✅ | python和docker旧版本需手动修改api_key，base_url，model等参数，v2.0.0版本均改为前端设置自动保存 | +| pdf（包含表格）解析效果+++ | ✅ | ❌ | ✅ | v1.4.2版本需手动设置，v2.0.0无需手动设置，pdf解析效果和性能均有提升 | +| 用户自定义embed，rerank配置（实验性：提升速度） | ✅ | ❌ | ✅ | v1.4.2需手动设置，v2.0.0默认使用最佳配置 | +| 其他文件类型解析效果+++ | ❌ | ❌ | ✅ | v2.0.0版本提升url，md，xlsx，docx等解析效果 | +| 支持独立服务调用 | ❌ | ❌ | ✅ | v2.0.0版本独立依赖服务，包括embed，rerank，ocr，pdf解析服务等，可独立调用（http） | +| 支持快速开始模式 | ❌ | ❌ | ✅ | 快速开始：无需创建知识库，支持文件即传即问，支持无文件问答 | +| 支持仅检索模式 | ❌ | ❌ | ✅ | 仅返回检索结果，不调用大模型进行问答 | +| 支持解析结果chunks内容可视化，手动编辑 | ❌ | ❌ | ✅ | v2.0.0版本支持手动编辑chunks内容，实时生效 | +| pdf解析支持图片,支持回答带图 | ❌ | ❌ | ✅ | | + +## V2.0.0版本新增细节优化： + +* 支持前端配置API_BASE，API_KEY，文本分片大小，输出token数量，上下文消息数量等参数 +* 优化Bot角色设定的指令遵循效果，每个Bot可单独配置模型参数 +* 支持创建多个对话窗口，同时保存多份历史问答记录 +* 支持问答记录保存成图片 +* 优化上传文件逻辑，解析文件与问答请求独立，上传文件不再影响问答 +* 优化镜像大小，旧版本镜像压缩后大小为18.94GB->新版镜像压缩后大小为4.88GB，降为原有的1/4，提供完整Dockerfile +* 检索优化，chunks新增片段融合与排序，聚合单文档或双文档 +* 检索阶段和问答阶段均嵌入metadata信息，提升检索和问答效果 + +### 各阶段数据展示： +* 知识库所有文件上传进度展示 +* 知识库单个文件上传进度展示，上传各阶段耗时 +* 问答信息统计，包含问答各阶段耗时，tokens消耗，模型信息等 +* 用户信息统计，包含上传文件总数量，总耗时，问答历史记录等（即将发布） + +### 问题修复 +* xlsx表格支持多sheet解析 +* 优化PDF表格漏识别问题 +* 修复部分DOCX文件解析出错问题 +* 优化FAQ匹配逻辑 +* 支持非utf-8编码的txt文件 + + +## 新旧解析效果对比 +* 首先针对在文档中大段表格，尤其是跨页表格的解析方面，2.0版本进行了显著的改进，新版本解析逻辑能够分析表格的结构，包括行和列的布局，并且能够自动识别出表头，将其置于每个切片分割出的表格的顶部。这样的改进防止了在解析长表格时，由于逻辑上的切割而导致的意义上的中断 + +| 原图 | 旧版本解析效果 | 新版本解析效果 | +|:----:|:--------------------------------------------:|:---------------------------------------------------------------------------------------------:| +| ![image.png](docs/assets/17244247170060.png) | ![image.png](docs/assets/17244247170067.png) | ![image.png](docs/assets/17244247170074.png) | + +* 此外，2.0版本在处理文本分栏和跨页布局的情况下也做了优化。它能够识别文本的双栏或多栏布局，并根据人类的阅读习惯来正确划分文本块的顺序。同时，该版本还能够保存文档中的图片，确保在文件解析过程中内容的完整性不会丢失。如在下图中，正确的排列顺序应该为一次排列编号1，2，3的文本为一大段并进行切块，而不是将1，2，3分别切块。 + * 在1.4版本解析结果中，跨页文本 “更高一些” 被分块到了下一文本块中，这对大模型语义理解是不利的，而在2.0版本解析结果中是正确划分的，并且也将穿插在文本段落中的图片解析到对应的chunk语句块中，非正文本文 “图1 鉴别与授权及其支持关系 37” 和 “Cover Story 封面专题” 也成功被过滤 + +| 原图 | 旧版本解析效果 | 新版本解析效果 | +|:----:|:--------------------------------------------:|:---------------------------------------------------------------------------------------------:| +| ![image.png](docs/assets/17244247170088.png) | ![image.png](docs/assets/17244247170101.png)
![image.png](docs/assets/17244247170115.png) | ![image.png](docs/assets/17244247170129.png) | + + +* 2.0版本在针对穿插在文本栏或文本块之间的表格解析也做了相应的优化，原先版本的解析无法识别表格，只会将其以文本段落的格式进行解析，这样不仅会破坏表格的逻辑结构，对大模型而言也是多了一段杂乱无用的文本，会影响大模型回答的准确度；而2.0版本能够识别并解析这些嵌入文本中的表格，从而提高了解析的质量和大模型回答的准确性。 + * 在1.4版本解析结果中，穿插在文本块中的表格被当做普通文本块解析，2.0版本这可以将这种表格“优雅地”解析，不仅提高了解析的质量也增加大模型回答的准确性； + * 此外，2.0版本在处理特定小标题下的文本时，会优先确保这些文本被分割到同一个chunk块中，以维持逻辑上的连贯性。当文本过长，需要进行分割时，2.0版本的解析逻辑会在每个分割后的文本块前重复相同的小标题，以示归属。例如，在示例中，三个文本块都加上了相同的小标题“欧洲会议：机器人的法律地位”（由于文本过长，在原文件截图时未能显示该标题）。这种处理方式有效避免了因文本过长而导致的分割后文本块语义逻辑不连贯的问题。 + +| 原图 | 旧版本解析效果 | 新版本解析效果 | +|:----:|:--------------------------------------------:|:---------------------------------------------------------------------------------------------:| +| ![image.png](docs/assets/17244247170148.png) | ![image.png](docs/assets/17244247170166.png) | ![image.png](docs/assets/17244247171267.png) | + +* 对于包含复杂格式的Excel（.xlsx）文档，2.0版本进行了一系列的优化措施，可以准确地识别和处理行列数据，包括合并单元格和跨行或跨列的文本等优化，具体可以看以下示例。 + * 在1.4版本中，解析Excel文档时可能存在一些限制，特别是对于包含特殊结构或格式的文档，解析结果可能不尽如人意，主要只能识别纯文本部分。这在处理复杂数据和格式时可能导致信息丢失或格式错乱。相比之下，2.0版本在解析能力上有了显著提升，能够更好地处理各种复杂格式的Excel文档，尽管可能还未达到完美，但已经能够解决绝大多数复杂情况。 + +| 原图 | 旧版本解析效果 | 新版本解析效果 | +|:----:|:--------------------------------------------:|:---------------------------------------------------------------------------------------------:| +| ![image.png](docs/assets/17244247170188.png) | ![image.png](docs/assets/17244247170208.png) | ![image.png](docs/assets/17244247170228.png) | + +* 同样，对于简单格式的xlsx文档，2.0版本的解析也做了优化。 + +| 原图 | 旧版本解析效果 | 新版本解析效果 | +|:----:|:--------------------------------------------:|:---------------------------------------------------------------------------------------------:| +| ![image.png](docs/assets/17244247170272.png) | ![image.png](docs/assets/17244247170298.png) | ![image.png](docs/assets/17244247170323.png) | + +* 在最新版本中，我们对URL解析功能也进行了显著改进。以下面的页面为例，旧版本在解析过程中可能会遗漏大量页面信息，并且无法有效处理表格、列表等较为复杂的页面元素。然而，新版本已经针对这些问题进行了优化，能够更准确地解析这些内容。 + +| 原图 | 旧版本解析效果 | 新版本解析效果 | +|:----:|:--------------------------------------------:|:---------------------------------------------------------------------------------------------:| +| ![image.png](docs/assets/17244247170443.png) | ![image.png](docs/assets/17244247170478.png)
![image.png](docs/assets/17244247170512.png) | ![image.png](docs/assets/17244247170546.png) | + +* 除此之外，对于绝大多数的文件，2.0版本也做了对应的优化，包括但不限于以下几点 + * 改进了chunk块的切割逻辑，避免了由于文档中的空白行或段落导致的语义块过短或逻辑中断，确保了文本块的连贯性和完整性 + * 新版本能够更准确地识别文档中的小标题，并根据这些小标题来定位和组织对应的文本块，有助于优化解析效果，使得解析的结构更加清晰，信息层次更加分明 + * 解析结果对比如下，1.4版本解析逻辑将文档分为了10个chunk块，而2.0版本解析后只有4个chunk块，更加合理且较少的chunk块极大的提高了内容的连贯性和完整性，有助于减少因切割不当而导致的语义断裂或逻辑混乱，从而提高了整体的解析效果和模型回答的效果 + +| 原图 | 旧版本解析效果 | 新版本解析效果 | +|:----:|:--------------------------------------------:|:---------------------------------------------------------------------------------------------:| +| ![image.png](docs/assets/17244247170352.png) | ![image.png](docs/assets/17244247170380.png) | ![image.png](docs/assets/17244247170406.png) | + +### 综上所述，2.0版本解析相较于1.4版本的解析优化了很多方面，包括但不限于 + +1. 通过更合理的分块长度，减少了因段落过小或段落不完整而导致的语义和逻辑上的丢失。 +2. 改进了对分栏文本的识别能力，能够智能判断阅读顺序，即使是跨页的段落也能做出正确处理。 +3. 新版本能够识别并保存文本段落中的图片和表格，确保不会遗漏任何重要的文本信息。 +4. 优化表格解析，包括超出chunk块限制的长表格和复杂结构的xlsx文件的解析和存储 +5. 根据识别文档中的小标题，定位和组织对应的文本块，使得解析的结构更加清晰，信息层次更加分明 +6. 优化对于网页url的解析结果，转为.md格式 +7. 支持更多编码格式的txt文件和docx文件 + + +## 安装 +### 必要条件 +| **系统** | **依赖** | **要求** | **说明** | +|-----------|-------------------------|-------------------|----------------------------------------------------------------------------------------| +| | RAM Memory | >= 20GB | | +| Linux/Mac | Docker version | >= 20.10.5 | [Docker install](https://docs.docker.com/engine/install/) | +| Linux/Mac | docker compose version | >= 2.23.3 | [docker compose install](https://docs.docker.com/compose/install/) | +| Windows | Docker Desktop | >= 4.26.1（131620） | [Docker Desktop for Windows](https://docs.docker.com/desktop/install/windows-install/) | + + +### step1: 下载本项目 +```shell +git clone https://github.com/netease-youdao/QAnything.git +``` +### step2: 进入项目根目录执行启动命令 +* 执行 docker compose 启动命令 + +```shell +cd QAnything +# 在 Linux 上启动 +docker compose -f docker-compose-linux.yaml up +# 在 Mac 上启动 +docker compose -f docker-compose-mac.yaml up +# 在 Windows 上启动 +docker compose -f docker-compose-win.yaml up +``` -## 安装 +（注意）如果启动失败，可以尝试将 `docker compose`改为 `docker-compose`。 + + +### step3: 开始体验 + +#### 前端页面 +运行成功后，即可在浏览器输入以下地址进行体验。 + +- 前端地址: http://localhost:8777/qanything/ + +### API +如果想要访问API接口，请参考下面的地址: +- API address: http://localhost:8777/api/ +- For detailed API documentation, please refer to [QAnything API 文档](docs/API.md) + +### DEBUG +##### 如果想要查看服务启动相关日志，请查看`QAnything/logs/debug_logs`目录下的日志文件。 +- **debug.log** + - 用户请求处理日志 +- **main_server.log** + - 后端服务运行日志 +- **rerank_server.log** + - rerank服务运行日志 +- **ocr_server.log** + - OCR服务运行日志 +- **embedding_server.log** + - 向量化服务运行日志 +- **rerank_server.log** + - 检索增强服务运行日志 +- **insert_files_server.log** + - 文件上传服务运行日志 +- **pdf_parser_server.log** + - pdf解析服务运行日志 +##### 详细上传文件日志请查看`QAnything/logs/insert_logs`目录下的日志文件。 +##### 详细问答日志请查看`QAnything/logs/qa_logs`目录下的日志文件。 +##### 详细embedding日志请查看`QAnything/logs/embed_logs`目录下的日志文件。 +##### 详细rerank日志请查看`QAnything/logs/rerank_logs`目录下的日志文件。 + +### 关闭服务 +```shell +# 前台启动服务方式如下： +docker compose -f docker-compose-xxx.yaml up # 关闭服务请按Ctrl+C +# 后台启动服务方式如下： +docker compose -f docker-compose-xxx.yaml up -d # 关闭服务请执行以下命令 +docker compose -f docker-compose-xxx.yaml down +``` -要求: +## 离线使用 +如果您想要离线使用QAnything，需要在断网机器提前部署本地的大模型（推荐使用ollama），随后可以使用以下命令启动服务。 +### windows离线使用 +```shell +# 先在联网机器上下载docker镜像 +docker pull quay.io/coreos/etcd:v3.5.5 +docker pull minio/minio:RELEASE.2023-03-20T20-16-18Z +docker pull milvusdb/milvus:v2.4.8 +docker pull mysql:8.4 +docker pull xixihahaliu01/qanything-win:v1.5.1 # 从 [https://github.com/netease-youdao/QAnything/blob/master/docker-compose-windows.yaml#L103] 中获取最新镜像版本号。 + +# 打包镜像 +docker save quay.io/coreos/etcd:v3.5.5 minio/minio:RELEASE.2023-03-20T20-16-18Z milvusdb/milvus:v2.4.8 mysql:8.4 xixihahaliu01/qanything-win:v1.5.1 -o qanything_offline.tar + +# 下载QAnything代码 +wget https://github.com/netease-youdao/QAnything/archive/refs/heads/master.zip + +# 把镜像qanything_offline.tar和代码QAnything-master.zip拷贝到断网机器上 +cp QAnything-master.zip qanything_offline.tar /path/to/your/offline/machine + +# 在断网机器上加载镜像 +docker load -i qanything_offline.tar + +# 解压代码，运行 +unzip QAnything-master.zip +cd QAnything-master +docker compose -f docker-compose-win.yaml up +``` - - Python 3.10+ (建议使用aoaconda3来管理Python环境) - - System - - Linux: glibc 2.28+ and Cuda 12.0+ (如果使用GPU) - - Windows: WSL with Ubuntu 20.04+ and GEFORCE EXPERIENCE 535.104+ (如果使用GPU) - - MacOS: M1/M2/M3 Mac with Xcode 15.0+ +### Linux离线使用 +```shell +# 先在联网机器上下载docker镜像 +docker pull quay.io/coreos/etcd:v3.5.5 +docker pull minio/minio:RELEASE.2023-03-20T20-16-18Z +docker pull milvusdb/milvus:v2.4.8 +docker pull mysql:8.4 +docker pull xixihahaliu01/qanything-linux:v1.5.1 # 从 [https://github.com/netease-youdao/qanything/blob/master/docker-compose-linux.yaml#L104] 中获取最新镜像版本号。 -请创建一个干净的Python虚拟环境，以避免潜在冲突（推荐使用Anaconda3）。 +# 打包镜像 +docker save quay.io/coreos/etcd:v3.5.5 minio/minio:RELEASE.2023-03-20T20-16-18Z milvusdb/milvus:v2.4.8 mysql:8.4 xixihahaliu01/qanything-linux:v1.5.1 -o qanything_offline.tar -安装软件包，请运行: -```bash -conda create -n qanything-python python=3.10 -conda activate qanything-python -git clone -b qanything-python https://github.com/netease-youdao/QAnything.git -cd QAnything -pip install -e . -``` +# 下载QAnything代码 +wget https://github.com/netease-youdao/QAnything/archive/refs/heads/master.zip -## PDF解析++ +# 把镜像qanything_offline.tar和代码QAnything-master.zip拷贝到断网机器上 +cp QAnything-master.zip qanything_offline.tar /path/to/your/offline/machine -如果你想使用更强大的pdf解析功能，请在modelscope下载相关的[解析模型](https://www.modelscope.cn/models/netease-youdao/QAnything-pdf-parser/files),并将其放置到根目录的qanything_kernel/utils/loader/pdf_to_markdown/checkpoints/下 +# 在断网机器上加载镜像 +docker load -i qanything_offline.tar -## 在Windows WSL或Linux环境下运行3B大模型（MiniChat-2-3B）要求显存>=10GB -```bash -bash scripts/run_for_3B_in_Linux_or_WSL.sh +# 解压代码，运行 +unzip QAnything-master.zip +cd QAnything-master +docker compose -f docker-compose-linux.yaml up ``` -## 在Windows WSL或Linux环境下运行7B大模型（自研Qwen-7B-QAnything）要求显存>=24GB -```bash -bash scripts/run_for_7B_in_Linux_or_WSL.sh -``` +## 常见问题 +[常见问题](FAQ_zh.md) -## 在Windows WSL或Linux环境下运行Openai API，仅使用CPU +## 贡献代码 +我们感谢您对贡献到我们项目的兴趣。无论您是修复错误、改进现有功能还是添加全新内容，我们都欢迎您的贡献！ -在scripts/run_for_openai_api_with_cpu_in_Linux_or_WSL.sh中补充api-key等参数 +### 感谢以下所有贡献者 + + + -```bash -bash scripts/run_for_openai_api_with_cpu_in_Linux_or_WSL.sh -``` +# 🛣️ 路线图 & 反馈 +🔎 想了解QAnything的未来规划和进展，请看这里： [QAnything Roadmap](https://qanything.canny.io/) -## 在Windows WSL或Linux环境下运行Openai API，使用GPU +🤬 想要给QAnything提交反馈，请看这里(可以给每个功能需求投票哦): [QAnything Feedbak](https://qanything.canny.io/feature-requests) -在scripts/run_for_openai_api_with_gpu_in_Linux_or_WSL.sh中补充api-key等参数 +# 交流 & 支持 -```bash -bash scripts/run_for_openai_api_with_gpu_in_Linux_or_WSL.sh -``` +## Discord +欢迎加入QAnything [Discord](https://discord.gg/5uNpPsEJz8) 社区！ -## 在M1Mac环境下使用Openai API -在scripts/run_for_openai_api_in_M1_mac.sh中补充api-key等参数 -```bash -bash scripts/run_for_openai_api_in_M1_mac.sh -``` +## 微信 +欢迎关注微信公众号，获取最新QAnything信息 -## 在M1Mac环境下使用Ollama API + -```bash -bash scripts/run_for_ollama_api_in_M1_mac.sh -``` +欢迎扫码进入QAnything交流群 -## 在M1Mac环境下使用3B LLM（(MiniChat-2-3B-INT8-GGUF） + -```bash -bash scripts/run_for_3B_in_M1_mac.sh -``` +## 邮箱 +如果你需要私信我们团队，请通过下面的邮箱联系我们： -## 在OpenCloud操作系统中运行 -OpenCloud 需要在 Docker 容器中运行，请先安装 Docker： -Docker 版本 >= 20.10.5 且 docker-compose 版本 >= 2.23.3 -```bash -docker-compose up -d -docker attach qanything-container -# 选择以下4个命令之一来运行： -bash scripts/run_for_3B_in_Linux_or_WSL.sh -bash scripts/run_for_7B_in_Linux_or_WSL.sh -bash scripts/run_for_openai_api_with_cpu_in_Linux_or_WSL.sh -bash scripts/run_for_openai_api_with_gpu_in_Linux_or_WSL.sh -``` +qanything@rd.netease.com -## 访问前端页面 -在浏览器中打开http://0.0.0.0:8777/qanything/ +## GitHub issues & discussions +有任何公开的问题，欢迎提交issues，或者在discussions区讨论 +- [Github issues](https://github.com/netease-youdao/QAnything/issues) +- [Github discussions](https://github.com/netease-youdao/QAnything/discussions) -或者打开http://{主机ip地址}:8777/qanything/ + + + Logo + -即可使用UI界面 +# Star History -注意末尾的斜杠不可省略，否则会出现404错误 +[![Star History Chart](https://api.star-history.com/svg?repos=netease-youdao/QAnything,netease-youdao/BCEmbedding&type=Date)](https://star-history.com/#netease-youdao/QAnything&netease-youdao/BCEmbedding&Date) -## API 文档 -[API.md](./docs/API.md) +# 协议 -### API访问示例 -```python -python scripts/new_knowledge_base.py # print kb_id -python scripts/upload_files.py scripts/weixiaobao.jpg # print file_id -python scripts/list_files.py # print files status -python scripts/stream_file.py # print llm res -``` +`QAnything` 依照 [AGPL-3.0](./LICENSE)开源。 + +# Acknowledgements +- [BCEmbedding](https://github.com/netease-youdao/BCEmbedding) +- [Qwen](https://github.com/QwenLM/Qwen) +- [Triton Inference Server](https://github.com/triton-inference-server/server) +- [vllm](https://github.com/vllm-project/vllm) +- [FastChat](https://github.com/lm-sys/FastChat) +- [FasterTransformer](https://github.com/NVIDIA/FasterTransformer) +- [Langchain](https://github.com/langchain-ai/langchain) +- [Langchain-Chatchat](https://github.com/chatchat-space/Langchain-Chatchat) +- [Milvus](https://github.com/milvus-io/milvus) +- [PaddleOCR](https://github.com/PaddlePaddle/PaddleOCR) +- [Sanic](https://github.com/sanic-org/sanic) +- [RAGFlow](https://github.com/infiniflow/ragflow) \ No newline at end of file diff --git a/docs/API.md b/docs/API.md index 4be47004..b8c08411 100644 --- a/docs/API.md +++ b/docs/API.md @@ -89,6 +89,21 @@ - [检索qa日志请求参数（Body）](#检索qa日志请求参数body) - [检索qa日志请求示例](#检索qa日志请求示例) - [检索qa日志响应示例](#检索qa日志响应示例) + - [获取文档完整解析内容（POST）](#获取文档完整解析内容post) + - [URL：http://{your_host}:8777/api/local_doc_qa/get_doc_completed](#urlhttpyour_host8777apilocal_doc_qaget_doc_completed) + - [获取文档完整解析内容请求参数（Body）](#获取文档完整解析内容请求参数body) + - [获取文档完整解析内容请求示例](#获取文档完整解析内容请求示例) + - [获取文档完整解析内容响应示例](#获取文档完整解析内容响应示例) + - [更新chunks（POST）](#更新chunkspost) + - [URL：http://{your_host}:8777/api/local_doc_qa/update_chunks](#urlhttpyour_host8777apilocal_doc_qaupdate_chunks) + - [更新chunks请求参数（Body）](#更新chunks请求参数body) + - [更新chunks请求示例](#更新chunks请求示例) + - [更新chunks响应示例](#更新chunks响应示例) + - [获取文件base64（POST）](#获取文件base64post) + - [URL：http://{your_host}:8777/api/local_doc_qa/get_file_base64](#urlhttpyour_host8777apilocal_doc_qaget_file_base64) + - [获取文件base64请求参数（Body）](#获取文件base64请求参数body) + - [获取文件base64请求示例](#获取文件base64请求示例) + - [获取文件base64响应示例](#获取文件base64响应示例) ##

全局参数

@@ -106,10 +121,12 @@ user_id 需要满足： 以字母开头，只允许包含字母，数字或下 ###

新建知识库请求参数（Body）

-| 参数名 | 示例参数值 | 是否必填 | 参数类型 | 描述说明 | -| ------- | ---------- | -------- | -------- | --------------------------------------- | -| user_id | "zzp" | 是 | String | 用户 id （如需使用前端填 zzp 不要更换） | -| kb_name | "kb_test" | 是 | String | 知识库名称 （可以随意指定） | +| 参数名 | 示例参数值 | 是否必填 | 参数类型 | 描述说明 | +| ------- | ----------------------------------------------- | ---- | ------ | ------------------------ | +| user_id | "zzp" | 是 | String | 用户 id （如需使用前端填 zzp 不要更换） | +| kb_name | "kb_test" | 是 | String | 知识库名称 （可以随意指定） | +| kb_id | "KB698428590b1e45108b63b943fac16559_240430_FAQ" | 否 | String | 指定知识库id，通常用于生成FAQ知识库 | +| quick | False | 否 | Bool | 是否是快速开始模式创建的知识库，默认为False | ###

新建知识库请求示例

@@ -154,12 +171,13 @@ Content-Type: multipart/form-data ###

上传文件请求参数（Body）

-| 参数名 | 参数值 | 是否必填 | 参数类型 | 描述说明 | -| ------- | ---------------------------------- | -------- | -------- | ------------------------------------------------------------------------------------------------------- | -| files | 文件二进制 | 是 | File | 需要上传的文件，可多选，目前仅支持[md,txt,pdf,jpg,png,jpeg,docx,xlsx,pptx,eml,csv] | -| user_id | zzp | 是 | String | 用户 id | -| kb_id | KBb1dd58e8485443ce81166d24f6febda7 | 是 | String | 知识库 id | -| mode | soft | 否 | String | 上传模式，soft：知识库内存在同名文件时当前文件不再上传，strong：文件名重复的文件强制上传，默认值为 soft | +| 参数名 | 参数值 | 是否必填 | 参数类型 | 描述说明 | +| ---------- | ---------------------------------- | ---- | ------ | ----------------------------------------------------------------- | +| files | 文件二进制 | 是 | File | 需要上传的文件，可多选，目前仅支持[md,txt,pdf,jpg,png,jpeg,docx,xlsx,pptx,eml,csv] | +| user_id | zzp | 是 | String | 用户 id | +| kb_id | KBb1dd58e8485443ce81166d24f6febda7 | 是 | String | 知识库 id | +| mode | soft | 否 | String | 上传模式，soft：知识库内存在同名文件时当前文件不再上传，strong：文件名重复的文件强制上传，默认值为 soft | +| chunk_size | 800 | 否 | Int | 文件切片大小，默认为800tokens | ###

上传文件同步请求示例

@@ -172,7 +190,7 @@ folder_path = "./docx_data" # 文件所在文件夹，注意是文件夹！！ data = { "user_id": "zzp", "kb_id": "KB6dae785cdd5d47a997e890521acbe1c9", - "mode": "soft" + "mode": "soft" } files = [] @@ -284,12 +302,15 @@ if __name__ == '__main__': ###

上传网页文件请求参数（Body）

-| 参数名 | 参数值 | 是否必填 | 参数类型 | 描述说明 | -| ------- | --------------------------------------------------------------- | -------- | -------- | ------------------------------------------------------------------------------------------------------- | -| url | "https://ai.youdao.com/DOCSIRMA/html/trans/api/wbfy/index.html" | 是 | String | html 网址，只支持无需登录的网站 | -| user_id | zzp | 是 | String | 用户 id | -| kb_id | KBb1dd58e8485443ce81166d24f6febda7 | 是 | String | 知识库 id | -| mode | soft | 否 | String | 上传模式，soft：知识库内存在同名文件时当前文件不再上传，strong：文件名重复的文件强制上传，默认值为 soft | +| 参数名 | 参数值 | 是否必填 | 参数类型 | 描述说明 | +| ---------- | --------------------------------------------------------------- | ---- | --------- | ------------------------------------------------------------------------- | +| url | "https://ai.youdao.com/DOCSIRMA/html/trans/api/wbfy/index.html" | 否 | String | html 网址，只支持无需登录且没有反爬虫的网站，不支持递归解析多级页面，解析单个url时使用此参数，解析多个url时可使用urls+titles | +| user_id | zzp | 是 | String | 用户 id | +| kb_id | KBb1dd58e8485443ce81166d24f6febda7 | 是 | String | 知识库 id | +| mode | soft | 否 | String | 上传模式，soft：知识库内存在同名文件时当前文件不再上传，strong：文件名重复的文件强制上传，默认值为 soft | +| urls | ["https://xxx", "https://xxx2"] | 否 | List[str] | 与titles配合使用，同时上传多个url | +| titles | ["页面1"，"页面2"] | 否 | List[str] | 与urls配合使用，同时上传多个url，可为[] | +| chunk_size | 800 | 否 | Int | 文件切片大小，默认为800tokens | ###

上传网页文件请求示例

@@ -303,8 +324,8 @@ headers = { } data = { "user_id": "zzp", - "kb_id": "KBb1dd58e8485443ce81166d24f6febda7", - "url": "https://ai.youdao.com/DOCSIRMA/html/trans/api/wbfy/index.html" + "kb_id": "KBb1dd58e8485443ce81166d24f6febda7", + "url": "https://ai.youdao.com/DOCSIRMA/html/trans/api/wbfy/index.html" } response = requests.post(url, headers=headers, data=json.dumps(data)) @@ -322,7 +343,8 @@ print(response.text) "data": [ { "file_id": "9a49392e633d4c6f87e0af51e8c80a86", - "file_name": "https://ai.youdao.com/DOCSIRMA/html/trans/api/wbfy/index.html", + "file_name": "index.html.web", + "file_url": "https://ai.youdao.com/DOCSIRMA/html/trans/api/wbfy/index.html", "status": "gray", "bytes": 0, // 网页文件无法显示大小 "timestamp": "202401261809" @@ -337,9 +359,9 @@ print(response.text) ###

查看知识库请求参数（Body）

-| 参数名 | 示例参数值 | 是否必填 | 参数类型 | 描述说明 | -| ------- | ---------- | -------- | -------- | -------- | -| user_id | "zzp" | 是 | String | 用户 id | +| 参数名 | 示例参数值 | 是否必填 | 参数类型 | 描述说明 | +| ------- | ----- | ---- | ------ | ----- | +| user_id | "zzp" | 是 | String | 用户 id | ###

查看知识库请求示例

@@ -381,10 +403,13 @@ print(response.text) ###

获取文件列表请求参数（Body）

-| 参数名 | 示例参数值 | 是否必填 | 参数类型 | 描述说明 | -| ------- | ------------------------------------ | -------- | -------- | --------- | -| user_id | "zzp" | 是 | String | 用户 id | -| kb_id | "KBb1dd58e8485443ce81166d24f6febda7" | 是 | String | 知识库 id | +| 参数名 | 示例参数值 | 是否必填 | 参数类型 | 描述说明 | +| ----------- | ------------------------------------ | ---- | ------ | ------------------------------------------ | +| user_id | "zzp" | 是 | String | 用户 id | +| kb_id | "KBb1dd58e8485443ce81166d24f6febda7" | 是 | String | 知识库 id | +| file_id | "m1xd58e8485443ce81166d24f6few33d" | 否 | String | 文件id，指定文件id则只返回单个文件的状态 | +| page_offset | 1 | 否 | Int | 默认值为1，结果太多时可指定分页id（从1开始），与page_limit配合使用 | +| page_limit | 10 | 否 | Int | 默认值为10，结果太多时可指定分页id（从1开始），与page_offset配合使用 | ###

获取文件列表请求示例

@@ -397,8 +422,8 @@ headers = { "Content-Type": "application/json" } data = { - "user_id": "zzp", //用户id - "kb_id": "KBb1dd58e8485443ce81166d24f6febda7" //知识库id + "user_id": "zzp", //用户id + "kb_id": "KBb1dd58e8485443ce81166d24f6febda7" //知识库id } response = requests.post(url, headers=headers, data=json.dumps(data)) @@ -411,37 +436,44 @@ print(response.text) ```json { - "code": 200, //状态码 - "msg": "success", //提示信息 - "data": { - "total": { // 知识库所有文件状态 - "green": 100, - "red": 1, - "gray": 1, - "yellow": 1, - }, - "details": { // 每个文件的具体状态 - { - "file_id": "21a9f13832594b0f936b62a54254543b", //文件id - "file_name": "有道知识库问答产品介绍.pptx", //文件名 - "status": "green", //文件状态（red：入库失败-切分失败，green，成功入库，yellow：入库失败-milvus失败，gray：正在入库） - "bytes": 177925, - "content_length": 3059, // 文件解析后字符长度，用len()计算 - "timestamp": "202401261708", - "msg": "上传成功" - }, - { - "file_id": "333e69374a8d4b9bac54f274291f313e", //文件id - "file_name": "网易有道智云平台产品介绍2023.6.ppt", //文件名 - "status": "green", //文件状态（red：入库失败-切分失败，green，成功入库，yellow：入库失败-milvus失败，gray：正在入库） - "bytes": 12379, - "content_length": 3239, // 文件解析后字符长度，用len()计算 - "timestamp": "202401261708", - "msg": "上传成功" - } - } - // ... - } + "code": 200, //状态码 + "msg": "success", //提示信息 + "data": { + "status_count": { // 知识库所有文件状态 + "green": 100, + "red": 1, + "gray": 1, + "yellow": 1, + }, + "details": { // 每个文件的具体状态 + { + "file_id": "21a9f13832594b0f936b62a54254543b", //文件id + "file_name": "有道知识库问答产品介绍.pptx", //文件名 + "status": "green", //文件状态（red：入库失败-切分失败，green，成功入库，yellow：入库失败-milvus失败，gray：正在入库） + "bytes": 177925, + "chunks_number": 16, + "content_length": 3059, // 文件解析后字符长度，用len()计算 + "file_location": "/workspace/QAnything/QANY_DB/content/zzp__1234/KBf593b232db0e454db43457a6d1c3632c_240625/63700516e25f4c81b9fd9f68b4a6410b/GeoAutoTrackHorizon.pdf" + "timestamp": "202401261708", + "msg": "上传成功" + }, + { + "file_id": "333e69374a8d4b9bac54f274291f313e", //文件id + "file_name": "网易有道智云平台产品介绍2023.6.ppt", //文件名 + "status": "green", //文件状态（red：入库失败-切分失败，green，成功入库，yellow：入库失败-milvus失败，gray：正在入库） + "bytes": 12379, + "chunks_number": 20, + "content_length": 3239, // 文件解析后字符长度，用len()计算 + "file_location": "/workspace/QAnything/QANY_DB/content/zzp__1234/KBf593b232db0e454db43457a6d1c3632c_240625/63700516e25f4c81b9fd9f68b4a6410b/GeoAutoTrackHorizon.pdf" + "timestamp": "202401261708", + "msg": "上传成功" + } + }, + "page_id": 1, + "total": 53, + "total_page": 1, + // ... + } } ``` @@ -451,18 +483,28 @@ print(response.text) ###

问答请求参数（Body）

-| 参数名 | 示例参数值 | 是否必填 | 参数类型 | 描述说明 | -|---------------|------------------------------------------------------------------------------|------|--------------|------------------------| -| user_id | "zzp" | 是 | String | 用户 id | -| kb_ids | ["KBb1dd58e8485443ce81166d24f6febda7", "KB633c69d07a2446b39b3a38e3628b8ada"] | 是 | Array | 知识库 id 的列表，支持多个知识库联合问答 | -| question | "保险单号是多少？" | 是 | String | 知识库 id 的列表，支持多个知识库联合问答 | -| history | [["question1","answer1"],["question2","answer2"]] | 否 | Array[Array] | 历史对话 | -| rerank | True | 否 | Bool | 是否开启 rerank，默认为 True | -| streaming | False | 否 | Bool | 是否开启流式输出，默认为 False | -| networking | False | 否 | Bool | 是否开启联网搜索，默认为 False | -| custom_prompt | "你是一个耐心、友好、专业的编程机器人，能够准确的回答用户的各种编程问题。" | 否 | String | 使用自定义prompt | - - +| 参数名 | 示例参数值 | 是否必填 | 参数类型 | 描述说明 | +| ------------------------ | ---------------------------------------------------------------------------- | ---- | ------------ | -------------------------------------------------------------- | +| user_id | "zzp" | 是 | String | 用户 id | +| kb_ids | ["KBb1dd58e8485443ce81166d24f6febda7", "KB633c69d07a2446b39b3a38e3628b8ada"] | 否 | Array | 知识库 id 的列表，支持多个知识库联合问答，仅在使用bot_id的情况下不需要填 | +| question | "保险单号是多少？" | 是 | String | 知识库 id 的列表，支持多个知识库联合问答 | +| history | [["question1","answer1"],["question2","answer2"]] | 否 | Array[Array] | 历史对话 | +| rerank | True | 否 | Bool | 是否开启 rerank，默认为 True | +| streaming | False | 否 | Bool | 是否开启流式输出，默认为 False | +| networking | False | 否 | Bool | 是否开启联网搜索，默认为 False | +| custom_prompt | "你是一个耐心、友好、专业的编程机器人，能够准确的回答用户的各种编程问题。" | 否 | String | 使用自定义prompt | +| only_need_search_results | False | 否 | Bool | 是否仅返回检索结果跳过大模型回答步骤，默认为False | +| source | paas | 否 | String | 可用于区分请求来源，目前共有paas，saas_bot，saas_qa，分别代表api调用，前端bot问答，前端知识库问答 | +| bot_id | 1234 | 否 | String | 问答若使用bot_id则相关bot设置自动生效，不需要传kb_ids和custom_prompt了 | +| hybrid_search | False | 否 | String | 是否开启混合检索，默认为False | +| max_token | 512 | 是 | int | LLM输出的最大token数，一般最大设置为上下文长度的1/4，比如4K上下文设置范围应该是[0-1024]，默认值为512 | +| api_base | "https://api.openai-proxy.org/v1" | 是 | String | | +| api_key | "sk-xxx" | 是 | String | | +| api_context_length | 16000 | 是 | String | 上下文总长度 | +| model | "gpt-3.5-turbo" | 是 | String | | +| top_p | 0.99 | 否 | Float | | +| temperature | 0.5 | 否 | Float | | +| web_chunk_size | 800 | 否 | Int | 开启联网检索后生效，web搜索到的内容文本分块大小 | ###

问答非流式请求示例

@@ -477,9 +519,24 @@ def send_request(): 'content-type': 'application/json' } data = { - "user_id": "zzp", - "kb_ids": ["KBf652e9e379c546f1894597dcabdc8e47"], - "question": "一嗨案件中保险单号是多少？", + "user_id":"zzp", + "kb_ids":["KBb85274e102df4de6aa22bf15a47d1c06_240625"], + "history":[], + "question":"蔡徐坤是谁？", + "streaming":true, + "networking":false, + "product_source":"saas", + "rerank":false, + "only_need_search_results":false, + "hybrid_search":false, + "max_token":512, + "api_base":"https://api.openai-proxy.org/v1", + "api_key":"sk-xxx", + "model":"gpt-3.5-turbo", + "api_context_length":16384, + "chunk_size":800, + "top_p":1, + "temperature":0.5 } try: start_time = time.time() @@ -530,7 +587,23 @@ if __name__ == '__main__': "score": "-8.9622345", "embed_version": "local_v0.0.1_20230525_6d4019f1559aef84abc2ab8257e1ad4c" } - ] //知识库相关文本内容 + ]， //retrieval_documents经过一系列后处理最终给到LLM的文本内容，可能是多个chunks组合而成 + "retrieval_documents": [...], // 检索到的文本chunks + "time_record": { + "time_usage": { + "preprocess": 0.01, + "retriever_search_by_milvus": 0.2, + "retriever_search": 0.2, + "rerank": 0.13, + "reprocess": 0.02, + "llm_first_return": 2.64 + }, + "token_usage": { + "prompt_tokens": 1847, + "completion_tokens": 50, + "total_tokens": 1897 + } + } } ``` @@ -617,11 +690,11 @@ if __name__ == '__main__': ###

删除文件请求参数（Body）

-| 参数名 | 示例参数值 | 是否必填 | 参数类型 | 描述说明 | -| -------- | ------------------------------------ | -------- | -------- | ----------------------------- | -| user_id | "zzp" | 是 | String | 用户 id | -| kb_id | "KB1271e71c36ec4028a6542586946a3906" | 是 | String | 知识库 id | -| file_ids | ["73ff7cf76ff34c8aa3a5a0b4ba3cf534"] | 是 | Array | 要删除文件的 id，支持批量删除 | +| 参数名 | 示例参数值 | 是否必填 | 参数类型 | 描述说明 | +| -------- | ------------------------------------ | ---- | ------ | ---------------- | +| user_id | "zzp" | 是 | String | 用户 id | +| kb_id | "KB1271e71c36ec4028a6542586946a3906" | 是 | String | 知识库 id | +| file_ids | ["73ff7cf76ff34c8aa3a5a0b4ba3cf534"] | 是 | Array | 要删除文件的 id，支持批量删除 | ###

删除文件请求示例

@@ -634,11 +707,11 @@ headers = { "Content-Type": "application/json" } data = { - "user_id": "zzp", //用户id - "kb_id": "KB1271e71c36ec4028a6542586946a3906", //知识库id - "file_ids": [ - "73ff7cf76ff34c8aa3a5a0b4ba3cf534" - ] //文件id列表 + "user_id": "zzp", //用户id + "kb_id": "KB1271e71c36ec4028a6542586946a3906", //知识库id + "file_ids": [ + "73ff7cf76ff34c8aa3a5a0b4ba3cf534" + ] //文件id列表 } response = requests.post(url, headers=headers, data=json.dumps(data)) @@ -661,10 +734,10 @@ URL： ###

删除知识库请求参数（Body）

-| 参数名 | 示例参数值 | 是否必填 | 参数类型 | 描述说明 | -| ------- | -------------------------------------- | -------- | -------- | ------------------------------- | -| user_id | "zzp" | 是 | String | 用户 id | -| kb_ids | ["KB1cd81f2bc515437294bda1934a20b235"] | 是 | Array | 要删除的知识库 id，支持批量删除 | +| 参数名 | 示例参数值 | 是否必填 | 参数类型 | 描述说明 | +| ------- | -------------------------------------- | ---- | ------ | ----------------- | +| user_id | "zzp" | 是 | String | 用户 id | +| kb_ids | ["KB1cd81f2bc515437294bda1934a20b235"] | 是 | Array | 要删除的知识库 id，支持批量删除 | ###

删除知识库请求示例

@@ -677,10 +750,10 @@ headers = { "Content-Type": "application/json" } data = { - "user_id": "zzp", //用户id - "kb_ids": [ - "KB1cd81f2bc515437294bda1934a20b235" - ] //知识库id列表 + "user_id": "zzp", //用户id + "kb_ids": [ + "KB1cd81f2bc515437294bda1934a20b235" + ] //知识库id列表 } response = requests.post(url, headers=headers, data=json.dumps(data)) @@ -704,9 +777,9 @@ print(response.text) ###

获取所有知识库状态请求参数（Body）

-| 参数名 | 示例参数值 | 是否必填 | 参数类型 | 描述说明 | -| ------- | ---------- | -------- | -------- | -------- | -| user_id | "zzp" | 是 | String | 用户 id | +| 参数名 | 示例参数值 | 是否必填 | 参数类型 | 描述说明 | +| ------- | ----- | ---- | ------ | ----- | +| user_id | "zzp" | 是 | String | 用户 id | ###

获取所有知识库状态请求示例

@@ -719,7 +792,7 @@ headers = { "Content-Type": "application/json" } data = { - "user_id": "zzp" + "user_id": "zzp" } response = requests.post(url, headers=headers, data=json.dumps(data)) @@ -759,11 +832,11 @@ print(response.text) ###

清理知识库请求参数（Body）

-| 参数名 | 示例参数值 | 是否必填 | 参数类型 | 描述说明 | -| ------- | ---------------------------------------------------------------------------- | -------- | -------- | ----------------------------------------------------------------------------------------------------------- | -| user_id | "zzp" | 是 | String | 用户 id | -| status | "gray" | 是 | String | 用清理何种状态的文件，red：入库失败-切分失败，green，成功入库，yellow：入库失败-milvus 失败，gray：正在入库 | -| kd_ids | ["KB0015df77a8eb46f6951de513392dc250", "KB6a4534f753b54c4198b62770e26b1a92"] | 否 | String | 需要清理的知识库 id 列表，默认为[]，此时默认清理 user_id 下全部的知识库 | +| 参数名 | 示例参数值 | 是否必填 | 参数类型 | 描述说明 | +| ------- | ---------------------------------------------------------------------------- | ---- | ------ | ------------------------------------------------------------------- | +| user_id | "zzp" | 是 | String | 用户 id | +| status | "gray" | 是 | String | 用清理何种状态的文件，red：入库失败-切分失败，green，成功入库，yellow：入库失败-milvus 失败，gray：正在入库 | +| kd_ids | ["KB0015df77a8eb46f6951de513392dc250", "KB6a4534f753b54c4198b62770e26b1a92"] | 否 | String | 需要清理的知识库 id 列表，默认为[]，此时默认清理 user_id 下全部的知识库 | ###

清理知识库请求示例

@@ -776,8 +849,8 @@ headers = { "Content-Type": "application/json" } data = { - "user_id": "zzp", - "status": "gray" + "user_id": "zzp", + "status": "gray" } response = requests.post(url, headers=headers, data=json.dumps(data)) @@ -812,11 +885,11 @@ print(response.text) ###

重命名知识库请求参数（Body）

-| 参数名 | 示例参数值 | 是否必填 | 参数类型 | 描述说明 | -| ----------- | ------------------------------------ | -------- | -------- | --------------------- | -| user_id | "zzp" | 是 | String | 用户 id | -| kb_id | "KB0015df77a8eb46f6951de513392dc250" | 是 | String | 需要重命名的知识库 id | -| new_kb_name | "新知识库" | 是 | String | 重命名后的知识库名字 | +| 参数名 | 示例参数值 | 是否必填 | 参数类型 | 描述说明 | +| ----------- | ------------------------------------ | ---- | ------ | ------------ | +| user_id | "zzp" | 是 | String | 用户 id | +| kb_id | "KB0015df77a8eb46f6951de513392dc250" | 是 | String | 需要重命名的知识库 id | +| new_kb_name | "新知识库" | 是 | String | 重命名后的知识库名字 | ###

重命名知识库请求示例

@@ -829,9 +902,9 @@ headers = { "Content-Type": "application/json" } data = { - "user_id": "zzp", - "kb_id": "KB0015df77a8eb46f6951de513392dc250", - "new_kb_name": "新知识库" + "user_id": "zzp", + "kb_id": "KB0015df77a8eb46f6951de513392dc250", + "new_kb_name": "新知识库" } response = requests.post(url, headers=headers, data=json.dumps(data)) @@ -855,16 +928,16 @@ print(response.text) ###

创建Bot请求参数（Body）

-| 参数名 | 示例参数值 | 是否必填 | 参数类型 | 描述说明 | -|-------------|---------------------------------|------|--------------|-----------------------| -| user_id | "zzp" | 是 | String | 用户 id | -| bot_name | "测试" | 是 | String | Bot名字 | -| description | "这是一个测试Bot" | 否 | String | Bot简介 | -| head_image | "zhiyun/xxx" | 否 | String | 头像nos地址 | -| prompt_setting | "你是一个耐心、友好、专业的机器人，能够回答用户的各种问题。" | 否 | String | Bot角色设定 | +| 参数名 | 示例参数值 | 是否必填 | 参数类型 | 描述说明 | +| --------------- | ------------------------------- | ---- | ------------ | --------------------- | +| user_id | "zzp" | 是 | String | 用户 id | +| bot_name | "测试" | 是 | String | Bot名字 | +| description | "这是一个测试Bot" | 否 | String | Bot简介 | +| head_image | "zhiyun/xxx" | 否 | String | 头像nos地址 | +| prompt_setting | "你是一个耐心、友好、专业的机器人，能够回答用户的各种问题。" | 否 | String | Bot角色设定 | | welcome_message | "您好，我是您的专属机器人，请问有什么可以帮您呢？" | 否 | String | Bot欢迎语 | -| model | "MiniChat-2-3B" | 否 | String | 模型选择，默认为MiniChat-2-3B | -| kb_ids | ["KB_xxx, KB_xxx, KB_xxx"] | 否 | List[String] | Bot关联的知识库id列表 | +| model | "MiniChat-2-3B" | 否 | String | 模型选择，默认为MiniChat-2-3B | +| kb_ids | ["KB_xxx, KB_xxx, KB_xxx"] | 否 | List[String] | Bot关联的知识库id列表 | ###

创建Bot请求示例

@@ -877,8 +950,8 @@ headers = { "Content-Type": "application/json" } data = { - "user_id": "zzp", - "bot_name": "测试机器人" + "user_id": "zzp", + "bot_name": "测试机器人" } response = requests.post(url, headers=headers, data=json.dumps(data)) @@ -891,13 +964,13 @@ print(response.text) ```json { - "code": 200, - "msg": "success create qanything bot BOTc87cc749a9844325a2e26f09bf8f6918", - "data": { - "bot_id": "BOTc87cc749a9844325a2e26f09bf8f6918", - "bot_name": "测试机器人", - "create_time": "2024-04-10 10:19:56" - } + "code": 200, + "msg": "success create qanything bot BOTc87cc749a9844325a2e26f09bf8f6918", + "data": { + "bot_id": "BOTc87cc749a9844325a2e26f09bf8f6918", + "bot_name": "测试机器人", + "create_time": "2024-04-10 10:19:56" + } } ``` @@ -907,10 +980,10 @@ print(response.text) ###

获取Bot信息请求参数（Body）

-| 参数名 | 示例参数值 | 是否必填 | 参数类型 | 描述说明 | -|-----------------|---------------------------------------|------| -------- |-----------------------| -| user_id | "zzp" | 是 | String | 用户 id | -| bot_id | "BOTc87cc749a9844325a2e26f09bf8f6918" | 否 | String | Bot id，不填则返回当前用户所有Bot | +| 参数名 | 示例参数值 | 是否必填 | 参数类型 | 描述说明 | +| ------- | ------------------------------------- | ---- | ------ | --------------------- | +| user_id | "zzp" | 是 | String | 用户 id | +| bot_id | "BOTc87cc749a9844325a2e26f09bf8f6918" | 否 | String | Bot id，不填则返回当前用户所有Bot | ###

获取Bot信息请求示例

@@ -923,8 +996,8 @@ headers = { "Content-Type": "application/json" } data = { - "user_id": "zzp", - "bot_id": "BOTc87cc749a9844325a2e26f09bf8f6918" + "user_id": "zzp", + "bot_id": "BOTc87cc749a9844325a2e26f09bf8f6918" } response = requests.post(url, headers=headers, data=json.dumps(data)) @@ -937,22 +1010,22 @@ print(response.text) ```json { - "code": 200, - "msg": "success", - "data": [ - { - "bot_id": "BOTc87cc749a9844325a2e26f09bf8f6918", - "user_id": "zzp", - "bot_name": "测试机器人", - "description": "一个简单的问答机器人", - "head_image": "", - "prompt_setting": "\n- 你是一个耐心、友好、专业的机器人，能够回答用户的各种问题。\n- 根据知识库内的检索结果，以清晰简洁的表达方式回答问题。\n- 不要编造答案，如果答案不在经核实的资料中或无法从经核实的资料中得出，请回答“我无法回答您的问题。”（或者您可以修改为：如果给定的检索结果无法回答问题，可以利用你的知识尽可能回答用户的问题。)\n", - "welcome_message": "您好，我是您的专属机器人，请问有什么可以帮您呢？", - "model": "MiniChat-2-3B", - "kb_ids": "", - "update_time": "2024-04-10 10:19:56" - } - ] + "code": 200, + "msg": "success", + "data": [ + { + "bot_id": "BOTc87cc749a9844325a2e26f09bf8f6918", + "user_id": "zzp", + "bot_name": "测试机器人", + "description": "一个简单的问答机器人", + "head_image": "", + "prompt_setting": "\n- 你是一个耐心、友好、专业的机器人，能够回答用户的各种问题。\n- 根据知识库内的检索结果，以清晰简洁的表达方式回答问题。\n- 不要编造答案，如果答案不在经核实的资料中或无法从经核实的资料中得出，请回答“我无法回答您的问题。”（或者您可以修改为：如果给定的检索结果无法回答问题，可以利用你的知识尽可能回答用户的问题。)\n", + "welcome_message": "您好，我是您的专属机器人，请问有什么可以帮您呢？", + "model": "MiniChat-2-3B", + "kb_ids": "", + "update_time": "2024-04-10 10:19:56" + } + ] } ``` @@ -962,17 +1035,17 @@ print(response.text) ###

修改Bot信息请求参数（Body）

-| 参数名 | 示例参数值 | 是否必填 | 参数类型 | 描述说明 | -|-----------------|--------------------------------|------| -------- |-----------------------| -| user_id | "zzp" | 是 | String | 用户 id | -| bot_id | "BOTc87cc749a9844325a2e26f09bf8f6918" | 是 | String | 用户 id | -| bot_name | "测试" | 否 | String | Bot名字 | -| description | "这是一个测试Bot" | 否 | String | Bot简介 | -| head_image | "zhiyun/xxx" | 否 | String | 头像nos地址 | -| prompt_setting | "你是一个耐心、友好、专业的机器人，能够回答用户的各种问题。" | 否 | String | Bot角色设定 | -| welcome_message | "您好，我是您的专属机器人，请问有什么可以帮您呢？" | 否 | String | Bot欢迎语 | -| model | "MiniChat-2-3B" | 否 | String | 模型选择，默认为MiniChat-2-3B | -| kb_ids | "KB_xxx, KB_xxx, KB_xxx" | 否 | String | Bot关联的知识库id列表 | +| 参数名 | 示例参数值 | 是否必填 | 参数类型 | 描述说明 | +| --------------- | ------------------------------------- | ---- | ------ | --------------------- | +| user_id | "zzp" | 是 | String | 用户 id | +| bot_id | "BOTc87cc749a9844325a2e26f09bf8f6918" | 是 | String | 用户 id | +| bot_name | "测试" | 否 | String | Bot名字 | +| description | "这是一个测试Bot" | 否 | String | Bot简介 | +| head_image | "zhiyun/xxx" | 否 | String | 头像nos地址 | +| prompt_setting | "你是一个耐心、友好、专业的机器人，能够回答用户的各种问题。" | 否 | String | Bot角色设定 | +| welcome_message | "您好，我是您的专属机器人，请问有什么可以帮您呢？" | 否 | String | Bot欢迎语 | +| model | "MiniChat-2-3B" | 否 | String | 模型选择，默认为MiniChat-2-3B | +| kb_ids | "KB_xxx, KB_xxx, KB_xxx" | 否 | String | Bot关联的知识库id列表 | ###

修改ot信息请求示例

@@ -1006,8 +1079,8 @@ print(response.text) ```json { - "code": 200, - "msg": "Bot BOTc87cc749a9844325a2e26f09bf8f6918 update success" + "code": 200, + "msg": "Bot BOTc87cc749a9844325a2e26f09bf8f6918 update success" } ``` @@ -1017,10 +1090,10 @@ print(response.text) ###

删除Bot请求参数（Body）

-| 参数名 | 示例参数值 | 是否必填 | 参数类型 | 描述说明 | -|-----------------|---------------------------------------|------| -------- |-----------------------| -| user_id | "zzp" | 是 | String | 用户 id | -| bot_id | "BOTc87cc749a9844325a2e26f09bf8f6918" | 否 | String | Bot id，不填则返回当前用户所有Bot | +| 参数名 | 示例参数值 | 是否必填 | 参数类型 | 描述说明 | +| ------- | ------------------------------------- | ---- | ------ | --------------------- | +| user_id | "zzp" | 是 | String | 用户 id | +| bot_id | "BOTc87cc749a9844325a2e26f09bf8f6918" | 否 | String | Bot id，不填则返回当前用户所有Bot | ###

删除Bot请求示例

@@ -1033,8 +1106,8 @@ headers = { "Content-Type": "application/json" } data = { - "user_id": "zzp", - "bot_id": "BOTc87cc749a9844325a2e26f09bf8f6918" + "user_id": "zzp", + "bot_id": "BOTc87cc749a9844325a2e26f09bf8f6918" } response = requests.post(url, headers=headers, data=json.dumps(data)) @@ -1047,8 +1120,8 @@ print(response.text) ```json { - "code": 200, - "msg": "Bot BOTc87cc749a9844325a2e26f09bf8f6918 delete success" + "code": 200, + "msg": "Bot BOTc87cc749a9844325a2e26f09bf8f6918 delete success" } ``` @@ -1060,12 +1133,12 @@ Content-Type: multipart/form-data ###

上传FAQ请求参数（Body）

-| 参数名 | 参数值 | 是否必填 | 参数类型 | 描述说明 | -|---------|--------------------------------------------------|------|------------|---------------------------| -| files | 文件二进制 | 否 | File | 需要上传的文件，可多选，仅支持xlsx格式 | -| user_id | zzp | 是 | String | 用户 id | -| kb_id | KBb1dd58e8485443ce81166d24f6febda7_FAQ | 是 | String | 知识库 id | -| faqs | [{"question": "你是谁？", "answer": "我是QAnything！"}] | 否 | List[Dict] | json格式的FAQ集合（与files参数二选一） | +| 参数名 | 参数值 | 是否必填 | 参数类型 | 描述说明 | +| ---------- | ------------------------------------------------ | ---- | ---------- | -------------- | +| user_id | zzp | 是 | String | 用户 id | +| kb_id | KBb1dd58e8485443ce81166d24f6febda7_FAQ | 是 | String | 知识库 id | +| faqs | [{"question": "你是谁？", "answer": "我是QAnything！"}] | 否 | List[Dict] | json格式的FAQ集合 | +| chunk_size | 800 | 否 | Int | 文件切片大小（tokens） | ###

上传FAQ请求示例

@@ -1074,20 +1147,13 @@ import os import requests url = "http://{your_host}:8777/api/local_doc_qa/upload_faqs" -folder_path = "./xlsx_data" # 文件所在文件夹，注意是文件夹！！ data = { "user_id": "zzp", "kb_id": "KB6dae785cdd5d47a997e890521acbe1c9_FAQ", + "faqs": [{"question": "北京3月13日天气"， "answer": "15-29度，小雨转晴"}] } -files = [] -for root, dirs, file_names in os.walk(folder_path): - for file_name in file_names: - if file_name.endswith(".xlsx"): - file_path = os.path.join(root, file_name) - files.append(("files", open(file_path, "rb"))) - -response = requests.post(url, files=files, data=data) +response = requests.post(url, data=data) print(response.text) ``` @@ -1095,41 +1161,41 @@ print(response.text) ```json { - "code": 200, - "msg": "success，后台正在飞速上传文件，请耐心等待", - "file_status": { - "QAnything_QA_模板.xlsx": "success" - }, - "data": [ - { - "file_id": "de9c63dfd2fe477394041612bdfbdec3", - "file_name": "FAQ_你今天吃了什么.faq", - "status": "gray", - "length": 11, - "timestamp": "202404121141" - }, - { - "file_id": "b3f9ac467ce14b0280e991c098298155", - "file_name": "FAQ_天气晴朗吗.faq", - "status": "gray", - "length": 9, - "timestamp": "202404121141" - }, - { - "file_id": "5a5ac165324940a8a62062a5335f5f92", - "file_name": "FAQ_你是谁.faq", - "status": "gray", - "length": 14, - "timestamp": "202404121141" - }, - { - "file_id": "562e9a2d1b994d619d4e1ebc85857c92", - "file_name": "FAQ_你好呀.faq", - "status": "gray", - "length": 7, - "timestamp": "202404121141" - } - ] + "code": 200, + "msg": "success，后台正在飞速上传文件，请耐心等待", + "file_status": { + "QAnything_QA_模板.xlsx": "success" + }, + "data": [ + { + "file_id": "de9c63dfd2fe477394041612bdfbdec3", + "file_name": "FAQ_你今天吃了什么.faq", + "status": "gray", + "length": 11, + "timestamp": "202404121141" + }, + { + "file_id": "b3f9ac467ce14b0280e991c098298155", + "file_name": "FAQ_天气晴朗吗.faq", + "status": "gray", + "length": 9, + "timestamp": "202404121141" + }, + { + "file_id": "5a5ac165324940a8a62062a5335f5f92", + "file_name": "FAQ_你是谁.faq", + "status": "gray", + "length": 14, + "timestamp": "202404121141" + }, + { + "file_id": "562e9a2d1b994d619d4e1ebc85857c92", + "file_name": "FAQ_你好呀.faq", + "status": "gray", + "length": 7, + "timestamp": "202404121141" + } + ] } ``` @@ -1139,17 +1205,17 @@ print(response.text) ###

检索qa日志请求参数（Body）

-| 参数名 | 示例参数值 | 是否必填 | 参数类型 | 描述说明 | -|---------------|----------------------------------------|------|--------------|-------------------------------------------------------| -| user_id | "zzp" | 是 | String | 用户 id | -| bot_id | "BOTc87cc749a9844325a2e26f09bf8f6918" | 否 | String | 非空则增加bot_id的筛选条件 | -| kb_ids | ["KBc87cc749a9844325a2e26f09bf8f6918"] | 否 | List[String] | 非空则增加kb_ids的筛选条件 | -| query | "你的名字" | 否 | String | 非空则增加query的筛选条件 | -| time_start | "2024-10-05" | 否 | String | 非空则增加time_start的筛选条件，需和time_end配合使用 | -| time_end | "2024-10-05" | 否 | String | 非空则增加time_end的筛选条件，需和time_start配合使用 | -| page_id | 0 | 否 | Int | 非空则增加page_id的筛选条件，结果超过100条时默认返回第0页（前100条），可用page_id换页 | -| need_info | ["query", "result", "timestamp"] | 否 | List[String] | 非空则增加need_info的筛选条件, 返回日志的key，空则返回全部的key | -| save_to_excel | True | 否 | Bool | 默认为False，为True时返回excel文件，可直接下载 | +| 参数名 | 示例参数值 | 是否必填 | 参数类型 | 描述说明 | +| ------------- | ------------------------------------- | ---- | ------------ | ----------------------------------------------------- | +| user_id | "zzp" | 否 | String | 用户 id，与any_kb_id二选一必填 | +| bot_id | "BOTc87cc749a9844325a2e26f09bf8f6918" | 否 | String | 非空则增加bot_id的筛选条件 | +| any_kb_id | "KBc87cc749a9844325a2e26f09bf8f6918" | 否 | String | 非空则增加kb_id的筛选条件 | +| query | "你的名字" | 否 | String | 非空则增加query的筛选条件 | +| time_start | "2024-10-05" | 否 | String | 非空则增加time_start的筛选条件，需和time_end配合使用 | +| time_end | "2024-10-05" | 否 | String | 非空则增加time_end的筛选条件，需和time_start配合使用 | +| page_id | 0 | 否 | Int | 非空则增加page_id的筛选条件，结果超过100条时默认返回第0页（前100条），可用page_id换页 | +| need_info | ["query", "result", "timestamp"] | 否 | List[String] | 非空则增加need_info的筛选条件, 返回日志的key，空则返回全部的key | +| save_to_excel | True | 否 | Bool | 默认为False，为True时返回excel文件，可直接下载 | ###

检索qa日志请求示例

@@ -1182,13 +1248,192 @@ print(response.text) ```json { - "code": 200, - "msg": "检索到的Log数为1，一次返回所有数据", - "page_id": 0, - "qa_infos": [ - { - "user_id": "zzp" - } - ] + "code": 200, + "msg": "检索到的Log数为1，一次返回所有数据", + "page_id": 0, + "qa_infos": [ + { + "user_id": "zzp" + } + ] +} +``` + +##

获取文档完整解析内容（POST）

+ +###

URL：

+ +###

获取文档完整解析内容请求参数（Body）

+ +| 参数名 | 示例参数值 | 是否必填 | 参数类型 | 描述说明 | +| ----------- | ------------------------------------------- | ---- | ------ | ------------------------------------------ | +| user_id | "zzp" | 是 | String | 用户 id | +| kb_id | "KB698428590b1e45108b63b943fac16559_240430" | 是 | String | 知识库id | +| file_id | "68d7e503cb79497f8416152bb6b773cd" | 是 | String | 文件id | +| page_offset | 1 | 否 | Int | 默认值为1，结果太多时可指定分页id（从1开始），与page_limit配合使用 | +| page_limit | 10 | 否 | Int | 默认值为10，结果太多时可指定分页id（从1开始），与page_offset配合使用 | + +###

获取文档完整解析内容请求示例

+ +```python +import requests +import json + +url = "http://{your_host}:8777/api/local_doc_qa/get_doc_completed" +headers = { + "Content-Type": "application/json" +} +data = { + "user_id": "zzp", + "kb_id": "KBe3f7b698208645218e787d2eee2eae41", + "file_id": "68d7e503cb79497f8416152bb6b773cd" +} + +response = requests.post(url, headers=headers, data=json.dumps(data)) + +print(response.status_code) +print(response.text) +``` + +###

获取文档完整解析内容响应示例

+ +```json +{ + "code": 200, + "msg": "success", + "completed_text": "\n6.全国两会会议精神.docx\n全国政协十四届二次会议开幕会会议精神\n\n6.全国两会会议精神.docx\n中国人民政治协商会议第十四届全国委员会第二次会议4日下午在人民大会堂开幕。2000多名全国政协委员将紧紧围绕中共中央决策部署，牢记政治责任，积极建言资政，广泛凝聚共识，共同谱写中国式现代化的壮美华章。\n\n6.全国两会会议精神.docx\n大会首先审议通过了政协第十四届全国委员会第二次会议议程。\n\n6.全国两会会议精神.docx\n王沪宁代表政协第十四届全国委员会常务委员会，向大会报告工作。\n\n6.全国两会会议精神.docx\n王沪宁表示，2023年是全面贯彻落实中共二十大精神的开局之年。以习近平同志为核心的中共中央团结带领全党全国各族人民，坚持稳中求进工作总基调，接续奋斗、砥砺前行，坚决克服内外困难，全面深化改革开放，新冠疫情防控平稳转段，高质量发展扎实推进，科技创新实现新突破，安全发展基础巩固夯实，民生保障有力有效，经济社会发展主要预期目标圆满完成，社会大局保持稳定，全面建设社会主义现代化国家迈出坚实步伐，极大增强了全国各族人民信心和底气。\n\n6.全国两会会议精神.docx\n王沪宁总结了过去一年来人民政协工作。他说，在以习近平同志为核心的中共中央坚强领导下，政协全国委员会及其常务委员会坚持以习近平新时代中国特色社会主义思想为指导，全面贯彻中共二十大和二十届二中全会精神，深刻领悟“两个确立”的决定性意义，增强“四个意识”、坚定“四个自信”、做到“两个维护”，坚持团结和民主两大主题，坚持人民政协性质定位，坚持在党和国家工作大局中谋划推进政协工作。坚持中国共产党对人民政协工作的全面领导、把牢履职正确政治方向，强化政治培训、提高政治能力和履职本领，完善工作制度体系、夯实履职制度基础，贯彻中共中央大兴调查研究要求、提高调研议政质量，围绕中共二十大重大部署协商议政、服务党和国家中心任务，践行以人民为中心的发展思想、助推保障和改善民生，守正创新、团结奋进，在历届全国政协打下的良好基础上，各项工作取得新成效，服务党和国家事业发展作出新贡献。\n\n6.全国两会会议精神.docx\n王沪宁表示，2024年是中华人民共和国成立75周年，是实现“十四五”规划目标任务的关键一年，也是人民政协成立75周年。人民政协要坚持以习近平新时代中国特色社会主义思想为指导，全面贯彻中共二十大和二十届二中全会精神，坚持党的领导、统一战线、协商民主有机结合，紧紧围绕推进中国式现代化履职尽责，推进思想政治引领，积极建言资政，广泛凝聚共识，加强自身建设，为实现全年经济社会发展目标任务汇聚智慧和力量。...\n", + "chunks": [ + { + "page_content": "6.全国两会会议精神.docx\n全国政协十四届二次会议开幕会会议精神", + "metadata": { + "user_id": "zzp", + "kb_id": "KB70b80588e3614fb9ac21581d38176a05_240430", + "file_id": "0a9f227fffd240aeb24d99e19b8e580e", + "file_name": "6.全国两会会议精神.docx", + "nos_key": "xxx/zzp/KB70b80588e3614fb9ac21581d38176a05_240430/0a9f227fffd240aeb24d99e19b8e580e/6.全国两会会议精神.docx", + "file_url": null, + "title_lst": [], + "has_table": false, + "images_number": 0, + "faq_dict": {}, + "source_info": { + "page_id": 0, + "pdf_nos_key": "zhiyun/docqa/qanything/local_file/zhiyun_paas_userId_test__93/KB70b80588e3614fb9ac21581d38176a05_240430/0a9f227fffd240aeb24d99e19b8e580e/6.全国两会会议精神.docx", + "chunk_id": -1, + "doc_id": "0a9f227fffd240aeb24d99e19b8e580e", + "bbox": null + } + } + }, + { + "page_content": "6.全国两会会议精神.docx\n中国人民政治协商会议第十四届全国委员会第二次会议4日下午在人民大会堂开幕。2000多名全国政协委员将紧紧围绕中共中央决策部署，牢记政治责任，积极建言资政，广泛凝聚共识，共同谱写中国式现代化的壮美华章。", + "metadata": { + "user_id": "zzp", + "kb_id": "KB70b80588e3614fb9ac21581d38176a05_240430", + "file_id": "0a9f227fffd240aeb24d99e19b8e580e", + "file_name": "6.全国两会会议精神.docx", + "nos_key": "xxx/zzp/KB70b80588e3614fb9ac21581d38176a05_240430/0a9f227fffd240aeb24d99e19b8e580e/6.全国两会会议精神.docx", + "file_url": null, + "title_lst": [], + "has_table": false, + "images_number": 0, + "faq_dict": {}, + "source_info": { + "page_id": 0, + "pdf_nos_key": "zhiyun/docqa/qanything/local_file/zhiyun_paas_userId_test__93/KB70b80588e3614fb9ac21581d38176a05_240430/0a9f227fffd240aeb24d99e19b8e580e/6.全国两会会议精神.docx", + "chunk_id": -1, + "doc_id": "0a9f227fffd240aeb24d99e19b8e580e", + "bbox": null + } + } + }, + ... + ], + "page_id": 1, + "total_count": 102, +} +``` + +##

更新chunks（POST）

+ +###

URL：

+ +###

更新chunks请求参数（Body）

+ +| 参数名 | 示例参数值 | 是否必填 | 参数类型 | 描述说明 | +| -------------- | ------------------------------------ | ---- | ------ | ------------------- | +| user_id | "zzp" | 是 | String | 用户 id | +| doc_id | "698428590b1e45108b63b943fac16559_0" | 是 | String | 修改的chunk对应的doc_id | +| update_content | "新的chunk内容" | 是 | String | 更新后的chunk文本内容 | +| chunk_size | 800 | 否 | Int | 文本切片大小，默认8000tokens | + +###

更新chunks请求示例

+ +```python +import requests +import json + +url = "http://{your_host}:8777/api/local_doc_qa/update_chunks" +headers = { + "Content-Type": "application/json" +} +data = { + "user_id": "zzp", + "doc_id": "698428590b1e45108b63b943fac16559_0", + "update_content": "新的chunk内容" +} + +response = requests.post(url, headers=headers, data=json.dumps(data)) + +print(response.status_code) +print(response.text) +``` + +###

更新chunks响应示例

+ +```json +{ + "code": 200, + "msg": "success update doc_id 698428590b1e45108b63b943fac16559_0", +} +``` + +##

获取文件base64（POST）

+ +###

URL：

+ +###

获取文件base64请求参数（Body）

+ +| 参数名 | 示例参数值 | 是否必填 | 参数类型 | 描述说明 | +| ------- | ---------------------------------- | ---- | ------ | ----- | +| file_id | "698428590b1e45108b63b943fac16559" | 是 | String | 文件 id | + +###

获取文件base64请求示例

+ +```python +import requests +import json + +url = "http://{your_host}:8777/api/local_doc_qa/get_file_base64" +headers = { + "Content-Type": "application/json" +} +data = { + "file_id": "698428590b1e45108b63b943fac16559" +} + +response = requests.post(url, headers=headers, data=json.dumps(data)) + +print(response.status_code) +print(response.text) +``` + +###

获取文件base64响应示例

+ +```json +{ + "code": 200, + "msg": "success", + "file_base64": "xxx" } ``` \ No newline at end of file diff --git a/docs/assets/17244247170060.png b/docs/assets/17244247170060.png new file mode 100644 index 00000000..b9742db8 Binary files /dev/null and b/docs/assets/17244247170060.png differ diff --git a/docs/assets/17244247170067.png b/docs/assets/17244247170067.png new file mode 100644 index 00000000..7c7d2788 Binary files /dev/null and b/docs/assets/17244247170067.png differ diff --git a/docs/assets/17244247170074.png b/docs/assets/17244247170074.png new file mode 100644 index 00000000..9a9ba7c5 Binary files /dev/null and b/docs/assets/17244247170074.png differ diff --git a/docs/assets/17244247170088.png b/docs/assets/17244247170088.png new file mode 100644 index 00000000..143cc4a1 Binary files /dev/null and b/docs/assets/17244247170088.png differ diff --git a/docs/assets/17244247170101.png b/docs/assets/17244247170101.png new file mode 100644 index 00000000..4a569f7d Binary files /dev/null and b/docs/assets/17244247170101.png differ diff --git a/docs/assets/17244247170115.png b/docs/assets/17244247170115.png new file mode 100644 index 00000000..23b54dd0 Binary files /dev/null and b/docs/assets/17244247170115.png differ diff --git a/docs/assets/17244247170129.png b/docs/assets/17244247170129.png new file mode 100644 index 00000000..aa944bc0 Binary files /dev/null and b/docs/assets/17244247170129.png differ diff --git a/docs/assets/17244247170148.png b/docs/assets/17244247170148.png new file mode 100644 index 00000000..481ed5f6 Binary files /dev/null and b/docs/assets/17244247170148.png differ diff --git a/docs/assets/17244247170166.png b/docs/assets/17244247170166.png new file mode 100644 index 00000000..792f3605 Binary files /dev/null and b/docs/assets/17244247170166.png differ diff --git a/docs/assets/17244247170188.png b/docs/assets/17244247170188.png new file mode 100644 index 00000000..0e2faddb Binary files /dev/null and b/docs/assets/17244247170188.png differ diff --git a/docs/assets/17244247170208.png b/docs/assets/17244247170208.png new file mode 100644 index 00000000..40700b2b Binary files /dev/null and b/docs/assets/17244247170208.png differ diff --git a/docs/assets/17244247170228.png b/docs/assets/17244247170228.png new file mode 100644 index 00000000..59e077bf Binary files /dev/null and b/docs/assets/17244247170228.png differ diff --git a/docs/assets/17244247170272.png b/docs/assets/17244247170272.png new file mode 100644 index 00000000..0ecd4597 Binary files /dev/null and b/docs/assets/17244247170272.png differ diff --git a/docs/assets/17244247170298.png b/docs/assets/17244247170298.png new file mode 100644 index 00000000..0f504b52 Binary files /dev/null and b/docs/assets/17244247170298.png differ diff --git a/docs/assets/17244247170323.png b/docs/assets/17244247170323.png new file mode 100644 index 00000000..c2e68e3a Binary files /dev/null and b/docs/assets/17244247170323.png differ diff --git a/docs/assets/17244247170352.png b/docs/assets/17244247170352.png new file mode 100644 index 00000000..280dc39e Binary files /dev/null and b/docs/assets/17244247170352.png differ diff --git a/docs/assets/17244247170380.png b/docs/assets/17244247170380.png new file mode 100644 index 00000000..ab73a66b Binary files /dev/null and b/docs/assets/17244247170380.png differ diff --git a/docs/assets/17244247170406.png b/docs/assets/17244247170406.png new file mode 100644 index 00000000..be8e618d Binary files /dev/null and b/docs/assets/17244247170406.png differ diff --git a/docs/assets/17244247170443.png b/docs/assets/17244247170443.png new file mode 100644 index 00000000..7bb759a4 Binary files /dev/null and b/docs/assets/17244247170443.png differ diff --git a/docs/assets/17244247170478.png b/docs/assets/17244247170478.png new file mode 100644 index 00000000..aafb5553 Binary files /dev/null and b/docs/assets/17244247170478.png differ diff --git a/docs/assets/17244247170512.png b/docs/assets/17244247170512.png new file mode 100644 index 00000000..3dbf3035 Binary files /dev/null and b/docs/assets/17244247170512.png differ diff --git a/docs/assets/17244247170546.png b/docs/assets/17244247170546.png new file mode 100644 index 00000000..6abcf709 Binary files /dev/null and b/docs/assets/17244247170546.png differ diff --git a/docs/assets/17244247171267.png b/docs/assets/17244247171267.png new file mode 100644 index 00000000..69f8bd68 Binary files /dev/null and b/docs/assets/17244247171267.png differ