diff --git a/demo_android_ncnn/.gitignore b/demo_android_ncnn/.gitignore new file mode 100644 index 000000000..6f6d9f761 --- /dev/null +++ b/demo_android_ncnn/.gitignore @@ -0,0 +1,15 @@ +*.iml +.gradle +/local.properties +/.idea +/.idea/caches +/.idea/libraries +/.idea/modules.xml +/.idea/workspace.xml +/.idea/navEditor.xml +/.idea/assetWizardSettings.xml +.DS_Store +/build +/captures +.externalNativeBuild +.cxx diff --git a/demo_android_ncnn/LICENSE b/demo_android_ncnn/LICENSE new file mode 100644 index 000000000..9e419e042 --- /dev/null +++ b/demo_android_ncnn/LICENSE @@ -0,0 +1,674 @@ +GNU GENERAL PUBLIC LICENSE + Version 3, 29 June 2007 + + Copyright (C) 2007 Free Software Foundation, Inc. + Everyone is permitted to copy and distribute verbatim copies + of this license document, but changing it is not allowed. + + Preamble + + The GNU General Public License is a free, copyleft license for +software and other kinds of works. + + The licenses for most software and other practical works are designed +to take away your freedom to share and change the works. By contrast, +the GNU General Public License is intended to guarantee your freedom to +share and change all versions of a program--to make sure it remains free +software for all its users. We, the Free Software Foundation, use the +GNU General Public License for most of our software; it applies also to +any other work released this way by its authors. You can apply it to +your programs, too. + + When we speak of free software, we are referring to freedom, not +price. Our General Public Licenses are designed to make sure that you +have the freedom to distribute copies of free software (and charge for +them if you wish), that you receive source code or can get it if you +want it, that you can change the software or use pieces of it in new +free programs, and that you know you can do these things. + + To protect your rights, we need to prevent others from denying you +these rights or asking you to surrender the rights. Therefore, you have +certain responsibilities if you distribute copies of the software, or if +you modify it: responsibilities to respect the freedom of others. + + For example, if you distribute copies of such a program, whether +gratis or for a fee, you must pass on to the recipients the same +freedoms that you received. You must make sure that they, too, receive +or can get the source code. And you must show them these terms so they +know their rights. + + Developers that use the GNU GPL protect your rights with two steps: +(1) assert copyright on the software, and (2) offer you this License +giving you legal permission to copy, distribute and/or modify it. + + For the developers' and authors' protection, the GPL clearly explains +that there is no warranty for this free software. For both users' and +authors' sake, the GPL requires that modified versions be marked as +changed, so that their problems will not be attributed erroneously to +authors of previous versions. + + Some devices are designed to deny users access to install or run +modified versions of the software inside them, although the manufacturer +can do so. This is fundamentally incompatible with the aim of +protecting users' freedom to change the software. The systematic +pattern of such abuse occurs in the area of products for individuals to +use, which is precisely where it is most unacceptable. Therefore, we +have designed this version of the GPL to prohibit the practice for those +products. If such problems arise substantially in other domains, we +stand ready to extend this provision to those domains in future versions +of the GPL, as needed to protect the freedom of users. + + Finally, every program is threatened constantly by software patents. +States should not allow patents to restrict development and use of +software on general-purpose computers, but in those that do, we wish to +avoid the special danger that patents applied to a free program could +make it effectively proprietary. To prevent this, the GPL assures that +patents cannot be used to render the program non-free. + + The precise terms and conditions for copying, distribution and +modification follow. + + TERMS AND CONDITIONS + + 0. Definitions. + + "This License" refers to version 3 of the GNU General Public License. + + "Copyright" also means copyright-like laws that apply to other kinds of +works, such as semiconductor masks. + + "The Program" refers to any copyrightable work licensed under this +License. Each licensee is addressed as "you". "Licensees" and +"recipients" may be individuals or organizations. + + To "modify" a work means to copy from or adapt all or part of the work +in a fashion requiring copyright permission, other than the making of an +exact copy. The resulting work is called a "modified version" of the +earlier work or a work "based on" the earlier work. + + A "covered work" means either the unmodified Program or a work based +on the Program. + + To "propagate" a work means to do anything with it that, without +permission, would make you directly or secondarily liable for +infringement under applicable copyright law, except executing it on a +computer or modifying a private copy. Propagation includes copying, +distribution (with or without modification), making available to the +public, and in some countries other activities as well. + + To "convey" a work means any kind of propagation that enables other +parties to make or receive copies. Mere interaction with a user through +a computer network, with no transfer of a copy, is not conveying. + + An interactive user interface displays "Appropriate Legal Notices" +to the extent that it includes a convenient and prominently visible +feature that (1) displays an appropriate copyright notice, and (2) +tells the user that there is no warranty for the work (except to the +extent that warranties are provided), that licensees may convey the +work under this License, and how to view a copy of this License. If +the interface presents a list of user commands or options, such as a +menu, a prominent item in the list meets this criterion. + + 1. Source Code. + + The "source code" for a work means the preferred form of the work +for making modifications to it. "Object code" means any non-source +form of a work. + + A "Standard Interface" means an interface that either is an official +standard defined by a recognized standards body, or, in the case of +interfaces specified for a particular programming language, one that +is widely used among developers working in that language. + + The "System Libraries" of an executable work include anything, other +than the work as a whole, that (a) is included in the normal form of +packaging a Major Component, but which is not part of that Major +Component, and (b) serves only to enable use of the work with that +Major Component, or to implement a Standard Interface for which an +implementation is available to the public in source code form. A +"Major Component", in this context, means a major essential component +(kernel, window system, and so on) of the specific operating system +(if any) on which the executable work runs, or a compiler used to +produce the work, or an object code interpreter used to run it. + + The "Corresponding Source" for a work in object code form means all +the source code needed to generate, install, and (for an executable +work) run the object code and to modify the work, including scripts to +control those activities. However, it does not include the work's +System Libraries, or general-purpose tools or generally available free +programs which are used unmodified in performing those activities but +which are not part of the work. For example, Corresponding Source +includes interface definition files associated with source files for +the work, and the source code for shared libraries and dynamically +linked subprograms that the work is specifically designed to require, +such as by intimate data communication or control flow between those +subprograms and other parts of the work. + + The Corresponding Source need not include anything that users +can regenerate automatically from other parts of the Corresponding +Source. + + The Corresponding Source for a work in source code form is that +same work. + + 2. Basic Permissions. + + All rights granted under this License are granted for the term of +copyright on the Program, and are irrevocable provided the stated +conditions are met. This License explicitly affirms your unlimited +permission to run the unmodified Program. The output from running a +covered work is covered by this License only if the output, given its +content, constitutes a covered work. This License acknowledges your +rights of fair use or other equivalent, as provided by copyright law. + + You may make, run and propagate covered works that you do not +convey, without conditions so long as your license otherwise remains +in force. You may convey covered works to others for the sole purpose +of having them make modifications exclusively for you, or provide you +with facilities for running those works, provided that you comply with +the terms of this License in conveying all material for which you do +not control copyright. Those thus making or running the covered works +for you must do so exclusively on your behalf, under your direction +and control, on terms that prohibit them from making any copies of +your copyrighted material outside their relationship with you. + + Conveying under any other circumstances is permitted solely under +the conditions stated below. Sublicensing is not allowed; section 10 +makes it unnecessary. + + 3. Protecting Users' Legal Rights From Anti-Circumvention Law. + + No covered work shall be deemed part of an effective technological +measure under any applicable law fulfilling obligations under article +11 of the WIPO copyright treaty adopted on 20 December 1996, or +similar laws prohibiting or restricting circumvention of such +measures. + + When you convey a covered work, you waive any legal power to forbid +circumvention of technological measures to the extent such circumvention +is effected by exercising rights under this License with respect to +the covered work, and you disclaim any intention to limit operation or +modification of the work as a means of enforcing, against the work's +users, your or third parties' legal rights to forbid circumvention of +technological measures. + + 4. Conveying Verbatim Copies. + + You may convey verbatim copies of the Program's source code as you +receive it, in any medium, provided that you conspicuously and +appropriately publish on each copy an appropriate copyright notice; +keep intact all notices stating that this License and any +non-permissive terms added in accord with section 7 apply to the code; +keep intact all notices of the absence of any warranty; and give all +recipients a copy of this License along with the Program. + + You may charge any price or no price for each copy that you convey, +and you may offer support or warranty protection for a fee. + + 5. Conveying Modified Source Versions. + + You may convey a work based on the Program, or the modifications to +produce it from the Program, in the form of source code under the +terms of section 4, provided that you also meet all of these conditions: + + a) The work must carry prominent notices stating that you modified + it, and giving a relevant date. + + b) The work must carry prominent notices stating that it is + released under this License and any conditions added under section + 7. This requirement modifies the requirement in section 4 to + "keep intact all notices". + + c) You must license the entire work, as a whole, under this + License to anyone who comes into possession of a copy. This + License will therefore apply, along with any applicable section 7 + additional terms, to the whole of the work, and all its parts, + regardless of how they are packaged. This License gives no + permission to license the work in any other way, but it does not + invalidate such permission if you have separately received it. + + d) If the work has interactive user interfaces, each must display + Appropriate Legal Notices; however, if the Program has interactive + interfaces that do not display Appropriate Legal Notices, your + work need not make them do so. + + A compilation of a covered work with other separate and independent +works, which are not by their nature extensions of the covered work, +and which are not combined with it such as to form a larger program, +in or on a volume of a storage or distribution medium, is called an +"aggregate" if the compilation and its resulting copyright are not +used to limit the access or legal rights of the compilation's users +beyond what the individual works permit. Inclusion of a covered work +in an aggregate does not cause this License to apply to the other +parts of the aggregate. + + 6. Conveying Non-Source Forms. + + You may convey a covered work in object code form under the terms +of sections 4 and 5, provided that you also convey the +machine-readable Corresponding Source under the terms of this License, +in one of these ways: + + a) Convey the object code in, or embodied in, a physical product + (including a physical distribution medium), accompanied by the + Corresponding Source fixed on a durable physical medium + customarily used for software interchange. + + b) Convey the object code in, or embodied in, a physical product + (including a physical distribution medium), accompanied by a + written offer, valid for at least three years and valid for as + long as you offer spare parts or customer support for that product + model, to give anyone who possesses the object code either (1) a + copy of the Corresponding Source for all the software in the + product that is covered by this License, on a durable physical + medium customarily used for software interchange, for a price no + more than your reasonable cost of physically performing this + conveying of source, or (2) access to copy the + Corresponding Source from a network server at no charge. + + c) Convey individual copies of the object code with a copy of the + written offer to provide the Corresponding Source. This + alternative is allowed only occasionally and noncommercially, and + only if you received the object code with such an offer, in accord + with subsection 6b. + + d) Convey the object code by offering access from a designated + place (gratis or for a charge), and offer equivalent access to the + Corresponding Source in the same way through the same place at no + further charge. You need not require recipients to copy the + Corresponding Source along with the object code. If the place to + copy the object code is a network server, the Corresponding Source + may be on a different server (operated by you or a third party) + that supports equivalent copying facilities, provided you maintain + clear directions next to the object code saying where to find the + Corresponding Source. Regardless of what server hosts the + Corresponding Source, you remain obligated to ensure that it is + available for as long as needed to satisfy these requirements. + + e) Convey the object code using peer-to-peer transmission, provided + you inform other peers where the object code and Corresponding + Source of the work are being offered to the general public at no + charge under subsection 6d. + + A separable portion of the object code, whose source code is excluded +from the Corresponding Source as a System Library, need not be +included in conveying the object code work. + + A "User Product" is either (1) a "consumer product", which means any +tangible personal property which is normally used for personal, family, +or household purposes, or (2) anything designed or sold for incorporation +into a dwelling. In determining whether a product is a consumer product, +doubtful cases shall be resolved in favor of coverage. For a particular +product received by a particular user, "normally used" refers to a +typical or common use of that class of product, regardless of the status +of the particular user or of the way in which the particular user +actually uses, or expects or is expected to use, the product. A product +is a consumer product regardless of whether the product has substantial +commercial, industrial or non-consumer uses, unless such uses represent +the only significant mode of use of the product. + + "Installation Information" for a User Product means any methods, +procedures, authorization keys, or other information required to install +and execute modified versions of a covered work in that User Product from +a modified version of its Corresponding Source. The information must +suffice to ensure that the continued functioning of the modified object +code is in no case prevented or interfered with solely because +modification has been made. + + If you convey an object code work under this section in, or with, or +specifically for use in, a User Product, and the conveying occurs as +part of a transaction in which the right of possession and use of the +User Product is transferred to the recipient in perpetuity or for a +fixed term (regardless of how the transaction is characterized), the +Corresponding Source conveyed under this section must be accompanied +by the Installation Information. But this requirement does not apply +if neither you nor any third party retains the ability to install +modified object code on the User Product (for example, the work has +been installed in ROM). + + The requirement to provide Installation Information does not include a +requirement to continue to provide support service, warranty, or updates +for a work that has been modified or installed by the recipient, or for +the User Product in which it has been modified or installed. Access to a +network may be denied when the modification itself materially and +adversely affects the operation of the network or violates the rules and +protocols for communication across the network. + + Corresponding Source conveyed, and Installation Information provided, +in accord with this section must be in a format that is publicly +documented (and with an implementation available to the public in +source code form), and must require no special password or key for +unpacking, reading or copying. + + 7. Additional Terms. + + "Additional permissions" are terms that supplement the terms of this +License by making exceptions from one or more of its conditions. +Additional permissions that are applicable to the entire Program shall +be treated as though they were included in this License, to the extent +that they are valid under applicable law. If additional permissions +apply only to part of the Program, that part may be used separately +under those permissions, but the entire Program remains governed by +this License without regard to the additional permissions. + + When you convey a copy of a covered work, you may at your option +remove any additional permissions from that copy, or from any part of +it. (Additional permissions may be written to require their own +removal in certain cases when you modify the work.) You may place +additional permissions on material, added by you to a covered work, +for which you have or can give appropriate copyright permission. + + Notwithstanding any other provision of this License, for material you +add to a covered work, you may (if authorized by the copyright holders of +that material) supplement the terms of this License with terms: + + a) Disclaiming warranty or limiting liability differently from the + terms of sections 15 and 16 of this License; or + + b) Requiring preservation of specified reasonable legal notices or + author attributions in that material or in the Appropriate Legal + Notices displayed by works containing it; or + + c) Prohibiting misrepresentation of the origin of that material, or + requiring that modified versions of such material be marked in + reasonable ways as different from the original version; or + + d) Limiting the use for publicity purposes of names of licensors or + authors of the material; or + + e) Declining to grant rights under trademark law for use of some + trade names, trademarks, or service marks; or + + f) Requiring indemnification of licensors and authors of that + material by anyone who conveys the material (or modified versions of + it) with contractual assumptions of liability to the recipient, for + any liability that these contractual assumptions directly impose on + those licensors and authors. + + All other non-permissive additional terms are considered "further +restrictions" within the meaning of section 10. If the Program as you +received it, or any part of it, contains a notice stating that it is +governed by this License along with a term that is a further +restriction, you may remove that term. If a license document contains +a further restriction but permits relicensing or conveying under this +License, you may add to a covered work material governed by the terms +of that license document, provided that the further restriction does +not survive such relicensing or conveying. + + If you add terms to a covered work in accord with this section, you +must place, in the relevant source files, a statement of the +additional terms that apply to those files, or a notice indicating +where to find the applicable terms. + + Additional terms, permissive or non-permissive, may be stated in the +form of a separately written license, or stated as exceptions; +the above requirements apply either way. + + 8. Termination. + + You may not propagate or modify a covered work except as expressly +provided under this License. Any attempt otherwise to propagate or +modify it is void, and will automatically terminate your rights under +this License (including any patent licenses granted under the third +paragraph of section 11). + + However, if you cease all violation of this License, then your +license from a particular copyright holder is reinstated (a) +provisionally, unless and until the copyright holder explicitly and +finally terminates your license, and (b) permanently, if the copyright +holder fails to notify you of the violation by some reasonable means +prior to 60 days after the cessation. + + Moreover, your license from a particular copyright holder is +reinstated permanently if the copyright holder notifies you of the +violation by some reasonable means, this is the first time you have +received notice of violation of this License (for any work) from that +copyright holder, and you cure the violation prior to 30 days after +your receipt of the notice. + + Termination of your rights under this section does not terminate the +licenses of parties who have received copies or rights from you under +this License. If your rights have been terminated and not permanently +reinstated, you do not qualify to receive new licenses for the same +material under section 10. + + 9. Acceptance Not Required for Having Copies. + + You are not required to accept this License in order to receive or +run a copy of the Program. Ancillary propagation of a covered work +occurring solely as a consequence of using peer-to-peer transmission +to receive a copy likewise does not require acceptance. However, +nothing other than this License grants you permission to propagate or +modify any covered work. These actions infringe copyright if you do +not accept this License. Therefore, by modifying or propagating a +covered work, you indicate your acceptance of this License to do so. + + 10. Automatic Licensing of Downstream Recipients. + + Each time you convey a covered work, the recipient automatically +receives a license from the original licensors, to run, modify and +propagate that work, subject to this License. You are not responsible +for enforcing compliance by third parties with this License. + + An "entity transaction" is a transaction transferring control of an +organization, or substantially all assets of one, or subdividing an +organization, or merging organizations. If propagation of a covered +work results from an entity transaction, each party to that +transaction who receives a copy of the work also receives whatever +licenses to the work the party's predecessor in interest had or could +give under the previous paragraph, plus a right to possession of the +Corresponding Source of the work from the predecessor in interest, if +the predecessor has it or can get it with reasonable efforts. + + You may not impose any further restrictions on the exercise of the +rights granted or affirmed under this License. For example, you may +not impose a license fee, royalty, or other charge for exercise of +rights granted under this License, and you may not initiate litigation +(including a cross-claim or counterclaim in a lawsuit) alleging that +any patent claim is infringed by making, using, selling, offering for +sale, or importing the Program or any portion of it. + + 11. Patents. + + A "contributor" is a copyright holder who authorizes use under this +License of the Program or a work on which the Program is based. The +work thus licensed is called the contributor's "contributor version". + + A contributor's "essential patent claims" are all patent claims +owned or controlled by the contributor, whether already acquired or +hereafter acquired, that would be infringed by some manner, permitted +by this License, of making, using, or selling its contributor version, +but do not include claims that would be infringed only as a +consequence of further modification of the contributor version. For +purposes of this definition, "control" includes the right to grant +patent sublicenses in a manner consistent with the requirements of +this License. + + Each contributor grants you a non-exclusive, worldwide, royalty-free +patent license under the contributor's essential patent claims, to +make, use, sell, offer for sale, import and otherwise run, modify and +propagate the contents of its contributor version. + + In the following three paragraphs, a "patent license" is any express +agreement or commitment, however denominated, not to enforce a patent +(such as an express permission to practice a patent or covenant not to +sue for patent infringement). To "grant" such a patent license to a +party means to make such an agreement or commitment not to enforce a +patent against the party. + + If you convey a covered work, knowingly relying on a patent license, +and the Corresponding Source of the work is not available for anyone +to copy, free of charge and under the terms of this License, through a +publicly available network server or other readily accessible means, +then you must either (1) cause the Corresponding Source to be so +available, or (2) arrange to deprive yourself of the benefit of the +patent license for this particular work, or (3) arrange, in a manner +consistent with the requirements of this License, to extend the patent +license to downstream recipients. "Knowingly relying" means you have +actual knowledge that, but for the patent license, your conveying the +covered work in a country, or your recipient's use of the covered work +in a country, would infringe one or more identifiable patents in that +country that you have reason to believe are valid. + + If, pursuant to or in connection with a single transaction or +arrangement, you convey, or propagate by procuring conveyance of, a +covered work, and grant a patent license to some of the parties +receiving the covered work authorizing them to use, propagate, modify +or convey a specific copy of the covered work, then the patent license +you grant is automatically extended to all recipients of the covered +work and works based on it. + + A patent license is "discriminatory" if it does not include within +the scope of its coverage, prohibits the exercise of, or is +conditioned on the non-exercise of one or more of the rights that are +specifically granted under this License. You may not convey a covered +work if you are a party to an arrangement with a third party that is +in the business of distributing software, under which you make payment +to the third party based on the extent of your activity of conveying +the work, and under which the third party grants, to any of the +parties who would receive the covered work from you, a discriminatory +patent license (a) in connection with copies of the covered work +conveyed by you (or copies made from those copies), or (b) primarily +for and in connection with specific products or compilations that +contain the covered work, unless you entered into that arrangement, +or that patent license was granted, prior to 28 March 2007. + + Nothing in this License shall be construed as excluding or limiting +any implied license or other defenses to infringement that may +otherwise be available to you under applicable patent law. + + 12. No Surrender of Others' Freedom. + + If conditions are imposed on you (whether by court order, agreement or +otherwise) that contradict the conditions of this License, they do not +excuse you from the conditions of this License. If you cannot convey a +covered work so as to satisfy simultaneously your obligations under this +License and any other pertinent obligations, then as a consequence you may +not convey it at all. For example, if you agree to terms that obligate you +to collect a royalty for further conveying from those to whom you convey +the Program, the only way you could satisfy both those terms and this +License would be to refrain entirely from conveying the Program. + + 13. Use with the GNU Affero General Public License. + + Notwithstanding any other provision of this License, you have +permission to link or combine any covered work with a work licensed +under version 3 of the GNU Affero General Public License into a single +combined work, and to convey the resulting work. The terms of this +License will continue to apply to the part which is the covered work, +but the special requirements of the GNU Affero General Public License, +section 13, concerning interaction through a network will apply to the +combination as such. + + 14. Revised Versions of this License. + + The Free Software Foundation may publish revised and/or new versions of +the GNU General Public License from time to time. Such new versions will +be similar in spirit to the present version, but may differ in detail to +address new problems or concerns. + + Each version is given a distinguishing version number. If the +Program specifies that a certain numbered version of the GNU General +Public License "or any later version" applies to it, you have the +option of following the terms and conditions either of that numbered +version or of any later version published by the Free Software +Foundation. If the Program does not specify a version number of the +GNU General Public License, you may choose any version ever published +by the Free Software Foundation. + + If the Program specifies that a proxy can decide which future +versions of the GNU General Public License can be used, that proxy's +public statement of acceptance of a version permanently authorizes you +to choose that version for the Program. + + Later license versions may give you additional or different +permissions. However, no additional obligations are imposed on any +author or copyright holder as a result of your choosing to follow a +later version. + + 15. Disclaimer of Warranty. + + THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY +APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT +HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY +OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, +THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR +PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM +IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF +ALL NECESSARY SERVICING, REPAIR OR CORRECTION. + + 16. Limitation of Liability. + + IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING +WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS +THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY +GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE +USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF +DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD +PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS), +EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF +SUCH DAMAGES. + + 17. Interpretation of Sections 15 and 16. + + If the disclaimer of warranty and limitation of liability provided +above cannot be given local legal effect according to their terms, +reviewing courts shall apply local law that most closely approximates +an absolute waiver of all civil liability in connection with the +Program, unless a warranty or assumption of liability accompanies a +copy of the Program in return for a fee. + + END OF TERMS AND CONDITIONS + + How to Apply These Terms to Your New Programs + + If you develop a new program, and you want it to be of the greatest +possible use to the public, the best way to achieve this is to make it +free software which everyone can redistribute and change under these terms. + + To do so, attach the following notices to the program. It is safest +to attach them to the start of each source file to most effectively +state the exclusion of warranty; and each file should have at least +the "copyright" line and a pointer to where the full notice is found. + + + Copyright (C) + + This program is free software: you can redistribute it and/or modify + it under the terms of the GNU General Public License as published by + the Free Software Foundation, either version 3 of the License, or + (at your option) any later version. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program. If not, see . + +Also add information on how to contact you by electronic and paper mail. + + If the program does terminal interaction, make it output a short +notice like this when it starts in an interactive mode: + + Copyright (C) + This program comes with ABSOLUTELY NO WARRANTY; for details type `show w'. + This is free software, and you are welcome to redistribute it + under certain conditions; type `show c' for details. + +The hypothetical commands `show w' and `show c' should show the appropriate +parts of the General Public License. Of course, your program's commands +might be different; for a GUI interface, you would use an "about box". + + You should also get your employer (if you work as a programmer) or school, +if any, to sign a "copyright disclaimer" for the program, if necessary. +For more information on this, and how to apply and follow the GNU GPL, see +. + + The GNU General Public License does not permit incorporating your program +into proprietary programs. If your program is a subroutine library, you +may consider it more useful to permit linking proprietary applications with +the library. If this is what you want to do, use the GNU Lesser General +Public License instead of this License. But first, please read +. \ No newline at end of file diff --git a/demo_android_ncnn/README.md b/demo_android_ncnn/README.md new file mode 100644 index 000000000..e9f433d71 --- /dev/null +++ b/demo_android_ncnn/README.md @@ -0,0 +1,17 @@ +# YOLOv5 NCNN Implementation + +This repo provides C++ implementation of [YOLOv5 model](https://github.com/ultralytics/yolov5) using +Tencent's NCNN framework. + +# Notes + +Currently NCNN does not support Slice operations with steps, therefore I removed the Slice operation +and replaced the input with a downscaled image and stacked it to match the channel number. This +may slightly reduce the accuracy. + +# Credits + +* [YOLOv5 by Ultralytics](https://github.com/ultralytics/yolov5) +* [NCNN by Tencent](https://github.com/tencent/ncnn) + +仅供学习。 diff --git a/demo_android_ncnn/app/.gitignore b/demo_android_ncnn/app/.gitignore new file mode 100644 index 000000000..796b96d1c --- /dev/null +++ b/demo_android_ncnn/app/.gitignore @@ -0,0 +1 @@ +/build diff --git a/demo_android_ncnn/app/build.gradle b/demo_android_ncnn/app/build.gradle new file mode 100644 index 000000000..70ed795b1 --- /dev/null +++ b/demo_android_ncnn/app/build.gradle @@ -0,0 +1,75 @@ +apply plugin: 'com.android.application' + +android { + compileSdkVersion 29 + buildToolsVersion "29.0.3" + defaultConfig { + applicationId "com.rangi.nanodet" + minSdkVersion 26 + targetSdkVersion 29 + versionCode 1 + versionName "1.0" + testInstrumentationRunner "androidx.test.runner.AndroidJUnitRunner" + externalNativeBuild { + cmake { + cppFlags "" + arguments '-DANDROID_PLATFORM=android-24', '-DANDROID_STL=c++_static', '-DANDROID_STL=c++_shared' + } + } + + ndk { + moduleName "WztNcnnJniLog" //编译后会生成 .so + ldLibs "log", "z", "m" + abiFilters "armeabi-v7a", "arm64-v8a" // opencv库太大只留需要的 + } + + multiDexEnabled true + } + buildTypes { + release { + minifyEnabled false + proguardFiles getDefaultProguardFile('proguard-android-optimize.txt'), 'proguard-rules.pro' + } + } + externalNativeBuild { + cmake { + path "src/main/cpp/CMakeLists.txt" + version "3.10.2" + } + } + sourceSets { + main { + jniLibs.srcDirs = ['libs'] + } + } + + repositories { + flatDir { + dirs 'libs' + } + } +} + +dependencies { + implementation fileTree(dir: 'libs', include: ['*.jar']) + implementation 'androidx.appcompat:appcompat:1.1.0' + implementation 'androidx.constraintlayout:constraintlayout:1.1.3' + testImplementation 'junit:junit:4.12' + androidTestImplementation 'androidx.test.ext:junit:1.1.1' + androidTestImplementation 'androidx.test.espresso:espresso-core:3.2.0' + + // Use the most recent version of CameraX, currently that is alpha04 + def camerax_version = "1.0.0-alpha05" + //noinspection GradleDependency + implementation "androidx.camera:camera-core:${camerax_version}" + //noinspection GradleDependency + implementation "androidx.camera:camera-camera2:${camerax_version}" + + implementation 'com.android.support:multidex:1.0.3' + // crash + implementation 'com.zxy.android:recovery:1.0.0' + // photoview + implementation 'com.github.chrisbanes:PhotoView:2.3.0' +// implementation 'com.bm.photoview:library:1.4.1' + +} diff --git a/demo_android_ncnn/app/proguard-rules.pro b/demo_android_ncnn/app/proguard-rules.pro new file mode 100644 index 000000000..f1b424510 --- /dev/null +++ b/demo_android_ncnn/app/proguard-rules.pro @@ -0,0 +1,21 @@ +# Add project specific ProGuard rules here. +# You can control the set of applied configuration files using the +# proguardFiles setting in build.gradle. +# +# For more details, see +# http://developer.android.com/guide/developing/tools/proguard.html + +# If your project uses WebView with JS, uncomment the following +# and specify the fully qualified class name to the JavaScript interface +# class: +#-keepclassmembers class fqcn.of.javascript.interface.for.webview { +# public *; +#} + +# Uncomment this to preserve the line number information for +# debugging stack traces. +#-keepattributes SourceFile,LineNumberTable + +# If you keep the line number information, uncomment this to +# hide the original source file name. +#-renamesourcefileattribute SourceFile diff --git a/demo_android_ncnn/app/src/androidTest/java/com/rangi/nanodet/ExampleInstrumentedTest.java b/demo_android_ncnn/app/src/androidTest/java/com/rangi/nanodet/ExampleInstrumentedTest.java new file mode 100644 index 000000000..f071f5be9 --- /dev/null +++ b/demo_android_ncnn/app/src/androidTest/java/com/rangi/nanodet/ExampleInstrumentedTest.java @@ -0,0 +1,27 @@ +package com.rangi.nanodet; + +import android.content.Context; + +import androidx.test.platform.app.InstrumentationRegistry; +import androidx.test.ext.junit.runners.AndroidJUnit4; + +import org.junit.Test; +import org.junit.runner.RunWith; + +import static org.junit.Assert.*; + +/** + * Instrumented test, which will execute on an Android device. + * + * @see Testing documentation + */ +@RunWith(AndroidJUnit4.class) +public class ExampleInstrumentedTest { + @Test + public void useAppContext() { + // Context of the app under test. + Context appContext = InstrumentationRegistry.getInstrumentation().getTargetContext(); + + assertEquals("gd.hq.yolov5", appContext.getPackageName()); + } +} diff --git a/demo_android_ncnn/app/src/main/AndroidManifest.xml b/demo_android_ncnn/app/src/main/AndroidManifest.xml new file mode 100644 index 000000000..281458369 --- /dev/null +++ b/demo_android_ncnn/app/src/main/AndroidManifest.xml @@ -0,0 +1,30 @@ + + + + + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/demo_android_ncnn/app/src/main/cpp/CMakeLists.txt b/demo_android_ncnn/app/src/main/cpp/CMakeLists.txt new file mode 100644 index 000000000..ba51a435c --- /dev/null +++ b/demo_android_ncnn/app/src/main/cpp/CMakeLists.txt @@ -0,0 +1,102 @@ +# For more information about using CMake with Android Studio, read the +# documentation: https://d.android.com/studio/projects/add-native-code.html + +# Sets the minimum version of CMake required to build the native library. + +cmake_minimum_required(VERSION 3.4.1) +set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -fopenmp") +set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -fopenmp") +if(DEFINED ANDROID_NDK_MAJOR AND ${ANDROID_NDK_MAJOR} GREATER 20) + set(CMAKE_SHARED_LINKER_FLAGS "${CMAKE_SHARED_LINKER_FLAGS} -static-openmp") +endif() +# Creates and names a library, sets it as either STATIC +# or SHARED, and provides the relative paths to its source code. +# You can define multiple libraries, and CMake builds them for you. +# Gradle automatically packages shared libraries with your APK. + +add_library( # Sets the name of the library. + yolov5 + + # Sets the library as a shared library. + SHARED + + # Provides a relative path to your source file(s). + jni_interface.cpp + YoloV5.cpp + YoloV4.cpp + NanoDet.cpp + ) + +include_directories( + ncnnvulkan/include +# ocr +) +# Searches for a specified prebuilt library and stores the path as a +# variable. Because CMake includes system libraries in the search path by +# default, you only need to specify the name of the public NDK library +# you want to add. CMake verifies that the library exists before +# completing its build. + +find_library(log-lib log) +find_library(android-lib android) +find_library(vulkan-lib vulkan) +find_library(jnigraphics-lib jnigraphics) +add_library( ncnn STATIC IMPORTED ) +set_target_properties( # Specifies the target library. + ncnn + + # Specifies the parameter you want to define. + PROPERTIES IMPORTED_LOCATION + + # Provides the path to the library you want to import. + ${CMAKE_SOURCE_DIR}/ncnnvulkan/${ANDROID_ABI}/libncnn.a ) + + +# ncnnvulkan +add_library(glslang STATIC IMPORTED) +add_library(OGLCompiler STATIC IMPORTED) +add_library(OSDependent STATIC IMPORTED) +add_library(SPIRV STATIC IMPORTED) +set_target_properties(glslang PROPERTIES IMPORTED_LOCATION ${CMAKE_SOURCE_DIR}/ncnnvulkan/${ANDROID_ABI}/libglslang.a) +set_target_properties(OGLCompiler PROPERTIES IMPORTED_LOCATION ${CMAKE_SOURCE_DIR}/ncnnvulkan/${ANDROID_ABI}/libOGLCompiler.a) +set_target_properties(OSDependent PROPERTIES IMPORTED_LOCATION ${CMAKE_SOURCE_DIR}/ncnnvulkan/${ANDROID_ABI}/libOSDependent.a) +set_target_properties(SPIRV PROPERTIES IMPORTED_LOCATION ${CMAKE_SOURCE_DIR}/ncnnvulkan/${ANDROID_ABI}/libSPIRV.a) + + +#include_directories( +# ${CMAKE_SOURCE_DIR}/opencv/include/ +#) +#add_library(libopencv_java4 STATIC IMPORTED) +#set_target_properties( +# libopencv_java4 +# PROPERTIES IMPORTED_LOCATION +# ${CMAKE_SOURCE_DIR}/opencv/${ANDROID_ABI}/libopencv_java4.so +# ) +target_link_libraries( # Specifies the target library. + yolov5 + + # Links the target library to the log library + # included in the NDK. + ${log-lib} + ${vulkan-lib} + ${android-lib} + ${jnigraphics-lib} +# libopencv_java4 + ) + + +# Specifies libraries CMake should link to your target library. You +# can link multiple libraries, such as libraries you define in this +# build script, prebuilt third-party libraries, or system libraries. + +target_link_libraries( # Specifies the target library. + yolov5 + + # Links the target library to the log library + # included in the NDK. + ${log-lib} + ${vulkan-lib} + ${android-lib} + ${jnigraphics-lib} + ncnn + glslang SPIRV OGLCompiler OSDependent ) \ No newline at end of file diff --git a/demo_android_ncnn/app/src/main/cpp/NanoDet.cpp b/demo_android_ncnn/app/src/main/cpp/NanoDet.cpp new file mode 100644 index 000000000..f53276896 --- /dev/null +++ b/demo_android_ncnn/app/src/main/cpp/NanoDet.cpp @@ -0,0 +1,202 @@ +// +// Create by RangiLyu +// 2020 / 10 / 2 +// +#include "NanoDet.h" + +bool NanoDet::hasGPU = true; +NanoDet* NanoDet::detector = nullptr; + +inline float fast_exp(float x) +{ + union { + uint32_t i; + float f; + } v{}; + v.i = (1 << 23) * (1.4426950409 * x + 126.93490512f); + return v.f; +} + +inline float sigmoid(float x) +{ + return 1.0f / (1.0f + fast_exp(-x)); +} + +template +int activation_function_softmax(const _Tp* src, _Tp* dst, int length) +{ + const _Tp alpha = *std::max_element(src, src + length); + _Tp denominator{ 0 }; + + for (int i = 0; i < length; ++i) { + dst[i] = fast_exp(src[i] - alpha); + denominator += dst[i]; + } + + for (int i = 0; i < length; ++i) { + dst[i] /= denominator; + } + + return 0; +} + +NanoDet::NanoDet(AAssetManager *mgr, const char *param, const char *bin, bool useGPU) { + this->Net = new ncnn::Net(); + // opt 需要在加载前设置 + hasGPU = ncnn::get_gpu_count() > 0; + this->Net->opt.use_vulkan_compute = false; //hasGPU && useGPU; // gpu + this->Net->opt.use_fp16_arithmetic = true; // fp16运算加速 + this->Net->opt.use_fp16_packed = true; + this->Net->opt.use_fp16_storage = true; + this->Net->load_param(mgr, param); + this->Net->load_model(mgr, bin); +} + +NanoDet::~NanoDet() +{ + delete this->Net; +} + +void NanoDet::preprocess(JNIEnv *env, jobject image, ncnn::Mat& in) +{ + in = ncnn::Mat::from_android_bitmap_resize(env, image, ncnn::Mat::PIXEL_RGBA2BGR, input_size, input_size); +// in = ncnn::Mat::from_pixels(image.data, ncnn::Mat::PIXEL_BGR, img_w, img_h); + //in = ncnn::Mat::from_pixels_resize(image.data, ncnn::Mat::PIXEL_BGR, img_w, img_h, this->input_width, this->input_height); + + const float mean_vals[3] = { 104.04f, 113.9f, 119.8f }; + const float norm_vals[3] = { 0.013569f, 0.014312f, 0.014106f }; + in.substract_mean_normalize(mean_vals, norm_vals); +} + +std::vector NanoDet::detect(JNIEnv *env, jobject image, float score_threshold, float nms_threshold) { + AndroidBitmapInfo img_size; + AndroidBitmap_getInfo(env, image, &img_size); + float width_ratio = (float) img_size.width / (float) this->input_size; + float height_ratio = (float) img_size.height / (float) this->input_size; + + ncnn::Mat input; + this->preprocess(env, image, input); + + auto ex = this->Net->create_extractor(); + ex.set_light_mode(true); + ex.set_num_threads(4); + hasGPU = ncnn::get_gpu_count() > 0; + //ex.set_vulkan_compute(hasGPU); + ex.input("input.1", input); + std::vector> results; + results.resize(this->num_class); + + for (const auto& head_info : this->heads_info) + { + ncnn::Mat dis_pred; + ncnn::Mat cls_pred; + ex.extract(head_info.dis_layer.c_str(), dis_pred); + ex.extract(head_info.cls_layer.c_str(), cls_pred); + + this->decode_infer(cls_pred, dis_pred, head_info.stride, score_threshold, results, width_ratio, height_ratio); + } + + std::vector dets; + for (int i = 0; i < (int)results.size(); i++) + { + this->nms(results[i], nms_threshold); + + for (auto box : results[i]) + { + dets.push_back(box); + } + } + return dets; +} + + +void NanoDet::decode_infer(ncnn::Mat& cls_pred, ncnn::Mat& dis_pred, int stride, float threshold, std::vector>& results, float width_ratio, float height_ratio) +{ + int feature_h = this->input_size / stride; + int feature_w = this->input_size / stride; + + //cv::Mat debug_heatmap = cv::Mat(feature_h, feature_w, CV_8UC3); + for (int idx = 0; idx < feature_h * feature_w; idx++) + { + const float* scores = cls_pred.row(idx); + int row = idx / feature_w; + int col = idx % feature_w; + float score = 0; + int cur_label = 0; + for (int label = 0; label < this->num_class; label++) + { + if (scores[label] > score) + { + score = scores[label]; + cur_label = label; + } + } + if (score > threshold) + { + //std::cout << "label:" << cur_label << " score:" << score << std::endl; + const float* bbox_pred = dis_pred.row(idx); + results[cur_label].push_back(this->disPred2Bbox(bbox_pred, cur_label, score, col, row, stride, width_ratio, height_ratio)); + //debug_heatmap.at(row, col)[0] = 255; + //cv::imshow("debug", debug_heatmap); + } + + } +} + +BoxInfo NanoDet::disPred2Bbox(const float*& dfl_det, int label, float score, int x, int y, int stride, float width_ratio, float height_ratio) +{ + float ct_x = (x + 0.5) * stride; + float ct_y = (y + 0.5) * stride; + std::vector dis_pred; + dis_pred.resize(4); + for (int i = 0; i < 4; i++) + { + float dis = 0; + float* dis_after_sm = new float[this->reg_max + 1]; + activation_function_softmax(dfl_det + i * (this->reg_max + 1), dis_after_sm, this->reg_max + 1); + for (int j = 0; j < this->reg_max + 1; j++) + { + dis += j * dis_after_sm[j]; + } + dis *= stride; + //std::cout << "dis:" << dis << std::endl; + dis_pred[i] = dis; + delete[] dis_after_sm; + } + float xmin = (std::max)(ct_x - dis_pred[0], .0f) * width_ratio; + float ymin = (std::max)(ct_y - dis_pred[1], .0f) * height_ratio; + float xmax = (std::min)(ct_x + dis_pred[2], (float)this->input_size) * width_ratio; + float ymax = (std::min)(ct_y + dis_pred[3], (float)this->input_size) * height_ratio; + + //std::cout << xmin << "," << ymin << "," << xmax << "," << xmax << "," << std::endl; + return BoxInfo { xmin, ymin, xmax, ymax, score, label }; +} + +void NanoDet::nms(std::vector& input_boxes, float NMS_THRESH) +{ + std::sort(input_boxes.begin(), input_boxes.end(), [](BoxInfo a, BoxInfo b) { return a.score > b.score; }); + std::vector vArea(input_boxes.size()); + for (int i = 0; i < int(input_boxes.size()); ++i) { + vArea[i] = (input_boxes.at(i).x2 - input_boxes.at(i).x1 + 1) + * (input_boxes.at(i).y2 - input_boxes.at(i).y1 + 1); + } + for (int i = 0; i < int(input_boxes.size()); ++i) { + for (int j = i + 1; j < int(input_boxes.size());) { + float xx1 = (std::max)(input_boxes[i].x1, input_boxes[j].x1); + float yy1 = (std::max)(input_boxes[i].y1, input_boxes[j].y1); + float xx2 = (std::min)(input_boxes[i].x2, input_boxes[j].x2); + float yy2 = (std::min)(input_boxes[i].y2, input_boxes[j].y2); + float w = (std::max)(float(0), xx2 - xx1 + 1); + float h = (std::max)(float(0), yy2 - yy1 + 1); + float inter = w * h; + float ovr = inter / (vArea[i] + vArea[j] - inter); + if (ovr >= NMS_THRESH) { + input_boxes.erase(input_boxes.begin() + j); + vArea.erase(vArea.begin() + j); + } + else { + j++; + } + } + } +} diff --git a/demo_android_ncnn/app/src/main/cpp/NanoDet.h b/demo_android_ncnn/app/src/main/cpp/NanoDet.h new file mode 100644 index 000000000..40759b97c --- /dev/null +++ b/demo_android_ncnn/app/src/main/cpp/NanoDet.h @@ -0,0 +1,60 @@ +// +// Create by RangiLyu +// 2020 / 10 / 2 +// + +#ifndef NANODET_H +#define NANODET_H + +#include "ncnn/net.h" +#include "YoloV5.h" + +typedef struct HeadInfo +{ + std::string cls_layer; + std::string dis_layer; + int stride; +}; + + +class NanoDet{ +public: + NanoDet(AAssetManager *mgr, const char *param, const char *bin, bool useGPU); + + ~NanoDet(); + + std::vector detect(JNIEnv *env, jobject image, float score_threshold, float nms_threshold); + std::vector labels{"person", "bicycle", "car", "motorcycle", "airplane", "bus", "train", "truck", "boat", "traffic light", + "fire hydrant", "stop sign", "parking meter", "bench", "bird", "cat", "dog", "horse", "sheep", "cow", + "elephant", "bear", "zebra", "giraffe", "backpack", "umbrella", "handbag", "tie", "suitcase", "frisbee", + "skis", "snowboard", "sports ball", "kite", "baseball bat", "baseball glove", "skateboard", "surfboard", + "tennis racket", "bottle", "wine glass", "cup", "fork", "knife", "spoon", "bowl", "banana", "apple", + "sandwich", "orange", "broccoli", "carrot", "hot dog", "pizza", "donut", "cake", "chair", "couch", + "potted plant", "bed", "dining table", "toilet", "tv", "laptop", "mouse", "remote", "keyboard", "cell phone", + "microwave", "oven", "toaster", "sink", "refrigerator", "book", "clock", "vase", "scissors", "teddy bear", + "hair drier", "toothbrush"}; +private: + void preprocess(JNIEnv *env, jobject image, ncnn::Mat& in); + void decode_infer(ncnn::Mat& cls_pred, ncnn::Mat& dis_pred, int stride, float threshold, std::vector>& results, float width_ratio, float height_ratio); + BoxInfo disPred2Bbox(const float*& dfl_det, int label, float score, int x, int y, int stride, float width_ratio, float height_ratio); + + static void nms(std::vector& result, float nms_threshold); + + ncnn::Net *Net; + int input_size = 320; + int num_class = 80; + int reg_max = 10; + std::vector heads_info{ + // cls_pred|dis_pred|stride + {"839", "842", 8}, + {"872", "875", 16}, + {"905", "908", 32}, + }; + +public: + static NanoDet *detector; + static bool hasGPU; +}; + + +#endif //NANODET_H diff --git a/demo_android_ncnn/app/src/main/cpp/YoloV4.cpp b/demo_android_ncnn/app/src/main/cpp/YoloV4.cpp new file mode 100644 index 000000000..ef2a87115 --- /dev/null +++ b/demo_android_ncnn/app/src/main/cpp/YoloV4.cpp @@ -0,0 +1,72 @@ +#include "YoloV4.h" + +bool YoloV4::hasGPU = true; +YoloV4 *YoloV4::detector = nullptr; + +YoloV4::YoloV4(AAssetManager *mgr, const char *param, const char *bin, bool useGPU) { + Net = new ncnn::Net(); + // opt 需要在加载前设置 + hasGPU = ncnn::get_gpu_count() > 0; + Net->opt.use_vulkan_compute = hasGPU && useGPU; // gpu + Net->opt.use_fp16_arithmetic = true; // fp16运算加速 + Net->load_param(mgr, param); + Net->load_model(mgr, bin); +} + +YoloV4::~YoloV4() { + delete Net; +} + +std::vector YoloV4::detect(JNIEnv *env, jobject image, float threshold, float nms_threshold) { + AndroidBitmapInfo img_size; + AndroidBitmap_getInfo(env, image, &img_size); + ncnn::Mat in_net = ncnn::Mat::from_android_bitmap_resize(env, image, ncnn::Mat::PIXEL_RGBA2RGB, input_size, + input_size); + float norm[3] = {1 / 255.f, 1 / 255.f, 1 / 255.f}; + float mean[3] = {0, 0, 0}; + in_net.substract_mean_normalize(mean, norm); + auto ex = Net->create_extractor(); + ex.set_light_mode(true); + ex.set_num_threads(4); + hasGPU = ncnn::get_gpu_count() > 0; + ex.set_vulkan_compute(hasGPU); + ex.input(0, in_net); + std::vector result; + ncnn::Mat blob; + ex.extract("output", blob); + auto boxes = decode_infer(blob, {(int) img_size.width, (int) img_size.height}, input_size, num_class, threshold); + result.insert(result.begin(), boxes.begin(), boxes.end()); +// nms(result,nms_threshold); + return result; +} + +inline float fast_exp(float x) { + union { + uint32_t i; + float f; + } v{}; + v.i = (1 << 23) * (1.4426950409 * x + 126.93490512f); + return v.f; +} + +inline float sigmoid(float x) { + return 1.0f / (1.0f + fast_exp(-x)); +} + +std::vector +YoloV4::decode_infer(ncnn::Mat &data, const yolocv::YoloSize &frame_size, int net_size, int num_classes, float threshold) { + std::vector result; + for (int i = 0; i < data.h; i++) { + BoxInfo box; + const float *values = data.row(i); + box.label = values[0] - 1; + box.score = values[1]; + box.x1 = values[2] * (float) frame_size.width; + box.y1 = values[3] * (float) frame_size.height; + box.x2 = values[4] * (float) frame_size.width; + box.y2 = values[5] * (float) frame_size.height; + result.push_back(box); + } + return result; +} + diff --git a/demo_android_ncnn/app/src/main/cpp/YoloV4.h b/demo_android_ncnn/app/src/main/cpp/YoloV4.h new file mode 100644 index 000000000..7eee00150 --- /dev/null +++ b/demo_android_ncnn/app/src/main/cpp/YoloV4.h @@ -0,0 +1,38 @@ +#ifndef YOLOV4_H +#define YOLOV4_H + +#include "ncnn/net.h" +#include "YoloV5.h" + + +class YoloV4 { +public: + YoloV4(AAssetManager *mgr, const char *param, const char *bin, bool useGPU); + + ~YoloV4(); + + std::vector detect(JNIEnv *env, jobject image, float threshold, float nms_threshold); + std::vector labels{"person", "bicycle", "car", "motorcycle", "airplane", "bus", "train", "truck", "boat", "traffic light", + "fire hydrant", "stop sign", "parking meter", "bench", "bird", "cat", "dog", "horse", "sheep", "cow", + "elephant", "bear", "zebra", "giraffe", "backpack", "umbrella", "handbag", "tie", "suitcase", "frisbee", + "skis", "snowboard", "sports ball", "kite", "baseball bat", "baseball glove", "skateboard", "surfboard", + "tennis racket", "bottle", "wine glass", "cup", "fork", "knife", "spoon", "bowl", "banana", "apple", + "sandwich", "orange", "broccoli", "carrot", "hot dog", "pizza", "donut", "cake", "chair", "couch", + "potted plant", "bed", "dining table", "toilet", "tv", "laptop", "mouse", "remote", "keyboard", "cell phone", + "microwave", "oven", "toaster", "sink", "refrigerator", "book", "clock", "vase", "scissors", "teddy bear", + "hair drier", "toothbrush"}; +private: + static std::vector + decode_infer(ncnn::Mat &data, const yolocv::YoloSize &frame_size, int net_size, int num_classes, float threshold); + +// static void nms(std::vector& result,float nms_threshold); + ncnn::Net *Net; + int input_size = 640 / 2; + int num_class = 80; +public: + static YoloV4 *detector; + static bool hasGPU; +}; + + +#endif //YOLOV4_H diff --git a/demo_android_ncnn/app/src/main/cpp/YoloV5.cpp b/demo_android_ncnn/app/src/main/cpp/YoloV5.cpp new file mode 100644 index 000000000..0a6642d56 --- /dev/null +++ b/demo_android_ncnn/app/src/main/cpp/YoloV5.cpp @@ -0,0 +1,132 @@ +// +// Created by 邓昊晴 on 14/6/2020. +// + +#include "YoloV5.h" + +bool YoloV5::hasGPU = true; +YoloV5 *YoloV5::detector = nullptr; + +YoloV5::YoloV5(AAssetManager *mgr, const char *param, const char *bin, bool useGPU) { + Net = new ncnn::Net(); + // opt 需要在加载前设置 + hasGPU = ncnn::get_gpu_count() > 0; + Net->opt.use_vulkan_compute = hasGPU && useGPU; // gpu + Net->opt.use_fp16_arithmetic = true; // fp16运算加速 + Net->load_param(mgr, param); + Net->load_model(mgr, bin); +} + +YoloV5::~YoloV5() { + delete Net; +} + +std::vector YoloV5::detect(JNIEnv *env, jobject image, float threshold, float nms_threshold) { + AndroidBitmapInfo img_size; + AndroidBitmap_getInfo(env, image, &img_size); +// ncnn::Mat in_net = ncnn::Mat::from_android_bitmap_resize(env,image,ncnn::Mat::PIXEL_BGR2RGB,input_size/2,input_size/2); + ncnn::Mat in_net = ncnn::Mat::from_android_bitmap_resize(env, image, ncnn::Mat::PIXEL_RGBA2RGB, input_size / 2, + input_size / 2); + float norm[3] = {1 / 255.f, 1 / 255.f, 1 / 255.f}; + float mean[3] = {0, 0, 0}; + in_net.substract_mean_normalize(mean, norm); + auto ex = Net->create_extractor(); + ex.set_light_mode(true); + ex.set_num_threads(4); + hasGPU = ncnn::get_gpu_count() > 0; + ex.set_vulkan_compute(hasGPU); + ex.input(0, in_net); + std::vector result; + for (const auto &layer: layers) { + ncnn::Mat blob; + ex.extract(layer.name.c_str(), blob); + auto boxes = decode_infer(blob, layer.stride, {(int) img_size.width, (int) img_size.height}, input_size, + num_class, layer.anchors, threshold); + result.insert(result.begin(), boxes.begin(), boxes.end()); + } + nms(result, nms_threshold); + return result; +} + +inline float fast_exp(float x) { + union { + uint32_t i; + float f; + } v{}; + v.i = (1 << 23) * (1.4426950409 * x + 126.93490512f); + return v.f; +} + +inline float sigmoid(float x) { + return 1.0f / (1.0f + fast_exp(-x)); +} + +std::vector +YoloV5::decode_infer(ncnn::Mat &data, int stride, const yolocv::YoloSize &frame_size, int net_size, int num_classes, + const std::vector &anchors, float threshold) { + std::vector result; + int grid_size = int(sqrt(data.h)); + float *mat_data[data.c]; + for (int i = 0; i < data.c; i++) { + mat_data[i] = data.channel(i); + } + float cx, cy, w, h; + for (int shift_y = 0; shift_y < grid_size; shift_y++) { + for (int shift_x = 0; shift_x < grid_size; shift_x++) { + int loc = shift_x + shift_y * grid_size; + for (int i = 0; i < 3; i++) { + float *record = mat_data[i]; + float *cls_ptr = record + 5; + for (int cls = 0; cls < num_classes; cls++) { + float score = sigmoid(cls_ptr[cls]) * sigmoid(record[4]); + if (score > threshold) { + cx = (sigmoid(record[0]) * 2.f - 0.5f + (float) shift_x) * (float) stride; + cy = (sigmoid(record[1]) * 2.f - 0.5f + (float) shift_y) * (float) stride; + w = pow(sigmoid(record[2]) * 2.f, 2) * anchors[i].width; + h = pow(sigmoid(record[3]) * 2.f, 2) * anchors[i].height; + //printf("[grid size=%d, stride = %d]x y w h %f %f %f %f\n",grid_size,stride,record[0],record[1],record[2],record[3]); + BoxInfo box; + box.x1 = std::max(0, std::min(frame_size.width, int((cx - w / 2.f) * (float) frame_size.width / (float) net_size))); + box.y1 = std::max(0, std::min(frame_size.height, int((cy - h / 2.f) * (float) frame_size.height / (float) net_size))); + box.x2 = std::max(0, std::min(frame_size.width, int((cx + w / 2.f) * (float) frame_size.width / (float) net_size))); + box.y2 = std::max(0, std::min(frame_size.height, int((cy + h / 2.f) * (float) frame_size.height / (float) net_size))); + box.score = score; + box.label = cls; + result.push_back(box); + } + } + } + for (auto &ptr:mat_data) { + ptr += (num_classes + 5); + } + } + } + return result; +} + +void YoloV5::nms(std::vector &input_boxes, float NMS_THRESH) { + std::sort(input_boxes.begin(), input_boxes.end(), [](BoxInfo a, BoxInfo b) { return a.score > b.score; }); + std::vector vArea(input_boxes.size()); + for (int i = 0; i < int(input_boxes.size()); ++i) { + vArea[i] = (input_boxes.at(i).x2 - input_boxes.at(i).x1 + 1) + * (input_boxes.at(i).y2 - input_boxes.at(i).y1 + 1); + } + for (int i = 0; i < int(input_boxes.size()); ++i) { + for (int j = i + 1; j < int(input_boxes.size());) { + float xx1 = std::max(input_boxes[i].x1, input_boxes[j].x1); + float yy1 = std::max(input_boxes[i].y1, input_boxes[j].y1); + float xx2 = std::min(input_boxes[i].x2, input_boxes[j].x2); + float yy2 = std::min(input_boxes[i].y2, input_boxes[j].y2); + float w = std::max(float(0), xx2 - xx1 + 1); + float h = std::max(float(0), yy2 - yy1 + 1); + float inter = w * h; + float ovr = inter / (vArea[i] + vArea[j] - inter); + if (ovr >= NMS_THRESH) { + input_boxes.erase(input_boxes.begin() + j); + vArea.erase(vArea.begin() + j); + } else { + j++; + } + } + } +} diff --git a/demo_android_ncnn/app/src/main/cpp/YoloV5.h b/demo_android_ncnn/app/src/main/cpp/YoloV5.h new file mode 100644 index 000000000..09397b0fa --- /dev/null +++ b/demo_android_ncnn/app/src/main/cpp/YoloV5.h @@ -0,0 +1,70 @@ +// +// Created by 邓昊晴 on 14/6/2020. +// + +#ifndef YOLOV5_H +#define YOLOV5_H + +#include "ncnn/net.h" + +namespace yolocv { + typedef struct { + int width; + int height; + } YoloSize; +} + +typedef struct { + std::string name; + int stride; + std::vector anchors; +} YoloLayerData; + +typedef struct BoxInfo { + float x1; + float y1; + float x2; + float y2; + float score; + int label; +} BoxInfo; + +class YoloV5 { +public: + YoloV5(AAssetManager *mgr, const char *param, const char *bin, bool useGPU); + + ~YoloV5(); + + std::vector detect(JNIEnv *env, jobject image, float threshold, float nms_threshold); + std::vector labels{"person", "bicycle", "car", "motorcycle", "airplane", "bus", "train", "truck", "boat", "traffic light", + "fire hydrant", "stop sign", "parking meter", "bench", "bird", "cat", "dog", "horse", "sheep", "cow", + "elephant", "bear", "zebra", "giraffe", "backpack", "umbrella", "handbag", "tie", "suitcase", "frisbee", + "skis", "snowboard", "sports ball", "kite", "baseball bat", "baseball glove", "skateboard", "surfboard", + "tennis racket", "bottle", "wine glass", "cup", "fork", "knife", "spoon", "bowl", "banana", "apple", + "sandwich", "orange", "broccoli", "carrot", "hot dog", "pizza", "donut", "cake", "chair", "couch", + "potted plant", "bed", "dining table", "toilet", "tv", "laptop", "mouse", "remote", "keyboard", "cell phone", + "microwave", "oven", "toaster", "sink", "refrigerator", "book", "clock", "vase", "scissors", "teddy bear", + "hair drier", "toothbrush"}; +private: + static std::vector + decode_infer(ncnn::Mat &data, int stride, const yolocv::YoloSize &frame_size, int net_size, int num_classes, + const std::vector &anchors, float threshold); + + static void nms(std::vector &result, float nms_threshold); + + ncnn::Net *Net; + int input_size = 640; + int num_class = 80; + std::vector layers{ + {"394", 32, {{116, 90}, {156, 198}, {373, 326}}}, + {"375", 16, {{30, 61}, {62, 45}, {59, 119}}}, + {"output", 8, {{10, 13}, {16, 30}, {33, 23}}}, + }; + +public: + static YoloV5 *detector; + static bool hasGPU; +}; + + +#endif //YOLOV5_H diff --git a/demo_android_ncnn/app/src/main/cpp/jni_interface.cpp b/demo_android_ncnn/app/src/main/cpp/jni_interface.cpp new file mode 100644 index 000000000..6e48460ff --- /dev/null +++ b/demo_android_ncnn/app/src/main/cpp/jni_interface.cpp @@ -0,0 +1,126 @@ +#include +#include +#include +#include +#include +#include "YoloV5.h" +#include "YoloV4.h" +#include "NanoDet.h" + + +JNIEXPORT jint JNI_OnLoad(JavaVM *vm, void *reserved) { + ncnn::create_gpu_instance(); + if (ncnn::get_gpu_count() > 0) { + YoloV5::hasGPU = true; + YoloV4::hasGPU = true; + NanoDet::hasGPU = true; + } + return JNI_VERSION_1_6; +} + +JNIEXPORT void JNI_OnUnload(JavaVM *vm, void *reserved) { + ncnn::destroy_gpu_instance(); +} + + +/********************************************************************************************* + Yolov5 + ********************************************************************************************/ +extern "C" JNIEXPORT void JNICALL +Java_com_rangi_nanodet_YOLOv5_init(JNIEnv *env, jclass, jobject assetManager, jboolean useGPU) { + if (YoloV5::detector == nullptr) { + AAssetManager *mgr = AAssetManager_fromJava(env, assetManager); + YoloV5::detector = new YoloV5(mgr, "yolov5.param", "yolov5.bin", useGPU); + } +} + +extern "C" JNIEXPORT jobjectArray JNICALL +Java_com_rangi_nanodet_YOLOv5_detect(JNIEnv *env, jclass, jobject image, jdouble threshold, jdouble nms_threshold) { + auto result = YoloV5::detector->detect(env, image, threshold, nms_threshold); + + auto box_cls = env->FindClass("com/rangi/nanodet/Box"); + auto cid = env->GetMethodID(box_cls, "", "(FFFFIF)V"); + jobjectArray ret = env->NewObjectArray(result.size(), box_cls, nullptr); + int i = 0; + for (auto &box:result) { + env->PushLocalFrame(1); + jobject obj = env->NewObject(box_cls, cid, box.x1, box.y1, box.x2, box.y2, box.label, box.score); + obj = env->PopLocalFrame(obj); + env->SetObjectArrayElement(ret, i++, obj); + } + return ret; +} + +/********************************************************************************************* + YOLOv4-tiny + yolov4官方ncnn模型下载地址 + darknet2ncnn:https://drive.google.com/drive/folders/1YzILvh0SKQPS_lrb33dmGNq7aVTKPWS0 + ********************************************************************************************/ + +// 20200813 增加 MobileNetV2-YOLOv3-Nano-coco + +extern "C" JNIEXPORT void JNICALL +Java_com_rangi_nanodet_YOLOv4_init(JNIEnv *env, jclass, jobject assetManager, jboolean v4tiny, jboolean useGPU) { + if (YoloV4::detector != nullptr) { + delete YoloV4::detector; + YoloV4::detector = nullptr; + } + if (YoloV4::detector == nullptr) { + AAssetManager *mgr = AAssetManager_fromJava(env, assetManager); + if (v4tiny == 1) { + YoloV4::detector = new YoloV4(mgr, "yolov4-tiny-opt.param", "yolov4-tiny-opt.bin", useGPU); + } else if (v4tiny == 0) { + YoloV4::detector = new YoloV4(mgr, "MobileNetV2-YOLOv3-Nano-coco.param", + "MobileNetV2-YOLOv3-Nano-coco.bin", useGPU); +// YoloV4::detector = new YoloV4(mgr,"export_demo.param","export_demo.bin"); + } + } +} + +extern "C" JNIEXPORT jobjectArray JNICALL +Java_com_rangi_nanodet_YOLOv4_detect(JNIEnv *env, jclass, jobject image, jdouble threshold, jdouble nms_threshold) { + auto result = YoloV4::detector->detect(env, image, threshold, nms_threshold); + + auto box_cls = env->FindClass("com/rangi/nanodet/Box"); + auto cid = env->GetMethodID(box_cls, "", "(FFFFIF)V"); + jobjectArray ret = env->NewObjectArray(result.size(), box_cls, nullptr); + int i = 0; + for (auto &box:result) { + env->PushLocalFrame(1); + jobject obj = env->NewObject(box_cls, cid, box.x1, box.y1, box.x2, box.y2, box.label, box.score); + obj = env->PopLocalFrame(obj); + env->SetObjectArrayElement(ret, i++, obj); + } + return ret; +} + +/********************************************************************************************* + NanoDet + ********************************************************************************************/ + +extern "C" JNIEXPORT void JNICALL +Java_com_rangi_nanodet_NanoDet_init(JNIEnv *env, jclass, jobject assetManager, jboolean useGPU) { + if (NanoDet::detector == nullptr) { + AAssetManager *mgr = AAssetManager_fromJava(env, assetManager); + NanoDet::detector = new NanoDet(mgr, "nanodet.param", "nanodet.bin", useGPU); + } +} + +extern "C" JNIEXPORT jobjectArray JNICALL +Java_com_rangi_nanodet_NanoDet_detect(JNIEnv *env, jclass, jobject image, jdouble threshold, jdouble nms_threshold) { + auto result = NanoDet::detector->detect(env, image, threshold, nms_threshold); + + auto box_cls = env->FindClass("com/rangi/nanodet/Box"); + auto cid = env->GetMethodID(box_cls, "", "(FFFFIF)V"); + jobjectArray ret = env->NewObjectArray(result.size(), box_cls, nullptr); + int i = 0; + for (auto &box:result) { + env->PushLocalFrame(1); + jobject obj = env->NewObject(box_cls, cid, box.x1, box.y1, box.x2, box.y2, box.label, box.score); + obj = env->PopLocalFrame(obj); + env->SetObjectArrayElement(ret, i++, obj); + } + return ret; +} + + diff --git a/demo_android_ncnn/app/src/main/java/com/rangi/nanodet/AppCrashHandler.java b/demo_android_ncnn/app/src/main/java/com/rangi/nanodet/AppCrashHandler.java new file mode 100644 index 000000000..3296f0be2 --- /dev/null +++ b/demo_android_ncnn/app/src/main/java/com/rangi/nanodet/AppCrashHandler.java @@ -0,0 +1,18 @@ +package com.rangi.nanodet; + +import androidx.annotation.NonNull; + +class AppCrashHandler implements Thread.UncaughtExceptionHandler { + + private Thread.UncaughtExceptionHandler uncaughtExceptionHandler = Thread.getDefaultUncaughtExceptionHandler(); + + @Override + public void uncaughtException(@NonNull Thread t, @NonNull Throwable e) { + uncaughtExceptionHandler.uncaughtException(t, e); + } + + public static void register() { + Thread.setDefaultUncaughtExceptionHandler(new AppCrashHandler()); + } + +} diff --git a/demo_android_ncnn/app/src/main/java/com/rangi/nanodet/Box.java b/demo_android_ncnn/app/src/main/java/com/rangi/nanodet/Box.java new file mode 100644 index 000000000..d065db1ef --- /dev/null +++ b/demo_android_ncnn/app/src/main/java/com/rangi/nanodet/Box.java @@ -0,0 +1,46 @@ +package com.rangi.nanodet; + +import android.graphics.Color; +import android.graphics.RectF; + +import java.util.Random; + +public class Box { + public float x0,y0,x1,y1; + private int label; + private float score; + private static String[] labels={"person", "bicycle", "car", "motorcycle", "airplane", "bus", "train", "truck", "boat", "traffic light", + "fire hydrant", "stop sign", "parking meter", "bench", "bird", "cat", "dog", "horse", "sheep", "cow", + "elephant", "bear", "zebra", "giraffe", "backpack", "umbrella", "handbag", "tie", "suitcase", "frisbee", + "skis", "snowboard", "sports ball", "kite", "baseball bat", "baseball glove", "skateboard", "surfboard", + "tennis racket", "bottle", "wine glass", "cup", "fork", "knife", "spoon", "bowl", "banana", "apple", + "sandwich", "orange", "broccoli", "carrot", "hot dog", "pizza", "donut", "cake", "chair", "couch", + "potted plant", "bed", "dining table", "toilet", "tv", "laptop", "mouse", "remote", "keyboard", "cell phone", + "microwave", "oven", "toaster", "sink", "refrigerator", "book", "clock", "vase", "scissors", "teddy bear", + "hair drier", "toothbrush"}; + public Box(float x0,float y0, float x1, float y1, int label, float score){ + this.x0 = x0; + this.y0 = y0; + this.x1 = x1; + this.y1 = y1; + this.label = label; + this.score = score; + } + + public RectF getRect(){ + return new RectF(x0,y0,x1,y1); + } + + public String getLabel(){ + return labels[label]; + } + + public float getScore(){ + return score; + } + + public int getColor(){ + Random random = new Random(label); + return Color.argb(255,random.nextInt(256),random.nextInt(256),random.nextInt(256)); + } +} diff --git a/demo_android_ncnn/app/src/main/java/com/rangi/nanodet/MainActivity.java b/demo_android_ncnn/app/src/main/java/com/rangi/nanodet/MainActivity.java new file mode 100644 index 000000000..d381c9b1d --- /dev/null +++ b/demo_android_ncnn/app/src/main/java/com/rangi/nanodet/MainActivity.java @@ -0,0 +1,506 @@ +package com.rangi.nanodet; + +import androidx.annotation.NonNull; +import androidx.annotation.Nullable; +import androidx.appcompat.app.AppCompatActivity; +import androidx.camera.core.CameraX; +import androidx.camera.core.ImageAnalysis; +import androidx.camera.core.ImageAnalysisConfig; +import androidx.camera.core.ImageProxy; +import androidx.camera.core.Preview; +import androidx.camera.core.PreviewConfig; +import androidx.camera.core.UseCase; +import androidx.core.app.ActivityCompat; +import androidx.lifecycle.LifecycleOwner; + +import android.Manifest; +import android.content.Intent; +import android.content.pm.PackageManager; +import android.database.Cursor; +import android.graphics.Bitmap; +import android.graphics.BitmapFactory; +import android.graphics.Canvas; +import android.graphics.ImageFormat; +import android.graphics.Matrix; +import android.graphics.Paint; +import android.graphics.Rect; +import android.graphics.YuvImage; +import android.media.ExifInterface; +import android.net.Uri; +import android.os.Bundle; +import android.provider.MediaStore; +import android.util.Size; +import android.view.TextureView; +import android.view.View; +import android.view.ViewGroup; +import android.widget.Button; +import android.widget.ImageView; +import android.widget.SeekBar; +import android.widget.TextView; +import android.widget.Toast; + +import java.io.ByteArrayOutputStream; +import java.io.IOException; +import java.nio.ByteBuffer; +import java.util.Locale; +import java.util.concurrent.ExecutorService; +import java.util.concurrent.Executors; +import java.util.concurrent.atomic.AtomicBoolean; + +public class MainActivity extends AppCompatActivity { + public static int YOLOV5S = 1; + public static int YOLOV4_TINY = 2; + public static int MOBILENETV2_YOLOV3_NANO = 3; + public static int NANODET = 9; + + public static int USE_MODEL = MOBILENETV2_YOLOV3_NANO; + public static boolean USE_GPU = false; + + public static CameraX.LensFacing CAMERA_ID = CameraX.LensFacing.BACK; + + private static final int REQUEST_CAMERA = 1; + private static final int REQUEST_PICK_IMAGE = 2; + private static String[] PERMISSIONS_CAMERA = { + Manifest.permission.CAMERA + }; + private ImageView resultImageView; + private SeekBar nmsSeekBar; + private SeekBar thresholdSeekBar; + private TextView thresholdTextview; + private TextView tvInfo; + private double threshold = 0.3, nms_threshold = 0.7; + private TextureView viewFinder; + + private AtomicBoolean detecting = new AtomicBoolean(false); + private AtomicBoolean detectPhoto = new AtomicBoolean(false); + + private long startTime = 0; + private long endTime = 0; + private int width; + private int height; + + double total_fps = 0; + int fps_count = 0; + + protected Bitmap mutableBitmap; + + ExecutorService detectService = Executors.newSingleThreadExecutor(); + + @Override + protected void onCreate(Bundle savedInstanceState) { + super.onCreate(savedInstanceState); + setContentView(R.layout.activity_main); + int permission = ActivityCompat.checkSelfPermission(this, Manifest.permission.CAMERA); + if (permission != PackageManager.PERMISSION_GRANTED) { + // We don't have permission so prompt the user + ActivityCompat.requestPermissions( + this, + PERMISSIONS_CAMERA, + REQUEST_CAMERA + ); + finish(); + } + if (USE_MODEL == YOLOV5S) { + YOLOv5.init(getAssets(), USE_GPU); + } else if (USE_MODEL == YOLOV4_TINY) { + YOLOv4.init(getAssets(), true, USE_GPU); + } else if (USE_MODEL == MOBILENETV2_YOLOV3_NANO) { + YOLOv4.init(getAssets(), false, USE_GPU); + } else if (USE_MODEL == NANODET) { + NanoDet.init(getAssets(), USE_GPU); + } + resultImageView = findViewById(R.id.imageView); + thresholdTextview = findViewById(R.id.valTxtView); + tvInfo = findViewById(R.id.tv_info); + nmsSeekBar = findViewById(R.id.nms_seek); + thresholdSeekBar = findViewById(R.id.threshold_seek); + if (USE_MODEL != YOLOV5S && USE_MODEL != NANODET) { + nmsSeekBar.setEnabled(false); + thresholdSeekBar.setEnabled(false); + } else if (USE_MODEL == YOLOV5S) { + threshold = 0.3f; + nms_threshold = 0.7f; + } else if (USE_MODEL == NANODET) { + threshold = 0.4f; + nms_threshold = 0.6f; + } + nmsSeekBar.setProgress((int) (nms_threshold * 100)); + thresholdSeekBar.setProgress((int) (threshold * 100)); + final String format = "Thresh: %.2f, NMS: %.2f"; + thresholdTextview.setText(String.format(Locale.ENGLISH, format, threshold, nms_threshold)); + nmsSeekBar.setOnSeekBarChangeListener(new SeekBar.OnSeekBarChangeListener() { + @Override + public void onProgressChanged(SeekBar seekBar, int i, boolean b) { + nms_threshold = i / 100.f; + thresholdTextview.setText(String.format(Locale.ENGLISH, format, threshold, nms_threshold)); + } + + @Override + public void onStartTrackingTouch(SeekBar seekBar) { + + } + + @Override + public void onStopTrackingTouch(SeekBar seekBar) { + + } + }); + thresholdSeekBar.setOnSeekBarChangeListener(new SeekBar.OnSeekBarChangeListener() { + @Override + public void onProgressChanged(SeekBar seekBar, int i, boolean b) { + threshold = i / 100.f; + thresholdTextview.setText(String.format(Locale.ENGLISH, format, threshold, nms_threshold)); + } + + @Override + public void onStartTrackingTouch(SeekBar seekBar) { + + } + + @Override + public void onStopTrackingTouch(SeekBar seekBar) { + + } + }); + Button inference = findViewById(R.id.button); + inference.setOnClickListener(new View.OnClickListener() { + @Override + public void onClick(View view) { + Intent intent = new Intent(Intent.ACTION_PICK); + intent.setType("image/*"); + startActivityForResult(intent, REQUEST_PICK_IMAGE); + } + }); + + resultImageView.setOnClickListener(new View.OnClickListener() { + @Override + public void onClick(View v) { + detectPhoto.set(false); + } + }); + + viewFinder = findViewById(R.id.view_finder); + viewFinder.addOnLayoutChangeListener(new View.OnLayoutChangeListener() { + @Override + public void onLayoutChange(View view, int i, int i1, int i2, int i3, int i4, int i5, int i6, int i7) { + updateTransform(); + } + }); + + viewFinder.post(new Runnable() { + @Override + public void run() { + startCamera(); + } + }); + } + + private void updateTransform() { + Matrix matrix = new Matrix(); + // Compute the center of the view finder + float centerX = viewFinder.getWidth() / 2f; + float centerY = viewFinder.getHeight() / 2f; + + float[] rotations = {0, 90, 180, 270}; + // Correct preview output to account for display rotation + float rotationDegrees = rotations[viewFinder.getDisplay().getRotation()]; + + matrix.postRotate(-rotationDegrees, centerX, centerY); + + // Finally, apply transformations to our TextureView + viewFinder.setTransform(matrix); + } + + private void startCamera() { + CameraX.unbindAll(); + // 1. preview + PreviewConfig previewConfig = new PreviewConfig.Builder() + .setLensFacing(CAMERA_ID) +// .setTargetAspectRatio() // 宽高比 + .setTargetResolution(new Size(320, 320)) // 分辨率 + .build(); + + Preview preview = new Preview(previewConfig); + preview.setOnPreviewOutputUpdateListener(new Preview.OnPreviewOutputUpdateListener() { + @Override + public void onUpdated(Preview.PreviewOutput output) { + ViewGroup parent = (ViewGroup) viewFinder.getParent(); + parent.removeView(viewFinder); + parent.addView(viewFinder, 0); + + viewFinder.setSurfaceTexture(output.getSurfaceTexture()); + updateTransform(); + } + }); + DetectAnalyzer detectAnalyzer = new DetectAnalyzer(); + CameraX.bindToLifecycle((LifecycleOwner) this, preview, gainAnalyzer(detectAnalyzer)); + + } + + + private UseCase gainAnalyzer(DetectAnalyzer detectAnalyzer) { + ImageAnalysisConfig.Builder analysisConfigBuilder = new ImageAnalysisConfig.Builder(); + analysisConfigBuilder.setImageReaderMode(ImageAnalysis.ImageReaderMode.ACQUIRE_LATEST_IMAGE); + analysisConfigBuilder.setTargetResolution(new Size(320, 320)); // 输出预览图像尺寸 + ImageAnalysisConfig config = analysisConfigBuilder.build(); + ImageAnalysis analysis = new ImageAnalysis(config); + analysis.setAnalyzer(detectAnalyzer); + return analysis; + } + + private Bitmap imageToBitmap(ImageProxy image) { + byte[] nv21 = imagetToNV21(image); + + YuvImage yuvImage = new YuvImage(nv21, ImageFormat.NV21, image.getWidth(), image.getHeight(), null); + ByteArrayOutputStream out = new ByteArrayOutputStream(); + yuvImage.compressToJpeg(new Rect(0, 0, yuvImage.getWidth(), yuvImage.getHeight()), 100, out); + byte[] imageBytes = out.toByteArray(); + + return BitmapFactory.decodeByteArray(imageBytes, 0, imageBytes.length); + } + + private byte[] imagetToNV21(ImageProxy image) { + ImageProxy.PlaneProxy[] planes = image.getPlanes(); + ImageProxy.PlaneProxy y = planes[0]; + ImageProxy.PlaneProxy u = planes[1]; + ImageProxy.PlaneProxy v = planes[2]; + ByteBuffer yBuffer = y.getBuffer(); + ByteBuffer uBuffer = u.getBuffer(); + ByteBuffer vBuffer = v.getBuffer(); + int ySize = yBuffer.remaining(); + int uSize = uBuffer.remaining(); + int vSize = vBuffer.remaining(); + byte[] nv21 = new byte[ySize + uSize + vSize]; + // U and V are swapped + yBuffer.get(nv21, 0, ySize); + vBuffer.get(nv21, ySize, vSize); + uBuffer.get(nv21, ySize + vSize, uSize); + + return nv21; + } + + private class DetectAnalyzer implements ImageAnalysis.Analyzer { + + @Override + public void analyze(ImageProxy image, final int rotationDegrees) { + detectOnModel(image, rotationDegrees); + } + } + + + private void detectOnModel(ImageProxy image, final int rotationDegrees) { + if (detecting.get() || detectPhoto.get()) { + return; + } + detecting.set(true); + startTime = System.currentTimeMillis(); + final Bitmap bitmapsrc = imageToBitmap(image); // 格式转换 + if (detectService == null) { + detecting.set(false); + return; + } + detectService.execute(new Runnable() { + @Override + public void run() { + Matrix matrix = new Matrix(); + matrix.postRotate(rotationDegrees); + width = bitmapsrc.getWidth(); + height = bitmapsrc.getHeight(); + Bitmap bitmap = Bitmap.createBitmap(bitmapsrc, 0, 0, Math.min(width, height), Math.min(width, height), matrix, false); + + Box[] result = null; + if (USE_MODEL == YOLOV5S) { + result = YOLOv5.detect(bitmap, threshold, nms_threshold); + } else if (USE_MODEL == YOLOV4_TINY || USE_MODEL == MOBILENETV2_YOLOV3_NANO) { + result = YOLOv4.detect(bitmap, threshold, nms_threshold); + } else if (USE_MODEL == NANODET) { + result = NanoDet.detect(bitmap, threshold, nms_threshold); + } + if (result == null) { + detecting.set(false); + return; + } + mutableBitmap = bitmap.copy(Bitmap.Config.ARGB_8888, true); + if (USE_MODEL == YOLOV5S || USE_MODEL == YOLOV4_TINY || USE_MODEL == MOBILENETV2_YOLOV3_NANO || USE_MODEL == NANODET) { + mutableBitmap = drawBoxRects(mutableBitmap, result); + } + runOnUiThread(new Runnable() { + @Override + public void run() { + detecting.set(false); + if (detectPhoto.get()) { + return; + } + resultImageView.setImageBitmap(mutableBitmap); + endTime = System.currentTimeMillis(); + long dur = endTime - startTime; + float fps = (float) (1000.0 / dur); + total_fps = (total_fps == 0) ? fps : (total_fps + fps); + fps_count++; + String modelName = getModelName(); + + tvInfo.setText(String.format(Locale.CHINESE, + "%s\nSize: %dx%d\nTime: %.3f s\nFPS: %.3f\nAVG_FPS: %.3f", + modelName, height, width, dur / 1000.0, fps, (float) total_fps / fps_count)); + } + }); + } + }); + } + + + protected Bitmap drawBoxRects(Bitmap mutableBitmap, Box[] results) { + if (results == null || results.length <= 0) { + return mutableBitmap; + } + Canvas canvas = new Canvas(mutableBitmap); + final Paint boxPaint = new Paint(); + boxPaint.setAlpha(200); + boxPaint.setStyle(Paint.Style.STROKE); + boxPaint.setStrokeWidth(4 * mutableBitmap.getWidth() / 800.0f); + boxPaint.setTextSize(40 * mutableBitmap.getWidth() / 800.0f); + for (Box box : results) { + if (USE_MODEL == MOBILENETV2_YOLOV3_NANO) { + if (box.getScore() < 0.3f) { + // 模型比较小,置信度太低就不要了 + continue; + } + // 有时候差太多了,手动改一下 + box.x0 = box.x0 < 0 ? box.x0 / 6 : box.x0; + box.y0 = box.y0 < 0 ? box.y0 / 6 : box.y0; + } + boxPaint.setColor(box.getColor()); + boxPaint.setStyle(Paint.Style.FILL); + canvas.drawText(box.getLabel() + String.format(Locale.CHINESE, " %.3f", box.getScore()), box.x0 + 3, box.y0 + 40 * mutableBitmap.getWidth() / 1000.0f, boxPaint); + boxPaint.setStyle(Paint.Style.STROKE); + canvas.drawRect(box.getRect(), boxPaint); + } + return mutableBitmap; + } + + + protected String getModelName() { + String modelName = "ohhhhh"; + if (USE_MODEL == YOLOV5S) { + modelName = "YOLOv5s"; + } else if (USE_MODEL == YOLOV4_TINY) { + modelName = "YOLOv4-tiny"; + } else if (USE_MODEL == MOBILENETV2_YOLOV3_NANO) { + modelName = "MobileNetV2-YOLOv3-Nano"; + } else if (USE_MODEL == NANODET) { + modelName = "NanoDet"; + } + return USE_GPU ? "GPU: " + modelName : "CPU: " + modelName; + } + + @Override + protected void onDestroy() { + if (detectService != null) { + detectService.shutdown(); + detectService = null; + } + CameraX.unbindAll(); + super.onDestroy(); + } + + + @Override + public void onRequestPermissionsResult(int requestCode, @NonNull String[] permissions, @NonNull int[] grantResults) { + super.onRequestPermissionsResult(requestCode, permissions, grantResults); + for (int result : grantResults) { + if (result != PackageManager.PERMISSION_GRANTED) { + Toast.makeText(this, "Camera Permission!", Toast.LENGTH_SHORT).show(); + this.finish(); + } + } + } + + @Override + protected void onActivityResult(int requestCode, int resultCode, @Nullable Intent data) { + super.onActivityResult(requestCode, resultCode, data); + if (data == null) { + return; + } + detectPhoto.set(true); + Bitmap image = getPicture(data.getData()); + if (image == null) { + Toast.makeText(this, "Photo is null", Toast.LENGTH_SHORT).show(); + return; + } + Bitmap mutableBitmap = image.copy(Bitmap.Config.ARGB_8888, true); + + Box[] result = null; + if (USE_MODEL == YOLOV5S) { + result = YOLOv5.detect(image, threshold, nms_threshold); + } else if (USE_MODEL == YOLOV4_TINY || USE_MODEL == MOBILENETV2_YOLOV3_NANO) { + result = YOLOv4.detect(image, threshold, nms_threshold); + } else if (USE_MODEL == NANODET) { + result = NanoDet.detect(image, threshold, nms_threshold); + } + if (USE_MODEL == YOLOV5S || USE_MODEL == YOLOV4_TINY || USE_MODEL == MOBILENETV2_YOLOV3_NANO || USE_MODEL == NANODET) { + mutableBitmap = drawBoxRects(mutableBitmap, result); + } + resultImageView.setImageBitmap(mutableBitmap); + } + + public Bitmap getPicture(Uri selectedImage) { + String[] filePathColumn = {MediaStore.Images.Media.DATA}; + Cursor cursor = this.getContentResolver().query(selectedImage, filePathColumn, null, null, null); + if (cursor == null) { + return null; + } + cursor.moveToFirst(); + int columnIndex = cursor.getColumnIndex(filePathColumn[0]); + String picturePath = cursor.getString(columnIndex); + cursor.close(); + Bitmap bitmap = BitmapFactory.decodeFile(picturePath); + if (bitmap == null) { + return null; + } + int rotate = readPictureDegree(picturePath); + return rotateBitmapByDegree(bitmap, rotate); + } + + public int readPictureDegree(String path) { + int degree = 0; + try { + ExifInterface exifInterface = new ExifInterface(path); + int orientation = exifInterface.getAttributeInt(ExifInterface.TAG_ORIENTATION, ExifInterface.ORIENTATION_NORMAL); + switch (orientation) { + case ExifInterface.ORIENTATION_ROTATE_90: + degree = 90; + break; + case ExifInterface.ORIENTATION_ROTATE_180: + degree = 180; + break; + case ExifInterface.ORIENTATION_ROTATE_270: + degree = 270; + break; + } + } catch (IOException e) { + e.printStackTrace(); + } + return degree; + } + + public Bitmap rotateBitmapByDegree(Bitmap bm, int degree) { + Bitmap returnBm = null; + Matrix matrix = new Matrix(); + matrix.postRotate(degree); + try { + returnBm = Bitmap.createBitmap(bm, 0, 0, bm.getWidth(), + bm.getHeight(), matrix, true); + } catch (OutOfMemoryError e) { + e.printStackTrace(); + } + if (returnBm == null) { + returnBm = bm; + } + if (bm != returnBm) { + bm.recycle(); + } + return returnBm; + } + + +} diff --git a/demo_android_ncnn/app/src/main/java/com/rangi/nanodet/NanoDet.java b/demo_android_ncnn/app/src/main/java/com/rangi/nanodet/NanoDet.java new file mode 100644 index 000000000..f593d6f30 --- /dev/null +++ b/demo_android_ncnn/app/src/main/java/com/rangi/nanodet/NanoDet.java @@ -0,0 +1,13 @@ +package com.rangi.nanodet; + +import android.content.res.AssetManager; +import android.graphics.Bitmap; + +public class NanoDet { + static { + System.loadLibrary("yolov5"); + } + + public static native void init(AssetManager manager, boolean useGPU); + public static native Box[] detect(Bitmap bitmap, double threshold, double nms_threshold); +} diff --git a/demo_android_ncnn/app/src/main/java/com/rangi/nanodet/NcnnApp.java b/demo_android_ncnn/app/src/main/java/com/rangi/nanodet/NcnnApp.java new file mode 100644 index 000000000..92c3a7f29 --- /dev/null +++ b/demo_android_ncnn/app/src/main/java/com/rangi/nanodet/NcnnApp.java @@ -0,0 +1,68 @@ +package com.rangi.nanodet; + +import android.app.Application; +import android.content.Context; +import android.util.Log; + +import androidx.multidex.MultiDex; + +import com.zxy.recovery.callback.RecoveryCallback; +import com.zxy.recovery.core.Recovery; + + +public class NcnnApp extends Application { + + @Override + public void onCreate() { + super.onCreate(); + + //崩溃界面 + initRecovery(); + } + + @Override + protected void attachBaseContext(Context base) { + super.attachBaseContext(base); + MultiDex.install(base); + } + + private void initRecovery() { + Recovery.getInstance() + .debug(BuildConfig.DEBUG) + .recoverInBackground(true) + .recoverStack(true) + .mainPage(MainActivity.class) + .recoverEnabled(true) + .callback(new MyCrashCallback()) + .silent(false, Recovery.SilentMode.RECOVER_ACTIVITY_STACK) +// .skip(TestActivity.class) + .init(this); + AppCrashHandler.register(); + } + + static final class MyCrashCallback implements RecoveryCallback { + @Override + public void stackTrace(String exceptionMessage) { + Log.e("wzt", "exceptionMessage:" + exceptionMessage); + } + + @Override + public void cause(String cause) { + Log.e("wzt", "cause:" + cause); + } + + @Override + public void exception(String exceptionType, String throwClassName, String throwMethodName, int throwLineNumber) { + Log.e("wzt", "exceptionClassName:" + exceptionType); + Log.e("wzt", "throwClassName:" + throwClassName); + Log.e("wzt", "throwMethodName:" + throwMethodName); + Log.e("wzt", "throwLineNumber:" + throwLineNumber); + } + + @Override + public void throwable(Throwable throwable) { + + } + } + +} diff --git a/demo_android_ncnn/app/src/main/java/com/rangi/nanodet/WelcomeActivity.java b/demo_android_ncnn/app/src/main/java/com/rangi/nanodet/WelcomeActivity.java new file mode 100644 index 000000000..cdcb019d6 --- /dev/null +++ b/demo_android_ncnn/app/src/main/java/com/rangi/nanodet/WelcomeActivity.java @@ -0,0 +1,90 @@ +package com.rangi.nanodet; + +import androidx.appcompat.app.AlertDialog; +import androidx.appcompat.app.AppCompatActivity; + +import android.content.Intent; +import android.os.Bundle; +import android.view.View; +import android.widget.Button; +import android.widget.CompoundButton; +import android.widget.ToggleButton; + + +public class WelcomeActivity extends AppCompatActivity { + + private ToggleButton tbUseGpu; + private Button yolov5s; + private Button yolov4tiny; + private Button mobilenetyolov3nano; + private Button nanodet; + + private boolean useGPU = false; + + @Override + protected void onCreate(Bundle savedInstanceState) { + super.onCreate(savedInstanceState); + setContentView(R.layout.activity_welcome); + + tbUseGpu = findViewById(R.id.tb_use_gpu); + tbUseGpu.setOnCheckedChangeListener(new CompoundButton.OnCheckedChangeListener() { + @Override + public void onCheckedChanged(CompoundButton buttonView, boolean isChecked) { + useGPU = isChecked; + MainActivity.USE_GPU = useGPU; + if (useGPU) { + AlertDialog.Builder builder = new AlertDialog.Builder(WelcomeActivity.this); + builder.setTitle("Warning"); + builder.setMessage("It may not work well in GPU mode, or errors may occur."); + builder.setCancelable(true); + builder.setPositiveButton("OK", null); + AlertDialog dialog = builder.create(); + dialog.show(); + } + } + }); + + yolov5s = findViewById(R.id.btn_start_detect1); + yolov5s.setOnClickListener(new View.OnClickListener() { + @Override + public void onClick(View v) { + MainActivity.USE_MODEL = MainActivity.YOLOV5S; + Intent intent = new Intent(WelcomeActivity.this, MainActivity.class); + WelcomeActivity.this.startActivity(intent); + } + }); + + yolov4tiny = findViewById(R.id.btn_start_detect2); + yolov4tiny.setOnClickListener(new View.OnClickListener() { + @Override + public void onClick(View v) { + MainActivity.USE_MODEL = MainActivity.YOLOV4_TINY; + Intent intent = new Intent(WelcomeActivity.this, MainActivity.class); + WelcomeActivity.this.startActivity(intent); + } + }); + + mobilenetyolov3nano = findViewById(R.id.btn_start_detect3); + mobilenetyolov3nano.setOnClickListener(new View.OnClickListener() { + @Override + public void onClick(View v) { + MainActivity.USE_MODEL = MainActivity.MOBILENETV2_YOLOV3_NANO; + Intent intent = new Intent(WelcomeActivity.this, MainActivity.class); + WelcomeActivity.this.startActivity(intent); + } + }); + + nanodet = findViewById(R.id.btn_start_detect10); + nanodet.setOnClickListener(new View.OnClickListener() { + @Override + public void onClick(View v) { + MainActivity.USE_MODEL = MainActivity.NANODET; + Intent intent = new Intent(WelcomeActivity.this, MainActivity.class); + WelcomeActivity.this.startActivity(intent); + } + }); + + } + + +} diff --git a/demo_android_ncnn/app/src/main/java/com/rangi/nanodet/YOLOv4.java b/demo_android_ncnn/app/src/main/java/com/rangi/nanodet/YOLOv4.java new file mode 100644 index 000000000..2b2ba8fbc --- /dev/null +++ b/demo_android_ncnn/app/src/main/java/com/rangi/nanodet/YOLOv4.java @@ -0,0 +1,13 @@ +package com.rangi.nanodet; + +import android.content.res.AssetManager; +import android.graphics.Bitmap; + +public class YOLOv4 { + static { + System.loadLibrary("yolov5"); // 存放在yolov5.so中 + } + + public static native void init(AssetManager manager, boolean v4tiny, boolean useGPU); + public static native Box[] detect(Bitmap bitmap, double threshold, double nms_threshold); +} diff --git a/demo_android_ncnn/app/src/main/java/com/rangi/nanodet/YOLOv5.java b/demo_android_ncnn/app/src/main/java/com/rangi/nanodet/YOLOv5.java new file mode 100644 index 000000000..b7b690378 --- /dev/null +++ b/demo_android_ncnn/app/src/main/java/com/rangi/nanodet/YOLOv5.java @@ -0,0 +1,13 @@ +package com.rangi.nanodet; + +import android.content.res.AssetManager; +import android.graphics.Bitmap; + +public class YOLOv5 { + static { + System.loadLibrary("yolov5"); + } + + public static native void init(AssetManager manager, boolean useGPU); + public static native Box[] detect(Bitmap bitmap, double threshold, double nms_threshold); +} diff --git a/demo_android_ncnn/app/src/main/res/drawable-v24/ic_launcher_foreground.xml b/demo_android_ncnn/app/src/main/res/drawable-v24/ic_launcher_foreground.xml new file mode 100644 index 000000000..1f6bb2906 --- /dev/null +++ b/demo_android_ncnn/app/src/main/res/drawable-v24/ic_launcher_foreground.xml @@ -0,0 +1,34 @@ + + + + + + + + + + + diff --git a/demo_android_ncnn/app/src/main/res/drawable-xxhdpi/cpu.png b/demo_android_ncnn/app/src/main/res/drawable-xxhdpi/cpu.png new file mode 100644 index 000000000..682c097dc Binary files /dev/null and b/demo_android_ncnn/app/src/main/res/drawable-xxhdpi/cpu.png differ diff --git a/demo_android_ncnn/app/src/main/res/drawable-xxhdpi/gpu.png b/demo_android_ncnn/app/src/main/res/drawable-xxhdpi/gpu.png new file mode 100644 index 000000000..dc8e249b0 Binary files /dev/null and b/demo_android_ncnn/app/src/main/res/drawable-xxhdpi/gpu.png differ diff --git a/demo_android_ncnn/app/src/main/res/drawable-xxhdpi/ncnn_icon.png b/demo_android_ncnn/app/src/main/res/drawable-xxhdpi/ncnn_icon.png new file mode 100644 index 000000000..40093faa2 Binary files /dev/null and b/demo_android_ncnn/app/src/main/res/drawable-xxhdpi/ncnn_icon.png differ diff --git a/demo_android_ncnn/app/src/main/res/drawable/cpu_gpu_bg.xml b/demo_android_ncnn/app/src/main/res/drawable/cpu_gpu_bg.xml new file mode 100644 index 000000000..197547a16 --- /dev/null +++ b/demo_android_ncnn/app/src/main/res/drawable/cpu_gpu_bg.xml @@ -0,0 +1,5 @@ + + + + + diff --git a/demo_android_ncnn/app/src/main/res/drawable/ic_launcher_background.xml b/demo_android_ncnn/app/src/main/res/drawable/ic_launcher_background.xml new file mode 100644 index 000000000..0d025f9bf --- /dev/null +++ b/demo_android_ncnn/app/src/main/res/drawable/ic_launcher_background.xml @@ -0,0 +1,170 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/demo_android_ncnn/app/src/main/res/layout/activity_main.xml b/demo_android_ncnn/app/src/main/res/layout/activity_main.xml new file mode 100644 index 000000000..28ab1f887 --- /dev/null +++ b/demo_android_ncnn/app/src/main/res/layout/activity_main.xml @@ -0,0 +1,110 @@ + + + +