diff --git a/.gitignore b/.gitignore index a7f7289ad0..3bca94ef00 100644 --- a/.gitignore +++ b/.gitignore @@ -54,3 +54,6 @@ gensim.egg-info .*.sw[op] data *.bak +/build/ +/dist/ + diff --git a/CHANGELOG.txt b/CHANGELOG.txt index 690755f3b6..047551c6d4 100644 --- a/CHANGELOG.txt +++ b/CHANGELOG.txt @@ -1,6 +1,24 @@ Changes ======= +0.8.7 + +* initial version of word2vec, a neural network deep learning algo +* make distributed gensim compatible with the new Pyro +* allow merging dictionaries (by Florent Chandelier) +* new design for the gensim website! +* speed up handling of corner cases when returning top-n most similar +* make Random Projections compatible with new scipy (andrewjOc360, PR #110) +* allow "light" (faster) word lemmatization (by Karsten Jeschkies) +* save/load directly from bzip2 files (by Luis Pedro Coelho, PR #101) +* Blei corpus now tries harder to find its vocabulary file (by Luis Pedro Coelho, PR #100) +* sparse vector elements can now be a list (was: only a 2-tuple) +* simple_preprocess now optionally de-accents letters (ř/š/ú etc.) +* better serialization of numpy corpora +* print_topics() returns the topics, in addition to printing/logging +* fixes for more robust Windows multiprocessing +* lots of small fixes, data checks and documentation updates + 0.8.6 * added HashDictionary (by Homer Strong) diff --git a/README.rst b/README.rst index 29fef211cb..18d3be1f27 100644 --- a/README.rst +++ b/README.rst @@ -3,7 +3,6 @@ gensim -- Python Framework for Topic Modelling ============================================== - Gensim is a Python library for *topic modelling*, *document indexing* and *similarity retrieval* with large corpora. Target audience is the *natural language processing* (NLP) and *information retrieval* (IR) community. @@ -33,6 +32,8 @@ Installation This software depends on `NumPy and Scipy `_, two Python packages for scientific computing. You must have them installed prior to installing `gensim`. +It is also recommended you install a fast BLAS library prior to installing NumPy. This is optional, but using an optimized BLAS such as `ATLAS `_ or `OpenBLAS `_ is known to improve performance by as much as an order of magnitude. + The simple way to install `gensim` is:: sudo easy_install gensim @@ -60,4 +61,4 @@ It is also included in the source distribution package. Gensim is open source software, and has been released under the `GNU LGPL license `_. -Copyright (c) 2009-2012 Radim Rehurek +Copyright (c) 2009-2013 Radim Rehurek diff --git a/docs/_sources/about.txt b/docs/_sources/about.txt deleted file mode 100644 index c6cb4a79b8..0000000000 --- a/docs/_sources/about.txt +++ /dev/null @@ -1,87 +0,0 @@ -.. _about: - -============ -About -============ - -History --------- - -Gensim started off as a collection of various Python scripts for the Czech Digital Mathematics Library `dml.cz `_ in 2008, -where it served to generate a short list of the most similar articles to a given article (gensim = "generate similar"). -I also wanted to try these fancy "Latent Semantic Methods", but the libraries that -realized the necessary computation were `not much fun to work with `_. - -Naturally, I set out to reinvent the wheel. Our `2010 LREC publication `_ -describes the initial design decisions behind gensim (clarity, efficiency and scalability) -and is fairly representative of how gensim works even today. - -Later versions of gensim improved this efficiency and scalability tremendously (in fact, -I made algorithmic scalability of distributional semantics the topic of my `PhD thesis `_). - -By now, gensim is---to my knowledge---the most robust, efficient and hassle-free piece -of software to realize unsupervised semantic modelling from plain text. It stands -in contrast to brittle homework-assignment-implementations that do not scale on one hand, -and robust java-esque projects that do scale, but only if you're willing to sacrifice -several weeks of your, your technician's as well as your local scientist's time just to run "hello world". - -In 2011, I started using `Github `_ for source code hosting, -and the gensim website moved from my university hosting to its present domain. - - -Licensing ----------- - -Gensim is licensed under the OSI-approved `GNU LGPL license `_. -This means that it's free for both personal and commercial use, but if you make any -modification to gensim that you distribute to other people, you have to disclose -the source code of these modifications. - -Apart form that, you are free to redistribute gensim in any way you like, though you're -not allowed to modify its license (doh!). - -My intent here is, of course, to get more help and community involvement with the development of gensim. -The legalese is therefore less important to me than your input and contributions. -Contact me if LGPL doesn't fit your bill but you'd still like to use it -- we'll work something out. - -.. seealso:: - - I also host a document similarity package `gensim.simserver`. This is a high-level - interface to `gensim` functionality, and offers transactional remote (web-based) - document similarity queries and indexing. It uses gensim to do the heavy lifting: - you don't need the `simserver` to use gensim, but you do need gensim to use the `simserver`. - Note that unlike gensim, `gensim.simserver` is licensed under `Affero GPL `_, - which is much more restrictive for inclusion in commercial projects. - -Contributors --------------- - -Credit goes to all the people who contributed to gensim, be it in `discussions `_, -ideas, `code contributions `_ or bug reports. -It's really useful and motivating to get feedback, in any shape or form, so big thanks to you all! - -Some honorable mentions are included in the `CHANGELOG.txt `_. - - -Academic citing ----------------- - -Gensim has been used in many students' final theses as well as research papers. When citing gensim, -please use `this BibTeX entry `_:: - - @inproceedings{rehurek_lrec, - title = {{Software Framework for Topic Modelling with Large Corpora}}, - author = {Radim {\v R}eh{\r u}{\v r}ek and Petr Sojka}, - booktitle = {{Proceedings of the LREC 2010 Workshop on New - Challenges for NLP Frameworks}}, - pages = {45--50}, - year = 2010, - month = May, - day = 22, - publisher = {ELRA}, - address = {Valletta, Malta}, - note={\url{http://is.muni.cz/publication/884893/en}}, - language={English} - } - - diff --git a/docs/_sources/apiref.txt b/docs/_sources/apiref.txt deleted file mode 100644 index 14ed4a8f7d..0000000000 --- a/docs/_sources/apiref.txt +++ /dev/null @@ -1,36 +0,0 @@ -.. _apiref: - -API Reference -============= - -Modules: - -.. toctree:: - :maxdepth: 0 - - interfaces - utils - matutils - corpora/bleicorpus - corpora/dictionary - corpora/hashdictionary - corpora/lowcorpus - corpora/mmcorpus - corpora/svmlightcorpus - corpora/wikicorpus - corpora/textcorpus - corpora/ucicorpus - corpora/indexedcorpus - models/ldamodel - models/lsimodel - models/tfidfmodel - models/rpmodel - models/hdpmodel - models/logentropy_model - models/lsi_dispatcher - models/lsi_worker - models/lda_dispatcher - models/lda_worker - similarities/docsim - similarities/simserver - diff --git a/docs/_sources/changes_080.txt b/docs/_sources/changes_080.txt deleted file mode 100644 index be5df9ad15..0000000000 --- a/docs/_sources/changes_080.txt +++ /dev/null @@ -1,80 +0,0 @@ -.. _changes_080: - -Change Set for 0.8.0 -============================ - -Release 0.8.0 concludes the 0.7.x series, which was about API consolidation and performance. -In 0.8.x, I'd like to extend `gensim` with new functionality and features. - -Codestyle Changes ------------------- - -Codebase was modified to comply with `PEP8: Style Guide for Python Code `_. -This means the 0.8.0 API is **backward incompatible** with the 0.7.x series. - -That's not as tragic as it sounds, gensim was almost there anyway. The changes are few and pretty straightforward: - -1. the `numTopics` parameter is now `num_topics` -2. `addDocuments()` method becomes `add_documents()` -3. `toUtf8()` => `to_utf8()` -4. ... you get the idea: replace `camelCase` with `lowercase_with_underscores`. - -If you stored a model that is affected by this to disk, you'll need to rename its attributes manually: - ->>> lsa = gensim.models.LsiModel.load('/some/path') # load old <0.8.0 model ->>> lsa.num_terms, lsa.num_topics = lsa.numTerms, lsa.numTopics # rename attributes ->>> del lsa.numTerms, lsa.numTopics # clean up old attributes (optional) ->>> lsa.save('/some/path') # save again to disk, as 0.8.0 compatible - -Only attributes (variables) need to be renamed; method names (functions) are not affected, due to the way `pickle` works. - -Similarity Queries -------------------- - -Improved speed and scalability of :doc:`similarity queries `. - -The `Similarity` class can now index corpora of arbitrary size more efficiently. -Internally, this is done by splitting the index into several smaller pieces ("shards") that fit in RAM -and can be processed independently. In addition, documents can now be added to a `Similarity` index dynamically. - -There is also a new way to query the similarity indexes: - ->>> index = MatrixSimilarity(corpus) # create an index ->>> sims = index[document] # get cosine similarity of query "document" against every document in the index ->>> sims = index[chunk_of_documents] # new syntax! - -Advantage of the last line (querying multiple documents at the same time) is faster execution. - -This faster execution is also utilized *automatically for you* if you're using the ``for sims in index: ...`` syntax -(which returns pairwise similarities of documents in the index). - -To see the speed-up on your machine, run ``python -m gensim.test.simspeed`` (and compare to my results `here `_ to see how your machine fares). - -.. note:: - This current functionality of querying is as far as I wanted to get with gensim. - More optimizations and smarter indexing are certainly possible, but I'd like to - focus on other features now. Pull requests are still welcome though :) - -Check out the :mod:`updated documentation ` of the similarity classes for more info. - -Simplified Directory Structure --------------------------------- - -Instead of the java-esque ``ROOT_DIR/src/gensim`` directory structure of gensim, -the packages now reside directly in ``ROOT_DIR/gensim`` (no superfluous ``src``). See the new structure `on github `_. - -Other changes (that you're unlikely to notice unless you look) ----------------------------------------------------------------------- - -* Improved efficiency of ``lsi[corpus]`` transformations (documents are chunked internally for better performance). -* Large matrices (numpy/scipy.sparse, in `LsiModel`, `Similarity` etc.) are now mmapped to/from disk when doing `save/load`. The `cPickle` approach used previously was too `buggy `_ and `slow `_. -* Renamed `chunks` parameter to `chunksize` (i.e. `LsiModel(corpus, num_topics=100, chunksize=20000)`). This better reflects its purpose: size of a chunk=number of documents to be processed at once. -* Also improved memory efficiency of LSI and LDA model generation (again). -* Removed SciPy 0.6 from the list of supported SciPi versions (need >=0.7 now). -* Added more unit tests. -* Several smaller fixes; see the `commit history `_ for full account. - -.. admonition:: Future Directions? - - If you have ideas or proposals for new features for 0.8.x, now is the time to let me know: - `gensim mailing list `_. diff --git a/docs/_sources/corpora/bleicorpus.txt b/docs/_sources/corpora/bleicorpus.txt deleted file mode 100644 index 91c02213b1..0000000000 --- a/docs/_sources/corpora/bleicorpus.txt +++ /dev/null @@ -1,8 +0,0 @@ -:mod:`corpora.bleicorpus` -- Corpus in Blei's LDA-C format -========================================================== - -.. automodule:: gensim.corpora.bleicorpus - :synopsis: Corpus in Blei's LDA-C format - :members: - :inherited-members: - diff --git a/docs/_sources/corpora/corpora.txt b/docs/_sources/corpora/corpora.txt deleted file mode 100644 index 3ea5151c96..0000000000 --- a/docs/_sources/corpora/corpora.txt +++ /dev/null @@ -1,8 +0,0 @@ -:mod:`corpora` -- Package for corpora I/O -========================================== - -.. automodule:: gensim.corpora - :synopsis: Package for corpora I/O - :members: - :inherited-members: - diff --git a/docs/_sources/corpora/dictionary.txt b/docs/_sources/corpora/dictionary.txt deleted file mode 100644 index 8542a61d6e..0000000000 --- a/docs/_sources/corpora/dictionary.txt +++ /dev/null @@ -1,8 +0,0 @@ -:mod:`corpora.dictionary` -- Construct word<->id mappings -========================================================== - -.. automodule:: gensim.corpora.dictionary - :synopsis: Construct word<->id mappings - :members: - :inherited-members: - diff --git a/docs/_sources/corpora/dmlcorpus.txt b/docs/_sources/corpora/dmlcorpus.txt deleted file mode 100644 index 3248522fa0..0000000000 --- a/docs/_sources/corpora/dmlcorpus.txt +++ /dev/null @@ -1,8 +0,0 @@ -:mod:`corpora.dmlcorpus` -- Corpus in DML-CZ format -==================================================== - -.. automodule:: gensim.corpora.dmlcorpus - :synopsis: Corpus in DML-CZ format - :members: - :inherited-members: - diff --git a/docs/_sources/corpora/indexedcorpus.txt b/docs/_sources/corpora/indexedcorpus.txt deleted file mode 100644 index 7f0a2764bf..0000000000 --- a/docs/_sources/corpora/indexedcorpus.txt +++ /dev/null @@ -1,12 +0,0 @@ -:mod:`corpora.indexedcorpus` -- Random access to corpus documents -================================================================= - -.. automodule:: gensim.corpora.indexedcorpus - :synopsis: Random access to corpus documents - :members: - :inherited-members: - - -.. autoclass:: IndexedCorpus - :members: - :inherited-members: \ No newline at end of file diff --git a/docs/_sources/corpora/lowcorpus.txt b/docs/_sources/corpora/lowcorpus.txt deleted file mode 100644 index 344495de31..0000000000 --- a/docs/_sources/corpora/lowcorpus.txt +++ /dev/null @@ -1,8 +0,0 @@ -:mod:`corpora.lowcorpus` -- Corpus in List-of-Words format -=========================================================== - -.. automodule:: gensim.corpora.lowcorpus - :synopsis: Corpus in List-of-Words format - :members: - :inherited-members: - diff --git a/docs/_sources/corpora/mmcorpus.txt b/docs/_sources/corpora/mmcorpus.txt deleted file mode 100644 index aec7b2d963..0000000000 --- a/docs/_sources/corpora/mmcorpus.txt +++ /dev/null @@ -1,8 +0,0 @@ -:mod:`corpora.mmcorpus` -- Corpus in Matrix Market format -========================================================== - -.. automodule:: gensim.corpora.mmcorpus - :synopsis: Corpus in Matrix Market format - :members: - :inherited-members: - diff --git a/docs/_sources/corpora/svmlightcorpus.txt b/docs/_sources/corpora/svmlightcorpus.txt deleted file mode 100644 index 9a2f8cb791..0000000000 --- a/docs/_sources/corpora/svmlightcorpus.txt +++ /dev/null @@ -1,8 +0,0 @@ -:mod:`corpora.svmlightcorpus` -- Corpus in SVMlight format -================================================================== - -.. automodule:: gensim.corpora.svmlightcorpus - :synopsis: Corpus in SVMlight format - :members: - :inherited-members: - diff --git a/docs/_sources/corpora/textcorpus.txt b/docs/_sources/corpora/textcorpus.txt deleted file mode 100644 index 01698db4de..0000000000 --- a/docs/_sources/corpora/textcorpus.txt +++ /dev/null @@ -1,8 +0,0 @@ -:mod:`corpora.textcorpus` -- Building corpora with dictionaries -================================================================= - -.. automodule:: gensim.corpora.textcorpus - :synopsis: Building corpora with dictionaries - :members: - :inherited-members: - diff --git a/docs/_sources/corpora/ucicorpus.txt b/docs/_sources/corpora/ucicorpus.txt deleted file mode 100644 index 3d48f81a1e..0000000000 --- a/docs/_sources/corpora/ucicorpus.txt +++ /dev/null @@ -1,8 +0,0 @@ -:mod:`corpora.ucicorpus` -- Corpus in UCI bag-of-words format -============================================================================================================== - -.. automodule:: gensim.corpora.ucicorpus - :synopsis: Corpus in University of California, Irvine (UCI) bag-of-words format - :members: - :inherited-members: - diff --git a/docs/_sources/corpora/wikicorpus.txt b/docs/_sources/corpora/wikicorpus.txt deleted file mode 100644 index 5dda3cb7fc..0000000000 --- a/docs/_sources/corpora/wikicorpus.txt +++ /dev/null @@ -1,8 +0,0 @@ -:mod:`corpora.wikicorpus` -- Corpus from a Wikipedia dump -========================================================== - -.. automodule:: gensim.corpora.wikicorpus - :synopsis: Corpus from a Wikipedia dump - :members: - :inherited-members: - diff --git a/docs/_sources/dist_lda.txt b/docs/_sources/dist_lda.txt deleted file mode 100644 index baf2d28aba..0000000000 --- a/docs/_sources/dist_lda.txt +++ /dev/null @@ -1,81 +0,0 @@ -.. _dist_lda: - -Distributed Latent Dirichlet Allocation -============================================ - - -.. note:: - See :doc:`distributed` for an introduction to distributed computing in `gensim`. - - -Setting up the cluster -_______________________ - -See the tutorial on :doc:`dist_lsi`; setting up a cluster for LDA is completely -analogous, except you want to run `lda_worker` and `lda_dispatcher` scripts instead -of `lsi_worker` and `lsi_dispatcher`. - -Running LDA -____________ - -Run LDA like you normally would, but turn on the `distributed=True` constructor -parameter:: - - >>> # extract 100 LDA topics, using default parameters - >>> lda = LdaModel(corpus=mm, id2word=id2word, num_topics=100, distributed=True) - using distributed version with 4 workers - running online LDA training, 100 topics, 1 passes over the supplied corpus of 3199665 documets, updating model once every 40000 documents - .. - - -In serial mode (no distribution), creating this online LDA :doc:`model of Wikipedia ` -takes 10h56m on my laptop (OS X, C2D 2.53GHz, 4GB RAM with `libVec`). -In distributed mode with four workers (Linux, Xeons of 2Ghz, 4GB RAM -with `ATLAS `_), the wallclock time taken drops to 3h20m. - -To run standard batch LDA (no online updates of mini-batches) instead, you would similarly -call:: - - >>> lda = LdaModel(corpus=mm, id2word=id2token, num_topics=100, update_every=0, passes=20, distributed=True) - using distributed version with 4 workers - running batch LDA training, 100 topics, 20 passes over the supplied corpus of 3199665 documets, updating model once every 3199665 documents - initializing workers - iteration 0, dispatching documents up to #10000/3199665 - iteration 0, dispatching documents up to #20000/3199665 - ... - -and then, some two days later:: - - iteration 19, dispatching documents up to #3190000/3199665 - iteration 19, dispatching documents up to #3199665/3199665 - reached the end of input; now waiting for all remaining jobs to finish - -:: - - >>> lda.print_topics(20) - topic #0: 0.007*disease + 0.006*medical + 0.005*treatment + 0.005*cells + 0.005*cell + 0.005*cancer + 0.005*health + 0.005*blood + 0.004*patients + 0.004*drug - topic #1: 0.024*king + 0.013*ii + 0.013*prince + 0.013*emperor + 0.008*duke + 0.008*empire + 0.007*son + 0.007*china + 0.007*dynasty + 0.007*iii - topic #2: 0.031*film + 0.017*films + 0.005*movie + 0.005*directed + 0.004*man + 0.004*episode + 0.003*character + 0.003*cast + 0.003*father + 0.003*mother - topic #3: 0.022*user + 0.012*edit + 0.009*wikipedia + 0.007*block + 0.007*my + 0.007*here + 0.007*edits + 0.007*blocked + 0.006*revert + 0.006*me - topic #4: 0.045*air + 0.026*aircraft + 0.021*force + 0.018*airport + 0.011*squadron + 0.010*flight + 0.010*military + 0.008*wing + 0.007*aviation + 0.007*f - topic #5: 0.025*sun + 0.022*star + 0.018*moon + 0.015*light + 0.013*stars + 0.012*planet + 0.011*camera + 0.010*mm + 0.009*earth + 0.008*lens - topic #6: 0.037*radio + 0.026*station + 0.022*fm + 0.014*news + 0.014*stations + 0.014*channel + 0.013*am + 0.013*racing + 0.011*tv + 0.010*broadcasting - topic #7: 0.122*image + 0.099*jpg + 0.046*file + 0.038*uploaded + 0.024*png + 0.014*contribs + 0.013*notify + 0.013*logs + 0.013*picture + 0.013*flag - topic #8: 0.036*russian + 0.030*soviet + 0.028*polish + 0.024*poland + 0.022*russia + 0.013*union + 0.012*czech + 0.011*republic + 0.011*moscow + 0.010*finland - topic #9: 0.031*language + 0.014*word + 0.013*languages + 0.009*term + 0.009*words + 0.008*example + 0.007*names + 0.007*meaning + 0.006*latin + 0.006*form - topic #10: 0.029*w + 0.029*toronto + 0.023*l + 0.020*hockey + 0.019*nhl + 0.014*ontario + 0.012*calgary + 0.011*edmonton + 0.011*hamilton + 0.010*season - topic #11: 0.110*wikipedia + 0.110*articles + 0.030*library + 0.029*wikiproject + 0.028*project + 0.019*data + 0.016*archives + 0.012*needing + 0.009*reference + 0.009*statements - topic #12: 0.032*http + 0.030*your + 0.022*request + 0.017*sources + 0.016*archived + 0.016*modify + 0.015*changes + 0.015*creation + 0.014*www + 0.013*try - topic #13: 0.011*your + 0.010*my + 0.009*we + 0.008*don + 0.008*get + 0.008*know + 0.007*me + 0.006*think + 0.006*question + 0.005*find - topic #14: 0.073*r + 0.066*japanese + 0.062*japan + 0.018*tokyo + 0.008*prefecture + 0.005*osaka + 0.004*j + 0.004*sf + 0.003*kyoto + 0.003*manga - topic #15: 0.045*da + 0.045*fr + 0.027*kategori + 0.026*pl + 0.024*nl + 0.021*pt + 0.017*en + 0.015*categoria + 0.014*es + 0.012*kategorie - topic #16: 0.010*death + 0.005*died + 0.005*father + 0.004*said + 0.004*himself + 0.004*took + 0.004*son + 0.004*killed + 0.003*murder + 0.003*wife - topic #17: 0.027*book + 0.021*published + 0.020*books + 0.014*isbn + 0.010*author + 0.010*magazine + 0.009*press + 0.009*novel + 0.009*writers + 0.008*story - topic #18: 0.027*football + 0.024*players + 0.023*cup + 0.019*club + 0.017*fc + 0.017*footballers + 0.017*league + 0.011*season + 0.007*teams + 0.007*goals - topic #19: 0.032*band + 0.024*album + 0.014*albums + 0.013*guitar + 0.013*rock + 0.011*records + 0.011*vocals + 0.009*live + 0.008*bass + 0.008*track - - - -If you used the distributed LDA implementation in `gensim`, please let me know (my -email is at the bottom of this page). I would like to hear about your application and -the possible (inevitable?) issues that you encountered, to improve `gensim` in the future. diff --git a/docs/_sources/dist_lsi.txt b/docs/_sources/dist_lsi.txt deleted file mode 100644 index 3cf8c5f065..0000000000 --- a/docs/_sources/dist_lsi.txt +++ /dev/null @@ -1,157 +0,0 @@ -.. _dist_lsi: - -Distributed Latent Semantic Analysis -============================================ - - -.. note:: - See :doc:`distributed` for an introduction to distributed computing in `gensim`. - - -Setting up the cluster -_______________________ - -We will show how to run distributed Latent Semantic Analysis by means of an example. -Let's say we have 5 computers at our disposal, all on the same network segment (=reachable -by network broadcast). To start with, install `gensim` and `Pyro` on each computer with:: - - $ sudo easy_install gensim[distributed] - -and run Pyro’s name server on exactly one of the machines (doesn’t matter which one):: - - $ python -m Pyro4.naming -n 0.0.0.0 & - -Let's say our example cluster consists of dual-core computers with loads of -memory. We will therefore run **two** worker scripts on four of the physical machines, -creating **eight** logical worker nodes:: - - $ python -m gensim.models.lsi_worker & - -This will execute `gensim`'s `lsi_worker.py` script (to be run twice on each of the -four computer). -This lets `gensim` know that it can run two jobs on each of the four computers in -parallel, so that the computation will be done faster, while also taking up twice -as much memory on each machine. - -Next, pick one computer that will be a job scheduler in charge of worker -synchronization, and on it, run `LSA dispatcher`. In our example, we will use the -fifth computer to act as the dispatcher and from there run:: - - $ python -m gensim.models.lsi_dispatcher & - -In general, the dispatcher can be run on the same machine as one of the worker nodes, or it -can be another, distinct computer (within the same broadcast domain). The dispatcher -won't be doing much with CPU most of the time, but pick a computer with ample memory. - -And that's it! The cluster is set up and running, ready to accept jobs. To remove -a worker later on, simply terminate its `lsi_worker` process. To add another worker, run another -`lsi_worker` (this will not affect a computation that is already running, the additions/deletions are not dynamic). -If you terminate `lsi_dispatcher`, you won't be able to run computations until you run it again -(surviving worker processes can be re-used though). - - -Running LSA -____________ - -So let's test our setup and run one computation of distributed LSA. Open a Python -shell on one of the five machines (again, this can be done on any computer -in the same `broadcast domain `_, -our choice is incidental) and try:: - - >>> from gensim import corpora, models, utils - >>> import logging - >>> logging.basicConfig(format='%(asctime)s : %(levelname)s : %(message)s', level=logging.INFO) - - >>> corpus = corpora.MmCorpus('/tmp/deerwester.mm') # load a corpus of nine documents, from the Tutorials - >>> id2word = corpora.Dictionary.load('/tmp/deerwester.dict') - - >>> lsi = models.LsiModel(corpus, id2word=id2word, num_topics=200, chunksize=1, distributed=True) # run distributed LSA on nine documents - -This uses the corpus and feature-token mapping created in the :doc:`tut1` tutorial. -If you look at the log in your Python session, you should see a line similar to:: - - 2010-08-09 23:44:25,746 : INFO : using distributed version with 8 workers - -which means all went well. You can also check the logs coming from your worker and dispatcher -processes --- this is especially helpful in case of problems. -To check the LSA results, let's print the first two latent topics:: - - >>> lsi.print_topics(num_topics=2, num_words=5) - topic #0(3.341): 0.644*"system" + 0.404*"user" + 0.301*"eps" + 0.265*"time" + 0.265*"response" - topic #1(2.542): 0.623*"graph" + 0.490*"trees" + 0.451*"minors" + 0.274*"survey" + -0.167*"system" - -Success! But a corpus of nine documents is no challenge for our powerful cluster... -In fact, we had to lower the job size (`chunksize` parameter above) to a single document -at a time, otherwise all documents would be processed by a single worker all at once. - -So let's run LSA on **one million documents** instead:: - - >>> # inflate the corpus to 1M documents, by repeating its documents over&over - >>> corpus1m = utils.RepeatCorpus(corpus, 1000000) - >>> # run distributed LSA on 1 million documents - >>> lsi1m = models.LsiModel(corpus1m, id2word=id2word, num_topics=200, chunksize=10000, distributed=True) - - >>> lsi1m.print_topics(num_topics=2, num_words=5) - topic #0(1113.628): 0.644*"system" + 0.404*"user" + 0.301*"eps" + 0.265*"time" + 0.265*"response" - topic #1(847.233): 0.623*"graph" + 0.490*"trees" + 0.451*"minors" + 0.274*"survey" + -0.167*"system" - -The log from 1M LSA should look like:: - - 2010-08-10 02:46:35,087 : INFO : using distributed version with 8 workers - 2010-08-10 02:46:35,087 : INFO : updating SVD with new documents - 2010-08-10 02:46:35,202 : INFO : dispatched documents up to #10000 - 2010-08-10 02:46:35,296 : INFO : dispatched documents up to #20000 - ... - 2010-08-10 02:46:46,524 : INFO : dispatched documents up to #990000 - 2010-08-10 02:46:46,694 : INFO : dispatched documents up to #1000000 - 2010-08-10 02:46:46,694 : INFO : reached the end of input; now waiting for all remaining jobs to finish - 2010-08-10 02:46:47,195 : INFO : all jobs finished, downloading final projection - 2010-08-10 02:46:47,200 : INFO : decomposition complete - -Due to the small vocabulary size and trivial structure of our "one-million corpus", the computation -of LSA still takes only 12 seconds. To really stress-test our cluster, let's do -Latent Semantic Analysis on the English Wikipedia. - -Distributed LSA on Wikipedia -++++++++++++++++++++++++++++++ - -First, download and prepare the Wikipedia corpus as per :doc:`wiki`, then load -the corpus iterator with:: - - >>> import logging, gensim, bz2 - >>> logging.basicConfig(format='%(asctime)s : %(levelname)s : %(message)s', level=logging.INFO) - - >>> # load id->word mapping (the dictionary) - >>> id2word = gensim.corpora.Dictionary.load_from_text('wiki_en_wordids.txt') - >>> # load corpus iterator - >>> mm = gensim.corpora.MmCorpus('wiki_en_tfidf.mm') - >>> # mm = gensim.corpora.MmCorpus(bz2.BZ2File('wiki_en_tfidf.mm.bz2')) # use this if you compressed the TFIDF output - - >>> print mm - MmCorpus(3199665 documents, 100000 features, 495547400 non-zero entries) - -Now we're ready to run distributed LSA on the English Wikipedia:: - - >>> # extract 400 LSI topics, using a cluster of nodes - >>> lsi = gensim.models.lsimodel.LsiModel(corpus=mm, id2word=id2word, num_topics=400, chunksize=20000, distributed=True) - - >>> # print the most contributing words (both positively and negatively) for each of the first ten topics - >>> lsi.print_topics(10) - 2010-11-03 16:08:27,602 : INFO : topic #0(200.990): -0.475*"delete" + -0.383*"deletion" + -0.275*"debate" + -0.223*"comments" + -0.220*"edits" + -0.213*"modify" + -0.208*"appropriate" + -0.194*"subsequent" + -0.155*"wp" + -0.117*"notability" - 2010-11-03 16:08:27,626 : INFO : topic #1(143.129): -0.320*"diff" + -0.305*"link" + -0.199*"image" + -0.171*"www" + -0.162*"user" + 0.149*"delete" + -0.147*"undo" + -0.144*"contribs" + -0.122*"album" + 0.113*"deletion" - 2010-11-03 16:08:27,651 : INFO : topic #2(135.665): -0.437*"diff" + -0.400*"link" + -0.202*"undo" + -0.192*"user" + -0.182*"www" + -0.176*"contribs" + 0.168*"image" + -0.109*"added" + 0.106*"album" + 0.097*"copyright" - 2010-11-03 16:08:27,677 : INFO : topic #3(125.027): -0.354*"image" + 0.239*"age" + 0.218*"median" + -0.213*"copyright" + 0.204*"population" + -0.195*"fair" + 0.195*"income" + 0.167*"census" + 0.165*"km" + 0.162*"households" - 2010-11-03 16:08:27,701 : INFO : topic #4(116.927): -0.307*"image" + 0.195*"players" + -0.184*"median" + -0.184*"copyright" + -0.181*"age" + -0.167*"fair" + -0.162*"income" + -0.151*"population" + -0.136*"households" + -0.134*"census" - 2010-11-03 16:08:27,728 : INFO : topic #5(100.326): 0.501*"players" + 0.318*"football" + 0.284*"league" + 0.193*"footballers" + 0.142*"image" + 0.133*"season" + 0.119*"cup" + 0.113*"club" + 0.110*"baseball" + 0.103*"f" - 2010-11-03 16:08:27,754 : INFO : topic #6(92.298): -0.411*"album" + -0.275*"albums" + -0.217*"band" + -0.214*"song" + -0.184*"chart" + -0.163*"songs" + -0.160*"singles" + -0.149*"vocals" + -0.139*"guitar" + -0.129*"track" - 2010-11-03 16:08:27,780 : INFO : topic #7(83.811): -0.248*"wikipedia" + -0.182*"keep" + 0.180*"delete" + -0.167*"articles" + -0.152*"your" + -0.150*"my" + 0.144*"film" + -0.130*"we" + -0.123*"think" + -0.120*"user" - 2010-11-03 16:08:27,807 : INFO : topic #8(78.981): 0.588*"film" + 0.460*"films" + -0.130*"album" + -0.127*"station" + 0.121*"television" + 0.115*"poster" + 0.112*"directed" + 0.110*"actors" + -0.096*"railway" + 0.086*"movie" - 2010-11-03 16:08:27,834 : INFO : topic #9(78.620): 0.502*"kategori" + 0.282*"categoria" + 0.248*"kategorija" + 0.234*"kategorie" + 0.172*"категория" + 0.165*"categoría" + 0.161*"kategoria" + 0.148*"categorie" + 0.126*"kategória" + 0.121*"catégorie" - -In serial mode, creating the LSI model of Wikipedia with this **one-pass algorithm** -takes about 5.25h on my laptop (OS X, C2D 2.53GHz, 4GB RAM with `libVec`). -In distributed mode with four workers (Linux, dual-core Xeons of 2Ghz, 4GB RAM -with `ATLAS`), the wallclock time taken drops to 1 hour and 41 minutes. You can -read more about various internal settings and experiments in my `research -paper `_. - diff --git a/docs/_sources/distributed.txt b/docs/_sources/distributed.txt deleted file mode 100644 index 5723937851..0000000000 --- a/docs/_sources/distributed.txt +++ /dev/null @@ -1,92 +0,0 @@ -.. _distributed: - -Distributed Computing -=================================== - -Why distributed computing? ---------------------------- - -Need to build semantic representation of a corpus that is millions of documents large and it's -taking forever? Have several idle machines at your disposal that you could use? -`Distributed computing `_ tries -to accelerate computations by splitting a given task into several smaller subtasks, -passing them on to several computing nodes in parallel. - -In the context of `gensim`, computing nodes are computers identified by their IP address/port, -and communication happens over TCP/IP. The whole collection of available machines is called -a *cluster*. The distribution is very coarse grained (not -much communication going on), so the network is allowed to be of relatively high latency. - -.. warning:: - The primary reason for using distributed computing is making things run faster. In `gensim`, - most of the time consuming stuff is done inside low-level routines for linear algebra, inside - NumPy, independent of any `gensim` code. - **Installing a fast** `BLAS (Basic Linear Algebra) `_ **library - for NumPy can improve performance up to 15 times!** So before you start buying those extra computers, - consider installing a fast, threaded BLAS that is optimized for your particular machine - (as opposed to a generic, binary-distributed library). - Options include your vendor's BLAS library (Intel's MKL, - AMD's ACML, OS X's vecLib, Sun's Sunperf, ...) or some open-source alternative (GotoBLAS, ALTAS). - - To see what BLAS and LAPACK you are using, type into your shell:: - - python -c 'import numpy; numpy.show_config()' - -Prerequisites ------------------ - -For communication between nodes, `gensim` uses `Pyro (PYthon Remote Objects) -`_, version >= 4.8. This is a library for low-level socket communication -and remote procedure calls (RPC) in Python. `Pyro` is a pure-Python library, so its -installation is quite painless and only involves copying its `*.py` files somewhere onto your Python's import path:: - - sudo easy_install Pyro4 - -You don't have to install `Pyro` to run `gensim`, but if you don't, you won't be able -to access the distributed features (i.e., everything will always run in serial mode, -the examples on this page don't apply). - - -Core concepts ------------------------------------ - -As always, `gensim` strives for a clear and straightforward API (see :ref:`design`). -To this end, *you do not need to make any changes in your code at all* in order to -run it over a cluster of computers! - -What you need to do is run a :term:`worker` script (see below) on each of your cluster nodes prior -to starting your computation. Running this script tells `gensim` that it may use the node -as a slave to delegate some work to it. During initialization, the algorithms -inside `gensim` will try to look for and enslave all available worker nodes. - -.. glossary:: - - Node - A logical working unit. Can correspond to a single physical machine, but you - can also run multiple workers on one machine, resulting in multiple - logical nodes. - - Cluster - Several nodes which communicate over TCP/IP. Currently, network broadcasting - is used to discover and connect all communicating nodes, so the nodes must lie - within the same `broadcast domain `_. - - Worker - A process which is created on each node. To remove a node from your cluster, - simply kill its worker process. - - Dispatcher - The dispatcher will be in charge of negotiating all computations, queueing and - distributing ("dispatching") individual jobs to the workers. Computations never - "talk" to worker nodes directly, only through this dispatcher. Unlike workers, - there can only be one active dispatcher at a time in the cluster. - - -Available distributed algorithms ---------------------------------- - -.. toctree:: - :maxdepth: 1 - - dist_lsi - dist_lda diff --git a/docs/_sources/index.txt b/docs/_sources/index.txt deleted file mode 100644 index f2f7e073de..0000000000 --- a/docs/_sources/index.txt +++ /dev/null @@ -1,63 +0,0 @@ -.. gensim documentation master file, created by - sphinx-quickstart on Tue Mar 16 19:45:41 2010. - You can adapt this file completely to your liking, but it should at least - contain the root `toctree` directive. - -Gensim -- Topic Modelling for Humans -===================================================== - - -.. raw:: html - :file: _static/tagcloud.html - - -.. raw:: html - - - - - - - - - - -
What is this?Installation
Get started!About
- - -Quick Reference Example ------------------------- - ->>> from gensim import corpora, models, similarities ->>> ->>> # Load corpus iterator from a Matrix Market file on disk. ->>> corpus = corpora.MmCorpus('/path/to/corpus.mm') ->>> ->>> # Initialize a transformation (Latent Semantic Indexing with 200 latent dimensions). ->>> lsi = models.LsiModel(corpus, num_topics=200) ->>> ->>> # Convert another corpus to the latent space and index it. ->>> index = similarities.MatrixSimilarity(lsi[another_corpus]) ->>> ->>> # determine similarity of a query document against each document in the index ->>> sims = index[query] - - -.. admonition:: What's new? - - * 15 Sep 2012: release 0.8.6 : added the `hashing trick `_ to allow online changes to the vocabulary; fixed parallel lemmatization + `other minor improvements `_ - * 22 Jul 2012: release 0.8.5 : better Wikipedia parsing, faster similarity queries, maintenance fixes - * 30 Apr 2012: William Bert's `interview with me `_ - * 9 Mar 2012: release 0.8.4: new model `Hierarchical Dirichlet Process `_ (full `CHANGELOG `_) - - -.. toctree:: - :hidden: - :maxdepth: 1 - - intro - install - tutorial - distributed - wiki - apiref diff --git a/docs/_sources/install.txt b/docs/_sources/install.txt deleted file mode 100644 index e4886b20b8..0000000000 --- a/docs/_sources/install.txt +++ /dev/null @@ -1,118 +0,0 @@ -.. _install: - -============= -Installation -============= - -Quick install --------------- - -Run in your terminal:: - - sudo easy_install -U gensim - -In case that fails, or you don't know what "terminal" means, read on. - ------ - -Dependencies -------------- -Gensim is known to run on Linux, Windows and Mac OS X and should run on any other -platform that supports Python 2.5 and NumPy. Gensim depends on the following software: - -* 3.0 > `Python `_ >= 2.5. Tested with versions 2.5, 2.6 and 2.7. -* `NumPy `_ >= 1.3. Tested with version 1.6.1rc2, 1.5.0rc1, 1.4.0, 1.3.0, 1.3.0rc2. -* `SciPy `_ >= 0.7. Tested with version 0.9.0, 0.8.0, 0.8.0b1, 0.7.1, 0.7.0. - -**Windows users** are well advised to try the `Enthought distribution `_, -which conveniently includes Python&NumPy&SciPy in a single bundle, and is free for academic use. - - -Install Python and `easy_install` ---------------------------------- - -Check what version of Python you have with:: - - python --version - -You can download Python from http://python.org/download. - -.. note:: Gensim requires Python 2.5 or greater and will not run under earlier versions. - -Next, install the `easy_install utility `_, -which will make installing other Python programs easier. - -Install SciPy & NumPy ----------------------- - -These are quite popular Python packages, so chances are there are pre-built binary -distributions available for your platform. You can try installing from source using easy_install:: - - sudo easy_install numpy - sudo easy_install scipy - -If that doesn't work or if you'd rather install using a binary package, consult -http://www.scipy.org/Download. - -Install `gensim` ------------------ - -You can now install (or upgrade) `gensim` with:: - - sudo easy_install --upgrade gensim - -That's it! Congratulations, you can proceed to the :doc:`tutorials `. - ------ - -If you also want to run the algorithms over a cluster -of computers, in :doc:`distributed`, you should install with:: - - sudo easy_install gensim[distributed] - -The optional `distributed` feature installs `Pyro (PYthon Remote Objects) `_. -If you don't know what distributed computing means, you can ignore it: -`gensim` will work fine for you anyway. -This optional extension can also be installed separately later with:: - - sudo easy_install Pyro4 - ------ - -There are also alternative routes to install: - -1. If you have downloaded and unzipped the `tar.gz source `_ - for `gensim` (or you're installing `gensim` from `github `_), - you can run:: - - sudo python setup.py install - - to install `gensim` into your ``site-packages`` folder. -2. If you wish to make local changes to the `gensim` code (`gensim` is, after all, a - package which targets research prototyping and modifications), a preferred - way may be installing with:: - - sudo python setup.py develop - - This will only place a symlink into your ``site-packages`` directory. The actual - files will stay wherever you unpacked them. -3. If you don't have root priviledges (or just don't want to put the package into - your ``site-packages``), simply unpack the source package somewhere and that's it! No - compilation or installation needed. Just don't forget to set your PYTHONPATH - (or modify ``sys.path``), so that Python can find the unpacked package when importing. - - -Testing `gensim` ----------------- - -To test the package, unzip the `tar.gz source `_ and run:: - - python setup.py test - - -Contact --------- - -Use the `gensim discussion group `_ for -any questions and troubleshooting. For private enquiries, you can also send -me an email to the address at the bottom of this page. diff --git a/docs/_sources/interfaces.txt b/docs/_sources/interfaces.txt deleted file mode 100644 index f0f932c50d..0000000000 --- a/docs/_sources/interfaces.txt +++ /dev/null @@ -1,8 +0,0 @@ -:mod:`interfaces` -- Core gensim interfaces -============================================ - -.. automodule:: gensim.interfaces - :synopsis: Core gensim interfaces - :members: - :inherited-members: - diff --git a/docs/_sources/intro.txt b/docs/_sources/intro.txt deleted file mode 100644 index 2166f21b14..0000000000 --- a/docs/_sources/intro.txt +++ /dev/null @@ -1,139 +0,0 @@ -.. _intro: - -============ -Introduction -============ - -Gensim is a :ref:`free ` Python framework designed to automatically extract semantic -topics from documents, as efficiently (computer-wise) and painlessly (human-wise) as possible. - - -Gensim aims at processing raw, unstructured digital texts ("*plain text*"). -The algorithms in `gensim`, such as **Latent Semantic Analysis**, **Latent Dirichlet Allocation** or **Random Projections**, -discover semantic structure of documents, by examining word statistical co-occurrence patterns within a corpus of training documents. -These algorithms are unsupervised, which means no human input is necessary -- you only need a corpus of plain text documents. - -Once these statistical patterns are found, any plain text documents can be succinctly -expressed in the new, semantic representation, and queried for topical similarity -against other documents. - -.. note:: - If the previous paragraphs left you confused, you can read more about the `Vector - Space Model `_ and `unsupervised - document analysis `_ on Wikipedia. - - -.. _design: - -Features ------------------- - -* **Memory independence** -- there is no need for the whole training corpus to - reside fully in RAM at any one time (can process large, web-scale corpora). -* Efficient implementations for several popular vector space algorithms, - including **Tf-Idf**, distributed incremental **Latent Semantic Analysis**, - distributed incremental **Latent Dirichlet Allocation (LDA)** or **Random Projection**; adding new ones is easy (really!). -* I/O wrappers and converters around **several popular data formats**. -* **Similarity queries** for documents in their semantic representation. - -Creation of `gensim` was motivated by a perceived lack of available, scalable software -frameworks that realize topic modelling, and/or their overwhelming internal complexity (hail java!). -You can read more about the motivation in our `LREC 2010 workshop paper `_. -If you want to cite `gensim` in your own work, please refer to that article (`BibTeX `_). - -You're welcome to share your results on the `mailing list `_, -so others can learn from your success :) - -The **principal design objectives** behind `gensim` are: - -1. Straightforward interfaces and low API learning curve for developers. Good for prototyping. -2. Memory independence with respect to the size of the input corpus; all intermediate - steps and algorithms operate in a streaming fashion, accessing one document - at a time. - -.. seealso:: - - If you're interested in document indexing/similarity retrieval, I also maintain a higher-level package - of `document similarity server `_. It uses gensim internally. - -.. _availability: - -Availability ------------- - -Gensim is licensed under the OSI-approved `GNU LPGL license `_ -and can be downloaded either from its `github repository `_ -or from the `Python Package Index `_. - -.. seealso:: - - See the :doc:`install ` page for more info on `gensim` deployment. - - -Core concepts -------------- - -The whole gensim package revolves around the concepts of :term:`corpus`, :term:`vector` and -:term:`model`. - -.. glossary:: - - Corpus - A collection of digital documents. This collection is used to automatically - infer structure of the documents, their topics etc. For - this reason, the collection is also called a *training corpus*. The inferred - latent structure can be later used to assign topics to new documents, which did - not appear in the training corpus. - No human intervention (such as tagging the documents by hand, or creating - other metadata) is required. - - Vector - In the Vector Space Model (VSM), each document is represented by an - array of features. For example, a single feature may be thought of as a - question-answer pair: - - 1. How many times does the word *splonge* appear in the document? Zero. - 2. How many paragraphs does the document consist of? Two. - 3. How many fonts does the document use? Five. - - The question is usually represented only by its integer id (such as `1`, `2` and `3` here), - so that the - representation of this document becomes a series of pairs like ``(1, 0.0), (2, 2.0), (3, 5.0)``. - If we know all the questions in advance, we may leave them implicit - and simply write ``(0.0, 2.0, 5.0)``. - This sequence of answers can be thought of as a high-dimensional (in this case 3-dimensional) - *vector*. For practical purposes, only questions to which the answer is (or - can be converted to) a single real number are allowed. - - The questions are the same for each document, so that looking at two - vectors (representing two documents), we will hopefully be able to make - conclusions such as "The numbers in these two vectors are very similar, and - therefore the original documents must be similar, too". Of course, whether - such conclusions correspond to reality depends on how well we picked our questions. - - Sparse vector - Typically, the answer to most questions will be ``0.0``. To save space, - we omit them from the document's representation, and write only ``(2, 2.0), - (3, 5.0)`` (note the missing ``(1, 0.0)``). - Since the set of all questions is known in advance, all the missing features - in a sparse representation of a document can be unambiguously resolved to zero, ``0.0``. - - Gensim is specific in that it doesn't prescribe any specific corpus format; - a corpus is anything that, when iterated over, successively yields these sparse vectors. - For example, `set([(2, 2.0), (3, 5.0)], ([0, -1.0], [3, -1.0]))` is a trivial - corpus of two documents, each with two non-zero `feature-answer` pairs. - - - - Model - For our purposes, a model is a transformation from one document representation - to another (or, in other words, from one vector space to another). - Both the initial and target representations are - still vectors -- they only differ in what the questions and answers are. - The transformation is automatically learned from the traning :term:`corpus`, without human - supervision, and in hopes that the final document representation will be more compact - and more useful: with similar documents having similar representations. - -.. seealso:: - - For some examples on how this works out in code, go to :doc:`tutorials `. diff --git a/docs/_sources/matutils.txt b/docs/_sources/matutils.txt deleted file mode 100644 index 45de7a14fa..0000000000 --- a/docs/_sources/matutils.txt +++ /dev/null @@ -1,8 +0,0 @@ -:mod:`matutils` -- Math utils -============================== - -.. automodule:: gensim.matutils - :synopsis: Math utils - :members: - :inherited-members: - diff --git a/docs/_sources/models/hdpmodel.txt b/docs/_sources/models/hdpmodel.txt deleted file mode 100644 index 1975b28fc4..0000000000 --- a/docs/_sources/models/hdpmodel.txt +++ /dev/null @@ -1,8 +0,0 @@ -:mod:`models.hdpmodel` -- Hierarchical Dirichlet Process -======================================================== - -.. automodule:: gensim.models.hdpmodel - :synopsis: Hierarchical Dirichlet Process - :members: - :inherited-members: - diff --git a/docs/_sources/models/lda_dispatcher.txt b/docs/_sources/models/lda_dispatcher.txt deleted file mode 100644 index f6dffd28e1..0000000000 --- a/docs/_sources/models/lda_dispatcher.txt +++ /dev/null @@ -1,8 +0,0 @@ -:mod:`models.lda_dispatcher` -- Dispatcher for distributed LDA -================================================================ - -.. automodule:: gensim.models.lda_dispatcher - :synopsis: Dispatcher for distributed LDA - :members: - :inherited-members: - diff --git a/docs/_sources/models/lda_worker.txt b/docs/_sources/models/lda_worker.txt deleted file mode 100644 index 381556eaa6..0000000000 --- a/docs/_sources/models/lda_worker.txt +++ /dev/null @@ -1,8 +0,0 @@ -:mod:`models.lda_worker` -- Worker for distributed LDA -====================================================== - -.. automodule:: gensim.models.lda_worker - :synopsis: Worker for distributed LDA - :members: - :inherited-members: - diff --git a/docs/_sources/models/ldamodel.txt b/docs/_sources/models/ldamodel.txt deleted file mode 100644 index 2d6ae3adf0..0000000000 --- a/docs/_sources/models/ldamodel.txt +++ /dev/null @@ -1,8 +0,0 @@ -:mod:`models.ldamodel` -- Latent Dirichlet Allocation -====================================================== - -.. automodule:: gensim.models.ldamodel - :synopsis: Latent Dirichlet Allocation - :members: - :inherited-members: - diff --git a/docs/_sources/models/logentropy_model.txt b/docs/_sources/models/logentropy_model.txt deleted file mode 100644 index 1466f6fd1b..0000000000 --- a/docs/_sources/models/logentropy_model.txt +++ /dev/null @@ -1,8 +0,0 @@ -:mod:`models.logentropy_model` -- LogEntropy model -====================================================== - -.. automodule:: gensim.models.logentropy_model - :synopsis: LogEntropy model - :members: - :inherited-members: - diff --git a/docs/_sources/models/lsi_dispatcher.txt b/docs/_sources/models/lsi_dispatcher.txt deleted file mode 100644 index 59bc80e35c..0000000000 --- a/docs/_sources/models/lsi_dispatcher.txt +++ /dev/null @@ -1,8 +0,0 @@ -:mod:`models.lsi_dispatcher` -- Dispatcher for distributed LSI -=============================================================== - -.. automodule:: gensim.models.lsi_dispatcher - :synopsis: Dispatcher for distributed LSI - :members: - :inherited-members: - diff --git a/docs/_sources/models/lsi_worker.txt b/docs/_sources/models/lsi_worker.txt deleted file mode 100644 index baf999f105..0000000000 --- a/docs/_sources/models/lsi_worker.txt +++ /dev/null @@ -1,8 +0,0 @@ -:mod:`models.lsi_worker` -- Worker for distributed LSI -====================================================== - -.. automodule:: gensim.models.lsi_worker - :synopsis: Worker for distributed LSI - :members: - :inherited-members: - diff --git a/docs/_sources/models/lsimodel.txt b/docs/_sources/models/lsimodel.txt deleted file mode 100644 index c0340db439..0000000000 --- a/docs/_sources/models/lsimodel.txt +++ /dev/null @@ -1,8 +0,0 @@ -:mod:`models.lsimodel` -- Latent Semantic Indexing -====================================================== - -.. automodule:: gensim.models.lsimodel - :synopsis: Latent Semantic Indexing - :members: - :inherited-members: - diff --git a/docs/_sources/models/models.txt b/docs/_sources/models/models.txt deleted file mode 100644 index f18032b7ee..0000000000 --- a/docs/_sources/models/models.txt +++ /dev/null @@ -1,8 +0,0 @@ -:mod:`models` -- Package for transformation models -====================================================== - -.. automodule:: gensim.models - :synopsis: Package for transformation models - :members: - :inherited-members: - diff --git a/docs/_sources/models/rpmodel.txt b/docs/_sources/models/rpmodel.txt deleted file mode 100644 index 47eba01262..0000000000 --- a/docs/_sources/models/rpmodel.txt +++ /dev/null @@ -1,8 +0,0 @@ -:mod:`models.rpmodel` -- Random Projections -====================================================== - -.. automodule:: gensim.models.rpmodel - :synopsis: Random Projections - :members: - :inherited-members: - diff --git a/docs/_sources/models/tfidfmodel.txt b/docs/_sources/models/tfidfmodel.txt deleted file mode 100644 index ea714e7268..0000000000 --- a/docs/_sources/models/tfidfmodel.txt +++ /dev/null @@ -1,8 +0,0 @@ -:mod:`models.tfidfmodel` -- TF-IDF model -====================================================== - -.. automodule:: gensim.models.tfidfmodel - :synopsis: TF-IDF model - :members: - :inherited-members: - diff --git a/docs/_sources/similarities/docsim.txt b/docs/_sources/similarities/docsim.txt deleted file mode 100644 index 9d330f23b9..0000000000 --- a/docs/_sources/similarities/docsim.txt +++ /dev/null @@ -1,8 +0,0 @@ -:mod:`similarities.docsim` -- Document similarity queries -======================================================================== - -.. automodule:: gensim.similarities.docsim - :synopsis: Document similarity queries - :members: - :inherited-members: - diff --git a/docs/_sources/similarities/simserver.txt b/docs/_sources/similarities/simserver.txt deleted file mode 100644 index 82b3101fc5..0000000000 --- a/docs/_sources/similarities/simserver.txt +++ /dev/null @@ -1,8 +0,0 @@ -:mod:`simserver` -- Document similarity server -====================================================== - -.. automodule:: simserver.simserver - :synopsis: Document similarity server - :members: - :inherited-members: - diff --git a/docs/_sources/simserver.txt b/docs/_sources/simserver.txt deleted file mode 100644 index de0785da57..0000000000 --- a/docs/_sources/simserver.txt +++ /dev/null @@ -1,330 +0,0 @@ -.. _simserver: - -Document Similarity Server -============================= - -The 0.7.x series of `gensim `_ was about improving performance and consolidating API. -0.8.x will be about new features --- 0.8.1, first of the series, is a **document similarity service**. - -The source code itself has been moved from gensim to its own, dedicated package, named `simserver`. -Get it from `PyPI `_ or clone it on `Github `_. - -What is a document similarity service? ---------------------------------------- - -Conceptually, a service that lets you : - -1. train a semantic model from a corpus of plain texts (no manual annotation and mark-up needed) -2. index arbitrary documents using this semantic model -3. query the index for similar documents (the query can be either an id of a document already in the index, or an arbitrary text) - - ->>> from simserver import SessionServer ->>> server = SessionServer('/tmp/my_server') # resume server (or create a new one) - ->>> server.train(training_corpus, method='lsi') # create a semantic model ->>> server.index(some_documents) # convert plain text to semantic representation and index it ->>> server.find_similar(query) # convert query to semantic representation and compare against index ->>> ... ->>> server.index(more_documents) # add to index: incremental indexing works ->>> server.find_similar(query) ->>> ... ->>> server.delete(ids_to_delete) # incremental deleting also works ->>> server.find_similar(query) ->>> ... - -.. note:: - "Semantic" here refers to semantics of the crude, statistical type -- - `Latent Semantic Analysis `_, - `Latent Dirichlet Allocation `_ etc. - Nothing to do with the semantic web, manual resource tagging or detailed linguistic inference. - - -What is it good for? ---------------------- - -Digital libraries of (mostly) text documents. More generally, it helps you annotate, -organize and navigate documents in a more abstract way, compared to plain keyword search. - -How is it unique? ------------------ - -1. **Memory independent**. Gensim has unique algorithms for statistical analysis that allow - you to create semantic models of arbitrarily large training corpora (larger than RAM) very quickly - and in constant RAM. -2. **Memory independent (again)**. Indexing shards are stored as files to disk/mmapped back as needed, - so you can index very large corpora. So again, constant RAM, this time independent of the number of indexed documents. -3. **Efficient**. Gensim makes heavy use of Python's NumPy and SciPy libraries to make indexing and - querying efficient. -4. **Robust**. Modifications of the index are transactional, so you can commit/rollback an - entire indexing session. Also, during the session, the service is still available - for querying (using its state from when the session started). Power failures leave - service in a consistent state (implicit rollback). -5. **Pure Python**. Well, technically, NumPy and SciPy are mostly wrapped C and Fortran, but - `gensim `_ itself is pure Python. No compiling, installing or root priviledges needed. -6. **Concurrency support**. The underlying service object is thread-safe and can - therefore be used as a daemon server: clients connect to it via RPC and issue train/index/query requests remotely. -7. **Cross-network, cross-platform and cross-language**. While the Python server runs - over TCP using `Pyro `_, - clients in Java/.NET are trivial thanks to `Pyrolite `_. - -The rest of this document serves as a tutorial explaining the features in more detail. - ------ - -Prerequisites ----------------------- - -It is assumed you have `gensim` properly :doc:`installed `. You'll also -need the `sqlitedict `_ package that wraps -Python's sqlite3 module in a thread-safe manner:: - - $ sudo easy_install -U sqlitedict - -To test the remote server capabilities, install Pyro4 (Python Remote Objects, at -version 4.8 as of this writing):: - - $ sudo easy_install Pyro4 - -.. note:: - Don't forget to initialize logging to see logging messages:: - - >>> import logging - >>> logging.basicConfig(format='%(asctime)s : %(levelname)s : %(message)s', level=logging.INFO) - -What is a document? -------------------- - -In case of text documents, the service expects:: - ->>> document = {'id': 'some_unique_string', ->>> 'tokens': ['content', 'of', 'the', 'document', '...'], ->>> 'other_fields_are_allowed_but_ignored': None} - -This format was chosen because it coincides with plain JSON and is therefore easy to serialize and send over the wire, in almost any language. -All strings involved must be utf8-encoded. - - -What is a corpus? ------------------ - -A sequence of documents. Anything that supports the `for document in corpus: ...` -iterator protocol. Generators are ok. Plain lists are also ok (but consume more memory). - ->>> from gensim import utils ->>> texts = ["Human machine interface for lab abc computer applications", ->>> "A survey of user opinion of computer system response time", ->>> "The EPS user interface management system", ->>> "System and human system engineering testing of EPS", ->>> "Relation of user perceived response time to error measurement", ->>> "The generation of random binary unordered trees", ->>> "The intersection graph of paths in trees", ->>> "Graph minors IV Widths of trees and well quasi ordering", ->>> "Graph minors A survey"] ->>> corpus = [{'id': 'doc_%i' % num, 'tokens': utils.simple_preprocess(text)} ->>> for num, text in enumerate(texts)] - -Since corpora are allowed to be arbitrarily large, it is -recommended client splits them into smaller chunks before uploading them to the server: - ->>> utils.upload_chunked(server, corpus, chunksize=1000) # send 1k docs at a time - -Wait, upload what, where? -------------------------- - -If you use the similarity service object (instance of :class:`simserver.SessionServer`) in -your code directly---no remote access---that's perfectly fine. Using the service remotely, from a different process/machine, is an -option, not a necessity. - -Document similarity can also act as a long-running service, a daemon process on a separate machine. In that -case, I'll call the service object a *server*. - -But let's start with a local object. Open your `favourite shell `_ and:: - ->>> from gensim import utils ->>> from simserver import SessionServer ->>> service = SessionServer('/tmp/my_server/') # or wherever - -That initialized a new service, located in `/tmp/my_server` (you need write access rights to that directory). - -.. note:: - The service is fully defined by the content of its location directory ("`/tmp/my_server/`"). - If you use an existing location, the service object will resume - from the index found there. Also, to "clone" a service, just copy that - directory somewhere else. The copy will be a fully working duplicate of the - original service. - - -Model training ---------------- - -We can start indexing right away: - ->>> service.index(corpus) -AttributeError: must initialize model for /tmp/my_server/b before indexing documents - -Oops, we can not. The service indexes documents in a semantic representation, which -is different to the plain text we give it. We must teach the service how to convert -between plain text and semantics first:: - ->>> service.train(corpus, method='lsi') - -That was easy. The `method='lsi'` parameter meant that we trained a model for -`Latent Semantic Indexing `_ -and default dimensionality (400) over a `tf-idf `_ -representation of our little `corpus`, all automatically. More on that later. - -Note that for the semantic model to make sense, it should be trained -on a corpus that is: - -* Reasonably similar to the documents you want to index later. Training on a corpus - of recipes in French when all indexed documents will be about programming in English - will not help. -* Reasonably large (at least thousands of documents), so that the statistical analysis has - a chance to kick in. Don't use my example corpus here of 9 documents in production O_o - -Indexing documents ------------------- - ->>> service.index(corpus) # index the same documents that we trained on... - -Indexing can happen over any documents, but I'm too lazy to create another example corpus, so we index the same 9 docs used for training. - -Delete documents with:: - - >>> service.delete(['doc_5', 'doc_8']) # supply a list of document ids to be removed from the index - -When you pass documents that have the same id as some already indexed document, -the indexed document is overwritten by the new input (=only the latest counts; -document ids are always unique per service):: - - >>> service.index(corpus[:3]) # overall index size unchanged (just 3 docs overwritten) - -The index/delete/overwrite calls can be arbitrarily interspersed with queries. -You don't have to index **all** documents first to start querying, indexing can be incremental. - -Querying ---------- - -There are two types of queries: - -1. by id: - - .. code-block:: python - - >>> print service.find_similar('doc_0') - [('doc_0', 1.0, None), ('doc_2', 0.30426699, None), ('doc_1', 0.25648531, None), ('doc_3', 0.25480536, None)] - - >>> print service.find_similar('doc_5') # we deleted doc_5 and doc_8, remember? - ValueError: document 'doc_5' not in index - - In the resulting 3-tuples, `doc_n` is the document id we supplied during indexing, - `0.30426699` is the similarity of `doc_n` to the query, but what's up with that `None`, you ask? - Well, you can associate each document with a "payload", during indexing. - This payload object (anything pickle-able) is later returned during querying. - If you don't specify `doc['payload']` during indexing, queries simply return `None` in the result tuple, as in our example here. - -2. or by document (using `document['tokens']`; id is ignored in this case): - - .. code-block:: python - - >>> doc = {'tokens': utils.simple_preprocess('Graph and minors and humans and trees.')} - >>> print service.find_similar(doc, min_score=0.4, max_results=50) - [('doc_7', 0.93350589, None), ('doc_3', 0.42718196, None)] - -Remote access -------------- - -So far, we did everything in our Python shell, locally. I very much like `Pyro `_, -a pure Python package for Remote Procedure Calls (RPC), so I'll illustrate remote -service access via Pyro. Pyro takes care of all the socket listening/request routing/data marshalling/thread -spawning, so it saves us a lot of trouble. - -To create a similarity server, we just create a :class:`simserver.SessionServer` object and register it -with a Pyro daemon for remote access. There is a small `example script `_ -included with simserver, run it with:: - - $ python -m simserver.run_simserver /tmp/testserver - -You can just `ctrl+c` to terminate the server, but leave it running for now. - -Now open your Python shell again, in another terminal window or possibly on another machine, and:: - ->>> import Pyro4 ->>> service = Pyro4.Proxy(Pyro4.locateNS().lookup('gensim.testserver')) - -Now `service` is only a proxy object: every call is physically executed wherever -you ran the `run_server.py` script, which can be a totally different computer -(within a network broadcast domain), but you don't even know:: - ->>> print service.status() ->>> service.train(corpus) ->>> service.index(other_corpus) ->>> service.find_similar(query) ->>> ... - -It is worth mentioning that Irmen, the author of Pyro, also released -`Pyrolite `_ recently. That is a package -which allows you to create Pyro proxies also from Java and .NET, in addition to Python. -That way you can call remote methods from there too---the client doesn't have to be in Python. - -Concurrency ------------ - -Ok, now it's getting interesting. Since we can access the service remotely, what -happens if multiple clients create proxies to it at the same time? What if they -want to modify the server index at the same time? - -Answer: the `SessionServer` object is thread-safe, so that when each client spawns a request -thread via Pyro, they don't step on each other's toes. - -This means that: - -1. There can be multiple simultaneous `service.find_similar` queries (or, in - general, multiple simultaneus calls that are "read-only"). -2. When two clients issue modification calls (`index`/`train`/`delete`/`drop_index`/...) - at the same time, an internal lock serializes them -- the later call has to wait. -3. While one client is modifying the index, all other clients' queries still see - the original index. Only once the modifications are committed do they become - "visible". - -What do you mean, visible? --------------------------- - -The service uses transactions internally. This means that each modification is -done over a clone of the service. If the modification session fails for whatever -reason (exception in code; power failure that turns off the server; client unhappy -with how the session went), it can be rolled back. It also means other clients can -continue querying the original index during index updates. - -The mechanism is hidden from users by default through auto-committing (it was already happening -in the examples above too), but auto-committing can be turned off explicitly:: - - >>> service.set_autosession(False) - >>> service.train(corpus) - RuntimeError: must open a session before modifying SessionServer - >>> service.open_session() - >>> service.train(corpus) - >>> service.index(corpus) - >>> service.delete(doc_ids) - >>> ... - -None of these changes are visible to other clients, yet. Also, other clients' -calls to index/train/etc will block until this session is committed/rolled back---there -cannot be two open sessions at the same time. - -To end a session:: - - >>> service.rollback() # discard all changes since open_session() - -or:: - - >>> service.commit() # make changes public; now other clients can see changes/acquire the modification lock - - -Other stuff ------------- - -TODO Custom document parsing (in lieu of `utils.simple_preprocess`). Different models (not just `lsi`). Optimizing the index with `service.optimize()`. -TODO add some hard numbers; example tutorial for some bigger collection, e.g. for `arxiv.org `_ or wikipedia. - diff --git a/docs/_sources/tut1.txt b/docs/_sources/tut1.txt deleted file mode 100644 index bd7fc87b0e..0000000000 --- a/docs/_sources/tut1.txt +++ /dev/null @@ -1,269 +0,0 @@ -.. _tut1: - -Corpora and Vector Spaces -=================================== - -Don't forget to set - ->>> import logging ->>> logging.basicConfig(format='%(asctime)s : %(levelname)s : %(message)s', level=logging.INFO) - -if you want to see logging events. - - -.. _second example: - -From Strings to Vectors ------------------------- - -This time, let's start from documents represented as strings: - ->>> from gensim import corpora, models, similarities ->>> ->>> documents = ["Human machine interface for lab abc computer applications", ->>> "A survey of user opinion of computer system response time", ->>> "The EPS user interface management system", ->>> "System and human system engineering testing of EPS", ->>> "Relation of user perceived response time to error measurement", ->>> "The generation of random binary unordered trees", ->>> "The intersection graph of paths in trees", ->>> "Graph minors IV Widths of trees and well quasi ordering", ->>> "Graph minors A survey"] - - -This is a tiny corpus of nine documents, each consisting of only a single sentence. - -First, let's tokenize the documents, remove common words (using a toy stoplist) -as well as words that only appear once in the corpus: - ->>> # remove common words and tokenize ->>> stoplist = set('for a of the and to in'.split()) ->>> texts = [[word for word in document.lower().split() if word not in stoplist] ->>> for document in documents] ->>> ->>> # remove words that appear only once ->>> all_tokens = sum(texts, []) ->>> tokens_once = set(word for word in set(all_tokens) if all_tokens.count(word) == 1) ->>> texts = [[word for word in text if word not in tokens_once] ->>> for text in texts] ->>> ->>> print texts -[['human', 'interface', 'computer'], - ['survey', 'user', 'computer', 'system', 'response', 'time'], - ['eps', 'user', 'interface', 'system'], - ['system', 'human', 'system', 'eps'], - ['user', 'response', 'time'], - ['trees'], - ['graph', 'trees'], - ['graph', 'minors', 'trees'], - ['graph', 'minors', 'survey']] - -Your way of processing the documents will likely vary; here, I only split on whitespace -to tokenize, followed by lowercasing each word. In fact, I use this particular -(simplistic and inefficient) setup to mimick the experiment done in Deerwester et al.'s -original LSA article [1]_. - -The ways to process documents are so varied and application- and language-dependent that I -decided to *not* constrain them by any interface. Instead, a document is represented -by the features extracted from it, not by its "surface" string form: how you get to -the features is up to you. Below I describe one common, general-purpose approach (called -:dfn:`bag-of-words`), but keep in mind that different application domains call for -different features, and, as always, it's `garbage in, garbage out `_... - -To convert documents to vectors, we'll use a document representation called -`bag-of-words `_. In this representation, -each document is represented by one vector where each vector element represents -a question-answer pair, in the style of: - - "How many times does the word `system` appear in the document? Once." - -It is advantageous to represent the questions only by their (integer) ids. The mapping -between the questions and ids is called a dictionary: - ->>> dictionary = corpora.Dictionary(texts) ->>> dictionary.save('/tmp/deerwester.dict') # store the dictionary, for future reference ->>> print dictionary -Dictionary(12 unique tokens) - -Here we assigned a unique integer id to all words appearing in the corpus with the -:class:`gensim.corpora.dictionary.Dictionary` class. This sweeps across the texts, collecting word counts -and relevant statistics. In the end, we see there are twelve distinct words in the -processed corpus, which means each document will be represented by twelve numbers (ie., by a 12-D vector). -To see the mapping between words and their ids: - ->>> print dictionary.token2id -{'minors': 11, 'graph': 10, 'system': 5, 'trees': 9, 'eps': 8, 'computer': 0, -'survey': 4, 'user': 7, 'human': 1, 'time': 6, 'interface': 2, 'response': 3} - -To actually convert tokenized documents to vectors: - ->>> new_doc = "Human computer interaction" ->>> new_vec = dictionary.doc2bow(new_doc.lower().split()) ->>> print new_vec # the word "interaction" does not appear in the dictionary and is ignored -[(0, 1), (1, 1)] - -The function :func:`doc2bow` simply counts the number of occurences of -each distinct word, converts the word to its integer word id -and returns the result as a sparse vector. The sparse vector ``[(0, 1), (1, 1)]`` -therefore reads: in the document `"Human computer interaction"`, the words `computer` -(id 0) and `human` (id 1) appear once; the other ten dictionary words appear (implicitly) zero times. - - >>> corpus = [dictionary.doc2bow(text) for text in texts] - >>> corpora.MmCorpus.serialize('/tmp/deerwester.mm', corpus) # store to disk, for later use - >>> print corpus - [(0, 1), (1, 1), (2, 1)] - [(0, 1), (3, 1), (4, 1), (5, 1), (6, 1), (7, 1)] - [(2, 1), (5, 1), (7, 1), (8, 1)] - [(1, 1), (5, 2), (8, 1)] - [(3, 1), (6, 1), (7, 1)] - [(9, 1)] - [(9, 1), (10, 1)] - [(9, 1), (10, 1), (11, 1)] - [(4, 1), (10, 1), (11, 1)] - -By now it should be clear that the vector feature with ``id=10`` stands for the question "How many -times does the word `graph` appear in the document?" and that the answer is "zero" for -the first six documents and "one" for the remaining three. As a matter of fact, -we have arrived at exactly the same corpus of vectors as in the :ref:`first-example`. - -Corpus Streaming -- One Document at a Time -------------------------------------------- - -Note that `corpus` above resides fully in memory, as a plain Python list. -In this simple example, it doesn't matter much, but just to make things clear, -let's assume there are millions of documents in the corpus. Storing all of them in RAM won't do. -Instead, let's assume the documents are stored in a file on disk, one document per line. Gensim -only requires that a corpus must be able to return one document vector at a time:: - ->>> class MyCorpus(object): ->>> def __iter__(self): ->>> for line in open('mycorpus.txt'): ->>> # assume there's one document per line, tokens separated by whitespace ->>> yield dictionary.doc2bow(line.lower().split()) - -Download the sample `mycorpus.txt file here <./mycorpus.txt>`_. The assumption that -each document occupies one line in a single file is not important; you can mold -the `__iter__` function to fit your input format, whatever it is. -Walking directories, parsing XML, accessing network... -Just parse your input to retrieve a clean list of tokens in each document, -then convert the tokens via a dictionary to their ids and yield the resulting sparse vector inside `__iter__`. - ->>> corpus_memory_friendly = MyCorpus() # doesn't load the corpus into memory! ->>> print corpus_memory_friendly -<__main__.MyCorpus object at 0x10d5690> - -Corpus is now an object. We didn't define any way to print it, so `print` just outputs address -of the object in memory. Not very useful. To see the constituent vectors, let's -iterate over the corpus and print each document vector (one at a time):: - - >>> for vector in corpus_memory_friendly: # load one vector into memory at a time - >>> print vector - [(0, 1), (1, 1), (2, 1)] - [(0, 1), (3, 1), (4, 1), (5, 1), (6, 1), (7, 1)] - [(2, 1), (5, 1), (7, 1), (8, 1)] - [(1, 1), (5, 2), (8, 1)] - [(3, 1), (6, 1), (7, 1)] - [(9, 1)] - [(9, 1), (10, 1)] - [(9, 1), (10, 1), (11, 1)] - [(4, 1), (10, 1), (11, 1)] - -Although the output is the same as for the plain Python list, the corpus is now much -more memory friendly, because at most one vector resides in RAM at a time. Your -corpus can now be as large as you want. - -Similarly, to construct the dictionary without loading all texts into memory:: - - >>> # collect statistics about all tokens - >>> dictionary = corpora.Dictionary(line.lower().split() for line in open('mycorpus.txt')) - >>> # remove stop words and words that appear only once - >>> stop_ids = [dictionary.token2id[stopword] for stopword in stoplist - >>> if stopword in dictionary.token2id] - >>> once_ids = [tokenid for tokenid, docfreq in dictionary.dfs.iteritems() if docfreq == 1] - >>> dictionary.filter_tokens(stop_ids + once_ids) # remove stop words and words that appear only once - >>> dictionary.compactify() # remove gaps in id sequence after words that were removed - >>> print dictionary - Dictionary(12 unique tokens) - -And that is all there is to it! At least as far as bag-of-words representation is concerned. -Of course, what we do with such corpus is another question; it is not at all clear -how counting the frequency of distinct words could be useful. As it turns out, it isn't, and -we will need to apply a transformation on this simple representation first, before -we can use it to compute any meaningful document vs. document similarities. -Transformations are covered in the :doc:`next tutorial `, but before that, let's -briefly turn our attention to *corpus persistency*. - - -.. _corpus-formats: - -Corpus Formats ---------------- - -There exist several file formats for serializing a Vector Space corpus (~sequence of vectors) to disk. -`Gensim` implements them via the *streaming corpus interface* mentioned earlier: -documents are read from (resp. stored to) disk in a lazy fashion, one document at -a time, without the whole corpus being read into main memory at once. - -One of the more notable file formats is the `Market Matrix format `_. -To save a corpus in the Matrix Market format: - ->>> from gensim import corpora ->>> # create a toy corpus of 2 documents, as a plain Python list ->>> corpus = [[(1, 0.5)], []] # make one document empty, for the heck of it ->>> ->>> corpora.MmCorpus.serialize('/tmp/corpus.mm', corpus) - -Other formats include `Joachim's SVMlight format `_, -`Blei's LDA-C format `_ and -`GibbsLDA++ format `_. - ->>> corpora.SvmLightCorpus.serialize('/tmp/corpus.svmlight', corpus) ->>> corpora.BleiCorpus.serialize('/tmp/corpus.lda-c', corpus) ->>> corpora.LowCorpus.serialize('/tmp/corpus.low', corpus) - - -Conversely, to load a corpus iterator from a Matrix Market file: - ->>> corpus = corpora.MmCorpus('/tmp/corpus.mm') - -Corpus objects are streams, so typically you won't be able to print them directly: - ->>> print corpus -MmCorpus(2 documents, 2 features, 1 non-zero entries) - -Instead, to view the contents of a corpus: - ->>> # one way of printing a corpus: load it entirely into memory ->>> print list(corpus) # calling list() will convert any sequence to a plain Python list -[[(1, 0.5)], []] - -or - ->>> # another way of doing it: print one document at a time, making use of the streaming interface ->>> for doc in corpus: ->>> print doc -[(1, 0.5)] -[] - -The second way is obviously more memory-friendly, but for testing and development -purposes, nothing beats the simplicity of calling ``list(corpus)``. - -To save the same Matrix Market document stream in Blei's LDA-C format, - ->>> corpora.BleiCorpus.serialize('/tmp/corpus.lda-c', corpus) - -In this way, `gensim` can also be used as a memory-efficient **I/O format conversion tool**: -just load a document stream using one format and immediately save it in another format. -Adding new formats is dead easy, check out the `code for the SVMlight corpus -`_ for an example. - -------------- - -For a complete reference (Want to prune the dictionary to a smaller size? -Convert between corpora and NumPy/SciPy arrays?), see the :doc:`API documentation `. -Or continue to the next tutorial on :doc:`tut2`. - - -.. [1] This is the same corpus as used in - `Deerwester et al. (1990): Indexing by Latent Semantic Analysis `_, Table 2. - diff --git a/docs/_sources/tut2.txt b/docs/_sources/tut2.txt deleted file mode 100644 index c130c23a1a..0000000000 --- a/docs/_sources/tut2.txt +++ /dev/null @@ -1,249 +0,0 @@ -.. _tut2: - -Topics and Transformations -=========================== - - -Don't forget to set - ->>> import logging ->>> logging.basicConfig(format='%(asctime)s : %(levelname)s : %(message)s', level=logging.INFO) - -if you want to see logging events. - -Transformation interface --------------------------- - -In the previous tutorial on :doc:`tut1`, we created a corpus of documents represented -as a stream of vectors. To continue, let's fire up gensim and use that corpus: - ->>> from gensim import corpora, models, similarities ->>> dictionary = corpora.Dictionary.load('/tmp/deerwester.dict') ->>> corpus = corpora.MmCorpus('/tmp/deerwester.mm') ->>> print corpus -MmCorpus(9 documents, 12 features, 28 non-zero entries) - -In this tutorial, I will show how to transform documents from one vector representation -into another. This process serves two goals: - -1. To bring out hidden structure in the corpus, discover relationships between - words and use them to describe the documents in a new and - (hopefully) more semantic way. -2. To make the document representation more compact. This both improves efficiency - (new representation consumes less resources) and efficacy (marginal data - trends are ignored, noise-reduction). - -Creating a transformation -++++++++++++++++++++++++++ - -The transformations are standard Python objects, typically initialized by means of -a :dfn:`training corpus`: - ->>> tfidf = models.TfidfModel(corpus) # step 1 -- initialize a model - -We used our old corpus from tutorial 1 to initialize (train) the transformation model. Different -transformations may require different initialization parameters; in case of TfIdf, the -"training" consists simply of going through the supplied corpus once and computing document frequencies -of all its features. Training other models, such as Latent Semantic Analysis or Latent Dirichlet -Allocation, is much more involved and, consequently, takes much more time. - -.. note:: - - Transformations always convert between two specific vector - spaces. The same vector space (= the same set of feature ids) must be used for training - as well as for subsequent vector transformations. Failure to use the same input - feature space, such as applying a different string preprocessing, using different - feature ids, or using bag-of-words input vectors where TfIdf vectors are expected, will - result in feature mismatch during transformation calls and consequently in either - garbage output and/or runtime exceptions. - - -Transforming vectors -+++++++++++++++++++++ - -From now on, ``tfidf`` is treated as a read-only object that can be used to convert -any vector from the old representation (bag-of-words integer counts) to the new representation -(TfIdf real-valued weights): - ->>> doc_bow = [(0, 1), (1, 1)] ->>> print tfidf[doc_bow] # step 2 -- use the model to transform vectors -[(0, 0.70710678), (1, 0.70710678)] - -Or to apply a transformation to a whole corpus: - ->>> corpus_tfidf = tfidf[corpus] ->>> for doc in corpus_tfidf: ->>> print doc -[(0, 0.57735026918962573), (1, 0.57735026918962573), (2, 0.57735026918962573)] -[(0, 0.44424552527467476), (3, 0.44424552527467476), (4, 0.44424552527467476), (5, 0.32448702061385548), (6, 0.44424552527467476), (7, 0.32448702061385548)] -[(2, 0.5710059809418182), (5, 0.41707573620227772), (7, 0.41707573620227772), (8, 0.5710059809418182)] -[(1, 0.49182558987264147), (5, 0.71848116070837686), (8, 0.49182558987264147)] -[(3, 0.62825804686700459), (6, 0.62825804686700459), (7, 0.45889394536615247)] -[(9, 1.0)] -[(9, 0.70710678118654746), (10, 0.70710678118654746)] -[(9, 0.50804290089167492), (10, 0.50804290089167492), (11, 0.69554641952003704)] -[(4, 0.62825804686700459), (10, 0.45889394536615247), (11, 0.62825804686700459)] - -In this particular case, we are transforming the same corpus that we used -for training, but this is only incidental. Once the transformation model has been initialized, -it can be used on any vectors (provided they come from the same vector space, of course), -even if they were not used in the training corpus at all. This is achieved by a process called -folding-in for LSA, by topic inference for LDA etc. - -.. note:: - Calling ``model[corpus]`` only creates a wrapper around the old ``corpus`` - document stream -- actual conversions are done on-the-fly, during document iteration. - We cannot convert the entire corpus at the time of calling ``corpus_transformed = model[corpus]``, - because that would mean storing the result in main memory, and that contradicts gensim's objective of memory-indepedence. - If you will be iterating over the transformed ``corpus_transformed`` multiple times, and the - transformation is costly, :ref:`serialize the resulting corpus to disk first ` and continue - using that. - -Transformations can also be serialized, one on top of another, in a sort of chain: - ->>> lsi = models.LsiModel(corpus_tfidf, id2word=dictionary, num_topics=2) # initialize an LSI transformation ->>> corpus_lsi = lsi[corpus_tfidf] # create a double wrapper over the original corpus: bow->tfidf->fold-in-lsi - -Here we transformed our Tf-Idf corpus via `Latent Semantic Indexing `_ -into a latent 2-D space (2-D because we set ``num_topics=2``). Now you're probably wondering: what do these two latent -dimensions stand for? Let's inspect with :func:`models.LsiModel.print_topics`: - - >>> lsi.print_topics(2) - topic #0(1.594): -0.703*"trees" + -0.538*"graph" + -0.402*"minors" + -0.187*"survey" + -0.061*"system" + -0.060*"response" + -0.060*"time" + -0.058*"user" + -0.049*"computer" + -0.035*"interface" - topic #1(1.476): -0.460*"system" + -0.373*"user" + -0.332*"eps" + -0.328*"interface" + -0.320*"response" + -0.320*"time" + -0.293*"computer" + -0.280*"human" + -0.171*"survey" + 0.161*"trees" - -(the topics are printed to log -- see the note at the top of this page about activating -logging) - -It appears that according to LSI, "trees", "graph" and "minors" are all related -words (and contribute the most to the direction of the first topic), while the -second topic practically concerns itself with all the other words. As expected, -the first five documents are more strongly related to the second topic while the -remaining four documents to the first topic: - ->>> for doc in corpus_lsi: # both bow->tfidf and tfidf->lsi transformations are actually executed here, on the fly ->>> print doc -[(0, -0.066), (1, 0.520)] # "Human machine interface for lab abc computer applications" -[(0, -0.197), (1, 0.761)] # "A survey of user opinion of computer system response time" -[(0, -0.090), (1, 0.724)] # "The EPS user interface management system" -[(0, -0.076), (1, 0.632)] # "System and human system engineering testing of EPS" -[(0, -0.102), (1, 0.574)] # "Relation of user perceived response time to error measurement" -[(0, -0.703), (1, -0.161)] # "The generation of random binary unordered trees" -[(0, -0.877), (1, -0.168)] # "The intersection graph of paths in trees" -[(0, -0.910), (1, -0.141)] # "Graph minors IV Widths of trees and well quasi ordering" -[(0, -0.617), (1, 0.054)] # "Graph minors A survey" - - -Model persistency is achieved with the :func:`save` and :func:`load` functions: - ->>> lsi.save('/tmp/model.lsi') # same for tfidf, lda, ... ->>> lsi = models.LsiModel.load('/tmp/model.lsi') - - -The next question might be: just how exactly similar are those documents to each other? -Is there a way to formalize the similarity, so that for a given input document, we can -order some other set of documents according to their similarity? Similarity queries -are covered in the :doc:`next tutorial `. - -.. _transformations: - -Available transformations --------------------------- - -Gensim implements several popular Vector Space Model algorithms: - -* `Term Frequency * Inverse Document Frequency, Tf-Idf `_ - expects a bag-of-words (integer values) training corpus during initialization. - During transformation, it will take a vector and return another vector of the - same dimensionality, except that features which were rare in the training corpus - will have their value increased. - It therefore converts integer-valued vectors into real-valued ones, while leaving - the number of dimensions intact. It can also optionally normalize the resulting - vectors to (Euclidean) unit length. - - >>> model = tfidfmodel.TfidfModel(bow_corpus, normalize=True) - -* `Latent Semantic Indexing, LSI (or sometimes LSA) `_ - transforms documents from either bag-of-words or (preferrably) TfIdf-weighted space into - a latent space of a lower dimensionality. For the toy corpus above we used only - 2 latent dimensions, but on real corpora, target dimensionality of 200--500 is recommended - as a "golden standard" [1]_. - - >>> model = lsimodel.LsiModel(tfidf_corpus, id2word=dictionary, num_topics=300) - - LSI training is unique in that we can continue "training" at any point, simply - by providing more training documents. This is done by incremental updates to - the underlying model, in a process called `online training`. Because of this feature, the - input document stream may even be infinite -- just keep feeding LSI new documents - as they arrive, while using the computed transformation model as read-only in the meanwhile! - - >>> model.add_documents(another_tfidf_corpus) # now LSI has been trained on tfidf_corpus + another_tfidf_corpus - >>> lsi_vec = model[tfidf_vec] # convert some new document into the LSI space, without affecting the model - >>> ... - >>> model.add_documents(more_documents) # tfidf_corpus + another_tfidf_corpus + more_documents - >>> lsi_vec = model[tfidf_vec] - >>> ... - - See the :mod:`gensim.models.lsimodel` documentation for details on how to make - LSI gradually "forget" old observations in infinite streams. If you want to get dirty, - there are also parameters you can tweak that affect speed vs. memory footprint vs. numerical - precision of the LSI algorithm. - - `gensim` uses a novel online incremental streamed distributed training algorithm (quite a mouthful!), - which I published in [5]_. `gensim` also executes a stochastic multi-pass algorithm - from Halko et al. [4]_ internally, to accelerate in-core part - of the computations. - See also :doc:`wiki` for further speed-ups by distributing the computation across - a cluster of computers. - -* `Random Projections, RP `_ aim to - reduce vector space dimensionality. This is a very efficient (both memory- and - CPU-friendly) approach to approximating TfIdf distances between documents, by throwing in a little randomness. - Recommended target dimensionality is again in the hundreds/thousands, depending on your dataset. - - >>> model = rpmodel.RpModel(tfidf_corpus, num_topics=500) - -* `Latent Dirichlet Allocation, LDA `_ - is yet another transformation from bag-of-words counts into a topic space of lower - dimensionality. LDA is a probabilistic extension of LSA (also called multinomial PCA), - so LDA's topics can be interpreted as probability distributions over words. These distributions are, - just like with LSA, inferred automatically from a training corpus. Documents - are in turn interpreted as a (soft) mixture of these topics (again, just like with LSA). - - >>> model = ldamodel.LdaModel(bow_corpus, id2word=dictionary, num_topics=100) - - `gensim` uses a fast implementation of online LDA parameter estimation based on [2]_, - modified to run in :doc:`distributed mode ` on a cluster of computers. - -* `Hierarchical Dirichlet Process, HDP `_ - is a non-parametric bayesian method (note the missing number of requested topics): - - >>> model = hdpmodel.HdpModel(bow_corpus, id2word=dictionary) - - `gensim` uses a fast, online implementation based on [3]_. - The HDP model is a new addition to `gensim`, and still rough around its academic edges -- use with care. - -Adding new :abbr:`VSM (Vector Space Model)` transformations (such as different weighting schemes) is rather trivial; -see the :doc:`API reference ` or directly the `Python code `_ -for more info and examples. - -It is worth repeating that these are all unique, **incremental** implementations, -which do not require the whole training corpus to be present in main memory all at once. -With memory taken care of, I am now improving :doc:`distributed`, -to improve CPU efficiency, too. -If you feel you could contribute (by testing, providing use-cases or code), -please `let me know `_. - -Continue on to the next tutorial on :doc:`tut3`. - ------- - -.. [1] Bradford. 2008. An empirical study of required dimensionality for large-scale latent semantic indexing applications. - -.. [2] Hoffman, Blei, Bach. 2010. Online learning for Latent Dirichlet Allocation. - -.. [3] Wang, Paisley, Blei. 2011. Online variational inference for the hierarchical Dirichlet process. - -.. [4] Halko, Martinsson, Tropp. 2009. Finding structure with randomness. - -.. [5] Řehůřek. 2011. Subspace tracking for Latent Semantic Analysis. diff --git a/docs/_sources/tut3.txt b/docs/_sources/tut3.txt deleted file mode 100644 index 2a0eca97ed..0000000000 --- a/docs/_sources/tut3.txt +++ /dev/null @@ -1,147 +0,0 @@ -.. _tut3: - -Similarity Queries -=========================== - - -Don't forget to set - ->>> import logging ->>> logging.basicConfig(format='%(asctime)s : %(levelname)s : %(message)s', level=logging.INFO) - -if you want to see logging events. - -Similarity interface --------------------------- - -In the previous tutorials on :doc:`tut1` and :doc:`tut2`, we covered what it means -to create a corpus in the Vector Space Model and how to transform it between different -vector spaces. A common reason for such a charade is that we want to determine -**similarity between pairs of documents**, or the **similarity between a specific document -and a set of other documents** (such as a user query vs. indexed documents). - -To show how this can be done in gensim, let us consider the same corpus as in the -previous examples (which really originally comes from Deerwester et al.'s -`"Indexing by Latent Semantic Analysis" `_ -seminal 1990 article): - ->>> from gensim import corpora, models, similarities ->>> dictionary = corpora.Dictionary.load('/tmp/deerwester.dict') ->>> corpus = corpora.MmCorpus('/tmp/deerwester.mm') # comes from the first tutorial, "From strings to vectors" ->>> print corpus -MmCorpus(9 documents, 12 features, 28 non-zero entries) - -To follow Deerwester's example, we first use this tiny corpus to define a 2-dimensional -LSI space: - ->>> lsi = models.LsiModel(corpus, id2word=dictionary, num_topics=2) - -Now suppose a user typed in the query `"Human computer interaction"`. We would -like to sort our nine corpus documents in decreasing order of relevance to this query. -Unlike modern search engines, here we only concentrate on a single aspect of possible -similarities---on apparent semantic relatedness of their texts (words). No hyperlinks, -no random-walk static ranks, just a semantic extension over the boolean keyword match: - ->>> doc = "Human computer interaction" ->>> vec_bow = dictionary.doc2bow(doc.lower().split()) ->>> vec_lsi = lsi[vec_bow] # convert the query to LSI space ->>> print vec_lsi -[(0, -0.461821), (1, 0.070028)] - -In addition, we will be considering `cosine similarity `_ -to determine the similarity of two vectors. Cosine similarity is a standard measure -in Vector Space Modeling, but wherever the vectors represent probability distributions, -`different similarity measures `_ -may be more appropriate. - -Initializing query structures -++++++++++++++++++++++++++++++++ - -To prepare for similarity queries, we need to enter all documents which we want -to compare against subsequent queries. In our case, they are the same nine documents -used for training LSI, converted to 2-D LSA space. But that's only incidental, we -might also be indexing a different corpus altogether. - ->>> index = similarities.MatrixSimilarity(lsi[corpus]) # transform corpus to LSI space and index it - -.. warning:: - The class :class:`similarities.MatrixSimilarity` is only appropriate when the whole - set of vectors fits into memory. For example, a corpus of one million documents - would require 2GB of RAM in a 256-dimensional LSI space, when used with this class. - Without 2GB of free RAM, you would need to use the :class:`similarities.Similarity` class. - This class operates in fixed memory, by splitting the index across multiple files on disk. - It uses :class:`similarities.MatrixSimilarity` and :class:`similarities.SparseMatrixSimilarity` internally, - so it is still fast, although slightly more complex. - -Index persistency is handled via the standard :func:`save` and :func:`load` functions: - ->>> index.save('/tmp/deerwester.index') ->>> index = similarities.MatrixSimilarity.load('/tmp/deerwester.index') - -This is true for all similarity indexing classes (:class:`similarities.Similarity`, -:class:`similarities.MatrixSimilarity` and :class:`similarities.SparseMatrixSimilarity`). -Also in the following, `index` can be an object of any of these. When in doubt, -use :class:`similarities.Similarity`, as it is the most scalable version, and it also -supports adding more documents to the index later. - -Performing queries -+++++++++++++++++++++ - -To obtain similarities of our query document against the nine indexed documents: - ->>> sims = index[vec_lsi] # perform a similarity query against the corpus ->>> print list(enumerate(sims)) # print (document_number, document_similarity) 2-tuples -[(0, 0.99809301), (1, 0.93748635), (2, 0.99844527), (3, 0.9865886), (4, 0.90755945), -(5, -0.12416792), (6, -0.1063926), (7, -0.098794639), (8, 0.05004178)] - -Cosine measure returns similarities in the range `<-1, 1>` (the greater, the more similar), -so that the first document has a score of 0.99809301 etc. - -With some standard Python magic we sort these similarities into descending -order, and obtain the final answer to the query `"Human computer interaction"`: - ->>> sims = sorted(enumerate(sims), key=lambda item: -item[1]) ->>> print sims # print sorted (document number, similarity score) 2-tuples -[(2, 0.99844527), # The EPS user interface management system -(0, 0.99809301), # Human machine interface for lab abc computer applications -(3, 0.9865886), # System and human system engineering testing of EPS -(1, 0.93748635), # A survey of user opinion of computer system response time -(4, 0.90755945), # Relation of user perceived response time to error measurement -(8, 0.050041795), # Graph minors A survey -(7, -0.098794639), # Graph minors IV Widths of trees and well quasi ordering -(6, -0.1063926), # The intersection graph of paths in trees -(5, -0.12416792)] # The generation of random binary unordered trees - -(I added the original documents in their "string form" to the output comments, to -improve clarity.) - -The thing to note here is that documents no. 2 (``"The EPS user interface management system"``) -and 4 (``"Relation of user perceived response time to error measurement"``) would never be returned by -a standard boolean fulltext search, because they do not share any common words with ``"Human -computer interaction"``. However, after applying LSI, we can observe that both of -them received quite high similarity scores (no. 2 is actually the most similar!), -which corresponds better to our intuition of -them sharing a "computer-human" related topic with the query. In fact, this semantic -generalization is the reason why we apply transformations and do topic modelling -in the first place. - - -Where next? ------------- - -Congratulations, you have finished the tutorials -- now you know how gensim works :-) -To delve into more details, you can browse through the :doc:`API documentation `, -see the :doc:`Wikipedia experiments ` or perhaps check out :doc:`distributed computing ` in `gensim`. - -Please remember that gensim is an experimental package, aimed at the NLP research community. -This means that: - -* there certainly are parts that could be implemented more efficiently (in C, for example), and there may also be bugs in the code -* your **feedback is most welcome** and appreciated, be it in code and - `idea contributions `_, - `bug reports `_ or just - `user stories and general questions `_. - -Gensim has no ambition to become an all-encompassing production level tool, with robust failure handling -and error recoveries. Its main goal is to help NLP newcomers try out popular algorithms -and to facilitate prototyping of new algorithms for NLP researchers. diff --git a/docs/_sources/tutorial.txt b/docs/_sources/tutorial.txt deleted file mode 100644 index a25f54f80b..0000000000 --- a/docs/_sources/tutorial.txt +++ /dev/null @@ -1,115 +0,0 @@ -.. _tutorial: - -Tutorials -========= - - -The tutorials are organized as a series of examples that highlight various features -of `gensim`. It is assumed that the reader is familiar with the Python language -and has read the :doc:`intro`. - -The examples are divided into parts on: - -.. toctree:: - :maxdepth: 2 - - tut1 - tut2 - tut3 - wiki - distributed - -Preliminaries --------------- - -All the examples can be directly copied to your Python interpreter shell (assuming -you have :doc:`gensim installed `, of course). -`IPython `_'s ``cpaste`` command is especially handy for copypasting code fragments which include superfluous -characters, such as the leading ``>>>``. - -Gensim uses Python's standard :mod:`logging` module to log various stuff at various -priority levels; to activate logging (this is optional), run - ->>> import logging ->>> logging.basicConfig(format='%(asctime)s : %(levelname)s : %(message)s', level=logging.INFO) - - -.. _first-example: - -Quick Example -------------- - -First, let's import gensim and create a small corpus of nine documents [1]_: - ->>> from gensim import corpora, models, similarities ->>> ->>> corpus = [[(0, 1.0), (1, 1.0), (2, 1.0)], ->>> [(2, 1.0), (3, 1.0), (4, 1.0), (5, 1.0), (6, 1.0), (8, 1.0)], ->>> [(1, 1.0), (3, 1.0), (4, 1.0), (7, 1.0)], ->>> [(0, 1.0), (4, 2.0), (7, 1.0)], ->>> [(3, 1.0), (5, 1.0), (6, 1.0)], ->>> [(9, 1.0)], ->>> [(9, 1.0), (10, 1.0)], ->>> [(9, 1.0), (10, 1.0), (11, 1.0)], ->>> [(8, 1.0), (10, 1.0), (11, 1.0)]] - -:dfn:`Corpus` is simply an object which, when iterated over, returns its documents represented -as sparse vectors. - -If you're familiar with the `Vector Space Model `_, -you'll probably know that the way you parse your documents and convert them to vectors -has major impact on the quality of any subsequent applications. If you're not familiar -with :abbr:`VSM (Vector Space Model)`, we'll bridge the gap between **raw strings** -and **sparse vectors** in the next tutorial -on :doc:`tut1`. - -.. note:: - In this example, the whole corpus is stored in memory, as a Python list. However, - the corpus interface only dictates that a corpus must support iteration over its - constituent documents. For very large corpora, it is advantageous to keep the - corpus on disk, and access its documents sequentially, one at a time. All the - operations and transformations are implemented in such a way that makes - them independent of the size of the corpus, memory-wise. - - -Next, let's initialize a :dfn:`transformation`: - ->>> tfidf = models.TfidfModel(corpus) - -A transformation is used to convert documents from one vector representation into another: - ->>> vec = [(0, 1), (4, 1)] ->>> print tfidf[vec] -[(0, 0.8075244), (4, 0.5898342)] - -Here, we used `Tf-Idf `_, a simple -transformation which takes documents represented as bag-of-words counts and applies -a weighting which discounts common terms (or, equivalently, promotes rare terms). -It also scales the resulting vector to unit length (in the `Euclidean norm `_). - -Transformations are covered in detail in the tutorial on :doc:`tut2`. - -To transform the whole corpus via TfIdf and index it, in preparation for similarity queries: - ->>> index = similarities.SparseMatrixSimilarity(tfidf[corpus]) - -and to query the similarity of our query vector ``vec`` against every document in the corpus: - ->>> sims = index[tfidf[vec]] ->>> print list(enumerate(sims)) -[(0, 0.4662244), (1, 0.19139354), (2, 0.24600551), (3, 0.82094586), (4, 0.0), (5, 0.0), (6, 0.0), (7, 0.0), (8, 0.0)] - -How to read this output? Document number zero (the first document) has a similarity score of 0.466=46.6\%, -the second document has a similarity score of 19.1\% etc. - -Thus, according to TfIdf document representation and cosine similarity measure, -the most similar to our query document `vec` is document no. 3, with a similarity score of 82.1%. -Note that in the TfIdf representation, any documents which do not share any common features -with ``vec`` at all (documents no. 4--8) get a similarity score of 0.0. See the :doc:`tut3` tutorial for more detail. - ------- - -.. [1] This is the same corpus as used in - `Deerwester et al. (1990): Indexing by Latent Semantic Analysis `_, Table 2. - - diff --git a/docs/_sources/utils.txt b/docs/_sources/utils.txt deleted file mode 100644 index ffc98e25f8..0000000000 --- a/docs/_sources/utils.txt +++ /dev/null @@ -1,7 +0,0 @@ -:mod:`utils` -- Various utility functions -========================================== - -.. automodule:: gensim.utils - :synopsis: Various utility functions - :members: - :inherited-members: diff --git a/docs/_sources/wiki.txt b/docs/_sources/wiki.txt deleted file mode 100644 index 3bb6aa097f..0000000000 --- a/docs/_sources/wiki.txt +++ /dev/null @@ -1,206 +0,0 @@ -.. _wiki: - -Experiments on the English Wikipedia -============================================ - -To test `gensim` performance, we run it against the English version of Wikipedia. - -This page describes the process of obtaining and processing Wikipedia, so that -anyone can reproduce the results. It is assumed you have `gensim` properly :doc:`installed `. - - - -Preparing the corpus ----------------------- - -1. First, download the dump of all Wikipedia articles from http://download.wikimedia.org/enwiki/ - (you want a file like `enwiki-latest-pages-articles.xml.bz2`). This file is about 8GB in size - and contains (a compressed version of) all articles from the English Wikipedia. - -2. Convert the articles to plain text (process Wiki markup) and store the result as - sparse TF-IDF vectors. In Python, this is easy to do on-the-fly and we don't - even need to uncompress the whole archive to disk. There is a script included in - `gensim` that does just that, run:: - - $ python -m gensim.scripts.make_wiki - -.. note:: - This pre-processing step makes two passes over the 8.2GB compressed wiki dump (one to extract - the dictionary, one to create and store the sparse vectors) and takes about - 9 hours on my laptop, so you may want to go have a coffee or two. - - Also, you will need about 35GB of free disk space to store the sparse output vectors. - I recommend compressing these files immediately, e.g. with bzip2 (down to ~13GB). Gensim - can work with compressed files directly, so this lets you save disk space. - -Latent Sematic Analysis --------------------------- - -First let's load the corpus iterator and dictionary, created in the second step above:: - - >>> import logging, gensim, bz2 - >>> logging.basicConfig(format='%(asctime)s : %(levelname)s : %(message)s', level=logging.INFO) - - >>> # load id->word mapping (the dictionary), one of the results of step 2 above - >>> id2word = gensim.corpora.Dictionary.load_from_text('wiki_en_wordids.txt') - >>> # load corpus iterator - >>> mm = gensim.corpora.MmCorpus('wiki_en_tfidf.mm') - >>> # mm = gensim.corpora.MmCorpus(bz2.BZ2File('wiki_en_tfidf.mm.bz2')) # use this if you compressed the TFIDF output (recommended) - - >>> print mm - MmCorpus(3931787 documents, 100000 features, 756379027 non-zero entries) - -We see that our corpus contains 3.9M documents, 100K features (distinct -tokens) and 0.76G non-zero entries in the sparse TF-IDF matrix. The Wikipedia corpus -contains about 2.24 billion tokens in total. - -Now we're ready to compute LSA of the English Wikipedia:: - - >>> # extract 400 LSI topics; use the default one-pass algorithm - >>> lsi = gensim.models.lsimodel.LsiModel(corpus=mm, id2word=id2word, num_topics=400) - - >>> # print the most contributing words (both positively and negatively) for each of the first ten topics - >>> lsi.print_topics(10) - topic #0(332.762): 0.425*"utc" + 0.299*"talk" + 0.293*"page" + 0.226*"article" + 0.224*"delete" + 0.216*"discussion" + 0.205*"deletion" + 0.198*"should" + 0.146*"debate" + 0.132*"be" - topic #1(201.852): 0.282*"link" + 0.209*"he" + 0.145*"com" + 0.139*"his" + -0.137*"page" + -0.118*"delete" + 0.114*"blacklist" + -0.108*"deletion" + -0.105*"discussion" + 0.100*"diff" - topic #2(191.991): -0.565*"link" + -0.241*"com" + -0.238*"blacklist" + -0.202*"diff" + -0.193*"additions" + -0.182*"users" + -0.158*"coibot" + -0.136*"user" + 0.133*"he" + -0.130*"resolves" - topic #3(141.284): -0.476*"image" + -0.255*"copyright" + -0.245*"fair" + -0.225*"use" + -0.173*"album" + -0.163*"cover" + -0.155*"resolution" + -0.141*"licensing" + 0.137*"he" + -0.121*"copies" - topic #4(130.909): 0.264*"population" + 0.246*"age" + 0.243*"median" + 0.213*"income" + 0.195*"census" + -0.189*"he" + 0.184*"households" + 0.175*"were" + 0.167*"females" + 0.166*"males" - topic #5(120.397): 0.304*"diff" + 0.278*"utc" + 0.213*"you" + -0.171*"additions" + 0.165*"talk" + -0.159*"image" + 0.159*"undo" + 0.155*"www" + -0.152*"page" + 0.148*"contribs" - topic #6(115.414): -0.362*"diff" + -0.203*"www" + 0.197*"you" + -0.180*"undo" + -0.180*"kategori" + 0.164*"users" + 0.157*"additions" + -0.150*"contribs" + -0.139*"he" + -0.136*"image" - topic #7(111.440): 0.429*"kategori" + 0.276*"categoria" + 0.251*"category" + 0.207*"kategorija" + 0.198*"kategorie" + -0.188*"diff" + 0.163*"категория" + 0.153*"categoría" + 0.139*"kategoria" + 0.133*"categorie" - topic #8(109.907): 0.385*"album" + 0.224*"song" + 0.209*"chart" + 0.204*"band" + 0.169*"released" + 0.151*"music" + 0.142*"diff" + 0.141*"vocals" + 0.138*"she" + 0.132*"guitar" - topic #9(102.599): -0.237*"league" + -0.214*"he" + -0.180*"season" + -0.174*"football" + -0.166*"team" + 0.159*"station" + -0.137*"played" + -0.131*"cup" + 0.131*"she" + -0.128*"utc" - -Creating the LSI model of Wikipedia takes about 4 hours and 9 minutes on my laptop [1]_. -That's about **16,000 documents per minute, including all I/O**. - -.. note:: - If you need your results even faster, see the tutorial on :doc:`distributed`. Note - that the BLAS libraries inside `gensim` make use of multiple cores transparently, so the same data - will be processed faster on a multicore machine "for free", without any distributed setup. - -We see that the total processing time is dominated by the preprocessing step of -preparing the TF-IDF corpus from a raw Wikipedia XML dump, which took 9h. [2]_ - -The algorithm used in `gensim` only needs to see each input document once, so it -is suitable for environments where the documents come as a non-repeatable stream, -or where the cost of storing/iterating over the corpus multiple times is too high. - - -Latent Dirichlet Allocation ----------------------------- - -As with Latent Semantic Analysis above, first load the corpus iterator and dictionary:: - - >>> import logging, gensim, bz2 - >>> logging.basicConfig(format='%(asctime)s : %(levelname)s : %(message)s', level=logging.INFO) - - >>> # load id->word mapping (the dictionary), one of the results of step 2 above - >>> id2word = gensim.corpora.Dictionary.load_from_text('wiki_en_wordids.txt') - >>> # load corpus iterator - >>> mm = gensim.corpora.MmCorpus('wiki_en_tfidf.mm') - >>> # mm = gensim.corpora.MmCorpus(bz2.BZ2File('wiki_en_tfidf.mm.bz2')) # use this if you compressed the TFIDF output - - >>> print mm - MmCorpus(3931787 documents, 100000 features, 756379027 non-zero entries) - -We will run online LDA (see Hoffman et al. [3]_), which is an algorithm that takes a chunk of documents, -updates the LDA model, takes another chunk, updates the model etc. Online LDA can be contrasted -with batch LDA, which processes the whole corpus (one full pass), then updates -the model, then another pass, another update... The difference is that given a -reasonably stationary document stream (not much topic drift), the online updates -over the smaller chunks (subcorpora) are pretty good in themselves, so that the -model estimation converges faster. As a result, we will perhaps only need a single full -pass over the corpus: if the corpus has 3 million articles, and we update once after -every 10,000 articles, this means we will have done 300 updates in one pass, quite likely -enough to have a very accurate topics estimate:: - - >>> # extract 100 LDA topics, using 1 pass and updating once every 1 chunk (10,000 documents) - >>> lda = gensim.models.ldamodel.LdaModel(corpus=mm, id2word=id2word, num_topics=100, update_every=1, chunksize=10000, passes=1) - using serial LDA version on this node - running online LDA training, 100 topics, 1 passes over the supplied corpus of 3931787 documents, updating model once every 10000 documents - ... - -Unlike LSA, the topics coming from LDA are easier to interpret:: - - >>> # print the most contributing words for 20 randomly selected topics - >>> lda.print_topics(20) - topic #0: 0.009*river + 0.008*lake + 0.006*island + 0.005*mountain + 0.004*area + 0.004*park + 0.004*antarctic + 0.004*south + 0.004*mountains + 0.004*dam - topic #1: 0.026*relay + 0.026*athletics + 0.025*metres + 0.023*freestyle + 0.022*hurdles + 0.020*ret + 0.017*divisão + 0.017*athletes + 0.016*bundesliga + 0.014*medals - topic #2: 0.002*were + 0.002*he + 0.002*court + 0.002*his + 0.002*had + 0.002*law + 0.002*government + 0.002*police + 0.002*patrolling + 0.002*their - topic #3: 0.040*courcelles + 0.035*centimeters + 0.023*mattythewhite + 0.021*wine + 0.019*stamps + 0.018*oko + 0.017*perennial + 0.014*stubs + 0.012*ovate + 0.011*greyish - topic #4: 0.039*al + 0.029*sysop + 0.019*iran + 0.015*pakistan + 0.014*ali + 0.013*arab + 0.010*islamic + 0.010*arabic + 0.010*saudi + 0.010*muhammad - topic #5: 0.020*copyrighted + 0.020*northamerica + 0.014*uncopyrighted + 0.007*rihanna + 0.005*cloudz + 0.005*knowles + 0.004*gaga + 0.004*zombie + 0.004*wigan + 0.003*maccabi - topic #6: 0.061*israel + 0.056*israeli + 0.030*sockpuppet + 0.025*jerusalem + 0.025*tel + 0.023*aviv + 0.022*palestinian + 0.019*ifk + 0.016*palestine + 0.014*hebrew - topic #7: 0.015*melbourne + 0.014*rovers + 0.013*vfl + 0.012*australian + 0.012*wanderers + 0.011*afl + 0.008*dinamo + 0.008*queensland + 0.008*tracklist + 0.008*brisbane - topic #8: 0.011*film + 0.007*her + 0.007*she + 0.004*he + 0.004*series + 0.004*his + 0.004*episode + 0.003*films + 0.003*television + 0.003*best - topic #9: 0.019*wrestling + 0.013*château + 0.013*ligue + 0.012*discus + 0.012*estonian + 0.009*uci + 0.008*hockeyarchives + 0.008*wwe + 0.008*estonia + 0.007*reign - topic #10: 0.078*edits + 0.059*notability + 0.035*archived + 0.025*clearer + 0.022*speedy + 0.021*deleted + 0.016*hook + 0.015*checkuser + 0.014*ron + 0.011*nominator - topic #11: 0.013*admins + 0.009*acid + 0.009*molniya + 0.009*chemical + 0.007*ch + 0.007*chemistry + 0.007*compound + 0.007*anemone + 0.006*mg + 0.006*reaction - topic #12: 0.018*india + 0.013*indian + 0.010*tamil + 0.009*singh + 0.008*film + 0.008*temple + 0.006*kumar + 0.006*hindi + 0.006*delhi + 0.005*bengal - topic #13: 0.047*bwebs + 0.024*malta + 0.020*hobart + 0.019*basa + 0.019*columella + 0.019*huon + 0.018*tasmania + 0.016*popups + 0.014*tasmanian + 0.014*modèle - topic #14: 0.014*jewish + 0.011*rabbi + 0.008*bgwhite + 0.008*lebanese + 0.007*lebanon + 0.006*homs + 0.005*beirut + 0.004*jews + 0.004*hebrew + 0.004*caligari - topic #15: 0.025*german + 0.020*der + 0.017*von + 0.015*und + 0.014*berlin + 0.012*germany + 0.012*die + 0.010*des + 0.008*kategorie + 0.007*cross - topic #16: 0.003*can + 0.003*system + 0.003*power + 0.003*are + 0.003*energy + 0.002*data + 0.002*be + 0.002*used + 0.002*or + 0.002*using - topic #17: 0.049*indonesia + 0.042*indonesian + 0.031*malaysia + 0.024*singapore + 0.022*greek + 0.021*jakarta + 0.016*greece + 0.015*dord + 0.014*athens + 0.011*malaysian - topic #18: 0.031*stakes + 0.029*webs + 0.018*futsal + 0.014*whitish + 0.013*hyun + 0.012*thoroughbred + 0.012*dnf + 0.012*jockey + 0.011*medalists + 0.011*racehorse - topic #19: 0.119*oblast + 0.034*uploaded + 0.034*uploads + 0.033*nordland + 0.025*selsoviet + 0.023*raion + 0.022*krai + 0.018*okrug + 0.015*hålogaland + 0.015*russiae + 0.020*manga + 0.017*dragon + 0.012*theme + 0.011*dvd + 0.011*super + 0.011*hunter + 0.009*ash + 0.009*dream + 0.009*angel - -Creating this LDA model of Wikipedia takes about 6 hours and 20 minutes on my laptop [1]_. -If you need your results faster, consider running :doc:`dist_lda` on a cluster of -computers. - -Note two differences between the LDA and LSA runs: we asked LSA -to extract 400 topics, LDA only 100 topics (so the difference in speed is in fact -even greater). Secondly, the LSA implementation in `gensim` is truly online: if the nature of the input -stream changes in time, LSA will re-orient itself to reflect these changes, in a reasonably -small amount of updates. In contrast, LDA is not truly online (the name of the [3]_ -article notwithstanding), as the impact of later updates on the model gradually -diminishes. If there is topic drift in the input document stream, LDA will get -confused and be increasingly slower at adjusting itself to the new state of affairs. - -In short, be careful if using LDA to incrementally add new documents to the model -over time. **Batch usage of LDA**, where the entire training corpus is either known beforehand or does -not exhibit topic drift, **is ok and not affected**. - -To run batch LDA (not online), train `LdaModel` with:: - - >>> # extract 100 LDA topics, using 20 full passes, no online updates - >>> lda = gensim.models.ldamodel.LdaModel(corpus=mm, id2word=id2word, num_topics=100, update_every=0, passes=20) - -As usual, a trained model can used be to transform new, unseen documents (plain bag-of-words count vectors) -into LDA topic distributions: - - >>> doc_lda = lda[doc_bow] - --------------------- - -.. [1] My laptop = MacBook Pro, Intel Core i7 2.3GHz, 16GB DDR3 RAM, OS X with `libVec`. - -.. [2] - Here we're mostly interested in performance, but it is interesting to look at the - retrieved LSA concepts, too. I am no Wikipedia expert and don't see into Wiki's bowels, - but Brian Mingus had this to say about the result:: - - There appears to be a lot of noise in your dataset. The first three topics - in your list appear to be meta topics, concerning the administration and - cleanup of Wikipedia. These show up because you didn't exclude templates - such as these, some of which are included in most articles for quality - control: http://en.wikipedia.org/wiki/Wikipedia:Template_messages/Cleanup - - The fourth and fifth topics clearly shows the influence of bots that import - massive databases of cities, countries, etc. and their statistics such as - population, capita, etc. - - The sixth shows the influence of sports bots, and the seventh of music bots. - - So the top ten concepts are apparently dominated by Wikipedia robots and expanded - templates; this is a good reminder that LSA is a powerful tool for data analysis, - but no silver bullet. As always, it's `garbage in, garbage out - `_... - By the way, improvements to the Wiki markup parsing code are welcome :-) - -.. [3] Hoffman, Blei, Bach. 2010. Online learning for Latent Dirichlet Allocation - [`pdf `_] [`code `_] - diff --git a/docs/_static/basic.css b/docs/_static/basic.css deleted file mode 100644 index 43e8bafaf3..0000000000 --- a/docs/_static/basic.css +++ /dev/null @@ -1,540 +0,0 @@ -/* - * basic.css - * ~~~~~~~~~ - * - * Sphinx stylesheet -- basic theme. - * - * :copyright: Copyright 2007-2011 by the Sphinx team, see AUTHORS. - * :license: BSD, see LICENSE for details. - * - */ - -/* -- main layout ----------------------------------------------------------- */ - -div.clearer { - clear: both; -} - -/* -- relbar ---------------------------------------------------------------- */ - -div.related { - width: 100%; - font-size: 90%; -} - -div.related h3 { - display: none; -} - -div.related ul { - margin: 0; - padding: 0 0 0 10px; - list-style: none; -} - -div.related li { - display: inline; -} - -div.related li.right { - float: right; - margin-right: 5px; -} - -/* -- sidebar --------------------------------------------------------------- */ - -div.sphinxsidebarwrapper { - padding: 10px 5px 0 10px; -} - -div.sphinxsidebar { - float: left; - width: 230px; - margin-left: -100%; - font-size: 90%; -} - -div.sphinxsidebar ul { - list-style: none; -} - -div.sphinxsidebar ul ul, -div.sphinxsidebar ul.want-points { - margin-left: 20px; - list-style: square; -} - -div.sphinxsidebar ul ul { - margin-top: 0; - margin-bottom: 0; -} - -div.sphinxsidebar form { - margin-top: 10px; -} - -div.sphinxsidebar input { - border: 1px solid #98dbcc; - font-family: sans-serif; - font-size: 1em; -} - -div.sphinxsidebar #searchbox input[type="text"] { - width: 170px; -} - -div.sphinxsidebar #searchbox input[type="submit"] { - width: 30px; -} - -img { - border: 0; -} - -/* -- search page ----------------------------------------------------------- */ - -ul.search { - margin: 10px 0 0 20px; - padding: 0; -} - -ul.search li { - padding: 5px 0 5px 20px; - background-image: url(file.png); - background-repeat: no-repeat; - background-position: 0 7px; -} - -ul.search li a { - font-weight: bold; -} - -ul.search li div.context { - color: #888; - margin: 2px 0 0 30px; - text-align: left; -} - -ul.keywordmatches li.goodmatch a { - font-weight: bold; -} - -/* -- index page ------------------------------------------------------------ */ - -table.contentstable { - width: 90%; -} - -table.contentstable p.biglink { - line-height: 150%; -} - -a.biglink { - font-size: 1.3em; -} - -span.linkdescr { - font-style: italic; - padding-top: 5px; - font-size: 90%; -} - -/* -- general index --------------------------------------------------------- */ - -table.indextable { - width: 100%; -} - -table.indextable td { - text-align: left; - vertical-align: top; -} - -table.indextable dl, table.indextable dd { - margin-top: 0; - margin-bottom: 0; -} - -table.indextable tr.pcap { - height: 10px; -} - -table.indextable tr.cap { - margin-top: 10px; - background-color: #f2f2f2; -} - -img.toggler { - margin-right: 3px; - margin-top: 3px; - cursor: pointer; -} - -div.modindex-jumpbox { - border-top: 1px solid #ddd; - border-bottom: 1px solid #ddd; - margin: 1em 0 1em 0; - padding: 0.4em; -} - -div.genindex-jumpbox { - border-top: 1px solid #ddd; - border-bottom: 1px solid #ddd; - margin: 1em 0 1em 0; - padding: 0.4em; -} - -/* -- general body styles --------------------------------------------------- */ - -a.headerlink { - visibility: hidden; -} - -h1:hover > a.headerlink, -h2:hover > a.headerlink, -h3:hover > a.headerlink, -h4:hover > a.headerlink, -h5:hover > a.headerlink, -h6:hover > a.headerlink, -dt:hover > a.headerlink { - visibility: visible; -} - -div.body p.caption { - text-align: inherit; -} - -div.body td { - text-align: left; -} - -.field-list ul { - padding-left: 1em; -} - -.first { - margin-top: 0 !important; -} - -p.rubric { - margin-top: 30px; - font-weight: bold; -} - -img.align-left, .figure.align-left, object.align-left { - clear: left; - float: left; - margin-right: 1em; -} - -img.align-right, .figure.align-right, object.align-right { - clear: right; - float: right; - margin-left: 1em; -} - -img.align-center, .figure.align-center, object.align-center { - display: block; - margin-left: auto; - margin-right: auto; -} - -.align-left { - text-align: left; -} - -.align-center { - text-align: center; -} - -.align-right { - text-align: right; -} - -/* -- sidebars -------------------------------------------------------------- */ - -div.sidebar { - margin: 0 0 0.5em 1em; - border: 1px solid #ddb; - padding: 7px 7px 0 7px; - background-color: #ffe; - width: 40%; - float: right; -} - -p.sidebar-title { - font-weight: bold; -} - -/* -- topics ---------------------------------------------------------------- */ - -div.topic { - border: 1px solid #ccc; - padding: 7px 7px 0 7px; - margin: 10px 0 10px 0; -} - -p.topic-title { - font-size: 1.1em; - font-weight: bold; - margin-top: 10px; -} - -/* -- admonitions ----------------------------------------------------------- */ - -div.admonition { - margin-top: 10px; - margin-bottom: 10px; - padding: 7px; -} - -div.admonition dt { - font-weight: bold; -} - -div.admonition dl { - margin-bottom: 0; -} - -p.admonition-title { - margin: 0px 10px 5px 0px; - font-weight: bold; -} - -div.body p.centered { - text-align: center; - margin-top: 25px; -} - -/* -- tables ---------------------------------------------------------------- */ - -table.docutils { - border: 0; - border-collapse: collapse; -} - -table.docutils td, table.docutils th { - padding: 1px 8px 1px 5px; - border-top: 0; - border-left: 0; - border-right: 0; - border-bottom: 1px solid #aaa; -} - -table.field-list td, table.field-list th { - border: 0 !important; -} - -table.footnote td, table.footnote th { - border: 0 !important; -} - -th { - text-align: left; - padding-right: 5px; -} - -table.citation { - border-left: solid 1px gray; - margin-left: 1px; -} - -table.citation td { - border-bottom: none; -} - -/* -- other body styles ----------------------------------------------------- */ - -ol.arabic { - list-style: decimal; -} - -ol.loweralpha { - list-style: lower-alpha; -} - -ol.upperalpha { - list-style: upper-alpha; -} - -ol.lowerroman { - list-style: lower-roman; -} - -ol.upperroman { - list-style: upper-roman; -} - -dl { - margin-bottom: 15px; -} - -dd p { - margin-top: 0px; -} - -dd ul, dd table { - margin-bottom: 10px; -} - -dd { - margin-top: 3px; - margin-bottom: 10px; - margin-left: 30px; -} - -dt:target, .highlighted { - background-color: #fbe54e; -} - -dl.glossary dt { - font-weight: bold; - font-size: 1.1em; -} - -.field-list ul { - margin: 0; - padding-left: 1em; -} - -.field-list p { - margin: 0; -} - -.refcount { - color: #060; -} - -.optional { - font-size: 1.3em; -} - -.versionmodified { - font-style: italic; -} - -.system-message { - background-color: #fda; - padding: 5px; - border: 3px solid red; -} - -.footnote:target { - background-color: #ffa; -} - -.line-block { - display: block; - margin-top: 1em; - margin-bottom: 1em; -} - -.line-block .line-block { - margin-top: 0; - margin-bottom: 0; - margin-left: 1.5em; -} - -.guilabel, .menuselection { - font-family: sans-serif; -} - -.accelerator { - text-decoration: underline; -} - -.classifier { - font-style: oblique; -} - -abbr, acronym { - border-bottom: dotted 1px; - cursor: help; -} - -/* -- code displays --------------------------------------------------------- */ - -pre { - overflow: auto; - overflow-y: hidden; /* fixes display issues on Chrome browsers */ -} - -td.linenos pre { - padding: 5px 0px; - border: 0; - background-color: transparent; - color: #aaa; -} - -table.highlighttable { - margin-left: 0.5em; -} - -table.highlighttable td { - padding: 0 0.5em 0 0.5em; -} - -tt.descname { - background-color: transparent; - font-weight: bold; - font-size: 1.2em; -} - -tt.descclassname { - background-color: transparent; -} - -tt.xref, a tt { - background-color: transparent; - font-weight: bold; -} - -h1 tt, h2 tt, h3 tt, h4 tt, h5 tt, h6 tt { - background-color: transparent; -} - -.viewcode-link { - float: right; -} - -.viewcode-back { - float: right; - font-family: sans-serif; -} - -div.viewcode-block:target { - margin: -1px -10px; - padding: 0 10px; -} - -/* -- math display ---------------------------------------------------------- */ - -img.math { - vertical-align: middle; -} - -div.body div.math p { - text-align: center; -} - -span.eqno { - float: right; -} - -/* -- printout stylesheet --------------------------------------------------- */ - -@media print { - div.document, - div.documentwrapper, - div.bodywrapper { - margin: 0 !important; - width: 100%; - } - - div.sphinxsidebar, - div.related, - div.footer, - #top-link { - display: none; - } -} \ No newline at end of file diff --git a/docs/_static/contents.png b/docs/_static/contents.png deleted file mode 100644 index 7fb82154a1..0000000000 Binary files a/docs/_static/contents.png and /dev/null differ diff --git a/docs/_static/default.css b/docs/_static/default.css deleted file mode 100644 index 611cb55720..0000000000 --- a/docs/_static/default.css +++ /dev/null @@ -1,574 +0,0 @@ -/** - * Alternate Sphinx design - * Originally created by Armin Ronacher for Werkzeug, adapted by Georg Brandl. - * Minor modifications by Radim Rehurek, who was too lazy to fork the theme. - */ - -table.links { - border-collapse: separate; - border-spacing: 15px; - margin: 10px auto 0px auto; -} - -table.links tr td { - -webkit-border-radius: 15px; - -moz-border-radius: 15px; - border-radius: 15px; - padding: 10px 30px; - color: white; -/* border: 1px solid #86989B;*/ - font-weight: bold; - background-color: #eee; - text-align: center; - background: -webkit-gradient(linear, left top, left bottom, from(#eee), to(#ddd)); - background: -moz-linear-gradient(top, #eee, #ddd); -} - -table.links a { text-decoration: none; } - -#tagcloud { - font-size: 50%; - width: 80%; - margin: 20px auto 0px auto; - padding: 15px; - line-height: 2.4em; - word-spacing: normal; - letter-spacing: normal; - text-transform: none; - text-align: justify; - text-indent: 0; - border: 1px dotted; -} - -.wrd { padding:0; position:relative } -.tagcloud0 { font-size:1.0em; color:#ACC1F3; z-index:10 } -.tagcloud1 { font-size:1.4em; color:#ACC1F3; z-index:9 } -.tagcloud2 { font-size:1.8em; color:#86A0DC; z-index:8} -.tagcloud3 { font-size:2.2em; color:#86A0DC; z-index:7} -.tagcloud4 { font-size:2.6em; color:#607EC5; z-index:6} -.tagcloud5 { font-size:3.0em; color:#607EC5; z-index:5} -.tagcloud6 { font-size:3.3em; color:#4C6DB9; z-index:4} -.tagcloud7 { font-size:3.6em; color:#395CAE; z-index:3} -.tagcloud8 { font-size:3.9em; color:#264CA2; z-index:2} -.tagcloud9 { font-size:4.2em; color:#133B97; z-index:1} -.tagcloud10 { font-size:4.5em; color:#002A8B; z-index:0} - - -body { - font-family: 'Lucida Grande', 'Lucida Sans Unicode', 'Geneva', 'Verdana', sans-serif; - font-size: 14px; - letter-spacing: -0.01em; - line-height: 150%; - text-align: center; - /*background-color: #AFC1C4; */ - background-color: #BFD1D4; - color: black; - padding: 0; - border: 1px solid #aaa; - - margin: 0px 80px 0px 80px; - min-width: 740px; -} - -a { - color: #CA7900; - text-decoration: none; -} - -a:hover { - color: #2491CF; -} - -pre { - font-family: 'Consolas', 'Deja Vu Sans Mono', 'Bitstream Vera Sans Mono', monospace; - font-size: 0.95em; - letter-spacing: 0.015em; - padding: 0.5em; - border: 1px solid #ccc; - background-color: #f8f8f8; -} - -td.linenos pre { - padding: 0.5em 0; - border: 0; - background-color: transparent; - color: #aaa; -} - -table.highlighttable { - margin-left: 0.5em; -} - -table.highlighttable td { - padding: 0 0.5em 0 0.5em; -} - -cite, code, tt { - font-family: 'Consolas', 'Deja Vu Sans Mono', 'Bitstream Vera Sans Mono', monospace; - font-size: 0.95em; - letter-spacing: 0.01em; -} - -hr { - border: 1px solid #abc; - margin: 2em; -} - -tt { - background-color: #f2f2f2; - border-bottom: 1px solid #ddd; - color: #333; -} - -tt.descname { - background-color: transparent; - font-weight: bold; - font-size: 1.2em; - border: 0; -} - -tt.descclassname { - background-color: transparent; - border: 0; -} - -tt.xref { - background-color: transparent; - font-weight: bold; - border: 0; -} - -a tt { - background-color: transparent; - font-weight: bold; - border: 0; - color: #CA7900; -} - -a tt:hover { - color: #2491CF; -} - -dl { - margin-bottom: 15px; -} - -dd p { - margin-top: 0px; -} - -dd ul, dd table { - margin-bottom: 10px; -} - -dd { - margin-top: 3px; - margin-bottom: 10px; - margin-left: 30px; -} - -.refcount { - color: #060; -} - -dt:target, -.highlight { - background-color: #fbe54e; -} - -dl.class, dl.function { - border-top: 2px solid #888; -} - -dl.method, dl.attribute { - border-top: 1px solid #aaa; -} - -dl.glossary dt { - font-weight: bold; - font-size: 1.1em; -} - -pre { - line-height: 120%; -} - -pre a { - color: inherit; - text-decoration: underline; -} - -.first { - margin-top: 0 !important; -} - -div.document { - background-color: white; - text-align: left; - background-image: url(contents.png); - background-repeat: repeat-x; -} - -/* -div.documentwrapper { - width: 100%; -} -*/ - -div.clearer { - clear: both; -} - -div.related h3 { - display: none; -} - -div.related ul { - background-image: url(navigation.png); - height: 2em; - list-style: none; - border-top: 1px solid #ddd; - border-bottom: 1px solid #ddd; - margin: 0; - padding-left: 10px; -} - -div.related ul li { - margin: 0; - padding: 0; - height: 2em; - float: left; -} - -div.related ul li.right { - float: right; - margin-right: 5px; -} - -div.related ul li a { - margin: 0; - padding: 0 5px 0 5px; - line-height: 1.75em; - color: #EE9816; -} - -div.related ul li a:hover { - color: #3CA8E7; -} - -div.body { - margin: 0; - padding: 0.5em 20px 20px 20px; -} - -div.bodywrapper { - margin: 0 280px 0 0; - border-right: 1px solid #ccc; -} - -div.body a { - text-decoration: underline; -} - -div.sphinxsidebar { - margin: 0; - padding: 0.5em 15px 15px 15px; - width: 250px; - float: right; - text-align: left; - background-color: #eee; -/* margin-left: -100%; */ -} - -div.sphinxsidebarl { - margin: 0; - padding: 0.5em 15px 15px 0px; - width: 210px; - float: left; - clear: left; - text-align: left; -/* margin-left: -100%; */ -} - -div.sphinxsidebar h4, div.sphinxsidebar h3, div.sphinxsidebarl h4, div.sphinxsidebarl h3 { - margin: 1em 0 0.5em 0; - font-size: 0.9em; - padding: 0.1em 0 0.1em 0.5em; - color: white; - border: 1px solid #86989B; - background-color: #AFC1C4; -} - -div.sphinxsidebar ul, div.sphinxsidebarl ul { - padding-left: 1.5em; - margin-top: 7px; - list-style: none; - padding: 0; - line-height: 130%; -} - -div.sphinxsidebar ul ul, div.sphinxsidebarl ul ul { - list-style: square; - margin-left: 20px; -} - -p { - margin: 0.8em 0 0.5em 0; -} - -p.rubric { - font-weight: bold; -} - -h1 { - margin: 0; - padding: 0.7em 0 0.3em 0; - font-size: 1.8em; - /*color: #11557C;*/ - color: #000; -} - -h2 { - margin: 1.3em 0 0.2em 0; - font-size: 1.35em; - padding: 0; -} - -h3 { - margin: 1em 0 -0.3em 0; - font-size: 1.2em; -} - -h1 a, h2 a, h3 a, h4 a, h5 a, h6 a { - color: black!important; -} - -h1 a.anchor, h2 a.anchor, h3 a.anchor, h4 a.anchor, h5 a.anchor, h6 a.anchor { - display: none; - margin: 0 0 0 0.3em; - padding: 0 0.2em 0 0.2em; - color: #aaa!important; -} - -h1:hover a.anchor, h2:hover a.anchor, h3:hover a.anchor, h4:hover a.anchor, -h5:hover a.anchor, h6:hover a.anchor { - display: inline; -} - -h1 a.anchor:hover, h2 a.anchor:hover, h3 a.anchor:hover, h4 a.anchor:hover, -h5 a.anchor:hover, h6 a.anchor:hover { - color: #777; - background-color: #eee; -} - -table { - border-collapse: collapse; - margin: 0 -0.5em 0 -0.5em; -} - -table td, table th { - padding: 0.2em 0.5em 0.2em 0.5em; -} - -div.footer { - background-color: #E3EFF1; - color: #86989B; - padding: 3px 8px 3px 0; - clear: both; - font-size: 0.8em; - text-align: right; -} - -div.footer a { - color: #86989B; - text-decoration: underline; -} - -div.pagination { - margin-top: 2em; - padding-top: 0.5em; - border-top: 1px solid black; - text-align: center; -} - -div.sphinxsidebar ul.toc, div.sphinxsidebarl ul.toc { - margin: 1em 0 1em 0; - padding: 0 0 0 0.5em; - list-style: none; -} - -div.sphinxsidebar ul.toc li, div.sphinxsidebarl ul.toc li { - margin: 0.5em 0 0.5em 0; - font-size: 0.9em; - line-height: 130%; -} - -div.sphinxsidebar ul.toc li p, div.sphinxsidebarl ul.toc li p { - margin: 0; - padding: 0; -} - -div.sphinxsidebar ul.toc ul, div.sphinxsidebarl ul.toc ul { - margin: 0.2em 0 0.2em 0; - padding: 0 0 0 1.8em; -} - -div.sphinxsidebar ul.toc ul li, div.sphinxsidebarl ul.toc ul li { - padding: 0; -} - -div.admonition, div.warning { - font-size: 0.9em; - margin: 1em 0 0 0; - border: 1px solid #86989B; - background-color: #f7f7f7; -} - -div.admonition p, div.warning p { - margin: 0.5em 1em 0.5em 1em; - padding: 0; -} - -div.admonition pre, div.warning pre { - margin: 0.4em 1em 0.4em 1em; -} - -div.admonition p.admonition-title, -div.warning p.admonition-title { - margin: 0; - padding: 0.1em 0 0.1em 0.5em; - color: white; - border-bottom: 1px solid #86989B; - font-weight: bold; - background-color: #AFC1C4; -} - -div.warning { - border: 1px solid #940000; -} - -div.warning p.admonition-title { - background-color: #CF0000; - border-bottom-color: #940000; -} - -div.admonition ul, div.admonition ol, -div.warning ul, div.warning ol { - margin: 0.1em 0.5em 0.5em 3em; - padding: 0; -} - -div.versioninfo { - margin: 1em 0 0 0; - border: 1px solid #ccc; - background-color: #DDEAF0; - padding: 8px; - line-height: 1.3em; - font-size: 0.9em; -} - - -a.headerlink { - color: #c60f0f!important; - font-size: 1em; - margin-left: 6px; - padding: 0 4px 0 4px; - text-decoration: none!important; - visibility: hidden; -} - -h1:hover > a.headerlink, -h2:hover > a.headerlink, -h3:hover > a.headerlink, -h4:hover > a.headerlink, -h5:hover > a.headerlink, -h6:hover > a.headerlink, -dt:hover > a.headerlink { - visibility: visible; -} - -a.headerlink:hover { - background-color: #ccc; - color: white!important; -} - -table.indextable td { - text-align: left; - vertical-align: top; -} - -table.indextable dl, table.indextable dd { - margin-top: 0; - margin-bottom: 0; -} - -table.indextable tr.pcap { - height: 10px; -} - -table.indextable tr.cap { - margin-top: 10px; - background-color: #f2f2f2; -} - -img.toggler { - margin-right: 3px; - margin-top: 3px; - cursor: pointer; -} - -img.inheritance { - border: 0px -} - -form.pfform { - margin: 10px 0 20px 0; -} - -table.contentstable { - width: 90%; -} - -table.contentstable p.biglink { - line-height: 150%; -} - -a.biglink { - font-size: 1.3em; -} - -span.linkdescr { - font-style: italic; - padding-top: 5px; - font-size: 90%; -} - -ul.search { - margin: 10px 0 0 20px; - padding: 0; -} - -ul.search li { - padding: 5px 0 5px 20px; - background-image: url(file.png); - background-repeat: no-repeat; - background-position: 0 7px; -} - -ul.search li a { - font-weight: bold; -} - -ul.search li div.context { - color: #888; - margin: 2px 0 0 30px; - text-align: left; -} - -ul.keywordmatches li.goodmatch a { - font-weight: bold; -} - -.twtr-ft { - display: none; -} diff --git a/docs/_static/file.png b/docs/_static/file.png deleted file mode 100644 index d18082e397..0000000000 Binary files a/docs/_static/file.png and /dev/null differ diff --git a/docs/_static/jquery.js b/docs/_static/jquery.js deleted file mode 100644 index 7c24308023..0000000000 --- a/docs/_static/jquery.js +++ /dev/null @@ -1,154 +0,0 @@ -/*! - * jQuery JavaScript Library v1.4.2 - * http://jquery.com/ - * - * Copyright 2010, John Resig - * Dual licensed under the MIT or GPL Version 2 licenses. - * http://jquery.org/license - * - * Includes Sizzle.js - * http://sizzlejs.com/ - * Copyright 2010, The Dojo Foundation - * Released under the MIT, BSD, and GPL Licenses. - * - * Date: Sat Feb 13 22:33:48 2010 -0500 - */ -(function(A,w){function ma(){if(!c.isReady){try{s.documentElement.doScroll("left")}catch(a){setTimeout(ma,1);return}c.ready()}}function Qa(a,b){b.src?c.ajax({url:b.src,async:false,dataType:"script"}):c.globalEval(b.text||b.textContent||b.innerHTML||"");b.parentNode&&b.parentNode.removeChild(b)}function X(a,b,d,f,e,j){var i=a.length;if(typeof b==="object"){for(var o in b)X(a,o,b[o],f,e,d);return a}if(d!==w){f=!j&&f&&c.isFunction(d);for(o=0;o)[^>]*$|^#([\w-]+)$/,Ua=/^.[^:#\[\.,]*$/,Va=/\S/, -Wa=/^(\s|\u00A0)+|(\s|\u00A0)+$/g,Xa=/^<(\w+)\s*\/?>(?:<\/\1>)?$/,P=navigator.userAgent,xa=false,Q=[],L,$=Object.prototype.toString,aa=Object.prototype.hasOwnProperty,ba=Array.prototype.push,R=Array.prototype.slice,ya=Array.prototype.indexOf;c.fn=c.prototype={init:function(a,b){var d,f;if(!a)return this;if(a.nodeType){this.context=this[0]=a;this.length=1;return this}if(a==="body"&&!b){this.context=s;this[0]=s.body;this.selector="body";this.length=1;return this}if(typeof a==="string")if((d=Ta.exec(a))&& -(d[1]||!b))if(d[1]){f=b?b.ownerDocument||b:s;if(a=Xa.exec(a))if(c.isPlainObject(b)){a=[s.createElement(a[1])];c.fn.attr.call(a,b,true)}else a=[f.createElement(a[1])];else{a=sa([d[1]],[f]);a=(a.cacheable?a.fragment.cloneNode(true):a.fragment).childNodes}return c.merge(this,a)}else{if(b=s.getElementById(d[2])){if(b.id!==d[2])return T.find(a);this.length=1;this[0]=b}this.context=s;this.selector=a;return this}else if(!b&&/^\w+$/.test(a)){this.selector=a;this.context=s;a=s.getElementsByTagName(a);return c.merge(this, -a)}else return!b||b.jquery?(b||T).find(a):c(b).find(a);else if(c.isFunction(a))return T.ready(a);if(a.selector!==w){this.selector=a.selector;this.context=a.context}return c.makeArray(a,this)},selector:"",jquery:"1.4.2",length:0,size:function(){return this.length},toArray:function(){return R.call(this,0)},get:function(a){return a==null?this.toArray():a<0?this.slice(a)[0]:this[a]},pushStack:function(a,b,d){var f=c();c.isArray(a)?ba.apply(f,a):c.merge(f,a);f.prevObject=this;f.context=this.context;if(b=== -"find")f.selector=this.selector+(this.selector?" ":"")+d;else if(b)f.selector=this.selector+"."+b+"("+d+")";return f},each:function(a,b){return c.each(this,a,b)},ready:function(a){c.bindReady();if(c.isReady)a.call(s,c);else Q&&Q.push(a);return this},eq:function(a){return a===-1?this.slice(a):this.slice(a,+a+1)},first:function(){return this.eq(0)},last:function(){return this.eq(-1)},slice:function(){return this.pushStack(R.apply(this,arguments),"slice",R.call(arguments).join(","))},map:function(a){return this.pushStack(c.map(this, -function(b,d){return a.call(b,d,b)}))},end:function(){return this.prevObject||c(null)},push:ba,sort:[].sort,splice:[].splice};c.fn.init.prototype=c.fn;c.extend=c.fn.extend=function(){var a=arguments[0]||{},b=1,d=arguments.length,f=false,e,j,i,o;if(typeof a==="boolean"){f=a;a=arguments[1]||{};b=2}if(typeof a!=="object"&&!c.isFunction(a))a={};if(d===b){a=this;--b}for(;b
a"; -var e=d.getElementsByTagName("*"),j=d.getElementsByTagName("a")[0];if(!(!e||!e.length||!j)){c.support={leadingWhitespace:d.firstChild.nodeType===3,tbody:!d.getElementsByTagName("tbody").length,htmlSerialize:!!d.getElementsByTagName("link").length,style:/red/.test(j.getAttribute("style")),hrefNormalized:j.getAttribute("href")==="/a",opacity:/^0.55$/.test(j.style.opacity),cssFloat:!!j.style.cssFloat,checkOn:d.getElementsByTagName("input")[0].value==="on",optSelected:s.createElement("select").appendChild(s.createElement("option")).selected, -parentNode:d.removeChild(d.appendChild(s.createElement("div"))).parentNode===null,deleteExpando:true,checkClone:false,scriptEval:false,noCloneEvent:true,boxModel:null};b.type="text/javascript";try{b.appendChild(s.createTextNode("window."+f+"=1;"))}catch(i){}a.insertBefore(b,a.firstChild);if(A[f]){c.support.scriptEval=true;delete A[f]}try{delete b.test}catch(o){c.support.deleteExpando=false}a.removeChild(b);if(d.attachEvent&&d.fireEvent){d.attachEvent("onclick",function k(){c.support.noCloneEvent= -false;d.detachEvent("onclick",k)});d.cloneNode(true).fireEvent("onclick")}d=s.createElement("div");d.innerHTML="";a=s.createDocumentFragment();a.appendChild(d.firstChild);c.support.checkClone=a.cloneNode(true).cloneNode(true).lastChild.checked;c(function(){var k=s.createElement("div");k.style.width=k.style.paddingLeft="1px";s.body.appendChild(k);c.boxModel=c.support.boxModel=k.offsetWidth===2;s.body.removeChild(k).style.display="none"});a=function(k){var n= -s.createElement("div");k="on"+k;var r=k in n;if(!r){n.setAttribute(k,"return;");r=typeof n[k]==="function"}return r};c.support.submitBubbles=a("submit");c.support.changeBubbles=a("change");a=b=d=e=j=null}})();c.props={"for":"htmlFor","class":"className",readonly:"readOnly",maxlength:"maxLength",cellspacing:"cellSpacing",rowspan:"rowSpan",colspan:"colSpan",tabindex:"tabIndex",usemap:"useMap",frameborder:"frameBorder"};var G="jQuery"+J(),Ya=0,za={};c.extend({cache:{},expando:G,noData:{embed:true,object:true, -applet:true},data:function(a,b,d){if(!(a.nodeName&&c.noData[a.nodeName.toLowerCase()])){a=a==A?za:a;var f=a[G],e=c.cache;if(!f&&typeof b==="string"&&d===w)return null;f||(f=++Ya);if(typeof b==="object"){a[G]=f;e[f]=c.extend(true,{},b)}else if(!e[f]){a[G]=f;e[f]={}}a=e[f];if(d!==w)a[b]=d;return typeof b==="string"?a[b]:a}},removeData:function(a,b){if(!(a.nodeName&&c.noData[a.nodeName.toLowerCase()])){a=a==A?za:a;var d=a[G],f=c.cache,e=f[d];if(b){if(e){delete e[b];c.isEmptyObject(e)&&c.removeData(a)}}else{if(c.support.deleteExpando)delete a[c.expando]; -else a.removeAttribute&&a.removeAttribute(c.expando);delete f[d]}}}});c.fn.extend({data:function(a,b){if(typeof a==="undefined"&&this.length)return c.data(this[0]);else if(typeof a==="object")return this.each(function(){c.data(this,a)});var d=a.split(".");d[1]=d[1]?"."+d[1]:"";if(b===w){var f=this.triggerHandler("getData"+d[1]+"!",[d[0]]);if(f===w&&this.length)f=c.data(this[0],a);return f===w&&d[1]?this.data(d[0]):f}else return this.trigger("setData"+d[1]+"!",[d[0],b]).each(function(){c.data(this, -a,b)})},removeData:function(a){return this.each(function(){c.removeData(this,a)})}});c.extend({queue:function(a,b,d){if(a){b=(b||"fx")+"queue";var f=c.data(a,b);if(!d)return f||[];if(!f||c.isArray(d))f=c.data(a,b,c.makeArray(d));else f.push(d);return f}},dequeue:function(a,b){b=b||"fx";var d=c.queue(a,b),f=d.shift();if(f==="inprogress")f=d.shift();if(f){b==="fx"&&d.unshift("inprogress");f.call(a,function(){c.dequeue(a,b)})}}});c.fn.extend({queue:function(a,b){if(typeof a!=="string"){b=a;a="fx"}if(b=== -w)return c.queue(this[0],a);return this.each(function(){var d=c.queue(this,a,b);a==="fx"&&d[0]!=="inprogress"&&c.dequeue(this,a)})},dequeue:function(a){return this.each(function(){c.dequeue(this,a)})},delay:function(a,b){a=c.fx?c.fx.speeds[a]||a:a;b=b||"fx";return this.queue(b,function(){var d=this;setTimeout(function(){c.dequeue(d,b)},a)})},clearQueue:function(a){return this.queue(a||"fx",[])}});var Aa=/[\n\t]/g,ca=/\s+/,Za=/\r/g,$a=/href|src|style/,ab=/(button|input)/i,bb=/(button|input|object|select|textarea)/i, -cb=/^(a|area)$/i,Ba=/radio|checkbox/;c.fn.extend({attr:function(a,b){return X(this,a,b,true,c.attr)},removeAttr:function(a){return this.each(function(){c.attr(this,a,"");this.nodeType===1&&this.removeAttribute(a)})},addClass:function(a){if(c.isFunction(a))return this.each(function(n){var r=c(this);r.addClass(a.call(this,n,r.attr("class")))});if(a&&typeof a==="string")for(var b=(a||"").split(ca),d=0,f=this.length;d-1)return true;return false},val:function(a){if(a===w){var b=this[0];if(b){if(c.nodeName(b,"option"))return(b.attributes.value||{}).specified?b.value:b.text;if(c.nodeName(b,"select")){var d=b.selectedIndex,f=[],e=b.options;b=b.type==="select-one";if(d<0)return null;var j=b?d:0;for(d=b?d+1:e.length;j=0;else if(c.nodeName(this,"select")){var u=c.makeArray(r);c("option",this).each(function(){this.selected= -c.inArray(c(this).val(),u)>=0});if(!u.length)this.selectedIndex=-1}else this.value=r}})}});c.extend({attrFn:{val:true,css:true,html:true,text:true,data:true,width:true,height:true,offset:true},attr:function(a,b,d,f){if(!a||a.nodeType===3||a.nodeType===8)return w;if(f&&b in c.attrFn)return c(a)[b](d);f=a.nodeType!==1||!c.isXMLDoc(a);var e=d!==w;b=f&&c.props[b]||b;if(a.nodeType===1){var j=$a.test(b);if(b in a&&f&&!j){if(e){b==="type"&&ab.test(a.nodeName)&&a.parentNode&&c.error("type property can't be changed"); -a[b]=d}if(c.nodeName(a,"form")&&a.getAttributeNode(b))return a.getAttributeNode(b).nodeValue;if(b==="tabIndex")return(b=a.getAttributeNode("tabIndex"))&&b.specified?b.value:bb.test(a.nodeName)||cb.test(a.nodeName)&&a.href?0:w;return a[b]}if(!c.support.style&&f&&b==="style"){if(e)a.style.cssText=""+d;return a.style.cssText}e&&a.setAttribute(b,""+d);a=!c.support.hrefNormalized&&f&&j?a.getAttribute(b,2):a.getAttribute(b);return a===null?w:a}return c.style(a,b,d)}});var O=/\.(.*)$/,db=function(a){return a.replace(/[^\w\s\.\|`]/g, -function(b){return"\\"+b})};c.event={add:function(a,b,d,f){if(!(a.nodeType===3||a.nodeType===8)){if(a.setInterval&&a!==A&&!a.frameElement)a=A;var e,j;if(d.handler){e=d;d=e.handler}if(!d.guid)d.guid=c.guid++;if(j=c.data(a)){var i=j.events=j.events||{},o=j.handle;if(!o)j.handle=o=function(){return typeof c!=="undefined"&&!c.event.triggered?c.event.handle.apply(o.elem,arguments):w};o.elem=a;b=b.split(" ");for(var k,n=0,r;k=b[n++];){j=e?c.extend({},e):{handler:d,data:f};if(k.indexOf(".")>-1){r=k.split("."); -k=r.shift();j.namespace=r.slice(0).sort().join(".")}else{r=[];j.namespace=""}j.type=k;j.guid=d.guid;var u=i[k],z=c.event.special[k]||{};if(!u){u=i[k]=[];if(!z.setup||z.setup.call(a,f,r,o)===false)if(a.addEventListener)a.addEventListener(k,o,false);else a.attachEvent&&a.attachEvent("on"+k,o)}if(z.add){z.add.call(a,j);if(!j.handler.guid)j.handler.guid=d.guid}u.push(j);c.event.global[k]=true}a=null}}},global:{},remove:function(a,b,d,f){if(!(a.nodeType===3||a.nodeType===8)){var e,j=0,i,o,k,n,r,u,z=c.data(a), -C=z&&z.events;if(z&&C){if(b&&b.type){d=b.handler;b=b.type}if(!b||typeof b==="string"&&b.charAt(0)==="."){b=b||"";for(e in C)c.event.remove(a,e+b)}else{for(b=b.split(" ");e=b[j++];){n=e;i=e.indexOf(".")<0;o=[];if(!i){o=e.split(".");e=o.shift();k=new RegExp("(^|\\.)"+c.map(o.slice(0).sort(),db).join("\\.(?:.*\\.)?")+"(\\.|$)")}if(r=C[e])if(d){n=c.event.special[e]||{};for(B=f||0;B=0){a.type= -e=e.slice(0,-1);a.exclusive=true}if(!d){a.stopPropagation();c.event.global[e]&&c.each(c.cache,function(){this.events&&this.events[e]&&c.event.trigger(a,b,this.handle.elem)})}if(!d||d.nodeType===3||d.nodeType===8)return w;a.result=w;a.target=d;b=c.makeArray(b);b.unshift(a)}a.currentTarget=d;(f=c.data(d,"handle"))&&f.apply(d,b);f=d.parentNode||d.ownerDocument;try{if(!(d&&d.nodeName&&c.noData[d.nodeName.toLowerCase()]))if(d["on"+e]&&d["on"+e].apply(d,b)===false)a.result=false}catch(j){}if(!a.isPropagationStopped()&& -f)c.event.trigger(a,b,f,true);else if(!a.isDefaultPrevented()){f=a.target;var i,o=c.nodeName(f,"a")&&e==="click",k=c.event.special[e]||{};if((!k._default||k._default.call(d,a)===false)&&!o&&!(f&&f.nodeName&&c.noData[f.nodeName.toLowerCase()])){try{if(f[e]){if(i=f["on"+e])f["on"+e]=null;c.event.triggered=true;f[e]()}}catch(n){}if(i)f["on"+e]=i;c.event.triggered=false}}},handle:function(a){var b,d,f,e;a=arguments[0]=c.event.fix(a||A.event);a.currentTarget=this;b=a.type.indexOf(".")<0&&!a.exclusive; -if(!b){d=a.type.split(".");a.type=d.shift();f=new RegExp("(^|\\.)"+d.slice(0).sort().join("\\.(?:.*\\.)?")+"(\\.|$)")}e=c.data(this,"events");d=e[a.type];if(e&&d){d=d.slice(0);e=0;for(var j=d.length;e-1?c.map(a.options,function(f){return f.selected}).join("-"):"";else if(a.nodeName.toLowerCase()==="select")d=a.selectedIndex;return d},fa=function(a,b){var d=a.target,f,e;if(!(!da.test(d.nodeName)||d.readOnly)){f=c.data(d,"_change_data");e=Fa(d);if(a.type!=="focusout"||d.type!=="radio")c.data(d,"_change_data", -e);if(!(f===w||e===f))if(f!=null||e){a.type="change";return c.event.trigger(a,b,d)}}};c.event.special.change={filters:{focusout:fa,click:function(a){var b=a.target,d=b.type;if(d==="radio"||d==="checkbox"||b.nodeName.toLowerCase()==="select")return fa.call(this,a)},keydown:function(a){var b=a.target,d=b.type;if(a.keyCode===13&&b.nodeName.toLowerCase()!=="textarea"||a.keyCode===32&&(d==="checkbox"||d==="radio")||d==="select-multiple")return fa.call(this,a)},beforeactivate:function(a){a=a.target;c.data(a, -"_change_data",Fa(a))}},setup:function(){if(this.type==="file")return false;for(var a in ea)c.event.add(this,a+".specialChange",ea[a]);return da.test(this.nodeName)},teardown:function(){c.event.remove(this,".specialChange");return da.test(this.nodeName)}};ea=c.event.special.change.filters}s.addEventListener&&c.each({focus:"focusin",blur:"focusout"},function(a,b){function d(f){f=c.event.fix(f);f.type=b;return c.event.handle.call(this,f)}c.event.special[b]={setup:function(){this.addEventListener(a, -d,true)},teardown:function(){this.removeEventListener(a,d,true)}}});c.each(["bind","one"],function(a,b){c.fn[b]=function(d,f,e){if(typeof d==="object"){for(var j in d)this[b](j,f,d[j],e);return this}if(c.isFunction(f)){e=f;f=w}var i=b==="one"?c.proxy(e,function(k){c(this).unbind(k,i);return e.apply(this,arguments)}):e;if(d==="unload"&&b!=="one")this.one(d,f,e);else{j=0;for(var o=this.length;j0){y=t;break}}t=t[g]}m[q]=y}}}var f=/((?:\((?:\([^()]+\)|[^()]+)+\)|\[(?:\[[^[\]]*\]|['"][^'"]*['"]|[^[\]'"]+)+\]|\\.|[^ >+~,(\[\\]+)+|[>+~])(\s*,\s*)?((?:.|\r|\n)*)/g, -e=0,j=Object.prototype.toString,i=false,o=true;[0,0].sort(function(){o=false;return 0});var k=function(g,h,l,m){l=l||[];var q=h=h||s;if(h.nodeType!==1&&h.nodeType!==9)return[];if(!g||typeof g!=="string")return l;for(var p=[],v,t,y,S,H=true,M=x(h),I=g;(f.exec(""),v=f.exec(I))!==null;){I=v[3];p.push(v[1]);if(v[2]){S=v[3];break}}if(p.length>1&&r.exec(g))if(p.length===2&&n.relative[p[0]])t=ga(p[0]+p[1],h);else for(t=n.relative[p[0]]?[h]:k(p.shift(),h);p.length;){g=p.shift();if(n.relative[g])g+=p.shift(); -t=ga(g,t)}else{if(!m&&p.length>1&&h.nodeType===9&&!M&&n.match.ID.test(p[0])&&!n.match.ID.test(p[p.length-1])){v=k.find(p.shift(),h,M);h=v.expr?k.filter(v.expr,v.set)[0]:v.set[0]}if(h){v=m?{expr:p.pop(),set:z(m)}:k.find(p.pop(),p.length===1&&(p[0]==="~"||p[0]==="+")&&h.parentNode?h.parentNode:h,M);t=v.expr?k.filter(v.expr,v.set):v.set;if(p.length>0)y=z(t);else H=false;for(;p.length;){var D=p.pop();v=D;if(n.relative[D])v=p.pop();else D="";if(v==null)v=h;n.relative[D](y,v,M)}}else y=[]}y||(y=t);y||k.error(D|| -g);if(j.call(y)==="[object Array]")if(H)if(h&&h.nodeType===1)for(g=0;y[g]!=null;g++){if(y[g]&&(y[g]===true||y[g].nodeType===1&&E(h,y[g])))l.push(t[g])}else for(g=0;y[g]!=null;g++)y[g]&&y[g].nodeType===1&&l.push(t[g]);else l.push.apply(l,y);else z(y,l);if(S){k(S,q,l,m);k.uniqueSort(l)}return l};k.uniqueSort=function(g){if(B){i=o;g.sort(B);if(i)for(var h=1;h":function(g,h){var l=typeof h==="string";if(l&&!/\W/.test(h)){h=h.toLowerCase();for(var m=0,q=g.length;m=0))l||m.push(v);else if(l)h[p]=false;return false},ID:function(g){return g[1].replace(/\\/g,"")},TAG:function(g){return g[1].toLowerCase()}, -CHILD:function(g){if(g[1]==="nth"){var h=/(-?)(\d*)n((?:\+|-)?\d*)/.exec(g[2]==="even"&&"2n"||g[2]==="odd"&&"2n+1"||!/\D/.test(g[2])&&"0n+"+g[2]||g[2]);g[2]=h[1]+(h[2]||1)-0;g[3]=h[3]-0}g[0]=e++;return g},ATTR:function(g,h,l,m,q,p){h=g[1].replace(/\\/g,"");if(!p&&n.attrMap[h])g[1]=n.attrMap[h];if(g[2]==="~=")g[4]=" "+g[4]+" ";return g},PSEUDO:function(g,h,l,m,q){if(g[1]==="not")if((f.exec(g[3])||"").length>1||/^\w/.test(g[3]))g[3]=k(g[3],null,null,h);else{g=k.filter(g[3],h,l,true^q);l||m.push.apply(m, -g);return false}else if(n.match.POS.test(g[0])||n.match.CHILD.test(g[0]))return true;return g},POS:function(g){g.unshift(true);return g}},filters:{enabled:function(g){return g.disabled===false&&g.type!=="hidden"},disabled:function(g){return g.disabled===true},checked:function(g){return g.checked===true},selected:function(g){return g.selected===true},parent:function(g){return!!g.firstChild},empty:function(g){return!g.firstChild},has:function(g,h,l){return!!k(l[3],g).length},header:function(g){return/h\d/i.test(g.nodeName)}, -text:function(g){return"text"===g.type},radio:function(g){return"radio"===g.type},checkbox:function(g){return"checkbox"===g.type},file:function(g){return"file"===g.type},password:function(g){return"password"===g.type},submit:function(g){return"submit"===g.type},image:function(g){return"image"===g.type},reset:function(g){return"reset"===g.type},button:function(g){return"button"===g.type||g.nodeName.toLowerCase()==="button"},input:function(g){return/input|select|textarea|button/i.test(g.nodeName)}}, -setFilters:{first:function(g,h){return h===0},last:function(g,h,l,m){return h===m.length-1},even:function(g,h){return h%2===0},odd:function(g,h){return h%2===1},lt:function(g,h,l){return hl[3]-0},nth:function(g,h,l){return l[3]-0===h},eq:function(g,h,l){return l[3]-0===h}},filter:{PSEUDO:function(g,h,l,m){var q=h[1],p=n.filters[q];if(p)return p(g,l,h,m);else if(q==="contains")return(g.textContent||g.innerText||a([g])||"").indexOf(h[3])>=0;else if(q==="not"){h= -h[3];l=0;for(m=h.length;l=0}},ID:function(g,h){return g.nodeType===1&&g.getAttribute("id")===h},TAG:function(g,h){return h==="*"&&g.nodeType===1||g.nodeName.toLowerCase()===h},CLASS:function(g,h){return(" "+(g.className||g.getAttribute("class"))+" ").indexOf(h)>-1},ATTR:function(g,h){var l=h[1];g=n.attrHandle[l]?n.attrHandle[l](g):g[l]!=null?g[l]:g.getAttribute(l);l=g+"";var m=h[2];h=h[4];return g==null?m==="!=":m=== -"="?l===h:m==="*="?l.indexOf(h)>=0:m==="~="?(" "+l+" ").indexOf(h)>=0:!h?l&&g!==false:m==="!="?l!==h:m==="^="?l.indexOf(h)===0:m==="$="?l.substr(l.length-h.length)===h:m==="|="?l===h||l.substr(0,h.length+1)===h+"-":false},POS:function(g,h,l,m){var q=n.setFilters[h[2]];if(q)return q(g,l,h,m)}}},r=n.match.POS;for(var u in n.match){n.match[u]=new RegExp(n.match[u].source+/(?![^\[]*\])(?![^\(]*\))/.source);n.leftMatch[u]=new RegExp(/(^(?:.|\r|\n)*?)/.source+n.match[u].source.replace(/\\(\d+)/g,function(g, -h){return"\\"+(h-0+1)}))}var z=function(g,h){g=Array.prototype.slice.call(g,0);if(h){h.push.apply(h,g);return h}return g};try{Array.prototype.slice.call(s.documentElement.childNodes,0)}catch(C){z=function(g,h){h=h||[];if(j.call(g)==="[object Array]")Array.prototype.push.apply(h,g);else if(typeof g.length==="number")for(var l=0,m=g.length;l";var l=s.documentElement;l.insertBefore(g,l.firstChild);if(s.getElementById(h)){n.find.ID=function(m,q,p){if(typeof q.getElementById!=="undefined"&&!p)return(q=q.getElementById(m[1]))?q.id===m[1]||typeof q.getAttributeNode!=="undefined"&& -q.getAttributeNode("id").nodeValue===m[1]?[q]:w:[]};n.filter.ID=function(m,q){var p=typeof m.getAttributeNode!=="undefined"&&m.getAttributeNode("id");return m.nodeType===1&&p&&p.nodeValue===q}}l.removeChild(g);l=g=null})();(function(){var g=s.createElement("div");g.appendChild(s.createComment(""));if(g.getElementsByTagName("*").length>0)n.find.TAG=function(h,l){l=l.getElementsByTagName(h[1]);if(h[1]==="*"){h=[];for(var m=0;l[m];m++)l[m].nodeType===1&&h.push(l[m]);l=h}return l};g.innerHTML=""; -if(g.firstChild&&typeof g.firstChild.getAttribute!=="undefined"&&g.firstChild.getAttribute("href")!=="#")n.attrHandle.href=function(h){return h.getAttribute("href",2)};g=null})();s.querySelectorAll&&function(){var g=k,h=s.createElement("div");h.innerHTML="

";if(!(h.querySelectorAll&&h.querySelectorAll(".TEST").length===0)){k=function(m,q,p,v){q=q||s;if(!v&&q.nodeType===9&&!x(q))try{return z(q.querySelectorAll(m),p)}catch(t){}return g(m,q,p,v)};for(var l in g)k[l]=g[l];h=null}}(); -(function(){var g=s.createElement("div");g.innerHTML="
";if(!(!g.getElementsByClassName||g.getElementsByClassName("e").length===0)){g.lastChild.className="e";if(g.getElementsByClassName("e").length!==1){n.order.splice(1,0,"CLASS");n.find.CLASS=function(h,l,m){if(typeof l.getElementsByClassName!=="undefined"&&!m)return l.getElementsByClassName(h[1])};g=null}}})();var E=s.compareDocumentPosition?function(g,h){return!!(g.compareDocumentPosition(h)&16)}: -function(g,h){return g!==h&&(g.contains?g.contains(h):true)},x=function(g){return(g=(g?g.ownerDocument||g:0).documentElement)?g.nodeName!=="HTML":false},ga=function(g,h){var l=[],m="",q;for(h=h.nodeType?[h]:h;q=n.match.PSEUDO.exec(g);){m+=q[0];g=g.replace(n.match.PSEUDO,"")}g=n.relative[g]?g+"*":g;q=0;for(var p=h.length;q=0===d})};c.fn.extend({find:function(a){for(var b=this.pushStack("","find",a),d=0,f=0,e=this.length;f0)for(var j=d;j0},closest:function(a,b){if(c.isArray(a)){var d=[],f=this[0],e,j= -{},i;if(f&&a.length){e=0;for(var o=a.length;e-1:c(f).is(e)){d.push({selector:i,elem:f});delete j[i]}}f=f.parentNode}}return d}var k=c.expr.match.POS.test(a)?c(a,b||this.context):null;return this.map(function(n,r){for(;r&&r.ownerDocument&&r!==b;){if(k?k.index(r)>-1:c(r).is(a))return r;r=r.parentNode}return null})},index:function(a){if(!a||typeof a=== -"string")return c.inArray(this[0],a?c(a):this.parent().children());return c.inArray(a.jquery?a[0]:a,this)},add:function(a,b){a=typeof a==="string"?c(a,b||this.context):c.makeArray(a);b=c.merge(this.get(),a);return this.pushStack(qa(a[0])||qa(b[0])?b:c.unique(b))},andSelf:function(){return this.add(this.prevObject)}});c.each({parent:function(a){return(a=a.parentNode)&&a.nodeType!==11?a:null},parents:function(a){return c.dir(a,"parentNode")},parentsUntil:function(a,b,d){return c.dir(a,"parentNode", -d)},next:function(a){return c.nth(a,2,"nextSibling")},prev:function(a){return c.nth(a,2,"previousSibling")},nextAll:function(a){return c.dir(a,"nextSibling")},prevAll:function(a){return c.dir(a,"previousSibling")},nextUntil:function(a,b,d){return c.dir(a,"nextSibling",d)},prevUntil:function(a,b,d){return c.dir(a,"previousSibling",d)},siblings:function(a){return c.sibling(a.parentNode.firstChild,a)},children:function(a){return c.sibling(a.firstChild)},contents:function(a){return c.nodeName(a,"iframe")? -a.contentDocument||a.contentWindow.document:c.makeArray(a.childNodes)}},function(a,b){c.fn[a]=function(d,f){var e=c.map(this,b,d);eb.test(a)||(f=d);if(f&&typeof f==="string")e=c.filter(f,e);e=this.length>1?c.unique(e):e;if((this.length>1||gb.test(f))&&fb.test(a))e=e.reverse();return this.pushStack(e,a,R.call(arguments).join(","))}});c.extend({filter:function(a,b,d){if(d)a=":not("+a+")";return c.find.matches(a,b)},dir:function(a,b,d){var f=[];for(a=a[b];a&&a.nodeType!==9&&(d===w||a.nodeType!==1||!c(a).is(d));){a.nodeType=== -1&&f.push(a);a=a[b]}return f},nth:function(a,b,d){b=b||1;for(var f=0;a;a=a[d])if(a.nodeType===1&&++f===b)break;return a},sibling:function(a,b){for(var d=[];a;a=a.nextSibling)a.nodeType===1&&a!==b&&d.push(a);return d}});var Ja=/ jQuery\d+="(?:\d+|null)"/g,V=/^\s+/,Ka=/(<([\w:]+)[^>]*?)\/>/g,hb=/^(?:area|br|col|embed|hr|img|input|link|meta|param)$/i,La=/<([\w:]+)/,ib=/"},F={option:[1,""],legend:[1,"
","
"],thead:[1,"","
"],tr:[2,"","
"],td:[3,"","
"],col:[2,"","
"],area:[1,"",""],_default:[0,"",""]};F.optgroup=F.option;F.tbody=F.tfoot=F.colgroup=F.caption=F.thead;F.th=F.td;if(!c.support.htmlSerialize)F._default=[1,"div
","
"];c.fn.extend({text:function(a){if(c.isFunction(a))return this.each(function(b){var d= -c(this);d.text(a.call(this,b,d.text()))});if(typeof a!=="object"&&a!==w)return this.empty().append((this[0]&&this[0].ownerDocument||s).createTextNode(a));return c.text(this)},wrapAll:function(a){if(c.isFunction(a))return this.each(function(d){c(this).wrapAll(a.call(this,d))});if(this[0]){var b=c(a,this[0].ownerDocument).eq(0).clone(true);this[0].parentNode&&b.insertBefore(this[0]);b.map(function(){for(var d=this;d.firstChild&&d.firstChild.nodeType===1;)d=d.firstChild;return d}).append(this)}return this}, -wrapInner:function(a){if(c.isFunction(a))return this.each(function(b){c(this).wrapInner(a.call(this,b))});return this.each(function(){var b=c(this),d=b.contents();d.length?d.wrapAll(a):b.append(a)})},wrap:function(a){return this.each(function(){c(this).wrapAll(a)})},unwrap:function(){return this.parent().each(function(){c.nodeName(this,"body")||c(this).replaceWith(this.childNodes)}).end()},append:function(){return this.domManip(arguments,true,function(a){this.nodeType===1&&this.appendChild(a)})}, -prepend:function(){return this.domManip(arguments,true,function(a){this.nodeType===1&&this.insertBefore(a,this.firstChild)})},before:function(){if(this[0]&&this[0].parentNode)return this.domManip(arguments,false,function(b){this.parentNode.insertBefore(b,this)});else if(arguments.length){var a=c(arguments[0]);a.push.apply(a,this.toArray());return this.pushStack(a,"before",arguments)}},after:function(){if(this[0]&&this[0].parentNode)return this.domManip(arguments,false,function(b){this.parentNode.insertBefore(b, -this.nextSibling)});else if(arguments.length){var a=this.pushStack(this,"after",arguments);a.push.apply(a,c(arguments[0]).toArray());return a}},remove:function(a,b){for(var d=0,f;(f=this[d])!=null;d++)if(!a||c.filter(a,[f]).length){if(!b&&f.nodeType===1){c.cleanData(f.getElementsByTagName("*"));c.cleanData([f])}f.parentNode&&f.parentNode.removeChild(f)}return this},empty:function(){for(var a=0,b;(b=this[a])!=null;a++)for(b.nodeType===1&&c.cleanData(b.getElementsByTagName("*"));b.firstChild;)b.removeChild(b.firstChild); -return this},clone:function(a){var b=this.map(function(){if(!c.support.noCloneEvent&&!c.isXMLDoc(this)){var d=this.outerHTML,f=this.ownerDocument;if(!d){d=f.createElement("div");d.appendChild(this.cloneNode(true));d=d.innerHTML}return c.clean([d.replace(Ja,"").replace(/=([^="'>\s]+\/)>/g,'="$1">').replace(V,"")],f)[0]}else return this.cloneNode(true)});if(a===true){ra(this,b);ra(this.find("*"),b.find("*"))}return b},html:function(a){if(a===w)return this[0]&&this[0].nodeType===1?this[0].innerHTML.replace(Ja, -""):null;else if(typeof a==="string"&&!ta.test(a)&&(c.support.leadingWhitespace||!V.test(a))&&!F[(La.exec(a)||["",""])[1].toLowerCase()]){a=a.replace(Ka,Ma);try{for(var b=0,d=this.length;b0||e.cacheable||this.length>1?k.cloneNode(true):k)}o.length&&c.each(o,Qa)}return this}});c.fragments={};c.each({appendTo:"append",prependTo:"prepend",insertBefore:"before",insertAfter:"after",replaceAll:"replaceWith"},function(a,b){c.fn[a]=function(d){var f=[];d=c(d);var e=this.length===1&&this[0].parentNode;if(e&&e.nodeType===11&&e.childNodes.length===1&&d.length===1){d[b](this[0]); -return this}else{e=0;for(var j=d.length;e0?this.clone(true):this).get();c.fn[b].apply(c(d[e]),i);f=f.concat(i)}return this.pushStack(f,a,d.selector)}}});c.extend({clean:function(a,b,d,f){b=b||s;if(typeof b.createElement==="undefined")b=b.ownerDocument||b[0]&&b[0].ownerDocument||s;for(var e=[],j=0,i;(i=a[j])!=null;j++){if(typeof i==="number")i+="";if(i){if(typeof i==="string"&&!jb.test(i))i=b.createTextNode(i);else if(typeof i==="string"){i=i.replace(Ka,Ma);var o=(La.exec(i)||["", -""])[1].toLowerCase(),k=F[o]||F._default,n=k[0],r=b.createElement("div");for(r.innerHTML=k[1]+i+k[2];n--;)r=r.lastChild;if(!c.support.tbody){n=ib.test(i);o=o==="table"&&!n?r.firstChild&&r.firstChild.childNodes:k[1]===""&&!n?r.childNodes:[];for(k=o.length-1;k>=0;--k)c.nodeName(o[k],"tbody")&&!o[k].childNodes.length&&o[k].parentNode.removeChild(o[k])}!c.support.leadingWhitespace&&V.test(i)&&r.insertBefore(b.createTextNode(V.exec(i)[0]),r.firstChild);i=r.childNodes}if(i.nodeType)e.push(i);else e= -c.merge(e,i)}}if(d)for(j=0;e[j];j++)if(f&&c.nodeName(e[j],"script")&&(!e[j].type||e[j].type.toLowerCase()==="text/javascript"))f.push(e[j].parentNode?e[j].parentNode.removeChild(e[j]):e[j]);else{e[j].nodeType===1&&e.splice.apply(e,[j+1,0].concat(c.makeArray(e[j].getElementsByTagName("script"))));d.appendChild(e[j])}return e},cleanData:function(a){for(var b,d,f=c.cache,e=c.event.special,j=c.support.deleteExpando,i=0,o;(o=a[i])!=null;i++)if(d=o[c.expando]){b=f[d];if(b.events)for(var k in b.events)e[k]? -c.event.remove(o,k):Ca(o,k,b.handle);if(j)delete o[c.expando];else o.removeAttribute&&o.removeAttribute(c.expando);delete f[d]}}});var kb=/z-?index|font-?weight|opacity|zoom|line-?height/i,Na=/alpha\([^)]*\)/,Oa=/opacity=([^)]*)/,ha=/float/i,ia=/-([a-z])/ig,lb=/([A-Z])/g,mb=/^-?\d+(?:px)?$/i,nb=/^-?\d/,ob={position:"absolute",visibility:"hidden",display:"block"},pb=["Left","Right"],qb=["Top","Bottom"],rb=s.defaultView&&s.defaultView.getComputedStyle,Pa=c.support.cssFloat?"cssFloat":"styleFloat",ja= -function(a,b){return b.toUpperCase()};c.fn.css=function(a,b){return X(this,a,b,true,function(d,f,e){if(e===w)return c.curCSS(d,f);if(typeof e==="number"&&!kb.test(f))e+="px";c.style(d,f,e)})};c.extend({style:function(a,b,d){if(!a||a.nodeType===3||a.nodeType===8)return w;if((b==="width"||b==="height")&&parseFloat(d)<0)d=w;var f=a.style||a,e=d!==w;if(!c.support.opacity&&b==="opacity"){if(e){f.zoom=1;b=parseInt(d,10)+""==="NaN"?"":"alpha(opacity="+d*100+")";a=f.filter||c.curCSS(a,"filter")||"";f.filter= -Na.test(a)?a.replace(Na,b):b}return f.filter&&f.filter.indexOf("opacity=")>=0?parseFloat(Oa.exec(f.filter)[1])/100+"":""}if(ha.test(b))b=Pa;b=b.replace(ia,ja);if(e)f[b]=d;return f[b]},css:function(a,b,d,f){if(b==="width"||b==="height"){var e,j=b==="width"?pb:qb;function i(){e=b==="width"?a.offsetWidth:a.offsetHeight;f!=="border"&&c.each(j,function(){f||(e-=parseFloat(c.curCSS(a,"padding"+this,true))||0);if(f==="margin")e+=parseFloat(c.curCSS(a,"margin"+this,true))||0;else e-=parseFloat(c.curCSS(a, -"border"+this+"Width",true))||0})}a.offsetWidth!==0?i():c.swap(a,ob,i);return Math.max(0,Math.round(e))}return c.curCSS(a,b,d)},curCSS:function(a,b,d){var f,e=a.style;if(!c.support.opacity&&b==="opacity"&&a.currentStyle){f=Oa.test(a.currentStyle.filter||"")?parseFloat(RegExp.$1)/100+"":"";return f===""?"1":f}if(ha.test(b))b=Pa;if(!d&&e&&e[b])f=e[b];else if(rb){if(ha.test(b))b="float";b=b.replace(lb,"-$1").toLowerCase();e=a.ownerDocument.defaultView;if(!e)return null;if(a=e.getComputedStyle(a,null))f= -a.getPropertyValue(b);if(b==="opacity"&&f==="")f="1"}else if(a.currentStyle){d=b.replace(ia,ja);f=a.currentStyle[b]||a.currentStyle[d];if(!mb.test(f)&&nb.test(f)){b=e.left;var j=a.runtimeStyle.left;a.runtimeStyle.left=a.currentStyle.left;e.left=d==="fontSize"?"1em":f||0;f=e.pixelLeft+"px";e.left=b;a.runtimeStyle.left=j}}return f},swap:function(a,b,d){var f={};for(var e in b){f[e]=a.style[e];a.style[e]=b[e]}d.call(a);for(e in b)a.style[e]=f[e]}});if(c.expr&&c.expr.filters){c.expr.filters.hidden=function(a){var b= -a.offsetWidth,d=a.offsetHeight,f=a.nodeName.toLowerCase()==="tr";return b===0&&d===0&&!f?true:b>0&&d>0&&!f?false:c.curCSS(a,"display")==="none"};c.expr.filters.visible=function(a){return!c.expr.filters.hidden(a)}}var sb=J(),tb=//gi,ub=/select|textarea/i,vb=/color|date|datetime|email|hidden|month|number|password|range|search|tel|text|time|url|week/i,N=/=\?(&|$)/,ka=/\?/,wb=/(\?|&)_=.*?(&|$)/,xb=/^(\w+:)?\/\/([^\/?#]+)/,yb=/%20/g,zb=c.fn.load;c.fn.extend({load:function(a,b,d){if(typeof a!== -"string")return zb.call(this,a);else if(!this.length)return this;var f=a.indexOf(" ");if(f>=0){var e=a.slice(f,a.length);a=a.slice(0,f)}f="GET";if(b)if(c.isFunction(b)){d=b;b=null}else if(typeof b==="object"){b=c.param(b,c.ajaxSettings.traditional);f="POST"}var j=this;c.ajax({url:a,type:f,dataType:"html",data:b,complete:function(i,o){if(o==="success"||o==="notmodified")j.html(e?c("
").append(i.responseText.replace(tb,"")).find(e):i.responseText);d&&j.each(d,[i.responseText,o,i])}});return this}, -serialize:function(){return c.param(this.serializeArray())},serializeArray:function(){return this.map(function(){return this.elements?c.makeArray(this.elements):this}).filter(function(){return this.name&&!this.disabled&&(this.checked||ub.test(this.nodeName)||vb.test(this.type))}).map(function(a,b){a=c(this).val();return a==null?null:c.isArray(a)?c.map(a,function(d){return{name:b.name,value:d}}):{name:b.name,value:a}}).get()}});c.each("ajaxStart ajaxStop ajaxComplete ajaxError ajaxSuccess ajaxSend".split(" "), -function(a,b){c.fn[b]=function(d){return this.bind(b,d)}});c.extend({get:function(a,b,d,f){if(c.isFunction(b)){f=f||d;d=b;b=null}return c.ajax({type:"GET",url:a,data:b,success:d,dataType:f})},getScript:function(a,b){return c.get(a,null,b,"script")},getJSON:function(a,b,d){return c.get(a,b,d,"json")},post:function(a,b,d,f){if(c.isFunction(b)){f=f||d;d=b;b={}}return c.ajax({type:"POST",url:a,data:b,success:d,dataType:f})},ajaxSetup:function(a){c.extend(c.ajaxSettings,a)},ajaxSettings:{url:location.href, -global:true,type:"GET",contentType:"application/x-www-form-urlencoded",processData:true,async:true,xhr:A.XMLHttpRequest&&(A.location.protocol!=="file:"||!A.ActiveXObject)?function(){return new A.XMLHttpRequest}:function(){try{return new A.ActiveXObject("Microsoft.XMLHTTP")}catch(a){}},accepts:{xml:"application/xml, text/xml",html:"text/html",script:"text/javascript, application/javascript",json:"application/json, text/javascript",text:"text/plain",_default:"*/*"}},lastModified:{},etag:{},ajax:function(a){function b(){e.success&& -e.success.call(k,o,i,x);e.global&&f("ajaxSuccess",[x,e])}function d(){e.complete&&e.complete.call(k,x,i);e.global&&f("ajaxComplete",[x,e]);e.global&&!--c.active&&c.event.trigger("ajaxStop")}function f(q,p){(e.context?c(e.context):c.event).trigger(q,p)}var e=c.extend(true,{},c.ajaxSettings,a),j,i,o,k=a&&a.context||e,n=e.type.toUpperCase();if(e.data&&e.processData&&typeof e.data!=="string")e.data=c.param(e.data,e.traditional);if(e.dataType==="jsonp"){if(n==="GET")N.test(e.url)||(e.url+=(ka.test(e.url)? -"&":"?")+(e.jsonp||"callback")+"=?");else if(!e.data||!N.test(e.data))e.data=(e.data?e.data+"&":"")+(e.jsonp||"callback")+"=?";e.dataType="json"}if(e.dataType==="json"&&(e.data&&N.test(e.data)||N.test(e.url))){j=e.jsonpCallback||"jsonp"+sb++;if(e.data)e.data=(e.data+"").replace(N,"="+j+"$1");e.url=e.url.replace(N,"="+j+"$1");e.dataType="script";A[j]=A[j]||function(q){o=q;b();d();A[j]=w;try{delete A[j]}catch(p){}z&&z.removeChild(C)}}if(e.dataType==="script"&&e.cache===null)e.cache=false;if(e.cache=== -false&&n==="GET"){var r=J(),u=e.url.replace(wb,"$1_="+r+"$2");e.url=u+(u===e.url?(ka.test(e.url)?"&":"?")+"_="+r:"")}if(e.data&&n==="GET")e.url+=(ka.test(e.url)?"&":"?")+e.data;e.global&&!c.active++&&c.event.trigger("ajaxStart");r=(r=xb.exec(e.url))&&(r[1]&&r[1]!==location.protocol||r[2]!==location.host);if(e.dataType==="script"&&n==="GET"&&r){var z=s.getElementsByTagName("head")[0]||s.documentElement,C=s.createElement("script");C.src=e.url;if(e.scriptCharset)C.charset=e.scriptCharset;if(!j){var B= -false;C.onload=C.onreadystatechange=function(){if(!B&&(!this.readyState||this.readyState==="loaded"||this.readyState==="complete")){B=true;b();d();C.onload=C.onreadystatechange=null;z&&C.parentNode&&z.removeChild(C)}}}z.insertBefore(C,z.firstChild);return w}var E=false,x=e.xhr();if(x){e.username?x.open(n,e.url,e.async,e.username,e.password):x.open(n,e.url,e.async);try{if(e.data||a&&a.contentType)x.setRequestHeader("Content-Type",e.contentType);if(e.ifModified){c.lastModified[e.url]&&x.setRequestHeader("If-Modified-Since", -c.lastModified[e.url]);c.etag[e.url]&&x.setRequestHeader("If-None-Match",c.etag[e.url])}r||x.setRequestHeader("X-Requested-With","XMLHttpRequest");x.setRequestHeader("Accept",e.dataType&&e.accepts[e.dataType]?e.accepts[e.dataType]+", */*":e.accepts._default)}catch(ga){}if(e.beforeSend&&e.beforeSend.call(k,x,e)===false){e.global&&!--c.active&&c.event.trigger("ajaxStop");x.abort();return false}e.global&&f("ajaxSend",[x,e]);var g=x.onreadystatechange=function(q){if(!x||x.readyState===0||q==="abort"){E|| -d();E=true;if(x)x.onreadystatechange=c.noop}else if(!E&&x&&(x.readyState===4||q==="timeout")){E=true;x.onreadystatechange=c.noop;i=q==="timeout"?"timeout":!c.httpSuccess(x)?"error":e.ifModified&&c.httpNotModified(x,e.url)?"notmodified":"success";var p;if(i==="success")try{o=c.httpData(x,e.dataType,e)}catch(v){i="parsererror";p=v}if(i==="success"||i==="notmodified")j||b();else c.handleError(e,x,i,p);d();q==="timeout"&&x.abort();if(e.async)x=null}};try{var h=x.abort;x.abort=function(){x&&h.call(x); -g("abort")}}catch(l){}e.async&&e.timeout>0&&setTimeout(function(){x&&!E&&g("timeout")},e.timeout);try{x.send(n==="POST"||n==="PUT"||n==="DELETE"?e.data:null)}catch(m){c.handleError(e,x,null,m);d()}e.async||g();return x}},handleError:function(a,b,d,f){if(a.error)a.error.call(a.context||a,b,d,f);if(a.global)(a.context?c(a.context):c.event).trigger("ajaxError",[b,a,f])},active:0,httpSuccess:function(a){try{return!a.status&&location.protocol==="file:"||a.status>=200&&a.status<300||a.status===304||a.status=== -1223||a.status===0}catch(b){}return false},httpNotModified:function(a,b){var d=a.getResponseHeader("Last-Modified"),f=a.getResponseHeader("Etag");if(d)c.lastModified[b]=d;if(f)c.etag[b]=f;return a.status===304||a.status===0},httpData:function(a,b,d){var f=a.getResponseHeader("content-type")||"",e=b==="xml"||!b&&f.indexOf("xml")>=0;a=e?a.responseXML:a.responseText;e&&a.documentElement.nodeName==="parsererror"&&c.error("parsererror");if(d&&d.dataFilter)a=d.dataFilter(a,b);if(typeof a==="string")if(b=== -"json"||!b&&f.indexOf("json")>=0)a=c.parseJSON(a);else if(b==="script"||!b&&f.indexOf("javascript")>=0)c.globalEval(a);return a},param:function(a,b){function d(i,o){if(c.isArray(o))c.each(o,function(k,n){b||/\[\]$/.test(i)?f(i,n):d(i+"["+(typeof n==="object"||c.isArray(n)?k:"")+"]",n)});else!b&&o!=null&&typeof o==="object"?c.each(o,function(k,n){d(i+"["+k+"]",n)}):f(i,o)}function f(i,o){o=c.isFunction(o)?o():o;e[e.length]=encodeURIComponent(i)+"="+encodeURIComponent(o)}var e=[];if(b===w)b=c.ajaxSettings.traditional; -if(c.isArray(a)||a.jquery)c.each(a,function(){f(this.name,this.value)});else for(var j in a)d(j,a[j]);return e.join("&").replace(yb,"+")}});var la={},Ab=/toggle|show|hide/,Bb=/^([+-]=)?([\d+-.]+)(.*)$/,W,va=[["height","marginTop","marginBottom","paddingTop","paddingBottom"],["width","marginLeft","marginRight","paddingLeft","paddingRight"],["opacity"]];c.fn.extend({show:function(a,b){if(a||a===0)return this.animate(K("show",3),a,b);else{a=0;for(b=this.length;a").appendTo("body");f=e.css("display");if(f==="none")f="block";e.remove();la[d]=f}c.data(this[a],"olddisplay",f)}}a=0;for(b=this.length;a=0;f--)if(d[f].elem===this){b&&d[f](true);d.splice(f,1)}});b||this.dequeue();return this}});c.each({slideDown:K("show",1),slideUp:K("hide",1),slideToggle:K("toggle",1),fadeIn:{opacity:"show"},fadeOut:{opacity:"hide"}},function(a,b){c.fn[a]=function(d,f){return this.animate(b,d,f)}});c.extend({speed:function(a,b,d){var f=a&&typeof a==="object"?a:{complete:d||!d&&b||c.isFunction(a)&&a,duration:a,easing:d&&b||b&&!c.isFunction(b)&&b};f.duration=c.fx.off?0:typeof f.duration=== -"number"?f.duration:c.fx.speeds[f.duration]||c.fx.speeds._default;f.old=f.complete;f.complete=function(){f.queue!==false&&c(this).dequeue();c.isFunction(f.old)&&f.old.call(this)};return f},easing:{linear:function(a,b,d,f){return d+f*a},swing:function(a,b,d,f){return(-Math.cos(a*Math.PI)/2+0.5)*f+d}},timers:[],fx:function(a,b,d){this.options=b;this.elem=a;this.prop=d;if(!b.orig)b.orig={}}});c.fx.prototype={update:function(){this.options.step&&this.options.step.call(this.elem,this.now,this);(c.fx.step[this.prop]|| -c.fx.step._default)(this);if((this.prop==="height"||this.prop==="width")&&this.elem.style)this.elem.style.display="block"},cur:function(a){if(this.elem[this.prop]!=null&&(!this.elem.style||this.elem.style[this.prop]==null))return this.elem[this.prop];return(a=parseFloat(c.css(this.elem,this.prop,a)))&&a>-10000?a:parseFloat(c.curCSS(this.elem,this.prop))||0},custom:function(a,b,d){function f(j){return e.step(j)}this.startTime=J();this.start=a;this.end=b;this.unit=d||this.unit||"px";this.now=this.start; -this.pos=this.state=0;var e=this;f.elem=this.elem;if(f()&&c.timers.push(f)&&!W)W=setInterval(c.fx.tick,13)},show:function(){this.options.orig[this.prop]=c.style(this.elem,this.prop);this.options.show=true;this.custom(this.prop==="width"||this.prop==="height"?1:0,this.cur());c(this.elem).show()},hide:function(){this.options.orig[this.prop]=c.style(this.elem,this.prop);this.options.hide=true;this.custom(this.cur(),0)},step:function(a){var b=J(),d=true;if(a||b>=this.options.duration+this.startTime){this.now= -this.end;this.pos=this.state=1;this.update();this.options.curAnim[this.prop]=true;for(var f in this.options.curAnim)if(this.options.curAnim[f]!==true)d=false;if(d){if(this.options.display!=null){this.elem.style.overflow=this.options.overflow;a=c.data(this.elem,"olddisplay");this.elem.style.display=a?a:this.options.display;if(c.css(this.elem,"display")==="none")this.elem.style.display="block"}this.options.hide&&c(this.elem).hide();if(this.options.hide||this.options.show)for(var e in this.options.curAnim)c.style(this.elem, -e,this.options.orig[e]);this.options.complete.call(this.elem)}return false}else{e=b-this.startTime;this.state=e/this.options.duration;a=this.options.easing||(c.easing.swing?"swing":"linear");this.pos=c.easing[this.options.specialEasing&&this.options.specialEasing[this.prop]||a](this.state,e,0,1,this.options.duration);this.now=this.start+(this.end-this.start)*this.pos;this.update()}return true}};c.extend(c.fx,{tick:function(){for(var a=c.timers,b=0;b
"; -a.insertBefore(b,a.firstChild);d=b.firstChild;f=d.firstChild;e=d.nextSibling.firstChild.firstChild;this.doesNotAddBorder=f.offsetTop!==5;this.doesAddBorderForTableAndCells=e.offsetTop===5;f.style.position="fixed";f.style.top="20px";this.supportsFixedPosition=f.offsetTop===20||f.offsetTop===15;f.style.position=f.style.top="";d.style.overflow="hidden";d.style.position="relative";this.subtractsBorderForOverflowNotVisible=f.offsetTop===-5;this.doesNotIncludeMarginInBodyOffset=a.offsetTop!==j;a.removeChild(b); -c.offset.initialize=c.noop},bodyOffset:function(a){var b=a.offsetTop,d=a.offsetLeft;c.offset.initialize();if(c.offset.doesNotIncludeMarginInBodyOffset){b+=parseFloat(c.curCSS(a,"marginTop",true))||0;d+=parseFloat(c.curCSS(a,"marginLeft",true))||0}return{top:b,left:d}},setOffset:function(a,b,d){if(/static/.test(c.curCSS(a,"position")))a.style.position="relative";var f=c(a),e=f.offset(),j=parseInt(c.curCSS(a,"top",true),10)||0,i=parseInt(c.curCSS(a,"left",true),10)||0;if(c.isFunction(b))b=b.call(a, -d,e);d={top:b.top-e.top+j,left:b.left-e.left+i};"using"in b?b.using.call(a,d):f.css(d)}};c.fn.extend({position:function(){if(!this[0])return null;var a=this[0],b=this.offsetParent(),d=this.offset(),f=/^body|html$/i.test(b[0].nodeName)?{top:0,left:0}:b.offset();d.top-=parseFloat(c.curCSS(a,"marginTop",true))||0;d.left-=parseFloat(c.curCSS(a,"marginLeft",true))||0;f.top+=parseFloat(c.curCSS(b[0],"borderTopWidth",true))||0;f.left+=parseFloat(c.curCSS(b[0],"borderLeftWidth",true))||0;return{top:d.top- -f.top,left:d.left-f.left}},offsetParent:function(){return this.map(function(){for(var a=this.offsetParent||s.body;a&&!/^body|html$/i.test(a.nodeName)&&c.css(a,"position")==="static";)a=a.offsetParent;return a})}});c.each(["Left","Top"],function(a,b){var d="scroll"+b;c.fn[d]=function(f){var e=this[0],j;if(!e)return null;if(f!==w)return this.each(function(){if(j=wa(this))j.scrollTo(!a?f:c(j).scrollLeft(),a?f:c(j).scrollTop());else this[d]=f});else return(j=wa(e))?"pageXOffset"in j?j[a?"pageYOffset": -"pageXOffset"]:c.support.boxModel&&j.document.documentElement[d]||j.document.body[d]:e[d]}});c.each(["Height","Width"],function(a,b){var d=b.toLowerCase();c.fn["inner"+b]=function(){return this[0]?c.css(this[0],d,false,"padding"):null};c.fn["outer"+b]=function(f){return this[0]?c.css(this[0],d,false,f?"margin":"border"):null};c.fn[d]=function(f){var e=this[0];if(!e)return f==null?null:this;if(c.isFunction(f))return this.each(function(j){var i=c(this);i[d](f.call(this,j,i[d]()))});return"scrollTo"in -e&&e.document?e.document.compatMode==="CSS1Compat"&&e.document.documentElement["client"+b]||e.document.body["client"+b]:e.nodeType===9?Math.max(e.documentElement["client"+b],e.body["scroll"+b],e.documentElement["scroll"+b],e.body["offset"+b],e.documentElement["offset"+b]):f===w?c.css(e,d):this.css(d,typeof f==="string"?f:f+"px")}});A.jQuery=A.$=c})(window); diff --git a/docs/_static/minus.png b/docs/_static/minus.png deleted file mode 100644 index da1c5620d1..0000000000 Binary files a/docs/_static/minus.png and /dev/null differ diff --git a/docs/_static/navigation.png b/docs/_static/navigation.png deleted file mode 100644 index 1081dc1439..0000000000 Binary files a/docs/_static/navigation.png and /dev/null differ diff --git a/docs/_static/plus.png b/docs/_static/plus.png deleted file mode 100644 index b3cb37425e..0000000000 Binary files a/docs/_static/plus.png and /dev/null differ diff --git a/docs/_static/pygments.css b/docs/_static/pygments.css deleted file mode 100644 index 1a14f2ae1a..0000000000 --- a/docs/_static/pygments.css +++ /dev/null @@ -1,62 +0,0 @@ -.highlight .hll { background-color: #ffffcc } -.highlight { background: #eeffcc; } -.highlight .c { color: #408090; font-style: italic } /* Comment */ -.highlight .err { border: 1px solid #FF0000 } /* Error */ -.highlight .k { color: #007020; font-weight: bold } /* Keyword */ -.highlight .o { color: #666666 } /* Operator */ -.highlight .cm { color: #408090; font-style: italic } /* Comment.Multiline */ -.highlight .cp { color: #007020 } /* Comment.Preproc */ -.highlight .c1 { color: #408090; font-style: italic } /* Comment.Single */ -.highlight .cs { color: #408090; background-color: #fff0f0 } /* Comment.Special */ -.highlight .gd { color: #A00000 } /* Generic.Deleted */ -.highlight .ge { font-style: italic } /* Generic.Emph */ -.highlight .gr { color: #FF0000 } /* Generic.Error */ -.highlight .gh { color: #000080; font-weight: bold } /* Generic.Heading */ -.highlight .gi { color: #00A000 } /* Generic.Inserted */ -.highlight .go { color: #303030 } /* Generic.Output */ -.highlight .gp { color: #c65d09; font-weight: bold } /* Generic.Prompt */ -.highlight .gs { font-weight: bold } /* Generic.Strong */ -.highlight .gu { color: #800080; font-weight: bold } /* Generic.Subheading */ -.highlight .gt { color: #0040D0 } /* Generic.Traceback */ -.highlight .kc { color: #007020; font-weight: bold } /* Keyword.Constant */ -.highlight .kd { color: #007020; font-weight: bold } /* Keyword.Declaration */ -.highlight .kn { color: #007020; font-weight: bold } /* Keyword.Namespace */ -.highlight .kp { color: #007020 } /* Keyword.Pseudo */ -.highlight .kr { color: #007020; font-weight: bold } /* Keyword.Reserved */ -.highlight .kt { color: #902000 } /* Keyword.Type */ -.highlight .m { color: #208050 } /* Literal.Number */ -.highlight .s { color: #4070a0 } /* Literal.String */ -.highlight .na { color: #4070a0 } /* Name.Attribute */ -.highlight .nb { color: #007020 } /* Name.Builtin */ -.highlight .nc { color: #0e84b5; font-weight: bold } /* Name.Class */ -.highlight .no { color: #60add5 } /* Name.Constant */ -.highlight .nd { color: #555555; font-weight: bold } /* Name.Decorator */ -.highlight .ni { color: #d55537; font-weight: bold } /* Name.Entity */ -.highlight .ne { color: #007020 } /* Name.Exception */ -.highlight .nf { color: #06287e } /* Name.Function */ -.highlight .nl { color: #002070; font-weight: bold } /* Name.Label */ -.highlight .nn { color: #0e84b5; font-weight: bold } /* Name.Namespace */ -.highlight .nt { color: #062873; font-weight: bold } /* Name.Tag */ -.highlight .nv { color: #bb60d5 } /* Name.Variable */ -.highlight .ow { color: #007020; font-weight: bold } /* Operator.Word */ -.highlight .w { color: #bbbbbb } /* Text.Whitespace */ -.highlight .mf { color: #208050 } /* Literal.Number.Float */ -.highlight .mh { color: #208050 } /* Literal.Number.Hex */ -.highlight .mi { color: #208050 } /* Literal.Number.Integer */ -.highlight .mo { color: #208050 } /* Literal.Number.Oct */ -.highlight .sb { color: #4070a0 } /* Literal.String.Backtick */ -.highlight .sc { color: #4070a0 } /* Literal.String.Char */ -.highlight .sd { color: #4070a0; font-style: italic } /* Literal.String.Doc */ -.highlight .s2 { color: #4070a0 } /* Literal.String.Double */ -.highlight .se { color: #4070a0; font-weight: bold } /* Literal.String.Escape */ -.highlight .sh { color: #4070a0 } /* Literal.String.Heredoc */ -.highlight .si { color: #70a0d0; font-style: italic } /* Literal.String.Interpol */ -.highlight .sx { color: #c65d09 } /* Literal.String.Other */ -.highlight .sr { color: #235388 } /* Literal.String.Regex */ -.highlight .s1 { color: #4070a0 } /* Literal.String.Single */ -.highlight .ss { color: #517918 } /* Literal.String.Symbol */ -.highlight .bp { color: #007020 } /* Name.Builtin.Pseudo */ -.highlight .vc { color: #bb60d5 } /* Name.Variable.Class */ -.highlight .vg { color: #bb60d5 } /* Name.Variable.Global */ -.highlight .vi { color: #bb60d5 } /* Name.Variable.Instance */ -.highlight .il { color: #208050 } /* Literal.Number.Integer.Long */ \ No newline at end of file diff --git a/docs/_static/searchtools.js b/docs/_static/searchtools.js deleted file mode 100644 index 663be4c909..0000000000 --- a/docs/_static/searchtools.js +++ /dev/null @@ -1,560 +0,0 @@ -/* - * searchtools.js_t - * ~~~~~~~~~~~~~~~~ - * - * Sphinx JavaScript utilties for the full-text search. - * - * :copyright: Copyright 2007-2011 by the Sphinx team, see AUTHORS. - * :license: BSD, see LICENSE for details. - * - */ - -/** - * helper function to return a node containing the - * search summary for a given text. keywords is a list - * of stemmed words, hlwords is the list of normal, unstemmed - * words. the first one is used to find the occurance, the - * latter for highlighting it. - */ - -jQuery.makeSearchSummary = function(text, keywords, hlwords) { - var textLower = text.toLowerCase(); - var start = 0; - $.each(keywords, function() { - var i = textLower.indexOf(this.toLowerCase()); - if (i > -1) - start = i; - }); - start = Math.max(start - 120, 0); - var excerpt = ((start > 0) ? '...' : '') + - $.trim(text.substr(start, 240)) + - ((start + 240 - text.length) ? '...' : ''); - var rv = $('
').text(excerpt); - $.each(hlwords, function() { - rv = rv.highlightText(this, 'highlighted'); - }); - return rv; -} - - -/** - * Porter Stemmer - */ -var Stemmer = function() { - - var step2list = { - ational: 'ate', - tional: 'tion', - enci: 'ence', - anci: 'ance', - izer: 'ize', - bli: 'ble', - alli: 'al', - entli: 'ent', - eli: 'e', - ousli: 'ous', - ization: 'ize', - ation: 'ate', - ator: 'ate', - alism: 'al', - iveness: 'ive', - fulness: 'ful', - ousness: 'ous', - aliti: 'al', - iviti: 'ive', - biliti: 'ble', - logi: 'log' - }; - - var step3list = { - icate: 'ic', - ative: '', - alize: 'al', - iciti: 'ic', - ical: 'ic', - ful: '', - ness: '' - }; - - var c = "[^aeiou]"; // consonant - var v = "[aeiouy]"; // vowel - var C = c + "[^aeiouy]*"; // consonant sequence - var V = v + "[aeiou]*"; // vowel sequence - - var mgr0 = "^(" + C + ")?" + V + C; // [C]VC... is m>0 - var meq1 = "^(" + C + ")?" + V + C + "(" + V + ")?$"; // [C]VC[V] is m=1 - var mgr1 = "^(" + C + ")?" + V + C + V + C; // [C]VCVC... is m>1 - var s_v = "^(" + C + ")?" + v; // vowel in stem - - this.stemWord = function (w) { - var stem; - var suffix; - var firstch; - var origword = w; - - if (w.length < 3) - return w; - - var re; - var re2; - var re3; - var re4; - - firstch = w.substr(0,1); - if (firstch == "y") - w = firstch.toUpperCase() + w.substr(1); - - // Step 1a - re = /^(.+?)(ss|i)es$/; - re2 = /^(.+?)([^s])s$/; - - if (re.test(w)) - w = w.replace(re,"$1$2"); - else if (re2.test(w)) - w = w.replace(re2,"$1$2"); - - // Step 1b - re = /^(.+?)eed$/; - re2 = /^(.+?)(ed|ing)$/; - if (re.test(w)) { - var fp = re.exec(w); - re = new RegExp(mgr0); - if (re.test(fp[1])) { - re = /.$/; - w = w.replace(re,""); - } - } - else if (re2.test(w)) { - var fp = re2.exec(w); - stem = fp[1]; - re2 = new RegExp(s_v); - if (re2.test(stem)) { - w = stem; - re2 = /(at|bl|iz)$/; - re3 = new RegExp("([^aeiouylsz])\\1$"); - re4 = new RegExp("^" + C + v + "[^aeiouwxy]$"); - if (re2.test(w)) - w = w + "e"; - else if (re3.test(w)) { - re = /.$/; - w = w.replace(re,""); - } - else if (re4.test(w)) - w = w + "e"; - } - } - - // Step 1c - re = /^(.+?)y$/; - if (re.test(w)) { - var fp = re.exec(w); - stem = fp[1]; - re = new RegExp(s_v); - if (re.test(stem)) - w = stem + "i"; - } - - // Step 2 - re = /^(.+?)(ational|tional|enci|anci|izer|bli|alli|entli|eli|ousli|ization|ation|ator|alism|iveness|fulness|ousness|aliti|iviti|biliti|logi)$/; - if (re.test(w)) { - var fp = re.exec(w); - stem = fp[1]; - suffix = fp[2]; - re = new RegExp(mgr0); - if (re.test(stem)) - w = stem + step2list[suffix]; - } - - // Step 3 - re = /^(.+?)(icate|ative|alize|iciti|ical|ful|ness)$/; - if (re.test(w)) { - var fp = re.exec(w); - stem = fp[1]; - suffix = fp[2]; - re = new RegExp(mgr0); - if (re.test(stem)) - w = stem + step3list[suffix]; - } - - // Step 4 - re = /^(.+?)(al|ance|ence|er|ic|able|ible|ant|ement|ment|ent|ou|ism|ate|iti|ous|ive|ize)$/; - re2 = /^(.+?)(s|t)(ion)$/; - if (re.test(w)) { - var fp = re.exec(w); - stem = fp[1]; - re = new RegExp(mgr1); - if (re.test(stem)) - w = stem; - } - else if (re2.test(w)) { - var fp = re2.exec(w); - stem = fp[1] + fp[2]; - re2 = new RegExp(mgr1); - if (re2.test(stem)) - w = stem; - } - - // Step 5 - re = /^(.+?)e$/; - if (re.test(w)) { - var fp = re.exec(w); - stem = fp[1]; - re = new RegExp(mgr1); - re2 = new RegExp(meq1); - re3 = new RegExp("^" + C + v + "[^aeiouwxy]$"); - if (re.test(stem) || (re2.test(stem) && !(re3.test(stem)))) - w = stem; - } - re = /ll$/; - re2 = new RegExp(mgr1); - if (re.test(w) && re2.test(w)) { - re = /.$/; - w = w.replace(re,""); - } - - // and turn initial Y back to y - if (firstch == "y") - w = firstch.toLowerCase() + w.substr(1); - return w; - } -} - - -/** - * Search Module - */ -var Search = { - - _index : null, - _queued_query : null, - _pulse_status : -1, - - init : function() { - var params = $.getQueryParameters(); - if (params.q) { - var query = params.q[0]; - $('input[name="q"]')[0].value = query; - this.performSearch(query); - } - }, - - loadIndex : function(url) { - $.ajax({type: "GET", url: url, data: null, success: null, - dataType: "script", cache: true}); - }, - - setIndex : function(index) { - var q; - this._index = index; - if ((q = this._queued_query) !== null) { - this._queued_query = null; - Search.query(q); - } - }, - - hasIndex : function() { - return this._index !== null; - }, - - deferQuery : function(query) { - this._queued_query = query; - }, - - stopPulse : function() { - this._pulse_status = 0; - }, - - startPulse : function() { - if (this._pulse_status >= 0) - return; - function pulse() { - Search._pulse_status = (Search._pulse_status + 1) % 4; - var dotString = ''; - for (var i = 0; i < Search._pulse_status; i++) - dotString += '.'; - Search.dots.text(dotString); - if (Search._pulse_status > -1) - window.setTimeout(pulse, 500); - }; - pulse(); - }, - - /** - * perform a search for something - */ - performSearch : function(query) { - // create the required interface elements - this.out = $('#search-results'); - this.title = $('

' + _('Searching') + '

').appendTo(this.out); - this.dots = $('').appendTo(this.title); - this.status = $('

').appendTo(this.out); - this.output = $('
    ').appendTo(this.out); - - $('#search-progress').text(_('Preparing search...')); - this.startPulse(); - - // index already loaded, the browser was quick! - if (this.hasIndex()) - this.query(query); - else - this.deferQuery(query); - }, - - query : function(query) { - var stopwords = ["and","then","into","it","as","are","in","if","for","no","there","their","was","is","be","to","that","but","they","not","such","with","by","a","on","these","of","will","this","near","the","or","at"]; - - // Stem the searchterms and add them to the correct list - var stemmer = new Stemmer(); - var searchterms = []; - var excluded = []; - var hlterms = []; - var tmp = query.split(/\s+/); - var objectterms = []; - for (var i = 0; i < tmp.length; i++) { - if (tmp[i] != "") { - objectterms.push(tmp[i].toLowerCase()); - } - - if ($u.indexOf(stopwords, tmp[i]) != -1 || tmp[i].match(/^\d+$/) || - tmp[i] == "") { - // skip this "word" - continue; - } - // stem the word - var word = stemmer.stemWord(tmp[i]).toLowerCase(); - // select the correct list - if (word[0] == '-') { - var toAppend = excluded; - word = word.substr(1); - } - else { - var toAppend = searchterms; - hlterms.push(tmp[i].toLowerCase()); - } - // only add if not already in the list - if (!$.contains(toAppend, word)) - toAppend.push(word); - }; - var highlightstring = '?highlight=' + $.urlencode(hlterms.join(" ")); - - // console.debug('SEARCH: searching for:'); - // console.info('required: ', searchterms); - // console.info('excluded: ', excluded); - - // prepare search - var filenames = this._index.filenames; - var titles = this._index.titles; - var terms = this._index.terms; - var fileMap = {}; - var files = null; - // different result priorities - var importantResults = []; - var objectResults = []; - var regularResults = []; - var unimportantResults = []; - $('#search-progress').empty(); - - // lookup as object - for (var i = 0; i < objectterms.length; i++) { - var others = [].concat(objectterms.slice(0,i), - objectterms.slice(i+1, objectterms.length)) - var results = this.performObjectSearch(objectterms[i], others); - // Assume first word is most likely to be the object, - // other words more likely to be in description. - // Therefore put matches for earlier words first. - // (Results are eventually used in reverse order). - objectResults = results[0].concat(objectResults); - importantResults = results[1].concat(importantResults); - unimportantResults = results[2].concat(unimportantResults); - } - - // perform the search on the required terms - for (var i = 0; i < searchterms.length; i++) { - var word = searchterms[i]; - // no match but word was a required one - if ((files = terms[word]) == null) - break; - if (files.length == undefined) { - files = [files]; - } - // create the mapping - for (var j = 0; j < files.length; j++) { - var file = files[j]; - if (file in fileMap) - fileMap[file].push(word); - else - fileMap[file] = [word]; - } - } - - // now check if the files don't contain excluded terms - for (var file in fileMap) { - var valid = true; - - // check if all requirements are matched - if (fileMap[file].length != searchterms.length) - continue; - - // ensure that none of the excluded terms is in the - // search result. - for (var i = 0; i < excluded.length; i++) { - if (terms[excluded[i]] == file || - $.contains(terms[excluded[i]] || [], file)) { - valid = false; - break; - } - } - - // if we have still a valid result we can add it - // to the result list - if (valid) - regularResults.push([filenames[file], titles[file], '', null]); - } - - // delete unused variables in order to not waste - // memory until list is retrieved completely - delete filenames, titles, terms; - - // now sort the regular results descending by title - regularResults.sort(function(a, b) { - var left = a[1].toLowerCase(); - var right = b[1].toLowerCase(); - return (left > right) ? -1 : ((left < right) ? 1 : 0); - }); - - // combine all results - var results = unimportantResults.concat(regularResults) - .concat(objectResults).concat(importantResults); - - // print the results - var resultCount = results.length; - function displayNextItem() { - // results left, load the summary and display it - if (results.length) { - var item = results.pop(); - var listItem = $('
  • '); - if (DOCUMENTATION_OPTIONS.FILE_SUFFIX == '') { - // dirhtml builder - var dirname = item[0] + '/'; - if (dirname.match(/\/index\/$/)) { - dirname = dirname.substring(0, dirname.length-6); - } else if (dirname == 'index/') { - dirname = ''; - } - listItem.append($('').attr('href', - DOCUMENTATION_OPTIONS.URL_ROOT + dirname + - highlightstring + item[2]).html(item[1])); - } else { - // normal html builders - listItem.append($('').attr('href', - item[0] + DOCUMENTATION_OPTIONS.FILE_SUFFIX + - highlightstring + item[2]).html(item[1])); - } - if (item[3]) { - listItem.append($(' (' + item[3] + ')')); - Search.output.append(listItem); - listItem.slideDown(5, function() { - displayNextItem(); - }); - } else if (DOCUMENTATION_OPTIONS.HAS_SOURCE) { - $.get(DOCUMENTATION_OPTIONS.URL_ROOT + '_sources/' + - item[0] + '.txt', function(data) { - if (data != '') { - listItem.append($.makeSearchSummary(data, searchterms, hlterms)); - Search.output.append(listItem); - } - listItem.slideDown(5, function() { - displayNextItem(); - }); - }, "text"); - } else { - // no source available, just display title - Search.output.append(listItem); - listItem.slideDown(5, function() { - displayNextItem(); - }); - } - } - // search finished, update title and status message - else { - Search.stopPulse(); - Search.title.text(_('Search Results')); - if (!resultCount) - Search.status.text(_('Your search did not match any documents. Please make sure that all words are spelled correctly and that you\'ve selected enough categories.')); - else - Search.status.text(_('Search finished, found %s page(s) matching the search query.').replace('%s', resultCount)); - Search.status.fadeIn(500); - } - } - displayNextItem(); - }, - - performObjectSearch : function(object, otherterms) { - var filenames = this._index.filenames; - var objects = this._index.objects; - var objnames = this._index.objnames; - var titles = this._index.titles; - - var importantResults = []; - var objectResults = []; - var unimportantResults = []; - - for (var prefix in objects) { - for (var name in objects[prefix]) { - var fullname = (prefix ? prefix + '.' : '') + name; - if (fullname.toLowerCase().indexOf(object) > -1) { - var match = objects[prefix][name]; - var objname = objnames[match[1]][2]; - var title = titles[match[0]]; - // If more than one term searched for, we require other words to be - // found in the name/title/description - if (otherterms.length > 0) { - var haystack = (prefix + ' ' + name + ' ' + - objname + ' ' + title).toLowerCase(); - var allfound = true; - for (var i = 0; i < otherterms.length; i++) { - if (haystack.indexOf(otherterms[i]) == -1) { - allfound = false; - break; - } - } - if (!allfound) { - continue; - } - } - var descr = objname + _(', in ') + title; - anchor = match[3]; - if (anchor == '') - anchor = fullname; - else if (anchor == '-') - anchor = objnames[match[1]][1] + '-' + fullname; - result = [filenames[match[0]], fullname, '#'+anchor, descr]; - switch (match[2]) { - case 1: objectResults.push(result); break; - case 0: importantResults.push(result); break; - case 2: unimportantResults.push(result); break; - } - } - } - } - - // sort results descending - objectResults.sort(function(a, b) { - return (a[1] > b[1]) ? -1 : ((a[1] < b[1]) ? 1 : 0); - }); - - importantResults.sort(function(a, b) { - return (a[1] > b[1]) ? -1 : ((a[1] < b[1]) ? 1 : 0); - }); - - unimportantResults.sort(function(a, b) { - return (a[1] > b[1]) ? -1 : ((a[1] < b[1]) ? 1 : 0); - }); - - return [importantResults, objectResults, unimportantResults] - } -} - -$(document).ready(function() { - Search.init(); -}); \ No newline at end of file diff --git a/docs/_static/sidebar.js b/docs/_static/sidebar.js deleted file mode 100644 index a45e1926ad..0000000000 --- a/docs/_static/sidebar.js +++ /dev/null @@ -1,151 +0,0 @@ -/* - * sidebar.js - * ~~~~~~~~~~ - * - * This script makes the Sphinx sidebar collapsible. - * - * .sphinxsidebar contains .sphinxsidebarwrapper. This script adds - * in .sphixsidebar, after .sphinxsidebarwrapper, the #sidebarbutton - * used to collapse and expand the sidebar. - * - * When the sidebar is collapsed the .sphinxsidebarwrapper is hidden - * and the width of the sidebar and the margin-left of the document - * are decreased. When the sidebar is expanded the opposite happens. - * This script saves a per-browser/per-session cookie used to - * remember the position of the sidebar among the pages. - * Once the browser is closed the cookie is deleted and the position - * reset to the default (expanded). - * - * :copyright: Copyright 2007-2011 by the Sphinx team, see AUTHORS. - * :license: BSD, see LICENSE for details. - * - */ - -$(function() { - // global elements used by the functions. - // the 'sidebarbutton' element is defined as global after its - // creation, in the add_sidebar_button function - var bodywrapper = $('.bodywrapper'); - var sidebar = $('.sphinxsidebar'); - var sidebarwrapper = $('.sphinxsidebarwrapper'); - - // for some reason, the document has no sidebar; do not run into errors - if (!sidebar.length) return; - - // original margin-left of the bodywrapper and width of the sidebar - // with the sidebar expanded - var bw_margin_expanded = bodywrapper.css('margin-left'); - var ssb_width_expanded = sidebar.width(); - - // margin-left of the bodywrapper and width of the sidebar - // with the sidebar collapsed - var bw_margin_collapsed = '.8em'; - var ssb_width_collapsed = '.8em'; - - // colors used by the current theme - var dark_color = $('.related').css('background-color'); - var light_color = $('.document').css('background-color'); - - function sidebar_is_collapsed() { - return sidebarwrapper.is(':not(:visible)'); - } - - function toggle_sidebar() { - if (sidebar_is_collapsed()) - expand_sidebar(); - else - collapse_sidebar(); - } - - function collapse_sidebar() { - sidebarwrapper.hide(); - sidebar.css('width', ssb_width_collapsed); - bodywrapper.css('margin-left', bw_margin_collapsed); - sidebarbutton.css({ - 'margin-left': '0', - 'height': bodywrapper.height() - }); - sidebarbutton.find('span').text('»'); - sidebarbutton.attr('title', _('Expand sidebar')); - document.cookie = 'sidebar=collapsed'; - } - - function expand_sidebar() { - bodywrapper.css('margin-left', bw_margin_expanded); - sidebar.css('width', ssb_width_expanded); - sidebarwrapper.show(); - sidebarbutton.css({ - 'margin-left': ssb_width_expanded-12, - 'height': bodywrapper.height() - }); - sidebarbutton.find('span').text('«'); - sidebarbutton.attr('title', _('Collapse sidebar')); - document.cookie = 'sidebar=expanded'; - } - - function add_sidebar_button() { - sidebarwrapper.css({ - 'float': 'left', - 'margin-right': '0', - 'width': ssb_width_expanded - 28 - }); - // create the button - sidebar.append( - '
    «
    ' - ); - var sidebarbutton = $('#sidebarbutton'); - light_color = sidebarbutton.css('background-color'); - // find the height of the viewport to center the '<<' in the page - var viewport_height; - if (window.innerHeight) - viewport_height = window.innerHeight; - else - viewport_height = $(window).height(); - sidebarbutton.find('span').css({ - 'display': 'block', - 'margin-top': (viewport_height - sidebar.position().top - 20) / 2 - }); - - sidebarbutton.click(toggle_sidebar); - sidebarbutton.attr('title', _('Collapse sidebar')); - sidebarbutton.css({ - 'color': '#FFFFFF', - 'border-left': '1px solid ' + dark_color, - 'font-size': '1.2em', - 'cursor': 'pointer', - 'height': bodywrapper.height(), - 'padding-top': '1px', - 'margin-left': ssb_width_expanded - 12 - }); - - sidebarbutton.hover( - function () { - $(this).css('background-color', dark_color); - }, - function () { - $(this).css('background-color', light_color); - } - ); - } - - function set_position_from_cookie() { - if (!document.cookie) - return; - var items = document.cookie.split(';'); - for(var k=0; kthis.time){this._setStyle(this.to);if(this.callback){this.callback.call(this)}clearInterval(this.timer);return }this.percentage=(Math.floor((this.diff/this.time)*100)/100);this.val=(this.animDiff*this.percentage)+this.from;this._setStyle(this.val)};A.prototype.start=function(){var C=this;this.startTime=new Date();this.timer=setInterval(function(){C._animate.call(C)},15)};TWTR.Widget=function(C){this.init(C)};(function(){var R={};var O=location.protocol.match(/https/);var Q=/^.+\/profile_images/;var W="https://s3.amazonaws.com/twitter_production/profile_images";var f={};var d=function(h){var g=f[h];if(!g){g=new RegExp("(?:^|\\s+)"+h+"(?:\\s+|$)");f[h]=g}return g};var D=function(l,p,m,n){var p=p||"*";var m=m||document;var h=[],g=m.getElementsByTagName(p),o=d(l);for(var j=0,k=g.length;j0&&n<13){h="am";return n}else{if(n<1){h="am";return 12}else{h="pm";return n-12}}}();var i=m.getMinutes();var l=m.getSeconds();function k(){var n=new Date();if(n.getDate()!=m.getDate()||n.getYear()!=m.getYear()||n.getMonth()!=m.getMonth()){return" - "+M[m.getMonth()]+" "+m.getDate()+", "+m.getFullYear()}else{return""}}return g+":"+i+h+k()};var K=function(m){var o=new Date();var k=new Date(m);if(e.ie){k=Date.parse(m.replace(/( \+)/," UTC$1"))}var n=o-k;var h=1000,i=h*60,j=i*60,l=j*24,g=l*7;if(isNaN(n)||n<0){return""}if(nl&&n'+((l.length>25)?l.substr(0,24)+"...":l)+"    "+h})},at:function(g){return g.replace(/\B[@＠]([a-zA-Z0-9_]{1,20})/g,function(h,i){return'@'+i+""})},list:function(g){return g.replace(/\B[@＠]([a-zA-Z0-9_]{1,20}\/\w+)/g,function(h,i){return'@'+i+""})},hash:function(g){return g.replace(/(^|\s+)#(\w+)/gi,function(h,i,j){return i+'#'+j+""})},clean:function(g){return this.hash(this.at(this.list(this.link(g))))}};function P(h,i,g){this.job=h;this.decayFn=i;this.interval=g;this.decayRate=1;this.decayMultiplier=1.25;this.maxDecayTime=3*60*1000}P.prototype={start:function(){this.stop().run();return this},stop:function(){if(this.worker){window.clearTimeout(this.worker)}return this},run:function(){var g=this;this.job(function(){g.decayRate=g.decayFn()?Math.max(1,g.decayRate/g.decayMultiplier):g.decayRate*g.decayMultiplier;var h=g.interval*g.decayRate;h=(h>=g.maxDecayTime)?g.maxDecayTime:h;h=Math.floor(h);g.worker=window.setTimeout(function(){g.run.call(g)},h)})},destroy:function(){this.stop();this.decayRate=1;return this}};function J(h,g,i){this.time=h||6000;this.loop=g||false;this.repeated=0;this.callback=i;this.haystack=[]}J.prototype={set:function(g){this.haystack=g},add:function(g){this.haystack.unshift(g)},start:function(){if(this.timer){return this}this._job();var g=this;this.timer=setInterval(function(){g._job.call(g)},this.time);return this},stop:function(){if(this.timer){window.clearInterval(this.timer);this.timer=null}return this},_next:function(){var g=this.haystack.shift();if(g&&this.loop){this.haystack.push(g)}return g||null},_job:function(){var g=this._next();if(g){this.callback(g)}return this}};function L(i){function g(){if(i.needle.metadata&&i.needle.metadata.result_type&&i.needle.metadata.result_type=="popular"){return''+i.needle.metadata.recent_retweets+"+ recent retweets"}else{return""}}if(O){i.avatar=i.avatar.replace(Q,W)}var h='
    '+i.user+' profile

    '+i.user+" "+i.tweet+' '+i.created_at+' · reply '+g()+"

    ";var j=document.createElement("div");j.id="tweet-id-"+ ++L._tweetCount;j.className="twtr-tweet";j.innerHTML=h;this.element=j}L._tweetCount=0;R.loadStyleSheet=function(i,h){if(!TWTR.Widget.loadingStyleSheet){TWTR.Widget.loadingStyleSheet=true;var g=document.createElement("link");g.href=i;g.rel="stylesheet";g.type="text/css";document.getElementsByTagName("head")[0].appendChild(g);var j=setInterval(function(){var k=G(h,"position");if(k=="relative"){clearInterval(j);j=null;TWTR.Widget.hasLoadedStyleSheet=true}},50)}};(function(){var g=false;R.css=function(j){var i=document.createElement("style");i.type="text/css";if(e.ie){i.styleSheet.cssText=j}else{var k=document.createDocumentFragment();k.appendChild(document.createTextNode(j));i.appendChild(k)}function h(){document.getElementsByTagName("head")[0].appendChild(i)}if(!e.ie||g){h()}else{window.attachEvent("onload",function(){g=true;h()})}}})();TWTR.Widget.isLoaded=false;TWTR.Widget.loadingStyleSheet=false;TWTR.Widget.hasLoadedStyleSheet=false;TWTR.Widget.WIDGET_NUMBER=0;TWTR.Widget.matches={mentions:/^@[a-zA-Z0-9_]{1,20}\b/,any_mentions:/\b@[a-zA-Z0-9_]{1,20}\b/};TWTR.Widget.jsonP=function(h,j){var g=document.createElement("script");var i=document.getElementsByTagName("head")[0];g.type="text/javascript";g.src=h;i.insertBefore(g,i.firstChild);j(g);return g};TWTR.Widget.prototype=function(){var j=O?"https://":"http://";var l=j+"search.twitter.com/search.";var m=j+"api.twitter.com/1/statuses/user_timeline.";var i=j+"twitter.com/favorites/";var k=j+"twitter.com/";var h=25000;var g=O?"https://twitter-widgets.s3.amazonaws.com/j/1/default.gif":"http://widgets.twimg.com/j/1/default.gif";return{init:function(o){var n=this;this._widgetNumber=++TWTR.Widget.WIDGET_NUMBER;TWTR.Widget["receiveCallback_"+this._widgetNumber]=function(p){n._prePlay.call(n,p)};this._cb="TWTR.Widget.receiveCallback_"+this._widgetNumber;this.opts=o;this._base=l;this._isRunning=false;this._hasOfficiallyStarted=false;this._hasNewSearchResults=false;this._rendered=false;this._profileImage=false;this._isCreator=!!o.creator;this._setWidgetType(o.type);this.timesRequested=0;this.runOnce=false;this.newResults=false;this.results=[];this.jsonMaxRequestTimeOut=19000;this.showedResults=[];this.sinceId=1;this.source="TWITTERINC_WIDGET";this.id=o.id||"twtr-widget-"+this._widgetNumber;this.tweets=0;this.setDimensions(o.width,o.height);this.interval=o.interval||6000;this.format="json";this.rpp=o.rpp||50;this.subject=o.subject||"";this.title=o.title||"";this.setFooterText(o.footer);this.setSearch(o.search);this._setUrl();this.theme=o.theme?o.theme:this._getDefaultTheme();if(!o.id){document.write('
    ')}this.widgetEl=H(this.id);if(o.id){b.add(this.widgetEl,"twtr-widget")}if(o.version>=2&&!TWTR.Widget.hasLoadedStyleSheet){if(O){R.loadStyleSheet("https://twitter-widgets.s3.amazonaws.com/j/2/widget.css",this.widgetEl)}else{if(o.creator){R.loadStyleSheet("/stylesheets/widgets/widget.css",this.widgetEl)}else{R.loadStyleSheet("http://widgets.twimg.com/j/2/widget.css",this.widgetEl)}}}this.occasionalJob=new P(function(p){n.decay=p;n._getResults.call(n)},function(){return n._decayDecider.call(n)},h);this._ready=I.fn(o.ready)?o.ready:function(){};this._isRelativeTime=true;this._tweetFilter=false;this._avatars=true;this._isFullScreen=false;this._isLive=true;this._isScroll=false;this._loop=true;this._showTopTweets=(this._isSearchWidget)?true:false;this._behavior="default";this.setFeatures(this.opts.features);this.intervalJob=new J(this.interval,this._loop,function(p){n._normalizeTweet(p)});return this},setDimensions:function(n,o){this.wh=(n&&o)?[n,o]:[250,300];if(n=="auto"||n=="100%"){this.wh[0]="100%"}else{this.wh[0]=((this.wh[0]<150)?150:this.wh[0])+"px"}this.wh[1]=((this.wh[1]<100)?100:this.wh[1])+"px";return this},setRpp:function(n){var n=parseInt(n);this.rpp=(I.number(n)&&(n>0&&n<=100))?n:30;return this},_setWidgetType:function(n){this._isSearchWidget=false,this._isProfileWidget=false,this._isFavsWidget=false,this._isListWidget=false;switch(n){case"profile":this._isProfileWidget=true;break;case"search":this._isSearchWidget=true,this.search=this.opts.search;break;case"faves":case"favs":this._isFavsWidget=true;break;case"list":case"lists":this._isListWidget=true;break}return this},setFeatures:function(o){if(o){if(I.def(o.filters)){this._tweetFilter=o.filters}if(I.def(o.dateformat)){this._isRelativeTime=!!(o.dateformat!=="absolute")}if(I.def(o.fullscreen)&&I.bool(o.fullscreen)){if(o.fullscreen){this._isFullScreen=true;this.wh[0]="100%";this.wh[1]=(V()-90)+"px";var p=this;E.add(window,"resize",function(s){p.wh[1]=V();p._fullScreenResize()})}}if(I.def(o.loop)&&I.bool(o.loop)){this._loop=o.loop}if(I.def(o.behavior)&&I.string(o.behavior)){switch(o.behavior){case"all":this._behavior="all";break;case"preloaded":this._behavior="preloaded";break;default:this._behavior="default";break}}if(I.def(o.toptweets)&&I.bool(o.toptweets)){this._showTopTweets=o.toptweets;var n=(this._showTopTweets)?"inline-block":"none";R.css("#"+this.id+" .twtr-popular { display: "+n+"; }")}if(!I.def(o.toptweets)){this._showTopTweets=true;var n=(this._showTopTweets)?"inline-block":"none";R.css("#"+this.id+" .twtr-popular { display: "+n+"; }")}if(I.def(o.avatars)&&I.bool(o.avatars)){if(!o.avatars){R.css("#"+this.id+" .twtr-avatar, #"+this.id+" .twtr-user { display: none; } #"+this.id+" .twtr-tweet-text { margin-left: 0; }");this._avatars=false}else{var q=(this._isFullScreen)?"90px":"40px";R.css("#"+this.id+" .twtr-avatar { display: block; } #"+this.id+" .twtr-user { display: inline; } #"+this.id+" .twtr-tweet-text { margin-left: "+q+"; }");this._avatars=true}}else{if(this._isProfileWidget){this.setFeatures({avatars:false});this._avatars=false}else{this.setFeatures({avatars:true});this._avatars=true}}if(I.def(o.hashtags)&&I.bool(o.hashtags)){(!o.hashtags)?R.css("#"+this.id+" a.twtr-hashtag { display: none; }"):""}if(I.def(o.timestamp)&&I.bool(o.timestamp)){var r=o.timestamp?"block":"none";R.css("#"+this.id+" em { display: "+r+"; }")}if(I.def(o.live)&&I.bool(o.live)){this._isLive=o.live}if(I.def(o.scrollbar)&&I.bool(o.scrollbar)){this._isScroll=o.scrollbar}}else{if(this._isProfileWidget){this.setFeatures({avatars:false});this._avatars=false}if(this._isProfileWidget||this._isFavsWidget){this.setFeatures({behavior:"all"})}}return this},_fullScreenResize:function(){var n=D("twtr-timeline","div",document.body,function(o){o.style.height=(V()-90)+"px"})},setTweetInterval:function(n){this.interval=n;return this},setBase:function(n){this._base=n;return this},setUser:function(o,n){this.username=o;this.realname=n||" ";if(this._isFavsWidget){this.setBase(i+o+".")}else{if(this._isProfileWidget){this.setBase(m+this.format+"?screen_name="+o)}}this.setSearch(" ");return this},setList:function(o,n){this.listslug=n.replace(/ /g,"-").toLowerCase();this.username=o;this.setBase(k+o+"/lists/"+this.listslug+"/statuses.");this.setSearch(" ");return this},setProfileImage:function(n){this._profileImage=n;this.byClass("twtr-profile-img","img").src=O?n.replace(Q,W):n;this.byClass("twtr-profile-img-anchor","a").href="http://twitter.com/"+this.username;return this},setTitle:function(n){this.title=n;this.widgetEl.getElementsByTagName("h3")[0].innerHTML=this.title;return this},setCaption:function(n){this.subject=n;this.widgetEl.getElementsByTagName("h4")[0].innerHTML=this.subject;return this},setFooterText:function(n){this.footerText=(I.def(n)&&I.string(n))?n:"Join the conversation";if(this._rendered){this.byClass("twtr-join-conv","a").innerHTML=this.footerText}return this},setSearch:function(o){this.searchString=o||"";this.search=encodeURIComponent(this.searchString);this._setUrl();if(this._rendered){var n=this.byClass("twtr-join-conv","a");n.href="http://twitter.com/"+this._getWidgetPath()}return this},_getWidgetPath:function(){if(this._isProfileWidget){return this.username}else{if(this._isFavsWidget){return this.username+"/favorites"}else{if(this._isListWidget){return this.username+"/lists/"+this.listslug}else{return"#search?q="+this.search}}}},_setUrl:function(){var o=this;function n(){return"&"+(+new Date)+"=cachebust"}function p(){return(o.sinceId==1)?"":"&since_id="+o.sinceId+"&refresh=true"}if(this._isProfileWidget){this.url=this._base+"&callback="+this._cb+"&include_rts=true&count="+this.rpp+p()+"&clientsource="+this.source}else{if(this._isFavsWidget||this._isListWidget){this.url=this._base+this.format+"?callback="+this._cb+p()+"&include_rts=true&clientsource="+this.source}else{this.url=this._base+this.format+"?q="+this.search+"&include_rts=true&callback="+this._cb+"&rpp="+this.rpp+p()+"&clientsource="+this.source;if(!this.runOnce){this.url+="&result_type=mixed"}}}this.url+=n();return this},_getRGB:function(n){return N(n.substring(1,7))},setTheme:function(t,n){var r=this;var p=" !important";var s=((window.location.hostname.match(/twitter\.com/))&&(window.location.pathname.match(/goodies/)));if(n||s){p=""}this.theme={shell:{background:function(){return t.shell.background||r._getDefaultTheme().shell.background}(),color:function(){return t.shell.color||r._getDefaultTheme().shell.color}()},tweets:{background:function(){return t.tweets.background||r._getDefaultTheme().tweets.background}(),color:function(){return t.tweets.color||r._getDefaultTheme().tweets.color}(),links:function(){return t.tweets.links||r._getDefaultTheme().tweets.links}()}};var q="#"+this.id+" .twtr-doc, #"+this.id+" .twtr-hd a, #"+this.id+" h3, #"+this.id+" h4, #"+this.id+" .twtr-popular { background-color: "+this.theme.shell.background+p+"; color: "+this.theme.shell.color+p+"; } #"+this.id+" .twtr-popular { color: "+this.theme.tweets.color+p+"; background-color: rgba("+this._getRGB(this.theme.shell.background)+", .3)"+p+"; } #"+this.id+" .twtr-tweet a { color: "+this.theme.tweets.links+p+"; } #"+this.id+" .twtr-bd, #"+this.id+" .twtr-timeline i a, #"+this.id+" .twtr-bd p { color: "+this.theme.tweets.color+p+"; } #"+this.id+" .twtr-new-results, #"+this.id+" .twtr-results-inner, #"+this.id+" .twtr-timeline { background: "+this.theme.tweets.background+p+"; }";if(e.ie){q+="#"+this.id+" .twtr-tweet { background: "+this.theme.tweets.background+p+"; }"}R.css(q);return this},byClass:function(q,n,o){var p=D(q,n,H(this.id));return(o)?p:p[0]},render:function(){var p=this;if(!TWTR.Widget.hasLoadedStyleSheet){window.setTimeout(function(){p.render.call(p)},50);return this}this.setTheme(this.theme,this._isCreator);if(this._isProfileWidget){b.add(this.widgetEl,"twtr-widget-profile")}if(this._isScroll){b.add(this.widgetEl,"twtr-scroll")}if(!this._isLive&&!this._isScroll){this.wh[1]="auto"}if(this._isSearchWidget&&this._isFullScreen){document.title="Twitter search: "+escape(this.searchString)}this.widgetEl.innerHTML=this._getWidgetHtml();var o=this.byClass("twtr-timeline","div");if(this._isLive&&!this._isFullScreen){var q=function(r){if(p._behavior==="all"){return }if(C.call(this,r)){p.pause.call(p)}};var n=function(r){if(p._behavior==="all"){return }if(C.call(this,r)){p.resume.call(p)}};this.removeEvents=function(){E.remove(o,"mouseover",q);E.remove(o,"mouseout",n)};E.add(o,"mouseover",q);E.add(o,"mouseout",n)}this._rendered=true;this._ready();return this},removeEvents:function(){},_getDefaultTheme:function(){return{shell:{background:"#8ec1da",color:"#ffffff"},tweets:{background:"#ffffff",color:"#444444",links:"#1985b5"}}},_getWidgetHtml:function(){var p=this;function r(){if(p._isProfileWidget){return'profile

    '}else{return"

    "+p.title+"

    "+p.subject+"

    "}}function o(){return p._isFullScreen?" twtr-fullscreen":""}var q=O?"https://twitter-widgets.s3.amazonaws.com/i/widget-logo.png":"http://widgets.twimg.com/i/widget-logo.png";if(this._isFullScreen){q="https://twitter-widgets.s3.amazonaws.com/i/widget-logo-fullscreen.png"}var n='
    '+r()+'
    ";return n},_appendTweet:function(n){this._insertNewResultsNumber();Z(n,this.byClass("twtr-reference-tweet","div"));return this},_slide:function(o){var p=this;var n=Y(o).offsetHeight;if(this.runOnce){new A(o,"height",{from:0,to:n,time:500,callback:function(){p._fade.call(p,o)}}).start()}return this},_fade:function(n){var o=this;if(A.canTransition){n.style.webkitTransition="opacity 0.5s ease-out";n.style.opacity=1;return this}new A(n,"opacity",{from:0,to:1,time:500}).start();return this},_chop:function(){if(this._isScroll){return this}var s=this.byClass("twtr-tweet","div",true);var t=this.byClass("twtr-new-results","div",true);if(s.length){for(var p=s.length-1;p>=0;p--){var r=s[p];var q=parseInt(r.offsetTop);if(q>parseInt(this.wh[1])){a(r)}else{break}}if(t.length>0){var n=t[t.length-1];var o=parseInt(n.offsetTop);if(o>parseInt(this.wh[1])){a(n)}}}return this},_appendSlideFade:function(o){var n=o||this.tweet.element;this._chop()._appendTweet(n)._slide(n);return this},_createTweet:function(n){n.timestamp=n.created_at;n.created_at=this._isRelativeTime?K(n.created_at):S(n.created_at);this.tweet=new L(n);if(this._isLive&&this.runOnce){this.tweet.element.style.opacity=0;this.tweet.element.style.filter="alpha(opacity:0)";this.tweet.element.style.height="0"}return this},_getResults:function(){var n=this;this.timesRequested++;this.jsonRequestRunning=true;this.jsonRequestTimer=window.setTimeout(function(){if(n.jsonRequestRunning){clearTimeout(n.jsonRequestTimer);n.jsonRequestTimer=null}n.jsonRequestRunning=false;a(n.scriptElement);n.newResults=false;n.decay()},this.jsonMaxRequestTimeOut);TWTR.Widget.jsonP(n.url,function(o){n.scriptElement=o})},clear:function(){var o=this.byClass("twtr-tweet","div",true);var n=this.byClass("twtr-new-results","div",true);o=o.concat(n);B(o,function(p){a(p)});return this},_sortByMagic:function(n){var o=this;if(this._tweetFilter){if(this._tweetFilter.negatives){n=n.filter(function(p){if(!o._tweetFilter.negatives.test(p.text)){return p}})}if(this._tweetFilter.positives){n=n.filter(function(p){if(o._tweetFilter.positives.test(p.text)){return p}})}}switch(this._behavior){case"all":this._sortByLatest(n);break;case"preloaded":default:this._sortByDefault(n);break}if(this._isLive&&this._behavior!=="all"){this.intervalJob.set(this.results);this.intervalJob.start()}return this},_loadTopTweetsAtTop:function(p){var q=[],r=[],o=[];B(p,function(s){if(s.metadata&&s.metadata.result_type&&s.metadata.result_type=="popular"){r.push(s)}else{q.push(s)}});var n=r.concat(q);return n},_sortByLatest:function(n){this.results=n;this.results=this.results.slice(0,this.rpp);this.results=this._loadTopTweetsAtTop(this.results);this.results.reverse();return this},_sortByDefault:function(o){var p=this;var n=function(r){return new Date(r).getTime()};this.results.unshift.apply(this.results,o);B(this.results,function(r){if(!r.views){r.views=0}});this.results.sort(function(s,r){if(n(s.created_at)>n(r.created_at)){return -1}else{if(n(s.created_at)r.views){return 1}}return 0});if(!this._isLive){this.results.reverse()}},_prePlay:function(o){if(this.jsonRequestTimer){clearTimeout(this.jsonRequestTimer);this.jsonRequestTimer=null}if(!e.ie){a(this.scriptElement)}if(o.error){this.newResults=false}else{if(o.results&&o.results.length>0){this.response=o;this.newResults=true;this.sinceId=o.max_id_str;this._sortByMagic(o.results);if(this.isRunning()){this._play()}}else{if((this._isProfileWidget||this._isFavsWidget||this._isListWidget)&&I.array(o)&&o.length){this.newResults=true;if(!this._profileImage&&this._isProfileWidget){var n=o[0].user.screen_name;this.setProfileImage(o[0].user.profile_image_url);this.setTitle(o[0].user.name);this.setCaption(''+n+"")}this.sinceId=o[0].id_str;this._sortByMagic(o);if(this.isRunning()){this._play()}}else{this.newResults=false}}}this._setUrl();if(this._isLive){this.decay()}},_play:function(){var n=this;if(this.runOnce){this._hasNewSearchResults=true}if(this._avatars){this._preloadImages(this.results)}if(this._isRelativeTime&&(this._behavior=="all"||this._behavior=="preloaded")){B(this.byClass("twtr-timestamp","a",true),function(o){o.innerHTML=K(o.getAttribute("time"))})}if(!this._isLive||this._behavior=="all"||this._behavior=="preloaded"){B(this.results,function(p){if(p.retweeted_status){p=p.retweeted_status}if(n._isProfileWidget){p.from_user=p.user.screen_name;p.profile_image_url=p.user.profile_image_url}if(n._isFavsWidget||n._isListWidget){p.from_user=p.user.screen_name;p.profile_image_url=p.user.profile_image_url}p.id=p.id_str;n._createTweet({id:p.id,user:p.from_user,tweet:F.clean(p.text),avatar:p.profile_image_url,created_at:p.created_at,needle:p});var o=n.tweet.element;(n._behavior=="all")?n._appendSlideFade(o):n._appendTweet(o)});if(this._behavior!="preloaded"){return this}}return this},_normalizeTweet:function(o){var n=this;o.views++;if(this._isProfileWidget){o.from_user=n.username;o.profile_image_url=o.user.profile_image_url}if(this._isFavsWidget||this._isListWidget){o.from_user=o.user.screen_name;o.profile_image_url=o.user.profile_image_url}if(this._isFullScreen){o.profile_image_url=o.profile_image_url.replace(/_normal\./,"_bigger.")}this._createTweet({id:o.id,user:o.from_user,tweet:F.clean(o.text),avatar:o.profile_image_url,created_at:o.created_at,needle:o})._appendSlideFade()},_insertNewResultsNumber:function(){if(!this._hasNewSearchResults){this._hasNewSearchResults=false;return }if(this.runOnce&&this._isSearchWidget){var q=this.response.total>this.rpp?this.response.total:this.response.results.length;var n=q>1?"s":"";var p=(this.response.warning&&this.response.warning.match(/adjusted since_id/))?"more than":"";var o=document.createElement("div");b.add(o,"twtr-new-results");o.innerHTML='
     
     
    '+p+" "+q+" new tweet"+n+"";Z(o,this.byClass("twtr-reference-tweet","div"));this._hasNewSearchResults=false}},_preloadImages:function(n){if(this._isProfileWidget||this._isFavsWidget||this._isListWidget){B(n,function(p){var o=new Image();o.src=p.user.profile_image_url})}else{B(n,function(o){(new Image()).src=o.profile_image_url})}},_decayDecider:function(){var n=false;if(!this.runOnce){this.runOnce=true;n=true}else{if(this.newResults){n=true}}return n},start:function(){var n=this;if(!this._rendered){setTimeout(function(){n.start.call(n)},50);return this}if(!this._isLive){this._getResults()}else{this.occasionalJob.start()}this._isRunning=true;this._hasOfficiallyStarted=true;return this},stop:function(){this.occasionalJob.stop();if(this.intervalJob){this.intervalJob.stop()}this._isRunning=false;return this},pause:function(){if(this.isRunning()&&this.intervalJob){this.intervalJob.stop();b.add(this.widgetEl,"twtr-paused");this._isRunning=false}if(this._resumeTimer){clearTimeout(this._resumeTimer);this._resumeTimer=null}return this},resume:function(){var n=this;if(!this.isRunning()&&this._hasOfficiallyStarted&&this.intervalJob){this._resumeTimer=window.setTimeout(function(){n.intervalJob.start();n._isRunning=true;b.remove(n.widgetEl,"twtr-paused")},2000)}return this},isRunning:function(){return this._isRunning},destroy:function(){this.stop();this.clear();this.runOnce=false;this._hasOfficiallyStarted=false;this._profileImage=false;this._isLive=true;this._tweetFilter=false;this._isScroll=false;this.newResults=false;this._isRunning=false;this.sinceId=1;this.results=[];this.showedResults=[];this.occasionalJob.destroy();if(this.jsonRequestRunning){clearTimeout(this.jsonRequestTimer)}b.remove(this.widgetEl,"twtr-scroll");this.removeEvents();return this}}}()})()})(); \ No newline at end of file diff --git a/docs/about.html b/docs/about.html deleted file mode 100644 index fb72969cc4..0000000000 --- a/docs/about.html +++ /dev/null @@ -1,215 +0,0 @@ - - - - - - - - - - About — gensim - - - - - - - - - - - - - - - - - - - - - - - - -
    -

    Navigation

    - -
    - - -
    -
    -

    Table Of Contents

    - - - - -
    -
    - - - - -
    -
    -
    -
    - -
    -

    About

    -
    -

    History

    -

    Gensim started off as a collection of various Python scripts for the Czech Digital Mathematics Library dml.cz in 2008, -where it served to generate a short list of the most similar articles to a given article (gensim = “generate similar”). -I also wanted to try these fancy “Latent Semantic Methods”, but the libraries that -realized the necessary computation were not much fun to work with.

    -

    Naturally, I set out to reinvent the wheel. Our 2010 LREC publication -describes the initial design decisions behind gensim (clarity, efficiency and scalability) -and is fairly representative of how gensim works even today.

    -

    Later versions of gensim improved this efficiency and scalability tremendously (in fact, -I made algorithmic scalability of distributional semantics the topic of my PhD thesis).

    -

    By now, gensim is—to my knowledge—the most robust, efficient and hassle-free piece -of software to realize unsupervised semantic modelling from plain text. It stands -in contrast to brittle homework-assignment-implementations that do not scale on one hand, -and robust java-esque projects that do scale, but only if you’re willing to sacrifice -several weeks of your, your technician’s as well as your local scientist’s time just to run “hello world”.

    -

    In 2011, I started using Github for source code hosting, -and the gensim website moved from my university hosting to its present domain.

    -
    -
    -

    Licensing

    -

    Gensim is licensed under the OSI-approved GNU LGPL license. -This means that it’s free for both personal and commercial use, but if you make any -modification to gensim that you distribute to other people, you have to disclose -the source code of these modifications.

    -

    Apart form that, you are free to redistribute gensim in any way you like, though you’re -not allowed to modify its license (doh!).

    -

    My intent here is, of course, to get more help and community involvement with the development of gensim. -The legalese is therefore less important to me than your input and contributions. -Contact me if LGPL doesn’t fit your bill but you’d still like to use it – we’ll work something out.

    -
    -

    See also

    -

    I also host a document similarity package gensim.simserver. This is a high-level -interface to gensim functionality, and offers transactional remote (web-based) -document similarity queries and indexing. It uses gensim to do the heavy lifting: -you don’t need the simserver to use gensim, but you do need gensim to use the simserver. -Note that unlike gensim, gensim.simserver is licensed under Affero GPL, -which is much more restrictive for inclusion in commercial projects.

    -
    -
    -
    -

    Contributors

    -

    Credit goes to all the people who contributed to gensim, be it in discussions, -ideas, code contributions or bug reports. -It’s really useful and motivating to get feedback, in any shape or form, so big thanks to you all!

    -

    Some honorable mentions are included in the CHANGELOG.txt.

    -
    -
    -

    Academic citing

    -

    Gensim has been used in many students’ final theses as well as research papers. When citing gensim, -please use this BibTeX entry:

    -
    @inproceedings{rehurek_lrec,
-      title = {{Software Framework for Topic Modelling with Large Corpora}},
-      author = {Radim {\v R}eh{\r u}{\v r}ek and Petr Sojka},
-      booktitle = {{Proceedings of the LREC 2010 Workshop on New
-           Challenges for NLP Frameworks}},
-      pages = {45--50},
-      year = 2010,
-      month = May,
-      day = 22,
-      publisher = {ELRA},
-      address = {Valletta, Malta},
-      note={\url{http://is.muni.cz/publication/884893/en}},
-      language={English}
-}
    -
    -
    -
    - - -
    -
    -
    -
    -
    - - -
    -

    Navigation

    - -
    - - -
    - © Copyright 2009-2012, Radim Řehůřek <radimrehurek(at)seznam.cz>. - Last updated on Sep 16, 2012. -
    - - \ No newline at end of file diff --git a/docs/apiref.html b/docs/apiref.html deleted file mode 100644 index b5d027d7f0..0000000000 --- a/docs/apiref.html +++ /dev/null @@ -1,190 +0,0 @@ - - - - - - - - - - API Reference — gensim - - - - - - - - - - - - - - - - - - - - - - - - - - -
    -

    Navigation

    - -
    - - -
    -
    -

    Previous topic

    -

    Experiments on the English Wikipedia

    -

    Next topic

    -

    interfaces – Core gensim interfaces

    - - -
    -
    - - - - -
    -
    -
    -
    - - - - -
    -
    -
    -
    -
    - - -
    -

    Navigation

    - -
    - - -
    - © Copyright 2009-2012, Radim Řehůřek <radimrehurek(at)seznam.cz>. - Last updated on Sep 16, 2012. -
    - - \ No newline at end of file diff --git a/docs/changes_080.html b/docs/changes_080.html deleted file mode 100644 index 6e3961af77..0000000000 --- a/docs/changes_080.html +++ /dev/null @@ -1,216 +0,0 @@ - - - - - - - - - - Change Set for 0.8.0 — gensim - - - - - - - - - - - - - - - - - - - - - - - - -
    -

    Navigation

    - -
    - - -
    -
    -

    Table Of Contents

    - - - - -
    -
    - - - - -
    -
    -
    -
    - -
    -

    Change Set for 0.8.0

    -

    Release 0.8.0 concludes the 0.7.x series, which was about API consolidation and performance. -In 0.8.x, I’d like to extend gensim with new functionality and features.

    -
    -

    Codestyle Changes

    -

    Codebase was modified to comply with PEP8: Style Guide for Python Code. -This means the 0.8.0 API is backward incompatible with the 0.7.x series.

    -

    That’s not as tragic as it sounds, gensim was almost there anyway. The changes are few and pretty straightforward:

    -
      -
    1. the numTopics parameter is now num_topics
      2. -
    2. addDocuments() method becomes add_documents()
      3. -
    3. toUtf8() => to_utf8()
      4. -
    4. ... you get the idea: replace camelCase with lowercase_with_underscores.
      5. -
    -

    If you stored a model that is affected by this to disk, you’ll need to rename its attributes manually:

    -
    >>> lsa = gensim.models.LsiModel.load('/some/path') # load old <0.8.0 model
->>> lsa.num_terms, lsa.num_topics = lsa.numTerms, lsa.numTopics # rename attributes
->>> del lsa.numTerms, lsa.numTopics # clean up old attributes (optional)
->>> lsa.save('/some/path') # save again to disk, as 0.8.0 compatible
-
    -
    -

    Only attributes (variables) need to be renamed; method names (functions) are not affected, due to the way pickle works.

    -
    -
    -

    Similarity Queries

    -

    Improved speed and scalability of similarity queries.

    -

    The Similarity class can now index corpora of arbitrary size more efficiently. -Internally, this is done by splitting the index into several smaller pieces (“shards”) that fit in RAM -and can be processed independently. In addition, documents can now be added to a Similarity index dynamically.

    -

    There is also a new way to query the similarity indexes:

    -
    >>> index = MatrixSimilarity(corpus) # create an index
->>> sims = index[document] # get cosine similarity of query "document" against every document in the index
->>> sims = index[chunk_of_documents] # new syntax!
-
    -
    -

    Advantage of the last line (querying multiple documents at the same time) is faster execution.

    -

    This faster execution is also utilized automatically for you if you’re using the for sims in index: ... syntax -(which returns pairwise similarities of documents in the index).

    -

    To see the speed-up on your machine, run python -m gensim.test.simspeed (and compare to my results here to see how your machine fares).

    -
    -

    Note

    -

    This current functionality of querying is as far as I wanted to get with gensim. -More optimizations and smarter indexing are certainly possible, but I’d like to -focus on other features now. Pull requests are still welcome though :)

    -
    -

    Check out the updated documentation of the similarity classes for more info.

    -
    -
    -

    Simplified Directory Structure

    -

    Instead of the java-esque ROOT_DIR/src/gensim directory structure of gensim, -the packages now reside directly in ROOT_DIR/gensim (no superfluous src). See the new structure on github.

    -
    -
    -

    Other changes (that you’re unlikely to notice unless you look)

    -
      -
    • Improved efficiency of lsi[corpus] transformations (documents are chunked internally for better performance).
      • -
    • Large matrices (numpy/scipy.sparse, in LsiModel, Similarity etc.) are now mmapped to/from disk when doing save/load. The cPickle approach used previously was too buggy and slow.
      • -
    • Renamed chunks parameter to chunksize (i.e. LsiModel(corpus, num_topics=100, chunksize=20000)). This better reflects its purpose: size of a chunk=number of documents to be processed at once.
      • -
    • Also improved memory efficiency of LSI and LDA model generation (again).
      • -
    • Removed SciPy 0.6 from the list of supported SciPi versions (need >=0.7 now).
      • -
    • Added more unit tests.
      • -
    • Several smaller fixes; see the commit history for full account.
      • -
    -
    -

    Future Directions?

    -

    If you have ideas or proposals for new features for 0.8.x, now is the time to let me know: -gensim mailing list.

    -
    -
    -
    - - -
    -
    -
    -
    -
    - - -
    -

    Navigation

    - -
    - - -
    - © Copyright 2009-2012, Radim Řehůřek <radimrehurek(at)seznam.cz>. - Last updated on Sep 16, 2012. -
    - - \ No newline at end of file diff --git a/docs/corpora/bleicorpus.html b/docs/corpora/bleicorpus.html deleted file mode 100644 index 42b78ed0b4..0000000000 --- a/docs/corpora/bleicorpus.html +++ /dev/null @@ -1,232 +0,0 @@ - - - - - - - - - - corpora.bleicorpus – Corpus in Blei’s LDA-C format — gensim - - - - - - - - - - - - - - - - - - - - - - - - - - - -
    -

    Navigation

    - -
    - - -
    -
    -

    Previous topic

    -

    matutils – Math utils

    -

    Next topic

    -

    corpora.dictionary – Construct word<->id mappings

    - - -
    -
    - - - - -
    -
    -
    -
    - -
    -

    corpora.bleicorpus – Corpus in Blei’s LDA-C format

    -

    Blei’s LDA-C format.

    -
    -
    -class gensim.corpora.bleicorpus.BleiCorpus(fname, fname_vocab=None)
    -

    Corpus in Blei’s LDA-C format.

    -

    The corpus is represented as two files: one describing the documents, and another -describing the mapping between words and their ids.

    -

    Each document is one line:

    -
    N fieldId1:fieldValue1 fieldId2:fieldValue2 ... fieldIdN:fieldValueN
    -
    -

    The vocabulary is a file with words, one word per line; word at line K has an -implicit id=K.

    -

    Initialize the corpus from a file.

    -

    fname_vocab is the file with vocabulary; if not specified, it defaults to -fname.vocab.

    -
    -
    -docbyoffset(offset)
    -

    Return the document stored at file position offset.

    -
    - -
    -
    -classmethod load(fname)
    -

    Load a previously saved object from file (also see save).

    -
    - -
    -
    -save(fname)
    -

    Save the object to file via pickling (also see load).

    -
    - -
    -
    -static save_corpus(fname, corpus, id2word=None)
    -

    Save a corpus in the LDA-C format.

    -

    There are actually two files saved: fname and fname.vocab, where -fname.vocab is the vocabulary file.

    -

    This function is automatically called by BleiCorpus.serialize; don’t -call it directly, call serialize instead.

    -
    - -
    -
    -classmethod serialize(serializer, fname, corpus, id2word=None, index_fname=None, progress_cnt=None, labels=None)
    -

    Iterate through the document stream corpus, saving the documents to fname -and recording byte offset of each document. Save the resulting index -structure to file index_fname (or fname.index is not set).

    -

    This relies on the underlying corpus class serializer providing (in -addition to standard iteration):

    -
      -
    • -
      save_corpus method that returns a sequence of byte offsets, one for
      -

      each saved document,

      -
      -
      -
      • -

    • the docbyoffset(offset) method, which returns a document -positioned at offset bytes within the persistent storage (file).

      -
      • -
    -

    Example:

    -
    >>> MmCorpus.serialize('test.mm', corpus)
->>> mm = MmCorpus('test.mm') # `mm` document stream now has random access
->>> print mm[42] # retrieve document no. 42, etc.
-
    -
    -
    - -
    - -
    - - -
    -
    -
    -
    -
    - - -
    -

    Navigation

    - -
    - - -
    - © Copyright 2009-2012, Radim Řehůřek <radimrehurek(at)seznam.cz>. - Last updated on Sep 16, 2012. -
    - - \ No newline at end of file diff --git a/docs/corpora/corpora.html b/docs/corpora/corpora.html deleted file mode 100644 index e4c164637a..0000000000 --- a/docs/corpora/corpora.html +++ /dev/null @@ -1,138 +0,0 @@ - - - - - - - - - - corpora – Package for corpora I/O — gensim - - - - - - - - - - - - - - - - - - - - - - - - -
    -

    Navigation

    - -
    - - -
    -
    - - -
    -
    - - - - -
    -
    -
    -
    - -
    -

    corpora – Package for corpora I/O

    -

    This package contains implementations of various streaming corpus I/O format.

    -
    - - -
    -
    -
    -
    -
    - - -
    -

    Navigation

    - -
    - - -
    - © Copyright 2009-2012, Radim Řehůřek <radimrehurek(at)seznam.cz>. - Last updated on Sep 16, 2012. -
    - - \ No newline at end of file diff --git a/docs/corpora/dictionary.html b/docs/corpora/dictionary.html deleted file mode 100644 index 229bf6965e..0000000000 --- a/docs/corpora/dictionary.html +++ /dev/null @@ -1,278 +0,0 @@ - - - - - - - - - - corpora.dictionary – Construct word<->id mappings — gensim - - - - - - - - - - - - - - - - - - - - - - - - - - - -
    -

    Navigation

    - -
    - - -
    -
    -

    Previous topic

    -

    corpora.bleicorpus – Corpus in Blei’s LDA-C format

    -

    Next topic

    -

    corpora.hashdictionary – Construct word<->id mappings

    - - -
    -
    - - - - -
    -
    -
    -
    - -
    -

    corpora.dictionary – Construct word<->id mappings

    -

    This module implements the concept of Dictionary – a mapping between words and -their integer ids.

    -

    Dictionaries can be created from a corpus and can later be pruned according to -document frequency (removing (un)common words via the Dictionary.filter_extremes() method), -save/loaded from disk (via Dictionary.save() and Dictionary.load() methods) etc.

    -
    -
    -class gensim.corpora.dictionary.Dictionary(documents=None)
    -

    Dictionary encapsulates the mapping between normalized words and their integer ids.

    -

    The main function is doc2bow, which converts a collection of words to its -bag-of-words representation: a list of (word_id, word_frequency) 2-tuples.

    -
    -
    -add_documents(documents)
    -

    Build dictionary from a collection of documents. Each document is a list -of tokens = tokenized and normalized utf-8 encoded strings.

    -

    This is only a convenience wrapper for calling doc2bow on each document -with allow_update=True.

    -
    >>> print Dictionary(["máma mele maso".split(), "ema má máma".split()])
-Dictionary(5 unique tokens)
-
    -
    -
    - -
    -
    -compactify()
    -

    Assign new word ids to all words.

    -

    This is done to make the ids more compact, e.g. after some tokens have -been removed via filter_tokens() and there are gaps in the id series. -Calling this method will remove the gaps.

    -
    - -
    -
    -doc2bow(document, allow_update=False, return_missing=False)
    -

    Convert document (a list of words) into the bag-of-words format = list -of (token_id, token_count) 2-tuples. Each word is assumed to be a -tokenized and normalized utf-8 encoded string. No further preprocessing -is done on the words in document; apply tokenization, stemming etc. before -calling this method.

    -

    If allow_update is set, then also update dictionary in the process: create -ids for new words. At the same time, update document frequencies – for -each word appearing in this document, increase its document frequency (self.dfs) -by one.

    -

    If allow_update is not set, this function is const, aka read-only.

    -
    - -
    -
    -filter_extremes(no_below=5, no_above=0.5, keep_n=100000)
    -

    Filter out tokens that appear in

    -
      -
    1. less than no_below documents (absolute number) or
      2. -
    2. more than no_above documents (fraction of total corpus size, not -absolute number).
      3. -
    3. after (1) and (2), keep only the first keep_n most frequent tokens (or -keep all if None).
      4. -
    -

    After the pruning, shrink resulting gaps in word ids.

    -

    Note: Due to the gap shrinking, the same word may have a different -word id before and after the call to this function!

    -
    - -
    -
    -filter_tokens(bad_ids=None, good_ids=None)
    -

    Remove the selected bad_ids tokens from all dictionary mappings, or, keep -selected good_ids in the mapping and remove the rest.

    -

    bad_ids and good_ids are collections of word ids to be removed.

    -
    - -
    -
    -static from_corpus(corpus)
    -

    Create Dictionary from an existing corpus. This can be useful if you only -have a term-document BOW matrix (represented by corpus), but not the -original text corpus.

    -

    This will scan the term-document count matrix for all word ids that -appear in it, then construct and return Dictionary which maps each -word_id -> str(word_id).

    -
    - -
    -
    -keys()
    -

    Return a list of all token ids.

    -
    - -
    -
    -classmethod load(fname)
    -

    Load a previously saved object from file (also see save).

    -
    - -
    -
    -static load_from_text(fname)
    -

    Load a previously stored Dictionary from a text file. -Mirror function to save_as_text.

    -
    - -
    -
    -save(fname)
    -

    Save the object to file via pickling (also see load).

    -
    - -
    -
    -save_as_text(fname)
    -

    Save this Dictionary to a text file, in format: -id[TAB]word_utf8[TAB]document frequency[NEWLINE].

    -

    Note: use save/load to store in binary format instead (pickle).

    -
    - -
    - -
    - - -
    -
    -
    -
    -
    - - -
    -

    Navigation

    - -
    - - -
    - © Copyright 2009-2012, Radim Řehůřek <radimrehurek(at)seznam.cz>. - Last updated on Sep 16, 2012. -
    - - \ No newline at end of file diff --git a/docs/corpora/indexedcorpus.html b/docs/corpora/indexedcorpus.html deleted file mode 100644 index 1123babac6..0000000000 --- a/docs/corpora/indexedcorpus.html +++ /dev/null @@ -1,244 +0,0 @@ - - - - - - - - - - corpora.indexedcorpus – Random access to corpus documents — gensim - - - - - - - - - - - - - - - - - - - - - - - - - - - -
    -

    Navigation

    - -
    - - -
    -
    -

    Previous topic

    -

    corpora.ucicorpus – Corpus in UCI bag-of-words format

    -

    Next topic

    -

    models.ldamodel – Latent Dirichlet Allocation

    - - -
    -
    - - - - -
    -
    -
    -
    - -
    -

    corpora.indexedcorpus – Random access to corpus documents

    -

    Indexed corpus is a mechanism for random-accessing corpora.

    -

    While the standard corpus interface in gensim allows iterating over corpus with -for doc in corpus: pass, indexed corpus allows accessing the documents with -corpus[docno] (in O(1) look-up time).

    -

    This functionality is achieved by storing an extra file (by default named the same -as the corpus file plus ‘.index’ suffix) that stores the byte offset of the beginning -of each document.

    -
    -
    -class gensim.corpora.indexedcorpus.IndexedCorpus(fname, index_fname=None)
    -

    Initialize this abstract base class, by loading a previously saved index -from index_fname (or fname.index if index_fname is not set). -This index will allow subclasses to support the corpus[docno] syntax -(random access to document #`docno` in O(1)).

    -
    >>> # save corpus in SvmLightCorpus format with an index
->>> corpus = [[(1, 0.5)], [(0, 1.0), (1, 2.0)]]
->>> gensim.corpora.SvmLightCorpus.serialize('testfile.svmlight', corpus)
->>> # load back as a document stream (*not* plain Python list)
->>> corpus_with_random_access = gensim.corpora.SvmLightCorpus('tstfile.svmlight')
->>> print corpus_with_random_access[1]
-[(0, 1.0), (1, 2.0)]
-
    -
    -
    -
    -classmethod load(fname)
    -

    Load a previously saved object from file (also see save).

    -
    - -
    -
    -save(fname)
    -

    Save the object to file via pickling (also see load).

    -
    - -
    -
    -static save_corpus(fname, corpus, id2word=None)
    -

    Save an existing corpus to disk.

    -

    Some formats also support saving the dictionary (feature_id->word mapping), -which can in this case be provided by the optional id2word parameter.

    -
    >>> MmCorpus.save_corpus('file.mm', corpus)
-
    -
    -

    Some corpora also support an index of where each document begins, so -that the documents on disk can be accessed in O(1) time (see the -corpora.IndexedCorpus base class). In this case, save_corpus is automatically -called internally by serialize, which does save_corpus plus saves the index -at the same time, so you want to store the corpus with:

    -
    >>> MmCorpus.serialize('file.mm', corpus) # stores index as well, allowing random access to individual documents
-
    -
    -

    Calling serialize() is preferred to calling save_corpus().

    -
    - -
    -
    -classmethod serialize(serializer, fname, corpus, id2word=None, index_fname=None, progress_cnt=None, labels=None)
    -

    Iterate through the document stream corpus, saving the documents to fname -and recording byte offset of each document. Save the resulting index -structure to file index_fname (or fname.index is not set).

    -

    This relies on the underlying corpus class serializer providing (in -addition to standard iteration):

    -
      -
    • -
      save_corpus method that returns a sequence of byte offsets, one for
      -

      each saved document,

      -
      -
      -
      • -

    • the docbyoffset(offset) method, which returns a document -positioned at offset bytes within the persistent storage (file).

      -
      • -
    -

    Example:

    -
    >>> MmCorpus.serialize('test.mm', corpus)
->>> mm = MmCorpus('test.mm') # `mm` document stream now has random access
->>> print mm[42] # retrieve document no. 42, etc.
-
    -
    -
    - -
    - -
    - - -
    -
    -
    -
    -
    - - -
    -

    Navigation

    - -
    - - -
    - © Copyright 2009-2012, Radim Řehůřek <radimrehurek(at)seznam.cz>. - Last updated on Sep 16, 2012. -
    - - \ No newline at end of file diff --git a/docs/corpora/lowcorpus.html b/docs/corpora/lowcorpus.html deleted file mode 100644 index c56ff70020..0000000000 --- a/docs/corpora/lowcorpus.html +++ /dev/null @@ -1,246 +0,0 @@ - - - - - - - - - - corpora.lowcorpus – Corpus in List-of-Words format — gensim - - - - - - - - - - - - - - - - - - - - - - - - - - - -
    -

    Navigation

    - -
    - - -
    -
    -

    Previous topic

    -

    corpora.hashdictionary – Construct word<->id mappings

    -

    Next topic

    -

    corpora.mmcorpus – Corpus in Matrix Market format

    - - -
    -
    - - - - -
    -
    -
    -
    - -
    -

    corpora.lowcorpus – Corpus in List-of-Words format

    -

    Corpus in GibbsLda++ format of List-Of-Words.

    -
    -
    -class gensim.corpora.lowcorpus.LowCorpus(fname, id2word=None, line2words=<function split_on_space at 0x10faabaa0>)
    -

    List_Of_Words corpus handles input in GibbsLda++ format.

    -

    Quoting http://gibbslda.sourceforge.net/#3.2_Input_Data_Format:

    -
    Both data for training/estimating the model and new data (i.e., previously
-unseen data) have the same format as follows:
-
-[M]
-[document1]
-[document2]
-...
-[documentM]
-
-in which the first line is the total number for documents [M]. Each line
-after that is one document. [documenti] is the ith document of the dataset
-that consists of a list of Ni words/terms.
-
-[documenti] = [wordi1] [wordi2] ... [wordiNi]
-
-in which all [wordij] (i=1..M, j=1..Ni) are text strings and they are separated
-by the blank character.
    -
    -

    Initialize the corpus from a file.

    -

    id2word and line2words are optional parameters. -If provided, id2word is a dictionary mapping between word_ids (integers) -and words (strings). If not provided, the mapping is constructed from -the documents.

    -

    line2words is a function which converts lines into tokens. Defaults to -simple splitting on spaces.

    -
    -
    -docbyoffset(offset)
    -

    Return the document stored at file position offset.

    -
    - -
    -
    -classmethod load(fname)
    -

    Load a previously saved object from file (also see save).

    -
    - -
    -
    -save(fname)
    -

    Save the object to file via pickling (also see load).

    -
    - -
    -
    -static save_corpus(fname, corpus, id2word=None)
    -

    Save a corpus in the List-of-words format.

    -

    This function is automatically called by LowCorpus.serialize; don’t -call it directly, call serialize instead.

    -
    - -
    -
    -classmethod serialize(serializer, fname, corpus, id2word=None, index_fname=None, progress_cnt=None, labels=None)
    -

    Iterate through the document stream corpus, saving the documents to fname -and recording byte offset of each document. Save the resulting index -structure to file index_fname (or fname.index is not set).

    -

    This relies on the underlying corpus class serializer providing (in -addition to standard iteration):

    -
      -
    • -
      save_corpus method that returns a sequence of byte offsets, one for
      -

      each saved document,

      -
      -
      -
      • -

    • the docbyoffset(offset) method, which returns a document -positioned at offset bytes within the persistent storage (file).

      -
      • -
    -

    Example:

    -
    >>> MmCorpus.serialize('test.mm', corpus)
->>> mm = MmCorpus('test.mm') # `mm` document stream now has random access
->>> print mm[42] # retrieve document no. 42, etc.
-
    -
    -
    - -
    - -
    - - -
    -
    -
    -
    -
    - - -
    -

    Navigation

    - -
    - - -
    - © Copyright 2009-2012, Radim Řehůřek <radimrehurek(at)seznam.cz>. - Last updated on Sep 16, 2012. -
    - - \ No newline at end of file diff --git a/docs/corpora/mmcorpus.html b/docs/corpora/mmcorpus.html deleted file mode 100644 index 406837fcb1..0000000000 --- a/docs/corpora/mmcorpus.html +++ /dev/null @@ -1,226 +0,0 @@ - - - - - - - - - - corpora.mmcorpus – Corpus in Matrix Market format — gensim - - - - - - - - - - - - - - - - - - - - - - - - - - - -
    -

    Navigation

    - -
    - - -
    -
    -

    Previous topic

    -

    corpora.lowcorpus – Corpus in List-of-Words format

    -

    Next topic

    -

    corpora.svmlightcorpus – Corpus in SVMlight format

    - - -
    -
    - - - - -
    -
    -
    -
    - -
    -

    corpora.mmcorpus – Corpus in Matrix Market format

    -

    Corpus in the Matrix Market format.

    -
    -
    -class gensim.corpora.mmcorpus.MmCorpus(fname)
    -

    Corpus in the Matrix Market format.

    -
    -
    -docbyoffset(offset)
    -

    Return document at file offset offset (in bytes)

    -
    - -
    -
    -classmethod load(fname)
    -

    Load a previously saved object from file (also see save).

    -
    - -
    -
    -save(fname)
    -

    Save the object to file via pickling (also see load).

    -
    - -
    -
    -static save_corpus(fname, corpus, id2word=None, progress_cnt=1000)
    -

    Save a corpus in the Matrix Market format to disk.

    -

    This function is automatically called by MmCorpus.serialize; don’t -call it directly, call serialize instead.

    -
    - -
    -
    -classmethod serialize(serializer, fname, corpus, id2word=None, index_fname=None, progress_cnt=None, labels=None)
    -

    Iterate through the document stream corpus, saving the documents to fname -and recording byte offset of each document. Save the resulting index -structure to file index_fname (or fname.index is not set).

    -

    This relies on the underlying corpus class serializer providing (in -addition to standard iteration):

    -
      -
    • -
      save_corpus method that returns a sequence of byte offsets, one for
      -

      each saved document,

      -
      -
      -
      • -

    • the docbyoffset(offset) method, which returns a document -positioned at offset bytes within the persistent storage (file).

      -
      • -
    -

    Example:

    -
    >>> MmCorpus.serialize('test.mm', corpus)
->>> mm = MmCorpus('test.mm') # `mm` document stream now has random access
->>> print mm[42] # retrieve document no. 42, etc.
-
    -
    -
    - -
    -
    -skip_headers(input_file)
    -

    Skip file headers that appear before the first document.

    -
    - -
    - -
    - - -
    -
    -
    -
    -
    - - -
    -

    Navigation

    - -
    - - -
    - © Copyright 2009-2012, Radim Řehůřek <radimrehurek(at)seznam.cz>. - Last updated on Sep 16, 2012. -
    - - \ No newline at end of file diff --git a/docs/corpora/svmlightcorpus.html b/docs/corpora/svmlightcorpus.html deleted file mode 100644 index 2cc2ab7e7a..0000000000 --- a/docs/corpora/svmlightcorpus.html +++ /dev/null @@ -1,237 +0,0 @@ - - - - - - - - - - corpora.svmlightcorpus – Corpus in SVMlight format — gensim - - - - - - - - - - - - - - - - - - - - - - - - - - - -
    -

    Navigation

    - -
    - - -
    -
    -

    Previous topic

    -

    corpora.mmcorpus – Corpus in Matrix Market format

    -

    Next topic

    -

    corpora.wikicorpus – Corpus from a Wikipedia dump

    - - -
    -
    - - - - -
    -
    -
    -
    - -
    -

    corpora.svmlightcorpus – Corpus in SVMlight format

    -

    Corpus in SVMlight format.

    -
    -
    -class gensim.corpora.svmlightcorpus.SvmLightCorpus(fname)
    -

    Corpus in SVMlight format.

    -

    Quoting http://svmlight.joachims.org/: -The input file contains the training examples. The first lines -may contain comments and are ignored if they start with #. Each of the following -lines represents one training example and is of the following format:

    -
    <line> .=. <target> <feature>:<value> <feature>:<value> ... <feature>:<value> # <info>
-<target> .=. +1 | -1 | 0 | <float>
-<feature> .=. <integer> | "qid"
-<value> .=. <float>
-<info> .=. <string>
    -
    -

    The “qid” feature (used for SVMlight ranking), if present, is ignored.

    -

    Although not mentioned in the specification above, SVMlight also expect its -feature ids to be 1-based (counting starts at 1). We convert features to 0-base -internally by decrementing all ids when loading a SVMlight input file, and -increment them again when saving as SVMlight.

    -

    Initialize the corpus from a file.

    -
    -
    -docbyoffset(offset)
    -

    Return the document stored at file position offset.

    -
    - -
    -
    -classmethod load(fname)
    -

    Load a previously saved object from file (also see save).

    -
    - -
    -
    -save(fname)
    -

    Save the object to file via pickling (also see load).

    -
    - -
    -
    -static save_corpus(fname, corpus, id2word=None, labels=False)
    -

    Save a corpus in the SVMlight format.

    -

    The SVMlight <target> class tag is set to 0 for all documents.

    -

    This function is automatically called by SvmLightCorpus.serialize; don’t -call it directly, call serialize instead.

    -
    - -
    -
    -classmethod serialize(serializer, fname, corpus, id2word=None, index_fname=None, progress_cnt=None, labels=None)
    -

    Iterate through the document stream corpus, saving the documents to fname -and recording byte offset of each document. Save the resulting index -structure to file index_fname (or fname.index is not set).

    -

    This relies on the underlying corpus class serializer providing (in -addition to standard iteration):

    -
      -
    • -
      save_corpus method that returns a sequence of byte offsets, one for
      -

      each saved document,

      -
      -
      -
      • -

    • the docbyoffset(offset) method, which returns a document -positioned at offset bytes within the persistent storage (file).

      -
      • -
    -

    Example:

    -
    >>> MmCorpus.serialize('test.mm', corpus)
->>> mm = MmCorpus('test.mm') # `mm` document stream now has random access
->>> print mm[42] # retrieve document no. 42, etc.
-
    -
    -
    - -
    - -
    - - -
    -
    -
    -
    -
    - - -
    -

    Navigation

    - -
    - - -
    - © Copyright 2009-2012, Radim Řehůřek <radimrehurek(at)seznam.cz>. - Last updated on Sep 16, 2012. -
    - - \ No newline at end of file diff --git a/docs/corpora/textcorpus.html b/docs/corpora/textcorpus.html deleted file mode 100644 index 46327deb2f..0000000000 --- a/docs/corpora/textcorpus.html +++ /dev/null @@ -1,238 +0,0 @@ - - - - - - - - - - corpora.textcorpus – Building corpora with dictionaries — gensim - - - - - - - - - - - - - - - - - - - - - - - - - - - -
    -

    Navigation

    - -
    - - -
    -
    -

    Previous topic

    -

    corpora.wikicorpus – Corpus from a Wikipedia dump

    -

    Next topic

    -

    corpora.ucicorpus – Corpus in UCI bag-of-words format

    - - -
    -
    - - - - -
    -
    -
    -
    - -
    -

    corpora.textcorpus – Building corpora with dictionaries

    -

    Text corpora usually reside on disk, as text files in one format or another -In a common scenario, we need to build a dictionary (a word->integer id -mapping), which is then used to construct sparse bag-of-word vectors -(= sequences of (word_id, word_weight) 2-tuples).

    -

    This module provides some code scaffolding to simplify this pipeline. For -example, given a corpus where each document is a separate line in file on disk, -you would override the TextCorpus.get_texts method to read one line=document -at a time, process it (lowercase, tokenize, whatever) and yield it as a sequence -of words.

    -

    Overriding get_texts is enough; you can then initialize the corpus with e.g. -MyTextCorpus(bz2.BZ2File(‘mycorpus.txt.bz2’)) and it will behave correctly like a -corpus of sparse vectors. The __iter__ methods is automatically set up, and -dictionary is automatically populated with all word->id mappings.

    -

    The resulting object can be used as input to all gensim models (TFIDF, LSI, ...), -serialized with any format (Matrix Market, SvmLight, Blei’s LDA-C format etc).

    -

    See the gensim.test.test_miislita.CorpusMiislita class for a simple example.

    -
    -
    -class gensim.corpora.textcorpus.TextCorpus(input=None)
    -

    Helper class to simplify the pipeline of getting bag-of-words vectors (= a -gensim corpus) from plain text.

    -

    This is an abstract base class: override the get_texts() method to match -your particular input.

    -

    Given a filename (or a file-like object) in constructor, the corpus object -will be automatically initialized with a dictionary in self.dictionary and -will support the iter corpus method. You must only provide a correct get_texts -implementation.

    -
    -
    -get_texts()
    -

    Iterate over the collection, yielding one document at a time. A document -is a sequence of words (strings) that can be fed into Dictionary.doc2bow.

    -

    Override this function to match your input (parse input files, do any -text preprocessing, lowercasing, tokenizing etc.). There will be no further -preprocessing of the words coming out of this function.

    -
    - -
    -
    -classmethod load(fname)
    -

    Load a previously saved object from file (also see save).

    -
    - -
    -
    -save(fname)
    -

    Save the object to file via pickling (also see load).

    -
    - -
    -
    -static save_corpus(fname, corpus, id2word=None)
    -

    Save an existing corpus to disk.

    -

    Some formats also support saving the dictionary (feature_id->word mapping), -which can in this case be provided by the optional id2word parameter.

    -
    >>> MmCorpus.save_corpus('file.mm', corpus)
-
    -
    -

    Some corpora also support an index of where each document begins, so -that the documents on disk can be accessed in O(1) time (see the -corpora.IndexedCorpus base class). In this case, save_corpus is automatically -called internally by serialize, which does save_corpus plus saves the index -at the same time, so you want to store the corpus with:

    -
    >>> MmCorpus.serialize('file.mm', corpus) # stores index as well, allowing random access to individual documents
-
    -
    -

    Calling serialize() is preferred to calling save_corpus().

    -
    - -
    - -
    -
    -gensim.corpora.textcorpus.getstream(input)
    -

    If input is a filename (string), return open(input). -If input is a file-like object, reset it to the beginning with input.seek(0).

    -
    - -
    - - -
    -
    -
    -
    -
    - - -
    -

    Navigation

    - -
    - - -
    - © Copyright 2009-2012, Radim Řehůřek <radimrehurek(at)seznam.cz>. - Last updated on Sep 16, 2012. -
    - - \ No newline at end of file diff --git a/docs/corpora/ucicorpus.html b/docs/corpora/ucicorpus.html deleted file mode 100644 index 06a4a90181..0000000000 --- a/docs/corpora/ucicorpus.html +++ /dev/null @@ -1,260 +0,0 @@ - - - - - - - - - - corpora.ucicorpus – Corpus in UCI bag-of-words format — gensim - - - - - - - - - - - - - - - - - - - - - - - - - - - -
    -

    Navigation

    - -
    - - -
    -
    -

    Previous topic

    -

    corpora.textcorpus – Building corpora with dictionaries

    -

    Next topic

    -

    corpora.indexedcorpus – Random access to corpus documents

    - - -
    -
    - - - - -
    -
    -
    -
    - -
    -

    corpora.ucicorpus – Corpus in UCI bag-of-words format

    -

    University of California, Irvine (UCI) Bag-of-Words format.

    -

    http://archive.ics.uci.edu/ml/datasets/Bag+of+Words

    -
    -
    -class gensim.corpora.ucicorpus.UciCorpus(fname, fname_vocab=None)
    -

    Corpus in the UCI bag-of-words format.

    -
    -
    -create_dictionary()
    -

    Utility method to generate gensim-style Dictionary directly from -the corpus and vocabulary data.

    -
    - -
    -
    -docbyoffset(offset)
    -

    Return document at file offset offset (in bytes)

    -
    - -
    -
    -classmethod load(fname)
    -

    Load a previously saved object from file (also see save).

    -
    - -
    -
    -save(fname)
    -

    Save the object to file via pickling (also see load).

    -
    - -
    -
    -static save_corpus(fname, corpus, id2word=None, progress_cnt=10000)
    -

    Save a corpus in the UCI Bag-of-Words format.

    -

    There are actually two files saved: fname and fname.vocab, where -fname.vocab is the vocabulary file.

    -

    This function is automatically called by UciCorpus.serialize; don’t -call it directly, call serialize instead.

    -
    - -
    -
    -classmethod serialize(serializer, fname, corpus, id2word=None, index_fname=None, progress_cnt=None, labels=None)
    -

    Iterate through the document stream corpus, saving the documents to fname -and recording byte offset of each document. Save the resulting index -structure to file index_fname (or fname.index is not set).

    -

    This relies on the underlying corpus class serializer providing (in -addition to standard iteration):

    -
      -
    • -
      save_corpus method that returns a sequence of byte offsets, one for
      -

      each saved document,

      -
      -
      -
      • -

    • the docbyoffset(offset) method, which returns a document -positioned at offset bytes within the persistent storage (file).

      -
      • -
    -

    Example:

    -
    >>> MmCorpus.serialize('test.mm', corpus)
->>> mm = MmCorpus('test.mm') # `mm` document stream now has random access
->>> print mm[42] # retrieve document no. 42, etc.
-
    -
    -
    - -
    - -
    -
    -class gensim.corpora.ucicorpus.UciWriter(fname)
    -

    Store a corpus in UCI Bag-of-Words format.

    -

    This corpus format is identical to MM format, except for -different file headers. There is no format line, and the first -three lines of the file contain number_docs, num_terms, and num_nnz, -one value per line.

    -

    This implementation is based on matutils.MmWriter, and works the same way.

    -
    -
    -update_headers(num_docs, num_terms, num_nnz)
    -

    Update headers with actual values.

    -
    - -
    -
    -write_headers()
    -

    Write blank header lines. Will be updated later, once corpus stats are known.

    -
    - -
    -
    -write_vector(docno, vector)
    -

    Write a single sparse vector to the file.

    -

    Sparse vector is any iterable yielding (field id, field value) pairs.

    -
    - -
    - -
    - - -
    -
    -
    -
    -
    - - -
    -

    Navigation

    - -
    - - -
    - © Copyright 2009-2012, Radim Řehůřek <radimrehurek(at)seznam.cz>. - Last updated on Sep 16, 2012. -
    - - \ No newline at end of file diff --git a/docs/corpora/wikicorpus.html b/docs/corpora/wikicorpus.html deleted file mode 100644 index caf0f5a00d..0000000000 --- a/docs/corpora/wikicorpus.html +++ /dev/null @@ -1,273 +0,0 @@ - - - - - - - - - - corpora.wikicorpus – Corpus from a Wikipedia dump — gensim - - - - - - - - - - - - - - - - - - - - - - - - - - - -
    -

    Navigation

    - -
    - - -
    -
    -

    Previous topic

    -

    corpora.svmlightcorpus – Corpus in SVMlight format

    -

    Next topic

    -

    corpora.textcorpus – Building corpora with dictionaries

    - - -
    -
    - - - - -
    -
    -
    -
    - -
    -

    corpora.wikicorpus – Corpus from a Wikipedia dump

    -

    Construct a corpus from a Wikipedia (or other MediaWiki-based) database dump.

    -

    If you have the pattern package installed, this module will use a fancy -lemmatization to get a lemma of each token (instead of plain alphabetic -tokenizer). The package is available at https://github.com/clips/pattern .

    -

    See scripts/process_wiki.py for a canned (example) script based on this -module.

    -
    -
    -class gensim.corpora.wikicorpus.WikiCorpus(fname, processes=None, lemmatize=True, dictionary=None)
    -

    Treat a wikipedia articles dump (*articles.xml.bz2) as a (read-only) corpus.

    -

    The documents are extracted on-the-fly, so that the whole (massive) dump -can stay compressed on disk.

    -
    >>> wiki = WikiCorpus('enwiki-20100622-pages-articles.xml.bz2') # create word->word_id mapping, takes almost 8h
->>> wiki.saveAsText('wiki_en_vocab200k') # another 8h, creates a file in MatrixMarket format plus file with id->word
-
    -
    -

    Initialize the corpus. Unless a dictionary is provided, this scans the -corpus once, to determine its vocabulary.

    -

    If pattern package is installed, use fancier shallow parsing to get -token lemmas. Otherwise, use simple regexp tokenization. You can override -this automatic logic by forcing the lemmatize parameter explicitly.

    -
    -
    -get_texts()
    -

    Iterate over the dump, returning text version of each article as a list -of tokens.

    -

    Only articles of sufficient length are returned (short articles & redirects -etc are ignored).

    -

    Note that this iterates over the texts; if you want vectors, just use -the standard corpus interface instead of this function:

    -
    >>> for vec in wiki_corpus:
->>>     print vec
-
    -
    -
    - -
    -
    -classmethod load(fname)
    -

    Load a previously saved object from file (also see save).

    -
    - -
    -
    -save(fname)
    -

    Save the object to file via pickling (also see load).

    -
    - -
    -
    -static save_corpus(fname, corpus, id2word=None)
    -

    Save an existing corpus to disk.

    -

    Some formats also support saving the dictionary (feature_id->word mapping), -which can in this case be provided by the optional id2word parameter.

    -
    >>> MmCorpus.save_corpus('file.mm', corpus)
-
    -
    -

    Some corpora also support an index of where each document begins, so -that the documents on disk can be accessed in O(1) time (see the -corpora.IndexedCorpus base class). In this case, save_corpus is automatically -called internally by serialize, which does save_corpus plus saves the index -at the same time, so you want to store the corpus with:

    -
    >>> MmCorpus.serialize('file.mm', corpus) # stores index as well, allowing random access to individual documents
-
    -
    -

    Calling serialize() is preferred to calling save_corpus().

    -
    - -
    - -
    -
    -gensim.corpora.wikicorpus.filter_wiki(raw)
    -

    Filter out wiki mark-up from raw, leaving only text. raw is either unicode -or utf-8 encoded string.

    -
    - -
    -
    -gensim.corpora.wikicorpus.process_article(args)
    -

    Parse a wikipedia article, returning its content as a list of tokens -(utf8-encoded strings).

    -
    - -
    -
    -gensim.corpora.wikicorpus.remove_file(s)
    -

    Remove the ‘File:’ and ‘Image:’ markup, keeping the file caption.

    -

    Return a copy of s with all the ‘File:’ and ‘Image:’ markup replaced by -their corresponding captions. See http://www.mediawiki.org/wiki/Help:Images -for the markup details.

    -
    - -
    -
    -gensim.corpora.wikicorpus.remove_template(s)
    -

    Remove template wikimedia markup.

    -

    Return a copy of s with all the wikimedia markup template removed. See -http://meta.wikimedia.org/wiki/Help:Template for wikimedia templates -details.

    -

    Note: Since template can be nested, it is difficult remove them using -regular expresssions.

    -
    - -
    -
    -gensim.corpora.wikicorpus.tokenize(content)
    -

    Tokenize a piece of text from wikipedia. The input string content is assumed -to be mark-up free (see filter_wiki()).

    -

    Return list of tokens as utf8 bytestrings. Ignore words shorted than 2 or longer -that 15 characters (not bytes!).

    -
    - -
    - - -
    -
    -
    -
    -
    - - -
    -

    Navigation

    - -
    - - -
    - © Copyright 2009-2012, Radim Řehůřek <radimrehurek(at)seznam.cz>. - Last updated on Sep 16, 2012. -
    - - \ No newline at end of file diff --git a/docs/dist_lda.html b/docs/dist_lda.html deleted file mode 100644 index f72cff64c8..0000000000 --- a/docs/dist_lda.html +++ /dev/null @@ -1,237 +0,0 @@ - - - - - - - - - - Distributed Latent Dirichlet Allocation — gensim - - - - - - - - - - - - - - - - - - - - - - - - - - - -
    -

    Navigation

    - -
    - - -
    -
    -

    Table Of Contents

    - - -

    Previous topic

    -

    Distributed Latent Semantic Analysis

    -

    Next topic

    -

    Experiments on the English Wikipedia

    - - -
    -
    - - - - -
    -
    -
    -
    - -
    -

    Distributed Latent Dirichlet Allocation

    -
    -

    Note

    -

    See Distributed Computing for an introduction to distributed computing in gensim.

    -
    -
    -

    Setting up the cluster

    -

    See the tutorial on Distributed Latent Semantic Analysis; setting up a cluster for LDA is completely -analogous, except you want to run lda_worker and lda_dispatcher scripts instead -of lsi_worker and lsi_dispatcher.

    -
    -
    -

    Running LDA

    -

    Run LDA like you normally would, but turn on the distributed=True constructor -parameter:

    -
    >>> # extract 100 LDA topics, using default parameters
->>> lda = LdaModel(corpus=mm, id2word=id2word, num_topics=100, distributed=True)
-using distributed version with 4 workers
-running online LDA training, 100 topics, 1 passes over the supplied corpus of 3199665 documets, updating model once every 40000 documents
-..
-
    -
    -

    In serial mode (no distribution), creating this online LDA model of Wikipedia -takes 10h56m on my laptop (OS X, C2D 2.53GHz, 4GB RAM with libVec). -In distributed mode with four workers (Linux, Xeons of 2Ghz, 4GB RAM -with ATLAS), the wallclock time taken drops to 3h20m.

    -

    To run standard batch LDA (no online updates of mini-batches) instead, you would similarly -call:

    -
    >>> lda = LdaModel(corpus=mm, id2word=id2token, num_topics=100, update_every=0, passes=20, distributed=True)
-using distributed version with 4 workers
-running batch LDA training, 100 topics, 20 passes over the supplied corpus of 3199665 documets, updating model once every 3199665 documents
-initializing workers
-iteration 0, dispatching documents up to #10000/3199665
-iteration 0, dispatching documents up to #20000/3199665
-...
-
    -
    -

    and then, some two days later:

    -
    iteration 19, dispatching documents up to #3190000/3199665
-iteration 19, dispatching documents up to #3199665/3199665
-reached the end of input; now waiting for all remaining jobs to finish
    -
    -
    >>> lda.print_topics(20)
-topic #0: 0.007*disease + 0.006*medical + 0.005*treatment + 0.005*cells + 0.005*cell + 0.005*cancer + 0.005*health + 0.005*blood + 0.004*patients + 0.004*drug
-topic #1: 0.024*king + 0.013*ii + 0.013*prince + 0.013*emperor + 0.008*duke + 0.008*empire + 0.007*son + 0.007*china + 0.007*dynasty + 0.007*iii
-topic #2: 0.031*film + 0.017*films + 0.005*movie + 0.005*directed + 0.004*man + 0.004*episode + 0.003*character + 0.003*cast + 0.003*father + 0.003*mother
-topic #3: 0.022*user + 0.012*edit + 0.009*wikipedia + 0.007*block + 0.007*my + 0.007*here + 0.007*edits + 0.007*blocked + 0.006*revert + 0.006*me
-topic #4: 0.045*air + 0.026*aircraft + 0.021*force + 0.018*airport + 0.011*squadron + 0.010*flight + 0.010*military + 0.008*wing + 0.007*aviation + 0.007*f
-topic #5: 0.025*sun + 0.022*star + 0.018*moon + 0.015*light + 0.013*stars + 0.012*planet + 0.011*camera + 0.010*mm + 0.009*earth + 0.008*lens
-topic #6: 0.037*radio + 0.026*station + 0.022*fm + 0.014*news + 0.014*stations + 0.014*channel + 0.013*am + 0.013*racing + 0.011*tv + 0.010*broadcasting
-topic #7: 0.122*image + 0.099*jpg + 0.046*file + 0.038*uploaded + 0.024*png + 0.014*contribs + 0.013*notify + 0.013*logs + 0.013*picture + 0.013*flag
-topic #8: 0.036*russian + 0.030*soviet + 0.028*polish + 0.024*poland + 0.022*russia + 0.013*union + 0.012*czech + 0.011*republic + 0.011*moscow + 0.010*finland
-topic #9: 0.031*language + 0.014*word + 0.013*languages + 0.009*term + 0.009*words + 0.008*example + 0.007*names + 0.007*meaning + 0.006*latin + 0.006*form
-topic #10: 0.029*w + 0.029*toronto + 0.023*l + 0.020*hockey + 0.019*nhl + 0.014*ontario + 0.012*calgary + 0.011*edmonton + 0.011*hamilton + 0.010*season
-topic #11: 0.110*wikipedia + 0.110*articles + 0.030*library + 0.029*wikiproject + 0.028*project + 0.019*data + 0.016*archives + 0.012*needing + 0.009*reference + 0.009*statements
-topic #12: 0.032*http + 0.030*your + 0.022*request + 0.017*sources + 0.016*archived + 0.016*modify + 0.015*changes + 0.015*creation + 0.014*www + 0.013*try
-topic #13: 0.011*your + 0.010*my + 0.009*we + 0.008*don + 0.008*get + 0.008*know + 0.007*me + 0.006*think + 0.006*question + 0.005*find
-topic #14: 0.073*r + 0.066*japanese + 0.062*japan + 0.018*tokyo + 0.008*prefecture + 0.005*osaka + 0.004*j + 0.004*sf + 0.003*kyoto + 0.003*manga
-topic #15: 0.045*da + 0.045*fr + 0.027*kategori + 0.026*pl + 0.024*nl + 0.021*pt + 0.017*en + 0.015*categoria + 0.014*es + 0.012*kategorie
-topic #16: 0.010*death + 0.005*died + 0.005*father + 0.004*said + 0.004*himself + 0.004*took + 0.004*son + 0.004*killed + 0.003*murder + 0.003*wife
-topic #17: 0.027*book + 0.021*published + 0.020*books + 0.014*isbn + 0.010*author + 0.010*magazine + 0.009*press + 0.009*novel + 0.009*writers + 0.008*story
-topic #18: 0.027*football + 0.024*players + 0.023*cup + 0.019*club + 0.017*fc + 0.017*footballers + 0.017*league + 0.011*season + 0.007*teams + 0.007*goals
-topic #19: 0.032*band + 0.024*album + 0.014*albums + 0.013*guitar + 0.013*rock + 0.011*records + 0.011*vocals + 0.009*live + 0.008*bass + 0.008*track
-
    -
    -

    If you used the distributed LDA implementation in gensim, please let me know (my -email is at the bottom of this page). I would like to hear about your application and -the possible (inevitable?) issues that you encountered, to improve gensim in the future.

    -
    -
    - - -
    -
    -
    -
    -
    - - -
    -

    Navigation

    - -
    - - -
    - © Copyright 2009-2012, Radim Řehůřek <radimrehurek(at)seznam.cz>. - Last updated on Sep 16, 2012. -
    - - \ No newline at end of file diff --git a/docs/dist_lsi.html b/docs/dist_lsi.html deleted file mode 100644 index a8d2d0f3e6..0000000000 --- a/docs/dist_lsi.html +++ /dev/null @@ -1,310 +0,0 @@ - - - - - - - - - - Distributed Latent Semantic Analysis — gensim - - - - - - - - - - - - - - - - - - - - - - - - - - - -
    -

    Navigation

    - -
    - - -
    -
    -

    Table Of Contents

    - - -

    Previous topic

    -

    Distributed Computing

    -

    Next topic

    -

    Distributed Latent Dirichlet Allocation

    - - -
    -
    - - - - -
    -
    -
    -
    - -
    -

    Distributed Latent Semantic Analysis

    -
    -

    Note

    -

    See Distributed Computing for an introduction to distributed computing in gensim.

    -
    -
    -

    Setting up the cluster

    -

    We will show how to run distributed Latent Semantic Analysis by means of an example. -Let’s say we have 5 computers at our disposal, all on the same network segment (=reachable -by network broadcast). To start with, install gensim and Pyro on each computer with:

    -
    $ sudo easy_install gensim[distributed]
    -
    -

    and run Pyro’s name server on exactly one of the machines (doesn’t matter which one):

    -
    $ python -m Pyro4.naming -n 0.0.0.0 &
    -
    -

    Let’s say our example cluster consists of dual-core computers with loads of -memory. We will therefore run two worker scripts on four of the physical machines, -creating eight logical worker nodes:

    -
    $ python -m gensim.models.lsi_worker &
    -
    -

    This will execute gensim‘s lsi_worker.py script (to be run twice on each of the -four computer). -This lets gensim know that it can run two jobs on each of the four computers in -parallel, so that the computation will be done faster, while also taking up twice -as much memory on each machine.

    -

    Next, pick one computer that will be a job scheduler in charge of worker -synchronization, and on it, run LSA dispatcher. In our example, we will use the -fifth computer to act as the dispatcher and from there run:

    -
    $ python -m gensim.models.lsi_dispatcher &
    -
    -

    In general, the dispatcher can be run on the same machine as one of the worker nodes, or it -can be another, distinct computer (within the same broadcast domain). The dispatcher -won’t be doing much with CPU most of the time, but pick a computer with ample memory.

    -

    And that’s it! The cluster is set up and running, ready to accept jobs. To remove -a worker later on, simply terminate its lsi_worker process. To add another worker, run another -lsi_worker (this will not affect a computation that is already running, the additions/deletions are not dynamic). -If you terminate lsi_dispatcher, you won’t be able to run computations until you run it again -(surviving worker processes can be re-used though).

    -
    -
    -

    Running LSA

    -

    So let’s test our setup and run one computation of distributed LSA. Open a Python -shell on one of the five machines (again, this can be done on any computer -in the same broadcast domain, -our choice is incidental) and try:

    -
    >>> from gensim import corpora, models, utils
->>> import logging
->>> logging.basicConfig(format='%(asctime)s : %(levelname)s : %(message)s', level=logging.INFO)
-
->>> corpus = corpora.MmCorpus('/tmp/deerwester.mm') # load a corpus of nine documents, from the Tutorials
->>> id2word = corpora.Dictionary.load('/tmp/deerwester.dict')
-
->>> lsi = models.LsiModel(corpus, id2word=id2word, num_topics=200, chunksize=1, distributed=True) # run distributed LSA on nine documents
-
    -
    -

    This uses the corpus and feature-token mapping created in the Corpora and Vector Spaces tutorial. -If you look at the log in your Python session, you should see a line similar to:

    -
    2010-08-09 23:44:25,746 : INFO : using distributed version with 8 workers
    -
    -

    which means all went well. You can also check the logs coming from your worker and dispatcher -processes — this is especially helpful in case of problems. -To check the LSA results, let’s print the first two latent topics:

    -
    >>> lsi.print_topics(num_topics=2, num_words=5)
-topic #0(3.341): 0.644*"system" + 0.404*"user" + 0.301*"eps" + 0.265*"time" + 0.265*"response"
-topic #1(2.542): 0.623*"graph" + 0.490*"trees" + 0.451*"minors" + 0.274*"survey" + -0.167*"system"
-
    -
    -

    Success! But a corpus of nine documents is no challenge for our powerful cluster... -In fact, we had to lower the job size (chunksize parameter above) to a single document -at a time, otherwise all documents would be processed by a single worker all at once.

    -

    So let’s run LSA on one million documents instead:

    -
    >>> # inflate the corpus to 1M documents, by repeating its documents over&over
->>> corpus1m = utils.RepeatCorpus(corpus, 1000000)
->>> # run distributed LSA on 1 million documents
->>> lsi1m = models.LsiModel(corpus1m, id2word=id2word, num_topics=200, chunksize=10000, distributed=True)
-
->>> lsi1m.print_topics(num_topics=2, num_words=5)
-topic #0(1113.628): 0.644*"system" + 0.404*"user" + 0.301*"eps" + 0.265*"time" + 0.265*"response"
-topic #1(847.233): 0.623*"graph" + 0.490*"trees" + 0.451*"minors" + 0.274*"survey" + -0.167*"system"
-
    -
    -

    The log from 1M LSA should look like:

    -
    2010-08-10 02:46:35,087 : INFO : using distributed version with 8 workers
-2010-08-10 02:46:35,087 : INFO : updating SVD with new documents
-2010-08-10 02:46:35,202 : INFO : dispatched documents up to #10000
-2010-08-10 02:46:35,296 : INFO : dispatched documents up to #20000
-...
-2010-08-10 02:46:46,524 : INFO : dispatched documents up to #990000
-2010-08-10 02:46:46,694 : INFO : dispatched documents up to #1000000
-2010-08-10 02:46:46,694 : INFO : reached the end of input; now waiting for all remaining jobs to finish
-2010-08-10 02:46:47,195 : INFO : all jobs finished, downloading final projection
-2010-08-10 02:46:47,200 : INFO : decomposition complete
    -
    -

    Due to the small vocabulary size and trivial structure of our “one-million corpus”, the computation -of LSA still takes only 12 seconds. To really stress-test our cluster, let’s do -Latent Semantic Analysis on the English Wikipedia.

    -
    -

    Distributed LSA on Wikipedia

    -

    First, download and prepare the Wikipedia corpus as per Experiments on the English Wikipedia, then load -the corpus iterator with:

    -
    >>> import logging, gensim, bz2
->>> logging.basicConfig(format='%(asctime)s : %(levelname)s : %(message)s', level=logging.INFO)
-
->>> # load id->word mapping (the dictionary)
->>> id2word = gensim.corpora.Dictionary.load_from_text('wiki_en_wordids.txt')
->>> # load corpus iterator
->>> mm = gensim.corpora.MmCorpus('wiki_en_tfidf.mm')
->>> # mm = gensim.corpora.MmCorpus(bz2.BZ2File('wiki_en_tfidf.mm.bz2')) # use this if you compressed the TFIDF output
-
->>> print mm
-MmCorpus(3199665 documents, 100000 features, 495547400 non-zero entries)
-
    -
    -

    Now we’re ready to run distributed LSA on the English Wikipedia:

    -
    >>> # extract 400 LSI topics, using a cluster of nodes
->>> lsi = gensim.models.lsimodel.LsiModel(corpus=mm, id2word=id2word, num_topics=400, chunksize=20000, distributed=True)
-
->>> # print the most contributing words (both positively and negatively) for each of the first ten topics
->>> lsi.print_topics(10)
-2010-11-03 16:08:27,602 : INFO : topic #0(200.990): -0.475*"delete" + -0.383*"deletion" + -0.275*"debate" + -0.223*"comments" + -0.220*"edits" + -0.213*"modify" + -0.208*"appropriate" + -0.194*"subsequent" + -0.155*"wp" + -0.117*"notability"
-2010-11-03 16:08:27,626 : INFO : topic #1(143.129): -0.320*"diff" + -0.305*"link" + -0.199*"image" + -0.171*"www" + -0.162*"user" + 0.149*"delete" + -0.147*"undo" + -0.144*"contribs" + -0.122*"album" + 0.113*"deletion"
-2010-11-03 16:08:27,651 : INFO : topic #2(135.665): -0.437*"diff" + -0.400*"link" + -0.202*"undo" + -0.192*"user" + -0.182*"www" + -0.176*"contribs" + 0.168*"image" + -0.109*"added" + 0.106*"album" + 0.097*"copyright"
-2010-11-03 16:08:27,677 : INFO : topic #3(125.027): -0.354*"image" + 0.239*"age" + 0.218*"median" + -0.213*"copyright" + 0.204*"population" + -0.195*"fair" + 0.195*"income" + 0.167*"census" + 0.165*"km" + 0.162*"households"
-2010-11-03 16:08:27,701 : INFO : topic #4(116.927): -0.307*"image" + 0.195*"players" + -0.184*"median" + -0.184*"copyright" + -0.181*"age" + -0.167*"fair" + -0.162*"income" + -0.151*"population" + -0.136*"households" + -0.134*"census"
-2010-11-03 16:08:27,728 : INFO : topic #5(100.326): 0.501*"players" + 0.318*"football" + 0.284*"league" + 0.193*"footballers" + 0.142*"image" + 0.133*"season" + 0.119*"cup" + 0.113*"club" + 0.110*"baseball" + 0.103*"f"
-2010-11-03 16:08:27,754 : INFO : topic #6(92.298): -0.411*"album" + -0.275*"albums" + -0.217*"band" + -0.214*"song" + -0.184*"chart" + -0.163*"songs" + -0.160*"singles" + -0.149*"vocals" + -0.139*"guitar" + -0.129*"track"
-2010-11-03 16:08:27,780 : INFO : topic #7(83.811): -0.248*"wikipedia" + -0.182*"keep" + 0.180*"delete" + -0.167*"articles" + -0.152*"your" + -0.150*"my" + 0.144*"film" + -0.130*"we" + -0.123*"think" + -0.120*"user"
-2010-11-03 16:08:27,807 : INFO : topic #8(78.981): 0.588*"film" + 0.460*"films" + -0.130*"album" + -0.127*"station" + 0.121*"television" + 0.115*"poster" + 0.112*"directed" + 0.110*"actors" + -0.096*"railway" + 0.086*"movie"
-2010-11-03 16:08:27,834 : INFO : topic #9(78.620): 0.502*"kategori" + 0.282*"categoria" + 0.248*"kategorija" + 0.234*"kategorie" + 0.172*"категория" + 0.165*"categoría" + 0.161*"kategoria" + 0.148*"categorie" + 0.126*"kategória" + 0.121*"catégorie"
-
    -
    -

    In serial mode, creating the LSI model of Wikipedia with this one-pass algorithm -takes about 5.25h on my laptop (OS X, C2D 2.53GHz, 4GB RAM with libVec). -In distributed mode with four workers (Linux, dual-core Xeons of 2Ghz, 4GB RAM -with ATLAS), the wallclock time taken drops to 1 hour and 41 minutes. You can -read more about various internal settings and experiments in my research -paper.

    -
    -
    -
    - - -
    -
    -
    -
    -
    - - -
    -

    Navigation

    - -
    - - -
    - © Copyright 2009-2012, Radim Řehůřek <radimrehurek(at)seznam.cz>. - Last updated on Sep 16, 2012. -
    - - \ No newline at end of file diff --git a/docs/distributed.html b/docs/distributed.html deleted file mode 100644 index 90b7be2939..0000000000 --- a/docs/distributed.html +++ /dev/null @@ -1,245 +0,0 @@ - - - - - - - - - - Distributed Computing — gensim - - - - - - - - - - - - - - - - - - - - - - - - - - -
    -

    Navigation

    - -
    - - -
    -
    -

    Table Of Contents

    - - -

    Previous topic

    -

    Distributed Latent Dirichlet Allocation

    -

    Next topic

    -

    Distributed Latent Semantic Analysis

    - - -
    -
    - - - - -
    -
    -
    -
    - -
    -

    Distributed Computing

    -
    -

    Why distributed computing?

    -

    Need to build semantic representation of a corpus that is millions of documents large and it’s -taking forever? Have several idle machines at your disposal that you could use? -Distributed computing tries -to accelerate computations by splitting a given task into several smaller subtasks, -passing them on to several computing nodes in parallel.

    -

    In the context of gensim, computing nodes are computers identified by their IP address/port, -and communication happens over TCP/IP. The whole collection of available machines is called -a cluster. The distribution is very coarse grained (not -much communication going on), so the network is allowed to be of relatively high latency.

    -
    -

    Warning

    -

    The primary reason for using distributed computing is making things run faster. In gensim, -most of the time consuming stuff is done inside low-level routines for linear algebra, inside -NumPy, independent of any gensim code. -Installing a fast BLAS (Basic Linear Algebra) library -for NumPy can improve performance up to 15 times! So before you start buying those extra computers, -consider installing a fast, threaded BLAS that is optimized for your particular machine -(as opposed to a generic, binary-distributed library). -Options include your vendor’s BLAS library (Intel’s MKL, -AMD’s ACML, OS X’s vecLib, Sun’s Sunperf, ...) or some open-source alternative (GotoBLAS, ALTAS).

    -

    To see what BLAS and LAPACK you are using, type into your shell:

    -
    python -c 'import numpy; numpy.show_config()'
    -
    -
    -
    -
    -

    Prerequisites

    -

    For communication between nodes, gensim uses Pyro (PYthon Remote Objects), version >= 4.8. This is a library for low-level socket communication -and remote procedure calls (RPC) in Python. Pyro is a pure-Python library, so its -installation is quite painless and only involves copying its *.py files somewhere onto your Python’s import path:

    -
    sudo easy_install Pyro4
    -
    -

    You don’t have to install Pyro to run gensim, but if you don’t, you won’t be able -to access the distributed features (i.e., everything will always run in serial mode, -the examples on this page don’t apply).

    -
    -
    -

    Core concepts

    -

    As always, gensim strives for a clear and straightforward API (see Features). -To this end, you do not need to make any changes in your code at all in order to -run it over a cluster of computers!

    -

    What you need to do is run a worker script (see below) on each of your cluster nodes prior -to starting your computation. Running this script tells gensim that it may use the node -as a slave to delegate some work to it. During initialization, the algorithms -inside gensim will try to look for and enslave all available worker nodes.

    -
    -
    Node
    -
    A logical working unit. Can correspond to a single physical machine, but you -can also run multiple workers on one machine, resulting in multiple -logical nodes.
    -
    Cluster
    -
    Several nodes which communicate over TCP/IP. Currently, network broadcasting -is used to discover and connect all communicating nodes, so the nodes must lie -within the same broadcast domain.
    -
    Worker
    -
    A process which is created on each node. To remove a node from your cluster, -simply kill its worker process.
    -
    Dispatcher
    -
    The dispatcher will be in charge of negotiating all computations, queueing and -distributing (“dispatching”) individual jobs to the workers. Computations never -“talk” to worker nodes directly, only through this dispatcher. Unlike workers, -there can only be one active dispatcher at a time in the cluster.
    -
    -
    - -
    - - -
    -
    -
    -
    -
    - - -
    -

    Navigation

    - -
    - - -
    - © Copyright 2009-2012, Radim Řehůřek <radimrehurek(at)seznam.cz>. - Last updated on Sep 16, 2012. -
    - - \ No newline at end of file diff --git a/docs/genindex.html b/docs/genindex.html deleted file mode 100644 index 9d39f50833..0000000000 --- a/docs/genindex.html +++ /dev/null @@ -1,1638 +0,0 @@ - - - - - - - - - - - - Index — gensim - - - - - - - - - - - - - -
    -

    Navigation

    - -
    -
    -
    - - - - - -
    -
    - -
    -
    -
    -
    - - -

    Index

    - -
    - A - | B - | C - | D - | E - | F - | G - | H - | I - | J - | K - | L - | M - | N - | O - | P - | Q - | R - | S - | T - | U - | V - | W - -
    -

    A

    - - - -
    - -
    add_documents() (gensim.corpora.dictionary.Dictionary method) -
    - -
    - -
    (gensim.corpora.hashdictionary.HashDictionary method) -
    - - -
    (gensim.models.lsimodel.LsiModel method) -
    - - -
    (gensim.similarities.docsim.Similarity method) -
    - -
    - -
    any2unicode() (in module gensim.utils) -
    - -
    - -
    any2utf8() (in module gensim.utils) -
    - -
    - -

    B

    - - - -
    - -
    BleiCorpus (class in gensim.corpora.bleicorpus) -
    - - -
    blend() (gensim.models.ldamodel.LdaState method) -
    - - -
    blend2() (gensim.models.ldamodel.LdaState method) -
    - -
    - -
    bound() (gensim.models.ldamodel.LdaModel method) -
    - - -
    buffer() (simserver.simserver.SessionServer method) -
    - -
    - -
    (simserver.simserver.SimServer method) -
    - -
    -
    - -

    C

    - - - -
    - -
    check_session() (simserver.simserver.SessionServer method) -
    - - -
    chunkize() (in module gensim.utils) -
    - - -
    chunkize_serial() (in module gensim.utils) -
    - - -
    clear() (gensim.models.ldamodel.LdaModel method) -
    - - -
    clip_spectrum() (in module gensim.models.lsimodel) -
    - - -
    close() (simserver.simserver.SessionServer method) -
    - -
    - -
    (simserver.simserver.SimIndex method) -
    - - -
    (simserver.simserver.SimModel method) -
    - - -
    (simserver.simserver.SimServer method) -
    - -
    - -
    close_shard() (gensim.similarities.docsim.Similarity method) -
    - - -
    Cluster -
    - -
    - -
    commit() (simserver.simserver.SessionServer method) -
    - - -
    compactify() (gensim.corpora.dictionary.Dictionary method) -
    - - -
    copytree_hardlink() (in module gensim.utils) -
    - - -
    Corpus -
    - - -
    corpus2csc() (in module gensim.matutils) -
    - - -
    corpus2dense() (in module gensim.matutils) -
    - - -
    CorpusABC (class in gensim.interfaces) -
    - - -
    create_dictionary() (gensim.corpora.ucicorpus.UciCorpus method) -
    - -
    - -

    D

    - - - -
    - -
    deaccent() (in module gensim.utils) -
    - - -
    decode_htmlentities() (in module gensim.utils) -
    - - -
    delete() (simserver.simserver.SessionServer method) -
    - -
    - -
    (simserver.simserver.SimIndex method) -
    - - -
    (simserver.simserver.SimServer method) -
    - -
    - -
    Dense2Corpus (class in gensim.matutils) -
    - - -
    dense2vec() (in module gensim.matutils) -
    - - -
    df2idf() (in module gensim.models.tfidfmodel) -
    - - -
    dict_from_corpus() (in module gensim.utils) -
    - - -
    Dictionary (class in gensim.corpora.dictionary) -
    - - -
    dirichlet_expectation() (in module gensim.models.hdpmodel) -
    - -
    - -
    (in module gensim.models.ldamodel) -
    - -
    -
    - -
    Dispatcher -
    - -
    - -
    (class in gensim.models.lda_dispatcher) -
    - - -
    (class in gensim.models.lsi_dispatcher) -
    - -
    - -
    do_estep() (gensim.models.ldamodel.LdaModel method) -
    - - -
    do_mstep() (gensim.models.ldamodel.LdaModel method) -
    - - -
    doc2bow() (gensim.corpora.dictionary.Dictionary method) -
    - -
    - -
    (gensim.corpora.hashdictionary.HashDictionary method) -
    - -
    - -
    doc2vec() (simserver.simserver.SimModel method) -
    - - -
    doc_e_step() (gensim.models.hdpmodel.HdpModel method) -
    - - -
    docbyoffset() (gensim.corpora.bleicorpus.BleiCorpus method) -
    - -
    - -
    (gensim.corpora.lowcorpus.LowCorpus method) -
    - - -
    (gensim.corpora.mmcorpus.MmCorpus method) -
    - - -
    (gensim.corpora.svmlightcorpus.SvmLightCorpus method) -
    - - -
    (gensim.corpora.ucicorpus.UciCorpus method) -
    - - -
    (gensim.matutils.MmReader method) -
    - -
    - -
    docs2vecs() (simserver.simserver.SimModel method) -
    - - -
    drop_index() (simserver.simserver.SessionServer method) -
    - -
    - -
    (simserver.simserver.SimServer method) -
    - -
    -
    - -

    E

    - - - -
    - -
    exit() (gensim.models.lda_dispatcher.Dispatcher method) -
    - -
    - -
    (gensim.models.lsi_dispatcher.Dispatcher method) -
    - -
    -
    - -
    expect_log_sticks() (in module gensim.models.hdpmodel) -
    - -
    - -

    F

    - - - -
    - -
    FakeDict (class in gensim.utils) -
    - - -
    filter_extremes() (gensim.corpora.dictionary.Dictionary method) -
    - -
    - -
    (gensim.corpora.hashdictionary.HashDictionary method) -
    - -
    - -
    filter_tokens() (gensim.corpora.dictionary.Dictionary method) -
    - - -
    filter_wiki() (in module gensim.corpora.wikicorpus) -
    - - -
    find_similar() (simserver.simserver.SessionServer method) -
    - -
    - -
    (simserver.simserver.SimServer method) -
    - -
    -
    - -
    flush() (simserver.simserver.SimServer method) -
    - - -
    from_corpus() (gensim.corpora.dictionary.Dictionary static method) -
    - - -
    full2sparse() (in module gensim.matutils) -
    - - -
    full2sparse_clipped() (in module gensim.matutils) -
    - -
    - -

    G

    - - - -
    - -
    gensim.corpora (module) -
    - - -
    gensim.corpora.bleicorpus (module) -
    - - -
    gensim.corpora.dictionary (module) -
    - - -
    gensim.corpora.hashdictionary (module) -
    - - -
    gensim.corpora.indexedcorpus (module) -
    - - -
    gensim.corpora.lowcorpus (module) -
    - - -
    gensim.corpora.mmcorpus (module) -
    - - -
    gensim.corpora.svmlightcorpus (module) -
    - - -
    gensim.corpora.textcorpus (module) -
    - - -
    gensim.corpora.ucicorpus (module) -
    - - -
    gensim.corpora.wikicorpus (module) -
    - - -
    gensim.interfaces (module) -
    - - -
    gensim.matutils (module) -
    - - -
    gensim.models (module) -
    - - -
    gensim.models.hdpmodel (module) -
    - - -
    gensim.models.lda_dispatcher (module) -
    - - -
    gensim.models.lda_worker (module) -
    - - -
    gensim.models.ldamodel (module) -
    - -
    - -
    gensim.models.logentropy_model (module) -
    - - -
    gensim.models.lsi_dispatcher (module) -
    - - -
    gensim.models.lsi_worker (module) -
    - - -
    gensim.models.lsimodel (module) -
    - - -
    gensim.models.rpmodel (module) -
    - - -
    gensim.models.tfidfmodel (module) -
    - - -
    gensim.similarities.docsim (module) -
    - - -
    gensim.utils (module) -
    - - -
    get_document_id() (gensim.similarities.docsim.Shard method) -
    - - -
    get_max_id() (in module gensim.utils) -
    - - -
    get_my_ip() (in module gensim.utils) -
    - - -
    get_similarities() (gensim.similarities.docsim.MatrixSimilarity method) -
    - -
    - -
    (gensim.similarities.docsim.SparseMatrixSimilarity method) -
    - -
    - -
    get_texts() (gensim.corpora.textcorpus.TextCorpus method) -
    - -
    - -
    (gensim.corpora.wikicorpus.WikiCorpus method) -
    - -
    - -
    getNS() (in module gensim.utils) -
    - - -
    getstate() (gensim.models.lda_dispatcher.Dispatcher method) -
    - -
    - -
    (gensim.models.lsi_dispatcher.Dispatcher method) -
    - -
    - -
    getstream() (in module gensim.corpora.textcorpus) -
    - - -
    getworkers() (gensim.models.lda_dispatcher.Dispatcher method) -
    - -
    - -
    (gensim.models.lsi_dispatcher.Dispatcher method) -
    - -
    - -
    grouper() (in module gensim.utils) -
    - -
    - -

    H

    - - - -
    - -
    HashDictionary (class in gensim.corpora.hashdictionary) -
    - - -
    hdp_to_lda() (gensim.models.hdpmodel.HdpModel method) -
    - -
    - -
    HdpModel (class in gensim.models.hdpmodel) -
    - -
    - -

    I

    - - - -
    - -
    index() (simserver.simserver.SessionServer method) -
    - -
    - -
    (simserver.simserver.SimServer method) -
    - -
    - -
    index_documents() (simserver.simserver.SimIndex method) -
    - - -
    IndexedCorpus (class in gensim.corpora.indexedcorpus) -
    - - -
    inference() (gensim.models.ldamodel.LdaModel method) -
    - -
    - -
    initialize() (gensim.models.lda_dispatcher.Dispatcher method) -
    - -
    - -
    (gensim.models.logentropy_model.LogEntropyModel method) -
    - - -
    (gensim.models.lsi_dispatcher.Dispatcher method) -
    - - -
    (gensim.models.rpmodel.RpModel method) -
    - - -
    (gensim.models.tfidfmodel.TfidfModel method) -
    - -
    - -
    is_corpus() (in module gensim.utils) -
    - - -
    iter_chunks() (gensim.similarities.docsim.Similarity method) -
    - -
    - -

    J

    - - - -
    - -
    jobdone() (gensim.models.lda_dispatcher.Dispatcher method) -
    - -
    - -
    (gensim.models.lsi_dispatcher.Dispatcher method) -
    - -
    -
    - -
    jobsdone() (gensim.models.lda_dispatcher.Dispatcher method) -
    - -
    - -
    (gensim.models.lsi_dispatcher.Dispatcher method) -
    - -
    -
    - -

    K

    - - -
    - -
    keys() (gensim.corpora.dictionary.Dictionary method) -
    - -
    - -
    (gensim.corpora.hashdictionary.HashDictionary method) -
    - - -
    (gensim.utils.FakeDict method) -
    - - -
    (simserver.simserver.SimServer method) -
    - -
    -
    - -

    L

    - - - -
    - -
    LdaModel (class in gensim.models.ldamodel) -
    - - -
    LdaState (class in gensim.models.ldamodel) -
    - - -
    lemmatize() (in module gensim.utils) -
    - - -
    load() (gensim.corpora.bleicorpus.BleiCorpus class method) -
    - -
    - -
    (gensim.corpora.dictionary.Dictionary class method) -
    - - -
    (gensim.corpora.hashdictionary.HashDictionary class method) -
    - - -
    (gensim.corpora.indexedcorpus.IndexedCorpus class method) -
    - - -
    (gensim.corpora.lowcorpus.LowCorpus class method) -
    - - -
    (gensim.corpora.mmcorpus.MmCorpus class method) -
    - - -
    (gensim.corpora.svmlightcorpus.SvmLightCorpus class method) -
    - - -
    (gensim.corpora.textcorpus.TextCorpus class method) -
    - - -
    (gensim.corpora.ucicorpus.UciCorpus class method) -
    - - -
    (gensim.corpora.wikicorpus.WikiCorpus class method) -
    - - -
    (gensim.interfaces.CorpusABC class method) -
    - - -
    (gensim.interfaces.SimilarityABC class method) -
    - - -
    (gensim.interfaces.TransformationABC class method) -
    - - -
    (gensim.models.VocabTransform class method) -
    - - -
    (gensim.models.hdpmodel.HdpModel class method) -
    - - -
    (gensim.models.ldamodel.LdaModel class method) -
    - - -
    (gensim.models.ldamodel.LdaState class method) -
    - - -
    (gensim.models.logentropy_model.LogEntropyModel class method) -
    - - -
    (gensim.models.lsimodel.LsiModel class method) -
    - - -
    (gensim.models.rpmodel.RpModel class method) -
    - - -
    (gensim.models.tfidfmodel.TfidfModel class method) -
    - - -
    (gensim.similarities.docsim.MatrixSimilarity class method) -
    - - -
    (gensim.similarities.docsim.Shard class method) -
    - - -
    (gensim.similarities.docsim.Similarity class method) -
    - - -
    (gensim.similarities.docsim.SparseMatrixSimilarity class method) -
    - - -
    (gensim.utils.RepeatCorpus class method) -
    - - -
    (gensim.utils.SaveLoad class method) -
    - - -
    (simserver.simserver.SessionServer class method) -
    - - -
    (simserver.simserver.SimModel class method) -
    - -
    -
    - -
    load_from_text() (gensim.corpora.dictionary.Dictionary static method) -
    - - -
    LogEntropyModel (class in gensim.models.logentropy_model) -
    - - -
    LowCorpus (class in gensim.corpora.lowcorpus) -
    - - -
    LsiModel (class in gensim.models.lsimodel) -
    - -
    - -

    M

    - - - -
    - -
    MatrixSimilarity (class in gensim.similarities.docsim) -
    - - -
    merge() (gensim.models.ldamodel.LdaState method) -
    - -
    - -
    (simserver.simserver.SimIndex method) -
    - -
    - -
    merge_sims() (in module simserver.simserver) -
    - - -
    MmCorpus (class in gensim.corpora.mmcorpus) -
    - -
    - -
    MmReader (class in gensim.matutils) -
    - - -
    MmWriter (class in gensim.matutils) -
    - - -
    Model -
    - -
    - -

    N

    - - -
    - -
    Node -
    - -
    - -

    O

    - - - -
    - -
    open_session() (simserver.simserver.SessionServer method) -
    - - -
    optimal_ordering() (gensim.models.hdpmodel.HdpModel method) -
    - -
    - -
    optimize() (simserver.simserver.SessionServer method) -
    - -
    - -
    (simserver.simserver.SimServer method) -
    - -
    -
    - -

    P

    - - - -
    - -
    pad() (in module gensim.matutils) -
    - - -
    pickle() (in module gensim.utils) -
    - - -
    precompute_idfs() (in module gensim.models.tfidfmodel) -
    - - -
    print_debug() (gensim.models.lsimodel.LsiModel method) -
    - -
    - -
    print_topics() (gensim.models.lsimodel.LsiModel method) -
    - - -
    process_article() (in module gensim.corpora.wikicorpus) -
    - - -
    pyro_daemon() (in module gensim.utils) -
    - -
    - -

    Q

    - - - -
    - -
    qr_destroy() (in module gensim.matutils) -
    - -
    - -
    query_shards() (gensim.similarities.docsim.Similarity method) -
    - -
    - -

    R

    - - - -
    - -
    remove_file() (in module gensim.corpora.wikicorpus) -
    - - -
    remove_template() (in module gensim.corpora.wikicorpus) -
    - - -
    RepeatCorpus (class in gensim.utils) -
    - - -
    reset() (gensim.models.lda_dispatcher.Dispatcher method) -
    - -
    - -
    (gensim.models.ldamodel.LdaState method) -
    - -
    -
    - -
    restricted_hash() (gensim.corpora.hashdictionary.HashDictionary method) -
    - - -
    revdict() (in module gensim.utils) -
    - - -
    rollback() (simserver.simserver.SessionServer method) -
    - - -
    RpModel (class in gensim.models.rpmodel) -
    - -
    - -

    S

    - - - -
    - -
    save() (gensim.corpora.bleicorpus.BleiCorpus method) -
    - -
    - -
    (gensim.corpora.dictionary.Dictionary method) -
    - - -
    (gensim.corpora.hashdictionary.HashDictionary method) -
    - - -
    (gensim.corpora.indexedcorpus.IndexedCorpus method) -
    - - -
    (gensim.corpora.lowcorpus.LowCorpus method) -
    - - -
    (gensim.corpora.mmcorpus.MmCorpus method) -
    - - -
    (gensim.corpora.svmlightcorpus.SvmLightCorpus method) -
    - - -
    (gensim.corpora.textcorpus.TextCorpus method) -
    - - -
    (gensim.corpora.ucicorpus.UciCorpus method) -
    - - -
    (gensim.corpora.wikicorpus.WikiCorpus method) -
    - - -
    (gensim.interfaces.CorpusABC method) -
    - - -
    (gensim.interfaces.SimilarityABC method) -
    - - -
    (gensim.interfaces.TransformationABC method) -
    - - -
    (gensim.models.VocabTransform method) -
    - - -
    (gensim.models.hdpmodel.HdpModel method) -
    - - -
    (gensim.models.ldamodel.LdaModel method) -
    - - -
    (gensim.models.ldamodel.LdaState method) -
    - - -
    (gensim.models.logentropy_model.LogEntropyModel method) -
    - - -
    (gensim.models.lsimodel.LsiModel method) -
    - - -
    (gensim.models.rpmodel.RpModel method) -
    - - -
    (gensim.models.tfidfmodel.TfidfModel method) -
    - - -
    (gensim.similarities.docsim.MatrixSimilarity method) -
    - - -
    (gensim.similarities.docsim.Shard method) -
    - - -
    (gensim.similarities.docsim.Similarity method) -
    - - -
    (gensim.similarities.docsim.SparseMatrixSimilarity method) -
    - - -
    (gensim.utils.RepeatCorpus method) -
    - - -
    (gensim.utils.SaveLoad method) -
    - - -
    (simserver.simserver.SessionServer method) -
    - - -
    (simserver.simserver.SimModel method) -
    - -
    - -
    save_as_text() (gensim.corpora.dictionary.Dictionary method) -
    - -
    - -
    (gensim.corpora.hashdictionary.HashDictionary method) -
    - -
    - -
    save_corpus() (gensim.corpora.bleicorpus.BleiCorpus static method) -
    - -
    - -
    (gensim.corpora.indexedcorpus.IndexedCorpus static method) -
    - - -
    (gensim.corpora.lowcorpus.LowCorpus static method) -
    - - -
    (gensim.corpora.mmcorpus.MmCorpus static method) -
    - - -
    (gensim.corpora.svmlightcorpus.SvmLightCorpus static method) -
    - - -
    (gensim.corpora.textcorpus.TextCorpus static method) -
    - - -
    (gensim.corpora.ucicorpus.UciCorpus static method) -
    - - -
    (gensim.corpora.wikicorpus.WikiCorpus static method) -
    - - -
    (gensim.interfaces.CorpusABC static method) -
    - -
    - -
    save_options() (gensim.models.hdpmodel.HdpModel method) -
    - - -
    save_topics() (gensim.models.hdpmodel.HdpModel method) -
    - - -
    SaveLoad (class in gensim.utils) -
    - - -
    scipy2sparse() (in module gensim.matutils) -
    - - -
    serialize() (gensim.corpora.bleicorpus.BleiCorpus class method) -
    - -
    - -
    (gensim.corpora.indexedcorpus.IndexedCorpus class method) -
    - - -
    (gensim.corpora.lowcorpus.LowCorpus class method) -
    - - -
    (gensim.corpora.mmcorpus.MmCorpus class method) -
    - - -
    (gensim.corpora.svmlightcorpus.SvmLightCorpus class method) -
    - - -
    (gensim.corpora.ucicorpus.UciCorpus class method) -
    - -
    - -
    SessionServer (class in simserver.simserver) -
    - - -
    set_autosession() (simserver.simserver.SessionServer method) -
    - - -
    Shard (class in gensim.similarities.docsim) -
    - - -
    show_topic() (gensim.models.lsimodel.LsiModel method) -
    - - -
    show_topics() (gensim.models.ldamodel.LdaModel method) -
    - -
    - -
    (gensim.models.lsimodel.LsiModel method) -
    - -
    - -
    Similarity (class in gensim.similarities.docsim) -
    - - -
    similarity_by_id() (gensim.similarities.docsim.Similarity method) -
    - - -
    SimilarityABC (class in gensim.interfaces) -
    - -
    - -
    SimIndex (class in simserver.simserver) -
    - - -
    SimModel (class in simserver.simserver) -
    - - -
    simple_preprocess() (in module gensim.utils) -
    - - -
    sims2scores() (simserver.simserver.SimIndex method) -
    - - -
    sims_by_id() (simserver.simserver.SimIndex method) -
    - - -
    sims_by_vec() (simserver.simserver.SimIndex method) -
    - - -
    SimServer (class in simserver.simserver) -
    - - -
    simserver.simserver (module) -
    - - -
    skip_headers() (gensim.corpora.mmcorpus.MmCorpus method) -
    - -
    - -
    (gensim.matutils.MmReader method) -
    - -
    - -
    Sparse vector -
    - - -
    Sparse2Corpus (class in gensim.matutils) -
    - - -
    sparse2full() (in module gensim.matutils) -
    - - -
    SparseMatrixSimilarity (class in gensim.similarities.docsim) -
    - - -
    stochastic_svd() (in module gensim.models.lsimodel) -
    - - -
    SvmLightCorpus (class in gensim.corpora.svmlightcorpus) -
    - - -
    synchronous() (in module gensim.utils) -
    - -
    - -

    T

    - - - -
    - -
    terminate() (simserver.simserver.SessionServer method) -
    - -
    - -
    (simserver.simserver.SimIndex method) -
    - -
    - -
    TextCorpus (class in gensim.corpora.textcorpus) -
    - - -
    TfidfModel (class in gensim.models.tfidfmodel) -
    - - -
    to_unicode() (in module gensim.utils) -
    - - -
    to_utf8() (in module gensim.utils) -
    - -
    - -
    tokenize() (in module gensim.corpora.wikicorpus) -
    - -
    - -
    (in module gensim.utils) -
    - -
    - -
    toptexts() (in module gensim.utils) -
    - - -
    train() (simserver.simserver.SessionServer method) -
    - -
    - -
    (simserver.simserver.SimServer method) -
    - -
    - -
    TransformationABC (class in gensim.interfaces) -
    - -
    - -

    U

    - - - -
    - -
    UciCorpus (class in gensim.corpora.ucicorpus) -
    - - -
    UciWriter (class in gensim.corpora.ucicorpus) -
    - - -
    unitvec() (in module gensim.matutils) -
    - - -
    unpickle() (in module gensim.utils) -
    - - -
    update() (gensim.models.ldamodel.LdaModel method) -
    - -
    - -
    update_expectations() (gensim.models.hdpmodel.HdpModel method) -
    - - -
    update_headers() (gensim.corpora.ucicorpus.UciWriter method) -
    - - -
    update_ids() (simserver.simserver.SimIndex method) -
    - - -
    update_mappings() (simserver.simserver.SimIndex method) -
    - - -
    upload_chunked() (in module gensim.utils) -
    - -
    - -

    V

    - - - -
    - -
    vec_by_id() (simserver.simserver.SimIndex method) -
    - - -
    Vector -
    - -
    - -
    vector_by_id() (gensim.similarities.docsim.Similarity method) -
    - - -
    VocabTransform (class in gensim.models) -
    - -
    - -

    W

    - - - -
    - -
    WikiCorpus (class in gensim.corpora.wikicorpus) -
    - - -
    Worker -
    - - -
    write_corpus() (gensim.matutils.MmWriter static method) -
    - -
    - -
    write_headers() (gensim.corpora.ucicorpus.UciWriter method) -
    - - -
    write_vector() (gensim.corpora.ucicorpus.UciWriter method) -
    - -
    - -
    (gensim.matutils.MmWriter method) -
    - -
    -
    - - - -
    -
    -
    -
    -
    -
    -

    Navigation

    - -
    -
    - © Copyright 2009-2012, Radim Řehůřek <radimrehurek(at)seznam.cz>. - Last updated on Sep 16, 2012. -
    - - \ No newline at end of file diff --git a/docs/index.html b/docs/index.html deleted file mode 100644 index 8c00681802..0000000000 --- a/docs/index.html +++ /dev/null @@ -1,257 +0,0 @@ - - - - - - - - - - Gensim – Topic Modelling for Humans — gensim - - - - - - - - - - - - - - - - - - - - - - - - - - - -
    -

    Navigation

    - -
    - - -
    -

    Download

    -

    Get gensim version 0.8.6 from the Python Package -Index or install directly with:

    -
    easy_install -U gensim
    -

    Table Of Contents

    - - - -

    Questions? Suggestions?

    - -

    Join the Google discussion group

    -
    - - -
    - - - -
    - - - - - - - - - -
    - - -

    -Check the source code at Github. Report bugs at the issue tracker. -

    -
    -
    - - - - -
    -
    -
    -
    - -
    -

    Gensim – Topic Modelling for Humans

    -
    - algorithms - analysis - answer - api - collection - concepts - corpus - design - documents - features - framework - human - index - infer - install - introduction - latent - dirichlet allocation - model - open-source - paragraphs - python - query - questions - random - reference - representation - semantic - similar - space - sparse - structure - SVD - text - thought - topic - training - tutorials - unsupervised - vector - words -
    - - - - - - - - - - -
    What is this?Installation
    Get started!About
    -

    Quick Reference Example

    -
    >>> from gensim import corpora, models, similarities
->>>
->>> # Load corpus iterator from a Matrix Market file on disk.
->>> corpus = corpora.MmCorpus('/path/to/corpus.mm')
->>>
->>> # Initialize a transformation (Latent Semantic Indexing with 200 latent dimensions).
->>> lsi = models.LsiModel(corpus, num_topics=200)
->>>
->>> # Convert another corpus to the latent space and index it.
->>> index = similarities.MatrixSimilarity(lsi[another_corpus])
->>>
->>> # determine similarity of a query document against each document in the index
->>> sims = index[query]
-
    -
    -
    -

    What’s new?

    - -
    -
    -
    -
    -
    - - -
    -
    -
    -
    -
    - - -
    - © Copyright 2009-2012, Radim Řehůřek <radimrehurek(at)seznam.cz>. - Last updated on Sep 16, 2012. -
    - - \ No newline at end of file diff --git a/docs/install.html b/docs/install.html deleted file mode 100644 index 99dfaf6e1e..0000000000 --- a/docs/install.html +++ /dev/null @@ -1,269 +0,0 @@ - - - - - - - - - - Installation — gensim - - - - - - - - - - - - - - - - - - - - - - - - - - -
    -

    Navigation

    - -
    - - -
    -
    -

    Table Of Contents

    - - -

    Previous topic

    -

    Introduction

    -

    Next topic

    -

    Tutorials

    - - -
    -
    - - - - -
    -
    -
    -
    - -
    -

    Installation

    -
    -

    Quick install

    -

    Run in your terminal:

    -
    sudo easy_install -U gensim
    -
    -

    In case that fails, or you don’t know what “terminal” means, read on.

    -
    -
    -
    -

    Dependencies

    -

    Gensim is known to run on Linux, Windows and Mac OS X and should run on any other -platform that supports Python 2.5 and NumPy. Gensim depends on the following software:

    -
      -
    • 3.0 > Python >= 2.5. Tested with versions 2.5, 2.6 and 2.7.
      • -
    • NumPy >= 1.3. Tested with version 1.6.1rc2, 1.5.0rc1, 1.4.0, 1.3.0, 1.3.0rc2.
      • -
    • SciPy >= 0.7. Tested with version 0.9.0, 0.8.0, 0.8.0b1, 0.7.1, 0.7.0.
      • -
    -

    Windows users are well advised to try the Enthought distribution, -which conveniently includes Python&NumPy&SciPy in a single bundle, and is free for academic use.

    -
    -
    -

    Install Python and easy_install

    -

    Check what version of Python you have with:

    -
    python --version
-
    -
    -

    You can download Python from http://python.org/download.

    -
    -

    Note

    -

    Gensim requires Python 2.5 or greater and will not run under earlier versions.

    -
    -

    Next, install the easy_install utility, -which will make installing other Python programs easier.

    -
    -
    -

    Install SciPy & NumPy

    -

    These are quite popular Python packages, so chances are there are pre-built binary -distributions available for your platform. You can try installing from source using easy_install:

    -
    sudo easy_install numpy
-sudo easy_install scipy
    -
    -

    If that doesn’t work or if you’d rather install using a binary package, consult -http://www.scipy.org/Download.

    -
    -
    -

    Install gensim

    -

    You can now install (or upgrade) gensim with:

    -
    sudo easy_install --upgrade gensim
    -
    -

    That’s it! Congratulations, you can proceed to the tutorials.

    -
    -

    If you also want to run the algorithms over a cluster -of computers, in Distributed Computing, you should install with:

    -
    sudo easy_install gensim[distributed]
    -
    -

    The optional distributed feature installs Pyro (PYthon Remote Objects). -If you don’t know what distributed computing means, you can ignore it: -gensim will work fine for you anyway. -This optional extension can also be installed separately later with:

    -
    sudo easy_install Pyro4
    -
    -
    -

    There are also alternative routes to install:

    -
      -

    1. If you have downloaded and unzipped the tar.gz source -for gensim (or you’re installing gensim from github), -you can run:

      -
      sudo python setup.py install
      -
      -

      to install gensim into your site-packages folder.

      -
      2. -

    2. If you wish to make local changes to the gensim code (gensim is, after all, a -package which targets research prototyping and modifications), a preferred -way may be installing with:

      -
      sudo python setup.py develop
      -
      -

      This will only place a symlink into your site-packages directory. The actual -files will stay wherever you unpacked them.

      -
      3. -

    3. If you don’t have root priviledges (or just don’t want to put the package into -your site-packages), simply unpack the source package somewhere and that’s it! No -compilation or installation needed. Just don’t forget to set your PYTHONPATH -(or modify sys.path), so that Python can find the unpacked package when importing.

      -
      4. -
    -
    -
    -

    Testing gensim

    -

    To test the package, unzip the tar.gz source and run:

    -
    python setup.py test
    -
    -
    -
    -

    Contact

    -

    Use the gensim discussion group for -any questions and troubleshooting. For private enquiries, you can also send -me an email to the address at the bottom of this page.

    -
    -
    - - -
    -
    -
    -
    -
    - - -
    -

    Navigation

    - -
    - - -
    - © Copyright 2009-2012, Radim Řehůřek <radimrehurek(at)seznam.cz>. - Last updated on Sep 16, 2012. -
    - - \ No newline at end of file diff --git a/docs/interfaces.html b/docs/interfaces.html deleted file mode 100644 index efb3b1039d..0000000000 --- a/docs/interfaces.html +++ /dev/null @@ -1,267 +0,0 @@ - - - - - - - - - - interfaces – Core gensim interfaces — gensim - - - - - - - - - - - - - - - - - - - - - - - - - - - -
    -

    Navigation

    - -
    - - -
    -
    -

    Previous topic

    -

    API Reference

    -

    Next topic

    -

    utils – Various utility functions

    - - -
    -
    - - - - -
    -
    -
    -
    - -
    -

    interfaces – Core gensim interfaces

    -

    This module contains basic interfaces used throughout the whole gensim package.

    -

    The interfaces are realized as abstract base classes (ie., some optional functionality -is provided in the interface itself, so that the interfaces can be subclassed).

    -
    -
    -class gensim.interfaces.CorpusABC
    -

    Interface (abstract base class) for corpora. A corpus is simply an iterable, -where each iteration step yields one document:

    -
    >>> for doc in corpus:
->>>     # do something with the doc...
-
    -
    -

    A document is a sequence of (fieldId, fieldValue) 2-tuples:

    -
    >>> for attr_id, attr_value in doc:
->>>     # do something with the attribute
-
    -
    -

    Note that although a default len() method is provided, it is very inefficient -(performs a linear scan through the corpus to determine its length). Wherever -the corpus size is needed and known in advance (or at least doesn’t change so -that it can be cached), the len() method should be overridden.

    -

    See the gensim.corpora.svmlightcorpus module for an example of a corpus.

    -

    Saving the corpus with the save method (inherited from utils.SaveLoad) will -only store the in-memory (binary, pickled) object representation=the stream -state, and not the documents themselves. See the save_corpus static method -for serializing the actual stream content.

    -
    -
    -classmethod load(fname)
    -

    Load a previously saved object from file (also see save).

    -
    - -
    -
    -save(fname)
    -

    Save the object to file via pickling (also see load).

    -
    - -
    -
    -static save_corpus(fname, corpus, id2word=None)
    -

    Save an existing corpus to disk.

    -

    Some formats also support saving the dictionary (feature_id->word mapping), -which can in this case be provided by the optional id2word parameter.

    -
    >>> MmCorpus.save_corpus('file.mm', corpus)
-
    -
    -

    Some corpora also support an index of where each document begins, so -that the documents on disk can be accessed in O(1) time (see the -corpora.IndexedCorpus base class). In this case, save_corpus is automatically -called internally by serialize, which does save_corpus plus saves the index -at the same time, so you want to store the corpus with:

    -
    >>> MmCorpus.serialize('file.mm', corpus) # stores index as well, allowing random access to individual documents
-
    -
    -

    Calling serialize() is preferred to calling save_corpus().

    -
    - -
    - -
    -
    -class gensim.interfaces.SimilarityABC(corpus)
    -

    Abstract interface for similarity searches over a corpus.

    -

    In all instances, there is a corpus against which we want to perform the -similarity search.

    -

    For each similarity search, the input is a document and the output are its -similarities to individual corpus documents.

    -

    Similarity queries are realized by calling self[query_document].

    -

    There is also a convenience wrapper, where iterating over self yields -similarities of each document in the corpus against the whole corpus (ie., -the query is each corpus document in turn).

    -
    -
    -classmethod load(fname)
    -

    Load a previously saved object from file (also see save).

    -
    - -
    -
    -save(fname)
    -

    Save the object to file via pickling (also see load).

    -
    - -
    - -
    -
    -class gensim.interfaces.TransformationABC
    -

    Interface for transformations. A ‘transformation’ is any object which accepts -a sparse document via the dictionary notation [] and returns another sparse -document in its stead.

    -

    See the gensim.models.tfidfmodel module for an example of a transformation.

    -
    -
    -classmethod load(fname)
    -

    Load a previously saved object from file (also see save).

    -
    - -
    -
    -save(fname)
    -

    Save the object to file via pickling (also see load).

    -
    - -
    - -
    - - -
    -
    -
    -
    -
    - - -
    -

    Navigation

    - -
    - - -
    - © Copyright 2009-2012, Radim Řehůřek <radimrehurek(at)seznam.cz>. - Last updated on Sep 16, 2012. -
    - - \ No newline at end of file diff --git a/docs/intro.html b/docs/intro.html deleted file mode 100644 index ad2d1f845a..0000000000 --- a/docs/intro.html +++ /dev/null @@ -1,283 +0,0 @@ - - - - - - - - - - Introduction — gensim - - - - - - - - - - - - - - - - - - - - - - - - - - -
    -

    Navigation

    - -
    - - -
    -
    -

    Table Of Contents

    - - -

    Previous topic

    -

    Gensim – Topic Modelling for Humans

    -

    Next topic

    -

    Installation

    - - -
    -
    - - - - -
    -
    -
    -
    - -
    -

    Introduction

    -

    Gensim is a free Python framework designed to automatically extract semantic -topics from documents, as efficiently (computer-wise) and painlessly (human-wise) as possible.

    -

    Gensim aims at processing raw, unstructured digital texts (“plain text”). -The algorithms in gensim, such as Latent Semantic Analysis, Latent Dirichlet Allocation or Random Projections, -discover semantic structure of documents, by examining word statistical co-occurrence patterns within a corpus of training documents. -These algorithms are unsupervised, which means no human input is necessary – you only need a corpus of plain text documents.

    -

    Once these statistical patterns are found, any plain text documents can be succinctly -expressed in the new, semantic representation, and queried for topical similarity -against other documents.

    -
    -

    Note

    -

    If the previous paragraphs left you confused, you can read more about the Vector -Space Model and unsupervised -document analysis on Wikipedia.

    -
    -
    -

    Features

    -
      -
    • Memory independence – there is no need for the whole training corpus to -reside fully in RAM at any one time (can process large, web-scale corpora).
      • -
    • Efficient implementations for several popular vector space algorithms, -including Tf-Idf, distributed incremental Latent Semantic Analysis, -distributed incremental Latent Dirichlet Allocation (LDA) or Random Projection; adding new ones is easy (really!).
      • -
    • I/O wrappers and converters around several popular data formats.
      • -
    • Similarity queries for documents in their semantic representation.
      • -
    -

    Creation of gensim was motivated by a perceived lack of available, scalable software -frameworks that realize topic modelling, and/or their overwhelming internal complexity (hail java!). -You can read more about the motivation in our LREC 2010 workshop paper. -If you want to cite gensim in your own work, please refer to that article (BibTeX).

    -

    You’re welcome to share your results on the mailing list, -so others can learn from your success :)

    -

    The principal design objectives behind gensim are:

    -
      -
    1. Straightforward interfaces and low API learning curve for developers. Good for prototyping.
      2. -
    2. Memory independence with respect to the size of the input corpus; all intermediate -steps and algorithms operate in a streaming fashion, accessing one document -at a time.
      3. -
    -
    -

    See also

    -

    If you’re interested in document indexing/similarity retrieval, I also maintain a higher-level package -of document similarity server. It uses gensim internally.

    -
    -
    -
    -

    Availability

    -

    Gensim is licensed under the OSI-approved GNU LPGL license -and can be downloaded either from its github repository -or from the Python Package Index.

    -
    -

    See also

    -

    See the install page for more info on gensim deployment.

    -
    -
    -
    -

    Core concepts

    -

    The whole gensim package revolves around the concepts of corpus, vector and -model.

    -
    -
    Corpus
    -
    A collection of digital documents. This collection is used to automatically -infer structure of the documents, their topics etc. For -this reason, the collection is also called a training corpus. The inferred -latent structure can be later used to assign topics to new documents, which did -not appear in the training corpus. -No human intervention (such as tagging the documents by hand, or creating -other metadata) is required.
    -
    Vector
    -

    In the Vector Space Model (VSM), each document is represented by an -array of features. For example, a single feature may be thought of as a -question-answer pair:

    -
      -
    1. How many times does the word splonge appear in the document? Zero.
      2. -
    2. How many paragraphs does the document consist of? Two.
      3. -
    3. How many fonts does the document use? Five.
      4. -
    -

    The question is usually represented only by its integer id (such as 1, 2 and 3 here), -so that the -representation of this document becomes a series of pairs like (1, 0.0), (2, 2.0), (3, 5.0). -If we know all the questions in advance, we may leave them implicit -and simply write (0.0, 2.0, 5.0). -This sequence of answers can be thought of as a high-dimensional (in this case 3-dimensional) -vector. For practical purposes, only questions to which the answer is (or -can be converted to) a single real number are allowed.

    -

    The questions are the same for each document, so that looking at two -vectors (representing two documents), we will hopefully be able to make -conclusions such as “The numbers in these two vectors are very similar, and -therefore the original documents must be similar, too”. Of course, whether -such conclusions correspond to reality depends on how well we picked our questions.

    -
    -
    Sparse vector
    -

    Typically, the answer to most questions will be 0.0. To save space, -we omit them from the document’s representation, and write only (2, 2.0), -(3, 5.0) (note the missing (1, 0.0)). -Since the set of all questions is known in advance, all the missing features -in a sparse representation of a document can be unambiguously resolved to zero, 0.0.

    -

    Gensim is specific in that it doesn’t prescribe any specific corpus format; -a corpus is anything that, when iterated over, successively yields these sparse vectors. -For example, set([(2, 2.0), (3, 5.0)], ([0, -1.0], [3, -1.0])) is a trivial -corpus of two documents, each with two non-zero feature-answer pairs.

    -
    -
    Model
    -
    For our purposes, a model is a transformation from one document representation -to another (or, in other words, from one vector space to another). -Both the initial and target representations are -still vectors – they only differ in what the questions and answers are. -The transformation is automatically learned from the traning corpus, without human -supervision, and in hopes that the final document representation will be more compact -and more useful: with similar documents having similar representations.
    -
    -
    -

    See also

    -

    For some examples on how this works out in code, go to tutorials.

    -
    -
    -
    - - -
    -
    -
    -
    -
    - - -
    -

    Navigation

    - -
    - - -
    - © Copyright 2009-2012, Radim Řehůřek <radimrehurek(at)seznam.cz>. - Last updated on Sep 16, 2012. -
    - - \ No newline at end of file diff --git a/docs/matutils.html b/docs/matutils.html deleted file mode 100644 index 29b8e8d65d..0000000000 --- a/docs/matutils.html +++ /dev/null @@ -1,304 +0,0 @@ - - - - - - - - - - matutils – Math utils — gensim - - - - - - - - - - - - - - - - - - - - - - - - - - - -
    -

    Navigation

    - -
    - - -
    -
    -

    Previous topic

    -

    utils – Various utility functions

    -

    Next topic

    -

    corpora.bleicorpus – Corpus in Blei’s LDA-C format

    - - -
    -
    - - - - -
    -
    -
    -
    - -
    -

    matutils – Math utils

    -

    This module contains math helper functions.

    -
    -
    -class gensim.matutils.Dense2Corpus(dense, documents_columns=True)
    -

    Treat dense numpy array as a sparse gensim corpus.

    -

    No data copy is made (changes to the underlying matrix imply changes in the -corpus).

    -
    - -
    -
    -class gensim.matutils.MmReader(input, transposed=True)
    -

    Wrap a term-document matrix on disk (in matrix-market format), and present it -as an object which supports iteration over the rows (~documents).

    -

    Note that the file is read into memory one document at a time, not the whole -matrix at once (unlike scipy.io.mmread). This allows us to process corpora -which are larger than the available RAM.

    -

    Initialize the matrix reader.

    -

    The input refers to a file on local filesystem, which is expected to -be in the sparse (coordinate) Matrix Market format. Documents are assumed -to be rows of the matrix (and document features are columns).

    -

    input is either a string (file path) or a file-like object that supports -seek() (e.g. gzip.GzipFile, bz2.BZ2File).

    -
    -
    -docbyoffset(offset)
    -

    Return document at file offset offset (in bytes)

    -
    - -
    -
    -skip_headers(input_file)
    -

    Skip file headers that appear before the first document.

    -
    - -
    - -
    -
    -class gensim.matutils.MmWriter(fname)
    -

    Store a corpus in Matrix Market format.

    -

    Note that the output is written one document at a time, not the whole -matrix at once (unlike scipy.io.mmread). This allows us to process corpora -which are larger than the available RAM.

    -

    NOTE: the output file is created in a single pass through the input corpus, so -that the input can be a once-only stream (iterator). -To achieve this, a fake MM header is written first, statistics are collected -during the pass (shape of the matrix, number of non-zeroes), followed by a seek -back to the beginning of the file, rewriting the fake header with proper values.

    -
    -
    -static write_corpus(fname, corpus, progress_cnt=1000, index=False)
    -

    Save the vector space representation of an entire corpus to disk.

    -

    Note that the documents are processed one at a time, so the whole corpus -is allowed to be larger than the available RAM.

    -
    - -
    -
    -write_vector(docno, vector)
    -

    Write a single sparse vector to the file.

    -

    Sparse vector is any iterable yielding (field id, field value) pairs.

    -
    - -
    - -
    -
    -class gensim.matutils.Sparse2Corpus(sparse, documents_columns=True)
    -

    Convert a matrix in scipy.sparse format into a streaming gensim corpus.

    -
    - -
    -
    -gensim.matutils.corpus2csc(corpus, num_terms=None, dtype=<type 'numpy.float64'>, num_docs=None, num_nnz=None, printprogress=0)
    -

    Convert corpus into a sparse matrix, in scipy.sparse.csc_matrix format, -with documents as columns.

    -

    If the number of terms, documents and non-zero elements is known, you can pass -them here as parameters and a more memory efficient code path will be taken.

    -
    - -
    -
    -gensim.matutils.corpus2dense(corpus, num_terms)
    -

    Convert corpus into a dense numpy array (documents will be columns).

    -
    - -
    -
    -gensim.matutils.dense2vec(vec, eps=1e-09)
    -

    Convert a dense numpy array into the sparse corpus format (sequence of 2-tuples).

    -

    Values of magnitude < eps are treated as zero (ignored).

    -
    - -
    -
    -gensim.matutils.full2sparse(vec, eps=1e-09)
    -

    Convert a dense numpy array into the sparse corpus format (sequence of 2-tuples).

    -

    Values of magnitude < eps are treated as zero (ignored).

    -
    - -
    -
    -gensim.matutils.full2sparse_clipped(vec, topn, eps=1e-09)
    -

    Like full2sparse, but only return the topn greatest elements (not all).

    -
    - -
    -
    -gensim.matutils.pad(mat, padrow, padcol)
    -

    Add additional rows/columns to a numpy.matrix mat. The new rows/columns -will be initialized with zeros.

    -
    - -
    -
    -gensim.matutils.qr_destroy(la)
    -

    Return QR decomposition of la[0]. Content of la gets destroyed in the process.

    -

    Using this function should be less memory intense than calling scipy.linalg.qr(la[0]), -because the memory used in la[0] is reclaimed earlier.

    -
    - -
    -
    -gensim.matutils.scipy2sparse(vec)
    -

    Convert a scipy.sparse vector to gensim format (list of 2-tuples).

    -
    - -
    -
    -gensim.matutils.sparse2full(doc, length)
    -

    Convert a document in sparse corpus format (sequence of 2-tuples) into a dense -numpy array (of size length).

    -
    - -
    -
    -gensim.matutils.unitvec(vec)
    -

    Scale a vector to unit length. The only exception is the zero vector, which -is returned back unchanged.

    -

    Output will be in the same format as input (i.e., gensim vector=>gensim vector, -or numpy array=>numpy array, scipy.sparse=>scipy.sparse).

    -
    - -
    - - -
    -
    -
    -
    -
    - - -
    -

    Navigation

    - -
    - - -
    - © Copyright 2009-2012, Radim Řehůřek <radimrehurek(at)seznam.cz>. - Last updated on Sep 16, 2012. -
    - - \ No newline at end of file diff --git a/docs/models/hdpmodel.html b/docs/models/hdpmodel.html deleted file mode 100644 index 0b7ce1208d..0000000000 --- a/docs/models/hdpmodel.html +++ /dev/null @@ -1,264 +0,0 @@ - - - - - - - - - - models.hdpmodel – Hierarchical Dirichlet Process — gensim - - - - - - - - - - - - - - - - - - - - - - - - - - - -
    -

    Navigation

    - -
    - - -
    -
    -

    Previous topic

    -

    models.rpmodel – Random Projections

    -

    Next topic

    -

    models.logentropy_model – LogEntropy model

    - - -
    -
    - - - - -
    -
    -
    -
    - -
    -

    models.hdpmodel – Hierarchical Dirichlet Process

    -

    This module encapsulates functionality for the online Hierarchical Dirichlet Process algorithm.

    -

    It allows both model estimation from a training corpus and inference of topic -distribution on new, unseen documents.

    -

    The core estimation code is directly adapted from the onlinelhdp.py script -by C. Wang see -Wang, Paisley, Blei: Online Variational Inference for the Hierarchical Dirichlet -Process, JMLR (2011).

    -

    http://jmlr.csail.mit.edu/proceedings/papers/v15/wang11a/wang11a.pdf

    -

    The algorithm:

    -
    -
      -
    • is streamed: training documents come in sequentially, no random access,
      • -
    • runs in constant memory w.r.t. the number of documents: size of the -training corpus does not affect memory footprint
      • -
    -
    -
    -
    -class gensim.models.hdpmodel.HdpModel(corpus, id2word, max_chunks=None, max_time=None, chunksize=256, kappa=1.0, tau=64.0, K=15, T=150, alpha=1, gamma=1, eta=0.01, scale=1.0, var_converge=0.0001, outputdir=None)
    -

    The constructor estimates Hierachical Dirichlet Process model parameters based -on a training corpus:

    -
    >>> hdp = HdpModel(corpus, id2word)
->>> hdp.print_topics(topics=20, topn=10)
-
    -
    -

    The model doesn’t support inference of topics on new, unseen documents, yet.

    -

    Model persistency is achieved through its load/save methods.

    -

    gamma: first level concentration -alpha: second level concentration -eta: the topic Dirichlet -T: top level truncation level -K: second level truncation level -kappa: learning rate -tau: slow down parameter -max_time: stop training after this many seconds -max_chunks: stop after having processed this many chunks (wrap around -corpus beginning in another corpus pass, if there are not enough chunks -in the corpus)

    -
    -
    -doc_e_step(doc, ss, Elogsticks_1st, word_list, unique_words, doc_word_ids, doc_word_counts, var_converge)
    -

    e step for a single doc

    -
    - -
    -
    -hdp_to_lda()
    -

    Compute the LDA almost equivalent HDP.

    -
    - -
    -
    -classmethod load(fname)
    -

    Load a previously saved object from file (also see save).

    -
    - -
    -
    -optimal_ordering()
    -

    ordering the topics

    -
    - -
    -
    -save(fname)
    -

    Save the object to file via pickling (also see load).

    -
    - -
    -
    -save_options()
    -

    legacy method; use self.save() instead

    -
    - -
    -
    -save_topics(doc_count=None)
    -

    legacy method; use self.save() instead

    -
    - -
    -
    -update_expectations()
    -

    Since we’re doing lazy updates on lambda, at any given moment -the current state of lambda may not be accurate. This function -updates all of the elements of lambda and Elogbeta -so that if (for example) we want to print out the -topics we’ve learned we’ll get the correct behavior.

    -
    - -
    - -
    -
    -gensim.models.hdpmodel.dirichlet_expectation(alpha)
    -

    For a vector theta ~ Dir(alpha), compute E[log(theta)] given alpha.

    -
    - -
    -
    -gensim.models.hdpmodel.expect_log_sticks(sticks)
    -

    For stick-breaking hdp, return the E[log(sticks)]

    -
    - -
    - - -
    -
    -
    -
    -
    - - -
    -

    Navigation

    - -
    - - -
    - © Copyright 2009-2012, Radim Řehůřek <radimrehurek(at)seznam.cz>. - Last updated on Sep 16, 2012. -
    - - \ No newline at end of file diff --git a/docs/models/lda_dispatcher.html b/docs/models/lda_dispatcher.html deleted file mode 100644 index 4806cbad45..0000000000 --- a/docs/models/lda_dispatcher.html +++ /dev/null @@ -1,219 +0,0 @@ - - - - - - - - - - models.lda_dispatcher – Dispatcher for distributed LDA — gensim - - - - - - - - - - - - - - - - - - - - - - - - - - - -
    -

    Navigation

    - -
    - - -
    -
    -

    Previous topic

    -

    models.lsi_worker – Worker for distributed LSI

    -

    Next topic

    -

    models.lda_worker – Worker for distributed LDA

    - - -
    -
    - - - - -
    -
    -
    -
    - -
    -

    models.lda_dispatcher – Dispatcher for distributed LDA

    -

    USAGE: %(program)s SIZE_OF_JOBS_QUEUE

    -
    -
    Dispatcher process which orchestrates distributed LDA computations. Run this script only once, on any node in your cluster.
    -

    Example: python -m gensim.models.lda_dispatcher

    -
    -
    -class gensim.models.lda_dispatcher.Dispatcher(maxsize=10)
    -

    Dispatcher object that communicates and coordinates individual workers.

    -

    There should never be more than one dispatcher running at any one time.

    -

    Note that the constructor does not fully initialize the dispatcher; -use the initialize() function to populate it with workers etc.

    -
    -
    -exit()
    -

    Terminate all registered workers and then the dispatcher.

    -
    - -
    -
    -getstate()
    -

    Merge states from across all workers and return the result.

    -
    - -
    -
    -getworkers()
    -

    Return pyro URIs of all registered workers.

    -
    - -
    -
    -initialize(**model_params)
    -

    model_params are parameters used to initialize individual workers (gets -handed all the way down to worker.initialize()).

    -
    - -
    -
    -jobdone(*args, **kwargs)
    -

    A worker has finished its job. Log this event and then asynchronously -transfer control back to the worker.

    -

    In this way, control flow basically oscillates between dispatcher.jobdone() -and worker.requestjob().

    -
    - -
    -
    -jobsdone()
    -

    Wrap self._jobsdone, needed for remote access through Pyro proxies

    -
    - -
    -
    -reset(state)
    -

    Initialize all workers for a new EM iterations.

    -
    - -
    - -
    - - -
    -
    -
    -
    -
    - - -
    -

    Navigation

    - -
    - - -
    - © Copyright 2009-2012, Radim Řehůřek <radimrehurek(at)seznam.cz>. - Last updated on Sep 16, 2012. -
    - - \ No newline at end of file diff --git a/docs/models/lda_worker.html b/docs/models/lda_worker.html deleted file mode 100644 index c25c7d00e7..0000000000 --- a/docs/models/lda_worker.html +++ /dev/null @@ -1,164 +0,0 @@ - - - - - - - - - - models.lda_worker – Worker for distributed LDA — gensim - - - - - - - - - - - - - - - - - - - - - - - - - - - -
    -

    Navigation

    - -
    - - -
    -
    -

    Previous topic

    -

    models.lda_dispatcher – Dispatcher for distributed LDA

    -

    Next topic

    -

    similarities.docsim – Document similarity queries

    - - -
    -
    - - - - -
    -
    -
    -
    - -
    -

    models.lda_worker – Worker for distributed LDA

    -

    USAGE: %(program)s

    -
    -
    Worker (“slave”) process used in computing distributed LDA. Run this script on every node in your cluster. If you wish, you may even run it multiple times on a single machine, to make better use of multiple cores (just beware that memory footprint increases accordingly).
    -

    Example: python -m gensim.models.lda_worker

    -
    - - -
    -
    -
    -
    -
    - - -
    -

    Navigation

    - -
    - - -
    - © Copyright 2009-2012, Radim Řehůřek <radimrehurek(at)seznam.cz>. - Last updated on Sep 16, 2012. -
    - - \ No newline at end of file diff --git a/docs/models/ldamodel.html b/docs/models/ldamodel.html deleted file mode 100644 index ce6f750f5d..0000000000 --- a/docs/models/ldamodel.html +++ /dev/null @@ -1,359 +0,0 @@ - - - - - - - - - - models.ldamodel – Latent Dirichlet Allocation — gensim - - - - - - - - - - - - - - - - - - - - - - - - - - - -
    -

    Navigation

    - -
    - - -
    -
    -

    Previous topic

    -

    corpora.indexedcorpus – Random access to corpus documents

    -

    Next topic

    -

    models.lsimodel – Latent Semantic Indexing

    - - -
    -
    - - - - -
    -
    -
    -
    - -
    -

    models.ldamodel – Latent Dirichlet Allocation

    -

    This module encapsulates functionality for the Latent Dirichlet Allocation algorithm.

    -

    It allows both model estimation from a training corpus and inference of topic -distribution on new, unseen documents.

    -

    The core estimation code is directly adapted from the onlineldavb.py script -by M. Hoffman [1], see -Hoffman, Blei, Bach: Online Learning for Latent Dirichlet Allocation, NIPS 2010.

    -

    The algorithm:

    -
    -
      -
    • is streamed: training documents come in sequentially, no random access,
      • -
    • runs in constant memory w.r.t. the number of documents: size of the -training corpus does not affect memory footprint, and
      • -
    • is distributed: makes use of a cluster of machines, if available, to -speed up model estimation.
      • -
    -
    - - - - - -
    [1]http://www.cs.princeton.edu/~mdhoffma
    -
    -
    -class gensim.models.ldamodel.LdaModel(corpus=None, num_topics=100, id2word=None, distributed=False, chunksize=2000, passes=1, update_every=1, alpha=None, eta=None, decay=0.5)
    -

    The constructor estimates Latent Dirichlet Allocation model parameters based -on a training corpus:

    -
    >>> lda = LdaModel(corpus, num_topics=10)
-
    -
    -

    You can then infer topic distributions on new, unseen documents, with

    -
    >>> doc_lda = lda[doc_bow]
-
    -
    -

    The model can be updated (trained) with new documents via

    -
    >>> lda.update(other_corpus)
-
    -
    -

    Model persistency is achieved through its load/save methods.

    -

    num_topics is the number of requested latent topics to be extracted from -the training corpus.

    -

    id2word is a mapping from word ids (integers) to words (strings). It is -used to determine the vocabulary size, as well as for debugging and topic -printing.

    -

    alpha and eta are hyperparameters that affect sparsity of the document-topic -(theta) and topic-word (lambda) distributions. Both default to a symmetric -1.0/num_topics (but can be set to a vector, for asymmetric priors).

    -

    Turn on distributed to force distributed computing (see the web tutorial -on how to set up a cluster of machines for gensim).

    -

    Example:

    -
    >>> lda = LdaModel(corpus, num_topics=100)
->>> print lda[doc_bow] # get topic probability distribution for a document
->>> lda.update(corpus2) # update the LDA model with additional documents
->>> print lda[doc_bow]
-
    -
    -
    -
    -bound(corpus, gamma=None)
    -

    Estimate the variational bound of documents from corpus.

    -

    gamma are the variational parameters on topic weights (one for each -document in corpus). If not supplied, will be automatically inferred -from the model.

    -
    - -
    -
    -clear()
    -

    Clear model state (free up some memory). Used in the distributed algo.

    -
    - -
    -
    -do_estep(chunk, state=None)
    -

    Perform inference on a chunk of documents, and accumulate the collected -sufficient statistics in state (or self.state if None).

    -
    - -
    -
    -do_mstep(rho, other)
    -

    M step: use linear interpolation between the existing topics and -collected sufficient statistics in other to update the topics.

    -
    - -
    -
    -inference(chunk, collect_sstats=False)
    -

    Given a chunk of sparse document vectors, estimate gamma (parameters -controlling the topic weights) for each document in the chunk.

    -

    This function does not modify the model (=is read-only aka const). The -whole input chunk of document is assumed to fit in RAM; chunking of a -large corpus must be done earlier in the pipeline.

    -

    If collect_sstats is True, also collect sufficient statistics needed -to update the model’s topic-word distributions, and return a 2-tuple -(gamma, sstats). Otherwise, return (gamma, None). gamma is of shape -len(chunk) x topics.

    -
    - -
    -
    -classmethod load(fname)
    -

    Load a previously saved object from file (also see save).

    -
    - -
    -
    -save(fname)
    -

    Save the object to file via pickling (also see load).

    -
    - -
    -
    -show_topics(topics=10, topn=10, log=False, formatted=True)
    -

    Print the topN most probable words for (randomly selected) topics -number of topics. Set topics=-1 to print all topics.

    -

    Unlike LSA, there is no ordering between the topics in LDA. -The printed topics <= self.num_topics subset of all topics is therefore -arbitrary and may change between two runs.

    -
    - -
    -
    -update(corpus, chunksize=None, decay=None, passes=None, update_every=None)
    -

    Train the model with new documents, by EM-iterating over corpus until -the topics converge (or until the maximum number of allowed iterations -is reached).

    -

    In distributed mode, the E step is distributed over a cluster of machines.

    -

    This update also supports updating an already trained model (self) -with new documents from corpus; the two models are then merged in -proportion to the number of old vs. new documents. This feature is still -experimental for non-stationary input streams.

    -

    For stationary input (no topic drift in new documents), on the other hand, -this equals the online update of Hoffman et al. and is guaranteed to -converge for any decay in (0.5, 1.0>.

    -
    - -
    - -
    -
    -class gensim.models.ldamodel.LdaState(eta, shape)
    -

    Encapsulate information for distributed computation of LdaModel objects.

    -

    Objects of this class are sent over the network, so try to keep them lean to -reduce traffic.

    -
    -
    -blend(rhot, other, targetsize=None)
    -

    Given LdaState other, merge it with the current state. Stretch both to -targetsize documents before merging, so that they are of comparable -magnitude.

    -

    Merging is done by average weighting: in the extremes, rhot=0.0 means -other is completely ignored; rhot=1.0 means self is completely ignored.

    -

    This procedure corresponds to the stochastic gradient update from Hoffman -et al., algorithm 2 (eq. 14).

    -
    - -
    -
    -blend2(rhot, other, targetsize=None)
    -

    Alternative, more simple blend.

    -
    - -
    -
    -classmethod load(fname)
    -

    Load a previously saved object from file (also see save).

    -
    - -
    -
    -merge(other)
    -

    Merge the result of an E step from one node with that of another node -(summing up sufficient statistics).

    -

    The merging is trivial and after merging all cluster nodes, we have the -exact same result as if the computation was run on a single node (no -approximation).

    -
    - -
    -
    -reset()
    -

    Prepare the state for a new EM iteration (reset sufficient stats).

    -
    - -
    -
    -save(fname)
    -

    Save the object to file via pickling (also see load).

    -
    - -
    - -
    -
    -gensim.models.ldamodel.dirichlet_expectation(alpha)
    -

    For a vector theta~Dir(alpha), compute E[log(theta)].

    -
    - -
    - - -
    -
    -
    -
    -
    - - -
    -

    Navigation

    - -
    - - -
    - © Copyright 2009-2012, Radim Řehůřek <radimrehurek(at)seznam.cz>. - Last updated on Sep 16, 2012. -
    - - \ No newline at end of file diff --git a/docs/models/logentropy_model.html b/docs/models/logentropy_model.html deleted file mode 100644 index 1cbac4efaa..0000000000 --- a/docs/models/logentropy_model.html +++ /dev/null @@ -1,223 +0,0 @@ - - - - - - - - - - models.logentropy_model – LogEntropy model — gensim - - - - - - - - - - - - - - - - - - - - - - - - - - - -
    -

    Navigation

    - -
    - - -
    -
    -

    Previous topic

    -

    models.hdpmodel – Hierarchical Dirichlet Process

    -

    Next topic

    -

    models.lsi_dispatcher – Dispatcher for distributed LSI

    - - -
    -
    - - - - -
    -
    -
    -
    - -
    -

    models.logentropy_model – LogEntropy model

    -
    -
    -class gensim.models.logentropy_model.LogEntropyModel(corpus, id2word=None, normalize=True)
    -

    Objects of this class realize the transformation between word-document -co-occurence matrix (integers) into a locally/globally weighted matrix -(positive floats).

    -

    This is done by a log entropy normalization, optionally normalizing the -resulting documents to unit length. The following formulas explain how -to compute the log entropy weight for term i in document j:

    -
    local_weight_{i,j} = log(frequency_{i,j} + 1)
-
-P_{i,j} = frequency_{i,j} / sum_j frequency_{i,j}
-
-                      sum_j P_{i,j} * log(P_{i,j})
-global_weight_i = 1 + ----------------------------
-                      log(number_of_documents + 1)
-
-final_weight_{i,j} = local_weight_{i,j} * global_weight_i
    -
    -

    The main methods are:

    -
      -
    1. -
      constructor, which calculates the global weighting for all terms in
      -

      a corpus.

      -
      -
      -
      2. -
    2. -
      the [] method, which transforms a simple count representation into the
      -

      log entropy normalized space.

      -
      -
      -
      3. -
    -
    >>> log_ent = LogEntropyModel(corpus)
->>> print = log_ent[some_doc]
->>> log_ent.save('/tmp/foo.log_ent_model')
-
    -
    -

    Model persistency is achieved via its load/save methods.

    -

    normalize dictates whether the resulting vectors will be -set to unit length.

    -
    -
    -initialize(corpus)
    -

    Initialize internal statistics based on a training corpus. Called -automatically from the constructor.

    -
    - -
    -
    -classmethod load(fname)
    -

    Load a previously saved object from file (also see save).

    -
    - -
    -
    -save(fname)
    -

    Save the object to file via pickling (also see load).

    -
    - -
    - -
    - - -
    -
    -
    -
    -
    - - -
    -

    Navigation

    - -
    - - -
    - © Copyright 2009-2012, Radim Řehůřek <radimrehurek(at)seznam.cz>. - Last updated on Sep 16, 2012. -
    - - \ No newline at end of file diff --git a/docs/models/lsi_dispatcher.html b/docs/models/lsi_dispatcher.html deleted file mode 100644 index fc28612823..0000000000 --- a/docs/models/lsi_dispatcher.html +++ /dev/null @@ -1,213 +0,0 @@ - - - - - - - - - - models.lsi_dispatcher – Dispatcher for distributed LSI — gensim - - - - - - - - - - - - - - - - - - - - - - - - - - - -
    -

    Navigation

    - -
    - - -
    -
    -

    Previous topic

    -

    models.logentropy_model – LogEntropy model

    -

    Next topic

    -

    models.lsi_worker – Worker for distributed LSI

    - - -
    -
    - - - - -
    -
    -
    -
    - -
    -

    models.lsi_dispatcher – Dispatcher for distributed LSI

    -

    USAGE: %(program)s SIZE_OF_JOBS_QUEUE

    -
    -
    Dispatcher process which orchestrates distributed LSI computations. Run this script only once, on any node in your cluster.
    -

    Example: python -m gensim.models.lsi_dispatcher

    -
    -
    -class gensim.models.lsi_dispatcher.Dispatcher(maxsize=0)
    -

    Dispatcher object that communicates and coordinates individual workers.

    -

    There should never be more than one dispatcher running at any one time.

    -

    Note that the constructor does not fully initialize the dispatcher; -use the initialize() function to populate it with workers etc.

    -
    -
    -exit()
    -

    Terminate all registered workers and then the dispatcher.

    -
    - -
    -
    -getstate()
    -

    Merge projections from across all workers and return the final projection.

    -
    - -
    -
    -getworkers()
    -

    Return pyro URIs of all registered workers.

    -
    - -
    -
    -initialize(**model_params)
    -

    model_params are parameters used to initialize individual workers (gets -handed all the way down to worker.initialize()).

    -
    - -
    -
    -jobdone(*args, **kwargs)
    -

    A worker has finished its job. Log this event and then asynchronously -transfer control back to the worker.

    -

    In this way, control flow basically oscillates between dispatcher.jobdone() -worker.requestjob().

    -
    - -
    -
    -jobsdone()
    -

    Wrap self._jobsdone, needed for remote access through proxies

    -
    - -
    - -
    - - -
    -
    -
    -
    -
    - - -
    -

    Navigation

    - -
    - - -
    - © Copyright 2009-2012, Radim Řehůřek <radimrehurek(at)seznam.cz>. - Last updated on Sep 16, 2012. -
    - - \ No newline at end of file diff --git a/docs/models/lsi_worker.html b/docs/models/lsi_worker.html deleted file mode 100644 index 125dc407a2..0000000000 --- a/docs/models/lsi_worker.html +++ /dev/null @@ -1,164 +0,0 @@ - - - - - - - - - - models.lsi_worker – Worker for distributed LSI — gensim - - - - - - - - - - - - - - - - - - - - - - - - - - - -
    -

    Navigation

    - -
    - - -
    -
    -

    Previous topic

    -

    models.lsi_dispatcher – Dispatcher for distributed LSI

    -

    Next topic

    -

    models.lda_dispatcher – Dispatcher for distributed LDA

    - - -
    -
    - - - - -
    -
    -
    -
    - -
    -

    models.lsi_worker – Worker for distributed LSI

    -

    USAGE: %(program)s

    -
    -
    Worker (“slave”) process used in computing distributed LSI. Run this script on every node in your cluster. If you wish, you may even run it multiple times on a single machine, to make better use of multiple cores (just beware that memory footprint increases accordingly).
    -

    Example: python -m gensim.models.lsi_worker

    -
    - - -
    -
    -
    -
    -
    - - -
    -

    Navigation

    - -
    - - -
    - © Copyright 2009-2012, Radim Řehůřek <radimrehurek(at)seznam.cz>. - Last updated on Sep 16, 2012. -
    - - \ No newline at end of file diff --git a/docs/models/lsimodel.html b/docs/models/lsimodel.html deleted file mode 100644 index 8a1d7ebc79..0000000000 --- a/docs/models/lsimodel.html +++ /dev/null @@ -1,356 +0,0 @@ - - - - - - - - - - models.lsimodel – Latent Semantic Indexing — gensim - - - - - - - - - - - - - - - - - - - - - - - - - - - -
    -

    Navigation

    - -
    - - -
    -
    -

    Previous topic

    -

    models.ldamodel – Latent Dirichlet Allocation

    -

    Next topic

    -

    models.tfidfmodel – TF-IDF model

    - - -
    -
    - - - - -
    -
    -
    -
    - -
    -

    models.lsimodel – Latent Semantic Indexing

    -

    Module for Latent Semantic Indexing.

    -

    This module actually contains several algorithms for decomposition of large corpora, a -combination of which effectively and transparently allows building LSI models for:

    -
      -
    • corpora much larger than RAM: only constant memory is needed, independent of -the corpus size (though still dependent on the feature set size)
      • -
    • corpora that are streamed: documents are only accessed sequentially, no -random-access
      • -
    • corpora that cannot be even temporarily stored: each document can only be -seen once and must be processed immediately (one-pass algorithm)
      • -
    • distributed computing for very large corpora, making use of a cluster of -machines
      • -
    -

    Wall-clock performance on the English Wikipedia (2G corpus positions, 3.2M -documents, 100K features, 0.5G non-zero entries in the final TF-IDF matrix), -requesting the top 400 LSI factors:

    - ----- - - - - - - - - - - - - - - - - -
    algorithmserialdistributed
    one-pass merge algorithm5h14m1h41m
    multi-pass stochastic algo (with 2 power iterations)5h39mN/A [1]
    -

    serial = Core 2 Duo MacBook Pro 2.53Ghz, 4GB RAM, libVec

    -

    distributed = cluster of four logical nodes on three physical machines, each -with dual core Xeon 2.0GHz, 4GB RAM, ATLAS

    - - - - - -
    [1]The stochastic algo could be distributed too, but most time is already spent -reading/decompressing the input from disk in its 4 passes. The extra network -traffic due to data distribution across cluster nodes would likely make it -slower.
    -
    -
    -class gensim.models.lsimodel.LsiModel(corpus=None, num_topics=200, id2word=None, chunksize=20000, decay=1.0, distributed=False, onepass=True, power_iters=2, extra_samples=100)
    -

    Objects of this class allow building and maintaining a model for Latent -Semantic Indexing (also known as Latent Semantic Analysis).

    -

    The main methods are:

    -
      -
    1. constructor, which initializes the projection into latent topics space,
      2. -
    2. the [] method, which returns representation of any input document in the -latent space,
      3. -
    3. add_documents() for incrementally updating the model with new documents.
      4. -
    -

    The left singular vectors are stored in lsi.projection.u, singular values -in lsi.projection.s. Right singular vectors can be reconstructed from the output -of lsi[training_corpus], if needed.

    -

    Model persistency is achieved via its load/save methods.

    -

    num_topics is the number of requested factors (latent dimensions).

    -

    After the model has been trained, you can estimate topics for an -arbitrary, unseen document, using the topics = self[document] dictionary -notation. You can also add new training documents, with self.add_documents, -so that training can be stopped and resumed at any time, and the -LSI transformation is available at any point.

    -

    If you specify a corpus, it will be used to train the model. See the -method add_documents for a description of the chunksize and decay parameters.

    -

    Turn onepass off to force a multi-pass stochastic algorithm.

    -

    power_iters and extra_samples affect the accuracy of the stochastic -multi-pass algorithm, which is used either internally (onepass=True) or -as the front-end algorithm (onepass=False). Increasing the number of -power iterations improves accuracy, but lowers performance. See [2] for -some hard numbers.

    -

    Turn on distributed to enable distributed computing.

    -

    Example:

    -
    >>> lsi = LsiModel(corpus, num_topics=10)
->>> print lsi[doc_tfidf] # project some document into LSI space
->>> lsi.add_documents(corpus2) # update LSI on additional documents
->>> print lsi[doc_tfidf]
-
    -
    - - - - - -
    [2]http://nlp.fi.muni.cz/~xrehurek/nips/rehurek_nips.pdf
    -
    -
    -add_documents(corpus, chunksize=None, decay=None)
    -

    Update singular value decomposition to take into account a new -corpus of documents.

    -

    Training proceeds in chunks of chunksize documents at a time. The size of -chunksize is a tradeoff between increased speed (bigger chunksize) -vs. lower memory footprint (smaller chunksize). If the distributed mode -is on, each chunk is sent to a different worker/computer.

    -

    Setting decay < 1.0 causes re-orientation towards new data trends in the -input document stream, by giving less emphasis to old observations. This allows -LSA to gradually “forget” old observations (documents) and give more -preference to new ones.

    -
    - -
    -
    -classmethod load(fname)
    -

    Load a previously saved object from file (also see save).

    -
    - -
    -
    -print_debug(num_topics=5, num_words=10)
    -

    Print (to log) the most salient words of the first num_topics topics.

    -

    Unlike print_topics(), this looks for words that are significant for a -particular topic and not for others. This should result in a more -human-interpretable description of topics.

    -
    - -
    -
    -print_topics(num_topics=5, num_words=10)
    -

    Alias for show_topics() which prints the top 5 topics to log.

    -
    - -
    -
    -save(fname)
    -

    Override the default save (which uses cPickle), because that’s -too inefficient and cPickle has bugs. Instead, single out the large transformation -matrix and store that separately in binary format (that can be directly -mmap’ed back in load()), under fname.npy.

    -
    - -
    -
    -show_topic(topicno, topn=10)
    -

    Return a specified topic (=left singular vector), 0 <= topicno < self.num_topics, -as string.

    -

    Return only the topn words which contribute the most to the direction -of the topic (both negative and positive).

    -
    >>> lsimodel.print_topic(10, topn=5)
-'-0.340 * "category" + 0.298 * "$M$" + 0.183 * "algebra" + -0.174 * "functor" + -0.168 * "operator"'
-
    -
    -
    - -
    -
    -show_topics(num_topics=-1, num_words=10, log=False, formatted=True)
    -

    Show num_topics most significant topics (show all by default). -For each topic, show num_words most significant words (10 words by defaults).

    -

    Return the shown topics as a list – a list of strings if formatted is -True, or a list of (value, word) 2-tuples if it’s False.

    -

    If log is True, also output this result to log.

    -
    - -
    - -
    -
    -gensim.models.lsimodel.clip_spectrum(s, k, discard=0.001)
    -

    Given eigenvalues s, return how many factors should be kept to avoid -storing spurious (tiny, numerically instable) values.

    -

    This will ignore the tail of the spectrum with relative combined mass < min(discard, 1/k).

    -

    The returned value is clipped against k (= never return more than k).

    -
    - -
    -
    -gensim.models.lsimodel.stochastic_svd(corpus, rank, num_terms, chunksize=20000, extra_dims=None, power_iters=0, dtype=<type 'numpy.float64'>, eps=1e-06)
    -

    Return (U, S): the left singular vectors and the singular values of the streamed -input corpus corpus [3].

    -

    This may actually return less than the requested number of top rank factors, -in case the input is of lower rank. The extra_dims (oversampling) and especially -power_iters (power iterations) parameters affect accuracy of the decomposition.

    -

    This algorithm uses 2+power_iters passes over the data. In case you can only -afford a single pass over the input corpus, set onepass=True in LsiModel -and avoid using this algorithm directly.

    -

    The decomposition algorithm is based on -Halko, Martinsson, Tropp. Finding structure with randomness, 2009.

    - - - - - -
    [3]If corpus is a scipy.sparse matrix instead, it is assumed the whole -corpus fits into core memory and a different (more efficient) code path is chosen.
    -
    - -
    - - -
    -
    -
    -
    -
    - - -
    -

    Navigation

    - -
    - - -
    - © Copyright 2009-2012, Radim Řehůřek <radimrehurek(at)seznam.cz>. - Last updated on Sep 16, 2012. -
    - - \ No newline at end of file diff --git a/docs/models/models.html b/docs/models/models.html deleted file mode 100644 index 5a99b51a41..0000000000 --- a/docs/models/models.html +++ /dev/null @@ -1,168 +0,0 @@ - - - - - - - - - - models – Package for transformation models — gensim - - - - - - - - - - - - - - - - - - - - - - - - -
    -

    Navigation

    - -
    - - -
    -
    - - -
    -
    - - - - -
    -
    -
    -
    - -
    -

    models – Package for transformation models

    -

    This package contains algorithms for extracting document representations from their raw -bag-of-word counts.

    -
    -
    -class gensim.models.VocabTransform(old2new, id2token=None)
    -

    Remap feature ids to new values.

    -

    Given a mapping between old ids and new ids (some old ids may be missing = these -features are to be discarded), this will wrap a corpus so that iterating over -VocabTransform[corpus] returns the same vectors but with the new ids.

    -

    Old features that have no counterpart in the new ids are discarded. This -can be used to filter vocabulary of a corpus “online”:

    -
    >>> old2new = dict((oldid, newid) for newid, oldid in enumerate(ids_you_want_to_keep))
->>> vt = VocabTransform(old2new)
->>> for vec_with_new_ids in vt[corpus_with_old_ids]:
->>>     ...
-
    -
    -
    -
    -classmethod load(fname)
    -

    Load a previously saved object from file (also see save).

    -
    - -
    -
    -save(fname)
    -

    Save the object to file via pickling (also see load).

    -
    - -
    - -
    - - -
    -
    -
    -
    -
    - - -
    -

    Navigation

    - -
    - - -
    - © Copyright 2009-2012, Radim Řehůřek <radimrehurek(at)seznam.cz>. - Last updated on Sep 16, 2012. -
    - - \ No newline at end of file diff --git a/docs/models/rpmodel.html b/docs/models/rpmodel.html deleted file mode 100644 index 92e1239822..0000000000 --- a/docs/models/rpmodel.html +++ /dev/null @@ -1,202 +0,0 @@ - - - - - - - - - - models.rpmodel – Random Projections — gensim - - - - - - - - - - - - - - - - - - - - - - - - - - - -
    -

    Navigation

    - -
    - - -
    -
    -

    Previous topic

    -

    models.tfidfmodel – TF-IDF model

    -

    Next topic

    -

    models.hdpmodel – Hierarchical Dirichlet Process

    - - -
    -
    - - - - -
    -
    -
    -
    - -
    -

    models.rpmodel – Random Projections

    -
    -
    -class gensim.models.rpmodel.RpModel(corpus, id2word=None, num_topics=300)
    -

    Objects of this class allow building and maintaining a model for Random Projections -(also known as Random Indexing). For theoretical background on RP, see:

    -
    -
    Kanerva et al.: “Random indexing of text samples for Latent Semantic Analysis.”
    -

    The main methods are:

    -
      -
    1. constructor, which creates the random projection matrix
      2. -
    2. the [] method, which transforms a simple count representation into the TfIdf -space.
      3. -
    -
    >>> rp = RpModel(corpus)
->>> print rp[some_doc]
->>> rp.save('/tmp/foo.rp_model')
-
    -
    -

    Model persistency is achieved via its load/save methods.

    -

    id2word is a mapping from word ids (integers) to words (strings). It is -used to determine the vocabulary size, as well as for debugging and topic -printing. If not set, it will be determined from the corpus.

    -
    -
    -initialize(corpus)
    -

    Initialize the random projection matrix.

    -
    - -
    -
    -classmethod load(fname)
    -

    Load a previously saved object from file (also see save).

    -
    - -
    -
    -save(fname)
    -

    Save the object to file via pickling (also see load).

    -
    - -
    - -
    - - -
    -
    -
    -
    -
    - - -
    -

    Navigation

    - -
    - - -
    - © Copyright 2009-2012, Radim Řehůřek <radimrehurek(at)seznam.cz>. - Last updated on Sep 16, 2012. -
    - - \ No newline at end of file diff --git a/docs/models/tfidfmodel.html b/docs/models/tfidfmodel.html deleted file mode 100644 index 6ce81bfd0a..0000000000 --- a/docs/models/tfidfmodel.html +++ /dev/null @@ -1,231 +0,0 @@ - - - - - - - - - - models.tfidfmodel – TF-IDF model — gensim - - - - - - - - - - - - - - - - - - - - - - - - - - - -
    -

    Navigation

    - -
    - - -
    -
    -

    Previous topic

    -

    models.lsimodel – Latent Semantic Indexing

    -

    Next topic

    -

    models.rpmodel – Random Projections

    - - -
    -
    - - - - -
    -
    -
    -
    - -
    -

    models.tfidfmodel – TF-IDF model

    -
    -
    -class gensim.models.tfidfmodel.TfidfModel(corpus=None, id2word=None, dictionary=None, wlocal=<function identity at 0x10ecc00c8>, wglobal=<function df2idf at 0x10fb58140>, normalize=True)
    -

    Objects of this class realize the transformation between word-document co-occurrence -matrix (integers) into a locally/globally weighted TF_IDF matrix (positive floats).

    -

    The main methods are:

    -
      -
    1. constructor, which calculates inverse document counts for all terms in the training corpus.
      2. -
    2. the [] method, which transforms a simple count representation into the TfIdf -space.
      3. -
    -
    >>> tfidf = TfidfModel(corpus)
->>> print = tfidf[some_doc]
->>> tfidf.save('/tmp/foo.tfidf_model')
-
    -
    -

    Model persistency is achieved via its load/save methods.

    -

    Compute tf-idf by multiplying a local component (term frequency) with a -global component (inverse document frequency), and normalizing -the resulting documents to unit length. Formula for unnormalized weight -of term i in document j in a corpus of D documents:

    -
    weight_{i,j} = frequency_{i,j} * log_2(D / document_freq_{i})
    -
    -

    or, more generally:

    -
    weight_{i,j} = wlocal(frequency_{i,j}) * wglobal(document_freq_{i}, D)
    -
    -

    so you can plug in your own custom wlocal and wglobal functions.

    -

    Default for wlocal is identity (other options: math.sqrt, math.log1p, ...) -and default for wglobal is log_2(total_docs / doc_freq), giving the -formula above.

    -

    normalize dictates how the final transformed vectors will be normalized. -normalize=True means set to unit length (default); False means don’t -normalize. You can also set normalize to your own function that accepts -and returns a sparse vector.

    -

    If dictionary is specified, it must be a corpora.Dictionary object -and it will be used to directly construct the inverse document frequency -mapping (then corpus, if specified, is ignored).

    -
    -
    -initialize(corpus)
    -

    Compute inverse document weights, which will be used to modify term -frequencies for documents.

    -
    - -
    -
    -classmethod load(fname)
    -

    Load a previously saved object from file (also see save).

    -
    - -
    -
    -save(fname)
    -

    Save the object to file via pickling (also see load).

    -
    - -
    - -
    -
    -gensim.models.tfidfmodel.df2idf(docfreq, totaldocs, log_base=2.0, add=0.0)
    -

    Compute default inverse-document-frequency for a term with document frequency doc_freq:

    -

    idf = add + log(totaldocs / doc_freq)

    -
    - -
    -
    -gensim.models.tfidfmodel.precompute_idfs(wglobal, dfs, total_docs)
    -

    Precompute the inverse document frequency mapping for all terms.

    -
    - -
    - - -
    -
    -
    -
    -
    - - -
    -

    Navigation

    - -
    - - -
    - © Copyright 2009-2012, Radim Řehůřek <radimrehurek(at)seznam.cz>. - Last updated on Sep 16, 2012. -
    - - \ No newline at end of file diff --git a/docs/modindex.html b/docs/modindex.html deleted file mode 100644 index 61840315ab..0000000000 --- a/docs/modindex.html +++ /dev/null @@ -1,200 +0,0 @@ - - - - - - - Global Module Index — gensim documentation - - - - - - - - - - - - - - - -
    -

    Navigation

    - -
    - -
    -
    - - -
    -
    - -
    -
    -
    -
    - - -

    Global Module Index

    - G -
    - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
     
    G
    - gensim -
        - gensim.corpora - Package for corpora I/O
        - gensim.corpora.bleicorpus - Corpus in Blei's LDA-C format
        - gensim.corpora.dictionary - Construct word<->id mappings
        - gensim.corpora.dmlcorpus - Corpus in DML-CZ format
        - gensim.corpora.lowcorpus - Corpus in List-of-Words format
        - gensim.corpora.mmcorpus - Corpus in Matrix Market format
        - gensim.corpora.svmlightcorpus - Corpus in SVMlight format
        - gensim.corpora.wikicorpus - Corpus from a Wikipedia dump
        - gensim.interfaces - Core gensim interfaces
        - gensim.matutils - Math utils
        - gensim.models - Package for transformation models
        - gensim.models.lda_dispatcher - Dispatcher for distributed LDA
        - gensim.models.lda_worker - Worker for distributed LDA
        - gensim.models.ldamodel - Latent Dirichlet Allocation
        - gensim.models.lsi_dispatcher - Dispatcher for distributed LSI
        - gensim.models.lsi_worker - Worker for distributed LSI
        - gensim.models.lsimodel - Latent Semantic Indexing
        - gensim.models.rpmodel - Random Projections
        - gensim.models.tfidfmodel - TF-IDF model
        - gensim.similarities.docsim - Document similarity queries
        - gensim.utils - Various utility functions
    - - -
    -
    -
    -
    -
    -
    -

    Navigation

    - -
    -
    - © Copyright 2010, Radim Řehůřek <radimrehurek(at)seznam.cz>. - Last updated on Nov 06, 2010. - Created using Sphinx 0.6.5. -
    - - \ No newline at end of file diff --git a/docs/mycorpus.txt b/docs/mycorpus.txt deleted file mode 100644 index d2bbafecd3..0000000000 --- a/docs/mycorpus.txt +++ /dev/null @@ -1,9 +0,0 @@ -Human machine interface for lab abc computer applications -A survey of user opinion of computer system response time -The EPS user interface management system -System and human system engineering testing of EPS -Relation of user perceived response time to error measurement -The generation of random binary unordered trees -The intersection graph of paths in trees -Graph minors IV Widths of trees and well quasi ordering -Graph minors A survey diff --git a/docs/objects.inv b/docs/objects.inv deleted file mode 100644 index 9e1b4674ed..0000000000 Binary files a/docs/objects.inv and /dev/null differ diff --git a/docs/py-modindex.html b/docs/py-modindex.html deleted file mode 100644 index a4fd87b083..0000000000 --- a/docs/py-modindex.html +++ /dev/null @@ -1,268 +0,0 @@ - - - - - - - - - - Python Module Index — gensim - - - - - - - - - - - - - - - - -
    -

    Navigation

    - -
    -
    -
    - - -
    -
    - -
    -
    -
    -
    - - -

    Python Module Index

    - -
    - g | - s -
    - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
     
    - g
    - gensim -
        - gensim.corpora - Package for corpora I/O
        - gensim.corpora.bleicorpus - Corpus in Blei's LDA-C format
        - gensim.corpora.dictionary - Construct word<->id mappings
        - gensim.corpora.hashdictionary - Construct word<->id mappings on the fly (the "hashing trick")
        - gensim.corpora.indexedcorpus - Random access to corpus documents
        - gensim.corpora.lowcorpus - Corpus in List-of-Words format
        - gensim.corpora.mmcorpus - Corpus in Matrix Market format
        - gensim.corpora.svmlightcorpus - Corpus in SVMlight format
        - gensim.corpora.textcorpus - Building corpora with dictionaries
        - gensim.corpora.ucicorpus - Corpus in University of California, Irvine (UCI) bag-of-words format
        - gensim.corpora.wikicorpus - Corpus from a Wikipedia dump
        - gensim.interfaces - Core gensim interfaces
        - gensim.matutils - Math utils
        - gensim.models - Package for transformation models
        - gensim.models.hdpmodel - Hierarchical Dirichlet Process
        - gensim.models.lda_dispatcher - Dispatcher for distributed LDA
        - gensim.models.lda_worker - Worker for distributed LDA
        - gensim.models.ldamodel - Latent Dirichlet Allocation
        - gensim.models.logentropy_model - LogEntropy model
        - gensim.models.lsi_dispatcher - Dispatcher for distributed LSI
        - gensim.models.lsi_worker - Worker for distributed LSI
        - gensim.models.lsimodel - Latent Semantic Indexing
        - gensim.models.rpmodel - Random Projections
        - gensim.models.tfidfmodel - TF-IDF model
        - gensim.similarities.docsim - Document similarity queries
        - gensim.utils - Various utility functions
     
    - s
    - simserver -
        - simserver.simserver - Document similarity server
    - - -
    -
    -
    -
    -
    -
    -

    Navigation

    - -
    -
    - © Copyright 2009-2012, Radim Řehůřek <radimrehurek(at)seznam.cz>. - Last updated on Sep 16, 2012. -
    - - \ No newline at end of file diff --git a/docs/search.html b/docs/search.html deleted file mode 100644 index 879e8556d8..0000000000 --- a/docs/search.html +++ /dev/null @@ -1,117 +0,0 @@ - - - - - - - - - - Search — gensim - - - - - - - - - - - - - - - - - -
    -

    Navigation

    - -
    -
    -
    -
    -
    - -
    -
    -
    -
    - -

    Search

    -
    - -

    - Please activate JavaScript to enable the search - functionality. -

    -
    -

    - From here you can search these documents. Enter your search - words into the box below and click "search". Note that the search - function will automatically search for all of the words. Pages - containing fewer words won't appear in the result list. -

    -
    - - - -
    - -
    - -
    - -
    -
    -
    -
    -
    -
    -

    Navigation

    - -
    -
    - © Copyright 2009-2012, Radim Řehůřek <radimrehurek(at)seznam.cz>. - Last updated on Sep 16, 2012. -
    - - \ No newline at end of file diff --git a/docs/searchindex.js b/docs/searchindex.js deleted file mode 100644 index 856b1eb65f..0000000000 --- a/docs/searchindex.js +++ /dev/null @@ -1 +0,0 @@ -Search.setIndex({objects:{"gensim.corpora.dictionary":{Dictionary:[42,3,1,""]},"gensim.similarities":{docsim:[10,0,1,""]},"gensim.models.logentropy_model.LogEntropyModel":{load:[32,4,1,""],initialize:[32,2,1,""],save:[32,2,1,""]},"gensim.corpora.lowcorpus.LowCorpus":{load:[2,4,1,""],save:[2,2,1,""],serialize:[2,4,1,""],save_corpus:[2,5,1,""],docbyoffset:[2,2,1,""]},"gensim.models.lda_dispatcher.Dispatcher":{reset:[18,2,1,""],getstate:[18,2,1,""],getworkers:[18,2,1,""],initialize:[18,2,1,""],jobdone:[18,2,1,""],exit:[18,2,1,""],jobsdone:[18,2,1,""]},"gensim.models.lsi_dispatcher.Dispatcher":{getstate:[24,2,1,""],getworkers:[24,2,1,""],initialize:[24,2,1,""],jobdone:[24,2,1,""],exit:[24,2,1,""],jobsdone:[24,2,1,""]},"gensim.corpora.mmcorpus":{MmCorpus:[39,3,1,""]},"simserver.simserver.SimServer":{index:[27,2,1,""],drop_index:[27,2,1,""],buffer:[27,2,1,""],keys:[27,2,1,""],train:[27,2,1,""],flush:[27,2,1,""],close:[27,2,1,""],find_similar:[27,2,1,""],optimize:[27,2,1,""],"delete":[27,2,1,""]},"gensim.corpora.wikicorpus":{WikiCorpus:[25,3,1,""],process_article:[25,1,1,""],filter_wiki:[25,1,1,""],remove_template:[25,1,1,""],tokenize:[25,1,1,""],remove_file:[25,1,1,""]},"gensim.interfaces.SimilarityABC":{load:[31,4,1,""],save:[31,2,1,""]},"gensim.matutils.MmWriter":{write_corpus:[8,5,1,""],write_vector:[8,2,1,""]},"gensim.similarities.docsim.SparseMatrixSimilarity":{load:[10,4,1,""],save:[10,2,1,""],get_similarities:[10,2,1,""]},"simserver.simserver.SimModel":{load:[27,4,1,""],close:[27,2,1,""],save:[27,2,1,""],doc2vec:[27,2,1,""],docs2vecs:[27,2,1,""]},simserver:{simserver:[27,0,1,""]},"gensim.similarities.docsim.Shard":{load:[10,4,1,""],save:[10,2,1,""],get_document_id:[10,2,1,""]},"gensim.corpora.hashdictionary.HashDictionary":{load:[38,4,1,""],doc2bow:[38,2,1,""],keys:[38,2,1,""],add_documents:[38,2,1,""],restricted_hash:[38,2,1,""],save:[38,2,1,""],save_as_text:[38,2,1,""],filter_extremes:[38,2,1,""]},"gensim.models.hdpmodel":{HdpModel:[4,3,1,""],dirichlet_expectation:[4,1,1,""],expect_log_sticks:[4,1,1,""]},"gensim.interfaces.CorpusABC":{load:[31,4,1,""],save:[31,2,1,""],save_corpus:[31,5,1,""]},"gensim.corpora.ucicorpus.UciCorpus":{load:[1,4,1,""],create_dictionary:[1,2,1,""],serialize:[1,4,1,""],save_corpus:[1,5,1,""],docbyoffset:[1,2,1,""],save:[1,2,1,""]},"gensim.corpora.svmlightcorpus":{SvmLightCorpus:[26,3,1,""]},"gensim.models.ldamodel":{LdaModel:[35,3,1,""],LdaState:[35,3,1,""],dirichlet_expectation:[35,1,1,""]},"gensim.models.rpmodel.RpModel":{initialize:[28,2,1,""],load:[28,4,1,""],save:[28,2,1,""]},"gensim.corpora.lowcorpus":{LowCorpus:[2,3,1,""]},"simserver.simserver":{SessionServer:[27,3,1,""],merge_sims:[27,1,1,""],SimModel:[27,3,1,""],SimServer:[27,3,1,""],SimIndex:[27,3,1,""]},"gensim.models.lsimodel":{clip_spectrum:[29,1,1,""],LsiModel:[29,3,1,""],stochastic_svd:[29,1,1,""]},"gensim.models.tfidfmodel":{df2idf:[3,1,1,""],precompute_idfs:[3,1,1,""],TfidfModel:[3,3,1,""]},"gensim.similarities.docsim.MatrixSimilarity":{load:[10,4,1,""],save:[10,2,1,""],get_similarities:[10,2,1,""]},"gensim.corpora.dictionary.Dictionary":{load:[42,4,1,""],load_from_text:[42,5,1,""],doc2bow:[42,2,1,""],from_corpus:[42,5,1,""],add_documents:[42,2,1,""],keys:[42,2,1,""],filter_tokens:[42,2,1,""],save:[42,2,1,""],save_as_text:[42,2,1,""],filter_extremes:[42,2,1,""],compactify:[42,2,1,""]},"simserver.simserver.SessionServer":{load:[27,4,1,""],index:[27,2,1,""],find_similar:[27,2,1,""],rollback:[27,2,1,""],buffer:[27,2,1,""],terminate:[27,2,1,""],check_session:[27,2,1,""],open_session:[27,2,1,""],drop_index:[27,2,1,""],train:[27,2,1,""],set_autosession:[27,2,1,""],close:[27,2,1,""],commit:[27,2,1,""],save:[27,2,1,""],optimize:[27,2,1,""],"delete":[27,2,1,""]},"gensim.corpora.bleicorpus.BleiCorpus":{load:[13,4,1,""],save:[13,2,1,""],serialize:[13,4,1,""],save_corpus:[13,5,1,""],docbyoffset:[13,2,1,""]},"gensim.utils.FakeDict":{keys:[6,2,1,""]},"gensim.matutils":{full2sparse_clipped:[8,1,1,""],unitvec:[8,1,1,""],Sparse2Corpus:[8,3,1,""],qr_destroy:[8,1,1,""],corpus2csc:[8,1,1,""],MmReader:[8,3,1,""],pad:[8,1,1,""],scipy2sparse:[8,1,1,""],corpus2dense:[8,1,1,""],MmWriter:[8,3,1,""],dense2vec:[8,1,1,""],Dense2Corpus:[8,3,1,""],full2sparse:[8,1,1,""],sparse2full:[8,1,1,""]},"gensim.corpora.ucicorpus":{UciCorpus:[1,3,1,""],UciWriter:[1,3,1,""]},"gensim.corpora.mmcorpus.MmCorpus":{load:[39,4,1,""],skip_headers:[39,2,1,""],serialize:[39,4,1,""],save_corpus:[39,5,1,""],docbyoffset:[39,2,1,""],save:[39,2,1,""]},"gensim.models.lsimodel.LsiModel":{load:[29,4,1,""],add_documents:[29,2,1,""],show_topics:[29,2,1,""],show_topic:[29,2,1,""],print_topics:[29,2,1,""],save:[29,2,1,""],print_debug:[29,2,1,""]},"gensim.corpora.bleicorpus":{BleiCorpus:[13,3,1,""]},"gensim.models.VocabTransform":{load:[33,4,1,""],save:[33,2,1,""]},"gensim.models.logentropy_model":{LogEntropyModel:[32,3,1,""]},"gensim.models.ldamodel.LdaModel":{load:[35,4,1,""],inference:[35,2,1,""],clear:[35,2,1,""],update:[35,2,1,""],bound:[35,2,1,""],do_mstep:[35,2,1,""],save:[35,2,1,""],do_estep:[35,2,1,""],show_topics:[35,2,1,""]},"gensim.corpora.indexedcorpus.IndexedCorpus":{load:[20,4,1,""],serialize:[20,4,1,""],save:[20,2,1,""],save_corpus:[20,5,1,""]},"gensim.corpora.indexedcorpus":{IndexedCorpus:[20,3,1,""]},"gensim.corpora.textcorpus":{TextCorpus:[22,3,1,""],getstream:[22,1,1,""]},"gensim.models.rpmodel":{RpModel:[28,3,1,""]},"gensim.corpora.hashdictionary":{HashDictionary:[38,3,1,""]},"gensim.corpora.svmlightcorpus.SvmLightCorpus":{load:[26,4,1,""],serialize:[26,4,1,""],save:[26,2,1,""],save_corpus:[26,5,1,""],docbyoffset:[26,2,1,""]},"gensim.interfaces":{TransformationABC:[31,3,1,""],CorpusABC:[31,3,1,""],SimilarityABC:[31,3,1,""]},"simserver.simserver.SimIndex":{vec_by_id:[27,2,1,""],sims2scores:[27,2,1,""],index_documents:[27,2,1,""],sims_by_vec:[27,2,1,""],update_ids:[27,2,1,""],update_mappings:[27,2,1,""],merge:[27,2,1,""],terminate:[27,2,1,""],close:[27,2,1,""],sims_by_id:[27,2,1,""],"delete":[27,2,1,""]},gensim:{models:[33,0,1,""],interfaces:[31,0,1,""],corpora:[37,0,1,""],matutils:[8,0,1,""],utils:[6,0,1,""]},"gensim.corpora.wikicorpus.WikiCorpus":{get_texts:[25,2,1,""],load:[25,4,1,""],save:[25,2,1,""],save_corpus:[25,5,1,""]},"gensim.models.tfidfmodel.TfidfModel":{load:[3,4,1,""],initialize:[3,2,1,""],save:[3,2,1,""]},"gensim.corpora.textcorpus.TextCorpus":{load:[22,4,1,""],get_texts:[22,2,1,""],save:[22,2,1,""],save_corpus:[22,5,1,""]},"gensim.models.hdpmodel.HdpModel":{load:[4,4,1,""],save_options:[4,2,1,""],hdp_to_lda:[4,2,1,""],doc_e_step:[4,2,1,""],update_expectations:[4,2,1,""],optimal_ordering:[4,2,1,""],save_topics:[4,2,1,""],save:[4,2,1,""]},"gensim.utils.SaveLoad":{load:[6,4,1,""],save:[6,2,1,""]},"gensim.corpora.ucicorpus.UciWriter":{write_vector:[1,2,1,""],write_headers:[1,2,1,""],update_headers:[1,2,1,""]},"gensim.models.lsi_dispatcher":{Dispatcher:[24,3,1,""]},"gensim.corpora":{hashdictionary:[38,0,1,""],svmlightcorpus:[26,0,1,""],dictionary:[42,0,1,""],ucicorpus:[1,0,1,""],wikicorpus:[25,0,1,""],bleicorpus:[13,0,1,""],lowcorpus:[2,0,1,""],indexedcorpus:[20,0,1,""],mmcorpus:[39,0,1,""],textcorpus:[22,0,1,""]},"gensim.models":{lsi_worker:[40,0,1,""],VocabTransform:[33,3,1,""],ldamodel:[35,0,1,""],lsi_dispatcher:[24,0,1,""],hdpmodel:[4,0,1,""],logentropy_model:[32,0,1,""],rpmodel:[28,0,1,""],lsimodel:[29,0,1,""],lda_worker:[21,0,1,""],lda_dispatcher:[18,0,1,""],tfidfmodel:[3,0,1,""]},"gensim.models.lda_dispatcher":{Dispatcher:[18,3,1,""]},"gensim.matutils.MmReader":{skip_headers:[8,2,1,""],docbyoffset:[8,2,1,""]},"gensim.similarities.docsim":{SparseMatrixSimilarity:[10,3,1,""],Shard:[10,3,1,""],MatrixSimilarity:[10,3,1,""],Similarity:[10,3,1,""]},"gensim.models.ldamodel.LdaState":{load:[35,4,1,""],reset:[35,2,1,""],merge:[35,2,1,""],save:[35,2,1,""],blend:[35,2,1,""],blend2:[35,2,1,""]},"gensim.similarities.docsim.Similarity":{load:[10,4,1,""],vector_by_id:[10,2,1,""],close_shard:[10,2,1,""],iter_chunks:[10,2,1,""],add_documents:[10,2,1,""],similarity_by_id:[10,2,1,""],query_shards:[10,2,1,""],save:[10,2,1,""]},"gensim.utils.RepeatCorpus":{load:[6,4,1,""],save:[6,2,1,""]},"gensim.interfaces.TransformationABC":{load:[31,4,1,""],save:[31,2,1,""]},"gensim.utils":{deaccent:[6,1,1,""],to_unicode:[6,1,1,""],revdict:[6,1,1,""],chunkize:[6,1,1,""],to_utf8:[6,1,1,""],RepeatCorpus:[6,3,1,""],pyro_daemon:[6,1,1,""],dict_from_corpus:[6,1,1,""],grouper:[6,1,1,""],copytree_hardlink:[6,1,1,""],any2unicode:[6,1,1,""],is_corpus:[6,1,1,""],get_max_id:[6,1,1,""],upload_chunked:[6,1,1,""],tokenize:[6,1,1,""],chunkize_serial:[6,1,1,""],toptexts:[6,1,1,""],synchronous:[6,1,1,""],FakeDict:[6,3,1,""],getNS:[6,1,1,""],unpickle:[6,1,1,""],decode_htmlentities:[6,1,1,""],any2utf8:[6,1,1,""],get_my_ip:[6,1,1,""],SaveLoad:[6,3,1,""],simple_preprocess:[6,1,1,""],lemmatize:[6,1,1,""],pickle:[6,1,1,""]}},terms:{entropi:32,get_text:[22,25],four:[5,15,7,29],secondli:0,prefix:10,update_id:27,ret:0,simindex:27,return_miss:[42,38],dynasti:7,categori:[0,5,29],lgpl:36,"const":[42,35],similaritydocu:27,tfidf_model:3,under:[29,36,6,27,9,10,41],preprocess:[0,22,38,6,42,15],logentropi:[30,32],worth:[15,16],digit:[36,6,9,16],everi:[0,27,21,7,40,10,34,16,17],delv:14,affect:[0,5,27,4,29,34,35,15],ala:6,num_best:10,"0x10faabaa0":2,sparse2corpu:8,upload:[0,6,7,16],id2word:[0,1,2,20,3,4,5,22,31,7,32,39,25,28,26,29,13,14,15,35],merge_sim:27,vector:[0,1,27,3,4,5,22,19,8,32,33,9,10,11,25,12,29,14,15,35,17],math:[30,3,8],reinvent:36,corpora:[0,1,2,3,5,6,8,9,11,12,13,14,15,16,17,19,20,22,25,26,27,29,30,31,34,36,37,38,39,42],direct:[5,15,7,34,29],commerci:36,consequ:15,second:[0,5,4,11,12,15,17],even:[0,29,27,21,36,40,10,15,16],supervis:9,nel:6,neg:[0,5,29],biject:38,mdhoffma:35,nez:6,"new":[0,5,2,4,6,7,8,9,10,11,12,14,15,16,18,19,34,27,29,33,35,36,38,42],net:[2,16],metadata:9,abov:[0,5,3,11,12,26,15,16],"komunist\u016f":6,never:[24,29,14,23,18],here:[0,27,7,8,36,9,10,11,12,34,14,15,16,17],dord:0,accur:[0,4],china:7,corpus_with_random_access:20,path:[19,29,23,8,10,11,41,12,34,14,15,16],bz2file:[0,5,8,22],interpret:[0,29,15,17],newsim:27,militari:7,postou:6,array_nam:10,precis:15,credit:36,"0x10fb58140":3,california:1,aka:[42,35,10],tremend:36,golden:15,wikimedia:[0,25],sqlitedict:[27,16],unix:6,qr_destroi:8,txt:[0,5,36,22,11,12],unit:[3,23,8,32,34,15,17],dnf:0,describ:[0,36,11,12,13,15],would:[5,29,22,7,14,15],palestinian:0,upload_chunk:[6,16,27],call:[1,2,7,8,9,10,11,12,13,15,16,20,22,23,26,27,31,32,42,38,39,25],recommend:[0,15,16],type:[29,23,8,10,14,16],until:[35,5,6,16],adler32:38,make_wiki:0,relat:[6,11,12,14,15,16],notic:34,warn:[14,23],"__iter__":[12,22,11],relai:0,unpack:41,must:[29,27,3,22,6,23,9,11,12,35,15,16,17],wikiproject:7,kumar:0,word:[0,1,2,3,5,6,7,9,10,11,12,13,14,15,17,19,20,22,28,29,30,31,32,33,25,35,38,42],multinomi:15,setup:[0,5,41,11,12],work:[0,1,36,30,23,9,10,41,34,14,16],libvec:[0,5,7,29],conceptu:16,root:[16,41],"16gb":0,overrid:[22,29,6,10,25],phd:36,wang:[15,4],give:[29,3,16],household:[0,5],india:0,all_token:[12,11],want:[0,36,20,4,22,31,7,9,11,41,25,12,34,14,15,16],basebal:5,toutf8:34,textcorpu:[22,30],end:[5,29,7,23,11,12,16],quot:[26,2],polic:0,hom:0,update_expect:4,"ne\u017e":6,how:[35,5,36,3,29,30,32,9,10,11,12,34,14,15,16,17],bleicorpu:[12,13,11,30],recoveri:14,answer:[19,27,9,11,12,14,16],negoti:23,split_on_spac:2,updat:[0,1,27,4,5,29,38,7,34,42,35,15,16],mess:6,after:[0,29,2,4,38,27,11,41,12,42,14,35],lab:[12,14,15,11,16],befor:[27,38,23,8,39,10,11,12,42,35,16],komunistu:6,id2token:[33,7],law:0,parallel:[23,5,6,10,19],averag:35,vfl:0,domin:0,classmethod:[28,1,2,20,3,4,29,22,38,6,27,32,25,33,39,10,26,42,13,35,31],"let\u011bt":6,revolv:9,exclud:0,receiv:14,maintain:[28,29,9],environ:0,enter:[14,27],greec:0,lambda:[35,14,6,4],order:[35,27,4,23,11,12,14,15,16],wine:0,indeped:15,oper:[29,14,9,17],feedback:[14,36],softwar:[36,9,41],over:[0,5,27,6,7,8,9,11,12,14,15,16,17,29,20,22,23,31,33,35,41,25],wiki_en_tfidf:[0,5],becaus:[0,29,27,8,10,11,12,14,15,16],sparsematrixsimilar:[14,10,17],appar:[0,14],japan:7,copyright:[0,5],vari:[12,11],suffix:[6,20],vocabtransform:33,fit:[29,36,10,11,12,34,14,35],fix:[19,14,38,34],better:[19,21,40,10,34,14],persist:[28,1,2,20,3,4,29,27,32,39,11,26,12,13,14,15,35],hidden:[15,16],easier:[0,38,41],descend:14,them:[35,27,6,23,8,9,10,11,41,25,12,26,14,15,16,17],thei:[35,2,6,27,9,26,14,15,16],proce:[29,41],safe:[27,16],"8gb":0,"break":4,band:[0,5,7],silver:0,choic:5,changelog:[19,36],unpickl:6,acml:23,each:[0,1,2,5,6,27,9,10,11,12,13,15,16,19,20,22,23,26,29,31,25,35,38,39,42],debug:[28,38,6,35],went:[5,16],mean:[0,5,36,3,38,7,35,9,11,41,12,34,14,15,16],resum:[29,27,16],ligu:0,min_scor:[27,16],docno:[1,20,8],extract:[0,5,6,7,33,9,11,12,25,35],network:[5,27,29,23,11,12,35,16],goe:36,basedir:27,content:[6,8,31,11,12,25,16],rewrit:8,lowercase_with_underscor:34,adapt:[35,6,4],reader:[8,17],cosin:[34,14,10,17],linear:[31,23,35],navig:16,columella:0,free:[0,36,9,41,25,14,35],standard:[1,2,20,7,39,25,26,13,14,15,17],prototyp:[14,9,41],speedi:0,topn:[29,35,4,8],"\u043a\u0430\u0442\u0435\u0433\u043e\u0440\u0438\u044f":[0,5],filter:[42,33,25],heck:[12,11],isn:[12,11],var_converg:4,onto:23,rang:[38,6,14],corpus2:[29,35],nhl:7,independ:[29,23,9,34,16,17],wast:6,rank:[26,14,29],necess:16,restrict:36,hook:0,unlik:[0,29,36,38,23,8,34,14,35],alreadi:[5,27,29,10,35,16],wrapper:[15,42,31,9,38],unlin:38,massiv:[0,25],primari:23,nomin:0,top:[0,29,27,4,6,10,15],"kateg\u00f3ria":5,sometim:15,toi:[12,15,11],too:[0,29,27,9,10,34,15,16],similarli:[12,7,11],new_doc:[12,11],listen:16,murder:7,jmlr:4,tool:[0,12,14,11],took:[0,7],docs2vec:27,technic:16,toronto:7,target:[15,26,9,41],keyword:[14,16],princeton:35,provid:[1,2,20,22,6,39,25,26,13,15,31],tree:[5,11,12,14,15,16],zero:[0,5,29,8,9,10,11,12,14,15,17],cite:[36,9],project:[28,5,36,29,30,7,24,9,15],matter:[12,5,11],minut:[0,5],idli:6,update_everi:[0,35,7],mini:7,fashion:[12,9,11],ran:16,ram:[0,5,27,29,38,7,8,35,9,10,11,12,34,14,16],mind:[12,11],spectrum:29,raw:[0,27,33,9,10,25,17],manner:16,increment:[0,29,27,9,26,15,16],"__main__":[12,11],seen:[29,38],incompat:34,seek:[22,8],especi:[5,17,29],sims_by_vec:27,bengal:0,huon:0,client:[27,16],fname:[1,2,3,4,6,8,10,13,29,20,22,26,27,28,31,32,33,42,35,38,39,25],simplifi:[22,34],germani:0,though:[5,36,34,29],object:[1,2,3,4,6,8,9,10,11,12,13,14,15,16,17,18,29,20,22,23,24,26,27,28,31,32,33,25,35,38,39,41,42],what:[19,38,23,9,11,41,12,14,15,16],regular:25,mouth:15,macbook:[0,29],bradford:15,broadcast:[5,7,16,23],simplic:[12,11],don:[0,1,2,38,3,23,6,7,27,39,36,10,11,41,26,12,13,14,15,16],doc:[27,20,4,6,8,31,11,12,14,15,16],tagcloud10:19,flow:[24,18],doe:[0,20,4,22,31,24,9,11,12,25,35,18],unchang:[8,16],docsim:[30,10],lsi_work:[5,40,7,30],random:[1,2,4,6,9,10,11,12,13,14,15,16,19,20,22,26,28,29,30,31,35,39,25],perenni:0,syntax:[34,20,10],mediawiki:25,radio:7,radim:36,shelv:27,earth:7,involv:[36,15,23,16],absolut:[42,38],latent:[0,5,36,19,29,23,7,35,9,10,11,30,12,28,14,15,16,17],acquir:16,line2word:2,explain:[16,32],deerwest:[5,11,12,14,15,17],rihanna:0,theme:0,latenc:23,folder:41,laptop:[0,5,7],ids_to_delet:16,thoroughbr:0,stop:[12,29,4,11],compli:34,shown:29,report:[14,36],reconstruct:29,symmetr:35,ema:42,sacrific:36,corpus_transform:15,method:[1,2,3,4,6,27,10,13,15,16,29,20,22,34,28,31,32,26,35,36,38,39,42],twice:5,get_my_ip:6,stead:31,discu:0,add_docu:[29,38,10,34,42,15],squadron:7,deaccent:6,fair:[0,5],num:16,result:[0,1,2,3,5,6,27,9,10,11,12,13,15,16,17,18,29,20,22,23,26,32,34,35,39,42],respons:[5,27,11,12,14,15,16],fail:[6,16,41],iran:0,o_o:16,blei:[0,4,22,30,11,12,13,35,15],best:0,said:7,xrehurek:29,enthought:41,hopefulli:[15,9],databas:[0,25,27],wikipedia:[0,5,19,30,7,25,9,38,29,14,15,16,17],score:[14,27,17],c2d:[5,7],petr:36,awai:16,approach:[12,34,15,11],attribut:[34,6,31],accord:[42,6,15,17],extend:[34,10],xrang:6,doc_e_step:4,extens:[14,15,41],lazi:[12,16,4,11],tisic:6,howev:[14,17],against:[0,19,29,31,9,10,34,14,16,17],clip_spectrum:29,fnc:[6,27],logic:[25,5,23,29],countri:0,com:[0,25,6],asctim:[0,5,11,12,14,15,16,17],"2nd":10,lpgl:9,diff:[0,5],guid:34,assum:[0,29,38,8,11,25,12,42,35,16,17],duplic:16,good_id:42,union:7,numpi:[29,23,8,10,11,41,12,34,16],three:[0,1,11,12,29],been:[29,36,10,42,15,16],accumul:35,much:[0,5,36,29,23,10,11,12,15,16],local_weight_:32,interest:[0,9,16],basic:[24,23,31,10,18],token_id:[42,38],pro:[0,29],tini:[12,29,14,11],quickli:16,"25h":5,worker:[5,21,29,7,23,24,40,30,18],anywher:6,lift:36,efficaci:15,log_ent_model:32,ident:[1,3],gnu:[36,9],shrink:42,servic:[27,16],sourceforg:2,air:7,aim:[15,14,9],euclidean:[15,17],calcul:[38,3,32],pairwis:34,simple_preprocess:[6,16],spawn:16,open_sess:[27,16],to_utf8:[34,6],player:[5,7],kwarg:[24,27,18],hebrew:0,sever:[29,36,38,23,9,10,11,12,34,15],perform:[0,29,27,31,23,35,10,34,14,16],make:[0,29,27,21,23,36,40,9,11,41,12,42,35,15,16,17],format:[0,1,2,5,8,9,10,11,12,13,14,15,16,17,29,20,22,26,30,31,25,35,37,38,39,42],airport:7,show_config:23,complex:[14,9],split:[2,6,23,27,10,11,34,12,42,14,16],complet:[5,7,10,11,12,35],wheel:36,nic:6,hand:[24,35,9,36,18],fairli:36,rais:27,nip:[29,35],studi:15,kept:[29,6,10],hdp_to_lda:4,thu:17,inherit:[6,31],contact:[36,41],greatest:8,thi:[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,31,32,33,34,35,36,37,38,39,40,41,42],mother:7,gzip:8,everyth:[23,16],left:[29,9,10],protocol:[6,16],just:[0,36,21,6,40,10,11,41,12,25,14,15,16],edmonton:7,nordland:0,human:[19,29,9,11,12,14,15,16],yet:[6,15,10,4,16],languag:[36,7,11,12,16,17],previous:[1,2,3,4,6,27,10,13,29,20,22,26,28,31,32,33,34,42,35,38,39,25],easi:[0,12,9,11,16],had:[0,5],num_term:[1,29,6,8,10,34],brisban:0,fortran:16,get_max_id:6,run_simserv:16,els:[6,16],save:[0,1,2,3,4,6,8,9,10,11,12,13,14,15,16,29,20,22,25,26,27,28,31,32,33,34,35,38,39,42],applic:[7,11,12,14,15,16,17],measur:[10,11,12,14,15,16,17],arab:0,leagu:[0,5,7],background:[28,6],training_corpu:[29,16],athlet:0,apart:36,use_lock:27,daemon:[6,16],document_numb:14,specif:[15,26,14,9],arbitrari:[35,29,6,16,34],contradict:15,manual:[34,27,16],documentm:2,remind:0,underli:[1,2,20,8,39,10,26,13,15,16],www:[0,5,7,41,25,35],right:[29,16],old:[29,27,33,34,35,15],czech:[36,7],maxim:6,dead:[12,11],resolv:[0,9],wwe:0,instabl:29,intern:[5,27,20,29,22,6,32,25,9,10,34,31,26,14,15,16],mmreader:8,doc2vec:27,total:[0,42,2,16,38],bottom:[7,41],deploy:9,subclass:[31,20],track:[5,15,7,38],foo:[28,6,3,32],corpusabc:31,corpus_tfidf:15,num_word:[5,29],core:[0,5,4,21,29,30,31,23,40,9,10,35,15,17],plu:[22,25,31,20],uncompress:0,subspac:15,discount:17,promot:17,repositori:9,"super":0,plug:3,obj:6,martinsson:[29,15],slightli:14,doc_count:4,distinct:[0,5,11,12],algo:[29,35],coars:23,commit:[34,27,16],marshal:16,match:[22,14],csail:4,duke:7,bad_id:42,basenam:27,"float":[10,6,3,32,26],encod:[25,42,6,16,38],bound:35,son:7,down:[0,24,27,4,18],paper:[5,36,9,4],lieu:16,wrap:[4,6,8,24,33,10,16,18],root_dir:34,precomput:[27,3],accordingli:[40,21],wai:[0,1,36,24,11,41,12,34,15,16,17,18],segment:5,support:[35,20,4,22,6,8,41,25,31,34,14,16,17],transform:[0,19,3,29,31,32,33,9,28,11,12,34,14,15,17],misconfigur:6,avail:[29,6,23,8,9,41,25,35,15,16,17],width:[12,14,15,11,16],reli:[1,2,20,39,13,26],model_param:[24,18],music:0,analysi:[0,5,19,23,7,9,28,11,12,29,14,15,16,17],bert:19,medium:6,max_result:[27,16],form:[36,6,7,10,11,12,14],offer:36,altogeth:14,write_head:1,back:[29,27,20,6,8,24,10,16,18],hear:7,"rychlost\u00ed":6,tasmania:0,"true":[5,27,3,29,38,6,7,8,32,25,42,14,15,35],document_similar:14,reset:[22,35,18],malaysia:0,maximum:[6,35],tell:[23,10],url:36,more:[35,5,36,3,29,38,8,24,9,10,11,34,12,42,14,15,16,17,18],featur:[0,5,19,29,6,23,8,33,9,10,11,41,34,12,26,14,15,16,17,35],estonia:0,qid:26,"abstract":[22,31,20,16],sidestep:6,fulltext:[6,14,27],write_corpu:8,wang11a:4,exist:[27,20,22,31,11,25,12,42,35,16],mold:[12,11],"mod\u00e8l":0,expresss:25,tagcloud:19,dostal:6,stake:0,when:[27,6,36,9,10,41,34,26,14,16,17],actor:5,lsi_vec:15,test:[0,1,2,20,5,22,39,34,10,11,41,26,12,13,14,15,16],sparse2ful:8,hoffman:[0,35,15],realiti:9,node:[0,5,21,29,23,24,40,35,18],"0x10d5690":[12,11],notwithstand:0,"35gb":0,eigenvalu:29,intent:36,do_mstep:35,consid:[0,14,23],pyrolit:16,rychlosti:6,constrain:[12,11],billion:0,femal:0,faster:[0,5,27,19,23,10,34],bullet:0,update_head:1,berlin:0,uciwrit:1,ignor:[29,27,3,8,11,41,25,12,26,35,15,16],time:[0,5,27,6,7,8,9,11,12,14,15,16,17,18,29,20,21,22,23,24,34,31,25,36,40,42],backward:34,hope:9,corpu:[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,19,20,22,23,25,26,27,28,29,30,31,32,33,34,35,37,38,39,42],concept:[0,19,38,23,9,42,17],chain:15,skip:[8,39],logentropy_model:[30,32],osi:[36,9],signific:[29,6],row:8,hierarch:[19,15,4,30],decid:[12,11],depend:[29,38,9,11,41,12,15],fancier:25,decim:6,father:7,certainli:[34,14],decis:36,terror:6,chunk_of_docu:34,sourc:[19,36,23,6,7,41,16],string:[28,29,2,22,38,6,8,27,25,35,11,26,12,42,14,15,16,17],irvin:1,size_of_jobs_queu:[24,18],cpast:17,exact:35,administr:0,level:[0,5,27,4,23,36,9,11,12,14,15,16,17],did:[9,16],die:0,cloudz:0,iter:[0,1,2,5,6,7,8,9,10,11,12,13,15,16,17,18,19,20,22,26,29,31,33,35,39,25],item:[14,27],documents_column:8,team:[0,7],quick:[12,19,41,11,17],div:19,dir:[35,4],slower:[0,29],trend:[29,15],cost:0,patient:7,docbyoffset:[1,2,20,8,39,13,26],targets:35,appear:[0,38,6,8,39,9,11,12,42,15],scaffold:22,sims2scor:27,current:[35,34,27,23,4],onlineldavb:35,attr_valu:31,nine:[12,5,14,11,17],mele:42,myhash:38,gener:[1,36,3,5,6,23,11,12,34,14,15,16],french:16,slow:[34,6,4],modif:[36,27,16,41],address:[12,36,23,11,41],coincid:16,stem:[42,6,38],wait:[5,6,7,16,27],unsupervis:[19,36,9],bot:0,alta:23,bow:[42,15],queue:[6,23],behav:22,extrem:35,reclaim:8,vec_bow:14,semant:[0,5,27,19,29,23,7,36,9,10,11,30,12,28,14,15,16,17],other_fields_are_allowed_but_ignor:16,extra:[29,20,23,38],doc_n:16,dtype:[29,10,8],modul:[29,4,22,30,6,8,25,10,38,31,42,35,16,17],prefer:[29,20,22,31,41,25],leav:[15,25,9,10,16],fake:8,instal:[0,5,19,23,9,41,25,16,17],matrixsimilar:[19,14,10,34],market:[19,22,30,8,39,11,12],memori:[35,5,27,4,21,29,38,6,8,31,40,9,10,11,12,34,14,15,16,17],calgari:7,bewar:[40,21],univers:[1,36],todai:36,tstfile:20,live:7,suppos:14,australian:0,corpus_lsi:15,club:[5,7],peopl:36,restricted_hash:38,doc_7:16,doc_5:16,doc_2:16,accept:[5,31,3],examin:9,doc_1:16,doc_8:16,fly:[0,25,6,15],tokyo:7,prepar:[0,5,14,35,17],uniqu:[12,42,15,11,16],transpos:8,can:[0,5,27,3,6,8,9,10,11,12,14,15,16,17,29,20,22,23,34,31,33,25,35,38,41,42],purpos:[12,34,9,11],set_autosess:[27,16],favourit:16,encapsul:[42,35,4,38],stream:[0,1,2,20,4,37,29,6,8,39,9,11,26,12,13,35,15,31,17],vec_lsi:14,levelnam:[0,5,11,12,14,15,16,17],topic:[0,5,36,4,19,29,7,9,10,11,12,28,14,15,35,17],occur:[12,11,32],alwai:[0,27,23,10,11,12,15,16],multipl:[0,27,21,23,40,34,14,15,16],"3h20m":7,fieldvalu:31,write:[1,9,10,8,16],enwiki:[0,25],anyon:0,fourth:0,doc_lda:[0,35],map:[0,5,2,20,3,22,30,6,27,38,25,33,28,11,42,12,13,35,31],remap:33,mar:19,mat:8,clone:[27,16],gradient:35,mac:41,mai:[0,29,27,4,21,38,6,23,36,40,33,9,10,41,26,42,14,15,35],data:[0,1,2,29,7,8,9,27,15,16],man:7,stress:5,practic:[15,9],lemmat:[19,6,25],produc:27,inform:35,combin:29,block:[6,7,16],manga:[0,7],talk:[0,23],lsa:[0,5,27,29,11,12,34,14,15,35],"aldil\u00e0":6,gradual:[0,29,15],approv:[36,9],lsi:[0,5,27,19,22,30,24,40,10,34,29,14,15,16],still:[35,5,36,29,9,34,14,15,16],dynam:[5,10,34],entiti:6,feature_id:[22,25,31,20],interspers:16,group:41,thank:[36,16],aviv:0,platform:[16,41],window:[16,41],pythonpath:41,mail:[34,9],main:[28,29,27,3,38,32,10,11,12,42,14,15],fieldvalue2:13,fieldvalue1:13,non:[0,5,29,8,9,11,12,14,15,35],ddr3:0,bow_corpu:15,initi:[2,3,7,8,9,10,13,14,15,16,17,18,19,20,22,23,24,26,28,29,32,36,25],enquiri:41,hamilton:7,now:[0,1,2,20,5,7,27,39,36,34,11,41,26,12,13,14,15,16],discuss:[0,36,41],introduct:[5,9,7,17,19],fieldvaluen:13,term:[2,3,7,8,32,42,15,17],"vy\u0161\u0161\u00ed":6,name:[0,5,20,6,7,34,16],tropp:[29,15],drop:[5,27,7],revert:7,separ:[29,2,22,38,6,10,11,41,12,16],magazin:7,rock:7,attributeerror:16,compil:[16,41],domain:[5,36,23,11,12,16],krai:0,replac:[34,27,25],individu:[20,22,31,23,24,25,18],continu:[12,15,11,16],contrib:[0,5,7],redistribut:36,toptext:6,stochastic_svd:29,year:36,"5h39m":29,happen:[23,16],dispos:[5,23],album:[0,5,7],corpus_memory_friendli:[12,11],"100k":[0,29],space:[0,5,2,3,19,29,8,27,32,9,10,11,12,28,14,15,17],filter_token:[12,42,11],storag:[1,2,20,39,13,26],bundesliga:0,saveload:[6,31],formula:[3,32],bla:[0,23,10],correct:[22,4],similarityabc:31,diseas:7,state:[0,27,4,31,35,16,18],runtimeerror:16,million:[0,5,23,11,12,14],seventh:0,inclus:36,lsi_dispatch:[24,5,7,30],zombi:0,org:[0,38,41,25,26,16],"byte":[1,2,20,6,8,39,25,26,13],care:[0,27,15,16],suffici:[25,35],prescrib:9,frequenc:[3,38,11,12,42,15],synchron:[5,6,27],thing:[12,14,23,11],place:[6,14,41],number_doc:1,think:[5,7],frequent:[42,38],first:[0,1,2,4,5,29,38,8,27,39,10,11,26,12,42,14,15,16,17],origin:[27,9,11,12,42,14,15,16],directli:[0,1,2,3,4,29,23,27,34,39,10,11,26,12,13,35,15,16,17],onc:[0,1,27,5,6,25,7,8,24,9,10,11,34,12,29,15,16,18],arrai:[12,9,10,8,11],revdict:6,yourself:38,submit:10,oppos:[6,23],index_fnam:[1,2,20,39,13,26],open:[5,27,19,22,23,11,12,16],size:[0,5,4,19,29,38,6,8,31,9,10,11,34,12,42,35,28,16,17],given:[0,29,27,4,22,38,6,23,36,33,10,35,15],token2id:[12,11],analog:7,citi:0,tragic:34,optimal_ord:4,conveni:[42,31,41,38],melbourn:0,okrug:0,forthcom:6,"2_input_data_format":2,fraction:[42,38],to_unicod:6,season:[0,5,7],moscow:7,copi:[0,27,6,23,8,25,16,17],specifi:[29,27,3,6,10,13,16],blacklist:0,"_jobsdon":[24,18],github:[36,6,9,41,25,34,16],mostli:[0,16],than:[29,36,38,8,24,10,25,42,16,18],png:7,serv:[36,27,15,16],instanc:[31,6,10,16],temporarili:29,were:[0,36,38,11,12,15],posit:[0,1,2,20,3,5,29,27,32,39,10,26,13],russia:[0,7],seri:[0,9,34,42,16,17],pre:[0,1,2,3,4,6,8,10,13,18,29,20,21,22,24,26,27,28,31,32,33,25,35,37,38,39,40,41,42],sai:[0,5],angel:0,molniya:0,ani:[0,1,4,5,6,8,9,10,11,12,14,15,16,17,18,29,22,23,24,31,35,36,41],himself:7,collect_sstat:35,rp_model:28,ambigu:6,sport:0,engin:[12,14,15,11,16],greek:0,destroi:8,fresh_doc:27,centimet:0,malta:[0,36],note:[0,5,6,7,8,9,10,11,12,14,15,16,17,18,24,34,31,42,36,38,41,25],forc:[25,10,35,7,29],take:[0,5,27,29,23,7,25,15,16,17],advis:41,beirut:0,noth:[12,27,10,11,16],channel:7,begin:[27,20,4,22,31,8,25],sure:[38,27],normal:[27,3,38,6,7,32,42,15],multipli:3,compress:[0,5,25],clearer:0,wire:[6,16],pair:[1,27,8,9,11,12,14],padcol:8,greyish:0,renam:34,jakarta:0,later:[0,1,27,5,7,36,9,10,11,41,12,42,14,16],wing:7,meanwhil:15,runtim:15,"0x10ecc00c8":3,nlp:[29,14,36],medalist:0,show:[0,5,14,15,29],german:0,painlessli:9,concurr:[27,16],hack:6,unfinish:10,fifth:[0,5],blend2:35,xml:[0,12,11,25],onli:[0,5,27,6,8,9,10,11,12,14,15,16,17,18,29,22,23,24,34,31,25,35,36,38,41,42],explicitli:[25,27,16],hockei:7,transact:[36,27,16],activ:[15,23,17],behind:[36,9],parametr:15,dict:[5,27,6,33,11,12,14,15],overwritten:16,sum_j:32,muni:[29,36],variou:[5,36,37,30,6,17],get:[0,19,36,4,22,6,7,8,24,10,11,25,12,34,35,15,16,17,18],mycorpu:[12,22,11],extra_sampl:29,cannot:[29,15,16],matutil:[1,8,30],theta:[35,4],gensim:[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,38,39,40,41,42],requir:[9,11,41,12,14,15],truli:0,median:[0,5],yield:[1,22,6,8,9,10,11,12,31],tfidf:[0,5,27,3,22,28,15,17],xeon:[5,7,29],priviledg:[16,41],doc_scor:27,where:[0,1,36,20,22,6,31,11,25,12,13,14,15,16,17],courcel:0,wiki:[0,38,25],uncopyright:0,kategorija:[0,5],assumpt:[12,11],index_of_docu:10,find_similar:[27,16],concern:[0,12,15,11],infinit:15,televis:[0,5],simultaneu:16,enumer:[14,33,16,17],label:[1,2,20,39,13,26],enough:[0,22,4],gaga:0,charad:14,between:[0,2,3,6,27,10,11,12,13,14,15,16,17,18,29,23,24,32,33,35,38,42],"import":[0,5,27,19,23,36,11,41,12,14,15,16,17],x27:6,paramet:[5,2,20,4,29,22,31,7,8,24,10,25,34,35,15,16,18],across:[29,27,24,10,11,12,14,15,18],spars:[0,1,3,19,22,31,8,9,10,11,34,12,29,35,17],wrd:19,guitar:[0,5,7],wikicorpu:[30,25],come:[0,5,4,22,6,14,15,35],reaction:0,northamerica:0,railwai:5,log_ent:32,pyro4:[5,23,16,41],tutori:[0,5,19,6,7,35,9,11,41,12,14,15,16,17],mani:[29,36,4,9,11,12],pakistan:0,acceler:[15,23],cancer:7,numterm:34,dispatch:[5,7,23,24,30,18],typic:[12,9,11,15],stretch:35,invers:[15,3],mark:[25,6,16],workshop:[36,9],valueerror:16,thousand:[15,10,16],resolut:[0,6],svmlightcorpu:[20,30,31,11,12,26],elogsticks_1st:4,remove_fil:25,aviat:7,futsal:0,tamil:0,those:[15,23],"case":[5,20,29,22,6,9,11,41,31,25,14,15,16],tokens_onc:[12,11],kappa:4,hdp:[15,4],superflu:[34,17],log_2:3,cast:7,margin:15,mmread:8,advantag:[12,34,11,17],ctrl:16,cluster:[0,5,21,29,23,7,24,40,41,35,15,18],scipi:[29,8,10,11,41,12,34,16],shutil:6,anemon:0,develop:[12,36,9,11,41],cpickl:[29,6,10,34],author:[36,7,16],alphabet:[25,6],medic:7,same:[0,1,2,5,6,8,9,10,11,12,14,15,16,17,20,22,23,34,27,31,33,25,35,38,42],check:[5,6,10,11,41,12,34,14],binari:[29,38,31,23,10,11,41,12,42,14,15,16],html:[19,6],pad:8,document:[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,19,20,22,23,25,26,27,29,30,31,32,33,34,35,36,38,39,42],save_opt:4,week:36,finish:[24,5,14,7,18],utf8:[38,6,16,25],nest:25,decompress:29,capabl:16,improv:[0,19,36,29,23,7,34,14,15,16],footprint:[29,4,21,38,40,35,15],appropri:[5,14],markup:[0,25],pep8:34,without:[0,38,9,11,12,14,15],model:[0,5,2,3,4,7,27,9,10,11,12,14,15,16,17,18,19,21,22,24,34,28,29,30,31,32,33,35,36,40],dimension:[15,6,9,14,16],tf_idf:3,execut:[5,15,16,34],resp:[12,11],harddisk:6,rest:[42,10,16,38],tasmanian:0,kill:[23,7],aspect:14,polish:7,speed:[0,29,27,10,34,35,15],concentr:[14,4],rover:0,death:7,get_document_id:10,lapack:23,mmap:[29,27,10,16,34],except:[1,27,6,7,8,15,16],littl:[15,16],treatment:7,query_docu:31,constant:[29,35,4,16],categoria:[0,5,7],around:[15,6,9,4],oop:16,read:[5,27,29,22,8,9,11,41,25,12,42,35,15,16,17],delhi:0,amd:23,"m\u00e1ma":42,traffic:[29,35],world:36,intel:[0,23],whitespac:[12,11],stationari:[0,35],integ:[28,2,3,22,38,6,32,9,11,26,12,42,35,15],server:[5,27,30,6,9,16],republ:7,newid:33,either:[0,29,27,6,8,9,25,15,16],output:[0,5,29,6,8,10,11,12,14,15,31,17],manag:[12,14,15,11,16],supplementari:38,lebanon:0,other_corpu:[35,16],wigan:0,intact:15,bigger:[29,16],slice:10,token:[0,5,2,22,38,6,27,11,25,12,42,16],moon:7,exit:[24,18],number_of_docu:32,notabl:[0,5,11,12],refer:[19,30,7,8,9,11,12,15,16],process:[0,5,27,4,6,8,9,10,11,12,15,16,18,19,21,22,23,24,34,29,30,25,38,40,42],no_abov:[42,38],power:[0,5,16,29],kyoto:7,garbag:[0,12,15,11],inspect:[6,15],fulli:[24,9,11,12,16,18],dict_from_corpu:6,regexp:25,"throw":15,earlier:[12,35,8,41,11],src:34,island:0,intens:8,stand:[12,36,15,11],act:[5,6,16],"cat\u00e9gori":5,routin:23,checkus:0,effici:[29,36,6,8,9,10,11,12,34,14,15,16],activest:6,quasi:[12,14,15,11,16],sstat:35,to_low:6,surviv:5,jerusalem:0,your:[0,5,36,3,21,22,23,7,24,40,9,10,11,41,12,34,14,15,16,17,18],fast:[14,15,23],log:[0,5,3,4,29,7,32,24,35,11,12,14,15,16,17,18],her:0,area:0,hex:6,overwrit:[27,16],start:[5,27,19,6,23,36,11,12,26,16],interfac:[36,20,30,6,31,9,11,12,25,14,15,16,17],low:[12,9,23,11],lot:[0,16],treat:[25,15,8],linalg:8,tupl:[35,29,22,38,6,8,10,31,42,14,16],bundl:41,full2spars:8,jul:19,oscil:[24,18],procedur:[35,23,16],buggi:34,congratul:[14,41],longer:[38,25],pull:34,nemuz:6,"0b1":41,dirti:15,possibl:[34,14,9,7,16],"default":[0,29,2,20,3,38,31,7,10,13,35,16],stochast:[29,35,15],connect:[23,16],creat:[0,5,27,23,7,8,25,9,28,11,34,12,42,14,15,16,17],accentu:6,strongli:15,my_serv:16,intro:19,decreas:[14,27],file:[0,1,2,3,4,6,7,8,10,11,12,13,14,16,19,20,22,23,26,27,28,29,31,32,33,25,35,38,39,41,42],encompass:14,proport:35,film:[0,5,7],fill:6,again:[5,27,6,10,34,26,15,16],tagcloud9:19,id_rang:38,event:[24,11,12,14,15,18],field:[1,8],cleanup:0,collis:38,you:[0,5,27,3,7,8,9,10,11,12,14,15,16,17,29,20,21,22,23,34,31,25,35,36,38,40,41,42],word_list:4,sekundu:6,lsimodel:[0,5,29,19,30,34,14,15],"pr\u00e1\u0161ek":6,sequenc:[1,2,20,22,6,8,31,39,9,10,11,26,12,13,27,16],drift:[0,35],tagcloud0:19,briefli:[12,11],vocabulari:[28,1,19,6,33,5,25,13,35],tagcloud1:19,queensland:0,tagcloud2:19,reduc:[6,15,35],allow_upd:[42,38],wife:7,maso:42,descript:[29,10],tagcloud5:19,inevit:7,ids_you_want_to_keep:33,mass:29,potenti:27,cpu:[5,15],represent:[28,19,27,3,29,38,31,23,8,32,33,9,10,11,12,42,15,16,17],all:[0,5,2,3,4,6,7,8,9,10,11,12,14,15,16,17,18,29,22,23,24,26,27,31,32,25,35,36,38,41,42],forget:[29,11,41,12,14,15,16],ali:0,illustr:16,lack:9,enslav:23,bayesian:15,docfreq:[12,3,11],follow:[2,8,32,11,41,12,26,14],disk:[0,19,27,20,29,22,6,8,31,39,25,10,11,34,12,42,14,15,16,17],ron:0,"divis\u00e3o":0,articl:[0,5,27,7,36,9,11,12,25,14],tail:29,program:[21,24,40,41,16,18],norm:17,sound:34,liter:[1,2,3,4,6,8,10,13,18,29,20,21,22,24,26,27,28,31,32,33,25,35,37,38,39,40,42],straightforward:[34,9,23],song:[0,5],fals:[29,27,3,38,6,8,26,42,35,16],util:[1,27,5,30,6,8,41,31,34,16],mechan:[20,16],fall:6,veri:[0,29,31,23,9,11,12,15,16,17],thesi:36,"b\u00edl\u00fd":6,kanerva:28,list:[0,29,2,20,30,6,25,8,27,38,36,9,10,11,34,12,42,14,16,17],adjust:0,saudi:0,fakedict:6,small:[0,5,16,17],dimens:[19,15,29],deacc:6,tel:0,unnorm:3,ten:[0,5,11,12],rate:4,design:[19,36,9],pass:[0,5,20,4,29,23,7,8,38,35,15,16],further:[22,42,6,15,38],bass:7,trick:[19,38],deleg:23,abc:[12,14,15,11,16],sub:10,clock:29,sun:[23,7],sum:[12,35,11],abl:[5,23,9,11,12,16],no_below:[42,38],delet:[0,5,27,16],version:[0,5,27,23,6,7,36,41,25,34,14,16],intersect:[12,14,15,11,16],hurdl:0,"public":[36,16],contrast:[0,36],full:[0,19,10,34],themselv:[0,31,10],variat:[35,15,4],decode_htmlent:6,ldastat:35,oldid:33,ineffici:[12,29,31,10,11],modifi:[5,27,3,7,36,41,34,35,15,16],valu:[1,27,29,6,8,33,10,26,15],"321d94e0e40d0acc92f5bf57d126b57369da70d":6,search:[31,14,16],prior:[35,23],mattythewhit:0,pick:[5,9],codebas:34,create_dictionari:1,magnitud:[35,8],via:[1,2,3,4,6,27,10,11,12,13,14,15,16,17,29,20,22,26,28,31,32,33,25,35,38,39,42],intermedi:9,filenam:[22,10],href:19,decrement:26,research:[5,14,36,41],select:[0,42,35],documet:7,proceed:[36,4],copytree_hardlink:6,template_messag:0,hdpmodel:[30,15,4],regist:[24,6,16,18],two:[0,1,27,5,6,7,35,9,10,13,14,15,16],taken:[5,15,7,8],forcefulli:6,minor:[5,19,11,12,14,15,16],vec:[25,27,8,17],reachabl:5,israel:0,process_articl:25,hundr:[15,10],probabilist:15,flag:7,stick:4,particular:[29,22,23,11,12,15],known:[0,1,29,31,8,9,28,41],knowl:0,cach:[27,31],dictat:[3,32,17],none:[1,2,3,4,6,8,10,13,16,29,20,22,26,27,28,31,32,33,25,35,38,39,42],hour:[0,5],der:0,histori:[34,36],remain:[12,5,15,7,11],paragraph:[19,9],del:34,learn:[0,35,9,4,15],write_vector:[1,8],male:0,def:[12,11],finland:7,bogu:6,scan:[42,6,10,31,25],challeng:[5,36],share:[14,9,27,17],shard:[34,27,10,16],templat:[0,25],corpus_with_old_id:33,huge:6,cours:[36,9,11,12,15,17],court:0,newlin:[42,38],divid:17,rather:[6,15,41],anoth:[0,5,4,19,22,6,31,9,11,25,12,13,35,15,16,17],comfort:10,divis:6,compactifi:[12,42,11],simpl:[28,2,3,22,6,32,10,11,12,25,35,17],csr:10,global_weight_i:32,resourc:[27,15,16],algebra:[29,23],semat:[0,17],reflect:[0,34],buffer:[27,10],blood:7,associ:16,"short":[0,25,6,36],confus:[0,9],preferr:15,dragon:0,caus:29,doc_id:[27,16],constitu:[12,11,17],help:[5,36,6,25,14,16],ith:2,mmcorpu:[0,1,2,20,5,22,19,31,39,25,11,26,12,13,14,15,30],letet:6,scientist:36,through:[35,1,2,20,4,38,31,23,8,39,24,26,13,14,15,16,18],wiki_en_vocab200k:25,testserv:16,islam:0,style:[12,1,34,11,19],brows:14,relev:[12,14,11],pca:15,might:[14,15],as_numpi:6,fool:6,tri:[6,23],good:[0,9,16],"return":[1,2,3,4,6,8,10,11,12,13,14,15,16,17,18,29,20,22,24,25,26,27,31,33,34,35,38,39,42],lowercas:[12,22,6,11],sentenc:[12,11],hunter:0,framework:[19,36,9],subtask:23,compound:0,oko:0,gethostbynam:6,bweb:0,mingu:0,troubleshoot:41,achiev:[28,29,20,3,4,6,8,32,10,35,15],found:[6,9,16],intervent:9,truncat:[27,4],ifk:0,drop_index:[27,16],weight:[35,15,3,32,17],extra_dim:29,osaka:7,hard:[29,16],idea:[34,14,36,38],functor:29,realli:[5,14,9,36,38],heavi:[36,16],expect:[26,15,8,16],more_docu:[15,16],slave:[40,23,21],energi:0,todo:16,orient:[0,29],docid:[6,27],get_similar:10,publish:[36,15,7],payload:16,health:7,print:[0,1,2,3,4,5,6,11,12,13,14,15,16,17,29,20,26,28,32,25,35,39,42],occurr:[9,3],wlocal:3,doc_tfidf:29,proxi:[24,6,10,16,18],advanc:[6,9,31],corpusmiislita:22,reason:[0,38,23,9,14,16],base:[1,36,20,4,29,22,6,32,10,25,26,35,15,31],basa:0,put:41,teach:16,ash:0,wglobal:3,document_freq_:3,thread:[27,23,16],english:[0,5,36,29,6,15,16,17],ldamodel:[0,30,35,15,7],omit:9,doc_0:16,halko:[29,15],caption:25,perhap:[0,14],perman:27,some_docu:16,assign:[12,42,36,9,11],mytextcorpu:22,feed:15,major:17,notifi:7,feel:15,check_sess:27,number:[35,29,2,4,38,8,9,10,11,34,12,42,14,15,16,17],elogbeta:4,extern:6,unique_word:4,done:[0,5,27,38,6,23,32,35,11,34,12,42,14,15,16],construct:[2,3,22,30,6,25,10,11,38,12,42],blank:[1,2],gibbslda:[12,2,11],miss:[15,33,9],fanci:[25,36],gpl:36,differ:[0,1,29,9,11,12,42,14,15,16],jew:0,script:[0,5,36,4,21,23,7,24,40,25,35,16,18],interact:[12,14,11],least:[12,27,11,31,16],stori:[14,7],station:[0,5,7],statement:7,scheme:15,store:[0,1,2,20,26,29,22,38,31,8,34,25,10,11,42,12,13,27,15,16,17],unitvec:8,option:[2,20,3,22,6,23,27,32,41,25,31,34,15,16,17],relationship:15,month:36,bowel:0,park:0,bgwhite:0,pars:[0,19,22,11,12,25,16,17],consult:41,vocab:[1,13],king:7,simplist:[12,11],dam:0,whenev:6,remot:[36,23,27,24,41,16,18],remov:[5,38,6,23,25,11,34,12,42,16],onlinelhdp:4,vocal:[0,5,7],bridg:17,str:[42,6],kategoria:[0,5],sims_by_id:27,toward:29,randomli:[0,35],comput:[0,5,27,3,4,6,7,9,10,11,12,14,15,16,17,18,29,21,23,24,32,35,36,38,40,41],fieldid1:13,test_miislita:22,fieldid2:13,beforehand:0,packag:[36,37,31,16,33,9,41,25,34,14,6],precompute_idf:3,dedic:16,lie:23,topicno:29,built:[38,10,41],equival:[4,17],self:[29,27,4,22,38,6,24,10,11,12,42,35,31,18],port:[6,23],also:[0,1,2,3,4,5,6,27,9,10,11,12,13,14,15,16,17,29,20,22,23,25,26,28,31,32,33,34,35,36,38,39,41,42],exampl:[1,2,4,5,7,9,11,12,13,14,15,16,17,18,19,20,21,22,23,24,26,29,31,35,39,40,25],build:[28,29,22,30,23,10,38,42],pipelin:[22,35],distribut:[0,5,36,4,21,29,23,6,7,24,40,9,41,30,14,15,35,17,18],previou:[15,14,9],reach:[5,6,7,35],mixtur:15,most:[0,5,27,29,38,6,23,36,9,10,11,12,42,14,15,35,17],plai:0,rho:35,fieldidn:13,alpha:[35,4],charg:[5,23],tlocknam:6,clear:[27,38,23,11,12,35],cover:[0,6,11,12,14,15,17],wordini:2,rpmodel:[28,30,15],part:[14,15,17],clean:[12,34,11],usual:[0,22,9],blend:35,visibl:[27,16],v15:4,session:[5,27,16],palestin:0,font:[19,9],fine:[16,41],find:[29,27,38,7,41,15],impact:[0,17],access:[1,2,4,9,10,11,12,13,16,17,18,29,20,22,23,24,26,30,31,35,39,25],ambit:14,pretti:[0,34],writer:7,experiment:[14,35],factor:29,do_estep:35,unhappi:16,booktitl:36,express:9,padrow:8,stopword:[12,11],twelv:[12,11],arxiv:16,lda_dispatch:[30,7,18],singapor:0,statist:[0,38,8,32,9,11,12,35,16],getwork:[24,18],river:0,set:[1,2,3,5,6,7,27,9,10,11,12,13,14,15,29,20,22,26,28,32,34,35,36,38,39,41,42],dump:[0,30,25],sep:19,overwhelm:9,sef:6,see:[0,1,2,3,4,5,6,7,27,9,10,11,12,13,14,15,16,17,29,20,22,23,25,26,28,31,32,33,34,35,36,38,39,42],arg:[24,25,27,18],close:[27,10],"po\u0161tou":6,expert:0,chart:[0,5],debat:[0,5],won:[12,5,6,23,11],fname_vocab:[1,13],experi:[0,5,11,12,14,15,17],altern:[23,35,10,41],latin:7,"1rc2":41,numer:[29,15],df2idf:3,inproceed:36,"4gb":[5,7,29],paislei:[15,4],hardlink:6,ipython:17,popul:[0,5,22,24,18],both:[0,5,2,4,29,27,9,36,14,15,35],last:[34,6,10],tagcloud8:19,hyperlink:14,lowcorpu:[12,30,2,11],coibot:0,output_prefix:10,context:23,pdf:[0,29,4],rehurek_nip:29,whole:[0,29,6,23,8,35,9,11,12,25,14,15,31,17],power_it:29,hierach:4,load:[0,1,2,3,4,5,6,27,10,11,12,13,14,15,19,20,22,25,26,28,29,31,32,33,34,35,38,39,42],episod:[0,7],consolid:[34,16],point:[29,6,15,27],schedul:5,final_weight_:32,sweep:[12,11],arbitrarili:16,header:[1,8,39],uniniti:38,suppli:[0,35,15,7,16],throughout:31,dream:0,backend:27,java:[34,36,9,16],duo:29,stamp:0,close_shard:10,due:[29,5,34,42],empti:[12,6,10,11],implicit:[13,9,16],newcom:14,unambigu:9,blob:6,modern:14,flight:7,fire:15,clariti:[14,36],imag:[0,5,7,25],convert:[0,19,2,38,6,8,27,9,10,11,26,12,42,14,15,16,17],tfidf_corpu:15,gap:[12,42,11,17],coordin:[24,8,18],nameserv:6,repetit:6,show_top:[29,35],look:[0,5,20,29,23,9,34],bill:36,batch:[0,10,27,7],bili:6,"while":[5,27,15,20,16],homework:36,behavior:4,error:[6,11,12,14,15,16],fun:36,real:[15,6,9,10,27],readi:[0,5,27],von:0,jpg:7,itself:[0,31,15,10,16],costli:15,from_corpu:42,print_debug:29,decor:6,obsolet:27,similarity_by_id:10,decod:6,sqlite3:16,multicor:0,higher:9,mountain:0,optim:[34,6,23,16,27],camera:7,painless:23,inflat:5,moment:4,user:[0,5,7,11,41,12,14,15,16],robust:[36,14,16],wherev:[31,14,16,41],"categor\u00eda":[0,5],hyperparamet:35,recent:16,stateless:27,lower:[5,29,6,11,12,14,15],task:23,unicod:[25,6],ampl:5,remove_templ:25,entri:[0,5,36,29,6,11,12,14,15],poland:7,chemistri:0,spent:29,pickl:[1,2,3,4,6,27,10,13,16,20,22,42,34,28,31,32,33,26,35,38,39,25],person:36,"2gb":[0,14],mmwriter:[1,8],five:[15,5,9],academ:[36,15,41],propos:34,lda:[0,27,4,21,22,30,7,9,11,34,12,13,35,15,18],shape:[36,35,10,8],autosess:27,wallclock:[5,7],cup:[0,5,7],scenario:22,theoret:28,tweak:15,input:[0,5,2,29,22,6,7,8,31,36,9,10,11,25,12,26,35,15,16],subsequ:[5,14,15,17],float32:10,indonesian:0,gotobla:23,vendor:23,transpar:[0,29,27,10],big:36,intuit:14,"1h41m":29,resid:[12,22,9,11,34],basicconfig:[0,5,11,12,14,15,16,17],"76g":0,document_id:27,formal:15,printprogress:8,docutil:[1,2,3,4,6,8,10,13,18,29,20,21,22,24,26,27,28,31,32,33,25,35,37,38,39,40,42],any2utf8:6,websit:36,collect:[19,27,22,38,6,23,8,36,9,10,11,12,42,35,16],princip:9,api:[19,30,23,9,11,12,34,14,15,16],filter_extrem:[42,38],popular:[15,14,9,41],"53ghz":[5,7,29],encount:7,often:6,"1st":10,some:[0,29,36,20,22,23,6,7,38,25,33,9,34,31,42,14,15,16,35],dinamo:0,bach:[0,35,15],global:[3,32],unspecifi:10,sampl:[12,28,11],mirror:42,legales:36,word_id:[22,42,2,25,38],scale:[36,4,8,9,10,15,17],chunksiz:[0,5,4,29,6,10,34,35,16],benefit:10,per:[0,1,5,11,12,13,16],mathemat:36,larg:[29,27,6,23,36,9,10,11,12,34,35,15,16,17],stoplist:[12,11],reproduc:0,oversampl:29,machin:[0,5,21,29,23,40,35,10,11,12,34,14,15,16],run:[0,5,36,4,21,23,6,7,24,40,10,41,38,34,35,15,16,17,18],progress_cnt:[1,2,20,8,39,13,26],"2ghz":[5,7],step:[0,4,38,31,9,35,15,16],prerequisit:[23,16,17],copytre:6,selsoviet:0,codestyl:34,prasek:6,materi:6,page:[0,36,7,23,9,41,25,15],idl:23,sysop:0,idf:[0,29,27,3,30,9,15,16,17],"1gb":10,gamma:[35,4],reduct:15,emphasi:29,corpus2dens:8,within:[1,2,20,5,23,39,9,10,26,13,16],is_corpu:6,malaysian:0,elra:36,contributor:36,chang:[0,19,27,7,31,23,8,41,34,35,16],wordi2:2,occupi:[12,11],requestjob:[24,18],ontario:7,span:[1,2,3,4,6,8,10,13,18,19,20,21,22,24,26,27,28,29,31,32,33,25,35,37,38,39,40,42],mkl:23,question:[19,7,9,11,41,12,14,15],"long":16,custom:[3,16],jockei:0,includ:[0,36,23,9,11,41,12,16,17],properli:[0,16],repeatedli:27,doc_3:16,jewish:0,poster:5,doh:36,decomposit:[5,8,29],link:[0,5,19],translat:27,russian:7,line:[1,2,5,22,34,11,26,12,13],isra:0,info:[0,5,38,9,11,34,12,26,14,15,16,17],concaten:27,utf:[25,42,6,38],consist:[5,2,9,11,12,15,16],caller:10,planet:7,highlight:17,similar:[5,27,19,30,6,36,31,9,10,11,12,34,14,15,16,17],irmen:16,curv:9,hash:[19,38],doesn:[5,36,4,38,31,9,11,41,12,16],repres:[36,26,38,9,10,11,42,12,13,14,15,17],splong:9,incomplet:10,save_top:4,grouper:6,titl:36,sequenti:[29,35,4,17],invalid:27,afford:29,kategori:[0,5,7],asymmetr:35,athen:0,william:19,meaning:[12,11],weight_:3,algorithm:[0,5,36,4,19,35,23,33,9,41,29,14,15,16,17],dvd:0,svd:[5,19],pyro:[5,27,6,23,24,41,16,18],tradeoff:29,far:[12,34,11,16],fresh:27,new_vec:[12,11],tagcloud3:19,tagcloud4:19,hello:36,tagcloud6:19,tagcloud7:19,code:[0,29,36,4,22,6,23,8,35,9,11,41,12,34,14,15,16,17],partial:27,edg:15,queri:[19,27,30,6,36,9,10,31,34,14,15,16,17],kilometru:6,edu:[1,35,4],compact:[15,42,9],privat:41,friendli:[12,6,15,11],send:[41,6,16,27],subcorpora:0,simmodel:27,sent:[29,6,35],unzip:41,rollback:[27,16],disclos:36,vec_with_new_id:33,implicitli:[12,27,11],batch_of_docu:10,recip:[6,16],"h\u00e5logaland":0,magic:14,parallel_shard:10,roll:[27,16],scalabl:[34,14,9,10,36],"try":[5,27,23,6,7,36,41,14,35],race:7,maxsiz:[24,6,18],process_wiki:25,copypast:17,tfidfmodel:[30,31,15,3,17],pleas:[15,14,9,7,36],impli:8,smaller:[0,29,38,6,23,10,11,12,34,16],natur:[0,36],focu:34,fold:15,download:[0,5,9,11,41,12],some_unique_str:16,acid:0,outputdir:4,valletta:36,compat:34,index:[1,2,6,8,9,10,11,12,13,14,15,16,17,19,20,22,26,27,28,29,30,31,34,36,39,25],estonian:0,compar:[35,34,14,10,16],veclib:23,cell:7,indian:0,attr_id:31,sixth:0,whatev:[12,22,11,16],iteritem:[12,11],len:[35,6,7,31],topsim:27,let:[0,5,7,11,12,34,14,15,16,17],stop_id:[12,11],hobart:0,becom:[34,14,9,16,27],sinc:[4,38,9,10,25,16],another_tfidf_corpu:15,didn:[0,12,11],survei:[5,11,12,14,15,16],hockeyarch:0,larger:[29,8,16],oblast:0,converg:[0,35],lrec:[36,9],opinion:[12,14,15,11,16],oldsim:27,honor:36,chanc:[16,41],ovat:0,although:[12,26,14,11,31],apr:19,appli:[38,23,10,11,12,42,14,15,17],approxim:[35,15],"kilometr\u016f":6,"boolean":14,simspe:[34,10],fed:22,from:[0,1,2,3,4,5,6,27,9,10,11,12,13,14,15,16,17,18,19,20,22,23,24,25,26,28,29,30,31,32,33,34,35,36,38,39,41,42],commun:[24,14,23,36,18],doubl:15,wordi1:2,upgrad:41,next:[5,27,11,41,12,14,15,17],"5h14m":29,few:34,doubt:14,wiki_corpu:25,sort:[14,15],word_frequ:[42,38],"chomutovsk\u00fdch":6,once_id:[12,11],shard_numb:10,mismatch:15,train:[0,19,2,3,4,29,6,7,27,32,35,9,26,14,15,16],actual:[1,29,6,11,41,12,13,14,15,31],iii:7,numtop:34,account:[29,34],retriev:[0,1,2,20,27,39,9,11,26,12,13],perceiv:[9,11,12,14,15,16],alia:29,annot:16,keep_n:[42,38],obvious:[12,11],wordij:2,wordid:6,sojka:36,control:[0,24,35,18],tau:4,patrol:0,tar:41,"3ghz":0,lock:[6,16,27],sudo:[5,23,16,41],logentropymodel:32,high:[0,14,9,23,36],tag:[26,9,16],tab:[42,38],onlin:[0,19,4,7,33,35,15],serial:[0,1,2,20,5,22,23,6,7,27,31,39,25,29,11,26,12,13,15,16],delai:6,surfac:[12,11],num_nnz:[1,10,8],six:[12,11],token_count:[42,38],brian:0,"13gb":0,instead:[1,2,4,5,29,38,6,7,39,34,25,10,11,26,42,12,13],num_top:[0,5,19,7,28,34,29,14,15,35],sunperf:23,convers:[12,15,11],fare:34,afl:0,overridden:31,singular:29,constructor:[28,29,3,4,22,7,32,24,10,35,18],affero:36,npy:[29,10],dirichlet_expect:[35,4],attent:[12,11],discard:[29,27,33,16],getstat:[24,18],physic:[5,23,16,29],alloc:[0,19,23,6,7,9,30,35,15,16,17],orchestr:[24,18],light:7,correspond:[27,23,9,10,25,35,14],element:[11,12,6,4,8],issu:[10,6,7,16],mainten:19,allow:[28,19,36,20,4,29,22,31,23,8,9,25,35,16],creation:[9,7],movi:[5,7],move:[36,27,16],onepass:29,graph:[5,11,12,14,15,16],"nem\u016f\u017e":6,lsi1m:5,caligari:0,doc_word_count:4,chosen:[29,10,16],decai:[29,35],tst:10,therefor:[5,27,36,9,11,12,35,15,16],pure:[23,16],handl:[2,14,27],auto:16,raion:0,overal:[38,16],handi:17,mention:[12,26,36,11,16],vyssi:6,facilit:14,front:29,bar:6,strive:23,multiprocess:10,locaten:16,somewher:[23,16,41],doc_:16,anyth:[38,6,9,16,27],edit:[0,5,7],sim:[19,14,27,17,34],interview:19,doc2bow:[22,38,11,12,42,14],subset:35,load_from_text:[0,5,42],hyun:0,chunk:[0,29,4,6,10,34,35,16],product:[14,16],related:14,consum:[15,23,16],whitish:0,meta:[0,25],"static":[1,2,20,22,31,8,39,25,10,26,42,13,14],brittl:36,our:[0,5,36,6,9,11,12,14,15,16,17],num_featur:[27,10],wander:0,special:10,out:[0,29,36,4,22,38,25,9,10,11,34,12,42,14,15],variabl:34,vector_by_id:10,contigu:6,bag:[0,1,22,30,33,10,11,38,12,42,15,17],influenc:0,indexedcorpu:[22,30,31,20,25],rep:6,keep_model:27,old2new:33,identifi:23,stub:0,suitabl:0,rel:[29,23],matric:34,common:[22,11,12,42,14,17],insid:[0,12,23,11],undo:[0,5],rhot:35,dictionari:[0,1,2,20,3,5,22,30,6,27,25,10,11,29,38,12,42,14,15,31],releas:[0,19,27,16,34],dirichlet:[0,19,4,23,7,9,30,35,15,16,17],spill:[27,10],could:[29,23,11,12,14,15],ask:[0,16],keep:[5,38,6,11,25,12,42,35,15,17],counterpart:33,length:[3,6,8,32,10,25,15,31,17],doc_bow:[0,35,15],"\u0161\u00e9f":6,south:0,csc_matrix:8,isbn:7,racehors:0,corpus2csc:8,qualiti:[0,17],muhammad:0,chunkize_seri:6,princ:7,utc:0,prioriti:17,salient:29,strict:6,licens:[0,36,9],perfectli:16,accent:6,system:[0,5,6,11,12,14,15,16],messag:[0,5,11,12,14,15,16,17],input_fil:[8,39],full2sparse_clip:8,termin:[5,27,6,24,41,16,18],"final":[5,36,3,29,6,24,9,14],shell:[5,23,16,17],shallow:25,doc_freq:3,diminish:0,exactli:[12,5,15,11],succinctli:9,prune:[12,42,11],structur:[1,2,20,5,29,19,39,9,34,26,13,14,15],charact:[25,2,7,6,17],sens:16,amount:0,exhibit:0,clip:[29,27,25],skip_head:[8,39],clearli:0,robot:0,have:[0,5,2,4,6,27,9,10,11,12,14,15,16,17,23,34,33,42,35,36,41,25],tabl:[12,19,11,17],need:[0,29,27,22,23,6,7,36,31,24,35,9,10,11,41,38,12,34,14,16,18],turn:[29,27,6,7,31,10,11,12,35,15,16],min:29,rout:[16,41],accuraci:29,atla:[5,7,29],which:[0,1,2,3,5,6,8,9,10,11,12,13,14,15,16,17,18,29,20,22,23,24,25,26,27,28,31,32,34,36,38,39,41,42],conclud:34,documenti:2,singh:0,tracklist:0,mit:4,singl:[0,1,27,4,21,5,6,23,8,40,9,10,11,41,12,29,14,35],unstructur:9,unless:[34,10,25],preliminari:17,mimick:[12,11],shardsiz:[27,10],who:36,discov:[15,9,23],fragment:17,eight:5,medal:0,why:[14,23,17],document1:2,document2:2,dens:[10,8],request:[29,27,6,7,10,34,35,15,16],uri:[24,18],capita:0,bui:23,bibtex:[36,9],determin:[28,19,6,35,10,25,14,31],iter_chunk:10,someth:[36,6,31],fact:[0,5,36,11,12,14],affair:0,lebanes:0,text:[0,19,2,22,38,6,27,36,9,28,11,25,12,42,14,16],bring:15,empir:[15,7],trane:9,freestyl:0,rough:15,trivial:[15,5,35,9,16],anywai:[34,41],redirect:25,footbal:[0,5,7],locat:[27,10,16],nois:[0,15],forev:23,incident:[5,14,15],should:[0,5,29,31,8,24,10,11,41,12,16,18],lemma:[25,6],reign:0,esqu:[34,36],rabbi:0,local:[36,3,6,32,8,41,16],index_docu:27,meant:[6,16],insight:6,contribut:[0,5,36,29,14,15],notat:[29,31],beat:[12,11],familiar:17,pypi:16,antarct:0,increas:[29,21,38,40,42,15],enabl:29,organ:[16,17],chunkiz:6,stuff:[23,16,17],she:0,contain:[0,1,27,37,29,6,8,33,10,26,31],view:[12,6,11],toe:16,legaci:4,knowledg:36,popup:0,jobdon:[24,18],"10h56m":7,bytestr:[25,6],"tis\u00edc":6,appreci:14,saveastext:25,asynchron:[24,18],statu:16,frequency_:[3,32],correctli:22,record:[1,2,20,7,39,13,26],pattern:[25,6,9],corpus1m:5,written:8,another_corpu:19,gzipfil:8,censu:[0,5],crude:16,chemic:0,email:[7,41],kei:[42,6,14,27,38],itertool:6,job:[23,5,24,7,18],entir:[0,27,6,8,10,11,12,15,16],addit:[0,1,2,20,5,29,8,39,35,34,26,13,14,15,16],float64:[29,8],admin:0,goal:[14,15,7],equal:35,drug:7,lda_work:[30,7,21],etc:[0,1,2,20,34,22,38,6,25,27,39,24,9,26,42,13,14,15,16,17,18],eta:[35,4],grain:23,any2unicod:6,comment:[5,14,26],wall:29,wonder:15,arriv:[12,15,11],walk:[12,14,11],rpc:[23,16],respect:9,uci:[0,1,30],quit:[0,14,15,23,41],save_corpu:[1,2,20,22,31,39,25,13,26],compon:3,json:16,hindi:0,"0rc2":41,"0rc1":41,immedi:[0,12,38,11,29],repeatcorpu:[5,6],vec_by_id:27,present:[26,36,15,8],multi:[29,15],novel:[15,7],query_shard:10,plain:[0,27,20,22,38,36,9,11,12,25,16],defin:[27,6,11,12,14,16],dense2vec:8,tokenid:[12,11],observ:[29,14,15],aldil:6,conclus:9,helper:[22,8],almost:[25,34,4,16],site:41,hashdictionari:[30,38],archiv:[0,1,7],motiv:[36,9],dual:[5,29],incom:[0,5],greater:[0,14,41],welcom:[0,34,14,9],getn:6,japanes:7,cross:[0,16],sqrt:3,python:[0,5,36,20,21,19,6,23,24,40,9,10,11,41,12,34,14,15,16,17,18],smarter:[34,6],und:0,failur:[14,15,16],probabl:[14,15,35,17],infer:[19,4,9,35,15,16],difficult:25,jobsdon:[24,18],http:[0,1,2,4,29,38,6,7,36,41,25,26,35],hostnam:6,ircbot:6,effect:29,coffe:0,dai:[36,7],filter_wiki:25,student:36,"0ghz":29,expand:0,total_doc:3,off:[36,29,27,16],fieldid:31,indonesia:0,well:[28,5,36,20,22,31,35,9,10,11,41,12,25,14,15,16],thought:[19,9],govern:0,command:17,interpol:35,tfidf_vec:15,soviet:7,latest:[0,27,10,16],docpo:10,distanc:15,less:[29,36,38,8,42,15],dense2corpu:8,obtain:[0,6,14],tcp:[23,16],increasingli:0,simpli:[5,27,6,23,31,9,10,11,41,12,15,16,17],simultan:16,save_as_text:[42,38],sku:6,web:[0,36,35,9,16],"ch\u00e2teau":0,wiki_en_wordid:[0,5],simserv:[36,30,6,16,27],lake:0,add:[0,5,27,3,29,8,10,16],book:7,kick:16,svmlight:[20,22,30,11,12,26],dest:6,piec:[34,36,10,25],camelcas:34,realiz:[27,3,31,32,9,36,6],xe0:6,know:[5,7,9,41,34,14,15,16,17],unseen:[0,29,2,4,35],press:7,recurs:6,sparsiti:35,doc_word_id:4,like:[0,5,27,29,22,6,7,8,36,9,10,11,12,34,14,15,16],success:[5,6,9],unord:[12,14,15,11,16],necessari:[38,36,9],soft:15,expect_log_stick:4,revers:6,twitter:6,linux:[5,7,41],hail:9,flush:27,proper:8,guarante:35,vsm:[15,9,17],librari:[0,23,36,7,16],tmp:[28,5,3,32,10,11,12,14,15,16],lead:17,avoid:[29,6],lean:35,joachim:[12,26,11],estim:[0,29,2,4,35,15],filesystem:8,prefectur:7,random_suffix:6,mode:[29,5,23,7,35,15],usag:[0,24,40,21,18],symlink:41,word_weight:22,host:[6,36],totaldoc:3,getstream:22,update_map:27,offset:[1,2,20,8,39,13,26],html_decod:6,about:[0,5,36,19,7,9,11,12,34,15,16],rare:[15,17],socket:[6,23,16],column:8,chomutovskych:6,list_of_word:2,rehurek_lrec:36,bz2:[0,5,25,8,22],emperor:7,metr:0,own:[9,3,16],easy_instal:[5,23,16,41],automat:[1,2,20,39,22,31,27,32,25,9,10,34,26,13,35,15,16],dataset:[0,1,2,15],print_top:[0,5,4,29,7,15],max_chunk:4,scipy2spars:8,maccabi:0,merg:[24,29,27,35,18],pictur:7,transfer:[24,18],ucicorpu:[1,30],hassl:36,stai:[25,41],"function":[1,2,3,4,6,8,10,11,12,13,14,15,18,20,22,24,25,34,27,30,31,26,35,36,38,39,42],"\u0159eh\u016f\u0159ek":15,aircraft:7,wrestl:0,matrixmarket:25,templ:0,spuriou:29,technician:36,highest:6,eau:6,bug:[29,14,10,36],count:[0,28,3,32,33,11,26,12,42,15,16,17],made:[36,27,8],wise:[9,17],temp:10,whether:[6,9,32],wish:[40,41,21],dml:36,displai:6,troubl:16,sockpuppet:0,directori:[27,6,11,41,12,34,16],below:[12,23,11],testfil:20,otherwis:[5,35,25],problem:5,log1p:3,sessionserv:[27,16],"int":6,dure:[27,15,23,8,16],run_serv:16,linguist:16,implement:[0,1,27,37,22,38,7,36,9,11,12,42,14,15,17],word_utf8:42,log_bas:3,some_doc:[28,3,32],pyro_daemon:6,bzip2:0,vivrai:6,transformationabc:31,detail:[25,14,15,16,17],virtual:27,other:[35,19,27,3,29,6,36,9,10,11,41,25,12,34,14,15,16],lookup:[6,16],futur:[12,34,7,11],rememb:[14,16],stat:[1,35],repeat:[0,5,6,15],star:7,adddocu:34,"class":[1,2,3,4,6,8,10,11,12,13,14,18,19,20,21,22,24,25,26,27,28,29,31,32,33,34,35,37,38,39,40,42],semin:14,max_tim:4,num_doc:[1,10,8],matrix:[0,19,3,29,22,30,8,32,39,10,11,12,42,28],snippet:6},objtypes:{"0":"py:module","1":"py:function","2":"py:method","3":"py:class","4":"py:classmethod","5":"py:staticmethod"},titles:["Experiments on the English Wikipedia","corpora.ucicorpus – Corpus in UCI bag-of-words format","corpora.lowcorpus – Corpus in List-of-Words format","models.tfidfmodel – TF-IDF model","models.hdpmodel – Hierarchical Dirichlet Process","Distributed Latent Semantic Analysis","utils – Various utility functions","Distributed Latent Dirichlet Allocation","matutils – Math utils","Introduction","similarities.docsim – Document similarity queries","Use Cases and Examples","Corpora and Vector Spaces","corpora.bleicorpus – Corpus in Blei’s LDA-C format","Similarity Queries","Topics and Transformations","Document Similarity Server","Tutorials","models.lda_dispatcher – Dispatcher for distributed LDA","Gensim – Topic Modelling for Humans","corpora.indexedcorpus – Random access to corpus documents","models.lda_worker – Worker for distributed LDA","corpora.textcorpus – Building corpora with dictionaries","Distributed Computing","models.lsi_dispatcher – Dispatcher for distributed LSI","corpora.wikicorpus – Corpus from a Wikipedia dump","corpora.svmlightcorpus – Corpus in SVMlight format","simserver – Document similarity server","models.rpmodel – Random Projections","models.lsimodel – Latent Semantic Indexing","API Reference","interfaces – Core gensim interfaces","models.logentropy_model – LogEntropy model","models – Package for transformation models","Change Set for 0.8.0","models.ldamodel – Latent Dirichlet Allocation","About","corpora – Package for corpora I/O","corpora.hashdictionary – Construct word<->id mappings","corpora.mmcorpus – Corpus in Matrix Market format","models.lsi_worker – Worker for distributed LSI","Installation","corpora.dictionary – Construct word<->id mappings"],objnames:{"0":["py","module","Python module"],"1":["py","function","Python function"],"2":["py","method","Python method"],"3":["py","class","Python class"],"4":["py","classmethod","Python class method"],"5":["py","staticmethod","Python static method"]},filenames:["wiki","corpora/ucicorpus","corpora/lowcorpus","models/tfidfmodel","models/hdpmodel","dist_lsi","utils","dist_lda","matutils","intro","similarities/docsim","examples","tut1","corpora/bleicorpus","tut3","tut2","simserver","tutorial","models/lda_dispatcher","index","corpora/indexedcorpus","models/lda_worker","corpora/textcorpus","distributed","models/lsi_dispatcher","corpora/wikicorpus","corpora/svmlightcorpus","similarities/simserver","models/rpmodel","models/lsimodel","apiref","interfaces","models/logentropy_model","models/models","changes_080","models/ldamodel","about","corpora/corpora","corpora/hashdictionary","corpora/mmcorpus","models/lsi_worker","install","corpora/dictionary"]}) \ No newline at end of file diff --git a/docs/similarities/docsim.html b/docs/similarities/docsim.html deleted file mode 100644 index 237ab1c335..0000000000 --- a/docs/similarities/docsim.html +++ /dev/null @@ -1,414 +0,0 @@ - - - - - - - - - - similarities.docsim – Document similarity queries — gensim - - - - - - - - - - - - - - - - - - - - - - - - - - - -
    -

    Navigation

    - -
    - - -
    -
    -

    Table Of Contents

    - - -

    Previous topic

    -

    models.lda_worker – Worker for distributed LDA

    -

    Next topic

    -

    simserver – Document similarity server

    - - -
    -
    - - - - -
    -
    -
    -
    - -
    -

    similarities.docsim – Document similarity queries

    -

    This module contains functions and classes for computing similarities across -a collection of documents in the Vector Space Model.

    -

    The main class is Similarity, which builds an index for a given set of documents. -Once the index is built, you can perform efficient queries like “Tell me how similar -is this query document to each document in the index?”. The result is a vector -of numbers as large as the size of the initial set of documents, that is, one float -for each index document. Alternatively, you can also request only the top-N most -similar index documents to the query.

    -

    You can later add new documents to the index via Similarity.add_documents().

    -
    -

    How It Works

    -

    The Similarity class splits the index into several smaller sub-indexes (“shards”), -which are disk-based. If your entire index fits in memory (~hundreds of thousands -documents for 1GB of RAM), you can also use the MatrixSimilarity or SparseMatrixSimilarity -classes directly. These are more simple but do not scale well (the entire index is -kept in RAM).

    -

    Once the index has been initialized, you can query for document similarity simply by:

    -
    >>> index = Similarity('/tmp/tst', corpus, num_features=12) # build the index
->>> similarities = index[query] # get similarities between the query and all index documents
-
    -
    -

    If you have more query documents, you can submit them all at once, in a batch:

    -
    >>> for similarities in index[batch_of_documents]: # the batch is simply an iterable of documents (=gensim corpus)
->>>     ...
-
    -
    -

    The benefit of this batch (aka “chunked”) querying is much better performance. -To see the speed-up on your machine, run python -m gensim.test.simspeed -(compare to my results here).

    -

    There is also a special syntax for when you need similarity of documents in the index -to the index itself (i.e. query=index documents themselves). This special syntax -already uses the faster, batch queries internally:

    -
    >>> for similarities in index: # return all similarities of the 1st index document, then 2nd...
->>>     ...
-
    -
    -
    -
    -class gensim.similarities.docsim.MatrixSimilarity(corpus, num_best=None, dtype=<type 'numpy.float32'>, num_features=None, chunksize=256)
    -

    Compute similarity against a corpus of documents by storing the index matrix -in memory. The similarity measure used is cosine between two vectors.

    -

    Use this if your input corpus contains dense vectors (such as documents in LSI -space) and fits into RAM.

    -

    The matrix is internally stored as a dense numpy array. Unless the entire matrix -fits into main memory, use Similarity instead.

    -

    See also Similarity and SparseMatrixSimilarity in this module.

    -

    num_features is the number of features in the corpus (will be determined -automatically by scanning the corpus if not specified). See Similarity -class for description of the other parameters.

    -
    -
    -get_similarities(query)
    -

    Return similarity of sparse vector query to all documents in the corpus, -as a numpy array.

    -

    If query is a collection of documents, return a 2D array of similarities -of each document in query to all documents in the corpus (=batch query, -faster than processing each document in turn).

    -

    Do not use this function directly; use the self[query] syntax instead.

    -
    - -
    -
    -classmethod load(fname)
    -

    Load a previously saved object from file (also see save).

    -
    - -
    -
    -save(fname)
    -

    Override the default save (which uses cPickle), because that’s -too inefficient and cPickle has bugs. Instead, single out the large index -matrix and store that separately in binary format (that can be directly -mmap’ed), under fname.npy. The rest of the object is pickled to fname.

    -
    - -
    - -
    -
    -class gensim.similarities.docsim.Shard(fname, index)
    -

    A proxy class that represents a single shard instance within a Similarity -index.

    -

    Basically just wraps (Sparse)MatrixSimilarity so that it mmaps from disk on -request (query).

    -
    -
    -get_document_id(pos)
    -

    Return index vector at position pos.

    -

    The vector is of the same type as the underlying index (ie., dense for -MatrixSimilarity and scipy.sparse for SparseMatrixSimilarity.

    -
    - -
    -
    -classmethod load(fname)
    -

    Load a previously saved object from file (also see save).

    -
    - -
    -
    -save(fname)
    -

    Save the object to file via pickling (also see load).

    -
    - -
    - -
    -
    -class gensim.similarities.docsim.Similarity(output_prefix, corpus, num_features, num_best=None, chunksize=1024, shardsize=32768)
    -

    Compute cosine similarity of a dynamic query against a static corpus of documents -(“the index”).

    -

    Scalability is achieved by sharding the index into smaller pieces, each of which -fits into core memory (see the (Sparse)MatrixSimilarity classes in this module). -The shards themselves are simply stored as files to disk and mmap’ed back as needed.

    -

    Construct the index from corpus. The index can be later extended by calling -the add_documents method. Note: documents are split (internally, transparently) -into shards of shardsize documents each, converted to a matrix, for faster BLAS calls. -Each shard is stored to disk under output_prefix.shard_number (=you need write -access to that location). If you don’t specify an output prefix, a random -filename in temp will be used.

    -

    shardsize should be chosen so that a shardsize x chunksize matrix of floats -fits comfortably into main memory.

    -

    num_features is the number of features in the corpus (e.g. size of the -dictionary, or the number of latent topics for latent semantic models).

    -

    If num_best is left unspecified, similarity queries will return a full -vector with one float for every document in the index:

    -
    >>> index = Similarity('/path/to/index', corpus, num_features=400) # if corpus has 7 documents...
->>> index[query] # ... then result will have 7 floats
-[0.0, 0.0, 0.2, 0.13, 0.8, 0.0, 0.1]
-
    -
    -

    If num_best is set, queries return only the num_best most similar documents, -always leaving out documents for which the similarity is 0. -If the input vector itself only has features with zero values (=the sparse -representation is empty), the returned list will always be empty.

    -
    >>> index.num_best = 3
->>> index[query] # return at most "num_best" of `(index_of_document, similarity)` tuples
-[(4, 0.8), (2, 0.13), (3, 0.13)]
-
    -
    -

    You can also override num_best dynamically, simply by setting e.g. -self.num_best = 10 before doing a query.

    -
    -
    -add_documents(corpus)
    -

    Extend the index with new documents.

    -

    Internally, documents are buffered and then spilled to disk when there’s -self.shardsize of them (or when a query is issued).

    -
    - -
    -
    -close_shard()
    -

    Force the latest shard to close (be converted to a matrix and stored -to disk). Do nothing if no new documents added since last call.

    -

    NOTE: the shard is closed even if it is not full yet (its size is smaller -than self.shardsize). If documents are added later via add_documents(), -this incomplete shard will be loaded again and completed.

    -
    - -
    -
    -iter_chunks(chunksize=None)
    -

    Iteratively yield the index as chunks of documents, each of size <= chunksize.

    -

    The chunk is returned in its raw form (matrix or sparse matrix slice). -The size of the chunk may be smaller than requested; it is up to the caller -to check the result for real length, using chunk.shape[0].

    -
    - -
    -
    -classmethod load(fname)
    -

    Load a previously saved object from file (also see save).

    -
    - -
    -
    -query_shards(query)
    -

    Return the result of applying shard[query] for each shard in self.shards, -as a sequence.

    -

    If PARALLEL_SHARDS is set, the shards are queried in parallel, using -the multiprocessing module.

    -
    - -
    -
    -save(fname=None)
    -

    Save the object via pickling (also see load) under filename specified in -the constructor.

    -

    Calls close_shard internally to spill any unfinished shards to disk first.

    -
    - -
    -
    -similarity_by_id(docpos)
    -

    Return similarity of the given document only. docpos is the position -of the query document within index.

    -
    - -
    -
    -vector_by_id(docpos)
    -

    Return indexed vector corresponding to the document at position docpos.

    -
    - -
    - -
    -
    -class gensim.similarities.docsim.SparseMatrixSimilarity(corpus, num_features=None, num_terms=None, num_docs=None, num_nnz=None, num_best=None, chunksize=500, dtype=<type 'numpy.float32'>)
    -

    Compute similarity against a corpus of documents by storing the sparse index -matrix in memory. The similarity measure used is cosine between two vectors.

    -

    Use this if your input corpus contains sparse vectors (such as documents in -bag-of-words format) and fits into RAM.

    -

    The matrix is internally stored as a scipy.sparse.csr matrix. Unless the entire -matrix fits into main memory, use Similarity instead.

    -

    See also Similarity and MatrixSimilarity in this module.

    -
    -
    -get_similarities(query)
    -

    Return similarity of sparse vector query to all documents in the corpus, -as a numpy array.

    -

    If query is a collection of documents, return a 2D array of similarities -of each document in query to all documents in the corpus (=batch query, -faster than processing each document in turn).

    -

    Do not use this function directly; use the self[query] syntax instead.

    -
    - -
    -
    -classmethod load(fname)
    -

    Load a previously saved object from file (also see save).

    -
    - -
    -
    -save(fname)
    -

    Override the default save (which uses cPickle), because that’s -too inefficient and cPickle has bugs. Instead, single out the large internal -arrays and store them separately in binary format (that can be directly -mmap’ed), under fname.array_name.npy.

    -
    - -
    - -
    -
    - - -
    -
    -
    -
    -
    - - -
    -

    Navigation

    - -
    - - -
    - © Copyright 2009-2012, Radim Řehůřek <radimrehurek(at)seznam.cz>. - Last updated on Sep 16, 2012. -
    - - \ No newline at end of file diff --git a/docs/similarities/simserver.html b/docs/similarities/simserver.html deleted file mode 100644 index 7d07ab9e88..0000000000 --- a/docs/similarities/simserver.html +++ /dev/null @@ -1,517 +0,0 @@ - - - - - - - - - - simserver – Document similarity server — gensim - - - - - - - - - - - - - - - - - - - - - - - - - - -
    -

    Navigation

    - -
    - - -
    -
    -

    Previous topic

    -

    similarities.docsim – Document similarity queries

    - - -
    -
    - - - - -
    -
    -
    -
    - -
    -

    simserver – Document similarity server

    -

    “Find similar” service, using gensim (=vector spaces) for backend.

    -

    The server performs 3 main functions:

    -
      -
    1. converts documents to semantic representation (TF-IDF, LSA, LDA...)
      2. -
    2. indexes documents in the vector representation, for faster retrieval
      3. -
    3. for a given query document, return ids of the most similar documents from the index
      4. -
    -

    SessionServer objects are transactional, so that you can rollback/commit an entire -set of changes.

    -

    The server is ready for concurrent requests (thread-safe). Indexing is incremental -and you can query the SessionServer even while it’s being updated, so that there -is virtually no down-time.

    -
    -
    -class simserver.simserver.SessionServer(basedir, autosession=True, use_locks=True)
    -

    Similarity server on top of SimServer that implements sessions = transactions.

    -

    A transaction is a set of server modifications (index/delete/train calls) that -may be either committed or rolled back entirely.

    -

    Sessions are realized by:

    -
      -
    1. cloning (=copying) a SimServer at the beginning of a session
      2. -
    2. serving read-only queries from the original server (the clone may be modified during queries)
      3. -
    3. modifications affect only the clone
      4. -
    4. at commit, the clone becomes the original
      5. -
    5. at rollback, do nothing (clone is discarded, next transaction starts from the original again)
      6. -
    -
    -
    -buffer(*args, **kwargs)
    -

    Buffer documents, in the current session

    -
    - -
    -
    -check_session(*args, **kwargs)
    -

    Make sure a session is open.

    -

    If it’s not and autosession is turned on, create a new session automatically. -If it’s not and autosession is off, raise an exception.

    -
    - -
    -
    -close(*args, **kwargs)
    -

    Don’t wait for gc, try to release important resources manually

    -
    - -
    -
    -commit(*args, **kwargs)
    -

    Commit changes made by the latest session.

    -
    - -
    -
    -delete(*args, **kwargs)
    -

    Delete documents from the current session.

    -
    - -
    -
    -drop_index(*args, **kwargs)
    -

    Drop all indexed documents from the session. Optionally, drop model too.

    -
    - -
    -
    -find_similar(*args, **kwargs)
    -

    Find similar articles.

    -

    With autosession off, use the index state before current session started, -so that changes made in the session will not be visible here. With autosession -on, close the current session first (so that session changes are committed -and visible).

    -
    - -
    -
    -index(*args, **kwargs)
    -

    Index documents, in the current session

    -
    - -
    -
    -classmethod load(fname)
    -

    Load a previously saved object from file (also see save).

    -
    - -
    -
    -open_session(*args, **kwargs)
    -

    Open a new session to modify this server.

    -

    You can either call this fnc directly, or turn on autosession which will -open/commit sessions for you transparently.

    -
    - -
    -
    -optimize(*args, **kwargs)
    -

    Optimize index for faster by-document-id queries.

    -
    - -
    -
    -rollback(*args, **kwargs)
    -

    Ignore all changes made in the latest session (terminate the session).

    -
    - -
    -
    -save(fname)
    -

    Save the object to file via pickling (also see load).

    -
    - -
    -
    -set_autosession(*args, **kwargs)
    -

    Turn autosession (automatic committing after each modification call) on/off. -If value is None, only query the current value (don’t change anything).

    -
    - -
    -
    -terminate(*args, **kwargs)
    -

    Delete all files created by this server, invalidating self. Use with care.

    -
    - -
    -
    -train(*args, **kwargs)
    -

    Update semantic model, in the current session.

    -
    - -
    - -
    -
    -class simserver.simserver.SimIndex(fname, num_features, shardsize=65536, topsims=100)
    -

    An index of documents. Used internally by SimServer.

    -

    It uses the Similarity class to persist all document vectors to disk (via mmap).

    -

    Spill index shards to disk after every shardsize documents. -In similarity queries, return only the topsims most similar documents.

    -
    -
    -close()
    -

    Explicitly release important resources (file handles, db, ...)

    -
    - -
    -
    -delete(docids)
    -

    Delete documents (specified by their ids) from the index.

    -
    - -
    -
    -index_documents(fresh_docs, model)
    -

    Update fresh index with new documents (potentially replacing old ones with -the same id). fresh_docs is a dictionary-like object (=dict, sqlitedict, shelve etc) -that maps document_id->document.

    -
    - -
    -
    -merge(other)
    -

    Merge documents from the other index. Update precomputed similarities -in the process.

    -
    - -
    -
    -sims2scores(sims, eps=1e-07)
    -

    Convert raw similarity vector to a list of (docid, similarity) results.

    -
    - -
    -
    -sims_by_id(docid)
    -

    Find the most similar documents to the (already indexed) document with docid.

    -
    - -
    -
    -sims_by_vec(vec, normalize=None)
    -

    Find the most similar documents to a given vector (=already processed document).

    -
    - -
    -
    -terminate()
    -

    Delete all files created by this index, invalidating self. Use with care.

    -
    - -
    -
    -update_ids(docids)
    -

    Update id->pos mapping with new document ids.

    -
    - -
    -
    -update_mappings()
    -

    Synchronize id<->position mappings.

    -
    - -
    -
    -vec_by_id(docid)
    -

    Return indexed vector corresponding to document docid.

    -
    - -
    - -
    -
    -class simserver.simserver.SimModel(fresh_docs, dictionary=None, method='lsi')
    -

    A semantic model responsible for translating between plain text and (semantic) -vectors.

    -

    These vectors can then be indexed/queried for similarity, see the SimIndex -class. Used internally by SimServer.

    -

    Train a model, using fresh_docs as training corpus.

    -

    If dictionary is not specified, it is computed from the documents.

    -

    method is currently one of “tfidf”/”lsi”/”lda”.

    -
    -
    -close()
    -

    Release important resources manually.

    -
    - -
    -
    -doc2vec(doc)
    -

    Convert a single SimilarityDocument to vector.

    -
    - -
    -
    -docs2vecs(docs)
    -

    Convert multiple SimilarityDocuments to vectors (batch version of doc2vec).

    -
    - -
    -
    -classmethod load(fname)
    -

    Load a previously saved object from file (also see save).

    -
    - -
    -
    -save(fname)
    -

    Save the object to file via pickling (also see load).

    -
    - -
    - -
    -
    -class simserver.simserver.SimServer(basename, use_locks=True)
    -

    Top-level functionality for similarity services. A similarity server takes -care of:

    -
      -
    1. creating semantic models
      2. -
    2. indexing documents using these models
      3. -
    3. finding the most similar documents in an index.
      4. -
    -

    An object of this class can be shared across network via Pyro, to answer remote -client requests. It is thread safe. Using a server concurrently from multiple -processes is safe for reading = answering similarity queries. Modifying -(training/indexing) is realized via locking = serialized internally.

    -

    All data will be stored under directory basename. If there is a server -there already, it will be loaded (resumed).

    -

    The server object is stateless in RAM – its state is defined entirely by its location. -There is therefore no need to store the server object.

    -
    -
    -buffer(*args, **kwargs)
    -

    Add a sequence of documents to be processed (indexed or trained on).

    -

    Here, the documents are simply collected; real processing is done later, -during the self.index or self.train calls.

    -

    buffer can be called repeatedly; the result is the same as if it was -called once, with a concatenation of all the partial document batches. -The point is to save memory when sending large corpora over network: the -entire documents must be serialized into RAM. See utils.upload_chunked().

    -

    A call to flush() clears this documents-to-be-processed buffer (flush -is also implicitly called when you call index() and train()).

    -
    - -
    -
    -close()
    -

    Explicitly close open file handles, databases etc.

    -
    - -
    -
    -delete(*args, **kwargs)
    -

    Delete specified documents from the index.

    -
    - -
    -
    -drop_index(*args, **kwargs)
    -

    Drop all indexed documents. If keep_model is False, also dropped the model.

    -
    - -
    -
    -find_similar(doc, min_score=0.0, max_results=100)
    -

    Find at most max_results most similar articles in the index, -each having similarity score of at least min_score.

    -

    doc is either a string (document id, previously indexed) or a -dict containing a ‘tokens’ key. These tokens are processed to produce a -vector, which is then used as a query.

    -

    The similar documents are returned in decreasing similarity order, as -(doc_id, doc_score) pairs.

    -
    - -
    -
    -flush(*args, **kwargs)
    -

    Commit all changes, clear all caches.

    -
    - -
    -
    -index(*args, **kwargs)
    -

    Permanently index all documents previously added via buffer, or -directly index documents from corpus, if specified.

    -

    The indexing model must already exist (see train) before this function -is called.

    -
    - -
    -
    -keys()
    -

    Return ids of all indexed documents.

    -
    - -
    -
    -optimize(*args, **kwargs)
    -

    Precompute top similarities for all indexed documents. This speeds up -find_similar queries by id (but not queries by fulltext).

    -

    Internally, documents are moved from a fresh index (=no precomputed similarities) -to an optimized index (precomputed similarities). Similarity queries always -query both indexes, so this split is transparent to clients.

    -

    If you add documents later via index, they go to the fresh index again. -To precompute top similarities for these new documents too, simply call -optimize again.

    -
    - -
    -
    -train(*args, **kwargs)
    -

    Create an indexing model. Will overwrite the model if it already exists. -All indexes become invalid, because documents in them use a now-obsolete -representation.

    -

    The model is trained on documents previously entered via buffer, -or directly on corpus, if specified.

    -
    - -
    - -
    -
    -simserver.simserver.merge_sims(oldsims, newsims, clip=100)
    -

    Merge two precomputed similarity lists, truncating the result to clip most similar items.

    -
    - -
    - - -
    -
    -
    -
    -
    - - -
    -

    Navigation

    - -
    - - -
    - © Copyright 2009-2012, Radim Řehůřek <radimrehurek(at)seznam.cz>. - Last updated on Sep 16, 2012. -
    - - \ No newline at end of file diff --git a/docs/simserver.html b/docs/simserver.html deleted file mode 100644 index 3710570d95..0000000000 --- a/docs/simserver.html +++ /dev/null @@ -1,462 +0,0 @@ - - - - - - - - - - Document Similarity Server — gensim - - - - - - - - - - - - - - - - - - - - - - - - -
    -

    Navigation

    - -
    - - -
    -
    -

    Table Of Contents

    - - - - -
    -
    - - - - -
    -
    -
    -
    - -
    -

    Document Similarity Server

    -

    The 0.7.x series of gensim was about improving performance and consolidating API. -0.8.x will be about new features — 0.8.1, first of the series, is a document similarity service.

    -

    The source code itself has been moved from gensim to its own, dedicated package, named simserver. -Get it from PyPI or clone it on Github.

    -
    -

    What is a document similarity service?

    -

    Conceptually, a service that lets you :

    -
      -
    1. train a semantic model from a corpus of plain texts (no manual annotation and mark-up needed)
      2. -
    2. index arbitrary documents using this semantic model
      3. -
    3. query the index for similar documents (the query can be either an id of a document already in the index, or an arbitrary text)
      4. -
    -
    >>> from simserver import SessionServer
->>> server = SessionServer('/tmp/my_server') # resume server (or create a new one)
-
    -
    -
    >>> server.train(training_corpus, method='lsi') # create a semantic model
->>> server.index(some_documents) # convert plain text to semantic representation and index it
->>> server.find_similar(query) # convert query to semantic representation and compare against index
->>> ...
->>> server.index(more_documents) # add to index: incremental indexing works
->>> server.find_similar(query)
->>> ...
->>> server.delete(ids_to_delete) # incremental deleting also works
->>> server.find_similar(query)
->>> ...
-
    -
    -
    -

    Note

    -

    “Semantic” here refers to semantics of the crude, statistical type – -Latent Semantic Analysis, -Latent Dirichlet Allocation etc. -Nothing to do with the semantic web, manual resource tagging or detailed linguistic inference.

    -
    -
    -
    -

    What is it good for?

    -

    Digital libraries of (mostly) text documents. More generally, it helps you annotate, -organize and navigate documents in a more abstract way, compared to plain keyword search.

    -
    -
    -

    How is it unique?

    -
      -
    1. Memory independent. Gensim has unique algorithms for statistical analysis that allow -you to create semantic models of arbitrarily large training corpora (larger than RAM) very quickly -and in constant RAM.
      2. -
    2. Memory independent (again). Indexing shards are stored as files to disk/mmapped back as needed, -so you can index very large corpora. So again, constant RAM, this time independent of the number of indexed documents.
      3. -
    3. Efficient. Gensim makes heavy use of Python’s NumPy and SciPy libraries to make indexing and -querying efficient.
      4. -
    4. Robust. Modifications of the index are transactional, so you can commit/rollback an -entire indexing session. Also, during the session, the service is still available -for querying (using its state from when the session started). Power failures leave -service in a consistent state (implicit rollback).
      5. -
    5. Pure Python. Well, technically, NumPy and SciPy are mostly wrapped C and Fortran, but -gensim itself is pure Python. No compiling, installing or root priviledges needed.
      6. -
    6. Concurrency support. The underlying service object is thread-safe and can -therefore be used as a daemon server: clients connect to it via RPC and issue train/index/query requests remotely.
      7. -
    7. Cross-network, cross-platform and cross-language. While the Python server runs -over TCP using Pyro, -clients in Java/.NET are trivial thanks to Pyrolite.
      8. -
    -

    The rest of this document serves as a tutorial explaining the features in more detail.

    -
    -
    -
    -

    Prerequisites

    -

    It is assumed you have gensim properly installed. You’ll also -need the sqlitedict package that wraps -Python’s sqlite3 module in a thread-safe manner:

    -
    $ sudo easy_install -U sqlitedict
    -
    -

    To test the remote server capabilities, install Pyro4 (Python Remote Objects, at -version 4.8 as of this writing):

    -
    $ sudo easy_install Pyro4
    -
    -
    -

    Note

    -

    Don’t forget to initialize logging to see logging messages:

    -
    >>> import logging
->>> logging.basicConfig(format='%(asctime)s : %(levelname)s : %(message)s', level=logging.INFO)
-
    -
    -
    -
    -
    -

    What is a document?

    -

    In case of text documents, the service expects:

    -
    >>> document = {'id': 'some_unique_string',
->>>             'tokens': ['content', 'of', 'the', 'document', '...'],
->>>             'other_fields_are_allowed_but_ignored': None}
-
    -
    -

    This format was chosen because it coincides with plain JSON and is therefore easy to serialize and send over the wire, in almost any language. -All strings involved must be utf8-encoded.

    -
    -
    -

    What is a corpus?

    -

    A sequence of documents. Anything that supports the for document in corpus: ... -iterator protocol. Generators are ok. Plain lists are also ok (but consume more memory).

    -
    >>> from gensim import utils
->>> texts = ["Human machine interface for lab abc computer applications",
->>>          "A survey of user opinion of computer system response time",
->>>          "The EPS user interface management system",
->>>          "System and human system engineering testing of EPS",
->>>          "Relation of user perceived response time to error measurement",
->>>          "The generation of random binary unordered trees",
->>>          "The intersection graph of paths in trees",
->>>          "Graph minors IV Widths of trees and well quasi ordering",
->>>          "Graph minors A survey"]
->>> corpus = [{'id': 'doc_%i' % num, 'tokens': utils.simple_preprocess(text)}
->>>           for num, text in enumerate(texts)]
-
    -
    -

    Since corpora are allowed to be arbitrarily large, it is -recommended client splits them into smaller chunks before uploading them to the server:

    -
    >>> utils.upload_chunked(server, corpus, chunksize=1000) # send 1k docs at a time
-
    -
    -
    -
    -

    Wait, upload what, where?

    -

    If you use the similarity service object (instance of simserver.SessionServer) in -your code directly—no remote access—that’s perfectly fine. Using the service remotely, from a different process/machine, is an -option, not a necessity.

    -

    Document similarity can also act as a long-running service, a daemon process on a separate machine. In that -case, I’ll call the service object a server.

    -

    But let’s start with a local object. Open your favourite shell and:

    -
    >>> from gensim import utils
->>> from simserver import SessionServer
->>> service = SessionServer('/tmp/my_server/') # or wherever
-
    -
    -

    That initialized a new service, located in /tmp/my_server (you need write access rights to that directory).

    -
    -

    Note

    -

    The service is fully defined by the content of its location directory (“/tmp/my_server/”). -If you use an existing location, the service object will resume -from the index found there. Also, to “clone” a service, just copy that -directory somewhere else. The copy will be a fully working duplicate of the -original service.

    -
    -
    -
    -

    Model training

    -

    We can start indexing right away:

    -
    >>> service.index(corpus)
-AttributeError: must initialize model for /tmp/my_server/b before indexing documents
-
    -
    -

    Oops, we can not. The service indexes documents in a semantic representation, which -is different to the plain text we give it. We must teach the service how to convert -between plain text and semantics first:

    -
    >>> service.train(corpus, method='lsi')
-
    -
    -

    That was easy. The method=’lsi’ parameter meant that we trained a model for -Latent Semantic Indexing -and default dimensionality (400) over a tf-idf -representation of our little corpus, all automatically. More on that later.

    -

    Note that for the semantic model to make sense, it should be trained -on a corpus that is:

    -
      -
    • Reasonably similar to the documents you want to index later. Training on a corpus -of recipes in French when all indexed documents will be about programming in English -will not help.
      • -
    • Reasonably large (at least thousands of documents), so that the statistical analysis has -a chance to kick in. Don’t use my example corpus here of 9 documents in production O_o
      • -
    -
    -
    -

    Indexing documents

    -
    >>> service.index(corpus) # index the same documents that we trained on...
-
    -
    -

    Indexing can happen over any documents, but I’m too lazy to create another example corpus, so we index the same 9 docs used for training.

    -

    Delete documents with:

    -
    >>> service.delete(['doc_5', 'doc_8']) # supply a list of document ids to be removed from the index
-
    -
    -

    When you pass documents that have the same id as some already indexed document, -the indexed document is overwritten by the new input (=only the latest counts; -document ids are always unique per service):

    -
    >>> service.index(corpus[:3]) # overall index size unchanged (just 3 docs overwritten)
-
    -
    -

    The index/delete/overwrite calls can be arbitrarily interspersed with queries. -You don’t have to index all documents first to start querying, indexing can be incremental.

    -
    -
    -

    Querying

    -

    There are two types of queries:

    -
      -

    1. by id:

      -
      >>> print service.find_similar('doc_0')
-[('doc_0', 1.0, None), ('doc_2', 0.30426699, None), ('doc_1', 0.25648531, None), ('doc_3', 0.25480536, None)]
-
      -
      -
      >>> print service.find_similar('doc_5') # we deleted doc_5 and doc_8, remember?
-ValueError: document 'doc_5' not in index
-
      -
      -

      In the resulting 3-tuples, doc_n is the document id we supplied during indexing, -0.30426699 is the similarity of doc_n to the query, but what’s up with that None, you ask? -Well, you can associate each document with a “payload”, during indexing. -This payload object (anything pickle-able) is later returned during querying. -If you don’t specify doc[‘payload’] during indexing, queries simply return None in the result tuple, as in our example here.

      -
      2. -

    2. or by document (using document[‘tokens’]; id is ignored in this case):

      -
      >>> doc = {'tokens': utils.simple_preprocess('Graph and minors and humans and trees.')}
->>> print service.find_similar(doc, min_score=0.4, max_results=50)
-[('doc_7', 0.93350589, None), ('doc_3', 0.42718196, None)]
-
      -
      -
      3. -
    -
    -
    -

    Remote access

    -

    So far, we did everything in our Python shell, locally. I very much like Pyro, -a pure Python package for Remote Procedure Calls (RPC), so I’ll illustrate remote -service access via Pyro. Pyro takes care of all the socket listening/request routing/data marshalling/thread -spawning, so it saves us a lot of trouble.

    -

    To create a similarity server, we just create a simserver.SessionServer object and register it -with a Pyro daemon for remote access. There is a small example script -included with simserver, run it with:

    -
    $ python -m simserver.run_simserver /tmp/testserver
    -
    -

    You can just ctrl+c to terminate the server, but leave it running for now.

    -

    Now open your Python shell again, in another terminal window or possibly on another machine, and:

    -
    >>> import Pyro4
->>> service = Pyro4.Proxy(Pyro4.locateNS().lookup('gensim.testserver'))
-
    -
    -

    Now service is only a proxy object: every call is physically executed wherever -you ran the run_server.py script, which can be a totally different computer -(within a network broadcast domain), but you don’t even know:

    -
    >>> print service.status()
->>> service.train(corpus)
->>> service.index(other_corpus)
->>> service.find_similar(query)
->>> ...
-
    -
    -

    It is worth mentioning that Irmen, the author of Pyro, also released -Pyrolite recently. That is a package -which allows you to create Pyro proxies also from Java and .NET, in addition to Python. -That way you can call remote methods from there too—the client doesn’t have to be in Python.

    -
    -
    -

    Concurrency

    -

    Ok, now it’s getting interesting. Since we can access the service remotely, what -happens if multiple clients create proxies to it at the same time? What if they -want to modify the server index at the same time?

    -

    Answer: the SessionServer object is thread-safe, so that when each client spawns a request -thread via Pyro, they don’t step on each other’s toes.

    -

    This means that:

    -
      -
    1. There can be multiple simultaneous service.find_similar queries (or, in -general, multiple simultaneus calls that are “read-only”).
      2. -
    2. When two clients issue modification calls (index/train/delete/drop_index/...) -at the same time, an internal lock serializes them – the later call has to wait.
      3. -
    3. While one client is modifying the index, all other clients’ queries still see -the original index. Only once the modifications are committed do they become -“visible”.
      4. -
    -
    -
    -

    What do you mean, visible?

    -

    The service uses transactions internally. This means that each modification is -done over a clone of the service. If the modification session fails for whatever -reason (exception in code; power failure that turns off the server; client unhappy -with how the session went), it can be rolled back. It also means other clients can -continue querying the original index during index updates.

    -

    The mechanism is hidden from users by default through auto-committing (it was already happening -in the examples above too), but auto-committing can be turned off explicitly:

    -
    >>> service.set_autosession(False)
->>> service.train(corpus)
-RuntimeError: must open a session before modifying SessionServer
->>> service.open_session()
->>> service.train(corpus)
->>> service.index(corpus)
->>> service.delete(doc_ids)
->>> ...
-
    -
    -

    None of these changes are visible to other clients, yet. Also, other clients’ -calls to index/train/etc will block until this session is committed/rolled back—there -cannot be two open sessions at the same time.

    -

    To end a session:

    -
    >>> service.rollback() # discard all changes since open_session()
-
    -
    -

    or:

    -
    >>> service.commit() # make changes public; now other clients can see changes/acquire the modification lock
-
    -
    -
    -
    -

    Other stuff

    -

    TODO Custom document parsing (in lieu of utils.simple_preprocess). Different models (not just lsi). Optimizing the index with service.optimize(). -TODO add some hard numbers; example tutorial for some bigger collection, e.g. for arxiv.org or wikipedia.

    -
    -
    - - -
    -
    -
    -
    -
    - - -
    -

    Navigation

    - -
    - - -
    - © Copyright 2009-2012, Radim Řehůřek <radimrehurek(at)seznam.cz>. - Last updated on Sep 16, 2012. -
    - - \ No newline at end of file diff --git a/docs/src/Makefile b/docs/src/Makefile index 713524614c..29bbc3559b 100644 --- a/docs/src/Makefile +++ b/docs/src/Makefile @@ -32,7 +32,8 @@ clean: html: $(SPHINXBUILD) -b html $(ALLSPHINXOPTS) $(BUILDDIR)/html - cp -r $(BUILDDIR)/html/* ../ + rm -r $(BUILDDIR)/html/_sources +# cp -r $(BUILDDIR)/html/* ../ @echo @echo "Build finished. The HTML pages are in ../" diff --git a/docs/src/_static/contents.png b/docs/src/_static/contents.png deleted file mode 100644 index 7fb82154a1..0000000000 Binary files a/docs/src/_static/contents.png and /dev/null differ diff --git a/docs/src/_static/css/anythingslider.css b/docs/src/_static/css/anythingslider.css new file mode 100644 index 0000000000..25ad8c53e6 --- /dev/null +++ b/docs/src/_static/css/anythingslider.css @@ -0,0 +1,291 @@ +/* + AnythingSlider v1.8+ Default theme + By Chris Coyier: http://css-tricks.com + with major improvements by Doug Neiner: http://pixelgraphics.us/ + based on work by Remy Sharp: http://jqueryfordesigners.com/ +*/ + +/***************************** + SET DEFAULT DIMENSIONS HERE + *****************************/ +/* change the ID & dimensions to match your slider */ +#slider { + width: 700px; + height: 390px; + list-style: none; + /* Prevent FOUC (see FAQ page) and keep things readable if javascript is disabled */ + overflow-y: auto; + overflow-x: hidden; +} + +/****************** + SET STYLING HERE + ****************** + ================================= + Default state (no keyboard focus) + ==================================*/ +/* Overall Wrapper */ +.anythingSlider-default { + margin: 0 auto; + /* 45px right & left padding for the arrows, 28px @ bottom for navigation */ + padding: 0 45px 28px 45px; +} +/* slider window - top & bottom borders, default state */ +.anythingSlider-default .anythingWindow { + border-top: 3px solid #777; + border-bottom: 3px solid #777; +} +/* Navigation buttons + start/stop button, default state */ +.anythingSlider-default .anythingControls a { + /* top shadow */ + background: #777 url(../images/default.png) center -288px repeat-x; + color: #000; + border-radius: 0 0 5px 5px; + -moz-border-radius: 0 0 5px 5px; + -webkit-border-radius: 0 0 5px 5px; +} +/* Make sure navigation text is visible */ +.anythingSlider-default .anythingControls a span { + visibility: visible; +} +/* Navigation current button, default state */ +.anythingSlider-default .anythingControls a.cur { + background: #888; + color: #000; +} + +/* start-stop button, stopped, default state */ +.anythingSlider-default .anythingControls a.start-stop { + background-color: #040; + color: #ddd; +} +/* start-stop button, playing, default state */ +.anythingSlider-default .anythingControls a.start-stop.playing { + background-color: #800; +} + +/* start-stop button, default hovered text color (when visible) */ +/* hide nav/start-stop background image shadow on hover - makes the button appear to come forward */ +.anythingSlider-default .anythingControls a.start-stop:hover, +.anythingSlider-default .anythingControls a.start-stop.hover, +.anythingSlider-default .anythingControls a.start-stop .anythingControls ul a:hover { + background-image: none; + color: #ddd; +} + +/* + ================================= + Active State (has keyboard focus) + ================================= +*/ +/* slider window - top & bottom borders, active state */ +.anythingSlider-default.activeSlider .anythingWindow { + border-color: #7C9127; +} +/* Navigation buttons, active state */ +.anythingSlider-default.activeSlider .anythingControls a { + /* background image = top shadow */ + background-color: #7C9127; +} +/* Navigation current & hovered button, active state */ +.anythingSlider-default.activeSlider .anythingControls a.cur, +.anythingSlider-default.activeSlider .anythingControls a:hover { + /* background image removed */ + background: #7C9127; +} + +/* start-stop button, stopped, active state */ +.anythingSlider-default.activeSlider .anythingControls a.start-stop { + background-color: #080; + color: #fff; +} +/* start-stop button, playing, active state */ +.anythingSlider-default.activeSlider .anythingControls a.start-stop.playing { + background-color: #d00; + color: #fff; +} +/* start-stop button, active slider hovered text color (when visible) */ +.anythingSlider-default.activeSlider .start-stop:hover, +.anythingSlider-default.activeSlider .start-stop.hover { + color: #fff; +} + +/************************ + NAVIGATION POSITIONING + ************************/ +/* Navigation Arrows */ +.anythingSlider-default .arrow { + top: 50%; + position: absolute; + display: block; +} + +.anythingSlider-default .arrow a { + display: block; + width: 45px; + height: 140px; + margin: -70px 0 0 0; /* half height of image */ + text-align: center; + outline: 0; + background: url(../images/default.png) no-repeat; +} + +/* back arrow */ +.anythingSlider-default .back { left: 0; } +.anythingSlider-default .back a { background-position: left top; } +.anythingSlider-default .back a:hover, +.anythingSlider-default .back a.hover { background-position: left -140px; } +/* forward arrow */ +.anythingSlider-default .forward { right: 0; } +.anythingSlider-default .forward a { background-position: right top; } +.anythingSlider-default .forward a:hover, +.anythingSlider-default .forward a.hover { background-position: right -140px; } + +/* Navigation Links */ +.anythingSlider-default .anythingControls { outline: 0; display: none; } +.anythingSlider-default .anythingControls ul { margin: 0; padding: 0; float: left; } +.anythingSlider-default .anythingControls ul li { display: inline; } +.anythingSlider-default .anythingControls ul a { + font: 11px/18px Georgia, Serif; + display: inline-block; + text-decoration: none; + padding: 2px 8px; + height: 18px; + margin: 0 5px 0 0; + text-align: center; + outline: 0; +} + +/* navigationSize window */ +.anythingSlider-default .anythingControls .anythingNavWindow { + overflow: hidden; + float: left; +} + +/* Autoplay Start/Stop button */ +.anythingSlider-default .anythingControls .start-stop { + padding: 2px 5px; + width: 40px; + text-align: center; + text-decoration: none; + float: right; + z-index: 100; + outline: 0; +} + +/*********************** + IE8 AND OLDER STYLING + ***********************/ + +/* Navigation Arrows */ +.as-oldie .anythingSlider-default .arrow { + top: 30%; +} +.as-oldie .anythingSlider-default .arrow a { + margin: 0; +} + +/* margin between nav buttons just looks better */ +.as-oldie .anythingSlider-default .anythingControls li { + margin-left: 3px; +} + +/* When using the navigationSize option, the side margins need to be zero + None of the navigation panels look good in IE7 now =( */ +.as-oldie .anythingSlider-default .anythingControls a { + margin: 0; +} +.as-oldie .anythingSlider-default .anythingNavWindow { + margin: 0 2px; +} +.as-oldie .anythingSlider-default .anythingNavWindow li { + padding: 3px 0 0 0; +} + +/*********************** + COMMON SLIDER STYLING + ***********************/ +/* Overall Wrapper */ +.anythingSlider { + display: block; + overflow: visible !important; + position: relative; +} +/* anythingSlider viewport window */ +.anythingSlider .anythingWindow { + overflow: hidden; + position: relative; + width: 100%; + height: 100%; +} +/* anythingSlider base (original element) */ +.anythingSlider .anythingBase { + background: transparent; + list-style: none; + position: absolute; + overflow: visible !important; + top: 0; + left: 0; + margin: 0; + padding: 0; +} + +/* Navigation arrow text; indent moved to span inside "a", for IE7; + apparently, a negative text-indent on an "a" link moves the link as well as the text */ +.anythingSlider .arrow span { + display: block; + visibility: hidden; +} +/* disabled arrows, hide or reduce opacity: opacity: .5; filter: alpha(opacity=50); */ +.anythingSlider .arrow.disabled { + display: none; +} +/* all panels inside the slider; horizontal mode */ +.anythingSlider .panel { + background: transparent; + display: block; + overflow: hidden; + float: left; + padding: 0; + margin: 0; +} +/* vertical mode */ +.anythingSlider .vertical .panel { + float: none; +} +/* fade mode */ +.anythingSlider .fade .panel { + float: none; + position: absolute; + top: 0; + left: 0; + z-index: 0; +} +/* fade mode active page - visible & on top */ +.anythingSlider .fade .activePage { + z-index: 1; +} + +/*********************** + RTL STYLING + ***********************/ +/* slider autoplay right-to-left, reverse order of nav links to look better */ +.anythingSlider.rtl .anythingWindow { + direction: ltr; + unicode-bidi: bidi-override; +} +.anythingSlider.rtl .anythingControls ul { float: left; } /* move nav link group to left */ +.anythingSlider.rtl .anythingControls ul a { float: right; } /* reverse order of nav links */ +.anythingSlider.rtl .start-stop { /* float: right; */ } /* move start/stop button - in case you want to switch sides */ + +/* probably not necessary, but added just in case */ +.anythingSlider, +.anythingSlider .anythingWindow, +.anythingSlider .anythingControls ul a, +.anythingSlider .arrow a, +.anythingSlider .start-stop { + transition-duration: 0; + -o-transition-duration: 0; + -moz-transition-duration: 0; + -webkit-transition-duration: 0; +} diff --git a/docs/src/_static/css/jquery.qtip.min.css b/docs/src/_static/css/jquery.qtip.min.css new file mode 100644 index 0000000000..9c454f3050 --- /dev/null +++ b/docs/src/_static/css/jquery.qtip.min.css @@ -0,0 +1 @@ +/*! qtip2 v2.0.0 | http://craigsworks.com/projects/qtip2/ | Licensed MIT, GPL */.qtip,.qtip{position:absolute;left:-28000px;top:-28000px;display:none;max-width:280px;min-width:50px;font-size:10.5px;line-height:12px;direction:ltr}.qtip-content{position:relative;padding:5px 9px;overflow:hidden;text-align:left;word-wrap:break-word}.qtip-titlebar{position:relative;padding:5px 35px 5px 10px;overflow:hidden;border-width:0 0 1px;font-weight:700}.qtip-titlebar+.qtip-content{border-top-width:0!important}.qtip-close{position:absolute;right:-9px;top:-9px;cursor:pointer;outline:medium none;border-width:1px;border-style:solid;border-color:transparent}.qtip-titlebar .qtip-close{right:4px;top:50%;margin-top:-9px}* html .qtip-titlebar .qtip-close{top:16px}.qtip-titlebar .ui-icon,.qtip-icon .ui-icon{display:block;text-indent:-1000em;direction:ltr;vertical-align:middle}.qtip-icon,.qtip-icon .ui-icon{-moz-border-radius:3px;-webkit-border-radius:3px;border-radius:3px;text-decoration:none}.qtip-icon .ui-icon{width:18px;height:14px;text-align:center;text-indent:0;font:normal bold 10px/13px Tahoma,sans-serif;color:inherit;background:transparent none no-repeat -100em -100em}.qtip-focus{}.qtip-hover{}.qtip-default{border-width:1px;border-style:solid;border-color:#F1D031;background-color:#FFFFA3;color:#555}.qtip-default .qtip-titlebar{background-color:#FFEF93}.qtip-default .qtip-icon{border-color:#CCC;background:#F1F1F1;color:#777}.qtip-default .qtip-titlebar .qtip-close{border-color:#AAA;color:#111}/*! Light tooltip style */.qtip-light{background-color:#fff;border-color:#E2E2E2;color:#454545}.qtip-light .qtip-titlebar{background-color:#f1f1f1}/*! Dark tooltip style */.qtip-dark{background-color:#505050;border-color:#303030;color:#f3f3f3}.qtip-dark .qtip-titlebar{background-color:#404040}.qtip-dark .qtip-icon{border-color:#444}.qtip-dark .qtip-titlebar .ui-state-hover{border-color:#303030}/*! Cream tooltip style */.qtip-cream{background-color:#FBF7AA;border-color:#F9E98E;color:#A27D35}.qtip-cream .qtip-titlebar{background-color:#F0DE7D}.qtip-cream .qtip-close .qtip-icon{background-position:-82px 0}/*! Red tooltip style */.qtip-red{background-color:#F78B83;border-color:#D95252;color:#912323}.qtip-red .qtip-titlebar{background-color:#F06D65}.qtip-red .qtip-close .qtip-icon{background-position:-102px 0}.qtip-red .qtip-icon{border-color:#D95252}.qtip-red .qtip-titlebar .ui-state-hover{border-color:#D95252}/*! Green tooltip style */.qtip-green{background-color:#CAED9E;border-color:#90D93F;color:#3F6219}.qtip-green .qtip-titlebar{background-color:#B0DE78}.qtip-green .qtip-close .qtip-icon{background-position:-42px 0}/*! Blue tooltip style */.qtip-blue{background-color:#E5F6FE;border-color:#ADD9ED;color:#5E99BD}.qtip-blue .qtip-titlebar{background-color:#D0E9F5}.qtip-blue .qtip-close .qtip-icon{background-position:-2px 0}.qtip-shadow{-webkit-box-shadow:1px 1px 3px 1px rgba(0,0,0,.15);-moz-box-shadow:1px 1px 3px 1px rgba(0,0,0,.15);box-shadow:1px 1px 3px 1px rgba(0,0,0,.15)}.qtip-rounded,.qtip-tipsy,.qtip-bootstrap{-moz-border-radius:5px;-webkit-border-radius:5px;border-radius:5px}.qtip-youtube{-moz-border-radius:2px;-webkit-border-radius:2px;border-radius:2px;-webkit-box-shadow:0 0 3px #333;-moz-box-shadow:0 0 3px #333;box-shadow:0 0 3px #333;color:#fff;border-width:0;background:#4A4A4A;background-image:-webkit-gradient(linear,left top,left bottom,color-stop(0,#4A4A4A),color-stop(100%,black));background-image:-webkit-linear-gradient(top,#4A4A4A 0,black 100%);background-image:-moz-linear-gradient(top,#4A4A4A 0,black 100%);background-image:-ms-linear-gradient(top,#4A4A4A 0,black 100%);background-image:-o-linear-gradient(top,#4A4A4A 0,black 100%)}.qtip-youtube .qtip-titlebar{background-color:#4A4A4A;background-color:rgba(0,0,0,0)}.qtip-youtube .qtip-content{padding:.75em;font:12px arial,sans-serif;filter:progid:DXImageTransform.Microsoft.Gradient(GradientType=0, StartColorStr=#4a4a4a, EndColorStr=#000000);-ms-filter:"progid:DXImageTransform.Microsoft.Gradient(GradientType=0, StartColorStr=#4a4a4a, EndColorStr=#000000);"}.qtip-youtube .qtip-icon{border-color:#222}.qtip-youtube .qtip-titlebar .ui-state-hover{border-color:#303030}.qtip-jtools{background:#232323;background:rgba(0,0,0,.7);background-image:-webkit-gradient(linear,left top,left bottom,from(#717171),to(#232323));background-image:-moz-linear-gradient(top,#717171,#232323);background-image:-webkit-linear-gradient(top,#717171,#232323);background-image:-ms-linear-gradient(top,#717171,#232323);background-image:-o-linear-gradient(top,#717171,#232323);border:2px solid #ddd;border:2px solid rgba(241,241,241,1);-moz-border-radius:2px;-webkit-border-radius:2px;border-radius:2px;-webkit-box-shadow:0 0 12px #333;-moz-box-shadow:0 0 12px #333;box-shadow:0 0 12px #333}.qtip-jtools .qtip-titlebar{background-color:transparent;filter:progid:DXImageTransform.Microsoft.gradient(startColorstr=#717171, endColorstr=#4A4A4A);-ms-filter:"progid:DXImageTransform.Microsoft.gradient(startColorstr=#717171, endColorstr=#4A4A4A)"}.qtip-jtools .qtip-content{filter:progid:DXImageTransform.Microsoft.gradient(startColorstr=#4A4A4A, endColorstr=#232323);-ms-filter:"progid:DXImageTransform.Microsoft.gradient(startColorstr=#4A4A4A, endColorstr=#232323)"}.qtip-jtools .qtip-titlebar,.qtip-jtools .qtip-content{background:transparent;color:#fff;border:0 dashed transparent}.qtip-jtools .qtip-icon{border-color:#555}.qtip-jtools .qtip-titlebar .ui-state-hover{border-color:#333}.qtip-cluetip{-webkit-box-shadow:4px 4px 5px rgba(0,0,0,.4);-moz-box-shadow:4px 4px 5px rgba(0,0,0,.4);box-shadow:4px 4px 5px rgba(0,0,0,.4);background-color:#D9D9C2;color:#111;border:0 dashed transparent}.qtip-cluetip .qtip-titlebar{background-color:#87876A;color:#fff;border:0 dashed transparent}.qtip-cluetip .qtip-icon{border-color:#808064}.qtip-cluetip .qtip-titlebar .ui-state-hover{border-color:#696952;color:#696952}.qtip-tipsy{background:#000;background:rgba(0,0,0,.87);color:#fff;border:0 solid transparent;font-size:11px;font-family:'Lucida Grande',sans-serif;font-weight:700;line-height:16px;text-shadow:0 1px black}.qtip-tipsy .qtip-titlebar{padding:6px 35px 0 10;background-color:transparent}.qtip-tipsy .qtip-content{padding:6px 10}.qtip-tipsy .qtip-icon{border-color:#222;text-shadow:none}.qtip-tipsy .qtip-titlebar .ui-state-hover{border-color:#303030}.qtip-tipped{border:3px solid #959FA9;-moz-border-radius:3px;-webkit-border-radius:3px;border-radius:3px;background-color:#F9F9F9;color:#454545;font-weight:400;font-family:serif}.qtip-tipped .qtip-titlebar{border-bottom-width:0;color:#fff;background:#3A79B8;background-image:-webkit-gradient(linear,left top,left bottom,from(#3A79B8),to(#2E629D));background-image:-webkit-linear-gradient(top,#3A79B8,#2E629D);background-image:-moz-linear-gradient(top,#3A79B8,#2E629D);background-image:-ms-linear-gradient(top,#3A79B8,#2E629D);background-image:-o-linear-gradient(top,#3A79B8,#2E629D);filter:progid:DXImageTransform.Microsoft.gradient(startColorstr=#3A79B8, endColorstr=#2E629D);-ms-filter:"progid:DXImageTransform.Microsoft.gradient(startColorstr=#3A79B8, endColorstr=#2E629D)"}.qtip-tipped .qtip-icon{border:2px solid #285589;background:#285589}.qtip-tipped .qtip-icon .ui-icon{background-color:#FBFBFB;color:#555}.qtip-bootstrap{font-size:14px;line-height:20px;color:#333;padding:1px;background-color:#fff;border:1px solid #ccc;border:1px solid rgba(0,0,0,.2);-webkit-border-radius:6px;-moz-border-radius:6px;border-radius:6px;-webkit-box-shadow:0 5px 10px rgba(0,0,0,.2);-moz-box-shadow:0 5px 10px rgba(0,0,0,.2);box-shadow:0 5px 10px rgba(0,0,0,.2);-webkit-background-clip:padding-box;-moz-background-clip:padding;background-clip:padding-box}.qtip-bootstrap .qtip-titlebar{padding:8px 14px;margin:0;font-size:14px;font-weight:400;line-height:18px;background-color:#f7f7f7;border-bottom:1px solid #ebebeb;-webkit-border-radius:5px 5px 0 0;-moz-border-radius:5px 5px 0 0;border-radius:5px 5px 0 0}.qtip-bootstrap .qtip-titlebar .qtip-close{right:11px;top:45%;border-style:none}.qtip-bootstrap .qtip-content{padding:9px 14px}.qtip-bootstrap .qtip-icon{background:transparent}.qtip-bootstrap .qtip-icon .ui-icon{width:auto;height:auto;float:right;font-size:20px;font-weight:700;line-height:18px;color:#000;text-shadow:0 1px 0 #fff;opacity:.2;filter:alpha(opacity=20)}.qtip-bootstrap .qtip-icon .ui-icon:hover{color:#000;text-decoration:none;cursor:pointer;opacity:.4;filter:alpha(opacity=40)}.qtip:not(.ie9haxors) div.qtip-content,.qtip:not(.ie9haxors) div.qtip-titlebar{filter:none;-ms-filter:none}.qtip .qtip-tip{margin:0 auto;overflow:hidden;z-index:10}.qtip .qtip-tip,.qtip .qtip-tip .qtip-vml{position:absolute;color:#123456;background:transparent;border:0 dashed transparent}.qtip .qtip-tip canvas{top:0;left:0}.qtip .qtip-tip .qtip-vml{behavior:url(#default#VML);display:inline-block;visibility:visible}#qtip-overlay{position:fixed;left:-10000em;top:-10000em}#qtip-overlay.blurs{cursor:pointer}#qtip-overlay div{position:absolute;left:0;top:0;width:100%;height:100%;background-color:#000;opacity:.7;filter:alpha(opacity=70);-ms-filter:"alpha(Opacity=70)"}.qtipmodal-ie6fix{position:absolute!important} \ No newline at end of file diff --git a/docs/src/_static/css/style.css b/docs/src/_static/css/style.css new file mode 100644 index 0000000000..7353756a00 --- /dev/null +++ b/docs/src/_static/css/style.css @@ -0,0 +1,759 @@ +a,a img { + text-decoration:none; + border:0; +} + +html, +body { + margin:0; + padding:0; + height:100%; + background: #fafafa url('../images/bg.png') top center repeat; + color:#333333; + font-family:arial; + font-size: 14px; +} + +p { + color:#333333; + font-family:arial; + font-size: 14px; +} + +ul li { + color:#333333; + font-family:arial; + font-size: 14px; +} + +.obsah ul li a,.obsah a{ + color:#006699; + font-weight:bold; + text-decoration:underline; +} + +.obsah ul li a:hover,.obsah a:hover{ + color:#000; + font-weight:bold; +} + +.h1gensim { + margin: 0; + padding: 0; + display: block; +} + +.h2gensim { + margin:0; + padding:0; + font-size:36px; + color:#fff; + text-shadow: 3px 3px 3px #404040; +} + +h1 { /* headings are already in a special banner... ignore the h1 generated by sphinx */ + display: none; +} + +.h3gensim { + margin:0; + padding:0 0 40px 0; + font-size:32px; + color:#333333; +} + +h4 { + margin:0; + padding:0 0 40px 0; + font-size:22px; + color:#333333; +} + +h5{ + margin:0; + padding:0 0 40px 0; + font-size:18px; + color:#333333; +} + +.qtip-content { + font-size: 12px; + line-height: 14px; +} + +/* LOGO, NADPIS A DOWNLOAD */ + +#topwrap{ + width:980px; + margin: 0 auto; +} + +#top{ + height:225px; +} +#left{ + float:left; + padding-top: 83px; + width: 120px; +} +#middleright{ + float:right; +} +#middle{ + float:left; + width:490px; + padding-top: 83px; +} +#right{ + float:right; + width:351px; + padding-top: 83px; +} +#design_tagline{ + padding-left:85px; +} +#design_install{ + float:right; + padding-right: 12px; + background: url('../images/direct-install.png') top center no-repeat; + width: 244px; + height: 52px; +} +#command{ + color:#485420; + font-size:14px; + font-family:arial; + font-weight:bold; + padding: 27px 0 0 72px; +} +.space10{ + height:10px; +} + +/* MENU, MODRY PRUH */ + +#menu{ + width: 974px; +} +#indentation{ + padding-top:90px; + float:right; + position:relative; + text-align:left; +} + +.menubuttons{ + list-style: none; + position:relative; +} + +.menubuttons li { + float:left; + position:relative; +} + +li.menubuttonblue a { + float:left; + display: inline-block; + width: 124px; + height: 39px; + background: url('../images/menubutton.png') top; + font-size:16px; + font-weight:bold; + color:#ffffff; + font-family:arial; + text-align:center; + line-height: 39px; +} + +li.menubutton a { + float:left; + display: inline-block; + width: 124px; + height: 39px; + background: url('../images/menubutton.png') bottom; + font-size:16px; + font-weight:bold; + color:#222222; + font-family:arial; + text-align:center; + line-height: 39px; +} + +.menubutton a:hover{ + background-position: 0 0; +} + +#design_banner{ + width: 100%; + height: 274px; + clear:both; + background: #006699; +} + +#thinbanner { + width: 100%; + height: 99px; + clear:both; + background: #006699; +} + +#bodythinbanner{ + width: 980px; + height: 99px; + padding-top:25px; + margin: 0 auto; +} + +#bodybanner{ + width: 980px; + height: 274px; + padding-top: 30px; + margin: 0 auto; +} + +.leftbanner{ + float:left; + width: 418px; + border-radius: 10px; + overflow: hidden; +} + +.leftbanner img { + border-radius: 10px; + overflow: hidden; + width: 418px; + margin: 10px 0 0 0; +} + +.rightbanner{ + float:right; +} + +ul.description li{ + list-style: none; + background: url('../images/checker.png') no-repeat left top; + font-size: 14px; + padding: 6px 0 0 45px; + height: 44px; + font-weight: bold; + color: #ffffff; +} + +ul.description{ + padding-left: 6px; + padding-top: 9px; +} + + +/* HLAVNI OBSAH, FEATURES, REFERENCE, GET STARTED */ + + +.podklad { + margin-top:80px; + margin-left:auto; + margin-right:auto; + width: 974px; + background: #ffffff; + -ms-filter: "progid:DXImageTransform.Microsoft.Shadow(Strength=4, Direction=135, Color='#cccccc')"; + filter: progid:DXImageTransform.Microsoft.Shadow(Strength=9, Direction=135, Color='#cccccc'); + -moz-border-radius: 5px; + -webkit-border-radius: 5px; + -khtml-border-radius: 5px; + border-radius: 5px; + -moz-box-shadow: 3px 2px 3px 4px #cccccc; + -webkit-box-shadow: 3px 2px 3px 4px #cccccc; + box-shadow: 3px 2px 3px 4px #cccccc; +} + +.podkladwrapper { + padding: 30px 0 20px 27px; +} + +.more-info { + font-size: 12px; + font-weight: bold; + color: #777777; + margin: 6px 2px; +} + +.obsah{ + margin-top:0; + margin-left:auto; + margin-right:auto; + width: 974px; + background: #ffffff; +} + +.obsahwrapper{ + padding:30px 27px 20px 27px; +} + +.feature{ + padding-top: 5px; + vertical-align: 100%; + font-size: 15px; + padding-left: 15px; + font-weight: bold; + color: #333333; +} + +.half1{ + width:520px; + float:left; +} + +.half2{ + width:400px; + float:right; +} +.lefthalf1{ + float:left; + display:block; +} +.lefthalf2{ + float:right; + display:block; +} +.quarter1{ + width:260px; + height: 90px; +} +.quarter2{ + width:250px; + height: 90px; +} +.quarter3{ + width:195px; + height: 90px; +} +.quarter4{ + width:160px; + height: 90px; +} +.clearit{ + clear:both; +} +.center{ + margin:0 auto; + text-align:center; +} + +ul.reference{ + list-style-type: none; + padding:0; + margin:0; +} + +.testimonial-author { + font-style: normal; + font-weight: normal; + display: block; +} + +ul.reference li { + display: inline-block; + padding-right: 20px; + font-size: 13px; + font-style: italic; + width: 281px; + height: 196px; + padding-top: 10px; + padding-bottom: 10px; + padding-left: 15px; + color: #333333; + font-weight: bold; + font-style: italic; +} + +.reference1 { + display: inline-block; + width: 247px; +/* background: url('../images/references/logo_ghent.png') top center no-repeat; +*/ background: url('../images/references/logo_ibcn.png') top center no-repeat; + background-size: 60%; + padding-top: 100px; + padding-bottom: 24px; +} + +.reference2 { + width: 247px; + background: url('../images/references/logo_roistr.png') top center no-repeat; + padding-top: 100px; + padding-bottom: 24px; +} + +.reference3 { + display: inline-block; + width: 247px; + background: url('../images/references/logo_sportsauthority.png') top center no-repeat; + padding-top: 100px; + padding-bottom: 24px; +} + +.reference4 { + display: inline-block; + width: 247px; + background: url('../images/references/logo_dynadmic.png') top center no-repeat; + background-size: 45%; + padding-top: 100px; + padding-bottom: 24px; +} + +.reference4 { + display: inline-block; + width: 247px; + background: url('../images/references/logo_dynadmic.png') top center no-repeat; + background-size: 45%; + padding-top: 100px; + padding-bottom: 24px; +} + +.reference5 { + display: inline-block; + width: 247px; + padding-top: 100px; + padding-bottom: 24px; +} + +.reference6 { + display: inline-block; + width: 247px; + background: url('../images/references/logo_eudml.png') top center no-repeat; + padding-top: 100px; + padding-bottom: 24px; +} + +.reference7 { + display: inline-block; + width: 247px; + background: url('../images/references/logo_dtu.gif') top center no-repeat; + background-size: 25%; + padding-top: 100px; + padding-bottom: 24px; +} + +.getstarted{ + text-align: center; + padding: 30px 0 20px 0; +} + + +/* FOOTER */ + + +#footer{ + margin: 0; + width: 100%; + height: 270px; + clear:both; + background: #072a47; + padding: 0; +} +#footerwrapper{ + width: 980px; + height: 270px; + padding-top:16px; + margin: 0 auto; +} + +#footerleft { +/* width:329px; +*/ float:left; +} + +#footermiddleright { +/* width:640px; +*/ float:right; +} + +#footermiddle { + width:355px; + float:left; + padding: 15px 10px; +} + +#footerright{ + width:285px; + float:right; +} +.smallerlogo{ + width: 44px; + height: 44px; + padding-right: 12px; +} +.copyright{ + font-size:14px; + color:#ffffff; + padding-top: 158px; +} +.navigation{ + list-style-type:none; + padding:0; + margin:0; +} +.navigation li{ + display:inline-block; + padding-right:5px; + font-size:14px; + color:#ffffff; +} +.navigation li a{ + display:inline-block; + font-size:14px; + color:#ffffff; +} + +.tweetodsazeni { + padding-top: 174px; +} + +.tweet { + width: 179px; + height: 39px; + padding-left: 50px; + padding-top: 0px; + font-size: 14px; + color: #ffffff; + background: url('../images/twitterbird.png') top left no-repeat; +} +a.tweet{ + font-size:14px; + color:#ffffff; +} +.footernadpis{ + font-size:34px; + color:#ffffff; +} +.googlegroupsodsazeni{ + padding-top:18px; +} + +.googlegroups{ + width: 300px; + height: 39px; + padding-left: 45px; + padding-top: 10px; + font-size: 14px; + color: #ffffff; + background: url('../images/googlegroups.png') top left no-repeat; + display: block; +} + +a.googlegroups{ + font-size:14px; + color:#ffffff; +} + +#design_download { + text-align:right; +} + +#design_install { + float:right; + padding-right: 12px; + background: url('../images/direct-install.png') top center no-repeat; + width: 207px; + height: 44px; +} + +#command { + color:#485420; + font-size:12px; + font-family:arial; + font-weight:bold; + padding: 23px 0 0 64px; +} + +#top1{ + height:148px; +} + +#left1{ + float:left; + padding-top: 53px; + width: 76px; +} + +#middle1{ + float:left; + width:540px; + padding-top: 53px; +} + +#tagline1{ + padding-left:50px; +} + +#indentation1{ + padding-top:10px; + float:right; + position:relative; + text-align:left; +} + +#right1 { + float:right; + width: 351px; + padding-top: 53px; +} + + +/* copied from sphinx themes */ + +a.headerlink { + color: #006699; + font-size: 0.8em; + padding: 0 4px 0 4px; + text-decoration: none; + visibility: hidden; +} + +h1:hover > a.headerlink, +h2:hover > a.headerlink, +h3:hover > a.headerlink, +h4:hover > a.headerlink, +h5:hover > a.headerlink, +h6:hover > a.headerlink, +dt:hover > a.headerlink { + visibility: visible; +} + +a.headerlink:hover { + background-color: #006699; + color: white; + visibility: visible; +} + +div.body p.centered { + text-align: center; + margin-top: 25px; +} + +table.docutils { + border: 0; + border-collapse: collapse; +} + +div.versioninfo { + margin: 1em 0 0 0; + border: 1px solid #ccc; + background-color: #DDEAF0; + padding: 8px; + line-height: 1.3em; + font-size: 0.9em; +} + +.document { + line-height: 150%; +} + +/*table.docutils td, table.docutils th { + padding: 1px 8px 1px 5px; + border-top: 0; + border-left: 0; + border-right: 0; + border-bottom: 1px solid #aaa; +} +*/ + +table.field-list td, table.field-list th { + border: 0 !important; +} + +table.footnote td, table.footnote th { + border: 0 !important; +} + +th { + text-align: left; + padding-right: 5px; +} + +table.citation { + border-left: solid 1px gray; + margin-left: 1px; +} + +table.citation td { + border-bottom: none; +} + +div.admonition, div.warning { + font-size: 0.9em; + margin: 1em 0 0 0; + border: 1px solid #86989B; + background-color: #f7f7f7; +} + +div.admonition p, div.warning p { + margin: 0.5em 1em 0.5em 1em; + padding: 0; +} + +div.admonition pre, div.warning pre { + margin: 0.4em 1em 0.4em 1em; +} + +div.admonition p.admonition-title, +div.warning p.admonition-title { + margin: 0; + padding: 0.1em 0 0.1em 0.5em; + color: white; + border-bottom: 1px solid #86989B; + font-weight: bold; + background-color: #AFC1C4; +} + +div.warning { + border: 1px solid #940000; +} + +div.warning p.admonition-title { + background-color: #CF0000; + border-bottom-color: #940000; +} + +div.admonition ul, div.admonition ol, +div.warning ul, div.warning ol { + margin: 0.1em 0.5em 0.5em 3em; + padding: 0; +} + +.first { + margin-top: 0 !important; +} + +pre { + font-family: 'Consolas', 'Deja Vu Sans Mono', 'Bitstream Vera Sans Mono', monospace; + font-size: 0.95em; + letter-spacing: 0.015em; + padding: 0.5em; + border: 1px solid #ccc; + background-color: #f8f8f8; + line-height: 120%; +} + +tt { + background-color: #ecf0f3; + padding: 0 1px 0 1px; + font-size: 0.95em; +} + +dt:target, +.highlight { + background-color: #fbe54e; +} + +dl.class, dl.function { + border-top: 2px solid #888; +} + +dl.method, dl.attribute { + border-top: 1px solid #aaa; +} + +dl.glossary dt { + font-weight: bold; + font-size: 1.1em; +} diff --git a/docs/src/_static/default.css b/docs/src/_static/default.css deleted file mode 100644 index 611cb55720..0000000000 --- a/docs/src/_static/default.css +++ /dev/null @@ -1,574 +0,0 @@ -/** - * Alternate Sphinx design - * Originally created by Armin Ronacher for Werkzeug, adapted by Georg Brandl. - * Minor modifications by Radim Rehurek, who was too lazy to fork the theme. - */ - -table.links { - border-collapse: separate; - border-spacing: 15px; - margin: 10px auto 0px auto; -} - -table.links tr td { - -webkit-border-radius: 15px; - -moz-border-radius: 15px; - border-radius: 15px; - padding: 10px 30px; - color: white; -/* border: 1px solid #86989B;*/ - font-weight: bold; - background-color: #eee; - text-align: center; - background: -webkit-gradient(linear, left top, left bottom, from(#eee), to(#ddd)); - background: -moz-linear-gradient(top, #eee, #ddd); -} - -table.links a { text-decoration: none; } - -#tagcloud { - font-size: 50%; - width: 80%; - margin: 20px auto 0px auto; - padding: 15px; - line-height: 2.4em; - word-spacing: normal; - letter-spacing: normal; - text-transform: none; - text-align: justify; - text-indent: 0; - border: 1px dotted; -} - -.wrd { padding:0; position:relative } -.tagcloud0 { font-size:1.0em; color:#ACC1F3; z-index:10 } -.tagcloud1 { font-size:1.4em; color:#ACC1F3; z-index:9 } -.tagcloud2 { font-size:1.8em; color:#86A0DC; z-index:8} -.tagcloud3 { font-size:2.2em; color:#86A0DC; z-index:7} -.tagcloud4 { font-size:2.6em; color:#607EC5; z-index:6} -.tagcloud5 { font-size:3.0em; color:#607EC5; z-index:5} -.tagcloud6 { font-size:3.3em; color:#4C6DB9; z-index:4} -.tagcloud7 { font-size:3.6em; color:#395CAE; z-index:3} -.tagcloud8 { font-size:3.9em; color:#264CA2; z-index:2} -.tagcloud9 { font-size:4.2em; color:#133B97; z-index:1} -.tagcloud10 { font-size:4.5em; color:#002A8B; z-index:0} - - -body { - font-family: 'Lucida Grande', 'Lucida Sans Unicode', 'Geneva', 'Verdana', sans-serif; - font-size: 14px; - letter-spacing: -0.01em; - line-height: 150%; - text-align: center; - /*background-color: #AFC1C4; */ - background-color: #BFD1D4; - color: black; - padding: 0; - border: 1px solid #aaa; - - margin: 0px 80px 0px 80px; - min-width: 740px; -} - -a { - color: #CA7900; - text-decoration: none; -} - -a:hover { - color: #2491CF; -} - -pre { - font-family: 'Consolas', 'Deja Vu Sans Mono', 'Bitstream Vera Sans Mono', monospace; - font-size: 0.95em; - letter-spacing: 0.015em; - padding: 0.5em; - border: 1px solid #ccc; - background-color: #f8f8f8; -} - -td.linenos pre { - padding: 0.5em 0; - border: 0; - background-color: transparent; - color: #aaa; -} - -table.highlighttable { - margin-left: 0.5em; -} - -table.highlighttable td { - padding: 0 0.5em 0 0.5em; -} - -cite, code, tt { - font-family: 'Consolas', 'Deja Vu Sans Mono', 'Bitstream Vera Sans Mono', monospace; - font-size: 0.95em; - letter-spacing: 0.01em; -} - -hr { - border: 1px solid #abc; - margin: 2em; -} - -tt { - background-color: #f2f2f2; - border-bottom: 1px solid #ddd; - color: #333; -} - -tt.descname { - background-color: transparent; - font-weight: bold; - font-size: 1.2em; - border: 0; -} - -tt.descclassname { - background-color: transparent; - border: 0; -} - -tt.xref { - background-color: transparent; - font-weight: bold; - border: 0; -} - -a tt { - background-color: transparent; - font-weight: bold; - border: 0; - color: #CA7900; -} - -a tt:hover { - color: #2491CF; -} - -dl { - margin-bottom: 15px; -} - -dd p { - margin-top: 0px; -} - -dd ul, dd table { - margin-bottom: 10px; -} - -dd { - margin-top: 3px; - margin-bottom: 10px; - margin-left: 30px; -} - -.refcount { - color: #060; -} - -dt:target, -.highlight { - background-color: #fbe54e; -} - -dl.class, dl.function { - border-top: 2px solid #888; -} - -dl.method, dl.attribute { - border-top: 1px solid #aaa; -} - -dl.glossary dt { - font-weight: bold; - font-size: 1.1em; -} - -pre { - line-height: 120%; -} - -pre a { - color: inherit; - text-decoration: underline; -} - -.first { - margin-top: 0 !important; -} - -div.document { - background-color: white; - text-align: left; - background-image: url(contents.png); - background-repeat: repeat-x; -} - -/* -div.documentwrapper { - width: 100%; -} -*/ - -div.clearer { - clear: both; -} - -div.related h3 { - display: none; -} - -div.related ul { - background-image: url(navigation.png); - height: 2em; - list-style: none; - border-top: 1px solid #ddd; - border-bottom: 1px solid #ddd; - margin: 0; - padding-left: 10px; -} - -div.related ul li { - margin: 0; - padding: 0; - height: 2em; - float: left; -} - -div.related ul li.right { - float: right; - margin-right: 5px; -} - -div.related ul li a { - margin: 0; - padding: 0 5px 0 5px; - line-height: 1.75em; - color: #EE9816; -} - -div.related ul li a:hover { - color: #3CA8E7; -} - -div.body { - margin: 0; - padding: 0.5em 20px 20px 20px; -} - -div.bodywrapper { - margin: 0 280px 0 0; - border-right: 1px solid #ccc; -} - -div.body a { - text-decoration: underline; -} - -div.sphinxsidebar { - margin: 0; - padding: 0.5em 15px 15px 15px; - width: 250px; - float: right; - text-align: left; - background-color: #eee; -/* margin-left: -100%; */ -} - -div.sphinxsidebarl { - margin: 0; - padding: 0.5em 15px 15px 0px; - width: 210px; - float: left; - clear: left; - text-align: left; -/* margin-left: -100%; */ -} - -div.sphinxsidebar h4, div.sphinxsidebar h3, div.sphinxsidebarl h4, div.sphinxsidebarl h3 { - margin: 1em 0 0.5em 0; - font-size: 0.9em; - padding: 0.1em 0 0.1em 0.5em; - color: white; - border: 1px solid #86989B; - background-color: #AFC1C4; -} - -div.sphinxsidebar ul, div.sphinxsidebarl ul { - padding-left: 1.5em; - margin-top: 7px; - list-style: none; - padding: 0; - line-height: 130%; -} - -div.sphinxsidebar ul ul, div.sphinxsidebarl ul ul { - list-style: square; - margin-left: 20px; -} - -p { - margin: 0.8em 0 0.5em 0; -} - -p.rubric { - font-weight: bold; -} - -h1 { - margin: 0; - padding: 0.7em 0 0.3em 0; - font-size: 1.8em; - /*color: #11557C;*/ - color: #000; -} - -h2 { - margin: 1.3em 0 0.2em 0; - font-size: 1.35em; - padding: 0; -} - -h3 { - margin: 1em 0 -0.3em 0; - font-size: 1.2em; -} - -h1 a, h2 a, h3 a, h4 a, h5 a, h6 a { - color: black!important; -} - -h1 a.anchor, h2 a.anchor, h3 a.anchor, h4 a.anchor, h5 a.anchor, h6 a.anchor { - display: none; - margin: 0 0 0 0.3em; - padding: 0 0.2em 0 0.2em; - color: #aaa!important; -} - -h1:hover a.anchor, h2:hover a.anchor, h3:hover a.anchor, h4:hover a.anchor, -h5:hover a.anchor, h6:hover a.anchor { - display: inline; -} - -h1 a.anchor:hover, h2 a.anchor:hover, h3 a.anchor:hover, h4 a.anchor:hover, -h5 a.anchor:hover, h6 a.anchor:hover { - color: #777; - background-color: #eee; -} - -table { - border-collapse: collapse; - margin: 0 -0.5em 0 -0.5em; -} - -table td, table th { - padding: 0.2em 0.5em 0.2em 0.5em; -} - -div.footer { - background-color: #E3EFF1; - color: #86989B; - padding: 3px 8px 3px 0; - clear: both; - font-size: 0.8em; - text-align: right; -} - -div.footer a { - color: #86989B; - text-decoration: underline; -} - -div.pagination { - margin-top: 2em; - padding-top: 0.5em; - border-top: 1px solid black; - text-align: center; -} - -div.sphinxsidebar ul.toc, div.sphinxsidebarl ul.toc { - margin: 1em 0 1em 0; - padding: 0 0 0 0.5em; - list-style: none; -} - -div.sphinxsidebar ul.toc li, div.sphinxsidebarl ul.toc li { - margin: 0.5em 0 0.5em 0; - font-size: 0.9em; - line-height: 130%; -} - -div.sphinxsidebar ul.toc li p, div.sphinxsidebarl ul.toc li p { - margin: 0; - padding: 0; -} - -div.sphinxsidebar ul.toc ul, div.sphinxsidebarl ul.toc ul { - margin: 0.2em 0 0.2em 0; - padding: 0 0 0 1.8em; -} - -div.sphinxsidebar ul.toc ul li, div.sphinxsidebarl ul.toc ul li { - padding: 0; -} - -div.admonition, div.warning { - font-size: 0.9em; - margin: 1em 0 0 0; - border: 1px solid #86989B; - background-color: #f7f7f7; -} - -div.admonition p, div.warning p { - margin: 0.5em 1em 0.5em 1em; - padding: 0; -} - -div.admonition pre, div.warning pre { - margin: 0.4em 1em 0.4em 1em; -} - -div.admonition p.admonition-title, -div.warning p.admonition-title { - margin: 0; - padding: 0.1em 0 0.1em 0.5em; - color: white; - border-bottom: 1px solid #86989B; - font-weight: bold; - background-color: #AFC1C4; -} - -div.warning { - border: 1px solid #940000; -} - -div.warning p.admonition-title { - background-color: #CF0000; - border-bottom-color: #940000; -} - -div.admonition ul, div.admonition ol, -div.warning ul, div.warning ol { - margin: 0.1em 0.5em 0.5em 3em; - padding: 0; -} - -div.versioninfo { - margin: 1em 0 0 0; - border: 1px solid #ccc; - background-color: #DDEAF0; - padding: 8px; - line-height: 1.3em; - font-size: 0.9em; -} - - -a.headerlink { - color: #c60f0f!important; - font-size: 1em; - margin-left: 6px; - padding: 0 4px 0 4px; - text-decoration: none!important; - visibility: hidden; -} - -h1:hover > a.headerlink, -h2:hover > a.headerlink, -h3:hover > a.headerlink, -h4:hover > a.headerlink, -h5:hover > a.headerlink, -h6:hover > a.headerlink, -dt:hover > a.headerlink { - visibility: visible; -} - -a.headerlink:hover { - background-color: #ccc; - color: white!important; -} - -table.indextable td { - text-align: left; - vertical-align: top; -} - -table.indextable dl, table.indextable dd { - margin-top: 0; - margin-bottom: 0; -} - -table.indextable tr.pcap { - height: 10px; -} - -table.indextable tr.cap { - margin-top: 10px; - background-color: #f2f2f2; -} - -img.toggler { - margin-right: 3px; - margin-top: 3px; - cursor: pointer; -} - -img.inheritance { - border: 0px -} - -form.pfform { - margin: 10px 0 20px 0; -} - -table.contentstable { - width: 90%; -} - -table.contentstable p.biglink { - line-height: 150%; -} - -a.biglink { - font-size: 1.3em; -} - -span.linkdescr { - font-style: italic; - padding-top: 5px; - font-size: 90%; -} - -ul.search { - margin: 10px 0 0 20px; - padding: 0; -} - -ul.search li { - padding: 5px 0 5px 20px; - background-image: url(file.png); - background-repeat: no-repeat; - background-position: 0 7px; -} - -ul.search li a { - font-weight: bold; -} - -ul.search li div.context { - color: #888; - margin: 2px 0 0 30px; - text-align: left; -} - -ul.keywordmatches li.goodmatch a { - font-weight: bold; -} - -.twtr-ft { - display: none; -} diff --git a/docs/src/_static/favicon.ico b/docs/src/_static/favicon.ico new file mode 100644 index 0000000000..3f458e4810 Binary files /dev/null and b/docs/src/_static/favicon.ico differ diff --git a/docs/src/_static/images/arrows.png b/docs/src/_static/images/arrows.png new file mode 100644 index 0000000000..fe7f378c40 Binary files /dev/null and b/docs/src/_static/images/arrows.png differ diff --git a/docs/src/_static/images/bg.png b/docs/src/_static/images/bg.png new file mode 100644 index 0000000000..af55f7ce54 Binary files /dev/null and b/docs/src/_static/images/bg.png differ diff --git a/docs/src/_static/images/bullets.png b/docs/src/_static/images/bullets.png new file mode 100644 index 0000000000..c6404fdf02 Binary files /dev/null and b/docs/src/_static/images/bullets.png differ diff --git a/docs/src/_static/images/checker.png b/docs/src/_static/images/checker.png new file mode 100644 index 0000000000..dffd774b74 Binary files /dev/null and b/docs/src/_static/images/checker.png differ diff --git a/docs/src/_static/images/default.png b/docs/src/_static/images/default.png new file mode 100644 index 0000000000..9fc837c804 Binary files /dev/null and b/docs/src/_static/images/default.png differ diff --git a/docs/src/_static/images/direct-install.png b/docs/src/_static/images/direct-install.png new file mode 100644 index 0000000000..5cfee8efe3 Binary files /dev/null and b/docs/src/_static/images/direct-install.png differ diff --git a/docs/src/_static/images/download.png b/docs/src/_static/images/download.png new file mode 100644 index 0000000000..a63d7242f3 Binary files /dev/null and b/docs/src/_static/images/download.png differ diff --git a/docs/src/_static/images/favicon.ico b/docs/src/_static/images/favicon.ico new file mode 100644 index 0000000000..3f458e4810 Binary files /dev/null and b/docs/src/_static/images/favicon.ico differ diff --git a/docs/src/_static/images/features/converters.png b/docs/src/_static/images/features/converters.png new file mode 100644 index 0000000000..08a07aa010 Binary files /dev/null and b/docs/src/_static/images/features/converters.png differ diff --git a/docs/src/_static/images/features/efficient_implementations.png b/docs/src/_static/images/features/efficient_implementations.png new file mode 100644 index 0000000000..e4d252ec81 Binary files /dev/null and b/docs/src/_static/images/features/efficient_implementations.png differ diff --git a/docs/src/_static/images/features/free_lgpl.png b/docs/src/_static/images/features/free_lgpl.png new file mode 100644 index 0000000000..0ba6d4cc6b Binary files /dev/null and b/docs/src/_static/images/features/free_lgpl.png differ diff --git a/docs/src/_static/images/features/memory_independence.png b/docs/src/_static/images/features/memory_independence.png new file mode 100644 index 0000000000..eea3acfca8 Binary files /dev/null and b/docs/src/_static/images/features/memory_independence.png differ diff --git a/docs/src/_static/images/features/platform_independence.png b/docs/src/_static/images/features/platform_independence.png new file mode 100644 index 0000000000..914dcf89d2 Binary files /dev/null and b/docs/src/_static/images/features/platform_independence.png differ diff --git a/docs/src/_static/images/features/robust.png b/docs/src/_static/images/features/robust.png new file mode 100644 index 0000000000..90687eb481 Binary files /dev/null and b/docs/src/_static/images/features/robust.png differ diff --git a/docs/src/_static/images/features/similarity_queries.png b/docs/src/_static/images/features/similarity_queries.png new file mode 100644 index 0000000000..c02b6f89a1 Binary files /dev/null and b/docs/src/_static/images/features/similarity_queries.png differ diff --git a/docs/src/_static/images/features/support.png b/docs/src/_static/images/features/support.png new file mode 100644 index 0000000000..7f31f2fae2 Binary files /dev/null and b/docs/src/_static/images/features/support.png differ diff --git a/docs/src/_static/images/forkme_left_white_ffffff.png b/docs/src/_static/images/forkme_left_white_ffffff.png new file mode 100644 index 0000000000..60c60a6342 Binary files /dev/null and b/docs/src/_static/images/forkme_left_white_ffffff.png differ diff --git a/docs/src/_static/images/gensim-footer.png b/docs/src/_static/images/gensim-footer.png new file mode 100644 index 0000000000..80bfa7a67d Binary files /dev/null and b/docs/src/_static/images/gensim-footer.png differ diff --git a/docs/src/_static/images/gensim.png b/docs/src/_static/images/gensim.png new file mode 100644 index 0000000000..72701bcb4f Binary files /dev/null and b/docs/src/_static/images/gensim.png differ diff --git a/docs/src/_static/images/gensim_code.png b/docs/src/_static/images/gensim_code.png new file mode 100644 index 0000000000..22dd4457ce Binary files /dev/null and b/docs/src/_static/images/gensim_code.png differ diff --git a/docs/src/_static/images/gensim_compact.png b/docs/src/_static/images/gensim_compact.png new file mode 100644 index 0000000000..908968c551 Binary files /dev/null and b/docs/src/_static/images/gensim_compact.png differ diff --git a/docs/src/_static/images/get-started.png b/docs/src/_static/images/get-started.png new file mode 100644 index 0000000000..38c7297524 Binary files /dev/null and b/docs/src/_static/images/get-started.png differ diff --git a/docs/src/_static/images/googlegroups.png b/docs/src/_static/images/googlegroups.png new file mode 100644 index 0000000000..719c37c044 Binary files /dev/null and b/docs/src/_static/images/googlegroups.png differ diff --git a/docs/src/_static/images/loading.gif b/docs/src/_static/images/loading.gif new file mode 100644 index 0000000000..1560b646cf Binary files /dev/null and b/docs/src/_static/images/loading.gif differ diff --git a/docs/src/_static/images/logo-gensim.png b/docs/src/_static/images/logo-gensim.png new file mode 100644 index 0000000000..738041df57 Binary files /dev/null and b/docs/src/_static/images/logo-gensim.png differ diff --git a/docs/src/_static/images/logo-gensim_compact.png b/docs/src/_static/images/logo-gensim_compact.png new file mode 100644 index 0000000000..7196c78668 Binary files /dev/null and b/docs/src/_static/images/logo-gensim_compact.png differ diff --git a/docs/src/_static/images/menubutton.png b/docs/src/_static/images/menubutton.png new file mode 100644 index 0000000000..0f5af34ee8 Binary files /dev/null and b/docs/src/_static/images/menubutton.png differ diff --git a/docs/src/_static/images/references/logo_dtu.gif b/docs/src/_static/images/references/logo_dtu.gif new file mode 100644 index 0000000000..ed61e62b51 Binary files /dev/null and b/docs/src/_static/images/references/logo_dtu.gif differ diff --git a/docs/src/_static/images/references/logo_dynadmic.png b/docs/src/_static/images/references/logo_dynadmic.png new file mode 100644 index 0000000000..6b862b7e1e Binary files /dev/null and b/docs/src/_static/images/references/logo_dynadmic.png differ diff --git a/docs/src/_static/images/references/logo_eudml.png b/docs/src/_static/images/references/logo_eudml.png new file mode 100644 index 0000000000..7c266bffaa Binary files /dev/null and b/docs/src/_static/images/references/logo_eudml.png differ diff --git a/docs/src/_static/images/references/logo_ghent.png b/docs/src/_static/images/references/logo_ghent.png new file mode 100644 index 0000000000..1f0a683b01 Binary files /dev/null and b/docs/src/_static/images/references/logo_ghent.png differ diff --git a/docs/src/_static/images/references/logo_ibcn.png b/docs/src/_static/images/references/logo_ibcn.png new file mode 100644 index 0000000000..c7ca533bf3 Binary files /dev/null and b/docs/src/_static/images/references/logo_ibcn.png differ diff --git a/docs/src/_static/images/references/logo_roistr.png b/docs/src/_static/images/references/logo_roistr.png new file mode 100644 index 0000000000..4be07bceee Binary files /dev/null and b/docs/src/_static/images/references/logo_roistr.png differ diff --git a/docs/src/_static/images/references/logo_sportsauthority.png b/docs/src/_static/images/references/logo_sportsauthority.png new file mode 100644 index 0000000000..0cbe8ae112 Binary files /dev/null and b/docs/src/_static/images/references/logo_sportsauthority.png differ diff --git a/docs/src/_static/images/tagline.png b/docs/src/_static/images/tagline.png new file mode 100644 index 0000000000..7f0f8567da Binary files /dev/null and b/docs/src/_static/images/tagline.png differ diff --git a/docs/src/_static/images/tagline_compact.png b/docs/src/_static/images/tagline_compact.png new file mode 100644 index 0000000000..0e11e243da Binary files /dev/null and b/docs/src/_static/images/tagline_compact.png differ diff --git a/docs/src/_static/images/twitterbird.png b/docs/src/_static/images/twitterbird.png new file mode 100644 index 0000000000..1adae57427 Binary files /dev/null and b/docs/src/_static/images/twitterbird.png differ diff --git a/docs/src/_static/images/ukazka.png b/docs/src/_static/images/ukazka.png new file mode 100644 index 0000000000..ffe2e18f83 Binary files /dev/null and b/docs/src/_static/images/ukazka.png differ diff --git a/docs/src/_static/images/ukazka2.png b/docs/src/_static/images/ukazka2.png new file mode 100644 index 0000000000..41b8ab9b10 Binary files /dev/null and b/docs/src/_static/images/ukazka2.png differ diff --git a/docs/src/_static/js/jquery-1.9.1.min.js b/docs/src/_static/js/jquery-1.9.1.min.js new file mode 100644 index 0000000000..006e953102 --- /dev/null +++ b/docs/src/_static/js/jquery-1.9.1.min.js @@ -0,0 +1,5 @@ +/*! jQuery v1.9.1 | (c) 2005, 2012 jQuery Foundation, Inc. | jquery.org/license +//@ sourceMappingURL=jquery.min.map +*/(function(e,t){var n,r,i=typeof t,o=e.document,a=e.location,s=e.jQuery,u=e.$,l={},c=[],p="1.9.1",f=c.concat,d=c.push,h=c.slice,g=c.indexOf,m=l.toString,y=l.hasOwnProperty,v=p.trim,b=function(e,t){return new b.fn.init(e,t,r)},x=/[+-]?(?:\d*\.|)\d+(?:[eE][+-]?\d+|)/.source,w=/\S+/g,T=/^[\s\uFEFF\xA0]+|[\s\uFEFF\xA0]+$/g,N=/^(?:(<[\w\W]+>)[^>]*|#([\w-]*))$/,C=/^<(\w+)\s*\/?>(?:<\/\1>|)$/,k=/^[\],:{}\s]*$/,E=/(?:^|:|,)(?:\s*\[)+/g,S=/\\(?:["\\\/bfnrt]|u[\da-fA-F]{4})/g,A=/"[^"\\\r\n]*"|true|false|null|-?(?:\d+\.|)\d+(?:[eE][+-]?\d+|)/g,j=/^-ms-/,D=/-([\da-z])/gi,L=function(e,t){return t.toUpperCase()},H=function(e){(o.addEventListener||"load"===e.type||"complete"===o.readyState)&&(q(),b.ready())},q=function(){o.addEventListener?(o.removeEventListener("DOMContentLoaded",H,!1),e.removeEventListener("load",H,!1)):(o.detachEvent("onreadystatechange",H),e.detachEvent("onload",H))};b.fn=b.prototype={jquery:p,constructor:b,init:function(e,n,r){var i,a;if(!e)return this;if("string"==typeof e){if(i="<"===e.charAt(0)&&">"===e.charAt(e.length-1)&&e.length>=3?[null,e,null]:N.exec(e),!i||!i[1]&&n)return!n||n.jquery?(n||r).find(e):this.constructor(n).find(e);if(i[1]){if(n=n instanceof b?n[0]:n,b.merge(this,b.parseHTML(i[1],n&&n.nodeType?n.ownerDocument||n:o,!0)),C.test(i[1])&&b.isPlainObject(n))for(i in n)b.isFunction(this[i])?this[i](n[i]):this.attr(i,n[i]);return this}if(a=o.getElementById(i[2]),a&&a.parentNode){if(a.id!==i[2])return r.find(e);this.length=1,this[0]=a}return this.context=o,this.selector=e,this}return e.nodeType?(this.context=this[0]=e,this.length=1,this):b.isFunction(e)?r.ready(e):(e.selector!==t&&(this.selector=e.selector,this.context=e.context),b.makeArray(e,this))},selector:"",length:0,size:function(){return this.length},toArray:function(){return h.call(this)},get:function(e){return null==e?this.toArray():0>e?this[this.length+e]:this[e]},pushStack:function(e){var t=b.merge(this.constructor(),e);return t.prevObject=this,t.context=this.context,t},each:function(e,t){return b.each(this,e,t)},ready:function(e){return b.ready.promise().done(e),this},slice:function(){return this.pushStack(h.apply(this,arguments))},first:function(){return this.eq(0)},last:function(){return this.eq(-1)},eq:function(e){var t=this.length,n=+e+(0>e?t:0);return this.pushStack(n>=0&&t>n?[this[n]]:[])},map:function(e){return this.pushStack(b.map(this,function(t,n){return e.call(t,n,t)}))},end:function(){return this.prevObject||this.constructor(null)},push:d,sort:[].sort,splice:[].splice},b.fn.init.prototype=b.fn,b.extend=b.fn.extend=function(){var e,n,r,i,o,a,s=arguments[0]||{},u=1,l=arguments.length,c=!1;for("boolean"==typeof s&&(c=s,s=arguments[1]||{},u=2),"object"==typeof s||b.isFunction(s)||(s={}),l===u&&(s=this,--u);l>u;u++)if(null!=(o=arguments[u]))for(i in o)e=s[i],r=o[i],s!==r&&(c&&r&&(b.isPlainObject(r)||(n=b.isArray(r)))?(n?(n=!1,a=e&&b.isArray(e)?e:[]):a=e&&b.isPlainObject(e)?e:{},s[i]=b.extend(c,a,r)):r!==t&&(s[i]=r));return s},b.extend({noConflict:function(t){return e.$===b&&(e.$=u),t&&e.jQuery===b&&(e.jQuery=s),b},isReady:!1,readyWait:1,holdReady:function(e){e?b.readyWait++:b.ready(!0)},ready:function(e){if(e===!0?!--b.readyWait:!b.isReady){if(!o.body)return setTimeout(b.ready);b.isReady=!0,e!==!0&&--b.readyWait>0||(n.resolveWith(o,[b]),b.fn.trigger&&b(o).trigger("ready").off("ready"))}},isFunction:function(e){return"function"===b.type(e)},isArray:Array.isArray||function(e){return"array"===b.type(e)},isWindow:function(e){return null!=e&&e==e.window},isNumeric:function(e){return!isNaN(parseFloat(e))&&isFinite(e)},type:function(e){return null==e?e+"":"object"==typeof e||"function"==typeof e?l[m.call(e)]||"object":typeof e},isPlainObject:function(e){if(!e||"object"!==b.type(e)||e.nodeType||b.isWindow(e))return!1;try{if(e.constructor&&!y.call(e,"constructor")&&!y.call(e.constructor.prototype,"isPrototypeOf"))return!1}catch(n){return!1}var r;for(r in e);return r===t||y.call(e,r)},isEmptyObject:function(e){var t;for(t in e)return!1;return!0},error:function(e){throw Error(e)},parseHTML:function(e,t,n){if(!e||"string"!=typeof e)return null;"boolean"==typeof t&&(n=t,t=!1),t=t||o;var r=C.exec(e),i=!n&&[];return r?[t.createElement(r[1])]:(r=b.buildFragment([e],t,i),i&&b(i).remove(),b.merge([],r.childNodes))},parseJSON:function(n){return e.JSON&&e.JSON.parse?e.JSON.parse(n):null===n?n:"string"==typeof n&&(n=b.trim(n),n&&k.test(n.replace(S,"@").replace(A,"]").replace(E,"")))?Function("return "+n)():(b.error("Invalid JSON: "+n),t)},parseXML:function(n){var r,i;if(!n||"string"!=typeof n)return null;try{e.DOMParser?(i=new DOMParser,r=i.parseFromString(n,"text/xml")):(r=new ActiveXObject("Microsoft.XMLDOM"),r.async="false",r.loadXML(n))}catch(o){r=t}return r&&r.documentElement&&!r.getElementsByTagName("parsererror").length||b.error("Invalid XML: "+n),r},noop:function(){},globalEval:function(t){t&&b.trim(t)&&(e.execScript||function(t){e.eval.call(e,t)})(t)},camelCase:function(e){return e.replace(j,"ms-").replace(D,L)},nodeName:function(e,t){return e.nodeName&&e.nodeName.toLowerCase()===t.toLowerCase()},each:function(e,t,n){var r,i=0,o=e.length,a=M(e);if(n){if(a){for(;o>i;i++)if(r=t.apply(e[i],n),r===!1)break}else for(i in e)if(r=t.apply(e[i],n),r===!1)break}else if(a){for(;o>i;i++)if(r=t.call(e[i],i,e[i]),r===!1)break}else for(i in e)if(r=t.call(e[i],i,e[i]),r===!1)break;return e},trim:v&&!v.call("\ufeff\u00a0")?function(e){return null==e?"":v.call(e)}:function(e){return null==e?"":(e+"").replace(T,"")},makeArray:function(e,t){var n=t||[];return null!=e&&(M(Object(e))?b.merge(n,"string"==typeof e?[e]:e):d.call(n,e)),n},inArray:function(e,t,n){var r;if(t){if(g)return g.call(t,e,n);for(r=t.length,n=n?0>n?Math.max(0,r+n):n:0;r>n;n++)if(n in t&&t[n]===e)return n}return-1},merge:function(e,n){var r=n.length,i=e.length,o=0;if("number"==typeof r)for(;r>o;o++)e[i++]=n[o];else while(n[o]!==t)e[i++]=n[o++];return e.length=i,e},grep:function(e,t,n){var r,i=[],o=0,a=e.length;for(n=!!n;a>o;o++)r=!!t(e[o],o),n!==r&&i.push(e[o]);return i},map:function(e,t,n){var r,i=0,o=e.length,a=M(e),s=[];if(a)for(;o>i;i++)r=t(e[i],i,n),null!=r&&(s[s.length]=r);else for(i in e)r=t(e[i],i,n),null!=r&&(s[s.length]=r);return f.apply([],s)},guid:1,proxy:function(e,n){var r,i,o;return"string"==typeof n&&(o=e[n],n=e,e=o),b.isFunction(e)?(r=h.call(arguments,2),i=function(){return e.apply(n||this,r.concat(h.call(arguments)))},i.guid=e.guid=e.guid||b.guid++,i):t},access:function(e,n,r,i,o,a,s){var u=0,l=e.length,c=null==r;if("object"===b.type(r)){o=!0;for(u in r)b.access(e,n,u,r[u],!0,a,s)}else if(i!==t&&(o=!0,b.isFunction(i)||(s=!0),c&&(s?(n.call(e,i),n=null):(c=n,n=function(e,t,n){return c.call(b(e),n)})),n))for(;l>u;u++)n(e[u],r,s?i:i.call(e[u],u,n(e[u],r)));return o?e:c?n.call(e):l?n(e[0],r):a},now:function(){return(new Date).getTime()}}),b.ready.promise=function(t){if(!n)if(n=b.Deferred(),"complete"===o.readyState)setTimeout(b.ready);else if(o.addEventListener)o.addEventListener("DOMContentLoaded",H,!1),e.addEventListener("load",H,!1);else{o.attachEvent("onreadystatechange",H),e.attachEvent("onload",H);var r=!1;try{r=null==e.frameElement&&o.documentElement}catch(i){}r&&r.doScroll&&function a(){if(!b.isReady){try{r.doScroll("left")}catch(e){return setTimeout(a,50)}q(),b.ready()}}()}return n.promise(t)},b.each("Boolean Number String Function Array Date RegExp Object Error".split(" "),function(e,t){l["[object "+t+"]"]=t.toLowerCase()});function M(e){var t=e.length,n=b.type(e);return b.isWindow(e)?!1:1===e.nodeType&&t?!0:"array"===n||"function"!==n&&(0===t||"number"==typeof t&&t>0&&t-1 in e)}r=b(o);var _={};function F(e){var t=_[e]={};return b.each(e.match(w)||[],function(e,n){t[n]=!0}),t}b.Callbacks=function(e){e="string"==typeof e?_[e]||F(e):b.extend({},e);var n,r,i,o,a,s,u=[],l=!e.once&&[],c=function(t){for(r=e.memory&&t,i=!0,a=s||0,s=0,o=u.length,n=!0;u&&o>a;a++)if(u[a].apply(t[0],t[1])===!1&&e.stopOnFalse){r=!1;break}n=!1,u&&(l?l.length&&c(l.shift()):r?u=[]:p.disable())},p={add:function(){if(u){var t=u.length;(function i(t){b.each(t,function(t,n){var r=b.type(n);"function"===r?e.unique&&p.has(n)||u.push(n):n&&n.length&&"string"!==r&&i(n)})})(arguments),n?o=u.length:r&&(s=t,c(r))}return this},remove:function(){return u&&b.each(arguments,function(e,t){var r;while((r=b.inArray(t,u,r))>-1)u.splice(r,1),n&&(o>=r&&o--,a>=r&&a--)}),this},has:function(e){return e?b.inArray(e,u)>-1:!(!u||!u.length)},empty:function(){return u=[],this},disable:function(){return u=l=r=t,this},disabled:function(){return!u},lock:function(){return l=t,r||p.disable(),this},locked:function(){return!l},fireWith:function(e,t){return t=t||[],t=[e,t.slice?t.slice():t],!u||i&&!l||(n?l.push(t):c(t)),this},fire:function(){return p.fireWith(this,arguments),this},fired:function(){return!!i}};return p},b.extend({Deferred:function(e){var t=[["resolve","done",b.Callbacks("once memory"),"resolved"],["reject","fail",b.Callbacks("once memory"),"rejected"],["notify","progress",b.Callbacks("memory")]],n="pending",r={state:function(){return n},always:function(){return i.done(arguments).fail(arguments),this},then:function(){var e=arguments;return b.Deferred(function(n){b.each(t,function(t,o){var a=o[0],s=b.isFunction(e[t])&&e[t];i[o[1]](function(){var e=s&&s.apply(this,arguments);e&&b.isFunction(e.promise)?e.promise().done(n.resolve).fail(n.reject).progress(n.notify):n[a+"With"](this===r?n.promise():this,s?[e]:arguments)})}),e=null}).promise()},promise:function(e){return null!=e?b.extend(e,r):r}},i={};return r.pipe=r.then,b.each(t,function(e,o){var a=o[2],s=o[3];r[o[1]]=a.add,s&&a.add(function(){n=s},t[1^e][2].disable,t[2][2].lock),i[o[0]]=function(){return i[o[0]+"With"](this===i?r:this,arguments),this},i[o[0]+"With"]=a.fireWith}),r.promise(i),e&&e.call(i,i),i},when:function(e){var t=0,n=h.call(arguments),r=n.length,i=1!==r||e&&b.isFunction(e.promise)?r:0,o=1===i?e:b.Deferred(),a=function(e,t,n){return function(r){t[e]=this,n[e]=arguments.length>1?h.call(arguments):r,n===s?o.notifyWith(t,n):--i||o.resolveWith(t,n)}},s,u,l;if(r>1)for(s=Array(r),u=Array(r),l=Array(r);r>t;t++)n[t]&&b.isFunction(n[t].promise)?n[t].promise().done(a(t,l,n)).fail(o.reject).progress(a(t,u,s)):--i;return i||o.resolveWith(l,n),o.promise()}}),b.support=function(){var t,n,r,a,s,u,l,c,p,f,d=o.createElement("div");if(d.setAttribute("className","t"),d.innerHTML="
    a",n=d.getElementsByTagName("*"),r=d.getElementsByTagName("a")[0],!n||!r||!n.length)return{};s=o.createElement("select"),l=s.appendChild(o.createElement("option")),a=d.getElementsByTagName("input")[0],r.style.cssText="top:1px;float:left;opacity:.5",t={getSetAttribute:"t"!==d.className,leadingWhitespace:3===d.firstChild.nodeType,tbody:!d.getElementsByTagName("tbody").length,htmlSerialize:!!d.getElementsByTagName("link").length,style:/top/.test(r.getAttribute("style")),hrefNormalized:"/a"===r.getAttribute("href"),opacity:/^0.5/.test(r.style.opacity),cssFloat:!!r.style.cssFloat,checkOn:!!a.value,optSelected:l.selected,enctype:!!o.createElement("form").enctype,html5Clone:"<:nav>"!==o.createElement("nav").cloneNode(!0).outerHTML,boxModel:"CSS1Compat"===o.compatMode,deleteExpando:!0,noCloneEvent:!0,inlineBlockNeedsLayout:!1,shrinkWrapBlocks:!1,reliableMarginRight:!0,boxSizingReliable:!0,pixelPosition:!1},a.checked=!0,t.noCloneChecked=a.cloneNode(!0).checked,s.disabled=!0,t.optDisabled=!l.disabled;try{delete d.test}catch(h){t.deleteExpando=!1}a=o.createElement("input"),a.setAttribute("value",""),t.input=""===a.getAttribute("value"),a.value="t",a.setAttribute("type","radio"),t.radioValue="t"===a.value,a.setAttribute("checked","t"),a.setAttribute("name","t"),u=o.createDocumentFragment(),u.appendChild(a),t.appendChecked=a.checked,t.checkClone=u.cloneNode(!0).cloneNode(!0).lastChild.checked,d.attachEvent&&(d.attachEvent("onclick",function(){t.noCloneEvent=!1}),d.cloneNode(!0).click());for(f in{submit:!0,change:!0,focusin:!0})d.setAttribute(c="on"+f,"t"),t[f+"Bubbles"]=c in e||d.attributes[c].expando===!1;return d.style.backgroundClip="content-box",d.cloneNode(!0).style.backgroundClip="",t.clearCloneStyle="content-box"===d.style.backgroundClip,b(function(){var n,r,a,s="padding:0;margin:0;border:0;display:block;box-sizing:content-box;-moz-box-sizing:content-box;-webkit-box-sizing:content-box;",u=o.getElementsByTagName("body")[0];u&&(n=o.createElement("div"),n.style.cssText="border:0;width:0;height:0;position:absolute;top:0;left:-9999px;margin-top:1px",u.appendChild(n).appendChild(d),d.innerHTML="
    t
    ",a=d.getElementsByTagName("td"),a[0].style.cssText="padding:0;margin:0;border:0;display:none",p=0===a[0].offsetHeight,a[0].style.display="",a[1].style.display="none",t.reliableHiddenOffsets=p&&0===a[0].offsetHeight,d.innerHTML="",d.style.cssText="box-sizing:border-box;-moz-box-sizing:border-box;-webkit-box-sizing:border-box;padding:1px;border:1px;display:block;width:4px;margin-top:1%;position:absolute;top:1%;",t.boxSizing=4===d.offsetWidth,t.doesNotIncludeMarginInBodyOffset=1!==u.offsetTop,e.getComputedStyle&&(t.pixelPosition="1%"!==(e.getComputedStyle(d,null)||{}).top,t.boxSizingReliable="4px"===(e.getComputedStyle(d,null)||{width:"4px"}).width,r=d.appendChild(o.createElement("div")),r.style.cssText=d.style.cssText=s,r.style.marginRight=r.style.width="0",d.style.width="1px",t.reliableMarginRight=!parseFloat((e.getComputedStyle(r,null)||{}).marginRight)),typeof d.style.zoom!==i&&(d.innerHTML="",d.style.cssText=s+"width:1px;padding:1px;display:inline;zoom:1",t.inlineBlockNeedsLayout=3===d.offsetWidth,d.style.display="block",d.innerHTML="
    ",d.firstChild.style.width="5px",t.shrinkWrapBlocks=3!==d.offsetWidth,t.inlineBlockNeedsLayout&&(u.style.zoom=1)),u.removeChild(n),n=d=a=r=null)}),n=s=u=l=r=a=null,t}();var O=/(?:\{[\s\S]*\}|\[[\s\S]*\])$/,B=/([A-Z])/g;function P(e,n,r,i){if(b.acceptData(e)){var o,a,s=b.expando,u="string"==typeof n,l=e.nodeType,p=l?b.cache:e,f=l?e[s]:e[s]&&s;if(f&&p[f]&&(i||p[f].data)||!u||r!==t)return f||(l?e[s]=f=c.pop()||b.guid++:f=s),p[f]||(p[f]={},l||(p[f].toJSON=b.noop)),("object"==typeof n||"function"==typeof n)&&(i?p[f]=b.extend(p[f],n):p[f].data=b.extend(p[f].data,n)),o=p[f],i||(o.data||(o.data={}),o=o.data),r!==t&&(o[b.camelCase(n)]=r),u?(a=o[n],null==a&&(a=o[b.camelCase(n)])):a=o,a}}function R(e,t,n){if(b.acceptData(e)){var r,i,o,a=e.nodeType,s=a?b.cache:e,u=a?e[b.expando]:b.expando;if(s[u]){if(t&&(o=n?s[u]:s[u].data)){b.isArray(t)?t=t.concat(b.map(t,b.camelCase)):t in o?t=[t]:(t=b.camelCase(t),t=t in o?[t]:t.split(" "));for(r=0,i=t.length;i>r;r++)delete o[t[r]];if(!(n?$:b.isEmptyObject)(o))return}(n||(delete s[u].data,$(s[u])))&&(a?b.cleanData([e],!0):b.support.deleteExpando||s!=s.window?delete s[u]:s[u]=null)}}}b.extend({cache:{},expando:"jQuery"+(p+Math.random()).replace(/\D/g,""),noData:{embed:!0,object:"clsid:D27CDB6E-AE6D-11cf-96B8-444553540000",applet:!0},hasData:function(e){return e=e.nodeType?b.cache[e[b.expando]]:e[b.expando],!!e&&!$(e)},data:function(e,t,n){return P(e,t,n)},removeData:function(e,t){return R(e,t)},_data:function(e,t,n){return P(e,t,n,!0)},_removeData:function(e,t){return R(e,t,!0)},acceptData:function(e){if(e.nodeType&&1!==e.nodeType&&9!==e.nodeType)return!1;var t=e.nodeName&&b.noData[e.nodeName.toLowerCase()];return!t||t!==!0&&e.getAttribute("classid")===t}}),b.fn.extend({data:function(e,n){var r,i,o=this[0],a=0,s=null;if(e===t){if(this.length&&(s=b.data(o),1===o.nodeType&&!b._data(o,"parsedAttrs"))){for(r=o.attributes;r.length>a;a++)i=r[a].name,i.indexOf("data-")||(i=b.camelCase(i.slice(5)),W(o,i,s[i]));b._data(o,"parsedAttrs",!0)}return s}return"object"==typeof e?this.each(function(){b.data(this,e)}):b.access(this,function(n){return n===t?o?W(o,e,b.data(o,e)):null:(this.each(function(){b.data(this,e,n)}),t)},null,n,arguments.length>1,null,!0)},removeData:function(e){return this.each(function(){b.removeData(this,e)})}});function W(e,n,r){if(r===t&&1===e.nodeType){var i="data-"+n.replace(B,"-$1").toLowerCase();if(r=e.getAttribute(i),"string"==typeof r){try{r="true"===r?!0:"false"===r?!1:"null"===r?null:+r+""===r?+r:O.test(r)?b.parseJSON(r):r}catch(o){}b.data(e,n,r)}else r=t}return r}function $(e){var t;for(t in e)if(("data"!==t||!b.isEmptyObject(e[t]))&&"toJSON"!==t)return!1;return!0}b.extend({queue:function(e,n,r){var i;return e?(n=(n||"fx")+"queue",i=b._data(e,n),r&&(!i||b.isArray(r)?i=b._data(e,n,b.makeArray(r)):i.push(r)),i||[]):t},dequeue:function(e,t){t=t||"fx";var n=b.queue(e,t),r=n.length,i=n.shift(),o=b._queueHooks(e,t),a=function(){b.dequeue(e,t)};"inprogress"===i&&(i=n.shift(),r--),o.cur=i,i&&("fx"===t&&n.unshift("inprogress"),delete o.stop,i.call(e,a,o)),!r&&o&&o.empty.fire()},_queueHooks:function(e,t){var n=t+"queueHooks";return b._data(e,n)||b._data(e,n,{empty:b.Callbacks("once memory").add(function(){b._removeData(e,t+"queue"),b._removeData(e,n)})})}}),b.fn.extend({queue:function(e,n){var r=2;return"string"!=typeof e&&(n=e,e="fx",r--),r>arguments.length?b.queue(this[0],e):n===t?this:this.each(function(){var t=b.queue(this,e,n);b._queueHooks(this,e),"fx"===e&&"inprogress"!==t[0]&&b.dequeue(this,e)})},dequeue:function(e){return this.each(function(){b.dequeue(this,e)})},delay:function(e,t){return e=b.fx?b.fx.speeds[e]||e:e,t=t||"fx",this.queue(t,function(t,n){var r=setTimeout(t,e);n.stop=function(){clearTimeout(r)}})},clearQueue:function(e){return this.queue(e||"fx",[])},promise:function(e,n){var r,i=1,o=b.Deferred(),a=this,s=this.length,u=function(){--i||o.resolveWith(a,[a])};"string"!=typeof e&&(n=e,e=t),e=e||"fx";while(s--)r=b._data(a[s],e+"queueHooks"),r&&r.empty&&(i++,r.empty.add(u));return u(),o.promise(n)}});var I,z,X=/[\t\r\n]/g,U=/\r/g,V=/^(?:input|select|textarea|button|object)$/i,Y=/^(?:a|area)$/i,J=/^(?:checked|selected|autofocus|autoplay|async|controls|defer|disabled|hidden|loop|multiple|open|readonly|required|scoped)$/i,G=/^(?:checked|selected)$/i,Q=b.support.getSetAttribute,K=b.support.input;b.fn.extend({attr:function(e,t){return b.access(this,b.attr,e,t,arguments.length>1)},removeAttr:function(e){return this.each(function(){b.removeAttr(this,e)})},prop:function(e,t){return b.access(this,b.prop,e,t,arguments.length>1)},removeProp:function(e){return e=b.propFix[e]||e,this.each(function(){try{this[e]=t,delete this[e]}catch(n){}})},addClass:function(e){var t,n,r,i,o,a=0,s=this.length,u="string"==typeof e&&e;if(b.isFunction(e))return this.each(function(t){b(this).addClass(e.call(this,t,this.className))});if(u)for(t=(e||"").match(w)||[];s>a;a++)if(n=this[a],r=1===n.nodeType&&(n.className?(" "+n.className+" ").replace(X," "):" ")){o=0;while(i=t[o++])0>r.indexOf(" "+i+" ")&&(r+=i+" ");n.className=b.trim(r)}return this},removeClass:function(e){var t,n,r,i,o,a=0,s=this.length,u=0===arguments.length||"string"==typeof e&&e;if(b.isFunction(e))return this.each(function(t){b(this).removeClass(e.call(this,t,this.className))});if(u)for(t=(e||"").match(w)||[];s>a;a++)if(n=this[a],r=1===n.nodeType&&(n.className?(" "+n.className+" ").replace(X," "):"")){o=0;while(i=t[o++])while(r.indexOf(" "+i+" ")>=0)r=r.replace(" "+i+" "," ");n.className=e?b.trim(r):""}return this},toggleClass:function(e,t){var n=typeof e,r="boolean"==typeof t;return b.isFunction(e)?this.each(function(n){b(this).toggleClass(e.call(this,n,this.className,t),t)}):this.each(function(){if("string"===n){var o,a=0,s=b(this),u=t,l=e.match(w)||[];while(o=l[a++])u=r?u:!s.hasClass(o),s[u?"addClass":"removeClass"](o)}else(n===i||"boolean"===n)&&(this.className&&b._data(this,"__className__",this.className),this.className=this.className||e===!1?"":b._data(this,"__className__")||"")})},hasClass:function(e){var t=" "+e+" ",n=0,r=this.length;for(;r>n;n++)if(1===this[n].nodeType&&(" "+this[n].className+" ").replace(X," ").indexOf(t)>=0)return!0;return!1},val:function(e){var n,r,i,o=this[0];{if(arguments.length)return i=b.isFunction(e),this.each(function(n){var o,a=b(this);1===this.nodeType&&(o=i?e.call(this,n,a.val()):e,null==o?o="":"number"==typeof o?o+="":b.isArray(o)&&(o=b.map(o,function(e){return null==e?"":e+""})),r=b.valHooks[this.type]||b.valHooks[this.nodeName.toLowerCase()],r&&"set"in r&&r.set(this,o,"value")!==t||(this.value=o))});if(o)return r=b.valHooks[o.type]||b.valHooks[o.nodeName.toLowerCase()],r&&"get"in r&&(n=r.get(o,"value"))!==t?n:(n=o.value,"string"==typeof n?n.replace(U,""):null==n?"":n)}}}),b.extend({valHooks:{option:{get:function(e){var t=e.attributes.value;return!t||t.specified?e.value:e.text}},select:{get:function(e){var t,n,r=e.options,i=e.selectedIndex,o="select-one"===e.type||0>i,a=o?null:[],s=o?i+1:r.length,u=0>i?s:o?i:0;for(;s>u;u++)if(n=r[u],!(!n.selected&&u!==i||(b.support.optDisabled?n.disabled:null!==n.getAttribute("disabled"))||n.parentNode.disabled&&b.nodeName(n.parentNode,"optgroup"))){if(t=b(n).val(),o)return t;a.push(t)}return a},set:function(e,t){var n=b.makeArray(t);return b(e).find("option").each(function(){this.selected=b.inArray(b(this).val(),n)>=0}),n.length||(e.selectedIndex=-1),n}}},attr:function(e,n,r){var o,a,s,u=e.nodeType;if(e&&3!==u&&8!==u&&2!==u)return typeof e.getAttribute===i?b.prop(e,n,r):(a=1!==u||!b.isXMLDoc(e),a&&(n=n.toLowerCase(),o=b.attrHooks[n]||(J.test(n)?z:I)),r===t?o&&a&&"get"in o&&null!==(s=o.get(e,n))?s:(typeof e.getAttribute!==i&&(s=e.getAttribute(n)),null==s?t:s):null!==r?o&&a&&"set"in o&&(s=o.set(e,r,n))!==t?s:(e.setAttribute(n,r+""),r):(b.removeAttr(e,n),t))},removeAttr:function(e,t){var n,r,i=0,o=t&&t.match(w);if(o&&1===e.nodeType)while(n=o[i++])r=b.propFix[n]||n,J.test(n)?!Q&&G.test(n)?e[b.camelCase("default-"+n)]=e[r]=!1:e[r]=!1:b.attr(e,n,""),e.removeAttribute(Q?n:r)},attrHooks:{type:{set:function(e,t){if(!b.support.radioValue&&"radio"===t&&b.nodeName(e,"input")){var n=e.value;return e.setAttribute("type",t),n&&(e.value=n),t}}}},propFix:{tabindex:"tabIndex",readonly:"readOnly","for":"htmlFor","class":"className",maxlength:"maxLength",cellspacing:"cellSpacing",cellpadding:"cellPadding",rowspan:"rowSpan",colspan:"colSpan",usemap:"useMap",frameborder:"frameBorder",contenteditable:"contentEditable"},prop:function(e,n,r){var i,o,a,s=e.nodeType;if(e&&3!==s&&8!==s&&2!==s)return a=1!==s||!b.isXMLDoc(e),a&&(n=b.propFix[n]||n,o=b.propHooks[n]),r!==t?o&&"set"in o&&(i=o.set(e,r,n))!==t?i:e[n]=r:o&&"get"in o&&null!==(i=o.get(e,n))?i:e[n]},propHooks:{tabIndex:{get:function(e){var n=e.getAttributeNode("tabindex");return n&&n.specified?parseInt(n.value,10):V.test(e.nodeName)||Y.test(e.nodeName)&&e.href?0:t}}}}),z={get:function(e,n){var r=b.prop(e,n),i="boolean"==typeof r&&e.getAttribute(n),o="boolean"==typeof r?K&&Q?null!=i:G.test(n)?e[b.camelCase("default-"+n)]:!!i:e.getAttributeNode(n);return o&&o.value!==!1?n.toLowerCase():t},set:function(e,t,n){return t===!1?b.removeAttr(e,n):K&&Q||!G.test(n)?e.setAttribute(!Q&&b.propFix[n]||n,n):e[b.camelCase("default-"+n)]=e[n]=!0,n}},K&&Q||(b.attrHooks.value={get:function(e,n){var r=e.getAttributeNode(n);return b.nodeName(e,"input")?e.defaultValue:r&&r.specified?r.value:t},set:function(e,n,r){return b.nodeName(e,"input")?(e.defaultValue=n,t):I&&I.set(e,n,r)}}),Q||(I=b.valHooks.button={get:function(e,n){var r=e.getAttributeNode(n);return r&&("id"===n||"name"===n||"coords"===n?""!==r.value:r.specified)?r.value:t},set:function(e,n,r){var i=e.getAttributeNode(r);return i||e.setAttributeNode(i=e.ownerDocument.createAttribute(r)),i.value=n+="","value"===r||n===e.getAttribute(r)?n:t}},b.attrHooks.contenteditable={get:I.get,set:function(e,t,n){I.set(e,""===t?!1:t,n)}},b.each(["width","height"],function(e,n){b.attrHooks[n]=b.extend(b.attrHooks[n],{set:function(e,r){return""===r?(e.setAttribute(n,"auto"),r):t}})})),b.support.hrefNormalized||(b.each(["href","src","width","height"],function(e,n){b.attrHooks[n]=b.extend(b.attrHooks[n],{get:function(e){var r=e.getAttribute(n,2);return null==r?t:r}})}),b.each(["href","src"],function(e,t){b.propHooks[t]={get:function(e){return e.getAttribute(t,4)}}})),b.support.style||(b.attrHooks.style={get:function(e){return e.style.cssText||t},set:function(e,t){return e.style.cssText=t+""}}),b.support.optSelected||(b.propHooks.selected=b.extend(b.propHooks.selected,{get:function(e){var t=e.parentNode;return t&&(t.selectedIndex,t.parentNode&&t.parentNode.selectedIndex),null}})),b.support.enctype||(b.propFix.enctype="encoding"),b.support.checkOn||b.each(["radio","checkbox"],function(){b.valHooks[this]={get:function(e){return null===e.getAttribute("value")?"on":e.value}}}),b.each(["radio","checkbox"],function(){b.valHooks[this]=b.extend(b.valHooks[this],{set:function(e,n){return b.isArray(n)?e.checked=b.inArray(b(e).val(),n)>=0:t}})});var Z=/^(?:input|select|textarea)$/i,et=/^key/,tt=/^(?:mouse|contextmenu)|click/,nt=/^(?:focusinfocus|focusoutblur)$/,rt=/^([^.]*)(?:\.(.+)|)$/;function it(){return!0}function ot(){return!1}b.event={global:{},add:function(e,n,r,o,a){var s,u,l,c,p,f,d,h,g,m,y,v=b._data(e);if(v){r.handler&&(c=r,r=c.handler,a=c.selector),r.guid||(r.guid=b.guid++),(u=v.events)||(u=v.events={}),(f=v.handle)||(f=v.handle=function(e){return typeof b===i||e&&b.event.triggered===e.type?t:b.event.dispatch.apply(f.elem,arguments)},f.elem=e),n=(n||"").match(w)||[""],l=n.length;while(l--)s=rt.exec(n[l])||[],g=y=s[1],m=(s[2]||"").split(".").sort(),p=b.event.special[g]||{},g=(a?p.delegateType:p.bindType)||g,p=b.event.special[g]||{},d=b.extend({type:g,origType:y,data:o,handler:r,guid:r.guid,selector:a,needsContext:a&&b.expr.match.needsContext.test(a),namespace:m.join(".")},c),(h=u[g])||(h=u[g]=[],h.delegateCount=0,p.setup&&p.setup.call(e,o,m,f)!==!1||(e.addEventListener?e.addEventListener(g,f,!1):e.attachEvent&&e.attachEvent("on"+g,f))),p.add&&(p.add.call(e,d),d.handler.guid||(d.handler.guid=r.guid)),a?h.splice(h.delegateCount++,0,d):h.push(d),b.event.global[g]=!0;e=null}},remove:function(e,t,n,r,i){var o,a,s,u,l,c,p,f,d,h,g,m=b.hasData(e)&&b._data(e);if(m&&(c=m.events)){t=(t||"").match(w)||[""],l=t.length;while(l--)if(s=rt.exec(t[l])||[],d=g=s[1],h=(s[2]||"").split(".").sort(),d){p=b.event.special[d]||{},d=(r?p.delegateType:p.bindType)||d,f=c[d]||[],s=s[2]&&RegExp("(^|\\.)"+h.join("\\.(?:.*\\.|)")+"(\\.|$)"),u=o=f.length;while(o--)a=f[o],!i&&g!==a.origType||n&&n.guid!==a.guid||s&&!s.test(a.namespace)||r&&r!==a.selector&&("**"!==r||!a.selector)||(f.splice(o,1),a.selector&&f.delegateCount--,p.remove&&p.remove.call(e,a));u&&!f.length&&(p.teardown&&p.teardown.call(e,h,m.handle)!==!1||b.removeEvent(e,d,m.handle),delete c[d])}else for(d in c)b.event.remove(e,d+t[l],n,r,!0);b.isEmptyObject(c)&&(delete m.handle,b._removeData(e,"events"))}},trigger:function(n,r,i,a){var s,u,l,c,p,f,d,h=[i||o],g=y.call(n,"type")?n.type:n,m=y.call(n,"namespace")?n.namespace.split("."):[];if(l=f=i=i||o,3!==i.nodeType&&8!==i.nodeType&&!nt.test(g+b.event.triggered)&&(g.indexOf(".")>=0&&(m=g.split("."),g=m.shift(),m.sort()),u=0>g.indexOf(":")&&"on"+g,n=n[b.expando]?n:new b.Event(g,"object"==typeof n&&n),n.isTrigger=!0,n.namespace=m.join("."),n.namespace_re=n.namespace?RegExp("(^|\\.)"+m.join("\\.(?:.*\\.|)")+"(\\.|$)"):null,n.result=t,n.target||(n.target=i),r=null==r?[n]:b.makeArray(r,[n]),p=b.event.special[g]||{},a||!p.trigger||p.trigger.apply(i,r)!==!1)){if(!a&&!p.noBubble&&!b.isWindow(i)){for(c=p.delegateType||g,nt.test(c+g)||(l=l.parentNode);l;l=l.parentNode)h.push(l),f=l;f===(i.ownerDocument||o)&&h.push(f.defaultView||f.parentWindow||e)}d=0;while((l=h[d++])&&!n.isPropagationStopped())n.type=d>1?c:p.bindType||g,s=(b._data(l,"events")||{})[n.type]&&b._data(l,"handle"),s&&s.apply(l,r),s=u&&l[u],s&&b.acceptData(l)&&s.apply&&s.apply(l,r)===!1&&n.preventDefault();if(n.type=g,!(a||n.isDefaultPrevented()||p._default&&p._default.apply(i.ownerDocument,r)!==!1||"click"===g&&b.nodeName(i,"a")||!b.acceptData(i)||!u||!i[g]||b.isWindow(i))){f=i[u],f&&(i[u]=null),b.event.triggered=g;try{i[g]()}catch(v){}b.event.triggered=t,f&&(i[u]=f)}return n.result}},dispatch:function(e){e=b.event.fix(e);var n,r,i,o,a,s=[],u=h.call(arguments),l=(b._data(this,"events")||{})[e.type]||[],c=b.event.special[e.type]||{};if(u[0]=e,e.delegateTarget=this,!c.preDispatch||c.preDispatch.call(this,e)!==!1){s=b.event.handlers.call(this,e,l),n=0;while((o=s[n++])&&!e.isPropagationStopped()){e.currentTarget=o.elem,a=0;while((i=o.handlers[a++])&&!e.isImmediatePropagationStopped())(!e.namespace_re||e.namespace_re.test(i.namespace))&&(e.handleObj=i,e.data=i.data,r=((b.event.special[i.origType]||{}).handle||i.handler).apply(o.elem,u),r!==t&&(e.result=r)===!1&&(e.preventDefault(),e.stopPropagation()))}return c.postDispatch&&c.postDispatch.call(this,e),e.result}},handlers:function(e,n){var r,i,o,a,s=[],u=n.delegateCount,l=e.target;if(u&&l.nodeType&&(!e.button||"click"!==e.type))for(;l!=this;l=l.parentNode||this)if(1===l.nodeType&&(l.disabled!==!0||"click"!==e.type)){for(o=[],a=0;u>a;a++)i=n[a],r=i.selector+" ",o[r]===t&&(o[r]=i.needsContext?b(r,this).index(l)>=0:b.find(r,this,null,[l]).length),o[r]&&o.push(i);o.length&&s.push({elem:l,handlers:o})}return n.length>u&&s.push({elem:this,handlers:n.slice(u)}),s},fix:function(e){if(e[b.expando])return e;var t,n,r,i=e.type,a=e,s=this.fixHooks[i];s||(this.fixHooks[i]=s=tt.test(i)?this.mouseHooks:et.test(i)?this.keyHooks:{}),r=s.props?this.props.concat(s.props):this.props,e=new b.Event(a),t=r.length;while(t--)n=r[t],e[n]=a[n];return e.target||(e.target=a.srcElement||o),3===e.target.nodeType&&(e.target=e.target.parentNode),e.metaKey=!!e.metaKey,s.filter?s.filter(e,a):e},props:"altKey bubbles cancelable ctrlKey currentTarget eventPhase metaKey relatedTarget shiftKey target timeStamp view which".split(" "),fixHooks:{},keyHooks:{props:"char charCode key keyCode".split(" "),filter:function(e,t){return null==e.which&&(e.which=null!=t.charCode?t.charCode:t.keyCode),e}},mouseHooks:{props:"button buttons clientX clientY fromElement offsetX offsetY pageX pageY screenX screenY toElement".split(" "),filter:function(e,n){var r,i,a,s=n.button,u=n.fromElement;return null==e.pageX&&null!=n.clientX&&(i=e.target.ownerDocument||o,a=i.documentElement,r=i.body,e.pageX=n.clientX+(a&&a.scrollLeft||r&&r.scrollLeft||0)-(a&&a.clientLeft||r&&r.clientLeft||0),e.pageY=n.clientY+(a&&a.scrollTop||r&&r.scrollTop||0)-(a&&a.clientTop||r&&r.clientTop||0)),!e.relatedTarget&&u&&(e.relatedTarget=u===e.target?n.toElement:u),e.which||s===t||(e.which=1&s?1:2&s?3:4&s?2:0),e}},special:{load:{noBubble:!0},click:{trigger:function(){return b.nodeName(this,"input")&&"checkbox"===this.type&&this.click?(this.click(),!1):t}},focus:{trigger:function(){if(this!==o.activeElement&&this.focus)try{return this.focus(),!1}catch(e){}},delegateType:"focusin"},blur:{trigger:function(){return this===o.activeElement&&this.blur?(this.blur(),!1):t},delegateType:"focusout"},beforeunload:{postDispatch:function(e){e.result!==t&&(e.originalEvent.returnValue=e.result)}}},simulate:function(e,t,n,r){var i=b.extend(new b.Event,n,{type:e,isSimulated:!0,originalEvent:{}});r?b.event.trigger(i,null,t):b.event.dispatch.call(t,i),i.isDefaultPrevented()&&n.preventDefault()}},b.removeEvent=o.removeEventListener?function(e,t,n){e.removeEventListener&&e.removeEventListener(t,n,!1)}:function(e,t,n){var r="on"+t;e.detachEvent&&(typeof e[r]===i&&(e[r]=null),e.detachEvent(r,n))},b.Event=function(e,n){return this instanceof b.Event?(e&&e.type?(this.originalEvent=e,this.type=e.type,this.isDefaultPrevented=e.defaultPrevented||e.returnValue===!1||e.getPreventDefault&&e.getPreventDefault()?it:ot):this.type=e,n&&b.extend(this,n),this.timeStamp=e&&e.timeStamp||b.now(),this[b.expando]=!0,t):new b.Event(e,n)},b.Event.prototype={isDefaultPrevented:ot,isPropagationStopped:ot,isImmediatePropagationStopped:ot,preventDefault:function(){var e=this.originalEvent;this.isDefaultPrevented=it,e&&(e.preventDefault?e.preventDefault():e.returnValue=!1)},stopPropagation:function(){var e=this.originalEvent;this.isPropagationStopped=it,e&&(e.stopPropagation&&e.stopPropagation(),e.cancelBubble=!0)},stopImmediatePropagation:function(){this.isImmediatePropagationStopped=it,this.stopPropagation()}},b.each({mouseenter:"mouseover",mouseleave:"mouseout"},function(e,t){b.event.special[e]={delegateType:t,bindType:t,handle:function(e){var n,r=this,i=e.relatedTarget,o=e.handleObj; +return(!i||i!==r&&!b.contains(r,i))&&(e.type=o.origType,n=o.handler.apply(this,arguments),e.type=t),n}}}),b.support.submitBubbles||(b.event.special.submit={setup:function(){return b.nodeName(this,"form")?!1:(b.event.add(this,"click._submit keypress._submit",function(e){var n=e.target,r=b.nodeName(n,"input")||b.nodeName(n,"button")?n.form:t;r&&!b._data(r,"submitBubbles")&&(b.event.add(r,"submit._submit",function(e){e._submit_bubble=!0}),b._data(r,"submitBubbles",!0))}),t)},postDispatch:function(e){e._submit_bubble&&(delete e._submit_bubble,this.parentNode&&!e.isTrigger&&b.event.simulate("submit",this.parentNode,e,!0))},teardown:function(){return b.nodeName(this,"form")?!1:(b.event.remove(this,"._submit"),t)}}),b.support.changeBubbles||(b.event.special.change={setup:function(){return Z.test(this.nodeName)?(("checkbox"===this.type||"radio"===this.type)&&(b.event.add(this,"propertychange._change",function(e){"checked"===e.originalEvent.propertyName&&(this._just_changed=!0)}),b.event.add(this,"click._change",function(e){this._just_changed&&!e.isTrigger&&(this._just_changed=!1),b.event.simulate("change",this,e,!0)})),!1):(b.event.add(this,"beforeactivate._change",function(e){var t=e.target;Z.test(t.nodeName)&&!b._data(t,"changeBubbles")&&(b.event.add(t,"change._change",function(e){!this.parentNode||e.isSimulated||e.isTrigger||b.event.simulate("change",this.parentNode,e,!0)}),b._data(t,"changeBubbles",!0))}),t)},handle:function(e){var n=e.target;return this!==n||e.isSimulated||e.isTrigger||"radio"!==n.type&&"checkbox"!==n.type?e.handleObj.handler.apply(this,arguments):t},teardown:function(){return b.event.remove(this,"._change"),!Z.test(this.nodeName)}}),b.support.focusinBubbles||b.each({focus:"focusin",blur:"focusout"},function(e,t){var n=0,r=function(e){b.event.simulate(t,e.target,b.event.fix(e),!0)};b.event.special[t]={setup:function(){0===n++&&o.addEventListener(e,r,!0)},teardown:function(){0===--n&&o.removeEventListener(e,r,!0)}}}),b.fn.extend({on:function(e,n,r,i,o){var a,s;if("object"==typeof e){"string"!=typeof n&&(r=r||n,n=t);for(a in e)this.on(a,n,r,e[a],o);return this}if(null==r&&null==i?(i=n,r=n=t):null==i&&("string"==typeof n?(i=r,r=t):(i=r,r=n,n=t)),i===!1)i=ot;else if(!i)return this;return 1===o&&(s=i,i=function(e){return b().off(e),s.apply(this,arguments)},i.guid=s.guid||(s.guid=b.guid++)),this.each(function(){b.event.add(this,e,i,r,n)})},one:function(e,t,n,r){return this.on(e,t,n,r,1)},off:function(e,n,r){var i,o;if(e&&e.preventDefault&&e.handleObj)return i=e.handleObj,b(e.delegateTarget).off(i.namespace?i.origType+"."+i.namespace:i.origType,i.selector,i.handler),this;if("object"==typeof e){for(o in e)this.off(o,n,e[o]);return this}return(n===!1||"function"==typeof n)&&(r=n,n=t),r===!1&&(r=ot),this.each(function(){b.event.remove(this,e,r,n)})},bind:function(e,t,n){return this.on(e,null,t,n)},unbind:function(e,t){return this.off(e,null,t)},delegate:function(e,t,n,r){return this.on(t,e,n,r)},undelegate:function(e,t,n){return 1===arguments.length?this.off(e,"**"):this.off(t,e||"**",n)},trigger:function(e,t){return this.each(function(){b.event.trigger(e,t,this)})},triggerHandler:function(e,n){var r=this[0];return r?b.event.trigger(e,n,r,!0):t}}),function(e,t){var n,r,i,o,a,s,u,l,c,p,f,d,h,g,m,y,v,x="sizzle"+-new Date,w=e.document,T={},N=0,C=0,k=it(),E=it(),S=it(),A=typeof t,j=1<<31,D=[],L=D.pop,H=D.push,q=D.slice,M=D.indexOf||function(e){var t=0,n=this.length;for(;n>t;t++)if(this[t]===e)return t;return-1},_="[\\x20\\t\\r\\n\\f]",F="(?:\\\\.|[\\w-]|[^\\x00-\\xa0])+",O=F.replace("w","w#"),B="([*^$|!~]?=)",P="\\["+_+"*("+F+")"+_+"*(?:"+B+_+"*(?:(['\"])((?:\\\\.|[^\\\\])*?)\\3|("+O+")|)|)"+_+"*\\]",R=":("+F+")(?:\\(((['\"])((?:\\\\.|[^\\\\])*?)\\3|((?:\\\\.|[^\\\\()[\\]]|"+P.replace(3,8)+")*)|.*)\\)|)",W=RegExp("^"+_+"+|((?:^|[^\\\\])(?:\\\\.)*)"+_+"+$","g"),$=RegExp("^"+_+"*,"+_+"*"),I=RegExp("^"+_+"*([\\x20\\t\\r\\n\\f>+~])"+_+"*"),z=RegExp(R),X=RegExp("^"+O+"$"),U={ID:RegExp("^#("+F+")"),CLASS:RegExp("^\\.("+F+")"),NAME:RegExp("^\\[name=['\"]?("+F+")['\"]?\\]"),TAG:RegExp("^("+F.replace("w","w*")+")"),ATTR:RegExp("^"+P),PSEUDO:RegExp("^"+R),CHILD:RegExp("^:(only|first|last|nth|nth-last)-(child|of-type)(?:\\("+_+"*(even|odd|(([+-]|)(\\d*)n|)"+_+"*(?:([+-]|)"+_+"*(\\d+)|))"+_+"*\\)|)","i"),needsContext:RegExp("^"+_+"*[>+~]|:(even|odd|eq|gt|lt|nth|first|last)(?:\\("+_+"*((?:-\\d)?\\d*)"+_+"*\\)|)(?=[^-]|$)","i")},V=/[\x20\t\r\n\f]*[+~]/,Y=/^[^{]+\{\s*\[native code/,J=/^(?:#([\w-]+)|(\w+)|\.([\w-]+))$/,G=/^(?:input|select|textarea|button)$/i,Q=/^h\d$/i,K=/'|\\/g,Z=/\=[\x20\t\r\n\f]*([^'"\]]*)[\x20\t\r\n\f]*\]/g,et=/\\([\da-fA-F]{1,6}[\x20\t\r\n\f]?|.)/g,tt=function(e,t){var n="0x"+t-65536;return n!==n?t:0>n?String.fromCharCode(n+65536):String.fromCharCode(55296|n>>10,56320|1023&n)};try{q.call(w.documentElement.childNodes,0)[0].nodeType}catch(nt){q=function(e){var t,n=[];while(t=this[e++])n.push(t);return n}}function rt(e){return Y.test(e+"")}function it(){var e,t=[];return e=function(n,r){return t.push(n+=" ")>i.cacheLength&&delete e[t.shift()],e[n]=r}}function ot(e){return e[x]=!0,e}function at(e){var t=p.createElement("div");try{return e(t)}catch(n){return!1}finally{t=null}}function st(e,t,n,r){var i,o,a,s,u,l,f,g,m,v;if((t?t.ownerDocument||t:w)!==p&&c(t),t=t||p,n=n||[],!e||"string"!=typeof e)return n;if(1!==(s=t.nodeType)&&9!==s)return[];if(!d&&!r){if(i=J.exec(e))if(a=i[1]){if(9===s){if(o=t.getElementById(a),!o||!o.parentNode)return n;if(o.id===a)return n.push(o),n}else if(t.ownerDocument&&(o=t.ownerDocument.getElementById(a))&&y(t,o)&&o.id===a)return n.push(o),n}else{if(i[2])return H.apply(n,q.call(t.getElementsByTagName(e),0)),n;if((a=i[3])&&T.getByClassName&&t.getElementsByClassName)return H.apply(n,q.call(t.getElementsByClassName(a),0)),n}if(T.qsa&&!h.test(e)){if(f=!0,g=x,m=t,v=9===s&&e,1===s&&"object"!==t.nodeName.toLowerCase()){l=ft(e),(f=t.getAttribute("id"))?g=f.replace(K,"\\$&"):t.setAttribute("id",g),g="[id='"+g+"'] ",u=l.length;while(u--)l[u]=g+dt(l[u]);m=V.test(e)&&t.parentNode||t,v=l.join(",")}if(v)try{return H.apply(n,q.call(m.querySelectorAll(v),0)),n}catch(b){}finally{f||t.removeAttribute("id")}}}return wt(e.replace(W,"$1"),t,n,r)}a=st.isXML=function(e){var t=e&&(e.ownerDocument||e).documentElement;return t?"HTML"!==t.nodeName:!1},c=st.setDocument=function(e){var n=e?e.ownerDocument||e:w;return n!==p&&9===n.nodeType&&n.documentElement?(p=n,f=n.documentElement,d=a(n),T.tagNameNoComments=at(function(e){return e.appendChild(n.createComment("")),!e.getElementsByTagName("*").length}),T.attributes=at(function(e){e.innerHTML="";var t=typeof e.lastChild.getAttribute("multiple");return"boolean"!==t&&"string"!==t}),T.getByClassName=at(function(e){return e.innerHTML="",e.getElementsByClassName&&e.getElementsByClassName("e").length?(e.lastChild.className="e",2===e.getElementsByClassName("e").length):!1}),T.getByName=at(function(e){e.id=x+0,e.innerHTML="
    ",f.insertBefore(e,f.firstChild);var t=n.getElementsByName&&n.getElementsByName(x).length===2+n.getElementsByName(x+0).length;return T.getIdNotName=!n.getElementById(x),f.removeChild(e),t}),i.attrHandle=at(function(e){return e.innerHTML="",e.firstChild&&typeof e.firstChild.getAttribute!==A&&"#"===e.firstChild.getAttribute("href")})?{}:{href:function(e){return e.getAttribute("href",2)},type:function(e){return e.getAttribute("type")}},T.getIdNotName?(i.find.ID=function(e,t){if(typeof t.getElementById!==A&&!d){var n=t.getElementById(e);return n&&n.parentNode?[n]:[]}},i.filter.ID=function(e){var t=e.replace(et,tt);return function(e){return e.getAttribute("id")===t}}):(i.find.ID=function(e,n){if(typeof n.getElementById!==A&&!d){var r=n.getElementById(e);return r?r.id===e||typeof r.getAttributeNode!==A&&r.getAttributeNode("id").value===e?[r]:t:[]}},i.filter.ID=function(e){var t=e.replace(et,tt);return function(e){var n=typeof e.getAttributeNode!==A&&e.getAttributeNode("id");return n&&n.value===t}}),i.find.TAG=T.tagNameNoComments?function(e,n){return typeof n.getElementsByTagName!==A?n.getElementsByTagName(e):t}:function(e,t){var n,r=[],i=0,o=t.getElementsByTagName(e);if("*"===e){while(n=o[i++])1===n.nodeType&&r.push(n);return r}return o},i.find.NAME=T.getByName&&function(e,n){return typeof n.getElementsByName!==A?n.getElementsByName(name):t},i.find.CLASS=T.getByClassName&&function(e,n){return typeof n.getElementsByClassName===A||d?t:n.getElementsByClassName(e)},g=[],h=[":focus"],(T.qsa=rt(n.querySelectorAll))&&(at(function(e){e.innerHTML="",e.querySelectorAll("[selected]").length||h.push("\\["+_+"*(?:checked|disabled|ismap|multiple|readonly|selected|value)"),e.querySelectorAll(":checked").length||h.push(":checked")}),at(function(e){e.innerHTML="",e.querySelectorAll("[i^='']").length&&h.push("[*^$]="+_+"*(?:\"\"|'')"),e.querySelectorAll(":enabled").length||h.push(":enabled",":disabled"),e.querySelectorAll("*,:x"),h.push(",.*:")})),(T.matchesSelector=rt(m=f.matchesSelector||f.mozMatchesSelector||f.webkitMatchesSelector||f.oMatchesSelector||f.msMatchesSelector))&&at(function(e){T.disconnectedMatch=m.call(e,"div"),m.call(e,"[s!='']:x"),g.push("!=",R)}),h=RegExp(h.join("|")),g=RegExp(g.join("|")),y=rt(f.contains)||f.compareDocumentPosition?function(e,t){var n=9===e.nodeType?e.documentElement:e,r=t&&t.parentNode;return e===r||!(!r||1!==r.nodeType||!(n.contains?n.contains(r):e.compareDocumentPosition&&16&e.compareDocumentPosition(r)))}:function(e,t){if(t)while(t=t.parentNode)if(t===e)return!0;return!1},v=f.compareDocumentPosition?function(e,t){var r;return e===t?(u=!0,0):(r=t.compareDocumentPosition&&e.compareDocumentPosition&&e.compareDocumentPosition(t))?1&r||e.parentNode&&11===e.parentNode.nodeType?e===n||y(w,e)?-1:t===n||y(w,t)?1:0:4&r?-1:1:e.compareDocumentPosition?-1:1}:function(e,t){var r,i=0,o=e.parentNode,a=t.parentNode,s=[e],l=[t];if(e===t)return u=!0,0;if(!o||!a)return e===n?-1:t===n?1:o?-1:a?1:0;if(o===a)return ut(e,t);r=e;while(r=r.parentNode)s.unshift(r);r=t;while(r=r.parentNode)l.unshift(r);while(s[i]===l[i])i++;return i?ut(s[i],l[i]):s[i]===w?-1:l[i]===w?1:0},u=!1,[0,0].sort(v),T.detectDuplicates=u,p):p},st.matches=function(e,t){return st(e,null,null,t)},st.matchesSelector=function(e,t){if((e.ownerDocument||e)!==p&&c(e),t=t.replace(Z,"='$1']"),!(!T.matchesSelector||d||g&&g.test(t)||h.test(t)))try{var n=m.call(e,t);if(n||T.disconnectedMatch||e.document&&11!==e.document.nodeType)return n}catch(r){}return st(t,p,null,[e]).length>0},st.contains=function(e,t){return(e.ownerDocument||e)!==p&&c(e),y(e,t)},st.attr=function(e,t){var n;return(e.ownerDocument||e)!==p&&c(e),d||(t=t.toLowerCase()),(n=i.attrHandle[t])?n(e):d||T.attributes?e.getAttribute(t):((n=e.getAttributeNode(t))||e.getAttribute(t))&&e[t]===!0?t:n&&n.specified?n.value:null},st.error=function(e){throw Error("Syntax error, unrecognized expression: "+e)},st.uniqueSort=function(e){var t,n=[],r=1,i=0;if(u=!T.detectDuplicates,e.sort(v),u){for(;t=e[r];r++)t===e[r-1]&&(i=n.push(r));while(i--)e.splice(n[i],1)}return e};function ut(e,t){var n=t&&e,r=n&&(~t.sourceIndex||j)-(~e.sourceIndex||j);if(r)return r;if(n)while(n=n.nextSibling)if(n===t)return-1;return e?1:-1}function lt(e){return function(t){var n=t.nodeName.toLowerCase();return"input"===n&&t.type===e}}function ct(e){return function(t){var n=t.nodeName.toLowerCase();return("input"===n||"button"===n)&&t.type===e}}function pt(e){return ot(function(t){return t=+t,ot(function(n,r){var i,o=e([],n.length,t),a=o.length;while(a--)n[i=o[a]]&&(n[i]=!(r[i]=n[i]))})})}o=st.getText=function(e){var t,n="",r=0,i=e.nodeType;if(i){if(1===i||9===i||11===i){if("string"==typeof e.textContent)return e.textContent;for(e=e.firstChild;e;e=e.nextSibling)n+=o(e)}else if(3===i||4===i)return e.nodeValue}else for(;t=e[r];r++)n+=o(t);return n},i=st.selectors={cacheLength:50,createPseudo:ot,match:U,find:{},relative:{">":{dir:"parentNode",first:!0}," ":{dir:"parentNode"},"+":{dir:"previousSibling",first:!0},"~":{dir:"previousSibling"}},preFilter:{ATTR:function(e){return e[1]=e[1].replace(et,tt),e[3]=(e[4]||e[5]||"").replace(et,tt),"~="===e[2]&&(e[3]=" "+e[3]+" "),e.slice(0,4)},CHILD:function(e){return e[1]=e[1].toLowerCase(),"nth"===e[1].slice(0,3)?(e[3]||st.error(e[0]),e[4]=+(e[4]?e[5]+(e[6]||1):2*("even"===e[3]||"odd"===e[3])),e[5]=+(e[7]+e[8]||"odd"===e[3])):e[3]&&st.error(e[0]),e},PSEUDO:function(e){var t,n=!e[5]&&e[2];return U.CHILD.test(e[0])?null:(e[4]?e[2]=e[4]:n&&z.test(n)&&(t=ft(n,!0))&&(t=n.indexOf(")",n.length-t)-n.length)&&(e[0]=e[0].slice(0,t),e[2]=n.slice(0,t)),e.slice(0,3))}},filter:{TAG:function(e){return"*"===e?function(){return!0}:(e=e.replace(et,tt).toLowerCase(),function(t){return t.nodeName&&t.nodeName.toLowerCase()===e})},CLASS:function(e){var t=k[e+" "];return t||(t=RegExp("(^|"+_+")"+e+"("+_+"|$)"))&&k(e,function(e){return t.test(e.className||typeof e.getAttribute!==A&&e.getAttribute("class")||"")})},ATTR:function(e,t,n){return function(r){var i=st.attr(r,e);return null==i?"!="===t:t?(i+="","="===t?i===n:"!="===t?i!==n:"^="===t?n&&0===i.indexOf(n):"*="===t?n&&i.indexOf(n)>-1:"$="===t?n&&i.slice(-n.length)===n:"~="===t?(" "+i+" ").indexOf(n)>-1:"|="===t?i===n||i.slice(0,n.length+1)===n+"-":!1):!0}},CHILD:function(e,t,n,r,i){var o="nth"!==e.slice(0,3),a="last"!==e.slice(-4),s="of-type"===t;return 1===r&&0===i?function(e){return!!e.parentNode}:function(t,n,u){var l,c,p,f,d,h,g=o!==a?"nextSibling":"previousSibling",m=t.parentNode,y=s&&t.nodeName.toLowerCase(),v=!u&&!s;if(m){if(o){while(g){p=t;while(p=p[g])if(s?p.nodeName.toLowerCase()===y:1===p.nodeType)return!1;h=g="only"===e&&!h&&"nextSibling"}return!0}if(h=[a?m.firstChild:m.lastChild],a&&v){c=m[x]||(m[x]={}),l=c[e]||[],d=l[0]===N&&l[1],f=l[0]===N&&l[2],p=d&&m.childNodes[d];while(p=++d&&p&&p[g]||(f=d=0)||h.pop())if(1===p.nodeType&&++f&&p===t){c[e]=[N,d,f];break}}else if(v&&(l=(t[x]||(t[x]={}))[e])&&l[0]===N)f=l[1];else while(p=++d&&p&&p[g]||(f=d=0)||h.pop())if((s?p.nodeName.toLowerCase()===y:1===p.nodeType)&&++f&&(v&&((p[x]||(p[x]={}))[e]=[N,f]),p===t))break;return f-=i,f===r||0===f%r&&f/r>=0}}},PSEUDO:function(e,t){var n,r=i.pseudos[e]||i.setFilters[e.toLowerCase()]||st.error("unsupported pseudo: "+e);return r[x]?r(t):r.length>1?(n=[e,e,"",t],i.setFilters.hasOwnProperty(e.toLowerCase())?ot(function(e,n){var i,o=r(e,t),a=o.length;while(a--)i=M.call(e,o[a]),e[i]=!(n[i]=o[a])}):function(e){return r(e,0,n)}):r}},pseudos:{not:ot(function(e){var t=[],n=[],r=s(e.replace(W,"$1"));return r[x]?ot(function(e,t,n,i){var o,a=r(e,null,i,[]),s=e.length;while(s--)(o=a[s])&&(e[s]=!(t[s]=o))}):function(e,i,o){return t[0]=e,r(t,null,o,n),!n.pop()}}),has:ot(function(e){return function(t){return st(e,t).length>0}}),contains:ot(function(e){return function(t){return(t.textContent||t.innerText||o(t)).indexOf(e)>-1}}),lang:ot(function(e){return X.test(e||"")||st.error("unsupported lang: "+e),e=e.replace(et,tt).toLowerCase(),function(t){var n;do if(n=d?t.getAttribute("xml:lang")||t.getAttribute("lang"):t.lang)return n=n.toLowerCase(),n===e||0===n.indexOf(e+"-");while((t=t.parentNode)&&1===t.nodeType);return!1}}),target:function(t){var n=e.location&&e.location.hash;return n&&n.slice(1)===t.id},root:function(e){return e===f},focus:function(e){return e===p.activeElement&&(!p.hasFocus||p.hasFocus())&&!!(e.type||e.href||~e.tabIndex)},enabled:function(e){return e.disabled===!1},disabled:function(e){return e.disabled===!0},checked:function(e){var t=e.nodeName.toLowerCase();return"input"===t&&!!e.checked||"option"===t&&!!e.selected},selected:function(e){return e.parentNode&&e.parentNode.selectedIndex,e.selected===!0},empty:function(e){for(e=e.firstChild;e;e=e.nextSibling)if(e.nodeName>"@"||3===e.nodeType||4===e.nodeType)return!1;return!0},parent:function(e){return!i.pseudos.empty(e)},header:function(e){return Q.test(e.nodeName)},input:function(e){return G.test(e.nodeName)},button:function(e){var t=e.nodeName.toLowerCase();return"input"===t&&"button"===e.type||"button"===t},text:function(e){var t;return"input"===e.nodeName.toLowerCase()&&"text"===e.type&&(null==(t=e.getAttribute("type"))||t.toLowerCase()===e.type)},first:pt(function(){return[0]}),last:pt(function(e,t){return[t-1]}),eq:pt(function(e,t,n){return[0>n?n+t:n]}),even:pt(function(e,t){var n=0;for(;t>n;n+=2)e.push(n);return e}),odd:pt(function(e,t){var n=1;for(;t>n;n+=2)e.push(n);return e}),lt:pt(function(e,t,n){var r=0>n?n+t:n;for(;--r>=0;)e.push(r);return e}),gt:pt(function(e,t,n){var r=0>n?n+t:n;for(;t>++r;)e.push(r);return e})}};for(n in{radio:!0,checkbox:!0,file:!0,password:!0,image:!0})i.pseudos[n]=lt(n);for(n in{submit:!0,reset:!0})i.pseudos[n]=ct(n);function ft(e,t){var n,r,o,a,s,u,l,c=E[e+" "];if(c)return t?0:c.slice(0);s=e,u=[],l=i.preFilter;while(s){(!n||(r=$.exec(s)))&&(r&&(s=s.slice(r[0].length)||s),u.push(o=[])),n=!1,(r=I.exec(s))&&(n=r.shift(),o.push({value:n,type:r[0].replace(W," ")}),s=s.slice(n.length));for(a in i.filter)!(r=U[a].exec(s))||l[a]&&!(r=l[a](r))||(n=r.shift(),o.push({value:n,type:a,matches:r}),s=s.slice(n.length));if(!n)break}return t?s.length:s?st.error(e):E(e,u).slice(0)}function dt(e){var t=0,n=e.length,r="";for(;n>t;t++)r+=e[t].value;return r}function ht(e,t,n){var i=t.dir,o=n&&"parentNode"===i,a=C++;return t.first?function(t,n,r){while(t=t[i])if(1===t.nodeType||o)return e(t,n,r)}:function(t,n,s){var u,l,c,p=N+" "+a;if(s){while(t=t[i])if((1===t.nodeType||o)&&e(t,n,s))return!0}else while(t=t[i])if(1===t.nodeType||o)if(c=t[x]||(t[x]={}),(l=c[i])&&l[0]===p){if((u=l[1])===!0||u===r)return u===!0}else if(l=c[i]=[p],l[1]=e(t,n,s)||r,l[1]===!0)return!0}}function gt(e){return e.length>1?function(t,n,r){var i=e.length;while(i--)if(!e[i](t,n,r))return!1;return!0}:e[0]}function mt(e,t,n,r,i){var o,a=[],s=0,u=e.length,l=null!=t;for(;u>s;s++)(o=e[s])&&(!n||n(o,r,i))&&(a.push(o),l&&t.push(s));return a}function yt(e,t,n,r,i,o){return r&&!r[x]&&(r=yt(r)),i&&!i[x]&&(i=yt(i,o)),ot(function(o,a,s,u){var l,c,p,f=[],d=[],h=a.length,g=o||xt(t||"*",s.nodeType?[s]:s,[]),m=!e||!o&&t?g:mt(g,f,e,s,u),y=n?i||(o?e:h||r)?[]:a:m;if(n&&n(m,y,s,u),r){l=mt(y,d),r(l,[],s,u),c=l.length;while(c--)(p=l[c])&&(y[d[c]]=!(m[d[c]]=p))}if(o){if(i||e){if(i){l=[],c=y.length;while(c--)(p=y[c])&&l.push(m[c]=p);i(null,y=[],l,u)}c=y.length;while(c--)(p=y[c])&&(l=i?M.call(o,p):f[c])>-1&&(o[l]=!(a[l]=p))}}else y=mt(y===a?y.splice(h,y.length):y),i?i(null,a,y,u):H.apply(a,y)})}function vt(e){var t,n,r,o=e.length,a=i.relative[e[0].type],s=a||i.relative[" "],u=a?1:0,c=ht(function(e){return e===t},s,!0),p=ht(function(e){return M.call(t,e)>-1},s,!0),f=[function(e,n,r){return!a&&(r||n!==l)||((t=n).nodeType?c(e,n,r):p(e,n,r))}];for(;o>u;u++)if(n=i.relative[e[u].type])f=[ht(gt(f),n)];else{if(n=i.filter[e[u].type].apply(null,e[u].matches),n[x]){for(r=++u;o>r;r++)if(i.relative[e[r].type])break;return yt(u>1&>(f),u>1&&dt(e.slice(0,u-1)).replace(W,"$1"),n,r>u&&vt(e.slice(u,r)),o>r&&vt(e=e.slice(r)),o>r&&dt(e))}f.push(n)}return gt(f)}function bt(e,t){var n=0,o=t.length>0,a=e.length>0,s=function(s,u,c,f,d){var h,g,m,y=[],v=0,b="0",x=s&&[],w=null!=d,T=l,C=s||a&&i.find.TAG("*",d&&u.parentNode||u),k=N+=null==T?1:Math.random()||.1;for(w&&(l=u!==p&&u,r=n);null!=(h=C[b]);b++){if(a&&h){g=0;while(m=e[g++])if(m(h,u,c)){f.push(h);break}w&&(N=k,r=++n)}o&&((h=!m&&h)&&v--,s&&x.push(h))}if(v+=b,o&&b!==v){g=0;while(m=t[g++])m(x,y,u,c);if(s){if(v>0)while(b--)x[b]||y[b]||(y[b]=L.call(f));y=mt(y)}H.apply(f,y),w&&!s&&y.length>0&&v+t.length>1&&st.uniqueSort(f)}return w&&(N=k,l=T),x};return o?ot(s):s}s=st.compile=function(e,t){var n,r=[],i=[],o=S[e+" "];if(!o){t||(t=ft(e)),n=t.length;while(n--)o=vt(t[n]),o[x]?r.push(o):i.push(o);o=S(e,bt(i,r))}return o};function xt(e,t,n){var r=0,i=t.length;for(;i>r;r++)st(e,t[r],n);return n}function wt(e,t,n,r){var o,a,u,l,c,p=ft(e);if(!r&&1===p.length){if(a=p[0]=p[0].slice(0),a.length>2&&"ID"===(u=a[0]).type&&9===t.nodeType&&!d&&i.relative[a[1].type]){if(t=i.find.ID(u.matches[0].replace(et,tt),t)[0],!t)return n;e=e.slice(a.shift().value.length)}o=U.needsContext.test(e)?0:a.length;while(o--){if(u=a[o],i.relative[l=u.type])break;if((c=i.find[l])&&(r=c(u.matches[0].replace(et,tt),V.test(a[0].type)&&t.parentNode||t))){if(a.splice(o,1),e=r.length&&dt(a),!e)return H.apply(n,q.call(r,0)),n;break}}}return s(e,p)(r,t,d,n,V.test(e)),n}i.pseudos.nth=i.pseudos.eq;function Tt(){}i.filters=Tt.prototype=i.pseudos,i.setFilters=new Tt,c(),st.attr=b.attr,b.find=st,b.expr=st.selectors,b.expr[":"]=b.expr.pseudos,b.unique=st.uniqueSort,b.text=st.getText,b.isXMLDoc=st.isXML,b.contains=st.contains}(e);var at=/Until$/,st=/^(?:parents|prev(?:Until|All))/,ut=/^.[^:#\[\.,]*$/,lt=b.expr.match.needsContext,ct={children:!0,contents:!0,next:!0,prev:!0};b.fn.extend({find:function(e){var t,n,r,i=this.length;if("string"!=typeof e)return r=this,this.pushStack(b(e).filter(function(){for(t=0;i>t;t++)if(b.contains(r[t],this))return!0}));for(n=[],t=0;i>t;t++)b.find(e,this[t],n);return n=this.pushStack(i>1?b.unique(n):n),n.selector=(this.selector?this.selector+" ":"")+e,n},has:function(e){var t,n=b(e,this),r=n.length;return this.filter(function(){for(t=0;r>t;t++)if(b.contains(this,n[t]))return!0})},not:function(e){return this.pushStack(ft(this,e,!1))},filter:function(e){return this.pushStack(ft(this,e,!0))},is:function(e){return!!e&&("string"==typeof e?lt.test(e)?b(e,this.context).index(this[0])>=0:b.filter(e,this).length>0:this.filter(e).length>0)},closest:function(e,t){var n,r=0,i=this.length,o=[],a=lt.test(e)||"string"!=typeof e?b(e,t||this.context):0;for(;i>r;r++){n=this[r];while(n&&n.ownerDocument&&n!==t&&11!==n.nodeType){if(a?a.index(n)>-1:b.find.matchesSelector(n,e)){o.push(n);break}n=n.parentNode}}return this.pushStack(o.length>1?b.unique(o):o)},index:function(e){return e?"string"==typeof e?b.inArray(this[0],b(e)):b.inArray(e.jquery?e[0]:e,this):this[0]&&this[0].parentNode?this.first().prevAll().length:-1},add:function(e,t){var n="string"==typeof e?b(e,t):b.makeArray(e&&e.nodeType?[e]:e),r=b.merge(this.get(),n);return this.pushStack(b.unique(r))},addBack:function(e){return this.add(null==e?this.prevObject:this.prevObject.filter(e))}}),b.fn.andSelf=b.fn.addBack;function pt(e,t){do e=e[t];while(e&&1!==e.nodeType);return e}b.each({parent:function(e){var t=e.parentNode;return t&&11!==t.nodeType?t:null},parents:function(e){return b.dir(e,"parentNode")},parentsUntil:function(e,t,n){return b.dir(e,"parentNode",n)},next:function(e){return pt(e,"nextSibling")},prev:function(e){return pt(e,"previousSibling")},nextAll:function(e){return b.dir(e,"nextSibling")},prevAll:function(e){return b.dir(e,"previousSibling")},nextUntil:function(e,t,n){return b.dir(e,"nextSibling",n)},prevUntil:function(e,t,n){return b.dir(e,"previousSibling",n)},siblings:function(e){return b.sibling((e.parentNode||{}).firstChild,e)},children:function(e){return b.sibling(e.firstChild)},contents:function(e){return b.nodeName(e,"iframe")?e.contentDocument||e.contentWindow.document:b.merge([],e.childNodes)}},function(e,t){b.fn[e]=function(n,r){var i=b.map(this,t,n);return at.test(e)||(r=n),r&&"string"==typeof r&&(i=b.filter(r,i)),i=this.length>1&&!ct[e]?b.unique(i):i,this.length>1&&st.test(e)&&(i=i.reverse()),this.pushStack(i)}}),b.extend({filter:function(e,t,n){return n&&(e=":not("+e+")"),1===t.length?b.find.matchesSelector(t[0],e)?[t[0]]:[]:b.find.matches(e,t)},dir:function(e,n,r){var i=[],o=e[n];while(o&&9!==o.nodeType&&(r===t||1!==o.nodeType||!b(o).is(r)))1===o.nodeType&&i.push(o),o=o[n];return i},sibling:function(e,t){var n=[];for(;e;e=e.nextSibling)1===e.nodeType&&e!==t&&n.push(e);return n}});function ft(e,t,n){if(t=t||0,b.isFunction(t))return b.grep(e,function(e,r){var i=!!t.call(e,r,e);return i===n});if(t.nodeType)return b.grep(e,function(e){return e===t===n});if("string"==typeof t){var r=b.grep(e,function(e){return 1===e.nodeType});if(ut.test(t))return b.filter(t,r,!n);t=b.filter(t,r)}return b.grep(e,function(e){return b.inArray(e,t)>=0===n})}function dt(e){var t=ht.split("|"),n=e.createDocumentFragment();if(n.createElement)while(t.length)n.createElement(t.pop());return n}var ht="abbr|article|aside|audio|bdi|canvas|data|datalist|details|figcaption|figure|footer|header|hgroup|mark|meter|nav|output|progress|section|summary|time|video",gt=/ jQuery\d+="(?:null|\d+)"/g,mt=RegExp("<(?:"+ht+")[\\s/>]","i"),yt=/^\s+/,vt=/<(?!area|br|col|embed|hr|img|input|link|meta|param)(([\w:]+)[^>]*)\/>/gi,bt=/<([\w:]+)/,xt=/\s*$/g,At={option:[1,""],legend:[1,"
    ","
    "],area:[1,"",""],param:[1,"",""],thead:[1,"","
    "],tr:[2,"","
    "],col:[2,"","
    "],td:[3,"","
    "],_default:b.support.htmlSerialize?[0,"",""]:[1,"X
    ","
    "]},jt=dt(o),Dt=jt.appendChild(o.createElement("div"));At.optgroup=At.option,At.tbody=At.tfoot=At.colgroup=At.caption=At.thead,At.th=At.td,b.fn.extend({text:function(e){return b.access(this,function(e){return e===t?b.text(this):this.empty().append((this[0]&&this[0].ownerDocument||o).createTextNode(e))},null,e,arguments.length)},wrapAll:function(e){if(b.isFunction(e))return this.each(function(t){b(this).wrapAll(e.call(this,t))});if(this[0]){var t=b(e,this[0].ownerDocument).eq(0).clone(!0);this[0].parentNode&&t.insertBefore(this[0]),t.map(function(){var e=this;while(e.firstChild&&1===e.firstChild.nodeType)e=e.firstChild;return e}).append(this)}return this},wrapInner:function(e){return b.isFunction(e)?this.each(function(t){b(this).wrapInner(e.call(this,t))}):this.each(function(){var t=b(this),n=t.contents();n.length?n.wrapAll(e):t.append(e)})},wrap:function(e){var t=b.isFunction(e);return this.each(function(n){b(this).wrapAll(t?e.call(this,n):e)})},unwrap:function(){return this.parent().each(function(){b.nodeName(this,"body")||b(this).replaceWith(this.childNodes)}).end()},append:function(){return this.domManip(arguments,!0,function(e){(1===this.nodeType||11===this.nodeType||9===this.nodeType)&&this.appendChild(e)})},prepend:function(){return this.domManip(arguments,!0,function(e){(1===this.nodeType||11===this.nodeType||9===this.nodeType)&&this.insertBefore(e,this.firstChild)})},before:function(){return this.domManip(arguments,!1,function(e){this.parentNode&&this.parentNode.insertBefore(e,this)})},after:function(){return this.domManip(arguments,!1,function(e){this.parentNode&&this.parentNode.insertBefore(e,this.nextSibling)})},remove:function(e,t){var n,r=0;for(;null!=(n=this[r]);r++)(!e||b.filter(e,[n]).length>0)&&(t||1!==n.nodeType||b.cleanData(Ot(n)),n.parentNode&&(t&&b.contains(n.ownerDocument,n)&&Mt(Ot(n,"script")),n.parentNode.removeChild(n)));return this},empty:function(){var e,t=0;for(;null!=(e=this[t]);t++){1===e.nodeType&&b.cleanData(Ot(e,!1));while(e.firstChild)e.removeChild(e.firstChild);e.options&&b.nodeName(e,"select")&&(e.options.length=0)}return this},clone:function(e,t){return e=null==e?!1:e,t=null==t?e:t,this.map(function(){return b.clone(this,e,t)})},html:function(e){return b.access(this,function(e){var n=this[0]||{},r=0,i=this.length;if(e===t)return 1===n.nodeType?n.innerHTML.replace(gt,""):t;if(!("string"!=typeof e||Tt.test(e)||!b.support.htmlSerialize&&mt.test(e)||!b.support.leadingWhitespace&&yt.test(e)||At[(bt.exec(e)||["",""])[1].toLowerCase()])){e=e.replace(vt,"<$1>");try{for(;i>r;r++)n=this[r]||{},1===n.nodeType&&(b.cleanData(Ot(n,!1)),n.innerHTML=e);n=0}catch(o){}}n&&this.empty().append(e)},null,e,arguments.length)},replaceWith:function(e){var t=b.isFunction(e);return t||"string"==typeof e||(e=b(e).not(this).detach()),this.domManip([e],!0,function(e){var t=this.nextSibling,n=this.parentNode;n&&(b(this).remove(),n.insertBefore(e,t))})},detach:function(e){return this.remove(e,!0)},domManip:function(e,n,r){e=f.apply([],e);var i,o,a,s,u,l,c=0,p=this.length,d=this,h=p-1,g=e[0],m=b.isFunction(g);if(m||!(1>=p||"string"!=typeof g||b.support.checkClone)&&Ct.test(g))return this.each(function(i){var o=d.eq(i);m&&(e[0]=g.call(this,i,n?o.html():t)),o.domManip(e,n,r)});if(p&&(l=b.buildFragment(e,this[0].ownerDocument,!1,this),i=l.firstChild,1===l.childNodes.length&&(l=i),i)){for(n=n&&b.nodeName(i,"tr"),s=b.map(Ot(l,"script"),Ht),a=s.length;p>c;c++)o=l,c!==h&&(o=b.clone(o,!0,!0),a&&b.merge(s,Ot(o,"script"))),r.call(n&&b.nodeName(this[c],"table")?Lt(this[c],"tbody"):this[c],o,c);if(a)for(u=s[s.length-1].ownerDocument,b.map(s,qt),c=0;a>c;c++)o=s[c],kt.test(o.type||"")&&!b._data(o,"globalEval")&&b.contains(u,o)&&(o.src?b.ajax({url:o.src,type:"GET",dataType:"script",async:!1,global:!1,"throws":!0}):b.globalEval((o.text||o.textContent||o.innerHTML||"").replace(St,"")));l=i=null}return this}});function Lt(e,t){return e.getElementsByTagName(t)[0]||e.appendChild(e.ownerDocument.createElement(t))}function Ht(e){var t=e.getAttributeNode("type");return e.type=(t&&t.specified)+"/"+e.type,e}function qt(e){var t=Et.exec(e.type);return t?e.type=t[1]:e.removeAttribute("type"),e}function Mt(e,t){var n,r=0;for(;null!=(n=e[r]);r++)b._data(n,"globalEval",!t||b._data(t[r],"globalEval"))}function _t(e,t){if(1===t.nodeType&&b.hasData(e)){var n,r,i,o=b._data(e),a=b._data(t,o),s=o.events;if(s){delete a.handle,a.events={};for(n in s)for(r=0,i=s[n].length;i>r;r++)b.event.add(t,n,s[n][r])}a.data&&(a.data=b.extend({},a.data))}}function Ft(e,t){var n,r,i;if(1===t.nodeType){if(n=t.nodeName.toLowerCase(),!b.support.noCloneEvent&&t[b.expando]){i=b._data(t);for(r in i.events)b.removeEvent(t,r,i.handle);t.removeAttribute(b.expando)}"script"===n&&t.text!==e.text?(Ht(t).text=e.text,qt(t)):"object"===n?(t.parentNode&&(t.outerHTML=e.outerHTML),b.support.html5Clone&&e.innerHTML&&!b.trim(t.innerHTML)&&(t.innerHTML=e.innerHTML)):"input"===n&&Nt.test(e.type)?(t.defaultChecked=t.checked=e.checked,t.value!==e.value&&(t.value=e.value)):"option"===n?t.defaultSelected=t.selected=e.defaultSelected:("input"===n||"textarea"===n)&&(t.defaultValue=e.defaultValue)}}b.each({appendTo:"append",prependTo:"prepend",insertBefore:"before",insertAfter:"after",replaceAll:"replaceWith"},function(e,t){b.fn[e]=function(e){var n,r=0,i=[],o=b(e),a=o.length-1;for(;a>=r;r++)n=r===a?this:this.clone(!0),b(o[r])[t](n),d.apply(i,n.get());return this.pushStack(i)}});function Ot(e,n){var r,o,a=0,s=typeof e.getElementsByTagName!==i?e.getElementsByTagName(n||"*"):typeof e.querySelectorAll!==i?e.querySelectorAll(n||"*"):t;if(!s)for(s=[],r=e.childNodes||e;null!=(o=r[a]);a++)!n||b.nodeName(o,n)?s.push(o):b.merge(s,Ot(o,n));return n===t||n&&b.nodeName(e,n)?b.merge([e],s):s}function Bt(e){Nt.test(e.type)&&(e.defaultChecked=e.checked)}b.extend({clone:function(e,t,n){var r,i,o,a,s,u=b.contains(e.ownerDocument,e);if(b.support.html5Clone||b.isXMLDoc(e)||!mt.test("<"+e.nodeName+">")?o=e.cloneNode(!0):(Dt.innerHTML=e.outerHTML,Dt.removeChild(o=Dt.firstChild)),!(b.support.noCloneEvent&&b.support.noCloneChecked||1!==e.nodeType&&11!==e.nodeType||b.isXMLDoc(e)))for(r=Ot(o),s=Ot(e),a=0;null!=(i=s[a]);++a)r[a]&&Ft(i,r[a]);if(t)if(n)for(s=s||Ot(e),r=r||Ot(o),a=0;null!=(i=s[a]);a++)_t(i,r[a]);else _t(e,o);return r=Ot(o,"script"),r.length>0&&Mt(r,!u&&Ot(e,"script")),r=s=i=null,o},buildFragment:function(e,t,n,r){var i,o,a,s,u,l,c,p=e.length,f=dt(t),d=[],h=0;for(;p>h;h++)if(o=e[h],o||0===o)if("object"===b.type(o))b.merge(d,o.nodeType?[o]:o);else if(wt.test(o)){s=s||f.appendChild(t.createElement("div")),u=(bt.exec(o)||["",""])[1].toLowerCase(),c=At[u]||At._default,s.innerHTML=c[1]+o.replace(vt,"<$1>")+c[2],i=c[0];while(i--)s=s.lastChild;if(!b.support.leadingWhitespace&&yt.test(o)&&d.push(t.createTextNode(yt.exec(o)[0])),!b.support.tbody){o="table"!==u||xt.test(o)?""!==c[1]||xt.test(o)?0:s:s.firstChild,i=o&&o.childNodes.length;while(i--)b.nodeName(l=o.childNodes[i],"tbody")&&!l.childNodes.length&&o.removeChild(l) +}b.merge(d,s.childNodes),s.textContent="";while(s.firstChild)s.removeChild(s.firstChild);s=f.lastChild}else d.push(t.createTextNode(o));s&&f.removeChild(s),b.support.appendChecked||b.grep(Ot(d,"input"),Bt),h=0;while(o=d[h++])if((!r||-1===b.inArray(o,r))&&(a=b.contains(o.ownerDocument,o),s=Ot(f.appendChild(o),"script"),a&&Mt(s),n)){i=0;while(o=s[i++])kt.test(o.type||"")&&n.push(o)}return s=null,f},cleanData:function(e,t){var n,r,o,a,s=0,u=b.expando,l=b.cache,p=b.support.deleteExpando,f=b.event.special;for(;null!=(n=e[s]);s++)if((t||b.acceptData(n))&&(o=n[u],a=o&&l[o])){if(a.events)for(r in a.events)f[r]?b.event.remove(n,r):b.removeEvent(n,r,a.handle);l[o]&&(delete l[o],p?delete n[u]:typeof n.removeAttribute!==i?n.removeAttribute(u):n[u]=null,c.push(o))}}});var Pt,Rt,Wt,$t=/alpha\([^)]*\)/i,It=/opacity\s*=\s*([^)]*)/,zt=/^(top|right|bottom|left)$/,Xt=/^(none|table(?!-c[ea]).+)/,Ut=/^margin/,Vt=RegExp("^("+x+")(.*)$","i"),Yt=RegExp("^("+x+")(?!px)[a-z%]+$","i"),Jt=RegExp("^([+-])=("+x+")","i"),Gt={BODY:"block"},Qt={position:"absolute",visibility:"hidden",display:"block"},Kt={letterSpacing:0,fontWeight:400},Zt=["Top","Right","Bottom","Left"],en=["Webkit","O","Moz","ms"];function tn(e,t){if(t in e)return t;var n=t.charAt(0).toUpperCase()+t.slice(1),r=t,i=en.length;while(i--)if(t=en[i]+n,t in e)return t;return r}function nn(e,t){return e=t||e,"none"===b.css(e,"display")||!b.contains(e.ownerDocument,e)}function rn(e,t){var n,r,i,o=[],a=0,s=e.length;for(;s>a;a++)r=e[a],r.style&&(o[a]=b._data(r,"olddisplay"),n=r.style.display,t?(o[a]||"none"!==n||(r.style.display=""),""===r.style.display&&nn(r)&&(o[a]=b._data(r,"olddisplay",un(r.nodeName)))):o[a]||(i=nn(r),(n&&"none"!==n||!i)&&b._data(r,"olddisplay",i?n:b.css(r,"display"))));for(a=0;s>a;a++)r=e[a],r.style&&(t&&"none"!==r.style.display&&""!==r.style.display||(r.style.display=t?o[a]||"":"none"));return e}b.fn.extend({css:function(e,n){return b.access(this,function(e,n,r){var i,o,a={},s=0;if(b.isArray(n)){for(o=Rt(e),i=n.length;i>s;s++)a[n[s]]=b.css(e,n[s],!1,o);return a}return r!==t?b.style(e,n,r):b.css(e,n)},e,n,arguments.length>1)},show:function(){return rn(this,!0)},hide:function(){return rn(this)},toggle:function(e){var t="boolean"==typeof e;return this.each(function(){(t?e:nn(this))?b(this).show():b(this).hide()})}}),b.extend({cssHooks:{opacity:{get:function(e,t){if(t){var n=Wt(e,"opacity");return""===n?"1":n}}}},cssNumber:{columnCount:!0,fillOpacity:!0,fontWeight:!0,lineHeight:!0,opacity:!0,orphans:!0,widows:!0,zIndex:!0,zoom:!0},cssProps:{"float":b.support.cssFloat?"cssFloat":"styleFloat"},style:function(e,n,r,i){if(e&&3!==e.nodeType&&8!==e.nodeType&&e.style){var o,a,s,u=b.camelCase(n),l=e.style;if(n=b.cssProps[u]||(b.cssProps[u]=tn(l,u)),s=b.cssHooks[n]||b.cssHooks[u],r===t)return s&&"get"in s&&(o=s.get(e,!1,i))!==t?o:l[n];if(a=typeof r,"string"===a&&(o=Jt.exec(r))&&(r=(o[1]+1)*o[2]+parseFloat(b.css(e,n)),a="number"),!(null==r||"number"===a&&isNaN(r)||("number"!==a||b.cssNumber[u]||(r+="px"),b.support.clearCloneStyle||""!==r||0!==n.indexOf("background")||(l[n]="inherit"),s&&"set"in s&&(r=s.set(e,r,i))===t)))try{l[n]=r}catch(c){}}},css:function(e,n,r,i){var o,a,s,u=b.camelCase(n);return n=b.cssProps[u]||(b.cssProps[u]=tn(e.style,u)),s=b.cssHooks[n]||b.cssHooks[u],s&&"get"in s&&(a=s.get(e,!0,r)),a===t&&(a=Wt(e,n,i)),"normal"===a&&n in Kt&&(a=Kt[n]),""===r||r?(o=parseFloat(a),r===!0||b.isNumeric(o)?o||0:a):a},swap:function(e,t,n,r){var i,o,a={};for(o in t)a[o]=e.style[o],e.style[o]=t[o];i=n.apply(e,r||[]);for(o in t)e.style[o]=a[o];return i}}),e.getComputedStyle?(Rt=function(t){return e.getComputedStyle(t,null)},Wt=function(e,n,r){var i,o,a,s=r||Rt(e),u=s?s.getPropertyValue(n)||s[n]:t,l=e.style;return s&&(""!==u||b.contains(e.ownerDocument,e)||(u=b.style(e,n)),Yt.test(u)&&Ut.test(n)&&(i=l.width,o=l.minWidth,a=l.maxWidth,l.minWidth=l.maxWidth=l.width=u,u=s.width,l.width=i,l.minWidth=o,l.maxWidth=a)),u}):o.documentElement.currentStyle&&(Rt=function(e){return e.currentStyle},Wt=function(e,n,r){var i,o,a,s=r||Rt(e),u=s?s[n]:t,l=e.style;return null==u&&l&&l[n]&&(u=l[n]),Yt.test(u)&&!zt.test(n)&&(i=l.left,o=e.runtimeStyle,a=o&&o.left,a&&(o.left=e.currentStyle.left),l.left="fontSize"===n?"1em":u,u=l.pixelLeft+"px",l.left=i,a&&(o.left=a)),""===u?"auto":u});function on(e,t,n){var r=Vt.exec(t);return r?Math.max(0,r[1]-(n||0))+(r[2]||"px"):t}function an(e,t,n,r,i){var o=n===(r?"border":"content")?4:"width"===t?1:0,a=0;for(;4>o;o+=2)"margin"===n&&(a+=b.css(e,n+Zt[o],!0,i)),r?("content"===n&&(a-=b.css(e,"padding"+Zt[o],!0,i)),"margin"!==n&&(a-=b.css(e,"border"+Zt[o]+"Width",!0,i))):(a+=b.css(e,"padding"+Zt[o],!0,i),"padding"!==n&&(a+=b.css(e,"border"+Zt[o]+"Width",!0,i)));return a}function sn(e,t,n){var r=!0,i="width"===t?e.offsetWidth:e.offsetHeight,o=Rt(e),a=b.support.boxSizing&&"border-box"===b.css(e,"boxSizing",!1,o);if(0>=i||null==i){if(i=Wt(e,t,o),(0>i||null==i)&&(i=e.style[t]),Yt.test(i))return i;r=a&&(b.support.boxSizingReliable||i===e.style[t]),i=parseFloat(i)||0}return i+an(e,t,n||(a?"border":"content"),r,o)+"px"}function un(e){var t=o,n=Gt[e];return n||(n=ln(e,t),"none"!==n&&n||(Pt=(Pt||b("'),u.bgiframe.appendTo(c),c.bind("tooltipmove"+h,o.adjustBGIFrame)),m=r("
    ",{id:"qtip-rcontainer"}).appendTo(t.body),o.redraw(),u.overlay&&!v&&(s=function(){u.overlay[0].style.top=n.scrollTop()+"px"},n.bind("scroll.qtip-ie6, resize.qtip-ie6",s),s(),u.overlay.addClass("qtipmodal-ie6fix"),v=i)},adjustBGIFrame:function(){var e=n.get("dimensions"),t=n.plugins.tip,r=u.tip,i,s;s=parseInt(c.css("border-left-width"),10)||0,s={left:-s,top:-s},t&&r&&(i=t.corner.precedance==="x"?["width","left"]:["height","top"],s[i[1]]-=r[i[0]]()),u.bgiframe.css(s).css(e)},redraw:function(){if(n.rendered<1||d)return o;var e=a.style,t=a.position.container,r,i,s,u;return d=1,e.height&&c.css(l,e.height),e.width?c.css(f,e.width):(c.css(f,"").appendTo(m),i=c.width(),i%2<1&&(i+=1),s=c.css("max-width")||"",u=c.css("min-width")||"",r=(s+u).indexOf("%")>-1?t.width()/100:0,s=(s.indexOf("%")>-1?r:1)*parseInt(s,10)||i,u=(u.indexOf("%")>-1?r:1)*parseInt(u,10)||0,i=s+u?Math.min(Math.max(i,u),s):i,c.css(f,Math.round(i)).appendTo(t)),d=0,o},destroy:function(){p&&u.bgiframe.remove(),c.unbind(h)}}),o.init()}var i=!0,s=!1,o=null,u="x",a="y",f="width",l="height",c="top",h="left",p="bottom",d="right",v="center",m="flip",g="flipinvert",y="shift",b,w,E,S="qtip",x={},T=["ui-widget","ui-tooltip"],N="div.qtip."+S,C=S+"-default",k=S+"-focus",L=S+"-hover",A="_replacedByqTip",O="oldtitle",M;b=r.fn.qtip=function(e,t,u){var a=(""+e).toLowerCase(),f=o,l=r.makeArray(arguments).slice(1),c=l[l.length-1],h=this[0]?r.data(this[0],"qtip"):o;if(!arguments.length&&h||a==="api")return h;if("string"==typeof e)return this.each(function(){var e=r.data(this,"qtip");if(!e)return i;c&&c.timeStamp&&(e.cache.event=c);if(a!=="option"&&a!=="options"||!t)e[a]&&e[a].apply(e[a],l);else{if(!r.isPlainObject(t)&&u===n)return f=e.get(t),s;e.set(t,u)}}),f!==o?f:this;if("object"==typeof e||!arguments.length)return h=D(r.extend(i,{},e)),b.bind.call(this,h,c)},b.bind=function(e,t){return this.each(function(o){function p(e){function t(){c.render(typeof e=="object"||u.show.ready),a.show.add(a.hide).unbind(l)}if(c.cache.disabled)return s;c.cache.event=r.extend({},e),c.cache.target=e?r(e.target):[n],u.show.delay>0?(clearTimeout(c.timers.show),c.timers.show=setTimeout(t,u.show.delay),f.show!==f.hide&&a.hide.bind(f.hide,function(){clearTimeout(c.timers.show)})):t()}var u,a,f,l,c,h;h=r.isArray(e.id)?e.id[o]:e.id,h=!h||h===s||h.length<1||x[h]?b.nextid++:x[h]=h,l=".qtip-"+h+"-create",c=H.call(this,h,e);if(c===s)return i;u=c.options,r.each(w,function(){this.initialize==="initialize"&&this(c)}),a={show:u.show.target,hide:u.hide.target},f={show:r.trim(""+u.show.event).replace(/ /g,l+" ")+l,hide:r.trim(""+u.hide.event).replace(/ /g,l+" ")+l},/mouse(over|enter)/i.test(f.show)&&!/mouse(out|leave)/i.test(f.hide)&&(f.hide+=" mouseleave"+l),a.show.bind("mousemove"+l,function(e){_(e),c.cache.onTarget=i}),a.show.bind(f.show,p),(u.show.ready||u.prerender)&&p(t)}).attr("data-hasqtip",i)},w=b.plugins={Corner:function(e){e=(""+e).replace(/([A-Z])/," $1").replace(/middle/gi,v).toLowerCase(),this.x=(e.match(/left|right/i)||e.match(/center/)||["inherit"])[0].toLowerCase(),this.y=(e.match(/top|bottom|center/i)||["inherit"])[0].toLowerCase();var t=e.charAt(0);this.precedance=t==="t"||t==="b"?a:u,this.string=function(){return this.precedance===a?this.y+this.x:this.x+this.y},this.abbrev=function(){var e=this.x.substr(0,1),t=this.y.substr(0,1);return e===t?e:this.precedance===a?t+e:e+t},this.invertx=function(e){this.x=this.x===h?d:this.x===d?h:e||this.x},this.inverty=function(e){this.y=this.y===c?p:this.y===p?c:e||this.y},this.clone=function(){return{x:this.x,y:this.y,precedance:this.precedance,string:this.string,abbrev:this.abbrev,clone:this.clone,invertx:this.invertx,inverty:this.inverty}}},offset:function(e,n){function c(e,t){i.left+=t*e.scrollLeft(),i.top+=t*e.scrollTop()}var i=e.offset(),s=e.closest("body"),o=r.browser.msie&&t.compatMode!=="CSS1Compat",u=n,a,f,l;if(u){do u.css("position")!=="static"&&(f=u.position(),i.left-=f.left+(parseInt(u.css("borderLeftWidth"),10)||0)+(parseInt(u.css("marginLeft"),10)||0),i.top-=f.top+(parseInt(u.css("borderTopWidth"),10)||0)+(parseInt(u.css("marginTop"),10)||0),!a&&(l=u.css("overflow"))!=="hidden"&&l!=="visible"&&(a=u));while((u=r(u[0].offsetParent)).length);(a&&a[0]!==s[0]||o)&&c(a||s,1)}return i},iOS:parseFloat((""+(/CPU.*OS ([0-9_]{1,5})|(CPU like).*AppleWebKit.*Mobile/i.exec(navigator.userAgent)||[0,""])[1]).replace("undefined","3_2").replace("_",".").replace("_",""))||s,fn:{attr:function(e,t){if(this.length){var n=this[0],i="title",s=r.data(n,"qtip");if(e===i&&s&&"object"==typeof s&&s.options.suppress)return arguments.length<2?r.attr(n,O):(s&&s.options.content.attr===i&&s.cache.attr&&s.set("content.text",t),this.attr(O,t))}return r.fn["attr"+A].apply(this,arguments)},clone:function(e){var t=r([]),n="title",i=r.fn["clone"+A].apply(this,arguments);return e||i.filter("["+O+"]").attr("title",function(){return r.attr(this,O)}).removeAttr(O),i}}},r.each(w.fn,function(e,t){if(!t||r.fn[e+A])return i;var n=r.fn[e+A]=r.fn[e];r.fn[e]=function(){return t.apply(this,arguments)||n.apply(this,arguments)}}),r.ui||(r["cleanData"+A]=r.cleanData,r.cleanData=function(e){for(var t=0,i;(i=e[t])!==n;t++)try{r(i).triggerHandler("removeqtip")}catch(s){}r["cleanData"+A](e)}),b.version="2.0.0-nightly-15f5c6bc20",b.nextid=0,b.inactiveEvents="click dblclick mousedown mouseup mousemove mouseleave mouseenter".split(" "),b.zindex=15e3,b.defaults={prerender:s,id:s,overwrite:i,suppress:i,content:{text:i,attr:"title",deferred:s,title:{text:s,button:s}},position:{my:"top left",at:"bottom right",target:s,container:s,viewport:s,adjust:{x:0,y:0,mouse:i,resize:i,method:"flipinvert flipinvert"},effect:function(e,t,n){r(this).animate(t,{duration:200,queue:s})}},show:{target:s,event:"mouseenter",effect:i,delay:90,solo:s,ready:s,autofocus:s},hide:{target:s,event:"mouseleave",effect:i,delay:0,fixed:s,inactive:s,leave:"window",distance:s},style:{classes:"",widget:s,width:s,height:s,def:i},events:{render:o,move:o,show:o,hide:o,toggle:o,visible:o,hidden:o,focus:o,blur:o}},w.svg=function(e,n,i,s){var o=r(t),u=n[0],a={width:0,height:0,position:{top:1e10,left:1e10}},f,l,c,h,p;while(!u.getBBox)u=u.parentNode;if(u.getBBox&&u.parentNode){f=u.getBBox(),l=u.getScreenCTM(),c=u.farthestViewportElement||u;if(!c.createSVGPoint)return a;h=c.createSVGPoint(),h.x=f.x,h.y=f.y,p=h.matrixTransform(l),a.position.left=p.x,a.position.top=p.y,h.x+=f.width,h.y+=f.height,p=h.matrixTransform(l),a.width=p.x-a.position.left,a.height=p.y-a.position.top,a.position.left+=o.scrollLeft(),a.position.top+=o.scrollTop()}return a},w.ajax=function(e){var t=e.plugins.ajax;return"object"==typeof t?t:e.plugins.ajax=new B(e)},w.ajax.initialize="render",w.ajax.sanitize=function(e){var t=e.content,n;t&&"ajax"in t&&(n=t.ajax,typeof n!="object"&&(n=e.content.ajax={url:n}),"boolean"!=typeof n.once&&n.once&&(n.once=!!n.once))},r.extend(i,b.defaults,{content:{ajax:{loading:i,once:i}}}),w.tip=function(e){var t=e.plugins.tip;return"object"==typeof t?t:e.plugins.tip=new F(e)},w.tip.initialize="render",w.tip.sanitize=function(e){var t=e.style,n;t&&"tip"in t&&(n=e.style.tip,typeof n!="object"&&(e.style.tip={corner:n}),/string|boolean/i.test(typeof n.corner)||(n.corner=i),typeof n.width!="number"&&delete n.width,typeof n.height!="number"&&delete n.height,typeof n.border!="number"&&n.border!==i&&delete n.border,typeof n.offset!="number"&&delete n.offset)},r.extend(i,b.defaults,{style:{tip:{corner:i,mimic:s,width:6,height:6,border:i,offset:0}}}),w.modal=function(e){var t=e.plugins.modal;return"object"==typeof t?t:e.plugins.modal=new I(e)},w.modal.initialize="render",w.modal.sanitize=function(e){e.show&&(typeof e.show.modal!="object"?e.show.modal={on:!!e.show.modal}:typeof e.show.modal.on=="undefined"&&(e.show.modal.on=i))},w.modal.zindex=b.zindex-200,w.modal.focusable=["a[href]","area[href]","input","select","textarea","button","iframe","object","embed","[tabindex]","[contenteditable]"],r.extend(i,b.defaults,{show:{modal:{on:s,effect:i,blur:i,stealfocus:i,escape:i}}}),w.viewport=function(n,r,i,s,o,m,b){function j(e,t,n,i,s,o,u,a,f){var l=r[s],c=x[e],h=T[e],p=n===y,d=-O.offset[s]+A.offset[s]+A["scroll"+s],m=c===s?f:c===o?-f:-f/2,b=h===s?a:h===o?-a:-a/2,w=_&&_.size?_.size[u]||0:0,E=_&&_.corner&&_.corner.precedance===e&&!p?w:0,S=d-l+E,N=l+f-A[u]-d+E,C=m-(x.precedance===e||c===x[t]?b:0)-(h===v?a/2:0);return p?(E=_&&_.corner&&_.corner.precedance===t?w:0,C=(c===s?1:-1)*m-E,r[s]+=S>0?S:N>0?-N:0,r[s]=Math.max(-O.offset[s]+A.offset[s]+(E&&_.corner[e]===v?_.offset:0),l-C,Math.min(Math.max(-O.offset[s]+A.offset[s]+A[u],l+C),r[s]))):(i*=n===g?2:0,S>0&&(c!==s||N>0)?(r[s]-=C+i,H["invert"+e](s)):N>0&&(c!==o||S>0)&&(r[s]-=(c===v?-C:C)+i,H["invert"+e](o)),r[s]N&&(r[s]=l,H=x.clone())),r[s]-l}var w=i.target,E=n.elements.tooltip,x=i.my,T=i.at,N=i.adjust,C=N.method.split(" "),k=C[0],L=C[1]||C[0],A=i.viewport,O=i.container,M=n.cache,_=n.plugins.tip,D={left:0,top:0},P,H,B;if(!A.jquery||w[0]===e||w[0]===t.body||N.method==="none")return D;P=E.css("position")==="fixed",A={elem:A,height:A[(A[0]===e?"h":"outerH")+"eight"](),width:A[(A[0]===e?"w":"outerW")+"idth"](),scrollleft:P?0:A.scrollLeft(),scrolltop:P?0:A.scrollTop(),offset:A.offset()||{left:0,top:0}},O={elem:O,scrollLeft:O.scrollLeft(),scrollTop:O.scrollTop(),offset:O.offset()||{left:0,top:0}};if(k!=="shift"||L!=="shift")H=x.clone();return D={left:k!=="none"?j(u,a,k,N.x,h,d,f,s,m):0,top:L!=="none"?j(a,u,L,N.y,c,p,l,o,b):0},H&&M.lastClass!==(B=S+"-pos-"+H.abbrev())&&E.removeClass(n.cache.lastClass).addClass(n.cache.lastClass=B),D},w.imagemap=function(e,t,n,i){function E(e,t,n){var r=0,i=1,s=1,o=0,u=0,a=e.width,f=e.height;while(a>0&&f>0&&i>0&&s>0){a=Math.floor(a/2),f=Math.floor(f/2),n.x===h?i=a:n.x===d?i=e.width-a:i+=Math.floor(a/2),n.y===c?s=f:n.y===p?s=e.height-f:s+=Math.floor(f/2),r=t.length;while(r--){if(t.length<2)break;o=t[r][0]-e.position.left,u=t[r][1]-e.position.top,(n.x===h&&o>=i||n.x===d&&o<=i||n.x===v&&(oe.width-i)||n.y===c&&u>=s||n.y===p&&u<=s||n.y===v&&(ue.height-s))&&t.splice(r,1)}}return{left:t[0][0],top:t[0][1]}}t.jquery||(t=r(t));var s=e.cache.areas={},o=(t[0].shape||t.attr("shape")).toLowerCase(),u=t[0].coords||t.attr("coords"),a=u.split(","),f=[],l=r('img[usemap="#'+t.parent("map").attr("name")+'"]'),m=l.offset(),g={width:0,height:0,position:{top:1e10,right:0,bottom:0,left:1e10}},y=0,b=0,w;m.left+=Math.ceil((l.outerWidth()-l.width())/2),m.top+=Math.ceil((l.outerHeight()-l.height())/2);if(o==="poly"){y=a.length;while(y--)b=[parseInt(a[--y],10),parseInt(a[y+1],10)],b[0]>g.position.right&&(g.position.right=b[0]),b[0]g.position.bottom&&(g.position.bottom=b[1]),b[1] - algorithms - analysis - answer - api - collection - concepts - corpus - design - documents - features - framework - human - index - infer - install - introduction - latent - dirichlet allocation - model - open-source - paragraphs - python - query - questions - random - reference - representation - semantic - similar - space - sparse - structure - SVD - text - thought - topic - training - tutorials - unsupervised - vector - words -
    - diff --git a/docs/src/_static/widget.js b/docs/src/_static/widget.js deleted file mode 100644 index aca590113a..0000000000 --- a/docs/src/_static/widget.js +++ /dev/null @@ -1,12 +0,0 @@ -/** - * Twitter - http://twitter.com - * Copyright (C) 2010 Twitter - * Author: Dustin Diaz (dustin@twitter.com) - * - * V 2.2.5 Twitter search/profile/faves/list widget - * http://twitter.com/widgets - * For full documented source see http://twitter.com/javascripts/widgets/widget.js - * Hosting and modifications of the original source IS allowed. - */ - -TWTR=window.TWTR||{};if(!Array.forEach){Array.prototype.filter=function(E,F){var D=F||window;var A=[];for(var C=0,B=this.length;Cthis.time){this._setStyle(this.to);if(this.callback){this.callback.call(this)}clearInterval(this.timer);return }this.percentage=(Math.floor((this.diff/this.time)*100)/100);this.val=(this.animDiff*this.percentage)+this.from;this._setStyle(this.val)};A.prototype.start=function(){var C=this;this.startTime=new Date();this.timer=setInterval(function(){C._animate.call(C)},15)};TWTR.Widget=function(C){this.init(C)};(function(){var R={};var O=location.protocol.match(/https/);var Q=/^.+\/profile_images/;var W="https://s3.amazonaws.com/twitter_production/profile_images";var f={};var d=function(h){var g=f[h];if(!g){g=new RegExp("(?:^|\\s+)"+h+"(?:\\s+|$)");f[h]=g}return g};var D=function(l,p,m,n){var p=p||"*";var m=m||document;var h=[],g=m.getElementsByTagName(p),o=d(l);for(var j=0,k=g.length;j0&&n<13){h="am";return n}else{if(n<1){h="am";return 12}else{h="pm";return n-12}}}();var i=m.getMinutes();var l=m.getSeconds();function k(){var n=new Date();if(n.getDate()!=m.getDate()||n.getYear()!=m.getYear()||n.getMonth()!=m.getMonth()){return" - "+M[m.getMonth()]+" "+m.getDate()+", "+m.getFullYear()}else{return""}}return g+":"+i+h+k()};var K=function(m){var o=new Date();var k=new Date(m);if(e.ie){k=Date.parse(m.replace(/( \+)/," UTC$1"))}var n=o-k;var h=1000,i=h*60,j=i*60,l=j*24,g=l*7;if(isNaN(n)||n<0){return""}if(nl&&n'+((l.length>25)?l.substr(0,24)+"...":l)+""+h})},at:function(g){return g.replace(/\B[@＠]([a-zA-Z0-9_]{1,20})/g,function(h,i){return'@'+i+""})},list:function(g){return g.replace(/\B[@＠]([a-zA-Z0-9_]{1,20}\/\w+)/g,function(h,i){return'@'+i+""})},hash:function(g){return g.replace(/(^|\s+)#(\w+)/gi,function(h,i,j){return i+'#'+j+""})},clean:function(g){return this.hash(this.at(this.list(this.link(g))))}};function P(h,i,g){this.job=h;this.decayFn=i;this.interval=g;this.decayRate=1;this.decayMultiplier=1.25;this.maxDecayTime=3*60*1000}P.prototype={start:function(){this.stop().run();return this},stop:function(){if(this.worker){window.clearTimeout(this.worker)}return this},run:function(){var g=this;this.job(function(){g.decayRate=g.decayFn()?Math.max(1,g.decayRate/g.decayMultiplier):g.decayRate*g.decayMultiplier;var h=g.interval*g.decayRate;h=(h>=g.maxDecayTime)?g.maxDecayTime:h;h=Math.floor(h);g.worker=window.setTimeout(function(){g.run.call(g)},h)})},destroy:function(){this.stop();this.decayRate=1;return this}};function J(h,g,i){this.time=h||6000;this.loop=g||false;this.repeated=0;this.callback=i;this.haystack=[]}J.prototype={set:function(g){this.haystack=g},add:function(g){this.haystack.unshift(g)},start:function(){if(this.timer){return this}this._job();var g=this;this.timer=setInterval(function(){g._job.call(g)},this.time);return this},stop:function(){if(this.timer){window.clearInterval(this.timer);this.timer=null}return this},_next:function(){var g=this.haystack.shift();if(g&&this.loop){this.haystack.push(g)}return g||null},_job:function(){var g=this._next();if(g){this.callback(g)}return this}};function L(i){function g(){if(i.needle.metadata&&i.needle.metadata.result_type&&i.needle.metadata.result_type=="popular"){return''+i.needle.metadata.recent_retweets+"+ recent retweets"}else{return""}}if(O){i.avatar=i.avatar.replace(Q,W)}var h='
    '+i.user+' profile

    '+i.user+" "+i.tweet+' '+i.created_at+' · reply '+g()+"

    ";var j=document.createElement("div");j.id="tweet-id-"+ ++L._tweetCount;j.className="twtr-tweet";j.innerHTML=h;this.element=j}L._tweetCount=0;R.loadStyleSheet=function(i,h){if(!TWTR.Widget.loadingStyleSheet){TWTR.Widget.loadingStyleSheet=true;var g=document.createElement("link");g.href=i;g.rel="stylesheet";g.type="text/css";document.getElementsByTagName("head")[0].appendChild(g);var j=setInterval(function(){var k=G(h,"position");if(k=="relative"){clearInterval(j);j=null;TWTR.Widget.hasLoadedStyleSheet=true}},50)}};(function(){var g=false;R.css=function(j){var i=document.createElement("style");i.type="text/css";if(e.ie){i.styleSheet.cssText=j}else{var k=document.createDocumentFragment();k.appendChild(document.createTextNode(j));i.appendChild(k)}function h(){document.getElementsByTagName("head")[0].appendChild(i)}if(!e.ie||g){h()}else{window.attachEvent("onload",function(){g=true;h()})}}})();TWTR.Widget.isLoaded=false;TWTR.Widget.loadingStyleSheet=false;TWTR.Widget.hasLoadedStyleSheet=false;TWTR.Widget.WIDGET_NUMBER=0;TWTR.Widget.matches={mentions:/^@[a-zA-Z0-9_]{1,20}\b/,any_mentions:/\b@[a-zA-Z0-9_]{1,20}\b/};TWTR.Widget.jsonP=function(h,j){var g=document.createElement("script");var i=document.getElementsByTagName("head")[0];g.type="text/javascript";g.src=h;i.insertBefore(g,i.firstChild);j(g);return g};TWTR.Widget.prototype=function(){var j=O?"https://":"http://";var l=j+"search.twitter.com/search.";var m=j+"api.twitter.com/1/statuses/user_timeline.";var i=j+"twitter.com/favorites/";var k=j+"twitter.com/";var h=25000;var g=O?"https://twitter-widgets.s3.amazonaws.com/j/1/default.gif":"http://widgets.twimg.com/j/1/default.gif";return{init:function(o){var n=this;this._widgetNumber=++TWTR.Widget.WIDGET_NUMBER;TWTR.Widget["receiveCallback_"+this._widgetNumber]=function(p){n._prePlay.call(n,p)};this._cb="TWTR.Widget.receiveCallback_"+this._widgetNumber;this.opts=o;this._base=l;this._isRunning=false;this._hasOfficiallyStarted=false;this._hasNewSearchResults=false;this._rendered=false;this._profileImage=false;this._isCreator=!!o.creator;this._setWidgetType(o.type);this.timesRequested=0;this.runOnce=false;this.newResults=false;this.results=[];this.jsonMaxRequestTimeOut=19000;this.showedResults=[];this.sinceId=1;this.source="TWITTERINC_WIDGET";this.id=o.id||"twtr-widget-"+this._widgetNumber;this.tweets=0;this.setDimensions(o.width,o.height);this.interval=o.interval||6000;this.format="json";this.rpp=o.rpp||50;this.subject=o.subject||"";this.title=o.title||"";this.setFooterText(o.footer);this.setSearch(o.search);this._setUrl();this.theme=o.theme?o.theme:this._getDefaultTheme();if(!o.id){document.write('
    ')}this.widgetEl=H(this.id);if(o.id){b.add(this.widgetEl,"twtr-widget")}if(o.version>=2&&!TWTR.Widget.hasLoadedStyleSheet){if(O){R.loadStyleSheet("https://twitter-widgets.s3.amazonaws.com/j/2/widget.css",this.widgetEl)}else{if(o.creator){R.loadStyleSheet("/stylesheets/widgets/widget.css",this.widgetEl)}else{R.loadStyleSheet("http://widgets.twimg.com/j/2/widget.css",this.widgetEl)}}}this.occasionalJob=new P(function(p){n.decay=p;n._getResults.call(n)},function(){return n._decayDecider.call(n)},h);this._ready=I.fn(o.ready)?o.ready:function(){};this._isRelativeTime=true;this._tweetFilter=false;this._avatars=true;this._isFullScreen=false;this._isLive=true;this._isScroll=false;this._loop=true;this._showTopTweets=(this._isSearchWidget)?true:false;this._behavior="default";this.setFeatures(this.opts.features);this.intervalJob=new J(this.interval,this._loop,function(p){n._normalizeTweet(p)});return this},setDimensions:function(n,o){this.wh=(n&&o)?[n,o]:[250,300];if(n=="auto"||n=="100%"){this.wh[0]="100%"}else{this.wh[0]=((this.wh[0]<150)?150:this.wh[0])+"px"}this.wh[1]=((this.wh[1]<100)?100:this.wh[1])+"px";return this},setRpp:function(n){var n=parseInt(n);this.rpp=(I.number(n)&&(n>0&&n<=100))?n:30;return this},_setWidgetType:function(n){this._isSearchWidget=false,this._isProfileWidget=false,this._isFavsWidget=false,this._isListWidget=false;switch(n){case"profile":this._isProfileWidget=true;break;case"search":this._isSearchWidget=true,this.search=this.opts.search;break;case"faves":case"favs":this._isFavsWidget=true;break;case"list":case"lists":this._isListWidget=true;break}return this},setFeatures:function(o){if(o){if(I.def(o.filters)){this._tweetFilter=o.filters}if(I.def(o.dateformat)){this._isRelativeTime=!!(o.dateformat!=="absolute")}if(I.def(o.fullscreen)&&I.bool(o.fullscreen)){if(o.fullscreen){this._isFullScreen=true;this.wh[0]="100%";this.wh[1]=(V()-90)+"px";var p=this;E.add(window,"resize",function(s){p.wh[1]=V();p._fullScreenResize()})}}if(I.def(o.loop)&&I.bool(o.loop)){this._loop=o.loop}if(I.def(o.behavior)&&I.string(o.behavior)){switch(o.behavior){case"all":this._behavior="all";break;case"preloaded":this._behavior="preloaded";break;default:this._behavior="default";break}}if(I.def(o.toptweets)&&I.bool(o.toptweets)){this._showTopTweets=o.toptweets;var n=(this._showTopTweets)?"inline-block":"none";R.css("#"+this.id+" .twtr-popular { display: "+n+"; }")}if(!I.def(o.toptweets)){this._showTopTweets=true;var n=(this._showTopTweets)?"inline-block":"none";R.css("#"+this.id+" .twtr-popular { display: "+n+"; }")}if(I.def(o.avatars)&&I.bool(o.avatars)){if(!o.avatars){R.css("#"+this.id+" .twtr-avatar, #"+this.id+" .twtr-user { display: none; } #"+this.id+" .twtr-tweet-text { margin-left: 0; }");this._avatars=false}else{var q=(this._isFullScreen)?"90px":"40px";R.css("#"+this.id+" .twtr-avatar { display: block; } #"+this.id+" .twtr-user { display: inline; } #"+this.id+" .twtr-tweet-text { margin-left: "+q+"; }");this._avatars=true}}else{if(this._isProfileWidget){this.setFeatures({avatars:false});this._avatars=false}else{this.setFeatures({avatars:true});this._avatars=true}}if(I.def(o.hashtags)&&I.bool(o.hashtags)){(!o.hashtags)?R.css("#"+this.id+" a.twtr-hashtag { display: none; }"):""}if(I.def(o.timestamp)&&I.bool(o.timestamp)){var r=o.timestamp?"block":"none";R.css("#"+this.id+" em { display: "+r+"; }")}if(I.def(o.live)&&I.bool(o.live)){this._isLive=o.live}if(I.def(o.scrollbar)&&I.bool(o.scrollbar)){this._isScroll=o.scrollbar}}else{if(this._isProfileWidget){this.setFeatures({avatars:false});this._avatars=false}if(this._isProfileWidget||this._isFavsWidget){this.setFeatures({behavior:"all"})}}return this},_fullScreenResize:function(){var n=D("twtr-timeline","div",document.body,function(o){o.style.height=(V()-90)+"px"})},setTweetInterval:function(n){this.interval=n;return this},setBase:function(n){this._base=n;return this},setUser:function(o,n){this.username=o;this.realname=n||" ";if(this._isFavsWidget){this.setBase(i+o+".")}else{if(this._isProfileWidget){this.setBase(m+this.format+"?screen_name="+o)}}this.setSearch(" ");return this},setList:function(o,n){this.listslug=n.replace(/ /g,"-").toLowerCase();this.username=o;this.setBase(k+o+"/lists/"+this.listslug+"/statuses.");this.setSearch(" ");return this},setProfileImage:function(n){this._profileImage=n;this.byClass("twtr-profile-img","img").src=O?n.replace(Q,W):n;this.byClass("twtr-profile-img-anchor","a").href="http://twitter.com/"+this.username;return this},setTitle:function(n){this.title=n;this.widgetEl.getElementsByTagName("h3")[0].innerHTML=this.title;return this},setCaption:function(n){this.subject=n;this.widgetEl.getElementsByTagName("h4")[0].innerHTML=this.subject;return this},setFooterText:function(n){this.footerText=(I.def(n)&&I.string(n))?n:"Join the conversation";if(this._rendered){this.byClass("twtr-join-conv","a").innerHTML=this.footerText}return this},setSearch:function(o){this.searchString=o||"";this.search=encodeURIComponent(this.searchString);this._setUrl();if(this._rendered){var n=this.byClass("twtr-join-conv","a");n.href="http://twitter.com/"+this._getWidgetPath()}return this},_getWidgetPath:function(){if(this._isProfileWidget){return this.username}else{if(this._isFavsWidget){return this.username+"/favorites"}else{if(this._isListWidget){return this.username+"/lists/"+this.listslug}else{return"#search?q="+this.search}}}},_setUrl:function(){var o=this;function n(){return"&"+(+new Date)+"=cachebust"}function p(){return(o.sinceId==1)?"":"&since_id="+o.sinceId+"&refresh=true"}if(this._isProfileWidget){this.url=this._base+"&callback="+this._cb+"&include_rts=true&count="+this.rpp+p()+"&clientsource="+this.source}else{if(this._isFavsWidget||this._isListWidget){this.url=this._base+this.format+"?callback="+this._cb+p()+"&include_rts=true&clientsource="+this.source}else{this.url=this._base+this.format+"?q="+this.search+"&include_rts=true&callback="+this._cb+"&rpp="+this.rpp+p()+"&clientsource="+this.source;if(!this.runOnce){this.url+="&result_type=mixed"}}}this.url+=n();return this},_getRGB:function(n){return N(n.substring(1,7))},setTheme:function(t,n){var r=this;var p=" !important";var s=((window.location.hostname.match(/twitter\.com/))&&(window.location.pathname.match(/goodies/)));if(n||s){p=""}this.theme={shell:{background:function(){return t.shell.background||r._getDefaultTheme().shell.background}(),color:function(){return t.shell.color||r._getDefaultTheme().shell.color}()},tweets:{background:function(){return t.tweets.background||r._getDefaultTheme().tweets.background}(),color:function(){return t.tweets.color||r._getDefaultTheme().tweets.color}(),links:function(){return t.tweets.links||r._getDefaultTheme().tweets.links}()}};var q="#"+this.id+" .twtr-doc, #"+this.id+" .twtr-hd a, #"+this.id+" h3, #"+this.id+" h4, #"+this.id+" .twtr-popular { background-color: "+this.theme.shell.background+p+"; color: "+this.theme.shell.color+p+"; } #"+this.id+" .twtr-popular { color: "+this.theme.tweets.color+p+"; background-color: rgba("+this._getRGB(this.theme.shell.background)+", .3)"+p+"; } #"+this.id+" .twtr-tweet a { color: "+this.theme.tweets.links+p+"; } #"+this.id+" .twtr-bd, #"+this.id+" .twtr-timeline i a, #"+this.id+" .twtr-bd p { color: "+this.theme.tweets.color+p+"; } #"+this.id+" .twtr-new-results, #"+this.id+" .twtr-results-inner, #"+this.id+" .twtr-timeline { background: "+this.theme.tweets.background+p+"; }";if(e.ie){q+="#"+this.id+" .twtr-tweet { background: "+this.theme.tweets.background+p+"; }"}R.css(q);return this},byClass:function(q,n,o){var p=D(q,n,H(this.id));return(o)?p:p[0]},render:function(){var p=this;if(!TWTR.Widget.hasLoadedStyleSheet){window.setTimeout(function(){p.render.call(p)},50);return this}this.setTheme(this.theme,this._isCreator);if(this._isProfileWidget){b.add(this.widgetEl,"twtr-widget-profile")}if(this._isScroll){b.add(this.widgetEl,"twtr-scroll")}if(!this._isLive&&!this._isScroll){this.wh[1]="auto"}if(this._isSearchWidget&&this._isFullScreen){document.title="Twitter search: "+escape(this.searchString)}this.widgetEl.innerHTML=this._getWidgetHtml();var o=this.byClass("twtr-timeline","div");if(this._isLive&&!this._isFullScreen){var q=function(r){if(p._behavior==="all"){return }if(C.call(this,r)){p.pause.call(p)}};var n=function(r){if(p._behavior==="all"){return }if(C.call(this,r)){p.resume.call(p)}};this.removeEvents=function(){E.remove(o,"mouseover",q);E.remove(o,"mouseout",n)};E.add(o,"mouseover",q);E.add(o,"mouseout",n)}this._rendered=true;this._ready();return this},removeEvents:function(){},_getDefaultTheme:function(){return{shell:{background:"#8ec1da",color:"#ffffff"},tweets:{background:"#ffffff",color:"#444444",links:"#1985b5"}}},_getWidgetHtml:function(){var p=this;function r(){if(p._isProfileWidget){return'profile

    '}else{return"

    "+p.title+"

    "+p.subject+"

    "}}function o(){return p._isFullScreen?" twtr-fullscreen":""}var q=O?"https://twitter-widgets.s3.amazonaws.com/i/widget-logo.png":"http://widgets.twimg.com/i/widget-logo.png";if(this._isFullScreen){q="https://twitter-widgets.s3.amazonaws.com/i/widget-logo-fullscreen.png"}var n='
    '+r()+'
    ";return n},_appendTweet:function(n){this._insertNewResultsNumber();Z(n,this.byClass("twtr-reference-tweet","div"));return this},_slide:function(o){var p=this;var n=Y(o).offsetHeight;if(this.runOnce){new A(o,"height",{from:0,to:n,time:500,callback:function(){p._fade.call(p,o)}}).start()}return this},_fade:function(n){var o=this;if(A.canTransition){n.style.webkitTransition="opacity 0.5s ease-out";n.style.opacity=1;return this}new A(n,"opacity",{from:0,to:1,time:500}).start();return this},_chop:function(){if(this._isScroll){return this}var s=this.byClass("twtr-tweet","div",true);var t=this.byClass("twtr-new-results","div",true);if(s.length){for(var p=s.length-1;p>=0;p--){var r=s[p];var q=parseInt(r.offsetTop);if(q>parseInt(this.wh[1])){a(r)}else{break}}if(t.length>0){var n=t[t.length-1];var o=parseInt(n.offsetTop);if(o>parseInt(this.wh[1])){a(n)}}}return this},_appendSlideFade:function(o){var n=o||this.tweet.element;this._chop()._appendTweet(n)._slide(n);return this},_createTweet:function(n){n.timestamp=n.created_at;n.created_at=this._isRelativeTime?K(n.created_at):S(n.created_at);this.tweet=new L(n);if(this._isLive&&this.runOnce){this.tweet.element.style.opacity=0;this.tweet.element.style.filter="alpha(opacity:0)";this.tweet.element.style.height="0"}return this},_getResults:function(){var n=this;this.timesRequested++;this.jsonRequestRunning=true;this.jsonRequestTimer=window.setTimeout(function(){if(n.jsonRequestRunning){clearTimeout(n.jsonRequestTimer);n.jsonRequestTimer=null}n.jsonRequestRunning=false;a(n.scriptElement);n.newResults=false;n.decay()},this.jsonMaxRequestTimeOut);TWTR.Widget.jsonP(n.url,function(o){n.scriptElement=o})},clear:function(){var o=this.byClass("twtr-tweet","div",true);var n=this.byClass("twtr-new-results","div",true);o=o.concat(n);B(o,function(p){a(p)});return this},_sortByMagic:function(n){var o=this;if(this._tweetFilter){if(this._tweetFilter.negatives){n=n.filter(function(p){if(!o._tweetFilter.negatives.test(p.text)){return p}})}if(this._tweetFilter.positives){n=n.filter(function(p){if(o._tweetFilter.positives.test(p.text)){return p}})}}switch(this._behavior){case"all":this._sortByLatest(n);break;case"preloaded":default:this._sortByDefault(n);break}if(this._isLive&&this._behavior!=="all"){this.intervalJob.set(this.results);this.intervalJob.start()}return this},_loadTopTweetsAtTop:function(p){var q=[],r=[],o=[];B(p,function(s){if(s.metadata&&s.metadata.result_type&&s.metadata.result_type=="popular"){r.push(s)}else{q.push(s)}});var n=r.concat(q);return n},_sortByLatest:function(n){this.results=n;this.results=this.results.slice(0,this.rpp);this.results=this._loadTopTweetsAtTop(this.results);this.results.reverse();return this},_sortByDefault:function(o){var p=this;var n=function(r){return new Date(r).getTime()};this.results.unshift.apply(this.results,o);B(this.results,function(r){if(!r.views){r.views=0}});this.results.sort(function(s,r){if(n(s.created_at)>n(r.created_at)){return -1}else{if(n(s.created_at)r.views){return 1}}return 0});if(!this._isLive){this.results.reverse()}},_prePlay:function(o){if(this.jsonRequestTimer){clearTimeout(this.jsonRequestTimer);this.jsonRequestTimer=null}if(!e.ie){a(this.scriptElement)}if(o.error){this.newResults=false}else{if(o.results&&o.results.length>0){this.response=o;this.newResults=true;this.sinceId=o.max_id_str;this._sortByMagic(o.results);if(this.isRunning()){this._play()}}else{if((this._isProfileWidget||this._isFavsWidget||this._isListWidget)&&I.array(o)&&o.length){this.newResults=true;if(!this._profileImage&&this._isProfileWidget){var n=o[0].user.screen_name;this.setProfileImage(o[0].user.profile_image_url);this.setTitle(o[0].user.name);this.setCaption(''+n+"")}this.sinceId=o[0].id_str;this._sortByMagic(o);if(this.isRunning()){this._play()}}else{this.newResults=false}}}this._setUrl();if(this._isLive){this.decay()}},_play:function(){var n=this;if(this.runOnce){this._hasNewSearchResults=true}if(this._avatars){this._preloadImages(this.results)}if(this._isRelativeTime&&(this._behavior=="all"||this._behavior=="preloaded")){B(this.byClass("twtr-timestamp","a",true),function(o){o.innerHTML=K(o.getAttribute("time"))})}if(!this._isLive||this._behavior=="all"||this._behavior=="preloaded"){B(this.results,function(p){if(p.retweeted_status){p=p.retweeted_status}if(n._isProfileWidget){p.from_user=p.user.screen_name;p.profile_image_url=p.user.profile_image_url}if(n._isFavsWidget||n._isListWidget){p.from_user=p.user.screen_name;p.profile_image_url=p.user.profile_image_url}p.id=p.id_str;n._createTweet({id:p.id,user:p.from_user,tweet:F.clean(p.text),avatar:p.profile_image_url,created_at:p.created_at,needle:p});var o=n.tweet.element;(n._behavior=="all")?n._appendSlideFade(o):n._appendTweet(o)});if(this._behavior!="preloaded"){return this}}return this},_normalizeTweet:function(o){var n=this;o.views++;if(this._isProfileWidget){o.from_user=n.username;o.profile_image_url=o.user.profile_image_url}if(this._isFavsWidget||this._isListWidget){o.from_user=o.user.screen_name;o.profile_image_url=o.user.profile_image_url}if(this._isFullScreen){o.profile_image_url=o.profile_image_url.replace(/_normal\./,"_bigger.")}this._createTweet({id:o.id,user:o.from_user,tweet:F.clean(o.text),avatar:o.profile_image_url,created_at:o.created_at,needle:o})._appendSlideFade()},_insertNewResultsNumber:function(){if(!this._hasNewSearchResults){this._hasNewSearchResults=false;return }if(this.runOnce&&this._isSearchWidget){var q=this.response.total>this.rpp?this.response.total:this.response.results.length;var n=q>1?"s":"";var p=(this.response.warning&&this.response.warning.match(/adjusted since_id/))?"more than":"";var o=document.createElement("div");b.add(o,"twtr-new-results");o.innerHTML='
     
     
    '+p+" "+q+" new tweet"+n+"";Z(o,this.byClass("twtr-reference-tweet","div"));this._hasNewSearchResults=false}},_preloadImages:function(n){if(this._isProfileWidget||this._isFavsWidget||this._isListWidget){B(n,function(p){var o=new Image();o.src=p.user.profile_image_url})}else{B(n,function(o){(new Image()).src=o.profile_image_url})}},_decayDecider:function(){var n=false;if(!this.runOnce){this.runOnce=true;n=true}else{if(this.newResults){n=true}}return n},start:function(){var n=this;if(!this._rendered){setTimeout(function(){n.start.call(n)},50);return this}if(!this._isLive){this._getResults()}else{this.occasionalJob.start()}this._isRunning=true;this._hasOfficiallyStarted=true;return this},stop:function(){this.occasionalJob.stop();if(this.intervalJob){this.intervalJob.stop()}this._isRunning=false;return this},pause:function(){if(this.isRunning()&&this.intervalJob){this.intervalJob.stop();b.add(this.widgetEl,"twtr-paused");this._isRunning=false}if(this._resumeTimer){clearTimeout(this._resumeTimer);this._resumeTimer=null}return this},resume:function(){var n=this;if(!this.isRunning()&&this._hasOfficiallyStarted&&this.intervalJob){this._resumeTimer=window.setTimeout(function(){n.intervalJob.start();n._isRunning=true;b.remove(n.widgetEl,"twtr-paused")},2000)}return this},isRunning:function(){return this._isRunning},destroy:function(){this.stop();this.clear();this.runOnce=false;this._hasOfficiallyStarted=false;this._profileImage=false;this._isLive=true;this._tweetFilter=false;this._isScroll=false;this.newResults=false;this._isRunning=false;this.sinceId=1;this.results=[];this.showedResults=[];this.occasionalJob.destroy();if(this.jsonRequestRunning){clearTimeout(this.jsonRequestTimer)}b.remove(this.widgetEl,"twtr-scroll");this.removeEvents();return this}}}()})()})(); \ No newline at end of file diff --git a/docs/src/_templates/download.html b/docs/src/_templates/download.html deleted file mode 100644 index d984610c76..0000000000 --- a/docs/src/_templates/download.html +++ /dev/null @@ -1,4 +0,0 @@ -

    Download

    -

    Get gensim version {{ version }} from the Python Package -Index or install directly with:

    -
    easy_install -U gensim
    diff --git a/docs/src/_templates/indexcontent.html b/docs/src/_templates/indexcontent.html new file mode 100644 index 0000000000..a967020f2e --- /dev/null +++ b/docs/src/_templates/indexcontent.html @@ -0,0 +1,166 @@ +{% extends "layout.html" %} + +{% set title = _('Topic modelling for humans') %} + +{% block header %} +
    +

    gensim

    + +
    +
    +
    gensim
    +
    gensim tagline
    +
    + + +
    +
    +{% endblock %} + + +{% block document %} + + + + Fork me on GitHub + + +
    +
    + +
    + gensim code example +
    + +
    + Gensim is a FREE Python library +
      +
    • Scalable statistical semantics
      • +
    • Analyze plain-text documents for semantic structure
      • +
    • Retrieve semantically similar documents
      • +
    +
    +
    +
    + +
    +
    +

    Features
    Hover your mouse over each feature for more info.

    + +
    +
    +
    + memory independenceScalability +
    +
    + efficient implementationsEfficient implementations +
    +
    +
    +
    + platform independentPlatform independent +
    +
    + converters and wrappersConverters & I/O formats +
    +
    +
    + +
    +
    +
    + robustRobust +
    +
    + similarity queriesSimilarity queries +
    +
    +
    +
    + open sourceOpen source +
    +
    + supportSupport +
    +
    +
    + +
    +
    +
    + +
    +
    +

    Who is using Gensim?
    Doing something interesting with gensim? Ask to be featured here.

    + +
    +
      +
    • + “I used gensim at Ghent university. I found it easy to build prototypes with various models, extend it with additional features and gain empirical insights quickly. It's a reliable library that can be used beyond prototyping too.” Dieter Plaetinck, IBCN group +
      • +
    • + “We used gensim in several text mining projects at Sports Authority. The data were from free-form text fields in customer surveys, as well as social media sources. Having gensim significantly sped our time to development, and it is still my go-to package for topic modeling with large retail data sets.” Josh Hemann, Sports Authority +
      • +
    • + “Gensim hits the sweetest spot of being a simple yet powerful way to access some incredibly complex NLP goodness.” Alan J. Salmoni, Roistr.com +
      • +
    • + “Semantic analysis is a hot topic in online marketing, but there are few products on the market that are truly powerful. +Gensim is undoubtedly one of the best frameworks that efficiently implement algorithms for statistical analysis. +Few products, even commercial, have this level of quality.” Bruno Champion, DynAdmic +
      • + +
    • + “Based on our experience with gensim on DML-CZ, we naturally opted to use it on a much bigger scale for similarity +of fulltexts of scientific papers in the European Digital Mathematics Library. In evaluation with other approaches, gensim became a clear winner, especially because of speed, scalability and ease of use.”Petr Sojka, EuDML +
      • +
    • + “We have been using gensim in several DTU courses related to digital media engineering and find it immensely useful as the tutorial material provides students an excellent introduction to quickly understand the underlying principles in topic modeling based on both LSA and LDA.”Michael Kai Petersen, Technical University of Denmark +
      • + +
    +
    + +
    +
    +
    + +
    + get started +
    +{% endblock %} diff --git a/docs/src/_templates/indexsidebar.html b/docs/src/_templates/indexsidebar.html deleted file mode 100644 index bd2c463fb7..0000000000 --- a/docs/src/_templates/indexsidebar.html +++ /dev/null @@ -1,33 +0,0 @@ -

    Questions? Suggestions?

    - -

    Join the Google discussion group

    -
    - - - - - - -
    - - - - - - - - - -
    - - -

    -Check the source code at Github. Report bugs at the issue tracker. -

    diff --git a/docs/src/_templates/layout.html b/docs/src/_templates/layout.html deleted file mode 100644 index dd7fbc627a..0000000000 --- a/docs/src/_templates/layout.html +++ /dev/null @@ -1,16 +0,0 @@ -{% extends "!layout.html" %} - -{% set script_files = ['http://ajax.googleapis.com/ajax/libs/jquery/1.4.2/jquery.min.js'] + script_files %} - -{% block rootrellink %} -
  • Gensim home| 
    • -
  • Tutorials| 
    • -
  • Support| 
    • -
  • Contribute| 
    • -
  • API reference»
    • -{% endblock %} - - -{# put the sidebar before the body #} -{% block sidebar1 %}{{ sidebar() }}{% endblock %} -{% block sidebar2 %}{% endblock %} diff --git a/docs/src/_templates/page.html b/docs/src/_templates/page.html deleted file mode 100644 index de5dfd798a..0000000000 --- a/docs/src/_templates/page.html +++ /dev/null @@ -1,96 +0,0 @@ -{% extends "!page.html" %} - -{% block extrahead %} - {{ super() }} - - - - - - - - {% if current_page_name == 'index' %} - - {% endif %} - -{% endblock %} - - -{% block sidebar1 %} - - {{ super() }} - {% if current_page_name == '_index' %} -
    -
    - -

    Download

    -

    Get gensim from the Python Package - Index or install directly with:

    - 
    easy_install -U gensim
    - -
    -
    - - -
    -
    - {% endif %} - -{% endblock %} - -{% block relbar2 %} - {% if current_page_name != 'index' %} - {{ super() }} - {% endif %} -{% endblock %} diff --git a/docs/src/about.rst b/docs/src/about.rst index c6cb4a79b8..c3e3808e29 100644 --- a/docs/src/about.rst +++ b/docs/src/about.rst @@ -16,17 +16,16 @@ Naturally, I set out to reinvent the wheel. Our `2010 LREC publication `_). +Later versions of gensim improved this efficiency and scalability tremendously. In fact, +I made algorithmic scalability of distributional semantics the topic of my `PhD thesis `_. By now, gensim is---to my knowledge---the most robust, efficient and hassle-free piece of software to realize unsupervised semantic modelling from plain text. It stands in contrast to brittle homework-assignment-implementations that do not scale on one hand, -and robust java-esque projects that do scale, but only if you're willing to sacrifice -several weeks of your, your technician's as well as your local scientist's time just to run "hello world". +and robust java-esque projects that take forever just to run "hello world". -In 2011, I started using `Github `_ for source code hosting, -and the gensim website moved from my university hosting to its present domain. +In 2011, I started using `Github `_ for source code hosting +and the gensim website moved to its present domain. In 2013, gensim got its current logo and website design. Licensing @@ -37,7 +36,7 @@ This means that it's free for both personal and commercial use, but if you make modification to gensim that you distribute to other people, you have to disclose the source code of these modifications. -Apart form that, you are free to redistribute gensim in any way you like, though you're +Apart from that, you are free to redistribute gensim in any way you like, though you're not allowed to modify its license (doh!). My intent here is, of course, to get more help and community involvement with the development of gensim. diff --git a/docs/src/apiref.rst b/docs/src/apiref.rst index 14ed4a8f7d..57a572f169 100644 --- a/docs/src/apiref.rst +++ b/docs/src/apiref.rst @@ -31,6 +31,7 @@ Modules: models/lsi_worker models/lda_dispatcher models/lda_worker + models/word2vec similarities/docsim similarities/simserver diff --git a/docs/src/conf.py b/docs/src/conf.py index 271d804f2c..0bce0ac24c 100644 --- a/docs/src/conf.py +++ b/docs/src/conf.py @@ -20,6 +20,8 @@ # -- General configuration ----------------------------------------------------- +html_theme = 'gensim_theme' + # Add any Sphinx extension module names here, as strings. They can be extensions # coming with Sphinx (named 'sphinx.ext.*') or your custom ones. extensions = ['sphinx.ext.autodoc'] @@ -35,20 +37,24 @@ #source_encoding = 'utf-8' # The master toctree document. -master_doc = 'index' +master_doc = 'indextoc' + +# Additional templates that should be rendered to pages, maps page names to +# template names. +html_additional_pages = {'index': './_templates/indexcontent.html'} # General information about the project. project = u'gensim' -copyright = u'2009-2012, Radim Řehůřek ' +copyright = u'2009-2013, Radim Řehůřek ' # The version info for the project you're documenting, acts as replacement for # |version| and |release|, also used in various other places throughout the # built documents. # # The short X.Y version. -version = '0.8.6' +version = '0.8.7' # The full version, including alpha/beta/rc tags. -release = '0.8.6' +release = '0.8.7' # The language for content autogenerated by Sphinx. Refer to documentation # for a list of supported languages. @@ -92,8 +98,7 @@ # The theme to use for HTML and HTML Help pages. Major themes that come with # Sphinx are currently 'default' and 'sphinxdoc'. -html_theme = 'default' -#html_theme = 'agogo' +#html_theme = 'default' # Theme options are theme-specific and customize the look and feel of a theme # further. For a list of options available for each theme, see the @@ -118,7 +123,7 @@ # Add any paths that contain custom themes here, relative to this directory. -#html_theme_path = [] +html_theme_path = ['.'] # The name for this set of Sphinx documents. If None, it defaults to # " v documentation". @@ -134,7 +139,7 @@ # The name of an image file (within the static path) to use as favicon of the # docs. This file should be a Windows icon file (.ico) being 16x16 or 32x32 # pixels large. -#html_favicon = None +html_favicon = 'favicon.ico' # Add any paths that contain custom static files (such as style sheets) here, # relative to this directory. They are copied after the builtin static files, @@ -150,13 +155,9 @@ #html_use_smartypants = True # Custom sidebar templates, maps document names to template names. -html_sidebars = {'index': ['download.html', 'globaltoc.html', 'searchbox.html', 'indexsidebar.html']} +html_sidebars = {} #{'index': ['download.html', 'globaltoc.html', 'searchbox.html', 'indexsidebar.html']} #html_sidebars = {'index': ['globaltoc.html', 'searchbox.html']} -# Additional templates that should be rendered to pages, maps page names to -# template names. -#html_additional_pages = {'index': 'index.html'} - # If false, no module index is generated. #html_use_modindex = True @@ -169,6 +170,8 @@ # If true, links to the reST sources are added to the pages. html_show_sourcelink = False +html_domain_indices = False + # If true, an OpenSearch description file will be output, and all pages will # contain a tag referring to it. The value of this option must be the # base URL from which the finished HTML is served. diff --git a/docs/src/dist_lsi.rst b/docs/src/dist_lsi.rst index 3cf8c5f065..ab14ed761b 100644 --- a/docs/src/dist_lsi.rst +++ b/docs/src/dist_lsi.rst @@ -13,11 +13,13 @@ _______________________ We will show how to run distributed Latent Semantic Analysis by means of an example. Let's say we have 5 computers at our disposal, all on the same network segment (=reachable -by network broadcast). To start with, install `gensim` and `Pyro` on each computer with:: +by network broadcast). To start with, install `gensim` and set up `Pyro` on each computer with:: $ sudo easy_install gensim[distributed] + $ export PYRO_SERIALIZERS_ACCEPTED=pickle + $ export PYRO_SERIALIZER=pickle -and run Pyro’s name server on exactly one of the machines (doesn’t matter which one):: +Then run Pyro’s name server on exactly one of the machines (doesn’t matter which one):: $ python -m Pyro4.naming -n 0.0.0.0 & diff --git a/docs/src/gensim_theme/domainindex.html b/docs/src/gensim_theme/domainindex.html new file mode 100644 index 0000000000..947a01ea8e --- /dev/null +++ b/docs/src/gensim_theme/domainindex.html @@ -0,0 +1,56 @@ +{# + basic/domainindex.html + ~~~~~~~~~~~~~~~~~~~~~~ + + Template for domain indices (module index, ...). + + :copyright: Copyright 2007-2011 by the Sphinx team, see AUTHORS. + :license: BSD, see LICENSE for details. +#} +{% extends "layout.html" %} +{% set title = indextitle %} +{% block extrahead %} +{{ super() }} +{% if not embedded and collapse_index %} + +{% endif %} +{% endblock %} +{% block body %} + + {%- set groupid = idgen() %} + +

    {{ indextitle }}

    + +
    + {%- for (letter, entries) in content %} + {{ letter }} + {%- if not loop.last %} | {% endif %} + {%- endfor %} +
    + +
    + {%- for letter, entries in content %} + + + {%- for (name, grouptype, page, anchor, extra, qualifier, description) + in entries %} + + + + {%- endfor %} + {%- endfor %} +
     
    + {{ letter }}
    {% if grouptype == 1 -%} + + {%- endif %}{% if grouptype == 2 %}   {% endif %} + {% if page %}{% endif -%} + {{ name|e }} + {%- if page %}{% endif %} + {%- if extra %} ({{ extra|e }}){% endif -%} + {% if qualifier %}{{ qualifier|e }}:{% endif %} + {{ description|e }}
    + +{% endblock %} diff --git a/docs/src/gensim_theme/genindex.html b/docs/src/gensim_theme/genindex.html new file mode 100644 index 0000000000..7bc002b6c1 --- /dev/null +++ b/docs/src/gensim_theme/genindex.html @@ -0,0 +1,77 @@ +{# + basic/genindex.html + ~~~~~~~~~~~~~~~~~~~ + + Template for an "all-in-one" index. + + :copyright: Copyright 2007-2011 by the Sphinx team, see AUTHORS. + :license: BSD, see LICENSE for details. +#} +{% macro indexentries(firstname, links) %} +
    + {%- if links -%} + + {%- if links[0][0] %}{% endif -%} + {{ firstname|e }} + {%- if links[0][0] %}{% endif -%} + + + {%- for ismain, link in links[1:] -%} + , {% if ismain %}{% endif -%} + [{{ loop.index }}] + {%- if ismain %}{% endif -%} + + {%- endfor %} + {%- else %} + {{ firstname|e }} + {%- endif %} +
    +{% endmacro %} + +{% extends "layout.html" %} +{% set title = _('Index') %} +{% block body %} + +

    {{ _('Index') }}

    + +
    + {% for key, dummy in genindexentries -%} + {{ key }} + {% if not loop.last %}| {% endif %} + {%- endfor %} +
    + +{%- for key, entries in genindexentries %} +

    {{ key }}

    + + {%- for column in entries|slice(2) if column %} + + {%- endfor %} +
    + {%- for entryname, (links, subitems) in column %} + {{ indexentries(entryname, links) }} + {%- if subitems %} +
    + {%- for subentryname, subentrylinks in subitems %} + {{ indexentries(subentryname, subentrylinks) }} + {%- endfor %} +
    + {%- endif -%} + {%- endfor %} +
    +{% endfor %} + +{% endblock %} + +{% block sidebarrel %} +{% if split_index %} +

    {{ _('Index') }}

    +

    {% for key, dummy in genindexentries -%} + {{ key }} + {% if not loop.last %}| {% endif %} + {%- endfor %}

    + +

    {{ _('Full index on one page') }}

    +{% endif %} + {{ super() }} +{% endblock %} diff --git a/docs/src/gensim_theme/layout.html b/docs/src/gensim_theme/layout.html new file mode 100644 index 0000000000..6ffdb20613 --- /dev/null +++ b/docs/src/gensim_theme/layout.html @@ -0,0 +1,218 @@ +{%- block doctype -%} + +{%- endblock %} + +{% set script_files = ["_static/js/jquery-1.9.1.min.js", "_static/js/jquery.qtip.min.js", "_static/js/jquery-migrate-1.1.1.min.js", "_static/js/jquery.anythingslider.min.js"] %} +{% set css_files = css_files + ["_static/css/jquery.qtip.min.css", "_static/css/anythingslider.css"] %} +{%- set url_root = pathto('', 1) %} + +{%- macro script() %} + + + {%- for scriptfile in script_files %} + + {%- endfor %} +{%- endmacro %} + +{%- macro css() %} + + {%- for cssfile in css_files %} + + {%- endfor %} + +{%- endmacro %} + + + + + + + + + {{ metatags }} + {%- block htmltitle %} + gensim: {{ title|striptags|e }} + {%- endblock %} + + {{ css() }} + + + + + + + + +
    + {% block header %} +
    +
    +

    + gensim logo +

    +
    + +
    +
    +
    gensim
    +
    gensim tagline
    +
    +
    +
    +
    +
    + {% endblock %} + + {% block navbar %} + + {% endblock %} + +
    +
    + + {{ script() }} + + {% block content %} +
    + {% block document %} +
    +
    + {{ title|striptags|e }} +
    +
    +
    +
    + {% block body %} {% endblock %} +
    +
    + {% endblock %} + +
    +
    + {% endblock %} + + {% block footer %} + + {% endblock %} + + + + + diff --git a/docs/src/gensim_theme/page.html b/docs/src/gensim_theme/page.html new file mode 100644 index 0000000000..17a9301653 --- /dev/null +++ b/docs/src/gensim_theme/page.html @@ -0,0 +1,4 @@ +{% extends "layout.html" %} +{% block body %} + {{ body }} +{% endblock %} diff --git a/docs/src/gensim_theme/search.html b/docs/src/gensim_theme/search.html new file mode 100644 index 0000000000..4cdc6935c0 --- /dev/null +++ b/docs/src/gensim_theme/search.html @@ -0,0 +1,56 @@ +{# + basic/search.html + ~~~~~~~~~~~~~~~~~ + + Template for the search page. + + :copyright: Copyright 2007-2011 by the Sphinx team, see AUTHORS. + :license: BSD, see LICENSE for details. +#} +{% extends "layout.html" %} +{% set title = _('Search') %} +{% set script_files = script_files + ['_static/searchtools.js'] %} +{% block extrahead %} + + {{ super() }} +{% endblock %} +{% block body %} +

    {{ _('Search') }}

    +
    + +

    + {% trans %}Please activate JavaScript to enable the search + functionality.{% endtrans %} +

    +
    +

    + {% trans %}From here you can search these documents. Enter your search + words into the box below and click "search". Note that the search + function will automatically search for all of the words. Pages + containing fewer words won't appear in the result list.{% endtrans %} +

    +
    + + + +
    + {% if search_performed %} +

    {{ _('Search Results') }}

    + {% if not search_results %} +

    {{ _('Your search did not match any results.') }}

    + {% endif %} + {% endif %} +
    + {% if search_results %} +
      + {% for href, caption, context in search_results %} +
    • {{ caption }} +
      {{ context|e }}
      +
      • + {% endfor %} +
    + {% endif %} +
    +{% endblock %} diff --git a/docs/_static/doctools.js b/docs/src/gensim_theme/static/doctools.js similarity index 100% rename from docs/_static/doctools.js rename to docs/src/gensim_theme/static/doctools.js diff --git a/docs/src/gensim_theme/static/jquery.js b/docs/src/gensim_theme/static/jquery.js new file mode 100644 index 0000000000..198b3ff07d --- /dev/null +++ b/docs/src/gensim_theme/static/jquery.js @@ -0,0 +1,4 @@ +/*! jQuery v1.7.1 jquery.com | jquery.org/license */ +(function(a,b){function cy(a){return f.isWindow(a)?a:a.nodeType===9?a.defaultView||a.parentWindow:!1}function cv(a){if(!ck[a]){var b=c.body,d=f("<"+a+">").appendTo(b),e=d.css("display");d.remove();if(e==="none"||e===""){cl||(cl=c.createElement("iframe"),cl.frameBorder=cl.width=cl.height=0),b.appendChild(cl);if(!cm||!cl.createElement)cm=(cl.contentWindow||cl.contentDocument).document,cm.write((c.compatMode==="CSS1Compat"?"":"")+""),cm.close();d=cm.createElement(a),cm.body.appendChild(d),e=f.css(d,"display"),b.removeChild(cl)}ck[a]=e}return ck[a]}function cu(a,b){var c={};f.each(cq.concat.apply([],cq.slice(0,b)),function(){c[this]=a});return c}function ct(){cr=b}function cs(){setTimeout(ct,0);return cr=f.now()}function cj(){try{return new a.ActiveXObject("Microsoft.XMLHTTP")}catch(b){}}function ci(){try{return new a.XMLHttpRequest}catch(b){}}function cc(a,c){a.dataFilter&&(c=a.dataFilter(c,a.dataType));var d=a.dataTypes,e={},g,h,i=d.length,j,k=d[0],l,m,n,o,p;for(g=1;g0){if(c!=="border")for(;g=0===c})}function S(a){return!a||!a.parentNode||a.parentNode.nodeType===11}function K(){return!0}function J(){return!1}function n(a,b,c){var d=b+"defer",e=b+"queue",g=b+"mark",h=f._data(a,d);h&&(c==="queue"||!f._data(a,e))&&(c==="mark"||!f._data(a,g))&&setTimeout(function(){!f._data(a,e)&&!f._data(a,g)&&(f.removeData(a,d,!0),h.fire())},0)}function m(a){for(var b in a){if(b==="data"&&f.isEmptyObject(a[b]))continue;if(b!=="toJSON")return!1}return!0}function l(a,c,d){if(d===b&&a.nodeType===1){var e="data-"+c.replace(k,"-$1").toLowerCase();d=a.getAttribute(e);if(typeof d=="string"){try{d=d==="true"?!0:d==="false"?!1:d==="null"?null:f.isNumeric(d)?parseFloat(d):j.test(d)?f.parseJSON(d):d}catch(g){}f.data(a,c,d)}else d=b}return d}function h(a){var b=g[a]={},c,d;a=a.split(/\s+/);for(c=0,d=a.length;c)[^>]*$|#([\w\-]*)$)/,j=/\S/,k=/^\s+/,l=/\s+$/,m=/^<(\w+)\s*\/?>(?:<\/\1>)?$/,n=/^[\],:{}\s]*$/,o=/\\(?:["\\\/bfnrt]|u[0-9a-fA-F]{4})/g,p=/"[^"\\\n\r]*"|true|false|null|-?\d+(?:\.\d*)?(?:[eE][+\-]?\d+)?/g,q=/(?:^|:|,)(?:\s*\[)+/g,r=/(webkit)[ \/]([\w.]+)/,s=/(opera)(?:.*version)?[ \/]([\w.]+)/,t=/(msie) ([\w.]+)/,u=/(mozilla)(?:.*? rv:([\w.]+))?/,v=/-([a-z]|[0-9])/ig,w=/^-ms-/,x=function(a,b){return(b+"").toUpperCase()},y=d.userAgent,z,A,B,C=Object.prototype.toString,D=Object.prototype.hasOwnProperty,E=Array.prototype.push,F=Array.prototype.slice,G=String.prototype.trim,H=Array.prototype.indexOf,I={};e.fn=e.prototype={constructor:e,init:function(a,d,f){var g,h,j,k;if(!a)return this;if(a.nodeType){this.context=this[0]=a,this.length=1;return this}if(a==="body"&&!d&&c.body){this.context=c,this[0]=c.body,this.selector=a,this.length=1;return this}if(typeof a=="string"){a.charAt(0)!=="<"||a.charAt(a.length-1)!==">"||a.length<3?g=i.exec(a):g=[null,a,null];if(g&&(g[1]||!d)){if(g[1]){d=d instanceof e?d[0]:d,k=d?d.ownerDocument||d:c,j=m.exec(a),j?e.isPlainObject(d)?(a=[c.createElement(j[1])],e.fn.attr.call(a,d,!0)):a=[k.createElement(j[1])]:(j=e.buildFragment([g[1]],[k]),a=(j.cacheable?e.clone(j.fragment):j.fragment).childNodes);return e.merge(this,a)}h=c.getElementById(g[2]);if(h&&h.parentNode){if(h.id!==g[2])return f.find(a);this.length=1,this[0]=h}this.context=c,this.selector=a;return this}return!d||d.jquery?(d||f).find(a):this.constructor(d).find(a)}if(e.isFunction(a))return f.ready(a);a.selector!==b&&(this.selector=a.selector,this.context=a.context);return e.makeArray(a,this)},selector:"",jquery:"1.7.1",length:0,size:function(){return this.length},toArray:function(){return F.call(this,0)},get:function(a){return a==null?this.toArray():a<0?this[this.length+a]:this[a]},pushStack:function(a,b,c){var d=this.constructor();e.isArray(a)?E.apply(d,a):e.merge(d,a),d.prevObject=this,d.context=this.context,b==="find"?d.selector=this.selector+(this.selector?" ":"")+c:b&&(d.selector=this.selector+"."+b+"("+c+")");return d},each:function(a,b){return e.each(this,a,b)},ready:function(a){e.bindReady(),A.add(a);return this},eq:function(a){a=+a;return a===-1?this.slice(a):this.slice(a,a+1)},first:function(){return this.eq(0)},last:function(){return this.eq(-1)},slice:function(){return this.pushStack(F.apply(this,arguments),"slice",F.call(arguments).join(","))},map:function(a){return this.pushStack(e.map(this,function(b,c){return a.call(b,c,b)}))},end:function(){return this.prevObject||this.constructor(null)},push:E,sort:[].sort,splice:[].splice},e.fn.init.prototype=e.fn,e.extend=e.fn.extend=function(){var a,c,d,f,g,h,i=arguments[0]||{},j=1,k=arguments.length,l=!1;typeof i=="boolean"&&(l=i,i=arguments[1]||{},j=2),typeof i!="object"&&!e.isFunction(i)&&(i={}),k===j&&(i=this,--j);for(;j0)return;A.fireWith(c,[e]),e.fn.trigger&&e(c).trigger("ready").off("ready")}},bindReady:function(){if(!A){A=e.Callbacks("once memory");if(c.readyState==="complete")return setTimeout(e.ready,1);if(c.addEventListener)c.addEventListener("DOMContentLoaded",B,!1),a.addEventListener("load",e.ready,!1);else if(c.attachEvent){c.attachEvent("onreadystatechange",B),a.attachEvent("onload",e.ready);var b=!1;try{b=a.frameElement==null}catch(d){}c.documentElement.doScroll&&b&&J()}}},isFunction:function(a){return e.type(a)==="function"},isArray:Array.isArray||function(a){return e.type(a)==="array"},isWindow:function(a){return a&&typeof a=="object"&&"setInterval"in a},isNumeric:function(a){return!isNaN(parseFloat(a))&&isFinite(a)},type:function(a){return a==null?String(a):I[C.call(a)]||"object"},isPlainObject:function(a){if(!a||e.type(a)!=="object"||a.nodeType||e.isWindow(a))return!1;try{if(a.constructor&&!D.call(a,"constructor")&&!D.call(a.constructor.prototype,"isPrototypeOf"))return!1}catch(c){return!1}var d;for(d in a);return d===b||D.call(a,d)},isEmptyObject:function(a){for(var b in a)return!1;return!0},error:function(a){throw new Error(a)},parseJSON:function(b){if(typeof b!="string"||!b)return null;b=e.trim(b);if(a.JSON&&a.JSON.parse)return a.JSON.parse(b);if(n.test(b.replace(o,"@").replace(p,"]").replace(q,"")))return(new Function("return "+b))();e.error("Invalid JSON: "+b)},parseXML:function(c){var d,f;try{a.DOMParser?(f=new DOMParser,d=f.parseFromString(c,"text/xml")):(d=new ActiveXObject("Microsoft.XMLDOM"),d.async="false",d.loadXML(c))}catch(g){d=b}(!d||!d.documentElement||d.getElementsByTagName("parsererror").length)&&e.error("Invalid XML: "+c);return d},noop:function(){},globalEval:function(b){b&&j.test(b)&&(a.execScript||function(b){a.eval.call(a,b)})(b)},camelCase:function(a){return a.replace(w,"ms-").replace(v,x)},nodeName:function(a,b){return a.nodeName&&a.nodeName.toUpperCase()===b.toUpperCase()},each:function(a,c,d){var f,g=0,h=a.length,i=h===b||e.isFunction(a);if(d){if(i){for(f in a)if(c.apply(a[f],d)===!1)break}else for(;g0&&a[0]&&a[j-1]||j===0||e.isArray(a));if(k)for(;i1?i.call(arguments,0):b,j.notifyWith(k,e)}}function l(a){return function(c){b[a]=arguments.length>1?i.call(arguments,0):c,--g||j.resolveWith(j,b)}}var b=i.call(arguments,0),c=0,d=b.length,e=Array(d),g=d,h=d,j=d<=1&&a&&f.isFunction(a.promise)?a:f.Deferred(),k=j.promise();if(d>1){for(;c
    a",d=q.getElementsByTagName("*"),e=q.getElementsByTagName("a")[0];if(!d||!d.length||!e)return{};g=c.createElement("select"),h=g.appendChild(c.createElement("option")),i=q.getElementsByTagName("input")[0],b={leadingWhitespace:q.firstChild.nodeType===3,tbody:!q.getElementsByTagName("tbody").length,htmlSerialize:!!q.getElementsByTagName("link").length,style:/top/.test(e.getAttribute("style")),hrefNormalized:e.getAttribute("href")==="/a",opacity:/^0.55/.test(e.style.opacity),cssFloat:!!e.style.cssFloat,checkOn:i.value==="on",optSelected:h.selected,getSetAttribute:q.className!=="t",enctype:!!c.createElement("form").enctype,html5Clone:c.createElement("nav").cloneNode(!0).outerHTML!=="<:nav>",submitBubbles:!0,changeBubbles:!0,focusinBubbles:!1,deleteExpando:!0,noCloneEvent:!0,inlineBlockNeedsLayout:!1,shrinkWrapBlocks:!1,reliableMarginRight:!0},i.checked=!0,b.noCloneChecked=i.cloneNode(!0).checked,g.disabled=!0,b.optDisabled=!h.disabled;try{delete q.test}catch(s){b.deleteExpando=!1}!q.addEventListener&&q.attachEvent&&q.fireEvent&&(q.attachEvent("onclick",function(){b.noCloneEvent=!1}),q.cloneNode(!0).fireEvent("onclick")),i=c.createElement("input"),i.value="t",i.setAttribute("type","radio"),b.radioValue=i.value==="t",i.setAttribute("checked","checked"),q.appendChild(i),k=c.createDocumentFragment(),k.appendChild(q.lastChild),b.checkClone=k.cloneNode(!0).cloneNode(!0).lastChild.checked,b.appendChecked=i.checked,k.removeChild(i),k.appendChild(q),q.innerHTML="",a.getComputedStyle&&(j=c.createElement("div"),j.style.width="0",j.style.marginRight="0",q.style.width="2px",q.appendChild(j),b.reliableMarginRight=(parseInt((a.getComputedStyle(j,null)||{marginRight:0}).marginRight,10)||0)===0);if(q.attachEvent)for(o in{submit:1,change:1,focusin:1})n="on"+o,p=n in q,p||(q.setAttribute(n,"return;"),p=typeof q[n]=="function"),b[o+"Bubbles"]=p;k.removeChild(q),k=g=h=j=q=i=null,f(function(){var a,d,e,g,h,i,j,k,m,n,o,r=c.getElementsByTagName("body")[0];!r||(j=1,k="position:absolute;top:0;left:0;width:1px;height:1px;margin:0;",m="visibility:hidden;border:0;",n="style='"+k+"border:5px solid #000;padding:0;'",o="
    "+""+"
    ",a=c.createElement("div"),a.style.cssText=m+"width:0;height:0;position:static;top:0;margin-top:"+j+"px",r.insertBefore(a,r.firstChild),q=c.createElement("div"),a.appendChild(q),q.innerHTML="
    t
    ",l=q.getElementsByTagName("td"),p=l[0].offsetHeight===0,l[0].style.display="",l[1].style.display="none",b.reliableHiddenOffsets=p&&l[0].offsetHeight===0,q.innerHTML="",q.style.width=q.style.paddingLeft="1px",f.boxModel=b.boxModel=q.offsetWidth===2,typeof q.style.zoom!="undefined"&&(q.style.display="inline",q.style.zoom=1,b.inlineBlockNeedsLayout=q.offsetWidth===2,q.style.display="",q.innerHTML="
    ",b.shrinkWrapBlocks=q.offsetWidth!==2),q.style.cssText=k+m,q.innerHTML=o,d=q.firstChild,e=d.firstChild,h=d.nextSibling.firstChild.firstChild,i={doesNotAddBorder:e.offsetTop!==5,doesAddBorderForTableAndCells:h.offsetTop===5},e.style.position="fixed",e.style.top="20px",i.fixedPosition=e.offsetTop===20||e.offsetTop===15,e.style.position=e.style.top="",d.style.overflow="hidden",d.style.position="relative",i.subtractsBorderForOverflowNotVisible=e.offsetTop===-5,i.doesNotIncludeMarginInBodyOffset=r.offsetTop!==j,r.removeChild(a),q=a=null,f.extend(b,i))});return b}();var j=/^(?:\{.*\}|\[.*\])$/,k=/([A-Z])/g;f.extend({cache:{},uuid:0,expando:"jQuery"+(f.fn.jquery+Math.random()).replace(/\D/g,""),noData:{embed:!0,object:"clsid:D27CDB6E-AE6D-11cf-96B8-444553540000",applet:!0},hasData:function(a){a=a.nodeType?f.cache[a[f.expando]]:a[f.expando];return!!a&&!m(a)},data:function(a,c,d,e){if(!!f.acceptData(a)){var g,h,i,j=f.expando,k=typeof c=="string",l=a.nodeType,m=l?f.cache:a,n=l?a[j]:a[j]&&j,o=c==="events";if((!n||!m[n]||!o&&!e&&!m[n].data)&&k&&d===b)return;n||(l?a[j]=n=++f.uuid:n=j),m[n]||(m[n]={},l||(m[n].toJSON=f.noop));if(typeof c=="object"||typeof c=="function")e?m[n]=f.extend(m[n],c):m[n].data=f.extend(m[n].data,c);g=h=m[n],e||(h.data||(h.data={}),h=h.data),d!==b&&(h[f.camelCase(c)]=d);if(o&&!h[c])return g.events;k?(i=h[c],i==null&&(i=h[f.camelCase(c)])):i=h;return i}},removeData:function(a,b,c){if(!!f.acceptData(a)){var d,e,g,h=f.expando,i=a.nodeType,j=i?f.cache:a,k=i?a[h]:h;if(!j[k])return;if(b){d=c?j[k]:j[k].data;if(d){f.isArray(b)||(b in d?b=[b]:(b=f.camelCase(b),b in d?b=[b]:b=b.split(" ")));for(e=0,g=b.length;e-1)return!0;return!1},val:function(a){var c,d,e,g=this[0];{if(!!arguments.length){e=f.isFunction(a);return this.each(function(d){var g=f(this),h;if(this.nodeType===1){e?h=a.call(this,d,g.val()):h=a,h==null?h="":typeof h=="number"?h+="":f.isArray(h)&&(h=f.map(h,function(a){return a==null?"":a+""})),c=f.valHooks[this.nodeName.toLowerCase()]||f.valHooks[this.type];if(!c||!("set"in c)||c.set(this,h,"value")===b)this.value=h}})}if(g){c=f.valHooks[g.nodeName.toLowerCase()]||f.valHooks[g.type];if(c&&"get"in c&&(d=c.get(g,"value"))!==b)return d;d=g.value;return typeof d=="string"?d.replace(q,""):d==null?"":d}}}}),f.extend({valHooks:{option:{get:function(a){var b=a.attributes.value;return!b||b.specified?a.value:a.text}},select:{get:function(a){var b,c,d,e,g=a.selectedIndex,h=[],i=a.options,j=a.type==="select-one";if(g<0)return null;c=j?g:0,d=j?g+1:i.length;for(;c=0}),c.length||(a.selectedIndex=-1);return c}}},attrFn:{val:!0,css:!0,html:!0,text:!0,data:!0,width:!0,height:!0,offset:!0},attr:function(a,c,d,e){var g,h,i,j=a.nodeType;if(!!a&&j!==3&&j!==8&&j!==2){if(e&&c in f.attrFn)return f(a)[c](d);if(typeof a.getAttribute=="undefined")return f.prop(a,c,d);i=j!==1||!f.isXMLDoc(a),i&&(c=c.toLowerCase(),h=f.attrHooks[c]||(u.test(c)?x:w));if(d!==b){if(d===null){f.removeAttr(a,c);return}if(h&&"set"in h&&i&&(g=h.set(a,d,c))!==b)return g;a.setAttribute(c,""+d);return d}if(h&&"get"in h&&i&&(g=h.get(a,c))!==null)return g;g=a.getAttribute(c);return g===null?b:g}},removeAttr:function(a,b){var c,d,e,g,h=0;if(b&&a.nodeType===1){d=b.toLowerCase().split(p),g=d.length;for(;h=0}})});var z=/^(?:textarea|input|select)$/i,A=/^([^\.]*)?(?:\.(.+))?$/,B=/\bhover(\.\S+)?\b/,C=/^key/,D=/^(?:mouse|contextmenu)|click/,E=/^(?:focusinfocus|focusoutblur)$/,F=/^(\w*)(?:#([\w\-]+))?(?:\.([\w\-]+))?$/,G=function(a){var b=F.exec(a);b&&(b[1]=(b[1]||"").toLowerCase(),b[3]=b[3]&&new RegExp("(?:^|\\s)"+b[3]+"(?:\\s|$)"));return b},H=function(a,b){var c=a.attributes||{};return(!b[1]||a.nodeName.toLowerCase()===b[1])&&(!b[2]||(c.id||{}).value===b[2])&&(!b[3]||b[3].test((c["class"]||{}).value))},I=function(a){return f.event.special.hover?a:a.replace(B,"mouseenter$1 mouseleave$1")}; +f.event={add:function(a,c,d,e,g){var h,i,j,k,l,m,n,o,p,q,r,s;if(!(a.nodeType===3||a.nodeType===8||!c||!d||!(h=f._data(a)))){d.handler&&(p=d,d=p.handler),d.guid||(d.guid=f.guid++),j=h.events,j||(h.events=j={}),i=h.handle,i||(h.handle=i=function(a){return typeof f!="undefined"&&(!a||f.event.triggered!==a.type)?f.event.dispatch.apply(i.elem,arguments):b},i.elem=a),c=f.trim(I(c)).split(" ");for(k=0;k=0&&(h=h.slice(0,-1),k=!0),h.indexOf(".")>=0&&(i=h.split("."),h=i.shift(),i.sort());if((!e||f.event.customEvent[h])&&!f.event.global[h])return;c=typeof c=="object"?c[f.expando]?c:new f.Event(h,c):new f.Event(h),c.type=h,c.isTrigger=!0,c.exclusive=k,c.namespace=i.join("."),c.namespace_re=c.namespace?new RegExp("(^|\\.)"+i.join("\\.(?:.*\\.)?")+"(\\.|$)"):null,o=h.indexOf(":")<0?"on"+h:"";if(!e){j=f.cache;for(l in j)j[l].events&&j[l].events[h]&&f.event.trigger(c,d,j[l].handle.elem,!0);return}c.result=b,c.target||(c.target=e),d=d!=null?f.makeArray(d):[],d.unshift(c),p=f.event.special[h]||{};if(p.trigger&&p.trigger.apply(e,d)===!1)return;r=[[e,p.bindType||h]];if(!g&&!p.noBubble&&!f.isWindow(e)){s=p.delegateType||h,m=E.test(s+h)?e:e.parentNode,n=null;for(;m;m=m.parentNode)r.push([m,s]),n=m;n&&n===e.ownerDocument&&r.push([n.defaultView||n.parentWindow||a,s])}for(l=0;le&&i.push({elem:this,matches:d.slice(e)});for(j=0;j0?this.on(b,null,a,c):this.trigger(b)},f.attrFn&&(f.attrFn[b]=!0),C.test(b)&&(f.event.fixHooks[b]=f.event.keyHooks),D.test(b)&&(f.event.fixHooks[b]=f.event.mouseHooks)}),function(){function x(a,b,c,e,f,g){for(var h=0,i=e.length;h0){k=j;break}}j=j[a]}e[h]=k}}}function w(a,b,c,e,f,g){for(var h=0,i=e.length;h+~,(\[\\]+)+|[>+~])(\s*,\s*)?((?:.|\r|\n)*)/g,d="sizcache"+(Math.random()+"").replace(".",""),e=0,g=Object.prototype.toString,h=!1,i=!0,j=/\\/g,k=/\r\n/g,l=/\W/;[0,0].sort(function(){i=!1;return 0});var m=function(b,d,e,f){e=e||[],d=d||c;var h=d;if(d.nodeType!==1&&d.nodeType!==9)return[];if(!b||typeof b!="string")return e;var i,j,k,l,n,q,r,t,u=!0,v=m.isXML(d),w=[],x=b;do{a.exec(""),i=a.exec(x);if(i){x=i[3],w.push(i[1]);if(i[2]){l=i[3];break}}}while(i);if(w.length>1&&p.exec(b))if(w.length===2&&o.relative[w[0]])j=y(w[0]+w[1],d,f);else{j=o.relative[w[0]]?[d]:m(w.shift(),d);while(w.length)b=w.shift(),o.relative[b]&&(b+=w.shift()),j=y(b,j,f)}else{!f&&w.length>1&&d.nodeType===9&&!v&&o.match.ID.test(w[0])&&!o.match.ID.test(w[w.length-1])&&(n=m.find(w.shift(),d,v),d=n.expr?m.filter(n.expr,n.set)[0]:n.set[0]);if(d){n=f?{expr:w.pop(),set:s(f)}:m.find(w.pop(),w.length===1&&(w[0]==="~"||w[0]==="+")&&d.parentNode?d.parentNode:d,v),j=n.expr?m.filter(n.expr,n.set):n.set,w.length>0?k=s(j):u=!1;while(w.length)q=w.pop(),r=q,o.relative[q]?r=w.pop():q="",r==null&&(r=d),o.relative[q](k,r,v)}else k=w=[]}k||(k=j),k||m.error(q||b);if(g.call(k)==="[object Array]")if(!u)e.push.apply(e,k);else if(d&&d.nodeType===1)for(t=0;k[t]!=null;t++)k[t]&&(k[t]===!0||k[t].nodeType===1&&m.contains(d,k[t]))&&e.push(j[t]);else for(t=0;k[t]!=null;t++)k[t]&&k[t].nodeType===1&&e.push(j[t]);else s(k,e);l&&(m(l,h,e,f),m.uniqueSort(e));return e};m.uniqueSort=function(a){if(u){h=i,a.sort(u);if(h)for(var b=1;b0},m.find=function(a,b,c){var d,e,f,g,h,i;if(!a)return[];for(e=0,f=o.order.length;e":function(a,b){var c,d=typeof b=="string",e=0,f=a.length;if(d&&!l.test(b)){b=b.toLowerCase();for(;e=0)?c||d.push(h):c&&(b[g]=!1));return!1},ID:function(a){return a[1].replace(j,"")},TAG:function(a,b){return a[1].replace(j,"").toLowerCase()},CHILD:function(a){if(a[1]==="nth"){a[2]||m.error(a[0]),a[2]=a[2].replace(/^\+|\s*/g,"");var b=/(-?)(\d*)(?:n([+\-]?\d*))?/.exec(a[2]==="even"&&"2n"||a[2]==="odd"&&"2n+1"||!/\D/.test(a[2])&&"0n+"+a[2]||a[2]);a[2]=b[1]+(b[2]||1)-0,a[3]=b[3]-0}else a[2]&&m.error(a[0]);a[0]=e++;return a},ATTR:function(a,b,c,d,e,f){var g=a[1]=a[1].replace(j,"");!f&&o.attrMap[g]&&(a[1]=o.attrMap[g]),a[4]=(a[4]||a[5]||"").replace(j,""),a[2]==="~="&&(a[4]=" "+a[4]+" ");return a},PSEUDO:function(b,c,d,e,f){if(b[1]==="not")if((a.exec(b[3])||"").length>1||/^\w/.test(b[3]))b[3]=m(b[3],null,null,c);else{var g=m.filter(b[3],c,d,!0^f);d||e.push.apply(e,g);return!1}else if(o.match.POS.test(b[0])||o.match.CHILD.test(b[0]))return!0;return b},POS:function(a){a.unshift(!0);return a}},filters:{enabled:function(a){return a.disabled===!1&&a.type!=="hidden"},disabled:function(a){return a.disabled===!0},checked:function(a){return a.checked===!0},selected:function(a){a.parentNode&&a.parentNode.selectedIndex;return a.selected===!0},parent:function(a){return!!a.firstChild},empty:function(a){return!a.firstChild},has:function(a,b,c){return!!m(c[3],a).length},header:function(a){return/h\d/i.test(a.nodeName)},text:function(a){var b=a.getAttribute("type"),c=a.type;return a.nodeName.toLowerCase()==="input"&&"text"===c&&(b===c||b===null)},radio:function(a){return a.nodeName.toLowerCase()==="input"&&"radio"===a.type},checkbox:function(a){return a.nodeName.toLowerCase()==="input"&&"checkbox"===a.type},file:function(a){return a.nodeName.toLowerCase()==="input"&&"file"===a.type},password:function(a){return a.nodeName.toLowerCase()==="input"&&"password"===a.type},submit:function(a){var b=a.nodeName.toLowerCase();return(b==="input"||b==="button")&&"submit"===a.type},image:function(a){return a.nodeName.toLowerCase()==="input"&&"image"===a.type},reset:function(a){var b=a.nodeName.toLowerCase();return(b==="input"||b==="button")&&"reset"===a.type},button:function(a){var b=a.nodeName.toLowerCase();return b==="input"&&"button"===a.type||b==="button"},input:function(a){return/input|select|textarea|button/i.test(a.nodeName)},focus:function(a){return a===a.ownerDocument.activeElement}},setFilters:{first:function(a,b){return b===0},last:function(a,b,c,d){return b===d.length-1},even:function(a,b){return b%2===0},odd:function(a,b){return b%2===1},lt:function(a,b,c){return bc[3]-0},nth:function(a,b,c){return c[3]-0===b},eq:function(a,b,c){return c[3]-0===b}},filter:{PSEUDO:function(a,b,c,d){var e=b[1],f=o.filters[e];if(f)return f(a,c,b,d);if(e==="contains")return(a.textContent||a.innerText||n([a])||"").indexOf(b[3])>=0;if(e==="not"){var g=b[3];for(var h=0,i=g.length;h=0}},ID:function(a,b){return a.nodeType===1&&a.getAttribute("id")===b},TAG:function(a,b){return b==="*"&&a.nodeType===1||!!a.nodeName&&a.nodeName.toLowerCase()===b},CLASS:function(a,b){return(" "+(a.className||a.getAttribute("class"))+" ").indexOf(b)>-1},ATTR:function(a,b){var c=b[1],d=m.attr?m.attr(a,c):o.attrHandle[c]?o.attrHandle[c](a):a[c]!=null?a[c]:a.getAttribute(c),e=d+"",f=b[2],g=b[4];return d==null?f==="!=":!f&&m.attr?d!=null:f==="="?e===g:f==="*="?e.indexOf(g)>=0:f==="~="?(" "+e+" ").indexOf(g)>=0:g?f==="!="?e!==g:f==="^="?e.indexOf(g)===0:f==="$="?e.substr(e.length-g.length)===g:f==="|="?e===g||e.substr(0,g.length+1)===g+"-":!1:e&&d!==!1},POS:function(a,b,c,d){var e=b[2],f=o.setFilters[e];if(f)return f(a,c,b,d)}}},p=o.match.POS,q=function(a,b){return"\\"+(b-0+1)};for(var r in o.match)o.match[r]=new RegExp(o.match[r].source+/(?![^\[]*\])(?![^\(]*\))/.source),o.leftMatch[r]=new RegExp(/(^(?:.|\r|\n)*?)/.source+o.match[r].source.replace(/\\(\d+)/g,q));var s=function(a,b){a=Array.prototype.slice.call(a,0);if(b){b.push.apply(b,a);return b}return a};try{Array.prototype.slice.call(c.documentElement.childNodes,0)[0].nodeType}catch(t){s=function(a,b){var c=0,d=b||[];if(g.call(a)==="[object Array]")Array.prototype.push.apply(d,a);else if(typeof a.length=="number")for(var e=a.length;c",e.insertBefore(a,e.firstChild),c.getElementById(d)&&(o.find.ID=function(a,c,d){if(typeof c.getElementById!="undefined"&&!d){var e=c.getElementById(a[1]);return e?e.id===a[1]||typeof e.getAttributeNode!="undefined"&&e.getAttributeNode("id").nodeValue===a[1]?[e]:b:[]}},o.filter.ID=function(a,b){var c=typeof a.getAttributeNode!="undefined"&&a.getAttributeNode("id");return a.nodeType===1&&c&&c.nodeValue===b}),e.removeChild(a),e=a=null}(),function(){var a=c.createElement("div");a.appendChild(c.createComment("")),a.getElementsByTagName("*").length>0&&(o.find.TAG=function(a,b){var c=b.getElementsByTagName(a[1]);if(a[1]==="*"){var d=[];for(var e=0;c[e];e++)c[e].nodeType===1&&d.push(c[e]);c=d}return c}),a.innerHTML="",a.firstChild&&typeof a.firstChild.getAttribute!="undefined"&&a.firstChild.getAttribute("href")!=="#"&&(o.attrHandle.href=function(a){return a.getAttribute("href",2)}),a=null}(),c.querySelectorAll&&function(){var a=m,b=c.createElement("div"),d="__sizzle__";b.innerHTML="

    ";if(!b.querySelectorAll||b.querySelectorAll(".TEST").length!==0){m=function(b,e,f,g){e=e||c;if(!g&&!m.isXML(e)){var h=/^(\w+$)|^\.([\w\-]+$)|^#([\w\-]+$)/.exec(b);if(h&&(e.nodeType===1||e.nodeType===9)){if(h[1])return s(e.getElementsByTagName(b),f);if(h[2]&&o.find.CLASS&&e.getElementsByClassName)return s(e.getElementsByClassName(h[2]),f)}if(e.nodeType===9){if(b==="body"&&e.body)return s([e.body],f);if(h&&h[3]){var i=e.getElementById(h[3]);if(!i||!i.parentNode)return s([],f);if(i.id===h[3])return s([i],f)}try{return s(e.querySelectorAll(b),f)}catch(j){}}else if(e.nodeType===1&&e.nodeName.toLowerCase()!=="object"){var k=e,l=e.getAttribute("id"),n=l||d,p=e.parentNode,q=/^\s*[+~]/.test(b);l?n=n.replace(/'/g,"\\$&"):e.setAttribute("id",n),q&&p&&(e=e.parentNode);try{if(!q||p)return s(e.querySelectorAll("[id='"+n+"'] "+b),f)}catch(r){}finally{l||k.removeAttribute("id")}}}return a(b,e,f,g)};for(var e in a)m[e]=a[e];b=null}}(),function(){var a=c.documentElement,b=a.matchesSelector||a.mozMatchesSelector||a.webkitMatchesSelector||a.msMatchesSelector;if(b){var d=!b.call(c.createElement("div"),"div"),e=!1;try{b.call(c.documentElement,"[test!='']:sizzle")}catch(f){e=!0}m.matchesSelector=function(a,c){c=c.replace(/\=\s*([^'"\]]*)\s*\]/g,"='$1']");if(!m.isXML(a))try{if(e||!o.match.PSEUDO.test(c)&&!/!=/.test(c)){var f=b.call(a,c);if(f||!d||a.document&&a.document.nodeType!==11)return f}}catch(g){}return m(c,null,null,[a]).length>0}}}(),function(){var a=c.createElement("div");a.innerHTML="
    ";if(!!a.getElementsByClassName&&a.getElementsByClassName("e").length!==0){a.lastChild.className="e";if(a.getElementsByClassName("e").length===1)return;o.order.splice(1,0,"CLASS"),o.find.CLASS=function(a,b,c){if(typeof b.getElementsByClassName!="undefined"&&!c)return b.getElementsByClassName(a[1])},a=null}}(),c.documentElement.contains?m.contains=function(a,b){return a!==b&&(a.contains?a.contains(b):!0)}:c.documentElement.compareDocumentPosition?m.contains=function(a,b){return!!(a.compareDocumentPosition(b)&16)}:m.contains=function(){return!1},m.isXML=function(a){var b=(a?a.ownerDocument||a:0).documentElement;return b?b.nodeName!=="HTML":!1};var y=function(a,b,c){var d,e=[],f="",g=b.nodeType?[b]:b;while(d=o.match.PSEUDO.exec(a))f+=d[0],a=a.replace(o.match.PSEUDO,"");a=o.relative[a]?a+"*":a;for(var h=0,i=g.length;h0)for(h=g;h=0:f.filter(a,this).length>0:this.filter(a).length>0)},closest:function(a,b){var c=[],d,e,g=this[0];if(f.isArray(a)){var h=1;while(g&&g.ownerDocument&&g!==b){for(d=0;d-1:f.find.matchesSelector(g,a)){c.push(g);break}g=g.parentNode;if(!g||!g.ownerDocument||g===b||g.nodeType===11)break}}c=c.length>1?f.unique(c):c;return this.pushStack(c,"closest",a)},index:function(a){if(!a)return this[0]&&this[0].parentNode?this.prevAll().length:-1;if(typeof a=="string")return f.inArray(this[0],f(a));return f.inArray(a.jquery?a[0]:a,this)},add:function(a,b){var c=typeof a=="string"?f(a,b):f.makeArray(a&&a.nodeType?[a]:a),d=f.merge(this.get(),c);return this.pushStack(S(c[0])||S(d[0])?d:f.unique(d))},andSelf:function(){return this.add(this.prevObject)}}),f.each({parent:function(a){var b=a.parentNode;return b&&b.nodeType!==11?b:null},parents:function(a){return f.dir(a,"parentNode")},parentsUntil:function(a,b,c){return f.dir(a,"parentNode",c)},next:function(a){return f.nth(a,2,"nextSibling")},prev:function(a){return f.nth(a,2,"previousSibling")},nextAll:function(a){return f.dir(a,"nextSibling")},prevAll:function(a){return f.dir(a,"previousSibling")},nextUntil:function(a,b,c){return f.dir(a,"nextSibling",c)},prevUntil:function(a,b,c){return f.dir(a,"previousSibling",c)},siblings:function(a){return f.sibling(a.parentNode.firstChild,a)},children:function(a){return f.sibling(a.firstChild)},contents:function(a){return f.nodeName(a,"iframe")?a.contentDocument||a.contentWindow.document:f.makeArray(a.childNodes)}},function(a,b){f.fn[a]=function(c,d){var e=f.map(this,b,c);L.test(a)||(d=c),d&&typeof d=="string"&&(e=f.filter(d,e)),e=this.length>1&&!R[a]?f.unique(e):e,(this.length>1||N.test(d))&&M.test(a)&&(e=e.reverse());return this.pushStack(e,a,P.call(arguments).join(","))}}),f.extend({filter:function(a,b,c){c&&(a=":not("+a+")");return b.length===1?f.find.matchesSelector(b[0],a)?[b[0]]:[]:f.find.matches(a,b)},dir:function(a,c,d){var e=[],g=a[c];while(g&&g.nodeType!==9&&(d===b||g.nodeType!==1||!f(g).is(d)))g.nodeType===1&&e.push(g),g=g[c];return e},nth:function(a,b,c,d){b=b||1;var e=0;for(;a;a=a[c])if(a.nodeType===1&&++e===b)break;return a},sibling:function(a,b){var c=[];for(;a;a=a.nextSibling)a.nodeType===1&&a!==b&&c.push(a);return c}});var V="abbr|article|aside|audio|canvas|datalist|details|figcaption|figure|footer|header|hgroup|mark|meter|nav|output|progress|section|summary|time|video",W=/ jQuery\d+="(?:\d+|null)"/g,X=/^\s+/,Y=/<(?!area|br|col|embed|hr|img|input|link|meta|param)(([\w:]+)[^>]*)\/>/ig,Z=/<([\w:]+)/,$=/",""],legend:[1,"
    ","
    "],thead:[1,"","
    "],tr:[2,"","
    "],td:[3,"","
    "],col:[2,"","
    "],area:[1,"",""],_default:[0,"",""]},bh=U(c);bg.optgroup=bg.option,bg.tbody=bg.tfoot=bg.colgroup=bg.caption=bg.thead,bg.th=bg.td,f.support.htmlSerialize||(bg._default=[1,"div
    ","
    "]),f.fn.extend({text:function(a){if(f.isFunction(a))return this.each(function(b){var c=f(this);c.text(a.call(this,b,c.text()))});if(typeof a!="object"&&a!==b)return this.empty().append((this[0]&&this[0].ownerDocument||c).createTextNode(a));return f.text(this)},wrapAll:function(a){if(f.isFunction(a))return this.each(function(b){f(this).wrapAll(a.call(this,b))});if(this[0]){var b=f(a,this[0].ownerDocument).eq(0).clone(!0);this[0].parentNode&&b.insertBefore(this[0]),b.map(function(){var a=this;while(a.firstChild&&a.firstChild.nodeType===1)a=a.firstChild;return a}).append(this)}return this},wrapInner:function(a){if(f.isFunction(a))return this.each(function(b){f(this).wrapInner(a.call(this,b))});return this.each(function(){var b=f(this),c=b.contents();c.length?c.wrapAll(a):b.append(a)})},wrap:function(a){var b=f.isFunction(a);return this.each(function(c){f(this).wrapAll(b?a.call(this,c):a)})},unwrap:function(){return this.parent().each(function(){f.nodeName(this,"body")||f(this).replaceWith(this.childNodes)}).end()},append:function(){return this.domManip(arguments,!0,function(a){this.nodeType===1&&this.appendChild(a)})},prepend:function(){return this.domManip(arguments,!0,function(a){this.nodeType===1&&this.insertBefore(a,this.firstChild)})},before:function(){if(this[0]&&this[0].parentNode)return this.domManip(arguments,!1,function(a){this.parentNode.insertBefore(a,this)});if(arguments.length){var a=f.clean(arguments);a.push.apply(a,this.toArray());return this.pushStack(a,"before",arguments)}},after:function(){if(this[0]&&this[0].parentNode)return this.domManip(arguments,!1,function(a){this.parentNode.insertBefore(a,this.nextSibling)});if(arguments.length){var a=this.pushStack(this,"after",arguments);a.push.apply(a,f.clean(arguments));return a}},remove:function(a,b){for(var c=0,d;(d=this[c])!=null;c++)if(!a||f.filter(a,[d]).length)!b&&d.nodeType===1&&(f.cleanData(d.getElementsByTagName("*")),f.cleanData([d])),d.parentNode&&d.parentNode.removeChild(d);return this},empty:function() +{for(var a=0,b;(b=this[a])!=null;a++){b.nodeType===1&&f.cleanData(b.getElementsByTagName("*"));while(b.firstChild)b.removeChild(b.firstChild)}return this},clone:function(a,b){a=a==null?!1:a,b=b==null?a:b;return this.map(function(){return f.clone(this,a,b)})},html:function(a){if(a===b)return this[0]&&this[0].nodeType===1?this[0].innerHTML.replace(W,""):null;if(typeof a=="string"&&!ba.test(a)&&(f.support.leadingWhitespace||!X.test(a))&&!bg[(Z.exec(a)||["",""])[1].toLowerCase()]){a=a.replace(Y,"<$1>");try{for(var c=0,d=this.length;c1&&l0?this.clone(!0):this).get();f(e[h])[b](j),d=d.concat(j)}return this.pushStack(d,a,e.selector)}}),f.extend({clone:function(a,b,c){var d,e,g,h=f.support.html5Clone||!bc.test("<"+a.nodeName)?a.cloneNode(!0):bo(a);if((!f.support.noCloneEvent||!f.support.noCloneChecked)&&(a.nodeType===1||a.nodeType===11)&&!f.isXMLDoc(a)){bk(a,h),d=bl(a),e=bl(h);for(g=0;d[g];++g)e[g]&&bk(d[g],e[g])}if(b){bj(a,h);if(c){d=bl(a),e=bl(h);for(g=0;d[g];++g)bj(d[g],e[g])}}d=e=null;return h},clean:function(a,b,d,e){var g;b=b||c,typeof b.createElement=="undefined"&&(b=b.ownerDocument||b[0]&&b[0].ownerDocument||c);var h=[],i;for(var j=0,k;(k=a[j])!=null;j++){typeof k=="number"&&(k+="");if(!k)continue;if(typeof k=="string")if(!_.test(k))k=b.createTextNode(k);else{k=k.replace(Y,"<$1>");var l=(Z.exec(k)||["",""])[1].toLowerCase(),m=bg[l]||bg._default,n=m[0],o=b.createElement("div");b===c?bh.appendChild(o):U(b).appendChild(o),o.innerHTML=m[1]+k+m[2];while(n--)o=o.lastChild;if(!f.support.tbody){var p=$.test(k),q=l==="table"&&!p?o.firstChild&&o.firstChild.childNodes:m[1]===""&&!p?o.childNodes:[];for(i=q.length-1;i>=0;--i)f.nodeName(q[i],"tbody")&&!q[i].childNodes.length&&q[i].parentNode.removeChild(q[i])}!f.support.leadingWhitespace&&X.test(k)&&o.insertBefore(b.createTextNode(X.exec(k)[0]),o.firstChild),k=o.childNodes}var r;if(!f.support.appendChecked)if(k[0]&&typeof (r=k.length)=="number")for(i=0;i=0)return b+"px"}}}),f.support.opacity||(f.cssHooks.opacity={get:function(a,b){return br.test((b&&a.currentStyle?a.currentStyle.filter:a.style.filter)||"")?parseFloat(RegExp.$1)/100+"":b?"1":""},set:function(a,b){var c=a.style,d=a.currentStyle,e=f.isNumeric(b)?"alpha(opacity="+b*100+")":"",g=d&&d.filter||c.filter||"";c.zoom=1;if(b>=1&&f.trim(g.replace(bq,""))===""){c.removeAttribute("filter");if(d&&!d.filter)return}c.filter=bq.test(g)?g.replace(bq,e):g+" "+e}}),f(function(){f.support.reliableMarginRight||(f.cssHooks.marginRight={get:function(a,b){var c;f.swap(a,{display:"inline-block"},function(){b?c=bz(a,"margin-right","marginRight"):c=a.style.marginRight});return c}})}),c.defaultView&&c.defaultView.getComputedStyle&&(bA=function(a,b){var c,d,e;b=b.replace(bs,"-$1").toLowerCase(),(d=a.ownerDocument.defaultView)&&(e=d.getComputedStyle(a,null))&&(c=e.getPropertyValue(b),c===""&&!f.contains(a.ownerDocument.documentElement,a)&&(c=f.style(a,b)));return c}),c.documentElement.currentStyle&&(bB=function(a,b){var c,d,e,f=a.currentStyle&&a.currentStyle[b],g=a.style;f===null&&g&&(e=g[b])&&(f=e),!bt.test(f)&&bu.test(f)&&(c=g.left,d=a.runtimeStyle&&a.runtimeStyle.left,d&&(a.runtimeStyle.left=a.currentStyle.left),g.left=b==="fontSize"?"1em":f||0,f=g.pixelLeft+"px",g.left=c,d&&(a.runtimeStyle.left=d));return f===""?"auto":f}),bz=bA||bB,f.expr&&f.expr.filters&&(f.expr.filters.hidden=function(a){var b=a.offsetWidth,c=a.offsetHeight;return b===0&&c===0||!f.support.reliableHiddenOffsets&&(a.style&&a.style.display||f.css(a,"display"))==="none"},f.expr.filters.visible=function(a){return!f.expr.filters.hidden(a)});var bD=/%20/g,bE=/\[\]$/,bF=/\r?\n/g,bG=/#.*$/,bH=/^(.*?):[ \t]*([^\r\n]*)\r?$/mg,bI=/^(?:color|date|datetime|datetime-local|email|hidden|month|number|password|range|search|tel|text|time|url|week)$/i,bJ=/^(?:about|app|app\-storage|.+\-extension|file|res|widget):$/,bK=/^(?:GET|HEAD)$/,bL=/^\/\//,bM=/\?/,bN=/)<[^<]*)*<\/script>/gi,bO=/^(?:select|textarea)/i,bP=/\s+/,bQ=/([?&])_=[^&]*/,bR=/^([\w\+\.\-]+:)(?:\/\/([^\/?#:]*)(?::(\d+))?)?/,bS=f.fn.load,bT={},bU={},bV,bW,bX=["*/"]+["*"];try{bV=e.href}catch(bY){bV=c.createElement("a"),bV.href="",bV=bV.href}bW=bR.exec(bV.toLowerCase())||[],f.fn.extend({load:function(a,c,d){if(typeof a!="string"&&bS)return bS.apply(this,arguments);if(!this.length)return this;var e=a.indexOf(" ");if(e>=0){var g=a.slice(e,a.length);a=a.slice(0,e)}var h="GET";c&&(f.isFunction(c)?(d=c,c=b):typeof c=="object"&&(c=f.param(c,f.ajaxSettings.traditional),h="POST"));var i=this;f.ajax({url:a,type:h,dataType:"html",data:c,complete:function(a,b,c){c=a.responseText,a.isResolved()&&(a.done(function(a){c=a}),i.html(g?f("
    ").append(c.replace(bN,"")).find(g):c)),d&&i.each(d,[c,b,a])}});return this},serialize:function(){return f.param(this.serializeArray())},serializeArray:function(){return this.map(function(){return this.elements?f.makeArray(this.elements):this}).filter(function(){return this.name&&!this.disabled&&(this.checked||bO.test(this.nodeName)||bI.test(this.type))}).map(function(a,b){var c=f(this).val();return c==null?null:f.isArray(c)?f.map(c,function(a,c){return{name:b.name,value:a.replace(bF,"\r\n")}}):{name:b.name,value:c.replace(bF,"\r\n")}}).get()}}),f.each("ajaxStart ajaxStop ajaxComplete ajaxError ajaxSuccess ajaxSend".split(" "),function(a,b){f.fn[b]=function(a){return this.on(b,a)}}),f.each(["get","post"],function(a,c){f[c]=function(a,d,e,g){f.isFunction(d)&&(g=g||e,e=d,d=b);return f.ajax({type:c,url:a,data:d,success:e,dataType:g})}}),f.extend({getScript:function(a,c){return f.get(a,b,c,"script")},getJSON:function(a,b,c){return f.get(a,b,c,"json")},ajaxSetup:function(a,b){b?b_(a,f.ajaxSettings):(b=a,a=f.ajaxSettings),b_(a,b);return a},ajaxSettings:{url:bV,isLocal:bJ.test(bW[1]),global:!0,type:"GET",contentType:"application/x-www-form-urlencoded",processData:!0,async:!0,accepts:{xml:"application/xml, text/xml",html:"text/html",text:"text/plain",json:"application/json, text/javascript","*":bX},contents:{xml:/xml/,html:/html/,json:/json/},responseFields:{xml:"responseXML",text:"responseText"},converters:{"* text":a.String,"text html":!0,"text json":f.parseJSON,"text xml":f.parseXML},flatOptions:{context:!0,url:!0}},ajaxPrefilter:bZ(bT),ajaxTransport:bZ(bU),ajax:function(a,c){function w(a,c,l,m){if(s!==2){s=2,q&&clearTimeout(q),p=b,n=m||"",v.readyState=a>0?4:0;var o,r,u,w=c,x=l?cb(d,v,l):b,y,z;if(a>=200&&a<300||a===304){if(d.ifModified){if(y=v.getResponseHeader("Last-Modified"))f.lastModified[k]=y;if(z=v.getResponseHeader("Etag"))f.etag[k]=z}if(a===304)w="notmodified",o=!0;else try{r=cc(d,x),w="success",o=!0}catch(A){w="parsererror",u=A}}else{u=w;if(!w||a)w="error",a<0&&(a=0)}v.status=a,v.statusText=""+(c||w),o?h.resolveWith(e,[r,w,v]):h.rejectWith(e,[v,w,u]),v.statusCode(j),j=b,t&&g.trigger("ajax"+(o?"Success":"Error"),[v,d,o?r:u]),i.fireWith(e,[v,w]),t&&(g.trigger("ajaxComplete",[v,d]),--f.active||f.event.trigger("ajaxStop"))}}typeof a=="object"&&(c=a,a=b),c=c||{};var d=f.ajaxSetup({},c),e=d.context||d,g=e!==d&&(e.nodeType||e instanceof f)?f(e):f.event,h=f.Deferred(),i=f.Callbacks("once memory"),j=d.statusCode||{},k,l={},m={},n,o,p,q,r,s=0,t,u,v={readyState:0,setRequestHeader:function(a,b){if(!s){var c=a.toLowerCase();a=m[c]=m[c]||a,l[a]=b}return this},getAllResponseHeaders:function(){return s===2?n:null},getResponseHeader:function(a){var c;if(s===2){if(!o){o={};while(c=bH.exec(n))o[c[1].toLowerCase()]=c[2]}c=o[a.toLowerCase()]}return c===b?null:c},overrideMimeType:function(a){s||(d.mimeType=a);return this},abort:function(a){a=a||"abort",p&&p.abort(a),w(0,a);return this}};h.promise(v),v.success=v.done,v.error=v.fail,v.complete=i.add,v.statusCode=function(a){if(a){var b;if(s<2)for(b in a)j[b]=[j[b],a[b]];else b=a[v.status],v.then(b,b)}return this},d.url=((a||d.url)+"").replace(bG,"").replace(bL,bW[1]+"//"),d.dataTypes=f.trim(d.dataType||"*").toLowerCase().split(bP),d.crossDomain==null&&(r=bR.exec(d.url.toLowerCase()),d.crossDomain=!(!r||r[1]==bW[1]&&r[2]==bW[2]&&(r[3]||(r[1]==="http:"?80:443))==(bW[3]||(bW[1]==="http:"?80:443)))),d.data&&d.processData&&typeof d.data!="string"&&(d.data=f.param(d.data,d.traditional)),b$(bT,d,c,v);if(s===2)return!1;t=d.global,d.type=d.type.toUpperCase(),d.hasContent=!bK.test(d.type),t&&f.active++===0&&f.event.trigger("ajaxStart");if(!d.hasContent){d.data&&(d.url+=(bM.test(d.url)?"&":"?")+d.data,delete d.data),k=d.url;if(d.cache===!1){var x=f.now(),y=d.url.replace(bQ,"$1_="+x);d.url=y+(y===d.url?(bM.test(d.url)?"&":"?")+"_="+x:"")}}(d.data&&d.hasContent&&d.contentType!==!1||c.contentType)&&v.setRequestHeader("Content-Type",d.contentType),d.ifModified&&(k=k||d.url,f.lastModified[k]&&v.setRequestHeader("If-Modified-Since",f.lastModified[k]),f.etag[k]&&v.setRequestHeader("If-None-Match",f.etag[k])),v.setRequestHeader("Accept",d.dataTypes[0]&&d.accepts[d.dataTypes[0]]?d.accepts[d.dataTypes[0]]+(d.dataTypes[0]!=="*"?", "+bX+"; q=0.01":""):d.accepts["*"]);for(u in d.headers)v.setRequestHeader(u,d.headers[u]);if(d.beforeSend&&(d.beforeSend.call(e,v,d)===!1||s===2)){v.abort();return!1}for(u in{success:1,error:1,complete:1})v[u](d[u]);p=b$(bU,d,c,v);if(!p)w(-1,"No Transport");else{v.readyState=1,t&&g.trigger("ajaxSend",[v,d]),d.async&&d.timeout>0&&(q=setTimeout(function(){v.abort("timeout")},d.timeout));try{s=1,p.send(l,w)}catch(z){if(s<2)w(-1,z);else throw z}}return v},param:function(a,c){var d=[],e=function(a,b){b=f.isFunction(b)?b():b,d[d.length]=encodeURIComponent(a)+"="+encodeURIComponent(b)};c===b&&(c=f.ajaxSettings.traditional);if(f.isArray(a)||a.jquery&&!f.isPlainObject(a))f.each(a,function(){e(this.name,this.value)});else for(var g in a)ca(g,a[g],c,e);return d.join("&").replace(bD,"+")}}),f.extend({active:0,lastModified:{},etag:{}});var cd=f.now(),ce=/(\=)\?(&|$)|\?\?/i;f.ajaxSetup({jsonp:"callback",jsonpCallback:function(){return f.expando+"_"+cd++}}),f.ajaxPrefilter("json jsonp",function(b,c,d){var e=b.contentType==="application/x-www-form-urlencoded"&&typeof b.data=="string";if(b.dataTypes[0]==="jsonp"||b.jsonp!==!1&&(ce.test(b.url)||e&&ce.test(b.data))){var g,h=b.jsonpCallback=f.isFunction(b.jsonpCallback)?b.jsonpCallback():b.jsonpCallback,i=a[h],j=b.url,k=b.data,l="$1"+h+"$2";b.jsonp!==!1&&(j=j.replace(ce,l),b.url===j&&(e&&(k=k.replace(ce,l)),b.data===k&&(j+=(/\?/.test(j)?"&":"?")+b.jsonp+"="+h))),b.url=j,b.data=k,a[h]=function(a){g=[a]},d.always(function(){a[h]=i,g&&f.isFunction(i)&&a[h](g[0])}),b.converters["script json"]=function(){g||f.error(h+" was not called");return g[0]},b.dataTypes[0]="json";return"script"}}),f.ajaxSetup({accepts:{script:"text/javascript, application/javascript, application/ecmascript, application/x-ecmascript"},contents:{script:/javascript|ecmascript/},converters:{"text script":function(a){f.globalEval(a);return a}}}),f.ajaxPrefilter("script",function(a){a.cache===b&&(a.cache=!1),a.crossDomain&&(a.type="GET",a.global=!1)}),f.ajaxTransport("script",function(a){if(a.crossDomain){var d,e=c.head||c.getElementsByTagName("head")[0]||c.documentElement;return{send:function(f,g){d=c.createElement("script"),d.async="async",a.scriptCharset&&(d.charset=a.scriptCharset),d.src=a.url,d.onload=d.onreadystatechange=function(a,c){if(c||!d.readyState||/loaded|complete/.test(d.readyState))d.onload=d.onreadystatechange=null,e&&d.parentNode&&e.removeChild(d),d=b,c||g(200,"success")},e.insertBefore(d,e.firstChild)},abort:function(){d&&d.onload(0,1)}}}});var cf=a.ActiveXObject?function(){for(var a in ch)ch[a](0,1)}:!1,cg=0,ch;f.ajaxSettings.xhr=a.ActiveXObject?function(){return!this.isLocal&&ci()||cj()}:ci,function(a){f.extend(f.support,{ajax:!!a,cors:!!a&&"withCredentials"in a})}(f.ajaxSettings.xhr()),f.support.ajax&&f.ajaxTransport(function(c){if(!c.crossDomain||f.support.cors){var d;return{send:function(e,g){var h=c.xhr(),i,j;c.username?h.open(c.type,c.url,c.async,c.username,c.password):h.open(c.type,c.url,c.async);if(c.xhrFields)for(j in c.xhrFields)h[j]=c.xhrFields[j];c.mimeType&&h.overrideMimeType&&h.overrideMimeType(c.mimeType),!c.crossDomain&&!e["X-Requested-With"]&&(e["X-Requested-With"]="XMLHttpRequest");try{for(j in e)h.setRequestHeader(j,e[j])}catch(k){}h.send(c.hasContent&&c.data||null),d=function(a,e){var j,k,l,m,n;try{if(d&&(e||h.readyState===4)){d=b,i&&(h.onreadystatechange=f.noop,cf&&delete ch[i]);if(e)h.readyState!==4&&h.abort();else{j=h.status,l=h.getAllResponseHeaders(),m={},n=h.responseXML,n&&n.documentElement&&(m.xml=n),m.text=h.responseText;try{k=h.statusText}catch(o){k=""}!j&&c.isLocal&&!c.crossDomain?j=m.text?200:404:j===1223&&(j=204)}}}catch(p){e||g(-1,p)}m&&g(j,k,m,l)},!c.async||h.readyState===4?d():(i=++cg,cf&&(ch||(ch={},f(a).unload(cf)),ch[i]=d),h.onreadystatechange=d)},abort:function(){d&&d(0,1)}}}});var ck={},cl,cm,cn=/^(?:toggle|show|hide)$/,co=/^([+\-]=)?([\d+.\-]+)([a-z%]*)$/i,cp,cq=[["height","marginTop","marginBottom","paddingTop","paddingBottom"],["width","marginLeft","marginRight","paddingLeft","paddingRight"],["opacity"]],cr;f.fn.extend({show:function(a,b,c){var d,e;if(a||a===0)return this.animate(cu("show",3),a,b,c);for(var g=0,h=this.length;g=i.duration+this.startTime){this.now=this.end,this.pos=this.state=1,this.update(),i.animatedProperties[this.prop]=!0;for(b in i.animatedProperties)i.animatedProperties[b]!==!0&&(g=!1);if(g){i.overflow!=null&&!f.support.shrinkWrapBlocks&&f.each(["","X","Y"],function(a,b){h.style["overflow"+b]=i.overflow[a]}),i.hide&&f(h).hide();if(i.hide||i.show)for(b in i.animatedProperties)f.style(h,b,i.orig[b]),f.removeData(h,"fxshow"+b,!0),f.removeData(h,"toggle"+b,!0);d=i.complete,d&&(i.complete=!1,d.call(h))}return!1}i.duration==Infinity?this.now=e:(c=e-this.startTime,this.state=c/i.duration,this.pos=f.easing[i.animatedProperties[this.prop]](this.state,c,0,1,i.duration),this.now=this.start+(this.end-this.start)*this.pos),this.update();return!0}},f.extend(f.fx,{tick:function(){var a,b=f.timers,c=0;for(;c-1,k={},l={},m,n;j?(l=e.position(),m=l.top,n=l.left):(m=parseFloat(h)||0,n=parseFloat(i)||0),f.isFunction(b)&&(b=b.call(a,c,g)),b.top!=null&&(k.top=b.top-g.top+m),b.left!=null&&(k.left=b.left-g.left+n),"using"in b?b.using.call(a,k):e.css(k)}},f.fn.extend({position:function(){if(!this[0])return null;var a=this[0],b=this.offsetParent(),c=this.offset(),d=cx.test(b[0].nodeName)?{top:0,left:0}:b.offset();c.top-=parseFloat(f.css(a,"marginTop"))||0,c.left-=parseFloat(f.css(a,"marginLeft"))||0,d.top+=parseFloat(f.css(b[0],"borderTopWidth"))||0,d.left+=parseFloat(f.css(b[0],"borderLeftWidth"))||0;return{top:c.top-d.top,left:c.left-d.left}},offsetParent:function(){return this.map(function(){var a=this.offsetParent||c.body;while(a&&!cx.test(a.nodeName)&&f.css(a,"position")==="static")a=a.offsetParent;return a})}}),f.each(["Left","Top"],function(a,c){var d="scroll"+c;f.fn[d]=function(c){var e,g;if(c===b){e=this[0];if(!e)return null;g=cy(e);return g?"pageXOffset"in g?g[a?"pageYOffset":"pageXOffset"]:f.support.boxModel&&g.document.documentElement[d]||g.document.body[d]:e[d]}return this.each(function(){g=cy(this),g?g.scrollTo(a?f(g).scrollLeft():c,a?c:f(g).scrollTop()):this[d]=c})}}),f.each(["Height","Width"],function(a,c){var d=c.toLowerCase();f.fn["inner"+c]=function(){var a=this[0];return a?a.style?parseFloat(f.css(a,d,"padding")):this[d]():null},f.fn["outer"+c]=function(a){var b=this[0];return b?b.style?parseFloat(f.css(b,d,a?"margin":"border")):this[d]():null},f.fn[d]=function(a){var e=this[0];if(!e)return a==null?null:this;if(f.isFunction(a))return this.each(function(b){var c=f(this);c[d](a.call(this,b,c[d]()))});if(f.isWindow(e)){var g=e.document.documentElement["client"+c],h=e.document.body;return e.document.compatMode==="CSS1Compat"&&g||h&&h["client"+c]||g}if(e.nodeType===9)return Math.max(e.documentElement["client"+c],e.body["scroll"+c],e.documentElement["scroll"+c],e.body["offset"+c],e.documentElement["offset"+c]);if(a===b){var i=f.css(e,d),j=parseFloat(i);return f.isNumeric(j)?j:i}return this.css(d,typeof a=="string"?a:a+"px")}}),a.jQuery=a.$=f,typeof define=="function"&&define.amd&&define.amd.jQuery&&define("jquery",[],function(){return f})})(window); \ No newline at end of file diff --git a/docs/_static/underscore.js b/docs/src/gensim_theme/static/underscore.js similarity index 100% rename from docs/_static/underscore.js rename to docs/src/gensim_theme/static/underscore.js diff --git a/docs/src/gensim_theme/theme.conf b/docs/src/gensim_theme/theme.conf new file mode 100644 index 0000000000..161d802a16 --- /dev/null +++ b/docs/src/gensim_theme/theme.conf @@ -0,0 +1,6 @@ +[theme] +inherit = none +stylesheet = css/style.css +pygments_style = sphinx + +[options] diff --git a/docs/src/index.rst b/docs/src/index.rst deleted file mode 100644 index f2f7e073de..0000000000 --- a/docs/src/index.rst +++ /dev/null @@ -1,63 +0,0 @@ -.. gensim documentation master file, created by - sphinx-quickstart on Tue Mar 16 19:45:41 2010. - You can adapt this file completely to your liking, but it should at least - contain the root `toctree` directive. - -Gensim -- Topic Modelling for Humans -===================================================== - - -.. raw:: html - :file: _static/tagcloud.html - - -.. raw:: html - -
    - - - - - - - - -
    What is this?Installation
    Get started!About
    - - -Quick Reference Example ------------------------- - ->>> from gensim import corpora, models, similarities ->>> ->>> # Load corpus iterator from a Matrix Market file on disk. ->>> corpus = corpora.MmCorpus('/path/to/corpus.mm') ->>> ->>> # Initialize a transformation (Latent Semantic Indexing with 200 latent dimensions). ->>> lsi = models.LsiModel(corpus, num_topics=200) ->>> ->>> # Convert another corpus to the latent space and index it. ->>> index = similarities.MatrixSimilarity(lsi[another_corpus]) ->>> ->>> # determine similarity of a query document against each document in the index ->>> sims = index[query] - - -.. admonition:: What's new? - - * 15 Sep 2012: release 0.8.6 : added the `hashing trick `_ to allow online changes to the vocabulary; fixed parallel lemmatization + `other minor improvements `_ - * 22 Jul 2012: release 0.8.5 : better Wikipedia parsing, faster similarity queries, maintenance fixes - * 30 Apr 2012: William Bert's `interview with me `_ - * 9 Mar 2012: release 0.8.4: new model `Hierarchical Dirichlet Process `_ (full `CHANGELOG `_) - - -.. toctree:: - :hidden: - :maxdepth: 1 - - intro - install - tutorial - distributed - wiki - apiref diff --git a/docs/src/indextoc.rst b/docs/src/indextoc.rst new file mode 100644 index 0000000000..7533955697 --- /dev/null +++ b/docs/src/indextoc.rst @@ -0,0 +1,11 @@ +.. toctree:: + :hidden: + :maxdepth: 1 + + intro + install + tutorial + distributed + support + wiki + apiref diff --git a/docs/src/install.rst b/docs/src/install.rst index e4886b20b8..bfcba77381 100644 --- a/docs/src/install.rst +++ b/docs/src/install.rst @@ -9,9 +9,13 @@ Quick install Run in your terminal:: - sudo easy_install -U gensim + easy_install -U gensim -In case that fails, or you don't know what "terminal" means, read on. +or, alternatively:: + + pip install --upgrade gensim + +In case that fails, make sure you're installing into a writeable location (or use `sudo`), or read on. ----- @@ -21,11 +25,11 @@ Gensim is known to run on Linux, Windows and Mac OS X and should run on any othe platform that supports Python 2.5 and NumPy. Gensim depends on the following software: * 3.0 > `Python `_ >= 2.5. Tested with versions 2.5, 2.6 and 2.7. -* `NumPy `_ >= 1.3. Tested with version 1.6.1rc2, 1.5.0rc1, 1.4.0, 1.3.0, 1.3.0rc2. -* `SciPy `_ >= 0.7. Tested with version 0.9.0, 0.8.0, 0.8.0b1, 0.7.1, 0.7.0. +* `NumPy `_ >= 1.3. Tested with version 1.7.1, 1.7.0, 1.6.2, 1.6.1rc2, 1.5.0rc1, 1.4.0, 1.3.0, 1.3.0rc2. +* `SciPy `_ >= 0.7. Tested with version 0.12.0, 0.11.0, 0.10.1, 0.9.0, 0.8.0, 0.8.0b1, 0.7.1, 0.7.0. **Windows users** are well advised to try the `Enthought distribution `_, -which conveniently includes Python&NumPy&SciPy in a single bundle, and is free for academic use. +which conveniently includes Python & NumPy & SciPy in a single bundle, and is free for academic use. Install Python and `easy_install` @@ -48,8 +52,8 @@ Install SciPy & NumPy These are quite popular Python packages, so chances are there are pre-built binary distributions available for your platform. You can try installing from source using easy_install:: - sudo easy_install numpy - sudo easy_install scipy + easy_install numpy + easy_install scipy If that doesn't work or if you'd rather install using a binary package, consult http://www.scipy.org/Download. @@ -59,7 +63,7 @@ Install `gensim` You can now install (or upgrade) `gensim` with:: - sudo easy_install --upgrade gensim + easy_install --upgrade gensim That's it! Congratulations, you can proceed to the :doc:`tutorials `. @@ -68,14 +72,14 @@ That's it! Congratulations, you can proceed to the :doc:`tutorials `. If you also want to run the algorithms over a cluster of computers, in :doc:`distributed`, you should install with:: - sudo easy_install gensim[distributed] + easy_install gensim[distributed] The optional `distributed` feature installs `Pyro (PYthon Remote Objects) `_. If you don't know what distributed computing means, you can ignore it: `gensim` will work fine for you anyway. This optional extension can also be installed separately later with:: - sudo easy_install Pyro4 + easy_install Pyro4 ----- @@ -85,14 +89,14 @@ There are also alternative routes to install: for `gensim` (or you're installing `gensim` from `github `_), you can run:: - sudo python setup.py install + python setup.py install to install `gensim` into your ``site-packages`` folder. 2. If you wish to make local changes to the `gensim` code (`gensim` is, after all, a package which targets research prototyping and modifications), a preferred way may be installing with:: - sudo python setup.py develop + python setup.py develop This will only place a symlink into your ``site-packages`` directory. The actual files will stay wherever you unpacked them. @@ -110,9 +114,8 @@ To test the package, unzip the `tar.gz source `_ for -any questions and troubleshooting. For private enquiries, you can also send -me an email to the address at the bottom of this page. +questions and troubleshooting. See the :doc:`support page `. diff --git a/docs/src/intro.rst b/docs/src/intro.rst index 2166f21b14..860cbee18e 100644 --- a/docs/src/intro.rst +++ b/docs/src/intro.rst @@ -41,8 +41,7 @@ frameworks that realize topic modelling, and/or their overwhelming internal comp You can read more about the motivation in our `LREC 2010 workshop paper `_. If you want to cite `gensim` in your own work, please refer to that article (`BibTeX `_). -You're welcome to share your results on the `mailing list `_, -so others can learn from your success :) +You're welcome to share your results and experiments on the `mailing list `_. The **principal design objectives** behind `gensim` are: diff --git a/docs/src/support.rst b/docs/src/support.rst new file mode 100644 index 0000000000..640c639537 --- /dev/null +++ b/docs/src/support.rst @@ -0,0 +1,30 @@ +.. _support: + +============= +Support +============= + +Open source support +-------------------- + +The main communication channel is the `gensim mailing list `_. +This is the preferred way to **ask for help**, **report problems** and **share insights** with the community. Newbie questions are perfectly fine, just make sure you've read the :doc:`tutorials `. + +I discourage sending private emails, because the mailing list serves as a knowledge base for all gensim users, cutting maintenance efforts needed for support. If you feel your problem is too special, data too sensitive, or you're just too shy, see the "commercial" section below. + +When posting on the mailing list, please try to include all relevant information, such as what it is you are trying to achieve, what went wrong, relevant gensim logs etc. + +**FAQ** and some useful **snippets of code** are maintained on GitHub: https://github.com/piskvorky/gensim/wiki/Recipes-&-FAQ. + +You can also try asking on StackOverflow, using the `gensim tag `_. + +Developer issues +---------------- + +Developers who `tweak gensim internals `_ are encouraged to report issues at the `GitHub issue tracker `_. +Note that this is not a medium for discussions or asking open-ended questions; please use the mailing list for that. + +Commercial support +------------------ + +In case you need deterministic response times or have extra support/development needs, `contact me `_ for a quote on commercial support and contracting. diff --git a/docs/src/tut1.rst b/docs/src/tut1.rst index bd7fc87b0e..09e4569f3c 100644 --- a/docs/src/tut1.rst +++ b/docs/src/tut1.rst @@ -257,10 +257,24 @@ just load a document stream using one format and immediately save it in another Adding new formats is dead easy, check out the `code for the SVMlight corpus `_ for an example. +Compatibility with NumPy and SciPy +---------------------------------- + +Gensim also contains `efficient utility functions `_ +to help converting from/to numpy matrices:: + +>>> corpus = gensim.matutils.Dense2Corpus(numpy_matrix) +>>> numpy_matrix = gensim.matutils.corpus2dense(corpus) + +and from/to `scipy.sparse` matrices:: + +>>> corpus = gensim.matutils.Sparse2Corpus(scipy_sparse_matrix) +>>> scipy_csc_matrix = gensim.matutils.corpus2csc(corpus) + ------------- For a complete reference (Want to prune the dictionary to a smaller size? -Convert between corpora and NumPy/SciPy arrays?), see the :doc:`API documentation `. +Optimize converting between corpora and NumPy/SciPy arrays?), see the :doc:`API documentation `. Or continue to the next tutorial on :doc:`tut2`. diff --git a/docs/src/tut3.rst b/docs/src/tut3.rst index 2a0eca97ed..aeaa4ce0c1 100644 --- a/docs/src/tut3.rst +++ b/docs/src/tut3.rst @@ -68,8 +68,9 @@ might also be indexing a different corpus altogether. The class :class:`similarities.MatrixSimilarity` is only appropriate when the whole set of vectors fits into memory. For example, a corpus of one million documents would require 2GB of RAM in a 256-dimensional LSI space, when used with this class. + Without 2GB of free RAM, you would need to use the :class:`similarities.Similarity` class. - This class operates in fixed memory, by splitting the index across multiple files on disk. + This class operates in fixed memory, by splitting the index across multiple files on disk, called shards. It uses :class:`similarities.MatrixSimilarity` and :class:`similarities.SparseMatrixSimilarity` internally, so it is still fast, although slightly more complex. @@ -133,15 +134,16 @@ Congratulations, you have finished the tutorials -- now you know how gensim work To delve into more details, you can browse through the :doc:`API documentation `, see the :doc:`Wikipedia experiments ` or perhaps check out :doc:`distributed computing ` in `gensim`. -Please remember that gensim is an experimental package, aimed at the NLP research community. -This means that: +Gensim is a fairly mature package that has been used successfully by many individuals and companies, both for rapid prototyping and in production. +That doesn't mean it's perfect though: -* there certainly are parts that could be implemented more efficiently (in C, for example), and there may also be bugs in the code -* your **feedback is most welcome** and appreciated, be it in code and +* there are parts that could be implemented more efficiently (in C, for example), or make better use of parallelism (multiple machines cores) +* new algorithms are published all the time; help gensim keep up by `discussing them `_ and `contributing code `_ +* your **feedback is most welcome** and appreciated (and it's not just the code!): `idea contributions `_, - `bug reports `_ or just + `bug reports `_ or just consider contributing `user stories and general questions `_. -Gensim has no ambition to become an all-encompassing production level tool, with robust failure handling -and error recoveries. Its main goal is to help NLP newcomers try out popular algorithms -and to facilitate prototyping of new algorithms for NLP researchers. +Gensim has no ambition to become an all-encompassing framework, across all NLP (or even Machine Learning) subfields. +Its mission is to help NLP practicioners try out popular topic modelling algorithms +on large datasets easily, and to facilitate prototyping of new algorithms for researchers. diff --git a/docs/src/tutorial.rst b/docs/src/tutorial.rst index a25f54f80b..db5bd4c702 100644 --- a/docs/src/tutorial.rst +++ b/docs/src/tutorial.rst @@ -5,8 +5,8 @@ Tutorials The tutorials are organized as a series of examples that highlight various features -of `gensim`. It is assumed that the reader is familiar with the Python language -and has read the :doc:`intro`. +of `gensim`. It is assumed that the reader is familiar with the `Python language `_, has :doc:`installed gensim ` +and read the :doc:`introduction `. The examples are divided into parts on: @@ -22,10 +22,7 @@ The examples are divided into parts on: Preliminaries -------------- -All the examples can be directly copied to your Python interpreter shell (assuming -you have :doc:`gensim installed `, of course). -`IPython `_'s ``cpaste`` command is especially handy for copypasting code fragments which include superfluous -characters, such as the leading ``>>>``. +All the examples can be directly copied to your Python interpreter shell. `IPython `_'s ``cpaste`` command is especially handy for copypasting code fragments, including the leading ``>>>`` characters. Gensim uses Python's standard :mod:`logging` module to log various stuff at various priority levels; to activate logging (this is optional), run @@ -39,7 +36,7 @@ priority levels; to activate logging (this is optional), run Quick Example ------------- -First, let's import gensim and create a small corpus of nine documents [1]_: +First, let's import gensim and create a small corpus of nine documents and twelve features [1]_: >>> from gensim import corpora, models, similarities >>> @@ -54,14 +51,10 @@ First, let's import gensim and create a small corpus of nine documents [1]_: >>> [(8, 1.0), (10, 1.0), (11, 1.0)]] :dfn:`Corpus` is simply an object which, when iterated over, returns its documents represented -as sparse vectors. +as sparse vectors. If you're not familiar with the `vector space model `_, we'll bridge the gap between **raw strings**, **corpora** and **sparse vectors** in the next tutorial on :doc:`tut1`. -If you're familiar with the `Vector Space Model `_, -you'll probably know that the way you parse your documents and convert them to vectors -has major impact on the quality of any subsequent applications. If you're not familiar -with :abbr:`VSM (Vector Space Model)`, we'll bridge the gap between **raw strings** -and **sparse vectors** in the next tutorial -on :doc:`tut1`. +If you're familiar with the vector space model, you'll probably know that the way you parse your documents and convert them to vectors +has major impact on the quality of any subsequent applications. .. note:: In this example, the whole corpus is stored in memory, as a Python list. However, @@ -91,7 +84,7 @@ Transformations are covered in detail in the tutorial on :doc:`tut2`. To transform the whole corpus via TfIdf and index it, in preparation for similarity queries: ->>> index = similarities.SparseMatrixSimilarity(tfidf[corpus]) +>>> index = similarities.SparseMatrixSimilarity(tfidf[corpus], num_features=12) and to query the similarity of our query vector ``vec`` against every document in the corpus: diff --git a/docs/tut1.html b/docs/tut1.html deleted file mode 100644 index d90097a0eb..0000000000 --- a/docs/tut1.html +++ /dev/null @@ -1,415 +0,0 @@ - - - - - - - - - - Corpora and Vector Spaces — gensim - - - - - - - - - - - - - - - - - - - - - - - - - - - -
    -

    Navigation

    - -
    - - -
    -
    -

    Table Of Contents

    - - -

    Previous topic

    -

    Tutorials

    -

    Next topic

    -

    Topics and Transformations

    - - -
    -
    - - - - -
    -
    -
    -
    - -
    -

    Corpora and Vector Spaces

    -

    Don’t forget to set

    -
    >>> import logging
->>> logging.basicConfig(format='%(asctime)s : %(levelname)s : %(message)s', level=logging.INFO)
-
    -
    -

    if you want to see logging events.

    -
    -

    From Strings to Vectors

    -

    This time, let’s start from documents represented as strings:

    -
    >>> from gensim import corpora, models, similarities
->>>
->>> documents = ["Human machine interface for lab abc computer applications",
->>>              "A survey of user opinion of computer system response time",
->>>              "The EPS user interface management system",
->>>              "System and human system engineering testing of EPS",
->>>              "Relation of user perceived response time to error measurement",
->>>              "The generation of random binary unordered trees",
->>>              "The intersection graph of paths in trees",
->>>              "Graph minors IV Widths of trees and well quasi ordering",
->>>              "Graph minors A survey"]
-
    -
    -

    This is a tiny corpus of nine documents, each consisting of only a single sentence.

    -

    First, let’s tokenize the documents, remove common words (using a toy stoplist) -as well as words that only appear once in the corpus:

    -
    >>> # remove common words and tokenize
->>> stoplist = set('for a of the and to in'.split())
->>> texts = [[word for word in document.lower().split() if word not in stoplist]
->>>          for document in documents]
->>>
->>> # remove words that appear only once
->>> all_tokens = sum(texts, [])
->>> tokens_once = set(word for word in set(all_tokens) if all_tokens.count(word) == 1)
->>> texts = [[word for word in text if word not in tokens_once]
->>>          for text in texts]
->>>
->>> print texts
-[['human', 'interface', 'computer'],
- ['survey', 'user', 'computer', 'system', 'response', 'time'],
- ['eps', 'user', 'interface', 'system'],
- ['system', 'human', 'system', 'eps'],
- ['user', 'response', 'time'],
- ['trees'],
- ['graph', 'trees'],
- ['graph', 'minors', 'trees'],
- ['graph', 'minors', 'survey']]
-
    -
    -

    Your way of processing the documents will likely vary; here, I only split on whitespace -to tokenize, followed by lowercasing each word. In fact, I use this particular -(simplistic and inefficient) setup to mimick the experiment done in Deerwester et al.’s -original LSA article [1].

    -

    The ways to process documents are so varied and application- and language-dependent that I -decided to not constrain them by any interface. Instead, a document is represented -by the features extracted from it, not by its “surface” string form: how you get to -the features is up to you. Below I describe one common, general-purpose approach (called -bag-of-words), but keep in mind that different application domains call for -different features, and, as always, it’s garbage in, garbage out...

    -

    To convert documents to vectors, we’ll use a document representation called -bag-of-words. In this representation, -each document is represented by one vector where each vector element represents -a question-answer pair, in the style of:

    -
    -
    “How many times does the word system appear in the document? Once.”
    -

    It is advantageous to represent the questions only by their (integer) ids. The mapping -between the questions and ids is called a dictionary:

    -
    >>> dictionary = corpora.Dictionary(texts)
->>> dictionary.save('/tmp/deerwester.dict') # store the dictionary, for future reference
->>> print dictionary
-Dictionary(12 unique tokens)
-
    -
    -

    Here we assigned a unique integer id to all words appearing in the corpus with the -gensim.corpora.dictionary.Dictionary class. This sweeps across the texts, collecting word counts -and relevant statistics. In the end, we see there are twelve distinct words in the -processed corpus, which means each document will be represented by twelve numbers (ie., by a 12-D vector). -To see the mapping between words and their ids:

    -
    >>> print dictionary.token2id
-{'minors': 11, 'graph': 10, 'system': 5, 'trees': 9, 'eps': 8, 'computer': 0,
-'survey': 4, 'user': 7, 'human': 1, 'time': 6, 'interface': 2, 'response': 3}
-
    -
    -

    To actually convert tokenized documents to vectors:

    -
    >>> new_doc = "Human computer interaction"
->>> new_vec = dictionary.doc2bow(new_doc.lower().split())
->>> print new_vec # the word "interaction" does not appear in the dictionary and is ignored
-[(0, 1), (1, 1)]
-
    -
    -

    The function doc2bow() simply counts the number of occurences of -each distinct word, converts the word to its integer word id -and returns the result as a sparse vector. The sparse vector [(0, 1), (1, 1)] -therefore reads: in the document “Human computer interaction”, the words computer -(id 0) and human (id 1) appear once; the other ten dictionary words appear (implicitly) zero times.

    -
    >>> corpus = [dictionary.doc2bow(text) for text in texts]
->>> corpora.MmCorpus.serialize('/tmp/deerwester.mm', corpus) # store to disk, for later use
->>> print corpus
-[(0, 1), (1, 1), (2, 1)]
-[(0, 1), (3, 1), (4, 1), (5, 1), (6, 1), (7, 1)]
-[(2, 1), (5, 1), (7, 1), (8, 1)]
-[(1, 1), (5, 2), (8, 1)]
-[(3, 1), (6, 1), (7, 1)]
-[(9, 1)]
-[(9, 1), (10, 1)]
-[(9, 1), (10, 1), (11, 1)]
-[(4, 1), (10, 1), (11, 1)]
-
    -
    -

    By now it should be clear that the vector feature with id=10 stands for the question “How many -times does the word graph appear in the document?” and that the answer is “zero” for -the first six documents and “one” for the remaining three. As a matter of fact, -we have arrived at exactly the same corpus of vectors as in the Quick Example.

    -
    -
    -

    Corpus Streaming – One Document at a Time

    -

    Note that corpus above resides fully in memory, as a plain Python list. -In this simple example, it doesn’t matter much, but just to make things clear, -let’s assume there are millions of documents in the corpus. Storing all of them in RAM won’t do. -Instead, let’s assume the documents are stored in a file on disk, one document per line. Gensim -only requires that a corpus must be able to return one document vector at a time:

    -
    >>> class MyCorpus(object):
->>>     def __iter__(self):
->>>         for line in open('mycorpus.txt'):
->>>             # assume there's one document per line, tokens separated by whitespace
->>>             yield dictionary.doc2bow(line.lower().split())
-
    -
    -

    Download the sample mycorpus.txt file here. The assumption that -each document occupies one line in a single file is not important; you can mold -the __iter__ function to fit your input format, whatever it is. -Walking directories, parsing XML, accessing network... -Just parse your input to retrieve a clean list of tokens in each document, -then convert the tokens via a dictionary to their ids and yield the resulting sparse vector inside __iter__.

    -
    >>> corpus_memory_friendly = MyCorpus() # doesn't load the corpus into memory!
->>> print corpus_memory_friendly
-<__main__.MyCorpus object at 0x10d5690>
-
    -
    -

    Corpus is now an object. We didn’t define any way to print it, so print just outputs address -of the object in memory. Not very useful. To see the constituent vectors, let’s -iterate over the corpus and print each document vector (one at a time):

    -
    >>> for vector in corpus_memory_friendly: # load one vector into memory at a time
->>>     print vector
-[(0, 1), (1, 1), (2, 1)]
-[(0, 1), (3, 1), (4, 1), (5, 1), (6, 1), (7, 1)]
-[(2, 1), (5, 1), (7, 1), (8, 1)]
-[(1, 1), (5, 2), (8, 1)]
-[(3, 1), (6, 1), (7, 1)]
-[(9, 1)]
-[(9, 1), (10, 1)]
-[(9, 1), (10, 1), (11, 1)]
-[(4, 1), (10, 1), (11, 1)]
-
    -
    -

    Although the output is the same as for the plain Python list, the corpus is now much -more memory friendly, because at most one vector resides in RAM at a time. Your -corpus can now be as large as you want.

    -

    Similarly, to construct the dictionary without loading all texts into memory:

    -
    >>> # collect statistics about all tokens
->>> dictionary = corpora.Dictionary(line.lower().split() for line in open('mycorpus.txt'))
->>> # remove stop words and words that appear only once
->>> stop_ids = [dictionary.token2id[stopword] for stopword in stoplist
->>>             if stopword in dictionary.token2id]
->>> once_ids = [tokenid for tokenid, docfreq in dictionary.dfs.iteritems() if docfreq == 1]
->>> dictionary.filter_tokens(stop_ids + once_ids) # remove stop words and words that appear only once
->>> dictionary.compactify() # remove gaps in id sequence after words that were removed
->>> print dictionary
-Dictionary(12 unique tokens)
-
    -
    -

    And that is all there is to it! At least as far as bag-of-words representation is concerned. -Of course, what we do with such corpus is another question; it is not at all clear -how counting the frequency of distinct words could be useful. As it turns out, it isn’t, and -we will need to apply a transformation on this simple representation first, before -we can use it to compute any meaningful document vs. document similarities. -Transformations are covered in the next tutorial, but before that, let’s -briefly turn our attention to corpus persistency.

    -
    -
    -

    Corpus Formats

    -

    There exist several file formats for serializing a Vector Space corpus (~sequence of vectors) to disk. -Gensim implements them via the streaming corpus interface mentioned earlier: -documents are read from (resp. stored to) disk in a lazy fashion, one document at -a time, without the whole corpus being read into main memory at once.

    -

    One of the more notable file formats is the Market Matrix format. -To save a corpus in the Matrix Market format:

    -
    >>> from gensim import corpora
->>> # create a toy corpus of 2 documents, as a plain Python list
->>> corpus = [[(1, 0.5)], []]  # make one document empty, for the heck of it
->>>
->>> corpora.MmCorpus.serialize('/tmp/corpus.mm', corpus)
-
    -
    -

    Other formats include Joachim’s SVMlight format, -Blei’s LDA-C format and -GibbsLDA++ format.

    -
    >>> corpora.SvmLightCorpus.serialize('/tmp/corpus.svmlight', corpus)
->>> corpora.BleiCorpus.serialize('/tmp/corpus.lda-c', corpus)
->>> corpora.LowCorpus.serialize('/tmp/corpus.low', corpus)
-
    -
    -

    Conversely, to load a corpus iterator from a Matrix Market file:

    -
    >>> corpus = corpora.MmCorpus('/tmp/corpus.mm')
-
    -
    -

    Corpus objects are streams, so typically you won’t be able to print them directly:

    -
    >>> print corpus
-MmCorpus(2 documents, 2 features, 1 non-zero entries)
-
    -
    -

    Instead, to view the contents of a corpus:

    -
    >>> # one way of printing a corpus: load it entirely into memory
->>> print list(corpus) # calling list() will convert any sequence to a plain Python list
-[[(1, 0.5)], []]
-
    -
    -

    or

    -
    >>> # another way of doing it: print one document at a time, making use of the streaming interface
->>> for doc in corpus:
->>>     print doc
-[(1, 0.5)]
-[]
-
    -
    -

    The second way is obviously more memory-friendly, but for testing and development -purposes, nothing beats the simplicity of calling list(corpus).

    -

    To save the same Matrix Market document stream in Blei’s LDA-C format,

    -
    >>> corpora.BleiCorpus.serialize('/tmp/corpus.lda-c', corpus)
-
    -
    -

    In this way, gensim can also be used as a memory-efficient I/O format conversion tool: -just load a document stream using one format and immediately save it in another format. -Adding new formats is dead easy, check out the code for the SVMlight corpus for an example.

    -
    -

    For a complete reference (Want to prune the dictionary to a smaller size? -Convert between corpora and NumPy/SciPy arrays?), see the API documentation. -Or continue to the next tutorial on Topics and Transformations.

    - - - - - -
    [1]This is the same corpus as used in -Deerwester et al. (1990): Indexing by Latent Semantic Analysis, Table 2.
    -
    -
    - - -
    -
    -
    -
    -
    - - -
    -

    Navigation

    - -
    - - -
    - © Copyright 2009-2012, Radim Řehůřek <radimrehurek(at)seznam.cz>. - Last updated on Sep 16, 2012. -
    - - \ No newline at end of file diff --git a/docs/tut2.html b/docs/tut2.html deleted file mode 100644 index b9ee456790..0000000000 --- a/docs/tut2.html +++ /dev/null @@ -1,426 +0,0 @@ - - - - - - - - - - Topics and Transformations — gensim - - - - - - - - - - - - - - - - - - - - - - - - - - - -
    -

    Navigation

    - -
    - - -
    -
    -

    Table Of Contents

    - - -

    Previous topic

    -

    Corpora and Vector Spaces

    -

    Next topic

    -

    Similarity Queries

    - - -
    -
    - - - - -
    -
    -
    -
    - -
    -

    Topics and Transformations

    -

    Don’t forget to set

    -
    >>> import logging
->>> logging.basicConfig(format='%(asctime)s : %(levelname)s : %(message)s', level=logging.INFO)
-
    -
    -

    if you want to see logging events.

    -
    -

    Transformation interface

    -

    In the previous tutorial on Corpora and Vector Spaces, we created a corpus of documents represented -as a stream of vectors. To continue, let’s fire up gensim and use that corpus:

    -
    >>> from gensim import corpora, models, similarities
->>> dictionary = corpora.Dictionary.load('/tmp/deerwester.dict')
->>> corpus = corpora.MmCorpus('/tmp/deerwester.mm')
->>> print corpus
-MmCorpus(9 documents, 12 features, 28 non-zero entries)
-
    -
    -

    In this tutorial, I will show how to transform documents from one vector representation -into another. This process serves two goals:

    -
      -
    1. To bring out hidden structure in the corpus, discover relationships between -words and use them to describe the documents in a new and -(hopefully) more semantic way.
      2. -
    2. To make the document representation more compact. This both improves efficiency -(new representation consumes less resources) and efficacy (marginal data -trends are ignored, noise-reduction).
      3. -
    -
    -

    Creating a transformation

    -

    The transformations are standard Python objects, typically initialized by means of -a training corpus:

    -
    >>> tfidf = models.TfidfModel(corpus) # step 1 -- initialize a model
-
    -
    -

    We used our old corpus from tutorial 1 to initialize (train) the transformation model. Different -transformations may require different initialization parameters; in case of TfIdf, the -“training” consists simply of going through the supplied corpus once and computing document frequencies -of all its features. Training other models, such as Latent Semantic Analysis or Latent Dirichlet -Allocation, is much more involved and, consequently, takes much more time.

    -
    -

    Note

    -

    Transformations always convert between two specific vector -spaces. The same vector space (= the same set of feature ids) must be used for training -as well as for subsequent vector transformations. Failure to use the same input -feature space, such as applying a different string preprocessing, using different -feature ids, or using bag-of-words input vectors where TfIdf vectors are expected, will -result in feature mismatch during transformation calls and consequently in either -garbage output and/or runtime exceptions.

    -
    -
    -
    -

    Transforming vectors

    -

    From now on, tfidf is treated as a read-only object that can be used to convert -any vector from the old representation (bag-of-words integer counts) to the new representation -(TfIdf real-valued weights):

    -
    >>> doc_bow = [(0, 1), (1, 1)]
->>> print tfidf[doc_bow] # step 2 -- use the model to transform vectors
-[(0, 0.70710678), (1, 0.70710678)]
-
    -
    -

    Or to apply a transformation to a whole corpus:

    -
    >>> corpus_tfidf = tfidf[corpus]
->>> for doc in corpus_tfidf:
->>>     print doc
-[(0, 0.57735026918962573), (1, 0.57735026918962573), (2, 0.57735026918962573)]
-[(0, 0.44424552527467476), (3, 0.44424552527467476), (4, 0.44424552527467476), (5, 0.32448702061385548), (6, 0.44424552527467476), (7, 0.32448702061385548)]
-[(2, 0.5710059809418182), (5, 0.41707573620227772), (7, 0.41707573620227772), (8, 0.5710059809418182)]
-[(1, 0.49182558987264147), (5, 0.71848116070837686), (8, 0.49182558987264147)]
-[(3, 0.62825804686700459), (6, 0.62825804686700459), (7, 0.45889394536615247)]
-[(9, 1.0)]
-[(9, 0.70710678118654746), (10, 0.70710678118654746)]
-[(9, 0.50804290089167492), (10, 0.50804290089167492), (11, 0.69554641952003704)]
-[(4, 0.62825804686700459), (10, 0.45889394536615247), (11, 0.62825804686700459)]
-
    -
    -

    In this particular case, we are transforming the same corpus that we used -for training, but this is only incidental. Once the transformation model has been initialized, -it can be used on any vectors (provided they come from the same vector space, of course), -even if they were not used in the training corpus at all. This is achieved by a process called -folding-in for LSA, by topic inference for LDA etc.

    -
    -

    Note

    -

    Calling model[corpus] only creates a wrapper around the old corpus -document stream – actual conversions are done on-the-fly, during document iteration. -We cannot convert the entire corpus at the time of calling corpus_transformed = model[corpus], -because that would mean storing the result in main memory, and that contradicts gensim’s objective of memory-indepedence. -If you will be iterating over the transformed corpus_transformed multiple times, and the -transformation is costly, serialize the resulting corpus to disk first and continue -using that.

    -
    -

    Transformations can also be serialized, one on top of another, in a sort of chain:

    -
    >>> lsi = models.LsiModel(corpus_tfidf, id2word=dictionary, num_topics=2) # initialize an LSI transformation
->>> corpus_lsi = lsi[corpus_tfidf] # create a double wrapper over the original corpus: bow->tfidf->fold-in-lsi
-
    -
    -

    Here we transformed our Tf-Idf corpus via Latent Semantic Indexing -into a latent 2-D space (2-D because we set num_topics=2). Now you’re probably wondering: what do these two latent -dimensions stand for? Let’s inspect with models.LsiModel.print_topics():

    -
    >>> lsi.print_topics(2)
-topic #0(1.594): -0.703*"trees" + -0.538*"graph" + -0.402*"minors" + -0.187*"survey" + -0.061*"system" + -0.060*"response" + -0.060*"time" + -0.058*"user" + -0.049*"computer" + -0.035*"interface"
-topic #1(1.476): -0.460*"system" + -0.373*"user" + -0.332*"eps" + -0.328*"interface" + -0.320*"response" + -0.320*"time" + -0.293*"computer" + -0.280*"human" + -0.171*"survey" + 0.161*"trees"
-
    -
    -

    (the topics are printed to log – see the note at the top of this page about activating -logging)

    -

    It appears that according to LSI, “trees”, “graph” and “minors” are all related -words (and contribute the most to the direction of the first topic), while the -second topic practically concerns itself with all the other words. As expected, -the first five documents are more strongly related to the second topic while the -remaining four documents to the first topic:

    -
    >>> for doc in corpus_lsi: # both bow->tfidf and tfidf->lsi transformations are actually executed here, on the fly
->>>     print doc
-[(0, -0.066), (1, 0.520)] # "Human machine interface for lab abc computer applications"
-[(0, -0.197), (1, 0.761)] # "A survey of user opinion of computer system response time"
-[(0, -0.090), (1, 0.724)] # "The EPS user interface management system"
-[(0, -0.076), (1, 0.632)] # "System and human system engineering testing of EPS"
-[(0, -0.102), (1, 0.574)] # "Relation of user perceived response time to error measurement"
-[(0, -0.703), (1, -0.161)] # "The generation of random binary unordered trees"
-[(0, -0.877), (1, -0.168)] # "The intersection graph of paths in trees"
-[(0, -0.910), (1, -0.141)] # "Graph minors IV Widths of trees and well quasi ordering"
-[(0, -0.617), (1, 0.054)] # "Graph minors A survey"
-
    -
    -

    Model persistency is achieved with the save() and load() functions:

    -
    >>> lsi.save('/tmp/model.lsi') # same for tfidf, lda, ...
->>> lsi = models.LsiModel.load('/tmp/model.lsi')
-
    -
    -

    The next question might be: just how exactly similar are those documents to each other? -Is there a way to formalize the similarity, so that for a given input document, we can -order some other set of documents according to their similarity? Similarity queries -are covered in the next tutorial.

    -
    -
    -
    -

    Available transformations

    -

    Gensim implements several popular Vector Space Model algorithms:

    -
      -

    • Term Frequency * Inverse Document Frequency, Tf-Idf -expects a bag-of-words (integer values) training corpus during initialization. -During transformation, it will take a vector and return another vector of the -same dimensionality, except that features which were rare in the training corpus -will have their value increased. -It therefore converts integer-valued vectors into real-valued ones, while leaving -the number of dimensions intact. It can also optionally normalize the resulting -vectors to (Euclidean) unit length.

      -
      >>> model = tfidfmodel.TfidfModel(bow_corpus, normalize=True)
-
      -
      -
      • -

    • Latent Semantic Indexing, LSI (or sometimes LSA) -transforms documents from either bag-of-words or (preferrably) TfIdf-weighted space into -a latent space of a lower dimensionality. For the toy corpus above we used only -2 latent dimensions, but on real corpora, target dimensionality of 200–500 is recommended -as a “golden standard” [1].

      -
      >>> model = lsimodel.LsiModel(tfidf_corpus, id2word=dictionary, num_topics=300)
-
      -
      -

      LSI training is unique in that we can continue “training” at any point, simply -by providing more training documents. This is done by incremental updates to -the underlying model, in a process called online training. Because of this feature, the -input document stream may even be infinite – just keep feeding LSI new documents -as they arrive, while using the computed transformation model as read-only in the meanwhile!

      -
      >>> model.add_documents(another_tfidf_corpus) # now LSI has been trained on tfidf_corpus + another_tfidf_corpus
->>> lsi_vec = model[tfidf_vec] # convert some new document into the LSI space, without affecting the model
->>> ...
->>> model.add_documents(more_documents) # tfidf_corpus + another_tfidf_corpus + more_documents
->>> lsi_vec = model[tfidf_vec]
->>> ...
-
      -
      -

      See the gensim.models.lsimodel documentation for details on how to make -LSI gradually “forget” old observations in infinite streams. If you want to get dirty, -there are also parameters you can tweak that affect speed vs. memory footprint vs. numerical -precision of the LSI algorithm.

      -

      gensim uses a novel online incremental streamed distributed training algorithm (quite a mouthful!), -which I published in [5]. gensim also executes a stochastic multi-pass algorithm -from Halko et al. [4] internally, to accelerate in-core part -of the computations. -See also Experiments on the English Wikipedia for further speed-ups by distributing the computation across -a cluster of computers.

      -
      • -

    • Random Projections, RP aim to -reduce vector space dimensionality. This is a very efficient (both memory- and -CPU-friendly) approach to approximating TfIdf distances between documents, by throwing in a little randomness. -Recommended target dimensionality is again in the hundreds/thousands, depending on your dataset.

      -
      >>> model = rpmodel.RpModel(tfidf_corpus, num_topics=500)
-
      -
      -
      • -

    • Latent Dirichlet Allocation, LDA -is yet another transformation from bag-of-words counts into a topic space of lower -dimensionality. LDA is a probabilistic extension of LSA (also called multinomial PCA), -so LDA’s topics can be interpreted as probability distributions over words. These distributions are, -just like with LSA, inferred automatically from a training corpus. Documents -are in turn interpreted as a (soft) mixture of these topics (again, just like with LSA).

      -
      >>> model = ldamodel.LdaModel(bow_corpus, id2word=dictionary, num_topics=100)
-
      -
      -

      gensim uses a fast implementation of online LDA parameter estimation based on [2], -modified to run in distributed mode on a cluster of computers.

      -
      • -

    • Hierarchical Dirichlet Process, HDP -is a non-parametric bayesian method (note the missing number of requested topics):

      -
      >>> model = hdpmodel.HdpModel(bow_corpus, id2word=dictionary)
-
      -
      -

      gensim uses a fast, online implementation based on [3]. -The HDP model is a new addition to gensim, and still rough around its academic edges – use with care.

      -
      • -
    -

    Adding new VSM transformations (such as different weighting schemes) is rather trivial; -see the API reference or directly the Python code -for more info and examples.

    -

    It is worth repeating that these are all unique, incremental implementations, -which do not require the whole training corpus to be present in main memory all at once. -With memory taken care of, I am now improving Distributed Computing, -to improve CPU efficiency, too. -If you feel you could contribute (by testing, providing use-cases or code), -please let me know.

    -

    Continue on to the next tutorial on Similarity Queries.

    -
    - - - - - -
    [1]Bradford. 2008. An empirical study of required dimensionality for large-scale latent semantic indexing applications.
    - - - - - -
    [2]Hoffman, Blei, Bach. 2010. Online learning for Latent Dirichlet Allocation.
    - - - - - -
    [3]Wang, Paisley, Blei. 2011. Online variational inference for the hierarchical Dirichlet process.
    - - - - - -
    [4]Halko, Martinsson, Tropp. 2009. Finding structure with randomness.
    - - - - - -
    [5]Řehůřek. 2011. Subspace tracking for Latent Semantic Analysis.
    -
    -
    - - -
    -
    -
    -
    -
    - - -
    -

    Navigation

    - -
    - - -
    - © Copyright 2009-2012, Radim Řehůřek <radimrehurek(at)seznam.cz>. - Last updated on Sep 16, 2012. -
    - - \ No newline at end of file diff --git a/docs/tut3.html b/docs/tut3.html deleted file mode 100644 index a1767e7475..0000000000 --- a/docs/tut3.html +++ /dev/null @@ -1,306 +0,0 @@ - - - - - - - - - - Similarity Queries — gensim - - - - - - - - - - - - - - - - - - - - - - - - - - - -
    -

    Navigation

    - -
    - - -
    -
    -

    Table Of Contents

    - - -

    Previous topic

    -

    Topics and Transformations

    -

    Next topic

    -

    Experiments on the English Wikipedia

    - - -
    -
    - - - - -
    -
    -
    -
    - -
    -

    Similarity Queries

    -

    Don’t forget to set

    -
    >>> import logging
->>> logging.basicConfig(format='%(asctime)s : %(levelname)s : %(message)s', level=logging.INFO)
-
    -
    -

    if you want to see logging events.

    -
    -

    Similarity interface

    -

    In the previous tutorials on Corpora and Vector Spaces and Topics and Transformations, we covered what it means -to create a corpus in the Vector Space Model and how to transform it between different -vector spaces. A common reason for such a charade is that we want to determine -similarity between pairs of documents, or the similarity between a specific document -and a set of other documents (such as a user query vs. indexed documents).

    -

    To show how this can be done in gensim, let us consider the same corpus as in the -previous examples (which really originally comes from Deerwester et al.’s -“Indexing by Latent Semantic Analysis” -seminal 1990 article):

    -
    >>> from gensim import corpora, models, similarities
->>> dictionary = corpora.Dictionary.load('/tmp/deerwester.dict')
->>> corpus = corpora.MmCorpus('/tmp/deerwester.mm') # comes from the first tutorial, "From strings to vectors"
->>> print corpus
-MmCorpus(9 documents, 12 features, 28 non-zero entries)
-
    -
    -

    To follow Deerwester’s example, we first use this tiny corpus to define a 2-dimensional -LSI space:

    -
    >>> lsi = models.LsiModel(corpus, id2word=dictionary, num_topics=2)
-
    -
    -

    Now suppose a user typed in the query “Human computer interaction”. We would -like to sort our nine corpus documents in decreasing order of relevance to this query. -Unlike modern search engines, here we only concentrate on a single aspect of possible -similarities—on apparent semantic relatedness of their texts (words). No hyperlinks, -no random-walk static ranks, just a semantic extension over the boolean keyword match:

    -
    >>> doc = "Human computer interaction"
->>> vec_bow = dictionary.doc2bow(doc.lower().split())
->>> vec_lsi = lsi[vec_bow] # convert the query to LSI space
->>> print vec_lsi
-[(0, -0.461821), (1, 0.070028)]
-
    -
    -

    In addition, we will be considering cosine similarity -to determine the similarity of two vectors. Cosine similarity is a standard measure -in Vector Space Modeling, but wherever the vectors represent probability distributions, -different similarity measures -may be more appropriate.

    -
    -

    Initializing query structures

    -

    To prepare for similarity queries, we need to enter all documents which we want -to compare against subsequent queries. In our case, they are the same nine documents -used for training LSI, converted to 2-D LSA space. But that’s only incidental, we -might also be indexing a different corpus altogether.

    -
    >>> index = similarities.MatrixSimilarity(lsi[corpus]) # transform corpus to LSI space and index it
-
    -
    -
    -

    Warning

    -

    The class similarities.MatrixSimilarity is only appropriate when the whole -set of vectors fits into memory. For example, a corpus of one million documents -would require 2GB of RAM in a 256-dimensional LSI space, when used with this class. -Without 2GB of free RAM, you would need to use the similarities.Similarity class. -This class operates in fixed memory, by splitting the index across multiple files on disk. -It uses similarities.MatrixSimilarity and similarities.SparseMatrixSimilarity internally, -so it is still fast, although slightly more complex.

    -
    -

    Index persistency is handled via the standard save() and load() functions:

    -
    >>> index.save('/tmp/deerwester.index')
->>> index = similarities.MatrixSimilarity.load('/tmp/deerwester.index')
-
    -
    -

    This is true for all similarity indexing classes (similarities.Similarity, -similarities.MatrixSimilarity and similarities.SparseMatrixSimilarity). -Also in the following, index can be an object of any of these. When in doubt, -use similarities.Similarity, as it is the most scalable version, and it also -supports adding more documents to the index later.

    -
    -
    -

    Performing queries

    -

    To obtain similarities of our query document against the nine indexed documents:

    -
    >>> sims = index[vec_lsi] # perform a similarity query against the corpus
->>> print list(enumerate(sims)) # print (document_number, document_similarity) 2-tuples
-[(0, 0.99809301), (1, 0.93748635), (2, 0.99844527), (3, 0.9865886), (4, 0.90755945),
-(5, -0.12416792), (6, -0.1063926), (7, -0.098794639), (8, 0.05004178)]
-
    -
    -

    Cosine measure returns similarities in the range <-1, 1> (the greater, the more similar), -so that the first document has a score of 0.99809301 etc.

    -

    With some standard Python magic we sort these similarities into descending -order, and obtain the final answer to the query “Human computer interaction”:

    -
    >>> sims = sorted(enumerate(sims), key=lambda item: -item[1])
->>> print sims # print sorted (document number, similarity score) 2-tuples
-[(2, 0.99844527), # The EPS user interface management system
-(0, 0.99809301), # Human machine interface for lab abc computer applications
-(3, 0.9865886), # System and human system engineering testing of EPS
-(1, 0.93748635), # A survey of user opinion of computer system response time
-(4, 0.90755945), # Relation of user perceived response time to error measurement
-(8, 0.050041795), # Graph minors A survey
-(7, -0.098794639), # Graph minors IV Widths of trees and well quasi ordering
-(6, -0.1063926), # The intersection graph of paths in trees
-(5, -0.12416792)] # The generation of random binary unordered trees
-
    -
    -

    (I added the original documents in their “string form” to the output comments, to -improve clarity.)

    -

    The thing to note here is that documents no. 2 ("The EPS user interface management system") -and 4 ("Relation of user perceived response time to error measurement") would never be returned by -a standard boolean fulltext search, because they do not share any common words with "Human -computer interaction". However, after applying LSI, we can observe that both of -them received quite high similarity scores (no. 2 is actually the most similar!), -which corresponds better to our intuition of -them sharing a “computer-human” related topic with the query. In fact, this semantic -generalization is the reason why we apply transformations and do topic modelling -in the first place.

    -
    -
    -
    -

    Where next?

    -

    Congratulations, you have finished the tutorials – now you know how gensim works :-) -To delve into more details, you can browse through the API documentation, -see the Wikipedia experiments or perhaps check out distributed computing in gensim.

    -

    Please remember that gensim is an experimental package, aimed at the NLP research community. -This means that:

    - -

    Gensim has no ambition to become an all-encompassing production level tool, with robust failure handling -and error recoveries. Its main goal is to help NLP newcomers try out popular algorithms -and to facilitate prototyping of new algorithms for NLP researchers.

    -
    -
    - - -
    -
    -
    -
    -
    - - -
    -

    Navigation

    - -
    - - -
    - © Copyright 2009-2012, Radim Řehůřek <radimrehurek(at)seznam.cz>. - Last updated on Sep 16, 2012. -
    - - \ No newline at end of file diff --git a/docs/tutorial.html b/docs/tutorial.html deleted file mode 100644 index ae2f077887..0000000000 --- a/docs/tutorial.html +++ /dev/null @@ -1,289 +0,0 @@ - - - - - - - - - - Tutorials — gensim - - - - - - - - - - - - - - - - - - - - - - - - - - -
    -

    Navigation

    - -
    - - -
    -
    -

    Table Of Contents

    - - -

    Previous topic

    -

    Installation

    -

    Next topic

    -

    Corpora and Vector Spaces

    - - -
    -
    - - - - -
    -
    -
    -
    - -
    -

    Tutorials

    -

    The tutorials are organized as a series of examples that highlight various features -of gensim. It is assumed that the reader is familiar with the Python language -and has read the Introduction.

    -

    The examples are divided into parts on:

    -
    - -
    -
    -

    Preliminaries

    -

    All the examples can be directly copied to your Python interpreter shell (assuming -you have gensim installed, of course). -IPython‘s cpaste command is especially handy for copypasting code fragments which include superfluous -characters, such as the leading >>>.

    -

    Gensim uses Python’s standard logging module to log various stuff at various -priority levels; to activate logging (this is optional), run

    -
    >>> import logging
->>> logging.basicConfig(format='%(asctime)s : %(levelname)s : %(message)s', level=logging.INFO)
-
    -
    -
    -
    -

    Quick Example

    -

    First, let’s import gensim and create a small corpus of nine documents [1]:

    -
    >>> from gensim import corpora, models, similarities
->>>
->>> corpus = [[(0, 1.0), (1, 1.0), (2, 1.0)],
->>>           [(2, 1.0), (3, 1.0), (4, 1.0), (5, 1.0), (6, 1.0), (8, 1.0)],
->>>           [(1, 1.0), (3, 1.0), (4, 1.0), (7, 1.0)],
->>>           [(0, 1.0), (4, 2.0), (7, 1.0)],
->>>           [(3, 1.0), (5, 1.0), (6, 1.0)],
->>>           [(9, 1.0)],
->>>           [(9, 1.0), (10, 1.0)],
->>>           [(9, 1.0), (10, 1.0), (11, 1.0)],
->>>           [(8, 1.0), (10, 1.0), (11, 1.0)]]
-
    -
    -

    Corpus is simply an object which, when iterated over, returns its documents represented -as sparse vectors.

    -

    If you’re familiar with the Vector Space Model, -you’ll probably know that the way you parse your documents and convert them to vectors -has major impact on the quality of any subsequent applications. If you’re not familiar -with VSM, we’ll bridge the gap between raw strings -and sparse vectors in the next tutorial -on Corpora and Vector Spaces.

    -
    -

    Note

    -

    In this example, the whole corpus is stored in memory, as a Python list. However, -the corpus interface only dictates that a corpus must support iteration over its -constituent documents. For very large corpora, it is advantageous to keep the -corpus on disk, and access its documents sequentially, one at a time. All the -operations and transformations are implemented in such a way that makes -them independent of the size of the corpus, memory-wise.

    -
    -

    Next, let’s initialize a transformation:

    -
    >>> tfidf = models.TfidfModel(corpus)
-
    -
    -

    A transformation is used to convert documents from one vector representation into another:

    -
    >>> vec = [(0, 1), (4, 1)]
->>> print tfidf[vec]
-[(0, 0.8075244), (4, 0.5898342)]
-
    -
    -

    Here, we used Tf-Idf, a simple -transformation which takes documents represented as bag-of-words counts and applies -a weighting which discounts common terms (or, equivalently, promotes rare terms). -It also scales the resulting vector to unit length (in the Euclidean norm).

    -

    Transformations are covered in detail in the tutorial on Topics and Transformations.

    -

    To transform the whole corpus via TfIdf and index it, in preparation for similarity queries:

    -
    >>> index = similarities.SparseMatrixSimilarity(tfidf[corpus])
-
    -
    -

    and to query the similarity of our query vector vec against every document in the corpus:

    -
    >>> sims = index[tfidf[vec]]
->>> print list(enumerate(sims))
-[(0, 0.4662244), (1, 0.19139354), (2, 0.24600551), (3, 0.82094586), (4, 0.0), (5, 0.0), (6, 0.0), (7, 0.0), (8, 0.0)]
-
    -
    -

    How to read this output? Document number zero (the first document) has a similarity score of 0.466=46.6%, -the second document has a similarity score of 19.1% etc.

    -

    Thus, according to TfIdf document representation and cosine similarity measure, -the most similar to our query document vec is document no. 3, with a similarity score of 82.1%. -Note that in the TfIdf representation, any documents which do not share any common features -with vec at all (documents no. 4–8) get a similarity score of 0.0. See the Similarity Queries tutorial for more detail.

    -
    - - - - - -
    [1]This is the same corpus as used in -Deerwester et al. (1990): Indexing by Latent Semantic Analysis, Table 2.
    -
    -
    - - -
    -
    -
    -
    -
    - - -
    -

    Navigation

    - -
    - - -
    - © Copyright 2009-2012, Radim Řehůřek <radimrehurek(at)seznam.cz>. - Last updated on Sep 16, 2012. -
    - - \ No newline at end of file diff --git a/docs/utils.html b/docs/utils.html deleted file mode 100644 index fe555788b6..0000000000 --- a/docs/utils.html +++ /dev/null @@ -1,458 +0,0 @@ - - - - - - - - - - utils – Various utility functions — gensim - - - - - - - - - - - - - - - - - - - - - - - - - - - -
    -

    Navigation

    - -
    - - -
    -
    -

    Previous topic

    -

    interfaces – Core gensim interfaces

    -

    Next topic

    -

    matutils – Math utils

    - - -
    -
    - - - - -
    -
    -
    -
    - -
    -

    utils – Various utility functions

    -

    This module contains various general utility functions.

    -
    -
    -class gensim.utils.FakeDict(num_terms)
    -

    Objects of this class act as dictionaries that map integer->str(integer), for -a specified range of integers <0, num_terms).

    -

    This is meant to avoid allocating real dictionaries when num_terms is huge, which -is a waste of memory.

    -
    -
    -keys()
    -

    Override the dict.keys() function, which is used to determine the maximum -internal id of a corpus = the vocabulary dimensionality.

    -

    HACK: To avoid materializing the whole range(0, self.num_terms), this returns -[self.num_terms - 1] only.

    -
    - -
    - -
    -
    -class gensim.utils.RepeatCorpus(corpus, reps)
    -

    Used in the tutorial on distributed computing and likely not useful anywhere else.

    -

    Wrap a corpus as another corpus of length reps. This is achieved by -repeating documents from corpus over and over again, until the requested -length len(result)==reps is reached. Repetition is done -on-the-fly=efficiently, via itertools.

    -
    >>> corpus = [[(1, 0.5)], []] # 2 documents
->>> list(RepeatCorpus(corpus, 5)) # repeat 2.5 times to get 5 documents
-[[(1, 0.5)], [], [(1, 0.5)], [], [(1, 0.5)]]
-
    -
    -
    -
    -classmethod load(fname)
    -

    Load a previously saved object from file (also see save).

    -
    - -
    -
    -save(fname)
    -

    Save the object to file via pickling (also see load).

    -
    - -
    - -
    -
    -class gensim.utils.SaveLoad
    -

    Objects which inherit from this class have save/load functions, which un/pickle -them to disk.

    -

    This uses cPickle for de/serializing, so objects must not contains unpicklable -attributes, such as lambda functions etc.

    -
    -
    -classmethod load(fname)
    -

    Load a previously saved object from file (also see save).

    -
    - -
    -
    -save(fname)
    -

    Save the object to file via pickling (also see load).

    -
    - -
    - -
    -
    -gensim.utils.any2unicode(text, encoding='utf8', errors='strict')
    -

    Convert a string (bytestring in encoding or unicode), to unicode.

    -
    - -
    -
    -gensim.utils.any2utf8(text, errors='strict', encoding='utf8')
    -

    Convert a string (unicode or bytestring in encoding), to bytestring in utf8.

    -
    - -
    -
    -gensim.utils.chunkize(corpus, chunksize, maxsize=0, as_numpy=False)
    -

    Split a stream of values into smaller chunks. -Each chunk is of length chunksize, except the last one which may be smaller. -A once-only input stream (corpus from a generator) is ok, chunking is done -efficiently via itertools.

    -

    If maxsize > 1, don’t wait idly in between successive chunk yields, but -rather keep filling a short queue (of size at most maxsize) with forthcoming -chunks in advance. This is realized by starting a separate process, and is -meant to reduce I/O delays, which can be significant when corpus comes -from a slow medium (like harddisk).

    -

    If maxsize==0, don’t fool around with parallelism and simply yield the chunksize -via chunkize_serial() (no I/O optimizations).

    -
    >>> for chunk in chunkize(xrange(10), 4): print chunk
-[0, 1, 2, 3]
-[4, 5, 6, 7]
-[8, 9]
-
    -
    -
    - -
    -
    -gensim.utils.chunkize_serial(iterable, chunksize, as_numpy=False)
    -

    Return elements from the iterable in chunksize-ed lists. The last returned -element may be smaller (if length of collection is not divisible by chunksize).

    -
    >>> print list(grouper(xrange(10), 3))
-[[0, 1, 2], [3, 4, 5], [6, 7, 8], [9]]
-
    -
    -
    - -
    - -

    Recursively copy a directory ala shutils.copytree, but hardlink files -instead of copying. Available on UNIX systems only.

    -
    - -
    -
    -gensim.utils.deaccent(text)
    -

    Remove accentuation from the given string. Input text is either a unicode string or utf8 encoded bytestring.

    -

    Return input string with accents removed, as unicode.

    -
    >>> deaccent("Šéf chomutovských komunistů dostal poštou bílý prášek")
-u'Sef chomutovskych komunistu dostal postou bily prasek'
-
    -
    -
    - -
    -
    -gensim.utils.decode_htmlentities(text)
    -

    Decode HTML entities in text, coded as hex, decimal or named.

    -

    Adapted from http://github.com/sku/python-twitter-ircbot/blob/321d94e0e40d0acc92f5bf57d126b57369da70de/html_decode.py

    -
    >>> u = u'E tu vivrai nel terrore - L&#x27;aldil&#xE0; (1981)'
->>> print decode_htmlentities(u).encode('UTF-8')
-E tu vivrai nel terrore - L'aldilà (1981)
->>> print decode_htmlentities("l&#39;eau")
-l'eau
->>> print decode_htmlentities("foo &lt; bar")
-foo < bar
-
    -
    -
    - -
    -
    -gensim.utils.dict_from_corpus(corpus)
    -

    Scan corpus for all word ids that appear in it, then construct and return a mapping -which maps each wordId -> str(wordId).

    -

    This function is used whenever words need to be displayed (as opposed to just -their ids) but no wordId->word mapping was provided. The resulting mapping -only covers words actually used in the corpus, up to the highest wordId found.

    -
    - -
    -
    -gensim.utils.getNS()
    -

    Return a Pyro name server proxy. If there is no name server running, -start one on 0.0.0.0 (all interfaces), as a background process.

    -
    - -
    -
    -gensim.utils.get_max_id(corpus)
    -

    Return highest feature id that appears in the corpus.

    -

    For empty corpora (no features at all), return -1.

    -
    - -
    -
    -gensim.utils.get_my_ip()
    -

    Try to obtain our external ip (from the pyro nameserver’s point of view)

    -

    This tries to sidestep the issue of bogus /etc/hosts entries and other -local misconfigurations, which often mess up hostname resolution.

    -

    If all else fails, fall back to simple socket.gethostbyname() lookup.

    -
    - -
    -
    -gensim.utils.grouper(iterable, chunksize, as_numpy=False)
    -

    Return elements from the iterable in chunksize-ed lists. The last returned -element may be smaller (if length of collection is not divisible by chunksize).

    -
    >>> print list(grouper(xrange(10), 3))
-[[0, 1, 2], [3, 4, 5], [6, 7, 8], [9]]
-
    -
    -
    - -
    -
    -gensim.utils.is_corpus(obj)
    -

    Check whether obj is a corpus. Return (is_corpus, new) 2-tuple, where -new is obj if obj was an iterable, or new yields the same sequence as -obj if it was an iterator.

    -

    obj is a corpus if it supports iteration over documents, where a document -is in turn anything that acts as a sequence of 2-tuples (int, float).

    -

    Note: An “empty” corpus (empty input sequence) is ambiguous, so in this case the -result is forcefully defined as is_corpus=False.

    -
    - -
    -
    -gensim.utils.lemmatize(content)
    -

    Use the English lemmatizer from the pattern package to extract tokens in -their base form (lemmas: “are, is, being”->”be” etc.). -This is a smarter version of stemming.

    -
    - -
    -
    -gensim.utils.pickle(obj, fname, protocol=-1)
    -

    Pickle object obj to file fname.

    -
    - -
    -
    -gensim.utils.pyro_daemon(name, obj, random_suffix=False, ip=None, port=None)
    -

    Register object with name server (starting the name server if not running -yet) and block until the daemon is terminated. The object is registered under -name, or name`+ some random suffix if `random_suffix is set.

    -
    - -
    -
    -gensim.utils.revdict(d)
    -

    Reverse a dictionary mapping.

    -

    When two keys map to the same value, only one of them will be kept in the -result (which one is kept is arbitrary).

    -
    - -
    -
    -gensim.utils.simple_preprocess(doc)
    -

    Convert a document into a list of tokens.

    -

    This lowercases, tokenizes, stems, normalizes etc. – the output are final, -utf8 encoded strings that won’t be processed any further.

    -
    - -
    -
    -gensim.utils.synchronous(tlockname)
    -

    A decorator to place an instance-based lock around a method.

    -

    Adapted from http://code.activestate.com/recipes/577105-synchronization-decorator-for-class-methods/

    -
    - -
    -
    -gensim.utils.to_unicode(text, encoding='utf8', errors='strict')
    -

    Convert a string (bytestring in encoding or unicode), to unicode.

    -
    - -
    -
    -gensim.utils.to_utf8(text, errors='strict', encoding='utf8')
    -

    Convert a string (unicode or bytestring in encoding), to bytestring in utf8.

    -
    - -
    -
    -gensim.utils.tokenize(text, lowercase=False, deacc=False, errors='strict', to_lower=False, lower=False)
    -

    Iteratively yield tokens as unicode strings, optionally also lowercasing them -and removing accent marks.

    -

    Input text may be either unicode or utf8-encoded byte string.

    -

    The tokens on output are maximal contiguous sequences of alphabetic -characters (no digits!).

    -
    >>> list(tokenize('Nic nemůže letět rychlostí vyšší, než 300 tisíc kilometrů za sekundu!', deacc = True))
-[u'Nic', u'nemuze', u'letet', u'rychlosti', u'vyssi', u'nez', u'tisic', u'kilometru', u'za', u'sekundu']
-
    -
    -
    - -
    -
    -gensim.utils.toptexts(query, texts, index, n=10)
    -

    Debug fnc to help inspect the top n most similar documents (according to a -similarity index index), to see if they are actually related to the query.

    -

    texts is any object that can return something insightful for each document -via texts[docid], such as its fulltext or snippet.

    -

    Return a list of 3-tuples (docid, doc’s similarity to the query, texts[docid]).

    -
    - -
    -
    -gensim.utils.unpickle(fname)
    -

    Load pickled object from fname

    -
    - -
    -
    -gensim.utils.upload_chunked(server, docs, chunksize=1000, preprocess=None)
    -

    Memory-friendly upload of documents to a SimServer (or Pyro SimServer proxy).

    -

    Use this function to train or index large collections – avoid sending the -entire corpus over the wire as a single Pyro in-memory object. The documents -will be sent in smaller chunks, of chunksize documents each.

    -
    - -
    - - -
    -
    -
    -
    -
    - - -
    -

    Navigation

    - -
    - - -
    - © Copyright 2009-2012, Radim Řehůřek <radimrehurek(at)seznam.cz>. - Last updated on Sep 16, 2012. -
    - - \ No newline at end of file diff --git a/docs/wiki.html b/docs/wiki.html deleted file mode 100644 index 69b293bce0..0000000000 --- a/docs/wiki.html +++ /dev/null @@ -1,368 +0,0 @@ - - - - - - - - - - Experiments on the English Wikipedia — gensim - - - - - - - - - - - - - - - - - - - - - - - - - - -
    -

    Navigation

    - -
    - - -
    -
    -

    Table Of Contents

    - - -

    Previous topic

    -

    Distributed Latent Dirichlet Allocation

    -

    Next topic

    -

    API Reference

    - - -
    -
    - - - - -
    -
    -
    -
    - -
    -

    Experiments on the English Wikipedia

    -

    To test gensim performance, we run it against the English version of Wikipedia.

    -

    This page describes the process of obtaining and processing Wikipedia, so that -anyone can reproduce the results. It is assumed you have gensim properly installed.

    -
    -

    Preparing the corpus

    -
      -

    1. First, download the dump of all Wikipedia articles from http://download.wikimedia.org/enwiki/ -(you want a file like enwiki-latest-pages-articles.xml.bz2). This file is about 8GB in size -and contains (a compressed version of) all articles from the English Wikipedia.

      -
      2. -

    2. Convert the articles to plain text (process Wiki markup) and store the result as -sparse TF-IDF vectors. In Python, this is easy to do on-the-fly and we don’t -even need to uncompress the whole archive to disk. There is a script included in -gensim that does just that, run:

      -
      $ python -m gensim.scripts.make_wiki
      -
      -
      3. -
    -
    -

    Note

    -

    This pre-processing step makes two passes over the 8.2GB compressed wiki dump (one to extract -the dictionary, one to create and store the sparse vectors) and takes about -9 hours on my laptop, so you may want to go have a coffee or two.

    -

    Also, you will need about 35GB of free disk space to store the sparse output vectors. -I recommend compressing these files immediately, e.g. with bzip2 (down to ~13GB). Gensim -can work with compressed files directly, so this lets you save disk space.

    -
    -
    -
    -

    Latent Sematic Analysis

    -

    First let’s load the corpus iterator and dictionary, created in the second step above:

    -
    >>> import logging, gensim, bz2
->>> logging.basicConfig(format='%(asctime)s : %(levelname)s : %(message)s', level=logging.INFO)
-
->>> # load id->word mapping (the dictionary), one of the results of step 2 above
->>> id2word = gensim.corpora.Dictionary.load_from_text('wiki_en_wordids.txt')
->>> # load corpus iterator
->>> mm = gensim.corpora.MmCorpus('wiki_en_tfidf.mm')
->>> # mm = gensim.corpora.MmCorpus(bz2.BZ2File('wiki_en_tfidf.mm.bz2')) # use this if you compressed the TFIDF output (recommended)
-
->>> print mm
-MmCorpus(3931787 documents, 100000 features, 756379027 non-zero entries)
-
    -
    -

    We see that our corpus contains 3.9M documents, 100K features (distinct -tokens) and 0.76G non-zero entries in the sparse TF-IDF matrix. The Wikipedia corpus -contains about 2.24 billion tokens in total.

    -

    Now we’re ready to compute LSA of the English Wikipedia:

    -
    >>> # extract 400 LSI topics; use the default one-pass algorithm
->>> lsi = gensim.models.lsimodel.LsiModel(corpus=mm, id2word=id2word, num_topics=400)
-
->>> # print the most contributing words (both positively and negatively) for each of the first ten topics
->>> lsi.print_topics(10)
-topic #0(332.762): 0.425*"utc" + 0.299*"talk" + 0.293*"page" + 0.226*"article" + 0.224*"delete" + 0.216*"discussion" + 0.205*"deletion" + 0.198*"should" + 0.146*"debate" + 0.132*"be"
-topic #1(201.852): 0.282*"link" + 0.209*"he" + 0.145*"com" + 0.139*"his" + -0.137*"page" + -0.118*"delete" + 0.114*"blacklist" + -0.108*"deletion" + -0.105*"discussion" + 0.100*"diff"
-topic #2(191.991): -0.565*"link" + -0.241*"com" + -0.238*"blacklist" + -0.202*"diff" + -0.193*"additions" + -0.182*"users" + -0.158*"coibot" + -0.136*"user" + 0.133*"he" + -0.130*"resolves"
-topic #3(141.284): -0.476*"image" + -0.255*"copyright" + -0.245*"fair" + -0.225*"use" + -0.173*"album" + -0.163*"cover" + -0.155*"resolution" + -0.141*"licensing" + 0.137*"he" + -0.121*"copies"
-topic #4(130.909): 0.264*"population" + 0.246*"age" + 0.243*"median" + 0.213*"income" + 0.195*"census" + -0.189*"he" + 0.184*"households" + 0.175*"were" + 0.167*"females" + 0.166*"males"
-topic #5(120.397): 0.304*"diff" + 0.278*"utc" + 0.213*"you" + -0.171*"additions" + 0.165*"talk" + -0.159*"image" + 0.159*"undo" + 0.155*"www" + -0.152*"page" + 0.148*"contribs"
-topic #6(115.414): -0.362*"diff" + -0.203*"www" + 0.197*"you" + -0.180*"undo" + -0.180*"kategori" + 0.164*"users" + 0.157*"additions" + -0.150*"contribs" + -0.139*"he" + -0.136*"image"
-topic #7(111.440): 0.429*"kategori" + 0.276*"categoria" + 0.251*"category" + 0.207*"kategorija" + 0.198*"kategorie" + -0.188*"diff" + 0.163*"категория" + 0.153*"categoría" + 0.139*"kategoria" + 0.133*"categorie"
-topic #8(109.907): 0.385*"album" + 0.224*"song" + 0.209*"chart" + 0.204*"band" + 0.169*"released" + 0.151*"music" + 0.142*"diff" + 0.141*"vocals" + 0.138*"she" + 0.132*"guitar"
-topic #9(102.599): -0.237*"league" + -0.214*"he" + -0.180*"season" + -0.174*"football" + -0.166*"team" + 0.159*"station" + -0.137*"played" + -0.131*"cup" + 0.131*"she" + -0.128*"utc"
-
    -
    -

    Creating the LSI model of Wikipedia takes about 4 hours and 9 minutes on my laptop [1]. -That’s about 16,000 documents per minute, including all I/O.

    -
    -

    Note

    -

    If you need your results even faster, see the tutorial on Distributed Computing. Note -that the BLAS libraries inside gensim make use of multiple cores transparently, so the same data -will be processed faster on a multicore machine “for free”, without any distributed setup.

    -
    -

    We see that the total processing time is dominated by the preprocessing step of -preparing the TF-IDF corpus from a raw Wikipedia XML dump, which took 9h. [2]

    -

    The algorithm used in gensim only needs to see each input document once, so it -is suitable for environments where the documents come as a non-repeatable stream, -or where the cost of storing/iterating over the corpus multiple times is too high.

    -
    -
    -

    Latent Dirichlet Allocation

    -

    As with Latent Semantic Analysis above, first load the corpus iterator and dictionary:

    -
    >>> import logging, gensim, bz2
->>> logging.basicConfig(format='%(asctime)s : %(levelname)s : %(message)s', level=logging.INFO)
-
->>> # load id->word mapping (the dictionary), one of the results of step 2 above
->>> id2word = gensim.corpora.Dictionary.load_from_text('wiki_en_wordids.txt')
->>> # load corpus iterator
->>> mm = gensim.corpora.MmCorpus('wiki_en_tfidf.mm')
->>> # mm = gensim.corpora.MmCorpus(bz2.BZ2File('wiki_en_tfidf.mm.bz2')) # use this if you compressed the TFIDF output
-
->>> print mm
-MmCorpus(3931787 documents, 100000 features, 756379027 non-zero entries)
-
    -
    -

    We will run online LDA (see Hoffman et al. [3]), which is an algorithm that takes a chunk of documents, -updates the LDA model, takes another chunk, updates the model etc. Online LDA can be contrasted -with batch LDA, which processes the whole corpus (one full pass), then updates -the model, then another pass, another update... The difference is that given a -reasonably stationary document stream (not much topic drift), the online updates -over the smaller chunks (subcorpora) are pretty good in themselves, so that the -model estimation converges faster. As a result, we will perhaps only need a single full -pass over the corpus: if the corpus has 3 million articles, and we update once after -every 10,000 articles, this means we will have done 300 updates in one pass, quite likely -enough to have a very accurate topics estimate:

    -
    >>> # extract 100 LDA topics, using 1 pass and updating once every 1 chunk (10,000 documents)
->>> lda = gensim.models.ldamodel.LdaModel(corpus=mm, id2word=id2word, num_topics=100, update_every=1, chunksize=10000, passes=1)
-using serial LDA version on this node
-running online LDA training, 100 topics, 1 passes over the supplied corpus of 3931787 documents, updating model once every 10000 documents
-...
-
    -
    -

    Unlike LSA, the topics coming from LDA are easier to interpret:

    -
    >>> # print the most contributing words for 20 randomly selected topics
->>> lda.print_topics(20)
-topic #0: 0.009*river + 0.008*lake + 0.006*island + 0.005*mountain + 0.004*area + 0.004*park + 0.004*antarctic + 0.004*south + 0.004*mountains + 0.004*dam
-topic #1: 0.026*relay + 0.026*athletics + 0.025*metres + 0.023*freestyle + 0.022*hurdles + 0.020*ret + 0.017*divisão + 0.017*athletes + 0.016*bundesliga + 0.014*medals
-topic #2: 0.002*were + 0.002*he + 0.002*court + 0.002*his + 0.002*had + 0.002*law + 0.002*government + 0.002*police + 0.002*patrolling + 0.002*their
-topic #3: 0.040*courcelles + 0.035*centimeters + 0.023*mattythewhite + 0.021*wine + 0.019*stamps + 0.018*oko + 0.017*perennial + 0.014*stubs + 0.012*ovate + 0.011*greyish
-topic #4: 0.039*al + 0.029*sysop + 0.019*iran + 0.015*pakistan + 0.014*ali + 0.013*arab + 0.010*islamic + 0.010*arabic + 0.010*saudi + 0.010*muhammad
-topic #5: 0.020*copyrighted + 0.020*northamerica + 0.014*uncopyrighted + 0.007*rihanna + 0.005*cloudz + 0.005*knowles + 0.004*gaga + 0.004*zombie + 0.004*wigan + 0.003*maccabi
-topic #6: 0.061*israel + 0.056*israeli + 0.030*sockpuppet + 0.025*jerusalem + 0.025*tel + 0.023*aviv + 0.022*palestinian + 0.019*ifk + 0.016*palestine + 0.014*hebrew
-topic #7: 0.015*melbourne + 0.014*rovers + 0.013*vfl + 0.012*australian + 0.012*wanderers + 0.011*afl + 0.008*dinamo + 0.008*queensland + 0.008*tracklist + 0.008*brisbane
-topic #8: 0.011*film + 0.007*her + 0.007*she + 0.004*he + 0.004*series + 0.004*his + 0.004*episode + 0.003*films + 0.003*television + 0.003*best
-topic #9: 0.019*wrestling + 0.013*château + 0.013*ligue + 0.012*discus + 0.012*estonian + 0.009*uci + 0.008*hockeyarchives + 0.008*wwe + 0.008*estonia + 0.007*reign
-topic #10: 0.078*edits + 0.059*notability + 0.035*archived + 0.025*clearer + 0.022*speedy + 0.021*deleted + 0.016*hook + 0.015*checkuser + 0.014*ron + 0.011*nominator
-topic #11: 0.013*admins + 0.009*acid + 0.009*molniya + 0.009*chemical + 0.007*ch + 0.007*chemistry + 0.007*compound + 0.007*anemone + 0.006*mg + 0.006*reaction
-topic #12: 0.018*india + 0.013*indian + 0.010*tamil + 0.009*singh + 0.008*film + 0.008*temple + 0.006*kumar + 0.006*hindi + 0.006*delhi + 0.005*bengal
-topic #13: 0.047*bwebs + 0.024*malta + 0.020*hobart + 0.019*basa + 0.019*columella + 0.019*huon + 0.018*tasmania + 0.016*popups + 0.014*tasmanian + 0.014*modèle
-topic #14: 0.014*jewish + 0.011*rabbi + 0.008*bgwhite + 0.008*lebanese + 0.007*lebanon + 0.006*homs + 0.005*beirut + 0.004*jews + 0.004*hebrew + 0.004*caligari
-topic #15: 0.025*german + 0.020*der + 0.017*von + 0.015*und + 0.014*berlin + 0.012*germany + 0.012*die + 0.010*des + 0.008*kategorie + 0.007*cross
-topic #16: 0.003*can + 0.003*system + 0.003*power + 0.003*are + 0.003*energy + 0.002*data + 0.002*be + 0.002*used + 0.002*or + 0.002*using
-topic #17: 0.049*indonesia + 0.042*indonesian + 0.031*malaysia + 0.024*singapore + 0.022*greek + 0.021*jakarta + 0.016*greece + 0.015*dord + 0.014*athens + 0.011*malaysian
-topic #18: 0.031*stakes + 0.029*webs + 0.018*futsal + 0.014*whitish + 0.013*hyun + 0.012*thoroughbred + 0.012*dnf + 0.012*jockey + 0.011*medalists + 0.011*racehorse
-topic #19: 0.119*oblast + 0.034*uploaded + 0.034*uploads + 0.033*nordland + 0.025*selsoviet + 0.023*raion + 0.022*krai + 0.018*okrug + 0.015*hålogaland + 0.015*russiae + 0.020*manga + 0.017*dragon + 0.012*theme + 0.011*dvd + 0.011*super + 0.011*hunter + 0.009*ash + 0.009*dream + 0.009*angel
-
    -
    -

    Creating this LDA model of Wikipedia takes about 6 hours and 20 minutes on my laptop [1]. -If you need your results faster, consider running Distributed Latent Dirichlet Allocation on a cluster of -computers.

    -

    Note two differences between the LDA and LSA runs: we asked LSA -to extract 400 topics, LDA only 100 topics (so the difference in speed is in fact -even greater). Secondly, the LSA implementation in gensim is truly online: if the nature of the input -stream changes in time, LSA will re-orient itself to reflect these changes, in a reasonably -small amount of updates. In contrast, LDA is not truly online (the name of the [3] -article notwithstanding), as the impact of later updates on the model gradually -diminishes. If there is topic drift in the input document stream, LDA will get -confused and be increasingly slower at adjusting itself to the new state of affairs.

    -

    In short, be careful if using LDA to incrementally add new documents to the model -over time. Batch usage of LDA, where the entire training corpus is either known beforehand or does -not exhibit topic drift, is ok and not affected.

    -

    To run batch LDA (not online), train LdaModel with:

    -
    >>> # extract 100 LDA topics, using 20 full passes, no online updates
->>> lda = gensim.models.ldamodel.LdaModel(corpus=mm, id2word=id2word, num_topics=100, update_every=0, passes=20)
-
    -
    -

    As usual, a trained model can used be to transform new, unseen documents (plain bag-of-words count vectors) -into LDA topic distributions:

    -
    >>> doc_lda = lda[doc_bow]
-
    -
    -
    - - - - - -
    [1](1, 2) My laptop = MacBook Pro, Intel Core i7 2.3GHz, 16GB DDR3 RAM, OS X with libVec.
    - - - - - -
    [2]

    Here we’re mostly interested in performance, but it is interesting to look at the -retrieved LSA concepts, too. I am no Wikipedia expert and don’t see into Wiki’s bowels, -but Brian Mingus had this to say about the result:

    -
    There appears to be a lot of noise in your dataset. The first three topics
-in your list appear to be meta topics, concerning the administration and
-cleanup of Wikipedia. These show up because you didn't exclude templates
-such as these, some of which are included in most articles for quality
-control: http://en.wikipedia.org/wiki/Wikipedia:Template_messages/Cleanup
-
-The fourth and fifth topics clearly shows the influence of bots that import
-massive databases of cities, countries, etc. and their statistics such as
-population, capita, etc.
-
-The sixth shows the influence of sports bots, and the seventh of music bots.
    -
    -

    So the top ten concepts are apparently dominated by Wikipedia robots and expanded -templates; this is a good reminder that LSA is a powerful tool for data analysis, -but no silver bullet. As always, it’s garbage in, garbage out... -By the way, improvements to the Wiki markup parsing code are welcome :-)

    -
    - - - - - -
    [3](1, 2) Hoffman, Blei, Bach. 2010. Online learning for Latent Dirichlet Allocation -[pdf] [code]
    -
    -
    - - -
    -
    -
    -
    -
    - - -
    -

    Navigation

    - -
    - - -
    - © Copyright 2009-2012, Radim Řehůřek <radimrehurek(at)seznam.cz>. - Last updated on Sep 16, 2012. -
    - - \ No newline at end of file diff --git a/gensim/corpora/bleicorpus.py b/gensim/corpora/bleicorpus.py index e38f4eb85b..b4c2405da4 100644 --- a/gensim/corpora/bleicorpus.py +++ b/gensim/corpora/bleicorpus.py @@ -11,6 +11,7 @@ from __future__ import with_statement +from os import path import logging from gensim import interfaces, utils @@ -46,7 +47,19 @@ def __init__(self, fname, fname_vocab=None): logger.info("loading corpus from %s" % fname) if fname_vocab is None: - fname_vocab = fname + '.vocab' + fname_base, _ = path.splitext(fname) + fname_dir = path.dirname(fname) + for fname_vocab in [ + fname + '.vocab', + fname + '/vocab.txt', + fname_base + '.vocab', + fname_dir + '/vocab.txt', + ]: + if path.exists(fname_vocab): + break + else: + raise IOError('BleiCorpus: could not find vocabulary file') + self.fname = fname words = [word.rstrip() for word in open(fname_vocab)] diff --git a/gensim/corpora/dictionary.py b/gensim/corpora/dictionary.py index d61997322b..7e02278d0f 100644 --- a/gensim/corpora/dictionary.py +++ b/gensim/corpora/dictionary.py @@ -11,7 +11,8 @@ Dictionaries can be created from a corpus and can later be pruned according to document frequency (removing (un)common words via the :func:`Dictionary.filter_extremes` method), -save/loaded from disk (via :func:`Dictionary.save` and :func:`Dictionary.load` methods) etc. +save/loaded from disk (via :func:`Dictionary.save` and :func:`Dictionary.load` methods), merged +with other dictionary (:func:`Dictionary.merge_with`) etc. """ from __future__ import with_statement @@ -111,7 +112,9 @@ def doc2bow(self, document, allow_update=False, return_missing=False): """ result = {} missing = {} - document = sorted(document) + if isinstance(document, basestring): + raise TypeError("doc2bow expects an array of utf8 tokens on input, not a string") + document = sorted(utils.to_utf8(token) for token in document) # construct (word, frequency) mapping. in python3 this is done simply # using Counter(), but here i use itertools.groupby() for the job for word_norm, group in itertools.groupby(document): @@ -208,6 +211,7 @@ def compactify(self): # reassign mappings to new ids self.token2id = dict((token, idmap[tokenid]) for token, tokenid in self.token2id.iteritems()) + self.id2token = {} self.dfs = dict((idmap[tokenid], freq) for tokenid, freq in self.dfs.iteritems()) @@ -219,11 +223,57 @@ def save_as_text(self, fname): Note: use `save`/`load` to store in binary format instead (pickle). """ logger.info("saving dictionary mapping to %s" % fname) - with open(fname, 'wb') as fout: + with utils.smart_open(fname, 'wb') as fout: for token, tokenid in sorted(self.token2id.iteritems()): fout.write("%i\t%s\t%i\n" % (tokenid, token, self.dfs.get(tokenid, 0))) + def merge_with(self, other): + """ + Merge another dictionary into this dictionary, mapping same tokens to the + same ids and new tokens to new ids. The purpose is to merge two corpora + created using two different dictionaries, one from `self` and one from `other`. + + `other` can be any id=>word mapping (a dict, a Dictionary object, ...). + + Return a transformation object which, when accessed as `result[doc_from_other_corpus]`, + will convert documents from a corpus built using the `other` dictionary + into a document using the new, merged dictionary (see :class:`gensim.interfaces.TransformationABC`). + + Example: + + >>> dict1 = Dictionary(some_documents) + >>> dict2 = Dictionary(other_documents) # ids not compatible with dict1! + >>> dict2_to_dict1 = dict1.merge_with(dict2) + >>> # now we can merge corpora from the two incompatible dictionaries into one + >>> merged_corpus = itertools.chain(some_corpus_from_dict1, dict2_to_dict1[some_corpus_from_dict2]) + + """ + old2new = {} + for other_id, other_token in other.iteritems(): + if other_token in self.token2id: + new_id = self.token2id[other_token] + else: + new_id = len(self.token2id) + self.token2id[other_token] = new_id + self.dfs[new_id] = 0 + old2new[other_id] = new_id + try: + self.dfs[new_id] += other.dfs[other_id] + except: + # `other` isn't a Dictionary (probably just a dict) => ignore dfs, keep going + pass + try: + self.num_docs += other.num_docs + self.num_nnz += other.num_nnz + self.num_pos += other.num_pos + except: + pass + + import gensim.models + return gensim.models.VocabTransform(old2new) + + @staticmethod def load_from_text(fname): """ @@ -231,7 +281,7 @@ def load_from_text(fname): Mirror function to `save_as_text`. """ result = Dictionary() - with open(fname, 'rb') as f: + with utils.smart_open(fname, 'rb') as f: for lineno, line in enumerate(f): try: wordid, word, docfreq = line[:-1].split('\t') @@ -239,6 +289,8 @@ def load_from_text(fname): raise ValueError("invalid line in dictionary file %s: %s" % (fname, line.strip())) wordid = int(wordid) + if word in result.token2id: + raise KeyError('token %s is defined as ID %d and as ID %d' % (word, wordid, result.token2id[word])) result.token2id[word] = wordid result.dfs[wordid] = int(docfreq) return result @@ -274,4 +326,3 @@ def from_corpus(corpus): (result, result.num_docs, result.num_pos)) return result #endclass Dictionary - diff --git a/gensim/corpora/hashdictionary.py b/gensim/corpora/hashdictionary.py index 5d851db596..dfca3be296 100644 --- a/gensim/corpora/hashdictionary.py +++ b/gensim/corpora/hashdictionary.py @@ -208,7 +208,7 @@ def save_as_text(self, fname): Note: use `save`/`load` to store in binary format instead (pickle). """ logger.info("saving HashDictionary mapping to %s" % fname) - with open(fname, 'wb') as fout: + with utils.smart_open(fname, 'wb') as fout: for tokenid in self.keys(): words = sorted(self[tokenid]) if words: diff --git a/gensim/corpora/indexedcorpus.py b/gensim/corpora/indexedcorpus.py index 51e2dd34ba..a9aaac71ee 100644 --- a/gensim/corpora/indexedcorpus.py +++ b/gensim/corpora/indexedcorpus.py @@ -114,4 +114,3 @@ def __getitem__(self, docno): raise RuntimeError("cannot call corpus[docid] without an index") return self.docbyoffset(self.index[docno]) #endclass IndexedCorpus - diff --git a/gensim/corpora/mmcorpus.py b/gensim/corpora/mmcorpus.py index eaa78e292d..dcd5be7d2d 100644 --- a/gensim/corpora/mmcorpus.py +++ b/gensim/corpora/mmcorpus.py @@ -45,7 +45,6 @@ def save_corpus(fname, corpus, id2word=None, progress_cnt=1000): call it directly, call `serialize` instead. """ logger.info("storing corpus in Matrix Market format to %s" % fname) - return matutils.MmWriter.write_corpus(fname, corpus, index=True, progress_cnt=progress_cnt) + num_terms = len(id2word) if id2word is not None else None + return matutils.MmWriter.write_corpus(fname, corpus, num_terms=num_terms, index=True, progress_cnt=progress_cnt) #endclass MmCorpus - - diff --git a/gensim/corpora/svmlightcorpus.py b/gensim/corpora/svmlightcorpus.py index a5b264a7a6..52f3f97033 100644 --- a/gensim/corpora/svmlightcorpus.py +++ b/gensim/corpora/svmlightcorpus.py @@ -14,7 +14,6 @@ import logging -from gensim import interfaces, utils from gensim.corpora import IndexedCorpus @@ -74,13 +73,14 @@ def save_corpus(fname, corpus, id2word=None, labels=False): """ Save a corpus in the SVMlight format. - The SVMlight `` class tag is set to 0 for all documents. + The SVMlight `` class tag is taken from the `labels` array, or set + to 0 for all documents if `labels` is not supplied. This function is automatically called by `SvmLightCorpus.serialize`; don't call it directly, call `serialize` instead. """ logger.info("converting corpus to SVMlight format: %s" % fname) - + offsets = [] with open(fname, 'w') as fout: for docno, doc in enumerate(corpus): @@ -100,13 +100,15 @@ def docbyoffset(self, offset): def line2doc(self, line): + """ + Create a document from a single line (string) in SVMlight format + """ line = line[: line.find('#')].strip() if not line: return None # ignore comments and empty lines parts = line.split() if not parts: - raise ValueError('invalid format at line no. %i in %s' % - (lineNo, self.fname)) + raise ValueError('invalid line format in %s' % self.fname) target, fields = parts[0], [part.rsplit(':', 1) for part in parts[1:]] doc = [(int(p1) - 1, float(p2)) for p1, p2 in fields if p1 != 'qid'] # ignore 'qid' features, convert 1-based feature ids to 0-based return doc @@ -114,7 +116,9 @@ def line2doc(self, line): @staticmethod def doc2line(doc, label=0): + """ + Output the document in SVMlight format, as a string. Inverse function to `line2doc`. + """ pairs = ' '.join("%i:%s" % (termid + 1, termval) for termid, termval in doc) # +1 to convert 0-base to 1-base return str(label) + " %s\n" % pairs #endclass SvmLightCorpus - diff --git a/gensim/corpora/wikicorpus.py b/gensim/corpora/wikicorpus.py index d6f2a84c29..4ea81d9bfd 100755 --- a/gensim/corpora/wikicorpus.py +++ b/gensim/corpora/wikicorpus.py @@ -285,7 +285,7 @@ def get_texts(self): pool = multiprocessing.Pool(self.processes) # process the corpus in smaller chunks of docs, because multiprocessing.Pool # is dumb and would load the entire input into RAM at once... - for group in utils.chunkize(texts, chunksize=10 * pool._processes, maxsize=1): + for group in utils.chunkize(texts, chunksize=10 * self.processes, maxsize=1): for tokens in pool.imap(process_article, group): # chunksize=10): articles_all += 1 positions_all += len(tokens) diff --git a/gensim/interfaces.py b/gensim/interfaces.py index 8eca51c730..49d19e41f5 100644 --- a/gensim/interfaces.py +++ b/gensim/interfaces.py @@ -119,9 +119,16 @@ class TransformationABC(utils.SaveLoad): """ Interface for transformations. A 'transformation' is any object which accepts a sparse document via the dictionary notation `[]` and returns another sparse - document in its stead. + document in its stead:: + + >>> transformed_doc = transformation[doc] + + or also:: + + >>> transformed_corpus = transformation[corpus] See the :mod:`gensim.models.tfidfmodel` module for an example of a transformation. + """ def __getitem__(self, vec): @@ -251,4 +258,3 @@ def __iter__(self): # restore old normalization value self.normalize = norm #endclass SimilarityABC - diff --git a/gensim/matutils.py b/gensim/matutils.py index 963998c1e8..03947b4c18 100644 --- a/gensim/matutils.py +++ b/gensim/matutils.py @@ -33,13 +33,34 @@ except ImportError: # numpy < 1.4 def triu_indices(n, k=0): - m = numpy.ones((n,n), int) + m = numpy.ones((n, n), int) a = triu(m, k) return numpy.where(a != 0) blas = lambda name, ndarray: scipy.linalg.get_blas_funcs((name,), (ndarray,))[0] +try: + # with bottleneck installed, we can use faster partial sorting + import bottleneck + def argsort(x, topn=None): + """Return indices of the `topn` greatest elements in numpy array `x`, in order.""" + if topn is None: + topn = x.size + if topn <= 0: + return [] + if topn >= x.size: + return numpy.argsort(x)[::-1] + biggest = bottleneck.argpartsort(x, x.size - topn)[-topn:] + # the indices in `biggest` are not sorted by magnitude => sort & return + return biggest.take(numpy.argsort(x.take(biggest))[::-1]) +except ImportError: + # no bottleneck => fall back to numpy + def argsort(x, topn=None): + if topn is None: + topn = x.size + return numpy.argsort(x)[::-1][:topn] + logger = logging.getLogger("gensim.matutils") @@ -74,8 +95,8 @@ def corpus2csc(corpus, num_terms=None, dtype=numpy.float64, num_docs=None, num_n if printprogress and docno % printprogress == 0: logger.info("PROGRESS: at document #%i/%i" % (docno, num_docs)) posnext = posnow + len(doc) - indices[posnow : posnext] = [feature_id for feature_id, _ in doc] - data[posnow : posnext] = [feature_weight for _, feature_weight in doc] + indices[posnow: posnext] = [feature_id for feature_id, _ in doc] + data[posnow: posnext] = [feature_weight for _, feature_weight in doc] indptr.append(posnext) posnow = posnext assert posnow == num_nnz, "mismatch between supplied and computed number of non-zeros" @@ -118,11 +139,20 @@ def ismatrix(m): return isinstance(m, numpy.ndarray) and m.ndim == 2 or scipy.sparse.issparse(m) -def scipy2sparse(vec): +def any2sparse(vec, eps=1e-9): + """Convert a numpy/scipy vector into gensim format (list of 2-tuples).""" + if isinstance(vec, numpy.ndarray): + return dense2vec(vec, eps) + if scipy.sparse.issparse(vec): + return scipy2sparse(vec, eps) + return [(int(fid), float(fw)) for fid, fw in vec if numpy.abs(fw) > eps] + + +def scipy2sparse(vec, eps=1e-9): """Convert a scipy.sparse vector to gensim format (list of 2-tuples).""" vec = vec.tocsr() assert vec.shape[0] == 1 - return zip(vec.indices, vec.data) + return [(int(pos), float(val)) for pos, val in zip(vec.indices, vec.data) if numpy.abs(val) > eps] class Scipy2Corpus(object): @@ -158,10 +188,9 @@ def full2sparse(vec, eps=1e-9): Values of magnitude < `eps` are treated as zero (ignored). """ - return [(pos, val) for pos, val in enumerate(vec) if numpy.abs(val) > eps] -# # slightly faster but less flexible: -# nnz = vec.nonzero()[0] -# return zip(nnz, vec[nnz]) + vec = numpy.asarray(vec, dtype=float) + nnz = numpy.nonzero(abs(vec) > eps)[0] + return zip(nnz, vec.take(nnz)) dense2vec = full2sparse @@ -174,13 +203,10 @@ def full2sparse_clipped(vec, topn, eps=1e-9): # this is about 40x faster than explicitly forming all 2-tuples to run sort() or heapq.nlargest() on. if topn <= 0: return [] - result = [] - for i in numpy.argsort(vec)[::-1]: - if abs(vec[i]) > eps: # ignore features with near-zero weight - result.append((i, vec[i])) - if len(result) == topn: - break - return result + vec = numpy.asarray(vec, dtype=float) + nnz = numpy.nonzero(abs(vec) > eps)[0] + biggest = nnz.take(argsort(vec.take(nnz), topn)) + return zip(biggest, vec.take(biggest)) def corpus2dense(corpus, num_terms): @@ -243,6 +269,7 @@ def veclen(vec): blas_nrm2 = blas('nrm2', numpy.array([], dtype=float)) blas_scal = blas('scal', numpy.array([], dtype=float)) + def unitvec(vec): """ Scale a vector to unit length. The only exception is the zero vector, which @@ -253,7 +280,7 @@ def unitvec(vec): """ if scipy.sparse.issparse(vec): # convert scipy.sparse to standard numpy array vec = vec.tocsr() - veclen = numpy.sqrt(numpy.sum(vec.data**2)) + veclen = numpy.sqrt(numpy.sum(vec.data ** 2)) if veclen > 0.0: return vec / veclen else: @@ -272,8 +299,8 @@ def unitvec(vec): except: return vec - if isinstance(first, tuple): # gensim sparse format? - length = 1.0 * math.sqrt(sum(val**2 for _, val in vec)) + if isinstance(first, (tuple, list)) and len(first) == 2: # gensim sparse format? + length = 1.0 * math.sqrt(sum(val ** 2 for _, val in vec)) assert length > 0.0, "sparse documents must not contain any explicit zero entries" if length != 1.0: return [(termid, val / length) for termid, val in vec] @@ -383,8 +410,7 @@ def write_vector(self, docno, vector): """ assert self.headers_written, "must write Matrix Market file headers before writing data!" assert self.last_docno < docno, "documents %i and %i not in sequential order!" % (self.last_docno, docno) - vector = [(i, w) for i, w in vector if abs(w) > 1e-12] # ignore near-zero entries - vector.sort() + vector = sorted((i, w) for i, w in vector if abs(w) > 1e-12) # ignore near-zero entries for termid, weight in vector: # write term ids in sorted order self.fout.write("%i %i %s\n" % (docno + 1, termid + 1, weight)) # +1 because MM format starts counting from 1 self.last_docno = docno @@ -392,7 +418,7 @@ def write_vector(self, docno, vector): @staticmethod - def write_corpus(fname, corpus, progress_cnt=1000, index=False): + def write_corpus(fname, corpus, progress_cnt=1000, index=False, num_terms=None): """ Save the vector space representation of an entire corpus to disk. @@ -405,7 +431,7 @@ def write_corpus(fname, corpus, progress_cnt=1000, index=False): mw.write_headers(-1, -1, -1) # will print 50 spaces followed by newline on the stats line # calculate necessary header info (nnz elements, num terms, num docs) while writing out vectors - num_terms, num_nnz = 0, 0 + _num_terms, num_nnz = 0, 0 docno, poslast = -1, -1 offsets = [] for docno, bow in enumerate(corpus): @@ -418,16 +444,17 @@ def write_corpus(fname, corpus, progress_cnt=1000, index=False): offsets.append(posnow) poslast = posnow max_id, veclen = mw.write_vector(docno, bow) - num_terms = max(num_terms, 1 + max_id) + _num_terms = max(_num_terms, 1 + max_id) num_nnz += veclen num_docs = docno + 1 + num_terms = num_terms or _num_terms if num_docs * num_terms != 0: - logger.info("saved %ix%i matrix, density=%.3f%% (%i/%i)" % - (num_docs, num_terms, - 100.0 * num_nnz / (num_docs * num_terms), - num_nnz, - num_docs * num_terms)) + logger.info("saved %ix%i matrix, density=%.3f%% (%i/%i)" % ( + num_docs, num_terms, + 100.0 * num_nnz / (num_docs * num_terms), + num_nnz, + num_docs * num_terms)) # now write proper headers, by seeking and overwriting the spaces written earlier mw.fake_headers(num_docs, num_terms, num_nnz) diff --git a/gensim/models/__init__.py b/gensim/models/__init__.py index 09a5a34291..0bbf507588 100644 --- a/gensim/models/__init__.py +++ b/gensim/models/__init__.py @@ -10,6 +10,7 @@ from tfidfmodel import TfidfModel from rpmodel import RpModel from logentropy_model import LogEntropyModel +from word2vec import Word2Vec from gensim import interfaces, utils @@ -46,5 +47,5 @@ def __getitem__(self, bow): if is_corpus: return self._apply(bow) - return [(self.old2new[oldid], weight) for oldid, weight in bow if oldid in self.old2new] + return sorted((self.old2new[oldid], weight) for oldid, weight in bow if oldid in self.old2new) #endclass VocabTransform diff --git a/gensim/models/hdpmodel.py b/gensim/models/hdpmodel.py index caabc98377..7a7ad33991 100644 --- a/gensim/models/hdpmodel.py +++ b/gensim/models/hdpmodel.py @@ -36,7 +36,7 @@ from __future__ import with_statement -import logging, os, itertools, time +import logging, itertools, time import numpy as np import scipy.special as sp @@ -47,24 +47,26 @@ meanchangethresh = 0.00001 rhot_bound = 0.0 + def log_normalize(v): log_max = 100.0 if len(v.shape) == 1: max_val = np.max(v) - log_shift = log_max - np.log(len(v)+1.0) - max_val + log_shift = log_max - np.log(len(v) + 1.0) - max_val tot = np.sum(np.exp(v + log_shift)) log_norm = np.log(tot) - log_shift v = v - log_norm else: max_val = np.max(v, 1) - log_shift = log_max - np.log(v.shape[1]+1.0) - max_val - tot = np.sum(np.exp(v + log_shift[:,np.newaxis]), 1) + log_shift = log_max - np.log(v.shape[1] + 1.0) - max_val + tot = np.sum(np.exp(v + log_shift[:, np.newaxis]), 1) log_norm = np.log(tot) - log_shift - v = v - log_norm[:,np.newaxis] + v = v - log_norm[:, np.newaxis] return (v, log_norm) + def dirichlet_expectation(alpha): """ For a vector theta ~ Dir(alpha), compute E[log(theta)] given alpha. @@ -73,6 +75,7 @@ def dirichlet_expectation(alpha): return(sp.psi(alpha) - sp.psi(np.sum(alpha))) return(sp.psi(alpha) - sp.psi(np.sum(alpha, 1))[:, np.newaxis]) + def expect_log_sticks(sticks): """ For stick-breaking hdp, return the E[log(sticks)] @@ -83,10 +86,11 @@ def expect_log_sticks(sticks): n = len(sticks[0]) + 1 Elogsticks = np.zeros(n) - Elogsticks[0:n-1] = ElogW + Elogsticks[0: n - 1] = ElogW Elogsticks[1:] = Elogsticks[1:] + np.cumsum(Elog1_W) return Elogsticks + def lda_e_step(doc_word_ids, doc_word_counts, alpha, beta, max_iter=100): gamma = np.ones(len(alpha)) expElogtheta = np.exp(dirichlet_expectation(gamma)) @@ -96,16 +100,16 @@ def lda_e_step(doc_word_ids, doc_word_counts, alpha, beta, max_iter=100): for _ in xrange(max_iter): lastgamma = gamma - gamma = alpha + expElogtheta * np.dot(counts/phinorm, betad.T) + gamma = alpha + expElogtheta * np.dot(counts / phinorm, betad.T) Elogtheta = dirichlet_expectation(gamma) expElogtheta = np.exp(Elogtheta) phinorm = np.dot(expElogtheta, betad) + 1e-100 - meanchange = np.mean(abs(gamma-lastgamma)) + meanchange = np.mean(abs(gamma - lastgamma)) if (meanchange < meanchangethresh): break likelihood = np.sum(counts * np.log(phinorm)) - likelihood += np.sum((alpha-gamma) * Elogtheta) + likelihood += np.sum((alpha - gamma) * Elogtheta) likelihood += np.sum(sp.gammaln(gamma) - sp.gammaln(alpha)) likelihood += sp.gammaln(np.sum(alpha)) - sp.gammaln(np.sum(gamma)) @@ -169,12 +173,12 @@ def __init__(self, corpus, id2word, max_chunks=None, max_time=None, self.m_alpha = alpha self.m_gamma = gamma - self.m_var_sticks = np.zeros((2, T-1)) + self.m_var_sticks = np.zeros((2, T - 1)) self.m_var_sticks[0] = 1.0 - self.m_var_sticks[1] = range(T-1, 0, -1) + self.m_var_sticks[1] = range(T - 1, 0, -1) self.m_varphi_ss = np.zeros(T) - self.m_lambda = np.random.gamma(1.0, 1.0, (T, self.m_W)) * self.m_D*100/(T*self.m_W)-eta + self.m_lambda = np.random.gamma(1.0, 1.0, (T, self.m_W)) * self.m_D * 100 / (T * self.m_W) - eta self.m_eta = eta self.m_Elogbeta = dirichlet_expectation(self.m_eta + self.m_lambda) @@ -254,7 +258,7 @@ def update_chunk(self, chunk, update=True, opt_o=True): self.m_lambda[:, word_list] *= np.exp(self.m_r[-1] - rw) self.m_Elogbeta[:, word_list] = \ sp.psi(self.m_eta + self.m_lambda[:, word_list]) - \ - sp.psi(self.m_W*self.m_eta + self.m_lambda_sum[:, np.newaxis]) + sp.psi(self.m_W * self.m_eta + self.m_lambda_sum[:, np.newaxis]) ss = SuffStats(self.m_T, Wt, len(chunk)) @@ -287,12 +291,12 @@ def doc_e_step(self, doc, ss, Elogsticks_1st, word_list, Elogbeta_doc = self.m_Elogbeta[:, doc_word_ids] ## very similar to the hdp equations - v = np.zeros((2, self.m_K-1)) + v = np.zeros((2, self.m_K - 1)) v[0] = 1.0 v[1] = self.m_alpha # back to the uniform - phi = np.ones((len(doc_word_ids), self.m_K)) * 1.0/self.m_K + phi = np.ones((len(doc_word_ids), self.m_K)) * 1.0 / self.m_K likelihood = 0.0 old_likelihood = -1e200 @@ -326,9 +330,9 @@ def doc_e_step(self, doc, ss, Elogsticks_1st, word_list, phi = np.exp(log_phi) # v - phi_all = phi * np.array(doc_word_counts)[:,np.newaxis] - v[0] = 1.0 + np.sum(phi_all[:,:self.m_K-1], 0) - phi_cum = np.flipud(np.sum(phi_all[:,1:], 0)) + phi_all = phi * np.array(doc_word_counts)[:, np.newaxis] + v[0] = 1.0 + np.sum(phi_all[:, :self.m_K - 1], 0) + phi_cum = np.flipud(np.sum(phi_all[:, 1:], 0)) v[1] = self.m_alpha + np.flipud(np.cumsum(phi_cum)) Elogsticks_2nd = expect_log_sticks(v) @@ -339,9 +343,9 @@ def doc_e_step(self, doc, ss, Elogsticks_1st, word_list, # v part/ v in john's notation, john's beta is alpha here log_alpha = np.log(self.m_alpha) - likelihood += (self.m_K-1) * log_alpha + likelihood += (self.m_K - 1) * log_alpha dig_sum = sp.psi(np.sum(v, 0)) - likelihood += np.sum((np.array([1.0, self.m_alpha])[:,np.newaxis]-v) * (sp.psi(v)-dig_sum)) + likelihood += np.sum((np.array([1.0, self.m_alpha])[:, np.newaxis] - v) * (sp.psi(v) - dig_sum)) likelihood -= np.sum(sp.gammaln(np.sum(v, 0))) - np.sum(sp.gammaln(v)) # Z part @@ -350,7 +354,7 @@ def doc_e_step(self, doc, ss, Elogsticks_1st, word_list, # X part, the data part likelihood += np.sum(phi.T * np.dot(var_phi, Elogbeta_doc * doc_word_counts)) - converge = (likelihood - old_likelihood)/abs(old_likelihood) + converge = (likelihood - old_likelihood) / abs(old_likelihood) old_likelihood = likelihood if converge < -0.000001: @@ -376,23 +380,23 @@ def update_lambda(self, sstats, word_list, opt_o): self.m_rhot = rhot # Update appropriate columns of lambda based on documents. - self.m_lambda[:, word_list] = self.m_lambda[:, word_list] * (1-rhot) + \ + self.m_lambda[:, word_list] = self.m_lambda[:, word_list] * (1 - rhot) + \ rhot * self.m_D * sstats.m_var_beta_ss / sstats.m_chunksize - self.m_lambda_sum = (1-rhot) * self.m_lambda_sum+ \ + self.m_lambda_sum = (1 - rhot) * self.m_lambda_sum + \ rhot * self.m_D * np.sum(sstats.m_var_beta_ss, axis=1) / sstats.m_chunksize self.m_updatect += 1 self.m_timestamp[word_list] = self.m_updatect - self.m_r.append(self.m_r[-1] + np.log(1-rhot)) + self.m_r.append(self.m_r[-1] + np.log(1 - rhot)) - self.m_varphi_ss = (1.0-rhot) * self.m_varphi_ss + rhot * \ + self.m_varphi_ss = (1.0 - rhot) * self.m_varphi_ss + rhot * \ sstats.m_var_sticks_ss * self.m_D / sstats.m_chunksize if opt_o: - self.optimal_ordering(); + self.optimal_ordering() ## update top level sticks - self.m_var_sticks[0] = self.m_varphi_ss[:self.m_T-1] + 1.0 + self.m_var_sticks[0] = self.m_varphi_ss[:self.m_T - 1] + 1.0 var_phi_sum = np.flipud(self.m_varphi_ss[1:]) self.m_var_sticks[1] = np.flipud(np.cumsum(var_phi_sum)) + self.m_gamma @@ -403,9 +407,9 @@ def optimal_ordering(self): """ idx = np.argsort(self.m_lambda_sum)[::-1] self.m_varphi_ss = self.m_varphi_ss[idx] - self.m_lambda = self.m_lambda[idx,:] + self.m_lambda = self.m_lambda[idx, :] self.m_lambda_sum = self.m_lambda_sum[idx] - self.m_Elogbeta = self.m_Elogbeta[idx,:] + self.m_Elogbeta = self.m_Elogbeta[idx, :] def update_expectations(self): @@ -420,7 +424,7 @@ def update_expectations(self): self.m_lambda[:, w] *= np.exp(self.m_r[-1] - self.m_r[self.m_timestamp[w]]) self.m_Elogbeta = sp.psi(self.m_eta + self.m_lambda) - \ - sp.psi(self.m_W*self.m_eta + self.m_lambda_sum[:, np.newaxis]) + sp.psi(self.m_W * self.m_eta + self.m_lambda_sum[:, np.newaxis]) self.m_timestamp[:] = self.m_updatect self.m_status_up_to_date = True @@ -474,13 +478,13 @@ def hdp_to_lda(self): Compute the LDA almost equivalent HDP. """ # alpha - sticks = self.m_var_sticks[0]/(self.m_var_sticks[0]+self.m_var_sticks[1]) + sticks = self.m_var_sticks[0] / (self.m_var_sticks[0] + self.m_var_sticks[1]) alpha = np.zeros(self.m_T) left = 1.0 - for i in range(0, self.m_T-1): + for i in range(0, self.m_T - 1): alpha[i] = sticks[i] * left left = left - alpha[i] - alpha[self.m_T-1] = left + alpha[self.m_T - 1] = left alpha = alpha * self.m_alpha # beta diff --git a/gensim/models/ldamodel.py b/gensim/models/ldamodel.py index c293dad9e4..d5da6d449d 100644 --- a/gensim/models/ldamodel.py +++ b/gensim/models/ldamodel.py @@ -203,7 +203,7 @@ def __init__(self, corpus=None, num_topics=100, id2word=None, distributed=False, raise ValueError('at least one of corpus/id2word must be specified, to establish input space dimensionality') if self.id2word is None: - logger.info("no word id mapping provided; initializing from corpus, assuming identity") + logger.warning("no word id mapping provided; initializing from corpus, assuming identity") self.id2word = utils.dict_from_corpus(corpus) self.num_terms = len(self.id2word) else: @@ -257,7 +257,7 @@ def __init__(self, corpus=None, num_topics=100, id2word=None, distributed=False, # Initialize the variational distribution q(beta|lambda) self.state = LdaState(self.eta, (self.num_topics, self.num_terms)) - self.state.sstats = numpy.random.gamma(100., 1./100., (self.num_topics, self.num_terms)) + self.state.sstats = numpy.random.gamma(100., 1. / 100., (self.num_topics, self.num_terms)) self.sync_state() # if a training corpus was provided, start estimating the model right away @@ -265,6 +265,11 @@ def __init__(self, corpus=None, num_topics=100, id2word=None, distributed=False, self.update(corpus) + def __str__(self): + return "LdaModel(num_terms=%s, num_topics=%s, decay=%s, chunksize=%s, alpha=%s)" % \ + (self.num_terms, self.num_topics, self.decay, self.chunksize, self.alpha) + + def sync_state(self): self.expElogbeta = numpy.exp(self.state.get_Elogbeta()) @@ -293,10 +298,11 @@ def inference(self, chunk, collect_sstats=False): _ = len(chunk) except: chunk = list(chunk) # convert iterators/generators to plain list, so we have len() etc. - logger.debug("performing inference on a chunk of %i documents" % len(chunk)) + if len(chunk) > 1: + logger.debug("performing inference on a chunk of %i documents" % len(chunk)) # Initialize the variational distribution q(theta|gamma) for the chunk - gamma = numpy.random.gamma(100., 1./100., (len(chunk), self.num_topics)) + gamma = numpy.random.gamma(100., 1. / 100., (len(chunk), self.num_topics)) Elogtheta = dirichlet_expectation(gamma) expElogtheta = numpy.exp(Elogtheta) if collect_sstats: @@ -535,7 +541,7 @@ def bound(self, corpus, gamma=None): def print_topics(self, topics=10, topn=10): - self.show_topics(topics, topn, True) + return self.show_topics(topics, topn, True) def show_topics(self, topics=10, topn=10, log=False, formatted=True): """ @@ -550,7 +556,7 @@ def show_topics(self, topics=10, topn=10, log=False, formatted=True): # print all topics if `topics` is negative topics = self.num_topics topics = min(topics, self.num_topics) - shown = [] + shown = [] for i in xrange(topics): if formatted: topic = self.print_topic(i, topn=topn) @@ -588,4 +594,12 @@ def __getitem__(self, bow, eps=0.01): topic_dist = gamma[0] / sum(gamma[0]) # normalize to proper distribution return [(topicid, topicvalue) for topicid, topicvalue in enumerate(topic_dist) if topicvalue >= eps] # ignore document's topics that have prob < eps + + + def save(self, fname): + dispatcher, self.dispatcher = self.dispatcher, None + try: + super(LdaModel, self).save(fname) + finally: + self.dispatcher = dispatcher #endclass LdaModel diff --git a/gensim/models/lsimodel.py b/gensim/models/lsimodel.py index ce24de705c..762f619cca 100644 --- a/gensim/models/lsimodel.py +++ b/gensim/models/lsimodel.py @@ -99,7 +99,7 @@ def ascarray(a, name=''): class Projection(utils.SaveLoad): - def __init__(self, m, k, docs=None, use_svdlibc=False): + def __init__(self, m, k, docs=None, use_svdlibc=False, power_iters=P2_EXTRA_ITERS, extra_dims=P2_EXTRA_DIMS): """ Construct the (U, S) projection from a corpus `docs`. The projection can be later updated by merging it with another Projection via `self.merge()`. @@ -109,12 +109,14 @@ def __init__(self, m, k, docs=None, use_svdlibc=False): the higher-level `LsiModel` class. """ self.m, self.k = m, k + self.power_iters = power_iters + self.extra_dims = extra_dims if docs is not None: # base case decomposition: given a job `docs`, compute its decomposition, # *in-core*. if not use_svdlibc: u, s = stochastic_svd(docs, k, chunksize=sys.maxint, num_terms=m, - power_iters=P2_EXTRA_ITERS, extra_dims=P2_EXTRA_DIMS) + power_iters=self.power_iters, extra_dims=self.extra_dims) else: try: import sparsesvd @@ -134,7 +136,7 @@ def __init__(self, m, k, docs=None, use_svdlibc=False): def empty_like(self): - return Projection(self.m, self.k) + return Projection(self.m, self.k, power_iters=self.power_iters, extra_dims=self.extra_dims) def merge(self, other, decay=1.0): @@ -282,14 +284,14 @@ def __init__(self, corpus=None, num_topics=200, id2word=None, chunksize=20000, raise ValueError('at least one of corpus/id2word must be specified, to establish input space dimensionality') if self.id2word is None: - logger.info("no word id mapping provided; initializing from corpus, assuming identity") + logger.warning("no word id mapping provided; initializing from corpus, assuming identity") self.id2word = utils.dict_from_corpus(corpus) self.num_terms = len(self.id2word) else: self.num_terms = 1 + max([-1] + self.id2word.keys()) self.docs_processed = 0 - self.projection = Projection(self.num_terms, self.num_topics) + self.projection = Projection(self.num_terms, self.num_topics, power_iters=self.power_iters, extra_dims=self.extra_samples) self.numworkers = 1 if not distributed: @@ -306,6 +308,7 @@ def __init__(self, corpus=None, num_topics=200, id2word=None, chunksize=20000, logger.debug("looking for dispatcher at %s" % str(dispatcher._pyroUri)) dispatcher.initialize(id2word=self.id2word, num_topics=num_topics, chunksize=chunksize, decay=decay, + power_iters=self.power_iters, extra_samples=self.extra_samples, distributed=False, onepass=onepass) self.dispatcher = dispatcher self.numworkers = len(dispatcher.getworkers()) @@ -370,7 +373,7 @@ def add_documents(self, corpus, chunksize=None, decay=None): logger.info("dispatched documents up to #%s" % doc_no) else: # serial version, there is only one "worker" (myself) => process the job directly - update = Projection(self.num_terms, self.num_topics, job) + update = Projection(self.num_terms, self.num_topics, job, extra_dims=self.extra_samples, power_iters=self.power_iters) del job self.projection.merge(update, decay=decay) del update @@ -386,7 +389,7 @@ def add_documents(self, corpus, chunksize=None, decay=None): else: assert not self.dispatcher, "must be in serial mode to receive jobs" assert self.onepass, "distributed two-pass algo not supported yet" - update = Projection(self.num_terms, self.num_topics, corpus.tocsc()) + update = Projection(self.num_terms, self.num_topics, corpus.tocsc(), extra_dims=self.extra_samples, power_iters=self.power_iters) self.projection.merge(update, decay=decay) logger.info("processed sparse job of %i documents" % (corpus.shape[1])) @@ -515,13 +518,14 @@ def save(self, fname): utils.pickle(self, fname) # first, remove the projection from self.__dict__, so it doesn't get pickled - u = self.projection.u + u, dispatcher = self.projection.u, self.dispatcher del self.projection.u + self.dispatcher = None try: utils.pickle(self, fname) # store projection-less object numpy.save(fname + '.npy', ascarray(u)) # store projection finally: - self.projection.u = u + self.projection.u, self.dispatcher = u, dispatcher @classmethod @@ -536,6 +540,7 @@ def load(cls, fname): result.projection.u = numpy.load(ufname, mmap_mode='r') # load back as read-only except: logger.debug("failed to load mmap'ed projection from %s" % ufname) + result.dispatcher = None # TODO load back incl. distributed state? will require re-initialization of worker state return result #endclass LsiModel diff --git a/gensim/models/rpmodel.py b/gensim/models/rpmodel.py index 2a103197f0..ef8f52b6e2 100644 --- a/gensim/models/rpmodel.py +++ b/gensim/models/rpmodel.py @@ -83,7 +83,7 @@ def __getitem__(self, bow): vec = matutils.sparse2full(bow, self.num_terms).reshape(self.num_terms, 1) / numpy.sqrt(self.num_topics) vec = numpy.asfortranarray(vec, dtype=numpy.float32) - topic_dist = scipy.linalg.fblas.sgemv(1.0, self.projection, vec) # (k, d) * (d, 1) = (k, 1) + topic_dist = numpy.dot(self.projection, vec) # (k, d) * (d, 1) = (k, 1) return [(topicid, float(topicvalue)) for topicid, topicvalue in enumerate(topic_dist.flat) if numpy.isfinite(topicvalue) and not numpy.allclose(topicvalue, 0.0)] diff --git a/gensim/models/word2vec.py b/gensim/models/word2vec.py new file mode 100755 index 0000000000..d011cf78d9 --- /dev/null +++ b/gensim/models/word2vec.py @@ -0,0 +1,459 @@ +#!/usr/bin/env python +# -*- coding: utf-8 -*- +# +# Copyright (C) 2013 Radim Rehurek +# Licensed under the GNU LGPL v2.1 - http://www.gnu.org/licenses/lgpl.html + + +""" +Module for deep learning via *hierarchical softmax skip-gram* from [1]_. +The algorithm is ported from the C package https://code.google.com/p/word2vec/ . + +Initialize a model with e.g.:: + +>>> model = Word2Vec(sentences, size=100, window=5, min_count=5) + +Store/load a model with:: + +>>> model.save(fname) +>>> model = Word2Vec.load(fname) + +The model can also be instantiated from an existing, trained file on disk in word2vec format:: + +>>> model = Word2Vec.load_word2vec_format('/tmp/vectors.txt', binary=False) # text format +>>> model = Word2Vec.load_word2vec_format('/tmp/vectors.bin', binary=True) # binary format + +.. [1] Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient Estimation of Word Representations in Vector Space. In Proceedings of Workshop at ICLR, 2013. + +""" + +import logging +import sys +import os +import heapq +import time + +from numpy import zeros_like, empty, exp, dot, outer, random, dtype,\ + float32 as REAL, seterr, array, uint8, vstack, argsort, fromstring + +from gensim import utils, matutils # utility fnc for pickling, common scipy operations etc + +logger = logging.getLogger(__name__) + +MIN_ALPHA = 0.0001 # don't allow learning rate to drop below this threshold + + +class Vocab(object): + """A single vocabulary item, used internally for constructing binary trees (incl. both word leaves and inner nodes).""" + def __init__(self, **kwargs): + self.count = 0 + self.__dict__.update(kwargs) + + def __lt__(self, other): # used for sorting in a priority queue + return self.count < other.count + + def __str__(self): + vals = ['%s:%r' % (key, self.__dict__[key]) for key in sorted(self.__dict__) if not key.startswith('_')] + return "<" + ', '.join(vals) + ">" + + +class Word2Vec(utils.SaveLoad): + """ + Class for training, using and evaluating neural networks described in https://code.google.com/p/word2vec/ + + The model can be stored/loaded via its `save()` and `load()` methods, or stored in a format + compatible with the original word2vec implementation via `save_word2vec_format()`. + + """ + def __init__(self, sentences=None, size=100, alpha=0.025, window=5, min_count=5, seed=1): + """ + Initialize a model from `sentences`. Each sentence is a list of words + (utf8 strings) that will be used for training. + + If you don't supply `sentences`, the model is left uninitialized -- use if + you plan to initialize it in some other way. + + `size` is the dimensionality of the feature vectors. + `window` is the maximum distance between the current and predicted word within a sentence. + `alpha` is the initial learning rate (will linearly drop to zero as training progresses). + `seed` = for the random number generator. + `min_count` = ignore all words with total frequency lower than this. + + """ + self.vocab = {} # mapping from a word (string) to a Vocab object + self.index2word = [] # map from a word's matrix index (int) to word (string) + self.layer1_size = int(size) + self.alpha = float(alpha) + self.window = int(window) + self.seed = seed + self.min_count = min_count + if sentences is not None: + self.build_vocab(sentences) + self.reset_weights() + self.train_model(sentences) + + + def create_binary_tree(self): + """ + Create a binary Huffman tree using stored vocabulary word counts. Frequent words + will have shorter binary codes. Called internally from `build_vocab()`. + + """ + logger.info("constructing a huffman tree from %i words" % len(self.vocab)) + + # build the huffman tree + heap = self.vocab.values() + heapq.heapify(heap) + for i in xrange(len(self.vocab) - 1): + min1, min2 = heapq.heappop(heap), heapq.heappop(heap) + heapq.heappush(heap, Vocab(count=min1.count + min2.count, index=i + len(self.vocab), left=min1, right=min2)) + + # recurse over the tree, assigning a binary code to each vocabulary word + if heap: + max_depth, stack = 0, [(heap[0], [], [])] + while stack: + node, codes, points = stack.pop() + if node.index < len(self.vocab): + # leaf node => store its path from the root + node.code, node.point = codes, points + max_depth = max(len(codes), max_depth) + else: + # inner node => continue recursion + points = array(list(points) + [node.index - len(self.vocab)], dtype=int) + stack.append((node.left, array(list(codes) + [0], dtype=uint8), points)) + stack.append((node.right, array(list(codes) + [1], dtype=uint8), points)) + + logger.info("built huffman tree with maximum node depth %i" % max_depth) + + + def build_vocab(self, sentences): + """Build vocabulary from a sequence of sentences.""" + logger.info("collecting all words and their counts") + sentence_no, vocab = -1, {} + total_words = lambda: sum(v.count for v in vocab.itervalues()) + for sentence_no, sentence in enumerate(sentences): + if sentence_no % 10000 == 0: + logger.info("PROGRESS: at sentence #%i, processed %i words and %i word types" % + (sentence_no, total_words(), len(vocab))) + for word in sentence: + v = vocab.setdefault(word, Vocab()) + v.count += 1 + logger.info("collected %i word types from a corpus of %i words and %i sentences" % + (len(vocab), total_words(), sentence_no + 1)) + + # assign a unique index to each word + self.vocab, self.index2word = {}, [] + for word, v in vocab.iteritems(): + if v.count >= self.min_count: + v.index = len(self.vocab) + self.index2word.append(word) + self.vocab[word] = v + logger.info("total %i word types after removing those with count<%s" % (len(self.vocab), self.min_count)) + + # add info about each word's Huffman encoding + self.create_binary_tree() + + + def train_sentence(self, words, alpha): + """ + Update skip-gram hierarchical softmax model by training on a single sentence, + where `sentence` is a list of Vocab objects (or None, where the corresponding + word is not in the vocabulary). Called internally from `train_model())`. + + """ + for pos, word in enumerate(words): + if word is None: + continue # OOV word in the input sentence => skip + reduced_window = random.randint(self.window) # `b` in the original word2vec code + + # now go over all words from the (reduced) window, predicting each one in turn + start = max(0, pos - self.window + reduced_window) + for pos2, word2 in enumerate(words[start : pos + self.window + 1 - reduced_window], start): + if pos2 == pos or word2 is None: + # don't train on OOV words and on the `word` itself + continue + l1 = self.syn0[word2.index] + # work on the entire tree at once, to push as much work into numpy's C routines as possible (performance) + l2a = self.syn1[word.point] # 2d matrix, codelen x layer1_size + fa = 1.0 / (1.0 + exp(-dot(l1, l2a.T))) # propagate hidden -> output + ga = (1 - word.code - fa) * alpha # vector of error gradients multiplied by the learning rate + self.syn1[word.point] += outer(ga, l1) # learn hidden -> output + + # TODO add negative sampling? + + l1 += dot(ga, l2a) # learn input -> hidden + + + def train_model(self, sentences, total_words=None): + """ + Train the model on a sequence of sentences, updating its existing neural weights. + Each sentence is a list of utf8 strings. + + """ + logger.info("training model with %i words and %i features" % (len(self.vocab), self.layer1_size)) + + # iterate over documents, training the model one sentence at a time + total_words = total_words or sum(v.count for v in self.vocab.itervalues()) + alpha = self.alpha + word_count, sentence_no, start = 0, -1, time.clock() + for sentence_no, sentence in enumerate(sentences): + if sentence_no % 100 == 0: + # decrease learning rate as the training progresses + alpha = max(MIN_ALPHA, self.alpha * (1 - 1.0 * word_count / total_words)) + + # print progress and training stats + elapsed = time.clock() - start + logger.info("PROGRESS: at sentence #%i, %.2f%% words, alpha %f, %.0f words per second" % + (sentence_no, 100.0 * word_count / total_words, alpha, word_count / elapsed if elapsed else 0.0)) + words = [self.vocab.get(word, None) for word in sentence] # replace OOV words with None + self.train_sentence(words, alpha=alpha) + word_count += len(filter(None, words)) # don't consider OOV words for the statistics + logger.info("training took %.1fs" % (time.clock() - start)) + + + def reset_weights(self): + """Reset all projection weights, but keep the existing vocabulary.""" + random.seed(self.seed) + self.syn0 = ((random.rand(len(self.vocab), self.layer1_size) - 0.5) / self.layer1_size).astype(dtype=REAL) + self.syn1 = zeros_like(self.syn0) + + + def save_word2vec_format(self, fname, binary=False): + """ + Store the input-hidden weight matrix in the same format used by the original + word2vec-tool, for compatibility. + + """ + logger.info("storing %sx%s projection weights into %s" % (len(self.vocab), self.layer1_size, fname)) + assert (len(self.vocab), self.layer1_size) == self.syn0.shape + with open(fname, 'wb') as fout: + fout.write("%s %s\n" % self.syn0.shape) + # store in sorted order: most frequent words at the top + for word, vocab in sorted(self.vocab.iteritems(), key=lambda item: -item[1].count): + row = self.syn0[vocab.index] + if binary: + fout.write("%s %s\n" % (word, row.tostring())) + else: + fout.write("%s %s\n" % (word, ' '.join("%f" % val for val in row))) + + + @classmethod + def load_word2vec_format(cls, fname, binary=False): + """ + Load the input-hidden weight matrix from the original word2vec-tool format. + + Note that the information loaded is incomplete (the binary tree is missing), + so while you can query for word similarity etc., you cannot continue training + with a model loaded this way. + + """ + logger.info("loading projection weights from %s" % (fname)) + with open(fname) as fin: + header = fin.readline() + vocab_size, layer1_size = map(int, header.split()) # throws for invalid file format + result = Word2Vec(size=layer1_size) + result.syn0 = empty((vocab_size, layer1_size), dtype=REAL) + if binary: + binary_len = dtype(REAL).itemsize * layer1_size + for line_no in xrange(vocab_size): + # mixed text and binary: read text first, then binary + word = [] + while True: + ch = fin.read(1) + if ch == ' ': + word = ''.join(word) + break + word.append(ch) + result.vocab[word] = Vocab(index=line_no, count=vocab_size - line_no) + result.index2word.append(word) + result.syn0[line_no] = fromstring(fin.read(binary_len), dtype=REAL) + fin.read(1) # newline + else: + for line_no, line in enumerate(fin): + parts = line.split() + assert len(parts) == layer1_size + 1 + word, weights = parts[0], map(REAL, parts[1:]) + result.vocab[word] = Vocab(index=line_no, count=vocab_size - line_no) + result.index2word.append(word) + result.syn0[line_no] = weights + logger.info("loaded %s matrix from %s" % (result.syn0.shape, fname)) + result.init_sims() + return result + + + def most_similar(self, positive=[], negative=[], topn=10): + """ + Find the top-N most similar words. Positive words contribute positively towards the + similarity, negative words negatively. + + This method computes cosine similarity between a simple mean of the projection + weight vectors of the given words, and corresponds to the `word-analogy` + script in the original word2vec implementation. + + Example:: + + >>> trained_model.most_similar(positive=['woman', 'king'], negative=['man']) + [('queen', 0.50882536), ...] + + """ + self.init_sims() + + # add weights for each word, if not already present; default to 1.0 for positive and -1.0 for negative words + positive = [(word, 1.0) if isinstance(word, basestring) else word for word in positive] + negative = [(word, -1.0) if isinstance(word, basestring) else word for word in negative] + all_words, mean = set(), [] + for word, weight in positive + negative: + if word in self.vocab: + mean.append(weight * matutils.unitvec(self.syn0[self.vocab[word].index])) + all_words.add(self.vocab[word].index) + else: + logger.warning("word '%s' not in vocabulary; ignoring it" % word) + if not mean: + raise ValueError("cannot compute similarity with no input") + mean = matutils.unitvec(array(mean).mean(axis=0)).astype(REAL) + dists = dot(self.syn0norm, mean) + if not topn: + return dists + best = argsort(dists)[::-1][:topn + len(all_words)] + # ignore (don't return) words from the input + result = [(self.index2word[sim], dists[sim]) for sim in best if sim not in all_words] + return result[:topn] + + + def init_sims(self): + if getattr(self, 'syn0norm', None) is None: + logger.info("precomputing L2-norms of word weight vectors") + self.syn0norm = vstack(matutils.unitvec(vec) for vec in self.syn0).astype(REAL) + + + def accuracy(self, questions, restrict_vocab=30000): + """ + Compute accuracy of the model. `questions` is a filename where lines are + 4-tuples of words, split into sections by ": SECTION NAME" lines. + See https://code.google.com/p/word2vec/source/browse/trunk/questions-words.txt for an example. + + The accuracy is reported (=printed to log and returned as list) for each + section separately, plus there's one aggregate summary at the end. + + Use `restrict_vocab` to ignore all questions containing a word whose frequency + is not in the top-N most frequent words (default top 30000). + + This method corresponds to the `compute-accuracy` script of the original C word2vec. + + """ + ok_vocab = dict(sorted(self.vocab.iteritems(), key=lambda item: -item[1].count)[:restrict_vocab]) + ok_index = set(v.index for v in ok_vocab.itervalues()) + + def log_accuracy(section): + correct, incorrect = section['correct'], section['incorrect'] + if correct + incorrect > 0: + logger.info("%s: %.1f%% (%i/%i)" % + (section['section'], 100.0 * correct / (correct + incorrect), + correct, correct + incorrect)) + + sections, section = [], None + for line_no, line in enumerate(open(questions)): + if line.startswith(': '): + # a new section starts => store the old section + if section: + sections.append(section) + log_accuracy(section) + section = {'section': line.lstrip(': ').strip(), 'correct': 0, 'incorrect': 0} + else: + if not section: + raise ValueError("missing section header before line #%i in %s" % (line_no, questions)) + try: + a, b, c, expected = [word.lower() for word in line.split()] # TODO assumes vocabulary uses lowercase, too... + except: + logger.info("skipping invalid line #%i in %s" % (line_no, questions)) + if a not in ok_vocab or b not in ok_vocab or c not in ok_vocab or expected not in ok_vocab: + logger.debug("skipping line #%i with OOV words: %s" % (line_no, line)) + continue + + predicted, ignore = None, set(self.vocab[v].index for v in [a, b, c]) + # go over predicted words, starting from the most likely, but ignoring OOV words and input words + for index in argsort(self.most_similar(positive=[b, c], negative=[a], topn=False))[::-1]: + if index in ok_index and index not in ignore: + predicted = self.index2word[index] + if predicted != expected: + logger.debug("%s: expected %s, predicted %s" % (line.strip(), expected, predicted)) + break + section['correct' if predicted == expected else 'incorrect'] += 1 + if section: + # store the last section, too + sections.append(section) + log_accuracy(section) + + total = {'section': 'total', 'correct': sum(s['correct'] for s in sections), 'incorrect': sum(s['incorrect'] for s in sections)} + log_accuracy(total) + sections.append(total) + return sections + + + +class BrownCorpus(object): + """Yield sentences from the Brown corpus (part of NLTK data).""" + def __init__(self, dirname): + self.dirname = dirname + + def __iter__(self): + for fname in os.listdir(self.dirname): + fname = os.path.join(self.dirname, fname) + if not os.path.isfile(fname): + continue + for line in open(fname): + # each file line is a single sentence in the Brown corpus + # each token is WORD/POS_TAG + token_tags = [t.split('/') for t in line.split() if len(t.split('/')) == 2] + # ignore words with non-alphabetic tags like ",", "!" etc (punctuation, weird stuff) + words = ["%s/%s" % (token.lower(), tag[:2]) for token, tag in token_tags if tag[:2].isalpha()] + if not words: # don't bother sending out empty sentences + continue + yield words + + +class Text8Corpus(object): + """Yield sentences from the "text8" corpus, unzipped from http://mattmahoney.net/dc/text8.zip .""" + def __init__(self, fname): + self.fname = fname + + def __iter__(self): + # the entire corpus is one gigantic line -- there are no sentence marks at all + # so just split the sequence of tokens arbitrarily: 1 sentence = 1000 tokens + sentence, rest, max_sentence_length = [], '', 1000 + with open(self.fname) as fin: + while True: + text = rest + fin.read(8192) # avoid loading the entire file (=1 line) into RAM + if text == rest: # EOF + sentence.extend(rest.split()) # return the last chunk of words, too (may be shorter/longer) + if sentence: + yield sentence + break + last_token = text.rfind(' ') # the last token may have been split in two... keep it for the next iteration + words, rest = (text[:last_token].split(), text[last_token:].strip()) if last_token >= 0 else ([], text) + sentence.extend(words) + while len(sentence) >= max_sentence_length: + yield sentence[:max_sentence_length] + sentence = sentence[max_sentence_length:] + + + +if __name__ == "__main__": + logging.basicConfig(format='%(asctime)s : %(levelname)s : %(message)s', level=logging.INFO) + logging.info("running %s" % " ".join(sys.argv)) + + # check and process cmdline input + program = os.path.basename(sys.argv[0]) + if len(sys.argv) < 3: + print globals()['__doc__'] % locals() + sys.exit(1) + infile, outfile = sys.argv[1:3] + + seterr(all='raise') # don't ignore numpy errors + + w = Word2Vec(BrownCorpus(infile), size=20, min_count=5) + w.save(outfile + '.model') + w.save_word2vec_format(outfile + '.model.bin', binary=True) + w.save_word2vec_format(outfile + '.model.txt', binary=False) + + logging.info("finished running %s" % program) diff --git a/gensim/scripts/make_wikicorpus.py b/gensim/scripts/make_wikicorpus.py index aee3442d83..5ebd3353f3 100755 --- a/gensim/scripts/make_wikicorpus.py +++ b/gensim/scripts/make_wikicorpus.py @@ -76,8 +76,8 @@ MmCorpus.serialize(outp + '_bow.mm', wiki, progress_cnt=10000) # ~4h on my macbook pro without lemmatization, 3.1m articles (august 2012) # with HashDictionary, the token->id mapping is only fully instantiated now, after `serialize` dictionary.filter_extremes(no_below=20, no_above=0.1, keep_n=DEFAULT_DICT_SIZE) - dictionary.save_as_text(outp + '_wordids.txt') - wiki.save(outp + '_corpus.pkl') + dictionary.save_as_text(outp + '_wordids.txt.bz2') + wiki.save(outp + '_corpus.pkl.bz2') dictionary.allow_update = False else: wiki = WikiCorpus(inp, lemmatize=lemmatize) # takes about 9h on a macbook pro, for 3.5m articles (june 2011) @@ -85,10 +85,10 @@ wiki.dictionary.filter_extremes(no_below=20, no_above=0.1, keep_n=DEFAULT_DICT_SIZE) # save dictionary and bag-of-words (term-document frequency matrix) MmCorpus.serialize(outp + '_bow.mm', wiki, progress_cnt=10000) # another ~9h - wiki.dictionary.save_as_text(outp + '_wordids.txt') + wiki.dictionary.save_as_text(outp + '_wordids.txt.bz2') # load back the id->word mapping directly from file # this seems to save more memory, compared to keeping the wiki.dictionary object from above - dictionary = Dictionary.load_from_text(outp + '_wordids.txt') + dictionary = Dictionary.load_from_text(outp + '_wordids.txt.bz2') del wiki # initialize corpus reader and word->id mapping diff --git a/gensim/similarities/docsim.py b/gensim/similarities/docsim.py index 8c141e4962..1b791a4e07 100755 --- a/gensim/similarities/docsim.py +++ b/gensim/similarities/docsim.py @@ -1,7 +1,7 @@ #!/usr/bin/env python # -*- coding: utf-8 -*- # -# Copyright (C) 2011 Radim Rehurek +# Copyright (C) 2013 Radim Rehurek # Licensed under the GNU LGPL v2.1 - http://www.gnu.org/licenses/lgpl.html """ @@ -73,7 +73,6 @@ pass - class Shard(utils.SaveLoad): """ A proxy class that represents a single shard instance within a Similarity @@ -84,33 +83,35 @@ class Shard(utils.SaveLoad): """ def __init__(self, fname, index): - self.fname = fname + self.dirname, self.fname = os.path.split(fname) self.length = len(index) self.cls = index.__class__ - logger.info("saving index shard to %s" % fname) - index.save(fname) + logger.info("saving index shard to %s" % self.fullname()) + index.save(self.fullname()) self.index = self.get_index() + def fullname(self): + return os.path.join(self.dirname, self.fname) def __len__(self): return self.length def __getstate__(self): result = self.__dict__.copy() - del result['index'] + # (S)MS objects must be loaded via load() because of mmap (simple pickle.load won't do) + if 'index' in result: + del result['index'] return result - def __setstate__(self, state): - self.__dict__.update(state) - self.index = self.get_index() - def __str__(self): - return ("%s Shard(%i documents in %s)" % (self.cls.__name__, len(self), self.fname)) + return ("%s Shard(%i documents in %s)" % (self.cls.__name__, len(self), self.fullname())) def get_index(self): - logger.debug("mmaping index from %s" % self.fname) - return self.cls.load(self.fname) + if not hasattr(self, 'index'): + logger.debug("mmaping index from %s" % self.fullname()) + self.index = self.cls.load(self.fullname()) + return self.index def get_document_id(self, pos): @@ -120,11 +121,11 @@ def get_document_id(self, pos): MatrixSimilarity and scipy.sparse for SparseMatrixSimilarity. """ assert 0 <= pos < len(self), "requested position out of range" - return self.index.index[pos] + return self.get_index().index[pos] def __getitem__(self, query): - index = self.index + index = self.get_index() try: index.num_best = self.num_best index.normalize = self.normalize @@ -362,6 +363,7 @@ def __getitem__(self, query): # gc doesn't seem to collect the Pools, eventually leading to # "IOError 24: too many open files". so let's terminate it manually. pool.terminate() + return result @@ -433,10 +435,19 @@ def iter_chunks(self, chunksize=None): # (unlike numpy). so, clip the end of the chunk explicitly to make # scipy.sparse happy chunk_end = min(query.shape[0], chunk_start + chunksize) - chunk = query[chunk_start : chunk_end] # create a view + chunk = query[chunk_start: chunk_end] # create a view yield chunk + def check_moved(self): + """ + Update shard locations, in case the server directory has moved on filesystem. + """ + dirname = os.path.dirname(self.output_prefix) + for shard in self.shards: + shard.dirname = dirname + + def save(self, fname=None): """ Save the object via pickling (also see load) under filename specified in @@ -475,7 +486,7 @@ class for description of the other parameters. """ if num_features is None: - logger.info("scanning corpus to determine the number of features") + logger.warning("scanning corpus to determine the number of features (consider setting `num_features` explicitly)") num_features = 1 + utils.get_max_id(corpus) self.num_features = num_features @@ -484,6 +495,8 @@ class for description of the other parameters. self.chunksize = chunksize if corpus is not None: + if self.num_features <= 0: + raise ValueError("cannot index a corpus with zero features (you must specify either `num_features` or a non-empty corpus in the constructor)") logger.info("creating matrix for %s documents and %i features" % (len(corpus), num_features)) self.index = numpy.empty(shape=(len(corpus), num_features), dtype=dtype) diff --git a/gensim/test/test_corpora_dictionary.py b/gensim/test/test_corpora_dictionary.py index d3c997eabe..11f0bf9408 100644 --- a/gensim/test/test_corpora_dictionary.py +++ b/gensim/test/test_corpora_dictionary.py @@ -102,6 +102,18 @@ def testFilter(self): expected = {0: 3, 1: 3, 2: 3, 3: 3} self.assertEqual(d.dfs, expected) + def test_doc2bow(self): + d = Dictionary([["žluťoučký"], ["žluťoučký"]]) + + # pass a utf8 string + self.assertEqual(d.doc2bow(["žluťoučký"]), [(0, 1)]) + + # doc2bow must raise a TypeError if passed a string instead of array of strings by accident + self.assertRaises(TypeError, d.doc2bow, "žluťoučký") + + # unicode must be converted to utf8 + self.assertEqual(d.doc2bow([u'\u017elu\u0165ou\u010dk\xfd']), [(0, 1)]) + def test_saveAsText_and_loadFromText(self): """`Dictionary` can be saved as textfile and loaded again from textfile. """ tmpf = get_tmpfile('dict_test.txt') diff --git a/gensim/test/test_corpora_hashdictionary.py b/gensim/test/test_corpora_hashdictionary.py index 262c54e219..be19db8e39 100644 --- a/gensim/test/test_corpora_hashdictionary.py +++ b/gensim/test/test_corpora_hashdictionary.py @@ -167,6 +167,16 @@ def test_saveAsText(self): d.save_as_text(tmpf) self.assertTrue(os.path.exists(tmpf)) + def test_saveAsTextBz2(self): + """ `HashDictionary` can be saved & loaded as compressed pickle. """ + tmpf = get_tmpfile('dict_test.txt.bz2') + # use some utf8 strings, to test encoding serialization + d = HashDictionary(['žloťoučký koníček'.split(), 'Малйж обльйквюэ ат эжт'.split()]) + d.save(tmpf) + self.assertTrue(os.path.exists(tmpf)) + d2 = d.load(tmpf) + self.assertEqual(len(d), len(d2)) + if __name__ == '__main__': diff --git a/gensim/utils.py b/gensim/utils.py index 0ce2767cf3..b659122a76 100644 --- a/gensim/utils.py +++ b/gensim/utils.py @@ -24,6 +24,8 @@ from htmlentitydefs import name2codepoint as n2cp # for `decode_htmlentities` import multiprocessing import shutil +import traceback + try: from pattern.en import parse @@ -49,6 +51,7 @@ def _synched(func): def _synchronizer(self, *args, **kwargs): tlock = getattr(self, tlockname) logger.debug("acquiring lock %r for %s" % (tlockname, func.func_name)) + with tlock: # use lock as a context manager to perform safe acquire/release pairs logger.debug("acquired lock %r for %s" % (tlockname, func.func_name)) result = func(self, *args, **kwargs) @@ -59,6 +62,10 @@ def _synchronizer(self, *args, **kwargs): class NoCM(object): + def acquire(self): + pass + def release(self): + pass def __enter__(self): pass def __exit__(self, type, value, traceback): @@ -119,14 +126,14 @@ def tokenize(text, lowercase=False, deacc=False, errors="strict", to_lower=False yield match.group() -def simple_preprocess(doc): +def simple_preprocess(doc, deacc=False): """ Convert a document into a list of tokens. This lowercases, tokenizes, stems, normalizes etc. -- the output are final, utf8 encoded strings that won't be processed any further. """ - tokens = [token.encode('utf8') for token in tokenize(doc, lower=True, errors='ignore') + tokens = [token.encode('utf8') for token in tokenize(doc, lower=True, deacc=deacc, errors='ignore') if 2 <= len(token) <= 15 and not token.startswith('_')] return tokens @@ -406,90 +413,110 @@ def chunkize_serial(iterable, chunksize, as_numpy=False): -def chunkize(corpus, chunksize, maxsize=0, as_numpy=False): - """ - Split a stream of values into smaller chunks. - Each chunk is of length `chunksize`, except the last one which may be smaller. - A once-only input stream (`corpus` from a generator) is ok, chunking is done - efficiently via itertools. +class InputQueue(multiprocessing.Process): + def __init__(self, q, corpus, chunksize, maxsize, as_numpy): + super(InputQueue, self).__init__() + self.q = q + self.maxsize = maxsize + self.corpus = corpus + self.chunksize = chunksize + self.as_numpy = as_numpy + + def run(self): + if self.as_numpy: + import numpy # don't clutter the global namespace with a dependency on numpy + it = iter(self.corpus) + while True: + chunk = itertools.islice(it, self.chunksize) + if self.as_numpy: + # HACK XXX convert documents to numpy arrays, to save memory. + # This also gives a scipy warning at runtime: + # "UserWarning: indices array has non-integer dtype (float64)" + wrapped_chunk = [[numpy.asarray(doc) for doc in chunk]] + else: + wrapped_chunk = [list(chunk)] - If `maxsize > 1`, don't wait idly in between successive chunk `yields`, but - rather keep filling a short queue (of size at most `maxsize`) with forthcoming - chunks in advance. This is realized by starting a separate process, and is - meant to reduce I/O delays, which can be significant when `corpus` comes - from a slow medium (like harddisk). + if not wrapped_chunk[0]: + self.q.put(None, block=True) + break - If `maxsize==0`, don't fool around with parallelism and simply yield the chunksize - via `chunkize_serial()` (no I/O optimizations). + try: + qsize = self.q.qsize() + except NotImplementedError: + qsize = '?' + logger.debug("prepared another chunk of %i documents (qsize=%s)" % + (len(wrapped_chunk[0]), qsize)) + self.q.put(wrapped_chunk.pop(), block=True) +#endclass InputQueue - >>> for chunk in chunkize(xrange(10), 4): print chunk - [0, 1, 2, 3] - [4, 5, 6, 7] - [8, 9] - """ - class InputQueue(multiprocessing.Process): - def __init__(self, q, corpus, chunksize, maxsize, as_numpy): - super(InputQueue, self).__init__() - self.q = q - self.maxsize = maxsize - self.corpus = corpus - self.chunksize = chunksize - self.as_numpy = as_numpy +if os.name == 'nt': + logger.info("detected Windows; aliasing chunkize to chunkize_serial") - def run(self): - if self.as_numpy: - import numpy # don't clutter the global namespace with a dependency on numpy - it = iter(self.corpus) + def chunkize(corpus, chunksize, maxsize=0, as_numpy=False): + for chunk in chunkize_serial(corpus, chunksize, as_numpy=as_numpy): + yield chunk +else: + def chunkize(corpus, chunksize, maxsize=0, as_numpy=False): + """ + Split a stream of values into smaller chunks. + Each chunk is of length `chunksize`, except the last one which may be smaller. + A once-only input stream (`corpus` from a generator) is ok, chunking is done + efficiently via itertools. + + If `maxsize > 1`, don't wait idly in between successive chunk `yields`, but + rather keep filling a short queue (of size at most `maxsize`) with forthcoming + chunks in advance. This is realized by starting a separate process, and is + meant to reduce I/O delays, which can be significant when `corpus` comes + from a slow medium (like harddisk). + + If `maxsize==0`, don't fool around with parallelism and simply yield the chunksize + via `chunkize_serial()` (no I/O optimizations). + + >>> for chunk in chunkize(xrange(10), 4): print chunk + [0, 1, 2, 3] + [4, 5, 6, 7] + [8, 9] + + """ + assert chunksize > 0 + + if maxsize > 0: + q = multiprocessing.Queue(maxsize=maxsize) + worker = InputQueue(q, corpus, chunksize, maxsize=maxsize, as_numpy=as_numpy) + worker.daemon = True + worker.start() while True: - chunk = itertools.islice(it, self.chunksize) - if self.as_numpy: - # HACK XXX convert documents to numpy arrays, to save memory. - # This also gives a scipy warning at runtime: - # "UserWarning: indices array has non-integer dtype (float64)" - wrapped_chunk = [[numpy.asarray(doc) for doc in chunk]] - else: - wrapped_chunk = [list(chunk)] - - if not wrapped_chunk[0]: - self.q.put(None, block=True) + chunk = [q.get(block=True)] + if chunk[0] is None: break + yield chunk.pop() + else: + for chunk in chunkize_serial(corpus, chunksize, as_numpy=as_numpy): + yield chunk - try: - qsize = self.q.qsize() - except NotImplementedError: - qsize = '?' - logger.debug("prepared another chunk of %i documents (qsize=%s)" % - (len(wrapped_chunk[0]), qsize)) - self.q.put(wrapped_chunk.pop(), block=True) - #endclass InputQueue - - assert chunksize > 0 - - if maxsize > 0: - q = multiprocessing.Queue(maxsize=maxsize) - worker = InputQueue(q, corpus, chunksize, maxsize=maxsize, as_numpy=as_numpy) - worker.daemon = True - worker.start() - while True: - chunk = [q.get(block=True)] - if chunk[0] is None: - break - yield chunk.pop() - else: - for chunk in chunkize_serial(corpus, chunksize): - yield chunk + +def smart_open(fname, mode): + from os import path + _, ext = path.splitext(fname) + if ext == '.bz2': + from bz2 import BZ2File + return BZ2File(fname, mode) + if ext == '.gz': + from gzip import GzipFile + return GzipFile(fname, mode) + return open(fname, mode) def pickle(obj, fname, protocol=-1): """Pickle object `obj` to file `fname`.""" - with open(fname, 'wb') as fout: # 'b' for binary, needed on Windows + with smart_open(fname, 'wb') as fout: # 'b' for binary, needed on Windows cPickle.dump(obj, fout, protocol=protocol) def unpickle(fname): """Load pickled object from `fname`""" - return cPickle.load(open(fname, 'rb')) + return cPickle.load(smart_open(fname, 'rb')) def revdict(d): @@ -586,15 +613,26 @@ def pyro_daemon(name, obj, random_suffix=False, ip=None, port=None): daemon.requestLoop() -# the following is only available when the optional 'pattern' package is installed if HAS_PATTERN: - ALLOWED_TAGS = re.compile('(NN|VB|JJ|RB)') # ignore everything except nouns, verbs, adjectives and adverbs - - def lemmatize(content): + def lemmatize(content, light=False, allowed_tags=re.compile('(NN|VB|JJ|RB)')): """ - Use the English lemmatizer from the `pattern` package to extract tokens in - their base form (lemmas: "are, is, being"->"be" etc.). - This is a smarter version of stemming. + This function is only available when the optional 'pattern' package is installed. + + Use the English lemmatizer from `pattern` to extract tokens in + their base form=lemma, e.g. "are, is, being" -> "be" etc. + This is a smarter version of stemming. Only consider nouns, verbs, adjectives + and adverbs by default (=all other lemmas are discarded). + + >>> lemmatize('Hello World! How is it going?! Nonexistentword, 21') + ['world/NN', 'be/VB', 'go/VB', 'nonexistentword/NN'] + + From http://www.clips.ua.ac.be/pages/pattern-en#parser : + + The parser is built on a Brill lexicon of tagged words and rules to + improve the tags context-wise. With light=False, it uses Brill's contextual + rules. With light=True it uses Jason Wiener's simpler ruleset. This + ruleset is 5-10x faster but also 25% less accurate. + """ # tokenization in `pattern` is weird; it gets thrown off by non-letters, # producing '==relate/VBN' or '**/NN'... try to preprocess the text a little @@ -604,12 +642,12 @@ def lemmatize(content): # use simpler, modified pattern.text.en.text.parser.parse that doesn't # collapse the output at the end: https://github.com/piskvorky/pattern - parsed = parse(content, lemmata=True, collapse=False) + parsed = parse(content, lemmata=True, collapse=False, light=light) result = [] for sentence in parsed: for token, tag, _, _, lemma in sentence: if 2 <= len(lemma) <= 15 and not lemma.startswith('_'): - if ALLOWED_TAGS.match(tag): + if allowed_tags.match(tag): lemma += "/" + tag[:2] result.append(lemma.encode('utf8')) return result