ch02_basics/Concept10_queue_text.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Ch `02`: Concept `10`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Queue of text from file"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The previous notebook (Concept 09) introduced queues. Let's see how we can use them with real world data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Download a corpus of text.\n",
    "\n",
    "    $ wget http://norvig.com/big.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a queue of filenames:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "\n",
    "filename_queue = tf.train.string_input_producer([\"big.txt\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Define an op to dequeue a line from file:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "reader = tf.TextLineReader()\n",
    "key_op, value_op = reader.read(filename_queue)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Start all queue runners collected in the graph:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "sess = tf.InteractiveSession()\n",
    "coord = tf.train.Coordinator()\n",
    "threads = tf.train.start_queue_runners(coord=coord)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Try reading lines from the file by dequeuing: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "b'big.txt:1' b'The Project Gutenberg EBook of The Adventures of Sherlock Holmes'\n",
      "b'big.txt:2' b'by Sir Arthur Conan Doyle'\n",
      "b'big.txt:3' b'(#15 in our series by Sir Arthur Conan Doyle)'\n",
      "b'big.txt:4' b''\n",
      "b'big.txt:5' b'Copyright laws are changing all over the world. Be sure to check the'\n",
      "b'big.txt:6' b'copyright laws for your country before downloading or redistributing'\n",
      "b'big.txt:7' b'this or any other Project Gutenberg eBook.'\n",
      "b'big.txt:8' b''\n",
      "b'big.txt:9' b'This header should be the first thing seen when viewing this Project'\n",
      "b'big.txt:10' b'Gutenberg file.  Please do not remove it.  Do not change or edit the'\n",
      "b'big.txt:11' b'header without written permission.'\n",
      "b'big.txt:12' b''\n",
      "b'big.txt:13' b'Please read the \"legal small print,\" and other information about the'\n",
      "b'big.txt:14' b'eBook and Project Gutenberg at the bottom of this file.  Included is'\n",
      "b'big.txt:15' b'important information about your specific rights and restrictions in'\n",
      "b'big.txt:16' b'how the file may be used.  You can also find out about how to make a'\n",
      "b'big.txt:17' b'donation to Project Gutenberg, and how to get involved.'\n",
      "b'big.txt:18' b''\n",
      "b'big.txt:19' b''\n",
      "b'big.txt:20' b'**Welcome To The World of Free Plain Vanilla Electronic Texts**'\n",
      "b'big.txt:21' b''\n",
      "b'big.txt:22' b'**eBooks Readable By Both Humans and By Computers, Since 1971**'\n",
      "b'big.txt:23' b''\n",
      "b'big.txt:24' b'*****These eBooks Were Prepared By Thousands of Volunteers!*****'\n",
      "b'big.txt:25' b''\n",
      "b'big.txt:26' b''\n",
      "b'big.txt:27' b'Title: The Adventures of Sherlock Holmes'\n",
      "b'big.txt:28' b''\n",
      "b'big.txt:29' b'Author: Sir Arthur Conan Doyle'\n",
      "b'big.txt:30' b''\n",
      "b'big.txt:31' b'Release Date: March, 1999  [EBook #1661]'\n",
      "b'big.txt:32' b'[Most recently updated: November 29, 2002]'\n",
      "b'big.txt:33' b''\n",
      "b'big.txt:34' b'Edition: 12'\n",
      "b'big.txt:35' b''\n",
      "b'big.txt:36' b'Language: English'\n",
      "b'big.txt:37' b''\n",
      "b'big.txt:38' b'Character set encoding: ASCII'\n",
      "b'big.txt:39' b''\n",
      "b'big.txt:40' b'*** START OF THE PROJECT GUTENBERG EBOOK, THE ADVENTURES OF SHERLOCK HOLMES ***'\n",
      "b'big.txt:41' b''\n",
      "b'big.txt:42' b''\n",
      "b'big.txt:43' b''\n",
      "b'big.txt:44' b''\n",
      "b'big.txt:45' b'(Additional editing by Jose Menendez)'\n",
      "b'big.txt:46' b''\n",
      "b'big.txt:47' b''\n",
      "b'big.txt:48' b''\n",
      "b'big.txt:49' b'THE ADVENTURES OF'\n",
      "b'big.txt:50' b'SHERLOCK HOLMES'\n",
      "b'big.txt:51' b''\n",
      "b'big.txt:52' b'BY'\n",
      "b'big.txt:53' b''\n",
      "b'big.txt:54' b'SIR ARTHUR CONAN DOYLE'\n",
      "b'big.txt:55' b''\n",
      "b'big.txt:56' b'CONTENTS'\n",
      "b'big.txt:57' b''\n",
      "b'big.txt:58' b'I.\\tA Scandal in Bohemia'\n",
      "b'big.txt:59' b'II.\\tThe Red-Headed League'\n",
      "b'big.txt:60' b'III.\\tA Case of Identity'\n",
      "b'big.txt:61' b'IV.\\tThe Boscombe Valley Mystery'\n",
      "b'big.txt:62' b'V.\\tThe Five Orange Pips'\n",
      "b'big.txt:63' b'VI.\\tThe Man with the Twisted Lip'\n",
      "b'big.txt:64' b'VII.\\tThe Adventure of the Blue Carbuncle'\n",
      "b'big.txt:65' b'VIII.\\tThe Adventure of the Speckled Band'\n",
      "b'big.txt:66' b\"IX.\\tThe Adventure of the Engineer's Thumb\"\n",
      "b'big.txt:67' b'X.\\tThe Adventure of the Noble Bachelor'\n",
      "b'big.txt:68' b'XI.\\tThe Adventure of the Beryl Coronet'\n",
      "b'big.txt:69' b'XII.\\tThe Adventure of the Copper Beeches'\n",
      "b'big.txt:70' b''\n",
      "b'big.txt:71' b''\n",
      "b'big.txt:72' b'ADVENTURE  I.  A SCANDAL IN BOHEMIA'\n",
      "b'big.txt:73' b''\n",
      "b'big.txt:74' b'I.'\n",
      "b'big.txt:75' b''\n",
      "b'big.txt:76' b''\n",
      "b'big.txt:77' b\"To Sherlock Holmes she is always the woman. I have seldom heard him mention her under any other name. In his eyes she eclipses and predominates the whole of her sex. It was not that he felt any emotion akin to love for Irene Adler. All emotions, and that one particularly, were abhorrent to his cold, precise but admirably balanced mind. He was, I take it, the most perfect reasoning and observing machine that the world has seen, but as a lover he would have placed himself in a false position. He never spoke of the softer passions, save with a gibe and a sneer. They were admirable things for the observer--excellent for drawing the veil from men's motives and actions. But for the trained reasoner to admit such intrusions into his own delicate and finely adjusted temperament was to introduce a distracting factor which might throw a doubt upon all his mental results. Grit in a sensitive instrument, or a crack in one of his own high-power lenses, would not be more disturbing than a strong emotion in a nature such as his. And yet there was but one woman to him, and that woman was the late Irene Adler, of dubious and questionable memory.\"\n",
      "b'big.txt:78' b''\n",
      "b'big.txt:79' b'I had seen little of Holmes lately. My marriage had drifted us away from each other. My own complete happiness, and the home-centred interests which rise up around the man who first finds himself master of his own establishment, were sufficient to absorb all my attention, while Holmes, who loathed every form of society with his whole Bohemian soul, remained in our lodgings in Baker Street, buried among his old books, and alternating from week to week between cocaine and ambition, the drowsiness of the drug, and the fierce energy of his own keen nature. He was still, as ever, deeply attracted by the study of crime, and occupied his immense faculties and extraordinary powers of observation in following out those clues, and clearing up those mysteries which had been abandoned as hopeless by the official police. From time to time I heard some vague account of his doings: of his summons to Odessa in the case of the Trepoff murder, of his clearing up of the singular tragedy of the Atkinson brothers at Trincomalee, and finally of the mission which he had accomplished so delicately and successfully for the reigning family of Holland. Beyond these signs of his activity, however, which I merely shared with all the readers of the daily press, I knew little of my former friend and companion.'\n",
      "b'big.txt:80' b''\n",
      "b'big.txt:81' b'One night--it was on the twentieth of March, 1888--I was returning from a journey to a patient (for I had now returned to civil practice), when my way led me through Baker Street. As I passed the well-remembered door, which must always be associated in my mind with my wooing, and with the dark incidents of the Study in Scarlet, I was seized with a keen desire to see Holmes again, and to know how he was employing his extraordinary powers. His rooms were brilliantly lit, and, even as I looked up, I saw his tall, spare figure pass twice in a dark silhouette against the blind. He was pacing the room swiftly, eagerly, with his head sunk upon his chest and his hands clasped behind him. To me, who knew his every mood and habit, his attitude and manner told their own story. He was at work again. He had risen out of his drug-created dreams and was hot upon the scent of some new problem. I rang the bell and was shown up to the chamber which had formerly been in part my own.'\n",
      "b'big.txt:82' b''\n",
      "b'big.txt:83' b'His manner was not effusive. It seldom was; but he was glad, I think, to see me. With hardly a word spoken, but with a kindly eye, he waved me to an armchair, threw across his case of cigars, and indicated a spirit case and a gasogene in the corner. Then he stood before the fire and looked me over in his singular introspective fashion.'\n",
      "b'big.txt:84' b''\n",
      "b'big.txt:85' b'\"Wedlock suits you,\" he remarked. \"I think, Watson, that you have put on seven and a half pounds since I saw you.\"'\n",
      "b'big.txt:86' b''\n",
      "b'big.txt:87' b'\"Seven!\" I answered.'\n",
      "b'big.txt:88' b''\n",
      "b'big.txt:89' b'\"Indeed, I should have thought a little more. Just a trifle more, I fancy, Watson. And in practice again, I observe. You did not tell me that you intended to go into harness.\"'\n",
      "b'big.txt:90' b''\n",
      "b'big.txt:91' b'\"Then, how do you know?\"'\n",
      "b'big.txt:92' b''\n",
      "b'big.txt:93' b'\"I see it, I deduce it. How do I know that you have been getting yourself very wet lately, and that you have a most clumsy and careless servant girl?\"'\n",
      "b'big.txt:94' b''\n",
      "b'big.txt:95' b'\"My dear Holmes,\" said I, \"this is too much. You would certainly have been burned, had you lived a few centuries ago. It is true that I had a country walk on Thursday and came home in a dreadful mess, but as I have changed my clothes I can\\'t imagine how you deduce it. As to Mary Jane, she is incorrigible, and my wife has given her notice, but there, again, I fail to see how you work it out.\"'\n",
      "b'big.txt:96' b''\n",
      "b'big.txt:97' b'He chuckled to himself and rubbed his long, nervous hands together.'\n",
      "b'big.txt:98' b''\n",
      "b'big.txt:99' b'\"It is simplicity itself,\" said he; \"my eyes tell me that on the inside of your left shoe, just where the firelight strikes it, the leather is scored by six almost parallel cuts. Obviously they have been caused by someone who has very carelessly scraped round the edges of the sole in order to remove crusted mud from it. Hence, you see, my double deduction that you had been out in vile weather, and that you had a particularly malignant boot-slitting specimen of the London slavey. As to your practice, if a gentleman walks into my rooms smelling of iodoform, with a black mark of nitrate of silver upon his right forefinger, and a bulge on the right side of his top-hat to show where he has secreted his stethoscope, I must be dull, indeed, if I do not pronounce him to be an active member of the medical profession.\"'\n",
      "b'big.txt:100' b''\n"
     ]
    }
   ],
   "source": [
    "for i in range(100):\n",
    "    key, value = sess.run([key_op, value_op])\n",
    "    print(key, value)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}