Query tuning / Advanced Query Tuning

Author: clarklindev

03.postgreSQL.md

@@ -3,12 +3,12 @@
 ---
 
 ## Table of contents
-  [1. What is postgreSQL](#what-is-postgresql)
-  [2. installation](#installation)
+  + [1. What is postgreSQL](#what-is-postgresql)
+  + [2. installation](#installation)
 
 ---
 
-#### What is PostgreSQL
+## What is PostgreSQL
 
 * Installing postgreSQL installs pgadmin.
 * Pgadmin is a web-based tool to manage and inspect a postgres database.
@@ -18,7 +18,7 @@
 * view table definition by clicking on database -> expanding schemas -> tables
   (right click on table) -> view/edit data -> all rows
 
-#### Installation
+## Installation
 
 * installation https://www.enterprisedb.com/downloads/postgres-postgresql-downloads
 * during install -> uncheck stack builder
@@ -27,7 +27,9 @@
 * clicking on servers -> requested password is password for localmachine.
 * have to refresh table to see updated info.
 
-#### creating database
+---
+
+## creating database
 
 1. servers -> localhost -> databases -> (right-click create)
 2. open query tool by right clicking on specific database
@@ -184,6 +186,10 @@ filename -> ... -> *(format -> all files or .sql) -> *(select file to restore fr
 4. disable -> trigger -> yes
 5. miscelleaneous / behavior -> verbose messages -> yes
 
+---
+
+## Understanding the Internals of PostgreSQL
+
 #### where postgreSQL stores data
 ```SQL
 show data_directory;
@@ -280,4 +286,172 @@ to calculate length of data. click on 3rd byte of 4 the bytes. look at int16 val
 length of first item is 172. 
 
 68.6.1 Table row layout - And item is further made up of a fixed-size header (23 bytes), followed by optional data, then is the actual user data.
-the pointer to the first value -> data inspector -> Int16 -> eg. 203 is the actual data is the id of the data in the postgreSQL table.
+the pointer to the first value -> data inspector -> Int16 -> eg. 203 is the actual data is the id of the data in the postgreSQL table.
+
+---
+
+## Indexes
+Indexes help with performance
+* without indexes, to search data the data has to be loaded up to memory (large performance cost)
+* then search through all data for what we are looking for (iteration of rows) also know as 'Full table scan'
+* indexes are data structures that efficiently tells us which block and index a record is stored in a heap file on harddrive.
+
+#### how an index works?
+* basically how index works is it picks a column we want to have a fast look up on.
+* you can have an index on multiple columns. 
+* then we look at our heap file and that property we want to extract for every row and we then record the block and index and that property (column) value.
+* we then sort in some meaningful way, eg. alphabetically.
+* put this list of data into a binary tree
+* root node gets instructions on where the new node should go eg. left or right (elimination of wrong node direction)
+* we cut down the number of blocks we have to read data from.
+
+#### creating an index
+* creating an index on users table -> username column.
+* view all indexes for a database -> schemas -> table -> users -> indexes -> *(refresh) -> 
+* naming convention: users_username_idx
+
+```SQL
+CREATE INDEX on users (username);
+```
+
+#### deleting index
+```SQL
+DROP INDEX users_username_idx;
+```
+
+#### Benchmarking queries with indexes
+Keyword : 'EXPLAIN ANALYZE' in front of SELECT statement
+Indexes make queries run 16X faster
+
+```SQL
+EXPLAIN ANALYZE SELECT * 
+FROM users
+WHERE username = 'Emil30';
+```
+
+#### downside of indexes
+Indexes are performance benefit, however it can sometimes slow down database.
+Indexes take up storage space.
+
+```SQL
+--Finding out how much space index use.
+SELECT pg_size_pretty(pg_relation_size('users_username_idx'))
+```
+
+Indexes slow down insert/update/delete because index has to be updated (especially tables that get updated frequently)
+
+Index might not get used. 
+
+#### PostgreSQL autogenerated indexes
+postgres does manage its own indexes.
+Idexes are created automatically by postgresql for columns (and therefore you never have to create your own indexes for these):
+
+* primary keys
+* column that has UNIQUE constraint
+
+Query to see which indexes (relkind = i) actually exist for database 
+
+```SQL
+SELECT relname, relkind
+FROM pg_class
+WHERE relkind = 'i'  -- type: index
+```
+
+---
+
+## Query tuning
+When postgres receives query
+1. goes into parser: splits up the query and figures out what each SQL keyword means
+2. after evaluation, it builds a query tree (programatic tree of query)
+3. rewriter takes tree and makes adjustments to it.
+4. planner takes a look at query tree and strategizes to get that information.
+5. planner picks a strategy 
+6. executer executes strategy
+
+
+keywords:
+
+* EXPLAIN - build query plan and display info about it. (plan)
+
+* EXPLAIN ANALYZE - build query plan, RUN IT!, and display info about it. (plan + run)
+
+both keywords are only used for performance optimization and never for production.
+pgadmin has a explain analyze button (make sure all options are checked).
+How to read explain analyze output:
+go to deepest level of query node of query plan (most indented), we can imagine that they keep passing information up to the nearest parent node.
+
+```SQL
+SELECT * 
+FROM pg_stats
+WHERE tablename = 'users';
+```
+
+postgres is able to make asumptions of (rows, width) in output of EXPLAIN ANALYZE because it actually keeps stats about whats going on in the database.
+
+##### Cost
+EXPLAIN ANALYZE -> cost
+amount of time to execute a part of a query.
+* query plan has something like (cost=9.4...1233.11) 
+	- 9.4 is the cost to calculate the first row value
+	- 1233.11 is the cost to calculate all rows
+* cost of parent node in query plan is the sum of all child nodes' costs.
+
+---
+
+## Advanced Query tuning 
+
+using index vs loading all up in memory and reading sequencially:
+
+* assumption that jumping to specific blocks/random child page is 4x as long (eg. x2 pages loaded) = 4 x 2 = 8 units
+
+* compared to sequencial reading x1 (assume 1x base value) which is loading for each page x 110 per file to search through list = 110 x 1 = 110 units
+
+eg. EXPLAIN ANALYZE - 
+
+Seq Scan on comments (cost=0.00...1589.10 rows=60410 width=72) (actual time = 0.008...14.29). 60410 rows, 985 pages.
+
+loading up a page to memory is more expensive than sequencial reading of rows
+attempting formula:
+
+1.0 (assume score is 1.0 as base to judge everything else in terms of estimate costs)
+0.01  (assume loading row is 1% cost of loading up row)
+
+(# pages) * 1.0 + (# rows) * (0.01)
+
+(985) * 1.0 + (60410) * (0.01) = 1589.1 (same estimate as EXPLAIN ANALYZE execution)
+
+##### actual formula for cost for any step of query plan (EXPLAIN ANALYZE)
+
+link specifics default values for costs to calculations. sequencial base cost (seq_page_cost) is the default and all other costs are relative to that.
+
+[runtime config query - postgresql.org/docs/current/runtime-config-query.html](http://postgresql.org/docs/current/runtime-config-query.html) 
+19.7 Query Planning -> 19.7.2 Planner cost constants
+
+random_page_cost -> 4x more expensive as fetching page in order (seq_page_cost)
+
+seq_page_cost -> 1x (default base that all other costs relative to)
+
+cpu_tuple_cost -> 0.1 (processing a single tuple (row) is 1% as expensive as fetching a page in order (seq_page_cost))
+
+cpu_index_tuple_cost - 0.005 (processing a tuple from an index is 50% as expensive as processing a real row (cpu_tuple_cost))
+
+cpu_operator_cost - 0.0025 (running an operator or function is 50% as expensive as processing an index tuple (cpu_index_tuple_cost))
+
+##### Cost for steps of query plan = 
+(# pages read sequentially) * seq_page_cost
++
+(# page read at random) * random_page_cost
++
+(# rows scanned) * cpu_tuple_cost
++
+(# index entries scanned) * cpu_index_tuple_cost
++
+(# times function/operator evaluated) * cpu_operator_cost
+
+
+Cost for sequential read = 
+(# pages read sequentially) * seq_page_cost
++ 
+(# rows scanned) * cpu_tuple_cost
+
+---