Feature/database operations #41

jakubg89 · 2023-01-20T17:15:04Z

Clearing old records.
Making copy of records with positive experience value.
Added charm points script.

…nes.

First db fill.

* File and directory creation * Adding proxy scrapper script. Result: list of elite proxy with port * Script edited. Deleted unneeded lines, deleted bs, function returns clear dataframe. * Saving to csv added * Delete duplicates script with update of list in file added * Exception FileNotFound added. Checking status code. * Result of proxy scrapper * Convert proxy to list. Files reorganization. * Header list. * ImportError fix csv file added to gitignore * Scrapping world's list from tibia.com * Scrapping character information from tibia.com. * Temporary data file * Highscores scrapper. Early version * Deleted proxy usage. Deleted unneeded code. * Deleted unneeded code. * Main page html code added. Footer changing font size. Added YouTube, Twitch buttons to nav bar * Fixing spacing, typo and content information for future work * Test news added * Database added. Creating models. * Test realase (#12) * File and directory creation * Adding proxy scrapper script. Result: list of elite proxy with port * Script edited. Deleted unneeded lines, deleted bs, function returns clear dataframe. * Saving to csv added * Delete duplicates script with update of list in file added * Exception FileNotFound added. Checking status code. * Result of proxy scrapper * Convert proxy to list. Files reorganization. * Header list. * ImportError fix csv file added to gitignore * Scrapping world's list from tibia.com * Scrapping character information from tibia.com. * Temporary data file * Highscores scrapper. Early version * Deleted proxy usage. Deleted unneeded code. * Deleted unneeded code. * Main page html code added. Footer changing font size. Added YouTube, Twitch buttons to nav bar * Fixing spacing, typo and content information for future work * Test news added * Database added. Creating models. * Pages/main page (#14) * Test commit * Init added to scripts * Gitignore updated * Models managed changed to true * Functions to get data from tibiadata API * Checking data from api and compare with database. If not exist, insert. * Views at main page for news ticker * Main page news ticker data loaded from database. * Added condition to selecting data for news ticker * Deleted unneeded code. * Adding condition to check if list is not empty. * Variable renamed * Getting data about boosted boss. * Boosted creature models created. * Adding boss to database. Replace function added * Display boosted boss. * Get boosted creature information * Adding boosted creature to database * Display information about boosted creature. * Fixed naming and spaces * Added getting news history * Html formating of tibia.com news. Inserting news into database. Some name changes. * Added column * Views for news * Added news on main page with testimonials * Pages/main page (#16) * Test commit * Init added to scripts * Gitignore updated * Models managed changed to true * Functions to get data from tibiadata API * Checking data from api and compare with database. If not exist, insert. * Views at main page for news ticker * Main page news ticker data loaded from database. * Added condition to selecting data for news ticker * Deleted unneeded code. * Adding condition to check if list is not empty. * Variable renamed * Getting data about boosted boss. * Boosted creature models created. * Adding boss to database. Replace function added * Display boosted boss. * Get boosted creature information * Adding boosted creature to database * Display information about boosted creature. * Fixed naming and spaces * Added getting news history * Html formating of tibia.com news. Inserting news into database. Some name changes. * Added column * Views for news * Added news on main page with testimonials * Names changed * Names changed * About page added. (#17) * .format changed to f * Typo fix * Collecting data from tibia.com about worlds, vocation, beprotection and categories with values used on tibia.com. (#18) Saved to json file. * Pages/worlds (#19) * Adding script to scrap world type values from tibia.com * Adding script to get worlds list and details from tibiadata api * Models, added auto increment * urls to worlds list added * Added loading settings from django * Added scripts for datatables to base * Added views for all worlds page * Added html page for all worlds * Orm added * Online history added, small changes in other models * creation date added * Charts deleted. Added loop for table creation with queryset data * Changed function name * Main() added for develop purpose * Charts added, fixed js inside django template with variables * Online history data for chart added. Creation history data for chart added. * JSON file with creation history added * Custom tag added. * Pages/single world (#20) * Url changed * Urls, added variable threw url * Dynamic views for single world added. * Urls added to single world * Single world site template added * Custom tag added ( iso date to y, m, d) * Import clean up * Arguments clean up * Test table added * Search button connected to views. * Search button connected to views. * Pages/find character (#21) * Get character information from api * Changed icon * Html site, display information from api. * Views for character search added. * Adding comment to unused buttons * Custom tag for data edited. * Forms file added * Updated working features. * New features (#22) * File and directory creation * Adding proxy scrapper script. Result: list of elite proxy with port * Script edited. Deleted unneeded lines, deleted bs, function returns clear dataframe. * Saving to csv added * Delete duplicates script with update of list in file added * Exception FileNotFound added. Checking status code. * Result of proxy scrapper * Convert proxy to list. Files reorganization. * Header list. * ImportError fix csv file added to gitignore * Scrapping world's list from tibia.com * Scrapping character information from tibia.com. * Temporary data file * Highscores scrapper. Early version * Deleted proxy usage. Deleted unneeded code. * Deleted unneeded code. * Main page html code added. Footer changing font size. Added YouTube, Twitch buttons to nav bar * Fixing spacing, typo and content information for future work * Test news added * Database added. Creating models. * Pages/main page (#14) * Test commit * Init added to scripts * Gitignore updated * Models managed changed to true * Functions to get data from tibiadata API * Checking data from api and compare with database. If not exist, insert. * Views at main page for news ticker * Main page news ticker data loaded from database. * Added condition to selecting data for news ticker * Deleted unneeded code. * Adding condition to check if list is not empty. * Variable renamed * Getting data about boosted boss. * Boosted creature models created. * Adding boss to database. Replace function added * Display boosted boss. * Get boosted creature information * Adding boosted creature to database * Display information about boosted creature. * Fixed naming and spaces * Added getting news history * Html formating of tibia.com news. Inserting news into database. Some name changes. * Added column * Views for news * Added news on main page with testimonials * Pages/main page (#16) * Test commit * Init added to scripts * Gitignore updated * Models managed changed to true * Functions to get data from tibiadata API * Checking data from api and compare with database. If not exist, insert. * Views at main page for news ticker * Main page news ticker data loaded from database. * Added condition to selecting data for news ticker * Deleted unneeded code. * Adding condition to check if list is not empty. * Variable renamed * Getting data about boosted boss. * Boosted creature models created. * Adding boss to database. Replace function added * Display boosted boss. * Get boosted creature information * Adding boosted creature to database * Display information about boosted creature. * Fixed naming and spaces * Added getting news history * Html formating of tibia.com news. Inserting news into database. Some name changes. * Added column * Views for news * Added news on main page with testimonials * Names changed * Names changed * About page added. (#17) * .format changed to f * Typo fix * Collecting data from tibia.com about worlds, vocation, beprotection and categories with values used on tibia.com. (#18) Saved to json file. * Pages/worlds (#19) * Adding script to scrap world type values from tibia.com * Adding script to get worlds list and details from tibiadata api * Models, added auto increment * urls to worlds list added * Added loading settings from django * Added scripts for datatables to base * Added views for all worlds page * Added html page for all worlds * Orm added * Online history added, small changes in other models * creation date added * Charts deleted. Added loop for table creation with queryset data * Changed function name * Main() added for develop purpose * Charts added, fixed js inside django template with variables * Online history data for chart added. Creation history data for chart added. * JSON file with creation history added * Custom tag added. * Pages/single world (#20) * Url changed * Urls, added variable threw url * Dynamic views for single world added. * Urls added to single world * Single world site template added * Custom tag added ( iso date to y, m, d) * Import clean up * Arguments clean up * Test table added * Search button connected to views. * Search button connected to views. * Pages/find character (#21) * Get character information from api * Changed icon * Html site, display information from api. * Views for character search added. * Adding comment to unused buttons * Custom tag for data edited. * Forms file added * Updated working features. * Pages/experience (#24) * Cleaning code * Cleaning code * Creating models for experience tracking * Collecting data from api, formating data from db for future df modification * collecting data changed * removed query * Filter latest data * Add id from db to latest data df * Select records that don't exist in db * Filter data and create name list * Check all names on tibia.com. If name was changed, if character is deleted. Creates list's for each option. * Insert all characters that are new and don't exist in DB and don't apear in deleted list or name change list. * Update name changes in character table. * Collect new id list from db and merge it to df * collect character data from day before * changing old names with new onews in df * Merge data for calculations * Filing up needed columns with calculated values for insert. * Generate world transfer df * Added coments for future steps * Changed naming, added working functions. Update db. Bulk data. Check what is missing in db. Check status of characters that dont exist in db. * Changed naming, added working functions. Update db. Bulk data. Check what is missing in db. Check status of characters that don't exist in db. * Experience tracking sites added. Views created. * Fixed long main page loading. * Select menu - added option * Models for record history. * Select menu - added blank option * Rollback changes * Getting daily records from each world. Insert into db. * Main page views changed to load immediately * Main page changed the best players to url * main function changed * Added top 5 in history * Added top of each world * Added top of each world and top 5 history to views. * Experience colored * Explore experience gain site added to sidebar * Explore experience gain site added * Explore experience gain views added. Filter form request and get data from db based on conditions * Explore experience gain urls added * Experience gain colored * Explore experience rewritten cuz of slow loading time and crashing. * Checkboxes checked based on filter request. * Explore html code formated * Explore views formatted. Queries changed for more clear view * Pages/world transfers (#25) * World transfers urls added * World transfers views created. * World transfers custom tags for displaying dict value and tibia coins amount added * World transfers html file added * Formatting file. * Pages/name changes (#26) * Name changes ulrs added. * Name changes custom tag for counting price added. * Name changes views added. * Name changes html site added. * Name changes html site added. * Various fixes (#29) * Get date putted into funciton * x * Adding news and news ticker to db refactored. Raw sql -> orm * Adding online players and boosted creatures to db refactored. Raw sql -> orm * Various fixes (#30) * Get date putted into funciton * x * Adding news and news ticker to db refactored. Raw sql -> orm * Adding online players and boosted creatures to db refactored. Raw sql -> orm * Adding requirements * Adding requirements * First deploy preparation (#32) * Pre deployment (#31) * File and directory creation * Adding proxy scrapper script. Result: list of elite proxy with port * Script edited. Deleted unneeded lines, deleted bs, function returns clear dataframe. * Saving to csv added * Delete duplicates script with update of list in file added * Exception FileNotFound added. Checking status code. * Result of proxy scrapper * Convert proxy to list. Files reorganization. * Header list. * ImportError fix csv file added to gitignore * Scrapping world's list from tibia.com * Scrapping character information from tibia.com. * Temporary data file * Highscores scrapper. Early version * Deleted proxy usage. Deleted unneeded code. * Deleted unneeded code. * Main page html code added. Footer changing font size. Added YouTube, Twitch buttons to nav bar * Fixing spacing, typo and content information for future work * Test news added * Database added. Creating models. * Test realase (#12) * File and directory creation * Adding proxy scrapper script. Result: list of elite proxy with port * Script edited. Deleted unneeded lines, deleted bs, function returns clear dataframe. * Saving to csv added * Delete duplicates script with update of list in file added * Exception FileNotFound added. Checking status code. * Result of proxy scrapper * Convert proxy to list. Files reorganization. * Header list. * ImportError fix csv file added to gitignore * Scrapping world's list from tibia.com * Scrapping character information from tibia.com. * Temporary data file * Highscores scrapper. Early version * Deleted proxy usage. Deleted unneeded code. * Deleted unneeded code. * Main page html code added. Footer changing font size. Added YouTube, Twitch buttons to nav bar * Fixing spacing, typo and content information for future work * Test news added * Database added. Creating models. * Pages/main page (#14) * Test commit * Init added to scripts * Gitignore updated * Models managed changed to true * Functions to get data from tibiadata API * Checking data from api and compare with database. If not exist, insert. * Views at main page for news ticker * Main page news ticker data loaded from database. * Added condition to selecting data for news ticker * Deleted unneeded code. * Adding condition to check if list is not empty. * Variable renamed * Getting data about boosted boss. * Boosted creature models created. * Adding boss to database. Replace function added * Display boosted boss. * Get boosted creature information * Adding boosted creature to database * Display information about boosted creature. * Fixed naming and spaces * Added getting news history * Html formating of tibia.com news. Inserting news into database. Some name changes. * Added column * Views for news * Added news on main page with testimonials * Pages/main page (#16) * Test commit * Init added to scripts * Gitignore updated * Models managed changed to true * Functions to get data from tibiadata API * Checking data from api and compare with database. If not exist, insert. * Views at main page for news ticker * Main page news ticker data loaded from database. * Added condition to selecting data for news ticker * Deleted unneeded code. * Adding condition to check if list is not empty. * Variable renamed * Getting data about boosted boss. * Boosted creature models created. * Adding boss to database. Replace function added * Display boosted boss. * Get boosted creature information * Adding boosted creature to database * Display information about boosted creature. * Fixed naming and spaces * Added getting news history * Html formating of tibia.com news. Inserting news into database. Some name changes. * Added column * Views for news * Added news on main page with testimonials * Names changed * Names changed * About page added. (#17) * .format changed to f * Typo fix * Collecting data from tibia.com about worlds, vocation, beprotection and categories with values used on tibia.com. (#18) Saved to json file. * Pages/worlds (#19) * Adding script to scrap world type values from tibia.com * Adding script to get worlds list and details from tibiadata api * Models, added auto increment * urls to worlds list added * Added loading settings from django * Added scripts for datatables to base * Added views for all worlds page * Added html page for all worlds * Orm added * Online history added, small changes in other models * creation date added * Charts deleted. Added loop for table creation with queryset data * Changed function name * Main() added for develop purpose * Charts added, fixed js inside django template with variables * Online history data for chart added. Creation history data for chart added. * JSON file with creation history added * Custom tag added. * Pages/single world (#20) * Url changed * Urls, added variable threw url * Dynamic views for single world added. * Urls added to single world * Single world site template added * Custom tag added ( iso date to y, m, d) * Import clean up * Arguments clean up * Test table added * Search button connected to views. * Search button connected to views. * Pages/find character (#21) * Get character information from api * Changed icon * Html site, display information from api. * Views for character search added. * Adding comment to unused buttons * Custom tag for data edited. * Forms file added * Updated working features. * New features (#22) * File and directory creation * Adding proxy scrapper script. Result: list of elite proxy with port * Script edited. Deleted unneeded lines, deleted bs, function returns clear dataframe. * Saving to csv added * Delete duplicates script with update of list in file added * Exception FileNotFound added. Checking status code. * Result of proxy scrapper * Convert proxy to list. Files reorganization. * Header list. * ImportError fix csv file added to gitignore * Scrapping world's list from tibia.com * Scrapping character information from tibia.com. * Temporary data file * Highscores scrapper. Early version * Deleted proxy usage. Deleted unneeded code. * Deleted unneeded code. * Main page html code added. Footer changing font size. Added YouTube, Twitch buttons to nav bar * Fixing spacing, typo and content information for future work * Test news added * Database added. Creating models. * Pages/main page (#14) * Test commit * Init added to scripts * Gitignore updated * Models managed changed to true * Functions to get data from tibiadata API * Checking data from api and compare with database. If not exist, insert. * Views at main page for news ticker * Main page news ticker data loaded from database. * Added condition to selecting data for news ticker * Deleted unneeded code. * Adding condition to check if list is not empty. * Variable renamed * Getting data about boosted boss. * Boosted creature models created. * Adding boss to database. Replace function added * Display boosted boss. * Get boosted creature information * Adding boosted creature to database * Display information about boosted creature. * Fixed naming and spaces * Added getting news history * Html formating of tibia.com news. Inserting news into database. Some name changes. * Added column * Views for news * Added news on main page with testimonials * Pages/main page (#16) * Test commit * Init added to scripts * Gitignore updated * Models managed changed to true * Functions to get data from tibiadata API * Checking data from api and compare with database. If not exist, insert. * Views at main page for news ticker * Main page news ticker data loaded from database. * Added condition to selecting data for news ticker * Deleted unneeded code. * Adding condition to check if list is not empty. * Variable renamed * Getting data about boosted boss. * Boosted creature models created. * Adding boss to database. Replace function added * Display boosted boss. * Get boosted creature information * Adding boosted creature to database * Display information about boosted creature. * Fixed naming and spaces * Added getting news history * Html formating of tibia.com news. Inserting news into database. Some name changes. * Added column * Views for news * Added news on main page with testimonials * Names changed * Names changed * About page added. (#17) * .format changed to f * Typo fix * Collecting data from tibia.com about worlds, vocation, beprotection and categories with values used on tibia.com. (#18) Saved to json file. * Pages/worlds (#19) * Adding script to scrap world type values from tibia.com * Adding script to get worlds list and details from tibiadata api * Models, added auto increment * urls to worlds list added * Added loading settings from django * Added scripts for datatables to base * Added views for all worlds page * Added html page for all worlds * Orm added * Online history added, small changes in other models * creation date added * Charts deleted. Added loop for table creation with queryset data * Changed function name * Main() added for develop purpose * Charts added, fixed js inside django template with variables * Online history data for chart added. Creation history data for chart added. * JSON file with creation history added * Custom tag added. * Pages/single world (#20) * Url changed * Urls, added variable threw url * Dynamic views for single world added. * Urls added to single world * Single world site template added * Custom tag added ( iso date to y, m, d) * Import clean up * Arguments clean up * Test table added * Search button connected to views. * Search button connected to views. * Pages/find character (#21) * Get character information from api * Changed icon * Html site, display information from api. * Views for character search added. * Adding comment to unused buttons * Custom tag for data edited. * Forms file added * Updated working features. * Pages/experience (#24) * Cleaning code * Cleaning code * Creating models for experience tracking * Collecting data from api, formating data from db for future df modification * collecting data changed * removed query * Filter latest data * Add id from db to latest data df * Select records that don't exist in db * Filter data and create name list * Check all names on tibia.com. If name was changed, if character is deleted. Creates list's for each option. * Insert all characters that are new and don't exist in DB and don't apear in deleted list or name change list. * Update name changes in character table. * Collect new id list from db and merge it to df * collect character data from day before * changing old names with new onews in df * Merge data for calculations * Filing up needed columns with calculated values for insert. * Generate world transfer df * Added coments for future steps * Changed naming, added working functions. Update db. Bulk data. Check what is missing in db. Check status of characters that dont exist in db. * Changed naming, added working functions. Update db. Bulk data. Check what is missing in db. Check status of characters that don't exist in db. * Experience tracking sites added. Views created. * Fixed long main page loading. * Select menu - added option * Models for record history. * Select menu - added blank option * Rollback changes * Getting daily records from each world. Insert into db. * Main page views changed to load immediately * Main page changed the best players to url * main function changed * Added top 5 in history * Added top of each world * Added top of each world and top 5 history to views. * Experience colored * Explore experience gain site added to sidebar * Explore experience gain site added * Explore experience gain views added. Filter form request and get data from db based on conditions * Explore experience gain urls added * Experience gain colored * Explore experience rewritten cuz of slow loading time and crashing. * Checkboxes checked based on filter request. * Explore html code formated * Explore views formatted. Queries changed for more clear view * Pages/world transfers (#25) * World transfers urls added * World transfers views created. * World transfers custom tags for displaying dict value and tibia coins amount added * World transfers html file added * Formatting file. * Pages/name changes (#26) * Name changes ulrs added. * Name changes custom tag for counting price added. * Name changes views added. * Name changes html site added. * Name changes html site added. * Various fixes (#29) * Get date putted into funciton * x * Adding news and news ticker to db refactored. Raw sql -> orm * Adding online players and boosted creatures to db refactored. Raw sql -> orm * Various fixes (#30) * Get date putted into funciton * x * Adding news and news ticker to db refactored. Raw sql -> orm * Adding online players and boosted creatures to db refactored. Raw sql -> orm * Adding requirements * Adding requirements * Pre deployment, adding missing files * Pre deployment, settings file added. * Pre deployment, requirements modified * Pre deployment, settings variables modified * Pre deployment, settings variables modified - test * Pre deployment, settings variables modified - test * Pre deployment, settings variables modified - test * Pre deployment, requirements - test * Pre deployment, settings - adjustment * Update django.yml * Update django.yml * Action runner test * Update django.yml * Update django.yml * Restarting server * Dotenv added to read .env files * All secret data moved to .env * Update visible folders. * Update first date update in views * Update data collecting by date. Bulk reduced to 1000 items per query. Disable all function call. * Test venv activation and updating files on host. * Uploading all static files to repository. * First deploy preparation (#35) * Restarting server * Dotenv added to read .env files * All secret data moved to .env * Update visible folders. * Update first date update in views * Update data collecting by date. Bulk reduced to 1000 items per query. Disable all function call. * Test venv activation and updating files on host. * Uploading all static files to repository. * Removing unneeded code * Pointing env file * Pointing env file location * Generated files for daily tasks to fill db * Gitactions configured properly * First deploy preparation - various fixes (#37) * Restarting server * Dotenv added to read .env files * All secret data moved to .env * Update visible folders. * Update first date update in views * Update data collecting by date. Bulk reduced to 1000 items per query. Disable all function call. * Test venv activation and updating files on host. * Uploading all static files to repository. * Removing unneeded code * Pointing env file * Pointing env file location * Generated files for daily tasks to fill db * Gitactions configured properly * File formatting * File formatting * Hide account login menu. * Prepared empty part for future develop * Fixed loading all static files. * Experience menu activated. * Custom views for error handler added * Custom views for error handler added. Reformatting views. * Reformatting file * Feature/database operations (#41) * Clear highscores from old records. * Copy only active players from day before. * Date upgraded for deploy * Date upgraded for deploy * Proper date value for insert * Added functions to copy only needed records to table and delete old ones. * Test * Charm points scrap added - test version. First db fill.

* File and directory creation * Adding proxy scrapper script. Result: list of elite proxy with port * Script edited. Deleted unneeded lines, deleted bs, function returns clear dataframe. * Saving to csv added * Delete duplicates script with update of list in file added * Exception FileNotFound added. Checking status code. * Result of proxy scrapper * Convert proxy to list. Files reorganization. * Header list. * ImportError fix csv file added to gitignore * Scrapping world's list from tibia.com * Scrapping character information from tibia.com. * Temporary data file * Highscores scrapper. Early version * Deleted proxy usage. Deleted unneeded code. * Deleted unneeded code. * Main page html code added. Footer changing font size. Added YouTube, Twitch buttons to nav bar * Fixing spacing, typo and content information for future work * Test news added * Database added. Creating models. * Test realase (#12) * File and directory creation * Adding proxy scrapper script. Result: list of elite proxy with port * Script edited. Deleted unneeded lines, deleted bs, function returns clear dataframe. * Saving to csv added * Delete duplicates script with update of list in file added * Exception FileNotFound added. Checking status code. * Result of proxy scrapper * Convert proxy to list. Files reorganization. * Header list. * ImportError fix csv file added to gitignore * Scrapping world's list from tibia.com * Scrapping character information from tibia.com. * Temporary data file * Highscores scrapper. Early version * Deleted proxy usage. Deleted unneeded code. * Deleted unneeded code. * Main page html code added. Footer changing font size. Added YouTube, Twitch buttons to nav bar * Fixing spacing, typo and content information for future work * Test news added * Database added. Creating models. * Pages/main page (#14) * Test commit * Init added to scripts * Gitignore updated * Models managed changed to true * Functions to get data from tibiadata API * Checking data from api and compare with database. If not exist, insert. * Views at main page for news ticker * Main page news ticker data loaded from database. * Added condition to selecting data for news ticker * Deleted unneeded code. * Adding condition to check if list is not empty. * Variable renamed * Getting data about boosted boss. * Boosted creature models created. * Adding boss to database. Replace function added * Display boosted boss. * Get boosted creature information * Adding boosted creature to database * Display information about boosted creature. * Fixed naming and spaces * Added getting news history * Html formating of tibia.com news. Inserting news into database. Some name changes. * Added column * Views for news * Added news on main page with testimonials * Pages/main page (#16) * Test commit * Init added to scripts * Gitignore updated * Models managed changed to true * Functions to get data from tibiadata API * Checking data from api and compare with database. If not exist, insert. * Views at main page for news ticker * Main page news ticker data loaded from database. * Added condition to selecting data for news ticker * Deleted unneeded code. * Adding condition to check if list is not empty. * Variable renamed * Getting data about boosted boss. * Boosted creature models created. * Adding boss to database. Replace function added * Display boosted boss. * Get boosted creature information * Adding boosted creature to database * Display information about boosted creature. * Fixed naming and spaces * Added getting news history * Html formating of tibia.com news. Inserting news into database. Some name changes. * Added column * Views for news * Added news on main page with testimonials * Names changed * Names changed * About page added. (#17) * .format changed to f * Typo fix * Collecting data from tibia.com about worlds, vocation, beprotection and categories with values used on tibia.com. (#18) Saved to json file. * Pages/worlds (#19) * Adding script to scrap world type values from tibia.com * Adding script to get worlds list and details from tibiadata api * Models, added auto increment * urls to worlds list added * Added loading settings from django * Added scripts for datatables to base * Added views for all worlds page * Added html page for all worlds * Orm added * Online history added, small changes in other models * creation date added * Charts deleted. Added loop for table creation with queryset data * Changed function name * Main() added for develop purpose * Charts added, fixed js inside django template with variables * Online history data for chart added. Creation history data for chart added. * JSON file with creation history added * Custom tag added. * Pages/single world (#20) * Url changed * Urls, added variable threw url * Dynamic views for single world added. * Urls added to single world * Single world site template added * Custom tag added ( iso date to y, m, d) * Import clean up * Arguments clean up * Test table added * Search button connected to views. * Search button connected to views. * Pages/find character (#21) * Get character information from api * Changed icon * Html site, display information from api. * Views for character search added. * Adding comment to unused buttons * Custom tag for data edited. * Forms file added * Updated working features. * New features (#22) * File and directory creation * Adding proxy scrapper script. Result: list of elite proxy with port * Script edited. Deleted unneeded lines, deleted bs, function returns clear dataframe. * Saving to csv added * Delete duplicates script with update of list in file added * Exception FileNotFound added. Checking status code. * Result of proxy scrapper * Convert proxy to list. Files reorganization. * Header list. * ImportError fix csv file added to gitignore * Scrapping world's list from tibia.com * Scrapping character information from tibia.com. * Temporary data file * Highscores scrapper. Early version * Deleted proxy usage. Deleted unneeded code. * Deleted unneeded code. * Main page html code added. Footer changing font size. Added YouTube, Twitch buttons to nav bar * Fixing spacing, typo and content information for future work * Test news added * Database added. Creating models. * Pages/main page (#14) * Test commit * Init added to scripts * Gitignore updated * Models managed changed to true * Functions to get data from tibiadata API * Checking data from api and compare with database. If not exist, insert. * Views at main page for news ticker * Main page news ticker data loaded from database. * Added condition to selecting data for news ticker * Deleted unneeded code. * Adding condition to check if list is not empty. * Variable renamed * Getting data about boosted boss. * Boosted creature models created. * Adding boss to database. Replace function added * Display boosted boss. * Get boosted creature information * Adding boosted creature to database * Display information about boosted creature. * Fixed naming and spaces * Added getting news history * Html formating of tibia.com news. Inserting news into database. Some name changes. * Added column * Views for news * Added news on main page with testimonials * Pages/main page (#16) * Test commit * Init added to scripts * Gitignore updated * Models managed changed to true * Functions to get data from tibiadata API * Checking data from api and compare with database. If not exist, insert. * Views at main page for news ticker * Main page news ticker data loaded from database. * Added condition to selecting data for news ticker * Deleted unneeded code. * Adding condition to check if list is not empty. * Variable renamed * Getting data about boosted boss. * Boosted creature models created. * Adding boss to database. Replace function added * Display boosted boss. * Get boosted creature information * Adding boosted creature to database * Display information about boosted creature. * Fixed naming and spaces * Added getting news history * Html formating of tibia.com news. Inserting news into database. Some name changes. * Added column * Views for news * Added news on main page with testimonials * Names changed * Names changed * About page added. (#17) * .format changed to f * Typo fix * Collecting data from tibia.com about worlds, vocation, beprotection and categories with values used on tibia.com. (#18) Saved to json file. * Pages/worlds (#19) * Adding script to scrap world type values from tibia.com * Adding script to get worlds list and details from tibiadata api * Models, added auto increment * urls to worlds list added * Added loading settings from django * Added scripts for datatables to base * Added views for all worlds page * Added html page for all worlds * Orm added * Online history added, small changes in other models * creation date added * Charts deleted. Added loop for table creation with queryset data * Changed function name * Main() added for develop purpose * Charts added, fixed js inside django template with variables * Online history data for chart added. Creation history data for chart added. * JSON file with creation history added * Custom tag added. * Pages/single world (#20) * Url changed * Urls, added variable threw url * Dynamic views for single world added. * Urls added to single world * Single world site template added * Custom tag added ( iso date to y, m, d) * Import clean up * Arguments clean up * Test table added * Search button connected to views. * Search button connected to views. * Pages/find character (#21) * Get character information from api * Changed icon * Html site, display information from api. * Views for character search added. * Adding comment to unused buttons * Custom tag for data edited. * Forms file added * Updated working features. * Pages/experience (#24) * Cleaning code * Cleaning code * Creating models for experience tracking * Collecting data from api, formating data from db for future df modification * collecting data changed * removed query * Filter latest data * Add id from db to latest data df * Select records that don't exist in db * Filter data and create name list * Check all names on tibia.com. If name was changed, if character is deleted. Creates list's for each option. * Insert all characters that are new and don't exist in DB and don't apear in deleted list or name change list. * Update name changes in character table. * Collect new id list from db and merge it to df * collect character data from day before * changing old names with new onews in df * Merge data for calculations * Filing up needed columns with calculated values for insert. * Generate world transfer df * Added coments for future steps * Changed naming, added working functions. Update db. Bulk data. Check what is missing in db. Check status of characters that dont exist in db. * Changed naming, added working functions. Update db. Bulk data. Check what is missing in db. Check status of characters that don't exist in db. * Experience tracking sites added. Views created. * Fixed long main page loading. * Select menu - added option * Models for record history. * Select menu - added blank option * Rollback changes * Getting daily records from each world. Insert into db. * Main page views changed to load immediately * Main page changed the best players to url * main function changed * Added top 5 in history * Added top of each world * Added top of each world and top 5 history to views. * Experience colored * Explore experience gain site added to sidebar * Explore experience gain site added * Explore experience gain views added. Filter form request and get data from db based on conditions * Explore experience gain urls added * Experience gain colored * Explore experience rewritten cuz of slow loading time and crashing. * Checkboxes checked based on filter request. * Explore html code formated * Explore views formatted. Queries changed for more clear view * Pages/world transfers (#25) * World transfers urls added * World transfers views created. * World transfers custom tags for displaying dict value and tibia coins amount added * World transfers html file added * Formatting file. * Pages/name changes (#26) * Name changes ulrs added. * Name changes custom tag for counting price added. * Name changes views added. * Name changes html site added. * Name changes html site added. * Various fixes (#29) * Get date putted into funciton * x * Adding news and news ticker to db refactored. Raw sql -> orm * Adding online players and boosted creatures to db refactored. Raw sql -> orm * Various fixes (#30) * Get date putted into funciton * x * Adding news and news ticker to db refactored. Raw sql -> orm * Adding online players and boosted creatures to db refactored. Raw sql -> orm * Adding requirements * Adding requirements * First deploy preparation (#32) * Pre deployment (#31) * File and directory creation * Adding proxy scrapper script. Result: list of elite proxy with port * Script edited. Deleted unneeded lines, deleted bs, function returns clear dataframe. * Saving to csv added * Delete duplicates script with update of list in file added * Exception FileNotFound added. Checking status code. * Result of proxy scrapper * Convert proxy to list. Files reorganization. * Header list. * ImportError fix csv file added to gitignore * Scrapping world's list from tibia.com * Scrapping character information from tibia.com. * Temporary data file * Highscores scrapper. Early version * Deleted proxy usage. Deleted unneeded code. * Deleted unneeded code. * Main page html code added. Footer changing font size. Added YouTube, Twitch buttons to nav bar * Fixing spacing, typo and content information for future work * Test news added * Database added. Creating models. * Test realase (#12) * File and directory creation * Adding proxy scrapper script. Result: list of elite proxy with port * Script edited. Deleted unneeded lines, deleted bs, function returns clear dataframe. * Saving to csv added * Delete duplicates script with update of list in file added * Exception FileNotFound added. Checking status code. * Result of proxy scrapper * Convert proxy to list. Files reorganization. * Header list. * ImportError fix csv file added to gitignore * Scrapping world's list from tibia.com * Scrapping character information from tibia.com. * Temporary data file * Highscores scrapper. Early version * Deleted proxy usage. Deleted unneeded code. * Deleted unneeded code. * Main page html code added. Footer changing font size. Added YouTube, Twitch buttons to nav bar * Fixing spacing, typo and content information for future work * Test news added * Database added. Creating models. * Pages/main page (#14) * Test commit * Init added to scripts * Gitignore updated * Models managed changed to true * Functions to get data from tibiadata API * Checking data from api and compare with database. If not exist, insert. * Views at main page for news ticker * Main page news ticker data loaded from database. * Added condition to selecting data for news ticker * Deleted unneeded code. * Adding condition to check if list is not empty. * Variable renamed * Getting data about boosted boss. * Boosted creature models created. * Adding boss to database. Replace function added * Display boosted boss. * Get boosted creature information * Adding boosted creature to database * Display information about boosted creature. * Fixed naming and spaces * Added getting news history * Html formating of tibia.com news. Inserting news into database. Some name changes. * Added column * Views for news * Added news on main page with testimonials * Pages/main page (#16) * Test commit * Init added to scripts * Gitignore updated * Models managed changed to true * Functions to get data from tibiadata API * Checking data from api and compare with database. If not exist, insert. * Views at main page for news ticker * Main page news ticker data loaded from database. * Added condition to selecting data for news ticker * Deleted unneeded code. * Adding condition to check if list is not empty. * Variable renamed * Getting data about boosted boss. * Boosted creature models created. * Adding boss to database. Replace function added * Display boosted boss. * Get boosted creature information * Adding boosted creature to database * Display information about boosted creature. * Fixed naming and spaces * Added getting news history * Html formating of tibia.com news. Inserting news into database. Some name changes. * Added column * Views for news * Added news on main page with testimonials * Names changed * Names changed * About page added. (#17) * .format changed to f * Typo fix * Collecting data from tibia.com about worlds, vocation, beprotection and categories with values used on tibia.com. (#18) Saved to json file. * Pages/worlds (#19) * Adding script to scrap world type values from tibia.com * Adding script to get worlds list and details from tibiadata api * Models, added auto increment * urls to worlds list added * Added loading settings from django * Added scripts for datatables to base * Added views for all worlds page * Added html page for all worlds * Orm added * Online history added, small changes in other models * creation date added * Charts deleted. Added loop for table creation with queryset data * Changed function name * Main() added for develop purpose * Charts added, fixed js inside django template with variables * Online history data for chart added. Creation history data for chart added. * JSON file with creation history added * Custom tag added. * Pages/single world (#20) * Url changed * Urls, added variable threw url * Dynamic views for single world added. * Urls added to single world * Single world site template added * Custom tag added ( iso date to y, m, d) * Import clean up * Arguments clean up * Test table added * Search button connected to views. * Search button connected to views. * Pages/find character (#21) * Get character information from api * Changed icon * Html site, display information from api. * Views for character search added. * Adding comment to unused buttons * Custom tag for data edited. * Forms file added * Updated working features. * New features (#22) * File and directory creation * Adding proxy scrapper script. Result: list of elite proxy with port * Script edited. Deleted unneeded lines, deleted bs, function returns clear dataframe. * Saving to csv added * Delete duplicates script with update of list in file added * Exception FileNotFound added. Checking status code. * Result of proxy scrapper * Convert proxy to list. Files reorganization. * Header list. * ImportError fix csv file added to gitignore * Scrapping world's list from tibia.com * Scrapping character information from tibia.com. * Temporary data file * Highscores scrapper. Early version * Deleted proxy usage. Deleted unneeded code. * Deleted unneeded code. * Main page html code added. Footer changing font size. Added YouTube, Twitch buttons to nav bar * Fixing spacing, typo and content information for future work * Test news added * Database added. Creating models. * Pages/main page (#14) * Test commit * Init added to scripts * Gitignore updated * Models managed changed to true * Functions to get data from tibiadata API * Checking data from api and compare with database. If not exist, insert. * Views at main page for news ticker * Main page news ticker data loaded from database. * Added condition to selecting data for news ticker * Deleted unneeded code. * Adding condition to check if list is not empty. * Variable renamed * Getting data about boosted boss. * Boosted creature models created. * Adding boss to database. Replace function added * Display boosted boss. * Get boosted creature information * Adding boosted creature to database * Display information about boosted creature. * Fixed naming and spaces * Added getting news history * Html formating of tibia.com news. Inserting news into database. Some name changes. * Added column * Views for news * Added news on main page with testimonials * Pages/main page (#16) * Test commit * Init added to scripts * Gitignore updated * Models managed changed to true * Functions to get data from tibiadata API * Checking data from api and compare with database. If not exist, insert. * Views at main page for news ticker * Main page news ticker data loaded from database. * Added condition to selecting data for news ticker * Deleted unneeded code. * Adding condition to check if list is not empty. * Variable renamed * Getting data about boosted boss. * Boosted creature models created. * Adding boss to database. Replace function added * Display boosted boss. * Get boosted creature information * Adding boosted creature to database * Display information about boosted creature. * Fixed naming and spaces * Added getting news history * Html formating of tibia.com news. Inserting news into database. Some name changes. * Added column * Views for news * Added news on main page with testimonials * Names changed * Names changed * About page added. (#17) * .format changed to f * Typo fix * Collecting data from tibia.com about worlds, vocation, beprotection and categories with values used on tibia.com. (#18) Saved to json file. * Pages/worlds (#19) * Adding script to scrap world type values from tibia.com * Adding script to get worlds list and details from tibiadata api * Models, added auto increment * urls to worlds list added * Added loading settings from django * Added scripts for datatables to base * Added views for all worlds page * Added html page for all worlds * Orm added * Online history added, small changes in other models * creation date added * Charts deleted. Added loop for table creation with queryset data * Changed function name * Main() added for develop purpose * Charts added, fixed js inside django template with variables * Online history data for chart added. Creation history data for chart added. * JSON file with creation history added * Custom tag added. * Pages/single world (#20) * Url changed * Urls, added variable threw url * Dynamic views for single world added. * Urls added to single world * Single world site template added * Custom tag added ( iso date to y, m, d) * Import clean up * Arguments clean up * Test table added * Search button connected to views. * Search button connected to views. * Pages/find character (#21) * Get character information from api * Changed icon * Html site, display information from api. * Views for character search added. * Adding comment to unused buttons * Custom tag for data edited. * Forms file added * Updated working features. * Pages/experience (#24) * Cleaning code * Cleaning code * Creating models for experience tracking * Collecting data from api, formating data from db for future df modification * collecting data changed * removed query * Filter latest data * Add id from db to latest data df * Select records that don't exist in db * Filter data and create name list * Check all names on tibia.com. If name was changed, if character is deleted. Creates list's for each option. * Insert all characters that are new and don't exist in DB and don't apear in deleted list or name change list. * Update name changes in character table. * Collect new id list from db and merge it to df * collect character data from day before * changing old names with new onews in df * Merge data for calculations * Filing up needed columns with calculated values for insert. * Generate world transfer df * Added coments for future steps * Changed naming, added working functions. Update db. Bulk data. Check what is missing in db. Check status of characters that dont exist in db. * Changed naming, added working functions. Update db. Bulk data. Check what is missing in db. Check status of characters that don't exist in db. * Experience tracking sites added. Views created. * Fixed long main page loading. * Select menu - added option * Models for record history. * Select menu - added blank option * Rollback changes * Getting daily records from each world. Insert into db. * Main page views changed to load immediately * Main page changed the best players to url * main function changed * Added top 5 in history * Added top of each world * Added top of each world and top 5 history to views. * Experience colored * Explore experience gain site added to sidebar * Explore experience gain site added * Explore experience gain views added. Filter form request and get data from db based on conditions * Explore experience gain urls added * Experience gain colored * Explore experience rewritten cuz of slow loading time and crashing. * Checkboxes checked based on filter request. * Explore html code formated * Explore views formatted. Queries changed for more clear view * Pages/world transfers (#25) * World transfers urls added * World transfers views created. * World transfers custom tags for displaying dict value and tibia coins amount added * World transfers html file added * Formatting file. * Pages/name changes (#26) * Name changes ulrs added. * Name changes custom tag for counting price added. * Name changes views added. * Name changes html site added. * Name changes html site added. * Various fixes (#29) * Get date putted into funciton * x * Adding news and news ticker to db refactored. Raw sql -> orm * Adding online players and boosted creatures to db refactored. Raw sql -> orm * Various fixes (#30) * Get date putted into funciton * x * Adding news and news ticker to db refactored. Raw sql -> orm * Adding online players and boosted creatures to db refactored. Raw sql -> orm * Adding requirements * Adding requirements * Pre deployment, adding missing files * Pre deployment, settings file added. * Pre deployment, requirements modified * Pre deployment, settings variables modified * Pre deployment, settings variables modified - test * Pre deployment, settings variables modified - test * Pre deployment, settings variables modified - test * Pre deployment, requirements - test * Pre deployment, settings - adjustment * Update django.yml * Update django.yml * Action runner test * Update django.yml * Update django.yml * Restarting server * Dotenv added to read .env files * All secret data moved to .env * Update visible folders. * Update first date update in views * Update data collecting by date. Bulk reduced to 1000 items per query. Disable all function call. * Test venv activation and updating files on host. * Uploading all static files to repository. * First deploy preparation (#35) * Restarting server * Dotenv added to read .env files * All secret data moved to .env * Update visible folders. * Update first date update in views * Update data collecting by date. Bulk reduced to 1000 items per query. Disable all function call. * Test venv activation and updating files on host. * Uploading all static files to repository. * Removing unneeded code * Pointing env file * Pointing env file location * Generated files for daily tasks to fill db * Gitactions configured properly * First deploy preparation - various fixes (#37) * Restarting server * Dotenv added to read .env files * All secret data moved to .env * Update visible folders. * Update first date update in views * Update data collecting by date. Bulk reduced to 1000 items per query. Disable all function call. * Test venv activation and updating files on host. * Uploading all static files to repository. * Removing unneeded code * Pointing env file * Pointing env file location * Generated files for daily tasks to fill db * Gitactions configured properly * File formatting * File formatting * Hide account login menu. * Prepared empty part for future develop * Fixed loading all static files. * Experience menu activated. * Custom views for error handler added * Custom views for error handler added. Reformatting views. * Reformatting file * Feature/database operations (#41) * Clear highscores from old records. * Copy only active players from day before. * Date upgraded for deploy * Date upgraded for deploy * Proper date value for insert * Added functions to copy only needed records to table and delete old ones. * Test * Charm points scrap added - test version. First db fill. * Logging added (#43) * Clear highscores from old records. * Copy only active players from day before. * Date upgraded for deploy * Date upgraded for deploy * Proper date value for insert * Added functions to copy only needed records to table and delete old ones. * Test * Charm points scrap added - test version. First db fill. * Basic sentry logging added. High bulk adjustments for vps. Some styling. * Basic sentry logging added. * Basic sentry logging added.

* File and directory creation * Adding proxy scrapper script. Result: list of elite proxy with port * Script edited. Deleted unneeded lines, deleted bs, function returns clear dataframe. * Saving to csv added * Delete duplicates script with update of list in file added * Exception FileNotFound added. Checking status code. * Result of proxy scrapper * Convert proxy to list. Files reorganization. * Header list. * ImportError fix csv file added to gitignore * Scrapping world's list from tibia.com * Scrapping character information from tibia.com. * Temporary data file * Highscores scrapper. Early version * Deleted proxy usage. Deleted unneeded code. * Deleted unneeded code. * Main page html code added. Footer changing font size. Added YouTube, Twitch buttons to nav bar * Fixing spacing, typo and content information for future work * Test news added * Database added. Creating models. * Test realase (#12) * File and directory creation * Adding proxy scrapper script. Result: list of elite proxy with port * Script edited. Deleted unneeded lines, deleted bs, function returns clear dataframe. * Saving to csv added * Delete duplicates script with update of list in file added * Exception FileNotFound added. Checking status code. * Result of proxy scrapper * Convert proxy to list. Files reorganization. * Header list. * ImportError fix csv file added to gitignore * Scrapping world's list from tibia.com * Scrapping character information from tibia.com. * Temporary data file * Highscores scrapper. Early version * Deleted proxy usage. Deleted unneeded code. * Deleted unneeded code. * Main page html code added. Footer changing font size. Added YouTube, Twitch buttons to nav bar * Fixing spacing, typo and content information for future work * Test news added * Database added. Creating models. * Pages/main page (#14) * Test commit * Init added to scripts * Gitignore updated * Models managed changed to true * Functions to get data from tibiadata API * Checking data from api and compare with database. If not exist, insert. * Views at main page for news ticker * Main page news ticker data loaded from database. * Added condition to selecting data for news ticker * Deleted unneeded code. * Adding condition to check if list is not empty. * Variable renamed * Getting data about boosted boss. * Boosted creature models created. * Adding boss to database. Replace function added * Display boosted boss. * Get boosted creature information * Adding boosted creature to database * Display information about boosted creature. * Fixed naming and spaces * Added getting news history * Html formating of tibia.com news. Inserting news into database. Some name changes. * Added column * Views for news * Added news on main page with testimonials * Pages/main page (#16) * Test commit * Init added to scripts * Gitignore updated * Models managed changed to true * Functions to get data from tibiadata API * Checking data from api and compare with database. If not exist, insert. * Views at main page for news ticker * Main page news ticker data loaded from database. * Added condition to selecting data for news ticker * Deleted unneeded code. * Adding condition to check if list is not empty. * Variable renamed * Getting data about boosted boss. * Boosted creature models created. * Adding boss to database. Replace function added * Display boosted boss. * Get boosted creature information * Adding boosted creature to database * Display information about boosted creature. * Fixed naming and spaces * Added getting news history * Html formating of tibia.com news. Inserting news into database. Some name changes. * Added column * Views for news * Added news on main page with testimonials * Names changed * Names changed * About page added. (#17) * .format changed to f * Typo fix * Collecting data from tibia.com about worlds, vocation, beprotection and categories with values used on tibia.com. (#18) Saved to json file. * Pages/worlds (#19) * Adding script to scrap world type values from tibia.com * Adding script to get worlds list and details from tibiadata api * Models, added auto increment * urls to worlds list added * Added loading settings from django * Added scripts for datatables to base * Added views for all worlds page * Added html page for all worlds * Orm added * Online history added, small changes in other models * creation date added * Charts deleted. Added loop for table creation with queryset data * Changed function name * Main() added for develop purpose * Charts added, fixed js inside django template with variables * Online history data for chart added. Creation history data for chart added. * JSON file with creation history added * Custom tag added. * Pages/single world (#20) * Url changed * Urls, added variable threw url * Dynamic views for single world added. * Urls added to single world * Single world site template added * Custom tag added ( iso date to y, m, d) * Import clean up * Arguments clean up * Test table added * Search button connected to views. * Search button connected to views. * Pages/find character (#21) * Get character information from api * Changed icon * Html site, display information from api. * Views for character search added. * Adding comment to unused buttons * Custom tag for data edited. * Forms file added * Updated working features. * New features (#22) * File and directory creation * Adding proxy scrapper script. Result: list of elite proxy with port * Script edited. Deleted unneeded lines, deleted bs, function returns clear dataframe. * Saving to csv added * Delete duplicates script with update of list in file added * Exception FileNotFound added. Checking status code. * Result of proxy scrapper * Convert proxy to list. Files reorganization. * Header list. * ImportError fix csv file added to gitignore * Scrapping world's list from tibia.com * Scrapping character information from tibia.com. * Temporary data file * Highscores scrapper. Early version * Deleted proxy usage. Deleted unneeded code. * Deleted unneeded code. * Main page html code added. Footer changing font size. Added YouTube, Twitch buttons to nav bar * Fixing spacing, typo and content information for future work * Test news added * Database added. Creating models. * Pages/main page (#14) * Test commit * Init added to scripts * Gitignore updated * Models managed changed to true * Functions to get data from tibiadata API * Checking data from api and compare with database. If not exist, insert. * Views at main page for news ticker * Main page news ticker data loaded from database. * Added condition to selecting data for news ticker * Deleted unneeded code. * Adding condition to check if list is not empty. * Variable renamed * Getting data about boosted boss. * Boosted creature models created. * Adding boss to database. Replace function added * Display boosted boss. * Get boosted creature information * Adding boosted creature to database * Display information about boosted creature. * Fixed naming and spaces * Added getting news history * Html formating of tibia.com news. Inserting news into database. Some name changes. * Added column * Views for news * Added news on main page with testimonials * Pages/main page (#16) * Test commit * Init added to scripts * Gitignore updated * Models managed changed to true * Functions to get data from tibiadata API * Checking data from api and compare with database. If not exist, insert. * Views at main page for news ticker * Main page news ticker data loaded from database. * Added condition to selecting data for news ticker * Deleted unneeded code. * Adding condition to check if list is not empty. * Variable renamed * Getting data about boosted boss. * Boosted creature models created. * Adding boss to database. Replace function added * Display boosted boss. * Get boosted creature information * Adding boosted creature to database * Display information about boosted creature. * Fixed naming and spaces * Added getting news history * Html formating of tibia.com news. Inserting news into database. Some name changes. * Added column * Views for news * Added news on main page with testimonials * Names changed * Names changed * About page added. (#17) * .format changed to f * Typo fix * Collecting data from tibia.com about worlds, vocation, beprotection and categories with values used on tibia.com. (#18) Saved to json file. * Pages/worlds (#19) * Adding script to scrap world type values from tibia.com * Adding script to get worlds list and details from tibiadata api * Models, added auto increment * urls to worlds list added * Added loading settings from django * Added scripts for datatables to base * Added views for all worlds page * Added html page for all worlds * Orm added * Online history added, small changes in other models * creation date added * Charts deleted. Added loop for table creation with queryset data * Changed function name * Main() added for develop purpose * Charts added, fixed js inside django template with variables * Online history data for chart added. Creation history data for chart added. * JSON file with creation history added * Custom tag added. * Pages/single world (#20) * Url changed * Urls, added variable threw url * Dynamic views for single world added. * Urls added to single world * Single world site template added * Custom tag added ( iso date to y, m, d) * Import clean up * Arguments clean up * Test table added * Search button connected to views. * Search button connected to views. * Pages/find character (#21) * Get character information from api * Changed icon * Html site, display information from api. * Views for character search added. * Adding comment to unused buttons * Custom tag for data edited. * Forms file added * Updated working features. * Pages/experience (#24) * Cleaning code * Cleaning code * Creating models for experience tracking * Collecting data from api, formating data from db for future df modification * collecting data changed * removed query * Filter latest data * Add id from db to latest data df * Select records that don't exist in db * Filter data and create name list * Check all names on tibia.com. If name was changed, if character is deleted. Creates list's for each option. * Insert all characters that are new and don't exist in DB and don't apear in deleted list or name change list. * Update name changes in character table. * Collect new id list from db and merge it to df * collect character data from day before * changing old names with new onews in df * Merge data for calculations * Filing up needed columns with calculated values for insert. * Generate world transfer df * Added coments for future steps * Changed naming, added working functions. Update db. Bulk data. Check what is missing in db. Check status of characters that dont exist in db. * Changed naming, added working functions. Update db. Bulk data. Check what is missing in db. Check status of characters that don't exist in db. * Experience tracking sites added. Views created. * Fixed long main page loading. * Select menu - added option * Models for record history. * Select menu - added blank option * Rollback changes * Getting daily records from each world. Insert into db. * Main page views changed to load immediately * Main page changed the best players to url * main function changed * Added top 5 in history * Added top of each world * Added top of each world and top 5 history to views. * Experience colored * Explore experience gain site added to sidebar * Explore experience gain site added * Explore experience gain views added. Filter form request and get data from db based on conditions * Explore experience gain urls added * Experience gain colored * Explore experience rewritten cuz of slow loading time and crashing. * Checkboxes checked based on filter request. * Explore html code formated * Explore views formatted. Queries changed for more clear view * Pages/world transfers (#25) * World transfers urls added * World transfers views created. * World transfers custom tags for displaying dict value and tibia coins amount added * World transfers html file added * Formatting file. * Pages/name changes (#26) * Name changes ulrs added. * Name changes custom tag for counting price added. * Name changes views added. * Name changes html site added. * Name changes html site added. * Various fixes (#29) * Get date putted into funciton * x * Adding news and news ticker to db refactored. Raw sql -> orm * Adding online players and boosted creatures to db refactored. Raw sql -> orm * Various fixes (#30) * Get date putted into funciton * x * Adding news and news ticker to db refactored. Raw sql -> orm * Adding online players and boosted creatures to db refactored. Raw sql -> orm * Adding requirements * Adding requirements * First deploy preparation (#32) * Pre deployment (#31) * File and directory creation * Adding proxy scrapper script. Result: list of elite proxy with port * Script edited. Deleted unneeded lines, deleted bs, function returns clear dataframe. * Saving to csv added * Delete duplicates script with update of list in file added * Exception FileNotFound added. Checking status code. * Result of proxy scrapper * Convert proxy to list. Files reorganization. * Header list. * ImportError fix csv file added to gitignore * Scrapping world's list from tibia.com * Scrapping character information from tibia.com. * Temporary data file * Highscores scrapper. Early version * Deleted proxy usage. Deleted unneeded code. * Deleted unneeded code. * Main page html code added. Footer changing font size. Added YouTube, Twitch buttons to nav bar * Fixing spacing, typo and content information for future work * Test news added * Database added. Creating models. * Test realase (#12) * File and directory creation * Adding proxy scrapper script. Result: list of elite proxy with port * Script edited. Deleted unneeded lines, deleted bs, function returns clear dataframe. * Saving to csv added * Delete duplicates script with update of list in file added * Exception FileNotFound added. Checking status code. * Result of proxy scrapper * Convert proxy to list. Files reorganization. * Header list. * ImportError fix csv file added to gitignore * Scrapping world's list from tibia.com * Scrapping character information from tibia.com. * Temporary data file * Highscores scrapper. Early version * Deleted proxy usage. Deleted unneeded code. * Deleted unneeded code. * Main page html code added. Footer changing font size. Added YouTube, Twitch buttons to nav bar * Fixing spacing, typo and content information for future work * Test news added * Database added. Creating models. * Pages/main page (#14) * Test commit * Init added to scripts * Gitignore updated * Models managed changed to true * Functions to get data from tibiadata API * Checking data from api and compare with database. If not exist, insert. * Views at main page for news ticker * Main page news ticker data loaded from database. * Added condition to selecting data for news ticker * Deleted unneeded code. * Adding condition to check if list is not empty. * Variable renamed * Getting data about boosted boss. * Boosted creature models created. * Adding boss to database. Replace function added * Display boosted boss. * Get boosted creature information * Adding boosted creature to database * Display information about boosted creature. * Fixed naming and spaces * Added getting news history * Html formating of tibia.com news. Inserting news into database. Some name changes. * Added column * Views for news * Added news on main page with testimonials * Pages/main page (#16) * Test commit * Init added to scripts * Gitignore updated * Models managed changed to true * Functions to get data from tibiadata API * Checking data from api and compare with database. If not exist, insert. * Views at main page for news ticker * Main page news ticker data loaded from database. * Added condition to selecting data for news ticker * Deleted unneeded code. * Adding condition to check if list is not empty. * Variable renamed * Getting data about boosted boss. * Boosted creature models created. * Adding boss to database. Replace function added * Display boosted boss. * Get boosted creature information * Adding boosted creature to database * Display information about boosted creature. * Fixed naming and spaces * Added getting news history * Html formating of tibia.com news. Inserting news into database. Some name changes. * Added column * Views for news * Added news on main page with testimonials * Names changed * Names changed * About page added. (#17) * .format changed to f * Typo fix * Collecting data from tibia.com about worlds, vocation, beprotection and categories with values used on tibia.com. (#18) Saved to json file. * Pages/worlds (#19) * Adding script to scrap world type values from tibia.com * Adding script to get worlds list and details from tibiadata api * Models, added auto increment * urls to worlds list added * Added loading settings from django * Added scripts for datatables to base * Added views for all worlds page * Added html page for all worlds * Orm added * Online history added, small changes in other models * creation date added * Charts deleted. Added loop for table creation with queryset data * Changed function name * Main() added for develop purpose * Charts added, fixed js inside django template with variables * Online history data for chart added. Creation history data for chart added. * JSON file with creation history added * Custom tag added. * Pages/single world (#20) * Url changed * Urls, added variable threw url * Dynamic views for single world added. * Urls added to single world * Single world site template added * Custom tag added ( iso date to y, m, d) * Import clean up * Arguments clean up * Test table added * Search button connected to views. * Search button connected to views. * Pages/find character (#21) * Get character information from api * Changed icon * Html site, display information from api. * Views for character search added. * Adding comment to unused buttons * Custom tag for data edited. * Forms file added * Updated working features. * New features (#22) * File and directory creation * Adding proxy scrapper script. Result: list of elite proxy with port * Script edited. Deleted unneeded lines, deleted bs, function returns clear dataframe. * Saving to csv added * Delete duplicates script with update of list in file added * Exception FileNotFound added. Checking status code. * Result of proxy scrapper * Convert proxy to list. Files reorganization. * Header list. * ImportError fix csv file added to gitignore * Scrapping world's list from tibia.com * Scrapping character information from tibia.com. * Temporary data file * Highscores scrapper. Early version * Deleted proxy usage. Deleted unneeded code. * Deleted unneeded code. * Main page html code added. Footer changing font size. Added YouTube, Twitch buttons to nav bar * Fixing spacing, typo and content information for future work * Test news added * Database added. Creating models. * Pages/main page (#14) * Test commit * Init added to scripts * Gitignore updated * Models managed changed to true * Functions to get data from tibiadata API * Checking data from api and compare with database. If not exist, insert. * Views at main page for news ticker * Main page news ticker data loaded from database. * Added condition to selecting data for news ticker * Deleted unneeded code. * Adding condition to check if list is not empty. * Variable renamed * Getting data about boosted boss. * Boosted creature models created. * Adding boss to database. Replace function added * Display boosted boss. * Get boosted creature information * Adding boosted creature to database * Display information about boosted creature. * Fixed naming and spaces * Added getting news history * Html formating of tibia.com news. Inserting news into database. Some name changes. * Added column * Views for news * Added news on main page with testimonials * Pages/main page (#16) * Test commit * Init added to scripts * Gitignore updated * Models managed changed to true * Functions to get data from tibiadata API * Checking data from api and compare with database. If not exist, insert. * Views at main page for news ticker * Main page news ticker data loaded from database. * Added condition to selecting data for news ticker * Deleted unneeded code. * Adding condition to check if list is not empty. * Variable renamed * Getting data about boosted boss. * Boosted creature models created. * Adding boss to database. Replace function added * Display boosted boss. * Get boosted creature information * Adding boosted creature to database * Display information about boosted creature. * Fixed naming and spaces * Added getting news history * Html formating of tibia.com news. Inserting news into database. Some name changes. * Added column * Views for news * Added news on main page with testimonials * Names changed * Names changed * About page added. (#17) * .format changed to f * Typo fix * Collecting data from tibia.com about worlds, vocation, beprotection and categories with values used on tibia.com. (#18) Saved to json file. * Pages/worlds (#19) * Adding script to scrap world type values from tibia.com * Adding script to get worlds list and details from tibiadata api * Models, added auto increment * urls to worlds list added * Added loading settings from django * Added scripts for datatables to base * Added views for all worlds page * Added html page for all worlds * Orm added * Online history added, small changes in other models * creation date added * Charts deleted. Added loop for table creation with queryset data * Changed function name * Main() added for develop purpose * Charts added, fixed js inside django template with variables * Online history data for chart added. Creation history data for chart added. * JSON file with creation history added * Custom tag added. * Pages/single world (#20) * Url changed * Urls, added variable threw url * Dynamic views for single world added. * Urls added to single world * Single world site template added * Custom tag added ( iso date to y, m, d) * Import clean up * Arguments clean up * Test table added * Search button connected to views. * Search button connected to views. * Pages/find character (#21) * Get character information from api * Changed icon * Html site, display information from api. * Views for character search added. * Adding comment to unused buttons * Custom tag for data edited. * Forms file added * Updated working features. * Pages/experience (#24) * Cleaning code * Cleaning code * Creating models for experience tracking * Collecting data from api, formating data from db for future df modification * collecting data changed * removed query * Filter latest data * Add id from db to latest data df * Select records that don't exist in db * Filter data and create name list * Check all names on tibia.com. If name was changed, if character is deleted. Creates list's for each option. * Insert all characters that are new and don't exist in DB and don't apear in deleted list or name change list. * Update name changes in character table. * Collect new id list from db and merge it to df * collect character data from day before * changing old names with new onews in df * Merge data for calculations * Filing up needed columns with calculated values for insert. * Generate world transfer df * Added coments for future steps * Changed naming, added working functions. Update db. Bulk data. Check what is missing in db. Check status of characters that dont exist in db. * Changed naming, added working functions. Update db. Bulk data. Check what is missing in db. Check status of characters that don't exist in db. * Experience tracking sites added. Views created. * Fixed long main page loading. * Select menu - added option * Models for record history. * Select menu - added blank option * Rollback changes * Getting daily records from each world. Insert into db. * Main page views changed to load immediately * Main page changed the best players to url * main function changed * Added top 5 in history * Added top of each world * Added top of each world and top 5 history to views. * Experience colored * Explore experience gain site added to sidebar * Explore experience gain site added * Explore experience gain views added. Filter form request and get data from db based on conditions * Explore experience gain urls added * Experience gain colored * Explore experience rewritten cuz of slow loading time and crashing. * Checkboxes checked based on filter request. * Explore html code formated * Explore views formatted. Queries changed for more clear view * Pages/world transfers (#25) * World transfers urls added * World transfers views created. * World transfers custom tags for displaying dict value and tibia coins amount added * World transfers html file added * Formatting file. * Pages/name changes (#26) * Name changes ulrs added. * Name changes custom tag for counting price added. * Name changes views added. * Name changes html site added. * Name changes html site added. * Various fixes (#29) * Get date putted into funciton * x * Adding news and news ticker to db refactored. Raw sql -> orm * Adding online players and boosted creatures to db refactored. Raw sql -> orm * Various fixes (#30) * Get date putted into funciton * x * Adding news and news ticker to db refactored. Raw sql -> orm * Adding online players and boosted creatures to db refactored. Raw sql -> orm * Adding requirements * Adding requirements * Pre deployment, adding missing files * Pre deployment, settings file added. * Pre deployment, requirements modified * Pre deployment, settings variables modified * Pre deployment, settings variables modified - test * Pre deployment, settings variables modified - test * Pre deployment, settings variables modified - test * Pre deployment, requirements - test * Pre deployment, settings - adjustment * Update django.yml * Update django.yml * Action runner test * Update django.yml * Update django.yml * Restarting server * Dotenv added to read .env files * All secret data moved to .env * Update visible folders. * Update first date update in views * Update data collecting by date. Bulk reduced to 1000 items per query. Disable all function call. * Test venv activation and updating files on host. * Uploading all static files to repository. * First deploy preparation (#35) * Restarting server * Dotenv added to read .env files * All secret data moved to .env * Update visible folders. * Update first date update in views * Update data collecting by date. Bulk reduced to 1000 items per query. Disable all function call. * Test venv activation and updating files on host. * Uploading all static files to repository. * Removing unneeded code * Pointing env file * Pointing env file location * Generated files for daily tasks to fill db * Gitactions configured properly * First deploy preparation - various fixes (#37) * Restarting server * Dotenv added to read .env files * All secret data moved to .env * Update visible folders. * Update first date update in views * Update data collecting by date. Bulk reduced to 1000 items per query. Disable all function call. * Test venv activation and updating files on host. * Uploading all static files to repository. * Removing unneeded code * Pointing env file * Pointing env file location * Generated files for daily tasks to fill db * Gitactions configured properly * File formatting * File formatting * Hide account login menu. * Prepared empty part for future develop * Fixed loading all static files. * Experience menu activated. * Custom views for error handler added * Custom views for error handler added. Reformatting views. * Reformatting file * Feature/database operations (#41) * Clear highscores from old records. * Copy only active players from day before. * Date upgraded for deploy * Date upgraded for deploy * Proper date value for insert * Added functions to copy only needed records to table and delete old ones. * Test * Charm points scrap added - test version. First db fill. * Logging added (#43) * Clear highscores from old records. * Copy only active players from day before. * Date upgraded for deploy * Date upgraded for deploy * Proper date value for insert * Added functions to copy only needed records to table and delete old ones. * Test * Charm points scrap added - test version. First db fill. * Basic sentry logging added. High bulk adjustments for vps. Some styling. * Basic sentry logging added. * Basic sentry logging added. * Update requirements.txt * Deploy (#45) * temp directory added to gitignore * Sentry added to req Psutil added to req * Task add_exp_highscores expanded. Checking status of steps in highscore task. Added ability to re-run script from last finished part. Removing files from temp directory. * Main script of collecting and inserting data from highscores re-written. Split into smaller parts to provide status information to task manager. Saving data to file as backup in case of crashed process. * Task process checker for longest script. * First changes of layout. * Tasks model added. * Adjusted checker. * Cleaned for deploy * Adding highscores prepared for deploy. Memory usage lowered. GC added. * Logs folder added to ignore * File formatted. Fixed get_daily_records function. Fixed move active players function. Clean db adjusted. * Removed timedelta * Logging information added * Logging msg changed. * Date fixed, Functions now can take arguments, task satus updated in all functions. * Delete files function stopped for future develop * Custom tag eddited to display big number in specific way * World transfer limit for display added. Name changes limit for display added. Added Charms views. * Layout modificatons. (colors, buttons, forms etc) * Charms urls added

jakubg89 added 8 commits January 20, 2023 11:49

Clear highscores from old records.

c9c8d30

Copy only active players from day before.

307375f

Date upgraded for deploy

016960f

Date upgraded for deploy

21dede6

Proper date value for insert

3d6e6b6

Added functions to copy only needed records to table and delete old o…

244b52b

…nes.

Test

66a1d23

Charm points scrap added - test version.

a991b18

First db fill.

jakubg89 merged commit 04acce3 into develop Jan 20, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feature/database operations #41

Feature/database operations #41

jakubg89 commented Jan 20, 2023

Feature/database operations #41

Feature/database operations #41

Conversation

jakubg89 commented Jan 20, 2023