Use requests, bug fix and additional user parameters

1. HTTP Library Migration: urllib → requests

• Replaced urllib.request with requests library for HTTP operations
• Benefits:
  • Built-in automatic retry mechanism with exponential backoff
  • Better handling of large HTTP responses (fixes IncompleteRead errors)
  • Cleaner, more Pythonic API (e.g., response.json() vs manual decoding)
  • Session-based connection pooling for improved performance

2. SSL Certificate Verification

• Replaced hardcoded verify=False with intelligent verification logic
• Features:
  • Verification enabled by default (secure by default)
  • Optional custom CA bundle path support
  • Explicit "ignore verification" flag only when requested
  • Automatic suppression of InsecureRequestWarning when verification is disabled

3. Improved Error Handling

• Fixed get_all_the_layers() function logic to properly distinguish between success and error responses
• Added response.raise_for_status() for explicit HTTP error detection
• Better error messages with included error details from API responses

4. Code Modernization

• Converted all .format() string operations to f-strings for improved readability

5. Automatic Workspace Creation

• Script now automatically creates output workspace (folder or fgdb) if it doesn't exist
• Supports:
  • File geodatabase creation (.gdb)
  • Folder creation
  • Error handling for SDE connections

6. Control overwrite

• Script now allows user to overwrite the default overwrite behavior. If unchecked it will skip if the output featureclass already exists.

7. Toolbox Parameter Updates

• Added new optional parameters:
  • ignore_ssl_verification (bool, default: False)
  • ca_bundle_path (string, default: empty)
  • overwrite_output (bool, default: True)

Dependency Changes

• Added: requests library
• Added: urllib3.exceptions.InsecureRequestWarning handling
• Retained: urllib.parse (for URL manipulation, not HTTP)
• Removed: ssl module (replaced by requests/urllib3 SSL handling)

Breaking Changes

None - fully backward compatible. Default behavior is more secure than previous versions.

Known Issues Fixed

• http.client.IncompleteRead errors when downloading large datasets
• SSL verification warnings in ArcGIS Pro
• Error handling in service layer discovery

SSL error fix

Skip SSL Certification to resolve issue with Python 3.6/3.7 in ArcGIS Pro with some installs. Tool reports an error with [SSL_CERTIFICATE_VERFY_FAILED]. Issue is related to lacking SSL CA Cert on the machine. Until a permanent fix is established, this release skips SSL cert verification.

Important Pro fix and add Desktop folder

Important fix to imports for ArcGIS Pro urllib, add folder of older dekstop software for easier transition, remove redundant items

ArcGIS Pro version

ArcGIS Pro version of the Data Pillager, use the ArcGIS Pro version toolbox. Breaking change, will not run in ArcGIS Desktop. Does not create a layer file in this release.

Last Desktop update

Last ArcGIS Desktop update. Compatible with ArcGIS Desktop 10.3 and up. The main branch will switch to ArcGIS Pro and Python 3 from this point on.

Add table handling

Adds ability to handle a table service as well as feature class. Also a nicer check for existence of drawingInfo - will skip creating a layer file if this doesn't exist rather than throwing an error

Additional functionality 10.3

Added query parameter, added ability to walk folders and multiple layers from single entry point, updated the authentication approach to fix bug with non standard server name, and a general refactoring of the code.

ArcGIS 10.3 and up compatible

Added ability to write out a layer file from the service description (layer file matches service symbology). This functionality is only available in ArcGIS 10.3 onwards, so the 10.1 toolbox was removed. If you need 10.1 compatibility please see the v1.0 release

ArcGIS 10.1 and up compatible

This is the first "release", compatible with ArcGIS 10.1 and up.