Mansi_Singh_DataEngineer.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "authorship_tag": "ABX9TyPgwm0Zj821CqmOh+B95ghP",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/singhmansi25/Mansi-Singh_-Data-Engineer/blob/main/Mansi_Singh_DataEngineer.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Brief:\n",
        "\n",
        "1. The requirement needs to be developed in Python 3\n",
        "2. Code should follow pep8 standards and should include pydoc, logging & unit tests\n",
        "3. Please provide github link for review.\n",
        "\n",
        "## Requirement:\n",
        "\n",
        "1. Download the xml from this link\n",
        "2. From the xml, please parse through to the first download link whose file_type is DLTINS and download the zip\n",
        "3. Extract the xml from the zip.\n",
        "4. Convert the contents of the xml into a CSV with the following header:\n",
        " - FinInstrmGnlAttrbts.Id\n",
        " - FinInstrmGnlAttrbts.FullNm\n",
        " - FinInstrmGnlAttrbts.ClssfctnTp\n",
        " - FinInstrmGnlAttrbts.CmmdtyDerivInd\n",
        " - FinInstrmGnlAttrbts.NtnlCcy\n",
        " - Issr\n",
        "5. Store the csv from step 4) in an AWS S3 bucket\n",
        "6. The above function should be run as an AWS Lambda (Optional)\n",
        "\n",
        "## Assessment criteria:\n",
        "\n",
        "1. Percentage of requirements satisfied\n",
        "2. How clean the code is - in particular simplicity, adhering to python code style conventions and error handling.\n",
        "3. Follows PEP 8 guidelines\n",
        "4. We expect pydoc for each class and function with optional type hints(nice to have)\n",
        "5. Follows standard logging (no print statements). Logs are essential part of troubleshooting application.\n",
        "6. Unit tests with good code coverage"
      ],
      "metadata": {
        "id": "QeJzzaR1o0OI"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Importing Libraries"
      ],
      "metadata": {
        "id": "hqesyxMIpfDs"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install boto3"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "1eshB6NRINal",
        "outputId": "4a474291-1738-45cb-9328-f6b4ecd4f966"
      },
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
            "Requirement already satisfied: boto3 in /usr/local/lib/python3.9/dist-packages (1.26.118)\n",
            "Requirement already satisfied: s3transfer<0.7.0,>=0.6.0 in /usr/local/lib/python3.9/dist-packages (from boto3) (0.6.0)\n",
            "Requirement already satisfied: botocore<1.30.0,>=1.29.118 in /usr/local/lib/python3.9/dist-packages (from boto3) (1.29.118)\n",
            "Requirement already satisfied: jmespath<2.0.0,>=0.7.1 in /usr/local/lib/python3.9/dist-packages (from boto3) (1.0.1)\n",
            "Requirement already satisfied: urllib3<1.27,>=1.25.4 in /usr/local/lib/python3.9/dist-packages (from botocore<1.30.0,>=1.29.118->boto3) (1.26.15)\n",
            "Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in /usr/local/lib/python3.9/dist-packages (from botocore<1.30.0,>=1.29.118->boto3) (2.8.2)\n",
            "Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.9/dist-packages (from python-dateutil<3.0.0,>=2.1->botocore<1.30.0,>=1.29.118->boto3) (1.16.0)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "import xml.etree.ElementTree as ET              ## module to parse and manipulate XML data in Python\n",
        "import unittest                                 ## module for writing and running tests in Python\n",
        "import logging                                  ## module to create log messages to help with debugging and troubleshooting\n",
        "import pydoc                                    ## module generates documentation from Python code\n",
        "import urllib.request                           ## module for opening URLs\n",
        "from urllib.request import urlopen as uReq      ## function from urllib.request that opens URLs\n",
        "from bs4 import BeautifulSoup as soup           ## module to parse HTML and XML documents\n",
        "import zipfile                                  ## module to work with zip files\n",
        "import csv                                      ## modeule for reading and writing CSV files\n",
        "import pandas as pd                             ## module for storing and manipulating datasets\n",
        "import boto3                                    ## module for using AWS S3 \n",
        "import os                                       ## os module\n",
        "import io                                       ## io module\n",
        "import botocore.session                         ## provides a session object to interact with AWS services \n",
        "from boto3.session import Session               ## boto3"
      ],
      "metadata": {
        "id": "t9wzVy7bNakR"
      },
      "execution_count": 2,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "class Xml_Csv_Converter(unittest.TestCase):\n",
        "  '''A module to extract zip file from url. Download XML file from ZIP file. Extract features from XML and convert it to CSV. Upload CSV to AWS S3 bucket'''\n",
        "\n",
        "  def __init__(self, url):\n",
        "    '''Constructor to initialize url and logging'''\n",
        "    self.url = url\n",
        "    # Create and configure logger\n",
        "    logging.basicConfig(filename=\"log_file.log\", format='%(asctime)s %(message)s', filemode='w')\n",
        "\n",
        "    # Creating a logging object\n",
        "    self.logger = logging.getLogger()\n",
        "    \n",
        "    # Setting the threshold of logger to DEBUG\n",
        "    self.logger.setLevel(logging.DEBUG)\n",
        "\n",
        "\n",
        "  def download_zip(self): \n",
        "    ''' Step-1:  Download the zip file from url link. '''\n",
        "    ## Read web page from url\n",
        "    web_page= uReq(self.url)\n",
        "    page = web_page.read()\n",
        "\n",
        "    ## XML parser to parse file using beautifulsoup\n",
        "    self.page_soup= soup(page, 'xml')\n",
        "    self.logger.debug(\"\\nExtracted page from given url\")\n",
        "\n",
        "\n",
        "  def get_file(self, n):\n",
        "    ''' Step-2:  From the xml, please parse through to the first download link whose file_type is DLTINS and download the zip. '''\n",
        "    ## find all the files with type as 'DLTINS'\n",
        "    filetyp_list = self.page_soup.findAll('str',{'name':'file_type'})\n",
        "\n",
        "    ## get the first file from all the files\n",
        "    print(\"First_file:\", filetyp_list[n])\n",
        "\n",
        "    ## find all the download links\n",
        "    link_list = self.page_soup.findAll('str',{'name':'download_link'})\n",
        "\n",
        "    ## get the first link from all the links\n",
        "    print(\"First_link:\", link_list[n])\n",
        "\n",
        "    ## get file type from file\n",
        "    filetyp = filetyp_list[n].get_text()\n",
        "    print(\"\\n file:\", filetyp)\n",
        "\n",
        "    ## get link from the first link\n",
        "    self.link = link_list[n].get_text()\n",
        "    print(\"\\n link:\", self.link)\n",
        "    ## Write Logs\n",
        "    self.logger.debug(\"\\nDownloaded file\")\n",
        "\n",
        "  def extractXML(self):\n",
        "    ''' Step-3: Extract the xml from the zip. '''\n",
        "    ## get file name from link\n",
        "    data_file = self.link.split('/')[-1]\n",
        "\n",
        "    ## download the zip file from the URL\n",
        "    zip_ = urllib.request.urlretrieve(self.link, data_file)\n",
        "\n",
        "    ## extract zip file from zip_\n",
        "    zip_file = zipfile.ZipFile(data_file, 'r')\n",
        "\n",
        "    ## create file xml from zip file \n",
        "    file = data_file.strip('.zip')+'.xml'\n",
        "\n",
        "    ## read xml file from zip file\n",
        "    xml_file = zip_file.open(file)\n",
        "    xml_data = xml_file.read()\n",
        "\n",
        "    ## store xml data in xml_data file\n",
        "    with open('xml_data.xml', 'wb') as f:\n",
        "      f.write(xml_data)\n",
        "\n",
        "    self.xml_path = '/content/xml_data.xml'\n",
        "    ## Write Logs\n",
        "    self.logger.debug(\"\\nRead XML file\")\n",
        "  \n",
        "  def parseXML(self):\n",
        "    ''' Step-4:  Convert the contents of the xml into a CSV with the following header:\n",
        "                 FinInstrmGnlAttrbts.Id\n",
        "                 FinInstrmGnlAttrbts.FullNm\n",
        "                 FinInstrmGnlAttrbts.ClssfctnTp\n",
        "                 FinInstrmGnlAttrbts.CmmdtyDerivInd\n",
        "                 FinInstrmGnlAttrbts.NtnlCcy\n",
        "                 Issr '''\n",
        "    ## Parse the XML file\n",
        "    xml_doc = ET.parse(self.xml_path)\n",
        "    root = xml_doc.getroot()\n",
        "\n",
        "    # Find all namespace prefixes used in the XML document\n",
        "    ns_prefixes = []\n",
        "    for elem in xml_doc.iter():\n",
        "        if elem.tag.startswith('{'):\n",
        "            ns_prefix = elem.tag.split('}')[0][1:]\n",
        "            if ns_prefix not in ns_prefixes:\n",
        "                ns_prefixes.append(ns_prefix)\n",
        "\n",
        "    ## Define namespaces used in the XML file\n",
        "    namespaces = {\n",
        "        'head': ns_prefixes[0],\n",
        "        'app':  ns_prefixes[1],\n",
        "        'auth': ns_prefixes[2]\n",
        "    }\n",
        "\n",
        "    ## Extract the required data from the XML file\n",
        "    data = []\n",
        "\n",
        "    ## Find all the data for corresponding columns\n",
        "    for fininstrm in root.findall('.//auth:FinInstrm', namespaces):\n",
        "        '''FinInstrmGnlAttrbts.Id'''\n",
        "        id = fininstrm.find('.//auth:Id', namespaces).text\n",
        "\n",
        "        '''FinInstrmGnlAttrbts.FullNm'''\n",
        "        fullnm = fininstrm.find('.//auth:FullNm', namespaces).text\n",
        "        \n",
        "        '''FinInstrmGnlAttrbts.ClssfctnTp'''\n",
        "        clssfctnTp = fininstrm.find('.//auth:ClssfctnTp', namespaces).text\n",
        "        \n",
        "        '''FinInstrmGnlAttrbts.CmmdtyDerivInd'''\n",
        "        cmmdtyDerivInd = fininstrm.find('.//auth:CmmdtyDerivInd', namespaces).text\n",
        "        \n",
        "        '''FinInstrmGnlAttrbts.NtnlCcy'''\n",
        "        ntnlCcy = fininstrm.find('.//auth:NtnlCcy', namespaces).text\n",
        "        \n",
        "        '''FinInstrmGnlAttrbts.Issr'''\n",
        "        issr = fininstrm.find('.//auth:Issr', namespaces).text\n",
        "        \n",
        "        '''create list of rows for the columns'''\n",
        "        row = [id, fullnm, clssfctnTp, ntnlCcy, cmmdtyDerivInd, issr]\n",
        "        data.append(row)\n",
        "\n",
        "    ## Write the extracted data to a CSV file\n",
        "    with open('xml_dataset.csv', 'w', newline='') as file:\n",
        "        writer = csv.writer(file)\n",
        "        writer.writerow(['Id', 'FullNm', 'ClssfctnTp', 'CmmdtyDerivInd', 'NtnlCcy', 'Issr'])\n",
        "        writer.writerows(data)\n",
        "    ## Write Logs\n",
        "    self.logger.debug(\"\\nWrite data from XML to CSV file\")\n",
        "\n",
        "  def readCSV(self):\n",
        "    ''' Read csv dataset using pandas and inspect first 5 rows. '''\n",
        "    ## Read csv using pd\n",
        "    self.xml_dt = pd.read_csv('/content/xml_dataset.csv')\n",
        "    print(self.xml_dt.head())\n",
        "    ## Write Logs\n",
        "    self.logger.debug(\"\\nRead CSV file\")\n",
        "\n",
        "  def aws_bucket(self, bucket_name, key):\n",
        "    ''' Step-5:  Store the csv from Step-4 in an AWS S3 bucket '''\n",
        "    # set the environment variables for AWS access keys\n",
        "    os.environ['AWS_ACCESS_KEY_ID'] = 'AKIAS4QDNRDD6D4FPTMX'\n",
        "    os.environ['AWS_SECRET_ACCESS_KEY'] = '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'\n",
        "    \n",
        "    # create boto session and AWS access keys\n",
        "    session = botocore.session.get_session()\n",
        "    AWS_ACCESS_KEY_ID = session.get_credentials().access_key\n",
        "    AWS_SECRET_ACCESS_KEY = session.get_credentials().secret_key\n",
        "\n",
        "    # create an S3 client with AWS access keys\n",
        "    s3 = boto3.client(\"s3\", aws_access_key_id=AWS_ACCESS_KEY_ID, aws_secret_access_key=AWS_SECRET_ACCESS_KEY)\n",
        "    \n",
        "    # read the CSV file from disk and create a buffer\n",
        "    with open('xml_dataset.csv', newline='') as csvfile:\n",
        "      csv_buffer = io.StringIO(csvfile.read())\n",
        "      # upload the buffer to S3\n",
        "      s3.put_object(Body=csv_buffer.getvalue(), Bucket=bucket_name, Key=key)\n",
        "    \n",
        "    # create boto session and S3 client with AWS access keys to upload CSV file\n",
        "    session = Session(aws_access_key_id=AWS_ACCESS_KEY_ID,aws_secret_access_key=AWS_SECRET_ACCESS_KEY)\n",
        "    s3 = session.resource('s3')\n",
        "    my_bucket = s3.Bucket('mansi-pythonengineer-project')\n",
        "\n",
        "    for s3_file in my_bucket.objects.all():\n",
        "      print('File uploaded to S3 in AWS: ',s3_file.key)\n",
        "\n",
        "    ## Write logs\n",
        "    self.logger.debug(\"\\nUploaded CSV to AWS S3 bucket\")\n"
      ],
      "metadata": {
        "id": "_Cd9aYfhim8U"
      },
      "execution_count": 3,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "if __name__ == '__main__':\n",
        "    unittest.main(exit=False)\n",
        "    # given url\n",
        "    url='https://registers.esma.europa.eu/solr/esma_registers_firds_files/select?q=*&fq=publication_date:%5B2021-01-17T00:00:00Z+TO+2021-01-19T23:59:59Z%5D&wt=xml&indent=true&start=0&rows=100'\n",
        "    # create class object\n",
        "    obj = Xml_Csv_Converter(url)\n",
        "    # call object methods - download_zip()\n",
        "    obj.download_zip()\n",
        "    # enter value of n for zip file, here we will use n=0 for first file\n",
        "    n=int(input(\"enter file number of zip: 0/1/2/3: \"))\n",
        "    obj.get_file(n)\n",
        "    obj.extractXML()\n",
        "    obj.parseXML()\n",
        "    obj.readCSV()\n",
        "    # set the name of the S3 bucket and key\n",
        "    bucket_name = 'mansi-pythonengineer-project'\n",
        "    key = 'xml-file-dataset.csv'\n",
        "    # call function aws_bucket\n",
        "    obj.aws_bucket(bucket_name, key)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "sWFkttLMk77D",
        "outputId": "bdaf0721-8af9-42ac-f2d1-6183085b58c5"
      },
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "E\n",
            "======================================================================\n",
            "ERROR: /root/ (unittest.loader._FailedTest)\n",
            "----------------------------------------------------------------------\n",
            "AttributeError: module '__main__' has no attribute '/root/'\n",
            "\n",
            "----------------------------------------------------------------------\n",
            "Ran 1 test in 0.003s\n",
            "\n",
            "FAILED (errors=1)\n",
            "DEBUG:root:\n",
            "Extracted page from given url\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "enter file number of zip: 0/1/2/3: 0\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "DEBUG:root:\n",
            "Downloaded file\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "First_file: <str name=\"file_type\">DLTINS</str>\n",
            "First_link: <str name=\"download_link\">http://firds.esma.europa.eu/firds/DLTINS_20210117_01of01.zip</str>\n",
            "\n",
            " file: DLTINS\n",
            "\n",
            " link: http://firds.esma.europa.eu/firds/DLTINS_20210117_01of01.zip\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "DEBUG:root:\n",
            "Read XML file\n",
            "DEBUG:root:\n",
            "Write data from XML to CSV file\n",
            "DEBUG:root:\n",
            "Read CSV file\n",
            "DEBUG:botocore.hooks:Changing event name from creating-client-class.iot-data to creating-client-class.iot-data-plane\n",
            "DEBUG:botocore.hooks:Changing event name from before-call.apigateway to before-call.api-gateway\n",
            "DEBUG:botocore.hooks:Changing event name from request-created.machinelearning.Predict to request-created.machine-learning.Predict\n",
            "DEBUG:botocore.hooks:Changing event name from before-parameter-build.autoscaling.CreateLaunchConfiguration to before-parameter-build.auto-scaling.CreateLaunchConfiguration\n",
            "DEBUG:botocore.hooks:Changing event name from before-parameter-build.route53 to before-parameter-build.route-53\n",
            "DEBUG:botocore.hooks:Changing event name from request-created.cloudsearchdomain.Search to request-created.cloudsearch-domain.Search\n",
            "DEBUG:botocore.hooks:Changing event name from docs.*.autoscaling.CreateLaunchConfiguration.complete-section to docs.*.auto-scaling.CreateLaunchConfiguration.complete-section\n",
            "DEBUG:botocore.hooks:Changing event name from before-parameter-build.logs.CreateExportTask to before-parameter-build.cloudwatch-logs.CreateExportTask\n",
            "DEBUG:botocore.hooks:Changing event name from docs.*.logs.CreateExportTask.complete-section to docs.*.cloudwatch-logs.CreateExportTask.complete-section\n",
            "DEBUG:botocore.hooks:Changing event name from before-parameter-build.cloudsearchdomain.Search to before-parameter-build.cloudsearch-domain.Search\n",
            "DEBUG:botocore.hooks:Changing event name from docs.*.cloudsearchdomain.Search.complete-section to docs.*.cloudsearch-domain.Search.complete-section\n",
            "DEBUG:botocore.utils:IMDS ENDPOINT: http://169.254.169.254/\n",
            "DEBUG:botocore.credentials:Looking for credentials via: env\n",
            "INFO:botocore.credentials:Found credentials in environment variables.\n",
            "DEBUG:botocore.hooks:Changing event name from creating-client-class.iot-data to creating-client-class.iot-data-plane\n",
            "DEBUG:botocore.hooks:Changing event name from before-call.apigateway to before-call.api-gateway\n",
            "DEBUG:botocore.hooks:Changing event name from request-created.machinelearning.Predict to request-created.machine-learning.Predict\n",
            "DEBUG:botocore.hooks:Changing event name from before-parameter-build.autoscaling.CreateLaunchConfiguration to before-parameter-build.auto-scaling.CreateLaunchConfiguration\n",
            "DEBUG:botocore.hooks:Changing event name from before-parameter-build.route53 to before-parameter-build.route-53\n",
            "DEBUG:botocore.hooks:Changing event name from request-created.cloudsearchdomain.Search to request-created.cloudsearch-domain.Search\n",
            "DEBUG:botocore.hooks:Changing event name from docs.*.autoscaling.CreateLaunchConfiguration.complete-section to docs.*.auto-scaling.CreateLaunchConfiguration.complete-section\n",
            "DEBUG:botocore.hooks:Changing event name from before-parameter-build.logs.CreateExportTask to before-parameter-build.cloudwatch-logs.CreateExportTask\n",
            "DEBUG:botocore.hooks:Changing event name from docs.*.logs.CreateExportTask.complete-section to docs.*.cloudwatch-logs.CreateExportTask.complete-section\n",
            "DEBUG:botocore.hooks:Changing event name from before-parameter-build.cloudsearchdomain.Search to before-parameter-build.cloudsearch-domain.Search\n",
            "DEBUG:botocore.hooks:Changing event name from docs.*.cloudsearchdomain.Search.complete-section to docs.*.cloudsearch-domain.Search.complete-section\n",
            "DEBUG:botocore.loaders:Loading JSON file: /usr/local/lib/python3.9/dist-packages/botocore/data/endpoints.json\n",
            "DEBUG:botocore.loaders:Loading JSON file: /usr/local/lib/python3.9/dist-packages/botocore/data/sdk-default-configuration.json\n",
            "DEBUG:botocore.hooks:Event choose-service-name: calling handler <function handle_service_name_alias at 0x7f9008491e50>\n",
            "DEBUG:botocore.loaders:Loading JSON file: /usr/local/lib/python3.9/dist-packages/botocore/data/s3/2006-03-01/service-2.json\n",
            "DEBUG:botocore.loaders:Loading JSON file: /usr/local/lib/python3.9/dist-packages/botocore/data/s3/2006-03-01/endpoint-rule-set-1.json.gz\n",
            "DEBUG:botocore.loaders:Loading JSON file: /usr/local/lib/python3.9/dist-packages/botocore/data/partitions.json\n",
            "DEBUG:botocore.hooks:Event creating-client-class.s3: calling handler <function add_generate_presigned_post at 0x7f900850db80>\n",
            "DEBUG:botocore.hooks:Event creating-client-class.s3: calling handler <function lazy_call.<locals>._handler at 0x7f900841b3a0>\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "             Id                                             FullNm ClssfctnTp  \\\n",
            "0  DE000A1R07V3    Kreditanst.f.Wiederaufbau     Anl.v.2014 (2021)     DBFTFB   \n",
            "1  DE000A1R07V3                                 KFW 1 5/8 01/15/21     DBFTFB   \n",
            "2  DE000A1R07V3        Kreditanst.f.Wiederaufbau Anl.v.2014 (2021)     DBFTFB   \n",
            "3  DE000A1R07V3        Kreditanst.f.Wiederaufbau Anl.v.2014 (2021)     DBFTFB   \n",
            "4  DE000A1X3J56  IKB Deutsche Industriebank AG Stufenz.MTN-IHS ...     DTVUFB   \n",
            "\n",
            "  CmmdtyDerivInd  NtnlCcy                  Issr  \n",
            "0            EUR    False  549300GDPG70E3MBBU98  \n",
            "1            EUR    False  549300GDPG70E3MBBU98  \n",
            "2            EUR    False  549300GDPG70E3MBBU98  \n",
            "3            EUR    False  549300GDPG70E3MBBU98  \n",
            "4            EUR    False  PWEFG14QWWESISQ84C69  \n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "DEBUG:botocore.hooks:Event creating-client-class.s3: calling handler <function add_generate_presigned_url at 0x7f900850d940>\n",
            "DEBUG:botocore.endpoint:Setting s3 timeout as (60, 60)\n",
            "DEBUG:botocore.loaders:Loading JSON file: /usr/local/lib/python3.9/dist-packages/botocore/data/_retry.json\n",
            "DEBUG:botocore.client:Registering retry handlers for service: s3\n",
            "DEBUG:botocore.utils:Registering S3 region redirector handler\n",
            "DEBUG:botocore.hooks:Event before-endpoint-resolution.s3: calling handler <function customize_endpoint_resolver_builtins at 0x7f90084b0940>\n",
            "DEBUG:botocore.hooks:Event before-endpoint-resolution.s3: calling handler <bound method S3RegionRedirectorv2.redirect_from_cache of <botocore.utils.S3RegionRedirectorv2 object at 0x7f900764dac0>>\n",
            "DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Bucket': 'mansi-pythonengineer-project', 'Region': 'us-east-1', 'UseFIPS': False, 'UseDualStack': False, 'ForcePathStyle': False, 'Accelerate': False, 'UseGlobalEndpoint': True, 'DisableMultiRegionAccessPoints': False, 'UseArnRegion': True}\n",
            "DEBUG:botocore.regions:Endpoint provider result: https://mansi-pythonengineer-project.s3.amazonaws.com\n",
            "DEBUG:botocore.regions:Selecting from endpoint provider's list of auth schemes: \"sigv4\". User selected auth scheme is: \"None\"\n",
            "DEBUG:botocore.regions:Selected auth type \"v4\" as \"v4\" with signing context params: {'region': 'us-east-1', 'signing_name': 's3', 'disableDoubleEncoding': True}\n",
            "DEBUG:botocore.hooks:Event before-parameter-build.s3.PutObject: calling handler <function validate_ascii_metadata at 0x7f90084af5e0>\n",
            "DEBUG:botocore.hooks:Event before-parameter-build.s3.PutObject: calling handler <function sse_md5 at 0x7f90084ac9d0>\n",
            "DEBUG:botocore.hooks:Event before-parameter-build.s3.PutObject: calling handler <function convert_body_to_file_like_object at 0x7f90084afee0>\n",
            "DEBUG:botocore.hooks:Event before-parameter-build.s3.PutObject: calling handler <function validate_bucket_name at 0x7f90084ac940>\n",
            "DEBUG:botocore.hooks:Event before-parameter-build.s3.PutObject: calling handler <function remove_bucket_from_url_paths_from_model at 0x7f90084b0790>\n",
            "DEBUG:botocore.hooks:Event before-parameter-build.s3.PutObject: calling handler <bound method S3RegionRedirectorv2.annotate_request_context of <botocore.utils.S3RegionRedirectorv2 object at 0x7f900764dac0>>\n",
            "DEBUG:botocore.hooks:Event before-parameter-build.s3.PutObject: calling handler <function generate_idempotent_uuid at 0x7f90084ac790>\n",
            "DEBUG:botocore.hooks:Event before-call.s3.PutObject: calling handler <function conditionally_calculate_md5 at 0x7f90085d9790>\n",
            "DEBUG:botocore.hooks:Event before-call.s3.PutObject: calling handler <function add_expect_header at 0x7f90084acca0>\n",
            "DEBUG:botocore.handlers:Adding expect 100 continue header to request.\n",
            "DEBUG:botocore.hooks:Event before-call.s3.PutObject: calling handler <function add_recursion_detection_header at 0x7f90084ac430>\n",
            "DEBUG:botocore.hooks:Event before-call.s3.PutObject: calling handler <function inject_api_version_header_if_needed at 0x7f90084b0040>\n",
            "DEBUG:botocore.endpoint:Making request for OperationModel(name=PutObject) with params: {'url_path': '/xml-file-dataset.csv', 'query_string': {}, 'method': 'PUT', 'headers': {'User-Agent': 'Boto3/1.26.118 Python/3.9.16 Linux/5.10.147+ Botocore/1.29.118', 'Content-MD5': 'fh8T66ftduGHQ820CZ1dZA==', 'Expect': '100-continue'}, 'body': <_io.BytesIO object at 0x7f9007663130>, 'auth_path': '/mansi-pythonengineer-project/xml-file-dataset.csv', 'url': 'https://mansi-pythonengineer-project.s3.amazonaws.com/xml-file-dataset.csv', 'context': {'client_region': 'us-east-1', 'client_config': <botocore.config.Config object at 0x7f9007680370>, 'has_streaming_input': True, 'auth_type': 'v4', 'signing': {'region': 'us-east-1', 'signing_name': 's3', 'disableDoubleEncoding': True}, 's3_redirect': {'redirected': False, 'bucket': 'mansi-pythonengineer-project', 'params': {'Body': <_io.BytesIO object at 0x7f9007663130>, 'Bucket': 'mansi-pythonengineer-project', 'Key': 'xml-file-dataset.csv'}}}}\n",
            "DEBUG:botocore.hooks:Event request-created.s3.PutObject: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x7f900778beb0>>\n",
            "DEBUG:botocore.hooks:Event choose-signer.s3.PutObject: calling handler <bound method ClientCreator._default_s3_presign_to_sigv2 of <botocore.client.ClientCreator object at 0x7f9008087790>>\n",
            "DEBUG:botocore.hooks:Event choose-signer.s3.PutObject: calling handler <function set_operation_specific_signer at 0x7f90084ac670>\n",
            "DEBUG:botocore.hooks:Event before-sign.s3.PutObject: calling handler <function remove_arn_from_signing_path at 0x7f90084b08b0>\n",
            "DEBUG:botocore.auth:Calculating signature using v4 auth.\n",
            "DEBUG:botocore.auth:CanonicalRequest:\n",
            "PUT\n",
            "/xml-file-dataset.csv\n",
            "\n",
            "content-md5:fh8T66ftduGHQ820CZ1dZA==\n",
            "host:mansi-pythonengineer-project.s3.amazonaws.com\n",
            "x-amz-content-sha256:UNSIGNED-PAYLOAD\n",
            "x-amz-date:20230423T165014Z\n",
            "\n",
            "content-md5;host;x-amz-content-sha256;x-amz-date\n",
            "UNSIGNED-PAYLOAD\n",
            "DEBUG:botocore.auth:StringToSign:\n",
            "AWS4-HMAC-SHA256\n",
            "20230423T165014Z\n",
            "20230423/us-east-1/s3/aws4_request\n",
            "26ee6dc421263ab6519f61b3830c863e3c74327991656d88b979e0390a1a6722\n",
            "DEBUG:botocore.auth:Signature:\n",
            "e9925b0881988f3a770d61b281c40bc3df92df6894606b836275ef710e925974\n",
            "DEBUG:botocore.hooks:Event request-created.s3.PutObject: calling handler <function add_retry_headers at 0x7f90084b0700>\n",
            "DEBUG:botocore.endpoint:Sending http request: <AWSPreparedRequest stream_output=False, method=PUT, url=https://mansi-pythonengineer-project.s3.amazonaws.com/xml-file-dataset.csv, headers={'User-Agent': b'Boto3/1.26.118 Python/3.9.16 Linux/5.10.147+ Botocore/1.29.118', 'Content-MD5': b'fh8T66ftduGHQ820CZ1dZA==', 'Expect': b'100-continue', 'X-Amz-Date': b'20230423T165014Z', 'X-Amz-Content-SHA256': b'UNSIGNED-PAYLOAD', 'Authorization': b'AWS4-HMAC-SHA256 Credential=AKIAS4QDNRDD6D4FPTMX/20230423/us-east-1/s3/aws4_request, SignedHeaders=content-md5;host;x-amz-content-sha256;x-amz-date, Signature=e9925b0881988f3a770d61b281c40bc3df92df6894606b836275ef710e925974', 'amz-sdk-invocation-id': b'937ebd31-21f9-4994-a76f-81b785fc2b0c', 'amz-sdk-request': b'attempt=1', 'Content-Length': '11064539'}>\n",
            "DEBUG:botocore.httpsession:Certificate path: /usr/local/lib/python3.9/dist-packages/certifi/cacert.pem\n",
            "DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): mansi-pythonengineer-project.s3.amazonaws.com:443\n",
            "DEBUG:botocore.awsrequest:Waiting for 100 Continue response.\n",
            "DEBUG:botocore.awsrequest:Received a non 100 Continue response from the server, NOT sending request body.\n",
            "DEBUG:urllib3.connectionpool:https://mansi-pythonengineer-project.s3.amazonaws.com:443 \"PUT /xml-file-dataset.csv HTTP/1.1\" 400 None\n",
            "DEBUG:botocore.parsers:Response headers: {'x-amz-request-id': 'NGPZEFSBSSCNZ2XG', 'x-amz-id-2': 'pcgw+EyRwMaD5im8k8BrsOsJMFnF8C8XPMs+VbkjY3bMbkXw3bcJRwSvg/m+k4LGez+fWTFOFjk=', 'Content-Type': 'application/xml', 'Transfer-Encoding': 'chunked', 'Date': 'Sun, 23 Apr 2023 16:50:14 GMT', 'Server': 'AmazonS3', 'Connection': 'close'}\n",
            "DEBUG:botocore.parsers:Response body:\n",
            "b'<?xml version=\"1.0\" encoding=\"UTF-8\"?>\\n<Error><Code>AuthorizationHeaderMalformed</Code><Message>The authorization header is malformed; the region \\'us-east-1\\' is wrong; expecting \\'ap-south-1\\'</Message><Region>ap-south-1</Region><RequestId>NGPZEFSBSSCNZ2XG</RequestId><HostId>pcgw+EyRwMaD5im8k8BrsOsJMFnF8C8XPMs+VbkjY3bMbkXw3bcJRwSvg/m+k4LGez+fWTFOFjk=</HostId></Error>'\n",
            "DEBUG:botocore.hooks:Event needs-retry.s3.PutObject: calling handler <botocore.retryhandler.RetryHandler object at 0x7f900764da00>\n",
            "DEBUG:botocore.retryhandler:No retry needed.\n",
            "DEBUG:botocore.hooks:Event needs-retry.s3.PutObject: calling handler <bound method S3RegionRedirectorv2.redirect_from_error of <botocore.utils.S3RegionRedirectorv2 object at 0x7f900764dac0>>\n",
            "DEBUG:botocore.utils:S3 client configured for region us-east-1 but the bucket mansi-pythonengineer-project is in region ap-south-1; Please configure the proper region to avoid multiple unnecessary redirects and signing attempts.\n",
            "DEBUG:botocore.hooks:Event before-endpoint-resolution.s3: calling handler <function customize_endpoint_resolver_builtins at 0x7f90084b0940>\n",
            "DEBUG:botocore.hooks:Event before-endpoint-resolution.s3: calling handler <bound method S3RegionRedirectorv2.redirect_from_cache of <botocore.utils.S3RegionRedirectorv2 object at 0x7f900764dac0>>\n",
            "DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Bucket': 'mansi-pythonengineer-project', 'Region': 'ap-south-1', 'UseFIPS': False, 'UseDualStack': False, 'ForcePathStyle': False, 'Accelerate': False, 'UseGlobalEndpoint': True, 'DisableMultiRegionAccessPoints': False, 'UseArnRegion': True}\n",
            "DEBUG:botocore.regions:Endpoint provider result: https://mansi-pythonengineer-project.s3.ap-south-1.amazonaws.com\n",
            "DEBUG:botocore.utils:Updating URI from https://mansi-pythonengineer-project.s3.amazonaws.com/xml-file-dataset.csv to https://mansi-pythonengineer-project.s3.ap-south-1.amazonaws.com/xml-file-dataset.csv\n",
            "DEBUG:botocore.regions:Selecting from endpoint provider's list of auth schemes: \"sigv4\". User selected auth scheme is: \"None\"\n",
            "DEBUG:botocore.regions:Selected auth type \"v4\" as \"v4\" with signing context params: {'region': 'ap-south-1', 'signing_name': 's3', 'disableDoubleEncoding': True}\n",
            "DEBUG:botocore.endpoint:Response received to retry, sleeping for 0 seconds\n",
            "DEBUG:botocore.awsrequest:Rewinding stream: <_io.BytesIO object at 0x7f9007663130>\n",
            "DEBUG:botocore.hooks:Event request-created.s3.PutObject: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x7f900778beb0>>\n",
            "DEBUG:botocore.hooks:Event choose-signer.s3.PutObject: calling handler <bound method ClientCreator._default_s3_presign_to_sigv2 of <botocore.client.ClientCreator object at 0x7f9008087790>>\n",
            "DEBUG:botocore.hooks:Event choose-signer.s3.PutObject: calling handler <function set_operation_specific_signer at 0x7f90084ac670>\n",
            "DEBUG:botocore.hooks:Event before-sign.s3.PutObject: calling handler <function remove_arn_from_signing_path at 0x7f90084b08b0>\n",
            "DEBUG:botocore.auth:Calculating signature using v4 auth.\n",
            "DEBUG:botocore.auth:CanonicalRequest:\n",
            "PUT\n",
            "/xml-file-dataset.csv\n",
            "\n",
            "content-md5:fh8T66ftduGHQ820CZ1dZA==\n",
            "host:mansi-pythonengineer-project.s3.ap-south-1.amazonaws.com\n",
            "x-amz-content-sha256:UNSIGNED-PAYLOAD\n",
            "x-amz-date:20230423T165015Z\n",
            "\n",
            "content-md5;host;x-amz-content-sha256;x-amz-date\n",
            "UNSIGNED-PAYLOAD\n",
            "DEBUG:botocore.auth:StringToSign:\n",
            "AWS4-HMAC-SHA256\n",
            "20230423T165015Z\n",
            "20230423/ap-south-1/s3/aws4_request\n",
            "7bcee9df73148b607afa864d88a02c3989f2faece9272724113622ed77148e31\n",
            "DEBUG:botocore.auth:Signature:\n",
            "5a5c4cdeaf6c8a528b086ecf7ae4b39186a360f58eaff1c2878e5d7bfd83e451\n",
            "DEBUG:botocore.hooks:Event request-created.s3.PutObject: calling handler <function add_retry_headers at 0x7f90084b0700>\n",
            "DEBUG:botocore.endpoint:Sending http request: <AWSPreparedRequest stream_output=False, method=PUT, url=https://mansi-pythonengineer-project.s3.ap-south-1.amazonaws.com/xml-file-dataset.csv, headers={'User-Agent': b'Boto3/1.26.118 Python/3.9.16 Linux/5.10.147+ Botocore/1.29.118', 'Content-MD5': b'fh8T66ftduGHQ820CZ1dZA==', 'Expect': b'100-continue', 'X-Amz-Date': b'20230423T165015Z', 'X-Amz-Content-SHA256': b'UNSIGNED-PAYLOAD', 'Authorization': b'AWS4-HMAC-SHA256 Credential=AKIAS4QDNRDD6D4FPTMX/20230423/ap-south-1/s3/aws4_request, SignedHeaders=content-md5;host;x-amz-content-sha256;x-amz-date, Signature=5a5c4cdeaf6c8a528b086ecf7ae4b39186a360f58eaff1c2878e5d7bfd83e451', 'amz-sdk-invocation-id': b'937ebd31-21f9-4994-a76f-81b785fc2b0c', 'amz-sdk-request': b'ttl=20230423T165114Z; attempt=2; max=5', 'Content-Length': '11064539'}>\n",
            "DEBUG:botocore.httpsession:Certificate path: /usr/local/lib/python3.9/dist-packages/certifi/cacert.pem\n",
            "DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): mansi-pythonengineer-project.s3.ap-south-1.amazonaws.com:443\n",
            "DEBUG:botocore.awsrequest:Waiting for 100 Continue response.\n",
            "DEBUG:botocore.awsrequest:100 Continue response seen, now sending request body.\n",
            "DEBUG:urllib3.connectionpool:https://mansi-pythonengineer-project.s3.ap-south-1.amazonaws.com:443 \"PUT /xml-file-dataset.csv HTTP/1.1\" 200 0\n",
            "DEBUG:botocore.parsers:Response headers: {'x-amz-id-2': 'LUfOGgHwwS3TEYtDqdeDXUUEbdHvNuQ1jXcU1vFA2m8q/+5me9JHnxVSddSpd4RFNcQdDbfgAYu8ow7cQ5YSWg==', 'x-amz-request-id': 'QAWKBT6H7Z2ZE905', 'Date': 'Sun, 23 Apr 2023 16:50:17 GMT', 'x-amz-version-id': 'w.TYPclZjlQzpiTETQ.S9kg.yYtyU4bA', 'x-amz-server-side-encryption': 'AES256', 'ETag': '\"7e1f13eba7ed76e18743cdb4099d5d64\"', 'Server': 'AmazonS3', 'Content-Length': '0'}\n",
            "DEBUG:botocore.parsers:Response body:\n",
            "b''\n",
            "DEBUG:botocore.hooks:Event needs-retry.s3.PutObject: calling handler <botocore.retryhandler.RetryHandler object at 0x7f900764da00>\n",
            "DEBUG:botocore.retryhandler:No retry needed.\n",
            "DEBUG:botocore.hooks:Event needs-retry.s3.PutObject: calling handler <bound method S3RegionRedirectorv2.redirect_from_error of <botocore.utils.S3RegionRedirectorv2 object at 0x7f900764dac0>>\n",
            "DEBUG:botocore.utils:S3 request was previously redirected, not redirecting.\n",
            "DEBUG:botocore.hooks:Changing event name from creating-client-class.iot-data to creating-client-class.iot-data-plane\n",
            "DEBUG:botocore.hooks:Changing event name from before-call.apigateway to before-call.api-gateway\n",
            "DEBUG:botocore.hooks:Changing event name from request-created.machinelearning.Predict to request-created.machine-learning.Predict\n",
            "DEBUG:botocore.hooks:Changing event name from before-parameter-build.autoscaling.CreateLaunchConfiguration to before-parameter-build.auto-scaling.CreateLaunchConfiguration\n",
            "DEBUG:botocore.hooks:Changing event name from before-parameter-build.route53 to before-parameter-build.route-53\n",
            "DEBUG:botocore.hooks:Changing event name from request-created.cloudsearchdomain.Search to request-created.cloudsearch-domain.Search\n",
            "DEBUG:botocore.hooks:Changing event name from docs.*.autoscaling.CreateLaunchConfiguration.complete-section to docs.*.auto-scaling.CreateLaunchConfiguration.complete-section\n",
            "DEBUG:botocore.hooks:Changing event name from before-parameter-build.logs.CreateExportTask to before-parameter-build.cloudwatch-logs.CreateExportTask\n",
            "DEBUG:botocore.hooks:Changing event name from docs.*.logs.CreateExportTask.complete-section to docs.*.cloudwatch-logs.CreateExportTask.complete-section\n",
            "DEBUG:botocore.hooks:Changing event name from before-parameter-build.cloudsearchdomain.Search to before-parameter-build.cloudsearch-domain.Search\n",
            "DEBUG:botocore.hooks:Changing event name from docs.*.cloudsearchdomain.Search.complete-section to docs.*.cloudsearch-domain.Search.complete-section\n",
            "DEBUG:botocore.loaders:Loading JSON file: /usr/local/lib/python3.9/dist-packages/boto3/data/s3/2006-03-01/resources-1.json\n",
            "DEBUG:botocore.loaders:Loading JSON file: /usr/local/lib/python3.9/dist-packages/botocore/data/endpoints.json\n",
            "DEBUG:botocore.loaders:Loading JSON file: /usr/local/lib/python3.9/dist-packages/botocore/data/sdk-default-configuration.json\n",
            "DEBUG:botocore.hooks:Event choose-service-name: calling handler <function handle_service_name_alias at 0x7f9008491e50>\n",
            "DEBUG:botocore.loaders:Loading JSON file: /usr/local/lib/python3.9/dist-packages/botocore/data/s3/2006-03-01/service-2.json\n",
            "DEBUG:botocore.loaders:Loading JSON file: /usr/local/lib/python3.9/dist-packages/botocore/data/s3/2006-03-01/endpoint-rule-set-1.json.gz\n",
            "DEBUG:botocore.loaders:Loading JSON file: /usr/local/lib/python3.9/dist-packages/botocore/data/partitions.json\n",
            "DEBUG:botocore.hooks:Event creating-client-class.s3: calling handler <function add_generate_presigned_post at 0x7f900850db80>\n",
            "DEBUG:botocore.hooks:Event creating-client-class.s3: calling handler <function lazy_call.<locals>._handler at 0x7f9007687790>\n",
            "DEBUG:botocore.hooks:Event creating-client-class.s3: calling handler <function add_generate_presigned_url at 0x7f900850d940>\n",
            "DEBUG:botocore.endpoint:Setting s3 timeout as (60, 60)\n",
            "DEBUG:botocore.loaders:Loading JSON file: /usr/local/lib/python3.9/dist-packages/botocore/data/_retry.json\n",
            "DEBUG:botocore.client:Registering retry handlers for service: s3\n",
            "DEBUG:botocore.utils:Registering S3 region redirector handler\n",
            "DEBUG:boto3.resources.factory:Loading s3:s3\n",
            "DEBUG:boto3.resources.factory:Loading s3:Bucket\n",
            "DEBUG:boto3.resources.model:Renaming Bucket attribute name\n",
            "DEBUG:botocore.hooks:Event creating-resource-class.s3.Bucket: calling handler <function lazy_call.<locals>._handler at 0x7f90076875e0>\n",
            "DEBUG:botocore.loaders:Loading JSON file: /usr/local/lib/python3.9/dist-packages/botocore/data/s3/2006-03-01/paginators-1.json\n",
            "DEBUG:boto3.resources.collection:Calling paginated s3:list_objects with {'Bucket': 'mansi-pythonengineer-project'}\n",
            "DEBUG:botocore.hooks:Event before-endpoint-resolution.s3: calling handler <function customize_endpoint_resolver_builtins at 0x7f90084b0940>\n",
            "DEBUG:botocore.hooks:Event before-endpoint-resolution.s3: calling handler <bound method S3RegionRedirectorv2.redirect_from_cache of <botocore.utils.S3RegionRedirectorv2 object at 0x7f9006bdd2e0>>\n",
            "DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Bucket': 'mansi-pythonengineer-project', 'Region': 'us-east-1', 'UseFIPS': False, 'UseDualStack': False, 'ForcePathStyle': False, 'Accelerate': False, 'UseGlobalEndpoint': True, 'DisableMultiRegionAccessPoints': False, 'UseArnRegion': True}\n",
            "DEBUG:botocore.regions:Endpoint provider result: https://mansi-pythonengineer-project.s3.amazonaws.com\n",
            "DEBUG:botocore.regions:Selecting from endpoint provider's list of auth schemes: \"sigv4\". User selected auth scheme is: \"None\"\n",
            "DEBUG:botocore.regions:Selected auth type \"v4\" as \"v4\" with signing context params: {'region': 'us-east-1', 'signing_name': 's3', 'disableDoubleEncoding': True}\n",
            "DEBUG:botocore.hooks:Event before-parameter-build.s3.ListObjects: calling handler <function set_list_objects_encoding_type_url at 0x7f90084afc10>\n",
            "DEBUG:botocore.hooks:Event before-parameter-build.s3.ListObjects: calling handler <function validate_bucket_name at 0x7f90084ac940>\n",
            "DEBUG:botocore.hooks:Event before-parameter-build.s3.ListObjects: calling handler <function remove_bucket_from_url_paths_from_model at 0x7f90084b0790>\n",
            "DEBUG:botocore.hooks:Event before-parameter-build.s3.ListObjects: calling handler <bound method S3RegionRedirectorv2.annotate_request_context of <botocore.utils.S3RegionRedirectorv2 object at 0x7f9006bdd2e0>>\n",
            "DEBUG:botocore.hooks:Event before-parameter-build.s3.ListObjects: calling handler <function generate_idempotent_uuid at 0x7f90084ac790>\n",
            "DEBUG:botocore.hooks:Event before-call.s3.ListObjects: calling handler <function add_expect_header at 0x7f90084acca0>\n",
            "DEBUG:botocore.hooks:Event before-call.s3.ListObjects: calling handler <function add_recursion_detection_header at 0x7f90084ac430>\n",
            "DEBUG:botocore.hooks:Event before-call.s3.ListObjects: calling handler <function inject_api_version_header_if_needed at 0x7f90084b0040>\n",
            "DEBUG:botocore.endpoint:Making request for OperationModel(name=ListObjects) with params: {'url_path': '', 'query_string': {'encoding-type': 'url'}, 'method': 'GET', 'headers': {'User-Agent': 'Boto3/1.26.118 Python/3.9.16 Linux/5.10.147+ Botocore/1.29.118 Resource'}, 'body': b'', 'auth_path': '/mansi-pythonengineer-project/', 'url': 'https://mansi-pythonengineer-project.s3.amazonaws.com/?encoding-type=url', 'context': {'client_region': 'us-east-1', 'client_config': <botocore.config.Config object at 0x7f9006c0da30>, 'has_streaming_input': False, 'auth_type': 'v4', 'signing': {'region': 'us-east-1', 'signing_name': 's3', 'disableDoubleEncoding': True}, 'encoding_type_auto_set': True, 's3_redirect': {'redirected': False, 'bucket': 'mansi-pythonengineer-project', 'params': {'Bucket': 'mansi-pythonengineer-project', 'EncodingType': 'url'}}}}\n",
            "DEBUG:botocore.hooks:Event request-created.s3.ListObjects: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x7f9006c0d760>>\n",
            "DEBUG:botocore.hooks:Event choose-signer.s3.ListObjects: calling handler <bound method ClientCreator._default_s3_presign_to_sigv2 of <botocore.client.ClientCreator object at 0x7f90075c7850>>\n",
            "DEBUG:botocore.hooks:Event choose-signer.s3.ListObjects: calling handler <function set_operation_specific_signer at 0x7f90084ac670>\n",
            "DEBUG:botocore.hooks:Event before-sign.s3.ListObjects: calling handler <function remove_arn_from_signing_path at 0x7f90084b08b0>\n",
            "DEBUG:botocore.auth:Calculating signature using v4 auth.\n",
            "DEBUG:botocore.auth:CanonicalRequest:\n",
            "GET\n",
            "/\n",
            "encoding-type=url\n",
            "host:mansi-pythonengineer-project.s3.amazonaws.com\n",
            "x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\n",
            "x-amz-date:20230423T165019Z\n",
            "\n",
            "host;x-amz-content-sha256;x-amz-date\n",
            "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\n",
            "DEBUG:botocore.auth:StringToSign:\n",
            "AWS4-HMAC-SHA256\n",
            "20230423T165019Z\n",
            "20230423/us-east-1/s3/aws4_request\n",
            "23196d469e4078d03047becc28e9e0eb53a1160f2ccf027d59ac3528b7075933\n",
            "DEBUG:botocore.auth:Signature:\n",
            "730cca723a3afcdec31b68b3a50e248480cc7f92b3d30ef71cb545f6302f575a\n",
            "DEBUG:botocore.hooks:Event request-created.s3.ListObjects: calling handler <function add_retry_headers at 0x7f90084b0700>\n",
            "DEBUG:botocore.endpoint:Sending http request: <AWSPreparedRequest stream_output=False, method=GET, url=https://mansi-pythonengineer-project.s3.amazonaws.com/?encoding-type=url, headers={'User-Agent': b'Boto3/1.26.118 Python/3.9.16 Linux/5.10.147+ Botocore/1.29.118 Resource', 'X-Amz-Date': b'20230423T165019Z', 'X-Amz-Content-SHA256': b'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855', 'Authorization': b'AWS4-HMAC-SHA256 Credential=AKIAS4QDNRDD6D4FPTMX/20230423/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=730cca723a3afcdec31b68b3a50e248480cc7f92b3d30ef71cb545f6302f575a', 'amz-sdk-invocation-id': b'7aef9ab0-b0e4-4745-be49-0bef84d3f93b', 'amz-sdk-request': b'attempt=1'}>\n",
            "DEBUG:botocore.httpsession:Certificate path: /usr/local/lib/python3.9/dist-packages/certifi/cacert.pem\n",
            "DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): mansi-pythonengineer-project.s3.amazonaws.com:443\n",
            "DEBUG:urllib3.connectionpool:https://mansi-pythonengineer-project.s3.amazonaws.com:443 \"GET /?encoding-type=url HTTP/1.1\" 400 None\n",
            "DEBUG:botocore.parsers:Response headers: {'x-amz-bucket-region': 'ap-south-1', 'x-amz-request-id': 'BEHJ483SN1JWN1KD', 'x-amz-id-2': 'FZ/t8dtaLnONRQoRLYzs09KA4ZgFny2DVLx54kUI94E8awBLR8yMhx8YyxQAMsr+yDA9Gzlqj24=', 'Content-Type': 'application/xml', 'Transfer-Encoding': 'chunked', 'Date': 'Sun, 23 Apr 2023 16:50:20 GMT', 'Server': 'AmazonS3', 'Connection': 'close'}\n",
            "DEBUG:botocore.parsers:Response body:\n",
            "b'<?xml version=\"1.0\" encoding=\"UTF-8\"?>\\n<Error><Code>AuthorizationHeaderMalformed</Code><Message>The authorization header is malformed; the region \\'us-east-1\\' is wrong; expecting \\'ap-south-1\\'</Message><Region>ap-south-1</Region><RequestId>BEHJ483SN1JWN1KD</RequestId><HostId>FZ/t8dtaLnONRQoRLYzs09KA4ZgFny2DVLx54kUI94E8awBLR8yMhx8YyxQAMsr+yDA9Gzlqj24=</HostId></Error>'\n",
            "DEBUG:botocore.hooks:Event needs-retry.s3.ListObjects: calling handler <botocore.retryhandler.RetryHandler object at 0x7f9006bdd220>\n",
            "DEBUG:botocore.retryhandler:No retry needed.\n",
            "DEBUG:botocore.hooks:Event needs-retry.s3.ListObjects: calling handler <bound method S3RegionRedirectorv2.redirect_from_error of <botocore.utils.S3RegionRedirectorv2 object at 0x7f9006bdd2e0>>\n",
            "DEBUG:botocore.utils:S3 client configured for region us-east-1 but the bucket mansi-pythonengineer-project is in region ap-south-1; Please configure the proper region to avoid multiple unnecessary redirects and signing attempts.\n",
            "DEBUG:botocore.hooks:Event before-endpoint-resolution.s3: calling handler <function customize_endpoint_resolver_builtins at 0x7f90084b0940>\n",
            "DEBUG:botocore.hooks:Event before-endpoint-resolution.s3: calling handler <bound method S3RegionRedirectorv2.redirect_from_cache of <botocore.utils.S3RegionRedirectorv2 object at 0x7f9006bdd2e0>>\n",
            "DEBUG:botocore.regions:Calling endpoint provider with parameters: {'Bucket': 'mansi-pythonengineer-project', 'Region': 'ap-south-1', 'UseFIPS': False, 'UseDualStack': False, 'ForcePathStyle': False, 'Accelerate': False, 'UseGlobalEndpoint': True, 'DisableMultiRegionAccessPoints': False, 'UseArnRegion': True}\n",
            "DEBUG:botocore.regions:Endpoint provider result: https://mansi-pythonengineer-project.s3.ap-south-1.amazonaws.com\n",
            "DEBUG:botocore.utils:Updating URI from https://mansi-pythonengineer-project.s3.amazonaws.com/?encoding-type=url to https://mansi-pythonengineer-project.s3.ap-south-1.amazonaws.com/?encoding-type=url\n",
            "DEBUG:botocore.regions:Selecting from endpoint provider's list of auth schemes: \"sigv4\". User selected auth scheme is: \"None\"\n",
            "DEBUG:botocore.regions:Selected auth type \"v4\" as \"v4\" with signing context params: {'region': 'ap-south-1', 'signing_name': 's3', 'disableDoubleEncoding': True}\n",
            "DEBUG:botocore.endpoint:Response received to retry, sleeping for 0 seconds\n",
            "DEBUG:botocore.hooks:Event request-created.s3.ListObjects: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x7f9006c0d760>>\n",
            "DEBUG:botocore.hooks:Event choose-signer.s3.ListObjects: calling handler <bound method ClientCreator._default_s3_presign_to_sigv2 of <botocore.client.ClientCreator object at 0x7f90075c7850>>\n",
            "DEBUG:botocore.hooks:Event choose-signer.s3.ListObjects: calling handler <function set_operation_specific_signer at 0x7f90084ac670>\n",
            "DEBUG:botocore.hooks:Event before-sign.s3.ListObjects: calling handler <function remove_arn_from_signing_path at 0x7f90084b08b0>\n",
            "DEBUG:botocore.auth:Calculating signature using v4 auth.\n",
            "DEBUG:botocore.auth:CanonicalRequest:\n",
            "GET\n",
            "/\n",
            "encoding-type=url\n",
            "host:mansi-pythonengineer-project.s3.ap-south-1.amazonaws.com\n",
            "x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\n",
            "x-amz-date:20230423T165020Z\n",
            "\n",
            "host;x-amz-content-sha256;x-amz-date\n",
            "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\n",
            "DEBUG:botocore.auth:StringToSign:\n",
            "AWS4-HMAC-SHA256\n",
            "20230423T165020Z\n",
            "20230423/ap-south-1/s3/aws4_request\n",
            "9ee3e8f4cf3ac8175f7263eec45a6dc21904e4c2d3262458431d9bca5e14e454\n",
            "DEBUG:botocore.auth:Signature:\n",
            "94a77265c20c6d7dfc62a4a243659781f06620eafb5238a7d1a16d7e4dcf7ecd\n",
            "DEBUG:botocore.hooks:Event request-created.s3.ListObjects: calling handler <function add_retry_headers at 0x7f90084b0700>\n",
            "DEBUG:botocore.endpoint:Sending http request: <AWSPreparedRequest stream_output=False, method=GET, url=https://mansi-pythonengineer-project.s3.ap-south-1.amazonaws.com/?encoding-type=url, headers={'User-Agent': b'Boto3/1.26.118 Python/3.9.16 Linux/5.10.147+ Botocore/1.29.118 Resource', 'X-Amz-Date': b'20230423T165020Z', 'X-Amz-Content-SHA256': b'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855', 'Authorization': b'AWS4-HMAC-SHA256 Credential=AKIAS4QDNRDD6D4FPTMX/20230423/ap-south-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=94a77265c20c6d7dfc62a4a243659781f06620eafb5238a7d1a16d7e4dcf7ecd', 'amz-sdk-invocation-id': b'7aef9ab0-b0e4-4745-be49-0bef84d3f93b', 'amz-sdk-request': b'ttl=20230423T165120Z; attempt=2; max=5'}>\n",
            "DEBUG:botocore.httpsession:Certificate path: /usr/local/lib/python3.9/dist-packages/certifi/cacert.pem\n",
            "DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): mansi-pythonengineer-project.s3.ap-south-1.amazonaws.com:443\n",
            "DEBUG:urllib3.connectionpool:https://mansi-pythonengineer-project.s3.ap-south-1.amazonaws.com:443 \"GET /?encoding-type=url HTTP/1.1\" 200 None\n",
            "DEBUG:botocore.parsers:Response headers: {'x-amz-id-2': 'aoQU/J773aZYROXo2E5URtDvwBTduDu1Q26HozQNQPcMjde0xWarip9vRn61oZhVvihs+o8Dap4=', 'x-amz-request-id': 'AX7985ZD610BJV4R', 'Date': 'Sun, 23 Apr 2023 16:50:22 GMT', 'x-amz-bucket-region': 'ap-south-1', 'Content-Type': 'application/xml', 'Transfer-Encoding': 'chunked', 'Server': 'AmazonS3'}\n",
            "DEBUG:botocore.parsers:Response body:\n",
            "b'<?xml version=\"1.0\" encoding=\"UTF-8\"?>\\n<ListBucketResult xmlns=\"http://s3.amazonaws.com/doc/2006-03-01/\"><Name>mansi-pythonengineer-project</Name><Prefix></Prefix><Marker></Marker><MaxKeys>1000</MaxKeys><EncodingType>url</EncodingType><IsTruncated>false</IsTruncated><Contents><Key>xml-file-dataset.csv</Key><LastModified>2023-04-23T16:50:17.000Z</LastModified><ETag>&quot;7e1f13eba7ed76e18743cdb4099d5d64&quot;</ETag><Size>11064539</Size><Owner><ID>c4bfbf6ccbbb854930909456ed5445c19a066258a9831791aa183616588e968e</ID></Owner><StorageClass>STANDARD</StorageClass></Contents></ListBucketResult>'\n",
            "DEBUG:botocore.hooks:Event needs-retry.s3.ListObjects: calling handler <botocore.retryhandler.RetryHandler object at 0x7f9006bdd220>\n",
            "DEBUG:botocore.retryhandler:No retry needed.\n",
            "DEBUG:botocore.hooks:Event needs-retry.s3.ListObjects: calling handler <bound method S3RegionRedirectorv2.redirect_from_error of <botocore.utils.S3RegionRedirectorv2 object at 0x7f9006bdd2e0>>\n",
            "DEBUG:botocore.utils:S3 request was previously redirected, not redirecting.\n",
            "DEBUG:botocore.hooks:Event after-call.s3.ListObjects: calling handler <function decode_list_object at 0x7f90084afca0>\n",
            "DEBUG:boto3.resources.factory:Loading s3:ObjectSummary\n",
            "DEBUG:boto3.resources.model:Renaming ObjectSummary attribute key\n",
            "DEBUG:botocore.hooks:Event creating-resource-class.s3.ObjectSummary: calling handler <function lazy_call.<locals>._handler at 0x7f90076879d0>\n",
            "DEBUG:root:\n",
            "Uploaded CSV to AWS S3 bucket\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "File uploaded to S3 in AWS:  xml-file-dataset.csv\n"
          ]
        }
      ]
    }
  ]
}