.json and .csv exports fail to apply base_url #1091

simonw · 2020-11-12T23:45:16Z

Just tested with the latest Docker image, and it works pretty much everywhere! THANK YOU!

I did notice that if I try to export json or csv, the base is not applied. Not sure if I should reopen this issue or open a new one.

To see this, go here: https://corpora.tika.apache.org/datasette/corpora-metadata/REF_PARSE_EXCEPTION_TYPES

Click/hover over json or CSV and you'll see that the 'datasette' base is not included.

Originally posted by @tballison in #865 (comment)

simonw · 2020-11-12T23:52:50Z

Hmm... it's not just the .csv and .json export links - it's the column headings (which can be clicked to change the sort order) as well. Here's an extract of the HTML from that page:

<p class="export-links">This data as 
  <a href="/corpora-metadata/REF_PARSE_EXCEPTION_TYPES.json">json</a>, 
  <a href="/corpora-metadata/REF_PARSE_EXCEPTION_TYPES.csv?_size=max">CSV</a> (
  <a href="#export">advanced</a>)
</p>
<div class="table-wrapper">
  <table class="rows-and-columns">
    <thead>
      <tr>
        <th class="col-Link" scope="col" data-column="Link" data-column-type="" data-column-not-null="0" data-is-pk="0">
          Link
        </th>
        <th class="col-rowid" scope="col" data-column="rowid" data-column-type="integer" data-column-not-null="0" data-is-pk="1">
          <a href="/corpora-metadata/REF_PARSE_EXCEPTION_TYPES?_sort_desc=rowid" rel="nofollow">rowid&nbsp;▼</a>
        </th>
        <th class="col-PARSE_EXCEPTION_ID" scope="col" data-column="PARSE_EXCEPTION_ID" data-column-type="INTEGER" data-column-not-null="0" data-is-pk="0">
          <a href="/corpora-metadata/REF_PARSE_EXCEPTION_TYPES?_sort=PARSE_EXCEPTION_ID" rel="nofollow">PARSE_EXCEPTION_ID</a>
        </th>
        <th class="col-PARSE_EXCEPTION_DESCRIPTION" scope="col" data-column="PARSE_EXCEPTION_DESCRIPTION" data-column-type="VARCHAR(128)" data-column-not-null="0" data-is-pk="0">
          <a href="/corpora-metadata/REF_PARSE_EXCEPTION_TYPES?_sort=PARSE_EXCEPTION_DESCRIPTION" rel="nofollow">PARSE_EXCEPTION_DESCRIPTION</a>
        </th>
      </tr>
    </thead>
    <tbody>
      <tr>
        <td class="col-Link type-pk">
          <a href="/datasette/corpora-metadata/REF_PARSE_EXCEPTION_TYPES/1">1</a>
        </td>
        <td class="col-rowid type-int">1</td>
        <td class="col-PARSE_EXCEPTION_ID type-int">0</td>
        <td class="col-PARSE_EXCEPTION_DESCRIPTION type-str">RUNTIME</td>
      </tr>
      <tr>
        <td class="col-Link type-pk">
          <a href="/datasette/corpora-metadata/REF_PARSE_EXCEPTION_TYPES/2">2</a>
        </td>
        <td class="col-rowid type-int">2</td>
        <td class="col-PARSE_EXCEPTION_ID type-int">1</td>
        <td class="col-PARSE_EXCEPTION_DESCRIPTION type-str">ENCRYPTION</td>
      </tr>
      <tr>
        <td class="col-Link type-pk">
          <a href="/datasette/corpora-metadata/REF_PARSE_EXCEPTION_TYPES/3">3</a>
        </td>
        <td class="col-rowid type-int">3</td>
        <td class="col-PARSE_EXCEPTION_ID type-int">2</td>
        <td class="col-PARSE_EXCEPTION_DESCRIPTION type-str">ACCESS_PERMISSION</td>
      </tr>
      <tr>
        <td class="col-Link type-pk">
          <a href="/datasette/corpora-metadata/REF_PARSE_EXCEPTION_TYPES/4">4</a>
        </td>
        <td class="col-rowid type-int">4</td>
        <td class="col-PARSE_EXCEPTION_ID type-int">3</td>
        <td class="col-PARSE_EXCEPTION_DESCRIPTION type-str">UNSUPPORTED_VERSION</td>
      </tr>
    </tbody>
  </table>
</div>
<div id="export" class="advanced-export">
  <h3>Advanced export</h3>
  <p>JSON shape:
    <a href="/corpora-metadata/REF_PARSE_EXCEPTION_TYPES.json">default</a>,
    <a href="/corpora-metadata/REF_PARSE_EXCEPTION_TYPES.json?_shape=array">array</a>,
    <a href="/corpora-metadata/REF_PARSE_EXCEPTION_TYPES.json?_shape=array&amp;_nl=on">newline-delimited</a>
  </p>
  <form action="/corpora-metadata/REF_PARSE_EXCEPTION_TYPES.csv" method="get">
    <p>
      CSV options:            
      <label>
        <input type="checkbox" name="_dl"> download file
        </label>
        <input type="submit" value="Export CSV">
          <input type="hidden" name="_size" value="max">
          </p>
        </form>
      </div>

But here's something really weird - the links to the individual rows DO include the /datasette/ prefix:

<td class="col-Link type-pk">
    <a href="/datasette/corpora-metadata/REF_PARSE_EXCEPTION_TYPES/2">2</a>
</td>

The navigation bar on that page is correct too:

<p class="crumbs">
        <a href="/datasette/">home</a> /
        <a href="/datasette/corpora-metadata">corpora-metadata</a>
</p>

I've also been unable to replicate this in my own local environment, running datasette fixtures.db --config base_url:/datasette/.

simonw · 2020-11-12T23:56:23Z

@tballison is there any chance you're running any custom templates in that installation? I'm really confused as to why I can't replicate the bug.

simonw · 2020-11-12T23:59:34Z

The sort headers are generated by this template code:

datasette/datasette/templates/_table.html

Lines 11 to 15 in 5eb8e9b

    
           {% if column.name == sort %} 
        
               <a href="{{ path_with_replaced_args(request, {'_sort_desc': column.name, '_sort': None, '_next': None}) }}" rel="nofollow">{{ column.name }}&nbsp;▼</a> 
        
           {% else %} 
        
               <a href="{{ path_with_replaced_args(request, {'_sort': column.name, '_sort_desc': None, '_next': None}) }}" rel="nofollow">{{ column.name }}{% if column.name == sort_desc %}&nbsp;▲{% endif %}</a> 
        
           {% endif %}

The export links use this code:

datasette/datasette/templates/table.html

Line 134 in 5eb8e9b

    
           <p class="export-links">This data as {% for name, url in renderers.items() %}<a href="{{ url }}">{{ name }}</a>{{ ", " if not loop.last }}{% endfor %}{% if display_rows %}, <a href="{{ url_csv }}">CSV</a> (<a href="#export">advanced</a>){% endif %}</p>

datasette/datasette/templates/table.html

Lines 180 to 201 in 5eb8e9b

    
           <div id="export" class="advanced-export"> 
        
               <h3>Advanced export</h3> 
        
               <p>JSON shape: 
        
                   <a href="{{ renderers['json'] }}">default</a>, 
        
                   <a href="{{ append_querystring(renderers['json'], '_shape=array') }}">array</a>, 
        
                   <a href="{{ append_querystring(renderers['json'], '_shape=array&_nl=on') }}">newline-delimited</a>{% if primary_keys %}, 
        
                       <a href="{{ append_querystring(renderers['json'], '_shape=object') }}">object</a> 
        
                   {% endif %} 
        
               </p> 
        
               <form action="{{ url_csv_path }}" method="get"> 
        
                   <p> 
        
                       CSV options: 
        
                       <label><input type="checkbox" name="_dl"> download file</label> 
        
                       {% if expandable_columns %}<label><input type="checkbox" name="_labels" checked> expand labels</label>{% endif %} 
        
                       {% if next_url and config.allow_csv_stream %}<label><input type="checkbox" name="_stream"> stream all rows</label>{% endif %} 
        
                       <input type="submit" value="Export CSV"> 
        
                       {% for key, value in url_csv_hidden_args %} 
        
                           <input type="hidden" name="{{ key }}" value="{{ value }}"> 
        
                       {% endfor %} 
        
                   </p> 
        
               </form> 
        
           </div>

simonw · 2020-11-13T00:00:43Z

Here's where url_csv comes from:

datasette/datasette/views/base.py

Lines 542 to 545 in 11eb1e0

    
           url_csv = path_with_format( 
        
               request=request, format="csv", extra_qs=url_csv_args 
        
           ) 
        
           url_csv_path = url_csv.split("?")[0]

tballison · 2020-11-13T14:35:22Z

I'm starting this with docker like so:

docker run --name datasette -d -p 8001:8001 -v pwd:/mnt datasetteproject/datasette datasette -p 8001 -h 0.0.0.0 /mnt/file_profiles.db --config sql_time_limit_ms:120000 --config max_returned_rows:100000 --config base_url:/datasette/ --config cache_size_kb:50000

I'm not doing any templating or anything else custom.

Apropos of nothing, I swapped out a simpler db, so this query should now work:

https://corpora.tika.apache.org/datasette/file_profiles?sql=select%0D%0A++*%0D%0Afrom%0D%0A++file_profiles+fp%0D%0Alimit%0D%0A++10

tballison · 2020-11-13T14:40:56Z

My headers aren't clickable/sortable with custom sql, but I think that's by design.

In the default view, https://corpora.tika.apache.org/datasette/file_profiles/file_profiles, ah, y, now I see that the headers should be sortable, but you're right the base_url is not applied.

base_url works with "View and Edit SQL" and with "(advanced)"

As you point out, does not work with the export csv, json, other or with the "Next page" navigational button at the bottom.

simonw · 2020-11-14T16:46:52Z

@tballison could I see the section of your Apache config that configures the proxying to /datasette/?

simonw · 2020-11-16T19:05:08Z

I have a hunch that there may be some extra configuration in play here - could Apache itself be rewriting some of the links using mod_proxy_html?

tballison · 2020-11-17T15:48:58Z

I don't think we are, but I'll check with Maruan.

I think this is the relevant part of our config?

  Alias "/base/" "/usr/share/corpora/"
  <Directory "/usr/share/corpora/">
    Options +Indexes -Multiviews
    AllowOverride None
  </Directory>

  ProxyPreserveHost On

  ProxyPass /datasette http://0.0.0.0:8001
  ProxyPassReverse /datasette http://0.0.0.0:8001

</VirtualHost>

tballison · 2020-11-17T16:31:00Z

We're using mod_proxy.

tballison · 2020-12-09T14:26:05Z

Anything we can do to help debug this? Thank you, again!

simonw · 2020-12-09T19:19:54Z

Could you try removing the ProxyPassReverse /datasette http://0.0.0.0:8001 line?

My hunch is that ProxyPassReverse is rewriting some of the links in the HTML (or maybe in the HTTP headers) in a way that breaks things.

Normally you would need ProxyPassReverse to compensate for the underlying application being unable to rewrite its links - but Datasette's base_url setting causes Datasette to rewrite all of the links for you, so ProxyPassReverse should be unneccessary.

tballison · 2020-12-09T19:36:42Z

I don't think this fixes it:

grep -R datasette .
./sites-available/000-default.conf:        ProxyPass /datasette http://127.0.0.1:8001/
./sites-available/000-default.conf:        #ProxyPassReverse /datasette http://127.0.0.1:8001/
./sites-available/corpora-le-ssl.conf:  ProxyPass /datasette http://0.0.0.0:8001
./sites-available/corpora-le-ssl.conf:  #ProxyPassReverse /datasette http://0.0.0.0:8001
./sites-enabled/corpora-le-ssl.conf:  ProxyPass /datasette http://0.0.0.0:8001
./sites-enabled/corpora-le-ssl.conf:  #ProxyPassReverse /datasette http://0.0.0.0:8001

And I confirmed that I actually restarted the server. 🤣

https://corpora.tika.apache.org/datasette/file_profiles

tballison · 2020-12-09T19:53:18Z

I can't imagine this helps (esp. given your point about potential rewrites), but you can see that /datasette/ was correctly added to the sql form, but not to the "export-links"

simonw · 2020-12-09T20:17:54Z

OK that is really weird. I'll have another go at replicating this locally.

henry501 · 2021-01-07T22:27:19Z

I found this issue while troubleshooting the same behavior with an nginx reverse proxy. The solution was to make sure I set:

proxy_pass http://server:8001/baseurl/
instead of just:

proxy_pass http://server:8001
The custom SQL query and header links are now correct.

simonw · 2021-01-07T23:39:58Z

@tballison I think that's the solution! It looks like you need to use this in your config:

ProxyPass /datasette http://127.0.0.1:8001/datasette

Instead of this:

ProxyPass /datasette http://127.0.0.1:8001/

Give that a go and let me know if it fixes it.

simonw · 2021-01-07T23:42:50Z

@henry501 it looks like you spotted a bug in the documentation - I just addressed that, the fix is now live here: https://docs.datasette.io/en/latest/deploying.html#running-datasette-behind-a-proxy

tballison · 2021-01-11T23:06:10Z

+1

Yep! Fixes it. If I navigate to https://corpora.tika.apache.org/datasette, I get a 404 (database not found: datasette), but if I navigate to https://corpora.tika.apache.org/datasette/file_profiles/, everything WORKS!

Thank you!

simonw · 2021-01-11T23:12:46Z

Fantastic!

henry501 · 2021-01-12T06:55:08Z

Great, really happy I could help! Reverse proxies get tricky.

tballison · 2021-01-12T13:52:29Z

Y, thank you to both of you!

Refs #1091, #1145, #1151, #1156, #1157, #1158, #1166, #1170, #1178, #1182, #1184, #1185, #1186, #1187

Refs #509, #1091, #1150, #1151, #1166, #1167, #1178, #1181, #1182, #1184, #1185, #1186, #1187, #1194, #1198

simonw added the bug label Nov 12, 2020

simonw mentioned this issue Nov 12, 2020

base_url doesn't seem to work when adding criteria and clicking "apply" #865

Closed

simonw added a commit that referenced this issue Jan 7, 2021

Fixed bug in example nginx config, refs #1091

4c0995e

simonw closed this as completed Jan 9, 2021

simonw added a commit that referenced this issue Jan 19, 2021

Release 0.54a0

57f4d7b

Refs #1091, #1145, #1151, #1156, #1157, #1158, #1166, #1170, #1178, #1182, #1184, #1185, #1186, #1187

simonw added this to the Datasette 0.54 milestone Jan 24, 2021

This was referenced Jan 25, 2021

Release notes for Datasette 0.54 #1201

Closed

Release 0.54 #1206

Merged

simonw added a commit that referenced this issue Jan 25, 2021

Release 0.54

0b9ac1b

Refs #509, #1091, #1150, #1151, #1166, #1167, #1178, #1181, #1182, #1184, #1185, #1186, #1187, #1194, #1198

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

.json and .csv exports fail to apply base_url #1091

.json and .csv exports fail to apply base_url #1091

simonw commented Nov 12, 2020

simonw commented Nov 12, 2020 •

edited

Loading

simonw commented Nov 12, 2020

simonw commented Nov 12, 2020

simonw commented Nov 13, 2020

tballison commented Nov 13, 2020

tballison commented Nov 13, 2020

simonw commented Nov 14, 2020

simonw commented Nov 16, 2020

tballison commented Nov 17, 2020

tballison commented Nov 17, 2020

tballison commented Dec 9, 2020

simonw commented Dec 9, 2020 •

edited

Loading

tballison commented Dec 9, 2020 •

edited

Loading

tballison commented Dec 9, 2020 •

edited

Loading

simonw commented Dec 9, 2020

henry501 commented Jan 7, 2021

simonw commented Jan 7, 2021 •

edited

Loading

simonw commented Jan 7, 2021

tballison commented Jan 11, 2021

simonw commented Jan 11, 2021

henry501 commented Jan 12, 2021

tballison commented Jan 12, 2021

.json and .csv exports fail to apply base_url #1091

.json and .csv exports fail to apply base_url #1091

Comments

simonw commented Nov 12, 2020

simonw commented Nov 12, 2020 • edited Loading

simonw commented Nov 12, 2020

simonw commented Nov 12, 2020

simonw commented Nov 13, 2020

tballison commented Nov 13, 2020

tballison commented Nov 13, 2020

simonw commented Nov 14, 2020

simonw commented Nov 16, 2020

tballison commented Nov 17, 2020

tballison commented Nov 17, 2020

tballison commented Dec 9, 2020

simonw commented Dec 9, 2020 • edited Loading

tballison commented Dec 9, 2020 • edited Loading

tballison commented Dec 9, 2020 • edited Loading

simonw commented Dec 9, 2020

henry501 commented Jan 7, 2021

simonw commented Jan 7, 2021 • edited Loading

simonw commented Jan 7, 2021

tballison commented Jan 11, 2021

simonw commented Jan 11, 2021

henry501 commented Jan 12, 2021

tballison commented Jan 12, 2021

simonw commented Nov 12, 2020 •

edited

Loading

simonw commented Dec 9, 2020 •

edited

Loading

tballison commented Dec 9, 2020 •

edited

Loading

tballison commented Dec 9, 2020 •

edited

Loading

simonw commented Jan 7, 2021 •

edited

Loading