Support for multiple syntaxes in a single file

[DrHyde]

I have a bunch of files like this. They're incoming web requests that I've captured to a file:

POST /foo/bar HTTP/1.1
X-Forwarded-For: 54.187.216.72
Content-Length: 3077
Cache-Control: no-cache
X-Forwarded-Proto: https
Content-Type: application/json; charset=utf-8
Host: 7261363d4b2c.ngrok.io
User-Agent: Stripe/1.0 (+https://stripe.com/docs/webhooks)
Accept: */*; q=0.5, application/xml
Accept-Encoding: gzip

{
  "id": "blahblahblahblah",
  "object": "event",
  "api_version": "2020-03-02",
  "created": 1626790174,
  "data": {
  ...

and it would be Dead Handy to be able to highlight the two sections independently, both the HTTP headers and the JSON body. Obviously this would require a manually specified -l and probably also telling bat what the boundary between the two languages is, something like this perhaps:

bat -l http+json --boundary='\r\n\r\n' filename

[keith-hall]

You may find that highlighting it as YAML will give reasonable results. I personally don't think we will complicate the parser or public API/CLI by adding an argument for this use case when we already support custom syntax definitions. I will see if I can make one for you when I get some spare time. Probably I can tweak some prior work: kylebebak/Requester@b6db9c0

[keith-hall]

I have created a custom syntax definition for this - it currently supports highlighting of json, xml and html request/response bodies. I wonder if we would want to include this in `bat` by default - I'm guessing not because we normally have a 15k package control download requirement, and this seems fairly niche, though there are some Sublime Text packages for making HTTP requests which do have over 15k downloads, especially collectively. Well, I may need to create a separate git repository for it if I decide I want to maintain it separately. Anyway, to use it, expand this "summary/details" to copy/paste the syntax definition into a `.sublime-syntax` file, and follow the instructions in the readme to make it available to bat.

%YAML 1.2
---
# See http://www.sublimetext.com/docs/3/syntax.html
scope: text.http-request-response
name: HTTP Request and Response
variables:
  content_type_sep: (?=;|$)
contexts:
  main:
    - match: (?:GET|HEAD|POST|PUT|DELETE)
      scope: meta.http-verb.http-request-response keyword.operator.word.http-request-response
    - match: \bHTTP/\d\.\d\b
      scope: constant.language.protocol-version.http-request-response
      push: maybe_response
    - include: url_escape_char
    - match: /
      scope: meta.url punctuation.separator.path.directory.http-request-response
    - match: \?
      scope: punctuation.separator.sequence.querystring.http-request-response
      push: querystring
    - match: '[^&\s?/]+'
      scope: meta.url.http-request-response
    - match: $
      set: http-headers

  maybe_response:
    - match: $
      pop: true
    - match: (?=[123])
      set: [info, response_code]
    - match: (?=4)
      set: [warning, response_code]
    - match: (?=5)
      set: [error, response_code]
    - include: response_code

  warning:
    - meta_scope: markup.warning.log
    - match: ''
      pop: true

  error:
    - meta_scope: markup.error.log
    - match: ''
      pop: true

  info:
    - meta_scope: markup.info.log
    - match: ''
      pop: true

  response_code:
    - match: \d+(\.\d+)?
      scope: meta.http-status-code.http-request-response meta.number.integer.decimal.http-request-response constant.numeric.value.http-request-response
    - match: \S.*$
      scope: meta.http-response-status-text.http-request-response string.unquoted.http-request-response
      pop: true

  querystring:
    - meta_content_scope: meta.querystring.http-request-response
    - match: '[^&\s=]+'
      scope: variable.parameter.http-request-response
    - match: \&
      scope: keyword.operator.querystring.and.http-request-response
    - match: =
      scope: keyword.operator.querystring.equals.http-request-response
      push:
        - match: '[^&\s%]+'
          scope: string.unquoted.value.http-request-response
        - include: url_escape_char
        - match: ''
          pop: true
    - match: ''
      pop: true

  url_escape_char:
    - match: (%)\h{2}
      scope: meta.url constant.character.escape.url.html
      captures:
        1: punctuation.definition.escape.html

  content-type:
    - meta_content_scope: meta.headers.http-request-response
    - match: (application/json|text/json){{content_type_sep}}
      set: [content-type-json, header-value]
    - match: (application/xml|text/xml){{content_type_sep}}
      set: [content-type-xml, header-value]
    - match: (application/xhtml+xml|text/html){{content_type_sep}}
      set: [content-type-html, header-value]

  http-headers:
    - meta_content_scope: meta.headers.http-request-response
    - match: (?i)^(Content-Type)(:)\s*
      captures:
        1: keyword.other.http-header.key.http-request-response
        2: punctuation.separator.key-value.http-request-response
      set: content-type
    - match: ([^:]+)(:)\s*(\d+)$
      captures:
        1: keyword.other.http-header.key.http-request-response
        2: punctuation.separator.key-value.http-request-response
        3: meta.number.integer.decimal.http-request-response constant.numeric.value.http-request-response
    - match: ([^:]+)(:)\s*
      captures:
        1: keyword.other.http-header.key.http-request-response
        2: punctuation.separator.key-value.http-request-response
      push: header-value
    - match: ^$\n
      set: scope:text.plain

  header-value:
    - meta_scope: string.unquoted.http-header.value.http-request-response
      pop: true
    - match: '[;,]'
      scope: punctuation.separator.mapping.key-value.http-request-response
    - match: \b([-\w]+)(=)
      captures:
        1: variable.parameter.http-request-response
        2: keyword.operator.assignment.http-request-response
      push: possible-number
    - match: $
      pop: true

  possible-number:
    - match: \b\d+(?!\.)\b
      scope: meta.number.integer.decimal.http-request-response constant.numeric.value.http-request-response
      pop: true
    - match: ''
      pop: true

  content-type-json:
    - meta_content_scope: meta.headers.http-request-response
    - match: ^$\n
      set: [response-body, embed_json]
    - include: http-headers

  content-type-xml:
    - meta_content_scope: meta.headers.http-request-response
    - match: ^$\n
      set: [response-body, embed_xml]
    - include: http-headers

  content-type-html:
    - meta_content_scope: meta.headers.http-request-response
    - match: ^$\n
      set: [response-body, embed_html]
    - include: http-headers

  response-body:
    - meta_content_scope: meta.response-body.http-request-response

  embed_json:
    - match: ''
      embed: scope:source.json
      escape: (?!)

  embed_xml:
    - match: ''
      embed: scope:text.xml
      escape: (?!)

  embed_html:
    - match: ''
      embed: scope:text.html.basic
      escape: (?!)

[sharkdp]

@keith-hall Awesome, I love it. I think the downloads policy is not always a great deciding factor due to the difference of tools: Maybe there are a lot of people who want to view HTTP responses (using bat), but probably only very few people who want to edit HTTP responses (in Sublime Text).

Having this in a separate git repository would be great, but if you don't want to, I'm fine with maintaining it in the bat repository.

Edit: I just now realized that this can also be used for HTTP requests, where it might make a bit more sense to edit in Sublime Text... anyway.

[DrHyde]

That's wonderful, thank you!

One minor issue - it doesn't show up in the output from bat -L but works just fine when I bat -l 'HTTP Request and Response' ....

[keith-hall]

One minor issue - it doesn't show up in the output from bat -L but works just fine when I bat -l 'HTTP Request and Response' ....

Thanks for letting me know. It seems that it will only show up if it has some file extensions associated with it. We could add http as one to get it to appear in the list.

[DrHyde]

Yep, that fixes it.

[keith-hall]

I have decided that I am interested in maintaining this syntax, it's quite fun to work on :)

I have created a repository for it here: https://github.com/keith-hall/http-request-response-syntax

I will add a PR to bat soon ™️ to add it as a submodule for syntax assets unless there are any objections :)