The -closing include should be part of all-years.journal, not yyyy-include.journal

[lestephane]

Description

In the situation where one has yyyy-opening.journal and yyyy-closing.journal for a specific year,
running one of the hledger built-in financial reports on one specific year, the user will be faced with empty reports

$ hledger bal 2017-include.journal 
--------------------
                   0

because hledger-flow adds an include to yyyy-closing.journal in the yyyy-include.journal, which effectively brings all the counters back to zero.

Expectation

I would expect hledger <any report> -f yyyy-include.journal to just work

Solution

The yyyy-opening.journal include stays in yyyy-include.journal, but yyyy-closing.journal include needs to move to the all-years.journal file, as evidenced by the all.journal in the full-fledged-hledger project. This ensures that reports work on a per-year yyyy-include.journal basis, and that reports also work on an aggregated all-years.journal basis, with the important caveat that the very last yyyy-closing.journal must not be part of the all-years.journal (even if one such file is present on the file system), otherwise we encounter the same problem of empty reports on an aggegated all-years.journal basis.

I hope that makes sense, it's a bit tricky to explain.

Version and Runtime Information

This include behaviour has been present in hledger-flow as far back as I can remember using it.

Additional context

I only found this bug because I'm trying to use hledger-flow import for my imports, while using full-fledged-hledger's export.hs for my reports, and I got confused that I was getting empty reports.

[apauley]

Thanks for double checking closing journals, this is something I haven't spent a lot of thinking time on yet - initially the whole concept of closing accounts seemed foreign to my non-accounting brain. But I'm warming up to it :-)

I especially would like to hear someone else's ideas on if it possible/feasible to use hledger close to automate some of this.

OK, back to your issue. While trying to replicate it I noticed that your initial command above does not have a -f:

$ hledger bal 2017-include.journal 
--------------------
                   0

My best guess on how hledger interprets this: it is trying to produce a balance for an account with 2017-include.journal in the name (eg expenses:2017-include.journal instead of expenses:Groceries).
And then an answer of 0 makes sense.

That's not to say there isn't some room for improvement in hledger-flow.
But please try again with the -f and tell me what you find. Or tell me if I've got my interpretation wrong.

I tried running some reports using -f on my yearly includes. Some work and some fail.
Those that fail show balance assertion errors. I think it is because I haven't spent the time to create closing account journals for December 31 in year x, with a matching opening journal for January 1 in year x+1.

I don't fully understand the proposed solution. I need to try it out myself I guess. But the proposed include logic sounds complicated. I hope we can achieve the end result (having working yearly include files) in a way that is simple to explain (and simple to implement). Especially since a change like this will have to be explained in the docs.

[lestephane]

uh you're right, too much bookkeeping yesterday.

I'll try to dig the exact command line that was generating an empty report and re-open with that.

[lestephane]

FYI this is how i implement closing the year...

#!/usr/bin/env bash

source common.sh

my_dir=$(readlink -e $(dirname "$0"))
top_dir="$(dirname "${my_dir}")"

usage() {
cat <<EOF 1>&2
Usage: $0 YEAR
EOF
exit 1
}

[ $# -eq 1 ] || usage
[ $1 -gt 2000 ] || usage

CLOSEYEAR="$1"
OPENYEAR=$((CLOSEYEAR + 1))

"${my_dir}/clear-year.sh" "${CLOSEYEAR}"

echo "closing year ${CLOSEYEAR}..." 1>&2

OPENING_FILE="${top_dir}/import/${OPENYEAR}-opening.journal"
CLOSING_FILE="${top_dir}/import/${CLOSEYEAR}-closing.journal"

../hledger close \
  --auto \
  -e "${OPENYEAR}" \
  -f "${CLOSEYEAR}.journal" \
  'Assets|liabilities|Transfers:' |
    awk \
    -v cf="${CLOSING_FILE}" \
    -v of="${OPENING_FILE}" '
      /closing balances/ { f = cf }
      /opening balances/ { f = of }
      { print > f }
    '

• I include Transfers: because that is money that might be in transit into and out of an assets account.
• clear-year just deletes the existing -opening and -closing journal

[lestephane]

Here is more information:

$ hledger bs -f includes.journal -f import/2017-include.journal 
Balance Sheet 2017/12/31

             || 2017/12/31 
=============++============
 Assets      ||            
-------------++------------
-------------++------------
             ||          0 
=============++============
 Liabilities ||            
-------------++------------
-------------++------------
             ||          0 
=============++============
 Net:        ||          0 

• import/2017-include.journalis the yearly include generated by hledger-flow import, which will include the -opening and -closing journals (if found to be present).

• includes.journal contains

  • account directives for my account hierarchy (they are not following the standard naming)

  $ cat include-accounts.journal | grep -E "type:(ASSET|LIABILITY)"
  account Personal:Assets                     ; type:ASSET
  account Personal:Liabilities                ; type:LIABILITY
  account Business-DE:Assets                 ; type:ASSET
  account Business-DE:Liabilities            ; type:LIABILITY
  account Business-CY:Assets                 ; type:ASSET
  account Business-CY:Liabilities            ; type:LIABILITY

  • historical price (P) directives

To test my theory that the closing journal inclusion is causing the empty report, i excluded the closing balances as follows, and suddenly the balance sheet report 'works', because the balance sheet has data (the closing balances were bringing everything down to zero, causing the empty report):

$ hledger bs -f includes.journal -f import/2017-include.journal "not:desc:closing balances" --depth 2 -XEUR cur:EUR | tr '[:digit:]' 'X'
Balance Sheet XXXX/XX/XX, current value

               ||      XXXX/XX/XX 
===============++=================
 Assets        ||                 
---------------++-----------------
 Business-DE   || -X.XXX,XXXX EUR 
   Assets      || -X.XXX,XXXX EUR 
 Personal      ||  X.XXX,XXXX EUR 
   Assets      ||  X.XXX,XXXX EUR 
---------------++-----------------
               ||  X.XXX,XXXX EUR 
===============++=================
 Liabilities   ||                 
---------------++-----------------
 Business-DE   ||     XX,XXXX EUR 
   Liabilities ||     XX,XXXX EUR 
---------------++-----------------
               ||     XX,XXXX EUR 
===============++=================
 Net:          ||  X.XXX,XXXX EUR 

Does my issue make more sense now?

[apauley]

Yes, I see the problem, I'll try to find out some more good practices around closing balances before implementing a solution.

[apauley]

The issue is discussed in the hledger documentation here:
https://hledger.org/hledger.html#close-usage

When playing around with bal and balancesheet, I get good results when excluding the opening and closing balances transactions with a bs:

$ hledger bs --yearly not:desc:"(opening|closing) balances"

It seems that the proposed solution where we craft includes so that only the all-years.journal have closing balances will only result in expected balances when using each individual yearly account as the journal - the zeroed out balances will still be present when using the default all-years.journal without any special exclusion tricks.

Possible approaches to deal with this:

• Use the existing include files - needs specially crafted reports when closing asset balances and usingall-years.journal. The individual yearly includes should work, also with specially crafted reports.
• Implement specially crafted include files so that individual yearly journals works without having to exclude opening/closing in reports. Still need specially crafted reports when using all-years.journal.
• Don't close asset balances - only all-years.journal works. No need for specially crafted reports.
• ???

Since this issue is generally present in hledger, and since the closing balances in hledger-flow generally follow the hledger docs, I don't think this is a bug.

The specific line in the docs, for reference:

save the closing transaction as last entry of the old file, and the opening transaction as the first entry of the new file

(the files mentioned are yearly include files)

And even though it isn't a bug, we could still implement something to make this aspect of hledger-flow better.

We should definitely point our closing balances docs to the hledger docs, and suggest some good options.

[apauley]

Here is a more detailed examination with some simple example journals:
https://github.com/apauley/hledger-acorns

[lestephane]

The approach I now take is as follows

• Have a close-years.sh script that, for each year, creates the opening and closing journal next to the hledger-flow yyyy-include.journal

readonly opening_file="${top_dir}/import/${open_year}-opening.journal"
readonly closing_file="${top_dir}/import/${close_year}-closing.journal"

hledger close \
  --auto \
  -e "${open_year}" \
  -f "${top_dir}/includes.journal" \
  -f "${top_dir}/import/${close_year}-include.journal" \
      'Assets|Liabilities|Transfers' |
        awk \
        -v cf="${closing_file}" \
        -v of="${opening_file}" '
          /closing balances/ { f = cf }
          /opening balances/ { f = of }
          { print > f }
        '

sed -e "s#\!include #\!include ${top_dir}/import/#g" \
    -e "s/.*closing.journal/#\0/g" \
    "${top_dir}/import/${close_year}-include.journal" >> "${my_dir}/${close_year}-includes.journal"

echo '!include' "${top_dir}/includes.journal" > "${my_dir}/${close_year}.journal"
hledger print -f "${my_dir}/${close_year}-includes.journal" >> "${my_dir}/${close_year}.journal"

• The "${my_dir}/${close_year}.journal"file is the yearly file used by full-fledged-hledger's export.hs file. export.hs now works (there is data for the balance sheet report), because the closing journal include has been omitted.

$ cat 2018-includes.journal
### Generated by hledger-flow - DO NOT EDIT ###

!include /scrubbed/import/2018-opening.journal
!include /scrubbed/import/x/2018-include.journal
!include /scrubbed/import/y/2018-include.journal
!include /scrubbed/import/z/2018-include.journal
#!include /scrubbed/import/2018-closing.journal

So now the 2017-include.journal from hledger-flow and the 2017.journal for full-fledged-hledger can live their own separate lives, fulfilling their own separate purposes.

[apauley]

I've updated the README section on closing balances to point to this issue:
https://github.com/apauley/hledger-flow#a-note-of-caution-regarding-closing-balances