'files_upload_v2' doesn't always return accurate `files.info` data

[eddyg]

Greetings.

I've been experimenting with files_upload_v2 and ran into situations where the response returned by the SDK doesn't contain the info necessary to determine the details about the shared file.

Here is an example exchange, with (redacted) debug logging enabled:

2022-10-15 22:13:18.278 [D] slack_sdk.web.async_base_client (async_internal_utils.py:_request_with_session:99)
                        Sending a request - url: POST https://www.slack.com/api/files.getUploadURLExternal, params:
                        {'filename': 'iyunejoslx.diff', 'length': 1139, 'snippet_type': 'diff', 'team_id': 'T043...'}, ...

2022-10-15 22:13:18.686 [D] slack_sdk.web.async_base_client (async_internal_utils.py:_request_with_session:143)
                        Received the following response - status: 200 ... body: {'ok': True, 'upload_url':
                        'https://files.slack.com/upload/v1/CwAB...jAA', 'file_id': 'F047...'}

2022-10-15 22:13:19.134 [D] slack_sdk.web.async_base_client (internal_utils.py:_upload_file_via_v2_url:384)
                         ('Received the following response - ', "status: 200 ... body: OK - 1139)

2022-10-15 22:13:19.135 [D] slack_sdk.web.async_base_client (async_internal_utils.py:_request_with_session:99)
                        Sending a request - url: POST https://www.slack.com/api/files.completeUploadExternal, params:
                        {'channel': 'C043..., 'files': '[{"id": "F047...", "title": "iyunejoslx.diff"}]',
                        'initial_comment': 'Here's the diff', 'team_id': 'T043...'}

2022-10-15 22:13:19.316 [D] slack_sdk.web.async_base_client (async_internal_utils.py:_request_with_session:143)
                        Received the following response - status: 200 ... body: {'ok': True, 'files': [{'id': 'F047...',
                        'title': 'iyunejoslx.diff'}]}

2022-10-15 22:13:19.317 [D] slack_sdk.web.async_base_client (async_internal_utils.py:_request_with_session:99)
                        Sending a request - url: GET https://www.slack.com/api/files.info, params: {'file': 'F047...',
                        'team_id': 'T043...'} ...

2022-10-15 22:13:19.664 [D] slack_sdk.web.async_base_client (async_internal_utils.py:_request_with_session:143)
                        Received the following response - status: 200 ... body: {'ok': True, 'file': {'id': 'F047...',
                        'created': 1665886398, 'timestamp': 1665886398, 'name': 'iyunejoslx.diff', 'title': 'iyunejoslx.diff',
                        'mimetype': '', 'filetype': '', 'pretty_type': '', 'user': 'U044...', 'user_team': 'T043...',
                        'editable': False, 'size': 1139, 'mode': 'hosted', 'is_external': False, 'external_type': '',
                        'is_public': False, 'public_url_shared': False, 'display_as_bot': False, 'username': '',
                        'url_private': 'https://files.slack.com/files-pri/T043...-F047.../iyunejoslx.diff',
                        'url_private_download': 'https://files.slack.com/files-pri/T043...-F047.../download/iyunejoslx.diff',
                        'media_display_type': 'unknown', 'permalink': 'https://xyzzy.slack.com/files/U044.../F047.../iyunejoslx.diff',
                        'permalink_public': 'https://slack-files.com/T043...-F047...06e', 'comments_count': 0, 
                        'is_starred': False, 'shares': {}, 'channels': [], 'groups': [], 'ims': [], 'has_rich_preview': False,
                        'file_access': 'visible'}, 'comments': [], 'response_metadata': {'next_cursor': ''}}

The file was successfully sent to the channel.

However, notice the response data of the last API call in the files_upload_v2 sequence (files.info). A lot of the fields are empty, in particular things like filetype and fields like shares and channels, while mode says "hosted" (but later changes to "snippet"), etc. Sometimes the response returned by files_upload_v2 does have valid data, but it seems like it can't be relied on.

Just wondering if this is a known limitation?

If I call files.info separately later on, I do get proper, accurate info about the file, but it would be nice to get this as the response to files_upload_v2:

{'ok': True, 'content': '...', 'is_truncated': False, 'content_highlight_html': '...', 'content_highlight_css': '...',
'file': {'id': 'F047...', 'created': 1665886398, 'timestamp': 1665886398, 'name': 'iyunejoslx.diff',
'title': 'iyunejoslx.diff', 'mimetype': 'text/plain', 'filetype': 'diff', 'pretty_type': 'Diff', 'user': 'U044...',
'user_team': 'T043...', 'editable': True, 'size': 1139, 'mode': 'snippet', 'is_external': False, 'external_type': '',
'is_public': True, 'public_url_shared': False, 'display_as_bot': False, 'username': '', 'url_private': '...',
'url_private_download': '...', 'permalink': '...', 'permalink_public': '...', 'edit_link': '...', 'preview': '...',
'preview_highlight': '...', 'lines': 20, 'lines_more': 15, 'preview_is_truncated': True, 'comments_count': 0,
'is_starred': False, 'shares': {'public': {'C043...': [{'reply_users': [], 'reply_users_count': 0, 'reply_count': 0,
'ts': '1665886407.800859', 'channel_name': '...', 'team_id': 'T043...', 'share_user_id': 'U044...'}]}},
'channels': ['C043...'], 'groups': [], 'ims': [], 'has_rich_preview': False, 'file_access': 'visible'}, 'comments': [],
'response_metadata': {'next_cursor': ''}}

Thanks for any insight!

Reproducible in:

slack-bolt==1.15.1
slack-sdk==3.19.1

[seratch]

Hi @eddyg, thanks for writing in (as always)!

Yes, this is a known limitation of the new way. There can be a visible delay until the shares in a file's metadata is properly updated (this means posting a message in the channel can be delayed too).

If the accuracy of the file metadata is critical for your use case, we recommend polling either files.info or conversations.history API until the file-sharing message is posted in the channel that you've specified. See also: #1165 (comment)

I hope this was helpful to you!

[eddyg]

Your comments are always helpful, @seratch! 😉

Do you know if listening for file_created events and/or message file_share sub-type events would be a usable as a "signal" to call files.info when the event indicates it was uploaded by the app? (I had planned on trying it myself, but have gotten busy with other things.)

[seratch]

@eddyg The file_share subtyped message events also work but I think that utilizing file_shared events would be straight-forward for this purpose. I personally feel that waiting for Events API request arrival can make the code complex. You have to store the tentative state of the file uploading operations somewhere such as database and proceed with the state in a different app.event listener (if you use bolt-python).

I know the issue is not resolved but since we don't have anything further to share here, let me close this issue now. If you have any follow-up comments, please feel free to write in!

[eddyg]

@seratch Based on the above, would you consider merging a PR that made calling _attach_full_file_metadata{_async} at the end of of files_upload_v2 optional (via a boolean parameter that defaults to True)? This would allow skipping the files.info API call for each uploaded file (since it isn't guaranteed to "complete") in situations where this information is desired and the user will have to get the information in a different way.

I'm thinking that when the full metadata isn't requested, the return value to files_upload_v2 could simply be the response from files.completeUploadExternal.

[seratch]

@eddyg Thanks for your comment. I still believe that providing other properties such as permalink is useful for many use cases but I agree that some developers like you would like to skip fetching file metadata that lacks valid shares property. Here is my pull request adding the option: #1282 Please feel free to add comments if you have any.