converting html tables to markdown creates unusable tables

[osmanburucu-ibm]

Describe the bug
cnverting from a wordpress based html content to markdown the tables are broken

To Reproduce
Based on this page: https://www.urbancode.com/plugindoc/accurev-scm#tab-settings
original HTML part which is converted:

Steps and Roles

Process steps in the AccuRev plug-in

• Import Version

Import Version

Creates a new component version and imports artifacts from AccuRev.

This step has no input properties.

Roles in the AccuRev plug-in

The plug-in adds these roles automatically to resources. You cannot add these roles
manually.

• AccuRevComponentProperties
• AccuRevImportProperties

AccuRevComponentProperties

Properties for the AccuRevComponentProperties role

Name	Type	Description
AccuRev Executable Path	String	Full path to the AccuRev executable on the agents file system.
AccuRev URL	String	Full URL for the AccuRev server. Hostname and port can be identified by running the `accurev info` command locally.
Depot	String	Specify a Depot to retrieve artifacts from.
Excludes	String	Specify a list of files to exclude during artifact import.
Import Issues	Boolean	Select to import latest Issue change set artifacts on automatic import. Artifacts will still be retrieved if the Issue number is specified on manual import. Only valid for a non-Workspace Stream.
Import Promote Transactions	Boolean	Select to import latest Promote Transaction artifacts on automatic import. Artifacts will still be retrieved if the Transaction number is specified on manual import. Only valid for a non-Workspace Stream.
Import Snapshots	Boolean	Select to import latest Snapshot artifacts on automatic import. Artifacts will still be retrieved if the Snapshot name is specified on manual import. Only valid for a non-Workspace Stream.
Includes	String	Specify a list of files to include during artifact import.
Password	Password	Specify the password of the user with permissions to access the AccuRev Depot and Stream.
Preserve Execute Permissions	Boolean	Select to save file execute permissions with files.
Stream	String	Specify a Stream or Workspace to retrieve artifacts from. If empty, the Depot name will be used. If a Workspace is specified, the checkboxes and manual import values for Promote Transactions, Issues, and Snapshot will be ignored. A version will be created based on the runtime timestamp and file contents of the Workspace.
Username	String	Specify a user with permissions to access the AccuRev Depot and Stream.

AccuRevImportProperties

Properties for the AccuRevImportProperties role

Name	Type	Description
Issue Numbers	String	Specify a Issue Number to import. Only valid for a non-Workspace Stream. Separate multiple Issue Numbers with commas. Example: 5, 16, 27
Stream Snapshots	String	Specify a Stream Snapshot to import. Only valid for a non-Workspace Stream. Separate multiple Snapshots with a comma. Example: snap-stream1, snap-stream2
Transaction Numbers	String	Specify a Promote Transaction Number to import. Only valid for a non-Workspace Stream. Separate multiple Transaction Numbers with commas. Example: 80, 120, 184
Version Name Tag	String	Specify a tag to append to the versions name.

I pre-process the html and remove the following:

Properties for the AccuRevComponentProperties role
------- This converts to this:

Process steps in the AccuRev plug-in

• Import
  Version

Import Version

Creates a new component version and imports artifacts from AccuRev.

This step has no input properties.

Roles in the AccuRev plug-in

The plug-in adds
these roles automatically to resources. You cannot add these roles manually.

AccuRevComponentProperties
*
AccuRevImportProperties

AccuRevComponentProperties

| Name | Type |
Description |
| --- | --- | --- |
| AccuRev Executable Path | String | Full path to the AccuRev executable on the agents
file system. |
| AccuRev URL | String | Full URL for the AccuRev server. Hostname and port can be identified by running
the accurev info command locally. |
| Depot | String | Specify a Depot to retrieve artifacts from. |
| Excludes |
String | Specify a list of files to exclude during artifact import. |
| Import Issues | Boolean | Select to import
latest Issue change set artifacts on automatic import. Artifacts will still be retrieved if the Issue number is
specified on manual import. Only valid for a non-Workspace Stream. |
| Import Promote Transactions | Boolean | Select to
import latest Promote Transaction artifacts on automatic import. Artifacts will still be retrieved if the Transaction
number is specified on manual import. Only valid for a non-Workspace Stream. |
| Import Snapshots | Boolean | Select to
import latest Snapshot artifacts on automatic import. Artifacts will still be retrieved if the Snapshot name is
specified on manual import. Only valid for a non-Workspace Stream. |
| Includes | String | Specify a list of files to
include during artifact import. |
| Password | Password | Specify the password of the user with permissions to access
the AccuRev Depot and Stream. |
| Preserve Execute Permissions | Boolean | Select to save file execute permissions with
files. |
| Stream | String | Specify a Stream or Workspace to retrieve artifacts from. If empty, the Depot name will be
used. If a Workspace is specified, the checkboxes and manual import values for Promote Transactions, Issues, and
Snapshot will be ignored. A version will be created based on the runtime timestamp and file contents of the Workspace. |

| Username | String | Specify a user with permissions to access the AccuRev Depot and Stream. |

AccuRevImportProperties

Name	Type	Description
Issue Numbers	String	Specify a Issue
Number to import. Only valid for a non-Workspace Stream. Separate multiple Issue Numbers with commas. Example: 5, 16,
27
Stream Snapshots	String	Specify a Stream Snapshot to import. Only valid for a non-Workspace Stream. Separate
multiple Snapshots with a comma. Example: snap-stream1, snap-stream2
Transaction Numbers	String	Specify a
Promote Transaction Number to import. Only valid for a non-Workspace Stream. Separate multiple Transaction Numbers with
commas. Example: 80, 120, 184
Version Name Tag	String	Specify a tag to append to the versions name.

[didix21]

So this output:

| Name | Type |
Description |
| --- | --- | --- |
| AccuRev Executable Path | String | Full path to the AccuRev executable on the agents
file system. |
| AccuRev URL | String | Full URL for the AccuRev server. Hostname and port can be identified by running
the accurev info command locally. |
| Depot | String | Specify a Depot to retrieve artifacts from. |
| Excludes |
String | Specify a list of files to exclude during artifact import. |
| Import Issues | Boolean | Select to import
latest Issue change set artifacts on automatic import. Artifacts will still be retrieved if the Issue number is
specified on manual import. Only valid for a non-Workspace Stream. |
| Import Promote Transactions | Boolean | Select to
import latest Promote Transaction artifacts on automatic import. Artifacts will still be retrieved if the Transaction
number is specified on manual import. Only valid for a non-Workspace Stream. |
| Import Snapshots | Boolean | Select to
import latest Snapshot artifacts on automatic import. Artifacts will still be retrieved if the Snapshot name is
specified on manual import. Only valid for a non-Workspace Stream. |
| Includes | String | Specify a list of files to
include during artifact import. |
| Password | Password | Specify the password of the user with permissions to access
the AccuRev Depot and Stream. |
| Preserve Execute Permissions | Boolean | Select to save file execute permissions with
files. |
| Stream | String | Specify a Stream or Workspace to retrieve artifacts from. If empty, the Depot name will be
used. If a Workspace is specified, the checkboxes and manual import values for Promote Transactions, Issues, and
Snapshot will be ignored. A version will be created based on the runtime timestamp and file contents of the Workspace. |

| Username | String | Specify a user with permissions to access the AccuRev Depot and Stream. |

is what you find on the markdown file once is generated with mdutils?

[osmanburucu-ibm]

Yes this is correct. I can provide you with the raw data if you like. The strange breaks in the line structure can be also seen on non-table paragraphs but as the markdown file is rendered then in GH Pages only the tables are seen as a mess.
here are two examplares of the raw data :

• https://raw.githubusercontent.com/UrbanCode/IBM-UCx-PLUGIN-DOCS/main/docs/UCD/AccuRevSourceConfig/steps%20and%20roles.md
• https://raw.githubusercontent.com/UrbanCode/IBM-UCx-PLUGIN-DOCS/main/docs/UCD/AccuRevSourceConfig/overview.md

[didix21]

The problem probably is due to wrap_width=120 from method new_paragraph(self, text='', bold_italics_code='', color='black', align='', wrap_width=120)`. Seems this is generating a lot of problems so maybe I would disable by default.

[osmanburucu-ibm]

Just another issue regarding tables:
if a caption style is in the html table, it also breaks the output. At the moment I delete the caption line from the html response before i convert it. Do not have the MD output available at the moment. It adds a Repository properties right before the table without any seperation. Maybe you could also check that? THANK YOU for your response!

Here is an sample of the HTML table->

Repository properties

Name	Type	Description
Command Path	String	The location of the AccuRev command executable file, if the location is not specified on the system PATH statement.
Depot Name	String	The name or ID of the AccuRev depot. The depot is the repository on the server for all related source code. The value for this property can be used as part of the source config stream name with ${source/repo/depotName}. If set, this will be recorded with any source changes for use by the AccuWork plug-in.
Password	Password	The password to be used to connect to the AccuRev server.
Password Script	String	The property to use for the password to connect to the AccuRev server. For example: ${p:AccuRevPassword}. If this property is specified, do not specify a value for the Password property.
Repository Host	String	The location of the AccuRev server. Specify in the format: host:port.
User Name	String	The user name to be used to connect to the AccuRev server.

Integration properties

Name	Type	Description
Directory Offset	String	The directory offset of the working directory for the current job. Use a period (.) to clone into the current working directory.
Exclude Filters	String	A list of user names to exclude.
File Filters	String	Restrict the changelog to files matching these patterns. Start each lines with a plus sign (+) to include or minus sign (- )to exclude. You can use the following wildcards in the pattern: *, **, and ?.
Populate Locations	String	A list of depot-relative path for the files that you want to populate the workspace or stream. The pop command is use to restore the listed files. Only the specified files are updated, without building the associated directory tree.
Repository		The AccuRev repository to be used.
Revision	String	The version that is to be checked out.
Stream	String	The stream that is to be checked out.

[didix21]

Which methods are you calling? new_header and new_paragraph?

[osmanburucu-ibm]

I first "cleanse the content" so that the tables look as good as possible and download all images in the content to local file system and link the local name instead of the href from original content.

contents = "".join(str(item) for item in soupcontent.contents)
logger1.info(f"contents-original={contents}")
contents = re.sub(r"<caption style=.*</caption>", '', contents)
contents = contents.replace("<br/>\n", " ")
contents = contents.replace("<br/>", " ")    
contents = contents.replace("\n </td>", "</td>")  
contents = contents.replace("\n</td>", "</td>")
contents = contents.replace("${", "``${")
contents = contents.replace("}", "}``")
list_of_src, list_of_images = download_all_images(plugin, soupcontent)
for index, value in enumerate(list_of_src):
    filename_parts = list_of_images[index].split("/")
    logger1.info (f"REPLACE {value} with {filename_parts[-1]}")
    logger1.info (f"POSITION={contents.find(value)}")
    contents = contents.replace(value, filename_parts[-1])
contents = contents.replace("/wp-content/uploads/plugindocs/", "")
soup2 = BeautifulSoup(contents, SOUP_PARSER)

return md(soup2)

then the return value is added to the markdownfile using new_paragraph..

[didix21]

Could @osmanburucu-ibm check if it's fixed on v1.4.0?

[osmanburucu-ibm]

Looks good so far! Thank You for your fast response!