Can I Replace or Strip characters from Strings?

[bldcm]

I am trying to replace "invalid characters" so they can be saved in paragraph fields like description. Below is a snipit of a V1 configuration with character maps. If only one of the characters is present then the mapping works fine. If multiple characters are present it does not work and logging seems to indicate that each mapping starts with the original source data to do the mapping, meaning only the last found mapping takes affect.

Can multiple mappings be done on a single field?
Can multiple fields use the same mapping (i.e. have multiple source fields and targetfields) with repeating the configuration for each field?

"FieldMaps": [
{
"$type": "RegexFieldMapConfig",
"WorkItemTypeName": "",
"sourceField": "System.Description",
"targetField": "System.Description",
"pattern": "\uFFFF",
"replacement": "U+FFFF"
},
{
"$type": "RegexFieldMapConfig",
"WorkItemTypeName": "",
"sourceField": "System.Description",
"targetField": "System.Description",
"pattern": "\uF1A4",
"replacement": "U+F1A4"
},
{
"$type": "RegexFieldMapConfig",
"WorkItemTypeName": "*",
"sourceField": "System.Description",
"targetField": "System.Description",
"pattern": "\uF210",
"replacement": "U+F210"
},

[MrHinsh]

Correct. this is not the tool for that job. Use the StringManipulatorTool instead!

 {
  "MigrationTools": {
    "Version": "16.0",
    "CommonTools": {
      "StringManipulatorTool": {
        "Enabled": "True",
        "Manipulators": [
          {
            "$type": "RegexStringManipulator",
            "Description": "Remove invalid characters from string",
            "Enabled": "True",
            "Pattern": "[^( -~)\n\r\t]+",
            "Replacement": ""
          }
        ],
        "MaxStringLength": "1000000"
      }
    }
  }
}

You can add multiple manipulators or create one uber pattern... there is also a fix in the latest preview to enable you to override the default rather than merge it in... note that the Tfs Object Model does not support some characters...

note: The workaround is to change the appsettings.json and remove the default manipulator if you don't have the version with the fix.

---

Funding for us doing work on this tool is generally through 1) customer funding, or 2) donated free time.

Sponcer a Feature | Hire us directly

[bldcm]

I want to match 7 things and replace with the same thing, "". I have tried the default pattern, a pattern I tested with a regex tester for my specific characters and a pattern with the whole string, each individually, not all at same time. None of them worked. If I remove the string from the source, the WI migrates, so I need a way to get rid of these individual characters.

These are the characters: "��O펧甛O"

"Pattern": "[^( -~)\n\r\t]+",
"Pattern": "[\uffff\u004f\uf1a4\ud3a7\u751b\uf210]+",
"Pattern": "(\uffff)?\uf1a4\u004f\ud3a7\u751b\uf210\u004f",

I've attached logs of all three failures, one for each pattern I have tested.

20241015153139-logOfIndividualCharactersPattern.zip
20241009112303-logOfWholeStringPatern.zip
20241015152932-logOfDefaultPattern.zip
configuration-TFS2ADOPreProd-PowerPDF-NOTO-upgraded-DIAGS.json

Side question, how do I increasing the logging verbosity to get more details in log if needed?

[MrHinsh]

The default pattern only allows through asking characters between - and ~, everything else will be stripped.

"Pattern": "[\uffff\u004f\uf1a4\ud3a7\u751b\uf210]+",

This is not a valid pattern acording to https://regex101.com/

"Pattern": "(\uffff)?\uf1a4\u004f\ud3a7\u751b\uf210\u004f",

This is also not a valid pattern.

In the config, set

"Serilog": {
  "MinimumLevel": "Verbose"
},

---

Funding for us doing work on this tool is generally through 1) customer funding, or 2) donated free time.

Sponcer a Feature | Hire us directly

[bldcm]

The website you referenced, https://regex101.com/, for regex supports multiple "flavors" and 5 of 8 found the reg expression valid as shown in the image below

I also believe it is valid because I have used similar regexpr in the migration tool as shown in the first post of this discussion and repeated here:

{
  "$type": "RegexFieldMapConfig",
  "WorkItemTypeName": "*",
  "sourceField": "System.Description",
  "targetField": "System.Description",
  "pattern": "\uFFFF",
  "replacement": "U+FFFF"
},    
{
  "$type": "RegexFieldMapConfig",
  "WorkItemTypeName": "*",
  "sourceField": "System.Description",
  "targetField": "System.Description",
  "pattern": "\uF1A4",
  "replacement": "U+F1A4"
},
{
  "$type": "RegexFieldMapConfig",
  "WorkItemTypeName": "*",
  "sourceField": "System.Description",
  "targetField": "System.Description",
  "pattern": "\uF210",
  "replacement": "U+F210"
},
{
  "$type": "FieldtoFieldMapConfig",
  "WorkItemTypeName": "*",
  "sourceField": "System.Team",
  "targetField": "System.Team",
  "defaultValue": "Power PDF"
},

It was after this I was directed to use the RegexStringManipulator tool.

So unless the two methods use different regular expression flavors, I believe it is valid.

[MrHinsh]

What are you expecting [\uffff\u004f\uf1a4\ud3a7\u751b\uf210]+ to match? Are those characters always together?

They use the same regex engine, but do not have the same match, this would be the same match:

 {
  "MigrationTools": {
    "Version": "16.0",
    "CommonTools": {
      "StringManipulatorTool": {
        "Enabled": "True",
        "Manipulators": [
          {
            "$type": "RegexStringManipulator",
            "Description": "Remove invalid characters from string",
            "Enabled": "True",
            "Pattern": "\uF1A4",
            "Replacement": "U+F1A4"
          }
        ],
        "MaxStringLength": "1000000"
      }
    }
  }
}

however [\uffff\u004f\uf1a4\ud3a7\u751b\uf210]+ does not match \uF1A4".

[MrHinsh]

The advantage of the StringManipulatorTool is that it applies to any string field.

[bldcm]

I'm expecting the pattern to match any instance of any individual character in the set defined by the pattern [\uffff\u004f\uf1a4\ud3a7\u751b\uf210]+, and replace each character found with "". In this example the characters happen to be together, but that is not necessarily the case.

I tested the pattern on the website you use for regex testing and it identifies each character in the string whether they are together or not. See screen capture below:

Not sure what what your statement, "however [\uffff\u004f\uf1a4\ud3a7\u751b\uf210]+ does not match \uF1A4".", is trying to say. Do you think there is a casing problem or that the set of \u characters is not valid for the StringManiuplatorTool?

[bldcm]

Does anyone have an example of using StringManipulatorTool to remove unicode characters defined in a RegEx set?