[Feature request] Extend profile manager to delete progress for specified CDLCs

[BuongiornoTexas]

Request from JustinAiken to be able to delete progress for custom DLC that won't be played again.

See rs-manager issue #55 for original discussion.

To implement this, I need to identify all json paths affected by CDLC. Ideally, it would be nice to identify these by song name, song key or arrangment id.

Pick this up after adding macos support sorting out importing song lists from rs-manager.

[JustinAiken]

But would be great to be able to delete progress for old customs.

This may be as simple as pruning out all json elements for those arrrangement ids? I'll put it in as a feature request for the profile manager.

Yes, in three places... for example, to remove Bad Religion - Sorrow (Lead) (not CDLC, but arrangement_id on my clipboard), which is 8C73667F0A0A601972496DBEFC42CF8B, you'd delete all three of these:

• "Songs" -> "8C73667F0A0A601972496DBEFC42CF8B"
• "SongsSA" -> "8C73667F0A0A601972496DBEFC42CF8B"
• "Stats" -> "Songs" -> "8C73667F0A0A601972496DBEFC42CF8B"

..not sure what it would do the calculated stats in the toplevel Stats key... 🤔

[BuongiornoTexas]

Great - it should be quick to do, although it may be a few weeks before I get to it.

The other thing I will do is a check for the arrangement id elsewhere in the save, and I'll raise an exception if this occurs.

On the stats, I think the effects will be minimal because most of the stats seem to be based on accumulators rather than recalculation. Either way, its not going to be perfect:

• If rocksmith calculates from scratch, you get the values reflecting your current library.
• If rocksmith just updates accumulators, you get the values reflecting your history over all songs.
• And possibly, there will be a mix of the two!

I can live with this.

However, rather than speculating, we will be able to test effects by: creating a test profile, deleting a couple of arrangements, and seeing what has changed.

[BuongiornoTexas]

@JustinAiken,,

I've found five places arrangement ids occur:

• "Songs"
• "Playnexts"->"Songs" - I'm pretty sure this is the set of Rocksmith Recommends items in Learn a song
• "Stats"->"Songs"
• "SongsSA"
• "Achievements" ->"SongStreakID"

As I'm not sure what the SongStreakId is for, my inclination would be to apply the following rules:

• If an arrangement id appears in "SongStreakId", don't delete this id anywhere.
• If the arrangement doesn't appear in SongStreakId, it is safe to delete from the other locations.

The alternative would be to delete only from the first four and ignore the fifth. Do you have a view on this?

Also, could you confirm my result by running the script scratch.py on a json dump of your profile (you will need to edit the json_file variable first)? This will find all arrangement ids in the first four locations and then walk the json object looking for more instances of arrangment data - hopefully it will only find the one entry at "Acheivements"->"SongStreakId". My output looks like:

      ('Achievements', 'SongStreakID') C2B8CA30003AEED822D8E50FECCC9B1C

[JustinAiken]

As I'm not sure what the SongStreakId is for

I think that's also for the RS Recommends, "Get a streak of 300 on this song" type of thing.

The alternative would be to delete only from the first four and ignore the fifth. Do you have a view on this?

Should be safe to delete all 5, if you're truly attempting to reset/remove all progress on it.

Also, could you confirm my result by running the script scratch.py on a json dump of your profile (you will need to edit the json_file variable first)?

('Achievements', 'SongStreakID') 2A39EA7FFC1B3BFD569316BCE3213CC6

That's a GVF I had played most recently.

..surprised I'm only getting one result, the .json is massive (817,853 lines!!!)

[BuongiornoTexas]

Thanks for the info, and good to see that you aren't finding anything else either. And that is a lot of json - my file is about quarter of the size! (but then I don't play around much with CDLC either.)

I think that's also for the RS Recommends, "Get a streak of 300 on this song" type of thing.

I don't think this will be it - there are three RS Recommends slots per song in Playnext->Songs. However, knowing it is something that changes based on recent plays, I'll just delete whatever is in my test profile and see what changes.

I should have something together in week or so.

[BuongiornoTexas]

@JustinAiken,

I've closed this with the 0.2.2 release (available on pypi). I would still use with caution and try a test profile first.

The SongStreakID looks as though it is used for tracking streaks across songs, as it is updated each time you exit.

I'm not confident about deleting or editing this field - I have tried the approach of not changing it at all when deleting arrangements (i.e. ignore this field even if the arrangement id is selected for deletion), and it looks as though Rocksmith just does the right thing and updates this field with whatever the next song you play turns out to be. But even so, I'm not confident this can be relied on.

So, on the basis that this should be a moderately infrequent occurrence, I've gone with the option of raising a warning if it happens. I then allow the user to decide if they want to take a chance on the edit, or exit and fix the problem by playing a song. The text of the warning is:

WARNING: One of the arrangement ids you want to delete was the last song played.
This is used for streak tracking, and I do not have a guaranteed safe way to fix this.
Your options are:

    1) Play it safe. Exit the program, play a song that you are not deleting in Score Attack,
       and then try again (Recommended).

    2) Continue with the deletion.
       This will *probably* be fine, but isn't guaranteed.


'y' to take the safe option (exit), 'n' to continue with deletion.

Y/n to confirm, N/n to reject >

Let me know if this feature needs more work.

Cheers,

Larry

[JustinAiken]

Alright, finally gave it a try...

• Calling --delete-arrangements without a arrangements_file_path doesn't seem to work:

  $ profilemanager --delete-arrangements .
  usage: profilemanager [-h]
                        [--clone-profile | --set-play-counts play_count_file_path | --delete-arrangements [arrangements_file_path]
                        | --dump-profile]
                        working_dir
  profilemanager: error: the following arguments are required: working_dir

• But if I touched a file with the id, it did.

First attempt

• First, I did a json dump of my profile
• Next, I removed arrangement 7BE964CC9036474CA2493C40198A564B (Stacy's Mom custom by firekorn - no more need for that CDLC now that there's an official version).
• Then I did another json dump.

Diff

Here's what json-diff saw changed:

 {
   Songs: {
-    7BE964CC9036474CA2493C40198A564B: {
-      TimeStamp: 1502423839
-      DynamicDifficulty: {
-        Avg: 1
-        Phrases: {
-        }
-        LevelUp: [
-        ]
-      }
-    }
   }
   Stats: {
     Songs: {
-      7BE964CC9036474CA2493C40198A564B: {
-        MasteryLast: 0.877037
-        PlayedCount: 1
-        AccuracyGlobal: 0.877037
-        DateLAS: "2017-08-10 21:57:19"
-        AccuracyChords: 0.032847
-        MasteryPrevious: 0
-        MasteryPeak: 0.877037
-        Streak: 224
-        ArticulationAccuracy: [
-          {
-            V: 0.5
-          }
-          {
-            V: -1
-          }
-          {
-            V: 0
-          }
-          {
-            V: 0.428571
-          }
-          {
-            V: -1
-          }
-          {
-            V: -1
-          }
-          {
-            V: -1
-          }
-          {
-            V: -1
-          }
-          {
-            V: 1
-          }
-          {
-            V: -1
-          }
-          {
-            V: 0.033333
-          }
-          {
-            V: -1
-          }
-          {
-            V: -1
-          }
-          {
-            V: -1
-          }
-          {
-            V: -1
-          }
-          {
-            V: -1
-          }
-          {
-            V: -1
-          }
-          {
-            V: -1
-          }
-          {
-            V: -1
-          }
-          {
-            V: -1
-          }
-          {
-            V: -1
-          }
-          {
-            V: -1
-          }
-          {
-            V: -1
-          }
-          {
-            V: -1
-          }
-          {
-            V: -1
-          }
-          {
-            V: -1
-          }
-          {
-            V: -1
-          }
-          {
-            V: 0.095238
-          }
-        ]
-        ChordsAccuracies: [
-          {
-            V: -1
-          }
-          {
-            V: -1
-          }
-          {
-            V: -1
-          }
-          {
-            V: -1
-          }
-          {
-            V: 0.03125
-          }
-          {
-            V: -1
-          }
-          {
-            V: 0.038462
-          }
-          {
-            V: -1
-          }
-          {
-            V: 1
-          }
-          {
-            V: -1
-          }
-          {
-            V: 1
-          }
-          {
-            V: -1
-          }
-          {
-            V: 1
-          }
-          {
-            V: -1
-          }
-          {
-            V: 1
-          }
-          {
-            V: 1
-          }
-          {
-            V: -1
-          }
-          {
-            V: 0.1
-          }
-          {
-            V: -1
-          }
-          {
-            V: 0.5
-          }
-          {
-            V: -1
-          }
-          {
-            V: 0.025
-          }
-          {
-            V: -1
-          }
-          {
-            V: 0.09375
-          }
-          {
-            V: -1
-          }
-          {
-            V: 1
-          }
-        ]
-      }
     }
   }
   Tones: [
     ...
     ...
     ...
-    {
-      GearList: {
-        Rack1: {
-          Type: "Racks"
-          KnobValues: {
-            Rack_StudioEQ_BassFreq: 270
-            Rack_StudioEQ_HiMidFreq: 2
-            Rack_StudioEQ_TrebleFreq: 5.6
-            Rack_StudioEQ_LoMidFreq: 210
-            Rack_StudioEQ_Treble: 5
-            Rack_StudioEQ_LoMidQ: 1
-            Rack_StudioEQ_Bass: 2
-            Rack_StudioEQ_HiMid: 6
-            Rack_StudioEQ_LoMid: -5
-            Rack_StudioEQ_HiMidQ: 0
-          }
-          Key: "Rack_StudioEQ"
-        }
-        Rack2: {
-          Type: "Racks"
-          KnobValues: {
-            Rack_StudioChamber_Mix: 16
-            Rack_StudioChamber_Depth: 72
-            Rack_StudioChamber_Tone: 26
-            Rack_StudioChamber_Time: 56
-          }
-          Key: "Rack_StudioChamber"
-          Category: "Reverb"
-        }
-        Amp: {
-          Type: "Amps"
-          KnobValues: {
-            Amp_MarshallPlexi_Loudness1: 88
-            Amp_MarshallPlexi_Loudness2: 59
-            Amp_MarshallPlexi_Mid: 85
-            Amp_MarshallPlexi_Bass: 82
-            Amp_MarshallPlexi_Pres: 10
-            Amp_MarshallPlexi_Treble: 89
-          }
-          Key: "Amp_MarshallPlexi"
-        }
-        Cabinet: {
-          Type: "Cabinets"
-          KnobValues: {
-          }
-          Key: "Cab_Marshall1960TV_Ribbon_Cone"
-        }
-        PostPedal1: {
-          Type: "Pedals"
-          KnobValues: {
-            Pedal_ModDelay_Time: 360
-            Pedal_ModDelay_Mix: 41
-            Pedal_ModDelay_Feedback: 24
-            Pedal_ModDelay_Rate: 0.5
-            Pedal_ModDelay_Depth: 38
-          }
-          Key: "Pedal_ModDelay"
-          Category: "30779"
-        }
-        PrePedal1: {
-          Type: "Pedals"
-          KnobValues: {
-            Pedal_GermaniumDrive_Gain: 80
-            Pedal_GermaniumDrive_Tone: 36
-          }
-          Key: "Pedal_GermaniumDrive"
-          Category: "Distortion"
-        }
-      }
-      IsCustom: false
-      Volume: "-19.479343"
-      Key: "ToneDefault_Lead"
-      Name: "$[36465]Guitar"
-    }
     ...
+    {
+      GearList: {
+        Rack1: {
+          Type: "Racks"
+          KnobValues: {
+            Rack_StudioEQ_BassFreq: 270
+            Rack_StudioEQ_HiMidFreq: 2
+            Rack_StudioEQ_TrebleFreq: 5.6
+            Rack_StudioEQ_LoMidFreq: 210
+            Rack_StudioEQ_Treble: 5
+            Rack_StudioEQ_LoMidQ: 1
+            Rack_StudioEQ_Bass: 2
+            Rack_StudioEQ_HiMid: 6
+            Rack_StudioEQ_LoMid: -5
+            Rack_StudioEQ_HiMidQ: 0
+          }
+          Key: "Rack_StudioEQ"
+        }
+        Rack2: {
+          Type: "Racks"
+          KnobValues: {
+            Rack_StudioChamber_Mix: 16
+            Rack_StudioChamber_Depth: 72
+            Rack_StudioChamber_Tone: 26
+            Rack_StudioChamber_Time: 56
+          }
+          Key: "Rack_StudioChamber"
+          Category: "Reverb"
+        }
+        Amp: {
+          Type: "Amps"
+          KnobValues: {
+            Amp_MarshallPlexi_Loudness1: 88
+            Amp_MarshallPlexi_Loudness2: 59
+            Amp_MarshallPlexi_Mid: 85
+            Amp_MarshallPlexi_Bass: 82
+            Amp_MarshallPlexi_Pres: 10
+            Amp_MarshallPlexi_Treble: 89
+          }
+          Key: "Amp_MarshallPlexi"
+        }
+        Cabinet: {
+          Type: "Cabinets"
+          KnobValues: {
+          }
+          Key: "Cab_Marshall1960TV_Ribbon_Cone"
+        }
+        PostPedal1: {
+          Type: "Pedals"
+          KnobValues: {
+            Pedal_ModDelay_Time: 360
+            Pedal_ModDelay_Mix: 41
+            Pedal_ModDelay_Feedback: 24
+            Pedal_ModDelay_Rate: 0.5
+            Pedal_ModDelay_Depth: 38
+          }
+          Key: "Pedal_ModDelay"
+          Category: "30779"
+        }
+        PrePedal1: {
+          Type: "Pedals"
+          KnobValues: {
+            Pedal_GermaniumDrive_Gain: 80
+            Pedal_GermaniumDrive_Tone: 36
+          }
+          Key: "Pedal_GermaniumDrive"
+          Category: "Distortion"
+        }
+      }
+      IsCustom: false
+      Volume: "-19.479343"
+      Key: "ToneDefault_Lead"
+      Name: "$[36465]Guitar"
+    }
   ]
 }

• Progress under Songs and Stats/Songs gone - cool
• Not sure what the tone diff is all about
• Next I launched RS, hit the stats screen once, then quit
• Did another profile dump

Diff

 {
   Stats: {
     SessionPoolWeights: [
-      {
-        V: 30
-      }
+      {
+        V: 40
+      }
       ...
-      {
-        V: 170
-      }
-      {
-        V: 50
-      }
+      {
+        V: 190
+      }
+      {
+        V: 60
+      }
     ]
-    StatsOnlineVersion: 27343
+    StatsOnlineVersion: 27347
   }
   Tones: [
     ...
     ...
     ...
-    {
-      GearList: {
-        Rack1: {
-          Type: "Racks"
-          KnobValues: {
-            Rack_StudioEQ_BassFreq: 270
-            Rack_StudioEQ_HiMidFreq: 2
-            Rack_StudioEQ_TrebleFreq: 5.6
-            Rack_StudioEQ_LoMidFreq: 210
-            Rack_StudioEQ_Treble: 5
-            Rack_StudioEQ_LoMidQ: 1
-            Rack_StudioEQ_Bass: 2
-            Rack_StudioEQ_HiMid: 6
-            Rack_StudioEQ_LoMid: -5
-            Rack_StudioEQ_HiMidQ: 0
-          }
-          Key: "Rack_StudioEQ"
-        }
-        Rack2: {
-          Type: "Racks"
-          KnobValues: {
-            Rack_StudioChamber_Mix: 16
-            Rack_StudioChamber_Depth: 72
-            Rack_StudioChamber_Tone: 26
-            Rack_StudioChamber_Time: 56
-          }
-          Key: "Rack_StudioChamber"
-          Category: "Reverb"
-        }
-        Amp: {
-          Type: "Amps"
-          KnobValues: {
-            Amp_MarshallPlexi_Loudness1: 88
-            Amp_MarshallPlexi_Loudness2: 59
-            Amp_MarshallPlexi_Mid: 85
-            Amp_MarshallPlexi_Bass: 82
-            Amp_MarshallPlexi_Pres: 10
-            Amp_MarshallPlexi_Treble: 89
-          }
-          Key: "Amp_MarshallPlexi"
-        }
-        Cabinet: {
-          Type: "Cabinets"
-          KnobValues: {
-          }
-          Key: "Cab_Marshall1960TV_Ribbon_Cone"
-        }
-        PostPedal1: {
-          Type: "Pedals"
-          KnobValues: {
-            Pedal_ModDelay_Time: 360
-            Pedal_ModDelay_Mix: 41
-            Pedal_ModDelay_Feedback: 24
-            Pedal_ModDelay_Rate: 0.5
-            Pedal_ModDelay_Depth: 38
-          }
-          Key: "Pedal_ModDelay"
-          Category: "30779"
-        }
-        PrePedal1: {
-          Type: "Pedals"
-          KnobValues: {
-            Pedal_GermaniumDrive_Gain: 80
-            Pedal_GermaniumDrive_Tone: 36
-          }
-          Key: "Pedal_GermaniumDrive"
-          Category: "Distortion"
-        }
-      }
-      IsCustom: false
-      Volume: "-19.479343"
-      Key: "ToneDefault_Lead"
-      Name: "$[36465]Guitar"
-    }
     ...
+    {
+      GearList: {
+        Rack1: {
+          Type: "Racks"
+          KnobValues: {
+            Rack_StudioEQ_BassFreq: 270
+            Rack_StudioEQ_HiMidFreq: 2
+            Rack_StudioEQ_TrebleFreq: 5.6
+            Rack_StudioEQ_LoMidFreq: 210
+            Rack_StudioEQ_Treble: 5
+            Rack_StudioEQ_LoMidQ: 1
+            Rack_StudioEQ_Bass: 2
+            Rack_StudioEQ_HiMid: 6
+            Rack_StudioEQ_LoMid: -5
+            Rack_StudioEQ_HiMidQ: 0
+          }
+          Key: "Rack_StudioEQ"
+        }
+        Rack2: {
+          Type: "Racks"
+          KnobValues: {
+            Rack_StudioChamber_Mix: 16
+            Rack_StudioChamber_Depth: 72
+            Rack_StudioChamber_Tone: 26
+            Rack_StudioChamber_Time: 56
+          }
+          Key: "Rack_StudioChamber"
+          Category: "Reverb"
+        }
+        Amp: {
+          Type: "Amps"
+          KnobValues: {
+            Amp_MarshallPlexi_Loudness1: 88
+            Amp_MarshallPlexi_Loudness2: 59
+            Amp_MarshallPlexi_Mid: 85
+            Amp_MarshallPlexi_Bass: 82
+            Amp_MarshallPlexi_Pres: 10
+            Amp_MarshallPlexi_Treble: 89
+          }
+          Key: "Amp_MarshallPlexi"
+        }
+        Cabinet: {
+          Type: "Cabinets"
+          KnobValues: {
+          }
+          Key: "Cab_Marshall1960TV_Ribbon_Cone"
+        }
+        PostPedal1: {
+          Type: "Pedals"
+          KnobValues: {
+            Pedal_ModDelay_Time: 360
+            Pedal_ModDelay_Mix: 41
+            Pedal_ModDelay_Feedback: 24
+            Pedal_ModDelay_Rate: 0.5
+            Pedal_ModDelay_Depth: 38
+          }
+          Key: "Pedal_ModDelay"
+          Category: "30779"
+        }
+        PrePedal1: {
+          Type: "Pedals"
+          KnobValues: {
+            Pedal_GermaniumDrive_Gain: 80
+            Pedal_GermaniumDrive_Tone: 36
+          }
+          Key: "Pedal_GermaniumDrive"
+          Category: "Distortion"
+        }
+      }
+      IsCustom: false
+      Volume: "-19.479343"
+      Key: "ToneDefault_Lead"
+      Name: "$[36465]Guitar"
+    }
   ]
 }

• I'm not sure what SessionPoolWeights orStatsOnlineVersion but they changed
• And RS fixed the tones

More deletes!

Then I went and deleted a batch of about 6 arrangements - some CDLCs that have been upgraded to ODLCs, and some with the 100% that's been driving me nuts.

• Launched RS
• All is well; songs I still have are reset to 0%/0 plays/etc
• I did notice that the in-game stats didn't change; "Songs Mastered" stayed the same, even though some 100%s went away
• But no biggie, since in-game stats suck; after a refresh in RS Manager, those numbers reported correct
• Next time I get a new song to 100% I'll see if it recalculates the stats or not.

TL;DR

• It worked! No more deleted CDLC or 100% bug ridden songs to bug me

[BuongiornoTexas]

It worked! No more deleted CDLC or 100% bug ridden songs to bug me

Great - that's the outcome I was hoping for.

Calling --delete-arrangements without a arrangements_file_path doesn't seem to work:

This is an optional command line oddity - you called:

  $ profilemanager --delete-arrangements .

The command line parser read this as a delete arrangements with the optional path to the list of arrangements specified as .. The arg parser then looks for the working directory and can't find it. This should give you the interactive session you wanted:

   $ profilemanager . --delete-arrangements

(Specify the working directory first, delete arrangements without argument last.)

I don't know what is going on with the Tones (and the profile manager definitely doesn't modify these), I did a side by side diff of the two sets you provided and couldn't see anything at all - possibly github eating white space? or maybe a false positive? I also didn't see these tone changes when I did my tests (7 arrangements removed, with changes only where expected). If it is white space, I doubt it's anything to worry about, and generally my attitude is if RS can load it and then save again, you are 99.99% good.

Next I launched RS, hit the stats screen once, then quit.

I'm not surprised to see changes - RS will modify things and resync to cloud even if you just open it, start a profile and close it. I haven't had the energy to see what is happening here (especially since I can't control it anyway).

Some with the 100% that's been driving me nuts.

I think Customs Forge Song Manager includes a fix that can get these CDLCs working properly again, and I think you can reset your progress via the riff repeater? (fixing and then deleting arrangement data, and restarting should work too).

I did notice that the in-game stats didn't change,

Most of the in-game stats are accumulators - totals and counts get incremented each time you do something (e.g get a plat badge, get a 100%), rather than recalculating from the raw data each time. So, yup, stats are going to reflect your total playing history rather that your current library (not too much of a problem if you are only deleting a small number of arrangements with low play counts).

Given all is working well enough and this is an advanced functionality, I'm inclined to leave this closed unless you want to some more digging on the Tone issues?

I'm also tending to the view that this is probably better as a minimally documented capability - good for people who know what they are doing, but possibly quite dangerous to your profile if you don't!

Cheers,

Larry

[JustinAiken]

Given all is working well enough and this is an advanced functionality, I'm inclined to leave this closed unless you want to some more digging on the Tone issues?

Oh yeah, it's definitely working; I was just over-reporting details. Thanks again!