Description
Due to security reasons I was attempting to update to UBI9.5, however I hit some issues. I hit an OpenSSL issue when I ran the line gem update --system 3.3.14:
OpenSSL is not available. Install OpenSSL and rebuild Ruby or use non-HTTPS sources (Gem::Exception)
To fix this I updated Ruby to 3.3.7, however I was faced with another issue in that the unit tests started to fail. Specifically the tests in ruby_lex_utils_spec.rb were failing as it appears that the class RubyLex has been refactored heavily:
https://msp-greg.github.io/ruby_3_3/irb/IRB/RubyLex.html
https://msp-greg.github.io/ruby_3_2/irb/RubyLex.html
I did some work to fix some of the tests by switching from using RubyLex to Prism. I have been stuck on the function each_lexed_segment as Prism for the most part has been a drop in replacement however in the implementation of this function it is a bit trickier. I am attaching the code that I have updated just to give it as an example, and it may be helpful in a future update.
It should be noted that Prism does not emit the on_sp event, so you will not be able to recreate your code with the spaces in it, thus some of the tests will need to be updated to account for this lack of spacing.
ruby/prism#722
# encoding: ascii-8bit
# Copyright 2022 Ball Aerospace & Technologies Corp.
# All Rights Reserved.
#
# This program is free software; you can modify and/or redistribute it
# under the terms of the GNU Affero General Public License
# as published by the Free Software Foundation; version 3 with
# attribution addendums as found in the LICENSE.txt
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU Affero General Public License for more details.
# Modified by OpenC3, Inc.
# All changes Copyright 2022, OpenC3, Inc.
# All Rights Reserved
#
# This file may also be used under the terms of a commercial license
# if purchased from OpenC3, Inc.
require 'prism'
class RubyLexUtils
# Regular expression to detect blank lines
BLANK_LINE_REGEX = /^\s*$/
# Regular expression to detect lines containing only 'else'
LONELY_ELSE_REGEX = /^\s*else\s*$/
KEY_KEYWORDS = [
'class'.freeze,
'module'.freeze,
'def'.freeze,
'undef'.freeze,
'begin'.freeze,
'rescue'.freeze,
'ensure'.freeze,
'end'.freeze,
'if'.freeze,
'unless'.freeze,
'then'.freeze,
'elsif'.freeze,
'else'.freeze,
'case'.freeze,
'when'.freeze,
'while'.freeze,
'until'.freeze,
'for'.freeze,
'break'.freeze,
'next'.freeze,
'redo'.freeze,
'retry'.freeze,
'in'.freeze,
'do'.freeze,
'return'.freeze,
'alias'.freeze
]
# @param text [String]
# @return [Boolean] Whether the text contains the 'begin' keyword
def contains_begin?(text)
lex = Prism.lex(text)
tokens = lex.value
tokens.each do |token|
token_object = token[0]
if token_object.type == :KEYWORD_BEGIN
return true
end
end
return false
end
# @param text [String]
# @return [Boolean] Whether the text contains the 'end' keyword
def contains_end?(text)
lex = Prism.lex(text)
tokens = lex.value
tokens.each do |token|
token_object = token[0]
if token_object.type == :KEYWORD_END
return true
end
end
return false
end
# @param text [String]
# @return [Boolean] Whether the text contains a Ruby keyword
def contains_keyword?(text)
lex = Prism.lex(text)
tokens = lex.value
tokens.each do |token|
token_object = token[0]
state_bits = token[1]
if token_object.type.start_with?("KEYWORD")
if KEY_KEYWORDS.include?(token_object.value)
return true
end
elsif token_object.type == :BRACE_LEFT and state_bits != (Ripper::EXPR_BEG | Ripper::EXPR_LABEL)
return true
end
end
return false
end
# @param text [String]
# @return [Boolean] Whether the text contains a keyword which starts a block.
# i.e. 'do', '{', or 'begin'
def contains_block_beginning?(text)
lex = Prism.lex(text)
tokens = lex.value
tokens.each do |token|
token_object = token[0]
if token_object.type == :KEYWORD_BEGIN || token_object.type == :KEYWORD_DO || token_object.type == :BRACE_LEFT
return true
end
end
return false
end
def continue_block?(text)
lex = Prism.lex(text)
tokens = lex.value
index = tokens.length - 1
while index > 0
token_object = tokens[index][0]
return true if token_object.type == :KEYWORD_DO
index -= 1
end
return false
end
# @param text [String]
# @param progress_dialog [OpenC3::ProgressDialog] If this is set, the overall
# progress will be set as the processing progresses
# @return [String] The text with all comments removed
def remove_comments(text, progress_dialog = nil)
lex = Prism.lex(text)
tokens = lex.value
comments_removed = ""
token_count = 0
progress = 0.0
tokens.each do |token|
token_object = token[0]
token_count += 1
if token_object.type != :COMMENT
comments_removed << token_object.value
else
newline_count = token_object.value.count("\n")
comments_removed << ("\n" * newline_count)
end
if progress_dialog and token_count % 10000 == 0
progress += 0.01
progress = 0.0 if progress >= 0.99
progress_dialog.set_overall_progress(progress)
end
end
return comments_removed
end