Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

NameError: instance variable @sequence_name not defined #81

Closed
NielsKSchjoedt opened this issue Jun 26, 2020 · 5 comments · Fixed by #89 or #187
Closed

NameError: instance variable @sequence_name not defined #81

NielsKSchjoedt opened this issue Jun 26, 2020 · 5 comments · Fixed by #89 or #187
Labels
bug Something isn't working

Comments

@NielsKSchjoedt
Copy link

After upgrading to ros-apartment from apartment, I get the above error at times:

[GEM_ROOT]/gems/ros-apartment-2.6.1/lib/apartment/adapters/postgresql_adapter.rb:124 :in `remove_instance_variable`
 [GEM_ROOT]/gems/ros-apartment-2.6.1/lib/apartment/adapters/postgresql_adapter.rb:124 :in `block in reset_sequence_names`
 [GEM_ROOT]/gems/ros-apartment-2.6.1/lib/apartment/adapters/postgresql_adapter.rb:123 :in `each`
 [GEM_ROOT]/gems/ros-apartment-2.6.1/lib/apartment/adapters/postgresql_adapter.rb:123 :in `reset_sequence_names`
 [GEM_ROOT]/gems/ros-apartment-2.6.1/lib/apartment/adapters/postgresql_adapter.rb:40 :in `reset`
 [GEM_ROOT]/gems/ros-apartment-2.6.1/lib/apartment/adapters/postgresql_adapter.rb:30 :in `initialize`
 [GEM_ROOT]/gems/ros-apartment-2.6.1/lib/apartment/adapters/postgresql_adapter.rb:11 :in `new`
 [GEM_ROOT]/gems/ros-apartment-2.6.1/lib/apartment/adapters/postgresql_adapter.rb:11 :in `postgresql_adapter`
 [GEM_ROOT]/gems/ros-apartment-2.6.1/lib/apartment/tenant.rb:56 :in `adapter`
/home/app_au/.rbenv/versions/2.6.5/lib/ruby/2.6.0/forwardable.rb:224 :in `switch!`
 [PROJECT_ROOT]/lib/country_manager.rb:188 :in `country=`
 [PROJECT_ROOT]/config/initializers/sidekiq.rb:55 :in `set_country_from_job`
 [PROJECT_ROOT]/config/initializers/sidekiq.rb:27 :in `call`
 [GEM_ROOT]/gems/sidekiq-6.0.7/lib/sidekiq/middleware/chain.rb:140 :in `block in invoke`
 [GEM_ROOT]/gems/sidekiq-6.0.7/lib/sidekiq/middleware/chain.rb:140 :in `block in invoke`
 [GEM_ROOT]/gems/sidekiq-6.0.7/lib/sidekiq/middleware/chain.rb:143 :in `invoke`
 [GEM_ROOT]/gems/sidekiq-6.0.7/lib/sidekiq/processor.rb:163 :in `block in process`
 [GEM_ROOT]/gems/sidekiq-6.0.7/lib/sidekiq/processor.rb:136 :in `block (6 levels) in dispatch`
 [GEM_ROOT]/gems/sidekiq-6.0.7/lib/sidekiq/job_retry.rb:111 :in `local`
 [GEM_ROOT]/gems/sidekiq-6.0.7/lib/sidekiq/processor.rb:135 :in `block (5 levels) in dispatch`
 [GEM_ROOT]/gems/sidekiq-6.0.7/lib/sidekiq/rails.rb:43 :in `block in call`
 [GEM_ROOT]/gems/activesupport-5.2.4.2/lib/active_support/execution_wrapper.rb:87 :in `wrap`
 [GEM_ROOT]/gems/activesupport-5.2.4.2/lib/active_support/reloader.rb:73 :in `block in wrap`
 [GEM_ROOT]/gems/activesupport-5.2.4.2/lib/active_support/execution_wrapper.rb:87 :in `wrap`
 [GEM_ROOT]/gems/activesupport-5.2.4.2/lib/active_support/reloader.rb:72 :in `wrap`
 [GEM_ROOT]/gems/sidekiq-6.0.7/lib/sidekiq/rails.rb:42 :in `call`
 [GEM_ROOT]/gems/sidekiq-6.0.7/lib/sidekiq/processor.rb:131 :in `block (4 levels) in dispatch`
 [GEM_ROOT]/gems/sidekiq-6.0.7/lib/sidekiq/processor.rb:257 :in `stats`
 [GEM_ROOT]/gems/sidekiq-6.0.7/lib/sidekiq/processor.rb:126 :in `block (3 levels) in dispatch`
 [GEM_ROOT]/gems/sidekiq-6.0.7/lib/sidekiq/job_logger.rb:13 :in `call`
 [GEM_ROOT]/gems/sidekiq-6.0.7/lib/sidekiq/processor.rb:125 :in `block (2 levels) in dispatch`
 [GEM_ROOT]/gems/sidekiq-6.0.7/lib/sidekiq/job_retry.rb:78 :in `global`
 [GEM_ROOT]/gems/sidekiq-6.0.7/lib/sidekiq/processor.rb:124 :in `block in dispatch`
 [GEM_ROOT]/gems/sidekiq-6.0.7/lib/sidekiq/logger.rb:10 :in `with`
 [GEM_ROOT]/gems/sidekiq-6.0.7/lib/sidekiq/job_logger.rb:33 :in `prepare`
 [GEM_ROOT]/gems/sidekiq-6.0.7/lib/sidekiq/processor.rb:123 :in `dispatch`
 [GEM_ROOT]/gems/sidekiq-6.0.7/lib/sidekiq/processor.rb:162 :in `process`
 [GEM_ROOT]/gems/sidekiq-6.0.7/lib/sidekiq/processor.rb:78 :in `process_one`
 [GEM_ROOT]/gems/sidekiq-6.0.7/lib/sidekiq/processor.rb:68 :in `run`
 [GEM_ROOT]/gems/sidekiq-6.0.7/lib/sidekiq/util.rb:15 :in `watchdog`
 [GEM_ROOT]/gems/sidekiq-6.0.7/lib/sidekiq/util.rb:24 :in `block in safe_thread`More (36 lines)
Nested Exceptions
NameError: instance variable @sequence_name not defined
 [GEM_ROOT]/gems/ros-apartment-2.6.1/lib/apartment/adapters/postgresql_adapter.rb:124 :in `remove_instance_variable`
 Sidekiq::JobRetry::Handled: Sidekiq::JobRetry::Handled
 [GEM_ROOT]/gems/sidekiq-6.0.7/lib/sidekiq/job_retry.rb:99 :in `rescue in global`
 NameError: instance variable @sequence_name not defined
 [GEM_ROOT]/gems/ros-apartment-2.6.1/lib/apartment/adapters/postgresql_adapter.rb:124 :in `remove_instance_variable`

Rails 5.2 and latest ros-apartment from rubygems.

Any idea what it could be?

@rpbaltazar
Copy link
Contributor

rpbaltazar commented Jun 26, 2020

You are running 2.6.1 which is not the latest but the issue should still be happening on the latest version as that particular piece of code has not changed.
By the logs that you've pasted, seems to me you should be doing something like:

Apartment.switch(tenant) do
<your business logic>
end

on a sidekiq worker. Is that the case?

looking at apartment gem itself, we're trying to reset the sequence names without hitting the database, but that does not explain why your code would break
https://github.com/rails-on-services/apartment/blob/development/lib/apartment/adapters/postgresql_adapter.rb#L124-L129

we are selecting explicitly active record descendants that have that instance variable.

@NielsKSchjoedt
Copy link
Author

Yeah, it's odd. I'm processing millions and millions of background jobs every day, where I start every job by switching schema (similar to the middleware plugin for sidekiq), and the problem only occurred a few times in total.

Here is the latest trace from Honeybadger:

NameError: instance variable @sequence_name not defined
 [PROJECT_ROOT]/lib/country_manager.rb:188 :in `country=`
186 
187       ::NewRelic::Agent.add_custom_attributes(:country => code)
188       Apartment::Tenant.switch!(code)
189 
190       search_path = Apartment.connection.schema_search_path
 [PROJECT_ROOT]/config/initializers/sidekiq.rb:55 :in `set_country_from_job`
 [PROJECT_ROOT]/config/initializers/sidekiq.rb:27 :in `call`
Full Backtrace
 [GEM_ROOT]/gems/ros-apartment-2.6.1/lib/apartment/adapters/postgresql_adapter.rb:124 :in `remove_instance_variable`
 [GEM_ROOT]/gems/ros-apartment-2.6.1/lib/apartment/adapters/postgresql_adapter.rb:124 :in `block in reset_sequence_names`
 [GEM_ROOT]/gems/ros-apartment-2.6.1/lib/apartment/adapters/postgresql_adapter.rb:123 :in `each`
 [GEM_ROOT]/gems/ros-apartment-2.6.1/lib/apartment/adapters/postgresql_adapter.rb:123 :in `reset_sequence_names`
 [GEM_ROOT]/gems/ros-apartment-2.6.1/lib/apartment/adapters/postgresql_adapter.rb:82 :in `connect_to_new`
 [GEM_ROOT]/gems/ros-apartment-2.6.1/lib/apartment/adapters/abstract_adapter.rb:77 :in `block in switch!`
 [GEM_ROOT]/gems/activesupport-5.2.4.2/lib/active_support/callbacks.rb:98 :in `run_callbacks`
 [GEM_ROOT]/gems/ros-apartment-2.6.1/lib/apartment/adapters/abstract_adapter.rb:76 :in `switch!`
/home/app_au/.rbenv/versions/2.6.5/lib/ruby/2.6.0/forwardable.rb:230 :in `switch!`
 [PROJECT_ROOT]/lib/country_manager.rb:188 :in `country=`
 [PROJECT_ROOT]/config/initializers/sidekiq.rb:55 :in `set_country_from_job`
 [PROJECT_ROOT]/config/initializers/sidekiq.rb:27 :in `call`
 [GEM_ROOT]/gems/sidekiq-6.0.7/lib/sidekiq/middleware/chain.rb:140 :in `block in invoke`
 [GEM_ROOT]/gems/sidekiq-6.0.7/lib/sidekiq/middleware/chain.rb:140 :in `block in invoke`
 [GEM_ROOT]/gems/sidekiq-6.0.7/lib/sidekiq/middleware/chain.rb:143 :in `invoke`
 [GEM_ROOT]/gems/sidekiq-6.0.7/lib/sidekiq/processor.rb:163 :in `block in process`
 [GEM_ROOT]/gems/sidekiq-6.0.7/lib/sidekiq/processor.rb:136 :in `block (6 levels) in dispatch`
 [GEM_ROOT]/gems/sidekiq-6.0.7/lib/sidekiq/job_retry.rb:111 :in `local`
 [GEM_ROOT]/gems/sidekiq-6.0.7/lib/sidekiq/processor.rb:135 :in `block (5 levels) in dispatch`
 [GEM_ROOT]/gems/sidekiq-6.0.7/lib/sidekiq/rails.rb:43 :in `block in call`
 [GEM_ROOT]/gems/activesupport-5.2.4.2/lib/active_support/execution_wrapper.rb:87 :in `wrap`
 [GEM_ROOT]/gems/activesupport-5.2.4.2/lib/active_support/reloader.rb:73 :in `block in wrap`
 [GEM_ROOT]/gems/activesupport-5.2.4.2/lib/active_support/execution_wrapper.rb:87 :in `wrap`
 [GEM_ROOT]/gems/activesupport-5.2.4.2/lib/active_support/reloader.rb:72 :in `wrap`
 [GEM_ROOT]/gems/sidekiq-6.0.7/lib/sidekiq/rails.rb:42 :in `call`
 [GEM_ROOT]/gems/sidekiq-6.0.7/lib/sidekiq/processor.rb:131 :in `block (4 levels) in dispatch`
 [GEM_ROOT]/gems/sidekiq-6.0.7/lib/sidekiq/processor.rb:257 :in `stats`
 [GEM_ROOT]/gems/sidekiq-6.0.7/lib/sidekiq/processor.rb:126 :in `block (3 levels) in dispatch`
 [GEM_ROOT]/gems/sidekiq-6.0.7/lib/sidekiq/job_logger.rb:13 :in `call`
 [GEM_ROOT]/gems/sidekiq-6.0.7/lib/sidekiq/processor.rb:125 :in `block (2 levels) in dispatch`
 [GEM_ROOT]/gems/sidekiq-6.0.7/lib/sidekiq/job_retry.rb:78 :in `global`
 [GEM_ROOT]/gems/sidekiq-6.0.7/lib/sidekiq/processor.rb:124 :in `block in dispatch`
 [GEM_ROOT]/gems/sidekiq-6.0.7/lib/sidekiq/logger.rb:10 :in `with`
 [GEM_ROOT]/gems/sidekiq-6.0.7/lib/sidekiq/job_logger.rb:33 :in `prepare`
 [GEM_ROOT]/gems/sidekiq-6.0.7/lib/sidekiq/processor.rb:123 :in `dispatch`
 [GEM_ROOT]/gems/sidekiq-6.0.7/lib/sidekiq/processor.rb:162 :in `process`
 [GEM_ROOT]/gems/sidekiq-6.0.7/lib/sidekiq/processor.rb:78 :in `process_one`
 [GEM_ROOT]/gems/sidekiq-6.0.7/lib/sidekiq/processor.rb:68 :in `run`
 [GEM_ROOT]/gems/sidekiq-6.0.7/lib/sidekiq/util.rb:15 :in `watchdog`
 [GEM_ROOT]/gems/sidekiq-6.0.7/lib/sidekiq/util.rb:24 :in `block in safe_thread`More (35 lines)
Nested Exceptions
NameError: instance variable @sequence_name not defined
 [GEM_ROOT]/gems/ros-apartment-2.6.1/lib/apartment/adapters/postgresql_adapter.rb:124 :in `remove_instance_variable`
 Sidekiq::JobRetry::Handled: Sidekiq::JobRetry::Handled
 [GEM_ROOT]/gems/sidekiq-6.0.7/lib/sidekiq/job_retry.rb:99 :in `rescue in global`
 [GEM_ROOT]/gems/sidekiq-6.0.7/lib/sidekiq/processor.rb:124 :in `block in dispatch`
 [GEM_ROOT]/gems/sidekiq-6.0.7/lib/sidekiq/logger.rb:10 :in `with`
 [GEM_ROOT]/gems/sidekiq-6.0.7/lib/sidekiq/job_logger.rb:33 :in `prepare`
 [GEM_ROOT]/gems/sidekiq-6.0.7/lib/sidekiq/processor.rb:123 :in `dispatch`
 [GEM_ROOT]/gems/sidekiq-6.0.7/lib/sidekiq/processor.rb:162 :in `process`
 [GEM_ROOT]/gems/sidekiq-6.0.7/lib/sidekiq/processor.rb:78 :in `process_one`
 [GEM_ROOT]/gems/sidekiq-6.0.7/lib/sidekiq/processor.rb:68 :in `run`
 [GEM_ROOT]/gems/sidekiq-6.0.7/lib/sidekiq/util.rb:15 :in `watchdog`
 [GEM_ROOT]/gems/sidekiq-6.0.7/lib/sidekiq/util.rb:24 :in `block in safe_thread`
 NameError: instance variable @sequence_name not defined
 [GEM_ROOT]/gems/ros-apartment-2.6.1/lib/apartment/adapters/postgresql_adapter.rb:124 :in `remove_instance_variable`
 [GEM_ROOT]/gems/ros-apartment-2.6.1/lib/apartment/adapters/postgresql_adapter.rb:123 :in `each`
 [GEM_ROOT]/gems/ros-apartment-2.6.1/lib/apartment/adapters/postgresql_adapter.rb:123 :in `reset_sequence_names`
 [GEM_ROOT]/gems/ros-apartment-2.6.1/lib/apartment/adapters/postgresql_adapter.rb:82 :in `connect_to_new`
 [GEM_ROOT]/gems/ros-apartment-2.6.1/lib/apartment/adapters/abstract_adapter.rb:77 :in `block in switch!`
 [GEM_ROOT]/gems/activesupport-5.2.4.2/lib/active_support/callbacks.rb:98 :in `run_callbacks`
 [GEM_ROOT]/gems/ros-apartment-2.6.1/lib/apartment/adapters/abstract_adapter.rb:76 :in `switch!`
/home/app_au.rbenv/versions/2.6.5/lib/ruby/2.6.0/forwardable.rb:230 :in `switch!`
 [PROJECT_ROOT]/lib/country_manager.rb:188 :in `country=`
 [PROJECT_ROOT]/config/initializers/sidekiq.rb:55 :in `set_country_from_job`
 [PROJECT_ROOT]/config/initializers/sidekiq.rb:27 :in `call`
 [GEM_ROOT]/gems/sidekiq-6.0.7/lib/sidekiq/middleware/chain.rb:140 :in `block in invoke`
 [GEM_ROOT]/gems/sidekiq-6.0.7/lib/sidekiq/middleware/chain.rb:140 :in `block in invoke`
 [GEM_ROOT]/gems/sidekiq-6.0.7/lib/sidekiq/middleware/chain.rb:143 :in `invoke`
 [GEM_ROOT]/gems/sidekiq-6.0.7/lib/sidekiq/processor.rb:163 :in `block in process`
 [GEM_ROOT]/gems/sidekiq-6.0.7/lib/sidekiq/processor.rb:136 :in `block (6 levels) in dispatch`
 [GEM_ROOT]/gems/sidekiq-6.0.7/lib/sidekiq/job_retry.rb:111 :in `local`
 [GEM_ROOT]/gems/sidekiq-6.0.7/lib/sidekiq/processor.rb:135 :in `block (5 levels) in dispatch`
 [GEM_ROOT]/gems/sidekiq-6.0.7/lib/sidekiq/rails.rb:43 :in `block in call`
 [GEM_ROOT]/gems/activesupport-5.2.4.2/lib/active_support/execution_wrapper.rb:87 :in `wrap`
 [GEM_ROOT]/gems/activesupport-5.2.4.2/lib/active_support/reloader.rb:73 :in `block in wrap`
 [GEM_ROOT]/gems/activesupport-5.2.4.2/lib/active_support/execution_wrapper.rb:87 :in `wrap`
 [GEM_ROOT]/gems/activesupport-5.2.4.2/lib/active_support/reloader.rb:72 :in `wrap`
 [GEM_ROOT]/gems/sidekiq-6.0.7/lib/sidekiq/rails.rb:42 :in `call`
 [GEM_ROOT]/gems/sidekiq-6.0.7/lib/sidekiq/processor.rb:131 :in `block (4 levels) in dispatch`
 [GEM_ROOT]/gems/sidekiq-6.0.7/lib/sidekiq/processor.rb:257 :in `stats`
 [GEM_ROOT]/gems/sidekiq-6.0.7/lib/sidekiq/processor.rb:126 :in `block (3 levels) in dispatch`
 [GEM_ROOT]/gems/sidekiq-6.0.7/lib/sidekiq/job_logger.rb:13 :in `call`
 [GEM_ROOT]/gems/sidekiq-6.0.7/lib/sidekiq/processor.rb:125 :in `block (2 levels) in dispatch`
 [GEM_ROOT]/gems/sidekiq-6.0.7/lib/sidekiq/job_retry.rb:78 :in `global`
 [GEM_ROOT]/gems/sidekiq-6.0.7/lib/sidekiq/processor.rb:124 :in `block in dispatch`
 [GEM_ROOT]/gems/sidekiq-6.0.7/lib/sidekiq/logger.rb:10 :in `with`
 [GEM_ROOT]/gems/sidekiq-6.0.7/lib/sidekiq/job_logger.rb:33 :in `prepare`
 [GEM_ROOT]/gems/sidekiq-6.0.7/lib/sidekiq/processor.rb:123 :in `dispatch`
 [GEM_ROOT]/gems/sidekiq-6.0.7/lib/sidekiq/processor.rb:162 :in `process`
 [GEM_ROOT]/gems/sidekiq-6.0.7/lib/sidekiq/processor.rb:78 :in `process_one`
 [GEM_ROOT]/gems/sidekiq-6.0.7/lib/sidekiq/processor.rb:68 :in `run`
 [GEM_ROOT]/gems/sidekiq-6.0.7/lib/sidekiq/util.rb:15 :in `watchdog`
 [GEM_ROOT]/gems/sidekiq-6.0.7/lib/sidekiq/util.rb:24 :in `block in safe_thread`

Any idea how I could further debug this?

@rpbaltazar
Copy link
Contributor

I'm as puzzled as you are. Can you somehow make it happen locally?
I'm wondering if there is any concurrency issue in that particular piece of code. The quick fix for that will be me checking again if the instance variable exists.

e.g
c.remove_instance_variable :@sequence_name if c.instance_variable_defined?(:@sequence_name)
and see if this solves it.

I can prepare a branch from your current version and add this line and you can check if it fixes it? would that help?

@rpbaltazar
Copy link
Contributor

If you can, please run this branch: https://github.com/rails-on-services/apartment/tree/test-reported-issue-81
I;ve added some put in case the var is not found. maybe it helps you understanding what is going on based on the context.

@rpbaltazar rpbaltazar added the bug Something isn't working label Jul 12, 2020
@rpbaltazar rpbaltazar linked a pull request Jul 12, 2020 that will close this issue
@rpbaltazar
Copy link
Contributor

For now we'll be avoiding the error by checking for the instance variable existence before removal. The only situation that i can think of is that somehow in a parallel execution the variable has been removed an not yet re-calculated thus throwing the error. I'll be investigating further the details of the implementation and see if there is a better longer term solution to avoid this whole logic in the first place

fsateler added a commit to fsateler/apartment that referenced this issue Jan 6, 2021
sequence_name is shared across threads because it is stored in a class
ivar. This means that on threaded servers, the sequence_name might point
to a different tenant, if another tenant made the switch at the right
time. This race is likely the cause of
rails-on-services#81 as well.

This reverts commit f8eefc4.
fsateler added a commit to fsateler/apartment that referenced this issue Jan 6, 2021
sequence_name is shared across threads because it is stored in a class
ivar. This means that on threaded servers, the sequence_name might point
to a different tenant, if another tenant made the switch at the right
time. This race is likely the cause of
rails-on-services#81 as well.

This reverts commit f8eefc4.
fsateler added a commit to fsateler/apartment that referenced this issue Jan 7, 2021
sequence_name is shared across threads because it is stored in a class
ivar. This means that on threaded servers, the sequence_name might point
to a different tenant, if another tenant made the switch at the right
time. This race is likely the cause of
rails-on-services#81 as well.

This reverts commit f8eefc4.
This was referenced Feb 2, 2022
rpbaltazar added a commit that referenced this issue Feb 7, 2022
**Implemented enhancements:**

- Increase errors visibility by showing more information on the underlying error rather than a generic error 'Apartment::TenantNotFound' (#176)
- Resolved #177 - Added rails 7 support (#178)

**Fixed bugs:**

- Fixing tenant_presence_check config in the README (#180)
- Resolved #161 and #81 - Fixed sequence name (#187)

**Closed issues:**

- Resolved #151 - removed reloader and console overwrite of reload method (#174)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
2 participants