serviceLocator: allow replacing an already-registered service (storage client / event manager / configuration)

[B4nan]

Problem

ServiceLocator throws a ServiceConflictError whenever a service that is already registered is set again to a different instance:

• setConfiguration(config)
• setStorageClient(client)
• setEventManager(manager)

The only way to clear a registered service is serviceLocator.reset(), which drops all services at once (configuration, storage client, event manager, logger, and the storage-instance-manager cache). There is no way to replace a single service.

The conflict guard usefully prevents accidental double-registration, but it also makes several legitimate replacement scenarios impossible without nuking everything.

Concrete impact (Apify SDK v4 migration)

1. Actor.init() cannot run when a storage client is already registered. On the platform, Actor.init() registers an ApifyStorageClient. If anything has already set a storage client — a test harness using @crawlee/memory-storage, a second Actor.init(), or a user who configured their own client before init — setStorageClient throws.

2. Tests can't swap the storage client. A common pattern is: let Actor.init() register the real client, then swap in an in-memory storage to assert against pushed data. Under v4 this is impossible — reset() would also discard the configuration and event manager that init() just established.

In crawlee v3 the equivalent (Configuration.useStorageClient) allowed replacement, so this is a behavioural regression for these flows.

Repro (sketch)

import { serviceLocator } from '@crawlee/core';
import { MemoryStorage } from '@crawlee/memory-storage';

serviceLocator.setStorageClient(new MemoryStorage());
serviceLocator.setStorageClient(new MemoryStorage());
// throws: Service StorageClient is already in use.

Possible solutions

1. Last-write-wins — make setX() overwrite an existing service (optionally with a log.warning on replacement). Simplest, but loses the accidental-double-set guard.
2. Explicit override flag — setStorageClient(client, { override: true }); keeps the guard by default, allows intentional replacement.
3. Granular per-service reset — add resetStorageClient() / resetEventManager() / resetConfiguration() so a caller can clear one service without dropping the rest.
4. Allow replacement until first read — track whether a service has been consumed (e.g. getStorageClient() called) and allow replacement until then, throwing only on replace-after-use.

Preference is (2) or (3): they keep the conflict guard for the common accidental case while making intentional replacement an explicit, first-class operation.

Happy to open a PR once we agree on the shape.

[B4nan]

Closing — this turned out not to be a blocker for the Apify SDK v4 migration. The SDK works around it in tests by fully resetting the serviceLocator (drop everything + re-register) rather than replacing a single service. The underlying ergonomic wart (no way to replace or reset an individual registered service without dropping all of them) still stands, but it's a nice-to-have DX improvement rather than a bug, so closing for now. Happy to reopen if it bites a real use case.

[janbuchar]

To make sure that we don't needlessly re-learn the same lessons as we did in crawlee-python, here's a chronological account of every meaningful iteration, with the reasoning behind each pivot. TL;DR up front: we tried essentially every option listed above, and the one we landed on is none of them — we made the global locator strictly set-once and moved the "replaceability" problem out of the locator entirely by giving each crawler its own ServiceLocator instance — that's what crawlee-js currently does.

---

Iteration 0 — service_container (pre-history)

Before there was a ServiceLocator, services lived in a module called service_container with free functions and a force=True flag on the setters. This is the ancestor; it never made it into _service_locator.py under that name.

---

Iteration 1 — Birth of ServiceLocator — "set-once, guarded by a flag"

• PR: #691 — fix!: Refactor service usage to rely on service_locator
• Commit: 1d31c6c7 (2024-12-13), seeded by 6ee11904 "better service locator"
• Closes: Streamline component management - unified access via service_container crawlee-python#699, Crawler doesn't respect configuration argument crawlee-python#539, How can I disable cache completely? crawlee-python#369 (unlocks Fix the usage of Configuration class crawlee-python#670)

Mechanism: Three services (Configuration, EventManager, BaseStorageClient), each lazily defaulted. Each setter guarded by a _X_was_set boolean. Setting an already-set service → ServiceConflictError. The only escape hatch was a force flag on the setters (explicitly "especially useful for testing").

Reasoning: Centralize service ownership. Components stopped storing their own service instances and started reading from the global container. This is the "last-write-wins guarded by an explicit override flag" shape — i.e. option (2) in the JS issue. The force flag was the override.

Why it didn't stick: A global mutable singleton with a force override is a footgun. Anyone, anywhere, could silently stomp the global config as a side effect. We backed away from force.

---

Iteration 2 — "Allow replacement until first read" (the _was_retrieved era)

• PR: #837 — fix: Relax ServiceLocator restrictions
• Commit: aa3667f3 (2024-12-20), via 19940d6b
• Closes: Refactor BasicCrawler so that it works better with Apify SDK init crawlee-python#806

Mechanism: Renamed every _X_was_set flag to _X_was_retrieved. The flag now flips inside get_*(), not set_*(). So you could call set_storage_client() as many times as you liked until something first read it; after the first get_*(), any set_*() threw ServiceConflictError.

Reasoning: The strict set-once guard (Iteration 1) was too rigid for legitimate setup flows — e.g. the SDK's Actor.init() wanting to register a client after the user had already touched the locator. "Until first read" felt like a safe middle ground: replacement is fine while nothing depends on the value yet.

This is precisely option (4) in #3710 ("Allow replacement until first read"). We shipped it, lived with it for ~7 months, and then deliberately removed it — see Iteration 4.

This _was_retrieved model survived several refactors unchanged:

• #828 (0ba04d16) — directory restructure, locator semantics untouched.
• #899 (0f1cf6f0) — wired Configuration into the lazy EventManager/StorageClient defaults; semantics untouched.

---

Iteration 3 — Storage client system rewrite (carried _was_retrieved forward)

• PR: #1194 — refactor!: Introduce new storage client system
• Commit: de1c03f7 (2025-07-01)
• Closes: Separate MemoryStorageClient and FilesystemStorageClient crawlee-python#92, Improve storage manager and merge it with creation_management module crawlee-python#147, Introduce new base & memory storage clients crawlee-python#783, Add a guide for storage clients crawlee-python#1247

Mechanism: Swapped BaseStorageClient → new StorageClient interface, default became FileSystemStorageClient (split out from the old "memory+persist" client), and added a storage_instance_manager property to the locator (instance caching). The _was_retrieved replace-until-read logic was kept verbatim.

Reasoning: This was a storage redesign, not a locator redesign — but it's the point where the locator gained ownership of the StorageInstanceManager cache, which becomes important next.

---

Iteration 4 — The big pivot: per-instance locators + strict set-once (current model)

• PR: #1386 — refactor!: Refactor storage creation and caching, configuration and services
• Commit: 04649bde (2025-09-16)
• Closes: StorageInstanceManager should support caching based on used configuration crawlee-python#1379 (connected to All crawlee persistence names should be valid names for Apify platform crawlee-python#1354, Add specialized/parametrized RequestQueueClients apify-sdk-python#513)

This is the most important commit. It explicitly abandons "replace-until-read" (option 4) and goes the opposite direction. Two changes, made together because they're inseparable:

A. The global locator became strictly set-once. Quoting the commit verbatim:

Services in ServiceLocator can be set only once. Any attempt to reset them will throw an Error... ServiceLocator is now way more strict... Previously, it allowed changing services until some service had _was_retrieved flag set to True. ...This led to hard-to-predict code as the global service_locator could be changed as a side effect from many places.

The _was_retrieved flags were deleted. The new guard is dead simple: if self._x is x → no-op; if self._x already set → ServiceConflictError. Implicit (lazy-default) services now emit a warning log because relying on them is "hard-to-predict."

B. ServiceLocator stopped being a pure singleton — every crawler gets its own. This is the part that actually solves the problem the JS issue is wrestling with. Instead of making the global locator replaceable, we made the locator cheap to instantiate per consumer:

# BasicCrawler.__init__ (current code)
if not storage_client:
    storage_client = service_locator.get_storage_client()   # fall back to global
if not event_manager:
    event_manager = service_locator.get_event_manager()
self._service_locator = ServiceLocator(
    configuration=configuration, storage_client=storage_client, event_manager=event_manager
)

Each crawler resolves its services from explicit constructor args, falling back to the global locator only for what wasn't provided. The crawler can no longer cause side effects in the global locator. This is what unblocked "multiple crawlers with different storage clients / configs / event managers running in the same process" — previously that threw ServiceConflictError.

The StorageInstanceManager stayed global (shared cache across all locators) and gained an additional_cache_key so two storages that differ only by Configuration (e.g. different storage_dir) don't collide in the cache.

Reasoning / the lesson for #3710: We concluded that making a global mutable service registry replaceable is the wrong layer to solve it at — it just relocates the side-effect footgun. The durable fix was to scope the locator to the consumer so replacement becomes "construct a new locator with the services you want" rather than "mutate the global one." The global locator is then free to be brutally strict (set-once) because nothing legitimately needs to mutate it after the fact.

---

Iteration 5 — Log-level cleanup

• #1489 — chore: Reduce log level in service_locator logs (e2beda10, 2025-10-16, closes [FIX] Update Dockerfile to fix build error #1468): the implicit-service warnings from Iteration 4 turned out to be too noisy → demoted to debug.
• #1746 — fix: Make log levels consistent in ServiceLocator (41634130 / 55727914, 2026-02-13): follow-up consistency pass.

Reasoning: Pure ergonomics fallout from the warning-on-implicit-use decision. Worth noting because it shows the "warn on implicit/default service" idea (the JS issue's option 1 "with a log.warning") has a real noise cost.

---

Iteration 6 — reset() (in-flight, not merged)

• Branch: add-reset-to-service-locator (commit 54d9a17c, 2026-02-19) — not on master.

def reset(self) -> None:
    """Reset the service locator."""
    self._configuration = None
    self._event_manager = None
    self._storage_client = None

A blunt all-services reset on the instance. This is the JS issue's option (3) at its crudest — note it's all-or-nothing, exactly the reset() the JS issue complains about, not the granular per-service version. It's sitting unmerged, which tells you we weren't satisfied with it either.

---

Summary table

#	PR	Date	Approach	Maps to #3710 option	Verdict
1	#691	2024-12	Set-once + force override flag	(2) explicit override	force abandoned
2	#837	2024-12	Replace-until-first-read (_was_retrieved)	(4) replace until read	shipped, later removed
3	#1194	2025-07	(carried #2 forward)	(4)	superseded
4	#1386	2025-09	Strict set-once global + per-crawler locator instances	none — sidesteps the question	current model
5	#1489, #1746	2025-10 → 2026-02	Log-level tuning	—	ergonomics
6	add-reset-to-service-locator	2026-02	All-services reset()	(3) crude	unmerged

---

Whatwe should take away

1. We literally shipped option (4) ("replace until first read") and then ripped it out in refactor!: Refactor storage creation and caching, configuration and services crawlee-python#1386. The stated reason: it makes the global registry mutable-as-a-side-effect from many call sites, producing "hard-to-predict code." If the JS serviceLocator is a true global singleton, option (4) will bite you the same way.

2. The force override (option 2) was our v1 and we walked away from it for the same footgun reason.

3. The fix that actually stuck wasn't any of the four options — it was making the locator instantiable per consumer (each crawler owns one, falling back to the global) so the global can stay strictly set-once. "Replacement" becomes "hand the consumer a different locator," not "mutate the shared one." For the SDK Actor.init() / test-swap scenarios in serviceLocator: allow replacing an already-registered service (storage client / event manager / configuration) #3710, that's the angle worth evaluating before adding override/resetX() to the global.

4. A blunt reset() exists but we left it unmerged — even we don't love all-or-nothing reset, which is exactly the complaint in the issue.

One asymmetry to flag: the Python Configuration/StorageClient/EventManager split maps cleanly onto the JS trio, but Python never had a per-service granular reset (option 3) merged — so if JS wants prior art for resetStorageClient(), we don't have it. We have prior art for why not to make the global mutable, and for the per-instance escape hatch instead.