Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Relay. This is a single entry in the upcoming DiskLruCache2. #2750

Merged
merged 1 commit into from
Aug 2, 2016

Conversation

swankjesse
Copy link
Collaborator

At the moment there are some things still incomplete:

  • Timeouts. Not yet tested, and no great implementation plan.
  • Thread interrupts. This has been quite problematic for
    the original DiskLruCache. We should test and support this
    in Relay.
  • File system problems. If the host file system is full or not
    accepting writes, we should fallback gracefully. This may
    be difficult.
  • Metadata updates. At the moment the metadata is permanent
    and immutable.

That said, many nice features are already implemented:

  • Metadata and stream data in one file.
  • No locks held during file I/O or network I/O.
  • Read from the buffer, the file, or the socket in that order.
  • Reference counting on open sources. It is assumed that the
    creator of each Relay will take a source, and that guarantees
    that the count will always reach 0.

}

// Read from the file.
if (source == SOURCE_FILE) {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice

@dave-r12
Copy link
Collaborator

I'm still absorbing some bits and pieces but the logic is straightforward given the problem this is solving. Good stuff.

@swankjesse
Copy link
Collaborator Author

After a tiny amount of soul searching I decided what to do about timeouts. I’ll do the work in a follow up PR.

Whenever a thread reads from upstream it’ll first configure upstream’s timeouts from its own timeouts. So if a thread has a 3 second timeout it’ll use that to read upstream, even if other waiting threads have longer or shorter timeouts.

I figure this is pretty reasonable because in practice all of the threads in the same HTTP client reading the same resource will inevitably have the same timeouts anyway.

@dave-r12
Copy link
Collaborator

Any reason to hold this one up? LGTM.

@swankjesse
Copy link
Collaborator Author

I pitched this design to Jake before I wrote the code so I’m hoping to get his eyes on it too. He’s been tied up but should be back soon.

At the moment there are some things still incomplete:
 - Timeouts. Not yet tested, and no great implementation plan.
 - Thread interrupts. This has been quite problematic for
   the original DiskLruCache. We should test and support this
   in Relay.
 - File system problems. If the host file system is full or not
   accepting writes, we should fallback gracefully. This may
   be difficult.
 - Metadata updates. At the moment the metadata is permanent
   and immutable.

That said, many nice features are already implemented:
 - Metadata and stream data in one file.
 - No locks held during file I/O or network I/O.
 - Read from the buffer, the file, or the socket in that order.
 - Reference counting on open sources. It is assumed that the
   creator of each Relay will take a source, and that guarantees
   that the count will always reach 0.
@swankjesse swankjesse force-pushed the jwilson.0725.source_relay branch from 6a3a6ad to 913f16b Compare July 30, 2016 14:54
@JakeWharton
Copy link
Collaborator

lgtm. this is dense. had to give it two passes.

@swankjesse
Copy link
Collaborator Author

Yeah! More exciting stuff to come. Thanks @dave-r12 and @JakeWharton for the reviews!

@swankjesse swankjesse merged commit 8504f3a into master Aug 2, 2016
@swankjesse swankjesse deleted the jwilson.0725.source_relay branch October 16, 2016 23:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants