Stream the result from Search Backend instead of use List. #481

davidclement90 · 2017-08-28T12:49:09Z

Issue : #480

I can not use cursors for Solr because sort clause must include the unique key field
(https://cwiki.apache.org/confluence/display/solr/Pagination+of+Results)

Signed-off-by: David Clement david.clement90@laposte.net

sjudeng

Nice work! I'm looking forward to reviewing in more detail. A couple general questions up front.

It looks like the scroll api is always used now? Or can standard search api still be used with configuration update or based on situation (e.g. result count)? It seems that there are use cases to prefer search over scroll api. In particular Elasticsearch scroll documentation says "scrolling is not intended for real time user requests". Also would users of some raw index query features (see Actually process raw query parameters. Issue 338 #423) require the search instead of scroll api?
Depending on the answer to the above, are there tests that hit both scroll/non-scroll branches?
Why was max-result-size configuration item removed? I think having an overall max-result-size configuration is still valuable, regardless of whether results are being paginated or scrolled, or is this enforced somewhere else now?

sjudeng · 2017-08-29T02:45:20Z

janusgraph-core/src/main/java/org/janusgraph/graphdb/transaction/StandardJanusGraphTx.java

@@ -1256,7 +1258,7 @@ else if (query.getResultType() == ElementCategory.VERTEX) {
            Iterator<JanusGraphElement> iter;
            if (!indexQuery.isEmpty()) {
                List<QueryUtil.IndexCall<Object>> retrievals = new ArrayList<QueryUtil.IndexCall<Object>>();
-                for (int i = 0; i < indexQuery.size(); i++) {
+                for (int i = 1; i < indexQuery.size(); i++) {


Why does the start index change from 0 to 1?

Because I use the first query to initialize SubqueryIterator on line 1280

iter = new SubqueryIterator(indexQuery.getQuery(0), indexSerializer, txHandle, indexCache, indexQuery.getLimit(), getConversionFunction(query.getResultType()), retrievals.isEmpty() ? null: QueryUtil.processIntersectingRetrievals(retrievals, indexQuery.getLimit()));

davidclement90 · 2017-08-30T08:43:47Z

It looks like the scroll api is always used now? Or can standard search api still be used with configuration update or based on situation (e.g. result count)? It seems that there are use cases to prefer search over scroll api. In particular Elasticsearch scroll documentation says "scrolling is not intended for real time user requests". Also would users of some raw index query features (see #423) require the search instead of scroll api?
Yes but from my experience, they just the that because keep scrool context for nothing take memory and user do not delete scroll context.
But I just change the code to not use scroll during totals method and if the limit is under nbResultByQuery.

Depending on the answer to the above, are there tests that hit both scroll/non-scroll branches?
There are already tests that cover both cases

Why was max-result-size configuration item removed? I think having an overall max-result-size configuration is still valuable, regardless of whether results are being paginated or scrolled, or is this enforced somewhere else now?
Why is still valuable ? Because for Elasticsearch with scroll it's useless. I already scroll over billions of results. For me max-result-size is use because you can not ask in one query a lot of results.
I did the PR for two goals :

Speed-up the first result
Deep paging

For Solr, I'm not sure because I can not use cursor. I may need to restore max-result-size for Solr ?

sjudeng · 2017-08-30T17:45:25Z

Thanks for your responses, they all make good sense. I like the change to use the search api if result is less than nbResultByQuery. Rather than adding the new parameter what about keeping max-result-size and just updating documentation to indicate that for index backends that support scrolling it will represent the number of results in each batch?

Regarding deleting the scroll context, it looks like you're just clearing after competing iteration over the given batch? Do you handle cases where iteration does not complete (e.g. client code stops iterating early, exceptions occur during processing, etc.), in terms of clearing the last scroll? If not I'm wondering about implications for the case where the same graph stays open during the (potentially long) lifetime of an application, where these uncleared scrolls might stack up.

sjudeng

Some feedback from first pass through.

sjudeng · 2017-08-30T21:52:07Z

janusgraph-core/src/main/java/org/janusgraph/graphdb/database/IndexSerializer.java

        IndexType index = query.getIndex();
        if (index.isCompositeIndex()) {
            MultiKeySliceQuery sq = query.getCompositeQuery();
            List<EntryList> rs = sq.execute(tx);
-            List<Object> results = new ArrayList<Object>(rs.get(0).size());
+            List<Object> results = new ArrayList<>(rs.get(0).size());


final here and throughout PR

sjudeng · 2017-08-30T22:06:13Z

janusgraph-solr/src/main/java/org/janusgraph/diskstorage/solr/SolrIndex.java

-        final QueryResponse response = runCommonQuery(query, informations, tx, collection, keyIdField);
-        logger.debug("Executed query [{}] in {} ms", query.getQuery(), response.getElapsedTime());
-        return response.getResults().getNumFound();
+    try {


Fix indentation

sjudeng · 2017-08-30T22:43:52Z

janusgraph-solr/src/main/java/org/janusgraph/diskstorage/solr/SolrResultIterator.java

+
+    @Override
+    public boolean hasNext() {
+        if(cpt!=0 && cpt % nbDocByQuery == nbDocByQuery/2 && nbDocByQuery * nbRequest < limit){


Can you add tests to hit any missing cases here (e.g. actual_count<nbDocByQuery, actual_count==nbDocByQuery, actual_count>nbDocByQuery, limit==actual_count,limit%nbDocByQuery==0, etc.)?

Why is the middle condition not cpt % nbDocByQuery == 0?

Add whitespace if (...) {

It's not already tested ? I can add tests.

At the begin of the PR, i would like to delegate the search to a stream (so I asked to the stream when i consumed half of the previous bulk), but I have some issue so I split the PR in two.

sjudeng · 2017-08-30T22:56:30Z

janusgraph-solr/src/main/java/org/janusgraph/diskstorage/solr/SolrResultIterator.java

+        this.collection = collection;
+        this.solrQuery = solrQuery;
+        this.function = function;
+        long nbFound = solrClient.queryAndStreamResponse(collection, solrQuery, new SolrCallbackHandler(this, function)).getResults().getNumFound() - offset;


IO/SolrExceptions here are propagated to SolrIndex, which wraps them as a PermanentBackendException, whereas below (hasNext/next) RuntimeExceptions are being thrown directly to the client (e.g. uncaught in SolrIndex). I think the behavior should be consistent with top-level exception as PermanentBackendException. I wouldn't catch generic RuntimeException in SolrIndex, but maybe propagate and catch a more specific/custom implementation below?

I'm in a stream inside a map or an iterator.
So I have to throw RuntimeException and It's have to be catch when the stream is collected ?
So I just create a specific RuntimeException

sjudeng · 2017-08-30T22:57:39Z

janusgraph-solr/src/main/java/org/janusgraph/diskstorage/solr/SolrResultIterator.java

+                solrClient.queryAndStreamResponse(collection, solrQuery, new SolrCallbackHandler(this, function));
+                nbRequest++;
+            } catch (SolrServerException | IOException e) {
+                throw new RuntimeException(e);


Emit something more specific and then catch and wrap as PermanentBackendException in SolrIndex?

sjudeng · 2017-08-30T23:09:04Z

janusgraph-solr/src/main/java/org/janusgraph/diskstorage/solr/SolrIndex.java

-                                .setRows(query.hasLimit() ? query.getLimit() : maxResults);
+                                .setStart(query.getOffset());
+        if (query.hasLimit()) solrQuery.setRows(Math.min(query.getLimit(), nbResultByQuery));
+        else solrQuery.setRows(nbResultByQuery);


Newlines and curly brackets

sjudeng · 2017-08-30T23:14:09Z

janusgraph-es/src/main/java/org/janusgraph/diskstorage/es/ElasticSearchIndex.java

-        if (query.hasLimit()) sr.setSize(query.getLimit());
-        else sr.setSize(maxResultsSize);
+        if (query.hasLimit()) sr.setSize(Math.min(query.getLimit(), nbResultByQuery));
+        else sr.setSize(nbResultByQuery);


Newlines and curly braces

sjudeng · 2017-08-30T23:16:32Z

janusgraph-es/src/main/java/org/janusgraph/diskstorage/es/ElasticSearchScroll.java

+/**
+ * @author David Clement (david.clement90@laposte.net)
+ */
+public class ElasticSearchScroll implements Iterator<RawQuery.Result<String>>{


White space before curly braces. Please fix here and throughout PR.

sjudeng · 2017-08-30T23:17:40Z

janusgraph-es/src/main/java/org/janusgraph/diskstorage/es/ElasticSearchScroll.java

+            if (isFinished) client.deleteScroll(scrollId);
+            return res.nbResults() > 0;
+        } catch (IOException e) {
+             throw new RuntimeException(e);


Consistent exception handling (see comments in SolrResultIterator)

sjudeng · 2017-08-30T23:17:44Z

janusgraph-es/src/main/java/org/janusgraph/diskstorage/es/ElasticSearchScroll.java

+         try {
+             return queue.take();
+         } catch (InterruptedException e) {
+              throw new RuntimeException(e);


Consistent exception handling (see comments in SolrResultIterator)

davidclement90 · 2017-08-31T15:14:54Z

Thanks for your responses, they all make good sense. I like the change to use the search api if result is less than nbResultByQuery. Rather than adding the new parameter what about keeping max-result-size and just updating documentation to indicate that for index backends that support scrolling it will represent the number of results in each batch?
Ok

Regarding deleting the scroll context, it looks like you're just clearing after competing iteration over the given batch? Do you handle cases where iteration does not complete (e.g. client code stops iterating early, exceptions occur during processing, etc.), in terms of clearing the last scroll? If not I'm wondering about implications for the case where the same graph stays open during the (potentially long) lifetime of an application, where these uncleared scrolls might stack up.
Elasticsearch do not keep the scroll context for life (https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-scroll.html#scroll-search-context), you can choose the duration with scroll-keep-alive by default it's 60 seconds.

sjudeng

Some more feedback is below. I think limiting the custom RuntimeException just to the Solr module would be preferable to PermanentBackendRuntimeException in core. In Elasticsearch and Lucene the IOExceptions can be wrapped in the existing UncheckedIOException. It's also important to catch whatever is thrown in ElasticsearchIndex and SolrIndex so the final exception to the client is consistent (e.g. PermanentBackendException).

I still think adding the scroll implementation around the nbResultsByQuery (or batchSize) does need tests that probably wouldn't have existed in the existing single-batch implementation. Cases to cover if they're not already: actual_count<batchSize, actual_count==batchSize, actual_count>batchSize, limit==actual_count, limit%batchSize==0, etc..

sjudeng · 2017-09-02T12:45:45Z

janusgraph-core/src/main/java/org/janusgraph/diskstorage/PermanentBackendRuntimeException.java

+        this("Permanent failure in storage backend", cause);
+    }
+
+


Delete extra newline

sjudeng · 2017-09-02T12:49:51Z

janusgraph-core/src/main/java/org/janusgraph/diskstorage/indexing/IndexProvider.java

@@ -79,11 +80,11 @@
     * @param query Query to execute
     * @param informations Information on the keys used in the query accessible through {@link KeyInformation.IndexRetriever}.
     * @param tx Enclosing transaction
-     * @return The ids of all matching documents
+     * @return An iterator of all matching documents ids


I think you could keep language as is "The ids of all matching documents" but if you want to change it looks like it should be stream, not iterator.

sjudeng · 2017-09-02T12:50:02Z

janusgraph-core/src/main/java/org/janusgraph/diskstorage/indexing/IndexProvider.java

@@ -92,11 +93,11 @@
     * @param query Query to execute
     * @param informations Information on the keys used in the query accessible through {@link KeyInformation.IndexRetriever}.
     * @param tx Enclosing transaction
-     * @return Results objects for all matching documents (i.e. document id and score)
+     * @return Iterator on objects for all matching documents (i.e. document id and score)


Ditto ... I think you could keep language as is but if you want to change it looks like it should be stream, not iterator.

sjudeng · 2017-09-02T12:50:32Z

...raph-core/src/main/java/org/janusgraph/graphdb/configuration/GraphDatabaseConfiguration.java

@@ -892,8 +892,9 @@ public boolean apply(@Nullable Integer uniqueIdBitWidth) {
            ConfigOption.Type.MASKABLE, String.class);

    public static final ConfigOption<Integer> INDEX_MAX_RESULT_SET_SIZE = new ConfigOption<Integer>(INDEX_NS, "max-result-set-size",
-            "Maxium number of results to return if no limit is specified",
-            ConfigOption.Type.MASKABLE, 100000);
+            "Maxium number of results to return if no limit is specified. For index backends that support scrolling, it represent " +


/represent/represents/

sjudeng · 2017-09-02T13:37:42Z

janusgraph-solr/src/main/java/org/janusgraph/diskstorage/solr/SolrIndex.java

@@ -218,7 +223,7 @@ public static Mode parse(String mode) {
    private final boolean dynFields;
    private final Map<String, String> keyFieldIds;
    private final String ttlField;
-    private final int maxResults;
+    private final int nbResultByQuery;


How about "batchSize" instead of "nbResultByQuery"?

sjudeng · 2017-09-02T17:31:07Z

janusgraph-core/src/main/java/org/janusgraph/graphdb/query/profile/QueryProfiler.java

@@ -110,4 +111,11 @@ public void setResultSize(long size) {
        return result;
    }

+    public static QueryProfiler startProfile(QueryProfiler profiler, Subquery query) {
+        QueryProfiler sub = profiler.addNested("backend-query");


sjudeng · 2017-09-02T17:32:05Z

janusgraph-core/src/main/java/org/janusgraph/graphdb/util/StreamIterable.java

+ */
+public class StreamIterable<E> implements Iterable<E>{
+
+    private Stream<E> stream;


sjudeng · 2017-09-02T17:32:35Z

janusgraph-core/src/main/java/org/janusgraph/graphdb/util/SubqueryIterator.java

+
+    private final Cache<JointIndexQuery.Subquery, List<Object>> indexCache;
+
+    private Iterator<? extends JanusGraphElement> itElement;


/itElement/elementIterator/

sjudeng · 2017-09-02T17:34:46Z

janusgraph-core/src/main/java/org/janusgraph/graphdb/util/SubqueryIterator.java

+            Function<Object, ? extends JanusGraphElement> function, List<Object> otherResults) {
+        this.subQuery = subQuery;
+        this.indexCache = indexCache;
+        List<Object> cacheResponse;


final here and throughout PR

This one is not final because cacheResponse is set on line 59 but for all the others and even more I update my eclipse save actions.
We need to work on #242...

It can be final. It's just declared here and only initialized once (on L59).

sjudeng · 2017-09-02T17:37:05Z

janusgraph-es/src/main/java/org/janusgraph/diskstorage/es/ElasticSearchIndex.java

@@ -187,7 +194,7 @@
    private final AbstractESCompat compat;
    private final ElasticSearchClient client;
    private final String indexName;
-    private final int maxResultsSize;
+    private final int nbResultByQuery;


/nbResultsByQuery/batchSize/

davidclement90 · 2017-09-04T13:55:56Z

I only add test for Solr implementation of Scroll, because ES one, it's base on Elasticsearch mechanism (respond empty when the scroll is over) and Lucene implementation does not change anymore.

sjudeng

Looking good. Some more feedback on latest updates.

sjudeng · 2017-09-04T21:27:54Z

janusgraph-core/src/main/java/org/janusgraph/diskstorage/UncheckedIOException.java

+ * @author davidclement@laposte.net
+ *
+ */
+public class UncheckedIOException extends RuntimeException {


Don't need this class as there's already a java.io.UncheckedIOException .

sjudeng · 2017-09-04T21:31:26Z

janusgraph-es/src/main/java/org/janusgraph/diskstorage/es/ElasticSearchSetup.java

@@ -91,7 +91,9 @@ public Connection connect(Configuration config) throws IOException {
            }
            RestClient rc = RestClient.builder(hosts.toArray(new HttpHost[hosts.size()])).build();

-            RestElasticSearchClient client = new RestElasticSearchClient(rc);
+            int scrollKeepAlive = config.get(ElasticSearchIndex.ES_SCROLL_KEEP_ALIVE);


sjudeng · 2017-09-04T21:31:33Z

janusgraph-es/src/main/java/org/janusgraph/diskstorage/es/ElasticSearchSetup.java

-            RestElasticSearchClient client = new RestElasticSearchClient(rc);
+            int scrollKeepAlive = config.get(ElasticSearchIndex.ES_SCROLL_KEEP_ALIVE);
+            Preconditions.checkArgument(scrollKeepAlive >= 1, "Scroll Keep alive should be greater or equals than 1");
+            RestElasticSearchClient client = new RestElasticSearchClient(rc, scrollKeepAlive);


sjudeng · 2017-09-04T21:31:46Z

janusgraph-es/src/main/java/org/janusgraph/diskstorage/es/rest/RestElasticSearchClient.java

@@ -274,6 +279,26 @@ public RestSearchResponse search(String indexName, String type, Map<String,Objec
        }
    }

+    @Override
+    public RestSearchResponse search(String scrollId) throws IOException {
+        Map<String, Object> request = new HashMap<>();


sjudeng · 2017-09-04T22:07:34Z

janusgraph-solr/src/test/java/org/janusgraph/diskstorage/solr/SolrIndexTest.java

@@ -151,4 +157,38 @@ public void testSupport() {
        assertTrue(index.supports(of(UUID.class, Cardinality.SINGLE), Cmp.NOT_EQUAL));
    }

+    @Test
+    public void testScroll() throws BackendException {


This test looks good (thanks for adding it). Can you elevate it to IndexProviderTest? I'd like to make sure we're covering all branches in ElasticSearchScroll as well. Though scrolling is not used in all implementations (e.g. Lucene) the test should still pass. If you want to exclude it from running under Lucene you could add an abstract boolean supportsBatching() or something to IndexProviderTest (or else add scrolling as an index feature).

sjudeng · 2017-09-05T00:59:11Z

janusgraph-core/src/main/java/org/janusgraph/graphdb/util/SubqueryIterator.java

+            Function<Object, ? extends JanusGraphElement> function, List<Object> otherResults) {
+        this.subQuery = subQuery;
+        this.indexCache = indexCache;
+        List<Object> cacheResponse;


It can be final. It's just declared here and only initialized once (on L59).

sjudeng · 2017-09-05T00:59:24Z

janusgraph-core/src/main/java/org/janusgraph/graphdb/util/SubqueryIterator.java

+        this.subQuery = subQuery;
+        this.indexCache = indexCache;
+        List<Object> cacheResponse;
+        Stream<?> stream;


Also can be final

sjudeng · 2017-09-05T01:06:08Z

janusgraph-es/src/main/java/org/janusgraph/diskstorage/es/ElasticSearchIndex.java


        ElasticSearchResponse response;
        try {
-            response = client.search(getIndexStoreName(query.getStore()), useMultitypeIndex ? query.getStore() : null, compat.createRequestBody(sr, NULL_PARAMETERS));
-        } catch (IOException e) {
+            response = client.search(getIndexStoreName(query.getStore()), useMultitypeIndex ? query.getStore() : null, compat.createRequestBody(sr, NULL_PARAMETERS), sr.getSize() >= batchSize);


Define index and type on separate lines to get this line length a little more manageable.

sjudeng · 2017-09-05T01:10:56Z

janusgraph-solr/src/main/java/org/janusgraph/diskstorage/solr/UncheckedSolrException.java

+    /**
+     * @param msg Exception message
+     */
+    public UncheckedSolrException(String msg) {


Change to public UncheckedSolrException(SolrServerException cause).

I keep this one but I change public UncheckedSolrException(Throwable cause) to public UncheckedSolrException(SolrServerException cause)

I'm looking at java.io.UncheckedIOException as a model here. Requiring SolrServerException in all constructors protects against the misuse of the class for something other than its intended purpose (e.g. to wrap a SolrServerException as a RuntimeException). It also provides an API guarantee for future clients that the cause of an UncheckedSolrException is always a SolrServerException.

Ok, I just make the change

sjudeng · 2017-09-05T01:12:49Z

janusgraph-solr/src/main/java/org/janusgraph/diskstorage/solr/UncheckedSolrException.java

+     * @param msg   Exception message
+     * @param cause Cause of the exception
+     */
+    public UncheckedSolrException(String msg, Throwable cause) {


Change to public UncheckedSolrException(String message, SolrServerException cause) (all constructors should take SolrServerException).

sjudeng

Thanks for all the updates. There are some new test failures under Elasticsearch 1.x. Is there an easy fix? Otherwise I think we'd have to (finally/again) remove Elasticsearch 1.x from the compatibility matrix.

sed -i 's/\${es.docker.image}:\${elasticsearch.test.version}/\${es.docker.image}:1.7.6/' janusgraph-es/pom.xml
mvn clean install -pl janusgraph-es -Pes-docker -Delasticsearch.test.groovy.inline=true -Des.docker.image=elasticsearch
...
Tests run: 21, Failures: 0, Errors: 4, Skipped: 0, Time elapsed: 16.006 sec <<< FAILURE! - in org.janusgraph.diskstorage.es.ElasticSearchMutliTypeIndexTest
singleStore(org.janusgraph.diskstorage.es.ElasticSearchMutliTypeIndexTest)  Time elapsed: 0.89 sec  <<< ERROR!
java.io.UncheckedIOException: POST http://127.0.0.1:9200_search/scroll: HTTP/1.1 400 Bad Request
{"error":"ElasticsearchIllegalArgumentException[Failed to decode scrollId]; nested: IOException[Bad Base64 input character decimal 123 in array position 0]; ","status":400}
...

davidclement90 · 2017-09-07T07:42:07Z

I find an easy fix. There have been breaking change in scroll API between ES 1 and ES 2.

sjudeng

Thanks for finding the fix. Great contribution.

sjudeng · 2017-09-07T10:40:12Z

janusgraph-es/src/main/java/org/janusgraph/diskstorage/es/ElasticSearchSetup.java

-            RestElasticSearchClient client = new RestElasticSearchClient(rc);
+            final int scrollKeepAlive = config.get(ElasticSearchIndex.ES_SCROLL_KEEP_ALIVE);
+            Preconditions.checkArgument(scrollKeepAlive >= 1, "Scroll Keep alive should be greater or equals than 1");
+            RestElasticSearchClient client = new RestElasticSearchClient(rc, scrollKeepAlive);


sjudeng · 2017-09-07T10:40:54Z

janusgraph-es/src/main/java/org/janusgraph/diskstorage/es/rest/RestElasticSearchClient.java

+             requestData = scrollId.getBytes();
+        } else {
+            path = "_search/scroll";
+            Map<String, Object> request = new HashMap<>();


Signed-off-by: David Clement <david.clement90@laposte.net>

sjudeng · 2017-09-14T01:45:07Z

Unless anyone wants to add their review or request more time to review I'll merge this in 48 hours.

…endResult Stream the result from Search Backend instead of use List.

janusgraph-bot added the cla: yes This PR is compliant with the CLA label Aug 28, 2017

sjudeng reviewed Aug 29, 2017

View reviewed changes

davidclement90 force-pushed the streamIndexBackendResult branch from b6d35e4 to b2da51c Compare August 30, 2017 08:24

sjudeng requested changes Aug 30, 2017

View reviewed changes

davidclement90 force-pushed the streamIndexBackendResult branch 2 times, most recently from 7ecd323 to 7bacf49 Compare August 31, 2017 15:32

sjudeng reviewed Sep 2, 2017

View reviewed changes

davidclement90 force-pushed the streamIndexBackendResult branch from 7bacf49 to 43755cf Compare September 4, 2017 13:47

sjudeng reviewed Sep 5, 2017

View reviewed changes

davidclement90 force-pushed the streamIndexBackendResult branch 3 times, most recently from e3dbd9a to e39fa64 Compare September 5, 2017 19:14

JanusGraph deleted a comment from codecov-io Sep 6, 2017

sjudeng reviewed Sep 6, 2017

View reviewed changes

davidclement90 force-pushed the streamIndexBackendResult branch from e39fa64 to 8e198fe Compare September 7, 2017 07:40

JanusGraph deleted a comment from codecov-io Sep 7, 2017

sjudeng approved these changes Sep 7, 2017

View reviewed changes

davidclement90 force-pushed the streamIndexBackendResult branch from 8e198fe to d67428f Compare September 7, 2017 16:57

Stream the result from Search Backend instead of use List.

d67428f

Signed-off-by: David Clement <david.clement90@laposte.net>

JanusGraph deleted a comment from codecov-io Sep 7, 2017

sjudeng merged commit f87917b into JanusGraph:master Sep 16, 2017

sjudeng mentioned this pull request Sep 23, 2017

Resolve numerous Coverity findings. #536

Merged

davidclement90 deleted the streamIndexBackendResult branch September 24, 2017 07:21

davidclement90 mentioned this pull request Nov 23, 2017

Search count bug #735

Open

bwatson-rti-org pushed a commit to bwatson-rti-org/janusgraph that referenced this pull request Mar 9, 2019

Merge pull request JanusGraph#481 from davidclement90/streamIndexBack…

be7449b

…endResult Stream the result from Search Backend instead of use List.

micpod pushed a commit to micpod/janusgraph that referenced this pull request Nov 5, 2019

Merge pull request JanusGraph#481 from davidclement90/streamIndexBack…

14b8b9d

…endResult Stream the result from Search Backend instead of use List.

This was referenced Mar 7, 2020

Performance issue for multiple index queries when limit is given #2032

Open

Refactor SubqueryIterator and the way we handle multiple index queries #2040

Draft


		private final Cache<JointIndexQuery.Subquery, List<Object>> indexCache;

		private Iterator<? extends JanusGraphElement> itElement;

Stream the result from Search Backend instead of use List. #481

Stream the result from Search Backend instead of use List. #481

Conversation

davidclement90 commented Aug 28, 2017

sjudeng left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

davidclement90 commented Aug 30, 2017

sjudeng commented Aug 30, 2017

sjudeng left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sjudeng Aug 30, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

davidclement90 commented Aug 31, 2017

sjudeng left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sjudeng Sep 2, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

davidclement90 commented Sep 4, 2017

sjudeng left a comment

Choose a reason for hiding this comment

sjudeng Sep 4, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sjudeng Sep 5, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sjudeng left a comment

Choose a reason for hiding this comment

davidclement90 commented Sep 7, 2017

sjudeng left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sjudeng commented Sep 14, 2017

sjudeng Aug 30, 2017 •

edited

Loading

sjudeng left a comment •

edited

Loading

sjudeng Sep 2, 2017 •

edited

Loading

sjudeng Sep 4, 2017 •

edited

Loading

sjudeng Sep 5, 2017 •

edited

Loading