Skip to content

Get Next Chunk

Description

If you wish to retrieve a large number of infotons, but you want to iterate over small "chunks" of data in a controlled fashion, you can use the create-iterator and next-chunk APIs. This allows you to request the number of infotons you want to process, and receive only that number during each iteration.

The process requires two different API calls:

  1. Call create-iterator to receive an iterator ID (in the iteratorId field) for the query.
  2. Repeatedly call next-chunk, while passing the iterator ID that you received in the iteratorId field in the previous chunk results. The process ends when CM-Well returns an empty list.

Syntax

URL: <cm-well-host>/<cm-well-path>

REST verb: GET

Mandatory parameters: op=next-chunk

Template:

<cm-well-host>/<cm-well-path>?op=next-chunk&format=<format>&iterator-id=<iteratorId>

URL example:

   <cm-well-host>/permid.org?op=next-chunk&format=json&iterator-id=YWtrYS50Y3A6Ly9jbS13ZWxsLXByb2RAMTAuMjA0LjE3Ny43OjUzMDY3L3VzZXIvJENscCM4MjY5NjY1NzQ

Curl example (REST API):

Curl -X GET <cm-well-host>/permid.org?op=next-chunk&format=json&iterator-id=YWtrYS50Y3A6Ly9jbS13ZWxsLXByb2RAMTAuMjA0LjE3Ny43OjUzMDY3L3VzZXIvJENscCM4MjY5NjY1NzQ

Note

An alternate syntax for retrieving the next chunk is:

Template: <cm-well-host>/_iterate?iterator-id=<token>

Example: curl "<cm-well-host>/_iterate?iterator-id=YWtrYS50Y3A6Ly9jbS13ZWxsLXByb2RAMTAuMjA0LjczLjE2MTo0MDEzMi91c2VyLyRHfmViIy04MDc5OTg5OTQ"

Special Parameters

Parameter Description       Values           Example
session-ttl The time, in milliseconds, until the iteration session expires. The iteration token is only valid for this length of time. The default value is 15 seconds; the maximal value is 60 seconds. A positive integer up to 60. session-ttl=20
format The output format of the streamed chunks. The default value is ntriples. See Query Parameters. format=ttl

Code Example

Call

curl <cm-well-host>/permid.org?op=next-chunk&iterator-id=YWtrYS50Y3A6Ly9jbS13ZWxsLXByb2RAMTAuMjA0LjE3Ny43OjUzMDY3L3VzZXIvJENscCM4MjY5NjY1NzQ&format=json

Results

{
      "type": "IterationResults",
      "iteratorId": "YWtrYS50Y3A6Ly9jbS13ZWxsLXByb2RAMTAuMjA0LjE3Ny41OjQxNDAwL3VzZXIvJGdicCMxMjI5ODU0NDI2",
      "totalHits": 91053224,
      "infotons": [
        {
            "type": "ObjectInfoton",
            "system": {
                "uuid": "b55c0bcc1b97d5488cef8619b7bde9cf",
                "lastModified": "2015-03-08T12:15:03.285Z",
                "path": "\/permid.org\/1-30064778964",
                "dataCenter": "dc1",
                "parent": "\/permid.org"
             }
        },
        {
           "type": "ObjectInfoton",
           "system": {
                "uuid": "f872f4986fd9f3aa5db39b6b492da7cf",
                "lastModified": "2015-04-01T02:05:08.331Z",
                "path": "\/permid.org\/1-21557484859",
                "dataCenter": "dc1",
                "parent": "\/permid.org"
           }
        },
        {
           "type": "ObjectInfoton",
           "system": {
                "uuid": "b5d00287d5de49bfdc583849416f7659",
                "lastModified": "2015-06-02T22:30:37.385Z",
                "path": "\/permid.org\/1-5045869909",
                "dataCenter": "dc1",
                "parent": "\/permid.org"
           }
        }
      ]
    }

Note

  • If the iteration process fails in the middle for any reason, you can retry the same request, as long as session-ttl has not passed. And in case the last iterator-id is no longer valid, you will have to restart the process from the beginning (that is, iterate again over all infotons that match the query).
  • An alternative is to use the consumer API, which allows you to save the iteration state and restart from the same point after a failure. See Create Consumer to learn more.

Create Iterator