[source-zendesk-support] streams do not have recordsEmitted values ​​greater than 100 or 1000

[tosh2230]

Connector Name

source-zendesk-support

Connector Version

2.2.4

What step the error happened?

During the sync

Relevant information

When I create a new zendesk support connection and execute a job, the number of records retrieved at one time for some streams does not exceed 100 or 1000.
Start date set in source is set to a date that exceeds the number of records listed above.

I think that CursorPaginationZendeskSupportStream and max_retries are related.

Relevant log output

"streamStats" : [ {
}, {
  "streamName" : "users",
  "stats" : {
    "bytesCommitted" : 1235504,
    "bytesEmitted" : 1235504,
    "recordsEmitted" : 1000,
    "recordsCommitted" : 1000
  }
}, {
  "streamName" : "tickets",
  "stats" : {
    "bytesCommitted" : 47253402,
    "bytesEmitted" : 47253402,
    "recordsEmitted" : 1000,
    "recordsCommitted" : 1000
  }
}, {
  "streamName" : "ticket_metrics",
  "stats" : {
    "bytesCommitted" : 89491,
    "bytesEmitted" : 89491,
    "recordsEmitted" : 100,
    "recordsCommitted" : 100
  }
}, {
  "streamName" : "satisfaction_ratings",
  "stats" : {
    "bytesCommitted" : 36035,
    "bytesEmitted" : 36035,
    "recordsEmitted" : 100,
    "recordsCommitted" : 100
  }
}, {
  "streamName" : "macros",
  "stats" : {
    "bytesCommitted" : 73880,
    "bytesEmitted" : 73880,
    "recordsEmitted" : 100,
    "recordsCommitted" : 100
  }
} ],

Contribute

• Yes, I want to contribute

[marcosmarxm]

@tosh2230 would be possible to you share the complete sync log to investigate if there are any API response causing the issue?

[rwask]

@tosh2230 I was able to set up the Zendesk source connector with PostgreSQL as the destination with over 1000 users synced without an issue.

Can you please follow up with the sync log to one of your jobs so we can take a closer look to see if we can determine what is causing only 100 or 1000 records to sync?

See my user's stream log below where 118757 records were committed.

{
    "streamName" : "users",
    "stats" : {
      "bytesCommitted" : 115147786,
      "bytesEmitted" : 115147786,
      "recordsEmitted" : 118757,
      "recordsCommitted" : 118757
    }

[tosh2230]

This is the sync log I got in my environment.
819a1403_48da_4199_9ce3_111b131fcf29_logs_13876_txt.txt

[marcosmarxm]

@tosh2230 it looks you're using an old version of Destination Bigquery, is it possible to you upgrade to latest version?

[saulgausin]

I'm actually having the same issue as @tosh2230 for users and tickets with S3 as destination.

This is our setup:

• Source: Zendesk Support v2.2.8
• Destination: S3 v0.5.8

  "streamStats" : [ {
    "streamName" : "users",
    "stats" : {
      "bytesCommitted" : 1093175,
      "bytesEmitted" : 1093175,
      "recordsEmitted" : 1000,
      "recordsCommitted" : 1000  
     }
  }, {
    "streamName" : "tickets",
    "stats" : {
      "bytesCommitted" : 39431179,
      "bytesEmitted" : 39431179,
      "recordsEmitted" : 1000,
      "recordsCommitted" : 1000
    }
  }, {
    "streamName" : "ticket_metrics",
    "stats" : {
      "bytesCommitted" : 20497,
      "bytesEmitted" : 20497,
      "recordsEmitted" : 21,
      "recordsCommitted" : 21
    }
  }, {
    "streamName" : "ticket_metric_events",
    "stats" : {
      "bytesCommitted" : 11942,
      "bytesEmitted" : 11942,
      "recordsEmitted" : 90,
      "recordsCommitted" : 90
    }
  }, {
    "streamName" : "ticket_fields",
    "stats" : {
      "bytesCommitted" : 0,
      "bytesEmitted" : 0,
      "recordsEmitted" : 0,
      "recordsCommitted" : 0
    }
  }, {
    "streamName" : "satisfaction_ratings",
    "stats" : {
      "bytesCommitted" : 0,
      "bytesEmitted" : 0,
      "recordsEmitted" : 0,
      "recordsCommitted" : 0
    }
  } ]
}