PageSize parameter is not supported for redshift-data

[twilson650]

I'm trying to query redshift tables and paginate the results with boto3 but I'm getting an error.

rsClient = boto3.client('redshift-data', region_name='us-east-1')

(...some code to query redshift and wait for the response, then extracting the requestId from the response...)

paginator = rsClient.get_paginator("get_statement_result")
page_iterator = paginator.paginate(Id=requestId, PaginationConfig={'PageSize': 500})

I'm getting this error on the last line:

Exception has occurred: PaginationError
Error during pagination: PageSize parameter is not supported for the pagination interface for this operation.
  File "/Users/twilson/code/fhir-pipeline/src/main/python/pipeline/mpiToFHIR.py", line 113, in <module>
    page_iterator = paginator.paginate(Id=requestId, PaginationConfig={'PageSize': 500})
botocore.exceptions.PaginationError: Error during pagination: PageSize parameter is not supported for the pagination interface for this operation.

If I omit the PageSize parameter, it returns all results successfully, but I haven't been able to paginate my results. What am I missing here?

[tim-finnigan]

Hi @twilson650 thanks for reaching out. It appears that the GetStatementResult API does not support MaxResults which is why PageSize isn't working. Interestingly it is the only exception in this regard for Redshift Data paginators.

I saw this topic came up here on Stack Overflow a few years ago, though the AWS Forums post referenced there is no longer working. I can try reaching out to the Redshfit Data team to get more clarity on the behavior here.

[twilson650]

Interesting, because when I include "MaxItems" that works:

paginator = rsClient.get_paginator("get_statement_result")
page_iterator = paginator.paginate(Id=requestId, PaginationConfig={"MaxItems": 500})
for page in page_iterator:
    print (len(page["Records"]))

Output is 500 in this case.