Why You Can't Query DynamoDB Without a Key and How to Iterate Instead
You cannot fetch all items from a DynamoDB table without specifying a primary key. The short answer is no, because that is how the underlying data model works. In DynamoDB, every item must have a unique logical address defined by its partition key (and optionally a sort key) before you can retrieve it. Your current code using batch_get_item attempts to solve this by guessing keysâconstructing an array of arbitrary IDs like '1', '2' and iterating manually. This approach is fundamentally flawed for two reasons: it is inefficient at scale, and it will eventually stop returning items once the batch limit is hit or you run out of guessed IDs.
The batch_get_item operation has a hard limit on the number of items you can request in a single call (typically 25 partitions) and a strict cap on the total size of the response data (400 KB). If your table contains thousands of items, your current script will return only a fraction of them before hitting these artificial ceilings. Furthermore, relying on hardcoded or guessed key values means you are ignoring the actual data distribution in the table. You are essentially trying to find needles in a haystack by randomly throwing darts at specific coordinates, which is not how indexing works.
The Correct Approach: Scan vs. Query
To retrieve all items without knowing their primary keys, you must use the scan operation or iterate through a query. These operations allow DynamoDB to traverse the table based on its internal storage structure rather than requiring you to know every specific ID.
If your goal is simply to read every item regardless of the key, the scan operation is the direct equivalent of what you are trying to achieve. Unlike batch_get_item, which requires explicit keys, scan reads items from the table sequentially.
$iterator = $dynamodb->getIterator([
'TableName' => 'products',
'ConsistentRead' => false // Use true for strong consistency at the cost of performance
]);
while ($item = $iterator->get()) {
// Process each item
echo json_encode($item);
}
This pattern is robust because it handles pagination automatically. The iterator manages the LastEvaluatedKey returned in the response, fetching the next set of items until there are no more results. This is the standard idiom for full table reads in DynamoDB.
Trade-offs: Scan vs. Query
While scan solves your immediate problem, it comes with significant performance implications that you need to understand before implementing this in production. A scan operation reads every item in the table, even if you only want a subset of data. This means your read capacity units (RCU) will be consumed linearly based on the total size of the table. If you have 10 million items and only need 1% of them, scanning is wasteful because you are paying for every single item read.
If your Id partition key represents a natural ordering or if you can create a Global Secondary Index (GSI) that supports your filtering requirements, a query might be more efficient. However, a standard query still requires specifying the partition key value to narrow down the search space. You cannot query for "all items" unless the table has only one unique partition key value.
Handling Pagination Manually
If you prefer not to use iterators or are working with an older version of the AWS SDK that lacks helper methods, you can handle pagination manually using the response metadata. The critical fields here are Items, which contains the data, and LastEvaluatedKey, which acts as a cursor for the next request.
$response = $dynamodb->scan([
'TableName' => 'products',
]);
if (!empty($response['LastEvaluatedKey'])) {
$nextPage = $dynamodb->scan([
'TableName' => 'products',
'ExclusiveStartKey' => $response['LastEvaluatedKey']
]);
// Merge items from both responses
$allItems = array_merge($response['Items'], $nextPage['Items']);
} else {
$allItems = $response['Items'];
}
Note that you must always check if LastEvaluatedKey is present. If it is missing, the scan has completed successfully. Relying on this mechanism ensures your application can handle tables of any size without hitting the 25-partition batch limit or the 400 KB response size restriction inherent to batch_get_item.
Summary of Limitations
It is important to note that DynamoDB does not support full table scans for large datasets if you are running on a serverless configuration like AWS Lambda. The default timeout and memory constraints often prevent the iterator from completing a scan of millions of rows before the function times out. In those scenarios, you might need to design your data model differently, perhaps using a GSI or breaking the data into smaller logical segments based on the partition key.
For most standard EC2 or ECS deployments, however, the scan iterator pattern remains the reliable method for retrieving all items without prior knowledge of their keys.
