@@ -130,7 +130,71 @@ and usage scenario.
130
130
operating system. This can manifest as ` Error: EMFILE, too many open files `
131
131
in Node.js.
132
132
133
- ## Example Scenario
133
+ ## Avoiding Streaming Deadlock
134
+
135
+ Because streaming responses stay open until you completely read or abandon the open stream connection,
136
+ the socket limit which you configure and how you handle streams can lead to a deadlock scenario.
137
+
138
+ #### Example deadlock
139
+
140
+ In this example, the max socket count is 1. When the Promise concurrency tries to ` await `
141
+ two simultaneous ` getObject ` requests, it fails because the streaming response of the first request
142
+ is ** not** read to completion, while the second ` getObject ` request is blocked indefinitely.
143
+
144
+ This can cause your application to time out or the Node.js process to exit with code 13.
145
+
146
+ ``` ts
147
+ // example: response stream deadlock
148
+ const s3 = new S3 ({
149
+ requestHandler: {
150
+ httpsAgent: {
151
+ maxSockets: 1 ,
152
+ },
153
+ },
154
+ });
155
+
156
+ // opens connection in parallel,
157
+ // but this await is dangerously placed.
158
+ const responses = await Promise .all ([
159
+ s3 .getObject ({ Bucket , Key: " 1" }),
160
+ s3 .getObject ({ Bucket , Key: " 2" }),
161
+ ]);
162
+
163
+ // reads streaming body in parallel
164
+ await responses .map (get => get .Body .transformToByteArray ());
165
+ ```
166
+
167
+ #### Recommendation
168
+
169
+ This doesn't necessarily mean you need to de-parallelize your program.
170
+ You can continue to use a socket count lower than your maxmimum parallel load, as long
171
+ as the requests do not block each other.
172
+
173
+ In this example, reading of the body streams is done in a non-blocking way.
174
+
175
+ ``` ts
176
+ // example: parallel streaming without deadlock
177
+ const s3 = new S3 ({
178
+ requestHandler: {
179
+ httpsAgent: {
180
+ maxSockets: 1 ,
181
+ },
182
+ },
183
+ });
184
+
185
+ const responses = ([
186
+ s3 .getObject ({ Bucket , Key: " 1" }),
187
+ s3 .getObject ({ Bucket , Key: " 2" }),
188
+ ]);
189
+ const objects = responses .map (get => get .Body .transformToByteArray ());
190
+
191
+ // `await` is placed after the stream-handling pipeline has been established.
192
+ // because of the socket limit of 1, the two streams will be handled
193
+ // sequentially via queueing.
194
+ await Promise .all (objects );
195
+ ```
196
+
197
+ ## Batch Workload Example Scenario
134
198
135
199
You have 10,000 files to upload to S3.
136
200
0 commit comments