I think because of a lot of gossip overhead among 100 nodes, t3.medium might not be able to handle 1000tx/s within several seconds.
practically, the validator nodes usually have much higher resource than t3.medium, and it usually has
only several very trusted internal peers to minimize gossip overheads.
so, i expect those are the reasons. From the number 10min(=600s), I expect only 1~2 nodes are succeeding for block creation in several seconds, rest of the attempts are failed because of lack of time I guess.
therefore, i suggest you to limit
- max_num_inbound_peers = 10
- max_num_outbound_peers = 10
and try the test again.
with this config, you might need sophisticated peering control to make all 100 nodes connected indirectly. (creating 5 clusters with whole connections and connecting 5 clusters can be a strategy.)