This pull request introduces a new integration test tool for the
StreamManager streaming system, adding a standalone test binary with
supporting modules for simulating and verifying high-throughput data
transfer. The changes include a test driver, a configurable in-memory
delivery pipe for simulating network conditions, a data generator, a
verifier for end-to-end integrity, and a metrics tracker. Additionally,
several improvements are made to the circular buffer and StreamManager
for better handling of blocking writes and out-of-order acknowledgments.
**New StreamManager Integration Test Tool**
* Added a new test binary `cmd/test-streammanager` with a main driver
(`main-test-streammanager.go`) that orchestrates end-to-end streaming
tests, including configuration for data size, delivery delay/skew,
window size, slow reader simulation, and verbose logging.
* Implemented a configurable `DeliveryPipe` (`deliverypipe.go`) for
simulating network delivery with delay and skew, supporting separate
data and ack channels, out-of-order delivery, and high water mark
tracking.
* Added `WriterBridge` and `ReaderBridge` modules for interfacing
between brokers and the delivery pipe, enforcing correct directionality
of data and acks.
* Created a sequential test data generator (`generator.go`) and a
verifier (`verifier.go`) for checking data integrity and reporting
mismatches.
[[1]](diffhunk://#diff-3f2d6e0349089e3748c001791a383687b33a2c2391fd3baccfceb83e76e6ee0dR1-R40)
[[2]](diffhunk://#diff-cb3aab0bae9bec15ef0c06fe5d9e0e96094affcf4720680605a92054ab717575R1-R61)
* Introduced a metrics module (`metrics.go`) for tracking throughput,
packet counts, out-of-order events, and pipe usage, with a summary
report at test completion.
**StreamManager and CirBuf Improvements**
* Refactored circular buffer (`pkg/jobmanager/cirbuf.go`) to replace
blocking writes with a non-blocking `WriteAvailable` method, returning a
wait channel for buffer-full scenarios, and removed context-based
cancellation logic.
* Updated StreamManager (`pkg/jobmanager/streammanager.go`) to track the
maximum acknowledged sequence/rwnd tuple, ignoring stale or out-of-order
ACKs, and resetting this state on disconnect.
* Modified StreamManager's data handling to use the new non-blocking
buffer write logic, ensuring correct signaling and waiting for space
when needed.
**Minor Cleanup**
* Removed unused context import from `cirbuf.go`.
* Minor whitespace cleanup in `streambroker.go`.
lots of stuff here.
introduces a streaming framework for the RPC system with flow control.
new authentication primitives for the RPC system. this is used to create
a persistent "job manager" process (via wsh) that can survive
disconnects. and then a jobcontroller in the main server that can
create, reconnect, and manage these new persistent jobs.
code is currently not actively hooked up to anything minus some new
debugging wsh commands, and a switch in the term block that lets me test
viewing the output.
after PRing this change the next steps are more testing and then
integrating this functionality into the product.
