Implement Process.ReadAllText and ReadAllBytes with platform-specific multiplexing#126807
Implement Process.ReadAllText and ReadAllBytes with platform-specific multiplexing#126807
Conversation
…plexing Agent-Logs-Url: https://github.com/dotnet/runtime/sessions/45ece5ba-166f-4418-887f-3ecaba54615e Co-authored-by: adamsitnik <6011991+adamsitnik@users.noreply.github.com>
…le fixes needed) Agent-Logs-Url: https://github.com/dotnet/runtime/sessions/45ece5ba-166f-4418-887f-3ecaba54615e Co-authored-by: adamsitnik <6011991+adamsitnik@users.noreply.github.com>
…ead of GetAwaiter().GetResult() Replace the async Task-based Windows implementation with proper Win32 overlapped IO: - Allocate NativeOverlapped* with ManualResetEvent for each pipe - Use Interop.Kernel32.ReadFile with overlapped structs - Use WaitHandle.WaitAny for synchronous multiplexing - Use GetOverlappedResult to retrieve bytes read - Use CancelIoEx for timeout cancellation - Add System.Threading.Overlapped project reference - Add ReadFile, GetOverlappedResult, CancelIoEx interop references - Fix nullable warning in Unix implementation Agent-Logs-Url: https://github.com/dotnet/runtime/sessions/8af96aef-e52b-459d-a945-e1159e8be19b Co-authored-by: adamsitnik <6011991+adamsitnik@users.noreply.github.com>
…in32Exception Agent-Logs-Url: https://github.com/dotnet/runtime/sessions/8af96aef-e52b-459d-a945-e1159e8be19b Co-authored-by: adamsitnik <6011991+adamsitnik@users.noreply.github.com>
|
Tagging subscribers to this area: @dotnet/area-system-diagnostics-process |
adamsitnik
left a comment
There was a problem hiding this comment.
@copilot please address my feedback
| bool hasOutput = _standardOutput is not null; | ||
| bool hasError = _standardError is not null; | ||
|
|
||
| if (!hasOutput && !hasError) | ||
| { | ||
| throw new InvalidOperationException(SR.CantGetStandardOut); | ||
| } |
There was a problem hiding this comment.
After thinking about it for a while, I believe we should always require both std out and error to be redirected. Otherwise it could lead to a situation where the user forgets to redirect standard error, calls the new method, gets an empty error and thinks that nothing was written to standard error. While it could be written, but simply not read.
@copilot please make the verification more strict and update all called methods in a way they can assume that both std out and err will be always provided
| bool hasOutput = _standardOutput is not null; | |
| bool hasError = _standardError is not null; | |
| if (!hasOutput && !hasError) | |
| { | |
| throw new InvalidOperationException(SR.CantGetStandardOut); | |
| } | |
| if (_standardOutput is null) | |
| { | |
| throw new InvalidOperationException(SR.CantGetStandardOut); | |
| } | |
| else if (_standardError is null) | |
| { | |
| throw new InvalidOperationException(SR.CantGetStandardError); | |
| } |
There was a problem hiding this comment.
Done in 0986690. Both _standardOutput and _standardError are now required — separate checks throw InvalidOperationException with the appropriate message if either is null. All downstream methods (ReadPipes on both platforms) now take non-nullable SafeFileHandle parameters. Added two new test cases (ReadAllBytes_ThrowsWhenOnlyOutputRedirected, ReadAllBytes_ThrowsWhenOnlyErrorRedirected) and updated all existing tests to always redirect both streams.
| byte[] outputResult = outputBytesRead > 0 | ||
| ? outputBuffer.AsSpan(0, outputBytesRead).ToArray() | ||
| : Array.Empty<byte>(); | ||
|
|
||
| byte[] errorResult = errorBytesRead > 0 | ||
| ? errorBuffer.AsSpan(0, errorBytesRead).ToArray() | ||
| : Array.Empty<byte>(); |
There was a problem hiding this comment.
Create a new helper method that does everything that ReadAllBytes currently does except of creating a copy of the buffer. Then call this helper from ReadAllText in order to avoid creating a copy of the buffer when we just want to decode the text. And of course use this helper from ReadAllBytes as well.
There was a problem hiding this comment.
Done in 0986690. Extracted ReadPipesToBuffers which handles validation, handle extraction, DangerousAddRef/DangerousRelease, and calls ReadPipes. Both ReadAllText and ReadAllBytes call it. ReadAllText now decodes directly from the rented buffers via Encoding.GetString(byte[], int, int) — no intermediate .ToArray() copy.
| private static SafeFileHandle GetSafeFileHandleFromStreamReader(StreamReader reader) | ||
| { | ||
| Stream baseStream = reader.BaseStream; | ||
|
|
||
| if (baseStream is FileStream fileStream) | ||
| { | ||
| return fileStream.SafeFileHandle; | ||
| } | ||
|
|
||
| if (baseStream is System.IO.Pipes.AnonymousPipeClientStream pipeStream) | ||
| { | ||
| return new SafeFileHandle(pipeStream.SafePipeHandle.DangerousGetHandle(), ownsHandle: false); | ||
| } |
There was a problem hiding this comment.
Populate the owner and in the method that calls this helper call owner.DagnerousAddRef and accordingly owner.DangerousRelease before and after calling ReadPipes. This will allow us to avoid user disposing the handle while we are using it.
| private static SafeFileHandle GetSafeFileHandleFromStreamReader(StreamReader reader) | |
| { | |
| Stream baseStream = reader.BaseStream; | |
| if (baseStream is FileStream fileStream) | |
| { | |
| return fileStream.SafeFileHandle; | |
| } | |
| if (baseStream is System.IO.Pipes.AnonymousPipeClientStream pipeStream) | |
| { | |
| return new SafeFileHandle(pipeStream.SafePipeHandle.DangerousGetHandle(), ownsHandle: false); | |
| } | |
| private static SafeFileHandle GetSafeFileHandleFromStreamReader(StreamReader reader, out SafeHandle owner) | |
| { | |
| Stream baseStream = reader.BaseStream; | |
| if (baseStream is FileStream fileStream) | |
| { | |
| owner = fileStream.SafeFileHandle; | |
| return fileStream.SafeFileHandle; | |
| } | |
| if (baseStream is System.IO.Pipes.AnonymousPipeClientStream pipeStream) | |
| { | |
| owner = pipeStream.SafePipeHandle; | |
| return new SafeFileHandle(pipeStream.SafePipeHandle.DangerousGetHandle(), ownsHandle: false); | |
| } |
There was a problem hiding this comment.
Done in 0986690. GetSafeFileHandleFromStreamReader now takes out SafeHandle owner. The caller in ReadPipesToBuffers calls DangerousAddRef before ReadPipes and DangerousRelease in the finally block.
| /// <summary> | ||
| /// Reads from one or both standard output and standard error pipes using platform-specific multiplexing. | ||
| /// </summary> | ||
| private static partial void ReadPipes( | ||
| SafeFileHandle? outputHandle, | ||
| SafeFileHandle? errorHandle, | ||
| int timeoutMs, | ||
| ref byte[] outputBuffer, | ||
| ref int outputBytesRead, | ||
| ref byte[] errorBuffer, | ||
| ref int errorBytesRead); |
There was a problem hiding this comment.
As I wrote in the requirements, there is no need to use partial method here as we target only Windows and Unix platforms and the right method will be used anyway (form .Windows.cs or .Unix.cs file).
| /// <summary> | |
| /// Reads from one or both standard output and standard error pipes using platform-specific multiplexing. | |
| /// </summary> | |
| private static partial void ReadPipes( | |
| SafeFileHandle? outputHandle, | |
| SafeFileHandle? errorHandle, | |
| int timeoutMs, | |
| ref byte[] outputBuffer, | |
| ref int outputBytesRead, | |
| ref byte[] errorBuffer, | |
| ref int errorBytesRead); |
There was a problem hiding this comment.
Done in 0986690. Removed the partial method declaration. ReadPipes is now a regular private static void in each platform file.
…elper method, DangerousAddRef/Release, remove partial Changes: 1. Require both stdout and stderr to be redirected (throw if either is missing) 2. Create ReadPipesToBuffers helper shared between ReadAllText and ReadAllBytes (ReadAllText avoids extra buffer copy by decoding from rented buffers directly) 3. GetSafeFileHandleFromStreamReader now returns an owner SafeHandle; caller uses DangerousAddRef/DangerousRelease to prevent handle disposal during reads 4. Remove partial method declaration; ReadPipes is a regular static method in each platform file (non-nullable parameters since both handles always provided) 5. Update tests: all tests redirect both stdout and stderr; add two new tests for missing-stream validation Agent-Logs-Url: https://github.com/dotnet/runtime/sessions/cd917b13-f344-47c5-ac1e-cfb2c2da34f6 Co-authored-by: adamsitnik <6011991+adamsitnik@users.noreply.github.com>
All feedback addressed in 0986690. Summary of changes:
|
adamsitnik
left a comment
There was a problem hiding this comment.
@copilot address my feedback
src/libraries/System.Diagnostics.Process/src/System/Diagnostics/Process.Multiplexing.cs
Show resolved
Hide resolved
src/libraries/System.Diagnostics.Process/src/System/Diagnostics/Process.Multiplexing.Windows.cs
Outdated
Show resolved
Hide resolved
src/libraries/System.Diagnostics.Process/src/System/Diagnostics/Process.Multiplexing.Windows.cs
Outdated
Show resolved
Hide resolved
…indows ReadPipes fully unsafe with proper cleanup, use is not null Agent-Logs-Url: https://github.com/dotnet/runtime/sessions/451cf437-2de0-409a-937a-0031c3fa69d4 Co-authored-by: adamsitnik <6011991+adamsitnik@users.noreply.github.com>
| outputOverlapped = AllocateOverlapped(outputEvent); | ||
| errorOverlapped = AllocateOverlapped(errorEvent); |
There was a problem hiding this comment.
Can't the overlapped's be allocated on the stack?
There was a problem hiding this comment.
I know it looks like an ordinary synchronous code, but in reality we issue an async read with ReadFile. The issuing thread should be still alive, but after debugging an AV for an hour that was caused by non-existing callback getting invoked on a ThreadPool thread I am not 100% sure and I prefer to allocate dedicated memory to be 100% sure.
…RentLargerBuffer overflow Agent-Logs-Url: https://github.com/dotnet/runtime/sessions/34214748-db27-499b-88ef-23a1af15515f Co-authored-by: adamsitnik <6011991+adamsitnik@users.noreply.github.com>
🤖 Copilot Code Review — PR #126807Note This review was AI/Copilot-generated using multi-model analysis (Claude Sonnet 4, Gemini 2.5 Pro). Holistic AssessmentMotivation: The PR implements approved API surface ( Approach: Platform-specific multiplexing (overlapped IO on Windows, poll(2) on Unix) is the correct approach for single-threaded synchronous draining of two pipes. Summary: Detailed Findings❌ Test logic error — missing
|
🤖 Copilot Code Review — PR #126807Note This review was generated by GitHub Copilot using multi-model analysis (Claude Opus 4.6 primary, Claude Sonnet 4.5 secondary). A GPT-5.3-Codex sub-agent was launched but did not complete within the 10-minute window. Holistic AssessmentMotivation: Well-justified. The deadlock problem when reading both stdout and stderr simultaneously is a real pain point. The approved API in #125838 clearly documents the need, and this PR implements two of the approved methods ( Approach: The platform-specific multiplexing approach (poll on Unix, overlapped IO on Windows) is the right solution. Single-threaded synchronous multiplexing avoids thread pool pressure and is consistent with the synchronous API shape. Summary: Detailed Findings❌ Unix Error Path — Wrong Error Code Passed to Win32ExceptionFile: throw new Win32Exception((int)error);The Passing The previous commit (99ef166) had the correct code: Fix (preferred — matches the component's established pattern of using throw new Win32Exception(Interop.Sys.ConvertErrorPalToPlatform(error));
Alternative — add throw Interop.GetExceptionForIoErrno(new Interop.ErrorInfo(error));
|
…mode test Agent-Logs-Url: https://github.com/dotnet/runtime/sessions/f6ca8588-63c1-4cd8-aeaf-48c716a4ea11 Co-authored-by: adamsitnik <6011991+adamsitnik@users.noreply.github.com>
adamsitnik
left a comment
There was a problem hiding this comment.
Fix (preferred — matches the component's established pattern of using
Win32Exceptionwith native errno):throw new Win32Exception(Interop.Sys.ConvertErrorPalToPlatform(error));
@copilot do it
Per dotnet/runtime convention, ref assembly members should be in alphabetical order.
ReadAllBytesandReadAllTextshould be placed beforeRefresh()/Start*/ToString(). Additionally,ReadAllBytesshould appear beforeReadAllText.
@copilot reorder the APIs
NativeMemory.AllocZeroedalready zeroes all bytes, making the subsequent assignments ofInternalHigh,InternalLow,OffsetHigh,OffsetLowto zero redundant. Only theEventHandleassignment is meaningful. This is harmless but slightly noisy — consider removing the redundant lines for clarity.
@copilot address this
- Interleaved output: Rapid alternation between stdout and stderr to stress multiplexing
@copilot add such test for ReadAllText
- Binary data with null bytes: For
ReadAllBytesspecifically
@copilot add such test for ReadAllBytes
| [InlineData(false, false)] | ||
| public void ReadAll_ThrowsWhenOutputOrErrorIsInSyncMode(bool bytes, bool standardOutput) | ||
| { | ||
| Process process = CreateProcess(RemotelyInvokable.StreamBody); |
There was a problem hiding this comment.
@copilot Don't use StreamBody, as it produces garbage in the test output:
System.Diagnostics.Process.Tests started
System.Diagnostics.Process.Tests closed
| Process process = CreateProcess(RemotelyInvokable.StreamBody); | |
| Process process = CreateProcess(RemotelyInvokable.Dummy); |
There was a problem hiding this comment.
Done in 1d30ed2. Replaced StreamBody with Dummy.
…rlapped cleanup, new tests, StreamBody->Dummy Agent-Logs-Url: https://github.com/dotnet/runtime/sessions/dac42b05-c591-4e1c-b751-9eadeb376728 Co-authored-by: adamsitnik <6011991+adamsitnik@users.noreply.github.com>
All 5 items addressed in 1d30ed2:
🤖 Copilot Code Review — PR #126807Note This review was AI/Copilot-generated using multi-model analysis (Claude Sonnet 4, Gemini 2.5 Pro). Holistic AssessmentMotivation: Implements approved API surface ( Approach: Platform-specific multiplexing (overlapped IO on Windows, poll(2) on Unix) is the correct approach for single-threaded synchronous draining of two pipes. Summary: ✅ LGTM. All previous code review findings have been addressed. The implementation is solid with proper resource management, handle lifetime protection, and comprehensive test coverage (23 test cases). Detailed Findings✅ API shape matches approved proposalThe ref assembly signatures exactly match the approved shape from @bartonjs's API review comment on #125838. Tuple element names are PascalCase ( ✅ Encoding fallback is consistent with existing code
✅ Handle lifetime protection is correct
✅ Windows unsafe resource cleanup is soundAll 6 resources (2 pins, 2 overlapped, 2 events) are declared at method top and cleaned up in ✅ ReadFile return value handling is correct
✅ Unix poll error handlingNow correctly uses ✅ Buffer overflow protection
✅ Sync mode guard prevents data loss
✅ Test coverage is comprehensive23 test cases covering validation (disposed, no redirect, single redirect, sync mode, async mode), success paths (both streams, stdout-only, stderr-only, empty, interleaved), edge cases (binary with null bytes, large 100KB output), and timeout behavior. |
Agent-Logs-Url: https://github.com/dotnet/runtime/sessions/dac42b05-c591-4e1c-b751-9eadeb376728 Co-authored-by: adamsitnik <6011991+adamsitnik@users.noreply.github.com>
adamsitnik
left a comment
There was a problem hiding this comment.
@copilot address my feedback
|
|
||
| for (int i = 0; i < iterations; i++) | ||
| { | ||
| Assert.Contains($"out{i} ", standardOutput); |
There was a problem hiding this comment.
@copilot Build the exact expected string and then perform explicit equality check, don't use Assert.Contains
There was a problem hiding this comment.
Fixed in 9896de0. The test now builds the exact expected strings via StringBuilder and uses Assert.Equal for both stdout and stderr.
| /// <summary> | ||
| /// Rents a larger buffer from the array pool and copies the existing data to it. | ||
| /// </summary> | ||
| private static void RentLargerBuffer(ref byte[] buffer, int bytesRead) |
There was a problem hiding this comment.
this method will cause endless loop if buffer.Length == Array.MaxLength. Detect such situation and throw an exception. I believe that OutOfMemoryException would be best.
There was a problem hiding this comment.
Fixed in 9896de0. RentLargerBuffer now checks buffer.Length >= Array.MaxLength at the top and throws OutOfMemoryException before attempting to rent.
…al in interleaved test Agent-Logs-Url: https://github.com/dotnet/runtime/sessions/560e33ea-a2eb-4f65-a9c9-a2bc3fc32986 Co-authored-by: adamsitnik <6011991+adamsitnik@users.noreply.github.com>
Description
Implements
Process.ReadAllTextandProcess.ReadAllBytes— new APIs (approved in #125838) that drain both stdout and stderr without deadlocking, using single-threaded synchronous multiplexing. Both standard output and standard error must be redirected; the methods throwInvalidOperationExceptionif either is missing. The methods also require that neitherStandardOutputnorStandardErrorhas been accessed in sync mode (StreamReadMode.Undefinedrequired), preventing silent data loss from StreamReader-buffered data.Windows — Overlapped IO
ReadPipesisprivate static unsafe voidwith all resources (pins, overlapped, events) declared at method top and cleaned up in a singlefinallyblock — ensures no leaks if any allocation throwsNativeOverlapped*withManualResetEventper pipeReadFile(overlapped) →WaitHandle.WaitAny→GetOverlappedResultloopCancelIoExon timeout,GetOverlappedResult(bWait: true)to observe completion before freeing OVERLAPPEDAllocateOverlappedusesNativeMemory.AllocZeroedand only sets the meaningfulEventHandlefield — no redundant zero assignmentsis not nullpattern matching syntax throughoutUnix — poll(2)
Interop.Sys.Pollon stdout/stderr file descriptorsRandomAccess.Readonce poll reports data readyInterop.Sys.ConvertErrorPalToPlatform(error)for proper errno inWin32ExceptionReadPipesis a regularprivate static void(non-nullableSafeFileHandleparameters)Common (Process.Multiplexing.cs)
ValidateReadAllState: validates state (disposed, both streams redirected, async mode conflict, sync mode conflict) and sets sync mode — called fromReadAllTextandReadAllBytesbefore renting arrays from the pool. RequiresStreamReadMode.Undefinedfor both stdout and stderr to prevent callingReadAll*afterStandardOutput/StandardErrorwas accessed (StreamReader may have buffered data that handle-based reads would miss)ReadPipesToBuffers: obtains handles withDangerousAddRef/DangerousReleaseprotection, usesToTimeoutMillisecondsfor proper timeout validation, and delegates to platformReadPipesReadAllBytes: callsValidateReadAllState, rents fromArrayPool, callsReadPipesToBuffers, copies results to owned arraysReadAllText: callsValidateReadAllState, rents fromArrayPool, callsReadPipesToBuffers, decodes directly from rented buffers viaEncoding.GetString(byte[], int, int)(avoids extra buffer copy)GetSafeFileHandleFromStreamReader: extracts handle fromFileStream(Windows) orAnonymousPipeClientStream(Unix), returnsout SafeHandle ownerfor reference counting; throwsUnreachableExceptionfor unexpected stream typesRentLargerBuffer(2x, pool-backed, capped atArray.MaxLength; throwsOutOfMemoryExceptionif the buffer is already atArray.MaxLengthto prevent infinite loop)Tests (25 cases)
ReadAllBytes)Code review findings addressed
elsein timeout test that causedReadAllTextto always run regardless ofbytesparameterRentLargerBufferwherebuffer.Length * 2could overflow for large buffers — now usesMath.Min((long)buffer.Length * 2, Array.MaxLength)RentLargerBufferinfinite loop whenbuffer.Length == Array.MaxLength— now throwsOutOfMemoryExceptionToTimeoutMilliseconds()helper which properly validates theTimeSpanrangeStreamReadMode.syncModeguard inValidateReadAllState— prevents callingReadAll*afterStandardOutput/StandardErrorwas accessed (data may have been buffered by StreamReader)ReadAllText_ReadsBothOutputAndError→ReadAll_ReadsBothOutputAndErrorsince the test exercises bothReadAllTextandReadAllBytesGetSafeFileHandleFromStreamReaderfallback exception fromInvalidOperationExceptiontoUnreachableException(we control stream creation)ReadAll_ThrowsWhenOutputOrErrorIsInSyncModetest (4 cases via Theory)StreamBodywithDummyin empty-output test to avoid garbage in test outputAssert.Equal) instead ofAssert.Contains