bpo-28724: Add methods send_fds, recv_fds to the socket module by nanjekyejoannah · Pull Request #12889 · python/cpython

nanjekyejoannah · 2019-04-20T22:44:32Z

I have added send_fds(), recv_fds() to the socket module.

https://bugs.python.org/issue28724

bedevere-bot · 2019-05-04T22:50:31Z

A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated.

Once you have made the requested changes, please leave a comment on this pull request containing the phrase I have made the requested changes; please review again. I will then notify any core developers who have left a review that you're ready for them to take another look at this pull request.

pablogsal · 2019-05-10T19:34:35Z

Why we need a loop of 1 iteration?

I want to return a list of 1 file descriptors for created files.

But you don't need a loop for that, right? Just substitute i by 0. I am missing something?

nanjekyejoannah · 2019-05-27T15:58:38Z

Needs, Mac OS fixes. Will get a Mac to test this later.

vstinner · 2019-08-20T15:08:01Z

@nanjekyejoannah: I think that these new functions are quite cool and I like to get them in Python 3.9. Do you plan to work on this PR? There are a few pending requests, but the main code is already written and has been reviewed, so this PR is close to be ready to be merged!

nanjekyejoannah · 2019-08-20T15:15:04Z

@nanjekyejoannah: I think that these new functions are quite cool and I like to get them in Python 3.9. Do you plan to work on this PR? There are a few pending requests, but the main code is already written and has been reviewed, so this PR is close to be ready to be merged!

I will look at this later this week maybe weekendish.

nanjekyejoannah · 2019-08-24T16:41:28Z

@pablogsal , Please help me confirm If the tests pass on MacOS now. I cant access a Mac now.

nanjekyejoannah · 2019-08-24T16:42:09Z

I have made the requested changes; please review again

bedevere-bot · 2019-08-24T16:42:13Z

Thanks for making the requested changes!

@tiran, @pablogsal, @vstinner, @ncoghlan: please review the changes made to this pull request.

vstinner · 2019-08-26T22:32:11Z

+ import array
+
+ def send_fds(sock, buffers, fds, flags=0, address=None):
+ """ send_fds(sock, buffers, fds[, flags[, address]]) -> socket object


It seems like sendmsg() returns an integer, not a socket object.

vstinner · 2019-08-26T22:33:08Z

+ import array
+
+ def recv_fds(sock, bufsize, maxfds, flags=0):
+ """ recv_fds(sock, bufsize, maxfds[, flags]) -> (socket object, socket object)


The doc of the return value looks wrong. The code uses a tuple of 4 times: return msg, list(fds), flags, addr.

vstinner · 2019-08-26T22:33:25Z

+ def recv_fds(sock, bufsize, maxfds, flags=0):
+ """ recv_fds(sock, bufsize, maxfds[, flags]) -> (socket object, socket object)
+
+ receive up to maxfds file descriptors returning the message


Upper case for the first word of the doc?
Suggested change
receiveuptomaxfdsfiledescriptorsreturningthemessage
Receiveuptomaxfdsfiledescriptorsreturningthemessage

vstinner · 2019-08-26T22:34:51Z

+ with sock1, sock2:
+ socket.send_fds(sock1, [MSG], fds)
+ # request more data and file descriptors than expected
+ msg, fds2, flags, addr = socket.recv_fds(sock2, len(MSG) * 2, len(fds) * 2)


nitpick (PEP 8): there is an useless space
Suggested change
msg, fds2, flags, addr=socket.recv_fds(sock2, len(MSG) *2, len(fds) *2)
msg, fds2, flags, addr=socket.recv_fds(sock2, len(MSG) *2, len(fds) *2)

vstinner · 2019-08-26T22:37:58Z

+ Receive up to *maxfds* file descriptors. Return ``(msg, list(fds), flags, addr)``. Consult
+ :meth:`recvmsg` for the documentation of these parameters.
+
+ .. availability:: Unix supporting :meth:`~socket.sendmsg` and :const:`SCM_RIGHTS` mechanism.


it requires recvmsg() instead of sendmsg(), no?

vstinner · 2019-08-26T22:38:32Z

+.. method:: socket.send_fds(sock, buffers, fds[, flags[, address]])
+
+ Send the list of file descriptors *fds* over an :const:`AF_UNIX` socket.
+ Consult :meth:`sendmsg` for the documentation of these parameters.


You should document at least what is the fds parameter. I suggest to document it as sequence of file descriptors. Is it possible to send 0 file descriptor?
What is the return value?
Does someone know if it's possible that the call send only a few file descriptors, but not all of them? What if it's called with 100 000 file descriptors for example?

What is the return value?
The return type for sendmsg() is int AFAIK. Am not sure about these other edge cases but I can look.

I tested to send 1000 file descriptors: sendmsg() fails with "OSError: [Errno 22] Invalid argument".
For partial send, it can happen, as in socket.send(). In that case, according to https://stackoverflow.com/questions/38431713/partial-read-write-issue-in-sendmsg-recvmsg the file descriptors are already sent and should be be send again, socket.sendmsg() should be called again, but only with buffers (adjusted to the offset equal to the send_fds() result).
In short, you don't have to do anything. I'm not comfortable to document how to handle partial send since I'm not sure that the behavior is portable.
sendmsg() doesn't warn about partial send. Maybe we should copy send() warning into sendmsg() and send_fds(): "Applications are responsible for checking that all data has been sent; if only some of the data was transmitted, the application needs to attempt delivery of the remaining data."

bedevere-bot · 2019-08-26T22:41:09Z

A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated.

Once you have made the requested changes, please leave a comment on this pull request containing the phrase I have made the requested changes; please review again. I will then notify any core developers who have left a review that you're ready for them to take another look at this pull request.

nanjekyejoannah · 2019-08-28T14:39:47Z

I have made the requested changes; please review again

bedevere-bot · 2019-08-28T14:39:51Z

Thanks for making the requested changes!

@ncoghlan, @pablogsal, @vstinner, @tiran: please review the changes made to this pull request.

vstinner

The test is skipped. When I enabled manually the test, it failed on Linux :-(

vstinner · 2019-08-28T21:20:29Z

+.. method:: socket.send_fds(sock, buffers, fds[, flags[, address]])
+
+ Send the list of file descriptors *fds* over an :const:`AF_UNIX` socket.
+ Consult :meth:`sendmsg` for the documentation of these parameters.


I tested to send 1000 file descriptors: sendmsg() fails with "OSError: [Errno 22] Invalid argument".
For partial send, it can happen, as in socket.send(). In that case, according to https://stackoverflow.com/questions/38431713/partial-read-write-issue-in-sendmsg-recvmsg the file descriptors are already sent and should be be send again, socket.sendmsg() should be called again, but only with buffers (adjusted to the offset equal to the send_fds() result).
In short, you don't have to do anything. I'm not comfortable to document how to handle partial send since I'm not sure that the behavior is portable.
sendmsg() doesn't warn about partial send. Maybe we should copy send() warning into sendmsg() and send_fds(): "Applications are responsible for checking that all data has been sent; if only some of the data was transmitted, the application needs to attempt delivery of the remaining data."

vstinner · 2019-08-28T21:22:47Z

+@requireAttrs(socket, "send_fds")
+@requireAttrs(socket, "recv_fds")
+@requireAttrs(socket, "AF_UNIX")
+class SendRecvFdsTests(unittest.TestCase):


Oh, this test case must be added manually into test_main(). Currently, the test is not run...
I added it manually and... the test fails :-( The test writes into the read end of the pipe, and reads from the write end of the pipe. It should be the opposite. Maybe on FreeBSD, pipe ends are not directional, but it seems to be the case on Linux.
So the test should be fixed on Linux a well.
Please move this class before test_main().

vstinner · 2019-08-28T21:24:12Z

+ _socket.CMSG_LEN(maxfds * fds.itemsize))
+ for cmsg_level, cmsg_type, cmsg_data in ancdata:
+ if (cmsg_level == _socket.SOL_SOCKET and cmsg_type == _socket.SCM_RIGHTS):
+ # Append data, ignoring any truncated integers at the end.


@nanjekyejoannah: Would you mind to document this behavior in recv_fds() documention?

vstinner · 2019-08-28T21:26:50Z

 socketpair() -- create a pair of new socket objects [*]
 fromfd() -- create a socket object from an open file descriptor [*]
+send_fds() -- Send file descriptor to the socket.
+recv_fds() -- Recieve file descriptor from the socket.


I suggest to replace "file descriptor" with "file descriptors" for both functions here (add an S), as in their name, since they can send and receive more than one file descriptor ;-)

bedevere-bot · 2019-08-28T21:27:30Z

A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated.

Once you have made the requested changes, please leave a comment on this pull request containing the phrase I have made the requested changes; please review again. I will then notify any core developers who have left a review that you're ready for them to take another look at this pull request.

vstinner · 2019-09-09T10:29:57Z

The test is skipped. When I enabled manually the test, it failed on Linux :-(

test_main() is awful... and it's no longer needed: unittest.main() can be used instead.

I created https://bugs.python.org/issue38063 "Modify test_socket.py to use unittest test discovery" and I wrote a PR to implement it.

nanjekyejoannah · 2019-09-11T15:55:14Z

@vstinner

Test passes with your fix.

Ran 580 tests in 24.587s
OK (skipped=118)
== Tests result: SUCCESS ==
1 test OK.
Total duration: 24 sec 956 ms
Tests result: SUCCES

Also the test ir running:
testSendAndRecvFds (test.test_socket.SendRecvFdsTests) ... ok

vstinner · 2019-09-11T17:09:29Z

I tested manually commit 1ac7fbf on Linux and FreeBSD. I merged the PR with the master branch, make && ./python -m test test_socket -m SendRecvFdsTests -v . The test pass on Linux and FreeBSD. Good!

According to Azure Pipelines: test_socket also pass on macOS (and the test is no longer skipped on macOS).

test_socket pass on Windows as well, even if the test is skipped on Windows.

vstinner · 2019-09-11T17:17:43Z

Thanks Shinya Okano for the original patch.

Well done Joannah @nanjekyejoannah! Thanks for your tenacity :-) This PR has 100 comments and 27 commits which shows the complexity of the feature.

Honestly, I'm not 100% happy with current documentation, but I chose to merge the PR anyway. Please leave https://bugs.python.org/issue28724 open until the documentation is completed to mention corner cases like partial send (similar to sock.send vs sock.sendall).

the-knights-who-say-ni added the CLA signed label Apr 20, 2019

bedevere-bot added the awaiting review label Apr 20, 2019

nanjekyejoannah changed the title ~~Add method send_fds, recv_fds to the socket module~~bpo-28724: Add method send_fds, recv_fds to the socket moduleApr 20, 2019

nanjekyejoannah changed the title ~~bpo-28724: Add method send_fds, recv_fds to the socket module~~bpo-28724: Add methodS send_fds, recv_fds to the socket moduleApr 20, 2019

nanjekyejoannah changed the title ~~bpo-28724: Add methodS send_fds, recv_fds to the socket module~~bpo-28724: Add methods send_fds, recv_fds to the socket moduleApr 22, 2019

ncoghlan requested changes May 4, 2019
View reviewed changes

Comment threadDoc/library/socket.rst Outdated
Comment threadDoc/library/socket.rst Outdated
Comment threadLib/socket.py Outdated
Comment threadLib/test/test_socket.py Outdated
Comment threadLib/socket.py Outdated
Comment threadLib/socket.py Outdated

bedevere-bot added awaiting changes and removed awaiting review labels May 4, 2019

nanjekyejoannah requested review from a team, 1st1, asvetlov, ericvsmith, gpshead, pablogsal and rhettinger as code owners May 6, 2019 17:09

nanjekyejoannah force-pushed the issue28724 branch 3 times, most recently from 40d7ea4 to f3e0822Compare May 10, 2019 19:26

pablogsal reviewed May 10, 2019
View reviewed changes

nanjekyejoannah requested review from brettcannon, encukou, ericsnowcurrently, gvanrossum, ilevkivskyi, terryjreedy, tiran and warsaw as code owners May 30, 2019 15:13

pablogsal reviewed Jun 13, 2019
View reviewed changes

Comment threadLib/test/test_socket.py Outdated

pablogsal removed request for berkerpeksag, methane and skrah June 13, 2019 17:30

Do not skip Macos in tests
0f03c42

bedevere-bot added awaiting change review and removed awaiting changes labels Aug 24, 2019

vstinner requested changes Aug 26, 2019
View reviewed changes

bedevere-bot removed the awaiting change review label Aug 26, 2019

bedevere-bot added the awaiting changes label Aug 26, 2019

Fix doc strings
bf67bd8

bedevere-bot added awaiting change review and removed awaiting changes labels Aug 28, 2019

vstinner requested changes Aug 28, 2019
View reviewed changes

bedevere-bot removed the awaiting change review label Aug 28, 2019

Fix test
1ac7fbf

ezio-melotti mentioned this pull request May 31, 2022
Remove execution bit from socket-related files. #93367
Closed

	receiveuptomaxfdsfiledescriptorsreturningthemessage
	Receiveuptomaxfdsfiledescriptorsreturningthemessage

	msg, fds2, flags, addr=socket.recv_fds(sock2, len(MSG) 2, len(fds) 2)
	msg, fds2, flags, addr=socket.recv_fds(sock2, len(MSG) 2, len(fds) 2)

Uh oh!

Uh oh!

Conversation

nanjekyejoannah commented Apr 20, 2019• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

bedevere-bot commented May 4, 2019

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

nanjekyejoannah commented May 27, 2019

Uh oh!

Uh oh!

vstinner commented Aug 20, 2019

Uh oh!

nanjekyejoannah commented Aug 20, 2019

Uh oh!

nanjekyejoannah commented Aug 24, 2019

Uh oh!

nanjekyejoannah commented Aug 24, 2019

Uh oh!

bedevere-bot commented Aug 24, 2019

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bedevere-bot commented Aug 26, 2019

Uh oh!

nanjekyejoannah commented Aug 28, 2019

Uh oh!

bedevere-bot commented Aug 28, 2019

Uh oh!

vstinner left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bedevere-bot commented Aug 28, 2019

Uh oh!

vstinner commented Sep 9, 2019

Uh oh!

nanjekyejoannah commented Sep 11, 2019• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vstinner commented Sep 11, 2019

Uh oh!

vstinner commented Sep 11, 2019

Uh oh!

Reviewers

nanjekyejoannah commented Apr 20, 2019•
edited
Loading

nanjekyejoannah commented Sep 11, 2019•
edited
Loading