gateway: expose commands for js-libp2p delegated routing

[ghost]

This enables js-libp2p/js-ipfs to delegate peer routing and content routing to a go-ipfs gateway (usually ipfs.io). It covers:

• Peer routing lookups (dht/findpeer)
• Content routing lookups (dht/findprovs)
• Content routing writes -- go-libp2p-kad-dht rejects provider records which originate from a different node than the message sender, so we're doing a creative hack here: we'll tell the delegatee to connect to us (swarm/connect with a /p2p-circuit address), and then to fetch the data from us (refs?r=true). This isn't technically content routing, but it makes the data available to the network which is all we need.

Peer routing writes will be a new command that'll be covered in a future PR.

(@diasdavid catch!)

[ghost]

Oh and this is already running on ipfs.io, so go take a shot at it @diasdavid

[Stebalien]

Would putting these behind a config option be problematic? I worry about increasing the default gateway API a bit.

[ghost]

Would putting these behind a config option be problematic? I worry about increasing the default gateway API a bit.

Not at all, you have a point. I'll update this in the next few days.

[Kubuxu]

@magik6k mind taking over?

[magik6k]

I don't see any reasonably simple way to put this behind a config option (rootROOldSubcommands is set up in init), one 'hack' I have in mind is to create a utility command which would perform that check and proxy to the proper command, does that sound reasonable?

[Stebalien]

Can the command itself not check and return a permission denied error? Not the best solution but it works.

[magik6k]

It would have to check if it was called from the gateway port in that case which I'm not sure is even possible.

[ghost]

We could maybe eliminate the root/rootRO duality and have :8080 call the same root cmds.Command as :5001, but with a configurable allowlist in front. The current rootRO would then be default config for that.

[ghost]

but with a configurable allowlist in front

and a hardcoded list of commands that should not ever be exposed to the public.

[ghost]

I'm picking this back up this week -- let me know if there's reservations about the approach mentioned back in March:

We could maybe eliminate the root/rootRO duality and have :8080 call the same root cmds.Command as :5001, but with a configurable allowlist in front. The current rootRO would then be default config for that.

[jacobheun]

@lgierth I think we may need to add some limiters on findProviders. I am testing this locally with an example of the current delegated routing implementation in js-libp2p. Other than limiting the number of providers there isn't currently any other limiter to the call. If I run ipfs dht findprovs for a newer CID and the number of providers is below 20 (the default), or the query takes a while to find 20, the connection stays open until at least 20 providers can be found.

For an http request to the delegated node, this can mean multiple http requests open for minutes at a time. For content I just added to my local go-ipfs instance, I've seen it exceed 30minutes for an open connection in browser.

[ghost]

@jacobheun In go-ipfs there's a -n/--num-providers (default=20) option on ipfs dht findprovs. But nothing similar on ipfs dht findpeer yet.

[ghost]

@jacobheun Just to confirm, do you need limiting on findprovs too?

[jacobheun]

@lgierth my primary concern is too many open connections to the server. The current code for findprovs keeps the connection open until the default 20 providers are returned.

In the event that findprovs is called for new content by several peers, the requests from those peers will stay open until resolved (which can take a significant amount of time).

Even though I can lower --num-providers, it's not the default so most people won't use it, unless we explicitly lower it in the js module.

I'm wondering if a default timeout is needed for these requests?

[ghost]

Yeah sounds like some combination of --num-providers and --timeout is needed, and a reasonable default should be used in the delegated routing module.

[ghost]

@jacobheun Just to confirm, do you need limiting on findprovs too?

Eeeh what I meant to say here: do you need the limiting on findpeer too?

[jacobheun]

@lgierth I think --timeout limiting for findpeer would be great.

[ghost]

Okay, cool -- --timeout already exists in go-ipfs, it's an option on the root command: ipfs --help

[jacobheun]

Perfect, I can work on adding support and sane defaults for those in the routing modules.

[jacobheun]

I've got the js-libp2p delegated routing work all ready to go and in review. I've got the timeouts configurable and currently defaulting to a max of 30 seconds. It's been verified against the example I demo'd a few weeks back, which is also in the PR for js-libp2p libp2p/js-libp2p#242

[ghost]

Update on the go side is that configurable Gateway API commands (aka "readonly API" commands) are ready, but I'm dealing with some nasty dependency issues - hopefully done by tomorrow.

[ghost]

Closing this in favor of #5565