SOLR-18413: Handle deleted MIGRATE routing targets

[JHSUYU]

JIRA: SOLR-18413

Description

A SolrCloud MIGRATE adds a temporary routing rule to the source slice so that matching updates are also forwarded to the target collection.

For example, migrating route key a! from source to target creates state similar to:

source/shard1.routingRules["a!"].targetCollection = "target"

The routing rule stores the target collection name as a string. Deleting target does not remove routing rules in other collections that refer to it, so the source can retain an unexpired rule whose target is no longer present in ClusterState.

A subsequent matching update, such as adding id=a!2 to source, is first applied locally on the source shard leader. DistributedZkUpdateProcessor.doDistribAdd() then resolves routing-rule destinations before distributing the update to the source shard replicas:

finalList<SolrCmdDistributor.Node> nodesByRoutingRules =
getNodesByRoutingRules(
clusterState, coll, cmd.getIndexedIdStr(), cmd.getSolrInputDocument());
...
if (nodes != null) {
cmdDistrib.distribAdd(...);
}

getNodesByRoutingRules() previously resolved the target using:

DocCollectiontargetColl =
cstate.getCollection(rule.getTargetCollectionName());

If the target had been deleted, this threw:

HTTP 400
Could not find collection : target

Because the source leader had already applied the update, but the exception occurred before source-replica distribution, the request could leave the shard in this state:

source shard leader: a!2 exists
source shard replica: a!2 is missing
client: HTTP 400

No request was sent to the source replica, so SolrCmdDistributor did not record a replica failure and this update did not trigger term demotion or recovery. Queries can then return different results depending on which replica serves them. If leadership subsequently moves to a replica that never received the update, the document is lost.

Solution

Resolve the routing-rule target with getCollectionOrNull():

DocCollectiontargetColl =
cstate.getCollectionOrNull(rule.getTargetCollectionName());
if (targetColl == null) {
removeRoutingRule(myShardId, routeKey);
break;
}

When the target collection no longer exists, the routing rule is treated as invalid. The existing routing-rule removal logic used for expired rules was extracted into removeRoutingRule(). Expired rules and rules with missing targets now share the same cleanup path. The cleanup remains guarded by the core's routing-rule lock.

The behavior for a target collection that exists but has no active slices is unchanged. That condition may be temporary and continues to produce an error rather than permanently removing the routing rule.

Tests

Added:

• MigrateRouteKeyTest.updateSucceedsAfterMigrateTargetIsDeleted

The test uses a real two-node MiniSolrCloudCluster and performs this sequence:

1. Creates a source collection.
2. Creates a target collection.
3. Adds and commits id=a!1 to the source.
4. Migrates route key a! from the source to the target.
5. Confirms that the source shard contains an active routing rule targeting the target collection.
6. Deletes the target collection and waits for it to disappear from ClusterState.
7. Adds id=a!2 to the source and commits.
8. Queries each source replica directly with distrib=false.
9. Confirms that both the leader and non-leader replica contain a!2.
10. Confirms that the dangling routing rule is removed.

Without the fix, the add in step 7 fails with:

Could not find collection : deletedMigrateTarget-target

[dsmiley]

Nice! I love your detailed explanation.

Note that you kept referring to "replica" in a way that meant to imply a "follower replica" (instead of the "leader replica"). Basically, the leader is a replica too!

Needs a changelog (see writeChangelog task). I'm not sure this is a bug, maybe it's just a "changed" (improvement). Assuming a user could repair the situation on their own with the HTTP API. If they delete the collection, they ought to first undo the migration. If they can't, then I'm more sympathetic to this being a bug.

[dsmiley]

Let's be clearer. I suggest "Removing shard update routing rule because the target collection ____ doesn't exist".

[JHSUYU]

JIRA: SOLR-18413

Description

A SolrCloud MIGRATE adds a temporary routing rule to the source slice so that matching updates are also forwarded to the target collection.

For example, migrating route key a! from source to target creates state similar to:

source/shard1.routingRules["a!"].targetCollection = "target"

The routing rule stores the target collection name as a string. Deleting target does not remove routing rules in other collections that refer to it, so the source can retain an unexpired rule whose target is no longer present in ClusterState.

A subsequent matching update, such as adding id=a!2 to source, is first applied locally on the source shard leader. DistributedZkUpdateProcessor.doDistribAdd() then resolves routing-rule destinations before distributing the update to the source shard replicas:

finalList<SolrCmdDistributor.Node> nodesByRoutingRules =
getNodesByRoutingRules(
clusterState, coll, cmd.getIndexedIdStr(), cmd.getSolrInputDocument());
...
if (nodes != null) {
cmdDistrib.distribAdd(...);
}

getNodesByRoutingRules() previously resolved the target using:

DocCollectiontargetColl =
cstate.getCollection(rule.getTargetCollectionName());

If the target had been deleted, this threw:

HTTP 400
Could not find collection : target

Because the source leader had already applied the update, but the exception occurred before source-replica distribution, the request could leave the shard in this state:

source shard leader: a!2 exists
source shard replica: a!2 is missing
client: HTTP 400

No request was sent to the source replica, so SolrCmdDistributor did not record a replica failure and this update did not trigger term demotion or recovery. Queries can then return different results depending on which replica serves them. If leadership subsequently moves to a replica that never received the update, the document is lost.

Solution

Resolve the routing-rule target with getCollectionOrNull():

DocCollectiontargetColl =
cstate.getCollectionOrNull(rule.getTargetCollectionName());
if (targetColl == null) {
removeRoutingRule(myShardId, routeKey);
break;
}

When the target collection no longer exists, the routing rule is treated as invalid. The existing routing-rule removal logic used for expired rules was extracted into removeRoutingRule(). Expired rules and rules with missing targets now share the same cleanup path. The cleanup remains guarded by the core's routing-rule lock.

The behavior for a target collection that exists but has no active slices is unchanged. That condition may be temporary and continues to produce an error rather than permanently removing the routing rule.

Tests

Added:

• MigrateRouteKeyTest.updateSucceedsAfterMigrateTargetIsDeleted

The test uses a real two-node MiniSolrCloudCluster and performs this sequence:

1. Creates a source collection.
2. Creates a target collection.
3. Adds and commits id=a!1 to the source.
4. Migrates route key a! from the source to the target.
5. Confirms that the source shard contains an active routing rule targeting the target collection.
6. Deletes the target collection and waits for it to disappear from ClusterState.
7. Adds id=a!2 to the source and commits.
8. Queries each source replica directly with distrib=false.
9. Confirms that both the leader and non-leader replica contain a!2.
10. Confirms that the dangling routing rule is removed.

Without the fix, the add in step 7 fails with:

Could not find collection : deletedMigrateTarget-target

[dsmiley]

Nice! I love your detailed explanation.

Note that you kept referring to "replica" in a way that meant to imply a "follower replica" (instead of the "leader replica"). Basically, the leader is a replica too!

Needs a changelog (see writeChangelog task). I'm not sure this is a bug, maybe it's just a "changed" (improvement). Assuming a user could repair the situation on their own with the HTTP API. If they delete the collection, they ought to first undo the migration. If they can't, then I'm more sympathetic to this being a bug.

[dsmiley]

Let's be clearer. I suggest "Removing shard update routing rule because the target collection ____ doesn't exist".

[JHSUYU]

JIRA: SOLR-18413

Description

A SolrCloud MIGRATE adds a temporary routing rule to the source slice so that matching updates are also forwarded to the target collection.

For example, migrating route key a! from source to target creates state similar to:

source/shard1.routingRules["a!"].targetCollection = "target"

The routing rule stores the target collection name as a string. Deleting target does not remove routing rules in other collections that refer to it, so the source can retain an unexpired rule whose target is no longer present in ClusterState.

A subsequent matching update, such as adding id=a!2 to source, is first applied locally on the source shard leader. DistributedZkUpdateProcessor.doDistribAdd() then resolves routing-rule destinations before distributing the update to the source shard replicas:

finalList<SolrCmdDistributor.Node> nodesByRoutingRules =
getNodesByRoutingRules(
clusterState, coll, cmd.getIndexedIdStr(), cmd.getSolrInputDocument());
...
if (nodes != null) {
cmdDistrib.distribAdd(...);
}

getNodesByRoutingRules() previously resolved the target using:

DocCollectiontargetColl =
cstate.getCollection(rule.getTargetCollectionName());

If the target had been deleted, this threw:

HTTP 400
Could not find collection : target

Because the source leader had already applied the update, but the exception occurred before source-replica distribution, the request could leave the shard in this state:

source shard leader: a!2 exists
source shard replica: a!2 is missing
client: HTTP 400

No request was sent to the source replica, so SolrCmdDistributor did not record a replica failure and this update did not trigger term demotion or recovery. Queries can then return different results depending on which replica serves them. If leadership subsequently moves to a replica that never received the update, the document is lost.

Solution

Resolve the routing-rule target with getCollectionOrNull():

DocCollectiontargetColl =
cstate.getCollectionOrNull(rule.getTargetCollectionName());
if (targetColl == null) {
removeRoutingRule(myShardId, routeKey);
break;
}

When the target collection no longer exists, the routing rule is treated as invalid. The existing routing-rule removal logic used for expired rules was extracted into removeRoutingRule(). Expired rules and rules with missing targets now share the same cleanup path. The cleanup remains guarded by the core's routing-rule lock.

The behavior for a target collection that exists but has no active slices is unchanged. That condition may be temporary and continues to produce an error rather than permanently removing the routing rule.

Tests

Added:

• MigrateRouteKeyTest.updateSucceedsAfterMigrateTargetIsDeleted

The test uses a real two-node MiniSolrCloudCluster and performs this sequence:

1. Creates a source collection.
2. Creates a target collection.
3. Adds and commits id=a!1 to the source.
4. Migrates route key a! from the source to the target.
5. Confirms that the source shard contains an active routing rule targeting the target collection.
6. Deletes the target collection and waits for it to disappear from ClusterState.
7. Adds id=a!2 to the source and commits.
8. Queries each source replica directly with distrib=false.
9. Confirms that both the leader and non-leader replica contain a!2.
10. Confirms that the dangling routing rule is removed.

Without the fix, the add in step 7 fails with:

Could not find collection : deletedMigrateTarget-target

[dsmiley]

Nice! I love your detailed explanation.

Note that you kept referring to "replica" in a way that meant to imply a "follower replica" (instead of the "leader replica"). Basically, the leader is a replica too!

Needs a changelog (see writeChangelog task). I'm not sure this is a bug, maybe it's just a "changed" (improvement). Assuming a user could repair the situation on their own with the HTTP API. If they delete the collection, they ought to first undo the migration. If they can't, then I'm more sympathetic to this being a bug.

[dsmiley]

Let's be clearer. I suggest "Removing shard update routing rule because the target collection ____ doesn't exist".

[JHSUYU]

JIRA: SOLR-18413

Description

A SolrCloud MIGRATE adds a temporary routing rule to the source slice so that matching updates are also forwarded to the target collection.

For example, migrating route key a! from source to target creates state similar to:

source/shard1.routingRules["a!"].targetCollection = "target"

The routing rule stores the target collection name as a string. Deleting target does not remove routing rules in other collections that refer to it, so the source can retain an unexpired rule whose target is no longer present in ClusterState.

A subsequent matching update, such as adding id=a!2 to source, is first applied locally on the source shard leader. DistributedZkUpdateProcessor.doDistribAdd() then resolves routing-rule destinations before distributing the update to the source shard replicas:

finalList<SolrCmdDistributor.Node> nodesByRoutingRules =
getNodesByRoutingRules(
clusterState, coll, cmd.getIndexedIdStr(), cmd.getSolrInputDocument());
...
if (nodes != null) {
cmdDistrib.distribAdd(...);
}

getNodesByRoutingRules() previously resolved the target using:

DocCollectiontargetColl =
cstate.getCollection(rule.getTargetCollectionName());

If the target had been deleted, this threw:

HTTP 400
Could not find collection : target

Because the source leader had already applied the update, but the exception occurred before source-replica distribution, the request could leave the shard in this state:

source shard leader: a!2 exists
source shard replica: a!2 is missing
client: HTTP 400

No request was sent to the source replica, so SolrCmdDistributor did not record a replica failure and this update did not trigger term demotion or recovery. Queries can then return different results depending on which replica serves them. If leadership subsequently moves to a replica that never received the update, the document is lost.

Solution

Resolve the routing-rule target with getCollectionOrNull():

DocCollectiontargetColl =
cstate.getCollectionOrNull(rule.getTargetCollectionName());
if (targetColl == null) {
removeRoutingRule(myShardId, routeKey);
break;
}

When the target collection no longer exists, the routing rule is treated as invalid. The existing routing-rule removal logic used for expired rules was extracted into removeRoutingRule(). Expired rules and rules with missing targets now share the same cleanup path. The cleanup remains guarded by the core's routing-rule lock.

The behavior for a target collection that exists but has no active slices is unchanged. That condition may be temporary and continues to produce an error rather than permanently removing the routing rule.

Tests

Added:

• MigrateRouteKeyTest.updateSucceedsAfterMigrateTargetIsDeleted

The test uses a real two-node MiniSolrCloudCluster and performs this sequence:

1. Creates a source collection.
2. Creates a target collection.
3. Adds and commits id=a!1 to the source.
4. Migrates route key a! from the source to the target.
5. Confirms that the source shard contains an active routing rule targeting the target collection.
6. Deletes the target collection and waits for it to disappear from ClusterState.
7. Adds id=a!2 to the source and commits.
8. Queries each source replica directly with distrib=false.
9. Confirms that both the leader and non-leader replica contain a!2.
10. Confirms that the dangling routing rule is removed.

Without the fix, the add in step 7 fails with:

Could not find collection : deletedMigrateTarget-target

[dsmiley]

Nice! I love your detailed explanation.

Note that you kept referring to "replica" in a way that meant to imply a "follower replica" (instead of the "leader replica"). Basically, the leader is a replica too!

Needs a changelog (see writeChangelog task). I'm not sure this is a bug, maybe it's just a "changed" (improvement). Assuming a user could repair the situation on their own with the HTTP API. If they delete the collection, they ought to first undo the migration. If they can't, then I'm more sympathetic to this being a bug.

[dsmiley]

Let's be clearer. I suggest "Removing shard update routing rule because the target collection ____ doesn't exist".

[JHSUYU]

JIRA: SOLR-18413

Description

A SolrCloud MIGRATE adds a temporary routing rule to the source slice so that matching updates are also forwarded to the target collection.

For example, migrating route key a! from source to target creates state similar to:

source/shard1.routingRules["a!"].targetCollection = "target"

The routing rule stores the target collection name as a string. Deleting target does not remove routing rules in other collections that refer to it, so the source can retain an unexpired rule whose target is no longer present in ClusterState.

A subsequent matching update, such as adding id=a!2 to source, is first applied locally on the source shard leader. DistributedZkUpdateProcessor.doDistribAdd() then resolves routing-rule destinations before distributing the update to the source shard replicas:

finalList<SolrCmdDistributor.Node> nodesByRoutingRules =
getNodesByRoutingRules(
clusterState, coll, cmd.getIndexedIdStr(), cmd.getSolrInputDocument());
...
if (nodes != null) {
cmdDistrib.distribAdd(...);
}

getNodesByRoutingRules() previously resolved the target using:

DocCollectiontargetColl =
cstate.getCollection(rule.getTargetCollectionName());

If the target had been deleted, this threw:

HTTP 400
Could not find collection : target

Because the source leader had already applied the update, but the exception occurred before source-replica distribution, the request could leave the shard in this state:

source shard leader: a!2 exists
source shard replica: a!2 is missing
client: HTTP 400

No request was sent to the source replica, so SolrCmdDistributor did not record a replica failure and this update did not trigger term demotion or recovery. Queries can then return different results depending on which replica serves them. If leadership subsequently moves to a replica that never received the update, the document is lost.

Solution

Resolve the routing-rule target with getCollectionOrNull():

DocCollectiontargetColl =
cstate.getCollectionOrNull(rule.getTargetCollectionName());
if (targetColl == null) {
removeRoutingRule(myShardId, routeKey);
break;
}

When the target collection no longer exists, the routing rule is treated as invalid. The existing routing-rule removal logic used for expired rules was extracted into removeRoutingRule(). Expired rules and rules with missing targets now share the same cleanup path. The cleanup remains guarded by the core's routing-rule lock.

The behavior for a target collection that exists but has no active slices is unchanged. That condition may be temporary and continues to produce an error rather than permanently removing the routing rule.

Tests

Added:

• MigrateRouteKeyTest.updateSucceedsAfterMigrateTargetIsDeleted

The test uses a real two-node MiniSolrCloudCluster and performs this sequence:

1. Creates a source collection.
2. Creates a target collection.
3. Adds and commits id=a!1 to the source.
4. Migrates route key a! from the source to the target.
5. Confirms that the source shard contains an active routing rule targeting the target collection.
6. Deletes the target collection and waits for it to disappear from ClusterState.
7. Adds id=a!2 to the source and commits.
8. Queries each source replica directly with distrib=false.
9. Confirms that both the leader and non-leader replica contain a!2.
10. Confirms that the dangling routing rule is removed.

Without the fix, the add in step 7 fails with:

Could not find collection : deletedMigrateTarget-target

[dsmiley]

Nice! I love your detailed explanation.

Note that you kept referring to "replica" in a way that meant to imply a "follower replica" (instead of the "leader replica"). Basically, the leader is a replica too!

Needs a changelog (see writeChangelog task). I'm not sure this is a bug, maybe it's just a "changed" (improvement). Assuming a user could repair the situation on their own with the HTTP API. If they delete the collection, they ought to first undo the migration. If they can't, then I'm more sympathetic to this being a bug.

[dsmiley]

Let's be clearer. I suggest "Removing shard update routing rule because the target collection ____ doesn't exist".

[JHSUYU]

JIRA: SOLR-18413

Description

A SolrCloud MIGRATE adds a temporary routing rule to the source slice so that matching updates are also forwarded to the target collection.

For example, migrating route key a! from source to target creates state similar to:

source/shard1.routingRules["a!"].targetCollection = "target"

The routing rule stores the target collection name as a string. Deleting target does not remove routing rules in other collections that refer to it, so the source can retain an unexpired rule whose target is no longer present in ClusterState.

A subsequent matching update, such as adding id=a!2 to source, is first applied locally on the source shard leader. DistributedZkUpdateProcessor.doDistribAdd() then resolves routing-rule destinations before distributing the update to the source shard replicas:

finalList<SolrCmdDistributor.Node> nodesByRoutingRules =
getNodesByRoutingRules(
clusterState, coll, cmd.getIndexedIdStr(), cmd.getSolrInputDocument());
...
if (nodes != null) {
cmdDistrib.distribAdd(...);
}

getNodesByRoutingRules() previously resolved the target using:

DocCollectiontargetColl =
cstate.getCollection(rule.getTargetCollectionName());

If the target had been deleted, this threw:

HTTP 400
Could not find collection : target

Because the source leader had already applied the update, but the exception occurred before source-replica distribution, the request could leave the shard in this state:

source shard leader: a!2 exists
source shard replica: a!2 is missing
client: HTTP 400

No request was sent to the source replica, so SolrCmdDistributor did not record a replica failure and this update did not trigger term demotion or recovery. Queries can then return different results depending on which replica serves them. If leadership subsequently moves to a replica that never received the update, the document is lost.

Solution

Resolve the routing-rule target with getCollectionOrNull():

DocCollectiontargetColl =
cstate.getCollectionOrNull(rule.getTargetCollectionName());
if (targetColl == null) {
removeRoutingRule(myShardId, routeKey);
break;
}

When the target collection no longer exists, the routing rule is treated as invalid. The existing routing-rule removal logic used for expired rules was extracted into removeRoutingRule(). Expired rules and rules with missing targets now share the same cleanup path. The cleanup remains guarded by the core's routing-rule lock.

The behavior for a target collection that exists but has no active slices is unchanged. That condition may be temporary and continues to produce an error rather than permanently removing the routing rule.

Tests

Added:

• MigrateRouteKeyTest.updateSucceedsAfterMigrateTargetIsDeleted

The test uses a real two-node MiniSolrCloudCluster and performs this sequence:

1. Creates a source collection.
2. Creates a target collection.
3. Adds and commits id=a!1 to the source.
4. Migrates route key a! from the source to the target.
5. Confirms that the source shard contains an active routing rule targeting the target collection.
6. Deletes the target collection and waits for it to disappear from ClusterState.
7. Adds id=a!2 to the source and commits.
8. Queries each source replica directly with distrib=false.
9. Confirms that both the leader and non-leader replica contain a!2.
10. Confirms that the dangling routing rule is removed.

Without the fix, the add in step 7 fails with:

Could not find collection : deletedMigrateTarget-target

[dsmiley]

Nice! I love your detailed explanation.

Note that you kept referring to "replica" in a way that meant to imply a "follower replica" (instead of the "leader replica"). Basically, the leader is a replica too!

Needs a changelog (see writeChangelog task). I'm not sure this is a bug, maybe it's just a "changed" (improvement). Assuming a user could repair the situation on their own with the HTTP API. If they delete the collection, they ought to first undo the migration. If they can't, then I'm more sympathetic to this being a bug.

[dsmiley]

Let's be clearer. I suggest "Removing shard update routing rule because the target collection ____ doesn't exist".

[JHSUYU]

JIRA: SOLR-18413

Description

A SolrCloud MIGRATE adds a temporary routing rule to the source slice so that matching updates are also forwarded to the target collection.

For example, migrating route key a! from source to target creates state similar to:

source/shard1.routingRules["a!"].targetCollection = "target"

The routing rule stores the target collection name as a string. Deleting target does not remove routing rules in other collections that refer to it, so the source can retain an unexpired rule whose target is no longer present in ClusterState.

A subsequent matching update, such as adding id=a!2 to source, is first applied locally on the source shard leader. DistributedZkUpdateProcessor.doDistribAdd() then resolves routing-rule destinations before distributing the update to the source shard replicas:

finalList<SolrCmdDistributor.Node> nodesByRoutingRules =
getNodesByRoutingRules(
clusterState, coll, cmd.getIndexedIdStr(), cmd.getSolrInputDocument());
...
if (nodes != null) {
cmdDistrib.distribAdd(...);
}

getNodesByRoutingRules() previously resolved the target using:

DocCollectiontargetColl =
cstate.getCollection(rule.getTargetCollectionName());

If the target had been deleted, this threw:

HTTP 400
Could not find collection : target

Because the source leader had already applied the update, but the exception occurred before source-replica distribution, the request could leave the shard in this state:

source shard leader: a!2 exists
source shard replica: a!2 is missing
client: HTTP 400

No request was sent to the source replica, so SolrCmdDistributor did not record a replica failure and this update did not trigger term demotion or recovery. Queries can then return different results depending on which replica serves them. If leadership subsequently moves to a replica that never received the update, the document is lost.

Solution

Resolve the routing-rule target with getCollectionOrNull():

DocCollectiontargetColl =
cstate.getCollectionOrNull(rule.getTargetCollectionName());
if (targetColl == null) {
removeRoutingRule(myShardId, routeKey);
break;
}

When the target collection no longer exists, the routing rule is treated as invalid. The existing routing-rule removal logic used for expired rules was extracted into removeRoutingRule(). Expired rules and rules with missing targets now share the same cleanup path. The cleanup remains guarded by the core's routing-rule lock.

The behavior for a target collection that exists but has no active slices is unchanged. That condition may be temporary and continues to produce an error rather than permanently removing the routing rule.

Tests

Added:

• MigrateRouteKeyTest.updateSucceedsAfterMigrateTargetIsDeleted

The test uses a real two-node MiniSolrCloudCluster and performs this sequence:

1. Creates a source collection.
2. Creates a target collection.
3. Adds and commits id=a!1 to the source.
4. Migrates route key a! from the source to the target.
5. Confirms that the source shard contains an active routing rule targeting the target collection.
6. Deletes the target collection and waits for it to disappear from ClusterState.
7. Adds id=a!2 to the source and commits.
8. Queries each source replica directly with distrib=false.
9. Confirms that both the leader and non-leader replica contain a!2.
10. Confirms that the dangling routing rule is removed.

Without the fix, the add in step 7 fails with:

Could not find collection : deletedMigrateTarget-target

[dsmiley]

Nice! I love your detailed explanation.

Note that you kept referring to "replica" in a way that meant to imply a "follower replica" (instead of the "leader replica"). Basically, the leader is a replica too!

Needs a changelog (see writeChangelog task). I'm not sure this is a bug, maybe it's just a "changed" (improvement). Assuming a user could repair the situation on their own with the HTTP API. If they delete the collection, they ought to first undo the migration. If they can't, then I'm more sympathetic to this being a bug.

[dsmiley]

Let's be clearer. I suggest "Removing shard update routing rule because the target collection ____ doesn't exist".

[JHSUYU]

JIRA: SOLR-18413

Description

A SolrCloud MIGRATE adds a temporary routing rule to the source slice so that matching updates are also forwarded to the target collection.

For example, migrating route key a! from source to target creates state similar to:

source/shard1.routingRules["a!"].targetCollection = "target"

The routing rule stores the target collection name as a string. Deleting target does not remove routing rules in other collections that refer to it, so the source can retain an unexpired rule whose target is no longer present in ClusterState.

A subsequent matching update, such as adding id=a!2 to source, is first applied locally on the source shard leader. DistributedZkUpdateProcessor.doDistribAdd() then resolves routing-rule destinations before distributing the update to the source shard replicas:

finalList<SolrCmdDistributor.Node> nodesByRoutingRules =
getNodesByRoutingRules(
clusterState, coll, cmd.getIndexedIdStr(), cmd.getSolrInputDocument());
...
if (nodes != null) {
cmdDistrib.distribAdd(...);
}

getNodesByRoutingRules() previously resolved the target using:

DocCollectiontargetColl =
cstate.getCollection(rule.getTargetCollectionName());

If the target had been deleted, this threw:

HTTP 400
Could not find collection : target

Because the source leader had already applied the update, but the exception occurred before source-replica distribution, the request could leave the shard in this state:

source shard leader: a!2 exists
source shard replica: a!2 is missing
client: HTTP 400

No request was sent to the source replica, so SolrCmdDistributor did not record a replica failure and this update did not trigger term demotion or recovery. Queries can then return different results depending on which replica serves them. If leadership subsequently moves to a replica that never received the update, the document is lost.

Solution

Resolve the routing-rule target with getCollectionOrNull():

DocCollectiontargetColl =
cstate.getCollectionOrNull(rule.getTargetCollectionName());
if (targetColl == null) {
removeRoutingRule(myShardId, routeKey);
break;
}

When the target collection no longer exists, the routing rule is treated as invalid. The existing routing-rule removal logic used for expired rules was extracted into removeRoutingRule(). Expired rules and rules with missing targets now share the same cleanup path. The cleanup remains guarded by the core's routing-rule lock.

The behavior for a target collection that exists but has no active slices is unchanged. That condition may be temporary and continues to produce an error rather than permanently removing the routing rule.

Tests

Added:

• MigrateRouteKeyTest.updateSucceedsAfterMigrateTargetIsDeleted

The test uses a real two-node MiniSolrCloudCluster and performs this sequence:

1. Creates a source collection.
2. Creates a target collection.
3. Adds and commits id=a!1 to the source.
4. Migrates route key a! from the source to the target.
5. Confirms that the source shard contains an active routing rule targeting the target collection.
6. Deletes the target collection and waits for it to disappear from ClusterState.
7. Adds id=a!2 to the source and commits.
8. Queries each source replica directly with distrib=false.
9. Confirms that both the leader and non-leader replica contain a!2.
10. Confirms that the dangling routing rule is removed.

Without the fix, the add in step 7 fails with:

Could not find collection : deletedMigrateTarget-target

[dsmiley]

Nice! I love your detailed explanation.

Note that you kept referring to "replica" in a way that meant to imply a "follower replica" (instead of the "leader replica"). Basically, the leader is a replica too!

Needs a changelog (see writeChangelog task). I'm not sure this is a bug, maybe it's just a "changed" (improvement). Assuming a user could repair the situation on their own with the HTTP API. If they delete the collection, they ought to first undo the migration. If they can't, then I'm more sympathetic to this being a bug.

[dsmiley]

Let's be clearer. I suggest "Removing shard update routing rule because the target collection ____ doesn't exist".