[ggiesen]

What does this PR do?

Makes Minion._mine_send() a coroutine that sends on an AsyncReqChannel bound to the minion's loop, and awaits it from handle_event. Before, it opened a synchronous ReqChannel and blocked in send() on the minion's main event loop, which a multi-master minion shares between all its masters. While a master was unreachable that took 15 seconds to almost 2 minutes per mine run, and the minion didn't answer the other masters in the meantime.

Events reach handle_event as separate tasks (on_recv_handler in salt/transport/tcp.py), so awaiting the send there doesn't hold up other events.

This also takes away the trigger for #70353 on this path: the SyncWrapper behind the ReqChannel was closed while the minion's loop was running, which left an AsyncAuth task and its socket behind. #70415 fixes that in SyncWrapper. The two PRs don't depend on each other.

With the loop no longer blocked, a mine run can start while the previous one is still waiting on an unreachable master. Sends for one minion ID go to a master one at a time, in the order the events arrived, so a newer update or a mine.delete isn't sent while an older update is still in flight. At most three are pending per minion ID and master; later ones are dropped with a warning, the same as a send that gives up on an unreachable master. Each send queues for the master that was current when its event arrived, so after a failover the sends stuck on the old master don't hold up the new one. A deltaproxy's sub-proxies have their own IDs and don't wait on each other.

With the TCP transport a send to a master that is down waits until the master is back, because the send timeout only starts once the connection is up (#66966). This PR doesn't change the transport; the cap limits how many sends that ties up.

What issues does this PR fix or reference?

Fixes #70421

Refs #70353, #70415, #66966

Previous Behavior

While a master was unreachable, each mine run (mine_interval or mine.update) blocked the minion's event loop until the send gave up, so the minion didn't process jobs from the masters that were up.

New Behavior

The mine send runs on the event loop without blocking it. Sends for one minion ID still go to the master one at a time and in order, and once three are pending for an ID and master, new ones are dropped with a warning.

Testing

• New tests/pytests/unit/test_minion.py::test_minion_mine_event_does_not_block_io_loop fails on 3008.x (assert 0 >= 10: no heartbeat ticks while the send runs) and passes with this change. test_mine_send_tries is updated for the coroutine.
• New tests for the ordering and the cap: test_mine_sends_for_one_minion_id_are_ordered, test_mine_send_queued_after_earlier_one_finished_keeps_order, test_mine_sends_after_failover_do_not_wait_on_old_master, test_mine_sends_pending_for_one_minion_id_are_capped, test_mine_sends_for_different_minion_ids_do_not_wait and test_mine_send_state_is_cleared. All but the different-IDs one fail without the ordering commit.
• tests/pytests/unit/test_minion.py, test_minion_resources.py, test_proxy_minion.py and modules/test_mine.py with --run-slow: 216 passed on Python 3.11, 3.12 and 3.14.

End to end, re-run on 2026-10-30 with this branch's salt/minion.py change, against the 3008.3 onedir packages (Python 3.14.7, Ubuntu 24.04 containers), with the setup from #70421: two masters, master_tries: -1, ping_interval: 5, a test.ping mine function, and minion -> master1 traffic on 4505/4506 dropped with iptables. 3008.x's salt/minion.py has moved on since 3008.3, so the PR's diff to that file is applied to the 3008.3 copy (it applies cleanly) and mounted over the onedir one. A loop runs salt '*' test.ping on master2 every 11 seconds or so (every 41 on plain 3008.3 over TCP, where each one ran into its 20 second timeout).

	transport, mine_interval, outage	test.ping via master2 during the outage	wedged in zmq_ctx_term()	after master1 is back
this PR, 12 minions	zeromq, 5, 57 min	279 of 279 samples 12/12, slowest 5s	0/12	12/12 via both masters
this PR, 12 minions	tcp, 1, 26 min	135 of 135 samples 12/12, slowest 2s	-	12/12 via both masters
plain 3008.3, 6 minions	tcp, 1, 29 min	0/6 in 35 of 37 samples, 1/6 in the first two	-	6/6 via both masters

For comparison, in the 2026-10-31 runs for #70415 (zeromq, mine_interval: 5, same sampler), plain 3008.3 wedged 12/12 about 22 minutes into the outage, and with #70415 alone 14 of 215 samples got no answer from any minion, all during mine runs.

On zeromq each mine send to master1 gave up after 15 to 30 seconds with Unable to send mine data to master., so the cap never came into play, and master1 had current mine data again from the first mine run after it was back. On TCP the sends to master1 never time out (#66966): the cap kicked in about 3 minutes into the outage, each later mine run logged Not sending mine data for <id> to master <master>, 3 earlier mine sends are still pending., and apart from the two minutes after the disconnect (about 14 minutes in) each minion had a single connection attempt to master1 open. Plain 3008.3 on TCP stopped answering at the first mine run of the outage and didn't answer again until master1 was back.

Merge requirements satisfied?

• Docs
• Changelog
• Tests written/updated

Commits signed with GPG?

No

[ggiesen]

The failing integration jobs here all fail in TestSyndic::test_fib and test_ping, which isn't this change. tests/filename_map.yml maps salt/minion.py to tests/integration/client/test_syndic.py, and that test fails whenever it runs without the other client tests ahead of it (#58975), so it fails the same way on 3008.x. #70441 fixes the test. Ubuntu 26.04 also had one failure in test_orchestration_with_pillar_dot_items, which I haven't looked into.

[twangboy]

Please resolve conflicts

[charzl]

Could we also keep Mine sends ordered per master and limit how many can pile up? Otherwise, an older update could finish after a newer one or a delete and bring stale data back. Also, TCP’s send timeout only starts after the connection succeeds, so sends can hang indefinitely when a master is down.

[ggiesen]

@charzl Both hold. The new commit handles the ordering and the pile-up.

Each mine send now holds a lock keyed on the master URI and minion ID from opening the channel until it's closed, retries included, so sends for one ID go to a master one at a time in the order the events arrived. At most three are pending per ID and master (one in flight, two waiting). Past that the new send is dropped with a warning (Not sending mine data for <id> to master <master>, 3 earlier mine sends are still pending.), which is the same outcome as a send that gives up on an unreachable master today. It drops the new send rather than a waiting one, so nothing gets cancelled and what does go out stays in order. mine_functions data comes back with the next mine.update; a dropped ad hoc mine.send or mine.delete is lost.

The key uses the ID from the load, so deltaproxy sub-proxies don't wait on each other. On failover eval_master updates self.opts in place, so each send takes a copy of the opts when its event arrives and queues for that master; a send stuck on the old master doesn't hold up the new one. Ordering is only kept on the minion side: a send that times out on the minion while the master is still processing it can still be applied after the next one, as before.

On TCP: RequestClient.send() awaits self.connect() before it arms the timeout, and getstream() retries every tcp_reconnect_backoff until it connects or the client is closed, so timeout and tries do nothing while the master is down:

salt/salt/transport/tcp.py

Lines 3100 to 3117 in 621f88e

	async def send(self, load, timeout=60):
	await self.connect()
	if self._closing:
	raise ClosingError()
	while not self._stream:
	await asyncio.sleep(0.03)
	message_id = self._message_id()
	header = {"mid": message_id}
	future = tornado.concurrent.Future()
	# Add this future to the mapping
	self.send_future_map[message_id] = future
	if self.opts.get("detect_mode") is True:
	timeout = 1
	if timeout is not None:
	self.io_loop.call_later(timeout, self.timeout_message, message_id, load)

That came in with #64705 (9683260), so 3007.0 onwards, and it's #66966. Job returns and AsyncAuth.sign_in() hang the same way.

I didn't wrap the send in wait_for. With this PR the mine channel shares the minion loop's AsyncAuth, and cancelling a send that's waiting in authenticate() cancels the shared _authenticate_future. The minion's req_channel then gets CancelledError on its next send, and auth.authenticated keeps raising after the sign-in completes.

I'd rather fix the timeout in salt/transport/tcp.py in a separate PR: arm it before connecting, don't write a request that has already timed out, and share one connect attempt between callers. #70602 and #70437 change the same send()/connect() code on master, so let me know if you'd rather it go into one of those. Until then a TCP outage leaves at most three mine sends pending per ID and master, with one open channel, instead of a new channel and reconnect loop for every mine run.

@twangboy I merged 3008.x into the branch; the conflict was only the imports in tests/pytests/unit/test_minion.py.