[Bug]: Mine send to an unreachable master blocks the minion's event loop #70421

What happened?

Minion.handle_event handles the _minion_mine event (fired by mine.update, including the __mine_interval schedule) by calling Minion._mine_send(), which is synchronous: it opens a ReqChannel (a SyncWrapper around AsyncReqChannel) and blocks in send(). handle_event runs on the minion's main event loop, which all the Minion objects of a multi-master MinionManager share. So while one master is unreachable, every mine run blocks that loop until the send gives up, and in the meantime the minion doesn't process anything else, including jobs published by the masters that are up.

Setup to reproduce, on the 3008.3 onedir packages (Python 3.14.7) in Ubuntu 24.04 containers:

master:
  - master1
  - master2
master_tries: -1
ping_interval: 5
mine_interval: 5
mine_functions:
  test.ping: []

Then drop the minion's traffic to master1 (iptables -I OUTPUT -d <master1> -p tcp -m multiport --dports 4505,4506 -j DROP) and run salt '*' test.ping on master2 in a loop.

What I see with 12 minions:

• Until the minion notices master1 is gone, each mine run blocks the loop for 15 to 30 seconds and ends with Unable to send mine data to master.
• Once the disconnect is handled (about 17 minutes in, when TCP keepalive gives up), each mine run blocks for about 105 seconds and ends with Error dispatching event. Attempt to authenticate with the salt master failed with timeout error.
• During those windows the minion doesn't answer master2. With the [Bug]: 3008.3 minion deadlocks in zmq_ctx_term() called from a GC weakref finalizer after a master outage #70353 fix (Fix minion deadlock in zmq_ctx_term() after a master outage #70415) applied, so that the minion doesn't wedge first, I sampled test.ping via master2 about once a minute from 17 minutes into a 53 minute outage until the end: 7 of 32 samples got no answer from any of the 12 minions, and each of the 7 fell inside a mine send.

This is also how #70353 starts: SyncWrapper.close() can't drain its private loop while the minion's loop is running in the same thread, which strands the AsyncAuth task that later wedges the minion.

_mine_send and its call site are the same on 3007.x and master.

Expected: a mine send to a master that is down shouldn't stop the minion from serving the other masters.

Type of salt install

Official deb

Major version

3008.x

What supported OS are you seeing the problem on? Can select multiple. (If bug appears on an unsupported OS, please open a GitHub Discussion instead)

ubuntu-24.04

salt --versions-report output

Salt Version:
          Salt: 3008.3
 
Python Version:
        Python: 3.14.7 (main, Sep 30 2026, 03:35:11) [GCC 11.2.0]
 
Dependency Versions:
          cffi: 2.0.0
      cherrypy: 18.10.0
  cryptography: 50.0.1
      dateutil: 2.9.0.post0
     docker-py: Not Installed
         gitdb: 4.0.12
     gitpython: Not Installed
        Jinja2: 3.1.6
       libgit2: Not Installed
  looseversion: 1.3.0
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.2.1
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 24.0
     pycparser: 3.00
      pycrypto: Not Installed
  pycryptodome: 3.23.0
        pygit2: Not Installed
  python-gnupg: 0.5.6
        PyYAML: 6.0.3
         PyZMQ: 27.1.0
        relenv: 0.22.27
         smmap: 5.0.2
       timelib: 0.3.0
       Tornado: 6.5.10
           ZMQ: 4.3.5
 
Salt Package Information:
  Package Type: onedir
 
System Versions:
          dist: ubuntu 24.04.5 noble
        locale: utf-8
       machine: x86_64
       release: 6.12.0-211.16.1.el10_2.0.1.x86_64
        system: Linux
       version: Ubuntu 24.04.5 noble