python3 多进程 multiprocess 任务均衡 task-balance 队列 queue V1

chrisplus

墨衍会员 · AI 创作全网分发码龄19年

关注

原创最新推荐文章于 2024-08-10 08:45:14 发布 616 阅读 AI 写同款· GEO 优化 ›

AI权益加码！Claude Code、Cursor等20+工具免费用！购周边限时加赠Coding Plan Lite，畅享主流AI工具！学习进阶更高效！阅读详情

python3多进程偶尔会有无法停下的bug，因为多进程竞争读取队列时会有Empty exception

所以最好的办法是人为设置进程控制信息，避免因为竞争导致读取队列为空的问题。

[Python,SQL,ETL]交流群 164142295

以下代码完美处理了这个问题

import time
from multiprocessing import cpu_count, Queue, Process
from queue import Empty
import traceback

def add(q, index, Q,waste):
r,b = 0,0
while 1:
try:
args = q.get(block=False)
if args[0] == 1:
Q.put(r)
break
else:
r += 1
except Empty:
b+=1
time.sleep(0.001)
continue
except:
print('Unknown Error:')
traceback.print_exc()
exit(1)
#time.sleep(0.00001)
waste.put(b)

if __name__ == '__main__':
multipy = 2
past, cpus = time.time(), cpu_count() * multipy
Q,waste = Queue(),Queue()
for j in range(1):
T, p = Queue(), []
# Data
for i in range(500000):
T.put((0, i, i + 1))
# Control Signal
for i in range(cpus):
T.put((1, i, i + 1))
for i in range(cpus):
p.append(Process(target=add, args=(T, i, Q,waste)))
for i in p:
i.start()
for i in p:
i.join()
print("JoinFinished")
tt = []
while not Q.empty():
tt.append(Q.get())
for i, j in enumerate(tt):
print(f'Process_{i} Return {j}')
print(f'Total Return {sum(tt)}')
w=0
while not waste.empty():
w+=waste.get()
print(f'ProgramRunToTheEnd Costs {round(time.time()*1000-past*1000,2)}MS,WasteOfLoop {w} Cycles')