Hovedproblemet ved dynamisk at sende job til en kø er, at det kan være svært at afgøre, hvornår jobs skal sendes. Hvis køen er for stor, kan opgaverne først sendes efter fristens udløb. Hvis køen er for lille, bliver opgaverne muligvis slet ikke sendt.
I have a Python program that needs to process a large number of files. Each file can be processed independently, so I would like to take advantage of parallel processing by sending each file to a job queue.
I would like the program to be able to dynamically add new jobs to the queue as they become available, without having to wait for all the current jobs in the queue to finish first. Is this possible? If so, how can I do it?
A:
You could use <code>multiprocessing</code>'s <code>Pool</code> class:
<blockquote>
<p><strong><a href="https://docs.python.org/3/library/multiprocessing.html#the-process-pool" rel="nofollow noreferrer">17.5.1 The Process Pool</a></strong></p>
<p>[...]</p>
<pre><code><code>class multiprocessing.pool.Pool(processes=None, initializer=None, initargs=(), maxtasksperchild=None) # noqa E501
Returns a process pool object.</code></code></pre>
<p>[...]</p>
<pre><code><code>apply(func[, args[, kwds]])
Equivalent of `func(*args, **kwds)`. It blocks until the result is ready.</code></code></pre>
<p>[...]</p>
<pre><code><code>apply_async(func[, args[, kwds[, callback]]])
A variant of the `apply()` method which returns an `AsyncResult` object.</ code ></code></pre>
<p>[...] </p>
<pre><code><code>map(func, iterable[, chunksize])
A parallel equivalent of `map()`. It blocks until the result is ready.</ code ></code></pre>
<p>[...] </p>
<pre><co de >map_async(func , iterable [ , chunksize [ , callback ] ] )
A variant of the map() method which returns an AsyncResult object.</ co de > </ pre > < p > [ ... ] < / p > < pre >< code > ; imap ( func , iterable [ , chunksize ] )
A lazy version of map(). The chunksize argument is the same as in get(). </ code & gt ; < / pre > < p > [ ... ] < / p > < pre >< code & gt ; imap_unordered ( func , iterable [ , chunksize ] )
A lazy version of map(). The chunksize argument is the same as in get(). </ code & gt ; < / pre > </blockquote>. . . . . . . . . .. .. .. .. .. .. .. ... ....... ....... ................. ................. ...................... ...................... ...................... ...................... .......... .......... .......... .......... .......... .......... ............ ............ .............. .............. .............. .............. .............. ...... ...... ...... ...... ...... ...... ....... ....... ....... ....... ....... ......... ............ ............ ............ ............ ............. .................. .................. .................. .................. ....................... ................................. ............................................... ………… ………… ………… ………… ………………. ………………. ………………. ……………….……………………………………………………………………………………………………………………………………………………………………………..
køer
En kø er en datastruktur, der gør det muligt at tilføje elementer foran og fjerne fra bagsiden. Køer bruges ofte i programmeringssprog for at tillade programmer at arbejde med indsamlinger af data.
Hvornår skal man bruge en kø
En kø er en datastruktur, der gør det muligt at tilføje elementer foran og fjerne fra bagsiden. Køer bruges ofte, når der er behov for at vente på et bestemt antal elementer, der skal behandles, før man gør noget andet.
