在Python中“不能启动新的线程错误”

时间:2022-06-01 21:24:47

I am running a script which extracts the information from debian packages and saves it in a database.

我正在运行一个脚本,它从debian包中提取信息并将其保存到数据库中。

After extracting information from about 100 packages an error occurs. The error is "can't start a new thread" Why am I facing this error? What can be the possible solution to it?

在从大约100个包中提取信息之后,会发生错误。错误是“无法启动新线程”为什么我要面对这个错误?有什么可能的解决办法呢?

This is the code used to save the data :

这是用来保存数据的代码:

for i in list_pack:

      if not i in oblist:        
        #Creating Packages
            slu=slugify(i)
            ob=Gbobject()
            ob.title=i
            ob.slug=slu
            ob.content=''
            ob.tags=tagname
        #with reversion.create_revision():
            ob.save()
            gb=Gbobject.objects.get(title=i)
            gb.objecttypes.add(Objecttype.objects.get(title='Packages'))
            gb.sites.add(Site.objects.get_current())
        #with reversion.create_revision():
            gb.save()
            gd=Gbobject.objects.get(title=i)
            print 'The Object created was',gd


            #Decsription
            try:
                atv=package_description_dict[i]
                atvalue=unicode(atv,'utf-8')
            except UnicodeDecodeError:
                pass
            try:
                lo=Gbobject.objects.get(title=i)
                loid=NID.objects.get(id=lo.id)
            except Gbobject.DoesNotExist:
                pass
            if atvalue<>'':
                slu=slugify(atvalue)
                at=Attribute()
                at.title=atvalue
                at.slug=slu
                at.svalue=atvalue
                at.subject=loid
                att=Attributetype.objects.get(title='Description')
                at.attributetype=att
                #with reversion.create_revision():
                at.save()                    
                print 'yeloow13'

Just like Description, there are around !2 more properties of the package which are saved in a similar way.

就像描述一样,包中还有2个属性以类似的方式保存。

This is the full traceback that I get when the error occurs :-

这是当错误发生时我得到的全部回溯:-

    error                                     Traceback (most recent call last)

/home/radhika/Desktop/dev_75/gnowsys-studio/demo/<ipython console> in <module>()

/usr/local/lib/python2.6/dist-packages/django_gstudio-0.3.dev-py2.6.egg/gstudio/Harvest/debdata.py in <module>()
   1086 # create_attribute_type()

   1087 # create_relation_type()

-> 1088 create_objects()
   1089 #create_sec_objects()

   1090 #create_relations()


/usr/local/lib/python2.6/dist-packages/django_gstudio-0.3.dev-py2.6.egg/gstudio/Harvest/debdata.py in create_objects()
    403             ob.sites.add(Site.objects.get_current())
    404             #with reversion.create_revision():

--> 405             ob.save()
    406             #time.sleep(10)

    407             #gd=Gbobject.objects.get(title=i)


/usr/local/lib/python2.6/dist-packages/django_reversion-1.6.0-py2.6.egg/reversion/revisions.pyc in do_revision_context(*args, **kwargs)
    298             try:
    299                 try:
--> 300                     return func(*args, **kwargs)
    301                 except:
    302                     exception = True

/usr/local/lib/python2.6/dist-packages/django_gstudio-0.3.dev-py2.6.egg/objectapp/models.pyc in save(self, *args, **kwargs)
    658     @reversion.create_revision()
    659     def save(self, *args, **kwargs):
--> 660         super(Gbobject, self).save(*args, **kwargs) # Call the "real" save() method.
    661 
    662 

/usr/local/lib/python2.6/dist-packages/django_reversion-1.6.0-py2.6.egg/reversion/revisions.pyc in do_revision_context(*args, **kwargs)
    298             try:
    299                 try:
--> 300                     return func(*args, **kwargs)
    301                 except:
    302                     exception = True

/usr/local/lib/python2.6/dist-packages/django_gstudio-0.3.dev-py2.6.egg/gstudio/models.pyc in save(self, *args, **kwargs)
    327     @reversion.create_revision()
    328     def save(self, *args, **kwargs):
--> 329         super(Node, self).save(*args, **kwargs) # Call the "real" save() method.
    330 
    331 

/usr/local/lib/python2.6/dist-packages/django/db/models/base.pyc in save(self, force_insert, force_update, using)
    458         if force_insert and force_update:
    459             raise ValueError("Cannot force both insert and updating in model saving.")
--> 460         self.save_base(using=using, force_insert=force_insert, force_update=force_update)
    461 
    462     save.alters_data = True

/usr/local/lib/python2.6/dist-packages/django/db/models/base.pyc in save_base(self, raw, cls, origin, force_insert, force_update, using)
    568         if origin and not meta.auto_created:
    569             signals.post_save.send(sender=origin, instance=self,
--> 570                 created=(not record_exists), raw=raw, using=using)
    571 
    572 

/usr/local/lib/python2.6/dist-packages/django/dispatch/dispatcher.pyc in send(self, sender, **named)
    170 
    171         for receiver in self._live_receivers(_make_id(sender)):
--> 172             response = receiver(signal=self, sender=sender, **named)
    173             responses.append((receiver, response))
    174         return responses

/usr/local/lib/python2.6/dist-packages/django_gstudio-0.3.dev-py2.6.egg/objectapp/signals.pyc in wrapper(*args, **kwargs)
     65             if inspect.getmodulename(fr[1]) == 'loaddata':
     66                 return
---> 67         signal_handler(*args, **kwargs)
     68 
     69     return wrapper

/usr/local/lib/python2.6/dist-packages/django_gstudio-0.3.dev-py2.6.egg/objectapp/signals.pyc in ping_external_urls_handler(sender, **kwargs)
     90         from objectapp.ping import ExternalUrlsPinger
     91 
---> 92         ExternalUrlsPinger(gbobject)
     93 
     94 

/usr/local/lib/python2.6/dist-packages/django_gstudio-0.3.dev-py2.6.egg/objectapp/ping.pyc in __init__(self, gbobject, timeout, start_now)
    153         threading.Thread.__init__(self)
    154         if start_now:
--> 155             self.start()
    156 
    157     def run(self):

/usr/lib/python2.6/threading.pyc in start(self)
    472         _active_limbo_lock.release()
    473         try:
--> 474             _start_new_thread(self.__bootstrap, ())
    475         except Exception:
    476             with _active_limbo_lock:

error: can't start new thread

I am not writing any code to handle the threads.

我没有编写任何代码来处理这些线程。

2 个解决方案

#1


13  

Sorry to have an incomplete solution here, I don't have the rating to post in the comment section.

抱歉,这里有一个不完整的解决方案,我没有在评论部分发布的评级。

One thing to check is the total number of threads you have running. I have some code that checks the count of cores (with sys) and then launches threads and checks core loading to test how the OS handles thread distribution and I've learned that windows 7 (for example) seems to throw an error beyond 32 threads on an 8 (logical) core CPU. [That's on Python 2.7, 32 bit within Win 7 64-bit, etc, YMMMV]. On other machines I can get past 1,000 Threads.

要检查的一件事是您正在运行的线程的总数。我有一些代码检查内核的计数(使用sys),然后启动线程并检查内核加载,以测试操作系统如何处理线程分布,我还了解到windows 7(例如)似乎在8(逻辑)核心CPU上抛出了32个线程之外的错误。[在Python 2.7上,win7 64位内32位,等等,YMMMV]。在其他机器上,我可以超过1000个线程。

So I guess the short version is: how many threads do you already have going when you get that error? You can check with

所以我猜简短的版本是:当你得到那个错误时,你已经有多少线程了?你可以去看看

threading.active_count()

Beyond that you don't really give the threading part of the code here so I'd direct you to this excellent Python Central page.

除此之外,您并没有给出代码的线程部分,因此我将引导您到这个优秀的Python中心页面。

You may also benefit from this previous Stack overflow discussion on large thread task counts and how to approach them.

您还可以从前面关于大型线程任务计数和如何处理它们的堆栈溢出讨论中获益。

Again, my apologies that this is more of a direction to look than a solution, I think that more information is likely needed to help us understand what you're running into.

再次,我很抱歉,这更像是一个方向,而不是一个解决方案,我认为需要更多的信息来帮助我们理解您遇到的问题。

#2


0  

You're using a 32bit system and running out of virtual memory. One of your libraries is likely spawning threads and not reclaiming them correctly. As a workaround, try reducing the default thread stack size with threading.stack_size.

您正在使用一个32位的系统并耗尽虚拟内存。您的一个库可能会生成线程,而不会正确地回收它们。作为解决方案,尝试使用thread .stack_size减少默认的线程堆栈大小。

#1


13  

Sorry to have an incomplete solution here, I don't have the rating to post in the comment section.

抱歉,这里有一个不完整的解决方案,我没有在评论部分发布的评级。

One thing to check is the total number of threads you have running. I have some code that checks the count of cores (with sys) and then launches threads and checks core loading to test how the OS handles thread distribution and I've learned that windows 7 (for example) seems to throw an error beyond 32 threads on an 8 (logical) core CPU. [That's on Python 2.7, 32 bit within Win 7 64-bit, etc, YMMMV]. On other machines I can get past 1,000 Threads.

要检查的一件事是您正在运行的线程的总数。我有一些代码检查内核的计数(使用sys),然后启动线程并检查内核加载,以测试操作系统如何处理线程分布,我还了解到windows 7(例如)似乎在8(逻辑)核心CPU上抛出了32个线程之外的错误。[在Python 2.7上,win7 64位内32位,等等,YMMMV]。在其他机器上,我可以超过1000个线程。

So I guess the short version is: how many threads do you already have going when you get that error? You can check with

所以我猜简短的版本是:当你得到那个错误时,你已经有多少线程了?你可以去看看

threading.active_count()

Beyond that you don't really give the threading part of the code here so I'd direct you to this excellent Python Central page.

除此之外,您并没有给出代码的线程部分,因此我将引导您到这个优秀的Python中心页面。

You may also benefit from this previous Stack overflow discussion on large thread task counts and how to approach them.

您还可以从前面关于大型线程任务计数和如何处理它们的堆栈溢出讨论中获益。

Again, my apologies that this is more of a direction to look than a solution, I think that more information is likely needed to help us understand what you're running into.

再次,我很抱歉,这更像是一个方向,而不是一个解决方案,我认为需要更多的信息来帮助我们理解您遇到的问题。

#2


0  

You're using a 32bit system and running out of virtual memory. One of your libraries is likely spawning threads and not reclaiming them correctly. As a workaround, try reducing the default thread stack size with threading.stack_size.

您正在使用一个32位的系统并耗尽虚拟内存。您的一个库可能会生成线程,而不会正确地回收它们。作为解决方案,尝试使用thread .stack_size减少默认的线程堆栈大小。