我在带有RedisToGo Nano插件的Heroku上使用celerybeat
有一个Web dyno和一个worker dyno
celerybeat工作者设置为每分钟执行一次任务。
问题是:每当我部署新的提交时,dynos重新启动,并且出现此错误
2014-02-27T13:19:31.552352+00:00 app[worker.1]: Traceback (most recent call last): 2014-02-27T13:19:31.552352+00:00 app[worker.1]: File "/app/.heroku/python/lib/python2.7/site-packages/celery/worker/consumer.py", line 389, in start 2014-02-27T13:19:31.552352+00:00 app[worker.1]: self.reset_connection() 2014-02-27T13:19:31.552352+00:00 app[worker.1]: File "/app/.heroku/python/lib/python2.7/site-packages/celery/worker/consumer.py", line 727, in reset_connection 2014-02-27T13:19:31.552352+00:00 app[worker.1]: self.connection = self._open_connection() 2014-02-27T13:19:31.552352+00:00 app[worker.1]: File "/app/.heroku/python/lib/python2.7/site-packages/celery/worker/consumer.py", line 792, in _open_connection 2014-02-27T13:19:31.552352+00:00 app[worker.1]: callback=self.maybe_shutdown) 2014-02-27T13:18:23.864287+00:00 app[worker.1]: self.on_connect() 2014-02-27T13:18:23.864287+00:00 app[worker.1]: File "/app/.heroku/python/lib/python2.7/site-packages/redis/connection.py", line 263, in on_connect 2014-02-27T13:18:23.864287+00:00 app[worker.1]: if nativestr(self.read_response()) != 'OK': 2014-02-27T13:18:23.864287+00:00 app[worker.1]: File "/app/.heroku/python/lib/python2.7/site-packages/redis/connection.py", line 314, in read_response 2014-02-27T13:18:23.864287+00:00 app[worker.1]: raise response 2014-02-27T13:18:23.864287+00:00 app[worker.1]: ResponseError: max number of clients reached 2014-02-27T13:19:31.552352+00:00 app[worker.1]: File "/app/.heroku/python/lib/python2.7/site-packages/kombu/connection.py", line 272, in ensure_connection 2014-02-27T13:19:31.552352+00:00 app[worker.1]: interval_start, interval_step, interval_max, callback) 2014-02-27T13:19:31.552591+00:00 app[worker.1]: File "/app/.heroku/python/lib/python2.7/site-packages/kombu/utils/__init__.py", line 218, in retry_over_time 2014-02-27T13:19:31.552591+00:00 app[worker.1]: return fun(*args, **kwargs) 2014-02-27T13:19:31.552591+00:00 app[worker.1]: File "/app/.heroku/python/lib/python2.7/site-packages/kombu/connection.py", line 162, in connect 2014-02-27T13:19:31.552591+00:00 app[worker.1]: return self.connection 2014-02-27T13:19:31.552591+00:00 app[worker.1]: File "/app/.heroku/python/lib/python2.7/site-packages/kombu/connection.py", line 617, in connection 2014-02-27T13:18:23.870811+00:00 app[worker.1]: [2014-02-27 13:18:23,870: ERROR/MainProcess] consumer: Connection to broker lost. Trying to re-establish the connection...
那些日志不断地继续下去。直到我停止两个测功机并重新启动它们。
这已经成为一个问题,因为它几乎在每次部署新提交时都会发生。
有什么想法为什么会发生以及如何解决呢?
纳米redistogo计划将同时进行的redis连接限制为 10 。
使用的redis连接数将根据您的前端和celery worker设置而有所不同。听起来您正在为生产堆栈使用 > = 5 redis连接。
部署新代码时,Heroku会启动一个全新的堆栈。这意味着您在部署时正在使用 > = 10 redis连接。
有两种方法可以解决此问题:
这是资源枯竭的简单问题。我只需要支付更大的RedisToGo计划。