刘凡
3282579ec1
first commit
|
2 年 前 | |
|---|---|---|
| .. | ||
| docs | 2 年 前 | |
| example | 2 年 前 | |
| scrapy_kafka | 2 年 前 | |
| .gitignore | 2 年 前 | |
| LICENSE | 2 年 前 | |
| README.rst | 2 年 前 | |
| requirements.txt | 2 年 前 | |
| setup.py | 2 年 前 | |
README.rst
scrapy-kafka
============
Kafka-based components for Scrapy. There are 2 components:
- A custom ``Spider`` that waits for URLs to crawl via a Kafka topic. When there are no more messages to read for the topic, the ``Spider`` just stays idle.
- A custom ``ItemPipeline`` component that stores a JSON-ified ``Item`` back into another Kafka topic.
Please see the `example`_ directory for how to use this.
.. _example: https://github.com/dfdeshom/scrapy-kafka/tree/master/example
Contributors
-------------
Contributors to `scrapy-kafka`, listed alphabetically:
* Matthew Daniel `@mdaniel`_
* Didier Deshommes `@dfdeshom`_
.. _@mdaniel: https://github.com/mdaniel
.. _@dfdeshom: https://github.com/dfdeshom
============
Kafka-based components for Scrapy. There are 2 components:
- A custom ``Spider`` that waits for URLs to crawl via a Kafka topic. When there are no more messages to read for the topic, the ``Spider`` just stays idle.
- A custom ``ItemPipeline`` component that stores a JSON-ified ``Item`` back into another Kafka topic.
Please see the `example`_ directory for how to use this.
.. _example: https://github.com/dfdeshom/scrapy-kafka/tree/master/example
Contributors
-------------
Contributors to `scrapy-kafka`, listed alphabetically:
* Matthew Daniel `@mdaniel`_
* Didier Deshommes `@dfdeshom`_
.. _@mdaniel: https://github.com/mdaniel
.. _@dfdeshom: https://github.com/dfdeshom