webmagic online with Winfy

We have hosted the application webmagic in order to run this application in our online workstations with Wine or directly.

Quick description about webmagic:

WebMagic is a scalable crawler framework. It covers the whole lifecycle of crawler, downloading, url management, content extraction and persistent. It can simplify the development of a specific crawler. WebMagic is a simple but scalable crawler framework. You can develop a crawler easily based on it. WebMagic has a simple core with high flexibility, a simple API for html extracting. It also provides annotation with POJO to customize a crawler, and no configuration is needed. Some other features include the fact that it is multi-thread and has distribution support. WebMagic is very easy to integrate. Add dependencies to your pom.xml. WebMagic use slf4j with slf4j-log4j12 implementation. If you customized your slf4j implementation, please exclude slf4j-log4j12. You can write a class implementation of PageProcessor.

Features:

• Simple core with high flexibility
• Simple API for html extracting
• Annotation with POJO to customize a crawler, no configuration
• Multi-thread and Distribution support
• Easy to be integrated
• It covers the whole lifecycle of crawler

Programming Language: Java.
Categories: