mirror of
https://github.com/netzbegruenung/green-spider.git
synced 2024-05-05 10:33:39 +02:00
Marian Steinbach
618e29d763
* CLI: remove 'jobs' command, add 'manager' * Add job definition * Move jobs to manage folder * Rename jobs to manager * Add rq and redis dependencies * Add docker-compose YAML * Downgrade to alpine 3.8 * Adjust paths in Dockerfile, remove entrypoint * Rename 'make spiderjobs' to 'make jobs' * Fix docker exectution * Adapt 'make jobs' * Fix metadata scheme * Add docker dependency * Rendomize queue (a bit) * Use latest image, remove debug output * Make docker-compose file downwards-compatible * Use latest instead of dev image tag * Update docker-compose.yaml * Adapt job start script * Fix redis connection in manager * Add support for increasing timeout via environment variable * Adapt load_in_browser to cookies table schema change * Fix execution * Mitigate yaml warning * Bump some dependency versions * Report resource usage stats for each job * checks/load_in_browser: Return DOM size, prevent multiple page loads * Update .dockerignore * Code update * Script update * Update README.md * WIP * WIP commit * Update Dockerfile to alpine:edge and chromium v90 * Update TestCertificateChecker * Set defaults for __init__ function * Detect sunflower theme * Update unit test for new datetime (zero-basing) * Set logging prefs from Chromium in a new way * Move datastore client instantiation As it is not needed for all commands * Change green-directory repository URL * Add git settings for cloning green-directory * Pin alpine version 3.14, fix py3-cryptography * Use plain docker build progress output * Add volumes to 'make test' docker run command * Fix bug * Update example command in README * Update dependencies * Add creation of Kubernetes jobs
43 lines
741 B
Plaintext
43 lines
741 B
Plaintext
beautifulsoup4==4.9.3
|
|
cachetools==4.2.2
|
|
certifi==2020.12.5
|
|
cffi==1.14.5
|
|
chardet==3.0.4
|
|
click==7.1.2
|
|
cssselect==1.1.0
|
|
dnspython==1.16.0
|
|
docker==4.4.1
|
|
feedparser==6.0.8
|
|
gitdb==4.0.7
|
|
GitPython==3.1.14
|
|
google-api-core==1.26.3
|
|
google-auth==1.30.0
|
|
google-cloud-core==1.6.0
|
|
google-cloud-datastore==1.15.3
|
|
google-cloud-storage==1.38.0
|
|
googleapis-common-protos==1.53.0
|
|
html-similarity==0.3.3
|
|
httpretty==0.9.7
|
|
idna==2.10
|
|
parsel==1.6.0
|
|
protobuf==3.15.8
|
|
pyasn1==0.4.8
|
|
pyasn1-modules==0.2.8
|
|
pycparser==2.20
|
|
pyOpenSSL==20.0.1
|
|
pytz==2021.1
|
|
PyYAML==5.4.1
|
|
redis==3.5.3
|
|
requests==2.25.1
|
|
responses==0.13.3
|
|
rq==1.8.0
|
|
rsa==4.7.2
|
|
selenium==3.141.0
|
|
smmap==3.0.4
|
|
smmap2==2.0.5
|
|
soupsieve==2.2.1
|
|
tenacity==5.1.5
|
|
urllib3==1.26.4
|
|
w3lib==1.22.0
|
|
websocket-client==0.59.0
|