py-scrapy: updated to 2.11.2 Scrapy 2.11.2 (2024-05-14) -------------------------- Security bug fixes ~~~~~~~~~~~~~~~~~~ - Redirects to non-HTTP protocols are no longer followed. Please, see the `23j4-mw76-5v7h security advisory`_ for more information. (:issue:`457`) .. _23j4-mw76-5v7h security advisory: https://github.com/scrapy/scrapy/security/advisories/GHSA-23j4-mw76-5v7h - The ``Authorization`` header is now dropped on redirects to a different scheme (``http://`` or ``https://``) or port, even if the domain is the same. Please, see the `4qqq-9vqf-3h3f security advisory`_ for more information. .. _4qqq-9vqf-3h3f security advisory: https://github.com/scrapy/scrapy/security/advisories/GHSA-4qqq-9vqf-3h3f - When using system proxy settings that are different for ``http://`` and ``https://``, redirects to a different URL scheme will now also trigger the corresponding change in proxy settings for the redirected request. Please, see the `jm3v-qxmh-hxwv security advisory`_ for more information. (:issue:`767`) .. _jm3v-qxmh-hxwv security advisory: https://github.com/scrapy/scrapy/security/advisories/GHSA-jm3v-qxmh-hxwv - :attr:`Spider.allowed_domains <scrapy.Spider.allowed_domains>` is now enforced for all requests, and not only requests from spider callbacks. (:issue:`1042`, :issue:`2241`, :issue:`6358`) - :func:`~scrapy.utils.iterators.xmliter_lxml` no longer resolves XML entities. (:issue:`6265`) - defusedxml_ is now used to make :class:`scrapy.http.request.rpc.XmlRpcRequest` more secure. (:issue:`6250`, :issue:`6251`) .. _defusedxml: https://github.com/tiran/defusedxml Bug fixes ~~~~~~~~~ - Restored support for brotlipy_, which had been dropped in Scrapy 2.11.1 in favor of brotli_. (:issue:`6261`) .. _brotli: https://github.com/google/brotli .. note:: brotlipy is deprecated, both in Scrapy and upstream. Use brotli instead if you can. - Make :setting:`METAREFRESH_IGNORE_TAGS` ``["noscript"]`` by default. This prevents :class:`~scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware` from following redirects that would not be followed by web browsers with JavaScript enabled. (:issue:`6342`, :issue:`6347`) - During :ref:`feed export <topics-feed-exports>`, do not close the underlying file from :ref:`built-in post-processing plugins <builtin-plugins>`. (:issue:`5932`, :issue:`6178`, :issue:`6239`) - :class:`LinkExtractor <scrapy.linkextractors.lxmlhtml.LxmlLinkExtractor>` now properly applies the ``unique`` and ``canonicalize`` parameters. (:issue:`3273`, :issue:`6221`) - Do not initialize the scheduler disk queue if :setting:`JOBDIR` is an empty string. (:issue:`6121`, :issue:`6124`) - Fix :attr:`Spider.logger <scrapy.Spider.logger>` not logging custom extra information. (:issue:`6323`, :issue:`6324`) - ``robots.txt`` files with a non-UTF-8 encoding no longer prevent parsing the UTF-8-compatible (e.g. ASCII) parts of the document. (:issue:`6292`, :issue:`6298`) - :meth:`scrapy.http.cookies.WrappedRequest.get_header` no longer raises an exception if ``default`` is ``None``. (:issue:`6308`, :issue:`6310`) - :class:`~scrapy.selector.Selector` now uses :func:`scrapy.utils.response.get_base_url` to determine the base URL of a given :class:`~scrapy.http.Response`. (:issue:`6265`) - The :meth:`media_to_download` method of :ref:`media pipelines <topics-media-pipeline>` now logs exceptions before stripping them. (:issue:`5067`, :issue:`5068`) - When passing a callback to the :command:`parse` command, build the callback callable with the right signature. (:issue:`6182`) Documentation ~~~~~~~~~~~~~ - Add a FAQ entry about :ref:`creating blank requests <faq-blank-request>`. (:issue:`6203`, :issue:`6208`) - Document that :attr:`scrapy.selector.Selector.type` can be ``"json"``. (:issue:`6328`, :issue:`6334`) Quality assurance ~~~~~~~~~~~~~~~~~ - Make builds reproducible. (:issue:`5019`, :issue:`6322`) - Packaging and test fixes.diff -r1.21 -r1.22 pkgsrc/www/py-scrapy/Makefile
(adam)
@@ -1,30 +1,34 @@ | @@ -1,30 +1,34 @@ | |||
1 | # $NetBSD: Makefile,v 1.21 2024/02/16 19:02:45 adam Exp $ | 1 | # $NetBSD: Makefile,v 1.22 2024/05/14 19:15:59 adam Exp $ | |
2 | 2 | |||
3 | DISTNAME= Scrapy-2.11.1 | 3 | DISTNAME= scrapy-2.11.2 | |
4 | PKGNAME= ${PYPKGPREFIX}-${DISTNAME:tl} | 4 | PKGNAME= ${PYPKGPREFIX}-${DISTNAME} | |
5 | CATEGORIES= www python | 5 | CATEGORIES= www python | |
6 | MASTER_SITES= ${MASTER_SITE_PYPI:=S/Scrapy/} | 6 | MASTER_SITES= ${MASTER_SITE_PYPI:=S/Scrapy/} | |
7 | 7 | |||
8 | MAINTAINER= pkgsrc-users@NetBSD.org | 8 | MAINTAINER= pkgsrc-users@NetBSD.org | |
9 | HOMEPAGE= https://scrapy.org/ | 9 | HOMEPAGE= https://scrapy.org/ | |
10 | COMMENT= High-level Web Crawling and Web Scraping framework | 10 | COMMENT= High-level Web Crawling and Web Scraping framework | |
11 | LICENSE= modified-bsd | 11 | LICENSE= modified-bsd | |
12 | 12 | |||
13 | WHEEL_NAME= Scrapy-${PKGVERSION_NOREV} | |||
14 | ||||
13 | TOOL_DEPENDS+= ${PYPKGPREFIX}-wheel-[0-9]*:../../devel/py-wheel | 15 | TOOL_DEPENDS+= ${PYPKGPREFIX}-wheel-[0-9]*:../../devel/py-wheel | |
14 | DEPENDS+= ${PYPKGPREFIX}-cssselect>=0.9.1:../../textproc/py-cssselect | 16 | DEPENDS+= ${PYPKGPREFIX}-cssselect>=0.9.1:../../textproc/py-cssselect | |
17 | DEPENDS+= ${PYPKGPREFIX}-defusedxml>=0.7.1:../../textproc/py-defusedxml | |||
15 | DEPENDS+= ${PYPKGPREFIX}-itemadapter>=0.1.0:../../textproc/py-itemadapter | 18 | DEPENDS+= ${PYPKGPREFIX}-itemadapter>=0.1.0:../../textproc/py-itemadapter | |
16 | DEPENDS+= ${PYPKGPREFIX}-itemloaders>=1.0.1:../../textproc/py-itemloaders | 19 | DEPENDS+= ${PYPKGPREFIX}-itemloaders>=1.0.1:../../textproc/py-itemloaders | |
17 | DEPENDS+= ${PYPKGPREFIX}-lxml>=4.3.0:../../textproc/py-lxml | 20 | DEPENDS+= ${PYPKGPREFIX}-lxml>=4.4.1:../../textproc/py-lxml | |
21 | DEPENDS+= ${PYPKGPREFIX}-cssselect>=0.9.1:../../textproc/py-cssselect | |||
18 | DEPENDS+= ${PYPKGPREFIX}-packaging-[0-9]*:../../devel/py-packaging | 22 | DEPENDS+= ${PYPKGPREFIX}-packaging-[0-9]*:../../devel/py-packaging | |
19 | DEPENDS+= ${PYPKGPREFIX}-parsel>=1.5.0:../../www/py-parsel | 23 | DEPENDS+= ${PYPKGPREFIX}-parsel>=1.5.0:../../www/py-parsel | |
20 | DEPENDS+= ${PYPKGPREFIX}-protego>=0.1.15:../../www/py-protego | 24 | DEPENDS+= ${PYPKGPREFIX}-protego>=0.1.15:../../www/py-protego | |
21 | DEPENDS+= ${PYPKGPREFIX}-pydispatcher>=2.0.5:../../devel/py-pydispatcher | 25 | DEPENDS+= ${PYPKGPREFIX}-pydispatcher>=2.0.5:../../devel/py-pydispatcher | |
22 | DEPENDS+= ${PYPKGPREFIX}-queuelib>=1.4.2:../../devel/py-queuelib | 26 | DEPENDS+= ${PYPKGPREFIX}-queuelib>=1.4.2:../../devel/py-queuelib | |
23 | DEPENDS+= ${PYPKGPREFIX}-service_identity>=18.1.0:../../security/py-service_identity | 27 | DEPENDS+= ${PYPKGPREFIX}-service_identity>=18.1.0:../../security/py-service_identity | |
24 | DEPENDS+= ${PYPKGPREFIX}-setuptools-[0-9]*:../../devel/py-setuptools | 28 | DEPENDS+= ${PYPKGPREFIX}-setuptools-[0-9]*:../../devel/py-setuptools | |
25 | DEPENDS+= ${PYPKGPREFIX}-tldextract-[0-9]*:../../net/py-tldextract | 29 | DEPENDS+= ${PYPKGPREFIX}-tldextract-[0-9]*:../../net/py-tldextract | |
26 | DEPENDS+= ${PYPKGPREFIX}-twisted>=18.9.0:../../net/py-twisted | 30 | DEPENDS+= ${PYPKGPREFIX}-twisted>=18.9.0:../../net/py-twisted | |
27 | DEPENDS+= ${PYPKGPREFIX}-w3lib>=1.17.0:../../www/py-w3lib | 31 | DEPENDS+= ${PYPKGPREFIX}-w3lib>=1.17.0:../../www/py-w3lib | |
28 | DEPENDS+= ${PYPKGPREFIX}-zope.interface>=5.1.0:../../devel/py-zope.interface | 32 | DEPENDS+= ${PYPKGPREFIX}-zope.interface>=5.1.0:../../devel/py-zope.interface | |
29 | 33 | |||
30 | USE_LANGUAGES= # none | 34 | USE_LANGUAGES= # none |
@@ -1,14 +1,14 @@ | @@ -1,14 +1,14 @@ | |||
1 | @comment $NetBSD: PLIST,v 1.11 2024/02/16 19:02:45 adam Exp $ | 1 | @comment $NetBSD: PLIST,v 1.12 2024/05/14 19:15:59 adam Exp $ | |
2 | bin/scrapy-${PYVERSSUFFIX} | 2 | bin/scrapy-${PYVERSSUFFIX} | |
3 | ${PYSITELIB}/${WHEEL_INFODIR}/AUTHORS | 3 | ${PYSITELIB}/${WHEEL_INFODIR}/AUTHORS | |
4 | ${PYSITELIB}/${WHEEL_INFODIR}/LICENSE | 4 | ${PYSITELIB}/${WHEEL_INFODIR}/LICENSE | |
5 | ${PYSITELIB}/${WHEEL_INFODIR}/METADATA | 5 | ${PYSITELIB}/${WHEEL_INFODIR}/METADATA | |
6 | ${PYSITELIB}/${WHEEL_INFODIR}/RECORD | 6 | ${PYSITELIB}/${WHEEL_INFODIR}/RECORD | |
7 | ${PYSITELIB}/${WHEEL_INFODIR}/WHEEL | 7 | ${PYSITELIB}/${WHEEL_INFODIR}/WHEEL | |
8 | ${PYSITELIB}/${WHEEL_INFODIR}/entry_points.txt | 8 | ${PYSITELIB}/${WHEEL_INFODIR}/entry_points.txt | |
9 | ${PYSITELIB}/${WHEEL_INFODIR}/top_level.txt | 9 | ${PYSITELIB}/${WHEEL_INFODIR}/top_level.txt | |
10 | ${PYSITELIB}/scrapy/VERSION | 10 | ${PYSITELIB}/scrapy/VERSION | |
11 | ${PYSITELIB}/scrapy/__init__.py | 11 | ${PYSITELIB}/scrapy/__init__.py | |
12 | ${PYSITELIB}/scrapy/__init__.pyc | 12 | ${PYSITELIB}/scrapy/__init__.pyc | |
13 | ${PYSITELIB}/scrapy/__init__.pyo | 13 | ${PYSITELIB}/scrapy/__init__.pyo | |
14 | ${PYSITELIB}/scrapy/__main__.py | 14 | ${PYSITELIB}/scrapy/__main__.py | |
@@ -160,26 +160,29 @@ ${PYSITELIB}/scrapy/downloadermiddleware | @@ -160,26 +160,29 @@ ${PYSITELIB}/scrapy/downloadermiddleware | |||
160 | ${PYSITELIB}/scrapy/downloadermiddlewares/downloadtimeout.pyo | 160 | ${PYSITELIB}/scrapy/downloadermiddlewares/downloadtimeout.pyo | |
161 | ${PYSITELIB}/scrapy/downloadermiddlewares/httpauth.py | 161 | ${PYSITELIB}/scrapy/downloadermiddlewares/httpauth.py | |
162 | ${PYSITELIB}/scrapy/downloadermiddlewares/httpauth.pyc | 162 | ${PYSITELIB}/scrapy/downloadermiddlewares/httpauth.pyc | |
163 | ${PYSITELIB}/scrapy/downloadermiddlewares/httpauth.pyo | 163 | ${PYSITELIB}/scrapy/downloadermiddlewares/httpauth.pyo | |
164 | ${PYSITELIB}/scrapy/downloadermiddlewares/httpcache.py | 164 | ${PYSITELIB}/scrapy/downloadermiddlewares/httpcache.py | |
165 | ${PYSITELIB}/scrapy/downloadermiddlewares/httpcache.pyc | 165 | ${PYSITELIB}/scrapy/downloadermiddlewares/httpcache.pyc | |
166 | ${PYSITELIB}/scrapy/downloadermiddlewares/httpcache.pyo | 166 | ${PYSITELIB}/scrapy/downloadermiddlewares/httpcache.pyo | |
167 | ${PYSITELIB}/scrapy/downloadermiddlewares/httpcompression.py | 167 | ${PYSITELIB}/scrapy/downloadermiddlewares/httpcompression.py | |
168 | ${PYSITELIB}/scrapy/downloadermiddlewares/httpcompression.pyc | 168 | ${PYSITELIB}/scrapy/downloadermiddlewares/httpcompression.pyc | |
169 | ${PYSITELIB}/scrapy/downloadermiddlewares/httpcompression.pyo | 169 | ${PYSITELIB}/scrapy/downloadermiddlewares/httpcompression.pyo | |
170 | ${PYSITELIB}/scrapy/downloadermiddlewares/httpproxy.py | 170 | ${PYSITELIB}/scrapy/downloadermiddlewares/httpproxy.py | |
171 | ${PYSITELIB}/scrapy/downloadermiddlewares/httpproxy.pyc | 171 | ${PYSITELIB}/scrapy/downloadermiddlewares/httpproxy.pyc | |
172 | ${PYSITELIB}/scrapy/downloadermiddlewares/httpproxy.pyo | 172 | ${PYSITELIB}/scrapy/downloadermiddlewares/httpproxy.pyo | |
173 | ${PYSITELIB}/scrapy/downloadermiddlewares/offsite.py | |||
174 | ${PYSITELIB}/scrapy/downloadermiddlewares/offsite.pyc | |||
175 | ${PYSITELIB}/scrapy/downloadermiddlewares/offsite.pyo | |||
173 | ${PYSITELIB}/scrapy/downloadermiddlewares/redirect.py | 176 | ${PYSITELIB}/scrapy/downloadermiddlewares/redirect.py | |
174 | ${PYSITELIB}/scrapy/downloadermiddlewares/redirect.pyc | 177 | ${PYSITELIB}/scrapy/downloadermiddlewares/redirect.pyc | |
175 | ${PYSITELIB}/scrapy/downloadermiddlewares/redirect.pyo | 178 | ${PYSITELIB}/scrapy/downloadermiddlewares/redirect.pyo | |
176 | ${PYSITELIB}/scrapy/downloadermiddlewares/retry.py | 179 | ${PYSITELIB}/scrapy/downloadermiddlewares/retry.py | |
177 | ${PYSITELIB}/scrapy/downloadermiddlewares/retry.pyc | 180 | ${PYSITELIB}/scrapy/downloadermiddlewares/retry.pyc | |
178 | ${PYSITELIB}/scrapy/downloadermiddlewares/retry.pyo | 181 | ${PYSITELIB}/scrapy/downloadermiddlewares/retry.pyo | |
179 | ${PYSITELIB}/scrapy/downloadermiddlewares/robotstxt.py | 182 | ${PYSITELIB}/scrapy/downloadermiddlewares/robotstxt.py | |
180 | ${PYSITELIB}/scrapy/downloadermiddlewares/robotstxt.pyc | 183 | ${PYSITELIB}/scrapy/downloadermiddlewares/robotstxt.pyc | |
181 | ${PYSITELIB}/scrapy/downloadermiddlewares/robotstxt.pyo | 184 | ${PYSITELIB}/scrapy/downloadermiddlewares/robotstxt.pyo | |
182 | ${PYSITELIB}/scrapy/downloadermiddlewares/stats.py | 185 | ${PYSITELIB}/scrapy/downloadermiddlewares/stats.py | |
183 | ${PYSITELIB}/scrapy/downloadermiddlewares/stats.pyc | 186 | ${PYSITELIB}/scrapy/downloadermiddlewares/stats.pyc | |
184 | ${PYSITELIB}/scrapy/downloadermiddlewares/stats.pyo | 187 | ${PYSITELIB}/scrapy/downloadermiddlewares/stats.pyo | |
185 | ${PYSITELIB}/scrapy/downloadermiddlewares/useragent.py | 188 | ${PYSITELIB}/scrapy/downloadermiddlewares/useragent.py |
@@ -1,5 +1,5 @@ | @@ -1,5 +1,5 @@ | |||
1 | $NetBSD: distinfo,v 1.16 2024/02/16 19:02:45 adam Exp $ | 1 | $NetBSD: distinfo,v 1.17 2024/05/14 19:15:59 adam Exp $ | |
2 | 2 | |||
3 | BLAKE2s (Scrapy-2.11.1.tar.gz) = ec247564bb7f25be4bca8e966e593c7c6c222b9644cf05686d6d9a0a4a436b07 | 3 | BLAKE2s (scrapy-2.11.2.tar.gz) = 43f17692a3ceb60435ed9e6a8f1921ac029b1e2f36689c992978c9cee3d47e26 | |
4 | SHA512 (Scrapy-2.11.1.tar.gz) = c33bf8fe45c96865483398920e823bd169d7d7e5d67dcfd5e57e4546f1016cfdcb404ebcbf67a6710a4597d5970f55481226fee25c27291dfaedfc00322327d9 | 4 | SHA512 (scrapy-2.11.2.tar.gz) = 488e3d1aff4f76ff795da7dbcaed6459d39ad45dc5ba5482f628b2d52dafbe740ee2f0c2caa4ffb6a4fe70b4268114877be833b6066a61f8d1fab8a95a74badf | |
5 | Size (Scrapy-2.11.1.tar.gz) = 1176726 bytes | 5 | Size (scrapy-2.11.2.tar.gz) = 1187710 bytes |