Replaces feedparser with a small, namespace-aware lxml parser tailored to the arXiv API (arxiv/_feed.py).
Why: arXiv's Atom feed declares three custom namespaces (arxiv:, opensearch:, dc:), and feedparser's lenient, schema-flattening model silently drops or collapses extension elements \u2014 most painfully the per-author <arxiv:affiliation> elements that <author> can carry. This is the root cause of #62 and is reported upstream in kurtmckee/feedparser#24 and kurtmckee/feedparser#145.
The arXiv API always returns well-formed XML, so feedparser's tolerance buys us nothing here. Parsing directly with lxml is both meaningfully faster and gives us full control over extension elements.
\narxiv._feed module exposing plain dataclasses (FeedEntry, FeedAuthor, Link, Category, FeedHeader, ParsedFeed) instead of feedparser's flattened dict-shape.Result.Author.affiliation: list[str] \u2014 populated from <arxiv:affiliation> children. Closes #62.arxiv/__init__.py translates parsed entries into the existing public Result / Result.Author / Result.Link types; no public field is removed.UnexpectedEmptyPageError.raw_feed is now a ParsedFeed rather than a FeedParserDict. Result._raw is now a FeedEntry.scripts/benchmark_parser.py parses every recorded golden-file fixture under tests/fixtures/ 50 times with each parser and reports the median.
```\nFixtures: 21 files, 506.6 KiB total\nIterations per fixture: 50
\nid_astro-ph_0601001_s0_m100_3150dca9dc76.json 3.5 1.456 0.163 8.9x\nid_hep-ex_0406020v1_s0_m100_84b15d66c0ce.json 1.8 1.075 0.129 8.3x\nid_1605.08386_s0_m100_199195aa31ad.json 1.9 1.148 0.135 8.5x\nq_testing_s0_m100_cfe7536492ca.json 226.1 83.024 10.866 7.6x\nq_testing_s0_m10_bc5a6c384fe9.json 24.4 8.676 1.128 7.7x\n... (full table in benchmark output)
\nTOTAL 506.6 190.913 25.413 7.5x\n```
\nA consistent ~7.5x speedup across the corpus, scaling cleanly with payload size.
\nOf course, these speedups are thumped by the network/API latency in practice.
\ntest_author_affiliations \u2014 uses astro-ph/0601001 (four authors, each with a distinct affiliation: Ohio State, Ohio State, CfA, Warsaw University Observatory). Regression test for #62.test_author_no_affiliation \u2014 confirms affiliation == [] on the common case using 1605.08386.test_from_feed_entry was previously calling _parse_feed with a bare ?search_query=testing URL; switched to a fixed id_list=[1605.08386] so the test is deterministic and shares an existing golden file.astro-ph/0601001; the now-unused q_testing_s0_m10_df962dbeea2e.json fixture was removed.pytest) and live (pytest --live).feedparser is no longer a dependency. Anything reaching into Result._raw.* and expecting FeedParserDict attribute names (updated_parsed, arxiv_comment, etc.) will need to migrate to the new FeedEntry field names (updated as datetime, comment, journal_ref, doi, primary_category, categories).UnexpectedEmptyPageError.raw_feed is now arxiv._feed.ParsedFeed rather than feedparser.FeedParserDict.lxml>=5.0,<7.0.Result.Author.affiliation.README.md example usage has been updated. (No README changes needed \u2014 feedparser isn't referenced there, and the public Result API is unchanged except for the additive Author.affiliation field.)PR drafted by Shelley (Claude) on behalf of @lukasschwab.
", "content_text": "# Description\r\n\r\nReplaces `feedparser` with a small, namespace-aware lxml parser tailored to the arXiv API ([`arxiv/_feed.py`](./arxiv/_feed.py)).\r\n\r\n**Why:** arXiv's Atom feed declares three custom namespaces (`arxiv:`, `opensearch:`, `dc:`), and feedparser's lenient, schema-flattening model silently drops or collapses extension elements \u2014 most painfully the per-author `Replaces feedparser with a small, namespace-aware lxml parser tailored to the arXiv API (arxiv/_feed.py).
Why: arXiv's Atom feed declares three custom namespaces (arxiv:, opensearch:, dc:), and feedparser's lenient, schema-flattening model silently drops or collapses extension elements \u2014 most painfully the per-author <arxiv:affiliation> elements that <author> can carry. This is the root cause of #62 and is reported upstream in kurtmckee/feedparser#24 and kurtmckee/feedparser#145.
The arXiv API always returns well-formed XML, so feedparser's tolerance buys us nothing here. Parsing directly with lxml is both meaningfully faster and gives us full control over extension elements.
\narxiv._feed module exposing plain dataclasses (FeedEntry, FeedAuthor, Link, Category, FeedHeader, ParsedFeed) instead of feedparser's flattened dict-shape.Result.Author.affiliation: list[str] \u2014 populated from <arxiv:affiliation> children. Closes #62.arxiv/__init__.py translates parsed entries into the existing public Result / Result.Author / Result.Link types; no public field is removed.UnexpectedEmptyPageError.raw_feed is now a ParsedFeed rather than a FeedParserDict. Result._raw is now a FeedEntry.scripts/benchmark_parser.py parses every recorded golden-file fixture under tests/fixtures/ 50 times with each parser and reports the median.
```\nFixtures: 21 files, 506.6 KiB total\nIterations per fixture: 50
\nid_astro-ph_0601001_s0_m100_3150dca9dc76.json 3.5 1.456 0.163 8.9x\nid_hep-ex_0406020v1_s0_m100_84b15d66c0ce.json 1.8 1.075 0.129 8.3x\nid_1605.08386_s0_m100_199195aa31ad.json 1.9 1.148 0.135 8.5x\nq_testing_s0_m100_cfe7536492ca.json 226.1 83.024 10.866 7.6x\nq_testing_s0_m10_bc5a6c384fe9.json 24.4 8.676 1.128 7.7x\n... (full table in benchmark output)
\nTOTAL 506.6 190.913 25.413 7.5x\n```
\nA consistent ~7.5x speedup across the corpus, scaling cleanly with payload size.
\nOf course, these speedups are thumped by the network/API latency in practice.
\ntest_author_affiliations \u2014 uses astro-ph/0601001 (four authors, each with a distinct affiliation: Ohio State, Ohio State, CfA, Warsaw University Observatory). Regression test for #62.test_author_no_affiliation \u2014 confirms affiliation == [] on the common case using 1605.08386.test_from_feed_entry was previously calling _parse_feed with a bare ?search_query=testing URL; switched to a fixed id_list=[1605.08386] so the test is deterministic and shares an existing golden file.astro-ph/0601001; the now-unused q_testing_s0_m10_df962dbeea2e.json fixture was removed.pytest) and live (pytest --live).feedparser is no longer a dependency. Anything reaching into Result._raw.* and expecting FeedParserDict attribute names (updated_parsed, arxiv_comment, etc.) will need to migrate to the new FeedEntry field names (updated as datetime, comment, journal_ref, doi, primary_category, categories).UnexpectedEmptyPageError.raw_feed is now arxiv._feed.ParsedFeed rather than feedparser.FeedParserDict.lxml>=5.0,<7.0.Result.Author.affiliation.README.md example usage has been updated. (No README changes needed \u2014 feedparser isn't referenced there, and the public Result API is unchanged except for the additive Author.affiliation field.)PR drafted by Shelley (Claude) on behalf of @lukasschwab.
", "content_text": "# Description\r\n\r\nReplaces `feedparser` with a small, namespace-aware lxml parser tailored to the arXiv API ([`arxiv/_feed.py`](./arxiv/_feed.py)).\r\n\r\n**Why:** arXiv's Atom feed declares three custom namespaces (`arxiv:`, `opensearch:`, `dc:`), and feedparser's lenient, schema-flattening model silently drops or collapses extension elements \u2014 most painfully the per-author `\n\nA clear and concise description of what the bug is.
\n
Author affiliations are available in raw arXiv API feeds, but are not exposed by this package's Result objects.
\n\nSteps to reproduce the behavior; ideally, include a code snippet.
\n
Apparent for any result set.
\n(Result)._raw.arxiv_affiliation is often defined, but it's a single string\u2013\u2013the affiliation of one author among several.\n\nA clear and concise description of what you expected to happen.
\n
Author affiliations should be exposed by the Result.Author class.
python version: *arxiv.py version: >= 1.0.0\n\nAdd any other context about the problem here.
\n
This is a long-open issue in feedparser, perhaps open since 2015: https://github.com/kurtmckee/feedparser/issues/24. There's a detailed breakdown of the interaction with arXiv results here: https://github.com/kurtmckee/feedparser/issues/145#issuecomment-821762233. I suspect arXiv will release their JSON API \u2013\u2013and this client library will be rewritten to use the JSON API\u2013\u2013before this feedparser bug is resolved.
This client library could expose the single author affiliation extracted by feedparser, but this has negative impacts:
If the single author affiliation is useful in your application, despite the noted downsides, access it with (Result)._raw.get('arxiv_affiliation').
when making a query with the length of id_list of 642 article names I get:
\nFile \"arxiv.py/arxiv/arxiv.py\", line 34, in query\n raise Exception(\"HTTP Error \" + str(results.get('status', 'no status')) + \" in query\")\nException: HTTP Error 414 in query
\nThis is probably due to some limit in the API. Does anybody know more about this?\nShould the library deal with this issue or is it more appropriate to leave it to the user?
", "content_text": "when making a query with the length of id_list of 642 article names I get:\r\n\r\nFile \"arxiv.py/arxiv/arxiv.py\", line 34, in query\r\n raise Exception(\"HTTP Error \" + str(results.get('status', 'no status')) + \" in query\")\r\nException: HTTP Error 414 in query\r\n\r\nThis is probably due to some limit in the API. Does anybody know more about this?\r\nShould the library deal with this issue or is it more appropriate to leave it to the user? ", "date_published": "2018-08-02T00:40:51Z", "date_modified": "2021-07-12T18:45:28Z", "tags": ["api"], "authors": [{"name": "luisberlioz", "url": "https://github.com/luisberlioz", "avatar": "https://avatars.githubusercontent.com/u/8170841?v=4"}]}]}