Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

First stab at chromote support #362

Merged
merged 56 commits into from
Feb 1, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
56 commits
Select commit Hold shift + click to select a range
1159580
First stab at chromote support
hadley Nov 15, 2022
06411d9
Merged origin/main into chromote
hadley Nov 15, 2022
bbcad1f
Fix tests
hadley Nov 15, 2022
5193896
More robust way to wait
hadley Nov 15, 2022
2b24002
Test and tweak print method
hadley Nov 15, 2022
602dfad
More fixture polishing
hadley Nov 15, 2022
7902416
Test (and fix) attribute retrieval
hadley Nov 15, 2022
f43ede4
Test handling of missing elements
hadley Nov 15, 2022
3e1d843
More robust on recovery
hadley Nov 16, 2022
dd06874
Add missing function
hadley Nov 16, 2022
3261b4b
Always try twice
hadley Nov 16, 2022
3b31e32
Simplify code
hadley Nov 16, 2022
604e235
WIP
hadley Nov 23, 2022
4fa59a1
Grab HTML eagerly
hadley Jan 10, 2023
18001fe
Add some basic examples
hadley Jan 11, 2023
c294276
Merged origin/main into chromote
hadley Jan 11, 2023
b35c696
Simplify clean up hack
hadley Jan 11, 2023
a65b96b
Convert to R6
hadley Jan 11, 2023
2bf6706
Implement click
hadley Jan 11, 2023
3866d58
Polish hack
hadley Jan 11, 2023
2272615
Implement double click
hadley Jan 11, 2023
7f58184
Initial scrolling support
hadley Jan 12, 2023
f8c6308
Refactoring
hadley Jan 12, 2023
d9c82b9
Move back to root note caching
hadley Jan 12, 2023
616f67e
Get scrolling working
hadley Jan 12, 2023
ec97ac4
Use private methods
hadley Jan 16, 2023
a28654e
Rename class
hadley Jan 16, 2023
ee9685f
Rename files
hadley Jan 16, 2023
fc8490a
Document LiveHTML object
hadley Jan 16, 2023
6260b7a
Try using mouseWheel for scrolling
hadley Jan 16, 2023
48a183e
Merged origin/main into chromote
hadley Jan 23, 2024
72ba563
Merged origin/main into chromote
hadley Jan 23, 2024
4f3d6ea
Polish docs
hadley Jan 25, 2024
dcf2087
Fix scroll_in_to_view
hadley Jan 25, 2024
3e91eb3
Get user-simulated click working
hadley Jan 25, 2024
6286baa
Override user agent
hadley Jan 26, 2024
f997620
Add html_table method
hadley Jan 26, 2024
84769bf
Add type and press methods
hadley Jan 26, 2024
9fe933a
Search with xpath
hadley Jan 26, 2024
7cdad68
Fix function name
hadley Jan 30, 2024
1e2796b
Redocument
hadley Jan 30, 2024
1ded591
Update spelling
hadley Jan 30, 2024
a7d02ce
Update reference page
hadley Jan 30, 2024
b2a9178
Tweak examples
hadley Jan 30, 2024
baa51a1
Oops
hadley Jan 30, 2024
3002bee
Fix pkgdown site
hadley Jan 30, 2024
1c96b52
Refactor live testing
hadley Feb 1, 2024
29e8d77
Test button clicking and fix bug thus revealed
hadley Feb 1, 2024
53e1424
Test scrolling
hadley Feb 1, 2024
397e03c
Remove no-longer used call_method method
hadley Feb 1, 2024
6d6fe05
Test pressing and typing
hadley Feb 1, 2024
b4a3940
Polish docs
hadley Feb 1, 2024
5d43eec
Use local page instead of hosted
hadley Feb 1, 2024
2c56755
Test tables and xpath
hadley Feb 1, 2024
f8a8a2d
Bump up scroll wait
hadley Feb 1, 2024
5bf52ca
Add news bullet
hadley Feb 1, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .Rbuildignore
Original file line number Diff line number Diff line change
Expand Up @@ -14,3 +14,4 @@
^LICENSE\.md$
^vignettes/articles$
^CRAN-SUBMISSION$
^data-raw$
3 changes: 2 additions & 1 deletion DESCRIPTION
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@ Imports:
tibble,
xml2 (>= 1.3)
Suggests:
chromote,
covr,
knitr,
R6,
Expand All @@ -41,4 +42,4 @@ Config/testthat/parallel: true
Encoding: UTF-8
Language: en-US
Roxygen: list(markdown = TRUE)
RoxygenNote: 7.3.0
RoxygenNote: 7.3.1
5 changes: 5 additions & 0 deletions NAMESPACE
Original file line number Diff line number Diff line change
Expand Up @@ -3,14 +3,17 @@
S3method(cookies,rvest_session)
S3method(format,rvest_field)
S3method(headers,rvest_session)
S3method(html_element,LiveHTML)
S3method(html_element,default)
S3method(html_element,rvest_session)
S3method(html_elements,LiveHTML)
S3method(html_elements,default)
S3method(html_elements,rvest_session)
S3method(html_form,rvest_session)
S3method(html_form,xml_document)
S3method(html_form,xml_node)
S3method(html_form,xml_nodeset)
S3method(html_table,LiveHTML)
S3method(html_table,rvest_session)
S3method(html_table,xml_document)
S3method(html_table,xml_node)
Expand All @@ -25,6 +28,7 @@ S3method(print,rvest_session)
S3method(read_html,rvest_session)
S3method(status_code,rvest_session)
export("%>%")
export(LiveHTML)
export(back)
export(follow_link)
export(forward)
Expand All @@ -50,6 +54,7 @@ export(is.session)
export(jump_to)
export(minimal_html)
export(read_html)
export(read_html_live)
export(repair_encoding)
export(session)
export(session_back)
Expand Down
5 changes: 5 additions & 0 deletions NEWS.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,10 @@
# rvest (development version)

* New `read_html_live()` reads HTML into a real, live, HTML browser, meaning
that you can scrape HTML generated by javascript. It returns a `LiveHTML`
object which you can also use to simulate user interactions with the page,
like clicking, typing, and scrolling (#245).

* `html_table()` discards rows without cells (@epiben, #360).

# rvest 1.0.3
Expand Down
Loading
Loading