Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix stack overflow when highlighting large source files #1277

Merged
merged 5 commits into from
Jan 16, 2025
Merged

Fix stack overflow when highlighting large source files #1277

merged 5 commits into from
Jan 16, 2025

Conversation

Julow
Copy link
Collaborator

@Julow Julow commented Jan 16, 2025

This is enough to highlight reason_parser.ml in reason.3.8.0, which is one very large file that Odoc has to process.

jonludlam and others added 5 commits January 16, 2025 11:09
@jonludlam @Julow
Make some function tail recursive
d3c8c0e 
When rendering source for large (~1M) source files, some List.map and
List.concat functions were running out of stack space. This commit converts
replaces them with tail-recursive equivalents.
@Julow
Driver: Collect stderr of failing commands
26b48cf 
Improve the output in case of an error and defer the failure to outside
of the Eio context to avoid the logs being lost.
@Julow
Utils: Vendor List.concat_map
d798571 
The function is taken from OCaml 4.14.2 and is tail-recursive. The label
is removed to make the code closer to idiomatic OCaml.
@Julow
Html: Fix stack overflow computing source code spans
459acd2
@Julow
Update CHANGES
3f11a66
@jonludlam
Copy link
Member

Thanks!

@jonludlam jonludlam merged commit 065dc72 into ocaml:master Jan 16, 2025
13 of 14 checks passed
jonludlam added a commit to jonludlam/opam-repository that referenced this pull request Jan 23, 2025
@jonludlam
[new release] odoc (6 packages) (3.0.0~beta1)
1bbe221 
CHANGES:

### Highlight

- Hierarchical documentation (@jonludlam, @panglesd, @Julow)
  Pages can now be organized in a directory tree structure.
  Relative and absolute references are added:
  `{!./other_page.label}`, `{!//other_page}`.

- Improved sidebar and breadcrumbs navigation (@panglesd, @gpetiot)
  The documentation pages and the libraries of the entire package are shown on
  the left sidebar.

- Added support for images, videos, audio and other assets
  The syntax is `{image!/reference/to/asset}` or `{image:URL}` for images.
  The syntax for `{video...}` and `{audio...}` is the same.
  (@panglesd, @EmileTrotignon, ocaml/odoc#1170, ocaml/odoc#1171, ocaml/odoc#1184, ocaml/odoc#1185)

- Search using Sherlodoc (@panglesd, @EmileTrotignon, @Julow)
  A new search bar that supports full-text and type-based search.

### Added

- Experimental driver (@jonludlam, @panglesd)
  The driver builds the documentation for a collection of Opam packages using
  the newer Odoc features. It supports linking external packages to ocaml.org
  and markdown files.
  This is experimental and will break in the future.

- Cross-package references (@panglesd, @Julow)
  Pages and modules from other packages can be referenced:
  `{!/otherpackage/page}`, `{!/otherpackage/Module.t}`.

- Option to remap links to other packages to ocaml.org or other site.
  See the `--remap` option of the driver or the `--remap-file` option of `odoc html-generate`.
  (@jonludlam, ocaml/odoc#1189, ocaml/odoc#1248)

- Option to compute occurrences of use of each identifiers
  The commands `aggregate-occurrences` and `count-occurrences` are added.
  (@panglesd, ocaml/odoc#976, ocaml/odoc#1076, ocaml/odoc#1206)

- Added the `odoc classify` command (@jonludlam, ocaml/odoc#1121)
  Helps driver detecting which modules belong to which libraries.
- Added `--suppress-warnings` to the CLI to remove warnings from a unit, even
  if they end up being raised in another unit through expansion
  (@jonludlam, ocaml/odoc#1260)
- Add clock emoji before `@since` tag (@yawaramin, ocaml/odoc#1089)
- Navigation for the search bar : use '/' to enter search, up and down arrows to
  select a result, and enter to follow the selected link. (@EmileTrotignon, ocaml/odoc#1088)
- Fix a big gap between the preamble and the content of a page (@EmileTrotignon, ocaml/odoc#1147)
- Add a marshalled search index consumable by sherlodoc (@EmileTrotignon, @panglesd, ocaml/odoc#1084)
- Allow referencing of polymorphic constructors in polymorphic variant type
  aliases (@panglesd, ocaml/odoc#1115)
- Added a home icon in the breacrumbs (@panglesd, ocaml/odoc#1251)
  It can be disabled with a CLI option.
- Add a frontmatter syntax for mld pages (@panglesd, ocaml/odoc#1187, ocaml/odoc#1193, ocaml/odoc#1243, ocaml/odoc#1246, ocaml/odoc#1251)
  Allows to specify the title of a page, the order of sub-pages and other
  behaviors in the sidebar.
- Added `odoc-md` to process standalone Markdown pages (@jonludlam, ocaml/odoc#1234)

### Changed

- The command line interface changed to support the new features.
  + Packages and libraries: `odoc link` must now be aware of packages and
    libraries with the `-L libname:path` and `-P pkgname:path` options. The
    module search path should still be passed with the `-I` option.
    The current package should be specified with `--current-package=pkgname`.
  + Hierarchy: `odoc compile` now outputs `.odoc` in the directory tree
    specified with `--output-dir=DIR` and the parent identifier must be
    specified with `--parent-id=PARENT`.
    The option `--source-parent-file` is removed.
  + Source code: Implementations are compiled with `compile-impl` instead of
    with `compile`. The options `--cmt=..` and `--source-name=..` are removed.
    Source code pages are generated with `html-generate-source`.
  + Assets: The commands `compile-asset`, `html-generate-asset` are added.
    The option `html-generate --asset` is removed.
  + Sidebar: The index is built using `compile-index`. The sidebar data is
    extracted from the index with `sidebar-generate` and passed to
    `html-generate --sidebar=..`.

- The syntax for `@tag` is now delimited (@panglesd, ocaml/odoc#1239)
  A `@tag` can now be followed by a paragraph or other elements.

- Updated colors for code fragments (@EmileTrotignon, ocaml/odoc#1023)
- Fixed complexity of looking up `.odoc` files (@panglesd, ocaml/odoc#1075)
- Normalize whitespaces in codespans (@gpetiot, ocaml/odoc#1085)
  A newline followed by any whitespaces is normalized as one space character.
- Reduce size of `Odoc_html_frontend` when compiled to javascript
  (@EmileTrotignon, ocaml/odoc#1072)
- Overhaul of module-type-of expansions and shadowing code (@jonludlam, ocaml/odoc#1081)
- Output file paths and labels in the man and latex backends changed to avoid
  name clashes (@Julow, ocaml/odoc#1191)

### Fixed

- Fix variant constructors being hidden if they contain hidden types
  (@jonludlam, ocaml/odoc#1105)
- Fix rare assertion failure due to optional parameters
  (@jonludlam, ocaml/odoc#1272, issue ocaml/odoc#1001)
- Fix resolution of module synopses in {!modules} lists that require --open
  (@jonludlam, ocaml/odoc#1104}
- Fix top comment not being taken from includes often enough (@panglesd, ocaml/odoc#1117)
- Fixed 404 links from search results (@panglesd, ocaml/odoc#1108)
- Fixed title content not being picked up across pages when rendering references
  (ocaml/odoc#1116, @panglesd)
- Fix wrong links to standalone comments in search results (ocaml/odoc#1118, @panglesd)
- Remove duplicated or unwanted comments with inline includes (@Julow, ocaml/odoc#1133)
- Fix bug where source rendering would cause odoc to fail completely if it
  encounters invalid syntax (@jonludlam ocaml/odoc#1208)
- Add missing parentheses in 'val (let*) : ...' (@Julow, ocaml/odoc#1268)
- Fix syntax highlighting not working for very large files
  (@jonludlam, @Julow, ocaml/odoc#1277)
jonludlam added a commit to jonludlam/opam-repository that referenced this pull request Jan 23, 2025
@jonludlam
[new release] odoc (5 packages) (3.0.0~beta1)
0ffc6db 
CHANGES:

- Hierarchical documentation (@jonludlam, @panglesd, @Julow)
  Pages can now be organized in a directory tree structure.
  Relative and absolute references are added:
  `{!./other_page.label}`, `{!//other_page}`.

- Improved sidebar and breadcrumbs navigation (@panglesd, @gpetiot)
  The documentation pages and the libraries of the entire package are shown on
  the left sidebar.

- Added support for images, videos, audio and other assets
  The syntax is `{image!/reference/to/asset}` or `{image:URL}` for images.
  The syntax for `{video...}` and `{audio...}` is the same.
  (@panglesd, @EmileTrotignon, ocaml/odoc#1170, ocaml/odoc#1171, ocaml/odoc#1184, ocaml/odoc#1185)

- Search using Sherlodoc (@panglesd, @EmileTrotignon, @Julow)
  A new search bar that supports full-text and type-based search.

- Experimental driver (@jonludlam, @panglesd)
  The driver builds the documentation for a collection of Opam packages using
  the newer Odoc features. It supports linking external packages to ocaml.org
  and markdown files.
  This is experimental and will break in the future.

- Cross-package references (@panglesd, @Julow)
  Pages and modules from other packages can be referenced:
  `{!/otherpackage/page}`, `{!/otherpackage/Module.t}`.

- Option to remap links to other packages to ocaml.org or other site.
  See the `--remap` option of the driver or the `--remap-file` option of `odoc html-generate`.
  (@jonludlam, ocaml/odoc#1189, ocaml/odoc#1248)

- Option to compute occurrences of use of each identifiers
  The commands `aggregate-occurrences` and `count-occurrences` are added.
  (@panglesd, ocaml/odoc#976, ocaml/odoc#1076, ocaml/odoc#1206)

- Added the `odoc classify` command (@jonludlam, ocaml/odoc#1121)
  Helps driver detecting which modules belong to which libraries.
- Added `--suppress-warnings` to the CLI to remove warnings from a unit, even
  if they end up being raised in another unit through expansion
  (@jonludlam, ocaml/odoc#1260)
- Add clock emoji before `@since` tag (@yawaramin, ocaml/odoc#1089)
- Navigation for the search bar : use '/' to enter search, up and down arrows to
  select a result, and enter to follow the selected link. (@EmileTrotignon, ocaml/odoc#1088)
- Fix a big gap between the preamble and the content of a page (@EmileTrotignon, ocaml/odoc#1147)
- Add a marshalled search index consumable by sherlodoc (@EmileTrotignon, @panglesd, ocaml/odoc#1084)
- Allow referencing of polymorphic constructors in polymorphic variant type
  aliases (@panglesd, ocaml/odoc#1115)
- Added a home icon in the breacrumbs (@panglesd, ocaml/odoc#1251)
  It can be disabled with a CLI option.
- Add a frontmatter syntax for mld pages (@panglesd, ocaml/odoc#1187, ocaml/odoc#1193, ocaml/odoc#1243, ocaml/odoc#1246, ocaml/odoc#1251)
  Allows to specify the title of a page, the order of sub-pages and other
  behaviors in the sidebar.
- Added `odoc-md` to process standalone Markdown pages (@jonludlam, ocaml/odoc#1234)

- The command line interface changed to support the new features.
  + Packages and libraries: `odoc link` must now be aware of packages and
    libraries with the `-L libname:path` and `-P pkgname:path` options. The
    module search path should still be passed with the `-I` option.
    The current package should be specified with `--current-package=pkgname`.
  + Hierarchy: `odoc compile` now outputs `.odoc` in the directory tree
    specified with `--output-dir=DIR` and the parent identifier must be
    specified with `--parent-id=PARENT`.
    The option `--source-parent-file` is removed.
  + Source code: Implementations are compiled with `compile-impl` instead of
    with `compile`. The options `--cmt=..` and `--source-name=..` are removed.
    Source code pages are generated with `html-generate-source`.
  + Assets: The commands `compile-asset`, `html-generate-asset` are added.
    The option `html-generate --asset` is removed.
  + Sidebar: The index is built using `compile-index`. The sidebar data is
    extracted from the index with `sidebar-generate` and passed to
    `html-generate --sidebar=..`.

- The syntax for `@tag` is now delimited (@panglesd, ocaml/odoc#1239)
  A `@tag` can now be followed by a paragraph or other elements.

- Updated colors for code fragments (@EmileTrotignon, ocaml/odoc#1023)
- Fixed complexity of looking up `.odoc` files (@panglesd, ocaml/odoc#1075)
- Normalize whitespaces in codespans (@gpetiot, ocaml/odoc#1085)
  A newline followed by any whitespaces is normalized as one space character.
- Reduce size of `Odoc_html_frontend` when compiled to javascript
  (@EmileTrotignon, ocaml/odoc#1072)
- Overhaul of module-type-of expansions and shadowing code (@jonludlam, ocaml/odoc#1081)
- Output file paths and labels in the man and latex backends changed to avoid
  name clashes (@Julow, ocaml/odoc#1191)

- Fix variant constructors being hidden if they contain hidden types
  (@jonludlam, ocaml/odoc#1105)
- Fix rare assertion failure due to optional parameters
  (@jonludlam, ocaml/odoc#1272, issue ocaml/odoc#1001)
- Fix resolution of module synopses in {!modules} lists that require --open
  (@jonludlam, ocaml/odoc#1104}
- Fix top comment not being taken from includes often enough (@panglesd, ocaml/odoc#1117)
- Fixed 404 links from search results (@panglesd, ocaml/odoc#1108)
- Fixed title content not being picked up across pages when rendering references
  (ocaml/odoc#1116, @panglesd)
- Fix wrong links to standalone comments in search results (ocaml/odoc#1118, @panglesd)
- Remove duplicated or unwanted comments with inline includes (@Julow, ocaml/odoc#1133)
- Fix bug where source rendering would cause odoc to fail completely if it
  encounters invalid syntax (@jonludlam ocaml/odoc#1208)
- Add missing parentheses in 'val (let*) : ...' (@Julow, ocaml/odoc#1268)
- Fix syntax highlighting not working for very large files
  (@jonludlam, @Julow, ocaml/odoc#1277)
@jonludlam jonludlam mentioned this pull request Jan 23, 2025
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@Julow @jonludlam