Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make it possible to increase the amount of memory assigned to the plaintext importer job #1518

Open
marekhorst opened this issue Jan 10, 2025 · 0 comments
Assignees

Comments

@marekhorst
Copy link
Member

The cache_builder workflow responsible for rebuilding PMC ingester cache from scratch failed with the following error message:

Container [pid=22105,containerID=container_e78_1715771856403_155795_02_000001] is running beyond physical memory limits. Current usage: 1.1 GB of 1 GB physical memory used; 2.8 GB of 2.1 GB virtual memory used. Killing container.

Job name:

oozie:action:T=map-reduce:W=importer_plaintext:A=content-retriever:ID=0004585-241122080234071-oozie-oozi-W

indicates plaintext importer phase has failed.

This is kind of similar to other recently introduced fix, #953, and should be solved in a similar way by increasing the amount of memory assigned to this job.

We could start with doubling the default value so increasing it to 2g.

@marekhorst marekhorst self-assigned this Jan 10, 2025
marekhorst added a commit that referenced this issue Jan 10, 2025
…ned to the plaintext importer job

Introducing `plaintext_content_retriever_java_opts` input parameter set to `-Xmx2g` value by default.
marekhorst added a commit that referenced this issue Jan 14, 2025
…ned to the plaintext importer job

Replacing `plaintext_content_retriever_java_opts` input parameter with `plaintext_content_retriever_appmaster_memory_mb` set to `2048` value by default.
After further analysis it turned out the problem was related to AM insufficient memory, not the mapper memory.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant