[FEATURE]: Browser Automation using AI

[surapuramakhil]

Feature summary

As of now AI is used for generating answers to the questions

Feature description

proposal:- AI to use as browser agent - viewing and acting directly on UI, performing job Application.

Motivation

No response

Alternatives considered

No response

Additional context

No response

[cjbbb]

After using this, can we apply for jobs on other job websites?

[surapuramakhil]

After using this, can we apply for jobs on other job websites?

cache is it's tough to implement 😅
@49Simon kindly comment on possibility - existing library support? I guess we send feed to AI (UI of browser)

[49Simon]

This would be something similar to Claude's Computer Use. Its early in development phase but by using tool use and vision models, it iteratively sends screenshots back to the model until the task is complete.

[feder-cr]

@surapuramakhil @cjbbb in this moment this tools not work correctly, also https://github.com/sentient-engineering/jobber isn't good, but if you want you can try!

[cjbbb]

@surapuramakhil @cjbbb in this moment this tools not work correctly, also https://github.com/sentient-engineering/jobber isn't good, but if you want you can try!

I will check the link

[surapuramakhil]

Another tool/lib bowser pilot which does same thing - https://github.com/handrew/browserpilot, once matured we can start using in this project.

[jfelten]

I am interested in pitching in. I went through the AIHawk code and have some ideas on how to use the existing classes to make a flexible architecture that allows for other sites. By this, I mean extending the current job_manager and LinkedIn_easy_applier classes to work for other sire like Amazon. I am the one who brought up browserpilot and would want to see if I could make it work. I wanted to see if anyone else has proposed similar ideas.

[surapuramakhil]

would want to see if I could make it work.

Appreciate that - Assigned this to you.

I wanted to see if anyone else has proposed similar ideas.

that was me, #401 found that it was never reopened.

If you wish I can also assign 401 for you. @jfelten

[surapuramakhil]

another project - https://github.com/gregpr07/browser-use

[feder-cr]

@jfelten @sarob @cjbbb try also to contact/work with @gregpr07

[feder-cr]

@jfelten https://github.com/Skyvern-AI/skyvern

[surapuramakhil]

@jfelten https://github.com/Skyvern-AI/skyvern

This one seems to be quite mature.

[sarob]

Looks very promising. Good find!Sean ***@***.*** On Nov 24, 2024, at 04:38, Akhil ***@***.***> wrote: @jfelten https://github.com/Skyvern-AI/skyvern This one seems to be quite mature. —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: ***@***.***>

[surapuramakhil]

Looks very promising. Good find!

@sarob what your thoughts on importing are lib / modules which are on AGPL? (Their strong copyleft) restrict our ability to update our own licenses in future?

Another is it's an end-to-end system; I hope they are happy / expanding it as lib - something like Lang chain (which is a library) or any other library.

[sarob]

AGPL supersedes GPL and MIT if their code is upstream. They have a generic job application capability already. browser-use is MIT, but less developed.

[suchintan]

Maintainer of https://github.com/Skyvern-AI/Skyvern here -- would love to see this integrated (thanks for messaging me @feder-cr!

Skyvern is an API-first product, so the easiest way to get integrated would be to use our API directly:

This works on most job boards. You just need to change the payload (ie users' information) and job URL. When changing the payload, we just need to make sure Skyvern is "oversupplied" information so it can pull from it as needed.

curl 'https://api.skyvern.com/api/v1/tasks' -X POST  // This can also be the self-hosted version
-H "Content-Type: application/json" 
-H "x-api-key: xxx" // get your own API key
--data-binary '
  {
   "url":"https://jobs.lever.co/leverdemo-8/45d39614-464a-4b62-a5cd-8683ce4fb80a/apply", 
   "webhook_callback_url":null, // Specify your own webhook here
   "navigation_goal":"Fill out the job application form and apply to the job. Fill out any public burden questions if they appear in the form. Your goal is complete when the page says you'\''ve successfully applied to the job. Terminate if you are unable to apply successfully.",
  "proxy_location":"RESIDENTIAL",
  "navigation_payload":"{\n  \"name\": \"John Doe\",\n  \"email\": \"[email protected]\",\n  \"phone\": \"6421440771\",\n  \"resume_url\": \"https://writing.colostate.edu/guides/documents/resume/functionalSample.pdf\",\n  \"cover_letter\": \"Generate a compelling cover letter for me\"\n}"
  }
'

[surapuramakhil]

Maintainer of https://github.com/Skyvern-AI/Skyvern here -- would love to see this integrated (thanks for messaging me @feder-cr!

Awesome - if you are discord can you join our discord community https://discord.gg/mMZcMTH9K6 and ping me once you are there? we might need much more flexibility, Glad to see both projects working together.

[sarob]

picking this up.

#967 is working on a similar purpose fix

[sameelarif]

Would definitely check out Stagehand for this: https://github.com/browserbase/stagehand