extract rules for using LLM, and use it for non-ai

[AzizNadirov]

Hi, can I extract information from source(crawling) by providing model(product_name, price etc), get result, and extract its css schema, for reusing for other pages in nonai - css manner ?

[aravindkarnam]

@AzizNadirov Let me try to unpack your question here. You expect to provide the product_name, price etc and you expect the model/algorithm to map the html id/class name etc of the divs to these specific data fields.
Then in the next go you expect the crawler to extract the data based on the html/styling tags and return it? Am I understanding this correctly?

[AzizNadirov]

Yes, exactly.

[aravindkarnam]

@AzizNadirov To best of my knowledge, we currently don't have a feature like this. We will certainly keep this use case in mind while planning our future roadmap.

However you can get the raw html from the crawler result using result.html then have a model(like chatGPT or Claude) to workout the mapping between classes/id/name etc attributes of divs vs desired data fields. Then you can extract using the help of JsonCssExtractionStrategy or JsonXPathExtractionStrategy .

You can find some useful examples here

Cc: @unclecode Interesting use case ☝🏼

[AzizNadirov]

Thanks, now I am working on it.

[unclecode]

@AzizNadirov Actually, there’s a helper static function in the JsonCss and JsonXpath extraction strategies. You pass the raw HTML, and it returns the schema, so you only need to call the LLM once, voilà! I’ve been testing it for a while and might merge it into the main branch in the next version.

@aravindkarnam