-
Notifications
You must be signed in to change notification settings - Fork 21
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[AI] Error #2
Comments
Thank you! It seems like it's a problem when the code calls the OpenAI API. Did you configure your OpenAI API Key in the plugin settings? Are the API keys still valid in OpenAI? And are you still under your rate limits? You can check it here |
That's strange, I am far below my rate limits, and I am using my API key with no issue on many other services. |
So at the moment, the error handling is quite aggressive in the fact that it stops processing if any one document has an error. It could be that the error is because the document is being parsed incorrectly, has some invalid characters, goes over the embedding API token limit or something like that. You said some of the other documents parsed fine and were populated into the supabase, am I right? If so, what happens if you add the Tweets directory to the excluded directories path (it's in the settings panel)? Does the error go away or change? It you don't mind, you can also email the problematic markdown file that is throwing the error in the log to me @ [email protected] and I can see if I can replicate it on my end. |
You were right! One single doc was crashing the indexing. I sent it to you by mail. |
Wonderful glad it works, I'll look into the problematic file. In the meantime I've patched the code to not be so strict on errors. Now, it continues processing the other files even if a file has an error in processing. And then it reports the problem via a UI notice for more feedback. I'll try to find ways to provide more info to users during the indexing process too. The patched release is here: https://github.com/solderneer/obsidian-ai-tools/releases/tag/1.1.4 |
Ah, I've looked into the problematic file. It turns out, it's because the context length limit for the embedding API endpoint is 8000 ish tokens, and that file exceeds it. Normally, the documents are separated into paragraphs, which prevents them from exceeding the context limit but that file has no paragraphs at all. The issue can be resolved by adding a new line in the file that separates the context into two chunks, which are sent to the API as 2 chunks. I'm reopening the issue so that I remember to add the code to automatically handle that, in the next patch. In the meantime, if you just add a newline in the middle of that file, everything should work without any errors. |
Looking into what's causing the errors to throw across my vault now. Do you log anything in the console for documents that were stored properly? It might be easier for me to track things down if I can find some working examples. |
The errors are logged to the console. Valid documents used to be logged to the documents until As an alternative, you can also just open up the supabase instance in Table View and see all the documents that were stored. The error in this issue is caused by files that are too long without paragraphing, so just need to break up the file into two or more. |
I found the root cause of my issue which was—without surprise—not following the setup instructions closely enough. Everything's working great now. Great work. |
It's always the setup instruction's fault for not being simple/clear enough 😄 . Which step tripped you up, maybe there's a way to rewrite the instructions to be easier to follow? |
Haha, the instructions were clear. The part that tripped me up was not reading them before I began trying to configure everything. Specifically, I didn't run the command to create the necessary tables in Supabase before I started the command to create the embeddings. 😅
|
I set up the plugin strictly following the instructions (using supabase CLI and downloading the last release from github).
The Table Editor shows the two tables
document
(7 records) anddocument_section
(30 records).When I run the
AI Tools: Refresh Index
, after a few seconds showing the🔮 [AI] Indexing ...
indicator, it finally switches to😔 [AI] Error
.I have no idea what went wrong.
The text was updated successfully, but these errors were encountered: